Multi-Index Grading Method for Pear Appearance Quality Based on Machine Vision

Abstract

The appearance quality of fruits affects consumers’ judgment of their value, and grading the quality of fruits is an effective means to improve their added value. The purpose of this study is to transform the grading of pear appearance quality into the classification of the categories under several quality indexes based on industry standards and design effective distinguishing features for training the classifier. The grading of pear appearance quality is transformed into the classification of pear shapes, surface colors and defects. The symmetry feature and quasi-rectangle feature were designed and the back propagation (BP) neural network was trained to distinguish standard shape, apical shape and eccentric shape. The mean and variance features of R and G channels were used to train support vector machine (SVM) to distinguish standard color and deviant color. The surface defect area was used to participate in pear appearance quality classification and the gray level co-occurrence matrix (GLCM) features of defect area were extracted to train BP neural network to distinguish four common defect categories: tabbed defects, bruised defects, abraded defects and rusty defects. The accuracy rates of the above three classifiers reached 83.3%, 91.0% and 76.6% respectively, and the accuracy rate of pear appearance quality grading based on grading rules was 80.5%. In addition, the hardware system prototype for experimental purpose was designed, which have certain reference significance for the further construction of the pear appearance quality grading pipeline.

1. Introduction

Manual fruit sorting has the disadvantages of low efficiency, high labor intensity and strong subjectivity in grading. It is of great significance to study the technology of fruit appearance quality detection and automatic grading to overcome the above disadvantages.

At present, fruit quality detection mainly focuses on fruit appearance (such as fruit shape, color, bruises, ripeness, etc.), internal defects (such as browning, cork, etc.) and internal components (such as soluble solids, brix, etc.), and the classification of fruits is carried out according to specific standards. Fruit quality detection techniques mainly include machine vision technology [1,2], spectral detection technology [3,4], X-ray detection technology [5,6], electronic nose detection technology [7], and nuclear magnetic resonance detection technology [8]. A comparison of the advantages and disadvantages of various detection techniques and the applicable occasions are shown in

Table 1. Comparison of fruit quality detection technology.

Detection Technology	Detection Object	Advantage	Disadvantage
Machine vision technology	Fruit size, shape, external damage, surface defects	Low cost, online inspection of production line	It is difficult to detect the entire surface of the fruit, and it is impossible to identify internal defects and components.
Spectroscopy	Surface and internal defects, composition	Detection information is richer than machine vision	The multi band spectral information needs to be screened, the data volume is large, and the processing time is long.
X-ray imaging technology	Internal defects, composition	Sensitive to bulk defects, can detect internal defects and components at the same time, especially distinguish between different densities	It is not sensitive to surface defects, the radiation source is expensive, and there is radiation risk.
Electronic nose technology	Internal composition, maturity	Evaluate the quality of fruit by its odor signal, easy to handle and fast to detect	It is easily affected by the environment, and the detection accuracy depends on the parameters of the gas sensor.
Nuclear Magnetic Resonance Technology	Surface and internal defects, composition	Strong penetrating power, not limited by peel thickness, safe and non-radiation	NMR instruments are expensive and time consuming to process.

. For fruit appearance quality detection, machine vision technology is the most direct, feasible and economical way.

Shi et al., used an image-based approach to classify the quality of apple appearance, and proposed a new multi-view spatial network to distinguish the size of apples and the presence of defects. The low-level features were extracted from the top-view, front-view and back-view images of apples which were obtained from each of the three cameras using three lightweight networks. The information related to multiple views was fused by the spatial feature aggregation module and the final classification accuracy rate reached 99.23% [9]. Sun et al., used the class balance loss to improve the MobileNet V2 model for jujube maturity classification, and used the transfer learning strategy to train the model. After the training, the model was fine-tuned using an imbalance data set. Compared with other networks, the classification accuracy of this model was higher, reaching 99.294% [10]. Nandi et al., extracted global Fourier descriptors of mango fruit shape for mango quality discrimination and used SVM to achieve the classification of mango fruit shape in good, medium and poor grades [11]. In addition, the overall or local color of some fruits can reflect their ripeness and defects. Rafeal et al. used color features to predict olive ripeness and defect recognition. This scheme was feasible when olives were less ripe, but olives with higher ripeness show black-purple color, which caused low accuracy of olive surface defect recognition [12]. And then other features need to be integrated for olive surface defect recognition. Kumar et al. used wavelet transform method to segment tomato defect areas and identified black spots, rotten spots and melanistic spots based on color and geometric features, with an accuracy of 98% [13]. Gu et al., combined the features extracted by the deep convolutional neural network (DCNN) with the k-nearest neighbor to distinguish the surface defect categories of pears and apples. They used ImageNet to pre-train the DCNN, and used the fruit surface defect data set for network fine-tuning to extract features. The extracted features were used to identify fruit surface defects by k-nearest neighbor, with an accuracy rate of 99.78% [14]. In aspect of fruit appearance quality detection, the relatively single grading index in the current study had insufficient responsiveness to overall fruit appearance quality. The fruit industry standard described the quality requirements of all aspects of fruit in a more comprehensive way, which can be referred to in the fruit classification. The fruit grading indexes in the industry standard mainly revolve around the fruit shape, fruit surface color and the size of the defects. In this paper, these three grading indexes were also used in the grading process of pear appearance quality. The final appearance quality of the pear is determined comprehensively according to the results of the above three grading indexes, which can reflect the appearance quality of the fruit in a more comprehensive way.

Sofu et al. designed an apple visual inspection and grading system, which can detect surface color spots, defects, crusts, stalks and calyxes of apples. In addition, apple size and weight were also used to grade apples, thus achieving high sorting efficiency [15]. Mon et al. proposed an algorithm for mango volume estimation based on machine vision technology. The length and width of the mango were obtained from the 2D mango image, and the thickness was estimated from the correlation between the light intensity distribution and the maximum width-thickness in the top view of the mango, so as to estimate the volume of mango, with an accuracy of 96.8% [16]. In the field of agricultural fruit grading and sorting, it is a trend to comprehensively grade fruits by integrating various evaluation indexes, which has richer grading connotation. Machine vision-based automatic fruit grading systems generally include a controller, an image acquisition device, a transfer device, a grading device and a human-machine interaction module [17]. The controller is responsible for the overall device operation control. The image acquisition device is usually installed with charge-coupled device (CCD) and light source system components to acquire the fruit appearance images. The conveyor and grading device are used to transport the fruits and sort them correctly, and the human-machine interaction module is used to realize the functions of setting equipment parameters and control command input.

In this paper, pears are employed as grading objects, whose variety is Xinjiang Kulle balsam pear. The shape, color and surface defects of the pear were used as grading indexes to visually detect the appearance quality of the pear. The content of this paper is organized as follows: Section 2 illustrates the pear appearance feature extraction and grading method with pear shape, color and surface defects as grading indexes. Section 3 is the experimental section, which introduces the designed hardware device and grading software, explains the involved image processing methods, followed by demonstrating the feasibility of the extracted features, and finally explains and evaluates the classification results. At last, a summary as well as an outlook is presented in Section 4.

2. Methods

Pear appearance quality grading refers to the national grading standard of fresh pears in China. The pear appearance quality indexes involved in the grading standard mainly include pear shape, surface color and defects. Abnormal pear shapes such as apical shape and eccentric shape, which are common in pear shapes, reduce the appearance quality of pears. The poor surface color and defects directly affect consumer behavior. The following will introduce the designed features corresponding to each appearance quality index and the pear appearance quality grading method.

2.1. Pear Feature Extraction

2.1.1. Pear Shape Feature Extraction

The proposed pear shape categories are standard shape, apical shape and eccentric shape, and the corresponding pear fruit shape images are shown in Figure 1. The symmetry feature and the quasi-rectangularity feature can distinguish the above three pear shapes. In Figure 2, the pear contour and the geometric outer rectangle $ABCD$ have been drawn with red lines, and the straight line $a$ is the midline of $AB$ side. In the image coordinate system $OXY$, the absolute value of the deviation of the pair of pixel points with the same vertical coordinate on the left and right half of the pear contour from the distance of the straight line $a$ can reflect the symmetry of the pear contour, if there are more pixel points with the same vertical coordinate on the left and right half of pear contour, we take the pixel points with the largest distance from the straight line $a$ for the calculation of the deviation, and the calculation is shown in Formula (1). In Figure 3, $OXY$ is the image coordinate system. The pear contour as well as the geometric outer rectangle $ABCD$ have been drawn with red lines. The ratio between the distance between the horizontal coordinates of the pair of pixel points with the same vertical coordinates on the left and right half of pear contour and the wide side of the geometric outer rectangle of the pear contour can reflect the quasi-rectangularity of the pear contour, which is calculated as shown in Formula (2). Similarly, if there are more pixel points with the same vertical coordinate on the left and right half of pear contour, we take the pair of pixel points with the largest difference in horizontal coordinates to calculate.

$$C_k=\left|d_{l,k}^{max}-d_{r,k}^{max}\right|,$$

(1)

$$R_k=\frac{d_{c,k}^{max}}{d_l},$$

(2)

In the above formulas, $k$ is the vertical coordinate of the point on the pear contour, $d_{l,k}^{max}$ is the maximum distance between the pixel point with vertical coordinate $k$ on the left half of pear contour and the line $a$. $d_{r,k}^{max}$ is the maximum distance between the pixel point with vertical coordinate $k$ on the right half of pear contour and the line $a$. $d_l$ is the length of the wide side of the geometric outer rectangle of the pear contour. $d_{c,k}^{max}$ is the maximum value of the absolute value of the difference between the horizontal coordinates of a pair of pixels with the same vertical coordinate on the pear contour.

2.1.2. Pear Surface Color Feature Extraction

The surface color of a pear affects the consumer’s judgment of its taste and sweetness. The normal color of a mature Kulle balsam pear is cyan, slightly reddish, with black round fine spots on the skin. The pear color categories proposed are standard color and deviation color, as shown in Figure 4. As shown in Figure 4a, the peels of standard-colored pears are mainly cyan and slightly red, while the peels of off-colored pears are all cyan, as shown in Figure 4b. It can be seen that there are differences in color distribution between the standard color pear and deviant color pear. The acquired pear image has BGR three channels, in which the combination feature of the mean and variance of the pixel points in R channel and G channel can distinguish the pear surface color. The means values $N_R$,$N_G$ and variances $S_R$,$S_G$ of the R and G channels corresponding to the pear surface area are calculated as shown in Formulas (3)–(6).

$$N_R=\frac{\sum R}{N},$$

(3)

$$N_G=\frac{\sum G}{N},$$

(4)

$$S_R=\frac{\sum R^2}{N}-N_R^2$$

(5)

$$S_G=\frac{\sum G^2}{N}-N_G^2$$

(6)

In the above formulas, $N$ is the total number of pixel points in the pear surface area, $R$ and $G$ are the values of each pixel point on the corresponding channels.

2.1.3. Pear Surface Defect Feature Extraction

The skin of pears is often damaged by external natural environment and mechanical friction, which reduces the commercial value of pears. The area of surface defects is an important factor affecting the appearance quality of the pear, which can correspond to the number of pixel points in the defective area of the pear image. The number of pixel points in the defective parts of the pear surface image are calculated as shown in Formula (7). It is stipulated that the area of mild defects is not more than $0.5 \mathrm{c}{\mathrm{m}}^2$. The area of moderate defects is not more than $1 \mathrm{c}{\mathrm{m}}^2$, and the area of serious defects is larger than $1 \mathrm{c}{\mathrm{m}}^2$.

$$S_{defect}={\sum }_m{\sum }_{n}f\left(x,y\right),$$

(7)

In the above formula, $f\left(x,y\right)$ is the binary image of the pear surface defect. $m$ and $n$ are the number of pixels in the defect area along the coordinate axes of the image coordinate system.

In order to further investigate the categories of pear surface defects, it is proposed to classify the categories of pear surface defects into stabbed defects, bruised defects, abraded defects and rusty defects. The appearance of defects is shown in Figure 5. Pear species can be distinguished according to the texture features of surface defects. Here, the angular second-order moment (ASM), contrast, inverse difference matrix (IDM) and entropy in the GLCM features of the defect area are selected and calculated as shown in Formulas (8)–(11).

$${\sum }_i{\sum }_{j}P{\left(i,j\right)}^2,$$

(8)

$${\displaystyle \sum }_i{\displaystyle \sum }_j{\left(i-j\right)}^{2}P\left(i,j\right)$$

(9)

$${\displaystyle \sum }_i{\displaystyle \sum }_j\frac{P\left(i,j\right)}{1+{\left(i-j\right)}^2}$$

(10)

$$-{\displaystyle \sum }_i{\displaystyle \sum }_{j}P\left(i,j\right)\log \left(P\left(i,j\right)\right)$$

(11)

In the above formulas, $\left(i,j\right)$ is the pixel point that deviates from the pixel point $\left(x,y\right)$ in the grayscale image $g\left(x,y\right)$ of the pear surface defect by a certain distance, and $P\left(i,j\right)$ is the probability of occurrence of the corresponding grayscale value of the pixel point $\left(i,j\right)$ in the whole grayscale image.

2.2. Pear Grading Method

Referring to the national standard for pear appearance quality grading GBT10650-2008, an intelligent grading method for pear appearance is proposed based on pear shape, surface color and defect information. The pears are divided into four categories: “premium pear”, “first-class pear”, “second-class pear” and “defective pear”. As shown in Figure 6, the main difference between “premium pear” and “first-class pear” is whether the pear shape is standard, the main difference between “first-class pear” and “second-class pear” is whether the pear surface color is standard, and the “second-class pear” and “defective pear” are mainly distinguished by the area of pear surface defects. In fact, this grading rule can be changed according to the requirements of users to achieve more standard grading of pears.

The specific implementation steps of pear appearance quality grading method are as follows:

Step 1:

Obtain pear shape features and train classifier;

Step 2:

Obtain pear surface color features and train classifier;

Step 3:

Input the test set of pear shape features and color features into the trained classifier to predict the pear shape category and color category;

Step 4:

Obtain the pear defect area, and judge the pear defect degree;

Step 5:

Judge the pear appearance quality category according to the output results of each classifier and the grading rules.

In this study, the relatively simple and effective classifiers were used to test the effectiveness of the above features for correct classification. In view of the advantages of BP neural network, such as strong nonlinear mapping ability, generalization ability, clustering analysis ability, parallel information processing ability and flexible network structure, BP neural network was used to identify the categories of pear shape and surface defect. Due to the good generalization performance of SVM and the binary classification of pear color, it is proposed to use SVM to classify the surface color of pears.

2.2.1. Classification Method of Pear Shape and Surface Defect

The neural network is a system that mimics the structure and function of the human brain and processes information through mathematical methods. The nodes in the neural network are called neurons. Each neuron has $n$ inputs $X_i$ with weight $W_i$. All the inputs received by the neuron ${\displaystyle \sum }_{i=1}^n{X}_i{W}_i$ will be linearly combined with the neuron threshold term $\theta$. Then the linear combination is mapped by the activation function $f$, and the final output of each neuron is $O=f\left({\displaystyle \sum }_{i=1}^n{X}_i{W}_i+\theta \right)$. Currently, the feedforward neural networks trained by back propagation algorithms are often used. In the feedforward neural network, there is no interconnection between the neuron nodes in each layer except for the neuron nodes between layers. The feedforward neural network model is shown in Figure 7, and the BP neural network consists of an input layer, a hidden layer, and an output layer [18]. The training process includes forward and backward phases. In the forward training phase, the feature vectors are input into the neural network and the loss function is calculated, and in the backward training phase, the deviation between the prediction result and the correct result is compared, and the weights of the network are updated based on this until the training process is completed.

For the pear shape BP neural network classifier, the model structure is shown in Figure 7. The algorithm steps are as follows.

Step 1:

Preprocess the collected pear appearance images to obtain the pear contour binary image.

Step 2:

Set the number of sampling points and then extract the symmetry feature and quasi-rectangularity feature of the pear shape contour.

Step 3:

Set the BP neural network hyperparameters and input the training set to complete the training of the pear shape classifier.

Step 4:

Input the test set into the trained model to test the classification performance.

For the pear surface defect BP neural network classifier, the model structure is shown in Figure 8, and the algorithm steps are as follows.

Step 1:

Preprocess the collected pear appearance images and segment the pear surface defect area images.

Step 2:

Extract the ASM, contrast, IDM and entropy features of the pear surface defect region.

Step 3:

Set the BP neural network hyperparameters, and input the training set to complete the training of the pear surface defect classifier;

Step 4:

Input the test set into the trained model to test the classification performance.

2.2.2. Classification Method of Pear Surface Color

SVM is a binary classification model trained on the basic idea of solving a separating hyperplane that can correctly distinguish data categories and with maximum geometric spacing. Due to the use of kernel skill, SVM can be used not only for linear classification, but also for nonlinear classification [19,20]. The process of using this algorithm is divided into two phases: training and testing. The training process of SVM is essentially to find the appropriate penalty factor $C$ and parameter $gamma$. The role of penalty factor $C$ is to balance the classification interval and the number of misclassified samples. The parameter $gamma$ is the coefficient of the kernel function. These two hyperparameters are determined by the grid search method. The algorithm steps are as follows.

Step 1:

Preprocess the collected pear appearance images and segment the pear surface image.

Step 2:

Extract the R and G channel means and variance features of the pear surface images.

Step 3:

Set the parameter range of the grid search method and train the SVM using the training set.

Step 4:

Input the test set into the trained SVM to test the classification performance.

2.2.3. Grading Rules

After the pear shape and surface color classifier models are trained, the pear shape and surface color category can be predicted according to the input features. The pear surface defect degree can be judged according to the area size of the pear surface defects. The grading rules are as follows.

Rule 1:

If the pear shape classifier outputs the standard shape category, the pear surface color classifier outputs the standard color category, and the pear surface is judged to be free of defects, then the final output of pear appearance quality category is “premium pear”.

Rule 2:

If the pear shape classifier outputs any shape category, the pear surface color classifier outputs the standard color category, and the judgment of the pear surface defect degree is mild or below, then the final output of pear appearance quality category is “first-class pear”.

Rule 3:

If the pear shape classifier outputs any fruit shape category, the pear surface color classifier outputs any color category, and the degree of pear surface defects is judged to be medium or below, then the final output of pear appearance quality category is “second-class pear”.

Rule 4:

If the pear shape classifier outputs any shape category, the pear surface color classifier outputs any color category, and the judgment of the pear surface defect degree is serious, then the final output of pear appearance quality category is “defective pear”.

3. Experiments and Results

3.1. Image Acquisition and Grading System

In general, the implementation of fruit grading algorithm needs to build an image acquisition system firstly [21]. This simple image acquisition system can only realize the image acquisition function but it cannot grade fruits according to the results of the classification algorithm. In this study, the designed experimental hardware system can simultaneously realize image acquisition and subsequent sorting functions.

The appearance of the designed pear appearance quality detection and grading system is shown in Figure 9. The step motor drives the conveyor, and the pears reach the image acquisition place to complete the appearance image acquisition. The sorting function of the device is realized by the push-pull electromagnets, which can push the pears into the corresponding pear storage boxes.

The composition of the whole hardware system is shown in Figure 10, which contains the transfer unit, the position detection unit, the image detection unit, the sorting unit, the control input unit and the power supply module. The image acquisition unit is a dark box made of black Arthur acrylic board, which can avoid the interference of external light. The LED light source and CCD camera are arranged inside the dark box to obtain pear images. The sorting unit adopts push-pull electromagnets. The schematic diagram of push-pull electromagnet sorting is shown in Figure 11. When the pear reaches the position sensor at the corresponding categories in the sorting area, the normally open (NO) contact of the relay in the control circuit of the push-pull electromagnet is closed, and the action of the push-pull electromagnet will push the pear into the corresponding storage box. The working flow of the designed device during the grading of the pear is shown in Figure 12.

An experimental prototype as shown in Figure 13 was built. The relevant hardware parameters are shown in

Table 2. Hardware parameters.

Device Name	Device Parameters
CCD color camera	Firm: Microvision, Camera model: EM-200C, Connection: RJ-45, Resolution: 1600 × 1200, Frame rate: 40, Lens model: BT-23C0828MP10, Focal length: 8 mm, Lens mount: C, Lens field of view: 67.6° × 57.6° × 44°, Adjustment range of aperture: 1:1.28~16
IPC	Network interface card: Gigabit NIC supporting jumbo frame, Communication interface: serial port
Controller	Firm: Siemens, Model: S7-200 smart, Communication protocols: RS485
Photoelectric sensor	Firm: OMCH, Model: E3F-DS10C4, Output model: NPN NO, Detection distance: 20~100 mm
Push-pull electromagnet	Stroke: 60 mm, Attraction: 20 N
Motor	Firm: PERFECT, Model: 57BYG250B, Motor driver: TB6600, Torque: 2.3 NM
Power supply	220 V AC power supply, 24 V DC power supply

.

The pear appearance quality grading software was developed for the designed hardware system. The flow of the grading software is shown in Figure 14.

3.2. Image Acquisition and Preprocessing

Using the built hardware device for pear appearance image acquisition. The number of images in different categories under the three grading indexes of pear shape, color and surface defects are shown in

Table 3. The number of samples in the corresponding categories of grading indicators.

Shape			Surface Color		Surface Defect					Number of Images
Standard Shape	Apical Shape	Eccentric Shape	Standard Color	Deviant Color	Flawless	Stabbed Defect	Bruised Defect	Abraded Defect	Rusty Defect
133	129	136	204	194	82	78	85	74	79	398

. The images are labeled by professionals. The labels of each pear image include shape label, color label, surface defect label and appearance quality label. The obtained data set is divided into training set and test set according to the ratio of 8:2. As can be seen from the number of samples in different categories reflected in Table 3, the samples of this classification task are roughly balanced.

The obtained images need to be preprocessed. For the analysis of pear shape features and color features, the complete pear surface image needs to be obtained. For the analysis of pear surface defect features, it is necessary to obtain the image of surface defects. The preprocessing process is shown in Figure 15. The original image (Figure 15a) is grayed to obtain the grayed image (Figure 15b). The Ostu threshold segmentation technique is used to segment the pear contour to obtain Figure 15c. The segmented pear contour image is morphologically processed to remove noise and stem parts, and the pear surface area mask (Figure 15d) is finally obtained. The pear surface area mask and the original image are Boolean operated to obtain the pear surface area (Figure 15e). The pear surface area image is grayed to obtain Figure 15f. The pear surface defect contour image (Figure 15g) with noise is obtained by threshold segmentation again. After morphological operation and contour screening, the pear surface defect mask (Figure 15h) is obtained. The pear defect area is obtained by Boolean operation between the mask image and the original image (Figure 15i).

In the above image preprocessing process, the pear surface area image obtained by the first threshold segmentation can be used to analyze the pear shape and color features. The pear defect area image obtained by the second threshold segmentation can further analyze the pear surface defect features.

3.3. Feature Analysis

The symmetry and quasi-rectangularity features of pear shape grading index were extracted. The R and G channel mean and variance features were extracted for the color grading index, and the ASM, contrast, IDM and entropy in the GLCM features were extracted for the pear surface defect grading index. If the extracted features perform well, they can produce a good distinction between different categories under the grading indexes.

For pear shape features, the symmetry feature comparison of the standard shape as well as the eccentric shape and the quasi-rectangularity feature comparison of the standard shape as well as the apical shape are shown in Figure 16. Comparing Figure 16a,b, it can be seen that the pair of pixel points with the same vertical coordinate in the left and right half of pear contour of the standard shape have approximately the same distance from the center line, so the two curves have a high degree of overlap. While the opposite is true for the eccentric shape, where the pair of pixel points with the same vertical coordinates in the left and right half of pear contour of the eccentric shape have a certain difference from the center line, so the two curves have a poor overlap. Comparing Figure 16c,d, the parts of the red curves framed by the green boxes are the significant distinction between the standard shape and the apical shape. The reason why the curves change so much is that the apical shape contour suddenly shrinks near the calyx. Moreover, there is also a difference between the two pictures in the red curve framed by the yellow boxes. The slope of the curve framed by the yellow box representing the apical shape is larger than that of the standard shape, because the apical shape contour expands faster closer to the pear stalk. Therefore, the designed symmetry feature and quasi-rectangularity feature can theoretically distinguish the standard shape, apical shape and eccentric shape.

In terms of pear grading with color, the BGR three-channel color histograms of the standard color and deviant color pear images are shown in Figure 17, which shows that the distribution range of the green component pixel values of the standard color pear image is narrower and the mean of pixel value is smaller than that of the deviant color pear image. However, the distribution range of the red component pixel value of the standard color pear image is wider than that of the deviant one. This is consistent with the human eye’s judgment of pear color.

Through the image preprocessing operation in Section 3.2, it can be concluded whether the pear has surface defects. However, the types of defects still cannot be distinguished. The GLCM feature of the pear surface defects were extracted, and 20 samples were randomly selected from the feature vectors of each category, which were reflected in Figure 18. The ASM, contrast, IDM and entropy eigenvalues of the rusty defect are larger than those of the stabbed defect, bruised defect and abraded defect, and they are more widely distributed in the [0,1] interval. The contrast eigenvalue of the abraded defect is larger than that of the stabbed defect and bruised defect, and the contrast eigenvalue of the stabbed defect is larger than that of bruised defect. In summary, the GLCM features of the defective region of pear surface can be used to distinguish the defect types well in theory.

3.4. Classification and Performance Evaluation

3.4.1. Classification Implementation and Hyperparameter Setting

The BP neural network was used to classify the categories of pear shapes and surface defects, and the SVM was employed to classify the categories of pear surface colors. The hyperparameters are shown in

Table 4. Classifier hyperparameter settings.

Classifier	Hyperparameters
BP neural network classifier of pear shape	Number of input layer nodes: 100, Number of hidden layer nodes: 50, Number of output layer nodes: 3, Learning rate: 0.3, Number of iterations: 300
SVM classifier of pear surface color	Penalty factor $C$:5, $Gamma$ = 1, Kernel: Gaussian kernel
BP neural network classifier of pear surface defect	Number of input layer nodes: 4, Number of hidden layer nodes: 85, Number of output layer nodes: 4, Learning rate: 0.23, Number of iterations: 300

. When classifying the categories of pear shapes, 50 sampling points were taken for calculating the symmetry and quasi-rectangularity features respectively. The grid search method was used to find the optimal hyperparameters for pear surface color classification. The above training process was performed on a computer with Intel Core i7-6700HQ CPU, 8GB of memory, NVIDIA GeForce GTX960 video card, and Python 3.6 development platform with Keras and scikit-learn.

3.4.2. Classification Performance Evaluation

The accuracy, weighted average recall, weighted average precision and weighted average F1-score were calculated to evaluate the classification performance of each classifier. The above four classification performance evaluation indicators were calculated from the confusion matrix of each classifier. The recall, precision and F1-score of each category are calculated as shown in Formula (12)–(14).

$$recal{l}_i=\frac{T{P}_i}{T{P}_i+F{N}_i}$$

(12)

$$precisio{n}_i=\frac{T{P}_i}{T{P}_i+F{P}_i}$$

(13)

$$F1-scor{e}_i=\frac{2\times precisio{n}_i\times recal{l}_i}{precisio{n}_i+recal{l}_i}$$

(14)

In the above formulas, $T{P}_i$ denotes the number of samples that actually belong to category $i$ and are predicted to be category $i$. $F{N}_i$ represents the number of samples that actually belong to category $i$ but are not predicted to be category $i$, and $F{P}_i$ denotes the number of samples that actually do not belong to category $i$ but are predicted to be category $i$.

The accuracy is calculated as shown in Formula (15). The weighted average recall, weighted average precision and weighted average F1-score are the average of the recall, precision and F1-score of each category. The calculation method is shown in Formula (16)–(18), where $I$ is the number of categories and $A$ is the total number of samples in the test set.

$$accuracy=\frac{{{\displaystyle \sum }}_{i=1}^{I}T{P}_i}{A}$$

(15)

$$recall=\frac{{{\displaystyle \sum }}_{i=1}^{I}recal{l}_i}{I}$$

(16)

$$precision=\frac{{{\displaystyle \sum }}_{i=1}^{I}precisio{n}_i}{I}$$

(17)

$$\mathrm{F}1-\mathrm{score}=\frac{{{\displaystyle \sum }}_{i=1}^{I}F1-scor{e}_i}{I}$$

(18)

Table 5. Classification performance of the four classifiers.

	$\mathit{A}\mathit{c}\mathit{c}\mathit{u}\mathit{r}\mathit{a}\mathit{c}\mathit{y}$	$\mathit{R}\mathit{e}\mathit{c}\mathit{a}\mathit{l}\mathit{l}$	$\mathit{P}\mathit{r}\mathit{e}\mathit{c}\mathit{i}\mathit{s}\mathit{i}\mathit{o}\mathit{n}$	$\mathit{F}1-\mathsf{Score}$	Running Time
Shape classifier	83.3%	83.3%	83.5%	83.4%	2.02 s
Surface color classifier	91.0%	91.1%	91.0%	91.9%	1.06 s
Surface defect classifier	76.6%	76.9%	81.5%	77.9%	2.34 s
Appearance quality classifier	80.5%	80.6%	81.3%	80.9%	—

shows the values of each classification performance evaluation indicator of the pear shape, color, and surface defect classifiers. The running times of each classifier are also illustrated in Table 5. To further track the misclassified samples, the confusion matrixes of the three classifiers were drawn, as shown in Figure 19a–c. According to the grading rules, the pear appearance quality grading was carried out, and the pear shape, color and surface defects were integrated. The confusion matrix of grading results is shown in Figure 19d.

3.4.3. Discussion

By analyzing the features extracted from the three grading indexes of pear shape, color and surface defect in Section 3.3. It can be seen that the extracted features can distinguish the categories under the three grading indexes. In order to further verify the classification performance, the design of the classifier and the evaluation of the classification performance were carried out. As can be seen from Table 5, the accuracy of three classifiers is above 75%, among which the pear surface color classifier has the best classification performance. And the performance of the surface defect classifier needs to be improved compared with the remaining two. The confusion matrix is used to track the misclassified samples. According to Figure 19a, some pear shapes are incorrectly classified. The reason may be that the image labeling is based on the actual overall shape of the pear, while the classification is based on a certain surface image of the pear, and the two may not be consistent. The misclassification of color categories may also be caused by the inconsistency between the overall color of the pear and the color of a surface of the pear (Figure 19b). In addition, the misclassification of pear shape and color may also be the result of insufficient discrimination of the designed features. For the misclassification of pear surface defect categories (Figure 19c), according to the analysis of the defect features in Section 3.3, the discrimination of some defect feature values is insufficient, which may be the reason for the misclassification of defect types. The pear appearance quality classification (Figure 19d) is finally determined by the output results of pear shape classifier, surface color classifier and the calculation of the defect area.

4. Conclusions

In this study, according to the classification standard of pear appearance quality, the corresponding features were designed and analyzed. Based on the three classification indexes of pear shape, surface color and surface defect, the features of distinguishing different categories were used for classifier training. In terms of pear shape, pear contour symmetry and quasi-rectangularity features were proposed to distinguish pear standard shape, apical shape and eccentric shape. The BP neural network classifier was trained and the classification accuracy reached 83.3%. In terms of pear surface color, the categories of pear color were divided into standard color and deviant color according to the color distribution when the pear was mature. The means and variances of G and R channel pixel values were extracted as features to train the SVM classifier and the classification accuracy reached 91.0%. In the aspect of pear surface defects, the size of defect area was used as the reference index of quality grading. In order to further distinguish the types of defects, the GLCM feature of pear surface defects was extracted and used for the training of the BP neural network, and the accuracy reached 80.5%. For the quality grading of pear appearance based on the three indexes of pear shape, surface color and defects, the accuracy reached 80.5% according to the grading rules. In this study, pear appearance quality grading was divided into several grading indexes. For the grading object of pears, the grading connotation is more abundant and comprehensive, which has certain positive significance for improving the grading efficiency of pear appearance quality and overcoming the shortcomings of strong subjectivity of manual grading. However, there are also some problems in the grading algorithm designed in this paper, such as the use of multiple classifiers in the classification process, which leads to the efficiency and accuracy to be further improved. In the future work, it is proposed to extract the features of the image set according to the pear shape, color and surface defects, make the category label of pear appearance image according to the factor analysis method, and use deep learning technology to grading pear appearance quality to overcome the shortcomings of low efficiency of multiple classifiers.

Moreover, the designed pear appearance quality grading system can realize the collection of pear appearance images, the grading of pear appearance quality, and can be used to verify the pear appearance quality grading algorithm in this paper. The designed system includes the transfer unit, the position detection unit, the image detection unit, the sorting unit, the control input unit and a power supply module. The sorting unit uses the push-pull electromagnets to perform the sorting function simply and reliably, which can basically meet the requirements of pear appearance quality detection and grading. It has certain reference significance for the future development of pear detection pipeline. However, there is still the disadvantage of not being able to obtain the full surface image of the pear. In the future work, more ingenious mechanical structures can be designed or multi-camera and image synthesis technology can be used as further improvement measures.