A Machine-Learning Approach for Automatic Grape-Bunch Detection Based on Opponent Colors

Abstract

This paper presents a novel and automatic artificial-intelligence (AI) method for grape-bunch detection from RGB images. It mainly consists of a cascade of support vector machine (SVM)-based classifiers that rely on visual contrast-based features that, in turn, are defined according to grape bunch color visual perception. Due to some principles of opponent color theory and proper visual contrast measures, a precise estimate of grape bunches is achieved. Extensive experimental results show that the proposed method is able to accurately segment grapes even in uncontrolled acquisition conditions and with limited computational load. Finally, such an approach requires a very small number of training samples, making it appropriate for onsite and real-time applications that are implementable on smart devices, usable and even set up by winemakers.

1. Introduction

The problem of a precise yield estimation in vineyards is of great interest for wine industry. Some data, such as a production of 6.04 million tons in 2020 only in the USA or a savings of about one hundred million dollars with a correct yield prediction, help to understand the importance of this topic [1,2]. As a result, increasing research work has been done in recent years on this topic through adopting different strategies. However, a precise yield prediction is as simple in theory as it is difficult in practice. Usually this task is accomplished by winemakers with unavoidable errors due to different factors that can be:

• Objective: non vineyard uniformity, weather conditions, different pruning techniques, etc. [3].
• Subjective: human overestimation, lack of attention, errors, etc. [4].

That is why an automatic image-based framework that replicates winemakers inspections but it is robust to external conditions, is becoming of great interest for the entire research field.

Objective factors make the task very difficult for any image-based framework exploiting deep-learning networks. In fact, the non uniformity of vineyards makes such an estimate complicated and strongly case dependent. For example, different kinds of pruning can amplify problems, such as object (i.e., grape) occlusions and so on. This difficulty is proven by the quantity of different kinds of deep neural networks (DNNs) proposed in the literature—see, for instance, [5,6] and the next section for a short review. Moreover, DNNs have a discrete computational burden (even though some approaches dealing with this problem have recently been proposed) [7,8,9,10,11] and need for a large and representative training set to guarantee an acceptable accuracy rate and to avoid overfitting [12]. Some recent approaches [13,14] have attempted to exploit pretrained convolutional neural networks (CNNs) to overcome their computational burden.

However, limits in finding useful images for an effective training for ‘in-the-wild’ cases along with the need for an RGB-d camera in place of a common RGB one prove that the grape bunch segmentation problem is far from being solved.

The goal of the proposed approach is to start from the aforementioned problems to produce a framework that:

• Is automatic, or at least, minimizes any human aid, while remaining effective for a reliable yield assessment.
• Is not computationally expensive, i.e., it allows for a fast response for each image and requires simple operations that facilitate its implementation on portable instrumentation.
• Requires a small training set, allowing its straightforward updating and adaptation to different conditions and use cases.

The aforementioned requirements come from the analysis of two possible practical scenarios. The first one accounts for unmanned aerial vehicle (UAV)-based applications where no web connections are available—very frequent in many practical cases. In this case, the software should run on the (small) computer that the UAV is equipped with. Hence, a simple artificial-intelligence (AI) tool that needs a low computational effort is required. The second scenario is the one where a winemaker uses their own smartphone for training and testing in the simplest way. In this case, very few examples for training the adopted AI tool are required, apart from a small computational effort. Both scenarios lead to a light and case-dependent approach based on a light machine-learning method (see Figure 1) [12]—with limited but good examples for the training set.

In addition to the considerations above, the subjective component plays a fundamental role. In fact, if on the one hand, the human approach has the drawback that winemakers ’tend to choose healthier and larger bunches when they are doing sampling in the field’ [4], it is also true that humans have an undoubtable ability in recognizing objects in very different and critical conditions: that is why DNNs attempt to simulate it. Only the good part of human activity, its early visual perception, should be accounted for [15].

The proposed approach attempts to embed all aforementioned requests. It is based on the main peculiarities of human perception exploiting both the early vision processes in terms of luminance and contrast and the opponent colors theory. The latter has been formalized in the past years and is currently under investigation [15,16,17,18,19]. Specifically, the proposed method consists of a supervised learning framework oriented to identify the areas containing (yellow or blue) bunches of grapes in an ’in-the-wild’ RGB vineyard image. With ’in the wild’, we mean an image containing grape bunches as well as foliage, ground, and sky and in uncontrolled light conditions. The proposed method takes advantage of the use of limited but distinctive features that are close to the ones that are encoded in the onsite visual inspection process.

Exploiting both multiscale analysis and opponent colors theory, a new feature space is defined where each transformed image is analyzed by a suitably trained support vector machine (SVM). The rationale is to exploit the fact that human perception works as an optimized encoder for processing and storing visual information [15,20,21,22]. This property contributes to determining the right features for very effective bunch detection. The achieved results show that the proposed method is able to outperform competing approaches in terms of accuracy, size of the training set and computing time.

2. Materials and Methods

This section is organized as follows. First, a short review on available approaches in the literature is presented. This helps the reader to better understand the main guidelines followed in the literature. Successively, a technical background useful to understand the proposed approach is offered. It contains a sketch of both multiscale analysis and opponent colors theory. Finally, the proposed approach is described, outlining the peculiarities of blue and yellow grape cases.

2.1. A Short Review on Automatic Yield Estimation

The increasing interest on this topic makes it difficult for any review to be exhaustive. In the following, only the approaches that are related (or involve topics close) to the proposed approach will be presented.

The approach in [23] is a good example to understand the difficulty of making any automatic yield estimation feasible: it is as effective as it is computationally demanding. If on the one hand, the use of a radial symmetry transform allows effective two-stage and large-scale wine images processing, on the other hand, the complexity is of about $O\left(KN\right)$, with K being the pixel number of the whole image while N is the size of the neighborhood [24]. The computational burden is not the only problem for automatic yield estimation. In fact, under the hypothesis of using an UAV (or any automatic vehicle) with a suitable camera, various problems, such as the view angle and size of the scene (in order to take grape bunches), calibration [25], image resolution, light conditions, and acquisition conditions (to avoid grape occlusion) are crucial for a successive but effective color image processing. All these problems make classical image processing infeasible on high-resolution images [1].

Once the acquisition phase has been made, the successive processing is, again, not simple, and different approaches have been proposed. Most of them are clearly oriented to feature extraction and classification, where a high accuracy (very often in ideal conditions) is combined with a high computational effort. An example is in [26], where different Fuzzy C-Means (FCM) clustering methods are compared: Robust Fuzzy Possibilistic C-Means, FCM and FCM-GK (FCM with Gustafson–Kessel)—with accuracy ranging from $85\%$ to $88\%$. Other interesting approaches are in: [27], which employs a 3D grapevines formation based on Structure-From-Motion followed by a saliency map analysis and SVM for classification; [26,28], where a combination of SVM, K-means and the scale-invariant feature transform (SIFT) for various vineyard components clustering is used; and [29,30], which classifies different vineyard objects by means of Mahalanobis measures.

As correctly outlined in [1], most of the computational effort is spent on processing unuseful scene components—estimated in at least 50% of the total information.

Some interesting approaches dealing with color images are found in [25,31,32]. The first two papers use RGB thresholding along with some morphological operations that speed up the processing phase even under controlled light conditions (artificial light in the night) and ideal acquisition conditions (controlled grape pose) with a manual RGB key value selection for successive SVM classification. It is straightforward that such an approach cannot be used in practice where large scale acquisitions are needed. From this point of view, an effort was made in [25] where a preliminary selection of bunch areas was made via color thresholding and morphological operations. This allows for successive feature selection and classification achieved via ReliefF [33], a sequential feature selection method [34] and SVM [35]. The preliminary selection of potential bunch areas allows for at least 70% of (useless) scene information to be discarded.

It is also worth mentioning the great effort in exploiting deep-learning techniques, very often in combination with computer vision technologies [12,36], to automate agricultural processes. In particular, many techniques have been inherited by a well-known field of computer vision: object detection. In this case, the very critical conditions (very different outdoor light conditions, different scale of the target, occlusions, and so on) in agriculture make this task very difficult. That is why various deep-learning approaches, mainly based on convolutional neural networks (CNN), have been proposed [5,6]. Specifically, deep-learning approaches for object detection can be split into:

• One-stage detectors: In this case, object classification and bounding-box regression are done directly without using pre-generated region proposals (candidate object bounding-boxes). Approaches belonging to this class are, for example, Single Shot multibox Detector (SSD) [37], RetinaNet [38], Fully Convolutional One-Stage (FCOS) [39], DEtection TRansformer (DETR) [40], EfficientDet [41], and the You Only Look Once (YOLO) family [42,43,44,45,46].
• Two-stage detectors: First, a generation of region proposals, e.g., by selective search as in R-CNN and Fast R-CNN or by a Region Proposal Network (RPN) as in Faster R-CNN, is made. Then, a second step oriented to object classification in each region proposal is applied. Sometimes, some additional phases, such as bounding-box regression for refining the region proposals, and binary-mask prediction, are performed. Examples of approaches belonging to this class are: region-based CNN (R-CNN) [47], Fast/Faster R-CNN [48,49], Spatial Pyramid Pooling Networks (SPPNet) [33], Feature Pyramid Network (FPN) [50], and CenterNet2 [51].

Usually, two-stage detectors perform better than one-stage ones in terms of the precision of localizing target in different conditions (see, for instance, [52,53]). However, two-stage detectors pay the price of ’slow inference speed and high requirement of computational resources’ for this specific field [2]. This is the reason why many approaches oriented their effort toward one-stage detectors: specifically, YOLO networks (see, for instance, [7,8,9,10,11,54,55,56] for specific applications to different kinds of fruit). Apart from the specific adopted strategies, it is worth noting that all these approaches, even though fast, show a high sensitivity to occlusion and should be combined with further computer vision tricks [57].

In particular, a trend of a certain success is represented by Swin-Transformer (hierarchical vision transformer network)-based approaches [57,58,59,60,61,62]. Despite the interesting philosophy on which they are based (self-attention mechanism for learning [63,64,65]), again, the computational effort is still high. Interesting approaches based on this strategy and concerning the agricultural field can be found in [66,67,68]. Finally, it should not be overlooked that the intensive use of deep-learning networks leads to the need for a great quantity of images for suitable training. That is why the need for populated and labeled databases is becoming very impelling—two very recent databases on grape bunches are described in [69,70].

2.2. Technical Background

This section focuses on two main topics that the proposed model is based on: multiscale analysis and the opponent colors theory. These will be the focus of the next two sections, respectively.

2.2.1. Multiresolution Analysis

The change of scale of the input image can be seen as simple application of the multiresolution analysis theory. The latter involves the formal definition of producing different scales of a given function that are correlated to each other by some mathematical properties [71]. This representation is oriented to highlight specific details of a given signal in agreement with the pioneering studies on multiresolution pyramids by Burt and Adelson [72] first and the formal construction of orthogonal wavelets later [73].

Coarsely speaking, a given function f at a resolution $2^{-j}$ can be seen as a (discrete) grid of samples where local function averages are considered—the size of the average domain is proportional to $2^j$. A multiresolution approximation of f is composed of different and embedded grids. Very often, this operation becomes more intuitive by considering each one of these grids (say, at resolution $2^{-j}$) as the orthogonal projection on the space $V_j\subset L^2\left(\mathbf{R}\right)$ (This functional space contains functions with finite energy.). $V_j$ includes all possible approximations at the resolution $2^{-j}$. Hence, starting from a given function f, its approximation $f_j$ at resolution $2^{-j}$ is the projection on the space $V_j$ constrained to minimize the following quantity: $\left|\right|f-f_j{\left|\right|}_2$.

More details concerning this theory and its applications can be found in [71,73,74]. However, it is possible to say that the aforementioned theory paves the way to orthonormal wavelets, i.e., orthonormal bases. More specifically, the approximation of a given function f at the resolution $2^{-i}$ can be defined as the orthogonal projection ${P_V}_{j}f$ on $V_j$. In order to find such a projection, an orthonormal basis of $V_j$ has to be looked for. Usually, this operation can be achieved by convolving f with a dilated and translated version of a scaling function $\mathsf{\Phi }$. It is possible to prove that, under suitable conditions, the family ${\left\{{\mathsf{\Phi }}_{j,n}\right\}}_{n\in \mathbf{Z}}$, with j and n, respectively, the dilation and shifting parameters, is an orthonormal basis of $V_j$ for all $j\in \mathbf{Z}$.

In the sequel, only one smoothed version of f will be used for each of the two phases of the model, and the Haar basis was selected as $\mathsf{\Phi }$ [71]. The latter can be simply seen as an operator that computes local averages of f with a fixed window that depends on the resolution level j. It is worth stressing that this smoothing is consistent with human vision mechanisms in the pre-attentive phase where redundant and not perceived frequencies are discarded [75,76]. Smoothing irregular areas has also the advantage of making regions more homogeneous and, thus, enhancing them and actually increasing their visual saliency [77,78]. The selection of a proper level of resolution quantifies the amount of information that can be lost in the visual coding process as formally studied in [76,78].

2.2.2. Opponent Colors Theory

It is well-known that the trichromatic theory of color vision explains how human beings (their cells) detect blue, red, and green wavelengths. The combination of these three main colors allows perception of the whole visible spectrum [15]. However, the current understanding of color perception is more complicated. In particular, a key role is played by the opponent colors theory. This theory was developed by Ewald Hering, who based it on the observation that specific color combinations cannot be seen [79]. The latter is based on the human ability to perceive color that is mainly based on three receptor complexes: the red–green complex, the blue–yellow complex, and the black–white complex [17]—see Figure 2. As matter of fact, recently, the pairings above have been refined as blue–yellow, red–cyan, and green–magenta.

According to the opponent colors theory, human brain can only register the presence of one color of a pair at a time. Specifically, for each receptor complex, the involved colors oppose one another. For example, the cell that activates for red will deactivate for green light and vice versa. Two opponent colors cannot survive in vision. This mechanism can be seen as a perceptive (and very effective) coding of visual information.

The whole theory is then based on the presence of two kinds of (opposite) cells for each receptor complex, which activate for a color while having an inhibitory response to the opposite one [17].

Though not used in this paper, it is worth outlining how this theory can explain the perceptual phenomena of negative afterimages—see [15,17,80] and the references therein.

It is possible to say that, while the classical trichromatic theory helps to explain how different types of cones detect different light wavelengths, the opponent colors theory says how the cones connect to the ganglion cells and then how opposing cells are excited or inhibited by certain wavelengths of light. In addition to these two theories, there is the complementary color theory that accounts for how and which wavelengths translate to which colors and then how the brain processes these colors.

2.3. The Proposed Method

The proposed model is based on human perception as it attempts to exploit the ability of human beings in recognizing objects in a small amount of time. Specifically, the ability of human early vision will be exploited for our study case.

It is well-known that early vision refers to those stages of vision that involve capturing, preprocessing, and coding visual information but do not involve the interpretation or other cognitive processing of visual information that requires further brain processing [15]. The proposed model follows some recent results proving that the early vision phase accounts for visual information that is mainly based on the luminance and contrast of the scene under study [75]. In fact, luminance gain control (known as light adaptation) is managed in the retina and is oriented to adjust the sensitivity to match the locally prevalent luminance (light intensity).

Coarsely speaking, the retina divides luminance by the local mean luminance [81,82,83]. On the contrary, contrast gain control starts in the retina and is strengthened at some successive stages of the visual system [81,84,85,86,87,88,89]. Apart from specific details (that can be found in [90]), the input signal is divided by a measure that grows with the locally prevalent root-mean-square (r.m.s.) contrast. In this way, a contrast invariance is produced for a better processing of the eye response: a contrast increase when the contrast is low and vice versa.

In addition to the aforementioned ’visual normalization’, it is worth highlighting another important aspect of visibility: the human eye works as a low pass filter at first glance [15,75]. This is due to the necessity of quickly understanding the content of the scene—for survival needs. That is why both luminance and contrast are considered at a given resolution in the proposed approach. This strategy has a double purpose: on the one hand, it replicates what happens in the early vision (the first 200 milliseconds) phase; on the other, it allows discarding a great deal of information that is not useful for the analysis of the scene content.

These two aspects are clearly linked each other and are implemented in a cascaded binary classification method in this paper. It is worth highlighting that the combination of more than one SVM classifier is not new in the literature, and it has been employed to improve classification accuracy. However, the optimization of a combination of more than one classifier is still matter of study for a wide part of the research community interested in effective classification tools—see, for instance, [91,92]. Figure 3 provides a graphical sketch of the proposed procedure that is described in the following.

First classification. From an algorithmic point of view, an RGB image I depicting a sketch of a vineyard will be transformed into another color space that emphasizes the contrast between object (grape) and background (leaves, grass, sky, etc.). The selected color space is CMY (cyan, magenta, and yellow) as this allows us to emphasize the features of blue and yellow grapes as it will become clearer in the following.

The achieved projected image J in the new color space will be then convolved with a suitable kernel ${\mathsf{\Phi }}_{\overline{j}}$ to simulate the low-pass filtering applied by the human visual system, thus, obtaining the image $\tilde{J}$. A multiscale analysis is then adopted where only one properly selected scale ($2^{\overline{j}}$) is considered. Hence, at each pixel location $\overline{x},\overline{y}$ in the image spatial domain $\mathsf{\Omega }$, the luminance $L(\overline{x},\overline{y})$ and the contrast $Con(\overline{x},\overline{y})$ can be considered as components of the following vector of features

$${\mathbf{v}}_1(\overline{x},\overline{y})=[L(\overline{x},\overline{y}),Con(\overline{x},\overline{y})]\forall (\overline{x},\overline{y})\in \mathsf{\Omega }.$$

(1)

By denoting with ${\mathsf{\Omega }}_G\subset \mathsf{\Omega }$ the grape image region and using the feature vector ${\mathbf{v}}_1$, it is then possible to classify the input image by means of an SVM binary classifier that produces the first binary map $M_1$, defined as follows:

$$M_1(\overline{x},\overline{y})=\left\{\begin{array}{cc}1\hfill & \mathrm{if}(\overline{x},\overline{y})\in {\mathsf{\Omega }}_G\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

The first phase accounts for the early vision mechanism on a (vineyard) color image. However, the role of the colors themselves is crucial in our problem as discriminant for grape recognition. That is why a second phase that refines the map $M_1$ is required. It accounts for the pure color information in order to discard non-grape pixels in $M_1$.

Second classification. For the second classification, the hue, saturation, and brightness components of the HSB space were used as features. HSB space (hue, saturation, and brightness) is also known as HSV (hue, saturation, and value). This space was introduced in the 1970s in order to better fit the way human vision perceives color-making attributes. Moreover, its definition is intuitive and, though debated, its cylindrical geometry (along with the HSL space) makes color perception more natural from a human point of view—see [15,93] for its formal and geometrical derivation.

Let K be the RGB image I projected in this color space; again, all color components are blurred in order to simulate the low-pass filtering performed by the naked eye. By denoting with $\tilde{K}$ the blurred image, for each $(\overline{x},\overline{y})\in {\mathsf{\Omega }}_G$, the feature vector ${\mathbf{v}}_2$ is defined as

$${\mathbf{v}}_2(\overline{x},\overline{y})=[\tilde{K}(\overline{x},\overline{y},1),\tilde{K}(\overline{x},\overline{y},2),\tilde{K}(\overline{x},\overline{y},3)]\forall (\overline{x},\overline{y})\in {\mathsf{\Omega }}_G,$$

(3)

where the indices $1,2,3$, respectively, refer to the hue, saturation, and value components of $\tilde{K}$ and are used as input for a second SVM binary classifier. The SVM-based classification restricted to the non-zero entries of $M_1$ provides the binary map $M_2$ defined as follows,

$$M_2(\overline{x},\overline{y})=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{M}_1(\overline{x},\overline{y})=1and(\overline{x},\overline{y})\in {\mathsf{\Omega }}_G\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(4)

which represents the output of the proposed procedure.

To properly assess if an image pixel depicts part of a grape, the scheme described above has to be adapted to blue and yellow grapes through proper definitions of luminance and contrast as discussed in the following subsections.

2.3.1. Blue Grapes

Blue grape detection inherits the mechanism that characterizes the opponent color theory. In particular, the color of a ripe bunch is usually blue, and the intensity depends on the kind of vineyard and on the lighting conditions. On the other hand, the grape background is composed of leaves that have a certain shade of yellow—a reddish-yellowish. This is the reason why the selected color space is CMY: it contains both a shade of blue (cyan) and yellow. Keeping in mind the opponent color mechanism and its inhibitory action, we propose to code the above mechanism in terms of classical Weber contrast [15]. As a result, the contrast for the blurred image $\tilde{J}$ in the CMY color space is defined as

$$Co{n}_b(\overline{x},\overline{y})=\frac{C(\overline{x},\overline{y})-Y(\overline{x},\overline{y})}{Y(\overline{x},\overline{y})},\forall (\overline{x},\overline{y})\in \mathsf{\Omega }.$$

(5)

In addition, bearing in mind that the human eye is designed to see luminance and contrast, the Y component is selected as luminance:

$$L_b(\overline{x},\overline{y})=Y(\overline{x},\overline{y}),\forall (\overline{x},\overline{y})\in \mathsf{\Omega }.$$

(6)

The rationale behind this choice is that the prevalent color detected by the human eye (while checking the yield) is the yellow color of the leaves—more than the blue grapes. It is worth outlining that the classical and more simple Weber contrast was selected in place of the Michelson one [15]. The motivation of such a choice is two-fold. It is simpler to use and characterize the relative opponent action of the two involved colors; color bands are smoothed with the aim of producing a uniform object on a uniform background as is the case in early vision.

That is why the role of contrast, as in the Michelson one, vanishes. In addition, even though there exist several studies concerning contrast in color images containing text, to the best of the authors’ knowledge, there is not an explicit formula for color contrast, and the matter is still debated (In this case, the minimum contrast ratio should be 4.5:1 for normal text and 3:1 for large text. Various software tools to check this [94] are also available.).

Using Equations (5) and (6), for blue grapes, Equation (1) becomes

$${\mathbf{v}}_1(\overline{x},\overline{y})=[L_b(\overline{x},\overline{y}),Co{n}_b(\overline{x},\overline{y})],\forall (\overline{x},\overline{y})\in \mathsf{\Omega }.$$

It represents the input feature vector for the first binary SVM classifier producing the first map $M_1$.

Regarding the second classification step, as already mentioned in the previous section, it is necessary to also account for the peculiar grape color that changes in agreement with both the stage of grape ripening and the kind of vineyard. Hence, the feature vector ${\mathbf{v}}_2$ is defined as in Equation (3) and feeds the SVM-based classifiers producing the final classification map $M_2$.

2.3.2. Yellow Grapes

Though a slight modification, yellow grape detection follows the same strategy adopted for blue grapes. Before showing it, two considerations have to be made. The first is that yellow grape detection is more difficult. As shown in Figure 4 (Right), grapes have a color that is a mixture of red and yellow, while leaves are characterized by a mixture of green and yellow. The contrast in this case is more light. The second consideration is relative to the selected CMY space, where the subtractive primaries of cyan, magenta, and yellow are the opposing colors to red, green, and blue. Specifically:

• Cyan is opposite to red.
• Magenta is opposite to green.
• Yellow is opposite to blue.

The following strategy was adopted to exploit the opposite components. In particular, magenta was selected as opposite to green (i.e., an approximation of leaves color), while cyan was selected as opposite to red (i.e., an approximation of grapes color). In this case, the common shade of yellow that characterizes both object (grape) and background (leaves) was considered to be self-vanishing.

The corresponding contrast and luminance, computed over the blurred color components, is:

$$Co{n}_y(\overline{x},\overline{y})=\frac{M(\overline{x},\overline{y})-C(\overline{x},\overline{y})}{C(\overline{x},\overline{y})},\forall (\overline{x},\overline{y})\in \mathsf{\Omega }$$

(7)

and

$$L_y(\overline{x},\overline{y})=C(\overline{x},\overline{y}),\forall (\overline{x},\overline{y})\in \mathsf{\Omega },$$

(8)

and then the feature vector ${\mathbf{v}}_1(\overline{x},\overline{y})=[Co{n}_y(\overline{x},\overline{y}),L_y(\overline{x},\overline{y})]$ is used as input for the first classifier. On the contrary, the second classifier works as for blue grapes.

3. Results

Experimental results and tests were performed in a vineyard located in Rome (San Cesareo), Italy in 2021. About 200 images were taken under natural light conditions. The adopted camera was a Kodak EasyShare V803. The image resolution was $3264\times 2448\times 3$. The vineyard was composed of different varieties of grape. In particular, Merlot, Cesanese, and Malvasia regarding blue grapes (oriented to wine production) and Uva Italia for yellow (table) grapes. The distance between the camera and grapes was about 2–3 m but was not expressly controlled. The same goes for the light conditions.

It is well-known in the literature that many available approaches have been tested in ideal conditions in terms of light, pose, without occlusions, and so on. This makes it very difficult to test a specific approach in real conditions [1]. Hence, in this paper, the choice of natural light conditions, no particular care to camera/grape distance as well as a vineyard with a type of pruning with many leaves was made in order to consider an ’in-the-wild’ test. The proposed approach was tested on several images; the algorithm was run on a laptop (1.8 GHz Intel Core i5 dual-core, RAM 8 GB) in the MATLAB environment. Only some examples will be shown here, and they are the blue and yellow grape cases shown in Figure 4.

3.1. Blue Grapes

Following the steps in Figure 3, the input image in Figure 4 (Left) was converted into the complementary CMY color space.

Each component was then filtered by means of 2-D Haar filter ${\mathsf{\Phi }}_{\overline{j}}$ with size $15\times 15$. This choice is the simplest among wavelet filters to obtain a specific scale from a multiresolution analysis [71,78]. From these color components, both the luminance in Equation (6) and contrast in Equation (5), depicted in Figure 5, were considered as elements of the feature vector. The output of the first classification is the map $M_1$ shown in Figure 6. It is worth outlining that, in this case, the adopted training set was composed of only 50 (suitably selected) pixels, where the first 25 refer to ’grape’ while the remaining 25 refer to ’other’—e.g., sky, soil, and foliage.

The second step of the proposed methodology requires the transformation of the RGB original image into the HSV color space. Each component has then been filtered by a Haar kernel of size $20\times 20$. The size of the two adopted blurring kernels was tuned accounting for the maximum visual attention scale (usually the third or fourth for Haar kernels) in a wavelet decomposition [15,77,95]. Hence, for each pixel classified as ‘grape’ in $M_1$, the feature vector was built according to Equation (3).

The second SVM-based classification led to the $M_2$ map shown in Figure 7 (Left). The post-processing step consisted of a morphological opening [96] on the resulting binary map $M_2$, where the radius of the disk was set equal to 10. This step eliminates some spurious and isolated points due to a bad classification, thus, leading to the final map in Figure 7 (Right). This step has not been inserted in the scheme in Figure 3 as it simply refines the achieved result without greatly increasing the framework performance.

In particular, this step simply avoids some annoying and spurious points in the final map. The final classification is in Figure 8. As far it concerns the training set of the second classification, it was built by randomly selecting points among the ‘good’ ones in the first classification. Specifically, only points classified as grapes in the first classification were considered. Among them, the true grape points were used as ‘grape’, while the remaining ones represent the ‘background’.

The most interesting result of this paper is possibly contained in

Table 1. Blue grapes: Classification accuracy (%) for a decreasing size of the training sets used in both phases of the proposed method. The training time and computing time, measured in seconds (s) and required for building the classification maps, have also been provided for each step of the proposed method. The last column refers to the processing time of the whole procedure.

	N° POINTS	ACCURACY	TIME (s)
			training	map	total
First classification	100	93.6	1.296	39.602
Second classification	100	96.5	3.064	23.93	67.89
First classification	90	93.0	1.037	33.742
Second classification	90	96.6	2.523	17.588	54.89
First classification	80	93.5	1.088	30.720
Second classification	80	96.6	2.065	23.758	57.63
First classification	70	93.4	1.105	30.824
Second classification	70	96.3	2.052	15.705	49.67
First classification	60	93.4	1.050	31.264
Second classification	60	96.3	2.057	17.725	52.09
First classification	50	93.5	1.056	31.078
Second classification	50	96.2	2.158	15.872	50.16
First classification	40	93.7	1.031	30.749
Second classification	40	96.2	2.061	14.055	47.89
First classification	30	90.8	1.060	30.626
Second classification	30	95.9	2.301	17.464	51.45
First classification	20	94.3	1.099	30.617
Second classification	20	95.2	2.089	13.708	47.51

, where the classification accuracy is shown for both classifiers of the adopted cascade (first and second classification). The results are ordered in terms of increasing number of adopted points in the training set. As can be observed, only a few points are required for a correct classification. To further stress this point,

Table 2. Blue grapes: Classification accuracy (%) for training sets composed of only 10 samples in both classifications—the result of six different runs is presented; each run requires about 51 s.

	N° POINTS	ACCURACY (%)
First classification	10	91.6
Second classification	10	96.0
First classification	10	92.5
Second classification	10	95.7
First classification	10	85.8
Second classification	10	95.7
First classification	10	95.2
Second classification	10	95.3
First classification	10	93.8
Second classification	10	95.6
First classification	10	92.1
Second classification	10	96.5

shows that having only 10 points in the training set can guarantee a classification accuracy greater than $95\%$.

This represents one of the main contributions of this work since it allows for a manual selection of points even by winemakers through a very fast procedure, making the proposed method easily and on-site adaptable to the different use cases and scenarios. Figure 9 shows the final map achieved using only 10 points, along with the corresponding classification accuracy, while Table 1 also contains the computing time. The latter refers to the main steps of the two cascaded classifications—the time required by the whole procedure is also provided. As can be observed, the whole process reaches high accuracy rates, especially after the second classification step, and is fast even on a moderately performing laptop, paving the way for a future real-time process.

3.2. Yellow Grapes

With regard to the yellow grapes, again, the steps in Figure 3 were performed. In particular, the input image was converted into the complementary CMY and filtered with the same filter. The feature vector, whose components are defined in Equations (7) and (8) and shown in Figure 10, was employed in the first classification whose result is depicted in Figure 11. The second classification in the filtered HSV space led to the $M_2$ map in Figure 12 (Left), while Figure 12 (Right) shows its post processed version. The final result is shown in Figure 13.

As for blue grapes, many classifications were made, and only a subset of them are shown in this section. In

Table 3. Yellow grapes: Classification accuracy (%) for a decreasing size of the training set in both phases. The training time and computing time, measured in seconds (s), required for building the classification maps were also provided for each step of the proposed method. The last column refers to the processing time of the whole procedure.

	N° POINTS	ACCURACY	TIME (s)
			training	map	total
First classification	100	92.2	1.20	31.44
Second classification	100	95.8	2.04	16.07	50.74
First classification	90	92.2	1.037	32.07
Second classification	90	95.8	2.271	15.413	50.79
First classification	80	91.7	1.053	31.930
Second classification	80	95.8	2.510	17.748	53.24
First classification	70	91.8	1.017	31.459
Second classification	70	95.8	2.170	17.117	51.76
First classification	60	91.8	1.107	30.480
Second classification	60	95.7	2.070	16.440	50.09
First classification	50	91.6	1.084	31.217
Second classification	50	95.6	2.240	16.308	50.85
First classification	40	92.3	1.111	30.643
Second classification	40	95.5	2.030	15.244	49.03
First classification	30	92.6	2.275	35.295
Second classification	30	95.2	3,058	21,588	62.22
First classification	20	92.3	1.418	36.640
Second classification	20	95.0	2.459	17.149	57.67

, the classification accuracy for a decreasing size of the training set is reported. The need for a small training set is confirmed even in this case. However,

Table 4. Yellow grapes: Classification accuracy (%) for only 10 points in the training set—the results for six different runs are presented.

	N° POINTS	ACCURACY (%)
First classification	10	93.0
Second classification	10	94.8
First classification	10	91.1
Second classification	10	94.3
First classification	10	89.9
Second classification	10	94.5
First classification	10	88.0
Second classification	10	93.7
First classification	10	92.2
Second classification	10	95.1
First classification	10	94.5
Second classification	10	94.8

shows that having only 10 points in the training set is not always sufficient to guarantee a classification accuracy greater than $95\%$. This can be easily explained with the intuitive observation that a fast recognition of yellow grapes on a general yellowish/greenish background makes the problem more difficult than for blue grapes.

A larger training set is required in this case.

Table 5. Yellow grapes: Classification accuracy (%) for 20 points in the training set.

	N° POINTS	ACCURACY (%)
First classification	20	93.5
Second classification	20	95.3
First classification	20	92.5
Second classification	20	95.5
First classification	20	92.9
Second classification	20	95.8
First classification	20	91.3
Second classification	20	95.2
First classification	20	92.3
Second classification	20	95.0
First classification	20	92.0
Second classification	20	95.1

refers to some trials where 20 points were used for training. As can be observed, 20 points in the training set guarantees a final classification rate greater than $95\%$ in this case. Finally, Figure 14 gives evidence of the better quality of the classification provided by the proposed method using 20 points in the training set when compared to the one achieved using only 10 points. As far it concerns the computing time, this is comparable to that required for processing blue grapes.

3.3. Comparative Studies and Discussions

The proposed approach is compared with the one presented in [1] since it is the most similar in spirit to the proposed one. This approach is based on three main phases: pre-processing, dataset training, and classification through an SVM classifier. Its main ingredients are the HSV space (as in the proposed approach) and Otsu’s threshold along with some morphological operations on the resulting binary maps. Successively, regions of interest (ROIs) are found and a classification on vectors involving various (i.e., 14) features, both geometrical and statistical, is performed.

Specifically, the adopted features are: closeness, extent, compactness, texture, H mean, H and S average contrast, H S and V smoothness, S third moment, H and V uniformity and H and S entropy. It is worth outlining that these features were selected among a larger set via two well-known techniques: ReliefF algorithm [97] and sequential feature selection [34]. The selected features are used for the SVM classification. The accuracy rates achieved on the ’in-the-wild’ images in Figure 15 by the method in [1] were, respectively, $52\%$ and $50\%$.

As can be observed, even with the use of a more populated training set (120 for the first classification and 179 for the second one), the method in [1] achieved a lower classification accuracy compared with the proposed approach, reaching accuracy rates of about $95\%$ for the two images in Figure 15 using 20-point training sets in both classification steps. This is the consequence of the use of the optimized visual-perception-based features in the proposed approach.

In fact, color perception and visual contrast play a significant role in the determination of the visual saliency of the objects under study, which represents one of the main ingredients in the naked eye yield analysis performed by a winemaker. As Figure 5 (Right) and Figure 10 (Right) show, the proposed visual contrast allows greatly emphasizing the grapes with respect to the remaining image components. The role of SVM-based classification is then to automatically define the separation threshold in a data-driven fashion. On the other hand, the ML-based approach benefits from the definition of specific and relevant features for the object under study, so the use of small training sets is allowed.

In addition to the benefits discussed in the previous subsections, the selection of points in the training set must be accurate in order to prevent misclassifications—this recommendation becomes fundamental whenever the proposed procedure is embedded in a smart application that enables the winemaker to retrain the classifier. On the other hand, this can be less troublesome than acquiring a large number of images as required in DNN-based approaches.

Finally, with regard to the computing time, although the proposed procedure shows some merits with respect to competing methods due to the very simple operations employed, some further work is required to optimized some of its steps, especially the testing phase. Region-based instead of the proposed pixelwise strategies could be employed for promoting real-time processing.

4. Conclusions

In this paper, we proposed a cascaded classification method of grape bunches. Its main peculiarity is the use of the human perception mechanism of early vision in order to define proper feature vectors to use as input for the two classifiers. This property enables replicating the almost straightforward grape-bunch detection process that is performed by a winemaker. As a consequence, a very small training set (a small number of image pixels) can be used in the learning phase of the classification procedures. In addition, the method is robust to “in-the-wild” videos that are acquired in uncontrolled acquisition conditions.

These two ingredients make the proposed method implementable on smart devices in a user-friendly fashion, making it directly usable and updatable even by the winemaker. When compared with similar methods, the proposed approach showed that the selection of a smaller number of features and the adoption of a small training set are possible by adopting visual-perception-based features that have been ‘naturally’ optimized over hundreds of thousands of years. Future research will be devoted to refining the proposed computational procedure to increase its accuracy as well as to defining a unified method for both blue and yellow grapes.