INNOVATIVE TECHNIQUES FOR FRUIT COLOR GRADING Paolo Gay, Remigio Berruto
1. INTRODUCTION
The external appearance is one of the most important factors that affects the value of the fruits. Consumers have developed distinct correlations between color and the overall quality of the specific product. In the case of apples, for example, the consumers are attracted by the color hue and its distribution on the surface. Such aspects are in general empirically graded by experts. Some research efforts are devoted to the development of new cultivars and the optimization of cultivation techniques that could improve color attributes [5],[6],[8]. One of the problems encountered by researchers is the evaluation and the comparison of the color properties of different lot of fruits. The standard method for the evaluation of the surface color is based on the use of colorimeters. Unfortunately they do not provide global indexes but only punctual information that could be biased by the operator’s choice of the measurements point on the surface. This paper presents a new device for the analysis of the color of the whole surface together with new indexes and algorithms for the characterization and classification of the fruits.
Memoria presentata il 04/12/2002. ING. PAOLO GAY, Ricercatore (
[email protected]); PROF. REMIGIO BERRUTO, Associato (
[email protected]); D.E.I.A.F.A. Sezione Meccanica, Università degli Studi di Torino. Il contributo al presente lavoro è da ritenersi equamente suddiviso tra gli autori.
The contribution of this paper differs from classical work [1][2][7][9][10][11][15] and [16] on fruit color analysis because the presented results are not intended for fruit sorting machine applications but could be useful for the development of measurement instruments for technicians and researchers. In this context, instead of speed of processing and machine reliability, aspects as high accuracy and measurements repeatability are of major interest. Two different indices are introduced: the first one is aimed to emulate human judgment and needs to be trained with an initial set of fruits. The second one allows to evidence color grade differences among different lots of fruits. The numerical evaluation of the whole fruit surface according the proposed methods makes the color assessment process reliable and repeatable. In order to show the effectiveness of the proposed techniques case studies, concerning different lots of apples, have been reported. 2. MATERIALS AND METHODS 2.1. THE IMAGE ACQUISITION SYSTEM A new device, reported in figure 1, has been designed and implemented for the image acquisition in controlled environmental conditions of fruit make rotating by an electrical motor. Images are synchronously acquired by a (fixed) CCD camera during the rotation of the fruit. The device is based on box containing the illumination system, a CCD camera (JAI 1
S3200) and a motorized rotating support for the fruit. The camera is connected to a National Instruments PCI1411 grabbing and acquisition board installed on a standard PC. The fruit is placed by the operator on the support which rotates at about one rpm speed (fig. 2.a). The camera acquires 30 frame per second that corresponds to a number of 1800 images per fruit. From this huge set of pictures, for each fruit a subset of 32 images only is extracted and stored. The 32 images are selected using a thin optical reference placed on the fruit support. In this way the image acquisition is synchronized with the rotation and does not need any external and auxiliary device. The rotation mechanism ensures that the almost-whole surface of the apple is homogeneously evaluated. The obtained images are numerically processed to obtain an approximated cylindrical projection of the fruit surface as will be discussed in the next section.
Fig. 1 - The device for the acquisition of the images of the fruit
The software for the acquisition, as well as the user interface, have been developed using LabView® 6.0, while the algorithms for the elaboration of the images have been implemented using Matlab® 6.0 together with the Image Processing Toolbox.
2.2. IMAGE PRE-PROCESSING
The 32 acquired images are used to compute an approximated cylindrical projection of the fruit. Each image concerns a thin strip of the fruit surface orthogonal to the optical line of the camera. This configuration reduces distortion and diffraction effects due to the curvature of the fruit surface. Assumed Fj, i the i -th out of 32 image of the surface of the j-th fruit and F j the image of the cylindrical projection of the j-th fruit, lets denote as F jRGB ?x, y ?? ? 3 and
FjHSV ?x , y ?? ? 3 the two vectors containing the three red-green-blue (RGB) and huesaturation-value (HSV) components respectively of a generic pixel at coordinate ?x, y ? of the image FJ . For more information about color space coordinates and conversion the reader could refer to [4][13][14] and [17]. Each strip Fj, i partially overlaps its two
adjacent images Fj , i? 1 and Fj , i? 1 . The size and the position of the overlapping region of any two successive Fj, i and Fj ,i ? 1 images vary for different indexes i and depend, among other, on the shape of the fruit, its symmetry and its position on the rotating support. The approximated cylindrical projection F j of the fruit can be obtained suitably connecting the 32 strips, eliminating possible overlapping regions. An efficient technique for registering a pair of gray-scale images subject to unknown displacement differences (see figure 2.b) can be found in [14]. The main idea consists in computing the normalized cross correlation function R?m, n ? between the image pair, determine the displacement offset coordinates ?m opt, nopt ? of the correlation function peak and then shift one of the images with respect to the other by the offset coordinates.
2
y
F j, i? 1 ?
Rotating support
F j ,i
CCD
n0 m0
apple
x Fig. 2.a (on the left) - The structure of the image acquisition system with the rotating support; (2.b – on the right) - the overlapping acquired images displaced of
This procedure can be extended to color images as described in the following. Given any couple of images Fj, i and Fj ,i ? 1 , the overlapping region can be found solving the optimization problem
?m
opt
The optimal merging of two adjacent strips can then be obtained joining the two images according the ?mopt, nopt ? displacement, as reported in figure 2.b. The connection of all the strips leads to the approximated cylindrical approximation of the fruit. An example of such a image is reported in figure 3. The second task of the preprocessing phase concerns the segmentation of the fruit image from the background. This has been performed applying a set of well established heuristic threshold rules on both the RGB and HSV representations of the image as reported in [4] and [14].
, nopt ? ? arg max R ?m, n ? (1) m ,n
where the normalized cross correlation R?m, n ? function is defined as (2) and is computed on a rectangular kernel region of the image of vertices coordinates ?xmin , y min ?, ?xmin , ymax ?, ?xmax , ymin ?and ?xmax , y max ?. The introduction of relation (2) allows to extend to color images the algorithm presented in [14].
? ? ?F ?x , y ?? x max
R?m, n ? ?
y max
x ? xmin y ? y min
? ?? ?x ? xmin x max
? ?F ?x, y ?? y max
y ? y min
RGB j, i
T
RGB j, i
1 2
?m0,n0? pixels.
? ? FjRGB ,i ?x, y ?? ? ? ? ?x ? x min x max
T
? ? FjRGB ,i ? 1 x ? m, y ? n
? ?F ?x ? m, y ? n ?? y max
y ? y min
RGB j, i ? 1
T
(2)
1 2
? FjRGB , i ? 1 ?x ? m , y ? n ?? ?
3
Fig. 3 – Approximated cylindrical projection obtained using the proposed algorithm for the optimal merging of frames.
3. FRUIT COLOR GRADING AND EVALUATION The obtained approximated cylindrical projection of the fruit is a huge set of data that could be not directly used by operators for fruit color evaluation and classification. The information contained in the data set has to be suitably reduced and organized in order to obtain an index or a simple curve describing the distribution of the color on the fruit surface. To this extent, in this section two new algorithms for direct color evaluation are presented. 3.1.
FRUIT EVALUATION
BUSH
COLOR
EXTENSION
One of the most diffused criteria for apple classification is the evaluation of the percentage of bush color surface. This percentage is generally evaluated by expert worker’s eye.
From a numeric point of view, the simplest and most effective way to quantify the area of foreground color of the surface of the fruit consists in the evaluation of the hue histogram. By defining hue threshold levels it is possible to get the percentage distribution of certain hue ranges within a fruit image. This could be performed, for example, by simply counting pixels with specified hues. The classification method is effective only if, as in our case, a constant color lighting system is used. The classification procedure performed by the operator’s eyes can be therefore reconduced to this approach. The definition of a suitably tuned threshold h , splitting red and yellows hues, allows the estimation of the ?h ? and (foreground) red area S FG j
?h ? as (background) yellow area S j ? S FG j S FG j ?h ? ?
? I ?F , x , y ; h ? ? ? j
x,y ? ?
(3)
j
where the indicator function I ?F j , x, y; h ? is defined as
? 1 F jH ?x, y ? ? h, I ?F j , x , y ; h ?? ? otherwise ?0
(4)
4
and ? j is the set of all the pixel constituting the surface of fruit and cardinality.
S j ? card ??
j
? is its
The key and crucial point of this approach relies in the choice of the threshold h . Beside a first possible alternative of arbitrarily fixing the hue threshold value, a second alternative consists in choosing the color threshold using a sample of color. This is the case, for example, where samples of fruit are directly used to tune grading machine or where the reference is set using standard color patterns [3]. In this way the adopted reference color patterns, which are commonly used to evaluate the ripening of fruit, are also processed by the CCD camera of the image acquisition system. Consequently, the color pattern will be processed in the same way as the fruit. This ensures the minimum possible chromatic distortion due to the illumination system and all the optic and electronic components. The third method, presented in this paper, is suitable to take into account the consumer point of view. Since the fruit is graded to better accomplish the preference of the consumers, the threshold could be tuned extracting some information from consumers themselves or, equivalently, from a panel of experts. This approach presents some analogies with [12] where an extensive study to characterize human being in fruit selection has been performed. The threshold can be tuned asking to a panel of consumers a judgment y j about the foreground color distribution percentage of any j-th fruit belonging to a lot of n fruits and then processing these information together with the n images F j , j ? 1,? n . The optimal threshold can then be obtained minimizing the discrepancies between the human judgment and the numerically evaluated foreground surface. Formally, this corresponds to solve the following minimization problem
?r? hopt ? arg min Q?h ? r
(5)
h
with Q?h ? ?
S FG ?h ? ?y S
for a suitable r norm, 1 ? r ? ? . For the practical case of r ? 2 and r ? ? , relation (5) respectively rewrites
?2?
hopt
? S FG ?h ? ?? ? arg min ? ? j ? yj ? ? h Sj j ?1 ? ?
2
n
(6)
and
?? ?
hopt ? arg min max h
j
S FG j ?h ? Sj
? yj
The case of r ? 2 provides, in some sense, an average solution and can be solved as a minimum least square problem. The ?? ? threshold hopt minimizes the worst occurrence of discrepancy between human and automatic judgments.
3.1.1. EXAMPLE OF APPLICATION OF THE THRESHOLD OPTIMIZATION
As an example, the computation of the optimal threshold, via the minimization of functions in (6), for a lot of thirty apples has been performed. In figure 4 the function Q?h ? r has been plotted for the cases of r ? 2 and r ? ? . In general, the optimal
5
70
Computation of the optimal threshold using the L2 norm criterion
100
Computation of the optimal threshold using the L? norm criterion
90
60
80 50 70 40
60
30 Error (%)
50 Error (%) 40
? h( )
20 h(2)
opt
30
opt
10
20 o
o
Hue (0-360 ) 0 0 10
20
30
40
50
60
Fig. 4 - Computation of the optimal threshold
?2 ? ?? ? thresholds hopt and hopt are next each other and, as can also be seen from the example of figure (4), allow to achieve small discrepancies in the consumer and automatic grading of the ratio of foreground and background fruit colors. The use of the ?2 ? optimal threshold hopt reported in figure 4 allows to estimate the percentage area of the foreground color with an error which is in the average less than 2%. This means that for the taken sample, the system grading error vs. the expert judgments of the red color surface extension is on average very small, about 2%. However, the most effective factors that affect the level of discrepancy are the objectiveness, the repeatability and the accuracy of the human judgments used to tune the threshold.
3.2. INDEX ON HUE OF THE FRUIT
The foreground color percentage distribution is an index that could be satisfactorily used for classification and sorting purpose but it could be not appropriate to evidence small color grade differences among different lots of fruits. This could be the case when lots of fruit of different cultivar
10
Hue (0-360 ) 0 10
20
30
40
50
60
(2 ) (? ) hopt (on the left) and hopt (on the right)
or grown according different agronomic techniques need to be compared in terms of external appearance. A typical example are the efforts made by the researchers to obtain apples with very intense and distributed red surface (see e.g. [5][6] and [8]). As already mentioned, the influence on the external appearance of fruit due to cultivar selection and the development of new techniques can not be evaluated using standard colorimeter because it provides only punctual information, while observations should concern the widest as possible surface of the fruit. An assessed method for the analysis of the color distribution of a fruit consists in the evaluation of the histogram of the hue in FjH . This corresponds in dividing the hue range (0?? 360?) in adjacent classes and evaluating the number of pixel of Fj belonging to any class. In the case of a lot of fruits, a mean histogram can be computed averaging the histogram of the fruits. An example of such a histogram for two different lots of thirty fruits apples is reported in figure (5). When two or more thesis need to be evaluated, their respective histograms should to be compared or intersected.
6
H
Histogram of FH for the first lot of fruits
Histogram of F for the second lot of fruits
0.05
0.05
0.04
0.04
0.03
0.03
0.02
0.02
0.01 Normalized frequencies 0 0
0.01 frequencies Normalized o
Hue (0-360 )
Hue (0-360o)
10
20
30
40
50
60
0 0
10
20
30
40
50
60
Fig. 5 - Histograms of the averaged distribution of hue for two different lot of fruits
However, the results of these operations could be highly influenced by the number, the size and the location of the classes constituting the histogram. Some applications in literature show the use of clusters on the hue values, for example to segregate red and green tomatoes, however they do not provide an index on the color of the single or of the tomato sample. To overcame this problem lets introduce the cumulative function
Comparison of the ? (H) functions for two lots of fruit 1 0.9 0.8 0.7 0.6 0.5 (H) Second lot of fruit ? 0.4 0.3
First lot of fruit
0.2 0.1
o
Hue (0-360 )
? j ?H ? ?
H
1 ?h ?dh S FG ? Sj 0 j
(7)
0 0
10
Fig. 6 - The
20
30
40
50
60
? ?H ? function relative to the two
different lot of fruits of Figure 5.
which describes the normalized area of the fruit surface that presents a hue less than H . When the surface of a lot of m fruits need to be analyzed, relation (7) could be generalized as
? ?H ? ?
1 m S j ?h ? ? S dh m? j 0 j ?1 H
FG
(8)
and could be used to derive the average distribution of the color over the entire lot of fruits. It obviously results, by construction, that ? j ?0? ? 0 and ? j 360 ? ? 1 (and ? j ?0? ? 0 ,
?
? j ?360 ?? ? 1 , respectively).
?
In figure (6) the two cumulative ? ?H ? functions relative to the two lots of apples whose histograms are reported in Figure (5) are plotted. The cumulative function indicates that the fruits of the second lot present an average 10% wider area characterized by the hue parameter less than 25?. This means that, at least in the average, fruits of the second lot present a foreground (red) area more extended than fruits of the first lot, making them best suited for the consumer.
7
3.2.1. EXAMPLE OF APPLICATION In this last example three different thesis have been evaluated. Each thesis is constituted by a lot of thirty apples harvested on August 2000 and concerns different soil maintenance techniques. The baseline [B] consists in apples grown in an orchard with grass cover. In the second experiment [T] the soil of the orchard was tilled while in the third one [W] wood chips as mulching media were used. These fruits samples were not distinguishable by the colorimeter, while they are differentially shown by the algorithm presented in this paper (Figure 7). In result, in fact, that the baseline curve is higher for hue less than 20?, while the others are indistinguishable. This means that the sample of apples taken from the orchard with grass ([B] thesis) was colored (red) better than the other two thesis (compare, in figure 7, the plots for hue less then 20?). Averaged ? (H) ? 1 std of three different lot of fruits 1 0.8 [B]
0.6
[W] [T]
0.4 0.2 Hue (0-360 o) 0 0
10
20
Fig. 7 - The averaged
30
40
50
60
? ?H ? function ? 1 standard
deviation – the baseline [B] (solid line), the wood chip mulching thesis [W] (dotted line) and the tilled thesis [T] (dashed line)
4. CONCLUSIONS Surface color is one of the external parameters that most influences consumers. The aim of this paper is to present a new methodology that could be used by
technicians and researchers for precise evaluation of the color distribution on the whole surface of the fruit. This hw/sw device allows to derive an approximation of the cylindrical projection of the fruit surface on the base of a set of images taken during the rotation of the fruit. This representation allows to improve the accuracy in the evaluation of surfaces, reducing optical distortions due to fruit curvature and irregularities. This approach extends the results obtained by the sorting devices based on two or more images of the fruit. Two different algorithms for color analysis and grading have been developed and presented together with some basic examples showing the way of using them. The first one allows to obtain a color threshold able to emulate the human being in fruit color evaluation. In the reported example the proposed algorithm achieve an average error less than 2% taking an average of bush color extension on samples of 30 apples, vs. the human evaluation given by the panel of experts. Beside this result, the key point of this promising technique relises in the fact that the system could be trained on different sets of data, arising from different classes of consumers of different countries. This could be the case, for example, of children, of aged people, Asian, European or US people. This flexibility allows to grade fruit according to the consumer group target expectation used as the reference, rather than on absolute values of red areas extension compared to color standards charts. The second application introduce a new method to represent the whole information about the fruit color in one compact, clear and understandable chart. Both these application are currently used to evaluate agronomic techniques effects and fruit ripeness in a nondestructive way. The system has been tested on the same cultivar of apples (Royal Gala), grown according different cultivation techniques. The samples taken from these trials have been analyzed and compared. The use of the proposed algorithm allows to detect and quantify the influence of the cultivation technique on the surface color of the fruit 8
sample which often could be not evidenced with standard colorimeter. The case study reported in this paper concerns comparative analysis of apples, the system can be easily trained for the analysis of other fruits, like peaches, apricots, plumes and oranges. Moreover, since the method is truly non-destructive, the same sample could be examined many times to test the effect of ripening and storage conditions on the fruit color. 5. ACKNOWLEDGEMENTS This work was partially supported by grant CNRG002E32 and COFIN 2000 MM07157922. SUMMARY This paper presents a new methodology that could be used by technicians and researchers for the precise evaluation of the color distribution on the whole surface of the fruit. This methodology is based on the computation of the approximation of the cylindrical projection of the fruit surface that allows to improve the accuracy, reducing optical distortions due to fruit curvature and irregularities. The obtained projection is then elaborated by two algorithms for color analysis and grading that have been developed and presented together with some basic examples showing the way of using them.
REFERENCES [1] CALPE J., P LA F., MONFORT J., DIAZ P., BOADA J.C., Robust low-cost vision system for fruit grading, 8th Mediterranean Electrotechnical Conference MELECON '96, (1996), 3, 1710-1713. [2] CHANG W.H., HUANG P.Y., CHEN Y.Y., Vision based fruit sorting system using measures of fuzziness and degree of matching, IEEE International Conference on Systems, Man, and Cybernetics, Humans, Information and Technology, (1994), 3, 2600 -2604. [3] DELWICHE, M.J., Grader performance using a peach ground color maturity chart. J. Americ. Soc. Hort. Sci., (1987),22(1): 87-89
[4] GONZALEZ R.C., W OODS R.E., Digital image processing, Addison Wesley, (1993). [5] IGLESIAS I., SALVIA J., TORGUET L., CABÙS C., Orchard cooling with overtree microsprinkler irrigation to improve fruit colour and quality of ‘Topred Delicious’ apples, Scientia Horticulturae, (2002), 93, 39-51. [6] JU Z., DUAN Y., JU Z., Effects of covering the orchard floor with reflecting films on pigment accumulation and fruit coloration in ‘Fuji’ apples, Scientia Horticulturae, (1999), 82, 47-56. [7] KAY G., DE JAGER G., A versatile colour system capable of fruit sorting and accurate obiect classification, (1992), Proceedings of South African Symposium on Communications and Signal Processing (COMSIG '92), (1992), 145 -148. [8] KAYS S.J., Preharvest factors affecting appearance, Postharvest Biology and Technology, (1999), 15, 233-247. [9] KONDO N., AHMAD U., MONTA M., MURASE H., Machine vision based quality evaluation of Iyokan orange fruit using neural networks, Computer and Electronic in Agriculture,(2000),29,135-147. [10] MILLER, B.K., DELWICHE M.J., A color vision system for peach grading. Trans. ASAE, (1989), 32(4), 1484-1490. [11] NAKANO K., Application of neural networks to the color grading of apples, Computers and electronics in agriculture, (1997), 18, 105-116. [12] P AULUS I., DE BUSSCHER R., SCHREVENS E., Use of image analysis to investigate human quality classification of apples. J. Agric. Engng. Res. (1997), 68,341-353. [13] P HILIPP I., RATH T., Improving plant discrimination in image processing by use of different colour space transformation, Computers and Electronics in Agriculture,(2002),35,1-15. [14] P RATT W.K., Digital image processing, (1991), New York, John Wiley & Sons. [15] REHKUGLER G.E., THROOP J.A., Apple sorting with machine vision, Trans. of ASAE, (1986), 29 (5), 1388-1397. [16] RENNICK G., A TTIKIOUZEL Y. Z AKNICH A., Machine grading and blemish detection in apples, Proc. Fifth Int. Symp. On Signal Processing and its Applications, (1999), Brisbane, Australia. 567-570.
9
[17] CERRUTO, E., SCHILLACI, G., Sistemi di visione per il riconoscimento di frutti in alberi di agrumi, Rivista di Ingegneria Agraria, (1996), 27, 245-254 (in italian).
Key words: fruit grading, image analysis, sorting, automatic color assessment,
10
