Professional Documents
Culture Documents
Automatic Detection of Surface Defects by Using A Vision System On Fruit
Automatic Detection of Surface Defects by Using A Vision System On Fruit
The OECD standards for the fresh market make it necessary to estimate the quality of fruit by
grading by size, colour and surface defects. While size and colour grading are now automated,
sorting of damaged fruit is still done manually.
A system for automatic detection of surface defects on Golden Delicious apples has been
developed and tested on line, on a conveyor system used for automatic colour grading. While the
fruit is rolling, a solid-state camera takes four pictures, and so is able to view most of the surface.
A hardware system has been developed to reduce the amount of data to be processed. This is
based on a model which assumes the apple to be spherical with variable diameter, according to
the fruit size. The system can analyse more than five fruit per second per grading line. The tests
on line showed that 69% of the fruit were correctly graded, but 26% were classified immediately
above or below the right grade. The rough model employed allowed only marketable fruit
(grades 1 and 2) to be distinguished from the unmarketable fruit (grades 3 and waste), and the
grade 3 to be distinguished from the waste. Consequently, this system can be used only during
the picking period, in order to sort the fruit unmarketable as fresh fruit. However, it can also be
used for the pre-size grading operations before storage, since it gives the weight with a standard
deviation of f 5 g (r = 0.985).
1. Introduction
The European Market standards for fresh fruit and vegetables require some precise
grading specifications. The sorting and packing plants select the fruit in order to constitute
homogeneous lots in the various grades. Such operations as size (or weight) and now colour
sorting are automated, but the selection according to surface defects, such as blemishes,
injuries and bad shape is still a tedious, difficult hand task, despite recent improvements.
These investigations were initiated to design, with the collaboration of an equipment
making company, an automatic system able to replace (totally or partially) the manual
sorting or grading of fruit according to the degree of damage. The objective was the
automatic detection and assessment of surface defects, which are currently assessed only by
the human eye.
From a commercial viewpoint, the most important fruit in France is the apple of which
1 800000 t/a are produced, the Golden Delicious variety alone representing 65% of this
production. As a first step, and in order to limit the specifications, it was decided to
experiment only with this fruit.
Paper presented at AG ENG 86, Noordwijkerhout, The Netherlands, 1-4 April 1986
Table 1
Apples: main European quality standards, according to the grades
I
Colour Dark green not
admitted
Size grade Greater than Greater than Greater than 1
(diameter) 65 mm 60 mm 55 mm
Defects None Total area under Total area under Total area under
1cm* 2.5 cm2 5 cm
Tolerance 5% of grade 1 10% of grade 2 10% of grade 3
processing. The main European quality standards for the various grades are briefly reported
in Table 1.
In France, at harvest, a first apple sorting is carried out before storage in a cold room. The
grade 3 fruit are sent for processing and the waste to the dump. Generally, the fruit are
stored by batches of similar size (pre-size grading). Before marketing, these batches are
sorted into the grades extra, 1 and 2, or often, only 1 and 2.
To sort according to the surface defects requires an analysis as complete as possible of the
entire fruit surface. The company working with the CEMAGREF in this research has
produced for its automatic colour sorting system (already marketed) a satisfactory conveyor
belt.
This conveyor belt, built with rotary rubber rollers, carries the fruit along while they
rotate. This rotation enables the colour detection system or a human observer to look at
most of the surface of the fruit. But, with this system, it is not possible to inspect the poles.
The shape of the rubber rollers (bicones) enables the fruit to be aligned in three to eight lines
across the conveyor belt as shown in Fig.r 2 and 2. The company had asked CEMAGREF to
integrate the new processing system for blemish detection into it.
The estimation of the surface defects, unlike the colour sorting, requires a very precise
analysis of the fruit. Even a small damaged area can downgrade the fruit. The system must
be able to analyse elementary surfaces of a few square millimetres. To do this, cameras were
used as sensors. The light colour of the Golden Delicious apples allows examination by a
monochromatic camera with a suitable filter.
The specifications agreed with the company were as follows:
automatic sorting by surface defects of the Golden Delicious apple;
elimination of the third-class fruit and of the rejection and partial definition of the
second-class fruit;
sorting before storage in a cold room (pre-size grading);
use of a black and white camera;
rate: around five fruit per second and per sorting line;
use of the present conveyor belt (bicones).
Based on these specifications, CEMAGREF designed and tested a prototype automatic
sorting unit on a conveying device provided by the company.
CCD camera
Anaiysed fruit
Rehkugler and Throops analysed the fruit surface with a linear photo diode array camera.
We consider that the use of this technique, in order to reach the rate of five fruits per second,
would require the fruit to be rotated at an excessive speed.
For this study, we used a 12S-200 solid-state camera. This camera is equipped with a
Thomson-CSF silicon chip with a 208 x 144 rectangular matrix of pixels. The signal
delivered by the camera is digitized in 64 light grey levels, i.e. on 6 bit data. This charge
coupled device (CCD) camera allows exposure times to be varied between 2 and 20 ms. Its
spectral range is from O-4 to 1.1 urn. In order to determine the most suitable spectral zone for
detection of the surface defects, reflectance measurements were made with a double-beam
spectrometer equipped with an integrating sphere coated with BaSO,. At picking time, the
Golden Delicious apples are normally green-yellow in colour. Fig. 3 shows that, at this stage
of maturity, the best contrast between the sound and damaged parts is obtained around
550 nm. The camera was equipped with a filter centred on 550 nm with a 100 nm bandwidth.
CYCLOPE, the processing software, has been designed by the CEMAGREF: it has
required a particular interface board between the camera and a BFM 186, a 16-bit colour
microcomputer equipped with an 8086 processor and an 8 MHz clock.
Rolling on rubber bicones, the fruit exhibit all their surface. In order to analyse the
surface, several images must be taken when the fruit are rotating. In theory, three images
3 to 8
grading Blcone Running
lmes dIrectIon
Blcone
-I
90 -
80-
10 -
I I I I I I I / I
500 600 700 800 900 1000 1100 1200 1300 1400
"fll
Fig. 3. Spectral reflectance of intact and bruised tissue (Golden Delicious apple). 1, Green apple;
2, green-yellow apple; 3, yellow-green apple; 4, yellow apple; 5, bruised tissue
should be enough, but, in order to take into account that the smaller fruit turn faster than
the bigger fruit, it was decided to take four images per fruit. This choice assumed that every
part of the fruit would be viewed by the camera. Some blemished parts of the fruit were
analysed twice and their surface overestimated by around 20x, but the following processing
method does not take this into account. The rotation system was adjusted so that an average
fruit with a diameter of 75 mm rotated 90 between two images, in the direction of travel.
The prototype and its processing system (BFM, processing electronic rack, control
monitor) was set up on the conveyor belt and consisted of the following:
a diffuse lighting system (six incandescent lamps with frosted glass);
a black background made of rubber bicones separated by sheet metal coated with non-
luminescent black coating;
a 12s camera mounted about 1 m above the conveyor (Fig. I).
The camera was able to view two sorting lines and four fruit per line (Fig. 2), so that the
definition of the camera was 1.6 x 1.6 mm.
Fruct to be analysed
Threshold
of surface defects
- - - - - - - Level 3
Greylevels
L
reductron
v--t. -------Level2
,
Background
___z - Level I
, ,-I
I Fruit site
I
Fig. 4. Scheme of the image processing
6 DETECTION OF FRUIT SURFACE DEFECTS
Sensor detects
the running blcones
I I
Control Addresses Controst Counter
unit generatlng filter arrays unit:
unit 8 arrays
GeometrIcal
reduction
filter
Doto bus
towards the edge of the picture often reflects less energy than a damaged part close to the
centre (Fig. 4). Therefore, it appeared impossible to devise a simple assessment reduced to
three increasing energy levels representing the background, the damaged part and the sound
part of the fruit.
The sorting rate had to reach five fruit per second and per line. With two lines and four
faces per apple, 5 x 2 x 4 = 40 apple faces must be processed per second. Since each image
contains eight apple faces, it was necessary to analyse 40 faces/8 = 5 images per second.
Now, each image is an array of 208 x 144 pixels, and each pixel contains 6 bits of
information. The number of bits to be processed per second is therefore:
5 x 208 x 144 x 6 = 898 560 z 900000.
It is at present impossible to obtain this rate with sophisticated software processing, and
so a system was developed which includes a stage of data reduction by specialized hardware,
followed by a stage of software processing of the remaining data.
For instance, maintaining 64 grey levels reduced by the camera would require, for 10
zones, more than 5000 counters! In the experiment N and 2 were reduced to 6; therefore,
288 counters were required on the eight electronic boards together (Fig. .5), each of these
boards corresponding to one fruit image. For each photograph, these counters are
incremented according to the geographic situation (fruit size, zone) and to the grey level of
each of the pixels.
An address generating software, assigning the pixels to the various zones, has been
developed on the BFM computer in high level language. An adaptable matrix results from
this software, depending on the geometry of the objects to be sorted. This matrix is then
transferred onto a read only memory (ROM), thus becoming a geometric reduction filter.
Fig. 5 shows the operation of the system.
A sensor detects the running bicones and triggers the image sensing of the eight fruit
(through the sorting control unit);
an address generating unit assigns each pixel to concentric zones according to the
geometrical model;
a contrasting filter reduces the 64 grey levels into six levels chosen by encoding wheels;
a unit of eight counters records information corresponding to the eight visualized fruit;
the processor board processes around 300 bits of data for each image taken.
Table 2
Predicted surface defects by automatic system compared with manual grading
Predicted grading
variable diameter. This model can function only if the fruit turn on the bicones around a
horizontal axis running through the stem and calix cavities, in order not to confuse them
with defects.
During the study, we recorded a video tape of a sorting line equipped with bicone
conveyors in a local fruit sorting plant. Unfortunately, the previous condition was not
satisfactorily realized. Thus, during the tests, the fruit were correctly hand-oriented on the
upper section of the conveyor.
Table 2 shows the results obtained on a set of 230 fruit. The numbers on the table
diagonal represent the correctly sorted fruit. The numbers set above this diagonal
correspond to the undergraded fruit, the numbers set under this diagonal correspond to the
overgraded fruit. Sixty-nine percent of the fruit were correctly graded, 26% were classified in
the gride just above or beneath the right one. Only 5% of the fruit were incorrectly classified
by two grades. However, these results show that the sorting quality is near to acceptable for
distinguishing the marketable fruit (grades 1 and 2) from the unmarketable fruit (grade 3 and
waste): less than 10% of grade 3 fruit were classified as grade 2. But the sorting quality is
poor between the grades 1 and 2: more than 22% of grade 2 fruit were classified as grade 1.
Consequently, this system can be used only for sorting the unmarketable fruit on the fresh
market, just before storage in a cold room.
The various tests showed that the system is able to grade the fruit according to their size:
these data are easily obtained by estimating the projected area, which is used for defect
processing. This area is estimated four times for each fruit and the weight of the fruit can be
found with a 5 g standard deviation and a correlation coefficient of 0.985.
The grading quality of this system could be greatly improved, by increasing the number of
zones and grey levels: nevertheless, this improvement would make the hardware price even
higher than for the present design. Moreover, idealizing the apple as a variable diameter
sphere is a rough approach, especially because of the stem and calix cavities.
In order to solve this problem, the company has tested a new system to correctly orientate
the fruit so that the camera does not view the cavities. The test results show that this
problem can largely be solved. Nevertheless, the vision system is unable to take into account
the defects located around or in these cavities, so that these continue to provide a source of
error.
6. Conclusions
A prototype automatic detection system for surface defects on Golden Delicious apples
was built and tested on line. The reduction of the data acquired by a CCD matrix black and
A. DAVENEL ET AL. 9
white camera enabled the rates fixed by the company of more than five fruit examined per
second to be attained. Nevertheless, this rough model enables only the acceptable detection
of the grade 3 and waste grade fruit, according to the European standards. Therefore, this
system can be used only during the picking period to sort the fruit unmarketable on the fresh
market and, at the same time, to determine size-grading before storage.
References
Official Journal of the EEC 20 April 1962 (Regulation 23), 27 October 1966 and 19 December 1969
(Regulation 158, modified) and 31 July 1971 (Regulation 1641)
z Read, W. S. Optical detection of apple skin, bruise, flesh, stem and calix. Journal of Agricultural
Engineering Research 1970, 21: 291-295
3 Rehkugler, G. E.; Millier, W. F.; Pellerin, R. A.; Throop, J. A. Design criterion for apple bruise
detection by infrared radiation. Paper presented at the 1st International Congress on Engineering
and Food, Boston, MA, 1976
4 Brown, G. K.; Segerlind, L. J.; Summit, R. Near infrared reflectance of bruised apples. Transactions
of the ASAE 1974, 17(l): 17-19
5 Rehkugler, G. E.; Throop, J. A. Apple sorting with machine vision. Transactions of the ASAE 1986,
29(5): 1388-1397
6 Graf, G. L.; Rehkugler, G. E. Automatic detection of surface flaws on apples using digital image
processing. Winter Meeting of the American Society of Agricultural Engineers, 1981, Paper
no. 81-3537