J. agric. Engng Res.

(1988) 41, l-9

Automatic Detection of Surface Defects on Fruit

by Using a Vision System

A. DAVENEL; CH. GUIZARD; T. LABARRE; F. SEVILA*

The OECD standards for the fresh market make it necessary to estimate the quality of fruit by
grading by size, colour and surface defects. While size and colour grading are now automated,
sorting of damaged fruit is still done manually.
A system for automatic detection of surface defects on Golden Delicious apples has been
developed and tested on line, on a conveyor system used for automatic colour grading. While the
fruit is rolling, a solid-state camera takes four pictures, and so is able to view most of the surface.
A hardware system has been developed to reduce the amount of data to be processed. This is
based on a model which assumes the apple to be spherical with variable diameter, according to
the fruit size. The system can analyse more than five fruit per second per grading line. The tests
on line showed that 69% of the fruit were correctly graded, but 26% were classified immediately
above or below the right grade. The rough model employed allowed only marketable fruit
(grades 1 and 2) to be distinguished from the unmarketable fruit (grades 3 and waste), and the
grade 3 to be distinguished from the waste. Consequently, this system can be used only during
the picking period, in order to sort the fruit unmarketable as fresh fruit. However, it can also be
used for the pre-size grading operations before storage, since it gives the weight with a standard
deviation of f 5 g (r = 0.985).

1. Introduction
The European Market standards for fresh fruit and vegetables’ require some precise
grading specifications. The sorting and packing plants select the fruit in order to constitute
homogeneous lots in the various grades. Such operations as size (or weight) and now colour
sorting are automated, but the selection according to surface defects, such as blemishes,
injuries and bad shape is still a tedious, difficult hand task, despite recent improvements.
These investigations were initiated to design, with the collaboration of an equipment
making company, an automatic system able to replace (totally or partially) the manual
sorting or grading of fruit according to the degree of damage. The objective was the
automatic detection and assessment of surface defects, which are currently assessed only by
the human eye.
From a commercial viewpoint, the most important fruit in France is the apple of which
1 800000 t/a are produced, the Golden Delicious variety alone representing 65% of this
production. As a first step, and in order to limit the specifications, it was decided to
experiment only with this fruit.

2. Details of the investigation

The apples are classified into four quality grades and waste, but only the first three of
these (extra, grades 1 and 2) can be sold on the fresh market. Grade 3 is allotted for

* CEMAGREF Division Technologie des Equipments Agricoles et Alimentaires, Domaine de Lavalette.

BP 5095, 34033 Montpellier Cedex, France

Paper presented at AG ENG 86, Noordwijkerhout, The Netherlands, 1-4 April 1986

Received 23 April 1987; accepted in revised form 19 April 1988

1
0021-8634/88/090001+ 09 %03,00/O 0 1988 The British Society for Research in Agricultural Engineering
2 DETECTION OF FRUIT SURFACE DEFECTS

Table 1
Apples: main European quality standards, according to the grades

Grades Extra Waste

I
Colour Dark green not
admitted
Size grade Greater than Greater than Greater than 1
(diameter) 65 mm 60 mm 55 mm
Defects None Total area under Total area under Total area under
1cm* 2.5 cm2 5 cm’
Tolerance 5% of grade 1 10% of grade 2 10% of grade 3

Destination Fresh market 1 Processing I lump

processing. The main European quality standards for the various grades are briefly reported
in Table 1.
In France, at harvest, a first apple sorting is carried out before storage in a cold room. The
grade 3 fruit are sent for processing and the waste to the dump. Generally, the fruit are
stored by batches of similar size (pre-size grading). Before marketing, these batches are
sorted into the grades extra, 1 and 2, or often, only 1 and 2.
To sort according to the surface defects requires an analysis as complete as possible of the
entire fruit surface. The company working with the CEMAGREF in this research has
produced for its automatic colour sorting system (already marketed) a satisfactory conveyor
belt.
This conveyor belt, built with rotary rubber rollers, carries the fruit along while they
rotate. This rotation enables the colour detection system or a human observer to look at
most of the surface of the fruit. But, with this system, it is not possible to inspect the poles.
The shape of the rubber rollers (bicones) enables the fruit to be aligned in three to eight lines
across the conveyor belt as shown in Fig.r 2 and 2. The company had asked CEMAGREF to
integrate the new processing system for blemish detection into it.
The estimation of the surface defects, unlike the colour sorting, requires a very precise
analysis of the fruit. Even a small damaged area can downgrade the fruit. The system must
be able to analyse elementary surfaces of a few square millimetres. To do this, cameras were
used as sensors. The light colour of the Golden Delicious apples allows examination by a
monochromatic camera with a suitable filter.
The specifications agreed with the company were as follows:
automatic sorting by surface defects of the Golden Delicious apple;
elimination of the third-class fruit and of the rejection and partial definition of the
second-class fruit;
sorting before storage in a cold room (“pre-size grading”);
use of a “black and white” camera;
rate: around five fruit per second and per sorting line;
use of the present conveyor belt (bicones).
Based on these specifications, CEMAGREF designed and tested a prototype automatic
sorting unit on a conveying device provided by the company.

3. Description of the detection system

Surface defects include mechanical damage (bruise) and evidence of diseases. Decreases in
diffuse surface reflectance of visible and near infrared light after bruising were observed by
Read,* Rehkugler et ~1.~and Brown et al.’
A. DAVENEL ET AL.

CCD camera

Anaiysed fruit

Fig. I. Lateral view of the conveyor belt

Rehkugler and Throops analysed the fruit surface with a linear photo diode array camera.
We consider that the use of this technique, in order to reach the rate of five fruits per second,
would require the fruit to be rotated at an excessive speed.
For this study, we used a 12S-200 solid-state camera. This camera is equipped with a
Thomson-CSF silicon chip with a 208 x 144 rectangular matrix of pixels. The signal
delivered by the camera is digitized in 64 light grey levels, i.e. on 6 bit data. This charge
coupled device (CCD) camera allows exposure times to be varied between 2 and 20 ms. Its
spectral range is from O-4 to 1.1 urn. In order to determine the most suitable spectral zone for
detection of the surface defects, reflectance measurements were made with a double-beam
spectrometer equipped with an integrating sphere coated with BaSO,. At picking time, the
Golden Delicious apples are normally green-yellow in colour. Fig. 3 shows that, at this stage
of maturity, the best contrast between the sound and damaged parts is obtained around
550 nm. The camera was equipped with a filter centred on 550 nm with a 100 nm bandwidth.
CYCLOPE, the processing software, has been designed by the CEMAGREF: it has
required a particular interface board between the camera and a BFM 186, a 16-bit colour
microcomputer equipped with an 8086 processor and an 8 MHz clock.
Rolling on rubber bicones, the fruit exhibit all their surface. In order to analyse the
surface, several images must be taken when the fruit are rotating. In theory, three images

3 to 8
grading Blcone Running
lmes dIrectIon

Blcone

Fig. 2. Plan view of the conveyor

4 DETECTION OF FRUIT SURFACE DEFECTS

-I

90 -

80-

10 -

I I I I I I I / I
500 600 700 800 900 1000 1100 1200 1300 1400
"fll

Fig. 3. Spectral reflectance of intact and bruised tissue (Golden Delicious apple). 1, Green apple;
2, green-yellow apple; 3, yellow-green apple; 4, yellow apple; 5, bruised tissue

should be enough, but, in order to take into account that the smaller fruit turn faster than
the bigger fruit, it was decided to take four images per fruit. This choice assumed that every
part of the fruit would be viewed by the camera. Some blemished parts of the fruit were
analysed twice and their surface overestimated by around 20x, but the following processing
method does not take this into account. The rotation system was adjusted so that an average
fruit with a diameter of 75 mm rotated 90” between two images, in the direction of travel.
The prototype and its processing system (BFM, processing electronic rack, control
monitor) was set up on the conveyor belt and consisted of the following:
a diffuse lighting system (six incandescent lamps with frosted glass);
a black background made of rubber bicones separated by sheet metal coated with non-
luminescent black coating;
a 12s camera mounted about 1 m above the conveyor (Fig. I).
The camera was able to view two sorting lines and four fruit per line (Fig. 2), so that the
definition of the camera was 1.6 x 1.6 mm’.

4. Hardware data reduction system

In the literature, several image analysis algorithms were designed and tested on apples
(Graf and Rehkugler3. In 1981, these authors used a multivariable classification algorithm
to provide an automatic apple bruise detection system. More recently, Rehkugler and
Throops used a non-statistical image analysis approach. Because of the simplification of the
image processing algorithm, apple digital image processing was possible in real time at a rate
of 30 apples/minute. The present work required a rate 10 times higher.
The spherical shape of the fruit gives a black and white image composed of pixels, whose
grey levels decrease from the centre of the fruit towards its edges. However, a sound part
A. DAVENEL ET AL.

Fruct to be analysed

Relative lntenslty of the 1

fruit reflectlon I Ideal response
, /

Threshold
of surface defects

- - - - - - - Level 3
Greylevels

L
reductron
v--t. -------Level2
,
Background
___z - Level I
, ,‘-I

GeometrIcal reduction filter

Zones for one srte (8 sttes per rmogel

I Fruit site
I
Fig. 4. Scheme of the image processing
6 DETECTION OF FRUIT SURFACE DEFECTS

Sensor detects
the running blcones

I I
Control Addresses Controst Counter
unit generatlng filter arrays unit:
unit 8 arrays

GeometrIcal
reduction
filter

Doto bus

Fig. 5. Block diagram of the monochromatic processing system

towards the edge of the picture often reflects less energy than a damaged part close to the
centre (Fig. 4). Therefore, it appeared impossible to devise a simple assessment reduced to
three increasing energy levels representing the background, the damaged part and the sound
part of the fruit.
The sorting rate had to reach five fruit per second and per line. With two lines and four
faces per apple, 5 x 2 x 4 = 40 apple faces must be processed per second. Since each image
contains eight apple faces, it was necessary to analyse 40 faces/8 = 5 images per second.
Now, each image is an array of 208 x 144 pixels, and each pixel contains 6 bits of
information. The number of bits to be processed per second is therefore:
5 x 208 x 144 x 6 = 898 560 z 900000.
It is at present impossible to obtain this rate with sophisticated software processing, and
so a system was developed which includes a stage of data reduction by specialized hardware,
followed by a stage of software processing of the remaining data.

4.1. Data reduction stage by specialized hardware

The apples were considered to be spheres of variable diameter. The photographs were
synchronized with the advancing conveyor belt, so that the site of each of the eight fruit in
the image was known, i.e. four fruit per line (Fig. 2). Each fruit site is represented by a
52 x 52 pixel square, i.e. 2704 pixels (Fig. 4). Each of the eight fruit positions is divided into
2 concentric zones (Fig. 4). To each of these concentric zones are allocated electronic
counters in equal number to the N grey levels: the number of counters needed is, therefore,
8xZxN.
A. DAVENEL ET AL. 7

For instance, maintaining 64 grey levels reduced by the camera would require, for 10
zones, more than 5000 counters! In the experiment N and 2 were reduced to 6; therefore,
288 counters were required on the eight electronic boards together (Fig. .5), each of these
boards corresponding to one fruit image. For each photograph, these counters are
incremented according to the geographic situation (fruit size, zone) and to the grey level of
each of the pixels.
An address generating software, assigning the pixels to the various zones, has been
developed on the BFM computer in high level language. An adaptable matrix results from
this software, depending on the geometry of the objects to be sorted. This matrix is then
transferred onto a read only memory (ROM), thus becoming a geometric “reduction” filter.
Fig. 5 shows the operation of the system.
A sensor detects the running bicones and triggers the image sensing of the eight fruit
(through the sorting control unit);
an address generating unit assigns each pixel to concentric zones according to the
geometrical model;
a contrasting filter reduces the 64 grey levels into six levels chosen by encoding wheels;
a unit of eight counters records information corresponding to the eight visualized fruit;
the processor board processes around 300 bits of data for each image taken.

4.2. Software processing of the remaining data

An estimation of the size of each fruit is given by the following expression:
2704-(total of the counter contents at the level 1). Level 1 represents the background
(the conveyor belt).
In the experiment, the fruit were graded into six size grades. Each fruit is assigned to one
of the six size grades. Each size grade undergoes a particular processing: for each concentric
zone, a grey level threshold is allocated. The pixels with a lower grey level are considered as
surface defects (Fig. 4). The task of the processor is thus limited to totalling the contents of
some of the 288 counters, according to the size grade of each of the fruit.
One image corresponds to eight different fruit (two lines of four fruit): in order to process
every fruit, it is necessary to take into account data from four successive images.

5. Results and discussion

There were two steps in the implementation of the project. In the first, before developing
the hardware system, software simulating the hardware system was developed on the BFM
microcomputer. This system was tested on the conveyor line. This line allowed accurate
simulation of the behaviour of the fruit on a conveyor belt. The sorting rate achieved on the
“software” sorting system was 1.5 fruit per second per line. This system enabled study of the
influence of the sorting parameters, such as the thresholds, the grey levels, the number and
the limits of the concentric zones, in order to approach the optimal value of these parameters
for the Golden Delicious apple. Then, the specialized hardware system designed around an
8086 microprocessor was built and tested on the conveyor unit. The exposure time per
photographic shot was reduced to 2 ms, this time corresponding to a fruit motion of less
than 0.8 mm.
The sorting rate of five fruit per second and line has been easily achieved: as a matter of
fact, the optoelectronic and hardware system could process more than 15 fruit per second
and per line.
The experiments showed that the grading quality is strongly influenced by the fruit
position. The processing system is based on the idealization of the apple as a sphere with
8 DETECTION OF FRUIT SURFACE DEFECTS

Table 2
Predicted surface defects by automatic system compared with manual grading

Predicted grading

Manual grading Extra and I 2 3 Waste Sub-total

Extra and 1 75 20 7 1 103

2 22 49 11 4 86
3 0 3 13 2 18
Waste 0 0 2 21 23
Sub-total 97 72 33 28 230

Grade limits 1 cm2 or 2.5 cm2 or 5 cm’ or

32 pixels 80 pixels 160 pixels

variable diameter. This model can function only if the fruit turn on the bicones around a
horizontal axis running through the stem and calix cavities, in order not to confuse them
with defects.
During the study, we recorded a video tape of a sorting line equipped with “bicone”
conveyors in a local fruit sorting plant. Unfortunately, the previous condition was not
satisfactorily realized. Thus, during the tests, the fruit were correctly hand-oriented on the
upper section of the conveyor.
Table 2 shows the results obtained on a set of 230 fruit. The numbers on the table
diagonal represent the correctly sorted fruit. The numbers set above this diagonal
correspond to the undergraded fruit, the numbers set under this diagonal correspond to the
overgraded fruit. Sixty-nine percent of the fruit were correctly graded, 26% were classified in
the gride just above or beneath the right one. Only 5% of the fruit were incorrectly classified
by two grades. However, these results show that the sorting quality is near to acceptable for
distinguishing the marketable fruit (grades 1 and 2) from the unmarketable fruit (grade 3 and
waste): less than 10% of grade 3 fruit were classified as grade 2. But the sorting quality is
poor between the grades 1 and 2: more than 22% of grade 2 fruit were classified as grade 1.
Consequently, this system can be used only for sorting the unmarketable fruit on the fresh
market, just before storage in a cold room.
The various tests showed that the system is able to grade the fruit according to their size:
these data are easily obtained by estimating the projected area, which is used for defect
processing. This area is estimated four times for each fruit and the weight of the fruit can be
found with a 5 g standard deviation and a correlation coefficient of 0.985.
The grading quality of this system could be greatly improved, by increasing the number of
zones and grey levels: nevertheless, this improvement would make the hardware price even
higher than for the present design. Moreover, idealizing the apple as a variable diameter
sphere is a rough approach, especially because of the stem and calix cavities.
In order to solve this problem, the company has tested a new system to correctly orientate
the fruit so that the camera does not view the cavities. The test results show that this
problem can largely be solved. Nevertheless, the vision system is unable to take into account
the defects located around or in these cavities, so that these continue to provide a source of
error.

6. Conclusions

A prototype automatic detection system for surface defects on Golden Delicious apples
was built and tested on line. The reduction of the data acquired by a CCD matrix black and
A. DAVENEL ET AL. 9

white camera enabled the rates fixed by the company of more than five fruit examined per
second to be attained. Nevertheless, this rough model enables only the acceptable detection
of the grade 3 and waste grade fruit, according to the European standards. Therefore, this
system can be used only during the picking period to sort the fruit unmarketable on the fresh
market and, at the same time, to determine size-grading before storage.

References
’ Official Journal of the EEC 20 April 1962 (Regulation 23), 27 October 1966 and 19 December 1969
(Regulation 158, modified) and 31 July 1971 (Regulation 1641)
z Read, W. S. Optical detection of apple skin, bruise, flesh, stem and calix. Journal of Agricultural
Engineering Research 1970, 21: 291-295
3 Rehkugler, G. E.; Millier, W. F.; Pellerin, R. A.; Throop, J. A. Design criterion for apple bruise
detection by infrared radiation. Paper presented at the 1st International Congress on Engineering
and Food, Boston, MA, 1976
4 Brown, G. K.; Segerlind, L. J.; Summit, R. Near infrared reflectance of bruised apples. Transactions
of the ASAE 1974, 17(l): 17-19
5 Rehkugler, G. E.; Throop, J. A. Apple sorting with machine vision. Transactions of the ASAE 1986,
29(5): 1388-1397
6 Graf, G. L.; Rehkugler, G. E. Automatic detection of surface flaws on apples using digital image
processing. Winter Meeting of the American Society of Agricultural Engineers, 1981, Paper
no. 81-3537