FPGA-based Real-Time Citrus Classication System

Marco Aurelio Nuno-Maganda,

Yahir Hernandez-Mier

Cesar Torres-Huitzil

Josue Jimenez-Arteaga

CINVESTAV-Tamaulipas,
Instituto de Estudios Superiores
Ciudad Victoria, Tamaulipas, Mexico
de Tamaulipas,
Universidad Politecnica de Victoria,
Email: ctorres@tamps.cinvestav.mx Av. Dr. Burton E. Grossman 501 Pte,
Av. Nuevas Tecnologas 5902,
Altamira, Tamaulipas, Mexico
Ciudad Victoria, Tamaulipas, Mexico
Email: josue.jimenez@icest.edu.mx
Email: {mnunom,yhernandezm}@upv.edu.mx

AbstractCommonly, in the citrus industry in Mexico, fruit

grading is performed by humans through visual inspection.
Manual inspection implies several problems in maintaining grading consistency and sorting uniformity. Citrus classication is
normally achieved based on external visible criteria, such as
size, shape and color. This problem has been addressed using
computer vision algorithms, but an automatic grading machine
based on computer vision usually requires large computational
resources in order to perform the classication due to the
complexity of the computer vision operators. This paper describes
an embedded color-based citrus selection system implemented on
a eld programmable gate array (FPGA) device. Experimental
results show that the proposed hardware architecture is able to
process 720p color images at 30 fps. The proposed architecture
was validated on a Spartan-6 FPGA and the hardware resource
utilization is reported.

I. I NTRODUCTION
Currently, the state of Tamaulipas is the second larger
producer of citrus in Mexico. In 2011, the total production
was approximately 500,000 tonnes [1]. Post-harvest treatment
has become an essential stage in the marketing of fresh fruits
in order to maintain the freshness of the product and reach
the consumer in the best conditions. Commonly, citrus quality
grading is performed by humans manually by visual inspection
of external visible criteria, such as size, shape and color.
Manual inspection implies several problems to maintain consistency in grading and uniformity in sorting due to observer
subjectivity. Thus an embedded automatic vision system for
citrus classication in the industry is highly desirable.
An automatic citrus grading machine based on computer
vision faces some technical issues, such as: 1) high resolution
images make difcult to apply image processing operators in
real-time (30 fps); 2) accurate classication demands very high
computational resources and since the application requires
a high-performance embedded implementation, conventional
computer and traditional microcontrollers are unsuitable for
implementing the classication system. However, image preprocessing operations can be performed using a dedicated
hardware implementation, which might exhibit a faster processing, by performing several tasks in parallel with the
additional advantages of low power consumption and small
size [2]. Considering the latter issues, FPGAs are suitable
for the addressed application since they potentially provide
c 2014 IEEE
978-1-4799-2507-0/14/$31.00 

a high computational power to achieve high performance with

reduced design cycles.
Several software implementations of fruit selection systems
using computer vision operators have been reported in the
past [3], [4], [5], [6], [7], [8]. Nevertheless, the embedded
implementation of any of these techniques using an FPGA
has not been explored. In this paper, a hardware architecture
for a color-based citrus classication system is described.
The proposed architecture can be congured to classify many
ripeness levels, but for this paper, only ve ripeness levels
were used in the experimentation. This rest of the paper is
organized as follows: Section II describes the background of
image processing, spatial ltering, and color spaces used in
the citrus classication approach. Section III introduces the
proposed hardware architecture for the citrus classication
system. FPGA implementation results and their discussion are
described in section IV. Finally, in section V, conclusions and
future work are presented.
II. C ITRUS C LASSIFICATION A LGORITHM
The citrus classication performed by the proposed hardware architecture is divided into the following stages: 1) The
input image is converted from RGB color space to CIELab
color space. For the specic citrus classication, only one of
the three channels of the CIELab color space is used (b*). This
simplies the implementation of the hardware block required
to compute this conversion; 2) The resulting image is blurred
using a 5-by-5 Gaussian lter to remove detail and noise; 3)
The blurred image is thresholded by a xed value to separate
objects from the background; 4) A morphological opening is
applied to the binary image obtained from thresholding in
order to remove some of the foreground pixels from the edges
of regions of foreground pixels; 5) The opened image is ltered
by applying a 5-by-5 mean lter to preserve edges in the
image while reducing noise; 6) Finally, a pixel classication
is performed by counting the number of pixel of the region
of interest that belongs to each one of the classes. The nal
classication is performed based on the color class that obtain
the maximum number of pixels.
A. RGB to CIELab Conversion Module
The three coordinates of the CIELab color space represent
the lightness of the color (L), its position between red/magenta

and green (a*) and its position between yellow and blue (b*).
Conversion of the RGB image pixel values to the CIELab color
space is performed in two stages [9]. Firstly, a conversion of
RGB image pixel values to the CIE-XYZ tri-stimulus values
is performed by using the matrix transformation given by
equation 1.

0.412
X
Yt = 0.213
Z
0.019

0.358
0.715
0.119

0.180
R
0.072 G
0.950
B

(1)

where X, Yt and Z are the desired CIE-XYZ values and R, G

and B are the pixel values to be transformed. The second stage
transforms the CIE-XYZ values to their CIELab equivalents.
To compute the L component, equation 2 is used.

116 (Yt )1/3 , if (Yt > 0.008856)
L=
(2)
903.3 Yt ,
if (Yt 0.008856)
The components a and b are dened by equations 3 and 4,
respectively.
(3)
a = 500(f (Xt ) f (Yt ))
b = 200(f (Yt ) f (Zt ))

(4)

where Xt = X/Xn , Xn = 0.950456, Zt = Z/Zn and Zn =

1.088754, and f (t) is given by equation 5.

f (t) =

(t)1/3 ,
7.787t + 16/116,

if (t > 0.008856)
if (t 0.008856)

(5)

B. Spatial ltering
Image spatial ltering consists in displacing a lter mask,
w(x, y), over every pixel f (x, y) of an image. At each pixel
coordinate (x, y), the response of the lter mask w(x, y) is
calculated by a predened relationship, given by a sum of
products. For a mask of m n, it can be assumed that m =
2a+1 and n = 2b+1, where a and b are non-negative integers.
In general, lineal ltering of an image f of M N pixels with
a lter mask of m n is given by equation 6 [10]:
g(x, y) =

a
b



w(s, t)f (x + s, y + t)

(6)

s=a t=b

where a = (m 1)/2 and b = (n 1)/2. The linear spatial

ltering process is similar to the convolution process, so it can
be referred as the convolution of a mask with an image.
C. Thresholding
Binarization is the operation of converting a gray-scale
image into a binary image to identify the objects of interest from background. Binarization is a widely applied preprocessing step for image segmentation. Often, the burden
of segmentation is on the threshold operation, so that a
properly thresholded image leads to better segmentation. In
the proposed architecture, in order to keep the hardware as
simple as possible, a global binarization method with a xed
threshold was used in this step.

D. Binary morphology
Erosion is one of the two basic operators in the area of
mathematical morphology, the other one being dilation. The
basic effect of erosion on a binary image is to reduce the
boundaries of regions of foreground pixels. The basic effect
of dilatation on a binary image is to enlarge the boundaries
of regions of foreground pixels. When an erosion operator
is applied to an image, followed by dilation, it is called a
morphological opening lter [11]. The opening lter is used
to smooth contours (round corners), reduce image noise, and
separate objects in narrow areas.
E. Pixel Classication
For the compactness of the proposed architecture, a very
simple scheme is proposed. Every pixel of the original image
masked by the binarized image is compared to 5 xed color
patterns. The pattern with the largest number of pixels is the
nal classication given to the object. In cases where none of
the color patterns have the largest number of the pixels, an
unknown color is assigned by the classier.
III. P ROPOSED SYSTEM
A. System Overview
The proposed citrus classication architecture has 6 hardware modules that implements the algorithm briey described
in section II. Figure 1 shows the dataow and conguration
of the main blocks of the proposed system.
B. Modules of the proposed architecture
In this section, only the complex modules of the proposed
architecture are explained. The thresholding module is excluded due to its simplicity. Only three modules are explained
in detail, because Gaussian Filtering, Mean Filtering, and
Opening are performed by three different instances of the same
module, each one with a different conguration.
1) RGB to CIELab Module: This module takes as input
the RGB pixels of the input image and converts them to the
CIELab color space, where only the b* component is used.
In gure 2, the internal structure of this module is shown.
For the proposed operation, a xed-point representation is
used. The xed point format used is U (16, 16), which is
suitable to perform the required mathematical operations with
an acceptable error. Constants for equation 5 are computed
at compilation time. To compute the cube root, a LUT is
implemented, each value using the U (16, 16) representation.
The main components of this module are:
Cube Root Module LUT (LU T CBRT ). To optimize
hardware resource utilization, the input range is divided
into 512 values. The cube root is computed and stored in
each RAM location.
Address Translation Module (Y and Z). This module
performs the conversion of each input value to a valid
address where its cube root is stored.
Combinational Components Module. This module comprises a Comparator (C), a Multiplexer (M), a Multiplier
(X), an Adder (+) and a Subtractor (-).

Fig. 1: Block diagram of the proposed citrus classication

system.

2) Filitering and Opening Module: This module performs

efciently time-consuming ltering, erosion and dilation operations. The module is congured in compilation time, depending on its generic parameters. When the module is congured
as a lter, ve buffers are used to store the input image pixels.
When the module is congured as a morphological operator,
only four buffers are used to receive image pixels. Each buffer
is mapped to a double port RAM memory, where it is possible
to perform both write a read operations on the memory in
parallel. The RAM stores a single image line with 720 pixels,
where 8 bits are used to store each image pixels. In gure
3, the internal structure of this module is shown. The main
components of this module are:
Input and output buffers. These elements store a xed
number of rows of the input image depending on the
size of the lter mask.
Convolution Processing Elements (CPEs). This module
computes the ltering operation or the morphological
operation, depending of the conguration of the module.
Input and output routers. These elements read from input
buffers and route them to the CPEs. Each CPE sends the
generated pixel to the output router.
Control unit. This module performs the synchronization
of the components. It takes as input the global synchronization and the generated control signals required by the
elements of this module.
3) Classication Module: This module takes as input the
original image pixels included in the binarized version of the
input image, and counts the number of pixels that belong to
each color group. The number of color groups and the color
of each group is predened in compilation time. In gure
4, the internal structure of this module is shown. The main
components of this module are: 1) a set of counters that store
the pixels belonging to each predened color 2) a comparator
that obtains the maximum value of the counters for the current
image frame that is used as the classication result.

Fig. 2: Color conversion module.

Fig. 3: Filtering and morphological operations module.

the used images were taken in almost optimal acquisition

conditions, i.e., with a high contrast between foreground and
a black background. The input images have a resolution of
720 1280 (720p) with a 24-bit color depth.
The overall experimental system comprises a digital camera
(Sony CX-130 with HDMI interface), an Avnet Industrial
Video Processing Kit development board and a HDMI-input
monitor. The proposed architecture for citrus classication was

IV. R ESULTS
The implemented prototype was tested under a controlled
environment, aiming to develop a fruit selection system for
the citrus industry. To reduce the image processing needed,

Fig. 4: Pixel classication module.

Resource
Utilization
(Total)
Slice
registers
(184,307)
Slice
LUTs
(92,152)
Occupied
Slices
(23,038)
18KB
BlockRAMs
(268)
DSP48A1
Slices
(180)

RGBCIELab

Gaussian
or median
ltering
174

Pixel
classif.

Overall

75

Morphological
lt
163

94

735

480

735

718

523

4,550

125

274

189

175

1,714

28

25

64

TABLE I: Hardware resources used by the modules of the

proposed hardware architecture.

designed using VHDL, synthesized using Xilinx XST and

implemented in the Spartan-6 FPGA (XC6SLX150T) of the
target platform. The hardware resource utilization for each
module and for the entire design is shown in table I.
A very compact hardware architecture was obtained. It
consumes less that 10% of devices slices, less than 10% of
devices BRAMs and less that 30% of the devices multipliers.
This makes this system suitable for the addition of more
complex operators which could improve the classication.
The performance of the proposed architecture for citrus
classication system was measured. The implemented system
was capable of processing images of resolution 720p at speeds
of 30 frames per second, at a maximum clock frequency of
123 MHz. Figure 5 shows two examples of fruit images used
as input of the system (gures 5a and 5d), the binarization of
each fruit (gures 5b and 5e), and the nal classication of
each fruit (gures 5c and 5f).
The performance of the software implementation is determined by measuring the computation time required to perform
the same color classication on a PC. The software implementation is capable of processing an image with a resolution of
720p at an average speed of 2.5 fps. The PC testing platform
has an Intel Pentium Processor (G6950@2.80GHz), 4 GB
DDR3 RAM and Ubuntu 11.04 operating system. The implementation was coded in C language without multithreading,
compiled with gcc with O3 optimization ag set and using
OpenCV 2.x libraries.
V. C ONCLUSIONS AND FUTURE WORK
In this paper, a hardware architecture for citrus classication
was proposed. The proposed architecture is able to perform a
real-time classication of fruits acquired in high resolution
images. The implementation was validated on a Spartan6 FPGA, and the nal design is compact. The resource
utilization shows that is possible to include more complex
modules and additional processing steps in order to improve
the classication performance.

(a) First ripeness level

(b) Binarization of 5a

(c) Classication of 5a

(d) Third ripeness level

(e) Binarization of 5d

(f) Classication of 5d

Fig. 5: Image of citrus segmented and classied by the

proposed architecture.

As future work, the evaluation of the proposed system and

the extension for the classication to other types of fruits
will be performed. Also, the evaluation of more complex
algorithms in the current hardware implementation and the
integration of the proposed system in an automatic conveyor
platform for fruit classication will be explored.
ACKNOWLEDGMENT
This work was supported by CONACyT, Mexico, under
research grants No. 133637 and No. 99912.
R EFERENCES
[1] SAGARPA, Servicio de Informacion Agropecuaria y Pesquera. Mexico,
D.F.: Secretara de Agricultura, Ganadera, Desarrollo Rural, Pesca y
Alimentacion, 2011.
[2] D. G. Bailey, Design for Embedded Image Processing on FPGAs, 1st ed.
John Wiley & Sons, August 2011.
[3] A. Martnez-Uso, F. Pla, and P. Garca-Sevilla, Multispectral image
segmentation for fruit quality estimation, in Proceeding of the 2005
conference on Articial Intelligence Research and Development. Amsterdam, The Netherlands, The Netherlands: IOS Press, 2005, pp. 5158.
[4] Y. A. Ohali, Computer vision based date fruit grading system: Design
and implementation, Journal of King Saud University - Computer and
Information Sciences, vol. 23, no. 1, pp. 29 36, 2011.
[5] D. Unay and B. Gosselin, Stem and calyx recognition on Jonagold
apples by pattern recognition, Journal of Food Engineering, vol. 78,
no. 2, pp. 597 605, 2007.
[6] W. C. Seng and S. Mirisaee, A new method for fruits recognition
system, in Electrical Engineering and Informatics, 2009. ICEEI 09.
International Conference on, vol. 01, aug. 2009, pp. 130 134.
[7] A. Rocha, D. C. Hauagge, J. Wainer, and S. Goldenstein, Automatic
fruit and vegetable classication from images, Comput. Electron.
Agric., vol. 70, no. 1, pp. 96104, Jan. 2010. [Online]. Available:
http://dx.doi.org/10.1016/j.compag.2009.09.002
[8] P. Annamalai and W. S. Lee, Citrus yield mapping system using
machine vision, Masters thesis, University of Florida, August 2004.
[9] A. Ford and A. Roberts, Colour Space Conversions, Aug. 1998.
[Online]. Available: http://www.poynton.com/PDFs/coloureq.pdf
[10] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 2nd ed.
Prentice Hall, January 2002.
[11] P. Soille, Morphological Image Analysis: Principles and Applications,
2nd ed. Secaucus, NJ, USA: Springer-Verlag New York, Inc., 2003.