Professional Documents
Culture Documents
IET Smart Cities - 2020 - Huang - Smart Agriculture Real Time Classification of Green Coffee Beans by Using A
IET Smart Cities - 2020 - Huang - Smart Agriculture Real Time Classification of Green Coffee Beans by Using A
IET Smart Cities - 2020 - Huang - Smart Agriculture Real Time Classification of Green Coffee Beans by Using A
Research Article
Nen-Fu Huang1 , Dong-Lin Chou1, Chia-An Lee1, Feng-Ping Wu1, An-Chi Chuang1, Yi-Hsien Chen1,
Yin-Chun Tsai1
1Department of Computer Science, National Tsing Hua University Hsinchu, Taiwan
E-mail: nfhuang@cs.nthu.edu.tw
Abstract: Coffee is an important economic crop and one of the most popular beverages worldwide. The rise of speciality
coffees has changed people's standards regarding coffee quality. However, green coffee beans are often mixed with impurities
and unpleasant beans. Therefore, this study aimed to solve the problem of time-consuming and labour-intensive manual
selection of coffee beans for speciality coffee products. The second objective of the authors’ study was to develop an automatic
coffee bean picking system. They first used image processing and data augmentation technologies to deal with the data. They
then used deep learning of the convolutional neural network to analyse the image information. Finally, they applied the training
model to connect an IP camera for recognition. They successfully divided good and bad beans. The false-positive rate was
0.1007, and the overall coffee bean recognition rate was 93%.
1 Introduction machine (SVM) [10] and near-infrared (NIR) [11]. Most methods
provide >90% of accuracy on classifying different species of coffee
Coffee is an important economic crop and one of the most popular beans. However, few methods are proposed to detect defects on
beverages in the human society [1]. Coffee is cultivated in over 70 unroasted green coffee beans. Before applying deep learning, area
countries, primarily in the equatorial regions of America, Southeast and circumference [12] of a coffee bean, are considered as
Asia, the Indian subcontinent, and Africa. Green coffee beans are important basis while judging, and gave us about 78.5% of
the most traded agricultural products in the world. With the rapid accuracy while inspecting coffee beans with defects. Image
increase in speciality coffee retailers and cafes in the 1990s, processing technique and threshold applying [13] can also help to
speciality coffee became one of the fastest-growing markets for the inspect defects on green coffee beans with about 83% of accuracy.
foodservice industry. Many countries have developed their own Currently, many vision sorting systems [14] are available in the
speciality coffee associations. According to the Specialty Coffee market. These systems can be used to distinguish good and bad
Association of America (SCAA), a cup of speciality coffee is not varieties of items such as peanuts, seeds, rice, and green coffee
defined as a cup of coffee that has been brewed and sent to beans mainly on the basis of colour. However, a range of colours
consumers. Instead, it emphasises the whole process of producing must be set for these systems, which require a robotic arm to pick
the cup of coffee. However, green coffee beans are often mixed out the items. This method is slow and inefficient. Accordingly, we
with impurities and unpleasant beans [2]. If these impurities and propose a method that involves using deep learning technology to
cowpeas are not manually picked out before roasting, the overall determine the standard of good and bad items.
coffee quality and flavour are affected. We preprocessed images of green coffee beans obtained through
In the present, the field of artificial intelligence applications can image processing technology by using the convolutional neural
be divided into three categories: speech recognition [3], image network (CNN), which is a popular technology in deep learning.
recognition, and natural language processing [4]. Image CNN is good at the colour and shape extraction of images.
recognition technology can be applied in areas such as smart cities, Therefore, we can easily get features of good and bad bean images
medical care, and agriculture. In smart cities [5], traffic flow such as partial black, broken and so on. We trained the exclusive
analysis can be used to improve traffic congestion and reduce coffee bean prediction model to quickly distinguish which raw
traffic accidents. Medical imaging [6] through MRI or computed beans were good and bad. By using this method, the considerable
tomography can be used in the early detection of disease roots and time required for the manual selection of coffee beans can be
in treatment. Agricultural imaging [7] can be used for the reduced and the development of speciality coffee beans can be
identification of crop pests, which can reduce the loss of crops promoted. An automatic green coffee bean identification system
during plantation and increase crop yields. Therefore, it is hoped
that image recognition technology can be applied for identifying
green coffee beans, thereby improving the quality and flavour of Table 1 Definition of a primary defect bean according to the
coffee. SCAA
The definition of defect beans [8] provided by the SCAA is Primary defect Number of occurrences equal to one full
presented in Tables 1 and 2. There are two types of defects: defect
primary defects and secondary defects. Through the definitions full black 1
provided by the SCAA, we can determine what type of green full sour 1
coffee beans are defective beans. If bad beans are roasted together
pod/cherry 1
with good beans, the coffee does not taste like speciality coffee. To
solve this problem, we must manually select green beans. large stones 2
However, this process involves considerable labour and time cost. medium stones 5
There are already plenty of different methods been proposed to large sticks 2
classify different species of coffee beans, including artificial neural medium sticks 5
network (ANN), K nearest neighbour (KNN) [9], support vector
The store helped us to divide the green coffee beans into good and
(i) Operating system: In our experiment, we use the Ubuntu 14.04 bad beans so that we could directly take pictures.
as the operating system for our central server. In the past, collecting data sets was a difficult job while
(ii) Hardware: In recent years, central processing unit (CPU) building a neural network. It sometimes took more time than the
performance and graphics processing unit (GPU) technology have training itself. We considered an automatic coffee bean data collect
been improved tremendously. Training time of the learning module mechanism with conveyor. In this mechanism, a vibrating bucket
through the GPU parallel computing technology can be greatly makes a pile of coffee beans line up in a lane and drop on the
reduced. We use the GPU developed by Nvidia to accelerate the conveyor. After dumping coffee beans into the vibration bucket,
training time in parallel. The hardware list of our central server is the bucket will make a pile of coffee beans line up in a lane and
shown in Table 3. Since we need to read and normalize all the send to the conveyor consecutively by vibrating. The vibrating
coffee bean images before the GPU training process, we need bucket is shown in Fig. 3.
larger memories to accommodate enough images and faster CPU A high-resolution IP camera is set above the conveyor in order
processing speed. to take a photo while coffee beans pass through, and it is connected
to a desktop computer. We have fine-tuned a pre-trained object
detection model Yolov3 with coffee beans. When the objects pass
2.2 Data collection through the camera, the model will recognise the object and then
give a command to the camera to take a photo. With this
Since insufficient pictures of coffee beans were available on the mechanism, we are able to collect data sets automatically instead of
Internet, we were required to buy green coffee beans from a coffee taking photos manually. The environment is shown in Fig. 4.
shop. Fig. 2 shows a few coffee beans we bought from the shop. We previously used a webcam to take the photos. However, we
found out that when coffee beans moved over a certain speed on
6 References
[1] Oder, T.: ‘How coffee changed the world’. Available at https://
www.mnn.com/food/beverages/stories/how-coffee-changed-the-world
[2] Pinto, C., Furukawa, J., Fukai, H., et al.: ‘Classification of green coffee bean
images based on defect types using convolutional neural network (CNN)’.
2017 Int. Conf. on Advanced Informatics, Concepts, Theory, and Applications
(ICAICTA), Bali, Indonesia, August 2017, pp. 1–5
[3] Gavat, I., Militaru, D.: ‘Deep learning in acoustic modeling for automatic
speech recognition and understanding – an overview’. 2015 Int. Conf. on
Speech Technology and Human-Computer Dialogue (SpeD), Bucharest,
Romania, October 2015, pp. 1–8
[4] Young, T., Hazarika, D., Poria, S., et al.: ‘Recent trends in deep learning
based natural language processing [review article]’, IEEE Comput. Intell.
Mag., 2018, 13, pp. 55–75
[5] Fadlullah, Z.M., Tang, F., Mao, B., et al.: ‘State-of-the-art deep learning:
evolving machine intelligence toward tomorrow's intelligent network traffic
control systems’, IEEE Commun. Surv. Tutorials, 2017, 19, pp. 2432–2455
[6] Ker, J., Wang, L., Rao, J., et al.: ‘Deep learning applications in medical image
analysis’, IEEE Access, 2018, 6, pp. 9375–9389
[7] Kamilaris, A., Prenafeta-Boldú, F.X.: ‘Deep learning in agriculture: a survey’,
Comput. Electron. Agric., 2018, 147, pp. 70–90
[8] ‘Specialty Coffee Association of America’. Available at http://
www.coffeeresearch.org/coffee/scaaclass.htm
[9] Arboleda, E.R., Fajardo, A.C., Medina, R.P.: ‘Classification of coffee bean
species using image processing, artificial neural network and k nearest
neighbors’. 2018 IEEE Int. Conf. on Innovative Research and Development,
Bangkok, Thailand, 2018, pp. 1–8
[10] Arboleda, E.R.: ‘Comparing performances of data mining algorithms for
classification of green coffee beans’, Int. J. Eng. Adv. Technol., 2019, 8, pp.
1563–1567
Fig. 14 Architecture of identification
[11] Okubo, N., Kurata, Y.: ‘Nondestructive classification analysis of green coffee
beans by using near-infrared spectroscopy’, Foods, 2019, 8, p. 82
sides of coffee beans; meanwhile, both side of the coffee bean will [12] Gunadi, I.G.A., Artha, I.P.M.K., Christyaditama, I.G.P., et al.: ‘Detection of
be inspected, respectively. coffee bean damage in the roasting process based on shape features analysis’.
2019 Int. Conf. on Mathematics and Natural Sciences, Bali, Indonesia, 2020,
The model currently runs on a desktop personal computer. vol. 1503
However, considering the cost and convenience, we are planning to [13] Arboleda, E.R., Fajardo, A.C., Medina, R.P.: ‘An image processing technique
move the whole process to an edge computing device like NVIDIA for coffee black beans identification’. 2018 IEEE Int. Conf. on Innovative
Jetson Nano or Raspberry Pi 4 with Intel Neural Compute Stick in Research and Development (ICIRD), Bangkok, Thailand, May 2018, pp. 1–5
[14] Tho, T.P., Thinh, N.T., Bich, N.H.: ‘Design and development of the vision
the future. After replacing the computing unit, the device will sorting system’. 2016 3rd Int. Conf. on Green Technology and Sustainable
become easier to transport and less costly for users. Development (GTSD), Kaohsiung, Taiwan, November 2016, pp. 217–223
Artificial intelligence is feasible for the image recognition of [15] Zhu, S., Xia, X., Zhang, Q., et al.: ‘An image segmentation algorithm in
green coffee beans, and it can provide accurate and efficient image processing based on threshold segmentation’. 2007 Third Int. IEEE
Conf. on Signal-Image Technologies and Internet-Based System, Shanghai,
results. Furthermore, good and bad beans can be accurately China, December 2007, pp. 673–678
distinguished by using a camera, which solves the problem of [16] Qin, Y., Sun, S., Ma, X., et al.: ‘A background extraction and shadow removal
spending considerable time and effort for selection. In the future, algorithm based on clustering for vibe’. 2014 Int. Conf. on Machine Learning
we hope to connect a robotic machine to select and remove bad and Cybernetics, Lanzhou, China, July 2014, vol. 1, pp. 52–57
[17] Mikołajczyk, A., Grochowski, M.: ‘Data augmentation for improving deep
beans. The blueprint of the architecture of our system is illustrated learning in image classification problem’. 2018 Int. Interdisciplinary PhD
in Fig. 14. Workshop (IIPhDW), Swinoujscie, Poland, May 2018, pp. 117–122
In Fig. 14, we combine our system architecture with colour [18] Zaheer, R., Shaziya, H.: ‘GPU-based empirical evaluation of activation
sorter machine architecture. We placed the background on the track functions in convolutional neural networks’. 2018 2nd Int. Conf. on Inventive
Systems and Control (ICISC), Coimbatore, India, January 2018, pp. 769–773
and instantly identified it through the original webcam. The brown [19] Kalman, B.L., Kwasny, S.C.: ‘Why tanh: choosing a sigmoidal function’.
objects are good coffee beans, and the yellow ones are bad coffee [Proc. 1992] IJCNN Int. Joint Conf. on Neural Networks, Beijing, China,
beans. We replaced the original CCD technology or infrared June 1992, vol. 4, pp. 578–581
technology with a deep learning model and used the colour sorter [20] Boughorbel, S., Jarray, F., El-Anbari, M.: ‘Optimal classifier for imbalanced
data using Matthews correlation coefficient metric’, PLOS ONE, 2017, 12, p.
machine selection method. Then, we used an air gun to separate the e0177678
coffee beans. Finally, we separated the coffee beans into two [21] Chou, D.L.: ‘Real-time classification of green coffee beans by using a CNN’.
containers. Available at https://youtu.be/aqzs3o8z08