A Deep Learning Approach to Smart Refrigerator System with the

assistance of IOT

Iftekhar Mohammad Md. Shah Imran Mazumder Eman Kumar Saha

Computer Science and Engineering Computer Science and Engineering Computer Science and Engineering
American International University- American International University- American International University-
Bangladesh Bangladesh Bangladesh
Dhaka, Bangladesh Dhaka, Bangladesh Dhaka, Bangladesh
im.toufiq@gmail.com fahim.mazumder73@gmail.com sahatonoy2@gmail.com

Syeda Tasnuva Razzaque Shahrin Chowdhury

Computer Science and Engineering Computer Science and Engineering
American International University- American International University-
Bangladesh Bangladesh
Dhaka, Bangladesh Dhaka, Bangladesh
tasnuvarazzaque@gmail.com shahrin.chowdhury@gmail.com

ABSTRACT ICCA 2020, January 10–12, 2020, Dhaka, Bangladesh

Smart refrigerator refers to refrigerators that are capable of taking
decisions without human interactions. It takes decision with the
help of sensors as perception medium and software as the logical
decision maker. The main challenge set for this work was to make
it cost efficient so that anyone can own this smart refrigerator. As
a result this work focuses on developing a smart refrigerator
which uses minimum amount of sensors to perceive its
surroundings and maximum efficiency from software to make up
for the lack of sensors and intelligent decision making. Also this
work uses convolutional neural network architecture and transfer
learning technique, Inception V3 pre-trained model to detect the
items present in the refrigerator and make a decision. It shows
faster precision performance with less consumption of other
resources. This work mainly focuses on developing a theoretical
model of proposed refrigerator which will automate our daily
refrigerator activity with a small demonstration of the proposed
model with developing a CNN model that can detect food. This
paper describes the work process in detail followed by found
result and future improvements.
CCS CONCEPT
Applied Computing ~ Life and medical science
KEYWORDS
IoT, Food Classification, CNN, Deep Learning, Inception, Smart
Refrigerator, Smart Devices, Automation.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full
citation on the first page. Copyrights for components of this work owned by others
than ACM must be honoured. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from Permissions@acm.org .
© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-7778-2/20/01
https://doi.org/ 10.1145/3377049.3377111
ACM Reference format:
I. Mohammad, M. S. I. Mazumder, E. K. Saha, S. T. Razzaque and S.
Chowdhury. 2020. A Deep Learning Approach to Smart Refrigerator
System with the assistance of IOT. In Proceedings of ACM ICCA
conference (ICCA’20), ACM 7 pages.
https://doi.org/ 10.1145/3377049.3377111
1 INTRODUCTION
The Internet of things (IoT ) is a system of interrelated computing
devices, mechanical and digital machines, objects, animals or
people that are provided with unique identifiers and ability to
transfer data over a network requiring human-to-human or human-
to-computer interaction [1]. Smart refrigerators feature a
touchscreen interface and ability to connect to the Internet through
Wi-Fi to provide a number of additional features. Smart
refrigerator include internal cameras more flexible user-controlled
cooling options, and the ability for you to interact with its features
using your smartphone or tablet when away from home. Smart
refrigerator runs apps for shopping lists and recipes [3]. We can
see inside the refrigerator from anywhere in mobile devices with
built-in cameras [2]. Smart refrigerator helps to set lights or
thermostat from anywhere and can identify who is at the front
door of the refrigerator. We can also check power consumption
from mobile devices from time-to-time in a smart refrigerator.
Since the first refrigerator in the world appeared in 1923, people
are increasingly inseparable from the refrigerator. Especially in
ICCA 2020, January 10 - January 12, 2020, Dhaka, Bangladesh I. Mohammad et. al

recent decades, the refrigerator is developing faster and faster. increasingly important role in our daily life and it has brought
Smart refrigerator is constantly evolving. Most of the functions of great amenities to people’s lives as more and more food is put into
intelligent refrigerator is only limited to the original traditional the refrigerator. The application of intelligent refrigerators will
refrigerator. With the development of information technology, if give busy people a very convenient life, so that those people can
the smart refrigerator and information technology can be put more time and energy into the work and study, let the life can
combined, then smart refrigerator will make people's lives more be in an orderly way.
convenient undoubtedly. Smart refrigerators are connected
appliances that know what items are going into the fridge, their
expiry dates. The type of person who frequently forgets what goes 2 PROBLEM FORMULATION
into the fridge it’s very beneficial for those kinds of people.
Smart Devices in general is a relatively new concept in the field of
Internet of things (IoT). Despite it being a new idea, many model
Under the hood of this focus the study examines the following set
has been proposed to make a workable smart refrigerator. But
of questions: use of sensitive scanners in the refrigerator for food
almost all of those models rely heavily on pricey sensors for
preservation without the help of user in intelligent refrigerator
perceiving the environment of the refrigerator to act upon it. Also
system, determine the way of combining smart refrigerator and
many of the proposed methods are incomplete in the sense that
information technology, convenient people live in intelligent
they only monitor the inner temperatures or provide cold water on
refrigerator system, demonstrate, explore and conceive the total
glass holds. But those are not fully smart. Also there are some
networking system in intelligent refrigerator system.
models that can do more but they are very expensive for day to
day use. Keeping those things in mind we asked ourselves, which
In previous works and studies, a model of smart refrigerator was
are the interactions people do the most with refrigerator and which
proposed based on the technology of radio frequency
is more time consuming? And the answer we got was tracking the
identification, combined with the Internet and information
product usages and refilling if necessary. That's why in this
processing technology. Main goal was to record the information
research we tried to propose a model that is not that costly and can
of foods with the assistance of sensors and Internet. When the
automate our interaction with refrigerator.
food is over or a week before shelf life of the food, a warning will
be sent to the user [4]. A design technique of smart refrigerator
was proposed which is able to sense the quantity as well as quality 3 PROPOSED MODEL
of the food items kept inside it. Its main goal was to notify the
current status of food items through an android app on mobile For developing this model, we divided the problem into two
phone. It will save money and food wastage as well as help us to sections. One handles Hardware and sensor related works and
live a healthier lifestyle [5]. Researchers also represented smart another handles software related works. We also divided food
refrigerator as a container of healthy food. A research was criteria into three parts packaged foods, raw foods and processed
focused on observing the suitable meal pattern that can be food. For packaged food we will use RFID to get details of the
suggested from the food currently present in the Smart food. But it is not possible for raw and processed food. For that
Refrigerator; enforcing the information regarding healthy recipes. we use camera and image processing techniques to determine the
This model of sensor-equipped food container of a smart food, quality of food and quantity of food.
refrigerator was proposed demonstrating the performance in
detection of food. This system consists of sensor system,
information server & user applications which will suggest the
automatic monitoring of food intake that can provide intuitive
feedback to users [9]. Another proposal of a smart refrigerator
system was to determine an identity of the food item, an
associated date of the food item and a compartment containing the
food item using one or more tag readers. Also, the system may
receive item usage information when a user wishes to use the food
item and can change the temperature of the food item [10]. In a
paper, to improve the accuracy in the case of fruit recognition;
weight information along with multi convolution neural models Figure 1: General Food Classification
have been combined which is named as integrated data fusion
approach by the authors. The authors claim that this approach has
We also propose to divide our system into six modules.
got a significant improvement in the fruit recognition system
compare to any other approach previously introduced [11].
Data Collection Module: Perception is the biggest constraint
when it comes to learning or making decision. It is not any
However, with the improvement of people’s living standards and different in this case as well. To perceive our model uses two
the accelerating pace of people’s life, the refrigerator is playing an sensors, RFID sensors for packaged foods and camera sensors for
 ICCA 2020, January 10 - January12, Dhaka, Bangladesh
raw and processed foods. The processing will be done on every
switch press by the door. Our system will do two tasks, it will
trigger RFID sensor to perceive packaged food changes and it will
trigger camera sensors to take a picture of the inner side of the
fridge. After that it will send the data to our cloud application for
information processing.
Information Processing Module: After Data Collection Module
sends the data, Information Processing Module will start
processing those data. There are two main information type we
will get from Data Collection Module. One is a JSON file
containing the information gathered by RFID sensor and another
will be a picture from the camera sensors. To make the analysis
easier the image will be divided into six smaller images. Those
processed information will be sent to Analysis Module.
Figure 2: Workflow of the system
Analysis Module: Analysis module is the main brain of our
system. Analysis Module is what does the main classification and
detection of the products present in Refrigerator. There are two
main components. One is a CNN architecture that classifies food.
It will also try to detect the quality of the food and the quantity of
the food present in the refrigerator with the help of image
processing. The processing will be done on the six divided
images. Another component is a JSON analyser which will parse
the JSON, compare the current data with the previous data taken
from previous JSON to detect changes. It will also track the
products expiry dates to detect if there is any spoiled food present
in the refrigerator.
I. Mohammad et al.
Learning Module: The Main task of the learning module is to
keep the CNN model up to date. It will be achieved with the help
of tensor serving. As the model stays in the cloud, it is easier for
us to keep the model trained with all the data coming from all the
users. As a result the model keeps learning from all the images
were not present in the training sets. Also as it is possible to get
the geographical data from the request, we will be able to train our
model in an optimized way to detect foods that look alike based
on geo-location data.
Notification Module: Notification module will handle the task of
notifying users about the food stocks, quality of food. It will also
keep users up to date with the vendor information so that users
can request the vendors to refill food stock from this module. This
module also plays a big role in teaching part of the model, as it
provides us with geological data as well as preference data. This
module will also feed our model with those data so that our model
can train on the preferences of users based on geographical data.
Connection Module: Connection module is the main module that
connects all those modules together. This module consists of one
Arduino mega and one powerful cloud service to run and train the
model. The Arduino helps sensors to gather and transfer data.
Also Arduino communicates with the cloud service to send data to
cloud, also to retrieve data from cloud. It also communicates with
user and vendors for notifying and requesting refills.
4 IMPLEMENTATION
Even though this is mainly a model based research, it comes with
a small demonstration to show that it is possible to implement this
as a full-fledged solution. For the demonstration, we implemented
CNN architecture to detect food from images. It was built with the
help of Google’s Inception V3 weight and Food 101 dataset from
Kaggle. A small implementation was also done on device end
with Arduino. Arduino detects the push on the switch, takes the
pictures and information from RFID and sends them to a central
server for processing. The trained CNN Model is also on the
central server.
Dataset: There are many dataset present for food classification
on Internet. But for this research we used Food-101 As Our
dataset [6]. Food 101 contains 101 classes of food item. Those
classes contain raw foods, processed foods, and mixed foods. Also
same class foods have pictures in multiple scenarios. Every Class
has 1000 samples in them.
Figure 3: Food-101 Dataset[6]
ICCA 2020, January 10 - January 12, 2020, Dhaka, Bangladesh I. Mohammad et. al

Every class has 250 manually reviewed test images and 750 train
images. The dataset has in total 101000 images with some noise
images in them. Noises come in form of not related images,
wrong labels and intense colours. All images have maximum
length of 512 pixels. Total Size of the dataset is 4.65GB.

Data Pre-processing and Augmentation: Data Pre-processing

and augmentation is a big part of any research. It is done to
remove noises from the dataset and preparing the data for
research. The same has been done for our dataset as well. After
collecting the dataset we cleaned all unrelated images from
dataset. We wrote a script to rename all the images to its
corresponding classes. Also we resized all the images to 299x299
pixel.

Figure 4: Pre-processed Image Sample

After that we used ImageGenerator API from Keras to generate

multiple copies of the same image from multiple angles and
orientation to fulfil as much as scenarios we can create.

Inception V3 Architecture: In a neural network, there are

multiple layers of neurons connected with each other to transfer
knowledge. For image classification we use Convolutional neural
network (CNN). In fully connected architecture every layer and
neurons are fully connected with each other. As a result it is very
time consuming and process heavy. But for images it is not an
optimal solution as every single image contains thousands of pixel
of data. To solve this problem, CNN was created. In CNN the
convolution layers are not fully connected. Every neuron in CNN
learns only a single pattern. Only the final layer is a fully
connected layer. As a result transfer learning is very easy in CNN.
We only need to train our model in fully connected layer and the
pattern knowledge can come from other previously trained
weights. In this research we used transfer learning approach to
train our model. For that we use Google’s Inception V3 model to
feed our model.
Inception usages only 42 layers and 7 million parameters which is
way less than AlexNet: 60 million parameters and VGGNet: 180
million parameters [7].
Figure 5: Inception Architecture [7]
 ICCA 2020, January 10 - January12, Dhaka, Bangladesh
Inception architecture usages less computation but gives more
accuracy. Inception is basically a convolutional neural network
(CNN) which has 27 layers.
Figure 6: Inception Architecture [7]
The main idea behind this architecture is a layer called inception
layer. The inception layer is the core concept of a sparsely
connected architecture.
Figure 7: Inception Layer [7]
Model Implementation: In this research we use Inception for our
transfer learning process. Weights from Inception model was
I. Mohammad et al.
used in this training process. The total number of parameters was
over 7 million. It was tuned using stochastic gradient descent with
a learning rate of 0.0001 for 40 epochs. This model also usages 64
batch size and 140 dense neurons. Increment rate of training
accuracy was reducing after each epoch. In our model dropout
layers were used to prevent overfitting. As the activation function,
ReLu was used. Because of transfer learning technique, decrement
of loss and increment of accuracy were achieved faster initially.
Data were split into training and validation set. After the training
process, the validation set was used for evaluation of the trained
model. Out of 101000 food images of the dataset, 75750 food
images have been used for training and the remaining 25250
images were validation set.
Prediction And Model Serving: After the final model was
produced, we served the model in a cloud with the help of tensor
serving [8]. The main benefit of tensor serving is that it lets the
model learn while it is serving [8]. Also after successful learning
tensor serving replaces the old model with the new model. That
new model also keeps learning on the new data. So our model
stays up to date with all those new data we get from users. The
production model gives us the prediction using REST API request
and responses. Arduino sends the request to get the prediction
with the image as parameter. In response the model returns a
JSON data.
Hardware and Communication Module Integration: The
communication is handled by the Arduino with the help of WIFI
shield. Arduino tracks the door open and closing and activates
RFID and camera to perceive the environment and get the data.
Then the Arduino requests our cloud service to predict and send
the JSON response. This way our system maintains a bi-
directional communication.
Notification: Notification is handled by the cloud service. After
the processing is done cloud service notifies Arduino through API
response and sends a mail to user with the new information. After
that user can request the vendors to refill the stock.
5 EXPERIMENTAL RESULTS
After the research was done, the model was tested on validation
and test datasets. For performance measurement and evaluation of
our model as a food classifier, we considered training accuracy,
training loss, validation accuracy, validation loss as parameters.
These factors of evaluation shall be analysed and studied to get a
better understanding of the performance of the model.
Due to resource scarcity, we were only able to train our model 13
epochs out of 40 epochs. From that training, we achieved training
accuracy and value of loss function 0.637 and 1.988. As for the
performance of the model against validation dataset, achieved
validation accuracy and validation loss was 0.726 and 1.606.
ICCA 2020, January 10 - January 12, 2020, Dhaka, Bangladesh I. Mohammad et. al

Evaluation Results

Training Accuracy 0.637

Validation Accuracy 0.726

Training Loss 1.988

Validation Loss 1.606

Table 1: Training Results

Refrigerator Components and Cost Breakdown: As this

Figure 8: Accuracy Curve
research proposed a theoretical model of a refrigerator and no
model was fully built, we took a three compartment refrigerator as
a standard (two normal compartments and one vegetable
compartment) model for our proposed model. For each
compartment we use three cameras as it will cause less occlusion
and angle problem in images. So in total nine cameras will be
used in the model which will be controlled by a single Arduino
Uno. Also we use 3 RFID readers on each compartment to read
data from three sides of the refrigerator. In total 18 lights (6 in
each compartment) will be used to get clearer picture of the
interior. We will also use 555 timers to trigger each camera to
take images. After all consideration we came up with a minimum
of 151.29$ for our model to work. The cost breakdown is given
below:

Equipment List Quantity Price (USD)

Arduino UNO 1 4.72 $
RFID Module 9 20.27 $
Wireless Sensor 1 5.48 $
CMOS Camera 9 111.42 $
Breadboard 1 2.12 $
LED 18 0.42 $
Resistors 36 2.12 $
Jumper Wire 90 2.46 $
Push switch 1 0.05 $
Figure 9: Loss Curve
555 Timer 1 0.11
AND Gate 9 2.12 $
Total 151.29 $

Table 2: Price list of equipment

 ICCA 2020, January 10 - January12, Dhaka, Bangladesh
6 FUTURE WORK
This research shows great performance in food classification. But
it can be improved upon. Right now this research proposes an
image slicing based approach to separate food items present in the
refrigerator. But slicing an image may destroy some important
features. So it is not a great solution. Instead of using pure CNN
and image slicing, using Regional-CNN (R-CNN) may be a better
solution.
This training was not fully complete due to resource scarcity.
Completing the model training is a future work we are looking
forward to.
This research also does not take into account the occlusion
problem that may happen in the refrigerator. This will be a future
work that needs to be focused on.
Right now research proposes 3 cameras per compartment
approach to handle orientation problem. This will increase the
cost of the system. It is also not a proper approach to handle
orientation problem. So handling orientation is a future work
needs to be focused on.
Also enriching our dataset with more scenarios like occlusion,
different backgrounds and lighting will be a future work we look
forward to.
7 CONCLUSION
In this work, we have developed a convolutional neural network
model from a pre-trained model with the help of transfer learning
techniques to detect food category from given 101 food classes. It
can be said that the convolutional neural network shows great
promise in food classification. It can also be said that transfer
learning is a great approach for food classification to achieve
faster performance as we only train our model in fully connected
layer.
As for food detection, our model achieved 94% training accuracy
and 92% validation accuracy. Overall performance of our model
is promising for food classification. It also proves that it is fully
valid to use convolutional neural network and transfer learning for
this kind of classification problem.
In this first paced world, basic interactions like checking the
validity of food, refilling food stock can be a chore. Automating
these interactions not only helps us in our daily life but also opens
a new possibility in the field of HCI. It also proves that many
basic interactions that we do daily can be automated with the help
of machine learning and the Internet of things. Therefore, this
convolutional neural network-based classifier can be considered a
significant contribution in automating those interactions.
I. Mohammad et al.
REFERENCES
[1] Rouse, M. 2019. What is Internet of things (IoT)? - Definition from WhatIs.com.
IoT Agenda. https://internetofthingsagenda.techtarget.com/definition/Internet-of-
Things-IoT?fbclid=IwAR1d7IXSB7FdzTJjJBZGSsX4-
c18TjRwfK6xVHaG5HQXs7m9IItM55q6arM[Accessed 10 Aug. 2019]
[2] It's more than a fridge, it's the Family Hub. 2019. Samsung Electronics America.
https://www.samsung.com/us/explore/family-hub-
refrigerator/overview/?fbclid=IwAR2uklQLj53q1nghO1BsCACwrtDUISswhDd_dp
Vs8OUsT8mGeAzl1wRWZK4[Accessed 10 Aug. 2019]
[3] Baker, N. 2013. Smart refrigerator runs apps for shopping lists, recipes. Reuters.
https://www.reuters.com/article/us-app-refrigerator/smart-refrigerator-runs-apps-for-
shopping-lists-recipes-idUSBRE90K0PX20130121?fbclid=IwAR25kAJGxqv-
qGkkztH61ZuwcT5HNo5O9jziQfSSLV7tKS4nIzC_uWwevNU. [Accessed 10 Aug.
2019]
[4] Shouming Qiao, Hongzhen Zhu, Lijuan Zheng and Jianrui Ding. Intelligent
Refrigerator based on Internet of Things. 2017 IEEE International Conference on
Computational Science and Engineering (CSE) and IEEE International Conference
on Embedded and Ubiquitous Computing (EUC).
https://ieeexplore.ieee.org/abstract/document/8006039
[5] Deepti Singh, Preet Jain. “IoT based smart refrigerator system.” ISSN: 2278 –
909X International Journal of Advanced Research in Electronics and Communication
Engineering (IJARECE) Volume 5, Issue 7, July 2016.
https://pdfs.semanticscholar.org/1b11/038f8b71a9fcac91f1b323d57d1bcdb60fba.pdf
[6] Lukas Bossard, Matthieu Guillaumin, Luc Van Gool. “Food-101 – Mining
Discriminative Components with Random Forests.” SpringerLink, Springer, Cham, 6
Sept. 2014, link.springer.com/chapter/10.1007/978-3-319-10599-4_29.
[7] Faizan Shaikh. “Understanding Inception Network from Scratch (with Python
Codes).” Analytics Vidhya, 6 May 2019,
www.analyticsvidhya.com/blog/2018/10/understanding-inception-network-from-
scratch/. [Accessed 10 Aug. 2019]
[8] TensorFlow. (2019). Serving Models | TFX | TensorFlow. [online] Available
at: https://www.tensorflow.org/tfx/guide/serving [Accessed 14 Aug. 2019].
[9] Kwon, Taein, Eunjeong Park, Hyukjae Chang. “Smart Refrigerator for Healthcare
Using Food Image Classification.” Proceedings of the 7th ACM International
Conference on Bioinformatics, Computational Biology, and Health Informatics -
BCB '16, 2016, doi:10.1145/2975167.2985644.
[10] Luk, Bryant Genepang, Yu Tang, and Richard Chapman Bates.
"Compartmentalized smart refrigerator with automated item management." U.S.
Patent No. 9,449,208. 20 Sep. 2016.
[11] Weishan Zhang,, Yuanjie Zhang, Jia Zhai, Dehai Zhao, Liang Xu, Jiehan Zhou,
Zhongwei Li, Su Yang, “Multi-source data fusion using deep learning for smart
refrigerators”, J. ELSEVIER, February 2018, Volume 95, Page 15-21,
https://www.sciencedirect.com/science/article/abs/pii/S0166361517303755