Demonstration Paper
Social Media Image Recognition for Food Trend Analysis
Giuseppe Amato Paolo Bolettieri
ISTI-CNR, Pisa, Italy ISTI-CNR, Pisa, Italy
Cristina Ioana Muntean Raffaele Perego
ISTI-CNR, Pisa, Italy ISTI-CNR, Pisa, Italy
ABSTRACT
An increasing number of people share their thoughts and the images
of their lives on social media platforms. People are exposed to food
in their everyday lives and share on-line what they are eating by
means of photos taken to their dishes. The hashtag #foodporn is
constantly among the popular hashtags in Twitter and food photos
are the second most popular subject in Instagram after selfies. The
system that we propose, WorldFoodMap, captures the stream of
food photos from social media and, thanks to a CNN food image
classifier, identifies the categories of food that people are sharing.
By collecting food images from the Twitter stream and associating
food category and location to them, WorldFoodMap permits to
investigate and interactively visualize the popularity and trends of
the shared food all over the world.
KEYWORDS
food image recognition; social media streaming; deep neural net-
work
1 INTRODUCTION
We are increasingly experiencing the phenomenon called eat and
tweet where a growing number of people is sharing photos of their
cooked and intaken food through social channels like Instagram,
Twitter, Facebook. Social media statistics say that there are cur-
rently about 600 Million users on Instagram1 and about 500 million
posts are posted daily on Twitter2 . Food photos are Instagram’s sec-
ond most popular subject, surpassed only by selfies, counting more
than 300 million photos3 . This phenomenon, commonly known also
as food porn, engages people to glamorize the food they are cooking
or eating by posting beautiful and attracting pictures. Instagram
counts over 100 million posts with the #foodporn hashtag4 . These
numbers give potentially a broad and interesting measure of what
people are eating worldwide at nearly real time. However, the photo
caption or post comments not always describe the food shot (e.g.
∗ also Federal University of Pernambuco, Brazil and University of Pisa, Italy
1 https://instagram-press.com/
2 goo.gl/SjX6jo
3 goo.gl/jGvkvD
4 Taken from the Instagram web site on February 26, 2017
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
SIGIR ’17, August 07-11, 2017, Shinjuku, Tokyo, Japan
© 2017 ACM. 978-1-4503-5022-8/17/08. . . $15.00
DOI: http://dx.doi.org/10.1145/3077136.3084142
1333
SIGIR’17, August 7-11, 2017, Shinjuku, Tokyo, Japan
Vinicius Monteiro de Lira∗
ISTI-CNR, Pisa, Italy
Chiara Renso
ISTI-CNR, Pisa, Italy
beef steak or pizza), but rather they provide generic and contextual
comments (e.g. “yesterday night dinner”). Therefore, analyzing this
social phenomenon requires to recognize the dish captured by the
photo by classifying the image into a number of food categories,
and associating it with the place where the photo was taken or
where the user is from. The proposed WorldFoodMap system has
been designed and developed with the objective of visualizing, in an
interactive way, the popularity and trends in the categories of food
shared in social media worldwide. WorldFoodMap is equipped
with a image recognition engine specifically trained on food images,
methods to detect and capture the posts with food images from
media streams, methods to properly locate them on the globe, and
analysis methods for computing popularity and trends measures.
The WorldFoodMap image recognition engine leverages Deep
Learning techniques based on Deep Convolutional Neural Networks
(CNNs) [10]. We use a pre-trained GoogLeNet [12] CNN, fine-
tuned using training images from the ETHZ Food-101 dataset [4],
containing in total 101.000 images belonging to 101 food categories.
Food recognition is obtained using a k-NN classifier on the deep
features extracted from image queries and the images of a training
set composed of ETHZ Food-101 and UPMC Food-101 [13] datasets.
The potential users of WorldFoodMap may vary from re-
searchers in social media mining to domain-specific stakeholders
like, for example, health and nutrition-based experts. Although in
this paper the tool is instantiated in the food domain, the proposed
solution is not domain dependent and the whole system can be
easily instantiated in other domains by properly training a specific
image recognition network. The potential users encompass thus
domain experts interested in having a global and timely vision of a
image-based social media phenomenon (e.g., traveling and cultural
heritage fruition, selfies, cats and dogs, etc).
To the best of our knowledge, WorldFoodMap is the first pro-
posal to interactively show food trends based on media stream
images recognition. However, the research work in the context of
food recognition from photos is receiving increasing interests in
the literature. Preliminary works on visual food recognition em-
ployed aggregation of hand-crafted features, capturing mainly color
and texture information. Leveraging on HOG features and color
histograms, Kawano et al. proposed FoodCam [8], a system for
smartphones for real-time user-aided visual food recognition using
CNNs. In PlateClick [14], a food preference elicitation system uses
a deep CNN to retrieve visually similar food images from visual
quizzes for recommendations purposes. Christodoulidis et al. [5]
trained a CNN to recognize patches of an image of a dish among
only seven different food classes, using a sliding-window classifica-
tion and a majority voting scheme to predict a food class. All these
solutions are limited in the number of different food classes they
can recognize, or require a high cognitive load from the user.
Demonstration Paper
Figure 1: WorldFoodMap Architecture
2 SYSTEM ARCHITECTURE
The architecture of WorldFoodMap, illustrated in Figure 1 is or-
ganized into four layers: the Data Gathering layer collects Twitter
data from the Streaming API, the Data Processing classifies the im-
ages according to 101 food categories and identifies the location of
the post; the Data Analysis computes trends and popularity mea-
sures of food categories at a per country level, and, finally, the
Interactive Visualization layer provides an interactive web interface
from which users can query tendencies and popularities of food
categories or visualize the incoming stream of food photos.
2.1 Data Gathering
WorldFoodMap continuously collects tweets through the Stream-
ing API provided by Twitter5 . The Twitter stream is filtered for
tweets related to food by means of a list of manually selected rel-
evant keywords and hashtags (e.g., #food, #foodporn, #instafood,
etc) that can be easily changed or extended. The tweets passing the
filter are further checked for the presence of images. We discard
tweets without images since there is no visual content to analyse.
2.2 Data Processing
There are two main processing units for the information extraction
from tweets: the Image Classifier and the Location Identifier.
The Image Classifier. Visual food recognition leverages Deep
Learning techniques based on CNNs. Deep learning techniques
have outperformed previous state of the art techniques in several
applications, such as image classification [1, 3, 6], image retrieval,
and object recognition. The activation values of internal layers of a
trained CNN have been effectively used as visual features to com-
pare objects for the task on which the neural network was trained
[2, 6]. If two images are similar, the corresponding activation values
of the internal layer are similar as well. The internal layers of the
neural network somehow learn to recognize the visual aspects of
5 dev.twitter.com
1334
SIGIR’17, August 7-11, 2017, Shinjuku, Tokyo, Japan
images that are salient to the training domain. The layers closer
to the output of the neural network carry out a more semantic
knowledge (similar objects, similar scenes). The layers closer to the
input of the neural network carry out a more syntactic knowledge
(shapes, geometry, colours). In many cases, the activation values of
the chosen layer of the neural network, to be used as features, are
treated as vectors of real values, and compared using the Eucledian
distance. The visual features, extracted using this approach are
often referred to as deep features. In order to perform food recog-
nition, we used a pre-trained GoogLeNet [12], fine-tuned with a
further training process on images from the ETHZ Food-101 dataset
[4]. The fine-tuned network is used to perform deep feature ex-
traction from food images. Specifically, the activation values of the
pool5/7x7 s1 layer of the network, obtained when receiving as input
an image, are used as food specialized image deep features. Food
recognition is obtained using a k-NN classifier on the deep features
extracted from image queries and the images of the training set. As
training set for the k-NN classifier we use the union of the ETHZ
Food-101 dataset (which was also used to execute the refinement of
the GoogLeNet) and the UPMC Food-101 [13] dataset.
This approach of using a k-NN classifier on the deep features
extracted from the GoogLeNet model tuned on food images, has the
advantage of being less prone to over-fitting, especially on classes
with few training examples. Moreover, it can be easily extended to
recognize new classes of food. The feature representation, learnt
by the neural network, is used for all training images of all classes.
Classes with fewer training examples can still use the full power
of the learnt features. In order to add an additional category of
food to be recognized, it is sufficient to extract the deep features
from the new training examples, and include them in the k-NN
classification process. From some preliminary experiments, best
accuracy is obtained with k equal to 15 and in 95% of the cases the
correct food class is among the three highest ranked classes.
Location Identifier. The Location Identifier extracts city and
country information from each tweet. As a primary source of
location information we use the GPS and Place fields in the tweet,
when present. However, these data are very sparse, and they are
present in less than 10% of the tweets. A second information source
used to infer the tweet location, when the primary source is not
available, is thus the free-text user location field in the profile of the
user posting the message. We identify the city and country from the
content of this field based on data from the Geonames6 dictionary
which fed a simple “parsing and matching” heuristic procedure. This
technique provides reliable geo-location information in presence
of meaningful user location data [11]. All the tweets for which the
above geo-referencing process succeeds are used for the time series
analysis discussed below. For the streaming visualization however,
only the tweets having the GPS or Place information are used, as it
requires a precise localization.
2.3 Data Analysis
The Data analysis layer materializes items in time series from the
Data processing layer and provides analytics functionalities. The
items encapsulate the timestamp of the tweet, the attached image,
the dish category and the location from where the tweet was posted.
6 http://www.geonames.org/
Demonstration Paper
This information is used to compute long-term trends for specific
food categories and to detect short-term popularity bursts.
Trend analysis. Trends can be identified by looking at time
series over a fixed time period [7]. A trend can be either positive
or negative and it determines a tendency over the observed period.
In WorldFoodMap, we observe food trends to see how the differ-
ent food categories are trending both worldwide and in a specific
country. For finding a trend, we first correlate time with the fre-
quency of a given food category in the relevant tweets posted in
the specific time and place. Then we fit the observed data with a
simple linear regression. The objective is to find the function that
best fits the described data: the function (at + b) that minimizes the
sum( of squared errors ) between the data and the line minimizing
t [(at + b) − yt ] . The trend is derived by calculating the slope
P 2
of the least squares between the actual values and the fitted line.
Popularity analysis. Another view on how culinary habits are
changing is obtained by a short-term analysis aimed at looking if a
certain food category presents a bursty behavior. We investigate
whether a food category is growing or losing popularity by look-
ing at the deviation of frequency from the mean, observed over
a determined time interval configurable by the user. This can be
expressed by calculating the standard score [9], a measure providing
an assessment of how off-target a process is operating. The standard
x −µ
score z is calculated as z = σ , where x is the value of the current
observation, µ is the mean of the population and σ is the standard
deviation of the population. This represents the signed number
of standard deviations and observations that are above/below the
mean. A positive score indicates a datum above the mean, whereas
a negative score indicates a datum below the mean. A score of 0
means the observation is the same as the mean.
A bursty phenomenon occurs far from the average, so the stan-
dard score can be a good indicator of a trending situation. A nega-
tive score can indicate a sudden fall in popularity, whereas a positive
one indicates an increasing number of observations. For example we
may see bursty dishes in periods close to seasonal holidays, when
tweets with typical dishes show increased frequencies. Around
Thanksgiving day, in the US, we may see many images of roasted
turkey, which may appear as a bursty dish in our analysis. In ad-
dition to popularity, WorldFoodMap provide a frequency-based
analysis visualizing the frequency of a given food category for a
given temporal interval.
2.4 Interactive Visualization
WorldFoodMap enables interactive analysis through a responsive
web interface adapted either for mobile and desktop web browsers.
Such interface is designed to be intuitive and simple. Among the
interactive functionalities available through WorldFoodMap web
interface, we highlight the Trends and Popularity depicting the
worldwide trending and popular foods, and the Streaming View.
For trends and popularity we have two visualization modes:
the trends reports illustrating in a tabular form the results of the
global or country-level trends and popularity analysis, and the
worldwide map plotting the same information in a choropleth map.
The Streaming View permits, instead, to point in a worldwide map
the punctual location of geotagged and classified images gathered
1335
SIGIR’17, August 7-11, 2017, Shinjuku, Tokyo, Japan
in near-real time from the Twitter stream. All these visualizations
can be personalized and refined by using a number of user-level
settings such as the focus on a specific food category (e.g. Lasagna
Bolognese, Steak, etc), or the temporal interval when to perform
popularity and trend analysis, or the focus on a specific country.
In addition, WorldFoodMap provides functionalities for real
time notifications of rapid changes in trends of observed food. Once
the user is logged in the system, she can set up custom alerts for
following the trends of food around the world. The user specifies the
category of food, the country, a time window (e.g. last semester, last
2 years) and a temporal interval (e.g. monthly, weekly) for observing
the trend and the percentage of variation. Those alerts are triggered
and notified to the users when, for the matched configuration of
country and food category, the trending rate is higher or lower
than the specified threshold.
3 THE DEMOSTRATION
Any user may interact with the demo to browse the trends of
selected food categories or visualize the streaming information,
where tweets with the relative location and classified food category
are plotted on the map in a near-real time fashion. These two views
correspond to the modules depicted in Figure 1 and discussed in
Section 2: the popularity and trends visualization and the streaming
visualization.
Trends and Popularity Visualization. WorldFoodMap
provides two visualization modes for trending and popularity anal-
ysis: the report with a tabular view and a worldwide choropleth
map. The reports view allow the user to perform custom analysis
either by country or by food category. The reports by country show
the ranking of trending food categories for the country selected
by the user. As for example, in the Figure 2(a) WorldFoodMap
reports the trending food categories in Japan. The reports by food
category instead, show the ranking of countries where the selected
food category is trending or not, Figure 2(b) reports the trending
countries for sharing ‘Sashimi’ food. Furthermore, for both reports
the ranking can also be ordered by popularity variation and the
results can be visualized in tables, summarizing the results, or on a
line chart, following the time-series style.
The visualization on the worldwide map has a more immediate
impact, as the user can see the upward and downward trends and
popularity at country level, for a given food category, thanks to
the use of a color scale, as in Figure 2(c). Darker colors represent a
higher value, while lighter colors present a lower trend or popularity
value. By selecting one option between trends or popularity, the
user indicates whether WorldFoodMap computes trending values
or popularity variances, as detailed in Section 2.3. The user can
change the food category by selecting an item in a list of pre-
existing categories (101 food categories) learned by our classifier
model. The user can set a time-window filter for computing the
trends and popularity variations. For example, as we can see in
Figure 2(c), the user chooses “Lasagna Bolognese” from the list
of food categories, and the map displays the countries where it is
trending up (USA, Iceland) or trending down (Canada, France).
Streaming visualization. Another visualization is the world-
wide stream map, giving a real-time view of the food tweets as they
Demonstration Paper
(a) Food trends in a country
(c) Map of trends for a food category by country
Figure 2: WorldFoodMap screenshots.
appear in the network. Through this visualization it is possible to
see what people are sharing “now” all around the world. Given the
location data, we can place the object on the world map and anno-
tate it with the food category label provided by the image classifier.
The tweets remain visible in the map for a fixed temporal window
or when the density of images on the map becomes unreadable. A
simple example selecting one tweet from Japan showing an image
classified as sashimi category can be seen in Figure 2(d).
From this interface the user can visualize the stream of tweets
by selecting among a list of relevant hashtags provided by
WorldFoodMap, as #food, #breakfast, #dinner, #lunch,
#foodporn, #instafood.
More details of the tool and additional screenshots are available
at the URL: http://worldfoodmap.isti.cnr.it
Acknowledgments. This work was supported by the EC H2020
INFRAIA-1-2014-2015 SoBigData (654024).
REFERENCES
[1] Giuseppe Amato, Fabio Carrara, Fabrizio Falchi, Claudio Gennaro, Carlo Meghini,
and Claudio Vairo. 2017. Deep learning for decentralized parking lot occupancy
detection. Expert Syst. Appl. 72 (2017), 327–334.
[2] Giuseppe Amato, Fabrizio Falchi, Claudio Gennaro, and Fausto Rabitti. 2016.
YFCC100M-HNfc6: A Large-Scale Deep Features Benchmark for Similarity
Search. In SISAP 2016.
1336
SIGIR’17, August 7-11, 2017, Shinjuku, Tokyo, Japan
(b) Food trends by dish category
(d) Streaming visualization of tweets with the classified image food category
[3] Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014.
Neural codes for image retrieval. In Computer Vision-ECCV 2014. Springer.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101 – Mining
Discriminative Components with Random Forests. In Eur. Conf. on Comp. Vision.
[5] Stergios Christodoulidis, Marios Anthimopoulos, and Stavroula Mougiakakou.
2015. Food Recognition for Dietary Assessment Using Deep Convolutional Neural
Networks. Springer.
[6] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric
Tzeng, and Trevor Darrell. 2013. Decaf: A deep convolutional activation feature
for generic visual recognition. arXiv preprint arXiv:1310.1531 (2013).
[7] James Douglas Hamilton. 1994. Time series analysis. Vol. 2. Princeton university
press Princeton.
[8] Yoshiyuki Kawano and Keiji Yanai. 2015. FoodCam: A real-time food recognition
system on a smartphone. Multimedia Tools and Applications 74, 14 (2015).
[9] Erwin Kreyszig. 2007. Advanced engineering mathematics. John Wiley & Sons.
[10] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifica-
tion with deep convolutional neural networks. In Advances in neural information
processing systems. 1097–1105.
[11] Jukka-Pekka Onnela, Samuel Arbesman, Marta C González, Albert-László
Barabási, and Nicholas A Christakis. 2011. Geographic constraints on social
network groups. PLoS one 6, 4 (2011), e16939.
[12] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir
Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.
Going Deeper with Convolutions. In Computer Vision and Pattern Recognition
(CVPR). http://arxiv.org/abs/1409.4842
[13] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frédéric Pre-
cioso. 2015. Recipe recognition with large multimodal food dataset. In 2015 IEEE
International Conference on Multimedia. 1–6.
[14] Longqi Yang, Yin Cui, Fan Zhang, John P. Pollak, Serge Belongie, and Deborah
Estrin. 2015. PlateClick: Bootstrapping Food Preferences Through an Adaptive
Visual Interface. In Proceedings of the 24th ACM CIKM.