Restaurant Recommendations with Implicit

User Behavior via Image Classification and

Sentiment Analysis from Social Media
Wei Chung Yap Kelvin Kai Jie Sing Ching Pang Goh
Faculty of Computing and Faculty of Computing and Faculty of Computing and
Information Technology Information Technology Information Technology
Tunku Abdul Rahman University of Tunku Abdul Rahman University of Tunku Abdul Rahman University of
Management and Technology Management and Technology Management and Technology
Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia
yapwc-wm20@student.tarc.edu.my kelvinskj-wm20@student.tarc.edu.my gohcp@tarc.edu.my
2024 3rd International Conference on Digital Transformation and Applications (ICDXA) | 979-8-3503-7342-4/24/$31.00 ©2024 IEEE | DOI: 10.1109/ICDXA61007.2024.10470702

Abstract—Traditional recommendation systems often fall
short in understanding users' true food preferences on social
media platforms. This paper introduces an innovative approach
to enhance the dining experience by leveraging implicit user
behavior via image classification and sentiment analysis. The
proposed system combines food image classification
(MobileNetV2) and sentiment analysis (Transformer Flair) on
social media comments, creating a robust framework for
implicit recommendation. The results demonstrate the system's
efficacy in accurately predicting food categories and extracting
positive sentiments from user comments. By systematically
linking these insights to user profiles, the paper establishes a
recommendation system that considers both culinary
preferences and geographical location. The seamless integration
of computer vision and natural language processing allows for a
more tailored and personalized restaurant suggestion,
addressing the limitations of existing recommendation systems.
The research highlights the potential of harnessing implicit user
data from social media to revolutionize the landscape of
personalized dining recommendations, setting the stage for
future advancements in understanding and adapting to users'
evolving preferences.
Keywords— restaurant recommendation, image classification,
sentiment analysis, social media, implicit behavior
I. INTRODUCTION
In the digital age, recommendation systems face
challenges in providing personalized service, often relying on
inadequate methods such as content-based or explicit data-
based approaches. These systems struggle to capture users'
true preferences and sentiments, especially in the context of
social media platforms [1],[2]. Consequently, users these days
are dissatisfied with the specific recommendation system
since it does not truly learn or understand what consumers
want in meals—rather, it simply makes recommendations
based on static data such as food quality, pricing, and service
[3]. To enhance the overall user experience, it is necessary to
improve the system by harnessing implicit user behavior and
including sentiment analysis from social media. The
objectives of this review paper include:
• Utilize Social Media Data: Leverage the wealth of data
users share on social media to understand their food
preferences more comprehensively via their posts.
• Implicit Behavior Analysis: Investigate the implicit
behavior exhibited by users on social media through
distinguishing the food they share on social media
posts with the Computer Vision technique (image
classification) and natural language processing
(sentiment analysis) on users' comments to gain deeper
insights into how they perceive and interact with
specific food-related content on social media.
• Implicit Recommendation System: The ultimate goal
is to create an implicit recommendation system to
provide users with more accurate and satisfying
restaurant suggestions, enhancing their overall dining
experience.
Recent developments in dermatology have acknowledged
that skin diseases, initially thought to be benign, can
significantly impact patients' physical, social, and
psychological well-being, as well as affect those around them
[1]. Furthermore, there is a growing belief that skin diseases
may reflect internal health issues, presenting visible and
detectable signs of underlying conditions [2]. This highlights
the importance of vigilant skin monitoring by individuals.
Consequently, a home-based solution for the detection and
classification of skin diseases has been proposed in this paper,
empowering users to proactively manage their skin health.
II. RELATED WORK
A. Image Classification
A modified MobileNet architecture for food image
classification had been introduced [4] to address challenges
related to diverse food perspectives and objects. This
approach, incorporating global average pooling, batch
normalization, ReLU activation, dropout layers, and a softmax
output layer, outperforms the original MobileNet when
combined with data augmentation techniques, achieving an
accuracy of 72.59%. Later, deep learning models has been
applied for food image classification, using Food-101 and an
augmented dataset [5]. The results show that MobileNet
achieved the highest accuracy among different models,
scoring 95% on the original Food-101 dataset, while
EfficientNetB1 excelled with 96.13% accuracy on the
augmented dataset. Both studies highlight the effectiveness of
data augmentation in improving model accuracy and the
potential of MobileNet in food image classification, despite
differing image quantities per class in the datasets, warranting
further analysis.
B. Sentiment Analysis
Sentiment analysis, a vital aspect of Natural Language
Processing (NLP) known as Opinion Mining, enables the
assessment of sentiment in text, aiding in understanding the
emotional tone or opinions expressed [6]. There are 3
sentiment labels used including Positive, Neutral and
Negative. It assists in promptly addressing concerns,

979-8-3503-7342-4/24/$31.00 ©2024 IEEE 39

Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 18,2024 at 07:40:46 UTC from IEEE Xplore. Restrictions apply.
 improving decision-making, and gathering insights from selected, each containing exactly 100 images, total is
diverse data sources such as call records, social media, 5000 images.
reviews, and more, with a focus on factors like customer
satisfaction [7]. Pretrained models such as VADER, TextBlob • Social Media Food Posts: The acquisition of social
and Flair are used to determine which is more suitable to media content involved the retrieval of food-related
analyze the particular data tweeted by developers. One of the posts and comments from two major social media
studies has used pre-trained sentiment models to identify platforms, Instagram and YouTube. By searching the
public opinion to public perceptions of a topic, product, and food names as keyword that are available in the food
service provided by industry players to the government in the image dataset, 20 food samples were captured using
Indonesian language [8]. The researchers had used a lot of text the "Snipping Tool". These collected social media food
preprocessing techniques and translation before the sentiment posts were designated for deployment purposes in the
modeling part for transforming raw text data into a more image classification phase.
appropriate format [9]. The comparison result of accuracy • Social Media Comments: In parallel, comments from
levels from each sentiment model shows that Vader was the Instagram were harvested using the
most accurate to estimate sentiment. In this paper, the user "IGCommentsExport" Chrome extension, while
comments under the food posts in social media platforms are comments from YouTube were obtained using the
used to undergo sentiment assessment with one of the NLP YouTube Data API v3 in conjunction with a Google
pre-trained models, Flair to understand what’s their real Sheet Extension and Apps Script. 20 sample comments
thoughts for each post. Flair is the most suitable for this related to crawled social media posts were earmarked
domain because it can capture contextual information of for subsequent sentiment analysis utilizing Natural
words in a comment and support diverse linguistic Language Processing (NLP).
environment especially in social media site.
C. Context-Aware Approach Recommendation System
The challenge of choosing dining options in a world with
numerous choices has been addressed by using a context-
aware recommendation system for restaurants [10]. The
limitations of existing recommender systems like Yelp and
Google Maps were highlighted. A novel recommendation
system called ILOVEFOOD! has been proposed in this paper,
implemented as a smartphone app, which takes into account
various contextual elements, including a user's current
location, time, weather, user preferences, and restaurant
reviews. By incorporating these elements, the system provides
complex and personalized restaurant recommendations,
enhancing user satisfaction. This innovative approach
leverages context awareness to modify the ranking of
suggestions based on the current context, demonstrating its
effectiveness in enhancing overall system usability. However,
it's crucial to consider users' implicit habits, ensuring that the
system can provide more precise and up-to-date
recommendations aligned with their true desires. Implicit
feedback's real-time nature distinguishes it from explicit data
in order to improve the recommendation system to be more
precise by considering what users really desire [11].
III. METHODOLOGY
A. Description of Dataset
This study encompasses four distinct datasets, each
serving specific purposes for the research objectives as shown
in Fig. 1, encompassing food image classification, social
media posts for deployment, social media comments for
sentiment analysis (NLP), and restaurant details.
• Food Images: The training dataset is made up of food
images from well-known culinary image sources, such
as food-101, uecfood100 and mafood121 datasets.
However, these pre-existing datasets exhibit a notable
deficiency in their representation of Malaysian
culinary diversity. To redress this shortfall, a
supplementary dataset was curated through web
scraping of images from Google, utilizing the
"Download All Images" browser extension. From the
combined dataset, 50 unique food categories were Fig. 1. Research Flowchart

40

Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 18,2024 at 07:40:46 UTC from IEEE Xplore. Restrictions apply.
 • Restaurant Information: To assemble comprehensive
restaurant details, the Place API, an integral
component of the Google Maps API suite, was
employed [12]. 50 restaurants that related to food
image dataset are scraped correspondingly. By
providing specific geolocation coordinates, a
maximum search radius, keywords, and restaurant
types to the Place API, it facilitated the extraction of
detailed restaurant information tailored to the specified
criteria. Geolocation coordinates for West Malaysia
were obtained from CalcMaps, a dedicated website for
retrieving longitude and latitude coordinates regarding
to various geographic locations within the state [13].
B. Food Image Classification
In the initial phase, a set of precisely 5000 food images was
assembled to serve as the foundational dataset for food image
classification. After that, a structured dataframe with the
image file paths and labels was created. This approach was
chosen to enhance data manipulation and facilitate data
preprocessing with greater flexibility and control. To further
enrich and diversify the dataset, data augmentation techniques
were applied. The objective was to expand the existing
dataset, amplifying each food category from its initial 100
images to a more extensive 500 images, total 25000 food
images. This augmentation strategy served to enrich the
dataset and furnish the model with an extensive range of
diverse training instances, thereby fortifying its ability to
generalize effectively. The data distribution for training,
testing, and validation was meticulously organized, allocating
70% of the data for training purposes, while allocating 15%
each for testing and validation. An image data generator was
employed to facilitate efficient data loading in batches [14]. In
the model architecture selection, MobileNetV2 was chosen, as
it demonstrates an improved overall performance while
maintaining computational efficiency [15]. Evaluation of the
model's performance encompassed key metrics such as
accuracy, generating a confusion matrix, and constructing a
classification report. These evaluations were crucial in
assessing the model's efficacy in classifying each food
category. Upon successful training and the attainment of
commendable performance on the test images, the model
transitioned to its deployment phase, wherein it undertook the
task of predicting food categories in the context of social
media posts.
C. Sentiment Analysis of User Comments
Since every social media comment scraped is considered
as unstructured data, it is necessary to make sure the whole
dataset is processed in an appropriate and structured format
consistently. Therefore, text preprocessing is a must to
undergo before analyzing them with sentiment models as well.
The social media comments are preprocessed with a translator
using Google Translate API. Social media platforms are
globally user-based and they are required to translate to a
common language like English to ensure the content can reach
a broader audience, facilitating communication and
interaction among users from different language backgrounds.
Next, the social media comments are processed with text
normalization including removing HTML tags, punctuations
and empty rows. HTML tags and punctuations are treated as
noise in the text or phrase whereas empty rows are the missing
values that might mislead the insights especially for sentiment
assessment [16]. These need to be removed so that those
comments are standardized for further sentiment analysis
41
Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 18,2024 at 07:40:46 UTC from IEEE Xplore. Restrictions apply.
purposes. Leftover social media comment texts from
commenters are converted to lowercase to avoid those texts
being treated as different tokens. For emoticon handling, those
emojis are demojized to text format for data consistency and
easier analysis for sentiment models [17]. In the modeling
section, one of the transformer models, Flair, is used for
determining the sentiment labels for social media user
comments and whether they like or dislike the particular food
posted. It is also a NLP sentiment pre-trained model which is
easy to use and considered as ready-to-use out of the box in
order to save developers’ time and resources on data
collection, feature engineering, particular model training and
also hyperparameter tuning. The sentiment evaluation on
social media comments is Human-Level Performance (HLP)
where the generated sentiment label is according to the
developer's own knowledge and understanding. The processed
social media comments with the sentiment “Positive” label are
exported to an Excel named “Processed Comments”.
D. Restaurant Recommendation System
From the "Processed Comments" excel dataset, the
extraction of usernames and corresponding user profile URLs
facilitated the systematic generation of individual user
profiles. Considering privacy constraints inherent to social
media data and the absence of explicit user address
information, a pseudorandom state allocation mechanism was
implemented, assigning a notional state to each user,
presuming their residence within the geographical bounds of
West Malaysia. Furthermore, leveraging the insights garnered
from the deployment of the food image classification model,
an automated assignment of food tags to users exhibiting
positive sentiment in their comments under specific posts was
executed. This process of attributing food tags to users
contributed to the refinement and enrichment of the
comprehensive user profiles. Consequently, the establishment
of a comprehensive User Profile repository was achieved,
wherein each user profile encompassed crucial details,
including username, user profile URL, assigned state, and
corresponding food tags. Moreover, through the utilization of
the username as an input parameter, the extraction of the user's
assigned state and associated food tags from the User Profile
repository was seamlessly executed. The extracted food tags
were subsequently employed to facilitate the retrieval of
precise restaurant details aligned with specific food
preferences, while the user's assigned state played a pivotal
role in facilitating location-based comparisons with the
geographical distribution of restaurants. This systematically
recommended restaurants in close proximity to the user's
assumed location, fostering an enhanced user experience and
personalized recommendation services.
IV. RESULT
Illustrated in Fig. 2, the deployed model demonstrates its
efficacy in accurately predicting the content of social media
posts pertaining to various food items. Notably, the model
accurately classifies "1.png" as Curry Mee, with a confidence
level of 72.97%, and "10.png" as Spaghetti Bolognese, with a
remarkably high confidence level of 99.99%. These precision
levels underscore the model's robust performance in
discerning intricate nuances within diverse culinary
compositions, thereby enabling precise classification
outcomes. Subsequent to the successful classification of food-
related posts, the corresponding food tags are systematically
stored within the user profile repository. This mechanism is
strategically designed to associate users who exhibit a positive
 sentiment in their comments under specific posts with the
relevant food tags.
In Fig. 3, the Excel spreadsheet stored commenters’ social
media comments with only a “Positive” sentiment label under
each particular food post. After the food post had detected
accurately in the food image classification deployment stage,
all of the crawled comments under the specific food post is
analyzed with the sentiment model, Transformer Flair, in
order to analyze every commenter’s own opinion in their
comment as implicit information. The transformer model
generated “Positive” and “Negative” sentiment labels for
each comment and all of the comments were evaluated with
Human Level Performance (HLP). Nevertheless, Flair does
not provide “Neutral” sentiment label as it might not have
been explicitly documented unless setting a threshold for
model’s confidence scores on “Neutral” labelling. After the
evaluation, the sentiment label with “Positive” as well as
having the sentiment confidence level higher than 85% were
exported to “Processed Comments” Excel format for further
restaurant recommendation purposes. Fig. 2. Food Image Classification

Fig. 3. Processed comments

Fig. 4. User profile

42

Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 18,2024 at 07:40:46 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5. Restaurant Recommendation System
Fig. 4 depicts the comprehensive user profiles including
usernames and corresponding user profile URLs, extracted
from the processed comments dataset, a result of the sentiment
analysis process. To address privacy concerns and the absence
of explicit geographical data, a pseudorandom allocation of
states is implemented to signify the assumed residence of each
user within the region of West Malaysia. Subsequently, food
tags are methodically assigned to users who express positive
sentiments in their comments beneath relevant posts. This
integrated approach to user profiling enhances the system's
capacity for personalized recommendations and augments the
user experience.
As depicted in Fig. 5, the system is designed to efficiently
retrieve user-specific information upon receiving a username
as input. This information includes the user's assigned state
and the associated food tags within their individual profile, as
extracted from the User Profile repository. Leveraging this
information, the system takes into account both the user's
culinary preferences, represented by food tags, and their
assumed geographical location, denoted by the assigned state.
This can facilitate the seamless curation of relevant and nearby
restaurant recommendations tailored to the user's individual
preferences and geographical proximity. For example, one of
the social media users named “burntfinger61” who lives in
Johor and loves to eat curry mee. The restaurant
recommendation system returns all restaurant information that
sells curry mee in Johor according to the rating popularity for
the particular user. There are 30 social media users used the
recommendation system and meet their expectation and
satisfaction on dining options.
V. CONCLUSION
The research paper concludes that the integration of image
classification (MobileNetV2) and sentiment analysis
(Transformer Flair) from social media data has advanced
personalized restaurant recommendations for social media
users. This more sophisticated recommendation system
utilizes implicit user behavior, extracting valuable insights
from image and sentiment data shared on social media. The
approach, achieving at least an 80% accuracy rate, goes
beyond explicit details like star ratings, likes, or follows,
focusing on capturing implicit preferences and contextual
complexities expressed in social media comments. The
findings suggest the potential for more accurate and tailored
dining suggestions, with future work focusing on improving
image classification accuracy, expanding food categories, and
43
Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 18,2024 at 07:40:46 UTC from IEEE Xplore. Restrictions apply.
analyzing Out of Vocabulary (OOV) words for a better
understanding of implicit social media user behavior in
context-aware restaurant recommendation systems.
ACKNOWLEDGMENT
The authors are grateful to the FOCS faculty for their great
support and direction during the research effort. With the
support of numerous libraries and internet resources, this
study has proven feasible. Lastly, we want to take this
opportunity to convey our sincere gratitude to everyone who
took part in the research's development and assessment stages,
without whom this research might not become achievable.
REFERENCES
[1] S. Sawant and G. Pai, “Yelp Food Recommendation System.”
Available: http://cs229.stanford.edu/proj2013/SawantPai-
YelpFoodRecommendationSystem.pdf.
[2] Y. Gao, W. Yu, P. Chao, R. Zhang, A. Zhou, and X. Yang, “A
Restaurant Recommendation System by Analyzing Ratings and
Aspects in Reviews,” Database Systems for Advanced Applications,
pp. 526–530, 2015, doi: https://doi.org/10.1007/978-3-319-18123-
3_33.
[3] M. Ahmed, M. D. Ansari, N. Singh, V. K. Gunjan, S. K. B. V., and M.
Khan, “Rating-Based Recommender System Based on Textual
Reviews Using IoT Smart Devices,” Mobile Information Systems, vol.
2022, p. e2854741, Jul. 2022, doi:
https://doi.org/10.1155/2022/2854741.
[4] S. Phiphitphatphaisit, & O. Surinta, “Food Image Classification with
Improved MobileNet Architecture and Data Augmentation,”
ResearchGate, 2020,
https://www.researchgate.net/publication/340470168_Food_Image_C
lassification_with_Improved_MobileNet_Architecture_and_Data_Au
gmentation.
[5] Doaa Al-Rubaye, & Serkan Ayvaz, “Deep Transfer Learning and Data
Augmentation for Food Image Classification.” Oclc.org, 2023,
https://ieeexplore.ieee.org/document/10010432.
[6] S. Arora, “Sentiment Analysis Using Python,” Analytics Vidhya, Jul.
07, 2022. https://www.analyticsvidhya.com/blog/2022/07/sentiment-
analysis-using-python/.
[7] Husna, “What is Sentiment Analysis? Types and Use Cases,” 2021,
www.taus.net. https://www.taus.net/resources/blog/what-is-sentiment-
analysis-types-and-use-cases.
[8] A. A. Arifiyanti, D. S. Y. Kartika and C. J. Prawiro, "Using Pre-Trained
Models for Sentiment Analysis in Indonesian Tweets," 2022 6th
International Conference on Informatics and Computational Sciences
(ICICoS), Semarang, Indonesia, 2022, pp. 78-83, doi:
10.1109/ICICoS56336.2022.9930599.
[9] G. Singhal, “Importance of Text Pre-processing | Pluralsight,”
www.pluralsight.com, Oct. 05, 2020.
https://www.pluralsight.com/guides/importance-of-text-pre-
processing.
 [10] K. Karmakar, “A Context-Aware Approach To Restaurant [15] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen,
Recommendations: System Algorithm and Case Study.”,2020, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” 2019.
ResearchGate, https://doi.org/10.13140/RG.2.2.12720.38409. Available: https://arxiv.org/pdf/1801.04381.pdf.
[11] P. Berkem, “Explicit Data vs Implicit Data: What Is the Difference?”, [16] C. Goyal, “Text Cleaning and Preprocessing | Guide to Master NLP
storyly, Nov. 14, 2022, https://www.storyly.io/post/explicit-vs- (Part 3),” Analytics Vidhya, Jun. 15, 2021.
implicit-data. https://www.analyticsvidhya.com/blog/2021/06/part-3-step-by-step-
[12] “Overview | Places API,” Google Developers. guide-to-nlp-text-cleaning-and-preprocessing/.
https://developers.google.com/maps/documentation/places/web- [17] V. Grover, “Exploiting Emojis in Sentiment Analysis: A Survey,”
service/overview. Journal of The Institution of Engineers (India): Series B, Jun. 2021,
[13] CalcMaps, “Map Radius Calculator,” CalcMaps, 2015. doi: https://doi.org/10.1007/s40031-021-00620-7.K. YW, “The impact
https://www.calcmaps.com/map-radius/. of skin disorders on patients’ quality of life in Malaysia,” Journal of
Clinical Intensive Care and Medicine, vol. 4, no. 1, pp. 001–009, Jan.
[14] A. Rosebrock, “Keras ImageDataGenerator and Data Augmentation,” 2019, doi: https://doi.org/10.29328/journal.jcicm.1001018
PyImageSearch, Jul. 08, 2019.
https://pyimagesearch.com/2019/07/08/keras-imagedatagenerator-
and-data-augmentation/.

44

Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 18,2024 at 07:40:46 UTC from IEEE Xplore. Restrictions apply.