Implementation of Multinomial Naïve Bayes
Algorithm for Sentiment Analysis of Applications’
User Feedback
Gabriella Putri Wiratama
Department of Informatics
Universitas Multimedia Nusantara
Tangerang, Indonesia
gabriella.wiratama@student.umn.ac.id
Abstract—User feedbacks can be used as a tool for application
developers to find out and understand users’ needs, preferences,
and complaints. It is important for developers to identify problems
that arise from the user-given feedbacks. This is very difficult to
do considering the amount of feedback received every day.
Reading and classifying each and every feedback takes a long time
and it is very ineffective. To overcome this problem, a sentiment
analysis system based on Multinomial Naïve Bayes classification
algorithm was built to determine whether a feedback has a positive
or negative sentiment. Naïve Bayes algorithm is generally used for
classification because it is very easy and effective. Based on
previous research, Multinomial Naïve Bayes algorithm gives off
the best performance compared to other traditional machine
learning algorithms. This study aims to implement Multinomial
Naïve Bayes classification algorithm on a web application and
calculate the accuracy of class predictions made by the system.
Based on the results of several test, the accuracy of class
predictions which were evaluated using confusion matrix, shows
that the model with training-testing comparison of 70:30, balanced
datasets, and over-sampling each dataset by 30% produces the
best performance, with 71.6% for accuracy, 76.92% for precision,
61.73% for recall and 68.49% for F1 score.
Keywords—Bahasa Indonesia; Multinomial Naïve Bayes; NLP;
sentiment analysis; user feedback;
I. INTRODUCTION
System objectives, conceptual structures, requirements and
assumptions that have been elicited, evaluated, specified and
analyzed may need to be changed for a variety of reasons,
including defects to be fixed; project fluctuations in terms of
priorities and constraints; better customer understanding of the
system’s actual features and so on [1]. As a developer, it is
important to anticipate changes that might occur. One of the
ways to identify the changes that have to be made is through
feedbacks given by users. User feedbacks are direct opinion
from the users who have experience the apps and reflect the
instant user experience [2]. Through user feedback, developers
can understand users’ needs, preferences, and complaints. The
emerging issues detected from user feedback can provide
informative evidence for developers in maintaining their apps
and scheduling the app updates [3].
Andre Rusli
Department of Informatics
Universitas Multimedia Nusantara
Tangerang, Indonesia
andre.rusli@umn.ac.id
Identifying issues from user feedbacks based on its
characteristics can be challenging. First, user feedback contains
numerous noise words, such as misspelled words, repetitive
words, not to mention, popular local language of slang words,
which is a popular language among Indonesian youngsters. This
makes it becomes difficult to categorize whether the user is
satisfied or dissatisfied with the application. Second, for some
popular apps, the volume of user feedbacks generated each day
is simply too large to be manually read and analyzed. Third, not
all feedback contains useful information for developers. Only a
third of user feedbacks contain information that can directly
help developers improve their apps [4].
As a first step towards a comprehensive support tool for
analyzing user feedbacks, this research proposes a tool to
automatically classify user feedbacks according to its overall
sentiment. Machine learning classifiers have been widely used
for the purpose of sentiment analysis providing good results [5].
Among various machine learning algorithm, Naïve Bayes (NB)
algorithm is generally used for the classification problem due
to its simplicity and effectiveness [6].
User feedbacks contain many words and sentences, which
are expressed in various ways. Therefore, the feedback is
processed first, before conducting sentiment analysis.
Preprocessing the user feedback is done with common Natural
Language Processing (NLP) technique. For text written in
English, there are several NLP toolkits available such as
Stanford Core NLP Toolkit [7], OpenNLP [8], or NLTK [9]. As
for Bahasa Indonesia, at the time our research was conducted,
it was still hard to find an integrated NLP toolkit available. The
available NLP modules for Bahasa Indonesia are separated
modules such as stemmer, POS tagger, syntactic parser and
other similar modules, thus, we conducted the text pre-
processing activities step by step, details on these activities will
be explained in the following sections.
This research aims to study, analyze and implement
Multinomial Naïve Bayes algorithm for analyzing sentiment of
application user feedbacks in Bahasa Indonesia. The paper is
organized as follows. In the following section we review some
related works to ours, then we present a brief overview of some
research in sentiment analysis. Section IV describe our
experiment and findings in implementing Multinomial Naïve
Bayes for sentiment analysis of application user feedback.
Section V concludes our work and proposes several future
works.
II. RELATED WORK
Sentiment analysis is the method used for enabling
computers to recognize and classifying opinions from big
unstructured texts datasets with machine language and
computer programming [10]. Sentiment analysis involves
classifying opinions into categories like “positive”, “negative”
or “neutral”. Many approaches and algorithm have been used
for sentiment analysis, most of them are machine learning
techniques. A comparative analysis of performance between
NB, Support Vector Machine and Maximum Entropy show that
NB classifiers outperforms others [11]. Another comparative
research on machine learning classifiers for Twitter sentiment
analysis show that Multinomial NB outperforms other
classifiers [5]. Many researches focus on sentiment analysis of
Twitter data due to simple and easy accessibility to massive
amount of data generated in real time [12, 13, 14]. Fig. 1. The operation flow of the proposed approach [6]

There isn’t as many researches on sentiment analysis of
application user feedback using NB algorithm, not to mention,
application user feedback in Bahasa Indonesia.
III. METHODOLOGY
Several literatures and existing tools are used in this
research. A previous research [15] has pointed out the potential
of using Naïve Bayes to classify feedbacks in Bahasa Indonesia.
In our research, in order to classify feedbacks for sentiment
analysis, we utilized a proposed classification method based on
Naïve Bayes algorithm proposed in another research [6] and
some adjustment were made to accommodate natural language
processing in Bahasa Indonesia. Researches related to natural
language processing in have also been conducted using other
algorithms [16, 17], displaying the potential of natural language
processing steps to process texts written in Bahasa Indonesia.
A. Natural Language Processing
Natural language processing (NLP) is a computer science
field dealing with human language processing in either text or
speech [18]. Preprocessing of feedback includes the stemming,
tokenization and removal of URL, numbers, punctuation and
special character, stop-words and duplicate words. In addition,
common slang and spelling mistake are replaced with the root
word. Stemming is done using an open-source stemmer library
[19], based on Enhanced Confix Stripping for Bahasa
Indonesia.
B. Multinomial Naïve Bayes
The classification method used in our research exploits a
novel attribute weighting and feature selection approach based
on the Multinomial NB. Fig. 1 illustrates the operation flow of
the proposed approach. The first method (attribute weighting)
aims to calculate the weights of words based on the training set
divided into positive and negative class, while the second
method (feature selection) aims to modify the weights using the
average of weight differences for automatic feature selection.
Multinomial NB expands the use of Naïve Bayes algorithm.
It implements NB for data distributed multinomially and is a
frequency-based model proposed for text classification, in
which word counts are used to represent data. Methods for
enhancing Naïve Bayes performance are classified into five
categories which are structure extension, feature selection,
attribute weighting, local learning and data expansion [20].
Among these methods, Song et al [6] used attribute weighting
method of using different weight for each attribute and feature
selection method of selecting a subset of attributes based on the
weight. MNB with attribute weighting can be modeled using
(1).
𝑐(𝑑) = arg max[log 𝑃(𝑐) + ∑𝑚
𝑖=1 𝑊𝑇𝑖 𝑓𝑖 log 𝑃(𝑤𝑖 |𝑐)]
𝑐∈𝐶
(1)
where 𝑊𝑇𝑖 is the weight of each word 𝑤𝑖 (𝑖 = 1,2, … , 𝑚).
C. Attribute weighting
Song et al [6] proposed an attribute weighting approach
similar to GRW. In the proposed approach, the training set is
first divided into positive and negative, and then the number of
positive words and negative words are counted separately from
each set. The weight 𝑊𝑇𝑖 of each word 𝑤𝑖,𝑐 in Dc can be
defined as:
𝐼𝐺𝑅(𝑐,𝑤 )×𝑚𝑐
𝑖,𝑐
𝑊𝑇𝑖,𝑐 = ∑𝑚𝑐 𝐼𝐺𝑅(𝑐,𝑤 (2)
𝑖=1 𝑖,𝑐 )
where c is the class label (∈ {𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒, 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒}), mc is the
number of different words in Dc. 𝐼𝐺𝑅(𝑐, 𝑤𝑖,𝑐 ), which is the
information gain ratio of 𝑤𝑖,𝑐 , can be obtained by (3).
𝐼𝐺(𝑐,𝑤𝑖,𝑐 )
𝐼𝐺𝑅(𝑐, 𝑤𝑖,𝑐 ) = 𝐻(𝑤𝑖,𝑐 )
(3)
𝐼𝐺(𝑐, 𝑤𝑖,𝑐 ) and 𝐻(𝑤𝑖,𝑐 ) are information gain and entropy common way to overcome this problem is to use Laplace
information of 𝑤𝑖,𝑐 , respectively. 𝐼𝐺(𝑐, 𝑤𝑖,𝑐 ) and 𝐻(𝑤𝑖,𝑐 ) can Smoothing. This technique is performed by assuming that the
be obtained by (4) and (5). training dataset is large enough so that when adding 1 to each
count, we only need to make a difference that is ignored in the
𝐼𝐺(𝑐, 𝑤𝑖,𝑐 ) = 𝐻(𝑐) − 𝐻(𝑐|𝑤𝑖,𝑐 ) (4) probability value of zero estimation. The smoothing is
calculated as follows.
|𝐷𝑣,𝑐 | |𝐷𝑣,𝑐 |
𝐻(𝑤𝑖,𝑐 ) = − ∑𝑉 |𝐷| log 2 |𝐷| (5)
𝑐𝑜𝑢𝑛𝑡(𝑤,𝑐)+1
𝑃(𝑤|𝑐) = (11)
𝑐𝑜𝑢𝑛𝑡(𝑐)+ |𝑉|
where 𝐻(𝑐) is the entropy of Dc, 𝐻(𝑐|𝑤𝑖,𝑐 ) is the conditional
entropy of Dc given the word 𝑤𝑖,𝑐 , |𝐷𝑣,𝑐 | is the size of Dc in where 𝑐𝑜𝑢𝑛𝑡(𝑤, 𝑐) is the number of words w in class c,
terms of the number of 𝑤𝑖,𝑐 for which v (∈ {0, 0̅}). 𝐻(𝑐) and 𝑐𝑜𝑢𝑛𝑡(𝑐) is the total of words in class c and |𝑉| is the number
𝐻(𝑐|𝑤𝑖,𝑐 ) are calculated as: of vocabularies in the training document.

𝐻(𝑐) = −𝑃(𝑐) log 2 𝑃(𝑐) (6)

|𝐷𝑣,𝑐 |
𝐻(𝑐|𝑤𝑖,𝑐 ) = − ∑𝑉 |𝐷| ∑𝑐 𝑃(𝑐|𝑣) log 2 𝑃(𝑐|𝑣) (7)
D. Feature selection
Typical meaningless words have larger weight value than
frequent or important word with the existing attribute weighting
approaches because they usually appear infrequently in the
training set. Song et al [6] proposed feature selection approach
that uses the difference between positive weight and negative
weight of all the words to modify the weights of meaningless
words. After obtaining the weights of all the words in the
training set, the average of the differences is calculated. Based
on that, some weight values are changed to zero. This lets the
meaningless words be effectively excluded when the class of
test document is predicted. The difference of weight, 𝑊𝐷𝑖 , of
𝑤𝑖 (𝑖 = 1,2, … , 𝑚) is defined as follows.
|𝑊𝑇𝑖,𝑝 − 𝑊𝑇𝑖,𝑛 |, if 𝑤𝑖 ∈ 𝐷𝑝 and 𝑤𝑖 ∈ 𝐷𝑛
𝑊𝐷𝑖 = { (8)
𝑊𝑇𝑖,𝑐 , otherwise
The average of 𝑊𝐷𝑖 ’s, 𝐴𝑣𝑔𝑊𝐷 , is computed using (9)
∑𝑚
𝑖=1 𝑊𝐷𝑖
𝐴𝑣𝑔𝑊𝐷 = 𝑚
(9)
𝐴𝑣𝑔𝑊𝐷 and the weight of all words are compared in each
training set, Dc. if the weight is higher than the average, it is
modified to zero. Otherwise, it is kept as follows.
0, if 𝑊𝑇𝑖,𝑐 > 𝐴𝑣𝑔𝑊𝐷
𝑊𝑇𝑖,𝑐 = { (10)
𝑊𝑇𝑖,𝑐 , otherwise
Finally, the class label of the test document is predicted using
non-zero weight words based on (1).
E. Laplace Smoothing
Conditional probability 𝑃(𝑐|𝑣) in (11) can cause problems
because it can produce a probability of zero for document that
have words that have never been encountered before. A
IV. RESULT AND DISCUSSION
A. Dataset
In the work conducted in this paper, we use feedbacks from
an existing web-based e-learning system in Universitas
Multimedia Nusantara. There is a total of 627 feedbacks
gathered on 8th of June 2018 which is comprised of 269 positive
and 358 negative feedbacks.
B. Performance Evaluation
Several scenarios are conducted to determine the best
configuration for classifying the feedback’s sentiment. An
experiment with two training-testing ratios, 70:30 and 80:20,
are conducted. First, we try to use the feedbacks as it is, which
was imbalanced (Scenario 1). Furthermore, we try to under-
sampling (Scenario 2) and over-sampling the dataset (Scenario
3).
In the experiment following Scenario 1 with imbalanced
dataset, the model achieves 50.88% for accuracy, 39.93% for
precision, 29.26% for recall and 33.57% for F1 score with 70:30
training-testing ratio. This is probably caused by the large
difference of number between positive and negative class, in
which the number of negative feedbacks is much higher than
the positive feedbacks. In Scenario 2, we try to balance the
dataset by under-sampling the negative feedbacks to match the
number of positive feedbacks. The model performs better, with
66.09% for accuracy, 66.1% for precision, 66.3% for recall and
66.06% for F1 score, by using 70:30 training-testing ratio. By
balancing the dataset, the prior probability for each class is
levelled. Moreover, by reducing the negative feedbacks, the
word occurrence in negative dataset is also reduced, resulting
in a more balance word weighting. However, removing
feedbacks could cause some important information to be loss,
especially in negative feedbacks that can contain useful
information, such as bug reports or feature request.
In Scenario 3, we try to balance the dataset, by randomly
resampled data from positive class training set to match the
number of negative feedbacks, which was 250 data. The model
performance performs better, though not much increment, with
67.2% for accuracy, 67.97% for precision, 67.04% for recall
and 66.71% for F1 score. We also try to resample data on both
classes to raise the performance. The resampling is done by
adding 10% to 50% of the initial amount of the negative training
dataset, 250 data. The result is model with 30% over-sampling
achieves the best performance, with 71.6% for accuracy,
76.92% for precision, 61.73% for recall and 68.49% for F1
score. Based on the experimentation, several feedbacks in the
dataset contain some ambiguous feedback, which might cause
difficulties for the machine in determining the feedback’s
overall sentiment. In addition, the sentence structure for Bahasa
Indonesia is lenient whether in text or speech, application
feedbacks included. An example of an ambiguous feedback is
as follows, one user feedback reads “Absensi”, meaning
“Attendance” in English. It is unclear whether the user’s
sentiment about the application is negative or positive.
Moreover, the feedback contains only one word, which is not
enough to accurately predict its sentiment. Feedbacks such as
the one previously mentioned (ones that only contain one word)
often appear in the dataset, henceforth, lowering the
performance of the classification model.
TABLE I. MODEL PERFORMANCE
Total Predicted: Predicted: Total Accuracy =
Testing Positive Negative 71.6%
Data = 162
Class: TP 50 FN 31 81 Precision =
Positive 76.92%
Class: FP 15 TN 66 81 Recall =
Negative 61.73%
Total 65 97 162 F1 Score =
68.49%
V. CONCLUSION AND FUTURE WORKS
In this research, we used an attribute weighting and feature
selection approach based on Multinomial Naïve Bayes in order
to get sentiment classification of application user feedbacks.
With this approach we used prior user feedbacks from an
existing web-based university e-learning system which then are
classified into positive and negative class.
The sentiment analysis model’s performance was tested and
evaluated with the best result is 71.6% for accuracy, 76.92% for
precision, 61.73% for recall and 68.49% for F1 score. The
results are calculated from the performance of the model which
is built by over-sampling both datasets up to 30% more than the
initial amount of the negative training set, with 70:30 training-
test ratio. The conducted research shows the potential of
supporting requirements engineering process in software
development by analyzing sentiment of application user
feedbacks in Bahasa Indonesia using Multinomial NB.
Future works include conducting the experiment on a larger
and more representative dataset and employ more advance
natural language processing methods and/or tools. Furthermore,
to enhance the proposed system, various N-gram approaches
such as bigram and trigram could be applied, together with a
more sophisticated attribute weighting and feature selection.
Additionally, semantic understanding could also be
incorporated to produce a more precise sentiment analysis.
Some positive terms can be used sarcastically to express
negative sentiment and some negative terms can be used vice
versa.
REFERENCES
[1] A. v. Lamsweerde, Requirements Engineering: From
System Goals to UML Models to Software
Specifications, Chichester: John Wiley & Sons Ltd, 2009.
[2] T.-S. Nguyen, H. W. Lauw and P. Tsaparas, "Review
Synthesis for Micro-Review Summarization," in Eighth
ACM International Conference on Web Search and Data
Mining, Shanghai, 2015.
[3] C. Gao, J. Zeng, M. R. Lyu and I. King, "Online App
Review Analysis for Indentifying Emerging Issues," in
International Conference on Software Engineering,
Gothenburg, 2018.
[4] N. Chen, J. Lin, S. C. H. Hoi, X. Xiao and B. Zhang, "AR-
Miner: Mining Informative Reviews for Developers from
Mobile App Marketplace," in 36th International
Conference on Software Engineering, Hyderabad, 2014.
[5] H. M. Ismail, S. Harous and B. Belkhouche, "A
Comparative Analysis of Machine Learning Classifiers
for Twitter Sentiment Analysis," Research in Computing
Science, vol. 110, pp. 71-83, 2016.
[6] J. Song, K. T. Kim, B. Lee, S. Kim and H. Y. Youn, "A
Novel Classification Approach based on Naive Bayes for
Twitter Sentiment Analysis," KSII Transactions on
Internet and Information System, vol. 11, no. 6, pp. 2996-
3011, 2017.
[7] C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J.
Bethard and D. McClosky, "The Standford CoreNLP
Natural Language Processing Toolkit," in 52nd Annual
Meeting of the Association for Computational
Linguistics: System Demonstrations, Baltimore, 2014.
[8] "OpenNLP," [Online]. Available:
https://opennlp.apache.org/. [Accessed 11 July 2019].
[9] "NLTK," [Online]. Available: http://www.nltk.org/.
[Accessed 11 July 2019].
[10] C. Fiarni, H. Maharani and R. Pratama, "Sentiment
Analysis System for Indonesia Online Retail Shop
Review Using Hierarchy Naive Bayes Technique," in
Fourth International Conference on Information and
Communication Technologies (ICoICT), Bandung, 2016.
[11] M. Tabra and A. Lawan, "A Comparative Analysis of the
Performance of Three Machine Learning Algorithms for
Tweets on Nigerian dataset," The International Journal
of E-Learning and Educational Technologies in the
Digital Media (IJEETDM), vol. 3, no. 1, pp. 23-30, 2017.
[12] N. A. Vidya, M. I. Fanany and I. Budi, "Twitter
Sentiment to Analyze Net Brand Reputation of Mobile
Phone Providers," in The Third Information Systems
International Conference, Surabaya, 2015.
[13] A. Alamsyah, W. Rahmah and H. Irawan, "Sentiment
Analysis Based on Appraisal Theory for Marketing
Intelligence in Indonesia's Mobile Phone Market,"
Theoritical and Applied Information Technology, vol. 82,
no. 2, pp. 335-340, 2015.
[14] I. Zulfa and E. Winarko, "Sentimen Analisis Tweet
Berbahasas Indonesia dengan Deep Belief Network,"
Indonesian Journal of Computing and Cybernetics
Systems, vol. 11, no. 2, pp. 187-198, 2017.
[15] A. Rusli, "Ekstraksi Kebutuhan Aplikasi Berdasarkan
Feedback Pengguna Menggunakan Naïve Bayes dan
Gamifikasi," Ultimatics: Jurnal Teknik Informatika, vol.
10, no. 1, pp. 34-40, 2018.
[16] S. Hansun and D. D. Sinaga, "Indonesia Text Document
Similarity Detection System Using Rabin-Karp and
Confix-Stripping Algorithms," International Journal of
Innovative Computing, Information and Control, vol. 14,
no. 5, pp. 1893-1903, 2018.
[17] S. Hansun and M. Widjaja, "Implementation of Porter's
Modified Stemming Algorithm in an Indonesian Word
Error Detection Plugin Application," International
Journal of Technology, vol. 2, pp. 139-150, 2015.
[18] A. Purwarianti, A. Andhika, A. F. Wicaksono, I. Afif and
F. Ferdian, "InaNLP: Indonesia Natural Language
Processing Toolkit, case study: Complaint Tweet
Classification," in International Conference On
Advanced Informatics: Concepts, Theory And
Application (ICAICTA), Penang, 2016.
[19] Sastrawi, "High quality stemmer library for Indonesian
Language (Bahasa)," Github Repository, 2017.
[20] L. Jiang, H. Zhang and Z. Cai, "A Novel Bayes Model:
Hidden Naive Bayes," IEEE Transactions on Knowledge
and Data Engineering, vol. 21, no. 10, pp. 1361-1371,
2009.