IEEE - 43488

Automated Topic Modeling and Sentiment Analysis of Tweets on

SparkR
Prema Monish Santoshi Kumari Narendra Babu C
Directorate of Training and Lifelong Department Of CSE Department Of CSE
Learning, M. S. Ramaiah University of M. S. Ramaiah University of
M. S. Ramaiah University of applied Applied Sciences, Applied Sciences,
Science, Bengaluru, India Bengaluru, India Bengaluru, India
premamonish12@gmail.com santoshik29@gmail.com narendrababu.c@gmail.com

Abstract—Advancement of mobile and internet technology
improves the communication and freedom of speaking in social
networks, blogs and websites. Twitter is one of the most common
and popular social media platform gives freedom to the people to
put their views, thoughts and opinion to the world. Analyzing
large scale tweets by putting together large scale individual’s
opinion on particular context will allows us to find various
hidden topics and insights. This paper proposes for developing an
automated topic modeling technique LDA to identify the
interested topics of discussion from large scale tweets related to
two famous political leaders of the county India. Paper
implements a topic modeling method on SparkR framework to
improve the speed and performance for large scale real time
social data processing and analysis. Finally sentiment analysis of
tweets is carried out using lexicon based approach to identify the
people sentiment towards these two leaders. Using empirical
results identified various unknown topics and people interest,
expectations and their concerns on various topics. Results also
shows automated topic modeling and sentiment analysis of tweets
on SparkR framework improves the speed compare to normal R
tool.
Keywords—Twitter data analytics; LDA; SparkR; big data;
sentiment analysis; topic modeling
I. INTRODUCTION
Social media analytics uses various techniques to analyze
social media data for knowledge discovery and information-
driven decision making in several fields like business,
healthcare, education, government, politics and social network
analysis. Social media like twitter, Facebook, WhatsApp,
LinkedIn, Instagram are generating large amount of real time
data which can be extracted for social media analytics to find
new insights. The volume of information is increasing
extensively in social media due to increased use of internet
and mobile technologies, hence analyzing such large volume
of data at real time becomes difficult. This big social media
data can be analyzed using a combination of methods of data
mining, web mining, text mining and natural language
processing on big data tools.
Topic modeling is a statistical machine learning approach
used to find semantics structures in email text, blog post,
speeches ,journal article ,book chapter, social media data and
many kind of unstructured text. Topic models cannot
understand the means and concepts of words in text
documents however this probabilistic model helps to discover
correlation among latent words and finding topic for group of
words that occur in set of document. Combing those word in
the document helps to find new hidden topics form group of
correlated words. Topic modeling is accomplished using
various methods and algorithms. One of the most commonly
used method is LDA.
In this paper LDA is considered for automated modeling
the content of tweets. However it is very challenging to
process unstructured twitter data contains linguistic content
which requires natural language processing and cleaning of
140 character tweets carrying individual’s opinion and
sentiment of peoples and wide discussion on various hidden
topics.
Latent Dirchlet Allocation (LDA) is a probabilistic topic
modeling technique introduced by Blei, Ng and Jordan [1]
which represents texts as mixture of topics that groups words
with certain probabilities. In LDA method assigned number
of topics K are checked repeatedly for each word in every
document .The entire set of documents is checked multiple
times and this iterative process helps LDA to generate a final
solution with consistent topics . Various methods like Gibbs
sampling, Expectation-Maximization (EM) algorithm and
Variational Bayes(VB) inference are proposed to estimate the
number of topics K, a LDA parameter [2] .Gibbs sampling is
an Monte Carlo Markov-chain algorithm for generating
samples from a joint distribution. EM uses an iterative method
for estimating parameters in statistical models depending on
certain latent variables [2]. VB is an alternative to Gibbs
sampling statistical inference over complex distribution which
are difficult to evaluate directly. VB requires large amount of
work compared to Gibbs sampling to derive set of equations
used to iteratively update the parameters .
Twitter is large online communication social networking
micro blogging site. Twitter has gained popularity over the
period of time .The statistics shows that usage of social media
and mobile communication has increased in number of active
twitter users in India from 26.7 million users in 2017 to 30.4
million users in 20181 .Thus lot of information is shared by
users during usage of twitter. In twitter users post and
communicate through messages called "tweets". The

9th ICCCNT 2018

July 10-12, 2018, IISC, Bengaluru
Bengaluru, India
 IEEE - 43488
information on the twitter such as tweets, retweets and user
profiles are analyzed to find personality trait, sentimental
analysis, particular opinion, political analytics, social network
analysis ,event analysis and many more [2]. Analyzing this
large volume of real time twitter data can be achieved using
big data tools like Hadoop and SparkR [3]. In this work we
analyzed real time twitter data on SparkR framework.
Whereas R is an open source statistical programming language
widely used for statistical data analysis. Spark combined with
R to improve the performance and speed of processing.
In present paper real-time tweets is extracted from twitter
of two famous Indian political party leaders Narendra Modi
and Rahul Gandhi. The amount of tweets about the two person
can be linked to a party performance and people opinion about
the person by performing sentimental analysis and topic
modeling. Extracted real time twitter data is analyzed on
SparkR framework by applying LDA topic modeling to find
the related topics of the person tweets. The data analyzed is
used as feedback about public needs and aspirations for taking
future political decisions.
This paper address mainly following three implementation.
1. Real time twitter data extraction and preprocessing
on SparkR framework to speed up the process
2. Automated topic modeling of tweets using LDA
Gibbs method to find hidden topics in corpus of
tweets.
3. Finally, sentiment analysis of tweets to identify
people sentiment on presented scenario.
The remaining paper is organized as follows. Section II
give an overview of past research work on twitter data
analytics and LDA. Section III gives the detailed description
of proposed system of this paper .Section IV gives the
visualization and explanation of experimental analysis
.Section V gives the conclusion and future work of this paper.
II. RELATED WORK
Several research work has been carried out in the area of
social media analysis, topic modeling, twitter data analytics
and big data analytics. Various methods of text mining, natural
language processing and topic modeling are applied in fields
like software engineering, social media analytics, Image
classification, and annotation, and political science, medical
and linguistic science. One of the popular latent semantic
analysis method is LDA used for automated topic
identification and modeling in large data set.
There are various models proposed on LDA [2] and
different information technique are used to extract data from
twitter like analysis of different hash tags, sentiment analysis
,event analysis , opinion mining ,identification of influence
,twitter network topology, business value analysis and many
more [4]. Social media analytics can be used for
identifying people opinion and sentiments towards political
party. Analyzing political party twitter data using text mining
and lexical method will help in understanding people emotions
towards political parties [5]. LDA is a well-known topic
modeling algorithm that can be effectively used for
discovering topics from social media content .This approach is
9th ICCCNT 2018
July 10-12, 2018, IISC, Bengaluru
Bengaluru, India
widely used in various domains to automatically mining the
hidden topics [6].
Another researcher uses a LDA topic modeling to analyze
the twitter message content of challenging social events in
Kenya. Normalized Mutual Information (NMI) and topic
coherence analysis were used to find the LDA parameters for
model that outputs more informative and coherent topics. The
change in the number of topics significantly resulted in change
of output [7].
Three-layer hierarchical Bayesian model is applied on
adverse drug reaction data which allowed to predict targeted
clinical safety in early preclinical stage. LDA with collapsed
Gibbs sampling method is applied for dimensionality
reduction and to obtain higher prediction accuracy with short
run time [8]. Securing networks from intrusions has been
critical issue. LDA-based Automatic Rule Generation
(LARGEN) with Gibbs sampling can be applied to analyze the
malicious traffic and to find the attack signatures that will be
used as Intrusion detection system rules. These rules can be
used to identify cyber attack with high accuracy rate[9].
Opinion-lexical approach is a text mining technique to find
the sentimental analysis such as positive, negative and neutral
tweets applied on two famous faces Narendra modi and
Arvind Kejriwal in country India. The results provide
feedback of public opinion, to deal with political affairs in
better way and identify areas they need to improve[10]. To
characterize malicious android apps advance LDA with
Genetic Algorithm (GA) is applied on app description and
sensitive data flow information to generate topic-specific data
flow signatures. These topic-specific data flow signatures can
be used to characterize malware and malicious behavior[11] .
Two layers of matrix factorization is proposed for topic
modeling method to detect latent variables in legislative
speeches from the period 1999-2014 of European Parliament
(EP). The results shows the members of EP have reacted to
external and internal stimulus when making parliament
speeches [12].
There is a wide application of the topic modeling in recent
years. LDA with various flavors are proposed based on their
application and performance of the algorithm like LDA with
Gibbs sampling, LDA with GA and LDAGEN
III. PROPOSED SYSTEM
Proposed system aims to develop, implement an automated
topic modeling technique LDA to identify the interesting
topics from large scale tweets related to two famous political
leaders of the county India. Interpreting the LDA output is
difficult by humans. LDAvis [13] package used to visualize
the results by creating JSON object which provides effective
visualization template. Also sentiment analysis of tweets is
carried out using NRC lexicon approach [5] to identify the
people sentiment towards those two leaders. Empirical system
is implement in following steps.
A. Connecting R with Spark using SparkR Package
R is open source software. R widely used for statistical
analysis, data analytics, visualization and reporting. R
 IEEE - 43488
Programming made its first debut in 1993 developed by Ross
Ihaka and Robert Gentleman at the Department of Statistics of
the University of Auckland in Auckland, New Zealand [3]. R
in most popular for analyzing statistical data and data science.
It is single threaded execution model that limits the data size
to be analyzed. To speed up there are many proposals to
develop parallelism in execution [14].
There are several ways for R to enable big data
processing like RHadoop, Rhipe and SparkR packages are
provided by R community. SaprkR provides outstanding
performance than other two approaches which used map
reduce[15]. There for this model uses SparkR which is a R
package that provides a light-weight frontend to use Apache
Spark in R tool. SparkR provides a distributed data frame that
supports data manipulation on large datasets like dplyr in R.
Spark depends on the SPARK_HOME environmental
variable, so before the spark session which connects SparkR
with R Program environmental variable need to be set. Then
load SparkR library to speed up the execution using multiple
threads.
B. Data Collection
Unstructured real time data is extracted from tweeter using
twitter API by authenticating with twitter API. 18000 tweets
and retweets are extracted for Rahul Gandhi and Narendra
Modi dated from 2017-04-18. Extracted data is stored in the
form data frame for further manipulation and analysis.
C. Processing and cleaning data
In this step only text (tweets) are extracted from data frame
to create corpus. As discussed before tweets are highly
unstructured which contains unwanted features like http, @, #,
www, url, .,"",via, stop words, numbers, white spaces which
does not carry any opinion or sentient. All unwanted features
are removed and cleaned from extracted data by applying text
mining technique.
D. Feature selection
In this step preprocessed and cleaned data is used for
feature selection.in which all the documents are converted to
lower case letters to maintain uniformity. Subsequently
document is tokenized into set of words or corpus. Using
lemmatization and stemming relative and similar feature are
tokenized into single type.
E. Sentimental analysis
In this step unsupervised lexicon based approach is used
for emotion classification [5]. Emotion level in calculated by
associating the emotions and sentiment score of collected
tweets generated by NRC lexicon method. Package Syuzhet is
used to perform this sentiment analysis. Calculated emotions
can be interpreted by graphical representation which makes
easy to understand the results.
F. Create Term document matrix TF-IDF
Considering corpus and dictionary of terms that contains
all words in corpus a document term matrix is created. It
contains a two dimensional matrix whose rows are the
9th ICCCNT 2018
July 10-12, 2018, IISC, Bengaluru
Bengaluru, India
documents and columns are terms in dictionary. Such that
each entry(i,j) in matrix represents the frequency of term i in
document j. TF-IDF(Term frequency - inverse document
frequency) represents impertinence of the word for the
document and relative document importance for overall
corpus.
G. Calculating LDA
Every document is mixture of topics, each topic consists of
some distribution words like selected dataset Narendra Modi
tweets consists of many related issues. LDA is topic modeling
technique which assumes probabilistic distribution over latent
topics is represented by each document. In proposed paper
uses Gibbs sampling and harmonic mean to find the k value.
Gibbs sampling is relatively faster than EM [16]. Gibbs
sampling is a marginal likelihood function which marginalizes
the posteriori log-likelihood ratio. It is adopted to find the best
assignment of related words into topic according to maximum
posteriori log-likelihood ratio [17]. Instead of applying k
values manually, harmonic mean is applied for log-likelihood
for each topic and find the maximum mean value which gives
the number of topics k [17].
A document is a sequence of N words denoted by d =
(w1,w2, . . . ,wN), where wn is the nth word in the sequence. A
corpus D is a collection of M documents denoted by D =
(d1,d2, . . . ,dM). where dm is the mth word in sequence. A
generative process of LDA [2] is as follows :
(a) Choose a multinomial distribution φt for topic t (t ∈
{1,2,..., T}) from a Dirichlet distribution with parameter β.
(b) Choose a multinomial distribution Θd for document d
(d ∈ {d1,d2, . . . ,dM}) from a Dirichlet distribution with
parameter α.
(c) For a word w n (n ∈ {1,..., N d }) in document d,
i. Select a topic z n from Θ d .
ii. Select a word w n from φ zn .
The probability of observed data D is computed and
maximized as follows:
Terminologies used
k - Number of topics
Ș - parameter of the Dirichlet prior on the per-document
topic distributions,
β - Parameter of the Dirichlet prior on the per-topic word
distribution,
ȟm -is the topic distribution for document m,
k -the word distribution for topic k,
Z mn -is the topic for the nth word in document m
Wmn- is the specific word.
 IEEE - 43488

LDA generates latent topics in whole corpus. As discussed

above interpreting LDA output is very challenging. The topics
identified by using LDA can be visualized using LDAvis
package. LDAvis Package [13] provides web based interactive
visualization of topics. Visualization plot using LDAvis has
two basic panel. Left panel present overall view of topic
model and relation between topics. Each circle corresponds to
a topic. Right panel has the horizontal bar plot which
represents usefulness of currently selected topic for selected
term independently.

IV. EXPERIMENTAL RESULT ANALYSIS

A. Extracting twitter data

For experiment 18000 real time tweets are extracted using
twitter API for each individual leaders dated from 2017-04-18
to 2018-04-30. Extracted data is stored as data frame. Fig 1. People emotions score for Narendra Modi tweets
Preprocessing data by removing noise and unwanted data .For
feature selection data is extracted from retweets, split into
words to find the insightful meaning.

B. Determine TF - IDF
Document term matrix finds the frequency of words
that occur in each document. Document term matrix is
calculated using package 'tm' .Using document term matrix
TF-IDF is calculated. Sparse entries in document term matrix
is removed. Document term matrix had 18000 tweets and
11104 terms after removing sparse terms it has 10172 tweets
and 11104 terms.

C. Sentimental analysis
Emotion level is calculated by associating the emotions
and sentiment of collected tweets generated by NRC lexicon Fig 2. People emotions score for Rahul Gandhi tweets
method. Package syuzhet is used to calculate sentiment and
emotions score. Below Table 1 illustrates the sentiment scores D. Applying and visualization LDA
of Narendra Modi and Rahul Gandhi. Figure 1 and figure 2
gives the sentiment and emotion visualization for both leader LDA with Gibbs sampling is applied for document term
tweets. matrix using package 'topic models'. The obtained LDA topics
are visualized using ‘LDAvis’ package. In this k=7 number of
topics obtained by applying harmonic mean for likelihood.
TABLE 1 SENTIMENT ANALYSIS SCORES Time elapsed to execute the model on sparkR is 127.33sec
Sentiment Narendra Rahul Gandhi to preprocessing, analyze and visualize 18000 tweets.
scores modi However time elapsed to preprocessing, analyze and visualize
anger 442 481 the same 18000 tweets without spark is 135 sec.
anticipation 380 386 In figure 3, 4, 5, 6 left panel present overall view of
disgust 348 368 topic model and relation between topics. Each circle
fear 474 526 corresponds to a topic. Right panel has the horizontal bar plot
joy 331 329 which represents usefulness of currently selected topic for
sadness 415 447 terms independently shows top 30 terms in topic. Below
figures 3, 4, 5, 6 shows the visualization of LDA plot for top
Surprise 220 221
eight topic discussed for two famous political faces of India.
trust 572 569 The figure 3 and 5 depict the overall term frequency in all the
negative 984 1104 topics. Figure 4 and 6 represents the frequent terms of the first
positive 1017 988 cluster.

9th ICCCNT 2018

July 10-12, 2018, IISC, Bengaluru
Bengaluru, India
 IEEE - 43488

Fig 3. Top eight clusters represents eight topics resulted to Narendra Modi tweets

Fig 4. Term frequency of first cluster representing first topic identified for Narendra Modi tweets

Fig 5. Top eight clusters represents eight topics resulted to Rahul Gandhi tweets

9th ICCCNT 2018

July 10-12, 2018, IISC, Bengaluru
Bengaluru, India
 IEEE - 43488

Fig 6. Term frequency of first cluster representing first topic identified for Rahul Gandhi tweets

E. Analysis of Result REFERENCES

Sentiment score of above Table.1 shows more positive
sentiment score toward Narendra Modi compered to Rahul
Gandhi. These results can be used identify people expectation
and trust toward both the political leaders. Further analysis of
large scale tweets would be helpful in predicting the next
prime minister of India.
Topic modeling results of figure 3, 4, 5, 6, will be helpful
to identify various topics discussed relative to different issues
and news related to Narendra Modi and Rahul Gandhi. Further
analysis of these topics will be used to identify opinion or
sentiment related to various hidden topics in different
applications such as healthcare, sports, education and
terrorism.
V. CONCLUSION
Advancement in internet, mobile technology and social
media gives a freedom for people to speak and put their
opinion and sentiment in front of the world. People
discussions, opinion and sentiment in the form of tweets on
two famous political face of county India are collected,
analyzed at real time by applying topic modeling technique
and sentiment analysis technique. Instead of reading
individual tweets of people, 18000 tweets are collected and
analyzed using LDA based topic molding method to identify
various topics of discussion related to two famous political
faces of country India. Further sentiment analysis of collected
tweets is carried out using lexicon based method to identify
inclination of people sentiment towards the each leaders.
Overall experimental analysis is carried out on SaprkR
framework to improve the preference and speed of processing.
To summarize a real time automated topic modeling and
sentiment analysis is performed over large set of related tweets
on SaprkR.
[1] D. Blei, A. Ng, and M. Jordan, “Latent dirichlet allocation,” Journal
of machine Learning research, 2003.
[2] H. Jelodar, Y. Wang, C. Yuan, and X. Feng, “Abstract :,” 2017.
[3] Y. G. Jung, K. T. Kim, B. Lee, and H. Y. Youn, “Enhanced Naive
Bayes Classifier for Real-time Sentiment Analysis with SparkR,” pp.
141–146, 2016.
[4] H. Anber, A. Salah, and A. A. A. El-aziz, “A Literature Review on
Twitter Data Analysis,” vol. 8, no. 3, pp. 241–249, 2016.
[5] S. Kuamri, “Real Time Analysis of Social Media Data to Understand
People Emotions Towards National Parties,” 2017.
[6] V. A. Rohani, S. Shayaa, G. Babanejaddehaki, and V. Ali, “Topic
Modeling for Social Media Content : A Topic Modeling for Social
Media Content : A Practical Approach Practical Approach,” pp. 397–
402, 2016.
[7] M. Sokolova, K. Huang, S. Matwin, J. Ramisch, R. Black, C. Orwa,
S. Ochieng, and N. Sambuli, “Topic Modelling and Event
Identification from Twitter Textual Data Keywords :”
[8] C. Xiao, P. Zhang, W. A. Chaowalitwongse, J. Hu, and F. Wang,
“Adverse Drug Reaction Prediction with Symbolic Latent Dirichlet
Allocation,” pp. 1590–1596, 2017.
[9] S. Lee, S. Kim, S. Lee, J. Choi, H. Yoon, D. Lee, and J. Lee,
“LARGen : Automatic Signature Generation for Malwares Using
Latent Dirichlet Allocation,” vol. X, no. X, 2016.
[10] Q. D. O. Vlviru, “6rfldo 0hgld 2slqlrq$qdo\vlviru ,qgldq 3rolwlfdo
’lsorpdwv,” pp. 681–686, 2017.
[11] X. Yang, D. Lo, L. Li, X. Xia, T. F. Bissyandé, and J. Klein,
“Characterizing malicious Android apps by mining topic-specific data
flow signatures,” vol. 90, pp. 27–39, 2017.
[12] D. Greene and J. P. Cross, “Unveiling the Political Agenda of the
European Parliament Plenary : A Topical Analysis,” 2015.
[13] C. Sievert, K. E. Shirley, and N. York, “LDAvis : A method for
visualizing and interpreting topics,” pp. 63–70, 2014.
[14] Y. El-khamra, N. Gaffney, D. Walling, E. Wernert, W. Xu, and H.
Zhang, “Performance Evaluation of R with Intel Xeon Phi
Coprocessor,” 2013.
[15] R. Huang, T. Advanced, W. Xu, and T. Advanced, “Performance
Evaluation of Enabling Logistic Regression for Big Data with R,” pp.
2517–2524, 2015.
[16] M. Banko, “A Comparison Of Expectation Maximization and Gibbs
Sampling Strategies for Motif Finding,” pp. 1–7, 2004.
[17] M. Ponweiser, “Latent Dirichlet Allocation in R,” no. May, 2012.

9th ICCCNT 2018

July 10-12, 2018, IISC, Bengaluru
Bengaluru, India
 IEEE - 43488

9th ICCCNT 2018

July 10-12, 2018, IISC, Bengaluru
Bengaluru, India