See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/264233723

Combining Classification and Clustering for Tweet Sentiment Analysis

Conference Paper · January 2014

DOI: 10.13140/2.1.1060.9281

CITATIONS READS

21 3,434

4 authors:

Luiz F. S. Coletta Nádia Félix

Dell Technologies Universidade Federal de Goiás
26 PUBLICATIONS 408 CITATIONS 52 PUBLICATIONS 687 CITATIONS

SEE PROFILE SEE PROFILE

Eduardo R Hruschka Estevam Rafael Hruschka

University of São Paulo 135 PUBLICATIONS 5,021 CITATIONS
114 PUBLICATIONS 4,744 CITATIONS
SEE PROFILE
SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Automatic Genetic Generation of Fuzzy Classification Systems. View project

Prophet View project

All content following this page was uploaded by Luiz F. S. Coletta on 26 July 2014.

The user has requested enhancement of the downloaded file.

 Combining Classification and Clustering for Tweet
Sentiment Analysis

Luiz F. S. Coletta∗ , Nádia F. F. da Silva∗ , Eduardo R. Hruschka∗ Estevam R. Hruschka Jr.†

∗ Instituteof Mathematics and Computer Science
University of Sao Paulo (USP) at Sao Carlos, Brazil
{luizfsc,nadia,erh}@icmc.usp.br
† Department of Computer Science

Federal University of Sao Carlos (UFSCAR) at Sao Carlos, Brazil

estevam@dc.ufscar.br

Abstract—The goal of sentiment analysis is to determine
opinions, emotions, and attitudes presented in source material. In
tweet sentiment analysis, opinions in messages can be typically
categorized as positive or negative. To classify them, researchers
have been using traditional classifiers like Naive Bayes, Maximum
Entropy, and Support Vector Machines (SVM). In this paper, we
show that a SVM classifier combined with a cluster ensemble can
offer better classification accuracies than a stand-alone SVM. In
our study, we employed an algorithm, named C3 E-SL, capable
to combine classifier and cluster ensembles. This algorithm can
refine tweet classifications from additional information provided
by clusterers, assuming that similar instances from the same
clusters are more likely to share the same class label. The
resulting classifier has shown to be competitive with the best
results found so far in the literature, thereby suggesting that the
studied approach is promising for tweet sentiment classification.
Keywords—Tweet Sentiment Analysis, Classification; Support
Vector Machines, Clustering, Cluster Ensemble.
I. I NTRODUCTION
Twitter is a popular microblogging service in which users
post status messages, called “tweets”, with no more than 140
characters. Such environment represents one of the largest
and most dynamic datasets of user generated content [1], [2].
Tweets can express opinions on different topics, which can
help to direct marketing campaigns to share consumer opinions
concerning brands and products [3], outbreaks of bullying
[4], events that generate insecurity [5], and acceptance or
rejection of politicians [6], all in an electronic word-of-mouth
way. Automatic tools have helped decision makers to ensure
efficient solutions for a plenty of problems. The focus of this
paper is on the sentiment analysis of tweets.
Sentiment analysis aims at identifying opinions, emotions,
and attitudes contained in source material like documents,
short texts, and sentences — e.g., from reviews [7], [8], [9],
[10], blogs [11], [12], and news [13]. In such domains, we
usually deal with large text corpora and “formal language”.
However, taking into account sentiment analysis of tweets, the
concerns are different, in particular due to: (i) the frequency
of misspellings and slang in tweets is much higher than in
other domains, since users frequently post messages (that can
contain a specific cultural vocabulary) using many different
electronic devices (e.g., cell phones and tablets); (ii) Twitter
users post messages on a variety of topics, unlike other sites,
which are tailored to specific topics. Tweet sentiment analysis
is considered a classification problem and, analogously to what
happens for large documents, sentiments of tweets can be
expressed in different ways [14]. Specifically, they can be
classified according to the existence of sentiment — i.e., if
there is sentiment in a message, such message is considered
polar (categorized as positive or negative), otherwise it is
considered neutral. Some authors consider six “universal”
emotions [15]: anger, disgust, fear, happiness, sadness, and
surprise. In our work, we adopt the view that tweet sentiments
can be either positive or negative, as in [16], [17], [18], [19].
Many researchers have been focusing on the use of tra-
ditional classifiers like Naive Bayes, Maximum Entropy, and
Support Vector Machines (SVM) to classify tweets. We show
that the combination of a supervised model and a cluster
ensemble [20], based on different data partitions, can improve
tweet classification accuracy. To the best of our knowledge,
no study has explored the combination of supervised and
unsupervised models for tweet classification. Our study takes
into account an algorithm introduced in [21], whose name is
C3 E (from Consensus between Classification and Clustering
Ensembles). This algorithm assumes that clusterers can provide
supplementary constraints that help to classify new data —
in particular, that similar instances are more likely to share
the same class label. To classify tweets, we have used a C3 E
version based on Squared Loss (SL) function, named C3 E-SL
[22]. Such an algorithm needs the information (a priori) of
two parameters, namely: the relative importance of classifier
and cluster ensembles, and the number of iterations of the
algorithm. To estimate these parameters, we have used a Dy-
namic Differential Evolution (D2 E) algorithm [22]. Our study
shows that the refinements provided by C3 E-SL (optimized by
means of D2 E) can improve the classification accuracy of a
stand-alone SVM classifier for tweet sentiment analysis.
II. R ELATED WORK
Several studies on the use of stand-alone classifiers for
tweet sentiment analysis are available in the literature. Some of
them propose the use of emoticons and hashtags for building
the training set, as Go et al. [23] and Davidov et al. [24], who
identified tweet polarity by using emoticons as class labels.
Others have used the characteristics of the social network as
networked data, such as Hu et al. [25]. In their paper, emotional
contagion theories are materialized based on a mathematical
optimization formulation for the supervised learning process.
 Approaches that integrate learning methods and opinion More precisely, the objective becomes to minimize J in (1)
mining lexicon-based techniques have been studied by Agar- with respect to {yi }ni=1 :
wal et al. [26], Read [27], Zhang et al. [28], and Saif et
al. [29]. They have used lexicons, part-of-speech, and writing
style as linguistics resources. In a similar context, Saif et al. 1X 1 X
J= kyi − π i k2 + α sij kyi − yj k2 (1)
[30] introduced an approach to add semantics to the sentiment 2 2
i∈X (i,j)∈X
analysis training set as an additional feature. For each extracted
entity (e.g., iPhone), they added its respective semantic concept
(like “Apple’s product”) and measured the correlation of the By keeping {yj }nj=1 \ {yi } fixed, we can minimize J in
representative concept as negative/positive sentiments. Equation (1) for every yi by setting:

To achieve better results in classification/clustering ap-

∂J
plications, many approaches based on ensembles have been = 0. (2)
developed and used in practice — for classification problems ∂yi
[31], [32], as for clustering problems [33], [34], [35], [36].
To take advantage of both labeled and unlabeled data, several Considering that the similarity matrix S is symmetric and
researches have designed ways of combining classifiers and observing that ∂kxk2
∂x = 2x, we obtain:
clusterers [22], [37], [38], [39], [40], [41], [21], [42], [43].
Acharya et al. [21] and Gao et al. [43], in particular, deal X
π i + α′ sij yj
with the combination of a handful of classifiers and clusterers
j6=i
with the ultimate goal of classifying new data. Coletta et yi = X , (3)
al. [22] used a Squared Loss (SL) function in the optimiza- 1 + α′ sij
tion algorithm introduced in [21]. Such an algorithm, named j6=i
C3 E-SL, has presented attractive empirical results, besides
being computationally efficient in practice [22], [37], [21]. where α′ = 2α has been set for mathematical conve-
Our work explores the combination of classifiers and nience. Equation (3) can be computed iteratively, for all
clusterers in tweet sentiment classification by using C3 E-SL i ∈ {1, 2, ..., n}, until a maximum number of iterations (I),
[22]. We consider that the additional information provided in order to obtain posterior class probability distributions for
by data partitions built from bag-of-words representation with the instances in X .
lexicons scores can boost classification accuracies yielded by Figure 1 shows an overview of our approach to refine tweet
classifiers. In this sense, clusterers can discover a type of “topic classification. In order to optimize the two user-defined param-
structure” present in the data encoding that is in the form of eters of C3 E-SL, namely the coefficient α, which controls the
meta-information. relative importance of classifier and cluster ensembles, and its
number of iterations, I, we have used a differential evolution
III. R EFINING TWEET CLASSIFICATION VIA CLUSTERING algorithm, as described next.

In [21], the authors designed a framework to combine clas-
sifiers and clusterers aiming at generating a more consolidated
classification. Such a framework, whose core is an optimization
algorithm named C3 E, assumes that an ensemble of classifiers
(consisting of one or more classifiers) have been previously in-
duced from a training set. This ensemble estimates initial class
probability distributions for every instance xi of a target/test
set X = {xi }ni=1 . These probability distributions are stored
as c-dimensional vectors {πi }ni=1 , where c is the number
of classes. Assuming that a cluster ensemble can provide
supplementary constraints for such a previous classification,
the probability distributions in {πi }ni=1 are refined taking
into account a similarity matrix S = {sij }ni,j=1 — with the
rationale that similar instances are more likely to share the
same class label. Matrix S captures similarities between the
instances of X . In particular, each entry corresponds to the
relative co-occurrence of two instances in the same cluster
[20], [36] — considering all the data partitions built from X .
The classification refinement is obtained from a posterior
class probability distribution for every instance in X — defined
as a set of vectors {yi }ni=1 — that is provided by C3 E.
In [21], C3 E exploits general properties of a large class of
loss functions, described by Bregman divergences [44]. If we
opt to use a Squared Loss (SL) function as the Bregman
divergence, the optimization problem can be simplified [22].
A. Estimating the C3 E-SL Parameters
The C3 E-SL user-defined parameters, α and I, can be
successfully estimated by means of a Dynamic Differential
Evolution (D2 E) [22] algorithm, which extends DE [45], [46]
by sampling values for its control parameters, F and Cr.
The dynamic adaptation takes place if in two consecutive
generations, there are no changes in the average fitness of the
current population. To do so, D2 E uses the following rules:
if (f¯G = f¯G−1 )

<d3 , d4 >,
<F, Cr>G+1 = (4)
<F, Cr>G , otherwise
where d3 ∈ [0.1, 1], d4 ∈ [0, 1] are values randomly chosen
from a uniform distribution, f¯G and f¯G−1 are the average fit-
ness of the population in generation G and G− 1, respectively.
It allows F and Cr to assume new values if there is no im-
provement in two consecutive generations. Therefore, D2 E can
avoid local minima (with sufficiently high values of F and/or
Cr), as well as it can emphasize exploitation (with sufficiently
low values of F and/or Cr), making the algorithm more robust
with respect to initial values of its control parameters. D2 E
is based on the strategy “DE/best/2/bin” [46] and starts with
N p = 20, F = 0.25, and Cr = 0.25.
Fig. 1. Improving tweet classification by using C3 E-SL algorithm [22]. Class probability distributions computed by SVM are refined taking into account the
information of clusters on the data to be classified.
To optimize the C3 E-SL parameters, D2 E aims at min-
imizing the misclassification rate provided by the algorithm
in a validation set by combining different values of α =
{0, 0.001, 0.002, ..., 0.15} and I = {1, 2, ..., 10}. After doing
that, C3 E-SL with α∗ and I ∗ can be used to classify tweets in
a target/test set.
IV. E XPERIMENTAL RESULTS
Experiments were conducted to evaluate the improvements
achieved when a SVM classifier is combined with a cluster
ensemble by means of C3 E-SL algorithm (described in Section
III). Comparisons were made between the stand-alone SVM
and its outcomes refined by the C3 E-SL approach. In addition,
for comparison purposes, the best classification results reported
in the literature are also presented. Details on the used datasets
and the performed preprocessing steps are described next.
A. Datasets
Experiments have been carried out on four representative
datasets obtained from tweets on different subjects:
1) Health Care Reform (HCR): This dataset was built by
crawling tweets containing the hashtag “#hcr” (health care
reform) in March 2010 [2]. A subset of this corpus was
manually annotated as positive, negative, and neutral. The
neutral tweets were not considered in the experiments. Thus
our training set contains 621 examples (215 positive and 406
negative) whereas the test set contains 665 (154 positive and
511 negative).
2) Obama-McCain Debate (OMD): This dataset was con-
structed from 3,238 tweets crawled during the first U.S.
presidential TV debate that took place in September 2008 [47].
The sentiment ratings of the tweets were acquired by using the
Amazon Mechanical Turk1. Each tweet was rated as positive,
negative, mixed, and other. “Other” tweets are those that could
not be rated. We kept only the positive and negative tweets
rated by at least three voters, which comprise a set of 1,906
tweets, from which 710 are positive and 1,196 are negative.
1 https://www.mturk.com/
3) Sanders - Twitter Sentiment Corpus: It consists of hand-
classified tweets collected from four Twitter search terms:
@apple, #google, #microsof t, #twitter. Each tweet has a
sentiment label: positive, neutral, negative, and irrelevant. As
in [48], we conducted experiments considering only the set
formed from 570 positive tweets and 654 negative ones.
4) Stanford - Twitter Sentiment Corpus: This dataset [23]
has 1.6 million training tweets (half tweets with positive
emoticons and another half with negative emoticons) collected
by using a scraper that queries the Twitter API. The test set was
collected by searching the Twitter API with specific queries
including product names, companies, and people. 359 tweets
were manually annotated — 177 with negative class label and
182 with positive class label. We used four training sets (with
500; 1,000; 2,000; and 3,000 tweets) sampled from the original
training set.
B. Preprocessing
Before addressing the preprocessing step, it is instructive to
describe the way the tweets were encoded. We employed bag-
of-words, where tweets are described by a table in which the
columns represent the terms (or existing words) in the tweets.
The values in the table represent the frequency of the terms in
the tweet. Hence, a collection of tweets can be represented as
illustrated in Table I, in which there are n tweets and m terms.
Each tweeti = (ai1 , ai2 , · · · , aim ), where aij is the frequency
of term tj . The frequency can be computed in various ways.
We used binary frequency, assuming that a term is considered
“frequent” if it occurs in more than one tweet.
TABLE I. R EPRESENTATION OF TWEETS AS BAG - OF - WORDS .
t1 t2 ··· tm
tweet1 a11 a12 ··· a1m
tweet2 a21 a22 ··· a2m
··· ··· ··· ··· ···
tweetn an1 an2 ··· anm
To preprocess the tweet datasets, we first removed retweets,
stop words, links, URLs, mentions, punctuation, and accentu-
ation. Stemming was performed in order to minimize sparsity.
 We also used the opinion lexicon2 proposed by Hu and Liu
[9], who created a list of 4,783 negative words and 2,006 pos-
itive ones. This list was compiled over many years, and each
word indicates an opinion. Positive opinion words are used
to express desired states, while negative opinion expresses the
opposite. Examples of positive opinion words are: beautiful,
wonderful, good, and amazing. Examples of negative opinion
words are: bad, poor, and terrible. We computed and included
the number of positive and negative lexicons in each tweet
in order to enrich the data. Emoticons available in the tweets
have also been used in the feature set. Thus, the number of
positive and negative emotions were counted to complement
the information provided by the bag-of-words and lexicons.
C. SVM and Cluster Ensemble Settings
For simplicity and aimed at showing a proof of concept,
we have used an ensemble formed by a single component,
namely a non-linear SVM with RBF kernel. The RBF kernel
parameters, C and γ, were estimated by grid-search on the
training set according to Hsu et al. [50]. As for the clustering
component, we built a similarity matrix from five data parti-
tions induced by using K-Medoids clustering algorithm [51]
with cosine similarity. Based on Kuncheva et al. [52], [53],
each data partition assumed a specific value for the number of
clusters k = {2,3,5,6,8}. These values were randomly selected
and fixed for all datasets.
D. Optimization of C3 E-SL
The C3 E-SL optimization consists of searching for values
of its parameters, α and I, that could yield to good classifi-
cation accuracy. Ideally, we are looking for an optimal pair of
values (α∗ and I ∗ ) that results in the most accurate classifier
for tweet sentiment analysis. In order to estimate these values,
we employ D2 E algorithm (addressed in Section III-A and that
shown good results in other application domains [22]).
The procedure adopted involves three datasets: training,
validation, and test [54]. The test set (with instances to be
classified) has not been used in the process of building the
ensemble of classifiers — i.e., it is an independent set of data
not used at all to optimize any parameter of the algorithm
and the resulting classifier. Thus, as usual and in conformity
with real-world applications (on tweet classification), only the
training and validation sets are used to optimize the parameters
of C3 E-SL. In this work, the validation set was formed with
half the training set available. In this sense, firstly a SVM is
built with 50% of the training set. Then, D2 E estimates α∗ and
I ∗ by minimizing the misclassification rate in the validation
set (formed with the other 50% of the available training set),
where a cluster ensemble was induced. Thereafter, the resulting
values for the parameters are finally employed to assess the
classification accuracy in the test set where, again and as
requested by C3 E-SL, a cluster ensemble similar to the one
built for the validation set must be induced. In this step, a new
SVM was built considering all the available labeled data (i.e.,
considering 100% of the original training set).
For Health Care Reform (HCR) [30] and Stanford [23]
datasets, results were obtained over 10 runs from the same
training and test sets available in their public resources.
2 Available at http://www.cs.uic.edu/∼ liub/FBS/sentiment-analysis.html.
For the latter, in particular, we sampled training sets (with
different sizes) from its original training set. For Obama-
McCain Debate (OMD) and Sanders datasets, results were also
obtained over 10 runs, but considering the 2×5 cross-validation
procedure [54].
E. Empirical Results
Table II shows results provided by SVM and their refine-
ments by using C3 E-SL on each dataset. The best results found
in the literature are also reported. The F-Measure (F1) has
been showed because some datasets have unbalanced data. By
observing these results, one can see that the combination of a
classifier with clusterers, as done by C3 E-SL algorithm, can
provide improvements in comparison with the stand-alone use
of SVM, which is considered a state-of-the-art classifier.
For the HCR dataset, C3 E-SL provided substantial en-
hancements over the SVM results. Also, better results than
those reported in the literature [2], [30] were obtained —
except for the F-Measure computed on positive tweets.
For OMD and Sanders datasets, we can note some slight
improvements, which are close to the best results in the
literature [25], [30], [48]. Considering the positive class F-
Measure on OMD, C3 E-SL achieved better results (80.58%)
in comparison to that found in the literature (70.30%).
For Stanford, we performed experiments on four different
training sets (sampled with 500; 1,000; 2,000; and 3,000
balanced tweet examples from the original training set). The
C3 E-SL classification refinements provided better results than
SVM in all these scenarios (except for that with 2,000 training
tweets). C3 E-SL also showed competitive accuracies in com-
parison with the results found in the literature. However, it
is worth noticing that the results reported in [49], [30] were
obtained from the whole training set (with 1.6 million tweets).
So our results are very encouraging, thus suggesting that
the algorithm being studied is promising for tweet sentiment
classification.
V. C ONCLUSIONS
Tweet sentiment analysis has been extensively studied in
recent years. To classify tweet messages, researchers have been
using stand-alone classifiers like Support Vector Machines
(SVM). We showed that the combination of a SVM classifier
with a cluster ensemble can boost the classification of tweets
— with the assumption that similar instances are more likely
to share the same class label. For tweet classification, in
particular, such assumption sounds reasonable, since tweet
messages of a specific topic (to be explored) usually have a
great number of opinions that can be (dis)similar to each other,
but either classified as positive or negative. Different clusters
can encompass similar positive/negative opinions, thereby pro-
viding additional information about the structure of the data,
which can help to classify tweets by using C3 E-SL.
Experimental results obtained on four datasets showed
that the combination of a SVM classifier with a ensemble
cluster, as done by C3 E-SL algorithm, can improve the tweet
classification. In most of the cases, C3 E-SL provided bet-
ter results than the stand-alone SVM. Furthermore, C3 E-SL
showed competitive results in comparison with those found in
TABLE II. AVERAGE CLASSIFICATION ACCURACIES (%) AND F-M EASURES (%) OBTAINED BY USING SVM AND C3 E-SL ON EACH DATASET. F OR THE
S TANFORD DATASET, WE SAMPLED TRAINING SETS WITH DIFFERENT SIZES ( BEST RESULTS IN BOLD ). F OR COMPARISON PURPOSES , THE BEST RESULTS
FROM THE LITERATURE ARE ALSO PRESENTED .

Accuracy (%) F1 - Positive Class (%) F1 - Negative Class (%)

Dataset
SVM C3 E-SL SVM C3 E-SL SVM C3 E-SL

Health Care Reform (HCR) 74.29 79.62 43.56 32.13 83.35 88.00

Literature [2], [30] 71.20 50.30 86.00

Obama-McCain Debate (OMD) 75.15 75.18 80.14 80.58 67.24 64.24

Literature [25], [30] 76.30 70.30 85.40

Sanders 82.11 82.15 80.43 80.80 83.48 83.27

Literature [48] 84.40 - -

Stanford - trained with 500 tweets 74.65 77.69 74.65 77.33 74.65 78.02
Stanford - trained with 1000 tweets 77.72 79.69 77.40 80.21 78.02 79.13
Stanford - trained with 2000 tweets 78.83 77.80 79.79 80.92 77.78 73.46
Stanford - trained with 3000 tweets 79.39 81.84 79.56 82.36 79.21 81.27
∗ ∗ ∗
Literature [49], [30] 87.20 82.50 85.30
∗
By using the whole training set — i.e., 1.6 million tweets.

the literature. We note that for the HCR dataset our accuracy is
the best one reported so far in the literature. For the Stanford
dataset, the best accuracy found, obtained with the complete
training set (formed by 1,600,000 tweets), is 87.20%, whereas
our approach trained with only 0.19% of these data achieved an
average accuracy of 81.84%. Although more experiments are
necessary before stating stronger claims, the studied algorithm
has shown to be promising for tweet sentiment analysis.
There are several aspects that can be investigated in the
future. For example, ways of inducing “good” data partitions
to compose a cluster ensemble deserve attention. In particular,
clustering algorithms such as those based on Latent Dirichlet
Allocation [55] can be used to explicitly find topics in data.
Also, the use of active learning to leverage the combination
results of classifiers and clusterers is a promising future work.
ACKNOWLEDGMENT
The authors would like to acknowledge Brazilian Re-
search Agencies Capes (Proc. DS-7253238/D), CNPq (Proc.
303348/2013-5) and FAPESP (Proc. 2013/07375-0 and
2010/20830-0) for the financial support.
R EFERENCES
[1] A. Ritter, S. Clark, Mausam, and O. Etzioni, “Named entity recognition
in tweets: An experimental study,” in Proceedings of the Conference
on Empirical Methods in Natural Language Processing, ser. EMNLP
’11. Stroudsburg, PA, USA: Association for Computational Linguistics,
2011, pp. 1524–1534.
[2] M. Speriosu, N. Sudan, S. Upadhyay, and J. Baldridge, “Twitter polarity
classification with label propagation over lexical links and the follower
graph,” in Proceedings of the First Workshop on Unsupervised Learning
in NLP, ser. EMNLP ’11. Stroudsburg, PA, USA: Association for
Computational Linguistics, 2011, pp. 53–63.
[3] B. J. Jansen, M. Zhang, K. Sobel, and A. Chowdury, “Twitter power:
Tweets as electronic word of mouth,” J. Am. Soc. Inf. Sci. Technol.,
vol. 60, no. 11, pp. 2169–2188, nov 2009.
[4] J.-M. Xu, K.-S. Jun, X. Zhu, and A. Bellmore, “Learning from
bullying traces in social media.” in HLT-NAACL. The Association
for Computational Linguistics, 2012, pp. 656–666.
[5] M. Cheong and V. C. Lee, “A microblogging-based approach to ter-
rorism informatics: Exploration and chronicling civilian sentiment and
response to terrorism events via twitter,” Information Systems Frontiers,
vol. 13, no. 1, pp. 45–59, Mar. 2011.
[6] N. A. Diakopoulos and D. A. Shamma, “Characterizing debate perfor-
mance via aggregated twitter sentiment,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ser. CHI ’10.
New York, NY, USA: ACM, 2010, pp. 1195–1198.
[7] P. D. Turney, “Thumbs up or thumbs down?: semantic orientation
applied to unsupervised classification of reviews,” in Proceedings of
the 40th Annual Meeting on Association for Computational Linguistics,
ser. ACL 02. Stroudsburg, PA, USA: Association for Computational
Linguistics, 2002, pp. 417–424.
[8] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? sentiment classifi-
cation using machine learning techniques,” in Proceedings of EMNLP,
2002, pp. 79–86.
[9] M. Hu and B. Liu, “Mining and summarizing customer reviews,” in
Proceedings of the tenth ACM SIGKDD international conference on
Knowledge discovery and data mining, ser. KDD ’04. New York, NY,
USA: ACM, 2004, pp. 168–177.
[10] ——, “Mining opinion features in customer reviews,” in Proceedings
of the 19th national conference on Artifical intelligence, ser. AAAI’04.
AAAI Press, 2004, pp. 755–760.
[11] B. He, C. Macdonald, J. He, and I. Ounis, “An effective statistical
approach to blog post opinion retrieval,” in Proceedings of the 17th
ACM conference on Information and knowledge management, ser.
CIKM ’08. New York, NY, USA: ACM, 2008, pp. 1063–1072.
[12] P. Melville, W. Gryc, and R. D. Lawrence, “Sentiment analysis of
blogs by combining lexical knowledge with text classification,” in
Proceedings of the 15th ACM SIGKDD international conference on
Knowledge discovery and data mining, ser. KDD ’09. New York, NY,
USA: ACM, 2009, pp. 1275–1284.
[13] A. Balahur, R. Steinberger, M. Kabadjov, V. Zavarella, E. van der
Goot, M. Halkia, B. Pouliquen, and J. Belyaeva, “Sentiment analysis
in the news,” in Proceedings of the Seventh International Conference
on Language Resources and Evaluation (LREC’10), N. C. C. Chair),
K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner,
 and D. Tapias, Eds. Valletta, Malta: European Language Resources
Association (ELRA), may 2010.
[14] A. Montoyo, P. Martnez-Barco, and A. Balahur, “Subjectivity and
sentiment analysis: An overview of the current state of the area and
envisaged developments,” Decision Support Systems, vol. 53, no. 4, pp.
675 – 679, 2012.
[15] P. Ekman, Emotion in the Human Face. Cambridge University Press,
1982, vol. 2.
[16] B. Liu, Sentiment Analysis and Opinion Mining, ser. Synthesis Lectures
on Human Language Technologies. Morgan & Claypool Publishers,
2012.
[17] ——, Web Data Mining: Exploring Hyperlinks, Contents, and Usage
Data, ser. Data-Centric Systems and Applications. Secaucus, NJ, USA:
Springer-Verlag New York, Inc., 2006.
[18] ——, Sentiment analysis and subjectivity. Taylor and Francis Group,
Boca, 2010.
[19] A. Agarwal and P. Bhattacharyya, “Sentiment analysis: A new approach
for effective use of linguistic knowledge and exploiting similarities in a
set of documents to be classified,” in Proceedings of the International
Conference on Natural Language Processing (ICON), 2005.
[20] J. Ghosh and A. Acharya, “Cluster ensembles,” Wiley Interdisc. Rew.:
Data Mining and Knowledge Discovery, vol. 1, no. 4, pp. 305–315,
2011.
[21] A. Acharya, E. R. Hruschka, J. Ghosh, and S. Acharyya, “C3E: A
framework for combining ensembles of classifiers and clusterers,” in
Multiple Classifier Systems. LNCS Vol. 6713, Springer, 2011, pp.
269–278.
[22] L. F. S. Coletta, E. Hruschka, A. Acharya, and J. Ghosh, “A differential
evolution algorithm to optimise the combination of classifier and cluster
ensembles,” International Journal of Bio-Inspired Computation, 2014,
(to appear).
[23] A. Go, R. Bhayani, and L. Huang, “Twitter sentiment classification us-
ing distant supervision,” Unpublished Manuscript, Stanford University,
pp. 1–6, 2009.
[24] D. Davidov, O. Tsur, and A. Rappoport, “Enhanced sentiment learning
using twitter hashtags and smileys,” in Proceedings of the 23rd Interna-
tional Conference on Computational Linguistics: Posters, ser. COLING
’10. Stroudsburg, PA, USA: Association for Computational Linguistics,
2010, pp. 241–249.
[25] X. Hu, L. Tang, J. Tang, and H. Liu, “Exploiting social relations for
sentiment analysis in microblogging,” in Proceedings of the sixth ACM
international conference on Web search and data mining, 2013.
[26] A. Agarwal, B. Xie, I. Vovsha, O. Rambow, and R. Passonneau,
“Sentiment analysis of twitter data,” in Proceedings of the Workshop
on Languages in Social Media, ser. LSM ’11. Stroudsburg, PA, USA:
Association for Computational Linguistics, 2011, pp. 30–38.
[27] J. Read, “Using emoticons to reduce dependency in machine learning
techniques for sentiment classification,” in Proceedings of the ACL
Student Research Workshop, ser. ACLstudent ’05. Stroudsburg, PA,
USA: Association for Computational Linguistics, 2005, pp. 43–48.
[28] L. Zhang, R. Ghosh, M. Dekhil, M. Hsu, and B. Liu, “Combining
lexicon-based and learning-based methods for twitter sentiment analy-
sis.” HP Laboratories, 2011, technical Report.
[29] S. Mohammad, S. Kiritchenko, and X. Zhu, “Nrc-canada: Building
the state-of-the-art in sentiment analysis of tweets,” in Proceedings of
the seventh international workshop on Semantic Evaluation Exercises
(SemEval-2013), Atlanta, Georgia, USA, June 2013.
[30] H. Saif, Y. He, and H. Alani, “Semantic sentiment analysis of twitter,” in
Proceedings of the 11th international conference on The Semantic Web
- Volume Part I, ser. ISWC’12. Berlin, Heidelberg: Springer-Verlag,
2012, pp. 508–524.
[31] N. C. Oza and K. Tumer, “Classifier ensembles: Select real-world
applications,” Information Fusion, vol. 9, no. 1, pp. 4–20, 2008.
[32] L. I. Kuncheva, Combining Pattern Classifiers: Methods and Algo-
rithms. Hoboken, NJ: Wiley-Interscience, 2004.
[33] R. Ghaemi, N. Sulaiman, H. Ibrahim, and N. Mustapha, “A Survey:
Clustering Ensembles Techniques,” Proceedings of World Academy of
Science, Engineering and Technology, vol. 38, February 2009.
View publication stats
[34] P. Hore, L. Hall, and D. Goldgof, “A scalable framework for cluster
ensembles,” Pattern Recognition, vol. 42, no. 5, pp. 676–688, 2009.
[35] A. Gionis, H. Mannila, and P. Tsaparas, “Clustering aggregation,” in
ACM Trans. Knowl. Discov. Data. New York, NY, USA: ACM, 2007,
vol. 1.
[36] A. Strehl and J. Ghosh, “Cluster Ensembles - A Knowledge Reuse
Framework for Combining Multiple Partitions,” Journal of Machine
Learning Research, vol. 3, pp. 583–617, 2002.
[37] A. Acharya, E. R. Hruschka, J. Ghosh, and S. Acharyya, “An optimiza-
tion framework for combining ensembles of classifiers and clusterers
with applications to non-transductive semi-supervised learning and
transfer learning,” ACM Transactions on Knowledge Discovery from
Data, 2014, (to appear).
[38] H. Gan, N. Sang, R. Huang, X. Tong, and Z. Dan, “Using clustering
analysis to improve semi-supervised classification,” Neurocomputing,
vol. 101, pp. 290 – 298, 2013.
[39] A. Rahman and B. Verma, “Cluster-based ensemble of classifiers,”
Expert Systems, vol. 30, no. 3, pp. 270–282, 2013.
[40] R. Soares, H. Chen, and X. Yao, “Semisupervised classification with
cluster regularization,” Neural Networks and Learning Systems, IEEE
Transactions on, vol. 23, no. 11, pp. 1779–1792, 2012.
[41] Q. Qian, S. Chen, and W. Cai, “Simultaneous clustering and classifica-
tion over cluster structure representation,” Pattern Recognition, vol. 45,
no. 6, pp. 2227 – 2236, 2012.
[42] W. Cai, S. Chen, and D. Zhang, “A multiobjective simultaneous learning
framework for clustering and classification,” Neural Networks, IEEE
Transactions, vol. 21, no. 2, pp. 185–200, 2010.
[43] J. Gao, F. Liangy, W. Fanz, Y. Suny, and J. Han, “Graph-based
Consensus Maximization among Multiple Supervised and Unsupervised
Models,” in 23rd Annual Conference on Neural Information Processing
Systems, December 2009.
[44] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh, “Clustering with
bregman divergences,” J. Mach. Learn. Res., vol. 6, pp. 1705–1749,
2005.
[45] R. Storn and K. Price, “Differential evolution - a simple and efficient
heuristic for global optimization over continuous spaces,” Jornal of
Global Optimization, vol. 11, no. 4, pp. 341–359, 1997.
[46] K. Price, “Differential evolution: a fast and simple numerical optimizer,”
in Fuzzy Information Processing Society, 1996. NAFIPS. 1996 Biennial
Conference of the North American. IEEE, 1996, pp. 524–527.
[47] D. A. Shamma, L. Kennedy, and E. F. Churchill, “Tweet the debates:
understanding community annotation of uncollected sources,” in Pro-
ceedings of the first SIGMM workshop on Social media. ACM, 2009,
pp. 3–10.
[48] D. Ziegelmayer and R. Schrader, “Sentiment polarity classification
using statistical data compression models.” in ICDM Workshops,
J. Vreeken, C. Ling, M. J. Zaki, A. Siebes, J. X. Yu, B. Goethals, G. I.
Webb, and X. Wu, Eds. IEEE Computer Society, 2012, pp. 731–738.
[49] A. Bakliwal, P. Arora, S. Madhappan, N. Kapre, M. Singh, and
V. Varma, “Mining sentiments from tweets,” in Proceedings of the 3rd
Workshop on Computational Approaches to Subjectivity and Sentiment
Analysis (WASSA), vol. 12, 2012.
[50] C.-W. Hsu, C.-C. Chang, C.-J. Lin et al., “A practical guide
to support vector classification, 2003,” Paper available at
http://www.csie.ntu.edu.tw/ cjlin/papers/guide/guide.pdf, 2010.
[51] L. Kaufman and P. Rousseeuw, Clustering by means of medoids. North-
Holland, 1987.
[52] L. Kuncheva, S. Hadjitodorov, and L. Todorova, “Experimental com-
parison of cluster ensemble methods,” in Information Fusion, 2006 9th
International Conference on, July 2006, pp. 1–7.
[53] L. I. Kuncheva and S. T. Hadjitodorov, “Using diversity in cluster
ensembles,” in Systems, man and cybernetics, 2004 IEEE international
conference on, vol. 2. IEEE, 2004, pp. 1214–1219.
[54] I. H. Witten and E. Frank, Data mining: practical machine learning
tools and techniques, 2nd ed., ser. Morgan Kaufmann Series in Data
Management Systems. Morgan Kaufmann, 2005.
[55] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,”
J. Mach. Learn. Res., vol. 3, pp. 993–1022, Mar. 2003.