Polarity Analysis of Editorial Articles towards Fake News
Detection
Mary Jane C. Samonte
Mapua University
Makati City, Philippines 1200,
+632475000
mjcsamonte@yahoo.com
ABSTRACT
The need in verifying online information is essential to identify
the lines between fake news and factual information. Social media
has become the platform for the digital production of news articles.
It can be found from various sources -- blogs to social networking
sites, or even online forums. This indicates how potentially fake
news can influence the overall opinion of the masses. This study
aims to create a model that categorizes online editorial articles and
use different classifier to determine its polarity through sentiment
analysis. This is a step first taken in order to detect fake against
real news online through data mining. In this study, online news
articles from various known websites were extracted in order to
develop a model. The researcher demonstrate that news articles
can be analyzed and showed effective results through the
performance of the classifiers used in this study.
CCS Concepts
Information systems→Information retrieval→Retrieval tasks
and goals→Sentiment analysis
Keywords
Fake News; Classifier; Editorial Articles; Model; Sentiment
Analysis.
1. INTRODUCTION
Applying Natural Language Processing (NLP) is one way of
identifying deception in news reports [1]. NLP is a ubiquitous
phenomenon that is used to provide meaningful results swiftly
such as autocomplete, spelling correctors, grammar checkers and
even technical paper critiques [2]. In 2015, Associated Press
published an entire financial story in an automated system of AP
Style Guide which was done by NLP bots [3]. NLP is accelerating
its research area as it expands its power to new industries in
forecasting unstructured data from social media like tweets and
Facebook posts [4].
But those mentioned social media platforms are also known in
proliferating fake news resulted to chaos and misinformed
decision makers. There are some news articles whose polarity is
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
ICIEB 2018, April 25–27, 2018, Singapore, Singapore
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-6375-4/18/04…$15.00
https://doi.org/10.1145/3230348.3230354
108
leading to predictions by what we call “social bots” or simply
“bots” [5].
The influence of fake online news in social media was seen in the
result of US presidential election and Brexit vote in 2016 [6].
Fake news and post-truth are the key factors of media-political
events. Fake news detection is defined as the prediction of the
chances of a particular news article, report editorial or expose
with an intention to deceive while post-truth is something that one
conveniently believe, politically, that can be based on emotion or
assumption. The era of post-truth is outside the conformity of
standards of evidentiary support [7], [8], and [9].
In a special online report by “Digital 2017: A Global Overview”,
Internet users grew by 10% in 2016, which is up 354 million
compared to 2015; Active social media users increased by 21%,
up 482 million versus 2015; Unique mobile users grew by 5%, up
222 million over the past 12 months; and, mobile social media
users grew by 30%, up an impressive 581 million in 2016. Social
media growth rates accelerated by more than 50% versus 2015,
with social media use overall up 25%, compared to 14% growth in
2016’s report. [10]. Today, there is an increase in the number of
Americans consumes daily news from mobile devices which gone
up from 54% in 2013 to 72% as of September 2017. So there are
many individuals that are exposed in reading not only fake online
news, but also other forms of “misinformation” [5], [11].
Journalistic deception has been defined as a “prima facie wrong”
which means “to deceive is morally wrong”. Journalism is a
responsibility where one should communicate messages, whether
verbal or nonverbal, without any intention of holding an
information that can be used to create a false belief [12].
Data mining has been interchange with machine learning. Data
mining refers to the practice of obtaining processed data through
the use of algorithms of machine learning [13]. Main examples of
machine learning algorithms are Decision trees, Neural Networks,
Support Vector Machines and Naïve Bayes. Sentiment analysis is
the process of computationally identifying and categorizing
opinions expressed in a piece of text, especially in order to
determine whether the writer’s attitude towards a particular topic,
product, etc. is positive, negative or neutral. In social media,
sentiment analysis can be used as a tool in simplifying this study
which is presumed as a complex topic [14].
Online news articles, defined as any document with a publicly-
accessible Uniform Resource Location (URL) with a headline,
posted on news sites that presents accumulated about a common
event or subject [15].
1.1 Objective of the Study
The objective of this study is to be able to determine a news
article’s overall sentiment through its editorial content. Datasets
used in this study was annotated by field experts. The study
developed a model that can correctly classify an article’s polarity
(negative/neutral/positive) using annotated dataset. Along the
experiment, the researcher determined which classifier is better
for detecting negative/neutral/positive opinions from news articles
based on accuracy rate and Kappa score [16].
1.2 Scope of the Study
For training the classifiers, the researcher used a dataset,
composing of 200 news articles that were gathered from the
current top 10 most frequently visited news websites for the
current month based on Alexa [17], a digital marketing tool that
can monitor internet traffic; and evaluated by 3 annotators. The
testing data were composed of 20 news articles gathered from the
same 10 sources. The websites chosen from the list are official
news sites such as Cable News Network (CNN), The New York
Times, etc. The news articles gathered are editorials only and shall
be gathered manually from the chosen websites. The articles are
then classified at the document-level. The study compares k-NN,
Naïve Bayes, and SVM as the classifiers in sentiment analysis in
RapidMiner [18]. The criteria for judging the performance of each
classifier are based on accuracy.
2. REVIEW OF PREVIOUS STUDIES
In order to define the term “fake news”, presentation of different
literatures can help us: 1) Checking of the news authenticity and
intent where false information can be verified and its intention to
mislead the reading public [19]; 2) Focus on either authenticity or
intent of the news content is deceiving to the readers which can be
entertainment-based [20]; and 3) The news includes grave
fabrications, hoaxes, and satires [21].
The impact of fake news extends to the current trend towards
political, emotional and financial aspect where editorial interests
is unfavorable to democratic values [5], [22], and [23]. Fake News
Challenge 1 (FNC-1) was launched in 2017 in Stanford University
with the goal of developing tools to help fact checkers tag fake
news. FNC-1. This works as there is a given article that acts
as ”ground truth” then a number of headlines was classified as
unrelated, agree, disagree, or discuss in relation to the article. But
the organizers of FNC-1 hoped stance detection problem would be
better if it was used to leverage as filters to limit the number of
articles the fact checkers will have to examine by hand [24].
Figure 1 describes the background of fake news detection problem
using theories and properties from psychology and social studies
[25]. Traditional news media is characterized by psychological
and social foundations of fake news while fake news on social
media include malicious accounts and echo chamber.
Psychological aspect refers to the cognitive weakness of human to
detect real and fake news. Social theory is all about decision
making of an individual’s gain or loss that applies to the reward
system or social acceptance that contributes to the spread of fake
news [26], [27]. Malicious accounts has been rampant due to the
low cost production of social bots, trolls and cyborg users in
social media. Studies shows that social bots. Human tend to trust
social contacts marked by the distorted 2016 U.S. presidential
election online discussions [28].
109
Figure 1. Fake news on social media: from characterization to
detection [25].
A study in “clickbait” was used in order to detect false news in
tabloids. Clickbait or headlines are used to tickle the curious
minds of the readers to clink the link in a particular website.
Clickbait has been linked to the rapid spread of rumor and
misinformation online. The paper discussed the possible methods
for the automatic detection of clickbait as a form of deception
through textual and non-textual clickbaiting cues [7].
A fake news detection system was developed by a group of
researchers which allows users to filter and mark possible
misleading information. They collected their data using web
scraping from various sites like Twitter feeds (real and fake),
BBC news, UCI news dataset and Kaggle news dataset. They
explored a variety of model from their classification such as
Support vector machines (SVM) and random forest. Under SVM,
they gained 7% improvement in performance from the baseline
model. A pattern of increase in the random forest classifier was
also observed.
Support vector machines (SVM) classifier have known to
outperformed other approaches in a number of text categorization
studies, but there has been some suggestion that they choose a
poor decision threshold when the numbers of positive and
negative examples are very different. SVM, which is highly
dimensional, decreasing the accuracy of the classifier [29]. Other
known classifiers are Naive Bayes and K-nearest neighbor. Naive
Bayes is often used in text classification because of its simplicity
and effectiveness. However, its performance is often degraded
because it does not model text well [30]. K-Nearest Neighbors
algorithm is a simple but effective method for text categorization
that uses all the training documents to predict labels of test
document but it has fatal defects like the complexity of its sample
similarity computing [31].
An article about fake news and post-truth explores the massive
increase of this problem and discussed how people react to
counter it. They examine impact of fake news in the society and
gave effective corrective measures. They suggested that this
problem needs a technological solution applied with
psychological principles, which they called it “technocognition”.
There are several techniques in correcting misinformation that are
helpful [32], [33], but they suggest to use basic tools.
Technocognition refers to the use of information architectures like
rating system, there should be a “Disinformation Charter” for
media to identify acceptable and unacceptable information, and
other general training in information literacy which includes what
to trust and not to believe [8].
3. METHODOLOGY
The study was conducted with three major stages: pre-processing,
processing, and post-processing stages.
3.1 Data Collection and Preparation
Data Gathering is consists of two parts: the gathering of the
editorial articles from the popular English news websites and the
acquisition of the annotated data set. The researcher looked for the
10 most popular English news websites according to Alexa
Internet, a California-based company that provides commercial
web traffic data and analytics. The editorial articles were gathered
from the top 10 websites in descending order and collected 20
each site’s most recent editorials and save it as hypertext mark-up
language format (HTML) readable by RapidMiner. The files are
saved in a folder named after the news website.
A worksheet has been prepared in order for the annotators’ tick
whether the polarity of an article is negative, neutral, or positive.
Annotation worksheets and copies of the 200 editorial articles for
training data and 20 articles for the testing data were distributed to
annotators with attached cover letter explaining the nature of the
study and the instructions for the worksheets. The annotators in
this study specialized in Journalism and have significant
experience in the said field. When the annotators were done,
results were tallied and the most occurring polarity for each article
are the one considered as its true polarity.
3.2 Data Processing
RapidMiner done the processing by “Transform” operator to
convert everything to lower case. The “Tokenize” operator splits
the text of a document into a sequence of tokens so as to be
readable. The “Filter Stop words (English)” operator filters
English stop words from the document by removing every token
which equals a stop word from the built-in stop word list. The
“Stem (Porter)” operator stems English words using the Porter
stemming algorithm applying an iterative, rule-based replacement
of word suffixes intending to reduce the length of the words until
a minimum length is reached.
The Porter Stemming involves a suffix stripping algorithm and to
present it a few definitions is needed. A \consonant\ in a word is a
letter other than A, E, I, O or U, and other than Y preceded by a
consonant. So in TOY the consonants are T and Y, and in
SYZYGY they are S, Z and G. If a letter is not a consonant it is a
\vowel\. A consonant will be denoted by “C”, a vowel by “V”. A
list ccc... of length greater than 0 will be denoted by C, and a list
vvv... of length greater than 0 will be denoted by V. Any word, or
part of a word, therefore has one of the four forms CVCV ... C,
CVCV ... V, VCVC ... C, VCVC ... V.
These may all be represented by the single for [C]VCVC ... [V]
where the square brackets denote arbitrary presence of their
contents. Using (VC) {m} to denote VC repeated m times, this
may again be written as [C](VC){m}[V]. m will be called the
\measure\ of any word or word part when represented in this form.
The case m = 0 covers the null word. Here are some examples:
m=0 TR, EE, TREE, Y, BY.
m=1 TROUBLE, OATS, TREES, IVY.
m=2 TROUBLES, PRIVATE, OATEN, ORRERY.
The \rules\ for removing a suffix will be given in the form
(condition) S1 -> S2. This means that if a word ends with the
suffix S1, and the stem before S1 satisfies the given condition, S1
is replaced by S2. The condition is usually given in terms of m,
e.g. (m > 1) EMENT ->.
If S1 is `EMENT' and S2 is null, this would map
REPLACEMENT to REPLAC, since REPLAC is a word part for
which m = 2. The `condition' part may also contain the following:
110
*S - the stem ends with S (and similarly for other letters)
*v* - the stem contains a vowel.
*d - the stem ends with a double consonant (e.g.-TT, -SS).
*o - the stem ends cvc, where the second c is not W, X or Y (e.g.-
WIL, -HOP)
Words’ suffixes are removed step by step. The first step of the
algorithm handles plurals, past participles, present participles, and
transforms a terminal “y” to an “I”. For example: “happy” to
“happi”. The second step deals with double suffixes to single ones.
For example: “generalization” is converted to “generalize”. The
third step removes other double suffixes not handled in the
previous step such as: “generalize” is changed into “general”. The
fourth step removes remaining suffixes such as: “general” is
transformed into “gener”. The fifth step treaties stems ending with
–e, and treats words ending in double consonant. For example:
“attribute” is recoded “attribut”.
The “Generate n-Grams” operator in RapidMiner creates term n-
Grams of tokens in a document or a series of consecutive tokens
of length n.
3.3 Model Building
The parameters for SVM dot Kernel type is C = 10.0. KNN has
been used in statistical estimation and pattern recognition already
in the beginning of 1970’s as a non-parametric technique. The
parameters for KNN is where k = 4 with weighted vote.
The 10-fold cross validation was used in the classification process
to train the model. With this method, the data set was divided into
10 subsets. Each time, one of the 10 subsets is used as the test set
and the other 9 subsets are the training set. Then the average error
across all 10 trials is computed.
RapidMiner was used to generate the process results log of the
threes classifiers in this study.
3.4 Model Testing
Once the news articles from a certain news website has been
processed, it can now be used for testing. The model is used for
the 3 classifiers that must be indicated in the “Retrieve”
parameters of Rapid Miner and pressed the Run button.
Testing was done by inserting additional negative bogus article
that was manually created by the researcher and put in the a
particular news folder, CNN for example, in order to verify if a
negative article can be read by the generated model of the study.
There were three (3) models generated, SVM, K-NN and Naïve
Bayes. Testing includes the model using the same dataset.
4. RESULTS AND ANALYSIS
In the training set, 200 is the total number of divided into three
categories: negative, neutral, and positive. In the dataset, 89
articles are negative, 67 articles are neutral, and 44 articles are
positive. The articles are subdivided by news source. In addition,
the testing set consists of 80 articles and is subdivided in the same
way. Table 1 displays the average percentage accuracy of the
models per news source data (training and testing) using
individual news sources for the training data and its corresponding
testing data.
Table 1. Average Percentage Accuracy per News Source Data 5. CONCLUSION
(Training and Testing). This research explored the creation of a model to analyze
Classifiers Average Accuracy (%) sentiment from popular news websites. The features of the
k-NN 40.00 editorial articles were extracted for classification by the chosen
Naïve Bayes 32.50 three classifiers; k-Nearest Neighbors, Naïve Bayes, and Support
SVM 38.77 Vector Machines. Based on the training data annotation on the
Table 2 displays the average percentage accuracy of the models selected 200 articles for this study, the emotion induced by
using all editorial articles for the training data and individual news editorial articles is, generally, negative. This can be attributed to
sources for the testing data. There was a significant change in the fact that, according to the annotators, negative articles let
accuracy for all the models, both k-NN and Naïve Bayes have alone an editorial one are more eye-grabbing in journalism. Thus,
significant increase. Only SVM experienced a decrease in raking in more readers. That is why this is the first step towards
accuracy. the detection of fake news.

Table 2. Average Percentage Accuracy for All Articles Data When doing batch processing per news source as testing set and
(Training) and Per News Source. the certain news sources’ training set SVM performed best with
an accuracy rate of 41.25%. When doing batch processing per
Classifiers Average Accuracy (%) news source as testing set and all the articles from the annotated
k-NN 42.50 training set, k-NN performed best with an accuracy rate of 42.5%.
Naïve Bayes 33.75 When using the entire dataset, k-NN performed best from the
SVM 38.75 three classifiers with an accuracy rate of 42.5%. In the
Table 3 displays the percentage accuracy of the models using all experiments, it was observed that with our imbalanced training
editorial articles for the training data and all the editorial articles data set, the polarity with the most training articles had the most
from the testing data. All the models have maintained their articles correctly predicted.
accuracy from the last table. It is recommended that further study would focus on injecting
Table 3. Summary of Results of the Second Iteration of Stress fake news and sees the polarity incurred upon the model to test
Testing. what polarity would still be dominant thus validating the result of
this study.
Classifiers Average Accuracy (%)
k-NN 42.50 6. REFERENCES
Naïve Bayes 33.75 [1] Pisarevskaya, D., Deception detection in news reports in the
SVM 38.75 Russian language: Lexics and discourse, In: 2017 EMNLP
Table 4 displays the number of correct predictions in the Workshop on Natural Language Processing meets
annotated test data using the models in this study. The same data Journalism, 74-79. Association for Computational
is presented in a different format in (Table 3). Linguistics (2017)
Table 4. Number of Correct Predictions of the Top [2] Hobson, L., Howard, C., Hapke, H.M., Natural language
Performing Models. processing in action, Manning Publications, New York,
USA (2017)
No. of Correct Predictions
Classifiers [3] Associated Press.
(Out of 2 articles per source)
Negative Neutral Positive http//www.theverge.com/2015/1/29/7939067/ap-journalism-
k-NN 32 7 4 automation-robots-financial-reporting
Naïve Bayes 12 9 6 [4] Volkova, S., Shaffer K., Jang, J.Y., Hodas, N., Separating
SVM 9 17 5 facts from fiction: Linguistic models to classify suspicious
Looking closely on the prediction results of the models, it can be and trusted news posts on Twitter, In: 55th Annual Meeting
derived that predicting the “Negative” sentiment label was the of Association for Computational Linguistics (2017)
easiest for the models as most correct predictions were under the [5] Shao, C., Ciampaglia, G. L., Varol, O., Flammini, A.,
“Negative” sentiment annotation, while the next sentiment easiest Menczer, F., The spread of fake news by social bots, arXiv
to predict is “Neutral”. Lastly, the sentiment that’s most hard to preprint arXiv: 1707.07592 (2017)
predict based on the 80 articles of the testing set is “Positive”.
This can be attributed to the fact that the 200 articles of the [6] Williamson, P., Take the time and effort to correct
annotated training set only contained 44 positive articles misinformation, Springer Nature - MacMillan Publishers
compared to 67 neutral articles and 89 negative articles. Thus, Limited, 540, 171 (2016)
affecting data training when being compared to other sentiment [7] Chen, Y., Conroy, N. J., Rubin, V. L., News in an online
labels. Finally, the researcher were able to test the over-all world: The need for an “Automatic crap detector”, In: The
performance of k-NN, Naïve Bayes, and SVM. Table 5 displays Proceedings of the Association for Information Science and
the accuracy rate and kappa scores of each classifier. Technology Annual Meeting (ASIST2015), Nov. 6-10, St.
Louis. (2015)
Table 5. Calculating Kappa.
[8] Lewandowsky, S., Ecker, U., Cook, J., Beyond
Classifiers Accuracy Rate (%) Kappa misinformation: Understanding and coping with post-truth
k-NN 40.00 0.186 era, J of Applied Research in Memory and Cognition. 6, 4,
Naïve Bayes 32.50 -0.009 418-424 (2017)
SVM 38.77 0.123

111
[9] Corner, J., Fake news, post-truth and media political change.
J in Media, Culture and Society, 39, 7, 1100-1107.
University of Leeds, UK (2017)
[10] Digital 2017: A Global Overview.
https://wearesocial.com/special-reports/digital-in-2017-
global-overview
[11] Gottfired, J., Shearer, E., Pew Research Center: Americans’
online news use is closing in on TV news use. (2017)
http://www.pewresearch.org/fact-
tank/2017/09/07/americans-online-news-use-vs-tv-news-
use/
[12] Elliot, D., Culver, C., Defining and analyzing journalistic
deception, In: Journal of Mass Media Ethics, 7, 2, 69-84.
(1992)
[13] Patterson, J., Gibson, A., Deep learning: A practitioner’s
approach. O’Reilly Media Inc. California, USA (2017)
[14] Medhat, W., Hassanb A., Korashyb, H., Sentiment analysis
algorithms and applications: A survey, Engineering Journal,
5, 1093-1113. (2014)
[15] Trumper, D., Castillo, C., Lalmas, M., Social media news
communities: Gatekeeping, coverage, and statement bias. In:
Proceedings of the 22nd ACM International Conference on
Conference on Information and Knowledge Management,
1679-1684, (2013)
[16] Tan, SS., Na, JC., Mining semantic patterns for sentiment
analysis of product reviews. In: Kamps J., Tsakonas G.,
Manolopoulos Y., Iliadis L., Karydis I. (eds) Research and
advanced technology for digital libraries, Lecture Notes in
Computer Science, 10450. Springer, Cham (2017)
[17] Alexa - Actionable Analytics for the Web.
http://www.alexa.com/
[18] Manek, A., Shenoy, P., Mohan, M., Aspect term extraction
for sentiment analysis in large movie reviews using Gini
Index feature selection method and SVM classifier, In:
World Wide Web 20, 135. (2017)
[19] Allcott H, Gentzkow M., Social media and fake news in the
2016 election, 31, 2, 211-236. (2017)
[20] Conroy, N., Rubin, V., Chen, Y., Automatic deception
detection: Methods for finding fake news, In: Proceedings
of the Association for Information Science and Technology,
52, 1, (2015)
[21] Jin, Z., Cao, J., Zhang, Y., Luo, J., News verification by
exploiting conflicting social viewpoints in microblogs, In:
Proceedings of the Thirtieth AAAI Conference on Artificial
Intelligence (AAAI-16). (2016)
112
[22] Couldry, N., Turow, J., Advertising, big data, and the
clearance of the public realm: Marketers’ new approaches to
the content subsidy, International Journal of
Communication, 8, 1710-1726. (2014).
[23] Kogan, S., Moskowitz, T., Niessner, M., Fake News in
Financial Markets. (2017)
[24] Chopra, S., Jain, S., Sholar, J., Towards automatic
identification of fake news: Headline-article stance
detection with LSTM attention models, In: Stanford CS224d
Deep Learning for NLP Final Project (2017).
https://web.stanford.edu/class/cs224n/reports/2761028.pdf.
[25] Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H., Fake news
detection on social media: A data mining perspective. In:
ACM SIGKDD Explorations Newsletter, 1, 19, 1, 22-36.
2017
[26] Tversky, A., Kahneman, D., Advances in prospect theory:
Cumulative representation of uncertainty, Journal of Risk
and Uncertainty, 5, 4, 297, 323, (1992)
[27] Kapferer, J., Rumors: Uses, Interpretation and Necessity,
Routledge, 2017.
[28] Bessi, A., Ferrara, E., Social bots distort the 2016 US
presidential election online discussion. First Monday, 21, 11.
(2016)
[29] Lenc, L., Král, P., Deep neural networks for czech multi-
label document classification, In: Proceedings of the 17th
International Conference on. Intelligent Text Processing
and Computational Linguistics. Turkey 1-12. (2016)
[30] Rashkin, H., Choi, E., Jang J., Volkova, S., Choi, Y., Truth
of varying shades: Analyzing language in fake news and
political fact-checking, In: Proceedings of the 2017
Conference on Empirical Methods in Natural Language
Processing, 2931-2937. (2017)
[31] Elmurngi, E., Gherbi, A., An empirical study on detecting
fake reviews using machine learning techniques, In:
Proceedings of the 2017 Seventh International Conference
on Innovative Computing Technology (INTECH). (2017)
[32] Cook, J., Lewandowsky, S., Ecker, U., Neutralizing
misinformation through inoculation: Exposing misleading
argumentation techniques reduces their influence, PLoS
ONE, 12, 5. (2017)
[33] Ecker, U., Hogan, J., Lewandowsky, S., Reminders and
repetition of misinformation: Helping or hindering its
retraction? In: Journal of Applied Research in Memory and
Cognition, 6, 2, 185-92. (2017).