Online Newspaper Analysis System

Hai Pham Van, Han Nguyen Nam, Huy Nguyen Quang, Kien Do Luong, Thanh
Nguyen Van, Tung Nguyen Duy,

Instructor Professor Pham Van Hai

Hanoi University of Science and Technology, Vietnam

ABSTRACT
Online newspapers are experiencing exponential growth in the digital age which
leads to the proliferation of fake news and information bias. Addressing these
challenges requires robust solutions leveraging advancements in Natural Language
Processing (NLP). In this paper, we present a comprehensive pipeline for detecting
fake news and performing Named Entity Recognition (NER) in online articles.
Our methodology integrates cutting-edge NLP techniques, including Bidirectional
Encoder Representations from Transformers (BERT), Transformer architecture,
BiLSTM, and Conditional Random Fields (CRF). We utilize BERT, a pre-trained
model, for understanding article semantics, enhancing accuracy in downstream KEYWORDS
tasks. For fake news detection, we employ BERT in a classification architecture, Fake news detection,
leveraging both headline and content information to significantly improve prediction sentiment analysis,
accuracy. Bidirectional Long Short-Term Memory (BiLSTM) is utilized for Named BERT, LSTM
Entity Recognition due to its bidirectional sequence modeling capabilities, identifying Transformer, NLP
potential entities efficiently. Additionally, we enhance NER predictions by
incorporating a CRF model for structured prediction tasks. We also use Scrapy as
a data crawling and preprocessing tool to extract features from online news articles,
enriching our dataset and enhancing model performance in prediction and labeling
tasks. The outcomes of our architecture are tested with contemporary datasets namely:
the Phobert Named Entity Reconigtion (PhoNER) datasets of Covid-19 and Kaggle
datasets relating to fake news.

I. Introduction pieces with non-essential content, sensational headlines, or

We are living in an era of industrialization and
modernization. Alongside scientific and technological
advancements, communication systems and media
platforms are rapidly developing, providing easier access to
information for people. Information and news are truly
booming with a massive volume.
Fig. 1. Amount of data being generated each minute (2019)
As a natural consequence, news websites and platforms
always deliver a huge amount of articles to readers. To
attract readers, many news sites provide numerous news
* Corresponding authors:
content that diverges from the article’s actual substance.
This inadvertently leads to readers being misled or
receiving inaccurate information. The recent advancement
of Natural Languge Processing modelsl, data mining and
processing methodologies, datahave also provided
important foundation and enable researchers to analyze
and extract information in large amount of text at ease
This paper aims to delve into the realm of online news
analysis, exploring various methodologies, techniques, and
challenges associated with analyzing news content
disseminated through digital platforms. By examining the
intersection of technology, media, and society, we seek to
uncover insights into how online news shapes public
opinion, influences decision-making processes, and reflects
societal trends and biases.
Through a comprehensive review of existing literature
and case studies, this paper will provide a
multidimensional understanding of the complexities
involved in analyzing online news. From sentiment analysis
and topic modeling to the detection of misinformation and
understanding algorithmic biases, we will explore the
diverse array of analytical approaches employed to dissect
digital news content.
Furthermore, we will discuss the ethical considerations
and implications of online news analysis, particularly
concerning privacy, data security, and the spread of
misinformation. By critically assessing the strengths and
limitations of current methodologies, we aim to offer

1
insights into future directions for research and practical
applications in the field of online news analysis.
A. General purpose and scope
The primary objective of this paper is to undertake a
multifaceted analysis of news articles, employing statistical
methodologies to discern their tonal attributes. Through
this analysis, the paper aims to provide readers with
recommendations for reputable news sources, thereby
facilitating informed consumption of information.
Additionally, the research endeavors to develop robust
mechanisms for the detection of both genuine and
counterfeit news items, while also delving into the
identification of their respective sources. Furthermore, the
paper seeks to devise sophisticated keyword sets utilizing
diverse combination rules, thereby enabling precise and
efficient retrieval of articles that adhere to predetermined
criteria. This research is delimited to the development of a
web-based system designed to execute the aforementioned
tasks with efficacy and accuracy. Emphasis will be placed
on the creation of a user-friendly interface that grants users
access to a curated selection of news articles, vetted for
authenticity and credibility. The system, while not
excessively intricate, will be endowed with a
comprehensive suite of functionalities essential for the
fulfillment of its objectives. This encompasses, but is not
limited to, statistical algorithms for tone analysis, machine
learning models for fake news detection, and sophisticated
search mechanisms based on predefined keyword sets. The
resultant system is envisioned as a reliable and
indispensable tool for individuals seeking to navigate the
vast landscape of digital news media with discernment and
confidence.
B. Data mining
The lack of training data for models is always an important
issue to address for any NLP projects. In our case , we
implemented Scrapy, a data crawling and processing tools
which can automate the extraction of features of online
news articles from numerous online news forums and
newspapers
By implementing Scrapy, the crawled data consists of
articles collected directly from over 100 news websites daily
and aggregated for statistical analysis and keyword
extraction. The data sources include articles from online
news websites on the internet such as:dantri.com.vn,
vnexpress.vn, soha.vn, kenh14.vn and other news websites.
The basic data fields that need to be collected are:
Field name Meaning
domain Domain name of the site
publishedTime Date on which the article
was posted
title Title of the article
content Contents of the news
authorDisplayName Name of the author
type The type of the news
... ...
Table 1: Data fields to be collected
C. General Architecture
In terms of architecture, our team have built an integrated
system consisting of 2 main modules:
a) Data collection and processing system: Data will be
collected from news websites on the internet using Scrapy,
2
then temporarily stored by Kafka. Logstash will retrieve
data from Kafka, process it through several steps, and
classify the data before adding it to Elasticsearch.
b) Website system: The website allows users to analyze and
track issues of interest from the data in Elasticsearch. The
backend utilizes Spring Boot to connect with both SQL and
Elasticsearch to retrieve data. The frontend is built using
ReactJS.
Our result is a complete web application that enables
users to create keyword sets for analyzing and tracking
data from news websites, as well as review articles collected
by the system.
II. Related Works
In order to perform statistical analysis, as well as to
detect fake news by topic and category, the first main task is
to determine what entities the news articles are targeting.
This task is essentially named entity recognition.
A. Named Entity Recognition Task (NER)
1. Overview of NER
In the current era of information explosion, useful
information extracting (IE) from data has become a
prerequisite activity in all existing fields.
According to the paper "A survey on Named Entity
Recognition — datasets, tools, and methodologies" [1],
Name Entity Recognition or NER "is the process of
identifying numerous segments of information referenced
in a text and then classifying them into pre-established
categories. Entities like “person”, “organization”, “region”,
and many more might be considered categories. Named
Entity Recognition is a broad category of NLP issues
known as sequence tagging.
NER is a subtask of IE, tasked with automatically
extracting named entities (such as person names, locations,
organizations, dates/times, quantities, etc.) by identifying
entities in a given text (Recognition) and determining their
types (Classification).
Through NER, we can build a knowledge base. It can
organize and arrange information in a way that is useful for
humans and also provide useful inference for subsequent
NLP algorithms.
NER can be implemented with the help of tool sets such
as:
a) SpaCy: [2] This Python package is designed for
high-level natural language processing (NLP) tasks and is
open source. It enables the creation of programs that can
effectively process and comprehend large volumes of text,
making it well-suited for deployment in production
environments. With support for over 72 languages and 80
pre-trained pipelines across nearly 24 languages, it
facilitates the development of systems for information
extraction and natural language understanding.
Additionally, it promotes multitask learning using
advanced models like BERT, offering state-of-the-art
processing speed. Furthermore, it allows integration of
custom models developed in TensorFlow, PyTorch, and
other libraries. For named entity recognition, it offers a
dedicated pipeline component called the entity recognizer.
The NER module [3] is built on a model that makes use of
residual convolutional neural networks and Bloom
embeddings. There are models for the English language
that were previously trained on the OntoNotes Release 5.0
package, including tagged text taken from news articles,
blogs, and phone conversations.
b) NLTK, recognized as a prominent Python platform for
developing applications that leverage human language
data, has gained prominence in the realm of natural
language processing (NLP). As cited by Bird et al.
(2009) [4], NLTK serves as a comprehensive toolkit
encompassing various text processing functionalities,
including parsing, categorization, tagging, stemming,
tokenization, and semantic reasoning. Moreover, it offers
named entity recognition (NER) capabilities alongside a
vibrant community forum, fostering active discussions and
knowledge exchange. NLTK is commonly utilized in both
research endeavors and educational settings, serving as a
valuable resource for training students in NLP techniques.
Notably, its NER module employs a Maximum Entropy
Classifier trained on the Automatic Content Extraction
corpus.
c) Pytorch, which was created by Facebook [5], and it was
another open-source deep-learning package designed only
for Python use. It serves as a premier platform for both
industry and education purposes. Pytorch can be mostly
used for image recognition and language processing by
making machine learning more scalable several AI models
can be built quickly and efficiently.
Some of the other tools offered for industry-based projects
are LingPipe, AllenNLP [6], IBM Watson [7], Intellexer [1],
ParallelDots [8], and Dandelion API [9]. Some applications
of NER include:
- Indexing documents and supporting search tools :
integrating NER into document indexing and search
tools enhances the efficiency, relevance, and
sophistication of information retrieval systems by
leveraging the semantic information encoded in text
documents [10] [11].
- Customer support chatbots: By integrating NER,
chatbots could potentially ask more relevant and
contextually appropriate questions during
conversations with users. This could lead to more
engaging and satisfying interactions [12] and
integrating NER into chatbots could enable them to
better understand and respond to user queries
involving named entities [13].
- Opinion mining: Named Entity Recognition (NER)
can aid in opinion mining by identifying and
extracting entities mentioned in opinions and also
linking them to find certain meaningful
relationships, which can provide valuable context
and insights for sentiment analysis [14].
2. Basic steps of NER
Fig. 2. Basic steps of NER
Sentence segmentation [15] [16]: is the process of
dividing a string of written language into its component
3
sentences. For example, sentences need to be separated
whenever punctuation or period marks are found. The
purpose of sentence segmentation is to delineate sentence
boundaries.
Tokenization [17] [18]: is the process of dividing a string
of written language into its constituent words.
Part-of-speech tagging [15] [19]: is the process of
assigning a word in a text to its corresponding part of
speech tag based on the context and its definition. It
describes the characteristic structure of the vocabulary
terms in a sentence or text.
Entity recognition [20]: is the process of identifying key
elements from the text, and then classifying them into
predefined categories. This is a crucial process of NER as it
fulfills the purpose of NER.
B. Overview of BiLSTM Architecture
1. LTSM
LSTM, a famous variant of RNN, was proposed by
Hochreiter and Schmidhuber in 1997 [21] as a solution to
the aforementioned problem. The LSTM model consists of
three gate networks, "input gate", "forget gate", and "output
gate", which perform better than RNN.
Fig. 3. Basic architecture of LSTM
LSTM operates in a three-step operation:
Step 1: The first step in LSTM is to decide which
information to discard from the cell at that specific time.
This is determined with the help of a sigmoid function. It
considers the previous state pht´1 q and the current input
pxt q and computes the function.
Step 2: There are two functions in the second step. The first
is the sigmoid function, and the second is the tanh
function. The sigmoid function decides which values to let
through (0 or 1). The tanh function assigns weights to the
passed values, determining their importance from -1 to 1.
Step 3: The third step is to decide what the final output will
be. First, a sigmoid layer needs to be run to determine
which part of the cell state goes to the output. Then, the cell
state needs to be passed through the tanh function to push
the values from -1 to 1 and multiply it with the output of
the sigmoid gate.
On the other hand, accurately identifying proper nouns
in a piece of text depends not only on the preceding
information of the word being considered but also on the
information following it. However, a traditional LSTM
architecture with a single layer can only predict the label of
the current word based on the information obtained from
the preceding words.
2. BiLSTM
BiLSTM [22] was created to address the aforementioned
weakness. A BiLSTM architecture typically contains 2
individual LSTM networks used simultaneously and
independently to model the input sequence in 2 directions:
from left to right (forward LSTM) and from right to left model. We apply it with the negative log-linear function of
(backward LSTM). the function above.
# +
m
ź
LpX, λ, yq “ ´ log k k
P py |X , λq (4)
k“1

m
ÿ
»
n
ÿ
fi

— 1
— ÿ ffi
log — exp λj fj pX m , i, yi´1
k
, yik qffi
ffi
“´
– ZpXm q j
fl
k“1 i“1
(5)
Fig. 4. BiLSTM = forward LSTM + backward LSTM Calculate the partial derivative with respect to lambda to
find the minimum value of the log function because finding
the argmin value will achieve the maximum value for the
However, the BiLSTM architecture still does not fully entire negative log-linear function.
exploit the contextual semantics of the text.
m
BLpX, y, λq 1 ÿ
“´ Fj py k , X k q
C. Overview of CRF Model Bλ m k“1
m
CRF [23] is a probabilistic model for structured ÿ
` ´1P py k |X k , λqFj py, X k q (6)
prediction tasks and has been applied successfully in many
k“1
fields such as computer vision, natural language
processing, bioinformatics, etc. In the CRF model, the where:
nodes containing input data and the nodes containing
n
output data are directly connected to each other, ÿ
Fj py, Xq “ Fj pX, i, yi´1 , yi q (7)
contrasting with the architecture of LSTM or BiLSTM i“1
where inputs and outputs are indirectly connected through
is the observed mean feature value,
memory cells. CRF can be used for named entity labeling
with input features being manually extracted features of a m
ÿ
word such as: capitalization of the first letter, all letters are P py k |X k , λqFj py, X k q (8)
capitalized, preceding word is capitalized, word under k“1

consideration, preceding word is under consideration, . . . is the expected feature value according to the model.
In the CRF algorithm, we aim to maximize the conditional We use partial derivatives as a step in the gradient
probability P py|xq - the probability of having output vector descent method. Gradient descent is an iterative
y, given input provided by random variable x; and we take optimization algorithm to update parameter values until
the sequence with the highest probability. convergence of lambda is found. The final gradient descent
update equation for the CRF model is:
ŷ “ argmaxy P py|xq (1)
« ff
In CRF, the input data is sequential, and we rely on the m
ÿ k k
m
ÿ k k k
previous context to make predictions about a data point. We λ“λ`α Fj py , X q ` P py |X , λqFj py, X q
use feature functions that have multiple input values: k“1 k“1
(9)
- The set of input vectors X. The CRF model used for entity labeling addresses the
- The position i of the data point we are predicting. drawback of label bias due to the independence of labels
- The label of the data point i ´ 1 in X. from each other in the Hidden Markov Model. CRF first
- The label of the data point i in X. identifies the necessary feature functions, initializes lambda
The feature function fj is defined as follows: weights for random values, and then applies the gradient
fj px, i, yi , yi´1 q. descent method iteratively until the parameter values
Each feature function is based on the label of the previous converge.
and current word, takes on values of 0 or 1. We assign a set However, a problem with linear chain CRF is that they
of weights (lambda values) to each function, the algorithm only capture the dependencies between labels in a forward
will learn: direction.

n
ÿ D. Overview of BERT Mechanism
1 ÿ
P py|X, λq “ exp λi fi pX, i, yi´1 , yi q (2) 1. Encoder and Decoder Process
ZpXq j
i“1
Computers cannot learn directly from raw data such as
where: images, text files, audio files, or video clips. Therefore, they
require a process of encoding information into numerical
n
ÿ form and decoding the numerical form into output results.
ZpXq “ řyPY λi fi pX, i, yi´1 , yi q (3) These are the two processes called encoder and decoder:
i“1
Encoder: It is the phase of transforming input into
is the normalized function. learning features capable of learning tasks. For Neural
To estimate the parameters lambda, we use Maximum Network models, the Encoder consists of hidden layers. For
Likelihood Estimation (MLE). It is a method in statistics for CNN models, the Encoder comprises a sequence of
estimating the parameter values of a CRF probability Convolutional + Maxpooling layers. For RNN models, the

4
Encoder process mainly involves Embedding layers and
Recurrent Neural Network layers.
Decoder: The output of the encoder is the input to the
decoders. This phase aims to determine the probability
distribution from the learning features at the encoder,
thereby identifying the output labels. The result can be a
single label for classification models or a sequence of labels
in chronological order for seq2seq model.
Fig. 5. seq2seq model with encoding and decoding, for
machine translation task
2. Transformer architecture
The Transformer architecture was first introduced in the
paper titled "Attention is All You Need" by Vaswani et
al [24]. This paper was presented at the Conference on
Neural Information Processing Systems (NeurIPS) in 2017
This architecture consists of two parts, an encoder on the
left and a decoder on the right.
Encoder: is a stacked combination of 6 defined layers.
Each layer consists of two sub-layers within it. The first
sub-layer is multi-head self-attention. The second sub-layer
is simply a fully connected feed-forward layer. A note is
that we will use a residual connection at each sub-layer
immediately after normalization layer. This architecture
has a similar idea to the resnet network in CNN. The output
of each sub-layer is LayerNorm (x + SubLayer(x)) with a
dimension of 512.
Decoder: The decoder is also a stacked combination of 6
layers. The architecture is similar to the sub-layers in the
Encoder except for the addition of 1 sub-layer representing
the attention distribution at the first position. This layer is
no different from the multi-head self-attention layer except
that it is adjusted to avoid including future words in the
attention. At the i-th step of the decoder, we only know the
words at positions smaller than i, so adjusting ensures that
attention is only applied to words smaller than position i.
The residual mechanism is also applied similarly to the
Encoder.
Note that there is always an additional step of adding
Positional Encoding to the inputs of the encoder and
decoder to incorporate the time factor into the model,
thereby increasing accuracy. This involves adding the
position encoding vector of the word in the sentence to the
5
word representation vector. We can encode it as a [0, 1]
position vector or use sine and cosine functions [24].
3. Attention mechanism
3.1 Scale dot product attention
This is a self-attention mechanism [24] where each word
can adjust its weight for other words in the sentence so that
the weight is larger for words closer to it and gradually
decreases for words further away.
After the word embedding step (passing through the
embedding layer), we have the input of the encoder and
decoder as a matrix X of size mxn, where m and n are the
length of the sentence and the dimension of a word
embedding vector, respectively.
The three matrices Wq , Wk , Wv are the parameters that
the model needs to train. After multiplying these matrices
with the input matrix X, we obtain the matrices Q, K, V
(corresponding to Query, Key, and Value matrices in the
figure). The Query and Key matrices are used to compute
the score distribution for word pairs. The Value matrix
utilizes the score distribution to calculate the probability
distribution output vector.
The input for calculating attention will include the matrix
Q (each of its rows is a query vector representing the input
words) and the matrix K (similarly to the matrix Q, each
row is a key vector representing the input words). These
two matrices Q, K are used to compute attention where
words in the sentence return attention to a specific word in
the sentence. The attention vector will be calculated based
on the weighted average of the value vectors in the matrix
V with attention weights (computed from Q, K).
The function of Attention is:
QK T
ˆ ˙
AttentionpQ, K, V q “ sof tmax ? V (10)
dk
Dividing by dk , which is the dimensionality of the key
vector, aims to prevent overflow if the exponent is too large.
3.2 Multi-head attention
So after the process of scale dot product, we will obtain
an attention matrix. The parameters that the model needs
to adjust are the matrices Wq , Wk , Wv . Each such process is
called a head of attention. When repeating this process
multiple times (in the paper, it’s 3 heads), we will obtain
the Multi-head Attention process as the transformation
below. Each branch of the input is a head of attention. In
this branch, we perform scaled dot production, and the
output is the attention matrices.
After obtaining the three attention matrices at the output,
we concatenate them along the columns to obtain the
aggregated multi-head matrix with the same height as the
input matrix, according to the formula:
M ultiHeadpQ, K, V q “ concatenatephead1 , . . . , headh qW0
where, headi “ AttentionpQi , Ki , Vi q
Note that, to return an output with the same size as the
input matrix, we just need to multiply it by the W0 matrix
with the same width as the input matrix.
4. Encoder and Decoder process in BERT mechanism
Thus, concluding the process 3, we have completed the
first sub-layer of the Transformer, which is the multi-head
Attention layer. In the second sub-layer, we will pass
through fully connected connections and return the output
with the same size as the input. The purpose is to be able to
iterate these blocks N times.
After the backward propagation transformations, we
often lose information about the position of words.
Therefore, it is necessary to apply a residual connection to
update previous information into the output. This method
is similar to ResNet in CNN. To ensure stable training, we
will also apply another normalization layer immediately
after the addition operation.
If we repeat the above layer block 6 times and denote them
as one encoder. We can simplify the graph of the encoder
process as shown below.
The decoder process is completely similar to the encoder
except for:
The decoder process will generate tokens sequentially from
left to right at each time step.
At each block layer of the decoder, we need to add the final
matrix from the encoder as an input to the multi-head
attention. Fig. 7. BERT architecture for MLM task
Add a layer of Masked Multi-head Attention sub-layer at
the beginning of each block layer. This layer is no different
from the Multi-head Attention except that it does not In which the task of masked language model has an
consider the attention of future words. output size equal to the maximum length of the sentence,
and the task of predicting whether the next sentence is
present or not has an output size of 2. Then, the loss value
5. Overview of BERT for updating the parameters is the total loss of both tasks.
BERT [25] is trained simultaneously on two different
tasks: Masked Language Model and Next Sentence
Prediction [25].
The BERTbase model utilizes 12 Transformer encoder
blocks, while the BERTlarge model uses 24 Transformer
encoder blocks (See Figure 6).
E. Evaluation metrics

The table below is called the Confusion Matrix and

Fig. 6. Encoder blocks in two BERT architectures
The BERT architecture in the Masked Language Model
task uses Transformer encoder blocks, which learn to
predict the missing words at the [MASK] positions in the
input, constituting 15% of the words in the sentence (See
Figure 7).
In the next sentence prediction task, BERT uses a
Transformer model with two sentences as input and
predicts the probability True/False of whether the two
sentences are consecutive or not.
During training, 50% of correct sentence pairs are mixed
with 50% of randomly chosen sentence pairs to help BERT
improve the accuracy of predicting the next sentence.
The BERT model uses the Cross Entropy loss function:
contains the definitions of True Positive (TP), False Positive
(FP), False Negative (FN), True Negative (TN). Confusion
matrix is also discussed in a paper called "Evaluation: From
precision, recall and F-measure to ROC, informedness,
markedness and correlation" by Powers (2011) [26]. This
provides an in-depth exploration of various evaluation
metrics used in classification tasks. The confusion matrix
provides a tabular representation of the actual versus
predicted class labels produced by a classifier. In the
context of the paper, the confusion matrix serves as the
foundation for calculating various evaluation metrics
discussed, including precision, recall, F-measure, ROC
curve, informedness, markedness, and correlation
coefficient. By organizing the true positive, false positive,
true negative, and false negative predictions into a matrix,
the confusion matrix allows for the computation of these
metrics, which offer different perspectives on the
performance of a classifier.

output size
Fig. 8. Confusion matrix
ÿ
Loss “ ´ yi ¨ log ŷi (11)
i“1

6
1. Precision
Precision is defined as the ratio of the number of True
Positives to the total number of predicted Positive instances
(i.e. take the number of correctly predicted labels and
divides it by the total number of predictions made for that
label).
A high precision means a high accuracy of correctly
labeled instances.
TP
P recision “ (12)
TP ` FP
2. Recall
Recall is defined as the ratio of the number of True
Positives to the total number of actual Positive instances
(i.e. take the number of correctly predicted labels and
divides it by the total number of labels that are actually
correct).
A high recall means a high proportion of True Positives,
indicating a low rate of entities not labeled as such.
website provides interaction with the for the following
services: Search, Generating statistics, News following and
other APIs for Administrators, User personal information
and Account.
In specific, APIs relating to user personal information will
help manage the users history in accessing and following
news. APIs for Admin and Account provide account
management, login/logout actions, authentication and
permission. Services and APIs will communicate through
mutual database. In addition a news database will be
constructed through a news integrated module with a
procedure containing the steps: retrieval, integration, and
data cleaning.
In terms of deployment the system will be deployed on
personal server and locally hosted. The SQL database is
deployed n PostgreSQL and other services are deployed on
Docker.
Actors accessing the components of the system from the
frontend to APIs and database are always required to
authenticate and given the right permission

TP 3. Building the system

Recall “ (13)
TP ` FN The system’s use cases: See Figure 10 and Figure 11.

3. F1 score
In practice, adjusting the model to increase Recall beyond
a certain point may lead to a decrease in Precision, and vice
versa, attempting to increase Precision can reduce Recall.
In such cases, F1 score serves as a useful measure of
prediction success when dealing with imbalanced classes,
calculated based on Precision and Recall.
2 1 1
“ `
F1 P recision Recall
P recision ˚ Recall
ðñ F1 “ 2 ˆ (14)
P recision ` Recall

III. Proposed Models

A. Integrated System Model proposal

1. Overview of the system
Our team has identified 3 main actors and our system
will consist of subsystems which will provide support for
the functionality of the actors such as: Query and search for
news, news statistics, management functionality.
In addition, our system will also be integrated with several
AI subsystems regarding object detection, keyword
detection as well as fake news detection.
The actors will interact with the system through the web
interface.
We would also provide database relating to news, Users
database and database for training the model.

2. Architectural System Design

Our system propose three main modules which are: (See
Figure 9) Data Mining and Preprocessing Module, AI
Module: this provides services for detecting fake news and
keyword extraction, Website Module which provides other
services.
As for the description of our architectural system design,
the actors include Customers, Users, Administrators
interacting with the system through the web browsers,
while with integrated platforms, the frontend server of the

7
Fig. 9. Architectural System Design

Fig. 10. Use cases for the Users Actor

8
 Fig. 11. Use cases for the Administrator Actor
B. Integrated architecture details
1. Data mining and processing module
1.1 Data flow
Data in the form of articles after being crawled will be
pushed into Kafka After that, these articles will be
classified into two states: Positive and Negative.
Within this project, we perform sentiment analysis based
on the set of laws. Data is processed through logstash with
the input as the data modified and processed through
Kafka’s filters. The output data will be pushed through
elastic search.
The data is stored in elastic search and indexed by the date
to make it convenient for data analysis since we regard date
here as the base time unit and also in order to reduce the
amount of data stored in one particular index.
1.2 Data mining and Processing Component
1.2.1 Data Crawling
In order to crawl data [27], we use the framework Scrapy
. Scrapy is a open source framework used to extract data
from websites on the internet. It can be used for different
purposes , from extracting data to monitoring and
automation testing [28]. First of all, Scrapy Engine is
responsible for controlling all the data flow from system
components and activate each events when particular
actions occur. The second component is a scheduler which
is similar to a queue, scheduler arrange the order of the
URL that are downloaded. And finally a downloader
performs downloading the websites and provide them as
data to the engine. Another important concept are the
9
Spiders: Spiders is a class written by the users, they are
responsible for necessary data parsing and create new
URLs to reload to scheduler through the engine. This is a
fundumental part of the Item Pipeline in which all the data
parsed by Spiders will be sent here. Item Pipeline has the
task of processing them and stored in the database.
Middlewares
This pipeline also includes middlewares which are the
components in between the Engine and other components.
They have the purpose of helping the user customize,
expand the processing capability for each components. E.g:
After downloading an URL, if you want to provide tracking
and send information right at that moment , you can
expand and modify the configuration.In this case Spider
middlewares are used: They are the component in between
the Scrapy Engine and Downloader, they handle the input
response from Spiders and the output (new items and
URLs). All three important of our pipeline all have
important middlewares. For example, Downloader
Middlewares sit between the Engine and Downloader,
effectively handling the pushed requests from the Engine
and the response generated from Downloader. While
Scheduler middlewares sit in between the Engine and
Scheduler to handle requests from both sides.
Data flow
As for the data flow, when beginning to crawl from a
website, the Engine will identify the domain name and find
the location of a Spider and request the Spider the first
URLs to crawl. After that, the Engine receive a list of the
first URLs from Spider and send this to the Scheduler to
make arrangement. Next, the Engine requests the next list
of URLs from Scheduler and send them to the Downloader
(requests). Downloader receives the request and perform
downloading the pages. After it has finished downloading,
it will create a response and send back to the Engine.
Responses from Downloader will be pushed from the the
Engine to the Spiders to analyze. At Spiders, when received
a response, they will begin to extract the information from
the reponse(title, content, author, data published,...) and
handle the urls that have potential to crawl and push back
to the Engine (requests). At this point, the Engine will
receive the results from Spiders regarding two main tasks:
pushing the parsed data to the Item Pipeline to process and
saved into the database, pushing new URLS to the
Scheduler and return to step 3.
1.2.2 Kafka
Apache Kafka [27] is a a distributed messaging system.
Kafka is developed and maintained by Apache, which is why
Kafka (message broker) has the name of Apache Kafka.
Similar to other message brokers, it is developed following
the model of public/subscibe. The party who public the data
is called producer and the party receiving the data divided
by topics is called consumer.
1.2.3 Logstash
Logstash [29] is an open source application, and belongs
to the ELK Stack ecosystem, with a very important aim
which consists of three stages in the log pipeline
corresponding to three modules:
• INPUT: Receive, collect raw event log data from various
sources such as files, Redis, RabbitMQ, Beats, syslog, ...
• FILTER: After receiving the data, perform event log data
operations (such as add, delete, replace, ... log content)
according to the administrator’s configuration to rebuild
the log event data structure as desired.
• OUTPUT: Finally, forward the event log data to other
services such as Elasticsearch for log reception, storage,
or display.
Fig. 12. HT LogStash Flow
At the INPUT stage, Logstash will be configured to select
the form of receiving log events or fetching log data
remotely as needed. After obtaining the log events, the
INPUT step will write the event data to a centralized queue
in RAM or on disk.
Each pipeline worker thread will continue to retrieve a
batch of events from this queue to process FILTER to help
restructure the log data to be sent at the OUTPUT stage.
The number of events processed per batch and the number
of pipeline worker threads can be configured for optimal
tuning.
By default, Logstash uses an in-memory queue in RAM
between stages (input -> filter and filter -> output) as a
buffer to store event data before processing. If the Logstash
service program is stopped for some reason in the middle,
then the event data in the buffer will be lost.
1.2.4 Elasticsearch
Elasticsearch [30] is a distributed search and analytics
tool built on Apache Lucene. Elasticsearch was first
introduced in the paper titled "ElasticSearch: A Distributed,
RESTful Search Engine" by Shay Banon [31]. This paper
was presented at the 2010 Berlin Buzzwords Conference. It
10
outlines the design, architecture, and features of
Elasticsearch, emphasizing its distributed nature,
scalability, and support for real-time search and analytics.
The paper highlights Elasticsearch’s RESTful API, which
allows users to interact with the search engine using HTTP
requests, and discusses its use cases in various domains,
including log analysis, full-text search, and business
intelligence. Since its launch in 2010, Elasticsearch has
quickly become the most popular search tool and is widely
used for use cases related to log analysis, full-text search,
security information, business analytics, and operational
information. Elasticsearch is considered a search engine
and inherits from Apache Lucene. Elasticsearch essentially
functions as a web server with fast search capabilities
through the RESTful protocol. It possesses highly efficient
data analysis and statistical capabilities and runs on its own
server platform and communicates via RESTful, so it is not
overly dependent on what client or existing system you
have written it in. Therefore, integrating it into your system
becomes easier, and you only need to send an HTTP
request, and it will return the results. Elasticsearch is a also
distributed system with incredible scalability and is an
open-source system
2. AI Module
2.1 BERT Model
BERT (Bidirectional Encoder Representation from
Transformer) stands for a model that represents words
bidirectionally using Transformer techniques. BERT is
designed to pre-train word embeddings. One notable
feature of BERT is its ability to balance context in both left
and right directions.
The attention mechanism of Transformer passes all
words in the sentence simultaneously into the model at
once without considering the direction of the sentence.
Therefore, Transformer is considered bidirectional training,
although in reality, it is more accurate to say it is
non-directional training. This feature allows the model to
learn the context of a word based on all surrounding words,
including both left and right words.
One unique aspect of BERT that previous embedding
models did not have is the ability for fine-tuning the
training results. We can add an output layer to the model
architecture to customize it for specific training tasks.
Fig. 13. Pre-train and Fine-tuning process of BERT
Currently there are many different versions of BERT. All
the versions are based on the change if Transformer
architecture focusing on three parameters:
• L: the number of block sub-layers inside transformer
• H: the size of the embedding vectors (aka hidden
size)
• A: the number of heads inside a multi-head layer,
each head will perform a self-attention mechanism
 There are two main architecture with different names:
BERTbase (L = 12, H = 768, A = 12) which has 110 millions
parameters in total and BERTlarge (L = 24, H = 1024, A = 16)
which has 340 millions parameters in total In BERT large
architecture, the number of layers has been doubled, the
hidden size has been increased 1.33 times and the number
of heads in multi-head layers are also 1.33x times the BERT
base.
BERT also take the classification token - CLS for short - as
the input and follwed by the word sequence. After that it
transports the input to the upper layers. Each layer applies
self-attention and passes the result through a feedforward
network, then it hands over to the next encoder layer. The
output of the model is a vector sized according to the
BERT’s dimension. If we want to extract a classifier from
this model, we can take the output corresponding to the
CLS classification token.
Fig. 14. Input illustration for BERT
2.2 Fake news detection with BERT
As stated in section 4.2.1, the input of the BERT model is a
classification token, so we need to tokenize the values to use
as input for the model. For the BERT Classification model
architecture for classifying fake news, our group proposes
the following:
Fig. 15. BERT Classfication architecture for fake news
detection
With the BERT model in use, the input is headlines and
content of the news article and the output is the classification
result over whether the news is fake or not.
2.3 Keyword Extraction using KeyBert
KeyBERT [32] is a minimalistic and easy-to-use keyword
extraction technique that leverages BERT embeddings to
generate keywords and keyword phrases most similar to
the document.
There are four steps that KeyBERT [33] has completed to
make predictions.
Step 1: The input document is embedded using a
11
pre-trained BERT model. This turns a piece of text into a
fixed-size vector that represents the semantic aspect of the
document.
Step 2: Keywords and phrases (n-grams) are extracted
from the same document using Bag of Words techniques
(such as TFIDF Vectorizer or CountVectorizer). This is a
classical step to perform keyword extraction.
Fig. 16. Demonstration for step 1 and 2 of KeyBert
Step 3: After that, each keyword is then embedded into a
vector with fixed size using the same model that we used to
embed the article
Fig. 17. Demonstration for step 3 of KeyBert
Step 4: Now that the keywords and the document are
represented in the same space, KeyBERT computes the
cosine similarity between the keyword embeddings and the
document embeddings. Then, the most similar keywords
(with the highest cosine similarity scores) are extracted.
Fig. 18. Demonstration for step 4 of KeyBert
Additionally, KeyBERT performs diversification of the
selected keywords. Without applying these techniques, the
model may select very similar words or phrases as
keywords.
For example, if there is an article about fruits containing
various types of fruits and this technique is not applied, the
top three keywords could be "green apple," "big apple," and
"yellow apple." Instead, we may want the model to include
various types of fruits in the top keywords, even if one type
is more common in the article. There are two main metrics
we would like to discuss. Firstly, Max Sum Similarity
(MSS): With this method, a parameter top_n, representing
the top keywords, is set to a value (e.g. 20). Then, 2 x top_n
keywords are extracted from the document. Pairwise
similarities are calculated between these keywords. Finally,
this method extracts the most relevant and least similar
keywords. The second metric is Maximal Marginal
Relevance (MMR): This method is similar to the previous
one, but it adds a parameter to represent diversity. MMR
aims to minimize redundancy and maximize diversity in
text summarization tasks. It starts by selecting keywords
most similar to the document. Then, it iteratively selects
new candidates that are similar to the document but
dissimilar to the already chosen keywords.
Fig. 19. Keyword extraction using KeyBERT
With the KeyBert model in use, the input of the model is
news headlines abd the output are keywords corresponding
to the headlines.
2.4 NER task with BiLSTM model combining CRF
The combined architecture of BiLSTM and CRF both
addresses the drawbacks and leverages the advantages of
each model in the NER task.
Fig. 20. Character Embeddings
Fig. 21. BiLSTM + CRF architecture
12
The model through two main steps. Firstly, word
embeddings of each word and character are fed into the
BiLSTM model layer to extract useful information about the
semantics and morphology of the word and the context
surrounding that word. Secondly, the CRF layer processes
this information as features to make predictions about NER
labels for each word. In addition to the information
received from the BiLSTM layer, CRF also relies on
information from previously predicted labels. For example,
if the previous label is B-LOCATION, it is highly likely that
the current word being considered will have the label
I-LOCATION.
The learning parameters of the BiLSTM+CRF architecture
include: the word embedding matrix, the weight matrix of
the BiLSTM layer, and the transition matrix of the CRF
layer. All of these parameters are updated during training
on labeled data through the Back-propagation algorithm
with Stochastic Gradient Descent (SGD).
3. Backend
To build backend, our group opted for Spring Boot [34].
Spring Boot is a module of Spring Framework, providing
feature RAD (Rapid Application Development). Spring
Boot is used to create independent applications based on
Spring. Spring Boot does not require XML configuration. It
is a standard configuration for software, which helps
increase productivity for developers. Advantages of Spring
Boot are that they have features of Spring Framework and
can create independent applications run by java -jar (also
for java web). Furthermore, Spring boot also possess
directly embedded applications server (Tomcat, Jetty...) so
no need to deploy war file. They also have few
configuration, automatically configure whenever possible
(Reduce coding time, increase productivity) and do not
require XML config. Spring Boot also provide many
plugins and is an industry standard for Microservices
(Cloud support; reduce setup, config; support libraries...)
4. Frontend
Our team uses ReactJS to develop the frontend. ReactJS is
an open-source library developed by Facebook, released in
2013. It is a JavaScript library used to build interactions with
components on websites. One of the most notable features
of ReactJS is the ability to render data not only on the server
side but also on the client side.
5. Docker
Our team utilizes Docker to deploy components in the
HT system. Each component is built into corresponding
containers.
Docker is a software platform that allows for fast
building, testing, and deployment of applications. Docker
packages software into standardized units called containers
that include everything the software needs to run,
including libraries, system tools, code, and runtime. By
using Docker, applications can be quickly deployed and
scaled into any environment, with the assurance that the
code will run.
 IV. Experimental Results

A. Model Evaluation
1. Experiment environment
The team is currently training models with the dataset in
the Python environment using deep learning libraries such
as Scikit-learn, Tensorflow [35], and Pytorch. The team
runs the training process on local machines and Google
Colab with corresponding configurations:

Fig. 22. Local computer configuration

Fig. 23. GPU configuration used on Colab

2. Dataset
2.1 PhoNER_COVID_19
PhoNER_COVID19 [36] is a dataset for identifying
named entities related to COVID-19 in Vietnamese, built
and published by VinAI Research.
The data was collected from articles tagged with the
keywords "COVID-19" or "COVID" from reputable online
news sources in Vietnam (VnExpress, ZingNews, BaoMoi,
and ThanhNien) from February 2020 to August 2020.
Subsequently, the main text content of the articles was
segmented into sentences using the RDRSegmenter from
VnCoreNLP. Sentences related to COVID-19 patients were
selected using BM25Plus. Then, they manually filtered out
sentences that did not contain relevant information about
COVID-19 patients in Vietnam, resulting in 10,027 raw
sentences.
Next, the data was manually labeled with a clear process.
The labeled results were reviewed and could be amended if
necessary. Finally, from the 10,000 raw text sentences,
35,000 entities were obtained. The authors divided the
dataset into corresponding Train/Validation/Test sets with
a ratio of 5/2/3.
PhoNER_COVID_19 is published at:
https://github.com/VinAIResearch/PhoNER_COVID19

2.2 Real-world Fake news:

Originating from a community competition on Kaggle in
2018, "Build a system to identify unreliable news articles,"
the dataset was published, consisting of 2 files: train.csv and
test.csv.
The news data collected is a collection of fake and real
news articles propagated during the 2016 United States
Presidential Election campaign.
The training data (train.csv) consists of 5 fields: news id
(id), news title (title), news author (author), news content
(text), and the label of the news (label). It contains 10,540
real news articles and 10,260 fake news articles.

13
3. Evaluation Results

PATIENT-ID AGE GENDER DATE OCCUPATION NAME LOCATION ORGANIZATION SYMPTOM TRANPORTATION

Istm-syllable 96 97 95 98 66 86 90 87 83 92

lstm-word 97 98 95 99 69 88 88 84 83 93

bert-syllable 95 93 94 98 56 90 84 77 84 83

bert-word 97 95 94 99 75 90 86 80 85 85

Table 2: Precision score

PATIENT-ID AGE GENDER DATE OCCUPATION NAME LOCATION ORGANIZATION SYMPTOM TRANPORTATION

Istm-syllable 96 91 90 96 55 82 94 78 88 93

lstm-word 95 88 86 96 57 85 93 80 86 95

bert-syllable 95 80 85 98 60 93 86 71 87 82

bert-word 96 82 86 99 65 93 86 76 88 83

Table 3: Recall score

PATIENT-ID AGE GENDER DATE OCCUPATION NAME LOCATION ORGANIZATION SYMPTOM TRANPORTATION

Istm-syllable 96 92 92 96 60 84 92 82 85 92

lstm-word 96 92 90 97 62 86 90 81 84 94

bert-syllable 95 86 89 98 58 91 85 74 85 82

bert-word 97 88 90 99 70 91 86 78 86 84

Table 4: F1 score

In the above tables, lstm-syllable means applying

BiLSTM+CRF model for syllable-level dataset and
lstm-word indicates applying BiLSTM+CRF model for
word-level dataset.

14
 Fig. 24. Comparing F1 score of BiLSTM + CRF model

Fig. 25. Accuracy comparison between our group’s model

15
3.1 NER task
It can be observed that the results of BiLSTM+CRF
applied at the word level are better than at the character
level for our team. However, the results of BiLSRM+CRF at
the word level from VinAI Research [36] [37] consistently
outperform ours, especially for the OCCUPATION (JOB)
label. Furthermore that predictions are far more accurate
with fields such as PATIENT_ID , AGE , GENDER and
achieve less correct guesses with fields like JOB.
These results demonstrate that (Figure 24) a more detailed
approach at the word level can potentially yield more
promising outcomes for our team in the future. Another
observation is the bert model at the word-level has also
shown good results, while syllable-level model are
out-performed in many categories such as OCCUPATION,
ORGANIZATION, LOCATION.

3.2 Fake news Detection task

Due to the lack of publicly available datasets for fake
news in Vietnamese, our team used fake news datasets in
English. The results (See Figure 25) show that the BERT
model our team used performs quite well since the
prediction results achieve an above 90% accuracy. However,
these are evaluation results on an English-language news
dataset, so it cannot be confirmed to have high accuracy as
on the Vietnamese-language dataset.

B. Experiment and Deployment

1. Deploy
Our teams have been able to deploy a website with
following interface (See Figure 26, Figure 27, Figure 28,
Figure 29):

Fig. 26. Login interface

16
Fig. 27. Dashboard interface containing all the dashboard in-
use of the user

Fig. 28. Home page after login

17
 Fig. 29. Detailed charts in a dashboard
This is the link of the demo video : https://youtu.be/
cbWDSplfBHY
2. Remarks
The web demo has fully met the functional requirements
of the tasks.
The web demo has a user-friendly and clear interface.
However, the functionality of detecting real/fake news is
not very easy to explain to users, meaning users do not fully
understand why the system classifies a news article A from
website B as real/fake.
V. Conlusion
A. Conclusion
The report has summarized the process that the team has
explored and developed an integrated system solution for
the news analysis problem. First of all, the team has
researched integrated system solutions and accompanying
technologies. Moreover, the team has explored some new
deep learning models and integrated AI solutions into the
system to enhance its usability. And the results obtained by
the team are quite satisfactory.
However, due to various objective and subjective reasons,
the team still has some limitations and shortcomings in
implementing this topic: the results from AI solutions are
not easy to explain to users.
B. Future Development
In the future, our team can explore and implement the
following directions:
Firstly, regarding integrated solutions, further explore
other integration solutions such as chatbots, news
recommendation systems, to personalize user experiences.
18
Secondly, concerning the quality of AI modules: delve
deeper into the domain of news data, gain a better
understanding of the structural levels of articles, texts;
explore and execute data enrichment to enable the model to
learn more. Implement the construction of real and fake
news datasets in Vietnamese and proceed with training.
References
[1] B. Jehangir, S. Radhakrishnan, R. Agarwal, “A survey
on named entity recognition—datasets, tools, and
methodologies,” Natural Language Processing Journal,
vol. 3, p. 100017, 2023.
[2] L. Tran, “Data scraping application with scrapy,” 2023.
[3] S. Vychegzhanin, E. Kotelnikov, “Comparison of
named entity recognition tools applied to news
articles,” in 2019 Ivannikov Ispras Open Conference
(ISPRAS), 2019, pp. 72–77, IEEE.
[4] S. Bird, E. Klein, E. Loper, Natural language processing
with Python: analyzing text with the natural language
toolkit. " O’Reilly Media, Inc.", 2009.
[5] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury,
G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga,
et al., “Pytorch: An imperative style, high-performance
deep learning library,” Advances in neural information
processing systems, vol. 32, 2019.
[6] M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi,
N. Liu, M. Peters, M. Schmitz, L. Zettlemoyer,
“Allennlp: A deep semantic natural language
processing platform,” arXiv preprint arXiv:1803.07640,
2018.
 [7] D. A. Ferrucci, “Introduction to “this is watson”,” IBM
Journal of Research and Development, vol. 56, no. 3.4,
pp. 1–1, 2012.
[8] A. Jain, I. Aggarwal, A. Singh, “Paralleldots at
semeval-2019 task 3: Domain adaptation with feature
embeddings for contextual emotion analysis,” in
Proceedings of the 13th International Workshop on Semantic
Evaluation, 2019, pp. 185–189.
[9] A. Dandelion, “Dandelion api,” línea]. Disponible:
https://dandelion. eu.[Último acceso: 23 de Enero de 2016],
2021.
[10] A. Brandsen, S. Verberne, K. Lambers, M. Wansleeben,
“Can bert dig it? named entity recognition for
information retrieval in the archaeology domain,”
Journal on Computing and Cultural Heritage (JOCCH),
vol. 15, no. 3, pp. 1–18, 2022.
[11] A. CARAVALE, P. MOSCATI, N. DURAN-SILVA,
B. GRIMAU, B. RONDELLI, “Developing a
digital archaeology classification system using
natural language processing and machine learning
techniques.,” Archeologia e Calcolatori, vol. 34, no. 2,
2023.
[12] S. Reshmi, K. Balakrishnan, “Enhancing
inquisitiveness of chatbots through ner integration,”
in 2018 International Conference on Data Science and
Engineering (ICDSE), 2018, pp. 1–5, IEEE.
[13] E. M. Kusumaningtyas, E. R. Laurentino, A. R.
Barakbah, “Responsive chatbot using named entity
recognition and cosine similarity,” 2023.
[14] J. Shin, E. Jo, Y. Yoon, J. Jung, “A system for
interviewing and collecting statements based on intent
classification and named entity recognition using
augmentation,” Applied Sciences, vol. 13, no. 20,
p. 11545, 2023.
[15] C. Manning, H. Schutze, Foundations of statistical natural
language processing. MIT press, 1999.
[16] B. Minixhofer, J. Pfeiffer, I. Vulić, “Where’s the
point? self-supervised multilingual punctuation-
agnostic sentence segmentation,” arXiv preprint
arXiv:2305.18893, 2023.
[17] D. Jurafsky, J. H. Martin, “Speech and language
processing: An introduction to natural language
processing, computational linguistics, and speech
recognition.”
[18] B. Elov, S. M. Khamroeva, Z. Xusainova, “The pipeline
processing of nlp,” in E3S Web of Conferences, vol. 413,
2023, p. 03011, EDP Sciences.
[19] P. Gholami-Dastgerdi, M.-R. Feizi-Derakhshi, “Part of
speech tagging using part of speech sequence graph,”
Annals of Data Science, vol. 10, no. 5, pp. 1301–1328,
2023.
[20] D. Nadeau, S. Sekine, “A survey of named
entity recognition and classification,” Lingvisticae
Investigationes, vol. 30, no. 1, pp. 3–26, 2007.
[21] S. Hochreiter, J. Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–
1780, 1997.
19
[22] M. Schuster, K. K. Paliwal, “Bidirectional recurrent
neural networks,” IEEE transactions on Signal
Processing, vol. 45, no. 11, pp. 2673–2681, 1997.
[23] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,
Z. Su, D. Du, C. Huang, P. H. Torr, “Conditional
random fields as recurrent neural networks,” in
Proceedings of the IEEE international conference on
computer vision, 2015, pp. 1529–1537.
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,
“Attention is all you need,” Advances in neural
information processing systems, vol. 30, 2017.
[25] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, “Bert:
Pre-training of deep bidirectional transformers
for language understanding,” arXiv preprint
arXiv:1810.04805, 2018.
[26] D. M. Powers, “Evaluation: from precision, recall
and f-measure to roc, informedness, markedness and
correlation,” arXiv preprint arXiv:2010.16061, 2020.
[27] V. Cothey, “Web-crawling reliability,” Journal of the
American Society for Information Science and Technology,
vol. 55, no. 14, pp. 1228–1238, 2004.
[28] J. Wang, Y. Guo, “Scrapy-based crawling and user-
behavior characteristics analysis on taobao,” in 2012
International Conference on Cyber-Enabled Distributed
Computing and Knowledge Discovery, 2012, pp. 44–52,
IEEE.
[29] J. Turnbull, The Logstash Book. James Turnbull, 2013.
[30] M.-P. Scott-Boyer, P. Dufour, F. Belleau, R. Ongaro-
Carcy, C. Plessis, O. Périn, A. Droit, “Use of
elasticsearch-based business intelligence tools for
integration and visualization of biological data,”
Briefings in Bioinformatics, vol. 24, no. 6, p. bbad348,
2023.
[31] A. ElasticSearch, “Distributed restful search engine,”
2012.
[32] https://maartengr.github.io/KeyBERT/.
[33] A. Besbes, “How to extract relevant
keywords with keybert.” [Online]. Available:
https://towardsdatascience.com/how-to-extract-
relevant-keywords-with-keybert-6e7b3cf889ae.
[34] M. Mythily, V. R. Kanakala, R. Nambiar, et al.,
“An extensive review of spring boot testing based
on business requirements of the software,” in 2023
4th International Conference on Smart Electronics and
Communication (ICOSEC), 2023, pp. 1547–1553, IEEE.
[35] S. Pattanayak, Pro deep learning with TensorFlow 2.0: A
mathematical approach to advanced artificial intelligence in
Python. Springer, 2023.
[36] D. Q. Nguyen, A. T. Nguyen, “Phobert: Pre-trained
language models for vietnamese,” arXiv preprint
arXiv:2003.00744, 2020.
[37] T. H. Truong, M. H. Dao, D. Q. Nguyen, “Covid-
19 named entity recognition for vietnamese,” arXiv
preprint arXiv:2104.03879, 2021.