Received 15 March 2023, accepted 14 May 2023, date of publication 29 May 2023, date of current version 7 June 2023.

Digital Object Identifier 10.1109/ACCESS.2023.3280556

Cyberbullying Detection Based on Emotion

MOHAMMED AL-HASHEDI1 , LAY-KI SOON2 , HUI-NGO GOH 1,

AMY HUI LAN LIM1 , AND EU-GENE SIEW 3

1 Facultyof Computing and Informatics, Multimedia University, Cyberjaya, Selangor 63100, Malaysia
2 School of Information Technology, Monash University Malaysia, Subang Jaya, Selangor 47500, Malaysia
3 School of Business, Monash University Malaysia, Subang Jaya, Selangor 47500, Malaysia

Corresponding author: Hui-Ngo Goh (hngoh@mmu.edu.my)

This work was supported by the Fundamental Research Grant Scheme, Ministry of Education, Malaysia, under
Grant FRGS/1/2017/ICT02/MMU/02/6.

ABSTRACT Due to the detrimental consequences caused by cyberbullying, a great deal of research has been
undertaken to propose effective techniques to resolve this reoccurring problem. The research presented in this
paper is motivated by the fact that negative emotions can be caused by cyberbullying. This paper proposes
cyberbullying detection models that are trained based on contextual, emotions and sentiment features.
An Emotion Detection Model (EDM) was constructed using Twitter datasets that have been improved in
terms of its annotations. Emotions and sentiment were extracted from cyberbullying datasets using EDM and
lexicons based. Two cyberbullying datasets from Wikipedia and Twitter respectively were further improved
by comprehensive annotation of emotion and sentiment features. The results show that anger, fear and guilt
were the major emotions associated with cyberbullying. Subsequently, the extracted emotions were used as
features in addition to contextual and sentiment features to train models for cyberbullying detection. The
results demonstrate that using emotion features and sentiment has improved the performance of detecting
cyberbullying by 0.5 to 0.6 recall. The proposed models also outperformed the state-of-the-art models by
a 0.7 f1-score. The main contribution of this work is two-fold, which includes a comprehensive emotion-
annotated dataset for cyberbullying detection, and an empirical proof of emotions as effective features for
cyberbullying detection.

INDEX TERMS Cyberbullying, BERT, emotion mining, sentiment analysis.

I. INTRODUCTION affective computing that aims to identify, analyze, and eval-

The advancement of information and communication tech- uate the human state of mind towards various events or
nologies has provided an avenue for the online community encounters [8]. Emotion analysis has significantly impacted
to publish and respond to user-generated content (UGC). different sectors such as the stock market, consumers’
Unfortunately, such convenience has been abused by online feedback, and recommendations [9], [10], [11]. However,
bullies, causing harm to others via threatening, harassing, researchers have not deeply taken emotion analysis into con-
humiliating, intimidating, manipulating, or controlling tar- sideration for cyberbullying detection. Therefore, including
geted victims [1]. These acts, termed ‘cyberbullying,’ are emotion features to facilitate cyberbullying detection can
defined as the willful and repeated harm inflicted using elec- improve detection accuracy due to the strong relationship
tronic devices [2], [3], [4]. between cyberbullying and negative emotions.
Cyberbullying (CB) can have a severe impact on a victim’s The textual format of cyberbullying comes in two main
mental health, ranging from negative emotions (anger, fear, forms, either via explicitly articulated profane words or
sadness, guilt, etc.) to depression, and even suicidal thoughts implicitly expressed ironic or sarcastic statements that do not
[5], [6], [7]. Emotion mining is one of the focus areas in contain vulgar words. Detecting sarcastic and ironic expres-
sions is a challenging and problematic task [12]. Based on
The associate editor coordinating the review of this manuscript and the literature review and experiments, the datasets used for
approving it for publication was Giacomo Fiumara . cyberbullying currently suffer from several limitations, which
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
VOLUME 11, 2023 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ 53907
 M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion

are data sparsity, a large imbalance between labels, and lim- • An improved and thoroughly processed cyberbullying
itations to specific social networking sites [13]. The efforts dataset annotated with emotion and sentiment features.
to create an autonomous cyberbullying detection model have • A label validation for the emotion dataset that discards
been unsuccessful so far due to the high speed and diversity instances with inaccurate annotations.
of user-generated content (UGC) on the Internet [14]. • A proposed DL-based cyberbullying model that incor-
The solution proposed in this work is driven by the hypoth- porates semantic, syntactic, emotion, and sentiment fea-
esis that the detection of cyberbullying can be improved tures.
through emotion mining. It consists of three stages. Table 1 defines the variables and notations used in this
The first stage is to obtain a clean, balanced, and feature- paper to serve readers as reference points to easily compre-
rich dataset. The effectiveness of machine learning mod- hend the meaning of these terms.
els is heavily dependent on the quality of the training
datasets [15], [16]. However, datasets that cover all forms TABLE 1. Variables and notations used in this paper.
of cyberbullying are rare [13], [17]. Additionally, the nature
of UGC varies across different social networking sites. For
example, tweets on Twitter have a different character limit
compared to Facebook posts. As a result, the first stage of
our proposed solution focuses on cleaning, transforming, and
integrating data from various sources to produce a clean,
balanced, and feature-rich dataset. This stage accounts for at
least 70% of the data lifecycle in this work.
The second stage focuses on extracting features from the
user-generated content (UGC), including semantic, syntactic,
emotional, and sentiment features. The correct extraction
and integration of these features will lead to a robust and
effective cyberbullying detection model, regardless of the
social media platform used. Semantic and syntactic features
can be obtained by using language models such as BERT
(Bidirectional Encoder Representations from Transformers).
In this paper, emotions are extracted from text using both
supervised and unsupervised approaches. The supervised
approach involves applying deep learning (DL) algorithms
on an emotion dataset obtained from Twitter. The tweets are
classified into specific emotions using hashtags. To validate
the automatic labeling for the emotion dataset, three valida-
tion steps have been proposed and implemented. These steps
aim to verify the sentiment and emotions of instances using
lexicons and to train emotion models using small, validated
datasets.
On the other hand, the unsupervised approach is based on a
predefined lexicon, which is a keyword-matching technique
that assigns the dominant emotions associated with a word.
This is a word-level technique that helps identify explicit
expressions in the text. Along with emotion features, senti-
ment features are extracted from text, and an overall polarity
is assigned to each instance.
After extracting the cyberbullying features, deep learning
DL models are trained in the third stage to classify the data
instances into the discrete categories of cyberbullying or
non-cyberbullying. Two sets of experiments are conducted.
The first set uses selected language models, including BERT
(BERT base and BERT large) and XLNet. The second set
includes additional features, namely emotions and sentiment.
The results show that the inclusion of these additional features The remainder of the paper is structured as follows.
has improved the performance of the cyberbullying detection Section II presents the related works and highlights the
models by 5% to 6%. In summary, the main contributions of research gaps. Section III discusses the preparation and pre-
this work are: processing of the datasets. Section IV presents an EDM

53908 VOLUME 11, 2023

M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
that aims to identify emotions associated with cyberbullying.
In Section V, a series of experiments were presented, fol-
lowed by the results and analysis in Section VI. Section VII
concludes and offers some recommendations for future work.
II. RELATED WORKS
In the past, different studies attempt to detect cyberbullying
through different sets of features [13], [18], [19], [20], [21].
These features varied and are dependent on the social media
platforms that are being explored. For instance, demographic
information such as gender, personality, number of follow-
ers, and date of account creation is adopted in some stud-
ies [22], [23], [24]. Other works have identified bullies using
tweets, user profile, and network-based attributes on Twitter.
These features focused on distinguishing bullies from regular
users [25]. Other features such as a number of comments and
likes are also taken into consideration by [26]. In addition,
features such as binary sentiment are used to identify cyber-
bullying by related works [17], [27], [28].
Profile and usage features such as gender, number of fol-
lowers, and hours spent online can be faked, or may not be
available due to data protection policies practiced by some
social media platforms. Furthermore, the available features
vary from one social media platform to another. Many works
rely on the textual UGC itself for cyberbullying detection.
The content is represented in word vectors or language mod-
els [13], [29]. In addition, there are other features that can
be extracted from the textual UGC. These features are inde-
pendent of the target social media platform since they are
generated purely from the UGC.
Generally, there are two categories of such features,
namely, syntactic, and semantic features. Syntactic features
are normally obtained from sentence parsing or depen-
dency structure. These include part-of-speech (POS) tag-
ging, named entity recognition (NER) and term frequency
inverse document frequency (TF-IDF). The TF-IDF is used
to weigh the importance of a word in a document and con-
vert words to numerical embedding. There are some stud-
ies that use this technique to convert UGC to embeddings
and encode them to machine learning classifiers to classify
cyberbullying [30], [31].
On the other hand, semantic features aim to enrich the
UGC from the meaning perspective. Word embeddings is
a feature extraction technique to calculate a word’s weight
based on its contexts. There are word embeddings models
such as word2vec and Glove. Works that have implemented
these models are [13] and [32]. Furthermore, more seman-
tic features were included in the UGC, for instance, some
works extend a corpus by including a dictionary of impor-
tant words [33], [34], [35], as well as the sentiment of the
text [36], [37], [38], [39]. The extracted syntactic and seman-
tic features are then transformed into textual representation
models through more advanced models such as BERT [40]
or XLNet [41]. These state-of-the-art (SOTA) language
VOLUME 11, 2023
models have shown a huge improvement in natural language
processing (NLP) applications such as sentiment analysis,
question answering, topic classification and machine trans-
lation [40], [42], [43]. With BERT, classification can be
done to classify text into the binary cyberbullying or non-
cyberbullying categories.
Since cyberbullying has a strong impact on the victim’s
psychology, emotions and sentiment can be extracted from
the content to further improve the detection [44], [45]. Sen-
timent analysis has been widely used in different domains,
such as products and service reviews, political analysis,
etc. [46], [47], [48]. However, there are very few studies that
incorporate emotion and sentiment in detecting cyberbully-
ing, particularly for emotions [20], [49], [50], [59].
For sentiment analysis, there are typically three main
classes, namely, positive, negative and neutral. Intuitively,
negative sentiment is more relevant for detecting cyberbully-
ing. For emotion categories, guilt and love with the six basic
emotions defined by [51] where emotions (anger, disgust,
fear, joy, sadness, and surprise) are used in this work. Emo-
tions are more refined, as they portray the exact feeling of the
victims. By identifying the type of emotion in a text, more
information can be revealed as to whether cyberbullying is
taking place. Therefore, this study focuses on emotions due
to the strong relationship between the impact of bullying and
negative emotions.
III. DATASETS
This section describes the datasets employed in this work.
Firstly, we examine the cyberbullying datasets used to train
cyberbullying detection models (CDMs). Secondly, we detail
the emotion dataset used to construct the emotion detection
model (EDM).
There are two major datasets used for CDMs. Firstly,
the toxic dataset was sourced from Wikipedia, and the hate
speech dataset was crawled from Twitter. Wikipedia and
Twitter are two different platforms in terms of text format.
Wikipedia comments consist of formal and lengthy com-
ments, while tweets are short, informal and may include
misspelled words. Therefore, it is essential to train CDMs
based on these two datasets.
To extract emotions from CB datasets, two emotion
datasets were combined to train EDM namely Cleaned Bal-
anced Emotional Tweets (CBET) and Twitter Emotion Cor-
pus (TEC). Both datasets were extracted from Twitter and
labeled using hashtag keywords, such as #anger, #fear, etc.
Human judgment was not examined during the labeling
process. Therefore, a validation procedure was followed to
discard any potentially incorrect instances. The reason for
combining the two datasets is to increase the size of the
datasets after discarding wrong labeled instances. A detailed
description of each dataset is provided in the following sec-
tions. Table 2 summarizes the four major datasets used for
CDMs and EDM.
53909

TABLE 2. Summary of datasets used in this work.
A. CYBERBULLYING DATASETS
In this study, considerable effort has been made to data pre-
processing, including acquisition, cleaning, and sampling.
Two main challenges faced by cyberbullying datasets are data
sparsity and imbalanced datasets. Data sparsity in cyberbul-
lying refers to a shortage in the number of instances that cover
both explicit and implicit forms of cyberbullying. Another
significant problem that can affect the performance of any
machine learning model is the imbalance in the ratio between
cyberbullying and non-cyberbullying classes. In cyberbul-
lying datasets, positive instances, which are cyberbullying,
are in the minority. For example, the Formspring dataset
mostly contains insults, lacking other forms of cyberbully-
ing [52]. Additionally, there is a large imbalance in the ratio
between the cyberbullying class (CB) and non-cyberbullying
class (non-CB), with positive instances consisting of only
6% of the total dataset. To overcome these limitations, this
study has used two datasets from two different social media
sources.
1) TOXIC DATASETS
a: DATASET ACQUISITION AND PRE-PROCESSING
The Toxic dataset is collected from the Wikipedia plat-
form by the Conversation AI team, which was founded by
Jigsaw and Google https://www.kaggle.com/c/jigsaw-toxic-
comment-classification-challenge/data. The datasets are pub-
licly shared on Kaggle for competition and research use.
The goal of these efforts is to support research towards
making the internet a safer place by identifying toxic com-
ments, which include rude, disrespectful, threatening, insult-
ing, and hate speech. This dataset is especially valuable
for detecting cyberbullying because it contains instances
of explicit and implicit expressions, overcoming the spar-
sity problem faced by previous cyberbullying datasets. The
dataset contains approximately 214,000 comments, with only
10% being classified as cyberbullying. Pre-processing steps
have been applied, such as removing punctuation, symbols,
53910
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
and non-English letters, lemmatizing the texts, and converting
the words to lowercase. Digits-containing words were also
removed as they may introduce noise to word representation
models. Stop words were not removed as BERT can handle
contextual information effectively. This dataset was chosen
for cyberbullying detection due to its rich content and appro-
priate size for building robust models
b: DATA SAMPLING
To address the imbalanced nature of the dataset, this study
proposes an under-sampling technique based on an anal-
ysis of the data. The non-cyberbullying class has about
193,000 comments, while the cyberbullying class has about
21,000 comments. The under-sampling approach applied to
the majority class aims to improve the class balance and
ensure relatively similar text lengths for both classes. Rather
than simply discarding instances randomly, a careful proce-
dure is designed to construct a useful dataset.
To under-sample the non-cyberbullying class, the number
of words in each instance is counted. Instances with less than
or equal to two words or more than 254 words are removed in
CB and Non-CB classes. This step is taken because instances
with fewer than two words are often meaningless and could
negatively impact the performance of the model. Instances
with more than 254 words may not be fully captured by the
DL vector due to its size. As a result of this step, 638 instances
were removed from CB class which represents only 2.8% of
all the CB instances.
Next, to ensure that the two classes have similar text
lengths, instances are categorized into 12 groups based on
the word count. Approximately 73% of the instances contain
3 to 60 words. These instances are divided into six categories,
each with a range of 10 words. Instances with more than
60 words are divided into six categories with different ranges,
as shown in Fig.1.
Thirdly, to achieve a similar length of instances and the
desired ratio of 25-75% between the CB and non-CB classes,
three times the number of CB instances are selected from
the non-CB instances for each category, while the remaining
instances are discarded. Fig. 1. displays the 12 categories
with the word ranges for each category on the x-axis, and the
number of instances for the CB, non-CB, and under-sampled
non-CB classes on the y-axis. The number of under-sampled
non-CB instances is calculated by:
samplednonCB = #CB × 3 (1)
Ultimately, the aim is to reduce the imbalance from
10-90% to 25-75%. In other words, the non-CB should be
triple the size of the cyberbullying class. The ratio is deter-
mined based on the performance of the BERT representation
in training the model. As the CB contains quite a good number
of instances, this improved ratio, albeit still imbalanced, does
not negatively affect the training, as shown in the experiment
results.
VOLUME 11, 2023
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
FIGURE 1. Dataset distribution for each category before and after
sampling for both cyberbullying and non-cyberbullying classes.
2) HATE SPEECH AND OFFENSIVE DATASET
(TWITTER DATASETS)
Due to the popularity of Twitter and the difference in nature
between it and Wikipedia (toxic dataset), it is necessary
to include tweets when building a cyberbullying detection
model. The toxic dataset, derived from the Wikipedia com-
ments page, is cleaner, longer, and contains fewer misspelled
words. Meanwhile, Twitter datasets often consist of short,
informal text and may include intentional spelling errors and
user tags, links, and hashtags.
In this study, a hate speech and offensive language dataset
is improved and utilized to create a more robust model. Hate
speech is another term for bullying and includes threats,
insults, and offensive language. This dataset was deemed suit-
able for building cyberbullying detection models with some
improvements. The dataset was collected using the Twitter
API and a hate speech lexicon compiled by hatespeech.org.
Approximately 25,000 tweets were selected and manually
labeled by CrowdFlower [53].
Out of the 25,000 tweets, 83% were classified as hate
speech or offensive, while the remaining 17% were catego-
rized as the negative class. To address the imbalance between
the two classes, another publicly available Twitter dataset
from Kaggle was utilized. This dataset was labeled as insults
or non-insults instances. The combined dataset named the
cyberbullying dataset (CD), is shown in Table 3 after over-
sampling to increase the size of the negative class.
TABLE 3. Data distribution before and after sampling of cyberbullying
datasets.
B. EMOTION DATASETS
Developing emotion models based on clean and accurately
labeled datasets is a critical step in identifying emotions
VOLUME 11, 2023
within cyberbullying instances. However, there are several
challenges in creating accurate emotion datasets. Some exist-
ing emotion datasets are automatically labeled by using
keyword hashtags, where an instance is tagged with a spe-
cific emotion class if it is hashtagged with an emotion term
(#anger, #fear, etc.). This approach is often used due to the
high cost of manual labeling by experts. The hashtags are
believed to accurately reflect the intent of the user who posted
the content.
However, there are many instances of tweets that do not
align with the emotion hashtags. Some examples of these
cases are listed in Table 4. Given the potential inaccuracies
of hashtags, this study proposes a process to validate the
instances’ labels.
TABLE 4. Some of the instances have been wrongly labeled by using
hashtags and ignored after the validation procedure.
In this study, two emotion datasets are validated and
utilized for the development of emotion models. The first
dataset is the Cleaned Balanced Emotional Tweets (CBET)
dataset [54], which is based on a basic emotions model [51]
that includes anger, disgust, fear, joy, sadness, and surprise.
This dataset also includes the emotions of thankfulness, love,
and guilt. The second dataset is the Twitter Emotion Corpus
(TEC) [55], which is also crawled from Twitter using the
Twitter API and labeled automatically using hashtags.
To produce more accurate labeled datasets, three validation
steps are carried out, as depicted in Fig. 2.
The first validation step is focused on checking the general
sentiment of each instance. Sentiment analysis is used to
determine the overall emotion expressed in the text, whether
it is positive, negative, or neutral. A lexicon-based approach
is used to identify the sentiment polarity of each instance.
The AFINN lexicon, which is developed based on microblog
English word lists [56], is utilized in this process. Each
word in the lexicon has been manually rated for valence,
with an integer score ranging from +5 (positive) to -5 (neg-
ative). Before identifying the sentiment of each instance,
misspelled words are corrected using the Norvig probabilistic
method [57]. A lexicon-based approach requires correcting
misspelled words to produce accurate results. If an instance
is labeled with a negative emotion (anger, fear, sadness, guilt,
53911

FIGURE 2. Validation steps to validate the annotation of emotion
datasets.
and disgust) but the identified sentiment is positive, it will be
discarded, and vice versa.
Secondly, an emotion lexicon is utilized to validate the
labeling of the automatic process. The NRC Emotion Lexicon
is employed to assess the overall emotion of a given instance,
which includes eight emotions (anger, fear, anticipation, trust,
surprise, sadness, joy, and disgust) and a binary value of
positive or negative. Each term in the lexicon is linked to one
or multiple emotions, which have been annotated manually
through crowdsourcing. The lexicon is used to generate a list
of emotions for every instance in the dataset through keyword
lookup. To determine the emotion type of an instance, the
emotions in the lexicon are categorized into negative and
positive emotions. Negative emotions include anger, fear,
sadness, disgust, and negative sentiment, while positive emo-
tions include the rest. A formula is then devised to classify
an instance as overall positive or negative based on the list
of emotions generated. If the majority of emotions for the
instance are negative, the instance is considered overall neg-
ative and vice versa. Finally, a comparison is made between
the labels of the dataset and the emotion class produced by the
lexicon. If there is any discrepancy, the instance is discarded.
The last validation step is performed through a trained DL
model using a short and manually labeled dataset. The name
of the dataset is Emotion Intensity (EmoInt) by [58]. The
EmoInt dataset contains four emotions: anger, fear, sadness
and joy. Since the priority is mainly on the emotions that
are associated with cyberbullying, it has been found to be a
significant dataset that contains three negative emotions. The
dataset consists of 7,000 tweets, which is sufficiently small
to build a robust emotion detection model. BERT is used for
word representation, and the data is split to 80% for training
53912
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
and 20% for testing. The results of the model show an 89%
recall and precision value. The model is then used to predict
the emotions for the instances produced in the previous step.
The correctly predicted instances are preserved, while the
others are discarded. The trained DL model is built to validate
instances with the four relevant emotions only. Due to the
unavailability of manually labeled datasets for the other emo-
tions, those instances are checked with the first two validation
steps only. The produced dataset after the validation process
is shown in Fig. 3.
There are 32,000 instances spanning eight fundamental
emotions. The validated emotion dataset (VED) covers five
negative emotions (anger, fear, sadness, guilt and disgust) and
three positive emotions (joy, love and surprise).
FIGURE 3. Validated emotion dataset instances distribution.
IV. EMOTION DETECTION MODEL (EDM)
This section outlines the training process for the emotion
detection model in our proposed solution, with a focus on
emotions related to cyberbullying. As shown in Fig. 4. the
EDM is trained using the validated emotion dataset (VED),
which is a multi-class classification where each instance is
assigned to a single emotion. Similar to the cleaning steps
applied to the Toxic Dataset, the VED dataset underwent data
cleaning. The data was split into 80% for training and 20% for
testing, and a stratification technique was employed to ensure
a fair representation of each emotion in the dataset. The BERT
base model was chosen for word representations, as it is
capable of effectively capturing the contextual features of
each emotion. The specific BERT base used in the experiment
has 768 feedforward network units, 12 attention heads, and
110 million parameters.
FIGURE 4. Emotion detection model (EDM).
The deep learning model for emotion detection is built
using the Keras libraries. Keras is a high-level library built
VOLUME 11, 2023
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion

on top of TensorFlow. During BERT tokenization, 15% of

the input words are masked, meaning that the masked words
must be predicted based on the surrounding words. The first
input token is supplied with the [CLS] token for classification
purposes. The input sequence of words is encoded into the
input layer, which then flows through the stack. Self-attention
is applied at each layer, and the results of each layer are passed
through the feed-forward network. The features produced
by BERT are encoded into hidden layers, which perform
activation functions to normalize the output of the nodes.
In this experiment, a Rectified Linear Unit (ReLU) activa- FIGURE 5. Emotion distribution for CB instances in the toxic and Twitter
tion function is attached to a dense layer with 256 nodes, datasets.

and the outputs are then forwarded to a classification layer

with eight nodes, each representing a distinct emotion. The
Softmax activation function is used in the classification layer
to normalize the outputs and convert them into probabilities
that sum up to one.
A. EMOTIONS ASSOCIATED WITH CYBERBULLYING
To empirically examine the emotions associated with cyber-
bullying, the EDM built using the VED was applied to clas-
sify the emotions of instances in the CD for the Toxic and
Twitter datasets. The labeled CD was fed into the EDM for
predicting the type of emotion each instance is associated
with. The Toxic dataset in the CD contains 21,830 instances,
with 92% classified as negative emotions (anger, fear, sad-
ness, disgust, and guilt), and 8% as positive emotions (joy,
love, and surprise). Anger was the most prevalent emotion,
followed by fear. The Twitter dataset in the CD contains
20,620 cyberbullying instances, with 88% categorized as neg-
ative emotions and the rest as positive. Guilt was the most
prevalent emotion, followed by disgust. Fig. 5. shows the
emotion distribution of cyberbullying for both datasets. The
results demonstrate that negative emotions can be used as
indicators in detecting cyberbullying
V. EXPERIMENTS
This section presents experiments conducted on the CD,
consisting of the Twitter and Toxic datasets, to evaluate the
efficacy of the proposed method. Given the challenges in
detecting cyberbullying content, this work investigated var-
ious features that could enhance the detection of cyberbul-
lying. The experiments were divided into two parts. In the
first part, only word representation models were used without
any additional features, while in the second part, emotion and
sentiment features were extracted from the text and combined
with the word representations.
A. LANGUAGE MODELS
To extract semantic features from text, words need to be
transformed into meaningful embeddings. Word embeddings
represent words as numerical vectors that can be fed into
neural networks. These embeddings capture the contex-
tual relationships between words in a sequence, and can
generate domain-specific textual features based on labeled
data. Advances in language representation extraction from
text have been made in recent years, and various types
of language models have been explored for cyberbullying
detection [13].
The most recent and highly regarded model is BERT, which
has created a sensation in the machine learning community
for a wide range of NLP tasks. BERT uses bidirectional
training of a transformer, which considers text sequences
from both the left and right directions, providing a deeper
understanding of each word and its context in a text sequence.
The pre-trained BERT model has been trained on large
corpora, such as the Book Corpus and English Wikipedia.
To fine-tune BERT for specific NLP tasks, a classification
layer is added on top of the pre-trained core model. BERT
has two versions: BERT base, which has 768 feedforward
network units, 12 attention heads, and 110 million param-
eters, and BERT large, which has 24 layers, 1024 feedfor-
ward network units, 16 attention heads, and 336 million
parameters [40].
Another language model used in the experiments is
XLNet, which is based on bidirectional encoding and
is trained through generalized autoregressive pre-training.
Unlike BERT, which predicts only 15% of the tokens in a
sequence, XLNet predicts all tokens in random order. The
XLNet base model has 12 layers, 768 feedforward network
units, and 110 million parameters [41].
B. PART 1 (USING WORD REPRESENTATIONS VIA
LANGUAGE MODELS ONLY)
In this work, experiments are conducted to detect cyber-
bullying using BERT base, BERT large, and XLNet pre-
trained word representation models. The experiments are
performed using the PyTorch machine learning framework
and Pytorch-transformers, an open-source Python library that
supports NLP applications. The high processing needs of
these pre-trained models are supported by Amazon Sage-
Maker, a cloud computing service for building, training, and
deploying machine learning models. The experiments are
conducted on the Toxic and Twitter datasets, with the data
split 80% for training and 20% for testing. Each model has
its own configuration, which is determined by the pre-trained
models.

VOLUME 11, 2023 53913


FIGURE 6. Extract emotion and sentiment features to be incorporated for
CB Detection Models (CDMs).
C. PART 2 (USING EMOTION AND SENTIMENT FEATURES
WITH WORD REPRESENTATIONS VIA LANGUAGE MODELS)
Fig. 6. shows our proposed solution, which includes senti-
ment and emotion features in the CB datasets. The proposed
solution starts with the aforementioned data pre-processing
steps. Each instance of the CB dataset is then fed into the
EDM to classify the type of emotion associated with it,
which can be anger, fear, sadness, guilt, disgust, joy, love,
or surprise. A list of emotions is also extracted based on the
NRC lexicon, matching each word in the instance to the types
of emotion it reveals. The AFFIN sentiment lexicon produces
a sentiment polarity and sentiment score for each instance.
These features - emotion based on EDM, list of emotions,
sentiment polarity, and sentiment score - were all added to
the CB datasets for each instance. We hypothesize that these
features provide more information for the trained model to
detect cyberbullying scenarios. The features which have been
considered are:
• Contextual features (BERT base): These features repre-
sent semantic, syntactic and contextual features that can
be extracted using word representations models.
• Emotions based on EDM: The overall emotion of an
instance that is detected based on the emotion detection
model (EDM).
• Emotions based on Lexicon: This is based on the NRC
Emotion Lexicon, which gives a list of emotions based
on the words of the cyberbullying instance. The list of
emotions is further collated to one single overall emo-
tion, which is either negative or positive, based on the
frequency formula as described in the second step of
validating the emotion dataset.
• Sentiment: The sentiment polarity is extracted based on
the AFINN lexicon. It contains a list of English words
that are manually assigned a score for valence.
Several experiments are conducted to determine the effec-
tiveness of the detection model based on the various textual
features. However, taking into consideration the strength of
the word representations in the pre-trained models, more
attention has been given to BERT. The list of settings con-
sidered are:
53914
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
• BERT base
• BERT base + emotions (EDM)
• BERT base + emotion (EDM) + lexicon emotions
• BERT base + Sentiment
• All features.
These experiments are conducted using the Keras open-
source library based on Tensorflow, which provides a Python
interface for artificial neural networks. The reason for using
Keras instead of Pytorch is the flexibility to include features
as inputs for the neural network built. The Keras functional
model API is used since it can handle models with multiple
inputs.
D. EVALUATION METRICS
Recall, precision, and F1 measure are used to measure the per-
formance of the detection models. These metrics are widely
used in classification models. The importance of each metric
depends on the classified topic.
Recall: Calculates the percentage of actual positives a
model correctly classified. In our case, the true positive is the
number of CB instances that have been correctly identified.
The recall is important since it identifies how many actual CB
instances are identified correctly. The false negatives are the
CB instances that have been flagged as non-CB.
Precision: Measures the percentage of predicted CB
instances that were correctly classified out of all the actual
CB instances. In real-time applications, precision is more
efficient to be used, since false positives are given higher
priority.
F1-score: measures the weighted average of precision and
recall. It is calculated as the following formula:
2 × ((precision × recall)/(precision + recall)) (2)
VI. RESULTS AND DISCUSSION
In this section, the results obtained during the experimen-
tal simulations are presented and analyzed. The results are
obtained using Pytorch and SageMaker for BERT base,
XLNet and BERT large.
A. RESULTS OF THE EMOTION DETECTION MODEL (EDM)
The performance of the model was measured using recall,
precision, and F1-score, which were calculated using the
Sklearn Python library. The model achieved a score of
0.84 for all three metrics, with only minor differences
between them. This result is considered efficient given the
number of classes and the size of the dataset in each class.
To gain a deeper understanding of the model’s performance,
we calculated the true positive rate (TPR) for each emotion.
Table 5 displays the TPR, the number of predicted instances,
and the total number of instances for each emotion. As seen
in the table, the model performed well in correctly classifying
instances of fear and joy, while love and guilt had the lowest
TPR with scores of 0.67 and 0.75, respectively. The results
indicate that the more instances a class has in the training
dataset, the better the model performs in predicting instances
VOLUME 11, 2023
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion

TABLE 5. True positive rate for each emotion. TABLE 7. Results for twitter hate speech wikipedia toxic dataset.

TABLE 8. Results of cyberbullying detection models with added features

for toxic dataset.

of that class. Despite this, the overall performance of the

model is good and it can be further utilized to detect emotions
in cyberbullying cases.

B. WORD REPRESENTATION MODELS

Table 6 and Table 7 present the results of using the pre-trained
models BERT base, XLNet, and BERT large for cyberbully-
ing detection. The differences between the models are minor
in both datasets. BERT base demonstrates consistency across
all three metrics in both datasets and outperforms XLNet and
BERT large in recall and F1-score on the Toxic dataset by 8%
recall and 4% F1-score. XLNet and BERT large, however,
achieve higher precision than BERT base by 1%. XLNet had
the lowest recall of 82%. For the Twitter dataset, the results
are high, and there is no significant difference between the
three models, as shown in Table 7. BERT base had a higher
precision than BERT large and XLNet, while BERT large had
a higher recall than BERT base and XLNet. All three models
scored 97% on the Twitter dataset in terms of F1-score. The
overall results indicate that these models can perform well in
detecting cyberbullying and there is no significant difference
in using BERT base, BERT Large, or XLNet to train the
models.
TABLE 6. Results for wikipedia toxic dataset.
C. BERT BASE RESULTS WITH ADDED FEATURES
Table 8 shows all the results of cyberbullying detection mod-
els with the additional extracted features for the Toxic dataset,
while Table 9 shows the results for the Twitter dataset. Three
metrics evaluate the performance of each model. Looking at
the results in general, added features have improved the per-
formance of the cyberbullying detection model. In the Toxic
dataset, all emotion features and sentiment features added
to BERT word representation models have outperformed the
BERT only model by 5% and 6% recall, and F1-score by 2%
and 3% respectively. Regarding the precision measure, BERT
with emotion detected by EDM has produced the highest
score of 0.95 on the Toxic dataset
Using all emotion features achieved the highest recall and
F1 measure values of 0.98 and 0.97 respectively. For the pre-
cision metric, BERT + emotion EDM + emotion extracted
from the NRC lexicon model and BERT + sentiment model
produced the highest precision of 0.97. Emotion features and
sentiment features added to BERT have performed consis-
tently well on both datasets.
The Toxic dataset shows relatively lower scores than the
Twitter dataset, due to the imbalance ratio between cyber-
bullying and non-cyberbullying classes, where the minority
instances belong to the cyberbullying class. However, it is
more practical for real-time applications to handle this imbal-
ance, because the majority of instances are not classified as
cyberbullying in some applications.
On the other hand, the Twitter dataset produced relatively
higher results than the Toxic dataset, due to the distribution
and the nature of the dataset. The cyberbullying class contains
the majority of the instances, and the Twitter instances are
relatively shorter than in the Toxic datasets. Therefore, mod-
els could perform very well using all features. Nevertheless,
understanding the cyberbullying language by a model which
contains the majority of cyberbullying instances can benefit
some applications where that kind of language is highly

VOLUME 11, 2023 53915


TABLE 9. Results of cyberbullying detection models with added features
for the twitter dataset.
occurring such as in online gaming, and video broadcasting
where users can leave comments.
D. COMPARISON OF RESULTS WITH RELATED STUDIES
A related study by Balakrishnan [20] investigated cyberbully-
ing through the use of features such as baseline features (text,
profile features, and network features), sentiment, emotion,
and the big 5 personality. The dataset used was extracted from
Twitter and contained 9,484 annotated tweet IDs. Random
Forest, Naive Bayes, and J48 machine learning algorithms
were applied to detect cyberbullying using the open-source
tool WEKA 3.8. Several experiments were conducted based
on the mentioned features. Although the datasets and clas-
sification algorithms used in the study differ from those in
our study, a comparison was made based on the performance
of the models for common features and Twitter datasets.
The F1-score was used for comparison since it is a common
evaluation metric in both studies.
TABLE 10. Comparison with related studies based on common features
and twitter datasets.
As shown in Table 10, our models outperformed the models
in the Balakrishnan study for both the baseline and sentiment
experiments, with a 7% improvement. Despite the difference
in baseline features between the two studies, our models still
scored 13% higher than the related study.
Another study by Maity et al. [59] used sentiment and
emotion features to improve the performance of their
cyberbullying detection models. The dataset consists of
6084 tweets in a mixture of English and Indian languages
and was processed using multilingual BERT for word
53916
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
representations. Their baseline model achieved an f1-score
of 0.80, while the model that incorporated sentiment and
emotion features had an f1-score of 0.82. Our models outper-
formed their models by 16% and 14% in both experiments,
respectively.
VII. CONCLUSION
This research paper proposes cyberbullying detection models
(CDMs) that utilize emotion features to enhance the effi-
ciency of detecting cyberbullying. In this work, all critical
steps were taken into consideration, from data preparation
to deep learning models. The preparation of the datasets
is a major challenge that all intelligent systems must over-
come, as their success is entirely dependent on the quality
of the datasets. Hence, meticulous attention was given to
the datasets, from the acquisition and pre-processing, up to
sampling. There is a sparsity issue in cyberbullying datasets
that encompasses all forms of cyberbullying, such as threat-
ening, harassing, humiliating, intimidating, and manipulating
or controlling targeted victims. To address this issue, this
research utilized two datasets. The first is the toxic dataset
collected by the Conversation AI team, and the second is
the Twitter dataset. The dataset of cyberbullying generally
faces an imbalance between its labels, therefore, sampling
techniques were developed to reduce the imbalance ratio. The
limitations of sparsity and imbalance in the cyberbullying
dataset were addressed and resolved.
After the preparation of the datasets, extracting textual
features was the second step in detecting cyberbullying. The
focus was on features that can be extracted from the text itself,
such as syntactic, semantic, contextual, emotion, and senti-
ment features. Features related to the perpetrator, such as gen-
der, age, and social media profiles, are not considered because
they are dependent on a specific social media platform and
can sometimes be difficult to obtain due to privacy policies.
Nevertheless, emotion features were thoroughly investigated
through the use of a deep learning model and lexicon-based
approach. To build an emotion detection model, the CBET
dataset which was collected from Twitter using hashtag key-
words was used. The dataset was labeled using hashtags as
the keywords. Due to the inaccuracy of the hashtag labeling,
a procedure was then carried out to validate the annotation
of the emotion dataset labels. The validated dataset was then
used to train the emotion detection model (EDM) using BERT
as a pre-trained word representation model. This model was
used to study and explore the emotions related to cyberbul-
lying texts. The results indicate that most cyberbullying texts
are categorized as negative emotions.
Emotions and sentiment were drawn out from cyberbul-
lying datasets through the use of EDM and NRC lexicon
for emotions and AFINN lexicon for sentiment. These fea-
tures were fed to deep learning models to train cyberbullying
detection models. A set of experiments were carried out with
different selections of features to investigate the best set of
features for cyberbullying detection. The results show that
emotions and sentiment features improve the precision of
VOLUME 11, 2023
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
cyberbullying detection and outperformed the use of BERT
contextual features. The use of emotion features added to
BERT resulted in a recall score of 0.87 on the Toxic dataset,
enhancing the performance of cyberbullying detection by
0.5 compared to using BERT alone. While the use of sen-
timent features scored 0.88 recall, improving the model by
0.6 recall compared to using BERT alone which in general is
greater than the baseline.
Future work can be focused on enhancing emotion datasets
in terms of size and annotations. Our study employed an auto-
matic validation procedure to eliminate instances with poten-
tially inaccurate annotations. This approach was adopted due
to the high cost of human annotation, particularly for large
datasets.
REFERENCES
[1] S. Modha, P. Majumder, T. Mandl, and C. Mandalia, ‘‘Detecting and visu-
alizing hate speech in social media: A cyber watchdog for surveillance,’’
Expert Syst. Appl., vol. 161, Dec. 2020, Art. no. 113725.
[2] S. Hinduja and J. W. Patchin, ‘‘Bullying, cyberbullying, and suicide,’’ Arch.
Suicide Res., vol. 14, no. 3, pp. 206–221, Jul. 2010.
[3] R. M. Kowalski, G.W. Giumetti, A.N. Schroeder, M.R. Lattanner, ‘‘Bully-
ing in the digital age: A critical review and meta-analysis of cyberbullying
research among youth,’’ Psychol. Bulletin, vol. 140, p. 1073, 2014, doi:
10.1037/a0035618.
[4] V. Balakrishnan, ‘‘Cyberbullying among young adults in Malaysia:
The roles of gender, age and internet frequency,’’ Comput. Hum. Behav.,
vol. 46, pp. 149–157, May 2015.
[5] S. M. B. Bottino, C. M. C. Bottino, C. G. Regina, A. V. L. Correia,
and W. S. Ribeiro, ‘‘Cyberbullying and adolescent mental health: Sys-
tematic review,’’ Cadernos Saude Publica, vol. 31, no. 3, pp. 463–475,
Mar. 2015.
[6] R. M. Kowalski, S. P. Limber, and P. W. Agatston, Cyberbullying: Bullying
in the Digital Age. Hoboken, NJ, USA: Wiley, 2012.
[7] X.-W. Chu, C.-Y. Fan, Q.-Q. Liu, and Z.-K. Zhou, ‘‘Cyberbullying
victimization and symptoms of depression and anxiety among Chi-
nese adolescents: Examining hopelessness as a mediator and self-
compassion as a moderator,’’ Comput. Hum. Behav., vol. 86, pp. 377–386,
Sep. 2018.
[8] A. Yadollahi, A. G. Shahraki, and O. R. Zaiane, ‘‘Current state of text
sentiment analysis from opinion to emotion mining,’’ ACM Comput. Surv.,
vol. 50, no. 2, pp. 1–33, Mar. 2018.
[9] Y. Ge, J. Qiu, Z. Liu, W. Gu, and L. Xu, ‘‘Beyond negative and positive:
Exploring the effects of emotions in social media during the stock market
crash,’’ Inf. Process. Manage., vol. 57, no. 4, Jul. 2020, Art. no. 102218.
[10] C. Yang, X. Chen, L. Liu, and P. Sweetser, ‘‘Leveraging semantic features
for recommendation: Sentence-level emotion analysis,’’ Inf. Process. Man-
age., vol. 58, no. 3, May 2021, Art. no. 102543.
[11] L. Jiang, L. Liu, J. Yao, and L. Shi, ‘‘A hybrid recommendation model
in social media based on deep emotion analysis and multi-source view
fusion,’’ J. Cloud Comput., vol. 9, no. 1, pp. 1–16, Dec. 2020.
[12] P. Parameswaran, A. Trotman, V. Liesaputra, and D. Eyers, ‘‘Detecting the
target of sarcasm is hard: Really?’’ Inf. Process. Manage., vol. 58, no. 4,
Jul. 2021, Art. no. 102599.
[13] M. Al-Hashedi, L.-K. Soon, and H.-N. Goh, ‘‘Cyberbullying detection
using deep learning and word embeddings: An empirical study,’’ in Proc.
2nd Int. Conf. Comput. Intell. Intell. Syst., Nov. 2019, pp. 17–21.
[14] J. Chun, J. Lee, J. Kim, and S. Lee, ‘‘An international systematic review of
cyberbullying measurements,’’ Comput. Hum. Behav., vol. 113, Dec. 2020,
Art. no. 106485.
[15] D. A. Winkler, ‘‘Role of artificial intelligence and machine learning in
nanosafety,’’ Small, vol. 16, no. 36, Sep. 2020, Art. no. 2001883.
[16] A. L’Heureux, K. Grolinger, H. F. Elyamany, and M. A. M. Capretz,
‘‘Machine learning with big data: Challenges and approaches,’’ IEEE
Access, vol. 5, pp. 7776–7797, 2017.
[17] S. Murnion, W. J. Buchanan, A. Smales, and G. Russell, ‘‘Machine learning
and semantic analysis of in-game chat for cyberbullying,’’ Comput. Secur.,
vol. 76, pp. 197–213, Jul. 2018.
VOLUME 11, 2023
[18] L. Cheng, J. Li, Y. N. Silva, D. L. Hall, and H. Liu, ‘‘XBully: Cyberbullying
detection within a multi-modal context,’’ in Proc. 12th ACM Int. Conf. Web
Search Data Mining, Jan. 2019, pp. 339–347.
[19] V. Balakrishnan, S. Khan, T. Fernandez, and H. R. Arabnia, ‘‘Cyberbully-
ing detection on Twitter using big five and dark triad features,’’ Personality
Individual Differences, vol. 141, pp. 252–257, Apr. 2019.
[20] V. Balakrishnan, S. Khan, and H. R. Arabnia, ‘‘Improving cyberbullying
detection using Twitter users’ psychological features and machine learn-
ing,’’ Comput. Secur., vol. 90, Mar. 2020, Art. no. 101710.
[21] H. Rosa, N. Pereira, R. Ribeiro, P. C. Ferreira, J. P. Carvalho, S. Oliveira,
L. Coheur, P. Paulino, A. M. Veiga Simão, and I. Trancoso, ‘‘Automatic
cyberbullying detection: A systematic review,’’ Comput. Hum. Behav.,
vol. 93, pp. 333–345, Apr. 2019.
[22] J. N. Navarro and J. L. Jasinski, ‘‘Going cyber: Using routine activities
theory to predict cyberbullying experiences,’’ Sociol. Spectr., vol. 32, no. 1,
pp. 81–94, Jan. 2012.
[23] V. Nahar, S. Al-Maskari, X. Li, and C. Pang, ‘‘Semi-supervised learning for
cyberbullying detection in social networks,’’ in Proc. Australas. Database
Conf. Cham, Switzerland: Springer, 2014, pp. 160–171.
[24] M. A. Al-garadi, K. D. Varathan, and S. D. Ravana, ‘‘Cybercrime detection
in online communications: The experimental case of cyberbullying detec-
tion in the Twitter network,’’ Comput. Hum. Behav., vol. 63, pp. 433–443,
Oct. 2016.
[25] D. Chatzakou, I. Leontiadis, J. Blackburn, E. D. Cristofaro, G. Stringh-
ini, A. Vakali, and N. Kourtellis, ‘‘Detecting cyberbullying and cyber-
aggression in social media,’’ ACM Trans. Web, vol. 13, no. 3, pp. 1–51,
Aug. 2019.
[26] H. Hosseinmardi, S. Arredondo Mattson, R. Ibn Rafiq, R. Han, Q. Lv, and
S. Mishra, ‘‘Detection of cyberbullying incidents on the Instagram social
network,’’ 2015, arXiv:1503.03909.
[27] J.-M. Xu, X. Zhu, and A. Bellmore, ‘‘Fast learning for sentiment analy-
sis on bullying,’’ in Proc. 1st Int. Workshop Issues Sentiment Discovery
Opinion Mining, Aug. 2012, pp. 1–6.
[28] J. A. Patch, ‘‘Detecting bullying on Twitter using emotion lexicons,’’
Ph.D. thesis, Dept. Sci., Univ. Georgia, Athens, GA, USA, 2015. [Online].
Available: https://getd.libs.uga.edu/pdfs/patch_jerrad_a_201505_ms.pdf
[29] M. J. Berger, ‘‘Large scale multi-label text classification with semantic
word vectors,’’ Stanford Univ., Stanford, CA, USA, Tech. Rep., 2015.
[Online]. Available: https://dspace.bracu.ac.bd/xmlui/handle/10361/6420
[30] Z. L. Chia, M. Ptaszynski, F. Masui, G. Leliwa, and M. Wroczynski,
‘‘Machine learning and feature engineering-based study into sarcasm and
irony classification with application to cyberbullying detection,’’ Inf. Pro-
cess. Manage., vol. 58, no. 4, Jul. 2021, Art. no. 102600.
[31] J. Eronen, M. Ptaszynski, F. Masui, A. Smywiński-Pohl, G. Leliwa, and
M. Wroczynski, ‘‘Improving classifier training efficiency for automatic
cyberbullying detection with feature density,’’ Inf. Process. Manage.,
vol. 58, no. 5, Sep. 2021, Art. no. 102616.
[32] Z. Mossie and J.-H. Wang, ‘‘Vulnerable community identification using
hate speech detection on social media,’’ Inf. Process. Manage., vol. 57,
no. 3, May 2020, Art. no. 102087.
[33] M. Dadvar, D. Trieschnigg, R. Ordelman, and F. de Jong, ‘‘Improving
cyberbullying detection with user context,’’ in Proc. Eur. Conf. Inf. Retr.
Cham, Switzerland: Springer, 2013, pp. 693–696.
[34] S. Mahbub, E. Pardede, and A. S. M. Kayes, ‘‘Detection of harassment type
of cyberbullying: A dictionary of approach words and its impact,’’ Secur.
Commun. Netw., vol. 2021, pp. 1–12, Jun. 2021.
[35] E. Hutson, ‘‘Cyberbullying in adolescence,’’ Adv. Nursing Sci., vol. 39,
no. 1, pp. 60–70, 2016.
[36] M. Z. Naf’an, A. A. Bimantara, A. Larasati, E. M. Risondang, and
N. A. S. Nugraha, ‘‘Sentiment analysis of cyberbullying on Instagram user
comments,’’ J. Data Sci. Appl., vol. 2, pp. 38–48, Jan. 2019.
[37] H. Dani, J. Li, and H. Liu, ‘‘Sentiment informed cyberbullying detection
in social media,’’ in Proc. Joint Eur. Conf. Mach. Learn. Knowl. Discovery
Databases. Cham, Switzerland: Springer, 2017, pp. 52–67.
[38] M. Sintaha, S. B. Satter, N. Zawad, C. Swarnaker, and A. Hassan, ‘‘Cyber-
bullying detection using sentiment analysis in social media,’’ Ph.D. the-
sis, Dept. Comput. Sci. Eng., BRAC Univ., Dhaka, Bangladesh, 2016.
[Online]. Available: https://dspace.bracu.ac.bd/xmlui/handle/10361/6420
[39] V. Nahar, S. Unankard, X. Li, and C. Pang, ‘‘Sentiment analysis for
effective detection of cyber bullying,’’ in Proc. Asia–Pacific Web Conf.
Cham, Switzerland: Springer, 2012, pp. 767–774.
[40] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-training
of deep bidirectional transformers for language understanding,’’ 2018,
arXiv:1810.04805.
53917

[41] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le,
‘‘XLNet: Generalized autoregressive pretraining for language understand-
ing,’’ in Advances in Neural Information Processing Systems, H. Wallach,
H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett,
Eds. Red Hook, NY, USA: Curran Associates, 2019. [Online]. Available:
https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733
e9ee67cc69-Paper.pdf
[42] Z. Xinxi, ‘‘Single task fine-tune BERT for text classification,’’ in Proc. 2nd
Int. Conf. Comput. Vis., Image, Deep Learn., Oct. 2021, pp. 194–206.
[43] S. González-Carvajal and E. C. Garrido-Merchán, ‘‘Comparing
BERT against traditional machine learning text classification,’’ 2020,
arXiv:2005.13012.
[44] M. N. Turliuc, C. Măirean, and M. Boca-Zamfir, ‘‘The relation between
cyberbullying and depressive symptoms in adolescence. the moderating
role of emotion regulation strategies,’’ Comput. Hum. Behav., vol. 109,
Jan. 2020, Art. no. 106341.
[45] A.-L. Camerini, L. Marciano, A. Carrara, and P. J. Schulz, ‘‘Cyberbul-
lying perpetration and victimization among children and adolescents:
A systematic review of longitudinal studies,’’ Telematics Informat., vol. 49,
Jun. 2020, Art. no. 101362.
[46] O. Araque, I. Corcuera-Platas, J. F. Sánchez-Rada, and C. A. Iglesias,
‘‘Enhancing deep learning sentiment analysis with ensemble techniques
in social applications,’’ Expert Syst. Appl., vol. 77, pp. 236–246, Jul. 2017.
[47] A. H. Alamoodi, B. B. Zaidan, A. A. Zaidan, O. S. Albahri,
K. I. Mohammed, R. Q. Malik, E. M. Almahdi, M. A. Chyad, Z. Tareq,
A. S. Albahri, H. Hameed, and M. Alaa, ‘‘Sentiment analysis and its
applications in fighting COVID-19 and infectious diseases: A systematic
review,’’ Expert Syst. Appl., vol. 167, Apr. 2021, Art. no. 114155.
[48] J. Guerreiro and P. Rita, ‘‘How to predict explicit recommendations in
online reviews using text mining and sentiment analysis,’’ J. Hospitality
Tourism Manage., vol. 43, pp. 269–272, Jun. 2020.
[49] M. Fortunatus, P. Anthony, and S. Charters, ‘‘Combining textual features to
detect cyberbullying in social media posts,’’ Proc. Comput. Sci., vol. 176,
pp. 612–621, Jan. 2020.
[50] W. A. Prabowo and F. Azizah, ‘‘Sentiment analysis for detecting cyber-
bullying using TF-IDF and SVM,’’ Jurnal Rekayasa Sistem Teknologi
Informasi, vol. 4, no. 6, pp. 1142–1148, Dec. 2020.
[51] P. Ekman, W. V. Friesen, and P. Ellsworth, Emotion in the Human Face:
Guidelines for Research and an Integration of Findings. Amsterdam,
The Netherlands: Elsevier, 2013.
[52] K. Reynolds, A. Kontostathis, and L. Edwards, ‘‘Using machine learning
to detect cyberbullying,’’ in Proc. 10th Int. Conf. Mach. Learn. Appl.
Workshops, vol. 2, Dec. 2011, pp. 241–244.
[53] T. Davidson, D. Warmsley, M. Macy, and I. Weber, ‘‘Automated hate
speech detection and the problem of offensive language,’’ in Proc. Int.
AAAI Conf. Web Social Media, 2017, vol. 11, no. 1, pp. 512–515.
[54] A. G. Shahraki and O. R. Zaiane, ‘‘Lexical and learning-based emotion
mining from text,’’ in Proc. Int. Conf. Comput. Linguistics Intell. Text
Process., vol. 9, 2017, pp. 24–55.
[55] S. Mohammad, ‘‘# emotional tweets,’’ in Proc. 6th Int. Workshop Semantic
Eval., 2012, pp. 246–255.
[56] F. Å. Nielsen, ‘‘A new ANEW: Evaluation of a word list for sentiment
analysis in microblogs,’’ 2011, arXiv:1103.2903.
[57] P. Norvig. (2007). How to Write a Spelling Corrector. [Online]. Available:
https://norvig.com/spell-correct.html
[58] N. Ide, A. Herbelot, and L. Màrquez, ‘‘Proceedings of the 6th joint confer-
ence on lexical and computational semantics (SEM 2017),’’ in Proc. 6th
Joint Conf. Lexical Comput. Semantics, 2017.
[59] K. Maity, A. Kumar, and S. Saha, ‘‘A multitask multimodal framework
for sentiment and emotion-aided cyberbullying detection,’’ IEEE Internet
Comput., vol. 26, no. 4, pp. 68–78, Jul. 2022.
MOHAMMED AL-HASHEDI received the bach-
elor’s degree in information technology from Mul-
timedia University, in 2017, where he is currently
pursuing the master’s degree with the Faculty of
Computing and Informatics. He was a Program-
mer and a Technical Manager with Al-Jifri Export
Company. His research interests include text min-
ing and machine learning.
53918
M. Al-Hashedi et al.: Cyberbullying Detection Based on Emotion
LAY-KI SOON received the Ph.D. degree (engi-
neering) in Web engineering from Soongsil Uni-
versity, South Korea, in 2009. She was a Senior
Lecturer with the Faculty of Computing and Infor-
matics (FCI), Multimedia University, where she
was also the Deputy Dean (research and inno-
vation) from 2016 to 2018. She was a Research
Fellow with the Telekom Malaysia Research and
Development on Database Optimization Project.
To date, she has graduated six Ph.D. and two mas-
ter’s students. Besides teaching at university, she has also conducted courses
on relational databases and NoSQL databases for corporate employees. Her
research interest includes data mining, with an emphasis on text mining.
HUI-NGO GOH received the bachelor’s degree
(Hons.) in computer science and the M.Sc. degree
from University Putra Malaysia, in 1999 and 2001,
respectively, and the Ph.D. degree in information
technology from Multimedia University, in 2014.
She has served as a Deputy Dean (academic) and
the Dean with the Faculty of Computing and Infor-
matics, Multimedia University, from 2016 to 2021,
where she is currently a Senior Lecturer. As a
Researcher, she is supervising and co-supervising
post-graduate students and a project leader and/or project member of gov-
ernment grants. As a trainer, she has conducted training on social media
analytics, big data essentials, text processing using Python, and advanced
data engineering. She is a certified HRDF trainer. Her research interest
includes text processing.
AMY HUI LAN LIM received the B.I.T. degree
(Hons) in information systems engineering, the
M.Sc. degree in information technology, and the
Ph.D. degree in information technology from
Multimedia University. She is currently a Senior
Lecturer with the Faculty of Computing and Infor-
matics, Multimedia University, Cyberjaya. Her
research interests include artificial intelligence,
information systems, data mining, and business
process management.
EU-GENE SIEW served as a Lecturer with KDU
Penang. He was an auditor in an audit firm and
an accountant in a manufacturing company. He is
currently a Senior Lecturer with the School of
Business, Monash University Malaysia. He has
researched and published in top-tier journals and
presented papers at conferences, including aca-
demic and industry seminars in the area of data
mining, Web mining, and accounting information
systems.
VOLUME 11, 2023