Displays 80 (2023) 102563

Contents lists available at ScienceDirect

Displays
journal homepage: www.elsevier.com/locate/displa

Review

Multimodal sentiment analysis: A survey✩

Songning Lai a , Xifeng Hu a , Haoxuan Xu a , Zhaoxia Ren b , Zhi Liu a ,∗
a
School of Information Science and Engineering, Shandong University, Qingdao, China
b
Assets and Laboratory Management Department, Shandong University, Qingdao, China

ARTICLE INFO ABSTRACT

Keywords: Multimodal sentiment analysis has emerged as a prominent research field within artificial intelligence,
Multimodal sentiment analysis benefiting immensely from recent advancements in deep learning. This technology has unlocked unprecedented
Multimodal fusion possibilities for application and research, rendering it a highly sought-after area of study. In this review, we
Affective computing
aim to present a comprehensive overview of multimodal sentiment analysis by delving into its definition,
Computer vision
historical context, and evolutionary trajectory. Furthermore, we explore recent datasets and state-of-the-art
models, with a particular focus on the challenges encountered and the future prospects that lie ahead. By
offering constructive suggestions for promising research directions and the development of more effective
multimodal sentiment analysis models, this review intends to provide valuable guidance to researchers in this
dynamic field.

1. Introduction Text-based sentiment analysis [13–15] has made significant strides in

Emotion refers to the subjective response of organisms to external
stimuli [1]. Humans possess a remarkable ability for sentiment anal-
ysis, and researchers are actively exploring methods to transfer this
capability to artificial agents [2].
Sentiment analysis involves the examination of sentiment polarity
using available information [3,4]. With the rapid advancements in
artificial intelligence, computer vision, and natural language process-
ing, the implementation of sentiment analysis by artificial agents is
becoming increasingly feasible. It is an interdisciplinary research field
that encompasses computer science, psychology, social science, and
other related disciplines [5–7].
For decades, scientists have been dedicated to equipping AI agents
with sentiment analysis capabilities, as it is a crucial aspect of achieving
human-like AI.
Sentiment analysis holds significant research value, evident from
a multitude of studies [8–11]. With the exponential growth of online
data, businesses can leverage evaluative information, such as reviews
and review videos, to enhance their products and services. Additionally,
sentiment analysis has found applications in diverse areas including lie
detection, interrogation, and entertainment. Subsequent sections will
delve into the practical applications and research potential of sentiment
analysis.
In the past, sentiment analysis primarily focused on a single modal-
ity, such as visual modality, speech modality, or text modality [12].
the field of natural language processing (NLP). Vision-based sentiment
analysis emphasizes the analysis of human facial expressions [16]
and body language. Speech-based sentiment analysis predominantly
extracts features like pitch, timbre, and intonation from speech signals
for sentiment analysis [17]. Thanks to advancements in deep learning,
these three modalities have made notable progress within the realm of
sentiment analysis.
Using a single modality for sentiment analysis has inherent limita-
tions [18–21]. Emotional information captured by a single modality is
often limited and incomplete. Conversely, leveraging multiple modal-
ities allows for a more comprehensive understanding of emotional
polarity. Analyzing only one modality yields restricted results and
hinders accurate emotion analysis.
Recognizing the significance of multi-modal sentiment analysis,
researchers have progressively developed numerous models to address
this need. Text features, for instance, dominate and serve as a crucial
component in deep emotional analysis [22]. Extracting expression and
pose features from the visual modality effectively complements text
sentiment analysis and judgment [23]. Speech modality, on one hand,
extracts textual features, while also enabling the recognition of speech
tones to unveil the temporal dynamics of the underlying text [24].
Fig. 1 illustrates the model architecture for a classical multi-modal sen-
timent analysis, comprising three essential parts: individual modality

✩ This paper was recommended for publication by Guangtao Zhai.

∗ Corresponding author.
E-mail addresses: 202000120172@mail.sdu.edu.cn (S. Lai), 201942544@mail.sdu.edu.cn (X. Hu), 202020120237@mail.sdu.edu.cn (H. Xu),
renzx@sdu.edu.cn (Z. Ren), liuzhi@sdu.edu.cn (Z. Liu).

https://doi.org/10.1016/j.displa.2023.102563
Received 28 June 2023; Received in revised form 2 October 2023; Accepted 23 October 2023
Available online 31 October 2023
0141-9382/© 2023 Published by Elsevier B.V.
S. Lai et al.
Fig. 1. The figure illustrates the model architecture for a classical approach to multi-modal sentiment analysis. The architecture comprises three essential components: feature
extraction for each modality, fusion of the extracted features, and sentiment analysis based on the fused features. These three components play a crucial role in the analysis process,
prompting researchers to focus on optimizing each part individually.
feature extraction, fusion of modality features, and sentiment analysis
based on the fused features. These three components are of utmost im-
portance, and researchers have initiated individual optimization efforts
for each part [25].
Our research focuses on multimodal sentiment analysis, providing a
comprehensive overview of the field, including its definition, historical
context, and evolutionary trajectory. It explores recent datasets and
state-of-the-art models, addressing challenges and future prospects.
Its unique contribution lies in offering constructive suggestions for
research directions. [26] discusses affective computing and automatic
emotion recognition, covering techniques, signal modalities, databases,
feature extraction, applications, and issues such as privacy and fair-
ness. [27] emphasizes emotion recognition based on modal informa-
tion, particularly video and audio, and reviews research methods and
fusion technologies. It also examines emotion recognition based on
deep learning and provides suggestions for future research. In compar-
ison to other surveys, our research stands out by providing a compre-
hensive overview and valuable guidance for researchers in multimodal
sentiment analysis.
This review provides a comprehensive overview of multimodal
sentiment analysis. It includes a summary and introduction to relevant
datasets, aiding researchers in selecting appropriate datasets for their
studies. We compare and analyze models that hold significant research
significance in multimodal sentiment analysis, offering suggestions for
model construction. In addition, we explore three types of modal
fusion methods, highlighting the advantages and disadvantages of each
approach. Finally, we discuss the challenges and future directions in
multimodal sentiment analysis, presenting several promising research
avenues. Distinct from other reviews in the field, our focus lies in
providing constructive suggestions for promising research directions
and constructing superior multimodal sentiment analysis models. We
accentuate the challenges and future prospects associated with these
technologies.
2. Multimodal sentiment analysis datasets
The rapid growth of the Internet has ushered in an era of data explo-
sion [28–30]. Researchers have extensively gathered these data from
various online sources such as videos and reviews, and have created
sentiment datasets tailored to their specific needs. Table 1 provides
a summary of commonly used multimodal datasets. The first column
lists the dataset names, while the second column indicates the release
year of the sentiment data. The third column categorizes the modalities
included in each dataset, and the fourth column specifies the platform
from which the dataset was obtained. The fifth column denotes the
language used in the dataset, and the sixth column quantifies the data
volume within each dataset. Each dataset possesses its own unique
2
Displays 80 (2023) 102563
characteristics. This section presents well-known datasets within the
research community, intending to assist researchers in understanding
the distinct features of each dataset and facilitating their selection
process.
2.1. IEMOCAP [31]
IEMOCAP, a sentiment analysis dataset released by the Speech
Analysis and Interpretation Laboratory in 2008, offers a comprehensive
multi-modal dataset. It consists of 1039 conversational segments, with
a total video duration of 12 h. The study involved participants engaging
in five different scenarios, expressing emotions based on predefined
scenarios. The dataset incorporates audio, video, and text information,
along with facial expression and posture data collected through addi-
tional sensors. Data points are categorized into ten emotions, including
neutral, happy, sad, angry, surprised, scared, disgusted, frustrated, ex-
cited, and other. IEMOCAP provides a valuable resource for researchers
investigating sentiment analysis across multiple modalities.
2.2. DEAP [32]
DEAP is a dataset specifically designed for sentiment analysis that
utilizes physiological signals as its source of data. The dataset examines
EEG data from 32 subjects, with a 1:1 ratio of male and female
participants. EEG signals were collected at 512 Hz from different areas
of the subjects’ brains, including the frontal, parietal, occipital, and
temporal lobes. To annotate the EEG signals, the subjects were asked
to rate the corresponding videos in terms of three aspects: Valence,
Arousal, and Dominance, on a scale of 1 to 9. This dataset provides
valuable resources for researchers to explore sentiment analysis using
physiological signals.
2.3. CMU-MOSI [33]
The CMU-MOSI dataset comprises 93 significant YouTube videos,
covering a diverse range of topics, as described by Zadeh et al. (2016).
The selection process ensured that each video featured a single speaker
facing the camera, enabling clear capture of facial expressions. While
there were no specific constraints on the camera model, distance, or
speaker scene, all presentations and comments in the videos were
conducted in English by 89 distinct speakers, including 41 women
and 48 men. The 93 videos were further divided into 2199 subjective
opinion segments, each annotated with sentiment intensity ranging
from strongly negative to strongly positive (−3 to 3). Overall, the CMU-
MOSI dataset serves as a valuable resource for researchers engaged in
the study of sentiment analysis.
S. Lai et al. Displays 80 (2023) 102563

Table 1
The table presented below showcases the utilized multimodal datasets.
Name Year Modalities Source Language Number
IEMOCAP 2008 A+V+T N/A English 10 039
A+V+T
DEAP 2011 A+V+T N/A English 10 039
A+V+T
CMU-MOSI 2016 A+V+T YouTube English 2199
CMU-MOSEI 2018 A+V+T YouTube English 23 453
MELD 2019 A+V+T The Friends English 13 000
Multi-ZOL 2019 V+T ZOL.com Chinese 5288
CH-SIMS 2020 A+V+T N/A Chinese 2281
Spanish
Portuguese
CMU-MOSEAS 2021 A+V+T YouTube German 40 000
French
Reddit
MEMOTION 2022 V+T Facebook English 10 000

2.4. CMU-MOSEI [34] 2.8. CMU-MOSEA [38]

CMU-MOSEI is a highly regarded dataset extensively used in sen-
timent analysis, comprising 3228 YouTube videos. These videos are
divided into 23,453 segments and encompass data from three dis-
tinct modalities: text, visual, and sound. With contributions from 1000
speakers and coverage of 250 different topics, this dataset offers a
diverse range of perspectives. All videos in the dataset are in English,
and annotations for both sentiment and emotion are available. The six
emotion categories include happy, sad, angry, scared, disgusted, and
surprised, while the sentiment intensity markers span from strongly
negative to strongly positive (−3 to 3). Overall, CMU-MOSEI stands as
an invaluable resource for researchers delving into sentiment analysis
across multiple modalities.
CMU-MOSEA is a versatile dataset that encompasses multiple lan-
guages, including Spanish, Portuguese, German, and French. The
dataset consists of 40,000 sentence fragments, covering 250 diverse
topics and featuring contributions from 1645 speakers. The annotations
within the dataset are classified into two categories: sentiment inten-
sity and binary. Each sentence fragment is annotated with sentiment
strength, quantified within the interval [−3, 3]. Additionally, the binary
category indicates whether the speaker expressed an opinion or made
an objective statement. For each sentence, emotions are categorized
into six categories: happiness, sadness, fear, disgust, surprise, and
annotation.
2.9. MEMOTION [39]

2.5. MELD [35]
MELD is a comprehensive dataset featuring video clips sourced from
the well-known television series Friends. The dataset encompasses tex-
tual, audio, and video information that aligns with the accompanying
textual data. It consists of 1400 videos, further divided into 13,000
individual segments. The dataset is meticulously annotated with seven
categories of emotions: Anger, Disgust, Sadness, Joy, Neutral, Surprise,
and Fear. Additionally, each segment is annotated with three sentiment
labels: positive, negative, and neutral.
2.6. Multi-ZOL [36]
Multi-ZOL is a dataset designed for bimodal sentiment classification
of images and text. The dataset consists of reviews of mobile phones col-
lected from ZOL.com. It contains 5288 sets of multimodal data points
that cover various models of mobile phones from multiple brands.
These data points are annotated with a sentiment intensity rating from
1 to 10 for six aspects.
2.7. CH-SIMS [37]
CH-SIMS is a distinctive dataset comprising 60 open-sourced videos
sourced from the web. These videos are further divided into 2281 video
clips. The dataset specifically focuses on the Chinese (Mandarin) lan-
guage and ensures that each segment features only one character’s face
and voice. It encompasses a diverse array of scenes and includes speak-
ers spanning various age groups. The dataset is meticulously labeled
for each modality, rendering it a valuable resource for researchers.
The annotations in the dataset encompass sentiment intensity ratings
ranging from negative to positive (−1 to 1), along with additional
annotations for attributes such as age and gender.
MEMOTION is a meme-based dataset that focuses on popular memes
associated with politics, religion, and sports. The dataset comprises
10,000 data points and is organized into three distinct sub-tasks: Senti-
ment Analysis, Emotion Classification, and Scale/Intensity of Emotion
Classes. Annotators approached each data point differently based on
the specific subtask. In subtask one, data points were annotated into
three categories: negative, neutral, and positive. Subtask two involved
annotating each data point into four categories: humor, sarcasm, of-
fense, and motivation. Lastly, in subtask three, each data point was
assigned a sentiment intensity value within the interval of [0, 4]. This
dataset presents a unique opportunity for researchers to explore the use
of memes as a form of communication and expression within modern
culture.
2.10. MULTIZOO & MULTIBENCH [40]
The MULTIBENCH dataset is a comprehensive and diverse collection
of datasets designed for multimodal learning research. It encompasses
15 datasets spanning six distinct research areas, including multimedia,
affective computing, robotics, finance, human–computer interaction,
and healthcare. With more than 10 input modalities available, such
as language, image, video, audio, time-series, tabular data, force sen-
sors, proprioception sensors, sets, and optical flow, the MULTIBENCH
dataset offers a wide range of real-world scenarios for studying mul-
timodal representation learning. Each dataset is carefully curated to
cover various prediction tasks, resulting in a total of 20 prediction
tasks for evaluation. This diversity allows researchers to explore the
integration of different data sources and investigate the performance
and robustness of multimodal models across domains and modali-
ties. The availability of MULTIBENCH dataset provides researchers
with a standardized and reliable benchmark for assessing the capabil-
ities and limitations of multimodal models, promoting accessible and
reproducible research in the field of multimodal learning.

3
S. Lai et al.
2.11. MER [41]
The Multimodal Emotion Recognition Challenge (MER 2023)
dataset is a high-quality benchmark designed to advance research in
multimodal emotion recognition, with a particular focus on system
robustness. Utilized in the MER 2023 challenge, the dataset comprises
three sub-challenges: MER-MULTI, MER-NOISE, and MER-SEMI. In the
MER-MULTI sub-challenge, participants recognized both discrete and
dimensional emotions, allowing for a comprehensive understanding
of emotional labels. The MER-NOISE sub-challenge evaluated sys-
tem robustness in the presence of modality perturbations, while the
MER-SEMI sub-challenge addressed limited labeled data by providing
unlabeled samples for semi-supervised learning. The dataset includes
labeled samples for training and validation, as well as a substantial
amount of unlabeled data. By signing a new End User License Agree-
ment, researchers can access this dataset, contributing to advancements
in multimodal emotion recognition and human–computer interaction.
In conclusion, these sentiment analysis datasets provide valuable
resources for researchers studying sentiment analysis across multiple
modalities. The datasets cover various modalities such as speech, video,
text, physiological signals, and memes, allowing researchers to explore
different aspects of sentiment analysis. They encompass diverse top-
ics, languages, and cultures, providing a comprehensive understanding
of sentiment expression. The datasets are annotated with emotions
and sentiment intensity, enabling researchers to analyze and classify
sentiment in a nuanced manner. Additionally, some datasets focus
on specific challenges, such as system robustness and limited labeled
data, promoting advancements in multimodal emotion recognition.
Overall, these datasets serve as valuable benchmarks for evaluating and
developing multimodal sentiment analysis models.
3. Multimodal fusion
Multimodal data provides a more comprehensive understanding of
objects by incorporating different perspectives, making it more infor-
mative compared to single-modal data. The information derived from
different modalities can complement each other. In the domain of
multimodal sentiment analysis, effectively fusing data features from
diverse modalities, preserving the semantic integrity of each modality,
and achieving robust fusion between them pose significant challenges.
Depending on the specific approach to modal fusion, it can be cat-
egorized into three stages: feature-based multimodal fusion in the
early stage, model-based multimodal fusion in the middle stage, and
decision-based multimodal fusion in the late stage. These stages repre-
sent different strategies for integrating modalities and optimizing the
overall sentiment analysis performance.
3.1. Early feature-based approaches for multimodal fusion
Early feature-based multimodal fusion methods aim to merge the
features of different modalities at a shallow level following feature
extraction in the initial stage. This fusion process involves bringing the
features of different single modalities into a unified parameter space.
However, since different modalities contain varying amounts of in-
formation, the resulting features often contain redundant information.
To address this, dimensionality reduction techniques are commonly
employed to eliminate this redundancy. The reduced features are then
fed into the model for subsequent feature extraction and prediction.
Early feature fusion ensures that the model considers input information
from multiple modalities right from the start of the feature modeling
process.
Nevertheless, unifying multiple parameter spaces in the input layer
might not always yield the desired outcomes due to differences in
the parameter spaces of the various modalities. While these models
effectively handle multimodal emotion recognition tasks with high
accuracy and robustness, they typically require a substantial amount
4
Displays 80 (2023) 102563
Fig. 2. The figure illustrates the overall framework of early feature-based approaches
for multimodal fusion. Once the features are extracted, this model employs dedicated
components to fuse the features from each modality. These components are designed
to integrate the information from different modalities in a cohesive manner, enabling
the model to capture the complementary aspects of the data and create a unified
representation. By fusing the features, the model can effectively leverage the combined
strength of the modalities and enhance the overall performance of the fusion process.
of training data to achieve optimal performance. Additionally, their
structures tend to be complex, resulting in longer training times. Fig. 2
illustrates the overall framework of early feature-based approaches for
multimodal fusion. Some notable representative models in this category
include:
3.1.1. THMM (Tri-modal Hidden Markov Model) [42]
Multimodal sequence modeling and analysis can be approached by
representing eigenvectors of multiple modalities as higher-order tensors
and utilizing tensor decomposition methods to extract hidden states
and transition probabilities. This approach effectively capitalizes on
the correlation and complementarity present in multimodal data, while
mitigating issues such as the curse of dimensionality and overfitting.
However, a drawback of this approach is that the order and rank of the
tensors, as well as the number of hidden states, must be predetermined,
which may potentially impact the model’s performance and efficiency.
It requires careful consideration and tuning to ensure optimal re-
sults and avoid potential limitations associated with predefining these
parameters.
3.1.2. RMFN (Recurrent Multistage Fusion Network) [43]
An alternative approach involves employing multiple recurrent neu-
ral network (RNN) layers to gradually fuse features derived from differ-
ent modalities. This fusion process occurs progressively, transitioning
from local to global information and from low-level to high-level
representations, ultimately yielding a comprehensive sentiment repre-
sentation. In this approach, an attention mechanism is incorporated to
dynamically adjust the relevance of features across semantic space for
each modality. This attention mechanism enables the model to capture
the nuanced variations in emotions exhibited by the same word under
different nonverbal behaviors. By adaptively focusing on the relevant
features, the model can effectively account for the diverse expressions
of sentiment across modalities, enhancing its overall performance in
multimodal sentiment analysis.
3.1.3. RAVEN (Recurrent Attended Variation Embedding Network) [44]
For multimodal sentiment analysis, a Hierarchical Fusion Network
has been proposed, featuring both a local fusion module and a global fu-
sion module. This network architecture aims to enhance the integration
of information across modalities. In particular, local cross-modal fusion
is achieved by employing a sliding window approach. This technique
effectively reduces computational complexity by enabling efficient fu-
sion of multimodal features within localized segments. By leveraging
the sliding window mechanism, the model can capture fine-grained
relationships between modalities while maintaining computational ef-
ficiency, thus contributing to the overall effectiveness of the fusion
process in multimodal sentiment analysis.
S. Lai et al.
Fig. 3. The figure illustrates the overarching framework of medium-term model-based
approaches for multimodal fusion. In this approach, the feature information from each
modality is inputted into multiple kernel learning, neural networks, graph models,
and alternative methods to accomplish the fusion of modalities. Notably, the majority
of the nodes involved in modal fusion are variable, indicating their adaptability and
flexibility in the fusion process. This variability allows the model to effectively integrate
and combine the information from different modalities, tailoring the fusion to suit the
specific characteristics and requirements of the multimodal data being processed.
3.1.4. HFFN(Hierarchical feature fusion network with local and global
perspectives for multimodal affective computing) [45]
An alternative approach involves utilizing RNNs in conjunction with
adversarial learning techniques to learn joint representations across
different modalities. This approach aims to enhance the effectiveness
of single-modal representations while also addressing challenges such
as missing modalities or noise in the data. By leveraging RNNs and in-
corporating adversarial learning, the model can effectively capture the
correlations and dependencies between modalities, leading to improved
multimodal representations. These joint representations not only en-
hance the individual modalities but also enable the model to handle
situations where certain modalities may be missing or corrupted. The
inclusion of adversarial learning further helps the model to learn robust
representations that are resilient to noise or inconsistencies across
modalities.
3.1.5. MCTN (Multimodal Cyclic Translation Network) [46]
A medium-term model-based multimodal fusion approach incorpo-
rates multimodal data into a network, where the intermediate layers
of the model facilitate the fusion of features from different modalities.
In this approach, modality feature fusion is strategically performed at
intermediate stages to enable interactions between modalities. Model-
based fusion methods encompass various techniques, such as multiple
kernel learning, neural networks, graph models, and alternative meth-
ods. These methods are utilized to determine the optimal fusion points
and facilitate the effective integration of information across modalities.
By leveraging model-based fusion, the approach enables comprehensive
and context-aware fusion that enhances the overall performance of
multimodal analysis tasks.
3.2. Medium-term model-based multimodal fusion method
A medium-term model-based multimodal fusion approach involves
inputting multimodal data into the network, where the intermediate
layers of the model carry out feature fusion between the different
modalities. Model-based modality fusion methods offer the flexibil-
ity to determine the optimal location for modality feature fusion,
enabling intermediate interactions. These fusion methods typically en-
compass multiple kernel learning, neural networks, graph models, and
alternative techniques. The overall framework of the medium-term
model-based multimodal fusion method is depicted in Fig. 3.
3.2.1. MKL (Multiple Kernel Learning) [47]
This model employs a multiple kernel learning approach to facil-
itate the fusion of multimodal information. It utilizes distinct kernel
functions to represent the diverse modal information and dynamically
selects an optimal combination of these functions by optimizing an
objective function. This methodology enhances the flexibility of the
5
Displays 80 (2023) 102563
Fig. 4. The figure illustrates the comprehensive framework of a multimodal model
based on decision fusion in the later stages. In this approach, individual classifiers are
trained for each modality. Subsequently, the outputs of these classifiers are integrated to
accomplish the task of multimodal sentiment analysis. By training separate classifiers
for each modality, the model can capture the unique characteristics and nuances of
different modalities. The fusion of the individual classifiers allows for a holistic analysis
that leverages the strengths of each modality, enhancing the overall performance of the
sentiment analysis task.
model, enabling it to adaptively choose different kernel functions based
on the given task. Consequently, this adaptability enhances the ro-
bustness and accuracy of the model. However, it is important to note
that the model structure is relatively straightforward and may have
limitations in handling complex emotional expressions that require
more intricate representations. While the model excels in its flexibility
and adaptability, it may benefit from further enhancements to handle
the intricacies of complex emotional expressions.
3.2.2. BERT-like (self supervised models to improve multimodal speech
emotion recognition) [48]
This model represents a Transformer-based approach for multi-
modal sentiment analysis, enabling the alignment and fusion of text
and image information through the utilization of a self-attention mech-
anism. By incorporating self-supervised learning techniques, the model
demonstrates effectiveness in addressing multimodal emotion recog-
nition tasks with high levels of accuracy and robustness. However, it
is worth noting that the quality of data and annotations can have an
impact on the model’s performance. The model’s ability to generalize
and make accurate predictions relies on the quality and reliability of
the input data as well as the accuracy of the annotations associated
with the multimodal samples. Ensuring high-quality data and precise
annotations is crucial to optimize the performance of the model in
multimodal sentiment analysis tasks.
3.3. Multimodal model based on decision fusion in the later stage
A decision-level fusion method is employed to integrate information
from various modalities in this approach. Decision-level fusion involves
training separate models on data from different modalities and com-
bining their outputs to make the final decision. Multimodal models
based on decision fusion typically employ techniques such as averaging,
majority voting, weighting, and learnable models to fuse the modalities.
These models are known for being lightweight and flexible, allowing
for efficient processing and adaptability. In scenarios where a modality
is missing, the decision can still be reached by utilizing the available
modalities. The overall framework of the multimodal model based on
decision fusion in the later stages is depicted in Fig. 4.
3.3.1. Deep multimodal fusion architecture [49]
In this model, each modality is equipped with its own independent
classifier. The prediction results are obtained by averaging the confi-
dence scores from each classifier. This model is characterized by its
simple structure, ease of implementation, and effectiveness in address-
ing multimodal emotion recognition tasks. However, it is important to
note that the model lacks the ability to capture the interaction infor-
mation between modalities, which may result in the loss of valuable
insights. The absence of explicit mechanisms for capturing modality
interactions can limit the model’s ability to fully exploit the synergis-
tic potential of multimodal data. Consequently, there is a possibility
of overlooking important contextual cues and interdependencies that
could contribute to a more comprehensive understanding of emotions.
S. Lai et al. Displays 80 (2023) 102563

3.3.2. SAL-CNN (Select-Additive Learning CNN) [50] Table 2

This table showcases a selection of recent and highly effective multimodal sentiment
This model is a multimodal sentiment analysis approach that com-
analysis models. The first column lists the model names, while the second column
bines CNN and attention mechanisms. It leverages an adaptive atten- indicates the respective publication years. The third column indicates the dataset
tion mechanism to fuse text and image features, a spatial attention utilized for model evaluation, and the fourth column presents the corresponding
mechanism to extract text-related regions within images, and a fully accuracy achieved on that dataset.
connected layer for classification. The incorporation of attention mech- Name Year Dataset Acc
anisms enables the model to effectively handle multimodal emotion MultiSentiNet-Att 2017 MVSA 68.86%
recognition tasks with high levels of accuracy and robustness. However, CMU-MOSI 80.98%
it is worth noting that the model’s performance is reliant on a substan- DFF-TMF 2019 CMU-MOSEI 77.15%
tial amount of training data to achieve optimal results. Additionally, Flickr 87.10%
due to the relatively complex structure of the model, the training AHRM 2020 Getty Image 87.80%
process may require a longer duration. It is important to consider these SFNN 2020 Yelp 62.90%
factors when implementing and training the model for multimodal MISA 2020 MOSI 83.40%
sentiment analysis tasks.
CMU-MOSI 84.10%
MAG-BERT 2020 CMU-MOSEI 84.50%
3.3.3. TSAM (Temporally Selective Attention Model) [51]
CMU-MOSI 92.28%
The proposed model introduces a time-selective attention mecha- TIMF 2021 CMU-MOSEI 79.46%
nism, which employs an attention mechanism to assign weights and
Auto-ML based Fusion 2021 B-T4SA 95.19%
guide the model in selecting the relevant time step. The selected
CMU-MOSI 84.00%
time step is then passed to a distinct SDL loss function model for CMU-MOSEI 82.81%
Self-MM 2022
sentiment analysis. SDL, known as Self-Distillation Learning, is a multi- CH-SIMS 80.74%
modal sentiment analysis method that leverages the complementar- CMU-MOSI 83.60%
ity among different modalities to enhance the model’s generalization DISRFN 2022 CMU-MOSEI 87.50%
ability and robustness. By integrating a time-selective attention mech- CMU-MOSI 84.05%
anism, the model demonstrates effectiveness in handling multimodal TETFN 2023 CMU-MOSEI 84.25%
emotion recognition tasks with high levels of accuracy and robustness. CMU-MOSI 89.30%
The incorporation of the time-selective attention mechanism allows the TEDT 2023 CMU-MOSEI 86.20%
model to focus on the most relevant temporal information, contributing CMU-MOSI 85.06%
to improved sentiment analysis performance. SPIL 2023 CMU-MOSEI 85.01%
There are three main approaches to multimodal fusion: early CH-SIMS 81.25%
feature-based fusion, medium-term model-based fusion, and decision-
level fusion.
Early feature-based fusion methods merge the features of different
feature fusion at intermediate layers, and decision-level fusion com-
modalities at a shallow level after feature extraction. The goal is to
bines outputs from separate models. Each approach has its advantages
bring the features of different modalities into a unified parameter space.
and considerations in terms of accuracy, robustness, training data
However, this approach often leads to redundant information in the
requirements, complexity, and flexibility.
resulting features due to variations in the amount of information con-
tained in different modalities. To address this, dimensionality reduction
4. Latest multimodal sentiment analysis models
techniques are commonly used to eliminate redundancy. The reduced
features are then fed into the model for further feature extraction and
prediction. Early feature fusion ensures that the model considers input In recent years, the field of multimodal sentiment analysis has
information from multiple modalities right from the start of the feature witnessed significant advancements, leading to the development of nu-
modeling process. However, these models require a substantial amount merous practical and efficient models and architectures. It is important
of training data to achieve optimal performance and tend to have to note that it is not feasible to cover all these models within the
complex structures, resulting in longer training times. scope of this chapter. Therefore, we will focus on presenting some of
Medium-term model-based fusion approaches involve inputting the recent and cutting-edge multimodal sentiment analysis models that
multimodal data into the network, where the intermediate layers of have emerged. These models have served as benchmarks for subsequent
the model perform feature fusion between different modalities. These researchers, providing valuable experimental references. A summary of
methods offer flexibility in determining the optimal location for modal- these models can be found in Table 2. The first column of the table
ity feature fusion, allowing for intermediate interactions. Techniques denotes the model names, while the second column indicates the year
such as multiple kernel learning, neural networks, graph models, and of publication. The third column provides information regarding the
alternative methods are often used in model-based modality fusion. The datasets employed for evaluation, and the fourth column presents the
overall framework of the medium-term model-based fusion method is corresponding accuracy achieved on these datasets.
shown in Fig. 3.
Decision-level fusion methods integrate information from various 4.1. MultiSentiNet-Att [52]
modalities by training separate models on data from different modali-
ties and combining their outputs to make the final decision. Techniques In this model, an LSTM network is employed to integrate textual
such as averaging, majority voting, weighting, and learnable models information with word vectors. Additionally, VGG is utilized to extract
are commonly employed to fuse the modalities. Decision-level fusion both target feature information and scene feature information from an
models are lightweight and flexible, enabling efficient processing and image. These target and scene feature vectors are then utilized in the
adaptability. In scenarios where a modality is missing, the decision cross-modal attention mechanism learning process alongside the word
can still be reached by utilizing the available modalities. The overall vectors. This mechanism assigns specific weights to the word vectors
framework of the multimodal model based on decision fusion in the that are associated with the sentiment expressed in the text, utilizing
later stages is depicted in Fig. 4. a combination of the target and scene feature information from the
In summary, early feature-based fusion focuses on merging fea- image. The resulting features are subsequently fed into a multi-layer
tures at a shallow level, medium-term model-based fusion performs perceptron to accomplish the sentiment analysis task.

6
S. Lai et al.
4.2. DFF-ATMF [53]
The proposed model primarily focuses on the text and audio modal-
ities. Its main contribution lies in the introduction of novel strategies
for multi-feature fusion and multimodal fusion. The model employs
two parallel branches to extract features specifically for the text and
audio modalities. Subsequently, a multimodal attention fusion module
is employed to effectively combine the features from these two modal-
ities, achieving multimodal fusion. This fusion process enhances the
model’s ability to capture complementary information and achieve a
comprehensive understanding of the input data from both the text and
audio sources.
4.3. AHRM [54]
This model primarily aims to capture the relationship between
the text and visual modalities. The authors introduce an attention
mechanism-based heterogeneous relation model, which effectively
combines high-quality representation information from both the text
and visual modalities. The progressive dual attention mechanism em-
ployed in this model effectively emphasizes channel-level semantic
information from both images and text. In order to incorporate social
attributes, social relations are introduced to capture complementary
information from the social context. Furthermore, heterogeneous net-
works are constructed to integrate features from different modalities,
facilitating a comprehensive understanding of the multimodal data.
4.4. SFNN [55]
The proposed model is a neural network based on semantic fea-
ture fusion. Convolutional neural networks and attention mechanisms
are used to extract visual features. Visual features are mapped to
text features and combined with text modality features for sentiment
analysis.
4.5. MISA [56]
The proposed model introduces an innovative framework for multi-
modal sentiment analysis. In this framework, each modality undergoes
feature extraction and is subsequently mapped into two separate fea-
ture spaces. One of these feature spaces primarily focuses on learning
the invariant features of the modality, while the other one empha-
sizes the acquisition of the modality’s unique features. This approach
allows the model to capture both the shared characteristics across
modalities and the distinctive aspects specific to each modality, thereby
facilitating a comprehensive analysis of the multi-modal data.
4.6. MAG-BERT [57]
The authors introduce a ‘‘multi-modal’’ adaptation architecture that
is specifically applied to BERT. This novel model is capable of ac-
commodating input from multiple modalities during the fine-tuning
process. The Multi-modal Adaptation Gateway (MAG) can be conceptu-
alized as a vector embedding structure, enabling the incorporation of
multimodal information and its subsequent embedding as a sequence
within BERT. This adaptation architecture enhances the model’s ability
to process and analyze data from diverse modalities, facilitating a more
comprehensive understanding of multimodal inputs within the BERT
framework.
4.7. TIMF [58]
The main idea of this model is that each modality learns features
separately and performs tensor fusion of the features of each modality.
In the dataset fusion stage, the feature fusion for each modality is
implemented by a tensor fusion network. In the decision fusion stage,
the upstream results are fused by soft fusion to adjust the decision
results.
7
Displays 80 (2023) 102563
4.8. Auto-ML based fusion [59]
The authors propose a novel approach that combines text and image
sentiment analysis through the utilization of AutoML for generating a
final fused classification. In this approach, individual classifiers for text
and image are combined using the best model selected by AutoML. This
decision-level fusion technique merges the outputs of the individual
classifiers to form the final classification. This approach showcases a
typical example of decision-level fusion, where the strengths of both
text and image analysis are leveraged to improve the overall sentiment
analysis performance.
4.9. Self-MM [60]
In this study, the authors propose a novel multi-modal sentiment
analysis architecture that combines self-supervised learning and multi-
task learning. To capture the private information specific to each
modality, a single-modal label generation module called ULGM is con-
structed based on self-supervised learning. The corresponding loss func-
tion for this module is designed to integrate the private features ac-
quired through three self-supervised learning subtasks into the original
multi-modal sentiment analysis model, employing a weight adjustment
strategy. The proposed model demonstrates strong performance, and
the ULGM module based on self-supervised learning also possesses
the capability of calibrating single-modal labels. This combination of
self-supervised learning and multi-task learning proves to be effec-
tive in enhancing the sentiment analysis performance while capturing
modality-specific information.
4.10. DISRFN [61]
The model is a dynamically invariant representation-specific fusion
network. The joint domain separation network is improved to obtain
a joint domain separation representation for all modalities, so that
redundant information can be effectively utilized. Second, a HGFN
network is used to dynamically fuse the feature information of each
modality and learn the features of multiple modal interactions. At the
same time, a loss function that improves the fusion effect is constructed
to help the model learn the representation information of each modality
in the subspace.
4.11. TEDT [62]
This model introduces a novel multimodal encoding-decoding trans-
lation network, incorporating a transformer, to tackle the challenges as-
sociated with multimodal sentiment analysis. Specifically, it addresses
the influence of individual modal data and the low quality of non-
natural language features. The proposed method assigns text as the
primary information, while sound and image serve as secondary in-
formation. To enhance the quality of non-natural language features, a
modality reinforcement cross-attention module is employed to convert
them into natural language features. Furthermore, a dynamic filtering
mechanism is implemented to filter out error information arising from
cross-modal interaction. The model’s key strength lies in its ability to
enhance multimodal fusion and accurately analyze human sentiment.
However, it should be noted that this approach may require signifi-
cant computational resources and might not be suitable for real-time
analysis.
4.12. TETFN [63]
The Text Enhanced Transformer Fusion Network (TETFN) presents a
novel approach to MSA that effectively addresses the challenges posed
S. Lai et al.
by the varying contributions of textual, visual, and acoustic modalities.
This method focuses on learning text-oriented pairwise cross-modal
mappings to acquire unified and robust multimodal representations.
It achieves this by incorporating textual information into the learning
process of sentiment-related nonlinguistic representations through text-
based multi-head attention. Additionally, the model incorporates uni-
modal label prediction to maintain differentiated information among
the various modalities. To capture both global and local information
of a human face, the vision pre-trained model Vision-Transformer is
employed to extract visual features from the original videos. The key
strength of this model lies in its ability to leverage textual information
to enhance the effectiveness of nonlinguistic modalities in MSA, while
simultaneously preserving inter- and intra-modality relationships.
4.13. SPIL [64]
This model introduces a deep modal shared information learning
module designed to enhance representation learning in multimodal
sentiment analysis tasks. The proposed module effectively captures
both shared and private information within a comprehensive modal
representation. It achieves this by utilizing a covariance matrix to
capture shared information across modalities, while employing a self-
supervised learning strategy to capture private information specific to
each modality. The module is flexible and can be seamlessly integrated
into existing frameworks, allowing for adjustable information exchange
between modalities to learn either private or shared information as
needed. Additionally, a multi-task learning strategy is implemented to
guide the model’s attention towards modal differentiation during train-
ing. The proposed model demonstrates superior performance compared
to current state-of-the-art methods across various metrics on three
publicly available datasets. Furthermore, the study explores additional
combinatorial techniques for leveraging the capabilities of the proposed
module.
5. Introduction to evaluation metrics
In the field of multimodal sentiment analysis, evaluating the per-
formance of sentiment analysis models is crucial for assessing their
effectiveness. Various evaluation metrics have been developed to mea-
sure the performance of these models, providing insights into their
accuracy, robustness, and generalizability. This chapter aims to intro-
duce and discuss the commonly used evaluation metrics in multimodal
sentiment analysis.
5.1. Accuracy
Accuracy is a fundamental evaluation metric that measures the
overall correctness of the sentiment predictions made by a model.
It calculates the ratio of correctly classified instances to the total
number of instances in the dataset. While accuracy provides a general
understanding of the model’s performance, it may not be sufficient for
assessing the model’s performance when class imbalances exist in the
dataset.
5.2. Precision and recall
Precision and recall are evaluation metrics commonly used in binary
sentiment classification tasks. Precision measures the proportion of
correctly predicted positive instances out of all instances predicted
as positive. Recall, on the other hand, calculates the proportion of
correctly predicted positive instances out of all actual positive instances
in the dataset. Precision and recall provide insights into the model’s
ability to correctly identify positive instances and avoid false positives.
8
Displays 80 (2023) 102563
5.3. F1 score
The F1 score is a metric that combines precision and recall into
a single value, providing a balanced evaluation of the model’s per-
formance. It is the harmonic mean of precision and recall and is
particularly useful when there is an imbalance between positive and
negative instances in the dataset. The F1 score ranges from 0 to 1, with
higher values indicating better performance.
5.4. Mean Absolute Error (MAE)
The mean absolute error is a metric commonly used in regression-
based sentiment analysis tasks. It measures the average absolute dif-
ference between the predicted sentiment values and the true sentiment
values. MAE provides a measure of the model’s accuracy in predicting
sentiment on a continuous scale.
5.5. Mean Squared Error (MSE)
Similar to MAE, mean squared error is another metric used in
regression-based sentiment analysis tasks. It calculates the average
squared difference between the predicted sentiment values and the true
sentiment values. MSE amplifies the impact of large errors, making it
more sensitive to outliers in the predictions.
5.6. Cross-entropy loss
Cross-entropy loss is a common metric used in sentiment analysis
tasks that involve probabilistic models, such as deep learning mod-
els. It measures the dissimilarity between the predicted probability
distribution and the true probability distribution of sentiment labels.
Minimizing cross-entropy loss encourages the model to produce more
accurate and confident predictions.
5.7. Cohen’s Kappa
Cohen’s Kappa is a statistical measure of inter-rater agreement, com-
monly used when evaluating sentiment analysis models in scenarios
where multiple annotators label the same dataset. It takes into account
the agreement between the model’s predictions and the average agree-
ment expected by chance. Cohen’s Kappa provides insights into the
model’s performance beyond chance agreement.
By using these evaluation metrics, researchers and practitioners can
assess the performance of multimodal sentiment analysis models and
compare their effectiveness. It is important to consider the specific
characteristics of the dataset and the task at hand when selecting
appropriate evaluation metrics to ensure a comprehensive evaluation
of model performance.
6. Challenges and future scope
The rapid development of deep learning has led to significant
advancements in multimodal sentiment analysis techniques [65–68].
However, despite these advancements, there are still numerous chal-
lenges that need to be addressed in the field of multimodal sentiment
analysis. Therefore, this subsection aims to analyze the current state
of research, identify the challenges faced, and explore potential future
developments in multimodal sentiment analysis.
6.1. Dataset
The dataset is a critical component in multimodal sentiment anal-
ysis. However, there is currently a lack of large-scale datasets that
S. Lai et al.
encompass multiple languages. Considering the linguistic and racial
diversity in many countries, it is crucial to have a large and diverse
dataset that can be used to train multimodal sentiment analysis models
with robust generalization capabilities and wide applicability. More-
over, existing multimodal datasets suffer from low annotation accuracy
and have not yet achieved absolute continuous values. To address this
issue, researchers need to focus on more precise labeling of multimodal
datasets. Furthermore, most current multimodal datasets only incorpo-
rate visual, speech, and text modalities, while neglecting the inclusion
of modal information combined with physiological signals such as brain
waves and pulses. Thus, there is a need to expand multimodal datasets
to include these additional modalities for more comprehensive analysis.
6.2. Detection of hidden emotions
Multimodal sentiment analysis tasks have long faced a recognized
challenge: the analysis of hidden emotions. Hidden emotions encom-
pass various aspects, including sarcastic emotions expressed through
the use of sarcastic words, emotions that require contextual analysis
for concrete understanding, and complex emotions such as happiness
and sadness experienced by individuals [69,70]. The exploration of
these hidden emotions is crucial as it represents a gap between human
and artificial intelligence capabilities in understanding and interpreting
emotions [71,72]. By delving into these hidden emotions, we can
bridge this gap and enhance the effectiveness of multimodal sentiment
analysis.
6.3. Multiple forms of video data
Video data presents unique challenges in multimodal sentiment
analysis tasks. Despite the speaker facing the camera and the high-
resolution quality of the video data, the actual situation is often more
complex, necessitating a model that is resilient to noise and capa-
ble of analyzing low-resolution video data. Furthermore, there is an
opportunity for researchers to delve into the capture and analysis of
micro-expressions and micro-gestures exhibited by speakers, as these
subtle cues can provide valuable insights for sentiment analysis. There-
fore, exploring techniques to address these challenges and harness the
potential of micro-expressions and micro-gestures is an important area
of focus for researchers in the field.
6.4. Multiform language data
In multimodal sentiment analysis tasks, text data is typically uni-
modal, focusing on a single language. However, in online communities,
evaluation texts often involve multiple languages, as reviewers em-
ploy different languages to express their comments more vividly. This
presents a challenge in analyzing text data with mixed emotions within
a multimodal context. Effectively utilizing memes that are embedded in
the text is an important research area, as memes often convey highly
impactful emotional messages from reviewers. Moreover, analyzing
emotions becomes more challenging when multiple individuals are
engaged in a conversation, as most text data is transcribed directly
from speech. Furthermore, considering the cultural characteristics of
different regions and countries, the same text data may evoke diverse
emotions. Therefore, understanding and addressing these complexi-
ties associated with cross-lingual text data, incorporating memes, and
accounting for cultural nuances are crucial areas of investigation in
multimodal sentiment analysis research.
6.5. Future prospects
The future of multimodal sentiment analysis techniques holds great
promise, with several potential applications on the horizon. Some
9
Displays 80 (2023) 102563
of these applications include multimodal emotion analysis for real-
time mental health assessment [73–75], multimodal criminal linguistic
deception detection models [76], and offensive language detection.
Another exciting prospect is the development of human-like emotion-
aware robots. Multimodal emotion analysis is a powerful technique
for recognizing and analyzing emotions. By leveraging multimodal
information and combining data from various modalities, sentiment
analysis models can significantly enhance the accuracy of emotion
recognition. As research progresses, multimodal sentiment analysis
techniques will continue to improve. It is conceivable that one day
we may witness the emergence of a multimodal sentiment analysis
model with a large number of parameters, possessing sentiment analysis
capabilities comparable to those of humans. Such a development would
be truly remarkable and pave the way for exciting advancements in the
field.
7. Conclusion
Multimodal sentiment analysis has garnered significant recogni-
tion in various research fields, establishing itself as a central topic
in natural language processing and computer vision. In this compre-
hensive review, we provide an in-depth exploration of various facets
of multimodal sentiment analysis, covering its research background,
definition, and developmental trajectory. Additionally, we present a
comprehensive overview of commonly used benchmark datasets in
Table 1, and conduct a comparative analysis of recent state-of-the-
art multimodal sentiment analysis models. Furthermore, we highlight
the existing challenges within the field and delve into potential future
developments.
Numerous promising endeavors are currently underway, with sev-
eral already implemented. However, there remain several challenges
that necessitate further investigation, giving rise to significant research
directions:
(1) Constructing large-scale multimodal sentiment datasets in mul-
tiple languages.
(2) Addressing the domain transfer problem across video, text, and
speech modal data.
(3) Developing a unified, large-scale multimodal sentiment analysis
model with exceptional generalization capabilities.
(4) Reducing model parameters, optimizing algorithms, and mini-
mizing algorithmic complexity.
(5) Tackling the challenges posed by multi-lingual hybridity in
multimodal sentiment analysis.
(6) Exploring the weighting problem in modal fusion and devising
rational weighting schemes for different modalities based on specific
cases.
(7) Investigating the inter-modal correlations and distinguishing
between shared and private information to enhance model performance
and interpretability.
(8) Constructing a multimodal sentiment analysis model capable of
effectively capturing and analyzing hidden emotions.
Addressing these research directions will undoubtedly contribute to
the advancement of multimodal sentiment analysis and enable the de-
velopment of more powerful and comprehensive models in the future.
Funding
This work was supported in part by Joint found for smart computing
of Shandong Natural Science Foundation under Grant ZR2020LZH013;
open project of State Key Laboratory of Computer Architecture
CARCHA202001; the Major Scientific and Technological Innovation
Project in Shandong Province under Grant 2021CXG010506 and
2022CXG010504; ‘‘New University 20 items’’ Funding Project of Jinan
under Grant 2021GXRC108 and 2021GXRC024.
S. Lai et al.
Declaration of competing interest
The authors declare no conflict of interest. This research was con-
ducted in an unbiased manner and the results presented in this paper
are based solely on the data and analysis conducted during the research
process. No financial or other relationships exist that could be perceived
as a conflict of interest.
Data availability
No data was used for the research described in the article.
References
[1] Julien Deonna, Fabrice Teroni, The Emotions: A Philosophical Introduction,
Routledge, 2012.
[2] Clayton Hutto, Eric Gilbert, Vader: A parsimonious rule-based model for sen-
timent analysis of social media text, in: Proceedings of the International AAAI
Conference on Web and Social Media, vol. 8, (no. 1) 2014, pp. 216–225.
[3] Soo-Min Kim, Eduard Hovy, Determining the sentiment of opinions, in: COLING
2004: Proceedings of the 20th International Conference on Computational
Linguistics, 2004, pp. 1367–1373.
[4] Erik Cambria, Björn Schuller, Yunqing Xia, Catherine Havasi, New avenues in
opinion mining and sentiment analysis, IEEE Intell. Syst. 28 (2) (2013) 15–21.
[5] Arshi Parvaiz, Muhammad Anwaar Khalid, Rukhsana Zafar, Huma Ameer,
Muhammad Ali, Muhammad Moazam Fraz, Vision transformers in medical
computer vision—A contemplative retrospection, Eng. Appl. Artif. Intell. 122
(2023) 106126.
[6] Bo Zhang, Jun Zhu, Hang Su, Toward the third generation artificial intelligence,
Sci. China Inf. Sci. 66 (2) (2023) 1–19.
[7] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham
Neubig, Pre-train, prompt, and predict: A systematic survey of prompting
methods in natural language processing, ACM Comput. Surv. 55 (9) (2023) 1–35.
[8] Jireh Yi-Le Chan, Khean Thye Bea, Steven Mun Hong Leow, Seuk Wai Phoong,
Wai Khuen Cheng, State of the art: A review of sentiment analysis based on
sequential transfer learning, Artif. Intell. Rev. 56 (1) (2023) 749–780.
[9] Mayur Wankhade, Annavarapu Chandra Sekhara Rao, Chaitanya Kulkarni, A
survey on sentiment analysis methods, applications, and challenges, Artif. Intell.
Rev. 55 (7) (2022) 5731–5780.
[10] Hui Li, Qi Chen, Zhaoman Zhong, Rongrong Gong, Guokai Han, E-word of mouth
sentiment analysis for user behavior studies, Inf. Process. Manage. 59 (1) (2022)
102784.
[11] Ashima Yadav, Dinesh Kumar Vishwakarma, Sentiment analysis using deep
learning architectures: A review, Artif. Intell. Rev. 53 (6) (2020) 4335–4385.
[12] Ganesh Chandrasekaran, Tu N. Nguyen, Jude Hemanth D, Multimodal senti-
mental analysis for social media applications: A comprehensive review, Wiley
Interdiscipl. Rev.: Data Mining Knowl. Discov. 11 (5) (2021) e1415.
[13] Bernhard Kratzwald, Suzana Ilic, Mathias Kraus, Stefan Feuerriegel, Helmut
Prendinger, Decision support with text-based emotion recognition: Deep learning
for affective computing, 2018, arXiv preprint arXiv:1803.06397.
[14] Carlo Strapparava, Rada Mihalcea, Semeval-2007 task 14: Affective text, in:
Proceedings of the Fourth International Workshop on Semantic Evaluations,
SemEval-2007, 2007, pp. 70–74.
[15] Yang Li, Quan Pan, Suhang Wang, Tao Yang, Erik Cambria, A generative model
for category text generation, Inform. Sci. 450 (2018) 301–315.
[16] Rong Dai, Facial expression recognition method based on facial physiological
features and deep learning, J. Chongqing Univ. Technol. (Natural Science) 34
(6) (2020) 146–153.
[17] Zhu Ren, Jia Jia, Quan Guo, Kuo Zhang, Lianhong Cai, Acoustics, content and
geo-information based sentiment prediction from large-scale networked voice
data, in: 2014 IEEE International Conference on Multimedia and Expo, ICME,
IEEE, 2014, pp. 1–4.
[18] Liu Jiming, Zhang Peixiang, Liu Ying, Zhang Weidong, Fang Jie, Summary of
multi-modal sentiment analysis technology, J. Front. Comput. Sci. Technol. 15
(7) (2021) 1165.
[19] Feiran Huang, Xiaoming Zhang, Zhonghua Zhao, Jie Xu, Zhoujun Li, Image–text
sentiment analysis via deep multimodal attentive fusion, Knowl.-Based Syst. 167
(2019) 26–37.
[20] Akshi Kumar, Geetanjali Garg, Sentiment analysis of multimodal twitter data,
Multimedia Tools Appl. 78 (2019) 24103–24119.
[21] Ankita Gandhi, Kinjal Adhvaryu, Vidhi Khanduja, Multimodal sentiment analysis:
review, application domains and future directions, in: 2021 IEEE Pune Section
International Conference, PuneCon, IEEE, 2021, pp. 1–5.
[22] Vaibhav Rupapara, Furqan Rustam, Hina Fatima Shahzad, Arif Mehmood, Imran
Ashraf, Gyu Sang Choi, Impact of SMOTE on imbalanced text features for toxic
comments classification using RVVC model, IEEE Access 9 (2021) 78621–78634.
10
Displays 80 (2023) 102563
[23] Jia Li, Ziyang Zhang, Junjie Lang, Yueqi Jiang, Liuwei An, Peng Zou, Yangyang
Xu, Sheng Gao, Jie Lin, Chunxiao Fan, et al., Hybrid multimodal feature
extraction, mining and fusion for sentiment analysis, in: Proceedings of the 3rd
International on Multimodal Sentiment Analysis Workshop and Challenge, 2022,
pp. 81–88.
[24] Anna Favaro, Chelsie Motley, Tianyu Cao, Miguel Iglesias, Ankur Butala,
Esther S. Oh, Robert D. Stevens, Jesús Villalba, Najim Dehak, Laureano Moro-
Velázquez, A multi-modal array of interpretable features to evaluate language
and speech patterns in different neurological disorders, in: 2022 IEEE Spoken
Language Technology Workshop, SLT, IEEE, 2023, pp. 532–539.
[25] Soujanya Poria, Erik Cambria, Rajiv Bajpai, Amir Hussain, A review of affective
computing: From unimodal analysis to multimodal fusion, Inf. Fusion 37 (2017)
98–125.
[26] Garima Sharma, Abhinav Dhall, A survey on automatic multimodal emotion
recognition in the wild, Adv. Data Sci.: Methodol. Appl. (2021) 35–64.
[27] Xin Gu, Yinghua Shen, Jie Xu, Multimodal emotion recognition in deep learning:
A survey, in: 2021 International Conference on Culture-Oriented Science &
Technology, ICCST, IEEE, 2021, pp. 77–82.
[28] Sathyan Munirathinam, Industry 4.0: Industrial internet of things (IIOT), in:
Advances in Computers, vol. 117, (no. 1) Elsevier, 2020, pp. 129–164.
[29] Esteban Ortiz-Ospina, Max Roser, The rise of social media, Our World Data
(2023).
[30] Abdul Haseeb, Enjun Xia, Shah Saud, Ashfaq Ahmad, Hamid Khurshid, Does
information and communication technologies improve environmental quality in
the era of globalization? An empirical analysis, Environ. Sci. Pollut. Res. 26
(2019) 8594–8608.
[31] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower,
Samuel Kim, Jeannette N. Chang, Sungbok Lee, Shrikanth S. Narayanan, IEMO-
CAP: Interactive emotional dyadic motion capture database, Lang. Resourc. Eval.
42 (2008) 335–359.
[32] Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan
Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, Ioannis Patras, Deap: A
database for emotion analysis; Using physiological signals, IEEE Trans. Affect.
Comput. 3 (1) (2011) 18–31.
[33] Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency, Mosi: Multi-
modal corpus of sentiment intensity and subjectivity analysis in online opinion
videos, 2016, arXiv preprint arXiv:1606.06259.
[34] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-
Philippe Morency, Multimodal language analysis in the wild: Cmu-mosei dataset
and interpretable dynamic fusion graph, in: Proceedings of the 56th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), 2018, pp. 2236–2246.
[35] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik
Cambria, Rada Mihalcea, Meld: A multimodal multi-party dataset for emotion
recognition in conversations, 2018, arXiv preprint arXiv:1810.02508.
[36] Nan Xu, Wenji Mao, Guandan Chen, Multi-interactive memory network for aspect
based multimodal sentiment analysis, in: Proceedings of the AAAI Conference on
Artificial Intelligence, vol. 33, (no. 01) 2019, pp. 371–378.
[37] Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun
Zou, Kaicheng Yang, Ch-sims: A Chinese multimodal sentiment analysis dataset
with fine-grained annotation of modality, in: Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics, 2020, pp. 3718–3727.
[38] Amir Zadeh, Yan Sheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria,
Louis-Philippe Morency, CMU-MOSEAS: A multimodal language dataset for
Spanish, Portuguese, German and French, in: Proceedings of the Conference
on Empirical Methods in Natural Language Processing. Conference on Empirical
Methods in Natural Language Processing, vol. 2020, NIH Public Access, 2020,
p. 1801.
[39] Sathyanarayanan Ramamoorthy, Nethra Gunti, Shreyash Mishra, S. Suryavardan,
Aishwarya Reganti, Parth Patwa, Amitava DaS, Tanmoy Chakraborty, Amit Sheth,
Asif Ekbal, et al., Memotion 2: Dataset on sentiment and emotion analysis of
memes, in: Proceedings of de-Factify: Workshop on Multimodal Fact Checking
and Hate Speech Detection, CEUR, 2022.
[40] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Arav Agarwal, Yun Cheng, Louis-Philippe
Morency, Ruslan Salakhutdinov, MULTIZOO & multibench: A standardized
toolkit for multimodal deep learning, J. Mach. Learn. Res. 24 (2023) 1–7.
[41] Zheng Lian, Haiyang Sun, Licai Sun, Jinming Zhao, Ye Liu, Bin Liu, Jiangyan
Yi, Meng Wang, Erik Cambria, Guoying Zhao, et al., MER 2023: Multi-label
learning, modality robustness, and semi-supervised learning, 2023, arXiv preprint
arXiv:2304.08981.
[42] Louis-Philippe Morency, Rada Mihalcea, Payal Doshi, Towards multimodal sen-
timent analysis: Harvesting opinions from the web, in: Proceedings of the 13th
International Conference on Multimodal Interfaces, 2011, pp. 169–176.
[43] Paul Pu Liang, Ziyin Liu, Amir Zadeh, Louis-Philippe Morency, Multimodal
language analysis with recurrent multistage fusion, 2018, arXiv preprint arXiv:
1808.03920.
[44] Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe
Morency, Words can shift: Dynamically adjusting word representations using
nonverbal behaviors, in: Proceedings of the AAAI Conference on Artificial
Intelligence, vol. 33, (no. 01) 2019, pp. 7216–7223.
S. Lai et al.
[45] Sijie Mai, Haifeng Hu, Songlong Xing, Divide, conquer and combine: Hierarchical
feature fusion network with local and global perspectives for multimodal affec-
tive computing, in: Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, 2019, pp. 481–492.
[46] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, Barnabás
Póczos, Found in translation: Learning robust joint representations by cyclic
translations between modalities, in: Proceedings of the AAAI Conference on
Artificial Intelligence, vol. 33, (no. 01) 2019, pp. 6892–6899.
[47] Soujanya Poria, Erik Cambria, Alexander Gelbukh, Deep convolutional neural
network textual features and multiple kernel learning for utterance-level multi-
modal sentiment analysis, in: Proceedings of the 2015 Conference on Empirical
Methods in Natural Language Processing, 2015, pp. 2539–2544.
[48] Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, Suranga
Nanayakkara, Jointly fine-tuning" bert-like" self supervised models to improve
multimodal speech emotion recognition, 2020, arXiv preprint arXiv:2008.06682.
[49] Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis,
Louis-Philippe Morency, Deep multimodal fusion for persuasiveness prediction,
in: Proceedings of the 18th ACM International Conference on Multimodal
Interaction, 2016, pp. 284–288.
[50] Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, Eric P. Xing, Select-
additive learning: Improving generalization in multimodal sentiment analysis, in:
2017 IEEE International Conference on Multimedia and Expo, ICME, IEEE, 2017,
pp. 949–954.
[51] Hongliang Yu, Liangke Gui, Michael Madaio, Amy Ogan, Justine Cassell, Louis-
Philippe Morency, Temporally selective attention model for social and affective
state recognition in multimedia content, in: Proceedings of the 25th ACM
International Conference on Multimedia, 2017, pp. 1743–1751.
[52] Nan Xu, Wenji Mao, Multisentinet: A deep semantic network for multimodal sen-
timent analysis, in: Proceedings of the 2017 ACM on Conference on Information
and Knowledge Management, 2017, pp. 2399–2402.
[53] Feiyang Chen, Ziqian Luo, Yanyan Xu, Dengfeng Ke, Complementary fusion of
multi-features and multi-modalities in sentiment analysis, 2019, arXiv preprint
arXiv:1904.08138.
[54] Jie Xu, Zhoujun Li, Feiran Huang, Chaozhuo Li, S. Yu Philip, Social image
sentiment analysis by exploiting multimodal content and heterogeneous relations,
IEEE Trans. Ind. Inform. 17 (4) (2020) 2974–2982.
[55] Weidong Wu, Yabo Wang, Shuning Xu, Kaibo Yan, SFNN: Semantic features
fusion neural network for multimodal sentiment analysis, in: 2020 5th Inter-
national Conference on Automation, Control and Robotics Engineering, CACRE,
IEEE, 2020, pp. 661–665.
[56] Devamanyu Hazarika, Roger Zimmermann, Soujanya Poria, Misa: Modality-
invariant and-specific representations for multimodal sentiment analysis, in:
Proceedings of the 28th ACM International Conference on Multimedia, 2020,
pp. 1122–1131.
[57] Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao,
Louis-Philippe Morency, Ehsan Hoque, Integrating multimodal information in
large pretrained transformers, in: Proceedings of the Conference. Association for
Computational Linguistics. Meeting, vol. 2020, NIH Public Access, 2020, p. 2359.
[58] Jianguo Sun, Hanqi Yin, Ye Tian, Junpeng Wu, Linshan Shen, Lei Chen, Two-
level multimodal fusion for sentiment analysis in public security, Secur. Commun.
Netw. 2021 (2021) 1–10.
[59] Vasco Lopes, António Gaspar, Luís A Alexandre, João Cordeiro, An automl-based
approach to multimodal image sentiment analysis, in: 2021 International Joint
Conference on Neural Networks, IJCNN, IEEE, 2021, pp. 1–9.
11
Displays 80 (2023) 102563
[60] Wenmeng Yu, Hua Xu, Ziqi Yuan, Jiele Wu, Learning modality-specific rep-
resentations with self-supervised multi-task learning for multimodal sentiment
analysis, in: Proceedings of the AAAI Conference on Artificial Intelligence, vol.
35, (no. 12) 2021, pp. 10790–10797.
[61] Jing He, Haonan Yanga, Changfan Zhang, Hongrun Chen, Yifu Xua, Dy-
namic invariant-specific representation fusion network for multimodal sentiment
analysis, Comput. Intell. Neurosci. 2022 (2022).
[62] Fan Wang, Shengwei Tian, Long Yu, Jing Liu, Junwen Wang, Kun Li, Yongtao
Wang, TEDT: Transformer-based encoding–decoding translation network for
multimodal sentiment analysis, Cogn. Comput. 15 (1) (2023) 289–303.
[63] Di Wang, Xutong Guo, Yumin Tian, Jinhui Liu, LiHuo He, Xuemei Luo, TETFN:
A text enhanced transformer fusion network for multimodal sentiment analysis,
Pattern Recognit. 136 (2023) 109259.
[64] Songning Lai, Xifeng Hu, Yulong Li, Zhaoxia Ren, Zhi Liu, Danmin Miao, Shared
and private information learning in multimodal sentiment analysis with deep
modal alignment and self-supervised multi-task learning, 2023, arXiv preprint
arXiv:2305.08473.
[65] Mahesh G. Huddar, Sanjeev S. Sannakki, Vijay S. Rajpurohit, A survey of
computational approaches and challenges in multimodal sentiment analysis, Int.
J. Comput. Sci. Eng. 7 (1) (2019) 876–883.
[66] Ramandeep Kaur, Sandeep Kautish, Multimodal sentiment analysis: A survey and
comparison, Res. Anthol. Implement. Sentiment Anal. Across Multiple Discipl.
(2022) 1846–1870.
[67] Lukas Stappen, Alice Baird, Lea Schumann, Schuller Bjorn, The multimodal
sentiment analysis in car reviews (muse-car) dataset: Collection, insights and
improvements, IEEE Trans. Affect. Comput. (2021).
[68] Anurag Illendula, Amit Sheth, Multimodal emotion classification, in: Companion
Proceedings of the 2019 World Wide Web Conference, 2019, pp. 439–449.
[69] Donglei Tang, Zhikai Zhang, Yulan He, Chao Lin, Deyu Zhou, Hidden topic–
emotion transition model for multi-level social emotion detection, Knowl.-Based
Syst. 164 (2019) 426–435.
[70] Petr Hajek, Aliaksandr Barushka, Michal Munk, Fake consumer review detection
using deep neural networks integrating word embeddings and emotion mining,
Neural Comput. Appl. 32 (2020) 17259–17274.
[71] Soonil Kwon, A CNN-assisted enhanced audio signal processing for speech
emotion recognition, Sensors 20 (1) (2019) 183.
[72] Umar Rashid, Muhammad Waseem Iqbal, Muhammad Akmal Skiandar, Muham-
mad Qasim Raiz, Muhammad Raza Naqvi, Syed Khuram Shahzad, Emotion
detection of contextual text using deep learning, in: 2020 4th International
Symposium on Multidisciplinary Studies and Innovative Technologies, ISMSIT,
IEEE, 2020, pp. 1–5.
[73] Zhentao Xu, Verónica Pérez-Rosas, Rada Mihalcea, Inferring social media users’
mental health status from multimodal information, in: Proceedings of the 12th
Language Resources and Evaluation Conference, 2020, pp. 6292–6299.
[74] Rahee Walambe, Pranav Nayak, Ashmit Bhardwaj, Ketan Kotecha, Employing
multimodal machine learning for stress detection, J. Healthc. Eng. 2021 (2021)
1–12.
[75] Nujud Aloshban, Anna Esposito, Alessandro Vinciarelli, What you say or how you
say it? Depression detection through joint modeling of linguistic and acoustic
aspects of speech, Cogn. Comput. 14 (5) (2022) 1585–1598.
[76] Safa Chebbi, Sofia Ben Jebara, Deception detection using multimodal fusion
approaches, Multimedia Tools Appl. (2021) 1–30.