Neural Computing and Applications (2021) 33:8335–8354

https://doi.org/10.1007/s00521-020-05587-y (0123456789().,-volV)
(0123456789().,-volV)

ORIGINAL ARTICLE

Student Class Behavior Dataset: a video dataset for recognizing,

detecting, and captioning students’ behaviors in classroom scenes
Bo Sun1 • Yong Wu1 • Kaijie Zhao1 • Jun He1 • Lejun Yu1 • Huanqing Yan1 • Ao Luo1

Received: 9 May 2020 / Accepted: 11 December 2020 / Published online: 30 January 2021
Ó The Author(s), under exclusive licence to Springer-Verlag London Ltd. part of Springer Nature 2021

Abstract
The massive increase in classroom video data enables the possibility of utilizing artificial intelligence technology to
automatically recognize, detect and caption students’ behaviors. This is beneficial for related research, e.g., pedagogy and
educational psychology. However, the lack of a dataset specifically designed for students’ classroom behaviors may block
these potential studies. This paper presents a comprehensive dataset that can be employed for recognizing, detecting, and
captioning students’ behaviors in a classroom. We collected videos of 128 classes in different disciplines and in 11
classrooms. Specifically, the constructed dataset consists of a detection part, recognition part, and captioning part. The
detection part includes a temporal detection data module with 4542 samples and an action detection data module with 3343
samples, whereas the recognition part contains 4276 samples and the captioning part contains 4296 samples. Moreover, the
students’ behaviors are spontaneous in real classes, rendering the dataset representative and realistic. We analyze the
special characteristics of the classroom scene and the technical difficulties for each module (task), which are verified by
experiments. Due to the particularity of classrooms, our datasets proposes increasing the requirements of existing methods.
Moreover, we provide a baseline for each task module in the dataset and make a comparison with the current mainstream
datasets. The results show that our dataset is viable and reliable. Additionally, we present a thorough performance analysis
of each baseline model to provide a comprehensive comparison for models using our presented dataset. The dataset and
code are available to download online: https://github.com/BNU-Wu/Student-Class-Behavior-Dataset/tree/master.

Keywords Student Class Behavior  Video dataset  Recognition dataset  Detection dataset  Caption dataset 
Baseline

1 Introduction
The rapid development of artificial intelligence (AI),
especially deep learning, has led to significant development
in the field of computer vision, and this has led to a wide
range of applications, such as object recognition [1–3],
object detection [4–7], object tracking [8–10], video
retrieval [11–16], and visual question answering (VQA)
[17]. Algorithms generally perform well on simple data-
sets, such as the UCF101 [18], HMDB [19] and MSVD
[20] datasets. However, owing to the complexity and
diversity of real life, models that perform well on simple
& Jun He
hejun@bnu.edu.cn
1
School of Artificial Intelligence, Beijing Normal University,
Xinjiekouwai Street No. 19, Beijing, China
datasets cannot meet real-world needs. Such needs and the
challenges associated with the algorithms used have
attracted the interest of researchers. Therefore, establishing
datasets that can provide a basis for research has gained
importance. Currently, remarkable progress has been made
in building increasingly complex and realistic datasets,
such as the AVA [21] and VATEX [22] datasets. However,
there still exists a lack of datasets in education, and this
strongly limits AI development in this field. Moreover,
since education scenarios are complex, algorithms face
many new challenges. Based on these findings, this study
establishes a comprehensive dataset for the field of
education.
Evaluations of education quality have attracted an
increasing amount of attention from researchers in fields
such as pedagogy and psychology. As a basic teaching
form, classroom teaching has always been the core of

123
8336
education. As part of a certain scenario, students’ behaviors
in a classroom are significant and not disregarded.
Acquiring information about student behaviors is not only
helpful for mastering students’ learning, personality and
psychological traits but also worthy of inclusion in evalu-
ations of education quality.
While many studies have addressed behavior recogni-
tion or detection, they are time-consuming or inefficient.
With the advent of the era of big data, classroom video data
are often available, e.g., at Beijing Normal University,
3.4 TB of videos are recorded every day in 244 classrooms,
thereby enabling advanced AI technology to automatically
recognize, detect and caption students’ behaviors. While
research on education using AI is still developing, research
on and prospective application of this technology are
worthwhile. Currently, a specific dataset of students’
classroom behaviors, which is necessary and primal for
deep learning, is not available.
As the main goal of computer vision, understanding
visual scenes involves many tasks, including recognition,
detection, and captioning. Although several datasets exist
for each task, few datasets exist simultaneously, especially
in the field of education. Clearly, a class contains multiple
subjects, i.e., students often simultaneously perform dif-
ferent types of actions that are often visually similar. The
tasks of automatically recognizing, detecting, and even
captioning these actions clearly pose strong challenges to
most traditional approaches. Thus, special research via
deep learning has considerable potential.
Since a dataset is primal for deep learning research, we
present a student class behavior dataset in this paper. This
dataset is a video collection of students’ spontaneous
actions in real classes obtained from 4.5 K student
behavior video clips from different disciplines and differ-
ent educational stages. The actions are divided into 11
typical behaviors, including taking notes, using computers,
and discussing. Each behavior has a set of four corre-
sponding descriptive sentences that are manually cap-
tioned. The dataset contains three submodules that depend
on different tasks, namely, (1) trimmed videos for recog-
nition; (2) untrimmed videos for detection; and (3) cap-
tioned tasks based on the recognition task. In addition to
providing the baseline algorithms, we conduct a compar-
ative experiment with current mainstream datasets to val-
idate its challenges.
Aimed at further education research, describing stu-
dents’ behaviors are necessary, for which detecting sub-
jects and their actions are always helpful. A video in a
classroom always contains multiple students who perform
different types of actions. The spatial detection of subjects
and the temporal detection of actions prior to multisubject
behavior recognition are necessary. Thus, the student class
behavior dataset proposed in this paper contains three parts,
123
Neural Computing and Applications (2021) 33:8335–8354
namely, a detection part, recognition part and caption part.
The former two parts are set up via action annotation, and
the latter is set up with caption annotation. The detection
part contains two data modules, while each of the other two
parts contains one module, as shown in Fig. 1.
The dataset has the following six characteristics: (1)
Special scenario. The dataset considers the core educa-
tional scene, i.e., a classroom, as the collection scenario,
which has excellent prospects for research and application.
(2) Real scene. We collect student’s spontaneous actions in
real classes using teaching videos. The videos show real
teaching scenes, and the students’ behaviors spontaneously
occur. Thus, the dataset is representative and realistic. (3)
Human-centric annotation. Rather than labeling an entire
video with atomic actions or temporal segmentations, this
dataset annotates students’ actions, each of which is com-
posed of several atomic actions, e.g., drinking. These
actions reflect the student’s states, which facilitate further
research on educational topics. (4) Comprehensive and
with multiple tasks. This dataset is a comprehensive
dataset for multiple tasks, namely, recognizing, detecting,
and captioning students’ behaviors. A number of tasks is
employed to automatically detect and caption students’
states by using AI technology. Because these collections
are closely related, multiple data modules are set up
simultaneously. (5) More complex caption annotations.
We provide more detailed captions than those of other
captioned datasets, and this allows us to better understand
students’ actions. (6) Challenging. The spontaneity of
students’ actions poses many challenges to existing meth-
ods. Because classroom scenes may contribute to low video
quality, with issues such as occlusion, the presence of
multiple subjects, large differences in subject sizes, and
few differences among action categories, an increased
requirement for algorithmic research is proposed for a
highly robust algorithm.
The remainder of this paper is organized as follows:
Sect. 2 reviews existing datasets for action recognition,
detection, and captioning. Section 3 describes the student
class behavior dataset. Section 4 introduces the character-
istics of the dataset and the challenges faced by existing
methods. Section 5 presents the baseline and analyzes the
difference between this dataset and other datasets, and
Sect. 6 concludes the paper.
2 Related work
2.1 Action recognition and detection datasets
The amazing results in the field of action recognition are
largely due to several mainstream behavioral recognition
datasets, such as the KTH [23], Weizmann [24],
Neural Computing and Applications (2021) 33:8335–8354
Fig. 1 Structure of the student class behavior dataset
Hollywood-2 [25], HMDB, and UCF101 datasets. The
limitations of the KTH and Weizmann datasets, which are
that they have the simple backgrounds, a lack of camera
movement, few kinds of actions, and only one person in
each video with a single movement, are obvious. The
scenes of these datasets are highly different from scenes in
reality. In video samples of the Hollywood-2 dataset, since
such factors as the expressions, postures, clothes, camera
movements, illumination changes, occlusions, and back-
grounds of the actors vary greatly, but are similar to situ-
ations in real scenes, the process of analysis or recognition
on this dataset is highly challenging. The maximum num-
ber of action categories in the UCF101 dataset and
HMDB51 dataset are 101 and 51, respectively. Their
uncertainties, which are induced by camera motion, clut-
tered backgrounds, different lighting conditions, occlusion,
and videos with low quality, have made research chal-
lenging and have attracted the attention of many
researchers. In addition to the previously mentioned data-
sets with short clips of manually tailored, individual
behavioral activities, some automatically labeled datasets,
such as the TrecVid MED [26], YouTube-8M [27],
something-something [28], Sports-1M [29], SLAC [30],
moments in time [31] and kinetics [32] datasets, have
emerged. The application of automatic labels arose due to
the high cost of manual investment. Clearly, this invest-
ment is performed at the expense of noisy labels, which
inevitably degrade the detection and recognition perfor-
mances of the model compared with those obtained by the
same model with completely artificially labeled datasets.
With further research, both datasets and methods have
begun to utilize temporal localization, e.g., the ActivityNet
[33], THUMOS [34], MultiTHUMOS [35] and Charades
[36] datasets, which use a large number of untrimmed
8337
videos, each of which contains at least one action. Activ-
ityNet is currently the largest dataset of this type, with
more than 648 h of untrimmed videos, approximately
20,000 videos, and 200 different daily activities. This
dataset provides temporal (but not spatial) localization for
each action of interest.
In recent years, some datasets based on spatiotemporal
annotations, such as the CMU [37], MSR Actions [38],
UCF Sports [39], JHMDB [40] and AVA datasets, have
emerged. Some datasets, such as the UCF101, DALY [41]
and Hollywood2Tubes [42] datasets, have been upgraded
from their original contents to spatiotemporal localization
in untrimmed videos.
2.2 Video caption datasets
It has been a fundamental yet evolving challenge for
computer vision to automatically caption visual content
with natural language. In particular, there has been enor-
mous interest in this area due to the presence of increas-
ingly large datasets. The caption datasets can be divided
into specific fields and open fields based on their applica-
tion backgrounds.
The specific field has two main categories: cooking and
watching movies. Cooking datasets mainly include the MP-
II Cooking [43], YouCook [44], TACoS [45], TACoS
Multilevel [46] and YouCook-II [47] datasets. The MP-II
Cooking dataset contains 65 cooking activities, in which
data are recorded in the same kitchen by a camera mounted
on the ceiling. The YouCook dataset contains 88 YouTube
cooking videos about the processes by which different
people cook a variety of recipes. Compared with the MP-II
Cooking dataset, the YouCook dataset is more challenging
and has different backgrounds and multiview shots.
123
8338
TACoS contains 26 fine cooking events in 127 videos; each
video contains 20 different text captions. TACoS Mul-
tilevel was upgraded based on TACoS; it provides three
levels of captioning for each video in the TACoS corpus,
including a detailed caption, a short caption, and a brief
caption of a single sentence. Typical movie-based datasets
are the MPII-Movie Description Corpus [48] and the
Montreal Video Annotation Dataset (M-VAD) [49]. The
former contains transcribed audio captions extracted from
94 Hollywood movies; the average length of each movie is
3.9 s with almost one sentence. The latter contains 48,986
video clips extracted from 92 different movies. Each clip
spans an average of 6.2 s, and the total number of sen-
tences is 55,904.
Datasets with open fields mainly include the Microsoft
Video Description (MSVD), the MSR-Video to Text
(MSR-VTT) [50], ActivityNet Captions [51], ANet-Enti-
ties [52], Charades [36] and VideoStory [53] datasets. The
MSVD dataset contains an average of 41 manually labeled
sentences per video. The videos are obtained from You-
Tube, and the length of each video, which mainly shows an
event, ranges between 10 and 25 s. The MSR-VTT dataset
is divided into 20 different categories; each category is
captioned by 20 Amazon Mechanical Turk (AMT) work-
ers. In addition to visual content, this dataset contains audio
information that may be employed for multimodal
research. Charades contains 9848 daily indoor family
activity videos. The average duration of each video, which
mainly depicts activities of daily living, is 30 s. Different
from other datasets, the ActivityNet Captions dataset
describes a certain activity in a video, that is, it describes a
certain piece of visual content in the video; therefore, a
corresponding start time and end time exist. The task
requires that the corresponding event be detected and then
captioned, which is clearly the more challenging part. The
ANet-Entities dataset, which was the first dataset with an
entity base and annotations, is based on the ActivityNet
Captions dataset. VideoStory was designed to solve the
narrative or caption generation problem of videos. Each
paragraph contains an average of 4.67 sentences, with an
average of 13.32 words per sentence. Table 1 [54] shows
information obtained from current mainstream video cap-
tion datasets.
3 Data collection
3.1 Action vocabulary generation
Since students may act randomly and arbitrarily, many
types of student behaviors are observed in classrooms. We
obtained a reasonable label vocabulary based on classroom
rules and regulations and the related literature [56–58]. We
123
Neural Computing and Applications (2021) 33:8335–8354
collected candidate action class sets with obvious visual
characteristics in mass classroom data. From these sets, we
selected typical classroom behaviors. Eventually, an action
vocabulary with 11 typical behaviors, e.g., taking notes,
using mobile phones, and carefully listening to the teacher,
became available.
3.2 Educational video selection
In Beijing Normal University, 244 classrooms are moni-
tored by 1920 * 1080 high-definition cameras (Brand:
HIKVISION; type: DS-2DF6223), which are deployed by
the Public Resources Management Center. Figure 2 shows
the deployment of one such camera. Each day, these
cameras produce 3.4 TB of data, i.e., daily classroom
videos, which provide the foundation for our data
collection.
Considering the illumination of and camera placement
in a classroom, videos of 11 classrooms were adopted. A
total of 128 untrimmed videos from different disciplines
and 13 courses (including computers, mathematics, and
geography) were collected. The students in the videos,
which comprise undergraduates, master’s students, and
doctoral candidates, are at different stages of education.
The number of participants in each class varies from 7 to
34.
3.3 Action annotation
As previously mentioned, describing a classroom video
often entails describing different types of actions per-
formed by multiple students and requires multisubject
behavior recognition, which involves not only the recog-
nition of certain actions but also the detection of their
temporal and spatial locations. Thus, action annotation is
related to the detection and recognition parts of our pro-
posed dataset. The former includes two data modules: a
module for temporal detection and a module for action
detection.
3.3.1 Recognition part
Students’ classroom behaviors are always considered as
one of their external performances with regard to student
states. Automatically obtaining their behaviors and manu-
ally recording them are difficult tasks. For further research
on recognition algorithms for students’ classroom behav-
iors, the recognition part of our proposed dataset is pro-
vided for the recognition task.
Recognition data module
The recognition part is one module that consists of many
video clips of single behaviors of a single subject. By
Neural Computing and Applications (2021) 33:8335–8354 8339

Table 1 Existing video caption

Dataset Domain Classes Videos Clips Sent Words Vocab
datasets
MP-II Cook Cooking 65 44 – 5609 – –
YouCook Cooking 6 88 – 2688 42,457 2711
TACoS Cooking 26 127 7206 18,227 146,771 28,292
TACosMlevel Cooking 1 185 14,105 52,593 – –
YouCook-II Cooking 89 2000 15.4 k 15.4 k – 2600
MPII-MD Movie – 94 68,337 68,375 653,467 24,549
M-VAD Movie – 92 48,986 55,904 519,933 17,609
VTW [55] Open – 18,100 – 44,613 – –
MSVD Open 218 1970 1970 70,028 607,339 13,010
MSR-VTT Open 20 7180 10 k 200 k 1856,523 29,316
ActyNet Cap Open – 20 k – 100 k 1348 k –
ANet-Entities Open – 14,281 52 k – – –
Charades Open 157 9848 – 27,847 – –
VideoStory Open – 20 k 123 k 123 k – –
Dataset: Dataset Name; Domain: The area of the video content; Classes: The number of video categories;
Videos: The number of videos collected; The number of video clips cropped out from the original video;
Sent: The number of captions in the dataset; Words: The number of words in the dataset; Vocab: The
number of vocabularies in the dataset

Fig. 2 Model diagrams of a classroom from different views. a Side view, b top view

manually annotating the categories of subjects’ actions,
subjects’ positions, and the start/end times for whole
videos, we can obtain the clips.
First, we numbered the original classroom videos. Sec-
ond, 128 videos were distributed to 12 professionally
trained annotators who watched and labeled the videos
according to our students’ action classes. Since the videos
contained all the students in the class, each spatial location
was marked with the coordinates of the upper left corner
and lower right corner and the start and end times of each
action. Last, with special software named Format Factory,
we obtained the partial clips based on the annotations and
then converted them into single-behavior video clips for a
single subject.
Therefore, the annotation for each clip contains nine
pieces of information, namely, the video clip number,
action number, action category label, start time, end time,
x-coordinate of the upper left corner of the action subject,
y-coordinate of the upper left corner of the action subject,
x-coordinate of the lower right corner of the action subject,
and y-coordinate of the lower right corner of the action
subject. For instance, 47-f-03-20-discuss-00_01_10-
00_01_35-263-160-784-533 corresponds to the time period
from 00:01:10 to 00:01:35 in Video No. 47-f-03. An action
of discussion is conducted by the subject with the coordi-
nates (263, 160) in the upper left corner and the coordinates
(784, 533) in the lower right corner.
3.3.2 Detection part
Automatically detecting students’ classroom behaviors in
original videos is essential for automatically understanding
students’ learning states and evaluating education quality.
For further research, the detection part of the dataset, which
is composed of two data modules, i.e., a data module for
temporal action detection and a data module for action
detection, is essential.
Detecting a student’s behavior state and its changes by
performing temporal action detection is helpful for ana-
lyzing his/her learning situation and psychological changes

123
8340 Neural Computing and Applications (2021) 33:8335–8354

to personalize his/her education. This dataset provides a Annotation

temporal detection data module.
The above processing provided us with a video clip
Detecting the states of students in the classroom by
sample; we used the action category and the coordinates of
action detection is helpful for evaluating teaching quality
the annotated action in the recognition module as its label.
and investigating students’ interests regarding different
As a result, a total of 3343 samples was collected to build
teaching methods or topics, adjusting the teaching plan and
the action detection data module.
improving teaching quality. This dataset provides an action
detection data module.
3.4 Caption annotation
Temporal detection data module
Collection of videos and clip selection A caption may precisely express the details of an activity.
Although a student’s behavior is available with the label,
Each collected original video is a 45-min video of the
his/her mental state and action details cannot be observed.
entire class. We acquired the space coordinates, as well as
They are the focus of our study. Based on the recognition
the starting and ending times of the action, based on the
data module, we captioned each video clip to build a
annotation of the recognition module. To obtain the sam-
caption data module.
ples for the temporal detection task, we cropped out the
videos of each student according to their coordinates. As
3.4.1 Caption data module
direct detection for the 45-min long video was challenging,
we divided the original video into several video clip sam-
Because the recognition module only recognizes a single
ples. Considering that approximately 90% of the actions
topic of the video, much other information could not be
annotated were within 40 s, we segmented them into
extracted at all. However, more specific descriptions of the
samples of 90 s. Once a cutting point was contained in an
visual content were demanded in the educational scene,
action, we extended it to the next clip to guarantee that
especially regarding student behaviors. Therefore, based on
each clip sample contained at least one annotated action.
the samples from the recognition module, we built the
The average length of the clip samples was 96.6 s.
caption module by deleting low-quality clips and adding
Annotations
some good clips, resulting in a total of 4296 samples.
The above processing provided us with a video clip
Collection of videos and clip selection
sample; we used the action category, together with the
Sentence annotation
starting and ending points of the annotated action in the
recognition module as its label. As a result, a total of 4542 Generally, most caption datasets describe a video in
samples was collected to build the temporal detection data only a simple way, but we needed a more detailed
module. Figure 3 shows an example. description. Therefore, we were strict with the annotator
who participated in the caption annotation process. The
Action detection data module
whole procedure can be described as follows: First, we
Collection of videos and clip selection
recruited and trained a group of annotators; second, we
To obtain the action detection sample, we divided the asked different annotators to describe the same video; and
original video into several video clip samples based on the finally, we asked reviewers to review the descriptions. The
starting and ending times of the annotated action in the details are as follows:
recognition module. There existed at least one annotated During annotator training, we conducted a qualification
action in each video clip sample. review (mainly to test their English level/ability) and

Fig. 3 Example of the temporal detection data module that shows a clip of 5 frames extracted from the video at a medium distance. The clip
shows that the subject has a reading action between 42 and 84 s

123
Neural Computing and Applications (2021) 33:8335–8354
professional training for all the annotators. All the qualified
people were divided into two groups, i.e., annotators and
reviewers. The former group was in charge of captioning
the videos, and the latter was responsible for checking the
caption quality, e.g., spelling and syntax.
Four annotators were asked to caption each labeled and
trimmed video with one sentence. The captioning process
had four requirements [59]:
1. Use actions and nouns that will appear frequently in a
class. For example, talk, hands up, and blackboard.
2. Do not describe any ‘‘predicted’’ information. For
example, it seems that the class will be over.
3. Do not use special nouns, such as people’s names.
4. Do not use simple descriptions. For example, a group
of people who are sitting.
After captioning the annotation, the reviewers examined
the spelling and sentence syntax of each annotations.
3.5 Quality control and privacy protection
Regarding privacy protection, the annotators were required
to sign an agreement before participating in our work, in
which they agreed to not disseminate or modify the original
video. In addition, all personal information of the subjects,
including names and addresses, were deleted or hidden
during the whole database construction process.
Three measures were taken to ensure the quality of the
annotations: (1) annotator training. A set of annotator
training programs was developed and conducted for each
volunteer. Only volunteers who passed the pre-annotating
appraisal could become annotators. (2) Pre-annotating and
annotator screening. After the training process was com-
pleted, small batches of videos were distributed to the
volunteers for pre-annotating, and their annotation work
was supervised and guided. With some evaluation, anno-
tators were screened. (3) Annotation and review. The
annotations were reviewed by special staff after recollec-
tion of the videos.
3.6 Dataset splits
Referring to the current mainstream dataset division prin-
ciple, we divided the samples of each module randomly
into a training set, validation set and test set at a ratio of
3:1:1, as shown in Table 2.
4 Student behavior dataset
The paucity of data has restricted the application of AI in
education. Therefore, we constructed a student behavior
dataset to provide a basis for such research. Given the
8341
complexity and specificity of educational scenes, this
dataset has some different characteristics, which could
bring new challenges to researchers. The details of the
dataset are introduced in the following sections.
4.1 Characteristics of the dataset
4.1.1 The classroom as the collection scenario
As the basic form, classroom teaching always has a core
role in education. Acquiring student behavior is the key,
which is not only helpful for mastering students’ learning,
personality and psychological traits but also worthy of
inclusion in evaluations of education quality. With the help
of massive amounts of classroom teaching videos gener-
ated in the era of big data, students’ classroom behaviors
can be automatically recognized, detected and captioned by
AI technology to evaluate and improve the quality of
classroom teaching. However, the required dataset has not
been developed in the educational scene, thereby hindering
related research in this respect. Therefore, the establish-
ment of a student behavior dataset is urgently needed.
4.1.2 Real videos as annotated material
In movie datasets, such as the MPII-Movie dataset, the
behavior of people is artificially limited. In reality, people’s
behavior is more complicated and abundant. To reflect
reality, our video materials were collected directly from
actual classroom scenes. Student behaviors in the videos
were spontaneous, which guaranteed the authenticity and
representativeness of our dataset.
4.1.3 Human-centric annotation
The objects of our research were students. We aimed to
reflect the students’ learning situations and psychological
states through their actions. Therefore, unlike other data-
sets captioning the entire video in general, our dataset is
human-centric with annotations based on students, whether
in the recognition, detection, or caption modules, and the
dataset effectively satisfies the requirements in the educa-
tional field.
4.1.4 Comprehensive and multitask dataset
At present, the lack of comprehensive datasets causes some
problems in AI research in the field of education. To
alleviate this problem, we built a comprehensive dataset
that can complete various tasks in this field.
123
8342 Neural Computing and Applications (2021) 33:8335–8354

Table 2 Partition sizes of the

Student Class Behavior Dataset
dataset
Recognition data module Action detection data module Caption data module
Temporal detection Action detection

Total sample 4276 4542 3343 4296

Training 2559 2725 2014 2602
Validation 858 908 659 862
Test 859 909 670 832

4.1.5 More complex caption annotations 4.1.6 Challenging

The use of richer language, such as adjectives, adverbs, and
more complex sentence structures, to caption a behavior
can better reflect the state and details of a subject. There-
fore, we provide more detailed captions than other caption
datasets, and this allows us to better understand the stu-
dents’ actions. Figure 4 shows an example of different
caption datasets. Obviously, our captions are more detailed,
and the sentence structure is more complicated than those
of other datasets.
Because classroom scenes always contribute to low-quality
and complex videos, our dataset is highly challenging, and
this is addressed in the next section.
4.2 Challenges of the dataset
4.2.1 Multiple subjects
A classroom is a multisubject environment in which dif-
ferent subjects perform different actions; for example, a
maximum of 34 subjects simultaneously appear in our

Fig. 4 Examples of three

different datasets. a and
b Captions from the MSVD
dataset and MSR-VTT dataset,
respectively, and c is a caption
from our dataset

123
Neural Computing and Applications (2021) 33:8335–8354
dataset. This situation creates great challenges for the
detection task.
4.2.2 Large differences in subject sizes (pixels)
In a classroom, students sit in different positions; therefore,
their picture sizes vary in the video, thereby creating extra
challenges for existing methods. Figure 5 shows an
example of a large difference in subject sizes in a video.
4.2.3 Less difference between action classes (similar
actions)
Limited by the characteristics of specific actions, some
behavior classes are visually similar, which undoubtedly
creates challenges for recognition and detection. Figure 6
shows the visual similarity between two different action
classes. We observe that their actions are very similar
visually.
4.2.4 Less difference between the foreground
and background
Due to the characteristics of classroom teaching, students’
activities are always restricted to a certain extent, and this
reduces the difference between the foreground and the
background. This restriction, which occurs for different
subjects and in different periods for the same subject,
hinders temporal and spatial action detection. Figure 7
shows an example of a similar foreground and
background.
4.2.5 Occlusion
Due to the limitations of the scene and the deployment
of cameras, some of the students’ behaviors are
Fig. 5 Examples of multisubject situations. The sizes of the subjects
in the two red boxes in the video differ substantially
8343
Fig. 6 Examples of the visual similarity between two different action
classes. a and b Show the ‘‘taking notes’’ behavior and the ‘‘using
phone’’ behavior, respectively
occluded to varying degrees, especially when many
students are in a classroom. This finding indicates the
need for higher requirements in algorithmic research
than the current requirements. Figure 8 shows two
examples of occlusion.
4.3 Annotation statistics
4.3.1 Recognition
The actions are divided into 11 classes, namely, listening
carefully, taking notes, using mobile phones, yawning,
eating or drinking, reading, discussing, looking around,
using computers, sleeping or snoring, and raising hands.
Figure 9 shows the number of samples for each class. The
class with the highest frequency is ‘‘listening carefully’’,
while the class with the lowest frequency is ‘‘raising
hands’’. Since the collected samples depict students with
high levels of education, the behavior of ‘‘raising hands’’ is
relatively rare. Thus, this class is removed from the base-
line for recognition and action detection.
4.3.2 Action detection
A total of 3343 video samples was collected in this module;
each sample contains multiple subjects, from 7 to 19
people. Figure 10 shows the distribution of the number of
subjects in each video clip sample. The average resolution
of a video sample is 1450 9 870, and the average length of
a video is 17.2 s.
4.3.3 Temporal detection
A total of 4542 untrimmed videos was collected in this
module, with a total capacity of 9.64 G. Each video con-
tains at least one annotation. The longest length of the
untrimmed videos is 187 s, the shortest length of the
untrimmed videos is 15 s, and the average length of the
untrimmed videos is 96.6 s. Figure 11 shows the distribu-
tion of the duration of each action. As shown, 95% of the
123
8344 Neural Computing and Applications (2021) 33:8335–8354

Fig. 7 Examples of the ‘‘taking notes’’ behavior. A clip of 5 frames was extracted from a video at a medium distance

4.3.4 Video captioning

A total of 4295 videos was captioned in this module; each

Fig. 8 Examples of occluded behaviors. a and b The ‘‘using phone’’
behavior and the ‘‘reading book’’ behavior, respectively
action durations are less than 50 s, among which 71% are
less than 20 s.
module contains four descriptive sentences with both
Chinese and English versions. In the English version, a
total of 261,582 words from 1850 vocabularies are inclu-
ded, with an average of 15.3 words per sentence. Compared
with several mainstream datasets, such as the MSVD
dataset with 8.7 words per sentence and the MSR-VTT
dataset with 9.3 per sentence, our dataset has more detailed
captions and more complicated sentence structures.

Fig. 9 Distribution of the 1200

numbers of behaviors in all 984
Number of each class

action classes 1000

800
582 545
600 520 515
365
400 265 252
168
200 80
15
0

Fig. 10 Distribution of the 800

number of subjects in this 700
Number of samples

dataset
600
500
400
300
200
100
0
7 8 9 10 12 13 14 15 16 17 18 19

Number of people

123
Neural Computing and Applications (2021) 33:8335–8354 8345

Fig. 11 Distribution of the 2000 1877

duration of each action
1800
1600

Number of samples
1346
1400
1200
1000
800 708
600
400 253
147 74
200 57 28 31 33
0
1~10 11~20 21~30 31~40 41~50 51~60 61~70 71~80 81~90 91~100
Duration of action(s)

5 Baseline and analysis 5.1.2 Datasets and metrics

5.1 Recognition We compare two standard datasets—HMDB51 and

5.1.1 Model and setting
The model employed for the baseline is the two-stream
network [60], which utilizes the notion that the appearance
of a subject in a static frame and motion information
between successive frames can complement the recogni-
tion of a specific behavior. The two-stream network takes
ResNet-101 as the backbone for both the spatial and tem-
poral streams. The spatial stream captures the spatial
information of behaviors from the static video frames,
while the temporal stream captures the motion information
of behaviors from the motion of the dense optical flow.
The training procedure is generally the same for both
spatial networks and temporal networks. The network
weights are learned using the mini-batch stochastic gradi-
ent descent with momentum (set to 0.9). At each iteration,
a mini-batch of 8 samples is constructed by sampling 8
training videos (uniformly across the classes); a single
frame is randomly selected from each video. In spatial
network training, a 2249224 sub-image is randomly
cropped from the selected frame; it then undergoes random
horizontal flipping and RGB jittering. In temporal network
training, we compute the optical flow volume I for the
selected training frame. From this volume, a fixed-size
224 9224 92 L image is randomly cropped and flipped.
For this task, L is set as 10. The learning rate is initially set
to 1023 and then decreased according to a fixed schedule,
which is kept constant for all training sets. In the fine-
tuning step, when training ConvNet from scratch, the rate
is changed to 1023 after 10 K iterations, changed to 1024
after 14 K iterations, and stopped after 20 K iterations. The
whole network is trained on an Nvidia GTX 1080Ti GPU
using the PyTorch deep learning framework.
UCF101—which are commonly employed for behavior
recognition. The HMDB51 dataset is mainly derived from
the video network site YouTube and digital cinema, and it
contains a total of 6676 videos of 51 categories with a
spatial resolution of 320 pixels 9 240 pixels. The UCF101
dataset contains a total of 13,320 videos in 101 categories.
Each type of video is divided into 25 groups, and each
group contains at least 100 videos, with video lengths from
29 to 1776 frames and a spatial resolution of 320 pix-
els 9 240 pixels. For these two datasets, we apply the
standard training/testing splits and protocols provided by
the datasets. For the HMDB51 and UCF101 datasets, we
report the average accuracy over the three splits. The for-
mula for average accuracy is shown below:
num of videos classified correctly
Average Acc ¼
num of all test videos
5.1.3 Evaluation and analysis
Table 3 shows the experimental results of the three datasets
for the two streams. As mentioned in Sect. 4.2, due to the
particularity of the classroom scene, the movements of
students’ classroom behaviors are relatively small, so they
generate relatively insufficient motion information in the
optical flow. The motion information of the ‘‘listening
carefully’’ sample in Fig. 12 is very small, and this causes
Table 3 Mean accuracy on UCF-101, HMDB-51 and our dataset
UCF101 (%) HMDB-51 (%) Our dataset (%)
Spatial stream 73 40.5 73.5
Temporal stream 83.7 54.6 51
Two-stream 86.9 58 61

123
8346
Fig. 12 Example of the ‘‘listening carefully’’ behavior. A clip of 5 frames was extracted from the ‘‘listening carefully’’ video at equal distances
poor performance on the temporal stream, thereby affecting
the effect of the two-stream network. Figure 13 shows
some failure cases of the baseline model in the recognition
module. We can see that some different behaviors of stu-
dents in the classroom are very similar visually. This kind
of similarity between classes brings great challenges to the
algorithm. We enumerate the comparison results of the
three datasets using the two-stream network, as shown in
Table 3. In addition, Table 4 shows the performance of the
model, which provides a benchmark for subsequent
research.
5.2 Action detection
5.2.1 Model and setting
The models utilized for the baseline are the ACAM [61]
and the MOC [62] models.
Label: playphone
Label: playphone
Fig. 13 The failure recognition of the two-stream model on our dataset
123
Neural Computing and Applications (2021) 33:8335–8354
ACAM model
The ACAM model generates attention maps conditioned
on each actor from contextual and actor features. Attention
maps are generated for each feature dimension, and they
determine the relation of the actor to every spatiotemporal
context location. This attention mechanism allows us to
focus on the actor without cropping, as in RoIPooling,
while capturing the spatiotemporal structure of the scene.
We initialize the models with I3D weights trained on the
Kinetics-400 dataset and train our models with an Adam
optimizer and a cosine learning rate between max (0.02)
and min (0.0001) for 40 epochs. We employ a batch size of
2 per GPU, and we have 2 Nvidia 1080 Ti’s (total batch
size of 4).
MOC model
The MOC model presents an action tubelet detection
framework, termed the moving center detector (MOC-
Two-stream: sleep
Two-stream: note
Neural Computing and Applications (2021) 33:8335–8354
Table 4 Performance of the
Parameters
two-stream model on our
dataset Spatial stream – 14.7
Temporal stream – 15.6
Two-stream 42,707,109
detector), by treating an action instance as a trajectory of
moving points based on the insight that movement infor-
mation could simplify and assist action tubelet detection. It
is composed of three crucial head branches: (1) a center
branch for instance center detection and action recognition,
(2) a movement branch for movement estimation at adja-
cent frames to form the trajectories of moving points, and
(3) a box branch for spatial extent detection by directly
regressing the bounding box size at each estimated center.
These three branches work together to generate the tubelet
detection results, which could be further linked to yield
video-level tubes with a matching strategy.
We choose DLA34 [63] as our backbone. The frame is
resized to 900 9900. The spatial downsampling ratio R is
set to 4, and the size of the resulting feature map is
225 9225. We use Adam with a learning rate of 5e-4 to
optimize the overall objective. The learning rate adjusts for
convergence on the validation set, and it decreases by a
factor of 10 when the model performance saturates. The
maximum number of iterations is set to 20 epochs. We
employ a batch size of 2 per GPU, and we have 2 Nvidia
1080 Ti’s (total batch size of 4).
5.2.2 Datasets and metrics
The J-HMDB dataset consists of 928 videos in 21 action
classes. Each video consists of 15–40 frames that span
1–2 s, and the time span of one action spans the entire
video clip. Thus, each video is only marked as an action
class. The atomic visual actions (AVA) dataset consists of
386 K video clips taken from 430 15-min movie clips at 1
FPS. The dataset consists of a 55:15:30 split, which yields
211 K training segments, 57 K validation segments and
118 K test segments. Each clip may contain multiple
subjects, and each subject has at least one of the 80 total
action classes.
The evaluation metric is the frame-level mean average
precision (frame-mAP) with an IOU threshold of 0.5, as
described in [21]. The frame-mAP measures the area under
the precision-recall curve of the detections for each frame.
A detection is correct if the intersection-over-union with
the ground truth at that frame is greater than r and the
action label is correctly predicted [64].
8347
Train time (h) Test time (s) Results (%)
4.1 73.5
12.5 51
– 9.2 61
5.2.3 Evaluation and analysis
Table 5 shows the experimental results of the three datasets
for the ACAM model and the MOC model. The results
show that detecting actions with our dataset is challenging.
As mentioned in Sect. 4.2, due to the particularity of
classrooms, such as occlusions, the presence of multiple
subjects, large differences in subject sizes, and small dif-
ferences among action categories, detecting actions by
using the algorithm is highly difficult, which indicates the
need for increased requirements in algorithmic research. In
addition, Table 6 shows the performance comparison
between the two models, which provides a benchmark for
subsequent research.
5.3 Temporal detection
5.3.1 Model and setting
The models utilized for the baseline are the BSN [65] and
the DBG [66] models.
BSN model
The BSN model consists of three modules: a temporal
evaluation module, a proposal generation module, and a
proposal evaluation module. Based on the extracted image
feature sequence obtained by the two-stream network, the
temporal evaluation module (TEM) in the BSN uses three
temporal convolution layers to generate the action start
probability sequence, the action end probability sequence,
and the action probability sequence. From the TEM results,
candidate time nodes can be obtained. Then, the candidate
start time node and the candidate end time node are com-
bined to retain the fragments with the required durations as
temporal action proposals and generate a proposal-level
feature (boundary-sensitive proposal feature) in the PGM
(proposal generation module). In the proposal evaluation
module (PEM), a multilayer perceptron (MLP) model is
Table 5 Comparison between the two datasets and our dataset with
the ACAM model and the MOC model in terms of frame-mAP
J-HMDB (%) AVA (%) Our dataset (%)
MOC 74 – 11.7
ACAM 78.90 22.67 15.84
123
8348
Table 6 Performance comparison between the MOC model and the
ACAM model on our dataset
Parameters Train time (h) Test time (s) Results (%)
MOC 20,544,180 78.1 1980 11.7
ACAM 12,290,234 23.3 900 15.84
used to estimate the confidence score for each proposal.
Finally, the soft-NMS algorithm is employed to suppress
overlapping results.
For visual feature encoding, we use the same strategy as
that employed in [65]. During training, the temporal eval-
uation module is trained with a batch size of 16 and a
learning rate of 0.001 for 10 epochs and then trained with a
learning rate of 0.0001 for another 10 epochs. The PEM is
trained with a batch size of 256 and the same learning rate.
For the soft-NMS algorithm, we set the threshold h to 0.8
and set the parameter e in the Gaussian function to 0.75 for
both datasets. We implement the algorithms using Ten-
sorFlow on a pair of Nvidia GTX 1080Ti GPUs.
DBG model
The DBG model mainly includes the following steps.
First, the dual-stream BaseNet is used to generate two
different levels and many discriminative features. Then, a
temporal boundary classification (TBC) module is adopted
to provide two temporal boundary confidence maps and
predict precise temporal boundaries. An action-aware
completeness regression (ACR) module is designed to
generate an action completeness score map and provide
reliable action completeness confidence. Finally, the soft-
NMS algorithm is used to suppress overlapping results.
For visual feature encoding, we use the same strategy as
that employed in [65]. During training, the DBG module is
trained with a batch size of 8 and a learning rate of 0.001
for 10 epochs and then trained with a learning rate of
0.0001 for another 2 epochs. For the soft-NMS algorithm,
we set the threshold h to 0.8 and set the parameter e in the
Gaussian function to 0.75 for both datasets. We implement
algorithms using Pytorch on a pair of Nvidia GTX 1080Ti
GPUs.
5.3.2 Datasets and metrics
ActivityNet-1.3 is a large-scale video action understanding
dataset that can be utilized to implement various video
tasks, such as action recognition, temporal detection, pro-
posal generation, and dense captioning. For temporal pro-
posal generation, the ActivityNet-1.3 dataset contains
19,994 temporal annotated, untrimmed videos with 200
123
Neural Computing and Applications (2021) 33:8335–8354
action categories, and the videos are divided into training,
validation and testing sets by the ratio 2:1:1.
A proposal is regarded as a true positive if it has a
temporal intersection-over-union (tIoU) with a ground-
truth segment that is greater than or equal to a given
threshold. The average recall (AR) is defined as the mean
of all recall values with diverse IoU thresholds between 0.5
and 0.95 with a step size of 0.05. The average number of
proposals (AN) is defined as the total number of proposals
divided by the number of videos in the testing subset. We
consider 100 bins for the AN. We evaluate the AR with
AN on the datasets to evaluate the relation between recall
and the number of proposal, which is represented as
AR@AN. Moreover, the area under the AR versus AN
curve (AUC) is also used as a metric, where the AN varies
from 0 to 100 with a step size of 1.
5.3.3 Evaluation and analysis
By calculating the average recall with different numbers of
proposals (AR@AN) and areas under the curve, we eval-
uate the abilities of the BSN model and the DBG model to
generate proposals with high recall for two datasets. We
enumerate the comparison results between ActivityNet-1.3
and our dataset using the BSN model and the DBG model
in Table 7. The results show that generating proposals with
our dataset is more difficult than doing so with Activi-
tyNet-1.3. As mentioned in Sect. 4.2, our foreground and
background are visually similar, and this prevents the
algorithm from generating proposals and adds high
requirements for algorithmic research. We show some
failure cases of the baseline model in the temporal detec-
tion module in Fig. 14. As analyzed above, because the
foreground and background are very similar, the algorithm
is prone to errors when detecting boundaries. In addition,
Table 8 shows the performance comparison between the
two models, which provides a benchmark for subsequent
research.
Table 7 Comparison between ActivityNet-1.3 and our dataset on the
BSN model and the DBG model in terms of the AR@AN and AUC
(%)
ActivityNet-1.3 Our dataset
AR@100 AUC(val) AR@100 AUC(val)
BSN 74.38 66.18 54.67 40.80
DBG 76.65 68.57 54.77 41.77
Neural Computing and Applications (2021) 33:8335–8354
Fig. 14 The failure cases of the BSN model and the DBG model on our dataset
Table 8 Performance
Parameters
comparison of the BSN model
and the DBG model on our
dataset
BSN 1,412,100
DBG 2,874,758
5.4 Video captioning
5.4.1 Model and setting
The models utilized for the baseline are the reconstruction
network (RecNet) [67] and HACA [68].
RecNet model
The model consists of three parts: an encoder, a decoder,
and a reconstructor. The main purpose of the encoder is to
extract the characteristics of videos. The decoder employs
long short-term memory (LSTM) to convert video features
into sentences. To further exploit the global time infor-
mation of the video, a time attention mechanism is
employed to encourage the decoder to select the most
important information. The reconstructor is built on top of
the encoder-decoder combination to reproduce the video
features from the sentences generated by the decoder. Via
this reconstruction process, the decoder is encouraged to
embed much information from the video. Therefore, the
relationship between the video sequence and the caption
can be further enhanced, thereby improving the video
captioning performance.
For the encoder, we extract the 28 equally spaced fea-
tures from one video; if the number of features is less than
28, we apply zero vectors to pad them and then feed them
into Inception-V4, which is pretrained on the ILSVRC-
8349
Train time (h) Test time (s) Results
AR@100 AUC(val)
4.3 175.43 54.67 40.80
4.1 447.83 54.77 41.77
2012-CLS for feature extraction after resizing the features
to a size of 299 9299 and extracting 1536 dimensional
semantic features of each frame from the last pooling layer.
The maximum length of a sentence is limited to 30 words.
Each word is embedded as 468 characters. For the decoder,
the input dimension is 468, while the hidden layer contains
512 units. For the reconstructor, the inputs are the hidden
states of the decoder and thus have dimensions of 512, and
the dimension of the hidden layer is set to 1536. During
training, AdaDelta is employed for optimization. During
testing, a beam search with a size of 5 is performed for the
final caption generation. We implement the algorithms
using Pytorch on a single Nvidia GTX 1080Ti GPU.
HACA model
The HACA model is an encoder-decoder framework
comprising multiple hierarchical recurrent neural networks.
Specifically, in the encoding stage, the model has one
hierarchical attentive encoder for each input modality, and
the encoder learns and outputs both the local and global
representations of the modality.
In the decoding stage, the model employs two cross-
modal attentive decoders: the local decoder and the global
decoder. The global decoder attempts to align the global
contexts of different modalities and learn the global cross-
modal fusion context. Correspondingly, the local decoder
learns a local cross-modal fusion context, combines it with
123
8350
the output from the global decoder, and predicts the next
word.
The maximum number of frames is 40. For the visual
hierarchical attentive encoders, the low-level encoder is a
bidirectional LSTM with a hidden layer dimension of 512,
and the high-level encoder is an LSTM with a hidden layer
dimension of 256 and a chunk size s of 10. The global
decoder is an LSTM with a hidden layer dimension of 256,
and the local decoder is an LSTM with a hidden layer
dimension of 1024. The maximum step size of the decoders
is 16. We use word embeddings of size 512. Moreover, we
adopt the dropout procedure with a value of 0.5 for regu-
larization. The AdaDelta optimizer is used with a batch
size of 64. The learning rate is initially set as 1 and then
reduced by a factor of 0.5 when the current CIDEr score
does not surpass the previous best for the previous 4
epochs. The maximum number of epochs is set as 50. We
implement the algorithms using Pytorch on a single Nvidia
GTX 1080Ti GPU.
5.4.2 Datasets and metrics
In this subsection, our dataset is compared with two
extensively employed video captioning benchmark data-
sets: the MSVD dataset and MSR-VTT dataset. MSVD is
an open-domain dataset for video captioning that covers
various topics, including sports, animals, and music. The
dataset contains 1970 YouTube video clips that are each
labeled with approximately 40 English descriptions col-
lected by AMT. We split the dataset into contiguous groups
of videos by index number: 1200 videos for training, 100
videos for validation and 670 videos for testing. The MSR-
VTT dataset is a set of 10,000 video clips (6513 video clips
for training, 497 video clips for validation, and the
remaining 2990 video clips for testing) from a commercial
video search engine. The dataset has twenty categories,
such as music, sports, and movies; each category has 20
human-annotated reference captions. Metrics: We use
several standard automated evaluation metrics: METEOR
[69], BLEU-4 [70], CIDEr-D [71], and ROUGE-L [72]. All
the metrics are computed by using the Microsoft COCO
[59] evaluation server. Due to the limited space in this
paper, specific indicator definitions and calculation meth-
ods can be found in [59].
5.4.3 Evaluation and analysis
Figure 15 provides some example results for the test sets of
the three datasets. An excessively short description may
neither accurately describe a video nor express the content
of a video in a rich language. The results show that our
dataset produces a more complex sentence description that
is closer to the human level than those of the other datasets.
123
Neural Computing and Applications (2021) 33:8335–8354
Compared with the other two datasets, the description in
our dataset is more reflective of the content in the video.
We enumerate the comparison results of the three datasets
using the RecNet model and the HACA model in Table 9.
Our dataset is more complex and realistic than other
datasets, and this makes the algorithm more demanding.
Figure 16 shows some failure cases of the models on our
dataset. The first video is about playing with mobile
phones, but it is described as ‘‘taking notes’’, and the
second video does not include yawning, but it is described
as containing this behavior. Due to the special character-
istics of the classroom scene and the length of the caption,
the algorithm is likely to lose important information during
the captioning process. In addition, Table 10 shows the
performance comparison between the two models, which
provides a benchmark for subsequent research.
6 Conclusion
Analyzing the learning and mental states of students
through their behavioral information are important for
teaching evaluations and for designing improved instruc-
tion plans. However, it is tedious and costly to record
students’ in-class behaviors through traditional manual
methods, such as field observations and questionnaire
surveys. It would be truly remarkable if AI technology
could provide automatic and accurate detection and
description of students’ classroom behaviors to provide a
basis for teaching evaluations and improving instruction
quality. Unfortunately, the lack of corresponding datasets
limits the study and application of AI technology in the
education field. Based on these findings, this study con-
structed a comprehensive dataset that can be used to rec-
ognize, detect, and caption student classroom behaviors,
and this can provide a data foundation for AI research in
the field of teaching. Moreover, we also provided a base-
line for each module of the dataset.
Compared with other datasets, our dataset has four main
contributions: (1) To the best of our knowledge, the pro-
posed dataset fills the gaps in student classroom behavior
research under teaching scenarios. (2) Classroom teaching
is a complex scenario that introduces many new challenges
to the algorithm and thus encourages people to explore
improved approaches for meeting real-world needs. (3) The
proposed dataset is a comprehensive dataset that contains
recognition, detection, and captioning tasks, which can
meet the research requirements of various tasks. (4) We
provided a comprehensive analysis of the dataset in the
paper and presented a baseline to verify our analysis. In
addition, each baseline model is evaluated and compared in
detail, thereby providing a multidimensional comparison
for algorithms using this dataset.
Neural Computing and Applications (2021) 33:8335–8354 8351

Fig. 15 Examples of sentence generation using the RecNet method with the three different datasets

Table 9 Performance evaluation of different datasets with the RecNet model and the HACA model in terms of BLEU-4, METEOR, ROUGE-L,
and the CIDEr scores (%)
MSVD MSR-VTT Our dataset
BLEU- METEOR ROUGE- CIDEr BLEU- METEOR ROUGE- CIDEr BLEU- METEOR ROUGE- CIDEr
4 L 4 L 4 L

HACA – – – – 39.6 27.4 59.7 45.8 28.4 23.1 48.1 43.1

RecNet 52.3 34.1 69.8 80.3 39.1 26.6 59.3 42.7 34.7 25.1 50.3 43.5

a girl in a grey jacket and glasses is taking notes with her head down.

a girl in a black coat is listening and yawns.

Fig. 16 Some failure cases of the HACA model and the RecNet model on our dataset

123
8352
Table 10 Performance
Parameters Train time (h)
comparison of the HACA model
and the RecNet model on our
dataset
HACA 25,103,345 4.9
RecNet 11,698,968 3 294
Our plan for future research is described as follows: with
regard to the datasets, we will discuss and analyze how to
provide a research basis for using AI to solve educational
problems, such as by analyzing and building related data-
sets. Regarding the research method, it can be seen from
the baseline experiment results that neither efficiency nor
effectiveness can meet practical applications at present.
Therefore, we will actively study and solve the current
problems of algorithms using this dataset such that it can be
applied to improve the quality of education.
Compliance with ethical standards
Conflict of interest The authors declared that they have no conflicts of
interest with regard to this work. We declare that we do not have any
commercial or associative interest that represents a conflict of interest
in connection with the work submitted.
References
1. Wang L, Qiao Y, Tang X (2015) Action recognition with tra-
jectory-pooled deep-convolutional descriptors. In: Proceedings of
the IEEE conference on computer vision and pattern recognition,
pp 4305–4314
2. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural
networks. In: Proceedings of the IEEE conference on computer
vision and pattern recognition, pp 7794–7803
3. Wang Y, Long M, Wang J, Yu PS (2017) Spatiotemporal pyra-
mid network for video action recognition. In: Proceedings of the
IEEE conference on computer vision and pattern recognition,
pp 1529–1538
4. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg
AC (2016) SSD: single shot multibox detector. In: European
conference on computer vision. Springer, pp 21–37
5. Lu X, Li B, Yue Y, Li Q, Yan J (2019) Grid r-cnn. In: Pro-
ceedings of the IEEE conference on computer vision and pattern
recognition, pp 7363–7372
6. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only
look once: unified, real-time object detection. In: Proceedings of
the IEEE conference on computer vision and pattern recognition,
pp 779–788
7. Shaoqing Ren, Kaiming He,Ross Girshick,Jian Sun (2016) Faster
R-CNN: towards real-time object detection with region proposal
networks. In: IEEE transactions on pattern analysis and machine
intelligence. arXiv:1506.01497
8. Berclaz J, Fleuret F, Fua P (2006) Robust people tracking with
global trajectory optimization. In: Proceedings of the IEEE
conference on computer vision and pattern recognition,
pp 744–750
123
Neural Computing and Applications (2021) 33:8335–8354
Test time (s) Results
BLEU-4 METEOR ROUGE-L CIDEr
468 28.4 23.1 48.1 43.1
34.7 25.1 50.3 43.5
9. Breitenstein MD, Reichlin F, Leibe B, Koller-Meier E, Gool L
(2009) Robust tracking-by-detection using a detector confidence
particle filter. In: Proceedings of the IEEE international confer-
ence on computer vision, pp 1515–1522
10. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional
neural networks on graphs with fast localized spectral filtering.
In: Advances in neural information processing systems,
pp 3844–3852
11. Ma L, Lu Z, Shang L, Li H (2015) Multimodal convolutional
neural networks for matching image and sentence. In: Proceed-
ings of the IEEE international conference on computer vision,
pp 2623–2631
12. Wang J, Liu W, Kumar S, Chang S (2016) Learning to hash for
indexing big data: a survey. Proc IEEE 104(1):34–57
13. Liu W, Zhang T (2016) Multimedia hashing and networking.
IEEE Multimed 23:75–79
14. Song J, Gao L, Liu L, Zhu X, Sebe N (2018) Quantization-based
hashing: a general framework for scalable image and video
retrieval. Pattern Recogn 75:175–187
15. Wang J, Zhang T, Song J, Sebe N, Shen H (2018) A Survey on
Learning to Hash. IEEE Trans Pattern Anal Mach Intell
40(4):769–790
16. Haijun Z, Yuzhu J, Wang H, Linlin L (2019) Sitcom-star-based
clothing retrieval for video advertising: a deep learning frame-
work. Neural Comput Appl. https://doi.org/10.1007/s00521-018-
3579-x
17. Ma L, Lu Z, Li H (2016) Learning to answer questions from
image using convolutional neural network. Assoc Adv Artif Intel
3:16
18. Soomro K, Zamir AR, Shah MJCe (2012) UCF101: a dataset of
101 human actions classes from videos in the wild. arXiv:1212.
0402
19. Kuehne, H, Jhuang, H, Garrote, E, Poggio, T.A., Serre, T (2011)
HMDB: a large video database for human motion recognition. In:
Proceedings of the IEEE international conference on computer
vision, pp 2556–2563
20. Chen D, Dolan WB (2011) Collecting highly parallel data for
paraphrase evaluation. In: Proceedings of the 49th annual meet-
ing of the association for computational linguistics: human lan-
guage technologies, pp 190–200
21. Gu C, Sun C, Ross DA, Vondrick C, Pantofaru C, Li Y,
Vijayanarasimhan S, Toderici G, Ricco S, Sukthankar R (2018)
Ava: a video dataset of spatio-temporally localized atomic visual
actions. In: Proceedings of the IEEE conference on computer
vision and pattern recognition, pp 6047–6056
22. Wang X, Wu J, Chen J, Li L, Wang YF, Wang WY (2019)
VATEX: a large-scale, high-quality multilingual dataset for
video-and-language research. In: Proceedings of the IEEE inter-
national conference on computer vision, pp 4580–4590
23. Schuldt C, Laptev I, Caputo B (2004) Recognizing human
actions: a local SVM approach. In: International conference on
pattern recognition, pp 32–36
24. Gorelick L, Blank M, Shechtman E, Irani M, Basri R (2007)
Actions as space-time shapes. IEEE Trans Pattern Anal Mach
Intel 29(12):2247–2253
Neural Computing and Applications (2021) 33:8335–8354
25. Marszalek M, Laptev I, Schmid C (2009) Actions in context. In:
Proceedings of the IEEE conference on computer vision and
pattern recognition, pp 2929–2936
26. Over P, Fiscus J, Sanders G, Joy D, Quénot G (2013) TRECVID
2013: an overview of the goals, tasks, data, evaluation mecha-
nisms and metrics. In: TRECVID 2013 workshop participants
notebook papers. http://www-nlpir.nist.gov/projects/tvpubs/tv13.
papers/tv13overview.pdf. Accessed 29 Dec 2020
27. Abu-El-Haija S, Kothari N, Lee J, Natsev P, Toderici G,
Varadarajan B, Vijayanarasimhan S (2016) YouTube-8M: a
large-scale video classification benchmark. arXiv:1609.08675
28. Goyal R, Kahou SE, Michalski V, Materzynska J, Westphal S,
Kim H, Haenel V, Fruend I, Yianilos P, Mueller-Freitag M,
Hoppe F, Thurau C, Bax I, Memisevic R (2017) The ‘‘something
something’’ video database for learning and evaluating visual
common sense. In: Proceedings of the IEEE international con-
ference on computer vision, pp 5843–5851
29. Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-
Fei L (2014) Large-scale video classification with convolutional
neural networks. In: Proceedings of the IEEE conference on
computer vision and pattern recognition, pp 1725–1732
30. Zhao H, Torralba A, Torresani L, Yan Z (2017) SLAC: a sparsely
labeled dataset for action classification and localization. In:
Proceedings of the IEEE conference on computer vision and
pattern recognition. arXiv:1712.09374
31. Monfort M, Andonian A, Zhou B, Ramakrishnan K, Bargal SA,
Yan T, Brown L, Fan Q, Gutfruend D, Vondrick C (2018)
Moments in time dataset: one million videos for event under-
standing. In: IEEE transactions on pattern analysis and machine
intelligence. arXiv:1801.03150
32. Kay W, Carreira J, Simonyan K, Zhang B, Zisserman A (2017)
The kinetics human action video dataset. In: Proceedings of the
IEEE conference on computer vision and pattern recognition.
arXiv:170506950
33. Caba Heilbron F, Escorcia V, Ghanem B, Carlos Niebles J (2015)
Activitynet: a large-scale video benchmark for human activity
understanding. In: Proceedings of the IEEE conference on com-
puter vision and pattern recognition, pp 961–970
34. Idrees H, Zamir AR, Jiang Y-G, Gorban A, Laptev I, Sukthankar
R, Shah M (2017) The THUMOS challenge on action recognition
for videos ‘‘in the wild’’. Comput Vis Image Underst 155:1–23
35. Yeung S, Russakovsky O, Jin N, Andriluka M, Mori G, Fei-Fei L
(2018) Every moment counts: dense detailed labeling of actions
in complex videos. In: Proceedings of the IEEE conference on
computer vision and pattern recognition, pp 375–389
36. Sigurdsson GA, Varol G, Wang X, Farhadi A, Laptev I, Gupta
(2016) A hollywood in homes: crowd sourcing data collection for
activity understanding. In: European conference on computer
vision. Springer, pp 510–526
37. Ke Y, Sukthankar R, Hebert M (2005) Efficient visual event
detection using volumetric features. In: Proceedings of the IEEE
international conference on computer vision, pp 166–173
38. Yuan J, Liu Z, Wu Y (2009) Discriminative subvolume search for
efficient action detection. In: Proceedings of the IEEE conference
on computer vision and pattern recognition, pp 2442–2449
39. Rodriguez MD, Ahmed J, Shah M (2008) Action mach a spatio-
temporal maximum average correlation height filter for action
recognition. In: Proceedings of the IEEE conference on computer
vision and pattern recognition, pp 1–8
40. Jhuang H, Gall J, Zuffi S, Schmid C, Black MJ (2013) Towards
understanding action recognition. In: Proceedings of the IEEE
international conference on computer vision, pp 3192–3199
41. Weinzaepfel P, Martin X, Schmid C (2016) Towards weakly-
supervised action localization. arXiv:1065.05197
8353
42. Mettes P, Van Gemert JC, Snoek CG (2016) Spot on: action
localization from pointly-supervised proposals. In: European
conference on computer vision. Springer, pp 437–453
43. Rohrbach M, Amin S, Andriluka M, Schiele B (2012) A database
for fine grained activity detection of cooking activities. In: Pro-
ceedings of the IEEE conference on computer vision and pattern
recognition, pp 1194–1201
44. Das P, Xu C, Doell RF, Corso JJ (2013) A thousand frames in just
a few words: lingual description of videos through latent topics
and sparse object stitching. In: Proceedings of the IEEE confer-
ence on computer vision and pattern recognition, pp 2634–2641
45. Rohrbach M, Regneri M, Andriluka M, Amin S, Pinkal M,
Schiele B (2012) Script data for attribute-based recognition of
composite activities. In: European conference on computer
vision, Springer, pp 144–157
46. Rohrbach A, Rohrbach M, Qiu W, Friedrich A, Pinkal M, Schiele
B (2014) Coherent multi-sentence video description with variable
level of detail. In: German conference on pattern recognition.
Springer, pp 184–195
47. Zhou L, Xu C, Corso J (2018) Towards automatic learning of
procedures from web instructional videos. In: Association for the
advancement of artificial intelligence, pp 7590–7598
48. Rohrbach A, Rohrbach M, Tandon N, Schiele B (2015) A dataset
for movie description. In: Proceedings of the IEEE conference on
computer vision and pattern recognition, pp 3202–3212
49. Torabi A, Pal C, Larochelle H, Courville A (2015) Using
descriptive video services to create a large data source for video
annotation research. arXiv:1503.01070
50. Xu J, Mei T, Yao T, Rui Y (2016) Msr-vtt: A large video
description dataset for bridging video and language. In: Pro-
ceedings of the IEEE conference on computer vision and pattern
recognition, pp 5288–5296
51. Krishna R, Hata K, Ren F, Fei-Fei L, Carlos Niebles J (2017)
Dense-captioning events in videos. In: Proceedings of the IEEE
international conference on computer vision, pp 706–715
52. Zhou L, Kalantidis Y, Chen X, Corso JJ, Rohrbach M (2019)
Grounded video description. In: Proceedings of the IEEE con-
ference on computer vision and pattern recognition,
pp 6578–6587
53. Gella S, Lewis M, Rohrbach M (2018) A dataset for telling the
stories of social media videos. In: Proceedings of the 2018 con-
ference on empirical methods in natural language processing,
pp 968–974
54. Aafaq N, Mian A, Liu W, Gilani SZ, Shah M (2019) Video
description: a survey of methods, datasets, and evaluation met-
rics. ACM Comput Surv 52(6):1–37
55. Zeng K-H, Chen T-H, Niebles JC, Sun M (2016) Generation for
user generated videos. In: European conference on computer
vision. Springer, pp 609–625
56. Wei Q, Sun B, He J, Yu LJ (2017) BNU-LSVED 2.0: Sponta-
neous multimodal student affect database with multi-dimensional
labels. Sig Process Image Commun 59:168–181
57. Wang Z, Pan X, Miller KF, Cortina KSJC, Education (2014)
Automatic classification of activities in classroom discourse.
Comput Educ 78:115–123
58. Sun B, Wei Q, He J, Yu L, Zhu X (2016) BNU-LSVED: a
multimodal spontaneous expression database in educational
environment. In: Optics and photonics for information processing
X, international society for optics and photonics, p 997016
59. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D,
Dollár P, Zitnick CL(2014) Microsoft coco: common objects in
context. In: European conference on computer vision. Springer,
pp 740–755
60. Simonyan K, Zisserman (2014) A Two-stream convolutional
networks for action recognition in videos. In: Advances in neural
information processing systems, pp 568–576
123
8354
61. Ulutan O, Rallapalli S, Srivatsa M, Torres C, Manjunath B (2020)
Actor conditioned attention maps for video action detection. In:
The IEEE winter conference on applications of computer vision,
pp 527–536
62. Li Y, Wang Z, Wang L, Wu G (2020) Actions as moving points.
In: Proceedings of the European conference on computer vision.
arXiv:2001.04608
63. Yu F, Wang D, Shelhamer E, Darrell T (2018) Deep layer
aggregation. In: Proceedings of the IEEE conference on computer
vision and pattern recognition, pp 2403–2412
64. Gkioxari G, Malik J (2015) Finding action tubes. In: Proceedings
of the IEEE international conference on computer vision,
pp 759–768
65. Lin T, Zhao X, Su H, Wang C, Yang M (2018) Bsn: boundary
sensitive network for temporal action proposal generation. In:
Proceedings of the European conference on computer vision,
pp 3–19
66. Lin C, Li J, Wang Y, Tai Y, Luo D, Cui Z, Wang C, Li J, Huang
F, Ji R (2020) Fast learning of temporal action proposal via dense
boundary generator. In: Association for the advancement of
artificial intelligence. https://doi.org/10.22648/ETRI.2020.J.
350303
67. Wang B, Ma L, Zhang W, Liu W (2018) Reconstruction network
for video captioning. In: Proceedings of the IEEE conference on
computer vision and pattern recognition, pp 7622–7631
123
Neural Computing and Applications (2021) 33:8335–8354
68. Wang X, Wang YF, Wang WY (2018) Watch, listen, and
describe: globally and locally aligned cross-modal attentions for
video captioning. In Proceedings of the 2018 conference of the
north american chapter of the association for computational lin-
guistics: human language technologies, pp 795–801
69. Denkowski M, Lavie (2014) A Meteor universal: language
specific translation evaluation for any target language. In: Pro-
ceedings of the ninth workshop on statistical machine translation,
pp 376–380
70. Papineni K, Roukos S, Ward T, Zhu W-J (2002) BLEU: a method
for automatic evaluation of machine translation. In: Proceedings
of the 40th annual meeting of the association for computational
linguistics, pp 311–318
71. Vedantam R, Lawrence Zitnick C, Parikh D (2015) Cider: con-
sensus-based image description evaluation. In: Proceedings of the
IEEE conference on computer vision and pattern recognition,
pp 4566–4575
72. Lin C-Y (2004) Rouge: a package for automatic evaluation of
summaries. In: Text summarization branches out, pp 74–81
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.