Video-based action recognition
Balgynbek Dikhan
School of Engineering and Digital Sciences
Data Science
balgynbek.dikhan@nu.edu.kz
Sultan Abylkairov
School of Sciences and Humanities
Mathematics
sultan.abylkairov@nu.edu.kz
1. Introduction
In today’s society the advancements in camera technol-
ogy have made it more affordable and accessible to every-
one. Every mobile phone now comes with its built-in cam-
era. Video devices are also widely used in fields such as
security, traffic management, healthcare, sports and more.
The abundance of video data generated from these sources
offers opportunities for research within the computer vision
community. For example, having the ability to understand
actions is crucial, in many important applications that im-
pact society greatly including intelligent surveillance [9],
patient monitoring, sports analysis and human computer in-
teraction [13].
2. Literature Review
In the field of recognizing human actions, researchers
have conducted numerous studies using various methods
and types of data. In the early stages of research, the focus
was on using RGB or grayscale video as input, mainly be-
cause of their wide availability and ease of use [11]. How-
ever, in recent years, there has been a notable shift towards
the exploration of alternative data modalities, as evidenced
by numerous works [1, 10, 12, 5, 17, 4, 19, 7]. These modal-
ities include, but are not limited to, skeletal data, depth in-
formation, infrared sequences, point clouds, event streams,
audio, acceleration, radar, and more. This trend has been
driven by the proliferation of accurate and cost-effective
sensor technologies. Moreover Convolutional Neural Net-
works (CNN) [18, 3, 6], and Long Short-Term Memory net-
works (LSTM) [2] have demonstrated remarkable perfor-
mance when applied to human action recognition.
In this project, we have chosen to utilize the freely avail-
able UCF Sports Dataset [16]. We find this dataset partic-
ularly valuable due to its rich diversity of actions, encom-
Rustem Sankubayev
School of Engineering and Digital Sciences
Computer Science
rustem.sankubayev@nu.edu.kz
passing a wide spectrum of sports and physical activities.
Furthermore, it presents challenging real-world scenarios,
including variations in lighting, backgrounds, camera an-
gles, and instances of occlusion. These inherent complexi-
ties make it an ideal testing ground for algorithms designed
to address real-world challenges.
3. Technical approach
3.1. Dataset
The first step is to download the UCF Sports Dataset
[14], which consists of video clips showcasing ten actions.
The dataset contains a total of 150 sequences with a reso-
lution of 720 x 480. It is. Covers actions, like diving, golf
swing, kicking, lifting riding horse running, skateboarding,
swinging bench, swinging side and walking. These actions
represent a range of scenarios and perspectives. The dataset
is divided into two subsets; training and testing. This initial
phase sets the foundation for model training and evaluation.
Data preprocessing plays a role in ensuring consistency
across the dataset. This involves standardizing video pa-
rameters such as resolutions and frame rates. Additionally
individual frames are extracted from the video clips to serve
as the basis for feature extraction.
3.2. Baseline
To capture information from the video frames we uti-
lize the Space Time Interest Point (STIP) feature extrac-
tion method [8]. STIP helps capture details about motion
patterns within each frame and enables us to identify pat-
terns that define different actions. We systematically extract
STIP features from every frame to create a representation
of action dynamics. The approach involves conducting a
series of experiments to find the parameters for extracting
features, such as scale and location thresholds.
1
 During data representation each video is converted into a
sequence of STIP feature vectors. To improve action recog-
nition techniques are used to segment videos into segments
or shots. This helps the model effectively identify action
sequences.
3.3. Main approach
In this step we choose the Support Vector Machine
(SVM) [15] as our classification model to differentiate be-
tween ten actions. SVM is a used algorithm in machine
learning and computer vision in video based action recog-
nition. Selecting the SVM configuration ensures perfor-
mance, in action recognition tasks.
We train our model using the SVM classifier and STIP
features extracted from video frames. The training process
involves refining the SVM model by experimenting with
regularization settings to enhance its accuracy and overall
performance.
3.4. Evaluation Metric
To thoroughly assess the models performance we imple-
ment Leave One Out (LOO) validation methodology [14].
This approach guarantees that every data point is utilized
for both training and testing resulting in an assessment of
the system. We. Record evaluation metrics such as accu-
racy, precision, recall and F1 score, for each action category
providing a comprehension of the model’s efficacy.
4. Preliminary results
In our project we have made progress in preparing and
planning our video based action recognition project. We
conducted a review of existing literature that focused on
methods for extracting features, designing classifiers and
evaluating the effectiveness of video based action recog-
nition. This initial step provided us with insights into the
practices and techniques used by researchers in this field.
After consideration we decided to use the UCF Sports
Dataset as our dataset for action recognition. We thoroughly
examined the dataset to gain an understanding of its con-
tents, which includes ten distinct actions.
For feature extraction we chose the Space Time Interest
Point (STIP) method. This method allows us to capture in-
formation within video frames making it ideal for our action
recognition task.
As for the classification component of our project we
opted to use Support Vector Machine (SVM) as our clas-
sifier. SVMs are known for their effectiveness in handling
action recognition tasks and their ability to capture linear
relationships within data.
Currently we are actively involved in developing the pro-
gram code for our action recognition system. We’re mak-
ing progress, with the coding process specifically working
on incorporating the chosen STIP feature extraction method
and SVM classifier.
As we move forward with developing and testing our
action recognition system we expect to see some results.
These findings will help shape our project and add to our
understanding of video based action recognition using the
UCF Sports Dataset.
5. Conclusion
The proposed technical method provides a structure
for video-based action recognition using the UCF Sports
Dataset. It emphasizes the significance of data prepara-
tion, extracting features, selecting models and conducting
evaluations to guarantee the recognition systems accuracy
and dependability. This project makes a contribution to the
field of computer vision and action recognition by provid-
ing insights into designing models, optimizing parameters
and employing evaluation strategies.
References
[1] J. Donahue, L. Anne Hendricks, S. Guadarrama,
M. Rohrbach, S. Venugopalan, K. Saenko, and T. Dar-
rell. Long-term recurrent convolutional networks for visual
recognition and description. In Proceedings of the IEEE
conference on computer vision and pattern recognition,
pages 2625–2634, 2015. 1
[2] H. Gammulle, S. Denman, S. Sridharan, and C. Fookes. Two
stream lstm: A deep fusion framework for human action
recognition. In 2017 IEEE winter conference on applications
of computer vision (WACV), pages 177–186. IEEE, 2017. 1
[3] C. Gao, Y. Du, J. Liu, J. Lv, L. Yang, D. Meng, and A. G.
Hauptmann. Infar dataset: Infrared action recognition at dif-
ferent times. Neurocomputing, 212:36–47, 2016. 1
[4] R. Ghosh, A. Gupta, A. Nakagawa, A. Soares, and
N. Thakor. Spatiotemporal filtering for event-based action
recognition. arXiv preprint arXiv:1903.07067, 2019. 1
[5] Z. Jiang, V. Rozgic, and S. Adali. Learning spatiotemporal
features for infrared action recognition with 3d convolutional
neural networks. In Proceedings of the IEEE conference on
computer vision and pattern recognition workshops, pages
115–123, 2017. 1
[6] R. Kavi, V. Kulathumani, F. Rohit, and V. Kecojevic. Mul-
tiview fusion for activity recognition using deep neural
networks. Journal of Electronic Imaging, 25(4):043010–
043010, 2016. 1
[7] Y. Kim and T. Moon. Human detection and activity classifi-
cation based on micro-doppler signatures using deep convo-
lutional neural networks. IEEE geoscience and remote sens-
ing letters, 13(1):8–12, 2015. 1
[8] I. Laptev. On space-time interest points. International jour-
nal of computer vision, 64:107–123, 2005. 1
[9] W. Lin, M.-T. Sun, R. Poovandran, and Z. Zhang. Human
activity recognition for video surveillance. 2008 IEEE Inter-
national Symposium on Circuits and Systems (ISCAS), pages
2737–2740, 2008. 1
[10] J. Liu, A. Shahroudy, D. Xu, A. C. Kot, and G. Wang.
Skeleton-based action recognition using spatio-temporal
lstm network with trust gates. IEEE transactions on pattern
analysis and machine intelligence, 40(12):3007–3021, 2017.
1
[11] R. Poppe. A survey on vision-based human action recog-
nition. Image and Vision Computing, 28(6):976–990, 2010.
1
[12] H. Rahmani and A. Mian. 3d action recognition from novel
viewpoints. In Proceedings of the IEEE conference on
computer vision and pattern recognition, pages 1506–1515,
2016. 1
[13] I. Rodomagoulakis, N. Kardaris, V. Pitsikalis, E. Mavroudi,
A. Katsamanis, A. Tsiami, and P. Maragos. Multimodal hu-
man action recognition in assistive human-robot interaction.
2016 IEEE International Conference on Acoustics, Speech
and Signal Processing (ICASSP), pages 2702–2706, 2016. 1
[14] M. D. Rodriguez, J. Ahmed, and M. Shah. Action mach
a spatio-temporal maximum average correlation height filter
for action recognition. In 2008 IEEE conference on computer
vision and pattern recognition, pages 1–8. IEEE, 2008. 1, 2
[15] K. Simonyan and A. Zisserman. Two-stream convolutional
networks for action recognition in videos. Advances in neu-
ral information processing systems, 27, 2014. 2
[16] K. Soomro and A. R. Zamir. Action recognition in realistic
sports videos. In Computer vision in sports, pages 181–208.
Springer, 2015. 1
[17] Y. Wang, Y. Xiao, F. Xiong, W. Jiang, Z. Cao, J. T. Zhou,
and J. Yuan. 3dv: 3d dynamic voxel for action recognition
in depth video. In Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pages 511–520,
2020. 1
[18] R. Yang and R. Yang. Dmm-pyramid based deep architec-
tures for action recognition with depth cameras. In Asian
Conference on Computer Vision, pages 37–49. Springer,
2014. 1
[19] M. Zeng, L. T. Nguyen, B. Yu, O. J. Mengshoel, J. Zhu,
P. Wu, and J. Zhang. Convolutional neural networks for hu-
man activity recognition using mobile sensors. In 6th inter-
national conference on mobile computing, applications and
services, pages 197–205. IEEE, 2014. 1