2020 International Conference on Pervasive Artificial Intelligence (ICPAI)

The Risk Classification of Ergonomic

Musculoskeletal Disorders in Work-related
Repetitive Manual Handling Operations with Deep
Learning Approaches
1st Yu-Wei Chan 2nd Tzu-Hsuan Huang
College of Computing and Informatics Department of Computer Science and Communication Engineering
Providence University Providence University
Taichung, Taiwan Taichung, Taiwan
ywchan@gm.pu.edu.tw s1052852@gm.pu.edu.tw

3rd Yu-Tse Tsan*

Institute of Occupational Medicine, Department of Emergency Medicine
Taichung Veterans General Hospital
Taichung, Taiwan
janyuhjer@gmail.com

4th Wei-Chen Chan 5th Chih-Hung Chang

Institute of Occupational Medicine, Department of Emergency Medicine College of Computing and Informatics
Taichung Veterans General Hospital Providence University
Taichung, Taiwan Taichung, Taiwan
fufu110017@gmail.com ch.chang@pu.edu.tw

6th Yin-Te Tsai

Department of Computer Science and Communication Engineering
Providence University
Taichung, Taiwan
yttsai@pu.edu.tw

Abstract—The injury resulted from the repetitive and load-
bearing works is the most frequent work-related musculoskeletal
disorders (WMSD) or cumulative trauma disorders (CTD). It
comes from the overload of repetitive load-bearing actions, which
resulting in fatigue, inflammation, even injuries of musculoskele-
tal system. According to the annular report of Labor Insurance
Bureau in Taiwan, WMSD is up to 85-88% payment. Thus,
the aim of this study is to evaluate the risk of WMSD during
work by using the simple, quick, and correct methods by using
the deep learning algorithms. In the proposed research method,
after collection the videos of hand repeated movements, the
ergonomic injuries are evaluated by using the 2D human pose
estimation method, which is based on the Key Indicator Method
- Manual Handling Operations (KIM-MHO). Then, a model of
predefined classifications through deep learning approaches for
manual handling operating tasks is built. The analysis results
show that the classification accuracy is more than 80%, compared
with the doctor’s judgment. The goal of this study is to get the
accuracy up to 90%, so as to achieve fast and accurate assistance
for deciding the risk of ergonomics, and immediately give proper
feedback.
Index Terms—ergonomics, work-related musculoskeletal disor-
ders, deep learning, occupational medicine;
I. I NTRODUCTION
Ergonomics [1] is the study of the interaction between
people, tools, machines, equipment and the environment in our
daily lives and work, so that the tools, machines, equipment
and the environment in which they are used can be related
to our own capabilities, limitations and needs. Ergonomics is
the design of the interactions between people, tools, machines,
equipment and the environment to achieve the best possible
fit. Poor ergonomics design has various direct and indirect
effects on workers, including causing/contributing to human
errors, accidents, musculoskeletal disorders, reduced quality
of work and life, poor production performance, and fatigue,
which seriously affect the health, safety, and well-being of
workers.
The causes of musculoskeletal disorders include work load,
work posture, repetition, and work schedule breaks, etc.
Among them, musculoskeletal disorders caused by repetitive

978-1-6654-0483-9/20/$31.00 ©2020 IEEE 268

DOI 10.1109/ICPAI51961.2020.00057

Authorized licensed use limited to: MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 05,2021 at 07:04:32 UTC from IEEE Xplore. Restrictions apply.
 and weight-bearing work are the most common occupational
diseases. Cumulative Trauma Disorders (CTD) is a disease
caused by fatigue, inflammation, and injury to musculoskeletal
or related tissues caused by repetitive work overload, which
accumulates over a long period of time. In Taiwan, since
the occupational safety and health action was amended and
enforced in 2003, it is stipulated that employers should plan
properly to prevent the occurrence of musculoskeletal injuries
and diseases caused by repetitive work. The risk of muscu-
loskeletal injury is increased when workers in various indus-
tries repeatedly engage in a certain work load and complete the
work in unnatural postures for a long period of time [2]–[7].
According to the statistics from 1990 to 2011, repetitive
musculoskeletal injuries and illnesses caused by work in
Taiwan accounted for 85% to 88% of all diseases covered
by workers’ compensation insurance, and they are the most
common occupational diseases. Due to the high prevalence and
long duration of the disease, it is the main factor that causes
long-term disablement of workers and has a great impact on
workers, enterprises, and the national society. For workers,
musculoskeletal injuries and diseases cause mobility problems,
reduced physical strength, poor quality of life, reduced income,
and even hardship. In addition, according to other related
surveys on musculoskeletal injuries, the seriousness of this
problem is also evident. According to the occupational injury
surveys in the United States, Japan, Europe, and Korea, the
number of lost workdays due to cumulative musculoskele-
tal injuries and illnesses accounts for a high proportion of
all occupational injuries and illnesses. The overall loss due
to repetitive musculoskeletal injuries and diseases is about
US$216 billion in the European Union (EU), accounting for
1.6% of EU GDP [4], and about US$168 billion in the
United States (US), accounting for 1.53% of US GDP [5].
According to a survey report [6], the number of work-related
musculoskeletal injuries in Korea increased from 124 in 1998
to 6,234 in 2009. The inclusion of musculoskeletal injuries in
the list of occupational diseases also indicates the prevalence
of musculoskeletal injuries. Therefore, in order to eliminate or
reduce the incidence of work-related musculoskeletal injuries
and diseases, all industrialized countries have been making
great efforts in recent years to promote the prevention of
repetitive musculoskeletal injuries and diseases.
In recent years, Taiwan has been actively promoting pre-
ventive measures, among which, employers should properly
plan for the prevention of musculoskeletal disorders caused by
repetitive work, and the contents should include the following
items:
1) Workflow, content, and action analysis.
2) Confirmation of human factors injuries.
3) Improvement methods and implementation.
4) Effectiveness evaluation and improvement.
5) Other related safety and health issues.
For companies with less than 100 employees, records or
documents are required instead. If the number of workers in a
company is less than 100, records or documents can be kept
269
Authorized licensed use limited to: MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 05,2021 at 07:04:32 UTC from IEEE Xplore. Restrictions apply.
instead. In complex work activities, how to effectively stream-
line and quickly decide the musculoskeletal injury caused by
the work becomes the key point to the implementation of the
plan. This problem is also the main problem of this work.
The goal of this work is to apply the deep learning technol-
ogy to the injury identification and risk assessment of repetitive
musculoskeletal injuries and diseases through the KIM-MHO
[8], [9]. In addition, the machine can automatically learn
and interpret the video recordings of workers’ daily work to
achieve fast, simple, and effective results, which can be used
as a reference for physicians to evaluate the risk of injury.
The rest of the paper is organized as follows. Section II
presents the proposed method. In Section III, the analysis
results are given. Finally, the concluding remark is given in
Section IV.
II. P ROPOSED M ETHOD
In recent years, with the improvement of computing power
of in hardware devices and algorithms, deep learning technol-
ogy has become one of the hot information technologies today.
Compared to the traditional method of manually selecting
specific features, deep learning has made many breakthroughs
and advances in the fields of image recognition and image
recognition, and has been applied to a variety of tasks. In
the field of human action recognition, many video datasets
have been published for researchers to study and use as
benchmarks for model accuracy, such as HMBD51, UCF101,
kinetics human action video datasets, and so on. These model
frameworks do not select features manually as input, but use
the video itself or different forms such as the theoretical flow
and the RGB difference of the continuous frame as model
input. Fully online learning on how to use the extracted
features to classify the final result.
In this study, we used the 2D human pose estimation
method [?], [10]–[12] to mark the coordinates of wrist, elbow,
shoulder, head, and foot in the video, and normalized all the
coordinates to the range of 0 1. The framework of the model
takes these coordinates as inputs, and learns to transform them
into features that can determine the risk level; these features
are transformed from the inputs and combined with data such
as the type of job, total number of working days, total distance
worked, load, etc., to determine the risk level of this operation
by using the holo-linked layer.
A. Data sets
The data sets mainly come from the following sources:
1) Raising and transporting action videos by public data
collection: The CRCV-University of Central Florida
provides 25 types of action videos, 101 types of action
videos, and each type of action video is composed of
4 7 videos. In this study, we will use public data to
determine the movements in the videos.
2) Video recordings of lifting and moving exercises by
researchers themselves: The video is recorded at 30fps,
640x480 or 480x640, and lasts 20 40 seconds. When
recording the video, the entire body of the operator must
 be filmed so that the overall posture and movement
of the operator can be identified from the video. The
recorded videos are classified into four categories: low,
medium, medium-high, and high-risk by using the KIM-
MHO (Kinetic Material Handling Checklist) as a key
indicator. Due to the lack of data, in order to have
sufficient data for training and sufficient data to reflect
the generalized performance of the model for model
selection, 50% of the data are used as training set and
the other 50% are used as validation set for model
performance after training.
3) Video recording of hand movements for clinical health
services: The occupational medicine physician conducts
on-site interviews with the employees in the course of
their work, and makes 20-40 seconds video recordings of
the entire body of the employees for lifting and carrying
operations.
B. Operation processes
The overall operation of the proposed method is divided
into the following four steps:
1) Data pre-processing: We use the OpenPose API to
retrieve the coordinates of key points of human body
from the video, because sometimes the key points of
human body cannot be marked correctly from the frame.
After filling in the missing values, divide the coordinates
by the video height and width, and normalize all coor-
dinates to a range of 0-1 for better learning and faster
training.
2) Data augmentation: Since it is not easy to record video
data and the amount of data is too small, we randomly
pan the body coordinates horizontally and vertically to
produce data about the human body in different parts
of the video; and zoom in and out to represent different
sizes of the human body.
3) Model structure: We select a fixed number of frames
from each movie, and input the coordinates of key points
of the human body in the frames at each time point as
one-dimensional data, such as (x1, y1), (x2, y2), (x3, y3)
combined into [x1, y1, x2, y2, x3, y3]. After the one-
dimensional (1D) convolution model, the characteristics
are combined with other values such as the load, the
total number of working days, etc., through the full
concatenation layer and the Softmax function to obtain
four values between 0 and 1, [p1, p2, p3, p4], which
represent the probability of each risk level.
4) Training process: Since the number of data for each
category is uneven, we use a weighted random selec-
tion method to select the training samples, so that the
probability of each category being selected is equal,
and the probability of each sample within the same
category being selected is also equal. During training,
the cross-entropy loss which is presented by Eq. (1)
between the predicted and actual values is reduced by the
gradient descent of the SGD. Set batch size=54, learning
rate=0.01, training response=400 to train, and train with
270
Authorized licensed use limited to: MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 05,2021 at 07:04:32 UTC from IEEE Xplore. Restrictions apply.
the best model parameter on the validation set to achieve
the best training result.
N
1 
J(w) = H(pn , qn ), (1)
N n=1
the equation (1) can be presented in detail as follows:
N
1 
J(w) = [yn log ŷn + (1 − yn )log(1 − ŷn )] (2)
N n=1
III. T HE A NALYSIS R ESULTS
A. Results of the risk assessment
In the section, we will introduce the risk assessment tool for
determining the risk level of human-induced injury by using
the KIM-MHO. The key indicators considered in the KIM-
MHO which are referred by [13] are given as follows:
1) Total duration of manual handling operations per shift.
2) Type, duration and frequency of force exertion(s).
3) Force transfer/gripping conditions.
4) Hand-arm position and movement.
5) Working organization.
6) Working conditions (e.g. noise, climatic conditions).
7) Body posture.
Different key indicators are classified according to different
levels. The total value can be calculated by multiplying the
daily activity time scale value (KI 1) by the sum of other
scale scores (KI 2 to KI 7). This score can be assigned to
one of the four risk categories, which is characterized by the
increasing intensity of manual operations and the increasing
risk estimates of overload and work-related adverse health
effects [13]:
• The risk category 1 (less than 10 points): Low exposure,
body overload caused by MHO is unlikely.
• The risk category 2 (10–24 points): With increased ex-
posure, certain groups of workers may be physically
overloaded due to MHO. Redesigning the workplace may
be especially useful for this group.
• The risk category 3 (25–49 points): Exposure is highly
increased, and there is a high possibility of body over-
load due to MHO. It is recommended to redesign the
workplace.
• The risk category 4 (greater than and equal to 50 points):
Under high exposure conditions, body overload due to
MHO is likely to occur. Redesigning the workplace is
necessary.
The KIM-MHO worksheet includes a short manual that ex-
plains how to conduct the assessment. The manual is provided
in [14].
B. The analysis results of the proposed method
This study was conducted in collaboration with the Depart-
ment of Occupational Medicine at Veterans General Hospital
in Taichung, and focused on the upper body tapping action
based on the KIM-MHO method for the classification of risk
 assessment of injury level. First, the video of the video record-
ing of the percussion action was made by our researchers.
Then, after the video input is modeled, the probability of
each of the 4 categories will be obtained and the category with
the highest probability will be selected as the classification
result. In the training process, the accuracy of the training
data and the validation data is calculated after each round of
training, and the model performance is based on this data. The
accuracy of the classification results was over 80%, which was
shown in Figure 1.
Fig. 1. The equation of cross-entropy loss.
IV. C ONCLUSIONS
In this work, we have proposed a deep learning approach
based on the 2D human pose estimation methods for train-
ing the recognition of occupational human factors injuries
and combines them with the commonly used KIM-MHO
worksheet. This work was performed by cooperating with
the doctors of the department of occupational medicine at
veterans general hospital in Taichung, Taiwan. The study
mainly focused on the upper body tapping movement of the
workers, and used the KIM-MHO assessment method to judge
the risk level of injury. With the proposed approach, the risk
level of injury was determined automatically, so as to assist the
doctors to judge the risk level efficiently. The current research
results were able to reach an accuracy of more than 80% for
the classification of injury level assessment with the doctor’s
judgment.
In the future work, we will continue our previous collabora-
tive research results and use deep learning methods to develop
a model for identifying and classifying lifting, holding, and
carrying operations on the KIM-LHC scale. The goal of this
project is to achieve a classification accuracy of more than
90% compared to that of expert interpretation.
271
Authorized licensed use limited to: MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on June 05,2021 at 07:04:32 UTC from IEEE Xplore. Restrictions apply.
R EFERENCES
[1] S. Schneider and P. Susi, “Ergonomics and construction: a review of
potential hazards in new construction,” American Industrial Hygiene
Association Journal, vol. 55, no. 7, pp. 635–649, 1994.
[2] S. P. Tsai, E. L. Gilstrap, S. R. Cowles, L. C. Waddell Jr, and C. E.
Ross, “Personal and job characteristics of musculoskeletal injuries in
an industrial population.” Journal of occupational medicine.: official
publication of the Industrial Medical Association, vol. 34, no. 6, pp.
606–612, 1992.
[3] B. Cagnie, L. Danneels, D. Van Tiggelen, V. De Loose, and D. Cambier,
“Individual and work related risk factors for neck pain among office
workers: a cross sectional study,” European Spine Journal, vol. 16, no. 5,
pp. 679–686, 2007.
[4] J. Å. Johansson, “Work-related and non-work-related musculoskeletal
symptoms,” Applied Ergonomics, vol. 25, no. 4, pp. 248–251, 1994.
[5] B. P. Bernard and V. Putz-Anderson, “Musculoskeletal disorders and
workplace factors; a critical review of epidemiologic evidence for work-
related musculoskeletal disorders of the neck, upper extremity, and low
back,” 1997.
[6] K. H. Kim, K. S. Kim, D. S. Kim, S. J. Jang, K. H. Hong, and S.-W. Yoo,
“Characteristics of work-related musculoskeletal disorders in korea and
their work-relatedness evaluation,” Journal of Korean medical science,
vol. 25, no. Suppl, pp. S77–S86, 2010.
[7] M. Ayoub, J. Selan, W. Karwowski, and H. Rao, “Lifting capacity
determination. paper in back injuries,” in Proceedings: Bureau of Mines
Technology Transfer Symposia, Pittsburgh, PA, 1983, pp. 54–63.
[8] J. O. Crawford, “The nordic musculoskeletal questionnaire,” Occupa-
tional medicine, vol. 57, no. 4, pp. 300–301, 2007.
[9] M. M. El-Bassoussi, “A biomechanical dynamic model for lifting in the
sagittal plane.” Ph.D. dissertation, Texas Tech University, 1974.
[10] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2d
pose estimation using part affinity fields,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2017, pp. 7291–
7299.
[11] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool,
“Temporal segment networks: Towards good practices for deep action
recognition,” in European conference on computer vision. Springer,
2016, pp. 20–36.
[12] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijaya-
narasimhan, F. Viola, T. Green, T. Back, P. Natsev et al., “The kinetics
human action video dataset,” arXiv preprint arXiv:1705.06950, 2017.
[13] A. Klussmann, F. Liebers, H. Gebhardt, M. A. Rieger, U. Latza, and
U. Steinberg, “Risk assessment of manual handling operations at work
with the key indicator method (kim-mho)—determination of criterion
validity regarding the prevalence of musculoskeletal symptoms and
clinical conditions within a cross-sectional study,” BMC musculoskeletal
disorders, vol. 18, no. 1, p. 184, 2017.
[14] A. Klussmann, U. Steinberg, F. Liebers, H. Gebhardt, and M. A. Rieger,
“The key indicator method for manual handling operations (kim-mho)-
evaluation of a new method for the assessment of working conditions
within a cross-sectional study,” BMC musculoskeletal disorders, vol. 11,
no. 1, pp. 1–8, 2010.
[15] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning
spatiotemporal features with 3d convolutional networks,” in Proceedings
of the IEEE international conference on computer vision, 2015, pp.
4489–4497.
[16] G. Varol, I. Laptev, and C. Schmid, “Long-term temporal convolutions
for action recognition,” IEEE transactions on pattern analysis and
machine intelligence, vol. 40, no. 6, pp. 1510–1517, 2017.