Human Activity Recognition with HMM-DNN Model

Licheng Zhang, Xihong Wu and Dingsheng Luo*

Key Lab of Machine Perception (Ministry of Education), Speech and Hearing Research Center
School of Electronics Engineering and Computer Science, Peking University
Beijing, 100871, China
Emails: {zhanglc.wxh.dsluo}@cis.pku.edu.cn
* Corresponding author

Ab stract-Activity recognition commonly made use of hidden
Markov models (HMMs) to exploit temporal
between activities. The emission distribution of HMMs could be
represented by generative models, such as Gaussian mixture
models (GMMs), or discriminative models, such as random forest
(RF). These models, especially discriminative ones, needed to
manually extract features from the sensor data, which relied on
the experience of the researchers, and usually was a time­
consuming task when complicated features
Furthermore, with these methods, the process of quantization of
the sensor data, i.e., manual feature extraction, might lose much
useful information and thus led to a performance debasement. In
this paper, we recommend deep neural networks (DNNs) for
modeling the emission distribution ofHMMs, which automatically
learn features suitable for classification from the raw sensor data
and then estimate the posterior probabilities of the HMM states.
We collected a dataset of daily activities and based on which
experiments were performed to compare our HMM-DNN model
with bothHMM-GMM andHMM-RF. The results illustrated that
HMM-DNN outperformed both HMM-GMM and HMM-RF.
extracted features. They compared the performance of HMM­
dependencies GMM with that of a large amount of static classifiers, which did
not take the temporal dependencies between activities into
consideration, including GMM, NaIve Bayes (NB), logistic
regression (LR), DT, support vector machine (SVM), k-nearest
neighbor (KNN), artificial neural network (ANN), and so on.
The results showed that HMM-GMM performed better than
static classifiers. Ataya et al. [5] collected triaxial accelerometer
are extracted.
data of nine activities and extracted several features for each
frame, including mean, energy, correlation, and so on. Then they
used several models to model the emission distribution of
HMMs, including DT, RF, SVM, KNN, GMM, and AdaBoost.
The results showed that taking the temporal dependencies
between activities into account improved classification
performance, and discriminative classifiers, such as HMM­
SVM, performed better than the generative one (HMM-GMM),
and among the discriminative models, HMM-RF performed
better than other models, i.e., HMM-DT, HMM-SVM, HMM­
KNN, HMM-AdaBoost.

Keywords-activity recognition; deep neural networks; hidden
Markov models; sensor data; accelerometer
I. INTRODUCTION
Activity recognition usually made use of hidden Markov
models (HMMs) to model temporal dependencies between
consecutive activities [1-5]. One critical step of activity
recognition with HMMs is to model the emission distribution of
HMMs. The emission distribution could be modeled by
generative models, such as Gaussian mixture models (GMMs)
[1, 2], or discriminative models, such as decision stump (DS) [3],
decision tree (DT) [4], random forest (RF) [5], and so on. These
models that modeled the emission distribution of HMMs,
especially discriminative models, needed to manually extract
features from the sensor data, which relied on the experience of
the researchers, and then modeled the extracted features. For
example, Lester et al. [3] collected sensor data and extracted
features from the data, including mean, variances, and so on.
Then they used DS and GMM to model the emission distribution
of HMMs based on the extracted features. The results showed
that HMM-DS performed better than HMM-GMM. Mannini et
al. [1] used a dataset of acceleration in [6] for activity
recognition. They extracted four kinds of features for each data
frame, including the DC component of acceleration which was
estimated by taking average of the data samples within each
frame, the energy, the frequency-domain entropy, and the
correlation coefficients. Then they used GMM to model the
From the aforementioned examples, we can find that
previous works needed to manually extract features from the
sensor data, which relied on the experience of the researchers.
The problem of these methods lies in that the quantization of the
sensor data, i.e., manual feature extraction, might lose much
useful information that would improve classification
performance.
In this paper, we recommend deep neural networks (DNNs)
for modeling the emission distribution of HMMs, which
automatically learn features suitable for classification from the
raw sensor data and then estimate the posterior probabilities of
the HMM states. We collected an acceleration dataset of daily
activities using the accelerometer on a smartphone and did
experiments on this dataset. We used DNN, GMM, and RF to
model the emission distribution of HMMs respectively, and
compared the performance of our HMM-DNN method with that
of HMM-GMM and HMM-RF. The experimental results
showed that HMM-DNN performed better than HMM-GMM
andHMM-RF.
The remainder of this paper is organized as follows. Section
II describes the related work. Section III gives a brief description
of HMMs and detail how to use DNN to model the emission
distribution ofHMMs. Section IV introduces the data collection
process, details the experimental design and gives the results of
experiments. Section V summarizes our conclusion and gives
the future work.

Proc. 2015111E 14th Int'! Conf. on Cognitive Informatics & Cognitive Computing IICCI'CC15J
H. Ge,l.lu, Y. Wang, H. Howard, P. Chen, II. Tao, a.Zhang, & U.Zadeh [Eds.!
918+4613·1290·91151$31.00 ©2015 IEEE

192
 II. RELATED WORK SVM, KNN, GMM, and AdaBoost. They compared the
performance of these models as follows: HMM-DT, HMM-RF,
Activity recognition that used HMMs to take the temporal
HMM-SVM, HMM-KNN, HMM-GMM, HMM-AdaBoost, and
dependencies between activities into consideration and used
static classifiers, i.e., DT, RF, SVM, KNN, GMM, and
generative models or discriminative models to model the
AdaBoost, which did not consider the temporal dependencies
emission distribution of HMMs has been researched for many
between activities. The results showed that taking the temporal
years. Lester et al. [3] collected sensor data using a multi-sensor
dependencies between activities into consideration improved
board mounted on the shoulder of the subject. They extracted a
classification performance, and discriminative classifiers such
total of 651 features for each frame. Then they selected several
as HMM-SVM, performed better than the generativ � one
features from them that enabled the classifiers to distinguish well
(HMM-GMM), and among the discriminative models, HMM­
betw�en activities. The recognized activities were sitting,
RF performed better than other models, i.e., HMM-DT, HMM­
standmg, walking, jogging, walking upstairs, walking
SVM, HMM-KNN, HMM-AdaBoost. Ronao et al. [8] used an
downstairs, riding a bicycle, driving car, riding elevator down,
acceleration and gyroscopic sensor dataset, namely Hwnan
and riding elevator up. They did several experiments. First of all,
Activity Recognition Using Smartphones Dataset from [9],
they compared the performance of static classifiers, i.e., a
collected using the accelerometer and gyroscope on a
discriminative DS and a generative NB, and the results showed
smartphone, for activity recognition. The sampling rate was 50
that the discriminative model performed better. Then they used
Hz. The frame length was 2.56 s and two consecutive frames
HMM to model the temporal dependencies between activities
overlapped by 50%. 561 features were extracted for each data
and used GMM and DS to model the emission distribution of
frame. The researchers used random forest variable importance
HMMs, and the results showed that HMM-DS performed better
measures to select important features and obtained a feature
than DS and NB, but DS and NB performed better than HMM­
subset. Six activities were recognized, including walking,
GMM. Mannini et al. [1] used an acceleration dataset from MIT
walki�g upstairs, walking downstairs (moving activities), sitting,
[6] for activity recognition. The sampling frequency was 76.25
standmg, and lying (stationary activities). They proposed two­
Hz. The frame length was 6.7 s. Two consecutive frames
stage HMMs for activity recognition. Specifically, they used
overlapped by 50%. They extracted four kinds of features for
each �ata frame, the DC component, the energy, the frequency­
GMM to model the selected features and HMM to model the
temporal dependencies between activities. HMM-GMM frrst
domam entropy, and the correlation coefficients. Then they used
classified a data frame into moving or stationary subclasses.
HMM to model the temporal dependencies between activities
Then, for each subclass, HMM-GMM classified the data frame
and used GMM to model the emission distribution of HMMs.
Th� rec?gn �zed activi�ies were siting, lying, standing, walking,
into a specific activity. They compared the performance of the
two-stage HMM model with that of DT, NB, and ANN. The
starr clrrnbmg, runmng, and cycling. They compared the
results showed that the two-stage HMM-GMM model
performance of HMM-GMM with a large amount of static
performed best.
classifiers, which did not consider the temporal dependencies
between activities, including GMM, NB, LR DT, SVM, KNN, The above works needed to manually extract features from
ANN, and so on. The results showed that HMM-GMM the sensor data relying on the experience of the researchers, and
performed better than static classifiers. Lee et al. [7] collected then modeled the emission distribution ofHMMs based on these
acceleration data from a triaxial accelerometer on a smartphone. features. The common problem of the above works is that the
The sampling rate of the accelerometer was 12 Hz and the frame quantization of the sensor data, i.e., manual feature extraction
length was five seconds, i.e., 60 samples. They used GMM to might lose much useful information that would improv �
model the raw acceleration data of each data frame. They also classification performance.
made use ofHMM to model the temporal dependencies between
activities. They recognized five activities: standing, running, In this paper, we recommend DNNs for modelling the
walking, ascending, and descending. They compared the emission distribution of HMMs, which automatically learn
performance of HMM-GMM with that of a static classifier features suitable for classification from the raw sensor data and
namely ANN. The results showed that HMM-GMM performe d then estimate the posterior probabilities of the HMM states. We
better than ANN when recognizing standing, running, walking, did experiments on our collected acceleration dataset and proved
and ascending, while ANN performed better than HMM-GMM that our recommended method performed better than traditional
when recognizing descending. Ataya et al. [5] collected methods.
acceleration data using a triaxial accelerometer sensor placed at
III. HMM-DNN BASED ACTIVITY RECOGNITION
the hip level. The sampling rate was 100Hz. Nine activities were
recognized, including lying down, slouching/sitting, standing We recommend HMM-DNN for activity recognition.
(stati� activities �, stamping, cycling, running, slow walking, fast Specifically, we use HMM to model the temporal dependencies
walkmg, and usmg stairs (dynamic activities). The frame length between activities and use DNN to model the emission
was 2 s. Two consecutive frames overlapped by 50 percent. probabilities ofHMMs.
They used a normalized Signal Magnitude Area value to
In this part, we first give a brief description of HMMs and
distinguish static and dynamic activities. For static activities,
then detail how to use DNN to model the emission distribution
they continued to extract 14 features for each data frame
including mean, energy, and so on. For dynamic activities, the ; ofHMMs.

extracted 18 features, including entropy, correlation between A. Using HMMs to model the temporal dependencies
axes, spectral energy, and so on. They used several models to HMMs are probabilistic models that model time sequences.
model the emission distribution of HMMs, including DT, RF, A HMM consists of a sequence of hidden states and a sequence

193
 Input Hidden Output
la er layers layer

Fig. 1. The graphical representaion of a HMM

of observations. The graphical representation of a HMM is

shown in Fig. 1. At each time step t, there is a hidden variable
Zt and an observed variable Xt in the HMM. In the field of
activity recognition from sensor data, the hidden variable is the
activities performed, such as walking and running, and the
observed variable is the data vector of each frame. HMMs make
two basic assumptions,

• The current hidden state depends only on the previous

hidden state.

• The observed variable xt is governed only by the
corresponding hidden state Zt.
A HMM is determined by prior probability distribution,
transition probability distribution and observation probability
distribution. The probability density P(Zl) is the prior
probability distribution. The conditional probability density
p(ZtIZt-l) is the transition model and the conditional
probability density p(xtIZt) is the observation model. Training
a HMM is to maximizing the joint probability P(X1:T,Zl:T),
with T to be the total time steps. Based on the above two
assumptions, the joint distribution is calculated using the above
three distributions as follows [10]:
T T
P(X1:T,Zl:T) =
P(Zl) n
t
p(ZtIZt-l) n p(xtlzt)· (1)
=2 t=l
After the training of the HMM, we get the parameters of the
HMM. Then, we used Viterbi algorithm [11] to infer the label
sequences that best explain a new sequence of observations.
Fig. 2. The graphical representaion of a DNN
parameters, DNNs possess the ability of automatically learning
features suitable for classification from the raw sensor data.
In a DNN, the data vector of a frame is fed into the input
units. For each hidden unit j, its activation probability Yj is
calculated using the inputs from the previous layer as follows
[12]:
where bj is the bias of unit j, i is an index over units in the
previous layer, Yi is the input of unit i in the previous layer, and
Wij is the weight on the connection from unit i to unit j. Then
the activation probabilities of the units of this layer are taken as
the inputs of the next layer. For multiclass classification, output
unit j uses "softmax" to convert the inputs from the previous
layer into a class probability P{

B. Using DNNs to model the emission distribution ofHMMs

One critical step of training a HMM is to learn the
observation model p(xtIZt). In this paper, we recommend
DNNs for modeling the observation probability distribution of
HMMs.
DNNs are a kind of the neural network model. It has an input
layer, two or more hidden layers, and an output layer. The
graphical representation of the DNN model is shown in Fig. 2.
The nodes represent the variables and the links between the
nodes represent the weight parameters. Arrows denote the
information flow direction through the network. Compared with
the traditional three-layer ANN, a DNN has many more
parameters because it has many more hidden layers and each
hidden layer has many more units. Due to the large amount of
where k is an index over all classes.
The parameters of a DNN, including the weights and the
biases, can be initialized using the method of random
initialization or using the method of pre-training by treating
every two adjacent layers as a Restricted Boltzmann Machine
(RBM), which was proposed by Hinton in [13]. After parameter
initialization, the whole network is trained using back­
propagation algorithm with the information of the labels, to get
the parameters optimized.
After DNN training, the DNN outputs the class probabilities

194
 Accuracy
0.9352

0.9 0.8488 0.8569

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0. 1

0
GMM I-IMM-GMM RF HMM-RF DNN I-IMM-DNN

Fig. 3. The accuracy of different methods

p(ZtIxt) given an observation xt at time step t. If the number of consecutive frames overlapped by 50 percent. We took the data
classes is N, p(ZtIxt) is (Pi' Pz, ... , PN) calculated using (3). in the fourth collection as the testing data. The data of the other
three collections were taken as the training data. TABLE I gives
But HMMs require p(xtlzt) as is shown in (1). We use
the details of the training set and testing set. The fIrst column
Bayes' theorem to calculate the emission probabilities required
represents different activities, the second column gives the
byHMMs:
number of frames of different activities in the training set, and
the third column gives the number of frames of different
(4) activities in the testing set. In the last row of TABLE I, we give
the total number of frames of training set and testing set
It should be noted that p(xt) has no effect on the results of respectively.
inferring the label sequences. Therefore, the emission
probabilities can be calculated by dividing the DNN outputs by
B. Experimental Design
the total number of the corresponding class. We compared our HMM-DNN method with traditional
methods. Among the traditional methods, HMM-RF and HMM-
IV. EXPERIMENTS
In this part, we introduce the data collection process, detail
the experimental design and give the results of experiments. TABLE !. THE AMOUNT OF DATA OF DIFFERENT ACTIVITIES IN OUR
DATASET
A. Data Collection
We collected acceleration data using the accelerometer on an
Number offrames Number offrames
android smartphone. The smartphone was attached to the left Activities
in the training set in the testing set
upper arm of one subject. The whole process is performed as
follows. The subject fIrst rode a bike to a building. After arriving
at the building, the subject began to walk and walked into the Riding 839 238
building. Then the subject walked upstairs to the eighth floor.
Then he walked into the office of his tutor and stood to listen to Walking 582 199
the tutor. After the teaching of the tutor, the subject walked out
of the office and then went downstairs to the ground floor. Then
Waking upstairs 783 237
he walked out the building and [mally rode the bike away from
the building. Five activities were considered: riding, walking,
walking upstairs, standing, and walking downstairs. The whole Standing 979 19 1
process is described as follows: riding-walking-walking upstairs
-walking-standing-walking-walking downstairs-walking-riding. Walking
797 246
downstairs
The process was performed for four times when the tutor asked
the subject to go to his offIce and taught him. The sampling rate
To tal amount 3980 1111
is 42 Hz. We set the frame length to be 1 s, which was proved to
be sufficient to distinguish different activities [14], and two

195
 Precision

0.9337
0.9 103
0.9 0.8472

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

o
GMM HMM-GMM RF HMM-RF DNN HMM-DNN

Fig. 4. The precision of different methods

GMM performed better than other methods as was shown in [1,
5, 9]. Therefore, we chose HMM-RF and HMM-GMM as the
baseline. We extracted four kinds of features for HMM-RF and
HMM-GMM: the DC component, the energy, the frequency­
domain entropy, and the correlation coefficients, which were
used in [1, 2]. For RF, the number of trees was set to 50 as was
used in [5]. For GMM, the number of components in the mixture
was set to 10. More components have no significant impact on
the accuracy.
For our HMM-DNN method, the parameters of the DNN
were randomly initialized. The DNN was trained using
stochastic gradient decent with a mini-batch size of 100 training
samples. We ran 1000 epochs at a learning rate of 0.1 with a
weight decay of 0.0002. An epoch refers to passing over all the
mini-batches, i.e., the entire training set. We adjusted the
structure of the DNN to achieve the best performance, including
the number of hidden layers and the number of units in each
hidden layer. When we adjusted the number of units of one layer,
we keep the number of units of the other layers fixed. For
example, when we adjusted the number of units of one of the
hidden layers, we fixed the number of units of the other hidden
layers and first set the number of units to be a small value such
as 50 and a large value such as 3072 respectively. Then we
gradually increased the value from 50 by a small value such as
100 and decreased the value from 3072 by a small value such as
100 and observed the results of experiments. When we found a
small interval for the optimal solution, for example, between 400
and 600, we set the small value for increasing and decreasing to
be smaller such as 10 rather than 100. Finally, the DNN that
achieved the best recognition accuracy had a structure of 126-
500-500-2000-5. That is to say, it had 126 units in the input layer,
500 units in the first hidden layer, 500 units in the second hidden
layer, 2000 units in the third hidden layer, and 5 units in the out-

Recall

0.9322
0.8656
0.9 0.8402

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0. 1

0
GMM HMM-GMM RF HMM-RF DNN HMM-DNN

Fig. 5. The recall of different methods

196
put layer, representing 5 categories. It should be noted that 126
data points of the inputs consist of 42 samples (1 second length)
of each axis for three axes.
During HMM training, we performed Laplace smoothing on
the prior and the transition model with a small value of 0.01, to
avoid zero probability.
C. Results
In this part, we give the results of HMM-GMM, HMM-RF
andHMM-DNN. Besides, we give the results of static classifiers,
i.e., GMM, RF and DNN, which did not consider the temporal
dependencies between activities. The accuracies of HMM­
GMM, HMM-RF, HMM-DNN and static classifiers are
presented in Fig. 3. The precisions of the different methods are
presented in Fig. 4. And the recalls of these methods are
presented in Fig. 5.
The results of HMM-GMM, HMM-RF and HMM-DNN
show that the accuracy, the precision and the recall of HMM­
DNN are all higher than those of HMM-GMM and HMM-RF.
The reason that our HMM-DNN method perform better than
traditional methods, i.e., HMM-GMM andHMM-RF, lies in that
manual feature extraction by traditional methods lose much
useful information that would improve classification
performance, while DNN used for modeling the emission
distribution by our HMM-DNN method automatically learn
features suitable for classification from the raw sensor data and
give better emission distribution ofHMMs. Besides, we can fmd
that bothHMM-DNN andHMM-RF perform better thanHMM­
GMM, and both DNN and RF perform better than GMM. It
proves once again that discriminative models perform better
than the generative one, which has been concluded in [3, 5]. In
addition, we can fmd that HMM-GMM perform better than
GMM, and HMM-RF perform better than RF, and HMM-DNN
perform better than DNN. It proves once again that taking the
temporal dependencies between activities into consideration
improves activity recognition performance, which has been
concluded in [5].
V. CONCLUTTON AND FUTURE WORK
Traditional methods needed to manually extract features
from the sensor data, which relied on the experience of the
researchers. The problem of these methods lies in that the
quantization of the sensor data, i.e., manual feature extraction
might lose much useful information that would improve the
classification performance. In this paper, we recommend DNNs
for modeling the emission distribution of HMMs, which
automatically learn features suitable for classification from the
raw sensor data. We collected an acceleration dataset and did
experiments on this dataset. We used DNN, GMM, and RF to
model the emission distribution of HMMs respectively, and
compared the performance of HMM-DNN with that of HMM­
GMM and HMM-RF. The results showed that HMM-DNN
performed better than HMM-GMM and HMM-RF.
In the future, we will make our collected acceleration dataset
public, and serve as a public dataset for researchers to compare
different methods in the field of activity recognition. Besides,
we will try to make use of recurrent neural networks to consider
more information in the past to contribute to the interpretation
197
of the future, and thus further improve the classification
performance. In addition, we will study whether the number of
hidden layers in DNNs would be the larger the better and
whether the number of units in each layer would be the larger
the better.
ACKNOWLEDGMENT
The work is supported in part by National Basic Research
Program (973 Program) of China (No. 2013CB329304), the
National Natural Science Foundation of China (No. 90920302,
No. 91120001, No.61121002), a "Twelfth Five-Year" National
Science & Technology Support Program of China (No.
2012BAIl2BOl), the Seeding Grant for Medicine and
Information Sciences of Peking University (No. 2 0 1 4- MI- I 0)
and the Key Program of National Social Science Foundation of
China (No. 12&ZD119).
REFERENCES
[I] A. Mannini and A. M. Sabatini, "Machine learning methods for
classifying human physical activity from on-body accelerometers,"
Sensors,vol. 10,no. 2,pp. 1 154- 1 175,2010.
[2] A. Mannini and A. M. Sabatini, "Accelerometry-based classification of
human activities using markov modeling," Computational intelligence
and neuroscience,vol. 20II,no. 4,20II.
[3] 1. Lester, T. Choudhury, N. Kern, G. Borriello, and B. Hannaford, "A
Hybrid Discriminative/Generative Approach for Modeling Human
Activities," Proceedings of the 19th international joint conference on
Artificial intelligence,vol. 5,pp. 766-772,2005.
[4] T. Maekawa, Y. Kishino, Y. Sakurai, and T. Suyama, "Activity
recognition with hand-worn magnetic sensors," Personal and ubiquitous
computing,vol. 17,no. 6,pp. 1085-1094,2013.
[5] A. Ataya, P. Jallon, P. Bianchi, and M. Doron, "Improving activity
recognition using temporal coherence," In 35th Annual International
Conference of the IEEE Engineering in Medicine and Biology Society
(EMBC),pp. 4215-4218,2013.
[6] L. Bao and S. S. Intille, "Activity recognition from user-annotated
acceleration data," Pervasive Computing, Lecture Notes in Computer
Science,vol. 300 1,pp. 1- 17,2004.
[7] Y. S. Lee and S. B. Cho, "Activity recognition using hierarchical hidden
markov models on a smartphone with 3D accelerometer," In Hybrid
Artificial Intelligent Systems, Lecture Notes in Computer Science,
Springer Berlin Heidelberg, vol. 6678,pp. 460-467,20 1 1.
[8] C. A. Ronao and S. B. Cho, "Human activity recognition using
smartphone sensors with two-stage continuous hidden Markov models,"
In 10th IEEE International Conference on Natural Computation (ICNC),
pp. 68 1-686,2014.
[9] D. Anguita,A. Ghio,L. Oneto,X. Parra,and 1. L. Reyes-Ortiz,"A public
domain dataset for human activity recognition using smartphones,"
European Symposium on Artificial Neural Networks (ESANN),pp. 437-
442,2013.
[ 10] K. P. Murphy, Machine learning: a probabilistic perspective,MIT press,
20 12.
[II] L. R. Rabiner, "A tutorial on hidden markov models and selected
applications in speech recognition," Proceedings of the IEEE,vol. 77,no.
2,pp. 257-286, 1989.
[ 12] G. Hinton,L. Deng,D. Yu,G. E. Dahl,A. Mohamed,N. Jaitly,A. Senior,
V. Vanhoucke,P. Nguyen,T. N. Sainath,and B. Kingsbury,"Deep neural
networks for acoustic modeling in speech recognition," Signal Processing
Magazine,IEEE,vol. 29,no. 6,pp. 82 -97,nov. 20 12.
[ 13] G. Hinton, S. Osindero, and Y. W. Teh, "A fast learning algorithm for
deep belief nets," Neural computation,vol. 18,pp. 1527-1554,2006.
[ 14] T. PlOtz, N. Y. Hammerla, and P. Olivier, "Feature learning for activity
recognition in ubiquitous computing," Proceedings of the Twenty-Second
international joint conference on Artificial Intelligence,vol. 22,pp. 1729-
1734,20 1 1.