2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)
Human Fall Detection during Activities of Daily
Living using Extended CORE9
Shobhanjana Kalita Arindam Karmakar
Dept. of Computer Science & Engg Dept. of Computer Science & Engg
Tezpur University, Assam, India Tezpur University, Assam, India
kalitas@tezu.ernet.in arindam@tezu.ernet.in
Abstract—There is an ever-increasing need for automated mon-
itoring systems to enable independent living in today’s elderly
population. Human fall detection is widely researched within
the field of assistive technologies. However, fall detection systems
based on wearable sensors is often incapacitated by inadvertent
neglect to wear the sensors. Computer vision based approaches
to detect a human fall in video holds promise. It has been
noted that an informative yet compact representation schema can
significantly improve the performance of video understanding.
For human activity recognition, Extended CORE9 has been used
for obtaining a qualitative spatial description of the video activity.
The spatial description of an activity obtained using Extended
CORE9 along with the temporal information can be encoded
within a graph structure. Extended CORE9 has been applied for
human-human and human-object interactions. In this paper, we
show how Extended CORE9 can be applied for single-person
activities. We evaluate our approach for detecting fall in an
assisted living environment. Experiments performed on the UR
Fall Detection dataset shows promising results.
Index Terms—Human fall detection, temporal graph, activities
of daily living, Extended CORE9
I. I NTRODUCTION
One of the fundamental aims of assistive technology is to
provide better health care to those in need. Such technologies
are expected to allow elderly, disabled, over weight or obese
people to live independently in their own home without chang-
ing their life style. To make that possible there is an increasing
demand for intelligent monitoring systems that can detect
emergency situations and notify concerned relatives or health-
care representatives [1]. This makes human fall detection
(during activities of daily living) an important functionality
in assisted living technologies. Research has shown that falls
are one of the main reasons for injuries and even death for
seniors or people in need [1]. Monitoring systems that are
able to detect human falls and raise appropriate alerts are being
widely researched.
Existing approaches for human fall detection can be widely
categorized into two groups: non-vision based and vision
based [2]. Non vision based technologies use a variety of
sensors, principal among which is the accelerometer. However,
such approaches require the patient or elderly being monitored
to actively cooperate or participate by wearing the sensors at
all times. This may be an inconvenience, because of several
reasons - the person may find it uncomfortable or forget to
wear it. Fall detection systems based on wearable sensors
978-1-5386-7989-0/19/$31.00 ©2019 IEEE
Shyamanta M Hazarika
Dept. of Mechanical Engg
IIT Guwahati, Assam, India
s.m.hazarika@iitg.ac.in
is often incapacitated by inadvertent neglect to wear the
sensors [3]. On the other hand, vision based approaches use
only video data and therefore are much less intrusive. Further,
it does not require active participation of the person being
monitored.
Vision-based fall detection may use videos from a single
RGB camera, multiple RGB cameras for depth perception or
depth cameras. Fall detections using a single RGB camera
have been extensively studied, as such systems are easy to set
up and are inexpensive [2]. In this paper we present a vision-
based approach for human fall detection that use the tracking
data of the videos for detecting fall. We use videos from
a single camera but do not specify any restrictions whether
the videos are from simple RGB cameras or depth cameras.
The only requirement of our approach is a decent underlying
part-based tracking mechanism [4], [5]. For videos obtained
using Kinect RGBD cameras, skeletal tracking is an inbuilt
functionality that can be used for obtaining a part-based model
of the human body.
Fig. 1 shows an example of a fall activity from the UR
Fall Detection dataset [6]. For human activity recognition,
abstracting the human body as a set of rectangles such that
each rectangle bounds a body part, performs better than
when the human body is abstracted using a single rectangular
bounding box [7]. Such an abstraction of the human body as
a set of related rectangles is termed extended object. Kalita
et. al. [7], proposed a framework for computing qualitative
spatial relations between a pair of extended objects. The
spatial description together with the temporal information can
be encoded within a graph representation [8]. Experiments
reported therein for multi-class human-human interactions and
human-object interactions are comparable to the state-of-the-
art. In this paper, we present an approach that makes it possible
to apply Extended CORE9 within the graph representation for
single-person activities. The paper explores a purely vision
based approach that uses Extended CORE9 based spatial
description that has been reported to give good results for
human interactions. We evaluate our approach for detecting
fall in an assisted living environment. Experiments performed
on the UR Fall Detection dataset [6] show promising results.
The rest of the paper is organized as follows: a brief
overview of vision based fall detection systems and human ac-
tivity recognition systems is given in Section II. In Section III
 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)
Fig. 1. Fall activity from the UR Fall Detection dataset [6]
we discuss the Extended CORE9 framework and the proposed
approach for using it in the context of single person activities.
The experimental set-up and the classification results obtained
are discussed in Section IV; Section V concludes the paper.
II. L ITERATURE R EVIEW
Human activity recognition (HAR) deals with recognition of
activities involving humans within a video [9]. In literature,
human activities have been classified into several categories
depending on their complexity and the number of people
involved. Interactions have been defined as human activities
involving two or more humans or humans and objects; whereas
single person activities have been termed as actions [9], [10].
HAR is a well-researched area of computer vision because
of its wide range of applications. Applications in automated
surveillance systems require recognition of suspicious human
interactions or group actions; patient monitoring systems and
assistive technologies require recognition of single person
activities or human-object interactions; human computer inter-
actions require recognition of human actions or gestures [9].
A. Vision based Fall Detection
Fall detection is an important aspect of patient monitoring
systems and assistive technologies. Fall has been classified
into - fall from sleeping, fall from standing or walking, fall
from sitting and fall from standing on support [1]. The UR Fall
detection dataset on which we have performed our experiments
include fall from standing or walking and fall from sitting.
Vision based fall detection techniques have three broad
approaches - detecting inactivity period of a person on the
floor, change of shape of the persons body when falling down
and analysis of the motion pattern of the head of the human
body [11]. Detecting the period of inactivity for fall detection
requires tracking of the human body and determining whether
the person is actively changing position or orientation [12].
Detection of change of body shape is a popular approach
that focuses on modelling how the shape of the body changes
during a fall event [13]. Analysis of the motion pattern of the
head involves the idea that in a fall activity the head of the
person moves downwards to the floor very fast, in contrast to
normal activities [14]. However, inactivity of a person could
also be due to sleeping or lying down. Therefore, detection
of body shape change and motion pattern of the head are
considered to be more reliable [11]. The approach presented
in this paper for fall detection, is a combination of modelling
change of body shape and motion pattern of the head. We
consider how the relative position of the head changes with
respect to the rest of the body during a fall scenario. This is
done by analyzing the change of qualitative direction relation
of the head with the rest of the body over time.
B. KR & R for Activity Recognition in Video
Knowledge Representation and Reasoning (KR & R) deals
with how symbolic knowledge can be used for automated
reasoning instead of quantitative information. Within KR & R,
Qualitative Reasoning deals with qualitative abstractions of the
quantitative information (obtained from the physical world) to
be used as symbolic knowledge. Qualitative Spatio-Temporal
Reasoning (QSTR), which deals with qualitative abstractions
of space and time, have often been used for a more intuitive
description of video activities [15], [16].
Within QSTR several different aspects of space and time
have been discussed in literature [17]. Topology is a popular
aspect of space that deals with relations unaffected by change
of shape or size of objects. The Region Connection Calculus
(RCC) is a notable framework for topological relations [18].
Fig. 2 shows the eight basic relations of the RCC framework
(termed RCC8). Another aspect of space for which qualita-
tive abstraction has been extensively discussed is direction.
In this context, the Cardinal Direction Calculus (CDC) is
one of the most intuitive notions [19]. Fig. 3 shows how
cardinal direction relations can be specified for a pair of
objects. Qualitative distance relations has also been discussed
in [20]. Such qualitative spatial relations computed using
Extended CORE9, are used within this work for the spatial
description of activities.
III. P ROPOSED M ETHOD
A. Extended CORE9
CORE9 is a comprehensive rectangle representation that al-
lows an integrated representation of several interesting spatial
information between a pair of single-piece, axis-aligned rect-
angles, viz. topology, direction, size, distance, and motion [21].
It uses the state information of the nine cores formed using
the pair of rectangles (see Fig. 4) for obtaining the qualitative
spatial relations. Since CORE9 is incapable of providing an
 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)
Fig. 2. The base relations of RCC8 [18]
Fig. 3. (a) Cardinal directions of b (b) a S b (c) a NE:E b
(d) a B:S:SW:W:NW:E:SE b [19]
efficient representation of extended objects, an extension of
CORE9 was proposed in [7]. In this context, extended objects
are defined as a set of axis-aligned rectangles, where each
rectangle is an approximation of a component of the object. An
extended object abstraction of the human body uses separate
bounding boxes for separate body parts allowing for a more
precise description of the body than compared to abstraction
using a single MBR for the whole body. Within the Extended
CORE9 framework, the extended state information of the nine
cores using the MBRs of the set of rectangles (as shown in
Fig. 5) is used along with a recursive algorithm to opportunisti-
cally compute the qualitative spatial relations, viz., topological,
directional and distance relations, between the objects. It has
been found that such an extended object representation leads
to better classification of human activities [7].
Fig. 4. The nine cores of CORE9 for a pair of rectangles A and B [21]. The
state information of a core can be: A, B, AB,  or φ
B. Single Person Activities through Extended CORE9
The Extended CORE9 framework described in [7], com-
putes binary qualitative spatial relations between a pair of
extended objects for description and classification of human-
human interactions and human-object interactions. However,
Fig. 5. The cores of Extended CORE9 for two extended objects A and B [7].
. xy is {ai , bj |ai , bj ∈ corexy } or φ
The extended state information of core
it is possible to use such a framework for description and
classification of single person human activities.
An extended object, A, comprising of m components can
be written as,
A = {ai |ai is an axis-aligned rectangle
bounding a component of A, 1 ≤ i ≤ m}
We can define extended objects as follows:
Definition 1. The collection of extended objects over a set of
axis-aligned rectangles can be recursively defined as follows:
1) the empty set φ is an extended object and is termed a
null object
2) a singleton set consisting of one axis-aligned rectangle
is an extended object
3) if A and B are extended objects then, A ∪ B, A ∩ B and
A − B are extended objects
Within the Extended CORE9 framework, humans and ob-
jects involved in human-human and human-object interactions
are abstracted as extended objects. The Extended CORE9
framework is then used to compute binary relations between
the pair of extended objects. In a single person activity, there is
only one human body involved in the activity, and traditionally
that would be abstracted as a single extended object. However,
to use the Extended CORE9 framework for such activities, we
decompose the set of axis-aligned rectangles into two sets. It
follows from Definition 1, that these subsets of the extended
object are also extended objects.
We partition the set of rectangles A abstracting the human
body into two disjoint subsets, H and B such that,
• Union of H and is A; H ∪ B = A
• Intersection of H and B is an empty set; H ∩ B 6= φ
• H and B are non empty sets; H 6= φ; B 6= φ
From Definition 1, H and B are extended objects. The set
of rectangles comprising H corresponds to a set of reference
components of A. The set B is obtained using the set differ-
ence A − H.
The Extended CORE9 framework can now be applied on
the two new extended objects, H and B, to compute the
topological, directional and distance relations between them.
 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)
In this paper, since we are trying to classify fall activities
within the UR Fall detection dataset we choose the head of
the human body to be the reference component. H is the set
of one rectangle corresponding to the head and B is the set
of rectangles corresponding to the rest of body parts.
Fig. 6 shows the sets H and B for the fall activity depicted
in Fig. 1. The corresponding Extended CORE9 matrix is
shown in Fig. 7. The qualitative relations computed using the
Extended CORE9 framework between the head and rest of the
body parts give a structural description of the body posture
during a fall. We represent the sequence of spatial relations
between the extended objects thus computed using graphs; a
graph-kernel based SVM classification is then used to classify
fall activities.
For example, the human bodies in the activities are ab-
stracted as extended objects which is a set of components
corresponding to the head, hands and legs. As discussed above,
we partition this set into two disjoint sets - set of reference
components and set of non-reference components. We consider
the head to be the reference component. The partitioned sets
- {head} and {lefthand, righthand, leftleg, rightleg} - are
also extended objects. The recursive algorithm of Extended
CORE9 is then used to compute the topological, directional
and distance relations between the two extended objects for
each frame considered [7]. The activities are represented as
graphs and the graph-kernel based SVM classifier discussed
in [8] is used to classify fall activities from ADL.
IV. E XPERIMENTAL S ET- UP
We evaluate our proposed approach for representing single
person activities using Extended CORE9 on the UR Fall
Detection Dataset [6]. The UR Fall Detection dataset consists
of RGBD videos for 30 instances of fall activities and 40
instances of activities of daily living. The activities of daily
living videos include instances that involve a single person
walking, sitting, lying down and picking up an object from the
floor. All activities in the dataset are single-person activities
that may or may not involve another object. For each fall ac-
tivity, there are two videos captured using cameras positioned
parallel to the ground and mounted on the ceiling. In addition
to the video data captured for each activity, accelerometric
data is captured using an accelerometer that is carried by the
person performing the activity. However, for our experiments
we consider only videos from camera parallel to the ground.
It has been shown that using the additional accelerometric
data significantly improves classification accuracy [6]. But we
do not consider the accelerometric data because in a general
assisted living environment, additional equipment may not
always be available or may raise the cost.
We take the RGBD videos and manually label the humans
for every 10th frame of the videos. As discussed in Sec-
tion III-B, the reference components and non-reference com-
ponents of the extended objects are identified. It is to be noted
that there are several automated part-based tracking algorithms
that can be used obtain the annotated body parts [4], [5]. Even
if the body parts are not labeled as head, hands, or legs, the
proposed method only requires a differentiation of the parts
into reference components and non-reference components.
The qualitative spatial relations between the two sets of
components is computed using Extended CORE9 and encoded
within a graph structure [8]. For example, the activity shown
in Fig. 1 can be represented as a temporal graph as shown
in Fig. 8. Here, nodes correspond to components; labels on
solid edges represent the qualitative spatial relation between
a pair of components; dotted edges represent the temporal
evolution of the component throughout the activity. The labels
on the solid edges between a pair of components, a and b,
representing the qualitative spatial relation between them is
a three tuple, htop, dir, disi. Where top is the RCC5 relation
between a and b, dir is the CDC relation and dis is qualitative
distance relation. We then use a graph-kernel based SVM to
classify the activities represented as graphs.
A. Experimental Results
We perform a 10-fold cross-validation and have obtained a
classification accuracy of 94.28%. The confusion matrix for
the experiments is given in Table I.
ADL Fall
ADL 38 2
Fall 2 28
TABLE I
C ONFUSION MATRIX FOR ACTIVITIES IN UR FALL D ETECTION DATASET
Features Sens(%) Spec(%) Acc (%)
Kwolek et al [6] Depth + 100 96.67 98.33
Acceleration
Kwolek et al [6] Depth 100 83 90
Yun et al [22] RGB 96.77 89.77 94
Proposed method RGB 93.33 95 94.28
TABLE II
C OMPARISON OF RESULTS REPORTED FOR UR FALL D ETECTION DATASET
B. Discussion
In our experiments, we have considered only the tracking
data for RGBD videos obtained from a single camera that is
parallel to the floor. Even though tracking is done manually for
our experiments, automated tracking can be easily done given
the RGBD videos obtained using a Kinect. As mentioned ear-
lier, we do not use the accelerometric data in our experiments.
Further, we perform our experiments on all 30 Fall sequences
and 40 ADL sequences, of which 11 ADL instances involve
the person lying down on the bed or couch or floor.
We have achieved a classification accuracy of 94.28% from
our experiments. The results reported in [6], show a very high
classification accuracy of 98.33% using depth and accelero-
metric data; the classification accuracy when using only depth
data is reported to be 90.0%. However, on close examination
our results are comparable to theirs. This is because their
results are based on experiments performed on 30 fall activity
sequences and only 30 ADL sequences. As shown in Table II,
the proposed method outperforms all results reported for UR
 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)

Fig. 6. Extended object abstraction of Fall activity shown in Fig. 1

Fig. 7. Extended CORE9 matrix of Fall activity shown in Fig. 1 and Fig. 6

Fig. 8. The temporal graph representation for the Fall activity depicted in Fig. 1

Fall detection dataset using only the video data, to the best of
our knowledge.
We have noted that the two misclassified ADL activities
in our experiments correspond to activities that involve the
person lying down on the floor. However, the difference
between a person falling down and lying down on the floor
is the speed at which the person reaches the floor. Therefore,
the accelerometric data could have provided the necessary
information to give a proper classification.
V. C ONCLUSION
For human activity recognition, abstraction human bodies
as extended objects has been shown to give better results
compared to abstracting human bodies only using a single
bounding box [7]. Encoding the spatial information within
a graph representation provides classification results that are
comparable to the state-of-the-art for human-human or human-
object interactions [8]. In this paper we have given an ap-
proach by which the Extended CORE9 framework can also
be applied for single person activities. We have shown via
experiments performed on the UR Fall Detection dataset that
such an approach can be effectively applied for detecting
fall scenarios during activities of daily living in an assisted
living environment. We have noted that our approach is not
always able to distinguish between a lying down activity
and fall activity because the speed of the activity has not
been taken into account. Research towards incorporating speed
information within the qualitative framework is part of future
 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP)

work. Another interesting and important direction of future [22] Y. Yun and I. Y. Gu, “Human fall detection via shape analysis on
work could be towards online recognition of fall activities. riemannian manifolds with applications to elderly care,” in 2015 IEEE
International Conference on Image Processing (ICIP), 2015, pp. 3280–
3284.
R EFERENCES
[1] M. Mubashir, L. Shao, and L. Seed, “A survey on fall detection:
Principles and approaches,” Neurocomputing, vol. 100, pp. 144 – 152,
2013.
[2] Z. Zhang, C. Conly, and V. Athitsos, “A survey on vision-based fall
detection,” in Proceedings of the 8th ACM International Conference on
PErvasive Technologies Related to Assistive Environments. ACM, 2015,
pp. 46:1–46:7.
[3] C. Lin, S. Wang, J. Hong, L. Kang, and C. Huang, “Vision-based fall
detection through shape features,” in 2016 IEEE Second International
Conference on Multimedia Big Data (BigMM), 2016, pp. 237–240.
[4] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan,
“Object Detection with Discriminatively Trained Part-Based Models,”
IEEE Transactions on Pattern Analysis and Machine Intelligence
(PAMI), vol. 32, no. 9, pp. 1627–1645, 2010.
[5] Y. Yang and D. Ramanan, “Articulated Pose Estimation with Flexible
Mixtures-of-parts,” in IEEE Conference on Computer Vision and Pattern
Recognition, ser. CVPR. IEEE Computer Society, 2011, pp. 1385–1392.
[6] B. Kwolek and M. Kepski, “Human fall detection on embedded platform
using depth maps and wireless accelerometer,” Computer Methods and
Programs in Biomedicine, vol. 117, pp. 489 – 501, 2014.
[7] S. Kalita, A. Karmakar, and S. M. Hazarika, “Efficient extraction of
spatial relations for extended objects vis-á-vis human activity recognition
in video,” Applied Intelligence, vol. 48, no. 1, pp. 204–219, 2018.
[8] ——, “A temporal activity graph kernel for human activity classifica-
tion,” in 11th Indian Conference on Computer Vision, Graphics, and
Image Processing, 2018.
[9] J. Aggarwal and M. Ryoo, “Human Activity Analysis: A Review,” ACM
Computing Surveys, vol. 43, no. 3, pp. 16:1–16:43, 2011.
[10] M. Vrigkas, C. Nikou, and I. A. Kakadiaris, “A review of human activity
recognition methods,” Frontiers in Robotics and AI, vol. 2, p. 28, 2015.
[11] M. M. A. J. Yoosuf Nizam, Mohd Norzali Haji Mohd, “A study on
human fall detection systems: Daily activity classification and sensing
techniques,” International Journal of Integrated Engineering, vol. 8,
no. 1, pp. 35 – 43, 2016.
[12] B. Jansen and R. Deklerck, “Context aware inactivity recognition for
visual fall detection,” in Pervasive Health Conference and Workshops,
2006.
[13] N. Thome and S. Miguet, “A hhmm-based approach for robust fall de-
tection,” in 2006 9th International Conference on Control, Automation,
Robotics and Vision, 2006, pp. 1–8.
[14] C. Rougier, J. Meunier, A. St-Arnaud, and J. Rousseau, “Fall detection
from human shape and motion history using video surveillance,” in Pro-
ceedings of the 21st International Conference on Advanced Information
Networking and Applications Workshops - Volume 02, ser. AINAW ’07.
IEEE Computer Society, 2007, pp. 875–880.
[15] K. S. R. Dubba, M. Bhatt, F. Dylla, D. C. Hogg, and A. G. Cohn, “In-
terleaved Inductive-Abductive Reasoning for Learning Complex Event
Models,” in International Conference on Inductive Logic Programming
(ILP), ser. LNCS, vol. 7207. Springer, 2012, pp. 113–129.
[16] M. Sridhar, A. G. Cohn, and D. C. Hogg, “Benchmarking Qualitative
Spatial Calculi for Video Activity Analysis,” in IJCAI Workshop Bench-
marks and Applications of Spatial Reasoning, 2011, pp. 15–20.
[17] A. G. Cohn and S. M. Hazarika, “Qualitative spatial representation and
reasoning: An overview,” Fundamenta Informaticae, vol. 46, no. 1-2,
pp. 1–29, 2001.
[18] D. A. Randell, Z. Cui, and A. Cohn, “A Spatial Logic Based on Regions
and Connection,” in 3rd International Conference on Principles of
Knowledge Representation and Reasoning (KR’92), B. Nebel, C. Rich,
and W. Swartout, Eds. Morgan Kaufmann, 1992, pp. 165–176.
[19] S. Skiadopoulos and M. Koubarakis, “On the consistency of cardinal
directions constraints,” Artificial Intelligence, vol. 163, pp. 91 – 135,
2005.
[20] T. Bittner and M. Donnelly, “A formal theory of qualitative size
and distance relations between regions,” in 21st Annual Workshop on
Qualitative Reasoning (QR07), 2007.
[21] A. G. Cohn, J. Renz, and M. Sridhar, “Thinking Inside the Box: A
Comprehensive Spatial Representation for Video Analysis,” in the 13th
International Conference on Principles of Knowledge Representation
and Reasoning (KR2012). AAAI Press, 2012, pp. 588 – 592.