Received: 26 November 2020 Revised: 16 March 2021 Accepted: 1 July 2021

DOI: 10.4218/etrij.2020-0446

ORIGINAL ARTICLE

Real-world multimodal lifelog dataset for human behavior

study

Seungeun Chung Chi Yoon Jeong Jeong Mook Lim Jiyoun Lim
Kyoung Ju Noh Gague Kim Hyuntae Jeong

Artificial Intelligence Laboratory,

Electronics and Telecommunications
Research Institute, Daejeon, Republic of
Korea
Correspondence
Chi Yoon Jeong, Artificial Intelligence
Laboratory, Electronics and
Telecommunications Research Institute,
Daejeon, Republic of Korea.
Email: iamready@etri.re.kr
Funding information
Electronics and Telecommunications
Research Institute, Grant/Award Number:
21ZS1100
Abstract
To understand the multilateral characteristics of human behavior and physiolog-
ical markers related to physical, emotional, and environmental states, extensive
lifelog data collection in a real-world environment is essential. Here, we pro-
pose a data collection method using multimodal mobile sensing and present a
long-term dataset from 22 subjects and 616 days of experimental sessions. The
dataset contains over 10 000 hours of data, including physiological, data such
as photoplethysmography, electrodermal activity, and skin temperature in addi-
tion to the multivariate behavioral data. Furthermore, it consists of 10 372 user
labels with emotional states and 590 days of sleep quality data. To demonstrate
feasibility, human activity recognition was applied on the sensor data using a
convolutional neural network-based deep learning model with 92.78% recog-
nition accuracy. From the activity recognition result, we extracted the daily
behavior pattern and discovered five representative models by applying spectral
clustering. This demonstrates that the dataset contributed toward understand-
ing human behavior using multimodal data accumulated throughout daily lives
under natural conditions.

K E Y WO R D S
data collection, human behavior pattern, lifelog, real-world dataset

1 I N T RO DU CT ION standing the multilateral characteristics of human behav-

With recent advancements in mobile devices and wear-
ables, including smartwatches, health trackers, and fitness
bands, collecting continuous sensor data at high resolu-
tion has become possible through off-the-shelf devices.
Since such devices sense various physiological and behav-
ioral information continuously over the long term, many
researchers have focused on improving human health and
well-being by analyzing lifelog data. Extensive real-world
lifelog datasets are an essential prerequisite for under-
ior and physiological markers correlating with the individ-
uals' physical, emotional, and environmental states. How-
ever, only a few lifelog datasets collected in the wild reflect
the multifarious aspects of human life.
Many publicly available activity datasets [1–5] contain
behavioral data measured using an inertial measurement
unit (IMU) and target human activity recognition (HAR)
through sensor data analysis [6,7]. Although vision-based
HAR [8] is another interesting research topic, here, we
only focus on on-body sensor data. Designing these activity

This is an open access article under the term of Korea Open Government License (KOGL) Type 4: Source Indication + Commercial Use Prohibition + Change Prohibition
(http://www.kogl.or.kr/info/licenseTypeEn.do).
1225–6463/$ © 2021 ETRI

426 wileyonlinelibrary.com/journal/etrij ETRI Journal. 2022;44(3):426–437.


CHUNG ET AL.
datasets follows from identifying activities of daily liv-
ing (ADL), recorded by following a dedicated protocol
in a controlled environment with a restricted number of
subjects. Because the dataset size is limited, data aug-
mentation is applied to expand the usable dataset [9].
Datasets collected under laboratory conditions are insuf-
ficient for representing natural behavioral traits during
everyday life.
Recently, research has shifted focus to lifelogging [10,11]
personal information, such as activities, locations, and
environments, and lifelog information retrieval [12,13].
Several lifelog datasets [14,15] are predominantly multi-
media data and semantic contexts, whereas available raw
sensor data are quantitatively small. Additionally, subject
numbers are significantly limited because of the privacy
concerns incurred by visual data collection. The ExtraSen-
sory [16–18] dataset is the most exhaustive study con-
ducted in a free-living condition and focused mainly on
accumulating behavioral sensor data.
A review [19] on mobile sensing for the behavior
study categorized assessable behavioral features into three:
the physical movements (physical activity and mobil-
ity patterns), social interactions (face-to-face encounters
and computer-mediated communication), and daily activ-
ities (mediated and non-mediated activities). Many public
datasets contain some aspects of the aforementioned fea-
tures, however, deficiencies exist in measuring multilateral
perspectives of daily behavioral patterns, emotional state,
and sleep quality.
We focus on the emotional state as one essential deter-
minant associated with behavioral phases of everyday life
[20,21], and quality of life [22,23]. Since physiological
responses are widely accepted vital markers corresponding
to emotional states [24,25], we actively accumulate multi-
ple physiological signals including photoplethysmography
(PPG), electrodermal activity (EDA), and skin temperature
naturally with self-reported emotional states. Although
there exist some physiological emotion datasets [26–28], of
emotions provoked by artificial stimuli, such as video clips
and music recordings, specifically targeted for the emotion
recognition domain.
Sequel to clinically investigated studies on the relation-
ship between the quality of life and sleep quality [29,30],
which correlates notably with non-sleep daily phenom-
ena [31], such as mood [32] and physical activities [33],
sleep quality is also considered in our lifelog dataset.
Although it is subjective, several efforts have tried to evalu-
ate sleep quality using objective measures [34,35] together
with subjective questionnaires [36,37]. We expand the
lifelog dataset by collecting comprehensive data, includ-
ing objective and subjective sleep qualities measured using
22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
427
a sleep-tracking sensor and ecological momentary assess-
ments [38] on sleep quality, respectively.
The contribution of this paper is as follows:
• We propose a lifelog data collection method that exten-
sively observes multilateral characteristics of behav-
ioral, emotional, and environmental states of human life
using multimodal mobile sensing.
• We present a long-term dataset including 28 consec-
utive days of experimental sessions obtained from 22
subjects each, to obtain 616 days. The dataset contains
more than 2.26 TB of data including 10 372 user labels
with emotional states, over 10 000 h of various sensor
data including physiological data, and 590 days of sleep
quality data.
• Finally, we demonstrate the feasibility of the dataset
through applications including HAR and human behav-
ior pattern extraction for practical use.
2 RELATED WORK
This section reviews previous works on activity and lifelog
datasets. Table 1 presents a summary of the datasets
reviewed in this section.
2.1 Activity datasets
The opportunity [5] dataset collected multiple on-body
IMUs and environmental sensor data from four subjects in
a living lab, with labels representing modes of locomotion,
actions, and objects. The PAMAP2 [4] dataset used three
IMUs from the wrist, chest, and ankle, added to a heart
rate monitor worn on nine subjects, containing 18 labels
on diverse activities, such as walking and playing soccer.
The IMU data contained in the UCI-HAR [1] dataset fol-
low from a smartphone mounted on the left side of the
waist, where 30 subjects performed six activity protocols.
Since these datasets were collected in a controlled environ-
ment from experimental protocols of simple and low-level
activities, they are inadequate for analyzing multilateral
characteristics of human behavior that are complex.
The mHealth [2] dataset gathered 12 common ADLs in
an out-of-lab environment from ten subjects, where the
smartphone app self-recorded the activity protocol exe-
cution. Afterward, the video-recorded data were used as
the ground-truth activity label. The experiment employed
three on-body IMUs on the chest, right wrist and left ankle,
and a 2-lead ECG sensor on the chest. Although the data
collection protocol was unconstrained, the dataset covers
only two h of activity data.
The ExtraSensory [16] dataset obtained labeled sen-
sor data from smartphones and smartwatches uncon-
strained for approximately one week (7.6 days on aver-
age) from 60 subjects, where the participation dura-
 22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
428 CHUNG ET AL.

TABLE 1 Summary of activity and lifelog datasets

Dataset Subjects Duration Labeling On-body Sensor Behavioral Data Physiological Data
Opportunity [5] 4 25.0 h Protocol 7 IMUs A, G, M -
PAMA2P [4] 9 10.0 h Protocol 3 IMUs, HR A, G, M HR
UCI-HAR [1] 30 1.5 h Protocol P A, G -
mHealth [2] 10 2.0 h Video 3 IMUs, ECG A, G, M ECG
NTCIR-14 [15] 2 30 days Metadataa P, W, camera Image, semantic contextb HR, blood glucose
ImageCLEF [14] 2 30 days Metadataa P, camera Image, semantic contextb -
ExtraSensory [16] 60 7 days Self-report P, W A, G, M, GPS, audioc , phone state -
ETRI (Ours) 22 28 days Self-report P, W A, G, M, GPS, audioc , phone state PPG, EDA, temperature

Smartphone (P), Smartwatch (W), Accelerometer (A), Gyroscope (G), Magnetometer (M), Heart Rate (HR)
a Attribute and category of the place and objects detected using image processing.
b Semantic locations, physical activities or transportation media captured by an application.
c Periodically recorded background noise.

tion varied from 2.9 to 28.1 days. The participants
self-reported their activity and context by selecting rele-
vant labels from the app that offers more than 100 labels.
Although data were collected in a real-world scenario,
neither the number of participants nor experiment dura-
tion was sufficient to completely reflect the long-term
aspects of human behavior under natural conditions.
Existing datasets focused on the activity or context and
lacked sensor data that indicated the psychological or
physiological states.
2.2 Lifelog datasets
The NII testbeds and community for information access
research (NTCIR) lifelog dataset [15] is provided by the
information retrieval research community, which treats
lifelogging as an application of information retrieval.
These datasets contain 30 days of data logged by two
subjects with approximately 26 GB of image, location,
and biometric information. Similarly, the cross-language
evaluation forum (CLEF) group supports the Image-
CLEF lifelog dataset [14], to encourage cross-language
annotation and image retrievals. The ImageCLEF lifelog
dataset contains 1.5 months of data from two partici-
pants, including visual data, biometric information, and
semantic contents.
Since participants are highly limited (two or three sub-
jects) in existing lifelogging datasets [15,39,40], it lacks
enough data to insufficiently derive a general lifelog
model. Additionally, existing datasets concentrate on
the surroundings [14,41,42] rather than the participants'
behavioral or emotional characteristics, thus, falling short
of understanding compound aspects of one's actual life.
3 ETRI LIFELOG DATA SET
This section first introduces our system design consider-
ation for collecting lifelog data under natural conditions.
Then, the system architecture and data collection protocol
are explained, to gather an extensive dataset represent-
ing the participants’ behavioral, emotional, and environ-
mental states using multimodal mobile sensing. Finally,
we discuss the details and statistics of the dataset. The
dataset is publicly available at https://nanum.etri.re.kr/
share/schung/ETRILifelogDataset2020?lang=En_us.
3.1 System design
The goal of our lifelogging system is to accumulate
large-scale lifelog data for the long term that offer
data-driven descriptions of human life from various per-
spectives. Therefore, we are primarily concerned with a
user-friendly interface for our system, thereby enhancing
experimental participation eventually and guaranteeing
the reliability of the data. Three requisite aspects of lifelog
data, the behavioral, environmental and social, and physi-
ological traits are defined, and the most suitable hardware
device combinations that meet our requirements are dis-
cussed. Furthermore, we organize the system with a min-
imum number of devices to reduce inconveniences that
occur during experimental sessions. The system is com-
posed of a smartphone, a wrist-worn health tracker, and a
sleep-quality monitoring sensor.
Repeatedly labeling activity variations in everyday life
is challenging and almost impossible to log each activ-
ity without exception. Since labeling rates are most cru-
cial for determining the accessibility and feasibility of
the experiment, frequently changing low-level activities
(for example, sitting, walking, standing, lying, and run-
ning) were omitted. We designed labels only for high-level
behaviors, including semantic contexts. Since long-term
experiments can deteriorate participation over time, main-
taining the fidelity of the experiment for a prolonged
period was considered. A user statistics page was embed-
ded in the lifelogging application that delivers the partici-
pant's daily and weekly statistics and compares it with that
 22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHUNG ET AL. 429

of other participants. First, the statistics are computed in a
central server that integrates lifelog data from participants,
and then, shared on the client device.
3.2 Lifelogging system architecture
Our proposed lifelogging system consists of a mobile appli-
cation on an android smartphone, a wrist-worn health
tracker, and a sleep-quality monitoring sensor, as shown
in Figure 1. The lifelogging server collects data from
our smartphone application and cloud servers, providing
FIGURE 1 Lifelogging system architecture
TABLE 2 Categories for activity, social state, and place labels
Label Category Sub-category (if exists)
Activity (16) Sleep, personal care, work, study,
housework, caregiving, media,
entertainment, sports, hobby, free
time, shopping, regular activity
Transport (Mode of transportation) On foot,
by bus, by car as a driver, by car
as a passenger, by subway or train,
by personal mobility
Meal (Amount of intake) Large, moder-
ate, small
Social (Conversation) Active, moderate,
passive
Socialstate (3) Alone -
Two With a family member, a friend,
a colleague, an acquaintance, a
stranger
Group With family members, friends,
colleagues, acquaintances,
strangers
Place (5) Home workplace, restaurant, other
indoor, outdoor
access to the collected data. To efficiently manage large
amounts of lifelog data, the server uses non-relational
MongoDB DBMS, widely used in big data and real-time
web applications. This server web application is imple-
mented using the spring framework. The smartphone
application mainly comprises the labeling user interface
(UI), data collection, user statistics, and data transfer
modules.
Labeling UI: User labels comprehend a broad range of
everyday activities and semantic contexts from the
time use survey published by Statistics Korea. Table 2
summarizes categories collected for activity, social
state, and place labels. To reduce the number of inter-
actions required for each label input, we designed the
labeling UI to integrate all elements in a single page,
as shown in Figure 2A. The labeling UI displays the
16 high-level activity options hierarchically. The user
first selects a category and then chooses the final label
from the sub-category options. The context labels are
designed to indicate a companion, if any, the intensity
of social interaction, and the semantic place informa-
tion. Emotion labels follow Russell's circumplex model
of affect [43], which presents emotion along two axes:
arousal and valence axes. Our application offers two
7-point Likert scales corresponding to these axes.
An interface for entering repeated routines or pre-
arranged schedules in advance was incorporated and
requests confirmation for each event to mitigate iter-
ated label input. A summary of the selected labels,
composed of a user's daily routine is visualized in
chronological order on the timeline screen, as shown
in Figure 2B. User labels appear as icons to represent
the context information, while emotions are mapped
as colors to display mood changes in a simple definite
way. Our application allows the user to interactively
insert new labels or modify existing ones by reviewing
the timeline to improve user convenience.
Data collection: The 3-axis inertial sensors were sampled
at 50 Hz, while the GPS coordinates were recorded
every five seconds. Audio input from a microphone
was recorded at 22 050 Hz every 30 min, cap-
turing background noise and ambient environment.
Recorded audio data were processed on the smart-
phone to extract audio features composed of 13 Mel
frequency cepstral coefficients (MFCCs) for every 40
ms frame with 30 ms overlap. Phone state information
includes activity types and application usage statis-
tics. The activity type was retrieved from the Google
awareness application programming interface (API)
every minute, indicating the device state, such as in a
vehicle, on a bicycle, on foot, still, or unknown. The
application usage statistics captured the duration of an
application running in the foreground every 30 min.

430
FIGURE 2 Lifelogging application screens. (A) Labeling UI, (B) Timeline, (C) User Statistics, (D) Data upload
Finally, the weather information, including tempera-
ture, humidity, and air pollutant concentration (that
is, PM 10 and PM 2.5), from the GPS location was
acquired using the AirVisual API.
User statistics: Our application visualizes various aspects
of statistical data to offer useful information related to
one's daily routine. Added to the number of daily labels
collected by the user, the distribution of aggregated
activity and emotion labels are presented by week to
deliver the user's lifelog trend in the longer term, as
shown in Figure 2C. The average of all participants is
also shown to compare statistics with others, which
inform data collection in the experiment.
Data transfer: User labels and sensor data are accumulated
in the local storage during the experimental session
and transferred to our data collection server via the
WiFi network whenever the user terminates the exper-
iment at day end. Figure 2D illustrates the data transfer
UI, which also shows a summary of daily labels before
uploading the data to the server.
We adopted an Empatica E4 [44] wristband to obtain
medical-grade physiological signals and behavioral data.
The 3-axis accelerometer data were sampled at 32 Hz,
while blood volume pressure was measured from a PPG
sensor at 64 Hz. Skin conductance values from an EDA
sensor and peripheral skin temperature from an infrared
thermopile were both recorded at a sampling rate of 4 Hz.
To retrieve the sensor data stored in the local memory of
the E4 device, the user connects the E4 device to a PC via a
USB interface. Then, Empatica's preinstalled E4 manager
software automatically uploads data to Empatica's cloud
22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHUNG ET AL.
server. Finally, the experiment operator manually fetches
data from the cloud to our lifelogging server.
To investigate the possible relationship between an
individual's daily routine and sleep quality, we collected
data during sleep at night. We adopted the Withings
sleep-tracking mat [45] that is placed under the mattress
during sleep once installed. Using a network connection
via the Withings health mate application installed on the
smartphone, sleep data were automatically uploaded to
the Withings cloud server where our lifelogging server
periodically fetches data through the Withings data API.
Sleep data provide various sleep quality measures, such as
the duration of wakefulness, light sleep, deep sleep, and
REM sleep during sleep. It also provides general informa-
tion, including the start and end times of sleep and the time
taken to fall asleep and awaken.
3.3 Data collection protocol
The duration of the experiment was from August 30, 2020,
to October 8, 2020, with 13 male and 9 female partici-
pants from a metropolitan city in South Korea. Their ages
ranged from 20 to 35 years and most were office workers
who commute periodically during weekdays. Reportedly,
all participants work more than three days a week. The
demographic information of the participants is summa-
rized in Table 3. The experiment was performed using
institutional review board approval (P01-202009-22-002).
We deployed our lifelogging application in person dur-
ing the orientation of our study. The participants were
rewarded daily, accumulating weekly incentives for com-
pleting each week's experiment without missing any day.
 22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHUNG ET AL. 431

TABLE 3 Demographic information of participants that they maintained their daily routine and that the
Participants Age Gender Height BMI Dominant number of devices managed during the experiment was
(yr) (cm) (kg/m2 ) hand reasonable, whereas 31.8% opined the labeling method
Participant 1 27 M 166 27.6 L caused some nuisance in their everyday life. E4 wrist-
Participant 2 25 F 168 29.1 L band recorded 63.6% respondents as the most disturb-
Participant 3 32 F 166 22.5 L ing device, because of the limited user interface of the
Participant 4 28 M 171 23.9 R research-purpose device. However, 90.9% of the partici-
Participant 5 31 M 177 23.9 R pants responded that statistical labels were informative
Participant 6 22 F 164 26.0 R
and useful in organizing their daily experiences. Addition-
Participant 7 30 M 173 28.1 R
Participant 8 32 M 180 26.2 R
ally, 95.5% of the participants wanted to join additional
Participant 9 20 M 170 27.7 R experiments and 59.1% answered affirmed willingness to
Participant 10 25 M 184 23.1 R continue the experiment over four weeks with the same
Participant 11 32 M 175 25.5 R protocol. This implies that our experimental design was
Participant 12 30 F 160 21.1 R acceptable for lifelogging in a real-world scenario. Overall,
Participant 13 35 M 177 27.1 R the survey results reflect the necessity of automatic lifel-
Participant 14 26 F 165 24.8 R ogging and self-quantification technologies for organizing
Participant 15 27 M 182 21.7 R
daily experiences.
Participant 16 27 F 160 18.8 R
Participant 17 28 F 170 20.8 R 3.4 Dataset statistics
Participant 18 31 M 169 27.7 R
Participant 19 29 M 169 22.1 R We accumulated over 2.26 TB of data with 10 372 user
Participant 20 26 M 174 19.2 R labels, representing the semantic activity, place, social sta-
Participant 21 27 F 166 20.7 R tus, and emotion at the time of labeling. Data representing
Participant 22 30 F 163 25.6 R behavioral traits consist of IMU data, GPS coordinates,
activity types, and phone states indicating the app usage
statistics, while environmental data contains audio fea-
tures reflecting ambient noise and weather information.
From Table 4, our dataset contains twice more data than
the ExtraSensory dataset. Also, our dataset contains 8,141
h of physiological data and 590 days of sleep-quality mea-
surements during night sleep. Formally, we assert owning
the largest dataset containing physiological data in addi-
tion to the activity and lifelog data collected in a real-world
FIGURE 3 Post-experiment survey results
scenario.
Figure 4 summarizes the occurrences and distributions
Thus, most participants completed the 28 consecutive days of activity, place, social state, and emotion labels. Among
of the experiment period. 16 labels listed as the activity option, eight activities includ-
The participants started the experiment in the morning ing transport (20.1%), work (17.9%), meal (14.0%), media
with their everyday routine and continued collecting data (11.8%), personal care (10.9%), free time (5.9%), housework
for at least 12-h daily. The sleep-quality questionnaires (5.1%), and social (3.5%) comprised a large proportion
were first self-reported in the morning. The participants (89.2%) of activity labels. Responding to the coronavirus
were instructed to insert a label for contextual changes, (COVID-19) pandemic, many participants unexpectedly
including physical activity, semantic place, social status, worked from home during the experiments. Consequently,
and emotion. At the end of the 12-h experiment, partici- 43.8% of the place labels were home, while only 18.8%
pants sent the data to the server. An experiment supervisor was denoted as the workplace. The social state distribution
managed the data quality daily and encouraged the par- showed that the vast majority of labels (58.0%) were tagged
ticipants to actively engage in the experiment. Finally, when participants were alone with little social interaction.
participants answered the second sleep-related survey that Figure 4D show the occurrence of emotion labels in
recorded the stress perception level (degree of physical and 2D according to Russell's circumplex model of affect.
emotional stress) and the amount of caffeine and alcoholic The 7-point valence and arousal scales range from cate-
beverage intake during the day before sleep. gory 1 (negative) to category 7 (positive) and category 1
After the experiment, we surveyed the experimental (relaxed) to category 7 (activated) on the x- and y-axes,
protocol and devices used. The results are summarized respectively. Category 4 denotes a neutral state for both
in Figure 3. Among the 22 participants, 81.8% affirmed arousal and valence scales and occurs most at (9.9%)
 22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
432 CHUNG ET AL.

TABLE 4 Overview of the dataset and details of the data

ExtraSensory [16] ETRI Dataset (Ours)
Sensor Data details (Sampling frequency) Hours Data details (Sampling frequency) Hours
Accelerometer 3-axis (40 Hz) 5138 3-axis (50 Hz) 9824
Gyroscope 3-axis (40 Hz) 4864 3-axis (50 Hz) 10 735
Magnetometer 3-axis (40 Hz) 4708 3-axis (50 Hz) 10 660
Watch Accelerometer 3-axis (25 Hz) 3512 3-axis (32 Hz) 8141
GPS Location Longitude, latitude, altitude (variable) 3512 Longitude, latitude, accuracy (5 s) 11 638
Audio 42-ms frame 13 MFCC (per min) 5036 40-ms frame 13 MFCC (30 min) 12 149
Phone State Physical phone state (per min) 5138 Foreground app usage time (30 min) 12 223
Activity type - - Transportation mode, activity (per min) 10 563
Weather - - Temperature, humidity, PM 10/2.5 (per h) 12 485
PPG - - Blood volume pressure (64 Hz) 8141
EDA - - Skin conductance (4 Hz) 8141
Infrared Thermopile - - Skin temperature (4 Hz) 8141
Sleep Sensor - - Time in bed, total sleep time, score (per day) 590 days

FIGURE 4 Histograms for user label occurrence and their distributions. (A) Activity labels, (B) Place labels, (C) Social labels, (D) Emotion
labels (Color bar presents the number of occurrences while the number stands for their distribution)

FIGURE 5 Deep learning network architecture for HAR
among 49 combinations of the 7-point A-V scale. The
low-arousal positive-valence state (quadrant IV) composed
the largest proportion (37.3%) of emotion labels, then the
high-arousal positive-valence state (quadrant I, 19.4%),
low-arousal negative-valence state (quadrant III, 6.0%),
and high-arousal negative-valence state (quadrant II, 4.5%)
in that order.
4 APPLICAT IONS OF THE
DATAS ET
To demonstrate the applicability of the dataset for practi-
cal use, we used the dataset for two applications: HAR and
human behavior pattern extraction.
4.1 HAR
We focus on HAR, which predicts low-level activities
(walking or sitting) by analyzing the sensor data from
on-body sensors. Since our experiment accumulated data
of real-world scenarios over the long term, the user label
can correspond more closely with the high-level activity
or semantic context (commuting to work) and inconsis-
tently with phase changes in low-level activities with high
precision. For example, walking and sitting can occur
alternately during commuting, but it is almost impossi-
ble to label the low-level activity promptly while in action.
Therefore, activity type data from the Google Awareness
API are the ground truth of low-level activity with four
classes: on foot, still, in vehicle, and unknown.
The numbers of activity data were highly imbalanced
following the human activity. Among 10 563 h of activity
type data, only 3% of data corresponded to “in vehicle” and
“on foot,” respectively, while 85% of data represented the
“still” state. To account for the label imbalance, the per-
formance and accuracy are presented using the F1-score
metric.
 22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHUNG ET AL. 433

FIGURE 6 Cumulative time of experiment days of each user (solid: weekdays, dotted: weekends)

FIGURE 7 (A) Silhouette coefficients according to the number of clusters. (B) Cumulative time distribution of the five clusters

FIGURE 8 Axes of the radar chart represent the experiment day count corresponding to each cluster

We performed activity recognition using accelerometer the sensor data using a 2.5 s-long window length with 50%
data collected from smartphones and wristbands, respec- overlap. The 3-axis sensor data were transformed into a 15
tively. Five seconds of sensor data were processed before × 15 matrix and fed into the convolutional neural network
the timestamp of the activity type data and we segmented architecture. A network with four convolutional layers and

434
two fully connected layers was used for model training, as
shown in Figure 5.
After over 5-fold cross-validation iterations, the average
recognition accuracy using the smartphone accelerometer
data was 92.78% with an F1-score of 0.92, while that using
the wristband accelerometer data was 82.9% accurate with
an F1-score of 0.78. Because Google’s activity type is deter-
mined by the sensor data embedded in the smartphone,
its accelerometer data showed better performance than the
wristband data, as expected. Since the accuracy of smart-
phone data is reliable, the discrepancy can be resolved
by learning the feature difference in wristband data from
the smartphone perspective and applying the model to the
wristband data using transfer learning, as future work.
4.2 Human behavior pattern extraction
From the activity recognition results, we further investi-
gated the daily behavioral patterns of each user, extracted
the activity recognition results from the sensor data stream
and accumulated the occurrence of activities every 10 min,
representing the distribution of activities in unit time. To
describe daily behavior patterns, we consider the active
portion by manipulating the cumulative time, recognized
as “walking”, as a feature vector. The feature vector con-
sists of 144 slots, representing 24 h in a 10-min unit,
starting from 5:00 AM.
Figure 6 shows the cumulative time of walking repre-
sented in the log scale for each user. Each line represents a
single experiment day, weekdays are solid lines, weekends
are denoted by dotted lines. Many users have a consistent
behavioral pattern, as most experimental results overlap
with other days in each figure. As expected, the behav-
ioral patterns during the weekends are different from those
of weekdays. As a result, we cluster the feature vector
to discover the difference in behavior patterns between
weekdays and weekends, and among users.
We use spectral clustering to group feature vectors. To
determine the number of clusters, the silhouette coeffi-
cient for measuring the separation distance between the
resulting clusters is computed. The coefficient is nearly
one when samples are far from the neighboring clusters,
while a negative value indicates that samples are assigned
to the wrong cluster. From Figure 7A, we plot the average
silhouette coefficient (in red) and the average silhouette
coefficients that are below zero (in blue) according to the
number of clusters. From the result, k = 5 is a reasonable
value for the number of clusters.
Figure 7B presents the clustering result indicating the
cumulative time of five representative clusters with diverse
behavioral patterns. The first cluster (solid line) represents
the experimental days that start early mornings and vig-
orously continues all day until the end of the experiment.
The second cluster (blue line) indicates the majority of
22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHUNG ET AL.
each experimental days, starting from about 8:00 AM and
remain active during working hours. The third cluster has
a similar trend at the beginning of the day but shows mod-
erate activity during working hours. The fourth cluster
starts the experiment late in the morning, while the last
cluster presents a behavior pattern that is inactive during
the whole day.
Finally, we visualized the experimental day count of
each cluster in radar charts for each user, as shown in
Figure 8, where the axes indicate five representative clus-
ters. The results show a sharp direction on a specific
cluster, interpreted as the primary behavior pattern of indi-
viduals. For example, user 02 tends to show moderate to
inactive behavior pattern during the experiment, while
users 26, 27, and 30 show distinct behavior patterns that
belong to a specific cluster such as idle, inactive, vigor-
ous pattern, respectively. In this way, users’ daily behavior
patterns can be recognized and classified by analyzing the
sensor data stream.
5 CO NCLUSIONS AND FUTURE
WO RK
In this paper, we proposed a long-term lifelog data collec-
tion method that minimizes data collection disturbance
during everyday life. We presented a lifelog dataset cov-
ering 616 days of experimental sessions obtained from 22
subjects. The dataset contains more than 2.26 TB of data
including 10 372 user labels with emotional state, over
10 000 h of various sensor data, and 590 days of sleep qual-
ity data. So far, our dataset is the largest that comprehends
multiple aspects of human life using behavioral, physiolog-
ical, emotional, and environmental lifelog data collected
in a real-world scenario. Among the 22 participants, 81.8%
maintained their daily routine and were comfortable with
the number of devices handled during the experiment.
Also, 90.9% of participants responded that the statistics
of the labels were informative and useful in organizing
their daily experiences. The survey result signifies that
our experimental design was satisfactory for lifelogging in
natural conditions.
To illustrate the practicability of our dataset, HAR was
applied to the dataset, especially the sensor data from
smartphones and wristbands. Our convolutional neural
network-based deep learning model recognized the activi-
ties with 92.78% accuracy using smartphone accelerometer
data while achieving 82.9% accuracy from using wristband
data. From the activity recognition result, we extracted the
daily behavior pattern as a feature vector and discovered
five representative patterns by applying spectral cluster-
ing. We confirmed that users' daily behavior patterns are
categorized by analyzing the sensor data stream.
Since our dataset contains several sensor data, we
propose that the dataset can be used to understand

CHUNG ET AL.
multilateral characteristics of behavioral, emotional, and
environmental states of human life. As a behavioral
trait, the application usage time can be further processed
to extract information about online activity and social
interaction from the application category and usage pat-
terns. Additionally, the transportation mode classifica-
tion and extracting points of interest from GPS data can
assist in understanding the mobility pattern and model
spatio-temporal routine [46]. Environmental traits such as
background audio recording can be used to understand
the ambiance of the venue, while weather information
can contribute to finding relevant cues affecting behavioral
or emotional tendencies. From physiological data, we dis-
cover some significant markers that correspond to specific
behavioral or emotional events. The sleep data provides
unusual daily traits or anomalies [47,48] that substantially
influence sleep quality. Furthermore, our dataset can be
exploited in developing personalized classifiers that find
the unique character of each participant and differenti-
ate one person from others. We leave the aforementioned
research topics as future work.
ACKNOWLEDGEMENT
This work was supported by Electronics and Telecom-
munications Research Institute (ETRI) grant funded by
the Korean government. [21ZS1100, Core Technology
Research for Self-Improving Integrated Artificial Intelli-
gence System].
CONFLICT OF INTEREST
The authors declare that there are no conflicts of interest.
ORCID
Seungeun Chung https://orcid.org/
0000-0001-9815-3985
Chi Yoon Jeong https://orcid.org/0000-0001-7089-2516
Jeong Mook Lim https://orcid.org/
0000-0002-5535-0417
Jiyoun Lim https://orcid.org/0000-0003-4246-3081
Kyoung Ju Noh https://orcid.org/0000-0001-8492-8612
Hyuntae Jeong https://orcid.org/0000-0003-4339-1673
REFERENCES
1. D. Anguita et al., A public domain dataset for human activity
recognition using smartphones, Esann 3 (2013), 3.
2. O. Banos et al., Design, implementation and validation of a novel
open framework for agile development of mobile health applica-
tions, Biomed. Eng. Online 14 (2015), no. S2, S6.
3. D. Micucci, M. Mobilio, and P. Napoletano, Unimib shar: A
dataset for human activity recognition using acceleration data
from smartphones, Appl. Sci. 7 (2017), no. 10, 1101.
22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
435
4. A. Reiss and D. Stricker, Introducing a new benchmarked dataset
for activity monitoring, in Proc. Int. Symp. Wearable Comput.,
(Newcastle, UK), June 2012, pp. 108–109.
5. D. Roggen et al., Collecting complex activity datasets in highly rich
networked sensor environments, in Proc. Int. Conf. Netw. Sens.
Syst. (INSS), (Kassel, Germany), June 2010, pp. 233–240.
6. S. Chung et al., Sensor data acquisition and multimodal sensor
fusion for human activity recognition using deep learning, Sensors
19 (2019), no. 7, 1716.
7. C. Y. Jeong and M. Kim, An energy-efficient method for human
activity recognition with segment-level change detection and deep
learning, Sensors 19 (2019), no. 17, article no. 3688.
8. O. F. İnce et al., Human activity recognition with analysis of angles
between skeletal joints using a rgb-depth sensor, ETRI J. 42 (2020),
no. 1, 78–89.
9. M. Kim and C. Y. Jeong, Label-preserving data augmentation for
mobile sensor data, Multidimens. Syst. Signal Process. 32 (2021),
no. 1, 115–129.
10. C. Gurrin, A. F. Smeaton, and A. R. Doherty, Lifelogging:
Personal big data, Found. Trends Inf. Ret. 8 (2014), no. 1, 1–125,
doi:10.1561/1500000033.
11. A. J. Sellen and S. Whittaker, Beyond total capture: A constructive
critique of lifelogging, Commun. ACM 53 (2010), no. 5, 70–77,
doi:10.1145/1735223.1735243.
12. SIGMM, Lsc ’18: Proceedings of the 2018 ACM Workshop on the
Lifelog Search Challenge, ACM, New York, NY, USA, 2018.
13. C. Gurrin et al., LTA 2016: The first workshop on lifelogging tools
and applications, in Proc. ACM Int. Conf. Multimed. (New York,
NY, USA), Oct. 2016, pp. 1487–1488.
14. D. T. Dang Nguyen et al., Overview of imageCLEFlifelog 2019:
Solve my life puzzle and lifelog moment retrieval, in Proc. Conf.
Labs Eval. Forum. (Lugano, Switzerland), Sept. 2019.
15. C. Gurrin et al., Overview of the NTCIR-14 lifelog-3 task, in Proc.
NTCIR Conf. (Tokyo, Japan), June 2019, pp. 14–26.
16. Y. Vaizman, K. Ellis, and G. Lanckriet, Recognizing detailed
human context in the wild from smartphones and smartwatches,
IEEE Pervasive Comput. 16 (2017), no. 4, 62–74.
17. Y. Vaizman et al., Extrasensory app: Data collection in-the-wild
with rich user interface to self-report behavior, in Proc. CHI Conf.
Hum. Factors Comput. Syst. (New York, NY, USA), Apr. 2018,
pp. 1–12.
18. Y. Vaizman, N. Weibel, and G. Lanckriet, Context recogni-
tion in-the-wild: Unified model for multi-modal sensors and
multi-label classification, Proc. ACM Interact. Mob. Wearable
Ubiquitous Technol. 1 (2018), no. 4, 1–22.
19. G. M. Harari et al., Smartphone sensing methods for studying
behavior in everyday life, Curr. Opin. Behav. Sci. 18 (2017), 83–90.
20. L. A. Clark and D. Watson, Mood and the mundane: Relations
between daily life events and self-reported mood, J. Pers. Soc.
Psychol. 54 (1988), no. 2, 296–308.
21. P. R. Giacobbi, H. A. Hausenblas, and N. Frye, A naturalis-
tic assessment of the relationship between personality, daily life
events, leisure-time exercise, and mood, Psychol. Sport Exerc. 6
(2005), no. 1, 67–81.
22. S. N. Rogers et al., The addition of mood and anxiety domains to
the university of washington quality of life scale, Head & Neck: J.
Sci. Specialties Head Neck 24 (2002), no. 6, 521–529.
23. K. J. Stewart et al., Are fitness, activity, and fatness associated
with health-related quality of life and mood in older persons?, J.
Cardiopulm. Rehabil. Prev. 23 (2003), no. 2, 115–121.

436
24. K. H. Kim, S. W. Bang, and S. R. Kim, Emotion recognition system
using short-term monitoring of physiological signals, Med. Biol.
Eng. Comput. 42 (2004), no. 3, 419–427.
25. L. Shu et al., A review of emotion recognition using physiological
signals, Sensors 18 (2018), no. 7, 2074.
26. J. Kim and E. André, Emotion recognition based on physiologi-
cal changes in music listening, IEEE Trans. Pattern Anal. Mach.
Intell. 30 (2008), no. 12, 2067–2083.
27. S. Koelstra et al., Deap: A database for emotion analysis; using
physiological signals, IEEE Trans. Affect. Comput. 3 (2011), no.
1, 18–31.
28. T. Song et al., Mped: A multi-modal physiological emotion
database for discrete emotion recognition, IEEE Access 7 (2019),
12177–12191.
29. M.-F. Shao et al., Sleep quality and quality of life in
female shift-working nurses, J. Adv. Nurs. 66 (2010), no. 7,
1565–1572.
30. J. Zeitlhofer et al., Sleep and quality of life in the austrian popu-
lation, Acta Neurol. Scand. 102 (2000), no. 4, 249–257.
31. B. Bower et al., Poor reported sleep quality predicts low positive
affect in daily life among healthy and mood-disordered persons, J.
Sleep Res. 19 (2010), no. 2, 323–332.
32. D. K. Thomsen et al., Rumination–relationship with negative
mood and sleep quality, Pers. Individ. Differ. 34 (2003), no. 7,
1293–1301.
33. K. J. Reid et al., Aerobic exercise improves self-reported sleep and
quality of life in older adults with insomnia, Sleep Med. 11 (2010),
no. 9, 934–940.
34. S. Ancoli-Israel et al., The role of actigraphy in the study of sleep
and circadian rhythms, Sleep 26 (2003), no. 3, 342–392.
35. A. D. Krystal and J. D. Edinger, Measuring sleep quality, Sleep
Med. 9 (2008), no. 1, S10–S17.
36. D. J. Buysse et al., The pittsburgh sleep quality index: A new instru-
ment for psychiatric practice and research, Psychiatry Res. 28
(1989), no. 2, 193–213.
37. G. Landry, J. Best, and T. Liu-Ambrose, Measuring sleep qual-
ity in older adults: A comparison using subjective and objective
methods, Front. Aging Neurosci. 7 (2015), article no. 166.
38. S. Shiffman, A. A. Stone, and M. R. Hufford, Ecological momen-
tary assessment, Annu. Rev. Clin. Psychol. 4 (2008), 1–32.
39. D.-T. Dang-Nguyen et al., in Proc. Int. Workshop Content-based
Multimed. Indexing (New York, NY, USA), June 2017,
pp. 1–6.
40. C. Gurrin et al., Ntcir lifelog: The first test collection for lifelog
research, in Proc. Int. ACM SIGIR Conf. Res. Dev. Inf. Retr.
(New York, NY, USA), July 2016, pp. 705–708.
41. D.-T. Dang-Nguyen et al., Overview of imageCLEFlifelog 2017:
Lifelog retrieval and summarization, in Proc. ImageCLEF 2017
(Dublin, Ireland), Sept. 2017.
42. D.-T. Dang-Nguyen et al., Overview of imageCLEFlifelog
2018: Daily living understanding and lifelog moment retrieval,
in Proc. Conf. Labs Eval. Forum. (Avignon, France),
Sept. 2018.
43. J. A. Russell, A circumplex model of affect, J. Pers. Soc. Psychol.
39 (1980), no. 6, article no. 1161.
44. Empatica, E4 wristband: Real-time physiological signals, Avail-
able from https://empatica.com/research/e4 [last accessed
March 2021].
22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHUNG ET AL.
45. Withings, Sleep tracking mat, Available from https://www.
withings.com/kr/en/sleep [last accessed March 2021].
46. M. Barzegar, A. Sadeghi-Niaraki, and M. Shakeri, Spatial expe-
rience based route finding using ontologies, ETRI J. 42 (2020), no.
2, 247–257.
47. F. Bergadano, Keyed learning: An adversarial learning
framework–formalization, challenges, and anomaly detection
applications, ETRI J. 41 (2019), no. 5, 608–618.
48. J. K. Bii, R. Rimiru, and R. W. Mwangi, Adaptive boosting in
ensembles for outlier detection: Base learner selection and fusion
via local domain competence, ETRI J. 42 (2020), no. 6, 886–898.
AU THOR BIOGRAPHIES
Seungeun Chung received her
BS and MS degrees in computer
science from Seoul National Uni-
versity, Seoul, Rep. of Korea,
in 2007 and 2009, respec-
tively, and her Ph.D. degree in
computer science from North
Carolina State University, NC, USA, in 2016. From
2009 to 2010, she worked as a junior researcher at Tmax
R&D Center, Seongnam, Republic of Korea. She is a
senior researcher in the artificial intelligence labora-
tory at Electronics and Telecommunications Research
Institute, Daejeon, Republic of Korea. Her research
interests are in the areas of machine learning with
expertise in human behavior and context recognition.
Chi Yoon Jeong received his BS
and MS degrees in electronic and
electrical engineering from Pohang
University of Science and Technol-
ogy, Pohang, Republic of Korea,
in 2002 and 2004, respectively,
and his Ph.D. degree in computer
science from the Korea Advanced Institute of Sci-
ence and Technology, Daejeon, Republic of Korea,
in 2018. He is currently a principal researcher in
the artificial intelligence laboratory at Electron-
ics and Telecommunications Research Institute,
Daejeon, Republic of Korea. His current research
interests include computer vision, pattern recognition,
machine learning, and emotion recognition.
Jeong Mook Lim received his BS
and MS degrees in computer science
from Chungnam National University,
Daejeon, Republic of Korea, in 1998
and 2000, respectively. He is currently
a principal research scientist in the
Human Enhancement & Assistive

CHUNG ET AL.
Technology research section at Electronics and
Telecommunications Research Institute, Daejeon,
Republic of Korea. His primary research interests
include understanding human behavioral patterns and
psychological states with smartphones.
Jiyoun Lim received her BS,
MS, and Ph.D. degrees in indus-
trial and systems engineering
from Korea Advanced Insti-
tute of Science and Technology,
Daejeon, Republic of Korea, in 2005,
2007, and 2017, respectively. She
worked at Korea University of Technology and Edu-
cation, Cheonan, from 2013 to 2017. She is a senior
researcher in the artificial intelligence laboratory
at Electronics and Telecommunications Research
Institute, Daejeon, Republic of Korea. Her research
interests include data engineering and human context
recognition.
Kyoung Ju Noh is currently a
principal researcher in the arti-
ficial intelligence laboratory at
Electronics and Telecommu-
nications Research Institute,
Daejeon, Republic of Korea. She
received her BS and MS degrees
in computer science from Chonbuk National Uni-
versity, Jeonju, Republic of Korea. She has been
involved in communications and personal computing
science-related projects for over a decade. She is cur-
rently developing affective computing technology for
human understanding. Her research interests include
human understanding, affective computing, artificial
intelligence, and emotion recognition.
Gague Kim is a principal researcher
in the artificial intelligence labora-
tory at Electronics and Telecommu-
nications Research Institute, Dae-
jeon, Republic of Korea. He received
his Ph.D. in electronic engineering
from Kyungpook National University,
22337326, 2022, 3, Downloaded from https://onlinelibrary.wiley.com/doi/10.4218/etrij.2020-0446 by Air University, Wiley Online Library on [22/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
437
Daegu, Republic of Korea, in 2000. His research inter-
ests include emotion recognition based on physiolog-
ical signals, classification with imbalanced datasets,
affective computing and interaction, gesture recogni-
tion, machine learning algorithms for emotion recog-
nition, and embedded software and systems.
Hyuntae Jeong received his BS
and MS degrees in electronic engi-
neering from Chungnam National
University, Daejeon, Republic of
Korea, in 1993 and 1995. From
1995 to 2000, he worked at Sam-
sung Heavy Industries R&D Center,
Daejeon, Republic of Korea, developing control
systems for ship experiments. Since 2001, he has
been worked at Electronics and Telecommuni-
cations Research Institute, Daejeon, Republic of
Korea. He has been involved in the development of
mobile and wearable computing technologies. His
current research interests include cognitive com-
puting, artificial intelligence, wearable computing
and HCI.
How to cite this article: S. Chung, J.M.
Lim, J. Lim, K. Ju Noh, G. Kim, H. Jeong.
Real-world multimodal lifelog dataset for human
behavior study, ETRI Journal 44(2022), 426–437.
https://doi.org/10.4218/etrij.2020-0446