Last lecture: location

1. Applications and Problem Specification

2. Location sensing
outdoor: GPS
indoor: Vision

indoor: Beacons

Indoor inertial
indoor: Contact/Proximity tracking

3. Use of Stochastic Filtering

Kalman

Particle Filter

1
 Summary: the 5 Dimensions

1. Extended time and location

2. User activity
3. User state
4. Environment state
5. System state

2
 User Activity
Modes of locomotion and posture

lying down, sitting, standing

walking, jogging, running
up-stairs, down stairs
bicycling, skateboarding,.

jumping on one leg and turning in the air
.......
3
 User Activity

Object Interaction
simple objects
pick up, put away, press
complex interaction with tools

4
 User Activity
Interaction with people
vocal interaction
speaking, listening
Discussion
physical interaction
shake hands, hug
gestures
Fight
social signals
smile

5
Modes of Locomotion

6
 User Activity
Modes of locomotion and posture

lying down, sitting, standing

walking, jogging, running
up-stairs, down stairs
bicycling, skateboarding,.

jumping on one leg and turning in the air
.......
7
 Walking
a typical human activity

8
 Walking
up-down motion skin/clothing
forward motion muscle flexion
shape change

rotation

contact/pressure distance change

footsteps sound/vibration; light/RF signal blocked; heart rate changes ...

9
 Motion Sensors

accelerometer: gyroscope (angular rate compass (magnetic field

change of speed sensor): sensor):
(acceleration) change of angle (angular direction of north
rate)
how rapid I am changing my or absolute angle my orientation
speed
how fast I am turning
10
 Acceleration Sensor
Acceleration tilts the plates changing the
distance to the substrate
The voltage across the capacitor changes with
the distance

11
MEMS Accelerometers

10-, 12- and 14-bit accelerometers

from Freescale MMA845xQ

analog & digital available

12
 Accelerometer

Acceleration sensors provides two types of information

13
 Accelerometer

Acceleration sensor provides two types of infomration

Z
n Angle towards the gravity vector (tilt) 1g

X

14
 Acceleration Sensor

Acceleration sensor provides two types of infomration

Z
n Angle towards the gravity vector (tilt
n Change of motion speed
X

15
 Acceleration Sensor

Acceleration sensors provides two types of infomration

n Angle towards the gravity vector (tilt)

n Change of motion speed
1g

n These components can not be easily separated !

16
 Example

+1g

-1g
+2g /+2g

0g /0g

-2g /-2g

greeting a person
17
 Filtering the Signal

Raw signal

Low pass (fcut=2Hz)

High pass (fcut=2Hz)

to learn more:
18
Matlab/toolbox/Filter Design and Analysis Tool
 Filtering the Signal

Raw signal

Low pass (fcut=2Hz) Low pass

High pass (fcut=2Hz) High pass

19
Acceleration on the Upper Leg
acceleration in direction along the leg

20
 Exercises now
plot out the output of accelerometer fixed along
the arm when
standing still
walking
sitting with lower arm lying on the table
scratching nose/head
reaching for the door

21
 Homework
put your smart phone in the pocket and collect
data when you
standing still
walking
running
go upstairs
go downstairs

22
 Problem type
Signal stream window size

situation

shift

state action

parameters
- separable in time invariant
feature space
- sliding window algorithms

23
 Recognition Method
Signal stream window size

shift
Features (min, max, mean variance, std deviation,)

c2 classifier (KNN, neural network, tree, SVM,)

x1 x x
x x
x x x
x x c1
x x
x
x x x
x
c3 x
x
Walk, run, sit, stand,.
x x
24
Walking Up and Down
Stairs
Acceleration on the upper leg

Down Up

25
Sliding Window Features

26
Sensor Requirements

27
Sensor Requirements

correctly classified segments

Recognition rate =
all segments

28
 Case Study
Recognition Task Sensor Setup
Modes of locomotion

Air pressure
Accelerometer

Accelerometer

Evaluation Method

Recognition using standard classifier (kNN) and 4 simple features

Evaluation based on data sets containing only targeted activities

29
30
 Example

Original Signal

Level walking, steps up, steps down

31
Frequency Based Features
122 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICI

and spectral entropies were estimat

local FFT respiratory effort signal.

D. Feature Selection
Feature selection was based on vis
The features were visually compare
good candidate features. Distribution
signal during different activities we
(Fig. 4). The plots show how the d
signal changes between different ac
tribution moves between activities a
overlap, the better it is for discrimin
A priori information was used in
tures. For example, during running th
Fig. 2. Spectogram of vertical acceleration on chest during walking, Nordic ment and thus more energy in acce
walking and running. Horizontal axis is time.
other activities. Based on a priori i
tures were calculated from raw da
Often more informative then simple statistical
C. Signal Processing and Feature Extraction selected based on the distribution ba
than one feature that could have be
The goal in context recognition is to develop algorithms that
can automatically infer features
sion, the feature with best discrimin
the annotated context from the collected
As a result of the feature selec
signals. The signals were first visually inspected and compared
(Fig. 4) were selected
32 for classifica
against the annotated contexts. This gave us the first impression
up-down chest acceleration Fpeak
on which signals are more useful than others. Feature signals
up-down chest acceleration Med(c
(1-Hz sampling rate) were calculated from the raw data.
 An Example of Frequency Analyze
Koji Yatani, Khai N. Truong, Bodyscope: A Wearable Acoustic Sensor for Activity Recognition

33
 Classifier Choice
PARKKA et al.: ACTIVITY CLASSIFICATION USING REALISTIC DATA FROM WEARABLE SENSORS

TABLE VI Walking is one of the

CLASSIFIER RESULTS [%]
set as in everyday life.
tion of decisions footst
walking. Distributions o
overlap when using the f
confuse walking mostly
ting/standing. The artific
mostly with sitting/stand
can be explained with in
notated as walking often
Very short periods of wa
ties, even if annotated co
A. Confusions
choice of classifier can have big impact on results
decision tree, are replaced
Much of our classifiers confusion seen in the results can processing method in the
be explained with transitions from one activity to another. The performance when lots o
annotator was not given the choice to annotate transition, but Nordic walking was de
he had to switch from one activity to another instantly at some approach34is successful w
point during the transition. The transition is sometimes gradual, they should be used. Peo
Time Series Recognition
Process P Process P

t
z1 z1

z2 z2

z3 z3

zN zN
different processes, gives different output series in time
35
 Time Series Recognition
Prozess P Prozess P

?????
t
z*1
Process models like
Hidden Markov models z*2
Conditional Random Fields
z*3

z*N
output series in time known, then what is the process behind?
36
 Example: The Dishonest Casino

A casino has two dice:

Fair die
P(1) = P(2) = P(3) = P(5) = P(6) = 1/6
Loaded die
P(1) = P(2) = P(3) = P(5) = 1/10
P(6) = 1/2
Casino player switches back-&-forth between
fair and loaded die once every 20 turns

Game:
1. You bet $1
2. You roll (always with a fair die)
3. Casino player rolls (maybe with fair die,
maybe with loaded die)
4. Highest number wins $2
 The dishonest casino
model

0.05
0.95 0.95

FAIR LOADED

P(1|F) = 1/6 P(1|L) = 1/10

P(2|F) = 1/6 P(2|L) = 1/10
P(3|F) = 1/6 0.05 P(3|L) = 1/10
P(4|F) = 1/6 P(4|L) = 1/10
P(5|F) = 1/6 P(5|L) = 1/10
P(6|F) = 1/6 P(6|L) = 1/2

38
HMM model

Probabilistic parameters of a
hidden Markov model (example)

x states (hidden)
y possible observations
a state transition probabilities
b output probabilities

39
Question # 1 Evaluation

GIVEN

A sequence of rolls by the casino player

1245526462146146136136661664661636616366163616515615115146123562344

Prob = 1.3 x 10-35

QUESTION

How likely is this sequence, given our model of how the casino works?

This is the EVALUATION problem in HMMs

40
Question # 2 Decoding

GIVEN

A sequence of rolls by the casino player

1245526462146146136136661664661636616366163616515615115146123562344

FAIR LOADED FAIR

QUESTION

What portion of the sequence was generated with the fair die, and what
portion with the loaded die?

This is the DECODING question in HMMs

41
Question # 3 Learning

GIVEN

A sequence of rolls by the casino player

1245526462146146136136661664661636616366163616515615115146123562344

Prob(6) = 64%

QUESTION

How loaded is the loaded die? How fair is the fair die? How
often does the casino player change from fair to loaded, and
back?

This is the LEARNING question in HMMs 42

 Window Size Effects 2 seconds
43

Optimum Actual activities

Recognition results for partitioned data set

100 samples per second

correctly classified segments

Recognition rate =
all segments
time
n Small window:
segments do not contain sufficient information
n Large window:
short activities not resolved anymore
what should be the window size if I want to detect walking/running/sitting?
 Findings
Sliding window approach
feasible approach for recognition of periodic
activities
window size has significant influence on recognition
shift: trade-off between resolution and computation
costs
major drawback: artificial partitioning where
parameters must be set explicitly

Alternative approach
event-based signal partitioning
recognition on step-basis
44
 Motion Sensors

accelerometer: gyroscope (angular rate compass (magnetic field

change of speed sensor): sensor):
(acceleration) change of angle (angular direction of north
rate)
how rapid I am changing my or absolute angle my orientation
speed
how fast I am turning
45
Gyroscope Background

YoYo
Factory
Spin
Top
Video

46
Traditional Gyroscope

47
 MEMS gyroscope
http://electroiq.com/blog/2010/11/introduction-to-mems-gyroscopes/

Coriolis Effect
An object moving in a reference frame
(coordinate) which is rotating, "receives" in this
coordinate a force when the moving direction is
different to the rotation axis, "pushing" the object
aside from its original path
48
 Coriolis Effect
Coriolis Effect
An object moving in a reference frame
(coordinate) which is rotating, "receives" in this
coordinate a force when the moving direction is
different to the rotation axis, "pushing" the object
aside from its original path

Examples in our life

49
 MEMS gyro
http://electroiq.com/blog/2010/11/introduction-to-mems-gyroscopes/

1) two masses excited to vibration in x direction

2) force in y direction is in linear with angular rate along z direction
3) vibration(distance between masses) in y direction in linear to angular rate
along z direction (dmax~ |z|)
4) capacitance between masses drops with distance between (C~ 1/d)
5) capacitance change is in linear with angular rate along z direction
(Cmax~dmax/d2~|z|)
6) fixed charge on masses -> voltage output shows the angular rate
50
 MEMs gyroscope

Inertial Measurement Unit (IMU) normally has 3

gyroscopes combined into 1 (or say 3 axis gyroscope),
and sense the angular rate

gyroscope in iPad
tremble hand compensation when taking pictures
51
 IMUs

XSENSE inertial Measurement Unit (IMU)

3 axis accelerometer
3 axis magnetic
3 axis gyro

absolute orientation of the device with respect to

gravity direction (based on accelerometer)
north (based on magnetic field sensor)

InvenSense MPU-6500
6-Axis (Gyro + Accel.) MEMS MotionTracking Devices
in a 3 x 3 x 0.9 mm package
52
Workflow Detection
With Accel. and Gyro

53
 IMU as a default

what is the typical readout of the IMU at typical locomotion?

. when lying in the pocket / on the table

. when the screen is turned 90 degree
. when walking
. when performing squats

54
 Hack it in 48 hours!

7 Universities partecipating:
48 hours to develop an Embedded System
Must use the given platform EFM32 Giant Gecko
Free food, coffee etc. and most important free Red Bulls J

55
EMECS-thon 2015 - TUKL - MIDFIT (Make it work! Don't fix it)

mobile phone
.raw data from accelerometer
.through Bluetooth out

processing unit
.accelerometer raw data to control information
(speed, angle)
.VGA control

play golf with your mobile now!

thanks to:
Fabian Faul
Johannes Stlp
Vasiar Allaj
56
EMECS-thon 2015 - TUKL - MIDFIT (Make it work! Don't fix it)
https://www.youtube.com/watch?v=pkLiHEAMqEo&t=3m42s 57
 FSR for Muscle Activity Monitoring

typical force curve

FSR: Force Sensing Resistors - thickness <0.5mm

- sampling rate >100Hz
- power loss <1mW

58
 Experiment

Level Walking

Ascending
Two FSRs placed under
right shoe of each subject
Each subject walked pre-
defined path without
Descending further instructions
Data was labeled during
experiment

59
 Lag Extraction

1. Find initial ground

contacts of heel and
ball

T2 2 T2 T2+1
2. Identify ball and heel
strikes that belong
together

T1 T2
3. Calculate lag (T2-T1)
between ball and
corresponding heel
strike

60
Lags

Lag correlates well with modes

of locomotion
Positive lags indicate level walk
Negative lags indicate initial ground
contact with forefoot

Change in lag values when

approaching stairs or walking
on landings o
Due to adaptation of stride length

Missing heel strikes during

ascending o

61
 Intra-variability

Interquartile range
(IQR)

Level Walk
Ascending
Descending

box plot is a convenient way of plotting statistics data:

http://en.wikipedia.org/wiki/Box_plot
62
Intra-and Inter-variability

Mean STD
Intra-Variability: Level + 0.12 0.055
Ascend -0.002 0.02
6 data sets of same subject
Descend - 0.084 0.032

Mean STD

Level + 0.12 0.05

Inter-Variability: Ascend +0.002 0.02
4 data sets from 4 subjects
Descend - 0.078 0.02

63
 Recognition
Bayes Classifier: arg max Class p(Class / Lag)

Evaluation: 6-fold cross validation

Confusion Matrix

Classification Level Walk Ascend Descend

Ground Truth
Level Walk 99.5 % 0.4 % 0.1 %

Ascend 3.2 % 94.2 % 2.6 %

Descend 1.2 % 1.6 % 97.2 %

Result: Excellent Recognition performance

64
 Muscle Movement

Motion is the result of cooperation of a group of muscles

Even small difference in the motion pattern is visible in muscles' behavior

Up vs. Downstairs

down up

. less effort of back muscles

. much less effort of back muscles when going upstairs
. relation between effort and pressure is not linear

66
Joints Friendly Downhill Walking

normal downstairs walking shock-absorbing downstairs walking

. more balanced front and back muscle effort when walking joint friendly

67
Long vs. Short Strides

short long

. longer strides train the front muscles

68
Modes of Locomotion Recognition

69
 Muscle Activity Intensity
Example

pushing up increasing weight stepper with increasing speed

70
 Squats Technique

correct asymmetric

cheat

71
 from FSR to Resistive Array

Hardware.jpg

Zhou,B., Sundholm, M., Cheng, J., Cruz, H., Lukowicz, P., Never Skip Leg Day: A Novel Wearable Approach to Monitoring Gym
Leg Exercises, PerCom2016 72
Sound Based Detection

SPECTRUM NOT WALKING

SPECTRUM WALKING

74
75
Further Examples

Each sensor has part of the information,

fuse them to get the big picture with confidence.
76
 Synesthesia
sensation together

I feel BLUE so SWEET

a letter has a color? a shape has a sound?

77
 Butoh Dance Visualization
Emotionen

Visualization

withProf. Gutknecht
real motion -> word, shape, color, background image...
ETH Zrich
78
Butoh Dance Visualization

79
 Homework
try sliding window approach
data recording
(standing *20s + walking *20s + running *20s ) *5
repeats
synchronization at start/end of each activity for
labelling
better share data also with others
data pre-processing
plot raw data with timestamps
create labels (ground truth by looking at the data)

80
 Homework
try sliding window approach
sliding window
window size and step as a parameter
short-time Fourier transform and plot result
calculate features
plot features with timestamps + labels
select 3 features and plot in 3-D space the
distribution (different labels with different markers)

81
 Homework
possible features
W. Dargie, "Analysis of Time and Frequency Domain Features of
Accelerometer Measurements," Computer Communications and
Networks, 2009. ICCCN 2009. Proceedings of 18th Internatonal
Conference on, San Francisco, CA, 2009, pp. 1-6.

82