My

PhD Project
Quantifying Visual Cues of
Psychological States in
Interview Situations (QVCPS)
18/05/2005
Enrica Dente
Email: enrica.dente@imperial.ac.uk
Website: http://www.enricadente.com/imperial/
Enrica Dente, QVCPS Project 2
Contents
Problem
Hypothesis
Project Objectives
Project Plan
Achievements to Date
Conclusion

Enrica Dente, QVCPS Project 3
Problem
Most lie detection approaches can only detect
psychological states, not deception (i.e. are less than
80% accuracy) [Vrij, 2004]
Manual observation of human behavior subjective and
limited [Burgoon, 2005]
Polygraph, 100 years old lie detector problematic but still in
use
Increased interest in alternative approaches:
Automated Face Analysis (i.e. changes in facial
expressions)
Thermal imaging (i.e. changes in blood flow around
eyes)
FMRI (i.e. BOLD)
Brain fingerprinting (i.e. brain waves)
Eye tracking (i.e. irregular eye movements in front of
images )
Intrusiveness yet to be addressed.

Enrica Dente, QVCPS Project 4
Hypothesis

Non skilled liars more likely to experience cognitive load
and controlled behavior

[Vrij, 2004]

Decrease of hand/finger movements when subjects lie yet
to be proved (or disproved)
Quantification of observations required
Focus on tracking of hand/finger movements in interview
situations
Real life cases can make a difference in results but
problematic [Vrij 2005]
Verbal behavior required to add context to the tracking
Enrica Dente, QVCPS Project 5
Project Objectives
To automate the tracking of finger/hand movements in
interview situations
To compute the frequency of hand and finger movements
normalised to the baseline of each individual
To build a biomechanical model to aid hand gesture
tracking
To provide a simple means of accessing and referring to
the verbal behaviour, when required
Later, to compare the movements of the interviewee with
the movements of the interviewer.

Enrica Dente, QVCPS Project 6
Project Plan
Year 1:
Skin Colour Modelling
Hand Tracking and Finger Position
Coding behaviour markup
Background Modelling
Graphical User Interface
Experimental Protocol
Year 2 and Year 3:
Kinematic models of upper body, limb and finger motion
Introduction of verbal behavior to add context to tracking
Train system to quantify different types of hand and finger
movements
Writing up.

Enrica Dente, QVCPS Project 7
Achievements to Date:
1) Segmentation
(a) Skin Colour Modelling
Color Predicate [Kieldsen, 1996]
Bayesian Posterior Map and Parzen colour
space probability density estimates more
accurate.
(b) Hand Tracking and Finger Position
Connected Component and central moments
Centroid correspondence
Hands separation based on area and distance
heuristics
Complex wavelet decomposition to detect
changes in finger position.

Next Step: background modelling using
parametric methods to address robustness.

Figure 1 - Hand Tracking
Figure 2 - Finger Orientations
Enrica Dente, QVCPS Project 8
1. Joint Conditional Density function in hue and sat for hand and non-
hands (i.e. Parzen density estimation)







where H and S represent hue and sat, Hn and Sn represent the observed
values amongst the N
H
hands training set pixels, K
H
is a normalised
constant and
0
species the width of the kernel used in smoothing

2. Bayesian Posterior Map to identify most probable locations of skin in
each frame





where we assume that p(H,S | Hands) is approximated by the left hand
side of Equation (1) and p(H,S | NonHands) is approximated by the left
hand side of Equation (2).

(a) Skin Colour Modelling
) , (
) ( ) | , (
) ,
S H p
Hands p Hands S H p
S (Hands|H p =

=
+

=
=
H
h
n
h
n
H
N
n
S S H H
H
N n
h
n
h
n
e
K
S ds, {H (H, S|Han p
1
2
) ( ) (
... 1
2
0
2 2
1
) } ,
o
(1)
(2)
Enrica Dente, QVCPS Project 9
(a) Skin Colour Modelling
(cont.)
Figure 3 - Likelihood function for hand regions in a), for non-hand regions in b)
Parzen colour space probability density estimates for
hand and for non-hand regions
There is overlap between the two classes

Enrica Dente, QVCPS Project 10
(b) Hand Tracking and Finger
Position
HAND TRACKING:
Centroid correspondence of each object across frames from
binary image created in (a)
Connected component analysis to label blobs.
Switch from a one-hand to two-hand state by using distance
and area heuristics.

FINGER POSITION:
Complex wavelet decomposition
Orientation of local image structure relatively invariant to
local phase
Filter bank of complex wavelets, each tuned to one of four
directions
We compare output of four complex masks to make estimate
of the presence of a line or an edge in each direction
Vector indicating the direction of local image structure
estimated by a weighted vector summation operation
Multirate scheme (i.e. series of different scales).


Enrica Dente, QVCPS Project 11
Finger Position Estimate
K/2 of the real and imaginary impulse responses of
wavelets at scale 1 for K = 8






Orientation field O(l) (m, n) is defined by:




2 / 1
1
0
2
) (
1
0
2 ) (
) (
) , (
) , (
) , (
|
|
.
|

\
|
+
=

=
K
k
l
k
K
k
j l
k
l
n m f p
e n m f
n m O
k
|
Figure 4 - Wavelet Kernels
Enrica Dente, QVCPS Project 12
Finger Position Estimate
(.cont)
Histogram of orientations for each hand in each frame
Weighting of the product of the orientation
Field magnitude and posterior skin tone map to appropriate
histogram bin
Orientation density functions can be used to detect finger
movement!
Figure 5 (a) Hands together (b) Hands moving
Enrica Dente, QVCPS Project 13
Achievements to Date:
2) XML Markup for Behavior Coding
Check Verbal behavior:
Check Non Verbal behavior:
Enrica Dente, QVCPS Project 14
Achievements to Date:
3) Graphical User Interface (GUI)
Enrica Dente, QVCPS Project 15
Achievements to Date:
4) Experimental Protocol
Stakes low in current lie detection experiments:
Interviewees are asked to lie on simulated actions
Interviewees less motivated to lie than in real-life
situations, performance less effective
Vrijs real-life police interview limited to visual
observation of one liar
Problem: questions based on prior knowledge of what
the interviewees will lie on and when
Our experimental protocol aims to raise the stakes of
deception detection in interview situations:
Solution: design experiments where interviewees are
not told in advance what they have to lie on.





Enrica Dente, QVCPS Project 16
Literature Review
1) Skin Colour Modelling
Non Parametric Methods
Create an histogram or lookup table (LUT) common and
simple
However, storage required, bins may miss pixel values,
noise removal and occlusion still an issue
Bayesian posterior map computed from hands and non
hands histograms more promising than Colour Predicate
Parametric Methods
Model skin colour as components of mixtures of Gaussians
Provide robustness, no need for storage space and ability to
interpolate or generalise the training data
However, they require accurate initialization and assume
number of components to be known in advance






Enrica Dente, QVCPS Project 17
Literature Review
1) Skin Colour Modelling
Parametric Methods (cont.)
Foreground and background modelled by a joint
probability density function
Model parameters (i.e. means and covariance) for the
Gaussians estimated from the training data using
maximum likelihood or Bayesian inference
Updated online based on colour, on position and/or on
motion information
Skin probability computed from Gaussian Probability
Density Functions (PDF's)
However, in interview situations update cannot be based
on assumption that foreground is moving.


Enrica Dente, QVCPS Project 18
Literature Review
2) Kinematic 3-D models
Need accurate tracking of hand/finger gestures
Choice of degrees of freedom crucial
Choice of type of model (i.e. stick figure or statistical)
Need markers for validation
Need simple and consistent movements
Does deception occur fast or slow in interviews?
How can we track several movements at the same time?
Number of cameras?
How visible can hand's edges be in interviews? Camera
resolution?



Enrica Dente, QVCPS Project 19
Conclusion
Problem and Hypothesis:
Most lie detection approaches detect psychological
states, not deception
Non skilled liars more likely to experience cognitive
load and controlled behavior
Quantification of observations required
Achievements to Date:
Hand Tracking using Bayesian Posterior Map
Finger Position using Complex Wavelets
XML markup for correlating visual and verbal behavior
Experimental Protocol
Next Step:
Background modelling to address robustness.


Enrica Dente, QVCPS Project 20
Questions?
Enrica Dente, QVCPS Project 21
References
[Vrij, 2004] Vrij Aldert, Psychology, Crime and Law, Challenging
Interviewees During Interviews, The Potential Effects on Lie
Detection, 2004
[Vrij2, 2004] Vrij Aldert, Why Professionals fail to catch liars
and how they can improve, Legal and Criminological
Psychology, 9, 159-181, 2004
[Vrij, 2002] Aldert Vrj, Detecting Lies and Deceit : The
Psychology of Lying and the Implications for Professional
Practice, John Wiley & Sons, Inc
[Watson, 1081] Watson, K. W. Oral and written linguistic indices
of deception during employment interviews. (Doctoral
dissertation, Louisiana State University, 1981). Dissertations
Abstracts International, 42, 06A 2367.
[Parker et al., 2000] Parker, A. & Brown, J., Detection of
deception: Statement Validity Analysis as a means of
determining truthfulness or falsity of rape allegations. Legal and
Criminological Psychology, 5, 237-259, 2000

Enrica Dente, QVCPS Project 22
References (cont.)
[Bradley, 1993] Bradley MT, Cullen MC, Polygraph lie detection
on real events in a laboratory setting, 76(3 Pt 1):1051-8, Percept
Mot Skills. June 1993
[Etcoff, 2000] Nancy L. Etcoff*, Paul Ekman, John J. Magee,
Mark G. Frank, NATURE , VOL 405, 11 MAY 2000, Lie
detection and language comprehension, 2000 Macmillan
Magazines Ltd, Available Online: http://www.nature.com/cgi-
taf/DynaPage.taf?file=/nature/journal/v405/n6783/full/405139a0_
fs.html&content_filetype=pdf
[Kieldsen, 1996] R. Kjeldsen and J. Kender. Finding skin in color
images. In Second International Conference on Automatic Face
and Gesture Recognition, 1996.
[Burgoon, 2005] J. Burgoon et al. An approach for intent identi-
cation by building on deception detection. In Proc. of the 38th
Annual Hawaii Int. Conf. on Detection of Deception:
Collaboration Systems and Technology, 2005.
[Bharath, 2003] A. Bharath and J. Ng. A Steerable Complex
Wavelet Construction and Its Application to Image Denoising.
IEEE Transactions on Image Processing. June 2003