Multimodal Corpora

Di r k H e y len1 C ost anza N avar r et t a2

Pat r izia Pag gio2
(1) University of Twente, Netherlands
(2) University of Copenhagen, Denmark

d.k.j.heylen@utwente.nl, costanza@hum.ku.dk, patrizia.paggio@um.edu.mt

ABSTRACT
The purpose of this tutorial is to introduce participants to the area of multimodal corpora,
covering central aspects of the way in which multimodal corpora are collected, annotated,
analysed and used to model human behaviour as well as to build multimodal systems. The
course will be a mixture of theoretical presentations, discussions of annotated video examples
and some practical annotation exercises, covering the following three parts:
1. Language and non-verbal behaviour in human communication
Theoretical approaches to multimodal communication from the literature will be presented and examples from video material will be shown and discussed. Methodologies for the
classification of different types of non-verbal behaviour will be introduced, and the interaction
of gesture with speech will be discussed.
2. Annotation models and procedures
Various approaches to gesture annotation will be described, covering issues related to
the identification, segmentation and interpretation of multimodal contributions. Examples of
multimodal annotation created with the ANVIL annotation tool (www.anvil-software.de) will
be shown. Reliability measures for multimodal annotation will be introduced.
3. Multimodal corpora, analyses and systems
Issues covered in this part include an overview of existing annotated multimodal corpora, a discussion of how multimodal corpora are analysed to gain insight into various aspects
of multimodal communication, how machine learning can be applied to multimodal data, and
how multimodal data are used to develop talking heads and embodied conversational agents.

!"
#$%&%'!$(!$)$*+',-!,'.+/)&!
/$!.&0$!1)00&$/1 /)$
02434
044526644786
934504864
72:8 ;<;<
0123457897
 3 0
9

0
  
  
 !42"#
$!
0% 
$ !

0123457897
 3
 0
7

   

  
 !""
"#"""$
%& "' """
'
"&"
" (" "
)*"

 )""
"

$

0123457897
 3
 0


5
3 !"
 ""
#$#

 " 

% 


!&
 !'

0123457897
 3
 0




 !" #$%&'()&*+,%$%-.

/

01
21

2

031
1

4&))5'%"+$%&'()&*+,%$%-.

10
06
217
0
/ 


4&)#5$!()&*+,%$%-(8%'#5$(+'*(
&5$#5$9.

:
6/1; 
60
<=>?@ABBC
1 
/1


1D

0123457897
 3
 0


 ! " "

#"!

! $$%%%&&" $%"!'()0*+,98./"

# 

! $$%%%&&" $%"!'()50(

123) 3
03 3!3"3!
4!('

0123457897
 3
 0


0 

  !!!!

" #$% 
%!&9''()*
+ 0!! 
," 
,

,
-
+ %# 
*$)!
#
$)
!#
$)
$,)
#
$,),0123457897
 3
 0


 
4 !"
# 
 $%
&
'(!)
' %

 *"

0123457897
 3
 0


0 !

"!!"!!

"#
$0 !%
$&!
$4 "!
$3%
!!!
$!"!%
$'
0123457897
 3
 0



42 ! "
# !$
! !
# $ % !
! !
#     &!
 
 

 
!'( ! !$

32)*+,-)
0123457897
 3
 0
98

0  

 

!"  

#"
 $%9&'8(
 
  
)
9* 5""+*
7* "  "*

0123457897
 3
 0
99

!!

!!"!"!!
#!!$%9&'&(!!"!!
!)!"! 
*!!"!!!! !
!%"4!9&'&
9&&7
788+(

0123457897
 3
 0
97

3

 
! "  #
!  # 
!  #" #$
# 
 %&9'(8
 49''7)
0123457897
 3
 0
9

 ! "#
$  % &

$  &'

0123457897
 3
 0
9

3!"

#$$

#%&$$!%!%
#%'
(%(
$ 
#$  
#)'
0123457897
 3
 0
9

!"#

$%&'&
!&''&!
$
$!('
)')*('&
&
$!!
0123457897
 3
 0
9

 !

" !

 
"  #
" 
" ! 
0123457897
 3
 0
9

3
! 
"#!
" #
"$!

0123457897
 3
 0
9

    

!"#$ "%
$& '(
)4#&" 
 
)*"&&
)3&$
)$ " + &
" ,  

$ "   

-

0123457897
 3
 0
9

 !"#$ %&''(#)*$+&,,#-../0

1234562738629:4;<=84<83>22?@
5AB
D  A

C A 
EF2FFA

1F1

H
GABAH
AABIAABI
AABI
AABI
0123457897
 3
 0
78

GABA

0

 


! "

 
9#$ %
7# 
&# 
'#"() 
 
*#+ (,
!
0123457897
 3
 0
79

0

5 !
" # 
5$
%&'(2
&)


&*
+ ("
%*

, 
-.
0123457897
 3
 0
77

0 

9!  " #

$%&'()*+,(-+(-),..'(+)/*+,(

0123457897
 3
 0
7

0 !
" #$%&'788()

''*!

+
,
''
  -.

/

01
0123457897
 3
 0
7

0 

9!  " #
7! 
$%&'()*+),-+./'

0123457897
 3
 0
7

 

 !

"#$%&#'()*+,-+.)(#/$01$&)123#(4)56(#7)08)
9:;
<99;=9
<>=?
@
 
;

0123457897
 3
 0
7

0
   

0123457897
 3
 0
7

  

 1 !"#
 $$!%!#
3!
"%&'()*+,
$3-.% "/01293495
6--%%%77! -%! #89:;09";
</%=$9
0123457897
 3
 0
7

 

!"#"" !$%&

" !'()
*+,-.+/012322451675175.,18955,2.:;1
<"! =>
"?@@<<<))A @<A"=BC  D74
E

0123457897
 3
 0
7

 

  !"#

 "$ "%&
3""' ()**+,-./-0.,1
0 23-/450.)61
7 8 
"9  "99  
 "
"

"
  
0123457897
 3
 0
8

3

0123457897
 3
 0
9

 

!""###$$% "#% &'()*+,

-.1/%

0123457897
 3
 0
7

3   !

"#$$$$ $$
$$$$%$
&$'$() $*$%$
*$$  $$$$$$+
, $
,
, $
,$

0123457897
 3
 0


0 !!

"#
$%&'()*+,
"-- ."/0"

12')*3)*4!00""!"#
156%7&')*3)*4"
"!-

0123457897
 3
 0


3

0123457897
 3
 0


 

0123457897
 3
 0




! "#  

  $%788&
! #
! # 
! ''

0123457897
 3
 0


3!

0123457897
 3
 0




0123457897
 3
 0




!!
 !

0123457897
 3
 0
8

 !788"

#4$ %&'()(*+,-./
#0 %&%
0&&1&"'
2345634.  "
2)378+78 &9$&"
253(+. &&9$$&"
278,:78
243+(*;,-78

0123457897
 3
 0
9


!"##$$$%% #$!&'()*+,
-$.
!"##$$$%% #$!&'( 02
/01*823(

0123457897
 3
 0
7

0!!

9" !!!# $

7" 
"! !

%&'()*+(,-./0*-+12*-3/4526/7)8/
89)76(,-/

0123457897
 3
 0




!" #
$
!$

$
%
!
"&


0123457897
 3
 0


 ! "#$



%$&'9()7

* +% 9,-,.

0123457897
 3
 0


 !"#$4 9%%7&

"#
"&
'( 
$
$' ) 
 

 ' ) 
0123457897
 3
 0


!2"#$%
&%%7889
'788
'788

5( %
5$
)#%
* $
0123457897
 3
 0


0!!

9"
7"
%"
"
)*

!!!# $
 
! !
&!'!!( 
 
+,-./0123

0123457897
 3
 0


 !

"  # $

%!!  # 
!$ !
!&% '&! '()
!%(
0! !
5& 
0123457897
 3
 0


 !"

 
##$


 %
&# '!'() *+,-./.01/2
394567 8


9
:!
!

 
 ###$
 

$
#'
0123457897
 3
 0
8

 ! 


   
"#$%&'()*+"%),'),'-).$(/0'+1$0'234,."'
9586
787! 



ABCDE

;<=>?;@

9

:
@;FD;GG;@

D;HA<;@

9

0123457897
 3
 0
9

 !  

 "#
0$ %  
 "#
4&4101'($
 )!! 

  *!* 
+

0123457897
 3
 0
7

   !"" "

" 
#$%&'&
41011(34)02*+34$!
""$ " 
 

0123457897
 3
 0


4 

! "#$#

1%34&02'(34)  
*
 *
 

0123457897
 3
 0


0 

!    "

#$  
#%&  &"
'# %


%
 (

#% 
 "
  )

0123457897
 3
 0


2

 !""#$%&'$%(%)*+,$%-'%./0012%3456,7,8%9"!%:"%;5,584%<"==>,7?5:7",'%@,%ABCDEFGHBIHJDFKLFMNOPQHR/$%SS'%/RTRU/R10'%
 !""#$%&'$%3'%<4665:"$%V'%&"W7,4,$%<'%X5Y5664::5%(%Z'%Z5887"%.[\\]2%^)4%;_;@X%<"#7,8%`?)4=4%a"6%:)4%,,":5:7",%"a%b44#c5?W$^>6,%
;5,584=4,:%5,#%`4d>4,?7,8'%@,%&'%<'%;56:7,%4:%5'%.4#*2%eCGMNLBfFGHgBDhBDFHIBDHeBfiGGNEKHjCLFEHeCGMNLBfFGHkilFmNBCD'%`S4?75%7**>4%
"a%:)4%@,:46,5:7",5%&">6,5%"a%35,8>584%n4*">6?4*%5,#%-Y5>5:7",'%`S67,846$%[\\]
o5:4*$%-'$%o4,78,7$%3'$%o64:)46:",$%@'$%<56,57",7$%3'%(%p":4665$%p'%./0]02'%qliHrLiDKiEOiHBIHstLuBGPvHgBKENMNBEHFEfHgBLLCENOFMNBEHNEH
wEIFEOt'%X4!%x"6Wy%?5#4=7?%Z64**'
-a6",$%z'%./0{/2'%|iPMCDiQH}FOiHFEfHgCGMCDi'%;">:",'
Z'%-W=5,%5,#%~'%b674*4,%./0]T2'%_,=5*W7,8%:)4%a5?4'%%8>7#4%:"%64?"8,77,8%4=":7",*%a6"=%a5?75%?>4*'%-,84!""#%<7aa*$%X4!%&46*4y%
Z64,:7?4U95$
V4,#",$%'%4*:>64%5,#%*S44?)y%:!"%5*S4?:*%"a%:)4%S6"?4**%"a%>::465,?4'%@,%;'n'%V4$%4#'$%BEmiDuFGHgBLLCENOFMNBEHFEfHFEKCFKi$%^)4%
958>4y%;">:",$%/01\$%SS'[\]U[[]'
V4,#",$%'%[\\{'%|iPMCDivHNPNuGiHOMNBEHFPHMMiDFEOi'%<5=c67#84y%<5=c67#84%_,7Y46*7:%Z64**'
V7SS$%;'%.[\\{2%|iPMCDiH|iEiDFMNBEHutHwLNMFMNBEHDBLHjCLFEHkilFmNBDMBHgBLhCMiDHglFDFOMiDHENLFMNBE'%o"?5%n5:",$%b"67#5$%
z7**46:5:7",'?"='
V7:5$%`'%.-#'2'%.[\\R2'%JBNEMNEKHliDiHGFEKCFKiQHOCGMCDiQHFEfHOBKENMNBEHLiiM'%97*#54%X&y%-6c5>='
;?X47$%z'%.[\\T2'%|iPMCDiHFEfHqlBCKlM'%<)7?58"y%_,7Y46*7:%"a%<)7?58"%Z64**'
;?X47$%z'%./00[2'%jFEfHFEfHLNEfvHlFMHKiPMCDiPHDimiFGHFuBCMHMlBCKlM'%<)7?58"y%_,7Y46*7:%"a%<)7?58"%Z64**'
;?X47$%z5Y7#%.?:"c46%/0]02'%qliHgBEOihMCFGHkFPNPHBIHFEKCFKi'%X4!%&46*4$%_`y%-6c5>=%%&"),%~74%(%`",*%@,?'
Z476?4$%<'%`'%./0R/UT12'%qliHgBGGiOMifHJFhiDPHBIHgHsHJiNDOi$%Y"*'%/U$%4#'%<)564*%956:*)"6,4%5,#%Z5>%~47**%Y"*'%]U1$%4#'%'%~'%
o>6W*$%<5=c67#84y%956Y56#'
Z"887$%@*5c45%.[\2%^)4%347?",%5,#%:)4%S)5c4:%"a%4*:>64$%54$%5,#%^">?)'%@,%wEMiGGNKiEMHNDMCFGHKiEMP34?:>64%X":4*%7,%<"=S>:46%
`?74,?4$%[\\/$%p">=4%[/0\[\\/$%[RTU[R'
n7=4$%o'$%5,#%`?)75:>65$%3'%./00/2'%4*:>64%5,#%*S44?)'%@,%n'`'%b4#=5,%5,#%o'%n7=4%.-#*'2$%CEfFLiEMFGPHBIHEBEmiDuFGHuilFmNBCD'%
956Y56#y%<5=c67#84%_,7Y46*7:%Z64**'
`?)4646$%V'%n'%%.[\\T2'%~)5:%564%4=":7",*%5,#%)"!%?5,%:)4%c4%=45*>64#%sBONFGHsONiEOiHwEIBDLFMNBE$%Y"'%{{$%,"'%{$%SS'%0T][0$%
z4?4=c46'
~>,#:$%~'%./0\T2%%|DCEfKifiDHhltPNBGBKNPOliEJPtOlBGBKNi'%347S78y%-,84=5,,'
0123457897
 3
 0


UNIVERSITY OF MALTA

Part 2

Annotation models and

procedures

Coling 2012, Mumbai

UNIVERSITY OF MALTA

Outline

Coling 2012, Mumbai

Dias 2

UNIVERSITY OF MALTA

Gesture annotation: what is it?

Coling 2012, Mumbai

Dias 3

UNIVERSITY OF MALTA

Annotation tools

Coling 2012, Mumbai

Dias 4

UNIVERSITY OF MALTA

An example: ANVIL

Coling 2012, Mumbai

Dias 5

UNIVERSITY OF MALTA

ANVIL file types

Contains the
speech stream of
the speaker(s)

Attributes and
values to be used in
the annotation

Coling 2012, Mumbai

Dias 6

UNIVERSITY OF MALTA

ANVIL the spec file

Coling 2012, Mumbai

Dias 7

UNIVERSITY OF MALTA

ANVIL annotation process

Coling 2012, Mumbai

Dias 8

UNIVERSITY OF MALTA

ANVIL - windows

From Kipps slides on www.anvil-software.de

Coling 2012, Mumbai

Dias 9

UNIVERSITY OF MALTA

ANVIL - elements

From Kipps slides on www.anvil-software.de

Coling 2012, Mumbai

Dias 10

UNIVERSITY OF MALTA

ANVIL more on elements

From Kipps slides on www.anvil-software.de

Coling 2012, Mumbai

Dias 11

UNIVERSITY OF MALTA

ANVIL - tracks

From Kipps slides on www.anvil-software.de

Coling 2012, Mumbai

Dias 12

UNIVERSITY OF MALTA

ANVIL - groups

From Kipps slides on www.anvil-software.de

Coling 2012, Mumbai

Dias 13

UNIVERSITY OF MALTA

ANVIL annotation example

UNIVERSITY OF MALTA

ANVIL annotation example

Coling 2012, Mumbai

Dias 15

UNIVERSITY OF MALTA

Identification of gestures

Coling 2012, Mumbai

Dias 16

UNIVERSITY OF MALTA

Identification, basic...

Coling 2012, Mumbai

Dias 17

UNIVERSITY OF MALTA

Identification, basic...

Coling 2012, Mumbai

Dias 18

UNIVERSITY OF MALTA

Segmentation

Coling 2012, Mumbai

Dias 19

UNIVERSITY OF MALTA

Segmentation

Coling 2012, Mumbai

Dias 20

UNIVERSITY OF MALTA

Segmentation

Coling 2012, Mumbai

Dias 21

UNIVERSITY OF MALTA

Segmentation

Coling 2012, Mumbai

Dias 22

UNIVERSITY OF MALTA

Segmentation, example

UNIVERSITY OF MALTA

Annotating gesture shape

Coling 2012, Mumbai

Dias 24

UNIVERSITY OF MALTA

Annotating shape... granularity

Coling 2012, Mumbai

Dias 25

UNIVERSITY OF MALTA

MacNeill (1992)
Coling 2012, Mumbai

Dias 26

UNIVERSITY OF MALTA

Annotating gesture... granularity

Coling 2012, Mumbai

Dias 27

UNIVERSITY OF MALTA

Annotating gesture... granularity

Coling 2012, Mumbai

Dias 28

UNIVERSITY OF MALTA

Annotating shape... granularity

Head movements in MUMIN

Nod

Movement down-up

Jerk

Sharp movement backwards-up

HeadBackward

Non-sharp movement backwards

HeadForward

Non-sharp movement downwards

Tilt

Movement leaning to one side

SideTurn

Rotation towards one side

Shake

Rotation from one side to the other

and back

Waggle

Mixture of shake and moving

backwards or forward

HeadOther

To be used if no other type applies

Coling 2012, Mumbai

Dias 29

UNIVERSITY OF MALTA

More gesture shape annotation

Coling 2012, Mumbai

Dias 30

UNIVERSITY OF MALTA

Gesture interpretation

Coling 2012, Mumbai

Dias 31

UNIVERSITY OF MALTA

Gesture interpretation
Gesture

Meta.cup

Description

Open, cup-shaped, palm up, held at

stomach height

Speech sample

was is das fr en Mann?

Form features

HS: open, Orient: PU, Loc:stomach

Meaning

Idea treated as a material

substance that can be presented in
a cupped hand.

Coling 2012, Mumbai

Dias 32

UNIVERSITY OF MALTA

Gesture interpretation

Attribute

Values

FeedbackBasic

CPU, Other

FeedbackDirection

Give, Elicit, GiveElicit

FeedbackAgreement

Agree, NonAgree

EmotionAttitude

Happy, Sad, Angry,

Interested, Bored,

Turn

Take, Yield, Accept, Elicit,

Hold, Complete

Sequencing

Open, Continue, Resume,

Close

Coling 2012, Mumbai

Dias 33

UNIVERSITY OF MALTA

Gesture interpretation

Coling 2012, Mumbai

Dias 34

UNIVERSITY OF MALTA

Gesture interpretation

Coling 2012, Mumbai

Dias 35

UNIVERSITY OF MALTA

Gestural interpretation

Coling 2012, Mumbai

Dias 36

UNIVERSITY OF MALTA

Gesture and speech

Gesture type

Corresponding speech

Emblem

Syntactic phrase, sentence

Deictic

Phrase denoting referent

Iconics/
metaphorics

Phrase whose referent

shares same semantic
properties as gesture

Beat

Stressed word

Coling 2012, Mumbai

Dias 37

UNIVERSITY OF MALTA

Gesture and speech

Coling 2012, Mumbai

Dias 38

UNIVERSITY OF MALTA

Reliability

Coling 2012, Mumbai

Dias 39

UNIVERSITY OF MALTA

Reliability: inter-annotator
agreement

Coling 2012, Mumbai

Dias 40

UNIVERSITY OF MALTA

Reliability: percentage
agreement

Coling 2012, Mumbai

Dias 41

UNIVERSITY OF MALTA

Reliability: percentage agreement

Coling 2012, Mumbai

Dias 42

UNIVERSITY OF MALTA

Reliability: percentage agreement

Coling 2012, Mumbai

Dias 43

UNIVERSITY OF MALTA

Reliability: the kappa score

P(A)-P(E)
K=
1 P(E)

Coling 2012, Mumbai

Dias 44

UNIVERSITY OF MALTA

Reliability: the kappa score

Coling 2012, Mumbai

Jerks

Nods

20

Jerks

Coder A

Coder B
Nods

10

15

Dias 45

UNIVERSITY OF MALTA

Reliability: the kappa score

1 P(E)

Coling 2012, Mumbai

Jerks

Nods

K=

Coder A

P(A)-P(E)

20

Jerks

Coder B
Nods

10

15

Dias 46

UNIVERSITY OF MALTA

Reliability: the kappa score

Coling 2012, Mumbai

Dias 47

UNIVERSITY OF MALTA

Reliability: the kappa score

Coder A

Coder B

Nod

Coling 2012, Mumbai

Jerk

Total

Nod

Jerk

Total

12

Dias 48

UNIVERSITY OF MALTA

Reliability: the kappa score

Coling 2012, Mumbai

Dias 49

UNIVERSITY OF MALTA

Conclusion

Coling 2012, Mumbai

Dias 50

UNIVERSITY OF MALTA

References (1)

Coling 2012, Mumbai

Dias 51

UNIVERSITY OF MALTA

References (2)

Coling 2012, Mumbai

Dias 52

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Part 3

Multimodal Corpora,
Analysis and Systems

Coling 2012, Mumbai

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Our goals

H
Coling 2012, Mumbai

H
Dias 2

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

This part

User
studies

Corpus

IPL

Machine
Learning

Coling 2012, Mumbai

Dias 3

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Beyond Corpus

Coling 2012, Mumbai

Dias 4

Evaluation

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Backchannels

Coling 2012, Mumbai

Dias 5

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

MultiLis corpus

Coling 2012, Mumbai

Dias 6

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

MultiLis corpus

Coling 2012, Mumbai

Dias 7

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Motivation: perception study

Coling 2012, Mumbai

Dias 8

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Random strategy

Coling 2012, Mumbai

Dias 9

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Copy strategy

Coling 2012, Mumbai

Dias 10

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Ward & Tsukahara strategy

Coling 2012, Mumbai

Dias 11

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Gaze strategy

Coling 2012, Mumbai

Dias 12

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Pitch & pause strategy

Coling 2012, Mumbai

Dias 13

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Pitch, pause & gaze strategy

Coling 2012, Mumbai

Dias 14

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Corpus

Coling 2012, Mumbai

Dias 15

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Stimuli

Coling 2012, Mumbai

Dias 16

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Procedure

Coling 2012, Mumbai

Dias 17

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Results strategy

100
90

Average score

80
70
60
50
40
30
20
10
0
Copy

Random

Ward &
Tsukahara

Gaze

Pitch & Pause

Overall: scores are low

Copy condition is highest (58%) but still low.

Coling 2012, Mumbai

Dias 18

Pitch, Pause &

Gaze

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Results timing

Copy

Random

W&T

Gaze

P&P

PP&G

51

47

54

25

25

49

Precision (%)

12.77

9.26

4.00

40.00

22.49

Recall (%)

11.76

9.80

1.96

19.60

21.57

F-score

0.122

0.095

0.026

0.263

0.220

Number

Coling 2012, Mumbai

Dias 19

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Perception experiment summary

Coling 2012, Mumbai

Dias 20

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Parasocial Consensus Sampling

Coling 2012, Mumbai

Dias 21

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Yucks

Coling 2012, Mumbai

Dias 22

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

ITERATIVE PERCEPTUAL LEARNING

Coling 2012, Mumbai

Dias 23

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Learning models from corpus data

Coling 2012, Mumbai

Dias 24

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Learning models from corpus data

Coling 2012, Mumbai

Dias 25

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Schematic setup

Coling 2012, Mumbai

Dias 26

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Schematic setup

Coling 2012, Mumbai

Dias 27

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Iterative perceptual learning: model learning

Coling 2012, Mumbai

Dias 28

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Iterative perceptual learning: experiment

Coling 2012, Mumbai

Dias 29

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Iterative perceptual learning: summary

Coling 2012, Mumbai

Dias 30

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

SWITCHING WIZARD OF OZ

Coling 2012, Mumbai

Dias 31

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Ingredients so far

Coling 2012, Mumbai

Dias 32

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Evaluation

Coling 2012, Mumbai

Dias 33

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Switching Wizard of Oz

Coling 2012, Mumbai

Dias 34

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Switching Wizard of Oz

Coling 2012, Mumbai

Dias 35

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Switching Wizard of Oz

Coling 2012, Mumbai

Dias 36

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Schematic setup

Coling 2012, Mumbai

Dias 37

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Experiment: results

Coling 2012, Mumbai

Dias 38

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Experiment: results

Coling 2012, Mumbai

Dias 39

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Switching Wizard of Oz: summary

Coling 2012, Mumbai

Dias 40

TUTORIAL ON MULTIMODAL CORPORA COLING 2012

Coling 2012, Mumbai

Dias 41