Learning

about Informa0on Searchers from Eye-Tracking

Jacek Gwizdka
Department of Library and Information Science School of Communication and Information Rutgers University

Monday, April 4, 2011

CONTACT:

www.jsg.tel

Outline
Overall research goals Eye-tracking fundamentals Eye-xa<on pa>erns: reading models (Exp 1; Exp 3) Search results presenta<on and cogni<ve abili<es (Exp 2) Summary and Challenges

Overall Research Goals

Characteriza*on and enhancement of human informa<on interac<on mediated by compu<ng technology Characteriza*on: cogni<ve and aec<ve user states tradi<onally: li>le access to the mental/emo<onal states of users while they are engaged in search process Implicit data collec<on about searchers cogni<ve and aec<ve states in rela<on to informa<on search phases Enhancement: Personaliza<on and Adapta<on

Example: Implicit Characteriza0on of Cogni0ve Load on Web Search

higher average cognitive load: Q & B

35% 27%

higher peak cognitive load: C

S T A R T

Q
formulate

97%

L view
search results list
7%

58%

view

content
page
42%

30%

query

bookmark page

E N D

95%

(Gwizdka, JASIST, 2010)

Eye-Tracking?
Early a>empts late XIX c.; early 1950s - a movie camera and hand-coding (Fi>s, Jones & Milton 1950) Now computerized and easy to use
infrared light sources and cameras sta<onary and mobile

Eye-tracking fundamental assump0ons

Top-down vs. bo>om-up control
in between: language processing (higher-level) controls when eyes move, visual processing (lower-level) controls where eyes move (Reichle et al., 1998)

Eye-mind link hypothesis: a>en<on is where eyes are focused (Just & Carpenter, 1980; 1987) Overt and covert a>en<on A>en<on can move with no eye movement BUT eyes cannot move without a>en<on

Data from Eye-tracking Devices

eye gaze points
eye gaze points in screen coordinates + distance eye xa<ons in screen coordinates + validity pupil diameter

[head posi<on 3D, distance from monitor] 50/60Hz; 300Hz; 1000-2000Hz eye-trackers common: 60Hz: one data record every 16.67ms

Eye-Tracking Can
Eye tracking can allow iden<ca<on of the specic content acquired by the person from Web pages Eye tracking enables high resolu<on analysis of searchers ac<vity during interac<ons with informa<on systems And more

Example: composing answer and from information on a Web page (video)

8

Related Work in Informa0on Science

Interac<on with search results
Interac<on with SERPs (Granka et al., 2004; Lorigo et al., 2007; 2008) Eects results presenta<on (Cutrell et al., 2007; Kammerer al., 2010) Relevance detec<on (Buscher, et al. 2009) Implicit Feedback (Fu, X., 2009); Query expansion (Buscher, et al. 2009)

Relevance detec<on
Pupillometry (Oliveira, Aula, Russell, 2009)

Detec<on of task dierences from eye-gaze pa>erns

Reading/reasoning/search/object manipula<on (Iqbal & Bailey, 2004) Informa<onal vs. transac<onal tasks (Terai , et al., 2008) Task detec*on is also one of our research interests
9

Experiment 1: Journalism tasks Open Web Search

32 journalism students 4 journalis<c tasks (realis<c, created by journalism faculty and journalists) Tasks: Task facets:
advanced obituary (OBI) interview prepara<on (INT) copy edi<ng (CPE) background informa<on (BIC)
product: factual vs. intellectual level: whole document vs. segment nature of task goal complexity number of steps needed

Note: OBI vs. CPE are most dissimilar

10

Experiment 1 Research Ques0ons

Can we detect task type (dierences in task facets) from implicit interac<on data (e.g., eye-tracking) ? How do we aggregate informa<on from eye-tracking data?

11

Eye-gaze paMerns
Eye-tracking research have frequently analyzed eye-gaze posi<on aggregates ('hot spots)
spa<otemporal-intensity heat maps also sequen<al scan paths

Higher-order pa>erns:
reading models

12

Reading Eye PaMerns

Reading and scanning have easily dis<nguished pa>erns of xa<ons and saccades. (Rayner & Fischer, 1996) Lexical Processing of Words
Reading research has established word availability is a func<on of xa<on dura<on: Orthographic recogni<on: 40-50 ms
<me to move data from eyes to mind

Phonological recogni<on: 55-70ms Lexical availability (typical): 113 ms 150ms (Rayner, 1998)
Unfamiliar or complex meanings require longer processing

Eyes do not saccade un<l the word has been processed

13

 Scan Fixa0ons vs. Reading Fixa0ons

Scanning xa<ons provide some seman<c informa<on, limited to foveal (1 visual acuity) visual eld (Rayner & Fischer, 1996) Fixa<ons in a reading sequence provide more informa<on than isolated scanning xa<ons:
informa<on is gained from the larger parafoveal (5 beyond foveal focus) region (Rayner et al., 2003) (asymmetrical, in dir of reading) richer seman<c structure available from text composi<ons (sentences, paragraphs, etc.)

Some of the types of seman<c informa<on available only through reading sequences may be crucial to sa<sfy task requirements.
14

Reading Models
We implemented the E-Z Reader reading model (Reichle et al., 2006)
Inputs: (eye xa<on loca<on, dura<on) Fixa<on dura<on >113 ms threshold for lexical processing (Reingold & Rayner, 2006) The algorithm dis<nguishes reading xa<on sequences from isolated xa<ons, called 'scanning' xa<ons Each lexical xa<on is classied to (S,R) (Scan, Reading) These sequences used to create a state model

15

Reading Model States and Characteris0cs

Two states: transi<on probabili<es Number of lexical xa<ons and dura<on

16

Example Reading Sequence

17

Results: Search Task Eect on Reading/Scanning

Task eects on transi<on probabili<es SR & RS (all subjects & pages)
For OBI, INT searchers biased to continue reading For CPE to continue scanning Searchers are adopting different reading strategies for different task types

(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, 2010)

18

Results: Search Task Facets and Text Acquisi0on

For highly a>ended pages

Total Text Acquired on SERPs and Content

Total Text Acquisition on SERPs and Content per page

19

Results: Search Task Facets and State Transi0ons

For highly a>ended pages
Read Scan Scan Read Read Scan Scan Read

State Transitions on SERPs per page

State Transitions on Content pages per page

20

Task Facets Eects - Summary

For highly a>ended pages

(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, submitted, 2011)

21

Scan<->Read Transi0on Probabili0es in 2 Experiments

Persons tendency to readscan related to scanread? (i.e., is p related to q ?) p ~ 1-q
Journalistic tasks (N=32) Genomics tasks (N=40)

correlation (Spearman ): 0.914 and 0.830

Experiment 1: Conclusions
Searchers reading / scanning behavior aected by task Tasks facets can be detected from eye-tracking data (from reading model proper<es) Reading models can be built on the y (during search) real- <me observa<ons of eye movements can be used by adap<ve search systems Challenge: Lack of baseline data about reading models of individuals

23

Experiment 2: Result List vs. Overview Tag-Cloud

37 par<cipants Everyday informa<on seeking tasks (travel, shopping) - two levels of task complexity Two user interfaces
1. List UI 2. Overview UI (Tag Cloud)

24

Experiment 2: User Ac0ons in Two Interfaces

1. List

2. Overview Tag Cloud

25

Experiment 2: Research Ques0ons

Does the search results overview benet users? Task eects? Individual dierences - cogni<ve ability eects?

26

General Results
Search results overview (tag cloud) beneted users
made them faster facilitated formula<on of more eec<ve queries

More complex tasks were indeed more demanding required more search eort

(Gwizdka, Information Research, 2009)

27

Task and UI and Reading Model dierences

Complex tasks required more reading eort
Longer max reading xa<on length and more reading xa<on regressions

Overview UI required less eort

Scanning more likely (S-S higher; S-R lower; R-S higher) Reading scan path length total lower but total scan paths were longer (including scanning) Less and shorter mean xa<ons per page visited
List Overview

28

Task and UI Interac0on and Reading model data

For complex tasks UI eect
Higher probability of short reading sequences in Overview UI

For simple tasks UI eect

Shorter length of reading scan paths per page and less xa<ons per page

Task & UI interac<on

Speed of reading:
for complex tasks faster reading in Overview than in List UI for simple tasks faster in List than in Overview UI

29

User Interface Features Individual Dierences

Two users, same UI and task

30

Individual Dierences Least Eort?

Higher cogni<ve ability searchers were faster in Overview UI and on simple tasks (same number of queries) Higher ability searchers did more in more demanding situa<ons
higher search eort did not seem to improve task outcomes
For tasks and working memory (WM)

F(144,1)=4.2; p=.042

F(144,1)=3.1; p=.08

31

Task and Working Memory Eye-tracking Data

High WM less likely to stay in Scanning state High WM higher reading speed (scan path/total xa<on dura<on)

Number and dura<on of reading sequences diers

(borderline: 0.05<p<0.1)

For high WM searchers:

for simple tasks less reading for complex more reading

For low WM no such dierence!

32

Experiment 2: Conclusions
Overview UI was faster reected in some eye-tracking measures Task complexity dierences reected in some eye-tracking measures Some eects of cogni<ve abili<es on interac<on
viola<on of the least eort principle not fully explained yet

33

34

Current Project: Can We Implicitly Detect Relevance Decisions?

Implicit characteriza<on of Informa<on Search Process using physiological devices Can we detect when searchers make informa<on relevance decisions? Start with pupillometry Also look at EEG, GSR
Funded by Google Research Award
GSR
pupil animation

Eye tracking

EEG

info relevance (Oliveria, Russell, Aula, 2009) low-level decision <ming (Einhuser, et al. 2010)

Summary & Conclusions

Eye tracking enables high resolu<on analysis of searchers ac<vity during interac<ons with informa<on systems Eye tracking can support iden<ca<on of search task types Eye tracking reects dierences in searcher performance on user interfaces Eye tracking reects individual dierences between searchers High poten<al for implicit detec<on of searcher states
36

Some Challenges
High-resolu<on data (low-level) How do we create higher-level pa>erns? How do we detect them computa<onally? How do we deal with ind. dis (baseline data)?

(Iqbal & Bailey, 2004)

(Terai et al., 2008)

(Lorigo et al., 2008)

37

High-resolu0on Eye-tracking is Coming Soon to You

Eye tracking technology is declining in price and in 2-3 years could be part of standard displays.
Already in luxury cars and semi-trucks (sleep detec<on) Computers with built in eye-tracking

Tobii / Lenovo proof of concept laptop March 2011

38

Thank you!

Ques<ons?

Jacek Gwizdka contact: h>p://jsg.tel

PoODLE Project: Personaliza<on of the Digital Library Experience

IMLS grant LG-06-07-0105-07

h>p://comminfo.rutgers.edu/research/poodle or for short: h>p://bit.ly/poodle_project PoODLE PIs: Nicholas J. Belkin, Jacek Gwizdka, Xiangmin Zhang Post-Doc: Ralf Bierig, PhD Students: Jingjing Liu, (now Asst Prof.), Chang Liu, Michael Cole

Other Eye-tracking Derived Measures

Speed of text processing
reading speed; scan path length / total xa<on dura<on

Perceptual span
number of characters processed at a <me English etc. ~ 19 characters (Rayner, 1998) decreases in normal reading when user encounters cogni<vely- demanding content

40