1

Modelling and Developing Mobile Applications

Paul Holleis September 13, 2009

My Background
From 2004, Comp. Science, Univ. Passau
9 months University of Edinburgh

Graph Drawing End User Programming Tangible Computing

Till Mar. 07, LMU Munich

DFG Group Embedded Interaction Media Informatics

Till Oct. 07, University of Bonn

b-IT Bonn-Aachen Centre of Technology Usability Research Group, Nokia Research Center, Helsinki

Wearable Computing
Prototyping and Modelling Pervasive Applications Mobile Computing

Till Sep. 08, University of Duisburg-Essen

Pervasive Computing and User Interface Engineering

From Oct. 08, DOCOMO Euro Labs, Munich

Smart and Secure Services Research Group

Structure
Introduction and motivation
User modelling for pervasive applications
Introduction, Keystroke-Level-Model Modelling Advanced Mobile Interactions

Development support for pervasive applications

Requirements and designs Architecture and implementation of the EIToolkit

Combination of toolkit, user models, and IDE

Design and architecture Programming by demonstration and state-graphs

Summary and Conclusion

Example Model 1: Three-State Model

Describes graphical input Simple, quick, expressive
Possible extensions:
multi-button interaction stylus input direct vs. indirect input

Buxton, W, 1990, A Three-State Model of Graphical Input In INTERACT'90, 449-456

Dragging tasks: (a) mouse (b) lift-and-tap touchpad. [MacKenzie 2003]

4

Example 2: Guiards Model of Bimanual Skill

Many tasks are asymmetric with regard to left / right hand Guiards model identifies the roles and actions of the non-preferred and preferred hands
Preferred hand follows the nonpreferred hand works within established frame of reference set by the non-preferred hand performs fine movements

Non-preferred hand leads the preferred hand sets the spatial frame of reference for the preferred hand performs coarse movements

Example 2: Guiards Model of Bimanual Skill

Microsoft Office Keyboard 6

The GOMS Model

G: goals
(Verbal) description of what a user wants to accomplish Various levels of complexity possible
Possible actions in the system Various levels of abstraction possible (sub-goals / ... / keystrokes) Sequences of operators that achieve a goal

O: operators M: methods

S: selection rules

Rules that define when a user employs which method

User tasks are split into goals which are achieved by solving sub-goals in a divide-and-conquer fashion
Card, S. K.; Newell, A.; Moran, T. P., 1983, The Psychology of Human-Computer Interaction (Book) 7

GOMS Example: ATM Machine

GOMS gives an early understanding of interactions How to not loose you card
GOAL: GET-MONEY . GOAL: USE-CASH-MACHINE . . . . INSERT-CARD ENTER-PIN SELECT-GET-CASH ENTER-AMOUNT GOAL: GET-MONEY . GOAL: USE-CASH-MACHINE . . . . INSERT-CARD ENTER-PIN SELECT-GET-CASH ENTER-AMOUNT

COLLECT-MONEY

.
.

COLLECT-CARD
COLLECT-MONEY

(outer goal satisfied!) . COLLECT-CARD

(outer goal satisfied!)

Models: Levels of Detail

Different levels of detail for the steps of a task performed by a user
Abstract: correct Concrete:
wrong spelling

mark-word delete-word type-word hold-shift ncursor-right recall-word del-key nletter-key

Keystroke-Level:

10

Keystroke-Level Model
Simplified version of GOMS
only operators on keystroke-level no sub-goals no methods

no selection rules

KLM predicts how much time it takes to execute a task Execution of a task is decomposed into primitive operators
Physical motor operators pressing a button, pointing, drawing a line, Mental operator preparing for a physical action System response operator user waits for the system to do something

10

11

Task
Switch your mobile phone to silent mode
Get your phone
Remove key lock Enter PIN Wait for confirmation Enter silent mode 3.89 sec. 0.99 sec.

1.72 sec.
2.70 sec. 0.99 sec.

Switch on key lock

0.99 sec.
-----11.28 seconds

Basics to Build On
Cognitive modelling (Anderson, J. R. Rules of the Mind. Lawrence Erlbaum Associates, 1993) GOMS (Card, S. K., Newell, A., and Moran, T. P. The Psychology of HumanComputer Interaction. Lawrence Erbaum Associates, 1983) Keystroke-Level Model (Card, S. K., Moran, T. P., and Newell, A. "The Keystroke-Level Model for User Performance Time with Interactive Systems." Communications of the ACM 23, no. 7. 1980. 396-410.)
KLM Operators: Mental Act, M System Response Time, R Keypress, K Homing, H Pointing, P Drawing, D

12

13

Task
Switch mobile phone to silent mode
Get your phone
Remove key lock Enter PIN Initial Preparation Hotkey + System Response 4*Key, Hotkey System Response I K, R 4*K, K R K, R K, R 3.89 0.99 1.72 2.70

Wait for confirmation

Enter silent mode Enter key lock

Hotkey + System Response

Hotkey + System Response

0.99
0.99

14

Calculate Interaction Time

Texecute nop op
op OP

OP
nop

set of available operators #occurrences of the operator op

I + 3*K[key] + 3*R + 4*K[hotkey] + K[hotkey] + R = 11.28 sec.

KLM In Use at CHI et al.

Systems and Applications

16

Gong, R., Elkerton, J., Designing Minimal Documentation Using a GOMS Model: a Usability Evaluation of an Engineering Approach. In Proc. CHI 1990. 99-107. 1990 John, B. E., Extensions of GOMS Analyses to Expert Performance Requiring Perception of Dynamic Visual and Auditory Information. In Proc. CHI 1990. 107-115. 1990 John, B. E., Vera, A. H., A GOMS Analysis of a Graphic Machine-paced, Highly Interactive Task. In Proc. CHI 1992. 251-258. 1992

Gong, R., Kieras, D., A Validation of the GOMS Model Methodology in the Development of a Specialized, Commercial Software Application. In Proc. CHI 1994. 351-357, 1994
Haunold, P., Kuhn W., A Keystroke Level Analysis of a Graphics Application: Manual Map Digitizing. In Proc. CHI 1994. 337-343. 1994 Blter, O., Keystroke Level Analysis of Email Message Organization. In Proc. CHI 2000

Manes, D., Green, P., and Hunter, D., Prediction of Destination Entry and Retrieval Times Using KeystrokeLevel Models (Technical Report UMTRI-96-37). The University of Michigan Transportation Research Institute
Hinckley, K., Guimbretiere, F., Baudisch, P., Sarin, R., Agrawala, M., and Cutrell, E., The Springboard: Multiple Modes in one Spring-loaded Control. In Proc. CHI 2006. 181-190. 2006

...

KLM In Use at CHI et al.

Mobile Phone Text Input
Dunlop, M.D., Crossan, A., Predictive Text Entry Methods for Mobile Phones. Personal Technologies, 4(2-3), 2000 Silfverberg M., MacKenzie, I. S., Korhonen, P. Predicting Text Entry Speed on Mobile Phones. In Proc. CHI 2000, 9-16. 2000 James, C.L., Reischel, K.M., Text Input for Mobile Devices: Comparing Model Prediction to Actual Performance. In Proc. CHI 2001, 365-371

17

Myung R., Keystroke-level Analysis of Korean Text Entry Methods on Mobile Phones. International Journal of Human-Computer Studies 60, 5-6, HCI Issues in Mobile Computing. 545-563. 2004
Pavlovych, A., Stuerzlinger, W., Model for Non-expert Text Entry Speed on 12-button Phone Keypads. In Proc. CHI 2004. ACM Press. 351-358. 2004 How Y., Kan M.Y., Optimizing Predictive Text Entry for Short Message Service on Mobile Phones. In Proc. HCII 2005. 2005 ...

KLM In Use at CHI et al.

Mobile Phone Interactions
Mori, R., Matsunobe, T., Yamaoka, T., A Task Operation Prediction Time Computation based on GOMS-KLM Improved for the Cellular Phone and the Verification of that Validity. 2003.10, Journal of the Asian Design International Conference. Vol.1. 2003 Luo, L., John, B. E., Predicting Task Execution Time on Handheld Devices Using the Keystroke-level Model. In Extended Abstracts CHI 2005. 1605-1608. 2005

18

Teo, L., John, B. E., Comparisons of Keystroke-level Model Predictions to Observed Data. In Extended Abstracts CHI 2006. ACM Press. 1421-1426, 2006 Holleis, P., Otto, F., Schmidt, A., Hussmann, H., Keystroke-Level Model for Advanced Mobile Phone Applications. In Proc. CHI 2007

19

Keystroke-Level Model Example Task

Task: in MS Word, add a 6pt space after the current paragraph
Word 2003:
Actions Locate menu Format Press ALT-o or mouse click Locate entry Paragraph Press p or mouse click Locate item in dialogue Point to item Enter a 6 for a 6pt space Close the dialogue (ENTER) Operator (keyboard) Estimated Time (in sec.) M 1.35 K,K 2*0.28 M 1.35 K 0.28 M 1.35 K,K 2*0.28 K 0.28 K 0.28 Sum (keyboard): 5.73 sec. Operator (mouse) Estimated Time (in sec.) M 1.35 P,B 1.10+0.10 M 1.35 P,B 1.10+0.10 M 1.35 P,B 1.10+0.10 K 0.28 K 0.28 Sum (mouse): 8.21 sec.

Word 2007:

Sum (keyboard): 7.22 sec.

Sum (mouse): 7.65 sec.

Operators: K (key press), P (mouse pointing), B (mouse button press), M (mental act)

19

20

KLM Application
Model and basic values validated in the last 25 years Valuable especially for experts / with many repetitions Available operators:
Mental Act, M System Response Time, R Keypress, K Homing, H Pointing, P Drawing, D

However: Only for desktop / mouse / keyboard setting

Extension for mobile interactions
20

21

Models for Mobile Applications Why?!?

Task completion time interesting for
Time critical applications
emergency Medical Interaction chains between tasks

User friendly applications

concentrate on important things when to introduce shortcuts

General advantages
Can helps in deciding between alternative design Can identify bottlenecks Can provides illustrative figures

Interactions with Mobile Phones

Input capabilities: Keypad, hotkeys (numbers, text, control) Microphone Camera Sensors (like accelerometers) Bluetooth etc. Readers (tags)

22

Point somewhere

Touch a tag Take a picture Scan a visual tag

Interactions with Mobile Phones

Input capabilities: Keypad, hotkeys (numbers, text, control) Microphone Camera Sensors (like accelerometers) Used as: Bluetooth etc. Remote control, data storage, personal key, organiser, payment, Readers (tags)

23

Enabled additional interactions: Speech Gestures Location sensitive Marker detection (RFID, NFC, barcodes, 2D visual codes) Pointing

browser physical mobile interactions (Rukzio et al. [32])

Peculiarities: Size More likely to be distracted Need to switch attention between phone and real world

24

Extensions for Novel Mobile Interactions

Current mobile interactions use
Keypad, hotkeys Microphone Sensors like accelerometers Visual markers, tag readers (NFC) Bluetooth

Method for Extending the Model:

Large set of studies Software on the phone Video frame-by-frame analysis Eye-tracker Total number of actions measured: 2134
24

25

Extended KLM Operators

Mental Act, M System Response, R Keystroke / button press, K Homing, H Pointing, P Micro attentions Shift, SMicro attentions Shift, SMicro Macro attention shift, SMacro attention shift, SMacro Finger movement F Distraction X Distraction X Gesture G Initial preparation I
unchanged

adapted operator value

newly added

25

26

Micro Attention Shift, SMicro

Switch attention between phone parts
hot keys

keypad

display

27

SMicro Operator Time Estimation

Task 1: write a text message

Task 2: change the ringtone

Task 3: set alarm clock

28

SMicro Operator Time Estimation

Study
10 participants, 24-34 years, 6 female 1500 shifts detected Using automatic eye-tracking 3 pre-set tasks
400
350

Micro Attention Shift: SMicro = 0.14 sec.

Micro Attention Shift

300 250 200

150

100 50 0 1 59 117 175 233 291 349 407 465 523 581 639 697 755 813 871 929 9871045 1103 1161 1219 1277 1335 1393 1451 1509

29

Macro Attention Shift, SMacro

Switch attention between phone and real world

30

SMacro Operator Time Estimation

Study
9 participants, 22-46 years, 2 female 121 detected shifts Frame-by-frame analysis of video footage No significant difference in the two directions (t=0.57, p>0.36)
0,90 0,80 0,70 0,60 0,50 0,40 0,30 0,20 0,10 0,00
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 Macro Attention Shift

Macro Shift: 0.36 sec. (quartiles: 0.24, 0.44)

31

Distraction, X
Study
10 participants, 24-33 years, 3 female

Distraction: multiplicative Xslight = 6%, Xstrong = 21%

Short message in 3 settings (quiet room, standing outside, walking)

Relative slow-down (significant: t=2.23, p<0.03 and t=3.28, p<0.01)
Time to type one character (in sec.)
1,8 1,6 1,4 1,2 1 0,8 0,6 0,4 0,2 0
1 2 3 4 5 Person 6 7 8 9 10

silent room stand on street walk on street

New KLM Operators: G

Gesture, G = 0.80 sec.
10 participants, 23-33 years, 5 female Movements with the phone treated as input

32

Samsung SGH-E760, built-in accelerometer, standard applications, games

Simple gestures only

New KLM Operators: F

Finger Movement, F = 0.23 sec. (hotkey region only: F = 0.16)
Analysed 323 key presses Used S[Micro] to calculate Holds for semi- and fully experienced people Silfverberg, MacKenzie, Korhonen: 0.27, Mori et al.: 0.19

33

New KLM Operators: I

Initial Preparation,
externally initiated:

34

I = 4.61
I = 5.32

internally initiated:
optimal setting:

I = 3.89
I = 1.18

10 participants, 25-54 years, 4 female Large differences due to different storage of phones

Depends on whether user starts or is, e.g., called

35

Extended KLM Time Prediction

Set of Available Operators:

OP

{A, F, G, H, I, K, M, P, R, SMicro, SMacro}

Total Execution Time:

Texecute

(nop
op OP

dop X slight

Dop X strong ) op

nop: #occurrences of the operator op dop: #occurrences of the operator op Dop: #occurrences of the operator op

OP without any distraction OP under slight distraction (modelled by Xslight) OP under strong distraction (modelled by Xstrong)

36

Advanced Mobile Phone KLM Values

some are to be revised every now and then ...

36

37

KLM Existing Extensions

Example: text input on small devices
Text Entry Method T9 T9 T9 T9 T9 Multitap Multitap Multitap Multitap Multitap Multitap Multitap Multitap index finger (thumb) expert (novice) novice WPM 25.0 45.7 (40.6) 17.6 4.80 (3.80) 7.58 18.4 14.9 22.5 (20.8) 14.9 8.20 (5.00) 7.15 5.87 30.4 (25.7) Reference [Dunlop and Crossan 1999] [Silfverberg, MacKenzie, and Korhonen 2000] [James and Reischel 2001] [Cockburn and Siresena 2003] [Pavlovych and Strzlinger 2004] [Dunlop and Crossan 1999] [Dunlop and Crossan 2000] [Silfverberg, MacKenzie, and Korhonen 2000] [James and Reischel 2001] [Cockburn and Siresena 2003] [Pavlovych and Strzlinger 2003] [Pavlovych and Strzlinger 2004] [Holleis, Otto, et al. 2007]

index finger (thumb) expert (novice) novice expert (novice)

38

Extended KLM Empirical Validation

Task: buy a public transportation ticket from A to B Implemented 2 ways of performing the task

Access through mobile web browser

Direct interaction with NFC tags Comparison:
Created the two Keystroke-Level Models Study: 9 people, 23-34 years, 3 female

39

Extended KLM Empirical Validation

Browser Interaction
250

NFC Interaction KLM Study

KLM Time in seconds

200 150 100 50 0

Study

123

117

147

137

Predicted speed loss: 20%

Actual speed loss: 17%

10

40

Experiscope Visualizing KLM

Analytical tool for designers to explore the results of quantitative evaluations Combines
Three-state Model KLM

Guimbretire et al. ExperiScope: An Analysis Tool for Interaction Data, CHI07

41

Models

John, B. E. (2003) Information processing and skilled behavior. In J. M. Carroll, (Ed.), Toward a multidisciplinary science of human computer interaction. Morgan Kaufman

42

CPM-GOMS
CriticalPathMethod

John, B. E. (2003) Information processing and skilled behavior. In J. M. Carroll, (Ed.), Toward a multidisciplinary science of human computer interaction. Morgan Kaufman

43

Example KLM Extension

KLEM: A Method for Predicting User Interaction Time and System Energy Consumption during Application Design, Luo, A., Siewiorek, D., ISWC07

Mapping between interaction and energy consumption

Busy(K, D, R): MOTOR, WAIT-FOR-SYSTEM Idle(M): VISION, PROCEDUAL

44

Weaknesses of GOMS et al.

Just spending time is not modelled Difficult to target specific users

No real users
Difficult to model novel interactions

Various variable parameters

Users like to have impact

44

45

Strengths of GOMS et al.

Good treatment of learning effects
Measurement of learnability Independence of sequences Measurement of knowledge requirements

Less cost in money and time

Quick to apply

Quick to prepare
Helpful to design Cheap to apply Easy to repeat Quick to analyse Precise to interpret Easy to convey

Good results
Gives reasons Helps in decision making Identifies bottlenecks Provides illustrative figures Combines various views Treats feasibility and cognitive load

45

46

GOMS / KLM Summary Example

Example prototype: the Combimouse

Ergonomic models followed

Follows Guiards model of bimanual control (for right handed people scrolling with the non-preferred hand) Removes KLMs Homing operator (H ~ 1 sec.)
http://www.combimouse.com

46

47

Structure
Introduction and motivation
User modelling for pervasive applications
Introduction, Keystroke-Level-Model Modelling Advanced Mobile Interactions

Development support for pervasive applications

Requirements and designs Architecture and implementation of the EIToolkit

Combination of toolkit, user models, and IDE

Design and architecture Programming by demonstration and state-graphs

Summary and Conclusion

47

EIToolkit Development Support for Pervasive Applications

General infrastructure for combining various applications Facilitates observation, debugging, simulation, exchange Abstracts from implementation details Support for several open standards Employed in several projects
Nabaztag

48

OSC
Skype Status

Serial Ports
MIDI UDP Winamp Keystrokes KLM

IYOUIT TCP Joysticks RTP Parts

www.eitoolkit.de

Power Socket

Particles

13

49

EIToolkit Idea and Architecture

... ...

MouseStub

...

ParticleStub

...

DisplayStub

Common Communication Area

Storage

Application 1

...

Application n

Remote Communication Area

14

50

EIToolkit Applied Architecture

WiimoteStub
button0:on button0:off

Communication Area

socket0:on socket0:off

PowersocketStub WinampStub

Application if (msg == button0:on) { if (msg == button0:on) send(socket0:on); send(socket0:on); send(play); }

15

Dev. Platform (e.g. EIToolkit) Important Features

Programming abstractions

51

Easy replacement of components

Simulation and replay

SerialStub

Comm unicatio n Area

PowersocketStub

Application if (msg == button0:on) send(socket0:on);

Open and replaceable protocols

Layered context processing

Platform, OS, and language independence

16

52

Whats Missing?
Have formal metrics to evaluate usability + Performance predictions using Keystroke-Level Model + Have KLM extensions for advanced mobile interactions Difficult and time consuming to know, create, and apply Only employed for academic reasons, post-hoc, no influence in development process Have EIToolkit + Easy creation of pervasive applications + Abstractions and simple component access + Simulation, logging, additional tools on top No integration in a whole development experience Need one integrated environment Development, modelling, iteration
17

53

Structure
Introduction and motivation
User modelling for pervasive applications
Introduction, Keystroke-Level-Model Modelling Advanced Mobile Interactions

Development support for pervasive applications

Requirements and designs Architecture and implementation of the EIToolkit

Combination of toolkit, user models, and IDE

Design and architecture Programming by demonstration and state-graphs

Summary and Conclusion

53

54

Graphical Development Tools

55

Graphical Development Tools

Simplifying Development: Eclipse

56

Integration into the Eclipse IDE Automatic class generation from EIToolkit interface descriptions Automatic code completion from Eclipse Automatic connection to EIToolkit functionality

57

Simplifying Development: d.tools

Integration into the d.tools IDE Using the original visual designer Using the original transition descriptions Augmented the graph capabilities Background connection to EIToolkit KLM component Added a view for KLM model output
Hartmann, B., Klemmer, S. R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., and Gee, J. 2006. Reflective physical prototyping through integrated design, test, and analysis. In UIST06. 299-308.

58

KLM Tool Support

This is not enough!

59

Integrated Development Environments

Prototyping for users: how to help people design/prototype/build applications taking people into account
MAKEIT

d.tools+

Eclipse+

60

MAKEIT Main User Interface

Screen content: Text, HTML, Web page, Drawing, Image

Underlying state graph

61

Automatically Adding KLM Operators

Keys: Marker: NFC:

7.0 sec. 6.0 sec. 5.1 sec.

Sum of times: 6.0

1C

62

Usability Analysis with State Graphs

Ensure no dead-ends and no unreachable states:
G is strongly connected

Cheapest way to achieve something:

Find the shortest path(s)

The expected average cost of a task:

1 p : p shortest Path p
p shortest Path

Cost to undo an action: p(sb, sa) for an action (sa, sb)

See also Thimbleby, H. and Gow, J. 2008. Applying Graph Theory to Interaction Design. In EIS 2007. LNCS 4940. Springer. 501-519

63

CogTool

Luo, L. and John, B. E. 2005. Predicting task execution time on handheld devices using the keystroke-level model. CHI05. 1605-1608

64

CogTool and Mobile Devices

Luo, L. and John, B. E. 2005. Predicting task execution time on handheld devices using the keystroke-level model. CHI05. 1605-1608

Quickly Create Applications Keeping the User in Mind

Programming by Demonstration
Easy to do, quick to change, easy to understand Good for demonstration and testing

65

Generate User Models on the Fly

Automatically for known actions Still adaptable by experts

Exploit Model Knowledge

Get an idea about task completion times Compare designs and interactions

Summary

Execute the application

Simulate it in the IDE Generate an executable phone application
1D

66

Modelling and Developing Mobile Applications

Paul Holleis September 13, 2009