Interaction Devices

Human Computer Interaction

CIS 6930/4930
Section 4188/4186
 Interaction Performance
► 60s vs. Today
 Performance
► Hz -> GHz
 Memory
►k -> GB
 Storage
►k -> TB
 Input
► punch cards ->
► Keyboards, Pens, tablets, mobile
phones, mice, digital cameras, web
cams
 Output
► 10 character/sec
► Megapixel displays, color laser,
surround sound, force feedback, VR
► Substantial bandwidth increase!
 Interaction Performance
► Future?
 Gestural input
 Two-handed input
 3D I/O
 Others: voice, wearable, whole
body, eye trackers, data gloves,
haptics, force feedback
 Engineering research!
 Entire companies created
around one single technology
► Current trend:
 Multimodal (using car
navigation via buttons or voice)
 Helps disabled (esp. those w/
different levels of disability)
 Keyboard and Keypads
► QWERTY keyboards been
around for a long time
 (1870s – Christopher Sholes)
 Cons: Not easy to learn
 Pros: Familiarity
 Stats:
► Beginners: 1 keystroke per
sec
► Average office worker: 5
keystrokes (50 wpm)
► Experts: 15 keystrokes per
sec (150 wpm)
► Is it possible to do better?
Suggestions?
 Keyboard and Keypads
► Look at the piano for possible
inspiration
► Court reporter keyboards (one
keypress = multiple letters or a
word)
 300 wpm, requires extensive
training and use
► Keyboard properties that matter
 Size
► large - imposing for novices,
appears more complex
► mobile devices
 Adjustable
► Reduces RSI, better
performance and comfort
 Mobile phone keyboards,
blackberry devices, etc.
► QWERTY
Keyboard Layouts
 Frequently used pairs far apart
 Fewer typewriter jams
 Electronic approaches don’t jam.. why use
it?
► DVOARK (1920s)
 150 wpm->200 wpm
 Reducing errors
 Takes about one week to switch
 Stops most from trying
► ABCDE – style
 Easier for non-typists
 Studies show no improvement vs. QWERTY
► Number pads
 What’s in the top row?
 Look at phones (slight faster), then look at
calculators, keypads
► Those for disabled
 Split keyboards
 KeyBowl’s orbiTouch (screenshot)
 Eyetrackers, mice
 Dasher - 2d motion with word prediction
 Keys
► Current keyboards have been
extensively tested
 Size
 Shape
 Required force
 Spacing
► Speed vs. error rates for
majority of users
► Distinctive click gives audio
feedback
 Why membrane keyboards are
slow (Atari 400?)
► Environment hazards might
necessitate
► Usually speed is not a factor
 Keys Guidelines
► Special keys should be denoted
► State keys (such as caps, etc.)
should have easily noted states
► Special curves or dots for home
keys for touch typists
► Inverted T Cursor movement
keys are important (though
cross is easier for novices)
► Auto-repeat feature
 Improves performance, but only
if repeat is customizable (motor
impaired, young, old)
► Two thinking points:
 Why are home keys fastest to
type?
 Why are certain keys larger?
(Enter, Shift, Space bar)
► This is called Fitt’s Law
 Keypads for small devices
► PDAs, Cellphones, Game consoles
► Fold out keyboards
► Virtual keyboard
► Cloth keyboards (ElekSen)
► Haptic feedback?
► Mobile phones
 Combine static keys with dynamic soft
keys
 Multi-tap a key to get to a character
 Study: Predictive techniques greatly
improve performance
 Ex. LetterWise = 20 wpm vs 15 wpm
multitap
► Draw keyboard on screen and tap w/
pen
 Speed: 20 to 30 wpm (Sears ’93)
► Handwriting recognition (still hard)
 Subset: Graffiti2 (uses unistrokes)
 Pointing Devices
► Direct manipulation needs some pointing device
► Factors:
 Size of device
 Accuracy
 Dimensionality
► Interaction Tasks:
 Select – menu selection, from a list
 Position – 1D, 2D, 3D (ex. paint)
 Orientation – Control orientation or provide direct
3D orientation input
 Path – Multiple poses are recorded
► ex. to draw a line
 Quantify – control widgets that affect variables
 Text – move text
► Faster w/ less error than keyboard
► Two types (Box 9.1)
 Direct control – device is on the screen surface
(touchscreen, stylus)
 Indirect control – mouse, trackball, joystick,
touchpad
 Direct-control pointing
► First device – lightpen
 Point to a place on screen and press a
button
 Pros:
► Easy to understand and use
► Very fast for some operations (e.g.
drawing)
 Cons:
► Hand gets tired fast!
► Hand and pen blocks view of screen
► Fragile
► Evolved into the touchscreen
 Pros: Very robust, no moving parts
 Cons: Depending on app, accuracy
could be an issue
► 1600x1600 res with acoustic wave
 Must be careful about software design
for selection (land-on strategy).
► If you don’t show a cursor of where you
are selecting, users get confused
 User confidence is improved with a
good lift-off strategy
 Direct-control pointing
► Primarily for novice
users or large user
base
► Case study: Disney
World
► Need to consider those
who are: disabled,
illiterate, hard of
hearing, errors in
usage (two touch
points), etc.
 Indirect-Control Pointing
► Pros:
 Reduces hand-fatigue
 Reduces obscuration problems
► Cons:
 Increases cognitive load
 Spatial ability comes more into play
► Mouse
 Pros:
► Familiarity
► Wide availability
► Low cost
► Easy to use
► Accurate
 Cons:
► Time to grab mouse
► Desk space
► Encumbrance (wire), dirt
► Long motions aren’t easy or obvious (pick up and replace)
 Consider, weight, size, style, # of buttons, force feedback
 Indirect-Control Pointing
► Trackball
 Pros:
► Smallphysical footprint
► Good for kiosks
► Joystick
 Easy to use, lots of buttons
 Good for tracking (guide or
follow an on screen object)
 Does it map well to your
app?
► Touchpoint
 Pressure-sensitive ‘nubbin’ on
laptops
 Keep fingers on the home
position
 Indirect-Control Pointing
► Touchpad
 Laptop mouse device
 Lack of moving parts,
and low profile
 Accuracy, esp. those w/
motor disabilities
► Graphics Tablet
 Screen shot
 comfort
 good for cad, artists
 Limited data entry
 Comparing pointing devices
► Direct pointing
 Study: Faster but less accurate than indirect (Haller ’84)
► Lots of studies confirm mouse is best for most tasks for
speed and accuracy
► Trackpoint < Trackballs & Touchpads < Mouse
► Short distances – cursor keys are better
► Disabled prefer joysticks and trackballs
 If force application is a problem, then touch sensitive is preferred
 Vision impaired have problems with most pointing devices
► Usemultimodal approach or customizable cursors
► Read Vanderheiden ’04 for a case study
► Designers should smooth out trajectories
► Large targets reduce time and frustration
 Example
► Five
fastest places to click on for a right-
handed user?
 Example
► What affects time?
 Fitts’s Law
► Paul Fitts (1954) developed a model of human hand
movement
► Used to predict time to point at an object
► What are the factors to determine the time to point to
an object?
 D – distance to target
 W – size of target
► Just from your own experience, is this function linear?
 No, since if Target A is D distance and Target B is 2D
distance, it doesn’t take twice as long
 What about target size? Not linear there either
► MT = a + b log2(D/W + 1)
 a = time to start/stop in seconds (empirically
measured per device)
 b = inherent speed of the device (empirically
measured per device)
 Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W =
2 cm
► Ans: 300 + 200 log2(14/2 + 1) = 900 ms
 Really a slope-intercept model
 Fitts’s Law
► MT = a + b log2(D/W + 1)
 a = time to start/stop in seconds (empirically measured
per device)
 b = inherent speed of the device (empirically measured
per device)
 Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm
► Ans:
300 + 200 log2(14/2 + 1) = 900 ms
 Question: If I wanted to half the pointing time (on average), how
much do I change the size?
► Proven to provide good timings for most age groups
► Newer versions taken into account
 Direction (we are faster horizontally than vertically)
 Device weight
 Target shape
 Arm position (resting or midair)
 2D and 3D (Zhai ’96)
 Very Successfully Studied
► Applies to
 Feet, eye gaze, head mounted sights
 Many types of input devices
 Physical environments (underwater!)
 User populations (even retarded and drugged)
 Drag & Drop and Point & Click
► Limitations
 Dimensionality
 Software accelerated pointer motion
 Training
 Trajectory Tasks (Accot-Zhai Steering Law)
 Decision Making (Hick’s Law)
► Results (what does it say about)
 Buttons and widget size?
 Edges?
 Popup vs. pull-down menus
 Pie vs. Linear menus
 iPhone/web pages (real borders) vs. monitor+mouse (virtual borders)
► Interesting readings:
 http://particletree.com/features/visualizing-fittss-law/
 http://www.asktog.com/columns/022DesignedToGiveFitts.html
 http://www.yorku.ca/mack/GI92.html
 Precision Pointing Movement Time
► Study: Sears and Shneiderman ’91
 Broke down task into gross and fine components for small targets
 PPMT = a + b log2(D/W+1) + c log2(d/W)
►c – speed for short distance movement
► d – minor distance
 Notice how the overall time changes with a smaller target.
► Other factors
 Age (Pg. 369)
► Research: How can we design devices that produce smaller
constants for the predictive equation
 Two handed
 Zooming
 Novel Devices
► Themes:
 Make device more diverse
► Users
► Task
 Improve match between task
and device
 Improve affordance
 Refine input
 Feedback strategies
► Foot controls
 Already used in music where
hands might be busy
 Cars
 Foot mouse was twice as slow as
hand mouse
 Could specify ‘modes’
 Novel Devices
► Eye-tracking
 Accuracy 1-2 degrees
 selections are by constant
stare for 200-600 ms
 How do you distinguish w/ a
selection and a gaze?
 Combine w/ manual input
► Multiple degree of freedom
devices
 Logitech Spaceball and
SpaceMouse
 Ascension Bird
 Polhemus Liberty and
IsoTrack
 Novel Devices
► Boom Chameleon
 Pros: Natural, good spatial
understanding
 Cons: limited applications,
hard to interact (very
passive)
► DataGlove
 Pinch glove
 Gesture recognition
 American Sign Language,
musical director
 Pros: Natural
 Cons: Size, hygiene,
accuracy, durability
 Novel Devices
► Haptic Feedback
 Why is resistance useful?
 SensAble Technology’s Phantom
 Cons: limited applications
 Sound and vibration are easier and
can be a good approximation
► Rumble pack
► Two-Handed input
 Different hands have different
precision
 Non-dominant hand selects fill, the
other selects objects
► Ubiquitous Computing and Tangible
User Interface
 Active Badges allows you to move
about the house w/ your profile
 Which sensors could you use?
 Elderly, disabled
 Research: Smart House
 Myron Kruger – novel user
participation in art (Lots of exhibit
art at siggraph)
 Novel Devices
► Paper/Whiteboards
 Video capture of annotations
 Record notes (special tracked pens
Logitech digital pen)
► Handheld Devices
 PDA
 Universal remote
 Help disabled
► Read LCD screens
► Rooms in building
► Maps

 Interesting body-context-sensitive.
► Ex.hold PDA by ear = phone call
answer.
 Novel Devices
► Miscellaneous
 Shapetape – reports 3D
shape.
► Tracks limbs
► Engineer for specific
app (like a gun trigger
connected to serial
port)
 Pros: good affordance
 Cons: Limited general
use, time
 Speech and Auditory Interfaces
► There’s the dream
► Then there’s reality
► Practical apps don’t really require freeform
discussions with a computer
 Goals:
► Low cognitive load
► Low error rates
► Smaller goals:
 Speech Store and Forward (voice mail)
 Speech Generation
 Currently not too bad, low cost, available
 Speech and Auditory Interfaces
► Bandwidth is much lower than visual displays
► Ephemeral nature of speech (tone, etc.)
► Difficulty in parsing/searching (Box 9.2)
► Types
 Discrete-word recognition
 Continuous speech
 Voice information
 Speech generation
 Non-speech auditory
► If you want to do research here, lots of research in the
audio, audio psychology, and DSP field you should
understand
 Discrete-Word Recognition
► Individual words spoken by a specific person
► Command and control
► 90-98% for 100-10000 word vocabularies
► Training
 Speaker speaks the vocabulary
 Speaker-independent
► Still requires
 Low noise operating environment
 Microphones
 Vocabulary choice
 Clear voice (language disabled are hampered, stressed)
 Reduce most questions to very distinct answers (yes/no)
 Discrete-Word Recognition
► Helps:
 Disabled
 Elderly
 Cognitive challenged
 User is visually distracted
 Mobility or space restrictions
► Apps:
 Telephone-based info
► Study: much slower for cursor movement than mouse or keyboard
(Christian ’00)
► Study: choosing actions (such as drawing actions) improved
performance by 21% (Pausch ’91) and word processing (Karl ’93)
 However acoustic memory requires high cognitive load (> than hand/eye)
► Toys are successful (dolls, robots). Accuracy isn’t as important
► Feedback is difficult
 Continuous Speech Recognition
► Dictation
► Error rates and error repair are still poor
► Higher cognitive load, could lower overall quality
► Why is it hard?
 Recognize boundaries (normal speech blurs them)
 Context sensitivity
 “How to wreck a nice beach”
► Much training
► Specialized vocabularies (like medical or legal)
► Apps:
 Dictate reports, notes, letters
 Communication skills practice (virtual patient)
 Automatic retrieval/transcription of audio content (like radio, CC)
 Security/user ID
 Voice Information Systems
► Use human voice as a source of info
► Apps:
 Tourist info
 Museum audio tours
 Voice menus (Interactive Voice Response IVR systems)
► Use speech recognition to also cut through menus
 If menus are too long, users get frustrated
 Cheaper than hiring 24 hr/day reps
► Voice mail systems
 Interface isn’t the best
► Get email in your car
 Also helps with non-tech savvy like the elderly
► Potentially aides with
 Learning (engage more senses)
 Cognitive load (hypothesize each sense has a limited ‘bandwidth’)
► Think ER, or fighter jets
 Speech Generation
► Playback speech (games)
► Combine text (navigation systems)
► Careful evaluation!
 Speech isn’t always great
► Door is ajar – now just a tone
► Use flash
► Supermarket scanners

 Often times a simple tone is better

 Why? Cognitive load
► Thuscockpits and control rooms need speech
► Competes w/ human-human communication
 Speech Generation
► Ex: Text-to-Speech (TTS)
► Latest TTS uses multiple syllabi to make generated speech sound
better
 Robotic speech could be desirable to get attention
 All depends on app
 Thus don’t assume one way is the best, you should user test
► Apps: TTS for blind, JAWS
► Web-based voice apps: VoiceXML and SALT (tagged web pages).
 Good for disabled, and also for mobile devices
► Use if
 Message is short
 Requires dynamic responses
 Events in time
► Good when visual displays aren’t that useful. When?
 Bad lighting, vibrations (say liftoff)
 Non-speech Auditory Interface
► Audio tones that provide information
► Major Research Area
 Sonification – converting information into audio
 Audiolization
 Auditory Interfaces
► Browsers produced a click when you clicked on a
link
 Increases confidence
 Can do tasks without visual cognitive load
 Helps figure out when things are wrong
 Greatly helps visually impaired
 Non-speech Auditory Interface
► Terms:
 Auditory icons – familiar sounds
(record real world sound and
play it in your app)
 Earcons – new learned sounds
(door ajar)
► Role in video games is huge
 Emotions, Tension, set mood
► To create 3D sound
 Need to do more than stereo
 Take into account Head-related
transfer function (HRTF)
► Ear and head shape
► New musical instruments
 Theremin
► New ways to arrange music
 Displays
► Primary Source of
feedback
► Properties:
 Physical Dimension
 Resolution
 Color Depth and correctness
 Brightness, contrast, glare
 Power
 Refresh rate
 Cost
 Reliability
 # of users
 Display
Technology
► Monochrome displays (single
color)
 Low cost
 Greater intensity range
(medical)
► Color
 Raster Scan CRT
 LCD – thin, bright
 Plasma – very bright, thin
 LED – large public displays
 Electronic Ink – new product w/
tiny capsules of negative black
particles and positive white
 Braille – refreshable cells with
dots that rise up
 Large Displays
► Wall displays
 Informational
► Control rooms, military, flight
control rooms, emergency
response
► Provides
 System overview
 Increases situational awareness
 Effective team review
► Old: Array of CRTs
 Interactive
► Require new interaction
methods (freehand sketch,
PDAs)
► Local and remote collaboration
► Art, engineering
 Large Displays
► Multiple Desktop Displays
 Multiple CRTs or Flat panels for
large desktops
 Cheap
 Familiar
 Spatial divide up tasks
 Comparison tasks are easier
 Too much info?
► HMD
► Eventually -> Every surface a
pixel
 Mobile device displays
► Applications
 Personal
► Reprogrammable picture
frames
 Digital family portrait
(GaTech)
 Business
► PDAs, cellphones
 Medical
► Monitor patients
 Research: Modality
Translation Services (Trace
Center – University of
Wisconsin)
► As you move about it auto
converts data, info, etc. for
you
 Mobile device displays
► Actions on mobile devices
 Monitor information and
alert (calendar)
 Gather then spread out
information (phone)
 Participate in groups and
relate to individual
(networked devices)
 Locate services and identify
objects (GPS car system)
 Capture and then share info
(phone)
 Mobile device displays
► Guidelines for design
 Bergman ’00, Weiss, ’02
 Industry led research and design case studies
(Lindholm ’03)
 Typically short in time usage (except handheld
games)
 Optimize for repetitive tasks (rank functions
by frequency)
 Research: new ways to organize large
amounts of info on a small screen
 Study: Rapid Serial Visual Presentation
(RSVP) presents text at a constant speed
(33% improvement Oquist ’03)
 Searching and web browsing still very poor
performance
 Promising: Hierarchical representation (show
full document and allow user to select where
to zoom into)
 Animation, Image, and Video
► Content quality has also greatly
increased
► 3D rendering is near life-like
► Digital Photography is common
► Scanned documents
► Video compression
► Multimedia considerations for the
disabled
► Printers
 3D Printers create custom objects
from 3D models