Designing

Human-Computer Systems

Ponnurangam  K  (“PK”)  
Spring  2015,  Lecture  13,    
March  17,  2015  
Recap!

2  
What do we expect by March 25th ?

– One  itera@on  complete  in  the  

implementa@on    
– Some  data  from  real  users    
– Close  to  deploying  the  final  setup  
– Want  to  see  some  data      
– Keep  at  least  7  –  10  –  14  days  for  data  
collec@on    

3  
Evaluation

4  
 Ethical Considerations
–  Sometimes tests can be distressing
–  users have left in tears
–  You have a responsibility to alleviate
–  make voluntary with informed consent
–  avoid pressure to participate
–  let them know they can stop at any time
–  stress you are testing the system, not them
–  make collected data anonymous as possible
–  Often must get human subjects approval
(IRB)

5  
Discuss IRB application
– Proposal    
– All  details    
– Consent  form  
– Discuss  with  example  documents      

6  
This lecture

7  
 Stanford Prison Experiment
–  Randomly assigned
participants as
guards or prisoners
–  Arrested prisoners
at home
–  Sadistic behavior
by guards
–  Fire extinguishers
–  Sexual humiliation
–  Sleep on concrete

8  
Principles of Research with Human Subjects

–  Respect for Persons

–  individuals have autonomy and choice
–  people can not be used as a means
to an end
–  provide protection to the vulnerable
–  provide informed consent and privacy
–  Beneficence
–  Justice

9  
Principles of Research with
Human Subjects
–  Respect for Persons
–  Beneficence
–  kindness beyond duty
–  obligation to do no harm
–  obligation to prevent harm
–  obligation to do good
–  minimize risks, maximize benefits
–  Justice

10  
Principles of Research with
Human Subjects
–  Respect for Persons
–  Beneficence
–  Justice
–  treat all fairly
–  share equitably burdens and benefits

11  
 Federal Regulations
Basic Ethical Principles Applications
1.  Respect for Persons 1.  Informed Consent
2.  Beneficence 2.  Assessment of Risk
and Benefits
Minimize Risk
Protect vulnerable
3.  Fair selection of
3.  Justice Subjects
Institutionalized in “Common Rule”
•  System of Institutional Review Boards (IRBs) to monitor
human subject research
•  Federal regulations for treatment of human subjects
•  http://ohrp.osophs.dhhs.gov/humansubjects/guidance/45cfr46.htm

12  
 Training to Do Human Subjects
Research

–  All human subjects research at many

schools must be reviewed by the IRB
–  E.g. http://www.cmu.edu/provost/spon-res/compliance/hs.htm

–  Researchers have a responsibility to know

how to conduct research ethically
–  Federal government requires training for
ethics in research
–  Many have an explicit health/medical focus

13  
Recent Research Failures
–  U.S. Office for Human Research
Protection suspended all research at John
Hopkins University after one research
participant died
–  May 1999, federal regulators temporarily
stopped research at Duke over concerns
of volunteer safety

14  
Debriefing
–  Conduct with evaluators, observers, and
development team members
–  Discuss general characteristics of UI
–  Suggest potential improvements to address
major usability problems
–  Dev team rates how hard things are to fix
–  Make it a brainstorming session
–  little criticism until end of session

15  
Deciding on Data to Collect
–  Two types of data
–  process data
–  observations of what users are doing & thinking
–  bottom-line data
–  summary of what happened (time, errors,
success)
–  independent and dependent variables

16  
Which Type of Data to Collect?

–  Always focus on process data first

–  gives good overview of where problems are
–  Bottom-line doesn’t tell you where to fix
–  just says “too slow”, “too many errors”, etc.
–  Hard to get reliable bottom-line results
–  need many users for statistical significance

17  
Measuring Bottom-Line Usability

–  Situations in which numbers are useful

–  time on task
–  # tasks successfully completed
–  # errors
–  define in advance what these mean
–  Do not measure bottom-line data with
think-aloud protocol. Why?
–  talking can affect speed & accuracy

18  
Measuring Subjective User
Preference

–  Can rate system on a Likert scale

–  Example: The user interface was easy to use
–  1 - Strongly disagree
–  2 - Disagree
–  3 - Neither agree nor disagree
–  4 - Agree
–  5 - Strongly agree
–  Can be hard to be sure what data means
–  novelty of UI, not realistic setting …

19  
Measuring Subjective User Preference

–  Can also get some useful data by asking

–  What they liked
–  Disliked
–  Where they had trouble
–  Best part
–  Worst part
–  etc. (redundancy is ok)

20  
 Comparing Two
Alternatives
–  Between groups experiment
–  two groups of test users
–  each group uses only 1 of the systems
–  Within groups experiment
–  one group of test users
–  each person uses both systems,
randomized ordering
–  can’t use the same tasks or order (learning)
–  Between groups requires many more
participants than within groups

21  
Discount Usability Engineering

–  Reaction to excuses for not doing

user testing
–  “too expensive”, “takes too long”, …
–  Cheap
–  no special labs or equipment needed
–  the more careful you are, the better it gets
–  Fast
–  on order of 1 day to apply
–  standard user tests may take week or more
–  Easy to use
–  some techniques can be taught in 2-4 hours

22  
Examples of Discount Usability

–  Low-fi prototyping

–  Action analysis (GOMS)
–  Heuristic evaluation
–  On-line, remote usability tests
–  Walkthroughs
–  put yourself in the shoes of a user
–  like a code walkthrough

23  
 Action Analysis & GOMS
–  Basic idea: uses a cognitive model to predict
quantitative (time) and qualitative use for expert
users
–  GOMS stands for
–  Goals – high level goal (and subgoals) in layman terms
–  Operators – low level, e.g. button press, menu select
–  Methods – well-learned sequences (e.g., delete para)
–  Selection – rules for deciding which method to use
–  Input: detailed description of UI / task(s)
–  list steps hierarchically
–  Output: quantitative time measures

24  
Stuart Card, Thomas P.
Moran and Allen Newell, 1983

25  
Non-Computer Example of
GOMS
–  Goal (the big picture)
–  go from hotel to the airport
–  Operators (specific actions)
–  locate bus stop; wait for bus; get on bus; ...
–  Methods
–  walk, take bus, take taxi, rent car, take train
–  Selection rules (choosing among methods)
–  Example: Walking is cheaper, but tiring and slow
–  Example: Taking a bus is complicated abroad

26  
GOMS Output
–  Execution time
–  add up times from operators
–  assumes experts (mastered the tasks)
–  error free behavior
–  absolute accuracy ~10-20%

27  
Using GOMS Output
–  Ensure frequent goals achieved quickly
–  If you want to make sure that a highly repetitive
task is done as quickly as possible, use GOMS
–  Making hierarchy also of value
–  functionality coverage & consistency
–  does UI contain needed functions?
–  consistency: are similar tasks performed
similarly?
–  operator sequence
–  in what order are individual operations done?

28  
Applications of GOMS
–  Comparing different UI designs
–  Estimating number of steps it will require
–  Estimating amount of time
–  Profiling an existing UI
–  Building a help system
–  Modeling makes user tasks & goals explicit
–  Can suggest questions users will ask & the
answers

29  
 Tradeoffs of Using GOMS
–  Advantages
–  Gives quantitative measures
–  In some cases, can be less work than user study
–  Easy to modify when UI is revised
–  Example (Interactions Magazine Oct 1995)
–  “NYNEX’s telephone-operator workstation is another
example in which every second counts because of the
sheer scale of people doing the same tasks so many
times each day. Again, GOMS models predicted a cost
increase of about $2 million per
year had NYNEX chosen to buy the less efficient
workstation.”

30  
 Tradeoffs of Using GOMS
–  Disadvantages
–  takes lots of time, skill, and effort
–  research: tools to aid modeling process
–  only works for goal-directed tasks
–  not problem solving or creative tasks
(design)
–  assumes expert performance w/o error
–  does not address several UI issues
–  readability, memorability of icons, commands

31  
Any questions, clarifications?

32  
Thank you

33