1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

Week 5: Usability Testing

Week 5: Remote user testing

Web services for user testing

http://www.usertesting.com/

Specify the tasks you want users to perform.

Test on all devices: mobile, tablet, desktop.
We recruit participants
Users are available to test 24/7.
You get results in 1 hour

Web services user testing

Let’s see those videos

http://www.usertesting.com/ (http://www.usertesting.com)

Another similar service:

http://www.trymyui.com/ (http://www.trymyui.com/)

Controlled experiment: A/B testing

From Ronny Kohavi, Microsoft’s General Manager of the Analysis and Experimentation team &
Stanford Professor

http://ai.stanford.edu/~ronnyk/ (http://ai.stanford.edu/~ronnyk/%20)

Concept is trivial
Set your OEC = Overall Evaluation Criterion.

Randomly split traffic between two (or more) versions

A (Control)
B (Treatment)

Collect metrics of interest

Analyze

http://www.exp-platform.com/Pages/KDD2015KeynoteExPKohavi.aspx (http://www.exp-
platform.com/Pages/KDD2015KeynoteExPKohavi.aspx)

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 1/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

The First Controlled Experiment

The earliest controlled experiment was a test for vegetarianism, suggested in the Old Testament's
Book of Daniel

Test your servants for ten days. Give us nothing but vegetables to eat and water to drink. Then
compare our appearance with that of the young men who eat the royal food, and treat your
servants in accordance with what you see

http://www.exp-platform.com/Pages/KDD2015KeynoteExPKohavi.aspx (http://www.exp-
platform.com/Pages/KDD2015KeynoteExPKohavi.aspx)

Real Example: MSN Home Page Search Box

OEC: Clickthrough rate for Search box and popular searches

Differences: A has taller search box (overall size is the same), has magnifying glass icon, “popular
searches” B has big search button, provides popular searches without calling them out

Real Example: Bing Ads with Site Links

Should Bing add “site links” to ads, which allow advertisers to offer several destinations on ads?

OEC: Revenue, ads constraint to same vertical pixels on avg

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 2/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

Pro adding: richer ads, users better informed where they land

Cons: Constraint means on average 4 “A” ads vs. 3 “B” ads Variant B is 5msc slower (compute +
higher page weight)

Case study: Facebook Social Experiments

 Facebook's social experiment (8:20 )

Tech Report: Facebook's Social Experiment Draws …

The Good and The Bad?

A/B testing?
Ethical issues?
Direct / indirect?

User-centred mobile design

Very agile model has to be used.

Be able to backtrack quickly and uses a flexible workflow system.
Feedback from users starts from idea conception all the way to release into the App Store.
In mobile development, usually even the good programmers have a great appreciation for quick
user feedback and testing methodologies.

Usability testing of mobile apps

Step-by-step guide: http://usabilitygeek.com/usability-testing-mobile-applications/

(http://usabilitygeek.com/usability-testing-mobile-applications/)

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 3/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

Mobile device usability testing

https://www.mrtappy.com/ (https://www.mrtappy.com/)

Attention span study

http://www.telegraph.co. uk/science/2016/03/12/humanshave-shorter-attention-span-thangoldfish-
thanks-to-smart/

Micro Moments

https://think.storage.googleapis.com/images/micromoments-guideto-winning-shift-to-mobile-
download.pdf

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 4/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

User testing in product development phase

Alpha, Beta testing

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 5/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

http://www.guru99.com/alpha-beta-testing-demystified.html (http://www.guru99.com/alpha-beta-
testing-demystified.html)

Google Analytics – sample interaction logging

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 6/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

Google Analytics – Behaviour Flow

Quantitative and Qualitative Research in Product Development Phase

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 7/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

Quantitative and Qualitative Data in UX research

Both are useful and complementary

Qualitative data describes (e.g. give insights to nuance)
Quantitative data counts (measures the significance)
UX/UCD research does both

 Qualitative and quantitative UX research (3:49)

Qualitative + Quantitative UX Research

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 8/9
1/26/2019 Week 5: Remote user testing: User-centred Design (1897)

https://rmit.instructure.com/courses/38071/pages/week-5-remote-user-testing?module_item_id=1298240 9/9