Data-driven Design

Making Software & Websites Better

Through Use of Statistics

Patrick McKenzie
April 15th, 2010
Creative Commons Licensed CC-BY. Please feel free to share.
 Outline


Introduction 
自己紹介

Art & Science 
芸術と技術

Rationale 
目的

Instrumentation 
動作監視

Funnel Analysis 
ファネル分析

A/B Testing 
A/B テスト

Conclusion 
まとめ
The Plan
What Happened
What Happened Next
Started Business
Art into Science 芸術から技術へ

Supplement the vision and craft of designers

with the logic, regimentation, and process of
science.

デザイナーの芸術を技術の倫理、数学、体
験方式で拡張する。
 Rationale 目的


Decrease Costs 
開発工数ダウン

Decrease 
不確実性ダウン
Uncertainty 
ユーザ足度アップ
満足 度ア ップ

Increase Satisfaction 
成功率アップ

Increase Success 
利益アップ

Increase Profits
 Designing With Data


Get Facts to Answer The Questions:

What are users doing? 何をしているか？

Which users are doing things? 誰がしている
か？

Where are they doing things? 何処にしている
か？

(Maybe) Why are they doing it? 何故している
か？
 Types of Instrumentation


Analytics software

Heatmapping

Surveys

Ad hoc analysis (Excel, etc)

Custom software
CrazyEgg
 Landing Page


Photo of problem solved goes here
Landing Page (Fixed)
 Paving The Trails


Picture of paving a path
Instrumenting Software
Instrumenting Software #2
 Funnel Analysis ファネル分析

Many applications have several steps followed
by a goal.

Only a percentage of users clear each step.

Funnel analysis finds where folks are falling
out, to concentrate efforts on improvement

Viewed Signed
Landing Did $
Up For
Page Work $
Trial
 Software ソフト


Google Analytics

Free but limited

Good for
starter

Mixpanel

Cheap

API

Harder to use

 Define Funnel

1.Signs In
2.Opens Word List
3.Saves Word List
4.Customizes Cards
5.Creates Cards
6.Downloads Card
 Collect Data

1.Signs In: 260

2.Opens Word List: 218 (83%)
3.Saves Word List: 191 (87%)
4.Customizes Cards 161 (84%)
5.Creates Cards: 131 (81%)
6.Downloads Card 126 (95%)
Total Conversion Rate 126 / 260 = ~48.4%
Identify Areas for Improvement

1.Signs In: 260

2.Opens Word List: 218 (83%)
3.Saves Word List: 191 (87%)
4.Customizes Cards 161 (84%)
5.Creates Cards: 131 (81%)
6.Downloads Card 126 (95%)
Total Conversion Rate 126 / 260 = ~48.4%
 Implement Improvements


Open Word List needlessly complicated: make
easier.

Save Button was unclear: tweak microcopy,
button design.

Customize Cards needlessly complicated:
show less options.
Before
After
 Collect Metrics

1.Signs In: 194

2.Opens Word List: 155 (79%)
Up from 83%
3.Saves Word List: 151 (97%)
4.Customizes Cards: 127 (84%)
5.Creates Cards: 112 (88%) Up from 81%

6.Downloads Card 126 (98%)

Total Conversion Rate goes from 48.5% to 56.7%
16.6% lift in customer success / revenue for 1 hour of work.
 Pretend You Own A Bar
Half of Customers Get... Half of Customers Get...
 A/B Testing


Divide similar population into two groups
randomly.

Treat the groups differently.

Measure difference in outcomes.

Scientific Method applied to design.
Eliminates bias, randomness, poor test
design.
 Actual A/B Test
Half of Users See... Half of Users See...
 Software Tools


Google Website Optimizer

Terrible. Do not use.

Visual Website Optimizer

visualwebsiteoptimizer.com

WYSIWYG alternative creation. Very nice.

Free for now.

A/Bingo or similar

Rails A/B testing library.

Programmer friendly, powerful, easy to use.
 Core A/B Testing Loop


Identify conversion and instrument.

Create alternatives.

Assign users randomly to alternatives

Watch behavior.

Identify winning alternative.

Iterate, iterate, iterate...
Landing Pages

Source: whichtestwon.com
Signup Forms
Signup Forms
Signup Forms
 New Features
60 ~ 80% of features do not change user behavior
Eric Ries
Old Version
Spot The New Feature
 Core A/B Testing Discipline


Accept that you do not know what is right.
(Hardest lesson.)

Test frequently, as a matter of routine. (Helps
to have one-line testing.)

Test changes in isolation.

Use evidence from tests to inform your
marketing and development plans.
 Conclusion


What we learned today:

Importance of tracking user interaction.

Funnel analysis – measuring user success and
goal completion.

A/B tests – improving marketing message and
product design.

Thanks for listening. ご清聴、ありがとうござ
いました。

patrick@kalzumeus.com ← My email.

http://www.kalzumeus.com ← My blog.

http://www.abingo.org ← A/Bingo software
(OSS)

http://www.startuplessonslearned.com ← Eric
Ries, very useful.
 Photo Credits
Microsoft logo: http://www.flickr.com/photos/scobleizer/2264763977/
Sleeping Salaryman: http://www.flickr.com/photos/kansai/3452417503/
Edamame (Green Soybeans): http://www.flickr.com/photos/kankan/33346213/
Salty Peanuts: http://www.flickr.com/photos/euromagic/2351628831/
Facepalm Statue: http://www.flickr.com/photos/cesarastudillo/3981364314/