Thank&you&for&having&me&here&today.&This&morning&Im&going&to&present&some&ideas&Im& working&through&about&designing&user&experiences&for&a&world&where&many,&if&not&all& objects&in&your&environment&have&some&form&of&networked&digital&component.

&

1&

Let&me&begin&by&telling&you&a&bit&about&my&background.&I m&a&user&experience&designer.&I& started&designing&Web&sites&back&in&the&Jurassic.&Heres&a&design&I&worked&on&for&HotBot,&an& early&search&engine.&

2&

Ive&also&worked&on&the&user&experience&design&of&a&lot&of&consumer&electronics&products& from&companies&youve&probably&heard&of.&

3 !

Along&the&way&I&wrote&a&couple&of&books&based&on&my&experience&as&a&designer.&One&is&a& cookbook&of&user&research&methodsthe&second&ediIon&came&out&last&falland&the& second&describes&what&I&think&are&some&of&the&core&concerns&when&designing&networked& computaIonal&devices.&It&came&out&a&couple&of&years&ago.&The&ideas&for&this&talk&originated& in&the&second&book.&

4&

I&also&started&a&couple&of&companies.&The&rst,&AdapIve&Path,&is&primarily&focused&on&design& for&the&Web.&With&the&second&one,&ThingM,&I&got&deep&into&developing&hardware.&

5 !

ThingM&primarily&designs&and&manufactures&a&range&of&RGB&LEDs&with&aRached&controllers& for&architects,&industrial&designers&and&hackers.&These&have&been&on&sale&for&about&ve& years&and&theyre&used&in&everything&from&ying&robot&prototypes&to&costumes&in&Lady& Gagas&stage&show.&

6&

We&also&spent&some&Ime&exploring&the&potenIal&uses&of&augmenIng&everyday&objects&with& embedded&technology.& & On&the&leU&is&an&RFID&wine&rack&thats&faceted&classicaIon&browser&to&a&cloudWbased& service.&For&example,&it&can&show&you&all&the&2003&Napa&Cabernet&Sauvignons&which&are& ready&to&drink&and&whose&current&market&price&is&under&$50.&On&the&right&is&a&capaciIve& sensing&kitchen&cabinet&knob&that&glows&when&you&touch&it.&Its&&an&exploraIon&in&making&a& digital&product&that&would&sIll&be&useful&20&years&aUer&it&was&made.& &

7&

In&January&of&this&year&I&joined&PARC&as&a&principal&in&its&InnovaIon&Services&group.&

8&

But&I&didnt&come&here&to&talk&about&myself.&&When&I&was&last&here&two& years&ago,&I&advocated&renaming&augmented&reality&as&the&less&exciIngW sounding,&but&perhaps&more&accurate&somaIc&data&percepIon.&

9&

My&argument&was&that&visual&AR,&informaIon&layered&onto&our&visual&eld,&faces&a&big& problem,&and&thats&the&limited&bandwidth&of&human&percepIon.&We&can&start&with&how& lowWresoluIon&our&eyes&are.&This&is&our&foveal&resoluIon,&which&you&will&noIce&wouldnt& even&cut&it&as&the&cheapest&mobile&phone.&

10&

To&compensate,&our&eyes&move&in&saccades&which&last&between&20ms&and&200ms,&or&5&to&50 & Imes&per&second.&

11&

So&this&leads&to&a&back&of&the&envelope&calculaIon&of&eye&bandwidth&between&100&bits&per& second&and&10K&bits&per&second& & Thats&around&4&orders&of&magnitude&slower&than&a&modern&frontWside&bus.&& & The&brain&deals&with&this&through&a&series&of&ingenious&lters&and&adaptaIons&to&create&the& illusion&of&an&experience&of&all&reality,&but&at&the&core&theres&only&a&limited&amount&of& bandwidth&available&and&our&visual&senses&are&easily&overwhelmed.&And&unfortunately&our& eyes&and&brains&are&not&subject&to&Moores&Law.&Were&not&going&to&get&more&foveal& resoluIon&or&cogniIve&processing&capability&any&Ime&soon.&

12&

My&argument&is&to&broaden&the&deniIon&of&AR&to&be&the&experience&of&any&kind&of&data&by& any&sense,&to&change&our&language&away&from&augmented&reality&as&primarily&visual& augmentaIon&to&something&thats&more&representaIve&of&the&whole&experience&of&data&in& the&environment,&the&bodily&experience&of&the&informaIon&shadows&that&objects&we& encounter&in&the&world&cast&into&the&internet,&the&personal&experience&of&the&Internet&of& Things.&I&called&this&SomaIc&Data&PercepIon.&

13&

Once&we&broaden&our&deniIon&to&include&other&senses,&we&can&start&creaIng&hierarchies& of&informaIon&that&match&the&capabiliIes&of&our&senses&and&avoid&pueng&secondary& informaIon&into&our&primary&sense.&We&can&begin&associaIng&secondary&data&with& secondary&senses.&

14&

There&are&plenty&of&ways&we&can&augment&the&world&that&doesnt&require&increasing&the& load&on&our&visual&cortex.&Were&already&suciently&experienced&with&phone&buzzing&that& we&can&feel&phantom&vibraIons&because&our&brains&have&deeply&embedded&that&sensaIon& as&meaningful&and&when&theres&a&misre,&we&jump.&Other&projects,&such&as&Jason&Wilsons& Audio&Perfume&create&shiUing&soundscapes&that&are&Ied&to&informaIon.&You&can&think&of& other&examples,&such&as&an&armband&that&gets&warmer&as&you&bike&past&an&apartment&thats & for&rent&in&your&price&range.&These&arent&nearly&as&high&resoluIon&signals&as&visual&ones,& which&makes&them&feel&less&important,&but&I&think&theyre&actually&incredibly&important& because&of&the&informaIon&they&remove&from&the&visual&channel.&

15&

Next,&I&want&to&say&that&Im&a&rm&believer&that&everything&is&going&to&be&augmented&with& networked&informaIon&processing&very&soon.&

16&

For&example,&adidas&has&the&miCoach&and&miCoach&elite&systems,&in&which&a&soccer&ball,&the& players&shirts,&their&shoes&and&someImes&the&space&theyre&in&are&all&networked&and& instrumented.&The&screen&devices&they&carry&are&the&displays&of&the&informaIon&shadows& all&of&the&things&are&casIng&onto&the&Net.&Very&soon&this&is&going&to&be&your&oce,&your& playground,&your&home&and&your&coee&shop.&Everything&that&can&be&connected&to&the&net,& including&things&that&probably&shouldnt&be,&will&be.&Full&stop.&

17&

Now&we&get&to&the&interesIng&user&experience&challenge.&How&do&we&design&for&the& environment&where&everything&has&a&data&shadow&and&can&be&perceived&through&any& sense?&Designers&work&well&with&constraints,&so&Im&focusing&on&two&ideas.&The&rst&is&that& the&scale&of&an&experience&is&a&criIcal&component&of&how&that&experience&is&perceived.&

18&

Computers&have&tradiIonally&had&interfaces&that&are&roughly&sized&to&our&bodies.&If&you& look&at&a&laptop,&the&inputs&are&the&size&of&your&hands&and&the&output&is&on&the&scale&of&your& head.&We&dont&have&great&models&for&designing&experiences&for&dierent&scales,&and&when& we&try&to&map&what&we&know&from&designing&PC&experiences&directly&onto&other&scales,&it& fails,&as&everyone&who&has&ever&designed&a&phone&interface&for&the&rst&Ime&knows.& & At&PARCs&ubicomp&program&in&the&late&80s&and&early&90s&they&dened&the&tab,&pad&and& board&as&names&for&the&scales&of&the&devices&they&were&developing,&leveraging&the&form& factors&of&oce&supplies&that&had&proven&to&be&at&useful&sizes.&

19&

This&is&the&scale&Ive&been&using.&Its&of&course&inspired&by&the&Eames&Powers&of&Ten&lms,& but&Ive&been&using&it&to&understand&the&constraints&and&potenIal&of&devices&at&dierent& scales.&Screens&dont&work&interacIvely&when&you&approach&the&covert&scale,&which&is&why& wristWmounted&devices&can&at&best&be&output&devices&displaying&a&limited&set&of&visual& informaIon,&but&theyre&great&for&sending&a&small&message&no&one&else&knows&about.&Thats& really&hard&to&do&with&a&big&display&such&as&a&TV.&BuRons&dont&work&well&controlling& experiences&on&the&environmental&scale&and&above,&because&theyre&too&small&relaIve&to& the&object.&

20&

The&two&screen&experience,&that&mythological&consumer&electronics&grail,&is&really&a& problem&of&mismatching&scales.&How&can&you&create&a&single&experience&that& simultaneously&spans&the&mobile&and&personal&scale?&

21&

To&me&this&brings&me&to&my&second&idea,&which&is&that&for&an&experience&to&make&sense& across&various&scales,&we&need&a&narraIve&to&Ie&it&together.&I&mean&this&quite&literally.&We& need&a&story&of&some&sort&that&will&help&us&make&sense&of&why&what&were&doing&at&any& given&moment&maRers.&The&desktop&metaphor&is&a&kind&of&a&narraIve.&When&it&was&applied& to&the&same&kinds&of&funcIons&that&had&existed&with&the&command&line,&it&made&them&much& easier&for&people&to&understand&because&it&told&a&story&about&oce&work.&It&wasnt&a& perfect&mapping,&because&it&was&a&metaphor,&but&it&was&beRer&than&what&it&replaced.&

22&

Look&at&the&way&we&explain&what&future&technology&is&going&to&do.&These&days&its&always&in& the&form&of&stories&that&use&the&technology,&whether&its&being&a&caring&parent&while& engaged&in&some&kind&of&serious&mulInaIonal&project,&or&opImizing&your&crop&planIng,&its& always&a&story&about&what&the&end&result&is&going&to&be.&

23&

Reeves&and&Nass,&two&Stanford&professors,&did&a&series&of&groundbreaking&experiments&that& showed&that&no&maRer&how&you&cut&it,&when&people&are&in&dialog&with&a&device,&they&treat& it&like&a&person.&And&since&the&kinds&of&relaIonships&we&have&with&people&are&essenIally& narraIves,&this&in&eect&means&that&all&of&our&experiences&with&devices&have&a&narraIve& component,&whether&we&put&one&in&or&not.&

24&

And&so&we&come&to&transmedia&storytelling.&

25&

Many&of&you&are&probably&familiar&with&Henry&Jenkins&work&on&how&stories&are&now&told,& with&bits&and&pieces&scaRered&across&many&media.&Jenkins&canonical&examples&is&the& Matrix,&where&characters&and&acIon&in&the&second&and&third&lm&didnt&really&make&sense& unless&you&had&watched&the&Animatrix&cartoon,&read&the&comics&and&played&the&games.&&A& more&recent&example&is&Jorgits&and&the&end&of&Winter,&which&is&a&set&of&childrens&stories& that&are&told&in&an&app&thats&mostly&a&novel,&but&has&elements&of&video&games;&however,&in& addiIon&to&that&the&characters&have&their&own&TwiRer&feeds,&Facebook&pages,&Soundcloud& streams&and&Pinterest&pages.&The&experience&is&carried&across&devices&and&media.&

26&

In&the&same&way&that&you&can&keep&watching&a&nenlix&movie&as&you&jump&from&device&to& device.&

27&

Or&read&a&Kindle&book&everywhere.&

28&

So&heres&my&thought.&That&the&way&that&we&can&bridge&the&gap&in&our&ability&to&create& compelling&user&experiences&that&span&scales&is&through&using&narraIve&to&tell&stories&that& link&together&the&individual&acIons&across&devices&and&between&screens.&I&think&that&this&is& already&happening.&

29&

One&of&my&favorite&ubicomp&products&from&the&toy&world&is&ve&years&old&and&was&a& commercial&failure,&but&I&think&it&points&to&whats&possible.&It&was&called&Clickables&and&it& was&developed&for&the&Disney&Fairies&lines&by&a&Hong&Kong&company&called&TechnoSource.&It & linked&the&virtual&world&Pixie&Hollow,&the&Fairies&virtual&social&network&and&chat&space,&with& the&physical&world&of&the&girls&who&were&the&target&audience&through&the&use&of&smart& jewelry.& & Source:&Disney&Clickables&

30&

When&two&kids&put&their&Clickables&bracelets&together,&their&avatars&would&link&up&in&Pixie& Hollow.&Moreover,&all&of&the&products&had&simultaneous&onlineWoine&existence,&so&for& example,&when&a&girl&got&one&of&the&charm&bracelets,&she&could&touch&the&USBWconnected& jewelry&box&and&her&fairy&avatar&gets&a&version&of&the&same&charm,&plus&some&unique&magic& buReries&she&could&only&get&by&geeng&one&of&these&charms,&or&being&friends&with& someone&who&did.&

31&

Now&the&execuIon&of&the&Clickables&experience&leU&much&to&be&desired,&but&I&think&that&the& basic&components&of&weaving&the&augmented&physical&device&with&the&social&network&was& powerful.&The&mode&of&use&explained&how&the&charms&and&the&jewelry&worked&provided&a& jusIcaIon&for&why&to&use&it.&The&narraIve&is&a&kind&of&scaolding&on&which&the&experience& is&built.&

32&

In&the&grownWup&world,&we&have&this&collaboraIon&between&Diageo&and&Evrythng,&a&Swiss& Internet&of&Things&company.&BoRles&of&a&giU&whiskey&have&a&unique&QR&code&that&leads&to& an&app&that&allows&the&buyer&to&record&a&video&for&the&recipient.&The&technologywhy&its& there&and&how&to&use&itis&embedded&in&a&narraIve.&Its&a&giU.&Youre&not&going&to&drink& this&special&boRle,&its&for&giving,&and&heres&some&technology&to&help&with&that.&

33&

This&is&Good&Night&Lamp,&a&set&of&paired&augmented&objects.&The&name&itself&creates&an& implicit&narraIve&that&tells&you&how&to&use&these.&The&narraIve&is&what&drives&the&design,& the&interacIon&and&what&jusIes&the&technology.&

34&

So,&in&conclusion,&we&need&to&think&about&transmedia&narraIve&as&a&means&to&unify& augmented&world&experiences&across&devices,&across&uses,&and&across&scales,&to&bridge&the& device&disconnects&that&vex&us&and&which&we&call&second&screen&experiences&and& appcessory&experiences.&As&designers&its&our&jobs&to&tell&people&why&and&how&they&need&to& use&our&technology.&In&eect,&we&already&tell&stories,&but&I&think&were&doing&a&terrible&job& of&telling&stories&about&the&augmented&world,&and&why&its&not&just&important,&but&criIcal& that&people&understand&and&engage&with&it.&We&didnt&need&the&desktop&metaphor&forever,& but&it&certainly&helped&during&a&criIcal&period&of&compuIng&to&give&people&a&structure&with& which&to&understand&the&potenIal&of&computers&and&how&they&could&take&advantage&of& that&potenIal.&Similarly&we&need&new&stories&to&tell&about&our&devices&today,&we&need& stories&about&the&augmented&world.& & Star&Wars&Imeline&(CC)&by&williampcoleman:&hRp://www.ickr.com/photos/ 20919058@N05/5850117502/&

35&

Thank&you&

36&