HCIBeyondtheGUI

TheMorganKaufmannSeriesinInteractiveTechnologiesSeriesEditors:StuartCard,PARC;Jo
nathanGrudin,Microsoft;JakobNielsen,NielsenNormanGroupMeasuringtheUserExperience
:Collecting,Analyzing,andPresentingUsabilityMetricsTomTullisandBillAlbertModerat
ingUsabilityTests:PrinciplesandPracticesforInteractingJosephDumasandBethLoringKe
epingFoundThingsFound:TheStudyandPracticeofPersonalInformationManagementWilliamJ
onesGUIBloopers2.0:CommonUserInterfaceDesignDontsandDosJeffJohnsonVisualThinkingf
orDesignColinWareUser-CenteredDesignStories:Real-WorldUCDCaseStudiesCarolRighian
dJaniceJamesSketchingUserExperiences:GettingtheDesignRightandtheRightDesignBillB
uxtonTextEntrySystems:Mobility,Accessibility,UniversalityScottMacKenzieandKumiko
Tanaka-ishiLettingGooftheWords:WritingWebContentthatWorksJaniceGinnyRedishPersonas
andUserArchetypes:AFieldGuideforInteractionDesignersJonathanPruittandTamaraAdlin
Cost-JustifyingUsabilityEditedbyRandolphBiasandDeborahMayhewUserInterfaceDesigna
ndEvaluationDebbieStone,CarolineJarrett,MarkWoodroffe,andShaileyMinochaRapidCont
extualDesignKarenHoltzblatt,JessamynBurnsWendell,andShelleyWoodVoiceInteractionD
esign:CraftingtheNewConversationalSpeechSystemsRandyAllenHarrisUnderstandingUser
s:APracticalGuidetoUserRequirements:Methods,Tools,andTechniquesCatherineCouragea
ndKathyBaxterTheWebApplicationDesignHandbook:BestPracticesforWeb-BasedSoftwareSu
sanFowlerandVictorStanwickTheMobileConnection:TheCellPhonesImpactonSocietyRichard
LingInformationVisualization:PerceptionforDesign,2ndEditionColinWareInteractionD
esignforComplexProblemSolving:DevelopingUsefulandUsableSoftwareBarbaraMirelTheCr
aftofInformationVisualization:ReadingsandReflectionsWrittenandeditedbyBenBederso
nandBenShneidermanHCIModels,Theories,andFrameworks:TowardsaMultidisciplinaryScie
nceEditedbyJohnM.CarrollWebBloopers:60CommonWebDesignMistakes,andHowtoAvoidThemJ
effJohnsonObservingtheUserExperience:APractitionersGuidetoUserResearchMikeKuniavs
kyPaperPrototyping:TheFastandEasyWaytoDesignandRefineUserInterfacesCarolynSnyder

HCIBeyondtheGUIDesignforHaptic,Speech,Olfactory,andOtherNontraditionalInterfaces
EditedbyPhilipKortum
AMSTERDAM BOSTON HEIDELBERGLONDONNEW YORK O XFORD PARIS SAN DIEGOSAN FRANCISCO Mo
rgan Kaufmann Publishers is an imprint of ElsevierSINGAPORE SYDNEY TOKYO

Publisher:DeniseE.M.PenrosePublishingServicesManager:GeorgeMorrisonProjectManage
r:MarilynE.RashAssistantEditor:MaryE.JamesCopyeditor:BarbaraKohlProofreader:Dian
neWoodIndexer:TedLauxCoverDesign:JayneJonesCoverDirection:AlisaAndreolaTypesetti
ng/IllustrationFormatting:SPiInteriorPrinter:SheridanBooksCoverPrinter:PhoenixCo
lorCorp.MorganKaufmannPublishersisanimprintofElsevier.30CorporateDrive,Suite400,
Burlington,MA01803Thisbookisprintedonacid-freepaper.Copyright#2008byElsevierInc.
Allrightsreserved.Designationsusedbycompaniestodistinguishtheirproductsareoftenc
laimedastrademarksorregisteredtrademarks.InallinstancesinwhichMorganKaufmannPubl
ishersisawareofaclaim,theproductnamesappearininitialcapitalorallcapitalletters.R
eaders,however,shouldcontacttheappropriatecompaniesformorecompleteinformationreg
ardingtrademarksandregistration.Nopartofthispublicationmaybereproduced,storedina
retrievalsystem,ortransmittedinanyformorbyanymeanselectronic,mechanical,photocopy
ing,scanning,orotherwisewithoutpriorwrittenpermissionofthepublisher.Permissionsma
ybesoughtdirectlyfromElseviersScience&TechnologyRightsDepartmentinOxford,UK:phone
:(44)1865843830,fax:(44)1865853333,e-mail:permissions@elsevier.com.Youmayalsocompl

eteyourrequeston-lineviatheElsevierhomepage(http://elsevier.com),byselectingSuppo
rt&ContactthenCopyrightandPermissionandthenObtainingPermissions.LibraryofCongressCata
loging-in-PublicationDataHCIbeyondtheGUI:designforhaptic,speech,olfactoryandothe
rnontraditionalinterfaces/editedbyPhilipKortum.p.cm.(TheMorganKaufmannseriesinint
eractivetechnologies)Includesbibliographicalreferencesandindex.ISBN-13:978-0-12374017-5(alk.paper)1.Human-computerinteraction.2.Graphicaluserinterfaces(Compute
rsystems)I.Kortum,Philip.QA76.9.H85H3972008005.437dc222007051584Forinformationonal
lMorganKaufmannpublications,visitourWebsiteatwww.mkp.comorwww.books.elsevier.com
.PrintedintheUnitedStates080910111254321

ContentsPrefaceixContributorsxi1IntroductiontotheHumanFactorsofNontraditionalInt
erfaces1PhilipKortum1.1StructureoftheBook11.2NontraditionalInterfaces31.3DesignP
rinciplesforNontraditionalInterfaces121.4TheFutureofNontraditionalInterfaceDesig
n18References232HapticInterfaces25MarciaK.OMalley,AbhishekGupta2.1NatureoftheInte
rface262.2TechnologyoftheInterface352.3CurrentInterfaceImplementations362.4Human
FactorsDesignofInterface512.5TechniquesforTestingtheInterface582.6DesignGuidelin
es602.7CaseStudies642.8FutureTrends64References653GestureInterfaces75MichaelNiel
sen,ThomasB.Moeslund,MoritzStorring,ErikGranum3.1Gestures753.2TechnologyandApplic
ability773.3FundamentalNatureoftheInterface803.4HumanFactorsInvolvedinInterfaceD
esign873.5DesignGuidelines94

3.6HowtoBuildandTestaGestureVocabulary983.7CaseStudy1023.8Summary1023.9FutureTre
nds103References1034LocomotionInterfaces107MaryC.Whitton,SharifRazzaque4.1Nature
oftheInterface1114.2TechnologyoftheInterface1174.3CurrentImplementationsoftheInt
erface1244.4HumanFactorsoftheInterface1284.5TechniquesforTestingtheInterface1324
.6DesignGuidelines1374.7CaseStudy1394.8FutureTrends141References1435AuditoryInte
rfaces147S.CamillePeres,VirginiaBest,DerekBrock,BarbaraShinn-Cunningham,Christop
herFrauenberger,ThomasHermann,JohnG.Neuhoff,LouiseValgerurNickerson,TonyStockman5
.1NatureoftheInterface1505.2TechnologyoftheInterface1565.3CurrentInterfaceImplem
entations1615.4HumanFactorsDesignofanAuditoryInterface1675.5TechniquesforTesting
theInterface1775.6DesignGuidelines1825.7CaseStudies1875.8FutureTrends187Referenc
es1896VoiceUserInterfaces197SusanL.Hura6.1AutomatedConversation:HumanversusMachi
ne1986.2TechnologyoftheInterface2086.3CurrentImplementationsoftheInterface:Onthe
Phone2136.4HumanFactorsDesignoftheInterface2146.5TechniquesforTestingtheInterfac
e217
Contentsvi

6.6DesignGuidelines2206.7CaseStudy2246.8FutureTrends224References2267Interactive
VoiceResponseInterfaces229JeffBrandt7.1NatureoftheInterface2297.2Technologyofthe
Interface2317.3CurrentImplementationsoftheInterface2327.4HumanFactorsDesignofthe
Interface2337.5TechniquesforTestingtheInterface2427.6DesignGuidelines2477.7CaseS
tudy2647.8FutureTrends264References2658OlfactoryInterfaces267YasuyukiYanagida8.1
NatureoftheInterface2678.2TechnologyoftheInterface2698.3CurrentImplementationsof
theInterface2718.4HumanFactorsDesignoftheInterface2838.5Interface-TestingTechniq
ues2858.6DesignGuidelines2868.7CaseStudies2898.8FutureTrends289References2899Tas
teInterfaces291HirooIwata9.1NatureoftheInterface2919.2TechnologyoftheInterface29
29.3CurrentImplementationsoftheInterface2939.4HumanFactorsDesignoftheInterface29
79.5TechniquesforTestingtheInterface3029.6DesignGuidelines3049.7CaseStudy3049.8F
utureTrends304References305

Contentsvii

10Small-ScreenInterfaces307DanielW.Mauney,ChristopherMasterton10.1NatureoftheInt
erface30710.2TechnologyoftheInterface31110.3CurrentImplementationsoftheInterface
31810.4HumanFactorsDesignoftheInterface32210.5TechniquesforTestingtheInterface33
910.6DesignGuidelines34310.7CaseStudy35110.8FutureTrends351References35411Multim
odeInterfaces:TwoorMoreInterfacestoAccomplishtheSameTask359AaronW.Bangor,JamesT.
Miller11.1NatureoftheInterface35911.2TechnologyoftheInterface36111.3CurrentImple
mentationsoftheInterface36311.4HumanFactorsDesignoftheInterface36911.5Techniques
forTestingtheInterface37711.6DesignGuidelines38111.7CaseStudy38611.8FutureTrends
386References38812MultimodalInterfaces:CombiningInterfacestoAccomplishaSingleTas
k391PauloBarthelmess,SharonOviatt12.1NatureoftheInterface39112.2TechnologyoftheI
nterface39412.3CurrentImplementationsoftheInterface40712.4HumanFactorsDesignofth
eInterface41512.5TechniquesforTestingtheInterface42312.6DesignGuidelines42612.7C
aseStudies43012.8FutureTrends430References432Index445
Contentsviii

PrefaceThecomputerrevolutionandthegraphicaluserinterfaces(GUIs)itusheredinhashel
peddefinetheworkofagenerationofhumanfactorsprofessionals.TheadventoftheInternete
stablishedthestandardGUIasoneoftheprimaryinter-facesthatbothusersanddesignersmus
tdealwith.Yet,despitetheubiquityoftheGUI,nontraditionalinterfacesabound,andarein
factsignificantlymorecom-monthanwemightfirstthink.Fromtheoft-reviledinteractivev
oiceresponsesys-temtothesmall-screeninterfacesonourcellphones,thesenontraditiona
linterfacesplayahugeroleinoureverydaylives.Thisbookwasbornoutofadesiretocollectt
hefundamentalwisdomthatmightbeneededtodothehumanfactorsworkonavarietyofnon-GUIin
terfacesintoasinglereferencesourceforpracticinghumanfactorsprofessionalsandtogiv
estudentsofpsychologyandengineeringanopportunitytobeexposedtothehumanfactorsfort
hemultitudeofnon-GUIinterfacesthattheywillmostlikelybeworkingonintherealworld.It
ismyhopethatthisbookservesbothofthesegroups.First,thechaptersarestructuredsoasto
providetheseasonedhumanfactorsprofessionalwithareadyreferencesourceforthoseoccas
ionswhentheprojectdemandsaninterfacethatisoutsidethecommonGUI.Theinclusionofthed
esignguidelinesandtheonlinecasestudieswasspecificallyintendedtogivethepracticing
humanfactorsprofes-sionaluseful,practicaladviceonimplementation.Second,thebookha
salsobeendesignedtobeusedasateachingtextforupper-divisionundergraduatesandgrad-u
atestudents,servingasanintroductiontothemanyfascinatinginterfacesthatexistbeyond
therealmofthewell-coveredGUI.Thediscussionoftheunderlyingtechnologies,thecurrent
implementationsandthefundamentalhumanfactorsoftheinterfacehavebeenwrittentohelpt
hestudentunderstandthenutsandboltsofeachinterfaceandgainanappreciationoftheroleoft
hehumanfactorsengineerinitsdesign.

AcknowledgmentsAswithanysuchendeavor,therearemanypeoplewhoplayedanimportantrolei
nhelpingtheprojectcometofruition.First,thankstomyfriendsandcolleagueswhocontribu
tedtothebookwithouttheirdedicatedeffortsandexpertise,thisbookwouldnotexist.Iwould
alsoliketothankmyeditorsatMorganKaufmann,DenisePenrose,MaryJames,andAsmaPalmeiro
,fortheirunendingpatienceinhelpingtomakethisbookareality.ArnieLund,CarolineJarre
tt,GavinLew,ChristineAlverado,andRandolphBiasprovidedenormouslyhelpfulreviews,an
dthebookisbetterfortheirsubstantialandcopiouscommentsontheearlyversions.Finally,
IwouldliketothankMichaelRiley,myfirsthumanfactorsprofessorattheUniversityofNebra
ska,forsparkingmyloveofhumanfactorsasadiscipline.DedicationToRebecca.

Prefacex

ContributorsAaronBangor,AT&TLaboratories,Austin,TX(bangor@labs.att.com)Bangorisa
principalmemberoftechnicalstaffatAT&TLabs,Inc.,inAustin.Hehasworkedonawidevariet
yofuserinterfacedesigns,includingapplicationsthathavemultipleinterfacesofdiffere
ntmodalities.HeearnedaPh.D.inhumanfac-torsengineeringfromVirginiaTechandisacerti
fiedhumanfactorsprofessional.BangorservesontheTexasGovernorsCommitteeonPeoplewith
Disabilities.HeisalsoactiveintheHumanFactorsandErgonomicsSociety,includingparted
itoroftheforthcomingAmericannationalstandard:HumanFactorsEngineeringofSoftwareUs
erInterfaces.(Chapter11)PauloBarthelmess,Adapx,Seattle,WA(Paulo.Barthelmess@natu
ralinteraction.com)Barthelmessisaresearchscientistworkingwithcollaborationtechno
logyatAdapx.Hisresearchinterestsareinhuman-centeredmultimodalsystems,explor-ingi
ntelligentinterfacestofacilitatetheworkofco-locatedordistributedgroupsofpeople.H
iscurrentfocusisonsupportingcollaborativedocument-centeredannotationworkusingdig
italpaper.Barthelmesshasanextensivesoftwareengi-neeringbackground,havingworkedin
industryinmanycapacitiesforover20years.HereceivedaPh.D.incomputersciencefromtheU
niversityofColoradoatBoulder.(Chapter12)VirginiaBest,BostonUniversity,Boston,MA(
ginbest@cns.bu.edu)BeststudiedmedicalscienceattheUniversityofSydney,andreceivedh
erPh.D.in2004afterspecializinginhumanauditoryspatialperception.Shethenworkedasar
esearchassociateatBostonUniversity,wheresheexaminedtheroleofspa-tialhearinginrea
listicmultiple-sourceenvironments,anddevelopedaninterestinhowspatialhearingisaff
ectedbyhearingimpairment.In2008,shewillcon-tinueherworkonhearingimpairmentasares
earchfellowattheUniversityofSydney.(Chapter2)

JeffBrandt,AT&TLaboratories,Austin,TX(brandt@labs.att.com)Brandtbeganhiscareerwi
ththeAT&TLabsHumanFactorsGroupin1996,ensuringthatnewproductsandservicesareuseful
toandusablebyAT&Tscus-tomers.HemanagestheAustinHumanFactorsLaboratoryfacilitiesan
dperformsinterfacedesignandusabilitytestingforInternetProtocolTelevisionapplicat
ions.Pastprojectsincludedisasterrecovery,privacymanagement,outgoingcallcon-trol,
voicedial,unifiedcommunications,andbillformatting.Brandtholds5patentsandhas43pat
entspending.HeearnedtheM.S.inindustrialengineeringfromtheUniversityofWashingtona
ndB.S.incognitive/experimentalpsychologyfromOregonStateUniversity.(Chapter7)Dere
kBrock,IntelligentSystemsSection,NavyCenterforAppliedResearchinArtificialIntelli
gence,U.S.NavalResearchLaboratory,Washington,DC(derek.brock@nrl.navy.mil)Brockis
acomputerscientistattheU.S.NavalResearchLaboratorysCenterforAppliedResearchinArti
ficialIntelligence.Hisworkinvolvestheapplicationofauditorydisplay,cognitivearchi
tectures,andmodelsofhumanlanguageusetothedesignofcollaborativeinterfacesfordeskt
op,immersive,mobile,androboticsystems.HeholdsB.S.andM.S.degreesincomputerscience
andcomputergra-phicsandmultimediasystemsfromGeorgeWashingtonUniversity.Brockisam
emberoftheAcousticalSocietyofAmerica(ASA),CognitiveScienceSociety,Associationfor
theAdvancementofArtificialIntelligence(AAAI),andInterna-tionalCommunityforAudito
ryDisplay(ICAD).(Chapter5)ChristopherFrauenberger,DepartmentofComputerScience,Qu
eenMary,UniversityofLondon,London,UK(frauenberger@dcs.qmul.ac.uk)Frauenbergerisa
Ph.D.studentintheInteractionMediaCommunicationGroupattheDepartmentofComputerScie
nce,QueenMary,UniversityofLondon.Hisresearchfocusesonalternativemodesofinteracti
ngwithtechnologywithaspecialinterestinthedesignofauditorydisplays.Since2006,heis
amemberoftheboardoftheInternationalCommunityforAuditoryDisplayandcontributestowa
rdestablishingaudioandsoundasahighlyefficientalternativeforhumancomputerinteracti
ondesigners.(Chapter5)ErikGranum,DepartmentofMediaTechnologyandEngineeringScienc
eatAalborgUniversity,Aalborg,Denmark(eg@vision.auc.dk)Granumisaprofessorofinform
ationsystemsandheadoftheDepartmentofMediaTechnologyandEngineeringScienceatAalbor
gUniversity,Denmark.Hisinterestscoverpatternrecognition,continuallyoperatingvisi
onsystems,motionanalysis,colorvision,multimediainterfaces,visualization,virtualr
eality,

Contributorsxii

andcreativeuseofmediatechnology.Hehasbeencoordinatorandpartnerofarangeofnational
andinternationalresearchprojectsandnetworksincomputervision,mediatechnologies,an
dvirtualreality.Hewasamajorcontributorintheestablishmentofamultimediaandvirtualr
ealitycenteratAalborgUniversity,andpursuesinterdisciplinaryeducationsandresearch
.(Chapter3)AbhishekGupta,RiceUniversity,Houston,TX(abhi@rice.edu)Guptareceivedth
ebacheloroftechnology(honors)degreeinmechanicalengi-neeringfromtheIndianInstitut
eofTechnology,Kharagpur,andtheM.S.degreeinmechanicalengineeringfromRiceUniversit
yin2004,whereheiscurrentlyadoctoralstudent.Hiscurrentresearchinterestsincludedes
ignandcontrolofhap-ticinterfaces,nanoroboticmanipulationwithhapticfeedback,andro
bot-assistedtrainingandrehabilitationinvirtualenvironments.(Chapter2)ThomasHerma
nn,NeuroinformaticsGroup,FacultyofTechnology,BielefeldUniversity,Bielefeld,Germa
ny(thermann@techfak.uni-bielefeld.de)HermannstudiedphysicsandreceivedaPh.D.incom
puterscienceatBielefeldUniversityin2002.HeisaresearchprofessoratBielefeldUnivers
itywherehelaunchedtheresearchonsonification.HermannservesasmemberoftheInterna-ti
onalCommunityforAuditoryDisplay(ICAD)boardofdirectorsandisGermandelegateandvicec
hairoftheEUCOSTActionIC0601(SID,sonicinteractiondesign).Heisinitiatorandorganize
roftheInternationalWorkshoponInteractiveSonificationandguesteditorofanIEEEMultim
ediaspecialissueoninteractionsonification.Hisresearchfieldsaresonification,datam
ining,andhumancomputerinteraction.(Chapter5)SusanL.Hura,SpeechUsability,Cumming,G
A(susan.hura@speechusability.com)HuraisthefounderofSpeechUsability,aconsultancyf
ocusedonimprovingcus-tomerexperiencebyincorporatinguser-centereddesignpracticesi
nspeechtech-nologyprojects.ShefoundedtheusabilityprogramatIntervoice,andpriortot
hatworkedamemberofthehumanfactorsteamatLucentTechnologies.AsfacultymemberatPurdu
eUniversity,shecofoundedamultidisciplinaryteamresearchingnovelapproachestospeech
recognition.HuraholdsaPh.D.inlinguisticsfromtheUniversityofTexasatAustin.Sheserv
edasco-chairofSpeechTEK2007and2008,andisamemberoftheboardofdirectorsofAVIOS.(Cha
pter6)HirooIwata,GraduateSchoolofSystemsandInformationEngineering,UniversityofTs
ukuba,Tsukuba,Japan(iwata@kz.tsukuba.ac.jp)IwataisaprofessorintheGraduateSchoolo
fSystemsandInformationEngineer-ing,UniversityofTsukuba.Hisresearchinterestsinclu
dehapticinterfaces,
Contributorsxiii

locomotioninterfaces,andspatiallyimmersivedisplays.IwatareceivedtheB.S.,M.S.,and
Ph.D.degreesinengineeringfromtheUniversityofTokyo.Heisafound-ingmemberoftheVirtu
alRealitySocietyofJapan.(Chapter9)PhilipKortum,RiceUniversity,Houston,TX(pkortum
@rice.edu)KortumiscurrentlyafacultymemberintheDepartmentofPsychologyatRiceUniver
sityinHouston.PriortojoiningRice,heworkedforalmostadecadeatSBCLaboratories(nowAT
&TLaboratories)doinghumanfactorsresearchanddevelopmentinallareasoftelecommunicat
ions.Kortumcontinuestodoworkintheresearchanddevelopmentofuser-centricsystemsinbo
ththevisual(webdesign,equipmentdesign,andimagecompression)andauditorydomains(tel
e-phonyoperationsandinteractivevoiceresponsesystems).HereceivedhisPh.D.fromtheUn
iversityofTexasatAustin.(Chapter1)MarciaOMalley,RiceUniversity,Houston,TX(omalley
m@rice.edu)OMalleyreceivedtheB.S.degreeinmechanicalengineeringfromPurdueUniver-si
ty,andtheM.S.andPh.D.degreesinmechanicalengineeringfromVanderbiltUniversity.Herc
urrentresearchinterestsincludenanoroboticmanipulationwithhapticfeedback,hapticfe
edbackandsharedcontrolbetweenroboticdevicesandtheirhumanusersfortrainingandrehab
ilitationinvirtualenvironments,andeducationalhaptics.Sheisco-chairoftheASMEDynam
icSystemsandControlsDivisionRoboticsTechnicalCommittee,andamemberoftheIEEETechni
calCommitteeonHaptics.(Chapter2)ChrisMasterton,OptimalInterfaces,Cary,NC(chris@o
ptimalinterfaces.com)Mastertonhasbeenapracticinginteractiondesignerandusabilitys

pecialistformorethan8years.Hisbroaduserinterfacedesignexperienceincludeslargee-c
ommercewebsitesforclientslikeIBMandLloydsofLondon;interactivesitesforTribalDDB,T
ourismBritishColumbia,OntarioTourism;mobilephoneinter-facedesignforNokiaandMotor
ola;andusabilitytestingforDirectTV,Clorox,Intrawest,andtheUniversityofMinnesota,
amongothers.In1997,Chrisachievedhisbachelorsdegreeincognitivesciencewithacertific
ateincomputingsciencefromSimonFraserUniversity.Forthepast7years,Chrishasalsobeen
theinstructorforuserinterfacedesignattheUniversityofBritishColumbiassoft-wareengi
neeringcontinuingstudiesprogram.(Chapter10)DanMauney,HumanCentric,Vancouver,BC,C
anada(dmauney@humancentrictech.com)Mauneyisa14-yearveteraninthewirelesstelecommu
nicationshumanfactorsprofession.Hehasdevelopedabroadviewandunderstandingofthewir
eless
Contributorsxiv

telecommunicationsmarketbyworkingdirectlyforamajorNorthAmericanwire-lessoperator
(SBCWireless,nowAT&T),amajorhandsetmanufacturer(Nokia),acontentprovider(Mobileum
),awirelessaccessorymanufacturer(JabraCorpora-tion),andcurrentlyforaserviceprovi
derspecializinginthewirelesstelecommu-nicationsfield(HumanCentricTechnologies).A
tHumanCentricTechnologies,Mauneyleadsateamofhumanfactorsprofessionalsspecializin
ginhelpingcli-entswithsmallscreendesignandevaluation.HeholdsaPh.D.andM.S.inindus
trialengineeringandhumanfactorsfromVirginiaTech.(Chapter10)JamesT.Miller,AT&TLab
oratories,Austin,TX(miller@labs.att.com)MillerisaprincipalmemberoftheTechnicalSt
affatAT&TLabs,Inc.Heispri-marilyresponsibleforthedevelopment,testing,andevaluati
onofwebpagesthatpresentconsumerandbusinessproductsforsaleandthatprovideonlinesup
portforthoseproducts.Inaddition,heisalsoresponsibleforthedevelopmentofinter-acti
vevoiceresponsesystems,includingsomethatusespeechrecognition.MillerearnedhisPh.D
.fromtheUniversityofColoradoatBoulder.(Chapter11)ThomasB.Moeslund,LaboratoryofCo
mputerVisionandMediaTechnology,AalborgUniversity,Aalborg,Denmark(tbm@cvmt.dk)Moe
slundisanassociateprofessorattheComputerVisionandMediaTechnologylabatAalborgUniv
ersity,Denmark.HeobtainedhisM.S.andPh.D.degreesin1996and2003,respectively,bothfr
omAalborgUniversity.Heisactivelyinvolvedinbothnationalandinternationalresearchpr
ojects,andiscurrentlycoordinatinganationalprojectandworkpackageleaderinaninterna
tionalproject.Hispri-maryresearchinterestsincludevisualmotionanalysis,patternrec
ognition,inter-activesystems,computergraphics,multimodalsystems,andmachinevision
.Moeslundhasmorethan70publicationsintheseareas.(Chapter3)JohnNeuhoff,Departmento
fPsychology,TheCollegeofWooster,Wooster,OH(jneuhoff@wooster.edu)Neuhoffisamember
oftheboardofdirectorsfortheInternationalCommunityforAuditoryDisplay(ICAD).Heplay
sthesaxophoneandteachesauditorydisplayandcognitivescienceatTheCollegeofWooster.H
isworkhasbeenpublishedinNature,Science,theProceedingsoftheNationalAcademiesofSci
ence,andhehaseditedabookonecologicalpsychoacoustics.HehasreceivedgrantsfromtheNa
tionalScienceFoundation,andtheNationalInstituteforOccupationalSafetyandHealth.Hi
ssaxophonecareerhasyettoblossom.(Chapter5)MichaelNielsen,LaboratoryofComputerVis
ionandMediaTechnology,AalborgUniversity,Aalborg,Denmark(mnielsen@cvmt.dk)
Contributorsxv

NielsenisanassistantprofessorinthestudyofmediaatAalborgUniversity.HisPh.D.thesis
wasfocusedonthree-dimensionalreconstruction-basedsensorsinprecisionagriculture,a
ndhehasalsoworkedwithgestureinterfacesandshadowsegmentation.Researchinterestsinc
ludeaspectsofmediatechnologysuchasinterfacedesign,games,camera-basedinterfaces,c
olorandlighttheory,andshadowsegmentation.(Chapter3)SharonOviatt,Adapx,Seattle,WA
(sharon.oviatt@adapx.com)OviattisadistinguishedscientistatAdapxandpresidentofInc
aaDesigns.Herresearchfocusesonhuman-centeredinterfacedesignandcognitivemodeling,
communicationtechnologies,spokenlanguage,pen-basedandmultimodalinter-faces,andmo

bileandeducationalinterfaces.Shehaspublishedover120scientificarticlesinawiderang
eofvenues,includingworkfeaturedinrecentspecialissuesofCommunicationsoftheACM,Hum
anComputerInteraction,TransactionsonHumanComputerInteraction,IEEEMultimedia,Procee
dingsofIEEE,andIEEETransactionsonNeuralNetworks.Shewasfoundingchairoftheadvisory
boardfortheInternationalConferenceonMultimodalInterfacesandGeneralChairoftheICMI
Conferencein2003.In2000,shewastherecipientofaNationalScienceFoundationCreativity
Awardforpioneeringworkonmobilemultimodalinter-faces.(Chapter12)S.CamillePeres,Ps
ychologyDepartment,UniversityofHouston-ClearLake,Houston,TX(peresSC@uhcl.edu)Per
esiscurrentlyanassistantprofessorinpsychologyattheUniversityofHouston-ClearLake.
Herresearchisgenerallyfocusedonthecognitivemechan-ismsassociatedwiththeacquisiti
onofnewskills,andspecificallymechanismsassociatedwithacquisitionanduseofefficien
tmethods,optimaldesignsforinter-activeauditorydisplays,andincorporationofsimulat
ionsintheteachingofstatis-tics.PeresreceivedherPh.D.inpsychologyfromRiceUniversi
tywithafocusonhumancomputerinteraction.(Chapter5)SharifRazzaque,ComputerScience,U
niversityofNorthCarolina,ChapelHill,NC(sharif@cs.unc.edu)Razzaqueisaresearchscie
ntistforInnerOpticTechnology,wherehedevelopsaugmented-realitysurgicaltoolsforsol
vingspatialcoordinationproblemsfacedduringsurgery.HereceivedhisPh.D.incomputersc
ienceattheUniversityofNorthCarolinaatChapelHillforworkinvirtualenvironmentlocomo
tioninter-faces.Hehaspreviouslyworkedonhapticinterfaces,physiologicalmonitoring,
Contributorsxvi

medicalimaging,collaborativesatelliteengineeringtooldevelopmentatLock-heedMartin,
andcochlearimplantsattheUniversityofMichigan.(Chapter4)BarbaraShinn-Cunningham,D
epartmentsofCognitiveandNeuralSystemsandBiomedicalEngineering,DirectorofCNSGradu
ateStudies,BostonUniversity,Boston,MA(shinn@cns.bu.edu)Shinn-Cunninghamisanassoc
iateprofessorincognitiveandneuralSystemsandbiomedicalengineeringatBostonUniversi
ty.Herresearchexploresspatialhearing,auditoryattention,auditoryobjectformation,e
ffectsofreverberantenergyonsoundlocalizationandintelligibility,perceptualplastic
ity,andotheraspectsofauditoryperceptionincomplexlisteningsituations.Shinn-Cunnin
ghamisalsoengagedincollaborativestudiesexploringphysiologicalcorrelatesofaudi-to
ryperception.ShereceivedtheM.S.andPh.D.inelectricalengineeringandcomputerscience
fromtheMassachusettsInstituteofTechnology.(Chapter5)TonyStockman,DepartmentofCom
puterScience,UniversityofLondon,London,UK(tonys@dcs.qmul.ac.uk)Stockmanisasenior
lectureratQueenMary,UniversityofLondon,andaboardmemberoftheInternationalCommunit
yforAuditoryDisplay(ICAD).Hefirstemployeddatasonificationtoassistintheanalysisof
physiologicalsignalsduringhisdoctoralresearchinthemid-1980s.Hehasover30yearsofex
perienceasaconsultantanduserofassistivetechnologyandhaspublishedover30papersonau
ditorydisplaysanddatasonification.(Chapter5)MoritzStorring,ICOSVisionSystems,Belg
ium(moritz.storring@icos.be)StorringstudiedelectricalengineeringattheTechnicalUniv
ersityofBerlin,andattheInstitutNationalPolytechniquedeGrenoble,France,andreceive
dthePhDfromAalborgUniversity,Denmark.AsanassociateprofessoratAalborgUniver-sity,
hisresearchinterestsincludedphysics-basedcolorvision,outdoorcomputervision,visio
n-basedhumancomputerinteraction,andaugmentedreality.In2006,Storringmovedtoindustry
whereheisfocusedonautomaticvisualinspec-tionofelectroniccomponentsandintellectua
lpropertyrightsIPR.(Chapter3)LouiseValgerurNickerson,DepartmentofComputerScience,
QueenMary,UniversityofLondon,London,UK(lou@dcs.qmul.ac.uk)ValgerurNickersonisaPh.
D.studentatQueenMary,UniversityofLondon,intheDepartmentofComputerScience.Herwork
focusesondevelopingauditoryoverviewsusingnonspeechsoundforthevisuallyimpairedand
formobileandwearablecomputing.SheholdsaB.A.inFrenchandItalianLanguageand
Contributorsxvii

LiteraturefromtheUniversityofVirginia,andaM.S.inadvancedmethodsincomputerscience

fromQueenMary.(Chapter5)MaryC.Whitton,ComputerScience,UniversityofNorthCarolina,
ChapelHill,NC(whitton@cs.unc.edu)WhittonisaresearchassociateprofessorintheDepart
mentofComputerScience,UniversityofNorthCarolinaatChapelHill.Shehasbeenworkinginh
igh-performancegraphics,visualization,andvirtualenvironmentssinceshecofoundedthe
firstofhertwoentrepreneurialventuresin1978.AtUNCsince1994,Whittonsresearchfocuses
onwhatmakesvirtualenvironmentsystemseffectiveandondevelopingtechniquestomakethem
moreeffectivewhenusedinapplicationssuchassimulation,training,andrehabilitation.S
heearnedM.S.degreesinguidanceandpersonnelservices(1974)andelectricalengineering(
1984)fromNorthCarolinaStateUniversity.(Chapter4)YasuyukiYanagida,DepartmentofInf
ormationEngineering,FacultyofScienceandTechnology,MeijoUniversity,Nagoya,Japan(y
anagida@ccmfs.meijo-u.ac.jp)YanagidaisaprofessorintheDepartmentofInformationEngi
neering,FacultyofScienceandTechnology,MeijoUniversity.HereceivedhisPh.D.inmathem
aticalengineeringandinformationphysicsfromtheUniversityofTokyo.Yanagidawasaresea
rchassociateattheUniversityofTokyoandaresearcheratAdvancedTelecommunicationResea
rchInstituteInternationalbeforehemovedtoMeijoUniversity.Hisresearchinterestsincl
udevirtualreality,telexistence,anddisplaytechnologiesforvarioussensorymodalities
.(Chapter8)
Contributorsxviii

1CHAPTERIntroductiontotheHumanFactorsofNontraditionalInterfacesPhilipKortum1.1ST
RUCTUREOFTHEBOOKAshumanfactorsprofessionals,wearetrainedintheartofinterfacedesig
n.However,moreandmoreofthattraininghascenteredoncomputerinterfaces.Morespecifica
lly,ithasfocusedonthegraphicaluserinterfaces(GUIs)thathavebecomeubiquitoussincet
headventofthecomputer.WhiletheGUIremainsthemostcommoninterfacetoday,ahostofother
interfacesarebecomingincreasinglyprevalent.HCIBeyondtheGUIdescribesthehumanfacto
rsofthesenontraditionalinterfaces.Ofcourse,thedefinitionofanontraditionalinterface
isratherarbitrary.Forthisbook,Iattemptedtoselectinterfacesthatcoveredallofthehum
ansenses,andincludednontraditionalinter-facesthatarewidelyused,aswellasthosethat
aresomewhat(ifnottotally)neglectedinmostmainstreameducationprograms.Manyofthesei
nterfaceswillevokeastrongwowfactor(e.g.,tasteinterfaces)sincetheyareveryrare,andco
mmercialapplicationsarenotgenerallyavailable.Others,suchasinteractivevoicerespon
seinterfaces,maynotseemasexciting,buttheyareincrediblyimportantbecausetheyarewid
elydeployed,andgenerallyverypoorlydesigned,anditislikelythateveryhumanfactorspro
fessionalwillbeaskedtoworkononeoftheseduringthecourseofhercareer.Thisbookbringst
ogetherthestateoftheartinhumanfactorsdesignandtestingof11majornontraditionalinte
rfaces,andpresentstheinformationinawaythatwillallowreaderswhohavelimitedfamil-ia
ritywiththeseinterfacestolearnthefundamentalsandseehowtheyareputintoactioninther
ealworld.Eachchapterinthebookisstructuredsimilarly,coveringthemostimportantinfor
mationrequiredtodesign,build,andtesttheseinterfaces.Specifically,eachchapterwill
addressthefollowingaspects.

Natureoftheinterface:Eachchapterbeginswithadescriptionofthefundamentalnatureofth
einterface,includingtheassociatedhumanperceptualcapabilities(psychophysics).Whil
ethedetailsofthesediscussionsmayseemunimportanttothepractitionerwhosimplywantsto
buildaninterface,anunderstandingofpertinenthumanstrengthsandlimitations,bothcogn
itiveandperceptual,iscriticalincreatingsuperiorinterfacesthatareoperationallyrob
ust.Interfacetechnology:Aswithanyinterface,technologyisoftenthelimitingfac-tor.S
omeoftheinterfacesdescribedinthisbookuseverymaturetechnology,whileothersareonthe
cuttingedgeoftheresearchdomain.Ineithercase,adetaileddescriptionofthetechnologie
susedandtheirappropriateimplementa-tionsareprovidedsothatthepractitionercanspeci
fyandconstructbasicinterfaces.Currentimplementationsoftheinterface:Thissectionde
scribeshowandwheretheinterfaceisusedtoday.Examplesofsuccessfulimplementationsfor
eachinterfacearegiven,aswellasexamplesoffailures(whereappropriate),whichcanbever

yinstructive.Anothertopicthatisincludedinthissectionisadiscus-sionoftheinterfaces
applicationtoaccessibility.Manyoftheseinterfacesareofspecialinterestbecausecerta
inimplementationsprovidecrucialinterfacesforpeoplewithphysicalorcognitivedisabil
ities.Forexample,Brailleisalow-techhapticinterfacethatallowsblinduserstoread.Thi
ssectionbrieflydiscussesthebenefitsofusingthetechnologytoassistindividualswhohav
ephysicalorcognitiveimpairments,andprovidesexamplesofspecialimplementationsofthe
technologyforsuchusers.Ifuseoftheinterfacehasanyspecialadversecon-sequencesforth
edisabledpopulation,thesearenotedaswell.Humanfactorsdesignoftheinterface:Thissec
tionwilltellyou,asthehumanfactorsdesigner,whatyoushouldbeconsideringasyouembarko
nthedesignorevaluationofagivennontraditionalinterface.Itdiscusseswhentoselectapa
rticularinterface,thedatarequiredtobuildtheinterface,anddetailsonwhatahumanfacto
rsprofessionalwouldneedtoknowinordertospecifysuchaninterfaceforuse.Techniquesinv
olvedintestingtheinterface:Specialinterfacesusuallyrequirespecialtestingmethodol
ogies.Thissectiondescribesspecialtestingcon-siderationsfortheinterface,specialte
chnologyorproceduresthatmightberequired,andmethodsofdataanalysisiftheyaresuffici
entlydifferentfromstandardanalysismethods.Evenifstandardtestingmeasuresareused,a
descriptionoftheseandwhentheyshouldbeappliedisincludedtoguidethepractitioner.Spe
cialattentionispaidtotheconceptofiterativetestingifitisapplicabletothespecificin
terface.Designguidelines:Forexperienceddesigners,guidelinescanappeartobetoosim-p
listicandinflexibletobeofanypracticalvalue.However,forthebeginningdesigner,theys
erveasaninvaluablewaytogenerateafirst-generationdesign
1IntroductiontotheHumanFactorsofNontraditionalInterfaces2

whileleveragingtheknowledgeofexpertdesigners.ItisinthisspiritthattheDesignGuidel
inessectionofeachchapterprovidessomeofthemostimportantlessonsthatshouldbeapplied
.Theguidelinespresentedforeachinterfacearenotmeanttobeexhaustiveandinclusive.Rat
her,thegoalofthissectionistolistthetop5to10itemsthatanexpertwouldpassalongtosome
onewhowaslookingforimportantadviceaboutthehumanfactorsimplementationoftheinterfa
ce.Casestudyofadesign:Thissectionpresentsacasestudyofthehumanfactorsspecificatio
n/implementation/evaluationoftheinterfaceoveritslifecycle.Wherepractical,thecase
studyisareal-worldimplementation.Forcertaininterfaces,however,proprietaryconside
rationsdictatedchangesinnames,dates,andidentifyingdetailstomasktheidentityofthei
nterface.Insomecases,theexamplehasbeenmadestrongerthoughtheuseofmultipleimple-me
ntationsratherthanasingle,lifecyclecasestudy.Futuretrends:Sincethefocusisonnontr
aditionalinterfaces,mostarestillevol-vingastechnologychangesandasusers(anddesign
ers!)becomemorefamiliarandcomfortablewiththeiruse.Thissectiondescribesthefutureo
ftheinterfaceinthenext10to20years.Whereistheinterfaceheaded?Howwillcurrentimplem
entationschange?Willcurrentimplementationssurviveorbesup-plantedbynewinnovations
?Whatistheendstateoftheinterfacewhenitisfullymature?Inthissection,theauthorsareg
ivenachancetospeculatehowaparticularinterfacewillmatureovertime,andwhatuserscanl
ookforwardto.Theauthorsofcertainchapters,particularlythosefocusedoninterfacestha
tusesound,haveprovidedaccesstoexamplesthatyoucanlistentobyvisitingthebookswebsite
atwww.beyondthegui.com.Thiswebsitealsocontainscasestudiesforeachinterface.Thesec
asestudiesprovideexamplesofhowtheinter-faceshavebeenimplemented,andhowhumanfacto
rscontributedtothoseimplementations.1.2NONTRADITIONALINTERFACESScarcityofimpleme
ntationwasnottheprimaryfactorindeterminingtheinter-facestobeincludedinthisbook,a
smanyofthemarenearlyubiquitous.Others,suchastasteinterfaces,arequiterare.Further
,eventhoughthenameofthebookisHCIBeyondtheGUI,severalchaptersdo,infact,dealwithGU
Is,butinaformthatmostdesignershavelittleexperiencewith(see,forinstance,Chapter10
onsmall-screendesign).The11interfacesselectedforinclusionrepresentthemostimporta
ntnontraditionalinterfacesthatahumanfactorsprofessionalshouldknowandunderstand.
1.2NontraditionalInterfaces3

1.2.1HapticUserInterfacesHapticinterfacesusethesensationoftouchtoprovideinformat
iontotheuser.Ratherthanvisuallyinspectingavirtualthree-dimensionalobjectonacompu
termonitor,ahapticdisplayallowsausertophysicallytouchthatobject.Theinter-facecanal
soprovideinformationtotheuserinotherways,suchasvibrations.Ofcourse,thegamingindu
stryhasledthewayinintroducingmanyofthesenon-traditionalinterfacestothegeneralpub
lic.Variousinterfacetechnologieshaveheightenedtherealismofgameplayandmakethegame
easierandmorecompel-ling.Oneoftheearlyinterfacestotakeadvantageofhapticscanbefou
ndinAtarisSteelTalonssit-downarcadegame(Figure1.1).Thegamewasfuntoplaybecausethec
ontrolswerereasonablyrealisticandtheactionwasnonstop;however,unlikeothercontempo
raryfirst-personshooter
FIGURE1.1AtarisSteelTalonshelicoptersimulation,circa1991.Whilethegraphicswereunre
markable(shadedpolygons),thegameemployedahapticinterfaceintheplayersseat(asindica
tedbythearrow)thatthumpedtheplayer(hard!)whenthehelicopterwasbeinghitbygroundfir
e.Theaddedinterfacedimensioncausedtheplayertoreactinmorerealisticwaystothethreatan
dmadetheinformationmoresalient.Source:Retrievedfromwww.mame.net.
1IntroductiontotheHumanFactorsofNontraditionalInterfaces4

games,Atariintegratedahapticfeedbackmechanismthatwasactivatedwhentheusershelicopt
erwashitbyenemyfire.Othercontemporarygamesusedsoundsandchangesinthegraphicalinterf
ace(flashing,bulletholes)toindicatethattheuserwastakingenemyfire.Atariintegrated
whatcanbestbedescribedasaknockerintheseatofthegame.Similarinsoundandfeeltothedevic
einpinballmachinesthatisactivatedwhentheuserwinsafreegame,thishapticinterfacewas
botheffectiveandcompelling.Althoughtheinformationprovidedtoplayerswasidenticalto
thatpresentedviasoundandsight,theyreactedtoitdifferentlytheirresponsewasevocative
ofthefight-or-flightresponseseenintherealworld,andtheyweremorereluctanttojustplay
throughthewarnings,asissooftenseeninstrictlyvisualgames.Byselectingtherightinterf
acetypefortheinformationthatmustbepre-sented,thedesignerscreatedabetterinterface
.Oncethesolepurviewofhigh-endsimulatorsandarcadegames,hapticscannowbefoundinhome
gameconsolesaswell(e.g.,NintendosRumblePac).Althoughgenerallynotassophisticatedor
realisticastheirmoreexpensivecounterparts,theuseofvibrationinthehandcontrollerpr
ovidestheplayerwithextrainformationabouttheenvironmentandgameplaythatwasnotprevi
ouslyavailable.Otherexamplesofcompellingimplementationsofthehapticinterfacecanbe
foundininterfacesasdiverseasautomobileantilockbrakingfeedbacksystemsandthreatide
ntificationsystemsforsoldiers.Chapter2willaddresstheentirespectrumofhapticinterf
aces,fromsimpleimplementations,suchasvibratingmobilephoneringers,tosomeofthemost
sophisticatedvirtual-touchsurgicalsimulators.1.2.2GestureInterfacesGestureinterf
acesusehandandfacemovementsasinputcontrolsforacomputer.Althoughrelatedtohapticin
terfaces,gestureinterfacesdifferinthenotedabsenceofmachine-mediatedproprioceptiv
eortactilefeedback.Thesimplestformofgestureinterfacescanbefoundinmotion-activate
dlightsthelightinter-pretstheusersmotionasthesignalthatitshouldturnitselfon.Otherc
ommercialimplementationsofgestureinterfaceshaverecentlybeguntomaketheirwayintoth
egameworldaswell.In2001,KonamireleasedagamecalledMoCapBoxing.Unlikeearlierver-si
onsofboxinggamesthatwerecontrolledwithjoysticksandbuttons,Konamisgamerequiredthep
layertoactuallybox.Theplayerdonnedglovesandstoodinaspecifiedareathatwasmonitored
withinfraredmotiondetectors.Bymovingandboxing,theplayercouldhittheopponent,duckthe
opponentshits,andpro-tecthisbodybysimplyreplicatingthemovesarealboxerwouldmake.Fi
gure1.2showsthegameinaction.Thistechnology,too,hasfounditswayintothehomewithther
ecentintroductionofNintendosWiisystem.Unlikeothercurrenthomegaming
1.2NontraditionalInterfaces5

systems,Wiimakesextensiveuseofthegestureinterfaceinavarietyofgames,frombowlingto
tennis,allowingtheusertointeractinamorenaturalstylethanpreviouslywheninteraction
wascontrolledviabuttonsinterfacedtotheGUI.Notonlyistheinterfacemorenaturalandapp
ropriateforcontrollingtheactioninthegames,butitalsohastheaddedbenefitofgettingpl
ayersoffthecouchandintotheaction,aninterfacefeaturethatisappreciatedbyparentswor
ldwide!AscanbeseeninFigure1.3,theWiibowlinggameenablestheplayertointer-actwithth
egameinamannerthatissimilartothatofreal-worldbowling.Thesenewinterfaceshavetheir
ownproblems,however.Forinstance,shortlyaftertheWiiwasreleasedtherewerereportsofu
sersaccidentallylettinggooftheremote
FIGURE1.2Konamisgestureinterfacegame,MoCapBoxing.Unlikepreviousgenerationsofsport
sgames,theuserdoesnotusebuttonstocodehisintentions.Instead,hedonsboxingglovesand
movesinamotioncapturearea(themattheuserisstandingon)tocontroltheinterface.Theend
effectisafairlyrealisticgamethatisintuitive(andtiring!)touse.(CourtesyofKonami.)
1IntroductiontotheHumanFactorsofNontraditionalInterfaces6

(particularlyinbowling,sincethatiswhatpeopledowhentheybowl)andhavingitcrashinto,
orthrough,thetelevision[Burnette,2006]).Gestureinterfacesrangefromsystemsthatemp
loyhandmotionforlanguageinputtothosethatusegesturestonavigate(e.g.,Iwanttogothatw
ay)andissuecommands(e.g.,Pickthatup)inavirtual-realityenvironment.Issuessurround-in
gthelimitationsofdiscriminatinggesturesandhowthoselimitationsguidethedesignofthe
seinterfacesareexplored.Inaddition,thepotentialforundesirableartifactswhenusingt
hesekindsofinterfaces(fatigue,misinterpretation,etc.)willbediscussedalongwithmet
hodsthathavebeendevelopedtomitigatethesepotentialdeficiencies.1.2.3LocomotionInt
erfacesLocomotioninterfaces,althoughsharingattributeswithbothhapticinterfacesand
gestureinterfaces,differbecausetheyrequiregrossmotormovementandtypicallydealwith
large-scalemovementornavigationthroughaninterface.Interfacesinresearchlabsnotonl
yincludetreadmill-typeinterfaces,buthavemovedinotherinterestingdirectionsaswell,
includingswimmingandhang-glidingapplications.Thesekindsofinterfacesarefrequently
associatedwith
FIGURE1.3Gesture-basedinterfaceinaction.TheNintendoWii,agesture-basedinterface,i
sbeingplayedinabowlingsimulation.Thesekindsofinterfacesbringtheirownuniqueissues
tothedesigner.
1.2NontraditionalInterfaces7

high-endsimulators,butthetechnologyhasrecentlymovedoutofthelaboratoryandintothec
ommercialrealmintheformofbodymotionarcadegamessuchasdancingandskiing.Aswithgestu
reinterfaces,severalofthemostcurrentgenera-tionsofhomegamingboxeshavebodymotionc
ontrollersavailableaswell.Issuessurroundingthephysicallimitationsofthehumanbodya
ndhowthoselimitationsguidethedesignoftheseinterfaceswillbeexplored.Inaddition,th
epotentialforundesirableartifactswhenusingthesekindsofinterfaces(fatigue,vertigo
,etc.)areconsidered.1.2.4AuditoryInterfacesAuditoryinterfaceshavelongbeenusedtos
endsimplecodedmessagesacrosswideareas(e.g.,tollingofchurchbells,wailingofthecivi
ldefensesirens).Auditoryinter-faceshavealsobeenusedextensivelytoaugmentcomplexin
terfacesandtospreadthecognitiveloadinhighlyvisualinterfaces,particularlyinthecon
veyanceofwarnings.Thesekindsofauditoryinterfacesarerelativelysimpletoimplement,b
utrequirethattheuserbeabletointerpretthemeaningofthecodedmessage.Recently,audito
ryinterfaceshavebeenemployedasasubstituteformorecomplexvisualinterfaces,andthete
rmsonificationhasbeencoinedtodescribethesekindsofauditoryinterfaces.Inasonifiedint
erface,representationsthataretypicallyvisual,suchasgraphsandicons,areturnedintos
ound,thatis,sonified,sothattheycanbeinterpretedintheauditoryratherthanthevisuald

omain.Thechapteronauditoryinterfaceswilldetailthefundamentalpsychophysicsofthehu
manauditorysystem,andthenrelatethattothedesignandimplemen-tationofauditoryinterf
aces.Issuesofoverload,humanlimitations,andappropri-ateselectionofauditoryfrequen
cyspaceforvariouskindsofauditoryandsonifiedinterfaceswillbediscussed.1.2.5Speech
UserInterfacesSinceAliBabaproclaimedOpenSesame!tomagicallygainentrancetotheDenofth
e40Thieves,speechinterfaceshavecaptivatedourimagination.Withthisimaginationfurth
erfueledbysciencefictiontelevisionseriessuchasStarTrek,wehavebeengenerallydisapp
ointedwiththereal-worldimplementationsofspeechinterfacesbecausetheyseemtolagsofa
rbehindourideaofhowwelltheyshouldwork.However,recentadvancesincomputingpowerhave
broughtthepossibilityofrobustspeechinterfacesintoreality,andcommercialsystemsare
nowreadilyavailableinmultiplerealms.Earlyimplementationsofsingle-wordspeechcomma
ndinterfaceshaveledtocontinuousspeechdictationsystemsandstate-of-the-artsystemsth
atemploypowerfulsemanticanalysistointerpretausersintentwithunstructuredspeech.
1IntroductiontotheHumanFactorsofNontraditionalInterfaces8

Thechapteronspeechinterfaceswilldiscussthetechnologybehindtheseinter-facesforbot
hspeakingandspeechrecognitionsystemsandcharttheprogressofboth.Itwilldiscusstheim
plementationofbothlimitedandunboundedvocabularyinterfaces,reviewtheadvantagesofs
peaker-dependentandspeaker-independentsystems,anddetailtheappropriatedesignofspe
echpromptsandnavigationstruc-tures.Sincespeechinterfaceshavebeenimplementedwidel
yandsuccessfullyintelephone-basedsystems,extensiveexamplesandcasestudiesofsucces
sfulinterfacesinthisrealmwillbeusedtohighlightthehumanfactorsofspeechuserinterfa
ces.1.2.6InteractiveVoiceResponseInterfacesInteractivevoiceresponsesystems(IVRs)
areinwidespreaduseinthecommercialworldtoday,yetreceivescantattentionintraditiona
lhumanfactors.Theinterfacehasbeenembracedbythebusinesscommunitybecauseofitshugep
otentialforcostsavingsandbecausewhenimplementedcorrectlyitcanresultinhighcus-tom
ersatisfactionratingsfromtheuseraswell.Becauseoftheubiquityoftheinter-face,howev
er,poorlydesignedIVRinterfacesabound,andusersarelefttosuffer.ThechapteronIVRswil
ldiscussthespecificationofappropriatenavigationstructures,promptconstruction,and
transactionmanagement,includinguserinputsandtime-outissues.Theselectionanduseofv
oicepersonaandtheimpactonbothuserpreferenceandperformancewillalsobeconsidered.Al
thoughIVRsaretypicallyusedforroutingcustomerstotherightserviceagent,ortoconveyli
mitedamountsofinformation(likeabillbalance),somecurrent-generationinterfacesprov
ideformoreinteractionandallowsignificantlymoreinformationtobedeliveredtotheuser.
ThesehighlyinteractiveIVRsandthespecialproblemsassociatedwiththemwillbeconsidere
daswell.1.2.7OlfactoryInterfacesOlfactoryinterfaceshavetypicallybeenusedinsituat
ionswherewidespreadcommunicationofamessageisrequired,butwheretraditionalinterfac
esarehamperedbytheenvironment.Stenchsystemstowarnminersofdanger(wheretraditional
auditoryandvisualwarningsdonotfunctionwell)andtheuseofwintergreenasafirealarmsig
nalinfactorieswherethereissignificantauditoryandvisualnoise(e.g.,aweldingshop)ar
etwoprimeexamplesofsuccessfulolfac-toryinterfaceimplementations.Aswithmanyofthei
nterfacesinthisbook,theadventofcomputersandsimulatedenvironmentshaspushedseconda
ryinterfaces,likeolfaction,tothefore.Firstintroducedinthe1960sasawaytoaddanother
dimensiontotheatermoviepresentations(whereitwasaresoundingfailure),researchiscon
tinuing
1.2NontraditionalInterfaces9

tobeconductedontheseso-calledsmell-o-visioninterfacesthatwillallowtheusertoexperie
ncescents.Unliketheearlyimplementations,researchisnowfocusedprimarilyontheIntern
etasadeliverymediuminsuchdiverseapplica-tionsasshopping(perfumefragrancesamples)
,entertainment(thesmellofburningrubberasyoudriveavideogameracecar),andambience(e
vergreeninanInternetChristmasshop).Ofcourse,theintroductionofsmellasaninterfacea

lsohastremendouspoten-tialintrainingapplicationsinareassuchasmedicineandequipmen
tmaintenance.Whencoupledwithvirtual-realitysimulators,theadditionofsmellmaybeabl
etosignificantlyenhancethetrainingexperience.Thepsychophysicsofolfactionwillbedi
scussedingreatdetail,sincethestrengthsandlimitationsofthehumanolfactorysystempla
yasignificantroleinthecorrectimplementationoftheseinterfaces.Theappropriateusesa
ndtechnologyimplementationsofcurrent-andnext-generationsystemswillalsobeconsider
ed.1.2.8TasteInterfacesWithoutadoubt,interfacesthatrelyontasteareoneoftheleastex
ploredofthenontraditionalinterfaces.Tasteinterfacesareusuallydiscussedintermsofs
imulation,inwhichaparticulartasteisaccuratelyrepresentedtosimulatearealtaste(e.g
.,forafoodsimulator).However,tastecanalsobeusedtoconveycodedinformation,muchlike
olfactorydisplays.Becausethisinterfaceisinitsinfancy,thechapterontasteinterfaces
willfocusprimarilyonthecurrentstate-of-the-arttastesimulatorresearchinthisareaan
dthepotentialforfutureuses.1.2.9Small-ScreenInterfacesMiniatureinterfaceshavebee
nenvisionedsinceDickTracyfirstusedhiswristpicturephone,andthisvisionhasbecomeare
alitywiththesuccessfulminiaturi-zationofelectroniccomponents.Whiledevicessuchasm
obiletelephonesandMP3playershavecontinuedtoshrink,theproblemswithcontrollingandu
singtheseminiaturedeviceshavegrown.Fromthephysicalergonomicsassociatedwithusingt
hesystemstothenavigationoftinymenus,thenewsystemshaveproventobesubstantiallydiff
erent,andmoredifficult,tousethantheirbiggerbrethren.Thechapteronsmall-screeninte
rfaceswilldiscusshowtheseminiatureGUIsaredesignedandtested,andhowspecialinterfac
emethods(predictivetyping,rapidserialpresentationoftext,etc.)canbeemployedtomake
theseinterfacessignificantlymoreusable.Thechapterwillalsodiscusshowtoimplementco
m-monmenu,navigation,andinformationdisplaystructuresonbothmonochromeandcolorscre
ensthatcontainextremelylimitedrealestate,fromtinycellphonescreenstomicrowavedotm
atrixdisplays.Specialemphasiswillbeplacedonhow
1IntroductiontotheHumanFactorsofNontraditionalInterfaces10

todesigntheseinterfacessothattheneedsofolderuserpopulations,whomayhavereducedvis
ualandmotorcapabilities,areaccommodated.1.2.10MultimodeInterfaces:TwoorMoreInter
facestoAccomplishtheSameTaskInmanyinstances,ataskcanbeaccomplishedusingoneormore
interfacesthatcanbeusedinamutuallyexclusivefashion.Forexample,youcanperformsimpl
ebankingtasksusingtheInternetoraninteractivevoiceresponsesystem.Youcanchoosetous
eeitherinterfacemethodforanygiventransaction,buttheyaremutuallyexclusiveinthesen
sethatthetaskdoesnotalloworrequirebothtobeusedsimultaneously.Providingmultiplein
terfacesforsinglesystemsmeansthattheseeminglyindependentinterfacesmustbedesigned
andtestedtogetherasasystem,toensurethatuserswhomovebackandforthbetweenthetwointe
r-facescandososeamlessly.Thischapterwillexplorethemorecommonmutuallyexclusivemul
timodeinterfaces(MEMM),includingIVR/GUI,speech/GUI,smallscreen/GUI,smallscreen/I
VR,andsmallscreen/speech,anddiscussthehumanfactorsassociatedwiththedesignandimpl
ementationofthesemultimodeinterfaces.Appropriateselectionofthedifferentmodeswill
bediscussed,aswellasconsiderationofimplementingunequalcapabilitiesinthesetypesof
systeminterfaces.1.2.11MultimodeInterfaces:CombiningInterfacestoAccomplishaSingl
eTaskIndirectcontrasttotheMEMMinterfacesdescribedinChapter11,multimodeinterfaces
thateitherrequireorallowtheusertointeractwiththesystemwithmorethanoneinterfaceat
thesametime(i.e.,amutuallyinclusivemultimode[MIMM]interface)aremuchmorecommonand
typicallyfallintotwodistinctclasses.ThefirstclassofMIMMinterfacesarethoseinwhich
theusercanchooseamongmultipleinterfacesduringatask,andcanmovebackandforthamongth
ematanytime.Forexample,systemsthatcombinespeechandinteractivevoiceresponseinterf
aceshavebecomemorecommon(e.g.,Pressorsayone),andthewaythesesystemsmustbedesignedan
dimplementedisdecidedlydifferentthanifeachinterfaceweretobeimplementedalone.Thes
econdclassofsystems,andbyfarthemostcommonofanydescribedinthisbook,arethoseinterf
acesthatemploymultipleinterfacesintoasinglesysteminterface.Auditoryandvisualinterf
acesarefrequentlycombinedtocreateeffectiveinterfaces.Virtual-realitysystemsareap
rimeexampleofthiskindofinterface,wherevision,audition,speech,haptic,andgesturein
terfacesarecombinedinasingleintegratedexperience.ThischapterwillfocusonMIMMinter
facesthatuseoneormoreoftheothernontraditionalinterfacesdescribed

1.2NontraditionalInterfaces11

inthebook.WhilethechapteronMIMMinterfacesisnotmeanttobeaprimeronsystemhumanfacto
rs,itwilldiscusshowtodeterminewhichinterfacemodestouse(interfaceallocation)andho
wandwhentoovercodetheinterfaces(inter-faceredundancy),anditwilldealwithissuessur
roundingthepresentationofinformationusingnon-nativeinterfaces(i.e.,usingtactiled
isplaystorepresentsound).1.3DESIGNPRINCIPLESFORNONTRADITIONALINTERFACESEachchapt
erdescribescertainhumanfactorsprinciplesanddesignconsidera-tionsthatshouldbetake
nintoaccountwhenworkingwithaparticularinterfacemodality.However,thefundamentalpr
inciplesfordesigningnontraditionalinter-facesarethesameasthosefordesigningtradit
ionalGUIs.SchneidermanandPlaisant(2004),Nielsen(1999),Raskin(2000),Norman(2002),
Mayhew(1997),andothershavedescribedtheseprinciplesingreatdetail.Thekeyistorememb
erthatthetraditional,fundamentalprinciplesstillapply,eveniftheinterfaceyouaredes
igningisanythingbuttraditional.Mostusersdonotcareabouttheinterfacetechnologytheys
implyhaveatasktheywanttoaccomplishandtheywanttheinterfacetosupportthemincom-plet
ingthattask.Toooften,however,thedesignerisledbyothergoals(corporateneeds,fascina
tionwithtechnology,coolfactor,etc.),andthehumanfactorsofthedesignsuffer.Thebottoml
ineisthis:Gooddesignsdonotjusthappen!Theyaretheresultofcarefulapplicationofnumer
ousdesignmethodologiesthatenableyou,asthedesigner,tounderstandtheuser,theenviron
ment,andhowtheyinteract.ISO9241:11(ISO,1998)specifiesthatusabledesignsshouldhave
threeimportantattributes.First,theyshouldbeeffective,meaningthattheusercansucces
sfullyusetheinterfacetoaccomplishagivengoal.Second,theinterfaceshouldbeefficient
.Thismeansthattheusernotonlycanaccomplishthegoal,butcandosoquicklyandeasilywitha
minimumoferrororinconvenience.Finally,theinterfaceshouldleavetheusersatisfiedwit
htheexperience.Thisdoesnotmeantheuserhastobehappy,butitdoesmeanthattheusershould
havehighconfidencethatthetaskwasaccomplishedaccordingtohisintentions.Forexample,
usingabanksautomatedtelephonesystemtotransfermoneyshouldbeeasytodo(effective),qui
ck(efficient),andleaveuserswiththecertaintythattheyreallyaccom-plishedthetaskand
thatthebankhastheircorrectinstructions(satisfaction).Thenextsectionsummarizesthe
fundamentalhumanfactorsdesignguide-linestokeepinmindwhenpursuinggeneralusability
goals,regardlessoftheinterfacemode(s)youeventuallydecidetouseinyourinterface.
1IntroductiontotheHumanFactorsofNontraditionalInterfaces12

1.3.1DesigntoSupporttheUsersGoalsWheneveryoustartadesign,youcertainlyhavetheuseri
nmind.Unfortunately,somewherealongtheway,designersoftenforgetthattheyshouldbetry
ingtoservetheuserratherthananothermaster,suchasthecorporation,vendor,ortechnolog
y.Thinkaboutwhattheuserwantsorneedstodoandfocusonthat.Agreatexampleofaviolationo
fthisprinciplecanbefoundinthebutterflyballotdebacleinthe2000Floridaelections.Alt
houghnotmaliciouslyconceived(unlessyouareaconspiracybuff),theballotwasdesignedto
supportamoremodestgoal:spaceefficiency.Theresults,asweallknow,weresignificantque
stionsaboutvotersintentintheseeminglysimpletaskofselectingtheirpreferredcan-didat
e.Eachtimeyouaretemptedtoadd,delete,change,prettify,orotherwisealteragooddesign,
askyourselfifthishelpstheuseraccomplishthegoal.Ifitdoesnot,thinktwice!1.3.2Desig
ntoSupporttheUsersConceptualModelWheneverpeopleuseasystem,theybringwiththemanidea
ofhowthatsystemworks.Ifyoursystemsdesigndoesnotmatchthisidea(theusersconceptualmod
el),thenthewaythatthepersonwillusethesystemcanbeunpredictable.Theclassicexampleo
fthisishowmanypeopleinteractwiththethermostatintheirhome.Forinstance,apersoncome
shomefromalongdayatwork,andthehouseishot.Hegoestothethermostatandturnsthedialall
thewaytothelefttoquicklycoolthehousedown.Whydoesheturnitallthewaytotheleft,rathe
rthancarefullyselecttheexacttemperaturethathewantsthehousetobe?Doesturningthedia
lallthewaytotheleftmakeitcoolerfaster?No!Thecool-ingsysteminthehomeisatwo-stated
eviceitiseitheronoroff.Itcoolsuntilitreachesthetemperaturesetonthethermostat.Howe

ver,themodelthatmanyofushaveisthatthefartheryouturnthedialtotheleft,thecooleritg
ets.Inonerespect,thatistrueleftiscoolerbuttherateofchangeisindependentofthesetting
.Whyisthisaproblem?Invariably,aftermakingthismake-it-as-cold-as-fast-as-possiblead
justment,onegetscalledawayfromthehouseandreturnstomeat-lockerliketempera-tures.Th
einterfacefailedtosupportwhattheuserthoughtthesystemwoulddo(andwhattheuserwanted
ittodo).Newerelectronicthermostatsseemtohavepreventedthisproblem,sinceaspecifict
emperaturesettingisrequiredandsoaquickcold-as-you-can-make-itcommandisharderandslo
wertoexe-cute.Figuringoutwhattheusersmodeliscanbedifficult(andmodelsmayvaryfrompe
rsontoperson).However,mappingthewaythesystemreallyworkstothewaypeoplethinkitwork
smakesforsuperiorusability.
1.3DesignPrinciplesforNontraditionalInterfaces13

1.3.3DesignfortheUsersKnowledgeEachuserofyourinterfacewillbringaspecificsetofknow
ledgeandexperiencetothetable.Ifyourinterfacerequiresknowledgethattheuserdoesnoth
ave,thenitwilllikelyfail(unlessyoucantraintheuseronthespot).Forinstance,asillu-s
tratedinFigure1.4,thedesignertookgreatcaretoensurethattheinterfacewouldsupportus
erswithmultiplelanguageskills(goodforthedesigner!).How-ever,hefailedtofollowthro
ughinhisexecutionofthedesign.Whatiswrongwiththisselectionmenu?TheusermustreadEng
lishinordertoselectthedesiredlan-guage!Determinewhatyourusersknow,andthendesignt
heinterfacetorequirenomoreknowledgethanthat.1.3.4DesignfortheUsersSkillsandCapabi
litiesInadditiontospecificknowledgesets,usersofyourinterfacewillalsohavespecific
setsofskillsandcapabilitiesthatlimithowtheycanuseyourinterface.Forexample,humans
havespecificlimitsonwhattheycanhear,sowhenyouaredesigninganauditorywarning,youwa
nttomakesurethatyouselectafrequencythatcanbeeasilyheardbythehumanear.Thinkofthed
ogwhistleasadogsbasicauditorydisplayitsaysAttention!tothedog.Theinterfaceisworthless
forusewithhumans(likeyourchildren)becausethewhistleoperatesoutsideofthehumanearsc
apabilitytodetectsound.Itis,however,aperfectinter-faceforthedog.Bythinkingofwhat
youruserscanandcannotdo,youcandesigntheinterfaceinthemosteffectivefashion.Figure
1.5showsanexampleofapoorlydesignedinterfaceinthatitignoredthespecificskillsandca
pabilitiesofitsintendedusers.Whiletheintentofthedesignerswasnoble,theexecutionof
thedesignresultedinaninterfacethatdidnotserveitsintendedpurpose.
FIGURE1.4Exampleofaclearviolationofdesigningfortheusersknowledge.UsersmustreadEng
lishinordertoselectthelanguageinwhichtheywouldliketousetheinterface.(Courtesyoft
heInterfaceHallofShame.)
1IntroductiontotheHumanFactorsofNontraditionalInterfaces14

1.3.5BeConsistentButNotattheExpenseofUsabilityConsistencyisoftenthehallmarkofausa
bledesign.Onceyoulearntouseoneinterface,thenothersimilarinterfacesareeasiertolea
rnanduse.Thisconsis-tencycanmanifestitselfasstandardizationamonginterfacesmadeby
differentdesignersorcanshowupasgoodinternalconsistencywithinagiveninterface.Inbo
thcasestheusabilityoftheinterfaceisincreasedbecauseusershavetolearnandrememberle
ssastheybecomefamiliarwiththeinterface.Unfortunately,blindadherencetotheconsiste
ncyprinciplecanactuallyleadtolessusableinter-faces.Sometimestheexistingstandardd
oesnotlenditselfwelltoanewproblemorsituation.Ifthedesignermakesthedecisionthatco
nsistencyisoftheutmostimportance,thentheentireinteractionmaybeforce-fitintothestan
dardinter-face.Whiletheendusermayenjoysomeofthebenefitsofaconsistentinterface,th
eforcedfitmayactuallydecreasetheoverallusabilityofthesystem.Small-screenimplemen
tationsoftheMicrosoftWindowsoperatingsystemareagoodexampleinwhichconsistencymayn
othavebeentherightchoice.
FIGURE1.5Exampleofaninterfacethatdoesnottakeintoaccounttheskillsandcapabilitieso

ftheusersforwhomitwasintended.Theintercomsystemissupposedtobeusedbyindividualswh
oaremobilityimpairedandcannotpumptheirowngas.Theproblemhereisthatitsplacementpre
cludesitsusebythisverypopulation.Further,theinstructionsthatexplainwhattheuseris
supposedtodoaretoosmalltobeeasilyreadfromthemostlikelylocationofauserinawheelcha
ir.
1.3DesignPrinciplesforNontraditionalInterfaces15

Clearly,theusersofthesedeviceshadthepotentialadvantageofbeingabletoseamlesslytra
nsitionbetweendesktopandhandheldapplications.However,manyoftheGUIelementsthatWin
dowsusesdonotscalewelltoverysmallscreensandsotheconsistencybeginstodegrade.Now,t
heinterfaceisonlypartiallyconsistent,somanyofthebenefitsderivedfromconsistencyar
elost(IknowIcandothisinWindows!Whereistheicon!?!).Othermanufacturesofsmall-screend
eviceshavedeterminedthattherearebetterinterfacemethodsandhavedesignedtheirownspe
cializedinterfaces(Chapter10addressestheseinmuchmoredetail).Whilethelearningcons
istencyhasbeenforfeited,theinterfaceisnotbeingaskedtodothingsthatitwasnotorigina
llydesignedtodo,andsooverallusabilityisenhanced.Intheend,youshouldseriouslyconsi
derwhetherconsistencyenhancesordetractsfromtheoverallusabilityoftheinterface,and
thenmaketheappropriateimplementationdecision.1.3.6GiveUseful,InformativeFeedback
WhenIfirstbegangivinglecturesthatincludedtheseeightdesignprinciples,thisguidelin
esimplyreadGiveFeedback.However,afterseeingtoomanyexamplesofuselessfeedbackmessage
sfilledwithcrypticerrorcodesorworthlessstateinformation(Figure1.6),theguidelinew
aschangedtoincludethewordsusefulandinformative.Usersofaninterfaceneedtoknowwherethey
areintheinter-face,whattheyaredoing,andwhatstatethesystemisin.Byprovidingfeedbac
k,thesystemaidstheuserinmakingcorrect,efficientdecisionsaboutwhatactionstotakene
xt.Thefeedbackmaybeexplicitandrequiredirectaction(e.g.,apop-updialogboxthatstate
s,Clickingyeswilleraseallthedata.Proceedanyway?),orit
FIGURE1.6Interfaceprovidingfeedbackthatisnotusefulorinformative.WhycantIdothis?Wh
atshouldIdotomakeitsothatIcan?Isthereaworkaround?Thisisthekindoffeedbackthatgive
sinterfacedesignersabadname.(CourtesyoftheInterfaceHallofShame.)
1IntroductiontotheHumanFactorsofNontraditionalInterfaces16

maybemoresubtleinnatureandprovidetheuserwithcuesthatthestateofthesystemhaschange
dandrequirestheattentionoftheuser(e.g.,antilock-brakefeedbacksystems).Asthedesig
ner,youshouldmakesurethatyouprovidebothkindsoffeedback,andtesttheeffectivenessof
thefeedbackwithactualuserswhoencounterthecaseswhereitisprovided.1.3.7DesignforEr
rorRecoveryPeoplemakemistakes.Withthatinmind,youshouldalwaysensurethattheinter-f
aceisdesignedtohelpusersbyminimizingthemistakestheymake(e.g.,theauto-spellfeatur
einaword-processingprogram)andbyhelpingthemrecoverfrommistakesthattheydomake.Int
hestrictestdefinitionoferror,noadverseconse-quenceisrequiredfortheaction(orlackt
hereof)tobeconsideredasanerror.Inpracticalterms,however,anerrorthatiscaughtandco
rrectedbeforeanyadverseimpactoccursisoffarlessconcernthananactionthathasanadvers
eout-come.Providinguserswithwaystoeasilycorrecttheirmistakes,evenlongaftertheyha
vehappenedcanbeverybeneficial.Thetrashcanthatisemployedinmanydesktopcomputerinte
rfacesisanexcellentexample.Adocumentmaybethrownawaybut,justlikeinthephysicalworl
d,canbepulledoutofthetrashandrecov-erediftheuserdiscoversthatthedeletionwasinerr
or.Contrastthatwiththedia-logwindowshowninFigure1.7.Eveniftheuserrecognizesthats
hehasperformedtheactioninerror,thesystemisofferingnorecourse.Addinginsulttoinjur
y,thepooruserhastoacknowledgeandaccepttheactionthatisnolongerdesired!1.3.8Design
forSimplicityAsAlbertEinsteinoncenoted,Makeeverythingassimpleaspossible,butnotsim
pler.Thisaxiomisespeciallytrueininterfacedesign.Simplicitymakes

FIGURE1.7Poorerrorrecoveryforacriticalaction.Imaginewhatyouwoulddo(besidesscream
)ifyouknewyouhadjustmadeamistake,andyoudidnotwanttooverwritetheoriginalfile.(Cou
rtesyoftheInterfaceHallofShame.)
1.3DesignPrinciplesforNontraditionalInterfaces17

interfaceseasytolearnandeasytousecomplexitydoestheopposite.Unfortu-nately,complex
ityissometimesanecessaryevilthatreflectsthenatureofthetask.Interfacesfornuclearp
owerplantsandsophisticatedfighterjetsjustcannotbeassimpleasthatforaniPod.Inthese
situations,youmuststrivetounderstandtheuserstasktosuchadegreeastobeabletodistillt
heinterfacedowntoitsessentialcomponents.Theresultmaystillbecomplex,butitsfunctio
nswillrep-resentwhattheuser(theoperatororthepilot)actuallyneedstodo.Inmanycases,
however,thecomplexityisduetotheinclusionoffeatures,usuallyovertime,thatenhancethep
roduct.Allofushaveseenthisfeaturecreep.Youonlyhavetolookasfarasyourwordprocessorf
eatures,features,andmorefeatures.IfonlyIcouldjustfigureouthowtousethem,theywould
begreat.Designsimplicityiswherethehumanfactorspractitionerandthemarketingfolksma
ybeatodds,sincemarketingisoftenlookingforfeaturesuperiority(Mytoyhasmorefeaturest
hanyourtoy!).Theresultcanbeaninterfacethatmasqueradesassimple,buthassignificanthi
ddenfunctionalityandcomplexitybehinditsbeautifullysimpleexterior.Themoderncellph
oneisaclassicexample.Initsmostbasicform,thecellphoneisdesignedtomakeandreceiveca
lls.Inmostcurrentimplementations,itisalsoacamera,anaddressbook,anMP3player,agame
platform,apager,andsoonandsoforth.Whileeachfeatureonitsownmaybeimportant,thecomb
inationofallthesefeatures,particularlyonasmall-screendevice,canleadtousabilityni
ghtmares.Sometimesthecomplexityissimplyamatterofspacetoomanyfeaturesandnotenoughs
pacetoimple-mentthem.Figure1.8showsanexampleofarelativelycomplexdesignthatisdriv
enbyspaceconsiderations.Asthedesigner,itwillbeuptoyoutotrytodeterminehowtomaximi
zebothfeatureinclusionandsimplicityinawaythatsupportsalloftheusersgoals.Any-oneca
ndesignacomplexinterfaceittakesrealskilltodesignasimpleone.Designingnontraditiona
linterfacesdoesnotreleaseyoufromthefundamentaldesignprinciplesdescribedhere.Byun
derstandingwhatyouruserswant,whattheyneed(whichisoftendifferentfromwhattheywant)
,whattheyknow,howtheydotheirwork,andwhattheirphysicalandcognitivelimitationsare,
you,thedesigner,cancreatesuperiorinterfacesthatsupporttheuserinaccomplishinghisg
oal.1.4THEFUTUREOFNONTRADITIONALINTERFACEDESIGNUndoubtedly,interfacedesigninthef
uturewillmovetowarddesignswheretherightmodalityforthejobisselected,ratherthanthe
modalityathandbeingforcedtofitthejob.Theresanoldexpressionthatsaysthatifallyouhav
eisahammer,theneverythinglookslikeanail,andthesamecanbesaidofGUIs.TheGUI
1IntroductiontotheHumanFactorsofNontraditionalInterfaces18

isapowerfulandadaptableinterfaceform,anddesignerswhoaretrainedintheartandscience
ofcreatingGUIshavetendedtoapproachanygiveninterfaceimplementationwithaGUIsolutio
n.However,carefulanalysisoftheneedsoftheuserinagiveninterfaceenvironmentmaysugge
stthatanother(nontradi-tional)formofuserinterfaceisrequired.Morelikelythannot,th
einterfacewillbemultimodal,asdescribedinChapters11and12,andthegoalofthedesignerw
illbetoensurethatthecorrectinterfacemodesareassignedtotheinformationinputsandout
putsthatprovideforthebest,mosteffectiveexperiencefortheuser.Muchofthismultimodal
interfaceevolutionisboundtomovetowardmorenaturalinteractiontechniques.Innaturalint
eractions,theuserdoesnotneedtomakeanytranslationofcodeddataordeterminehowtoperfo
rmafunctiontheusersimplyinteractswiththeinterfaceasshewouldinthephysicalworld.Ana
logs(liketheWindowsdesktop)willdisappear,andtheinterfacewillbecomeindistinguisha
blefromitsrepresentativemodel.ImagineiftheWindowsdesktopwerereplacedwithanimmers
ivevirtualdesktopthatwasindistinguishablefromthereal-wordanaloginformandbasicfun
ctionbuthadallthebenefitsthatcanbederivedinacomputer-drivenmodel.Thisvisionismos
tcertainlynotareversiontothemuchmalignedvirtual-worldinterfaceproposedbyMicrosof

tinthemid-1990sthatwassupposedtotakethedesktopmetaphortothenextlevel.MicrosoftBo
b(Figure1.9)triedtorepli-catetheofficeandhomeenvironmentthrougharichmetaphorthat
hadallthe
FIGURE1.8ComplexinterfaceformultiplefeaturesontheturnsignalstalkinaChevroletSubu
rban.Thestalkusedtobereservedforasinglefunctionactivatingtheturnsignal.Nowitsuppo
rtstheblinker,cruisecontrol,variablefrontwipers,high-beamheadlights,rearwiper,an
dwindowwashersystem.Theresultisasystemthatishardtouseandcontributestoinadvertent
activationofundesiredfunctions.
1.4TheFutureofNontraditionalInterfaceDesign19

objectsthatonewouldexpectintheseenvironments.However,themetaphorwasnotquiteperfe
ct,sincetherearemanythingsthatcan(andshould)bedoneonacomputerthatjustdonotfitint
othephysicalmetaphor.Theimplementationsofthesenonconformingfunctions(nottheleast
ofwhichwasthetalkingdogthatwasneededtohelpnoviceusersgettheirbearings)causeduser
stohavedifficultyinlearningandusingtheinterface.Thiscomplexity,alongwithaplethor
aofhiddenfunctions,ledtheinterfacetoanearlydemisedespiteitspromiseasaresearchpla
tformintohigh-performancemetaphoricalinterfaces.Oneofthemostexcitingdevelopments
innontraditionalinterfacesistherecentadvancesthatarecurrentlybeingmadeinbraincomp
uterinterfaces.Thisclassofinterfacecompletelybypassesthehumanmusculoskeletalbody
asamechanismforinput/output,andinsteadinterfacesdirectlywiththebrain.Theinterfac
ethattheuseristryingtouseinterpretstheserawbrainwavesandthenperformstheappropria
teaction.Thesekindsofinterfacesrepresentavery
FIGURE1.9HomepageforMicrosoftBob.MSBobwasanextensionofthedesktopmetaphorthatwass
upposedtobeextremelyeasytolearnanduse.Unfortunately,itsufferedfrommanydeficienci
es,andMicrosoftpulleditfromproductionshortlyafteritsrelease.Source:Mid-1990sscre
enshotofMicrosoftsBobOperatingSystem.
1IntroductiontotheHumanFactorsofNontraditionalInterfaces20

specialized,newlyemerginginterfacedomain,andsotheyarenotaddressedinaseparatechap
terinthisbook(justwaitforthesecondedition!),buttheyholdsig-nificantpromiseforthe
future.Muchofthecurrentbraincomputerresearchisfocusedonusingbrainwavestocontrolsc
reencursormovements(Friedrich,2004),althoughrecentadvanceshaveledtolimitedsucces
sintheoperationofroboticarms(Donoghueetal.,2007).Ofcourse,aswithmanyfuturisticin
terfaces,HollywoodhasbeenthinkingaboutthisforsometimerecallClintEastwoodflyingthe
super-secretRussianplaneinthe1980sthrillerFirefox,inwhichtheaircraftiscontrolled
bythethoughtsofthepilotbutofcoursethosethoughtsmustbeinRussianGjvjubnt!Zytdb;edhec
crbq!Help!IcantthinkinRussian!.Althoughthetechnologyisnowherenearthislevelofsophistic
ation,Tanakaandhiscolleagues(Tanaka,Matsunaga,&Wang,2005)havehadtremendoussucces
swithresearchinwhichpatientscouldcontrolthenavigationoftheirpoweredwheelchairsus
ingbraininterfaces,eliminatingtheneedforjoysticksorotherinputdevices.Directbrain
interfacesmayprovetobeaboonforthephysicallydisabled,allowingthemtocontrolcompute
rsandotherassistivedeviceswithoutphysicalmovement.Holographicinterfacesareanothe
rtechnologythatmaybecomeimportantinfutureapplications.Oneofthebiggestproblemswit
hcurrentGUIsisthattheydemandphysicalspace.Ifthephysicalspaceisrestricted,asinamo
bilephoneforinstance,thentheinterfacemustconformtothereducedspace.Holographyhast
hepotentialtoovercomethislimitationbyusingtheairastheinterfacemedium.Byusingholo
graphy,nophysicalinterfacewouldberequired.Simplecommercialholographicinterfacesa
rejustnowbecomingavailable,andresearchintomorecomplexholographicinterfacescontin
ues(e.g.,Bettioetal.,2006;Kurtenbach,Balakrishnan,&Fitzmaurice,2007).Whenmention
ingholographicdisplays,theonesenvisionedbyGeorgeLucasandhisteamofspecialeffectsw

izardsimmediatelycometomind.Heandhisassociatesshowedushowfutureholographicinterf
acesmightincludegames,per-sonalcommunicationsdevices,andbattlefieldtacticaldispl
ays.Realityhasbeenlessforthcomingaholographic-likegamewasintroducedintheearly1980
sbyCinematronicsthathadtheillusionofbeingaprojectedholographicdisplay(itwasanill
usionbasedon,literally,mirrors).DragonsLairwasimmenselypop-ularatthetime,buttheli
mitationsofthegameanditsdisplayinterfacemadeitthelastcommerciallyavailablegameof
itstypesincethen.AmorerealisticdepictionofwhatthefuturemightholdisshowninFigure1
.10.Althoughnotholographic,thenewprojectedkeyboardsaresimilarinconceptkeyboardsta
keupspace,andsoforsmalldevices,akeyboardthatcouldappearoutofthinairmightbeausefu
linterfacemode(althoughspeechmightbeaviablealternativeaswell).Figure1.11showsone
ofthesevirtualkeyboardsbeingprojectedonaflatsurface.
1.4TheFutureofNontraditionalInterfaceDesign21

Withanyluck,thesefutureinterfaceswillbesignificantlymoreintuitivefortheaverageus
erandbecomeeasiertouseandhencemorepowerful.Earlycom-puterdesignersoriginallyenvi
sionedthatcomputersandautomationwouldreduceoreliminatethedifficultiesofperformin
gtasks,andfreepeopletobecomecreativeproblemsolvers;mundane,repetitive,andinsigni
ficantjobswouldberelegatedtothemachines.AnyonewhohastriedtosetaVCRclockorfigureo
uthowtoformatapaperinaword-processingprogramcaneasilyattesttothefactthattheinter
faceoftencomplicatestheproblemratherthansimplifyit.Thesenewtechnologiesandtheint
erfacestheymayleadtoholdthepromiseofallowingtheusertofinallyfocusonthetask,notth
einterface.Whilethenontraditionalinterfacespresentedinthisbookareastart,thereiss
tillatremendousamountofworkthatmustbedone,inboththeresearchandapplicationrealms,
ifweasinterfacedesignersaregoingtohelpourusersmovebeyondtheGUI.
FIGURE1.10Artistsdepictionofwhatarealdesktopholographicinterfacemightlooklike.Not
ethattheinterfacehasfewerrestrictionsonspacethanphysicaldevices,andyoucouldeasil
yimaginethatitmightbesharedwithagesture-basedinterfacetoallowmanipulationofspace
withthehands.(CourtesyofApimac.)
1IntroductiontotheHumanFactorsofNontraditionalInterfaces22

REFERENCESBettio,F.,Frexia,F.,Giachetti,A.,Gobbetti,E.,Pintore,G.,Zanetti,G.,Bal
ogh,T.,Forgacs,T.,Agocs,T.,&Bouvier,E.(2006).Aholographiccollaborativemedicalvisu
alizationsystem.StudiesinHealthTechnologyandInformatics119:5254.Burnette,E.(2006)
.NintendotoGamers:DoNotLetGoofWiiRemote.EdBurnettesDevConnection,ZDNetBloghttp://blo
gs.zdnet.com/Burnette/?p=210.Donoghue,J.P.,Nurmikko,A.,Black,M.,&Hochberg,L.R.(2
007).Assistivetechnologyandroboticcontrolusingmotorcortexensemble-basedneuralint
erfacesystemsinhumanswithtetraplegia.JournalofPhysiology579(3):60311.Friedrich,M.
J.(2004).Harnessingbrainsignalsshowspromiseforhelpingpatientswithparalysis.Journ
aloftheAmericanMedicalAssociation291(18):217981.InternationalStandardsOrganizatio
n.(1998).ErgonomicRequirementsforOfficeWorkwithVisualDisplayTerminal(VDTs)Part11:G
uidanceonUsability(ISO9241-11(E)).Geneva,Switzerland.Kurtenbach,G.P.,Balakrishna
n,R.,&Fitzmaurice,G.W.(2007).Graphicaluserinter-facewidgetsviewableandreadablefr
ommultipleviewpointsinavolumetricdisplay.USPatent720599.Washington,DC:U.S.Patent
andTrademarkOffice.
FIGURE1.11Projectedkeyboardinterface.Whilenotholographicinthestricttechnicalsens
eoftheterm,thesekindsofinterfacesallowlargerinterfacestobecreatedwheretherearesp
acelimitations,asmightbethecaseinaPDA.Source:Retrievedfromwww.virtuallaserkeyboa
rd.com.
References23

Mayhew,D.J.(1997).PrinciplesandGuidelinesinSoftwareUserInterfaceDesign.Englewood
Cliffs,NJ:PrenticeHall.Nielsen,J.(1999).DesigningWebUsability:ThePracticeofSimpl
icity.Berkeley,CA:Peach-pitPress.Norman,D.A.(2002).TheDesignofEverydayThings.New
York:BasicBooks.Raskin,J.(2000).TheHumaneInterface:NewDirectionsforDesigningInte
ractiveSystems.Boston:Addison-Wesley.Schneiderman,B.,&Plaisant,C.(2004).Designin
gtheUserInterface:StrategiesforEffectiveHumanComputerInteraction.Boston:Addison-W
esley.Tanaka,K.,Matsunaga,K.,&Wang,H.O.(2005).Electroencephalogram-basedcontrolo
fanelectricwheelchair.IEEETransactionsonRobotics21:76266.
1IntroductiontotheHumanFactorsofNontraditionalInterfaces24

2CHAPTERHapticInterfacesMarciaK.OMalley,AbhishekGuptaIngeneralthewordhapticrefersto
thesenseoftouch.Thissenseisessentiallytwofold,includingbothcutaneoustouchandkine
sthetictouch.Cutaneoustouchreferstothesensationofsurfacefeaturesandtactilepercep
tionandisusuallyconveyedthroughtheskin.Kinesthetictouchsensations,whicharisewith
inthemusclesandtendons,allowustointerpretwhereourlimbsareinspaceandinrelationtoo
urselves.Hapticsensationcombinesbothtactileandkinestheticsensations.Thesenseofto
uchisoneofthemostinformativesensesthathumanspossess.Mechanicalinteractionwithagi
venenvironmentisvitalwhenasenseofpresenceisdesired,orwhenauserwishestomanipulate
objectswithinaremoteorvirtualenvironmentwithmanualdexterity.Thehapticdisplay,orf
orce-reflectinginterface,istheroboticdevicethatallowstheusertointeractwithavirtu
alenvironmentorteleoperatedremotesystem.Thehapticinterfacecon-sistsofareal-timed
isplayofavirtualorremoteenvironmentandamanipulator,whichservesastheinterfacebetw
eenthehumanoperatorandthesimulation.Theusermoveswithinthevirtualorremoteenvironm
entbymovingtheroboticdevice.Hapticfeedback,whichisessentiallyforceortouchfeedbac
kinamanmachineinterface,allowscomputersimulationsofvarioustaskstorelayrealistic,t
angiblesensationstoauser.Hapticfeedbackallowsobjectstypicallysimulatedvisuallyto
takeonactualphysicalproperties,suchasmass,hardness,andtexture.Itisalsopossibleto
realisticallysimulategravitationalfieldsaswellasanyotherphysicalsensationthatcan
bemathematicallyrepresented.Withtheincorpora-tionofhapticfeedbackintovirtualorre
moteenvironments,usershavetheabilitytopush,pull,feel,andmanipulateobjectsinvirtu
alspaceratherthanjustseearepresentationonavideoscreen.

Theapplicationofhapticinterfacesinareassuchascomputer-aideddesignandmanufacturin
g(CAD/CAM),designprototyping,andallowinguserstomanip-ulatevirtualobjectsbeforema
nufacturingthemenhancesproductionevaluation.Alongthesamelines,theusersofsimulato
rsfortraininginsurgicalprocedures,controlpaneloperations,andhostileworkenvironme
ntsbenefitfromsuchacapability(Meech&Solomonides,1996).Hapticinterfacescanalsobee
mployedtoprovideforcefeedbackduringexecutionofremotetasks(knownasteleopera-tion)
suchastelesurgeryorhazardouswasteremoval.Withsuchawiderangeofapplications,theben
efitsofhapticfeedbackareeasilyrecognizable.2.1NATUREOFTHEINTERFACEThissectiondes
cribesthefundamentalnatureofhapticinterfaces,introducingthebasiccomponentsofahap
ticdisplaysystemanddescribingindetailthecap-abilitiesofthehumanhapticsensingsyst
em.2.1.1FundamentalsofHapticInterfacesAhapticinterfacecomprisesaroboticmechanism
alongwithsensorstodeterminethehumanoperatorsmotionandactuatorstoapplyforcestotheo
perator.Acontrollerensurestheeffectivedisplayofimpedances,asgovernedbytheopera-t
orsinteractionwithavirtualorremoteenvironment.Impedanceshouldbeunder-stoodtorepre
sentadynamic(history-dependent)relationshipbetweenvelocityandforce.Forinstance,i
fthehapticinterfaceisintendedtorepresentmanipulationofapointmass,itmustexertonth
eusershandaforceproportionaltoacceleration;ifitistorepresentsqueezingofaspring,it

mustgenerateaforceproportionaltodisplace-ment(Colgate&Brown,1994).Finally,thehap
ticvirtualenvironmentisrenderedsoastoimplementthedesiredrepresentation.HapticInt
erfaceHardwareHapticinterfacehardwareconsistsofthephysicalmechanismthatisusedtoc
ouplethehumanoperatortothevirtualorremoteenvironment.Thishardwaremaybeacommoncom
putergamingjoystick,amultiple-degree-of-freedom(DOF)stylus,awearableexoskeletond
evice,oranarrayoftactorsthatdirectlystimulatetheskinsurface.Thebasiccomponentsof
thehardwaresystemincludethemech-anism,whichdefinesthemotioncapabilitiesofthehuma
noperatorwheninter-actingwiththedevice;thesensors,whichtrackoperatormotioninthev
irtualenvironment;andtheactuators(motors),whichdisplaythedesiredforcesortextures
totheoperatorasdefinedbytheenvironmentmodel.Thefinalselectionofaparticularmechan
ism,sensor,oractuatoristypicallygovernedbythetargetapplication.Tactileandkinesth
eticinterfacesprovidetactileandkinestheticfeed-backtotheoperator,respectively,an
dwillbetreatedseparatelythroughoutthe
2HapticInterfaces26

chapter.Applicationswherebothtactileandkinestheticfeedbackisdesiredcanemploytact
iledisplaysmountedonakinestheticdisplay.HapticInterfaceControlHapticdevicesarety
picallycontrolledinoneoftwomannersimpedanceoradmittance.Impedancecontrolofaroboti
nvolvesusingmotioninputfromthemanipulatorandcalculatingthecorrespondingforcesspe
cifictoagivensys-temmodel.Forexample,whensimulatingavirtualspring,whentheusercom
-pressesaspringinthevirtualenvironment,theinterfaceappliesforcestotheusershandtha
topposehandmotionandareproportionaltospringdisplace-ment.Motiondataareavailablef
romsensorsontheroboticdeviceandaresenttosignal-conditioningboardstypicallywithina
desktoppersonalcomputerforprocessing.Theprocessingcalculationsinvolvetwodifferent
iationsoftheposi-tiondatainordertofindvelocityandacceleration,oronedifferentiati
ontogetaccelerationifvelocitysignalsareavailabledirectly(e.g.,fromatachometer).M
ostsimplesimulatedenvironmentsconsistonlyofspringsthatproduceaforceproportionalt
odisplacement,anddampersthatgenerateforcesproportionaltovelocity.Thus,ifposition
andvelocitysignalscanbeobtaineddirectlywithoutanydifferentiation,impedancecontro
loftherobotisthedesiredapproach.Admittancecontrolofarobotistheoppositeoperation.
Forcesaremeasured,usuallywithaloadcell,andarethensenttothecomputer.Calculationsa
reperformedtofindthecorrespondingmotionoftheendpointaccordingtothesimulationsequa
tionsofmotion,andpositioncontrolapproachesareusedtomovetherobotaccordingly.Solvi
ngfortheoutputpositioninvolvesoneortwointegrationsteps,dependingontheenvironment
model.Typically,integrationisamuchcleaneroperationthandifferentiation,butproblem
swithoffsetsandintegratorwinduparecommonanddetractfromthismethodofrobotcontrol.I
npractice,impedance-controlledinterfacesarebetteratsimulatingsoft,spongyenvironm
ents,whereasadmittance-controlleddevicesperformbetterwhendisplayinghardsurfaces.
CreatingaHapticEnvironmentAhapticenvironmentisdefinedviaamathematicalmodel.Forth
esimplecaseofavirtualwall,themodelofaspringandadamperinparallelistypicallyused.T
hehigherthestiffnessofthespring,thestifferthevirtualwallappearstotheuser.Usingth
eimpedancecontrolmode,whereendpointmotionismeasuredandforceisdisplayed,thepositi
onoftheendpointistrackedtodetermineiftheuserispushingonthevirtualwall.Whenthepla
neofthewalliscrossed,thecorrespondingforcethattheusershouldfeeliscalculatedaccor
dingtothemodelequation,usingpositionsensordataandvelocitydatatocalculatethemodel
unknowns.ThisvirtualwallmodelisillustratedinFigure2.1,andservesasthebuildingbloc
kformanyvirtualenvironments.Hapticrenderingwillnotbea
2.1NatureoftheInterface27

focusofthischapter.However,thoroughreviewsandintroductionstothebasiccon-ceptsofh
apticrenderingareavailable,suchastheworkbySalisburyetal.(2004).2.1.2HumanHapticS
ensingTouchcanbedefinedasthesensationevokedwhentheskinissubjectedtomechani-cal,t

hermal,chemical,orelectricalstimuli(Cholewiak&Collins,1991).Touchisunlikeanyothe
rhumansenseinthatsensoryreceptorsrelatedtotoucharenotasso-ciatedtoformasingleorg
an.Hapticreceptorsareofthreeindependentmodalities:pressure/touch(mechanoreceptio
n),heatandcold(thermoreception),andpain(nociception)(Schmidt,1977).Asthemechanor
eceptorsareresponsiblefortactilesensationofpressure/touch,andaretheprimarytarget
softactilehapticdevices,thissectionwillfocusonthepressure/touchmodality.Kinesthe
tichapticfeedbackissensedthroughreceptorsinmusclesandtendons,andisdiscussedinSec
tion2.1.3.
kbx
Wall positionActual positionof end point
Fx = kx + bx.
FIGURE2.1Graphicalrepresentationofavirtualwallmodel.Thevirtualwallisafundamental
buildingblockofahapticvirtualenvironment.Itistypicallyimplementedasaspringanddam
perinparallel.
2HapticInterfaces28

Mechanoreceptioncomprisesfoursensations:pressure,touch,vibration,andtickle.Thedi
stributionofthesereceptorsisnotuniformoverthebody.Hairless(glabrous)skinhasfivek
indsofreceptors:freereceptors(ornerveendings),Meissnerscorpuscles,Merkelsdisks,Pac
iniancorpuscles,andRuffiniendings.Inadditiontothesereceptors,hairyskinhasthehair
rootplexus(orfollicle)thatdetectsmovementonthesurfaceoftheskin.Figure2.2depictst
helocationofthesereceptorsintheskin.Eachofthesemechanoreceptorsrespondsdifferent
lytoappliedpressure/touchstimuliandtheircombinedbehaviordetermineshumanperceptio
nofpressureandvibrations.Thestudyofthesepropertiesiscrit-icaltosuccessfuldesigno
fhapticinterfacesfortemporalaswellasspatialdetec-tionand/ordiscriminationbytheus
er.Sensoryadaptationisthetendencyofasensorysystemtoadjustasaresultofrepeatedexpo
suretoaspecifictypeofstimulus.Basedontherateofsensoryadaptation,thereceptorsarec
lassifiedasslow-adapting(SA)orrapid-adapting
Horny layerEpidermicCoriumMeissnerscorpuscleMerkelsdisksPaciniancorpuscleHair foll
iclereceptorTactilediskRuffiniendingSubcutaneoustissue
FIGURE2.2Structureandlocationoftactilereceptorsintheskin.Source:FromSchmidt(1977
),withkindpermissionofSpringerScience&BusinessMedia.
2.1NatureoftheInterface29

(RA)receptors.Merkelsdisksproducealongbutirregulardischargerateinresponsetoforces
ontheskin,whereasRuffiniendingsproducearegulardischargeforasteadyload.Meissnersco
rpusclesdischargemostlyattheonsetofstimulus,andhencebestrespondtovelocity.Finall
y,Paciniancorpusclesrespondonceforeverystimulusandaregoodonlyasvibrationdetector
s,respondingbesttofrequenciesof200Hz,whichistheirloweststimulusamplitudethreshol
d(Schmidt,1977).Hence,whendesigninghigh-frequencyvibrotactilefeedback,forexample
,thebehaviorofPaciniancorpusclesmustbeconsideredtoensureproperdetectionanddiscri
minationofthestimulibytheuser,whereasatlowerfrequen-cies,thebehaviorofotherrecep
torsneedstobeconsideredaswell.Mechanoreceptorscanalsobecharacterizedbasedontheir
receptivefieldsize.Thisistheareainwhichastimuluscanexcitethereceptor,andvariesfr
om1to2mm2toupto45mm2dependingonthereceptoranditslocationonthebody.PacinianandRuf
finicorpuscleshavelargefieldsizeandhencelowspatialresolution.Ontheotherhand,Merk

eldisksandMeissnersendingsprovidemoreaccuratespatiallocalization.Thisisparticular
lyimportantintactiledisplaydesign,ascuesthatcannotbediscriminatedbytheuserwillfa
iltoconveyanyadditionalinformationaboutthesimulatedenvironment.Theskinsthermorece
ptorsaredividedintocold-andwarmth-sensitiverecep-tors.Theformerarelocatedjustben
eaththeepidermis,whilethelatterarelocatedinthedermis.Thesereceptorshaveareceptiv
efieldof1to2mmindiam-eterandaspatialresolutionthatislessthanthatofpainreceptorso
rmechano-receptors.Tissue-damagingstimulitriggernociceptors.Thesehaveareceptivef
ieldofapproximately25mm2.Hapticinterfacedesignersshouldensurethattheforcefeedbac
kissufficientforsatisfactorycompletionofataskwhileatthesametimebeingcomfortablef
ortheuser.Thisrequiresparticularattentiontotheperceptualcapabilitiesofthehumanse
nsorimotorloop,whicharediscussedinthefollowingsections.2.1.3HumanHapticPerceptio
nHumanhapticperception(ratherthansensing,discussedpreviously)istheprocessofacqui
ring,interpreting,selecting,andorganizinghapticsensoryinformation,andiscomprised
oftactileperceptionandkinesthesia(includingpropriocep-tion).Kinesthesiareferstot
hesenseofforcewithinthemusclesandtendons,andproprioceptionreferstothehumanpercep
tionofonesownbodypositionandmotion.Thesenseofpositionreferstotheangleofvariousske
letaljoints,andthesensitivityorresolutionofjointpositiondeterminestheaccuracywit
hwhichwecancontrolourlimbs.Tactileperceptionspecificallyconcernstheacquisitionan
dinterpretationofsensationsrealizedthroughthemechanorecep-torsoftheskin.Manyscie
ntistshavestudiedhumanperceptionthresholdsinordertounder-standthelimitsofourabil
ities.Sincethehumansenseoftouchinherentlytakesplace
2HapticInterfaces30

throughtwoseparatepathways,namelykinestheticandtactileinformationpathways,percep
tionstudiesinthehumansenseoftouchcanalsobecategorizedwithrespecttothefundamental
informationcontainedwithinthestimuli.Irrespectiveofwhetherthedominantpathwayiski
nestheticortactile,existingstudieshavelookedatdiscrim-inationoridentificationofs
urfaceproperties(e.g.,shapeandsurfacetexture)andvol-umetricproperties(e.g.,massa
ndsponginess)ofobjects.Currentstudiesofthejustnoticeabledifferences(JNDs)forkine
stheticandtactilesenseshavefocusedondiscriminationofgeometries,textures,andvolumetricpropertiesofobjectsheldbythehuman,orhavefocusedondiscriminationofthesubjec
tsownlimbmovements;seeDurlachandMavor(1995)foracompre-hensivereview.TheJNDisthesm
allestdifferenceinaspecifiedmodalityofsensoryinputthatisdetectablebyahuman.Itisa
lsoreferredtoasthedifferencelimenorthedifferentialthreshold.Earlykinestheticstud
iesbyClarkandcolleagueandJonesandHunter(Clark&Horch,1986;Clark,1992;Jones&Hunter
,1992)investigatedhumanperceptionoflimbpositionsandconcludedthathumansarecapable
ofdetectingjointrota-tionsofafractionofadegreeperformedoverasecondoftimeinterval
.JonesandHunter(1992)alsoreportedthedifferentialthresholdforlimbmovementas8perce
nt.FurtherpsychophysicalexperimentsconductedbyTanandcolleagues(1994)determinedth
eJNDforthefingerjointsas2.5percent,forthewristandelbowas2percent,andfortheshould
eras0.8percent.Durlachandcolleagues(1989)investigatedthelengthresolutionforrigid
objectsheldinapinchgraspbetweenthethumbandtheforefinger(Durlachetal.,1989).Commo
nlyacceptedperceptionthresholdsforlengthresolutionaregivenasabout1mmforareferenc
elengthof10mm,increasingto2to4mmforareferencelengthof80mm.Forpurposesofcompariso
n,thethicknessofapennyisapproximately1.57mm,whereasitsdiameterisabout19mm.Latere
xperimentsfocusingonobjectsizecharacterizedtheeffectofvaryinglevelsofforceoutput
andvirtualsurfacestiffnessontheabilityofhumansubjectstoperformsizeidentification
andsizediscriminationtasksinasimulatedenviron-ment(OMalley&Goldfarb,2002,2004;Upp
ermanetal.,2004;OMalley&Upper-man,2006).Inanapplicationwherehapticcuesareprovided
fornavigation,detectionofthestimuliisimportantandnottheirdiscriminationfromeacho
ther.Insuchascenario,lowforcesandvirtualsurfacestiffnessmaysuffice.Notethatthese
cueswillfeelsoftorsquishyduetolowforceandstiffnesslevels.Ontheotherhand,taskstha
trequiresizediscrimination,suchaspalpationinamedicaltrainer,requirelargerforcean
dstiffnessvaluesand,consequently,ahapticinter-facecapableoflargerforceoutputando
fhigherquality.Recently,McKnightandcolleagues(2004)extendedthesepsychophysicalsi
zediscriminationexperimentstoincludetwo-andthree-fingergrasps.Thebandwidthofthek

inestheticsensingsystemhasbeenestimatedat20to30Hz(Brooks,1990).Inotherwords,thek
inestheticsensingsystemcannotsensemovementsthathappenmorefrequentlythan30timesin
asecond.Hence,in
2.1NatureoftheInterface31

studiesonperceptionofhigh-frequencyvibrationsandsurfacetexture,thetactilepathway
servesastheprimaryinformationchannel,whereasthekinestheticinformationissupplemen
tary.Earlytactileperceptionstudiesconcludedthatthespatialresolutiononthefingerpa
disabout0.15mmforlocalizationofapointstimulus(Loomis,1979)andabout1mmforthetwo-p
ointlimen(Johnson&Phil-lips,1981).Otherpartsofthebodyhavemuchlessspatialresoluti
on.Forexam-ple,thepalmcannotdiscriminatebetweentwopointsthatarelessthan11mmapart
(Shimoga,1993).Arelatedmeasure,thesuccessivenesslimen(SL),isthetimethresholdforw
hichsubjectsareabletodetecttwosuccessivestimuli.AnapproximateSLvalueforthemechan
oreceptorsis5msec,witharequiredintervalof20msectoperceivetheorderofthestimuli.Th
ehumanthresholdforthedetectionofvibra-tionofasingleprobeisreportedtobeabout28dBf
orthe0.4-to3-Hzrange.Anincreaseinlevelof6dBrepresentsadoublingofamplitude,regard
lessoftheinitiallevel.Achangeof20dBrepresentsachangeinamplitudebyafactorof10.Thi
sthresholdisshowntodecreaseattherateof5dBperoctaveinthe3-to30-Hzrange,andatthera
teof12dBperoctaveinthe30-toabout250-Hzrange,withanincreaseforhigherfrequencies(R
abinowitzetal.,1987;Bolanowskietal.,1988).2.1.4SensoryMotorControlInadditiontota
ctileandkinestheticsensing,thehumanhapticsystemincludesamotorsubsystem.Explorato
rytasksaredominatedbythesensorialpartofthesensorymotorloop,whereasmanipulationta
sksaredominatedbythemotorpart(Jandura&Srinivasan,1994).Thekeyaspectsofhumansenso
rimotorcon-trolaremaximumforceexertion;force-trackingresolution;compliance,force
,andmassresolution;fingerandhandmechanicalimpedance;andforcecontrolbandwidth.Max
imumForceExertionAmaximumgraspingforceof400Nformalesand228Nforfemaleswasmeasured
inastudybyAnandcoworkers(Anetal.,1986).Inastudyonmaxi-mumforceexertionbythepoint
er,index,andringfingers,itwasfoundthatthemaximumforceexertedbythepointerandindex
fingerswasabout50N,whereastheringfingerexertedamaximumforceof40N(Sutteretal.,198
9).Theseforceswerefoundtobeconstantover0to80degreesofthemetacarpal(MCP)jointangl
e.Thisworkwaslaterextendedtoincludetheproximal-interphalangeal(PIP)jointsandMCPj
oints,aswellasthewrist,elbow,andshoul-der(witharmextendedtothesideandinfront)(Ta
netal.,1994).Notethatthemaximumforceexertioncapabilityisdependentontheuserspostur
e.Itwasfoundthatmaximumforceexertiongrowsfromthemostdistaljointinthepalmtothemos
tproximaljoint(shoulder).Inaddition,itwasfoundthatcontrollability
2HapticInterfaces32

overthemaximumforcedecreasedfromtheshouldertothePIPjoint.Inordertoensureusersafe
ty,ahapticinterfaceshouldneverapplyforcesthattheusercannotsuccessfullycounter.Su
stainedForceExertionProlongedexertionofmaximumforceleadstofatigue.Fatigueisanimp
ortantconsiderationwhendesigningfeedbackforappli-cationslikedatavisualizationwhe
reforcefeedbackmaybepresentforextendedperiodsoftime.Wikerandcolleagues(1989)perf
ormedastudyoftherelationshipbetweenfatigueduringgraspingasafunctionofforcemagnit
ude,restduration,andprogressionofthetask.Thetestsshowedadirectcorrelationbetween
magni-tudeofdiscomfortandmagnitudeofpinchforce.Thework-versus-restratiowasnotfou
ndtobeimportantforlowforcesbutwaseffectiveinreducingfatigueforhighpinchforces.Fo
rce-TrackingResolutionForce-trackingresolutionrepresentsthehumanabilitytocontrol
contactforcesinfollowingatargetforceprofile.SrinivasanandChen(1993)studiedfinger
tipforcetrackinginsubjectsusingbothconstantandtime-varying(rampandsinusoid)force
s.Forsomeparticipants,acomputermonitorprovidedadisplayofboththeactualandtargetfo
rces.Subjectsalsoperformedthetestsunderlocalcutaneousanesthesia.Itwasfoundthatwh
ennovisualfeedbackwasavailable,theabsoluteerrorrateincreasedwithtargetmagnitude.

Whenvisualfeedbackwaspresent,theerrorratedidnotdependontargetmagnitude.Complianc
eResolutionCompliance,orsoftness,resolutioniscriticalincertainapplicationssuchas
train-ingforpalpationtasksortelesurgery,sincemanymedicalproceduresrequireaccurat
ediscriminationoftissueproperties.Thefollowingdiscussionpresentsashortsummaryoft
heliteratureoncomplianceresolutionbothwithandwithoutthepresenceofadditionalvisua
lorauditoryclues.Ifahapticinterfaceistobeusedforexploratorytasksthatrequirediscr
iminationamongobjectsbasedontheircompliance,thendesignersshouldensurethatthesimu
latedvirtualobjectsappearsufficientlydifferenttothehumanoperator.Humanperception
ofcomplianceinvolvesboththekinestheticandtactilechannelssincespatialpressuredist
ributionwithinthecontactregionsensedthroughthetactilereceptorsplaysafundamentalr
oleincomplianceperception.However,fordeformableobjectswithrigidsurfaces,theinfor
mationavailablethroughthetactilesenseislimitedandkinestheticinformationagainbeco
mesthedominantinformationchannel.Insuchcases,humanperceptualresolutionismuchlowe
rthanthecasesforcompliantobjectswithdeformablesurfaces(Srinivasan&LaMotte,1995).
Instudiesinvolvingdeformableobjectswithrigidsurfaces,JonesandHunter(1990,1992)re
portedthedifferentialthresholdsforstiffnessas23percent.Tanandcolleagues(1992,199
3)observedthatforsuch
2.1NatureoftheInterface33

objectsheldinapinchgrasp,theJNDforcomplianceisabout5to15percentwhenthedisplaceme
ntrangeisfixed,and22percentwhenthedisplacementrangeisrandomlyvaried.Moreover,the
yreportedaminimumstiffnessvalueof25N/m(Newtonspermeter)foranobjecttobeperceiveda
srigid(Tanetal.,1994).Infurtherstudies,Tanandcoworkers(1995)investigatedtheeffec
tofforceworkcuesonstiffnessperceptionandconcludedthatJNDcanbecomeashighas99perce
ntwhenthesecuesareeliminated.Investigatingtheeffectofothercuesoncompliancepercep
tion,DiFrancoandcolleagues(1997)observedtheimportanceofauditorycuesassociatedwit
htappinghardersurfacesandcon-cludedthattheobjectsareperceivedtobestifferwhensuch
auditorycuesarepresent.Inasimilarstudy,Srinivasanandcoworkers(1996)reporteddomin
ancevisioninhumanstiffnessperception.Inrelatedstudies,Durfeeetal.(1997)inves-tig
atedtheinfluenceofhapticandvisualdisplaysonstiffnessperception,whileOMalleyandGol
dfarb(2004)studiedtheimplicationsofsurfacestiffnessforsizeidentificationandperce
ivedsurfacehardnessinhapticinterfaces.Observingtheimportanceoftheinitialforcerat
eofchangeinstiffnessperception,Lawrenceandcolleagues(2000)proposedanewmetricforh
umanperceptionofstiffness,calledratehardness.ForceResolutionInrelatedexperiments
,humanperceptuallimitationsofcontactforceperceptionwhenthekinestheticsenseactsast
heprimaryinformationchannelhavebeenstudied.TheJNDforcontactforceisshowntobe5to15p
ercentofthereferenceforceoverawiderangeofconditions(Jones,1989;Pangetal.,1991;Ta
netal.,1992).AccompanyingexperimentsrevealedaJNDvalueofabout10percentformanualre
solutionofmass(Tanetal.,1994),whileaJNDvalueofabout34percenthasbeenobservedforma
nualresolutionofviscosity(Jones&Hunter,1993).Recently,Barbagliandcolleagues(2006
)studiedthediscriminationthresh-oldsofforcedirectionandreportedvaluesof25.6perce
ntand18.4percentforforcefeedbackonlyandvisuallyaugmentedforcefeedbackconditions,
respec-tively.Specialattentionisrequiredwhendesigningfeedbackforapplicationslike
graspingandmanipulation,wheresubtlechangesinforcecanbeimportant,suchasinmakingth
edifferencebetweenholdinganddroppinganobjectinthevirtualenvironment.MechanicalIm
pedanceTheimpedanceofthehumanoperatorsarmorfingerplaysanimportantroleindeterminin
ghowwelltheinterfaceperformsinreplicatingthedesiredcontactforceatthehumanmachinec
ontactpoint.HajianandHowe(1997)studiedthefingertipimpedanceofhumanstowardbuildin
gafingerhapticinterface.Overallsubjectsandforces,theyestimatedtheequivalentmasst
ovaryfrom3.5to8.7g,theequivalentdampingfrom4.02to7.4Ns/m,andstiffnessfrom255to
2HapticInterfaces34

1,255N/m.Itwasnotedthatthedampingandstiffnessincreasedlinearlywithforce.Insimila
rwork,Speichandcolleagues(2005)builtandcomparedtwo-andfive-DOFmodelsofahumanarmt
owarddesignofateleoperationinterface.SensingandControlBandwidthSensingbandwidthr
eferstothefrequencywithwhichtactileand/orkinestheticstimuliaresensed,andcontrolb
andwidthreferstothefrequenciesatwhichthehumancanrespondandvoluntarilyinitiatemot
ionofthelimbs.Inhumans,theinput(sensory)bandwidthismuchlargerthantheoutputbandwi
dth.Asnotedearlier,itiscriticaltoensurethatthelevelofhapticfeedbackissufficientf
ortaskcompletionwhilebeingcomfortablefortheuser.Inareviewpaper,Shimogashowedthat
thehandsandfingershaveaforceexertionbandwidthof5to10Hz,comparedtoakinestheticsen
singbandwidthof20to30Hz(Shimoga,1992).Tactilesensinghasabandwidthof0to400Hz.Keep
ingthisinmind,ifwedesignanapplicationthatrequiresrepetitiveforceexertionbytheuse
r,toguaranteeusercomforttherequiredrateshouldnotbemorethan5to10timesasecond.Simi
larly,anykinestheticfeedbacktotheusershouldbelimitedto20to30Hz.2.2TECHNOLOGYOFTH
EINTERFACEHapticinterfacesarearelativelynewtechnology,withincreaseduseforhumanin
teractionwithvirtualenvironmentssincetheearly1990s.Aprimaryindicatoroftheprolife
rationofhapticdevicesisthenumberofcompaniesthatnowmarketdevices,includingSensabl
eTechnologies,Immersion,ForceDimension,Quanser,andNovint,amongothers.Thecommerci
alizationofhapticdevicesisdueprimar-ilytotechnologicaladvancesthathavereducedthe
costofnecessarycomponentsinhapticsystems,includingmaterials,actuation,sensing,an
dcomputercontrolplatforms.Novelmaterials,suchascarbonfibertubing,haveenabledthed
esignandfab-ricationoflight-weightyetstiffkinematicmechanismsthatarewellsuitedto
thekinesthetictypeofhapticdisplay.Power-denseactuators,suchasbrushlessDCmotors,h
aveallowedforincreasedmagnitudeforceoutputfromhapticdeviceswithminimaltrade-offs
intermsofweight.However,itshouldbenotedthatactuationtechnologyisstillakeylimitat
ioninhapticdevicedesign,sincelargeforcesandtorquesobtainedviadirectdriveactuatio
nareoftendesired,whilestillachievingminimalinertia(mass)inthemechanism.Improveds
ensortechnologyhasalsoenabledanincreaseintheavailabilityofhigh-qualityhapticinte
rfacehardware.Thekeyrequirementofsensorsforhapticapplicationsishighresolu-tion,a
ndmanysolutionssuchasopticalencodersandnoncontactpotentiometersareprovidingincre
asedresolutionwithoutcompromisingtheback-drivabilityofhapticdevicesduetotheirnon
contactnature.
2.2TechnologyoftheInterface35

Thefinalsetoftechnologicaladvancesisintheareaofcomputationalplat-forms.First,dat
aacquisitionsystems,whichenabletransformationfromanaloganddigitalsignalscommonto
thesensorsandactuatorstothedigitalcomputationcarriedoutbythecontrolcomputer,area
chievinghigherandhigherresolutions.Second,real-timecomputationplatformsandincrea
singprocessorspeedsareenablinghapticdisplays(typicallyrenderedatarateof1,000Hz)t
oexhibitincreasinglygreatercomplexityintermsofcomputationandmodelrealism.Thisint
urnbroadenstherangeofapplicationsforwhichhapticfeedbackimplementa-tionisnowfeasi
ble.Finally,embeddedprocessorsandembeddedcomputingareenablinghapticdevicestobemo
reportable.2.3CURRENTINTERFACEIMPLEMENTATIONSOverthelastseveralyears,avarietyofh
apticinterfaceshavebeendevelopedforvariousapplications.Theyrangefromsimplesingle
-DOFdevicesforresearch(Lawrence&Chapel,1994)tocomplex,multi-DOFwearabledevices(F
risolietal.,2005;Kimetal.,2005;Gupta&OMalley,2006).DOFreferstothenumberofvari-abl
esrequiredtocompletelydefinetheposeofarobot.Ahigher-DOFdevicehasalargerworkspacet
hephysicalspacewithinwhichtherobotendpointmovesascomparedtoalow-DOFdeviceofsimila
rsize.Hapticdevicesarealsousedinvariousapplications(Haywardetal.,2004).Forinstan
ce,hapticinterfaceshavebeenemployedforaugmentationofgraphicaluserinterfaces(GUIs
)(Smyth&Kirkpatrick,2006),scientificdatavisualization(Brooksetal.,1990),enhancem
entofnanomanipulationsystems(Falvoetal.,1996),visualarts(OModhrain,2000),CAD/CAM(
Nahvietal.,1998;McNeelyetal.,1999),educationandtraining,partic-ularlysurgicaltra
ining(Delpetal.,1997),masterinterfacesinteleoperation(Kimetal.,2005),rehabilitat
ion(Bergamasco&Avizzano,1997;Krebsetal.,1998),andthescientificstudyoftouch(Hogan
etal.,1990;Weisenbergeretal.,2000).ThePHANToMdesktophapticinterface(SensableTech
nologies),showninFigure2.3,isprobablythemostcommonlyusedhapticinterface.Itisapen

-orstylus-typehapticinterface,wheretheoperatorgripsthestylusattheendoftherobotdu
ringhapticexploration.ThePHANToMdesktopdevicehasaworkspaceofabout160W120H120Dmm.
Thedeviceprovidesfeedbacktotheopera-torinthreedimensionswithamaximumexertablefor
cecapabilityof1.8foot-pounds(lbf)(7.9N)andacontinuousexertableforcecapability(ov
er24hours)of0.4lbf(1.75N).Anumberofdevicemodelsareavailablethatvaryinworkspacean
dforceoutputspecifications.Severalotherhapticinterfacesarecommerciallyavailable,
suchasthesix-DOFDeltahapticdevice(ForceDimension),thethree-DOFplanarpantograph(Q
uanser),andtheforcefeedbackhandcontroller(MPBTechnologies)(Figure2.3).Thecommonf
eatureofmostcommerciallyavailablehapticdevicesisthattheyarepointcontactdevices,i
nthattheendpointoftherobotismappedtoa
2HapticInterfaces36

FIGURE2.3Selectedcommerciallyavailablehapticinterfaces.(a)PHANToMdesktop,(b)sixDOFDeltahapticinterface,(c)three-DOFplanarpantograph,and(d)forcefeedbackhandcont
roller.(Courtesyof(a)SensAbleTechnologies,Inc.;(b)FrancoisConti;(c)QuanserInc.;a
nd(d)MPBTechnologiesInc.
2.3CurrentInterfaceImplementations37

positioninthevirtualenvironmentandforcesareappliedbacktotheuseratthesamepoint.Th
us,withinthevirtualenvironment,theusercaninteractwithonlyonepoint.Onecanthinkoft
hisasbeingsimilartointeractingwithobjectsintherealworldwiththeaidoftoolslikeapen
,screwdriver,orscalpel.Evenwiththislimitation,thesetypesofdeviceshavebeenemploye
dinapplicationssuchassci-entificvisualization,augmentationofGUIs,CAD,andpsychoph
ysicalstudies.ThechoiceofaspecificdevicedependsonthedesiredworkspaceandDOF,thety
peofforcefeedbackdesired,andthemagnitudeofforcestobedisplayed.Forexample,thePHAN
ToMcanmovewithinthethree-dimensional(3D)physicalspace,butcanapplyforcesonlyonthe
usershands.Incomparison,thethree-DOFpanto-graphisrestrictedtomovinginaplane,butca
napplyatorqueorwrenchtotheusershandinadditiontoforcesinthetwoplanardirections.Int
hefollowingsubsections,wetakeacloserlookatselectedcurrentimple-mentationsofhapti
cinterfaces.Theexamplespresentedhavebeenchosentodem-onstrateessentialfeatures,an
ddonotnecessarilyrepresentthestateoftheartbutratherbasicfeaturesoftheirrespectiv
ecategories.Thesedevicesdemonstratethewiderangeoftechnologiesinvolvedinhaptics.2
.3.1NonportableHapticInterfacesHapticJoysticksJoysticksarewidelyusedassimpleinpu
tdevicesforcomputergraphics,indus-trialcontrol,andentertainment.Mostgeneral-purp
osejoystickshavetwoDOFwithahandlethattheusercanoperate.Thehandleissupportedatone
endbyasphericaljointandattheotherbytwoslidingcontacts(Figure2.4).Hapticjoy-stick
svaryinbothmechanicaldesignandactuationmechanisms.AdelsteinandRosen(1992)develop
edasphericalconfigurationhapticjoystickforastudyofhandtremors.Sphericaljoysticks
,asthenameimplies,haveasphere-shapedworkspace.Cartesianjoysticks,ontheotherhand,
havetwoorthreeorthogonalaxesthatallowtheentirebaseofthehandletotranslate.Anexamp
leisthethree-DOFCartesianjoystickcomprisingamovingplatformthatslidesusingguiding
blocksandrailsproposedbyEllisandcolleagues(1996).Themovingblocksupportsanelectri
cmotorthatactuatesthethirdDOF(rotationaboutthez-axis).ThisCartesianjoystick(Figu
re2.5)hasasquare-shapedplanarworkspace,andeachaxisisactuatedusingDCmotorsandacab
letransmission.Otherexamplesofhapticjoysticksincludeafour-DOFjoystickbasedontheS
tewartplatform(Millmanetal.,1993)andamagneticallylevitatedjoystick(Salcudean&Vla
ar,1994).Themagneticallylevitatedjoystickhasnofrictionatallandisparticularlysuit
edfordisplayofsmallforcesandstiffcontact.Thesehapticjoysticksarepointcontactdevi
ces,andeachtypevariesaccord-ingtoworkspaceshapeandsize.Whilethesphericaljoystick
canbeusedwithjustwristmovements,Cartesianjoysticksrequiretheusertoemployotherjoi
ntsof

2HapticInterfaces38

thearm,liketheelbowortheshoulder.Consequently,theworkspaceandforceoutputofCartes
ianjoystickscanbegreaterthanthatofsimilarlysizedsphericalmodels.Notethatmostcomm
erciallyavailableforcefeedbackjoysticks,typicallymarketedforcomputergamingapplic
ations,lackthequalitynecessarytoachieve
HandleSwing armAxis #1Axis #2SphericaljointPotentiometerYXZO
FIGURE2.4Two-DOFslottedswing rmjoystick.Source:FromAdelstein ndRosen(1992).
FIGURE2.5Three-DOFC rtesi njoystick.Thejoystickcomprises centr lst geth tmovesus
ing guiding
lock ndr ils.Source:FromElliset l.(1996).
2.3CurrentInterf ceImplement tions39

high-qu lityfeed
ckforexplor toryorm nipul tivet sks.Whiletheym ysuf-ficetoprov

ideh pticcuesth tc n
edetected,high-qu lityh rdw re nd f stcomputerpl tform rene
cess rytoensureproperdiscrimin tionofcues.Pen-B sedM stersPen-
sedh pticdevices
llowinter ctionwiththevirtu lenvironmentthroughtoolssuch s pen(orpointer)orsc l
pel(insurgic lsimul tions).Thesedevices recomp ctwith worksp cel rgerth nth tofs
pheric l ndm gnetic llylevi-t tedjoysticks ndh vethreetosixDOF.The
est-knownex m
pleof pen-
sedh pticinterf ceisthePHANToM,mentionede rlierinthissection.Origin
llydeveloped
yM ssie ndS lis
ury(1994),thePHANToMis nelectric lly ctu- tedseri l
-feed
ckro
otic rmth tendswith fingergim
lsupportth tc n
erepl cedwith stylus(
Figure2.6).Thegim
lorient tionisp ssive ndtheseri l rm ppliestr nsl tion lforce
stotheoper torsfingertiporh nd.Asix-DOFinterf ceth tc n pplyforces swell storques
totheoper torispresented
T
le tt chmentFinger attachmentThimblegimbalCable and pulleyForce feedback armP
osition encoderAnalog controllinesDC electricalactuatorPHANTOM
FIGURE2.6SchematicofthePHANToMdesktophapticinterface.Source:FromMassieandSalisbu
ry(1994).
2HapticInterfaces40

inIwata(1993),extendingthecomplexityofforceandtorqueinteractionsbetweenahumanope
ratorandtheremoteorvirtualenvironment.Floor-andCeiling-MountedInterfacesGenerall
y,floor-andceiling-mountedinterfacesarelargerandmorecomplexandexpensivethandeskt
opdevices.Theyhavealargeforceoutput,andasaresultusersafetybecomescritical.Thisis
especiallytrueforexoskeletonswheretheoperatorisinsidethedeviceworkspaceatalltime
s.Figure2.7showsoneofthefirstgeneralizedmasterarmsthatwasdevelopedattheNationalA
eronauticalandSpaceAdministration(NASA)JetPropulsionLaboratory(JPL)(Bejczy&Salis
bury,1980).Itisasix-DOFinterfacewithathree-axishandgripthatslidesandrotatesabout
afixedsupportattachedtothefloor.Thehandgripcanapplyforcesupto9.8Nandtorquesupto0
.5N/m.TheJPLdeviceisanotherexampleofapointcontacthapticinterfacewheretheforcesar
eappliedattheusershands.Ascomparedtojoysticksordesktopdevices,though,itprovidesam
uchlargerworkvolumewithgreaterforceoutputcapabilities,coupledwithgreaterfreedomo
farmmovement.Theselargerdevicesareusefulforremotelymanipulatinglargeroboticmanip
ulatorslikethoseusedinspace.AnexampleofagroundedexoskeletalhapticinterfaceistheM

AHIarmexo-skeleton(Figure2.8)builtatRiceUniversity(Gupta&OMalley,2006;Sledd&OMalle
y,2006).Thisfive-DOFexoskeletonwasdesignedprimarilyforrehabilita-tionandtraining
invirtualenvironments.Thedeviceencompassesmostofthehumanarmworkspaceandcanindepe
ndentlyapplyforcestotheelbow,forearm,orwristjoints.Notethatthisisnolongerapointc
ontactdevice,butcanprovideindependentlycontrolledfeedbacktovarioushumanjoints.Th
isfeaturemakesitextremelysuitableasarehabilitationinterfacethatallowsthetherapis
ttofocustreatmentonisolatedjoints.
FIGURE2.7Six-DOFJPLarmmaster.Twohandcontrollersusedbyahumanoperator.Source:FromOM
alleyandAmbrose(2003).
2.3CurrentInterfaceImplementations41

2.3.2PortableHapticInterfacesAllelementsofportablehapticinterfacesarewornbytheus
er.Basedontheirmechanicalgrounding,theycanbeclassifiedasarmexoskeletonsorhandmas
ters.Armexoskeletonsaretypicallyattachedtoabackplateandtotheforearm.Handmas-ters
,ontheotherhand,areattachedtotheuserswristorpalm.Ascomparedtopointcontactdevices,
exoskeletaldevicesarecapableofmeasuringthelocationofvarioushumanjointsandcanprov
idefeedbackatmultiplelocations.Thus,withanexoskele-ton-typeinterfacetheuserisnol
ongerrestrictedtointeractingwithasinglepointintheworkspace,butcanusethewholearm,
aswithanarmexoskeleton,orgraspandmanipulatemultidimensionalobjectsusingahandmast
er.Inaddition,wearabledeviceshaveaworkspacethatiscomparabletothenaturalhumanwork
space.OneoftheearliestmodernhapticarmexoskeletonswasdevelopedbyBerga-mascoandcol
leagues(Bergamascoetal.,1994).Thefive-DOFarmprovidesfeedbacktotheshoulder,elbow,
andforearmjointsusingDCmotorsandacomplexcabletrans-mission.Theusercontrolstheexo
skeletonthroughahandleattachedtothelastrigidlink.Thedeviceweighs10kgandcanapplyt
orqueupto20N/mattheshoulder,10N/mattheelbow,and2N/matthewristjoint.Recently,Berg
amascoandcollea-gues(Frisolietal.,2005)developedanewerversionofthedevice,theLigh
tExoskeleton(Figure2.9),whichhasimprovedweightandtorqueoutputproperties.
FIGURE2.8MAHIhapticarmexoskeleton.Thefive-DOFarmexoskeletonappliesforcestotheope
ratorselbow,forearm,andwristjoints.Source:FromSleddandOMalley(2006);#2006IEEE.
2HapticInterfaces42

AnexampleofaportablehandmasteristheRutgersMasterIIbuiltatRutgersUniversity(Figur
e2.10).TheRutgersMasterIIincorporatestherobotsactuatorsintothepalmoftheuser,there
byeliminatingtheneedforatransmissionorbulkycableroutingoverthebacksofthefingerti
ps.Thetotalweightoftheinterfaceisabout100g,anditcanapplyforcesofupto4Natthefinge
rtip.Withthishandmaster,thepositionsofeachofthefourfingerscanbeseparatelymappedi
nthevirtualorremoteenvironmentandrespectiveforcesdisplayedbacktotheuser.Thismake
sitanidealinterfacefortaskswheregraspormanipulationofobjectsisdesirable.Examples
ofsuchanapplicationincludepalpation,virtualtoursofhomesandmuseums,andremotemanip
ulationofroboticgrippers.Onedrawbackofthedesignisalimitationonthetightnessoftheg
ripthatcanbeachievedduetolocationoftheactuatorswithinthepalm.2.3.3TactileInterfa
cesTactileinterfacesconveytactualinformation,thatis,informationrelatedtoheat,pre
ssure,vibration,andpain.Justlikethewiderangeofstimulidisplayedbytac-tileinterfac
es,theinterfacesthemselvescomeinvariousdesignswithavarietyofsensingandactuationt
echnologies.Hence,classificationoftactileinterfacesisnearlyimpossible,andnosingl
einterfaceisrepresentativeofthestateofartintactileinterfacedesign.Mosttactileint
erfacesprovidefeedbacktothefingertipsoftheoperator,althoughsomeinterfacesintende
dforotherpartsofthebody,like
(a)

Link 1AXIS 1AXIS 2AXIS 4AXIS 3

AXIS 51 serisorizedaxis
Link 2Link 3Link 4Link 5Link 0
FIGURE2.9LightExoskeleton(L-Exos).(a)Kinematics.(b)Finalexoskeleton.Source:FromF
risolietal.(2005);#2005IEEE.
2.3CurrentInterfaceImplementations43

theback,havealsobeenimplemented.Inthissection,wetakealookatvibrotac-tiledisplays
,whichareoneofthemostcommonformsoftactileinterfaces,andtactileinterfacesfortheto
rso.VibrotactileInterfacesThesearetactileinterfacesforconveyingvibratoryinformat
iontoanoperator.Applicationswherevibrotactiledisplayscanbeparticularlyusefulincl
udeinspec-tiontasks,textureperception,scientificdatavisualization,andnavigationa
laids.KontarinisandHowe(1995)werethefirsttopresentdesignguidelinesforimple-menta
tionofvibrationdisplays.Basedonthepropertiesofthehumantactilesystem,theynotedtha
tavibrationdisplaydeviceshouldproducemechanicalvibrationsintherangeof60to1,000Hz
withvariableamplitudeandfrequency.Inordertoachievethisgoal,theyemployedmodified0
.2-Wattloudspeakers.ThecompletesetupisshowninFigure2.11.Therangeofmotionofthedev
iceis3mm,anditcanproduceupto0.25Npeakforceat250Hz.Theusergraspsthemastermanipula
torasshowninthefigure.Anothersimilarrobotmanipula-tor,knownastheslave,thathasacc
elerationsensorsmountedonitsends,sends
FIGURE2.10RutgersMasterII.Thishandmasteremployspneumaticactuatorsinthepalm,there
byeliminatingtheneedforatransmission.Source:FromBouzitetal.(2002);#2002IEEE.
2HapticInterfaces44

backdatarelatedtothevibrationsfeltwhilegraspinganobject.Thesedataarethendisplaye
dtothehumanoperator.Okamuraandcolleagues(2001)developeddecayingsinusoidalwavefor
m-basedmodelsforvibrationfeedbackduringhapticinteraction.Throughexperi-mentsperf
ormedwithrealmaterials,theyrecordedamplitude,frequency,anddecayratesofvibrations
duringimpactevents.Theynotedthatforsomemate-rialstheparameterswerebeyondthebandw
idthoftheirhapticdisplayandhencetheinterfacewasnotcapableofdisplayingthosevibrat
ions.Theseauthorsreportedthatincorporationofvibrationfeedbackalongwithforcefeedb
ackledtoimprovedperformanceduringmaterialdiscriminationtasks.Experimentswerecond
uctedusingthe3GMhapticinterfacebyImmersion.Sim-ilarresultswerepresentedinOkamura
etal.(1998)usingtheIE2000joystick,alsobyImmersion.WearableTactileInterfacesMostt
actileinterfacesaremadeforthefingertip,givenitshighperceptualsensi-tivity.Howeve
r,giventhesmallsizeofthefingertip,tactileinterfacesforotherpartsofthebody,includ
ingthetorso(Ertanetal.,1998;Traylor&Tan,2002)
AccelerationsensorVibrationdisplaySlavemanipulatorMastermanipulator
FIGURE2.11Loudspeaker-basedvibrotactiledisplay.Vibrationsareconveyedtoahumanoper
atorthroughbracketsmountedattheendofthemastermanipulator.Subjectsperformvarioust
asksinateleoperationsetting.Source:FromKontarinisandHowe(1995).
2.3CurrentInterfaceImplementations45

andthemouth(Tang&Beebe,2006),havealsobeenexplored.Thetorsoisprimarilyattractiveb
ecausethesurfaceareaofskinonthebackcanconveytwicetheamountofinformationasthefing
ertips(Jonesetal.,2004).TanandPentland(1997)provideanoverviewoftechnologiesforwe
arabletactiledisplays.Theresearchersalsodevelopedadirectionalhapticdisplay,calle
dtherabbit,composedofa33arrayofninevibrotactileactuatorsforthebackofauser.Thedevic
emakesuseofthesensorysaltationphenomenon(Gerald,1975)toprovidedirectionalinformati
ontothehumanuser.Iftactilecuesaresequentiallyappliedtothreespatiallyseparatedpoi
ntsonthearmofasubject,thentheactualperceptionofthesubjectisofthecuestobeuniforml
ydistributedoverthedistancebetweenthefirstandthelasttactileactuator.Thisspatialr
esolutionofthediscretecuesfeltbythesubjectisalsomorethanthespatialres-olutionoft
heappliedcuesthemselves.Thisphenomenonisknownassensorysaltation,andallowsresearche
rstoachievehighspatialresolutionwiththeuseoffewactuators.Insimilarwork,Jonesandc
olleagues(2004)presentedthedesignofatactilevestusingvibrotactileactuatorsfordire
ctionaldisplayofspatialinformationtotheblind.Thevestprovideddirectionalcuestothe
blinduserthroughthetactileactuatorsmountedontheback.Theyevaluatedandcomparedfour
differentelec-tricalactuatorsforthevibrotactilevest,andchoseapancakemotorafterco
nsider-ingpeakfrequency,powerrequirements,andsizeoftheactuators.Theyfoundthatthe
participantsidentifiedthedirections85percentofthetime,withmosterrorsbeinginthedi
agonaldirection.Theseresultsindicatethatwearabletactileinterfacesarepromisingcan
didatestoserveasnavigationalaidsforthedisabled.Jonesandcolleagues(2004)alsobuilt
andtestedashapememoryalloy(SMA)basedtactorunitfortactilefeedbacktothetorso(Figure
2.12).Theunithadanoverallheightof17mm,lengthof42mm,andwidthof22mm.Inexperi-ments
,thetactorunitproducedapeakforceintherangeof5to9N,withanaver-ageof7Nwithadisplac
ementof3mm.Thebandwidthofthetactorswaslessthan0.3Hz.Inexperimentalstudies,tactor
swerearrangedin14and22arrays,andwereactivatedsequentiallyaswellastogether.Theyno
tedthatalthoughtheusersperceivedthestimulation,itwasnotwelllocalizedandfeltlikef
irmpressuresuchasafingerproddingtheskin.Furthermore,stimulationsonfleshierareaso
fthebackwerefoundtobemoreeasilydetectiblethannearthespinalcord.Theseexperimentss
uggestthatSMAtactorsmaybeusedfortactilefeedbacktothetorso,whichcanleadtolighter,
morecompactvestsduetobetterpower-to-weightcharacteristicsoftheSMAascomparedtoele
ctricalactuators.2.3.4ApplicationsofInterfacetoAccessibilityThemodalitiesofbothh
apticsandvisionarecapableofencodinganddecodingimportantstructuralinformationonth
ewayobjectpartsrelatetoeachotherina3Dworld(Ballestero&Heller,2006).Duetothissimi
larityintheroleofthehap-ticandvisualmodalities,engineersandresearchershavebeenin
terestedinthe
2HapticInterfaces46

useoftouchintheabsenceofvision.TheTactileVisionSubstitutionSystem(TVSS)was,perha
ps,oneofthemostdramaticearlyexamplesofthisinterest(Bach-y-Rita,1972).Theoriginal
TVVSemployedacameraconnectedtoacom-puterandanarrayofvibratingstimulatorsontheski
noftheback.Thebasicideawastoallowpeopletoseewiththeirskin.AderivativeoftheTVVSwast
heOptacon,aportabletactiledisplaytopermitblindpeopletoreadprintedmaterial.Themai
nunitoftheOptaconcontainedatemplateorarraywith144tinypins.Thepinsofthearrayvibrate
dtocreateatactileimageofalphabetsandlettersasacameralenswasmovedoverthem.Recenta
dvancesinhapticinterfaceshaveledtorenewedresearcheffortstobuildhapticinterfacesf
ortheblindorvisuallyimpaired.Therearethreetypesofhapticinterfacesforaccessibilit
y:devicesliketheOptaconthattactuallydisplaymaterialtoberead,hapticnavigationalai
dsfornavigationwithoutsight,andhapticinterfacesforweborcomputeraccess.Christian(
2000)providesabroadoverviewofhapticdisplaydesignforblindusers.Henotesthateventho
ughlittleresearchhasfocusedonthedesignoftactiledisplaysfortheblind,alreadythetac
tiledisplaysoutperformspeechinterfacesbothintermsofspeedand
FIGURE2.12SingleSMAtactorunit.SMA-basedvestshavethepotentialtobelighterandmoreco
mpactthanvibrotactileones.Source:FromJonesetal.(2004);#2004IEEE.

2.3CurrentInterfaceImplementations47

performance.Also,tactiledisplays,ascomparedtoauditoryones,speakauniver-sallangua
ge.Theycanbeunderstoodbyanyblinduser,regardlessoflanguage.HapticBrailleDisplaysA
Brailledisplayisadevice,typicallyattachabletoacomputerkeyboard,thatallowsablindp
ersontoreadthetextualinformationfromacomputermonitoronelineatatime.EachBraillech
aracterconsistsofsixoreightmovablepinsinarectangulararray.Thepinscanriseandfalld
ependingontheelectricalsignalstheyreceive.Thissimulatestheeffectoftheraiseddotso
fBrailleimpressedonpaper.SeveralBrailledisplaysarecommerciallyavailable.Pantobra
illeisasingle-cellbidirectionalBrailledisplaydevelopedattheCentreforInformationT
echnologyInnovation,Canada(Ramstein,1996).ABraillemoduleiscoupledwiththepantogra
ph,aplanarhapticdisplay.Thedeviceprovidestheuserwithacombinationoftactilestimula
tionandstrongfeedback.Ithasaworkspaceof10cm16cmand,unliketraditionalBrailledispl
ays,allowsthereadertomoveoverthematerialinabidirectionalfashion.Inapilotstudywit
htwousers,RamsteinfoundthattheuserspreferredtheinterfaceovertheOptacon,eventhoug
hnosignif-icantimprovementinperformanceoverlargeBrailledisplayswasrealized.Hyper
Brailleisanothertextscreenorientedapplicationthatintegratestoolsforcreating,retri
eving,andsharingprintedandelectronicdocuments(Kieninger,1996).Ascomparedtoothers
imilartoolsthatprovideblindusersaccesstospecificapplications,theHyperBraillesyst
empromotestheuseofstandarddocumentfor-matsandcommunicationprotocols.Forexample,v
ariouspartsofalettersuchasthesender,recipient,andbodyareprelabeled,andHyperBrail
leautomaticallygenerateslinkstotaketheblindreadertothosepartsofthedocuments.Hapt
icAccesstoGraphicalInformationUnliketheBrailledisplaysthatprovidetextualinformat
iontotheblind,somehapticinterfacesprovideblindusersaccesstotheelementsofaregular
GUI.AnexampleofsuchadeviceistheMoose,ahapticmouse(OModhrain&Gillespie,1997).TheMo
ose,showninFigure2.13,iseffectivelyapoweredmousethatdisplayselementsofaGUIusingh
apticcues.Forexample,windowedgesarerepresentedbygrooves,andcheckboxesuseattracti
veandrepulsiveforcefields.Yuandcolleagues(2000)haveinvestigatedtheuseofhapticgra
phsfordatavisualizationinblindusers.Basedonexperimentsonblindandsightedusers,the
yrecommendtheuseofengravingsandtexturestomodelcurvedlinesinhapticgraphs.Furtherm
ore,theyproposetheintegrationofsurfacepropertiesandauditorycuestoaidtheblinduser
.ThepreviouslymentionedinterfacesallowexplorationofaGUIintwodimen-sions.TheHapti
candAudioVirtualEnvironment(HAVE)developedaspartoftheEuropeanUnionGRABprojectsee
kstoprovideblindusersaccessto3Dvirtualenvironments.Thisisachievedbythemeansofadu
al-fingerhapticinterface,asshowninFigure2.14.Thehapticdisplayisfurtheraugmentedw
iththeuseofaudioinputandoutput.Woodandcolleagues(2003)evaluatedtheinterfacethrou
ghasimplecomputergameforblindusersandconcludedthatuserscaneasilyfindandidentifyo
bjectswithin
2HapticInterfaces48

FIGURE2.13Moosehapticinterface.TheMoosereinterpretstheMicrosoftWindowsscreenforb
lindusers.Source:FromOModhrainandGillespie(1997).
FIGURE2.14HapticandAudioVisualEnvironment.HAVEprovidesamultimodaldisplayfor3Dexp
lorationforblindusers.Thehapticinterfaceispictured.(CourtesyofScuolaSuperioreSan
tAnna,Lab.PERCRO.)
2.3CurrentInterfaceImplementations49

thegame,andcancausechangesinthegameenvironmentandperceivethem.Inaddition,alluser
simprovedquicklyandreportedanimmersiveexperience.HapticNavigationalAidsHapticnav
igationalaids,asthenameimplies,attempttoprovidenavigationalinformationtoblinduse
rs.Ascomparedtoauditoryaids,thehapticsignalsprovidedbythehapticaidscannotbeconfu
sedwithenvironmentalsignalsbytheblinduser.Thetwocanalsobeusedtogethertoaugmentea
chother.Ertanandcolleagues(1998)presentedawearablehapticguidancesystemthatusesa4
4gridofmicromotorstotactuallyprovidenavigationalinformationtoausersback.Inadditio
ntothetactiledisplay,theproposedinterfacecomprisesaninfra-red-basedsystemtolocat
etheuserintheenvironmentandacomputerforrouteplanning.Insimilarwork,atactilevest(
seeSection2.3.3)hasbeendevelopedattheMassachusettsInstituteofTechnology(MIT)that
providesnavigationalinfor-mationtotheuser(Jonesetal.,2004).Theauthorsnotethatift
hetotalareaoftheskinisconsidered,thetorsocanconveytwicetheamountofinformationoft
hefingertips.Inmorerecentwork,Tangandcolleagues(2006)presentedanoraltactilenavig
ationalaidfortheblind(Figure2.15).
FIGURE2.15Oralnavigationalaidfortheblind.(a)Aidfitstheupperteethandtonguetouchke
ypadforthebottom(left).(b)Movinglinesandarrowsareusedtoprovidedirectionalcuesass
hownontheright.Source:FromTangandBeebe(2006);#2006IEEE.
2HapticInterfaces50

Thedeviceisamouthpiecewithamicrofabricatedelectrotactiledisplaytoprovidetactilei
nformationtotheroofofthemouth.Atonguetouchkeypadisprovidedforsimultaneousoperati
on.Thedeviceprovidesdirectionalcuestotheoperatorinfourdifferentdirectionsleft,rig
ht,forward,andbackwardusinglinesandarrows.Inpreliminaryexperiments,theresearchers
foundthatsmallelectricalsignalsweresufficientforstimulatingtheroofofthemouth.Ina
prelim-inaryexperimentalstudy,userperformancewasfoundtobegoodfordiscrimina-tiono
fleftandrightsignals,andmixedfortheforwardandbackwardones.2.4HUMANFACTORSDESIGNO
FTHEINTERFACETherearetwobasicfunctionsofhapticinterfaces.First,thedeviceisusedto
measurethemotion(position,velocity,andpossiblyacceleration)andthecontactforcesof
theusersentirebodyorarm,foot,orhand.Second,thedeviceisusedtodisplaycontactforcesa
ndmotionalongwithspatialandtemporaldis-tributionstotheuser(Tanetal.,1994).Whilec
urrenttechnologyinhapticinter-facesislimited,allowingthedisplayofonlyapproximate
interactionswithacomparablerealenvironment,thefeedbackexperiencedviaahapticdevic
ecanfeelveryrealistic,andcanindeedimprovehumanperformanceandsensa-tionsofrealism
wheninteractingwithavirtualenvironment.Theseexperiencesareprimarilyattributedtot
hedevicesabilitytoexploitlimitationsofthehumantactileandkinestheticsensorychannel
s.Tospecifyahapticinterfaceforagivenapplication,itisthereforenecessarytoundersta
ndthebiomechanical,sensori-motor,andcognitiveabilitiesofthehumansystem(Tanetal.,
1994).Sections2.4.1and2.4.2discussthepossibleusesofhapticinterfacesandthefactors
tobeconsideredwhenselectinginterfacesforaparticularapplication,respec-tively.Sec
tion2.4.3isaimedatreadersinterestedindesigningandbuildingtheirowninterfaces.2.4.
1WhentoSelectaHapticInterfaceHapticinterfaceshaveanumberofbeneficialcharacterist
ics,suchasenablingperceptionoflimbmovementandposition,improvingskilledperformanc
eoftasks(typicallyintermsofincreasedprecisionandspeedofexecutionofthetask),anden
ablingvirtualtraininginasafeandrepeatableenvironment.Forcefeed-backhasbeenshown,
specificallyforteleoperatorsystems,toimproveperfor-manceoftheoperatorintermsofre
ducedcompletiontimes,decreasedpeakforcesandtorques,anddecreasedcumulativeforcesa
ndtorques(Hill,1979;Draperetal.,1987;Hannafordetal.,1991;Kim,1991;Massimino&Sher
idan,1994;Murrayetal.,1997;Williamsetal.,2002;OMalley&Ambrose,2003).Fortraining,v
irtualenvironmentscanprovideasettingforsafe,repeatablepractice,andtheinclusionof
hapticfeedbackinsuchenvironmentsimprovesfeelingsof
2.4HumanFactorsDesignoftheInterface51

realisminthetask,increasingthelikelihoodforskilltransferfromthevirtualtothereale
nvironment.Hapticfeedbackisalsoshowntosupporthandeyecoordinationtasks,specif-ical
lyimprovingperformanceindexterousmanipulation(Hale&Stanney,2004).Broadlyspeaking
,hapticfeedbackcaneffectivelyalertpeopletocriticaltasks,pro-videaspatialframeofr
eferencefortheoperator,andimproveperformanceoftasksrequiringhandeyecoordination(H
ale&Stanney,2004).Specifically,tactilecues,suchasvibrationsorvaryingpressuresapp
liedtothehandorbody,areeffectiveassimplealerts,whilekinestheticfeedbackiskeyfort
hemoredexteroustasksthathumanscarryout(Biggs&Srinivasan,2002;Hale&Stanney,2004).
HaleandStanney(2004)provideanexcellentsummaryofthebenefitsofaddingtactileandkine
stheticfeedbackviadiverseinterfacetypes,whicharesum-marizedhere.First,theydiscus
stextureperception.Theadditionoftactilefeed-backviaatactiledisplay,inadditiontov
isualfeedback,resultsinmoreaccuratejudgmentofsoftnessandroughnesscomparedtohuman
performanceofsuchtaskswithvisualfeedbackalone.Ifthetactilefeedbackisaddedtothevi
sualdisplaybymeansofaprobe-baseddeviceratherthanatactiledisplay,researchshowstha
titispossiblefortheoperatortojudgesoftnessandroughnesswiththesameaccuracyaswhenu
singthefingertipdirectly.Finally,ifthetactilefeedbackisdisplayedtotheoperatorvia
anexoskeletondevicewithtactileactua-torsinthefingertips,itispossiblefortheperson
tojudgetexture.Textureper-ceptioncouldbeimportantinapplicationsthatinvolveexplor
ationorobjectmanipulation.Tactilefeedbackcanalsobeusedtoassistintwo-dimensional(
2D)formper-ception.Forsuchtasks,visualfeedbackaloneenablesperceptionoftheformsrel
-ativedepthwithinthefieldofview.Theadditionoftactilefeedbackviaatactiledisplay,e
itherdirectlytothefingertiporthroughaprobedevice,doesnotdomuchtoimprove2Dformper
ception,andcanbeignoredwhenirrelevant.Forexample,vibrotactile(binary)feedbackadd
edtovisualfeedbackduringapick-and-placetaskdoesnotsignificantlyimproveperformanc
ebecausetheinforma-tionisnotrichenoughfortheoperator(Murrayetal.,1997).However,t
hereissomebenefittohavingcross-modalcueingforexample,iftactileactuatorsareusedwit
hinthefingersofanexoskeleton,itispossibletojudge2Dformperception.Theseresearcher
salsosummarizethebenefitsofaddingkinestheticfeedbacktovisualdisplaysforvariousta
sks(Hale&Stanney,2004).Forthepurposeofspa-tialawarenessintermsofposition(ofobjec
tsintheenvironmentorofself),visualdisplaysaloneenablecomprehensionoftherelatived
epthofobjectsandvisualproprioceptionwithinthefieldofview.Theadditionofkinestheti
cfeedbackviaapositionalactuatorfurtherallowsforanegocentricframeofreferencewithi
ntheoperatorspersonalspace,gesturesfornavigationoftheenvironment,andtar-getlocati
on(withlessdecaythanvisualtargetlocation).Whenusingaprobe-baseddeviceoranexoskel
etontoprovidekinestheticfeedback,theoperatorwill
2HapticInterfaces52

experienceenhanceddistancejudgmentswithintheworkspace.Forexample,perceptionofcon
tactcanbegreatlyimprovedwiththeuseofhapticfeedbackovervisual-onlydisplaysinsitua
tionswheretheviewingangledoesnotpermitaclearviewofthecontactingpoints.Whenthetas
kis3Dformperception,visualfeedbackalonewillresultinidentificationanddiscriminati
onperformancethatisdependentonviewingangle,andtheuserwillhavenoindicationofthewe
ightofobjectsintheenvironment.Byincludingkinestheticfeedbackviaaprobe-basedsyste
m,theoperatorwillexperiencedeformabilityofobjectsintheenvironmentthroughtheforce
feed-back,therebyaidingdiscriminationandidentification.Withanexoskeletonsys-tem,
theuserwillexperienceimprovedweightdiscriminationofobjectsalongwithimprovedobjec
tinteraction.Thiskindofinformationcanbecriticalinappli-cationslikesurgicaltraini
ngandtelesurgery,wherehighmanualdexterityisdesired.Forbothprobe-basedandexoskele
tondevices,theinclusionofhapticfeedbacktoavisualvirtualenvironmentresultsinincre
asedsensationsofpresenceorembodimentwithinthevirtualworld.Finally,hapticinterfaces
couldbeincludedtoaugmentvisualinformation.Examplesofsuchapplicationsincludeaugme
ntationofaGUI(seeSection2.3.4),scientificdatavisualization,orCAD.2.4.2DataNeeded
toBuildtheInterfaceUpondeterminingthattheinclusionofhapticfeedbackisbeneficialto
avirtualorremoteenvironmentdisplay,anumberofdecisionsmustbemadeinordertobuildaha
pticinterfacesystem,evenifcommercialhardwareistobeselected.First,thedesignermust

determineiftactileorkinestheticfeedbackispreferred.Thesedecisionsaredependentont
hetypeoffeedbackthatthedesignerwishestoprovide.Forexample,ifthedesireistoprovide
asimplealerttotheuserortodisplaytexturesorsurfaceroughness,thenatactiledeviceism
ostappropriate.Incontrast,if2Dor3Dshapeperception,discrimination,orpresenceinthe
vir-tualorremoteenvironmentisthegoal,thenkinestheticdevicesarepreferred.RefertoS
ection2.4.1foradiscussionofsituationswherehapticfeedbackmightbebeneficial.Ifkine
sthetic,thenthedesignermustselectaprobe-orjoystick-typedevicethatisgraspedbytheu
ser,oranexoskeletondevicethatiswornbytheuser.Whenselectingadesktopdeviceversusaw
earableexoskeletondeviceforkines-theticforcedisplay,thedesignermustdecideontheim
portanceofmobilitywhenusingtheinterfaceandthenatureofthefeedbacktobedisplayed.Of
ten,wechoosetosimulateinteractionswithanenvironmentthroughuseofatoolwieldedinthe
hand,inwhichcasethedesktopdevicesareentirelysuitable.Exoskeletondevices,ontheoth
erhand,enablejoint-basedfeedbacktosimulategraspingofobjectsforhandexoskeletons,o
rmanualobjectmanipulationforthemorecomplexexoskeletons.
2.4HumanFactorsDesignoftheInterface53

Forallapplications,thedesignermustconsidertheperceptualelementsoftheapplicationf
orwhichhapticfeedbackistobedesigned.Forexample,doestheuserneedtodiscriminatebetw
eentwocuesorjustdetectthem?Isthefeed-backwithinhumanperceptualcapabilities?Willt
heuserbefatiguedafterprolongeduse?Cantheuserdiscriminateamongthedifferentvibrato
rycuesprovided?Thesefactorsarehighlytaskdependent,andansweringsuchquestionsforan
arrayofpossibleapplicationsisbeyondthescopeofthischapter.Designersareencouragedt
oconsulttheliteraturefornumerousexamplesofapplicationsofhapticinterfacesandtoref
ertoSection2.1.3forasummaryofhumanperceptualcapabilities.Duetothewiderangeofappl
icationsforwhichhapticfeedbackmaybedesired,variationsintroducedbydifferentroboti
cdevices,partsofthebodywithwhichthedevicemayinterface,thenatureofthefeedback,the
preciseroleofacueinaparticularapplication,andusers,thedesignersmayneedtoconductt
heirownexperimentsanduserstudiestofullyanswersomeofthesequestions.KinestheticFee
dbackDevicesForbothprobe-basedandexoskeletondevices,thedesigndecisionsthatmustbe
madeinordertoimplementthedevicearesimilar.Again,aswiththenatureofthefeedback(tac
tileversuskinesthetic),decisionsareoftentaskdependent.Thedesignermustdeterminean
appropriateDOFnumberforthedevicetoofewandtheflexibilityofthedevicewillbecompromis
ed,whiletoomanywillovercomplicatetheimplementation.Forexample,isplanarmotioninth
evirtualenvironmentsufficient,ordoesitrequiremotioninthree-dimensionalspace?Itis
alsoimportanttoconsiderhowmanyDOFtheuserrequires,andhowmanyDOFofforcefeedbackare
appropriate.Forexample,thepopularPHANToMhapticdevices(SensableTechnologies)allow
forsixDOFofmotioncontrol(threeCartesianmotionsandthreerotations),whileforcefeedb
ackisprovidedonlyforthetranslationalDOF,withlimiteddetrimentaleffectsonsensation
ofpresenceinthevirtualenvironmentbytheuser.Thesizeoftheworkspaceisanothersignifi
cantdesigndecisionforanyhapticdevice,andshouldbeconsideredcarefully.Scalingofope
ratormotionfromthedevicetothevirtualenvironmenteithertoamplifyoperatormotionorsca
ledownmotionsifworkinginamicroscopicenvironment,forexampleisafeasiblesolutioninma
nycases.Itisnotalwaysnecessarythatmotionswiththehapticinterfaceexactlymimicrealworldtasks.However,iftheapplicationofthesystemisfortraining,thencarefulconsidera
tionoftheworkspacesizeandscalingshouldbepracticed.Exoskeletondevicestypicallydon
otinvolvescalingoftheusermotion,exceptinsomeapplicationsforupperextremityrehabil
itation(Breweretal.,2005).However,decisionsaboutDOFandrequiredworkspaceintermsof
jointrangeofmotionshouldbedetermined.Decisionsregardingtherangeofforcestobedispl
ayedbythehapticdeviceareoftendirectlycoupledtoworkspacesizeandtothesizeandcostof
thehapticinterfacehardware.Designersshouldconsiderthetypicalrangeofinteraction
2HapticInterfaces54

forcesthatarerequiredforthedesiredtaskstoberenderedintheenvironment,butconsidera
tionofmaximumhumanforceoutputcapabilities(toensureusersafetyandreducefatigue)alo
ngwithknowledgeofthelimitsofhumanforce-sensingcapabilitiesforsmallforces(asprese
ntedinthesectiononhumanhapticsensing)willinfluencedecisions.Designersshouldbecar
efultoensurethattheselecteddeviceiscapableofprovidingaqualityoffeedbackthatallow
sforsuc-cessfultaskexecution.Forexample,isavibratorycuelikethatofacellphoneringe
rallthatisrequired,orshouldtheuserbeabletodiscriminatebetweentwoseparatecues?Fin
ally,thedesignermustimplementthevirtualenvironmentviacomputercontrolofthehaptici
nterfaceand(typically)acoupledvisualandhapticdisplay.Basichapticrenderingisnotaf
ocusofthischapter;however,thereaderisreferredtoanexcellentintroductiontohapticre
nderingconceptspresentedbySalisburyandcolleagues(2004).Designersarestronglyencou
ragedtoreviewthecommerciallyavailablehapticdevices,astheirimplementationwillbemo
restraightforwardthanbuildinghapticdevicesforcustomapplications.Asnotedpreviousl
y,thereareawidevari-etyofdevicesavailableonthemarketwithvaryingDOFandworkspacedi
men-sions.Theremainderofthissectionfocusesonadditionaldecisionsthatmustbemadewhe
nfabricatingacustomhapticinterface.Whenbuildingacustomhapticinterface,thedesigne
rmustselectthebasiccontrolapproachforthesystemimpedanceoradmittance.Impedancedevi
ceswillrequiresensorstorecordoperatormotions,andwillbecontrolledbyspecify-ingthe
forcesandtorquesthattheenvironmentwillapplytotheuserbasedonhisorherinteractions.
Suchdevicesrequiremechanicaldesignsthatareverylight-weight,stiff,andeasilyback-d
rivable.Admittancedeviceswillrequiresensorstorecordoperatorforces,inadditiontopo
sitionsensorstoallowforclosed-looppositioncontrolofthedevice.Oftenthesesystemsar
enonbackdrivableandexhibitpropertiesoftypicalindustrialrobotsduetothepositioncont
rolapproachofdisplayofthevirtualenvironment.Mostcommercialhapticinterfacesareoft
heimpedancedisplaytype.TactileFeedbackDevicesThespecificationsrequiredfortactile
feedbackdevicesarefewerinnumberthanthoserequiredforkinesthetichapticdevices.Thed
esignermustdeterminethebodylocationforthetactiledisplay.Typically,thisisthefinge
rpadoftheopera-tor,althoughtactiledisplayshavealsobeendevelopedforthetorso(Tan&P
entland,1997;Traylor&Tan,2002).Mosttactilefeedbackdevicesarepinarrays;therefore,
thedesignermustspecifythedensityofthepinarrayandthemethodofactuationforthepins.F
orexample,ifthepinsaretooclose,theusermaynotbeabletodiscriminatesimultaneouscues
fromtwoadjacentpins,whereasiftheyaretoofarapartthecuesmayappeardisjointed.Tactil
earraysforthefingerpadcanbestatic,orcanbecoupledtomouse-likedevicesthatallow
2.4HumanFactorsDesignoftheInterface55

translationinaplane.Thedesignermustdetermineifthiscapabilityisrequiredfortheappl
ication.Finally,lateralskinstretch(inadditionto,orinplaceof,fingerpaddeflectionn
ormaltotheskinsurface)cangenerateverydifferentsensationsforthehumanoperator(Hayw
ard&Cruz-Hernandez,2000).Again,theappropri-atenessofthemethodoffeedbackandthesel
ectionofactuatorsandmechanicaldesignwillbetask-dependent,anddesignersarestrongly
encouragedtoreviewcommerciallyavailablesolutionsalongwithothertactiledisplaydevi
cesthathavebeenpresentedintheliterature.2.4.3DetailedDescriptionofWhatIsNeededto
SpecifySuchInterfacesforUseThefeaturesofkinestheticandtactilehapticinterfacesare
describedseparatelyinthissection.Sensingandactuationarediscussedspecificallyfork
inesthetichapticinterfaces,sinceselectionofcomponentsforthesetasksiscloselycoupl
edtothemechanicaldesignofsuchdevices.MechanicalDesignofKinestheticHapticInterfac
esThemechanicaldesignofahapticinterfaceincludesspecificationofthedeviceDOF,theki
nematicmechanism,andportability.Themostprominentfeatureofahapticinterfaceisthenu
mberandthenatureoftheDOFattheactiveendorends.Theactiveendreferstothepartoftherob
otthatisconnectedtothebodyoftheoperator.Attheactiveend,thehandholdsthedeviceorth
edevicebracesthebody;otherwise,theinteractionisunilateral(Hayward&Astley,1996).T
heDOFthatareactuatedoractiveandothersthatarepassivearealsocritical.Forexample,th
ePHANToMisapen-basedmechanismthathassixDOFattheendpoint,butonlythreeoftheseareac
tuated(Massie&Salisbury,1994).ThroughthePHANToMhapticinterface,anoperatorcanexpl
oreavirtualenvironmentinsixDOF(threeintranslationandthreeinrotation),butreceives
forcefeedbackonlyinthethreetranslationalDOF.ThechoiceofDOFofaparticularhapticint

er-facedependsprimarilyontheintendedapplication.Hapticinterfacesrangefromsimples
ingle-DOFdevicesbuiltforresearch(Lawrence&Chapel,1994)toa13-DOFexoskeletonmaster
armforforce-reflectiveteleoperation(Kimetal.,2005).Thechoiceofmechanismforahapti
cinterfaceisinfluencedbyboththeapplicationandthepartofthebodyinterfacedwith.Robo
ticmechanismscanbeserialorparallel.Aserialmechanismiscomposedofasequenceoflinksc
onnectedendtoend,withoneendoftheresultinglinkageconnectedtotheground(base)andthe
otherbeingfree.Serialmechanismsprovidesimplicityofdesignandcontrol,buttypicallyr
equirelargeractuatorsthanparallelmechan-isms.Inaddition,errorsinthemotionoflinks
nearthebaseoftherobotarepropagatedtotheendeffector,resultinginlossofprecision.Ap
arallelmechanism,ontheotherhand,containsclosedloopsoftheselinkageswithtwoormore
2HapticInterfaces56

connectionstoground.Parallelmechanismsofferhighstructuralstiffness,rigid-ity,pre
cision,andlowapparentinertia,whicharedesirableforthedisplayofhigh-fidelityvirtua
lenvironments,butthesemechanismstendtohavesingulari-ties,limitedworkspaces,andmo
recomplexcontrolschemesthantheirserialcounterparts.Hapticinterfacesapplyforcesto
thehumanoperator.Asaresult,equalandoppositeforcesactontheinterfaceandneedtobedis
tributedinordertomaintainforceequilibrium.Basedonthegroundingofthesefeedbackforc
es,hapticdevicescanbeclassifiedasnonportable(grounded)orportable(ungrounded).Agr
oundedhapticdeviceisaffixedtoarigidbase,transferringreactionforcestoground.Anung
roundedhapticdeviceisattachedonlytotheoperatorsbody,exertingreactionforcesontheus
eratthepoint(s)ofattachment.Mostoftodayshapticinterfaces,likepen-basedhapticdevic
esandjoysticks,aregrounded.Typically,ungroundedhapticinterfacesaregoodatprovidin
gfeedbacksuchasgraspingforcesduringobjectmanipulation,andhaveworkspacesthatpermi
tnatu-ralmovementduringhapticinteractionsbutattheexpenseofdesignsimplicity.Altern
atively,groundeddevicesperformbetterwhendisplayingkinestheticforcestotheuser,lik
eforcesthatarisewhensimulatingstaticsurfaces(Burdea,1996).Theworkspaceofagrounde
ddeviceislimitedbythemanipulatorslinklengthsandjointlimits,suchasincommondesktopi
nterfaceslikethePHANToMDesktopbySensableTechnologies(workspace:6.4inW4.8inH4.8in
D)ortheImpulseEngine2000byImmersionCorporation(workspace:6in6in).Somehapticinter
faces,mostlyexoskeleton-typeinterfaces,canbewearable.Examplesofsuchinterfacesinc
ludetheRutgersMasterIIforcefeedbackglove(Bouzitetal.,2002),theSalfordarmexoskele
ton(Tsagarakis&Caldwell,2003),theL-Exosforcefeedbackexoskeleton(Frisolietal.,200
5),andtheMAHIarmexoskeleton(Gupta&OMalley,2006).SensingandActuationSensingandactu
ationarecriticalcomponentsofahapticinterface.Section2.1presentedthehumansensorya
ndsensorymotorcapabilities.Aneffectivehapticinterfaceneedstomatchtheserequiremen
tsthroughitssensorsandactuators.Forhigh-qualityhapticdisplay,theactuatorsofahapt
icinterfaceshouldhaveahighpower-to-weightratio,highforce/torqueoutput,andhighban
dwidth.Thebandwidthofanactuatorreferstotherangeoffrequencyofforcesthatcanbeappli
edwiththeactuator.Inaddition,theactuatorshouldhavelowfrictionandinertiaastheseca
nmasksmallfeedbackforces,therebydestroyingthesenseofrealism.Sensorsforhapticinte
rfacesshouldhavehighresolution.Duetothedif-ferenceinhumantactileandkinestheticse
nsing,tactileandkinestheticdisplaystypicallyemploydifferentsetsofsensorsandactua
tors.Kinestheticinterfacesmayuseelectricalactuators,hydraulicactuators,orpneumat
icactuators.Electricalactuatorsarecurrentlythemostusedhapticactua-tors.Theseincl
udeDCmotors(bothbrushedandbrushless),magnetic-particle
2.4HumanFactorsDesignoftheInterface57

brakes,andSMA.Specifictrade-offsofthesetypesofactuatorsarediscussedindetailinthe
onlinecasestudiesmentionedinSection2.7.High-resolutionsensorsforkinestheticinter
facesarereadilyavailable,andgenerallynoncontactopticalencodersthatmeasurepositio
nofamotorshaftareused.Theseencoderstypicallyhavearesolutionoflessthanhalfadegree

,whichissufficientformostapplications.Anotheroptionforpositionsensingisnoncontac
trotarypotentiometersthat,liketherotaryencoders,areplacedinlinewiththeactuatorsh
afts.MechanicalDesignofTactileHapticInterfacesTactilefeedbackcanbeprovidedusingp
neumaticstimulationbyemployingcompressedairtopressagainsttheskin(Satoetal.,1991)
,vibrotactilestimulationbyapplyingavibrationstimuluslocallyorspatiallyovertheuse
rsfingertipsusingvoicecoils(Patrick,1990)ormicropinarrays(Hasser&Weisenberger,199
3),orelectrotactilestimulationthroughspeciallydesignedelectrodesplacedontheskint
oexcitethereceptors(Zhu,1988).Inaddition,single-stagePeltierpumpshavebeenadapted
asthermal/tactilefeedbackactuatorsforhapticsimula-tions,suchasinZerkusetal.(1993
).Atactiledisplayusinglateralskinstretchhasalsobeendeveloped(Hayward&Cruz-Hernan
dez,2000).Sensingintactileinterfacescanbeachievedviaforce-sensitiveresistors(Sto
ne,1991),miniaturepressuretransducers(MPT)(Burdeaetal.,1995),theultrasonicforces
ensor(Burdeaetal.,1995),andthepiezoelectricstressratesensor(Sonetal.,1994).Forat
horoughsurveyofcurrenttechnologiesrelatedtotactileinterfacedesign,seePasquero(20
06).2.5TECHNIQUESFORTESTINGTHEINTERFACEAswithallinterfaces,hapticinterfacesneedt
obeevaluatedtoensureoptimalperformance.Sections2.5.1and2.5.2describesomeofthespe
cialconsiderationsthatmustbetakenintoaccountwhentestingandevaluatinghapticinterf
aces.2.5.1TestingConsiderationsforHapticDevicesTheevaluationcanfocusonthehardwar
ealone,orperformancecanbemeasuredbytestinghumaninteractionwiththeenvironment.Har
dwaretestingisrequiredtodeterminethequalityofforcefeedbackachievablebytheinterfa
ce.Thisinvolvesexaminationoftherangeoffrequenciesofforcesthedevicecandisplayandt
heaccuracywithwhichthoseforcescanbedisplayed.Intestingofthemachine,whichisapplic
ationindependent,thecapabilitiesofthedevicesthem-selvesaremeasuredandcomparisona
mongvariousdevicesisallowed.User-basedtestingistaskdependentandcarriedouttostudy
theperceptualeffectivenessoftheinterface,whichisimportantfromahumanfactorspointo
fview.
2HapticInterfaces58

Whentestingthehardware,theproceduresarefairlystraightforwardandshouldsimplyfollo
wbestpracticesforexperimentdesign,datacollection,anddataanalysis,includingstatis
ticalconsiderations.Fortestsinvolvinghumansub-jects,firstitisimportanttofollowgo
vernmentalregulationsforhuman-subjecttesting,oftenoverseenbyanInstitutionalRevie
wBoard.Second,asufficientnumberofsubjectsshouldbeenrolledandtrialsconductedforth
eresultstohavestatisticalsignificance.Third,andspecificallyforhapticdevices,itis
necessarytoisolateonlythosesensoryfeedbackmodalitiesthatareofinterestinagivenstu
dy.Forexample,oftenthehapticinterfacehardwarecanprovideunwantedauditorycuesdueto
theamplifiersandactuatorsthatprovidetheforcesensationsfortheoperator.Insuchstudi
es,itisoftenbeneficialtousenoise-isolatingheadphonesorsomeothermethodofmaskingth
eunwantedauditoryfeedback.Similarly,iffocusingonthehapticfeedbackcapabilitiesofa
particulardevice,itmaybenecessarytoremovevisualcuesfromtheenvironmentdisplay,sin
ceinmanycasesofhumanperceptionthevisualchanneldominatesthehapticchannel.2.5.2Eva
luatingaHapticInterfaceSeveralapproachescanbetakenwhenevaluatingahapticinterface
.First,perfor-manceofthehardwarecanbeassessedusinghuman-subjecttesting,usuallyvi
amethodsthatassesshumanperformanceoftasksinthehapticvirtualenviron-mentorthatmea
suretheindividualsperceptionofthequalitiesofthevirtualenvironment.Tothisend,resea
rchershavestudiedtheeffectsofsoftwareonthehapticperceptionofvirtualenvironments(
Rosenberg&Adelstein,1993;Millman&Colgate,1995;Morgenbesser&Srinivasan,1996).Morg
enbesserandSrinivasan(1996),forexample,lookedattheeffectsofforce-shadingalgorith
msontheperceptionofshapes.Itisalsocommontocompareperformanceoftasksinasimulatede
nvironmentwithaparticulardevicetoperformanceinanequivalentreal-worldenviron-ment
(Buttoloetal.,1995;West&Cutkosky,1997;Richardetal.,1999;Shimojoetal.,1999;Ungere
tal.,2001).WorkbyOMalleyandGoldfarb(2001,2005)andOMalleyandUpperman(2006)extendedt
hesecomparisonstoincludeperfor-manceinhigh-andlow-fidelityvirtualenvironmentsver
susthoseinrealenviron-ments,demonstratingthatalthoughperformanceofsomeperceptual
tasksmaynotbedegradedwithlower-fidelityhapticdevices,humanoperatorscanstillperce
ivedifferencesinqualityoftherenderedvirtualenvironmentsintermsoftheperceivedhard

nessofsurfaces.Suchstudiescangiveanindicationoftheextenttowhichaparticulardevice
anditsaccompanyingrenderedenvironmentmimicreal-worldscenariosandenablehumanstope
rceivethevirtualenvironmentwiththesameaccuracyasispossibleinthenaturalworld.Fina
lly,performancecanbeassessedusingtypicalmeasuresandcharacter-isticsofqualityrobo
tichardware.Primaryrequirementsforahapticsystemaretheabilitytoconveycommandstoth
eremoteorvirtualplantandtoreflect
2.5TechniquesforTestingtheInterface59

relevantsensoryinformation,specificallyforcesintheremoteorvirtualenvi-ronment,ba
cktotheoperator.Inessence,thedynamicsofthedevicemustnotinterferewiththeinteracti
onbetweentheoperatorandtheenvironment.Jex(1988)describesfourteststhatahapticinte
rfaceshouldbeabletopass.First,itshouldbeabletosimulateapieceoflightbalsawood,wit
hnegligibleinertia,friction,orperceivedfrictionbytheoperator.Second,thedevicesho
uldbeabletosimulateacrisphardstop.Itshouldsimulatecoulombfriction,thatis,thedevi
ceshoulddroptozerovelocitywhentheoperatorletsgoofthehandle.Finally,thedeviceshou
ldbeabletosimulatemechanicaldetentswithcrisptransitionandnolag.Inpractice,perfor
manceofahapticinterfaceislimitedbyphysicalfactors,suchasactuatorandsensorquality
,devicestiffness,friction,deviceworkspace,forceisotropyacrosstheworkspace,backla
sh,computationalspeed,andusersactions(handgrip,muscletone).Fromthepreviousdiscuss
ion,itisclearthatthereisawiderangeofhapticdevicesbothintermsoftheirDOFandapplica
tions,makingthetaskofgeneratingacommonperformancefunctionparticularlychallenging
.Variousresearchershaveattemptedtodesignasetofperformancemeasurestocomparehaptic
devicesindependentofdesignandapplication(Ellisetal.,1996;Hayward&Astley,1996),in
cludingkinematicperformancemeasures,dynamicperformancemeasures(Colgate&Brown,199
4)(Lawrenceetal.,2000),andapplication-specificperformancemeasures(Kammermeier&Sc
hmidt,2002;Kirkpatrick&Douglas,2002;Chunetal.,2004).Additionally,whendevelopingc
ustomhardware,sensorresolutionshouldbemaximized,structuralresponseshouldbemeasur
edtoensurethatdisplaydistortionisminimized,andclosed-loopperformanceofthehaptici
nterfacedeviceshouldbestudiedtounderstanddevicestabilitymargins.Detaileddiscussi
onofthetechniquesnecessaryformeasuringthesequantitiesisbeyondthescopeofthischapt
er.2.6DESIGNGUIDELINESThissectionprovidesguidanceonhowtoeffectivelydesignahaptic
interfacethatisbothsafeandeffectiveinitsoperation.2.6.1BaseYourMechanicalDesigno
ntheInherentCapabilitiesoftheHumanOperatorBecausethehapticdevicewillbemechanical
lycoupledtothehumanoperator,itisimportanttoensurethatthecharacteristicsofthesyst
em,suchaswork-spacesize,positionbandwidth,forcemagnitude,forcebandwidth,velocity
,acceleration,effectivemass,accuracy,andotherfactors,arewellmatchedto
2HapticInterfaces60

thehumanoperator(Stocco&Salcudean,1996).Thedesigngoalsforthesystem,ifbasedonthei
nherentcapabilitiesofthehumanhand(orotherbodypartusingthedisplay),willensureasaf
eandwell-designedsystemthatisnotoverqualifiedforthejob.2.6.2ConsiderHumanSensiti
vitytoTactileStimuliSensitivitytotactilestimuliisdependentonanumberoffactorsthat
mustbeconsidered.Forexample,thelocationofapplicationofthestimulioreventhegendero
ftheusercanaffectdetectionthresholds(Sherrick&Cholewiak,1986).Stimulimustbeatlea
st5.5msecapart,andpressuremustbegreaterthan0.06to0.2N/cm2(Hale&Stanney,2004).Add
itionally,vibrationsmustexceed28dBrelativetoa1-microsecondpeakfor0.4-to3-Hzfrequ
enciesforhumanstobeabletoperceivetheirpresence(Biggs&Srinivasan,2002).2.6.3UseAc
tiveRatherThanPassiveMovementToensuremoreaccuratelimbpositioning,useactivemoveme
ntratherthanpas-sivemovementofthehumanoperator.Additionally,avoidminute,precisej
ointrotations,particularlyatthedistalsegments,andminimizefatiguebyavoidingstatic
positionsatorneartheendrangeofmotion(Hale&Stanney,2004).2.6.4AchieveMinimumForce
andStiffnessDisplayforEffectiveInformationTransferfromtheVirtualEnvironmentWheni

mplementingthevirtualenvironmentandselectingactuatorforceoutputandsimulationupda
terates,ensurethattheminimumvirtualsurfacestiffnessis400N/m(OMalley&Goldfarb,2004
)andminimumendpointforcesare3to4N(OMalley&Goldfarb,2002)toeffectivelypromotehapti
cinformationtransfer.2.6.5DoNotVisuallyDisplayPenetrationofVirtualRigidObjectsAv
irtualenvironmentsimulationwithbothvisualandhapticfeedbackshouldnotshowtheoperat
orsfingerpenetratingarigidobject,evenwhenthestiffnessofthevirtualobjectislimiteds
uchthatpenetrationcanindeedoccurbeforesignificantforcesareperceivedbytheoperator
(Tanetal.,1994).Thisisbecausewhenno
2.6DesignGuidelines61

visualfeedbackisavailable,peopletendtofailtodifferentiatethedeformationofthesoft
fingerpadfrommovementsofthefingerjoints.2.6.6MinimizeConfusionandControlInstabil
itiesInmultimodalsystems,itisimportanttominimizeconfusionoftheoperatorandlimitco
ntrolinstabilitiesbyavoidingtimelagsamonghaptic/visualloops(Hale&Stanney,2004).2
.6.7EnsureAccuracyofPositionSensinginDistalJointsSeriallinkagesrequirethatthedis
taljointshavebetteraccuracyinsensingangu-larpositionthanproximaljoints,iftheaccu
racyofalljointsisconstrainedbycostorcomponentavailability(Tanetal.,1994).Thisisb
ecausejointangleresolutionofhumansisbetteratproximaljointsthanatdistalones.2.6.8
ForExoskeletonDevices,MinimizetheContactAreaatAttachmentPointsforMechanicalGroun
dItisimportanttominimizethecontactareaforgroundattachmentpointsbecausehumansarel
esssensitivetopressurechangeswhenthecontactareaisdecreased(Tanetal.,1994).2.6.9E
nsureRealisticDisplayofEnvironmentswithTactileDevicesNotethatahumanoperatormustm
aintainactivepressuretofeelahardsurfaceaftercontact,andmaintainingthesensationof
texturedsurfacesrequiresrelativemotionbetweenthesurfaceandtheskin(Hale&Stanney,2
004).2.6.10KeepTactileFeaturesFixedRelativetotheObjectsCoordinateFrameItisimporta
nttomaintainthisfixedrelativepositionforrealisticperceptionofobjectswithatactile
display.Thisrequirementtranslatestoaneedforhightem-poralbandwidthofthepins(imagi
nefastfingerscanning)(Peineetal.,1997).Matchingmaximumfingerspeedsduringnaturale
xplorationisausefulgoal.
2HapticInterfaces62

2.6.11MaximizeRangeofAchievableImpedancesBecauseitisjustasimportantforthehapticd
evicetobelightandback-drivableasitisforthedevicetobestiffandunyielding,itcanbedi
fficulttooptimizedesignparametersforaspecifichardwaresystem.Therefore,itisrecomm
endedtogenerallyachieveabroadrangeofimpedanceswiththedevice(Colgate&Schenkel,199
7).Sucharangecanbeachievedbycarefullyselectingrobotconfig-uration,defininggeomet
ricparameters,usingtransmissionratios,incorporatingexternaldampers,orenablingact
uatorredundancy(Stoccoetal.,2001).Specifi-cally,techniquesincludeloweringtheeffe
ctivemassofthedevice(Lawrence&Chapel,1994),reducingvariationsinmass(Ma&Angeles,1
993;Haywardetal.,1994;Massie&Salisbury,1994),designingadevicesuchthatitexhibitsa
nisotro-picJacobian(Kurtz&Hayward,1992;Zanganeh&Angeles,1997),oraddingphys-icald
amping(mechanicalorelectrical)tothesystem(Colgate&Schenkel,1997;Mehlingetal.,200
5).2.6.12LimitFrictioninMechanismsToreducenonlinearitiesinthehapticdevice,limiti
ngfrictionisimportant.Ifusingimpedancecontroltechniques,minimalfrictioniskeytoba
ck-drivabilityofthedeviceaswell.Frictioncanbelimitedthroughtheuseofnoncontacting
supportslikeairbearingsormagneticlevitation,byincorporatingdirectdriveactuationt
echniques,or,iftransmissionsareneededtoachievedesiredforcesandtorques,byselectin
gcabledrivesovergearsorothertransmissionmethods.2.6.13AvoidSingularitiesintheWor
kspaceIncertainpartsoftheworkspace,therobotendpointmaylose(inversekinemat-icssin
gularity)orgain(forwardkinematicssingularity)adegreeoffreedom.Forexample,ifarobo
tarmwithrevolutejointsisfullystretched,thentheendpointoftherobotlosesadegreeoffr
eedomasitcannotbemovedalongthelineconnectingthejoints.Similarly,forparallelmecha

nismsincertainconfigurations,itispossiblethattheendpointgainsadegreeoffreedom,th
atis,itcanbeinstan-taneouslymovedwithoutaffectingtheactuatedjoints.Nearworkspace
locationsthatexhibitinversekinematicssingularities,significantlyhightorquesarere
quiredtomovetherobotinthesingulardirection.Nearthesepoints,evendur-ingfreemoveme
nt,theoperatorofahapticinterfacewouldneedtoexertconsid-erableforcestomove,thereb
yreducingtherealismofdisplay.Conversely,ataforwardkinematicssingularity,itisposs
ibletoinitiateendpointmotionwithlittleforce,whichisespeciallydetrimentalforhapti
cinterfacesasitisnotpossibletodisplayanyforcetotheoperatorattheselocations.Hence
,singularitiesintherobotworkspaceshouldbeavoided.
2.6DesignGuidelines63

2.6.14MaximizePinDensityofTactileDisplaysObjectsfeelmorerealisticasthespatialden
sityofthepinsisincreased,althoughthiswillbelimitedbythesizeoftheactuatorsselecte
d.Verticaldisplacementofpinsshouldbe2to3mmwhileproviding1to2Nofforcetoimposeskin
deflectionsduringlargeloads(Peineetal.,1997).2.7CASESTUDIESCasestudiesforseveral
hapticinterfacescanbefoundatwww.beyondthegui.com.2.8FUTURETRENDSThecommercialapp
licationsofhapticandtactiledisplayshavebeensimpleandinexpensivedevices,suchasthe
vibrationsofacellulartelephoneorpager,orforcefeedbackjoystickscommontovideogames
.Hapticinterfaces,bothkines-theticandtactiledisplays,whichhavegreatercapabilitya
ndfidelitythantheseexamples,haveseenlimitedapplicationbeyondtheresearchlab.Thepr
imarybarrierhasbeencost,sincehigh-fidelitydevicestypicallyexhibithighernumbersof
DOF,power-denseactuation,andhigh-resolutionsensing.Wearablehapticdevices,specifi
callywearabletactiledisplays,willalsolikelyseeincreaseddemandfromdefensetoconsum
erapplicationsforthepurposeofsituationalawareness,withdevelopmentsinflexiblemate
rialsthatcanbewovenintofabric.Therefore,apredictionforthenext10to20yearsismuchgr
eateraccessibilitytohapticdevicesincommercialapplicationsasthepriceofimprovedsen
sorandactuatortechnologycomesdown.Suchwidespreadapplicabilityofhapticinter-facet
echnology,especiallyingaming,willbecatalyzedbytherecentincreaseinvideogamesthate
ncourageandevenrequireactivehumanintervention(e.g.,DanceDanceRevolution,Wii).The
seconddriverofhapticdeviceproliferationwillbethesheernumberofapplicationswhereha
pticfeedbackwillproveitselfbeneficial.Improveddatavisualizationbyuseofhaptic(inc
ludingkinestheticandtactile)displaysthatenableincreasedchannelsofinformationconv
eyancetotheuserwillberealized.Hapticdevicesarealreadyunderdevelopmentingeoscienc
eandpharmaceuticalresearchandtestingviahaptic-enrichedprotein-dockingdisplays(Sa
lisbury,1999;Fritz&Barner,1999).Themostlikelydisciplineforwidespreadadoptionofha
ptictechnologieswillbemedicine.Fromrobot-assistedrehabilitationinvirtualenvironm
entswithvisualandhapticfeedback,tosurgicalroboticsthatenablerealistictouchintera
ctionsdisplayedtotheremotesurgeon,tohardwareplat-formsthat,duetotheirreconfigura
bilityandflexibility,willenablenewdiscoveries
2HapticInterfaces64

incognitiveneuroscience,totheincorporationofhapticsensoryfeedbackinpros-theticli
mbsfortheincreasingpopulationofamputees,hapticdeviceswillenableanewdimensionofin
teractionwithourworld.REFERENCESAdelstein,B.,&Rosen,M.(1992).Designandimplementa
tionofaforcereflectingmanip-ulandumformanualcontrolresearch.ProceedingsofASME,112
.An,K.-N.,Askew,L.,&Chao,E.Y.(1986).Biomechanicsandfunctionalassessmentofupperex
tremities.InKarwowski,W.,ed.,TrendsinErgonomics/HumanFactorsIII.Amsterdam:Elsevi
erSciencePublishers,57380.Bach-y-Rita,P.(1972).BrainMechanismsinSensorySubstituti
on.NewYork:AcademicPress.Ballestero,S.,&Heller,M.A.(2006).Conclusions:Touchandbl
indness.InHeller,M.A.,&Ballestero,S.,eds.,TouchandBlindness.Mahwah,NJ:Erlbaum,19
7219.Barbagli,F.,Salisbury,K.,etal.(2006).Hapticdiscriminationofforcedirectionand
theinfluenceofvisualinformation.ACMTransactionsonAppliedPerception3(2):12535.Bejc

zy,A.,&Salisbury,K.(1980).Kinematiccouplingbetweenoperatorandremotemanipulator.A
dvancesinComputerTechnology1:197211.Bergamasco,M.,Allotta,B.,etal.(1994).Anarmexo
skeletonsystemforteleoperationandvirtualenvironmentsapplications.ProceedingsofIE
EEInternationalConferenceonRoboticsandAutomation,144954.Bergamasco,M.,&C.A.Avizza
no,C.A.(1997).Virtualenvironmenttechnologiesinreha-bilitation.ProceedingsofIEEERSJ-SICESymposiumonRobotandHumanInteractiveCommunication.Biggs,J.,&Srinivasan,M.
(2002).Hapticinterfaces.InStanney,K.,ed.,HandbookofVirtualEnvironments.London:Er
lbaum,93116.Bolanowski,S.J.,Gescheider,G.A.,etal.(1988).Fourchannelsmediatethemec
hanicalaspectsoftouch.JournaloftheAcousticalSocietyofAmerica84(5):168094.Bouzit,M
.,Burdea,G.,etal.(2002).TheRutgersMasterIInewdesignforce-feedbackglove.IEEE/ASMET
ransactionsonMechatronics7(2):25663.Brewer,B.R.,Fagan,M.,etal.(2005).Perceptualli
mitsforaroboticrehabilitationenviron-mentusingvisualfeedbackdistortion.IEEETrans
actionsonNeuralSystemsandRehabil-itationEngineering13(1):111.Brooks,F.P.,Ouh-Youn
g,M.,etal.(1990).ProjectGROPE:Hapticdisplaysforscientificvisualization.Proceedin
gsoftheAssociationforComputingMachinerySpecialInterestGrouponGraphicsandInteract
iveTechniques(SIGGRAPH)Conference,17795.Brooks,T.L.(1990).Teleroboticresponserequ
irements.ProceedingsofIEEEInternationalConferenceonSystems,ManandCybernetics,1132
0.Burdea,G.,Goratowski,R.,etal.(1995).Atactilesensinggloveforcomputerizedhanddia
g-nosis.TheJournalofMedicineandVirtualReality1(1):4044.Burdea,G.C.(1996).Forceand
TouchFeedbackforVirtualReality.NewYork:JohnWiley&Sons.Buttolo,P.,Kung,D.,etal.(1
995).Manipulationinreal,virtualandremoteenvironments.ProceedingsofIEEEInternatio
nalConferenceonSystems,ManandCybernetics,465661.
References65

Cholewiak,R.,&Collins,A.(1991).Sensoryandphysiologicalbasesoftouch.InHeller,M.,&
Schiff,W.,eds.,ThePsychologyofTouch.Hillsdale,NJ:Erlbaum,2360.Christian,K.(2000).
Designofhapticandtactileinterfacesforblindusers.CollegePark,MD:DepartmentofCompu
terScience,UniversityofMaryland.Chun,K.,Verplank,B.,etal.(2004).Evaluatinghaptic
sand3DstereodisplaysusingFittslaw.ProceedingsofThirdIEEEInternationalWorkshoponHa
ptic,AudioandVisualEnvironmentsandTheirApplications,Ottawa,5358.Clark,F.J.(1992).
Howaccuratelycanweperceivethepositionofourlimbs?BehavioralBrainScience15:72526.Cl
ark,F.J.,&Horch,K.(1986).Kinesthesia.InBoff,K.,Kauffman,L.,&Thomas,J.,eds.,Handb
ookofPerceptionandHumanPerformance.NewYork:JohnWiley,162.Colgate,J.E.,&Brown,J.M.
(1994).FactorsaffectingtheZ-widthofahapticdisplay.ProceedingsofIEEEInternational
ConferenceonRoboticsandAutomation,320510.Colgate,J.E.,&Schenkel,G.G.(1997).Passiv
ityofaclassofsampled-datasystems:Applicationtohapticinterfaces.JournalofRoboticS
ystems14(1):3747.Delp,S.L.,Loan,J.P.,etal.(1997).Surgicalsimulation:Anemergingtec
hnologyfortrain-inginemergencymedicine.Presence-TeleoperatorsandVirtualEnvironme
nts6:14759.DiFranco,D.E.,Beauregard,G.L.,etal.(1997).Theeffectofauditorycuesonthe
hapticperceptionofstiffnessinvirtualenvironments.ProceedingsofASMEDynamicSystems
andControlDivision,1722.Draper,J.,Herndon,J.,etal.(1987).Theimplicationsofforcere
flectionforteleoperationinspace.ProceedingsofGoddardConferenceonSpaceApplication
sofArtificialIntelli-genceandRobotics.Durfee,W.K.,Hendrix,C.M.,etal.(1997).Influ
enceofhapticandvisualdisplaysontheestimationofvirtualenvironmentstiffness.Procee
dingsofASMEDynamicSystemsandControlDivision,13944.Durlach,N.I.,Delhorne,L.A.,etal
.(1989).Manualdiscriminationandidentificationoflengthbythefinger-spanmethod.Perc
eption&Psychophysics46(1):2938.Durlach,N.I.,&Mavor,A.S.,eds.(1995).VirtualReality
:ScientificandTechnologicalChallenges.Washington,DC:NationalAcademyPress.Ellis,R
.E.,Ismaeil,O.M.,etal.(1996).Designandevaluationofahigh-performancehapticinterfa
ce.Robotica14:32127.Ertan,S.,Lee,C.,etal.(1998).Awearablehapticnavigationguidance
system.ProceedingsofSecondInternationalSymposiumonWearableComputers,16465.Falvo,M
.R.,Finch,M.,Superfine,R.,etal.(1996).TheNanomanipulator:Ateleoperatorformanipul
atingmaterialsatnanometerscale.ProceedingsofInternationalSymposiumontheSciencean
dEngineeringofAtomicallyEngineeredMaterials,57986.Frisoli,A.,Rocchi,F.,etal.(2005
).Anewforce-feedbackarmexoskeletonforhapticinterac-tioninvirtualenvironments.Pro
ceedingsofFirstJointEurohapticsConferenceandSym-posiumonHapticInterfacesforVirtu

alEnvironmentandTeleoperatorSystems,195201.Fritz,J.P.,&Barner,K.E.(1999).Designof
ahapticvisualizationsystemforpeoplewithvisualimpairments.ProceedingsofIEEETransa
ctionsonNeuralSystemsandRehabilitationEngineering7(3):37284.Gerald,F.A.(1975).Sen
sorySaltation:MetastabilityinthePerceptualWorld.Hillsdale,NJ:Erlbaum,133.
2HapticInterfaces66

Gillespie,B.,OModhrain,S.,etal.(1998).Thevirtualteacher.ProceedingsofASMEDynamicS
ystemsandControlDivision,17178.Gupta,A.,&OMalley,M.K.(2006).Designofahapticarmexos
keletonfortrainingandrehabilitation.IEEE/ASMETransactionsonMechatronics11(3):2808
9.Hajian,A.,&Howe,R.(1997).Identificationofthemechanicalimpedanceofthehumanfinge
rtips.ASMEJournalofBiomechanicalEngineering119(1):10914.Hale,K.S.,&Stanney,K.M.(2
004).Derivinghapticdesignguidelinesfromhumanphysiolog-ical,psychophysical,andneu
rologicalfoundations.IEEEComputerGraphicsandApplica-tions24(2):3339.Hannaford,B.,
Wood,L.,etal.(1991).Performanceevaluationofasix-axisgeneralizedforce-reflectingt
eleoperator.IEEETransactionsonSystems,ManandCybernetics21(3):62033.Hasser,C.,&Wei
senberger,J.(1993).Preliminaryevaluationofashape-memoryalloytactilefeedbackdispl
ay.ProceedingsofSymposiumonHapticInterfacesforVirtualEnvironmentsandTeleoperator
Systems,AmericanSocietyofMechanicalEngineers,WinterAnnualMeeting,7380.Hayward,V.,
&Astley,O.R.(1996).Performancemeasuresforhapticinterfaces.InGiralt,G.,&Hirzinger
,G.,eds.,RoboticsResearch:TheSeventhInternationalSymposium.NewYork:Springer,19520
7.Hayward,V.,Astley,O.R.,etal.(2004).Hapticinterfacesanddevices.SensorReview24(1
):1629.Hayward,V.,Choksi,J.,etal.(1994).Designandmulti-objectiveoptimizationofali
nkageforahapticinterface.InLenarcic,J.,&Ravani,B.,eds.,AdvancesinRobotKinematics
.NewYork:KluwerAcademic,35968.Hayward,V.,&Cruz-Hernandez,M.(2000).Tactiledisplayd
eviceusingdistributedlateralskinstretch.ProceedingsofSymposiumonHapticInterfaces
forVirtualEnvironmentsandTeleoperatorSystems,Orlando,130916.Hill,J.W.(1979).Study
ofmodelingandevaluationofremotemanipulationtaskswithforcefeedback.Pasadena,CA:SR
IInternationalforJetPropulsionLaboratory.Hogan,N.,Kay,B.A.,etal.(1990).Hapticill
usions:Experimentsonhumanmanipulationandperceptionofvirtualobjects.Proceedingsof
ColdSpringHarborSymposiaonQuantitativeBiology,92531.Iwata,H.(1993).Pen-basedhapti
cvirtualenvironment.ProceedingsofIEEEVirtualRealityAnnualInternationalSymposium,
28792.Jandura,L.,&Srinivasan,M.(1994).Experimentsonhumanperformanceintorquediscri
minationandcontrol.ProceedingsofASMEDynamicSystemsandControlDivision,NewYork,3697
5.Jeong,Y.,Lee,Y.,etal.(2001).A7DOFwearableroboticarmusingpneumaticactuators.Pro
ceedingsofInternationalSymposiumonRobotics.Jex,H.(1988).Fourcriticaltestsforcont
rol-feelsimulators.Proceedingsof23rdAnnualConferenceonManualControl.Johnson,K.O.
,&Phillips,J.R.(1981).TactilespatialresolutionI.twopointdiscrimination,gapdetecti
on,gratingresolutionandletterrecognition.JournalofNeurophysiology46(6):117192.Jon
es,L.A.,&Hunter,I.W.(1990).Aperceptualanalysisofstiffness.ExperimentalBrainResea
rch79(1):15056.
References67

Jones,L.A.,Hunter,I.W.,&Irwin,R.J.(1992).Differentialthresholdsforlimbmovementme
asuredusingadaptivetechniques.Perception&Psychophysics52:52935.Jones,L.A.&Hunter,
I.W.(1993).Aperceptualanalysisofviscosity.ExperimentalBrainResearch94(2):34351.Jo
nes,L.A.,etal.(2004).Developmentofatactilevest.ProceedingsofTwelfthInterna-tiona
lSymposiumonHapticInterfacesforVirtualEnvironmentandTeleoperatorSystems,8289.Jone
s,L.A.(1989).Matchingforces:Constanterrorsanddifferentialthresholds.Perception18
(5):68187.Kammermeier,P.,&Schmidt,G.(2002).Application-specificevaluationoftactil
earraydisplaysforthehumanfingertip.ProceedingsofIEEE/RSJInternationalConferenceo
nIntelligentRobotsandSystems,Lausanne,Switzerland,3:293742.Kieninger,T.G.(1996).T
hegrowingupofHyperBrailleanofficeworkspaceforblindpeople.ProceedingsofAssociation

forComputingMachinerySymposiumonUserInter-faceSoftwareandTechnology,6773.Kim,W.S.
(1991).Anewschemeofforcereflectingcontrol.ProceedingsofFifthAnnualWorkshopofSpac
eOperationsApplicationsandResearch,NASA,LyndonB.JohnsonSpaceCenter,25461.Kim,Y.S.
,Lee,J.,etal.(2005).Aforcereflectedexoskeleton-typemasterarmforhuman-robotintera
ction.IEEETransactionsonSystems,ManandCybernetics,PartA35(2):198212.Kirkpatrick,A
.E.,&Douglas,S.A.(2002).Application-basedevaluationofhapticinterfaces.Proceeding
sofTenthSymposiumonHapticInterfacesforVirtualEnvironmentandTeleoperatorSystems,I
EEEComputingSociety,Orlando,3239.Kontarinis,D.A.,&Howe,R.D.(1995).Tactiledisplayo
fvibratoryinformationintele-operationandvirtualenvironments.Presence-Teleoperato
rsandVirtualEnvironments4(4):387402.Krebs,H.I.,Hogan,N.,etal.(1998).Robot-aidedne
urorehabilitation.IEEETransactionsonRehabilitationEngineering6(1):7587.Kurtz,R.,&
Hayward,V.(1992).Multiple-goalkinematicoptimizationofaparallelsphericalmechanism
withactuatorredundancy.IEEETransactionsonRoboticsandAutomation8(5):64451.Lawrence
,D.A.,&Chapel,J.D.(1994).Performancetrade-offsforhandcontrollerdesign.Proceeding
sofIEEEInternationalConferenceonRoboticsandAutomation,4:321116.Lawrence,D.A.,Pao,
L.Y.,etal.(2000).Rate-hardness:Anewperformancemetricforhap-ticinterfaces.IEEETra
nsactionsonRoboticsandAutomation16(4):35771.Lee,S.,Park,S.,etal.(1998).Designofaf
orcereflectingmasterarmandmasterhandusingpneumaticactuators.ProceedingsofIEEEInt
ernationalConferenceonRoboticsandAutomation,3:257479.Loomis,J.M.(1979).Aninvestig
ationoftactilehyperacuity.SensoryProcesses3(4):289302.Ma,O.,&Angeles,J.(1993).Opt
imumdesignofmanipulatorsunderdynamicisotropyconditions.ProceedingsofIEEEInternat
ionalConferenceonRoboticsandAutoma-tion,1:47075.Massie,T.,&Salisbury,J.K.(1994).T
hePHANToMhapticinterface:Adeviceforprobingvirtualobjects.ProceedingsofASME,Winte
rAnnualMeeting,NewYork,55(1):295300.
2HapticInterfaces68

Massimino,M.J.,&Sheridan,T.B.(1994).Teleoperatorperformancewithvaryingforceandvi
sualfeedback.HumanFactors36(1):14557.McKnight,S.,Melder,N.,etal.(2004).Psychophys
icalsizediscriminationusingmultifin-geredhapticinterfaces.ProceedingsofEuroHapti
csConference.McNeely,W.,Puterbaugh,K.,etal.(1999).Sixdegree-of-freedomhapticrend
eringusingvoxelsampling.ProceedingsofAssociationforComputingMachinerySpecialInte
restGrouponGraphicsandInteractiveTechniquesConference.Meech,J.F.,&Solomonides,A.
E.(1996).Userrequirementswheninteractingwithvirtualobjects.ProceedingsofIEEEColl
oquiumonVirtualRealityUserIssues(DigestNo.1996/068),3/13/3.Mehling,J.S.,Colgate,J.
E.,etal.(2005).Increasingtheimpedancerangeofahapticdis-playbyaddingelectricaldam
ping.ProceedingsofFirstJointEurohapticsConferenceandSymposiumonHapticInterfacesf
orVirtualEnvironmentandTeleoperatorSystems,25762.Millman,P.,Stanley,M.,etal.(1993
).Designofahighperformancehapticinterfacetovir-tualenvironments.ProceedingsofIEE
EVirtualRealityAnnualInternationalSymposium.Millman,P.A.,&Colgate,J.E.(1995).Eff
ectsofnon-uniformenvironmentdampingonhapticperceptionandperformanceofaimedmoveme
nts.ProceedingsofASMEDynamicSystemsandControlDivision,57(2):70311.Morgenbesser,H.
B.,&Srinivasan,M.A.(1996).Forceshadingforhapticshapeperception.ProceedingsofASME
DynamicSystemsandControlDivision,40712.Moy,G.,Singh,U.,etal.(2000).Humanpsychophy
sicsforteletactionsystemdesign.Haptics-e,TheElectronicJournalofHapticsResearch,1
(3).Murray,A.M.,Shimoga,K.B.,etal.(1997).Touchfeedbackusingbinarytactordisplays:
Unexpectedresultsandinsights.ProceedingsofASMEDynamicSystemsandControlDivision,39
.Nahvi,A.,Nelson,D.,etal.(1998).Hapticmanipulationofvirtualmechanismsfrommechani
-calCADdesigns.ProceedingsofIEEEInternationalConferenceonRoboticsandAutomation,3
7580.Nakai,A.,Ohashi,T.,etal.(1998).7DOFarmtypehapticinterfaceforteleoperationandv
irtualrealitysystems.ProceedingsofIEEE/RSJInternationalConferenceonIntelligentRo
botsandSystems.Okamura,A.M.,Cutkosky,M.R.,etal.(2001).Reality-basedmodelsforvibr
ationfeedbackinvirtualenvironments.IEEE/ASMETransactionsonMechatronics6(3):24552.
Okamura,A.M.,Dennerlein,J.T.,etal.(1998).Vibrationfeedbackmodelsforvirtualenvironments.ProceedingsofIEEEInternationalConferenceonRoboticsandAutomation,67479.OMal
ley,M.K.,&Ambrose,R.O.(2003).HapticfeedbackapplicationsforRobonaut.IndustrialRob
ot:AnInternationalJournal30(6):53142.OMalley,M.K.,&Goldfarb,M.(2001).Forcesaturati

on,systembandwidth,informationtransfer,andsurfacequalityinhapticinterfaces.Proce
edingsofIEEEInternationalConferenceonRoboticsandAutomation,138287.OMalley,M.K.,&Go
ldfarb,M.(2002).Theeffectofforcesaturationonthehapticperceptionofdetail.IEEE/ASM
ETransactionsonMechatronics7(3):28088.OMalley,M.K.,&Goldfarb,M.(2004).Theeffectofv
irtualsurfacestiffnessonthehapticperceptionofdetail.IEEE/ASMETransactionsonMecha
tronics9(2):44854.
References69

OMalley,M.K.,&Goldfarb,M.(2005).Ontheabilityofhumanstohapticallyidentifyanddiscri
minaterealandsimulatedobjects.Presence:TeleoperatorsandVirtualEnvironments14(3):
36676.OMalley,M.K.,&Gupta,A.(2003).Passiveandactiveassistanceforhumanperformanceof
asimulatedunderactuateddynamictask.ProceedingsofEleventhSymposiumonHapticInterfa
cesforVirtualEnvironmentandTeleoperatorSystems,34855.OMalley,M.K.,Gupta,A.,etal.(2
006).Sharedcontrolinhapticsystemsforperformanceenhancementandtraining.ASMEJourna
lofDynamicSystems,Measurement,andControl128(1):7585.OMalley,M.K.,&Upperman,G.(2006
).Astudyofperceptualperformanceinhapticvirtualenvironments.JournalofRoboticsandM
echatronics,SpecialIssueonHaptics:Interfaces,Applications,andPerception,18(4):46
775.OModhrain,M.S.(2000).PlayingbyFeel:IncorporatingHapticFeedbackintoComputer-Bas
edMusicalInstruments.PhDdiss.,StanfordUniversity.OModhrain,M.S.,&Gillespie,B.(199
7).TheMoose:Ahapticuserinterfaceforblindpersons.ProceedingsofSixthWorldWideWebCo
nference,SantaClara,CA.Pang,X.D.,Tan,H.Z.,etal.(1991).Manualdiscriminationofforc
eusingactivefingermotion.Perception&Psychophysics49:53140.Pasquero,J.(2006).Surve
yonCommunicationthroughTouch.Montreal:CenterforIntelligentMachines,McGillUnivers
ity.Patrick,N.J.M.(1990).Design,ConstructionandTestingofaFingertipTactileDisplay
forInteractionwithVirtualandRemoteEnvironments.Mastersthesis,DepartmentofMechanic
alEngineering,MassachusettsInstituteofTechnology.Peine,W.J.,Wellman,P.S.,etal.(1
997).Temporalbandwidthrequirementsfortactileshapedisplays.ProceedingsofASMEDynam
icSystemsandControlDivision,61:10714.Rabinowitz,W.M.,Houtsma,A.J.M.,etal.(1987).M
ultidimensionaltactiledisplays:Iden-tificationofvibratoryintensity,frequency,and
contactorarea.JournaloftheAcousticalSocietyofAmerica82(4):124352.Ramstein,C.(1996
).CombininghapticandBrailletechnologies:Designissuesandpilotstudy.ProceedingsofA
ssociationforComputingMachineryConferenceonAssistiveTechnologies,3744.Richard,P.,
Coiffet,Ph.,etal.(1999).Humanperformanceevaluationoftwohandlehapticdevicesinadex
trousvirtualtelemanipulationtask.ProceedingsofIEEE/RSJInterna-tionalConferenceon
IntelligentRobotsandSystems,154348.Rosenberg,L.B.(1993).Virtualfixtures:Perceptua
ltoolsforteleroboticmanipulation.ProceedingsofIEEEVirtualRealityAnnualInternatio
nalSymposium,7682.Rosenberg,L.B.,&Adelstein,B.D.(1993).Perceptualdecompositionofv
irtualhapticsurfaces.ProceedingsofIEEESymposiumonResearchFrontiersinVirtualReali
ty,4653.Salcudean,S.,&Vlaar,T.(1994).Ontheemulationofstiffwallsandstaticfrictionw
ithamagneticallylevitatedinput/outputdevice.ProceedingsofASMEDynamicSystemsandCo
ntrolDivision,303309.Salisbury,J.K.(1999).Makinggraphicsphysicallytangible.Commun
icationsoftheACM42(8):7481.Salisbury,K.,Conti,F.,etal.(2004).Hapticrendering:Intr
oductoryconcepts.IEEECom-puterGraphicsandApplications24(2):2432.
2HapticInterfaces70

Sato,K.,Igarashi,E.etal.(1991).Developmentofnon-constrainedmasterarmwithtac-tile
feedbackdevice.ProceedingsofIEEEInternationalConferenceonAdvancedRobotics1:33438.
Schmidt,R.(1977).Somatovisceralsensibility.InSchmidt,R.,ed.,FundamentalsofSensor
yPhysiology.NewYork:Springer-Verlag,81125.Sherrick,C.E.,&Cholewiak,R.W.(1986).Cut
aneoussensitivity.InBoff,K.,Kaufman,L.,&Thomas,J.L.,eds.,HandbookofPerceptionand
HumanPerformance.NewYork:Wiley,158.Shimoga,K.(1992).Fingerforceandtouchfeedbackis
suesindexteroustelemanipulation.ProceedingsofNASA-CIRSSEInternationalConferenceo

nIntelligentRoboticSystemsforSpaceExploration,15978.Shimoga,K.(1993).Asurveyofper
ceptualfeedbackissuesindexteroustelemanipulation.PartII:Fingertouchfeedback.Proc
eedingsofIEEEVirtualRealityAnnualInternationalSymposium,NewYork.Shimojo,M.,Shino
hara,M.,etal.(1999).Humanshaperecognitionperformancefor3Dtac-tiledisplay.IEEETra
nsactionsonSystems,ManandCybernetics,PartA29(6):63744.Sledd,A.,&OMalley,M.K.(2006)
.Performanceenhancementofahapticarmexoskeleton.ProceedingsofFourteenthSymposiumo
nHapticInterfacesforVirtualEnvironmentandTeleoperatorSystems,37581.Smyth,T.N.,&Ki
rkpatrick,A.E.(2006).AnewapproachtohapticaugmentationoftheGUI.ProceedingsofInter
nationalConferenceonMultimodalInterfaces,37279.Son,J.,Monteverde,E.,etal.(1994).A
tactilesensorforlocalizingtransienteventsinmanipu-lation.ProceedingsofIEEEIntern
ationalConferenceonRoboticsandAutomation,47176.Speich,J.E.,Shao,L.,etal.(2005).Mo
delingthehumanhandasitinteractswithatelemanipulationsystem.Mechatronics15(9):112
742.Srinivasan,M.,&Chen,J.(1993).Humanperformanceincontrollingnormalforcesofcon-t
actwithrigidobjects.ProceedingsofAmericanSocietyofMechanicalEngineersConferenceo
nAdvancesinRobotics,MechatronicsandHapticInterfaces,NewYork,11925.Srinivasan,M.A.
,Beauregard,G.L.,etal.(1996).Theimpactofvisualinformationonthehapticperceptionof
stiffnessinvirtualenvironments.ProceedingsofASMEDynamicSystemsandControlDivision
,55559.Srinivasan,M.A.,&LaMotte,R.H.(1995).Tactualdiscriminationofsoftness.Journa
lofNeurophysiology73(11):88101.Stocco,L.,&Salcudean,S.E.(1996).Acoarse-fineapproa
chtoforce-reflectinghandcontrol-lerdesign.ProceedingsofIEEEInternationalConferen
ceonRoboticsandAutomation,40410.Stocco,L.J.,Salcudean,S.E.,etal.(2001).Optimalkin
ematicdesignofahapticpen.IEEE/ASMETransactionsonMechatronics6(3):21020.Stone,R.(1
991).Advancedhuman-systeminterfacesforteleroboticsusingvirtual-reality&teleprese
ncetechnologies.ProceedingsofFifthInternationalConferenceonAdvancedRobotics,16873
.Sutter,P.,Iatridis,J.,etal.(1989).Responsetoreflected-forcefeedbacktofingersint
eleopera-tion.ProceedingsofNASAConferenceonSpaceTelerobotics,15766.Tan,H.Z.,Durla
ch,N.I.,etal.(1995).Manualdiscriminationofcomplianceusingactivepinchgrasp:Therol
esofforceandworkcues.Perception&Psychophysics57(4):495510.
References71

Tan,H.Z.,Durlach,N.I.,etal.(1994).Tactualperformancewithmotionalstimulationofthe
indexfinger.JournaloftheAcousticalSocietyofAmerica95(5):298687.Tan,H.Z.,Durlach,N
.I.,etal.(1993).Manualresolutionofcompliancewhenworkandforcecuesareminimized.Pro
ceedingsofASMEDynamicSystemsandControlDivision,99104.Tan,H.Z.,Eberman,B.,etal.(19
94).Humanfactorsforthedesignofforce-reflectinghap-ticinterfaces.ProceedingsofInt
ernationalMechanicalEngineersCongressandExposi-tion,Chicago,35359.Tan,H.Z.,Pang,X
.D.,etal.(1992).Manualresolutionoflength,force,andcompliance.ProceedingsofAmeric
anSocietyofMechanicalEngineers,WinterAnnualMeeting,Ana-heim,CA,1318.Tan,H.Z.,&Pen
tland,A.(1997).Tactualdisplaysforwearablecomputing.DigestoftheFirstInternational
SymposiumonWearableComputers,8489.Tang,H.,&Beebe,D.J.(2006).Anoraltactileinterfac
eforblindnavigation.IEEETransactionsonNeuralSystemsandRehabilitationEngineering1
4(1):11623.Traylor,R.,&Tan,H.Z.(2002).Developmentofawearablehapticdisplayforsitua
tionawarenessinaltered-gravityenvironment:Someinitialfindings.ProceedingsofTenth
SymposiumonHapticInterfacesforVirtualEnvironmentandTeleoperatorSystems,15964.Tsag
arakis,N.,Caldwell,D.G.,etal.(1999).A7DOFpneumaticmuscleactuator(pMA)poweredexos
keleton.ProceedingsofEighthIEEEInternationalWorkshoponRobotandHumanInteraction,3
2733.Tsagarakis,N.G.,&Caldwell,D.G.(2003).Developmentandcontrolofasoft-actuatedexos
keletonforuseinphysiotherapyandtraining.AutonomousRobots15(1):2133.Unger,B.J.,Nic
olaidis,A.,etal.(2001).Comparisonof3-Dhapticpeg-in-holetasksinrealandvirtualenvi
ronments.ProceedingsofIEEE/RSJInternationalConferenceonIntelli-gentRobotsandSyst
ems,175156.Upperman,G.,Suzuki,A.,etal.(2004).Comparisonofhumanhapticsizediscrimin
ationperformanceinsimulatedenvironmentswithvaryinglevelsofforceandstiffness.ProceedingsofInternationalSymposiumonHapticInterfacesforVirtualEnvironmentandTeleop
eratorSystems,16975.Verillo,R.T.(1966).Vibrotactilesensitivityandthefrequencyresp
onseofthePaciniancorpuscle.PsychonomicScience4:13536.Weisenberger,J.M.,Krier,M.J.
,etal.(2000).Judgingtheorientationofsinusoidalandsquarewavevirtualgratingspresen

tedvia2-DOFand3-DOFhapticinterfaces.Haptics-e1(4).West,A.M.,&Cutkosky,M.R.(1997)
.Detectionofrealandvirtualfinesurfacefeatureswithahapticinterfaceandstylus.Proce
edingsofASMEInternationalMechanicalEngi-neersCongressDynamicSystemsandControlDiv
ision,Dallas,15965.Wiker,S.,Hershkowitz,E.,etal.(1989).Teleoperatorcomfortandpsyc
hometricstability:Criteriaforlimitingmaster-controllerforcesofoperationandfeedba
ckduringtelemanipu-lation.ProceedingsofNASAConferenceonSpaceTelerobotics,Greenbe
lt,MD,99107.Williams,L.E.P.,Loftin,R.B.,etal.(2002).Kinestheticandvisualforcedisp
layfortelero-botics.ProceedingsofIEEEInternationalConferenceonRoboticsandAutomat
ion,124954.WilliamsII,R.L.,Murphy,M.A.,etal.(1998).Kinestheticforce/momentfeedbac
kviaactiveexoskeleton.ProceedingsofImageSocietyConference.
2HapticInterfaces72

Wood,J.,Magennis,M.,etal.(2003).Thedesignandevaluationofacomputergamefortheblind
intheGRABhapticaudiovirtualenvironment.ProceedingsofEurohapticsConference.Yu,W.,
Ramloll,R.,etal.(2000).Hapticgraphsforblindcomputerusers.LectureNotesinComputerS
cience2058:4151.Zanganeh,K.E.,&Angeles,J.(1997).Kinematicisotropyandtheoptimumdes
ignofparallelmanipulators.InternationalJournalofRoboticsResearch16(2):18597.Zerku
s,M.,Becker,B.,etal.(1993).Temperaturesensinginvirtualrealityandtelerobotics.Vir
tualRealitySystems1(2):8890.Zhu,H.(1988).Electrotactilestimulation.InWebster,J.,e
d.,TactileSensorsforRoboticsandMedicine.NewYork:JohnWiley&Sons,34153.
References73

Thispageintentionallyleftblank

3CHAPTERGestureInterfacesMichaelNielsen,ThomasB.Moeslund,MoritzStorring,ErikGranu
mThischapterprovidesanintroductiontothedomainofgestureinterfaces.Webeginwiththef
oundationofgestureinterfacesandhowandwheretheyarecom-monlyused,followedbybasicth
eorythatwillhelpyouselectanddesignagestureinterface,andrelatedadviceandwarnings.
Next,wepresentaproceduretoiden-tifyandtestgoodgesturesforadesignandapracticalexa
mpleofhowtousethisprocedure.Finally,weprovideashortperspectiveonourvisionofthefu
tureastheinterfaceevolves.3.1GESTURESGesturesoriginatefromnaturalinteractionbetw
eenpeople.Theyconsistofmovementsofthebodyandfaceasnonverbalcommunicationthatcomp
lementsverbalcommunication.Thisistheinspirationbehindusinggestureinterfacesbetwe
enmanandmachine.Figure3.1showsaninteractionbetweentwopeoplethatreliesonnonverbal
communication.Thecameraismountedontheearofoneperson.Asseenhere,interpretationisc
ontextdependent.Theycannotheareachotherbecauseofloudmusic,andthepersonwiththecam
erawantstoorderadrink.ThegesturebythepersonattheleftmeansIcannothearyou.Theper-son
(attheright)withthecameraonhisheadisusinganiconicgesturetoshowwhathewants.Theoth
erpersonimitateshisgesture.Agestureinterfacecanbeseenasanalternativeorcomplement
toexistinginterfacetechniques,suchastheolddesktopparadigm.Goodexamplesarethenewa
lternativestothemouse,suchasergonomictrackballs,mousetabletpens(e.g.,fromPalmand
aTabletPC),andtheiGesturePad(Figure3.2).

GestureinterfacescannavigateaWindowsinterfacejustaswellorbetterthanthemousecurso
r,whiletheymaybemoreorlessuselesswhenitcomestofastcomputergames,suchasthree-dime
nsional(3D)shootersandairplanesimula-tors.Fortheseapplications,specializedinterf

aceshavebeendevelopedthatarebasedonwheelsandflightsticksfromtherealworld.Whendev
elopingagestureinterface,theobjectiveshouldnotbetomakeagestureinterface.Agesturein
terfaceisnotuniversallythebestinterfaceforanyparticularapplication.Theobjectivei
stodevelopamoreefficientinterfacetoagivenapplication.Thiscanbeillustratedbyanexamp
leofaninterfaceforartisticmodelingofasculpture.Theartistmaybegivenamouseandakeyb
oardforaCADprogram.Theresultisperfecttothesmallestdetailregardingaccuracyoftheli
nes,because
Mouse4Mouse3Mouse5WheelupWheeldowniGesture Pad
FIGURE3.2TheiGesturePadreactstotherelativepositionsandmovementsoftheusersfingerti
psastheytouchthepad.(CourtesyofFingerworks.)
FIGURE3.1Interactionbasedonnonverbalcommunication.Gesturesbetweentwopeoplewhocan
notheareachother.
3GestureInterfaces76

itispossibletosetcoordinatesexplicitly.Iftheartistisprovidedwithagestureinterfac
einwhichavirtualclaymodelcanbealteredbytouchingandsqueezingit,itwillnotbeaccurat
eintermsofcoordinatesandstraightlines,butitmightbeaestheticallyclosertotheartists
vision.Thus,selectinganinterfaceisamatterofwhichoutcomeoftheapplicationisdesired
.Consequently,thefirststepisanalysisofthekindofinterfacethatismostsuitableforthi
stask.Suchanalysismayleadtotheconclusionthatagestureinter-faceisthemostsuitablet
ypeofinterface.3.2TECHNOLOGYANDAPPLICABILITYMuchworkhasbeendoneintheinvestigatio
nanddevelopmentofnaturalinterac-tioninterfaces,includinggestureinterfaces(Cassel
l,1998;Freemann&Weissman,1995;Hummels&Stapiers,1998;Storringetal.,2001;Paggio&Mu
sic,2000;Steiningeretal.,2002;Streitzetal.,2001).Gestureinterfaceshavealsobeende
vel-opedinsciencefictionliteratureandmovies,suchasthemoviesJohnnyMnemonic(1995),
FinalFantasy(2001),andMinorityReport(2002).Furthermore,gestureinterfacesareappli
edtosolveproblemsforpeoplewithphysicaldisabilities(Keates&Robinson,1998;Jaimes&S
ebe,2005).Themostinterestingpotentialinthisfieldofresearchistomakeaccessory-free
andwirelessgestureinterfaces,suchasinvirtual-realityandintelligentrooms,becauset
heuseofphysicalandwiredgadgetsmakestheinterfaceandges-turingtediousandlessnatura
l.Thefirstsolutionsrequiredexpensivedataglovesorothersuchintrusiveequipmentwithw
iresthatmadetheuserfeeluncomfort-able.Greatersuccesscamewithpen-basedgestures(e.
g.,Palmhandhelddevices),wheretrajectorieswererecognizedasgestures.Acommonmotivat
ionbehindtheanalogybetweennonverbalcommunica-tionandhumancomputercommunicationist
hatitallowsforbetter,morenatu-ralandintuitiveinteraction.However,Cassell(1998)cl
aimsthatthiscannotbeliberallyassumedtoalwaysbethecase.Forexample,acommandlanguag
eisasnaturalaswritingato-dolist.Anotherideainvolvedaccessibilitystudies,wherepeo
plewithphysicaldisabilitiesarenotcapableofusingtactileinputmodalities.Whilenatur
algesturesareoftensubtle,gestureinterfacesrelyonemphasizedgestures.Interfaceshav
emostlyrevolvedaroundsimplehandgesturesforpoint-ingoutobjectsorcontrollingamouse
-likepointerandafewgesturesthatarelinkedtospecificfunctionsintheapplication.Simp
leusageofthesemotion-trackinginterfacesisfoundingameconsoles(e.g.,PlaystationEye
Toy).Recentworkhasalsofocusedonfacialgestures(faceexpressionsandposes)detectingre
actionsandemotions.Thisinformationcanbeusedforautomaticannotationinhumanbehavior
studies(EmotionToolfromiMotions,http://www.imotions.dk),accessibilityforparalyze
dpeople(Betkeetal.,2002),andfeedbacktoanintelligentlearningsystem(Pantic&Patras,
2006).
3.2TechnologyandApplicability77

3.2.1MechanicalandTactileInterfacesEarlygestureinterfacesreliedonmechanicalormag
neticinputdevices.Examplesincludethedataglove(Figure3.3),thebodysuit(SuguruGoto,
2006),andNintendoWii.Single-pointtouchinterfacesarewellknownaspengestures(Longet
al.,2000),mostcommonlyseeninPalmhandhelddevices.Butrecentresearchhasdevelopedmul
tipointtouchesdirectlyontothescreen,usedintheiGesturePad(JeffersonHan,2005;Tseet
al.,2006),whichopenupanewandmoreefficientinterfacepotential.Thereareexamplesofre
searchinmakingthecomputerawareofhumanemo-tionsshowninbodylanguage.DeSilvaetal.(2
006)detectedemotionintensityfromgesturesusingsensorsthatreadgalvanicskinresponse
.However,peopleingeneraldonotliketheideaofhavingsuchsensorsorattachmentsontheirb
odies.3.2.2ComputerVisionInterfacesWhentheaimistomakegestureinterfacesinvisiblet
otheuser(Jaimes&Sebe,2005),computervisionisanicewaytodetectgestures.Computervisi
onisinher-entlywireless,andpeoplehavebecomeaccustomedtosurveillancecamerasinreta
ilandairportenvironments.Computervisionalgorithmsoftenconsistofthreeparts:segmen
tationthatspotsrelevantpartsinthefieldofview,trackingthatfollowsthemovements,and
classificationthatfindsmeaningfulinformation.
FIGURE3.3Dataglovefrom5DT.(Courtesyofwww.5DT.com.)
3GestureInterfaces78

Segmentationdetectsthebodypartsintheimages,suchasrelyingonshapeorskincolor,ormot
iondetectioninvideosequencesfordynamicgestures(Stoer-ringetal.,2004).Inordertomo
vepastthisstep,manysolutionshavesimplifiedthesegmentationbyusinginfraredreflecto
rsorothersuchmarkingsthatareeasytosegmentandtrack.Trackingfollowsasetoffeaturesw
henitmoveswith,forexample,condensa-tionorKalmanfiltering(Rasmussenetal.,2006).Cl
assificationdetectstheactualgesturesusinghiddenMarkovmodels(Sageetal.,2003;DeSil
vaetal.,2006;Minetal.,1999),templateormodelmatching(Moe-slundetal.,2003;Riviera&
Guitton,2004;Fihletal.,2006),orfuzzyorBayesianlogic(Aveles-Arrigaetal.,2006;Wach
setal.,2006;Moustakasetal.,2006).Anewapproach(Wangetal.,2006)tracksthemovementof
acameraphonebyanalyzingitsimagesofsurroundings.Thesephonemovementsareusedasgestu
res.3.2.3FaceGazeandExpressionFacegazeandexpressionareasubdomainofgestureresearc
h(Jaimes&Sebe,2005).Facegazetrackingistraditionallyusedforviewpointestimationofv
irtual-realityandstereodisplays,butrecentresearchaimstoextractallvisualinformati
onfromtheface.Facialexpressionsmaybeusedasinputmodalitiesinaccessibilityresearch
(Betkeetal.,2002),suchasfordisabledpeoplewhocannotmakeanymovementotherthanfacial
,suchasblinkingorsmiling.Furthermore,facialmovementscon-veyinformationabouttheem
otionsoftheuser(Pantic&Rothkrantz,2003).Thesecanbeusedtodetectconfusion,userreac
tions,andintentions,andforpur-posesofsurveillanceandautomaticvideofeedannotation
suchasinhumanbehaviorresearch.Acomputerwithoutempathyismuchlikeanapatheticperson
,andinteractingwithitcancausefrustration(Kleinetal.,2002).Computervisionisalsous
edfordetectionoffacialexpressions,whichallowsforanimpressiveperceptiveinterfacea
ndefficientvideoannotationtools,andcangiveemotionalawarenesstovirtualagents(Betk
eetal.,2002;Bowyeretal.,2006;Pantic&Patras,2006).However,thetechnologyfordoingth
isisstillunderdevelopmentandresultsareusuallygivenastechnicaldetectionrates.Test
inguserexperiencewithsuchsystemsisgoingtobeveryinteresting.3.2.4ApplicabilityGes
tureinterfacesarepopularwherevertheinterfacerequiressomefreedomofmovementorimmer
sivefeelingsuchasinvirtual-realityenvironments(Stoerringetal.,2001),intelligentr
ooms(Streitzetal.,2001),medicaltoolsduringsurgery(Wachsetal.,2006),andmedicalsim
ulators(Tsagarakisetal.,2006).Othertypicalapplicationsareadvancedelectronicwhite
boardsandhandlingof3Dobjects(Muller-Tomfelde&Steiner,2001;Moustakasetal.,2006).
3.2TechnologyandApplicability79

Anexampleofagestureinterfacesystem(Moustakasetal.,2006)isshowninFigure3.4.Theuse

rcanretrieve3Dobjectsfromadatabaseandmanipulatethem.Commandssuchasretrieve,rotat
e,andsoon,areaccessedthroughspeech.Theusermanipulatesthe3Dspacewithgesturessucha
sshowinghowtorotate,scale,move,anddrawprimitives.Avirtualagentactsasahelpwizardi
ftheuserforgetsthecommandsandgestures.Gestureinterfacesystemsarealsoveryapplicab
leinaccessibilityresearchbecausetheyarenaturallyhands-offandhands-freeinteractio
nsthatcanbedonewithanyfunctionalpartofthebody.Interfacesforaccessibilityfocusonc
reatingalternativeinputmodestoWIMP(windows,icons,mouse,pointers)applications,suc
hascontrollingthemousepointerwiththeface(Figure3.5)oranysinglelimbthatthedisable
dpersoncanuse(Betkeetal.,2002).3.3FUNDAMENTALNATUREOFTHEINTERFACEThesimplestgest
ureinterfaceisthewell-knownmotiondetectorthatturnsonalightinaroomortriggersthewa
terinawaterfountainatapubliclavatory.Wewillfocusonthemoreexplicitgestureinterfac
esinwhichspecificgesturessuchaspos-turesormovementsaretobeinterpretedbyacomputer
tomotivatesomethinginanapplication.Thissectiondescribesgeneraltheoryrelatedtosuc
hgestureinterfaces.
InteractionProcessingMicrophoneVirtualguideStereo cameraLarge displaySpeechrecei
verTCPTCPTalking-head virtualguideGesturespeechrecognitionVirtual assembly,search
platform,sketch recognition3D modeldatabaseTargetvirtualenvironmentSpeechtransmitter
FIGURE3.4Exampleofagestureinterfacesystem.Objectsearchusingspeechandgesturesanda
virtualagentinMasterpiece.Source:AdaptedfromMoustakasetal.(2006);#2006IEEE.
3GestureInterfaces80

3.3.1HowGesturesRelatetoOtherModesItisnotalwaysfeasibletodesignaninterfacethatre
liescompletelyongesturesbecausetheremaybeissuesrelatingtouserorcomputersystempre
cisionandaccu-racyorthecomplexityoftheinputrequired.Hence,itisoftenbeneficialtom
akegesturespartofamultimodalsystem.Toknowwhentouseonemodeortheother,weneedtobeaw
areofperceptiveabilitiesinrelationtoothersenses.Table3.1showsthesensesthatcontai
ninformationtobeinterpretedinhumanperception.Audible,visual,andtactilesensesaret
hemostimportantinhumancomputerinteraction,andthesearethecommonlyusedsensesthusfar
tomakeuseofininterfacedesign.Scentandtastearelessrelevantinacomputerinteractionc
ontext,buttheymaybecomemoreimportantinthefutureastechnologyfordetectionandsynthe
sisemerges.Thesensescanobtainconflicting,complementary,irrelevant,inadequate,orr
edundantinformation.Thesedatamustbefilteredinthecognitivesystemandprocessedinord
ertobeunderstood(Figure3.6).Thisisdoneusingexpectationsbasedonaknownorassumedcon
textregardingtheworld,history,andpersonalattributes.Sometimestheseexpectationsar
ewrong,whichleadstoprejudice,amongotherthings.Figure3.6alsolistswhatthoseexpecta
tionscouldconsistofifasimilarinterpretationsystemweredevelopedonacomputer.Ifthec
omputerwouldbeabletointerpretthehumanbehaviorandprovidenaturalsensoryfeedback,it
wouldenhancetheimmersivefeelingoftheuser.
FIGURE3.5Controllingamousepointerwiththeface.Thecameramouseiscontrolledbya30-mon
th-olduser.Source:FromBetkeetal.(2002);#2002IEEE.
3.3FundamentalNatureoftheInterface81

Animmersiveinterfacewouldbelikedrivingacar,wherethedriverdoesnotfocusontheintera
ctionofthecarbutbecomesasonewithit.Itischallengingtousedevice-freeinteraction,as
thetactilesensoryfeedbackisdifficulttoprovide.Itisasifthesomestheticsenseislacki
ng,suchasinadisabledpatient,asfollows:
ExpectationsContextWorld GeometryApplicationactivitySensesExpectationsMemoryGram

marSemanticsHistoryPersonalAttributionUserconfiguration
Interpretation
Sensoryfeedback
FIGURE3.6Understandingsensorydata.Interpretationofintegratedsensesreliesonexpect
ations,whichalsodirectthefocusofattentionofthesenses.Possibleexpectationsofinter
pretationsystemonacomputer.
TABLE3.1TheFiveSensesandRelevantInformationaboutSurroundingsSenseRelevantInforma
tionAudioSpeechIdentityIntonationPrecision(timing)VisionIdentityFacialexpression
BodylanguageGestureAccuracy(spatial)Tactile/somestheticTabs,pads,devicesTextureP
recision(timing)Accuracy(spatial)ScentAtmosphere,likabilityTasteClarification,en
joyment
3GestureInterfaces82

FLossofthecapabilitytosenselimbmovementandposition.FMajorimpairmentinskilledperf
ormance,evenwithfullvisionandhearing.Thisisworsenedasvisualinformationdegrades.F
Abnormalmovementsandtheinabilitytowalkfollowingthelossofsomesthesis.FPatientsmus
texertimmenseefforttorelearnhowtowalk.FMajorlossofprecisionandspeedofmovement,pa
rticularlyinthehands.FMajordifficultyperformingtasksthatcombinesignificantcognit
iveloadsandfinemotorskillssuchaswritingminutesduringmeetings.FMajordifficultylea
rningnewmotortasks,relearninglostones,orusingpreviousexperiencetoguidetheseproce
sses(Tsagarakisetal.2006).Figure3.7showsresultsfromTsagarakisetal.(2006),whodesi
gnedavirtualsur-gicalknifethroughaphysicalhandhelddeviceposingastheknife.Theyfou
nd
0.450.400.350.300.250.200.150.100.050Distance error (mm)(b)User12345Distance err
or (mm)0.0250.0200.0150.0100.0050User(a)12345
FIGURE3.7Resultsoftestingavirtualsurgicalknife.Theaccuracyofthesurgicalknifeisme
asuredinmillimetersforfivetestusers.(a)Testperformedusinghapticandvisualfeedback
.(b)Testusingonlyvisualfeedback.Source:FromTsagarakisetal.(2006);#2006IEEE.
3.3FundamentalNatureoftheInterface83

thatthegestureaccuracyimproved15-foldbyaddingtactilefeedbackwhenthevirtualknifeh
itthevirtualflesh.Applyingsuchfeedbacktoaccessory-freegestureinterfacessuchasavi
rtualrealityCAVE(anenvironmentinwhichtheuserissurroundedbyback-projectedscreenso
nallfoursides,ceiling,andfloor)withcamera-basedtrackingofbodypartswillbechalleng
ing.Tactile(orhaptic)feedbackisoftengiventhroughphys-icalcontactwithaccessorydev
icesviavibrations(Caldwelletal.,1999;Langdonetal.,2000).However,device-freetacti
lefeedbackcouldalsobepossiblethroughsubsonicsoundwaves(Muller-Tomfelde&Steiner,20
01).Usingthisapproachcouldenableusageinapplicationswherephysicalandwireddevicesw
ouldbeunnatural.Acommonsolutionistousevisualandaudiofeedback,butthesearenottac-t
ile.Anothersolutioncouldbetoresearchtheusageofdirectedlow-frequencyaudiowaves,as
thiswouldfeelliketactilefeedback.SpatialversusTemporalPerceptiveRelationandPreci
sionApopularsolutioninmultimodalinterfacestudiesisthecomplementaryusageofspeecha
ndgesture.Thesemodalitiescomplementeachotherwellbecausevisionrelatesmainlytospat
ialperception,whilesoundrelatesmainlytotempo-ralperception.Forexample,anexperime
ntwasconductedwheretestsubjectssawadotblinkingonceonthemonitorwhilehearingtwocli
ckswithinacertaintimeframe(Vroomen&Gelder,2004).Theresultwasthatthetestsubjectsp

erceivedtwoblinksandeventhreeblinkswhenthesoundclickedthrice.Thisdemonstratedacl
earcomplementarymergingofsenseswithadominantaudiocuefortemporalcognition.Buthuma
nsaremuchbetteratestablishingdistanceanddirectionfromvisualcuesthanfromauditoryc
ues(Loomisetal.,1998).Whenyoudesignavisual-andanaudio-baseddetectionandsynthesis
systemsynchronizationproblemscanarisebecausetheirresponsetimesmaybedifferent.Aph
ysicallyaccuratedetectionandsynthesismodeltendstoreduceresponsetimeperformance.L
ongresponsetimecancauseambiguityanderrorbetweenthemodes,whilegoodsynchronization
solvesambiguityandminimizeserrors.Consideranelaboratedexamplesuchasavirtual-real
ityapplicationwhereyoucanpickupitemsforvarioustasks.Avoicerecognitionsystemreact
stothephrasePickuptotriggerthisaction,andtheapplicationusesyourhandpositiontoident
ifythevirtualitem.However,youmoveyourhandslightlyfast,becauseyouaregoingtodoalot
ofwork.Thevisualrecognitionsystemlagsbehind,whichcausesa0.5-seconddiscrepancybet
weenthehandpositionandwheretheapplicationthinksyouare.Furthermore,thespeechrecog
nitionmighthavea1.5-secondlag.IfyousayPickupwhenyourrealhandisontopofthevirtualobj
ect,yourhandhasmovedonfromtheitemthatyouwantedandyoumaypickupanotheritemornoneat
all.Ifitwereamenuitem,youmighthavechosenDeleteeverythinginsteadofSave.
3GestureInterfaces84

3.3.2GestureTaxonomiesAnumberoftaxonomiesongesturesarefoundintheliterature.Thebe
stoneforuseinagivenapplicationdependsontheconcretecontext.Basically,therearethre
ewaysoflabelinggestures:Semanticlabelsdescribethemeaningofthegesture,thatis,what
itcommunicatesanditspurpose.Commonlyusedinnonverbalcommunicationstudies.Function
allabelsdescribewhatthegesturedoesinaninterface.Commonlyusedintechnicalhumancompu
ter(HCI)descriptions.Descriptivelabelsdescribehowthegestureisperformed,suchasits
movement.CommonlyusedintechnicalHCIdescriptions.SemanticLabelsSemanticlabels,asd
escribedbyCassell(1998),canbeconsciousorspontaneous.Furthermore,theycanbeinterac
tionalorpropositional.Consciousgestureshavemeaningwithoutspeech,whilespontaneous
gesturesonlyhavemeaninginthecontextofspeech.Examplesofconsciousgesturesfollow:Em
blemsareconsciouscommunicativesymbolsthatrepresentwords.Theseareinteractionalges
tures.Anexampleisaringformedbythethumbandindexfin-ger.InWesternculturethismeansOk
ay,andinJapanitmeansMoney.Propositionalgesturesconsciouslyindicateplacesinthespacea
roundtheper-formerandcanbeusedtoillustratesizeormovement.ExamplesincludeItwasthis
big(usingapropositionalgesturetoshowthesize)orPutthatthere.Spontaneousgesturesarele
sscontrollableandemergespontaneouslyasthesenderisengagedinaninteraction:FIconicg
esturesareillustrationsoffeaturesineventsandactions,orhowtheyarecarriedout.Examp
lesaredepictinghowahandlewastriggeredorlookingaroundacorner.FMetaphoricgesturesa
relikeiconicgestures,butrepresentabstractdepictionsofnonphysicalform.Anexampleis
circlingthehandtorepresentthatThemeetingwentonandon.FDeicticgesturesrefertothespac
ebetweenthenarratorandthelistener(s).Forinstance,thesecanpointtoobjectsorpeopleb
eingdiscussedorrefertomovementordirections.Theycanalsobeusedinamoreabstractway,s
uchasinwavingawaymethods,Wedontusethose,andpickingdesiredmethodsasinThesearewhatweus
e.Thesegesturesaremainlyspontaneous,butcanalsooccurconsciously,suchaswhenpointing
atanobject.FBeatgesturesareusedtoemphasizewords.Theyarehighlydynamic,astheydonot
depictthespokenmessageswithpostures.Anexampleiswhenthespeakermakesamistakeandthe
ncorrectshim-orherselfwhilepunchingwiththehand.
3.3FundamentalNatureoftheInterface85

Cassell(1998)statesthatemblemsandmetaphoricgesturesareculturallydependent,whileo
thertypesaremostlyuniversal,althoughsomeculturesusethemmorethanothers.Asiancultu
resuseverylittlegesturingwhileSouthernEuropeanculturesuseagreatdeal.Thegesturest
hataregenerallyrelevantformachineinteractionsaredeictic,iconic,propositional,and
emblems.Thisisbecausetheapplicationshavehadcommonthemessuchasconveyingcommandsan

dmanipulatingandpointingoutentities.Thisobservationleadstofunctionallabels.Funct
ionalLabelsFunctionallabelsexplainintendedusageinanapplication.Commandgesturesac
cesssystemfunctionsinanapplication,suchasQuit,Undo,andConfigure.Typicallyemblems
canbeusedforthesebecausetheirappear-ancesignalssomethingspecific.Pointinggesture
sarecommonlydeicticorpropositionalgesturesthatpointoutentitiesinthespacearoundth
euser.Thisisacoregesturetypethatisusedforselection.Forexample,ifyouwanttobuysome
thinginamarketandyoudonotspeakthelocallanguage,youmightpointtoit.Manipulationges
turesrelatetofunctionsthatmanipulateoreditthedataspace,suchasscalingorrotatingan
imageor3Dmodel.Propositionalgesturesarecloselyrelatedtothistypeofinteractionbeca
usetheyresembletheusageofthosegesturesinnonverbalcommunication.Controlgesturesmi
micthecontroloverentitiesintheapplicationsuchasavatarorcameramovements.Iconic,pr
opositional,anddeicticgesturescanbeusedbecauseallareusedwhenonepersonimitatesano
ther.DescriptiveLabelsDescriptivelabelsrefertothemannerinwhichthegesturesareperf
ormedinaspatiotemporalsense.Staticgesturesarepostures,thatis,relativehandandfing
erpositions,thatdonottakemovementsintoaccount.Anyemblemcanbeseenasanexampleofast
aticgesture,asinThumbsup.Dynamicgesturesaremovements,thatis,handtrajectoryand/orpo
stureswit-chingovertime.Innonverbalcommunication,dynamicsinagesturecanalteritsme
aning(Hummels&Stapiers,1998)asitiscommonlyusedinsignlan-guage.Adynamicgesturecan
bedefinedbyatrajectorythatresemblesafigure(Figure3.8)orsimplyasequenceofstaticge
stures.Spatiotemporalgesturesarethesubgroupofdynamicgesturesthatmovethroughthewo
rkspaceovertime.
3GestureInterfaces86

3.4HUMANFACTORSINVOLVEDININTERFACEDESIGNOneofthemostdifficulttasksistofindafeasi
blegesturevocabularythatiseasyfortheusertorememberandperform(Keates&Robinson,199
8).Signlanguageisnotconvenientbecausethegesturesarerathercomplicated,andsignlang
uagesdifferaccordingtotheunderlyingvocallanguage.Limitingthevocabularyisimportan
t,andwillbenefitbothusersanddesigners.Methodsthatcanbeusedforthispurposefollow:C
ontextdependence:Availableoptionsvarywiththecontextofthecurrentselection.Thisiss
imilartothecontextmenuinWindowsapplications,wherearightclickspawnsamenuofitemsth
atarerelevantonlytotheselectedobject.Itisimportanttomakethestateofthesystemvisib
letotheuseratalltimesinsuchinterfaces.Spatialzones:Thismethodisknownasthesurroun
duserinterface(Figure3.9).Thespacearoundtheuserisdividedintospatialzones,eachofw
hichhasitsowncontextthatdefinesthefunctionsofthegestures.Forexample,commandgestu
resareclosetotheuserandmanipulationgesturesarefartheraway.Specialutilityfunction
scouldbestowedtobothsidesoftheuser,muchlikestowingthingsawayinpockets.3.4.1Techn
ology-versusHuman-BasedGesturesAtypicalapproachtodefininganapplicationsgesturevoc
abularyistomakeiteasyforthecomputersystemsrecognitionalgorithmtorecognizethegestu
res.Theresultofthisapproachforfindinggesturescanbecalledatechnology-based
StartStart cutPauseEnd cutCutStopPrintClearQuit
FIGURE3.8Dynamicgestures.Examplesofcommandgesturesthataredynamicandspatiotempora
l,astheytrackthepositionofahandorhandhelddevice.
3.4HumanFactorsInvolvedinInterfaceDesign87

gesturevocabulary.Anexampleistodefineagesturebyhowmanyfingersarestretched(Figure
3.10)becauseanalgorithmhasbeendevelopedthatcancountextendedfingers.Thesegestures
togethercreateavocabularythatmightbeusedwherethereisnoparticularmeaningtothegest
ureitself;inbrief,theassociationbetweengestureandmeaningisarbitrary.Ontheotherha
nd,itiseasytoimple-mentacomputervisionbasedrecognizerforsuchgestures.Table3.2show
stwoexampleapplicationsinwhichfunctionalitieshavebeenassignedtogestures.Aneutral

handpositionthatisnottobeinterpretediscalledresidue.Applicationscontainingtechnolo
gy-basedgestureshavebeenimplementedandtested.Figure3.11showstheinterfaceoftheapp
lication.Thetestsshowedthatthegestureswereinappropriateforoneormoreofthefollowin
greasons:FStressful/fatigueproducingfortheuserFNearlyimpossibleforsomepeopletope
rformFIllogicallyimposedfunctionalityAsanalternative,thehuman-basedgestureapproa
chinvestigatesthepeoplewhoaregoingtousetheinterface,andmakesuseofhumanfactorsder
ivedfromHCIresearch,user-centereddesign,ergonomics,andbiomechanics.3.4.2HCIHeuri
sticsandMetaphorsNumerousguidelinesforuser-friendlydesignsareavailable.Usability
canbedescribedusingthefollowingfiveparameters(Federico,1999;Bevan&Curson,
ManipulationCommandUtilityUtility
FIGURE3.9Exampleofasurrounduserinterface.Left:Thecommandgestureshellisclosetothe
user.Gesturesareinterpretedascommandgestureswhenthehandsareinsidethebentcube.Rig
ht:Gesturesperformedfartherawayareinterpretedasdatamanipulationgestures.Utilityf
unctionsareavailableatthesidesoftheuser.
3GestureInterfaces88

T0
T1
T2
T3
T4
T5
FIGURE3.10Technology-basedgestures.Thesegesturesareeasyforacomputertorecognizeus
ingcomputervision,buttheyaredifficulttorelatetofunctionsandsomeofthemaredifficul
ttoperform.
TABLE3.2ImposedFunctionalitiesinTwoDemoApplicationsApplicationGesture*PaintingOb
jectHandlingT0Residue/releaseResidueT1Paint/selectSelectT2CopyandpasteT3DeleteT4T
5MenuRelease
3.4HumanFactorsInvolvedinInterfaceDesign89

1997;Nielsen,1992):learnability,efficiency,memorability,errors,andcoverage.Allpa
rametersareimportantfortheentireinterfaceinwhichthegesturesareused,encompassingt
hestructure,sequence,andfeedbackfromtheuserinter-faceaswellasthegesturesthemselv
es.Sequencingreferstothestepsrequiredtogetfromonestatetothegoalstate.Anexampleis
wordprocessingorfilemanagement.Therearetwomethodstomoveasectionorfile:Copy-Paste
-Delete,orCut-Paste.Thisisanexampleofminimizingthestepsinthesequencing.However,s
omepeoplemaypreferthethreestepsbecauseitprovidesaddedsecuritybyleavingtheorigina
linplaceincasesomethinggoeswronginthetransaction.Thecoreofthehuman-basedapproach
comprisesthefollowingcharacteris-ticsforthechosengestures:Intuitive:Userscanuset
heinterfacewithlittleornoinstruction.Metaphoricallyoriconicallylogicaltowardfunc
tionality:Userscaneasilyseewhatthegesturesarefor.Easytorememberwithouthesitation
:Usersfocusontheirtasksratherthanontheinterface.Ergonomic:Notphysicallystressful
whenusedoften.Becauseofthetechnicallimitationsofgesturerecognitiontechnology,iti

sevenmorenecessarytosimplifythedialogandsequencingthanitiswithconventionalinputm
ethods.Furthermore,itiswisetokeepinmindthattheapplicationmustbeabletodistinguish
thedifferentgestures.Hence,theprinciplefromthetech-nology-basedgesturesmayhaveto
directthechoiceinthedevelopmentofhuman-basedgesturestosomeextent.
FIGURE3.11Technology-basedgesture.Apointinggestureisusedforshapecreationinacompu
ter-aideddesigntool.Source:FromMoeslundetal.(2004).
3GestureInterfaces90

3.4.3ErgonomicsandIntuitivityAnotherwaytoapproachthedevelopmentofagestureinterfa
ceistolookintotheergonomicsandbiomechanicsofgesturingtoensurethataphysicallystre
ssinggestureisavoided.Inthissectionthebiomechanicsofthehand(Linetal.,2001;Lee&Ku
njii,1995)isdescribed.Figures3.12and3.13showthetermsusedinthissection.Asahandyre
ference,Table3.3liststherangesofmotionforthejoints(Eaton,1997)fortheaveragehandi
ndegrees,wherezerodegreesinalljointanglesisastretchedhand.Anexampleofhowtoreadth
enumbersfollows:Thewristextension/flexionof70/75meansthatthewristcanextend70degr
eesupwardandflex75degreesdownward.Hyperextensionmeansextendingthejointfarthertha
nnaturallybyexternalforce.Adductionmeansmovingthebodyparttowardthecentralaxis,wh
ichinthehandisbetweenthemiddleandringfingers(i.e.,gatheringthefingers).Abduction
meansmovingthebodypartawayfromthecentralaxis(i.e.,spreadingthefingers).Pronation
andsupinationmeanrotatingthewristaroundthefore-arm.Iftheneutralpositionfacesthep
almsideways,pronationfacesthepalmdownwardandsupinationfacesitupward.
FIGURE3.12X-rayofarighthandshowingjointnameabbreviationsandwristmovement.
3.4HumanFactorsInvolvedinInterfaceDesign91

ThenumbersinTable3.3arethestaticconstraintsforgesturepostures.However,thereareal
sodynamicconstraintsthatis,intra-andinterfingerconstraints,asinthefollowinglist.
FIGURE3.13Sideviewofahand.Fingerandwristmotiondirectionisshownwitharrows.Forexam
ple,bendingfingersintoafistisflexionofjoints.Stretchingfingersisextensionofjoint
s.
TABLE3.3RangeofMotioninDegreesinHandandWristJointsJointExtension/FlexionHyperext
ensionAdduction/AbductionMetacarpophalangeal(MCP)19004530Proximalinterphalangeal(
PIP)10000Distalinterphalangeal(DIP)8000Trapezio-metacarpophalangeal(TMCP)90100Th
umbdistalinterphalangeal(TDIP)80150PalmarAdduction/abductionRadialAdduction/abdu
ctionThumbbasaljoint(TBJ)Contact/45Contact/60Pronation/supination2Radial/ulnarWr
ist370/7570/8520/351SeeFigure3.12.2Rotationisdonewiththeentireforearm.3Morethano
nejoint.
3GestureInterfaces92

FIntrafingerconstraintsarethedependenciesbetweenthefingerjoints.FInterfingercons
traintsarethedependenciesbetweentheposturesofneighboringfingers.Furthermore,fing
erposturesaffectpressureinthecarpaltunnel(Keiretal.,1998),mostseverelywithmetaca
rpophalangealjointanglesatzerodegrees(fingersextended)andatleast45degrees.Majorp
rinciplesinergonomics(seeGrant,2002;Hedgeetal.,1999;Shaw&Hedge,2002)follow:FAvoi
douterpositions.FAvoidrepetition.FRelaxmuscles.FRelaxedneutralpositionisinthemid

dlebetweenouterpositions.FAvoidstayinginastaticposition.FAvoidinternalandexterna
lforceonjointsandstoppingbodyfluids.Giventheseguidelines,agestureinterfaceispote
ntiallyergonomicallysuperiortophysicalhandhelddevices,whichintroduceexternalforc
e.Whenyouconsiderandcomparebiomechanicsingestures,itisimportanttoevaluateinterna
lforcesandpostureangles.Suchevaluationisappliedinthefollowingparagraphstoacommon
lyusedgesture.TheTVcontrolgestureinFreeman&Weissman(1995)wasatightfist.Feed-back
wasprovidedtonavigateapointeronthescreen.Thedialogwasverysim-ple,buttestsshowedt
hatthegesturewasverytiring.Theergonomicsofthegesturesupportthisconclusion:Theneu
tralpositionisforalljointstobeinthemiddleofrespectiverangesofmotion.Thismeanstha
tafistisaforcedpositionbythemusclesinthehand.Furthermore,thehandisraisedtoheadhe
ight,orhigher,whichplacestheshoulderinorclosetoanouterpositionofexternalrotation
(rotatedawayfromthebody),withtheweightofthearmandhandonit.Onthepositiveside,thew
ristsarekeptstraightwhileoperatingtheTV.ThegesturesinFigure3.8arestressingbecaus
etheydonotfollowtheinter-fingerconstraintsofthehand.Whileconstraintlimitationsar
edifferentforeachperson,theusermustusemoreorlessforcetopositionthefingersforthes
ystemtorecognizethecorrectposture.Furthermore,therecognitionisvulnerabletohowstr
etchedthefingersare.Thismeansthattheusersmuststretchthemtoouterpositions.Theergo
nomicsshowthatmakingtherecognitionalgorithmstoleranttonon-stressingmovementsisim
portant,whichallowstheusertoavoidremaininginafixedorstaticresidueorpointinggesture.T
olerancefordeviationsingesturesisdesirablewhenimplementinggestureinterfaces,also
becauseofvaryinghandshapesandpostureperformance.
3.4HumanFactorsInvolvedinInterfaceDesign93

3.5DESIGNGUIDELINESInthissectionwediscusssomeimportantissuesindesigningagesturei
nterface.3.5.1TheMidasTouchInGreekmythology,thehedonistKingMidashelpedDionysius,
thegodofwine.AsarewardMidasaskedthatDionysiusmakeitpossibleforallthathe,Midas,to
uchedtoturnintogold.Andsoitwasuntilherealizedthateverythingturnedtogold,includin
ghisdaughterandhisfoodanddrink.Seeinghisgiftasacurse,heprayedtoDionysiusandhewas
toldtopassonthepowertoariver,Pactolus,whichhistoricallyhasbeenrichingoldandother
metals.Therelevanceofthisstorytogestureinterfacedesignisimportant:TheMidastouchr
eferstotheever-returningproblemofwhentostartandstopinterpretingagesture.Asadesig
neryoucanexpectspontaneousgesturesfromusersallthetimeifthegoalisnaturalimmersive
behavior.Therefore,thegesturerecognitionmustbeverytolerant.Otherwise,userswoulds
ufferrigidconstraintstotheirbehaviorwhileinthesystem.Unfortunately,designersofte
nselectrigidconstraintsassolu-tions,suchasforcingtheusernottomovebetweengestures
.Alternatively,usersmayhaveamanualtriggerthattellsthesystemwhenagesturestartsand
stops.3.5.2CulturalIssuesItiswidelyrecognizedthatnonverbalcommunicationiscultura
llydependent(Cassell,1998;Agliatietal.,2006)intypology(semantics),rhythm,andfrequency.Perhapsthereareevengenderdifferences,butthisisaninfanttheoryinHCI(Hall,20
06).Animportantquestiontoaskiswhetherculturaldependenceisaproblem.Conventionalin
terfacesthatareinternationalaregenerallyinEnglish,butmostsoftwareisavailablewith
aseriesofnationallanguagepackages,andpeopleinsomenationsusedifferentkeyboards.In
agestureinterface,thiscanbetranslatedtoselectablegesturevocabulariesifitshouldbe
comeaproblemthatanemblemisillogicaltoanotherculture.Furthermore,ifaculturallydep
endentgestureisused,thisdoesnotnecessarilymeanthatitisutterlyillogicalforpeopleo
fotherculturestolearnit.Itiscriticaltoconsiderculturalaspectswhenanalyzinganddev
elopinggestureinterfaces/detectorswithafocusonnaturalhumanconversationandbehavio
r.Thesystemmustbeabletodistinguish(and/orsynthesize)thoseparametersonrhythm,freq
uency,andtypology.3.5.3SequencingThechoiceofgesturesmaydependonyourchoiceofseque
ncing.Avoidingproblemslaterintheprocessiseasierifyoudesignthesequencingfromthest
art.Sequence
3GestureInterfaces94

designinvolvesdecidingeachstepthatauserandthesystemwillgothroughtoaccomplishagiv
entask.Forexample,onewaytodesignaninserttaskfollows:1.Theusermakesacommandgestur
eforinsertandthegestureisperformedatthelocationoftheobjectsplacement.2.Thesystemf
eedsbackalistofobjectstoinsert.3.Theuserusesapointinggesturetoselectthetype.4.Th
esystemremovestheobjectlistandinsertstheobject.Analternativesequencingcouldbethe
following:1.Theusermakesapointinggestureatanobjectzonenexttotheworkspace.2.Thesy
stemhighlightstheobject.3.Theuserusesapointinggestureintheworkspacetoplacetheobj
ect.4.Thesysteminsertstheobject.Notethatanemblemforinsertisonlyneededinthefirste
xample.Itisobviousthatsequencingcanaffectinterfaceefficiencyandlearnability.This
iswhyclassi-calWIMPinterfacesusuallyprovidemultiplepathstothesameobjectiveusingt
hemouseonly,thekeyboardonly,oracombinationofthetwo.Inagestureinterface,whichisqu
itelimited,thismaynotbeafeasiblesolution.Inconclusion,chooseasequencingthatworks
wellfortheintendedusergroupandtheirpreferredgesturevocabulary.Itmaybeamatterofha
bit,whichyoucandeterminefromaWizardofOzexperimentorbyobservingusersbehaviorinexis
tingapplications.3.5.4UseContextAwarenessAsdiscussedinSection3.3.1,humanpercepti
onreliesoncontextawarenessandexpectationsrelatedtothiscontextandpersonalattribut
es.Furthermore,aWIMPinterfacehasacontext-awaremenubyright-clickingonanobject.Thi
smetaphorcanbeimplementedintoyourgestureinterfaceaswell.Context-dependentrecogni
-tioncanbeusedforlimitingthegesturevocabulariesandtosolveambiguity.SpatialZones(
SurroundGUI)Onetechnicallytangiblewayofusingcontextawarenessisbyusingspatialzone
s(Figure3.7).Thespacearoundtheusercanbedividedintovariousinteractionzonessuchasa
workspace,menuspace,templatespace,andcommandspace.Thepositionwhereagestureisperf
ormeddirectstheexpectationofthegesturerecognition.InputModesAnothermethodistoreu
segesturesindifferentinputmodes.Metaphoricallythisissimilartochanginginputmodesf
orthemouseintheWIMPinterface.Forexample,themouseinAdobePhotoshopcanbeusedfordraw
ingorforcutting,dependingontheinputmode.
3.5DesignGuidelines95

UseofArgumentsAthirdmethodistoassignargumentstogestures.Anargumentcanbeaphysicalor
avirtualplaceholderthataltersthemeaningofagesture.IntheprojectcalledARTHUR(Aishe
tal.,2004),theseargumentswerephys-icalplaceholderobjectsmadeoutofcoloredwoodenbr
icks.Architectswouldusetheseplaceholderstomovevirtualbuildingsaroundinavirtualmo
delofacity.Otherplaceholderswereusedforparameterselectionortomodifythevisualappe
aranceofthevirtualbuildings.Thissolutionprovidedrobustselectionandplacementaswel
lasintuitiveinteractionfortheusers,whohadnaturaltactileandvisualfeedbackwhenthey
movedobjects(Figure3.14).3.5.5FeedbackUsingahands-freeinterfacecanmaketheuserfee
ldisconnectedfromthesys-tem.Itisimportanttomaintainaconnectionbysupplyingtheuser
withfeedbackthattellsherthatthesystemisawareofherandwhatsheisdoing.Thisfeedbackc
analsotelltheuserinwhichcontextsheisactingandhowthesysteminterpretstheactions.Th
efeedbackcanbeanysensoryinput(tactile,auditory,orvisual)dependingonthenatureofth
econveyedinformation.Forexample,iftheuserinteractswith3Dobjectsinaworkspace,theo
bjectunderthehandishighlightedinawayotherthantheselectedobjectsare.Thisservestwo
goals:(1)theuserknowsthatthesystemisawareofthehand,and(2)anydiscrepancybetweenth
ephysicalworld
FIGURE3.14ARTHURproject.Left:Architectsinvestigatethelocationandsizeofanewbuildi
nginLondon.Right:Physicalplaceholderobjectsaremovedinplacetoselectthenumberofbea
msinarooftop.Source:FromMoeslundetal.(2004).
3GestureInterfaces96

andthevirtualenvironmentiscalibratedintheconsciousnessoftheuser.Theoutcomeisthat

theuserfeelsfreertomovelesscarefullyintheinterface.Whentheusermovesanobjectclose
toanotherobject,asoundalertstheuseratthetimetheobjectsintersectwhilesimultaneous
lytheintersectedpartsarecoloredred.Thesoundtellstheuserexactlywhentheintersectio
noccursandallowstheusertoreactpromptly.Thecolorshowsthespatialextentoftheinterse
ction.3.5.6IntegrationandDisambiguationDesigningagestures-onlyinterfaceisdifficu
lt.Thetaxonomyforcooperatingwithothermodalitiesiscalledtypesofcooperation(TYCOON
):Transfer:Serialchronology,thatis,firstoneandthentheothermode.Equivalence:Eithe
ror,thatis,useoneortheothermode.Complement:Putthatthere(e.g.SayPutwhilepointingatth
atandmovingthehandtothedestinationandsayingtheretocompletetheaction).Redundancy:Sim
ultaneousmodestoensurerobustness.Specialization:Monopoly,thatis,onlyonemodehasacce
sstothefunction.Onewaytoorganizethetypesofcooperationforanimaginaryapplicationis
showninFigure3.15.Considerhavingthreeinputmodes:speech,vision,andtraditionalWIMP
.EachfunctionintheapplicationshouldbeassignedtooneormoremodesusingTYCOON.Forexam
ple,Exitisasensitivefunction,whichshouldnotbeaccessedbyaccident.Thus,itisdetermine
dthatallthreemodes
Confirm (E)Pause (S)TransferEquivalenceComplementaryRedundancySpecializationSele
ct all (E)Configuration (S)Assign PHO(T)Moving objects (C)Exit (R/T)Vision-based
SpeechTycoonTraditionalScaling (S)
FIGURE3.15Organizingtypesofcooperation.Assignmentoffunctionstothreemodalitiesusi
ngtheTYCOONtaxonomy.
3.5DesignGuidelines97

shouldbeusedsimultaneouslytoexittheapplication.TheusermustsayExitandmakeanexitgest
ure(redundancy),andthenconfirmviathekeyboard(transfer).Whenusingmultiplemodaliti
es,thereisariskofconflictinginformation.Anadvanceexpectationmodulecanpreventsuch
conflicts,ortwomodescanalsodisambiguateinformation.Anothersolutiontoconflictingi
nformationistoassignmodepriorityandactionpriority.Onemodeoractioncanhavepreceden
ceovertheother.Forexample,whenonemodesaysYesandtheothersaysNo,modepri-oritygiven
tothefirstleadstosystemexecutionoftheYescommand.IftheactionNoisgivenpriority,int
hisinstancetheNocommandisexecuted.3.6HOWTOBUILDANDTESTAGESTUREVOCABULARYSection3
.4outlinedtheimportanceofchoosingalogicalandergonomicgesturevocabulary,anddiscus
sedparametersthatcanbetunedtoachievingthesame.Thissectionpresentsaprocedureforho
wtofindanappropriategesturevocabularyforagivenapplication.Thedescriptionisaccomp
aniedbyapracticalexamplethatshowshowitcanbeused.Toensureintuitiveandlogicalmappi
ng,aninvestigationintotheinterfacesofknownapplicationsshouldbeconductedtoidentif
ytheneededfunctionalities.Itisimportanttorememberthatagesturevocabularymustbetai
loredforthepur-poseoftheapplicationandfortheusergroupoftheapplication.Agesturevo
cab-ularyshouldnotbeimposedarbitrarilyonanygivenapplication.Inthedevelopmentofin
terfaces,scenarioshaveprovenvaluable(Dzida&Freitag,1998)todefinethecontextandfun
ctionalitiesandtoinvestigatetheuserandproblemdomains.Ascenariocanbeusedtoexamine
humanhumannonverbalcommunication.Userswouldparticipateinscenariosinwhichtheycommu
nicatethesamethingstoapersonthattheywouldcommunicatetothecomputerapplication.The
rearebottom-upandtop-downapproachestothisinvestigation.Thebottom-uptypefindsmatc
hinggesturestofunctions,whilethetop-downapproachpresentsgesturesandidentifieslog
icallymatchedfunctions.Outlinesofvariationsontheseapproachesfollow:Subconscious,
bottom-up:Createscenariosofcommunicationbetweenpeople.Recordthetransactionsonvid
eoandanalyzethesubconsciousgesturingthatoccurs.Conscious,bottom-up:Askforeachfun
ctionalityforwhichagesturewouldbeused,ormakeaguideddrawingtest.Atesteereceivesar
andomdrawingprojectandguidestheoperatorwithgesturesonwhattodraw.Ifanobjecttempla
tebasedapplicationisconsidered,thoseobjectscanbeprovidedonthetable.
3GestureInterfaces98

Top-Down:Askeachtesteewhatagivengesturecouldmean,ordoadrawingtest.Theoperatorgui
desthetesteewithgesturingwhattoputonpaperusingapre-definedgesturevocabulary.This
isusefulfortestingagesturevocabulary.Anothernecessarytoolisabenchmarktomeasureth
equalityofagesturevocabularyaccordingtoprinciplesinthehuman-basedapproach.3.6.1T
heProceduretoFindGesturesThissectiondescribestheproposedprocedureandbenchmarkwhe
ndevelopingagestureinterface.IntheonlinecasestudyreferredtoinSection3.7,wedemonstrateourownexperimentusingtheprocedure.Itisaniterativeapproach,wheregesturesare
foundandtestedandresultsareusedtorefinethechoices.StepA:FindtheFunctionsFindthef
unctionsthatthegestureswillhavetoaccess.Keepinmindtheuserinterfaceinexistingsimi
larapplicationsonstandardinterfaces.(Forexample,weweredesign-inganarchitecturald
esignapplication,sowelookedat3DStudio,amongothers.)StepB:FindLogicalGesturesFind
thelogicalgesturesthatrepresentthefunctionsfoundinStepA.Thiscanbedonethroughinte
rviewsorexperimentswithpeoplebytakingthemthroughsce-nariosundercamerasurveillanc
ewheretheycommunicatethesamemessagesthattheywouldcommunicatetothecomputertotheope
rator(i.e.,thepersonwhoconductstheexperiment).Itisimportantthatthescenariostaketh
etesteesawayfromnormaltechnicalthinking,especiallywhenconductingthetestsontechni
calpeople.Ifyoutellanengineerthatitisatestforthegesturerecognitionsystem,theengi
neerwillhaveaconsciousdesigninmindandactaccordingtothehypotheticalrestrictionsof
thatdesign.Inbrief,youriskthatsuchapersonwillstillthinkintermsofinter-facesandal
gorithms.Ifyouwanttowriteascenariowithatechnicalinterfaceaspect,itcanbeperformed
asaWizardofOzexperiment,whichtestsnotonlythegesturingbutalsothedesignoftheentire
interface,includingthefeedbackfromthesystemandthesequencingintheinterface.Thenum
berofpeoplerequiredforthisinvestigationdependsonhowbroadtheusergroupisandhowdive
rsetheresultsofthetestare.Becauseitisatime-consumingprocess,usingupto10participa
ntsistypical.StepC:ProcesstheDataNoteandcapturetheframeswiththecommonlyusedgestu
res,andnotehowcon-sistentlythevarioustesteesusethem.Noteiftheyareusedonlyasstati
cposturesorifdynamicsplayanimportantpartininterpretingthegesture.
3.6HowtoBuildandTestaGestureVocabulary99

ThetheorydiscussedinSection3.4.3shouldbetakenintoaccountintheselectionofgestures
:FEvaluateinternalforcecausedbyposture.DeviationfromneutralpositionOuterlimitsForce
sfrominterjointrelationsFEvaluatefrequencyanddurationofthatgesture.FConsidereffe
ctonwristfromwristandfingerposture.Seetheonlinecasestudy(StepC)toexaminehowthisi
sdoneinpraxis.StepD:BenchmarktheChosenGestureVocabularyThefinalstepistotestthere
sultinggesturevocabulary.TheattributesshowninTable3.4aretobetestedinthebenchmark
.Test1:GuesstheFunctionGivethetesteealistoffunctions.Presenttheges-turesandexpla
inthegeneralideaoftheapplication.Askthepersontoguessthefunctions.Gesturesthatdep
endoncontextmustbepresentedincontext,butnothowtheapplicationreactstothem.Explain
carefullythatitistheges-turesthatarebeingtested,nottheintelligenceofthetestee.Th
istakestensionawayfromthetestee.ScorewrongguessesdividedbynumberofgesturesTest2:M
emoryThistestmeasureshowlongittakestobecomecomfortablewiththegestures.Onlywhenth
egesturesareperformedwithouthesitationwilltheuserkeeptheapplicationinfocusrather
thantheinterface.Givethegesturevocabularytothetestee,whowillexperimentwiththeges
turestomakesuretheyareunderstood.Remindthetesteeagainthatitistheges-turevocabula
rythatissubjecttothetest,notthetesteesabilitytoperformit.
TABLE3.4OverviewofWhatisActuallyTestedinEachTestAttributeEvaluatedinTest___Seman
ticinterpretation1Generalization1Intuitivity1,2Memory,learningrate2Stress3
3GestureInterfaces100

Presentaslideshowoffunctionsataswiftpace,twosecondsperfunction.Thetesteemustperf
ormthemcorrectly.Continueuntiltheyareallcorrect.Theordershouldbelogicaltowardseq
uencesintheapplication,butitshouldnotbethesameeverytime.Thenthetesteewillsimplym
emorizetheorderofgesturesandnotthemapping.Restarttheslideshowaftereverymistake,a
ndshowthegesturevocab-ularytothetesteebetweeneachretry.ScorenumberofrestartsTheou
tcomeisanindicatorofhowdifficultitwillbeforanewusertobecomecomfortablewiththeges
turevocabularytotheextentthatthegestureswillnotstealthefocusfromtheuserstasks.Tes
t3:StressThisisasubjectiveevaluationofergonomics.Presentthetesteewithalistofaseq
uenceofgestures.ThetesteemustperformthesequenceXtimes,whereXtimesthesizeoftheges
turevocabularyequalsatleast200.Gobacktotheneutralhandposition(residue)betweeneac
hgesture.Afterwardaskthetesteehowstressfuleachgesturewasandtoprovideanoverallrat
ingforusingeachgestureoveralongtime.Meanwhiletheoperatorshouldnotehowergonomical
lythetesteeisperformingthegesturesandhowthetesteeissittingatthetable.Weusedthefo
llowingscorelistforeachgestureandoverallforthesequence:1.Noproblem2.Mildlytiring
/stressing3.Tiring/stressing4.Veryannoying/painful5.ImpossibleToplacetheprocessi
ncontext,asitisverysubjective,acomputermousewouldbescoredataround2fornormallyabl
epeople.Amoreobjectivestresstestwouldrequireelectrodesonthenervesatthewristandfo
rearmtomeasurestrain.Suchtestingmaybeofinteresttopainresearchersinparticular.The
benchmarkcanbeusedtocomparemultiplegesturevocabulariesorusergroups.Test2isonlyco
mparableifthevocabulariesareofthesamesize.Iftest-ingasinglevocabulary,reasonable
successcriteriamustbeascertained.Theseaimsdependonthegesturevocabularyathand.See
theonlinecasestudy(StepD)toseehowthisisdoneinpraxis.WizardofOzexperimentshavepro
venvaluableinthedevelopmentofges-tures(Beringer,2002;Carbinietal.,2006).Theexper
imentssimulatetheresponseofthesystembyhavingapersonrespondtotheusercommands.This
approachtestsadevelopedinterfaceandcanidentifyproblems,withthechoiceofgesturesan
dsequencingasthenecessaryfeedback.
3.6HowtoBuildandTestaGestureVocabulary101

3.6.2TechnicalTestsOncetheinterfacehasbeenimplemented,itisimportanttoconductaser
iesoftechnicalperformancetests.Aninterfacethatrespondsinadelayedfashionorisimpre
cisewillnotbesuccessful.Interfacesthatassumesloworfairlystaticmovementsarelessse
nsitivetothisthanthoseinwhichmovementisallowedtobefluentandnatural.PrecisionofPo
sition,Orientation,andTimingTheimportantprecisionparametersthatshouldbetestedfol
low:Position:Measureanydiscrepancybetweenwherethesystemthinksthegestureisperform
edandwhereitreallyisinreal-worldunits.Orientation:Whenapplicable,measureanydiscr
epancybetweenthedirectionthatagestureispointingandtherealdirection.Timing:Measur
etheresponsedelayintimeunits.Notestart,stroke,andendtimes,wherestrokeistheclimax
ofthegesture.Forexample,thestrokeofrais-ingahandintheairiswhenthehandreachesitsh
ighestposition.Evaluatewhethertheresultsarecriticaltotheapplicationandwhetheriti
ssafetohavetheusercompensatebyusingagivenfeedback.Thisfeedbackcouldbesoundtomark
themomentthattheapplicationtriggersanaction,oravisualmarkerthatshowswheretheappl
icationthinksthefingersare.AccuracyofClassificationsThisparameterisameasureofges
turedetectionrobustness.Accuracyistheper-centageofgesturesdetectedbythecomputerb
asedonalargeseriesofgestureclassifications.3.7CASESTUDYAcasestudyforagestureinte
rfacecanbefoundatwww.beyondthegui.com.3.8SUMMARYAgestureinterfacecanbeagoodwayto
enhancetheimmersivefeelinginavirtual-realityapplicationandagoodmeansofmakingtheu
sersexperiencefeelmorenatural.Thischapterprovidedbackgroundinformationonvariouswo
rkandtypesofgestureinterfaces.Italsooutlinedfundamentaltheoryandissuestobeaddres
sedwhenconsideringanddesigningsuchaninterface.Gesturesareparticularlyusefulwhenr
elatingtothespacearoundtheuseraswellasforconveyingcommands.Inparticular,westress
edthatthegesture
3GestureInterfaces102

vocabularymustbeconcise,intuitive,andergonomictoperform,becauseitcanbetiringtous
esuchaninterfaceforaprolongedtime.Aprocedureforachievingthiswasproposedanddemons
trated,andtheresultinggesturevocabularywastestedonvarioususergroupsandcomparedto
apurelytechnical-basedgesturevocabulary.Thesuccessofthegesturevocabu-larydepende
dontheusergroups,andthetechnicalgestureswerenotaseasytouseastheintuitiveset.3.9F
UTURETRENDSThefuturewilllikelybringmanyinterestingbreakthroughsinthisarea.Rawcom
-putervisionandotherdetectionsystemswillevolveintorobustclassifierswithlessobstr
uctivedesigns.Thismeansthattherewillbenodevicesattachedtotheuser,sothattheuserca
nfeelcompletelyimmersedinthevirtualenviron-mentthatblendsperfectlywiththerealwor
ld.Classicalcomputersystemsseeinputmodesandoutputmodesasinterchangeableinputmodu
lestothesystem.Thisdefinesaclearboundarybetweentheuserandthesystem.Weshouldstopt
hinkingofaninterfaceasjust...aninterface.Wedonotsaythatweinterfacewithacar.Wesay
thatwedrivetosomelocation,thatis,thetaskofdrivingtothelocationisinfocus.Ifwecanb
ringtheapplicationandtheuserclosertoeachotherinaninvisibleinterface,theuserfocus
esontheobjective,notthecomputerorinterface,whichisjustanaturalpartoftheenvironme
nt.Wepredictthathands-freegestureinterfaceswillbeassmoothasJeffersons(Han,2005)mu
ltitouchinterface,andwillbeusedtocreateintelligentroomswithubiquitouscomputingan
dadvancedimmersivevirtual-andmixed-realityenvir-onments,whichwillbeawareofpeople
intherealworld.Theseenvironmentswillbeawarenotonlyofpeoplespresencebutoftheiremot
ions,actions,andintentionsaswell.Computerswillbeabletoadapttouserstosuchadegreet
hattheybecomeanaturalpartoflifealmostlikealivingorganism.REFERENCESAgliati,A.,Vesco
vo,A.,&Anolli,L.(2006).Anewmethodologicalapproachtononverbalbehavioranalysisincu
lturalperspective.BehaviorResearchMethods38(3):36471.Aish,F.,Broll,W.,Stoerring,M
.,Fatah,A.,&Mottram,C.(2004).ARTHUR:Anaugmentedrealitycollaborativedesignsystem.
ProceedingsofFirstEuropeanConferenceonVisualMediaProduction,London.Aviles-Arriag
a,H.H.,Sucar,L.E.,&Mendoza,C.E.(2006).Visualrecognitionofsimilargestures.Proceed
ingsofEighteenthInternationalConferenceonPatternRecognition1:1100103.Beringer,N.(
2002).EvokinggesturesinSmartKomDesignofthegraphicaluserinterface.LectureNotesinAr
tificialIntelligence2298,22840.
3.9FutureTrends103

Betke,M.,Gips,J.,Fleming,P.(2002).Thecameramouse:visualtrackingofbodyfeaturestop
rovidecomputeraccessforpeoplewithseveredisabilities.IEEETransactionsonNeuralSyst
emsandRehabilitationEngineering10(1):120.Bevan,N.,&Curson,I.(1997).Methodsformeas
uringusability.ProceedingsofSixthInter-nationalFederationofInformationProcessing
ConferenceonHumanComputerInterac-tion,Sydney,Australia.Bowyer,K.W.,Chang,K.,&Flyn
n,P.2006.Asurveyofapproachesandchallengesin3Dandmulti-modal3D+2Dfacerecognition.
ComputerVisionandImageUnderstanding101:115.Caldwell,D.G.,Tsagarakis,N.,&Giesler,C
.(1999).Anintegrationtactile/shearfeedbackarrayforsimulationoffingermechanorecep
tor.Proceedingsof1999IEEEInternationalConferenceonRoboticsandAutomation,Detroit.
Carbini,S.,Delphin-Poulat,L.,Perron,L.,&Viallet,J.E.(2006).FromaWizardofOzexperi
-menttoareal-timespeechandgesturemultimodalinterface.SignalProcessing86:355977.Ca
ssell,J.(1998).Aframeworkforgesturegenerationandinterpretation.InCipolla,R.,&Pen
tland,A.,eds.,ComputerVisioninHumanMachineInteraction.NewYork:CambridgeUniversity
Press,191215.DeSilva,P.R.,Marasinghe,C.A.,Madurapperuma,A.P.,Lambacher,S.G.,&Osan
o,M.(2006).Modelingcognitivestructureofemotionfordevelopingapedagogicalagentbase
dinteractivegame.IEEEInternationalConferenceonSystems,ManandCybenetics1(811):33944
.Dzida,W.,&Freitag,R.(1998).Makinguseofscenariosforvalidatinganalysisanddesign.I
EEETransactionsonSoftwareEngineering24:118296.Eaton,C.(1997).ElectronicTextbookon
HandSurgery.Availableathttp://www.eatonhand.com/.Federico,M.(1999).UsabilityEval
uationofaSpokenData-EntryInterface.Povo,Trento,Italy:ITC-IrstCentroperlaRiceraSc
ientificaeTechnologica.Fihl,P.,Holte,M.B.,Moeslund,T.B,&Reng,L.(2006).Actionreco
gnitionusingmotionprimitivesandprobabilisticeditdistance.ProceedingsofFourthInte
rnationalConferenceonArticulatedMotionandDeformableObjects.Mallorca,Spain.Finger
works.(2002).iGesturePad.Availableathttp://www.fingerworks.com/igesture.html.Fre

eman,W.T.,&Weissman,C.D.(1995).Televisioncontrolbyhandgestures.Proceed-ingsofIEE
EInternationalWorkshoponAutomaticFaceandGestureRecognition.Goto,S.(2006).Thecase
studyofanapplicationofthesystem,BodySuitandRobotic-Music:Itsintroductionandaesthetic
s.ProceedingsofConferenceonNewInterfacesforMusicalExpression,Paris,29295.Grant,C.
(2002).TenThingsYouShouldKnowaboutHandandWristPain.AnnArbor,MI:F-OneErgonomics.H
all,J.(2006).Nonverbalbehavior,statusandgender:Howdoweunderstandtheirrelations?P
sychologyofWomenQuarterly30:38491.Han,J.Y.(2005).Low-costmulti-touchsensingthroug
hfrustratedtotalinternalreflection.ProceedingsofEighteenthAnnualAssociationforCo
mputingMachinerySymposiumonUserInterfaceSoftwareandTechnology,Seattle,11518.Hedge
,A.,Muss,T.M.,&Barrero,M.(1999).ComparativeStudyofTwoComputerMouseDesigns.Ithaca
,NY:CornellHumanFactorsLaboratory.Hummels,C.,&Stapers,P.J.(1998).Meaningfulgestu
resforhumancomputerinteraction:Beyondhandpostures.ProceedingsofThirdInternationa
lConferenceonAutomaticFace&GestureRecognition,Nara,Japan,59196.
3GestureInterfaces104

Jaimes,A.,&Sebe,N.(2005).Multimodalhumancomputerinteraction:Asurvey.Proceed-ings
ofIEEEInternationalWorkshoponHumanComputerInteractioninConjunctionwithInternation
alConferenceonComputerVision,Beijing.Keates,S.,&Robinson,P.(1998).Theuseofgestur
esinmultimodalinput.ProceedingsofAsso-ciationforComputingMachinerySpecialInteres
tGrouponAccessibleComputing,3542.Keir,P.J.,Bach,J.M.,&Rempel,D.M(1998).Effectsoff
ingerpostureoncarpaltunnelpressureduringwristmotion.SanFrancisco:DivisionofOccup
ationalMedicine,UniversityofCalifornia.Klein,J.,Moon,Y.,&Picard,R.W.(2002).Thisc
omputerrespondstouserfrustration:Theory,design,andresults.InteractingwithCompute
rs14(2):11940.Langdon,P.,Keates,S.,Clarkson,P.J.,&Robinson,P.(2000).Usinghapticfe
edbacktoenhancecomputerinteractionformotion-impairedusers.ProceedingsofThirdInte
rna-tionalConferenceonDisability,VirtualRealityandAssociatedTechnologies,Alghero
,Italy.Lee,J.,&Kunjii,T.(1995).Model-basedanalysisofhandposture.IEEEComputerGrap
hicsandApplications15:7786.Lin,J.,Wu,Y.,&Huang,T.S.(2001).Modelingthenaturalhandm
otionconstraints.ProceedingsofFifthAnnualARLFederatedLaboratorySymposium,10510.Lo
ng,A.C.,Landay,J.A.,Rowe,L.A.,&Michiels,J.(2000).Visualsimilarityofpengestures.C
HILetters:HumanFactorsinComputingSystems(SIGCHI)2(1):36067.Loomis,J.M.,Klatzky,R.
L.,Philbeck,J.W.,&Golledge,R.G.(1998).Assessingauditorydistanceperceptionusingpe
rceptuallydirectedaction.Perception&Psychophysics60:96680.Min,B.W.,Yoon,H.S.,Soh,
J.,Ohashi,T.,&Ejima,T.(1999).Gesture-basededitingsystemforgraphicprimitivesandal
phanumericcharacters.EngineeringApplicationsofArtificialIntelligence12(4):42941.M
oeslund,T.B.,&Granum,E.(2003).SequentialMonteCarlotrackingofbodyparametersinasub
-space.IEEEInternationalWorkshoponAnalysisandModelingofFacesandGestures,8491.Moes
lund,T.B.,Storring,M.,Broll,W.,Aish,F.,Liu,Y.,&E.Granum.(2004).TheARTHURsystem:An
augmentedroundtable.ProceedingsofEighthWorldMulti-ConferenceonSystemics,Cybernet
icsandInformatics,Orlando.Moeslund,T.B.,Storring,M.,&Granum,E.(2004).Pointingandc
ommandgesturesforaugmentedreality.ProceedingsofFGnetWorkshoponVisualObservationo
fDeicticGestures,Pointing,inAssociationwithSeventeenthInternationalConferenceonP
atternRecognition,Cambridge.Moustakas,K.,Strintzis,M.G.,Tzovaras,D.,Carbini,S.,B
ernier,O.,Viallet,J.E.,Raidt,S.,Mancas,M.,Dimiccoli,M.,Yagdi,E.,Balci,S.,&Leon,E
.I.(2006).Masterpiece:Physicalinteractionand3DcontentbasedsearchinVRapplications.
IEEEMultimedia13(6):92100.Muller-Tomfelde,C.,&Steiner,S.(2001).Audio-enhancedcolla
borationataninteractiveelectronicwhiteboard.ProceedingsofInternationalConference
onAuditoryDisplay,Espoo,Finland.Nielsen,J.(1992).Theusabilityengineeringlifecycl
e.Computer25:1222.Paggio,P.,&Music,B.(2000).LinguisticinteractioninstagingAlanguag
eengineeringview.InQvortrup,L.,ed.,VirtualInteraction:Interactionin/withVirtualI
nhabited3DWorlds.NewYork:Springer-Verlag,23549.
References105

Pantic,M.,&Patras,I.(2006).Dynamicsoffacialexpression:Recognitionoffacialactions
andtheirtemporalsegmentsfromfaceprofileimagesequences.IEEETransactionsonSystems,
ManandCybernetics,PartB36(2):43349.Pantic,M.,&Rothkrantz,L.J.M.(2003).Towardanaff
ect-sensitivemultimodalhumancomputerinteraction.ProceedingsofIEEE91(9):137090.Rasm
ussen,N.T.,Storring,M.,Moeslund,T.B.,&Granum,E.(2006).Real-timetrackingforvirtual
environmentsusingSCAATKalmanfilteringandunsynchronizedcameras.ProceedingsofInter
nationalConferenceonComputerVisionTheoryandApplications,Setubal,Portugal.Riviera,
J-B.,&Guitton,P.(2004).Handposturerecognitioninlarge-displayVRenviron-ments.InGe
sture-BasedCommunicationinHuman-ComputerInteraction.Berlin:Springer,25968.Sage,K.
,Howell,A.J.,&Buxton,H.(2003).DevelopingcontextsensitiveHMMgesturerec-ognition.P
roceedingsofFifthInternationalWorkshoponGestureandSignLanguage-BasedHumanComputer
Interaction,Genova,Italy.Shaw,G.,&Hedge,A.(2002).Theeffectofkeyboardandmouseplac
ementonshouldermuscleactivityandwristposture.ErgoInfo,availableathttp://www.huma
nscale.com/ergo_info/articles.cfm?type=studies.Steininger,S.,Lindemann,B.,&Paetz
old,T.(2002).LabelingofgesturesinSmartKomThecodingsystem.ProceedingsofGestureandS
ignLanguageinHumanComputerInteraction,InternationalGestureWorkshop,London,21527.St
orring,M.,Andersen,H.J.,&Granum,E.(2004).Amultispectralapproachtorobusthumanskind
etection.ProceedingsofSecondEuropeanConferenceonColorinGraphics,ImagesandVision.
Storring,M.,Granum,E.,&Moeslund,T.B.(2001).Anaturalinterfacetoavirtualenvironment
throughcomputervision-estimatedpointinggestures.ProceedingsofWorkshoponGesturean
dSignLanguage-BasedHCI,London.Streitz,N.A.,etal.(2001).Roomware:TowardstheNextGe
nerationofHumanComputerInteractionBasedonanIntegratedDesignofRealandVirtualWorlds
.GermanNationalResearchCenterforInformationTechnology,IntegratedPublicationandIn
formationSystemsInstitute.Tsagarakis,N.G.,Gray,J.O.,Caldwell,D.G.,Zannoni,C.,Pet
rone,M.,Testi,D.,&Viceconti,M.(2006).Ahaptic-enabledmultimodalinterfaceforthepla
nningofhiparthroplasty.IEEEMultimedia13(6):4048.Tse,E.,Shen,C.,Greenberg,S.,&Forl
ines,C.(2006).Enablinginteractionwithsingleuserapplicationsthroughspeechandgestu
resonamulti-usertabletop.ProceedingsofWork-ingConferenceonAdvancedVisualInterfac
esTableofContents,Venezia,Italy,33643.Vroomen,J.,&deGelder,B.(2004).Temporalventr
iloquism:Soundmodulatestheflash-lageffect.JournalofExperimentalPsychology:HumanP
erceptionandPerformance30(3):51318.Wachs,J.,Stern,H.,Edan,Y.,Gillam,M.,Feied,C.,S
mith,M.,&Handler,J.(2006).Areal-timehandgestureinterfaceforamedicalimage-guideds
ystem.ProceedingsofNinthIsraeliSymposiumonComputer-AidedSurgery,MedicalRobotics,
andMedicalImaging.Wang,S.B.,Quattoni,A.,Morency,L-P.,Demirdjian,D.,&Darrell,T.(2
006).Hiddenconditionalrandomfieldsforgesturerecognition.IEEEComputerSocietyConfe
renceonComputerVisionandPatternRecognition2:152127.
3GestureInterfaces106

4CHAPTERLocomotionInterfacesMaryC.Whitton,SharifRazzaqueThischapterisaboutlocomo
tioninterfacesinterfacesthatbothenableuserstomovearoundinrealorvirtualspacesandma
keusersfeelasiftheyaremoving.Locomotionisaspecialtypeofmovement:Locomotion,asuse
dbylifescientists,referstotheactofanorganismmovingitselffromoneplacetoanother.Th
isincludesactionssuchasflying,swimming,andslithering.Forhumans,locomo-tioniswalk
ing,running,crawling,jumping,swimming,andsoon.InFigure4.1,theusermovesfromonepla
cetoanotherbyleaningtocontrolspeedanddirec-tionoftheSegwayPersonalTransporter(PT
);inFigure4.2,theuseris(really)walkingonthetreadmill(shownindetailontheright)tom
ovethroughavirtuallandscape.Thefocusofthischapteriscomputer-basedlocomotioninter
facesformovingaboutincomputer-generatedscenes.Onewaytothinkoftheseinterfacesisth
attheyarevirtual-locomotioninterfacesforvirtualscenes.Wemayoccasionallyuseanexam
pleoflocomotionintherealworld,butwhenwesaylocomotioninterface,wemeanvirtual-loco
motioninterface.Manycommonvirtual-locomotioninterfacestakeinputfromonlytheusersha
nds;forinstance,three-dimensional(3D)computervideogamesareoperatedbyjoystick,key
board,mouse,and/orgamecontroller.Otherinterfacesusemoreofthebody.Forexample,inth
erealworld,SegwayPTs,hanggliders,andskate-boardsrequireuserstoleanandshifttheirw
eighttocontrolmotion.Inavirtual-realitysystemthathasareal-walkinglocomotioninter

face,peopleusetheirbodiesinacompletelynaturalfashiontowalkaboutthevirtualscene(F
igure4.3).Thischapterconcentratesonvirtual-locomotioninterfacesthatinvolveallof,
oratleastmuchof,theusersbody.

Eventhoughbotharmsandbothlegsmaybeinvolvedincontrollingavehicleandthewholebodyma
ybeinvolvedinreceivingfeedbackaboutthestateofthevehicle,simulatingmovementresult
ingfromoperatingavehicleisconceptuallydifferentfromsimulatingthemovementofahuman
walkingabouthuman-scalespacessuchasbuildingsandships.Inavehicle,usersintentionsof
whatdirectiontomoveandhowfasttogoarespecifiedunambiguouslybytheirinteractionwith
thecontrolsofthevehicle.Thechallengeforvirtual-locomotioninterfacesistocaptureth
eusersintentusingdatathatcanbederivedfromsensingthepose(positionandorientation)an
dmovementoftheusersbody.Thescopeofthischapteris,then,interfacesbetweenhumanusersa
ndcomputer-generatedscenesthatenableuserstocontroltheirlocomotionthroughthe(virt
ual)scenebymovingtheirbodies.Thebodyposeandmovementspecifyhowtheywanttomovebothth
edirectiontheywanttomoveinandhowfasttheywanttomove.
FIGURE4.1Segwayi2self-balancingPersonalTransporter.Theusercontrolsthespeedbylean
ingforwardorbackwardandturnsbyleaningtotherightorleftwiththeLeanSteerframe.(Cour
tesyofSegwayInc.)
4LocomotionInterfaces108

FIGURE4.2UniversityofUtahsTreadportvirtual-locomotioninterface.Left:Theuserwalkso
natreadmillwhileviewingthemovingvirtuallandscapeonthelargeprojectorscreens.Right
:Tosimulatehills,theentiretreadmillcantiltup.Tosimulatetheusersvirtualinertia,the
Treadportphysicallypushesorpullstheuserviaalargerodthatconnectstoauser-wornharne
ss.(CourtesyofUniversityofUtah,SchoolofComputing;#JohnM.Hollerbach.)
FIGURE4.3Real-walkingvirtuallocomotioninterface.Tomoveaboutthevirtualscene(right
),theuseractuallywalksaboutthereal-worldlaboratory(left).(ImagescourtesyoftheDep
artmentofComputerScience,UNCatChapelHill.)
4LocomotionInterfaces109

Therearethreedominantmetaphorsforvirtuallocomotion.Inreal-walkingstylesystems,inp
utmovementsandresultingmovementthroughthespaceareasnaturalandasmuchlikereallywal
kingaspossible.Examplesaretwo-dimensional(2D)treadmills,walking-in-place,andreal
-walkinginterfaces.Invehicle-styleinterfaces,inputmovementsandresponsearesimilar
todrivingavehicle.Forexample,Flemingandcolleagues(2002)developedajoystickinter-f
aceinwhichpushingthejoystickup(awayfromyou)movesyouforward,andpushingthejoystick
leftrotatesyouleftbutdoesnotmoveyouforward.Tomoveleft,youmustfirstturn(rotate)le
ftusingthejoystickandthenmoveforward.Thisissimilartohowadriveroperatesatank.Both
real-walkingandvehicle-styleinterfacesfallintoacategoryoflocomo-tiontechniquesth
atSlaterandUsoh(1994)callmundane,thatis,theyattempttomimic,ascloselyaspossible,h
owwewalkorcontrolvehiclesintherealworld.Incontrast,magical-styleinterfacesaretho
sethatpermitmovementsthathavenonaturalcorollaryintherealworldbutthatserveauseful
purposewhenyouaremovingaboutinavirtualscene.Forinstance,theabilitytoteleportbetw
eentwodistantlocationsisamagicalcomponentofalocomotioninterface.Similarly,theabi
litytoadaptthelengthofyourvirtualstridesothatyoumovemileswitheachstepisamagicalp
roperty.Regardlessofthechoiceofmetaphor,theinterfacehastoconverttheusersphysicalm
ovement(intherealworld)intovirtualmovementbothdirectionandspeedinthevirtualscene.T

able4.1givessomeexamplesofbodymove-mentsandhowtheymightbeusedtocontrollocomotion
directionandspeed.Notethatlocomotionisdistinctfromwayfinding,whichreferstothecog
nitivetaskofdeterminingarouteforreachingadestination.Weavoidusingwayfindingbecau
seitisusedintheliteraturetorefertobothwayfindingandlocomotion.Otherauthorshavere
ferredtomovementssuchasgoingfromavirtualdesktoavirtualbookshelfastravel(e.g.,Bow
man,Koller,&Hodges,1997),butweprefer
TABLE4.1ExamplesofHowBodyMovementMightBeInterpretedtoControlDirectionandSpeedofL
ocomotionBodyPartDirectionControlSpeedControlHandsMoveindirectionuserispointingD
istancebetweenhandscontrolsstepsizeFeetMoveindirectionfeetarepointingStepfastero
rslowertospeedorslowpaceLegsSwinglegsidewaysfromhiptomovesidewaysStepfasterorslo
wertospeedorslowpaceArmsMakebankinggesturetoturnlikebirdFlaparmsfasterorslowertosp
eedorslowpace
4LocomotionInterfaces110

locomotion1becausetravelimpliesalargedistanceonetravelsfromNewYorktoChicago,butdo
esnottravelacrossaroom.Hollerbachs(2002)definitionofalocomotioninterfacerequirest
hattheusermakerepetitivemotionssuchastappingakeyonakeyboardorwalking-in-place.We
donotuseHollerbachsdefinitioninthischapter.4.1NATUREOFTHEINTERFACETheessenceofalo
comotioninterface(forbothrealandvirtualworlds)canbegeneralized.Userscommunicatet
heirintentionstothesystem(computerorvehicle)bymakingreal-worldmotionstheypresskey
s,moveajoystick,movetheirhands,leanforward,pushwithafoot,andsoon.Theseuserinputm
otionsarethenconvertedthroughaninterfacesystemtomovementthroughtherealorvirtualw
orld.Thesystemmayalsoprovidefeedbacktotheusers,indicatinghowtheyaremovingorhavem
oved.Usersusethefeedbacktoplantheirnextmotionand,forvisualfeedback,tohelpmaintai
nbalanceandtoupdatetheirmentalmodeloftheirsurroundings.4.1.1ThePhysiologicalNatu
reoftheInterface:PerceivingSelf-MotionThereareseveralsensorychannelsthatprovideu
swithinformationonhowandwherewearemoving.Eachofthesesensorychannelsauditory,visua
l,vestibular(fromtheinnerear),andproprioceptive(thesenseofbodyposition)contribute
sinformationtoawarenessofself-motion.Humansrelyonsensorycuestodeter-minewhethert
heythemselvesaremoving(self-motion)oriftheobjectsaroundthemaremoving(externalmot
ion)andforbalanceandorientation(Dichgans&Brandt,1977).Auditory:Humanshavetheabil
itytodeducequalitiesoftheirenvironmentfromthewaytheenvironmentsounds;forinstance
,largeroomssounddifferentfromsmallrooms.Wealsohavetheabilitytolocalizesoundsourc
es,andaswemovetheperceivedlocationofastationarysoundmovesinrelationtoourhead.Amo
vingsoundsourcecan,byitself,causeastationarypersontofeelasifsheismoving(Lackner,
1977).Tactile,proprioceptive,andpodokinetic:Humanscansensemovementintheirjoints,
muscles,andviscera,andcansensepressureandslippageontheskin.Thesecuesareimportant
forwalking,astheytellapersonwhereherlimbsare1Researchersmayhaveadoptedthetermtra
velbecausetheverbformoflocomotion,tolocomote,soundsawkward.
4.1NatureoftheInterface111

andwhenherfeetaretouchingtheground.Thesecuesalsocommunicatetherelativemotionofth
epersonsbody(i.e.,howthelimbsmoverelativetothetorso).Vestibular:Thevestibularsyst
emisabletosensemotionoftheheadwithrespecttotheworld.Physically,thesystemconsists
oflabyrinthsinthetemporalbonesoftheskull,justbehindandbetweentheears.Thevestibul
arorgansaredividedintothesemicircularcanals(SCCs)andthesacculeandutricle(Figure4
.4).Asafirst-orderapproximation,thevestibularsystemsensesmotionbyactingasathreeaxisrategyroscope(measuringangularvelocity)andathree-axislinearaccelerometer(How
ard,1986).TheSCCssenserotationoftheheadandaremoresensitivetohigh-frequency(quick
lychanging)componentsofmotion(aboveroughly0.1Hz)thantolow-frequency(slowlychangi
ng)components.Becauseoftheseresponsecharacteristics,itisoftennotpossibletodeterm

ineabsoluteorientationfromvestibularcuesalone.Humansusevisualinformationtocomple
mentanddisambiguatevestibularcues.Visual:Visualcuesalonecaninduceasenseofself-mo
tion,whichisknownasvection.Thekindsofvisualprocessingcanbeseparatedintolandmarkr
ecognition(orpiloting),wherethepersoncognitivelyidentifiesobjects(e.g.,chairs,wi
ndows)inhervisualfieldandsodeterminesherlocationandopticalflow.
PinnaAuditoryossiclesCartila
g
eTympanicmembraneExternalauditorycanalVestibularcomplexInternaljugularveinRoundw
indowAuditory tubeTopharyn
x
NerveSemicircular canals
FIGURE4.4Vestibularsystem.Acut-awayillustrationoftheouter,middle,andinnerear.Sou
rce:AdaptedfromMartini(1999).Copyright#1998byFredericH.Martixi,Inc.Reprintedbype
rmissionofPearsonEducation,Inc.
4LocomotionInterfaces112

Opticalflowisalow-levelperceptualphenomenonwhereinthemovementoflightpatternsacro
sstheretinaissensed.Inmostsituations,theoptical-flowfieldcorrespondstothemotionf
ield.Forexample,iftheeyeisrotatinginplace,totheleft,theoptical-flowpatternisalam
inartranslationtotheright.Whenapersonismovingforward,theoptical-flowpatternradia
tesfromacenterofexpansion(Figure4.5).Bothoptical-flowandlandmarkrecognitioncontr
ibutetoapersonssenseofself-motion(Riecke,vanVeen,&Bulthoff,2002;WarrenJr.etal.,20
01).VisualandVestibularSensesAreComplementaryAsmentionedbefore,thevestibularsyst
emismostsensitivetohigh-frequencycomponentsofmotions.Ontheotherhand,thevisualsys
temismostsensitivetolow-frequencycomponentsofmotion.Thevestibularandvisualsystem
sarecom-plementary(Figure4.6).2Thecrossoverfrequencyofthetwosenses(Figure4.7)has
beenreportedtobeabout0.07Hz(Duhetal.,2004).CombiningInformationfromDifferentSens
esintoaCoherentSelf-MotionModelEachsensorymodalityprovidesinformationaboutaparti
cularqualityofaper-sonsmotion.Thesepiecesofinformationarefusedtocreateanoverallse
nseof
FIGURE4.5Threeopticalflowpatterns.Left:Laminartranslationresultfromturningthehea
dleft.Center:Radialexpansionresultingfrommovingforward.Right:Circularflowresulti
ngfromrollingabouttheforwardaxis.2Thishasastrikingsimilaritytothehybridmotiontra
ckersmentionedinSection4.2.1,whichuseaccelerometersandgyrostosensehigh-frequency
componentsofmotionwhilecorrectingforlow-frequencydriftbyusinglow-frequencyoptica
loracousticsensors.
4.1NatureoftheInterface113

Stimulus3.53.02.52.01.51.00.500510152030253540Perceived with visualand inertial

cuesPerceived with visual cues onlyPerceived with inertial cues onlyTime (s)Angu
lar velocity (degrees)
FIGURE4.6Contributionofthevisualandvestibular(orinertial)systemstoperceivingaste
pfunctioninangularvelocity.Thevestibularsystemdetectstheinitial,high-frequencyst
ep,whereasthevisualsystemperceivesthesustained,low-frequencyrotation.Source:Adap

tedfromRolfeandStaples(1988).ReprintedwiththepermissionofCambridgeUniversityPres
s.
1.000.800.600.400.20Normalized relative response0.000.010.050.100.20Frequency-hz
0.400.80
FIGURE4.7Crossoverfrequencyofvisualandvestibularsenses.Visual(solidline)andvesti
bular(dashedline)responsesasafunctionofmotionfrequency.Source:AdaptedfromDuhetal
.(2004);#Dr.DonaldParkerandDr.HennyBeen-LimDuh.
4LocomotionInterfaces114

self-motion.Therearetwochallengesinthisprocess.First,theinformationmustbefusedqu
icklysothatitisuptodateandrelevant(e.g.,thepersonmustknowthatandhowshehastripped
intimetoregainbalanceandfootingbeforehittingtheground).Second,thetotalinformatio
nacrossallthesensorychannelsisoftenincomplete.Onetheorystatesthat,atanygiventime
,apersonhasamodelorhypothesisofhowsheandsurroundingobjectsaremovingthroughthewor
ld.Thismodelisbasedonassumptions(someofwhichareconsciousandcognitive,whileothers
areinnateorhardwired)andprevioussensoryinformation.Newincomingsensorycuesareeval
uatedintermsofthismodel,ratherthannewmodelsbeingcontinuouslyconstructedfromscrat
ch.Themodelisupdatedwhennewinformationarrives.Considerthefamiliarexampleofaperso
nonastoppedtrainthatisbesideanotherstoppedtrain.Whenthetrainontheadjacenttrackst
artstomove,thepersonmighthaveashortsensationthathertrainhasstartedmovinginstead.
Thisbrief,visuallyinducedillusionofself-motion(vection)isconsistentwithallofhers
ensoryinformationthusfar.Whenshelooksouttheothersideofhertrainandnoticesthetrees
arestationary(relativetohertrain),shehasamomentofdisorientationorconfusionandthe
n,inlightofthisnewinformation,shereviseshermotionmodelsuchthathertrainisnowconsi
deredstationary.Inshort,onetendstoperceivewhatoneisexpectingtoperceive.Thisisane
xplanationforwhysomanyvisualillusionswork(Gregory,1970).Anillusionissimplythebra
inswayofmakingsenseofthesensoryinformation;thebrainbuildsamodeloftheworldbasedona
ssumptionsandsensoryinformationthathappentobewrong(Berthoz,2000).Foravisuallocom
otioninterfacetoconvincinglygiveuserstheillusionthattheyaremoving,itshouldreduce
thingsthatmakethe(false)belief(thattheyaremoving)inconsistent.Forexample,asmalld
isplayscreengivesanarrowfieldofviewofthevirtualscene.Theusersseetheedgeofthescre
enandmuchoftherealworld;therealworldisstationaryandthusinconsistentwiththebelief
thattheyaremoving.Alargescreen,ontheotherhand,blocksoutmoreoftherealworld,andmak
estheillusionofself-motionmoreconvincing.Perceptionisanactiveprocess,inseparably
linkedwithaction(Berthoz,2000).Becausesensoryinformationisincomplete,onesmotionmo
deliscon-stantlytestedandrevisedviainteractionwithandfeedbackfromtheworld.Theint
erplayamongcuesprovidesadditionalself-motioninformation.Forexample,ifapersonsees
thescenery(e.g.,sheisstandingonadock,seeingthesideofalargeshiponlyafewfeetaway)s
hifttotheleft,itcouldbebecausesheherselfturnedtoherrightorbecausetheshipactually
startedmovingtoherleft.Ifshehasconcurrentproprioceptivecuesthatherneckandeyesare
turningtotheright,sheismorelikelytoconcludethattheshipisstillandthatthemotioninh
ervisualfieldisduetoheractions.Theactiveprocessofself-motionpercep-tionreliesonp
rediction(ofhowtheincomingsensoryinformationwillchangebecauseofthepersonsactions)
andfeedback.Locomotioninterfacesmustmaintainsuchafeedbackloopwiththeuser.
4.1NatureoftheInterface115

4.1.2LocomotionInterfacesasInteractionLoopsInthecaseofawhole-bodyvirtuallocomoti
oninterface,theinputstotheinter-facearetheposeofoneormorepartsoftheusersbody,andt
heoutputoftheinterfaceisthepointofview(POV)inthevirtualscene.Thechangeineyeposit
ionbetweensamplesisavectorquantitythatincludesbothdirectionandmagnitude.Theprima

rymodeoffeedbackinmostvirtuallocomotionsystemsisvisual:Whatyouseechangeswhenyoum
ove.Inthelocomotionsystem,thechangeinPOVcausesthescenetobe(re)drawnasitisseenfro
mtheneweyeposition,andthentheuserseesthescenefromtheneweyepointandsensesthatsheh
asmoved.Figure4.8showstheinteractionloopthatisrepeatedeachtimetheposeissampledan
dthesceneredrawn.Delvingaleveldeeper,alocomotioninterfacemustdetectwhetheritisth
eusersintenttobemovingornot,and,ifmoving,determinethedirectionandspeedoftheintend
edmotionandinitiateachangeinPOV.Figure4.9showsthisasatwo-stepprocess.First,senso
rsofvarioustypes(describedinmoredetailinthenextsection)areusedtocapturedatasucha
sthepositionandorientationofpartsoftheusersbody,ortocaptureinformationaboutthespe
edand/oracceler-ationofthemotionofpartsoftheusersbody.Thesedataareprocessedtogene
r-atesignalsthatspecifywhethertheuserismoving,thedirectionofthemotion,andthespee
dofthemovement.ThosesignalsareinterpretedandchangedintoaformthatspecifieshowtheP
OVismovedbeforethesceneisnextrendered.Weshowexamplesofthisinthecasestudy(availab
leatwww.beyondthegui.com).
FeedbackIs moving?Which direction?How far/fast?DisplaysystemLocomotionInterfaceS
ensedposeinformationChangein POV
FIGURE4.8Dataabouttheusersbodyposeareinputstotheinterface.Theinterfacecomputesthe
directionanddistanceofthevirtualmotionfromtheinputs.Thedirectionanddistance,intu
rn,specifyhowthePOV(fromwhichthevirtualsceneisdrawn)moves.Bydisplayingthescenefr
omanewPOV,theinterfaceconveys,totheuser,thatshehasmoved.Basedonthisfeedback,she,
inturn,updateshermodelofspaceandplanshernextmove.
4LocomotionInterfaces116

4.2TECHNOLOGYOFTHEINTERFACEWhole-bodylocomotioninterfacesrequiretechnologytosens
euserbodypositionandmovementandtodisplaytofeedbacktheresultsofthelocomotiontothe
user.4.2.1PoseandMotionSensorsSensorsthatmeasureandreportbodymotionsareoftencall
edtrackers.Trackerscanmeasurethepositionandorientationofpartsofthebody,orcanmeas
urebodymovement(e.g.,displacement,rotationalvelocity,oracceleration).Trackerscom
einamyriadofformfactorsandtechnologies.Mosthavesomepiecesthatareattachedtopartso
ftheusersbody(e.g.,thehead,hands,feet,orelbows)andpiecesthatarefixedintheroomorla
boratory.AdetailedoverviewoftrackingtechnologycanbefoundinFoxlin(2002).Commoncat
egoriesoftrackingsystemsaretrackerswithsensorsandbeacons(includingfull-bodytrack
ers)andbeaconlesstrackers.TrackerswithSensorsandBeaconsOneclassoftracker,commonl
yusedinvirtual-realitysystems,hasoneormoresensorswornontheusersbody,andbeaconsfix
edintheroom.Beaconscanbeactive,suchasblinkingLEDs,orpassive,suchasreflectivemark
ers.Trackerswithsensorsontheuserarecalledinside-looking-outbecausethesensorslook
outtothebeacons.ExamplesaretheInterSenseIS-900,the3rdTechHiBall-3100,andthePolhe
musLIBERTY.Untilrecently,body-mountedsensorswereoftenconnectedtothecontrollerand
room-mountedbeaconswithwirescarryingpoweranddata;newmodelsarebatterypoweredandwi
reless.Ininside-looking-outsystems,eachsensorreportsitsposerelativetotheroom.Man
yvirtual-environment(VE)systemsuseasensorontheusershead
SensorsignalprocessingIs moving?MovedirectionMovespeedSensors on or around userI
nterpretsignalsandconvert to POV POV
FIGURE4.9Detailsofvirtual-locomotioninterfaceprocessing.Thesensorsmeasurepositio
n,orientation,velocity,and/oraccelerationofbodyparts.Signalprocessingofthesensor
datadetermineswhethertheuserintendstomoveanddeterminesthedirectionandspeedofheri
ntendedmovement.SpeedanddirectionareinterpretedandconvertedintochangesinthePOV.
4.2TechnologyoftheInterface117

toestablishwhichwaytheheadispointing(whichisapproximatelythewaytheuserislooking)
andasensoronahandhelddevicethathasbuttonsorswitchesthatallowtheusertointeractwit
htheenvironment.InFigure4.3,thesensoronthetopoftheheadisvisible;thesensorfortheh
andcontrollerisinsidethecase.Asensorisrequiredoneachbodypartforwhichposedataaren
eeded.Forexample,ifalocomotioninterfaceneedstoknowwherethefeetare,thenusersmustw
earsensorsorbeaconsontheirfeet.Figure4.10showsawalking-in-place
FIGURE4.10Sensorsonauserofawalking-in-placelocomotioninterface.Thissystemhasahea
dpositionandorientationsensoronthebackoftheheadset,atorsoorientationsensorhangin
gfromtheneck,kneepositionsensors,andpressuresensorsundereachshoe.(Photocourtesyo
ftheDepartmentofComputerScience,UNCatChapelHill.)
4LocomotionInterfaces118

userwithsensorsonherhead,torso,andbelowthekneeofeachleg.Thetrackerontheheaddeter
minestheusersviewdirection;thetorsotrackerdetermineswhichwayshemoveswhenshepushes
thejoystickawayfromher;thetrackersatthekneesestimatethepositionandvelocityoftheh
eels,andthosedataareusedtoestablishhowfasttheusermoves.Sensorbeacontrackerscanbe
builtusinganyoneofseveraltechnologies.Inmagnetictrackers,thebeaconscreatemagneti
cfieldsandthesensorsmeasurethedirectionandstretchofthemagneticfieldstodeterminel
ocationandorienta-tion.Inopticaltrackers,thebeaconsarelightsandthesensorsactasca
meras.Opticaltrackersworkabitlikecelestialnavigationthesensorscanmeasuretheangles
tothreeormorestars(beacons)ofknownposition,andthenthesys-temtriangulatestofindth
epositionofthesensor.Acoustictrackersmeasurepositionwithultrasonicpulsesandmicro
phones.Anotherclassoftrackersiscalledoutside-looking-inbecausethebeaconsareonthe
userandthesensorsarefixedintheroom.Camera-basedtrackersystemssuchastheWorldVizPr
ecisionPositionTracker,theViconMX,andthePhase-SpaceIMPULSEsystemareoutside-looki
ng-insystemsandusecomputervisiontechniquestotrackbeacons(sometimescalledmarkersi
nthiscontext)wornbytheuser.Thistechnologyiscommoninwhole-bodytrackers(alsocalled
motioncapturesystems)wherethegoalistomeasure,overtime,thepathoftensofmar-kers.Fi
gure4.11showsauserwearinghighlyreflectivefoamballsandthreecam-erasontripods.Moti
oncapturesystemscanalsobebuiltusingmagnetic,mechanical(Figure4.12onpage121),ando
thernonopticaltechnologies.Motioncapturesystemsoftenrequiresignificantinfrastruc
tureandindividualcalibration.BeaconlessTrackersSometrackingtechnologiesdonotrely
onbeacons.Some,forexample,determinetheorientationofthebodypartfromtheEarthsmagnet
icfield(likeacompass),ortheEarthsgravitationalfield(alevel-liketiltsensor).Others
areinertialsensorstheymeasurepositionalaccelerationand/orrotationalvelocityusingg
yroscopes.Someinertialtrackingsystemsintegratethemeasuredaccelerationandrotation
datatwicetocomputepositionordisplacement.However,thiscomputationcausesanyerrorsi
ntheaccelerationmeasurement(calleddrift)toresultinrapidlyincreasingerrorsinthepo
sitiondata.Sometrackersystems,suchastheIntersenseIS-900,correctforaccelerometerd
riftbyaugmentingtheinertialmeasurementswithmeasurementsfromopticaloracoustictrac
kers.Suchsystemswithtwo(ormore)trackingtechnologiesarecalledhybridtrackers.Trade
-offsinTrackingTechnologiesEachtypeoftrackerhasadvantagesanddisadvantages.Wiresa
ndcablesareausabilityissue.Manymagnetictrackersaresusceptibletodistortionsinmagneticfieldscausedbymetalobjectsintheroom(e.g.,beams,metalplatesinthefloor,othere
lectronicequipment).Somesystemsrequirerecalibrationwhenever
4.2TechnologyoftheInterface119

themetalobjectsintheroomaremoved.Opticalsystems,ontheotherhand,requirethatthesen
sorshaveadirectlineofsighttothebeacons.Ifauserhasanopticalsensoronherhead(pointi
ngatabeaconintheceiling)andbendsovertopicksomethingoffthefloor,thehead-mountedse

nsormightnotbeabletoseetheceilingbeaconanymoreandthetrackerwillnolongerreportthepo
seofherhead.OtherMotionSensorsInadditiontoposition-trackingmotionsensors,therear
esensorsthatmea-sureother,simplercharacteristicsoftheusersbodymotion.Examplesincl
udepressuresensorsthatreportif,where,orhowhardauserissteppingonthefloor.Thesesen
sorscanbemountedintheshoes(Yan,Allison,&Rushton,2004)orintheflooritself.Thefloor
switchescanbefarapart,asinthematsusedinKonamiCorporationsDanceDanceRevolutionmusi
cvideogame(Figure4.13onpage122)orclosertogethertoprovidehigherpositionalresolu-t
ion(Bouguilaetal.,2004).
FIGURE4.11Opticalmotioncapturesystem.Inthissystem,theuserwearsfoamballsthatrefle
ctinfraredlighttothecamerasmountedonthetripods.Thebrightringsareinfraredlightsou
rces.Camerasareembeddedinthecenteroftherings.(CourtesyofVicon.)
4LocomotionInterfaces120

4.2.2FeedbackDisplaysAsusersspecifyhowtheywanttomove,thesystemmustprovidefeedbac
kviaadisplaytoindicatehowandwheretheyaremoving.Feedbackclosesthelocomo-tioninter
faceloop(Figure4.8).Displayisageneralterm,andcanrefertovisualdisplaysaswellastom
eansofpresentingotherstimulitoothersenses.Manylocomotioninterfacesmakeuseofheadwornvisualdisplays,asinFigure4.3,toprovidevisualfeedback.Theseareoftencalledhead
-mounteddis-plays(HMDs).Usingsuchadisplay,nomatterhowtheuserturnsherbodyorhead,t
hedisplayisalwaysdirectlyinfrontofhereyes.Onthenegativeside,manyHMDsareheavytowe
arandthecablesmayinterferewiththeusersheadmotion.Anothertypeofvisualdisplayhasala
rgeflatprojectionscreenorLCDorplasmapanelinfrontoftheuser.Thesedisplaysareusuall
yofhigherimagequality(e.g.,greaterresolution,highercontrastratio,andsoon)thanHMD
s.However,iftheuserphysicallyturnsawayfromthescreen(perhapstolookaroundinthevirt
ualsceneortochangeherwalkingdirection),shenolongerhasvisualimageryinfrontofher.T
oaddressthisproblem,severalsystemssurroundtheuserwithprojectiondis-plays.Oneexam
pleofthisisaCAVE,whichisessentiallyasmallroomwhereoneormorewalls,thefloor,andsom
etimestheceiling,aredisplaysurfaces(Figure4.14).Mostsurroundscreendisplaysdonotc
ompletelyenclosetheuser,butsomedo.
FIGURE4.12Mechanicalmotioncapturesystem.Thismechanicalfull-bodysystemisproducedb
ySONALOG.(CourtesyofSONALOG.)
4.2TechnologyoftheInterface121

FIGURE4.13KonamiDanceDanceRevolutionsensorpad.Thegameplayerpressesswitcheswithhe
rfeet(andsometimeshands)tocoordinatewithon-screendancesteps.(ImagecourtesyoftheD
epartmentofComputerScience,UNC,ChapelHill.)
FIGURE4.14Surround-screendisplay.Thethree-wall-plus-floorprojectionsystemisseen(
left)withtheprojectorsoffand(right)displayingascenewithavirtualhumanontheleftand
arealhumanuserontheright.(CourtesyofUniversityCollegeLondon,DepartmentofComputer
Science.)
4LocomotionInterfaces122

Inadditiontovisualdisplays,therearedisplaysforothersenses.Asmentionedearlier,peo
plecangetthesensationthattheyaremovingfromauditorycuesalone,withoutanyvisuals(La
ckner,1977).Whilewehaveseenonevirtual-environmentsystemthatusesonlyauditorydispl

ays(fromAuSIM),mostsystemsuseauditorydisplaysasasupplementtotheprimaryvisualdisp
lay.Hearingyourownfootstepsisonecuethathelpsyouremainorientedduringmovement,ands
ubtleechoestellyoumuchaboutthesizeoftheroomyouareinandyourpositioninit.Anotherdi
splayusedinwhole-bodylocomotioninterfacesisamotionplat-formtheuserissittingorstan
dingonaplatformthatisphysicallymovedundercomputercontrol.Someplatformstiltalongo
neormoreaxes,whereasotherstiltandtranslate.Motionplatformsaremostcommonlyusedinf
lightsimulators.ThedeckoftheshipinDisneysPiratesoftheCaribbeanridehasamotionplatf
ormaswellasasurroundscreenvisualdisplay(Schell&Shochet,2001).Themotionplat-formt
iltsthedecksothatthereisamatchbetweenthevirtualwavestheuserseesandtherealrolling
oftheshipsdeck(Figure4.15).
FIGURE4.15DisneysPiratesoftheCaribbeanride.Thisridehassurroundscreens(inthedistan
ce)andamotionplatform(intheshapeofapirateshipsdeck).Playersaimthecannonsandsteert
heshipusingthephysicalprops.(CourtesyofDisney.)
4.2TechnologyoftheInterface123

Finally,amotorizedtreadmill,wherethecomputersystemcontinuallyadjuststhespeedandd
irectionofthemovingbeltinresponsetotheusersinputsandposition,canalsobeconsidereda
sakindofmotiondisplay.TheUniversityofUtahTreadport(Figure4.2)combinesseveralkind
sofdisplaysintoasingleloco-motioninterface:Ithasasurroundscreenvisualdisplay,amo
torizedtreadmill,amotionplatformthattiltstheentiretreadmillupanddown(tosimulated
iffer-entslopes),andaforcedisplaythatpushesandpullsaharnesswornbytheuser(tosimul
atemomentum).4.3CURRENTIMPLEMENTATIONSOFTHEINTERFACEThissectiondiscussesthecurre
ntstateoftheartoflocomotioninterfaces.Foralloftheexamplesdescribed,atleastonepro
totypehasbeenbuilt.Unfortunately,severalofthemnolongerexist(asfarasweknow).4.3.1
FlyingandLeaningThemostcommonlocomotiontechniqueinvirtual-realitysystemsisflying
usingajoystickorsomeotherhandcontroller.Whentheuserpushesajoystickorpressesabutt
on,shemovesforwardinthevirtualscene.Shecanstillmoveaboutlocallybyleaningortaking
arealstepinanydirection,assumingherheadistrackedbytheVEsystem.Theeffectissimilar
tothatofwalkingaboutonamovingflatbedtruckorflyingcarpet(Robinett&Holloway,1992).
Whentheuserpressesthebutton,thetruckmovesforwardinthevirtualsceneandtheusercansi
multaneouslymoveaboutonthetruckbed.Therearesignificantvariationsinhowflyingisimp
lemented.Pushingthejoystickorthebuttonsignalsthattheuserintendstomove.Thedirecti
onofmotioncanbeestablishedinvariousways.InsomeVEsystems,theusermovesinthedirecti
onherhead(ornose)ispointing(headdirected).Otherinterfacesmovetheuserinthedirecti
oninwhichsheispointingwithherhandcontroller.Stillothersinterpretforwardusingaveh
iclemetaphor:Forwardistowardthecenter-frontwallofthemultiwallprojectionsystem.Si
milartoflying,leaningtechniquesmovetheuserinthevirtualsceneinthedirectioninwhich
sheisleaning(LaViolaJr.etal.,2001;Petersonetal.,1998).Mostleaninginterfacesalsoc
ontrolspeedthefarthertheuserleans,thefastershemoves.Leaninghastheadvantageofnotre
quiringahandcontroller,leavingthehandsfree.4.3.2SimulatedWalkingandWalkingLocomo
tioninterfacesthatrequireuserstomakewalkingmotionswiththeirlegsareattractivebeca
useoftheirsimilaritytorealwalking.Twoprimarymethods
4LocomotionInterfaces124

allowapersontofeelheiswalkingbutkeephiminarestrictedspace:treadmillsandwalking-i
n-place.TreadmillsandTreadmill-LikeDevicesThereareanumberoftechniquesthatsimulat
ethephysicalactofwalkingwithtreadmills(Brooks1998;Hollerbachetal.,2000).Asusersw
alkforwardonthetreadmill,theymoveforwardinthevirtualscene.Usingapassivetreadmill
(i.e.,withoutamotor)requiresaphysicaleffortbeyondthatofwalkingintherealworldbeca
useofthemassandfrictionofthebeltandrollers.Motorizedtreadmillsaddressthisproblem
andstillallowtheusertochangespeed.Motorizedtreadmillssensetheuserspositiononthebe

ltandvarythebeltmotorspeedtokeeptheusercentered.Thedisadvantageofmotorizedtreadm
illsisthenoiseandpotentialforinjurymanyvirtual-locomotiontreadmillshavetheuserswe
arasafetytethertopreventthemfromfalling.Allofthetreadmillsdiscussedthusfarhaveth
elim-itationthatthetreadmillhasapreferredorientation;itisdifficult,disorienting,
andoftenimpossibletoturnonthespotinthevirtualscene.TheUniversityofNorthCarolinast
readmill(Brooks,1988),forexample,hadhandlebarstosteerlikeabicycle.Toallowturning
onthespot,severalgroupshavedeveloped2Dtreadmills(Darken,Cockayne,&Carmein,1997;I
wata,1999)oromnidirectionaltreadmills(VirtualDevices)wheretheusercanwalkinanydir
ectiononthegroundplane.AscanbeseeninFigure4.16,thesedevicesarelarge,noisy,andmec
hanicallycomplex.Darkenandcolleagues(1997)commentedabouttheomnidirectionaltreadm
illthatitwashardtochangedirectionswhilerunningbecauseitfeltlikeslipperyice.Another
approachto2Dtreadmill-likelocomotionishavingtheusercontainedinsideaverylargesphe
re(akintoahuman-sizedhamsterball)thatisheldinplacewhileitrotates.Thereareatleasttw
oimplementationsofthisideatheVirtuSphereandtheCyberSphere.TheVirtuSphere(Figure4.
17)ispassive,whereastheCyberSphereismotorized.Asofthiswriting,severalVirtu-Spher
eshavebeeninstalled,butweareunawareofanyworkingCyberSphereshavingbeenbuilt.Walki
ngInterfacesWhenusingawalking-in-place(WIP)interface,theusermakeswalkingorstep-p
ingmotions,thatis,liftingthelegs,butstaysonthesamespotphysically.Thelocomotionin
terfacedetectsthismotionandmovesherforwardinthevirtualscene(Slater,Usoh,&Steed,1
995;Templeman,Denbrook,&Sibert,1999;Usohetal.,1999).Likeflying,WIPdoesnotrequire
alargelabortrackingspace.Likeatreadmill,theusermakesmotionsthataresimilartorealw
alking.Virtual-realitysystemuserswhowalk-in-placehaveagreatersenseofpresencethan
thosewhoflywithajoystickorhandcontroller(Slateretal.,1995;Usohetal.,
4.3CurrentImplementationsoftheInterface125

FIGURE4.162Dtreadmills.Left:TheTorustreadmill,whichconsistsofmanysmallone-dimens
ionaltreadmillsconnectedtogethersuchthattheybecomethelinksofagianttreadmill.Cent
er:Whenthesmalltreadmillsarerotatingatthesamespeedandthelargetreadmillisstopped,
theuserwalksinthedirectionofthesmalltreadmills.Right:Whenthesmalltreadmillsarest
oppedbutthelargeoneismoving,theuserwalksinthedirectionofthelargetreadmill.Source
:FromIwata(1999);#NivooIwata.
FIGURE4.17TransparentversionoftheVirtuSphere(standardversionsaremadeofamesh).The
usersarewearingwirelessheadsetsthatprovidevisualfeedbackwhilemoving.(CourtesyofV
irtuSphere,Inc.)
4LocomotionInterfaces126

1999).Walking-in-placeisthesubjectofthischapterscasestudyandisexploredingreaterde
tailthere.Ifthevirtualsceneisthesamesizeasorsmallerthanthetrackedspace,thenrealw
alkingisfeasible.Heretheusersmovementinthevirtualscenecorre-spondsexactlytohermov
ementinthelab:Ifshewalks5minthelab,shealsowalks5minthevirtualscene.Whenvirtualsp
acesgrowverylarge,itissometimenecessarytoaugmentareal-walkinginterfacewithamagic
alinterface.4.3.3MagicalandUnnaturalInterfacesWhenthevirtualsceneisverylargeandt
hetaskrequirestheusertovisitfarawayplaces,mundaneinterfacesmaybeaugmentedwithmag
icalinterfacetechniquesthatallowtheusertoquickly,oreveninstantaneously,moveoverl
ongdistancesortoacompletelydifferentscene.Currenttechniquesincludeteleportationt
oandfromspeciallocationswithinthescene,goingthroughspecialportalsthattaketheuser
toanewscene,oraspeciallocomotionmodethatswitchestheuserspacefromanormalwalkingspe
edtoahyperdrivemode.Aconcernwithmagicallocomotiontechniquesisthattheusermaybecom
edisorientedwhenthescenechangesrapidly.Alertingtheuserthatsomethingspecialisgoin
gtohappenforinstance,byrequiringhertogotoaspecialplaceorthroughadoorwaypreparesher
foranabruptchangeoflocation.Graduallyspeedingupandslowingdownatthestartandendofa

high-speedmoveislessdisturbingtousersthanabruptspeedchanges.Alovelyexampleofcont
ext-preservingmovementovergreatdistanceistheGoogleEarthmappingservice:Theviewpoi
ntzoomsawayuntilthewholeEarthisvisible,theEarththenrotatesuntilthenewtargetlocat
ioniscentered,andthentheviewpointzoomsbackin.TheDigitalArtFormsSmartSceneVisuali
zerproduct3andMine(1997)havedemonstratedtechniqueswheretheusercangrabthevirtualsc
eneandmoveittowardhimself.Byrepeatedlygrabbingpointsinthevirtualsceneandpullingt
hemin,theusercanlocomotefromoneplacetoanother.Evenlessliketherealworld(butnoless
fun),StoakleysWorlds-in-Miniaturetechniquehastheusermanipulateahandheld,dollhouses
izedmodelofthevirtualscene.Theusermovesinthevirtualscenebymovingadoll(representi
ngherself)tothedesiredlocationintheminiaturevirtualscene.Thentheminiaturevirtual
scenegrowstobecomehumanscaleandtheuserfindsherselfinthedesirednewlocationinthevi
rtualscene(Stoakley,Conway,&Pausch,1995).Figure4.18illustrateswhattheuserseeswit
htheWorlds-in-Miniaturetechnique.3SmartScenewasoriginallydevelopedbyMultiGen.
4.3CurrentImplementationsoftheInterface127

4.4HUMANFACTORSOFTHEINTERFACESection4.1.1showedthathumansrelyonacomplexcombinati
onofsensoryinputtoplanandcontroltheirmovement.Thesensorycomplexityistherewhether
themotioniscontrollingarealoravirtualvehicleorcontrollingrealorvirtualmove-mento
nfoot.Inthissection,weagainfocusonlocomotiononfoot,andwilldis-cussfactorsthatmak
etheinterfacenatural(oratleastintuitive)touseandelementsofsystemdesignthatminimi
zetheriskofsimulatorsickness.4.4.1MakingtheInterfaceFeelNaturalWearealongwayfrom
beingabletoplacesensorsonusersthatevenapproxi-matethenumber,variety,andsensitivi
tyofthesensoryinputsthatcontrolnatu-ralhumanmovement.Thesensorinputstothelocomot
ioninterfaceareimpoverishedwhencomparedtotheinputsthatcontrolrealmovement.Thejob
FIGURE4.18Worlds-in-Miniatureinterfaceasseenbytheuser.Thesmallroomwiththechecker
boardfloor(intheforeground)isheldintheusershand(invisiblehere),andisidenticaltoth
efull-scaleroominwhichtheuserisstanding(background).Tomoveherselfinthefull-scale
room,shemovesatinydoll(notpictured)intheminiaturehandheldroom.Source:Adaptedfrom
Stoakleyetal.(1995);#RandyPausch.
4LocomotionInterfaces128

oftheinterfacedesigner,then,istomaketrade-offstomaximizethecorrespondenceofvirtu
allocomotiontoreallocomotionintermsofthemovementstheusermakesandthemovementstheu
serperceivesthroughfeedbacktohervarioussenses.Ifyouaredesigningalocomotioninterf
ace,youhavetodecidewhatmotionbywhichpartoftheusersbodywillgenerateaparticulartype
oflocomotion.Fac-torsinthischoicearenotonlyhowlikenaturalmotionitis,butalsowheth
erthismotionwillstimulateothersensesinparticularthevestibularsysteminthesamewaytha
tnaturalmotionwould.First,astheinterfacedesigner,youhavetochooseoneormoretracker
sys-tems,decidewhatpartsofthebodytotrack,anddeterminehowtointerpretthosetrackerr
eadings.Naturalforwardmotionisrelativelyeasytoaccomplishinawalking-in-placeinter
facewithonlyafewsensorinputsignals,suchasiftrackersonthekneesshowthefeetaregoing
upanddown,thenmoveforward.Ifthedesigncriteriarequirethattheuserbeabletomovesidew
aysandbackwardwhilelookingforward,itbecomesmuchmoredifficulttodefineanaturalmoti
onthatcanbedetectedfromtheoutputofonlytwosensors.TheGaitersystemenabledsidewaysa
ndbackwardmotionbyhavingtheusermakedistinctivegestureswithherlegs(Templemanetal.
,1999).Theuserswingsherlegsidewaysfromthehiptomovesidewaysandkicksherheelsupbehi
ndherwhensteppingtomovebackward.Infact,userscanmoveinanydirectionbyextendingthei
rlegsinthedesireddirec-tion.Whilethesemovementsarenotthewaywenormallymovesideway
sorback-ward,thegesturesaresomewhatconsistentwiththeintendeddirectionofmotion.Wa
lking-in-placeprovidesmanyofthesamesensorycuesasreallywalking,exceptthattheusere
xperiences,andsenses,noneofthephysicalforcesasso-ciatedwithforwardmomentum.Thepa

rtoftheinterfacemostcriticalformakingfeedbacknaturalarethealgorithmsexecutedinth
efunctionboxinFigure4.9labeledInterpretsignalsandconverttoDPOV.Todate,almostalloft
hesealgo-rithmshavebeenverysimpledetectastep,moveforwardastep.Afewinter-faceshave
beguntoimprovethenaturalnessofthevisualfeedbackwithamorecomplexmodelthatreflects
,frametoframe,thesubtlevariationsinvelocitythatoccurinnaturalhumanwalking.4.4.2H
igher-FidelityReproductionofHumanGaitWhenwewalkthroughtherealworld,thepositionof
ourviewpointchangescontinuously,butnotataconstantspeed.Ourspeedgoesthroughcycles
ofaccel-erationanddecelerationaswestep.ThegraphinFigure4.19(right)showshowourspe
edvaries:Thereisadominantpatternofspeedingupaswestartfollowedbyarhythmicphaseofn
earconstantspeedthatlastsuntilweslowdowntostop.Thesmallerperturbationsinspeedare
thesmallaccelerationsanddecelerationsthatoccuraswetakesteps(Inman,1981).Ideally,
ifwemeasuredandplottedthespeedforusersofourvirtual-locomotioninterface,itwouldlo
okmuchlikethisfigure.Theidealisdifficulttoachieve,butourapproximationsaregetting
better.
4.4HumanFactorsoftheInterface129

Wehaveexperimentedwiththreemethodsofallocatingmovementtothechangeineyepositionpe
rframe:Inserttheentiresteplengthinoneframe,dividethesteplengthintouniformsegment
sandinsertthepartsoverseveralframes,anddividethesteplengthintononuniformsegments
andinserttheseg-mentsoverseveralframes(Figure4.19,left).Ininformaltesting,usersp
referredthelastmethod.Notethateventhelastwaveformonlycrudelyapproximatesthepatte
rnoftherhythmicphaseofrealhumanwalking.4.4.3AvoidingSimulatorSicknessLocomotioni
nterfaces,justlikeflightsimulators,havetheriskofmakingtheusersick.Afterexposuret
ovirtualenvironments,somepercentageofusersreportsomesicknesssymptoms:drymouth,na
usea,dizziness,visualaftereffects(flashbacks),pallor,sweating,ataxia(lossofbalan
ce),andevenvomiting.VEpractitionerscom-monlyrefertothisphenomenonassimulatorsick
ness,orcybersickness.Theoccur-renceofthesesymptomsvarieswidelyfrompersontoperson
andamongVEsystems.Twentytofortypercentofmilitarypilotssufferfromsimulatorsicknes
s,dependingonthesimulator(Kolasinski,1995).
VelocityTime (frames)(a)
(c)Time (frames)Velocity
(b)Time (frames)Velocity
TimeAverage speed duringrhythmic phaseSpeedDevelopmentphaseRhythmicphaseDecaypha
se
FIGURE4.19Walkingviewpointandstart-upspeedvariations.Left:Threemethodsofmovingth
eviewpointoverthecourseofafewvirtualsteps.(a)Allchangeineyepositionismadeatasing
leinstantuserappearstosnapforwardwitheachstep.Thismakesthemotionseemjerky.(b)Thec
hangeisdistributeduniformlyoverseveralframes(afewhundredmsec).Usersfoundtheglide
forward/stop/glideforward/stopnatureofthistechniqueverydisturbing.(c)Changeisdis
tributedoverseveralframes,withtheamountdecreasingovertime.Velocityishighestimmed
iatelyafterthefootstrike,andthenitdecays.Right:Speedvariationsasapersonstartswal
king,walksatasteadypace,andstops.Notethesmallvariationsinspeedthatareassociatedw
itheachstep,evenduringtherhythmicphase.Source:AdaptedfromInmanetal.(1981).
4LocomotionInterfaces130

ConsequencesofSimulatorSicknessSometimessimulatorsicknesssymptomslingerfordaysaf
terusingalocomotioninterface,affectmotorcontrolandcoordination(Draper,1998),andc

animpairapersonsperformanceintherealworld(e.g.,whiledrivingacarorflyingaplane).Ke
nnedyandLilienthal(1995)andothers(Cobb&Nichols,1998)haveproposedquantifyingthese
verityofsimulatorsicknessasamapping(usingmeasuresofapilotsabilitytokeepbalance)to
bloodalcohollevelthelegalmetricofalcoholintoxicationusedtodetermineifapersonisfit
todrive.FactorsThatAggravateSimulatorSicknessItisusefultoconsiderhowcharacterist
icsoftheuserandofthesystemaffectthelevelofsicknesssuffered.Thefactorscanbedivide
dintothoseoftheindividualuserandthoseofthesystemorsimulator.Qualitiesthatcorrela
tewithdecreasedsusceptibilitytosimulatorsicknessincludebeingingoodhealth(hangove
rs,med-ications,stress,illness,andfatigueallincreasesickness)andpreviousexperien
cewithsimulatorsorlocomotion.Table4.2liststhecharacteristicsofsystemsdividedinto
twocategories.Thecharacteristicsintheupperlistaretechnicalshortcomingsoftheequip
ment.Astechnologiesimprove,oneexpectstheseshortcomingstobereducedand
TABLE4.2QualitiesofVESystemsandFlightSimulatorsThatIncreaseSimulatorSicknessEqui
pmentShortcomingsFTrackerinaccuracies(temporalandspatial)FLowupdaterateFHighlate
ncy/lagFMismatchesbetweendisplayparametersandimagegeneratorsparameters(e.g.,incor
rectFOVsetting)FDisplayflickerFHeadsetweightDesirableFunctionsofSystemFStereodis
playFLongexposuredurationFWideFOVdisplayFFreeheadmovementFViewpointmotioncontrol
ledbysomeoneotherthantheviewerFHighratesofsimulatedlinearorrotationalacceleratio
nSource:AdaptedfromKolasinski(1995).
4.4HumanFactorsoftheInterface131

thusdecreasetheresultingsimulatorsickness.Thecharacteristicsinthelowerlistarequa
litiesthatareoftendesiredinsystems.Longerexposuresmightberequiredtosimulatelongm
issionsortoallowuserstocarryoutmeaningfultasks.Stereoscopicdisplayscanimprovetas
kperformance(Pang,Lim,&Quek,2000)butincreasesimulatorsickness,andhigherfield-ofview(FOV)displaysresultinbettertrainingandhigherlevelsofuserimmersion(Arthur,200
0).Therearemanytheoriesastowhatcausessimulatorsickness;seeLaViolaJr.(2000)forabr
iefintroductiontoseveraltheories,andRazzaque(2005)formorein-depthdiscussion.From
apracticalpointofview,ouradvicetothedesigneristobeonthelookoutforsymptomsofsimul
atorsickness(particularlynauseaanddizziness)inyourtestusers,andthentoinvestigate
moredeeplyifyoususpectaproblem.4.5TECHNIQUESFORTESTINGTHEINTERFACEWhyaretheresom
anydifferentlocomotioninterfaces?Eachmusthaveitsownadvantagesandshortcomings.How
ever,itisverydifficulttomakequantitativecomparisonsamongthem.Evenwhentheresearch
ersdoconductuserstudiestocomparetechniques,theresultsareoftennotwidelycomparable
becausetheystudydifferentuserpopulationsanddifferenttasks,andevaluateusingdiffer
entmetrics.Furthermore,nonlocomotionparametersofthesystems,suchasframerate,field
ofview,latency,andimagequality,aredifferentand,inmanycases,notmeasured.Thesefact
orscanconfoundanystudyresults,particularlytaskperformancemeasures.Oneparticularl
yimpressivecomparisonoflocomotiontechniquesisastudybyBowmanandcoworkers(1997).Th
eyevaluatedseveralvariationsofflyingbyreimplementingeachtechniquetorunonthesameh
ardwareandhavingeachsubjectperformthesametaskusingalltheflyingtechniques.Theseau
thorseval-uatedeachtechniqueusingseveralcriteria:easeoflearning,spatialawareness
,speed,accuracy,andcognitiveload,andpointedoutthatdifferentapplicationshavediffe
rentneeds.Aphobiatreatmentapplicationmaybemoreconcernedwithevaluatingnaturalness
andpresence,whereasagamemaybemoreconcernedwithspeedandagilitysothattheusercanget
tothetargetquickly.Astheapplica-tiondesigner,youshouldfirstdecidewhichattributes
areimportant,choosealoco-motiontechniquethatoptimizestheparticularattributesimpo
rtantforthatapplication,andthenevaluatethoseattributes.4.5.1GuidelinesforTesting
Onenavewaytotestaninterfacemightbetoassignatasktoasampleofusers,andthenmeasurehowq
uicklytheyperformthistask.Butfastertaskperformance
4LocomotionInterfaces132

doesnotalwaysimplyabetterinterface.Forexample,imagineasimulatorintendedtohelpapa
tientgetoverafearofheights,whereintheuserstaskistocrossariverusinganarrowropeandp
lankbridge.Oneinterfacedesignmightrequiretheusertoholdhandrails,andlookdownandsl
owlystepfromplanktoplank,whereasanotherinterfacedesignmighthavetheuserpointherha
ndattheoppositeendofthebridgeinordertoinstantlyteleportthere.Eventhoughthelatter
hasherattheothersideofthebridgesooner,itiscontrarytotheobjec-tiveoftheinterfaceto
exposethepatienttothestimulusthatevokesthefearofheightsresponse.Standardpractice
inusabilitytestingdictatesthatthedesignerusetesttasksthatareappropriatetothegoal
oftheinterface,andthatthetestingmustincluderealuserswhocarryouttheentiretaskfore
ntiresessions.Tasksthatseemeasyinshortsessionscanbecomefatiguingorevenundoablein
longersessions.Manyofthespecifictestsdescribedinthenextsectionresultinquantitati
vedata,butareexpensivetoconduct.Whilequantitativetestswithalargenumberofusersres
ultsinmoreconvincingstudyresults,doingmoreinformaltestsand/orqualitativetestscan
resultinmoreinsight,andcansavetimeandresources,allowingformoreiterationsofdesign
ingandtesting.Inthisregard,thedesignandtestingoflocomotioninterfacesisnodifferen
tthanforanyotherkindofuserinterface.4.5.2QualitiestoConsiderforTestingandSpecifi
cTestsInthissection,weenumeratequalitiesofawhole-bodylocomotioninterfacethatyoum
ightwanttotest,andprovidespecificmethodsfortestingeach.DistractionThefirsttestin
gcategoryissimplyeffectiveness:Cantheusermovearoundsothatshecanperformhertasks,o
rdoestheinterfaceinterferewithordistractherfromthetasks?Distractionsmaybecogniti
vetheuserhastopayattentiontousingtheinterfaceratherthantodoinghertask,ortheycanbe
physicaltheinterfacedeviceoritscablesencumbertheuserinwaysthatinterferewithtaskpe
rformance.Oneexampleofacognitivedistractionisaninterfacewithpoormotioncon-trol.E
videnceofpoorcontrol(inadditiontouserssimplycomplainingaboutit)includestheuserov
ershootingherintendedtargetorunintentionallycollidingwithobjects,walls,ortheedge
sofdoorways.Suchalocomotioninterfaceisnottransparenttotheuser,butconstantlyrequi
restheusersattention,reducingthecognitiveresourcesavailableforherprimarytask.Then
umberofovershoots(wheretheusergoespastthetargetandmustreverseorturnaround),the
4.5TechniquesforTestingtheInterface133

distanceofeachovershoot,andthenumberofunintendedcollisionscanberecordedandcounte
dduringtesting.Whentheuserperformsthatsametaskindifferentversionsoftheinterface,
thesecountscanbeusedtocomparethedifferentversions.Asanexampleofphysicaldistracti
on,considerthecaseofasoldierorgameplayerwhoissupposedtowieldagunwhilemovingthrou
ghavirtualbuilding,butmustusebothhandstocontrolthelocomotioninterface.Inthissitu
ation,theusermaywellnotbeableaimorshootherguneffectivelywhilemoving.Onewaytotest
forthiskindofphysicaldistractionistomeasuretheusersperfor-manceonthenonlocomotion
partofhertaskwhileusingdifferentversionsofthelocomotioninterface,andthentoattrib
utethedifferencesinperformancetothedistractioncausedbythelocomotioninterfaceitse
lf.TrainingTransferSomesystemsthatincludelocomotioninterfacesareintendedtotrainu
sersforsometask,suchasevacuatingabuildingduringafire.Onewayoftestingtheeffective
nessofsuchasystemisbymeasuringtrainingtransferhowwelldoesittraintheuser?Thisisdon
ebyusingthesystem(oraprototype)totrainagroupofusers,andthenmeasuringtheirperform
anceontherealtaskafterward.Thistaskperformanceiscomparedagainsttheperformanceofo
theruserswhoweretrainedusingsomeothersystem(orapreviousversion).Asanexample,Insk
o(2001)testedasystem(designedtotrainuserstolearnavirtualmaze)toseeiflettingtheus
ersfeelthevirtualwalls(viahapticfeedback)astheymovedthroughthevirtualmazewouldre
sultintheirhavingbetterlearnedthemaze.Itturnedoutthatthoseuserswhousedtheinterfa
cewithhap-ticsmadefewererrorsintherealmaze.Thespecificmethodstotesttrainingtrans
fer,ofcourse,dependonthespecifictasksthattheinterfaceisintendedtotrain.PresenceI
nthecontextofavirtualscene,presenceisthefeeling,intheuser,ofbeinginthevirtualsce
ne,asopposedtobeingintheroomwheresheisphysicallystand-ing.Forapplicationssuchasp
hobiadesensitization,presenceisthemostimpor-tantqualityofasystem,andthereforemer
itstesting.Onewayoftestingpresenceistouseaquestionnaire,suchastheSlater-Usoh-Ste
edquestionnaire(Slater&Usoh,1993;Slateretal.,1995;Usohetal.,1999).Anotherinvolve

scountingbreak-in-presence(BIP)events(Slater&Steed,2000).BIPsaredefinedaseventsw
heretheillusionofbeingsomewhereelseisbrokenandtheusersattentionisbroughtbacktoher
physicalsurroundings.OneexampleofaBIPeventiswhentheusersvirtualbodypenetratesorin
tersectsavirtualobject(Figure4.20).Theusercanpushabuttonorcallouttotheexperiment
ereachtimesheexperienceaBIPevent,andthenthecountscanbecomparedagainstthecountsfr
omusersinotherinterfaces.Usingthe
4LocomotionInterfaces134

BIPmeasures,Slateretal.(1995)determinedthatthewalking-in-placeinterfaceresultsin
greaterpresencethanlocomotionviaflyingwithhandcontroller.Wehavealsoexploredphysi
ologicalindicatorsofpresence,suchasheartrateandskinconductance.Thegeneralideaist
oplacetheuserinasimulatedstress-fulsituationandrecordphysiologicaldata.Themoresi
milarherphysicalresponsesinthevirtualscenearetoresponsestoacorrespondingstressfu
lsitua-tionintherealworld,themorepresenceinducingthesimulatoris.Physiologicalmea
sureshavebeenusedtotesttheeffectsofinterfacedesignparameterssuchasthedisplaysfram
erateandlag(Meehanetal.,2003,2005).Oneproblemwithallthreewaysoftestingpresenceme
ntionedinthissectionisthatthereislargevariationbetweenindividualsandfromsessiont
osession.Whileyoumighthaveaqualitativefeelforwhichlocomotioninterfaceisbetteraft
ersam-plinghalfadozenusers,severaldozentohundredsofusersmightneedtobetestedtoget
astatisticallysignificantquantitativeindicationfromphysiologicaldata.4.5.3Locomo
tionRealismandPreservationofSpatialUnderstandingAnothertestablequalityofawhole-b
odylocomotioninterfaceishowrealisticthemotiondisplayedtotheuseris.Rememberthatdi
splayisagenericterm
FIGURE4.20Exampleofaneventthatcausesabreakinpresence.Unnaturalevents,suchaswalki
ngthroughawall,breaktheusersillusionthatsheisinthevirtualscene,andremindherthatshei
sreallyinalaboratory.(CourtesyoftheDepartmentofComputerScience,UNCatChapelHill.)
4.5TechniquesforTestingtheInterface135

andmayrefertoavisualdisplay,anaudiodisplay,oramotiondisplay(motionplatformortrea
dmill).Whenauseriswalkingonatreadmill,doesvisualdis-playmatchhowfastshefeelsshei
swalking?Withawalk-in-placeinterface,whentheusertakesavirtualstep,doesthesteplen
gthfeelconsistentwithherexpectation?Someresearchershavefoundthatwhenauserisphysi
callypushedinacartatsomeacceleration,andthevirtualviewpointmovesatthesameacceler
ationinthevirtualscene,theuseroftenfeelsthatthevirtualaccel-eration(seen)andther
ealacceleration(felt)donotmatchup(Harris,Jenkin,&Zikovitz,1999).Onetheoryisthatt
hemismatchcouldbeduetothenarrowFOVinsystemsthathavetheuserwearanHMD.Whateverther
easons,youshouldqueryyourtestusersastowhetherthedistance,velocity,and/oraccel-er
ationofthedisplayedmovementfeelstoofastortooslow,andadjustitaccordingly.Youcanal
sotesthowwellthelocomotioninterfacepreservestheusersspatialawareness.Petersonandc
olleagues(1998)hadusersmovethroughavir-tualmazeandthenhadthempointtowardthelocat
ioninthevirtualmazefromwhichtheystarted.Hecomparedthepointingaccuracyoftwodiffer
entloco-motioninterfaces.IwataandYoshida(1999)comparedtheshapeofthepaththatusers
tookwhenwalkingintherealworldandwhenusingtheTorustreadmill.4.5.4SimulatorSicknes
sWhendesigningawhole-bodylocomotioninterface,youshouldpaycarefulatten-tiontothee
xtenttheinterfaceinducessimulatorsicknessinusers,anddecidewhetheracertainamounto
fsicknessisacceptableforaparticularapplication.Themostcommonwayofmeasuringsimula
torsicknessisKennedyssimulatorsicknessquestionnaire(SSQ)(Kennedyetal.,2003).TheSS
Qconsistsof16multiple-choiceitems,andresultsinanoverallsick-nessscoreaswellasthr
eesubscores.Itcanbeadministeredtoatestuserveryquickly,butbecausethescoresvarygre
atlyfrompersontoperson,largenumbers(manydozens)oftestusersmustbeused.Also,theexp
erimentermustnotadministertheSSQtotestusersbeforetheyusethelocomotioninterface,a

sthereisevidencethatusingtheSSQinapre-exposure/postexposurefashioncausesuserstor
eportmoresicknessinthepostexposuretestthanuserswhohavethesamevirtuallocomotionex
perience,buttakethetestonlyafterexposure(Young,Adelstein,&Ellis,2006).TheSSQwasd
evelopedfromananalysisofoverathou-sandflightsimulatorsessionsbyU.S.NavyandMarine
pilots.Inourexperiencewithwhole-bodylocomotioninterfaces,ratherthancomparingtheS
SQscoresoftestuserstothesicknesslevelsreportedbymilitarypilots,itismoreusefultoc
ompareSSQscoresbetweendifferentlocomotioninterfaces(ordifferentversionsofthesame
interface).
4LocomotionInterfaces136

Researchershaveproposedotherwaysofmeasuringsimulatorsickness,suchaslookingatthec
hangeinausersposturalstabilityherabilitytobalancewhenmeasuredbeforeandafterusingthe
locomotioninterface(Stoffregenetal.,2000).Finally,whenyoudesignaninterface,yoush
ouldalsoconsidertestingthequalitiesofthelocomotionsystemthatmightleadtoincreases
insimulatorsickness.Forexample,thedisplayframerateandend-to-endsystemlatency(Mee
hanetal.,2003)canbemeasuredwithouthavingtorunuserstudies.Wefindthataframerateof6
0framespersecondorgreaterandanend-to-endlatencyoflessthan100millisecondsaregener
allyacceptable.Valueshigherthantheseshouldcauseyoutolookmorecarefullyforsymptoms
ofsicknessinusers.4.6DESIGNGUIDELINESGeneralinterfacedesignrulesapplytolocomotio
ninterfacesaswell:1.Alwaysconsidertheusersgoalsfortheinterfaceasthehighestpriorit
y.2.Performmanyiterationsofthedesign!test!revisecycle.3.Alwaysincludetestswithac
tualusersandtheactualtasksthoseuserswillperform.Wealsosuggestguidelinesthataresp
ecifictowhole-bodylocomotioninterfaces.4.6.1MatchtheLocomotionMetaphortotheGoals
fortheInterfaceYoushouldalwaysconsiderwhetherthelocomotionmetaphorsuitsthegoalof
thewhole-bodylocomotioninterface.Forexample,iftheinterfacesgoalistosim-ulaterealw
alking,thentheinterfaceshouldrequiretheusertoreallyturnherbodytoturn(walkingmeta
phor),ratherthanturningbymanipulatingasteeringwheelorhandcontroller(vehiclemetap
hor).4.6.2MatchtheInterfaceCapabilitiestotheTaskRequirementsBecausetheuseremploy
stheinterfacetospecifylocomotion,youmustconsiderhowfasttheuserwillneedtomove,how
finelyoraccuratelyshemustbeabletocontrolherlocomotion(e.g.,willsheneedtowalkthro
ughtightspaces?),andifshewillbeperformingothertaskswhilesheismoving.Notbeingable
tomove
4.6DesignGuidelines137

fastenoughwillresultintheuserfindingtheinterfacetedious,whereasnotprovidingfinee
noughcontrolwillresultinunintendedcollisionsandinabilitytostopexactlywheredesire
d.4.6.3ConsiderSupplementingVisualMotionCuesUsingOtherSensesThevisualdisplayalon
emaynotalwayscommunicatesomekindsofmovement.Forexample,iftheuserofagameishitwith
bullets,thequickandtransientshockmovementwillbeimperceptibleinavisualdisplaybeca
usethevisualsystemisnotsensitivetothosekindsofmotions.Anauditorydisplayandmotion
plat-form(e.g.,ashaker)wouldworkbetter.Ontheotherhand,amotionplatformwouldnotbea
ppropriateforconveyingveryslowandsmoothmotionsuchascanoeingonastillpond.4.6.4Con
siderUserSafetySincetheuserwillbephysicallymoving,considerhersafety.Onethingyoum
ustalwaysconsiderasadesigneristhesetofcablesthatmightconnecttoequipmentwornbythe
user.Willthecablesunintentionallyrestricthermovementorentan-gleherorcausehertofa
ll?Whatpreventsherfromfalling?Andifshedoesfall,doestheequipmentsheiswearingcause
heradditionalinjury?Howwillthecablesbemanaged?4.6.5ConsiderHowLongandHowOftenthe
InterfaceWillBeUsedAsadesigneryoumustalsoconsiderthelengthoftimethepersonwillbeu
singtheinterfaceandhowphysicallyfatiguingandstressfulthemotionsshemustmakeare.Fo
rexample,iftheinterfacerequireshertoholdherarmsoutinfrontofher,shewillquicklytir
e.Iftheinterfacerequireshertoslamherfeetintotheground,repeatedusemightaccelerate
kneeandjointinjury.Asdiscussedabove,youshouldalsobeonthelookoutforsymptomsofsimu

latorsickness,andifyoususpectthatyourlocomotioninterfaceiscausingsomeuserstobeco
mesick,investigatewaysofchangingtheinterface(e.g.,reducingtheexposuretime,reduci
ngthedisplaysFOV)toaddressit.Forexample,theIMAX3Dversionofthemovie,HarryPotterand
theOrderofthePhoenix,limitedthe3Dvisualstoonlythefinal20minutesofthemovie.Theapp
licationsofwhole-bodylocomotioninterfacesaresovariedthatthereareveryfewthingstha
tadesignshouldneverdo,nomatterwhatthecircumstanceorapplication.
4LocomotionInterfaces138

4.6.6DoNotAssumethatMoreSophisticatedTechnology(HigherTech)IsBetterOneassumption
thatwehaveseennoviceinterfacedesignersmakethatoftenresultsinabadinterfacedesigni
sthataninterfaceisbettersimplybecauseitishighertech.Youmustconsiderthegoalsoftheus
erandmakedesignchoicesbasedonthat.4.6.7DoNotDeployHead-DirectedInterfacesAlthoug
heasytoimplement,alocomotioninterfacethatforcestheusertomoveinthedirectioninwhic
hherheadispointing(or,inversely,forceshertopointherheadinthedirectionshewishesto
move)isalmostalwayssuboptimal.Onfoot,humanslookinthedirectiontheyaremovingroughl
y75percentofthetime(Hollands,Patla,&Vickers,2002);about25percentofthetimewearemo
vinginonedirectionandlookinginanother.Forexample,peoplelookaroundwhilecrossingas
treetormergingintoalaneoftraffic.Ifyoucanonlymoveinthedirectionyouarefacing,youc
annotlookaroundandmoveatthesametime.Templemanetal.(2007)provideadetailedanalysis
ofwhymotiondirectionandheaddirectionmustbedecouplediflocomotionsystemsfortrainin
gsoldiersonfootaretobeeffective.Manydesigners(includingtheauthors)whofirstimplementedahead-directedlocomotioninterfacethendiscoveredthisproblemandhadtoredevelo
ptheinterface.Onthispoint,werecommendthatyoulearnfromtheexperienceofothersandavo
idhead-directedlocomotioninterfacesinproductionsystems.4.6.8SummaryofPracticalCo
nsiderationsTherearemyriadpracticalconsiderationswhendesigning,implementing,andd
eployingalocomotioninterface.Table4.3summarizessomeofthefactorsthatshouldbeconsi
deredtoensurethataninterfacewillbesuccessfulthatuserswillacceptitandthatthevirtua
llocomotioninterfaceiseffective.Thefactorsarecategorizedasfunctionality,systemsa
ndsensorcharacteris-tics,andencumbrancesandrestraints.TheCommentscolumnvariously
explainswhyalocomotioninterfacedesignershouldcareaboutthefactor,expandsonwhatthe
factormeans,orgivesanexampleofthekindsofproblemsthatcanariseifthefactorisignored
.4.7CASESTUDYAcasestudyforawalking-in-placelocomotioninterfacecanbefoundatwww.be
yondthegui.com.
4.7CaseStudy139

TABLE4.3ItemsConsideredinDesign,Implementation,andDeploymentofVirtualLocomotionS
ystemsPracticalConsiderationsCommentsFunctionalityHowhardisittolearntheinterface
?Ifanaturalinterfaceishardtouse,userswillpreferalessnaturalbuteasy-to-learninterfa
ce.Isviewingdirectionindependentofmotiondirection?Allowsyoutolookaroundatthesame
timeyouaremoving.Arehandsusedforlocomotion?Itisgoodtohavehandsfreeforuseduringta
sks.Abletomoveinanydirectionandchangedirectioneasily?Itisdifficulttomakeasharpco
rneronsometreadmillsandwithsomejoysticks.Abletomoveinanydirectionwhilebodyfacesf
orward?Backward?Sideways?Canbasewalkingspeedorsteplengthbesetforeachindividual?F
amiliaritywithonesownsteplengthisafactorcontributingtospatialunderstanding.Ableto
moveatdifferentpaces,andmovequickly(run)orslowly(creep)?Extendsrangeofapplicatio
nsthatcanbesupported.Abletoadjuststeplength,thatis,takelargerorsmallersteps?With
outtheabilitytotakesmallsteps,userscannotfine-tunewheretheystop.Magicalinterface
sforfastmovementbetweendistantplaces?Mostoftenneededforverylargevirtualscenes;li
kelyusedwithamundane-styleinterfaceforhuman-scalemovements.SystemandSensorCharac
teristicsAretheremorethanoneortwoframesoflatencybetweensignalingintenttomoveandv
isualfeedbackofmovement?Trade-offbetweenstart-uplaganddetectingunintendedsteps.D
oestheusermoveoneormorestepsfurtherthanintendedwhenstopping(stoppinglag)?Trade-o

ffbetweenstoppinglagandmissingintendedsteps.Istrackertechnologycompatiblewiththe
physicalenvironment?Forexample,steelraisedflooringcompromisesaccuracyofmagnetict
rackers.Aretrackersensorsimmunetoothersignalsintheroom?Aretheresourcesintheroom(
otherthanthetrackerbeacons)thatthesensorswilldetect,thuscompromisingtrackingaccu
racy?Arewirelesstrackersused?Istheresufficientbandwidth?Isroomelectricallynoisyfro
motherdevicesandwirelesshub?EncumbrancesandRestraintsWhatpartsofthebodywillhaves
ensorsormarkersonthem?Willuserbeabletomakeappropriatemotionswithsensorson?Cancab
lesbemanagedsotheydonotinterferewithusersmotion?Willusergetwoundupinthecableswhen
performingthetask?Doyouneedafixturetorestrictusersrealmovementinthelab?Thismaybea
safety/institutionalreviewboardissueinadditiontokeepingtheuserwithinrangeofthetr
ackingsystem.
4LocomotionInterfaces140

4.8FUTURETRENDSFuturelocomotioninterfacedesignerswillstillfacethesameproblemswed
otoday:howtosensebodypositionandmovementandmapintolocomotiondirectionandspeedwhi
lemeetingapplicationconstraintsthatmayvaryfrommaximizingnaturalnesstominimizingt
herangeofmovementrequired.4.8.1NewTrackerTechnologiesOurbodieshaveonlyasetrangeo
fmotions;so,toincreasethenumberofpara-metersthatcanbecontrolledwithbodymovementa
nd/ortoincreasethepreci-sionwithwhichthoseparameterscanbecontrolled,wemusthavemo
reprecisedataaboutthepositionandmotionoftheuser.Thetrendtosmallerandwirelesstrac
kersmeansthatthesedevicescanbeputinmanymoreplacesonthebody.Sincehavingmoremarker
scanresultinslowertrackerupdates,data-miningtech-niquesarebeingappliedtologsofda
tafrommotioncapturesystemstoidentifythesetofmarkerlocationsthataremostcriticalfo
rdeterminingbodypose(Liuetal.,2005).Thevisionforthefutureisfullyunencumberedtrac
kingofthewholebodyforseveraluserssharingthesamephysicalspaceatthesametime:nowire
s,nobodysuits,andnomarkersorotherdevicesplacedontheusersbody.Computervisionalgori
thmswilluseasinputsynchronizedframesofvideotakenfromanumberofcamerassurroundingt
hespace.Theoutputpertimestepwillbeanestimationofthebodyposition(pose)ofeachusert
hatiscomputedbytrackingfeaturesbetweenframesandbetweencameraviews.Therearesucces
sfulvision-basedtechniquestodaythatusemarkers;thechallengeforthefutureisamarkerl
esssystem.4.8.2ReducingthePhysicalSpaceNeededforReal-WalkingInterfacesTheadvanta
gesofthenaturalnessofareal-walkingvirtual-locomotioninter-faceareoffsetbythefact
thatthevirtualsceneislimitedtothesizeofthephysicalspacecoveredbythetrackingsyste
m.Forexample,toallowtheusertoexploreavirtualmuseumwhilereallywalking,thetracking
spacemustbeaslargeasthemuseum.Severaleffortsareunderwaytoovercomethislimitation.
Someofthenewtechniquesstrivetobeimperceptible,suchasredirectedwalking(Razzaque,2
001);somestrivetomaintaintheactualdistancewalked,suchasmotioncom-pression(Nitzsc
he,Hanebeck,&Schmidt,2004),andothersexplicitlyinterruptuserstoresettheminthephysic
alspacewhilenotchangingtheirlocationinthevirtualscene(Mohleretal.,2004).
4.8FutureTrends141

4.8.3ChangingEconomicsHistoricalexperiencehasledpractitionersincomputer-relatedt
echnologiestoassumecontinuousimprovementsinperformanceanddecreasesincost.Thereha
venotyetbeensuchtrendsfortechnologiessupportingwhole-bodyinterfaces.Thepricesofm
otiontrackers,immersivedisplays,andmotionplatformoutputdeviceshavenotdroppedmuch
inthelast10years.Since3Dgraphicshardwarewasadoptedbymass-marketvideogames,ithasb
ecomecheap,widelyavailable,andveryhighperformance.Asofthiswritinghigh-endtracker
systemsstillhavefive-tosix-digitprices(inU.S.dollars),butNintendosnewestconsole,W
ii,isamass-marketsuccess.TheWiigameconsoleincludeshandheldcontrollersthatsenseha
ndacceler-ationand,whenpointedattheTVscreen,alsosensehandorientation.InthegameWi
i-SportsTennis,thesensedaccelerationisusedtocontrolthemotionoftheracquet(Figure4
.21).Theorientationfeatureenablesuserstopointatobjects

FIGURE4.21NintendoWiigamesystem.Wiiincludesrudimentarymotiontrackinginthehandhel
dcontrollers.Aplayerswingsavirtualtennisracketbyswingingherhandandarm.(Courtesyo
ftheDepartmentofComputerScience,UNCatChapelHill.)
4LocomotionInterfaces142

displayedontheTV.TheWii-Fitgameincludesaplatformthatreportstheweightandcenter-of
-balancepositionoftheuserwhoisstandingonit.Whilewehavenotyetseenanywhole-bodyloc
omotioninterfacesinmass-marketvideogames,wesuspecttheywillsoonappear.Weourselves
arecurrentlydevelopinginterfacesthatuseWiicontrollers.WearehopefulthatWiiwillbet
hebreakthroughproductthatwill,inareasonablyshorttime,resultinwidelyavail-able,ch
eap,andhigh-performancewhole-bodytrackersforallapplicationsandinparticularforvir
tuallocomotion.REFERENCESArthur,K.(2000).EffectsofFieldofViewonPerformancewithHe
ad-MountedDisplays.CSTechnicalReportTR00-019.ChapelHill:DepartmentofComputerScie
nce,UniversityofNorthCarolina.Berthoz,A.(2000).TheBrainsSenseofMovement.Weiss,G.,
trans.Cambridge,MA:HarvardUniversityPress.Bouguila,L.,Evequoz,F.,Courant,M.,&Hir
sbrunner,B.(2004).Walking-pad:Astep-in-placelocomotioninterfaceforvirtualenviron
ments.ProceedingsofSixthInternationalConferenceonMultimodalInterfaces.StateColle
ge,PA,ACM,7781.Bowman,D.A.,Koller,D.,&Hodges,L.F.(1997).Travelinimmersivevirtuale
nviron-ments:Anevaluationofviewpointmotioncontroltechniques.ProceedingsofIEEEVir
-tualRealityAnnualInternationalSymposium,4552.Brooks,F.P.(1988).Graspingrealityth
roughillusioninteractivegraphicsservingscience.ProceedingsofSIGCHIConferenceonHum
anFactorsinComputingSystems,Washing-ton,DC,ACM,1-11.Cobb,S.V.G.,&Nichols,S.C.(19
98).Staticposturetestsfortheassessmentofposturalinstabilityaftervirtualenvironme
ntuse.BrainResearchBulletin47(5):45964.Darken,R.P.,Cockayne,W.R.,&Carmein,D.(1997
).Theomni-directionaltreadmill:Alocomotiondeviceforvirtualworlds.ProceedingsofAs
sociationforComputingMachin-eryUserInterfaceSoftwareandTechnology.Banff,Canada,A
CM,21321.Dichgans,J.,&Brandt,T.(1977).Visualvestibularinteraction:Effectsonself-mo
tionper-ceptionandposturalcontrol.Perception8:755804.Draper,M.(1998).TheAdaptiveE
ffectsofVirtualInterfaces:Vestibulo-OcularReflexandSimulatorSickness.PhDdiss.,Un
iversityofWashington,Seattle.Duh,H.B.-L.,Parker,D.E.,Phillips,J.,&Furness,T.A.(2
004).Conflictingmotioncuesatthefrequencyofcrossoverbetweenthevisualandvestibularse
lf-motionsystemsevokesimulatorsickness.HumanFactors46:14253.Fleming,S.A.,Krum,D.M
.,Hodges,L.F.,&Ribarsky,W.(2002).Simulatorsicknessandpresenceinahighfield-of-vie
wvirtualenvironment[abstract].ProceedingsofComputerHumanInteraction,HumanFactorsi
nComputingSystems.Minneapolis,ACM,78485.Foxlin,E.(2002).Motiontrackingrequirement
sandtechnologies.InStanney,K.M.,ed.HandbookofVirtualEnvironments.Mahwah,NJ:Erlba
um,163210.Gregory,R.L.(1970).TheIntelligentEye.NewYork:McGraw-Hill.
References143

Harris,L.,Jenkin,M.,&Zikovitz,D.C.(1999).Vestibularcuesandvirtualenvironments:Ch
oosingthemagnitudeofthevestibularcue.ProceedingsofIEEEVirtualRealityConfer-ence,
Houston,22936.Hollands,M.A.,Patla,A.E.,&Vickers,J.N.(2002).Lookwhereyouregoing!:Gaze
beha-viourassociatedwithmaintainingandchangingthedirectionoflocomotion.Experimen
-talBrainResearch143(2):22130.Hollerbach,J.M.(2002).Locomotioninterfaces.InStanne
y,K.M.,ed.,HandbookofVirtualEnvironments.Mahwah,NJ:Erlbaum,23954.Hollerbach,J.M.,
Christensen,R.R.,Xu,Y.,&Jacobsen,S.C.(2000).Designspecificationsforthesecondgene
rationSarcosTreadportlocomotioninterface.ProceedingsofASMEDynamicSystemsandContr
olDivision,HapticsSymposium,Orlando,129398.Howard,I.P.(1986).Thevestibularsystem.
InBoff,K.R.,Kaufman,L.,&Thomas,J.P.,eds.,HandbookofPerceptionandHumanPerformance
,1:1126.NewYork:Wiley-Interscience.Inman,V.T.,Ralston,H.J.,&Frank,T.(1981).HumanW
alking.Baltimore/London:WilliamsandWilkins.Insko,B.(2001).PassiveHapticsSignific

antlyEnhancesVirtualEnvironments.CSTechnicalReportTR01-017.ChapelHill:University
ofNorthCarolina,DepartmentofComputerScience.Iwata,H.(1999).Walkingaboutvirtualen
vironmentsonaninfinitefloor.ProceedingsofIEEEVirtualRealityConference,Houston,28
695.Iwata,H.,&Yoshida,Y.(1999).PathreproductiontestsusingaTorustreadmill.Presence
:TeleoperatorsandVirtualEnvironments8(6):58797.Kennedy,R.S.,Drexler,J.M.,Compton,
D.E.,Stanney,K.M.,Lanham,D.S.,&Harm,D.L.(2003).Configuralscoringofsimulatorsickn
ess,cybersicknessandspaceadaptationsyndrome:Similaritiesanddifferences.InHetting
er,L.J.,&Haas,M.W.,eds.,VirtualandAdaptiveEnvironments:Applications,Implications
,andHumanPerformance.Mahwah,NJ:Erlbaum,24778.Kennedy,R.S.,&Lilienthal,M.G.(1995).
Implicationsofbalancedisturbancesfollowingexposuretovirtualrealitysystems.Procee
dingsofIEEEVirtualRealityAnnualInterna-tionalSymposium,ResearchTrianglePark,NC,3
539.Kolasinski,E.(1995).SimulatorSicknessinVirtualEnvironments.TechnicalReport102
7.Alexandria,VA:U.S.ArmyResearchInstitutefortheBehavioralandSocialSciences.Lackn
er,J.R.(1977).Inductionofillusoryself-rotationandnystagmusbyarotatingsound-field
.Aviation,Space,andEnvironmentalMedicine48(2):12931.LaViolaJr.,J.J.(2000).Adiscus
sionofcybersicknessinvirtualenvronments.SIGCHIBulletin32(1):4755.LaViolaJr.,J.J.,
Feliz,D.A.,Keefe,D.F.,&Zeleznik,R.C.(2001).Hands-freemulti-scalenavigationinvirt
ualenvironments.ProceedingsofAssociationforComputingMachinerySymposiumonInteract
ive3DGraphics,ResearchTrianglePark,NC,915.Liu,G.,Zhang,J.,Wang,W.,&McMillan,L.(20
05).Asystemforanalyzingandindexinghuman-motiondatabases.ProceedingsofAssociation
forComputingMachinerySIGMODInternationalConferenceonManagementofData,Baltimore,9
2426.Martini,F.(1999).FundamentalsofAnatomyandPhysiology,4thed.UpperSaddleRiver,N
J:PrenticeHall.
4LocomotionInterfaces144

Meehan,M.,Razzaque,S.,Insko,B.,Whitton,M.C.,&Brooks,F.P.J.(2005).Reviewoffourstu
diesontheuseofphysiologicalreactionasameasureofpresenceinastressfulvirtualenviro
nment.AppliedPsychophysiologyandBiofeedback30(3):23958.Meehan,M.,Razzaque,S.,Whit
ton,M.,&Brooks,F.(2003).Effectsoflatencyonpresenceinstressfulvirtualenvironments
.ProceedingsofIEEEVirtualReality,LosAngeles,14148.Mine,M.R.(1997).ExploitingPropri
oceptioninVirtual-EnvironmentInteraction.CSTech-nicalReportTR97-014.ChapelHill:U
niversityofNorthCarolina,DepartmentofCom-puterScience.Mohler,B.,Thompson,W.B.,Cr
eem-Regehr,S.,Pick,H.L.,Warren,W.,Rieser,J.J.,&Willemsen,P.(2004).Virtualenviron
mentsI:Visualmotioninfluenceslocomotioninatreadmillvirtualenvironment.Proceeding
sofFirstSymposiumonAppliedPerceptioninGraphicsandVisualization.LosAngeles,1922.Ni
tzsche,N.,Hanebeck,U.D.,&Schmidt,G.(2004).Motioncompressionfortelepresentwalking
inlargetargetenvironments.Presence:TeleoperatorsandVirtualEnvironments13(1):4460.
Pang,T.K.,Lim,K.Y.,&Quek,S.M.(2000).Designanddevelopmentofastereoscopicdisplayri
gforacomparativeassessmentofremotefreighthandlingperformance.ProceedingsofJointC
onferenceofAPCHIandASEANErgonomics,Singapore,6267.Peterson,B.,Wells,M.,Furness,T.
,&Hunt,E.(1998).Theeffectsoftheinterfaceonnavi-gationinvirtualenvironments.Proce
edingsofHumanFactorsandErgonomicsSocietyAnnualMeeting,1496505.Razzaque,S.(2005).R
edirectedWalking.CSTechnicalReportTR05-018.ChapelHill:Uni-versityofNorthCarolina
,DepartmentofComputerScience.Razzaque,S.,Kohn,Z.,Whitton,M.(2001).Redirectedwalk
ing.ProceedingsofEuro-graphicsWorkshop,28994.Riecke,B.E.,vanVeen,H.A.H.C.,&Bultho
ff,H.H.(2002).Visualhomingispossiblewithoutlandmarks:Apathintegrationstudyinvirt
ualreality.Presence:TeleoperatorsandVirtualEnvironments11(5):44373.Robinett,W.,&H
olloway,R.(1992).Implementationofflying,scalingandgrabbinginvir-tualenvironments
.ProceedingsofAssociationforComputingMachinerySymposiumonInteractive3DGraphics,C
ambridge,MA,18992.Rolfe,J.M.,&Staples,K.J.(Eds.)(1988).FlightSimulation.Cambridge
:CambridgeUniver-sityPress.Schell,J.,&Shochet,J.(2001).Designinginteractivetheme
parkrides.IEEEComputerGra-phicsandApplications21(4):1113.Slater,M.,&Steed,A.(2000
).Avirtualpresencecounter.Presence:TeleoperatorsandVir-tualEnvironments9(5):41334
.Slater,M.,&Usoh,M.(1993).Presenceinimmersivevirtualenvironments.ProceedingsofIE
EEVirtualRealityAnnualInternationalSymposium,9096.Slater,M.,&Usoh,M.(1994).Body-c
enteredinteractioninimmersivevirtualenviron-ments.InMageneat-Thalman,N.,&Thalman

n,D.,eds.,ArtifcialLifeandVirtualReality.NewYork:JohnWileyandSons,12548.Slater,M.
,Usoh,M.,&Steed,A.(1995).Takingsteps:Theinfluenceofawalkingtechniqueonpresencein
virtualreality.ACMTransactionsonComputerHumanInteraction2(3):20119.
References145

Stoakley,R.,Conway,M.J.,&Pausch,R.(1995).VirtualrealityonaWIM:Interactiveworldsi
nminiature.ProceedingsofHumanFactorsandComputingSystems,26572.Stoffregen,T.A.,Het
tinger,L.J.,Haas,M.W.,&Smart,L.J.(2000).Posturalinstabilityandmotionsicknessinaf
ixed-baseflightsimulator.HumanFactors42:45869.Templeman,J.N.,Denbrook,P.S.,&Siber
t,L.E.(1999).Virtuallocomotion:Walkinginplacethroughvirtualenvironments.Presence
:TeleoperatorsandVirtualEnvironments8(6):598617.Templeman,J.N.,Sibert,L.,Page,R.C
.,&Denbrook,P.S.(2007).PointmanAdevice-basedcontrolforrealistictacticalmovement.P
roceedingsofIEEESymposiumon3DUserInterfaces,Charlotte,NC.Usoh,M.,Arthur,K.,Whitt
on,M.C.,Bastos,R.,Steed,A.,Slater,M.,&Brooks,F.P.(1999).Walking>walking-in-place
>flyinginvirtualenvironments.ProceedingsofSIGGRAPH.LosAngeles,35964.WarrenJr.,W.H
.,Kay,B.A.,Zosh,W.D.,Duchon,A.P.,&Sahuc,S.(2001).Opticflowisusedtocontrolhumanwa
lking.NatureNeuroscience4(2):21316.Yan,L.,Allison,R.S.,&Rushton,S.K.(2004).Newsim
plevirtualwalkingmethodwalkingonthespot.PaperpresentedatNinthAnnualImmersiveProje
ctionTechnology(IPT)Symposium.Young,L.R.,Adelstein,B.D.,&Ellis,S.R.(2006).Demand
characteristicsofquestionnaireusedtoassessmotionsicknessinavirtualenvironment.Pr
oceedingsofIEEEVirtualReality,Alexandria,VA,97102.
4LocomotionInterfaces146

5CHAPTERAuditoryInterfacesS.CamillePeres,VirginiaBest,DerekBrock,BarbaraShinn-Cu
nningham,ChristopherFrauenberger,ThomasHermann,JohnG.Neuhoff,LouiseValgerurNicker
son,TonyStockmanAuditoryinterfacesarebidirectional,communicativeconnectionsbetwe
entwosystemstypicallyahumanuserandatechnicalproduct.Thesidetowardthemachineinvolv
esmachinelistening,speechrecognition,anddialogsystems.Thesidetowardthehumanusesa
uditorydisplays.Thesecanusespeechorpri-marilynonspeechaudiotoconveyinformation.T
hischapterwillfocusonthenonspeechaudiousedtodisplayinformation,althoughinthelast
sectionofthechapter,someintriguingpreviewsintopossiblenonspeechaudio-receptivein
terfaceswillbepresented.Auditorydisplaysarenotnewandhavebeenusedasalarms,forcomm
unica-tion,andasfeedbacktoolsformanydecades.Indeed,inthemid-1800sanauditorydispl
ayusingMorsecodeandthetelegraphusheredinthefieldoftelecommu-nications.Astechnolo
gyhasimproved,ithasbecomeeasiertocreateauditorydis-plays.Thus,theuseofthistechno
logytopresentinformationtousershasbecomecommonplace,withapplicationsrangingfromc
omputerstocrosswalksignals(Brewster,1994;Massof,2003).Thissameimprovementintechn
ologyhascoinci-dentallyincreasedtheneedforauditorydisplays.Someoftheneedsthatcan
bemetthroughauditorydisplaysinclude(1)pre-sentinginformationtovisuallyimpairedpe
ople,(2)providinganadditionalinfor-mationchannelforpeoplewhoseeyesarebusyattendi
ngtoadifferenttask,(3)alertingpeopletoerrororemergencystatesofasystem,and(4)prov
idinginformationviadeviceswithsmallscreenssuchasPDAsorcellphonesthathavealimited
abilitytodisplayvisualinformation.Furthermore,theauditorysystemiswellsuitedtodet
ectandinterpretmultiplesourcesandtypesofinformationaswillbedescribedinthechapter
.

Audio displays (or auditory interfaces) as they are experienced now, at thebegin
ningof the 21st century, are more sophisticated and diverse than the bellsandcli
cks of the past. As mentioned previously, this is primarily due to theincreasing

availability of powerful technical tools for creating these displays. How-ever,

thesetools have only recently become accessible to most engineers anddesigners,
and thus the field of auditory displays is somewhatin its adolescenceand for ma
ny is considered relatively new. Nevertheless, there is asubstantialandgrowingbo
dyofworkregardingallaspectsoftheauditoryinterface.Becausethisfieldisyoungandcurr
entlyexperiencingexponentialgrowth,the lexicon andtaxonomy for the various types
of auditory displays is still in development. A dis-cussion of the debates and
nuances regardingthe development ofthis taxonomy isnot appropriate for this chap
ter, but the interested reader will find details on thistopic in Kramers book on
auditorydisplays (1994), distributed by the Interna-tional Community on Auditory
Displays (ICAD) (www.icad.org ), as well as othersonification sites (e.g., http
://sonification.de ).The terms used in this chapter are outlined and defined int
he following sec-tions. We have organized the types of auditory displays primari
ly by the methodor technique used to create the display. Additionally, all of th
e sound examplesare availableat www.beyondthegui.com .Sonification of Complex Da
taSonification is the useofnonspeechsoundtorenderdata,eithertoenhancethevisualdi
splayorasapurelyaudiodisplay.Sonificationisroutinelyusedinhospi-talstokeeptracko
fphysiologicalvariablessuchasthosemeasuredbyelectrocar-diogram(ECG)machines.Audi
ooutputcandrawtheattentionofmedicalstafftosignificantchangesinpatientswhiletheya
reotherwiseoccupied.Othersonifica-tionsincludetherhythmsinelectroencephalogram(E
EG)signalsthatcanassistwiththepredictionandavoidanceofseizures(Baier,Hermann,Sah
le,&Ritter,2006)andsonificationoftheexecutionofcomputerprograms(Berman&Gallagher
,2006).Therearevarioustypesofsonificationtechniques,andthesewillbeelaboratedthro
ughoutthechapterandparticularlyinSection5.2.4.Considerableresearchhasbeendonereg
ardingquestionsthatariseindesign-ingeffectivemappingsforsonificationofdata.Among
thekeyissuesarevoicing,property,polarity,andscaling/context.Voicingdealswiththem
appingofdatatothesounddomain;thatis,givenasetofinstruments,whichoneshouldbeassoc
iatedwithwhichvariable?Propertydealswithhowchangesinavariableshouldberepresented
,suchasshouldchangesinadatavariablebemappedtopitch,amplitude,ortempo?Polar-ityde
alswiththewayapropertyshouldbechanged(Walker,2002),thatis,shouldachangeinavariab
lecausetheassociatedsoundpropertytoriseorfall?Supposeweightismappedtotempo:Shoul
danincreaseinweightleadtoanincreaseintempo,oradecreasegiventhatincreasingweightg
enerallywouldbeassociatedwithaslowerresponse?
5AuditoryInterfaces148

Scalingdealswithhowquicklyasoundpropertyshouldchange.Forinstance,howcanweconveya
nunderstandingoftheabsolutevaluesbeingdisplayed,whichrequiresestablishingmaximum
andminimumvaluesdisplayed,wherethestartingvalueisinthescale,andwhenthedatacrossz
ero?Howmuchofachangeinanoriginaldatavariableisindicatedbyagivenchangeinthecorres
pondingauditorydisplayparameter?AudificationAverybasictypeofauditorydisplay,call
edaudification,issimplypresentingrawdatausingsound.Thiswillbedescribedinmoredeta
ilinSection5.2.4,butessentiallyeverythingfromvery-low-frequencyseismicdata(Haywa
rd,1994)tovery-high-frequencyradiotelescopedata(Terenzi,1988)canbetransformedint
operceptiblesound.Listenerscanoftenderivemeaningfulinformationfromtheaudificatio
n.Symbolic/SemanticRepresentationsofInformationSimilartoothertypesofinterfaces,a
uditoryinterfacesordisplaysareoftenneededforataskotherthandataanalysisorperceptu
alization.Forinstance,GUIsuseiconstorepresentdifferentsoftwareprogramsorfunction
swithinaprogram.Thistypeofvisualdisplayismoresemanticinnatureanddoesnotrequirean
alysisonthepartoftheuser.Theauditoryequivalentsofvisualiconsareauditorydisplaysk
nownasaudi-toryiconsandearcons.Thesedisplaysareveryusefulintranslatingsymbolicvi
sualartifactsintoauditoryartifactsandwillbediscussedseveraltimesthroughthechapte
r.Anexampleofanauditoryiconisthepaper-crinklingnoisethatisdisplayedwhenauseremptie
stheTrashfolder.Thistechniqueemployssoundsthathaveadirectandthusintuitiveconnect
ionbetweentheauditoryiconandthefunctionoritem.Earconsareatechniqueofrepresenting
functionsoritemswithmoreabstractandsymbolicsounds.Forexample,justasMorsecodesoun
dpatternshaveanarbitraryconnectiontothemeaningofthelettertheyrepresent,themeanin

gofanearconmustbelearned.Thesetypesofdisplayscanbeparticularlyappropriateforprog
ramsthathaveahierarchicalstructureastheyallowforthecommunica-tionofthestructurei
nadditiontotherepresentationofthefunctions.Anewsymbolic/semantictechniqueforaudi
torydisplaysthatshowssomepromiseistheuseofspearcons.Thesearenonspeechcuesusedint
hewaythaticonsorearconswouldbe.Theyarecreatedbyspeedingupaspokenphraseuntilitisn
otrecognizedasspeech(Walker,Nance,&Lindsay,2006).Thisrepresenta-tionofthespokenp
hrase,suchasapersonsnameinaphonelist,canbesloweddowntoarecognizableleveltofacilit
atelearningtheassociationbetweenthespearconandthename.Oncethisassociationismade,
thespearconcanbeplayedusingtheshortestdurationtoreducetheamountoftimenecessaryto
presenttheauditorydisplay.
5AuditoryInterfaces149

Itisveryimportanttounderstand,andshouldbeclearafterreadingtheentirechapter,thatt
hesetechniquesarenotnormallyandsometimesnotevenideallyusedexclusively.Furthermor
e,theyarenotnecessarilyindependentofeachother.Forinstance,indailyweathersonifica
tions(Hermann,Drees,&Ritter,2005),mostdatavariablesweredisplayedviaparameterized
auditoryicons(e.g.,watersoundsgaveaniconiclinktorain,andthedurationofthesoundall
owedtheusertojudgetheamountofrainperunitoftime).Whendesigningandtestingauditoryd
isplays,designerscanconsiderthreedifferentdimensionsoraxes:theinterpretationleve
l,fromanalogictosymbolic;interactivity,fromnoninterac-tivetotightlyclosedinterac
tion;andhybridness,fromisolatedtechniquestocomplexmixturesofdifferenttechniques.Th
eabilitytousethesethreedimen-sionsgivesthedesignerawideandintriguingrangeoftools
tobettermeetusersneeds.5.1NATUREOFTHEINTERFACEItisimportantforthosewhowilldesigna
ndtestauditorydisplaystounderstandsomeofthebasicperceptualpropertiesofthehumanau
ditorysystem.Thesepropertiesdictatehowhumansexperienceandinterpretsounds,andthis
sectiondescribessomeofthemorefundamentalelementsoftheauditoryperceptionprocess.5
.1.1BasicPropertiesofSoundSoundarisesfromvariationsinairpressurecausedbythemotio
norvibrationofanobject.Soundsareoftendescribedaspressurevariationsasafunctionoft
imeandareplottedasawaveform.Figure5.1showsthewaveformforapuresinusoid,whichisper
iodic(repeatsthesamepatternoverandoverintime)andischarac-terizedbyitsfrequency(n
umberofrepetitionspersecond),amplitude(sizeofthepressurevariationaroundthemean),
andphase(howthewaveformisalignedintimerelativetoareferencepoint).Allcomplexsound
scanbedescribedasthesumofaspecificsetofsinusoidswithdifferentfrequencies,amplitu
des,andphases.Sinusoidsareoftenanaturalwaytorepresentthesoundswehearbecausetheme
chanicalpropertiesofthecochleabreakdowninputsoundsintothecompo-nentsatdifferentf
requenciesofvibration.Anyincomingsoundisdecomposedintoitscomponentfrequencies,re
presentedasactivityatspecificpointsalongthecochlea.Manynaturalsoundsareperiodic(
suchasspeechormusic)andcon-tainenergyatanumberofdiscretefrequenciesthataremultip
lesofacommon(fundamental)frequency.Othersarenonperiodic(suchasclicksorwhitenoise
)andcontainenergythatismoreevenlydistributedacrossfrequency.
5AuditoryInterfaces150

5.1.2HumanSensitivitytoAuditoryDimensionsThebasicpropertiesofsoundgiverisetoanum
berofperceptualdimensionsthatformthebasisofauditorydisplays.Theeffectivenessofan
auditorydisplaydependscriticallyonhowsensitivelistenersaretochangesalongthedimen
sionsusedtorepresenttherelevantinformation.Thefollowingsectionspresentabriefexam
inationofhumansensitivitytothedimensionsoffrequencyandpitch,loud-ness,timbre,and
spatiallocation.(Foranextensivediscussion,seeMoore,2003.)FrequencyandPitchAsment
ionedbefore,soundfrequencyisafundamentalorganizingfeatureoftheauditorysystemanda
naturaldimensionforcarryinginformationinauditorydisplays.Effectiveuseofthefreque
ncydomainmusttakeintoaccountboththerangeoffrequenciestowhichhumanearsrespondandh
owwelllistenerscandistinguishneighboringfrequencieswithinthisrange.Humanlistener

sareabletodetectsoundswithfrequenciesbetweenabout16Hzand20kHz,withsensitivityfal
lingoffattheedgesofthisrange.Fre-quencyresolution(i.e.,theabilitytodistinguishdi
fferentfrequencies)withintheaudiblerangeisdeterminedbytheperipheralauditorysyste
m,whichactslikeaseriesofoverlappingfilters.Eachfilterisresponsivetothesoundenerg
y
Period T = 1/frequency100TTime2TAmplitude1
FIGURE5.1Thewaveformforapuresinusoid.Theelementsofawaveformarefrequency(numberof
repetitionspersecond),amplitude(sizeofthepressurevariationaroundthemean),andphas
e(howthewaveformisalignedintimerelativetoareferencepoint).
5.1NatureoftheInterface151

inanarrowrangeoffrequenciesandhasabandwidththatvarieswithfrequency.Forfrequencie
sbelow200Hz,thebandwidthisconstantataround90Hz,whileforhigherfrequenciesitisappr
oximately20percentofthecenterfrequency.Thewidthofthesefilters,orcriticalbands,dete
rminestheminimumfrequencyspacingbetweentwosoundsrequiredtoperceivethemseparately
.Soundsthatfallwithinthesamecriticalbandwillgenerallybedifficulttoseparateperceptu
ally.Pitchisthesubjectiveattributeofperiodicsoundthatallowsittobeorderedonamusic
alscaleortocontributetoamelody.Forpuresinusoids,thepitchismonotonicallyrelatedto
thefrequency,withhigherfrequenciesgivingrisetohigherpitches.Forcomplexperiodicso
unds,pitchismonotonicallyrelatedtothefundamentalfrequency.Mostmusicalinstruments
produceperiodicsoundsthathaveastrongpitch.Periodicsoundswithoutastrongpitchcanno
nethelessbetoldapart(discriminated)basedontheirspectralcontent.LoudnessThesubjec
tiveexperienceofloudnessisrelatedtoitsphysicalcorrelate,theintensityofasoundwav
eform(orsquareofthepressureamplitude)(Figure5.1),andhasbeendescribedusingseveral
differentscales(suchasthesonescale)(Stevens,1957).Humanlistenersaresensitivetoal
argerangeofintensities,withsensitivitythatisroughlylogarithmicsothatthesmallestd
etectablechangeinloudnessforwidebandsoundsisapproximatelyaconstantfractionofther
eferenceloudness.Listenerscandetectintensitychangesofjustafewdecibels(dB)defineda
s10timesthelogarithmoftheratiooftheintensitiesformanytypesofstimuli.TimbreForacom
plexsound,energycanbedistributedindifferentwaysacrossfrequencyandtime,givingthes
ounditsqualityortimbre.Forexample,twocomplextonesofthesamepitchandloudnessmaydif
ferintheirtimbreduetothedetailedstructureofthewaveform(e.g.,differencesintherel
ativemagnitudesofthevariousfrequencycomponents).Inintuitiveterms,thesedifference
sarewhatdistinguishinstruments(e.g.,thebowedviolinfromthepercussivepiano)playing
thesamenoteorchordfromthesamelocationwiththesameloudness.TemporalStructureMostso
undsarenotstationary,butturnonandofforfluctuateacrosstime.Humanlistenersareexqui
sitelysensitivetosuchtemporalchanges.Temporalresolutionisoftenquantifiedasthesma
llestdetectablesilentgapinastimulus,andisontheorderofafewmillisecondsforhumanlis
teners(Plomp,1964).Temporalresolutioncanalsobedescribedintermsofhowwelllisteners
candetectfluctuationsintheintensityofasoundovertime(amplitudemodulation).Modulat
iondetectionthresholdsareconstantforratesuptoabout16Hz,but
5AuditoryInterfaces152

sensitivitydecreasesforratesfrom16to1,000Hz,wheremodulationcannolongerbedetecte
d.Forhumanlisteners,relativelyslowtemporalmodulationsareparticularlyimportant,as
theycontributesignificantlytotheintelligibilityofnaturallyspokenspeech(Shannon,Z
eng,Kamath,Wygonski,&Ekelid,1995).SpatialLocationSoundscanarisefromdifferentloca
tionsrelativetothelistener.Localizationofsoundsourcesispossibleduetoanumberofphy
sicalcuesavailableatthetwoears(seeCarlile,1996,forareview).Differencesinthearriv
altimeandintensityofasoundatthetwoears(causedbytheheadactingasanacousticobstacle

)allowanestimationoflocationinthehorizontalplane.Forexample,asoundoriginatingfro
mtheleftsideofalistenerwillarriveattheleftearslightlyearlierthantheright(bytenst
ohundredsofmilliseconds)andwillbemoreintenseintheleftearthanintheright(byuptoten
sofdecibels).Inaddition,thephysicalstructureoftheouterearaltersincomingsignalsan
dchangestherelativeamountofenergyreachingtheearateachfrequency.Thisspectralfilte
ringdependsonthedirectionofthesourcerelativetothelistener,andthusprovidesdirecti
onalinformationtocomplementthatprovidedbytheinterauralcues(e.g.,allowingelevatio
nandfrontbackdiscrimination).Forestimatingthedistanceofsoundsources,listenersuse
anumberofcues,includingloudness,frequencycontent,and(whenlisteninginanenclosedsp
ace)theratioofthedirectsoundtotheenergyreflectedfromnearbysurfacessuchaswallsand
floors(Bronkhorst&Houtgast,1999).Thespatialresolutionofthehumanauditorysystemisp
oorcomparedtothatofthevisualsystem.Forpairsofsoundsourcespresentedinsuccession,h
umanlistenersarejustabletodetectchangesinangularlocationofaround1degreeforsource
slocatedinthefront,butrequirechangesof10degreesormorefordiscriminationofsourcest
otheside(Mills,1958).Forthecaseofsimultaneoussources,localizationiswellpreserve
daslongasthesourceshavedifferentacousticstructuresandformclearlydistinctobjects(
Best,Gallun,Carlile,&ShinnCunningham,2007).5.1.3UsingAuditoryDimensionsWhenusin
gsoundtodisplayinformation,theavailableauditorydimensionsandhumansensitivitytoth
esedimensionsarecriticalfactors.However,therearealsoanumberofdesignquestionsrela
tedtohowtomapdatatothesedimensionsinaneffectiveway.Asanexample,itisnotalwaysclea
rhowdatapolarityshouldbemapped.Intuitively,increasesinthevalueofadatadimensionse
emasthoughtheyshouldberepresentedbyincreasesinanacousticdimension.Indeed,manyson
ificationexampleshavetakenthisapproach.Forexample,inthesonificationofhistorical
5.1NatureoftheInterface153

weatherdata,dailytemperaturehasbeenmappedtopitchusingthispositivepolarity,wherehig
hpitchesrepresenthightemperaturesandlowpitchesrepresentlowtemperatures(Flowers,
Whitwer,Grafel,&Kotan,2001).Ontheotherhand,anegativepolarityismostnaturalwhensonif
yingsize,wherebydecreasingsizeisbestrepresentedbyincreasingpitch(Walker,2002).T
oaddtothecomplexityofdecisionsregardingpolarity,insomecasesindividuallistenersva
ryconsiderablyintheirpreferredpolarities(Walker&Lane,2001).Asanotherexample,redu
ndantmappingscansometimesincreasetheeffectivenesswithwhichinformationisconveyed.
Recentwork(Peres&Lane,2005)hasshownthattheuseofpitchandloudnessinconjunctionwhen
sonifyingasimpledatasetcanleadtobetterperformance,butotherconjunctionsmaynot.5.
1.4PerceptualConsiderationswithComplexDisplaysMultipleMappingsWithmultidimension
aldatasets,itmaybedesirabletomapdifferentdatadimensionstodifferentperceptualdim
ensions.Asanexample,thepitchandloudnessofatonecanbemanipulatedtosimultaneouslyre
presenttwodifferentparametersintheinformationspace(seePollack&Ficks,1954).Howeve
r,recentworkhasshownthatperceptualdimensions(suchaspitchandloudness)caninteracts
uchthatchangesinonedimensioninfluencetheperceptionofchangesintheother(Neuhoff,20
04).Inmanycases,themosteffectivewayofpresentingmultipledatasetsmaybetomapthemtoa
uditoryobjectswithdistinctidentitiesanddistinctspatiallocations.Theseobjectscanb
edefinedonthebasisoftheiridentity(e.g.,ahightoneandalowtone)ortheirlocation(e.g.
,asourcetotheleftandasourcetotheright).Thisapproachtheoreticallyallowsanunlimite
dnumberofsourcestobepresented,andoffersthelisteneranatural,intuitivewayoflisteni
ngtothedata.MaskingMaskingdescribesareductioninaudibilityofonesoundcausedbythepr
esenceofanother.Aclassicexampleofthisisthereductioninintelligibilitywhenspeechis
presentedagainstabackgroundofnoise.Inauditorydisplayswithspatialcapabilities,sep
aratingsourcesofinterestfromsourcesofnoisecanreducemasking.Forexample,speechpres
entedagainstabackgroundofnoiseiseasiertounderstandwhenthespeechandthenoiseareloc
atedindifferentplaces(e.g.,oneontheleftandoneontheright).Insuchasituation,theaud
itorysystemisabletousedifferencesbetweenthesignalsattheearstoenhancetheperceptio
nofaselectedsource.Inparticular,itisable
5AuditoryInterfaces154

tomakeuseofthefactthatoneofthetwoears(theonenearestthespeechtarget)isbiasedacous
ticallyinfavorofthetargetsoundduetotheshadowingofthenoisebythehead(Bronkhorst,20
00).AuditorySceneAnalysisandAttentionAnothercrucialconsiderationwhendeliveringmu
ltiplesignalstoalistenerishowtheauditorysystemorganizesinformationintoperceptuals
treamsorobjects.Abasicprincipleofauditorysceneanalysis(Bregman,1990)isthattheaudito
rysystemusessimplerulestogroupacousticelementsintostreams,wheretheelementsinast
reamarelikelytohavecomefromthesameobject.Forexample,soundsthathavethesamefrequen
cycontentorarerelatedharmonicallyarelikelytobegroupedintothesameperceptualstream
.Similarly,soundsthathavesynchronousonsetsandoffsetsandcommonamplitudeandfrequen
cymodulationsarelikelytobegroupedtogetherintoasingleperceptualobject.Inaddition
,soundsthatareperceivedasevolvingovertimefromthesamespatiallocationtendtobeperce
ivedasarelatedstreamofevents.Groupingrulescanbeusedinauditorydisplaydesignwhenit
isdesirablethatdifferentsignalsbeperceivedasacoherentstream,butunwantedgroupings
canleadtodisruptionsintheprocessingofindividualsignals.Confusionaboutwhichpieces
oftheacousticmixturebelongtowhichsoundsourcearequitecommoninauditoryscenescontai
ningsoundsthataresimilaralonganyofthesedimensions(Kidd,Mason,&Arbogast,2002).Rel
atedtothisissue,sourcesinamixturecancompeteforattentionifeachsourceisparticular
lysalientorcontainsfeaturesthatmayberelevanttothelistenersbehavioralgoals.Bymakin
gatargetsourcedistinctalongoneoftheperceptualdimensionsdiscussedpreviously(e.g.,
bygivingitadistinctpitchorspatiallocation),confusioncanbereducedbecausetheliste
nersattentionwillbeselectivelydirectedalongthatdimension.Forexample,whenalistener
mustattendtoonevoiceinamixtureofcompetingvoices,thetaskismucheasier(andlessconfu
sionoccurs)whenthetargetvoicediffersingenderfromitscompetitors(Darwin&Hukin,2000
).AuditoryMemoryIncomplexauditorydisplays,thecapabilitiesandlimitationsofauditor
ymemoryareimportantconsiderations.Theauditorysystemcontainsabriefauditorystore(im
mediateorechoicmemory)whereacruderepresentationofthesensorystimulusismaintained,nor
mallyfornolongerthantwoseconds(Neisser,1967).Thisstoremakesitpossibletoretainaso
undtemporarilyinordertomakecomparisonswithlaterarrivingsounds,aswellastoprocess
simultaneousstimuliinaserialfashion(Broadbent,1958).Whenstimuliareprocessedinmor
edetail(suchasthesemanticprocessingofspeechorthelearningofasoundpatternintheenvi
ronment),thereisthepossibilityformorepermanent,categoricalrepresentationsandlon
gtermstorage.
5.1NatureoftheInterface155

5.2TECHNOLOGYOFTHEINTERFACEThissectionprovidesabriefoverviewofthetechnologyrequi
redtocreateauditorydisplays.Atypicalauditorydisplaysystemencompassesthevariousc
omponentssketchedinFigure5.2.Thesecomponentsestablishaclosedloopsystemthatinte
gratestheuserandcanberecreatedinalmostanyauditoryinterface.Itisoftenpossiblefort
heusertointeractwiththesystemthiscanbeaverysimpleinteractionlikestartingthesonif
icationplayback,oritcaninvolvemorecomplexcontinuousinteractions,whicharethepart
icularfocusofinteractivesonification(describedlater).5.2.1AuditoryDisplaySystems
Theloudspeakersorphysicaldisplaysaretheonlyvisible(andaudible)partinthischainandar
eoftenreferredtoasthefrontendoftheauditorydisplaysystem(ADS).Mostofthetechnologyfo
rtheADSishiddenbehindthecurtainofcomputation,algorithms,andsignalprocessing.Thec
urrentsectionfocusesmostlyonthesebackendorinvisibleparts.(Forfurtherreadingonthes
oundengineeringandhardwareofADSs,thereadershouldconsultspecializedliteraturesuch
asMiranda,1998.)Concerningthesoundhardware(orfrontendoftheADS),thechoiceofwhethe
rtouseheadphonesorspeakersisdeterminedbasicallybyissuessuchasprivacy,mobility,pr
acticality,isolation,and/orusergoals.Loudspeakers,forinstance,donotrequirethelis
tenertowearanyelectronicequipmentandthus
DataA: Representation andpreprocessingInteractionsB: Application,processing loop
C: Sonificationtechniques (rendering)D: Technicalsound display

FIGURE5.2Sketchoftheinformationflowinatypicalauditorydisplaysystem.A:Datareprese
ntation.B:Mainapplication(processingloop),whichusesthedatatodeterminewhensoundss
houldbecreated.C:Auditorydisplaytechniquestorenderanacousticsignal(digitalaudios
ignal)basedonthedata.D:Technicalsounddisplaysystemssuchassoundcards,mixers,ampli
fiers,headphones,orloudspeakerstoconvertthedigitalaudiosignalstoaudiblevibration
sattheuserseardrums.
5AuditoryInterfaces156

increasetheusersmobility,yettheirsoundsareaudibletoeveryoneintheroom.Headphones,i
ncontrast,allowapersonalauditorydisplayyetmaybeimpracticalsincetheymayinterferew
ithauditoryperceptionofthenaturalsurroundingsandthusisolatetheuserfromhisorheren
vironment.Loudspeakerandheadphonesystemsalsodifferintheirabilitytocommu-nicateth
esourcelocationofsound.Thisaspectofauditoryinterfacesishighlyrelevantforapplicat
ionswherethegoalistodirecttheusersfocusofattention.Multispeakersystems(Pulkki,199
7)arewellsuitediftheusercanbeassumedtobestationaryandlocatedinasweetspot.Typically
,headphonesleadtotheuserperceivingthesourcewithinhisorherhead;however,three-dime
nsional(3D)spatializationwithheadphonescanbeachievedbymodelingthefiltereffectoft
heouterear(seeSpatialLocationinSection5.1.2).Thisisdonemathemati-callywithanadju
stmentofthesourcesoundusinghead-relatedtransferfunctions(HRTFs)(Carlile,1996).To
beconvincing,however,theheadpositionandorien-tationofthelistenerhavetobetrackeds
othattheperceivedsoundsourcepositioncanbekeptconstantwhiletheusermoveshisorherhe
ad.ThetechnologyforthefrontendoftheADS(DinFigure5.2)iswelldevel-opedandcontinues
toadvanceduetotheworkofsoundandproductengineers.However,thelargerbackendoftheADS
,consistingofthetechnologiestocomputesoundsignalsforauditorydisplays(AthroughCin
Figure5.2),ismuchyoungerandthusnotaswelldeveloped.ThisbackendwillbethefocusofSec
tions5.2.2through5.2.4.Anoteforthehumanfactors/usabilitypractitioner:Inordertode
scribethetechnologyoftheauditoryinterface,theinformationthatwillbedisplayedtothe
usermustoftenbedescribedintermsofprogrammingcodeorcomputerscience.Thus,fortheben
efitofreaderswantingtocreateauditorydisplays,inthesetechnology-orientedsectionsw
ewillusemathematicaldescrip-tionsandprogramcodeexamples.However,readerslessfamil
iarwithprogrammingmaywanttoskipthemathematics/programmingdetails.5.2.2DataRepres
entationandProcessingLetusstartthediscussionoftechnologiesfromthepointintheinfor
mationcirclewhereinformationiscreated,measured,orbecomesavailablewithinacomputer
system(depictedinFigure5.2asA).Informationmayhaveahighlydifferentappear-ance,fro
masymbolic(textual)level(e.g.,analarmconditionthatacoolingsystemfailed)toamorean
alogiclevelofrawmeasurements(e.g.,temperaturevalues).Dataarefrequentlyorganizeda
satableofnumbers,eachcolumnrepresentingadifferentfeaturevariable,eachrowrepresen
tingmeasurementsforasinglerecord.Inacen-susdataset,forinstance,columnscouldbefea
turessuchasincome,weight,andgender,whilerowscouldrepresentdifferentpersons.Wecal
lsucharepresentationdatasetX,andinmostcasesdataorinformationcanberecodedintosuch
aform.WewillrefertorowsofXas~x,andtothei-thfeatureasxi.Manyauditorydisplaysrangin
gfromapplicationsinprocessmonitoringandexploratorydataanalysissonificationstotab
leviewersforblindusersoperate
5.2TechnologyoftheInterface157

usingthistypeofdatarepresentation.Communicationofsingleevents(e.g.,tosignalthata
ne-mailarrived)isalsocommon.Themessagecanbecharacterizedbyasetoffeaturevalues(e.
g.,sender,e-maillength,urgency,existinge-mailattachments,etc.).Thesevaluesformar
owvector~xfollowingthepreviousrep-resentation.Anauditorydisplaytechniqueshouldbe
abletorepresentallpossiblefeaturevectorsusingthesystematictransformationofeventd
atatosound.Statisticsanddatamining(Fayyad,Piatetsky-Shapiro,Smyth,&Uthurusamy,19

96)arethedisciplinesconcernedwithexplainingallpeculiaritiesofhowdataarestructure
dandsummarized,andathoroughintroductionwouldexceedthescopeofthischapter.Centrala
spectsofdata,though,aretherange(minimum/maximumvalues),whetherthedataarediscrete
orcontinuous,andwhetheravariableisnominalorordinal.Itisimportanttoidentifythesea
spectsofdatabecausecertainauditoryvariablesareoftenconsideredbettersuitedtorepre
sentcertaindatatypes;forexample,timbreisabettermatchfornominalvariableswhereasfr
equencyfitswellwithordinalvariables.Datafeatureswithazerovalue(e.g.,velocityofac
ar)matchwithacousticvariablesthathaveazerovalue(e.g.,loudness,vibratofrequency,p
ulsingrate,etc.).5.2.3SoundSynthesisSoundsynthesisisthetechnologicalbasisforcont
rollingsoundcharacteristicswithdata.Inrarecases,itmightbepossibletosimplyrecords
oundsforeverypos-sibleconditionorevent.Butwheneverfullcontroloverallsoundcharact
eristicsiswishedorneeded,soundsynthesisisessential.Digitalsoundsignalsarevectors
ofnumbersthatdescribethesoundpressureateverymoment.Real-timesoundcomputingisthus
computationallyquitedemandingandscaleswiththenumberofindependentaudiochannels.Po
werfulprogrammingsystemsforsoundsynthesisareavailable.Thefollowingcodeexam-plesd
erivefromSuperCollider(McCartney,1996),aversatile,compact,powerful,andopen-sourc
etextualprogrammingsystem.PureDataisanothergraphicalengine(alsocross-platformand
open-source)(Puckette,1997).Additive/SubtractiveSynthesisAdditivesynthesisisthec
reationofcomplexsoundsfromsimpleingredients.Theseingredientsarethebuildingblocks
inabottom-upapproach.Thebuildingblocksaresimplesignals(suchassinewavesbo(t)sin(otj
)),andtheirsuper-positionrepresentstheresultofadditivesoundsynthesisstwtXNi1akbokt
noktjkToobtainharmonictimbres,frequenciesokarechosenasintegermultiplesofafundamenta
lfrequencyo1.Thecoefficientsaideterminehowstronglyeach
5AuditoryInterfaces158

component contributes to the sound. An example of achieving additive synthesisin

SuperCollider follows: For only two partial tones of 440 Hz, the designer would
use{SinOsc.ar(440, mul: 0.4) SinOsc.ar(880, mul: 0.2)}.playSound example S1 (at
http://sonification.de/publications/BeyondGUI ) providesonesecondofthesound.Sou
ndexamplesS2toS4demonstratesometypicaladdi-tivesynthesissounds.Subtractivesynthe
sistakestheopposite(top-down)approachandcreatescomplextimbresbyremovingmaterialf
romaspectrallyrichsourcesuchassawtooth-shapedsignals(refertoSection5.1formoreons
ignalshapes),pulsetrains,orspectrallyrichnoisesignals.Agoodintroductiontosubtrac
tivesynthesiscanbefoundinMoore(1990).SoundexamplesS5toS8demonstratesometypicalso
undsofsubtractivesynthesis,andsoundexamplesS9toS12demonstratedifferentfilters.Wa
vetableSynthesisOneofthemostpracticallyrelevantsynthesistechniquesiswavetablesyn
thesis,inwhicharecordedversionofareal-worldsoundisusedtogeneratethesynthe-sizedo
utput.Mixingandmanipulatingthesynthesisalgorithmallowstheaudiodesignertocreateno
velsoundsortoplayasampleatdifferentmusicalnotes.Manycommercialsoundsynthesizersr
elyonthistechnique,andmostoftheauditoryicons(discussedlater)areproducedusingwave
tablesynthesis.InSuperCollider,awavetableisrepresentedbyaBuffer,andaPlayBufunitg
eneratorcanbeusedtoplaythebufferatarbitraryspeed,asdemonstratedinthefollowingcod
eexample(soundexampleS13),wheretheplaybackrateisslowlymodulatedbythesineoscillat
or:bBuffer.read(s,soundsample.wav);{PlayBuf.ar(1,b.bufnum,SinOsc.kr(0.2,mul:0.4,add
:1)*BufRateScale.kr(b.bufnum),loop:1)}.playOtherSynthesisTechniquesOthersynthesi
stechniquesincludegranularsynthesis,physicalmodelingsoundsynthesis,FM-synthesis,
andnonlinearsynthesis.Discussionofthesetechniqueseasilyfillsbooks,andtheinterest
edreadershouldlooktoMoore(1990),Roads(2001),andCook(2002)formoreinformation.5.2.
4AuditoryDisplayTechniquesinaNutshellThissectionfocusesonvariousauditorydisplayt
echniques.Ingeneral,thesetechniquesarealgorithmsthatconnectthedatatobedisplayedt
osoundsynthesistechniques(describedinSection5.2.3).Asmentionedpreviously,auditor
ydisplay
5.2TechnologyoftheInterface159

techniquescanroughlybecharacterizedassymbolicoranalogic.Westartherewithsymbolics
onificationtechniques.Auditoryicons:Auditoryicons,asmentionedearlier,representsp
ecificmessagesviaanacousticeventthatshouldenablethequickandeffortlessidentificat
ionandinterpretationofthesignalwithrespecttotheunderlyinginformation.Thesesounds
needtobeselectedfromadatabaseofrecordings,orsynthesizedaccordingtothedatafeature
s,whichispracticallyachievedbyadaptingappro-priatesoundsynthesisalgorithms(seeSe
ction5.2.3).Earcons:Differentfromauditoryicons,earconsusemusicalmotifstorepresen
tmessagesandrequiretheusertolearnthemeaningforeachearcon.Asabenefit,earconscanin
heritstructuralpropertiesfromlanguageasamoreabstractandhighlysymbolicformofcommu
nication(Blattner,Papp,&Glinert,1994).Thesesoundscanbebuiltusingconcatenation,wh
ichallowsthedesignertocomposemorecomplexmessagesfromsimplebuildingblocks.Audific
ation:Inaudification,thedataspeakforthemselvesbyusingeverydatavalueasasoundsamplei
nasoundsignals(t).Sinceonlyvariationsabove50Hzareacousticallyperceptible(seeSect
ion5.1),audificationsoftenconsumethousandsofsamplespersecond.Thus,thetechniqueis
suitableonlyif(1)enoughdataareavailable,(2)datacanbeorganizedinacanonicalfashion
(e.g.,time-indexedmeasurements),and(3)datavaluesexhibitvariationsintheselectedfe
aturevariable.Mathematically,audificationcanbeformalizedasthecreationofasmoothin
terpolationfunctiongoingthroughasampleof(time,value)pairs(ta,xa)foralldataitemsa
.Thesimplestimplementationofaudification,however,isjusttousethemeasurementsdirec
tlyasvaluesinthedigitalsoundsignalbysettings[n]xn.Somesoundexamplesforaudificatio
nsdemonstratethetypicalacousticresult(S14,S15).S14isanaudificationofEEGmeasureme
ntsoneelec-trodemeasuresthebrainactivityofabeginningepilepticattack(roughlyinthem
iddleofthesoundexample).S15playsthesamedataatlowertimecompression.Clearlythepitc
hdropsbelowthewell-audiblefrequencyrangeandtheepilepticrhythmisperceivedasanaudi
blerhythmofevents.Parametermappingsonification(PMS):Thisisthemostwidelyusedsonif
icationtechniqueforgeneratinganauditoryrepresentationofdata.Conceptually,thetech
niqueisrelatedtoscatterplotting,wherefeaturesofadatasetdeterminegraphicalfeature
sofsymbols(suchasx-position,y-position,color,size,etc.)andtheoveralldisplayisare
sultofthesuperpositionofthesegraphicalele-ments.Forexample,imagineadatasetofmeas
urementsfor150irises.Foreachflower,measurementsofthepetallength,sepallength,peta
lwidth,andsepalwidtharelisted.Aparametermappingsonification(S16)could,forinstanc
e,mapthepetallengthtotheonsettimeofsonicevents,thesepallengthtothepitchofsonicev
ents,thepetalwidthtobrilliance,andthesepalwidthtodura-tion.Theresultingsonificat
ionwouldallowthelistenertoperceivehowthedata
5AuditoryInterfaces160

areorganizedintimeorchangewithtime.Eachsoundeventrepresentsasin-gleflower,whilet
hePMSdisplaystheentiredatasetofmeasurementsofall150flowers!Model-basedsonificati
on(MBS):ThisisastructurallyverydifferentapproachfromPMS(Hermann,2002).InPMS,data
directlycontrolacousticattributes,whereasinMBSthedataareusedtocreateasound-capab
ledynamicmodel.Theresultofthisisthatasonificationmodelwillnotsoundatallunlessexc
itedbytheuserandthusputsinteractionintothefore.Thesetofrulesregardinghowtocreate
avir-tualsoundobjectfromdataiscalledasonificationmodel,andtherulescanbedesignedi
natask-orientedway.Forexample,imaginethateverytextmessageinamobilephoneislikeama
rbleinabox.Byshakingthephone,themarblesmove,interact,andtherebycreateanacousticr
esponsefromwhichyoucaninferhowmanytextmessages,size,andsoon,havearrived(Williams
on,Mur-ray-Smith,&Hughes,2007).Theexcitationhereisshaking,thedynamicsarethephysica
llawsthatdescribethemarblemotionandinteractions,andsoforth.Thesonificationmodels
imulatestheentirephysicalprocess,andthuscreatesaninteractiveandinformativesonifi
cation.Interactivesonification:Thisisaspecialfocusinauditoryinterfacesandcanbeus
edwithmanytypesofsonificationtechniques.Oftenaparticularbenefitresultsfromtightl
yclosedinteractionloopsbetweentheuserandasonificationsystem(Hunt&Hermann,2004).A
llsonificationtechniquescanbemodifiedtobemoreinteractive;forinstance,foraudifica
tion,interactivesonificationcanenabletheusertoactivelynavigatethedatawhilegenera

tingthesoundandsoon.Therationalebehindinteractivesonificationisthatpeopletypical
lygetlatency-freeacousticresponsesasby-productsoftheirinteractionactivity,andthe
yusetheacousticfeedbackcontinuouslytorefinetheiractivity,beitwithinasearch,scan,
discovery,oranyothertask.5.3CURRENTINTERFACEIMPLEMENTATIONSThefollowingsectionpr
ovidesdescriptionsofwhenandwhyauditoryinterfacesarecurrentlyutilized,bothforsigh
tedandvisuallyimpairedusers.5.3.1ABitofHistorySoundoftenhelpsdirectourfocusandde
scribeswhatisgoingon.Listeningtoacarengineorthespinningofaharddrivecanoffervital
cluesaboutwhetherthecarisingoodmechanicalconditionorthecomputerisfinishedsavinga
file.Inearlycomputing,suchincidentalsoundswereoftenusedforexample,beepswereintrod
ucedtoindicateerrorsintheprogramorthebeginningofanewitera-tionofaprogramloop.The
reisalongtraditionofaudioindicatingwarningsand
5.3CurrentInterfaceImplementations161

alerts.Soundhasthepotentialtobeusedinmanymoresophisticatedways,rang-ingfromshort
soundstoindicatespecificeventstofullyimmersivespatialsoundenvironments.5.3.2WhyS
oundIsUsedInadditiontothosementionedintheintroductionofthischapter,therearenumer
ousreasonswhysoundmaybeusedinaninterface.Averycommononeistoreinforceavisualmessa
ge,suchasanalert.Otherreasonsareoutlinednext.Reducingvisualoverload:Visualinterf
acestendtobebusy,fillingasmuchofthescreenaspossiblewithinformation.Constantlycha
ngingvisualinformationcanbedistractingandcanlimittheamountofinformationthatreach
estheuser.Whereapplicable,thecognitiveloadcanbereducedbychannelinginfor-mationto
theears(Brown,Newsome,&Glinert,1989).Reinforcingvisualmessages:Sendingthesameinf
ormationtomorethanonesensecanensurethattheuserreceivestheinformation,makingthein
terfacemoreeffective.Forexample,whenenteringapersonalidentificationnum-ber(PIN)a
tanautomatedtellermachine(ATM),themachinebeepsforeachnumberenteredandthevisualin
terfaceusesanasterisktorepresenteachnumber.Thisdualfeedbackcanreassureusersthatt
heirinputwasreceived.Wheneyesareelsewhere:Sincesoundcanbeperceivedfromalldirecti
ons,itisidealforprovidinginformationwhentheeyesareotherwiseoccupied.Thiscouldbew
heresomeonesvisualattentionshouldbeentirelydevotedtoaspecifictasksuchasdrivingors
urgicallyoperatingonapatient(Recarte&Nunes,2003).Whenaudioismoreinformative:Huma
nsareverygoodathearingpatternsinsound.Thismeansthatattimesitiseasiertounderstand
informationwhenitissonified(Bly,1982).Twoprimeexamplesofthisareseismicdata(Haywa
rd,1994)andmedicalmonitoringdata(Baier&Hermann,2004).Userscanveryquicklynoticeac
hangethatmaynotbeaseasilynoticedwhenlookingatnum-bersoragraph.Smallornovisualdis
play:Unlikevisualdisplays,wherethesizeoftheinterfaceisdeterminedbythesizeofthede
vice,audioislimitedonlybythesoundqualitythatthedevicecanprovide.Itisthereforeago
odcandidateforaugmentingorreplacingavisualinterface.Conveyingemotion:Theaestheti
csofsoundcanhaveagreatimpactontheusersimpressionofanapplication.Thisisparticularl
yobviousinvideogames,wherethesounddesigniscarefullyorchestratedtomakeplayersenjo
ythegameandtoimpacttheamountoftensiontheplayerexperiences.
5AuditoryInterfaces162

5.3.3DrawbackstoUsingSoundSomeofthemajordisadvantagestousingsoundareannoyanceand
lackofpri-vacy.Therearealsodangerstousingtoomuchsound.Ifsoundispoorlydesignedoru
sedatthewrongtime,usersareveryquicktoturnitoff.Someofthedraw-backsareoutlinednex
t:Annoyance:Soundisverygoodatdrawingtheusersattention.However,theurgencyofthesoun
dshouldberelativetotheimportanceoftheinformation.Theuseofobtrusivesoundstorepres
entsomethingoflowimportancecanquicklyannoytheuser.Privacy:Soundisomnidirectional
andthuscannotbedirectedatasingleuserun-lesstheyareusingaheadsetorahandset.Theref
ore,ifheadsetsarenotemployed,anauditoryinterfacecanpublicizewhatauserisdoinginan
undesirablemanner.Auditoryoverload:Displayingtoomuchinformationinsoundcanresulti
nthatinformationlosingmeaningandbeinginterpretedasnoise.Interference/masking:Asm

entionedinSection5.1.3,soundscaninterfereormaskoneanother.Likeauditoryoverload,t
hiscanresultinlossofinformation.Environmentalsoundscanalsocauseinterferenceand/o
rmasking.Lowresolution:Withvisualdisplays,objectscanbelocatedverypreciselyonthes
creen;withspatialsoundinterfaces,thereisagreaterareathatcanbeusedbuttheresolutio
nislower(seeSection5.1.2).Thisdifferenceisevident,forexample,invisualandaudiogam
es.Inavisualgame,itispossibletopinpointatargetwithgreataccuracyonacomputerscreen
.Inanauditorygame,how-ever,althoughtheauditorydisplaymaybepresentedinanareamuchl
argerthanacomputerscreen,thelowerresolutionthathearingaffordsinlocatingaudiosour
cesgreatlyreducestheaccuracywithwhichtheycanbelocated.Impermanence:Unlikevisuald
isplays,wheretheinformationonthescreenremains,audioisserialandonceplayediseasily
forgotten(seeSection5.1).Lackoffamiliarity:Soundstakeawhiletogettoknowandcanbein
itiallyconfus-ingtousersuntiltheyknowwhattheyarelisteningto.Aswithvisualicons,th
esoundsinanaudiointerfaceneedtobeeasytolearn.5.3.4AdvancedAudioInterfacesTheavai
labilityofincreasinglysophisticatedaudiohardwareandsoftwarehasprovidedthepossibi
lityformorewidespreaduseofaudioininterfaces.Thissophisticatedtechnologyhascontri
butedtousingaudiotosupportricherhumancomputerinteractionsaswellasincreasedthequal
ityoftheaudiothatcanbeused,whichcanbetheequivalentofthatusedinafilmorradioproduc
tion.Thismoresophisticateduseofaudiomayofcoursebepartofamultimodal
5.3CurrentInterfaceImplementations163

interface or the basis of an audio-only display. Althoug h much of t his potenti

alre mains underexploited, in this subsection we exami ne several application ar
easwhe re these improved audi o capa bili ties have been us ed to good effect.Au
dio for Moni toringAudio is a well-known method for monitoring in specialized en
vironments such ashospitals and environmental monitoring facilities such as for
weather and seismicactivity. Other applications are stock market and network mon
itoring.Accentus ( http://www.accentus.com) makes Sonify!, a product for traders
to(amongotherthings)monitorstockpricechangesandprogresstowardatarget(Janata&Chi
lds,2004).Sonify!usesrealinstrumentsandshortmelodiestocreatepleasanttonesthatare
informativeyetremainunobtrusive.Theuseofsoundisparticularlyappropriateastradersa
reusuallysurroundedbynumerousscreensandworkinaveryvisuallyintensiveenvironment.T
heyareoftenrequiredtomonitordatawhiletalkingonthephone.Thenonspeechaudiocanreduc
ethevisualoverloadandcanbeheardinthebackgroundofotherauditoryeventssuchastelepho
neconversations.Numerousprogramsthatusesoundtomonitornetworktraffichavebeendevel
oped.TheSheridanInstituteinCanadahasdevelopediSIC(Farkas,2006),whichusesmathemat
icalequationsonnetworktrafficdatatocreatemusic.TheSoundofTraffic(Weir,2005)usesM
IDIsoundstotracktraffictospecificportsonacomputer.Thesenetworkmonitorsallowadmin
istratorstohearreal-timedataabouttheirsitesandnetworkswithouthavingtopurposelygo
toaninterfaceorlogtocheckactivity;theyallowmonitoringtobeabackgroundprocess.Whil
etheseprogramsarenotcommerciallyavailable,theyshowagrowingtrendinusingaudioindiv
erseapplications.AudioinGamesAudioplaysaveryimportantroleinvideogames,usingmusic
tosetthemoodandsoundeffectstobringrealismtotheaction.Audioinmostmoderngamesisamu
ltilayeredproductionusedtorendercomplexauditoryscenes,creatinganimmersiveenviron
ment.SoundusuallytakestheformofeffectssuchasonewouldhearinfilmknownasFoleyeffects
1andhumandialogtoimbuesceneswithrealism.Thisaudioisincreasinglysynthesizeddynamic
allyratherthanprerecorded.Aparticularproblemfacedbygamedesignersisthefactthat,wh
ilebroachedearlyoninthedesignphase,oftensoundsareonlyincorporatedintothegameafte
rthewholeoftheinteractivegraphicdesignhasbeenimple-mented.Thus,propersoundtestin
gisdoneataverylatestage.Theendresult1JackFoleypioneeredthecraftofcreatingsoundef
fects,usingallkindsofmaterialstoimi-tatesounds,intheearlydaysofthetalkiesforUniver
salStudios.
5AuditoryInterfaces164

i s that the sound desi gn is mostl y aest hetic as opposed to informative and n
otnearly as powerful as it could be.Audio-Only GamesAudio-only games are often,
but not exclusively, targeted at the visually impaired.Evolving from mostly spe
ech and simple sound effects, todays audio games,such as Shades of Doom (first-pe
rson shooter) and Lone Wolf (submarine simula-tor), use sophisticated sound, tra
nsitioning smoothly between multilayered andimmersive soundscapes. Many of todays
audio-only games also provide a visualinterface for collaborative game play bet
ween visually impaired and sightedgamers. A typical issue that audio game design
ers face is supporting orientationand navigation in a 3D world (Andresen, 2002)fo
r example, which way is theplayer facing, where are the walls/entrances/exits, a
nd what is the state of otherplayers/objects? The AudioGames website ( http://ww
w.audiogames.net) is dedi-catedtoblind-accessiblecomputergamesandincludesissueso
ftheaudiogamingmagazineAudyssey(2006).5.3.5ApplicationsofAuditoryInterfacestoAcc
essibilityNonspeechsoundhassignificantpotentialforimprovingaccessibility,eithero
nitsownorasacomplementtoothermedia,particularlyforuserswithvisualimpairments,bec
auseofitsabilitytoconveylargeamountsofinformationrapidlyandinparallel.Imagineavi
suallyimpairedteachermonitoringtheprogressofstudentstakinganelectronicmultiple-c
hoicetest.Theteachercouldgetarapidauditoryglanceofapproximatelyhowfarthroughthetes
tstudentswereandtheproportionofcorrectanswers,individuallyandasawhole,usingtheda
tasonificationconceptsdescribedatthebeginningofthechapter.AccessibilityandtheDes
ktopVisuallyimpairedusersemploysoftwarecalledascreenreader,whichusessyn-theticsp
eechtospeaktextsuchasmenus,thestateofGUIwidgets,andtextindocuments.Themostpopula
rscreenreadersusedworldwideareJawsforWin-dows(JFW)byFreedomScientific(2005)andWi
ndow-EyesbyGWMicro(2006).Theuseofnonspeechaudiohasseenrelativelylittlecommercial
uptake.ThereleaseofJFWversion5in2003representedthefirstsignificantuseofnonspeech
soundintheformoftheabilitytocustomizefeedback.Thesecustomizations,calledbehaviors
,aredefinedinschemes.Eachschemecanbeassociatedwithaspecificapplication.Examplesin
cludefocusonaparticulartypeofwidget,thestateofacheckbox,upper/lowercase,degreeof
indentation,andvaluesofHTMLattributes.
5.3CurrentInterfaceImplementations165

The inclusion of nonspeech sound is intended to improve efficiency andeffectiven

ess, such as in navigation of the GUI or screen-based proofreading oflengthy doc
uments. However, significant effort is required by users or interfacedevelopers
to associate specific sounds with events or symbols, and further todevelop a coh
erent set of these associations into an overall sound scheme for anapplication.O
ther uses of nonspeech sound to improve accessibility have been reported inthe I
CAD literature ( http://www.icad.org). These include supporting the naviga-tion
of structured documents and the Web, auditory progress bars, auditorygraphs, aud
itory widgets, auditory access to the widgets in a word processor,and use of aud
io to review data in spreadsheets.Mobile AccessibilityNonspeech sound has been e
mployed in mobility devices, such as giving longer-range warnings to a blind use
r of an approaching obstacle than can be obtainedusing a white cane. A study car
ried out by Walker and Lindsay (2004), where par-ticipants were guided along rou
tes by sound beacons, showed good performanceeven in the worst case, proving tha
t the nonspeech auditory interface couldsuccessfully be used for navigation.The
K-Sonar (Bay Advanced Technologies, 2006), intended to supplement awhite cane, i
s like a flashlight that sends out a beam of ultrasound instead of light.Using t
he ultrasound waves reflected from the objects in the beams path, K-Sonarprovides
complex audible representations of the objects that users can learn torecognize
and use to build mental maps of their environment.Despite the success of these
applications, problems can arise with the useof audio in such mobility and orien
tation devices. Sounds that are fed back tothe user can be masked or made less c
lear by ambient sound such as traffic orconstruction noise. Although some users
might choose to reduce this problemwith earphones, many visually impaired users

are wary of anything that limitstheirperceptionoftheirenvironment.Thisproblemcan

beresolvedthroughtheuseofasmallspeakerpositionedjustafewinchesawayfromoneoftheus
ersears.The Trekker, developed by Humanware (www.humanware.com), a largelyspeechbasedgeographicpositioningsystem(GPS)mobilityandorientationsys-temforvisuallyimp
airedusers,usessuchasystem.Thisapproachkeepstheaudiooutputaudibleandrelativelypr
ivatewithoutmaskingotherenvironmentalsounds.Becauseoftheproblemsofsoundinterfere
nce,anumberofotherultra-soundorinfraredsystemsemployhapticfeedbackratherthansoun
d,usingthestrengthandtypeofvibrationstoprovideinformationaboutthesizeandproxim-i
tyofnearbyobjects.Apromisingapproachtothesoundinterferenceproblemisofferedbytheu
seofboneconductionheadphones.Ratherthangoingthroughtheeardrum,boneconductionhead
phonesconvertsoundsintomechanicalvibrationsgoingthroughtheskullstraighttotheaudi
torynerve.
5AuditoryInterfaces166

5.4HUMANFACTORSDESIGNOFANAUDITORYINTERFACESoundiscommonlythoughtofasawaytoenhanc
ethedisplayofvisualinforma-tion,butastheprecedingsectionsinthischapterhavedemons
trated,italsohasarangeofinformationalcapacitiesandadvantagesthatmakeitparticular
lyusefulinavarietyofinteractivesettings,and,insomecontexts,makeitthebestoronlych
oiceforconveyinginformation.Thus,thedesignproblemisnotjustoneofevaluatingcriteri
afortheuseofsound,butisalsoaquestionofdetermininghowsoundwillbeusedtoitsbesteffe
ct.However,itisimportantthatdesignersdonotuseauditorydisplaysunlesstheauditorydi
splaysarebeneficialand/ornecessary.Therearemanycircum-stanceswhereauditorydispla
yscouldbeutilized,butitisonlythroughtheutiliza-tionofsoundusercentereddesignprinc
iplesthatthepractitionercandeterminewhetheranauditorydisplayshouldbeutilized.Asw
ithallgooduser-centereddesign,thedecisiontousesoundasadis-playtechniqueshouldbeb
asedonacomprehensivetaskandneedsanalysis.Thisanalysiswillinformdesignspecificati
onandtheultimateimplementation.Thissectionprovidesinformationaboutsomeoftheuniqu
eissuesandchallengesthatpractitionersanddesignersmayfacewhenconsideringauditoryi
nterfaces.5.4.1TaskAnalysisHowDoesItDifferforanAuditoryInterface?Taskanalysisisac
ommontechniqueusedaspartoftheuser-centereddesignpro-cessforstandardGUIs.However,
theuniquenatureofauditoryinterfacesrequiresspecialconsiderationsinthedevelopment
andutilizationofthetaskanalysis.DescriptionoftheUserInadditiontotheusualvariable
ssuchasgender,age,experiencewithinforma-tiontechnology,andsoon,thatdeveloperssho
uldconsider,thedevelopmentofauditorydisplaysneedstotakespecificaccountoftheexper
ienceofaudioofthetargetuserpopulationfortheapplicationasawhole.Thisinturnbreaksd
ownintoanumberofdistinctvariables,includingthefollowing:Musicalexpertise:Whatist
heculturalbackground,level,andrangeofmusicalexpertiseofparticipants?Userswithmus
icaltrainingarebetterindiscriminat-ingvariationsofpitchortemporalfeatureslikerhy
thm.Familiaritywithauditorydisplays:Irrespectiveofmusicalability,whatlevelofexpe
riencewilltheusershavewithusingauditorydisplays?(SeealsoSection5.6.1.)
5.4HumanFactorsDesignofanAuditoryInterface167

Hearingabilities:Willtheuser/groupofusershaveahigherlikelihoodofhearingimpairmen
tthanotherpopulations(e.g.,workerswhooperateloudmachineryonaregularbasis)?Dothey
havelimitedspatialresolutionbecauseofhearinglossinoneear?Aretheuserscapableofana
lyticallistening?WorktheUserMustAccomplishThegoalofarticulatingthespecifictaskst
heuserwillbeperformingistwofold.First,itdevelopsanaccountofbothwhattheuserisexpe
ctedtodointhetargetedactivityandhowthatistobeaccomplished.Second,itprovidesdetai
leddescrip-tionsoftheinformationthatmustbeavailabletotheuserinordertoperformthet
asks.Generally,thisprocessentailscreatingdetaileddescriptionsoftheproceduresthat
usersmustfollowforaccomplishingcertaintasks.Forauditorydisplays,however,thesepro
ceduresarenotnecessarilyvisualorpsychomotor(e.g.,lookattheinformationdisplayedon

thespeedometerandadjustthespeedaccordinglywitheitherthebrakeorthegaspedal).Thus,
additionalelementsofthetaskmustbeconsidered.WhatTasksWilltheUserPerformUsingtheA
uditoryDisplay?Ifanaudi-torydisplaywillbepartofalargermultimodalinterface,whichi
smoreoftenthannotthecase,thetaskanalysismustaccountforallfacetsoftheuserstask,not
justtheauralcomponent.Exploringdatawithinteractivesonificationtechniques,forexam
ple,usuallyentailsmorethanjustinteractivelistening(Hermann&Hunt,2005).Manyusersn
eediterativeaccesstothedataaswellaswaystostipulateandrevisethemannerofauditoryin
teraction,bethisvariableparametermappingsorasonificationmodel.Eachoftheseaspects
ofthetaskmustbeidentifiedanddescribedprocedurally.Inparticular,theuseandroleofan
ynonaudiointerfacecomponents,suchasacommandlineorgraphicaldisplay,commonornoveli
nputdevices,andsoon,shouldbespelledoutintheanalysis,aswellasthekindofcognitive,p
erceptual,andmotoractionsthattheusermustperformtodefineandinteractwiththesonific
ation.Theresultingdescriptionofthecom-pletetaskthenservesastheprimarybasisforspe
cifyingthecompositionaldetailsoftheuserinterface,thatis,whattheuserwillactuallyh
ear,see,andphysicallymanipulate.Inaddition,someauditorytasks,particularlythosedr
ivenbyexternaleventsordata,maybecriticalcomponentsoflargeroperationsthataresubje
cttovari-ableratesofactivityorpriority.Whenthisislikelytobeanissue,identifyingpo
tentialperformanceboundaryconditionsisimportant.Itmaybepossible,forinstance,fore
ventpacingtoexceedtheabilitiesofuserstorespondtothedemandsofthetaskorfortheusersa
ttentiontobeoverwhelmedbytheprior-ityofotherstimuliduringperiodsofhighoperationa
lactivity.Concernwith
5AuditoryInterfaces168

performancelimitsandtaskprioritiesiscloselyrelatedtootherconcernsthatcomplexuser
environmentsraise,butiseasilyoverlooked.Bymakingnoteofhowandwhenthesefactorsmaya
rise,thetaskanalysishelpstoidentifywheretheauditoryinterfacedesignislikelytorequ
irethegreatesteffort.Indeed,insomesituations,auditorydisplaysareusedtoreducecogn
itiveloadatpointsinthetaskwhenthecognitiveloadisthehighest.WhatInformationShould
theSoundConvey?Adetailedaccountoftheinformationthatsoundwillbeusedtoconveyinthea
uditoryinterfacemustalsobedevelopedinconjunctionwiththetaskanalysis.Thisinformationanalysisspecifieswhattheusermustknowtoachievehisorherapplica-tiongoals,andis
especiallyimportantinthedesignofauditorydisplaysbecauseofthesomewhatuniquechalle
ngesthatauditoryinformationdesignposes.Unlikeotherrepresentationaltechniquesthat
arecommonlyusedinthedesignofinformationtasks,particularlythoseusedinthedesignofv
isualdisplays,themappingofdatato,andtheperceptionofmeaningin,soundcanbesubjectto
arangeofindividualperformancedifferences(see,inparticular,Sections5.1and5.3).The
informationanalysisisbestorganizedtocorrespondtotheorganizationofthetaskanalysis
.Descriptionsoftheinformationinvolvedateachstepinthetaskshouldbedetailedenoughto
addresstherepresentationalandperceptualconsiderationsthatwillariseattheauditoryd
esignstage,manyofwhichhavebeendescribedinprecedingsections.Inparticular,theinfor
mationtobeconveyedtotheusershouldbecharacterizedinthefollowingways:FApplicationt
oelementsofthetask(one,several,many,etc.)and/oritscon-ceptualpurposeFClassandorg
anization:qualitative(nominal,categorical,hierarchical);spatial(direction,distan
ce);temporal;quantitative(nominal,binary,ordinal,integral,ratio,etc.);rangeFExpe
cteddegreeofuserfamiliarity(Isthisinformationthattheuserwilleasilyrecognizeorwil
ltrainingberequired?)FMeta-leveldescriptionofwhattheinformationwillbeusedfor(e.g
.,whentheinformationisrelevanttootheraspectsofthetask,andwhenitisredundantorrein
forcesotherdisplayedinformation,orhasramificationsinsomebroadercontext)Anadditio
nalfacetoftheinformationanalysisthatisoftenneededfortheauditoryinformationdesign
isanexplicitspecificationoftheunderlyingdata(seeSection5.2.2).Thisisgenerallythe
caseforqualitativeinformationandfortask-specificsubsetsofnumericalinformationtha
tarenonlinear(e.g.,noncontig-uous)inoneormoreways.
5.4HumanFactorsDesignofanAuditoryInterface169

ContextandEnvironmentoftheAuditoryDisplaySincecertainoperationalenvironmentshave
implicationsforhowanauditorydisplayshouldbeorganizedandpresented,thetaskanalysis
shouldalsoconsiderthefollowingquestions.First,willthetaskbeperformedinasingleors
hareduserspace?Thisques-tionbearsdirectlyonhowtheauditorydisplayisrenderedforind
ividualorgroupuse,thatis,throughheadphonesorloudspeakers.Second,willtheuserbeeng
agedinmorethanonetask,andifsowhatistheroleofconcurrency?Themostimportantquestion
hereishowauditoryinfor-mationinthetaskbeinganalyzedislikelytointeractwiththeperf
ormanceofanothertasktheuserisexpectedtoperformandviceversa,especiallyifsoundisal
sousedbytheothertask.Particularattentionshouldbegiventothepotentialforconflictin
gorambiguoususesofsound.Third,willexternalsoundbepresentintheenvironment,andifso
whatisitsinformationalrole?Thisconcernissomewhatlikethepreviousconsideration,onl
yatalargerscale.Iftheauditorytaskwillbeembeddedinalargerenviron-mentinwhichnoise
orintentionalusesofsoundarepresent,thetaskanalysisshouldaddressthepotentialforma
skingorotherdisruptiveimpactsontheauditorytask.5.4.2ConstraintsonInterfaceDesign
Afteruser,task,andenvironmentalconsiderations,designersofauditoryinter-facesmust
considerthepracticallimitsassociatedwiththecapacitiesandcostsofcurrentcomputeran
daudiotechnologies.Theprimaryissuesthatdesignersmustconfrontareprocessingpoweran
dmodeofrendering,particularlyinmid-dle-andlower-tierplatforms.Therearesomeenviro
nmentsthatwillbecondu-civetoacomputationallyintensiveauditorydisplay,suchasasyst
emthathasbeenengineeredtomeettherequirementsofavirtualimmersiveenvironmentinalab
oratoryoraresearchanddevelopmentfacility.However,manyhandhelddevicesandportablec
omputersdonothavethecapacityforadisplaythatrequiresmuchcomputationalpower,andthi
swilllikelybethecaseforanumberofyearstocome.Forinstance,manyportabledevicescurre
ntlylackthethrough-putand/ortheprocessingrequirementsforreal-time3Dauditoryrende
ringofmultiplesources.Furthermore,widelyavailablesolutionsfortailoringbinauralpr
ocessingforindividuallistenersarestillseveralyearsaway.Thus,ifthetaskrequirestha
tthedisplaydevicebeportable,thedesignershouldavoidcreatinganauditoryinterfacetha
tdependsonconveyingaccuratespatialinformationtothelistener.Otherpracticalconside
rationsthatareuniquetoauditoryinterfacedesignsarefactorsassociatedwithloudspeake
randheadphonerendering,personaliza-tion,andchoiceofsoundfileformat.Accurateperce
ptionofauditorydetailin
5AuditoryInterfaces170

loudspeakerrendering,forinstance,suchaslocation,motion,andhigh-frequencyinformat
ion,requiresthatthelistenerbeideallypositionedinrelationtotheout-putoftheloudspe
akersystem.Renderingwithheadphonesand/orearbudsmaybeinappropriateinsomeoperation
alsettingsbecausethiscandefeatthelistenersabilitytohearotherimportantsoundsinthei
mmediateenvironment.Designersalsomustconsiderhowuserswillaccessandcontrolparamet
erssuchasloudness,equalization,programselection,andotherfeaturesofanauditoryinte
rfacethattheusermaywishtoindividualize.Arelated,ergonomicissuethatdesignersmustb
eawareofistheriskofhearinglossassociatedwithrepeatedexposuretoexcessivelyloudsou
nds.Finally,designersshouldunderstandhowMIDI(musicalinstrumentdigitalinterface)f
ilesworkandwhattheperceptualadvantagesandfunctionaltrade-offsarebetweenuncompres
sedauditoryfileformats,suchastheWAVandAIFFspe-cifications,andcompressionformats,
suchasWMAandMP3(e.g.,seeLieder,2004).Iftheuserofanauditoryinterfacemustbeabletoe
xchangesoundswithcollaborators,cross-platformcompatibilityisacrucialconsideratio
n.Whenthisisthecase,thedesignershouldensurethattheinterfacesupportsarangeofsound
fileformatsthatareappropriatefortheperceptualrequirementsofthetask.TheMP3format,
forexample,isadequateforauditoryalertsandiconsbutinferiortotheWAVformatformanysc
ientificapplications.5.4.3WhentoUseanAuditoryInterfaceDesignmotivationsforaudito
ryinterfacescanbebroadlyorganizedintofourfunctionalcategories:managinguserattent
ion,workingwithsounddirectly,usingsoundinconjunctionwithotherdisplays,andusingso
undastheprimarydisplaymodality.Althougheachofthesehasbeenmentionedanddescribedin
previoussections,inthissubsectionwebrieflyidentifyspecifichumanfactorsrelevantfo

reachapplicationtype.ManyofthesehumanfactorshavebeendescribedinsomedetailinSecti
on5.1,buttheusabilitypractitionermayrequiremoreinformationwhenactuallydesigninga
ndimplementingtheseapplica-tions.Consequently,additionalissuesandsourcesofinform
ationareprovidedhere.ManagingUserAttentionVariousauditorymaterialscanbeusedtoman
ageattention,andthemannerinwhichinformationisconveyedcanbeeitherdiscreteorcontin
uous,andmayormaynotinvolvethemanipulationofauditoryparameters.Forinstance,asubst
an-tialbodyofhumanfactorsresearchrelatingtheparametersofauditorysignalstoperceiv
edurgencyexistsforthedesignofauditorywarnings,whichfallunderthisdesigncategory(s
eeStantonandEdworthy,1999,forareview).Similarly,changesinthecharacterofcontinuou
sorstreamingsoundsallowlistenerstoperipherally
5.4HumanFactorsDesignofanAuditoryInterface171

orpreattentivelymonitorthestateofongoingbackgroundprocesseswhiletheyattendtoothe
rfunctions.Wheneverdesignerscreatesoundsforattentionalpur-poses,theperceptualstr
engthsandweaknessesoftheauditorymaterialsmustbecarefullyevaluatedinthecontextoft
helargertaskenvironment.Inaddition,arangeofrelatednonauditoryhumanfactorsmayalso
needtobeconsidered.Theseincludeinterruptions(McFarlane&Latorella,2002),situation
awareness(Jones&Endsley,2000),timesharing(Wickens&Hollands,2000),andstress(Staal
,2004).WorkingwithSoundDirectlyManyactivitiesinvolveworkingwithsounditselfasinfo
rmation,asamedium,orboth.Inthisdesigncategory,perceptionand/ormanipulationofsoun
datameta-levelisthefocusofthetask.Forexample,ausermayneedtobeabletomonitororrevi
ewaliveorrecordedaudiostreamtoextractinformationorotherwiseannotateauditoryconte
nt.Similarly,soundmaterials,particularlyinmusic,film,andscientificresearch,often
mustbeedited,filtered,orprocessedinspecificways.Interfacesforsonifyingdata,cover
edatlengthpreviously,alsofallunderthisheading.Knowledgeofhumanauditoryperceptual
skills(Bregman,1990)andprocessesinauditorycognition(McAdams&Bigand,1993)arebothi
mpor-tantprerequisitesforthedesignofanytaskthatinvolvesend-userdevelopmentormani
pulationofauditorymaterials.UsingSoundinConjunctionwithOtherDisplayModalitiesSou
ndisperhapsmostfrequentlycalledontocomplementthepresentationofinformationinanoth
ermodality.Althoughhapticinterfacesarebeginningtomakeuseofsound,(see,e.g.,McGook
in&Brewster,2006a),moreoftenthannotsoundisusedinconjunctionwithavisualdisplayofs
omesort.Likeitsfunc-tionintherealworld,soundnotonlyreinforcesandoccasionallydisa
mbiguatestheperceptionofdisplayedeventsbutalsooftenaugmentsthemwithadditionalinf
ormation.Inadditiontopreviouslymentionedhumanfactorsissues,aesthetics(Leplatre&M
cGregor,2004),multisensoryprocessing(Calvert,Spence,&Stein,2004),andthepsycholog
yofmusic(Deutsch,1999)couldberelevantforthesetypesofapplications.UsingSoundasaPr
imaryDisplayModalityInarangeofcontexts,soundmaybethemostversatileortheonlymodeav
ail-ableforrepresentinginformation.Intheseauditoryapplications,auserpopula-tionw
ithitsown,oftenunique,setofinteractiongoalsandexpectationsistargeted.Manyofthehu
manfactorsconsiderationsthatarerelevantforthepre-cedingdesigncategoriescanalsobe
applicablehere,particularlythoserelevanttoworkingwithsoundandthoserelevanttousin
gsoundinlargeroperationalcontexts.
5AuditoryInterfaces172

5.4.4SpecifyingRequirementsforanAuditoryInterfaceAnimportantstepinthedevelopment
processistoturntheproductsofthecon-textualtaskanalysisanditerativeprototypetesti
ngintoacoherentspecification.Dependingonthesizeandscopeoftheproject,formalmethod
s(e.g.,Habrias&Frappier,2006)orasimplesoftwarerequirementsdocumentcanbeusedforth
ispurpose.However,thevalueofthisexerciseanditsimportanceshouldnotbeunderestimate
d.Thespecificationisbothablueprintfortheinterfaceandaroadmapfortheimplementation
process.Inwhateverformittakes,thespecificationshoulddetailhowtheinterfacewillbeo
rganized,howitwillsoundandappear,andhowitwillbehaveinresponsetouserinputwellenou

ghtoprototypeorimplementtheauditorytasktoapointthatissufficientforsubsequentdeve
lopmentandevaluations.Theinterfaceshouldorganizeandpresenttheactionsandgoalsenum
eratedinthetaskanalysisinamannerthatisintuitiveandeasytounderstand.Anyauditoryma
terials,sound-processingspecifications,andexamplesthatweredevelopedfortheauditor
ydesignshouldbereferencedandappendedtothespecificationsdocument.Inmanycases,itwi
llalsobeeffectivetosketchthesoundequivalentofavisuallayoutwithanauditoryprototyp
ingtool.Thedesignershouldtakecaretoensurethatthespecifiedbehavioroftheinterfacei
sorderlyandpredictable,and,becauseaudiocanbeunintentionallytooloud,theinterfacei
deallyshouldcaptheamplitudeofauditorypresentationsandshouldalwaysincludeaclearme
thodfortheusertocancelanyactionatanypoint.Anumberofauditorypro-totypingtools,ran
gingfromsimplesoundeditorstofull-blownapplicationdevel-opmentenvironments,areava
ilablebothcommerciallyandasopen-sourceprojectsthatarefreelyavailableontheInterne
tfordownloadingandpersonaluse.Section5.2.3providedprogrammingexamplesforsonifyin
gdataintheopensourceSuperColliderenvironment(McCartney,1996),andalsomentionedPur
eData,anotherpopularandpowerfulopen-sourceprogrammingenvironmentthatcanbeusedfor
prototypingaudio,video,andgraphicalprocessingapplications(Puckette,1997).Arecent
,commerciallyavailabletoolfordevelopingimmersivevirtualauditoryenvironmentsisVib
eStudio(VRSonic,2007).Last,agoodunderstandingofcurrentauditorydisplaytechnologyi
simpor-tantforachievinggoodhumanfactorsinauditorytasks.Thedesignershouldweighthe
advantagesofcommercialversusopen-sourceaudiosynthesisandsignal-processinglibrari
esandshouldalsogiveattentiontotheimplicationsofvariousaudiofileformatsandrenderi
ngmethodsforauditorytasks.Soundsren-deredbinaurallywithnonindividualizedHRTFs,fo
rinstance,areperceivedbymostlistenerstohavefunctionalspatialpropertiesbutareaccu
ratelylocalizedbyonlyasmallpercentageoflisteners.Technicalknowledgeatthislevelis
integral
5.4HumanFactorsDesignofanAuditoryInterface173

todevelopingeffectivespecificationandensuringthatusabilityconcernsremaincentrald
uringtheiterativestagesofimplementationandformativeevaluationthatfollow.5.4.5Des
ignConsiderationsforAuditoryDisplaysOneofthegoalsofthischapterssectionsonthenatur
e,technology,andcurrentimplementationsofauditoryinterfacesistogivethereaderatang
iblesenseoftheexcitingscopeandrangeofchallengesthatauditoryinformationdesignspos
e,especiallywithregardtohumanfactors.Auditorydesignisstillmoreofanartthanascienc
e,anditisstillverymuchthecasethatthosewhochoosetoimple-mentauditoryinterfacesare
likelytofindtheywillhavetodoabitoftrailblazing.Sections5.2and5.6providedetailedd
evelopmentanddesignguidelinesthatthedesignerandpractitionershouldfindusefulwhend
evelopingsoundsforaudi-toryinterfaces.Becauseauditoryinterfacesusesoundsindiffer
entparadigmsthanmany(indeedmost)peoplearefamiliarwith,thecurrentsectionisdevoted
toprovidingthereaderwithawayofthinkingaboutsoundfromaninformationalperspectivean
dhowthedesignandproductionofsoundsshouldbeinfluencedbythisdifferentwayofthinking
.ThinkingaboutSoundasInformationIntheprocessofdesigningauditorymaterialstoconvey
task-relatedinformation,itisimportanttokeepinmindanumberofconceptualnotionsabout
soundasinformation.First,soundcanbeusefullycategorizedinanumberofways,non-speech
versusspeech,naturalversussynthetic,nonmusicalversusmusical,andsoon.Listenersgen
erallygraspsuchdistinctionswhentheyareobviousinthecontextofanauditorytask,sothis
sortofpartitioninginanauditorygraphingapplication,forinstancecanbequiteusefulasade
signconstruct.Anothervaluabledesignperspectiveonauditoryinformation,introducedin
Kramer(1994),istheauditoryanalogic/symbolicrepresentationcontinuummentionedatthe
beginningofthechapter.Soundsareanalogicwhentheydisplayrelationshipstheyrepresent
,andsymbolicwhentheydenotewhatisbeingrepre-sented.Muchofthedisplayedsoundinforma
tionthatpeopleregularlyexperiencefallssomewherebetween,andtypicallycombines,thes
etwoideals.TheclassicGeigercounterexamplecanbeunderstoodinthiswaytherateofsounded
clicksisanalogicofthelevelofradiation,whiletheclicksthemselvesaresymbolicofradia
tionevents,whicharesilentintherealworld.Theanalogic/symbolicdistinctioncanalsobe
usefullyconceptualizedintermsofsemiotics(i.e.,thestudyofsymbolsandtheiruseorinte
rpretation).Whensoundisdesignedtoconveyinformationtoalistener,theintendedresult,

ifsuccessful,isanindex,anicon,and/orasymbol(e.g.,seeClark,1996).Indicesworkbydir
ectinglistenersattentiontotheinformationthattheyareintended
5AuditoryInterfaces174

tosignal,andiconsworkbyaurallyresemblingordemonstratingtheinforma-tiontheyaremea
nttoconvey.Bothofthesetypesofsoundsfunctioninanana-logicmanner.Aremarkableexampl
eofanauditorysignalthatisbothindexicalandiconicisaversionofUlfvengrens(2003)slurp,w
hich,whenrenderedspatiallyinanairplanecockpit,isintendedtodrawthepilotsattentiont
oandresemblealow-fuelcondition.Notehowtheinformationaluseofthissoundinthiscontex
talsomakesitasymbol.Soundsthataresymbolsworkbyrelyingonanassociativeruleorconven
tionthatisknownbythelistener.Outsideofacockpit,mostlistenerswouldnotreadilyassoc
iateaslurpingsoundwiththeirsupplyoffuel!Afinalmeta-levelaspectofsoundasinformati
onthatshouldbekeptinmindastheauditorydesignprocessbeginsispeoplesexperienceandfam
iliaritywiththemeaningsofeverydayandnaturallyoccurringsounds(Ballas,1993).Muchof
theinformationconveyedbytheseclassesofsoundsisnotintentionallysig-naledbutisaper
ceptualby-productofactivityintherealworld,andisunderstoodassuch.Footsteps,thesou
ndofrain,thewhineofajet,thegrowlofadogallhavecontextualmeaningsanddimensionsthatl
istenersreadilycompre-hendandmakesenseofonthebasisoflifelongexperienceandnativel
isteningskills.Theinherenteaseofthisperceptualfacilitysuggestsanimportantrangeof
strategiesforauditorydesignerstoexplore.FitchandKramer(1994)usedanalogsofnatural
soundsinasuccessfulpatient-monitoringapplicationtorenderconcurrent,self-labeling
auditorystreamsofphys-iologicaldata.EvenmoreinnovativeisworkbyHermannetal.(2006)
inwhichpathologicalfeaturesinEEGdatathatarediagnosticofseizuresarerenderedwithrh
ythmsandtimbrescharacteristicofhumanvocalizations.Incontrast,soundsthatareunusua
lornovelforlisteners,includingmanysyntheticandeditedsoundsaswellasmusic,haveanim
portantplaceinauditorydesign,primarilyfortheirpotentialforcontextualsalienceand,
inmanycases,theirlackofidentityinothersettings.Ingeneral,though,unfamiliarusesof
soundsrequiremoretrainingforlisteners.DesigningtheSoundTurningnowtopractice,once
theinformationtobeconveyedbysoundhasbeenanalyzed,thedesignershouldbegintheproces
sofselectingand/ordevelopingappropriatesounds.Ausefulperspectiveonthedesignprobl
ematthispointistothinkoftheauditorycontentofaninterfaceasakindofsoundecology(Wal
ker&Kramer,2004).Ideally,theinterfaceshouldbecompelling,inventive,andcoherentitsh
ouldtellakindofstoryandthesoundsitemploysshouldhaveacollectiveidentitythatlistene
rswillhavelittledifficultyrecognizing,inmuchthesamewaythatpeopleeffortlesslyreco
gnizefamiliarvoices,music,andthecharacteristicsoundsoftheirdailyenvironments.Goo
dauditorydesignpracticeinvolvescriticallistening(toboththeusersofthesoundsandthe
sounds
5.4HumanFactorsDesignofanAuditoryInterface175

themselves!)andstrivesabovealltoaccommodatetheauralskills,expectations,andsensib
ilitiesthatlistenersordinarilypossess.Itiseasierthanmanypeoplemightthinktocreate
anauditoryinterfacethatisunintentionallytiresomeorinter-nallyinconsistentorthatr
equiresextensivetrainingorspeciallisteningskills.Oncesomeinitialthoughthasbeengi
ventotheorganizationandcharacterofthelisteningenvironment,thefirstcomponentofthe
auditorydesignprocessistoworkouthowsoundwillbeusedtoconveythetask-relatedinforma
tionthatisidentifiedinthetaskanalysis.Often,itisalsousefultobegindevelopingcandi
datesoundsfortheinterfaceatthistime,becausethiscanhelptocrystallizeideasaboutthe
design;however,thismaynotalwaysbepossible.Themappingfrominformationtosoundshould
,inmanycases,berelativelystraightforward,butinothercasesforinstance,withcomplexda
tarelationsitwillgenerallybenecessarytoexperimentwithanumberofideas.Severalexampl
esfollow:FEventonsetsintuitivelymaptosoundonsets.FLevelofpriorityorurgencycanber
epresentedsystematicallywithavarietyofparameters,includingrhythm,tempo,pitch,and

harmoniccomplexity(e.g.,Guillaume,Pellieux,Chastres,&Drake,2003).FDrawingattenti
onto,orindexing,aspecificlocationinspaceaformofdeixis(Ballas,1994)canbeaccomplishe
dwiththree-dimensionalaudio-renderingtechniques.FEmotionalcontextcanbeconveyedwi
thmusicormusicalidioms.FDistinctsubclassesofinformationcanbemappedtodifferenttim
bres;rangescanbemappedtolinearlyvaryingparameters.FPeriodicitycanbemappedtorhyth
m.Manymoreexamplescouldbegiven.Often,therewillbemorethanonedimen-siontoconveyabo
utaparticularpieceofinformationandinsuchinstancesauditoryparametersarefrequently
combined.Anauditoryalert,forexample,canpackonset,eventidentity,location,level(s)
ofurgency,duration,andconfirmationofresponseintoasingleinstanceofsound(Brock,Bal
las,Stroup,&McClimens,2004).ProducingtheSoundAsmappingsandcandidatesoundsforthei
nterfacearedeveloped,anotherfactortheauditorydesignermustaddressishowthefinalsou
ndswillbeproduced,processed,andrendered.Althoughanintroductiontothetechnologyofs
oundproductionwasgiveninSection5.2,theemphasistherewasprimarilyoncompu-tationalt
echniquesforthesynthesisofsound.Othermeansofsoundproductionincludelivesourcesand
playbackofrecordedand/oreditedmaterial.Inaddition,manyauditoryapplicationsrequir
esoundstobelocalizedforthelistener,usuallywithbinauralfilteringorsomeformoflouds
peakerpanning.Andsometasksalloworrequiretheusertocontrol,manipulate,assign,orcho
oseaportionorallofitsauditorycontent.
5AuditoryInterfaces176

Consequently,vettinganauditorydesigntoensurethatitsdisplayimplemen-tationwillfun
ctionasintendedcanrangefromassemblingafixedsetofaudiofilesforamodestdesktopappli
cationtospecifyingasetofaudiosourcesandpro-cessingrequirementsthatcanhavesubstan
tialimplicationsforanapplicationssupportingcomputationalarchitecture.Inthelatters
ituation,onewouldreason-ablyexpecttobepartofacollaborativeprojectinvolvinganumbe
rofspecialistsandpossiblyotherdesigners.Choosingbetweenoneconstructandanotherinr
ichapplicationdomainsandknowingwhatisnecessarytoormostlikelytomeettheusersneedsis
notalwaysamatterofjustknowingorgoingtotheliterature.Alloftheseadvancedconsid-era
tions,howevertheproduction,filtering,augmentation,timing,andmixingofvarioustypesa
ndsourcesofsoundareproperlypartoftheauditorydesignandshouldbeidentifiedasearlyasp
ossibleinacomplexdesignprojectbecauseoftheirimplicationsforthesubsequentimplemen
tationandevaluationphasesoftheauditoryinterface.5.4.6IterativeEvaluationThefinal
andindispensablecomponentoftheauditorydesignprocessisforma-tiveevaluationviauser
testing(Hix&Hartson,1993).Targetedlisteningstudieswithcandidatesounds,contextual
mock-ups,orprototypesoftheauditoryinter-face,designedtodemonstrateorrefutetheeff
icacyofthedesignoritsconstituentparts,shouldbecarriedouttoinformandrefineiterati
vedesignactivities.Formoreonevaluation,seeSection5.5.5.5TECHNIQUESFORTESTINGTHEI
NTERFACEAswitheveryinteractivesystem,theevaluationofauditorydisplaysshouldensure
ahighdegreeofusability.Forauditorydisplays,findingmethodstoevaluateusabilityisno
tatrivialtask.Thefollowingsectionshighlightsomeofthespecificapproachesandissuesr
elevanttotheevaluationofauditorydisplays.5.5.1IssuesSpecifictoEvaluationofAudito
ryDisplaysFromprototypingtodatacapture,anumberofissuesuniquetodisplaysthatusesou
ndneedtobeconsideredwhenevaluatinganauditorydisplay.EarlyPrototypingTherearefewg
enerallyavailabletoolsfortheearlyprototypingofconceptsinaudi-torydisplays,butthe
desirabilityofobtainingearlyfeedbackonauditoryinterface
5.5TechniquesforTestingtheInterface177

designsis,ifanything,evenmoreimportantthanonprototypingvisualdisplaysbecausemany
usersarerelativelyunfamiliarwiththeuseofaudio.WizardofOztechniques(Dix,Finlay,Ab
owd,&Beale,2004)andtheuseoflibrariesofsoundsavailableontheInternet(FindSounds,20
06)canprovidethebasisofwaysaroundthisdilemma.Thequalityandamplitudeofthesoundsem
ployedinprototypesmustbeclosetothoseanticipatedinthefinalsysteminordertodrawconc

lusionsabouthowwelltheyarelikelytoworkincontext.Forinstance,researchbyBallas(199
3)showsthatthewayanindividualsoundisinterpretedisaffectedbythesoundsheardbeforea
ndafterit,soaccuratesimulationofthepaceoftheinteractionisalsoimportant.Oneearlystagetech-niqueistovocalizewhattheinterfaceisexpectedtosoundlike;forexample,Herm
annetal.(2006)usedsuchatechniquetodevelopsonificationsofEEGdata.Bymakingitpossib
leforpeopletoreproducethesonifications,userswereabletomoreeasilydiscusswhattheyh
eardandthesediscussionsfacilitatedtheiterativetestingprocess.ContextofUseEvaluat
ionofthedisplayintheenvironmentwherethesystemwillbeusedandinthecontextofusersper
formingnormaltasksisparticularlyimportantforauditorydisplays.Theprimaryfactorsas
sociatedwithcontextareprivacyandambientnoise.Forinstance,althoughthetaskanalysis
(describedinSection5.4.1)mayhavedeterminedthatprivacyisnecessaryandthusthatheadp
honesarethebestdeliverymethodforthesounds,anevaluationofthedisplayintheenviron-m
entmaydeterminethatwearingheadphonesinterfereswiththeuserstask.Conversely,ifpriva
cyisnotanissueandspeakersarebeingusedfortheauditorydisplay,anevaluationofthedisp
layinthecontextwhereitwillbeusedcoulddeterminetheappropriateplacementandpowerofs
peakers.CognitiveLoadIfanauditorydisplayisbeingusedtoreducecognitiveload,theeval
uationpro-cessshouldconfirmthatloadreductionoccurs.Onewaytomeasurecognitiveloadi
sHartandStavelands(1988)NASATaskLoadIndex(TLX).Ifthismeasureisnotsensitiveenoughf
orthetasksassociatedwiththeauditorydisplay,itmaybebettertouseaccuracyand/ortimep
erformancemeasuresasindirectmeasuresofwhetherthedisplayhasreducedthecognitiveloa
d.ChoiceofParticipantsWhenconductingevaluationsofauditorydisplays,aswithallevalu
ationsofalltypesofdisplays,thecharacteristicsoftheparticipantsshouldmatchthoseof
theintendedusersascloselyaspossible.Someoftheobviousvariablesthatshouldbeconside
redaregender,age,andexperiencewithinformationtechnol-ogy.Furthermore,evaluatorsa
lsoneedtomatchparticipantsonthespecificvariablesassociatedwithaudition(listedinS
ection5.4.1).
5AuditoryInterfaces178

Whenconductingevaluations,therearealsodangersofmakingfalseassump-tionsconcerning
theapplicabilityofevaluationdataacrossdifferentusertypes.Forexample,inthedevelop
mentofsystemsforvisuallyimpairedusers,itisnotunusualforsighteduserswhohavehadthe
irviewofthedisplayobscuredinsomewaytobeinvolvedintheevaluation.However,inastudyi
nvolvingjudgmentsconcerningtherealismofsoundsandsoundmappings,PetrieandMorley(19
98)concludedthatthefindingsfromsightedparticipantsimaginingthemselvestobeblindco
uldnotbeusedasasubstitutefordatafromparticipantswhowereactuallyblind.Finally,eva
luatorsneedtobeparticularlydiligentaboutdeterminingwhetherparticipantshaveanyhea
ringloss.Obviously,hearinglossesarelikelytoimpactparticipantsinteractionswiththes
ystemandthusshouldbetakenintoaccount.DataCaptureAsmightbeexpected,problemscanari
sewiththeuseofthink-aloudprotocolsforcapturingtheresultsofformativeevaluationsof
auditorydisplays.Participantsarelikelytoexperienceproblemswhenaskedtoarticulatet
heirthoughtswhileatthesametimetryingtolistentothenextaudioresponsefromtheinterfa
ce.Thisisnottoruleouttheuseofthink-aloudprotocolsaltogether.Forexample,Walkerand
Stamper(2005)describe,intheevaluationofMobileAudioDesigns(MAD)Monkey,anaudio-aug
mentedrealitydesignerstool,thattheremaybesituationswheretheaudiooutputisintermitt
entandallowssufficienttimeforparticipantstoarticulatetheirthoughtsinbetweenaudio
outputfromthesystem.Anotheralter-nativewouldbetousewhatiscommonlyknownasaretrosp
ectivethink-aloudpro-tocol,inwhichparticipantsdescribethethoughtstheyhadwhenusin
gthesystemtotheevaluatorwhilereviewingrecordingsoftheevaluationsession.LearningE
ffectsImprovementinperformanceovertimeislikelytobeimportantinmostsystems,butther
eiscurrentlylittleknownaboutlearningeffectsobservedinusersofaudi-torydisplays,ot
herthanthefactthattheyarepresentandneedtobeaccountedfor.Inexperimentsconductedby
severaloftheauthors,significantlearningeffectshavefrequentlybeenseenearlyintheus
eofauditorydisplaysasuserstransitionfromneverhavingusedacomputer-basedauditorydi
splaytogainingsomefamiliarityinreactingtothedisplay.Forinstance,astudybyWalkeran
dLindsay(2004)ofawearablesystemforaudio-basednavigationconcludedthatpracticehasam
ajoreffectonperformance,whichisnotsurprising,giventhatnoneoftheparticipantshadex

periencedanauditoryway-findingsystembefore.Thusitiscriticaltoexamineperformancel
ongitudinallywhenevaluatingaudi-torydisplaydesigns.HeuristicEvaluationsItisoneoft
hegreatadvantagesofsoundthattheauditorycuesemployedcanbedesignedtobebackgroundno
ises;hence,auditorydisplaysareoftenusedas
5.5TechniquesforTestingtheInterface179

ambient displays. Mankoff and colleagues (2003) developed heuristics for revealing usability issues in such ambient displays. Heuristic evaluation of user inte
r-faces is a popular method because it comes at very low cost. For example,Niels
en and Molich (1990) found that a panel of three to five novice evaluatorscould
identify 40 to 60 percent of known issues when applying heuristic evalua-tion. H
owever, doubts have been expressed about the results of some studiesinvestigatin
g its effectiveness and some usability professionals argue that heuristicevaluat
ion is a poor predictor of actual user experience (e.g., see http://www.usabilit
ynews.com/news/article2477.asp ).5.5.2ExampleofaCross-ModalCollaborativeDisplay:
TowersofHanoiOnecommonconceptinaccessibilityisthat,givenacollaborativesituationb
etweensightedandvisuallyimpairedusers,thedifferenceininteractiondevicescancausep
roblemswiththeinteraction.WinbergandBowers(2004)developedaTowersofHanoigamewithb
othagraphicalandanaudiointerfacetoinvestigatecollaborationbetweensightedandnonsi
ghtedworkers.Toeliminateanypro-blemsassociatedwithhavingdifferentdevicesforsight
edandblindusers,bothinteractionsweremousebased,employingafocusfeaturetoenablethe
mousetotrackthecursor.Thesightedplayerworkedwithascreen,andtheblindonehadheadpho
nes.Inordertoencouragecollaboration,therewasonlyonecursorforthetwomice.TestingSe
tupTokeepthingsequal,neitherplayerhadaccesstotheothersinterface.Thesightedplayerh
adascreenandcouldseetheblindparticipantbutnothisorhermousemovement;bothcouldhear
eachother,asthiswasnecessaryforcollaboration.Eachplayerwastrainedindependentlyan
dhadnoknowledgeoftheothersinterface.EvaluationTheentireinteractionbetweenthetwopl
ayerswasvideotapedandthegamewindowwasrecorded.ThevideoenabledWinbergandBowers(20
04)tostudytheentireinteraction,andthescreencaptureallowedthemtoseethestateoftheg
ameatalltimes.Theplayersplayedthreegameswiththree,four,andfivedisks,respectively
.AnalysisInthisexperimentoncross-modalcollaboration,WinbergandBowers(2004)stud-i
edthefollowingaspectsoftheinteraction:turntaking,listeningwhilemoving,monitoring
theothersmove,turn-takingproblemsandrepair,reorientationand
5AuditoryInterfaces180

re-establishingsense,engagement,memoryandtalk,anddisengagement.Themajormethoduse
dtoevaluatetheinteractionwasconversationanalysis(tenHave,1999).Thetranscriptiona
ndsubsequentstudyoftheconversationpairedwiththeplayersactionsprovidedanin-depthqu
alitativeanalysisofproblemsintheinterac-tion.Anyproblemswiththeinterfacesbecamea
pparentfromstumblingandconfu-sionintheconversation.Actionswerealsotimed,andthish
elpedtopinpointproblemswiththedirectmanipulationintheauditoryinterface.Conclusio
nsThesystemdevelopedandevaluatedbyWinbergandBowers(2004)enabledtheexaminationofs
omebasicissuesconcerningthecooperationbetweenpeopleofdifferentphysicalabilitiess
upportedbyinterfacesindifferentmodalities.Theyconcludedthatsonicinterfacescouldb
edesignedtoenableblindparticipantstocollaborateonthesharedgame:Intheirevaluation
,allpairscompletedallgames.Theauditoryinterfaceenabledblindplayerstosmoothlyinte
rleavetheirtalkandinteractionswiththeinterface.Theprincipleofcontinuouspresentat
ionofinter-faceelementsemployedinthegameallowedblindplayerstomonitorthestateofth
egameinresponsetomovesastheyweremade.Thisenabledtheblindplayertoparticipatefully
intheworkingdivisionoflabor.Bothblindandsightedcollaboratorsthereforehadresource
stomonitoreachothersconductandhelpeachotheroutifrequired.However,problemswereseen
whentheblindplayersstoppedmanipulatingthedisplayandlisteningtotheconsequentchang

es.Inthesesituationsthestateofthegamebecameuncleartotheblindplayersanddifficulti
eswereexperiencedinreestablishingtheirunderstandingofthecur-rentstateofthegame.I
mportantfindingsresultingfromthestudybyWinbergandBowers(2004)canbesummarizedasfo
llows:1.Themanipulabilityofanassistiveinterfaceiscritical,notonlyforthepurposeof
completingtasksbutalsotoenableacross-modalunderstandingofthesys-temstatetobeesta
blished.Thisunderstandingcanbecomecompromisedifthelinkageamonggesture,sound,ands
ystemstatebecomesunreliable.2.Whendecidingwhethertoimplementfunctionalityinsound
,theavailabilityofotherchannelsofcommunicationandtheappropriatenessofaudioforrep
re-sentingthefunctionshouldbekeptinmind.Forexample,therecouldbesitua-tionswheret
hesonificationofaninterfaceartifactmaysimplytaketoolongandmaybebetterreplacedbyt
alkbetweencollaborators.3.Itisnotenoughtodesignanassistiveauditoryinterfacesotha
titfacilitatesthedevelopmentofthesamementalmodelastheinterfaceusedbysightedindiv
iduals.Additionally,itisessentialtoexaminehowtheassistiveinter-facewillbeusedand
howthisusageisintegratedwiththevariousthingsthatparticipantsdo,suchasdesigningges
tures,monitoringeachother,establishingthestateofthingsandonesorientationinit,reas
oningand
5.5TechniquesforTestingtheInterface181

describing(WinbergandBowers,2004).Inordertodothiseffectively,itbecomesessentialto
studyhowpeopleuseassistiveinterfacesincollaborativesituations.5.6DESIGNGUIDELINE
SAlthoughthepotentialofaudioasaninteractionmodalityinHCIishighandmanyapplication
shaveshownthis(e.g.,Brewster,2002),theefficientdesignofaudioremainssomethingofam
ysteriousprocessandguidanceisoftenscarce.Hence,inthissectionweattempttodescribee
xistingguidelines,principles,anddesigntheories.5.6.1AnalysisandRequirementSpecif
icationInauditorydesign,someaspectsoftherequirementspecificationsdemandspecialat
tention.AsmentionedinSection5.4.1,itisimportanttohaveaclearunder-standingoftheus
erslisteningbackgroundandabilities.InadditiontotheissueslistedinSection5.4.1,desi
gnersshouldconsiderusersopennesstoanalternativedisplaymodality.Audioaspartofhumant
echnologyinteractionisacompara-tivelynewfield,andthususershavelittleexperiencewi
thusingit.Thisalsomeansthatdesignersmayencounterskepticismandprejudiceagainstusi
ngaudio,notonlyfromusersbutfromallstakeholdersinthedesignprocess.Althoughtheremi
ghtbestrongargumentsforusingaudioinaspecificapplica-tion,aclientmightstillreques
tavisualsolutionbecauseheorshecannotimagineanauditorysolution.Avaluableconceptfo
rauditoryinformationdesigninthisearlyphasewasproposedbyBarrass(1997)andiscalledTa
Daanalysis.Itisamethodfordescribingthetaskandthedatatoberepresentedinaformalwayin
cludingastoryabouttheusageandpropertiesthatareindicatorsforauditorycueslikeatten
tionlevelsordatatypes.Barrass(1997)usedtheseTaDadescriptionsasastartingpointfort
heselectionofsounds,andthenemployedthesedescriptionstomatchthemwithsoundsstoredi
nadatabase(EarBender).TheEarBenderdatabasecontainsalargenumberofsoundstaggedwith
semanticandotherpropertiesthatcanbematchedwiththerequirementsfromaTaDaanalysis(B
arrass,1997).Theauthoralsoproposedthecreationofauditorydesignprinciplesbasedonpr
inciplesforgenericinformationdesign,suchasdirectnessortheleveloforganization.Bar
rass(1997)linkedthesewiththeprop-ertiesofauditoryperceptiontocreateauditorydesig
nprinciples.TheTaDatechniquehasbeenusedbyanumberofauditorydesigners.Nota-bly,att
heSciencebyEarworkshopthattookplaceattheInstituteofElectronicMusic(IEM)inGrazin2
006,thetechniquewasusedtoformalizerequirements
5AuditoryInterfaces182

foranumberofcasestudiesforwhichmultidisciplinaryteamswereformedtodesigndatasonif
ications.Thecasestudiesincludeddatadrawnfromparticlephysics,electricalpowersyste
ms,EEGdata,globalsocialdata,andrainfalldata.TheTaDatechniqueprovedhelpfulinprovi
dingastandardformatforrepresent-ingtheinformationrequirementsofeachsonificationt

obedesignedforusebythemultidisciplinaryteams.Examplesofthecasestudiesemployedatt
hework-shopcanbefoundinthepapersbydeCampo(2007)andotherspresentedattheInternatio
nalConferenceonAuditoryDisplays(ICAD,2007).Ofcourse,othermethodsandguidelinescan
andshouldbeappliedattherequirementspecificationstageofthedesign.Taskanalysis,use
rscenarios,perso-nae,andotherconceptshavebeensuccessfullyappliedtovisualdesignan
ddonotinvolvetheneedtospecifyanyinteractionmodality.Exampleswherethesecon-ceptsh
avebeenappliedinauditorydesignsaretheuseofrichuserscenariosinthedesignofanaudito
rywebbrowser(Pirhonen,Murphy,McAllister,&Yu,2006)andthefirststageinthedesignmeth
odologyproposedbyMitsopoulos(2000).Bothapproachesareelaboratedinthenextsection.5
.6.2ConceptDesignConceptdesigniswhenhigh-leveldesigndecisionsaremadewhileleaving
mostdetailsstillunspecified.Thisphaselinksthedesignproblemwithconceptsofauditory
displays.Thefirstandforemosttaskinthisphaseistodecidewhichpartsoftheuserinterfac
eaudiowillbeusedandwhichauditoryconceptsmatchtherequirementsandconstraintsdefine
dintherequirementsphase.Brewster(1994)addressedthisissueinabottom-upapproach:Fin
derrorsinindividualpartsofanexistinginterfaceandtrytofixthembytheadditionofsound
.Headoptedtheeventandstatusanalysisandextendedittobeapplicabletodifferentinterac
tionmodalities(i.e.,toaccommodateaudio).Thiswasanengi-neeringapproachtorevealinf
ormationhiddeninaninterfacethatcouldcauseerrors.Brewster(1994)suggestedusingsoun
dtomakethisinformationaccessibleandlinkedtheoutputoftheanalysistohisguidelinesfo
rthecreationofearcons.Asmentionedintheprevioussection,Pirhonenetal.(2006)propose
dadesignmethodthatlinkedusertasksandauditorycuesthroughrichusecasescenarios.Theu
secasewasdevelopedwithavirtualpersonathatrepresentedthetargetgroupandtoldthestor
yofhowthispersonacarriedoutaspecifictask.Thestorywasenrichedwithasmuchdetailabou
ttheenvironmentandtheback-groundoftheuseraspossibletocreateacompellingscenario;t
heauthorspro-posedthattheusescenarioshouldhavequalitiesthatenabletheinterpreter(t
o)identifyhim/herselfwiththecharacter(Pirhonenetal.,2006:136).Thenapaneloffourtof
ivedesignerswentthroughthisscenarioandtriedtoproducedescriptionsofsoundstosuppor
tthetask.Aftercreatingthesoundsassuggestedbythedesigners,anotherpanelwasorganize
dandwentthroughtheusecase
5.6DesignGuidelines183

scenariothatwasenrichedbytheinitialsounddesigns.Thisprocedurewasiter-ateduntilaw
orkingdesignwasidentified.Themethodstressedtheimportanceoflinkingusertaskswithth
edesign,butalsoreliedheavilyontheavailabilityofexpertdesignersforapanelandonthei
rexperienceandideas.Anothertoolforconceptdesignisdesignpatterns(Frauenberger,Hol
drich&deCampo,2004).However,therearenotyetenoughsuccessfulimplementationsormetho
dologicalframeworksforauditorydisplaysthatincorporatedesignpat-ternsintheauditor
ydesignprocess.Nevertheless,thistoolmostlikelywillprovebeneficialinthefuture.Ano
ther,moretheoreticalapproachhasbeenproposedbyMitsopoulos(2000),whofoundedhismeth
odologyonaframeworkfordialogdesign.Mitsopou-lossmethodologyconsistsofthreelevels:
(1)theconceptlevelinwhichthecon-tentoftheinterfaceisspecifiedintermsofsemanticenti
ties,(2)thestructurallevelinwhichsoundsarestructuredovertime,and(3)theimplementa
tionlevelinwhichthephysicalfeaturesofthesoundaredetermined.Mitsopoulos(2000)prop
osedguidelinesforeachoftheselevelsthatarederivedfromtheoriesofaudi-toryperceptio
nandattention(e.g.,Arons,1992;Bregman,1990).Byapplyingthesetheories,Mitsopoulosi
ntendedtonarrowthedesignspacebyeliminatingdesignsthatwouldviolatepsychologicalpr
inciples.Notablyhearguedfortwofun-damentalmodesofpresentationofinformationbyaudi
o:fastpresentation,thatis,ataglance,andtheinteractivepresentationformoredetailedus
erinteraction.Eachrepresentationisdefinedinallthreelevels.Althoughhismethodology
andguidelinesareproperlyfoundedintheory,itisimportanttonotethattheapproachrequir
esasteeplearningcurve.Ingeneral,decisionsintheconceptdesignphasearecrucialforsuc
cessfulauditorydesign,butthereislittleguidanceavailablethatmayhelpnovicedesigner
s.Itisimportanttonotethatmostflawsinauditorydesignsderivefromdecisionsmadeduring
theconceptphaseasnovicestendtobeoverlyinfluencedbyvisualthinking.Goodauditorydes
ignemphasizesthecharacteristicsandstrengthsofaudioandadoptsvisualconceptsonlyift
hereisevidencethattheyworkintheauditorydomain.5.6.3DetailedDesignManyspecificati

onsresultingfromthepreviousdesignstagedescribethesoundsvaguelyordescribeonlycert
ainproperties.Inthedetaileddesignstage,thesespe-cificationsaremappedontophysical
propertiesofsound.AuditoryDisplay:Sonification,AudificationandAuditoryInterfaces
,editedbyKramer(1994),isoftenseenasalandmarkpublicationinauditorydisplaydesign.T
hevolumecontainstheproceedingsofthefirstmeetingoftheICADin1992,andincludesacompa
nionCDofaudioexamplesillustratingmanyofthepsychologicalphenomena,techniques,anda
pplicationsdiscussed.Several
5AuditoryInterfaces184

chapters in the book present principles for use in representing information inau
dio. The book presents a number of methods for associating perceptual issuesin a
uditory display with techniques for their practical implementation. This workint
roduces fundamental sonification techniques such as the direct representationof
data in sound (or audification), as well as a number of approaches to mappingdat
a variables into a range of sound parameters such as pitch, loudness, timbre,and
tempo. The book also provides an overview of many other relevant issuesin audit
ory display design such as the following:F As would be expected, concurrency is
an issue in auditory display design.Clearly there is a limit to how much auditor
y information human beings canperceive and process concurrently. Nevertheless, c
oncurrency is potentiallya powerful tool in auditory display design, as evidence
d by the ease withwhich even untrained musicians can detect a single instrument
playing outof tune in an entire orchestra of players.F Metaphor, as in the rest
of user interface design, can be an effective mecha-nismfordevelopingandsupporti
ngtheusersmentalmodelofthesystem.Seetheuseofthegroupconversationmetaphortosupport
multitaskingdescribed in the clique case study ( www.beyondthegui.com) as a part
icularlyeffectiveexampleofthis.Muchofthebookfocusesonapplicationsinvolvingthedes
ignofsonificationsofcomplexdata,thatis,applicationsrepresentingeitherrawdataorin
formationtobepresentedinsound.Additionally,thereisagooddealofvaluableguidanceint
hebookforthoseinvolvedinthedesignofmoresymbolicauditoryinterfaceelements.Acontri
butortoAuditoryDisplay,Gaver(1994)providesaclearuserinter-facefocus.Hepresentste
chniquestocreateauditoryiconsforuserinterfacesincomputingsystems.Asmentionedprev
iously,auditoryiconsarebasedonoureverydayhearingexperience;thus,familiarityandin
herentmeaningmakethemhighlyefficientauditorycues.Hence,whencreatingauditoryicons
,theyarenotdescribedintheusualdimensionsofsoundlikepitchortimbre,butratheraccord
-ingtopropertiesofthereal-worldobjectthatcausesthesound.Withregardtodetaildesign
,auditoryiconscanbeparameterizedbydimensionssuchasmate-rial,size,orforce,andwhen
synthesizingauditoryicons,designersseektousealgorithmsthatallowthemtoinfluenceth
eseparametersinsteadofthephysicalpropertiesofthesounddirectly(e.g.,pitch,loudnes
s,etc.).Gaver(1994)providesawiderangeofsuchalgorithmsforimpactsounds,breaking,bo
uncing,andspillingeffects,scraping,andothermachinesounds.Blattneretal.developedg
uidelinesforconstructingearconsbasedonvisualicons(Blattner,Sumikawa,&Greenberg,1
989).Intheseauthorsterminology,representationalearconsaresimilartoauditoryiconsan
darebuiltonmetaphorsandinherentmeaning.Forabstractearcons,theyusemusicalmotifs(a
brief
5.6DesignGuidelines185

successionofideallynotmorethanfourtones)asastartingpoint,anddefinerhythmandpitch
asthefixedparameters.Timbre,register,anddynamicsarethevariableparametersofthemot
if.Bysystematicallyalteringthefixedparameters,designerscouldcreatedistinctiveear
cons,whilealteringthevariableparameterswouldproduceearconswithperceivablesimilar
ityandmaybeusedtocreaterelatedfamiliesofearcons.Intheirguidelines,Blattneretal.(
1989)suggestchoos-ingthetonesaccordingtotheculturalbackgroundofthetargetgroup(e.

g.,Westerntonalmusic),andtheyelaborateonexploitinghierarchicalstructuresinearcon
familiesforbetterlearnability.Suchcompoundearconscanbecreatedthroughcombination,
transformation,andinheritanceofsingle-elementearcons.Inhisworkonguidelinesforcre
atingearcons,Brewster(1994)refinestheguidelinesmentionedpreviouslyandprovidesmor
especificguidanceregardingrhythm,timbre,pitch,amongotherproperties.Keyguidelines
givenbyBrewsterfollow:FUsemusicalinstrumenttimbrestodifferentiatebetweenearconso
rgroupsofearcons,aspeoplecanrecognizeanddifferentiatebetweentimbresrelativelyeas
ily.FDonotusepitchorregisteralonetodifferentiatebetweenearconswhenusersneedtomak
eabsolutejudgmentsconcerningwhattheearconisrepresenting.FIfregistermustbeusedoni
tsown,thereshouldbeadifferenceoftwoorthreeoctavesbetweenearcons.FIfpitchisused,i
tshouldnotbelowerthan125Hzandnothigherthan5kHztoavoidthemaskingoftheearconbyothe
rsounds,andmustbeeasilywithinthehearingrangeofmostusers.FIfusingrhythmtodistingu
ishbetweenearcons,maketherhythmsasdifferentfromeachotheraspossiblebyputtingdiffe
rentnumbersofnotesineachearcon.FIntensity(loudness)shouldnotbeusedtodistinguishb
etweenearcons,asmanyusersfindthisannoying.FKeepearconsshortinordernottoslowdownt
heusersinteractionwiththesystem.FTwoearconsmaybeplayedatthesametimetospeedupthein
teraction.Brewster(1994)alsoinvestigatedtheconcurrentuseofearcons,andMcGookinand
Brewster(2006b)summarizedsomeoftheissueswithusingconcurrentaudiopresentationsina
uditorydisplays.Lumsdenandcolleaguesprovidedguidelinesforamorespecificscenario,t
hatis,theenhancementofGUIwidgets,suchasbut-tons,byearcons(Lumsden,Brewster,Creas
e,&Gray,2002).Althoughadetaileddescriptionofthedesignguidelinesoftheseadditional
considerationsforearconsisbeyondthescopeofthischapter,thestudiesbyBrewsterwithMc
GookinandLumsdenetal.aregoodresourcesfortheearcondesigner.
5AuditoryInterfaces186

For more on auditory information and interface design, a number of excellentreso

urces can easily be found on the Web, including some mentioned in this sec-tion.
De Campo (2007) presents a useful design space map for data sonification,and re
ferences numerous examples that are available at the SonEnvir projectwebsite ( h
ttp://sonenvir.at ). A new online edition of The Handbook for AcousticEcology (T
ruax, 1999) provides an invaluableglossary for acoustic concepts andterminology,
as well as hyperlinks to relevant sound examples. Additionally,a wealth of rese
archpapers and other resources for auditory design aswell asan active online des
ign community can be found at theICAD website.5.7 CASE STUDIESTwocase studies of
auditory interfaces canbe found at www.beyondthegui.com .5.8 FUTURE TRENDSThema
terialpresentedinthischapterillustratesthatauditoryinterfacesareaversatileandeff
icientmeansforcommunicatinginformationbetweencomputersystemsandusers.Listeningis
perhapsthecommunicationchannelwiththehighestbandwidthaftervision,andcertainlyach
annelwhosecharacteristicsaresodifferentfromvisualperceptionthatthecombinationofv
isualandauditorydis-playsinparticularcoversaverywiderangeofinterfacepossibilitie
s.Whereasvisionisafocusedsense(weonlyseewhatwelookat),soundsurroundstheuser;whil
evisionstopsatsurfaces,soundallowsustodiscoverinnerworlds,beyondvisiblelimits;wher
easvisionispersistent,soundisintrinsicallyalignedtochangesovertimeanddynamicinte
raction,andourauditoryskillsareparticularlygoodatdiscerningthesepropertiesinsoun
d.Lookingatauditoryinterfacesfromamoreremovedperspective,weseethattwodirectionsa
repossible:soundasadisplayorasaninputmodality.Thischap-terfocusedonthedisplaymod
e.However,theinputmodecanbeequallycom-pelling.Forexample,aninterfacecouldemployn
onspeechsoundsandvocalizationsasauditoryinputs.Thishaspotentialforthedevelopment
ofaudi-toryinteractionsinvolvingsonification.Ausercouldanalogicallyspecifyorsele
ctthetempoofasonificationbytappingorutteringarhythmintoamicrophone.Similarly,mus
icalidiomscouldbequeriedand/orselectedbyhumming.Anotheravenuefornonsymbolic,voca
lauditoryinputinsonificationsystemsisalreadybeingexplored.Hermannetal.(2006)desc
ribesthevocalsonificationofEEGs,inspiredbytheexceptionalhumancapacityfordiscerni
ngandmemorizingstructureinvocalsounds.Becausehumansareeasilyabletomimicvocal
5.8FutureTrends187

patterns,theycanusevocalizationstoactivelyreferencecertainpatternsinthesonificat
ion.Thisprocessmaysimplifythecommunicationofdatapatterns.Auditoryinterfacesinvol
vinganalogicandsymbolicinputandoutputarelikelytogainrelevanceinfutureinteraction
designforseveralreasons.First,thetechnologicaldevelopmentisjuststartingtoenablet
heseinteractionsatasuf-ficientlevelofsophistication.Second,therearemanysituation
swherethevisualsenseisnotavailableorotherwiseused.Finally,wearesimplywastinganex
cel-lentandhighlydevelopedcommunicationchannelifsoundisneglectedintheuserinterfa
ce.Additionalfuturetrendsthatmaydevelopinauditorydisplaysinclude(1)interactiveso
nificationabetterclosureofinteractionloopsbyinteractivesonifica-tion,and(2)anamal
gamationofauditorydisplaywithothermodalitiessuchasvisualdisplayandtactile/haptic
interfacesthatwouldresultintrulymultimodalinterfaces.Interactivesonification(Her
mann&Hunt,2005)bearsthepotentialtocreateintuitivecontrolofsystemsatalevelbeyonda
rational(logic)analysisofsteps,moreasanintuitive,creative,andsynergeticapproacht
osolvingproblems.Forinstance,indataanalysisviainteractivesonification,discoverin
gpatternswouldturnfromastep-by-stepanalysisintoacontinuousmovementthroughthedata
spaceorsonificationspace.Theuserwouldintegrateanylocallygainedinsightregardingth
estructureofthedataintohisorherexploratoryactivityinacontin-uousway,withoutdisru
ptingtheactivityandexperience.Suchacontinuous,interruption-freemodemaybettercrea
tetheexperienceofflow,thedissolvingofauserintheactivity,whichinturnmaygiveabette
raccesstotheusersoftencoveredcreativepotential.Multimodalinterfaces,ontheotherhan
d,willallowthedesignertocombinethestrengthsofvariousmodalities,sothatthecommunic
ationofdataissimplifiedandachievesabettermatchwiththeusersperceptioncapabilities.
Forinstance,iftheusersvisualfocusisalreadyhighlyused,multimodaldisplayenginescana
utomaticallyselectauditorycomponents,orevenjustemphasizethemagainstthevisualcoun
terpart,sothatthecommunicationbetweentheuserandacomputersystemisoptimized.Anothe
rtrendinauditoryinterfacesthatisgainingmomentumistheconceptofintelligentoradapti
veauditoryenvironmentsanddisplays.Advancesinrender-ing,signalprocessing,usermode
ling,machinelistening(Wang&Brown,2006),andnoninvasiveuserandcontextualmonitoring
technologiesmeanthatintherelativelynearfuturemanyinteractivedevicesandenvironmen
tswilltranspar-entlyadapttheaudiocomponentoftheirinformationdisplaystomatchthene
edsofusers,muchlikepeoplerelyoneachotherslisteningskillsinsocialsettingstocoordin
atetheauraldimensionofconversationandothersharedformsofsoundinformation.Threeare
asinwhichadaptivesoundtechnologyisalreadybeingexploredaremobiletelephony,pervasi
vecomputing,andsocialrobotics.Mobilephoneshavearguablybecomethemostcommonauditor
yinterfacepeopleencounterintheirday-to-daylives.Tocompensatefornoiseindynamic
5AuditoryInterfaces188

environments, new wireless headsets are already being marketed that adaptivelyal
ter a mobile phones outgoing and incoming audio signals to improve speechcommunic
ations (e.g., see Mossberg, 2006).As mobile phones move beyond telephony into ar
eas as diverse as Internetaccess, personal entertainment, content creation, and
interaction with so-calledsmart and pervasive computing environments, exciting n
ew opportunities forintelligent auditory presentation behaviors arise. In recent
pervasive-computingresearch, for instance, users intuitively navigated their wa
y to undisclosed out-door locations using a context-dependent, directionally ada
ptive auditory display(Etter & Specht, 2005). The underlying system uses global
positioning data and ageographical information system to infer the mobile users g
eographical context.Navigation cues are then rendered by adaptively panning and
filtering musicselected by the user to correspond with his or her direction of t
ravel.Intelligent, adaptive auditory displays are also expected to be an importa
nttechnology for social and service robots. Recent humanrobot interaction workby
Martinson and Brock (2007) explores several strategies for adaptively improv-ing

a robots presentation of auditory information for users, including user tracking

,ambient noise level monitoring, and mapping of auditory environments.In summary
, auditory interfaces are a rapidly evolving mode for humancomputer interaction,
with a huge potential for better use of peoples communica-tive skills and percept
ual resources.Acknow ledgmentsWe want to acknowledge the primary sources of cont
ribution for each section. Wealso wish to thank the reviewers for their insights
and feedback on the chapter.Introduction: S. Camille PeresSection 5.1: Virginia
Best, Barbara Shinn-Cunningham, and John NeuhoffSection 5.2: Thomas HermannSect
ions 5.3, 5.5, 5.6, and 5.7: Tony Stockman, Louise Valge@ ur Nickerson,and Chris
topher FrauenbergerSection 5.4: Derek Brock and S. Camille PeresSection 5.8: Tho
mas Hermann and Derek BrockREFERENCESAndresen, G. (2002). Playing by ear: Creati
ng blind-accessible games. Gamasutraavailableat http://www.gama sutra.com/resourc
e_guide/20020520/andresen_01.htm .Arons, B. (1992). A review of the cocktail par
ty effect. Journal of the American Voice I/OSociety 12:3550.Audyssey Magazine . (
19962006). See http://www.audiogames.net/page.php ?pagefile=audyssey.
References189

Baier, G., & Hermann, T. (2004). The sonification of rhythms in human electroenc
ephalo-gram. Proceedings of International Conference on Auditory Display , Sydne
y.Baier, G., Hermann, T., Sahle, S., & Ritter, H. (2006). Sonified epileptic rhy
thms. Proceed-ings of International Conference on Auditory Display , London.Ball
as, J. A. (1993). Common factors in the identification of an assortment of brief
everydaysounds. Journal of Experimental Psychology: Human Perception and Perfor
mance19:25067.Ballas, J. A. (1994). Delivery of information through sound. In Kra
mer, G., ed., AuditoryDisplay: Sonification, Audification and Auditory Interface
s. Reading, MA: Addison-Wesley, 7994.Barras, S. (1997). Auditory information desi
gn. PhD diss., Australian National University.Bay Advanced Technologies. (2006).
BAT K-Sonaravailable at http://www.batforblind.co.nz/.Berman, L., & Gallagher, K
. (2006). Listening to program slices. Proceedings of Interna-tional Conference
on Auditory Display , London.Best, V., Gallun, F. J., Carlile, S., & Shinn-Cunni
ngham, B. G. (2007). Binaural interferenceand auditory grouping. Journal of the
Acoustical Society of America 121:107076.Blattner, M. M., Sumikawa, D. A., & Gree
nberg, R. M. (1989). Earcons and icons: Theirstructure and common design princip
les. HumanComputer Interaction 4(1):1144.Blattner, M., Papp, A., & Glinert, E. P.
(1994). Sonic enhancement of two-dimensional gra-phics displays. In Kramer, G.,
ed., Auditory Display: Sonification, Audification, and Audi-tory Interfaces . Re
ading, MA: Addison-Wesley, 44770.Bly, S. (1982). Presenting information in sound.
Proceedings of SIGCHI Conference onHuman Factors in Computing Systems .Bregman,
A. S. (1990). Auditory Scene Analysis: The Perceptual Organization of Sound .Ca
mbridge, MA: MIT Press.Brewster, S. A. (1994). Providing a structured method for
integrating non-speech audio intohumancomputer interfaces. PhD diss., University
of York. Available at http://www.dcs.gla.ac.uk/~stephen/papers/Brewster_thesis.
pdf .Brewster,S.A.(2002).Nonspeechauditoryoutput.InTheHumanComputerInteractionHan
dbook:Fundamentals,EvolvingTechnologiesandEmergingApplicationsArchive.Mahwah,NJ:
Erlbaum,22039.Broadbent,D.E.(1958).PerceptionandCommunication.NewYork:OxfordUnive
rsityPress.Brock,D.,Ballas,J.A.,Stroup,J.L.,&McClimens,B.(2004).Thedesignofmixed
-use,vir-tualauditorydisplays:Recentfindingswithadual-taskparadigm.Proceedingsof
Interna-tionalConferenceonAuditoryDisplay.Sydney.Bronkhorst,A.W.,&Houtgast,T.(19
99).Auditorydistanceperceptioninrooms.Nature397:51720.Bronkhorst,A.W.(2000).Theco
cktailpartyphenomenon:Areviewofresearchonspeechintelligibilityinmultiple-talkerc
onditions.Acustica86:11728.Brown,M.L.,Newsome,S.L.,&Glinert,E.P.(1989).Anexperime
ntintotheuseofaudi-torycuestoreducevisualworkload.ProceedingsofSIGCHIConferenceo
nHumanFac-torsinComputingSystems.Calvert,G.A.,Spence,C.,&Stein,B.E.,eds.(2004).T
heHandbookofMultisensoryPro-cesses.Cambridge,MA:MITPress.
5AuditoryInterfaces190

Carlile, S. (1996). The physical and psychophysical basis of sound localization.

In Carlile, S.,ed., Virtual Auditory Space: Generation and Applications . Austi
n, TX: RG Landes; NewYork: Chapman & Hall (distributor).Clark, H. H. (1996). Usi
ng Language . Cambridge: Cambridg e University Press.Cook, P. R. (2002). Real So
und Synthesis for Interactive Applications . Wellesley, MA: A. K.Peters Ltd.Darw
in, C. J., & Hukin, R. W. (2000). Effectiveness of spatial cues, prosody and tal
ker char-acteristics in selective attention. Journal of the Acoustical Society o
f America 107:97077.de Campo, A. (2007). Toward a sonification design space map.
Proceedings of InternationalConference on Auditory Display , Montreal.Deutsch, D
., ed. (1999). The Psychology of Music , 2nd ed. San Diego: Academic Press.Dix,
A., Finlay, J., Abowd, G. D., & Beale, R. (2004). HumanComputer Interaction , 3rd
ed.London: Prentice Hall Europe.Etter, R., & Specht, M. (2005). Melodious walka
bout: Implicit navigation with contextuali zedpersonal audio contents. Adjunct P
roceedings of Third International Conference on Per-vasive Computing , Munich.Fa
rkas, W. (2006). iSICData Modeling through Musicavailable at http://www.vagueterra
in.net/content/archives/journal03/farkas01.html.Fayyad, U. M., Piatetsky-Shapiro
, G., Smyth, P., & Uthurusamy, R. (1996). Advances inKnowledge Discovery and Dat
a Mining . Cambridge, MA: AAAI/MIT Press.Fin dSou nds. (2 0 06). Se arch t he We
b for Soun ds. H o m e page a vailab le at http :/ / www.f i n d s o u n d s .c o
m / .Fitch, W. T., & Kramer, G. (1994). Sonifying the body electric: Superiorit
y of an auditory dis-play over a visual display in a complex, multivariate syste
m. In Kramer, G., ed., AuditoryDisplay: Sonification, Audification and Auditory
Interfaces . Reading, MA: Addison-Wesley, 30726.Flowers, J. H., Whitwer, L. E., G
rafel, D. C., & Kotan, C. A. (2001). Sonification of dailyweather records: Issue
s of perception, attention and memory in design choices. Proceed-ings of Interna
tional Conference on Auditory Display , Espoo, Finland.Frauenberger, C., Ho ldric
h, R., & de Campo, A. (2004). A generic, semantically based designapproach for s
patial auditory computer displays. Proceedings of International Conferenceon Aud
itory Display , Sydney.Freedom Scientific. (2005). Jaws R for Windowsavailable at
http://www.freedomscientific.com/fs_products/JAWS_HQ.asp.Gaver, W. W. (1994). U
sing and creating auditory icons. In Kramer, G., ed., AuditoryDisplay: Sonificat
ion, Audification and Auditory Interfaces . Reading, MA: Addison-Wesley, 41746.Go
ffin, V., Allauzen, C., Bocchieri, E., Hakkani-Tu r, D., Ljolje, A., Parthasarath
y, S., Rahim, M.,Riccardi, G., & Saraclar, M. (2005). The AT&T WATSON speech rec
ognizer. Proceedings ofIEEE International Conference on Acoustics, Speech and Si
gnal Processing, P hi la de l p hi a.Guillaume, A., Pellieux, L., Chastres, V.,
& Drake, C. (2003). Judging the urgency of non-vocal auditory warning signals: P
erceptual and cognitive processes. Journal of Experi-mentalPsychology9(3):196212.
GW Micro. (2006). Window-Eyesavailable at http://www.gw micro.com/Window-Eyes/ .H
abrias,H.,andFrappier,M.(2006).SoftwareSpecificationMethods:AnOverviewUsingaCase
Study.London:ISTEPublishing.
References191

Hart, S. G., and Staveland, L. E. (1988). Development of NASA TLX (Task Load Ind
ex):Results of empirical and theoretical research. In Hancock, P. A., & Meshkati
, N., eds.,Human Mental Workload . Amsterdam: North-Holland, 13983.Hayward, C. (1
994). Listening to the earth sing. In Kramer, G., ed., Auditory Display:Sonifica
tion, Audification and Auditory Interfaces . Reading, MA: Addison-Wesley,369404.H
ermann, T. (2002). Sonification for exploratory data analysis. PhD diss., Bielef
eldUniversity.Hermann, T., & Hunt, A. (2005). An introduction to interactive son
ific ation. IEEE Multi-media 12(2):2024.Hermann, T., Bai er, G ., Step hani, U .,
& Ritter, H. (2006). Vocal sonification of patho-logic EEG features. Proceeding

s of Internati onal Conference on Auditory Display,Lond on.Hermann, T., Drees, J

. M., & Ritter, H. (2005). Broadcasting auditory weather reportsapilot project. P
roceedings of International Conference on Auditory Display , Boston.Hix, D., & H
artson, H. R. (1993). Formative evaluation: Ensuring usability in user inter-fac
es. In Bass, L., & Dewan, P., eds., User Interface Software . Chichester, Englan
d; NewYork: Wiley.Hunt, A., & Hermann, T. (2004). The importance of interaction
in sonification. ProceedingsofInternationalConferenceonAuditoryDisplay,Sydney.In
ternational Community for Auditory Display (ICAD). (2007). Home page http://www.i
cad.org/ .Janata,P.,&Childs,E.(2004).Marketbuzz:Sonificationofreal-timefinancial
data.Proceed-ingsofInternationalConferenceonAuditoryDisplay,Sydney.Jones,D.G.,an
dEndsley,M.R.(2000).Overcomingrepresentationalerrorsincomplexenvironments.HumanF
actors42(3):36778.KiddJr.,G.,Mason,C.R.,&Arbogast,T.L.(2002).Similarity,uncertain
ty,andmaskingintheidentificationofnonspeechauditorypatterns.JournaloftheAcoustic
alSocietyofAmerica111(3):136776.Kramer,G.,ed.(1994).AuditoryDisplay:Sonification,
AudificationandAuditoryInterfaces.Reading,MA:Addison-Wesley.Leplatre,G.,&McGrego
r,I.(2004).Howtotackleauditoryinterfaceaesthetics?Discussionandcasestudy.Proceed
ingsofInternationalConferenceonAuditoryDisplay,Sydney.Lieder,C.N.(2004).DigitalA
udioWorkstation.NewYork:McGraw-Hill.Lumsden,J.,Brewster,S.A.,Crease,M.,&Gray,P.D
.(2002).Guidelinesforaudio-enhancementofgraphicaluserinterfacewidgets.Proceeding
sofPeopleandComputersXVIMemorableYetInvisible:HumanComputerInteraction,London.Man
koff,J.,Dey,A.K.,Hsieh,G.,Kientz,J.,Ames,M.,&Lederer,S.(2003).Heuristiceval-uati
onofambientdisplays.ProceedingsofSIGCHIConferenceonHumanFactorsinComputingSystem
s,FortLauderdale.Martinson,E.,&Brock,D.(2007).Improvinghumanrobotinteractionthrou
ghadaptationtotheauditoryscene.ProceedingsofInternationalConferenceonHumanRobotIn
terac-tion,Washington,DC.Massof,R.W.(2003).Auditoryassistivedevicesfortheblind.P
roceedingsofInternationalConferenceonAuditoryDisplay,Boston.
5AuditoryInterfaces192

McAdams, S., & Bigand, E., eds. (1993). Thinking in Sound: The Cognitive Psychol
ogy ofHuman Audition . Oxford; New York: Oxford University Press.McCartney, J. (
1996). SuperCollider Hubavailable at http://supercollider.sourcefor ge.net.McFarl
ane, D. C., & Latorella, K. A. (2002). The scope and importanc e of human interr
up-tion in HCI design. HumanComputer Interaction 17(1):161.McGookin, D. K., & Brew
ster, S. A. (2006a). Haptic and audio interaction design. First Inter-national W
orkshop, HAID 2006, Glasgow. Proceedings (Lecture Notes in Computer Sci-ence) .
New York: Springer-Verlag.McGookin, D. K., & Brewster, S. A. (2006b). Advantages
and issues with concurrent audiopresentation as part of an auditory display. Pr
oceedings of International Conference onAuditory Displays , London.Mills, A. W.
(1958). On the minimum audible angle. Journal of the Acoustical Society ofAmeric
a 30(4):23746.Miranda, E. R. (1998). Computer Sound Synthesis for the Electronic
Musi cian.Oxford:FocalPress.Mitsopoulos, E. N. (2000). a principled approach to
the design of auditory interaction in thenon-visual user interface. PhD diss., U
niversity of York.Moore, B. C. J. (2003). An Introduction to the Psychology of H
earing, 5th ed. London: Aca-demic Press.Moore, F. R. (1990). Elements of Compute
r Music . Upper Saddle River, NJ: Prentice Hall.Mossberg, W. (2006). New earphon
e devices let you go cordless on iPods, cellphones. Per-sonal Technology. The Wa
ll Street Journal Online , December 21; available at http://ptech.wsj.com/archiv
e/ptech20061221.html .Mynatt, E. (1995). Transforming graphical interfaces into a
uditory interfaces. PhD diss.,Georgia Institute of Technology, Atlanta.Neisser,
U. (1967). Cognitive Psychology . New York: Appleton-Century-Crofts.Neuhoff, J.
(2004). Interacting auditory dimensions. In J. Neu hoff, ed., Ecological Psychoacoustics. Boston: Elsevier/Academic Press.Nielsen, J. (1994). Ten usability heu
risticsavailable at http://www.useit.com/papers/heuristic/heuristic_list.htm .Nie
lsen,J.,&Molich,R.(1990).Heuristicevaluationofuserinterfaces.ProceedingsofSIGCHI
ConferenceonHumanFactorsinComputingSystems.Peres,S.C.,&Lane,D.M.(2005).Auditoryg
raphs:Theeffectsofredundantdimensionsanddividedattention.ProceedingsofInternatio

nalConferenceonAuditoryDisplays,Limerick,Ireland.Petrie,H.,&Morley,S.(1998).Theu
seofnon-speechsoundsinnon-visualinterfacestotheMS-WindowsGUIforblindcomputeruser
s.ProceedingsofInternationalConferenceonAuditoryDisplays,Glasgow.Pirhonen,A.,Mur
phy,E.,McAllister,G.,&Yu,W.(2006).Non-speechsoundsaselementsofausescenario:Asemi
oticperspective.ProceedingsofInternationalConferenceonAuditoryDisplays,London.Pl
omp,R.(1964).Rateofdecayofauditorysensation.JournaloftheAcousticalSocietyofAmeri
ca36:27782.Pollack,I.,&Ficks,L.(1954).Informationofelementarymultidimensionalaudi
torydis-plays.JournaloftheAcousticalSocietyofAmerica26:15558.
References193

Puckette, M. S. (1997). Pure Data. Proceedings of International Computer Music C

onfer-ence , 22427.Pulkki, V. (1997). Virtual sound source positioning using vect
or base amplitude panning.Journal of the Audio Engineering Society 45(6):45666.Re
carte, M. A., & Nunes, L. M. (2003). Mental workload while driving: Effects on v
isualsearch, discrimination, and decision making. Journal of Experimental Psycho
logy,Applied 9(2):11937.Roads, C. (2001). Microsound . Cambridge, MA: MIT Press.S
awhney, N., & Schmandt, C. (2003). Nomadic Radio: Wearable Audio Computing.Cambr
idge, MA: Speech Interface Group, MIT Media Laboratory; available at http://web.
media.mit.edu/~nitin/NomadicRadio/ .Schmandt, C. (1993). Phoneshell: The telepho
ne as computer terminal. Proceedings of Asso-ciation for Computing Machinery Int
ernational Conference on Multimedia .Shannon, R. V., Zeng, F., Kamath, V., Wygon
ski, J., & Ekelid, M. (1995). Speech recognitionwith primarily temporal cues. Sc
ience 270(5234):303304.Staal, M. A. (2004). Stress, cognition, and human performa
nce: A literature review and con-ceptual framework. National Aeronautics and Spa
ce Administration (NASA) TechnicalMemorandum, TM-2004-212824. Moffett Field, CA:
Ames Research Center.Stanton, N. A., & Edworthy, J., eds. (1999). Human Factors
in Auditory Warnings . Alder-shot, Hampshire, England: Ashgate Publishers.Steve
ns, S. S. (1957). On the psychophysical law. Psychological Review 64(3):15381.ten
Have, P. (1999). Doing Conversation Analysis: A Practical Guide . London: Sage.T
erenzi, F. (1988). Design and realization of an integrated system for the compos
ition ofmusical scores and for the numerical synthesis of sound (special applica
tion for transla-tion of radiation from galaxies into sound using computer music
procedures). PhysicsDepartment, University of Milan, Milan, Italy.Truax, B., ed
. (1999). Handbook for Acoustic Ecology; see http://www2.sfu.ca/sonic-studio/han
dbook/ .Ulfvengren, P. (2003). Design of natural warning sounds in human machine
systems. PhDdiss., Royal Institute of Technology, Stockholm.VRSonic. (2007). Vi
beStudioavailabl e at http://www.vrs onic.com.Walker,B.N.(2002).Magnitudeestimati
onofconceptualdatadimensionsforuseinsonifi-cation.JournalofExperimentalPsycholog
y:Applied8(4):21121.Walker,B.N.,&Kramer,G.(2004).Ecologicalpsychoacousticsandaudi
torydisplays:Hearing,grouping,andmeaningmaking.InNeuhoff,J.,ed.,EcologicalPsycho
acoustics.NewYork:AcademicPress,15075.Walker,B.N.,&Lane,D.M.(2001).Psychophysical
scalingofsonificationmappings:Acomparisonofvisuallyimpairedandsightedlisteners.P
roceedingsofInternationalConferenceonAuditoryDisplay,Espoo,Finland.Walker,B.N.,&
Lindsay,J.(2004).Auditorynavigationperformanceisaffectedbyway-pointcaptureradius
.ProceedingsofInternationalConferenceonAuditoryDisplay,Sydney.Walker,B.N.,&Stamp
er,K.(2005).Buildingaudiodesignsmonkey:Anaudioaugmentedrealitydesignerstool.Proce
edingsofInternationalConferenceonAuditoryDisplay,Limerick,Ireland.
5AuditoryInterfaces194

Walker, B. N., Nancy, A., & Lindsay, J. (2006). Spearcons: Speech-based earcons
improvenavigation performance in auditory menus. Proceedings of International Co
nference onAuditory Display , London.Wang, D., & Brown, G. J. (2006). Computatio

nal Auditory Scene Analysis: Principles, Algo-rithms, and Applications . Hoboken

, NJ: John Wiley & Sons.Weir, J. (2005). The sound of trafficavai lable at http:/
/www.smokinggun.com/projects/soundoftraffic/ .Wickens,C.D.,&Hollands,J.G.(2000).
EngineeringPsychologyandHumanPerformance,3rded.UpperSaddleRiver,NJ:PrenticeHall.
Williamson,J.,Murray-Smith,R.,&Hughes,S.(2007).Shoogle:Multimodalexcitatoryinter
-facesonmobiledevices.ProceedingsofSIGCHIConferenceonHumanFactorsinCom-putingSys
tems.Winberg,F.,&Bowers,J.(2004).Assemblingthesenses:Towardsthedesignofcooperati
veinterfacesforvisuallyimpairedusers.ProceedingsofAssociationforComputingMachineryConferenceonComputerSupportedCooperativeWork.
References195

Thispageintentionallyleftblank

6CHAPTERVoiceUserInterfacesSusanL.HuraAvoiceuserinterface(VUI)isthescripttoaconv
ersationbetweenanautomatedsystemandauser.Thisscriptcontainsalltheutterancesthatt
heautomatedsys-temwillspeaktotheuserandthelogictodecidewhichutterancestospeakinr
esponsetouserinput.Underlyingthevoiceuserinterfaceisspeechrecognitiontechnologyt
hathastheabilitytocaptureanddecodetheusersspokeninputtoallowthesystemtounderstandwh
attheuserhassaid.ThemajorityoftheseautomatedconversationsbetweenauserandaVUItake
placeoverthephone,asmostspeechtechnologyiscurrentlydeployedinspeech-enabledinter
activevoiceresponseinterfaces(seeChapter7).Increasingly,VUIsarebeingaddedtotheus
erexperienceofmobileandhandhelddevices,in-vehiclenavigationsystems,anddesktopcom
puterapplications.Userscomeintoautomatedconversationswithasetofexpectationsabout
howspokenconversationshouldworkandtheappropriatewaytobehaveasacooperativespeaker
andlistener.Theoverwhelmingmajorityofusersexperiencecomesfromunscriptedhuman-to-h
umanspeech,afeatthatisfaroutsidethecap-abilitiesoftodaysspeechtechnology.Thislimi
tationislargelyunknowntomostusers,agreatmanyofwhomapproachVUIswiththeStarTrekmod
elinmind:Tosuchusers,speech-enabledsystemsareanalways-listeningintelligenceawait
ingoneseverycommand.WhatCaptainPicardrequestsisalwaysunderstood,andhisspokeninter
actionswithComputerappeareffortless.Therefore,itisnosur-prisethatusersexpectationsa
resomewhatoutofsyncwiththewaytodaysVUIsactuallybehave(Hura,2005).Theexpectationmi
smatchisparticularlyproblematicbecauseitleadsuserstoproduceunpredictablespeechin
putthatisoftenhandledpoorlybyspeechsys-tems.Inthischapter,Ioffermyexperiencesasa
nevaluatoranddesignerofVUIs.

Theunifyingthemeforthechapteristheexpectationsmismatchandhowtomin-imizeit.Ialsop
rovideanoverviewofspeechtechnologyfromtheperspectiveofaVUIdesignerwiththegoalofa
chievingeffortlessspokeninteractionwithauto-matedsystems.6.1AUTOMATEDCONVERSATIO
N:HUMANVERSUSMACHINECommercialapplicationsofspeechrecognitionarelargelybottom-up
systemsthatachieveunderstandingviastatisticallybasedmatchingoftheusersspokeninput
withstoredacousticmodelsofspeechsounds,words,andpronunciations(seeSection6.2form
oredetail).Humanbeings,bycontrast,relyheavilyontop-downknowledgeaboutmeaningandc
ontextwhenrecognizingspeech(Massaro,2006).Thatis,whenhumanbeingsarelisteningtosp
okenlanguage,wearenottryingtomatchtheincomingspeechsignalagainstanypossiblewordi
nthelan-guage.Instead,weautomaticallyandunconsciouslyexpecttohearwordsthatfitintot
heoverallconversationalcontext.Contextincludesmicrolevelinforma-tionsuchaswhichp
honemecombinationsarepermissible,rulesaboutsentencestructurethatdeterminewhatsor
tofwordislikelytocomenext,andnonlinguis-ticworldknowledgethatrulesoutnonsensical
interpretationsofunclearwords.Becauseweknowhowtheworldworks,weknowthatBanjoisahigh
lyunlikelyresponsetothequestionWhatdoyouwantforlunch?nomatterhowmuchtheacousticsig

nalpointsustowardthatinterpretation.Automatedspeechrecogni-tionsystemshavenoknow
ledgeoftheworldandmustrelysolelyontheacousticsignalandthebitsofintelligencebuilt
inbyVUIdesigners.Evenwithoutthebenefitofhuman-liketop-downknowledge,manyspeechre
cognitionsystemsperformquitewell,andthroughcleverVUIsappeartobecar-ryingontheirh
alfoftheconversationadmirably.Wheninteractionswithauto-matedspeechsystemsgowell,
usersassumethattheirinterlocutor(thesystem)isconversinginthesamemannertheyareperf
ormingthetasksofconversationinawaysimilartohowtheydoitandplayingbythesamerulesof
conver-sation.Grice(1975)definedasetofmaximsofconversationthatdescribetheunconsc
iousrulesbywhichweparticipateinconversations.Accordingtothemaximsofconversation,
weexpectutterancestobeofhighquality(truthfulandhonest),toofferanappropriatequant
ityofinformation,toberelevanttothecon-versation,andtobespokeninaclear,unambiguou
smanner.Automatedspeechsystemsoftendonotfollowthesecooperativeprinciplesaccidenta
llyorbydesignyetuserstendtogivethesystemsthebenefitofthedoubtandassumethattheyare
followingtherules.Thismisapprehensiononthepartofusersisthefundamentalchallengein
designingeffectiveVUIs.Whydousersexpectsomuchofautomatedspeechsystems?Inpartitis
becauseVUIssoundveryreal.Mostcommercialspeechapplicationsrelyona
6VoiceUserInterfaces198

setofrecordedprompts,spokenbyavoiceactorwhohasbeencarefullycoachedtoproduceeachu
tteranceasifitwerepartofanongoingconversation(seeadditionalinformationonpromptsi
nSection6.1.3).Thebiggerreasonbehindusershighexpectationsisthevastexperiencetheyh
avewithspokenlanguage.Languageisahugelycomplexsystemgovernedbymyriadrules,butpeo
plearelargelyunawareoftheirownlinguisticprowess.Humanbeingsareconstantlyimmersed
inspokenlanguage.Exposurebeginsbeforebirthandthereisevidencethathumaninfantscome
intotheworldwiththeabilityandexperiencetobeginimmediatelymakingsenseoftheirlingu
isticenvironment.Languageisacquiredautomaticallybyhumaninfantsthroughsimpleexpos
ureandrequiresnoadultintervention,otherthansimplyconversingwiththechild.Language
isoftencharacterizedasaninnateabilityofhumanbeings(Pinker,1994;Reeves&Nass,1996)
;thus,itisdifficultformanypeopletounderstandwhysimpleconversationcanbesodaunting
forautomatedsystems.Talkingandlisteningareoverlearnedtasksforhumansweconverseeffo
rtlessly,movingourvocalmusculaturetoproducespeechsoundsarrangedintofluentutteran
ceswithoutconsciouseffort.Itssoeasyforuswhyisitsohardforcomputers?Anotherreasontha
tusersexpectalotfromspeechsystemsisthattheyviewtheabilitytoengageinconversationa
sevidenceoftheinherentintelligenceofthespeechsystem.VUIsoftenseemsmart,friendly,
andevenhumantheyunderstandwhatyousayandrespondinwaysthatapersonmightinthatcircumstance.WhenusersdiscusstheirexperienceswithVUIs,theyareoftensomewhatdisconcerted
whentheynoticethemselvesattributingpersonalcharacteristics(smart,friendly)towhat
theyknowtobeacomputersystem.Theseembarrassedusersdonotbelievetheyareactuallytalk
ingtoaperson,buttheycannothelptalkingabouttheVUIassheratherthanit.Whenusersthinkthis
way,theyaresuccumbingtoattributiontheory(Heider,1958).Attributiontheorydescribes
thephenomenonofviewinganindi-vidualsactionsasevidenceofhisorherinherentcharacteri
stics.Thatis,weseetheactionsofothersasevidenceofstablefactorsthatarepartoftheirp
erson-alities.Humanbeingshaveastrongpropensitytooverattributeindividualbeha-vior
stotheinherentcharacterofanindividual.Thepullofattributiontheoryissointenseandso
unconsciousthatweattributepersonalitieseventononhumanswithwhomweinteract,suchasc
omputersandVUIs.ResearchbyNassandcol-leagues(Reeves&Nass,1996;Nass&Brave,2005)de
monstratesthatwetendtointeractwithnonhumanactorsaccordingtothesamesocialprincipa
lsweapplyinhuman-to-humaninteractions.Thefundamentalattributionerrorisrelevantto
VUIsinthatuserstendtoseeeveryactionofaVUIasevidenceoftheperson-alityandabilities
ofthespeechsystem.WhenaVUIoffersrelevantinformationandgivesappropriateresponsest
ousers,theytendtoattributethistotheintelli-genceandsensibilityofthesystem.WhenaV
UIsayspleaseandthankyou,webelievethatthesystemispoliteandfriendly.
6.1AutomatedConversation:HumanversusMachine199

6.1.1VUIPersonaVoiceuserinterfacedesignersoftencapitalizeonuserstendencytoattribu
techaracteristicsbasedonbehaviorinordertoattachaparticularpersonatotheVUI.Here,p
ersonaisnotusedtodescribeusercharacteristics(Pruitt&Adlin,2006)inthemorefamiliar
senseoftheterminthegeneralhumanfactorscom-munity.Instead,intheworldofspeechtechn
ology,personareferstotheperson-alityofaspeechinterfaceinferredbyusersbasedontheb
ehavioroftheVUI.Factorsthatcontributetotheperceptionofpersonaarethestyleandtoneo
fprompts,theflowofdialog,andtheresponsivenessofthesystem.SomeVUIdesignersconceiv
eofpersonaasanimaginarycustomerservicerepresentativebeingemulatedbytheVUI.Thecon
ceptofpersonahasbeenacontinuingsourceofcontroversyintheVUIcommunity.VUIdesigners
disagreevigorouslyontheimportanceofpersonainoverallVUIdesign(Klie,2007;Rolandi,2
007).Somedesignersdenythatpersonaexistsatall,andotherssuggestthatpersonaisthemos
tsignificantcharacteristicofanapplication.Therealityofpersonaalmostcertainlylies
somewherebetweenthesetwoextremes.Attributiontheorysuggeststhatitmaybeimpossiblef
orusersnottoinferapersonawhenparticipatinginaspokenconversation,whetherwithapersonorwithanautomatedsystem.Partofthehumanlanguagefacultyinvolvestheprocessofsizi
ngupyourinterlocutorbasedonthewayshespeaks.Thisisnotsomethingthathumanscanturnon
andoff;thus,itisimpossiblefortheretobenopersonainaVUI.VUIssupposedlydesignedwithno
personatendtosoundeitherschizophrenic(asifavarietyofindividualsarespeakingrather
thanonecoherentvoice),robotic(sodevoidofnormalhumanspeechpatternsthattheyremindy
ouofbad1970ssciencefiction),orboth.TherelationshipofpersonatoVUIisakintotherelat
ionshipofcolortotheoverallGUIdesignofawebpage(Hura,Polkosky,&Gilbert,2006).Color
isnotthesingledefiningqualityofuserspercep-tionsofthewebpage,buttheirimpressionsa
reaffectedinsignificantwaysbycolor.Thereisalsoevidencethatpersonaandtheusability
ofaVUIaretightlycou-pled.Usersattributepersonathroughevidencetheyobserveinthevoi
ceofthesys-tem(e.g.,pitch,pitchrange,loudness,loudnessrange,articulatoryprecisio
n),thelinguisticstyleofprompts(e.g.,wordchoice,syntax,content,andstructureoferro
rrecoveryprompts),andtheoverallorganizationofaVUI.ResearchbyPolkosky(2005)sugges
tsthatthesefactorspredictanimpressionofdialogefficiency,inter-activeease,andaffe
ctiveresponseintheuser.Thereisstrongsimilarityheretothecharacteristicsofgeneralu
sabilitypresentedinthestandardISO9241:11.6.1.2JekyllandHydePersonasBecausetheten
dencytoattributeapersonalitytoaVUIissostrong,onemightthinkthatchangingausersviews
onthispersonalitywouldbedifficult.Instead,
6VoiceUserInterfaces200

usersperceptionsofaVUIarenotoriouslyfragile.Onebadturnoftheconversa-tionandusersc
anloseallconfidenceinthesystemasanintelligentactorintheconversation.Speechrecogn
itionfailurescancausethiserosionofconfidence,butthisisnotalwaysthecase.Similarly
,promptsthatelicitunexpectedresponsescan,butdonotalways,causeuserperceptionstoch
angesignificantly.Theexam-plesthatfollowdemonstratethesetwoclasses.Thefirstcasei
swhentheuserspeaksanappropriateresponseclearlyandaudibly,butthesystemdoesnotreco
gnizeit.Usershaveagoodsenseofhowintelligibletheirutterancesare,andwhenaspeechsys
temfailstorecognizewell-formedresponsesuserscanbecomefrustrated.Thefrustrationca
nbeexa-cerbatedbylongandgenericerrorpromptsthatdonotallowforquickrepairoftheconv
ersation.Asanexample,considerthefollowing:SYSTEM:Pleaseselectoneofthefollowing:a
ccountbalances,transactionhistory,transfers...USER:Accountbalances.SYSTEM:Imsorry
.Ididnotunderstandyourresponse.Pleasechooseoneofthefollowing:Youcansay:accountba
lances,transactionhistory...Thiserrorpromptbelaborstherecognitionfailureanddraws
theusersattentiontoit.Infact,inhuman-to-humanconversationthissortofmisunderstandi
ngiscommon,andconversationsarerepairedandcontinuedwithonlyminimalinter-ruption.T
hedifferenceisthathumansusebriefconversationalrepairtechniquesratherthantakingaw
holestepbackinthedialog.Ifafriendasks,Whereareyougoingonvacation?andfailstounderst
andyourresponse,thefriendwilllikelysay,Where?asafollow-upratherthanrepeatingtheori
ginalquestioninitsentirety.Well-designedVUIsmimicconversationalrepairbyusingquic
krepromptsthatallowtheconversationtogetbackontrackquickly.Thesecondcaseiswhenapr

omptelicitsunexpectedresponses.Thesystemasksanopen-endedquestion,andbasedonprevi
ousgoodexperiencewiththeVUI,theuserrespondsappropriately.Thesystem,however,isinc
apableofdeal-ingwiththeresponseandgoesintoerrorprompting.Thisisessentiallyacaseo
ffoolingtheuserintothinkingtheVUIissmarterthanitreallyis(Rolandi,2002).Thetypica
lsituationhereisthatauserhasbeensuccessfullyinteractingwithasimplemenu-basedVUIf
orseveralturnsoftheconversation(thesystemsoffersanumberofdiscretechoices,andtheu
serrespondsbyspeakingoneofthesechoices).Theuseristhenconfrontedbyamoreopen-ended
prompt,liketheoneinthefollowingexample.Becausethepreviousinteractionhasbeensosuc
-cessful,theuserattributesintelligenceandcooperationtothesystem,andcooper-ativel
yanswersthequestionbeingasked.
6.1AutomatedConversation:HumanversusMachine201

SYSTEM:WhatelsecanIhelpyouwith?[pause]USER:Ineedtogetthebalanceonanotheraccount.
SYSTEM:Imsorry.Pleasechooseoneofthefollowing...Inthefirstprompt,theopen-endedques
tionisintendedsimplyasafriendlyandnatural-soundingintroductiontothelistofchoices
thatwouldhavefollowedthepause.Buttheuserassumesthatheshouldjustanswerthequestion
beingaskedandbargesinbeforehearingthechoices.Thisisnotacaseoftheuserincor-rectly
havinghighexpectationsforasystem.Onthecontrary,theusersexpecta-tionshaverightlybe
enbuiltupthroughaseriesofpositiveinteractions.Theproblemhereisthataskingtheopenendedquestionleadstheusertobelievethatthesystemwillbeabletounderstandhisresponse
.1Thetypeofinteractionshownherecanbeparticularlydamagingbecauseiterodesthetrusta
ndconfidencetheuserhasbuiltupwiththesystem.Askingaquestionandthennotbeingabletod
ealwiththeanswermakesusersfeelasiftheyhavebeendeliberatelytrickedintodoingsometh
ingfoolish.Onceanerrorlikethisoccurs,itcanbeextremelydifficulttogettheinterac-ti
onflowingsmoothlyagain.Onceusersbecomeawareofthepreviouslyhiddenlimitationsofthe
system,theybecomeattentiveforotherinstancesinwhichthesystemshowsitslackofconvers
ationalability.Astheuserbuildsuppositiveinteractionswithasystem,theconversations
eemsmoreandmorerealisticandthesystemseemsmorehuman.Butbeforetheusercanreachtheun
cannyvalley(inwhichalmost-humancomputersystemsgofromarousingpositiveemotionstone
gativeones[Mori,1970]),theillusionoftenshattersduetothetypesofissuesdescribedint
hissection.Theendresultistoooftenwhathasbeencalledthecon-versationaldeathspiral(At
twateretal.,2005)inwhichthesystemandtheuserbothtryunsuccessfullytorepairtheconve
rsationviadisparatestrategies.6.1.3ThePresentationLayer:PromptsUsersexperienceth
eVUIprimarilythroughitsprompts.PromptsarewhattheVUIsaysandaretheentiretyoftheevi
denceusersreceiveabouttheirinterlocu-tor.Throughprompts,usersexperiencethevoiceo
fthesystem,aswellaslinguis-ticandprosodicfeaturesthatcontributetotheattributiono
fapersona.Thebestwaytoappreciatetheimpactofpromptsistoexperiencetheminanautomate
dconversation.Imaginethefollowingsituation.Auser,Nick,ishavingtroublewithhissate
llitetelevisionservicehistelevisionscreengoesblankintermittentlyon1Therearespeech
recognitionsystemsdesignedtohandleresponsestoopen-endedques-tions.Thesystemdescr
ibedhereistypicalofgrammar-basedrecognitionsystemsthatrec-ognizeonlyitemspreload
edintothegrammar.
6VoiceUserInterfaces202

severalchannels.Nickgoesonlineandreadsthetroubleshootinginformationavailablether
e,butdoesnotfindanythinguseful.Hethendecidestocallhissatelliteprovideratthenumbe
rhelpfullyprovidedonthewebpage.Table6.1containsanaccountofhisphonecall.Nickscalli
stypicaloftheinteractionsthatmanyusershavewithVUIs.Usersbegintheconversationcoop
eratively,andwhensomethinggoeswrongtheyturntoconversationrepairtechniquesthatwor
kinotherconversations(e.g.,speakingmoreslowlyandclearlyasinStep12inthetable).Iro
nicallytheserepair

TABLE6.1HypotheticalCalltoaSatelliteTelevisionProviderStepSpeakerUtteranceUsersTh
oughtsandActions/Comments1SystemHelloandthankyouforcallingBestChoiceSatellite(br
andingsound),thebestchoiceforallyoursatellitetelevisionneeds.Sofar,sogood.2Syste
mYoucanalsocontactusviaourwebsite,w-w-wdotbestchoicedotcom.Ourwebsiteofferscompl
eteinformationaboutsettingupsatelliteserviceatyourhomeandoffersvaluableinstallat
ioncouponsworthover$40.Thisisreallyannoying!Ijustspent10minutesonthewebsitewitho
utsolvingmyproblem.3SystemIcanhelpyouwiththefollowingfiveoptions.Youcaninterrupt
meandspeakyourchoiceatanytime.Pleaseselectoneofthefollowing:Signupfornewservice,
addfeaturestomyservice,movemyexistingservice,problemswithmysatelliteservice,oras
kabillingquestion.ImgladtheguybehindthecounteratthelocalsatelliteTVofficedoesnttal
kthismuch....Ohwait,whichonedoIwant?4NickServiceproblem.Finally!5SystemImsorry.Id
idnotunderstandyourresponse.Icanhelpyouwiththefollowingfiveoptions.Youcaninterru
ptmeandspeakyourchoiceatanytime.Pleaseselectoneofthefollowing:Signupfornewservic
e,addfeaturestomyservice,movemyexistingservice,problemswithmysatelliteservice,or
askabillingquestion.Huh?Betterlistenagaintofigureoutexactlywhattosay.6NickProble
mswithmysatelliteservice.(Continued)
6.1AutomatedConversation:HumanversusMachine203

techniquesoftenexacerbatetheoriginalproblemorcreatenewones.Inthefol-lowing,theco
nversationisanalyzedturnbyturntohighlightwhatwentwrongandhowtoavoidtheproblem.Th
edesignguidelinesinSection6.6detailmanyoftheprinciplesintroducedhere.Step1ofthec
onversationactuallybeginswithausefulpractice.Therecordedgreetingincorporatesanon
speechbrandingsound(e.g.,theIntelnoise).Thisservestheusefulfunctionofdemonstrati
ngthattheuserisinteractingwithanautomatedsystem,notarealperson.Withoutthenonspee
chaudio,theusermaybeinitiallyunsurewhetherheistalkingtoaliveagentfaithfullyrecit
ingtheofficialcompanygreetingorlisteningtoarecording.Somesystemschoosetoannounce
theirnonhu-manstatus(e.g.,YouhavereachedABCBanks24-hourautomatedaccountaccess
TABLE6.1ContdStepSpeakerUtteranceUsersThoughtsandActions/Comments7SystemPleasesayo
renteryouraccountnumber.Iguesssheheardme.ButIdontknowmyaccountnumber!WhereamIgoin
gtofindit?8Nick(mutters)Justasecond...(Nickfranticallyshufflespaperstryingtofind
hislastsatelliteTVbillandmuttersJustasecondunderhisbreath.)9SystemImsorry.Thenumber
youenteredisnotavalidaccountnumber.Pleasesayorenteryouraccountnumber.Whydidsheth
inkthatwasanaccountnumber?AnyonecantellthatJustasecondisnotanaccountnumber!Whateve
r,Illjusttryagain.10Nick0-0-0-0-dash-4-3-5-7-2-2-7-6(Nickreadswhathebelievestobeh
isaccountnumberfromhisprintedbillingstatement.)11SystemImsorry.Thenumberyouentere
disnotavalidaccountnumber.Pleasesayorenteryouraccountnumber.(Systemfailstorecogn
izebecauserecognizedaccountnumbersareeightdigitsanddonotincludeleadingzerosandth
ehyphen.Systemgoestoareprompt.)Givemeabreak.Ijusttoldyoutheaccountnumber.12Nick0
...0...0...4...3...5...7...2...Thismachineisidiotic.Imgoingtogoextraslowsoshecantm
essupagain.13SystemImsorry,thatisnotavalidaccountnumber.OnemomentwhileItransferyo
urcall.
6VoiceUserInterfaces204

system).Althoughagreetinglikethiscanservethesameinformativepurpose,non-speechaudi
ointhegreetingavoidsimmediatelyarousingthenegativeemotionssomeusersexperiencewhe
ntheyheartheywillbetalkingtoasystem.Step2demonstratesthecommonpracticeofusingthe
momentjustfollowingthegreetingtoinsertadvertisementsorgiveinstructionsonhowtouse
thesystem.Theideahereisthatusersshouldbegiventhisinformationearly,beforetheystar
tusingthesystem,justincasetheyneeditlater.However,informationpresentedtousersout
ofcontextwillnotseemrelevantandisunlikelytoberetained.Whenusersbeginaninteractio
nwithanautomatedsystem,theytendtobeextremelytaskfocusedtheyarethinkingaboutwhatth

eyneedtoaccomplishandnothingelse.Myexperiencefromusabilitytestingsuggeststhatuse
rsignoreinformationpresentedbeforetheylocatethefunctionalitytheyneedtoaccomplish
theirtask.Inthecaseofadvertisingmessages,itisclearthattheplacementofthemessageis
ineffectivebecausetheaudienceisprimedtoignoreituntiltheyreachtheirgoal.Thisreaso
ningtendstobepersuasivewhennegotiatingwithmarketingdepartmentssimplyaskiftheyinte
ndtheiradtoactivelyannoytheuser.ThefictionalmenushowninStep3isbasedonthemenusone
oftenhearsinVUIs:Icanhelpyouwiththefollowingfiveoptions.Youcaninterruptmeandspea
kyourchoiceatanytime.Pleaseselectoneofthefollowing:signupfornewservice,addfeatur
estomyservice,movemyexistingservice,problemswithmysatelliteservice,oraskabilling
question.Thereareonlyfivemenuoptions,butthemenustillseemsquitelong.(SeeSection6.
6.1foradiscussionofmenusandmemory.)Onereasonisthattherearethreesentencesofpreamb
lebeforetheoptionsareactuallypresented.Aswithadvertisementsdiscussedinthepreviou
ssection,instructionspre-sentedbeforethemainmenuaretypicallyignoredbyusers.These
introductorypromptsareoftenthoughttobenecessarytoprepareuserswhomaybeusingaspeec
hsystemforthefirsttimeandwhoaremoreaccustomedtotouch-tonesys-tems.VUIbestpractic
essuggestthatmostuserswillbeabletogivesatisfactoryspokenresponsesaslongastheprom
ptsincludewordinglikeYoucansay...beforemenuoptionsarepresented.Anotherproblemwitht
hemenupresentedinStep3isthemenuoptionnamesthemselves.Eachoptionislongandsyntacti
-callycomplex,andtheoptionnamesarepartiallyoverlapping(fourusethetermservice).This
makesitdifficultforuserstolocateandrememberthemenuoptiontheywant.Theusersresponse
atStep4Serviceproblemdemonstratesthesortofresponsesgeneratedbylong,complexmenuoption
names.Theuseraccuratelyidentifiedtheoptionhewanted,andproducedanabbreviatedversi
onofitashisresponse,whichwasunfortunatelynotrecognizedbythesystem.Becausetheuser
accuratelyselectedtheappropriateoption,themenuoptioninitselfis
6.1AutomatedConversation:HumanversusMachine205

notfullyatfault.Instead,thisrepresentsafailureinthegrammarforthismenu.Agrammaris
thesetofallitemsthatcanberecognizedataspecificrecognitionstate,asdefinedbytheVUI
designer.ItiseasyforapersontounderstandthattheusersresponseServiceproblemwasintende
dasasynonymforthemenuoptionProblemswithmysatelliteservice.However,thespeechsystemi
sunabletomakethesamejudgmentunlessthatparticularsynonymisbuiltintothegram-mar.Th
elongerandmorecomplexthemenuoptionnames,themorepossiblesynonymsthereare,andthusl
ongerandmorecomplexthegrammarsarerequiredtosupportthemenuoptions.Step5isanexampl
eofineffectiveerrorhandling:Imsorry.Ididnotunderstandyourresponse.Icanhelpyouwith
thefol-lowingfiveoptions.Youcaninterruptmeandspeakyourchoiceatanytime.Pleasesele
ctoneofthefollowing:signupfornewservice,addfea-turestomyservice,movemyexistingse
rvice,problemswithmysatelliteservice,oraskabillingquestion.Thisrepromptblamesthe
userfortheerrorbysuggestingthathefailedtocor-rectlychooseanitemfromthemenu,wheni
nfacttheuserisgivingapredictablevariantofamenuoptionnamethatshouldhavebeencovere
dinthegrammar.Therepromptthenrepeatstheentireinitialprompt,includingtheintroduct
orysentences,butdoesnotofferanyinformationtotheuseraboutspecificallyhowtogiveawe
ll-formedresponse.Simplerepetitionisaneffectiverepromptingstrat-egyonlyforthosec
asesinwhichtheuserfailstorespondbecausehedidnothearthechoicesinitially.Here,theu
serclearlytriedtomakearesponse,sotherepromptissimplyannoying.Whyistheuserfeeling
uncomfortableinStep7?Pleasesayorenteryouraccountnumber.Inpartitisbecauseheisbein
gaskedforinformationthathedoesnothaveathand.Evenbeforethis,though,theuserisuncom
fortablewiththeflowoftheconversa-tionbecauseheisnotsurewhetherhisresponseinthepr
eviousstepwasaccepted.Whatsmissingisaseeminglyunimportantconnectorword,acknowledg
ingtheusersresponse.SimplyaddingawordorphrasesuchasAllrightbeforeaskingfortheaccoun
tnumberwouldhavelettheuserknowthathewasheardandunder-stood,andthattheconversatio
nwouldbemovingontoanewtopic.Theseconnectorwords,knownasdiscoursemarkers,serveman
yfunctionsinspokenconversation,includingacknowledgment(okay,allright),changingto
pics(otherwise,instead),emphasizingorreinforcing(also,aboveall),andservingassequ
encemarkers(first,next,finally)(Schiffrin,1987).Discoursemar-kersconveyvitalinfo
rmationaboutthestructureandflowoftheoveralldialog.Withoutdiscoursemarkers,aconve
rsationfeelsincomplete,andwillleaveusersuncomfortablewithoutknowingexactlywhy.

6VoiceUserInterfaces206

Step8,inwhichtheusermuttersJustasecond,isbothanindicationoftheineffectivenessofthe
accountnumberpromptinStep7andanillustrationoftheblatantlackofreal-worldknowledge
ofspeechrecognitionsystems.Toanyper-son,itisobviousthattheuserwasnotintendingJust
asecondasaresponsetotheaccountnumberprompt.Hewassimplymumblingashetriedtolocatehi
saccountnumber.Thesystemcannotatfacevaluedistinguishsystem-directedspeechfromsid
espeech(definedasspeechaudibleoverthetelephoneconnec-tionthatisnotintendedaspart
ofthetelephoneconversation).Thefactthattheuserismutteringhereisasignthatthereisa
problemwiththeaccountnum-berprompt.Theuserhesitatesbecauseheisunpreparedtogiveth
einformationheisbeingaskedtoprovide.Thebestwaytopreventthissortofproblemistoavoi
daskingforinformationthattheuserwillneedtolookup,infavorofsome-thingtheuserwillh
avememorizedsuchasatelephonenumber.Whenthisisimpossible,givetheuserpracticaladvi
ceforwheretheycanfindandrecognizetherequestedinformation.Step9isanotherpoorlydes
ignedreprompt:Imsorry.Thenumberyouenteredisnotavalidaccountnumber.Pleasesayorente
ryouraccountnumber.First,theuserdidnotinfactenteranaccountnumber,sothepromptshou
ldnotaccusehimofenteringaninvalidone.Moreimportantly,therepromptfailstooffertheu
serassistanceinfindingtheaccountnumber.Especiallywhenrequest-ingspecificpiecesof
information,itisvitaltowriterepromptsthatgiveusersadditionalinstructionthatwillh
elpthemprovideawell-formedresponse.Theuserwouldhavebeenmoresuccessfulandmorecomf
ortableiftheprompthadsaid,forexample,Pleasesayorenteryoureight-digitaccountnumber
.Youcanfindyouraccountnumberintheblueboxunderyournameonyourstatement.Step10showst
heresultsofgivingtheuserinsufficientinstructionaboutwhatconstitutesanaccountnumb
er.Nicklocatedthenumberandsaid,0-0-0-0-dash-4-3-5-7-2-2-7-6becausehewasnotinstruct
edtospeakonlythelasteightdigits.Thegrammarcontributedtotheproblematthisstepbecau
seitdidnotaccommodatethefull12-digitversionoftheaccountnumber.Step11istheidentic
alrepromptplayedinStep9.Whatwasannoyingandunhelpfulonfirstpresentationcanbecomei
nfuriatingwithrepetition.Repeatingtheidenticalreprompt,withtheidenticalcheerfult
oneofvoice,iscompletelyoutofsyncwiththeconversation.Theuserhasjusttriedmultiplet
imestogiveapieceofinformationandfailedineachcase.Whenthesystemcalmlyrepeatstheid
enticalprompt,theusercanfeelasifheisbeingpatronized.Whenwritingreprompts,bemindf
uloftheuserspotentialfrustrationlevel.Twotriesatanygivenrecognitionstateisaboutth
emaximumthatuserscantoleratewith-outsignificantannoyance,inmyexperience.EarlierI
recommendedbriefrepromptingforarepromptatamenu(seeSection6.1.2).Here,andforother
6.1AutomatedConversation:HumanversusMachine207

states when the user is inputting data (account or ID numbers, etc.), repromptss
hould actually be more expansive than the initial prompt, adding informationto h
elp the user provide a well-formed response.In Step 12 when Nick slowly says, 0..
. 0... 0... 4... 3... 5... 7... 2..., wesee the expectations mismatch in action.
In frustration, the user falls back on astrategy for giving information that wor
ks in human-to-human conversationsunder these circumstancesspeaking slowly and cl
early. In the automated con-versation, however, this strategy makes the recognit
ion problem worse. Acousticmodels used in recognition are built on normal conver
sational speech, not onthe deliberate, slow, overenunciated speech spoken here;
thus, recognition maysuffer. In addition, the user is speaking so slowly that he
has exceeded the sys-tems time limit for giving a response. Consequently, the sy
stem interrupts theuser before he is done speaking, adding a turn-taking error (
Sachs, Schleghoff, &Jefferson, 1974) to the recognition problems. The problem ca
n be reduced byadjusting the parameter that specifies the amount of time the use
r has to respondbefore the system times out at a dialog state (sometimes labeled
max speechtimeout). It is far preferable to avoid causing the user frustration in

the first placeby limiting the number of retries and using more effective repro
mpts.In Step 13 the user is released from speech technology jail and sent to wai
t ina queue for a live representative after several minutes of torture and no cl
oser tosolvinghisproblem.Thesortoffrustrationandpoorcustomerserviceexperi-encedb
ytheimaginaryuserhereisthemotivationforrealuserstoadvocateforIVRreform(forbothsp
eechandtouch-tone)throughthegethumaninitiative(http://www.gethuman.com ).6.2TECHNO
LOGYOFTHEINTERFACEThetechnologybehindVUIsisautomaticspeechrecognition(ASR).Thesy
stemisoftenaspeech-enabledIVR,butincreasinglymayalsobeapersonalcomputer,in-carna
vigationsystem,orotherspeech-enabledhandhelddevice.Allthesesys-temsdependonthesa
mebasictechnologytorecognizespeechinput.Ineachcase,thespeechsignaliscaptured,dig
itized,segmented,andthencomparedagainstasetofstoredacousticmodelsforspeechsounds
.Thesoundsarethenbuiltupintopotentialwordsthatarecomparedwithagrammarofwordsthat
aretoberecognizedatthatpointinthedialog.Therecognitionprocessisacous-ticallydriv
enthereisnotop-downsemanticorlogicalanalysistohelpthesys-temdecodeuserinput.Thus,
thewaythatcomputersystemsrecognizespeechisfundamentallydifferentfromthewaythathu
mansdo.MostASRsystemsincommercialusetodayrelyonHiddenMarkovModels(Rabiner,1989)t
orecognizewords.Inthe1980sand1990s,agreatdealofresearchwasfocusedonbuildingmoreh
uman-likespeechrecognitionssystems(e.g.,McClelland&Elman,1986).These
6VoiceUserInterfaces208

systemsaimedatmimickingthemultiplesourcesofnonacousticinformationusedbyhumanswhe
ndecodingtheacousticspeechsignal.Suchsystemscontainedhugeamountsoflinguisticinfo
rmationaboutthestructureofwords,sentencestructure,andsemantics(meaningatthelevel
ofwordsandsentences).Eventhiswasinsufficienttomodeleverythingthathumansbringtoth
etaskofspeechrecognition,somodelsofdialogstructureandworldknowledgewerealsoincor
porated.Thecomplexityandcomputationalrequirementsofsuchsystemshavegenerallykeptt
heminthelaboratoryratherthanallowingthemtomigrateintocommercialapplications.6.2.
1TheRecognitionProcessFigure6.1depictsthestepsintherecognitionprocess.Userinputi
scapturedviaamicrophone(eitherbuiltintothephoneorconnectedtothecomputerorotherde
vicebeingused).Thefirstmajortechnicalhurdleforspeechsystemsisencoun-teredatthisv
eryearlystageoftheprocess:Whatcountsasuserinput?Thepro-cessofmakingthisdecisioni
scalledend-pointing.Amicrophonewillcaptureanysoundintheenvironment,notjustthosed
eliberatelyutteredbytheusertothesystem.Innoisyenvironments,thequalityoftheaudios
ignalispoorerthanthesignalcapturedinaquietenvironment.Wehavealltriedusingmobilep
honesinnoisyenvironments(inacarintraffic,atacrowdedairport)andexperiencedproblem
srelatedtobackgroundnoise,evenwhenspeakingtoanotherhuman.Forspeechrecognitionsys
tems,thisproblemisamplifiedbecausesystemscannotautomaticallydistinguishspeechfro
mnonspeech.Thisissuecanbeimprovedbysimplymakingthethresholdforrecognitionofanyso
undhigher,sothatlow-levelbackgroundnoisesarenotdetected.Thereisatrade-off,howeve
r,inthismethodbecauseuserswhospeaksoftlywillsimplynotbeheardbyasystemwithtoohigh
athreshold.
End-pointingSpeechEnd-pointedutteranceWordstrin
g
MeaningActions
/
promptsFeaturevectorsFeatureextractionRecognitionNatural-languageunderstandingI w
anna goto DallasDestination: DallasPlay> OK, whatday do you wantto leave?Dialogma
nagement
FIGURE6.1Speechrecognitionprocess.Sequenceofstepsinautomaticspeechrecognition.So
urce:FromCohen,Giangola,andBalogh(2004).

6.2TechnologyoftheInterface209

Inadditiontobackgroundnoise,otherclassesofsoundcomplicatethecap-tureprocess.Nons
peechmouthsoundssuchascoughing,clearingthethroat,andlaughingareparticularlyhardtod
istinguishbecausetheysharesomeoftheacousticqualitiesofspeechinthattheyareproduce
dbythehumanvocaltract.Themostdisruptivetoconversationswithspeechsystemsissidespe
echspeechthatiswithinrangeofthemicrophonebutisnotdirectedatthesystem.Sidespeechma
ybespokenbytheuserorbyanotherpersonintheroomwiththeuser.Sidespeechisdisruptiveno
tonlybecauseisbeginsaroundoferrorhandling,butalsobecauseitemphasizesthelimitatio
nsofthesystem.Whenthesystemcannotdistinguishbetweenspeechthatislogicallypartofth
econversationandspeechthatismeanttobeignored(e.g.,NicksJustasecondinTable6.1),theus
erissuddenlyacutelyawareofhowstupidthesystemreallyis.Somethingusersper-ceiveascomp
letelyeffortlessknowingwhatisandisnotpartofthecurrentcon-versationisbeyondthecapab
ilitiesoftheautomatedsystem,sousersbegintodoubtthesystemsabilitiesoverall.Oncethe
signaliscaptured,itisdigitizedanddividedintoasetofshortseg-mentsforacousticanaly
sis.Eachsegmentistransformedintoafeaturevectoranumericalrepresentationofthespeech
signalthatcontainsinformationrelevanttorecognition.Thesequenceoffeaturevectorsis
thebasisforrecognitionatthelevelofwords.Theelementsthatcontributetotherecognitio
nmodelareacousticmodels,dictionaries,grammars,andasearchalgorithm.Acousticmodels
specifythepro-nunciationofphonemesthespeechsoundsthatcomprisealanguageintermsofseq
uencesoffeaturevectors.Acousticmodelsaregenerallybuiltbytrainingthemagainstlarge
setsoftranscribedandlabeledacousticdata.Thus,acousticmodelsonlysupportpronunciat
ionsthatarewellrepresentedinthetrainingdata.Thisisthereasonthatunusualaccents,or
particularvoices,aresometimespoorlyrecognizedbyspeechsystems.Adictionaryinaspeec
hrecognitionsystemisalistofwordsandassociatedpronunciations.Pronunciationsarerep
resentedusingaphoneticalphabet,andtheremaybeseveralalternatepronunciationsforasi
ngleword.Pronunciationofwordsmayvarydependingonfactorslikedialect,rateofspeech,a
ndspeakingstyle.Foraspecificpronunciationtoberecognized,itmustbeaddedtothedictio
nary.Agrammaristhelistofutterancesthattherecognitionsystemisexpectedtoreceiveasi
nputateachrecognitionstate.Everytimeweintendtherecognizertoacceptincomingspeech,
wemustdefinewhatitisthatcallersarelikelytosay.Agrammarshouldlistallthelikelyvari
antsofresponsesthatusersmightproduceatagivenstate(recallStep4inNicksconversation,
Table6.1).Thereisacosttoaddingitemstoagrammar,however;themoreitemsthereareinagra
mmarandthemoresimilarthegrammaritemsareacoustically,themoredif-ficultrecognition
willbe.GrammarsthatarewrittenbyaVUIdesignerordevel-operarecalledrule-basedgramma
rs,andareusedinthemajorityofrecognitionapplicationsincommercialuse.
6VoiceUserInterfaces210

Asecondsortofgrammarisdefinedstatisticallyratherthanbyrule.Statisti-callanguagem
odels(SLMs)arecreatedautomaticallyfromtranscribedspeechdata.TheSLMlooksatallthep
ossiblesequencesofwordsandcomputestheprob-abilityofwordsappearinginparticularcon
texts.SLMsallowuserstogiverela-tivelyfree-formresponsesratherthansimplyrepeating
menuoptionsofferedbythesystem.ThismakesSLMsattractivetomanyorganizationsdeployin
gspeechtechnology.(RememberthatStarTrekmodel?Speechrecognitionsoftwaresalesfolks
canconvinceyouthatSLMsalmostmakeitpossible.)Tobeeffective,how-ever,SLMsrequireva
stamountsoftrainingdata(tensorhundredsofthousandsofutterances),whichmustallbetra
nscribedandlabeledbyaperson.SLMsarethereforeseveralordersofmagnitudemoretimecons
umingandcostlythanrule-basedgrammars.Onlylargeorganizationstendtoseeapositiveret
urnoninvestmentfromSLM-basedspeechrecognition.Afinaltypeofgrammartonoteisslotted
grammars,inwhichtheusercanspeakseveralindividualpiecesofinformationinasingleutte
rance.Slottedgram-marsareoftenusedforsimpletransactionssuchastransferringfundsbe
tweenaccountsorrequestingatravelreservationbetweentwocities.Userutterancesforslo

ttedgrammarscanseemquitefree-form,similartothepossibleresponsesforstatisticallan
guagemodels,suchasIwanttotransfer$3,000fromMarketGrowthFundtoInternationalEquityF
und.However,slottedgrammarsaremuchmorelimitedbecausetheyaredesignedtocollectmulti
plediscretechunksofinformationthatcanbeconnectedbyfillerwords.Inthesimpletransfe
rexampleabove,aslottedgrammarwouldbebuilttosearchtheincomingaudiosignaltofillana
mountslot,afrom-fundslot,andthenato-fundslot.Ifauserresponsedevi-atesfromthisamo
unt!from-fund!to-fundmodel,aslottedgrammarwillfailtorecognizeit,showingtheveryli
mitedrangeofinputflexibilitypossibleforslot-tedgrammars.Thislimitationmakesslott
edgrammarsagoodfitfortaskswhereuserresponsesareformulaicandfollowasetpattern(suc
hassimpletransfers),butlesssuitableforsituationsinwhichusersmightphrasetheirresp
onseinmanydifferentways.Usingtheacousticmodels,dictionaries,andgrammars,thespeec
hrecogni-tionsystemthenlooksformatchesbetweenthesequenceoffeaturevectors(thatrep
resentincomingspeech)andpossiblewordsandpronunciationsinthegram-mar.Thebestmatch
ingpaththroughthesequenceoffeaturevectorsisthewordthatisrecognized.Mostspeechrec
ognizershavetheabilitytoreturnnotjustthebestmatchingpath,butn-bestpaths(then-bes
tmatchingpathsratherthanjustthetopofthelist).Associatedwitheachiteminthen-bestli
stwillbeaconfi-dencescore,ameasureofhowconfidenttherecognizeristhatthematchiscor
-rect.Inpractice,then-bestlistisarank-orderedlistofwordsandtheprobabilityforeach
.Themethodforsearchingforbestpathsusedmostoftenincurrentspeechrecognitionsystems
iscalledHiddenMarkovModels(HMMs).HMMsareastatisticaltechniqueforcomputingtheprob
abilityofamatchatastateandbetweenstates.
6.2TechnologyoftheInterface211

6.2.2SomeHistoricalNotesonSpeechRecognitionHistorically,speechrecognitionsystems
wereconstrainedbythelackofproces-singspeedandcapacity.Speechrecognitioniscomputa
tionallyintensive,andasrecentlyasthe1990s,thewayspeechsystemsfunctionedwaslimite
dbylackofprocessingpower.(SeeBaberandNoyes[1993]andLea[1980]forilluminatingdis-c
ussionsofthefutureofspeechtechnologywrittenintherecentpast.)Onewayofdealingwithp
rocessinglimitationswastorestrictrecognitiontotheutterancesofasingleindividual.T
hesespeaker-dependentsystemsarebuilttorecognizethespeechofanindividual,andthusre
quiretrainingdata(intheformofspokenutterances)fromthisindividualinordertofunctio
n.Incontrast,speaker-independentsystemsdonotrequiretheusertoprovidetrainingdatab
eforebeingrecognized.Instead,speaker-independentrecognitionproceedsfromasetoftra
iningdatafromalargenumberofindividualswhomayormaynotbeusersofthesystem.Theadvant
ageofspeaker-dependentsys-temswasthattheycouldsupportlargervocabulariesthanspeak
er-independentones,givenafixedamountofprocessingpower.Ifprocessingpowerislimited
,thereisadirecttrade-offbetweenthenumberofwordsinasystemsvocabularyandthenumberof
differentvoicesthatcanberecognized.Today,thisdistinctionhasallbutdisappeared,and
thevastmajorityofcommerciallydeployedspeechsystemsarespeakerindependentandrecogn
izealimitednumberofitems(althoughthisnumbermaybeinthethousands)foranyrecognition
state.Oneimportantcounterexampleofspeakerdependenceinusecurrentlyisspeakerverifi
cationsystems(Campbell,1997;Furui,1996).Thesesystemsaretrainedtoidentifyusersbyt
heuniquecharacteristicsoftheirvoices.Theusersvoicethereforebecomesamethodofauthen
ticationthatcanbeusedalongsidelessreliableauthenticatorslikepasswordsoruserIDs.F
orspeakerverification,usersareenrolled(typicallyoverthephone)byspeakingaparticul
arutterancemultipletimes.Representationsoftheseutterances(sometimescalledvoicepr
ints)arestoredbytheorganizationdeployingtheverificationsystemandareusedasonemeth
odofauthenticatingtheuserwhenshenextcallsintothesystem.Speakerverificationhasbee
nshowntoberobustanddifficulttofool,andisadvantageousbecauseusersnolongerneedtore
memberpasswords.Instead,bysimplyspeakingapieceofidentifyinginformationsuchasanac
countnumber,thesystemisabletobothrecognizethedatabeingspokenandrecognizethevoice
itself.Therearemanypossiblecombinationsofdataandvoiceprintspossibletoprovidediff
erentlevelsofsecurityforspeechapplications.Anotherhistoricaldistinctionisbetween
systemsthatrecognizeisolatedwordsandthosethatrecognizeconnected(orcontinuous)spe
ech.Previously,rec-ognitionwasmuchmoreaccurateforisolatedwordsbecauseisolatedwor
dselim-inatetheneedtosegmenttheincomingspeechsignalintowords.Itcanbenotoriouslyd

ifficulttoaccuratelylocatewordboundariesinconnectedspeech,andeliminatingthisdist
inctionsignificantlyimprovedrecognitionaccuracy.The
6VoiceUserInterfaces212

majorityofspeechsystemsinusetodaystillrelyonisolatedwordrecognition(althoughthes
ystemscannowrecognizelongerphrases,notjustwords).Statisti-callanguagemodels,whic
haretrainedoncontinuousspeech,offerthenearestapproximationtocontinuousspeechreco
gnitionincommercialusetoday.Foran excellent history of speech technologies, see
appendix A in Weinshenk andBarker(2000).6.3CURRENTIMPLEMENTATIONSOFTHEINTERFACE:
ONTHEPHONEVoiceuserinterfacesareoverwhelminglyencounteredoverthephone.Speechreco
gnitiontechnologyhasbeenadoptedinthecallcenterasthesuccessortotouch-toneIVRsyste
ms.Speechhasanumberofinherentadvantagesovertouch-tone.Allowinguserstospeaktheirr
esponsesratherthankeytheminonthetelephonekeyboardmakestheinteractioneasierineyes
-busy,hands-busysituations.Theprevalenceofcordlessandmobilephonesalsogivesanadva
ntagetospeechinputovertouch-tone.Speechdoesnotrequirethatuserspullthehand-setawa
yfromtheirear,pressakey,andthenreturnthephonetotheearrepeatedly.Speechalsoenable
sfunctionalitythatisawkwardorimpossibleusingtouch-tone.Enteringnames,cities,fund
names,oranystringthatcontainsbothnum-bersanddigitsisverydifficultintouch-tone,wh
ichrequiresuserstoeitherspellwordsonthetelephonekeypadorchoosefromverylonglistso
fitems.Speechrecognitionsimplifiestaskssuchascollectingnameandaddress,transferri
ngfundsbetweenaccounts,andmakingtravelreservations.Speechalsoallowsforshallowerm
enuhierarchies,astheinteractionisnolongerconstrainedbythelim-itsofthe12-digittel
ephonekeypad.VUIshavegrownandmaturedinthetele-phonyworld,andmuchofVUIdesignistot
allyphone-centric.Salespeoplealsopitchspeechsystemsasmorenaturalthantouch-tone,sin
ceweallknowhowtotalk.(Isincerelyhopethatthischapterwillconvincethereaderofthefal
lacyofthissalespitch.)Voiceuserinterfacescanbefoundoutsidethecallcenter,although
inmuchsmallernumbers.Oneapplicationthatisrapidlybecomingmorewidespreadisspeech-e
nabledin-vehiclenavigationsystems.Thesesystemsallowthedrivertocontrolthenavigati
onsystemandvariousfeaturesofthevehicle(soundsystem,airconditioning)throughvoicec
ommands.Thesevehicle-basedsystemsaremul-timodalinthatthereisascreenavailablethro
ughouttheinteraction.Thismakesthespeechpartoftheuserinterfacesomewhatlessrichtha
nwhatisseenintele-phonyapplications.Speechrecognitiononthepersonalcomputerdeskto
pactuallyprecededwide-spreaduseofspeechinthecallcenter,butdesktopspeechrecogniti
onusehasyet
6.3CurrentImplementationsoftheInterface:OnthePhone213

tobecomethenormforthevastmajorityofusers.Dictationprogramshavebeenavailableforde
cadesandsomeusersarecommittedtothem.Theseprogramsarebuilttorecognizefree-forminp
utfromasingleuserratherthanasmallgrammarofwordsspokenbyanyuser.Becauseofthis,dic
tationprogramsmustbetrainedtothevoiceoftheuser,whichentailsaninitialtimeinvestme
ntbeforetheprogramisusable.Dictationprogramscanworkquitewell,butsufferfromapoorr
eputationofbeingtime-consumingandinaccurate.Lesscommonlyused(thoughnowwidelyavai
lablethroughWindowsVista)aredesktopcommandandcontrolprograms.Withtheseprograms,u
serscanspeakcommandstoperformthesametaskstheyusuallyaccomplishwithkeyboardandmou
se.Again,thevoicecomponentoftheinterfacehereisratherminimaltheuserhasasetofcomman
dsthatcanbespo-ken.Thereisnoconversationasthereisintelephonesystems.Speechisalsofo
undisanumberoftinynichemarkets.Therearededicatedspeechrecognitionsystemsinfields
suchasmedicaldictationinwhichthereisacomplexbutlimitedspecializedvocabulary.Spee
chsystemsarealsousedbyworkersinwarehousestoassistwithinventoryandorderfulfillmen
ttasks.AlthoughthesenicheapplicationsareonthefringeofVUItoday,theyarelikelytopro
liferateincom-ingyears.Nichemarketsareconducivetospeechtechnologybecauseanicheis
alimiteddomainthatconstrainsthevocabulary(thusenablingsmallergrammarsandbetterre

cognitionperformance),andbecausenicheuserstendtobemorewill-ingtolivewithlimitati
onsthatthegeneralpopulationmightnottolerate.Nicheusersmaybewillingtowearheadsets
orotherdevicestoenablespeechrecognition,andmaybeamenabletohavingtolearnhowtouset
hesystem(asopposedtothegeneralpopulation,whotendtodemandimmediateeaseofuseforspe
echtechnologies.)6.4HUMANFACTORSDESIGNOFTHEINTERFACEVoiceuserinterfacesoftensuff
erfromalackofhumanfactorsinvolvementdur-ingdesignanddevelopment.Thereasonsforthi
sarelogisticalandtechnical:Speechsystemsaremostoftendeployedinatelephonyenvironm
ent,typicallyinthecallcenter,andVUIsrelyonadifferenttechnologythanGUIsandthusreq
uiredifferentdeveloperskillsets.Speechsystemdevelopersareoftennotaccustomedtowor
kingwithhumanfactorsprofessionalsandmayhavedifficultyadjustingtonewproceduresand
timelinesimposedbyuser-centereddesignprac-tices.Inlargeorganizations,theremaybea
nexistingstaffofhumanfactorspro-fessionalsworkingonwebsiteandsoftwareissues.Howe
ver,thecallcenterisnotthetypicalrealmforhumanfactorsspecialists,andIVRmanagersma
ybeunawareofthehumanfactorsskillsavailabletothem.Consequently,VUIdesignoftenproc
eedswithoutanyhumanfactorsinvolvementatall.Sadly,evenwhenhumanfactorsfolksareinv
olvedinspeechprojects,mostlackexpertiseinspeechtechnologiesandhumanconversationa
lbehaviorandthusdonotcontributeasmuchastheydoinGUIprojects.
6VoiceUserInterfaces214

In the vast majority of cases, the sole representative of human factors onspeech
system projects is the VUI designer (VUID). Much like the early humanfactors pi
oneers, VUIDs are a heterogeneous group made up of ex-developers, way-ward lingu
ists, telephony managers, assorted psychologists, and many others withlittle or
no relevant formal training in speech technology or human factors. Thereare curr
ently no formal degree programs specifically in VUI design; however,Auburn Unive
rsity has an engineering program with a strong VUI emphasis underthe direction o
f Juan Gilbert. What VUIDs may lack in formal human factors train-ing, most make
up for in their enthusiasm to bring strong user-centered designpracticestospeec
hprojects.TogetasenseoftheVUIDcommunity,considervisiting the VUIDs Yahoo group a
t http://tech.groups.yahoo.com/group/vuids .Strongsupportforhumanfactorsinspeech
technologysystemshascomefromAVIOS,theAppliedVoiceInputOutputSociety.AVIOSisanonp
rofitmembershiporganizationforspeechtechnologyprofessionalsthathasbeeninfluentia
lforover20years.AVIOSoftensponsorsspeechtechnologyconferencesandpreviouslyhadedi
torialcontroloftheInternationalJournalofSpeechTechnology,animportantsourceofrigo
roustechnicalanduserinterfaceinformationforspeechsystems.MoresothanintheGUIworld
,VUIDsneedamixofbothtechnicalanddesignskills.GUIinteractionsoccurusingamouse,key
board,orsomeotherpre-ciseinputdevice,andGUIdesignerscanreasonablyassumethatuseri
nputwillbeinterpretedcorrectlybythesystem.Inspeech-enabledapplications,well-form
eduserinputwillbemisinterpretedinasignificantproportionofcases.Mostcurrentspeech
recognitionalgorithmsboastnear100percentrecognitionaccu-racy,butinpractice,error
ratesof10to20percentormoreatagivenrecognitionstatearenotuncommon.Therefore,theVU
Idesignermustunderstandthelimita-tionsofthetechnologyanddesignusingstrategiestha
tminimizetechnology-levelerrors.VUIDsalsoserveasthevoiceoftheuserinspeechproject
s,muchasotherhumanfactorsprofessionalsdointheGUIworld.Onmanyprojects,theVUIdesig
nerisworkingalonewithoutanyotherhumanfactorssupport,soitisvitaltohavefirmguideli
nesforappropriateuser-centereddesignprocesses,aswellasguerrillatechniquestofallb
ackonwhentheidealisnotpossible.Theabilitytojustifyandevangelizeforuser-centeredd
esign(UCD)processesisvitalforVUIDswhoareoftenthefirstexposureprojectmembershavet
ohumanfactors.6.4.1User-CenteredDesignforSpeechProjectsUser-centereddesignpracti
ceforspeechtechnologiesissimilarbutnotidenticaltoestablishedUCDtechniques.Atagen
erallevel,theprocessissimilar:Gatherinformationfromusersandthebusiness,designapr
eliminaryinterface,testwithrealusers,modifythedesign,anditerate.Therearedifferen
ttwistsateachstage,however,becauseoftheuniquepositionofspeechtechnologies.During
requirementsgathering,VUIdesignershaveuniquesourcesofinfor-mationrelevanttothevo
icechannel.Sincemostspeechtechnologyisdeployed

6.4HumanFactorsDesignoftheInterface215

overthephoneincallcenters,callcenteragentsandmanagerscanprovidevalu-ablebackgrou
ndontheusersstateofmindwhencalling,terminologyusedandunderstoodbycallers,andnuanc
esofcallingpatternsthatmaynotbeobviousfromothersources(e.g.,userswhocallduringbu
sinesshourstendtobemoreimpatientthanuserswhocalllaterintheday).Incallcenterswith
existingIVRs,usagestatisticscanrevealfrequencyofuseforvariousfunctions,patternso
fuseovertime,andpointsatwhichuserstendtoabandonself-serviceinfavorofliveagents.E
xistingIVRsalsooffertheopportunitytolistentorealcallssothatdesignersmayunderstan
dthecurrentuserexperienceandthusgaininsightonhowtoimproveit.Anotherdatapointinth
erequirementsprocessisthedetailsoftheparticularspeechrecognitionsystembeingusedo
naproject.Whileallrecognitionengineshavethesamebasiccapabilities,thereareamultit
udeofdifferencesinsystemssoldbydifferentvendorsthatcanaffecttheVUI.Understanding
whatspeechtechnologyisavailableandhowtoputittobestuseisafoundationfordesigningan
effectiveVUI.TheprocessofVUIdesignissimilartoGUIdesignexcept,ofcourse,forthemoda
lityofthepresentationlayeroftheinterface.Inspiteofbeingauditory,VUIsneedtobedocu
mentedvisually.MostVUIdesignersrelyonacombinationofflowchartstorepresentthelogic
alflowoftheapplication(inaprogramsuchasVisio)andtextdocumentstospecifytheprompts
anddetailedbehaviorateachrecognitionstate.Thiscombinedcall-flowplustext-documentar
rangementissuboptimalanddifficulttomaintain,butmostVUIdesignershavenobetteralter
native.Thereisnotoolinwideusethatcapturesbothcallflowanddetailedspecificationinf
ormationinasingleprogram.6.4.2HowMuchSpeechShouldWeUse?Oneofthemostimportantques
tionsforVUIdesignersishowtousespeechtech-nologysmartlytoachievebusinessgoalsandm
eetuserneeds.Theaimistoavoidusingspeechtechnologyforitsownsake,andinsteadlookfor
opportunitiesinwhichspeechprovidesdistinctvaluetotheuserandtothebusiness.Theseop
por-tunitiesariseunderseveralconditions:FThecurrenttechnologyunderaddressesusero
rbusinessneeds,suchastypingcitynamesusingthenumericaltelephonekeypad.FThetypical
contextofuseisaneyes-busy,hands-busysituation,suchasanyapplicationwhereusersarel
ikelytobemobile.FUsershaveadisabilitythatlimitstheirabilitytousetheirhands.(Seet
heUnited States Access Board, Section 508 , and the Web Accessibility Initiative
oftheW3Cforfurtherrecommendationsforspeechandaccessibility.)FThewaittimetospeak
toaliverepresentativeisunacceptabletotheusersfortheirtask.
6VoiceUserInterfaces216

Somefactorsweighagainstusingspeech.Manyusershaveconcernsaboutpri-vacyandmaynotbe
willingtospeakcertaininformationaloud(e.g.,passwords,SocialSecuritynumbers,accou
ntnumbers,medicaldata,otherfinancialinforma-tion).Ifyouchoosetospeech-enablecoll
ectingthistypeofsensitiveinformation,besuretoallowusersanalternatemethodtoentert
heirinformation.Speechisalsoapoorfitforcollectingunconstrainedalphanumericsequen
cesbecauseofthediffi-cultyofrecognizingsequencesofvariablelengthscomposedoflette
rsandnumbersthatarenotoriouslyconfusable.Variousnoisyacousticenvironmentsalsomak
espeechalessdesirablesolution.Inmanyothersituations,speechworksreasonablywellfor
thetaskathand,butmaynotofferanyrealbenefitsoveranothertechnology(liketouch-tonei
nput).Inthesescenarios,thecaseforusingspeechismoreabouttheabilityofspeechinterfa
cestogeneratepositivecustomerexperienceforusers.Whenitisclearthatspeechrecogniti
ontechnologyisagoodfitforthetask,thereisstillthechoiceofthetypeofdialogandgramma
rtouse.Fortelephone-basedspeechsystems,adirecteddialogapproachisoftenappropriate
.DirecteddialogreferstotheclassofVUIsinwhichthesystemoffersusersthechoiceamongaf
initesetofoptions,andthoseoptionsarerecognizedagainstastandardgrammardefinedinad
vance.Directeddialogcanprovideeffectiveandsatisfyinginteractionsforwidevarietyof
routineinformationretrievaltasks(e.g.,gettinganaccountbalance,check-ingflightsta
tus)andformanytransactionssuchasplacinganorderorchanginganaddress.Inthespeechind
ustrypress,directeddialogissometimesportrayedasunnaturalorinferiortoso-callednatu

ral-languagesystems.Naturallanguageisanimprecisetermthatisuseddifferentlybyeachau
thor.Acommoninterpretationfornaturallanguageisthesortofminimallycon-strainedrespon
sespossiblewhenusingastatisticallanguagemodel.Asexplainedabove,SLMsaresimplyamor
eexpansivegrammar,nottruelanguageunder-standing,andcanbecostlytoimplement.Andbec
ausemostusersareunaccus-tomedtospeakingnaturallytoautomatedsystems,theydonotknow
howtobestformtheirresponses,andarethusuncomfortablewiththesemorenaturalinterfaces.
Tothecontrary,myexperienceshowsthatwell-designeddirecteddialoginterfacesareoften
themostsuccessfulandcomfortableforusers.6.5TECHNIQUESFORTESTINGTHEINTERFACEColle
ctingdatafromendusersofspeech-enabledapplicationsinusabilitytestsisavitalstepina
nyspeechproject,asitisforanyautomatedsystem.Usabilitytestingforspeech-enabledapp
licationsshouldbeginattheearlieststagesofaprojectinordertobemaximallyeffective.E
arlyuserinputallowsthedesignertomakechangeswhenitisstillrelativelyquickandinexpe
nsivetomakethem.Moreover,subjectivedatafromendusersisanimportantfactorininterpre
tingobjectivemeasuresofthetechnicalperformanceofaspeechapplication.Inisolation,m
ea-suressuchasrecognitionrateorcallcontainmentdonotgiveorganizations
6.5TechniquesforTestingtheInterface217

ameaningfulwayofmeasuringtheoverallsuccessofaspeechapplication.Onlythecombinatio
nofobjectiveperformancemeasures(suchasrecognitionrate)coupledwithsubjectivemeasu
resofperformance(suchasperceivedfrequencyofbeingmisunderstoodbythesystem)allowsu
stoknowifaspeech-enabledapplicationisgoodenough(Hura,2007a).Asinanyusabilityproj
ect,thereisatrade-offbetweenhowearlyyoutestandhowrepresentativethecollecteddataa
re.TherearedirectcorrelatesinspeechtomanyofthetestingtechniquesthatareusedinGUIp
rojects.ForformalVUIusabilitytests,thereisachoicebetweenusingactualspeechrecogni
tionduringtestingandmimickingit.Thisisasignificantdecisionbecausefullyfunctional
speechprototypesaretypicallycodedbydevelopers(ratherthanusabilityspecialists)and
thususuallyoccurlaterintheprojectlifecycle.Thealternativeisaso-calledWizardofOzt
estinwhichnoapplicationhasbeendevelopedatall.Instead,ausabilityspecialist(thewizard)suppliesspeechrecognitionbylisteningtouserinputandthenplayingsoundfilesofprom
ptstomimicthebehaviorofthespeechapplication(Hura,Polkosky,&Gilbert,2006).Therear
ebenefitsanddownfallstoeachtypeoftesting.WizardofOz(WOZ)testingisoftendiscusseda
sbeingadiscountusabilitymethodequivalenttoprototypetesting,butnocodingisrequired.A
lthoughthereisnocoding,thebenefitisexaggeratedinrelationtotheoverallamountofwork
torunaWOZtestwhichisequivalenttoaprototypetest.WOZtestingrequiresacompleteVUIdes
ign,professionallyrecordedprompts,asystemforplayingpromptsoverthephone,andtheful
l-timeinvolvementoftwohumanfactorspro-fessionalsforeverytestsession(onetoactasth
ewizardandonetofacilitatetestsessionsbothdemandingjobs).Itiscertainlypossibletoru
nWOZteststhatcompromiseononeormoreofthesefactors,butthetestisthennotatruesubstit
uteforprototypetesting.Forexample,ifyouchoosetohavethewizardsimplyspeaktheprompt
sratherthanusingrecordedprompts,therearemultipledisadvantages:First,youlosetheab
ilitytogatherfeed-backaboutthevoicetalent;second,thepromptwillbespokendifferentl
yeverytimeitisuttered,whichintroducesunwelcomevariabilityintothetestingprocess.B
ecausesomuchoftheuserexperienceisboundupintheminutedetailsoftheprompts,thiscompr
omiseishardlyworthwhileinmanycases.Inspiteoftheinten-sivepreparationrequiredbyth
eusabilityteam,WOZtestsaretypicallyconductedearlierintheprojectthantestsusingfun
ctionalspeechrecognitionprototypes.Testingwithafunctioningspeechprototypeisthego
ldstandardforassessingVUIusability(Hura,2003).Prototypesgivearealisticviewofthef
lowoftheauto-matedconversation,andthedatathatcanbegatheredinaprototypeusabilityt
estareanorderofmagnitudemorerealisticthanthedatafromaWOZtest.Thetimingoftheinter
actionisalsomorerealisticinprototypetesting.Itisnotori-ouslydifficultinaWOZtestt
oaccuratelysimulatetheresponselatencyofaspeechapplication,andwizardstypicallyres
pondmorequicklythanthesysteminsomesituationsandmoreslowlyinothers.Onelater-stage
techniquethatisunderusedinVUItestingisfieldtests,inwhichusersinteractwithaVUIfro
mtheirtypicalusageenvironment.Fieldtests
6VoiceUserInterfaces218

arelogisticallycomplicatedbecausetelephone-basedsystemscanbeusedany-whereandwhil
einmotion(asopposedtocomputers,whicharestationarywhenused).Additionally,becausem
anycommercialspeechapplicationsareusedrarelybyagivenindividual,itcanbedifficultt
oarrangetobepresentatthetimeofuse.Instead,mostVUIprojectsrelyonlivecallmonitorin
g,orcallrecording,inwhichusabilityspecialistscanlistentousersastheyhavereal-life
interactionswithaVUI.Thesetechniquesdonotallowthedesignertogivetheuserasetoftask
stocomplete(hencewecanneverbetotallysureofausersmotivationintheinteraction),andth
ereisgenerallynoabilitytospeaktousersdirectlytounderstandreactionstotheirexperie
ncefollowinganinteraction(althoughsurveysaresometimesadministeredatthistime).Afi
nalconsiderationforVUIusabilitytestingislogistics:VUIusabilitytestingcantakeplac
ewiththeuserandfacilitatorinthesamelocationorremotelyoverthephone.Thetechnologyi
sreadilyavailabletoallowuserstointeractwithaspeechapplicationwhileusabilityspeci
alists(andothers)listenin,alloverastan-dardtelephoneconnection.Thebenefitsofover
-the-phonetestingarethatitsavestheconsiderableexpenseofsecuringatestingfacility,
andthetimeandcostasso-ciatedwithhavingparticipantstraveltothisfacility.Thedownfa
llsallrelatetothefactthatwecanonlylistentotheinteraction,notwatchit,whichdeprive
susofcuesthatusersgivethroughtheirfacialexpressions,postures,andbodylanguage.InpersonusabilitytestingforVUIsallowsustowatchandlistentointerac-tions,andtherefor
erecordbothaudioandvideooftestsessions.Buildingrapportwithauseriseasierwhenyouar
ephysicallyinthesameroomwithoneanother,whichencouragesuserstogiveuncensoredrespo
nsesduringinterviews.Thedownfallsofin-persontestingarethetimeandcostassociatedwi
thbringingagroupofparticipantstoonelocationfortesting.In-persontestingalsolimits
testparticipantstothosewhoarewillingtotraveltothetestlocation,asopposedtoover-th
e-phonetestingwhereremoteuserscanbeincludedeasily.OnefinalVUItestingmethodistuni
ng.Tuningistheprocessofoptimizingperformanceofaspeechapplicationoverall.Thefocus
intuningisverybroad:Everythingfromlow-levelrecognitionparameterstopromptsisevalu
ated.Tuninginvolvesrecordingliveuserinteractionswithaspeechsystemseveralhundredin
teractionsareatypicalanalysissetforaroundoftuning.Recordedcallsaretran-scribedan
dtranscriptionsareusedasabasisfordeterminingrecognitionaccuracyintheapplicationo
verallandstatebystate.Utterancesarecategorizedasingram-maroroutofgrammarandascor
rectlyrecognized,incorrectlyrecognized(orfalseacceptinwhichanout-of-grammarutter
anceismistakenlyrecognizedasanin-grammarutterancethatfailstoberecognized),orcorr
ectlyrejected.Alongwithcomputingstatistics,tuningalsoinvolveslisteningtoasmanyus
ercallsaspossible.TuningcallsaresometimestheonlyopportunitythatVUIdesignershavet
oobservetheirdesigninactionanddeterminewhetherconclusionsfromusabilitytest-ingho
ldintherealworld.Basedonthecomputedstatisticsandinsightsgainedfrommonitoringcall
s,designerscanidentifyissuesandproposesolutions.Tuningofferstheopportunitytotest
sometypesofsolutionsbeforetheyareputintopractice.Itis
6.5TechniquesforTestingtheInterface219

oftenpossibletomodifyagrammar,runtherecordedspeechsamplesthrough,andobservethere
sults.Iftuningdataareinterpretednarrowly(applyingresultsonlytospecificdialogstat
esratherthanoverallacrosstheapplication)andusedjudi-ciouslywithoutmakingtoomanyas
sumptionsaboutwhatusersarethinkingitispossibletocometosomeconclusionsaboutusabili
tyfromtuningdata.TherearefewestablishedmetricsofVUIusabilityavailableorinwideuse
.GuptaandGilbert(2005)proposetheholisticusabilitymetric,whichallowsthetestertoap
plyweightstofactorsjudgedtobemostimportantforindividualinter-faces.Becauseeachte
stermayassignweightsdifferently,thismetriccanbetrickytointerpretuniversally,butc
anbequitevaluablefortestingoneapplicationovertimeorasetofsimilarapplicationsusin
gthesameversionofthemetric.Polkosky(2005)hasconstructedanopinionmetricspecificto
VUIsforcollectingsubjectiveuserdata.Herresearchshowsthatusersperceivethebehavior
ofaVUIasevi-denceofhigher-orderfactors,suchastheoveralllevelofcustomerserviceori

en-tationwithinanorganization.Thismetricisvaluableinthatitaddressesissuesbeyondp
ureusability,suchasdesirabilityandcustomersatisfaction,andithaswideimplicationsf
orexactlywhichfactorsinaVUIinfluenceuseropinions.NeitheroftheseVUImetricshasbeen
puttowidescaleuseintheindustry.Instead,organizationstendtorelyonhome-grownsurvey
sorcustomersatisfac-tionsurveysdesignedforWeborothermedia,withafewminoradjustmen
tsforspeech,toevaluatethequalityandusabilityofspeech-enabledsystems.Theitemsonsu
chsurveysarerarelyvettedforvalidityorreliability,andassuchthedatagatheredfromthe
marelessthanideal.OnenotableexceptiontothispatternistheservicesofferedbyVocalLab
oratories,whichoffersprofessionallydesignedsurveysandtheabilitytoquicklycollectd
atafromverylargepopulationsofusers.6.6DESIGNGUIDELINESThereareanumberofreference
sthatprovideanoverallphilosophyofVUIdesign,includingCohen,Giangola,andBalogh(200
4),Ballentine(2001),Kotelly(2003),andLarsonetal.(2005).Theguidelinespresentedher
eareintendednotasall-inclusive,butratherasacollectionofpracticaltipsforwritingef
fectiveprompts.Thedatasup-portingthesetipsderivefromobservationsofhundredsofuser
sinteractingwithdozensofspeech-enabledIVRapplicationsduringusabilitytests.6.6.1D
oNotWorrySoMuchabouttheNumberofMenuOptionsGeorgeMillers50-year-oldresearchonshort
-termmemory(1956)isoftenportrayedasthefinalwordonthepermissiblenumberofauditoril
ypresentedmenuoptions.Thethinkinghereisthatitisprimarilythelimitsofauditoryshort
-termmemorythat
6VoiceUserInterfaces220

determinetheusabilityofavoicemenu.Thereisakerneloftruthhereitiscer-tainlypossible
tooverloaduserswithtoomuchauditoryinformationbutthestrictlimitofseven-plus-or-min
us-twosimplydoesnotholdforVUImenus(Hura,2007b).TheflawinapplyingMillerverbatimis
thatusingamenuisnotarecalltaskbutratheraselectiontask.Usersshouldneverneedtohold
anentireVUImenuinmemoryinordertomaketheirselection.Thegoalistohavethemevaluatean
dacceptmenuoptionsonebyoneor,atmost,rememberafewlikelycandidates.6.6.2PresentMen
uOptionsinaWayThatMakesThemComprehensibleandEasytoRetainEspeciallyinmenuswithmor
ethanasmallhandfulofoptions,thebestwaytoenablesuccessfulselectionistoensurethatt
heoverallsetofmenuoptionsisdescriptiveanddistinct.Descriptivemenuoptionsgiveuser
stheimmediatesenseofIknowwhatthatis!Thisenablesuserstoevaluateanddiscardnondesired
optionswithoutholdingtheminmemory.Similarly,userswillimmediatelyknowthedifferenc
esbetweendistinctmenuoptions,whicheliminatestheneedtoremembertheentirelistwhilem
akingtheirmenuselection.Currentshort-termmemorytheoristsagreethattherearemanyfac
torsbeyondthenumberofitemsthataffectauditoryshort-termmemory,suchaswordlength,fr
equency,familiarity,andinter-itemdiscriminability(i.e.,howdistinctitemsonalistar
efromoneanother[Nairne,2002]).Theimportanceofdescriptivenessissupportedbyrecentr
esearchinhumancomputerinteractionshowingthatmeaningfulmenuoptionlabelingismoreimp
ortantthaninformationarchitectureforusers(Resnick&Sanchez,2004).6.6.3DoNotLetPer
sonaGetintheWayofEffectivenessUserswillalwaysattributeapersonatoaVUI,butpersonas
houldalwaysbesubor-dinatetoeffectiveness.SomedesignersarguethataVUIshouldneversa
yanythingthatapersonwouldnotsay,butfollowingthisrulewilloftenleadtheVUIdesignert
owritepromptsthatencouragetheusertooverestimatetheabilitiesofthespeechsystem.Ins
tead,focusoncraftingpromptsthatleaduserstogiveappropriateresponses.Itisfarbetter
tobesomewhatunnaturalthantotakeauserintoanerrorstateforthesakeofanice-soundingpr
ompt(Klie,2007;Rolandi,2007).6.6.4ConsiderErrorSourcesWhenWritingError-HandlingP
romptsUsersfindthemselvesinerrorconditionsforavarietyofreasons,manyofthemunrelat
edtotheVUIitself.Therearedistractionsintheenvironment
6.6DesignGuidelines221

thatmaytaketheusersattentionawayfromtheVUI,andbackgroundnoisethatmaytriggermisrec
ognitionsortheremaytrulybeanissuewiththeinterface.Errorpromptingmustworkforalluse

rs.Agoodpracticeforerrorpromptsistoallowuserstobargeinquicklyandself-correct(for
userswhoknowwhattheywantandsimplyneedachancetorepeatit),buttogiveaddi-tionalinfo
rmationtohelpuserswhowereunabletogiveagoodresponseattheinitialprompt.6.6.5Provid
eLandmarksBecauseaudio-onlyinterfacesarenonpersistent,oneofthemostchallengingthi
ngsforusersisstayingoriented.Usinglandmarkpromptscanserveasawaytoacknowledgeause
rschoiceandreinforceasenseofplace.Forexample:SYSTEM:Youcansay:accountbalances,las
tcontribution,transfers,allocations,orlifeinsurance.USER:Transfers.SYSTEM:Transf
ers.Youcansay:hearpendingtransfersormakeatransfer.6.6.6UseHintsandRemindersPrese
ntinginformationtousersbeforetheyareinacontextinwhichtheycanmakeuseofitisineffec
tive.Themoreusablealternativeistoprovideinstructiontousersinsmallchunksattimeswh
entheyarelikelytoneedtheinformation(Hura,2006).Hintsareinformationpresentedtouse
rstogivethemcontext-specificinstructionforthecurrentdialogstateortoinformthemofs
hortcutsoralternatemethodsofcompletingtasksinthesystem.Forexample,ifauserhasreac
hedtheorderstatusstateintheapplicationbynavigatingthroughsev-eralmenus,consideroff
eringahintattheorderstatusmodule:Nexttime,youcansayorderstatusatthemainmenutogether
equicker.Reminders,notsurprisingly,arebriefpromptsthatremindtheuserofarelevantbit
ofinforma-tionataparticularstateinthedialog,suchasYoucanspeakyouraccountnumberork
eyitin.6.6.7FavorModelsoverInstructionsItissometimesnecessarytoprovideanexampleto
usersofhowtoformaresponse.Inthesecases,itisofteneasierandmoreeffectivetosimplymo
delthedesiredsortofresponsethantoexplainit.Considertwoversionsofapromptrequestin
gdateofbirthtobeenteredasaneight-digitnumber.Thesecondver-sion,usingamodel,ismor
econciseandmucheasiertounderstandthanthefirstversion.
6VoiceUserInterfaces222

Version1:Pleaseenteryourdateofbirthasaneight-digitnumber,withtwodigitsforthemont
hfollowedbytwodigitsforthedayandfourdigitsfortheyear.Version2:Pleaseenteryourdat
eofbirthasaneight-digitnumber.So,March14,1986,wouldbe03141986.6.6.8UseDiscourseM
arkersLiberallyRemember,thisisaconversation!Discoursemarkersservetoconnectuttera
nceswithinadialog.Discoursemarkershelpestablishasequence,changethetopic,providea
cknowledgment,andconveythespeakersattitudetowardwhatsbeingsaid.Thisdoesnotmeanthat
theVUIshouldrespondtoeveryuserutterancewithAllrightorOkaythistendstomaketheVUIsoundov
erlyperky.Instead,varythediscoursemarkerswiththeconversationalcontextasinthefoll
owingconversation(discoursemarkersareinitalics):SYSTEM:Mainmenu.Youcansay:balanc
es,transfers,transactionhistory,orloans.USER:Balances.SYSTEM:Allright,balances.T
ogetyouthatinformation,weneedtogetyouloggedin.First,pleaseenteryouraccountnumber
.USER:12345678.SYSTEM:Next,enteryourpassword.USER:54321.SYSTEM:Thanks.Thebalanceon
account123-45-678asofMay23,2007is...6.6.9AllowforConversationalRepairHumanconver
sationisfullofsmallproblems.Speakershesitate,startanutter-anceandthenbackoffands
tartagain,andmispronouncewords;listenersmisun-derstandandrequestclarification.We
doallofthiswithoutthinkingaboutitandwithouttheconversationfailinginavastmajority
ofcases.WhenthereisasmallprobleminaVUIinteraction,usersnaturallyfallbackontheirt
echniquestofixconversations(recalltheexampleoftheuserspeakingmoreslowlyandclearl
yafterbeingmisunderstood).Incraftingerrorprompts,beawareofthistendencyandwritepr
omptsandgrammarsthataccommodatetheuser.Acommonexamplethatiseasytoaccom-modateisp
romptingfollowinganexplicitconfirmation.Ifwhatisbeingcon-firmediswrong,usersofte
nwanttocorrectthemistakeinasingleturn:SYSTEM:Toconfirm,yourelookingforflightsfrom
LosAngelestoBostononMay23rd.USER:No,toAustin.
6.6DesignGuidelines223

To accommodate such corrections, build grammars that include no plus any cor-recti
on the use might offer (here, no plus city names).6.6.10 Be Careful with Terminolo
gyUsing terminology that is familiar and comfortable for users is a basic guidel

ine forany automated system. In the VUI world, this guideline takes on more impo
rtancebecause the interface is nonpersistent. If users do not understand a term
on aweb page, they can rescan the options with minimal effort. In a VUI, if user
s missa term, they have to spend the time to listen to the entire list of option
s again.Steer clear of jargon and branded or technical terms in a VUI unless use
rs usethese terms.6.6.11 Give Users an Escape HatchUsers detest being held capti
ve in an automated system. Most users are willing totry automation to accomplish
simple tasks, but when the automation fails, theyjust want a real person. Users
also tend to pick up the phone when they havealready established that they have
a problemmeaning that a user may be enter-ing his interaction with a call center
VUI already certain that automation cannotsolve his problem. Businesses deployi
ng speech often want to deflect as manycalls as possible away from live agents a
nd want to completely prevent users fromrequesting a live agent. These efforts a
re often counterproductive in that usersalways find the escape hatch, and if you
have made them work hard to find it,they will be frustrated and angry when they
get there. Take the middle ground:Offer to transfer to agents after you have ev
idence that the user is having a prob-lemsuchasrepeatedtime-outsornomatches.Itis
helpfulnotonlytocountthenumberoferrorsatasingledialogstatebutalsotouseaglobalcou
nterthatkeepstrackoferrorsfortheentireinteraction.6.7CASESTUDYA case study for a
speech interface can be found at www.beyondthegui.com .6.8FUTURETRENDSWhatsthene
xtbigthingforVUIs?Thisisnecessarilylinkedtothenextbigthingforspeechtechnologyove
rall.Onetrendthathasemergedrecentlyisvoicesearchusingspeechrecognitiontospecifyas
earchtermandthenreceivingtheoutputauditorilyordisplayedastextonsomedevice.Onehig
hlyanticipated
6VoiceUserInterfaces224

voice search application is 1-800-GOOG-411, Google Voice Local Search, from Google Labs ( http://labs.google.com/goog411/ ). Google is the search engine giant
ontheWeb.TheGoogleLabsdecisiontoofferaspeechoptionforspecifyingsearchesovertheph
onemakesaportionoftheWebaccessibletomobileusersinawaythatithasnotbeenuntilnow.Se
archingtheWebusingeventhemostadvancedsmartphoneispainful.Typingsearchtermsonatin
ykeyboardisoverwhelm-inglymoredifficultthansimplyspeakingtheterm.Speech-enabling
searchisalogicalstep.Itisinterestingtonotethattheuserinterfaceof1-800-GOOG-411is
quitemin-imal.Thereisnothingflashyorchattyaboutit.Thepromptsarebriefandtothepoin
tandarenotnecessarilyphrasedexactlythewayapersonwouldphrasethingsinahuman-to-hum
anconversation.TheVUIdesignersatGoogleclearlyconceiveof1-800-GOOG-411asatool,not
asaliveoperatoremulator.Thepromptsfor1-800-GOOG-411arecertainlypleasantandnotata
llterse,buttheyrepresentastylisticshiftawayfromthemoreverbose,morehuman-aspiring
systemsofthelastfiveyears.Asofthiswriting,1-800-GOOG-411isstilldescribedasanexpe
rimentalser-viceandislimitedtoYellowPagestypesearchesforlocalbusinesses.Anumberof
thefeaturesof1-800-GOOG-411arenovelandwouldbeusefulinbroadersearchesaswell.Forex
ample,theusercanbrowsetheresultsofasearchinmul-tipleways;usingsimplevoiceortouch
-tonecommands,userscannavigatethelistofresultsandselecttheresultstheywant.Altern
ately,userscanchoosetohavethelistofresultssentviashortmessageservice(SMS,ortextm
essaging)totheirmobilephone.Googlegivesusersthechoiceofmodality(speechandtouch-t
oneforinput,audioandSMSforoutput)throughouttheapplication.Thisdecisiontobemodali
tyagnosticshouldservetomorrowsincreasinglymobileuserswell.Whenthecontextofuseisany
where,anytime,theuserinterfacemustallowuserstoselectthebestwaytointeractbasedonth
eircurrentcircumstances.Speechtechnologyisalsopoisedtomoveintothehome.Appliances
,homesecuritysystems,andsmarthomesystemswillsoonbesupplementedwithspeechtechnology
.Inthesecases,speechrecognitionalgorithmsareunlikelytoresideonthedevicesthemselv
es,butwillbehousedonacentralservertowhichthelocaldevicecommunicatesinrealtime.Th
esedevelopmentswillbepropelledinitiallybytheadvantagesthatspeechoffersintermsofa
ccessibilityforthemobilityimpaired.Beingabletospeakcommandsinordertocontrollight
s,tele-visions,heating,cooling,andthelike,wouldprovidecontrolandfreedomtoindi-vi
dualswhotodaycannotmanagethesetasksindependently.Itislikelythatspeechinthehomewi

llbeaphenomenon(likecurbcuts)inwhichanaffordancedesignedtoassistasmallgroupendsu
pbenefitingthelargercommunityinunex-pectedways.Finally,thereisthepromiseofspeech
formobileandhandhelddevices.Mul-timodalspeech,touch,andgraphicsdeviceshavebeenpr
omisedbymanycompanieslargeandsmallformorethan10years,buttherehasyettobeatrue
6.8FutureTrends225

multimodalbreakthrough.Theproliferationofmobilewirelessdeviceswithsmallscreensan
daudioinputandoutputmakesthepossibilityofrealmultimodalitymorelikelyoverthenext1
0years.Likeanytechnology,breakthroughswillcomewhenmultimodalityenableseasyaccess
toanactivitythatiscumbersomeorimpossibletoday.REFERENCESAttwater,D.,Ballentine,B
.,Graham,M.,&Hura,S.(2005).Voiceuserinterfacedesign:Whatworks,whatdoesnt.Paperpre
sentedatSpeechTEKWestConference.Baber,C.,&Noyes,J.(1993).InteractiveSpeechTechno
logy:HumanFactorsIssuesintheApplicationofSpeechInput/OutputtoComputers.London:Ta
ylor&Francis.Ballentine,B.,&Morgan,D.(2001).HowtoBuildaSpeechRecognitionApplicat
ion.SanRamon,CA:EnterpriseIntegrationGroup.Blumstein,S.,&Stevens,K.(1981).Phonet
icfeaturesandacousticinvarianceinspeech.Cognition10:2532.Campbell,J.(1997).Speake
rrecognition:Atutorial.ProceedingsoftheIEEE85:143762.Cohen,M.,Giangola,J.,&Balogh
,J.(2004).VoiceUserInterfaceDesign.Boston:Addison-Wesley.Denes,P.,&Pinson,E.(199
3).TheSpeechChain.NewYork:WorthPublishers.Furui,S.(1996).Anoverviewofspeakerreco
gnitiontechnology.InLee,C.,&Soong,F.,eds.AutomaticSpeechandSpeakerRecognition.Bo
ston:KluwerAcademic,31-57.Gupta,P.,&Gilbert,J.(2005).Speechusabilitymetric:Evalu
atingspokenlanguagesystems.Proceedings11thInternationalConferenceonHumanComputerI
nteraction,LasVegas,NV(CD-ROM).Grice,H.P.(1975).Logicandconversation.InCole,P.,&
Morgan,J.,eds.,SyntaxandSemantics,Vol.3:SpeechActs.NewYork:AcademicPress,41-58.H
eider,F.(1958).ThePsychologyofInterpersonalRelations.NewYork:JohnWiley&Sons.Hura
,S.(2003).Passingthetest.SpeechTechnology8.5(September/October).Hura,S.(2005).Gr
eatexpectations.SpeechTechnology10.6(November/December).Hura,S.(2006).Givemeahin
t.InMeisel,W.,ed.,VUIVisions:ExpertViewsonEffectiveVoiceUserInterfaceDesign.Vict
oria,BC:TraffordPublishing.Hura,S.(2007a).Howgoodisgoodenough?SpeechTechnology12
.1(January/February).Hura,S.(2007b).Mybigfatmainmenu.SpeechRecognitionUpdate.Hur
a,S.,Polkosky,M.,&Gilbert,J.(2006).Point,counterpoint:Whatthreeexpertssayaboutsp
eechusability.UserExperience5:2.InternationalOrganizationforStandardization.(199
8).ISO9241:11.Ergonomicrequire-mentsforofficeworkwithvisualdisplayterminals(VDTs
)Part11:Guidanceonusabil-ity.Geneva,Switzerland:ISO.Klie,L.(2007).Itsapersona,nota
personality.SpeechTechnology12.5(June).Kotelly,B.(2003).TheArtandBusinessofSpeec
hRecognition:CreatingtheNobleVoice.Boston:Addison-Wesley.Larson,J.,Applebaum,T.,
Byrne,B.,Cohen,M.,Giangola,J.,Gilbert,J.,NowlinGreen,R.,Heb-ner,J.,Houwing,T.,Hu
ra,S.,Issar,S.,Kaiser,L.,Kaushansky,K.,Kilgore,R.,Lai,J.,Leppik,D.,
6VoiceUserInterfaces226

Mailey,S.,Margulies,E.,McArtor,K.,McTear,M.,&R.Sachs.(2005).Tenguidelinesfordesi
gningasuccessfulvoiceuserinterface.SpeechTechnology9.7(January/February).Lea,W.(
1980).TrendsinSpeechRecognition.EnglewoodCliffs,NJ:PrenticeHall.Massaro,D.(2006)
.Speechperceptionandrecognition:Theoriesandmodels.EncyclopediaofCognitiveScience
.NewYork:JohnWiley&Sons,1-9.McClelland,J.,&Ellman,J.(1986).TheTRACEmodelofspeech
perception.CognitivePsy-chology18:186.Mori,M.(1970).Bukiminotani.Theuncannyvalley
.MacDorman,K.F.,&Minato,T.,trans.Energy7(4):3335.Nairne,J.(2002).Rememberingovert
heshort-term:Thecaseagainstthestandardmodel.AnnualReviewofPsychology53:5381.Nass,
C.,&Brave,S.(2005).WiredforSpeech:HowSpeechActivatesandAdvancestheHumanComputerRe
lationship.Cambridge,MA:MITPress.Nielsen,J.(1993).UsabilityEngineering.SanDiego:
AcademicPress.Pinker,S.(1994).TheLangaugeInstinct.NewYork:HarperCollins.Pitt,I.,
&Edwards,E.(2003).TheDesignofSpeech-BasedDevices.London:Springer-Verlag.Polkosky

,M.(2005).TowardaSocialCognitivePsychologyofSpeechTechnology:AffectiveResponsest
oSpeech-Basede-Service.PhDdiss.,UniversityofSouthFlorida.Pruitt,J.,&Adlin,T.(200
6).ThePersonaLifeCycle:KeepingPeopleinMindThroughoutProductDesign.SanFrancisco:M
organKaufmann.Rabiner,L.(1989).AtutorialonHiddenMarkovModelsandselectedapplicati
onsinspeechrecognition.ProceedingsofIEEE77:25786.Reeves,B.,&Nass,C.(1996)TheMedia
Equation:HowPeopleTreatComputers,TelevisionandNewMediaLikeRealPeopleandPlaces.St
anford,CA:CenterfortheStudyofLan-guageandInformation.Resnick,M.,&Sanchez,J.(2004
).Effectsoforganizationalschemeandlabelingontaskper-formanceinproduct-centeredan
duser-centeredwebsites.HumanFactors46:10417.Rolandi,W.(2002).Doyourusersfeelsilly
?SpeechTechnologyMagazine,November/Decemberavailableathttp://www.speechtechmag.co
m/Articles/ReadArticle.aspx?ArticleID=29420.Rolandi,W.(2007).Thepersonacrazenear
sanend.SpeechTechnology12.5(June).Sacks,H.,Schegloff,E.,&Jefferson,G.(1974).Asim
plestsystematicsfortheorganizationofturn-takingforconversation.Language50:701.Sc
hegloff,E.,Jefferson,G.,&Sacks,H.(1977).Thepreferenceforself-correctionintheorga
nizationofrepairinconversation.Language53:36182.UnitedStatesAccessBoard.(2007).Se
ction508AccessibilityStandardavailableathttp://www.section508.gov/.Shiffrin,D.(19
87).DiscourseMarkers.Cambridge:CambridgeUniversityPress.Stevens,K.,&Blumstein,S.
(1981).Thesearchforinvariantacousticcorrelatesofphoneticfeatures.InEimas,P.,&Mil
ler,J.,eds,PerspectivesontheStudyofSpeech.Hillsdale,NJ:Erlbaum,1-38.Weinshenk,S.
,&Barker,D.(2000).DesigningEffectiveSpeechInterfaces.NewYork:Wiley.WorldWideWebC
onsortium.(2007).WebAccessibilityInitiativeavailableathttp://www.w3.org/WAI/.
References227

Thispageintentionallyleftblank

7CHAPTERInteractiveVoiceResponseInterfacesJeffBrandt7.1NATUREOFTHEINTERFACEInter
activevoiceresponse(IVR)interfacesarechieflytelephonyinterfaces.Theyarethemuchma
lignedsystemsthatyoureachwhencallingabusinessanddonotconnecttoaliveperson,butins
teadheararecordingsomethinglikethis:Yourcallisveryimportanttous....TocontinueinEn
glish,press1.MostpeoplecanreadilyrecallinstancesofterribleexperienceswithIVRs.IVR
s,however,arenotinherentlybad.Moreoftenthannot,theyarecreatedthatwaybytheirdesig
ners.ThischapterwillexplorethechallengeofdesigningIVRsthatareusableanduseful,and
itwilloffertechniquesfortestingIVRsthatwillgivethedesignercon-fidencethat,oncefi
elded,theIVRwillfulfillitsdesigngoalsandleaveitsuserssmiling(oratleastnotcursing
)(Figure7.1).ThemostcommonIVRapplicationisvoicemail.Anotherwell-knownexam-pleisa
24-hourbankingapplicationwhereaccountholderscancheckbalancesandtransferfundsfrom
oneaccounttoanotherusingthephone.Theuserdialsaphonenumberassociatedwiththebank,w
hichisansweredbyanautomatedsystem(theIVR),whichimmediatelyplaysoutarecordedmessa
ge:WelcometotheBigBank.Pleaseenteryouraccountnumber.Usingthekeysonthetele-phone,th
ecallerentersanaccountnumber.TheIVRreceivesthesekeypressesandcanthenchecktheente
redaccountnumberagainstadatabaseandcontinuewiththeinteraction.Itistheplayingofre
cordedmessagesorpromptsandthecollectionofresponsesthroughkeypressesthatdefineaus
erinterfaceasanIVR.Onaphonethekeypressesaremadeonthedual-tonemultifrequency(DTMF
)keypad(DTMFwasalsotrademarkedforatimeasTouchTone).TheprinciplesofIVRdesigncanalso
beappliedtootherinterfacesthatcombinetheelementsofan

IVRwithotherssuchasthoseinaself-servicecheck-outsystem.Thesesystemstypicallyplay
promptstocommunicatewiththeshopper,andoftenhaveanumerickeypadusedforresponses,ju
stlikeanIVR.Additionally,self-servicecheckoutshavebarcodescannersandfrequentlyat
ouchscreenthatcanbothdisplayinformationandacceptinput.Ingeneral,thepsychophysica

lrequirementsfromanIVRuserareminimal.Thebuttonsonphonesaredesignedtobeusedbypeop
lewithabroadrangeofphys-icalabilities(seeChapter2formoredetailonhapticinterfaces
).Individualsarefreetoselectaphonewithkeysthatmatchtheirphysicalneedsforsize,lab
eling,andpushforce,solongasthephoneproducesstandardDTMFtones.Userswhocancarryona
conversationwithanotherpersononaphoneareusuallyabletohearthepromptsfromanIVR.Som
eIVRsprovideameansforausertoadjustthevolumeorpaceofprompts.Usersrequiringamplifi
cationrequireitofallcalls,notjustcallsreachingIVRs.Callerswiththeseneedstypicall
ymakeuseofaphonewithadjusta-blevolume,oruseahearingaidandahearingaidcompatiblepho
ne(seeChapter5formoredetailonpsychophysicalconsiderationsofauditoryinterfaces).W
hiletheusersphysicalcapabilitiesmustbetakenintoconsideration,themostchallenginghu
mancharacteristictoconsiderinIVRdesignisthecognitivecapabilitiesoftheexpecteduse
rsofthesystem.Quitefrequentlytheintendeduserpopulationisthegeneralpopulation,wit
hitsinherentrangeofcognitiveabil-ity.Differencesbetweentheusersexpectedtaskflow,t
helanguagemodelsheusestodescribetheflow,andthevocabularypresentedbytheIVRcancaus
edif-ficulties.Theserialnatureofrecordedpromptsalsopresentsauniquecognitivechall
enge.Thedesignermustconsiderthefactthatausermusthearanoptionandtheassociatedkeyt
opressforthatoption,onefollowinganother,andmayhavetotrytoholdinshort-termmemoryo
neormorepossiblechoiceswhilesimultaneouslyevaluatingthecontinuingoptionsonamenu.
Thefactthatmost
FIGURE7.1MostpeopleexpectterribleIVRs.PoordesignshaveconditioneduserstodislikeIV
Respeciallywhentheyareexpectingaperson.UserstendtobesurprisedwhenanIVRhasbeenwell
designedtomeettheirneeds.Source:FromBlondie,February7,2007;#KingFeaturesSyndicate.
7InteractiveVoiceResponseInterfaces230

IVRsystemshaveafinitewaitingperiodforaresponsebeforetimingoutandhangingupcreates
afurthercognitiveload.TechniquestomakeIVRsmoreacces-siblearediscussedinSection7.
3.1.7.2TECHNOLOGYOFTHEINTERFACEHistorically,IVRswerebuiltontelephonecompanies(tel
cos)class-5switchessuchastheNortelDMS100ortheLucent5E.Theseswitcheswereusedtoprov
idetheIVRsthatthetelcosthemselvesmightneed,andinsomecircumstancestopro-videIVRsf
orlargecorporatecustomers.Class-5switcheshavemanylimitationsthatmustbeconsidered
whendesigninganIVR.Typically,recordedpromptshavelengthlimitationsoftenasshortas17
to20seconds.Promptsmayhavetoberecordedoveraphonehandsetasaliverecording,ormayhav
etoberecordedbytheswitchmanufacturerontoanannouncementcard.Somekeysonthephonekey
padmayhavereservedfunctionsthatcannotbechanged,andthetimingpara-metersmaynotbesu
fficientlyadjustable.Theabilitytointerfacewithexternaldatasources(informationnot
ontheswitchorinthetelephonynetwork)islimitedandinsomecasesnotpossibleornotinreal
time.Later,IVRcapabilitieswerebuiltintosmallertelephonyplatformsincludingCENTREX
platformsandkeysystems,allowingmedium-sizedbusinessestocreatetheirownIVRs.Manyof
thelimitationsimposedbytheclass-5switcheswerereducedontheseplatforms.Currently,I
VRscanbebuiltonhardwareassmallasaconsumer-gradePC.Outfittedwithatelephonymodemor
apurpose-builttelephonylinecardandsoftware,asystemtoserveoneormorelinescanbecreatedforlessthan$10,000.Linecardsallowphonecallstoconnecttothecomputerthatishosti
ngtheIVRandplayingtheprompts.Linecardsaremanufacturedforbothanalog(plainoldtelep
honeservice[POTS]),anddigital(ISDNorT1)phonelines.VoiceoverInternetProtocol(VoIP
)callscanbeterminatedthroughthecomputersEthernet/Internetnetworkconnection.Voicee
XtensibleMarkupLanguage(VXML)nowallowsIVRserviceproviderstohostyourIVRcodeonthei
rhardwareeliminatingtheneedforsmall-andmid-sizedbusinessestomaintaintelephonyserv
ers.CurrentIVRtechnologyaffordsthedesignervirtuallycompletecontrolovertheinterac
tion.ThelimitationspresentinIVRstodayaremainlythosethatareinherenttothetechnolog
y.First,informationmustbepresentedseriallyduetotheauditorynatureofprompts.Second
,responsesarelimitedtothe12keysonthephonekeypad,hangingup,ordoingnothing,whichev
entuallyresultsinatime-out.Third,thekeypadmakesnumberentryeasy,butalphanumericen
tryistedious,slow,anderrorprone.Fourth,althoughthephonesupportsfullduplexaudio(b
othsidescantalkandhear),IVRscannothearanythingotherthantheDTMFtonesproducedbykeypr
essesalltheunkindwordsshoutedatIVRsfallondeafears.Speechrecognitioninterfaces,where

thesystemcanhear(andhopefullyunderstand)whatissaid,werecoveredinChapter6.
7.2TechnologyoftheInterface231

7.3CURRENTIMPLEMENTATIONSOFTHEINTERFACEInteractivevoiceresponseinterfacesareubiq
uitous.ItisnearlyimpossibletocallaphonenumberwithoutthelikelihoodofreachinganIVR
.ThisboldstatementistruebecausevoicemailisanIVRapplication,probablythemostcommon
ofthemall.EverysizeablebusinessseemstohaveanIVRcallrouterthatattemptstodirectyou
rcalltotheproperpersonordepartment.Onceyourcallhasbeenrouted,itmaygonottoaperson
buttoanotherIVR,suchasaninformationser-vicethatallowsyoutoretrievedatayourbankbal
ance,thestatusofanairlineflight,ormovieshowtimes,forexample.TherearealsoIVRsyste
msthatallowyoutoperformactions.ThesevaryfromhomeautomationIVRsthatturnlightsonan
dofftoastock-tradingsystemthatallowsacallertoplacebuyandsellorders.Thelistgoeson
andon.UserswillgrudginglyacceptanIVRsolongasthetasktheycalledforcanbecompletedne
arlyaswellwiththeIVRastheuserexpectsitwouldbebyaperson.Inmanycases,however,IVRso
fferbetterservicethanwouldotherwisebeavailable.Theyaretypicallyavailable24/7anda
renotaffectedbyweather,ill-ness,strikes,ornewmoviereleases.Balance-checkingIVRsa
tbanksareagreatexamplebankershoursarenotwhenmostofusneedtoknowifthatlastdeposithas
beencreditedornot.AlthoughonlinesystemsperformmanyofthesamefunctionsasIVRsdo,the
rearemanytimeswhenandplaceswherepeoplearewithoutInternetaccess,althoughmostcarry
acellphonethatcanbeusedtocallanywhere,anytime.Insomecases,anIVRispreferredoverap
erson.IVRsofferinherentprivacy(Tourangeau&Smith,1996).WhenyouuseanIVR,nopersonis
listeningtoyourSocialSecuritynumber,yourPIN,oryouraccountnumber.TheIVRwillnotrec
-ognizeyourvoiceoryournameandwillnottalkoverlunchaboutyourmedicalproblems,financ
ialissues,testresults,orscandalousinterestsandanIVRwillnotcomplainthatyoushouldha
vewrittenallthisdownwhenyoucallasecondorthirdtime.CreatinganIVRcanreducethecosto
fofferingaservice,possiblyresultinginacostsavingspassedontothecustomer.Theperson
altouch,however,islost.Noautomatoncanreplaceapersonwhoisgenuinelyhappyorsadforyo
u,andnotoneofusputsmuchstockintheprerecordedPleases,Thankyous,andWeresorrys.7.3.1
onsoftheInterfacetoAccessibilityForapersonwithaspeechproductiondeficitapersonwhoh
asdifficultyspeaking,difficultybeingunderstood,oriscompletelyunabletospeakIVRsare
ablessing.Inthiscase,IVRscreateameansforaccomplishingataskthatdoes
7InteractiveVoiceResponseInterfaces232

notrequirespeaking.Goodpointsnotwithstanding,IVRstendtocreatemoreacces-sibilityp
roblemsthantheysolve.IVRstendtoincreasethecognitiverequirementsforcompletingatas
kwhencomparedwithacustomerservicerepresentative.Forsomeuserswithcognitivedeficit
s,thepredictabilityofanIVR(Dialthisnumber,press1andthen3)maybeeasierthaninteractin
gwithaperson.How-ever,shorttime-outsrequiringquickresponsesandtheinabilitytogeta
dditionalexplanationsmaymakeIVRsmoredifficultforothers.Inavery,veryfewcases,IVRs
havebeenbuiltthatcaninteractwiththeTTY/TDDmachinesusedbytheDeaftocommunicateviap
hone.However,mostIVRsareunusablebypeoplewithsevereortotalhearingloss.IVRsalsoint
eractpoorlywithTTYrelayserviceandcommunicationsassistants(CAs),asthetime-outsonm
ostIVRsaretooshorttoallowtheCAtotypethetextoftheprompttotherelaycaller,receivear
eply,andthenpressakeyinresponse.TTYisaslowmeansofcommunication,andintheUnitedSta
tes,bylaw,CAsarerequiredtotranscribewhattheyhearfromtheIVRverbatim.Itmightbeassu
medthatifapersoncandialaphone,hecanusetheIVRreached.Whiletherearemanydialingaids
forpeoplewithmotordisabilities,mostofthesefunctionasspeeddialersanddonothavethec
apabilitytostayonlineafterthecallcompletes,anddonotprovideafacultywithwhichtoper
formpostdialingkeypresses.Byandlarge,thoseusingadaptivetechnologytodialphonecall
shavedifficultieswithIVRs.7.4HUMANFACTORSDESIGNOFTHEINTERFACEThehumanfactorsdisc
iplineiswhatmakesthedifferencebetweenanIVRthatpeopledonotmindusingandonetheylove
tocomplainabout.AdvancesinIVRsoftwarehavemadeitpossibleforpeoplewithlowlevelsofs

killincomputerprogramming,inhumanfactors,andinuserinterfacedesigntoquicklyandeas
ilysetupaterribleIVRandinflictitontheunsuspectingpublic.Thissectionwilldescribet
hehumanfactorsinvolvedindesigninganIVR.Whilemuchofthissec-tionisbasedon10yearsof
experienceasahumanfactorsengineerwithAT&T,manyothershavepublishedonthistopic,suc
hasGardner-BonneauandBlanchard(2007),SchwartzandHardzinski(1993),Schumacher(1992
),andMarics(1991).IoweagreatdealofwhatIknowtothosewhohavelaboredinthisareabefore
me.7.4.1WhentoSelectanIVRInterfaceRarely,ifever,willtheIVRbetheonlyinterfaceaime
datcaringfortheusersneeds.Mostcommonly,IVRssharetheburdenwithcustomerservicerepre
senta-tives(CSRs)withthegoalofachievingsomelevelofcustomerself-supportthroughthe
IVR,whilethechallenging,unusual,ornonstandardissuesdropout
7.4HumanFactorsDesignoftheInterface233

oftheIVRfortheCSRstohandle.Inthesecases,itisalsocommontohaveawebsitethatisdesign
edtofulfillthesamegoaltoenablecustomerself-servicetoreducetheneedforandcostofCSRs
.BoththeIVRandthewebsitetendtocostlessthanstaffingacallcenterforthefullload,24/7
.Fullautomationand100percentcoverageofallissuesisrarelyarealisticgoal,andinmostc
asesnotappro-priate.ThismeansthattheIVRcannotbedesignedinavacuum,butmustbeconsid
eredaspartofasystem.Chapter11discussesingreaterdetailthehumanfactorsofofferingth
esameserviceovermultipleinterfaces.Onceithasbeendeterminedthatuserswillwanttocal
l,themostcommontrade-offtobeconsiderediswhetherthephoneinterfaceshouldbeanIVRora
speechrecognitionsystem.ManypeopleerroneouslyassumethatIfonlythecallerscouldtalkt
othesystem,itwouldbeeasier.AlthoughspeechrecognitioninterfacesarethesubjectofChap
ter6,afewcommentsherewouldbeappropri-ate.Inthisauthorsexperience,therearethreekey
factorstoconsiderwhenchoosingbetweenspeechandIVRsystems:accuracy,privacy,andthen
atureofthemenucontent.First,althoughitwillbearguedbyeveryspeechrecognitionvendor
youencounter,anIVRshouldbethedefaultchoiceonthebasisofaccuracy.TheDTMFkeysofanIV
Rarerecognizedwithnear-perfectaccuracy;speech,ontheotherhand,advertisesrecogniti
onaccuraciesinthemiddle-toupper90thpercen-tile.Inpractice,thislevelofaccuracyisd
ifficulttoachieveevengrantingit,thislevelofaccuracypertainstotherecognitionofsing
leutterances.Thecumulativechanceformisrecognitiongrowswitheachutterancerequiredt
oadvancetheuserthroughthemenustocompleteatask.Thetypicaltaskofacallerrequiressev
eralmenustobetraversed,andthushisorhersuccessandsatisfactionwiththesys-temisdepe
ndentonmultiplerecognitionevents.Givenatheoreticalspeechrecognitionsystemwith99p
ercentrecognitionaccuracyforconnecteddigits,thecumulativeprobabilityofcollecting
a10-digitphonenumberwithoutarecognitionerrorfallsto90percent.Iftheapplicationnee
dsa16-digitcreditcardnumbertobespokenanda9-digitSocialSecuritynum-ber(SSN)forcon
firmation,theprobabilityofanerror-freeattemptfallsto82per-cent.Errorrecoveryisan
important,difficult,andtime-consumingpartofdesigninganIVRoraspeechsystem,butgive
nthehigherprobabilityoferrorsinaspeechsystem,moreattentionisrequiredcomparedtoan
IVR.Thisbringsustothesecondconsiderationprivacy.Intheprecedingexam-ple,theuserofa
speechinterfacewouldhavehadtospeakhiscreditcardnumberandSSNaloud.Frequently,call
stosystemssuchasthesearemadefrompublicplacessuchascoffeeshopsorairportswhereitwo
uldbeunwisetospeakaloudpersonalandotherwiseprivatedata.IVRsaremoresecureinthesec
ases,asitisachallengetodeterminefromwatchingsomeonewhatthekeystheyarepress-ingme
an,whereasphonenumbers,SSNs,andcreditcardnumberseachhaveauniquecadencethatwhensp
okenmakesthemeasytoidentify.Andwhileitwouldbeachallengetounobtrusivelyattempttow
atchapersonenterkey
7InteractiveVoiceResponseInterfaces234

presses,itisoftendifficultnottoheartheconversationsofotherswhetheronewishestoeav
esdropornot.Givepeopleacellphoneandtheywilltalkloud.Givethemaspeechsystemonacell
phoneandtheywilltalklouderstill.Letthesys-temmakeanerrorandtheywillshouttheirSSN

,loudandslow,sothestupidsystem(andeveryoneelseintheroom)canclearlyhearthem.Goodspe
echdesignerswillfallbacktoDTMFinthesecasesratherthanrequirespeech,cre-atingahybrid
interface.Atacertainpointonehastoaskhowmuch,inwhatway,andatwhatcostisspeechrecog
nitionhelpingtheuser?Hopefully,thehonestanswertotheabovequestionisthatspeechisal
lowingyourinterfacetodosomethingwellthatanIVRcannot.Thisisthethirdconsider-ationt
henatureofthemenucontent.IVRmenusdobestwiththingsthatarenaturalorderedlists,andp
referablylistswithfewerthan10items.Agoodexam-plewouldbechoosingadayoftheweek:ForM
onday,press1.ForTuesday,press2,andsoonthrough7forSunday.IVRsalsodookaywithshortme
nusthatcanbeordered,althoughnonaturalorderexists:Forvanilla,press1;forchoco-late,
press2;forstrawberry,press3.IVRsfalldownwhenthecategoriesarelarge,creatinganartif
icialneedtobreakthemintomultiplemenus.Onecouldimaginehavingtoselectoneofthe50U.S
.statesfromanIVR.Howmanytimeswouldyouhavetopress9formoreoptionstogettothemenuthatw
ouldletyouselectTexas?Whatwouldtheorderonthemenube?Shoulditbealphabetical,orwoul
ditmakemoresensetoorderbysize,population,orfre-quencyofselection?Itsoundsawfuland
itwouldbe,inanIVR.Choosingastateistheperfecttaskforaspeechrecognitionsystem.Thev
ocabularyisclearlydefinedandbounded,butistoolargetobeeasilydoneinanIVR.Ifyourdes
ignneedisonethatwillcausetheusertomakechoicesfromlarge,difficult-to-organizegrou
ps,speechshouldbeconsidered.Iftheneedcanbecompletedwithtasksthatarelimitedtomenu
sthatareshortandcanbesensiblyorganizednumerically,anIVRshouldbeconsidered.Thepre
cedingisnotmeanttoscaredesignersawayfromspeechrecognitionsystems,astheyhavetheir
place;rather,itismeanttoallowforareasonedchoicewhenspeechisinevitablypresentedas
thecure-allthatwillmakeusershappierandmoresuccessfulthanastodgyoldIVR.Newerandfl
ashieralmostalwaysmeansmoreexpensive,andbettercanonlyreallybejudgedbyusers.7.4.2Wh
atDataShouldBeCollectedtoDesignanIVRThesinglemostimportantthingtoknowisthatyouar
enotrepresentativeoftheusersofyourIVR.Youwillhavetofindoutwhotheusersareandasmuc
hasyoucanaboutthem.WhydidtheycallthenumberthatgotthemconnectedtotheIVR?Whatgoals
dotheyhaveandwhattasksaretheyhopingtocomplete?Whatdotheyknow?Havetheydonethisbef
ore?Whatleveloffunctionalvocabulary
7.4HumanFactorsDesignoftheInterface235

mighttheypossess?Whenyoubegintolookreallyhardattheanswerstothesequestions,theimp
licationscanbestaggering.Whoaretheusersofvoicemail?Therearetwokindsofvoicemailus
ers:thesubscriberwhoisretrievingmessages,andcallerswholeavemessagesforthesub-scr
iber.Therearenotmanygoodassumptionsyoucanmakeabouteitheruser.Bothcouldbeofnearly
anyage,6to106,ofanyvocabularylevel,andneithermayspeaktheprimarylanguageofyourare
a.AusefultrickistousegrammarcheckersavailableontheWeborinmanywordprocessorstogiv
eyouthereadinglevelofyourpromptscript.InMicrosoftWord,gotoTools>Options>Spelling
&Grammar,andplacecheckmarksintheboxesforCheckgrammarwithspelling,andShowreadability
statistics.Inmostcases,forgeneraltelephonyapplicationswehavefoundthatpromptswritt
enatthethird-gradelevelorlowerperformwellinusabilitytesting.Thesubscribercouldha
vereceivedsomematerialsfromyousuchasauserguideandtheymayormaynothavereadthem.Voic
emailillustratestwobroadcategoriesofIVRs:subscribedtoandwalk-up-and-use.Inasubscri
bed-tointerface,theusermayhavereceivedtrainingmaterials,andislikelytousetheIVRre
peatedlywithsomelearning.Inawalk-up-and-useinterface,theusermayhavenoideawhythey
arehearingarecording,notraining,noexperience,andnodesiretoparticipateinwhatisbei
ngaskedofthem.Forthesereasons,theuserinterfacedesignbarismuchhigherforawalk-up-a
nd-useinterface.InmostIVRdesigns,therewillmorethanonesetofusers,soyoumustbecerta
inthattheuserinterfacegetsdesignedfortherealusers.Forexample,con-sideranIVRbuilt
fortheaccountingdepartmentthatkeepstrackofthehoursthatemployeesworkandthenfeedst
hosedatatoapayrollsystem.Withoutconsider-ingthelargenumberofemployeeswhowillhave
tousethesystemtoentertime,theIVRwillgetdesignedbyandforthefewpayrollpersonnelwho
needtouseittopullreports.Itwillcontainjargonandacronymsspecifictoaccountantsthat
willbeunfamiliartothemajorityofusers.Theprocessofenteringtimewillbeconstructedar
oundthedatabaseusedtostoretimedata,notthewayanormalpersonthinksoftheworkweek.Itisp
aramounttounderstandthatyourIVRwilllikelyhavemorethanonetypeofuser,andthateachty
pehasdifferinggoalsandcharacteristics.Ifonegroupofuserscontrolsthedesign,thereis

ahighriskthatwhiletheirneedswillbeconsid-ered,theneedsofanyotheruserpopulationwi
llbeleftunconsidered,orpoorlycaredforinthedesign.Thetypicalendresultisasystemtha
tfunctions,thatis,thetaskcanbecompletedusingtheIVR,butwithhightrainingcosts,lowsuc
cess,higherrorrates,andlowsatisfaction.Recognizingthattherearemultipleusertypesa
ndallowingfortheirneedsinthedesignwillleadtoamoresuccessfuldesign.EvenanIVRthati
sdesignedtosimplyroutecallerstoanappropriatepersonordepartmentrequireshumanfacto
rstechniquestosucceed.Determiningwhattocallitemsonamenu,andwhatsubitemsfallunder
eachmenuheading,shouldbeahumanfactorsexercise,notsimplywhatthedesignerthinkswill
7InteractiveVoiceResponseInterfaces236

work.Cardsorttechniquesarequiteusefulherefordeterminingmenuterminol-ogyandmappin
g.Weoncehadanincomingcallrouterinoneofourregionsthatwasintendedtoallowcallerstor
eachapersoninoneofourdepartmentswhocouldsolvethecallersproblem.Theoriginalcallrou
terhadbeendevelopedwith-outtheaidofhumanfactors,andhadbeencreatedfromthecompanysa
ndnotthecustomerspointofview.ThemainmenuofferedchoicesofDepartmentsA,B,andC.Custo
mersfrequentlyhaddifficultydeterminingwhichofourdepart-mentsshouldhandletheirpro
blem;theymadeachoicebutreachedthewrongdepartment.TheCSRtheyreachedhadtodetermine
thenatureoftheproblem,thecorrectdepartment,andthenmanuallytransferthecall.Everyo
newasfrus-trated:thecustomer,theCSR,andthecorporationbecauseofthewastedtime.Staf
finourhumanfactorsgroupsampledapproximately20,000callstodeter-minethereasonforcal
ling.Webroughtinparticipantsandaskedthemtosortthe30orsomostcommonreasonsforcallin
gintogroups.Wedidnotspecifythenumberofgroups,justthattherebeatleasttwo.Wethenana
lyzedthegroup-ingsforcommonpatternsandfoundasetofthreegroupsthatappearedtofitmos
tparticipantsviewofhowtoorganizethereasonsforcalling.Asecondgroupwasbroughtintovi
ewthegroupsandthereasonsinthemandtoprovidesuggestednamesforthegroups.Athirdgroup
wasthenbroughtintovalidateboththenamesandthegroupings.Givenareasonforcallingandt
henamesofthethreegroups,weaskedtheparticipantstotellusunderwhichgroupnametheywou
ldexpecttofindeachreasonforcalling.Thisprocessmayberepeateduntilasatisfactoryres
ultisobtainedthatis,thatagroupofnaveuserscanreliablyselectanamedmenuoptionthatwills
olvetheirgivenproblem.Themenuforthiscallrouterwasrecordedusingthenamesresultingf
romthestudyintheformofForaproblemoftype1,press1;foraproblemoftype2,press2;orforap
roblemoftype3,press3.Nonewtechnologywasapplied;theroutingdidnotevenhavetobechange
d.Theimprovementinefficiencyresultedinarecurring$29millionannualcostsavings.Thec
osttoachievethissavings?About6to8weeksoftimeforonehumanfactorsengineer,andlessth
an$10,000inrecruitingandincentivesforparticipants.Customerswerelessfrustratedthey
routedthemselvescorrectlymoreoften.TheCSRswerehappytheygotcreditforservingmorecus
tomersratherthanspendingtimetransferringmisdirectedcalls.Thecorporationwasplease
dtohavehappycustomersandhappyemployees,andtohaveimprovedefficiency.7.4.3HowtoSpe
cifyanIVRSmallIVRscanbespecifiedinatextoutlineorasapageortwoofpseudocode.Theseme
thodsaresufficientforaone-totwo-leveldeepmenu.Anexamplemightbeasmallcompanycallro
uterwhereyouhavethreedepartmentsandonetofiveemployeesineachdepartment.Foranythin
glargerormorecomplex,flowchartingorstatediagrammingwillberequiredtofullydescribe
thedesired
7.4HumanFactorsDesignoftheInterface237

behavioroftheIVR.Therearemanydifferentstylesandconventionsforhowtolayoutthediagr
amsthatdescribeausersprogressthroughanIVR.Considerusingaformatthatcompletelydescr
ibesonlyonemenuperpage.Formatsthatspreadlogicacrossmultiplepagesoftenmakeitchall
engingtounderstandwhatthedesiredbehavioris.7.4.4WhattheFlowchartShouldShowWhatis
thephonenumberthatmustbedialedtoreachtheIVR?HowmanytimeswillitringbeforetheIVRco
mesontheline?AssoonastheIVRanswers,whatisthefirstprompttobeplayed?Typically,thef
irstpromptidentifiestheorganiza-tionthatyouhavecalled:WelcometoAAAPlumbing.Immedia

telyfollowingthefirstpromptisusuallyachoiceofsometype.Inordertofullydescribethea
ctionsonamenu,eachofthe12DTMFkeys(0,1to9,*and#)mustbeassignedafunctionorpath.Ava
lueforhowlongthesystemwillwaitforaresponse(5to6secondsisgood)beforetimingoutanddoi
ngsomethingfortheuserisneeded,aswellastheactiontobeperformedonatime-out.Whatthes
ystemshoulddoiftheuserhangsupatthispointintheprocessmustbedescribedaswell.Inthee
xampleofFigure7.2,theuserhasmadeaselectiononthemainmenuindicatingthatshewishesto
changeherlanguageselection(currentlyEnglish).Uponentry,theerrorcountissettozeroa
ndthemenuisplayed.TheIVRthenwaitsforakeypress(hopefully0,1,2,3,or*).Iftheuserpre
sses*(star),sheisgivenastatusmessagethatthelanguageselectionhasnotchangedandthen
returnedtothemainmenu.Uponpressing1,theflowcontinuestoanotherpagewherethelanguag
ewillbechangedtoEnglish.Uponpressing2,theflowcontinuestoanotherpagewherethelangu
agewillbechangedtoSpanish.Uponpressing3,theflowcontinuestoanotherpagewherethelan
guagewillbechangedtobilingual,whichwillallowacallertochooseeitherEnglishorSpan-i
shwhenshecalls.Iftheuserpressesakeywithnofunction(4,5,6,7,8,9,or#),sheisinformed
thatthekeyshepressedisnotoneoftheoptions.Theerrorcountisthenincremented.Onthefir
sterror,theuserisreturnedtothemenutotryagain.Onthesecondandthirderrors,theuseris
playedacontext-specifichelppromptbeforebeingreturnedtothemenutotryagain.Onthefou
rtherror,theuserreceivesastatusreportthatthelanguagesettingshavenotbeenchangedan
dthecallisdiscon-nected.CallsaredisconnectedaftermultipleerrorstoprotecttheIVRpl
atformfrombeingoverloadedbycallsthatarenotabletousethesystem.Mostofthesecallsare
fromautodialerstryingtomakesalescalls,faxmachines,orhang-upcalls.Itistheresponsi
bilityoftheIVRdesignertoensurethatitiseasyenoughtousethatfewofthedisconnectedcal
lsarerealuserswhoarefrustrated.Iftheuserpresses0(zero)ortimesoutwithnoresponseaf
ter5seconds,theerrorcountisincrementedanderrorhandlingfollows.Thisillustrateshow
asimplethree-choicemenuismorecomplicatedtospecifythanonemightexpect.
7InteractiveVoiceResponseInterfaces238

LANGUAGEANN 949) Language Menu:To use English only, press 1.To use Spanish only,
press 2.For you and your callers to have the choice between English and Spanish
on each call, press 3.For help, press 0.To go back to the main menu without mak
ing any changes, press star.NIA 951) Your languagesettings have notbeen changed.
MAIN MENUENGLISHSPANISHBILINGUALDISCONNECTANN 950) LanguageHelp Menu: From this
menu you can change the language that both you and your callers will hear. To ke
ep the setting you have now, press star.NIA 099)The [DTMF] key is notone of the
choices.NIA 951)Your languagesettings have notbeen changed.Error count = 0FLUSHB
UFFERDTMF 0for help?Incrementerror count0 or 5 sec Time-out4-9, or #1230, 1,2, 3
, **Error #1Error #Error #4NoYesError #2,#3
FIGURE7.2Exampleofaflowchartforathree-choiceIVRmenu.Theuserhasmadeaselectiononth
emainmenuindicatingadesiretochangethelanguageselection(currentlyEnglish).
7.4HumanFactorsDesignoftheInterface239

7.4.5RecordedPromptsBythetimethehardworkofdesigningthelogicoftheIVRiscomplete,pr
ototyp-inghasbeendone,ithaspassedusabilitytesting,andpeoplearereadytogettheIVRou
tthedoor.Consequently,oneofthemostimportantaspectsofanIVRtherecordedpromptsisoften
givenverylittleconsideration.Thepromptswillhavebeenword-smithedduringusabilitytest
ing,sothatwhatissaidworkswell.Howitissaid,though,isofequalimportance.Itisnotunco
mmonforthetaskofrecordingthepromptstofalltosomeoneonthedevelopmentteam,thesecret
arywhoanswersthephonesintheoffice,orthetechnicianinstallingthehardware.Onlyoccas
ionallydoesthisresultingoodrecordings.Thepersonreadingthepromptscript,yourvoiceta
lentneednotbeaprofessional,buthisorhervoicemustbepleasant,friendly,warm,engaging,
rel-ativelyfreefromaccents,andshouldgiveyourcallerstheimpressionofyourcompanyoro

rganizationthatyoudesire.Irritatingvoices;boredorhaughtyvoices;speechimpediments
;andgumsmackingoughttobeavoided.Thecharac-terofthevoiceshouldmatchyourapplicatio
n.Achirpy,gushingcheerleadervoiceisprobablynotthebestchoiceforasomberapplication
likeahospital,church,orbank.Ontheotherhand,thevoicetalentforahealthclub,cardeale
r,ordanceclubmayneedtobeengagingandupbeat.IfyourIVRmaychangeovertime,selectavoic
etalentwhoislikelytobeavailableinthefutureforadditionalrecordings.Itisbesttohave
theentireIVRrecordedbyasinglevoicetalent(Christodoulou,1996).Ifyoucannotusethepr
evioustalentagain,youmayincurtheexpenseofrerecordingtheentirescripttoeffectasimp
leadditionorchange.Insomecases,thevoicetalentbecomesalargepartofthebrandbycreating
apersonathatcomestorepresentthecompanyororganizationthattheIVRserves.Manycelebriti
eslendtheirvoicestocorporationshopingtoinfluenceusersoftheirIVRsthroughfamiliari
ty,recognition,andassociationswiththevoicetalent.VerizonhasusedJamesEarlJonessdis
tinctivevoice(thevoiceofStarWarsDarthVader)primarilyinadvertising,butalsoinsomeof
theirIVRs.Whencreatingapersona,itisimportantthatthepersonareinforces,ratherthanc
hallenges,thepoliticsandcultureoftheorganizationitisintendedtorepresent.Oneareat
hatisparticularlychallengingforIVRsishandlingbilingualormul-tilingualneeds.Desig
nersoftenflippantlysuggestthatwelljusttranslateitandrecordthescriptintwo,three,ort
enlanguages,andthenallofourcustomerswillbeabletouseit.Thereareseveralfallaciouspo
intsintheprecedingstatement.Evenhigh-qualitytranslationsdonotoftenresultinusable
IVRs.Oneissueisthatconceptsandmetaphorsthatworkinonelanguagearenotalwaysappropri
ateinanother.IfEnglishistheoriginallanguageoftheIVR,translationsoftenresultinlon
gerprompts,insomecasesexceedinglimitationsthatwerenotaprobleminEnglish.Withoutus
abilitytestingofthetranslatedscript,howitwillperformisunknown.Thepoolofhumanfact
orsprofessionalsissmalltobeginwith;fewer
7InteractiveVoiceResponseInterfaces240

stillareabletoperformtestinginmultiplelanguages.Suddenlyoureasytaskisbecomingmorec
omplicated.Thehardestpartofamultilingualinterfaceissimplygettinguserstostaywithi
tlongenoughtouseit.InordertotelltheIVRthelanguagethatyouwanttouse,youmustwaitunt
ilyouhearinalanguageyouunderstandtheinstructiontochoosealanguageandthekeytopressto
selectit.Anicegraduatethesiscouldbebuiltaroundansweringthequestionofhowmanyoptio
nsforalanguageauserwilllistentoinalanguagetheydonotspeakbeforehangingup.Itisther
eforeimportanttounderstandtheexpecteduserpopulationbeforereachingforamul-tilingu
alsolution.Becarefultoevaluateyourdecisionbasedonthenumberofnon-nativespeakersra
therthanonethnicity.Experiencetellsusthatmostvoicetalentsarefemale.Forbetterorwo
rse,ouroldculturalstereotypesarestillwithus.Unpublishedresearchfromourlabsug-ges
tsthatfemalevoicetalentsareperceivedasfriendlierthanmalevoices,whilemalevoicesarem
orecrediblethanfemalevoices.Inonestudy,wevariedthegenderofthevoicetalentacrosstwot
ypesofrecordings:menusthatofferedchoicesandresponsesthatprovidedinformation.Nots
urprisingly,theIVRthatsubjectsinthestudyhadthemostconfidenceinandratedasthefrien
dliesthadafemalevoicetalentofferingchoicesandamaletalentprovidingtheinformationt
hattheuserwastryingtoretrieve.Whatwassurprisingwasthisalmostallsubjectsreportedhe
aringonlyasinglevoicetalent!Thescriptthatthetalentwillfollowtorecordyourpromptsi
sanimportanttoolforincreasingthequalityoftheIVR.Inmanycases,asimplelistofphrases
isprovidedtothevoicetalent.Whilethiswillgetthejobdone,thereisnocon-textthatis,nom
eansforthevoicetalenttotrytounderstandwhatishappeningorhowhecanmakeiteasierforth
eusertounderstandbypausingorpacingtheprose.Itcanhelptoprovidesomesidenotestothet
alent,suchasThishelloisastatementofgreeting,whilethishelloisaquestionareyouthere?Scr
softenbreakpromptsintoshortphrases,anditisoftennotobvioustothevoicetalenthowphra
seswillbecombined.Withoutthisknowledge,thecombinedphrasemaysoundawkwardwhenplaye
dinsequence.Ifpossible,itisproductivetohavesomeonefromthedesignteamwhoisintimate
lyfamiliarwiththeflowandpromptsoftheapplicationsitwiththevoicetalenttocoachheras
promptsarerecorded.Regardlessofthemethodsusedtoprocuretheprompts,itisnecessaryto
per-formafulltestofthecompleteapplicationtoensurethattherightpromptplaysattherig
httime,soundsgood,andcontainsthecorrectverbiage.Whenproducingascript,bestpractic
eistospelloutnumbersasyouwantthemread.Insteadof20,writeouttwentyortwo-zeroortwo-oh.

UnitedStates,thenumber0isusuallyreadaszeroandinsomecasesasoh,whileinothercountriesit
maybereadaszed.U.S.conventionforthe#keyispound,andthe*keyisreadasstar.Thereareinterna
ionaldifferencesintheseconventions.Youmustnotonlyknowyouruserssothatacorrectspec
ificationcanbecreated,butwith
7.4HumanFactorsDesignoftheInterface241

outsourcingandglobalization,youmustknowyourpartnersandvendorssothattheydonotunwi
ttinglyintroduceinappropriatetermsintothescript.Promptsneedtoberecordedinasuitab
lyquietenvironment.Backgroundnoiseincludedinyourpromptsonlymakesthealreadychalle
ngingtaskoflisten-ingtothemenuchoicesharder.Fromarecordingpointofview,noofficeisq
uiet,andeventhehushofthecoolingfanonaPCwilllowerthequalityofyourrecordings.Holdmus
icisfineforbeingonhold,butithasnoplaceasaback-groundinanIVRprompt.Yourpromptswill
beplayedoveratelephonewithabout3kHzofdynamicrange(Careyetal.,1984),whichisoflowa
udioqualitywhencomparedtoaCDrecordingwitha44-kHzrange.Productionvaluesshouldrefl
ectthis.IVRpromptsaretypicallyrecordedat8-kHzsamplinginmono,andtheydonotneedtobe
CDquality.Yourpromptswillnotsoundasgoodplayedoveratelephoneastheydointhestudioor
as.wavfilesonyourPC,butthisisnotanexcusetoacceptpoorrecordingquality.TheIVRwilln
eversoundbetterthanthesourcerecordings,sotherecordingsmustbeascleanaspracticable
.AnimportantfinalstepinproducingpromptrecordingsforanIVRistofiltertherecordingsf
orDTMFfrequencies.SomevoicetalentsspeechnaturallyincludesthesamefrequenciesasaDTM
Fkeypress.SomerecordingtoolssuchasAdobeAudition(formerlyCoolEdit)haveDTMFfilters
availableasafunctionthatcanbeappliedtoasoundfile.FilteringtheDTMFfrequenciesouto
fthepromptspreventstheIVRfrombeingtrickedbyitsownpromptsintothinkingthattheuserh
aspressedakey.7.5TECHNIQUESFORTESTINGTHEINTERFACEThedeceptivelysimplenatureofIVR
srequiresthatspecialattentionbepaidtothemethodsthatareemployedinthetestingofthei
nterfacetoensurethatvalid,reliable,andusefulresultsareobtained.7.5.1Task-BasedPr
otocolThebestresultsintheAT&THumanFactorsLabhavealwaysbeengainedwhenatask-basedt
estingprotocolwasused.Eachparticipantisgivenanumberoftasksthatalignwiththereason
sapersonmightcallthenumbertoreachtheIVRundertest.If,forexample,theIVRundertestis
acallblockingservicethatallowsyoutoblockincomingcallsfromspecifiedphonenumbers,o
netaskmightbetoaddaphonenumbertothelistofnumbersthatareblocked.Abettertaskwouldb
eanarrativethatgiveslessdirectionlikethefollowing:Yourdaughterhasanex-boy-friendw
hokeepscalling.Seeifyoucansetupthecall-blockingservicesothatifhecallsyourdaughte
r,thephonewontring.Theleastinformativetasksaretheonesthatprovidetoomuchinformation
andleadthetestparticipantthroughtheinterfacestepbystep.Createscenarios
7InteractiveVoiceResponseInterfaces242

thatwillexercisethefeaturesoftheIVR,andprovideaslittleinstructionaspos-sible.Itisf
airandofteninformativetoincludeataskthattheIVRwillnotsupport.Itallowsyoutoseeifa
personcandeterminethattheIVRwillnotdowhatshewastryingtoaccomplishorifbetterfeedb
ackisrequired.Tasksthatcontainincorrectorpartialinformationcanbeusedtotesttheerr
orrecoveryelementsoftheIVR.Itisagoodideatotesttheerrorrecoverypathsinadditiontot
hesunny-daypath(inwhichtheuserdoeseverythingjustastheIVRexpects).Ourpreferredtestr
egimenisoneormoreiterativeusabilitytestsonproto-typesoftheIVRtoimproveusability,
utility,andaccessibility,followedbyasum-mativeusabilitytestonthelastprototypeorp
referablytheproductionsystemtocharacterizetheexpectedperformanceoftheIVR.Iterati
vetestingofaprototypecanquicklyimproveadesign.Itisespeciallyeffectiveifadevelope
rwhocanmodifytheprototypeisonhandduringtesting.Usingthismethod,weaknessinthedesi
gncanbecorrectedovernightbetweenparticipantsor,insomecases,assoonasbetweentasks.
WithIVRs,itisalsohelpfulifthevoicetalentwhorecordedtheprototypeisavailableduring
thestudy.Inmanycasesyoumayfilltherolesofdesigner,tester,voicetalent,andprototype
r,givingyoufullcontrolandresponsibilitytousetheiterativeprocesstogettheIVRinfina

lform.7.5.2SignalDetectionAnalysisMethodOnetechniquethatcanbeusedtogreateffectin
IVRtestingborrowsfromsignaldetectiontheory,whichisexplainedquitewellinWickenssEng
ineeringPsychol-ogyandHumanPerformance(1984).Byobservationitcanbedeterminedifthe
usersuccessfullycompletesatask.Itcanbequiteinstructive,however,toasktheuserifheb
elieveshehascompletedthetasksuccessfully.Onehopesthateachtaskwillresultinbothsuc
cessfultaskcompletionandperceivedsuccess,ahitinsignaldetectionterms.Thesecondbesto
utcomeisacorrectrejection,wheretheuserfailsthetaskandcorrectlybelievesthatshehasfa
iled.Inthesetwocases,theuserhasanaccuratepictureofhersituationandcanmakeanintelligentdecisionastowhattodonext.Theothertwoconditions,actualfailurewithperceiveds
uccess(afalsealarm)andactualsuccessperceivedasafailure(amiss),causesignificantpr
oblemsiftheyoccurindeployedsystems.Giventhetaskofmakingacarreservationatanairpor
t,imaginewhathap-penstotheuser.Inasystemthatgeneratesfalsealarmstates,theuserbel
ievesthathehasrentedacarwheninfacthehasnot.Mostoften,somepartoftheinteractionhas
giventheusertheimpressionthathehasfinishedbeforehavingcompletedallnecessarysteps
.Perhapsacaseoftoomuchfeedback,toosoonorapoorlyorganizedprocessthatdoesnotlinearl
ydrivetheusertoasuccessfulconclusion.Inanyevent,theuserconfidentlyhangsup,boards
aflight,andlandsthousandsofmilesfromhomewithoutacarreserved.Systemsthatgeneratem
issescauseuserstomistakenlyrepeataprocesstoasuccessfulconclusionmoretimesthanthe
yintend.Theuserisnotgettingthefeedbackthatheisdone,or
7.5TechniquesforTestingtheInterface243

thatthesystembelievesheisdone.Herepeatsthestepslookingforapositiveconfirmationth
atthereservationhasbeenaccepted,oftenmultipletimes.Thispersonarrivesathisdestina
tiononlytofindthathehasreservedfourcars,per-hapswithnonrefundabledeposits.7.5.3P
rototypingTherearefourmethodsoftesting,eachwithincreasinglevelsoffidelity,thatca
nbeemployedtotestanIVR:WizardofOz(WOZ),WOZwithsoundfiles,afunctionalprototype,an
dbetacodeonaproductionplatform.WOZThelowestleveloffidelity,andalsothelowestcostt
orunisaWOZstudy.AWOZstudyisoftendoneforspeechrecognition,butcanalsobeemployedfor
IVRs.Likethewizardinthemovie,theexperimentertakesontheroleofthetechnol-ogy.Youre
adfromthepromptscriptandhavetheparticipantindicatethechoicetheywouldmakeoneachme
nu.Followingafloworoutline,youthenreadthenextpromptandcollectthenextresponseunti
leachtaskiscomplete.Itischeap,fast,andeasytomakeiterativechanges.Thefacevalidity
isabitlackingandawizardmayintroducebiaswithorwithoutintentionorawarenessthroughn
on-verbalcues,coaching,orgivingabitofthebenefitofthedoubt.Nonetheless,thismethod
isusefulifyourmeansarelimited,orasafirstvettingoftheinterfacebeforeadditionalres
ourcesarespentonprototyping.Notethatthelowerleveloffidelityoptionofhandingaparti
cipanttheoutlineorflowandthepromptscriptwasnotofferedasaviablemethod,asitisnot.T
hefirststepinvettingpromptsistoreadthemoutloud.Manythingsthatwillpassbydesigntea
msandcommitteesonpapersoundasawfulastheyarewhenreadaloudthefirsttime.Onediscount
methodistoleaveyourpromptscriptasavoicemailtoyourselfandthenlistentoit.Whenyouar
eforcedtolistentoandnotallowedtoreadaprompt,itssuitabilitycanbebetterjudged.Prom
ptswithawk-wardstructure,repeatedwords,ortoomanyoptionswillbegintomakethemselves
apparent.WOZwithSoundFilesTheveracityofaWOZcanbeimprovedbyrecordingthepromptsass
oundfilesonaPCthatcanbeplayedratherthanread,asinthesimplestWOZ.Inthisway,eachpar
ticipanthearsthemenusexactlythesameasanotherparticipantwiththesameinflection,emp
hasis,andphrasing.Therecordingsprovidesomesocialdistancebetweentheexperimenteran
dtheparticipantandgivetheimpres-sionofworkingwithanIVRsystem.Ameansoforganizingt
hesoundfilessothattheycanbeaccessedeasilyandplayedcorrectlyinresponsetotheusersin
dicatedkeypressesisneeded.Addingrecordingsforthewizardtouseisarelativelysmallste
pintheefforttoincreasetestvalidity.
7InteractiveVoiceResponseInterfaces244

FunctionalPrototypesFunctionalprototypesallowthetestparticipanttoexperienceyourd
esignwithoutyourintervention.AsoftprototypecouldbebuiltonaPCinascriptinglanguage
oreveninHTMLorVXMLthattiesyourmenulogictosoundfilesthatplayinsuccessionasuserscl
icktheirwaythrough.Thenextlevelofprototypeisonethatwillanswerphonecalls.Nowyouha
vehighfidelityinthatusersareinteractingwithaphonehandset,justliketheywillwiththe
realIVR.ThisprototypemaybebuiltonasmallPCwithatelephonymodem,orwithlinecards,ort
hroughawebsitethatwillhostVXMLscriptsandprovideacall-innumber.Ifyourpromptsarere
cordedwell,yourtestparticipantsmaynotknowtheyarecallingaproto-type.Onemajorheada
cheinpromptscriptingcannowbeevaluated.Whenusingawirelessoracordlessphonethekeysc
annotbeusedwhilelis-tening.Thehandsetmustbeheldawayfromtheeartopressakeyandthenr
eturnedtotheeartolistentothenextprompt.Ifcriticalinformationresidesinthefirstfew
wordsoftheprompt,manyuserswillnothearitastheirphonewillbeonitswaybacktotheear.Te
stingwithacordlessorprincess-stylehandsetinsteadofadesktoporspeakerphonewillhelp
identifypromptsthatarevulnera-bletothisissue.Inafunctionalprototype,connectionst
odatasourcesmayhavetobefaked.ThePCinyourtestlabisunlikelytohaveaccesstothebanksac
countdatabase.Fakedatabasesorevenstaticdataarenotusuallyaproblem,andclevertaskde
signcanoftenavoidexposingaprototypeslimitations.ProductionSystemPrototypesThegran
dluxuryintestingistobeabletoruntestcodeorbetadesignsonaproductionsystem.Onthepro
ductionsystem,youwilllikelyhaveaccesstodata-basesandlivefeeds.Everythingisofthes
amequalityandcharacterthatitwillbeonceitgoeslive.Audiolevels,delaystoreachdata,s
lowdownsduetotrafficorloadscanallbeincludedaspartofyourtest.Youmayevenbeabletous
ealoadbalancertosendeveryn-thcalltoyourprototypeinsteadofthecurrentIVR.Thedownsi
deisthatyourtestmayaccessliveaccounts,placerealorders,andincurrealcharges.Itcanb
edifficulttoobtaintestaccountsortestprocessesthatrequirearealcreditcardorSSNonpr
oductionsystems.Insomecases,testingiseasierandsaferonaprototype.7.5.4TestingEqui
pmentTherearecertainpiecesofequipment,suchasphonetaps,DTMFkeystrokeloggers,andco
mpressor/limiters,thatmaketestingprototypeorproductionsys-temseasierandmoreinfor
mative.Phonetapscanbelegalorillegaldependingonhowtheyareused:Consultlocallawenfo
rcementtodeterminethepropermethodinyourarea.Therearethreetypesofphonetaps:contac
tmicrophone,handsetcord,andlinetaps.Contactmicrophonetapsaretheleastexpensiveand
tendtohavethelowestaudioquality.Asofthiswriting,RadioShackcarried
7.5TechniquesforTestingtheInterface245

amodelthatadherestothehandsetwithasuctioncupnearthespeaker.Betteraudioqualitycan
beobtainedwithatapthatplugsintothecordthatgoesbetweenthetelephonebaseandthehands
et(e.g.,fromamanufacturersuchasJKAudio)(Figure7.3).Bothofthesetapsmustbeusedonth
ephonethatyourcallerisusing.Thelinetapgoesonthephonelineitself,andcanbeusedonthe
callerslineorthelinethattheIVRisconnectedto.Linetapsdoabetterjobofseparatingtheau
dioofthecallerandtheIVR.LinetapssuchasGentnersHybridCouplerorJKAudiosInnkeeperrunf
romthelowhundredstothelowthousandsofdollars;thisisapieceofequipmentwherehigherco
sttendstoequatetohigherquality.DTMFkeystrokeloggerscantieintothephonelineortheau
diofromaphonetap,andthenlogordisplaythekeycorrespondingtoaDTMFtone.Thismakestest
ingIVRssimplerasyoudonothavetowatchwhatkeytheuserpresses.Atonetimeduringaperiodo
fheavyIVRtesting,IfoundthatIhadmemorizedthesoundofmostoftheDTMFkeysperhapsasignth
atIneededmorevarietyinmywork.DTMFkeypressesareloud,especiallywhenplayedthroughhe
adphones.Acompressor/limiterisacommonpieceofaudiogearusedintherecordingindustrya
ndonperformancestageslargeandsmall.Thecompressor/limitertakesanaudioinputandshap
estheoutputtoconformtolimitsthatyouset,suchasnomatterwhattheinputlevelnothinglea
vesthecompressorover65dB.Thisisdonenotbycuttingoffloudpartsoftheinputbutbycompre
ssingtheaudiotofitwithinthelimitsyouset.Thebenefitisthatyoucanrunmicrophonesorph
one
FIGURE7.3TheTHAT-1phonetap.Becausethetapplugsintothehandsetcord,theaudioqualityi
sbetterthancontacttaps.(CourtesyofJKAudio,Inc.www.jkaudio.com.)
7InteractiveVoiceResponseInterfaces246

tapshot(athighvolumelevels)sothatyoucanhearquietinputswell,withouttheworrythatwh
ensomethingloud(likeaDTMFkeypress)comesalongyourearswillgetblasted.TheAlesis3630
compressor/limiterisaworkhorseinourlabthatcanoftenbesourcedforaslowas$100.7.6DES
IGNGUIDELINESIVRsysteminterfacesneedtobeinherentlyeasytouse.Thissectionisintende
dtogiveIVRdevelopersasetoftoolsthattheycanuseduringthedesignstagetohelpensuretha
ttheirproductmeetsfunctionalandusabilityexpectationsforabroadrangeofusers.Itshou
ldbenotedthatnosetofrulesorguidelinesalonewillleadtothedevelopmentofanoptimallyu
sableandusefulIVR.Thegoalinanyinterfacedesigneffortistofindthebestfitbetweenasyste
msspecificrequire-mentsandtheconsistencyandusabilitybenefitsthatwillresultfromthe
applica-tionofdesignguidelines.Toachievethisgoal,itisrecommendedthatguidelinesbe
usedaspartofabroaderprocess,includingusabilitytesting.7.6.1GeneralIVRDesignInfor
mationTheguidelinesthatfollowaredrawnfromseveralsources(HFES/ANSI,2006;ANSI,1992
;ANSI/ISO/IEC,1995;Davidson&Persons,1992;Engelbeck&Roberts,1989;PacificBell,1992
;Shovar,Workman,&Davidson,1994;Simon&Davidson,1990;PacificBell,1990),andrepresen
tconclusionsthatinmanycaseswerereachedindependentlyinmultiplelabsandthatoftencro
ss-referenceeachother.DoNotLoseDataWhenTransferringInformationenteredbytheusersh
ouldnotbelostinthetransitionbetweensystems,norshoulditbelostwhenroutedfromanauto
matedsystemtoaliverep-resentative.Theusershouldneverhavetogivethesameinformation
twiceinthesamesession(exceptforrequiredverificationorotherspecialcircumstances).
MakeWalk-up-and-UseApplicationsSimpleWalk-up-and-useapplicationsshouldhavesimple
interfacesaccessibletothenoviceuser.Menusshouldcontainfewchoices,possiblycontain
ingonlybinaryones(e.g.,yes/nooraccept/cancel).Systemoptionsshouldbeexplainedfull
yincompleteornearlycompletesentences.Usersshouldnotneedtoaskforhelporforaddition
alinformation,althoughitshouldbeavailable.ProvideShortcutsinSubscribed-toApplica
tionsSubscribed-toapplicationscanhavemorecomplexinterfaces.Whiletheyshouldstillb
eaccessibletofirst-timeusers,shortcutsshouldbeprovidedtoallowexperi-encedusersto
operatetheinterfacemoreefficiently.
7.6DesignGuidelines247

UsetheBestAnnouncementVoiceFUsehumanspeechwheneverpossible.FUsetrainedvoicetalen
twheneverpossible.FUsesynthesizedspeechonlyfortextthatcannotbeprerecordedorstore
d.7.6.2OpeningMessageTheopeningmessageisnotathrowawaygreeting;ithasworktodo.Open
ingMessageShouldGiveandGatherInformationFPresentuserswithanopeningmessagewhenthe
yfirstdialintothesystem.Exceptinsystemswheresubscribersareallowedtorecordtheirow
ngreetingmessages(e.g.,voicemail).FTheopeningmessageshouldbewordedtoidentifyther
elevantcompany(e.g.,AT&T,RiceUniversity)andtheapplication.FForwalk-upservicestha
tacceptonlytouch-toneinput,verifyasearlyaspossibleinthescriptthattheuserisatatou
ch-tonephone,andifnot,presentalive-agentroutingoptionforrotaryphoneusers.Theuser
shouldbeinstructedtopress1toindicatetouch-toneservice(Ifyouareusingatouch-tonepho
ne,press1;otherwise,pleasehold),butthesystemshouldrecognizeanytouch-toneinput(not
justthe1key)asevidenceofatouch-tonephone.FIfatouch-tonephoneisnotdetectedintheti
me-outperiodattheopeningmessage,itshouldbeassumedthattheuserhasrotaryservice(ori
sunwillingtousetheIVR)andshouldbeconnectedwithaliveagent.7.6.3UserControlGiveuse
rstoolstoallowthemtosucceed.GivetheUserControlFTheusershouldhavecontrolofthesyst
emwhereverpossible.Thisincludestheabilitytocontrolthestartandendofthesystemsactio
ns,theabilitytocanceltransactionsinprogress,andtheabilitytoexitfromthesystemand/
orspeakwithaservicerepresentative.FPresentuserswiththeoptiontochoosebetweenaself
-servicechannelandalive-agentchannel,withtheoptionsbeingpresentedasearlyaspossib
leintheapplicationconsistentwithorganizationalobjectives.FAlwayspromptuserswitha
lloftheavailableoptions.FThereshouldbeacommon,universalsetoffunctionsthatarealwa
ysavailablewithinthesystem.Thesefunctionsshouldbeeasytouse,andbe

7InteractiveVoiceResponseInterfaces248

consistentthroughouttheinterface.Anexampleisusingthe0(zero)keytotransfertoaservi
cerepresentativeortohearahelpmessage.FGivetheuserimmediatefeedbackforeachaction.
AnexampleoffeedbackisYourorderhasbeencanceled.Beepsandtonesbythemselvesareinsuffic
ientfeedbackandshouldbepairedwithanannouncementwhenused.AllowDial-Through:Interr
uptingSystemOutputFAllowtheusertooverrideprompts,menus,andotherstatementsatvirtu
allyanypointbeforeorduringthetimethattheyareoffered.Allowtheusertoselectamenuopt
ionimmediatelyafterithasbeenheardwithouthavingtowaitfortheentiremenutobepresente
d.FSometypesofsystemoutputmustnotbeinterruptible.Theseincludeerrortonesandthenon
interruptibleportionofanerrormessage.FUserinputshouldterminatethepromptaskingfor
userinputwithin0.3seconds,andthesystemshouldthenactonthatinput.FErrormessageshav
ethreeparts.Thefirstpartofanerrormessagestateswhatwentwrong.Thesecondpartstatesw
hyithappened.Thethirdparttellstheuserhowtocorrecttheproblemifitiscorrectable,and
whatoptionstheymayhaveorwhomtocallforhelp.Thewhatportionoftheannouncementshouldn
otbeinterruptible.Itmaybeappropriatetocombinethewhatandwhyinsomecases.Error:User
entersanincorrectpassword.What:Accesstoyouraccounthasbeendenied.Why:Thephonenumbera
ndPINyouentereddonotmatchourrecords.How:Pleasere-enteryourareacodeandphonenumber.Al
lowDial-Ahead:Pre-emptingSystemOutputFAllowuserstojumpaheadthroughseveralmenus.F
Userinputshouldbequeuedforprocessingwhentheuserisworkingaheadofthesystem.FIfanin
validinputisencounteredbythesystemwhileinputremainsqueuedforprocessing,thequeues
houldbeclearedatthetimetheuserstartstoreceiveerrorfeedback.AllowtheUsertoCancelA
ctionsafterDataInputAlwaysprovideawayforuserstocanceltheirinput.Thiscanbeprovide
daseitherapromptedoption(Ifthisiscorrect,press1.Ifincorrect,press2.),and/orasanimp
licitoption(e.g.,byenablingthe*[star]keytocanceland/orstartdataentryoveragain).
7.6DesignGuidelines249

DoNotStrandtheUserForoptimalusercontrol,theusershouldbereturnedtoafamiliarandmea
ningfullocationafteranactivityiscompleted(e.g.,tothesystemmainmenuorthecur-rentp
rimarytopicmenu).Thelocationshouldbetitledandcontainpromptsforwhattodonext.Whena
ppropriate,presenttheuserwithanoptiontoexitthesystem,orinstructionsonhowtoexit.7
.6.4MenusMenusmustbepurposefullydesignedtobeeasy.RarelydogoodIVRmenusorga-nizeth
emselveslikeexistingbusinessprocesses.MakeMenusInformativeandOrderlyFPromptedcho
icesshouldbenumberedsequentially,inascendingnumericalorderbeginningwith1.FTheuse
rshouldnothearpromptsforcommandsthatarenotcurrentlyavailable.Itispreferabletoski
pnumbersinthemenuthantopresentanonfunctionaloption.However,skippingnumbersshould
beconsideredonlyforoptionsthatarestatedependentandareavailableatothertimes.(Fore
xample,onamenuthathasadeletefunction,theoptiontodeletemayonlybeofferedwhentherea
reitemstodelete.)FWhenenteringanewmenu,theusershouldbegivenatitleandabriefinstru
ctionwiththeappropriateinflectiontoseparatetitlefrominstruction(e.g.,ForwardingSt
artDay.<pause>Choosethedaytostartforwardingon.ForMonday,press1...).FForwalk-up-an
d-useservices,telltheuserhowmanyentriestoexpect.Thiscuestheusernottorespondtooso
onifthereisuncertainty(e.g.,Pleasemakeyourselectionfromthefollowingfourchoices.).F
Presentthenextpromptorstatementwithin750millisecondsafterausermakesamenuselectio
n.OrganizetheMenutoHelptheUserFAsageneralrule,placethemostfrequentlyselectedchoi
cesfirst.Forexample,ifmostincomingcallsareforbilling,thebillingoptionshouldbethe
firstonepresented.FFollowanatural/logicalorder.Forexample,askfortheuserstelephone
numberbeforeaskingwhetherthereistroubleontheline.Askfortheareacodeandthenthetele
phonenumber.Askfortheitemandthenthedesiredquantity.Daysoftheweekshouldbeinorderr
atherthanbymostfrequentlyselected.
7InteractiveVoiceResponseInterfaces250

FFollowafunctionalorder.Relatedmenuitemsshouldbepresentedtogetherinthesamemenu.F
Theremaybecaseswheregroupingoptionsbasedontheusersknowledgeofcategoriesismoreeffe
ctiveandmayleadtobetterperformancethangroupingoptionsbyfrequencyalone.FThestruct
ureofmenuscanbelargelydependentonapplicationrequirements.Ahierarchicalmenustruct
ureisgenerallythemostappropriateforaninterfacewithalargenumberofavailableoptions
.Ontheotherhand,aflatmenustructure(i.e.,justonemenu)shouldbeusedwhenonlyafewopti
onsareavailable.LimittheNumberofItemsonMenusFThereisnouniversaloptimumnumberofch
oicesonamenuormenulevelsinahierarchy.Ingeneral,feweroptionsandfewerlevelswillbee
asier.Gooddesignisacompromisebetweenthenumberofoptionsinamenu,thenumberofmenuson
alevel,andthenumberoflevelsofmenus.Givenasetofrequirements,createthesmallestsyst
emthatgivesusersaccesstotheoptionsthattheywant.FAsthenumberofchoicesinamenuincre
ases,expectincreasesinthenumberoferrorsandrequestsforhelp,andanincreaseinuserres
ponsetimes.Designthemenustructuretobreakuptasksorcategoriesintosubtasksorsubcate
goriesthatcanbeaccomplishedwithshortermenus.Thisstructureshouldmatchwithareprese
ntativeusersmodelofthesystem.FNomenushouldeverrequiredouble-digitentry.Menusmustb
edesignedtousethe0to9and*and#keysassinglekeypressoptions.Amenudesignthathasmoret
han12optionsmustbesplitupintoseparatemenus,whichmayrequireaddinglevelsofhierarch
yoraddingitemstohigher-levelmenus.FAvoidusingcatchallalternativessuchasFormoreoptio
ns,press3.Instead,provideadescriptionoftheoptionsothatausercouldselect3onpurpose,
notjustbecause1and2didnotsoundright.MapBinaryMenuChoicestothe1and2KeysFThelabels
orchoicesonabinarymenushouldreflecttheusers,notthesystems,viewofthechoice.FManyyes
/nodecisionscanberewrittenforclarity.Considerthisexample:Doyouhavedialtone?Foryes
,press1.Forno,press2.Thispromptcanbeimprovedasfollows:Ifyouhavedialtone,press1.Ifn
ot,press2.FWhenayes/nobinarychoiceispresented,yesmustbeassignedtothe1key,andnomustbea
ssignedtothe2key.
7.6DesignGuidelines251

FIfpossible,phrasethequestionssothatyesisthemostfrequentresponse.FForconfirmations
,starcanbeusedinplaceof2togobackandfixtheentry:Youentered800-555-1212.Ifthisiscor
rect,press1;ifnot,press*.7.6.5StatementPhrasingPromptstatementsneedtobeshortandcl
eartheuserwillbehearing,notreading,them.KeepStatementsBriefPrompts,messages,andme
nuselectionsshouldbeasbrief,simple,andunambig-uousaspossible(e.g.,Toconfirmyouror
der,press1).Statementsshouldbeshortenoughnottodiscourageexpertusers,butcontaineno
ughinformationtobecleartonovices.FormatStatementsforSuccessFSystempromptsshouldb
eoftheformaction-object(i.e.,TodoX,pressY.).Inotherwords,presenttheavailableactionf
irst,andthekeyneededtotaketheactionsecond(Forcallwaiting,press1.).FAvoidthepassive
voice.Forexample,donotsay,Atelephonenumbermustbeenterednow.Instead,say,Enterthearea
codeandphonenumbernow.FAvoidnegativeconditionals.Forexample,donotsay,Iftheseinstru
ctionswerenotclear,press1.Instead,say,Toheartheseinstructionsagain,press1.FSystemst
atementsshouldpresentimportantinformationfirsttoallowuserswhodonotwishtolistento
thefullsetofstatementsandpromptstoquicklyidentifykeyinformation.FCriticalinforma
tionshouldnotbetheveryfirstwordinanannouncement.Userswithkeypadsembeddedinthehan
dsetmaynothearthefirsttwowordsfollowingakeypressasthehandsetisbroughtbacktotheea
r.Therefore,introducecriticalinformationwithphrasessuchasYouhavefive...,Therearefiv
e...,Thenumberfive...FTerminologyshouldbeconsistentthroughoutthesystem.FTerminolog
yshouldmatchtheintendedaudienceoftheinterface.Avoidjargonrelatedtotelecommunicat
ionsandcomputerswhentheintendedaudienceisoutsideoftheseindustries.Forexample,useE
ntertheareacodeandthefirstthreedigitsofyourphonenumber,ratherthansomethinglikeEnte
ryourareacodeandprefix.FAvoidprompts,statements,andmenulabelsthatsoundtoogeneral,
asuserswilloftenselectanincorrectoptionbeforelisteningforonemore
7InteractiveVoiceResponseInterfaces252

appropriate.Forexample,whenusersarepromptedwithastatementthatincludeswordinglikeF
orallotherbillingquestions...manyuserswillselecttheoptionforavarietyofquestionsun
relatedtobillingneeds.FWhenreferringtothesystemitself,ortothecompanyoroneofitsre
presentatives,usethepronounwe.(Weresorry,weareunabletoprocessyourrequestatthistime.).
Anexceptioniswhenreferringtoaspecificindividualorgroupwithinthecompany(Arepairtec
hnicianwillcontactyouwithin24hours.).7.6.6WordingConventionsFollowingconventionsa
ndusingstandardterminologybuildsonwhattheuseralreadyknows.UseStandardTerminology
inPromptsFUsethetermpressforsingle-digitentry(Ifyes,press1.).FUsethetermenterformulti
le-digitentryofphonenumbers(Pleaseentertheareacodeandthenthetelephonenumberforwhi
chyouarereportingtrouble.)FUseentertorequestinformation(Enteryourzipcodenow.).FWhenpr
omptingtheuserforasinglekeypress(e.g.,asinamenuprompttopressakeyforamenuselectio
n),usetheTodoX,pressYformat(Forcallwaiting,press1.).FRefertotheDTMFkeyswithstandardn
ames.FReferto*asstar(Tocancelthisorder,pressstar.).FReferto#asthepoundkeyorsimplypou
tertheamountofyourpaymentindollarsandcents,thenpresspound.).FReferto0aszero(Tospeakw
ithaservicerepresentative,presszero.).FRefertothenumberkeys(0to9)bytheirdigitname
s(Forsales,press3.).7.6.7MetaphorsMetaphorscanbepowerful.Whileapropermetaphorcanai
dtheuser,apoormetaphorwilladdconfusion.UseMetaphorsAppropriatelyFChooseanappropr
iatemetaphorsothattheusersdevelopareasonableconceptualmodelofthesystem(e.g.,voic
emailisbasedonametaphorof
7.6DesignGuidelines253

thepapermailsystem).Byusingmetaphors,userscandevelopasetofexpectationsthatcanthe
nbeappliedtothesystemenvironment,andwillhelpthemtohaveavalidconceptualmodel.FRes
ervetheuseofametaphorforplaceswhereittrulyapplies.Donotlabelfunctionswithnamesth
atdonotmakesenseinthecontextoftheapplicationsimplytofitagivenmetaphor.Thiswillco
nfusetheuserwhenthenamedoesnotrepresenttheexpectedaction.FTheuseofametaphorshoul
dbeconsistentwithineachinterface.Donotmixmetaphorswithinasingleapplication.FTheu
seofmetaphorsshouldbeconsistentamongapplications.Thisisespeciallyimportantininte
rfaceswithinteractingoridenticalfunctions.Otherwise,userswhotransferbetweensyste
mswithdifferentmetaphorsmaybecomelostandconfused.ApplyDirectionalMetaphorsConsis
tentlyFIfadirectionalmetaphorisused(e.g.,backward,forward),usethenumbersonthelef
tsideofthestandard34keypadtoindicateprevious,lower,slower,backward,andsoon,andth
enumbersontherightofthekeypadtoindicatenext,higher,faster,forward,andsoon.Thispr
ovidesthebestpossiblematchwithlearnedexpectationsandshouldminimizeanyconflictswi
thotherfactors.FThisisacasewhereitmaybedesirabletoviolatetherecommendationonordi
nalnumberingofoptions.Forexample,if4and6areusedforpreviousandnext,thereisnorequi
rementthat5haveanassignedfunction.Insomecases,itmayalsobeacceptabletovoicethesec
hoicesinorderofmostfrequentuseratherthannumerically.Forexample,Forthenextmessage,
press6;forthepreviousmessage,press4.7.6.8KeyAssignmentKeyassignmentsshouldbechose
ntosimplifytheinteractionfortheuser.DoNotUseMnemonicsFChoicesinamenushouldbenumb
eredconsecutivelyinascendingorder,startingwith1.FMenuchoicesshouldbenumberedtoco
rrespondwiththenumbersonthetelephonekeypad,ratherthannamedwithmnemonics,suchasTos
endamessage,press1,ratherthanTosendamessage,presss.Whileusingwforcallwaiting,fforcal
rwarding,orstosendamessagemightseemlikeagoodwayforuserstoremembermenucommands,itwi
llincreaseuserinputtimeandleadtoerrors.Some
7InteractiveVoiceResponseInterfaces254

handsetsdonothavelettersonthekeys.Inaddition,personswithvisualdisabilitiesmemori
zethepositionofthenumbers,notthelettersonthekeypad.FThemeaningofakeypressmustnev

erdependonitsduration.Callingcardsthatrequiretheusertopressandholdstarfortwoseco
ndshaveviolatedthisrecommendationandshouldnotbeemulated.Usingkeypressdurationtoc
hangefunctioncreatesaccessibilityissuesforuserswithmotorcontroldifficultiesandof
tenforallusersduetolackofclearinstructions.Manyusersholdkeypresseslongerwhenfrus
tratedhavingthefunctionchangewhentheydosowillnotimprovesatisfaction.CreateFunctio
nalConsistencyFMakekeyassignmentsconsistentwithinapplications,andwhereverpossibl
e,acrossdifferentapplications.Keysshouldbeconsistentlyassignedtothesameactions.F
Recognizethatinnewversionsorupgradestoaservice,reassigningakeyassociatedwithacor
efunctiontoadifferentfunctionmaycausedifficultiesforusers;theywillhavetothenunle
arntheoriginalassociationandlearnthenewone,resultinginerrors.FIfacorefunctionisk
nowntobecominginthefuture,butisnotyetimplemented,trytoreservethekeyforthefuturei
mplementationofthatfunction.FWithinamenuthefunctionassignedtoakeyshouldnotchange
asaresultoftimepassingorbackgroundstatechanges.Ifthefunctionassignedtoakeyneedst
ochange,thesystemshouldplayanewmenuwiththenewkeyassignment.AvoidLetterEntryonthe
KeypadWheneverpossibleavoidletterentryonthekeypad.Ifpossiblechoosenumericrathert
hanalphanumericUserIDsorpasswordsforIVRsystems.Ifthereisadatabasetomatchtextentr
iesagainst,theusershouldbeabletoentertextwithasinglekeystrokeperletter.Forexampl
e,ifacustomersnameisPat,thecus-tomershouldberequiredtoonlypressthenumbers7,2,and8.L
ettersmapontokeypadnumbers,asshowninTable7.1.Whilemostkeypadsbeginletteringwitht
he2key(A,B,C),someassignQandZtothe1key.Therefore,althoughthepreferredwayistomapQ
andZontothe7and9keys,respectively,thesystemshouldalsomapthe1keytoQandZ.Whenitisn
otpossibletouseasinglekeystrokeperletter(e.g.,whennamesarebeingenteredforthefirs
ttimeandthesystemcouldinterprettheinputinseveralways),thesystemshouldclearlydesc
ribethemultikeymethodoftextentrytousers.
7.6DesignGuidelines255

AvoidTerminatorKeysWhenPossibleFDonotrequireuserstoenteraterminatoraspartofamenu
selection.Forexample,ifthepromptisForbilling,press1,thentheusershouldonlyhavetoent
er1,not1followedby#(where#istheterminator).Iftheusermistakenlyentersaterminatorw
hereoneisnotaskedfor,thesystemshouldignoreitandnotflagitasanerror.FIfpossible,do
notrequireuserstoenteraterminatortosignaltheendofmultiple-digitinput.Forexample,
thesystemshouldbeabletodetecttheninthandfinaldigitofanine-digitSocialSecuritynum
berentry.7.6.9The0KeyWhenhelpisprovided,the0(zero)keyshouldbeusedtoaccesssystemh
elp.Typically,systemhelptakestheformofbeingtransferredtoaliveattendantorofreceiv
ingrecordedhelpinformation.FIftransferstoaliveattendantareavailable,pressingthe0
keyshouldterminatethecurrentactionandtaketheusertoaservicerepresentative.Thespec
ificfacilitytowhichthesystemtransferstheusershouldbedeter-minedbythecontextofwhe
reinthesystemtheuseriscurrentlylocated(e.g.,iftheuserrequestsassistancewhileinth
ebillingportionofanIVRflow,theusershouldberoutedtoabillingrepresentative).FAvoid
transferringtheusertomorethanonerepresentative.FWhennorepresentativeiscurrentlya
vailable(busyorafterbusinessofficehours),the0keyshouldleadtoappropriatestatement
sormessages.FIfthesystemprovidesonlinehelp,thehelpmessageshouldbespecifictotheus
erscontextintheIVR.Oncefinishedwithhelp,theusershouldbereturned
TABLE7.1LettersMappedtoDTMFKeyNumbersKeyLetterMapping1None2A,B,C3D,E,F4G,H,I5J,K
,L6M,N,O7P,Q,R,S8T,U,V9W,X,Y,Z
7InteractiveVoiceResponseInterfaces256

tothesamesystemstatethatheorshewasinpriortothehelprequest.Incom-pleteactionsmayb
ecancelled,however.Iftheyare,theusershouldbeaskedtostarttheactionoveragain.FIfmu
ltidigitinputisexpectede.g.aphonenumber,andtheuserentersZEROandthentimesout,thes
ystemshouldtreatthisasaZEROpressedforhelpandtransfertoaservicerepresentativeorgive

context-specifichelp.7.6.10The*KeyThe*(star)keyistypicallyusedtocanceluserinput,
movebackwardinasequence,ormoveupwardinthemenustructure.FCancelinputstring:Iftheu
serisenteringastringsuchasatelephonenumberorpasscode,pressing*shoulddiscardthedi
gitsthathavebeenenteredandreturntheusertothepromptprecedingtheentryofthedigits.F
Backuptopreviousaction:Infunctionsthatrequiremanysteps,pressing*shouldtaketheuse
rbackpastthelastcompletedstep,andreturntotheprompttoallowtheusertoperformthatste
poveragain.FExitsubmenu:Iftheuserisatalower-levelmenu,pressingthe*keyshouldtaket
heuserbacktoahigher-levelmenu,perhapseventothesystemsmainmenu.FTerminatecall:The*
keycanbeusedtoexitsystemandterminatecall.Forsomewalk-upapplicationswhereusersare
expectedtobeunfamiliarwithIVRconventions,acontext-specifichelpmessagetellsusersw
herethe*keyislocatedonthekeypadthefirsttimethatitismentionedorused.Usethe*KeytoI
tsBestEffectSincethe*keycannotprovideallofthefunctionalityoutlinedintheaboveexam
-plesatanyonepointintime,useofthekeyinanyparticularinstanceshoulddependonwhereth
euserisintheapplication.Inotherwords,thestarkeywillhavedifferentresultsdepending
onwhethertheuseriscurrentlyFEnteringanumericstring(theuserwillbetakenbacktothepr
omptforthenumericentryfield)FInthemiddleofamultistepfunction(theuserwouldbetaken
backtothepromptprecedingthelastcompletedstep)FAtalower-levelmenu(theuserwouldber
eturnedtothehigher-levelmenuthatthelower-leveloneisnestedwithin)FAtthehighest-le
velmenuintheapplication(theuserwouldthenexitthesystem)
7.6DesignGuidelines257

Developersshouldcarefullyevaluatetheentireapplicationtodeterminewherethe*keyshou
ldreturntheuseratanygivenpoint,andassignthefunc-tionalityofthekeyaccordingly.Spe
cialUsesofthe*KeyinVoiceMessagingSystemsIftheuserisrecordingamessage,greeting,or
name,pressingthe*keyshouldstoptherecording,discardwhathadbeenrecorded,andreturnt
heusertothepromptprecedingtherecordtone.Intheprocessofloggingin,iftheuserrealize
sthatthesystemisassumingthewrongmailbox,heorsheshouldbeallowedtopress*duringthep
asswordpromptandbetakenbacktothemailboxprompt.7.6.11The#Key(Terminate,SkipAhead)
The#(pound)keyistypicallyusedasaterminatorforvariable-lengthuserinput,toconfirmu
serinput,ortomoveforwardinthesystem.Usethe#KeyasaTerminatorFWhenaterminatorisnee
ded,usethe#key.Time-outsarealwaysacceptedasterminatorsinthesecasesaswell.FUse#to
indicatetheendofavariable-lengthinputstring(Entertheamountindollarsandcents,thenp
resspound).Aterminatorisrequiredwhenevertheinputcanbeofvariablelength.Aterminator
shouldnotbeusedwithfixed-lengthentries(e.g.,phonenumbers).Whenuseofthepoundkeyis
required,thepromptmustexplicitlystatethisrequirement,asintheexampleabove.FDonotu
sethe#keyasaterminatoraspartofmenuselection.Forexample,ifanoptionisTouseourautoma
tedsystem,press1,thentheusershouldonlyhavetoenter1,not1followedby#.FIfauserenters
the#keyduringtheplayingofaninterruptibleannouncement,theannouncementshouldimmedi
atelyceaseandthesystemshouldproceedwiththenextstep.The#keyisnotinthedial-aheadbu
ffer,asitsactionwastointerrupttheannouncementandithasacted.FThe#keycanalsobeused
todotheusualormostcommonchoiceatamenu.Anexamplewouldbeselectingadayoftheweek,wheret
odayvaries,butisthemostcommonchoice.Fortoday,presspound.ForMonday,press1.ForTuesda
y...SpecialUsesofthe#KeyinVoiceMessagingSystemsFThe#keyshouldbeusedwhenaterminato
risrequiredtoendarecording(atime-outshouldalsobeacceptedastheterminator).
7InteractiveVoiceResponseInterfaces258

FIftheuserislisteningtoapromptthatprecedesarecordtone,pressingthe#keyshouldskipt
herestofthepromptandtaketheusertotherecordtone(e.g.,whenrecordingamessage,greeti
ng,orname).Thisalsoappliesiftheuserislisteningtoasubscribersgreeting.The#keyshoul
dskipthegreetingandgostraighttotherecordtonetoleaveamessageforthesubscriber.FIft
heuserislisteningtoavoicemailmessage,pressingthe#keyshouldstopplayingthemessagea
ndtaketheusertothenextmessage.7.6.12PromptsandFeedbackTheusershouldreceivefeedba

ckfromthesystemtoconfirmactions,warnaboutdangerousactions(e.g.,requestexplicitco
nfirmationfromtheuser),echouserinput,explainerrorconditions(e.g.,anoptionisnotav
ailable),andtelltheuserwhatthesystemisdoing.FIfsomethinghasbeenaddedordeleted,co
nfirmeachadditionordeletionasitisrequested,thensummarizeoverallchangesatendofadd
/deleteactivities.FPresentthenextpromptorstatementwithin0.75ofasecond(750millise
conds)afterausermakesamenuselection.FWhenpossible,avoidusingthewordsto,tooorforinmenu
romptswheretheymaybeconfusedwiththeoptionkeys2and4.ProvideAppropriateInformation
aboutSystemDelaysFAsmalldelayisdefinedasbetween750millisecondsand3seconds.Thisde
layisnoticeableandundesirabletocustomers,butnoannouncementexplainingitisrequired
.FAmediumdelayisdefinedasbetween3and8seconds.Thisdelayisnoticeableandundesirable
tocustomers;ashortannouncementexplainingthedelayisrequired.FAlongdelayisdefineda
sbetween8and30seconds.Thisdelayisnoticeableandundesirabletocustomers;ashortannou
ncementwithanestimateofthedelayisrequired.Musicshouldbeplayedduringthewait.Music
shouldnotbeacannedfilethatstartsinthesameplaceeverytimethecustomerexperiencesadela
y.FAnextremedelayisdefinedasanydelaylongerthan30seconds.Thisdelayisnoticeableand
undesirabletocustomers;itisrequiredthatevery30secondstheuserhearanupdatedannounc
ementwithanestimateoftheremainingdelay,andmusicshouldbeplayedbetweentheseannounc
ements.Extremedelayswillcausecustomerdissatisfaction.Musicshouldnotbeacannedfileth
atstartsinthesameplaceeverytimethecustomerexperiencesadelay.
7.6DesignGuidelines259

GetRequiredInformationEarlyIfanapplicationrequirescustomerinformation,collectitb
eforecompletingstepsthataredependentoncorrectentryofthatinformation.Forexample,i
fthetelephonenumberisrequiredtoaccessaccountinformation,getandconfirmitbeforeask
ingwhattypeofinformationthecustomerdesires.TelltheUsertheNumberofDigitstoEnterWh
entheuserispromptedtoenternumericinputofafixedlength(e.g.,aSocialSecuritynumber)
,thepromptshouldstatethenumberofdigitsthatmustbeentered,asinPleaseenteryournine-d
igitSocialSecuritynumber.Exception:Incaseswhereafixed-digitnumberisbeingcollected
forsecuritypurposes,thenumberofdigitsrequiredshouldnotbestated.TelltheUserHowtoI
ndicateThatNumberEntryIsFinishedWhentheuserispromptedtoenternumericinputofavaria
blelength,thepromptshouldasktheusertoenteraterminatorwhendatainputiscompleted.Ho
wever,atime-outwillalsobeacceptedtoterminateinput.The#keyshouldbeusedasthetermin
atorkey.AlwaysConfirmUserInputFAlwaysprovideawayfortheusertoconfirm(orcancel)inp
ut.Thiscanbeprovidedasanexplicit,promptedoption,suchasIfthisiscorrect,press1.Ifno
t,press2.FThesystemshouldreadbackuserinputinproperphrasing.Inmanycases,thesystems
houldjustreadbacktheusersinputexactlyasitwasentered.Forexampleiftheuserenters1-80
0-555-1212,thesystemmayignorethe1,butwhenconfirmingtheentryitmustasktheusertoconfi
rmall11digitsentered,asinYouentered1,8,0,0...However,ifthesystemreportsthetimetoth
euser,itshouldbereadbackaselevenoclock,not1,1,0,0.FRepeatbacktotheuserlongdataentries
forconfirmationwithappropriatepauses.Forinstance,iftheuserisaskedtoenteraSocialS
ecuritynumber,thesystemshouldrespondwithsomethinglikeYouentered123<pause>45<pause
>6789.Ifthatiscorrect,press1.Tore-enter,press2.Itiscriticalthatphonenumbersberead
withproperphrasingandnotasa10-digitstring.UseaTonetoSignaltheStartofaRecordingWi
ndowVoicemessagingsystemsshouldhaveadistinctrecordtone,distinguishablefromanyoth
ertonesusedinthesystem.Thetoneusedtoindicatethattherecordingisstartingshouldbeas
inglebeep,of440Hzfrequency,and0.5second(500millisecond)duration.
7InteractiveVoiceResponseInterfaces260

7.6.13ErrorsErrorstatementsandpromptsshouldbeofthefollowingformat:FOptionalerror
tone(standarddoublebeep,asinthefollowingexample).FWhatwentwrong(andthecurrentsys
temstatus).FWhytheerrorhappened.FWhattheusershoulddonext,possiblyincludingmorein
formationthanwaspresentedintheoriginalprompt.Example:<beep-beep>Accesstoyouraccou

nthasbeendenied.(what)ThephonenumberandPINyouentereddonotmatchourrecords.(why)Please
re-enteryourareacodeandphonenumber.(how)FAfteraparticularnumberofconsecutiveerror
s(twoorthree),thesystemshouldtakespecialaction,suchastransferringusertoaliveatte
ndant.Anerrorcountononepromptshouldgenerallynotcarryovertootherpartsoftheinterfa
ce,butshouldbeclearedwhenuserresolvestheerror.Forexample,inanorderingsystem,ause
rmightenteraphonenumberincorrectlytwicebeforegettingitright.Thesystemshouldnotth
enimme-diatelytransfertheuserifheorshemakesasubsequenterrorintheorder-ingprocess
,eventhoughthatwouldbethethirderrorsinceenteringthesystem.FAsingleusererrorshoul
dnotcausethesystemtodisconnect,unlessitisasecurityrequirement.UsetheStandardErro
rToneIfatoneisusedtoindicateanerror,thetoneshouldbeadoublebeep,andshouldbe440-Hz
frequency,100millisecondson,50millisecondsoff,and100milli-secondson.DoNotMakethe
ErrorMessageBadFAvoidnegativewordsandphraseslikeinvalidresponse,error,andsoon.FErrorm
ssagesshouldbewrittensoastoavoidcompromisingsystemsecurity.Inpartsoftheapplicati
onwheresecurityisanissue(e.g.,enteringthesystem,changingaccountstatus),thesystem
responsesshouldnotprovidemoreinformationthanisneeded.IfaphonenumberandPINarerequ
iredtoauthorizeuseofasystem,donotprompttheuserwiththedigitlengthofthePIN,anddono
tgiveindividualconfirmationsoneitherthephonenumberorthePIN.Ifeitherorbothareinco
rrect,usetheerrormessageThephonenumberandPINyouentereddonotmatchourrecords.
7.6DesignGuidelines261

7.6.14Time-outsWhentheuserhastakennoactionafterhearingaprompt,thesystemshouldres
pondafterthedesignatedtime-outperiod.Thesystemresponsefollowinganunansweredpromp
tshouldtaketheformofFFirsttime-outorerror:Repeattheprompt.FSecondtime-outorerror
:Playthehelpmessageandrepeattheprompt.FThirdtime-outorerror:Playthehelpmessagean
drepeattheprompt.FFourthtime-outorerror:Transfertoacustomerservicerepresentative
ifavailable;otherwise,playapolitemessageaskingtheusertotryagainlater,andthenterm
inatethecall.Time-outscanbeofvariablelength;atpresent,thereisnosinglestandardfor
theamountoftimetoallowuserstocomprehendandactonapromptbeforethesystemresponds.Fa
ctorssuchasthecontextoftheprompt,thefrequencythatthepromptwillbeused,andthecompl
exityoftheactionshouldbeconsideredindeterminingthelengthofthetime-out.Apostmenut
ime-outisthetimethatauserisgiventoinitiatearesponseafteralistofmenuchoices.There
com-mendedvalueforpostmenutime-outsisfivetosixseconds.Aninterdigitkeypresstime-o
utisthetimeallowedbetweenkeypressesinamultiple-digitentry.Italsofunctionsasthete
rminatingtime-outforvariable-digitentry.Therecom-mendedvalueforinterdigittime-ou
tsissixseconds.Thesearegeneralguidelinesanditisstronglysuggestedthattimeinterval
susedinIVRinterfacesbecarefullytunedfortheintendedaudience,aswellasforthesituati
oninwhichtheywillbepresented.Also,designersshouldbeawarethatuserswhosephoneshave
keypadsinthehandset(cordlessandcellularphones)maytakelongertorespondtopromptsdue
tothetimenecessarytolowerthehandsetfromtheeartoaccessthekeys.Finally,individuals
withperceptualormotordisabilitiesmayalsoexperiencedifficultyinrespondingtoprompt
s,andtheirspecificneedsshouldbeconsidered.7.6.15AnnouncementVoiceThesoundofthedi
gitizedvoiceshouldbeconsistentwithintheapplication,aswellaswithallothersofferedb
ythecompanyororganization.Wheneverpossible,thesamepersonshouldbeusedfortherecord
ings,andtherecordingsforagivenapplicationshouldtakeplaceinasinglesession.However
,asitoftenbecomesnecessarytorecordoverseveralsessions(e.g.,aschangesandupdatesto
anapplicationbecomenecessary),itisunderstoodthatthesameindividualmaynotbeavailab
lewhenneeded.Whenadifferentindividualwilldosubsequentrecording,trytoselectaspeak
erinsuchawayastominimizedifferencesinpitch,
7InteractiveVoiceResponseInterfaces262

accent,andotherindividualvoicecharacteristics,asthesedifferencescandistractusers
frompayingattentiontothecontentofthestatementsorprompts.Whenrecordingspokendigit

ssuchastelephonenumbers,thedigitsmustberecordedforeachpositioninwhichthatdigitwi
lloccur.Thatis,anyparticulardigitwillhavedifferentintonationsdependingonwhetheri
tappearsatthebegin-ning,middle,orattheendofaspokendigitstring.Silenceandpausesar
ejustasimportantasthewordsbeingspoken.Inthefollowingexample,pausesoccuratdiffere
ntplaces.Inthefirstcase,theinappro-priatelocationofthepausemightcausesomeuserstop
ress1tosend.Theappropriatepausesinthesecondcasereducethepossibilityofthaterror.In
appropriate:Tolisten<pause>press1.Tosend<pause>press2.Appropriate:Tolisten,press1.<
pause>Tosend,press2.<pause>7.6.16TechniquesforCollectingInformationThetechniquesd
escribedbelowhavebeensubjecttomultipleusabilitytests(Miller,1996)andarewidelyand
successfullydeployed.CollectPersonalIdentificationNumbersSecurelyForsecurityreas
ons,donottelltheuserhowmanydigitstheyaretoenter.Theexceptioniswhentheuserischang
ingapersonalidentificationnumber(PIN)thenthesystemmusttelltheuserhowmanydigitstoe
nter.IftheenteredPINdoesnotmatchthephonenumberandPINpairmaintainedbythesystemfor
theuser,informtheuserandrepromptforbothphonenumberandPIN.Donotvoicebacktheerrone
ousdata.Sayinstead,ThephonenumberandPINyouentereddonotmatchourrecords.BeFlexibleWh
enCollectingPhoneNumbersFNeveraskuserfora10-digitphonenumber.Whena10-digitphonenum
berisneeded,promptuserwithEnterjusttheareacodeandphonenumber(ortheregionallyapprop
riatedescriptionofthephonenumber).FIftheuserentersa1ora0asthefirstdigit,thesyste
mshouldignoreitandcollect10moredigits.FIftheuserentersthewrongnumberofdigits,inf
ormher:Youenteredonly8digits.Repromptforthephonenumber.Donotrepeattheshortsetofcol
lecteddigits.FIftheuserleavesofftheareacodewhenaskedfortheareacodeandphonenumber
,prompthimtoaddjusttheareacode.However,thesystemshouldanticipateandaccepteithert
he3-digitareacodeorthefull10-digitnumberasaresponsetothisprompt.
7.6DesignGuidelines263

Collect Days of the Week in OrderPrompt the user by reading the days, with each
day followed by a numbercorresponding to the days position in the week. Monday is
day 1 and Sundayis day 7. If today is a likely response, assign it to the pound k
ey and offer itas the first option; otherwise, start with Monday. For today, pres
s pound. ForMonday, press 1. For Tuesday, press 2.Be Flexible in Colle cting Time
sF The system should accept input in military time (24-hour clock) format.(Howev
er, the user should not be asked to enter military time.)F Ask for hours and min
utes, followed by 1 for A .M. or 2 for P.M.F If the user leaves out the A .M ./P
.M. indicator, prompt the user for itagain while confirming the hours/minutes p
art of the entry. For example, ifthe user entered 2, 3, 0 corresponding to 2:30, p
rompt the user with For2:30 A .M., press 1. For 2:30 P.M., press 2. For a differe
nt time, press star.F The 0 (zero) key can still provide assistance, but only if
it is the first andonly digit entered after the prompt for the time entry. After
a time-out, theuser is either routed to a live agent or receives some other for
m ofassistance.Define Time Periods with a Start Time and a Stop TimeF A start ti
me and a stop time, not a start time and duration, define timeperiods.F The stop
day is the same as the start day if the start time is earlier than thestoptime.
Ontheotherhand,ifthestarttimeislaterthanthestoptime,thestopdayisthefollowingday.
FIdentifyoverlappingtimeperiodsandprovideoptionstocorrectthem.7.7CASESTUDYA case
study for an IVR can be found at www.beyondthegui.com .7.8FUTURETRENDSItisunlik
elythatpeoplewillbewillingtolivewithoutvoicemailinthefuture,andsoIVRswillcontinu
esolongasphonesexist.Inthenext10to20years,Iwouldexpectlittletochangeinthefield,a
sIVRtechnologyisalreadyrelativelymature.VXMLhasopenedmanydoors,makingiteasiertha
nevertocreateandoperateanIVR.However,thathasbeenamixedblessing.Manyeasilycreated
7InteractiveVoiceResponseInterfaces264

IVRsarepoorlyconceivedandcreateheadachesfortheirusers.Promotersofspeechrecogniti

onsystemshavebeenpredictingthedemiseofIVRsandtheriseofspeechforthelasttwodecades
.Currentlythereislittleevidencethatspeechisabouttotakeovertheworld,orthatIVRsare
abouttogoaway.IVRs,oncebuilt,tested,andputintooperationtendtobequiterobust,andar
efairlystraightforwardtoupdate.Speechrecogni-tionsystemsrequireprofessionaltunin
gandcarefulselectionofgrammarmodelstobesuccessful,andseeminglyinsignificantchang
esoftenrequirethesestepstoberepeated.OneareaforgrowthintheIVRfieldisstandards.Cu
rrentlytherearefew,ANSI/ISO/IEC13714-1995beingone,andintheexperienceofthisauthor
,exist-ingstandardsoccasionallypromulgatesuboptimalsolutions.Whatwouldbeofgreatv
aluetothedesignersofIVRsisatoolkitofprovensolutionsthatcouldbeadoptedandcopied.The
open-sourcemovementandVXMLhavethepossibil-ityofcreatingapubliclibraryofusability
-testedsolutionstocommonIVRtaskssuchascollectingaphonenumber,collectingadateandt
ime,andenteringadol-laramount.Withsuchalibraryinwidespreaduse,usersmaybeabletoco
untonsomesimilarityacrossvariousIVRsratherthenhavingtolearnforeachonehowwedoither
e.AcknowledgmentTheauthorisindebtedtothemanyhumanfactorsprofessionalswhohaveadvan
cedthefieldofIVRs.ResearchersfromSouthwesternBell/SBC,PacificBell,andAmeritechha
veallmadecontributionstothebodyofknowledgethatiscar-riedforwardintheAT&THumanFac
torsLaboftoday.REFERENCESAmericanNationalStandardsInstitute(ANSI).(1992).Standar
dUserInterfacetoVoiceMessaging.ANSIWorkingDocument,Project0976-D.ANSI/Internatio
nalStandardsOrganization(ISO)/InternationalElectrotechnicalCommis-sion(IEC).(199
5).UserInterfacetoTelephone-BasedServicesVoiceMessagingApplica-tions.ANSI/ISO/IEC
13714-1995.Carey,M.B.,Chen,H.T.,Descloux,A.,Ingle,J.F.,&Park,K.I.(1984).1982/83E
ndofficeconnectionstudy:Analogvoiceandvoicebanddatatransmissionperformancecharac
terizationofthepublicswitchednetwork.AT&TBellLaboratoriesTechnicalJournal63(9):2
059-2119.Christodoulou,J.(1996).StrategiesforEffectiveVoiceResponseScripting.Dra
ft.Austin,TX:SouthwesternBellTechnologyResources,Inc.Davidson,J.,&Persons,K.S.(1
992).QuickserviceBilledTollInquiry:TechnologyAssess-mentandRecommendations.SanRa
mon,CA:PacificBellHumanFactorsEngineeringLaboratory.
References265

Engelbeck,G.,&Roberts,T.L.(1989).TheEffectsofSeveralVoice-MenuCharacteristicsonM
enu-SelectionPerformance.Boulder,CO:U.S.WestAdvancedTechnologies.Gardner-Bonneau
,D.(1999).HumanFactorsandVoiceInteractiveSystems.NewYork:Springer.Gardner-Bonnea
u,D.,&Blanchard,H.E.(2007).HumanFactorsandVoiceInteractiveSystems,2nded.NewYork:
Springer.HumanFactorsandErgonomicsSociety(HFES)/ANSI.(2006).HumanFactorsEngineer
ingofSoftwareUserInterfaces.HFES/ANSI200(Canvassdraft2006).SantaMonica:HFES.Mari
cs,M.A.(1991).InteractiveTelephoneServices:DesigningfortheUser.C3F.91.008.Austin
,TX:SouthwesternBellTechnologyResources,Inc.Miller,J.T.(1996).AnInterimSummaryan
dProposedStudiesforStandardizingInteractiveVoiceResponseandSpeech-BasedUserInter
facesforAdvancedIntelligentNetworkServices.A3.126.96.79.Austin,TX:SouthwesternBe
llTechnologyResources,Inc.PacificBell.(1990).VoiceMessagingUserInterfaceForum:Sp
ecificationDocument.SanRamon,CA:PacificBellHumanFactorsEngineeringLaboratory.Pac
ificBell.(1992).GuidelinesforInterfaceDesignofAutomatedResponseUnits.SanRamon,CA
:PacificBellHumanFactorsEngineeringLaboratory.Schumacher,R.M.(1992).Phone-basedi
nterfaces:Researchandguidelines.ProceedingsofHumanFactorsSociety,36thAnnualMeeti
ng.SantaMonica,CA:HumanFactorsSociety,105056.Schwartz,A.L.,&Hardzinski,M.L.(1993)
.AmeritechPhone-BasedUserInterfaceStan-dardsandDesignGuidelines.HoffmanEstates,I
L:AmeritechServices.Shovar,N.,Workman,L.,&Davidson,J.(1994).UsabilityTestofAlter
nativeCallDirectorPrototypes.SanRamon,CA:PacificBellHumanFactorsEngineeringLabor
atory.Simon,S.,&Davidson,J.(1990).LosAngelesCSOAudioResponseUnit(ARU)Implemen-ta
tion:CustomerImpactAssessmentandRecommendations.SanRamon,CA:PacificBellHumanFact
orsEngineeringLaboratory.Tourangeau,R.,&Smith,T.W.(1996).Askingsensitivequestion
s:Theimpactofdatacol-lectionmode,questionformat,andquestioncontext.PublicOpinion
Quarterly60:275304.Wickens,C.(1984).EngineeringPsychologyandHumanPerformance.Colu
mbus,OH:Merrill.
7InteractiveVoiceResponseInterfaces266

8CHAPTEROlfactoryInterfacesYasuyukiYanagida8.1NATUREOFTHEINTERFACEOlfactionisthe
senserelatedtosmell,andolfactoryinterfacesrefertodevicesand/orsystemsthatprovide
userswithinformationthroughsmells.Olfactioniscategorizedasakindofspecialsensation
,forhumanshaveadedicatedsensingorgantodetectthestimuli.Humansperceivesmellbydetec
tingcertainmoleculesininspiredairviatheolfactoryorganinthenose.Theolfactoryinter
faceisarelativelynoveldevice.Inmoderncomputersys-tems,visualandauditoryinterface
sarehighlydeveloped,andtherearenumerouswaysforuserstoobtaininformationthroughvis
ualandauditorychannels.Recently,hapticinterfaceshavebeenactivelydevelopedsuchtha
t,incertainenvironments,usersareabletotouchandfeelvirtualobjectsorenvironments.A
mongtheso-calledfivesenses,onlyolfactionandgustation(senseoftaste)havebeenleftunex
amined.Theseunexploitedsensationsarechemicalsenses,whereastherelativelywell-deve
lopedsensoryinterfaces(visual,auditory,andhaptic)arerelatedtophysicalstimuli.Thi
sfactmakesitdifficulttointroduceolfactoryandgustatoryinterfacesinasimilarmannert
osensorychannelsbasedonphysicalstimuli.However,researchershaverecentlybeguntoact
ivelydiscussanddeveloptheuseofolfactoryinterfaces,includingtheintroductionofanol
fac-torydisplayinvirtual-reality(VR)systems(Barfield&Danas,1996),andtransmit-tin
golfactoryinformationovertelecommunicationlines(Davideetal.,2001).Figure8.1(a)sh
owsthestructureofthehumannose.Inthenasalcavity,therearethreenasalturbinatessepar
atingthepathoftheairflowinthecavity.Thereisanorgancalledtheolfactoryepitheliumat
thetopofcavity,wheresmellsaredetected.Figure8.1(b)showsthedetailedstructureofthe
olfactoryepithelium.Odormoleculesarecapturedbythemucuslayerandthendetectedbyolfa
ctorycells.

Materialsthathavesmellsarequitelimited.Carbon(C),hydrogen(H),oxygen(O),nitrogen(
N),phosphorus(P),sulfur(S),chlorine(Cl),bromine(Br),andiodine(I)aretheelementsre
latedtosmell,butonlythelastthree(halogensCl,Br,andI)havesmellsassingleelements.M
ostmoleculeswithsmellsareorganiccompoundsandhavealimitedrangeofmolecularweight.O
nlymoleculeswhoseweightisapproximately30to300havesmells;heavierandlightermolecul
esdonot.Fromtheperspectiveofhumancomputerinteraction,itmaybeinterestingtoexamineh
owmanykindsofsmellarerequiredtosynthesizearbitrarysmells.However,beforediscussin
gthisimportantissue,letusreviewthecaseofvision.Itissaidthatwecansynthesizearbitrar
ycolorbymixingthreeprimarycolors:red,green,andblue.Physicallightconsistsofelectr
omagneticwaveswhosepowerisdescribedasacontinuousdistributionfunctionofwavelength
orfre-quency.Thisdistributioniscalledaspectrum.Ourvisualsystem,however,mightrega
rdlightwithdifferentspectrumsasasinglecolorifthebalanceofpowerofthethreerepresen
tativewavelengths(red,green,andblue)isthesame.Hence,thephenomenonofthethreeprima
rycolorsisnotthenatureofphysicalcoloritself,butderivesfromthemechanismofvisualse
nsation.Thismechanismisnowusedbyvariousvisualrecordinganddisplaysystems,includin
gcolortelevi-sionsets,totransmitcolorinformationamongremoteplacesbycodingcoloras
athree-dimensionalvectorinsteadofacontinuousdistributionfunctionthatcanbeconside
redtohaveinfinitedimensions.Inotherwords,wecancompresscolorinformationbymakingus
eofourvisualmechanism.Thequestionnowiswhetherolfactionhasamechanismcomparabletot
hethreeprimarycolorsthatis,arethereprimaryodors?Theansweris,
Olfactory epitheliumOlfactory bulbUpper nasal turbinateMiddle nasal turbinateLow
er nasal turbinateNostril(a)
Mucus layerOlfactory ciliaOlfactory cellBasal cellSupporting cell(b)
FIGURE8.1Noseanatomy.(a)Structureofthenose.(b)Detailedstructureoftheolfactoryepi
thelium.Source:FromOoyama,HandbookofSenseandPerception(1994).
8OlfactoryInterfaces268

unfortunately,no.Humanscandetectthesmellsofapproximately0.4millionchemicalcompou
nds,anddistinguishthousands(insomecases,tensofthousands)ofsmells.Amoore(1970),in
histheoryofstereochemistry,sophisticat-edlycategorizedvariousodorsintosevenmajor
groupsofsmells.Someresearch-ersconsideredthatthesesevengroupscorrespondtothethre
eprimarycolors,butrecentprogressinolfactionresearchshowsthatthiswasnotsufficient
toexplainhumanolfaction.Later,Amooreincreasedthenumberofprimaryodorstoapproximatel
y20to30,butthisnumberhasnotbeenclearlytested.Abreakthroughinolfactoryresearchocc
urredin1991.Usinganapproachfromthefieldofgenetics,BuckandAxel(1991)estimatedthen
umberofreceptorproteinsinmicetobeapproximately1,000.Forthisdiscovery,theyreceive
dthe2004NobelPrizeinphysiologyormedicine.Usingrecentdataobtainedinhumangenomeana
lysis,thisnumberisestimatedtobe350inhumans.Eachreceptorproteinrespondstomultiple
molecules,resultingintheverycomplexfunctionsofhumanolfaction.Currently,thisnumbe
r(350)isconsideredtobeanindexthatshowstheorderofrequiredodorcomponentsforobtaini
ngarbitrarysmells.8.2TECHNOLOGYOFTHEINTERFACEThereareseveraltechnologiesapplicab
letothedevelopmentofolfactoryinter-faces.Becausepeoplesmellbyusingtheirnose,odor
antsshouldbevaporizedanddeliveredtothenose.Thus,themajortechnicalfieldsinvolvedi
ncreatingolfactoryinterfacesarethegenerationanddeliveryofsmells.8.2.1GeneratingS
mellsMostextractedsmellsourcesareinliquidform(e.g.,essentialoils)or,occasion-all
y,solidsubstancessoakedinliquid.However,humanscandetectairbornesmells.Thismeanst
hatolfactoryinterfacesshouldincludeadevicetovaporizesourcesofsmellsofwhichtherea
reseveralmethods.VaporizationAstraightforwardwayofvaporizingsmellsisnaturalevapo
ration.Thismethodcanbeusedforliquidsourceswithrelativelyhighvolatility.Bystoring
odorsourcesinasmallvessel,suchasanessentialoiladheredtocottonorporouscera-mics,a
ndlettingairflowthroughthevessel,wecangenerallyobtainscentedairinsufficientconce
ntrations.Ifsuchnaturalevaporationisnotsufficient,themate-rialcanbeheatedtoaccel
eratetheevaporation.Certaintypesofodormolecules,however,areeasilydestroyedbyhigh
temperatures,soheatingofmaterialsshouldbeavoidedwheneverpossible.Anotherwayofacc
eleratingthevaporizationofaliquidsourceisbymakingafinemistoftheliquid.Onecommonl
yuseddeviceisasprayer.Bycontinuously
8.2TechnologyoftheInterface269

pushingtheliquidoutofafinenozzle,theliquidbecomesafinemistthatiseas-ilyvaporized
.Electromechanicalactuatorscanalsobeusedtomakeafinemistfromliquids.Piezoelectric
actuatorsareoftenusedtopushasmallliquiddropoutofthenozzle.Withthismethod,thenumb
erofdropspersecondtobegen-eratedcanbecontrolledsothattheprecisecontrolofthearoma
intensitycanbeachieved.BlendingTherearetwowaystoblendcomponentodorsataspecifiedr
atio:inliquidsta-tusandaftervaporization.Toblendodorswhiletheyareinliquidform,di
lutionfluids(e.g.,alcohol)aretypicallyused,becausearomaticsources(essentialoils)
arecondensedandblendinggranularitywouldnotbeidealiftheundilutedsolu-tionsaredire
ctlymixed.Dilutedsolutionscontainingessentialoilsaremixedindropunitstomakeamixed
solutioncontainingcomponentodorsatthedesiredratio.Anotherwayofblendingcomponento
dorsistoblendthemaftereachodorisvaporized.Airflowingthroughavesselcontainingaliq
uidodorsourcebecomessaturatedwithvapor,whichisthenmixedwithairatdesiredmassflowr
ate.Thismix-ingisusuallyperformedbyvalves.Valvularsystemsthatcancontinuouslycont
roltheamountofairflowarecalledmassflowcontrollers(MFCs).AscommerciallyavailableM
FCsareoftenlargeandexpensive,itmaybedifficulttointegratemanyMFCsintoodor-blendin
gsystems.Instead,amethodtoblendmultiplecomponentodorsbydigitallycontrollingsolen
oidvalveshasbeenproposed(Yamanakaetal.,2002).8.2.2DeliveringSmellsOnceanodorsour
ceisvaporized,itshouldbedeliveredtothenose.Themethodofdeliverydependsontheapplic
ationstyle,whichinvolveswhetherusersmoveorwalkaround,andhowquicklythevarioussmel
lsshouldbedeliveredtotheuser.Thetraditionalwayofenjoyingsmellsinvolvesdiffusinga
romasthroughoutaroom,wheretheyarecontinuouslyenjoyedbyusersforasufficientperiodo

ftime.Thisisthereasonwhytraditionalaromageneratorsaregenerallycalleddif-fusers.Wid
espread,long-termaromasaresufficientforcommunicatingenviron-mentalorambientinfor
mation.Suchsmellsdonotdisturbusersexplicitly,butcanstillsubtlycommunicatewhatish
appeninginthesurroundingworld.Mostofthearomageneratorsdescribedintheprevioussect
ionarecategorizedintothistypebecausethey,bythemselves,donothaveanexplicitfunctio
nforcontrollingthespatiotemporaldistributionofsmells.Thistraditionalwayofenjoyin
garomas,however,maynotbesufficientifwewanttomakeuseofsmellsasapartofmediatechnol
ogies.Ifweconsiderenjoyingsmellsinconjunctionwithaudiovisualentertainmentsuchasm
ovies,TVprograms,andinteractivevideogames,thesmellcorrespondingtoeach
8OlfactoryInterfaces270

scenemightneedtobereplacedimmediately.Theproblemwithtraditionalscattering-typearom
ageneratorsisthattheycannoterasethesmellonceitisdiffusedwithinthespace.Hence,ame
chanismthatlocalizesscentdeliverytoalimitedrange(intimeand/orinspace)isrequiredt
ousesmellasapartofmultimodalmedia.Therearetwoapproachestospatiotemporalcontrolof
smells.Oneistoemittheminimumamountofsmellbeingdeliveredtothenose.Anotheristoinco
rpo-ratesmelleliminationequipmentintothesystem.Theformerapproachhassev-eralvaria
tions(seeSection8.3).Themostdirectwayofdeliveringsmallamountsofsmelltothenoseist
ousetubesthatcarryscentedgasfromthescentgenerator.Ifthescentgeneratorisplacedont
hedesktop,theusersworkspace(movablerange)islimitedbythelengthofthetube.Therefore,
makingscentgen-eratorscompactenoughtobewearableisaneffectivesolution(Yamadaetal.
,2006).Ifthescentgeneratorsbecomeextremelycompact,theycanbedirectlyattachedtothe
nosetoconfigureadirect-injectiontypeofolfactorydisplay(Yamadaetal.,2006).Compact
scentgeneratorscanalsobemountedattheshouldersorchest(Morieetal.,2003).Mountingsc
entgeneratorstothesepositionsreducestheinconvenienceofwearingdevicesandcanresult
innaturalhumancomputerinteractions.Therehavealsobeenattemptstodeliversmallclumpso
fscentedairthroughfreespacewithoutusingtubes.Byusingtheprincipleofavortexring,la
unchedfromtheapertureofabox,scentedaircanbecarriedthroughfreespace(Watkins,2002;
Yanagidaetal.,2004).Thisisanuntetheredapproachforspatio-temporallylocalizedolfac
torydisplays.Thesecondoptionforachievingspatiotemporallylocalizedolfactorydispla
ysystemsistoincorporatesmelleliminationequipmentintothesystem.Withoutsuchequipme
nt,theemittedscentwillbegraduallydiffusedandreducedinden-sity.Suchnaturaldiffusi
onisadequateiftheamountofemittedscentissuffi-cientlysmall.Forrelativelylargeamou
ntsofscent,systemdesignersshouldalsodesignsmell-eliminatingequipment.Iftheamount
ofthesmellismoderate,suctionpumpsandfilters(suchasacharcoalfilter)canbeused.Fors
ystemsthatemitamassiveamountofscent,ventilatingfunctionsshouldbeincorporatedinto
thesystem.Ifthesesuctionorventilationfunctionsareincludedinthesystem,thescent-em
ittingsubsystemislesscriticalindeliveringsmellstotheuser.8.3CURRENTIMPLEMENTATIO
NSOFTHEINTERFACEAlthougholfactoryinterfacesarestillatanearlystageofdevelopment,s
everalsuccessfulimplementationscanbefound.Systemsarefocusedongenerationofaromaor
spatiotemporalcontrolofsmells.
8.3CurrentImplementationsoftheInterface271

8.3.1HistoricalBackgroundAnearlyapproachtoincorporatingsmellswithotherkindsofsen
sorydisplayscanbefoundinHeiligsSensorama(Figure8.2),developedaround1960(Heilig,19
62,1992).Userscouldenjoymultimodalmoviesincorporatingbreezesandsmells,althoughth
edisplayswerenotinteractive.Therehavealsobeensomeentertainmentattractionsusingsc
ent;forexample,McCarthy(1984)developedascent-emittingsystemcalledaSmellitzer(Figur
e8.3)thatcouldemitaselectedscentandproduceasequenceofvarioussmells.8.3.2Systemsf
orScentGenerationandBlendingAnolfactometerisareliableinstrumentusedinexperiments
onhumanolfaction.Experimentersconstructolfactometersystemsverycarefullytoensuret
hatthe

FIGURE8.2Sensoramasimulator.TheSensoramawasdevelopedbyMortonHeiligaround1960.(Ph
otofromhttp://www.telepresence.org/.)
8OlfactoryInterfaces272

experimentisbothsafeandaccurate.Airflowisdividedintoseveralchannels,eachofwhichc
onsistsofapairoftubesequippedwithsolenoidvalves.Apairofsolenoidvalvesiscontrolle
dtoopenexclusivelysothatthetotalairflowofthechanneliskeptconstant.Oneofthetubesi
neachchannelisconnectedtoanodorvesseltoproducescentedair.Theintensityofeachodori
scontrolledbythetimingratioaccordingtowhichthevalveconnectedtotheodorantvesselis
opened.Allvalvesinthesystemarecontrolledsimultaneouslybyahostcomputertoproducebl
endedsmellsthatconsistofthedesireddensityofselectedodorant(s).Figure8.4showsanex
ampleofanolfactometerdevelopedbyLorigetal.(1999).Althoughcarefullydesignedandimp
lementedolfactometershavesuccessfullyperformedstrictcontrolofsmelltype,concentra
tion(intensity),andtemporalprofile,thesesystemsmaybecumbersomeforordinaryusers.R
ecently,severaltrialshavebeenmadetoconstructeasy-to-use,computer-controlledodoremittingsystems.In1999,DigiScentsannouncedplanstoreleaseascentdiffuser,callediSm
ell,thatcanemitmultiplesmells.Shapedlikeasharkfin,thediffuserfunctionedasa
FIGURE8.3Smellitzer.Thisscent-emittingsystemwasdevelopedbyMcCarthy(1984).Source:
FromMcCarthy(1984);U.S.Patent4,603,030.
8.3CurrentImplementationsoftheInterface273

peripheraldeviceforpersonalcomputers(Figure8.5).Asofthiswriting,iSmellisstillnot
commerciallyavailable.Kaye(2001,2004)hasdevelopedvarioussystemstoemitscentsinthe
contextofambientmedia(Figure8.6).AirbrushesareusedinthesystemnamedinStinktoemit12k
indsofsmells.SpraysdrivenbysolenoidsareusedintheDollars&Scents,ScentReminder,and
Honey,ImHomesystems,whichemittwosmells,fivesmells,andasinglesmell,respectively.Sc
entDome,developedbyTriSenx(2007),isacomputer-controlledscentdif-fuserthatcanblen
d20differentscents.TriSenxalsoprovidesexchangeablecar-tridges,sothatuserscancust
omizethearomas(Figure8.7).FranceTelecomdevelopedmultichannelscentdiffusersthatco
nsistofves-selscontainingaromaticgelsandafancontrolledviaaUSBcommunicationline.

Teflon odorantvessels/valvesTeflon solenoidvalvesNO - Normally openNC - Normally

closedPumpFiltersFlow-metersExhaustExhaustExhaustExhaustExhaustExhaustExhaust
IaIbIIaIIbIIIaIIIbIIIIIINONONONONONOControl flowConstant flowHousing2.0 L/min1.0
L/min
FIGURE8.4Olfactometer.Aschematicdiagramofanolfactometer.Source:FromLorigetal.(19
99).
8OlfactoryInterfaces274

FIGURE8.5iSmellscentdiffuser.DevelopedbyDigiScents,iSmellemitsmultiplesmellsandf
unctionsasaPCperipheral.(CourtesyofDigiScents.)
(a)(b)
FIGURE8.6Computer-controlledscentemitters.(a)inStink.(b)Dollars&Scents.Source:Fr

omKaye(2001).
8.3CurrentImplementationsoftheInterface275

Thesystemconfigurationissimpleandstraightforward,andthissystemwasincorporatedwit
hpracticalaudiovisualprogramsanddemonstratedatvarioussites.Inatypicalprogram,cal
ledKaori-Web(KaoriisaJapanesewordmeaningaroma),thecookingprocessisdisplayedbyimage
s,sounds,andsmellsofthefoodbeingcooked.Thesemultimodalstimuliaresynchronouslycon
trolledbyaPC(Figure8.8).Nakamotoetal.(2007)developedareal-timeodor-blendingsyste
masapartofanodor-recordingsystem(Figure8.9).Theseauthorsusedsolenoidvalvesthatco
uldopenandcloseveryrapidly,andsucceededincontrollingthedensityofmultipleodorsinr
ealtime.Althoughthevalvesonlyhadadigitalfunction(openandclose),theyintroducedthe
conceptofdeltasigmamodulation,whichhasbeenusedindigital-to-analogconversionsforau
dioplayers,totheirreal-timeblender(Yamanakaetal.,2002).Thehigh-speedswitchingoft
hevalvesmadeitpossibletoquantitativelycontrolodordensitybycompactdigitaldevicesa
ndcom-puterinterfaces,providingintegrationofupto32-channelblendersinacompactencl
osure(Nakamotoetal.,2007).8.3.3SystemsforSpatiotemporalControlofScentsAlthoughsp
atiotemporallycontrollableolfactoryinterfaceshaveonlyrecentlybeendeveloped,there
areavarietyofpotentialapplications.Cater(1992,1994)constructedafirefightingsimul
atorthatemitsfire-relatedodors,andembeddedallnecessarypartsinthesetofequipmentus
edbyfirefigh-ters(Figure8.10).
FIGURE8.7ScentDome.DevelopedbyTriSenx,thisscentdiffuserblends20differentscents.(
ImagecourtesyTriSenxHoldings,Inc,fromhttp://www.trisenx.com/.)
8OlfactoryInterfaces276

FIGURE8.8Kaori-Web(Scent-Web).DevelopedbytheTsujiWellnessCorporationandFranceTel
ecomR&D,thissystemisamultimodalprogramwithaudio,images,andscents.(CourtesyofExha
liaJapan.)
(a)(b)
FIGURE8.9Real-timeodorblender.Theblenderuseshigh-speedswitchinginsolenoidvalves.
(a)Prototype,partoftheodorrecordersystem.(b)Commercialversionofthereal-timeodorb
lender.(Photo(a)courtesyofNakamotoLab,TokyoInstituteofTechnology.)
8.3CurrentImplementationsoftheInterface277

Tanikawaandcolleagues(Yamadaetal.,2006)havedevelopedawearableolfactorydisplaysys
tem.Byfocusingonthespatialdistributionofvirtualolfactoryspace,andbymakingtheenti
resystem(includingscentgenerators)compactandwearable,theirsystemallowsuserstomov
earoundtheenvironmentandactivelyexplorevirtualolfactoryspace.Thisolfactorydispla
ysystemhasfourodorves-sels,andamicropump(drivenbyaDCmotor)isusedtoproduceairflow
foreachvessel(Figure8.11).Wearableolfactorydisplaysunderdevelopmentaredirect-inj
ectiontypedis-plays.Theseusepiezoelectricactuators(inkjethead)toproducesmalldrops
ofessentialoilthataredirectlyinjectedintotheusersnostril(Yamadaetal.,2006).Althou
ghthisresearchisongoing,thegoalistoultimatelycreateacompactolfactorydisplaythatw
illmakeauserfeelunencumberedandabletomovefreelyaroundtheenvironment.Ifthissystem
isdeveloped,olfactory-augmentedrealitywillberealizedandvarioustypesofinformation
canthenbecommunicatedbyodor.Mochizukietal.(2004)developedanarm-mountedolfactoryd

isplaysystem,focusingonthehumanactionofholdingobjectsinthehand,bringingthemclose
tothenose,andthensniffing.Theyconfiguredthesystemsothattheodorvesselsoftheolfact
ometeraremountedonthearmandtheodor-emittingendofthetubeispositionedontheuserspalm
(Figure8.12).Thisarrangementenabledquickswitch-ingamongdifferentsmellssothatanin
teractivegameinterfaceusingsmellwasachieved.AttheUniversityofSouthernCaliforniaI
nstituteforCreativeTechnologies(ICT),Morieandcoworkers(2003)designedandimplement
edanecklace-shapedunitequippedwithfoursmallwireless-controlledscentemitters.This
systemiscalledtheScentCollar(Figure8.13,page281).Byemittingscentsclosetothe
OdorPakOdorPak controllerValve manifold panelAir hoseAir pressure tankAspriation
air blower
FIGURE8.10Firefightingsimulator.TheD.I.V.E.firefightertrainingsystembyCater.Sour
ce:FromKaye(2001).
8OlfactoryInterfaces278

nose,thecollarenableslocation-specificolfactorystimuliwithoutforcinguserstoweare
ncumberingtubesordevicesneartheface.MicroScentdevelopedascentgeneratorthatcandel
iverasmallamountofscentlocallybymakinguseofthevortexringlaunchedfromtheapertureo
fanenclosedspace(Figure8.14).Thisprinciple(alsoknownasanaircannon)isthebasisofapop
ularscientificdemonstrationforchildren.Inthissystem,multi-plediffusersareembedde
dintheenclosureandmultiplesmellscanbeemitted.Withthisconfiguration,thescentedair
intheenclosedspacehastobecleanedbyasuctionfilterbeforeeachscentisemitted.Yanagid
aetal.(2004)proposedamethodofscentdeliverythatalsousestheprincipleofthevortexrin
g.Theirsystemhasatrackingfunctionthatisaimedattheusersnoseandascent-switchingmech
anismthatcanprovideadifferentsmellforeachlaunchofthevortexring(Figure8.15,page28
2).Theusersnoseistrackedbydetectingitspositionthroughimageprocessing,andthetwo-de
grees-of-freedomplatformcarryingthebodyoftheaircannonismaneuveredtodirecttheairc
annonatthenose.Thescent-switchingmechanismisimplementedbyplacingasmall
FIGURE8.11Wearableolfactorydisplay.DevelopedbyTanikawaandcolleagues(Yamadaetal.,
2006)thissystemallowsuserstomovearoundtheenvironmentandexplorevirtualolfactorysp
ace.(CourtesyofHiroseTanikawaLab,GraduateSchoolofInformationScienceandTechnology
.)
8.3CurrentImplementationsoftheInterface279

cylinderchamberinfrontoftheapertureoftheaircannon.Thescentedairistheninjectedint
othischamberinsteadofintotheaircannonbodysothatthescentedairiscompletelypushedou
tofthechamberforeachshotandnoscentedairremainsintheaircannonbody.Thissystemwasna
medtheScentProjector.Duringthedevelopmentofscentprojectors,severalissuesemerged.
Forinstance,thefeelingofsuddenlybeingwind-blastedimpairsthenaturalolfactoryexper
ienceoftheuser.Tosolvethisproblem,amethodhasbeenproposedtousetwoaircannonsandlet
thevortexringscollidewitheachothersothattheycollapseinfrontoftheuserandthehigh-s
peedairflowcomposingthevortexringsisreduced(Figure8.16,page282).
(c)
(a)(b)
FIGURE8.12Arm-mountedtypeofolfactorydisplayforactivesniffing.Thisdisplaywasdevel
opedbyMochizukiandcolleagues(2004).(CourtesyofNaraInstituteofScienceandTechnolog
y,Japan.)

8OlfactoryInterfaces280

11711930242254
FIGURE8.14MicroScentsscentgenerator.Thisdevicedeliversscentlocally.(CourtesyofMic
roScent;U.S.Patent6,357,726.)
FIGURE8.13ScentCollar.DevelopedbyMorieandcolleagues(2003)attheInstituteforCreati
veTechnologies,thisdevicepermitslocation-specificstimuliwithoutclumsytubesordevi
cesneartheface.
8.3CurrentImplementationsoftheInterface281

8.3.4ApplicationsoftheInterfacetoAccessibilityOlfactoryinterfacescanbeanintuitiv
emeanstocommunicatedangertoothers.Forexample,peoplenoticeleaksofinflammable(buti
nvisible)gasbysmell.Usually,methyl-mercaptan,towhichhumansarehighlysensitive,isb
lended
FIGURE8.16Generatinganolfactoryfieldinfreespacewithtwoaircannons.Thevortexringsc
ollapseinfrontoftheuserandthehigh-speedairflowisreduced.Source:FromNakaizumietal
.(2006);courtesyofJosephJofishKay.
(a)(b)
TubeShuttersFresh airAir intake valveScented airPush
FIGURE8.15ScentProjector,withnose-trackingandscent-switchingfunctions.(a)Systemo
verview.(b)Scent-switchingmechanism.ThesystemwasdevelopedattheAdvancedTelecommun
icationsResearchInstituteInternational.
8OlfactoryInterfaces282

withmethaneorpropanegas.Theolfactorycommunicationfunctioniseffec-tiveforbothsigh
tedandvisuallyimpairedpeople.Ifcomputer-controlledolfac-toryinterfacesbecomecomm
on,awidevarietyofphysiologicallyintuitivesignalscanbeprovided,especiallyforcommu
nicatingdangeroussituations.8.4HUMANFACTORSDESIGNOFTHEINTERFACEAlthoughhumansare
saidtobeabletodistinguishthousandsofodors,thereiscurrentlynomethod,asmentionedpr
eviously,tosynthesizearbitrarysmellsbymixingasmallsetofprimaryodors.Inthissection,
variousaspectsofolfactionaredescribed.8.4.1OdorIntensityThesubjectiveintensityof
asmellisrelatedtotheconcentrationofodormole-cules,butthereisawidevarietyinsubjec
tiveintensityforvariousodormaterialsatidenticalconcentrations.Whenmultipleodormo
leculescoexist,odorsathigherconcentrationsarenotalwayssubjectivelydominant.Physi
calaspectsthatarerelatedtosubjectiveodorintensityfollow:Concentration:Themostbas
icvalue,describedingrams,molarconcentration,percentages,orpartspermillion(ppm).D
iffusivity:Thisparameterreferstohowfasttheodordiffusesintotheatmosphere,andisdef
inedasthedistancethattheodormoleculecanreachwithinaspecifiedtimeorthetimerequire
dfortheodormoleculetotravelacertaindistance.Thehighertheconcentrationandthelower
themolecularweight,thelargerthedif-fusivityobtained.However,airflowhasasignifica
nteffectonodordiffusionwhencomparedtostaticdiffusionofodormaterial.Boilingpoint:
Thetemperatureatwhichvaporpressurebecomesequaltoenviron-mentalpressure.Theboilin

gpointformostodorcompoundsiswithintherangeof150to300C.Odorswithasignificantlylow
boilingpointtendtohaveanintensesmell,andodorswithahigherboilingpointhaveaweakers
mell.Volatility:Theamountofodormaterialthatisconsumedinagiventimeperiod.Generall
y,Weber-Fechnerslawcanbeappliedtotherelationshipbetweentheconcentrationofodormate
rialandsubjectiveintensity.8.4.2ThresholdValueTheminimalconcentrationatwhichhuma
nscandetectasmelliscalledtheodorthresholdvalue.Therearemultipledefinitionsofthresh
oldvalue:detection,recognition,anddifferential.
8.4HumanFactorsDesignoftheInterface283

Detectionthresholdvalue:Smellscannotbedetectedatverylowodorconcentra-tions.Ifthe
concentrationisgraduallyincreased,subjectscandetectsomekindofsmell,eventhoughtheyc
annotdistinguishthetypeofsmell.Thisconcen-trationiscalledthedetectionthresholdva
lue.Recognitionthresholdvalue:Byincreasingtheodorconcentrationbeyondthedetection
threshold,thequalityorimpressionofthesmellatacertaincon-centrationcanbedescribed
.Theminimalconcentrationatwhichthequalityorimpressionofasmellcanbedescribediscal
ledtherecognitionthresholdvalue.Differentialthresholdvalue:Theminimumdifferencei
nconcentrationthatcanbedetectediscalledthedifferentialthresholdvalue.Usually,iti
sdescribedinpercentagesastheratiooftheminimaldetectabledifferencetotheoriginalin
tensity(Weberratio).Thisratiovariesdependingonthetypeofodor.Thedif-ferentialthre
sholdvaluegenerallyrangesfrom10to30percentforolfaction,asopposedto1to2percentfor
differencesinlightintensity.8.4.3AdaptationAftercontinuousexposuretoanodor,thesu
bjectiveintensityoftheodorgradu-allydecreases;afteracertainperiod(typicallysever
alminutes),theodorisnolongerconsciouslydetected.Thisphenomenoniscalledadaptation
,andisconsid-eredtobeamechanismofthesensingorganforautomaticallyadjustingsensiti
v-itytoanodor.Throughadaptation,smellsthatcontinuouslyexist,suchasbodyodor,areno
tdetected,whileodorsthatoccasionallyexistareclearlynoticed.Thelevelandthetempora
laspectofadaptationvarydependingontheodormate-rial.Theadaptationcausedbyasingleo
dormaterialiscalledself-adaptation.Therecoveryphaseisgenerallyfasterthantheadapt
ationphase.Cross-adaptationisthephenomenonofdecreasedsensitivityofoneodorthrough
exposuretoanother.Thisadaptationoccursforsomecombinationofodorsbutnotothers.Anot
hermechanismiscalledhabitation,whichisoftenusedinasimilarcon-texttoadaptation.Ha
bitationisconsideredtooccurwithinthebrain,whileadap-tationisafunctionoftheolfact
orycells.However,thedefinitionofhabitationappearstovaryamongresearchers,anditsde
finitionhasnotbeenstandardized.8.4.4MultiplierEffectandMaskingEffectWhentwodiffe
rentodorsaremixed,theintensityisenhancedinsomecasesandsuppressedinothers.Enhance
mentiscalledthemultipliereffectandsuppressioniscalledthemaskingeffect.Themasking
effectinparticularisusedindailylife.Whenanaromaisintroducedintoaspacewhereanunpl
easantsmellexists,the
8OlfactoryInterfaces284

unpleasantsmellisperceivedtodecrease.Thisisasubjectivephenomenonknownassensoryde
odorization,astheunpleasantsmellisnotactuallyremoved.Typicalexamplesofsensorydeo
dorizationareperfumesanddeodorantsinlavatories.8.4.5NonlinearityChangesinconcent
rationsometimesresultinqualitativeratherthanquantitativechanges,especiallyforver
ylowconcentrations.Forexample,atypeoffecalodor(indole)changestoaflowerysmellatve
rylowconcentrations.Inaddition,amix-tureofsmellssometimesresultsinaqualitativech
angeofsmell.Thisphenome-noniscalledmodification.Usually,surroundingodorsconsisto
fmultiplecomponents,andtheseodorsaremodified.Theblendingofaromasisconsideredtobe
asophisticatedtechniqueincreatingdesiredaromasviamodification.8.4.6SpatialPercep
tionThequestionofhowhumansperceivethesurroundingenvironmentthrougholfactionisani
nterestingtopicforinteractivehumancomputerinterfaces,espe-ciallyinvirtualreality.
Findingthelocationofasmellsourceisanimportanttaskrelatedtospatialperceptionbyolf

action.Bymovingandcomparingthetrajectoryandthetemporalprofileoftheintensity,thes
mellsourcecanbelocated(Yamadaetal.,2006).Thismightbeconsideredtobeacounterpartto
themotionparallaxcueinvisualperception.Anotherinterestingquestioniswhethertherea
reolfactorycounterpartstobinocularcuesthroughtheuseoftwonostrils.Inhishistoricre
search,Bekesy(1964)reportedthatthetimedifferenceoftheolfactorystimuliforeachnostri
lcontributestothedirectionallocalizationofthesmellsource,justasthedirectionofaud
itorysourcescanbedeterminedbythedif-ferenceinsoundsperceivedbyeachear.Morerecent
ly,Porteretal.(2007)reportedthatscenttrackingisaidedbyinternostrilcomparisons.Co
nsideringthattheinternostrildistanceisonly2to3cm,whichislessthantheinterpupillar
y(6to7cm)andinteraural(approximately15cm)distances,thestereoeffectinolfac-tionmayn
otbeasstrongasthemotioneffect.However,itisveryinterestingtonotethathumansmakeuseof
twonostrilstohelpunderstandtheenvironmentbyolfaction.8.5INTERFACE-TESTINGTECHNIQ
UESHistorically,theevaluationofodorhaslongreliedonsensoryanalysisbyexperi-encedh
umantesters.Thismeansthatitwasdifficulttoevaluateasmellobjec-tively.Eventhoughex
periencedtestersweretrainedtoprovideobjectiveresultsratherthansubjectiveimpressi
ons,theseresultsareassumedtobeaffectedbyenvironmentalconditions,thetestersphysica
lcondition,andsoon.
8.5Interface-TestingTechniques285

However,recentprogressinchemicalsensorshasresolvedthisproblem.Thetechnicalfieldr
elatedtosmell-sensingmachineryiscalledelectronicnoseore-nose(Pearceetal.,2003).T
heelectronicnoseisasystemtoidentifythecomponentsofodors.Multiplegassensors(chemo
sensors)areusedinthesystemtodetect,ana-lyze,andcategorizemultiplecomponentodors.
Amongthevarioustypesofchemo-sensors,metaloxidesemiconductorsensors,conductingpol
ymersensors,andquartzmicrobalancesensorshavebeenfrequentlyusedforelectronicnoses
ystems.Metaloxidegassensorshavebeenusedtodetectflammablegasessuchasmethaneandpro
paneorpoisonousgassuchashydrogensulfide,andarenowusedasodor-sensingelementsaswel
l.Whenodormoleculesareabsorbedontothesurfaceofasemiconductor,theelectricconducti
vityoftheelementischangedaccordingtotheamountofabsorbedmolecules.Conductingpolym
ersensorsalsousetheprincipleofelectricconductivityvariation,whilequartzmicrobala
ncesensorshavearesonancefrequencyoutputthatvarieswhenodormoleculesareabsorbedint
hesensitivemembraneonthesurfaceofaquartzresonator.Oneproblemwithodor-sensingequi
pmentisthatthetemporalresponseisstillinsufficientforinteractiveuse.Theresponseti
meforatypicalgassensorisdozensofseconds.However,somerecentsensorshaveimprovedres
ponsetimes;iftheresponseofgassensorsinparticularbecomesfasterthanhumanolfact-ion
,thesesensorscouldbeincorporatedintotheinteractionloopofanolfactorydisplaysystem
.8.6DESIGNGUIDELINESAlthoughthedevelopmentofcomputer-controlledolfactoryinterfac
esisstillatanearlystage,someguidelinescanbeindicatedwhendesigningandimplementing
them.8.6.1ExpectedEffectofAppendinganOlfactoryInterfaceFirst,thedesignershouldta
keintoaccountwhetheritismeaningfuloreffectivefortheentiresystemtoappendanolfacto
rychannel.Iftheolfactoryinterfaceisinherenttothedesignersintention,suchascreating
odorasthemaincontentoftheinterface,thisdecisionisreasonable.However,theremaybeis
sueswhentheolfactoryinterfaceisnottheprimarymechanismofinteraction.Inthiscase,de
signersshoulddecidewhethertheolfactoryeffectisworththeadditionalcostandcomplexit
ytothesystem.
8OlfactoryInterfaces286

8.6.2NumberofOdorsAsmentionedpreviously,thereisnomethodforsynthesizingarbitrarys
mellsfromasmallsetofcomponentodors.Hence,determiningthenumberofodorsrequiredfort
heapplicationisimportant.Eventhoughhumanscandistinguishthousandsofsmells,thenumb
erofsmellsrequiredforasingleapplicationmayberelativelysmall.Forexample,ifwewantt
oemitsmellsaccordingtotheusersbehaviorinaninteractivegame,severaltoafewdozenodors

maybesufficient.Inthiscase,itissufficienttoblendarelativelysmallnumberofsmellsor
selectpre-blendedaromas;blendinghundredsorthousandsofodorsisnotnecessary.Thenumb
erofsimultaneouslyrequiredodorsdirectlyaffectsthecomplexityofthesystem;thus,care
fulselectionoftheodorsisessentialforcreatingcost-effectivesystems.8.6.3NumberofU
sersandSpatialDimensionsThenextpointtoconsideristhenumberoftargetusersforsimulta
neoussmelldeployment.Iftheapplicationisforasingleuser,adeviceofpersonalsize,such
asasmallscentgeneratorlocatedclosetotheuser,canbeapplied.Iftheapplica-tionisform
anysimultaneoususers,odorsshouldbedeliveredtoalargerspace.Therearetwoapproachest
oachievemultiuserolfactoryinterfaces:providingapersonaldeviceforeachuserorcontro
llingalarge-volumeolfactoryspaceatanyonetime.Theeffectsofthevolumeofspaceonthete
mporalaspectsofodorarediscussedinthenextsubsection.8.6.4TemporalAspectsofOdorCon
trolWhenapplyingolfactoryinterfaceswithaudiovisualprogramsorinteractiveappli-cat
ions,thespeedofchangesorreplacementsinolfactorystimuliisanimportantissue.Iftheod
oristobeusedasbackgroundorambientmedia,thetimeconstantisrelativelylargewithconst
antorslow/gradualchangesinodor.Thisisaneffec-tivewayofusingodortoprovideinformat
ioninasubtlemanner.However,theuseofolfactoryinterfacestogetherwithaudiovisualpro
gramsorapplicationswithrapidinteractionrequiresshort-termchangingorswitchingofod
ors.Inthiscase,designersfacetheproblemthatemittedsmellscannotbeerasedinstantaneo
usly.Onceanodorisdiffusedintheair,itremainsunlessitismovedbyaircurrents.Possible
solutionsare(1)designtheentirefacilitytoincludeventilationsystemsand(2)emitassma
llanamountofscentaspossible.Thefirstapproachiseffectivewhentheolfactoryinterface
sareusedindedi-cated,speciallydesignedfacilities,suchasamusementparks.Theolfacto
ryinter-facesusedatsuchlocationscansimplyemitalargeamountofodormaterials,
8.6DesignGuidelines287

whichensurestheolfactoryexperience.Theemittedodorsarethenremovedbytheventilation
systemandfreshairoranotherodorisprovided.Thisapproachcanalsobeappliedtoaclosed-b
oothconfiguration.AnexampleofthistypeistheinteractivesmellsystemshowninFigure8.1
7,whichmakesuseofaslow,undetectableairflowtoproducespatiotemporalpatternsofsmell
inarectangularparallel-pipedbooth.Ifonecannotmakededicatedfacilities,suchasforho
meuse,thenthesec-ondapproachwouldbeeffective.Becausetheamountoftheemittedsmellis
small,theodoremissiondeviceshouldbeplacedclosetothenose.Althoughtheemittedsmelli
sdiffusedintothesurroundingatmosphere,theamountissufficientlysmallthattheconcent
rationquicklydiffusesbelowthedetectionthresholdandthesmellisunnoticed.Therearese
veralapproachesofthistype,dependingonthelocationoftheodor-emittingdevice.Nose-mo
untedandface-mountedtypesarepossiblesolutions(Yamadaetal.,2006).Odoremissionpoin
tscanincludeotherpartsofthebody,suchasthechest(Morieetal.,2003)orarm(Mochizukiet
al.,2004).Foruserswhodonotwanttoweardevices,alocalscentdeliverytechniquethroughf
reespacecanbeadopted(Watkins,2002;Yanagidaetal.,2004).
SweatHigh-pressure chamberFan array intakeDiffusion chambersInteraction zoneAir
movement0.2 M/SLow-pressure chamberExtract ductSmell wallCoffee shopLaundromatTi
resPetrolFresh airRubbishCut
g
rass
FIGURE8.17ScentsofSpace.Aninteractivesmellsystem.Source:FromHaque(2004),HaqueDes
ignandResearch/JosephinePletts.
8OlfactoryInterfaces288

8.7CASE STUDIESTwo case studies exploring olfactory interfaces can be found at w

ww.beyond-thegui.com.8.8FUTURETRENDSAsmentionedpreviously,olfactoryinterfacesare
stillinanearlystageofdevelop-mentcomparedtovisual,auditory,andhapticinterfaces.I
nrecentyears,however,researchpapersand/ordemonstrationshavebeenpresentedinalmost
everyacademicconferencerelatedtohumancomputerinteraction,includingthefieldofvirtu
alreality.Thisimpliesthatmanyresearchersinvariouscountriesarenowinterestedinolfa
ctoryinterfacesandthatresearchersinthisfieldarecur-rentlynetworkingtoestablishar
esearchcommunity.Thefutureofolfactoryinterfacesdependsonatleasttwofactors(Wasbur
n&Jones,2004):thedevelopmentofpracticalapplicationstomakeolfactoryinterfacesmore
common,andfundamentalimprovementsinolfactoryinterfaces,includingtheelucidationof
humanolfactionatboththephysiologicalandpsychologicallevels.Fromatechnicalviewpoi
nt,itisveryimportanttoknowhowmanytypesofodorsarenecessaryforacertainapplicationa
ndforproducingarbitrarysmells.Ifthisnumberislessthanahundred,constructingpractic
alinterfaceswillnotbedifficult.REFERENCESAmoore,J.E.(1970).MolecularBasisofOdor.
Springfield,IL:CharlesC.Thomas.Barfield,E.,&Danas,E.(1996).Commentsontheuseofolf
actorydisplaysforvirtualenvir-onments.Presence5(1):10921.Bekesy,G.(1964).Olfactorya
naloguetodirectionalhearing.JournalofAppliedPhysiology19(3):36373.BuckL.,&Axel,R.
(1991).Anovelmultigenefamilymayencodeodorantreceptors:Amolecularbasisforodorreco
gnition.Cell65:17587.Cater,J.P.(1992).Thenosehaveit!Presence1(4):49394.Cater,J.P.(
1994).Smell/taste:Odorsinvirtualreality.ProceedingsofIEEEInternationalConference
onSystems,ManandCybernetics,SanAntonio,TX.Davide,F.,Holmberg,M.,&Lundstrom,I.(200
1).Virtualolfactoryinterfaces:Electronicnosesandolfactorydisplays.InRiva,G.,&Dav
ide,F.,eds.,CommunicationsThroughVirtualTechnologies:Identity,CommunityandTechno
logyintheCommunicationAge.Amsterdam:IOSPress,193219.Dinh,H.Q.,Walker,N.,Song,C.,K
obayashi,A.,&Hodges,L.(1999).Evaluatingtheimport-anceofmulti-sensoryinputonmemor
yandthesenseofpresenceinvirtualenvironments.ProceedingsofIEEEVirtualReality,Hous
ton.
8.8FutureTrends289

Haque,U.(2004).ScentsofSpace:Aninteractivesmellsystem.ProceedingsofACMSIG-GRAPH2
004Sketches,LosAngeles.Heilig,M.L.(1962).SensoramaSimulator,U.S.Patent3,050,870.
Heilig,M.L.(1992).Elcinedelfuturo:Thecinemaofthefuture.Presence1(3):27994.Kaye,J.
N.(2001).SymbolicOlfactoryDisplay.Mastersthesis,MassachusettsInstituteofTechnolog
y.Kaye,J.N.(2004).Makingscents.ACMInteractions11(1):4961.Lorig,T.S.,Elmes,D.G.,Za
ld,D.H.,&Pardo,J.V.(1999).Acomputer-controlledolfac-tometerforfMRIandelectrophys
iologicalstudiesofolfaction.BehaviorResearchMeth-ods,Instruments&Computers31(2):
37075.McCarthy,R.E.(1984).Scent-EmittingSystems,U.S.Patent4,603,030.Mochizuki,A,A
mada,T.,Sawa,S.,Takeda,T.,Motoyashiki,S.,Kohyama,K.,Imura,M.,&Chihara,K.(2004).F
ragra:Avisual-olfactoryVRgame.ProceedingsofACMSIGGRAPH2004Sketches,LosAngeles.Mo
rie,J.F.,Iyer,K.,Valanejad,K.,Sadek,R.,Miraglia,D.,Milam,D.,Williams,J.,Luigi,DP.
,&Leshin,J.(2003).Sensorydesignforvirtualenvironments.ProceedingsofACMSIGGRAPH20
03Sketches,SanDiego.Nakaizumi,F,Yanagida,Y,Noma,H.,&Hosaka,K.(2006).SpotScents:A
novelmethodofnaturalscentdeliveryusingmultiplescentprojectors.ProceedingsofIEEEV
irtualReality2006,Arlington,VA,20712.Nakamoto,T.,&Hiramatsu,H.(2002).Studyofodorr
ecorderfordynamicalchangeofodorusingQCMsensorsandneuralnetwork.SensorsandActuato
rsB85(3):26369.Nakamoto,T.,&Minh,H.P.D.(2007).Improvementofolfactorydisplayusings
olenoidvalves.ProceedingsofIEEEVirtualReality,Charlotte,NC.Ooyama,A.(1994).Handb
ookofSensationandPerception.Tokyo:SeishinShobor.Pearce,T.C.,Schiffman,S.S.,Nagle
,H.T.,&Gardner,J.W.,eds.(2003).HandbookofMachineOlfaction.Weinheim:Wiley-VCH.Pla
ttig,K.H.(1995).SpurnasenuntFeinschmecker.Berlin:Springer-Verlag.Porter,J.,Craven
,B.,KhanR.M.,ChangS.J.,Kang,I.,Judkewitz,B.,Volpe,J.,Settles,G.,&Sobel,N.(2007).
Mechanismofscent-trackinginhumans.NatureNeuroscience10(1):2729.Tonoike,M.,ed.(200
7).[InformationandCommunicationTechnologyofOlfaction](inJap-anese).Tokyo:Fragran
ceJournalLtd.TriSenx.(2007).Homepageavailableathttp://www.trisenx.com/.Yamada,T.,
Yokoyama,S.,Tanikawa,T.,Hirota,K.,&Hirose,M.(2006).Wearableolfac-torydisplay:Usi
ngodorinoutdoorenvironments.ProceedingsofIEEEVirtualReality2006,Charlotte,NC,1992

06.Yamanaka,T.,Matsumoto,R.,&Nakamoto,T.(2002).Studyofodorblenderusingsolenoidva
lvescontrolledbydeltasigmamodulationmethod.SensorsandActuatorsB87:45763.Yanagida,Y
.,Kawato,S.,Noma,H.,Tomono,A.,&Tetsutani,N.(2004).Projection-basedolfactorydispl
aywithnosetracking.ProceedingsofIEEEVirtualReality2004,Chicago.Yatagai,M.,ed.(20
05).Kaorinohyakka-jiten[Encyclopediaofaromas].Tokyo:Maruzen.WashburnD.A.,&Jones,
L.M.(2004).Couldolfactorydisplaysimprovedatavisualization?ComputinginScience&Eng
ineering6(6):8083.Watkins,C.J.(2002).Methodsandapparatusforlocalizeddeliveryofsce
ntedaerosols,U.S.Patent6,357,726.
8OlfactoryInterfaces290

9CHAPTERTasteInterfacesHirooIwata9.1NATUREOFTHEINTERFACETasteisanimportantsenset
hatisseldomusedintraditionaldisplays.Itisthefundamentalnatureoftheunderlyingsens
eandthedifficultyofimplementingunobtrusivedevicesthathavemadetasteinterfacessodi
fficult.9.1.1TheSenseofTasteHumansdetectchemicalfeaturesoffoodwithtastereceptorc
ells.Theyareassem-bledintotastebuds,whicharedistributedacrossthetongue.Althoughh
umanscantasteavastarrayofchemicalentities,theyevokefewdistincttastesensations:sw
eet,bitter,sour,salty,andumami.Distincttastereceptorcellsdetecteachofthefivebasict
astes(Chandrashekaretal.,2006).Examplesofthesebasictastesfollow:Sweet:Acommonswe
etsubstanceissucrose,knownastablesugar.Bitter:Astrongbittertastepreventsingestio
noftoxiccompounds.Asmallamountofabittersubstancecontributestoperceivedinterestin
gorgoodtaste/flavor.Caffeineincoffeeisanexample.Sour:Soursubstancesaremostlyacid
s.Vinegarisanexample.Salty:AcommonsaltysubstanceisNaCl,knownastablesalt.Umami:Th
isisaJapanesewordmeaningsavory,andthusappliestothesensa-tionofsavoriness.Typicalum
amisubstancesareglutamates,whichareespe-ciallycommoninmeat,cheese,andotherprotei
n-richfoods(Kawamura&Kare,1987).

9.1.2WhyIsaTasteInterfaceDifficult?Tasteisthelastfrontierofvirtualreality.Tastei
sverydifficulttodisplaybecauseitisamultimodalsensationcomprisingchemicalsubstanc
e,sound,smell,andhapticsensations.Theliteratureonvisualandauditorydisplaysisexte
nsive.Smelldisplayisnotcommon,butsmellcaneasilybedisplayedusingavaporizer.Althou
ghnotasextensiveasvisualandauditorydisplays,theliteratureonolfac-torydisplaysisg
rowing(e.g.,Nakamotoetal.,1994;Davideetal.,2001).Tasteperceivedbythetonguecanbem
easuredusingabiologicalmembranesensor(Tokoetal.,1994,1998).Thesensormeasuresthec
hemicalsubstanceofthefivebasictastes.Anyarbitrarytastecaneasilybesynthesizedfrom
thefivetastesbasedonsensordata.Anotherimportantelementintasteisfoodtexture.Measu
rementofbitingforcehasbeenstudied.Amultipointforcesensorhasbeenusedtomeasureforc
edistributionontheteeth(Khoyamaetal.,2001,2002).Thesensorypropertiesofthetexture
ofrealfoodhavealsobeenstudied(Szczesniak,1963,2002).Dentalschoolshavebeenworking
onthetrainingofpatientstochewprop-erly.Amasterslavemanipulatorformasticationhasbe
endeveloped(Takanobuetal.,2002).Arobotmanipulatedbythedoctorappliesappropriatefo
rcetothepatientsteeth.Hapticinterfacesforhandsorfingersarethefocusofmajorresearch
effortsinvirtualreality.Incontrast,researchintohapticdisplaysforbitingisveryrare
.9.2TECHNOLOGYOFTHEINTERFACETherehavebeennointeractivetechniquesdevelopedtodatet
hatcandisplaytaste.TheFoodSimulatorProjectwasanefforttodevelopjustsuchaninterfac
e.9.2.1ExistingTechniquesforDisplayingTasteThetraditionalmethodfordisplayingtast
eistheuseoffilterpaperdiscs,suchasforpatientswithtastedisorders(Tsuruoka,2003).A
smallfilterpaperdiscissoakedwithachemical(taste)substanceandputonthepatientstongu
e.Filterpaperdiscswithvariousdensityofthechemicalsubstanceareusedtotesttastediso
rders.However,therehasbeennotechniquetodisplaytasteforhumancomputerinteraction.9.
2.2FoodSimulatorProject:ATasteInterfaceChallengeIn2001,Ilaunchedaprojectcalledth
eFoodSimulatortodevelopaninterfacedevicethatpresentsfoodtextureaswellastaste.Theun
solvedproblemintaste

9TasteInterfaces292

displayisahapticsissue.Thefirstgoaloftheprojectwasdevelopingahapticinterfaceforb
iting.Thedeviceshouldbesuitablyshapedforplacinginthemouth,andbeeffectivelycontro
lledtosimulatefoodtexture.Thesecondgoalwastopresentamultimodalsensationtotheuser
.Tothisend,bitingsoundsandchemicaltastesshouldbeintegratedwiththehapticinterface
.Toachievethesegoals,wedevelopedahapticinterfacetopresentthebitingforce.TheFoodS
imulatorgeneratesaforcesimulatingthepreviouslycapturedforceprofilesofanindividua
lbitingrealfood.Afilm-likeforcesensorisusedtomeasurebitingforceassociatedwithrea
lfood.AforcesensorisinstalledintheFoodSimulatorandthedeviceisactuatedusingforcec
ontrolmethods.TheFoodSimulatorisintegratedwithauditoryandchemicalsensationsassociatedwithtaste.Thesoundofbitingiscapturedbyabonevibrationmicrophone.Thesoundist
henreplayedusingabonevibrationspeakersynchronizedwiththebitingaction.Thechemical
sensationoftasteisproducedusinganinjectionpump,withatubeinstalledattheendeffecte
r.9.3CURRENTIMPLEMENTATIONSOFTHEINTERFACETheFoodSimulatoremploysmechanicallinkag
estoapplybitingforcetotheteeth.Itisintegratedwithsound,vibration,andchemicaltast
e.9.3.1HapticDeviceinaFoodSimulatorThehapticdeviceiscomposedofaonedegree-of-free
dom(DOF)mechanismthatisdesignedtofitintheusersmouth.Thedeviceisaimedatapplyingafo
rcerepresentingthefirstbite.Chewingisnotaddressedinthisprototype.Thus,thedevicei
sdesignedtoapplyforceinadirectionnormaltotheteeth.Theconfigu-rationofthemechanic
allinkagetakesintoconsiderationthejawstructure.Theshapeofthelinkageenablestheapp
licationofforcetothebackteeth.Thewidthoftheendeffecteris12mm,andthedeviceapplies
forcetotwoorthreeteeth.Regardinghygiene,theendeffecterincludesadisposablecoverof
clothandrubber.Figure9.1showstheoverallviewoftheapparatus.Thehapticdeviceiscompo
sedofaonesDOFmechanismthatemploysfourlinkages.Figure9.2illustratesitsmechanicalc
onfiguration.ThelinkagesaredrivenbyaDCservomotor(MAXONMotor,RE25).Theuserbitesth
eendeffec-terofthedevice.Theworkingangleoftheendeffecteris35degrees,andthemax-im
umforceappliedtotheteethis135N.Aforcesensorthatdetectsforceappliedbytheusersteeth
isattachedtotheendeffecter.ThedeviceiscontrolledbyaPC(Pentium4,2GHz).Theupdatera
teofforcecontrolis1,700Hz,whichissuffi-cientforcontrollingahapticinterface.
9.3CurrentImplementationsoftheInterface293

9.3.2MultisensoryDisplayThesecondgoaloftheFoodSimulatorprojectwastopresentmultim
odalsensa-tion.Wealsotriedtointegrateamultimodaldisplaywiththehapticinterface.So
undSoundiscloselyrelatedtobitingaction.Masticationofahardfoodgeneratessound.This
soundisperceivedbyvibration,mostlyinthejawbone.Soundsofbitingrealfoodwererecorde
dusingabonevibrationmicrophone.Figure9.3showsarecordingsession.Thebonevibrationm
icrophonewasinsertedintotheear.Anaccelerometer,installedinthemicrophone,pickedup
vibrationofthejawbone.
FIGURE9.1Overallviewoftheapparatus.The1-DOFhapticdeviceisacoreelementofthesystem
.
MotorForcesensorBitingforce
FIGURE9.2Mechanicalconfigurationofthehapticdevice.Fourlinkagesareemployedinthede
vice,whichisdesignedtofitintothemouth.
9TasteInterfaces294

FIGURE9.3Measuringbitingsound.Anaccelerometerpicksupvibrationofthejawbonewhilebi
tingrealfood.
FIGURE9.4Bonevibrationspeaker.Therecordedsoundissynchronizedwiththebitingaction.
Therecordedsoundwasthendisplayedusingabonevibrationspeaker.Figure9.4showsthespea
kerthatgeneratedvibrationsinthejawbone.Thesoundwassynchronizedwiththebitingactio
n.Forexample,thesoundofavirtualcrackerwasdisplayedatthebeginningofthesecondstage
oftheforcecontrolshowninFigure9.7(seepage298).
9.3CurrentImplementationsoftheInterface295

ChemicalTasteChemicalsensationsperceivedbythetonguecontributegreatlytothesenseof
taste.Anarbitrarytastecanbesynthesizedfromthefivebasictastes.Table9.1summarizesc
ommonchemicalsubstancesforthebasictastes.Thechemicalsensationoftastewaspresented
byinjectingasmallamountofliquidintothemouth.Atubewasattachedtotheendeffecterofth
ehapticinter-face.Figure9.5showsthetubeatthetopendofthelinkage.Theliquidwastrans
-ferredusinganinjectionpump(Nichiryo,DDU-5000).Figure9.6showsanoverallviewofthei
njectionpump.Injectionoftheliquidwassynchronizedwiththebitingaction.Thepumpprovi
ded0.5mlofsolutionforeachbite.VisionandSmellVisualandolfactorysensationsoffoodoc
curindependentlyofthebitingaction.Thus,theFoodSimulatorprototypedidnotsupportthe
sesensations.However,head-mounteddisplayscanprovidevisualizationoffood.Also,smel
lcanbedisplayedusingavaporizer.Thesedisplayscouldeasilybeintegratedwiththehaptic
interface.
TABLE9.1ChemicalSubstancesUsedinSynthesisofFiveBasicTastesBasicTasteChemicalSubs
tanceSweetSucroseSourTartaricacidSaltySodiumchlorideBitterQuininesulfateUmamiSod
iumglutamate
FIGURE9.5Tubeforinjectionofchemicaltaste.Asmallamountofliquidisinjectedfromtheto
pendofthetubeintothemouth.
9TasteInterfaces296

9.4HUMANFACTORSDESIGNOFTHEINTERFACETheFoodSimulatorisdesignedtogeneratebitingfor
ceaccordingtothatofrealfood.Thereareanumberofhumanfactorsissuesthatmustbeconside
redwhendesigningatasteinterface.9.4.1MeasurementofBitingForceofRealFoodTheFoodSi
mulatorgeneratesforce,simulatingtheforcerecordedfromanindi-vidualbitingrealfood.
Afilm-likeforcesensor(FlexiForce,KamataIndustries)isusedtomeasurethisbitingforce
.Thesensorhasathicknessof0.1mmandacircularsensitiveareaof9.5mmindiameter.Thesens
ingrangeis0to110,withthemaximumsamplingrateof5,760Hz.Figure9.7showsanoverallview
ofthesensor.Figure9.8showsaviewoftheparticipantintheexperiment,whoisbitingacrack
erandtheFlexiForcesensor.Figure9.9showsmeasuredbitingforceassociatedwitharealcra
cker.Twopeaksappearedintheprofileofthemeasuredforce.Thefirstpeakrepresentsdestru
ctionofthehardsurfaceofthecracker.Thesecondpeakrepresentsdestructionofitsinterna
l
FIGURE9.6Injectionpump.Theinjectionoftheliquidissynchronizedwiththebitingaction.
Thepumpprovides0.5mlofsolutionforeachbite.
9.4HumanFactorsDesignoftheInterface297

FIGURE9.8Forcesensorandrealfood.Theparticipantisbitingrealfoodalongwiththeforces
ensor.
0012345[N][s]67890.10.20.30.40.5
FIGURE9.9Measuredforceofacracker.Thefirstpeakrepresentsdestructionofthehardsurfa
ceofthecracker.Thesecondpeakrepresentsdestructionofitsinternalstructure.
FIGURE9.7Film-likeforcesensor.Thesensorisplacedinthemouthwithrealfoodtomeasurebi
tingforce.
9TasteInterfaces298

structure.Figure9.10showsthemeasuredbitingforceassociatedwithrealcheese.Theslope
observedinthefigurerepresentstheelasticdeformationofthecheese.9.4.2MethodofGener
atingBitingForceTwofoodtypesaresimulated:crackersandcheese.CrackersTheforcecontr
olofthedevicehastwostages(Figure9.11).First,thedeviceappliesforcetomaintainitspo
sition.Thisprocessrepresentsthehardsurfaceofthevirtualcracker.Whenthebitingforce
exceedsthefirstpeak,theforcecontrolmovestothesecondstage.Thedevicegeneratesthesa
meforceasthatmeasuredfromtherealcracker.Thisstageissubjecttoopen-loopcontrol.Des
tructionoftheinternalstruc-tureofthevirtualcrackerissimulatedbythesecondstage.Fi
gure9.12showsthemeasuredforceofthevirtualcrackerdisplayedbytheFoodSimulator.Thep
rofileofthemeasuredforcehastwopeaks,asintherealcracker.However,theshapeoftheprof
ileisdifferentfromthatassociatedwiththerealcracker.Thisdifferenceseemstobecaused
bytheopen-loopcontrol.CheeseFigure9.13showstwostagesofforcecontrolforbitingrealc
heese.Inordertosim-ulateelasticdeformationofthecheese,thespringconstantofthechee
sewasesti-matedfromthecollecteddata.Thedevicegeneratesforceaccordingtothisspring
constanttodisplaytheelasticityofthevirtualcheese.Whenthebitingforceexceedsthepea
kforce,thecontrolentersthesecondstage.Thedevicegeneratesthesame
00246[N][s]81012140.20.40.60.811.2
FIGURE9.10Measuredforceofcheese.Thesloperepresentstheelasticdeformationofthechee
se.
9.4HumanFactorsDesignoftheInterface299

0012345[N][s]Stage 1Stage 267890.10.20.30.40.5

FIGURE9.11Methodofsimulationofacracker.First,thedeviceappliesforcetomaintainitsp
osition.Whenthebitingforceexceedsthefirstpeak,thedevicegeneratesthesameforceasth
atmeasuredfromtherealcracker.
0012345[N][s]6789100.20.40.60.8
FIGURE9.12Measuredforceofavirtualcracker.Twopeaksareobserved.
9TasteInterfaces300

00246[N][s]Spring constantStage 1Stage 281012140.20.40.60.811.2

FIGURE9.13Simulationmethodforcheese.Thedevicegeneratesforceaccordingtothespringc
onstantoftherealcheese.
00246[N][s]810120.20.40.60.811.41.2
FIGURE9.14Measuredforceofvirtualcheese.Theprofileresemblesthemeasurementsforreal
cheese.forceasthatmeasuredfromtherealcheese.Thisstageissubjecttoopen-loopcontrol
.Destructionofthevirtualcheeseissimulatedbythesecondstage.Figure9.14showsthemeas
uredforceofthevirtualcheesedisplayedbytheFoodSimulator.Theprofileofthemeasuredfo
rceissimilartotherealone.
9.4HumanFactorsDesignoftheInterface301

9.5TECHNIQUESFORTESTINGTHEINTERFACETheabilityoftheFoodSimulatortorepresentvirtua
lfoodswasevaluated.Themajorobjectiveofthisexperimentwastodiscoverwhetherthedevic
ecouldrepresentdifferencesinfoodtexture.Experiment1Task:Participantswereaskedtob
itethedeviceandanswerwhetherthevirtualfoodwasacrackerorcheese.Participants:Thepa
rticipants(22to24yearsofage)weresixuniversitystudents(male)whovoluntarilypartici
patedintheexperiment.Procedure:Eachparticipanttried10virtualfoods.Crackersandche
eseswererandomlydisplayed,withthesubjectsbeingrequestedtodistinguishbetweenthetw
ofoods.Results:Theanswerswere98.3percentcorrect.Discussion:Alloftheparticipantsw
ereperfectlyabletodistinguishbetweenthetwovirtualfoods.Thedevicethussucceededinr
epresentinghardnessofthefood,perceivedbythefirstbite.Experiment2Task:Thefollowin
gthreefoodsweredisplayed:FFoodA:Cracker(sameasinthepreviousexperiment)FFoodB:Bis
cuit(CalorieMate)FFoodC:Japanesecrispysnack(Ebisen)Thesefoodspossessasimilartexture.
Figures9.15and9.16showtheforceprofilesofFoodsBandC,respectively.Thesameforcecont
rolmethodasshowninFigure9.11wasusedtosimulateFoodsBandC.Theparticipantswereasked
tobitethedeviceandselectoneofthethreefoods.Participants:Theexperimentwasperforme
dusingthesameparticipantsasinExperiment1.Procedure:Eachfoodwasdisplayed60times;t
hus,eachparticipantsampled180virtualfoods.Thesubjectswererequestedtodistinguisha
mongthreerandomlydisplayedfoods.Results:Thepercentagesofcorrectanswersfollowed:F
FoodA:69.1percentFFoodB:59.1percentFFoodC:28.2percentDiscussion:Forceprofilesoft
hecrackerandtheEbisenwereverysimilar.ThepeakforceoftheEbisenwasgreaterthanthatof
thecracker,butmost
9TasteInterfaces302

0051015[N][s]20250.511.5
FIGURE9.15Measuredforceofabiscuit.Onlyonepeakisobserved.
002610144[N][s]812160.511.5
FIGURE9.16MeasuredforceofaJapanesesnack.Twopeaksareobservedthatareverysimilartot
hoseofthecracker.
9.5TechniquesforTestingtheInterface303

participants were unable to distinguish the Ebisen from the cracker. The peakfor
ce of the Calorie Mate was the greatest of the three foods. The internalstructur
e of the Calorie Mate is soft so that the force profile did not exhibita second

peak. This difference contributed to the 60 percent of correctanswers.9.6DESIGN

GUIDELINESDesign of the Food Simulator is in a very preliminary state and it has
much roomfor improvement. However, the following items should be considered for
design-ing the device.F Linkages of the force feedback device should be fit to
the structure of theusers jaw.F The mouthpiece of the force feedback device shoul
d be thin, but it has to sup-port large biting force. The material of the linkag
e should be carefully chosen.F Due to sanitary issues, the mouthpiece should be
covered by disposable mate-rial. The material should not infringe taste perceive
d by the tongue.F The injector tube for the chemical taste should be set at an a
dequate positionsothattheliquiddropsonthetongue.9.7CASESTUDYA case study of a ta
ste interface can be found at www.beyondthegui.com .9.8FUTURETRENDSAmajoradvanta
geoftheFoodSimulatoristhatitcandisplaybitingforcebytheuseofasimplemechanism.Alth
oughtheapparatusisinexpensive,itcaneffec-tivelydisplayfoodtexture.Themethodoffor
cecontrolrequiresimprovement,butmostoftheparticipantsintheSpecialInterestGroupon
GraphicsandInterac-tiveTechniques(SIGGRAPH)greatlyenjoyedthedemonstration.Ontheo
therhand,majorlimitationsofthecurrentprototypeareconcernedwithitsthicknessandthe
weightofthelinkages.Thelinkagesare2mmthick.Becausegreatforcemustbesupported,thed
evicemustbefabricatedinsteel.Whenusersfinishbiting,theycanfeeltheunnaturalsensat
ionofthelinkagethickness.Theweightofthelinkagecausesunwantedvibration.Theuserstee
thcontacttheflatsurfaceofthelinkage.Thisflatsurfacedegradesfoodtexture.Apersonfe
elstheforceonindependentteethwhilebitingrealfood.Moreover,thecurrentdeviceapplie
sforceonlytotheteeth.Thetexture
9TasteInterfaces304

ofrealfoodisperceived,inpart,bythetongue.However,displayingfoodtexturetothetongu
eisverydifficult.Bitingexercisescontributetohumanhealth.Someofthemanyapplication
areasfortheFoodSimulatorfollow:Training:TheFoodSimulatorcanbeprogrammedtogenerat
evariousforcesotherthanthatofrealfood.Elderlypeoplecanpracticebitingwithreducedr
esistancetotheteeth.Ontheotherhand,increasedresistanceenablesyoungerpeopletoperc
eivethedifficultyinbitingexperiencedbyelderlypeople.Entertainment:TheFoodSimulat
orcanchangethepropertiesoffoodwhilechew-ing.Acrackercanbesuddenlychangedtoagel.T
heusercanenjoyanovelexpe-riencewhilechewing.Thiskindofentertainmentcontributesto
thechewingcapabilityofchildren.Fooddesign:Preferredresistancebytheteethcanbefoun
dusingtheFoodSimu-lator.Suchfindingscouldcontributetothedesignofnewfoods.Thischa
pterhasdemonstratedahapticdevicethatsimulatesthesenseofbit-ing.Thedevicepresents
aphysicalpropertyoffood.TheFoodSimulatorhasbeenintegratedwithauditoryandchemical
tastesensations.Thechemicalsensationwassuccessfullydisplayedbyatubeandaninjectio
npump.Thedevicecanbeusedinexperimentsofhumantasteperception.Tasteisamultimodalse
nsation,makingitverydifficulttocontrolmodalitybyusingrealfood.Thedevicecandispla
yfoodtexturewithoutchemicaltaste,unlikerealfood.Thischaracteristiccontributestoe
xperimentsinmodalityintegration.Futureworkwillincludethepsychologicalstudyofsens
oryintegrationregardingtaste.REFERENCESChandrashekar,J.,Hoon,M.A.,Ryba,N.J.,&Zuk
er,C.S.(2006).Thereceptorsandcellsformammaliantaste.Nature444:28894.Davide,F.,Hol
mberg,M.,&Lundstrom,I.(2001).Virtualolfactoryinterfaces:Electronicnosesandolfacto
ry.InRiva,G.,&Davide,F.,eds.,CommunicationsthroughVirtualTechnologies:Identity,C
ommunityandTechnologyintheCommunicationAge.Amsterdam:IOSPress,16072.Kawamura,Y.,&
Kare,M.R.(1987).Umami:ABasicTaste.NewYork:MarcelDekker.Kohyama,K.,Ioche,L.,&Mart
in,J.F.(2002).Chewingpatternsofvarioustexturedfoodsstudiedbyelectromyographyinyo
ungandelderlypopulations.JournalofTextureStud-ies33(4):26983.Kohyama,K.,Sakai,T.,
&Azuma,T.(2001).Patternsobservedinthefirstchewoffoodswithvarioustextures.FoodSci
enceandTechnologyResearch7(4):29096.Nakamoto,T.,Ustumi,S.,Yamashita,N.,Moriizumi,
T.,&Sonoda,Y.(1994).Activegassensingsystemusingautomaticallycontrolledgasblender
andnumericaloptimizationtechnique.SensorsandActuatorsB20:13137.
References305

Szczesniak,A.S.(1963).Classificationoftexturalcharacteristics.JournalofFoodScien
ce28:38589.Szczesniak,A.S.(2002).Textureisasensoryproperty.FoodQualityandPreferen
ce13:21525.Takanobu,H.,Takanishi,A.,Ozawa,D.,Ohtsuki,K.,Ohnishi,M.,&Okino,A.(2002
).Integrateddentalrobotsystemformouthopeningandclosingtraining.ProceedingsofICRA
,142833.Toko,K.,Matsuno,T.,Yamafuji,K.,Hayashi,K.,Ikezaki,H.,Sato,K.,&Kawarai,S.(
1994).Multichanneltastesensorusingelectricpotentialchangesinlipidmembranes.Biose
n-sorsandBioelectronics9:35964.Toko,K.(1998).Electronictongue.Biosensors&Bioelect
ronics13:701709.Tsuruoka,S.,Wakaumi,M.,Nishiki,K.,Araki,N.,Harada,K.,Sugimoto,K.,
&Fujimura,A.(2003).Subclinicalalterationoftastesensitivityinducedbycandesartanin
healthysubjects.BritishJournalofClinicalPharmacology305:80712.
9TasteInterfaces306

10CHAPTERSmall-ScreenInterfacesDanielW.Mauney,ChristopherMasterton10.1NATUREOFTH
EINTERFACETheuseofsmallscreensasatoolfordisplayingdynamicinformationisbecom-ingu
biquitous.Displaysrangefromverysimplescreensasseenonclocks,microwaves,alarmsyste
ms,andsoon,tohighlycapablegraphicaldisplaysasseenonmobilephones,medicaldevices,h
andheldgamingdevices,andper-sonaldigitalassistants(PDAs).Thenumberofproductssold
withsmallscreensisstaggering.Mobilephonesales,justonecategoryofsmall-screenprodu
cts,wereestimatedat825millionunitsin2005(Gohring,2006)andhaveover2.6billionsubsc
ribersworldwide(Nystedt,2006).Incomparison,personalcom-puter(PC)saleswereestimat
edat208.6millionunitsin2005(Williams&Cowley,2006).Small-screendesignprimarilymak
esuseofthevisualsystem.Questionsinsmall-screendesignoftencenteronhowsmalldataele
mentscanbedisplayedsothattheyareproperlyseenorrecognized.Togainadeeperunderstand
ingoftheanswerstothisquestion,abriefbackgroundoflight,eyeanatomy,andthesensitivi
tyandacuityoftheeyearediscussed.10.1.1LightLightisgenerallydescribedbyitswavelen
gthandintensity.Thesubjectiveorpsychologicalcorrelatetowavelengthiscolor(moreacc
uratelyknownashue).Thus,thecolorthatpeopleseeistheirperceptionofthelightswaveleng
th.Wave-lengthismeasuredinnanometers(nm).Thesubjectiveorpsychologicalcorrelateto
intensityisbrightness.Brightnessistheimpressionproducedbytheintensityoflightstri
kingtheeyeandvisualsystem.

Peopledonotnormallylookdirectlyatalightsource.Mostofthelightseenisreflectedfroms
urroundingsurfaces.Themeasurementoftheintensityoflightfocusesontwoaspects:theamo
untoflightfallingonanobject,calledillumi-nance,andtheamountoflightemittedfromorr
eflectedfromasurface,calledluminance.ThecommonEnglishunitofilluminanceisthefootcandle(ft-c),whilethemetricunitofilluminanceisthemeter-candle(m-c).ThecommonEngl
ishunitofluminanceisthefoot-Lambert(ft-L),whilethemetricunitoflumi-nanceisthemil
lilambert(mL).Theamountoflightemanatingfromasourceiscalledradiance.Toputallofthi
stogether,considertheactofreadingabook.Theamountoflightgeneratedbythelightbulbis
calledradiance,theamountoflightfallingonthebookspagesiscalledilluminance,theamoun
toflightreflectedfromthosepagesistheluminance,andtheamountoflightperceivedbythev
isualsystemiscalledbrightness.Contrastisameasurementoftheluminancedifferencebetw
eenatargetanditsbackground.Itisoftenexpressedasaratio,suchas10:1.10.1.2TheEyeAsl
ightraysentertheeye,theyfirstpassthroughthecornea,thepupil,andthelenstoultimatel
yfallontheretina.Thecorneaandthelensrefract(orbend)thelightsothatimagescomeintof
ocusontheretina.Theiriscontrolstheamountoflightenteringtheeyebydilatingorconstri
ctingthesizeofthepupil(theroundblackopeningsurroundedbytheiris).Theretinaiscompo
sedofnervecellsandphotoreceptorsthataresensitivetolight.Therearetwotypesofphotor
eceptors:rodsandcones.Rodsareheavilyconcentratedintheperipheralregionoftheretina
,whileconesareconcentratedprimarilyinasmallpitcalledthefovea.Manyrodsshareacommo

nopticnervefiber,whichpoolstheirstimulationandaidssensitivitytolowerlevelsofligh
t.Incontrast,coneshaveamoreorlessone-to-onerelationshipwithopticnervefibers,whic
haidstheirimageresolution,oracuity.Visionaccomplishedprimarilywithconesiscalledp
hotopicvision,andvisionaccomplishedprimarilywithrodsiscalledscotopicvision.Onlyi
nphotopicvisiondopeopleactuallyperceivecolors.Inscotopicvision,theweaklightsarev
isible,butnotascolors.Instead,allwavelengthsareseenasaseriesofgreys.10.1.3Sensit
ivityInpoorlylitenvironments,scotopicvisiondominatesbecauserodsarefarmoresensiti
vetolight.Sincerodsarelocatedprimarilyintheperipheryoftheretina,sourcesoflowligh
tarebestseeninthevisualperiphery.Thisexplainswhyitiseasiertoseeafaintstaratnight
whennotfixatingdirectlyonit.Thesensitivityoftheeyetolightisheavilydependentonthe
wavelengthofthelight.Figure10.1showsthespectralthresholdcurvesforphotopicandscot
opic
10Small-ScreenInterfaces308

vision.Thesecurvesshowthattherodsaremaximallysensitivetowavelengthsofaround500nm
(yellowgreen),whiletheconesaremaximallysensitivetowave-lengthsofaround550nm(green
).Italsoshows,asnotedbefore,thatscotopicvisionisgenerallymoresensitivetolighttha
nphotopicvision.10.1.4VisualAcuityThesmallestdetailtheeyeiscapableofresolvingata
particulardistanceisknownastheminimumvisualacuity(Grether&Baker,1972).Visualacui
tyofanobjectdependsonbothitssizeanditsdistancefromtheviewer.Asmallobjectthatis
Logarithm of relative radiant flux requiredfor threshold vision(arbitrary units)
43210400500Scotopic(rod)visionPhotopic(cone)vision600Wavelength in nanometers700
FIGURE10.1Spectralthresholdcurvesforphotopicandscotopicvision.Rodsaremaximallyse
nsitivetowavelengthsofaround500nm,comparedto550nmforcones.Scotopicvisionisgenera
llymoresensitivetolightthanphotopicvision.Source:FromSchiffman(1990),asadaptedfr
omChapanis(1949);courtesyJohnWileyandSons,Ltd.
10.1NatureoftheInterface309

veryclosemayappearlargerthanalargeobjectthatisfaraway.Thedistin-guishingfeatureo
fanobjectssize,therefore,isthesizetheobjectprojectsontheretina,alsoknownasitsvisu
alangle(Figure10.2).Theprobabilityofanobjectbeingdetectedisdirectlyrelatedtothev
isualangle,expressedinseconds,minutes,anddegrees,ofthatobjectsubtendedattheeye.F
igure10.3graphstheprobabilityofdetectinganobjectatdifferentvisualangles.FactorsA
ffectingAcuityAnumberoffactorsaffectacuity,includingillumination,contrast,time,a
ndwave-length.Ingeneral,asillumination,contrast,andtimespentfocusingonatarget
Target 1Size = 2Distance = 25Target 2Size = 1Distance = 12.5Target 3Size = .5Dis
tance = 6.25Visual angleb
FIGURE10.2Impactofvisualangle.Allthreetargetsappeartobethesamesizebecausetheirvi
sualangleisthesame.Source:FromSchiffman(1990);courtesyJohnWileyandSons,Ltd.
100755025000.20.40.60.8Visual an
g
le (min)Probability of detection (%)1.01.21.41.6
FIGURE10.3Probabilityofdetectionforobjectsofvaryingvisualangles.Thisprobabilityi
sdirectlyrelatedtothevisualangleofanobjectsubtendedattheeye.Source:FromGretheran

dBaker(1972),asadaptedfromBlackwell(1946);courtesyJohnWileyandSons,Ltd.
10Small-ScreenInterfaces310

increase,acuityimproves.Wavelengthhaslittleimpactonacuityifthereishighluminancec
ontrast.However,ifthereislowcontrast,colorcontrastwillimproveacuity.Aredsignalis
mosteasilyseeninlow-contrastconditions,followedbygreen,yellow,andwhiteinthatorde
r(Sanders&McCormick,1993).10.1.5ColorColorperceptionistheabilitytodiscriminateam
ongdifferentwavelengthsoflight.Itisusefultonotethatthelightitselfisnotcolored.In
stead,differentwavelengthsoflightproducethesensationofdifferentcolors.Thus,color
isapsychologicalexperi-encethatdifferentwavelengthsoflighthaveonthenervoussystem
.Therearethreepsychologicaldimensionsofcolor:hue,brightness,andsaturation.Huevar
ieswithchangesinwavelength,andthetermisoftenusedinterchangeablywiththewordcolor.
Brightnessvarieswiththephysicalintensityofthelight.Saturationreferstothespectral
purityofthewavelength.Theadditionofotherwavelengths,whitelight,orgreytoasinglewa
velengthwilldesaturatethecolor(Schiffman,1990).Approximately8percentofmenand0.5p
ercentofwomenhavesomeformofcolorvisiondeficiency(Sanders&McCormick,1993).Themost
commonformofcolorweaknessinvolvestheinabilitytodifferentiatebetweenredandgreen.A
ssuch,colorshouldneverbeusedasaprimarycue(theprimarydistinguishingfeaturebetween
twoitems),andrequiringuserstodistinguishbetweenthecolorsofredandgreenshouldbeavo
ided.10.2TECHNOLOGYOFTHEINTERFACEUntil1970,theprimarydisplaytechnologyavailablet
oelectronicsdesignerswasthecathoderaytube(CRT).AlthoughtheCRTisstillinusetodayan
dstillout-performscompetingtechnologiesinsomeareas(especiallymanufacturingcost),
ithastwomajordrawbacks:sizeandpowerdraw.Cathoderaytechnologyrequiresagreatdealof
powertofireelectronsatthescreenandarelativelygreatdistanceinordertodisplaythosee
lectronsonareasonablysizedscreen.DespitecontinuedimprovementsintheCRTengineering
process,thesetwoimportantflawshavenotbeenovercome.In1970,theSwissteamofSchadtand
Helfrichbuilttheworldsfirstliquidcrystaldisplay(LCD).WhileoriginalLCDdesignshadth
eirownshortcomings(namelycost,responsetime,andcontrast),thetechnologyovercamethe
limita-tionsoftheCRTbyrequiringmuchlesspowerandavastlydecreaseddepth.Thecontinue
dimprovementoftheLCDhasledtotheemergenceofanenormousrangeofsmall-screenproducts,
fromthesimpledigitalwatchtopower-fullaptopcomputers.Inthissection,weexploretheva
rioustechnologiesusedincurrentsmall-screendesign.
10.2TechnologyoftheInterface311

10.2.1DisplayTechnologiesToday,LCDtechnologycanbebrokenintotwomajorcategories:ac
tive-andpassive-matrixdisplays.Passive-matrixdisplaysworkbyrapidlyactivatingcorr
espondingrowsorcolumnsofthedisplaysequentiallytoupdatethescreenwithnewimages.The
tech-nologyiscalledpassivebecauseeachpixelmustretainitsstatebetweenrefresheswithou
tanelectricalcharge.Passive-matrixscreensrequirelesspowerandarelesscostlytomanuf
acturethanactive-matrixdisplays,butastheresolutionofthedisplayincreases,therespo
nsetimeandthecontrastofthepassivedisplaybecomeworse.Active-matrixLCDsemployanind
ividualtransistorforeachscreenpixel.Active-matrixdisplaysusemorepowerbecauseeach
transistorrequiresasteadyelectricchargetomaintainorupdateitsstate.However,becaus
eeachpixeliscontrolledseparately,active-matrixdisplaysaretypicallybrighter,sharp
er,andhaveafasterresponsetimethanpassivescreens.BothactiveandpassiveLCDtechnolog
iescanbeusedincolorormono-chromedisplays.Monochromedisplaysuselesspower,havehigh
ercontrast,andcostlesstomanufacturethancolordisplays(Wright&Samei,2004).Obvi-ous
ly,colorLCDsprovidearicherandmorecompellinggraphicaluserinterface(GUI)andaretypi
callymoredesiredbyusers.WhileLCDscreensaretheprimarydisplaytechnologyinusetodayi
nsmall-screendevices,othertechnologiesareincommonusedependingonthedevicesapplicat
ion.Thesimplestdisplaytechnologyisthevacuumfluorescentdisplay(VFD).VFDsareusedin
microwaveovens,hometheatercomponents,andotherconsumerelectronicdevicesthatdonotr

equirecomplexuserinterfaces.VFDsemitverybrightlight,areveryhighcontrast,andarein
expensivetomanufacture.ThedownsideisthatasingleVFDscreenelementcanonlydisplayasi
nglecoloratatimeandistooslowformorecomplexcomputer-likeinterfaces.VFDdisplaysare
frequentlydesignedinasegmentedfashionwithrelativelylargesegments(comparedtoLCDor
CRTpixels),thuslimitingtherangeofshapesthatcanbecreated.Figure10.4showsaclose-up
ofaVCRssegmentedVFDdisplay.
FIGURE10.4Close-upofavacuumfluorescentdisplay.VFDshavebeenadaptedtodisplaydynami
cdatabybreakingdownthesegmentsintoindividualpixels.Source:FromWikipedia.org,2007
.
10Small-ScreenInterfaces312

Anotherstraightforwardtechnologythathasblossomedintoaviabledisplaymodalityisthel
ight-emittingdiode(LED).FromthesingleLEDlightthattellsyouthatyoursmokealarmisrec
eivingpowertofiber-opticlightpipesthatallowforhigh-resolutionLEDdisplays,theLEDh
asevolvedintoanextremelyusefuldisplaytechnology.LEDsuseaverysmallamountofpower,a
ndhavealonglifespanandquickresponsetimes.Figure10.5showsacar-parkingassistanttha
ttellstheuserhowfarawayheorsheisfrombackingintoobjectsbydisplayingthedistanceona
nLEDdisplay.OneofthemostexcitingdevelopmentsindisplaytechnologyistheorganicLED(O
LED)display.Organicdyesaredepositedonalayerofglassorplasticthatemitslightwhenexc
itedbyelectrons.OLEDdisplayshavegreatpotentialasadisplaytechnologybecausetheyreq
uirelittlepower,produceabrightpicture,andcanbemadeverythin.Themajordisadvantageo
fcurrentimple-mentationsofOLEDdisplaysistheshortlifespanoftheorganicmaterialsuse
d(currentimplementationsarelimitedtoamaximumof10,000hours).Ifthislimitationcanbe
overcome,OLEDdisplaysmayquicklyovertaketheLCDastheprimarydisplaytechnology.Comme
rcialimplementationsofOLEDtechnologyarealreadybecomingavailable;Figure10.6showsa
SamsungOLEDwatch.AnadditionalbenefitofOLEDdisplaysisthattheentirescreenmaybemade
flexible.Electronicpaper(alsocallede-paper)isanOLEDdisplaytechnologydesignedtolo
okandactlikeprintedpaperbutwiththeabilitytodisplaydynami-callychangingdata.Theke
ydifferencebetweene-papertechnologyandotherdisplaysisthatnopowerisneededtokeeppi
xelsonthescreen(atechniqueknownasimagestability);powerisonlyneededtochangethepic
tureonthedis-play.Theimagestabilitypropertiesofe-papermeanthatthesedisplaysrequi
reverylittlepowertooperateoverlongperiodsoftime.
FIGURE10.5Car-parkingassistantwithanLEDdisplay.Thenumberonthedisplayisthedistanc
etheuserisfromanyobjectsbehindhim.Source:FromAlibaba.com,2007.
10.2TechnologyoftheInterface313

10.2.2ScreenSize,Resolution,andDotsperInchScreensaredefinedbyboththeirphysicalsi
zeandtheirresolution(thenumberofpix-elstheycandisplay).Physicalscreensizeismeasu
redininchesbydiagonallength.Typicaldisplaysusearatioofhorizontaltoverticalsizeof
4:3.Newer,so-calledwide-screendisplays,usea16:9ratio,andotherlesscommonratios(e.g.
,3:2,5:4,or16:10)areemployeddependingontheapplicationordevice(Figure10.7).Theres
olutionofthedisplayalsoplaysanimportantroleintheamountofinformationthatcanbepres
entedonscreen.Dotsperinch(DPI)(alsocalledpixelsperinch)isameasureofthenumberofpi
xelsineachsquareinchofthedisplay.ThehighertheDPIvalue,thedenserthepixelsareinthe
displayandthusthebetterthevisualqualityofthedisplay.AtypicalDPIvalueforacomputer
screenis72,butnewerscreensarealreadyexceeding200indensity.Anothermorecommonmeasu
rementofresolutionisthenumberofhorizon-talandverticalpixelsthescreeniscapableofd
isplaying.Small-screendevicereso-lutionscanrangefromaslowas1616pixelsforwatchdis
playsandother

FIGURE10.6CommercialimplementationofOLED.TheSamsungGPRSClass10WatchPhoneusesanOL
EDdisplay.Source:FromEngadget.com,2004;courtesySamsung.
10Small-ScreenInterfaces314

similarverysmallscreens,to800600pixels(SVGA)foundinsomehigh-enddevices(liketheMo
tionLS800TabletPC).Newerdevicesarecontinuallypushingthislimitforhigher-resolutio
ndisplays.VariousresolutionstandardsareshowninTable10.1.10.2.3InputTechnologiesI
tisdifficulttodesignforsmall-screendeviceswithoutconsideringhowinforma-tionisinp
uttedtothedevice.Thedesignofthedisplayinterfaceisheavily
TABLE10.1StandardSmall-ScreenResolutionsNameAspectRatioResolutionQQVGA4:3160120Q
CIF1.22:1176144QCIF4:5176220QVGA4:3320240CGA16:5640200EGA64:35640350VGA4:3640480S
VGA4:3800600XGA4:31024768
FIGURE10.7Fourcommonaspectratios.Typicaldisplaysusearatioof4:3.Source:FromCambri
dgeincolour.com,2007.
10.2TechnologyoftheInterface315

dependentonthedataentrymethod.Infact,humandatainputforsmall-screendevicesisargua
blythebiggestchallengefacingdevicedesigners.Today,theQWERTYkeyboardandmouseareth
eubiquitousinputdevicesforthePC;how-ever,thesmallsizeoftodaysmobiledevicesrequire
sdifferentinputtechniques.Formobilephonesthestandardtelephonekeypad(knownasthedu
al-tonemultifrequency,orDTMF,keypad)isastandardinputdevice.Forenteringnum-bers,t
heDTMFkeypadisfastandeffective,butforalphabetictextualinput,itisslowandinefficie
nt.Themostcommonmethodoftextualinputiscalledmultitapentry.Itrequirestheusertopre
ssthenumberkeythatrepresentsaparticularlet-teranumberoftimesinordertoproducethed
esiredcharacter.Forexample,pressingthe2keythreetimeswouldproducetheletterC.Clear
lythismethodrequiresseveraltimesmoreeffortthanatypicalcomputerkeyboard.Figure10.
8showstheedgyDTMFkeypadonaMotorolaRAZRphone.Severalmethodsofpredictivetextinputh
avebeendevelopedtoovercomethelimitationsoftheDTMFkeyboard,includingtheT9,SureTyp
e,andiTap(tonameonlyafew).Generallythesetechnologiesrelyondictionarycompletion,m
eaningthatwordsareguessedfromadictionarybasedonthekeyspressed.IntheT9systemtheus
erpressesthenumberkeyrepresentingthelettersintheword(onceforeachletter).Thesoftw
arethenguessesthemostappropriatewordbasedondictionarymatchesofthelettersentered.
Ifthewordisnotcorrect,theuserisgiventheoptionofcyclingthroughotherwordsthatmatch
thekeys
FIGURE10.8DTMFkeypadfortheMotorolaRAZR.Thismodelhasafive-wayrocker,twosoftkeys,a
ndfourhardkeys.Source:FromWikimedia.org,2007;courtesyMotorola.
10Small-ScreenInterfaces316

pressed.Whilepredictivetechniquesarefasterthanmultitapinput,theyarestillslowinco
mparisonwithastandardcomputerQWERTYkeyboard.Arecentcon-testbetweenachampionpredi
ctive-textphoneuseranda93-year-oldtelegraphoperatorfoundthatMorsecodewasafastert
extentrymethodthanpredictivetext(Engadget.com,2005).EventhesmallQWERTYkeyboardse
mployedinsomephonestendtomakethephonevolumetoolargetobedesirableforsomeconsumers
.Theneedforsmallerformfactorshasledtotheexplorationofdifferentinputtechnologies.
TechniquessuchasthePalmshandwritingrecognitionsoftwareandvirtualsoftkeyboards(whi

chdisplayaQWERTYkeyboardonatouch-sensitivescreen)havebeenimple-mented.However,We
sternconsumershavenottakentothesenewinputstylesenmasseduetotheirlimitedinputspee
dandpooraccuracyrates(Bohan,2000).ItisclearthatafullQWERTYkeyboardisaquickandusa
bleinputdevice,socompanieslikeLumiohavedevelopedprojectionkeyboards.Theprojectio
nkeyboard(liketheoneshowninFigure10.9)isadevicethatprojectslightintheshapeofareg
ularkeyboardontoaflatsurface.Theuserinteractswithitbyvirtuallypressingthekeysont
hesurface.Basedonwhichlightbeamsarebroken,thedevicecantellwhichkeysarebeingpresse
d.Sofar,fewcommercialdevicesusetheprojectionkeyboard,butasthetechnologymaturesitw
illnodoubtbeintegratedintomoredevices.Clearly,textualinputisnottheonlywayinwhich
usersinputdatatotheirdevices.Formanyyears,hardkeyshavebeenusedindevicesfrommicro
waves
FIGURE10.9Simulationofaprojectionkeyboard.ThisoneisinuseonaWindowsMobiledevice.S
ource:FromRoeder-Johnson.com,2007;courtesyCayerta,Inc.
10.2TechnologyoftheInterface317

andwatchestophonesandmediadevices.Ahardkeyisadedicatedbuttonthatwhenpressedactiv
atesoneormorefunctionswithinthedevice.Asimilartech-nologyisthesoftkey,whichisade
dicatedbuttonwithasoftware-determinedlabel.Thesoftkeyperformscontext-sensitiveac
tionsbasedonthestateofthesoft-ware,andisverycommoninmobilephones.Apopulartrendin
small-screendevicesistheuseoftouch-sensitivescreens.Recentlythetouchscreenhasbec
omeamoreviableinputoptionbecauseofboththereductioninproductioncostandtheimproved
sensitivityofthescreensthemselves.Thereisagoodreasonwhykeyboardshavenotbeenrepla
cedbytouchscreensinlargerdevices;tactilefeedbackisanimportantpartoftheinputexper
ience.Thislimitationoffeedbackhashamperedthedevelopmentoftouchscreensuntilrecent
ly,whenimprovedsensitivityandresponsiveinterfaceshavemadethetouchscreenareasonab
leinputtechnology.OthercommoninputdevicesincludetheclickwheelusedinthepopulariPo
d,thefour-andfive-wayrockerusedinalargenumberofphones,thetrackwheelusedinBlackBe
rries,andthejoystickusedinphonesandmediadevices.Asyetnoidealall-aroundinputdevic
ehasbeendiscovered.Whendeterminingadevicesinputtechnology,thehardwareshouldbedesi
gnedincooperationwiththesoftwaresothattheprimaryusecasesofthedevicearemadeeasyan
defficient.10.3CURRENTIMPLEMENTATIONSOFTHEINTERFACESmall-screeninterfaceshavebec
omeapervasivepartoftodayssociety.Fromthesimpledigitalwatchtoin-cardisplays,almost
everyoneinteractswithsmall-screendevicesonaregularbasis.Despitetheprevalenceofth
esedevices,theircomplexityandusabilityvarywidelyacrossthespectrumofimplementatio
ns.Diversityofhardwareaside,severaltrendscanbeseenthroughouttheindustry:acontinu
edreductioninthesizeofdevices,increasingcomputingpowerwithinthedevice,growingsof
twareanduserinterfacecomplexityandtheenhancedconvergenceoffeatures.10.3.1MobileP
honesInmanyways,mobilephonesareresponsiblefortodaysexplosiveproliferationofsmallscreen,handheldcommunicationdevices.In2005,worldwidemobilephonesaleswereestimate
datmorethan825millionunitssold(Gohring,2006).Thismassivemarkethasdrivenmanufactu
ringcostsofsmall-screendevicesdowntoalevelwheretheyareaccessibletonewusergroups.
Thereductioninthecostofpartshasalsomadeothersmalldevicesmorecost-effectivetomanu
facture.
10Small-ScreenInterfaces318

Typicallymobilephonesaresmaller,lessfeaturefilled,andlesscostlythantheirlargerPD
Acousins.Mobilephonesinthepastusuallyperformedalimitednumberofkeyfunctionalities
suchasmakingphonecalls,writingshortmessageservice(SMS)messages,andmanagingcontac
ts.Screensonmobilephonesarefoundinbothmonochromeandcolorbuttendtobelowerresoluti
onandtendtohavesimplerandmorestraightforwardsoftwareinteractiondesignthanPCs.New
erphonesfollowthetrendtowardmorefeatures:cameraintegration,videoplayback,high-re

solutionscreens,andmultipledisplays,tonameafew.10.3.2SmartPhonesandPersonalDigit
alAssistantsNotonlydouserswantsmallerdevices,theyalsowantdevicesthathaveadvanced
technologicalandnetworkingcapabilities(Sarker&Wells,2003).NokiasSeries60baseddevic
esallowuserstoperformanearlyunlimitednumberoftaskspre-viouslyonlyavailableinfull
-blownPCs.AccordingtothepromotionalmaterialfortheNokiaN95,ItsGPS.Itsaphotostudio.Its
amobiledisco.ItstheWorldWideWeb.Itsanythingyouwantittobe(s60.com,2007).Thecombinati
onofimprovedbatterytechnology,miniaturizationofcomponents,andaseeminglyendlessnu
mberofnewhardwareandsoftwarefeaturesaremakingsmall-screendevicesintotrueportable
PCs.Thedownsideofincreasedtechnologicalcomplexity,generally,isareductioninusabil
ity.Thisistrueofmanyofthecutting-edgesmall-screendevices.How-ever,improvedintegr
ationoffeaturesismakingiteasierforuserstounderstandandevenmanagethecomplexitieso
fmultimediadata.TheResearchInMotion(RIM)BlackBerryPearlkeepstheuserincontrolofhi
sorherbusylifebyprovidingsimplebuteffectivefiltersfore-mail,integrationofinstant
messaging,andtheabilitytoeasilysynchronizethedevicewithaPC.Despitethecomplexityo
ftheunderlyingtechnology,usability-focusedPDAsliketheBlackberrycanprovideaneasyto-useintegratedcomputingexperience.10.3.3DigitalAudioPlayersIna2003study,Sarker
andWellsfoundthatreduceddevicesizeplayedamajorroleintheadoptionofmobiledevices.D
evelopmentsinhardwareminiaturizationhaveallowedmanufacturerstocontinuetoreduceth
eformfactorsofsmall-screendeviceswhileincreasingtheircomputingpower.TheiPodiscur
rentlyoneofthemostpopularmobiledevicesonthemarketwithanestimatedinstallbaseof40m
illionunits(AppleInsider.com,2006).TheiPodNanousesatouch-sensitiveclickwheel,hol
dsupto8gigabytesofmusic(oraround2,000songs),providesavibrantcolorLCDdisplay,ando
ffersmanydifferentwaystoenjoymusic.TheNanoisbarelybiggerthanacreditcardat3.5in1.
6in0.26in.TheiPodssuccessmakesitclearthatusershaveastrongdesireforsmalldevices,an
dthenumberofsimilardeviceshasskyrocketed.
10.3CurrentImplementationsoftheInterface319

10.3.4MobileGamingDevicesMobilegamingdevicesareanothersmall-screeninformationapp
liancethathasfoundwideacceptanceinthemarketplace.Whilemobilegamingdeviceshavebee
naroundsincethe1980s,newerdevicesplaymultiplegames,havebiggerscreens,andarepacke
dwithadvancedfeatures.TheNintendoDSLitewithitstwo3-inLCDscreens,touch-screeninpu
t,four-wayrocker,andWiFiconnectivityisaperfectexampleofapowerfulcomputingdeviced
esignedtobeusedbychil-drenandadults.DSgamesseamlesslyintegratethetouch-screenint
erfaceintoplay,encouraginganimmersiveinteractionexperience.10.3.5ScientificandMe
dicalEquipmentScientificequipmentalsoemployssmall-screentechnologytoconveycomple
xinformation.Progressinscreentechnologyhasvastlyimprovedthereliabilityandaccurac
yofvisualdisplayreliantscientificequipment.Smallscreensarefoundinamyriadofscienti
ficdevices,includingoximeters,signalanalyzers,infu-sionpumps,andoscilloscopes.Fo
rexample,moderndigitalstorageoscilloscopes(DSOs)employsmallcolorCRTsorLCDstodisp
laydetailedtestdata.TheTektronixTDS3052BshowninFigure10.10hasanactive-matrixVGA(
640480pixelres-olution)LCDscreen,numeroushardkeysandsoftkeys,andanEthernetnetwor
kinterface,andcanevenconnecttolargermonitors.Anotherareaofprolificsmall-screenus
ageisconsumermedicaldevices.Modernthermometers,insulinpumps,glucosemeters,andmed
ical-alertdevicesallmakeuseofsmall-screendevicetechnology.Forexample,glucosemete
rsfordiabeticshaveadvancedsignificantlyinthepast10yearsintodigitaldevicesthat
FIGURE10.10TektronixTDS3052BLCD-baseddigitalstorageoscilloscope.Thisdevicehasana
ctive-matrixVGALCDscreenandmanyothersophisticatedfeatures.(CourtesyTektronix,Inc
.)
10Small-ScreenInterfaces320

candisplayglucoselevelswithinsecondsoftesting.TheOneTouchUltrameternotonlydispla

ysthedetectedglucoselevelsonahigh-contrastmonochromeLCDscreen,butalsoallowstheus
ertokeeptrackofexercise,health,andmedicationbydownloadingthedatafromthedevicetoa
PC(Onetouch.ca,2007).10.3.6InformationAppliancesDeviceslikewatches,microwaves,al
armclocks,cameras,stereoequipment,digi-talthermostats,andevenexercisemachinesall
makeuseofsmallscreens.DonaldNorman,aleadingexpertincognitivepsychologyandinterfa
cedesign,callsthesedevicesinformationappliances,devicesthatperformalimitednumberof
func-tionsbutcarryoutthosefunctionsinasimpleandusableway(Norman,1999).TheLogitec
hHarmonyseriesofremotecontrolsemploysasmallscreenaccom-paniedbyaseriesofsoftkeys
andhardkeystoperformtask-basedcontrolovereventhemostcomplicatedhometheatersetups
.ThedesignoftheHarmonyRemotemirrorshowusersactuallyinteractwiththeirhometheater,
allowingtheusertorequestsimpletaskssuchasWatchamovieorWatchTV.Hiddenfromtheuser,theH
armonyintelligentlyperformsanumberofdifferentactionssuchaschangingTVinputsandrec
eiverinputsandturningdevicesonoroff(Logitech.com,2007).10.3.7MediaCenterInterfac
esSmall-screendevicessharemanydesignsimilaritieswithoneparticularlarge-screenint
erfaceparadigm:themediacenter.Mediacentertechnologyhasevolvedoutoftheusersneedtoc
ontrolthevarioustypesofhometheatermedia.Pro-ductsliketheTivoSeries3,theMotorola6
200series,andMicrosoftsWindowsXPMediaCenterEditionallowtheusertowatchtelevision,w
atchrecordedshows,watchmovies,andlistentomusicthroughasingleintegratedinterface.
Eventhoughtelevisionsarefarfromsmallscreens,thedesignguidelinesusedforthesedevices
arenearlyidenticaltothoseusedforsmall-screendevicesbecausetheusersitsagreatdista
ncefromthedisplayandtheinterfacemustbedesignedtoaccommodatethelowerresolutionoft
hemajorityoftelevisionsintodayshomes.10.3.8HybridDevicesTheincreasingdemandbyuser
sforsimplicityhasdrivenmanufacturersofsmall-screendevicestoattemptcombiningvario
ussmall-screentechnologiesintohybrids.Typicallythesedevicesarenotasrobustorsimpl
easdedicatedsmall-screendevices.However,duetotheirconvenience,thisisarapidly
10.3CurrentImplementationsoftheInterface321

growingmarket.TheAppleiPhoneisagoodexampleofahybriddevice.Itcon-tainsthefunction
alityofaphone,aniPod,andaPDAinasinglesmallpackage.TheiPhoneemploysalargemultitou
chscreeninterface,WiFiandcellularconnectivity,andabuilt-inharddrive.TheiPhoneeve
nhasagyroscopethatrotatesthescreenbasedonwhichdirectionthephoneisbeingheld.Addit
ionally,likeasmallPC,externalprogramscanbeaddedtotheiPhonetoexpanditsfunctionali
ty.10.3.9ImplicationsoftheInterfaceforAccessibilityWhilesmallscreenstendtohavepo
oraccessibilitythemselves(duetothefactthatdesignerslooktomakethingssmallinordert
odisplayasmuchinformationaspossibleonthescreen),theycanbeusedtoaidaccessibilityt
ootherpro-ducts.TwoexamplesarethesmallscreensemployedinTeletypewriters(TTYs)(Fig
ure10.11)andtheuseofthebottomofthetelevisionscreenforclosedcap-tioning.Theuseofs
mallscreensintheseexamplesaidsthosewhohavedifficultyhearingthetelephoneortelevis
ionaudiobyprovidingalternativemodalitiesoffeedback.10.4HUMANFACTORSDESIGNOFTHEIN
TERFACEBeforebeginningadesign,itisimportanttoestablishguidingprinciplesthatwillb
eusedtofocusthedesigninadirectionthatwillultimatelyyieldasuccessfulproduct.There
aremanysourcesforguidingprinciples.Guidingprinciplescan
FIGURE10.11ExampleTTYs.Left:Uniphone1140TTY.Right:Compact/CcellulartelephoneTTY.
Source:FromEnablemart.com,2007.
10Small-ScreenInterfaces322

bestyleguidelikeinnature(i.e.,alltextshouldbeinthe8-pointArialfontbecauseresearch
hasdeterminedthatthissizeisreadableatthedistancethedevicewillbeused),ortheycanbe
interactionguidelinelikeinnature(i.e.,menustructuresshouldfavordepth,notbreadth;o
rtheuserinterfaceshouldmakeprimaryfunctionseasyandsecondaryfunctionsshouldberemo
ved).Acompletesetofguidingprinciplesshouldbeacombinationofboth.Thissection,toget

herwithSection10.6,willprovideinformationtohelpthedesignerdeveloptheprinciplesth
atguidesmall-screeninterfacedesign.Thiswillbeaccomplishedbypresentingimportantin
formationeverypractitionershouldknowaboutsmall-screendesign.Thissectionwillfocus
largelyonthescholarlyworkintheareaofsmall-screeninterfacedesign,coveringtopicssu
chascharactersize,contrast,andfonttoenablereadabilityonsmallscreens,andonorganiz
ationalstructuresforaccessingtheavailablefunctionality.Itwillalsocoverconsiderat
ionsforreadingandcomprehensiononsmalldisplays,andincludeadiscussionontheadaptati
onofcommonGUIelementsforsmallscreens.Section10.6willfocusonprovidingguidelinesfo
rcreatingsuccessfuluserinterfacedesignsonsmallscreens.Together,thisinformationca
nbeusedtodeveloptheguidingprinciplesthatdrivethedesignofthesmall-screeninterface
.10.4.1InformationGatheringOneofthefirststepsinsmall-screendesignistodetermineth
echaracteristicsofthescreenitself.Todefinethosecharacteristics,itiswisetofirstco
llectthebackgroundinformationneededtomakeaninformeddecision.Thisincludescollecti
ngdetailsaboutthefollowing,whichisadaptedinpartfromMutoandMaddox(inpress).Userch
aracteristics:Informationaboutusers,suchastheirvisualacuity,colordeficiencies,an
danthropometriccharacteristics(particularlyinformationrelevanttoeyeposition)shou
ldbecollected.Thesecharacteristicswillbeasourceofrequirementsusedtodefinescreenc
haracteristics.Apracticalwaytocollectthisinformationistonotehowusersdifferfromth
egeneralpopulationandusepublishedmeasurementsofthegeneralpopulationforinstancesi
nwhichtherearenodifferences.Position(s)ofdisplay:Informationaboutpossiblepositio
nsandlocationsofthedisplay.Position(s)ofeye:Informationabouttheuserseyepositionsr
elativetothedisplayisveryimportant.Knowledgeofthefarthestdistancefromthedisplayt
hatshouldbeaccommodatedinthedesignisveryimportantforcomputing
10.4HumanFactorsDesignoftheInterface323

theappropriatesizeofcharactersandobjectspresentedonthedisplay.Knowledgeofthewide
stangleofviewthatshouldbeaccommodatedinthedesignisalsovaluableforensuringtheprop
erchoiceofdisplaytechnology.Lightingandenvironmentalconditions:Informationaboutl
ightingintheexpectedenvironmentisuseful.Ifarangeofenvironmentsisexpected,identif
ytheextremesandensurethatthedisplaypropertiesaresatisfactoryinthoseextremes.Type
ofinformationtobedisplayed:Thetypeofinformationtobedisplayedshoulddrivethecharac
teristicsofthedisplay,suchasresolution,size,andcolorrange.Forexample,iffinedetai
lneedstobedisplayed,ascreenwithhighresolutionwillberequired.Butifpurelyalphanume
riccharactersneedtobedisplayed,alow-costsegmenteddisplaymaysuffice.10.4.2Informa
tionFormatForsmall-screeninterfaces,humanfactorsdesignersfacethechallengeofdis-p
layingalltheinformationtheywanttopresent,whilemakingsurethatwhattheypresentisnot
toosmalltobeseenbytheuser.Asaresult,theyoftenaskquestionsabouttheminimumsizeofob
jectsdisplayedonthescreen,particularlytext.Asdescribedearlier,theminimumsizeofan
yobjectdisplayedonascreenisdirectlydependentonthevisualangle,contrast,andluminan
ce.Theimpactthesehaveondeterminingthesizeoftextonsmall-screendisplaysisdiscussed
inthefollowingsubsections.Notethatviewingdistanceisakeycomponentinmanyofthesizec
alculationsdemonstratedinthissection.Theviewingdistanceusedshouldbethemaximumdis
tanceutilizedinatypicalinstallation.Specifyingatyp-icalinstallationisimportantbe
causedevicescanbeplacedinlocationsnotantici-patedbydesignersorinlocationsthatmay
notbeoptimal.Nevertheless,theselocationsneedtobeaccountedforwhenspecifyingthemax
imumdistanceinthefollowingcalculations.VisualAngleManyoftherecommendationsformin
imumsizeofanobjectarebasedontheanglethattheobjectsubtendsattheeye,measuredinminu
tesofarc.Theformulaforcomputingthevisualangle,inminutesofarc,isshowninthisequati
on:VisualAngleminarctan
LD6010:1whereListhesizeoftheobjectmeasuredperpendiculartothelineofsight,andDisthed
istancefromtheeyetotheobject(Figure10.12).
10Small-ScreenInterfaces324

Forangleslessthan600minutes,thefollowingsimplerformofthepreviousequationcanbeuse
d(Grether&Baker,1972):VisualAnglemin

57360LD10:2The57.3and60intheformulaareconstantsforangleslessthan600minutes.Solvingfor
,Equation10.2canberearrangedasfollows:L

VisualAngleD57:36010:3L,orcharacterheight,isafunctionofvisualangleanddistance.Thi
illenablethedesignertodeterminetheobjectsizewhenthedistancefromtheeyetotheobject
andthevisualangleareknown.Fortext,however,aconversionofthisobjectsizetofontsizem
easuredinpointsisdesirable.Thefollowingformulamakesthisconversion:F
L0:013910:4whereFisthefontsizeinpointsandListhesizeoftheobjectmeasuredperpen-dicul
artothelineofsightininches.ViewingDistanceViewingdistancescanvarygreatly,butthem
inimumviewingdistanceshouldbegreaterthan12inches.Itcanbeassumedthatmostofficewor
kershaveacor-rected,oruncorrected,visualabilityforcomfortablereadingat12to16inch
es(ANSI/HFS1001988,1988).
Visual angleDL
FIGURE10.12Computingthevisualangleofanobjectsubtendedattheeye.Relationshipbetwee
nvisualangle,sizeoftheobject,anddistancefromtheobjecttotheeye.
10.4HumanFactorsDesignoftheInterface325

CharacterHeightTextisoneofthemostimportantobjecttypesdisplayedonascreen,andthesi
zeofthattextaffectstheidentificationofcharactersandthereadabilityofwords.Themini
mumcharacterheightshouldbe16minutesofarctosupporttherapididentificationofcharact
ers,andpreferably22to30minutesofarcatthemaxi-mumviewingdistance.Forcharactersinw
hichspeedofrecognitionisnotimpor-tant(suchasfootnotes,superscripts,orsubscripts)
,thecharacterheightshouldbeatleast10arcminutes(BSR/HFES100,2006).Characterheight
sof20to22arcminutesarepreferredforreadability,whilethethresholdforreadabilityis1
6to18arcminutes.Largecharacterheights,morethan24arcminutes,mayinhibitreadability
byreducingthenumberofcharactersthatmaybeviewedperfixation(ANSI/HFS100-1988,1988)
.Fordiscriminationofthecolorofanalphanumericstring,20arcminutesareneeded,andford
iscriminationofthecolorofasinglecharacterorsymbol,30arcminutesarerequired(BSR/HF
ES100,2006).Inmanysituations,thedesignerhasdeterminedtheviewingdistanceand,fromt
herecommendedvisualanglesforcharacterheightjustpresented,thedesignercanchooseapr
eferredvisualangle.Withthesetwopiecesofinforma-tion,thedesiredcharacterheightand
theresultantfontsizecanbecalculatedwithEquations10.3and10.4,respectively.Table10
.2showscharacterheightsandfontsizesforasampleofvisualanglesandviewingdistances.C
haracterWidth-to-HeightRatioBasedontheuppercaseletterHwithoutserifs(thesmallline
sextendingfromthemainstrokesofaletter),thewidth-to-heightratioshouldbebetween0.5
:1and1:1.Foroptimallegibilityandreadability,itshouldbebetween0.6:1and0.9:1(BSR/H
FES100,2006).StrokeWidthCharactersaremadeupoflines,calledstrokes.Thestrokewidtho
fcharactersshouldbefrom
16to
112ofthemaximalcharacterheight(BSR/HFES100,2006).SpacingbetweenCharactersThespac
ingbetweencharacterswithoutserifsshouldbeatleastequaltothestrokewidth,andprefera
bly25percentto60percentofthewidthoftheupper-caseletterH.Thespacingbetweencharact
erswithserifsshouldbeatleast1pixel(BSR/HFES100,2006).SpacingbetweenLinesThespace
betweenlinesoftext,includingdiacritics,shouldbeatleast1pixelandpreferablyatleast
15percentofthemaximalcharacterheight.Foruserswithpar-tialvision,uselargerspacing
,or25percentto30percentofthecharacterheight(BSR/HFES100,2006).

10Small-ScreenInterfaces326

SpacingbetweenWordsThespacingbetweenwordsshouldexceedthespacingbetweencharacters
,andpreferablybeatleasthalfthewidthofanuppercaseletterHwithoutserifs(BSR/HFES100
,2006).LuminanceInthecaseoflight-emittingvisualdisplays,luminanceisameasureofthe
inten-sityofthelightemittedfromthedisplay.Itisanobjectivemeasurementofwhatmostpe
oplethinkofasbrightness,andisusuallymeasuredascandelaspersquaremeter(cd/m2).Thedis
playshouldbecapableofproducingaluminanceofatleast35cd/m2,andpreferably,100cd/m2.
Usersoftenpreferahighluminance(>100cd/m2),asreadingspeedandaccuracyincreasewithi
ncreas-ingluminance,andlegibilitydecreaseswhenluminancefallsbelow35cd/m2(BSR/HFE
S100,2006).LuminanceContrastTwoofthemostimportantcomponentsthatinfluencerecognit
ionareobjectsizeandcontrast.Forcontrast,thedisplayshouldexhibitaminimumcontrast
TABLE10.2CharacterHeightandFontSizeasFunctionofViewingDistanceandVisualAngleView
ingDistance(inches)VisualAngle(minutes)CharacterHeight1(inches)ApproximateFontSi
ze2(points)16160.07455180.08386200.09317220.10247240.1117824160.11178180.1257920
0.139610220.153611240.16751236160.167512180.188514200.209415220.230417240.251318
1CalculatedwithEquation10.3.2CalculatedwithEquation10.4.
10.4HumanFactorsDesignoftheInterface327

ratioof3to1underallilluminationconditions(BSR/HFES100,2006).Alumi-nancecontrasto
f7to1ispreferred(ANSI/HFES100-1988,1988).Notethatreflectionsofexternallightsourc
esfromthedisplayreducecontrastandthereforelegibility,soitisimportanttomeasurecon
trastundertheproperenvironmentalconditions.FontThedeterminationofafontshouldtake
intoconsiderationtheavailablepixelstodisplayacharacter.BSR/HFES100(2006)recommen
dsaminimumpixelheightof9pixelsandaminimumpixelwidthof7pixelsforLatinalphanumeric
char-actersusedintasksrequiringcontinuousreadingoridentificationofindividualchar
acters(Figure10.13).Forpurelyuppercaselettersandnumerals,charactersshouldbe7pixe
lshighand5pixelswide,whilesuperscripts,subscripts,andthenumeralsinfractionsshoul
dbeatleast5pixelshighand4pixelswide.BSR/HFES100(2006)alsostatesthattheminimumhei
ghtshouldbeincreasedby2pixelsifdiacriticmarksareused.However,manyfontsaredesigne
dforprintedmaterials,andthelegibilityofthesefontssufferswhenfontsizefallsbelow14
pixelsinheightbecausethesetypefacescannotbedisplayedfullyonapixelgridsmallerthan
thissize(Zwicketal.,2005).Thetypefacewillotherwiseappearjagged.Asaresult,pixelfo
ntshavebeendesignedfordisplayonasmallpixelgrid.Theseareoftenoptimizedforaparticu
larpixelgridandarenotscalable.Table10.3presentsafewexamplesofpixelfontsthatareop
timizedforaparticularpixelgrid.
97
FIGURE10.13Heightandwidthofcharacters.Eachcirclerepresents1pixel.
10Small-ScreenInterfaces328

Pixelheightdiffersfromafontsizeinpoints.Fontsizeinpointsisameasure-mentofthephys
icalheightofacharacter(1point0.0139in).Forexample,a12-pointfontis0.1668intall.Pix
elheightisthenumberofverticalpixelsonadis-playusedtorenderthecharacter.Theactual
heightofapixelfontisdependentonthenumberofpixelsperinchthedisplayiscapableofrend
ering,alsoknownasDPI.Tocalculatetheactualheightofapixelfont,useL

PA10:5whereListheheightofthecharacterininches,Pistheheightofthecharacterinpixels,a
ndAisthenumberofpixelsperinchthedisplayiscapableofrendering(DPI).At72DPI,a12-pix
elfontis0.1667in,approximatelythesameheightofa12-pointfont.IncreasingtheDPIofasc
reenwillresultindecreasingtheheightofapixelfont.Anotherattributeoffontsiswhether
thefontisaserif(hasthesmalllinesextendingfromthemainstrokesofacharacter)orsansse
rif(doesnothavethoselines).Researchontheuseofseriffontsfordisplaysissomewhatcont
radictory.MutoandMaddox(inpress),intheirreviewandsummarizationofthatresearch,rec
ommendthefollowing:FFordisplaysofhighresolution(>150DPI),seriffontsaremorereadab
lefortasksthatrequireextendedreading.FFordisplaysoflowerresolution(<80DPI),sanss
eriffontsareoftenpreferredbyusers.DisplayPlacementForhandhelddevices,displayplac
ementisatthediscretionoftheuser.However,formounteddisplaysinafixedlocation,displ
aysshouldbeplacedintheoptimumvisualzone(Figure10.14).
TABLE10.3ExamplePixelFontsFontExampleUppercasePixelHeightLowercasePixelHeightAld
ebraRegular
97Argon
65Axxell
75Foxley816
86Mini7
5NotapplicableMiniMicra
64Pico12
75Source:MiniFonts.com(2006).
10.4HumanFactorsDesignoftheInterface329

10.4.3AccessingFunctionalityThecomputingpowerofdevicestodayenablesthecreationofh
ighlyfunctionalsoftware.Evendeviceswithsmallscreenshavethecapabilityofperforming
alargenumberoffeaturesandfunctions.Thechallenge,then,becomeshowtheuseraccessesth
esefeatures.Menusarecurrentlythedominantdesignforgivingusersaccesstoalargenumber
offeaturesforbothdesktopcomputersaswellasforhandhelddevices.Forbothapplications,
menusenableuserstorecognizethefunctiontheyneed
15 optimum15 optimum35 maximum35 maximum60 maximum90 maximum85 maximum85 maximum
65 maximum35 maximumHead rotationHead and eye rotationNormal line of sightNormal
line of sight60 maximumEye rotation40 maximum20 maximum0 optimum15 optimum15 opt
imum15 optimum15 optimum15 optimumNormal lineof sight15 optimum
FIGURE10.14Mounteddisplaysinfixedlocations.Verticalandhorizontalvisualfields.Sou
rce:AdaptedfromU.S.DepartmentofDefense(1999).
10Small-ScreenInterfaces330

andselectitfromalist(ratherthanrecallitandtypeitin).Theavoidanceoftextentryisesp
eciallybeneficialondevicesthatdonothavefullkeyboards.However,eventhoughmenusaret
hedominantdesignchoice,manysoftwaredesignspoorlyimplementmenus,particularlyonhan
dhelddevices,leadingsomeresearcherstoinvestigatealternativemethods.Thissectionwi

llpresentresearchonmenudesignandpresentfactorsthatcontributetoausablemenudesign,
aswellasintroduceresearchonalternativestomenus.MenuOrganizationTheprimarygoalofm
enudesignistoorganizethemenuitemssuchthatusersareabletofindtheitemstheyneedquick
lyandwithfewerrorsandfewkey-strokes.LeeandRaymond(1993)referenceabodyofresearchs
howingthatmenuselectionerrorsaremostoftencausedbymiscategorization,overlappingca
te-gories,and/orvaguecategorytitles.Focusingonavoidingthesepitfallswillsignif-ic
antlyimprovetheoverallmenudesign.Therearetwolevelsofmenuorganization:overallmenu
organization(oftentakestheformofhierarchical,branchingtreestructures)andwithin-m
enuplace-ment(organizesitemswithinamenulist).Studieshaveshownthatthebestwaytoavo
idthecommonpitfallsofmenuorganizationmentionedpreviouslyistoincludeendusersinthe
designprocess.Tohelpavoidoverlappingcategoriesandmiscategorizationoftheoverallme
nustructure,studieshaveshownthatmenuorganizationsgeneratedbyuserdataaresuperiort
othosegeneratedbydesignerintuition(Hayhoe,1990;Roske-Hofstrand&Paap,1986).Captur
ingtheusersperceivedorganizationandorganizingthemenusinthiswaycanimprovetheoveral
lusabilityofthemenu.Anin-depthdiscussionoftechniquesforcollectinguserdataisbeyon
dthescopeofthischapter,butcanbefoundinPaapandCooke(1997),Cooke(1994),andRuggandM
cGeorge(1997),andadiscussionofscalingtechniquesforreduc-inguserdataintomeaningfu
lcategoriescanbefoundinPaapandCooke(1997)andHayhoe(1990).Onegoaltostriveforinthe
hierarchicalstructureistodeveloporthogonalcategories(mutuallyexclusivecategories
thathavenocom-moncharacteristics)thatareclearlywordedsothatitemsundereachcategor
ycannotbeperceivedasbelongingtoanothercategoryandsothateveryitemfitswellintoacat
egory.Toavoiderrorsduetovaguecategorytitles,menutitlesshouldbeclearlywordedandac
curatelyconveytheircontents.AccordingtoresearchreferencedbyLeeandRaymond(1993),c
learlywordedmenuitemscanimproveuserperfor-manceandthebestmenutitlesarederivedfro
muserdatacombinedwithdesignerexpertise.Forwithin-menuplacement,itemscanbearrange
dinseveralways(Marics&Engelbeck,1997;Paap&Cooke,1997):FAlphabeticallyFTemporally
(daysofweek,monthsofyear)
10.4HumanFactorsDesignoftheInterface331

FAccordingtomagnitude(small,medium,large)FConsistently(ifitemsappearinmultiplepl
aces,keeptheorderthesame)FCategoricallyFAccordingtofrequencyofuse(placethemostfr
equentlyaccesseditemsatthetop)Frequencyofuseisoftenthepreferredchoice.Organizing
menusitemsinthiswaycansavetimeanderrors,asitemslocateddeepwithinamenustructurear
eoftendifficulttofind.Inourexperiencefromviewinghundredsofusersnavi-gatingmenus,
userswilloftenselectthefirstcategorythatcouldcontaintheitemforwhichtheyarelookin
g.Thus,orderingitemsaccordingtofrequencyofusewillaidthissearchtechnique,aswellas
provideamoreefficientinterfaceforcontinueduse.Thisguidelinecanbeadaptediftheitem
shaveanaturalordertothem(suchastemporalormagnitude),ortomaintainconsistencywitho
thermenus.Forexample,ifSavecommonlyappearsaboveDeleteinothermenusintheinterface,
theneveninamenuinwhichDeletemaybemorefrequentlychosen,itmaybebesttoagainplaceSav
eaboveDeletefromaconsistencystandpoint.Usealpha-beticalorderingonlyifthosepartic
ularitemsareoftenpresentedinthisway(e.g.,alistofstates).MenuEnhancementsOnepromi
singapproachtoenhancingmenusistoenablekeywordentry.LeeandRaymond(1993)notetwofac
torsthatcausedifficultyforusersofmenusystems:FAdisproportionatenumberoferrorsocc
uratthetopmenulevels.FMenusystemscanbetedious,inefficient,andboringforexpertuser
s.Menukeywordscanassistinbothsituations.Thesesystemsassociateakeywordwitheachmen
u,enablingexpertuserstobypasstheproblematicupperlevelsandreducekeystrokestoimmed
iatelyaccesslowerlevels.LeeandRaymond(1993)presentaseriesofstudiesshowingareduct
ioninsearchtime,failures,andnumberofframesaccessed,withamarkedimprovementinuserp
references.MenuDepthversusMenuBreadthMenuscanbebroad,withmanyoptionspermenucateg
ory,ornarrow,withfewoptionspercategory.Theycanalsobedeep,withmanylevelsofsubcate
gories,orshallow,withfewlevelsofsubcategories(Figure10.15).Foragivensetoffeature
s,thefollowingquestionisoftenraised:Isitbettertohaveabroadandshallowmenuhierarch
y,oranarrowanddeepmenuhierarchy?Thisquestionhasbeenheavilystudiedintheliterature
.Earlystudiesondesk-topcomputermenusshowthatbroadhierarchiesareusuallythemosteff
icientforusers,especiallywhentheoptionsareunambiguous.Broadhierarchiesalsoseem

10Small-ScreenInterfaces332

bestforexpertusers.However,iferrorsarecostlyorthenumberoferrorsishigh,anarrowhie
rarchythatfunnelsusersintoachoicecanbebest.(AsummaryofthesestudiescanbefoundinBe
rnard,2002a,andGevenetal.,2006.)Themajorityofstudiesconductedonthisissue,however
,havefocusedpri-marilyonevaluatingstructuresofrelativelyconstantshapes(thesamenu
mberofitemsperlevel).Fewhaveexaminedtheshapeofthestructure.NormanandChin(1988)an
dBernard(2002b)examineddifferentlyshapedstructures,includ-ingaconstantbreadth(sa
menumberofitemsperlevel),adecreasingstructure(moreitemsinupperlevelsthaninlowerl
evels),anincreasingstructure(moreitemsinlowerlevelsthaninupperlevels),aconcavest
ructure(moreitemsintopandbottomlevels,feweritemsinmiddlelevels),andaconvexstruct
ure(moreitemsinmiddlelevels,feweritemsintopandbottomlevels).Bothstudiesfoundthat
concavestructuresaremorenavigationallyefficientthanrelativelyconstantshapesofthe
samesizeanddepth.Thus,theyassertthatbroadmenusatthemiddlelevelswillincreasemenus
electionerrors.Themajorityoftheabovestudieswereperformedondesktopcomputerswherem
oreinformationcanbepresentedonthescreen.Ifallitemscannotbedisplayedonthescreenat
atime,thedepth-versus-breadthtrade-offshifts.Gevenetal.(2006)re-evaluatedthisiss
uewithmenusonmobilephonesandPDAs.TheyfoundthatwithmobilephonesandPDAs,themosteff
ectivehierarchyhasonlyfourtoeightitemsperlevel,andifalargenumberofitemsareneeded
,itisbettertoincreasethenumberoflevelsthantoincreasethenumberofitemsperlevel.
Narrow structure: 6 levels, 2 items each (2  2  2  2  2  2) 64 items total Str
ucture: 3 levels, 4 items each (4  4  4) 64 items totalStructure: 2 levels, 8 it
ems each (8  8) 64 items totalBroad structure: 1 level, 64 items (1  64)

64 items total
FIGURE10.15Breadthversusdepth.Examplesofbroad,narrow,deep,andshallowmenustructur
es.
10.4HumanFactorsDesignoftheInterface333

Thus,adeeperhierarchyismoreeffectiveforusersofproductswithsmallscreensthanabroad
hierarchy,forexpertusersandnoviceusersalike.IconsIconsareoftenusedinmenudesign,p
articularlyatthetoplevelsofahierarchy.Researchintothevalueoftheicons,however,ism
ixed.MacGregor(1992)foundthatwheniconsshowanexampleofaniteminthatcategory,theyre
duceerrorsby40to50percent.Thegeneralizabilityofthisresult,however,isdependentont
heicondesignanditssuccessatunambiguouslyrepresentingacategorymem-ber.Iconsareals
ousefulinmenusasameansforachievingincidentallearningforlateruseintoolbarsorother
isolatedconditions(Dix,1995).Baeckerandcolleagues(1991)assertedthatthemeaningofa
niconshouldbeobvioustoexperiencedusersandself-evidenttonoviceusers.Theydeclareth
atmanyiconsdonotmeettheformercriterion,andmostdonotmeetthelatter.Asaresult,theye
valuatedtheimpactofanimatediconsinreducingthisproblem,andfoundthatanimatediconsw
ereusefulandhelpful.Ineverycaseintheirstudywhereastaticiconwasnotunderstood,thea
nimatediconwassuccessful.Again,thegeneralizabilityofthisresultisdependentonthean
imatedicondesign.Theoverallvalueoficons,however,isstilldebated.Somebelievetheirg
reat-estvalueisinvisualentertainment,addingsomeinteresttoanotherwisedullpieceofe
quipment,andthatintermsofusability,theyareatbestirrelevantandatworstdistracting(
Jones&Marsden,2006).MenuAlternativesMenusareadefinitiveimprovementovercommandlin
einterfacesorfunctionkeybasedinterfaces(e.g.,pressingtheFCNkeysimultaneouslywitha
notherkeytoseethephonebook,acommonearlyinteractionwithcellphones).How-ever,canth

eybeimproved?JonesandMarsden(2006)proposedanalternativemethodformobilephonesastru
ctureinspiredbyBTreeinterfacesusedindatabases.Thisstructureinvolvesenablinguserso
fmobilephonestobeginenter-ingthenameofthefunctiontheywishtoaccessandthentoselect
thatfunctionfromthebest-matchlist.Forexample,ifuserswantedtoaccessCallDivert,the
ywouldstartspellingthefunctionnameusingthekeypad(inthiscase,pressingthe2keyforaC
andpressingitagainforanA).Thesystemwouldthendisplayascrollablelistofallpossiblec
hoices,suchasCallDivert,CallIdentification,CallBarring,Banner,Backlight,andsoon.
Theuserscouldcontinuespellingtoreducethenumberofmatches,orselectthedesiredoption
fromthelistatanytime.Userscanalsomovethroughtheentirelistusingthescrollkeyswitho
utattempt-ingtospellanythingfirst.Thus,thisdesignsupportsthenoviceusersdesiretoex
ploreeveryfunction,aswellastheexpertusersneedtorapidlychooseaknownfunction.Usingt
heNokia5110mobilephoneasatestcase,theyfoundthatusersintheirstudyrequiredanaverag
eof9.54keystrokestocompletetheirtasksinthe
10Small-ScreenInterfaces334

BTree-inspiredstructure,comparedto16.52keystrokesontheNokia5110,andthatthemeantas
ktimewasreducedfrom42.02secondsto33.42seconds.MenuDesignGuidelinesBasedontherese
archdiscussedaboveandotherknownprinciplesformenudesign,thefollowingcompriseafewb
est-practiceguidelinesformenudesign:1.Organizemenuitemsintoorthogonalcategoriest
hatfollowthetargetusersperceivedorganization.Unfortunately,beingrealistic,attimes
thiscanbeimpossibletofullyachieve.Often,throughcarefulconsideration,repeatedatte
mpts,andwiththehelpofuserstudies,nearorthogonalcategoriescanbecreated.Itisatthis
stagethattheuserstudieswillfurtheraidmenudesignbecausemenuitemsthatcouldappearun
dermultiplecategoriescanbeassignedtothecategorytowhichmostoftheusersinthestudype
rceivedittobelong.Forexample,shouldRingVolumeforamobilephoneappearunderRingtones
orSoundSettings?Validargumentscouldbemadeeitherway,butifthemajorityofusersinastu
dyplaceditunderSoundSettings,thenthatiswhereisshouldgo.Ofcourse,thisproblemonlyo
ccurswhencategoriesarenotorthogonal,sowhenpossible,identifyingorthogonalcategori
esisalwaysthebestsolutionforissueslikethis.2.Createmenutitlesthatareclearlyworde
dandaccuratelyconveytheircontents.Utilizeuserdataincreatingthemenutitles.3.Usesi
mpleactionverbstodescribemenuoptions.4.Forsubmenus,titlethetopofeachmenu(perhaps
intheheader)withthenameofthemenuitempreviouslyselected.Thiswillaidusersinknowing
theirlocationwithinthemenuhierarchyaswellasconfirmtheirselection.5.Whenusingloop
edmenus,ensurethattheusersknowwhenthemenuhasloopedandreturnedtothefirstitem.Onew
aytoaccomplishthisistoplacethefirstmenuitemonlyatthetopofthescreen,evenifitisscr
olledtobypressingthedownarrowkey.Thus,thefirstmenuitemwillneverappearatanypositi
ononthescreenbutthetopposition.Thiswillresultinajumpofthehighlightfromthebottomo
fthescreentothetopwhentheuserpressesthedownbuttonfromthelastmenuitemtothefirstme
nuitem.Thisjumpwillcausetheusertotakenotethatsomethingdifferenthashappenedandsig
nifythattheuserisnowlookingatthefirstiteminthelist.6.Makeitveryeasyforuserstonav
igateupwardinthemenuhierarchy.Doingsowillenableuserstorecoverfromamistakeiftheyr
ealizetheitemtheyarelookingforisnotinthecategorytheyselected.7.Provideawayforthe
usertoexitthemenuwithoutmakingaselection.Doingsowillalsoenabletheuserstorecoverf
romamistakeiftheyrealizetheitemtheyarelookingforisnotinthemenuatall.
10.4HumanFactorsDesignoftheInterface335

10.4.4ReadingandComprehensiononSmallScreensResearchintoreadingandcomprehensionon
smallscreenshasfoundthatreadingwillbeonlyslightlysloweronsmallscreensandthatthee
xperienceisnotdramaticallydifferentfromreadingonalargescreen.DuchnickyandKolers(
1983)studiedthereadabilityandcomprehensionoftextwhenvaryingscreensize.Whenvaryin
gscreenheight(1,2,4,and20lines),theyfoundthatwhenonly1to2linesweredisplayed,perf
ormancewassignificantlypoorerthanwhen4or20linesoftextwerevisible,andthattherewas

nodifferencebetween4and20lines.Althoughthedifferencewasstatisticallysignificant,
themagnitudeoftheperformancedecrementwasnotlarge.Theyfoundthatreadingwasonly9per
-centslowerthanwhendisplayheightwasincreasedfrom1to20lines,andtherewasnodifferen
ceincomprehension.Varyingscreenwidth,however,hadalargerimpact.Thefull-widthand
23-widthdisplaysread25percentfasterthanthe
13-widthdisplay.ElkertonandWilliges(1984)alsostudiedtheimpactofvaryingscreenheig
htandfoundasignificantperformancedecrementwithscreensofonelineoftext.Basedonthes
eresults,ifdisplayswillbeusedheavilyforreading,preserveasmuchofthewidthofthedisp
layforreadingaspossible.Thisalsoimpliesplacingnavigationalcontrols,menubars,ands
oon,intheverticalplane.Evenifthereislittleperformancedegradationinreadingandcomp
rehensiononsmallscreens,experiencehastaughtusthatusersprefernottoreadonsmallscre
ens.MarshallandRuotolo(2002)providedtwoclassroomsofstudentsthesameclassmaterials
ine-bookformatonPocketPCsandotherelectronicandpaperformatsandexaminedhowstudents
usedthePocketPCs.Theyfoundthatthestudentsemployedthemprimarilyforreadingsecondar
ymaterials,excerpts,andshorterreadings.Theyusedthemwhenportabilitywasimportant,s
uchaswaitinginline,intheclassroom,whentraveling,whenwaitingforthebus,andsoon.The
yfoundthatonthePocketPC,usersrarelyreadlinearly;insteadtheytendedtoskimtoparticu
larspotsandthenreadshortpassages.Finally,thestudyfoundthatsearchwasakeyfunctionf
orfindingfamiliarmaterialsandfornavigat-ingwithinthedocument.Thisstudygivessomei
nsightintohowpeoplewillusesmallscreenstoreadwhensuchmaterialsareincompetitionwit
hothersourcesofthesamematerial.Thisconfirmsthebeliefthatthetruestrengthofsmall-s
creenreadingisincasual,opportunisticreading.10.4.5UseofCommonGUIElementsUserinte
rfacedesignsthatbuildonpeoplespreviousexperiencesareoftenquickertolearnthroughtra
nsferoftrainingandincreasedinitialfamiliarity.Itisoftenwisewhendesigningauserint
erfacetoexaminethebackgroundknowledgeandexperiencesofthetargetusergroupanddeterm
inewhether
10Small-ScreenInterfaces336

thereareopportunitiestobenefitfromthistransferoftraining.Inmanymar-ketsaroundthe
world,thetargetusergroupofsmall-screenGUIsoftenhasexperiencewithdesktopcomputerG
UIs.Thus,thecommondesktopGUIele-mentscanoftenbeutilizedinsmall-screendesigntoena
blequickerlearning.However,someofthecommonelementsintheircurrentimplementationfo
rdesktopGUIsrequireadaptationtothesmallscreen.Thissectionwillpresentsomeoftheval
uableadaptationsofcommonGUIelementsthatcanbeusedinsmall-screenGUIs.WindowsInthei
rimplementationondesktopGUIs,windowsareresizableandmovable.Onsmallscreens,thescr
eensizeisgenerallylimitedandtheabilitytomoveandresizeislessvaluable.Inaddition,t
heseactivitiesoftenrequiretwohands,whichisalsonotidealforhandheldsmall-screendev
ices.Thus,generallyspeaking,itiswisetoavoidmovableandresizablewindows.TabsTabsar
eanelementoftenfoundindesktopGUIs.Theycanbesimilarlyvaluableinsmallscreens;howev
er,tabswithtwoorthreerowsmayrequiretoomuchscreenrealestateandshouldbeavoided.Pul
l-DownandPop-UpMenusPull-downandpop-upmenusareestablishedwidgetsinsmall-screende
sign.Fortouchscreeninterfaces,pop-upmenusareoftenpreferredoverpull-downmenusbeca
usethehandwillnotcoverthemenu.Itisbesttoavoidscrollingmenusifpossiblebecausethey
requireagreatdealofmanualskillfromtheuser,particu-larlyfortouchscreens(Zwicketal
.,2005).DialogBoxesDialogboxesareoftenusedinsmallscreensandareeasilytransferredf
romdesk-topGUIs.Theycanbeusedforpersistentsystemnotes(Youhaveanewtextmessage),forq
ueries(PleaseenteryourPINorDeletethismessage?),forlistselection(callJanesmobilenumber
,worknumber,orhomenumber),andformanyotherpurposes.Mobilephonesintroduceanewtypeo
fdialogbox,typicallycalledatransient.Transientsarepop-updialogboxesthatareautoma
ticallydis-missedafteraspecifiedperiodoftime.Theyareusedtoconveysystemstatuschan
gessuchasFilesavedorMessagesdeleted.Transientsarevaluableaddi-tionstothemobilephoneu
serinterfacebecausetheyareabletoprovideimpor-tantbutnoncriticalinformationtotheu
serwithoutrequiringspecificuseractiontodismiss.However,theyshouldonlybeusedwheni
tisnotessentialthattheuserseestheinformationtheyconveybecauseitispossiblethatthe

userwillmissthetransient.
10.4HumanFactorsDesignoftheInterface337

ListsListsareaprimaryinteractionmethodforsmall-screenGUIs.Listsareoftenusedforme
nus,singleselectionofanitemfromalist,multipleselectionofitems,andinforms.ScrollB
arsScrollbarscanbeeffectivelyusedinsmall-screendesigntoindicatetotheuserthatscro
llingispossible.However,thereisatendencytoshrinktheirsizetoapointwheretheyaredif
ficulttonotice.Asaresult,secondaryindicatorsoftheabilitytoscrollcanbehelpful(suc
haspresentingapartiallineoftextatthebottomofthescreen).Combiningverticalscrollin
gwithhorizontalscrolling,however,isdif-ficultforusers,particularlyonsmallscreens
,andshouldbeavoidedifpossible.Ifavoidingisnotpossible,Zwicketal.(2005)recommendu
singanavigationborderthatrunsallthewayaroundascreentoenablethecontenttobemovedfr
eelyinanydirection.Scrollingonsmallscreenswillbediscussedinmoredetaillaterinthis
chapter.ZoomZoomisatechniquethatholdspromiseforsmallscreensinsomeapplicationspro
videdthatthehardwarecanperformthisfunctionsmoothly.Itinvolvespre-sentinganovervi
ewofalargespaceandenablingtheusertozoomintoanareaofinterest.Zoomingisabletocreatea
logicallinkbetweenanoverviewanddetailedcontentandisthereforebeneficialinpresenti
nglargeamountsofinfor-mationinlimitedspace(Zwicketal.,2005).DynamicOrganizationo
fSpaceOneheavilyresearchedareainvolvestechniquesfordisplayingwebsitesonsmallscre
ens,mostnotablyPDAs.Nodominantdesignhasemergedtodate,asmanyofthesetechniquesares
tillunderinvestigation.Aquicklookintoafewofthesetechniques,though,canprovideuniq
ueandinventiveideasforpresentinglargeamountsofinformationonsmallscreensandprovid
eameansforuserstointeractwithit.Someofthemostresearchedtechniquesprovideuserswit
hanoverviewvisu-alizationofthelargerwebpagewhileallowingthemtorapidlyzoominontil
esofrelevantcontent.Theoverviewvisualizationallowsuserstogetanunderstandingofthe
ircontextwithinthewebpageandzoomingallowstheusertoviewsectionsofthepageatareadab
lesize.Differenttechniquesutilizedifferentapproachesforzooming.Thecollapse-to-zo
omtechnique(Baudishetal.,2004)allowsuserstoselectareasofthewebpagethatareirrelev
ant,suchasbanneradvertisementsormenus,andremovethem,therebyenablingtherelevantco
ntenttoutilizemorescreenspace.Thecollapse-to-zoomtechniqueusesgesturesforcollaps
ingandexpandingcontent.
10Small-ScreenInterfaces338

Anothertechnique,summarythumbnails(Lam&Baudish,2005),isessen-tiallythumbnailview
sthatareenhancedwithreadabletextfragments.Theread-abletextwithinthethumbnailisin
tendedtoenableuserstomorequicklyidentifytheareaofinterestwithouthavingtozoominan
doutofthepage.Azoomingcapabilityisstillprovided,allowingusersaccesstothefull,rea
dabletext.OthertechniquesincludeWebThumb(Wobbrocketal.,2002)andfisheye,zoom,orpa
n-ningtechniques(Gutwin&Fedak,2004).10.5TECHNIQUESFORTESTINGTHEINTERFACEUsabilit
ytestingofsmall-screendevicesisanimportantstepintheuser-centereddesignprocess.Th
emethodsandprocessesfortestingsmall-screeninterfacedesignsaresimilartothoseforot
herproducts.Testpreparation,protocoldevelop-ment,participantselectionandrecruiti
ng,conductingofthetestitself,andanaly-sisofresultsarethesameforusabilitytestingo
fsmallscreensastheyareforusabilitytestingofanyotheruserinterface.Sincetheseitems
arecoveredindepthinanumberofsources,afulldiscussionhereisbeyondthescopeofthischa
pter(seeinsteadDumas&Redish,1994,andRubin,1994).However,theadaptationsandvariati
onsofequipmentsetupneededtoevaluatesmall-screendevicesarespecialized.Asaresult,t
hecurrentsectionwillfocusonthisaspectoftesting.Oneofthemostchallengingaspectsofu
sabilitytestingofsmall-screendevicesisenablingthemoderatortoseewhatishappeningon
thescreenandvideo-recordit.Therearemanypossiblesolutionstothischallenge,butthree
solutionsthathaveprovensuccessfulinthepastinvolvetheuseofanover-the-shouldercame
ra,adocumentcamera,orasoftwareutilitythatwillreplicatethescreenofadeviceonacompu

ter.Thereareseveralsuchutilitiesforsomemodelsofmobilephones,forexample.Thissecti
onwilldiscusseachofthesetechniquesinsomedetail.10.5.1Over-the-ShoulderCameraWith
theover-the-shouldercameraoption,typicallythecameraismountedonatripodbehindthepa
rticipantsshoulderandraisedtoaheightwhereitcanbefocusedonthescreen.Inalaboratoryw
heresmallscreensareoftentested,thiscameracanbemountedonthewallorceilinginsteadof
onatripod.Withacam-eracapableofhighzoom(e.g.,intherangeof16),aclearandclosepictu
reofthedevicesscreencanbeobtained.Ifthemoderatorwishestositintheroomwiththepartic
ipant,thevideoout-putfromthiscameracanbesplitandruntotherecordingdeviceaswellast
oatelevisionormonitor.Thismonitorcanbeplacedintheparticipantroomtoenablethemoder
atortoseetheparticipantsscreenonthemonitorwithout
10.5TechniquesforTestingtheInterface339

havingtocrowdtheparticipantbytryingtoviewhis/herscreendirectly(seeFigure10.16).T
headditionalchallengeforhandheldsmall-screendevicesthatanover-the-shouldercamera
mustsolveisthatthedevicecanbemovedoutofthefocusofthecameraastheparticipantnatura
llymovestoadjustpostureandsoon.Oneapproachtomitigatethisissueistoplaceamarkeront
hetableinfrontofthepar-ticipant(withinstructionstotrytoholdthedeviceoverthemarke
r)andalsotoorientthemoderatorsmonitorsoboththeparticipantandthemoderatorcanviewit
.Orientingthemonitorinthiswayenablestheparticipanttobetterunder-standtheneedtoho
ldthedeviceoverthemarkerandeasesanythoughtsshemayhaveastowherethatcameraisfocuse
d.Toachieveapicture-in-pictureviewofthescreenandtheparticipantsface,acameracanbel
ocatedacrosstheroomfocusedontheparticipant(Figure10.16).Lightingisanotheritemtoc
onsiderandcontrol.Thescreensonthedevicescanreflectanoverheadlightorotherlightsou
rce.Testthelightingofthepartici-pantslocationbysittingdownatthatlocationwhilewatc
hingthecameraimageonthemonitor.Checktoensurethattheimageisglare-freeandnotwashed
out.Adisadvantageoftheover-the-shouldercameraisthatdespitethemarkerandthemonitor
facingtheparticipant,userswillstillmoveoutofthedesignedareasattimesandneedtobere
mindedtokeepthedeviceinthefocusofthecamera.Theadvantageofthissetupisthattheuseri
sabletoholdthedevicenaturallywithoutanythingattachedtoitorinfrontofit,andaviewof
thefingerspressingthekeyscanalsobecapturedbythecamera.
FIGURE10.16Examplesetupofanover-the-shouldercamera.Thecameraismountedonatripodin
aconferenceroomwithanin-roommoderator.
10Small-ScreenInterfaces340

10.5.2DocumentCameraAdocumentcamera(Figure10.17)canbeusedinsteadoftheover-the-sh
ouldercamera.Therearemultiplepossibilitiesforthiscamera,rangingfromalightweightw
irelesscamerathatcanbeattachedtothedeviceitself,toplacingthedeviceinafixedholder
(e.g.,avise)andutilizingastationarycamera.Infact,itisevenpossibletodesignanddeve
lopacustomversionofthisequipmenttosuitspecificneeds(Schusteritschetal.,2007).Any
oftheseoptions,andothervariationsoftheseoptions,willgiveaverygoodviewofthedevices
screenand,dependingonthecameraangle,thefingersaswell.Theycanalsobeusedintestings
etupssimilartothatdescribedpreviouslywithanin-roommonitorforanin-roommoderatorto
view,andanothercameraforcapturingvideooftheparticipantsexpressions.Theadvantageof
thedocumentcameraisthatthecamerawillmovewiththeparticipant,alwaysstayinginfocuso
nthedeviceandscreen.Thedisadvantageis
FIGURE10.17Documentcamera.ThisisawirelessmobiledevicecamerabyNoldus(2007).(Court
esyNoldusInformationTechnology.)
10.5TechniquesforTestingtheInterface341

thatitdoesnotallowthedeviceundertesttobeheldasitwasdesignedtobeheld(althoughthee
xampleinFigure10.17doesattempttoapproximateit),anditplacesthecamerainthedirectvi
ewoftheparticipant,whichcanbeannoyingorgetinthewayoftheparticipantsview.Afurtherd
isadvantageformobilephonesisthatthecameracangetinthewayofplacingphonecalls,iftha
tisoneofthetasksundertest.10.5.3ScreenReplicationUtilityIfthedevicesupportsit,as
creenreplicationutilitycanprovideadetailedandclearviewofthesmallscreenonalargerc
omputermonitor.Onceonacomputermoni-tor,numerousscreen-recordingutilitiescanbeuse
dtorecordwhatishappeningonthescreenforlaterviewing.However,thisoptionrequiressup
portbythesmall-screendevice.ThereareseveralutilitiesthatcanservethispurposeforWi
ndowsMobileSmartphonesandPocketPCs,aswellasNokiaSeries60devicesandselectSonyEric
ssonphones.Theadvantagetothisoptionisthatitprovidesanunobstructedviewofthescreen
withouthavingtoremindaparticipanttoputitinacertainpositionandwithouthavingtoatta
chacameratothedevice.However,thedisadvantageisthatitisdependentonsupportbythedev
iceundertest,anditcannotshowtheparticipantsfingers.10.5.4UsabilityLaboratoriesMan
yusabilitytestsoccurinausabilitylaboratory.Usabilitylaboratoriesoftenuti-lizetwo
rooms:aparticipantroomandanobserverroom.Aone-waymirroroftenseparatesthetworoomsa
ndenablestheobserverstoseeintotheparticipantroom,butpreventstheparticipantsfroms
eeingintotheobserverroom.Micro-phonescanbemountedinvariouslocationswithinthepart
icipantroomtoenabletheobserverstohearconversations,andcamerascanbemountedinvario
usloca-tionstoenablevideorecordingsoftheevents.10.5.5TestingwithoutaLaboratoryUs
abilitytestingdoesnotrequireanelaboratelaboratory,however.Manytimes,usabilitytes
tscanberunjustaseffectivelyinconferenceroomswithminimalequipment.Thisguidelineis
justastrueforusabilitytestingofsmall-screeninter-faces(seeFigure10.16).Thetechni
quesdescribedabovecanbeusedinanyconfer-enceroom:Thecameracanbeacamcorder(unlesst
hedocumentcameraisused),andthemicrophonecanbedirectlyconnectedtothecamcorder(orc
anbethecam-cordersmicrophoneitself).Infact,muchoftheequipmentdescribedabovehasopt
ionsfortheuseofinexpensiveequipment.Themainpointisthatusabilitytest-ingcanberuno
nanyequipmentbudget.Itisjustimportantthatitisrun.
10Small-ScreenInterfaces342

10.6DESIGNGUIDELINESThefundamentalsofsoftwaredesignremainconstantacrossvariousim
plementa-tionswhethertheyaresmall-screen,web,ortraditionalPCapplications.Thefocu
softhissectionwillbeondesignguidelinesthatareuniqueorimportanttosmall-screendesi
gn.Alldesignprinciplesmustbeadaptedtothestrengthsandlimita-tionsoftheimplementat
ionmedium.Aswithallinterfacedesignprojects,userinterfacedesignforsmalldisplayssh
ouldfollowtheuser-centereddesign(UCD)process.Thismeansinvolvingusersineachstepof
therequirements-gatheringanddesignprocess,plusperformingiterativedesignwithusabi
litytesting.AfulldiscussionoftheUCDprocessisout-sidethescopeofthischapter;seeLew
isandReiman(1993)foradetailedexplora-tionofthesubject.AsdemonstratedinSection10.
4,thesizeofthevisualdisplayofsmall-screendevicesisaprimaryrestrictiontointerface
design.However,otherfactorsmustalsobeconsidered,suchastheimperfectionofcurrentin
puttechnologiesandtheoftenlimitedattentionoftheuserinamobilesetting.Withthoseres
traintsinmind,severaloverarchingdesignprinciplesemerge:Designforsimplicity,desig
nwiththesmallscreeninmind,provideusefulfeedback,maintaininteractiondesignstandar
ds,andrespecttheusersphysicalandmentaleffort.10.6.1DesignforSimplicityAsdiscussed
inSection10.3,small-screendevicestendtowardtwoextremes:thedoes-everythingmobilePCr
eplacementandthedoes-one-thing-wellinforma-tionappliance.Inbothcases,theunderlying
technologyisusuallycomplexandtheinterfacemuststrivetowardtheprincipalgoalofsimpl
icity.Inhisclassicbook,TheDesignofEverydayThings,DonaldNormandefinestheparadoxof
technol-ogyasfollows:Wheneverthenumberoffunctionsandrequiredoperationsexceedsthen
umberofcontrols,thedesignbecomesarbitrary,unnaturalandcomplicated(1998:29).Thehar
dwaredesigntrendtowardmorecomplicatedandfeature-filleddevicesmakestheguidelineof
keepingthingssimpleaverydifficulttaskindeed.RelateVisualPrecedencetoTaskImportan

ceUnderstandingwhattheuserwantsthedevicetodoshouldbethefirststepinthesimplificat
ionprocess.Manyinterfacedesignprojectshavegonewrongbymakinguninformedassumptions
aboutuserneeds(Nielsen,1993).Theoverarch-ingneedforsimpledesignmakesunderstandin
gboththeprimarytasksandtheirpriorityofuseabsolutelycriticaltogooddesign.Oncetask
sandtaskpriorityareunderstood,thedesignoftheinterfacecanbeorganizedaccordingly.T
hemostfrequentlyusedtasksandthosethatareusedbythemajorityofusersshouldbe
10.6DesignGuidelines343

themostvisibleintheinterface(Isaacs&Walendowski,2002).Figure10.18showsthesuggest
edvisibilityoftasksbasedontheirusage.ReduceFunctionalityStronglyconsiderremoving
functionalitythatfallsintotheOccasionalbyfewcategoryinthefigure.Infact,thedesigner
shouldagonizeovertheadditionofeachindividualfeaturetothedesignnomatterwhereitfal
lsonthefrequencyofusescale(37signals,2006).Bearinmindthateachadditionalfeaturema
kestheoverallproductmorecomplexandmoredifficulttouse.KeepNavigationNarrowandShal
lowSmall-screendevicesaregenerallyusedinsettingswheretheusersattentionisatapremiu
m,suchasmakingacallonabusybusormakingamealinthemicro-wave.Inthesecases,usersdono
wanttochoosefromadozenoptions;theywanttocompletetheirprimarytaskasquicklyaspossi
ble.Taskefficiencyisanextremelyimportantdesigngoalandoftenhelpsenforcetheprincip
leofsimplic-ity.Thisimpliesthattheinformationhierarchyoftheinterfaceshouldbekept
bothnarrow(fewerchoicesateachlevel)andshallow(fewerlevelsofchoicetothebottomofth
ehierarchy).AsdiscussedinSection10.4,ifthenumberofoptionsmustexceedanarrowandsha
llowhierarchy,thenitismoreefficienttodesignadeeperstructurethanawiderone.AvoidEx
traneousInformationonEachScreenTheprincipleofsimplicityextendsnotonlytotheoveral
leaseofuseofthedevicebutalsotothecomplexityofeachindividualscreen.Notethelimited
numberofcontrols,interfacewidgets,andtextusedintheNowPlayingviewoftheAppleiPodsh
owninFigure10.19.TheNowPlayingscreencouldshowotherpotentiallyusefuldataitems,suc
hasdetailedmusicmetadata,nextandprevioustrackinformation,trackratings,repeat/shu
fflestatus,andplaylistname,amongothers.However,thedesigner
MoreclicksFrequentBy manyFrequent by manyVisible, few clicksOccasional by manySu
ggested, more clicksFrequent by fewSuggested, few clicksOccasional by fewHidden,
more clicksBy fewOccasional
Less visible
FIGURE10.18Visibilitytasksbasedonusage.Thetasksintheinterfacearebasedontheamount
ofusageandthenumberofusers.Source:FromIsaacsandWalendowski(2002).
10Small-ScreenInterfaces344

hasintentionallyshownonlytheinformationcriticaltotheusersprimarytask.Notealsothes
impleiPodhardwarecontrols:Eachhardkeyonthedeviceservesadedicatedandimportantpurp
osewithineachcontext,andtherearenoextrane-ouscontrols.ReduceorRemovePreferencesA
dvancedcustomizationisastandardaspectoftraditionalapplicationdesignthatisnotsuit
edtothesmallscreen.Manytraditionalapplicationsallowtheusertocustomizetheuserexpe
riencethroughcomplex(andoftenunused)controlssuchaspreferencedialogs.Whilekeeping
theuserincontroloftheinterfaceisimpor-tant,providingmanydifferentwaystocustomize
thewaytheapplicationworksincorporatestoomuchcomplexityintoasmall-screeninterface
.Typically,small-screenapplicationsaresimplerthantraditionalapplications;thuscus
tomi-zationislessimportanteventoadvancedusers.Theexceptiontothisruleispotentiall
yannoyingfeaturessuchaswarningmessagesthatoftenpopup.Inthesecases,asimplercustom
ization,suchasallowingtheusertopermanentlydismissthedialog,canbeusefulbecauseits
implifiestheentireexperience.UseProgressiveDisclosurePerhapsthemostpowerfultooli

ninterfacesimplificationisprogressivedis-closure.Progressivedisclosureinvolvesbr
eakingdowncomplextasksintoseparateunderstandableparts.Eachstepinthetaskissplitin
toscreensthatleadtheuserstowardtheirgoal.Manyinterfaceguidelineshavetheirbasisin
progressive
FIGURE10.19NowPlayingviewofApplesiPod.Thisviewdemonstrateshoweachscreenoftheinter
facecanbemadebothsimpleandusefulatthesametime.Source:FromApple.com,2007;courtesy
AppleInc.
10.6DesignGuidelines345

disclosure;perhapsthebestexampleisthehierarchicalnavigationfoundinnearlyeverymod
erninterface.Insteadofgivingtheuser50differentoptionstochoosefrom,thosechoicesar
ebrokenupintomeaningfulgroupsandsubgroups.Onthesurface,progressivedisclosureappe
arstobreaktheguidelineofefficiency;however,inpractice,progressivedisclosureactua
llyimprovestheusersspeedincompletingtasks(Buyukkokten,Garcia-Molina,&Paepcke,2001
).Withaflatlistofoptions,theuserisforcedtodecidebetweenmyriad(oftenambiguous)opt
ions.Ittakestimefortheusertodecidewhichoptiontochooseand,ifthechosenoptionisinco
rrect,theusermustscanthelonglistandchooseagain.AsdiscussedinSection10.4,aproperl
ydesignedprogressivedisclosureemploysmutuallyexclusive(orthogonal)choicesateachs
teptoensurethattheuserisalwaysprogressingforwardtowardhergoal.Onasmall-screendev
ice,progressivedisclosureisevenmoreimportantthanintraditionalapplica-tionsbecaus
eofthelimitedscreenspaceavailabletodisplayinformation.10.6.2DesignwiththeSmallSc
reeninMindMinimizeUserInputAsdiscussedinSection10.2,userinputisanevenmoredifficu
ltdesignchallengethanvisualoutputforsmall-screendevices.Thisimpedimentimpliestha
tthedesignermustbethoughtfulwheneveraddinguserinputwidgetstotheinterface.Ifpossi
ble,avoiduserinputalltogether.Forexample,ifthedeviceisgivingfeed-backtotheuserth
atisnoncritical,itmaymakesensetouseatransientdialogboxandhavethedialogautomatica
llydismissitselfafterashortperiodoftime.Incaseswhereuserinputisrequired,keepther
equireduserefforttoamini-mum.Ifaparticularscreenrequirestheusertoenterhiscity(sa
ytodeterminetimezoneonamobilephone),donotpresentafree-formtextentryfield.Whilefr
ee-formentrymightbefairlyeasyforauserwithafull-sizedQWERTYkeyboard,usingtheDTMFk
eypadfoundonatypicalmobilephonetoenteralongcitynamewillrequireagreatdealofeffort
andtime.Widgetsthatusethecognitiveprincipleofrecognition(suchasradiobuttons,drop
-downs,andcheckboxes),ratherthanrecall(suchasfree-formentryfields),aremuchmoreef
fective.Inthepreviousexample,usingadrop-downmenuthatdisplayscitynamesorafieldtha
tenablesZIP/postalcodeentry(andperformsalookuptodeterminecityname)wouldbemuchmor
eappropriate.MinimizeVerticalScrollingandAvoidHorizontalScrollingIdeallyscreenss
houldbedesignedsothatthereisnoneedforscrollingofanykind.Inlarge-screenapplicatio
nsandontheWeb,applicationsareoftenheavywithinstructionalorinformationaltext.Thes
mall-screendesignerdoesnothavethisluxuryandinstructionaltextshouldbelessnecessar
ysincefunctionalityisnor-mallysimpleronthesmallscreen.Adesignthatrequireslengthy
textualinstruc-tionsisanindicationthattheinterfaceistoocomplexandshouldberevised
.
10Small-ScreenInterfaces346

Insomemorecomplicatedapplications,itisunavoidabletopresentascrol-lingscreen.Inth
esecasesitsimportanttouseverticalratherthanhorizontalscrolling.Theproliferationof
theWebhasmadeverticalscrollingawell-under-stoodinterfaceparadigmforthemajorityof
users.Westernusersreadfromtoptobottomandfromlefttoright,soverticalscrollingmapst
ranslatewelltothereal-worldtaskofreadingabook.Horizontalscrollingisfarlessunders
toodbyusersandbreaksboththemetaphorofbookreadingandtheperceptualprocessofvisuals
canning.Insomecases,horizontalscrollingisunavoidable(e.g.,whenscrollingaroundama

poranimagetoolargetofitonthescreen).Inthesecases,useascrollingborder(asdiscussed
inSection10.4).UseHyperlinkingEffectivelyInmanycases,itisdifficulttoprovideenoug
hinformationtotheuserinsideasin-glesmallscreen(e.g.,someerrormessagesrequiredeta
iledtextualadvice).Inthesesituations,hyperlinkingcanbeused.Thebeautyofthehyperli
nkisitsabil-itytohidedetailswhilestillmakingthemavailabletothosewhowantmoreinfor
-mation.Hyperlinkingisespeciallyusefulincaseswhereusersmayseethesametextmanytime
s(e.g.,acommonerrormessage).Inthesecases,theusermayneeddetailedinformationthefir
sttimethemessageisdisplayed,butdoesnotwanttobeforcedtoscrollthroughthesamelongte
xteverytimetheerroroccurs.Ofcourse,ifanerrorissocommonthattheuserisseeingitallth
etime,thisisagoodindicationthatthedesignisflawed.Alwaysstriveforadesignthatminim
izesthenumberofpossibleuser-initiatederrors.ProvideUsefulErrorMessagesSimplified
contentcansometimesleadtoconfusioniftakentoanextreme.Errormessagesneedtobehelpfu
levenonthesmallscreen.Ideallyanerrormessagecontainsthreedistinctparts:(1)aunique
erroridentifier,(2)adescriptionoftheproblem,and(3)possiblesolutions.Item1isimpor
tantinthecasewheretheuserisunabletoquicklyfigureoutasolution.Auniqueidentifieral
lowsuserstoeasilyseekhelpabouttheirspecificproblemfromanexternalsource(e.g.,aweb
siteorcustomerservice).Items2and3needtobewrittenwithouttechnicaljargonandinalang
uagethattheusercanunderstand.Properlywrittenerrormes-sagescanmakeasignificantdif
ferenceintheusabilityofacomplicatedsystem.PrioritizeInformationonEachScreenEvenm
inimalcontentcanbeconfusingtoauserifitisnotpresentedinamean-ingfulway.Asmentione
dearlier,displaythemostimportantinformationfirstandlargestonthescreen(alwaysbase
donyourknowledgeoftheusersprimarytasks).Ifnoprioritycanbegiventotheinformation,or
derthedatainalogicalmanner(e.g.,alphabeticallyorchronologically)toreducetheusersm
emoryloadandincreasetaskefficiency.Similarly,groupinglikedataelementstogethermak
esvisuallyscanningitemsfaster.
10.6DesignGuidelines347

10.6.3ProvideUsefulFeedbackVisual,auditory,andsensoryfeedbacktotheuserisoneofthe
mostcriticalaspectsoftheinteractiondesignofasmall-screendevice.Providingjustther
ightamountoffeedbackwithoutoverwhelmingtheusercanbeatrickytask.IdentifyCriticalF
eedbackAnotherguidelinebasedonthelimitedattentionthatuserstypicallyprovidetheirs
mall-screendevicesisthatuserfeedbackmustbeverystrong.Usersneedtobemadecontinuall
yawareofwhatthesystemisdoingwhiletheyinteractwiththedevice.Identifytheinformatio
nthatiscriticaltotheusersneedsandpresentonlythatinformationtotheuser.Fortherelati
velysimpleexampleofamicro-wavedisplay,itisimportantthatusersknowtheheatsettingof
themicrowaveandtheremainingtimeleftintheheatingprocess.Removingorobscuringeither
ofthosetwocriticalitemsfromthescreentodisplayotherinformationwillconfuseandfrust
rateusers.EmployAlternativeFeedbackModalitiesIntelligentlyTheuseofalternativefor
msoffeedbackcanbeverypowerfulonsmall-screendevices(oftenincontrasttotraditionalc
omputerapplications).Usingsoundandtactilefeedbackisastandardtechniqueonmanysmall
-screendevicesbecausetheusersattentionisrarelyfocusedexclusivelyonthedevice.Imagi
nehowuselessamobilephonewouldbeifitonlypresentedvisualfeedbackofanincomingcall!H
owever,thedecisiontousealternativefeedbackshouldnotbemadelightly.Usersdonotwantt
heirattentiondivertedunlessanimportanteventhasoccurred.Theguidelinehereistouseal
ternativefeedbackintelligentlyandsparingly.EnsureQuickSystemResponseTimeRelatedt
otaskefficiencyistheamountoftimethatittakestheinterfaceitselftorespondtotheuser.
Thereareveryfewtasksthataretechnicallycomplexenoughthattheusershouldhavetowaitfo
rtheinterfacetorespond.TheBlackBerryStyleGuidestates,userinterfaceresponsetimesho
uldbe,atworst,200ms(1/5ofasecond)orbetter.Ifanoperationisexpectedtotakealongtime
(e.g.,atextsearchoradatabasequery),thenitshouldbeexecutedinthebackground(Research
inMotion,2003).Ifforsomereasonalongtaskcannotbeperformedinthebackgroundandtheuse
risforcedtowait,ensurethatdetailedfeedbackispresented.Ideallysuchfeedbackinclude
sanestimateoftheamountoftimetheprocesswilltaketofinish.Theprogressbarwidgetisani
dealfeedbackcontrolforsuchsituationssinceitgivestheuseranindicationofhowlongthep
rocesswilltakeandprovidesfeedbackthattheprocessisstillmovingforward.Forany

10Small-ScreenInterfaces348

processthatforcestheusertowait,evenwithstrongfeedback,ensurethatyouprovideawayto
cancelthetask.10.6.4MaintainInteractionDesignStandardsAcrossthewiderangeofsmallscreendevices,manydifferentuserinterfacestylesareused.However,standardsdoexistan
dinsomecasestheyareverywelldevel-opedanddocumented.Wherestandardsdonotexist,usin
gconceptsthatusersalreadyunderstandcandramaticallyimprovetheusabilityofacomplexi
nteraction.UseExistingStandardsWhendesigningforthesmallscreen,aswithanyothertype
ofinterfacedesign,thedesignshouldbeginwithexistingstandards.Manyofthemoreadvance
dsmall-screendevices(suchastheWindowsMobileplatform)havedetailedinter-faceguidel
inedocumentsthatarecreatedtoensureconsistencyacrossallplat-formapplications.Less
complexdevicesalsohavestandards,buttheyareoftenimplicitandrarelywrittendown.Fore
xample,usershaveanexpectationthatpressingabuttononawatchwillcausesomeactiontoocc
urinthewatchdisplay.Thismayseemobvious,butmanywatchesbreakthisbasicstandardbyinc
luding(oftenunlabeled)buttonsthatonlyworkincertaincontexts.Itisimportanttoexplor
ethepresentation,widget,andinteractionstandardsofthedevicebeforedesignbegins.Avo
idingthesefundamentalbuildingblocksoftheinterfacecancausesignificantreworkwhenus
ertestinguncoversproblemswiththenonstandarddesign.Similarly,existingdevicevisual
-designstandardsshouldonlybebrokeninthemostextremecases,suchaswhentheexistingfra
meworksimplywillnotsupportthetaskyouaretryingtoachieve.UseReal-WorldMetaphorsExi
stingreal-worldmetaphorsthattranslateintotheworldofsoftwarearesomeofthemostpower
fuluserinterfaceparadigms.Takeforexampletheplay,pause,trackforward,andbackbutton
sfoundonalmostalldigitalmusicplayers(Figure10.19).Thesecontrolshavebeencarriedfo
rwardsincestand-alonetapeplayerswerefirstreleased.Makingthedecisiontobreakawellunderstoodmetaphorwilltaketheinterfaceintounknownwaters;thedesignerwillhavenoide
awhetherthenewinterfacewillworkeffectivelyuntilusertestingisperformed.Thesamegoe
sforcontrolwidgets.Makingadecisiontocreateormodifyanexisting,well-understoodwidg
etisextremelydangeroustotheusabilityofthedevice.InhisfamousbookTogonInterface,Ap
pleMacintoshGUIdesignerBruceTognazzini(1992)chronicleshisexperiencetryingtocreat
eahybridcheckbox/radiobuttonwidget.Aftermanyfrustratingiterationsofdesignandtest
ing,hecompletelyabandonedtheseeminglysimplecontrolbecauseuserscouldneverfigureou
thowthenewwidgetworked.
10.6DesignGuidelines349

10.6.5RespectBothMentalandPhysicalEffortDuetothelimitedamountofattentionusersgiv
etotheirsmall-screendevices,itisimportanttorespecttheamountofbothmentalandphysic
alefforttheusermustemploytointeractwiththeinterface.Insteadofforcingtheusertodot
hework,agooddesignwillmaketheuserstasksalmosteffortless.Forexample,severalcameras
providepanoramamodesthataidtheuserinthecreationofpanoramicpicturesbyshowingthesi
deofthepreviouspictureonthescreentohelptheuserlineupthenextpictureintheseries(Is
aacs&Walendowski,2002).UseWizardstoSimplifyComplexInteractionsThesoftwarewizardi
sapotentnavigationparadigmthatembodiestheguidelineofprogressivedisclosureandredu
cestheusersmemoryloadbysplittingupcom-plextasksintomultiplesteps.Awell-designedwi
zardincludestheabilitytomoveforwardandbackthroughthevariousstepsandalsokeepstheu
sersinformedaboutwheretheyareinthewizardprocess.Thekeywithsmall-screenwizarddesi
gnistoeliminateasmanyinputfieldsaspossiblethroughsimplificationbyputtingtheonuso
nthesystemsoftwaretodoasmuchoftheworkaspossible.Forexample,iftheinterfaceiscaptu
ringanaddress,askfortheZIP/postalcodefirst,thenprepopulatethecityandstatefieldsw
iththeappropriatedatatoreduceuserinput.UseMultitaskingOperatingSystemsEffectivel
yManymodernsmall-screenoperatingsystemsnativelysupportmultitasking.Whilemultitas
kingisaterrificboontosoftwaredesign,caremustbetakeninitsinterfaceimplementation.
Duetofactthatusershavelimitedattentionfortheirdevices,itisnotagoodideatoforceaus
ertomultitasktocompleteatask.Whenusingasmall-screendevice,userstypicallyhaveasin

gletaskinmindandtheywanttocompletethattaskasquicklyaspossible.Multitaskingisapow
erfultool,butitshouldbeusedprimarilybythesystemandnotbytheuser.Forexample,system
tasksthatcannotberesolvedinatimelymannershouldbeperformedinthebackgroundsothatth
eydonotinterferewiththeuserscontinueduseofthedevice.DesignforEfficiencyTheNokiaSe
ries60ContactsapplicationshowninFigure10.20isastrongexam-pleofefficientandsimple
small-screendesign.Theuserisabletoscrollverticallythroughthelistofalphabetizedco
ntacts.However,iftheuserhasalargenumberofcontacts,thelistgetstoolongtoscrollthro
ugh.Inthiscasethedesignershaveincludedanalphabeticalfilterthatworkswiththekeypad
.PressingtheletterMwillfiltertheentirelisttoshowonlythosenamesthathaveanMbeginni
ngthefirstorlastname.Thistechniqueissurprisinglyeffectiveatimprovingtask-complet
iontimeandisintuitivetolearnevenbynoviceusers.
10Small-ScreenInterfaces350

10.7CASESTUDYAcasestudyofasmall-screeninterfacecanbefoundatwww.beyondthegui.com.
10.8FUTURETRENDSAsdiscussedinSection10.3ofthischapter,thereareseveraltrendsthatw
illcon-tinueintothefutureofsmall-screendevices.Themostprevalentoneaffectsalltech
nologicalfields:increasedcapabilityofbothsoftwareandhardware.Whenmakingpurchased
ecisions,usersoftenbuybasedonthefeaturelistofthedevice(Sarker&Wells,2003).Thistr
endencourageshardwaremanufacturerstointe-grateadditionalfunctionalityintotheirsm
all-screendevices.High-endsmartphonesandhybriddevicesareconstantlyincorporatingf
unctionalitypreviouslyfoundonlyindedicateddevices,suchaslocation-basedservices,h
igh-resolutioncameras,multimediamessaging,andimprovedtextinput.Technologiesforme
rlyfoundsolelyintraditionalpersonalcomputerswillalsocontinuetobeintegratedintosm
all-screendevices:multimediaediting,increasedwirelessdatabandwidth,largerdevicem
emory,andopensoftwarearchitectures.Theoppositetrendwillalsocontinueinthefuture:s
implededicatedinforma-tionappliancesthatperformalimitednumberoffunctionalitiesbu
tdosoinaflexible,feature-rich,andusablemanner.Whilethoseonthecuttingedgewillcontinuetopurchaseleading-edgedevices,theaverageconsumerwantslesssophisti-catedande
asier-to-usetechnology.Thematuringworldofmultimediainvolvestheblendingofseveralc
omplextechnologies,makingtheconsumerslearningcurvesteep.Mediacentersystemswillcon
tinuetopresentstraightforward,uncomplicatedinterfaceshidingtheintricatemediaplay
back,wirelessnetworking,
FIGURE10.20NokiaSeries60Contactsapplication.Thefieldwiththemagnifyingglassallows
theusertoquicklyfindacontactusingfiltering.Source:Fromallaboutsymbian.com,2007.
10.7CaseStudy351

mediasharing,andincreasedsoftwarecontrolfoundunderthesurface.Thenumberofinformat
ionappliancestargetedtowardallareasofsmall-screentechnologywillcontinuetoblossom
overtime.Thetrendtolarger,higher-resolutiondisplayswillalsocontinue.Interfacesth
atusevectorgraphicstoscaletoanyresolutionwillbecomemorecommon.FlexibleOLEDdispla
yswillmeanthatbigscreenswillbecomeportableanduselessbatterypowerthanLCDswhiledis
playingricherdataandmedia.Figure10.21showstwoconceptdevicesusingrollabledisplays
.Inaddition,eyewearenablinguserstoprivatelyviewabig-screen-likedisplayhasalready
becomeavailable.In2005,OrangeSAlaunchedvideoeyewearwithaSamsungphoneenablinguser
stoviewTV,movies,photos,andbroadbandInternetcontentwithabig-screenviewingeffect(
DigitalCamera@101review,2005).Thesmall-screendevicewillcontinuetocatchuptothePCi
ntermsofCPUspeedandfunctionality,andmobiledeviceswillrunanincreasinglycomplexvar
ietyofsoftware.Inthemedium-termfuture,displayinginformationonatraditionalscreenw
illbeeclipsedbynewdisplaytechnologies.Theproblemwithcurrentlarger-but-portablesc
reentechnologyisthatnomatterhowsmallindividualcomponentscanbemade,thescreenwilla
lwayslimitthereductioninsizeoftheformfactor.Peoplewantsmalldevicesthatareportabl

e.Makingthedisplayexternaltothedeviceremovesthisrestriction.Threeinterestingtech
nologiesareprojectiondisplays,heads-updisplays,andopen-airholographic-likedispla
ys.Projectiontechnologymeansthatthescreenitselfcanbeaslargeastheuserrequires,but
thedeviceitselfisstillverysmallandthushighlyportable.Heads-updisplaysuseanextern
aldevicetodisplayimportantinformationtotheuser.Theopen-airholographic-likedispla
yprojectsanimageintheairjustaboveit.Oneexampleofthisistheheliodisplaydevelopedby
IO2Technologythatiscapableofproject-inganinteractivetwo-dimensionalimage(Genuth,
2006).Allofthesedisplaytechnologieswillseefurtherrefinementincomingyears.
FIGURE10.21Conceptdevicesusingrollabledisplays.Theconceptatrightshowsdeviceopena
ndclosed.Source:FromDeRossi(2005);courtesyPolymerVision.
10Small-ScreenInterfaces352

AsdemonstratedinSection10.2,theparamountchallengeinsmall-screendevicedesignisnot
outputbutuserinput.Theprimarymethodsofinputonapersonalcom-puter(themouseandkeybo
ard)donottranslatewellintotherealmofmobiledevices.Inthenext10years,threeareasofr
esearchwilldominatesmall-deviceinput:spatialinterfaces,eye-trackinginterfaces,an
dnatural-languagecomprehension.Aspatialinterfaceisaninputtechnologythatisawareof
devicemovement.Asthedevice(oradedicatedinputdevice)ismoved,theinterfacerespondsa
ppropriately.Spatialinterfacesareeasytouseandwellunderstoodevenbynov-iceusers.Ty
pically,aspatialinterfaceworksinthreedimensions,allowingtheinputdevicetobemoveda
nywhereinspace.Thesudden-motionsensorbuiltintoApplelaptops(originallydesignedtop
reventdamagetotheharddiskifdropped)hasalreadybeenco-optedasaspatialinput.Enterpr
isingengineershavealreadybuiltsoftwareforthemotionsensortoscrollwindowsandplayga
mes(Singh,2006).Dedicatedspatial-interfacedevicesliketheWiiRemoteincludedwiththe
NintendoWiiprovideacompletelynewwaytointeractwithgamesbyincludingagyroscopetotra
ckhandmovement.Motion-sensingtechnologyisapracticalinputmethodologyforsmalldevic
esbecausethehardwarerequiredcanbemadeverysmallandtheinteractionisnaturalforusers
.Anothertechnologywithconsiderablepotentialrequiresevenlessusereffortthanaspatia
linterface.Eyetracking(sometimescalledagazeinterface)monitorsthemovementoftheeye
acrossavisualdisplay.Thetrackedmovementoftheeyeistranslatedintoinputwithintheint
erface.Eyetrackinghasbeentradition-allyusedinperceptionandadvertisingstudiesbuts
howspromiseforvisualsearchtaskssuchasexploringimagesandforsoftwarenavigation.The
downsideofeye-trackinginterfacesisthattheyrequiresometrainingtouseeffectively.Ho
wever,eye-trackinginterfacescanbemadesmallenoughforsmall-screendevicesandarealso
appealingfromanaccessibilitystandpoint.Atechnologywithgreatpotential,bothfromane
ase-of-useandanaccessibil-ityperspective,isnatural-languagerecognitionandcompreh
ension.AsdiscussedinSection10.2,somemobilephonesalreadyincorporaterudimentarylan
guagerecognitionwithlimitedsuccess.Aslinguistsbetterunderstandhowtoteachcom-pute
rssemanticandgrammaticalunderstanding,languageinterfaceswillimprovetothepointtha
ttheyareapowerfulinputmodality(Yates,Etzioni,&Weld,2003).Alreadytechnologiessuch
asIBMsViaVoiceforthePCareapproachingfunc-tionaleverydayuse,anditisonlyamatterofti
mebeforethetechnologyisfoundinsmallerdevices.Evenfurtherinthefuture,thequintesse
ntialhumancomputerinterfacewillcommunicatedirectlywiththebrain.Whilevirtualrealit
ypumpeddirectlyintothebrain(liketheinterfacedepictedinthemovieTheMatrix)maybeman
yyearsaway,elementarybraininputinterfacesarealreadybeingstudiedinlabssuchastheUn
iversityofBritishColumbiasBrainComputerInterfacelab(Birch,2007).Clearlyadirect-tobraininterfaceinvolvesalmostnoefforttocontrolinputoroutputdata,andisaccessibleto
anyonewithorwithoutphysicaldisabilities.Thebraininterfacetrulyrepresentsthepanac
eaofcomputinginterfaces.
10.8FutureTrends353

REFERENCES37signals.(2006).GettingReal;availableathttp://gettingreal.37signals.c
om/toc.php.Alibaba.com.(2007).CarParkingSensorSystemwithLEDDisplay;availableatht
tp://cyberfair.trustpass.alibaba.com/product/10885882/Car_Parking_Sensor_System_
With_LED_Display.html.Allaboutsymbian.com.(2007).Nokiae61ContactsApplication;ava
ilableathttp://allabout-symbian.com/images/devices/e61/nokaie61_review2_07.JPG.A
mericanNationalStandardsInstitute(ANSI)/HumanFactorsandErgonomicsSociety(HFES)10
0-1988.(1988).AmericanNationalStandardforHumanFactorsEngineeringofVisualDisplayT
erminalWorkstations.SantaMonica:HumanFactorsandErgonomicsSociety.Apple.(2007).So
undandHearing;availableathttp://www.apple.com/sound/.AppleInsider.com.(2006).iPo
d:Howbigcanitget?AppleInsider.com,Mayhttp://www.appleinsider.com/article.php?id=1
770.Av.it.pt.(2007).TektronixTDS3052B;availableathttp://www.av.it.pt/wcas/images
/PIC0014.JPG.Baecker,R.,Small,I.,&Mander,R.(1991).Bringingiconstolife.Proceeding
softheSIGCHIConferenceonHumanFactorsinComputingSystems:ReachingthroughTechnology
.NewYork:ACMPress,16.Baudish,P.,Xie,X.,Wang,C.,&Ma,W.(2004).Collapse-to-zoom:View
ingWebpagesonsmallscreendevicesbyinteractivelyremovingirrelevantcontent.Proceedi
ngsofSev-enthAnnualACMSymposiumonUserInterfaceSoftwareandTechnology.NewYork:ACMP
ress,9194.Bernard,M.L.(2002a).Examiningametricforpredictingtheaccessibilityofinfo
rmationwithinhypertextstructures.PhDdiss.WichitaStateUniversity.Bernard,M.L.(200
2b).Examiningtheeffectsofhypertextshapeonuserperformance.Usabil-ityNews4.2http://
psychology.wichita.edu/surl/usabilitynews/42/hypertext.htm.Blackwell,H.R.(1946).
Contrastthresholdsofthehumaneye.JournaloftheOpticalSoci-etyofAmerica36:624.Birch
,G.(2000).TheBrainComputerInterfaceProject;availableathttp://www.ece.ubc.ca/~gary
b/BCI.htm.Bohan,M.(2000).Enteringtextintohand-helddevices:Comparingtwosoftkeyboa
rds.UsabilityNews2.1;availableathttp://psychology.wichita.edu/surl/usabilitynews
/2W/softkeyboards.htm.BoardofStandardsReview(BSR)/HFES100.(2006).HumanFactorsEng
ineeringofCom-puterWorkstations:SecondCanvassReview,2006.SantaMonica,CA:HumanFac
torsandErgonomicsSociety.Buyukkokten,O.,Garcia-Molina,H.,&Paepcke,A.(2001).Seein
gtheWholeinParts:TextSummarizationforWebBrowsingonHandheldDevices.DigitalLibrary
Project(InfoLab),StanfordUniversity.Availableathttp://www10.org/cdrom/papers/594
/.Cambridgeincolour.com.(2007).BasicsofDigitalCameraPixels;availableathttp://www
.cambridgeincolour.com/tutorials/digital-camera-pixel.htm.Chapanis,A.(1949).Howw
esee:Asummaryofbasicprinciples.InHumanFactorsinUnderseaWarfare.Washington,DC:Nat
ionalResearchCouncil,12.
10Small-ScreenInterfaces354

Cooke,N.J.(1994).Varietiesofknowledgeelicitationtechniques.InternationalJournalo
fHumanComputerStudies41:80149.DeRossi,L.C.(2005).Portablerollabledisplays.Foldable
screensthatfitinyourpocket;availableathttp://www.masternewmedia.org/video_displa
y/portable_screens/rollable_portable_screens_prototypes_first_looks_20050712.htm
.DigitalCamera@101reviews.(2005).Bigscreenviewingeffectformobilephonevideos;avai
lableathttp://digitalcamera.101reviews.com/news/big-screen-viewing-effect-for-mo
bile-phone-videos.Dix,D.A.(1995).Accelerationandtoolbars:Learningfromthemenu.Pro
ceedingsofPeopleandComputersIX(Adjunct):BritishComputerSocietyConferenceonHumanC
omputerInteraction.London:BritishComputerSociety,13843.Duchnicky,R.L.,&Kolers,P.A
.(1983).Readabilityoftextscrolledonvisualdisplayterm-inalsasafunctionofwindowsiz
e.HumanFactors25(6):68392.Dumas,J.S.,&Redish,J.C.(1994).APracticalGuidetoUsabilit
yTesting.Norwood,NJ:Ablex.Elkerton,J.,&Williges,R.(1984).Informationretrievalstr
ategiesinafilesearchenviron-ment.HumanFactors26(2):17184.Enablemart.com.(2007).He
aring;availableathttp://www.enablemart.com/productGroup-Detail.aspx?store=10&dep
t=15&group=29.Engadget.com.(2005).MorseCodeTrumpsSMSinHead-to-HeadSpeedTextingCo
mbat.May26http://www.engadget.com/2005/05/06/morse-code-trumps-sms-in-head-to-hea
d-speed-texting-combat/.Engadget.com.(2004).WatchthisWednesday:TheSamsungGPRSCla
ss10WatchPhone.November17http://www.engadget.com/2004/11/17/watch-this-wednesdaythe-samsung-gprs-class10-watch-phone/.Genuth,I.(2006).HeliodisplayFloatingFree-Spa
ceInteractiveDisplay;availableathttp://www.tfot.info/content/view/101/.Geven,A.,
Sefelin,R.,&Tscheligi,M.(2006).Depthandbreadthawayfromthedesktoptheoptimalinforma

tionhierarchyformobileuse.InProceedingsofEighthConferenceonHumanComputerInteracti
onwithMobileDevicesandServices.NewYork:ACMPress,15764.Gohring,N.(2006).Mobilephon
esalestopped800millionin2005,saysIDC.InfoWorldhttp://www.infoworld.com/article/06
/01/27/74861_HNmobilephonesales_1.html.Grether,W.F.,&Baker,C.A.(1972).Visualpres
entationofinformation.InVanCott,H.P.,&Kinkade,R.G.,eds.,HumanEngineeringGuidetoE
quipmentDesign.NewYork:JohnWiley&Sons,41121.Gutwin,C.,&Fedak,C.(2004).Interacting
withbiginterfacesonsmallscreens:Acompari-sonoffisheye,zoom,andpanningtechniques.
Proceedingsof2004ConferenceonGra-phicsInterfaces.Waterloo,Ontario,Canada,14552.Gw
micro.com.(2007).GWMicroSmallTalkUltra;availableathttp://www.gwmicro.com/Small-Ta
lk_Ultra/.Hayhoe,D.(1990).Sorting-basedmenucategories.InternationalJournalofManMa
chineStudies33:677705.Isaacs,E.,&Walendowski,A.(2002).DesignfromBothSidesoftheScr
een.Indianapolis:NewRiders.Jones,M.,&Marsden,G.(2006).MobileInteractionDesign.Lo
ndon:JohnWiley&Sons.
References355

Lam,H.,&Baudisch,P.(2005).Summarythumbnails:Readableoverviewforsmallscreenwebbro
wsers.ProceedingsofSIGCHIConferenceonHumanFactorsinComputingSys-tems.NewYork:ACM
Press,68190.Lee,E.S.,&Raymond,D.R.(1993).Menu-drivensystems.InKent,A.,&Williams,J
.,eds.,TheEncyclopediaofMicrocomputers11:10127.Lewis,C.,&Reiman,J.(1993).Task-Cen
teredUserInterfaceDesign;availableathttp://hcibib.org/tcuid/index.html.Logitech.
com.(2007).HarmonyW670AdvancedUniversalRemote;availableathttp://www.logitech.com
/index.cfm/products/detailsharmony/US/EN,CRID=2084,CONTENTID=12701.MacGregor,J.N
.(1992).Acomparisonoftheeffectsoficonsanddescriptorsinvideotextmenuretrieval.Int
ernationalJournalofManMachineStudies37:76777.Marics,M.A.,&Engelbeck,G.(1997).Desig
ningvoicemenuapplicationsfortelephones.InHelander,M.,Landauer,T.K.,&Prabhu,P.,ed
s.,HandbookofHumanComputerInter-action.Amsterdam:ElsevierScience,1085102.Marshall,
C.C.,&Ruotolo,C.(2002).Reading-in-the-small:Astudyofreadingonsmallformfactordevi
ces.InProceedingsofSecondACM/IEEE-CSJointConferenceonDigitalLibraries.NewYork:AC
MPress,5664.MiniFonts.com.(2006).Homepage.http://www.minifonts.com/.Muto,W.H.,&Ma
ddox,M.E.(2007,inpress).Visualdisplays.InWeinger,M.B.,ed.,Hand-bookonHumanFactor
sinMedicalDeviceDesign.Hillsdale,NJ:Erlbaum.Nielsen,J.(1993).UsabilityEngineerin
g.SanFrancisco:MorganKaufmann.Noldus.(2007).Mobiledevicecamera;seehttp://www.nol
dus.com/site/doc200402054.Norman,D.(1999).TheInvisibleComputer.Cambridge,MA:MITP
ress.Norman,D.(1988).TheDesignofEverydayThings.NewYork:BasicBooks.Norman,K.L.,&C
hin,J.P.(1988).Theeffectoftreestructureonsearchperformanceinahierarchicalmenusel
ectionsystem.BehaviourandInformationTechnology7:5165.Nystedt,D.(2006).Mobilesubsc
riberstoreach2.6Bthisyear.InfoWorldhttp://www.infoworld.com/article/06/11/10/HNmo
bilesubscribers_1.html.Onetouch.ca.(2007).OneTouchUltra;availableathttp://www.on
etouch.ca/english/product_detail.asp?gr=0&cat=1&pid=47.Paap,K.R.,&Cooke,N.J.(199
7).Designofmenus.InHelander,M.,Landauer,T.K.,&Prabhu,P.,eds.,HandbookofHumanCompu
terInteraction.Amsterdam:ElsevierSci-ence,53372.ResearchinMotion.(2003).Blackberr
yWirelessHandheldUserInterfaceStyleGuide;availableathttp://www.blackberry.com/de
velopers/na/java/doc/bbjde/BlackBerry_Wireless_Hand-held_User_Interface_Style_Gu
ide.pdf.Roeder-johnson.com.(2007).ProjectionKeyboardPDA;availableathttp://www.ro
eder-johnson.com/RJDocs/CAProjectionKeyboardPDA.jpg.Roske-Hofstrand,R.J.,&Paap,K
.R.(1986).Cognitivenetworksasaguidetomenuorgani-zation:Anapplicationintheautomat
edcockpit.Ergonomics29(11):130112.Rubin,J.(1994).HandbookofUsabilityTesting:Howto
Plan,Design,andConductEffec-tiveTests.NewYork:JohnWiley&Sons.Rugg,G.,&McGeorge,P
.(1997).Thesortingtechniques:Atutorialpaperoncardsorts,pic-turesorts,anditemsort
s.ExpertSystems14(2):8093.
10Small-ScreenInterfaces356

S60.com.(2007).Nokia95;availableathttp://www.s60.com/life/s60phones/displayDevic
e-Overview.do?deviceId=1065.Sanders,M.S.,&McCormick,E.J.(1993).HumanFactorsinEng
ineeringandDesign.NewYork:McGraw-Hill.Sarker,S.,&Wells,J.D.(2003).Understandingm
obilehandhelddeviceuseandadoption.CommunicationsoftheACM46(12):3540.Schiffman,H.R
.(1990).SensationandPerception:AnIntegratedApproach.NewYork:JohnWiley&Sons.Schus
teritsch,R.,Wei,C.,&LaRosa,M.(2007).Towardstheperfectinfrastructureforusabilityt
estingonmobiledevices.ProceedingsofConferenceonHumanFactorsinComputingSystems.CH
I07ExtendedAbstractsonHumanFactorsinComputingSys-tems.NewYork:ACMPress,183944.Sing
h,A.(2006).TheAppleMotionSensorasaHumanInterfaceDevice;availableathttp://www.ker
nelthread.com/software/ams2hid/ams2hid.html.Tognazzini,B.(1992).TogonInterface.R
eading,MA:Addison-Wesley.U.S.DepartmentofDefense.(1999).DepartmentofDefenseDesig
nCriteriaStandard:HumanEngineering.MIL-STD-1472F.Washington,DC:U.S.DOD.Wikimedia
.org.(2007).MotorolaRAZRKeypad;availableathttp://upload.wikimedia.org/wikipedia/
en/thumb/1/10/RazrKeypad.jpg/180px-RazrKeypad.jpg.Wikipedia.org.(2007).Vacuumflu
orescentdisplay;availableathttp://en.wikipedia.org/wiki/Vacuum_fluorescent_displ
ay.Williams,M.,&Cowley,S.(2006).Update:PCmarketachieveddouble-digitgrowthin2005.
Info-World;availableathttp://www.infoworld.com/article/06/01/19/74317_HNgrowthin
2005_1.html.Wobbrock,J.O.,Forlizzi,J.,Hudson,S.E.,&Myers,B.A.(2002).WebThumb:Int
eractiontechniquesforsmall-screenbrowsers.ProceedingsofFifteenthAnnualACMSymposi
umonUserInterfaceSoftwareandTechnology.NewYork:ACMPress,20508.Wright,S.L.,&Samei,
E.(2004).Liquid-crystaldisplaysformedicalimaging:Adiscussionofmonochromeversusco
lor.MedicalImaging2004:Visualization,Image-GuidedProce-dures,andDisplay.Proceedi
ngsofSPIE5367:44455.Yates,A.,Etzioni,R.,&Weld,D.(2003).Areliablenaturallanguagein
terfacetohouseholdappliances.InProceedingsofEighthInternationalConferenceonIntel
ligentUserInter-faces.NewYork:ACMPress,18996.Zwick,C.,Burkhard,S.,&Kuhl,K.(2005).D
esigningforSmallScreens.Switzerland:AVA.
References357

Thispageintentionallyleftblank

11CHAPTERMultimodeInterfaces:TwoorMoreInterfacestoAccomplishtheSameTaskAaronW.Ba
ngor,JamesT.Miller11.1NATUREOFTHEINTERFACEToday,technologyisprovidingconsumerand
businessuserswithanincreasingarrayofdevicesandsoftwareapplicationsthatcanbeusedt
oorganize,entertain,andinform.Becausethesetechnologysolutionshavebecomemoreubiqu
itous,flexible,andcapable,itisreasonabletoaskiftheuserinterface(UI)thatallowsthe
usertotakeadvantageofthetechnologyhaskeptpace.Acoupleofscenariosmayhelpillustrat
ethis.Inthefirstscenario,letussupposethatyouareridingthebushomefromworkand,justa
sMurphysLawmighthavepredicted,yougetstuckintrafficandwillnotbehomeuntilmuchlatert
hanyouexpected.Youwill,infact,besolatethatyouaregoingtomissyourfavoriteTVshow.Yo
udidnotsetituptorecordbecauseyouthoughtyouwouldbehomeinplentyoftime.Nooneishomet
orecorditforyou,soyounowwishyouhadawaytorecordtheshow,suchascallingyourdigitalvi
deorecorder(DVR)fromyourcellphonetosetuptherecording.Anotherscenarioisthatyourbe
stfriendjusthadabigargumentwithherroommateandyouwanttocallhertogivehersomesuppor
t.Shehasmovedtoanotherplaceandemailedyouhernewphonenumber.Youknowyouchangedtheph
onenumberwhenyougotamessagefromher,butyoucannotrememberwhereyouchangeditinyoure-m
ailaddressbook,yourcellphonespeeddial,yourvoicediallistonyourhomephone,and/oryou
rPDA.Andyoudefinitelydonotwanttocalltheoldnumberandspeakwiththeformerroommate.Th
efirstscenariopointstousersneedtobeabletomanagetheirservicesusinganumberofdiffere
ntdevices,dependingonthesituation.Thesecondsce-narioillustratesthefrustrationsth
atmanyuserscanexperiencewhenthese

needsaremet,butwithoutforethoughtabouthowthemultipledevicesthatpro-videthesamefu
nctionalitymightworktogether.Thatisthetopicofthischapterhowtodesignapplicationsth
atprovideuserswithmorethanoneUImodality,sothattheycaninteractwiththesameapplicat
ionanddothesametaskswithwhateverdeviceandinterfacearemostconvenientatthetime.For
example,inthefirstscenariothevisualon-screeninterfaceforthehometelevisionissuffi
cientforthemostcommonusersituations.However,anothervisualinterfaceaccessedbyacom
puter,awirelessphone,orPDAwouldbeusefulinothersituations.Itmightevenbebettertopr
ovideanauditoryinterfaceviaaphonesothattheuserwouldnotbeforcedtoentertherelevant
informationbutcouldjustspeakit.Inthesecondscenario,themultiplemodalitiesarealrea
dypresent,butthedesignofeachdevicewasdonewithoutconsideringthatthedeviceandtheinf
ormationusedbythedevicemightalsoapplytoseveralotherdevicesusedtoaccomplishthesame
tasks.Ideally,theinformationcommontoallofthesedevicesshouldbeavailabletoallofthe
msothatachangeintheinformationforonedevicewouldresultinachangeforall.Thenatureof
theindividualUIsthatareapartofmultimodeUIs(MMUIs)is,forthemostpart,notdifferentf
romtheotherUImodalitiesdiscussedinthisbook.Andmuchhasbeenresearchedandwrittenabo
utsingle-user,single-modeUIscomparedtoMMUIs(Kray,Wasinger,&Kortuem,2004).However
,thefocusofthischapterisontheuniquenatureofdesigningUIsforthesameapplicationwher
etheuserisexpectedtousemultiplemodalitiestointeractwiththesameserviceandaccompli
shthesametasks.Researchrequirements,criteriatocon-siderwhenelectingtocreatemulti
plemodalities,anddesignandtestingmethodsareincludedinthischapter.Asastart,Seffah
,Forbrig,andJavahery(2004)providethefollowingele-mentsthatdefineanMMUI:FAllowsas
ingleoragroupofuserstointeractwithserver-sideservicesandinformationusingdifferen
tinteraction/UIstyles.FAllowsanindividualoragrouptoachieveasequenceofinterrelate
dtasksusingmultipledevices.FPresentsfeaturesandinformationthatbehavethesameacros
splatforms,eventhougheachplatform/devicehasitsspecificlookandfeel.FFeelslikeavar
iationofasingleinterfacefordifferentdeviceswiththesamecapabilities.Notethatthese
elementsaretechnologyindependentandfocusonauserorgroupofusersaccomplishingtasks,al
thoughtheremaybevariationsonhowthisisimplemented,dependingontheexactdeviceandUIt
echnology.ThatisnottosaythatanMMUIisimplementedwithoutconsideringthespecificUIte
ch-nologiesavailable(Paterno,2004).
11MultimodeInterfaces:TwoorMoreInterfaces360

Typically,anMMUIisimplementedusingvisualand/orauditoryUIs.Otherchaptersofthisboo
kcovertheseinmoredetail,includingspeech(Chapter6),touch-tone(Chapter7),andsmallscreenvisualdisplays(Chapter10).Chapter12alsohasusefulinformationondesigningform
orethanonemodality,althoughitcoversintegratingmorethanonemodalityintothesameUI,r
atherthanprovidingmultiple,yetseparate,modalities.Therearenumeroussourcesavail-a
bleforhowtodesignvisualuserinterfaces.Pleaseseetheendofthischapterforgeneralrefe
rencesregardinggraphicalUIs(GUIs)(Carroll,2003;ISO,1998;Thatcheretal.,2006;Weins
chenk,Jamar,&Yeo,1997).11.2TECHNOLOGYOFTHEINTERFACEAnyUItechnologycanbeacandidat
eforanMMUI.Generallysomecombinationofgraphical,web,touch-tone,speechrecognition,
andsmall-screenvisualUIsisconsideredwhendesigningMMUIs.Itispossiblethatothermoda
lities,suchastactile,olfactory,andgustatory,couldbeused,butthesegenerallyhavenot
beenconsideredbecausetheylackthepowerfulinput/outputcapabilitiesthatthevisualand
auditorytechnologieshave.Otherchaptersofthisbookdiscussthetechnologiesusedinnonv
isualUItechnologiesandmanyotherworkscanprovidethesameforvisualUIs.How-ever,there
arestillsomeimportanttechnologyconsiderationsforMMUIs.First,itshouldbeunderstood
thatwithMMUIsthereisagreatlyincreasedchanceofsimultaneousinteractionbetweenappli
cationanduserormultipleusersonthesameaccounteitherintentionallyorunintentionally.
Whilethesesituationsoccurwithsingle-modalityUIs,greaterattentionneedstobegivento
theminthecaseofMMUIsbecauseofthemultipleaccessmethods.Majorconcernsinthesescenar
iosfollow:FEnsureaccuratepresentationofinformationtotheuserbecauseofupdatesmadei
notherUIs.FPreventuserconfusionaboutdisplayedinformationorapplicationbehaviorbec
auseofchangesmadethroughotherUIsthatchangedthestateoftheinformationorapplication
.FEstablishtheprecedencegiventochangesmadetotheaccountoraccountdatafromthediffer

entUIs.FPreventdatacorruptionorlostworkwhenaccessingthesamedatafrommorethanoneUI
.FAllowtheusertoquicklyswitchbetweenUIstoaccomplishataskoraseriesoftasks,ifitisa
ppropriatetothetask(s).Asanexampleofthefirstconsideration,thinkofasituationinwhi
chchangestoanaddressbookononedevicearepropagatedtootherdevicesthathaveaddressboo
ks.
11.2TechnologyoftheInterface361

Ifalldevicesarenotquicklyupdatedassoonasachangetoanaddresshasbeenmade,andtheuser
triestousetheaddressonanotherdevice,thenthesubsequentselectionofthataddressmayno
tbeaccurate.Additionalconfusionmayoccur,dependingonwhethertheuserrecognizesthatt
heupdatehasnotyetoccurred.Consequently,itisimportantthatinformationbeupdatedinat
imelymannersoausercantrustthatchangesmadeusingonedevicewillbeavailableonalldevic
es.ConfusionaboutdataorapplicationbehaviorbecauseofchangesmadeinotherUIscanhappe
ninmanysituations.Forinstance,inavoicemailapplicationthatcanbereachedovertheWebo
rthephone,onememberofahouseholdmightlistentoamessagestreamedovertheWebandthendel
eteit.Ataboutthesametimeanothermemberofthehouseholdischeckingmessagesfromhercell
phoneandhearsthatthereisanewmessage.However,whenshechoosestolistentothemessage,t
herewillbenomessagetoplaybecauseithasbeendeleted.Typically,theprecedenceforchang
estoanaccountoraccountdatafromdiffer-entUIsiswhicheveristhemostrecent.So,inthepr
eviousexample,usuallythehouseholdmemberwhogavethecommandtodeleteorsavethemessage
firstwouldestablishtheprecedenceforthisspecificsetofinformation.However,othercri
teriaforprecedencecouldbeestablishedbasedonuserneedsandcontextsofusefortheapplic
ation.Forexample,iftherearemasterandsecondaryaccounts,changesmadebythemasteruser
wouldoverridechangesbythesecondaryaccountusers.Preventingdatacorruptionorlostwor
kwhenaccessingthesamedatafrommorethanoneUIcanbecritical.Therearefewsituationsmor
edishearteningandfrustratingforthetypicaluserwhohasjustspenttimeandeffortaccompl
ishingatask,onlytoseeitlost.Worseyetiswhenauserthinkshehadsuccessfullyfinishedat
ask,thanfindingoutlaterthathedidnot.Forexample,acaseworkerinthefieldcallsintoana
utomatedsystemtomakeupdatestoapatientsfileimmediatelyafteranin-homevisit.Shemakes
changesandhangsup,believingthatthefilehasbeenupdated.Meanwhile,herprocrastinatin
gcoworkerisattheirofficecomputerandjustgettingaroundtocompletingupdatesfromthepr
eviousweeksvisittothesamepatient.Thecoworkerschangesaremadelaterandoverwritethecha
ngesofthefirstcaseworker.Inthiscase,alockoutsystemtopreventaccessingthesamerecor
datthesametimecouldpreventthisscenario,oranapplicationthatcouldtrackahistoryofch
angesandtheindividualswhomadethemcouldhelp.Althoughmostofthesescenarioshavebeena
boutpreventingnegativeconse-quencesofauserorusersaccessingtheapplicationthroughm
ultipleUIs,MMUIscanbebeneficial,iftakenadvantageof,fortheappropriatetasks.Auserm
aywanttheabilitytorepeatedlyswitchbackandforthbetweentheUImodalitiestomorequickl
yoreasilyaccomplishatask.Forexample,althoughtheprimarymethodforsettingupavoicema
ilapplicationisasetupwizardonawebsite,itmaybemuchmoreconvenientfortheusertorecor
dgreetingsusingaphoneinterfaceattheappropriatetime.Inthiscase,accomplishingtheta
sk(settingupthevoicemailinterface)usestheappropriateinterfacemodalitytofacilitat
ethecomponentsofthetask.
11MultimodeInterfaces:TwoorMoreInterfaces362

AnothermajortechnologicalconsiderationforMMUIapplicationsistheunderlyingsoftware
andsystemsthatenabletheUI.AlthoughnotdirectlyrelatedtoUIdesign,thesupportinginfr
astructurecanhaveamajorimpactonthequalityofuserinteractionwiththeapplication.Thi
sisapracticalconsiderationforanyUIdesign,butitisevenmoreimportantforMMUIs.Thisis
becausethereisafargreaterpossibilityforadisjointeduserexperienceifanMMUIiscreate
dbystitch-ingtogetherseveralindependentlydevelopedUIstocreatethemultimodalapplication.IftherequirementtodevelopthenecessaryinfrastructuretosupportMMUIsisadhere

dto,thenitislikelythattheresultingMMUIswillbecharacter-izedbydescriptionslikeseam
less,fullyintegrated,andconverged.Ifnot,thenthesetermswillonlybebuzzwordsforapplicati
onsthatwillultimatelydisappointusers.Consequently,goodMMUIdesignisinextricablyli
nkedtounderstandingtheabilityoftheunderlyingsystemstoaggregate,store,andprovided
atatomultipleUIsinaconsistent,timely,anddevice-independentmanner.Tohelpunderstan
dthesecomplexrelationshipsamongusers,contextsofuse,multipleUIs,andunderlyinginfr
astructure,Figure11.1depictsanexampleofhowanapplicationthatusesmultipleUIsmightb
eimplemented.TheuniquenessofdesigninganMMUIisfactoredinfromthebeginningwithconsi
derationoftheuser(althoughsometimestherearemultipleusers)andtheirdifferentcontex
tsofuseandtasks.Becauseofthis,theremaybeaneedtointeractwithmultipledevicesand,co
nsequently,multipleUIs,althoughsomecontextsofusemaylimitsuchinteraction.Thesedev
icesinteractwithvarioussoftwareappli-cationstoprovideservicefunctionality.Depend
ingontheimplementation,thesameapplicationmaysupportonlyonedevice/UI,butitcouldsu
pportmorethanone.Iftherearemultipleapplications,theirabilitytocommunicatewitheac
hotherisafactorintheoveralllevelofintegrationthattheuserexperiences.Finally,appl
icationsnetworkwithback-endsystems(e.g.,databases)tocarryoutuserrequests.Again,s
omeimplementationsmayusemultipleback-endsys-tems,andtheirabilitytosynchronizedat
aalsowillhaveanimpactontheuser.11.3CURRENTIMPLEMENTATIONSOFTHEINTERFACEThepreval
enceofMMUIsismuchlessthansingle-modeUIs.Seffah,Forbrig,andJavahery(2004)suggestt
hatthismaybeduetothedifficultyofintegratingtheoverwhelmingnumberoftechnological,p
sychological,andsociologicalfactorsneededtocreateagoodapplication.Butwhilenotaspr
evalentassingle-modalityUIs,theuseofmorethanoneinterfacemodalitytocompletethesam
etaskorsetoftasksismorecommonthanonemightthink.Forexample,checkingontheshowtimes
foramoviecaneasilybedoneoverthephone(usingspeechoreven
11.3CurrentImplementationsoftheInterface363

touch-tone),ontheWeb,andusingasmall-screendevicelikeacellphoneorPDA.Infact,infor
mationretrievaliscurrentlyoneofthemostcommonusesofMMUIs.11.3.1InformationPortals
Webportals,suchasYahoo!,arehubsontheInternetthatcollectandorganizesomeofthevasta
rrayofinformationsourcesandstreamsavailabletowebusers(Figure11.2).Overthephone,v
oiceportalsperformmuchthesamefunction,offeringuserstheabilitytohearinformationab
outnews,stocks,weather,sportsscores,movielistings,phonenumberlookup,andinformati
ononothertopicsthatauserwouldwanttoknowatanytimeorplace.Thelattermotivationis
Context-of-use 1Context-of-use 2User 1User 1User interface 1User interface 2User
interface 3Device 3Device 2Device 1Service application 1Service application 2Ba
ck-end system 2Back-end system 1IntegrationS
y
nchronization
User 2
FIGURE11.1ExampleofanMMUI.Interrelationshipamongusers,contextsofuse,multipleUIs,
andtheunderlyinginfrastructureforanexampleMMUI.
11MultimodeInterfaces:TwoorMoreInterfaces364

oneofthereasonsthatvoiceportalshavetraditionallybeenassociatedwithwire-lessphone
services.However,thelowerinformation-processingbandwidthoverthephonegenerallylen
dsitselftofarfewertopicsinvoiceportals,ascomparedtoWebportals.11.3.2AutomatedSel
f-HelpAlthoughsomecompanies,suchasAmazon.com,haveestablishedtheirbusinesswithlit

tleornodirecthumaninteraction,manytraditionalcompanieswithmajorcallcentershavebe
ensubstitutingautomatedself-helpforthetraditionalcareprovidedbycustomerservicere
presentativesinordertoreducecompanycosts.Therelativelyhighcostofprovidinglive-ag
entcallcentersupportforawidearrayofproductsandservices,especiallyina24/7environm
ent,hasresultedintheuseofautomatedsystemstoprovidesupportfortasksrelatedtobillin
g,payments,technicalsupport,accountinquiries,andsales.
FIGURE11.2TraditionalinformationportalfortheWeb.Theportalorganizesdisparatesourc
esofinformationfortheuser.(CourtesyYahoo!Inc.)
11.3CurrentImplementationsoftheInterface365

Manyoftheearlyautomatedself-helpapplicationsweretouch-toneinteractivevoicerespon
se(IVR)systems(seeChapter7formoredetailsondesigningthiskindofapplication).Therea
sonforthis,simply,wasthatcustomerswerealreadycallingandthesimplestwaytoengagethe
minanautomatedinteractionwastohandletheirtaskoverthephone.Nowadays,manyoftheseta
sksarealsobeingintegratedintothecompanyswebsite.Customerscannowcallorvisitawebsit
etopayabill,orderproductsandservices,gettechnicalsupport,anddootherroutinetasks.
11.3.3AddressBooksUsershaveaddressbooksinmyriadlocationsintheelectronicworld.The
yhavetheminaGUIenvironmentwithintheire-mailprogramsandinstantmessengers.Somearei
ncellphonesforvoicedialingandtextmessaging.OthersareinonlineaccountsontheWebfors
hopping.Thediversetasksthatbenefitfromhavinganaddressbookhavedriventheneedtohave
asingle,synchronizedaddressbookthatcanbeaccessedfromalmostanyendpoint:computer,m
obiledevice,telephone,orTV.11.3.4MessageMailboxesHandlingmessages,suchase-mail,s
hortmessageservice(SMS)text,instantmes-sages(IMs),voicemails,andfaxes,hasbecomea
partofeverydaylife.Messagesallowpeopletokeepincontactevenwhentheyarenotavailable
atthatmoment.Thisneedforubiquity,combinedwithverydifferentoriginsforeachtypeofme
ssage,hasledtotheuseofMMUIstobetterhandleand,moreimportantly,expandtheirusefulne
ssbygivingusersmoreflexibilityinhowandwhentheymessageotherpeople.11.3.5RemoteAcc
esstoServicesInthebusinesssettingmanynetworkmanagershavetoleavetheirdeskstotroubleshoot,butneedtotaketheiraccesstoserviceswiththem.Othersinbusinessneedmobility
inordertodotheirjobs,butneedaccesstothesameresourcesthattheyhaveintheiroffice.An
d,increasingly,homeautomationisprovidingmorecapabilitiesforthehomethatcanberemot
elycontrolledand/ormonitored.Inallofthesecases,theuserhasanormalenvironmentinwhi
charobustsetofapplicationsorservicesisavailabletothem,butalsoneedtomoveawayfroma
wireddevice.Thishasresultedinacorrespondingneedtoremotelyaccesstheseservices.The
usualmediumofinteraction,typicallyaGUIorfull-screenwebsite,isnotportableinthesec
asesandsootherinterfaces,suchasphone-basedIVRsorsmall-screendevices,becomeagooda
lternative.Figure11.3(a)showsacommontaskcheckingtheweatherradarasausermightseeitwh
enusingaPCwithabroadbandconnection.Figure11.3(b)showswhatmightbeseenwhenusingamo
bilewirelessdeviceforthesametask.
11MultimodeInterfaces:TwoorMoreInterfaces366

FIGURE11.3Checkingtheweatherradar.(a)UsingaPCwithabroadbandconnection.Theweather
radarfromtheNationalWeatherServiceonthewebsitehasnumerousoptionsandalargedisplay
.(b)Usingamobilewirelessdevice.TheweatherradarfromtheNationalWeatherServiceformo
biledeviceshasfewoptionsandasmalldisplay.(CourtesyNOAA/NWS.)

11.3.6ApplicationsoftheInterfacetoUniversalDesignandAccessibilityItisworthtaking
sometimetodiscusstheimpactofMMUIsonpeoplewithdisabilities.Havingadisability,whet

herinherent(e.g.,visionorhearingdifficul-ties,dyslexia,etc.)orduetoonescircumstan
ces(e.g.,inabilitytoseesecondarytaskswhiledriving,tooperatesmallcontrolswhilewea
ringgloves,etc.),canpresentbarrierstotheuseofcomputertechnology.However,computer
technol-ogycanbeoneofthegreatenablersforpeopleofallabilities.Whenthedesignhascon
sideredthefullrangeofusercapabilitiesandlimitations,thisiscalleduniversaldesign.
Andthedegreetowhichsomeonecanusethedesignispartofaccessibility.Amajortenetofuniv
ersaldesignandaccessibilityisflexibilityofuserinput/output(ANSI/HFES,2005).Thedi
stinguishingcharacteristicofMMUIsisthat,bytheirverynature,theyoffermultipleinput
/outputmodalitiestotheuser.OnegoodexampleofMMUIshelpingtoachieveamoreuniversalde
signistheuseoftouch-tonecommandsinconjunctionwithaspeech-enabledUI(Fig-ure11.4).
Whetherofferedconcurrentlyasabackup(althoughthisismoreofacombinedinterface,asdis
cussedinChapter12)orasanoptionalsetting,offer-ingthesetwoUItechnologiescangreatl
yimprovetheperformanceforawiderangeofusers.Theuseofspeechrecognitionhelpsthosewi
thacognitivedisabilitywhowouldhavetroublerememberingthemappingoftouch-toneoption
splayedinamenu,whileanotheruserwithlimiteduseofthehands(eitherbecausetheyhave
<says nothing>CallerOkay, which are you calling about: your bill, an order, or t
echnicalsupport?SystemCaller<presses the 3 key>SystemIm sorry, I didnt hear. If yo
ure calling about your bill, say mybill or press one. If its about an order, say orde
r or press two.Otherwise, say technical support or press three.
FIGURE11.4MMUIexample.Thisisabriefcallerexchangewithanautomatedsystemthatdemonst
ratestheuseofbothspeechandtouch-toneinputoptions.
11MultimodeInterfaces:TwoorMoreInterfaces368

adisabilityorarewearingglovesorholdingsomething)canusespeechtoprovidetheinput.In
contrast,offeringtouch-toneinputasanoptionhelpsauserwithaspeechimpairmentoradiff
icultaccent,andprovidesotheruserswhoareinanoisyenvironmentorhaveaneedforprivacy,
theabilitytorespondtotheappli-cation.Offeringbothmodesofinteractionenablesasmany
usersaspossibletointeractwiththeapplicationinthemannerthatworksbestforthem.While
MMUIsshouldnotbethoughtofasameanstocircumventmakingeachUIaccessible,theinclusion
ofmultiplemodalitiesforinteractiongreatlyincreasestheflexibilityforuserstointera
ctinwaysthatmatchtheirparticularcapabilitiesandlimitations.11.4HUMANFACTORSDESIG
NOFTHEINTERFACEBeginningtodesignanapplicationthatusesmorethanoneUImodalityisbase
dontheassumptionthatmorethanoneisneeded.Oftenthisisbasedonabeliefthatbecauseitca
nbedoneitshouldbedone.Aswiththechoiceofaninterfacemodalityandtechnologyforanyapp
lication,theexpectedusersandtheirneeds,tasks,andcontextsofusefirstneedtobeidenti
fiedandevaluated.11.4.1ResearchRequirementsbeforeChoosingtheInterfaceModalityThe
user-centereddesignprocessbeginswithusers,nottechnology.Thisisthecorebeliefofuse
r-centereddesign,butitismentionedherebecausewhendealingwiththemyriadUItechnologi
esandthecomplextechnologicalinfrastructureneededforMMUIs,itisevenmorelikelythatt
hispointwillbelost.ThefirststepinthedesignofMMUIsisthesameasdesigninganyotherUI:
Definetheuserpopulation.Thisprocessdefinesthecharacteristicsofthepoten-tialusers
,includingtheircapabilitiesandlimitationsandtheenvironmentsinwhichtheapplication
isexpectedtobeused.Thisprocessincludesconsideringpeoplewithdisabilities.Oncethet
argetusershavebeendefined,thenuserresearchcanbegin.Whiledoingtheusualneedsandcon
textualanalysis,beawarethatsomeuserneedsmaybespecifictodifferentenvironmentsinwh
ichtheapplicationwillbeusedforexample,checkinge-mailfromtheofficewhileatonesdeskan
dcheckinge-mailonacrowdedsubwaytoandfromtheoffice.Someneedsmaybebasedonthegoalst
hatusersmightwanttoaccomplishusingthepro-posedapplication.
11.4HumanFactorsDesignoftheInterface369

Oncetheneedsarewelldefinedandunderstood,applicationfeaturescanbeadded,removed,or
refined.Matchingthesefeatureswithuserneedscre-atesasetofcommongoalsfortheuserand
theapplication.Next,theusualtaskanalysisshouldbeperformedtodeterminehowusersacco
mplishthesegoalsandtolookforopportunitiestoimproveexistingmethods.Thetaskanal-ys
isshouldalsobeusedtodeterminewhichgoalscanorcannotbeaccom-plishedusingspecificin
terfacemodalities.Forexample,settingupagroupofaddressbookcontactshavingacommonpr
operty,suchasyourdaughterssoccerteam,mayberelativelyeasytodowithavisualinterface,
keyboard,andmouse,butverydifficultorimpossibletodousingatelephonekeypad.However,
itmaybepossibletoeditateammemberse-mailaddressusingspeechrecognition.Thereissomer
esearchintotheareaofdevelopmentconsiderationsparticulartoMMUIs.Forexample,Kray,W
asinger,andKortuem(2004)identifyataxonomyofmanagement(e.g.,conflictresolutionamo
ngusersand/ortasks),technical(e.g.,synchronization),andsocialissues(e.g.,privacy
),butnotethattheyareoftenhardtoclassifyasduetooverlappingcategories.Seffah,Forbrig
,andJavahery(2004)discusstheuseofvariousmodelsfordevelopment,includingtaskmodelba
seddevelopment,UIpatterndrivendevelopment,anddevice-independentdevelopmentlanguag
es.Theyfoundthatpatternmodelsdohelpwithunderstandingthefullcomplexityoftasksandt
heinterrelationshipsparti-culartoMMUIs.Toassistdesign,Paterno(2004)describesanau
thoringenviron-mentforMMUIs.Someofitskeycomponentsaregrouping,relation,ordering,
andhierarchy.WhilethesetechniquesarenotpeculiartoMMUIs,theoverridingthemeisthatt
hedesignanddevelopmentofapplicationsusingMMUIsarecomplexandrequireaplanned,metho
dicalapproach.TopreventbecomingoverwhelmedwhendesigninganddevelopinganMMUI,thete
chniqueusedislessimportantthanwhetheritisthorough.Criticallyevaluatingthesepoint
swillprovideabasisforaninformeddecisionaboutwhatmodalityandassociatedtechnologyc
anbestenabletheuserstoperformtheirtasksintheexpectedsituations.11.4.2ChoosingOne
orMoreUserInterfaceModalitiesOncebackgroundresearchandanalysishavebeenconducted,
thenitisappropri-atetoselectofoneormoreUImodalities.ThequestionsinTable11.1helpg
uidethedecisionofwhethermultipleUIsareneeded,althoughtheydonotmakeupanexhaustive
list.Iftheconclusionisthatmultiplemodalitiesarewarranted,thenchoosingwhichmodali
tiestouseisthenextstep.
11MultimodeInterfaces:TwoorMoreInterfaces370

TABLE11.1QuestionstoAidinChoosinganMMUIEvaluationQuestionExplanationExampleCanmo
rethanonemodalitybecost-justified?Designinganddevelopingmorethanoneuserinterface
iscostly.Cantheaddedexpensebebalancedwithagreatercustomerbase,greateruseoftheser
vice,increasedcustomerretention,andsoforth?(SeeBiasandMayhew[2005].)Somecustomer
swanttopaytheirbillovertheWeb,butotherscalltopaytheirbillandcoulduseanIVR.Offeri
ngbothwillautomatemorebillpaymentsoverall,savingmoneyandincreasingrevenuecollect
ion.Canallofthetasksbeperformedinonlyonemodality?Basedonthetaskanalysis,canallof
thetasksbeadequatelyperformedusingonlyonemodality?Ifso,theremaynotbeacompellingr
easontousemultiplemodalities.Allvoicemailtaskscanbedoneoverthephoneusingtouch-to
nes.Wouldmorethanoneinterfacemodalitydirectlyfulfilluserneeds?Sometimesoneuserin
terfacemodalitycannotsupportalloftheneedsandtasksoftheuserpopulation.Sendingalon
ge-mailinfestivecolorstoanentirelistofpartyguestswithnumerousdetailsandamapmayca
llforawebsiteorGUI.Alternatively,sendingaquicke-mailtoafriendlettingthemknowyoua
restuckintrafficandwillbe15minuteslatemaycallforsomethingmobileandsimple,likeace
llphoneorPDAUI.Dothecontextsofuseand/orenvironmentsnecessitatedifferentmodalitie
s?Whenusersperformthesametask(orsetoftasks)indifferentsituationstheymaynothaveth
esameneeds,sothesamemodalitymaynotwork,dependingonwheretheyareandwhattheyaredoin
g.AtooltohelpcustomersresolveproblemswiththeirInternetserviceoffersbothweb-based
helpforthosewhocanstillaccesstheInternetandover-the-phonehelpforthosewhoseIntern
etconnectionsaretotallydown.Willthenewapplicationsfunctionalitybecomposedofmyriad
taskscurrentlyperformedbydifferentmodalities?Aretheuserneedsandtaskscurrentlyhan
dledbydifferentapplicationsusingdifferentmodalitiesthatuserswillnoteasilygiveup?
Anapplicationthatprovidesmovielistingsusesawebuserinterfaceandatelephoneuserinte
rfacebecausesomeusershavealwaysusedoneoftheseanddonotwanttobeforcedtochange.(Con
tinued)

11.4HumanFactorsDesignoftheInterface371

11.4.3SelectingAppropriateUserInterfaceModalitiesSometimesthechoiceofUImodalitym
aybeclear-cut,andsometimesfurtheranalysisisnecessarytoensurethattherighttoolisbe
ingusedforthejob.Thissectionprovidessomeoftherelativestrengthsandweaknessesofthe
UImodal-ities(Table11.2)thataretypicallyusedtogethertoformMMUIs.Pleaseseetheothe
rchaptersofthebookandthereferencesattheendofthischapterforamoreexhaustivelistoft
hestrengthsandweaknessesofthevariousUImodalities.Thereareotherstrengthsandweakne
ssesassociatedwitheachmodality,butTable11.2shouldprovideagoodstartingpointforthe
trade-offsinvolvedwhendecidingwhichareappropriateforaparticularapplication.11.4.
4TaskAllocationtoMultipleModalitiesOncetheappropriateUImodalitieshavebeenselecte
dfortheapplication,thenextstepistoassignwhichtaskswillbeperformedwithwhichmodali
ty.Justashumanfactorshavetraditionallyperformedfunctionallocation(seeWright,Dear
den,&Fields,2000;Kantowitz&Sorkin,1987)forthehumanoperatorandmachines,taskalloca
tionamongtheselectedUImodalitiesneedstobe
TABLE11.1ContdEvaluationQuestionExplanationExampleAretheuserssodiversesoastowarra
ntmultiplemodalities?Someapplicationsarenarrowlyfocusedwithwhattheydoandsotheiru
serstendtobealmostasnarrowlyfocused.However,otherapplicationshavebroadappealandt
heirusersarediverse.Anaddressbookisusedbysomepeopleonlyinconjunctionwithsendinge
-mailstoaclosecircleoffamilyandfriends.Othersusetheiraddressbooktokeeptrackofall
businessacquaintances,includingmultiplephonenumbers,e-mailaddresses,streetaddres
ses,personaldetails,andsoforth.Dothecapabilitiesandlimitationsoftheuserspreclude
usinganymodality?Willuserswithdisabilities,eitherinherentorbasedontheirsituation
,havetroubleusingoneormoreofthemodalities?Ifso,aretheothermodalitiesaccessibleto
them?Someonewithhandtremorsorsomeonewhoisdrivingcaninteractusingspeechratherthan
pressingkeysonaphonetocheckmessages.
11MultimodeInterfaces:TwoorMoreInterfaces372

performed.Thisallocationshouldbebasedontheneedsandtasksoftheusers,thereasonsbehi
ndselectingmultiplemodalities,andthestrengthsandweak-nessesofthosemodalitiestofu
lfilltheneedsoftheusers.Inmostcases,alltaskswillbeassignedtoeachoftheUImodalitie
s.Themostimportantbenefitofthispracticeisthatusershavefullchoiceofwhichmodalityt
heycanuseforanytaskthattheapplicationcanperform.Thiscanbeveryimportantforunivers
aldesignobjectives.Italsoeliminatestheneedforuserstorememberwhetheraspecificfunc
tionissupportedwithinthecur-rentUIandavoidstheaccompanyingconfusionthatarisesifa
userisinthemid-dleofataskandthenrealizesthatheneedstostopandswitchmodalities.Ani
llustrationofthismightbeabillpaymentapplicationthatallowspaymentwithacreditcard,
debitcard,checkingaccount,orgiftcard.Ifbothatouch-toneIVRandawebsitepermitallfor
msofpayment,theuserisfreetochoosetocallorgoonlinetopayabill.However,iftheIVRcann
otacceptgiftcards,thentheuser
TABLE11.2StrengthsandWeaknessesofUIModalitiesUserInterfaceStrengthsWeaknessesGUI
/WebUsesthehigh-bandwidthvisualchannelwithalargescreen.Usuallynottimedependent.C
anprovidemultimediainteraction.HardwareassociatedwiththeseUIsusuallyisnoteasilyp
ortable.Usuallyrequiresvisionandmanualdexterityfortheusertointeractwithit,unless
designedwithaccessibilityinmind.Small-screendevicesEasilyportable.Typicallyenabl
edforaudioinputandoutput.Limitedscreenrealestatetoworkwith.Inputmethodsareusuall
ymoredifficult.Smallsizemakesitdifficulttoseeandmanipulate.Touch-toneIVRRecognit
ionofuserinputisnearlyperfect.Notsusceptibletobackgroundnoise(Paterno,2004).Lend
sitselftosimplifiedmenuchoicesandlineartasks(Lee&Lai,2005).Inputoptionsareconstr
ainedandhavetobemappedtoarbitrarynumbers.Enteringcharactersotherthannumeralsisdi

fficult.Contextsofuseordisabilitycanpreventtheuseofhandsforinput.Speech-enabledI
VRAnaturalmeansofcommunicationformostusers(Cohenetal.,2005).Betteratperformingno
nlineartasks(Lee&Lai,2005).Hands-freeandgoodforwhenthevisualchannelisoverloaded(Pa
terno,2004).Recognitionofuserinputisnot100%.Susceptibletonoisyenvironments(Cohen
,Giangola,&Balogh,2005).Notalluserscanproducerecognizablespeech.
11.4HumanFactorsDesignoftheInterface373

maycall,enterhisaccountnumber,theamounthewouldliketopay,andonlythenfindoutthatth
egiftcardhewantedtousewillnotworkandhemusthangupandvisittheWeb.Althoughtherearem
anyadvantagestoimplementingallfunctionsinalloftheUIs,thisstrategyisoftencostly.I
naddition,therearetasksthatmaynotbeappropri-ateforeveryUImodality.Furthermore,in
cludingallfeaturesfromGUIsandwebsites(i.e.,highinformation-processingbandwidthmo
dalities[Wickens,1992])intosmall-screendevicesorIVRsthatuseaudio(i.e.,lowerinfor
mation-processingmodes)mayresultinuseroverloadinthelatter.Figure11.3(a)and(b)(on
page367)illustratethispoint.Whiletheweathermapforthefullwebsitehascontrolstoaddo
rremovenumerousfeatures,theweathermapformobiledevicespickedonlythosethatareneede
dfortheessentialtaskandremovedthecustomizationcontrols.Whateverthechoice,itmustb
emadeduringtheearlystagesofanalysisanddesignandthentestedforvalidationasaworkabl
esolution.11.4.5PerformDetailedDesign,Test,andIterateDesignThelaststepsinthedesi
gnprocessforMMUIsarenotmuchdifferentthaninanyotheruser-centereddesignprocess.The
inputsfromtheprevioushumanfactorsactivitiesprovidethebasisfordetailedUIdesign,th
edesignmustbetestedwithrepresentativeusers,andthedesigniteratedbasedonuserinput.
Whilethestepsmaybethesame,considerableeffortmustbedevotedtothesesteps.Onespecial
considerationduringthedesignprocessforMMUIsisthat,totheextentpossible,allUIsshou
ldbedesignedatthesametime.DoingthishelpstoensurethatthereisconsistencyamongtheUI
sfromtheverybeginning.Eachdesignshouldtakeintoaccountthespecificcapabilitiesandco
nstraintsofthedevicewhilemaintainingcross-platformconsistencyanduniversalusabili
ty(Seffah,Forbrig,&Javahery,2004).Inaddition,thelearningprocessthatoccurswhiledes
igningtaskflowsforoneUIcanbetransferredtotheotherUIs.Concur-rentdesignalsohelpst
oensurethattheuserneedssupportedbyoneUIdonotmarginalizetheneedssupportedbytheoth
erUIs.Ifthiscannotbedone,thenthepredominantUIshouldbedesignedfirst.Thepredominan
tUIcanbedeter-minedbasedonwhichUIwillsupportthehighest-frequencytasksandanyother
tasksthatarecriticaltothesuccessoftheapplication(e.g.,setup).Whileperformingconc
urrentdesign,Seffah,Forbrig,andJavahery(2004)proposedthefollowingusabilityfactor
sspecifictoMMUIs:FVariousinteractionstylesamongUIsFInformationscalability/adapta
bilitytovariousdevicesandUIsFTaskstructureadaptabilityFConsistentapplicationbeha
vior
11MultimodeInterfaces:TwoorMoreInterfaces374

FUserawarenessoftrade-offsduetotechnicalcapabilitiesandconstraintsFApplicationav
ailabilityFOverallinter-usabilityThelastpointisoneSeffah,Forbrig,andJavahery(2004)
definedashorizontalusability.Eithernameissuitable,butthekeypointisthatitisadimensi
onspe-cifictoMMUIs.ItisimportanttounderstandthatwhiletheusabilityoftheUIsmaybesu
fficientseparately,totrulyknowtheusabilityofanMMUI,allUIsmustbeassessedtogethert
odeterminetheirintegrateduserfriendliness.AnotherspecialconsiderationforMMUIsist
hatalltheUIsshouldbejointlyuser-tested.Forthesestudies,exposingallparticipantsto
allofthedesignsprovidesthemostflexibility,allowingthesameparticipanttomakedirect
comparisonsamongthevariousUIsbeingtested.ThisalsoallowsthedesignertoseewhichUI,i
fany,isinitiallypickedbytheusersortoseeifuserswillspontaneouslyabandononeoftheUI
sforanotherwhencompletingtasks.Beyondtheusualbenefitsofusertesting(Hix&Hartson,1
993),concurrentlytestingtheUIshelpsinseveralways.Itcanbeusedtovalidatethetaskall
ocationthatwasperformed,eitherbyevaluatingwhetherthesametaskinallUIscanbeaccompl

ishedorwhetherthechoicetoallocateatasktoonlyoneUIisappropriateortoolimiting.Depe
ndingonthereasonforneedingmorethanoneUI,jointtest-ingprovidesrealisticscenariosf
orparticipants.Inmanycases,userpreferencesforaccomplishingtaskswithparticularUIs
canbeveryhelpfulwhenconsideringapplicationdefaultsandfurtherunderstandinguserbeh
avior.Finally,justasconcurrentlydesigningtheUIsoffersonetheabilitytolearnfromthe
experiencewithonemodalityandapplylessonstotheothermodalities,thesameistrueforthe
datacollectedduringusertesting.Havingparticipantsper-formthesametaskinallofthemo
dalitiesallowsonetoobserveinconsistenciesinthedesign,aswellasthestrengthsandweak
nessesoftheUImodalitiesthemselves.11.4.6ASpecialCase:AddingaModalitytoanExisting
ApplicationSometimesthedesignprocessforamultimodeapplicationdoesnotstartfromtheb
eginning.Therearetimeswhenanapplicationhasalreadybeendevelopedthatusesonemodalit
y,butwhereadecisionhasbeenmadetoaddanothermodeofinteraction.Asmentionedbefore,th
edecisionmaybebasedonamisguidedbeliefthatbecauseanotherUIcanbedone,itisagoodidea
todoit.OthertimesthereasonsforaddinganotherUIarebasedonthetypesofdatamentionedin
thepre-vioussection:usersneeds,theircontextsofuse,and/ortheenvironmentsinwhichthe
yarecurrentlyusingtheapplication.Oneoftheopportunitiesinthissituationthatmaynotb
easavailablewhendesigningfromscratchisthattheusers,theircontextsofuse,andtasksar
e
11.4HumanFactorsDesignoftheInterface375

alreadyavailableforstudy.Atsomepointafterdeploymentofanapplication,stud-iesofthe
applicationmayuncoversignificantgapsinitsusefulnesstotheusersoritsusability.Anal
yzinghowtoaddresstheseproblemsmayshowthatiterativeimprovementstotheoriginalUIare
sufficient.Othertimes,however,theoriginalmodalitymaynotbethebestchoiceforthebest
solutionsandtheselectionofanothermodalityisappropriate.Butaswithdesigningallthem
odalitiesinitially,thedecisiontouseanothermodalityandwhichonetouseshouldbebasedond
ataabouttheusers,whattheyneed,andhowtheyperformtheirtasks.Taskallocationamongmod
alitiesforanMMUIdesignedfromthebeginningissomewhatdifferentthanwhenaddinganother
modalitytoanexistingapplication.ThecruxofthedifferenceisthatoneoftheUIshasalread
ybeendesignedandthushasalreadybeenallocateditstasks.Theprincipaldesignproblemfor
thenewUIisdeterminingwhatothertasksshouldbeincludedinthenewUI.Thereisnodefinitea
nswer,buttimeandcostconsiderationsusuallymeanthatonlythetasksthatwouldmakethenec
essaryimprovementstotheoriginalapplicationareincludedinthenewmodality.Inaddition
,tasksallocatedtotheoriginalUIcouldbeaddedorremoved,butusuallynotatthispoint.Alt
ernatively,acompletetaskallocationcanbeperformed,treatingtheinclusionofanewUIasa
chancetoperformacom-pleteredesign.Thisisusuallymoretimeconsumingandcostly,butsom
etimesathoroughredesignisneededtoachievethedesiredimprovements.Theothermajordiff
erencebetweendevelopinganMMUIfromscratchandaddinganothermodalityafterapplication
deploymentisthatjointdesignisnotpossible.Inthiscase,thenewUIshouldbedesignedinli
ghtoftheexistingmodality,understandingthattheoriginalUIprobablywillnotfundamenta
llychangebutalsorecognizingthatmanyusersareaccustomedtotheexistinginterfaceandth
atanychangestothatinterfacearelikelytohavenegativeconsequences.Jointtestingofthe
originalandnewUIsshouldtakeplaceinthesamewayasforatotallynewapplication,butwitho
necaveat:Bothexistingusersandnewuserswithnoknowledgeoftheapplicationshouldbepart
icipantsinthesestudies.Perspectivesofbothusergroupsareveryimportantbecausewhilei
tisimportantthatcurrentcustomersarenotalienatedbythenewdesign,itisalsocrucialtha
tnewusersareattractedtotheimprovedapplication.Whentestingcurrentusers,payattenti
ontoanynegativetransfertothenewUIfromtheirexperienceswiththeexistingone,especial
lyregardingconsistency.Withuserswhohavenoknowledgeoftheapplica-tionneworoldconcent
rateonwhethertheadditionofthenewUImeetstherequirementsthatwereestablishedforaddi
ngthenewmodality.Thisearlytestingshouldbedesignedtodecidewhetherthetaskallocatio
niscorrect,especiallyiftheminimalistapproachwastakenduetotimeandcost.Moreover,it
willalsopointoutwhethertheoriginalUIneedstobesignificantlychangedtoaccommodateth
enewmodality.Finally,asisthecasewithdesigningfromscratch,thistesting,redesign,an
dretestingshouldcontinueuntilasatisfac-torydesignhasbeenachieved.

11MultimodeInterfaces:TwoorMoreInterfaces376

11.4.7SummaryofDesignStepsforMultimodeUserInterfacesStepsforanalyzinganddesignin
gmultipleUImodalitiesare1.Definetheuserpopulation,includingpeoplewithdisabilitie
s.2.Determinecapabilitiesandlimitationsofusers.3.Collectuserneeds,includingspeci
ficcontextsofuseandenvironments.4.Matchuserneedswithsystemfunctionality.5.Perfor
mtaskanalysis.6.EvaluatetheneedformorethanoneUI.7.Selectthespecificinterfacemoda
lity(s)basedonstrengthsandweaknessesofeach.8.Assigntasks/functionalitytoallmodal
itiesoronlysomeofthem.9.ConcurrentlyperformthedetaileddesignoftheUIs.10.Testwith
representativeusers,collectingdataaboutusability/accessibilityfortasksandmodalit
ies.11.Iteratedesignandretestasneeded.11.5TECHNIQUESFORTESTINGTHEINTERFACEBecaus
ethischapterdoesnotdiscussauniqueinterfacebutratheracombinationofinterfaces,thet
estingtechniquesdescribedherewillfocusontestinganappli-cationthatusesmultipleUIs
andhowtotestwhetherthecorrectUIsarebeingusedratherthanhowtotestaparticularmodali
ty.Seeotherchaptersinthisbookforadiscussionofnon-GUI/Webtestingtechniques,orther
eferencesattheendofthischapterfordiscussionsoftestingGUI/WebUIs.WhenevaluatingMM
UIs,manyofthestandardusabilityevaluationtechni-quesareused.Earlytestingwillgener
allybedonewithprototypesoftheapplica-tionandfewerparticipants;thefocusisonfindin
gwaystomaketheUIdesignsbetter.Latertestingwillgenerallybewithafullydevelopedappl
icationandhavemoreparticipants;thegoalherewillbetoironoutdetailsandbegintopredic
toveralluserperformance.Inbothcases,comparativetechniquesareespeciallyimportant.
Aswithanyapplication,themostimportanttestingisthatdoneearlyanditeratively,withre
presentativeusers.11.5.1EarlyTestingTheprimarygoalsofearlytestingaretovalidateth
echoiceofmodalitiesandanytaskallocationthathasbeendone.Thefirstroundoftestingide
allyshouldbecon-ductedwhilethedetailedUIdesignprocessisunderway.Atthisstage,test
ing
11.5TechniquesforTestingtheInterface377

withthrowaway(orexploratory)prototypesisbestbecausetheycanbebuiltandchangedquickly
,evenduringtesting.Forvisualinterfaces,theseprototypesareusuallywireframesorsimple
clickableprototypescreatedinHTMLorrapidpro-totypinglanguages.Forspeechrecognitio
nsystems,aWizardofOztechniquecanbeveryusefulforcreatingquickmock-upsofearlydesig
ns(Cohen,Giangola,&Balogh,2005).(SeealsoChapter6.)Inthissameway,eventouch-toneIV
Rscanbeprototyped(i.e.,byhavingthecallersaythekeytheywanttopress).Theessentialth
emeforthistestingisthattheearlierrepresentativeusershaveachancetoprovidemeaningf
ul,validinputintothedevelopmentprocess.Theclosertheseearlyinterfacesresemblethet
argetUIdesign,thegreaterthelikelihoodthattheproductwillbedesignedwithfeweriterat
ions.Thisisespe-ciallyimportantforMMUIsbecausewithmultipleUIs,anyoneUIwilllikely
getfewerresourcestomakechangesthanifitwereastand-aloneapplication.Inaddition,MMU
Istypicallyhaveamorecomplexsupportinfrastructure,whoseowndesigncannotchangeeasil
y.Becausefinedetailsarenotimportantatthisstage,asmallsamplesizeofrep-resentative
usersisappropriate.Thetestshouldexposealloftheparticipantstoeachofthedesignalter
nativessodirectcomparisonscanbemade.Thetasksforparticipantsshouldbeselectedtofoc
usontheonesmostrelevanttotheongoingdetaileddesign,whichincludethefollowing:FMost
frequentlyperformedFMostcriticaltotheoverallsuccessoftheapplicationFMostfullyexp
lorethechoiceofaparticularmodalityFSpecificallyallocatedtoonlyoneoftheUIsFMostli
kelytousemorethanonemodalityatatimeorconsecutivelyTestingthemostfrequentandcriti
caltasksiscommoninusabilitytestingandwillnotbediscussedhere(Hix&Hartson,1993).Wh
entestingMMUIs,thegoalshouldbetochooseasampleoftasksthatallowstheexperimentertoo
bservethebehaviorwitheachoftheUImodalities.Notonlydoesthisprovideperfor-mancedat
aforthetasks,butitalsoprovidestheopportunitytogetabettersenseofwhatdoesanddoesno
tworkforthatmodalityintheapplication,aswellasachancetoaskparticipantswhichmodali
tytheypreferredandwhy.Tasksthathavebeenallocatedtoonlyonemodality,especiallyifth

osearetasksthatcanonlybeperformedusingthatmodality,shouldbethoroughlytested.Thes
etasksareincludedbecausethereisnoalternativemodalityintheapplica-tionforthemandt
hereforetheyabsolutelymustworkinthechosenmodality.Ifnot,oneormoreoftheothermodal
itiesneedstobeconsideredforthetask.Tasksthatrequireusingmorethanonemodalityatati
meorconsecutivelymayormaynotberelevanttoaparticularapplication.Sometimesthereare
tasksthatmayrequireausertointeractwithmorethanonemodalityatthesametime(e.g.,regi
steringasmall-screendevicewiththeapplicationwhile
11MultimodeInterfaces:TwoorMoreInterfaces378

loggedintoaGUI/WebUI)orconsecutively(e.g.,oncesetupusingatelephoneUIreachesalogi
calendpoint,theusermayberequiredtousetheWebtofinishtheinstallation).Ifthisisthec
ase,thenthesetasksshouldbetestedearlywiththebestprototypesavailable(evenifthestu
dyparticipantsareaskedtomakebelieveaboutsomethings)becausethesetaskscanbeverycompl
exandthetransitionbetweenmodalitiesmaynotbeclear.Itisimportanttodeterminewhether
theinteractionsinbothmodalitiesareconsistentwithusersexpecta-tionsandthattheUIsin
bothmodalitiesareconsistentandmeettherequire-mentsforgoodusability.Ifthisisnotth
ecase,thenoneorbothinterfacesmayneedtoberedesigned,andthisshouldbedoneasearlyint
hedesignpro-cessaspossible.Basedontheresultsofearlytesting,itisimportanttoapplyt
heresultstotheongoingdesignprocess.Adding,changing,orremovingfunctionalitymayber
equired,aswellasrevisingtheUIdesigns.Atthispoint,evenstartingoverandtryinganewap
proachmaybeappropriate.11.5.2LaterTestingAfterearlytestinghashelpedshapethedesig
nsoftheUIs,aworkabledesignshouldbereadytobebuilt.Testingonasemi-functioningsyste
meitheranevo-lutionaryprototypeoranearlyalphaversionoftheapplicationshouldtakeplac
eassoonaspossible.Whenthissystemisready,thenextmajorroundoftestingshouldtakeplac
e.Therearetwomajorgoalsforthisroundoftesting:validatingtheUIdesignsthatwerebased
onearlyusertestingandtestingtasksthatcouldnotbeeasilyoraccuratelytestedwithlower
-fidelityprototypes.Again,standardusabilitytechni-quesareused,althoughmorepartic
ipantsareprobablywarrantedhereastheneedgrowstobecomemoreconfidentwiththeresults.
Thesametypeoftasksthatwereusedinearlytestingarestillapplicable,especiallythehigh
-frequencyandcriticaltasks.Tasksforexploringamodalityandthoseassignedtoasinglemo
dalityarelesscriticalbecausethatpartofthedesignneedstobemostlysettledatthispoint
butstillshouldbevalidated.SomeofthenewtasksimportantinthisroundoftestingincludeF
Tasksthatareheavilydependentonback-endsystemsorotherdependenciesthatmadethemdiff
icultorunreliabletoevaluateinearlytestingFTasksthatcanbeperformedonmorethanoneUI
Theformerisfortasksthatcouldnotadequatelybecapturedinearlytestingwithlower-fidel
ityprototypes.InmanycaseswithMMUIs,theuserinteractionisheavilydependentonintegra
tedsystemsand/ordynamicdatathatcannotalwaysbesufficientlyreplicatedinearlytestin
g.Thisisthetimetotestuserswiththosetasks.
11.5TechniquesforTestingtheInterface379

Thisisalsowhereinthetestingprocesstodeterminetherelativeperfor-manceofthesametas
kwithmultipleUIs.Thiscanbeimportantwhenassessingtherelativestrengthsandweaknesse
softhevariousUIsoftheapplication,forsettingapplicationdefaults,andforprojectingu
sagepatterns.Atestduringwhichallparticipantsexperiencealloftheexperimentaloption
sisstillappropriateatthispoint.Thistimethedirectcomparisonsareusuallymadebetween
performanceonthesametaskwithdifferentmodalities,althoughifalterna-tivedesignsare
availableatthisstage,acomparativeevaluationisalsouseful.How-ever,iftherearenumer
oustaskswithseveralUIs,atestinwhichvariousgroupsofparticipantsperformdiversetask
swithdifferentmodalitiesmaybetheonlypracticalexperimentaldesignatthispoint.Withm
ultipleUImodalitiestobefactoredin,thetestsizecanquicklygrowoutofhandwhenallparti
cipantsdoallofthetasks.Uptothispointtheusertestingcouldbedonejustaboutanywherebe
causethefocuswasonvalidatingthefundamentalsoftheUIdesigns.Butoftentheneedsandtas

ksoftheusersthatdrovetheneedformultipleUImodalitiesareunusualand/orvaried.Inthes
ecasesamorenaturalisticsettingmaybeneces-sary.Thecontextofuseandtheenvironmentsi
nwhichtheusersareexpectedtousetheapplicationneedtobeconsideredaspartofthetesting
methodology.Forexample,ifaUIforasmall-screendeviceispartoftheapplicationbecauseo
fastrongneedformobility/ubiquity,thensomeofthetestingshouldbecon-ductedinanenvir
onmentthatreplicatestheexpectedusesthatoriginallyrequiredtheuseofmultipleUIs.11.
5.3InclusionofPeoplewithDisabilitiesinUserTestingDuringallphasesofdesignandtesti
ngtheinclusionoftheneedsandinputofpeo-plewithdisabilitiesisimportant.Overlooking
thispartofdesignandtestingisoftenaccompaniedbytheexcuseoftryingtogetmostoftheUIr
ightfornondis-abledusers,withthepromiseofmakingthenecessaryadjustmentstoaccommodatethedisableduserlater.ThisbeliefisevenlesstruewithMMUIsbecausethemultiplemode
softenareaboontopeoplewithdisabilities.Moreover,MMUIstendtobeusedmoreoftenindive
rsecontextsofuseandenvironments,creatingsituationsthatarefunctionallyequivalentt
oadisability(e.g.,notbeingabletohearbecauseofanoisyenvironment).Peoplewithdisabi
litiesnotonlyprovidefeedbackduringtestingaboutaccessibilityfeatures,butfrequentl
yaremoresusceptibletomarginaldesignsandcanuncoverusabilityproblemsthatwouldnotne
cessarilybeuncoveredwithasmallsamplesize.Assuch,itisvaluabletoconsidertheseneeds
duringdesign,butitisalsoimportanttounderstandthattheimprovementstothedesignresul
tingfromusertestingwiththisgroupimprovetheoveralluser-centeredprocess.
11MultimodeInterfaces:TwoorMoreInterfaces380

11.6DESIGNGUIDELINESMultimodeUIsareextremelyvaried.Becausetheyarecomposedofmoret
hanoneUI,theyaregenerallymorecomplexandyethaveallofthesameissuesastheircomponent
UIs.Thesumcanbegreaterthanitsindividualparts,butonlyifdoneright.Althoughthereisn
orecipeforhowtodesignagoodMMUI,thissectionlistsanddescribesselectedguidelinesonw
aystoimprovethechanceforcreatingauseful,usable,andaccessibleone.11.6.1OnlyChoose
MMUIsIfThereIsaCompelling,Real,andBeneficialNeedHavingmultipleUImodalitiesforthe
sameapplicationmakesalmosteverythingaboutthedesignanddevelopmentmoredifficult,co
mplex,andcostly.AndiftheUIdesignerisnotcareful,thisburdencanbetransferredtotheus
ers.ThegoalofdesignistolettheusersbenefitfromthechoiceofmultipleUImodalities,but
notletthemsufferbecauseofit.Todothat,firstbejudiciousaboutthedecisiontousemultip
leUImodalitiesandaboutwhicharechosen.IfthedatasupporttheneedtohaveanMMUI(i.e.,ba
sedontheuserpopulationandanalysisoftheirneedsandcontextsofuse)isstrongenough,the
ntheUIdesignshouldfollowthatcourse.Example:Adiverseuserpopulation,includingindiv
idualswithvisionandhearingloss,needtobeabletofindouttheshowtimesfornumerousmovie
satmultipletheaters,whethertheyareathome,work,orinthecar.Basedonthesevariedusers
andcontextsofuse,webandspeechrec-ognitionIVRUIsareselected.11.6.2ChooseModalitie
sWhoseStrengthsandWeaknessesMatchtheNeedsandTasksoftheIntendedUserPopulationThee
ntirepurposeofhavingmorethanoneUImodalityforanapplicationistotakeadvantageofthec
apabilitiesthatmultiplemodalitiesprovide.Consequently,usersneedsandthetaskstheype
rformthatcannotadequatelybehandledbyonemodalitymustbeassignedtoamodalitywhosecap
abilitiesareagoodfit.Example:Foramovieshowtimesapplication,aspeechrecognitionIVR
ischosenovertouch-tonebecausetherearenumerousmovienames,sosayingthenameofthemovi
eismucheasierthanselectingfrommultiplehierarchicalmenus.
11.6DesignGuidelines381

11.6.3ConcurrentlyDesignUserInterfacesAsdiscussedinSection11.4,theanalysisthatah
umanfactorsspecialistconductsbeforethedesignoftheMMUIisirrespectiveofhavingmoret
hanoneUImodal-ityandanyparticulartechnology.Oncetheneedformorethanonemodalityise
stablished,thetendencytosilotheirdesignanddevelopmentmustberesisted.ThemultipleU
Ismayhavedifferenttasks,designdocuments,developmentteamsandtimelines,anddifferen
ttechnologies,butthehumanfactorsspecialistmustchampionthefactthatitisstillauserp

erformingtaskswithasingleapplication.Tohelppreventfalseassumptionsanddesignerror
sandtobolsterconsistency,thehigh-levelanddetaileddesignoftheUIsshouldbeexecuteda
tthesametime.Example:Forthemovieshowtimesapplication,designdocumentsforawebUI(e.
g.,wireframes)andspeechrecognitionIVRUI(e.g.,callflows)arecreatedatthesametime.1
1.6.4ConductIterativeUserTestingEarlyandOftenThisisavaluableguidelineforanyUIdes
ign,butitisespeciallyimportantwhendesigninganapplicationwithmultipleUIs.Notonlyi
sitnecessarytomakemorethanoneUIusableandaccessible,buttheinteractionbetweentheus
erandtheUIsgoesfrombeingasimpleback-and-forthstyletoamultifaceted,multimodeinter
actionwhoseinterplayisverydifficulttopredict.Thisamountofdesignworknecessitatesm
ultipleiterationsofusertestingtoensureoptimalperformanceonanyoftheinterfaces,but
itisalsocriticaltodoitearlybecausethereisevenmoreinfrastructuretoanMMUIapplicati
onthatcannoteasilybechangedifdevelop-menthasprogressedtoofar.Onthepositiveside,t
hough,earlytestingwithuserslikelywillprovideuniqueandvaluableperspectivesaboutho
wtomaketheUIsworkbettertogetherbecausetheparticipantswillbelessfocusedontheUItec
h-nologyandmoreonwhattheywanttodo.Example:Theusertestingofamovieshowtimesapplica
tionbeginswithapaperprototypeofawebUIandaWizardofOzprototypeforthespeechrecognit
ionIVR.Resultsofthesetestsareusedtorefinethedesigns,eitherbyimprovingandthenrete
stingthelow-fidelityprototypesorbycreatinghigher-fidelityprototypesbasedonthefin
dings.11.6.5JointlyTestMultipleUserInterfacesasEarlyasPossibleJustasitisimportan
ttoconcurrentlydesigntheUIs,itisimportanttojointlytestthem.Andaswithanyuser-cent
ereddesignprocess,generallytheearlier
11MultimodeInterfaces:TwoorMoreInterfaces382

andmoreoftentestingisdone,thebetter.Jointtestinghelpstoproviderealisticscenarios
forperformingtasks.Italsohelpstoprovidedataaboutuserpreferencesforaccomplishingc
ertaintaskswiththevariousUIs.Lastly,thelessonslearnedfromtestingoneUIcanbetransf
erredtotheotherUIswithintheproducttovalidatewhethertheyareappropriateinallcases.
Example:Amessagingapplicationthatcombinesvoicemails,faxes,ande-mailsistestedusin
gaprototypeofawebUIandatouch-toneIVRUI.Par-ticipantsperformthesametaskswithbothU
Is,suchassettingupanewmail-box,checkingforavoicemailfromacoworker,deletingane-ma
il,changingthevoicemailgreeting,andsettingupnewmessagenotification.Therela-tivep
erformanceanduserpreferenceforeacharecollectedforcomparison.11.6.6BeConsistentam
ongtheInterfacestotheGreatestExtentPossibleGivenLimitsofModalitiesConsistencyisv
italforanyUI,butforanMMUIitiscritical.Emphasizerobust-nessandconsistencyevenmoret
hanintraditionalinterfacedesign(Kray,Wasinger,&Kortuem,2004).Itisimportantthatthe
usernotberequiredtolearninfactorinperceptionhowtousedifferentUIsforthesameapplicat
ion.TheoverridingphilosophyindesigningtasksforalloftheUIsistomakethemassimilaras
possible.AlthoughitcanbetemptingtooptimizetasksforeachUImodality,inmostcasesthev
aluetotheuserfromcross-learningamongalloftheapplicationsUIs,thepredictabilityofco
nsistency,andthequickerformationofanaccuratementalmodelusuallyoutweighsthegainfr
omadaptingtaskstoeachmodality.Thecaveat,however,isthatthedifferentUImodalitiesar
e,infact,different(afterall,morethanonewasneeded),andthevariouscapabilitiesandco
nstraintsofeachmodalityandUItechnologyhavetobeconsidered(Seffah,Forbrig,&Javaher
y,2004).Consequently,theimplementationoffunctionalitycannotbeexactlythesameforea
chmodality.Occasionally,itwillbeclearlybettertodesignataskspecifictoasingleUIkno
wingthatthesamedesigncannotbeappliedtoanotherUI.Nevertheless,thegoalistoachievea
Pareto-optimaldesignamongtheUImodalities,whichismucheasiertoachieveifconcurrentd
esignoftheUIsisdone.Example:Anautomatedbillingapplicationhasthesametaskflowforth
ewebUIandforaspeechrecognitionIVRbyfirstpresentingthebillamountandduedate,andthe
npromptingforthepaymentinformationinthesameorder.
11.6DesignGuidelines383

11.6.7UseConsistentLabelsamongModalitiesOneofthemostimportantdimensionsofconsist
encyamongthedifferentUImodalitiesistousethesamenamesanddescriptionsforthesameite
ms,func-tions,data,andeventhemodalitiesthemselves.Bydoingthis,userswillnotonlybe
abletocross-learnthemodalities,butwillalsohavealevelofcomfortwiththeapplicationb
ecause,eventhoughtheirtasksmaylookand/orsounddifferentamongtheUIs,thesametermsar
eusedtorefertothevariouscomponentsoftheapplication.Example:Thetermcellphoneisusedf
ortheweb,IVR,andsmall-screenUIsaswellastheuserdocumentation.Thetermsmobilephoneorwi
re-lessphonearenotusedanywherewithintheapplication.11.6.8AdheretoConventionsofMod
alitiesUsedWhileMaintainingConsistencywithOtherModalitiesintheApplicationWhendes
igningmorethanoneUImodalityforanapplication,therecanbesuchathingastoomuchconsist
encyamongtheUIs.Asdiscussedearlier,con-sistencyhelpsuserstolearnandtransferknowl
edgefromoneUItoanother.However,eachUImodalityhasitsownstrengths,weaknesses,idios
yncrasies,andsometimesexpectationsoftheuserabouthowitworks(Seffah,Forbrig,&Javah
ery,2004).Sometimesbeingconsistentwiththerestoftheapplicationwillproduceasubopti
maldesignforoneoftheUIs.Inthosecases,itmaybemoreeffectivetocreateaUIforonemodali
tythatisnotthesameastheothermodalities.Example:Whencheckingvoicemailmessagesover
thephone,thenewmessagesarefirstplayedtogether,followedbyallsavedmessages.Whenvie
wingalistofvoicemailmessagesontheweb,thenewandsavedmes-sagesareshownmixedtogethe
r,inreversechronologicalorder.11.6.9DoNotMakeUsersReenterInformationacrossModali
tiesIftheuserentersinformation,includinguserdata,preferences,andsoon,thenthatinf
ormationshouldbeavailablewhentheapplicationisaccessedviaanyofthemodalities.Noton
lydoesthissavetheuserstimeandreinforcethefactthattheyareusingoneapplication(rath
erthanseveralapplicationscobbledtogether),butitalsopreventshavingdifferentversio
nsofthesamedatawithintheapplication.
11MultimodeInterfaces:TwoorMoreInterfaces384

Example:Theuserentershisfavoritesportsteamsintoawebportalsothathecanseeacustomiz
edlistofscores,whichthevoiceportalalsousestocreatealistofscores.11.6.10MakeShari
ngandUpdatingofUserDataamongUserInterfaceModalitiesTransparenttoUsersOneofthemaj
orbenefitsofanMMUIisthatitsupplantsmultipleapplicationseachwithitsownUI.Buttoach
ievethebenefitsofasingleapplication,thedatacannotbesegmentedorhandledonaper-moda
litybasis.Iftheapplicationneedstousemultipledatabasesorotherdatastores,thentheap
plicationneedstounob-trusivelyperformthenecessarytransactionssoastoappearasifitw
ereasingleapplicationtotheuser.Example:Whenauserchangesthephonenumberforoneofher
contactsinhercellphonesaddressbook,thephoneautomaticallysendsames-sagetoacentrali
zeddatabaseforupdating,whichinturnschedulesitforsynchronizationwithherPDA.11.6.1
1ProvideGlobalPreferencesThatApplytoAllModalitiesSomeuserpreferencesshouldapplyt
otheentireapplication,regardlessofwhichUIisbeingused.Thispracticesavestheusertim
eandreducescomplexitybecauseanychangesneedtobemadeinonlyoneplaceandtherearefewer
over-allsettingstotrack.Example:Thereisausersettingtoemploythesamelanguage(e.g.,
English)inallUIs.11.6.12JudiciouslyProvidePreferencesThatApplytoOneorOnlySomeMod
alitiesAlthoughinmostcasesitisbeneficialtohaveglobalpreferences,duetothenatureof
thevariousmodalitiesthereissometimesaneedtohavespecialprefer-encesforasinglemoda
lity.Inaddition,preferenceswithdifferentsettingsmaybeuseful,dependingonwhichmoda
lityisbeingused.Example:Bydefaultwhenviewingstockquotesonawebportal,theopeningpr
ice,dailyhigh,dailylow,52-weekhigh,52-weeklow,closingprice,changeinprice,andperc
entagechangeareallshownforeach
11.6DesignGuidelines385

stock. Alternatively, by default when listening to stock quo tes over t hephone
via a sp eech r ecognition IVR, only the closing price and cha nge inprice are g
iven, with an option to he ar more det ails. The us er is al so giventhe prefere

nces about which pieces of info rmation t o display on the webportal an d the I
VR.11.6.13 Use a Multimode User Interface to EnhanceUniversal Design and Accessi
bility, Not as aSubstitute for Making Each User InterfaceModality AccessiblePr o
viding more than one modality for use r intera cti on c an great ly benef itpeop
le with disabilities, a s well a s all users who need or want to use the appli-c
ation in restricted contexts of use. But j ust because one type of di sability m
a ynot often use one of the modalities, it does not me an t ha t reasonable e ff
ortshould notbe madetomakealloftheUImodalitiesaccessible.Doingsowillmean that pe
ople wi th disabi lit ies are n ot l imit ed t o on e mo d al ity, but maychoose
the mo dal ity that best sui ts their needs at t he ti me t ha t they use theap
pl ication .Example: In a messaging application, web and speech recognition IVR
UIsare used to fulfill the needs of users and give them more flexibility overhow
they interact with their messages. However, the W3C Web ContentAccessibilityGui
delinesarefollowedforthewebsitetomakeitaccessibletoblinduserswithscreenreaders,r
atherthanforcingthemtouseonlythespeechrecognitionIVR.11.7CASESTUDYA case study o
f an MMUI can be found at www.beyondthegui.com .11.8FUTURETRENDSInthefutureitisl
ikelythattheuseofMMUIsforthesameapplicationwillgrow.Thegrowthprobablywillnotbeph
enomenalbecauseofthecomplexityandcostinvolvedindevelopingthem.However,theirincre
aseduseprobablywillbesurprisingbecauseoftheinnovativewaystheywillbeusedtofulfill
theneedsofusers.Infact,thegrowthwillbespurredbythegrowingneedofusersforbothmobil
ityandflexibility.Astheneedforinformationandserviceswhileonthegoincreases,newmet
hodsofsupplyingtheinformationinthefuturewillberequired,including
11MultimodeInterfaces:TwoorMoreInterfaces386

wirelessbroadbanddataandthematurationofthetechnologyusedinwearablecom-puting.Thi
s,inturn,willbedrivenbyalargerrangeoftasksthatpeoplewillbeabletoperformandalsoby
thegreaterdiversityofpeopleandcontextsofuse.Oneofthekeydriversofflexibilitywillb
etheincreasedfocusonuniversaldesignandaccessibility.Tosomeextentthiswillbedriven
bycivilrightslegisla-tionforpeoplewithdisabilities,suchastheAmericanswithDisabil
itiesAct,Section255oftheTelecommunicationsAct,Section508oftheRehabilitationAct,a
ndsimilarlegislationaroundtheworld.Butfromtheotherside,consumerdemandwillgrowfor
moreinclusivedesigns.Thisdemandismostlygoingtocomefromtheagingpopulation,asBabyB
oomersageandlongerlifeexpectan-ciesincreasetherelativenumberofolderconsumers.Thi
sfundamentalchangeforalmostallapplicationsfortheuserpopulationwillmakeaccessibil
ityissuespartofthemainstreamdesignprocess.Astheseneeds,bothregulatoryandmarketdr
iven,cometotheforefront,UIdesignwillneedtoprovideconsumerswithproductsthatmeetth
oseneeds.Ashasbeendis-cussedearlierinthischapter,MMUIsarenotasubstituteformaking
UIsaccess-ible.However,whenselectedproperly,theyprovidegreaterflexibilityandchoi
ceoverhowusersinteractwithanapplication,whichcanbeaboonforpeoplewithdisabilities
andotherswithtemporaryimpairmentsorreducedabilitiesbecauseoftheirenvironment.Fro
matechnicalstandpoint,thegrowthindemandforapplicationswithmorethanoneUImodalityn
eedstobeandisbeingsupportedbyacorrespondingimprovementintechnologytoenableMMUIs.Th
emostimportantoftheseadvance-mentsistheincreasinguseofstandards.Thewidespreadado
ptionofstandardshelpsintwoimportantways.First,makingdatadeviceindependentresults
ineasiermanipulationbyanyUItechnology.Forexample,eXtensibleMarkupLanguage(XML)ca
nberenderedontheWebusinghypertextmarkuplanguage(HTML)orwithanIVRusingVoiceXML(Pa
terno,2004).Italsoenablesback-endsysteminter-operability,whichhelpstoachievemuch
oftheseamlessnessthatiscriticalfortheeaseofuseofMMUIs.Theothermajortechnologicalch
angethatisoccurringtoincreasethedevel-opmentofMMUIsisincomputingtechnologyitself
.Ascomputingpowerrapidlygrows,withanoppositedropinpriceandsize,thedemandofconsum
ersforaccesstoinformationandservicesathomeorwork(wherearicherinteractioncanbedes
igned)andonthego(wheremobilityrestrictsinteractiontechniques)willincrease.Contex
t-dependentandmigratoryUIsthatexploitdifferentmodesofinputwillbeamajordriveroftheu
seofMMUIs(Paterno,2004).Evenwiththisincreaseincomputingabilityandothertechnologi
caldevelop-ments,itisstilllikelythatMMUIswillcontinuetobealmostexclusivelycombin
a-tionsofvisualandauditoryinterfaces.Althoughthefuturenodoubtholdsimprovementsin

ourunderstandingandinnovationswiththeothersenses,thisisnotlikelytobeenoughinthen
eartermtobesufficientforastand-alonecomponentofanMMUI.
11.8FutureTrends387

OnefinalthoughtaboutthefutureofMMUIsisinorder.Astechnologybecomesmoreubiquitousa
ndthetoolstobuildtheseinterfacesbecomemorewidelyavailable,therewilllikelybeaninc
reaseincompetition.Whenthetech-nologyandfeaturesarenolongerthedefiningreasonsfor
apurchaseandpricepointsconverge,consumerswilldemandandstartmakingpurchasingdecisi
onsbasedontheeaseofuseoftheproductorservice.Thewell-trainedanddata-drivenhumanfac
torsprofessionalcanbethebridgefromgrandideasofwhatanapplicationcanbetomakingitsi
mpleandenjoyableforpeopletouse.REFERENCESAmericanNationalStandardsInstitute&Huma
nFactorsandErgonomicsSociety.(2005).HumanFactorsEngineeringofSoftwareUserInterfa
ces.SantaMonica,CA:HumanFactorsandErgonomicsSociety.Bias,R.G.,&Mayhew,D.J.(2005)
.Cost-JustifyingUsability:AnUpdatefortheInternetAge,2nded.SanFrancisco:MorganKau
fmann.Brooke,J.(1996).SUS:Aquickanddirtyusabilityscale.InJordan,P.W.,Thomas,B.,Wee
rdmeester,B.A.,&McClelland,A.L.,eds.,UsabilityEvaluationinIndustry.London:Taylor
andFrancis.Carroll,J.M.,ed.(2003).HCIModels,Theories,andFrameworks:TowardaMultid
isciplin-aryScience.SanFrancisco:MorganKaufmann.Cohen,M.H.,Giangola,J.P.,&Balogh
,J.(2005).VoiceUserInterfaceDesign.Boston:Addison-Wesley.Hix,D.,&Hartson,H.R.(19
93).DevelopingUserInterfaces:EnsuringUsabilityThroughProduct&Process.NewYork:Joh
nWiley&Sons.InternationalStandardsOrganization.(2006).ErgonomicsofHumanSystemInte
ractionPart110:DialoguePrinciples.Geneva:ISO.InternationalStandardsOrganization.(
1998).ErgonomicRequirementsforOfficeWorkwithVisualDisplayTerminals(VDTs).Part11:
GuidanceonUsability.Geneva:ISO.Kantowitz,B.H.,&Sorkin,R.D.(1987).Allocationoffun
ctions.InSalvendy,G.,ed.,Hand-bookofHumanFactors.NewYork:Wiley.Kray,C.,Wasinger,
R.,&Kortuem,G.(2004).Conceptsandissuesininterfacesformultipleusersandmultipledev
ices.ProceedingsofWorkshoponMulti-UserandUbiquitousUserInterfaces(MU3I)atIUI2004
,Funchal,Madeira,Portugal,711.Laurel,B.(1990).TheArtofHumanComputerInterfaceDesign
.Reading,MA:Addison-Wesley.Lee,K.M.,&Lai,J.(2005).Speechversustouch:Acomparative
studyoftheuseofspeechandDTMFkeypadfornavigation.InternationalJournalofHumanComput
erInteraction19(3):34360.Paterno,F.(2004).Multimodalityandmulti-deviceinterfaces.
W3CWorkshoponMultimodalInteraction.SophiaAntipolis,France:WorldWideWebConsortium
.Raskin,J.(2000).TheHumaneInterface:NewDirectionsforDesigningInteractiveSystems.
NewYork:ACMPress.
11MultimodeInterfaces:TwoorMoreInterfaces388

Seffah, A., Forbrig, P., & Javahery, H. (2004). Multi-devices Multiple user interf
aces:Development models and research opportunities. Journal of Systems and Softw
are73:287300.Seffah, A., & Javahery, H. (2004). Multiple User Interfaces: Cross-P
latform Applications andContext-Aware Interfaces . Hoboken, NJ: Wiley.Thatcher,
J., Burks, M. R., Heilmann, C., Henry, S. L., Kirkpatrick, A., Lauke, P. H., Law
son, B.,Regan, B., Rutter, R., Urban, M., & Waddell, C. D. (2006). Web Accessibi
lity: Web Standardsand Regulatory Compliance. Berkeley, CA: Friends of Ed.Weinsc
henk, S., Jamar, P, & Yeo, S. C. (1997). GUI Design Essentials . New York: John
Wiley& Sons.Wickens, C. D. (1992). Engineering Psychology and Human Performance
, 2nd ed. NewYork:HarperCollins.World Wide Web Consortium. (2006). Web Content A
ccessibility Guidelines http://www.w3.org/WAI/intro/wcag.php .Wright,P.,Dearden,A
.,&Fields,R.(2000).Functionallocation:Aperspectivefromstudiesofworkpractice.Inte
rnationalJournalofHumanComputerStudies52(2):33555.
References389

Thispageintentionallyleftblank

12CHAPTERMultimodalInterfaces:CombiningInterfacestoAccomplishaSingleTaskPauloBar
thelmess,SharonOviatt12.1NATUREOFTHEINTERFACEAnessentialdistinguishingcapability
ofmultimodalsystemsistheusethesesystemsmakeoftwoormorenaturalinputmodalitiessuch
asspeech,handwriting,gestures,facialexpressions,andotherbodymovements(Oviatt,200
7).Usingsuchsystems,usersmayforexampledealwithacrisismanagementsituationusingspe
echandpeninputoveramap.Anemergencyresponseroutecanforexamplebeestablishedbysketc
hingalineonamapalongthedesiredroute,usingadigitalpenoraTabletPCstylus,whilespeak
ingCreateemergencyroutehere(Oviattetal.,2003).Inthisexample,amultimodalsystemanaly
zesthespeechandsketch-ingcapturedviasensors(amicrophoneandaninstrumentedwritings
urface),andinterpretstheusersintentionbycombining(orfusing)thecomplementaryinformationprovidedviathesetwomodalities.Theresultinthiscaseisaninterpretationthattak
eslocationinformationfromthesketchedlineandtheattributesoftheobjectbeingcreated(
thatthisisanemergencyroute)fromthespeech.Multimodalinterfacesmaytakemanyformsandas
pects(Figure12.1).Systemsmaybehostedbysmallportabledevices,suchasonaPDAorcellpho
nethatistakentothefield,oronatabletcomputerusedwithinofficesorincafeterias.Group
sofpeoplemayinteractmultimodallyvialargeinteractiveboardsorsheetsofdigitalpaper.
Thecommonaspectinallthesecasesistheinterfacesupportforinteractionvianaturalmodes
ofcommunicationinvolvingcombinations,forexample,ofspeech,peninput,gestures,gaze,
orothermodalities.12.1.1AdvantagesofMultimodalSystemsTheintrinsicadvantageofmult
imodalsystemsisthattheyallowuserstoconveytheirintentionsinamoreexpressiveway,bet
termatchedtothewaytheynaturally

communicate.Awell-designedmultimodalsystemgivesusersthefreedomtochoosethemodalit
ythattheyfeelbestmatchestherequirementsofthetaskathand.Usershavebeenshowntotakea
dvantageofmultimodalsystemcapabilitieswithoutrequiringextensivetraining.Givenach
oice,userspreferredspeechinputfordescribingobjectsandeventsandforissuingcommands
foractions(Cohen&Oviatt,1995;Oviatt&Cohen,1991).Theirpreferenceforpeninputincrea
sedwhenconveyingdigits,symbols,graphiccontent,andespeciallywhenconveyingthelocat
ionandformofspatiallyorientedinformation(Oviatt,1997;Oviatt&Olsen,1994;Suhm,1998
).Asaresultofthechoiceofinputtheyprovidetousers,multimodalsystemsmakecomputingmo
reaccessible,loweringinputrequirementbarrierssothatabroaderrangeofuserscanbeacco
mmodated(e.g.,byallowingauserwithasen-sorydeficittousethemodalitysheismostcomfor
tablewith).Asaconsequence,usersbecomeabletocontrolabroaderrangeofchallenging,com
plexapplicationsthattheymightnototherwisebeabletocommandviaconventionalmeans.The
sefactorsleadtothestronguserpreferenceforinteractingmultimodallythathasbeendocum
entedintheliterature.Whengiventheoptiontointeractviaspeechorviapeninputinamap-ba
seddomain,95to100percentoftheuserschosetointeractmultimodally(Oviatt,1997).While
manipulatinggraphicobjectsonaCRTscreen,71percentoftheuserspreferredtocombinespee
chandmanualgestures(Hauptmann,1989).ErrorReductionviaMutualDisambiguationArichse
tofmodalitiesnotonlyprovidesforenhancedfreedomofexpression,allowinguserstochoose
themodalitiesthatbestfittheirparticularsituation,butmayalsoleadtobetterrecogniti
onwhencomparedforinstancetospeech-only
(a)
(b)
FIGURE12.1Multimodalinterfacesspanavarietyofusecontexts.(a)Mobileinterface.(b)Co
llaborativeinterface.

12MultimodalInterfaces:CombiningInterfaces392

interfaces(Oviatt,1999a).Mutualdisambiguationisusedtorefertothepositiveeffecttha
trecognitioninonemodemayhaveinenhancingrecognitioninanothermode.Analysisofaninpu
tinwhichausermaysayPlacethreeemergencyhos-pitalshere<point>here<point>andhere<poi
nt>providesamultimodalsystemwithmultiplecuesindicating,forinstance,thenumberofobj
ectstobeplaced,asthespokenwordthreecanbematchedtothenumberofpointinggesturesthatar
eprovidedviaanothermodality(Oviatt,1996a).Similarly,misrecognizedcommandsineachm
odalitymayinsomecases(e.g.,tutoriallectures)becorrectedbyexploitingredundancy.Em
piricalresultsdemonstratethatawell-integratedmultimodalsystemcanyieldsignificant
levelsofmutualdisambiguation(Cohenetal.,1989;Oviatt,1999a,2000).Studiesfoundmutu
aldisambiguationanderrorsuppressionrangingbetween19and41percent(Oviatt,1999a,200
0,2002).Similarrobustnessisfoundaswellinsystemsthatintegratespeechandlipmovement
s.Improvedspeechrecognitionresultscanoccurduringmultimodalprocessing,bothforhuma
nlisteners(McLeod&Summerfield,1987)andsys-tems(Adjoudani&Benoit,1995;Tomlinson,R
ussell,&Brooke,1996).Asimilaropportunityfordisambiguationexistsinmultipartysetti
ngsforinstance,whencommunicatingpartnershandwritetermsonasharedspace,suchasawhite
-boardorashareddocument,whilealsospeakingthesesameterms.Theredun-dantdeliveryoft
ermsviahandwritingandspeechissomethingthathappenscommonlyinlecturesorsimilarpres
entationscenarios(Anderson,Andersonetal.,2004;Anderson,Hoyeretal.,2004;Kaisereta
l.,2007).Bytakingadvantageofredundancywhenitoccurs,asystemmaybeabletorecoverthei
ntendedtermseveninthepresenceofmisrecognitionsaffectingbothmodalities(Kaiser,200
6).PerformanceAdvantagesEmpiricalworkhasalsodeterminedthatmultimodalsystemscanof
ferperfor-manceadvantages(e.g.,duringvisual-spatialtasks)whencomparedtospeech-on
lyprocessingorconventionaluserinterfaces.Multimodalpenandspeechhavebeenshowntore
sultin10percentfastercompletiontime,36percentfewertask-criticalcontenterrors,50p
ercentfewerspontaneousdisfluencies,andshorterandsimplerlinguisticconstructions(O
viatt,1997;Oviatt&Kuhn,1998).Whencom-paringtheperformanceofmultimodalsystemstoac
onventionaluserinterfaceinamilitarydomain,experimentsshowedafour-tonine-foldspee
dimprovementoveragraphicaluserinterfaceforacomplexmilitaryapplication(Cohen,McGe
e,&Clow,2000).Multimodalsystemsarepreferableinvisual-spatialtasksprimarilybecaus
eofthefacilitiesthatmultimodallanguageprovidestospecifycomplexspatialinformation
suchaspositions,routes,andregionsusingpenmode,whilepro-vidingadditionaldescripti
vespokeninformationabouttopicandactions(Oviatt,1999b).
12.1NatureoftheInterface393

12.2TECHNOLOGYOFTHEINTERFACEMultimodalsystemstakeadvantageofrecognition-basedcom
ponenttechnologies(e.g.,speech,drawing,andgesturerecognizers).Advancesinrecognit
iontechno-logiesmakeitincreasinglypossibletobuildmorecapablemultimodalsystems.Th
eexpressivepowerofrichmodalitiessuchasspeechandhandwritingisnone-thelessfrequent
lyassociatedwithambiguitiesandimprecisioninthemessages(Bourguet,2006).Theseambig
uitiesarereflectedbythemultiplepotentialinter-pretationsproducedbyrecognizersfor
eachinput.Recognitiontechnologyhasbeenmakingsteadyprogress,butisstillconsiderabl
ylimitedwhencomparedtohuman-levelnatural-languageinterpretation.Multimodaltechno
logyhasthere-foredevelopedtechniquestoreducetheuncertainty,attemptingtoleveragem
ultimodalitytoproducemorerobustinterpretations.Inthefollowing,ahistoricperspecti
veofthefieldispresented(Section12.2.1).Sections12.2.2and12.2.3thenpresenttechnic
alconceptsandmechanismsandinformationflow,respectively.12.2.1HistoryandEvolution
BoltsPutthatthereisoneofthefirstdemonstrationsofmultimodaluserinter-faceconcepts(19
80).Thissystemalloweduserstocreateandcontrolgeometricshapesofmultiplesizesandcol
ors,displayedoveralarge-formatdisplayembed-dedinaninstrumentedmediaroom(Negropon
te,1978).Thenoveltyintro-ducedwasthepossibilityofspecifyingshapecharacteristicsv

iaspeech,whileestablishinglocationviaeitherspeechordeictic(pointing)gestures.The
positionofusersarmswastrackedusingadeviceattachedtoabracelet,displayinganxonthesc
reentomarkthepositionofaperceivedpointlocationwhenauserspokeanutterance.Tocreate
newobjects,userscouldforinstancesayCreateabluesquareherewhilepointingattheintended
locationofthenewobject.Suchmultimodalcommandswereinterpretedbyresolvingtheuseofhe
rebyreplacingitwiththecoordinatesindicatedbythepointinggesture.ByspeakingPutthatth
ere,userscouldselectanobjectbypointing,andthenindicatewithanadditionalgesturethed
esirednewlocation(Figure12.2).Thisdemonstrationsystemwasmadepossiblebytheemergen
ceofrecognizers,particularlyforspeech,thatforthefirsttimesupportedmultiple-words
entenceinputandavocabularyofafewhundredwords.EarlyYearsInitialmultimodalsystemsm
adeuseofkeyboardandmouseaspartofatraditionalgraphicalinterface,whichwasaugmented
byspeech.Spokeninputprovidedanalternativetotypedtextualinput,whilethemousecouldb
eusedtoprovide
12MultimodalInterfaces:CombiningInterfaces394

pointinginformation.ExamplesofearlysystemsincludeXTRA,amultimodalsys-temforcompl
etingtaxforms(Allegayeretal.,1989),VoicePaintandNotebook(Nigay&Coutaz,1993),Geor
al(Sirouxetal.,1995),MATIS(Nigay&Coutaz,1995),CUBRICON(Neal&Shapiro,1991),andSho
pTalk(Cohen,1992).Othersys-temsofthisperiodincludeVirtualWorld(Codellaetal.,1992
),Finger-Pointer(Fukumoto,Suenaga,&Mase,1994),VisualMan(Wang,1995),Jeannie(Vo&Wo
od,1996),QuickTour,JeannieII,andQuarterback(Vo,1998),andamultimodalwindowmanager
(Bellik&Teil,1993).Thesophisticationofthespeechrecognitioncomponentsvaried,withs
omesystemsofferingelaboratesupportforspokencommands.CUBRICONandShop-Talkallowedu
serstoasksophisticatedquestionsaboutmapsandfactorypro-ductionflow,respectively.T
heinterfacesofthesesystemsactivatedextensivedomainknowledgesothatuserscouldpoint
toalocationonamap(inCUBRI-CON)andaskIsthisanairbase?orpointtoaspecificmachineinadi
agram(inShopTalk)andcommandthesystemtoShowallthetimeswhenthismachinewasdown.
FIGURE12.2Boltsmultimodaldemonstrator.Ausermovesanobjectbypointingtoitandthenthed
esirednewlocationwhilesaying,Putthatthere.Source:FromBolt(1980).(CourtesyACM.)
12.2TechnologyoftheInterface395

SpeechandPenSystemsSystemsstartedtomoveawayfromaugmentinggraphicalinterfacesbyin
corpor-atinginputwithafullspoken-languagesystem.Duringthistime,equipmentbecameav
ailabletohandlepeninputviaastylus,providinganaturalsecondsourceofrichinformation
thatcomplementsspeech(Oviattetal.,2000).Peninputmadeitpossibleforuserstoinputdia
gramsandhandwrittensymbolicinformation,inadditiontopointingandselection,replacin
gsimilarmousefunc-tionality.Theavailabilityofmorepowerfulrecognitioncomponentsle
dinturntothedevelopmentofnewtechniquesforhandlingparallel,multimodalinput.Pen-ba
sedinputoveratablet-likesurfacestartedtobeusedtoprovideaddi-tionalcommandsviaske
tchedgestures(e.g.,inTAPAGE[Faure&Julia,1994]).Peninputthenevolvedtoprovideaseco
ndsemanticallyrichmodality,comple-mentingandreinforcingthespokeninput.Handwritin
grecognition(e.g.,inCheyerandJulia,1995)andmoresophisticatedgesturerecognition(i
nMVIEWS[Cheyer&Luc,1998]andQuickset[Cohenetal.,1997])provideduserswithsig-nifica
ntexpressivepowerviainputthatdidnotrequirethekeyboardorthemousetobeused,providin
ginsteadaflexible,trulymultimodalalternativeinputviaspeechandpen.Empiricalworkin
theearly1990s(Oviattetal.,1992)laidthefoundationformoreadvancedspeechandpensyste
mssuchasQuickset(Figure12.3).Quicksetprovidesagenericmultimodalframeworkthatisst
illcurrentlyused(e.g.,byCharter[Kaiseretal.,2004],asystemforcollaborativeconstru
ctionofprojectschedules).OneoftheinitialQuicksetapplicationswasamap-basedmilitar
ysystem(whichevolvedintoacommercialapplication,NISMap[Cohen&McGee,2004])(seeSect
ion12.3.3).Usingthissystem,militarycommanderscanenterbattleplanfragmentsusingmul

timodallanguage(Figure12.3).Morespecifically,
FIGURE12.3Quickset/NISMapinterfaceonaTabletPC.Thekeyboardandmousearereplacedbype
nandspeechinput.Sketchesandhandwritingonamaparefusedwithspokeninput.
12MultimodalInterfaces:CombiningInterfaces396

usersofthesystemcandrawstandardmilitarysymbolscomprisingsketchedandhandwrittenel
ements,complementedbyspeech.Forexample,acommandercanestablishthepositionofaforti
fiedlinebysketchingthelineonamapandspeakingFortifiedline.MVIEWS(Cheyer&Luc,1998)de
monstratedmultimodalfunctionalityrelatedtovideoanalysisactivitiesformilitaryinte
lligencepurposes.MVIEWSintegratedaudioannotationsofvideostreams,allowinguserstom
arkanddescribespecifictargetsofinterest.Bycirclinganobjectonavideoframeusingasty
lus,thesystemcouldbecommandedviaspeechtotracktheobjectinsubsequentframes.Usersco
uldalsorequesttobenotifiedwheneveranobjectselectedviaastylusmoved,orwheneveracti
vitywithinaregionwasdetected(e.g.,byspeakingIfmorethanthreeobjectsenterthisregion
,alertme.).OthersystemsofthesamegenerationincludeIBMsHuman-centricWordProcessor(Ov
iattetal.,2000),BoeingsVirtualReality(VR)AircraftMaintenanceTraining(Duncanetal.,
1999),FieldMedicInformationSystem(Oviattetal.,2000),andthePortableVoiceAssistant
(Bers,Miller,&Makhoul,1998).TheHuman-centricWordProcessorprovidesmultimodalcapab
ilitiesfordic-tationcorrection.Userscanuseapentomarkasegmentoftext,issuingspoken
commandssuchasUnderlinefromheretohere<drawline>,orMovethissen-tence<pointtosentence
>here<pointtonewlocation>(Oviattetal.,2000).TheFieldMedicInformationSystemisamob
ilesystemforuseinthefieldbymedicsattendingtoemergencies.Twocomponentsareprovidedo
neallowingspokeninputtobeusedtofilloutapatientsrecordform,andasecondmultimodaluni
ttheFMAthatletsmedicsenterinformationusingspeechoralternativelyhandwritingtoperfor
mannotationsofahumanbodydiagram(e.g.,toindicatepositionandnatureofinjuries).TheF
MAcanalsoreceivedatafromphysiologicalsensorsviaawirelessnetwork.Finally,thepatie
ntreportcanberelayedtoahospitalviaacellular,radio,orsatellitelink.Morerecently,t
angiblemultimodalinterfaceshavebeenexploredinRasa(Cohen&McGee,2004;McGee&Cohen,2
001).Rasaisatangibleaugmented-realitysystemthatenhancesexistingpaper-basedcomman
dandcontrolcapabil-itiesinamilitarycommandpost(McGee&Cohen,2001).Rasainterpretsa
ndintegratesinputsprovidedviaspeech,pen,andtouch.Thesystemreactsbyproject-inginf
ormationoverapapermapmountedonaboard,producingspokenprompts,andinsertingandupdat
ingelementsinadatabaseandmilitarysimulators.Rasadoesnotrequireuserstochangetheir
workpracticesitoperatesbytransparentlymonitoringusersactionsastheyfollowtheirusual
routines.Offi-cersincommandpostsoperatebyupdatinginformationonasharedmapbycre-at
ing,placing,andmovingstickynotes(Post-its)annotatedwithmilitarysymbols.Astickyno
temayrepresentasquadofanopposingforce,representedviastan-dardizedsketchednotatio
n(Figure12.4).Asusersspeakamongthemselveswhileperformingupdates,sketchingonthema
pandonthestickynotesplacedonthemap,Rasaobservesandinterpretstheseactions.Themapi
saffixedtoa
12.2TechnologyoftheInterface397

touch-sensitivesurface(aSmartBoard),sothatwhenstickynotesareplacedonit,thesystem
isabletosensethelocation.Theattributesofintroducedentitiesareintegratedbyfusingt
helocationinformationwiththetypeinformationprovidedviathestickynotesketch.Asketc
hrepresentinganenemycompanymaybeintegratedwithaspokenutter-ancesuchasAdvancedguar
d,identifyingtype,location,andidentityofaunit.Rasaconfirmstheinterpretationviaasp
okenpromptofthekindConfirm:Enemyreconnaissancecompanycalledadvancedguardhasbeensigh
tedatnine-sixnine-four.Userscancancelthisinterpretationorconfirmiteitherexplic-it
lyorimplicitly,bycontinuingtoplaceormoveanotherunit.SpeechandLipMovementsAconsid
erableamountofworkhasbeeninvestedaswellinexaminingthefusionofspeechandlipmovemen

ts(Benoit&LeGoff,1998;Bernstein&Benoit,1996;Cohen&Massaro,1993;Massaro&Stork,199
8;McGrath&Summerfield,1985;McGurk&MacDonald,1976;McLeod&Summerfield,1987;RobertRibesetal.,1998;Sumby&Pollack,1954;Summerfield,1992;Vatikiotis-Batesonetal.,1996
).Thesetwomo-dalitiesaremoretightlycoupledthanspeechandpeninputorgestures.System
sthatprocessthesetwohighlycorrelatedmodalitiesstrivetoachievemorerobustrecogniti
onofspeechphonemesbasedontheevidenceprovidedbyvisemesduringphysicallipmovement,p
articularlyinchallengingsituationssuchasnoisyenviron-ments(Dupont&Luettin,2000;M
eier,Hurst,&Duchnowski,1996;Potamianosetal.,2004;Rogozan&Deleglise,1998;Sumby&Poll
ack,1954;Vergo,1998).
FIGURE12.4UserscollaboratingwithRasa.Sketchesonstickynotesrepresentmilitaryunits
,placedonspecificlocationsonamap.Source:FromMcGeeandCohen(2001).(CourtesyACM.)
12MultimodalInterfaces:CombiningInterfaces398

Comprehensivereviewsofworkinthisareacanbefoundelsewhere(Benoitetal.,2000;Potamia
nosetal.,2004).MultimodalPresentationTheplanningandpresentationofmultimodaloutpu
t,sometimescalledmulti-modalfission,employsaudioandgraphicalelementstodeliversys
temresponsesacrossmultiplemodalitiesinasynergisticway.Someofthesystemsandframe-w
orksexploringmultimodalpresentationandfissionareMAGIC(Dalaletal.,1996),whichempl
oysspeechandagraphicaldisplaytoprovideinformationonapatientscondition;WIP(Wahlste
retal.,1993);andPPP(Andre,Muller,&Rist,1996).SmartKom(Wahlster,2006)isanencompas
singmultimodalarchitecturethatincludesfacilitiesformultimodalfissionaswell.Multi
modalpresentationsometimesmakesuseofanimatedcharactersthatmayspeak,gesture,anddi
splayfacialexpressions.Thestudyofthecorrelationofspeechandlipmovementsdiscussedi
ntheprevioussectionisalsorelevantininformingtheconstructionofrealisticanimatedch
aractersabletosynchronizetheirlipmovementswithspeech(Cohen&Massaro,1993).Further
workonani-matedcharactershasexploredwaystogenerateoutputinwhichcharactersareable
togesticulateandusefacialexpressionsinanaturalway,providingforrichmultimodaloutp
ut(Nijholt,2006;Oviatt&Adams,2000).Foroverviewsofthearea,see,forexample,Andre(200
3)andStockandZancanaro(2005).Vision-BasedModalitiesandPassiveInterfacesFacialexp
ressions,gaze,headnodding,andgesturinghavebeenprogressivelyincorporatedintomulti
modalsystems(Flanagan&Huang,2003;Koons,Sparrell,&Thorisson,1993;Lucente,Zwart,&G
eorge,1998;Morencyetal.,2005;Morimotoetal.,1999;Pavlovic,Berry,&Huang,1997;Turk&
Robertson,2000;Wang&Demirdjian,2005;Zhai,Morimoto,&Ihde,1999).Poppe(2007)present
sarecentoverviewofthearea.Furtherincorporationofmodalitiesisexpectedasvision-bas
edtrackingandrecognitiontechniquesmature.Ofparticularinterestarethepossibilities
thatvision-basedtechniquesintroduceameansforsystemstounobtrusivelycollectinforma
tionbypassivelymonitoringuserbehaviorwithoutrequiringexplicituserengagementiniss
uingcommands(Oviatt,2006b).Systemscanthentakeadvan-tageoftheinterpretationofnatu
raluserbehaviortoautomaticallyinferuserstate,tohelpdisambiguateuserintentions(e.
g.,byusingevidencefromnaturalgazetorefinepointinghypotheses),andsoforth.Eventual
ly,itisexpectedthatsystemsmaytakeadvantageoftheinformationprovidedbymultiplesens
orstoautomaticallyprovideassistancewithoutrequir-ingexplicituseractivationofinte
rfacefunctions.Theabilitytopassivelyobserveandanalyzeuserbehaviorandproactivelyr
eact(e.g.,Danningeretal.,2005;Oliver&Horvitz,2004)areofspecialinterestforgroupso
fusersthatmaybecollaborating,aselaboratedinthenextsubsection.
12.2TechnologyoftheInterface399

CollaborativeMultimodalSystemsRecently,considerableattentionhasbeengiventosystem
sthatinterpretthecom-municationtakingplaceamonghumansinmultipartycollaborativesc
enarios,suchasmeetings.Suchcommunicationisnaturallymultimodalpeopleemployspeech,g
estures,andfacialexpressions,takenotes,andsketchideasinthecourseofgroupdiscussio

ns.AnewbreedofresearchsystemssuchasRasa(McGee&Cohen,2001),Neem(Barthelmess&Ellis
,2005;Ellis&Barthelmess,2003),andothers(Falconetal.,2005;Pianesietal.,2006;Rienk
s,Nijholt,&Barthelmess,2006;Zancanaro,Lepri,&Pianesi,2006)havebeenexploitingmult
imodalcollaborativescenarios,aimingatprovidingassistanceinwaysthatleveragegroupc
ommunicationandattempttoavoidadverselyimpactingperformance.Processingunconstrain
edcommunicationamonghumanactorsintroducesavarietyoftechnicalchallenges.Conversat
ionalspeechoveranopenmicrophoneisconsiderablyhardertorecognizethanmoreconstraine
dspeechdirectedtoacomputer(Oviatt,Cohen,&Wang,1994).Theinterpretationofothermoda
litiesissimilarlymorecomplex.Sharedcontextthatmaynotbedirectlyaccessibletoasyste
misreliedonveryheavilybycommunicatingpartners(Barthelmess,McGee,&Cohen,2006;McGe
e,Pavel,&Cohen,2001).Devisingwaystoextracthigh-valueitemsfromwithinthecomplexgro
upcommunicationstreamsconstitutesaprimarychallengeforcollaborativemulti-modalsys
tems.Whereasinasingle-usermultimodalinterface,ahighdegreeofcontroloverthelanguag
eemployedcanbeexerted,eitherdirectlyorindirectly,systemsdealingwithgroupcommunic
ationneedtobeabletoextracttheinforma-tiontheyrequirefromnaturalgroupdiscourse,am
uchharderproposition.Collaborativesystemsarefurthermorecharacterizedbytheirvulne
rabilitytochangesinworkpractices,whichoftenresultfromtheintroductionoftechnology
(Grudin,1988).Asaconsequence,interruptionsbyasystemlookingforexplicitconfirmatio
nofpotentiallyerroneousinterpretationsmayprovetoodisruptive.Thisinturnrequiresth
edevelopmentofnewapproachestosystemsupportthatarerobusttomisrecognitionsanddonot
interferewiththenaturalflowofgroupinteractions(Kaiser&Barthelmess,2006).Automati
cextractionofmeetinginformationtogeneraterichtranscriptshasbeenoneofthefocusarea
sinmultimodalmeetinganalysisresearch.Thesetran-scriptsmayincludevideo,audio,andn
otesprocessedbyanalysiscomponentsthatproducetranscripts(e.g.,fromspeech)andprovi
desomedegreeofsemanticanalysisoftheinteraction.Thisanalysismaydetectwhospokewhen
(whichissome-timescalledspeakerdiarization)(VanLeeuwen&Huijbregts,2006),whattopics
werediscussed(Purveretal.,2006),thestructureoftheargumentation(Verbree,Rienks,&H
eylen,2006b),rolesplayedbytheparticipants(Banerjee&Rudnicky,2004),actionitemstha
twereestablished(Purver,Ehlen,&Niekrasz,2006),structureofthedialog(Verbree,Rienk
s,&Heylen,2006a),andhigh-levelturnsofameeting
12MultimodalInterfaces:CombiningInterfaces400

(McCowanetal.,2005).Thisinformationisthenmadeavailableforinspectionviameetingbro
wsers(Ehlen,Purver,&Niekrasz,2007;Nijholtetal.,2006).Someofthisworkstillreliespr
imarilyontheanalysisofspeechsignals,althoughsomeadditionalworkfocusesonincorpora
tionofadditionalmodalities.Multimodalcollaborativesystemshavebeenimplementedtomo
nitorusersactionsastheyplanaschedule(Barthelmessetal.,2005;Kaiseretal.,2004).Othe
rsystemsobservepeoplediscussingphotosandautomaticallyextractfrommulti-modallangu
agekeyterms(tags)usefulfororganizationandretrieval(Barthelmessetal.,2006).Pianes
iandcolleagues(2006)andZancanaroandcoworkers(2006)haveproposedsystemsthatanalyze
collaborativebehaviorasagroupdiscusseshowtosurviveinadisasterscenario.Thesesyste
mspassivelymonitorgroupinter-actions,analyzespeech,handwriting,andbodymovements,
andprovideresultsaftertheobservedmeetingshaveconcluded.TheChartersystem(Kaiseret
al.,2004,2005),derivedfromtheQuicksetsys-tem(Cohenetal.,1997),observesuseraction
swhilebuildingaGanttchart.Thisdiagram,astandardrepresentationofschedules,isbased
onplacinglines,repre-sentingtasks,andmilestones,representedbydiamondshapes,withi
natemporalgrid.UsersbuildachartbysketchingitonaninstrumentedwhiteboardoraTabletP
C(Figure12.5).Sketchrecognitionidentifiesthegraphicalsymbols.Amultimodalapproach
tolabelrecovery(Kaiser,2006)matcheshandwrittenlabelstotheredundantspeechinthetem
poralvicinityofthehandwriting,asuserssay,forinstance,Theprototypeneedstobereadyby
thethirdweek<writeprototypenexttomilestonesymbol>.Themutualdisambiguationprovided
bythisredun-dantspeechandhandwritingaccountsforupto28percentimprovementinlabelre
cognitionintestcases(Kaiseretal.,2007).Oncethemeetingisover,thesystemautomatical
lyproducesaMS-Projectrenditionofthesketchedchart.TheChartersystemhasbeenmorerece
ntlyextendedtosupportremotecol-laborationbylettinguserswhoarenotcolocatedvisuali
zesketchesmadeataremotesiteandcontributeelementstothesharedGanttchart,byusingast

ylusonaTabletPCorsheetsofdigitalpaper(Barthelmessetal.,2005).Thesupportforremote
collaborationalsoincorporatesastereovisionbasedgesturerecognizerMITsVTracker(Demird
jian,Ko,&Darrell,2003).Thisrecognizeridentifiesnat-uralgesturesmadetowardthewhit
eboardonwhichtheGanttchartissketched,andautomaticallypropagatesthegestureasahalo
sothatremoteparticipantsbecomeawareofpointingperformedatremotesites.Themultimoda
lphotoannotatordescribedbyBarthelmess,Kaiser,andcol-leagues(2006)exploitssocials
ituationsinwhichgroupsofpeoplegettogethertodiscussphotostoautomaticallyextractta
gsfromthemultimodalcommunica-tiontakingplaceamonggroupmembers.Thesetagsarerepres
entativetermsthatareusedtoidentifythesemanticcontentofthephotos.Tagscanbeusedtop
er-formclusteringortofacilitatefutureretrieval.Byleveragingtheredundancybetweenh
andwrittenandspokenlabels,thesystemisabletoautomaticallyextractsomeofthesemeanin
gfulretrievalterms(Kaiser,2006;Kaiseretal.,2007).
12.2TechnologyoftheInterface401

Automatictagextractionaddressesthereluctancemanyusershaveofspendingtimeperformin
gmanualannotationofdigitalphotos.Multimodalgroupcommunicationhasalsobeenusedtoan
alyzegroupbehav-iorinmeetings(McCowanetal.,2005;Pianesietal.,2006;Rienks&Heylen,
2005;Wainer&Braga,2001;Zancanaroetal.,2006).PianesiandZancanaroandrespec-tivecol
leagueshavebeenexploringamultimodalapproachtotheautomaticdetectionandclassificat
ionofsocial/relationalfunctionalrolesassumedbypeopleduringmeetings.Detectedroles
donotcorrespondtothehierarchicalrolesparti-cipantsmayhaveinanorganization,butrat
herreflectattitudesduringameeting.Twomaincategoriesareidentified:taskandsocial-e
motional.Whereastheformerhastodowithfacilitationandcoordinationoftaskperformance
(e.g.,thedefini-tionofgoalsandprocedures),thelatterisconcernedwiththerelationshi
pamonggroupmembers(e.g.,moderationofgroupdiscussionsorcooperativeattitudetowardo
therparticipants).Usinginformationaboutspeechactivity(whoistalkingtowhomateachmo
ment)andlocalizedrepetitivemotions(fidgeting)obtainedfromavisualchannel,thebehav
iorofparticipantsisclassifiedwithinfivetaskrolesandfivesocialemotionalroles(Zanca
naroetal.,2006).Theultimateproductofthis
FIGURE12.5Agroupofcolocatedandremoteparticipants(inset)usingChartertodefineaproj
ectschedule.Sketchesoverasharedspacearepropagatedtoremoteusers,whocanaddtheircon
tribution.ChartermonitorsthisinteractionandproducesanMS-Projectchartattheendofth
emeeting.
12MultimodalInterfaces:CombiningInterfaces402

systemisasemiautomatedcoachthatpresentstoparticipantsepisodesofthemeetingduringwhi
chdysfunctionalbehaviorisdetected(e.g.,dominantoraggressivebehavior),withthegoal
ofimprovingmeetingparticipationbehaviorovertimebyallowingparticipantstoreflectup
ontheirownactions.Otherworklooksintotheroleofspeechamplitude,lexicalcontent,andg
azetoautomaticallydetectwhotheintendedaddresseeisininteractionsinvolv-inggroupso
fpeopleandcomputationalassistants(Jovanovic,opdenAkker,&Nijholt,2006;Katzenmaier
,Stiefelhagen,&Schultz,2004;Lunsford,Oviatt,&Arthur,2006;Lunsford,Oviatt,&Coulst
on,2005;vanTurnhoutetal.,2005).Speechamplitudeisfoundtobeastrongindicatorofwhopa
rticipantsintendtoaddressinsituationsinwhichacomputationalassistantisavailable(L
unsfordetal.,2005,2006).Directivesintendedtobehandledbythecomputeraredeliv-eredw
ithamplitudesignificantlyhigherthanspeechdirectedtohumanpeers,asindicatedbystudi
esofusersengagedinaneducational-problemsolvingtask.Leveragingsuchatechnique,asyst
emisabletoautomaticallydeterminewhetherspecificspokenutterancesshouldbeinterpret
edascommandsrequiringaresponsefromthesystem,separatingthesefromtheremainingconve
rsationintendedtoberespondedtobyhumanpeers.Thisopen-microphoneengagementproblemi
soneofthemorechallengingbutfundamentalissuesremainingtobesolvedbynewmultimodalco

llaborativesystems.12.2.2ConceptsandMechanismsAprimarytechnicalconcernwhendesign
ingamultimodalsystemisthedefinitionofthemechanismusedtocombineorfuseinputrelatedto
multiplemodalitiessothatacoherentcombinedinterpretationcanbeachieved.Systemssuch
asBolts(1980)Putthatthereandotherearlysystemsmainlyprocessedspeech,andusedgesturesj
usttoresolvex,ycoordinatesofpointingevents.Systemsthathandlemodalitiessuchasspee
chandpen,eachofwhichisabletoprovideseman-ticallyrichinformation,orspeechandlipmo
vements,whicharetightlycorrelated,requireconsiderablymoreelaboratefusionmechanis
ms.Thesemechanismsincluderepresentationformalisms,fusionalgorithms,andentirelyne
wsoftwarearchitectures.Multimodalfusionemergesfromtheneedtodealwithmultiplemodal
itiesnotonlyasindependentinputalternatives,butasalsoabletocontributepartsoreleme
ntsofexpressionsthatonlymakesensewheninterpretedsynergistically(Nigay&Coutaz,199
3).WhenausertracesalineusingapenwhilespeakingEvacuationroute,amultimodalsystemisre
quiredtosomehowcomposetheattributesofthespatialcomponentgivenviathepenwiththemea
ningassignedtothislineviaspeech.Awell-designedmultimodalsystemoffers,totheextent
possible,thecapabil-ityforcommandstobeexpressiblethroughasinglemodality.Userssho
uldbeabletospecifymeaningsusingonlythepenorusingjustspeech.Itmustbenoted,
12.2TechnologyoftheInterface403

though,thatspecifyingspatialinformationviaspeechisnotpreferredbyusersingeneral(O
viatt,1997;Oviatt&Olsen,1994;Suhm,1998).Itmayberequiredinsituationsinwhichusersa
reconstrainedtousingspeech(e.g.,whiledriving).EarlyandLateFusionInordertobeablet
ointegrateinterpretationsgeneratedbymultipledisparateunimodalrecognizers,thesein
terpretationsneedtoberepresentedinacommonsemanticrepresentationformalism.Amultim
odalintegrationelementthenana-lyzestheseuniformsemanticrepresentationsanddecides
howtointerpretthem.Inparticular,itdecideswhethertofusemultipleindividualelements
,forexampleaspokenutteranceandsomepointinggesturesorhandwriting,orwhethersuchinp
utshouldbeconsideredtorepresentindividualcommandsthatshouldnotbecomposedintoacom
binedinterpretation.Intheformercase,wesaythatamultimodalinterpretationwasapplied
,andinthelatterthattheinterpretationwasunimodal.Twostrategiesthathavebeenexplore
dtodealwiththecombinationofmod-alitiesarelate,orfeaturefusion,andearly,orsemanti
cfusion.Early-fusiontechniquesareusuallyemployedwhenmodalitiesaretightlycoupled,
asisthecaseforinstanceforspeechandlipmovements.Earlyfusionisachievedbytrainingas
inglerecognizeroveracombinationoffeaturesprovidedbyeachofthemodalities.Suchfeatu
resmaybecomposedfromsomerepresenta-tionofphonemesfromaspeechmodalityandvisemesch
aracterizinglipmove-mentsprovidedbyavisualmodality.Theadvantageofearlyfusionisth
atparallelinputfromeachmodalityisusedconcurrentlytoweightinterpretationhypothese
sastherecognitiontakesplace.Thisresultsinmanycasesinenhancedrecognitionaccuracy,
evenwhenoneofthesignalsiscompromised(e.g.,speechinnoisyenvironments)(Dupont&Luet
tin,2000;Meieretal.,1996;Potamianosetal.,2004;Rogozan&Deleglise,1998;Sumby&Pollac
k,1954;Vergo,1998).Late-orsemantic-fusiontechniquesemploymultipleindependentreco
gni-zers(atleastonepermodality).Asequentialintegrationprocessisappliedovertherec
ognitionhypothesesgeneratedbythesemultiplerecognizerstoachieveacombinedmultimoda
linterpretation.Latefusionisusuallyemployedwhentwoormoresemanticallyrichmodaliti
essuchasspeechandpenareincorporated.Thisapproachworkswellinsuchcasesbecauseeachr
ecognizercanbetrainedindependentlyoverlargeamountsofunimodaldata,whichisusuallym
orereadilyavailable,ratherthanrelyingontypicallymuchsmallermultimodalcorpora(set
sofdata).Giventhedifferencesininformationcontentandtimescalecharacteristicsofnon
coupledmodalities,theamountoftrainingdatarequiredtoaccountforthewidevarietyofcom
binationsusersmightemploywouldbeprohi-bitivelylarge.Asystemwouldforinstanceneedt
obetrainedbyalargenumberofpossiblevariationsinwhichpenandspeechcouldbeusedtoexpr
esseverysinglecommandacceptedbyasystem.Givenindividualdifferencesinstyleanduse
12MultimodalInterfaces:CombiningInterfaces404

conditions,thenumberofalternativesispotentiallyverylarge,particularlyforsystemss
upportingexpressivemodalitiessuchaspenandspeech.Byemployingindependentunimodalre
cognizers,systemsthatadoptalate-fusionapproachareabletouseoff-the-shelfrecognize
rs.Thatmakesitpossibleforlate-fusionsystemstobemoreeasilyscaledandadaptedasindiv
idualrecogni-zersarereplacedbynewer,enhancedones.Intherestofthisdiscussion,wecon
centrateprimarilyonlate-fusionmechanisms.12.2.3InformationFlowIngeneralterms,ina
multimodalsystemmultiplerecognizersprocessinputgen-eratedbysensorssuchasmicropho
nes,instrumentedwritingsurfacesandpens,andvision-basedtrackers(Figure12.6).These
recognizersproduceinterpretationhypothesesabouttheirrespectiveinput.TemporalCons
traintsTemporalconstraintsplayanimportantroleisdecidingwhethermultipleuni-modale
lementsaretobeintegratedornot(Bellik,1997;Johnston&Bangalore,2005;Oviatt,DeAngel
i,&Kuhn,1997).Inmostsystems,afixedtemporalthresh-olddetermines,forinstance,wheth
eraspokenutteranceistobefusedwithpengestures,orwhethermultiplepengesturesshouldb
efusedtogether.Ingeneral,ifinputfromdifferentmodalitieshasbeenproducedwithintwot
ofoursecondsofeachother,integrationisattempted.Inpracticalterms,thatmeansthatsys
-temswillwaitthismuchtimeforuserstoprovideadditionalinputthatmightbepotentiallyu
sedincombinationwithaprecedinginput.Whilethisapproachhasbeensuccessfullyusedinsy
stemssuchasQuickset(Cohenetal.,1997)andMATCH(Johnston&Bangalore,2005),itsuseintr
oducesdelaysinprocessingusercommandsthatequalthechosenthreshold.Morerecently,lea
rning-basedmodelshavebeendevelopedthatadapttouser-specificthresholds(Huang&Oviat
t,2005).Byemployinguser-specificmodelsthattakeintoaccounttheempiricalevidenceofm
ultimodalproduction,systemdelaysareshowntobereduced40to50percent(Gupta&Anastasak
os,2004;Huang,Oviatt,&Lunsford,2006).ResponsePlanningIngeneral,amultimodalintegr
ationelementproducesapotentiallylargenumberofinterpretations,someofwhichmaybemul
timodalandothersmaybeunimo-dal.Themostlikelyinterpretationhypothesisisthenchosen
asthecurrentinputinterpretation,andtheotherlesslikelyonesareremovedfromconsidera
tion.Thischoiceissometimesinfluencedbyadialogmanager,whichmayexploitadditionalco
ntextualinformationtoselectthemostappropriateinterpretation(Johnston&Bangalore,2
005;Wahlster,2006).
12.2TechnologyoftheInterface405

Onceaninterpretationischosen,asystemresponsemaybegenerated.Responseplanningdepen
dsveryheavilyonthespecificsofeachsystemanditsdomainandintendedfunctionality.Ares
ponsemayconsistofmultimodal/multime-diadisplayduringwhichusersarepresentedwithgr
aphicalandaudio/spokenout-putonacomputerorwithinavirtual-realityenvironment,some
timesembodiedasananimatedcharacter(Casselletal.,2000;Nijholt,2006).Otherresponse
smayincludeinteractingwithanotherapplication,suchasupdatingamilitaryplanninginfo
rmationsystemordrivingsimulation(Cohen&McGee,2004;Johnstonetal.,1997),updatingan
MS-Projectschedule(Kaiseretal.,2004),orsupportingremotecollaboration(Barthelmess
etal.,2005).Aconsiderableamountofworkonplanningandpresentationhasbeenpur-suedbys
ystemsandframeworkssuchasMAGIC(Dalaletal.,1996),WIP(Wahlsteretal.,1993),PPP(Andr
e,Muller,&Rist,1996),andSmartKom(Wahlster,2006).
Feature/framestructuresFeature/frame structuresGestureUnderstandingGestureRecogn
itionMultimodal IntegrationDialog ManagerApp1App2App3Response PlanningGraphicsVR
TTSApplication Invocationand CoordinationFeature/frame structuresContextManageme
ntPenGloveLaserTouchMicrophoneSpeechRecognitionNatural-LanguageProcessing
FIGURE12.6Genericconceptualarchitectureofamultimodalsystem.Multiplerecognizersar
eusedforgestureandspeechinput.Source:FromOviattetal.(2000).
12MultimodalInterfaces:CombiningInterfaces406

12.3CURRENTIMPLEMENTATIONSOFTHEINTERFACEEventhoughfewcommercialmultimodalapplica
tionsareavailable,multimodal-ityhasbeenexploredinavarietyofdifferentareasandform
ats,includinginformationkiosks,mobileapplications,andultramobileapplicationsbase
dondigital-papertechnology.Multimodalinterfaceshavealsobeenusedtopromoteaccessib
ility.Thissectionexaminessomerepresentativeexamplesofinterfacesineachofthesearea
sandformfactors.12.3.1InformationKiosksKiosksprovidingavarietyofservices,suchast
ouristandmuseuminformation,banking,airportchecking,andautomatedcheck-outinretail
stores,arebecomingmoreprevalent.Themajorityofthesedevicesusetouchandkeypadasdata
entrymechanisms,producingresponsesviagraphicaldisplaysandsometimesspeechandaudio
.Giventherequirementsforrobustnessandspaceconstraints,key-boardsandmicearenotusu
allyavailable(Johnston&Bangalore,2004).Multi-modalinterfacesprovideadditionalmea
nsofinteractionthatdonotrequirekeyboardormiceaswell,andmaythereforeprovideanidea
linterfaceoptionforkiosks.Inthissection,somerepresentativeexamplesarepresented.M
ATCHKioskMATCHKiosk(Johnston&Bangalore,2004)isamultimodalinteractivecityguidefor
NewYorkCityandWashington,DC,providingrestaurantandsubway/metroinformation.Thekio
skimplementationisbasedonthemobileMATCHmultimodalsystem(Johnstonetal.,2002).This
interactiveguideallowsuserstointeractviaspeech,pen,andtouch.Responsesarealsomult
imodal,presentingsynchronizedsyntheticspeech,alife-likevirtualagent,anddynamical
lygeneratedgraphics(Figure12.7).Thesystemhelpsusersfindrestaurantsbasedonlocatio
n,price,andtypeoffoodserved.Ausermayaskforinformationusingspeech,asinFindmemodera
telypricedItalianrestaurantsinAlexandria.Thesamequerycanbeexpressedmultimod-ally,
forinstancebyspeakingModerateItalianrestaurantsinthisarea<circleareaonthemap>,orby
usingjustthepen,forinstancebycirclinganareainthemapandhandwritingCheapItalian.Ital
soprovidessubwaydirectionsbetweenlocations,forinstance,whenauserasks,HowdoIgetfro
mhere<pointtomaplocation>tohere?<pointtomaplocation>,orjustcirclesaregionwiththep
enandhandwritesRoute.Systemoutputcombinessyntheticspeechsynchronizedwiththeanimate
dcharactersactionsandcoordinatedwithgraphicalpresentations.Thelattermay
12.3CurrentImplementationsoftheInterface407

consistofautomaticpanningandzoomingofthemapportionofthedisplay,suchasshowingrout
esegments.Avarietyofmechanismsareimplementedtoletusersfine-tunetheirqueriesandco
rrectmisrecognitions.Anintegratedhelpmechanismisalsoavailable,andisautomatically
activatedwhenrepeatedinputfailuresaredetected.SmartKom-PublicSmartKomisaflexible
platformthatincludesfacilitiesforhandlingmulti-modalinputandoutputgeneration(Wah
lster,2006).SmartKom-Publicisakioskinstantiation(Figure12.8)ofthearchitecture(Ho
rndasch,Rapp,&Rottger,2006;Reithingeretal.,2003;Reithinger&Herzog,2006).Thiskios
k,mountedinsideatelephonebooth,providesgestureandfacialrecognitionviacameras,mic
rophone,graphicaldisplay,andaudiooutput.Thekioskprovidesinformationaboutmoviesin
thecityofHeidelberg,Germany,aswellascommunicationfacilities,suchasdocumenttransm
ission.Thisdiscussionwillfocusonthemovieinformationfunctionality.Ausermayiniti-a
teadialogwiththesystembyspeaking,Whatsplayingatthecinemastonight?Thesystemrespondsb
yshowingalistingofmovies,aswellasamapdisplayingthelocationsofthetheaters(Figure1
2.9onpage410).AnanimatedcharacterSmartakusprovidesspokeninformationsuchasTheseareth
emoviesplayingtonight.Thecinemasaremarkedonthemap.Userscanthenaskforspecificinfor
-mationmultimodallyGivemeinformationaboutthisone<pointtothedisplay>forexamplecausing
thesystemtodisplaythetextshowingdetails
(a)
(b)
FIGURE12.7MATCHKiosk.(a)Hardware.(b)Userinterface.Source:FromJohnstonandBangalor
e(2004).(CourtesyAssociationofComputationalLinguistics.)
12MultimodalInterfaces:CombiningInterfaces408

oftheselectedmovie.Thesystemalsoprovidesticketreservations,supportedbydialogsinw
hichusersmaychooseseats.Finally,thesystemisabletopro-videinformationabouthowtoge
ttoaspecificlocation,suchasaparticularmovietheater.12.3.2MobileApplicationsMulti
modalinterfacesprovideanaturalwaytoovercometheintrinsiclimitationsofdisplays(usu
allysmall)andinputmechanismsofmobiledevices.Theabilitytousespeechorpenisalsoimpo
rtantinaccommodatingthevarietyofmobileusecontexts.Speechcanbeused,forinstance,wh
ilewalkingordriving;apeninterface
FIGURE12.8SmartKom-Public.Thisplatformisakioskinstantiationmountedinatelephonebo
oth.Fromhttp://www.smartkom.org/eng/project_en_frames.pl?public_en.html.(Courtes
yGermanResearchCenterforArtificialIntelligenceGmbHo.)
12.3CurrentImplementationsoftheInterface409

mayprovidetheprivacyrequiredinpublicsettingsinwhichspeechisnotappro-priate.Thech
allengeindevelopingmobilemultimodalinterfacesisinbuildingsystemsthatarecompatibl
ewiththelow-poweredprocessingunitsfoundinmostmobiledevices.Thefollowingtwointerf
acesillustratethecharacteristicsandapproachesusedbymobilemultimodalsystems.Micro
softMiPadMicrosoftdevelopedamultimodalmobileapplicationMiPad(MultimodalInter-acti
veNotepad,http://research.microsoft.com/srg/mipad.aspx)(Dengetal.,2002;Huangetal
.,2001)thatdemonstratespenandspeechinputonaportabledigitalassistant(PDA).Peninput
isusedtoselecticonsandtoactivatevoicerecognitionviaatap-to-talksoftwarebutton(Figu
re12.10).Thesysteminterpretsspokencommands,andisabletoinitiatee-mail,setupappoin
tments,andmanagecontactlists.Usingthissystem,usersareabletosay,forexample,Sendmai
ltoNicky,causingthesystemtoopenupane-maildialogwiththerecipientalreadyfilledout(K
.Wang,2004).Theusercanthendictatethecontentofthemessage.Userscanalsotapwiththest
ylusonaformfield,suchastheTofieldofane-mail.Thisprovidesthesystemwithcontextualinf
ormationthatwillhelpthesystemselectinterpretations,asin,forexample,preferringHele
naBayertoHellothereasaninterpretationwhenane-mailRecipientfield
FIGURE12.9SmartKom-Publicinterface.Theinterfacepresentsalisting(inGerman)ofthelo
cationsshowingTerminator3.Source:FromReithingerandHerzog(2006);courtesyofSpringe
r.
12MultimodalInterfaces:CombiningInterfaces410

isselected.Finally,correctioncanbeperformedmultimodallybyselectingamisrecognized
wordbytappingwiththestylusandthenspeakingitagain.KirusasMultimodalInterfaceKirusa
offersaplatformforthedevelopmentofmultimodalmobileapplicationsintendedforwireles
sphoneproviders.Itsfunctionalityisillustratedbyasportssampleapplication(http://w
ww.kirusa.com/demo3.htm).Thisapplicationpro-videsaninterfacestructuredasmenusand
formsthatcanbeactivatedandfilledoutmultimodally.Menuitemscanbeselectedbypentapor
byspeech.Formfieldscanbeselectedviatapsandfilledviaspeech;fieldscanalsobeselecte
dbyspeakingthenameofthefield.Besidesthestructuredgraphicalinformation(menusandfo
rms)theinterfacepresentssynthesizedspeech,hyperlinks,andvideos.Multimodalinputis
demon-stratedbyvideocontrolcommands,suchaszoomingbyissuingthecommandZoominhere<ta
ponlocation>whileavideoisbeingdisplayed(Figure12.11).12.3.3UltramobileDigital-Pap
erBasedInterfacesDespitethewidespreadintroductionoftechnology,aconsiderablenumber
ofusersstillprefertooperateonpaper-baseddocuments.Ratherthandiminishing,theworld
wideconsumptionofpaperisactuallyrising(Sellen&Harper,2003).Paperislightweight,hi

ghdefinition,highlyportable,androbustpaperstillworksevenaftertornandpunctured.Paper
doesnotrequirepowerandisnot
FIGURE12.10MicrosoftsMiPad.Theinterface,displayingiconsandaformthatcanbeactivated
multimodally.Fromhttp://research.microsoft.com/srg/mipad.aspx.(CourtesyMicrosoft
.)
12.3CurrentImplementationsoftheInterface411

subjecttosystemcrashes,whichmakesitsuseidealinsafety-criticalscenarios(Cohen&McGee
,2004).Theseaspectsmakethismediumhighlycompatiblewithmobilefielduseinharshcondit
ions.NISMapisacommercialmultimodalapplicationproducedbyAdapx(http://www.adapx.co
m)thatcapturesstandardmilitaryplanfragmentsanduploadstheseplanstostandardmilitar
ysystemssuchasCPOF.Officersoperatethesys-tembyspeakingandsketchingonapapermap(Fi
gure12.12).Thesysteminter-pretsmilitarysymbolsbyfusinginterpretationsofasketchre
cognizerandaspeechrecognizer.Ausermayforinstanceaddabarbed-wirefencemultimodally
bydrawingalineandspeakingBarbedwire.Asanalternative,thiscommandcanbeissuedusingske
tchonly,bydecoratingthelinewithanalphasymbol.Themultimodalintegrationtechnologyuse
dbyNISMapisbasedonQuickset(Cohenetal.,1997).NISMapusesdigitalpaperandpentechnolo
gydevelopedbyAnoto(2007).Anoto-enableddigitalpaperisplainpaperthathasbeenprinted
withaspecial
FIGURE12.11Kirusassportsdemo.Thedemoshowstheselectionofalocationwhiletheusercomma
ndstheinterfacetozoominviaspeech.Fromhttp://www.kirusa.com/demo3.htm.(CourtesyKiru
sa,Inc.)
12MultimodalInterfaces:CombiningInterfaces412

pattern,likeawatermark.Thepatternconsistsofsmalldotswithanominalspacingof0.3mm(0
.01inch).Thesedotsareslightlydisplacedfromagridstruc-turetoformtheproprietaryAno
topattern.AusercanwriteonthispaperusingapenwithAnotofunctionality,whichconsistso
faninkcartridge,acamerainthepenstip,andaBluetoothwirelesstransceiversendingdatato
apaireddevice.Whentheuserwritesonthepaper,thecameraphotographsmovementsacrossthe
gridpatternandcandeterminewhereonthepaperthepenhastraveled.InadditiontotheAnotog
rid,whichlookslikealightgrayshading,thepaperitselfcanhaveanythingprintedonitusin
ginksthatdonotcontaincarbon.12.3.4ApplicationsoftheInterfacetoAccessibilityMulti
modalinterfaceshavethepotentialtoaccommodateabroaderrangeofusersthantraditionali
nterfaces.Theflexibleselectionofmodalitiesandthecontroloverhowthesemodalitiesare
usedmakeitpossibleforawiderrangeofuserstobenefitfromthiskindofinterface(Felletal
.,1994).Thefocusonnaturalmeansofcommu-nicationmakesmultimodalinterfacesaccessibl
etousersofdifferentages,skilllevels,native-languagestatus,cognitivestyles,sensor
yimpairments,andothertemporaryillnessesorpermanenthandicaps.Speechcanforinstance
bepreferredbyuserswithvisualormotorimpairments,whilehearingimpairedorheavilyacce
ntedusersmayprefertouch,gesture,orpeninput(Oviatt,1999a).Somemultimodalsystemsha
veaddressedissuesrelatedtodisabilitydirectly.BellikandBurger(1994,1995)havedevel
opedmultimodalinterfacesfortheblind.Thisinterfacesupportsnonvisualtextmanipulati
onandaccesstohyperlinks.
FIGURE12.12NISMapapplicationusingadigitalpenandpaper.Source:FromCohenandMcGee(20
04);courtesyACM.
12.3CurrentImplementationsoftheInterface413

TheinterfacecombinesspeechinputandoutputwithaBrailleterminalandkey-board.Usingth
isinterface,userscanpointtoplacesintheBrailleterminalandcommandthesystemviaspeec
htounderlinewordsortoselecttextthatcanthenbecut,copied,andpasted(Figure12.13).Mu
ltimodallocomotionassistancedeviceshavealsobeenconsidered(Bellik&Farcy,2002).Ass
istanceisprovidedviahapticpresentationofreadingscom-ingfromalasertelemeterthatde
tectsdistancestoobjects.Hinaetal.(Hina,Ramdane-Cherif,&Tadj,2005)presentamultimo
dalarchitecturethatincorpo-ratesausermodelwhichcanbeconfiguredtoaccountforspecif
icdisabilities.Thismodelthendrivesthesysteminterpretationstoaccommodateindividua
ldifferences.FacetopTablet(Miller,Culp,&Stotts,2006;Stottsetal.,2005)isaninterfa
cefortheDeaf.Thisinterfaceallowsdeafstudentstoaddhandwrittenandsketchednotes,whi
charesuperimposedbyatranslucentimageofasigninginterpreter(Figure12.14).Deafstude
ntsareabletocontinuetomonitorongoinginterpreta-tionwhilestillbeingabletotakenote
s.Conventionalsystemswouldrequirestudentstomovetheireyesawayfromtheinterpreter,w
hichcausessegmentsoftheinterpretationtobelost.Otherprojects,suchastheVisicastand
theeSignProjects(http://www.visicast.sys.uea.ac.uk/)addresssignlanguageproductio
nviaananimatedcharacterdrivenbyasigndescriptionnotation.Theobjectiveoftheseproje
ctsistoprovidedeafusersaccesstogovernmentservicesforinstance,inapostoffice(Bangha
metal.,2000)oronwebsites.Verlindenandcol-leagues(2005)employedsimilaranimationte
chniquestoprovideforamultimodal
FIGURE12.13Brailleterminal.Theterminalisinusewithamultimodalinterfacefortheblind
.Source:FromBellikandBurger(1995);courtesyRESNA.
12MultimodalInterfaces:CombiningInterfaces414

educationalinterface,inwhichgraphicalandtextualinformationispresentedinsynchrony
withsigningproducedbyananimatedcharacter.12.4HUMANFACTORSDESIGNOFTHEINTERFACEMul
timodalinterfacedesigndependsonfindingbalancebetweentheexpressivepowerandnatural
nessofthelanguagemadeavailabletousersontheonehandandtherequirementsofprocessabil
ityontheotherhand.Therecognitiontech-nologythatmultimodalsystemsdependonischarac
terizedbyambiguitiesanduncertainty,whichderivedirectlyfromtheexpressivenessofcom
municationthatisoffered.Currentrecognitiontechnologyisconsiderablylimitedwhencom
paredtohuman-levelnaturallanguageprocessing.Particularlyfornaveusers,thefactthatas
ystemhassomenaturallanguagecapabilitiesmayleadtoexpectationsthataretoohighforasy
stemtofulfill.Thechallengeindesigningmultimodalinterfacesconsiststhereforeoffind
ingwaystotransparentlyguideuserinputinawaythatagreeswiththelimitedcapabil-itieso
fcurrenttechnology.Empiricalevidenceaccumulatedoverthepastdecadeshasprovidedawid
evarietyofimportantinsightsintohowusersemploymultimodallanguagetoperformtasks.Th
esefindingsareoverviewedinSection12.4.1,inwhich
FIGURE12.14FacetopTablet.Handwrittennotesaresuperimposedwiththeimageofasigningin
terpreter.Source:FromStottsetal.(2005).
12.4HumanFactorsDesignoftheInterface415

linguisticandcognitivefactorsarediscussed.Section12.4.2thenpresentstheprinci-ple
sofahuman-centereddesignprocessthatcanguideinterfacedevelopment.12.4.1Linguistic
andCognitiveFactorsHumanlanguageproduction,onwhichmultimodalsystemsdepend,involv
esahighlyautomatizedsetofskills,manyofwhicharenotunderfullconsciouscon-trol(Ovia
tt,1996b).Multimodallanguagepresentsconsiderablydifferentlinguisticstructurethan
unimodallanguage(Oviatt,1997;Oviatt&Kuhn,1998).Eachmodal-ityisusedinamarkedlydif
ferentfashion,asusersself-selecttheiruniquecapabil-itiestosimplifytheiroverallco

mmands(Oviattetal.,1997).Individualdifferencesdirectlyaffecttheforminwhichmultim
odalconstructsaredelivered(Oviattetal.,2003;Oviatt,Lunsford,&Coulston,2005;Xiao,
Girand,&Oviatt,2002),influencingtimingandpreferenceforunimodalormultimodaldelive
ry(Epps,Oviatt,&Chen,2004;Oviatt,1997;Oviattetal.,1997;Oviatt,Lunsford,&Coulston
,2005).IndividualDifferencesinTemporalIntegrationPatternsAccumulatedempiricalevi
dencedemonstratesthatusersemploytwoprimarystylesofmultimodalintegration.Thereare
thosewhoemployapredominantlysequentialintegrationpattern,whileothersemployapredo
minantlysimulta-neouspattern(Oviattetal.,2003;Oviatt,Lunsford,&Coulston,2005;Xia
oetal.,2002).Themaindistinguishingcharacteristicofthesetwogroupsisthatsimultaneo
usintegratorsoverlapmultimodalelementsatleastpartially,whilesequentialintegrator
sdonot.Inaspeechpeninteraction,forexample,simulta-neousintegratorswillbeginspeaki
ngwhilestillwriting;sequentialintegrators,ontheotherhand,willfinishwritingbefore
speaking(Figure12.15).Thesepatternshavebeenshowntoberemarkablyrobust.Userscaninm
ostcasesbecharacterizedaseithersimultaneousorsequentialintegratorsbyobservingthe
irveryfirstmultimodalcommand.Dominantintegrationpatternsarealsoverystable,with88
to97percentofmultimodalcommandsbeingconsis-tentlydeliveredaccordingtothepredomin
antintegrationstyleovertime(Oviatt,Lunsford,&Coulston,2005).Thesestylesoccuracro
ssagegroups,includingchildren,adults,andtheelderly(Oviattetal.,2003),withinavari
etyofdifferentdomainsandinterfacestyles,includingmap-based,realestate,crisismana
ge-ment,andeducationalapplications(Oviattetal.,1997;Xiaoetal.,2002;Xiaoetal.,200
3).Integrationpatternsarestrikinglyresistanttochangedespiteexplicitinstructionan
dselectivereinforcementencouraginguserstoswitchtoastylethatisnottheirdominantone
(Oviatt,Coulston,&Lunsford,2005;Oviattetal.,2003).Onthecontrary,thereisevidencet
hatusersfurtherentrenchintheirdominantpatternsduringmoredifficulttasksorwhendeal
ingwitherrors.Inthesemoredemandingsituations,sequentialintegratorswillfurtherinc
reasetheirintermodallag,andsimultaneousintegratorswillmoretightlyoverlap.
12MultimodalInterfaces:CombiningInterfaces416

Integrationpatternsarealsocorrelatedtootherlinguisticandperformanceparameters.Se
quentialintegratorspresentmoreprecisearticulationwithfewerdisfluencies,adoptinga
moredirectcommand-stylelanguagewithsmallerandlessvariedvocabulary.Sequentialinte
gratorsalsomakeonlyhalfasmanyerrorsassimultaneousintegrators(Oviatt,Lunsford,&Co
ulston,2005).Thesestrongindividualdifferencesinpatternsmaybeleveragedbyasystemto
betterdeterminewhichelementsshouldbefusedandtheappropriatetimetoperforminterpret
ation(Section12.2.3).Asystemcanreducetheamountoftimethatitwaitsforadditionalinpu
tbeforeinterpretingacommandwhendealingwithsimultaneousintegrators,givenpriorknow
ledgethattheseusersprimarilyover-laptheirinputs.Morerecently,integrationpatterns
havebeenexploitedwithinamachinelearningframeworkthatcanprovideasystemwithrobustp
redictionofthenextinputtype(multimodalorunimodal)andadaptivetemporalthresholdsba
sedonusercharacteristics(Huangetal.,2006).RedundancyandComplementarityMultimodal
inputoffersopportunitiesforuserstodeliverinformationthatiseitherredundantacrossm
odalities,suchaswhenauserspeaksthesamewordsheishandwriting,orcomplementary,sucha
swhendrawingacircleonamapandspeaking,Addhospital.Empiricalevidencedemonstratesthat
thedominantthemeinusersnaturalorganizationofmultimodalinputtoasystemiscomple-ment
arityofcontent,notredundancy(McGurk&MacDonald,1976;Oviatt,2006b;Oviattetal.,1997
;Oviatt&Olsen,1994;Wickens,Sandry,&Vidulich,1983).Usersnaturallymakeuseofthediff
erentcharacteristicsofeachmodality
Lets have an evacuation routeMake a route0.00.51.01.52.0Time (seconds)2.53.03.5
FIGURE12.15Modeloftheaveragetemporalintegrationpatternforsimultaneousandsequenti
alintegratorstypicalconstructions.Top:Simultaneousintegrator.Bottom:Sequentialint
egrator.Source:FromOviatt,Lunsford,andCoulston(2005);courtesyACM.
12.4HumanFactorsDesignoftheInterface417

todeliverinformationtoaninterfaceinaconciseway.Whenvisual-spatialcon-tentisinvol
ved,forexample,userstakeadvantageofpeninputtoindicateloca-tionwhileusingthestron
gdescriptivecapabilitiesofspeechtospecifytemporalandothernonspatialinformation.T
hisfindingagreeswiththebroaderobserva-tionbylinguiststhatduringinterpersonalcomm
unicationspontaneousspeechandmanualgesturinginvolvecomplementaryratherthanduplic
ateinformationbetweenmodes(McNeill,1992).Speechandpeninputexpressesredundantinfo
rmationlessthan1percentofthetimeevenwhenusersareengagedinerrorcorrection,asituat
ioninwhichtheyarehighlymotivatedtoclarifyandreinforceinformation.Insteadofrelyin
gonredundancy,usersemployacontrastiveapproach,switchingawayfromthemodalityinwhic
htheerrorwasencountered,suchascorrectingspokeninputusingthepenandviceversa(Oviat
t&Olsen,1994;Oviatt&VanGent,1996).Preliminaryresultsindicatefurthermorethatthede
greeofredundancyisaffectedbythelevelofcognitiveloadusersencounterwhileperforming
atask(Ruiz,Taib,&Chen,2006),withasignificantreductioninredundancyastasksbecomech
allenging.Therelationshipofmultimodallanguageandcognitiveloadisfurtherdiscussedi
nfollowingsections.Whilecomplementarityisamajorintegrationthemeinhumancomputerint
eraction,thereisagrowingbodyofevidencepointingtotheimportanceofredundancyinmulti
partysettings,suchaslectures(Anderson,Andersonetal.,2004;Anderson,Hoyeretal.,200
4)orotherpublicpresentationsduringwhichoneormoreparticipantswriteonasharedspacew
hilespeaking(Kaiseretal.,2007).Inthesecases,ahighdegreeofredundancybetweenhandwr
ittenandspokenwordshasbeenfound.Redundancyappearstoplayaroleoffocusingtheattenti
onofagroupandhelpingthepresenterhighlightpointsofimportanceinhermessage.Techniqu
esthatleveragethisredundancyareabletorobustlyrecoverthesetermsbyexploitingmutual
disambiguation(Kaiser,2006).LinguisticStructureofMultimodalLanguageThestructureo
fthelanguageusedduringmultimodalinteractionwithacomputerhasalsobeenfoundtopresen
tpeculiarities.Userstendtoshifttoalocative-subject-verb-objectwordorderstrikingl
ydifferentfromthecanonicalEnglishsubject-verb-objective-locativewordorderobserve
dinspokenandformaltextuallanguage.Infact,thesameusersperformingthesametaskshaveb
eenobservedtoplace95percentofthelocativesinsentence-initialpositionsduringmultim
odalinter-actionandinsentence-finalpositionswhenusingspeechonly(Oviattetal.,1997
).Thepropositionalcontentthatistransmittedisalsoadaptedaccordingtomodality.Speec
handpeninputconsistentlycontributedifferentandcomplemen-tarysemanticinformationwi
ththesubject,verb,andobjectofasentencetypi-callyspokenandlocativeinformationwrit
ten(Oviattetal.,1997).Providedthatarichsetofcomplementarymodalitiesisavailableto
theusers,multimodallanguagealsotendstobesimplifiedlinguistically,briefer,syntact
icallysimpler,andlessdisfluent(Oviatt,1997),containinglesslinguisticindirection
12MultimodalInterfaces:CombiningInterfaces418

andfewerco-referringexpressions(Oviatt&Kuhn,1998).Thesefactorscontrib-utetoenhan
cedlevelsofrecognitioninmultimodalsystemswhencomparedtounimodal(e.g.,speech-only
)interfaces.Peninputwasalsofoundtocomebeforespeechmostofthetime(Oviattetal.,1997
),whichagreeswiththefindingthatinspontaneousgesturingandsignedlanguages,gestures
precedespokenlexicalanalogs(Kendon,1981;Naughton,1996).Duringspeechandthree-dime
nsionalgestureinteractions,pointinghasbeenshowntobesynchronizedwitheitherthenomin
alordeicticspokenexpres-sions(e.g.,this,that,here).Thetimingofthesegesturescanbefurth
rmorepredictedinatimewindowof200to400millisecondsaroundthebeginningofthenominalor
deicticexpression(Bourguet,2006;Bourguet&Ando,1998).ChoiceofMultimodalandUnimoda
lInputOnlyafractionofusercommandsaredeliveredmultimodally,withtherestmakinguseof
asinglemodality(Oviatt,1999b).Thenumberofmultimodalcom-mandsdependshighlyontheta
skathand,andonthedomain,varyingfrom20to86percent(Eppsetal.,2004;Oviatt,1997;Ovia
ttetal.,1997;Oviatt,Lunsford,&Coulston,2005),withhigherratesinspatialdomains.Fig
ure12.16presentsagraphshowingthepercentageoccurrenceofmultimodalcommandsacrossva
rioustasksanddomains.

SpecifyconstraintOver-laysLocatePrintScrollControltaskZoomLabelDeleteQueryCalcul
atedistanceModifyMoveAdd80706050403020100General action commandsSelection comman
dsSpatial location commandsPercentage of constructions
FIGURE12.16Percentageofcommandsthatusersexpressedmultimodallyasafunctionoftypeof
taskcommand.Negligiblelevelscanbeobservedduringgeneralcommands,withincreasinglev
elsforselectioncommandsandthehighestlevelforlocationcommands(Oviatt,1999b);court
esyACM.
12.4HumanFactorsDesignoftheInterface419

InfluenceofCognitiveLoadAnaspectofgrowinginterestistherelationshipbetweenmultimo
dalityandcognitiveload.Onegoalofawell-designedsystem,multimodalorotherwise,shoul
dbetoprovideuserswithmeanstoperformtheirtasksinawaythatdoesnotintroduceextraneou
scomplexitythatmightinterferewithperformance(Oviatt,2006a).Wickensandcolleaguesco
gnitiveresourcetheory(1983)andBad-deleystheoryofworkingmemory(1992)provideinteres
tingtheoreticalframe-worksforexaminingthisquestionanditsrelationstomultimodalsys
tems.Baddeley(1992)maintainsthatshort-termorworkingmemoryconsistsofmulti-pleinde
pendentprocessorsassociatedwithdifferentmodes.Thisincludesavisual-spatialsketchpa
dthatmaintainsvisualmaterialssuchaspicturesanddia-gramsinoneareaofworkingmemory,a
ndaseparatephonologicalloopthatstoresauditory-verbalinformation.Althoughthesetwo
processorsarebelievedtobecoordinatedbyacentralexecutive,intermsoflower-levelmoda
lityproces-singtheyareviewedasfunctioninglargelyindependently,whichiswhatenables
theeffectivesizeofworkingmemorytoexpandwhenpeopleusemultiplemodal-itiesduringtas
ks(Baddeley,2003).Empiricalevidenceshowsthatusersself-manageloadbydistributingin
forma-tionacrossmultiplemodalities(Calvert,Spence,&Stein,2004;Mousavi,Low,&Swell
er,1995;Oviatt,Coulston,&Lunsford,2004;Tangetal.,2005).Astaskcom-plexityincrease
s,sodoestherateatwhichuserschoosetoemploymultimodalratherthanunimodalcommands(Ov
iattetal.,2004;Ruizetal.,2006).Inanexperi-mentwithacrisismanagementdomaininvolvi
ngtasksoffourdistinctdifficultylevels,theratioofusersmultimodalinteractionincreas
edfrom59.2percentduringlow-difficultytasksto65.5percentatmoderatedifficulty,68.2
percentathigh,and75.0percentatveryhighdifficultyanoverallrelativeincreaseof27perc
ent.Analysisofuserstask-criticalerrorsandresponselatenciesacrosstaskdifficultylev
elsincreasedsystematicallyandsignificantlyaswell,corroboratingthemani-pulationof
cognitive-processingload(Oviattetal.,2004).Intermsofdesign,thesefindingspointtot
headvantageofprovidingmultiplemodalities,particularlyininterfacessupportingtasks
withhighercognitivedemands.12.4.2AHuman-CenteredDesignProcessGiventhehighvariabi
lityandindividualdifferencesthatcharacterizemultimodaluserinteraction,successful
developmentrequiresauser-centeredapproach.Ratherthanrelyingoninstruction,trainin
g,andpracticetomakeusersadapttoasystem,auser-centeredapproachadvocatesdesignings
ystemsbasedonadeeperunder-standingofuserbehaviorinpractice.Empiricalandethnograp
hicworkshouldpro-videinformationonwhichmodelsofuserinteractioncanbebased.Onceuse
rsnaturalbehaviorisbetterunderstood,includingtheirabilitytoattend,learn,andperfor
m,interfacescanbedesignedthatwillbeeasiertolearn,moreintuitive,andfreerofperform
anceerrors(Oviatt,1996b,2006).
12MultimodalInterfaces:CombiningInterfaces420

Whiletheempiricalworkdescribedintheprevioussectionhaslaidthefoun-dationforthedev
elopmentofeffectivemultimodalinterfaces,thereisstillaneedforcarefulinvestigation
oftheconditionssurroundingnewapplications.Inthefollowingsections,stepsaredescrib
edthatcanbefollowedwheninvestigatingissuesrelatedtoanewinterfacedesign.Theproces
sdescribedhereisfurtherillu-stratedbytheonlinecasestudy(seeSection12.7).Addition
alrelatedconsiderationsarealsopresentedasdesignguidelines(Section12.6).Understan

dingUserPerformanceinPracticeAfirststepwhendevelopinganewmultimodalinterfaceisto
understandthedomainunderinvestigation.Multimodalinterfacesdependveryheavilyonlan
-guageasthemeansthroughwhichtheinterfaceisoperated.Itisimportantthere-foretoanal
yzethedomaintoidentifytheobjectsofdiscourseandpotentialstandardizedlanguageeleme
ntsthatcanbeleveragedbyaninterface.Whenanalyzingabattlemanagementtask,forexample
,McGee(2003)usedethnographictechniquestoidentifythelanguageusedbyofficerswhilewo
rkingwithtangiblematerials(stickynotesovermaps).Whenperformedinalaboratory,thisi
nitialanalysismaytaketheformofsemistructuredpilots,duringwhichsub-jectsperformat
askwithsimulatedsystemsupport.Thatis,tofacilitateevolution,asimulated(WizardofOz)s
ystemmaybeused(seeSection12.5.2foradditionaldetails).IdentificationofFeaturesand
SourcesofVariabilityOnceadomainisbetterunderstood,thenextstepconsistsofidentifyi
ngthespecificsofthelanguageusedinthedomain.Inspecializedfields,itisnotun-commonf
orstandardizedlanguagetobeemployed.Themilitaryemployswell-understoodproceduresan
dstandardizedlanguagetoperformbattleplanning(McGee&Cohen,2001;McGeeetal.,2001).S
imilarly,healthprofessionals(Cohen&McGee,2004)andengineershavedevelopedmethodsan
dlanguagespecifictotheirfields.Evenwhendealingwithdomainsforwhichthereisnostanda
rdizedlan-guage,suchasphototagging,theanalysisofinteractionscanrevealusefulrobus
tfeaturesthatcanbeleveragedbyamultimodalinterface.Inacollaborativephotoannotatio
ntaskinwhichusersdiscussedtravelandfamilypictures,analysisrevealedthatthetermsth
atarehandwrittenarealsoredundantlyspoken.Asauserpointstoapersoninaphotoandhandwr
iteshisname,shewillalsoinmostcasesspeakthepersonsnamerepeatedlywhileexplainingwho
thatis.Furtheranalysisrevealedthatredundantlydeliveredtermswerefrequentduringthe
dis-cussionandthattheyweregoodretrievalterms(Barthelmess,Kaiseretal.,2006;Kaiser
etal.,2007).Manytimeslanguagesandfeaturesidentifiedduringthisphaseprovetoocomple
xtobeprocessableviacurrenttechnology.Animportanttaskisthereforetorecognizeandmod
elthesourcesofvariabilityanderror(Oviatt,1995).
12.4HumanFactorsDesignoftheInterface421

InSection12.7,thisisexaminedinthecontextofidentifyingcausesofspeechdis-fluencies
,whichintroducehard-to-processfeaturesintothespokenlanguage.TransparentGuidanceo
fUserInputAstrategyfordealingwithhard-to-processfeatures(e.g.,disfluencies)prese
ntinusersinputistodesigninterfacesthattransparentlyguideusersinputtoreduceerrors(O
viatt,1995).Theessenceofthisapproachistogetuserstosay[anddo]whatcomputerscanunder
stand(Zoltan-Ford,1991).Themaingoalofthedesignatthisstageistoidentifymeanstoguide
usersinputtowardsimplerandmorepredictablelanguage.Howthisisachievedmaydependonthe
specificsofthedomain.Determiningdomain-specificfactorsrequiresdetailedanalysisof
naturallinguisticfeaturesmanifestedbyusers.Basedonthisanalysis,experimentationwi
thalternativeinterfacedesignsrevealsmeth-odstoreducecomplexityinwaysthatarenotob
jectionabletousers.Thisisillu-stratedintheonlinecasestudy(seeSection12.7).Techni
questhathaveledtotransparentguidanceofuserinputincludechoos-inginterfacemodaliti
esthatmatchtherepresentationalsystemsusersrequirewhileworkingonatask(Oviatt,Arth
ur,&Cohen,2006;Oviatt&Kuhn,1998),structuringtheinterfacetoreduceplanningload(Ovi
att,1995,1996b,1997),andexploitinguserslinguisticconvergencewithasystemsoutput(Lar
son,2003;Oviattetal.,2005).Intermsofdesign,thelanguageusedinasystemspre-sentation
shouldbecompatiblewiththelanguageitcanprocess.TheseaspectsarefurtherdiscussedinS
ection12.6.DevelopmentofFormalModelsLinguisticandbehavioralregularitiesdetectedd
uringtheanalysisofadomaincanberepresentedviaavarietyofformalmodelsthatdirectlyin
fluenceanddrivetheinterpretationprocessofasystem.Giventhesemodels,asystemisablew
ithhigherlikelihoodtodistinguishtheactualusersintentionsfromavarietyofalternative
interpretations.Formalmodelsincludeprimarilymultimodalgrammarsusedtodriveinterpr
e-tation.Thesegrammarsproviderulesthatguidetheinputinterpretation,whichinturndri
vesinterfaceresponsesoractivationofotherapplications.Morerecently,therehasbeenag
rowinginterestintheapplicationofmachinelearningtechniquestomodelaspectsofmultimo
dalinteractions.Oneexampleistheuser-adaptedmodeldevelopedbyHuangandOviatt(2006)t
opre-dictthepatternsofusersnextmultimodalinteraction.Othermachinelearningmodelsha
vebeenusedtointerpretuseractionsduringmultimodalmultipartyinteractions(e.g.,McCo

wanetal.,2005;Pianesietal.,2006;Zancanaroetal.,2006).Theadvantageofthesesystemsi
sthattheyareabletoadapttoindividualuserscharacteristicsandarethereforemorecapable
ofprocessinginputwithfewererrors.
12MultimodalInterfaces:CombiningInterfaces422

12.5TECHNIQUESFORTESTINGTHEINTERFACEIterativetestingthroughoutthedevelopmentcycl
eisoffundamentalimportancewhendesigningamultimodalinterface.Empiricalevaluationa
ndusermodelingaretheproactivedrivingforcesduringsystemdevelopment.Inthissection,
thegenericfunctionalityrequiredforsuccessfullytestingamultimodalinterfaceisdescr
ibed.Aninitialstepintestinganinterfaceisthecol-lectionofuserdataforanalysis.Give
ntheneedtoexamineavarietyofmodalitieswhendesigningamultimodalinterface,anappropr
iatedatacollectioninfrastruc-tureisrequired(Section12.5.1).Astrategythathasprove
nveryfruitfulforproto-typingnewmultimodalinterfacesistoexploithigh-fidelitysimul
ationtechniques,whichpermitcomparingtrade-offsassociatedwithalternativedesigns(S
ection12.5.2).Analysisofmultimodaldataalsorequiressynchronizingmultipledatastrea
msanddevelopmentofannotationtools(asdescribedinSection12.5.3).12.5.1DataCollecti
onInfrastructureThefocusofdatacollectionisonthemultimodallanguageandinteractionp
at-ternsemployedbyusersofasystem.Adatacollectionfacilitymustthereforebecapableof
collectinghigh-qualityrecordingsofmultiplestreamsofsynchronizeddata,suchasspeech
,peninput,andvideoinformationconveyingbodymotions,gestures,andfacialexpressions.
Besidesproducingrecordingsforfurtheranalysis,thecollectioninfrastruc-turealsohas
toprovidefacilitiesforobservationduringthecollectioninordertosupportsimulationst
udies.Sinceviewsofeachmodalityarenecessaryduringsomesimulations,capabilitiesforr
eal-timedatastreaming,integration,anddis-playarerequired.Buildingacapableinfrast
ructuretoprototypenewmultimodalsystemsrequiresconsiderableeffort(Arthuretal.,200
6).Ofprimaryimportanceisthatdatabenaturalisticandthereforerepresentativeoftaskpe
rformancethatisexpectedonceasystemisdeployed.Thus,anydatacol-lectiondevicesorins
trumentationneedtobeunobtrusive.Thisischallenging,giventherequirementforrichcoll
ectionofavarietyofdatastreamsthateachcanrequireacollectiondevice(e.g.,cameras,mi
crophones,pens,physiologicalsensors).Thiscanbeparticularlyproblematicwhentheaimi
stocollectrealisticmobileusageinformation(Figure12.17).12.5.2High-FidelitySimula
tionsAtechniquethathasbeenveryvaluableindesigningmultimodalinterfacesistheconstr
uctionofhigh-fidelitysimulations,orWizard-of-Ozexperiments(Oviattetal.,1992;Salb
er&Coutaz,1993a,1993b).Theseexperimentsconsistofhavingsubjects
12.5TechniquesforTestingtheInterface423

interactwithasystemforwhichfunctionalityisatleastpartiallyprovidedbyahumanoperat
or(i.e.,wizard)whocontrolstheinterfacefromaremotelocation.Themainadvantageofprotot
ypingviasimulationsisthattheyarefasterandcheaperthandevelopingafullyfunctionalsy
stemandtheyprovideadvancedinformationaboutinterfacedesigntrade-offs.Hard-to-impl
ementfunctionalitycanbedelegatedtothewizard,whichmayshortenthedevelopmentcycleco
nsid-erablyviarapidprototypingoffeatures.Simulationsmakeitpossibleforinterfaceop
tionstobetestedbeforecommittingtoactualsystemconstruction,andmayevensupportexper
imentsthatwouldnototherwisebepossiblegiventhestateoftheart(onecan,forinstance,ex
aminetherepercussionsofmuchenhancedrecognitionlevelsonaninterface).Onechallengew
hensettingupaWizard-of-Ozexperimentismakingitbeliev-able(Oviatt,1992;Arthuretal.
,2006).Inordertobeeffective,workingprototypesthatusersinteractwithmustbecredible
;forexample,makingsomeerrorsaddstocredibility.Thewizardneedstobeabletoreactquick
lyandaccuratelytouseractions,whichrequirestrainingandpractice.Techniquesforfurth
erfacilitatingthistypeofexperimentincludesimplifyingthewizardsinterfacethroughsem
i-automation(e.g.,byautomaticallyinitiatingdisplayactions)(Oviattetal.,1992).Ana
utomaticrandomerrormodulescanbeusedtointroducesystemmisrecogni-tions,whichcanbes

etatanylevelandcontributetothesimulationscredibility.
FIGURE12.17Mobiledatacollection.Thesubject(ontheleft)carriesavarietyofdevices,in
cludingaprocessingunitinabackpack.Source:FromOulasvirtaetal.(2005);courtesyACM.
12MultimodalInterfaces:CombiningInterfaces424

Figure12.18showsaninterfaceviewedbygeometrystudentsandthecorrespond-ingwizardint
erface.Toaddressthecomplexityofmultimodalinteractions,multiplewizardshavebeenuse
dinthepastinsomestudies,especiallyincasesinvolvingcollaboration(Arthuretal.,2006
;Oviattetal.,2003;Salber&Coutaz,1993a).Eachwizardthenconcentratesonprovidingsimu
lationfeedbackforparticularaspectsoftheinterac-tion,whichassistsinmakingtheovera
lltaskofdrivinganinterfacemanageable.12.5.3SupportforDataAnalysisTheassessmentof
theeffectivenessofaninterfacedesignorthecomparisonofalternativedesignsrequiresde
tailedexaminationofthedatathathavebeencol-lected.Analysistoolsthereforeneedtopro
videmeansforplaybackandnavigation
(a)
(b)
FIGURE12.18Simulatedmultimodalinterface.Theinterfaceas(a)seenbyusersand(b)usedby
thewizardtocontroltheflowofinteractioninabelievableway.Thisinterfacewasuseddurin
gacollaborativegeometryproblemsolvinginteraction.Source:FromArthuretal.(2006);cou
rtesyACM.
12.5TechniquesforTestingtheInterface425

ofmultipledatastreams.High-fidelitysynchronizationisalsorequired.Audio,video,and
otherinput(e.g.,pen)shouldbealignedwellenoughthatdifferencesarenotbeperceptiblet
oahumananalyst.Theanalysisprocessusuallyincorporatesmark-ups(orannotations)ofsel
ectedpartsofaninteraction.Theseannotationsmightincludespeechtranscriptsandsemant
icannotationofgestures,gaze,orprosodiccharacteristicsofthespeech.Thespecificsofw
hatisannotateddependonthepurposeoftheresearchandinter-facebeingdesigned.Annotate
ddatacanbeexaminedintermsofcharacteristicsoftheirlanguageproduction,performance,
anderrorcharacteristicsundervaryingcircumstances.Avarietyofdifferentplaybackanda
nnotationtoolsareavailable,suchasNomos(Gruenstein,Niekrasz,&Purver,2005),Anvil(M
artin&Kipp,2002),andAmiGram(Laueretal.,2005).Arthuretal.(2006)describeatoolforan
nota-tionandplaybackofmultiplehigh-definitionvideoandaudiostreamsappropriatefort
heanalysisofmultipartyinteractions.12.6DESIGNGUIDELINESTheguidelinespresentedint
hissectionaregroupedintwoprimaryclasses:thosethathavetodowithissuesrelatedtotheu
ncertaintyofrecognition(Section12.6.1),andthoseconcernedwiththecircumstancesguid
ingtheselectionofmodalities(Section12.6.2).12.6.1DealingwithUncertaintyOneessent
ialaspectthathastobeconsideredwhenbuildingsystemsthatrelyonambiguousinterpretati
onishowuncertaininterpretationsaredealtwith(Bourguet,2006;Mankoff,Hudson,&Abowd,
2000).Shieldingusersfromerrorsandprovidinggracefulwaysforhandlingunavoidablemisi
nterpretationsareessentialusabilityconcernsinthisclassofinterfaces.Asdiscussed(S
ection12.4.1),humanlanguageproductioninvolvesahighlyautomatizedsetofskillsnotund
erusersfullconsciouscontrol(Oviatt,1996b).Themosteffectivestrategyforerroravoidan
ceistodesigninterfacesthatlever-ageusersengrainedcognitiveandlinguisticbehaviorin
ordertotransparentlyguideinputthatavoidserrors.Infact,trainingandpracticeinaneff
orttochangeengrainedbehaviorpatternsoftenproveuseless(Oviattetal.,2003,2005).Whi
lethisstrategycanbethemosteffective,itisalsothemostdemandingintermsofusermodelin
gandimplementationeffort.Inordertodeterminetherootcauseoferrorsanddesignaneffect

ivestrategyforavoidingandresolvingthem,acycleofexperi-mentsisrequired,asillustra
tedbytheonlinecasestudy(seeSection12.7).Inthissection,theeffectiveprinciplesofmu
ltimodalinteractionaredistilledasasetofguidelines.
12MultimodalInterfaces:CombiningInterfaces426

SupporttheRangeofRepresentationalSystemsRequiredbytheTaskThestructuralcomplexity
andlinguisticvariabilityofinputgeneratedbyusersareimportantsourcesofprocessingdi
fficulties.Aprimarytechniquetoelicitsimpler,easier-to-processlanguageisrelatedto
thechoiceofmodalitiesthataninterfacesupports.Userswillnaturallychoosethemodaliti
esthataremostappro-priateforconveyingcontent.Forexample,userstypicallyselectpeni
nputtopro-videlocationandspatiallyorientedinformation,aswellasdigits,symbols,and
graphiccontent(Oviatt,1997;Oviatt&Olsen,1994;Suhm,1998).Incontrast,theywillusesp
eechfordescribingobjectsandeventsandforissuingcommandsforactions(Cohen&Oviatt,19
95;Oviatt&Cohen,1991).Aprimaryguidelineisthereforetosupportmodalitiessothatthere
presenta-tionalsystemsrequiredbyusersareavailable.Thelanguagethatresultswhenadeq
uatecomplementarymodalitiesareavailabletendstobesimplifiedlinguisti-cally,briefe
r,syntacticallysimpler,andlessdisfluent(Oviatt,1997),anditcon-tainslesslinguisti
cindirectionandfewerco-referringexpressions(Oviatt&Kuhn,1998).Oneimplicationofth
isisthatthefundamentallanguagemodelsneededtodesignamultimodalsystemarenotthesame
asthoseusedinthepastforprocessingtextuallanguage.StructuretheInterfacetoElicitSi
mplerLanguageAkeyinsightindesigningmultimodalinterfacesthatleadtosimpler,morepro
cess-ablelanguageisthatthelanguageemployedbyuserscanbeshapedverystronglybysystem
presentationfeatures.Addingstructure,asopposedtohavinganuncon-strainedinterface,
hasbeendemonstratedtobehighlyeffectiveinsimplifyingthelanguageproducedbyusers,re
sultinginmoreprocessablelanguageandfewererrors.Aforms-basedinterfacethatguidesus
ersthroughthestepsrequiredtocompleteataskcanreducethelengthofspokenutterancesand
eliminateupto80percentofhard-to-processspeechdisfluencies(Oviatt,1995).Similarbe
nefitshavebeenidentifiedinmap-baseddomains.Amapwithmoredetailedinformationdispla
yingthefullnetworkofroads,buildings,andlabelscanreducedisfluenciescomparedtoamin
imalistmapcontainingone-thirdoftheroads(Oviatt,1997).Othertechniquesthatmayleadu
serstowardexpectedlanguageareguideddialogsandcontext-sensitivecues(Bourguet,2006
).Theseprovideadditionalinformationthathelpsusersdeterminewhattheirinputoptionsa
reateachpointofaninteraction,leadingtomoretargetedproductionoftermsthatareexpect
edbytheinterfaceatagivenstate.Thisisusuallyimplementedbyhavingapromptthatexplici
tlyliststheoptionstheusercanchoosefrom.ExploitNaturalAdaptationApowerfulmechanis
mfortransparentlyshapinguserinputreliesontheten-dencythatusershaveofadaptingtoth
elinguisticstyleoftheirconversational
12.6DesignGuidelines427

partners(Oviatt,Darvesetal.,2005).Astudyinwhich24childrenconversedwithacomputergeneratedanimatedcharacterconfirmedthatchildrensspeechsignalfeatures,amplitude,du
rationalfeatures,anddialogresponselatenciessponta-neouslyadapttothebasicacoustic
-prosodicfeaturesofasystemstext-to-speechoutput,withthelargestadaptationsinvolvin
gutterancepausestructureandampli-tude.Adaptationsoccurredrapidly,bidirectionally
,andconsistently,with70to95percentofchildrensspeechconvergingwiththatoftheircompu
terpartners.Similarconvergencehasbeenobservedinusersresponsestodiscretesystemprom
pts.Peoplewillrespondtosystempromptsusingthesamewordingandsyn-tacticstyle(Larson
,2003).Thissuggeststhatsystempromptsshouldbematchedtothelanguagethatthesystemide
allywouldreceivefromusersusuallypresent-ingasimplestructure,restrictedvocabulary,
andrecognizablesignalfeatures.OfferAlternativeModalitiesUsersCanSwitchtoWhenCorr
ectingErrorsThereisevidencethatusersswitchmodalitiestheyareusingtocorrectmisreco
g-nitionsafterrepeatedfailures(Oviatt&VanGent,1996).Userscorrectingaspo-kenmisre

cognitionwillattempttorepeatthemisrecognizedwordviaspeechafewtimes,butwillthensw
itchtoanothermodalitysuchashandwritingwhentheyrealizethatthesystemisunabletoacce
ptthecorrection.Thisbehaviorappearstobemorepronouncedforexperienceduserscompared
tonovices.Thelattertendtocontinuetousethesamemodalitydespitethefailures(Halverso
netal.,1999).Therefore,awell-designedsystemshouldofferalternativemodalitiesforco
rrectingmisrecognitions.Theabsenceofsuchafeaturemayleadtoerrorspirals(Halversoneta
l.,1999;Oviatt&VanGent,1996)situationsinwhichtheuserrepeatedlyattemptstocorrectan
error,butduetoincreasedhyperarticulation,thelikelihoodofcorrectsystemrecognition
actuallydegrades.MakeInterpretationsTransparentButNotDisruptiveOnedisconcertinge
ffectofsystemsthatrelyoninterpretationofusersambig-uousinputiswhenuserscannotclea
rlyconnecttheactionsperformedbythesystemwiththeinputtheyjustprovided.Thatisparti
cularlydisconcertingwhenthesystemdisplaydisagreeswiththeexpectationsthatusershad
whenprovidingtheinput,aswouldhappenwhenausercommandsasystemtopaintatablegreenand
seesthefloorturningblueasaresult(Kaiser&Barthelmess,2006).Onetech-niquetomakesys
temoperationmoretransparentistogiveuserstheopportunitytoexaminetheinterpretation
andpotentiallycorrectit,suchasviaagraphicaldisplayoftheinterpretation.Onepopular
wayofmakingusersawareofasystemsinterpretationistomakeavailablealistofalternaterec
ognitions.Theselists,sometimescalledn-bestlists,representalimitednumberofthemostli
kely(best)interpretationsidentifiedbya
12MultimodalInterfaces:CombiningInterfaces428

system.Thisstrategyneedstoconsiderverycarefullyhoweasyitistoaccessanddismissthel
ist,andalsohowaccuratethelistis.Incasesinwhichtheaccuracyissuchthatmostofthetime
thecorrectinterpretationisnotpresent(forveryambiguousinput),thisstrategycanbecom
ecounterproductive(Suhm,Myers,&Waibel,1999).Whenwelldesigned,makingalternativere
cognitionsavailablecanbeeffective,tothepointthatusersmaytrythelistfirstandjustat
tempttorepeatmisrecognizedtermsasasecondaryoption(Larson&Mowatt,2003).Oneexample
ofasimpleinterfacethatmakesanalternativeinterpretationavailableisGoogle.Ahyperli
nklistsapotentialalternativespellingofquerytermsandmaybeactivatedbyasingleclick.
Displayingthestateofrecognitioncansometimesprovedisruptive,suchasduringmultipart
yinteractions.Displayingmisrecognitionsorrequiringtheuserstochooseamongalternati
verecognitionsinthecourseofameetinghasbeenshowntodisrupttheinteraction,asuserstu
rntheirattentiontocorrectingthesystem.Afruitfulapproach(explored,forexample,inth
eDistributedCharterSystem)istopresentthestateofrecognitioninalessdistractingorfo
rcefulway,suchasviasubtlecolorcodingofdisplayelements.Usersarethenfreetochooseth
emomentthatismostappropriatetodealwithpotentialissues(Kaiser&Barthelmess,2006).1
2.6.2ChoosingModalitiestoSupportThechoiceofwhichmodalitiestosupportisnaturallyan
importantonewhenamultimodalinterfaceisbeingdesigned.Theappropriatemodalitiesandc
haracter-isticsofthelanguagesupportwithineachmodalityareinfluencedbythetasksthat
usersaretoface,conditionsofuse,andusercharacteristics.ContextofUseConditionsofus
edetermine,forinstance,whetherasystemisrequiredtobemobileorwhetheritistobeusedwi
thinanofficeenvironmentorameetingroom.Thatinturndeterminesthenatureandcapability
ofthedevicesthatcanbeemployed.Mobileuserswillcertainlynotacceptcarryingaroundhea
vyloadsandwillthereforenotbeabletotakeadvantageofmodalitiesthatrequireproces-sin
gpowerorsensorsthatcannotbefitintoacellphoneorPDA(portabledigitalassistant),asis
thecaseformostvision-basedtechniques.Meetingrooms,ontheotherhand,maymakeuseofamu
chlargersetofmodalities,evenincasesinwhichseveralcomputersarerequiredtorunasyste
m.Mostcurrentmobilesystemsprovidesupportforpeninputandarepowerfulenoughtoexecute
atleastacertainlevelofspeechrecognition.Speechinterfacesareidealformanymobilecon
ditionsbecauseofthehands-and-eyes-freeusethatspeechaffords.Peninputisusedinmanyd
evicesasanalternativeforkeyboardinputinsmalldevices.
12.6DesignGuidelines429

Otherconsiderationsassociatedwithusagecontextthataffectchoiceofmod-alitiesarepri
vacyandnoise.Speechislessappropriatewhenprivacyisaconcern,suchaswhentheinterface
istobeusedinapublicsetting.Speechisalsonotindi-catedwhennoisyconditionsaretobeex
pected(e.g.,ininterfacesforconstructionsitesornoisyfactories).Awell-designedinte
rfacetakesadvantageofmultiplemodalitiestoprovidesupportforavarietyofusagecontext
s,allowinguserstochoosethemodalitythatismostappropriategivenaparticularsituation
.UserCharacteristicsAwell-designedinterfaceleveragestheavailabilityofmultiplemod
alitiestoaccommodateusersindividualdifferencesbysupportinginputoveralternativemod
alities.Motorandcognitiveimpairments,age,nativelanguage(e.g.,accent),andotherind
ividualcharacteristicsinfluenceindividualchoiceofinputmodalities.Forexample,peni
nputandgesturingarehardtousewhenthereisdiminishedmotoracuity.Conversely,spokenin
putmayproveproblematictouserswhohavespeechorhearingimpairmentsorwhospeakwithanac
cent.Aparticularkindoftemporaryimpairmentoccurswhenusersaremobileorarerequiredto
keepahighlevelofsituationalawareness,suchasonabattlefieldorinanemergencyresponse
situation,orevenwhileoperatingavehicle(Oviatt,2007).Supportingspokeninteractiont
henbecomesanattractiveoption.12.7CASESTUDIESCasestudiesforthesekindsofmultimodal
interfacescanbefoundatwww.beyondthegui.com.12.8FUTURETRENDSMultimodalinterfacesd
ependultimatelyonthequalityoftheunderlyingrecogni-zersformultiplemodalitiesthata
reused.Thus,theseinterfacesbenefitfromtheadvancesofnaturallanguageprocessingtech
niquesandadvancesinhardwarecapabilitiesthatmakeitpossibleformorechallengingrecog
nitionstobesuccessfullyachieved.12.8.1MobileComputingMobilecomputingisoneofthear
easinwhichmultimodalinterfacesareexpectedtoplayanimportantroleinthefuture.Multim
odalinterfacessuitmobility
12MultimodalInterfaces:CombiningInterfaces430

particularlywellbecauseofthesmallfactorrequirementsthatareusuallyimposedbyon-the
-moveoperation.Multimodalinterfacesprovideexpressivemeansofinteractiontouserswit
houtrequiringbulky,hard-to-carryequipmentsuchaskeyboardsormice.Theflexibilitypro
videdbyinterfacesthatsupportmultiplemodesalsofitsthedemandsintroducedbymobilityv
iatheadaptationtoshiftingcontextsofuse.Amultimodalinterfacecan,forinstance,provi
despokenoperationforuserswhosehandsandeyesarebusy,suchaswhiledriving,ormaintaini
ngsituationalawarenessindangerousenvironments,suchasdisasterareas.Theseinterface
scanalsoadapttonoisyenvironmentseitherbyleveragingmutualdisambiguationtoenhancer
ecognitionorbyprovidingmeansforcommandstobegivenvianonspokenmeanssuchasapen.Spee
chandpeninputarealreadysupportedbyavarietyofdevices(smartphones,PDAs),andsuccess
fulcommercialimplementationsareavailable.Furthercomputationalpowershouldleadtoan
erainwhichamajorityofthemobiledevicesmightoffermultimodalcapabilities,includingi
ncreasinglevelsofvideo-basedmodalities.12.8.2CollaborationSupportCollaborativehu
maninteractionisintrinsicallymultimodal(Tang,1991).Groupsofpeoplecollaboratingma
keampleuseofeachothersspeech,gaze,pointing,andbodymotiontofocusattentionandestabl
ishrequiredsharedcontexts.Collabora-tivemultimodalinterfacesarethereforeanatural
fitforgroupware.Interestingchallengesareintroducedbytheshiftfromsingle-usertomul
ti-userinterfaces.Thelanguageusedbyhumanstocommunicateamongthem-selvescanbeconsi
derablymorecomplexthanthatemployedwhenaddressingcomputers(Oviatt,1995).Systemsth
atarebasedonobservationofhumanhumaninteractionwillneedtoemploynoveltechniquesfore
xtractingusefulinformation.Theintroductionoftechnologyintocollaborativesettingsh
asnotbeenwith-outproblems.Theseinteractionsarealsoknowntobebrittleinthefaceoftec
h-nology(Grudin,1988)andthepotentialdisruptionsofsubtlesocialprocessesthattechno
logymayintroduce.Considerablecareisthereforerequiredtoexam-inehowsystemsthatoper
atevianaturallanguagecanbestbeintegrated.Pioneeringsystemshavemostlyemployedapas
siveapproach,inwhichobser-vationsperformedbyasystemarecollectedwithminimaldirect
interactionbetweenthesystemandagroupofusers.Systemresultsaredeliveredaftertheint
eractionshaveconcluded,intheformofbrowsableinformation(Ehlen,Purver,&Niekrasz,20
07)orMS-Projectcharts(Kaiseretal.,2004)orviaasemiautomatedcoachthatpresentsepisode
softhemeetingtoparticipantsduringwhichdysfunctionalbehaviorisdetected(Zancanaroe

tal.,2006).
12.8FutureTrends431

Workisongoingtoleveragepara-linguisticcuessuchasamplitudetofacilitatethedetectio
nofwhoisbeingaddressed(humansorcomputer)inamultipartysetting(Lunsfordetal.,2005,
2006)andtoexaminetheimpactoftheintroductionofcomputationalassistantsduringandaft
ermeetings(Falconetal.,2005;Pianesietal.,2006;Rienks,Nijholt,&Barthelmess,2006;Z
ancanaroetal.,2006).Giventhetimeandexpensethatareinvolvedincollaborativeinteract
ionsingeneral,advancesinsupportingtechnologyhaveapotentialforhighpayoffs.Itisexp
ectedthatthisareawillcontinuetoattractattentioninthefuture.12.8.3ConclusionTheco
ntinuousadvancesinhardwareandofthebasicrecognitiontechnologiesuponwhichmultimoda
lsystemsrelyshouldmakeitincreasinglypossiblefordesignerstotakeadvantageofmultimo
dalitytoprovideuserswithmoreintuitive,easier-to-useinterfacesthatwillimpactuserp
erformanceandsatisfactioninapositiveway.Systemsthattakeadvantageofabroaderrangeo
fnaturalhumancommunica-tioncapabilities,suchasspeech,handwriting,andgesturing,ar
eexpectedtobecomeincreasinglyimportantastheytransfertheburdenofcommunicationfrom
userstothesystemsthemselves.Thisshifttowardsystemsthatarebetterabletounderstandt
heirusers,ratherthanrequiringthemtooperateinwaysthatfitthesystemsnarrowlimitation
s,isexpectedtoeventuallyresultinunprecedentedgainsinproductivityasperformancebec
omeslesshinderedbytechnologyandusersbecomefreertoconcentrateontheirtasksratherth
anontheintricaciesofthetechnology.REFERENCESAdjoudani,A.,&Benoit,C.(1995).Audiovisualspeechrecognitioncomparedacrosstwoarchitectures.Paperpresentedatthe4thEuro
peanConferenceonSpeechCommunicationandTechnology.Allegayer,J.,Jansen-Winkeln,R.,
Reddig,C.,&Reithinger,N.(1989).Bidirectionaluseofknowledgeinthemulti-modalNLacce
sssystemXTRA.Paperpresentedatthe11thInter-nationalJointConferenceonArtificialInt
elligence.Anderson,R.J.,Anderson,R.,Hoyer,C.,&Wolfman,S.A.(2004).Astudyofdigital
inkinlecturepresentation.Paperpresentedatthe2004SIGCHIConferenceonHumanFactorsin
ComputingSystems.Availableathttp://doi.acm.org/10.1145/985692.985764.Anderson,R.
J.,Hoyer,C.,Prince,C.,Su,J.,Videon,F.,&Wolfman,S.A.(2004).Speech,inkandslides:Th
einteractionofcontentchannels.Paperpresentedatthe12thAnnualACMInternationalConfe
renceonMultimedia.Availableathttp://doi.acm.org/10.1145/1027527.1027713.Andre,E.
(2003).Naturallanguageinmultimedia/multimodalsystems.InMitkov,R.,ed.,TheOxfordHa
ndbookofComputationalLinguistics.NewYork:OxfordUniversityPress,65069.
12MultimodalInterfaces:CombiningInterfaces432

Andre,E.,Muller,J.,&Rist,T.(1996).ThePPPpersona:Amultipurposeanimatedpresen-tati
onagent.Paperpresentedatthe1996WorkshoponAdvancedVisualInterfaces.Avail-ableatht
tp://doi.acm.org/10.1145/948449.948486.Anoto.(2007).Anototechnologyhttp://www.ano
to.com/.Arthur,A.M.,Lunsford,R.,Wesson,R.M.,&Oviatt,S.L.(2006).Prototypingnovelc
ollabo-rativemultimodalsystems:Simulation,datacollection,andanalysistoolsforthen
extdecade.Paperpresentedatthe8thInternationalConferenceonMultimodalInterfaces,Or
alSession4:CollaborativeSystemsandEnvironments.Availableathttp://portal.acm.org/
citation.cfm?id=1181039&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.Baddele
y,A.D.(1992).Workingmemory.Science255(5044):55659.Baddeley,A.D.(2003).Workingmemo
ry:Lookingbackandlookingforward.NatureReviewsNeuroscience44(10):82939.Banerjee,S.
,&Rudnicky,A.I.(2004).Usingsimplespeech-basedfeaturestodetectthestateofameetinga
ndtherolesofthemeetingparticipants.Paperpresentedatthe8thInter-nationalConferenc
eonSpokenLanguageProcessing.Availableathttp://www.cs.cmu.edu/~banerjee/Publicati
ons/icslp2004.pdf.Bangham,J.A.,Cox,S.J.,Elliott,R.,Glauert,J.R.W.,Marshall,I.,Ra
nkov,S.,etal.(2000).Virtualsigning:Capture,animation,storageandtransmissionanover
viewoftheViSi-CASTproject.PaperpresentedattheIEESeminaronSpeechandLanguageProces
singforDisabledandElderlypeople,London.Barthelmess,P.,&Ellis,C.A.(2005).TheNeemP

latform:Anevolvableframeworkforperceptualcollaborativeapplications.JournalofInte
lligentInformationSystems25(2):20740.Barthelmess,P.,Kaiser,E.C.,Huang,X.,&Demirdj
ian,D.(2005).Distributedpointingformultimodalcollaborationoversketcheddiagrams.P
aperpresentedatthe7thInterna-tionalConferenceonMultimodalInterfaces,Recognitiona
ndMultimodalGesturePat-terns.Availableathttp://doi.acm.org/10.1145/1088463.10884
69.Barthelmess,P.,Kaiser,E.C.,Huang,X.,McGee,D.R.,&Cohen,P.R.(2006).Collaborativ
emultimodalphotoannotationoverdigitalpaper.Paperpresentedatthe8thInternationalCo
nferenceonMultimodalInterfaces,PosterSession1.Availableathttp://portal.acm.org/c
itation.cfm?id=1181000&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.Barthelm
ess,P.,McGee,D.R.,&Cohen,P.R.(2006).Theemergenceofrepresentationsincollaborative
spaceplanningoverdigitalpaper:Preliminaryobservations.Paperpresentedatthe2006ACM
ConferenceonComputer-SupportedCooperativeWork,WorkshoponCollaborationoverPaperan
dDigitalDocuments(CoPADD).Bellik,Y.(1997).Mediaintegrationinmultimodalinterfaces
.PaperpresentedattheIEEEFirstWorkshoponMultimediaSignalProcessing,Princeton,NJ.B
ellik,Y.,&Burger,D.(1994).Multimodalinterfaces:Newsolutionstotheproblemofcompute
raccessibiltyfortheblind.Paperpresentedatthe1994SIGCHIConferenceonHumanFactorsin
ComputingSystems,ConferenceCompanion.Availableathttp://portal.acm.org/citation.c
fm?id=260482&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.Bellik,Y.,&Burger,
D.(1995).Thepotentialofmultimodalinterfacesfortheblind:Anexploratorystudy.Paperp
resentedatthe18thAnnualRESNAConference.Availableathttp://www.limsi.fr/Individu/b
ellik/publications/resna95.rtf.
References433

Bellik,Y.,&Farcy,R.(2002).Comparisonofvariousinterfacemodalitiesforalocomotionas
sistancedevice.Paperpresentedatthe8thInternationalConferenceonComputersHelpingPe
oplewithSpecialNeeds,ICCHP2002,Linz,Austria.Bellik,Y.,&Teil,D.(1993).Amultimodal
dialoguecontrollerformultimodaluserinterfacemanagementsystemapplication:Amultimo
dalwindowmanager.Paperpresentedatthe1993SIGCHIConferenceonHumanFactorsinComputin
gSystems,Amsterdam.Benoit,C.,&LeGoff,B.(1998).Audio-visualspeechsynthesisfromFre
nchtext:Eightyearsofmodels,designsandevaluationattheICP.SpeechCommunication26(12)
:11729.Benoit,C.,Martin,J.-C.,Pelachaud,C.,Schomaker,L.,&Suhm,B.(2000).Audio-visu
alandmultimodalspeech-basedsystems.InGibbon,D.,Mertins,I.,&Moore,R.,eds.,Handboo
kofMultimodalandSpokenDialogueSystems:Resources,TerminologyandProductEvaluation.
Boston:KluwerAcademicPublishers,102203.Bernstein,L.E.,&Benoit,C.(1996).Forspeechp
erceptionbyhumansormachines,threesensesarebetterthanone.Paperpresentedatthe4thIn
ternationalConferenceonSpokenLanguageProcessing,Philadelphia.Bers,J.,Miller,S.,&
Makhoul,J.(1998).Designingconversationalinterfaceswithmulti-modalinteraction.Pro
ceedingsofDARPAWorkshoponBroadcastNewsUnderstandingSystems,Lansdowne,VA,31921.Bol
t,R.A.(1980).Put-that-there:Voiceandgestureatthegraphicsinterface.Paperpre-senteda
tthe7thAnnualConferenceonComputerGraphicsandInteractiveTechniques,Seattle.Bourgu
et,M.-L.(2006).Towardsataxonomyoferror-handlingstrategiesinrecognition-basedmult
i-modalhumancomputerinterfaces.SignalProcessingJournal86(12):362543.Bourguet,M.-L.
,&Ando,A.(1998).Synchronizationofspeechandhandgesturesduringmul-timodalhumancompu
terinteraction.Paperpresentedatthe1998SIGCHIConferenceonHumanFactorsinComputingS
ystems.Availableathttp://portal.acm.org/citation.cfm?id=286726&coll=portal&dl=AC
M&CFID=6362723&CFTOKEN=90890358.Calvert,G.A.,Spence,C.E.D.T.,&Stein,B.E.,eds.(20
04).TheHandbookofMultisensoryProcessing.Cambridge,MA:MITPress.Cassell,J.,Sulliva
n,J.W.,Prevost,S.,&Churchill,E.F.,eds.(2000).EmbodiedConversa-tionalAgents.Cambr
idge,MA:MITPress.Cheyer,A.J.,&Julia,L.(1995).Multimodalmaps:Anagent-basedapproac
h.Paperpre-sentedatthe1995InternationalConferenceonCooperativeMultimodalCommunic
ation,Eindhoven,TheNetherlands.Cheyer,A.J.,&Luc,J.(1998).MVIEWS:Multimodaltoolsf
orthevideoanalyst.Paperpre-sentedatthe4thInternationalConferenceonIntelligentUse
rInterfaces,SanFrancisco.Codella,C.,Jalili,R.,Koved,L.,Lewis,J.B.,Ling,D.T.,Lips
comb,J.S.,etal.(1992).Inter-activesimulationinamulti-personvirtualworld.Paperpre
sentedatthe1992SIGCHIConferenceonHumanFactorsinComputingSystems,Monterey,CA.Cohe
n,M.M.,&Massaro,D.W.(1993).Modelingcoarticulationinsyntheticvisualspeech.InMagne

nat-Thalmann,M.,&Thalmann,D.,eds.,ModelsandTechniquesinComputerAnimation.Tokyo:S
pringer-Verlag,13959.Cohen,P.R.(1992).Theroleofnaturallanguageinamultimodalinterf
ace.Paperpresentedatthe5thAnnualACMSymposiumonUserInterfaceandSoftwareTechnology
,Monterey,CA.
12MultimodalInterfaces:CombiningInterfaces434

Cohen,P.R.,Dalrymple,M.,Moran,D.B.,Pereira,F.C.N.,Sullivan,J.W.,Gargan,R.A.,Schl
ossberg,J.L.,&Tyler,S.W.(1989).Synergisticsuseofdirectmanipulationandnat-urallan
guage.ProceedingsofHumanFactorsinComputingSystems(CHI89).NewYork:ACMPress,22734.Co
hen,P.R.,Johnston,M.,McGee,D.R.,Oviatt,S.L.,Pittman,J.,Smith,I.A.,etal.(1997).Qu
ickSet:Multimodalinteractionfordistributedapplications.Paperpresentedatthe5thACM
InternationalConferenceonMultimedia,Seattle.Cohen,P.R.,&McGee,D.R.(2004).Tangibl
emultimodalinterfacesforsafety-criticalappli-cations.CommunicationsoftheAssociat
ionforComputingMachinery47(1):4146.Cohen,P.R.,McGee,D.,&Clow,J.(2000).Theefficien
cyofmultimodalinteractionforamap-basedtask.Paperpresentedatthe6thAppliedNaturalL
anguageProcessingConference,Seattle.Cohen,P.R.,&Oviatt,S.L.(1995).Theroleofvoice
inputforhumanmachinecommunication.ProceedingsofNationalAcademyofSciencesoftheUnit
edStatesofAmerica92(22):992127.Dalal,M.,Feiner,S.K.,McKeown,K.R.,Pan,S.,Zhou,M.X.
,Hollerer,T.,etal.(1996).Negotiationforautomatedgenerationoftemporalmultimediapr
esentations.Paperpre-sentedattheFourthAnnualACMInternationalConferenceonMultimed
ia,Boston.Danninger,M.,Flaherty,G.,Bernardin,K.,Ekenel,H.,Kohler,T.,Malkin,R.,et
al.(2005).Theconnector:Facilitatingcontext-awarecommunication.Paperpresentedatth
e7thInternationalConferenceonMultimodalInterfaces,Trento,Italy.Demirdjian,D.,Ko,
T.,&Darrell,T.(2003).Constraininghumanbodytracking.Paperpre-sentedattheNinthIEEE
InternationalConferenceonComputerVision,Nice.Deng,L.,Wang,K.,Acero,A.,Hon,H.-W.,
Droppo,J.,Boulis,C.,etal.(2002).Tap-to-talkinaspecificfield:Distributedspeechpro
cessinginmiPadsmultimodaluserinterface.IEEETransactionsonComputerSpeechandAudioPr
ocessing10(8):60519.Duncan,L.,Brown,W.,Esposito,C.,Holmback,H.,&Xue,P.(1999).Enha
ncingvirtualmaintenanceenvironmentswithspeechunderstanding.TechnicalReportTECHNE
T-9903,BoeingMathematicsandComputingTechnology.Dupont,S.,&Luettin,J.(2000).Audio
-visualspeechmodelingforcontinuousspeechrecog-nition.IEEETransactionsonMultimedi
a2(3):14151.Ehlen,P.,Purver,M.,&Niekrasz,J.(2007).Ameetingbrowserthatlearns.Paper
pre-sentedattheAAAI2007SpringSymposium:InteractionChallengesforArtificialAssis-t
ants,Stanford,CA.Ellis,C.A.,&Barthelmess,P.(2003).TheNeemdream.Paperpresentedatt
he2ndTapiaConferenceonDiversityinComputing,Atlanta.Epps,J.,Oviatt,S.L.,&Chen,F.(
2004).Integrationofspeechandgestureinputsduringmultimodalinteraction.Paperpresen
tedatthe2004AustralianInternationalConferenceonComputerHumanInteraction.Available
athttp://www.cse.unsw.edu.au/~jepps/ozchi04.pdf.Falcon,V.,Leonardi,C.,Pianesi,F.
,Tomasini,D.,&Zancanaro,M.(2005).Co-locatedsup-portforsmallgroupmeetings.Paperpr
esentedatthe2005SIGCHIConferenceonHumanFactorsinComputingSystems,Workshop:TheVir
tualityContinuumRevisited.Availableathttp://portal.acm.org/citation.cfm?id=10571
23&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.Faure,C.,&Julia,L.(1994).Ana
gent-basedarchitectureforamultimodalinterface.Paperpresentedatthe1994AAAISpringS
ymposium,PaloAlto,CA.
References435

Fell,H.J.,Delta,H.,Peterson,R.,Ferrier,L.J.,Mooraj,Z.,&Valleau,M.(1994).Usingthe
baby-babble-blanketforinfantswithmotorproblems:Anempiricalstudy.Paperpresentedat
theFirstAnnualACMConferenceonAssistiveTechnologies,MarinadelRey,CA.Flanagan,J.L.
,&Huang,T.S.(2003).Scanningtheissue:Specialissueonhumancomputermultimodalinterfac
es.ProceedingsoftheIEEE91(9):126771.Fukumoto,M.,Suenaga,Y.,&Mase,K.(1994).Fingerpointer:Pointinginterfacebyimageprocessing.Computers&Graphics18(5):63342.Grudin,J

.(1988).WhyCSCWapplicationsfail:Problemsinthedesignandevaluationoforganizational
interfaces.Paperpresentedatthe1988ACMConferenceonComputer-SupportedCooperativeWo
rk,Portland,OR.Gruenstein,A.,Niekrasz,J.,&Purver,M.(2005).Meetingstructureannota
tion:Dataandtools.Paperpresentedatthe9thEuropeanConferenceonSpeechCommunicationa
ndTechnology,6thSIGdialWorkshoponDiscourseandDialogue.Availableathttp://www.mit.
edu/~alexgru/pubs/sigdial05.pdf.Gupta,A.K.,&Anastasakos,T.(2004).Integrationpatt
ernsduringmultimodalinteraction.Paperpresentedatthe8thInternationalConferenceonS
pokenLanguageProcessing.Halverson,C.A.,Horn,D.B.,Karat,C.-M.,&Karat,J.(1999).The
beautyoferrors:Patternsoferrorcorrectionindesktopspeechsystems.Paperpresentedatt
he7thIFIPInterna-tionalConferenceonHumanComputerInteraction,Edinburgh.Hauptmann,A
.G.(1989).Speechandgesturesforgraphicimagemanipulation.Paperpresentedatthe1989SI
GCHIConferenceonHumanFactorsinComputingSystems,Austin,TX.Hina,M.D.,Ramdane-Cheri
f,A.,&Tadj,C.(2005).Aubiquitouscontext-sensitivemulti-modalmultimediacomputingsy
stemanditsmachinelearning-basedreconfigurationatthearchitecturallevel.Paperprese
ntedattheSeventhIEEEInternationalSymposiumonMultimedia,Irvine,CA.Horndasch,A.,Ra
pp,H.,&Rottger,H.(2006).SmartKom-Public.InWahlster,W.,ed.,SmartKom:Foundationsof
MultimodalDialogueSystems.Berlin:Springer,47192.Huang,X.,Acero,A.,Chelba,C.,Deng,
L.,Droppo,J.,Duchene,D.,etal.(2001).MiPad:Amultimodalinteractionprototype.Paperp
resentedattheAcoustics,Speech,andSignalProcessingConference(ICASSP).Huang,X.,&Ov
iatt,S.L.(2005).Towardadaptiveinformationfusioninmultimodalsys-tems.Paperpresent
edattheSecondJointWorkshoponMultimodalInteractionandRelatedMachineLearningAlgori
thms.Huang,X.,Oviatt,S.L.,&Lunsford,R.(2006).Combiningusermodelingandmachinelear
ningtopredictusersmultimodalintegrationpatterns.Paperpresentedatthe3rdJointWorksh
oponMultimodalInteractionandRelatedMachineLearningAlgorithms.Johnston,M.,&Bangal
ore,S.(2004).MATCHKiosk:Amultimodalinteractivecityguide.Paperpresentedatthe42ndA
nnualMeetingoftheAssociationofComputationalLinguis-tics.Availableathttp://acl.ld
c.upenn.edu/acl2004/postersdemos/pdf/johnston.pdf.Johnston,M.,&Bangalore,S.(2005
).Finite-statemultimodalintegrationandunderstand-ing.NaturalLanguageEngineering1
1(2):15987.Johnston,M.,Bangalore,S.,Vasireddy,G.,Stent,A.,Ehlen,P.,Walker,M.,etal
.(2002).MATCH:Anarchitectureformultimodaldialoguesystems.Paperpresentedatthe40th
AnnualMeetingoftheAssociationofComputationalLinguistics,DiscourseandDialogue.Ava
ilableathttp://portal.acm.org/citation.cfm?id=1073083.1073146.
12MultimodalInterfaces:CombiningInterfaces436

Johnston,M.,Cohen,P.R.,McGee,D.R.,Oviatt,S.L.,Pittman,J.A.,&Smith,I.A.(1997).Uni
fication-basedmultimodalintegration.Paperpresentedatthe35thAnnualMeetingoftheAss
ociationforComputationalLinguistics.Availableathttp://portal.acm.org/citation.cf
m?id=976909.979653&dl=GUIDE&dl=%23url.dl.Jovanovic,N.,opdenAkker,R.,&Nijholt,A.(
2006).Addresseeidentificationinface-to-facemeetings.Paperpresentedatthe11thConfe
renceoftheEuropeanChapteroftheAssoci-ationforComputationalLinguistics,Trento,Ita
ly.Kaiser,E.C.(2005).Multimodalnewvocabularyrecognitionthroughspeechandhandwritinginawhiteboardschedulingapplication.Paperpresentedatthe10thInternationalConfer
enceonIntelligentUserInterfaces,SanDiego.Kaiser,E.C.(2006).Usingredundantspeecha
ndhandwritingforlearningnewvocabularyandunderstandingabbreviations.Paperpresente
datthe8thInternationalConferenceonMultimodalInterfaces,OralSession5:SpeechandDia
logueSystems.Availableathttp://portal.acm.org/citation.cfm?id=1181060&coll=porta
l&dl=ACM&CFID=6362723&CFTOKEN=90890358.Kaiser,E.C.,&Barthelmess,P.(2006).Edge-sp
littinginacumulativemultimodalsystem,forano-waittemporalthresholdoninformationfu
sion,combinedwithanunder-specifieddisplay.Paperpresentedatthe9thInternationalCon
ferenceonSpokenLanguageProcessing.Kaiser,E.C.,Barthelmess,P.,Erdmann,C.G.,&Cohen
,P.R.(2007).Multimodalredun-dancyacrosshandwritingandspeechduringcomputermediate
dhumanhumaninterac-tions.Paperpresentedatthe2007SIGCHIConferenceonHumanFactorsinC
omputingSystems,SanJose,CA.Kaiser,E.C.,Barthelmess,P.,Huang,X.,&Demirdjian,D.(20
05).Ademonstrationofdistributedpointingandreferencingformultimodalcollaborationo
versketcheddia-grams.Paperpresentedatthe7thInternationalConferenceonMultimodalIn
terfaces,WorkshoponMultimodal,MultipartyMeetingProcessing.Availableathttp://www.

csee.ogi.edu/~kaiser/distgest_demo.3.pdf.Kaiser,E.C.,Demirdjian,D.,Gruenstein,A.
,Li,X.,Niekrasz,J.,Wesson,R.M.,etal.(2004).Amultimodallearninginterfaceforsketch
,speakandpointcreationofaschedulechart.Paperpresentedatthe6thInternationalConfer
enceonMultimodalInterfaces,DemoSession2.Availableathttp://portal.acm.org/citatio
n.cfm?id=1027992&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.Katzenmaier,M.
,Stiefelhagen,R.,&Schultz,T.(2004).Identifyingtheaddresseeinhumanhumanrobotinterac
tionsbasedonheadposeandspeech.Paperpresentedatthe6thInternationalConferenceonMul
timodalInterfaces,StateCollege,PA.Kendon,A.(1981).Gesticulationandspeech:Twoaspe
ctsoftheprocessofutterance.InKey,M.R.,ed.,TheRelationshipofVerbalandNonverbalCom
munication.TheHague:MoutondeGruyter,20727.Koons,D.B.,Sparrell,C.J.,&Thorisson,K.R
.(1993).Integratingsimultaneousinputfromspeech,gaze,andhandgestures.InMaybury,M.
T.,ed.,IntelligentMultimediaInter-faces.Cambridge,MA:AAAIPress/MITPress,25776.Lar
son,J.A.(2003).VoiceXML:IntroductiontoDevelopingSpeechApplications.UpperSaddleRi
ver,NJ:PrenticeHall.Larson,K.,&Mowatt,D.(2003).Speecherrorcorrection:Thestoryoft
healternateslist.InternationalJournalofSpeechTechnology8(2):18394.
References437

Lauer,C.,Frey,J.,Lang,B.,Becker,T.,Kleinbauer,T.,&Alexandersson,J.(2005).AmiGram
:Ageneral-purposetoolformultimodalcorpusannotation.Paperpresentedatthe2ndJointWo
rkshoponMultimodalInteractionandRelatedMachineLearningAlgorithms.Lucente,M.,Zwar
t,G.-J.,&George,A.(1998).VisualizationSpace:Atestbedfordevicelessmultimodaluseri
nterface.Paperpresentedatthe1998AAAISpringSymposiumonApplyingMachineLearningandD
iscourseProcessing,PaloAlto,CA.Lunsford,R.,Oviatt,S.L.,&Arthur,A.M.(2006).Toward
open-microphoneengagementformultipartyinteractions.Paperpresentedatthe8thInterna
tionalConferenceonMultimodalInterfaces,Banff,Alb.Lunsford,R.,Oviatt,S.L.,&Coulst
on,R.(2005).Audio-visualcuesdistinguishingself-fromsystem-directedspeechinyounge
randolderadults.Paperpresentedatthe7thInternationalConferenceonMultimodalInterfa
ces,Trento,Italy.Mankoff,J.,Hudson,S.E.,&Abowd,G.D.(2000).Interactiontechniquesf
orambiguityresolutioninrecognition-basedinterfaces.Paperpresentedatthe13thAnnual
ACMSymposiumonUserInterfaceSoftwareandTechnology,SanDiego.Martin,J.-C.,&Kipp,M.(
2002).AnnotatingandmeasuringmultimodalbehaviourTYCOONmetricsintheAnviltool.Paperp
resentedatthe3rdInternationalConferenceonLanguageResourcesandEvaluation.Availabl
eathttp://www.dfki.uni-sb.de/~kipp/public_archive/MartinKipp2002-lrec.ps.Massaro
,D.W.,&Stork,D.G.(1998).Speechrecognitionandsensoryintegration.Ameri-canScientis
t86(3):23644.McCowan,I.,Gatica-Perez,D.,Bengio,S.,Lathoud,G.,Barnard,M.,&Zhang,D.
(2005).Automaticanalysisofmultimodalgroupactionsinmeetings.IEEETransactionsonPat
ternAnalysisandMachineIntelligence27(3):30517.McGee,D.R.(2003).AugmentingEnvironm
entswithMultimodalInteraction.PhDdiss.,OregonGraduateInstituteofScienceandTechno
logy.McGee,D.R.,&Cohen,P.R.(2001).Creatingtangibleinterfacesbytransformingphysic
alobjectswithmultimodallanguage.Paperpresentedatthe6thInternationalConferenceonI
ntelligentUserInterfaces,SantaFe,NM.McGee,D.R.,Pavel,M.,Adami,A.,Wang,G.,&Cohen,
P.R.(2001).Avisualmodalityfortheaugmentationofpaper.Paperpresentedatthe3rdWorksh
oponPerceptual/PerceptiveUserInterfaces,Orlando.McGee,D.R.,Pavel,M.,&Cohen,P.R.(
2001).Contextshifts:Extendingthemeaningofphysicalobjectswithlanguage.HumanCompute
rInteraction16(24):35162.McGrath,M.,&Summerfield,Q.(1985).Intermodaltimingrelation
sandaudio-visualspeechrecognitionbynormal-hearingadults.JournaloftheAcousticalSo
cietyofAmerica77(2):67885.McGurk,H.,&MacDonald,J.(1976).Hearinglipsandseeingvoice
s.Nature264:74648.McLeod,A.,&Summerfield,Q.(1987).Quantifyingthecontributionofvis
iontospeechperceptioninnoise.BritishJournalofAudiology21(2):13141.McNeil,D.(1992)
.HandandMind:WhatGesturesRevealaboutThought.Chicago:Univer-sityofChicagoPress.Me
ier,U.,Hurst,W.,&Duchnowski,P.(1996).Adaptivebimodalsensorfusionforauto-maticspee
chreading.Paperpresentedatthe1996IEEEInternationalConferenceonAcoustics,Speech,a
ndSignalProcessing,Atlanta.
12MultimodalInterfaces:CombiningInterfaces438

Miller,D.,Culp,J.,&Stotts,D.(2006).Facetoptablet:Note-takingassistancefordeafper
sons.Paperpresentedatthe8thInternationalACMSIGACCESSConferenceonComputersandAcce
ssibility,Portland,OR.Morency,L.-P.,Sidner,C.L.,Lee,C.,&Darrell,T.(2005).Context
ualrecognitionofheadgestures.Paperpresentedatthe7thInternationalConferenceonMult
imodalInterfaces,Trento,Italy.Morimoto,C.H.,Koons,D.B.,Amir,A.,Flickner,M.,&Zhai
,S.(1999).KeepinganeyeforHCI.Paperpresentedatthe12thBrazilianSymposiumonComputer
GraphicsandImageProcessing.Availableathttp://doi.ieeecomputersociety.org/10.1109
/SIBGRA.1999.805722.Mousavi,S.Y.,Low,R.,&Sweller,J.(1995).Reducingcognitiveloadb
ymixingauditoryandvisualpresentationmodes.JournalofEducationalPsychology87(2):31
934.Naughton,K.(1996).Spontaneousgestureandsign:AstudyofASLsignsco-occurringwiths
peech.Paperpresentedatthe1996WorkshopontheIntegrationofGestureinLanguage&Speech.
Availableathttp://www.sign-lang.uni-hamburg.de/BibWeb/LiDat.acgi?ID=40674.Neal,J
.G.,&Shapiro,S.C.(1991).Intelligentmulti-mediainterfacetechnology.InSullivan,J.W
.,&Tyler,S.W.,eds.,IntelligentUserInterfaces.NewYork:Addison-Wesley,1143.Negropon
te,N.(1978).TheMediaRoom:ReportforONRandDARPA.Cambridge,MA:MIT,ArchitectureMachi
neGroup.Nigay,L.,&Coutaz,J.(1993).Adesignspaceformultimodalsystems:Concurrentpro
ces-singanddatafusion.Paperpresentedatthe1993SIGCHIConferenceonHumanFactorsinCom
putingSystems,Amsterdam.Nigay,L.,&Coutaz,J.(1995).Agenericplatformforaddressingt
hemultimodalchallenge.Paperpresentedatthe1995SIGCHIConferenceonHumanFactorsinCom
putingSystems,Denver.Nijholt,A.(2006).Towardstheautomaticgenerationofvirtualpres
enteragents.InformingScience9:97110.Nijholt,A.,Rienks,R.J.,Zwiers,J.,&Reidsma,D.(
2006).Onlineandoff-linevisualizationofmeetinginformationandmeetingsupport.Visual
Computer22(12):96576.Oliver,N.,&Horvitz,E.(2004).S-SEER:Selectiveperceptioninamul
timodalofficeactivityrecognitionsystem.PaperpresentedattheFirstInternationalWork
shoponMachineLearningforMultimodalInteraction,Martigny,Switzerland.Oulasvirta,A.
,Tamminen,S.,Roto,V.,&Kuorelahti,J.(2005).Interactionin4-secondbursts:Thefragmen
tednatureofattentionalresourcesinmobileHCI.Paperpresentedatthe2005SIGCHIConferen
ceonHumanFactorsinComputingSystems,Portland,OR.Oviatt,S.L.(1995).Predictingspoke
ndisfluenciesduringhumancomputerinteraction.ComputerSpeechandLanguage9(1):1935.Ovi
att,S.L.(1996a).Multimodalinterfacesfordynamicinteractivemaps.Paperpresentedatth
e1996SIGCHIConferenceonHumanFactorsinComputingSystems.Availableathttp://portal.a
cm.org/citation.cfm?id=238438&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.O
viatt,S.L.(1996b).User-centeredmodelingforspokenlanguageandmultimodalinter-faces
.IEEETransactionsonMultimedia3(4):2635.Oviatt,S.L.(1997).Multimodalinteractivemap
s:Designingforhumanperformance.HumanComputerInteraction12(12):93129.
References439

Oviatt,S.L.(1999a).Mutualdisambiguationofrecognitionerrorsinamultimodalarchitecture.Paperpresentedatthe1999SIGCHIConferenceonHumanFactorsinComputingSystems,CHI
Letters.Availableathttp://portal.acm.org/citation.cfm?id=303163&coll=portal&dl=A
CM&CFID=6362723&CFTOKEN=90890358.Oviatt,S.L.(1999b).Tenmythsofmultimodalinteract
ion.CommunicationsoftheAssoci-ationforComputingMachinery42(11):7481.Oviatt,S.L.(2
000).Tamingrecognitionerrorswithamultimodalinterface.Communica-tionsoftheAssocia
tionforComputingMachinery43(9):4551.Oviatt,S.L.(2002).Breakingtherobustnessbarrie
r:Recentprogressonthedesignofrobustmultimodalsystems.AdvancesinComputers56:30541.
Oviatt,S.L.(2006a).Human-centereddesignmeetscognitiveloadtheory:Designinginterfa
cesthathelppeoplethink.Paperpresentedatthe14thAnnualACMInternationalConferenceon
Multimedia,SantaBarbara,CA.Oviatt,S.L.,ed.(2006b).EncyclopediaofMultimedia,2nded
.Boston:KluwerAcademicPublishers.Oviatt,S.L.,(2007).Multimodalinterfaces.InChen,
F.,&Jokinen,J.,eds.NewTrendsinSpeech-BasedInteractiveSystems,2nded.NewYork:Sprin
ger.Oviatt,S.L.,&Adams,B.(2000).Designingandevaluatingconversationalinterfaceswi
thanimatedcharacters.InCassell,J.,Sullivan,J.W.,Prevost,S.,&Churchill,E.F.,eds.,
EmbodiedConversationalAgents.Cambridge,MA:MITPress,31943.Oviatt,S.L.,Arthur,A.M.,

&Cohen,J.(2006).Quietinterfacesthathelpstudentsthink.Paperpresentedatthe19thAnnu
alACMSymposiumonUserInterfaceSoftwareandTechnology,Montreaux,Switzerland.Oviatt,
S.L.,&Cohen,P.R.(1991).Discoursestructureandperformanceefficiencyininteractivean
dnoninteractivespokenmodalities.ComputerSpeechandLanguage5(4):297326.Oviatt,S.L.,
Cohen,P.R.,Fong,M.W.,&Frank,M.P.(1992).Arapidsemi-automaticsim-ulationtechniquef
orinvestigatinginteractivespeechandhandwriting.Paperpresentedatthe2ndInternation
alConferenceonSpokenLanguageProcessing,Banff,Alb.Oviatt,S.L.,Cohen,P.R.,&Wang,M.
Q.(1994).Towardinterfacedesignforhumanlan-guagetechnology:Modalityandstructureas
determinantsoflinguisticcomplexity.SpeechCommunication15:283300.Oviatt,S.L.,Cohen
,P.R.,Wu,L.,Duncan,L.,Suhm,B.,Bers,J.,etal.(2000).Designingtheuserinterfaceformu
ltimodalspeechandgestureapplications:State-of-the-artsystemsandresearchdirection
s.HumanComputerInteraction15(4):263322.Oviatt,S.L.,Coulston,R.,&Lunsford,R.(2004).
Whendoweinteractmultimodally?Cog-nitiveloadandmultimodalcommunicationpatterns.Pa
perpresentedatthe6thInterna-tionalConferenceonMultimodalInterfaces,Pittsburgh.Ov
iatt,S.L.,Coulston,R.,&Lunsford,R.(2005).JustdowhatItellyou:Thelimitedimpactofin
structionsonmultimodalintegrationpatterns.Paperpresentedatthe10thInterna-tionalC
onferenceonUserModeling,Edinburgh.Oviatt,S.L.,Coulston,R.,Tomko,S.,Xiao,B.,Lunsf
ord,R.,Wesson,R.M.,etal.(2003).Towardatheoryoforganizedmultimodalintegrationpatt
ernsduringhumancomputerinteraction.Paperpresentedatthe5thInternationalConferenceo
nMultimodalInter-faces,Vancouver,BC.
12MultimodalInterfaces:CombiningInterfaces440

Oviatt,S.L.,Darves,C.,Coulston,R.,&Wesson,R.M.(2005).Speechconvergencewithanimat
edpersonas.SpokenMultimodalHumanComputerDialogueinMobileEnviron-ments28:37997.Ovia
tt,S.L.,DeAngeli,A.,&Kuhn,K.(1997).Integrationandsynchronizationofinputmodesduri
ngmultimodalhumancomputerinteraction.Paperpresentedatthe1997SIGCHIConferenceonHum
anFactorsinComputingSystems.Availableathttp://portal.acm.org.liboff.ohsu.edu/cit
ation.cfm?id=258821&coll=portal&dl=ACM&CFID=6362723&CFTOKEN=90890358.Oviatt,S.L.
,&Kuhn,K.(1998).Referentialfeaturesandlinguisticindirectioninmulti-modallanguage
.Paperpresentedatthe5thInternationalConferenceonSpokenLanguageProcessing,Sydney.
Oviatt,S.L.,Lunsford,R.,&Coulston,R.(2005).Individualdifferencesinmultimodalinte
grationpatterns:Whataretheyandwhydotheyexist?Paperpresentedatthe2005SIGCHIConfer
enceonHumanFactorsinComputingSystems,Portland,OR.Oviatt,S.L.,&Olsen,E.(1994).Int
egrationthemesinmultimodalhumancomputerinterac-tion.Paperpresentedatthe3rdInterna
tionalConferenceonSpokenLanguageProcessing.Oviatt,S.L.,&VanGent,R.(1996).Errorre
solutionduringmultimodalhumancomputerinteraction.Paperpresentedatthe4thInternatio
nalConferenceonSpokenLanguageProcessing,Philadelphia.Pavlovic,V.I.,Berry,G.A.,&H
uang,T.S.(1997).Integrationofaudio/visualinformationforuseinhumancomputerintellig
entinteraction.Paperpresentedatthe1997IEEEInternationalConferenceonImageProcessi
ng,SantaBarbara,CA.Pianesi,F.,Zancanaro,M.,Falcon,V.,&Not,E.(2006).Towardsupport
inggroupdynamics.Paperpresentedatthe3rdIFIPConferenceonArtificialIntelligenceApp
licationsandInnovations,Athens.Poppe,P.(2007).SpecialIssueonVisionforHumanCompute
rInteraction.ComputerVisionandImageUnderstanding108(1-2):418Potamianos,G.,Neti,C.
,Luettin,J.,&Matthews,I.(2004).Audio-visualautomaticspeechrecognition:Anoverview
.InBailly,G.,Vatikiotis-Bateson,E.,&Perrier,P.,eds.,IssuesinVisualandAudio-Visua
lSpeechProcessing.Cambridge,MA:MITPress.Purver,M.,Ehlen,P.,&Niekrasz,J.(2006).De
tectingactionitemsinmulti-partymeetings:Annotationandinitialexperiments.InRenals
,S.,Bengio,S.,&Fiscus,J.,eds.,MachineLearningforMultimodalInteraction.NewYork:Sp
ringer-Verlag,20011.Purver,M.,Kording,K.P.,Griffiths,T.L.,&Tenenbaum,J.B.(2006).Un
supervisedtopicmodellingformulti-partyspokendiscourse.Paperpresentedatthe21stInt
ernationalConferenceonComputationalLinguistics,44thAnnualMeetingoftheACL,Sydney.
Reithinger,N.,Alexandersson,J.,Becker,T.,Blocher,A.,Engel,R.,Lockelt,M.,etal.(200
3).SmartKom:Adaptiveandflexiblemultimodalaccesstomultipleapplications.Paperpre-s
entedatthe5thInternationalConferenceonMultimodalInterfaces,Vancouver,BC.Reithing
er,N.,&Herzog,G.(2006).AnexemplaryinteractionwithSmartKom.InWahlster,W.,ed.,Smar
tKom:FoundationsofMultimodalDialogueSystems.Berlin:Springer,4152.Rienks,R.J.,&Hey

len,D.K.J.(2005).Dominancedetectioninmeetingsusingeasilyobtainablefeatures.Paper
presentedatthe2ndJointWorkshoponMultimodalInterac-tionandRelatedMachineLearningA
lgorithms,Edinburgh.
References441

Rienks,R.J.,Nijholt,A.,&Barthelmess,P.(2006).Pro-activemeetingassistants:Attenti
onplease!Paperpresentedatthe5thWorkshoponSocialIntelligenceDesign,Osaka.Robert-R
ibes,J.,Schwartz,J.-L.,Lallouache,T.,&Escudier,P.(1998).Complementarityandsynerg
yinbimodalspeech:Auditory,visual,andauditory-visualidentificationofFrenchoralvow
elsinnoise.JournaloftheAcousticalSocietyofAmerica103(6):367789.Rogozan,A.,&Delegli
se,P.(1998).Adaptivefusionofacousticandvisualsourcesforauto-maticspeechrecogniti
on.SpeechCommunication26(12):14961.Ruiz,N.,Taib,R.,&Chen,F.(2006).Examiningtheredu
ndancyofmultimodalinput.Paperpresentedatthe2006AustralianInternationalConference
onComputerHumanInterac-tion,Sydney.Salber,D.,&Coutaz,J.(1993a).ApplyingtheWizardo
fOztechniquetothestudyofmulti-modalsystems.PaperpresentedattheThirdInternational
ConferenceonHumanComputerInteraction,Moscow.Salber,D.,&Coutaz,J.(1993b).AWizardof
Ozplatformforthestudyofmultimodalsys-tems.Paperpresentedatthe1993SIGCHIConferenc
eonHumanFactorsinComputingSystems,Amsterdam.Sellen,A.J.,&Harper,R.H.(2003).TheMy
thofthePaperlessOffice.Cambridge,MA:MITPress.Siroux,J.,Guyomard,M.,Multon,F.,&Re
mondeau,C.(1995).ModelingandprocessingoftheoralandtactileactivitiesintheGeoralta
ctilesystem.PaperpresentedattheFirstInternationalConferenceonCooperativeMultimod
alCommunication,Eindhoven.Stock,O.,&Zancanaro,M.,eds.(2005).MultimodalIntelligen
tInformationPresentations.NewYork:Springer.Stotts,D.,Bishop,G.,Culp,J.,Miller,D.
,Gyllstrom,K.,&Lee,K.(2005).FacetopontheTabletPC:AssistiveTechnologyinSupportofC
lassroomNote-TakingforHearing-ImpairedStudents.TechnicalReport.ChapelHill:Univer
sityofNorthCarolina.Suhm,B.(1998).MultimodalInteractiveErrorRecoveryforNon-Conve
rsationalSpeechUserInterfaces.PhDdiss.,UniversityofKarlsruhe(Germany).Suhm,B.,My
ers,B.A.,&Waibel,A.(1999).Model-basedandempiricalevaluationofmul-timodalinteract
iveerrorcorrection.Paperpresentedatthe1999SIGCHIConferenceonHumanFactorsinComput
ingSystems,Pittsburgh.Sumby,W.H.,&Pollack,I.(1954).Visualcontributiontospeechint
elligibilityinnoise.JournaloftheAcousticalSocietyofAmerica26(2):21215.Summerfield
,Q.(1992).Lipreadingandaudio-visualspeechperception.PhilosophicalTransactionsoft
heRoyalSocietyofLondon:BiologicalSciences335(1273):7178.Tang,A.,McLachlan,P.,Lowe
,K.,Saka,C.R.,&MacLean,K.(2005).Perceivingordinaldatahapticallyunderworkload.Pap
erpresentedatthe7thInternationalConferenceonMultimodalInterfaces,Trento,Italy.Ta
ng,J.C.(1991).Findingfromobservationalstudiesofcollaborativework.InternationalJo
urnalofManMachineStudies34(2):14360.Tomlinson,M.J.,Russell,M.J.,&Brooke,N.M.(1996)
.Integratingaudioandvisualinfor-mationtoprovidehighlyrobustspeechrecognition.Pap
erpresentedatthe1996IEEEInternationalConferenceonAcoustics,Speech,andSignalProce
ssing,Atlanta.Turk,M.,&Robertson,G.(2000).Perceptualuserinterfaces(introduction)
.Communica-tionsoftheAssociationforComputingMachinery43(3):3234.
12MultimodalInterfaces:CombiningInterfaces442

VanLeeuwen,D.A.,&Huijbregts,M.A.H.(2006).TheAIMspeakerdiarizationsystemforNISTRT
06smeetingdata.PaperpresentedattheNISTRichTranscription2006SpringMeetingRecogniti
onEvaluation.vanTurnhout,K.,Terken,J.,Bakx,I.,&Eggen,B.(2005).Identifyingtheinte
ndedaddresseeinmixedhumanhumanandhumancomputerinteractionfromnon-verbalfeatures.Pa
perpresentedatthe7thInternationalConferenceonMultimodalInterfaces,Trento,Italy.V
atikiotis-Bateson,E.,Munhall,K.G.,Hirayama,M.,Lee,Y.,&Terzopoulos,D.(1996).Thedy
namicsofaudiovisualbehaviorofspeech.SpeechreadingbyHumansandMachines:Models,Syst
emsandApplications150:22132.Verbree,D.,Rienks,R.J.,&Heylen,D.K.J.(2006a).Dialogue
-acttaggingusingsmartfea-tureselection:Resultsonmultiplecorpora.Paperpresentedat

theFirstInternationalIEEEWorkshoponSpokenLanguageTechnology,PalmBeach,FL.Verbree
,D.,Rienks,R.J.,&Heylen,D.K.J.(2006b).Firststepstowardstheautomaticcon-struction
ofargument-diagramsfromrealdiscussions.FrontiersinArtificialIntelligenceandAppli
cations144:18394.Vergo,J.(1998).Astatisticalapproachtomultimodalnaturallanguagein
teraction.PaperpresentedattheAAAIWorkshoponRepresentationsformulti-modalHumanComp
uterInteraction,Madison,WI.Verlinden,M.,Zwitserlood,I.,&Frowein,H.(2005).Multime
diawithAnimatedSignLanguageforDeafLearners.PaperpresentedattheWorldConferenceonE
ducationalMultimedia,Hypermedia&Telecommunications,ED-MEDIA,Montreal.Vo,M.T.(1998
).AFrameworkandToolkitfortheConstructionofMultimodalLearningInterfaces.PhDdiss.,
CarnegieMellonUniversity.Vo,M.T.,&Wood,C.(1996).Buildinganapplicationframeworkfo
rspeechandpeninputintegrationinmultimodallearninginterfaces.Paperpresentedatthe1
996IEEEInterna-tionalConferenceonAcoustics,Speech,andSignalProcessing,Atlanta.Wa
hlster,W.(2006).Dialoguesystemsgomultimodal:TheSmartKomexperience.InWahlster,W.,
ed.,SmartKom:FoundationsofMultimodalDialogueSystems.Berlin:Springer,327.Wahlster,
W.,Andre,E.,Finkler,W.,Profitlich,H.-J.,&Rist,T.(1993).Plan-basedintegra-tionofn
aturallanguageandgraphicsgeneration.ArtificialIntelligence63:387427.Wainer,J.,&Br
aga,D.P.(2001).Symgroup:Applyingsocialagentsinagroupinteractionsystem.Paperprese
ntedatthe2001InternationalACMSIGGROUPConferenceonSupportingGroupWork,Boulder,CO.
Wang,J.(1995).Integrationofeye-gaze,voiceandmanualresponseinmultimodaluserinterf
aces.Paperpresentedatthe1995IEEEInternationalConferenceonSystems,ManandCyberneti
cs,Vancouver,BC.Wang,K.(2004).Frommultimodaltonaturalinteractions.Paperpresented
attheW3Work-shoponMultimodalInteraction.PositionPaper.Availableathttp://www.w3.o
rg/2004/02/mmi-workshop/wang-microsoft.html.Wang,S.,&Demirdjian,D.(2005).Inferri
ngbodyposefromspeechcontent.Paperpresentedatthe7thInternationalConferenceonMulti
modalInterfaces.Wickens,C.D.,Sandry,D.,&Vidulich,M.(1983).Compatibilityandresour
cecompetitionbetweenmodalitiesofinput,centralprocessing,andoutput.HumanFactors25
(2):22748.
References443

Xiao,B.,Girand,C.,&Oviatt,S.L.(2002).Multimodalintegrationpatternsinchildren.Pap
erpresentedatthe7thInternationalConferenceonSpokenLanguageProcessing,Denver.Xiao
,B.,Lunsford,R.,Coulston,R.,Wesson,R.M.,&Oviatt,S.L.(2003).Modelingmulti-modalin
tegrationpatternsandperformanceinseniors:Towardadaptiveprocessingofindividualdif
ferences.Paperpresentedatthe5thInternationalConferenceonMulti-modalInterfaces,Va
ncouver,BC.Zancanaro,M.,Lepri,B.,&Pianesi,F.(2006).Automaticdetectionofgroupfunc
tionalrolesinfacetofaceinteractions.Paperpresentedatthe8thInternationalConferenc
eonMultimodalInterfaces,Banff,Alb.Zhai,S.,Morimoto,C.H.,&Ihde,S.(1999).Manualand
gazeinputcascaded(MAGIC)pointing.Paperpresentedatthe1999SIGCHIConferenceonHumanF
actorsinComput-ingSystems,Pittsburgh.Zoltan-Ford,E.(1991).Howtogetpeopletosayand
typewhatcomputerscanunderstand.InternationalJournalofManMachineStudies34(4):52747.
12MultimodalInterfaces:CombiningInterfaces444

IndexAccelerometers,112113drift,119FoodSimulatorproject,294295Accessibilityapplica
tionsauditoryinterfaces,165166hapticinterfaces,4651IVRinterfaces,232233mobileapplic
ations,413415multimodeinterfaces,368369,386olfactoryinterfaces,282283small-screenin
terfaces,321322,330335Accuracyhapticpositionsensing,62multimodeinterfaces,361362Aco
usticmodels,210211Actionpriority,ingestureinterfaces,98Active-matrixdisplays,312A
ctivemovement,inhapticinterfacedesign,61Actuationhapticinterfaces,5758smellgenera
tion,270Adaptationauditoryenvironmentsanddisplays,188189multimodalinterfaces,42742
8olfactoryinterfaces,284Additivesoundsynthesis,158159Addressbooks,366Adduction,919
2Admittancecontrol,27AdobeAuditiontool,242Aircannons,279280,282Alesis3630compress
or/limiter,247Alternativefeedback,insmall-screeninterfaces,348Alternativemodalit

ies,inmultimodalinterfaces,428AmericanswithDisabilitiesAct,387AmiGramtool,426Amo
ore,J.E.,269Animatedcharacters,399Annotations,inmultimodalinterfaces,426Announce
mentvoice,forIVRinterfaces,240241,248,262263Annoyance,sound,163Anviltool,426Applie
dVoiceInputOutputSociety(AVIOS),215Arguments,forgestureinterfaces,96Armexoskelet
ons,4043Aromas.SeeOlfactoryinterfacesARTHURproject,96Aspectratios,315Attributiont
heory,199200Audification,149,160Audio-onlygames,165Auditorycues,inlocomotioninter
faces,123Auditorydisplaysystems(ADSs),156157Auditoryicons,149,160Auditoryinterfac
es,8,147148,161162accessibility,165166advanced,163165applications,171172audification,
149,160casestudies,187constraints,170171cross-modalcollaborativedisplay,180182data
representationandprocessing,157158

Auditoryinterfaces(Continued)designguidelines,182187dimensions,153154displaysystem
s,156157,159161,174175,177182drawbacks,163futuretrends,187189humanfactorsdesign,16717
humansensitivity,151153implementations,161166perceptualconsiderations,154156require
ments,173174,182183sonification,148149,161,188soundproduction,176177soundproperties,
150151soundsynthesis,158159symbolic/semanticrepresentations,149150,175taskanalysis,
167170testing,177182Auditoryoverload,163Auditorysenses,inlocomotion,111Automatedco
nversations,198199designguidelines,220224errorhandling,202208,221224escapehatches,22
4personas,200202,221prompts,201208terminology,224Automatedself-help,365366Automatic
speechrecognition(ASR),208Axel,R.,269Backgroundnoise,154155Baddeley,A.D.,420Baeck
er,R.,334Ballas,J.A.,178Ballotdebacle,13Bandwidthsensorymotorcontrol,35sound,152
Bankingapplications,229,232Barbagli,F.,34Barras,S.,182Beaconlesstrackers,119Beac
ons,117119Beatgestures,85Bekesy,G.,285Bellik,Y.,413Benchmarks,forgestureinterface
s,99101Bergamasco,M.,42Bernard,M.L.,333Binarymenuchoices,inIVRinterfaces,251252BIP
.SeeBreak-in-presenceeventsBitingforcegenerations,299301measurements,297299Bittert
aste,291BlackBerryPearl,319Blattner,M.,185186Blending,inolfactoryinterfaces,270,2
72277Blindnesscomputergamesfor,165hapticinterfacesfor,4750multimodalinterfacesfor,
413414Boilingpoint,ofodorintensity,283Bolt,R.A.,394,403Bonevibrationmicrophones,2
93295Bottom-upVUIsystems,198Bowers,J.,180182Bowman,D.A.,132Boxinggame,56Brailledisp
lays,48,414Braininterfacesbraincomputer,2021small-screen,353Breadth,ofmenus,332333B
reak-in-presence(BIP)events,134135Brewster,M.,183,186Brightness,insmall-screenint
erfaces,307308,311BTreeinterfaces,334Buck,L.,269Bufferunitgenerators,159Burger,D.,
413Butterflyballotdebacle,13Cameras,insmall-screeninterfacetesting,339342Cancelin
gIVRinterfaceactions,249,257Capabilitiesdesignfor,1415hapticinterfaces,6061locomot
ioninterfaces,137138Carbonfibertubing,35Cardsorttechniques,237Cartesianjoysticks,
3839Casestudiesauditoryinterfaces,187gestureinterfaces,102hapticinterfaces,64
Index446

IVRinterfaces,264locomotioninterfaces,139multimodalinterfaces,430multimodeinterf
aces,386olfactoryinterfaces,289small-screeninterfaces,351tasteinterfaces,304VUIs
,224Cassell,J.,77,8586Cater,J.P.,276Cathoderaytubes(CRTs),311CAVEsystem,121Ceilin
g-mountedhapticinterfaces,41Charactersize,insmall-screeninterfaces,326327Charters
ystem,401Cheese,inbitingforcegeneration,299301Chemicalsenses,267268,296Chen,J.,3334
Chin,J.P.,333Christian,K.,4748Classificationscomputervisioninterfaces,7879gesturei
nterfacedevelopment,102Clickwheels,318Cognitivefactorsauditoryinterfaces,178loco
motioninterfaces,133134multimodalinterfaces,416420Collaborationauditoryinterfaces,
180182multimodalinterfaces,400403,431432Collapse-to-zoomtechnique,338Collars,scent,
278279Colorperception,310311Commandgestures,86Commercialspeech.SeeVoiceuserinterfa
ces(VUIs)CommonGUIelements,insmall-screeninterfaces,336339Communicationsassistant
s(CAs),233Compactscentgenerators,271Compass,119Complementaritygestureinterfaces,
97multimodalinterfaces,417418Complexdata,sonificationof,148149Complianceresolution
,3334Comprehensiononsmall-screeninterfaces,336Computationalplatforms,insensorymot
orcontrol,36Computervisioninterfaces,7879Concentration,inodorintensity,283Concept

designandmodelsauditoryinterfaces,183184multimodalinterfaces,403405supportfor,13Co
ncurrencyauditoryinterfaces,185multimodeinterfaces,382Cones,eye,308Confirminguse
rinput,inIVRinterfaces,260Confusion,inmultimodeinterfaces,362Conscious,bottom-up
,gestureinterfacedevelopment,98Consistencydesignfor,1516IVRinterfaces,254255multim
odeinterfaces,383384Constraints,inauditoryinterfacedesign,170171Contactarea,inexos
keletonsdevices,62Contactmicrophonetaps,245246Contextauditoryinterfaces,170,175,1
78conversations,198gestureinterfaces,87,9596multimodalinterfaces,429430Contrast,in
small-screeninterfaces,308,327328Controlbandwidth,insensorymotorcontrol,35Control
gestures,86Control,inhapticinterfaces,27Convergedmultimodeinterfaces,363Conversa
tionaldeathspirals,202Conversationsautomated.SeeAutomatedconversationsvoice.SeeV
oiceuserinterfaces(VUIs)Coordinateframes,inhapticinterfacedesign,62Cordlessphone
s,245Corneas,eye,308Corrupteddata,inmultimodeinterfaces,362Crackers,inbitingforc
egeneration,299301Criticalfeedback,insmall-screeninterfaces,348Cross-adaptation,i
nolfactoryinterfaces,284Cross-modaldisplay,inauditoryinterfaces,180182CUBRICONmul
timodalsystem,395Culturalissues,ingestureinterfaces,94
Index447

Customerservicerepresentatives(CSRs),233234Cutaneoustouch,25Cybersickness,130132,1
36137CyberSphere,125126DanceDanceRevolutiongame,120,122Dataaccuracy,inmultimodeint
erfaces,361362Dataanalysis,inmultimodalinterfaces,425426Datacollectionandminingaud
itoryinterfaces,158,179IVRinterfaces,235237,263264multimodalinterfaces,423Dataglov
e,78Datapolarity,ofsound,153154Datarepresentationandprocessing,inauditoryinterfac
es,157158Datarequirementshapticinterfaces,5356specifications,5658Datatransfer,inIVR
interfaces,247Daysofweekcollection,inIVRinterfaces,264Deafness.SeealsoAuditoryin
terfacesIVRinterfacesfor,233multimodalinterfacesfor,414Deathspirals,conversation
al,202DegreesoffreedomFoodSimulatorproject,293294hapticinterfacedevices,3638,56Dei
cticgestures,85Delays,inIVRinterfaces,259Delta-sigmasimulation,276Depth,ofmenus,
332333Descriptivelabels,ingestureinterfaces,8587Descriptivemenuoptions,inVUIs,221D
esignguidelinesauditoryinterfaces,182187gestureinterfaces,9498hapticinterfaces,6064
IVRinterfaces,247264locomotioninterfaces,137139multimodalinterfaces,426430multimode
interfaces,374375,377,381386olfactoryinterfaces,286288small-screeninterfaces,343351t
asteinterfaces,304VUIs,220224Designprinciples,12consistency,1516errorrecovery,17fe
edback,1617future,1823simplicity,1718userconceptualmodelandgoals,13userknowledge,14
userskillsandcapabilities,1415Desktop,inauditoryinterfaceaccessibility,165166Detai
ldesign,inauditoryinterfaces,184187Detectionthresholdvalue,inolfactoryinterfaces,
284Dial-through,inIVRinterfaces,249250Dialogboxes,insmall-screeninterfaces,337Dia
logdesign,184Dialogmanagers,inmultimodalinterfaces,405Dialogs,directed,217Dictat
ionsystems,214Dictionaries,inspeechrecognitionsystems,210211Differentialthreshold
value,inolfactoryinterfaces,284Diffusers,insmellgeneration,270,273275Diffusivity,
inodorintensity,283Digitalaudioplayers,319Digitalpaper-basedinterfaces,411413Digi
talstorageoscilloscopes(DSOs),320Dimensions,auditory,153154Directeddialogs,217Dir
ection,inlocomotioninterfaces,108,110DisabilitiesBrailledisplays,48computergames
for,165IVRinterfaces,233multimodalinterfaces,413414multimodeinterfaces,380Disambi
guationgestureinterfaces,9798multimodalinterfaces,392393Discoursemarkers,223224Disp
laysauditoryinterfaces,156157,159161,174175,177182locomotioninterfaces,121small-scre
eninterfaces,312314,329330Disruptiveinterpretations,428429Distinctmenuoptions,inVUI
s,221Distraction,inlocomotioninterfaces,133134DMS100system,231Documentcameras,ins
mall-screeninterfacetesting,341342Dollars&Scentssystem,274275
Index448

Dotsperinch(DPI),insmall-screeninterfaces,314315,329DragonsLair,21Drift,119DSLite,
320Dualtonemultifrequency(DTMF)keypadsIVRinterfaces,229230,238,242,246247small-scr

eeninterfaces,316Duchnicky,R.L.,336Durlach,N.I.,31Dynamicgestures,86Dynamicorgan
ization,ofsmall-screenspace,338EarBenderdatabase,182Earcons,149,160Earlyfusion,i
nmultimodalinterfaces,404Earlyprototyping,177178Earlytesting,inmultimodeinterface
s,377378Echoicmemory,155Economics,inlocomotioninterfaces,142143Effectivedesigns,12
Efficientdesigns,12small-screeninterfaces,350Electrocardiogram(ECG)machines,148E
lectronicnose(e-nose),286Electronicpaper(e-paper),313Elkerton,J.,336Emblems,inge
stureinterfaces,85Emotion,auditoryinterfacesfor,162EmotionTool,77Environmentsaud
itoryinterfaces,170hapticinterfaces,2728small-screeninterfaces,324EquipmentIVRint
erfacetesting,245247small-screeninterfaces,320321Equivalence,ingestureinterfaces,9
7Ergonomics,ingestureinterfaces,9193Errorhandlingandcorrection,17automatedconvers
ations,202208,221224IVRinterfaces,238239,261multimodalinterfaces,392393,428small-scr
eeninterfaces,347Errorspirals,428Ertan,S.,50Escapehatches,inautomatedconversatio
ns,224eSignproject,414Evaluatinghapticinterfaces,5960Evaporation,insmellgeneratio
n,269Exoskeletons,4043,5354,62Exploratoryprototypes,378ExtensibleMarkupLanguage(XM
L),387Extraneousinformation,insmall-screeninterfaces,344345Extremedelays,inIVRint
erfaces,259Eyepositionandtracking,insmall-screeninterfaces,323324,353Eyes,308Face
gazeandexpression,79Face-mountedodor-emittingdevices,288FacetopTablet,414415Facia
lexpressionsgestureinterfaces,77multimodalinterfaces,399Facialgestures,77Familia
rityofusers,inauditoryinterfaces,167Featurecreep,18Features,inmultimodalinterfac
esfusion,404identification,421422Feedbackdesignfor,1617gestureinterfaces,9697IVRint
erfaces,259260kinesthetic,5257locomotioninterfaces,116,121124small-screeninterfaces
,348349tactile,52,5556,58FieldMedicInformationSystem,397Field-of-view(FOV),inlocom
otioninterfaces,132Filemanagement,ingestureinterfaces,90Filteringsound,153Finger
s,ingestureinterfaces,9193Firefightingsimulators,276,278Five-DOFarmexoskeletons,4
2Fixedtemporalthreshold,inmultimodalinterfaces,405Flatdisplays,inlocomotioninter
faces,121Fleming,S.A.,110Flexibility,inmultimodeinterfaces,387FlexibleOLEDdispla
ys,352FlexiForcesensors,297Flightsimulators,123Floor-mountedhapticinterfaces,41F
loorswitches,formotionsensing,120Flowcharts,inIVRinterfaces,238239
Index449

Flying,inlocomotioninterfaces,124,132Foleyeffects,164Fonts,forsmall-screeninterf
aces,327329FoodSimulatorproject,292293bitingforcegeneration,299301bitingforcemeasur
ements,297299designguidelines,304futuretrends,304305hapticinterfaces,293294multisen
sorydisplay,294297testing,302304Foot-candles(ft-c),308Foot-Lamberts(ft-L),308Forbr
ig,P.,360,374375Forcebitingmeasurements,297299FoodSimulatorproject,293handcontroll
ers,3637hapticinterfacedesign,61sensorymotorcontrol,3334Formalmodels,formultimodal
interfaces,422Forms-basedinterfaces,427Four-DOFjoysticks,38Freeman,W.T.,93Freque
ncy,sound,150152Friction,inhapticinterfacedesign,63Fullyintegratedmultimodeinterf
aces,363Functionalconsistency,inIVRinterfaces,255Functionallabels,ingestureinter
faces,8586Functionalprototypes,245Functionality,insmall-screeninterfaces,344Fusio
n,inmultimodalinterfaces,403404Futuretrendsauditoryinterfaces,187189design,1823gest
ureinterfaces,103hapticinterfaces,6465IVRinterfaces,264265locomotioninterfaces,1411
43multimodalinterfaces,430432multimodeinterfaces,386388olfactoryinterfaces,289smal
l-screeninterfaces,351353tasteinterfaces,304305VUIs,224226Gaitreproduction,128129Gai
tersystem,128Games,56audioin,164165mobile,320Ganttcharts,401Gassensors,286Gaver,W.
W.,185Gazeinterface,353Geigercounterexample,174Geographicpositioningsystems(GPS)
,166Gestureinterfaces,57,7577,102103applicability,77,7980computervisioninterfaces,787
9contextawareness,9596culturalissues,94designguidelines,9498ergonomicsandintuitivi
ty,9193facegazeandexpression,79feedback,9697future,103heuristicsandmetaphors,8890hu
manfactorsdesign,8793integrationanddisambiguation,9798mechanicalandtactileinterfac
es,78Midastouch,94multimodalinterfaces,399400nature,8081processandbenchmarks,99101r
elationtoothermodes,8184,9798sequencing,9495taxonomies,8587technical-vs.human-basedg
estures,8790technology,77vocabulary,98102Geven,A.,333Globalpreferences,inmultimode
interfaces,385Goalsdesignfor,13locomotioninterfaces,137GoogleVoiceLocalSearch,22
5Grammars,inspeechrecognitionsystems,210211Graphicalinformation,hapticaccessto,484
9Graphicaluserinterfaces(GUIs),forsmall-screeninterfaces,312,336339Gravitationalf

ield,119Grice,H.P.,198Grounding,inhapticinterfaces,57Groupbehavior,multimodalcom
munication,402403Groupingsauditoryinterfaces,155IVRinterfacedatacollection,237Gui
danceofuserinput,inmultimodalinterfaces,422
Index450

Habitation,inolfactoryinterfaces,284Hajian,A.,34Hale,K.S.,52Handeyecoordinationta
sks,52Handmasters,4244Handsgestureinterfaces,9193hapticinterfaces,3637,4244Handwriti
ngrecognition,396HapticandAudioVirtualEnvironment(HAVE),4849Hapticinterfaces,45,252
6,5658accessibilityapplications,4651casestudies,64datarequirements,5356designguidel
ines,6064FoodSimulatorproject,293294fundamentals,2628futuretrends,6465humanfactorsde
sign,5158humanperception,3032humansensing,2830implementations,3638nonportable,3842por
table,4243sensorymotorcontrol,3235tactile,4346technology,35testing,5860Hardkeys,318H
ardwareauditoryinterfaces,156157hapticinterfaces,2627HarmonyRemotemirrors,321Headd
irectedsystems,124,139Head-mounteddisplays(HMDs),121Head-relatedtransferfunction
s(HRTFs),157Headphones,inauditoryinterfaces,156157Heads-updisplays,352Hearing.See
alsoAuditoryinterfacesIVRinterfaces,233multimodalinterfacesfor,414Heilig,Morton,
272Helfrich,W.,311Helicoptersimulation,4Helio-displays,352HelpIVRinterfaces,256m
ultimodeinterfaces,365366Hermann,T.,187Heuristicsauditoryinterfaces,179180gesturei
nterfaces,8890HiBall3100tracker,117HiddenMarkovModels(HMMs),208,211High-fidelitysi
mulations,423425High-resolutionsensors,58Hints,inautomatedconversations,222Histor
icalnotesauditoryinterfaces,161162multimodalinterfaces,394403olfactoryinterfaces,2
72speechrecognition,212213Holisticusabilitymetrics,220Holographicinterfaces,2122Ho
meapplications,forspeechrecognition,225Honey,ImHomesystem,274Horizontalscrolling,
insmall-screeninterfaces,346347Howe,R.,34,44Hue,insmall-screeninterfaces,311Human
-basedgestures,8788Human-centricWordProcessor,397Humanissues,inhapticinterfaces,6
061perception,3032sensing,2830Humansensitivity,inauditoryinterfaces,151153Hunter,I.W
.,31HybridCouplerproduct,247Hybriddevices,insmall-screeninterfaces,321322Hybridtr
ackers,119HyperBrailleapplication,48Hyperextension,9192Hyperlinking,insmall-scree
ninterfaces,347Hypertextmarkuplanguage(HTML),387Iconicgestures,85Icons,insmall-s
creeninterfaces,334iGesturePad,7576,78Illuminance,insmall-screeninterfaces,308Ill
usions,115Immediatememory,inauditoryinterfaces,155Immersiveinterfaces,82Impedanc
eshapticinterfaces,27,63sensorymotorcontrol,3435Impermanence,inauditoryinterface,
163IMPULSEsystem,119In-personusabilitytesting,forVUIs,219
Index451

Inertialtrackingsystems,119Informationappliances,321Informationflowauditoryinter
faces,169,174175multimodalinterfaces,405406small-screeninterfaces,323330Information
kiosks,407409Informationportals,364365InformativeIVRinterfacemenus,250Injectionpum
ps,297Innkeeperproduct,247Inputmodes,ingestureinterfaces,95Inputtechnologies,for
small-screeninterfaces,315318Inside-looking-outtrackers,117Insko,B.,134Instabilit
ies,inhapticinterfacedesign,62inStinksystem,274275InstituteofElectronicMusic(IEM)
workshop,182183Instructions,forautomatedconversations,222223Integration,ofgesturei
nterfaces,9798Intelligentauditoryenvironmentsanddisplays,188189Interactionloops,lo
comotioninterfacesas,116117Interactionstandards,forsmall-screeninterfaces,349Inte
ractivesonification,161,188Interactivevoiceresponse(IVR)interfaces,9,229230,237238
accessibilityapplications,232233announcementvoice,240241,248,262263casestudy,264dat
acollection,235237,263264designguidelines,247264dial-through,249250errorstatements,2
38239,261flowcharts,238239futuretrends,264265humanfactorsdesign,233242implementation
s,232233keyassignments,254259menus,250252metaphors,253254openingmessage,248promptsan
dfeedback,240242,252253,259261technology,231testing,242247timeouts,262usercontrol,24
8249uses,233235wordingconventions,253Intercomsystem,15Interference/masking,inaudit
oryinterfaces,163InternationalCommunityonAuditoryDisplays(ICAD),148International

issues,inIVRinterfaces,240241Interpretations,inmultimodalinterfaces,428429Interrup
tingsystemoutput,inIVRinterfaces,249InterSenseIS900trackers,117,119Intuitivity,in
gestureinterfaces,9193iPhone,321322iPodNano,319IS900trackers,117,119iSICsystem,164i
Smellscentdiffuser,273275Isolatedwords,inspeechrecognitionsystems,212iTapinputdev
ices,316Iterativeevaluationandtestingauditoryinterfaces,177multimodeinterfaces,3
81IVR.SeeInteractivevoiceresponseinterfacesJavahery,H.,360,374375JawsforWindows(J
FW)screenreader,165JekyllandHydepersonas,200202Jointtesting,formultimodeinterface
s,376,382383Joints,ingestureinterfaces,9193Jones,L.A.,31,46Jones,M.,334Joysticks,3
839Justnoticeabledifferences(JNDs),inhapticinterfaces,31,34K-Sonarsystem,166Kaori
-Webprogram,276277Kaye,J.N.,274Keyassignments,inIVRinterfaces,254259Keyboards,insm
all-screeninterfaces,316317Keypadnumbers,inIVRinterfaces,255Kinesthesia,3032Kinest
heticdevices,25,5257Kiosks,information,407409Kirusamultimodalsystem,411412
Index452

Kolers,P.A.,336Kontarinis,D.A.,44Kramer,G.,174,184Labelsgestureinterfaces,8587mul
timodeinterfaces,384Laboratories,forsmall-screeninterfacetesting,342Landmarksloc
omotioninterfaces,112VUIs,222Languageissuesautomatedconversation,199IVRinterface
s,240241multimodalinterfaces,416420,427speechsystems,217Latefusion,404Latertesting
,inmultimodeinterfaces,379380Leaning,inlocomotioninterfaces,124Learningeffects,in
auditoryinterfaces,179Lee,E.S.,331332Lenses,eye,308Letterkeypadentries,inIVRinter
faces,255256LIBERTYtrackers,117Lightcharacteristics,307308Light-emittingdiode(LED)
displays,313LightExoskeletondevice,4243Lighting,forsmall-screeninterfaces,324,340
Lindsay,J.,166,179Linespacing,insmall-screeninterfaces,326Linetaps,245246Linguist
icissues.SeeLanguageissuesLipmovements,inmultimodalinterfaces,398399Liquidcrystal
displays(LCDs),311312Lists,insmall-screeninterfaces,338Localization,ofsoundsource
s,153Lockoutsystems,inmultimodeinterfaces,362Locomotioninterfaces,78,107111casestu
dy,141cybersickness,130132,136137designguidelines,137139feedbackdisplays,116,121124f
lyingandleaning,124,132futuretrends,141143gaitreproduction,128129humanfactors,12713
2implementations,124128asinteractionloops,116117magicalandunnaturalinterfaces,12712
8multimodalinterfaces,414naturalness,127128physiologicalnature,111115poseandmotion
sensors,108,117121realismandpreservationofspatialunderstanding,135136testing,132137
walking,124127Logicalgestures,99Logistics,forVUIs,219LogitechHarmonyremotecontrol
s,321LoneWolfgame,165Longdelays,inIVRinterfaces,259Lostwork,inmultimodeinterface
s,362Loudness,152Loudspeakersauditoryinterfaces,156157vibrotactiledisplays,4445Low
resolution,inauditoryinterfaces,163Lucent5Esystem,231Luminance,insmall-screenint
erfaces,308,327328MacGregor,J.N.,334Maddox,M.E.,323MAGICsystem,399,406Magicalinte
rfaces,110,127128Magnetictrackers,119MAHIarmexoskeleton,4041Mailboxes,inmultimodei
nterfaces,366Manipulationgestures,86Mapsauditoryinterfaces,154multimodalinterfac
es,412413Mark-ups,formultimodalinterfaces,426Markovmodels,79,208,211Marsden,G.,33
4Marshall,C.C.,336Maskingauditoryinterfaces,154155olfactoryinterfaces,284285Massfl
owcontrollers(MFCs),270Massie,T.,40Masterarms,4043Masticationsounds,294295MATCHsys
tem,405MATCHKiosksystem,407408Matsunaga,K.,21Maximumforceexertion,insensorymotorc
ontrol,32
Index453

McCarthy,R.E.,272Mechanicalimpedance,insensorymotorcontrol,3435Mechanicalinterfac
es,forgestures,78Mechanicalmotiontrackers,119Mechanoreceptors,2830Mediacentertech
nology,321Medicalequipment,small-screeninterfacesfor,320321Mediumdelays,inIVRinte
rfaces,259Meissnercorpuscles,30Memoryauditory,155gestureinterfacedevelopment,100
short-termmemorystudies,220Mentaleffort,forsmall-screeninterfaces,350351MenusIVRi
nterfaces,250252small-screeninterfaces,330335,337VUIs,220221Merkeldisks,30Messagema
ilboxes,inmultimodeinterfaces,366Metaloxidegassensors,286Metaphorsauditoryinterf

aces,185gestureinterfaces,85,8890IVRinterfaces,253254locomotioninterfaces,109,137s
mall-screeninterfaces,349Meter-candles(m-c),308Microphonetaps,245246MicrophonesFo
odSimulatorproject,293295VUIrecognitionprocess,209MicrosoftBob,1920Midastouch,94MI
DI(musicalinstrumentdigitalinterface)files,171Miller,G.,220Millilamberts(mLs),30
8Mineproduct,127MiPadapplication,410411Mitsopoulos,E.N.,184MMUIexample,368Mnemoni
cs,inIVRinterfaces,254255Mobileapplicationsaccessibilityapplications,413415auditor
yinterfaces,166multimodalinterfaces,409413,430431small-screeninterfaces,320MobileA
udioDesigns(MAD)Monkey,179Mobilephonesauditoryinterfaces,188189keypads,316sales,3
07small-screeninterfaces,318319MoCapBoxinggame,56Mochizuki,A.,278,280Modepriority,
ingestureinterfaces,98Model-basedsonification(MBS),161Modelsautomatedconversatio
ns,222223self-motion,113115Molecules,odor,267268Monitoring,audiofor,164Moosemouse,4
849Morie,J.F.,278,281Morsecode,317Motioncapturesystems,119Motiondata,inhapticinte
rfaces,27Motioneffect,inolfactoryinterfaces,285Motioninterfaces.SeeLocomotionint
erfacesMotionplatforms,123Motionranges,ingestureinterfaces,9193Motionsensors,80,1
17121Motorizedtreadmills,123,125126Mousehaptic,4849small-screeninterfaces,316Mousta
kas,K.,80Multimodalfusion,403404Multimodalinterfaces,391,432advantages,391393audit
oryinterfaces,188casestudies,430collaborativesystems,400403,431432conceptsandmecha
nisms,403405dataanalysis,425426datacollection,423designguidelines,426430futuretrend
s,430432gestureinterfacesfor,81high-fidelitysimulations,423425historyandevolution,
394403humanfactorsdesign,415422informationflow,405406informationkiosks,407409linguis
ticandcognitivefactors,416420mobileapplications,409413,430431pensystems,396398,418419
,431presentation,399speechsystems,396399,418,431
Index454

technology,394testing,423426vision-basedmodalitiesandpassiveinterfaces,399Multimo
deinterfaces,1112,359361accessibilityapplications,368369,386addressbooks,366automat
edself-help,365366casestudy,386design,374375,377,381386existingapplications,375376fu
turetrends,386388humanfactorsdesign,369377informationportals,364365messagemailboxes
,366modalitychoices,370372remoteaccesstoservices,366researchrequirements,369370tas
kallocation,372374technology,361363testing,376383Multiplelanguages,inIVRinterfaces,
240241Multiplemappings,inauditoryinterfaces,154Multipliereffect,inolfactoryinterf
aces,284285Multisensorydisplay,intasteinterfaces,294297Multispeakersystems,157Mult
itapentries,316Multitaskingoperatingsystems,350Mundanelocomotioninterfaces,110Mu
sicalexpertise,ofauditoryinterfaceusers,167Muto,W.H.,323Mutualdisambiguation,3923
93MVIEWSsystem,397Nakamoto,T.,276Nanoaudioplayer,319Narrowmenus,332333Naturaladap
tation,inmultimodalinterfaces,427428Naturallanguages,inspeechsystems,217Naturalne
ss,inlocomotioninterfaces,127128Navigationhaptic,5051menus.SeeMenussmall-screenint
erfaces,344Neemsystem,400Negativeconditionals,inIVRstatements,252Negativepolarit
y,insound,154Negativewordsandphrases,inIVRerrormessages,261NicheVUImarkets,214Ni
ntendoDSLite,320NISMapapplication,396,412413Nociceptors,30Noise,inauditoryinterfa
ces,154155Nokia5110mobilephone,334335NokiaSeries60,350351NOMOStool,426Nonlinearity,
inolfactoryinterfaces,285Nonportablehapticinterfaces,3842Nonspeechsoundsforaccess
ibility,165166speechrecognition,210Norman,D.,343Norman,K.L.,333NortelDMS100system
,231Nose-mountedodor-emittingdevices,288Nosestructure,267268NowPlayingscreen,344O
dors.SeeOlfactoryinterfacesOkamura,A.M.,45OLEDdisplays,313,352Olfactometersystem
s,272274Olfactoryinterfaces,910,267269,272accessibilityapplications,282283adaptation
,284casestudies,289designguidelines,286288futuretrends,289humanfactorsdesign,28328
5multipliereffectandmaskingeffect,284285nonlinearity,285odorintensity,283scentgen
erationandblending,270,272277smelldelivery,270271smellgeneration,269270spatialperce
ption,285,287spatiotemporalscentcontrol,276282technology,269temporalaspects,287288
testing,285286thresholdvalues,283284Omnidirectionaltreadmills,1251800-GOOG411voicese
archapplication,225One-degree-of-freedom(DOF)mechanisms,293294OneTouchUltrameters
,320321
Index455

Openingmessages,inIVRinterfaces,248OpinionVUImetrics,220Optacontactiledisplay,47
Opticnervefibers,308Opticalsystems,inlocomotioninterfaces,112113,120Opticaltracke
rs,119Orderlymenus,inIVRinterfaces,250OrganicLED(OLED)displays,313,352Orientatio
ntests,ingestureinterfacedevelopment,102Outside-looking-intrackers,119Over-the-s
houldercameratesting,339340Oviatt,S.L.,416428Paciniancorpuscles,30PantoBrailledisp
lay,48Pantographdisplay,48Paper-basedinterfaces,411413Parallelmechanisms,inhaptic
interfaces,5657Parametermappingsonification(PMS),160161Participantchoices,inaudito
ryinterfaces,178179Passive-matrixscreens,312Passivemultimodalinterfaces,399Passiv
evoice,inIVRstatements,252Pauses,inIVRinterfaces,263Peltierpumps,58Pensystemshap
ticdevices,4041multimodalinterfaces,396398,418419,431PDAs,410Penetrationofobjects,i
nhapticinterfaces,6162Pentland,A.,46Perceptualconsiderations,inauditoryinterfaces
,154156Performancehapticinterfaces,5960multimodalinterfaces,393,421Periodicsound,1
50Personalidentificationnumbercollection,263PersonalTransporter(PT),Segway,107108
Personas,inautomatedconversations,200202,221Peterson,B.,136PHANToMdesktophapticin
terface,3638,4041,56Phase,insound,150151PhaseSpaceIMPULSEsystem,119Phonenumbercolle
ction,263Phonetaps,245246Phonesmobile.SeeMobilephonesVUIsystems.SeeVoiceuserinter
facesPhotoannotators,401402Photopicvision,308309Photoreceptors,eye,308Physicaldist
raction,inlocomotioninterfaces,134Physicaleffort,insmall-screeninterfaces,350351P
hysicalspacerequirements,forreal-walkinginterfaces,141Physiologicalnature,ofloco
motioninterfaces,111115Pilotinglocomotioninterfaces,112Pindensity,inhapticinterfa
cedesign,64PiratesoftheCaribbeanride,123Pirhonen,A.,183Pitch,sound,151152,154Pixe
ls,insmall-screeninterfaces,328329Placeholders,ingestureinterfaces,96PlayBufunitg
enerator,159PocketPCs,336,342Podokineticcues,111112Pointofview(POV),inlocomotioni
nterfaces,116Pointinggestures,86Polarity,sound,148,153154PolhemusLIBERTYtrackers,
117Polkosky,M.,220Pop-upmenus,insmall-screeninterfaces,337Portabledigitalassista
nt(PDA)penandspeechinput,410small-screeninterfaces,319Portablehapticinterfaces,4
243PortableVoiceAssistant,397Porter,J.,285Pose,inlocomotioninterfaces,108,117121Po
sition-sensingaccuracy,inhapticinterfacedesign,62Positiontests,ingestureinterfac
edevelopment,102Positioning,insmall-screeninterfaces,323324Positivesoundpolarity,
154Pound(#)keys,inIVRinterfaces,258259Power-denseactuators,35PPPsystem,406
Index456

PrecisionPositionTracker,119Preferencesmultimodeinterfaces,385386small-screeninte
rfaces,345Presence,inlocomotioninterfaces,134135Presentation,multimodal,399Presen
tationlayer,inautomatedconversations,202208Preservation,ofspatialunderstanding,13
5136Pressuresensors,120Primaryodors,269Prioritizinginformation,insmall-screeninte
rfaces,347Privacyauditoryinterfaces,163IVRinterfaces,232,234235speechsystems,217P
roductionsystemprototypes,inIVRinterfacetesting,245Progressivedisclosure,insmall
-screeninterfaces,345346Projectionkeyboards,317Projectiontechnology,352Promptsaut
omatedconversations,201208IVRinterfaces,240242,252253,259260Pronation,91Pronunciatio
ns,inspeechrecognitionsystems,210Propertyissues,insonification,148Propositionalg
estures,85Proprioceptivecues,111112Prototypesauditoryinterfaces,177178IVRinterface
s,244245multimodalinterfaces,424multimodeinterfaces,378VUIs,218Psychologicaldimen
sions,ofcolor,311Pull-downmenus,insmall-screeninterfaces,337Pupils,eye,308PureDa
taenvironment,173Put-that-theredemonstration,394395,403Qualitativefeel,inlocomotion
interfaces,135Quantitativeindication,inlocomotioninterfaces,135Quickreprompts,in
automatedconversations,201Quickresponsetime,insmall-screeninterfaces,348349Quicks
etsystem,396,405QWERTYkeyboards,insmall-screeninterfaces,316317Rabbittactiledevic
e,46Radiance,insmall-screeninterfaces,308Rangesofmotion,ingestureinterfaces,9193R
apid-adapting(RA)mechanoreceptors,2930Rasasystem,397398,400Raymond,D.R.,331332RAZRp
hone,316Re-enteringinformation,inmultimodeinterfaces,384Readingandcomprehension,
insmall-screeninterfaces,336Real-walkinglocomotionsystems,109110,141Realism,inloc
omotioninterfaces,135136Recognitionprocess,inVUIs,209211Recognitionthresholdvalues

,inolfactoryinterfaces,284Recordedprompts,inIVRinterfaces,240242Recordingwindows,
inIVRinterfaces,260Redirectedwalking,141Redundancygestureinterfacemodalitycooper
ation,97multimodalinterfaces,417418RehabilitationAct,387Reminders,inautomatedconv
ersations,222Remoteaccesstoservices,366367Remotecollaboration,401Representational
systems,inmultimodalinterfaces,427Reprompts,inautomatedconversations,201Requirem
entsauditoryinterfaces,173174,182183hapticinterfaces,5358IVRinterfaceinformation,26
0multimodeinterfaces,369370Researchrequirements,inmultimodeinterfaces,369370Resolu
tionsensorymotorcontrol,3334small-screeninterfaces,314315Responseplanning,inmultim
odalinterfaces,405406Responsetime,insmall-screeninterfaces,348349Retinas,eye,308
Index457

Rhythmicphase,ingaitreproduction,129Rods,eyes,308Ruffinicorpuscles,30Ruotolo,C.,
336RutgersMasterIIhandmaster,4344Safety,inlocomotioninterfaces,138Salisbury,K.,40
Saltytaste,291Sarker,M.S.,319Satisfactorydesigns,12Saturation,insmall-screeninte
rfaces,311Savorinesstaste,291Scaling/contextissues,insonification,148149Scatterin
g-typearomagenerators,271Sceneanalysis,inauditoryinterfaces,155Scentcollars,27827
9,281ScentDomesystem,274,276ScentProjector,280,282ScentRemindersystem,274Scents.
SeeOlfactoryinterfacesSchadt,M.,311Scientificequipment,small-screeninterfacesfor
,320321Scotopicvision,308309Screenreaders,165Screenreplicationutility,342Screensiz
e,insmall-screeninterfaces,314315Scripts,forIVRinterfaces,241242Scrolling,insmallscreeninterfaces,338,346347Seamlessmultimodeinterfaces,363Seffah,A.,360,374375Segm
entation,incomputervisioninterfaces,7879SegwayPersonalTransporter,107108Self-adapt
ation,inolfactoryinterfaces,284Self-help,inmultimodeinterfaces,365366Self-motionp
erception,111115Self-servicecheckoutsystems,230Semanticfusion,inmultimodalinterfa
ces,404Semanticlabels,ingestureinterfaces,8586Semanticrepresentations,inauditoryi
nterfaces,149150Senses,inself-motionmodel,113115SenseViewsmallscreens,322Sensingba
ndwidth,insensorymotorcontrol,35Sensitivity,insmall-screeninterfaces,308309Sensor
-beacontrackers,119Sensoramasimulator,272Sensorsbitingforce,297gas,286gestureint
erfaces,8182hapticinterfaces,3235,5758locomotioninterfaces,117121Sensorysaltationphe
nomenon,46Sequencing,ingestureinterfaces,9495Sequentialintegrationpatterns,inmult
imodalinterfaces,416417Serialmechanisms,inhapticinterfaces,56ShadesofDoomgame,165
Shallowmenus,332333Shapememoryalloy(SMA)tactorunits,4647Sharingdata,inmultimodeint
erfaces,385Shimoga,K.,35ShopTalkmultimodalsystem,395Shortmessageservice(SMS)mess
ages,319Short-termmemorystudies,220Shortcuts,inIVRinterfaces,247Sidespeech,inspe
echrecognitionsystems,210Sightenhancementcomputergamesfor,165hapticinterfacesfor
,4750multimodalinterfacesfor,413414Signaldetectionanalysis,243244Simplicity,1718IVRi
nterfaces,247small-screeninterfaces,343346Simulationsfood.SeeFoodSimulatorproject
multimodalinterfaces,424425walking,124127Simulatorsickness,130132,136137Simulatorsic
knessquestionnaire(SSQ),136Simultaneouspatterns,inmultimodalinterfaces,416417Sing
ularities,inhapticinterfacedesign,63Sinusoids,150151Six-DOFDeltahapticinterface,3
637Sketchrecognition,401Skills,designfor,1415Slater-Usoh-Steedquestionnaire,134Slo
ttedgrammars,211Slow-adapting(SA)mechanoreceptors,2930Slurpsignal,175
Index458

Smalldelays,inIVRinterfaces,259Small-screeninterfaces,1011,307,311accessibility,3
21322,330335casestudy,351colorperception,310311commonGUIelements,336339designguideli
nes,343351digitalaudioplayers,319displaytechnologies,312314,329330eye,308feedback,3
48349futuretrends,351353humanfactorsdesign,322339hybriddevices,321322informationappl
iances,321informationformat,324330informationgathering,323324inputtechnologies,3153
18interactionstandards,349light,307308mediacentertechnology,321mentalandphysicale
ffort,350351menus,330335,337mobilegamingdevices,320mobilephones,318319readingandcom
prehensionon,336scientificandmedicalequipment,320321screenreplicationutility,342s

creensize,resolution,anddotsperinch,314315sensitivity,308309simplicity,343346testin
g,339342visualacuity,309311SmartKom-Publickiosk,408410SmartKomsystem,399,406,408410S
martphones,319SmartSceneVisualizerproduct,127Smell-eliminationequipment,271Smell
-o-visioninterfaces,10Smellitzersystem,272273SmellsFoodSimulatorproject,296olfact
oryinterfaces.SeeOlfactoryinterfacesSoftkeys,318Softnessresolution,3334Sonificati
on,8,148149,160161,188Sonify!product,164Soundfiles,inIVRinterfacetesting,244245Soun
dofTrafficprogram,164Sounds.SeealsoAuditoryinterfacesFoodSimulatorproject,293295p
roperties,150151synthesis,158159Sourtaste,291Spacing,insmall-screeninterfaces,32632
7Spatialinterfaces,353Spatialperceptionandlocationgestureinterfaces,84locomotion
interfaces,135136olfactoryinterfaces,285,287sound,153Spatialzones,ingestureinterf
aces,87,95Spatiotemporalgestures,86Spatiotemporalsmellcontrols,271,276282Speakerd
iarization,400Speaker-independentspeechrecognitionsystems,212Spearcons,149Specia
lportals,inlocomotioninterfaces,127Specialization,ingestureinterfacemodalitycoop
eration,97Specificationshapticinterfaces,5658IVRinterfaces,237238Spectralfiltering
,153Speechgestureinterfaces,84multimodalinterfaces,396399,403,418,431overview,89PD
As,410user-centereddesignfor,215216VUIs.SeeVoiceuserinterfacesSphericaljoysticks,
38Spontaneousgestures,85Srinivasan,M.A.,3334Standarderrortones,inIVRinterfaces,26
1Standardterminology,inIVRinterfaces,253Stanney,K.M.,52Star(*)key,inIVRinterface
s,257258Startperiods,inIVRinterfacedatacollection,264Statementphrasing,inIVRinter
faces,252253Staticgestures,86Statisticallanguagemodels(SLMs),211Statistics,inaudi
toryinterfaces,158SteelTalonssimulation,4Stereoeffect,inolfactoryinterfaces,285S
toakley,R.,127
Index459

Stopperiods,inIVRinterfacedatacollection,264Stresstests,ingestureinterfacedevelo
pment,101Strokewidth,insmall-screeninterfaces,326Subconscious,bottom-up,gesturei
nterfacedevelopment,98Subscribed-toIVRinterface,236Subtractivesoundsynthesis,1581
59Successivenesslimen(SL),32Summarythumbnails,339SuperCollider,159,173Supination
,91Supportchoices,inmultimodalinterfaces,429430SureTypeinputdevices,316Surgicalkn
ives,8384Surround-screendisplays,121122Surrounduserinterfaces,8788,95Sustainedforce
exertion,33Sweettaste,291Switches,telephone,231Symbolicrepresentations,ofauditor
yinterfaces,149150,175Synchronousonsetsandoffsets,inauditoryinterfaces,155Synthes
is,inauditoryinterfaces,158159Systemdelays,inIVRinterfaces,259Systemhelp,inIVRint
erfaces,256T9inputdevices,316Tabs,337Tactilecuesandinterfaces,4344feedbackdevices
,52,5556,58gestureinterfaces,78hapticinterfacedesign,61locomotioninterfaces,111112
vibrotactile,4445wearable,4546,50Tactileperception,3032TactileVisionSubstitutionSys
tem(TVSS),47TaDaanalysis,182183Tags,inmultimodalinterfaces,401402Tan,H.Z.,3334,46Ta
naka,K.,21Tanikawa,T.,278Taps,inIVRinterfacetesting,245247TaskLoadIndex(TLX),178T
asksauditoryinterfaces,167170IVRinterfacetesting,242243multimodeinterfaces,372374Ta
steinterfaces,10,291,292bitingforcegeneration,299301bitingforcemeasurements,297299
casestudy,304designguidelines,304difficulties,292existingtechniques,292FoodSimul
atorproject,292293futuretrends,304305humanfactorsdesign,297301multisensorydisplay,2
94297testing,302304Taxonomies,ingestureinterfaces,8587Technical-basedgestures,8790Te
chnicaltests,ingestureinterfacedevelopment,102TektronixTDS3052Boscilloscope,320T
elecommunicationsAct,387Teleportation,127Teletypewriters(TTYs),322Temporalaspect
sodorcontrol,287288sound,152153Temporalconstraints,inmultimodalinterfaces,405Tempo
ralintegrationpatterns,inmultimodalinterfaces,416417Temporalperception,ingesturei
nterfaces,84Terminatorkeys,inIVRinterfaces,256,258260TerminologyIVRinterfaces,2522
53VUIs,224Testingauditoryinterfaces,177182gestureinterfaces,100102hapticinterfaces
,5860IVRinterfaces,242247locomotioninterfaces,132137multimodalinterfaces,423426multi
modeinterfaces,376383olfactoryinterfaces,285286small-screeninterfaces,339342tastein
terfaces,302304VUIs,217220Textinput,insmall-screeninterfaces,316318Textureperceptio
n,52Thermoreceptors,30Thermostats,133rdTechHiBall3100trackers,117
Index460

Three-DOFjoysticks,3839Three-DOFplanarpantographs,3638Thresholdvalues,inolfactoryi
nterfaces,283284Throwawayprototypes,378Tiltsensors,119Timbre,sound,152Timeouts,in
IVRinterfaces,262Timescollection,inIVRinterfaces,264Timingtests,ingestureinterfa
cedevelopment,102TLX.SeeTaskLoadIndexTognazzini,B.,349Tonesignals,260Top-down,ge
stureinterfacedevelopment,99Top-downknowledge,forconversations,198Torsotactilein
terfaces,4546Torustreadmill,126Touch.SeeHapticinterfacesTouch-tonephonesystems,21
3214TowersofHanoigame,180182Trackers,inlocomotioninterfaces,117121,141Trackingcompu
tervisioninterfaces,7879Tradeoffs,intrackingtechnologies,119120TrainingFoodSimulat
orproject,305locomotioninterfaces,134speechrecognitionsystems,212Transfer,ingest
ureinterfaces,97Transferringdata,inIVRinterfaces,247Transparentguidance,ofuserin
put,422Transparentinterpretations,428429Travel,inlocomotioninterfaces,110Treadmil
ls,motorized,123126Treadportsystem,109,124Trekkersystem,166Tsagarakis,N.G.,83TTY/
TDDmachines,233TuningVUIs,2192202Dtreadmills,125126TVcontrolgesture,93TYCOONtaxono
my,97Ultramobileinterfaces,411413Umamitaste,291Uncertainty,inmultimodalinterfaces
,426429Unimodalinputversusmultimodal,419Unimodalinterpretations,404Unnaturalinter
faces,forlocomotion,127128Updatingdata,inmultimodeinterfaces,385Usabilityandusabi
litytestingdesignfor,1516gestureinterfaces,8890IVRinterfaces,243small-screeninterf
aces,339,342VUIs,219Userattentionfactor,inauditoryinterfaces,171172User-centeredd
esign(UCD)processsmall-screeninterfaces,343speechprojects,215216Usercharacteristi
csmultimodalinterfaces,430small-screeninterfaces,323Userconsiderationsandfactors
auditoryinterfaces,167168indesign,1315IVRinterfaces,248249locomotioninterfacesafety
,138multimodeinterfaces,381,421olfactoryinterfaces,287small-screeninterfaces,346
Vacuumfluorescentdisplays(VFDs),312Vaporization,insmellgeneration,269Variability
sources,inmultimodalinterfaces,421422Vection,inlocomotioninterfaces,112Vehiclesty
le,inlocomotioninterfaces,110Verticalscrolling,insmall-screeninterfaces,346347Ves
tibularsystem,112114Vests,tactile,50ViaVoice,353VibeStudioenvironment,173Vibrotac
tileinterfaces,4445ViconMXtrackersystems,119Viewingdistances,insmall-screeninterf
aces,325Virtualenvironment(VE)systems,117118VirtualReality(VR)AircraftMaintenance
Trainingsystem,397Virtual-realitysystems,inlocomotioninterfaces,124Virtualsurgic
alknives,8384Virtualwalls,2728VirtuSphere,125126Visemes,398,404Visicastproject,414
Index461

Visioncomputergamesfor,165computerinterfaces,7879eyecharacteristics,308inFoodSimu
latorproject,296hapticinterfacesfor,4750multimodalinterfacesfor,413414Visualaspect
s,insmall-screeninterfaces,309311,324325Visualcues,inlocomotioninterfaces,112113,13
8Visualoverload,auditoryinterfacesfor,162Vocabulary,ingestureinterfaces,98102Voic
eExtensibleMarkupLanguage(VXML),231,387Voicemail,229,232,236VoiceoverInternetPro
tocol(VoIP),231Voiceportals,364Voicerecognitionsystems,84Voicetalent,forIVRinter
faces,240241,248,262263Voiceuserinterfaces(VUIs),197198applications,216217automatedc
onversations.SeeAutomatedconversationscasestudies,224designguidelines,220224desig
ners,215futuretrends,224226humanfactorsdesign,214217phonesystems,213214recognitionp
rocess,209211speechrecognitionhistoricalnotes,212213technology,208209testing,217220u
ser-centereddesign,215216VoiceXMLsystem,387Voicing,forsonification,148Volatility,
ofodorintensity,283VTrackersystem,401VUIdesigners(VUIDs),215VUIs.SeeVoiceuserint
erfacesWalk-up-and-useIVRinterfaces,236,247Walker,B.N.,166,179Walking,inlocomoti
oninterfaces,109110,124127,141Walking-in-place(WIP)interfaces,125126,129Walls,inhap
ticenvironments,2728Wang,H.O.,21Wavetablesynthesis,159Wayfinding,110Wearableolfac
torydisplays,278279Wearabletactileinterfaces,4546Weatherradar,366367Webportals,364W
eber-Fechnerslaw,283Weissman,C.D.,93Wells,J.D.,319Wickens,C.,243Wiisystem,56,142143
,353Wiker,S.,33Williges,R.,336Winberg,F.,180182Window-Eyesscreenreader,165Windows
,insmall-screeninterfaces,337WindowsMobileSmartphones,342WIPsystem,406Wirelessph
ones,245Within-menuplacement,insmall-screeninterfaces,331WizardofOz(WOZ)testinga
ndexperimentsauditoryinterfaces,178gestureinterfaces,99,101multimodalinterfaces,

423424VUIs,218Wizardsmultimodalinterfaces,425small-screeninterfaces,350Wood,J.,48
Wordprocessing,gestureinterfacesfor,90Wordspacing,insmall-screeninterfaces,327Wo
rdingconventions,inIVRinterfaces,253Worlds-in-Miniaturetechnique,127128WorldVizPr
ecisionPositionTracker,119Wrists,ingestureinterfaces,9193XML.SeeExtensibleMarkupL
anguageXTRAmultimodalsystem,395Yanagida,Y.,279Yu,W.,48Zero(0)key,inIVRinterfaces
,256257Zoom,insmall-screeninterfaces,338
Index462