TMT4053 Multimodal Interaction Technology

Unit 11:
Multimodal Interfaces
Nature of the Interface
 Technology solutions have become more ubiquitous, flexible,
and capable, it is reasonable to ask if the user interface (UI) that
allows the user to take advantage of the technology has kept
pace.
 The following elements that define an MMUI:
 Allows users to interact with server-side services and information
using different interaction/UI styles.
 Allows user to achieve a sequence of interrelated tasks using
multiple devices.
 Presents features and information that behave the same across
platforms
 even though each platform/device has its specific look and feel.
 Feels like a variation of a single interface for different devices with
the same capabilities.
 Typically, an MMUI is implemented using visual and/or auditory
UIs.
Technology of the Interface

 Consideration when designing MMUIs

 combination of graphical, web, touch-tone, speech recognition, and small-screen
visual UIs
 Used of other modalities, such as tactile, olfactory, and gustatory
 Drawback: lack of powerful input/output capabilities that the visual and
auditory technologies have.
 MMUIs allow multiple access methods. Major concerns in MMUIs scenarios
follow:
 Ensure accurate presentation of information to the user
 Prevent user confusion
 Establish the precedence given to changes made
 Prevent data corruption or lost work
 Allow the user to quickly switch between UIs
Fig. 1. Shows how an application that uses multiple UIs implemented
Fig. 2. Shows how an application that uses multiple UIs implemented
Fig. 3. Shows how an application that uses multiple UIs implemented
Technology of the Interface (cont.)
 Figures 1-3: Examples of an MMUI.
 Interrelationship among users, contexts of use, multiple UIs and the underlying
infrastructure for an example MMUI
 Factors in uniqueness of designing an MMUI
 User’s consideration and different contexts of use and tasks.
 May need to interact with multiple devices as well as multiple UIs.

 These devices interact with various software applications to provide service

functionality.
 Depending on the implementation, the same application may support only one
device/UI, but it could support more than one.
 If there are multiple applications, their ability to communicate with each other is a
factor in the overall level of integration that the user experiences.
 Finally, applications network with back-end systems to carry out user requests,
i.e. databases
 Some implementations may use multiple back-end systems, and their ability to
synchronize data also will have an impact on the user.
CURRENT IMPLEMENTATIONS OF THE
INTERFACE
i. Information/Web Portals

 Specially designed web page that brings information together from

diverse sources in a uniform way, i.e. Google, Yahoo!, Bing

 Each information source gets its dedicated area on the page for
displaying information
 ii. Automated Self-Help
 Interactive Voice Recognition (IVR)
 One of the early automated self-help applications
 Integrated into the company’s website
 Voice User Interface

 Big companies established business with little or no direct human

interaction
 Automated systems to provide support for tasks (i.e. billing, payments,
technical support, account inquiries and sales)
 Examples: Amazon, Lazada, internet banking – CIMBClicks, Maybank2u
iii. Address Books
 The diverse tasks that benefit from having an address book
have driven the need to have a single, synchronized address
book that can be accessed from almost any end point

 MMUI allows one synchronized address book accessible from

any mobile devices, computers, phones or even TV
 Sync.me (Android, iOS)  Contacts+ (Android)

 pulls contact information from your Google+,  an address book replacement that
helps organize your contacts and
Facebook, LinkedIn etc. accounts to keep
integrates with your social media
your contacts automatically updated with the
accounts and communications.
latest profile photos and other related

contact details.
 iv. Message Mailboxes
 Messages allow people to keep in contact even when they are
not available/ unreachable.

 It needs to be ubiquity (omnipresence)

 It has different origins for each type of message

 Create the use of MMUIs to better handle messages better and

expand its usefulness and functionality
O Access Email on your O iOS & android support synchronization of SMS

smartphone through with email using Microsoft Exchange

email apps O When Exchange account is added using

ActiveSync on an Android phone, some
phones will begin forwarding a copy of every
SMS message that the phone receives to user
email account.
 v. Remote Access to Services

 Problems:

 Accessibility to company’s network & files – i.e. business trips

 Need mobility to do their works – i.e. working remotely

 The users has the ability to get access to services from a remote
distance unattached to a wired device

 Automation - the method, or system of operating or controlling a

process by highly automatic means, as by electronic devices,
reducing human intervention to a minimum.
 (a) (b)

Locating a place (i.e. UNIMAS)

a) Using a PC with an internet connection. The UNIMAS’s map on the website has numerous
options and has large display.
b) Using a mobile wireless device. The UNIMAS’s map for mobile devices has few options
and has small display. (Credit: Google Maps)
Applications of the Interface to
Universal Design and Accessibility

 The impact of MMUIs on people with disabilities

 Design consideration covered the full range of user capabilities and

limitations = Universal Design

 MMUIs helping to achieve a more universal design. Examples:-

 The use of touch-tone commands in conjunction with a speech-

enabled UI.

 The use of speech recognition helps those with a cognitive disability –

people have trouble remembering the mapping of touch-tone options
played in a menu.

 Touch-tone input – helps a user with a speech impairment

 Summary of Design Steps for MMUIs
1. Define the user population, including people with disabilities.
2. Determine capabilities and limitations of users.
3. Collect user needs, including specific contexts of use and
environments.
4. Match user needs with system functionality.
5. Perform task analysis.
6. Evaluate the need for more than one UI.
7. Select the specific interface modality(s) based on strengths and
weaknesses of each.
8. Assign tasks/functionality to all modalities or only some of them.
9. Concurrently perform the detailed design of the UIs.
10. Test with representative users, collecting data about
usability/accessibility for tasks and modalities.
11. Iterate design and retest as needed.
Future Trends

 MMUIs have emerged during the last decade as a major worldwide

trend for of supplying the information

 One of the key drivers of flexibility will be the increased focus on

universal design and accessibility.

 This fundamental change for almost all applications for the user
population will make accessibility issues part of the mainstream design
process.

 From a technical standpoint, the growth in demand for applications

with more than one UI modality needs to be—and is being—supported
by a corresponding improvement in technology to enable MMUIs.
Future Trends (cont.)

 The other major technological change that is occurring to

increase the development of MMUIs is in computing technology
itself.

 MMUIs will continue to be almost exclusively combinations of

visual and auditory interfaces.

 As technology becomes more ubiquitous and the tools to build

these interfaces become more widely available, there will likely
be an increase in competition (rapidly growing).
Future Trends (cont.)

 In conclusion, multimodal interfaces are just beginning to

model human-like sensory perception and communication
patterns.

 They are recognising and identifying actions (Gesture &

Locomotive UI), language and people that have been seen,
heard (VUI) or in other ways experienced in the past.

 They literally reflect and acknowledge the existence of human

users, empower them in new ways.

 They also can be playful and self-reflective interfaces that

suggest new forms of human identity as we interact face-to-
face with animated personas representing our own kind.
THANK YOU