+

Nokia Morph Concept (2008)

Melissa Densmore, UCT CS2002S, 2020

1
+

Mobile
Input

CS2002S: Mobile
Development and Design
Melissa Densmore
University of Cape Town

CS2002S, 2020 Melissa Densmore, UCT

2
 +
Impoverished or Extraordinary?
interaction on mobile devices

Small screens, hopeless keyboards

vs.

Challenging us towards the extraordinary

Melissa Densmore, UCT CS2002S, 2020

In this lecture I want to challenge you to think about how we approach mobile
design.

Is mobile interaction impoverished – meaning that we have a compromised

experience, or is it extraordinary? Mobile design challenges how we look at human-
computer interaction, and provides opportunities for amazing innovation.

3
 +
ISO keypad
• 12 keys 0-9, *, #
• 20 words per minute
• Predictive text (T9)
• 45 words per minute

Touch typing on full qwerty keyboard is

around 60 wpm.

Melissa Densmore, UCT CS2002S, 2020

Let’s roll back a bit and think about mobile input just 15 years ago, before touch
screens. Then, phones were optimized for calling phone numbers.
Think for a little bit – how do those optimizations embody some of the design
principles?
• The power off button is hard to reach (and press)
• There’s a big button, presumably for answering a call, and a smaller one for
hanging up.
• Make frequent things (like pressing number buttons easy) and less frequent things
(like letters, and power buttons) harder
• Phone size affords holding.

But then short message system - SMS became a thing, and suddenly people were
sending text messages to one another, using the keyboard not just for giving names
to their contacts, but also to communicate.

There were two main ways to support this. The first one built upon existing
conventions from phones - set up to facilitate the use of letters instead of numbers
for phone numbers to make them more memorable. For example, calling
0860JOBURG reaches Johannesburg’s Call Centre.
What design principle does that evoke?
Recognition vs recall. A phone word is typically easier to recall than a number.

4
So to type a particular character you could press a given button multiple times to
cycle to the letter of interest.

Another approach that became popular was T9, in which you didn’t press numbers
multiple times for each letter, but instead you just used each number key as a letter
key, and the phone would guess based on the combinations of numbers what word
you wanted to type. Surprisingly, with predictive text you could type at up to 45
words per minute. Interestingly – without ever looking at a keyboard. This is harder
now that feedback is typically visual rather than by touch.

By the way I owned this exact model of phone, or something very close to it, back in
1999. My next phone lost the external antenna but was much the same. Then they
got prettier and had stuff like flashlights and integrated radios.

4
 +
FasTap Keypad
6.3 words per minute

Melissa Densmore, UCT CS2002S, 2020

At the time and even still, there’s a ton of research looking into ways to make text
entry on phones easier.

Take for this example the FastTap keyboard which instead of overloading the number
buttons, added a second set of buttons at the intersections. You’d think that having
the keys all accessible would make it easier to type, but the average WPM with this
keyboard is slower.

Would this have been predictable using design principles? What does Fitz’s law say
about it? Or Hick’s law?

5
 +
Graffiti
For Palm and WebOS Devices
21-25 wpm

Melissa Densmore, UCT CS2002S, 2020

Another alternative was handwriting recognition. This is better supported in the

cloud space these days, using resources like Anoto or Apple Pencil, some apps have
handwriting recognition built in, with varying levels of accuracy.

At the time though, natural handwriting recognition was hard to do, but easier if you
prescribed simpler gestures that were maybe more easily recognizable by an offline
computing device. Gestures become extremely important for mobile design,
especially with touch screens.

You see also that the use of handwriting recognition frees up more screen real estate
and lowers hardware costs over having a full keyboard. It was brilliant.

However, these devices were almost always used hand-in-hand with PCs, designed to
sync and charge connected to your computer, and not exist independently.

I have a couple of these devices in my office, and a lot of my early research was
around Java applications written for Palm Pilots and Palm phones.

6
 + Text Entry in other
Languages
Our initial assumptions were trumped when
we observed that eight of the ten CNEs took
only a week of regular usage and practice to
“comfortably” enter names in freeform Hindi
text… instances of non-work related usage of
the CommCare phone were an indirect
motivation for the CNEs to learn the
challenging native text input…
Medhi et al, NordiCHI 2012

Melissa Densmore, UCT CS2002S, 2020

One might think that complicated keyboards can inhibit adoption or make phones
difficult to learn. But as a counterpoint to what I have been saying for this course, if
the application is sufficiently useful, or obligatory, we find that users will overcome
these difficulties. In this case, the researchers asked rural community health workers
to track their patient records using a phone, fearing that unfamiliarity with the device
might make the project fail. But because the phone was perceived to be useful the
health workers went out of their way to learn the native text input quickly, so the
project would succeed and they would be permitted to keep the phone.

Mobile experience is extraordinary – and humans are even more so.

7
 +

Alternative Keyboards
chording
laser projection
mini-keyboards

Melissa Densmore, UCT CS2002S, 2020

Twiddler

8
+

Pocketables
Comparison
UMPC mode = thumbs
Laptop mode = traditional

Melissa Densmore, UCT CS2002S, 2020

9
+

Keyboard Modes
Same user input (or system) produces distinct
results depending on system state
- classic example: [CAPS] or [NUM] lock
- mobile affords a more clear way
- need clues about current mode
background/header color
light indicators
toggle switches

Melissa Densmore, UCT CS2002S, 2020

10
 +

Smartphone Input
Fewer physical keys means greater
ambiguity
Software screens allow for easier innovation

virtual keyboards
handwriting recognition
voice recognition
different alphabets
predictive text

Melissa Densmore, UCT CS2002S, 2020

Keyboards and glossaries adapt according to language, personal vocabularies, letter

order in words.

These are some examples of touch-keyboards people have been working on.
Everything from handwriting recognition to keyboards designed specifically for

But caveats – how well does this work for multi-lingual users, or in locales with
different recognized names. Existing algorithms and dictionaries can arguably be
considered biased against anyone who types words differently and hasn’t had input
into design of these systems. I don’t know how many times I’ve had to undo
autocorrect of African names. Even just the names of local suburbs get butchered.
(pause for dramatic effect and empathy here, but unfortunately no virtual students to
pipe in with real-life examples, alas)

If text is a component of an app you are designing, think about how you can make
that easiest for your users – from reducing the typing necessary to using the right
keyboard options to make it work.

SwipeKey Yuan-Fu Shao MobileHCI 2016

11
Melissa Densmore, UCT CS2002S, 2020

12
 +
10B Speech Recognition

CS2002S, 2020 Melissa Densmore, UCT

13
 +
Speech Recognition

n Uses cloud service

n English recognition heavily biased

towards American and UK accents

n Low-resource languages neglected

n Privacy and social

restrictions

n Ambiguity

Melissa Densmore, UCT CS2002S, 2020

A simple human workaround for keyboards is just recording voice notes. This is
common with WA.

But I’ve always struggled with receiving voice messages on WA because I’m not
usually in a place where I can listen. And it takes me longer to listen to a long
message than to skim a long text message. Easier to record, hard to listen. This
highlights an instance in which an interaction must be well-designed, not just
between the computer and the human but across a wider use case.

An alternative approach may be to combine with speech-to-text, even bad speech-to-

text, I could quickly scan the message and decide whether to prioritize listening. But
is this something I want from WA if it will compromise the encryption of the
message? One also needs to weigh the tradeoffs between implementation difficulty,
and potentially conflicting user requirements.

There is some work that allows small vocab speech recognition within a device rather
than remotely, but it’s limited in application.

However importantly – for connected users, speech is a rapidly improving mechanism

for mobile input.

14
 +
Pervasive Speech Recognition

Melissa Densmore, UCT CS2002S, 2020

Speech recognition is now an indelible part of mobile devices and commonly used to
support input.

15
 +
Security Considerations

Melissa Densmore, UCT CS2002S, 2020

Speech recognition poses a number of cyber-security questions, especially with the

always-on home devices.
• Always on and always listening
• Knows more about what’s happening in your home than you do
• Exploitable – even by kids

• Some practical advice

• Don’t put the device visible from a window
• Keep it away from the TV, change the default keyword
• Whitelist only the features and services you intend to use.

Image source:
https://www.digitalinformationworld.com/2020/03/creepy-things-you-didn-t-know-
your-smart-home-tech-could-do.html

16
 +

Melissa Densmore, UCT CS2002S, 2020

Every week I get another email from Amazon giving me hints about how to use the
our (now decommissioned) echo devices. Why do you think this is?

Speaking from the perspective of visual affordances, everything one of these can do
is effectively a hidden affordance. Perhaps we learn that the device will respond to
us and make suggestions. This is a learned affordance, that speaking evokes some
sort of response. But how does one identify key words – pure guessing?

These suggestions are effectively a user manual to bootstrap exploration and

discovery.

But they also highlight the ways in which home speech devices have failed to achieve
the goal of natural user interaction.

17
+

Melissa Densmore, UCT CS2002S, 2020

18
 +
10C Touchscreen Input

CS2002S, 2020 Melissa Densmore, UCT

19
 +
Function input: Menus and Buttons
windows icons menus pointers (wimp)

Android iOS

Melissa Densmore, UCT CS2002S, 2020

The core of non-text input in HCI is really icons and menus – things you can click on
and point to, to evoke a response.
The vocabulary of windows, icons, menus and pointers is effectively a learned
language that we pick up as we use computing devices.
What are the affordances and mappings you see in these screenshots (taken from
Google Play Store and Apple App Store listings for Facebook)

• hamburger for additional menu options (which is not universally recognized btw)
• A series of icons invites exploration (luckily pressing on any is always undoable
without consequence)
• The picture then helps users map buttons to particular views.
• Note use of color or highlight to indicate state for this modal interface
• Use of screen space hopefully puts most important info at the most visible.

20
 +
Direct Manipulation

The criteria for direct manipulation are the continuous

representation of objects of interests and the fast, incremental,
undoable actions, which have an immediate visual impact on
the object itself. The goal is to allow the user to directly interact
with the object.

- Ben Shneiderman 1983

Melissa Densmore, UCT CS2002S, 2020

The key goal in design of interfaces is to support direct manipulation. This is even
more important for touchscreen.

Key points
• Fast and incremental allows for continuous representation. Like resizing a window.
Zooming in on a map. Scrolling a screen.
• Incremental affords learning – initiating an action that’s undoable allows people to
learn quickly what that action will achieve.
• The immediate visual impact allows people to immediately know the effects of
their action, making it seem like they are interacting with a real object, not just a
virtual representation
• Undoable actions allows people to go back and forth with the manipulation

• Something that happens to me recently is that when I’m on a Google Meets call,
and sometimes Zoom, my computer becomes unresponsive. Then, something I’ve
typed only appears a few seconds later, or a window opens in response to a click I
did a while ago and then closes again because I clicked twice thinking I missed the
click the first time.

21
 +
Timers

Melissa Densmore, UCT CS2002S, 2020

Consider various timer interfaces and whether they are easy to use.

This one on the left – the visual timer is a great example of direct manipulation.
Dragging and edge of the red bit around the circle is a direct manipulation of the
interface that also tells you exactly what you’re setting,

Most apps will have you set the hms separately either by typing in the numbers or
using various widgets to adjust the number up and down.

This is also a form of direct manipulation - and akin to how you might have set an
analog timer once upon a time. But it is clunky and requires more interactions. And
it misses Schneiderman’s point about direct interaction with the object, as well as
incremental interactions.

https://thebetterparent.com/2019/03/the-7-best-timer-apps-for-android/

22
+
Canonical Operations
multitouch à “stroking, not poking”

n Graphics: translate, rotate, and scale

n Text: cut, copy, paste

Melissa Densmore, UCT CS2002S, 2020

23
+

Melissa Densmore, UCT CS2002S, 2020

24
 +
10D Augmented Reality

CS2002S, 2020 Melissa Densmore, UCT

25
+
Infinite Screens, Halos, and
Peepholes, Augmented Reality

Melissa Densmore, UCT CS2002S, 2020

26
+
Sensors and Attachments

Heartrate Monitor Cellscope Stress Detection

camera camera • Microphone

• Temperature

Payments Pedometer Level

NFC/Bluetooth Gyroscope Gyroscope

Melissa Densmore, UCT CS2002S, 2020

27
+
Camera Input
not just for pictures

Melissa Densmore, UCT CS2002S, 2020

28
+
Proximity: Familiar Strangers

Melissa Densmore, UCT CS2002S, 2020

29
+
Natural User Interface (NUIs)
Interacting without having to think hard about how

n Kinect – but not Wii?

n Contrast with keyboard/mouse or indirect mappings

n Manipulating photos – e.g. rotate

n Lower learning curves

n Using sensors (pressure, etc) to interpret user requirements

with more depth

What is natural?

Melissa Densmore, UCT CS2002S, 2020

30
+
Shoogle
combining content with motion sensors and audio
feedback

Melissa Densmore, UCT CS2002S, 2020

31
+
Google Now
Minimizing the need for input by predicting
information requirements

Melissa Densmore, UCT CS2002S, 2020

32
+
On Mobile Inputs

n Small keyboards and display are constraining but not

limiting

n Our aim is direct manipulation or a natural user interface

n Just because you can build something doesn’t mean you

should!

Melissa Densmore, UCT CS2002S, 2020

33