Usability analysis and evaluation of mobile

ICT systems
Asbjørn Følstad, Petter Bae Brandtzæg and Jan Heim
SINTEF Telecom and Informatics, PO Box 124, N-0314 Oslo, Norway
asbjorn.folstad@informatics.sintef.no

Abstract
There is a need for new usability methods tailored to suit the development of mobile ICT.
In response to this, three methods that have been developed to a prototype level are
presented. The methods are designed to be cost-effective and provide information on (1)
user problems as experienced in users’ everyday lives, (2) contexts of use, and (3)
integration of technology in users’ patterns of activity. The methods have been tried out in
a small-scale evaluation in conjunction with the introduction of PDAs and mobile
telephones with PDAs in an organisation.

Key words: Usability, method development, mobile, ICT, PDA

1.Introduction
A major challenge for future usability research is to develop new methods for usability
analysis and evaluation of ICT (Information Communication Technology) systems
[Abowd & Mynatt (2000); Monk (2000)]. The users of ICT systems are no longer only
business users, but belong to the general population. And the technology is becoming
increasingly mobile.

Existing usability methods and tools are suitable for testing stationary in-office ICT, but
do not cover the necessary span of mobile ICT use. Opposed to traditional desktop
computation, mobile ICT involves issues like use in multiple contexts, multiple users,
and continuous everyday use [Abowd & Mynatt (2000)]. Traditional usability work
typically concerns single users using one single technology to solve well-defined tasks in
an undisturbed environment. Solving tasks in this manner requires different strategies,
other sources of support, and is accompanied by problems other than the solving of tasks
in complex and shifting environments [Zsambok (1997)]. The traditional usability
framework needs to be supplemented by methods tailored to gain knowledge of user
problems in the ‘real world’, actual user contexts, and users’ integration of mobile ICT in
their everyday lives.

Mobile computing systems typically have short product realisation cycles. Consequently,
cost and time efficient methods and tools for usability analysis and evaluation are
required (Millen, 2000). What is needed are ‘quick and clean’ (as opposed to ‘quick and
dirty’) methods of usability analysis and evaluation [Wichansky (2000)]. One direction
with potential that has been identified is that of training users to report their own critical
incidents [Castillo, Hartson, & Hix (1998); Palen (2000)].

In response to these challenges, some early work on the development of methods

particularly tailored for mobile usability analysis and evaluation has been initiated. The
focus of attention has been on users performing real tasks in their normal environment,
and the methods include:
• Angry Face. User-initiated usability evaluation by way of an Angry Face button.
• User Context Logging. Selected users’ reporting their contexts of use ‘on the fly’.
• Patterns of activity. Monitoring the integration of the technology and user
satisfaction through interviews and questionnaires.

The three methods was used in a small-scale investigation, conducted in conjunction with
the introduction of PDAs (Palm IIIc) and mobile phones with PDAs (Nokia 9110
Communicator) in a department of a Norwegian research organisation.

2.Method development
The development and implementation of the methods will be presented in regard to:
• Generic description. Presentation of the method on a general level.
• Purpose. The kind of data that may be collected, and why these data are important.
• Ideal implementation. How the method ultimately may be implemented, given future
time and resources.
• Actual implementation. Implementation of the method at this stage of development.
• Validation. How the method has been-, or may be validated.

2.1Angry Face user feedback

In reaction to the tedium and occasional impossibility of bringing users to the lab, remote
user initiated evaluation is a tempting option. In Angry Face users are sensitised to
usability problems through short initial training. They are then equipped with the
technology to be evaluated, and urged to use it in their everyday life for a period of time.
The technology has an Angry Face button included in its interface, and the users are
instructed to press Angry Face whenever they experience user problems. Pressing Angry
Face causes automatic capture of recent on-screen action. The user is also given the
opportunity to make ‘on the fly’ comments on her experienced difficulty.

The purpose of Angry Face is to collect data on user problems as they appear in real life.
The method is designed to be low-cost, and should be easy to implement in a
development process providing that the product is sufficiently operative.

The ideal implementation of ‘Angry Face’ would be an on-screen button shaped as an

angry Smiley (see Figure 1). Pushing the button should save the last few screen-changes
to allow analysis of what has gone wrong in the interaction. The user should be allowed
to give input regarding the experienced problem in a voice-mail fashion.
 Figure 1: Example of a possible Angry Face interface implementation

The prototype of Angry Face used in the investigation was far more mundane than the
ideal implementation. The PDAs were equipped with a screen-dump program, allowing
users to capture screen images associated with user problems by pushing a single button.
Comments were entered in the notebook function of the PDA.

Angry Face was validated through semi-structured interviews with the participants,
following their period as testers. Future validation may include the comparing of results
from Angry Face with results from more structured testing.

2.2User Context Logging

Context of use is an important designator of the suitability of a user interface. The
following list includes aspects of context that are commonly taken into account:
• Activity – what I am doing
• Time – when I am doing it
• Location – where I am doing it
• Social situation – ... and with whom

Among the above contextual variables, only one (‘time’) is easily logged automatically.
Location, activity, and social situation may only partially be automatically logged, and
this with an overwhelming amount of technical fuzz1.

The method of User Context Logging involves users giving ‘on the fly’ descriptions of
their contexts of use. The technology in question is equipped with a device for recording
users’ context descriptions and given to a predefined set of users. The users are urged to
report the context of every incident of use as they occur, for a given period of time.

An ideal implementation of the data collection tool for User Context Logging would be a
recording device activated by the onset of interaction with the technology, allowing the
user to register the necessary information in the simplest possible way. More down to
earth, the prototype data collection tool consisted of context templates stored in the
notebook function of the participants PDAs. The templates included slots for recording
1
 Meaningful location terms (e.g. ‘at the office’, ‘on the bus’ etc) may to a certain degree be inferred from 
geographical positioning data. Activities may be partially inferred from what functionality is used in the 
mobile device. Social situation may be waverly induced from the combination of time and location.
time, task, location and social situation, and the participants entered their contextual
information by using the stylus. The collected data were validated through semi-
structured interviews.

2.3Patterns of Activity
Mobile technology is often introduced to support users in activities they are already
engaged in. As an example, a PDA may be introduced to help the user updating her
calendar; a task she may be used to do with the help of a Time Manager or PC software.
A beneficial introduction of the PDA requires that she changes her pattern of use in
regard to aids that have previously supported the activity, and that the new technology is
integrated in her pattern of activity.

Patterns of Activity involves the collection of data on how mobile technology is

integrated in the users’ activities, and how satisfactory the user perceives this integration
to be. Data are collected when the users have reached a level of stable use. The primary
data collection tool is thought to be questionnaires, supplemented with interviews.

The questionnaires are divided in parts corresponding to the activities that the mobile
technology is likely to support. The activities are identified through an initial task
analysis. For each identified activity, the respondent is asked to indicate
• How often the technology is used in conjunction with the activity
• How often other identified aids are used in conjunction with the activity, as compared
to before the new technology was introduced
• Satisfaction with the present way of the activity

The questionnaire includes a measure of general user satisfaction, providing ipsative data
on the perceived Usefulness, Control, Ease of Use, and Affect. The supplementing
interviews are conducted with a subset of the participants and should capture user activity
patterns not addressed in the questionnaires.

Data collected through Patterns of Activity may generate useful knowledge for deciding
on future developments of the technology, as well as for deciding the focus of attention of
further usability testing.

For the present investigation of PDAs, the tasks identified through the initial task
analyses included viewing and updating of calendar, contact lists, and task lists, in
addition to reading e-mail. The satisfaction measure consisted of 12 items corresponding
to the four factors mentioned above. The items were presented in four groups of three in a
way that each factor was represented with one item in three of the groups, and no two
groups had the same combination of factors. The participants were asked to rank the
items of each group in regard to how well they described the participants’ relation to the
technology. Factor scores were calculated as the sum of the rankings given to each of its
items.

Patterns of Activity may be validated through comparisons with related measurements

e.g. SUMI [Porteous, Kirakowsky, & Corbett (1993)] for the satisfaction measure.
3.Method utilisation and results
The three prototype methods were used in a small-scale usability investigation, primarily
motivated by the desire to try out the methods. However, the results also gave insight on
the use of PDAs, and their introduction in an organisation. The data collected in the
investigation are reported under the headings of the three methodical approaches.

3.1Angry Face
Angry Face was conducted with four novice users of Palm IIIc, over a period of two
weeks. The users totally reported 21 instances of usability problems. A sample of the
reported problems is given in Table 1.

• Cumbersome to give written input, in particular the Norwegian letters ‘æ’, ‘ø’, and ‘å’
• Cumbersome and difficult to register an event in the calendar for two consecutive days
• Difficult to understand how to categorise to-do lists
• Difficult to understand how to turn off alert function in calendar
• Lack date (in addition to time) on ‘Home’ screen
• Cumbersome to delete notes the hard way (through ‘details’), and difficult to understand how to do it
the easy way (through the menu or using graffiti)
• Dysfunctional when used as shopping-list.
Table 1: A sample of reported usability problems

Five of the reported problems were overlapping, and indicates that these may be
experienced by a majority of users. Nine of the reported problems were accompanied by
a relevant screen image, stored through the screen-dump function. Figure 2 shows the
screen-dump recorded by one user, indicating where she expected to find the function for
registering an event in the calendar for two consecutive days.

Figure 2: Screen-dump made by frustrated user

To get an indication of what categories of problems the users perceived to be most

prevalent, the reported usability problems were made subject of a post hoc categorisation.
It turned out that more than half the reported problems (11) were related to the
technology being cumbersome to use. This may indicate that usability-wise, the greatest
impediment of Palm IIIc is not that its functions are hard to understand, but that they are
slow to use. The distribution of usability problems is presented in Table 2 below.
 Category Reported instances
Cumbersome to use function 11
Hard to understand function 4
Particular function is missing 2
Does not support particular task 2
Lack of control 1
Not compatible with function in PC 1
software
Table 2: Distribution of reported usability problems across categories.

The real nature of the reported problems was analysed through subsequent interviews,
and by having an expert user interpret the reported incidents. Through this process,
incidents could be re-categorised. An example of this was that a function reported as
missing was only hard to understand.

In the interviews the participants reported that they were able to use Angry Face to
indicate usability problems they would otherwise forget. Even so, they claimed to be
hampered from giving a complete review of their user problems when using the current
version of Angry Face. Partly because its rudimentary implementation made the input
process cumbersome.

3.2User context logging

User Context Logging was conducted with the four users participating in Angry Face.
The participants reported their contexts of use for two 24-hour periods each. The reported
contexts, and their distribution across general categories, are summarised in Table 3.

Task Total Work Mobile Home Alone Meeting

Check calendar 8 2 2 4 7 1
Update calendar 4 1 3 4
Check tasks 3 2 1 3
Update tasks 4 1 2 1 4
Check contacts 1 1 1
Read e-mail 1 1 1
Make note 2 1 1 2
Fiddle with PDA 1 1 1
Total 24 6 7 11 17 7
Table 3: Results of User Context Logging

As for tasks, the data did not provide much in the line of surprises: Checking calendar is
the most frequently registered activity on the PDA. What is interesting however, is that
reported use in home and mobile contexts by far outnumbers use in a work setting. Also it
may seem that solitaire use is far more frequent than use in the company of others.

In subsequent interviews, the participants claimed that their assignment in User Context
Logging was easy to carry through. It was held to be easy to remember reporting the
instances of use, in particular since they were urged to glue a yellow post-it reminder on
the cover of the PDA. The participants also reported that the logging procedure did not
interfere with the task at hand.
3.3Patterns of activity
Patterns of activity were conducted with 13 users. Three of the participants were provided
Nokia 9110 while the 10 others got Palm IIIc. Each of the participants responded to a
questionnaire distributed 12 weeks upon receiving the mobile terminals. The participants
had also responded to a similar questionnaire distributed on the day of receiving the
terminals. Four of the respondents participated in interviews.

Regarding activities related to viewing and updating calendar, contact list and task list,
the users reported an increased use of these applications on their PC (MS Outlook), and a
decreased use of all other aids (like filofax, paper notebook etc.). This indicates a mutual
reinforcement between the use of PDAs and corresponding PC applications. Further, the
PDAs were more closely integrated in activities regarding the viewing and updating of
calendar, and less so in other activities. The participants, when asked before the
introduction of the technology, were better at predicting what effect they would get from
the calendar function of the PDAs than they were at predicting the effect of the other
functions. As for satisfaction, Palm IIIc users generally scored high on perceived usability
and low on affect, while users of Nokia 9110 scored high on affect.

Quite a few respondents reported that they were unable to express their individual use of
the technology through the questionnaire, and the interpretation of data generated through
the questionnaires relied heavily on the supplementary interviews.

4.Discussion

4.1Experiences and lessons learnt

The three methods are being developed as a response to demands from the field of mobile
ICT. It must be discussed whether the methods presented may help us to meet these
methodological demands.

Angry Face and User Context Logging seem to be interesting methodological concepts.
They both enable the gathering of valuable information from real users in the world with
a minimum of fuzz and resources, and the information collected could not easily be
gathered through other means. When this is said, both methods have their limitations.
Both methods require that the technology to be evaluated is sufficiently developed to be
tried out on naive users without the benefit of supervision. Contextual User Logging also
has the problem that it may bias the participant, since the method may impose an
exaggerated consciousness regarding the use of the technology. Angry Face on the other
hand may suffer from the possible misconception that it is a regular usability evaluation,
only with the laboratory and mentor removed. This is not so. Data from Angry Face
address usability problems as experienced by the users in the real world, and should not
be expected to have the same focus or detail as data from the usability lab.

The success of Patterns of Activity relied partly on the authors’ ability to generate a
standardised form for collecting the necessary data associated with each participant’s
individual pattern of activity. This proved to be no easy task. The questionnaire generated
interesting data, but the interpretation of these required extensive analyses and
supplementary semi-structured interviews. A possible lesson to be learnt from this is that
since the use of mobile ICT in real life is complex partly unpredictable, structured
questionnaires alone may not be the optimal way to collect data on this.

4.2The case of introducing PDAs

The investigation of the introduction of PDAs in an organisation provided some
interesting insights. Among these the mutually stimulating effect between PDAs and the
corresponding desktop application was particularly interesting. It seems like the
opportunity to use for example task lists in a mobile situation, inspires users to begin the
use of the same lists while in the office as well.

4.3Future challenges
Future development of methods for analysis and evaluation of mobile ICT involves
several challenges. The methods presented in this paper may be interesting to pursue, but
require further efforts in the line of validation. Furthermore, nothing has yet been said
about the number of users required to get reliable results, or how interesting user groups
should be identified. A final point that needs to be addressed is that the development of
‘quick and clean’ methods alone is not enough to ensure the usability of future mobile
ICT. Future work should also focus on a tight integration of these usability methods in the
product development cycle.

References
Abowd, G. D., & Mynatt, E. D. (2000). Charting Past, Present, and Future Research in
Ubiquitous Computing. ACM Transactions on Computer-Human Interaction,
7(1), 29-58.
Castillo, J. C., Hartson, H. R., & Hix, D. (1998, 18-23 April). Remote usability
evaluation: Can users report their own critical incedents? Paper presented at the
CHI, Conference on Human Factors in Computing Systems.
Millen, D. R. (2000, 17-19 August). Rapid Ethnography: Time Deepening Strategies for
HCI Field Research. Paper presented at the The International DIS 2000
Conference on Designing Interactive Systems, New York.
Monk, A. (2000). User-Centred Design: The Home Use Challenge. In F. v. R. A. Sloane
(Ed.), Home informatics and telematics: Information technology and society (pp.
181-190). Boston: Kluwer Academic Press.
Palen, L., Salzman, M., Youngs, E. (2000, December 2 - 6). Going Wireless: Behavior &
Practice of New Mobile Phone Users. Paper presented at the CSCW '2000, ACM
Conference on Computer Supported Cooperative Work, Philadelphia,
Pennsylvania, USA.
Porteous, M., Kirakowsky, J., & Corbett, M. (1993). SUMI user handbook. Cork: Human
Factors Research Group, University College Cork.
Wichansky, A. M. (2000). Usability testing in 2000 and beyond. Ergonomics, 43(7), 998-
1006.
Zsambok, C. E. (1997). Naturalistic Decision Making: Where Are We Now? In C. E.
Zsambok, Klein, G. (Ed.), Naturalistic Decision Making (pp. 3-16). Mahwah, NJ:
Lawrence Erlbaum Associates.