Klia I Ka Nuu:

Evaluating Usability of a Financial Aid

and Scholarship Resource Website

Terri Saragosa
University of Hawaii at Mnoa
Learning Design & Technology
tks@hawaii.edu

The Problem?

ESTIMATED cost to attend

UH Mnoa for 4 years

to earn an Undergraduate Degree?

A.
B.
C.
D.

$78,400
$10,584
$115,776
$28,944

(2015-2016 resident tuition, fees, books, meals/housing, personal expenses)

http://www.hawaii.edu/fas/downloads/2015-2016/2015-2016%20Cost%20of%20Attendance.pdf

ESTIMATED cost to attend

UH Mnoa for 4 years

to earn an Undergraduate Degree?

C. $115,776

FINANCIAL AID INCLUDES WHICH

OF THE FOLLOWING:
SCHOLARSHIPS

WORK
STUDY

LOANS
Grants

FINANCIAL AID INCLUDES WHICH

OF THE FOLLOWING:
SCHOLARSHIPS

WORK
STUDY

LOANS
Grants

DID YOU RECEIVE ANY TYPE OF

FINANCIAL AID IN COLLEGE?

A. YES
B. NO
C. UNSURE

DID YOU RECEIVE ANY TYPE OF

FINANCIAL AID IN COLLEGE?

of college students
receive some form
of financial aid
SOURCE: U.S. Department of Education, National Center for Education Statistics. (2014). The Condition of Education
2014 (NCES 2014-083), Grants and Loan Aid to Undergraduate Students.

Purpose:
To develop and evaluate the ease of use
and effectiveness of a website designed to
inform Native Hawaiian College Students
about Financial Aid and Scholarship Basics
and Resources.
www.finaidresources.weebly.com

Website
Design

Instructional Design
ADDIE
Mayers Principles of
Multimedia Design
Dont Make Me Think!

www.finaidresources.weebly.com

Target Audience:
Adult College
Students
Native Hawaiian
Plan to apply for
financial aid

Olelo Noeau

Nature Theme

Tech Tools

Content

Infographics

Segmenting

Videos

Annotated Resources

Scholarships

Basic Eligibility

Click-through Links

Resources

Short overview of site

Links

Multiple Modes of Learning:

watch read printout

Interactivity:

Slideshow

Self-Assessment

Discussion

Usability Testing

What?
Watching people try to use what
youve designed with the intention of:
a.
b.

Making it easier to use and

Proving that it is easy to use.

-Steve Krug

Why?
If you want a great site, youve got to
test. After youve worked on a site for
even a few weeks, you cant see it freshly
anymore. You know too much. The only
way to find out if it really works is to test it.

-Steve Krug

Methods

Instruments

I think this
looks like...

We are
evaluating
the SITE, not
you

I think I would
click on this
tab

There are no
right or wrong
answers

Usability Protocol

Post-Study Survey

Two Rounds

Results

Participant Demographics

ALL Native Hawaiian College Students

Planning to apply for financial aid

5 female, 3 male

Computer Skill Level

Poor (1) to Excellent (6)

Technology
Use

6
5
4
3
2
1
0
3

Level of Comfort Using the

Internet
Very Uncomfortable (1) to Very Comfortable(6 )

6
5
4

3
2
1
0
3

ALL use internet

daily

How do you usually access

the internet?

Technology
Use

5
4
3
2
1
Desktop/Laptop

Mobile Device (tablet,

ipad, smartphone)

What do you do on the internet?

8
7

6
5
4
3
2
School

Email

Social Media

Shopping

News

Entertainment

Information

QUANTITATIVE RESULTS
Task Success Rate
User Satisfaction Survey

6-point Likert Scale

Design Layout
Navigation
Ease of Use
Effectiveness

Round 1 : Task Success Rate

Percent

100
75
50

25
0
1-NAVIGATE
HOME PAGE

2-LOCATE TYPES
OF FIN AID

3-LOCATE HOW
TO APPLY FOR
AID

4-LOCATE BASIC
TERMS

5-LOCATE
SCHOLARSHIPS

OVERALL EXPERIENCE
6.00

MEAN

5.50

6.00
5.75

5.75

LAYOUT

NAVIGATION

5.75

5.00
4.50
4.00
EASE OF USE

EFFECTIVENESS

6 point Likert Scale: Strongly Disagree to Strongly Agree

QUALITATIVE FEEDBACK
Comments during test
Survey feedback
Post-test interview

Helpful Feedback

Some of these links don't work

How do I stop the slideshow?
Make eligibility BOLD!
Make handouts bigger, so I dont
need to scroll so much

When fixing problems, try

to do the least you can do.
Tweak, dont redesign.
-Steve Krug

Issues and revisions from Round 1:

Navigation/Ease of Use
Broken Links
Layout/Design Modifications
Tab
Slide Show
Infographic
Simplify

PROBLEM: Broken Links

BEFORE

AFTER

REVISION: Updated Links

Problem: Layout/Design
BEFORE

Simplify Tab:
Change FIN AID BASICS to FINANCIAL AID INFO

AFTER

PROBLEM: Layout/Design

BEFORE

REVISION: Change slideshow

to gallery

AFTER

Click on slide to enlarge for

easy reading

PROBLEM: Layout/Design REVISION: Update Infographic

BEFORE

AFTER

Problem: Layout/Design
BEFORE

SIMPLIFY and CLARIFY

Add explanation

Problem: Layout/Design
BOLD text to make
ELIGIBILITY stand out

Show more of Info

Sheets

FINDINGS

Task Success Rates

100

Percent

75
50

25
0
Navigate Home
Page

Locate Types of
Fin Aid

Locate How to
Apply for Aid
Round 1

Locate Basic
Terms

Locate
Scholarships

Round 2

Round 2: ALL tasks completed by ALL

OVERALL EXPERIENCE
ROUND 1 MEAN
6.00
5.50
5.00
4.50

6.00
5.75

5.75

ROUND 2 MEAN
6.00

6.00

5.75

6.00
5.75

DESIGN LAYOUT

4.00

LAYOUT
Very
confusing Very clear

NAVIGATION
Very
difficultVery
easy

EASE OF USE
Very
difficult Very
easy

EFFECTIVENESS OF
WEBSITE
Not at all
effectiveVery effective

All categories stayed the same or increased

Positive Feedback
Clean and neat
Easy to navigate
Basic, straight to the
point
Very easy to understand
Good information
I like the pictures and
quotes
Tabs are clear
I wish I had this!
Im going to share this
with my friends.

Why Usability?
Allowed testing of site on target users
Well worth the time and effort
Helped identify problems

Mahalo!
www.finaidresources.weebly.com

Questions?

References:
Krug, S. (2006). Dont Make Me Think! A Common Sense Approach to Web Usability.
Berkeley,
CA: New Riders.
Krug, S. (2009). Rocket surgery made easy: The do-it-yourself guide to finding and
fixing usability problems. Berkeley, CA: New Riders.
Makuakane-Drechsel, T., & Hagedorn, L. S. (2000). Correlates of retention among
Asian Pacific Americans in community colleges: The case for Hawaiian
students. Community College Journal of Research & Practice, 24(8), 639655.
Mayer, R. E. (2005). Cognitive theory of multimedia learning. In The Cambridge
handbook of multimedia learning (pp. 3148). Cambridge: Cambridge
University Press.