USABILITY ASSESSMENT

TESTING REPORT FOR OIT

1
 Contents
Executive Summary ............................................................................................................................................................................3
Summaries of Data ..............................................................................................................................................................................4
Task Accuracy ..................................................................................................................................................................................4
Task Timings ....................................................................................................................................................................................5
Preference Data ...............................................................................................................................................................................6
Suggestions for improvement ...................................................................................................................................................7
Analysis of Data ....................................................................................................................................................................................9
Source of Error Analysis ..............................................................................................................................................................9
Findings and Recommendations................................................................................................................................................ 11
General Findings and Recommendations (global issues for OIT Help system)................................................ 11
Finding #1: Many students are unfamiliar with the OIT help page. ................................................................. 11
Finding #2: User experiences were highly variable based on search terms ................................................ 12
Finding #3: Most users didn’t want to have to go to a 3rd party site or submit a form, they just
wanted to perform the task from the OIT site. .......................................................................................................... 12
Finding #4: Successful task completion not clearly identified or universally understood by
moderators. Lots of variance in how far testers went before indicating task completion. .................... 12
Finding #5: Users seem to prefer using search tool rather than digging through links on the front
page. ............................................................................................................................................................................................. 12
Specific Findings and Recommendations ......................................................................................................................... 13
Finding #1: PSU email account expiration information not explicitly stated on OIT help topic for
ODIN account. Hard to find correct search terms. ................................................................................................... 13

2
 Executive Summary

This test was designed to examine the usability of the OIT help website and associated supporting
documentation. The research objectives were to expose major usability deficiencies and their causes
related to the most common tasks that users complete and to determine whether the supporting
documentation is being used as designed. To this end, ten test participants were drawn from students in
the WR-533 Research Methods for Technical Writers class and divided into groups of two. Tests were
performed remotely via Zoom, with one member of each team acting as the tester, and the other as
moderator and notetaker. Four tasks were tested by each team, so that each task was tested five times in
total. The tasks were:

• Checking print credits in your quota.

• Connecting a Nintendo Switch to wi-fi from a dorm.
• Researching options for using Microsoft Word free via PSU.
• Researching how long you can use your PSU computer account to access pdx.edu email
after graduation.

The data collected during the tests included the users self-reported level of experience with the OIT help
website and support documentation, think-aloud commentary while performing the tasks, as well as
observations made by the test moderator. Tests were timed, and following the tests users were asked to
rate their level of success and how well they felt the documentation supported completion of the task.

Data analysis supported the following findings which identify recommended areas for further
research/improvement:

• Some students were unaware of the OIT help website.

• User experiences were highly variable based on search terms.
• Most users didn’t want to have to go to a 3rd party site or submit a form, they just
wanted to perform the task from the OIT site.
• Users seem to prefer using search tool rather than digging through links on the front
page.

3
 Summaries of Data
Task Accuracy

Task Set and Time to Complete User Rates Success Moderator Rates
Number Task of Task Completion Success of Task
Completion

Set #1, Task #1 3:05 5 5

Set #1, Task #1 1:00 3 4

Set #1, Task #1 0:10 5 5

Set #1, Task #1 1:32 5 5

Set #1, Task #1 2:00 5 5

Set #1, Task #2 1:00 4 5

Set #1, Task #2 1:40 5 5

Set #1, Task #2 2:17 5 5

Set #1, Task #2 3:12 5 5

Set #1, Task #2 5:00 4 5

Set #2, Task #1 2:47 5 5

Set #2, Task #1 0:50 5 5

Set #2, Task #1 1:30 5 5

Set #2, Task #1 0:25 5 5

Set #2, Task #1 5:05 5 4

4
 Set #2, Task #2 1:39 5 5

Set #2, Task #2 0:43 5 5

Set #2, Task #2 3:30 3 3

Set #2, Task #2 2:35 3 4

Set #2, Task #2 3:58 4 5

Task Timings

The most important data points are the min/max values, given the small sample size. These represent
the clearest insight into instances where user experiences measurably differed.

Time to completion (Seconds)

Set 1 Set 2

Task 1 Task 2 Task 1 Task 2

Average 153 158 126 151

Median 92 137 90 155

Min 10 60 25 43

Max 185 300 305 238

Std D. 65.51946 93.17832 119.8453 81.72209

5
 Preference Data

How well do you feel the OIT Help documentation

Experience Level
supported the task?

Set 1 Task 1 Not well at all Very well

1 2 3 4 5

Never Used 0 0 0 0 0 0

Some Experience 5 0 0 2 1 2

Experienced/Confident 0 0 0 0 0 0

Set 1 Task 2 Not well at all Very well

1 2 3 4 5

Never Used 0 0 0 0 0 0

Some Experience 5 0 1 1 2 1

Experienced/Confident 0 0 0 0 0 0

Set 2 Task 1 Not well at all Very well

1 2 3 4 5

Never Used 2 0 0 0 0 2

Some Experience 2 0 0 0 0 2

Experienced/Confident 1 0 0 0 1 0

Set 2 Task 2 Not well at all Very well

1 2 3 4 5

6
 Never Used 2 1 0 1 0 0

Some Experience 1 0 0 0 1 0

Experienced/Confident 1 0 0 0 1 0

No Answer 1 1 0 0 0 0

Totals (across tasks) 20 2 1 4 6 7

Suggestions for improvement

Set/ Task Suggestion

I should have had options about printing on campus without having to enter "print on campus". A
lot of people are probably wondering how many pages they have left to print in a computer lab. Just
Set #1 Task #1 putting in "print" should have been sufficient to bring me to the page I eventually found.

Set #1 Task #1 Took too long, info should all be in the same spot.

Set #1 Task #1 None. It seemed to have a good idea of what users were searching for and why.

Maybe reorder the welcome page - make common questions more available - have explicit prompt
Set #1 Task #1 "search here for..." - need more direction

Set #1 Task #1 Clearly found help. No real improvement needed.

It is too difficult to find the help website, without knowing to go there. Once there, finding help is
easy- but I would have never known about this site otherwise. If there was an explanation of why I
need to register my device to Wi-Fi, that would be helpful. Because I don't know why that is
Set #1 Task #2 necessary.

It might have been helpful to provide an article telling people that they need to fill out a form, with a
link to the form. When I came into the form, I wasn't even sure it was what I was looking for.
Otherwise I would have just assumed that I could look up the SSID and password and get on alone
Set #1 Task #2 without requesting help.

7
 "Register device to Wi-Fi" might be confusing to inexperienced users. Needs link to define the term
Set #1 Task #2 "MAC address".

Searched for Nintendo Switch- none of the choices had Nintendo switch- clicked on MAC address
article because had previous experience with MAC address. Otherwise, wouldn't have had a clue.
Maybe would have gone to register a device first - but that is to send a request, not how to register.
Set #1 Task #2 Need more direction from the site.

Set #1 Task #2 Feels like instructions should be on OIT site and shouldn't involve the user leaving without warning.

First attempt was about forms and she wasn't sure that was what she needed. Note on search results
Set #2 Task #1 that it should say Free Microsoft Software for Students.

Make help website easier to find or locate because never been to that website and I consider myself
Set #2 Task #1 experienced.

Set #2 Task #1 Add hyperlink to emboldened "download the software".

"There should be something that tells me I'm going to a trusted site." "Not a lot of trust instilled" in
Set #2 Task #1 the help topic.

The main thing is the only option from the main page of the help site is to search. This task may be
straight forward in terms of knowing to enter Word as a search term. The actual instructions were
Set #2 Task #1 fine. The actual install office link was up in the corner, could have been more prominent.

Better if “Learn More About” sections were clearer about the fact that you were going into that
Set #2 Task #2 section. Not clear what all is contained in the links. Description is incomplete for each article.

Create subsections in the article organized by year like "for graduating students" because many
Set #2 Task #2 students don't think about accessing their ODIN account after graduation.

Add keywords for graduation related topics to link the expiration of IT accounts to the expiration of
Set #2 Task #2 student status.

Set #2 Task #2 "Wish it was clear to include email." Had to use "heavy inference."

The actual documentation is fine but getting to the right documentation is hard. It was hard to find
the right search terms. Info about email address and ODIN account should have been on the
homepage because it is so common and important. If tester had not thought of the right term, she
Set #2 Task #2 would have stopped. At one point she almost gave up.

8
 Analysis of Data
Source of Error Analysis

Set #1-Task #1
Task Source of Error/Difficulty Severity Ranking

Checking print credits in Counterintuitive search engine: 3

your quota.
User tried multiple queries to find what
they were looking for.
“Lots of clicks, answers fairly embedded”
Target article was buried at the bottom of
the page; the user did not notice it at first.
“First article doesn’t seem to be helpful. I
would click the first thing that comes up.”

Needless extra steps: 1

User was prompted to fill out one of
several suggested forms to get help.

Confusing/disorganized layout 2
“Make common questions more available
- have explicit prompt ‘search here for’”
“Took too long, info should all be in the
same spot”

Set #1-Task #2
Task Source of Error/Difficulty Severity Ranking

Setting up Nintendo Switch Unintuitive search engine response 3

9
on wi-fi in dorm. Does not warn user before redirecting to 1
3rd party website

User expects integration between 3rd 1

party site data and OIT form if being
redirected back and forth

Participants expected to be able to 1

perform task without having to submit a
request

Set #2-Task #1
Task Source of Error/Difficulty Severity Ranking

Researching options for Instead of finding correct help topic, some 1

using MS Word free via testers initially found an unrelated form for
PSU. installing other software.

Some testers were confused about using 2

PDX email with Outlook.

Tester thought it felt sketchy being directed 1

to external site to download software.

Some testers thought the MS Office 1

download link wasn’t prominent enough.

Set #2-Task #2
Task Source of Error/Difficulty Severity Ranking

Researching how long you Hard to find. 2

can use your PSU computer
account to access pdx.edu Not explicitly on front page so most testers
email after graduation. turned to search tool.

10
 One tester went to the “Odin Account and
Password” link but had trouble deciding
where to go from there.

Unfinished/cut-off article description on 1

“Odin computer accounts and passwords”
doesn’t explain what is on the page and
creates confusion.

Search engine issues. 3

Many testers had difficulty finding the

appropriate search terms.

Odin vs. email: language around Odin and

email are unclear. “Email” can lead you to
D2L or other irrelevant places, makes
searching difficult.

Some users had trouble locating

information based on “graduate” key terms.

Findings and Recommendations

General Findings and Recommendations (global issues for OIT Help
system)
Finding #1: Many students are unfamiliar with the OIT help page.
• “Make help website easier to find or locate because I’ve never been to that website and I
consider myself experienced.”

• “It is too difficult to find the help website without knowing to go there. Once there, finding help
is easy- but I would have never known about this site otherwise.”

11
Finding #2: User experiences were highly variable based on search
terms.
• “Search terms didn't seem to have much connection to the task. However, she still found this
very, very quickly. I wonder what other search terms would have brought up the same
information.”

• “User struggled with search term and search results a bit.”

• “Search for Nintendo Switch- Nintendo Switch was not mentioned in the results.”

Finding #3: Most users didn’t want to have to go to a 3rd party site or
submit a form, they just wanted to perform the task from the OIT site.

• "Wait, why am I leaving the OIT site?" "I was not reassured that this was a safe site to visit"
"Why couldn't OIT give me this information without sending me elsewhere?" "Wait... now I have
to GO BACK to the OIT site?"

• "I just want to add my device!"

• You have to send a help request to get this done?

Finding #4: Successful task completion not clearly identified or

universally understood by moderators. Lots of variance in how far
testers went before indicating task completion.

● Some testers never reached an external website and ended at the OIT help topic.

● For the MS Word scenario and the Nintendo Switch scenario, some testers stopped at the OIT
help topic.

Finding #5: Users seem to prefer using search tool rather than digging
through links on the front page.

12
 Specific Findings and Recommendations
Finding #1: PSU email account expiration information not explicitly
stated on OIT help topic for ODIN account. Hard to find correct search
terms.
• “I'm assuming if I lose access to ODIN, I lose access to email.”

• “Wish it was clear to include email”

• “It was hard to find the right search terms.”

• “Add keywords for graduation related topics to link the expiration of IT accounts to the
expiration of student status”

• “Tried searching ‘email account.’ Everything coming up references D2L, not email.” Need email
specific search terms.

13