Usability Test for the Hacking Your College Textbook RLO Prototype

Developed by Tim Lockman for EDUC 767 project

Peer feedback by Nadine Anderson

Test Setting and Participants

Setting for the user test
The user test will take place in a staff workroom in the campus Library, on a computer equipped with a
standard mouse and keyboard, running on a Microsoft Windows operating system, with a standard
software image that is very similar to public workstations. The monitor will be a standard wide-screen
model used in campus computer labs. This setup was chosen in order to emulate an authentic setting
that the target audience will learn in, at least when on campus.
Description of the test population
The test will be administered to ten users, selected from the general population of students at the
college. Every attempt will be made to choose users from a variety of backgrounds, but with novice-level
experience with the subject matter.
Administration of the tests
Tests will be administered by the colleges instructional designer.
General instructions for users
Users will receive a set of tasks and instructions, listed below. They also will be instructed to think out
loud as they complete the tasks, so that test monitors can capture their covert performances as well as
the overt ones. All participants will receive an identical set of tasks, and the test administrator will read
any verbal directions from a script, in order to limit bias and help ensure valid results. After each test is
complete, the user will be thanked for his/her participation and given the agreed-upon compensation.

The User Test

General Usability Questions
1. Does the number of slides make sense? (that is, are there any unnecessary slides, or should
there be additional slides that are not present?)
Feedback: The number of slides makes sense, though I think an additional slide explaining
the Navigation options available and how to use them at the beginning could be helpful for
students
ME: Our general recommendation for this [giving directions] and other factors (such as the
amount of detail in text) is to start with minimal content and determine through user
testing if more detail is needed (Alessi & Trollip, 2000, p. 50). I plan to keep the directions
for this short RLO minimal, with the expectation that user testing would tell me what further
directions would be needed.
2. Is the sequence of the slides logical and understandable?
Feedback: Yes, I think the way the content was presented and sequenced made sense and
helped support the learning of the lesson outcome
3. Is there too much stuff on any of the slides, or does the amount of stuff seem about right?
Feedback: I like having the Rosario example (I may steal this idea for a course Im
developing). However, I felt like having the Rosario examples on the same pages as the
 instruction meant that there was too much on the slides. Perhaps the content could be
chunked a bit more by moving the Rosario examples to separate slides. ME: Just the Intro
slide, or also the Overview?
4. Are the images helpful?
Feedback: The images were engaging, and I liked the way you used the Google Maps
screenshot to provide visual explanation of what you were teaching
5. Are there enough images?
Feedback: If you do break up the Rosario examples from the instruction slides, perhaps
some images on the instruction slides illustrating your instruction would be helpful
6. Do the images need further explanation?
Feedback: No, I think the images are fine. I cant tell if they have alt text, though, so that
should be added if you havent added alt text to them already ME: Confirmed that alt text is
present
7. Is there sufficient information in the training to answer the quiz questions?
Feedback: Yes, I felt that I had learned enough information from your instruction and
Rosarios examples to answer the questions. I also liked the way that you branched your
Knowledge Checks so that students would be sent back to the relevant content slide to
review before trying to answer the question again.
8. Would the training benefit from adding points of reflection that ask the student to think over
various points of information?
Feedback: I think the Knowledge Checks already do a good job of that perhaps thats
something you would integrate into the theoretical Connect activity for this lesson?
9. Are the color choices appropriate or are they distracting?
Feedback: I thought the color choices were great; they were engaging but not distracting
Usability Tasks
1. Navigate to the slide titled The Table of Contents. After reading the slide, go to the next slide,
and answer the Knowledge Check question. Continue trying until you get the correct answer.
1) Were you able to complete the task?
Feedback: Yes, I was able to complete the task
2) What path did they take in completing the task?
Feedback: Table of Contents Next - Knowledge Check Submit wrong answer
feedback window - Continue - Table of Contents Next Knowledge Check
Submit correct answer feedback window Continue Introduction and
Conclusion
3) Did the path taken seem efficient? (that is, no excessive searching or clicking required)
Feedback: Yes, the path was an efficient way to deliver the feedback and relevant
lesson review
4) Did you have any problems completing the task?
Feedback: No, I did not have any problems completing the task
5) Do you have any suggestions?
Feedback: None, the navigation was clear and it seemed to work in the way that
you planned

2. Navigate to the slide titled Intro and Conclusion. After reading the slide, to go to the next
slide, and answer the Knowledge Check question. Continue trying until you get the correct
answer.
1) Were you able to complete the task?
 Feedback: Yes, I was able to complete the task
2) What path did they take in completing the task?
Feedback: Intro and Conclusion Next Knowledge Check Submit wrong
answer feedback window Continue Intro and Conclusion Next Knowledge
Check Submit correct answer feedback window Continue Review Questions
3) Did the path taken seem efficient? (that is, no excessive searching or clicking required)
Feedback: Yes, the path was an efficient way to deliver the feedback and relevant
lesson review
4) Did you have any problems completing the task?
Feedback: No, I did not have any problems completing the task
5) Do you have any suggestions?
Feedback: None, the navigation was clear and it seemed to work in the way that
you planned

3. Navigate to the slide titled Review Questions. Follow these steps:

1) Play the sound file that is on this slide.
2) As soon as you start hearing sound, pause the sound file.
3) Start the sound file over again, and listen carefully to the end.
Please answer the following questions:
a. Were you able to play the sound file?
Feedback: Yes, I was able to play the sound file
b. Were you able to hear the sound at a comfortable listening level?
Feedback: No, I turned the sound on my computer all the way up, but it still
wasnt very clear or loud
c. Was the sound quality good, or was it poor? (that is, was there hissing, humming,
distortion, etc., or was the sound clear?)
Feedback: The sound quality was not good there was humming and the sound
wasnt clear. The narration quality was good, though you enunciated clearly and
spoke at an engaging pace. It just wasnt recorded well do you have access to a
headset that would improve the sound quality of the recording?
d. Did you have any problems starting the sound file over again and listening from the
beginning?
Feedback: Yes, I wasnt actually able to figure out how to start the sound file over
again and listen to it from the beginning. Some instructions on how to do so
would be helpful.

4. On the Review Questions slide, follow these steps:

1) View the text version of the characters story (not the sound file).
2) Read the characters story carefully.
3) Close the text box that contains the characters story.
Please answer the following questions:
a. Did you have any problems viewing the text version of the characters story?
Feedback: No, I didnt have any trouble viewing the text version of Rosarios story
b. Were you able to read the text comfortably? (that is, did the color, size, font,
background, etc. make it easy or difficult to read?)
Feedback: Yes, the text in this story was actually the clearest to read of any in the
lesson. You may want to apply your text choices for this story to the rest of your
slides.
c. Did you have any problems closing the text box that contains the characters story?
Feedback: No, the instructions within the text box made it quite clear what I
needed to do to close the text box.