Running head: LEGO INSTRUCTION SET TEST

Lego Instruction Set Test

Chad Goss
Sharon Stetov
Vincent Tong
Tre Hawkins
Matthew Gardner
Eleanor Chambers
Miami University

October 24, 2016

LEGO INSTRUCTION SET TEST

Table of Contents
I.
II.
III.
IV.
V.
VI.
VII.

Introduction pg 3
Executive Summary pg 3 & 4
Methodology pg 4 & 5
Results pg 5 & 6
Test Reader Comments pg 6
Recommendations pg 6
Conclusion pg 7

LEGO INSTRUCTION SET TEST

I.

Introduction

We created a helicopter using the Lego Set number 6911. Our instruction set was split up
into different phases that helped the reader create different aspects of the helicopter. We included
pictures of each step when additional lego pieces were added on. The purpose of our instruction
set was to help readers build the helicopter and notify readers of the dangers that small legos
could have on young children playing with the set.
We had another group conduct the test. There were five individuals in this group and
three actually worked hands on to conduct the test. The only materials used were the instructions
that are in an online slideshow and the lego set. Before the three individuals produced the
helicopter, we asked them Likert-Scale questions. After testing, we asked them the same LikertScale questions for comparison.

II.

Executive Summary

The usability test took place in Bachelor Hall Room 252 on Wednesday, October 5th,
2016. The purpose of the test was to ensure the readability of the instructions and gather
feedback on the quality of our format. We also wanted to verify there were no errors in the
instructions as the users went through the document and assembled the helicopter.
There were five participants. Three of them went through the steps to build the helicopter
for about ten minutes. They answered some Likert-scale questions before and after the test. They
also answered a few more questions during the following class session for about five minutes.
The three participants that actually worked with the legos had no complications when it
came to building the helicopter. The participants were able to build the helicopter by just using
the pictures presented in our slideshow. They worked together and helped each other out when
one of them jumped to a conclusion about a certain aspect of the project design. The participants
were all diverse. One participant is an international student who speaks English as a second
language. The two other participants are not international students and had a little more
experience building legos. All three of the participants building the legos were male and either
3rd or 4th year students.
There were several issues with the instructions:
When finding the pieces, they needed to flip between slides often.
Parts not used were skipped
Parts needed for phase 1 were skipped.
The pictures were relied upon much more than the words.
Parts were missed when being pulled out in beginning. (user error, were included)
Incorrect assumptions were made between phases as to how the phases fit
together.
There was one extra 1x1 black dot piece that was unaccounted for in the parts
lists.

LEGO INSTRUCTION SET TEST

This document details our methodology, the results of the test, the participants
comments, our recommendations for the instructions and a conclusion. The answers the
participants gave for the Likert-scale questions before and after the test are provided afterwards
as an attached file. We also included the participants responses to additional questions after the
test.

III. Methodology
Participants were recruited from students from the ENG 313 Section G class. These
students were previously broken into smaller groups so that they could complete a group writing
project. One of our group members approached their group and asked them to test the instruction
set that our group had created for the 6911 Lego instruction set. These students all agreed to test
our instructions as long as our group agreed to test their lego instructions. This agreement for
mutual testing of each others instructions was the point that secured their cooperation.
First, the test group was asked to fill out a pre-test questionnaire that took less than five
minutes. After the pre-test questionnaire was complete, the test group was instructed to build the
lego set following our instructions list. It took the test group about ten minutes to complete the
build and about five minutes to fill out our post-build questionnaire, which we used to evaluate
how helpful our instruction set was to the build.
Participants were asked to use provided lego pieces to construct a helicopter to a
specified final product. They were supposed to use the pieces given to them and the instructions
that our group created and provided, and they were asked to refrain from asking questions or
using any aides outside of the given questions. The post-test session involved members of our
group asking members from the participating group to answer questions relating to the ease of
use of the instructions we created. These sessions were generally short and involved us asking
their group questions that were meant to gather honest feedback about the effectiveness of our
instructions. Theses questions were filled out on Google Docs and are provided below as an
attachment so that their responses can be seen.
There was a pre-test and a post-test questionnaire. The pre-test questionnaire consisted of
five Likert Scale questions concerning the testers ability and perceived competence in building
lego sets. The questionnaire also asked about the testers ability to work in a group. The post-test
questionnaire asked the same Likert Scale questions, as well as several open-ended questions
which aimed to discover any issues or problems that appeared while following the instructions.

A. Participants
There are five participants in the group that we worked with. Three of them worked
together to complete the helicopter. The date that they worked on the project October 5th during
our English 313 class from 4:00-5:20. One participant is a sophomore international student with
English as a second language and a physics major. Another participant is a junior student that
said he had experience building legos when he was younger. His major is sports leadership and

LEGO INSTRUCTION SET TEST

management. The last participant is a sophomore student majoring in kinesiology.

B. Evaluation Tasks
The group, after a period of intense brainstorming, created a set of 6 standardized tasks that were
deemed as critical to the viability of the project. They are listed below:
Task 1: Constructing the base (phase 1) of the lego
Task 2: Constructing the cockpit portion (phase 2) of the lego
Task 3: Constructing the rotor portion (phase 3) of the lego
Task 4: Constructing the tail portion (phase 4) of the lego
Task 5: Image use
Task 6: Assembling the phases into the final lego set

IV. Results
A. Task Completion Success Rate
The ability of the participants to complete the task was observed by group members Chad
Goss and Sharon Svetov. This was done by having both members of our group stand behind the
participants and observe the ways in which their team worked together, interacted with the
instructions, made mistakes, and completed the lego set.
Participants

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

Success

Completion
Rate

100%

100%

100%

100%

0%

66%

B. Errors
There were two observers during the usability test, both of whom were members of the
instructions team. They took detailed observation notes using the observation sheet found in the
Attachments section, and using these notes, they compiled a list of issues that needed to be
resolved.
There were few errors made during the test; the major errors made were during task 6,
which was assembling the 4 phases into the completed lego. Users made several incorrect
assumptions about the final assembly before the they reached the appropriate instructions.

LEGO INSTRUCTION SET TEST

Task 1 through 4 involved assembling the 4 sections of the lego. Task 5 rated whether the
users understood and followed the pictorial instructions correctly. All of these tasks were
performed without any issues.

V.

Test Reader Comments

The participants indicated that they really enjoyed the pictures we included throughout
our presentation. They thought that the different views in each step really helped view all the
pieces that were needed for the specific step and phase. They also thought that the phases were a
great idea because it helped spread out the different parts of the helicopter and make the
directions easier to follow. They also specifically made comments that they enjoyed the level of
detail that was given in the powerpoint
The participants had only a few comments about what they didnt like about the
instruction set. They indicated that going back and forth through the Parts Used and Parts Not
Used list was a little confusing and time consuming. They also indicated that the directions that
were written in words, although very detail, were unused/unneeded because of how specific the
pictures we included were. One last comment they had was in putting the helicopter together,
they had some confusion when attaching the correct pieces to one another.
Some recommendations that the group had to improve our presentation included making
the pictures more emphasized due to the fact that they utilized that part of the instruction set
more so than the textual part of the instruction set. Seeing as though they very much enjoyed the
different views in each step, these instructions can be even more efficient by making the pictures
our primary form of instruction, and using the text as helpful guidelines if the pictures are not
clear to the viewer.

VI. Recommendations
After hearing the overall feedback from the group, we decided to make a couple of minor
changes to the instructions to accommodate their comments and concerns. To start, we are going
to correct the Parts Used and Parts Not Used list so that there are an accurate amount of 1x1
black pieces on each sheet. Next, we should also emphasize the pictures more so than the textual
instructions, due to the fact that the group told us that they mainly used the pictures, and not the
text to create the helicopter. Lastly, we also think that we should test these instructions out with a
younger age group in order to confirm the accuracy of our instructions. We recognize that a
twenty year old age group is very different than the elementary-aged age group that would more
frequently be utilizing and building our product.

VII. Conclusion
Our participants found that it was very easy to build the helicopter based off of the
pictures that we have included in the instruction set. We found that including all the pictures was

LEGO INSTRUCTION SET TEST

very helpful in creating a helicopter. Young kids who normally play with legos have little
patience and are more likely to use pictures when building a lego set than reading through
instructions, although both were included. We also found that it takes about 10 minutes for 18-22
year old students to build the lego which means that young kids ages 6-12 could spend about half
an hour working through the lego set.

LEGO INSTRUCTION SET TEST

Attachments
I. Writers Guide for Testing Drafts
ii. Likert-Scale Test Questions
Likert Scale Questions Before

1. I am confident that I can create a lego set:

1. St
rongly
Agree

2. Agree

3. Undecided

4. Disagree

5. Strongly
Disagree

4. Disagree

5. Strongly
Disagree

4. Disagree

5. Strongly
Disagree

Answer: 2
2. I often construct things easily without instructions:
1. St
rongly
Agree

2. Agree

3. Undecided

Answer: 2
3. I can easily finish a project with a group:
1. St
rongly
Agree

2. Agree

3. Undecided

Answer: 1
4. I am confident I can finish the product in an efficient amount of time:
1. St
rongly
Agree

2. Agree

3. Undecided

4. Disagree

5. Strongly
Disagree

Answer: 2
5. I am able to clearly communicate the process that I consider the best way forward:
1. St
rongly
Agree
Answer: 2

2. Agree

3. Undecided

4. Disagree

5. Strongly
Disagree

LEGO INSTRUCTION SET TEST

LEGO INSTRUCTION SET TEST

10

Likert Scale Questions After

1. I am confident that I can create a lego set:

2. St
rongly
Agree

2. Agree

3. Undecided

4. Disagree

5. Strongly
Disagree

4. Disagree

5. Strongly
Disagree

4. Disagree

5. Strongly
Disagree

Answer: 1
2. I often construct things easily without instructions:
2. St
rongly
Agree

2. Agree

3. Undecided

Answer: 1
3. I can easily finish a project with a group:
2. St
rongly
Agree

2. Agree

3. Undecided

Answer: 1
4. I am confident I can finish the product in an efficient amount of time:
2. St
rongly
Agree

2. Agree

3. Undecided

4. Disagree

5. Strongly
Disagree

Answer: 1
5. I am able to clearly communicate the process that I consider the best way forward:
2. St
rongly
Agree

2. Agree

3. Undecided

4. Disagree

5. Strongly
Disagree

Answer: 1

iii. Observation Sheet for User Test

Problems
1. Did the users encounter any obstacles during the test? At which stages did they occur?
When finding the pieces they needed they had to flip between slides often.

LEGO INSTRUCTION SET TEST

11

Parts not used was totally skipped

Parts needed for phase 1 was skipped.
Pictures are much more relied upon than words.
Parts were missed when being pulled out in beginning. (user error, were included)

2. Were there any other issues?

Incorrect assumptions were made between phases as to how the phases fit together.
There was one extra 1x1 black dot. (This must be added to the parts not used list and
picture)

Interpretation
1. Did the users complete any steps out of order?
For the most part the users completed the instructions in order. For each of the phases they followed each
step in order, however when constructing the final product they did assume they knew how to put the
phases together prior to getting to that step and attempted to put two phases together incorrectly.
2. Could the current order of steps be improved upon?
Due to the complexity of the build, the current order of the steps cannot truly be improved upon. As
demonstrated in the test, when users follow the instructions as they are presented in the instruction set the
final product is easy to get to. This being said, the wording of the slides could be clearer, and state that
more steps must be completed prior to the steps already built going together.
3. Any other results to note?
There was one mistake in the picture and pieces needed, however this did not hinder the ability of the
participants to finish the build. Likewise, the fact that the words were hardly used in comparison to the
pictures is a very noteworthy conclusion from the test. Finally, the fact that the pictures were so heavily
relied upon for each step shows that their detail and accuracy cannot be understated when it comes to the
final set of instructions.

Solution
1. How could the issues above be resolved?
The issues could be resolved by adding the missing black stud to the unused parts list and
remove it from the used parts list. It is possible that the text instructions can be removed but they
dont detract from the instructions and may be useful to other readers. There is little else to
improve upon based on the test data we received.
2. What areas of the instructions need attention based on what you observed?
The parts lists need edited. Otherwise there are not many places that need improved. Perhaps
the transition between phases could be looked at as well.
iv. Follow-up Interview Questions for Readers

LEGO INSTRUCTION SET TEST

12

Post-Interview Question
1. Do you feel the instructions provided allowed you to completely construct the final
product?
Answer:

yes

2. Were there were parts of the instructions that were confusing or unclear?
Answer:

no

3. Did you have many questions about what you was supposed to be doing as you were constructing the
final product?
Answer:

no

4. Were the pictures well related to the written instructions and accurately represent what was being
described?
Answer:

yes

5. Was the wording of instructions clear, free of mistakes, had no ambiguous wording, and were free of
grammatical errors.
Answer:

Only problem was within the different phases, got stumped with connecting
propeller with the actual base

6. Was the layout of instructions consistent throughout the entire instruction set? Did it lead to
confusion between phases or steps?
Answer:

The phases, confusing at the end putting it together

7. Was it easy to continue on from where one phase or step finished, to where the next one began.
Answer:

yeah