Gonzales, Sherry Mae R_Laboratory Experiment 6_Usability

Republic of the Philippines
BATANGAS STATE UNIVERSITY

The National Engineering University
Alangilan Campus
Golden Country Homes, Alangilan Batangas City, Batangas, Philippines 4200
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
E-mail Address: coe.alangilan@g.batstate-u.edu.ph | Website Address: http://www.batstate-u.edu.ph
College of Engineering - Department of Industrial Engineering
LABORATORY EXPERIMENT #6
Course Code IE415

Course Title Ergonomics 2: Cognitive Ergonomics
Section IE 3205
Semester Second Semester, AY 2023-2024
Date May 12, 2024
Name of Student Gonzales, Sherry Mae R.
SR Code 21-05062
Instructor Engr. Princess Arianne E. Esmana
Experiment Title USABILITY

Category Individual Report #3
Leading Innovations, Transforming Lives, Building the Nation

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
I. OBJECTIVES
Students should be able to:

1. Understand the interaction between human and machine by conducting a simple
usability test of a website of choice.
2. Create a usability report.
3. Recommend solutions to improve usability of the website under study based from
the findings.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
USABILITY REPORT
I. BACKGROUND OF THE STUDY
The primary objective of this study is to evaluate the usability of Pinterest

application for their target market: people who finds Lifestyle ideas. Three aspects of
usability, namely effectiveness, efficiency, and satisfaction, will be tested: (1) to
determine effectiveness through error rate, (2) to determine efficiency through time-study
of completion of tasks, and (3) to determine satisfaction through the score generated from
the participant’s responses on the System Usability Scale (SUS) questionnaire. The
participants of the study were divided into two groups: novices without prior background
or experience with the Pinterest application, and the experts who have used the system
at least once. Each participant was given the same three tasks on the same experimental
setup.
II. METHODOLOGY
1. Description of Tasks
The three (3) tasks that the participants need to complete were common tasks that
a customer usually does in the website being tested. Shown in Table 1 are the instructions
for each task as well as the expected minimum number of mouse clicks needed to
accomplish it.
Table 1: Task Description with Mouse Clicks
Task No. Task Instruction Expected/Minimum

Mouse Clicks
1 Share a photo from Pinterest to Messenger 4 clicks
Search ‘Waves’ and visit the comment section of the
2 11 clicks
first photo that will appear
3
Create a secret ‘Board’ in the application 6 clicks

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
2. Selection of Participants
Conducting the experiment required a total of six (6) participants: three (3)
novices that have not yet experienced using the Pinterest application, and three (3)
experts who have used the website at least once. The participants chosen are middle aged
people with small age gap. According to Becker (2004), this is to remove possible errors
committed in the age gap, mainly age factors (vision, cognition, motor skills, and
literacy) of the web design.
Participants’ Profile
Table 2: Subject Profile for Novice Users
Name Age Brief Description
Roma 34 Female, married, a mom and not familiar with Pinterest app
Jestine 32 Female, single, and not familiar with Pinterest app
Maribeth 36 Female, married, a mom and not familiar with Pinterest app
Table 3: Subject Profile for Expert Users
Name Age Brief Description
Daniella 21 3rd year Industrial Engineering student in BSU
Sophia 21 3rd year Industrial Engineering student in BSU
Aiyana 21 3rd year Industrial Engineering student in BSU

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Apparatus /Materials Used
a. Tally Sheet
To be able to determine the number of errors and number of click of each
respondent, I have decided to make a tally sheet. By that, I was able to record the necessary
data needed for this report.
b. Smartphone
Stopwatch
It is also necessary to know the length of time that the respondents consumed to
accomplish the task. For this reason, I used the stopwatch in my smartphone to identify the
task time of each to the participant.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Pinterest Application
To be able to assess the two group of respondents, a Pinterest application was also
used. It was downloaded in App Store for free.
Procedure
Step 1:
In a random order, all six (6) participants were asked if they have experienced
using the Pinterest application and were told that the experiment results will be kept
confidential. The subjects were then asked to do the experiment using their smart phone
and an open Pinterest application with the said Pinterest application.
The first task assign to the respondents is to share a photo on Messenger
application. Icon or image to be click in chronological order are: certain photo, share icon,
Messenger application, sent button. Second task is to search the word ‘waves in the search
bar of the Pinterest application and visit the comment section of the first photo that will
appear. The clicking process goes like this: tap search bar, type ‘waves’, click search icon,
tap the first photo, click the comment icon. Lastly, the participants are requested to create
a secret board in the application. They need to click the create icon, choose the board
image, type the desire board name, invite friends, and slide the ‘make this board as a
secret’ button.
For every task, their number of errors were tallied using a tally sheet. Then, the
time they consumed to finish each task was also recorded using a timer. All of the gathered
data were summarized in tables.
Step 2:
After finishing all tasks, the participants were required to answer the System
Usability Scale (SUS), a 10-item survey questionnaire with a 5-point Likert scale response
from strongly disagree (1) to strongly agree (5).

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
III. RESULTS AND ANALYSIS OF THE STUDY
a. Summary of Data Gathered

All data recordings gathered were compiled and analyzed one by one. The
participant's' facial expressions, mode of language, keystrokes, mouse clicks, and errors
were observed and marked in the video clip. Each marker was given a score (0 - severe,
1 - medium, 2 - minor) for data evaluation later on. Shown in Table 4 are the observations
encountered in the experiment and their classification as minor, medium, or severe.
Table 4: Observations During Tasks
MINOR (2) MEDIUM (1) SEVERE (0)

• Confused
expression
• Wrong tap on
• Short pause • Asking questions
screen
• Frequent errors
b. Performance Measures
1. Effectiveness through error rate
Table 5
ERROR RATE: NOVICE GROUP
Expected
Participant Participant Participant Total
TASK Number of Error Rate
1 2 3 Number of
NUMBER Click per (%)
Error Error Error Errors
Task
Task 1 2 1 1 4 4 33%
Task 2 4 3 2 9 11 27%
Task 3 1 2 2 5 6 28%
Table 6
ERROR RATE: EXPERT GROUP
Expected
Participant Participant Participant Total
TASK Number of Error Rate
1 2 3 Number of
NUMBER Click per (%)
Error Error Error Errors
Task
Task 1 0 1 0 1 4 8%
Task 2 1 1 0 2 11 6%
Task 3 0 0 0 0 6 0%

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Figure 1: Error Rate
As presented on the figure above, the novice groups produce high errors on three
different tasks. On the other hand, the expert group produces only very minimal error to
none. It means that the first-time user of Pinterest tends to commit more error than those
people who usually use the application.
2. Efficiency through time-study of completion of tasks
Table 7
TASK TIMES: NOVICE GROUP
Task Time Task Time Task Time Average Task
TASK
Participant Participant Participant Time
NUMBER
1 2 3 (seconds)
Task 1 12 9 7 9.3
Task 2 16 14 11 13.7
Task 3 11 13 12 12
Table 8
TASK TIMES: EXPERT GROUP
Task Time Task Time Task Time Average Task
TASK
Participant Participant Participant Time
NUMBER
1 2 3 (seconds)
Task 1 7 9 8 8
Task 2 9 10 8 9
Task 3 7 6 6 6.3

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Figure 2: Average Task Time
Analyzing the average task time of the respondents, it can be observed that novice
group have higher task time the three-task compared to the expert group. The greatest
number of clicks task tends to have high average task times. Hence it can be concluded that
the first-time user of the application has higher task time compared to the expert users.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
3. Satisfaction through the score generated from the participant’s responses

on the System Usability Scale (SUS) questionnaire.
The industry standard metric for evaluating the usability of products and websites
is called the SUS score. The System Usability Scale, or SUS as it is frequently called, was
developed in 1986 by John Brooke for evaluating the usability of brand-new computer
systems that were being put into use in DEC offices at the time. After that, it was rapidly
adapted for many different kinds of purposes. Participants respond to ten statements on
the usability of the system to determine the SUS score. A total score can be determined by
giving each question a 1–5 rating based on how strongly the respondent agrees or disagrees
with the statements.
Table 9
SUS Questionnaire Result: Novice
Response of Participant (Actual Response): Likert Scale
Number of Number of Number of Number of Number of
TEN SUS QUESTIONS Participant Participant Participant Participant Participant
Who Who Who Who Who
Answered 1 Answered 2 Answered 3 Answered 4 Answered 5
1. I think that I would like to
0 0 0 2 1
use this system frequently.
2. I found the system
0 1 1 1 0
unnecessarily complex.
3. I thought the system was
0 0 0 1 2
easy to use.
4. I think that I would need
the support of a technical
2 1 0 0 0
person to be able to use this
system.
5. I found the various
functions in this system were 0 0 0 1 2
well integrated.
6. I thought there was too
much inconsistency in this 2 1 0 0 0
system.
7. I would imagine that most
people would learn to use this 0 0 0 2 1
system very quickly.
8. I found the system very
2 1 0 0 0
cumbersome to use.
9. I felt very confident using
0 0 1 2 0
the system.
10. I needed to learn a lot of
things before I could get going 2 0 1 0 0
with this system.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Figure 3: SUS Questionnaire Result: Novice
Based on the constructed SUS questionnaire, even number questions pertain to

negativity experience on the applications or how unsatisfied the respondents are on the
application being used to perform the tasks. On the other hand, odd numbered questions
have something to do to the positive experience on the application. This pertains on how
satisfied they are. Based on response summary, most of the respondents answered high
number, between strongly agree to agree, on the odd number questions. While on the
questions on even number, the responses of the respondents are mostly between strongly
disagree to disagree.
It means that the respondents had a good experience on using the application. They
easily understand on how to use the Pinterest application and experience only minimal
inconvenience despite using it for the first time. The application satisfied their
expectations.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Table 10
SUS Questionnaire Result: Expert
Response of Participant (Actual Response): Likert Scale

Number of Number of Number of Number of Number of
TEN SUS QUESTIONS Participant Participant Participant Participant Participant
Who Who Who Who Who
Answered 1 Answered 2 Answered 3 Answered 4 Answered 5
1. I think that I would like to
0 0 0 0 3
use this system frequently.
2. I found the system
2 1 0 0 0
unnecessarily complex.
3. I thought the system was
0 0 0 1 2
easy to use.
4. I think that I would need
the support of a technical
3 0 0 0 0
person to be able to use this
system.
5. I found the various
functions in this system were 0 0 0 1 2
well integrated.
6. I thought there was too
much inconsistency in this 2 1 0 0 0
system.
7. I would imagine that most
people would learn to use this 0 0 0 2 1
system very quickly.
8. I found the system very
2 1 0 0 0
cumbersome to use.
9. I felt very confident using
0 0 0 1 2
the system.
10. I needed to learn a lot of
things before I could get going 2 1 0 0 0
with this system.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Figure 4: SUS Questionnaire Result: Expert
Even number questions on the SUS questionnaire relate to negative experiences or

unsatisfaction with the applications. Conversely, questions with odd numbers are linked to
a satisfying application experience. According to the summary of responses, the majority
of participants provided a high number for the odd numbered questions, ranging from
strongly agree to agree. When asked even-numbered questions, most respondents' answers
fell into the disagree to strongly disagree range.
It indicates that the responders found the application to be user-friendly. Since they
are already familiar on the application, it was easier for them to navigate through the
application and accomplish the tasks given.
Table 11
SUS Questionnaire Score
SUS
Category of
Questionnaire
Respondents
Score
Novice 80
Expert 92.5

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Figure 5: SUS Questionnaire/Usability Score per Group
Figure 6: ANOVA Result of Usability Score per Group
Hypothesis:
H0: The average SUS usability score on different groups is equal.
H1: The average SUS usability score on different groups is not equal.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Decision and Conclusion:

Since F-computed (2.8862) < F-critical (7.70865), Accept H0.
Therefore, the average SUS usability score on different groups is equal.
Since P-value (0.16466) > alpha (0.05), Accept H0.
Therefore, the average SUS usability score on different groups is equal.
The usability score of 80.3 and above means that people love the application being
used. For the score near 68, it means that you are doing good on using the application and
could improve. Lastly, score of 51 and under suggests that there’s a problem on usability
of the application and it needs to be fixed quickly.
Based on the average usability score of novice group, which is 80, the respondents
are doing okay on using the Pinterest applications. While for expert group, they have an
average of 92.5, which means that they love the application and would recommend it to
other people. Both groups perform well on using the application. This means that the
application has good usability function.
c. Usability Problems Identified
The Pinterest application’s development started on December 2009 and it was

launched on March 2010. Currently, it has 518 million active users. Navigating the whole
function of the application can take for about less than 10 minutes. If you are an active
user, doing task in the application is just a piece of cake. Since the Pinterest application is
existing for almost 15 years, several improvements are already made in the application
which was based on the feedbacks and reviews of the users.
Those novice group and expert group were asked on the problems they have
encountered on using the application. Aside from that, there were also problems that was
observed by the person doing the experiment. Only few problems were identified since the
application was convenient to navigate and the respondents achieve high usability score.
Here are some of the identified usability problems in the Pinterest application:
1. Some icons are unfamiliar and unrecognizable, just like the comment icon.
2. The captions that are not in English language cannot be translated.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
3. Not all of the photos available are not having good quality or high resolution.
Having an application that was existing for a long time and undergo several
revisions already, a few to none problems are expected.
IV. RECOMMENDATIONS FOR IMPROVEMENT OF THE WEBSITE
To be able to make the Pinterest application more interesting and gain more users, it is
necessary that the application undergo an improvement process. Through this, the usability of
the application has a chance to increase by considering some of the problems being identified
and observed. Here are the lists of the recommendations to attain bitterness of Pinterest:
1. Change the comment icon to look like the common comment icon. Applies this
concept on other icon so that the user will easily recognize the purpose of each icon.
2. Have a translation button to be used on translating captions on every photo. Provide

different choices of language to be used.
3. Strictly filter the photo to be uploaded by the users by setting minimum quality
resolution requirement.
These suggestions can be applied by the developer of the applications to make the
Pinterest application be more usable. Both the users and developers will be benefited to these
ideas. Feedbacks of the users can be a great help to improve a website or an application. This
can lead to positive result and success of the developers of the application.

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
V. BIBLIOGRAPHY and APPENDICES
Novice Group’s Respondents:
Maribeth Roma Jestine
Expert Group’s Respondents:
Sophia Aiyana Daniella

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Name of Respondent: Maribeth Dimailig Age: 36 years old
Respondent Category: ✔ Novice ____ Expert
The System Usability Scale (SUS) Questionnaire
LIKERT SCALE 1 2 3 4 5
Strongly Strongly
TEN SUS QUESTIONS Disagree Agree
1. I think that I would like to use this system

frequently. ✔
2. I found the system unnecessarily complex. ✔
3. I thought the system was easy to use. ✔

4. I think that I would need the support of a
technical person to be able to use this system. ✔
5. I found the various functions in this system
were well integrated. ✔
6. I thought there was too much inconsistency in
this system. ✔
7. I would imagine that most people would learn
to use this system very quickly. ✔
8. I found the system very cumbersome to use. ✔
9. I felt very confident using the system. ✔

10. I needed to learn a lot of things before I
could get going with this system. ✔
USABILITY SCORE 75

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Name of Respondent: Roma Reyes Age: 34 years old
Strongly Strongly

frequently. ✔

this system. ✔

USABILITY SCORE 72.5

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Name of Respondent: Jestine Gonzales Age: 32 years old
Strongly Strongly

frequently. ✔

this system. ✔


Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Name of Respondent: Daniella Broncano Age: 21years old
Respondent Category: _____Novice ✔ Expert
Strongly Strongly

frequently. ✔

this system. ✔

USABILITY SCORE 100

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Name of Respondent: Ma. Sophia Amado Age: 21years old
Strongly Strongly

frequently. ✔

this system. ✔

USABILITY SCORE 90

Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221
Name of Respondent: Aiyana Jasmin Holgado Age: 21years old
Strongly Strongly

frequently. ✔

this system. ✔


Alangilan Campus
Tel Nos.: (+63 43) 425-0139 local 2121 / 2221

Gonzales, Sherry Mae R_Laboratory Experiment 6_Usability

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Gonzales, Sherry Mae R_Laboratory Experiment 6_Usability

Uploaded by

Copyright:

Available Formats

Republic of the Philippines

BATANGAS STATE UNIVERSITY

College of Engineering - Department of Industrial Engineering

Course Code IE415

Experiment Title USABILITY

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Students should be able to:

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

I. BACKGROUND OF THE STUDY

The primary objective of this study is to evaluate the usability of Pinterest

Task No. Task Instruction Expected/Minimum

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Table 2: Subject Profile for Novice Users

Name Age Brief Description

Jestine 32 Female, single, and not familiar with Pinterest app

Table 3: Subject Profile for Expert Users

Name Age Brief Description

Daniella 21 3rd year Industrial Engineering student in BSU

Sophia 21 3rd year Industrial Engineering student in BSU

Aiyana 21 3rd year Industrial Engineering student in BSU

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Apparatus /Materials Used

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

III. RESULTS AND ANALYSIS OF THE STUDY

a. Summary of Data Gathered

Table 4: Observations During Tasks

MINOR (2) MEDIUM (1) SEVERE (0)

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Figure 1: Error Rate

2. Efficiency through time-study of completion of tasks

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Figure 2: Average Task Time

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

3. Satisfaction through the score generated from the participant’s responses

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Figure 3: SUS Questionnaire Result: Novice

Based on the constructed SUS questionnaire, even number questions pertain to

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Response of Participant (Actual Response): Likert Scale

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Figure 4: SUS Questionnaire Result: Expert

Even number questions on the SUS questionnaire relate to negative experiences or

SUS Questionnaire Score

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Figure 5: SUS Questionnaire/Usability Score per Group

Figure 6: ANOVA Result of Usability Score per Group

Leading Innovations, Transforming Lives, Building the Nation

College of Engineering - Department of Industrial Engineering

Decision and Conclusion:

Therefore, the average SUS usability score on different groups is equal.