Chandler Chamber of Commerce Site

Project Report
3.29.18
4.18.18
Daniel Joji Koons
Table of Contents
1 Introduction ........................................................................................................................................................1
1.1 Purpose .......................................................................................................................................................1
1.2 Tasks Identified ...........................................................................................................................................1
1.3 Assumptions ...............................................................................................................................................1
2 Analysis ...............................................................................................................................................................2
2.1 Personas .....................................................................................................................................................2
Task Analysis Tools .................................................................................................................................................3
2.2 Task #1 ........................................................................................................................................................3
2.2.1 Task Detail #1......................................................................................................................................3
2.2.2 Task #1 Analysis ..................................................................................................................................4
2.2.3 Task #1 Discussion ..............................................................................................................................4
2.3 Task #2 ........................................................................................................................................................4
2.3.1 Task Detail #2......................................................................................................................................4
2.3.2 Task #2 Analysis ..................................................................................................................................4
2.3.3 Task #2 Discussion ..............................................................................................................................4
2.4 Task #3 ........................................................................................................................................................4
2.4.1 Task Detail #3......................................................................................................................................4
2.4.2 Task #3 Analysis ..................................................................................................................................5
2.4.3 Task #3 Discussion ..............................................................................................................................5
2.5 Task #4 ........................................................................................................................................................5
2.5.1 Task Detail #4......................................................................................................................................5
2.5.2 Task #4 Analysis ..................................................................................................................................5
2.5.3 Task #4 Discussion ..............................................................................................................................5
2.6 Task #5 ........................................................................................................................................................5
2.6.1 Task Detail #5......................................................................................................................................5
2.6.2 Task #5 Analysis ..................................................................................................................................6
2.6.3 Task #5 Discussion ..............................................................................................................................6
3 Prototype and Design .........................................................................................................................................6
3.1 Overview of Prototype and Design Features..............................................................................................6
3.2 Task #1 ........................................................................................................................................................6
3.2.1 Task #1 Design ....................................................................................................................................6
3.2.2 Task #1 Design Justifications ..............................................................................................................7
3.2.3 Task #1 Prototype ...............................................................................................................................7
 3.2.4 Task #1 Prototype Rational.................................................................................................................8
3.3 Task #2 ........................................................................................................................................................8
3.3.1 Task #2 Design ....................................................................................................................................8
3.3.2 Task #2 Design Justifications ..............................................................................................................8
3.3.3 Task #2 Prototype ...............................................................................................................................9
3.3.4 Task #1 Prototype Rational.............................................................................................................. 10
3.4 Task #3 ..................................................................................................................................................... 10
3.4.1 Task #3 Design ................................................................................................................................. 10
3.4.2 Task #3 Design Justifications ........................................................................................................... 10
3.4.3 Task #3 Prototype ............................................................................................................................ 10
3.4.4 Task #3 Prototype Rational.............................................................................................................. 11
3.5 Task #4 and 5 ........................................................................................................................................... 11
3.5.1 Task #4 and #5 Design ..................................................................................................................... 11
3.5.2 Task #4 and #5 Design Justifications ............................................................................................... 11
3.5.3 Task #4 and #5 Prototype ................................................................................................................ 11
3.5.4 v Task #4 and #5 Prototype ............................................................................................................. 12
4 A/B Testing ...................................................................................................................................................... 12
4.1 Participants .............................................................................................................................................. 12
4.2 Scenarios.................................................................................................................................................. 12
4.2.1 Task #1 ............................................................................................................................................. 12
4.2.2 Task #2 ............................................................................................................................................. 13
4.2.3 Task #3 ............................................................................................................................................. 13
4.2.4 Task #4 ............................................................................................................................................. 13
4.2.5 Task #5 ............................................................................................................................................. 13
4.3 Equipment ............................................................................................................................................... 13
4.4 Subjective Metrics ................................................................................................................................... 14
4.5 Quantitative Metrics................................................................................................................................ 15
4.6 Test results .............................................................................................................................................. 15
5 Conclusions ...................................................................................................................................................... 18
5.1 Discussion of Results ............................................................................................................................... 18
5.2 Lessons Learned ...................................................................................................................................... 18
5.3 Conclusion ............................................................................................................................................... 18
6 Appendixes ...................................................................................................................................................... 19
6.1 Heuristic Evaluation ................................................................................................................................. 19
6.2 Cognitive Walk-through........................................................................................................................... 22

Page 1 of 39
6.3 New GUI snapshots ................................................................................................................................. 26
6.4 Instructions for participants .................................................................................................................... 30
6.5 Researcher guidelines.............................................................................................................................. 31
6.6 Background questionnaire ...................................................................................................................... 31
6.7 Post-session questionnaire...................................................................................................................... 32
6.8 YouTube Links .......................................................................................................................................... 34

Page 2 of 39
 Daniel Joji Koons CSE463: Project Report

1 Introduction
1.1 Purpose
The main objective of this project is to evaluate and improve the site for the city of Chandler's
Chamber of Commerce http://chandlerchamber.com. This site is the online headquarters for the
chamber, and helps provide support to local businesses as well as the downtown Chandler community,
as well as creating events and projects to help strengthen the diversity and overall vitality of the town.
While the purpose for this kind of site is good, the site is a bit overwhelming to new users. If you were
looking for specific information about events or meetings on the site, it may take a good while to find
those sources as the site is very broad and spread out. The site's UI is also a bit random at times, with
widgets thrown around the page and colors and text not looking to great either. As an example, there
are bits of old information that are still on the site which could be either tossed or pushed back to let
the new sources of information be on the front.
After performing a heuristic evaluation on the site, there were a great amount of design flaws
that would hinder a new user who either wishes to join the chamber as a member, or a user that wishes
to participate more in the Chandler community. With the help of this evaluation, a redesign of the site
and some of its features was created, using the help of testing different types of users to see if either
the redesign is the most efficient design for the site, the redesign is a more inefficient design for the site,
or if both designs could possibly contribute some of their features to create a more logical design for the
site. With these outcomes in mind, it is possible that more problems could result with these
implementations which could be improved in the future.

1.2 Tasks Identified

Most the tasks that the users will be testing will be finding specific information on this site, since
this site is a local-business and community directory for the city of Chandler. The idea is that most users
would be using this site to find out what events are happening in the area, whether they are talks about
improving local business, celebrations on new businesses opening, festivals, etc. These types of
information can be accessed from the site, but there are times where simply googling the event would
be a faster and orderly way to get dates and times for these events. Due to the fact that the last two
tasks are difficult to complete, there will be a 1-minute time limit for the user for the original site and
the prototype before the researcher steps in to help. The user will be directed back to the home page
after each task.
List of tasks:
1. Log in to the site.
2. Become a member of the Chamber (fill-out-form).
3. Navigate to the Chamber Business Calendar and find out what the Wednesday Lunch Leads
Group Meeting is on the 6th of March in 2019 (without clicking the "Events List View" button).
4. Navigate to the Chambers' Leadership Institute application on the site.
5. Navigate to the Women in Leadership Application to become a speaker for the event.

1.3 Assumptions
It is assumed that most of the participants have used some of the basic functions the site contains
and have a general idea of what a chamber of a city is and does. It is also assumed that the participant
will complete the tasks in an efficient and coherent manner, as the site's functions do not require

Page 1 of 39
 Daniel Joji Koons CSE463: Project Report

specific knowledge of a typical user, e.g. a business owner. Participants will be asked what a chamber of
commerce is before being tested and the results will be counted. If the participant does in fact not know
what a chamber of commerce is, they will be given a description.

2 Analysis
2.1 Personas
Persona 1: "The Motivator"

Fictional Name: Sally Sales

Job Title/Responsibilities: Successful Business Owner and Public Speaker

Demographics:  50 years old

 Married

Goals and Tasks: Her goals are oriented in the Woman in Leadership talks and events the
chamber has. Wants information on:
 Becoming a WIL speaker.
 Finding information on specific events in business.

Environment: She is of the chamber's current demographic, is a consumer of the

internet (social media, entertainment, blogs, etc).

Quote: "How can I How do I go about publicly giving my advice to local business
owners in Chandler?"

Persona 2: "The Business Owner"

Fictional Name: Timothy King

Job Title/Responsibilities: Local Business Owner

Demographics:  35 years old

 Single

Goals and Tasks: He is a business owner looking to become a future member of the
Chandler Chamber of Commerce and would like to:
 Apply to be a member.

Page 2 of 39
 Daniel Joji Koons CSE463: Project Report

 Find out about the education resources the chamber has for
business owners.

Environment: He is a very optimistic person, who wishes to take his business to the next
level via promotion with the chamber and have more

Quote: "I hope the chamber can bring me more success for my business!"

Persona 3: "The newbie"

Fictional Name: Keith D. Nicolas

Job Title/Responsibilities: New Resident of Downtown Chandler

Demographics:  Single
 40 years old

Goals and Tasks: He is a new resident to the Chandler community and would like to find
out more information on what type of business are around the area, as
well as restaurants, bars, entertainment, public events, etc. He would like
to know what the chamber can offer him in any of these topics.

Environment: Very relaxed environment, as he just moved to the area. Generally, likes
to know what types of businesses are around his new residence. He is
looking for things to do around the area for after work entertainment as
well as goods and resources these business offer.

Quote: "I'd like to know what downtown Chandler has to offer"

Task Analysis Tools

The tools I had used for analyzing this website were a heuristic analysis as well as a cognitive walk
through. Since the site's functionality is minimalistic and considered a source for information, these
evaluations seemed the most appropriate to see how well-designed the site is. The site scored a
substandard rating of 55 on the heuristic evaluation. The poor rating is due to the fact that the site's
user interface is a little sporadic, the design does not give cohesion to the site, and there is a large
amount of disorganization in links, titles, and functions. Most of the tasks chosen for this project deal
with these problems, and the prototype will correct these problems, but may or may not slow down the
typical user's efficiency. The tasks involve the user looking up different bits of information, such as
applications or events, as well as the fill-out-form to join the chamber as a member.

2.2 Task #1
2.2.1 Task Detail #1
The user will need to locate the "member login" function on the home page, enter the correct
information to the corresponding text boxes (the user will be given a fake username and password for

Page 3 of 39
 Daniel Joji Koons CSE463: Project Report

the original design and prototype design; however, the user will not have to truly log in to either site),
and proceed to click login.

2.2.2 Task #1 Analysis

This first task is very simple and is more of a test of the original design's UI. Because of the way the
homepage is setup the login is placed in a poor position, and makes the user search around the page,
rather than just simply having it at the top of the page, next to the "become a member" button.

2.2.3 Task #1 Discussion

This task could be more methodical by having the login button placed in a more coherent position,
such as at the top of the screen, in the nav-bar, next to the "become a member" button, etc. This way,
the user is not too distracted from any of the other functions or buttons the site's homepage contains.
Again, the task is not too tedious or workload-heavy, but rather just seems like a simple fix would help
the sites UI and UX improve slightly.

2.3 Task #2
2.3.1 Task Detail #2
The second is task is also straightforward. The participant must simply become a member of the
chamber by clicking on the big, green "Become a member" button and fill out the following application.
The user will be given a Notepad or TextEdit file with pre-typed information and, without copy and
pasting, must fill out the form with the information, however, the participant will not have to click the
submit button at the end of the application.

2.3.2 Task #2 Analysis

This task is focused on both the user experience of filling out the application, and a test of
efficiency on the length of time it takes to fill out the application. When the application is shown to the
user, it is a long, extensive fill-out form that takes a few scrolls of the mouse, or couple presses of the
tab button, to get through in a sufficient amount of time. The main worry for this task is that if a user
were to make an error, they would have to scroll all the way up to the error's source, and the fact that
the user may be overwhelmed by the amount of information the chamber is asking for.

2.3.3 Task #2 Discussion

A possible solution for this task would be to implement a step-by-step process for the application,
as well as a progress bar to give the user a sense of confirmation that they are progressing through the
application. There would be a step for the address, username, and business topics so that the user could
focus on each section without distractions or any feeling of being overwhelmed. While this idea sounds
like it would cater to the user for errors and ease-of-use, it may slow down the time it takes to fill out
the entire application due to multiple pages.

2.4 Task #3
2.4.1 Task Detail #3
In the third task, the user is asked to navigate the site to find the Chamber Business Calendar and,
without clicking the Events List View button, find the Wednesday Lunch Leads Group Meeting on the 6th
of March in 2019. This task's focus is the efficiency of the calendars functionality. The calendar can be
found by hovering the mouse over the "events" option on the nav-bar and clicking "chamber business

Page 4 of 39
 Daniel Joji Koons CSE463: Project Report

calendar". The page will open to a calendar, where the only options are to click the left arrow, which
shows the previous month and its corresponding events, or the right arrow, which shows the next
month.

2.4.2 Task #3 Analysis

This task shows how inefficient the calendars functionality is. There are no options to choose the
separate day, week, month, or year on the calendar, and the user must continuously click on the right
arrow to get to the tasks set date. One option is to click the Events List View button, but the user has to
type in the date in the search boxes to get to their specific date. If the user clicks on the tiny calendar
icons next to the search boxes, the user is given a plainer version of the main calendar.

2.4.3 Task #3 Discussion

This task could be easily improved by allowing the user to have drop down menus to specify the
date, month, and year to find specific events instead of making the user click the Events List View and
leaving the page, or making the user click continuously through the calendar. Also, clicking on the Events
tab on the nav-bar brings the user back to the homepage instead of a separate page for events,
requiring the user to hover over the tab and find the Chamber Business Calendar.

2.5 Task #4
2.5.1 Task Detail #4
In the fourth task the user is asked to find the Leadership Institute application on the chambers site.
Because of how hidden the application is, and to increase the pressure of the user, they are given one
minute to navigate the site on their own before given help. This allows the user to truly focus on the
design and placement of certain buttons or pages of the site. The correct path to find this application is
to hover over the About Us tab, click the Community Foundation button, and the only post on that page
is the Leadership Institute Info and application.

2.5.2 Task #4 Analysis

This task can seem almost impossible because the application is so hidden in the site. There are no
hints or clues as to what the Leadership Institute is, and the only way to find the application or any
information as to what the Leadership Institute provides is by following the path in the previous task
detail. A new user of the site would never know the Leadership Institute existed unless told from an
employee, told from a member, googling, or by randomly navigating through the site.

2.5.3 Task #4 Discussion

The biggest solution that could be implemented is getting rid of the title of Community Foundation
and renaming it to Institute Leadership. Since there is only one sentence about the idea of a "community
foundation" on this page, it seems useless to have a page dedicated to that idea. A simple renaming of
the page and getting rid of the text and replacing it with full information, videos, pictures, events that
deal with the Leadership Institute would benefit this design significantly.

2.6 Task #5
2.6.1 Task Detail #5
The fifth and final task is also a navigation task. The user is asked to find the application to become
a speaker (not register) for the Women in Leadership event. Like the previous task, the user must

Page 5 of 39
 Daniel Joji Koons CSE463: Project Report

navigate the site to find the application and will be given one minute to try and find it on their own
before help is given. This differs from task four because there are pages dedicated to this event and is
one of the chambers most well-known events. To find the application, the user must navigate to the
"signature events" page under the events tab and look under the Women in Leadership title to find the
application to become a speaker.

2.6.2 Task #5 Analysis

This task is catered to any members or users that would like to partake in the chambers events or
contribute more to the chamber and their community. The main problem with this task lies in the fact
that there is a button under the "Events" tab that explicitly says, "Women in Leadership". This is a step
up from the previous task, but once the user clicks this button, they are taken to a page where the user
can only find the venue for the event, register for the event, and find a fill out form titled "Interested in
Women in Leadership's Committee" to submit. The application to become a public speaker for the event
is not found on this page.

2.6.3 Task #5 Discussion

The simple solution would be to create a link to the application on this page. The signature
events page could simple be hotlinks to the separate pages on the site, rather than adding more to the
nav-bar that could potentially distract or impede the users path to complete the task. The main
demographic for this task are female users who are local business owners in the area that wish to either
give or gain advice from these events through the chamber, but if the site acts as a negative catalyst,
some of the users may lose interest or simply not find the application.

3 Prototype and Design

3.1 Overview of Prototype and Design Features
The main problems and challenges with the sites original design deal with the disarranged nav-
bar, inefficient functions, and most of the sites UI not working as intended or provide a poor user
experience. There are many design errors that are not specifically mentioned in the five tasks listed
above, such as hotlinks not working (for example: at the bottom of the screen there is a link that is titled
"dining" however, when clicked, it directs the user to a real estate page). The prototype will condense
the nav-bar to help new users be able to navigate the site in an efficient and coherent manner, provide
options to the user for better functionality, give the user a clean design that does not look outdated, and
overall improve the experience for members and new users.

3.2 Task #1
3.2.1 Task #1 Design
In the original design of the website the user would be brought to the homepage and the login
screen would not be at the top of the screen. This would make the user scroll down passed numerous
widgets and functions to find the login widget, which was placed in the middle-area of the homepage.
Instead of having the user scroll down to find the login screen, this design would implement a top
navigation bar with important features, such as the login button.

Page 6 of 39
 Daniel Joji Koons CSE463: Project Report

3.2.2 Task #1 Design Justifications

Some of the results I received from the evaluation dealt with the login screen. It is a strange design
flaw to have the login screen nowhere near the top of the screen and having the user search for it
instead. This new design will give the site a more clean and organized look, as well as giving the login
button a convenient and common placement.

3.2.3 Task #1 Prototype

While the old design did not have a separate page for a login, it is arguable that having the login
screen at the top of the page where the user may commonly look for the login is more beneficial than
having the user look for the login info in the middle of the homepage. In the prototype there is a
navigation bar added to the top of the page with the login button, as well as other important buttons
that are meant to grab the users' attention. Here is an image from the original design:

(it should be noted that the web browser was zoomed out to 67% to capture the full page). As can be
seen from this image, there is only one nav-bar, which does not contain a login button. The only way to
login to the site is from the homepage.

Here is the image from the prototype:

Page 7 of 39
 Daniel Joji Koons CSE463: Project Report

And the actual login page:

This design is much more sleek and organized, and is in a more convenient placement than in
the middle of the page.

3.2.4 Task #1 Prototype Rational

This design is much more organized than the original design. It allows regular and new users to find
and login to the site in a more sufficient and timely manner, as well as making the sites UI look clean and
organized. It also allowed for a completely separate nav-bar that follows all pages so that a user can log
in from any page of the site instead of only the homepage.

3.3 Task #2
3.3.1 Task #2 Design
The task was to improve the “become a member” application that the site has. Again, like the login
button, the original design only had the become a member option on the homepage. When the button
is clicked the user is brought to an extensive and long fill-out-form. The user must fill out all the
information and does not need to click submit to complete the task. Taking into account the length of
the fill-out-form, the prototype design has separated the form into separate sections and pages, and
contains a progress bar to show what section the user is currently on, how many more sections are left,
and how many sections are completed.

3.3.2 Task #2 Design Justifications

The justifications for this design are due to the feedback I received from the heuristic evaluation.
Instead of filling out a long-drawn-out form, the form will be broken up into sections to not give the user
the feeling of being overwhelmed, as well as accounting for any errors the user makes. In the original
design, if there was an error, the user had to scroll to where the error was made. In this new design, the
error only lies in the section that the user is currently in, allowing them to take their time to make sure
they did not make mistakes.

Page 8 of 39
 Daniel Joji Koons CSE463: Project Report

3.3.3 Task #2 Prototype

The redesign of this task was creating a section-by-section format for the “become a member”
form. The overall information is still the same, but the prototype breaks the information into certain
sections so that the experience is more organized and clear. There is also a progress bar that gives
confirmation to the user that they are indeed progressing and completing the task.
Here is the form from the original design (zoomed out to 25%):

And here is a section from the prototype design:

As you can see, the original design’s form is a bit over-bearing while the new design is much more
sleek and organized. The user simply inputs their information, and clicks the “next” button at the

Page 9 of 39
 Daniel Joji Koons CSE463: Project Report

bottom of the screen (not shown in picture). Instead of worrying about submitting the form with any
errors, the prototype design gives the user a condensed version of each section so that they may focus
on that particular sections information.

3.3.4 Task #1 Prototype Rational

The purpose of this redesign was to improve the UI and UX of the “become a member” form, one of
the sites most important and well-used features. With the improved prototype design, the idea is to
reduce the cognitive load on the user, and allow them to see progression through the form, and account
for any errors in any section.

3.4 Task #3
3.4.1 Task #3 Design
This task asks the user to find the Chamber Business Calendar on the site, and navigate to the
Wednesday Lunch Leads Group event on March 6th of 2019. While the event is not too important in this
task, the calendar functionality is. The only options on the original design are to click two left and right
arrows, which only progress the calendar by the next or previous month. The time it takes for a user to
click through all the way to the next year is very illogical, and requires a simple fix. A better design is to
add an option to change the year directly.

3.4.2 Task #3 Design Justifications

Adding the option to directly change the calendars year and month saves the user time if planning
to attend an event in the future. The time difference between the original sites design and the
prototypes design is significant.

3.4.3 Task #3 Prototype

The prototype has an effective design as in the calendar smoothly transitions through the months,
as well as years, instead of taking the user to a new page each time. Along with changing the year, the
transitions help the user save a large amount of time.
Here is the original calendar design:

Page 10 of 39
 Daniel Joji Koons CSE463: Project Report

And here is the prototype design:

The prototype version of the calendar is much more timely and satisfactory, looks modernized, and
allows the user to navigate to specific dates easily.

3.4.4 Task #3 Prototype Rational

The year functionality is a simple fix, but creates significant differences and changes from the
original design. Users will no longer have to click a the “Events List View” to be able to get full
functionality over a calendar, and can now navigate via year and month. If the specified date was further
than the tasks, the user may feel annoyed clicking through each month, or just use a search engine to
find the date.

3.5 Task #4 and 5

3.5.1 Task #4 and #5 Design
These next two tasks essentially have the same idea- simply fetching information from the site. The
idea of these tasks is to test the sites navigation bar, and the coherence and comprehensiveness of
placement of specific information. This design would need a better version of their nav-bar, which in the
prototype there is now two nav-bars, one at the top of the screen and right below it. While the direct
task does not mention the nav-bar, the intention is for the users to get acclimated with the nav-bar in
both designs.

3.5.2 Task #4 and #5 Design Justifications

The heuristic evaluation results showed how poor the navigation bar was in the original design, to
the point where most of the information was very hard to find. The information for the next two tasks
are both applications that returning users or new users may want to find more about but if they are
trying to access the information from the homepage, it is almost impossible to find unless randomly
clicking pages.

3.5.3 Task #4 and #5 Prototype

The new prototype shows the improved second navigation bar. It is more organized, condensed,
and coherent than the original design. The user simply must make an educated guess as to where the
applications or information is and, using the updated nav-bar, find the information.

Page 11 of 39
 Daniel Joji Koons CSE463: Project Report

Here is the original designs nav-bar:

And the prototype nav-bar(s):

The new nav-bar makes much more sense in coherence and intuition, as the titles of the menus are
matched with the correct information and does not send the user on a clicking-adventure or ditch the
site completely for a search engine. It also includes the site's social media, as well as a search option for
easy navigating through the site (participants in testing will not be asked to use the search function).

3.5.4 v Task #4 and #5 Prototype

With the help of a top nav-bar, the information can be nicely spread between the two nav-bars,
reducing the user’s cognitive load. It is much more efficient to have a condensed and smart nav-bar than
to overload a singular nav-bar full of information that could distract the user from the task, or mislead
them to unneeded information.

4 A/B Testing
4.1 Participants
The participants for this experiment will be college students that are between the ages of 18 and
22. The participants were recruited by asking friends, as well as other classmates, to participate in the
experiment. The participants were not compensated for their time except for a good pat on the back
and a thank you.

4.2 Scenarios
The following scenarios are the situations where a user will complete the type of tasks that have
been chosen for this experiment. Instructions and information that the users are given can be
found in

4.2.1 Task #1
The user is a newly-joined member of the chamber, and would like to login to his member account
to view options and information that only members receive. They would like to login to the site as soon
as the homepage is loaded up. The user would input their login info in the correct text boxes and does
not need to click submit to complete the task. The user will be directed back to the homepage after
completion.

Page 12 of 39
 Daniel Joji Koons CSE463: Project Report

4.2.2 Task #2
The user is a local business owner residing in downtown Chandler and would like to join the
chamber as a member for promotion and funding for events. The user would find the “become a
member” form on the site and fill out the correct information in the text boxes. The user does not need
to click submit to complete this task. The user will be directed back to the homepage after completion.

4.2.3 Task #3
The user is already a member of the chamber or a returning user of the site, and would like to find
out the date of an event in the future (the next year). The user would need to locate the Chamber
Business Events calendar on the site from the homepage. This information is easy to find by clicking
through the months, but the year needs to be able to be changed as well. An option is to give the year
arrows, so the user can navigate through the calendar in an efficient manner. The user will be directed
back to the homepage after completion.

4.2.4 Task #4
The user will be given a 1-minute time limit to complete the task on their own. The user is either a
new user, returning user, or member that wishes to access more information on the Leadership Institute
that the chamber offers. The user will need to go through the nav-bar and click what they think are the
right buttons to find the application. Once the user finds the application, they will click on the link to the
application. The user will be directed back to the homepage after completion.

4.2.5 Task #5
The user will be given a 1-minute time limit to complete the task on their own. The user is either a
new user, returning user, or member that wishes to access more information on the Women in
Leadership Speaker Series the chamber offers. The user wishes to register as a speaker for the event,
not register to attend the event. The user will, again, need to go through the nav-bar and click what they
think are the right buttons to find the application to be a speaker. Once the user finds the application,
they will click the link to the application. The user will be directed back to the homepage after
completion.

4.3 Equipment
The users will be using the original website on a laptop that is rented from the digital culture
equipment room. The prototype will only be tested on the personal desktop of the researcher. The
laptop is running OS High Sierra and the web browser that the users will use to complete the tasks will
be Google Chrome. The screen size for the laptop is 15.4” and the resolution is 2880 x 1880. The
personal desk top will have two monitors, one with a screen size of 21.5" and a resolution of 1920x1080,
the other will be 24” and the resolution is also 1920 x 1080. The second monitor will have the login info,
as well as the "become a member" application info. Each computer will have a wired mouse to help with
the differences between resolutions. The Chrome extension “Screencastify” will be running to record
the tests and actions. On the personal desktop only the monitor with the prototype design will be
recorded. An iPhone will be used to time the last two tasks before the user is given help. Audio and
Webcam will not be recorded to help keep all users anonymous.

Page 13 of 39
 Daniel Joji Koons CSE463: Project Report

4.4 Subjective Metrics

The users will be given a background questionnaire as well as a post session questionnaire. The
background questionnaire will be used to see if the site appeals to new users, find out what
demographic the site caters to, find out if the site is easier to use if the user has knowledge of what a
chamber is, and if the user can find themselves using a site like this regularly. The background
questionnaire will not be scored as it is not designed as a quiz. The post questionnaire will be a System
Usability Scale Test and will be used to get feedback on the user’s satisfaction with completing the tasks
on the original and prototype design. The post session questionnaire will provide results and feedback of
either designs strengths and weaknesses.
Post Session Questionnaire Score – Original Design (A) :

Post Session Questionnaire Score – Prototype Design (B) :

Page 14 of 39
 Daniel Joji Koons CSE463: Project Report

The bar graphs are the total scores of the system usability scale test/post session questionnaire
given to the participants of both the original design and prototype design. The scores were much higher
on the prototype design, showing that the improvements made based on the flaws on the original
design did in fact benefit the user experience. The original design caused some annoyance and
frustration in the participants, which felt the website was very complex and hard to navigate through
and was incomprehensive. The scores and feedback from the participants of the prototype design stated
that it was much more organized and easier to use, and the functionality was effective and made the
experience smooth.

4.5 Quantitative Metrics

The tasks will all be timed, as that is the biggest metric that the experiment and results are based
on. The completion time of the tasks between the original design and the prototype design will be
compared. The users’ errors will also be counted to see if the design of either the original or prototype
create or reduce the amount of distractions or misguided actions the user has. Each time a user clicks a
link or button that does not count towards the task it is counted as an error.

4.6 Test results

The main test for this project was to see if the tasks could be completed in a more, timely manner
in the prototype design than the original sites' design. The average time to complete the tasks in the
original design was 597 seconds, and for the new design it was 357.4 seconds. The extreme difference
between average times already gives an idea of how much improvement the prototype had caused.
Original design task completion time graph-

Page 15 of 39
 Daniel Joji Koons CSE463: Project Report

Prototype design task completion time graph-

Since the overall time it took to complete all the tasks are so vastly different from the original
design and the prototype, it seemed appropriate to list all the participants and their times in each

Page 16 of 39
 Daniel Joji Koons CSE463: Project Report

design. While they are included in the graph, it should be noted that both participant 1 and 4 testing the
old design stopped their recording by mistake in the middle of their recordings, resulting in some
incorrect data.
The participants were accounted for all errors they had made. If the participant clicked an
option or button that did not lead to the progression of completing a task they were given one error.
The following tables are the participants and the error count for each task. The first table is for the
original design and the second table is for the prototype design.

Original design error table-

Participant Task 1 Task 2 Task 3 Task 4 Task 5

1 0 0 0 1 3

2 0 0 0 3 3

3 1 0 1 2 4

4 1 0 0 0 1

5 2 0 1 4 3

Prototype design error table-

Participant Task 1 Task 2 Task 3 Task 4 Task 5

1 0 0 0 0 0

2 0 0 0 1 0

3 0 0 0 1 1

4 0 0 0 2 0

5 0 0 0 2 0

As can be seen, the original design had many errors across the board for almost all tasks and all
participants. It seems that only task 2, filling out the "become a member" form, was the only task in
both designs that did not receive a single error, however, this was expected as both sites clearly showed
a button to become a member at the top of the screen, and the following actions were simply filling out
the form with pre-typed information. Task 4, finding the Leadership Institute application. was also the
most difficult to complete, as the original site did not explicitly have a tab or button that lead to the
application. The case seems to be the same for the prototype design, although there are less errors,
perhaps caused by the user not reading through the nav-bar carefully, or the design not meeting up to
standards.

Page 17 of 39
 Daniel Joji Koons CSE463: Project Report

A lot of the participants for the original design expressed frustrated emotions or statements,
explaining how the site did not meet their standards in terms of organization, placement, usability, etc.
The participants for the prototype did not express too many emotions, only stoic faces and confirmation
terms (i.e. "Yes", "okay", "alright").

5 Conclusions
5.1 Discussion of Results
The results from the prototype design were very significant in comparison to the original design.
The prototype was built with the expectation to repair and refine most of the original sites' design flaws
or annoyances, which, from the graphs, not all the flaws were completely fixed. Some factors play into
these results from the testing, such as the idea of mis-clicking. In the prototype, only specific links and
buttons led to another page, while the other buttons were place holders. Some participants during the
testing would click on the buttons before the end of the test and would figure out that since those
buttons did not lead to a page on the site, they must not have a use in any of the tasks, reducing their
overall completion time. If the prototype site was fully designed with the idea of completely replicating
the original design, the participants would have an increased completion time. The prototype did
improve the website, but there were still a few UI aspects missing, such as the "become a member"
button blending in to the top nav-bar when it should be a call-out button to differentiate from the
others. Also, some of the participants had stopped recording mid-testing, which was taken into account
for the timing results.

5.2 Lessons Learned

There were a lot of lessons to be learned while doing this project. For example, the experiment
allowed the participants of the new design to use a personal desktop with two monitors, wider screen,
and a larger resolution, while the old design only had a 15.4" 2880x1880 resolution laptop (MacBook
pro). The tasks seemed to go by much quicker on the desktop than the laptop, showing that the
hardware difference played a major role in the experiment. For any future experiments, I would like to
have all the hardware alike and identical, so that they only play a side role in the testing. I also learned
about numerous techniques and cognitive skills web designers use to give the user immediate
recognition to buttons, widgets, tabs, options, etc., all while keeping the format clean and organized.

5.3 Conclusion
The prototype proved to be the better of the two designs, allowed the tasks to be completed in a
sufficient time, repaired most, if not, some of the flaws the original design contained, and gave a better
insight on how a site for a chamber of commerce should be presented. Of course, the prototype is not
perfect, and would still need some UI fixes on the design, but the limited repairs made did enhance the
user experience. The experiment is considered successful, since most of the errors from the original
design were eliminated, and the tasks were completed in a significantly shorter amount of time.

Page 18 of 39
Daniel Joji Koons CSE463: Project Report

6 Appendixes
6.1 Heuristic Evaluation

Page 19 of 39
Daniel Joji Koons CSE463: Project Report

Page 20 of 39
Daniel Joji Koons CSE463: Project Report

Page 21 of 39
 Daniel Joji Koons CSE463: Project Report

6.2 Cognitive Walk-through

Tasks:
1.) Login to the site as a member.
Steps:
1. Enter in the URL of the (chandlerchamber.com) or go to the homepage of the site.
2. Scroll down to the middle of the page and find the login widget.
3. Enter the correct username and password.

1 2 3

Know the action? Yes Yes Yes

See the action? Yes Depends/Maybe Yes

Understand the Yes Depends/Maybe Yes

control?

Page 22 of 39
Daniel Joji Koons CSE463: Project Report

See the feedback? Yes No Yes

Recommendations:
A suggestion for the site to have would be to place the login widget near the top of the screen
next to the “become a member” button instead of requiring the user to search around the
homepage to find it.

2.) Click on the “become a member” button and fill out the form.
Steps:
1.) Find the “become a member” button at the top of the screen.
2.) Fill out the form with the correct information.

1 2

Know the action? Yes Yes

See the action? Yes Yes

Understand the control? Yes Yes

See the feedback? Yes No

Recommendations:
Since step 2 is most of the task, the recommendations mostly deal with the length of the form.
There is a lot of information to fill in on this form, which could cause the split-attention effect of
the user. Splitting the sections up in separate pages could reduce the effect. The user sees no
progression of the task except for the text fields that are filled or unfilled, perhaps a progression
bar would fix this design.
3.) Using the Chamber Business Calendar, locate the Wednesday Lunch Leads Group event that
falls on the 6th of March in 2019 (without using Events List View).
Steps:
1.) Find the "Chamber Business Calendar" under the Events tab.
2.) Clicking the right arrow, keep clicking until the correct day, month, and year are in place.
3.) Click on the event.

1 2 3

Know the action? Yes Yes Yes

See the action? Yes Yes Yes

Understand the control? Yes Yes Yes

Page 23 of 39
 Daniel Joji Koons CSE463: Project Report

See the feedback? Yes Yes Yes

Recommendations:
The calendar is very tedious to use in its current state unless the "Events List View" is clicked. A
quick fix would be adding a function that allows the user to change the current year, so the user could
access a future date in a short amount of time.

4.) Find the Leadership Institute application for the program that the chamber offers.
Steps:
1.) Locate the "community foundation" button under the "About Us" tab on the nav-bar.
2.) Click on the application link.

1 2

Know the action? No Yes

See the action? Yes Yes

Understand the control? Yes Yes

See the feedback? Yes Yes

Recommendations:
While the cognitive walkthrough shows the answer "yes" for nost of the questions, the
application is quite hard to find without any sort of help or keen intuition. Since it is the only application
or program mentioned in the "community foundation" page, the Leadership Institute should have its
own separate page and the community foundation page should be deleted or reworked.
5.) Find the application to register as a public speaker at a Women in Leadership event (not
simply register for the event),
Steps:
1.) Locate the "Networking Events" page under the "Events" tab.
2.) Scroll down to the Women in Leadership Event description.
3.) Click on the link to the public speaker application.

1 2 3

Know the action? No Yes Yes

See the action? Yes Yes Yes

Understand the control? Yes Yes Yes

Page 24 of 39
 Daniel Joji Koons CSE463: Project Report

See the feedback? Yes Yes Yes

Recommendations:
The same as in task 4, this application is hard to find. There is a separate page under the "Events"
tab for Women in Leadership, however this page only links to the venue, a registration for the event,
and a fill-out-form for more information about the event. There are no links to the application to be a
speaker for the event on the actual event's page. Instead the user must go to a separate page and find
the link. The simple fix would be to add the link to the actual event's main page to simplify the task.

Page 25 of 39
Daniel Joji Koons CSE463: Project Report

6.3 New GUI snapshots

Page 26 of 39
Daniel Joji Koons CSE463: Project Report

Page 27 of 39
Daniel Joji Koons CSE463: Project Report

Page 28 of 39
Daniel Joji Koons CSE463: Project Report

Page 29 of 39
 Daniel Joji Koons CSE463: Project Report

6.4 Instructions for participants

After each task, please return to the homepage of the site.
Task 1.) Find the login function and login to the site. Do not press submit or attempt to actually
login to the site.

Task 2.) Locate the "become a member" button and fill out the form, do not press submit.

Task 3.) Locate the Chamber Business Calendar on the site and click on the Wednesday Lunch
Leads Group Event that is happening on the 6th of March of 2019. Do not click on "Events List
View".

Task 4.) Locate the application to the Leadership Institute program that the chamber offers.
Click on the link to the app once it appears.

Task 5.) Locate the application to register as a speaker for the Women in Leadership event. Do
not click the link to register for the event. Click on the link to the app once it appears.

Page 30 of 39
 Daniel Joji Koons CSE463: Project Report

6.5 Researcher guidelines

Prelude:
Thank you for participating in my experiment. You will be tested in completing five separate tasks
on either the original design or prototype design for a website. The website is for the Chandler Chamber
of Commerce with information for local business owners, new users, and new residents of Chandler. The
tasks are all information based, meaning that the user must navigate through the site finding or filling
out pieces of information on the site. Before testing, you will be given a short background questionnaire
testing your knowledge of what a chamber is, does, and their online record. The second part of the
experiment will be the actual experiment. Your screen will be recorded; however, your audio and
webcam will not be used. The tasks will be read aloud to you at the beginning, and again after you finish
each task. You will be given a one-minute timer for the last two tasks. When the time is up, I, the
researcher, will step in to help. After the testing, you will be given a post-session questionnaire asking
about your experiences, feedback, and comments of the site's design and your completion of the tasks.
All information shared will be kept confidential and anonymous and will be removed once the
experiment is over. Any questions? Thank you.
Background Questionnaire:
The purpose of this questionnaire is to see your knowledge and understanding of what a chamber
of commerce is and its role in society, both physically and online. If you do not wish to answer the
question, or simply do not know the answer, please answer with "c". It is okay if you do not know what
the answers are to the questions, you can still participate in the experiment. Please answer to the best
of your ability, and if you are confused by a question, do not be afraid to ask for help. Thank you.
Experiment:
You will simply complete the tasks in order, one-by-one. If you cannot complete the task, please
ask the researcher (me) for help. The last two tasks will have a time limit of one-minute due to the
difficulty of the tasks. It is okay if you do not complete the last two tasks in under one-minute. After
each task, please go back to the homepage before starting the next task. When done with all tasks,
please raise your hand for further instruction. Thank you.
Post-Session Questionnaire:
Thank you for participating in the experiment. Your participation and helpfulness are very
beneficial to my research. To get more personable answers, please answer this post-session
questionnaire. If the questions are confusing, do not be afraid to ask me for help. All your information
will remain confidential and anonymous. Thank you for participating in my experiment again. Have a
nice day!

6.6 Background questionnaire

Answer the following questions:
1. Do you know what a chamber of commerce is?
a. Yes
b. No
c. Does not apply

2. Have you ever visited a chamber of commerce in person?

Page 31 of 39
Daniel Joji Koons CSE463: Project Report

a. Yes
b. No
c. Does not apply

3. Have you ever visited a website belonging to a chamber of commerce?

a. Yes
b. No
c. Does not apply

4. A chamber of commerce is known to promote local businesses. If you owned a local business,
would you apply to become a part of a chamber?
a. Yes
b. No
c. Does not apply

5. Do you believe a bad web design for a chambers website would hinder the experience of a new,
future member?
a. Yes
b. No
c. Does not apply

6. Do you believe a bad web design for a chambers website would hinder the experience of a
regular, current member?
a. Yes
b. No
c. Does not apply

7. Do you believe a good web design for a chambers website would improve the experience of a
new, future member?
a. Yes
b. No
c. Does not apply

8. Do you believe a good web design for a chambers website would improve the experience of a
regular, current member?
a. Yes
b. No
c. Does not apply

6.7 Post-session questionnaire

1. I think that I would like to use this website frequently.

Page 32 of 39
Daniel Joji Koons CSE463: Project Report

1.) Strongly disagree

2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
2. I found the website unnecessarily complex.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
3. I thought the website was easy to use.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
4. I think that I would need the support of a technical person to be able to use this website.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
5. I found the various functions in this website were well integrated.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
6. I thought there was too much inconsistency in this website.
1.) Strongly disagree
2.) Disagree
3.) Neutral

Page 33 of 39
 Daniel Joji Koons CSE463: Project Report

4.) Agree
5.) Strongly agree
7. I would imagine that most people would learn to use this website very quickly.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
8. I found the website very cumbersome to use.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
9. I felt very confident using the website.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree
10. I needed to learn a lot of things before I could get going with this website.
1.) Strongly disagree
2.) Disagree
3.) Neutral
4.) Agree
5.) Strongly agree

6.8 YouTube Links

Old Design Links:
Participant 1 – pt. 1 https://youtu.be/y_38sFkdGWg
pt. 2 https://youtu.be/9GucyDtjFdQ
Participant 2 - https://youtu.be/DAryyhU28zs
Participant 3 - https://youtu.be/nTQdww2dSMQ

Page 34 of 39
 Daniel Joji Koons CSE463: Project Report

Participant 4 – pt. 1 https://youtu.be/K1P_65LMvps

Pt. 2 https://youtu.be/fUnXbA5sNeg
Participant 5 - https://youtu.be/te3r-ro_MqE

New Design Links:

Participant 1 - https://youtu.be/d8uIeiDviDY
Participant 2 – https://youtu.be/IVbx1Y4FDpA
Participant 3 - https://youtu.be/4xMefs3W6aE
Participant 4 - https://youtu.be/3sVKQBVQBoY
Participant 5 - https://youtu.be/xO4u6dMyJhI

Page 35 of 39