ACVREP Case Study

Online Testing vs. Paper and Pencil - The Difference is Night and Day

Overview
This case study details needs, process for change, results and lessons learned
that one organization experienced when moving from traditional Paper and Pencil
Testing to Online Testing.

Background

The Academy for Certification of Vision Rehabilitation and Education Professionals

(ACVREP) is a private, not-for-profit 501 c6 status certification board. ACVREP
certifies over 3,000 professionals within the United States and Canada working in
the vision rehabilitation and low vision services field. ACVREP offers the following
three (3) professional certificates:

Certified Low Vision Therapist® (CLVT®)

Certified Orientation and Mobility Specialist® (COMS®)
Certified Vision Rehabilitation Therapist® (CVRT®)

As part of ACVREP’s certification requirements, the individual candidate must sit

for and pass the applicable ACVREP certification exam (CLVT, COMS, or CVRT)
for the professional discipline in which the candidate is seeking to obtain
certification. Historically, the administration of ACVREP certification exams was
facilitated through paper and pencil format.

www.KryterionOnline.com
With the introduction of new executive leadership in April 2008, ACVREP
completed an organizational effectiveness study. One of the areas of significant
need identified within the study was the administration of the certification exams
via paper and pencil format.

Areas of Need

The organizational effectiveness study

completed by ACVREP identified the ACVREP Areas of Need
following concerns related to the
administration of certification exams via paper
and pencil format: x Exam Scheduling
x Distribution of Exam
Materials
Exam Scheduling
x Cost to Candidate
ACVREP certification exams are
administered at various testing centers
x Reporting of Results
located in the U.S. and Canada. In x Accessibility of
accordance to organizational policy and Materials
procedure, ACVREP staff would coordinate
the scheduling of exam dates and times,
which resulted in the need to support several points of contact between the testing
center and the individual candidate. This action represented a significant burden to
ACVREP staff. To further complicate the matter, if an individual candidate had to
reschedule the administration of the exam, the entire process would be repeated,
resulting in additional stress and burden on staff and candidate. Furthermore, this
action served to reduce the amount of time available to staff to answer stakeholder
inquiries regarding certification programs and related requirements, resulting in
poor responsiveness to stakeholders’ needs and requests and decreased
customer satisfaction and autonomy in the scheduling process.

Distribution of Exam Materials

Given the paper-based format of the exam, ACVREP was responsible for the
distribution of hard copy exam materials. As per organizational policy and
procedure, ACVREP would distribute materials via Fed-Ex delivery. Given the
physical transport, several concerns were noted. First, the increased risk relative to
test security during transport. Second, increased burden on staff in terms of
preparing materials, quality checks, and coordinating scheduled deliveries and
pick-up of exam related materials. Lastly, a lack of control in stabilizing costs
related to the transport of exam-related materials.

www.KryterionOnline.com Page 2 of 7
Relative Costs to the Candidate
In addition to paying the certification exam fee, the candidate was also required to
pay for the costs of proctoring the exam. In review of the organizational
effectiveness study, it was found that proctoring costs varied widely depending
upon geographic location and that costs were increasing at an unprecedented rate.
Furthermore, there was no method available to ACVREP to help stabilize costs to
the candidate relative to proctor fees given the current method of exam
administration (including format).

Reporting of Exam Results

Given organizational policy and procedure, the reporting of exam results to
candidates for certification would take up to three (3) weeks. This action
represented undue burden on the candidate, resulting in poor customer
satisfaction and delay in obtaining certification.

Accessibility of Exam Materials

Given that ACVREP provides professional certification to individuals working in the
vision rehabilitation and low vision field, it is critical to recognize that many
candidates for ACVREP certification are blind or visually impaired. Given the
paper-based format of the exam, as per organizational policy and procedure,
ACVREP was able to provide a Braille format, a large print format, and or a test
reader option to accommodate test-taker needs. However, these formats were
often identified as cumbersome to the test taker due to the volume of paper need
ed to support the Braille and large print formats, in consideration of exam
administration time requirements.

In reviewing the findings of the study with ACVREP’s testing vendor (Prometric),
the executive leadership of ACVREP was unsatisfied with the recommendations
provided, as said recommendations did not take into consideration ACVREP’s
size, significant increases in cost to administer the exam, or the needs for
individual test takers, including individuals who are blind or visually impaired.

www.KryterionOnline.com Page 3 of 7
Resolution

Beginning in 2009, the executive leadership of ACVREP began to research options

to improve the organization’s exam administration methodology (format,
procedures, etc.) to address the specific areas of
need identified through the organizational
effectiveness study. After extensive research of “... the utilization of the
and discussion with several potential vendors, Webassessor™ online
ACVREP selected Kryterion™ as its vendor for testing portal has allowed
test development and delivery. The selection of ACVREP to stabilize exam
Kryterion™ was based on the following factors:
related costs for the
The Webassessor™ online testing platform consumer, specifically
allows for candidates to schedule testing dates, proctor costs.”
times and locations. While ACVREP staff is
available to help with the scheduling if requested Garett A. Holm, President
by an individual candidate, this action has of ACVREP
dramatically reduced the time spent by staff
relative to scheduling and coordinating exams for
individual candidates. Furthermore, candidates report to ACVREP that the online
portal is “user friendly” and provides them greater “control” in the process,
specifically if the administration of the exam needs to be rescheduled. Since
moving to the online format, reported customer satisfaction has increased
significantly and ACVREP staff has reported that the scheduling process is now
less burdensome to manage, allowing for more time to respond to stakeholder
questions and requests.

The Webassessor™ online testing platform significantly decreases risks relative to

test security, as it eliminates the need to transport hard copy exam related
materials. In addition, this action allows ACVREP to stabilize costs, given
consistently increasing organizational costs related to the transport of said
materials.

With the costs of proctoring the exam included in the per-test delivery fee, the
utilization of the Webassessor™ online testing portal has allowed ACVREP to
stabilize exam related costs for the consumer, specifically proctor costs.

With the utilization of the Webassessor™ online testing platform, exam results are
reported immediately to the test taker and to ACVREP. This action has significantly
improved processing timelines for certification, which, in turn, increased
stakeholder satisfaction. Furthermore, with the incorporation of said technology,

www.KryterionOnline.com Page 4 of 7
ACVREP now has the opportunity to review its certification policies and procedures
with the intent to quicken the timelines relative to certification. This would not be
possible without the implementation of Webassessor™.

Given the Webassessor™ online testing platforms functionality, accessibility of

exam forms can be supported with zoom text, voice, and other mediums.

Additional factors relative to the decision to work with Kryterion™ include:

x The integration of development and delivery functions on the Webassessor

platform, allowing for the opportunity to reduce costs over time;
x Greater control over the management (form creation, item banking, etc.) of
exam programs;
x The ability to start implementation of the platform quickly (less than a month
in ACVREP’s case);
x Customer support that is responsive to the needs of the organization and
individual test takers.

www.KryterionOnline.com Page 5 of 7
ACVREP Lessons Learned

Given ACVREP’s traditional utilization of a paper and pencil format to facilitate the
administration of its certification exams, ACVREP was placed in a position to
maintain policies and procedures to support said exam format that proved
detrimental to organizational efficiency and stakeholder satisfaction. Given the
costs involved provided by various vendors in switching to an online platform and
the relatively low numbers of exams administered per year by ACVREP, it did not
seem possible to enhance the testing program. In light of these circumstances,
ACVREP learned the following lessons during the process of identifying its
organizational needs and resolving needs relative to the testing program:

Carefully Identify Stakeholder Needs

Carefully identify your organizational and stakeholder needs relative to your testing
program. This process will assist you in articulating program objectives and goals.
Furthermore, without carefully articulated objectives and goals, you will not be able
to communicate with potential vendors or select the best options for your testing
program. Taking this concept one step forward, it is important to make these
needs, objectives and goals transparent throughout the organization, including the
Board and other stakeholders.

Find a Provider That Meets Your Needs

Speaking metaphorically, if you feel like a small fish in a big pond, there are two
options available to you. You can stay in the pond and hope for the best. Or, you
can start to look for a different pond. Your satisfaction (in terms of cost, ease of
administration, security, customer service, etc.) is what matters most. If you feel
that you are sacrificing satisfaction for maintaining something that works at a basic
level, then you are sacrificing your ability to enhance your programs and improve
customer satisfaction. Take the time to identify program needs, articulate program
objectives and goals, research providers, and select a provider that recognizes
your specific needs, understands your objectives and goals, and is willing to work
with you to accomplish said objectives and goals.

Invest Time in the Beginning

Do not be afraid of a little hard work in the short-term to enhance your programs,
as it will save you a lot of work down the road in maintaining operations for
something that may only meet some of your overall program goals and objectives.
The same can be said for start-up costs and long-term costs.

www.KryterionOnline.com Page 6 of 7
Demand Responsiveness
If it takes more than two business days to get a response from your provider, this is
one day too long. Truth be told, testing programs are difficult to manage for the
largest of certifying agencies. For smaller certifying agencies, like ACVREP, the
task can be daunting. Responsiveness of the vendor should be a primary
consideration when selecting a vendor or deciding to maintain the relationship.

Conclusion
The positive results of ACVREP’s decision to move to the Webassessor™ online
testing platform through Kryterion™ were noticed immediately. Garett A. Holm,
President of ACVREP, states, “Prior to making the move to online testing, it
seemed that no one was satisfied with the administration of the exam. Staff
expressed concerns about security and staff time. Candidates also expressed
great frustration with the time it took to schedule the exam and then to receive
exam results. Plain and simple, the program was just not working.” Holm
continues, “Since moving to online testing, all parties have expressed greater
satisfaction regarding the administration of exam. It’s like night and day.”

Written by Garett A. Holm, President of ACVREP, November 2010.

About Kryterion
Kryterion is a full-service test development and delivery company that serves the
Association, Technology and Education markets with world-class secure testing
solutions. Our solutions include the five key elements of certification and testing:
Psychometric services, exam management, security, real time access and test
delivery, all managed through our proprietary Webassessor test development and
deployment system. Our clients test with confidence and convenience.

For more information on how Kryterion can streamline your testing program
contact::

Kryterion Certification and Licensure Sales

Sales@KryterionOnline.com

www.KryterionOnline.com Page 7 of 7