August 27, 2020

Chief Justice Ralph D. Gants, Supreme Judicial Court

Marilyn J. Wellington, Executive Director, Board of Bar Examiners
One Pemberton Square
Boston, Massachusetts 02108

Dear Chief Justice Gants and Ms. Wellington:

We would like to thank the Board of Bar Examiners and the Supreme Judicial Court for their responses
to our petition. We recognize that adapting a complex examination scheme in a short time frame must
demand significant time and resources from both the Board and the Court, and we are grateful for the
hard work of both bodies. Respectfully, we were disappointed that the Court and the Board did not
respond to many of the concerns that we raised in our petition. That being said, we write today for the
limited purpose of addressing the Board’s claims and plans regarding ExamSoft.

In the Board’s memorandum to the Supreme Judicial Court, it stated that ExamSoft is “well tested with
proven success” due to its experience delivering “167,000 synchronous, remotely-proctored exams for
a single high-stakes assessment to examinees dispersed across geographic locations.”

Because ExamSoft has often touted this figure regarding a 167,000-person synchronous remote exam
without providing further detail, we hope that the Board of Bar Examiners is aware that ExamSoft is
apparently referring to a university admissions exam administered by the Kingdom of Saudi Arabia,
during which 8% of examinees experienced technical troubles.1 (We say “apparently” because there is
no public information indicating that ExamSoft has been involved with any other synchronous remote
exams nearing that scale, and we presume that ExamSoft would not have the ability or desire to conceal
involvement with an exam of equivalent size, especially given that ExamSoft claims it as a success.)

If a similar rate of technological issues occurs during the October bar exams, in which approximately
30,000 to 40,000 examinees will be using ExamSoft simultaneously, then 2,400 to 3,200 examinees
will face technological problems. We hope that the Board agrees that this number is unacceptable.

In addition, there are several important factors that distinguish an exam administration in Saudi Arabia
from those in the United States. For one thing, the Kingdom of Saudi Arabia maintains centralized
control over the country’s internet infrastructure, and thus, “we can presume that unlike in the United
States, ExamSoft was not responsible for spinning up the server-side capacity required for test

1
See Education Training and Evaluation Commission, “AI and TM monitored all irregularities and we will reveal
them with the results announcement” (June 12, 2020), https://www.etec.gov.sa/en/Media/News/Pages/Remote-
testing2.aspx?utm_content=132287259&utm_medium=social&utm_source=twitter&hss_channel=tw-123858468.
It is also worth considering that the officially reported rate of 8% does not reflect the full nature and scope of the
technological problems that occurred during this examination date. Unfortunately, ExamSoft has not been transparent
about this exam and many other issues. Further, the exam was administered by an authoritarian government known
for its lack of transparency and its willingness to suppress those who criticize its regime.
administration” in Saudi Arabia.2 The Saudi government also has the ability to curtail non-exam-
related internet traffic during the administration of the exam, a capability that governments in the
United States do not have.3

To the extent that ExamSoft disputes any of the above or wishes to further clarify, examinees would
greatly welcome such transparency.

We also wish to encourage the Board to consider the full repercussions of exam administrations in
which examinees encounter technological problems that are disruptive, but that are ultimately resolved
(in the sense that examinees can still submit their exams in some fashion). Such situations are not
success stories. Although the Board notes that the technological failures in the Michigan bar exam
were “stressful,” we would suggest that the psychological impacts of technological disruption can
severely damage an examinee’s ability to focus and perform on an incredibly high-stakes exam.

For the foregoing reasons, it is imperative that the states using ExamSoft during the October 5 and 6
bar exams insist on a live beta trial with all 30,000-40,000 examinees registered for these exams
completing mock exams at the same time, as will be the case during the actual exam. Any mock exams
that do not test ExamSoft’s capacity in this fashion, including its capacity to field technical support
phone calls from thousands of applicants, will not provide sufficient assurance that ExamSoft is
capable of performing during the October exam.

We are not rooting for ExamSoft to fail. Examinees across the country would be incredibly relieved if
the ExamSoft technology successfully performs in such a mock exam, enabling us to fully focus on
our studies and other professional and familial obligations, without the added stress caused by the
substantial probability of technological issues and eleventh-hour changes to the exam format.

We would like to thank the Board for its transparency in explaining that it has a back-up plan, should
ExamSoft technology appear to be infeasible. To the extent that the Board becomes persuaded that
ExamSoft is unlikely to function successfully, we would urge the Board to adopt its back-up plan at
the earliest possible occasion, in order to give examinees as much notice as possible of this additional
significant change to the exam format. Alternatively, it would be helpful for examinees to have a sense
of the Board’s timeline for making a final determination regarding the viability of ExamSoft.

Once again, we would like to thank both the Board and the Court for their thorough and thoughtful
engagement on these issues, especially as facts on the ground continue to change. Thank you for
hearing our concerns.

2
McMasters, Feasibility of a Mass Online California Bar Exam: Part 2: Technical Feasibility (July 28, 2020),
https://docs.google.com/document/d/1xfNmDsP0fnHqk6se2vUF54T9ZSf9bNr8oisR9aDjVwg/edit?usp=sharing, at
12. This report provides further analysis of the exam in Saudi Arabia that the Board may find instructive. See id. at
11–13. ExamSoft has shared very little about the extent of its involvement and technological role in this exam: the
most information that appears to be available is that ExamSoft stated that it was “honored to help make [the exam in
Saudi Arabia] happen” on LinkedIn. See ExamSoft, LinkedIn, https://www.linkedin.com/posts/examsoft_ai-
technology-to-enable-350000-students-activity-6676479099927719936-AcS0.
3
See McMasters, supra note 2, at 12.
Sincerely,

Rachel Davidson
Jordana Douglas
Sam Feigenbaum
Brianna Paton
Natalia Pena

Signatories are law graduates registered for the October bar exam and are also petitioners in matter
number SJ-2020-0556.

cc: Donna Wardynski, Jennifer MacBeth