Legal updates TECHNOLOGY
PREDICTIVE CODING:
MACHINE LEARNING
DISRUPTS DISCOVERY
By Thomas Davey and Michael Legg*
T
he decision in McConnell
Dowell Constructions (Aust)
Pty Ltd v Santam Ltd (No 1)
[2016] VSC 734 (‘McConnell
Dowell’) signals the emerging judicial
acceptance of predictive coding and
should encourage practitioners in New
South Wales to embrace the technology.
Predictive coding provides solicitors
with a rare opportunity to dramatically
improve the time efficiency and cost
effectiveness of court proceedings while
also enhancing justice.
Machine learning, predictive
coding and technology assisted
review
Predictive coding is the application
of machine learning to the process
of discovery. It is a process which
uses statistical modelling to make
predictions about the relevance of
documents in discovery in lieu of human
review. Predictive coding has been
frequently referred to as ‘technology
assisted review’ or ‘TAR’ by the legal
community. Unfortunately, this term
obfuscates the range of ways in which
technology assists in the review process.
TAR is frequently and indiscriminately
used to refer to keyword searches,
concept searching, predictive coding,
or all of the above. In practice, each of
these technologies works differently,
produces different results, and achieves
different levels of accuracy. This article
champions predictive coding, the most
groundbreaking and disruptive of those
technologies, which is changing the way
in which large scale electronic discovery
takes place.
Predictive coding addresses the
challenges posed by electronically
stored information (‘ESI’) during the
process of discovery. ESI is the cause
of significant cost and delay in the
discovery process due to its volume,
duplicability, and dispersion. Rapid and
continuing developments in information
technology have increased the number
of potentially relevant documents that
need to be analysed. Estimates suggest
that 90 per cent of business records
82 LSJ I ISSUE 32 I APRIL 2017
Thomas Davey is
a law clerk, Jones
Day, and Michael
Legg is Associate
Professor UNSW Law
and member of the
Law Society’s Future
Committee.
• Predictive coding can offer a
fifty-fold improvement over
manual review in terms of
efficiency for cases with large
quantities of Electronically
Stored Information.
• The Victorian decision of
McConnell Dowell represents
the first judicial approval of
predictive coding in Australia
and creates persuasive
authority for New South Wales.
• Predictive coding better
adheres to the principle of
proportionality than the
alternatives in many cases.
are already held in electronic format
only and individuals within businesses
will send and receive an average of 140
emails a day by 2018.
Predictive coding is concerned with
the collection phase of discovery in
which documents are gathered and
reviewed for relevance. Frequently,
the desired documents only make up
a small proportion of the documents
initially obtained. This process has
been traditionally undertaken by junior
associates and paralegals who manually
review documents to identify those
which may be relevant. Predictive
coding, on the other hand, uses machine
learning to identify relevant documents.
The process sees a senior lawyer or small
team review and code a ‘seed set’ of
documents. A computer then identifies
similarities and patterns within those
documents and attempts to predict the
coding for additional samples. When
the coding of the human reviewers and
the computer sufficiently coincides,
the computer is able to make confident
predictions for the balance of the
documents.
The total number of documents
reviewed by the senior lawyer is typically
only a few thousand. Studies have
determined that predictive coding
can yield more accurate results than
exhaustive manual review and, in some
cases, offer a fifty-fold saving in terms
of documents reviewed manually (see
Maura Grossman and Gordon Cormack,
‘Technology-Assisted Review in
E-Discovery Can Be More Effective and
More Efficient Than Exhaustive Manual
Review’, (2011) 17 Richmond Journal of
Law and Technology 11).
Has TAR been judicially approved?
The first judicial decision concerning
the use of predictive coding was made
by Magistrate Judge Peck in the United
States decision of Da Silva Moore v
Publicis Groupe et al (2012) 287 F.R.D
182 (‘Da Silva Moore’). In that case,
Judge Peck held that predictive coding
was better able to process the three
million electronically stored documents
than any of the available alternatives.
Statistical evidence examined during the
trial revealed that the concept of manual
review as the ‘gold standard’ was a myth
and computerised searches were ‘at least
as accurate, if not more so.’
Judge Peck criticised the utility of
keyword searches (which are widely
used in Australia). He stated that such
searches require guesses to be made by
legal practitioners about which keywords
might support their clients’ case. The
results are often inaccurate and, in many
cases, counsel doesn’t know what is in
its own client’s documents. Keyword
searches frequently achieve a mere 25
per cent recall and 25 per cent precision.
In contrast, the machine learning
technology that underpins predictive
coding sees the computer learn from
experience over time and produce
results that are better in every measure.
When used in discovery, predictive
coding extrapolates human judgments
to a broad collection of documents,
emulating human decision-making and
reducing the cost and time of review.
Similar technology has been used in self-
driving cars, language translation, and
speech and vision recognition.

In what may be a relief to many lawyers,
Judge Peck held that the Court should
be less interested in the science behind
the ‘black box’ of the predictive coding
vendor’s software than its production of
responsive documents with reasonably
high recall and precision.
The Da Silva Moore ruling was made in
pre-trial discovery proceedings, of which
Magistrate Judge Peck was supervising.
District Judge Carter went on to affirm
Peck’s decision stating that it was well
reasoned and considered the potential
advantages and pitfalls of the predictive
coding software.
From novel technology to black letter law
In 2014, the United States Tax Court held
that predictive coding had become a
widely accepted method for facilitating
effective discovery of electronically
stored information without undue
burden (see Dynamo Holdings Ltd.
P’Ship v Comm’r of Internal Revenue,
2014 WL 4636526). In Rio Tinto PLC v
Vale S.A 306 F.R.D 125 (S.D.N.Y. 2015)
(‘Rio Tinto’), three years after Da Silva
Moore, Judge Peck stated that the use
of predictive coding in the United States
had become ‘black letter law’.
This string of cases was picked up
in Ireland (see Irish Bank Resolution
Corporation Limited v Quinn [2015]
IEHC 175) and England (see Pyrrho
Investments Ltd v MWB Property Ltd
[2016] EWHC 256 (‘Pyrrho’)) where the
courts acknowledged the connection
between the advantages of predictive
coding and the broader purposes of the
judicial system. Namely, that it provides
a proportionate solution to the discovery
of expansive ESI.
Australia’s cautious approach
Ultimately, it was from the ratio of Pyrrho
that Justice Vickery drew the majority
of his reasoning in McConnell Dowell.
In that case, the central issue was how
discovery of approximately four million
electronic documents could and should
be managed consistently with the
principles of proportionality and s 9 of the
Civil Procedure Act 2010 (Vic).
Vickery J noted that principles of civil
procedure referred to in Pyrrho were
closely aligned to those in Victoria. The
vast quantity of documents and the
fact that it would be near impossible
to accurately review them through
traditional means also reflected the
issues in Pyrrho. Vickery J appointed a
senior barrister as a Special Referee to
answer questions as to the appropriate
Predictive coding is the
application of machine
learning to the process of
discovery. It is a process
which uses statistical
modelling to make
predictions about the
relevance of documents in
discovery in lieu of human
review.
management of discovery. In answering
the questions, the parties came
together to undertake a ‘due diligence’
examination of predictive coding. The
conclusion of the special referee, the
parties and the Court was that just,
efficient, timely and cost-effective
discovery would be best achieved
through the sophisticated process of
predictive coding.
Vickery J held that predictive coding
facilitated a reasonable search for
documents and that the costs associated
with doing so were proportionate. Not
only was the process more sophisticated
than a word search facility it was also
able to dramatically reduce the search
volume and bring it within ‘reasonable
and manageable bounds.’ The fact that
traditional review of the documents
would have taken approximately 583
working weeks clearly demonstrated that
manual discovery was not appropriate.
The law as it stands
Vickery J’s judgment foreshadowed
Practice Note SC Gen 5 – Technology
in Civil Litigation which was issued
earlier this year. The note states that
the use of technology in civil litigation
should facilitate the just, efficient,
timely and cost-effective resolution of
the real issues in dispute. The Court
expects parties to ‘acquit their obligation
to ensure costs are reasonable and
proportionate by employing technology
to save time and cost wherever possible.’
The note also states that ‘the use of
common technologies is a core skill for
lawyers and a basic component of all
legal practice’ and that ‘the inability or
reluctance of a lawyer to use common
technologies should not occasion
additional costs for other parties’.
In relation to discovery, it states that
TECHNOLOGY
‘technology assisted review will ordinarily
be an accepted method of conducting
a reasonable search in accordance with
the Rules of Court.’ The note makes
direct reference to the decision in Pyrrho
and states that the costs of manually
searching documents in larger cases may
not be reasonable and proportionate.
The Federal Court and the Supreme Court
of New South Wales
Given the acceptance of predictive
coding in Victoria and across the world,
it is likely that the technology will receive
judicial recognition in other Australian
jurisdictions soon. With judicial approval
mounting, the relevant practice notes
of the Federal Court of Australia and the
Supreme Court of NSW both appear to
be broad enough to permit TAR without
express judicial intervention. Practice
Note SC Gen 7 of the NSW Supreme
Court and the Technology and the Court
Practice Note of the Federal Court of
Australia both encourage the use of
technology in civil litigation and require
all parties to consider the prospect of
using technology at various stages of the
trial. The Federal Court note in particular
states that technology should be used to
assist in achieving the quick, inexpensive
and efficient resolution of proceedings.
Opportunities
It may be that solicitors are now under
an obligation to consider predictive
coding in cases where large quantities
of ESI exist. Whichever path is chosen,
solicitors will need to satisfy the court
that their selected process meets the
standards of each jurisdiction. When
appropriately implemented, predictive
coding should enable a document
review process that is more just,
time efficient, and cost effective than
traditional manual review.
Justice
Predictive coding can enhance the just
resolution of a dispute by improving
the quality of discovery and, indirectly,
by reducing cost and delay. The quality
of predictive coding in relation to large
sets of documents is superior to that of
manual review. As Judge Peck said in Da
Silva Moore, ‘even if all parties here were
willing to entertain the notion of manually
reviewing the documents, such review is
prone to human error and marred with
inconsistencies from the various attorneys’
determination of whether a document is
responsive.’

ISSUE 32 I APRIL 2017 I LSJ 83
 Legal updates TECHNOLOGY
When it comes to large data sets,
‘technology assisted review using
predictive coding is at least as accurate
as, and, probably more accurate than, the
manual or linear method in identifying
relevant documents’. Researchers have
concluded that human reviewers often
suffer from fatigue, inattention and
boredom and that predictive coding
exceeds the effectiveness of human
review with a small fraction of the effort
(see Maura Grossman and Gordon
Cormack, ‘A Tour of Technology-Assisted
Review’ in Jason Baron, Ralph Losey
and Michael Berman (eds), Perspectives
on Predictive Coding (American Bar
Association, 2016) chapter 3).
Time efficiency
The concept of justice is also closely
intertwined with cost and delay. Courts
have long struggled with the causes of
delay, but have recognized justice is
illusory if it cannot be achieved within
a reasonable time. In Aon Risk Services
Australia Ltd v Australian National
University (2009) 239 CLR 175 it was held
that there is ‘an irreparable element of
unfair prejudice in unnecessarily delaying
proceedings.’ Predictive coding presents
a solution to delays caused by expansive
discovery. The cost of the computer
processing power required to conduct
predictive coding has become affordable
to a large percentage of litigants and the
bulk of the time required for it to work
can be attributed to the human review
component. With the reductions in
human review on offer, predictive coding
is able to significantly reduce the time of
the discovery process.
Cost effectiveness
It is estimated that document review
accounts for 70 per cent of all discovery
costs. The suitability of any given
technology in litigation should be
measured with reference to its ability
to make the proceedings more cost-
effective. Doing so enhances justice for
both the participating parties and the
community at large. It prevents premature
settlement and improves access to justice
for those who would otherwise be unable
to bear the burden of an ESI discovery.
Predictive coding reduces costs by
simply reducing the number of hours
needed for fee-earners to review
documents. Using standard rates, it
would take over 1,000 hours and cost
over $250,000 for a human to review
100,000 documents, at a speedy rate
of 100 documents an hour. The same
review completed using predictive
84 LSJ I ISSUE 32 I APRIL 2017
coding would cost a mere $50,000 in end, they are duty bound to adhere to
labor as humans review just one fifth instructions of the court, including those
of the original set. In many cases, even contained in practice notes. Interestingly,
greater savings may be achieved. a recent survey of US Federal Court
judges revealed a belief that a majority of
Risks lawyers are failing to make proper use of
technology when it comes to discovery.
Security
It is likely that this holds true for the
Like all cloud based technologies, Australian legal profession. This trend
predictive coding presents a cyber poses a risk for lawyers.
security risk. Stacey Blaustein, the head
It may be that solicitors are obliged to
of IBM’s eDiscovery program, has noted
consider the use of predictive coding in
that the volume and severity of attacks
all cases that have a large amount of ESI.
on ESI has increased exponentially.
The Supreme Court of Victoria already
According to Blaustein, 80 per cent
‘expects parties to be in a position to
of the United States’ largest law firms
address questions [of technology in
have experienced a data security
review] at the point at which discovery
incident. The risk can be mitigated
is sought.’ The solicitor’s obligation to
by pre-engagement risk analysis, the
deliver services competently, diligently
implementation of appropriate cyber
and as promptly as reasonably possible
security measures, and the taking out
may encompass an obligation to advise
of suitable cyber insurance policies
clients of the avenues available to
(see: www.insurancelawtomorrow.
manage discovery and their respective
com/2016/12/how-to-protect-your-
costs. Considering predictive coding is
business-from-cyber-threats ‘).
often able to provide significant cost
Procedural risks savings, practitioners may be under an
The overriding purpose of the justice ethical obligation to consider its use. The
American Bar Association has recently
system is facilitated through careful case
management by the presiding judge. In updated their rules to ensure lawyers
keep abreast of benefits and risks of
this regard, predictive coding presents
unique challenges for the judiciary. relevant technology.
Complicated ESI and varied technologies
Conclusion
make case management a difficult
task. There is a risk that the benefits of It is certainly not the case that
predictive coding will be appropriate in
predictive coding will not be realised if
all circumstances, but its widespread
judges are unable to encourage parties
adoption suggests that it should be
and their lawyers to cooperate. It is likely
seriously considered in cases where
that the parties rather than the judge
large amounts of ESI are likely to cause
will be best placed to determine what
significant cost or delay.
procedure is appropriate. In Rio Tinto,
* The views in this article are those of the
the approved protocol incorporated a
collaborative process of review which authors and not any organisation they are
associated with.
saw both parties check the relevancy
of seed documents. This process
assisted in satisfying the court and the
requesting party that the computer
It may be that solicitors are
had been appropriately trained. Such now under an obligation to
collaboration may not be common in
Australian courts; however, it may be consider predictive coding
that documents produced through a
process of predictive coding without
in cases where large
collaboration are more likely to be quantities of Electronically
challenged. If the judge is unable to
maintain genuine cooperation between Stored Information exist.
the parties the overriding purpose may
not be met and the benefits of predictive
Whichever path is chosen,
coding may be lost. solicitors will need to
Ethical obligations satisfy the court that their
The availability of predictive coding
presents ethical risks for technophobic
selected process meets
lawyers. Solicitors are under a prevailing the standards of each
and paramount duty to the court and
the administration of justice. To this jurisdiction.