Free Gift - Project Failure Case Studies V1.8

Project Failure Case Studies
Lessons learned from other people’s mistakes
by
Henrico Dolfing
www.henricodolfing.com
Version 1.8
Copyright © 2020 Henrico Dolfing. All rights reserved.
2
Content
Introduction 5
Case Study 1: The £10 Billion IT Disaster at the NHS 6

Timeline of Events 7
What Went Wrong 9
How NHS Could Have Done Things Differently 11
Closing Thoughts 14
References 15
Case Study 2: The Epic Meltdown of TSB Bank 16

The Basic Principles of a System Migration 19
Testing a System Migration 20
What Went Wrong 21
How TSB Could Have Done Things Differently 22
Closing Thoughts 22
References 23
Case Study 3: How a Screwed-Up SAP Implementation Almost Brought Down

National Grid 24
What Went Wrong 28
Closing Thoughts 30
References 31
Case Study 4: The $440 Million Software Error at Knight Capital 32

What Went Wrong 34
How Knight Capital could have done things differently 37
Closing Thoughts 41
References 41
Case Study 5: Workday did not work for Sacramento Public Schools 42
What Went Wrong 45
How SCUSD Could Have Done Things Differently 48
Closing Thoughts 49
References 50
Case Study 6: How Revlon Got Sued by Its Own Shareholders Because of a Failed SAP
Implementation 51
3
What Went Wrong 55
How Revlon Could Have Done Things Differently 56
Closing Thoughts 58
References 58
Case Study 7: Target’s $2.5 Billion Cross-Border Expansion Mistake 60

What Went Wrong 66
How Target Could Have Done Things Differently 70
Closing Thoughts 73
References 73
Case Study 8: How Hertz Paid Accenture $32M for a Website That Never Went Live 74
What Went Wrong 78
How Hertz Could Have Done Things Differently 81
Closing Thoughts 82
References 83
Case Study 9: The Payroll System That Cost Queensland Health AU$1.25 Billion 84
What Went wrong 88
How Queensland Health Could Have Done Things Differently 90
Closing Thoughts 92
References 93
Case Study 10: Leaseplan Paid $100 Million for a SAP System That Never Went Live 94
What Went Wrong 97
How LeasePlan Could Have Done Things Differently 98
Closing Thoughts 100
References 101
Conclusion 102
About the author 105
4
Introduction
“Management is, above all, a practice where art, science, and craft meet.”
— Henry Mintzberg
Lessons Learned are experiences distilled from a project that should be actively taken
into account in future projects.
There are several definitions of the concept. The one used by NASA is as follows:
“A lesson learned is knowledge or understanding gained by experience. The

experience may be positive, as in a successful test or mission, or negative, as in
a mishap or failure. A lesson must be significant in that it has a real or assumed
impact on operations; valid in that is factually and technically correct; and
applicable in that it identifies a specific design, process, or decision that reduces
or eliminates the potential for failures and mishaps, or reinforces a positive
result.”
Personally I like the following definition:
“Generalizations based on evaluation experiences with projects, programs, or

policies that abstract from the specific circumstances to broader situations.
Frequently, lessons highlight strengths or weaknesses in preparation, design,
and implementation that affect performance, outcome, and impact.”
I think most organizations feel that project sponsors, managers, and teams can reduce
project costs and duration by learning from past projects, by implementing past
successes, and by avoiding past failures.
And of course instead of learning from your own failures, it is even better to learn from
past failures from others. I do this by researching project failures and then writing case
studies about them because it is a great way (for both of us) to learn from others'
mistakes.
This eBook is an ever-growing collection of such project failure case studies.
Henrico Dolfing
5
Case Study 1: The £10 Billion IT Disaster at the NHS
“One of the worst and most expensive contracting fiascos in public sector history” —
The Public Accounts Committee (PAC)
The National Program for IT (NPfIT) in the National Health Service (NHS) was the
largest public-sector IT program ever attempted in the UK, originally budgeted to cost
approximately £6 billion over the lifetime of the major contracts.
These contracts were awarded to some of the biggest players in the IT industry,
including Accenture, CSC, Atos Origin, Fujitsu and BT.
After a history marked by delays, stakeholder opposition and implementation issues,

the program was dismantled by the Conservative-Liberal Democrat government in
2011, almost 10 years after Prime Minister Tony Blair initiated it at a seminar in
Downing Street in 2002.
The core aim of the NPfIT was to bring the NHS’ use of information technology into the
21st century, through the introduction of integrated electronic patient records
systems, online ‘choose and book’ services, computerized referral and prescription
systems and underpinning network infrastructure.
Despite the failure of many of these services to be delivered, the government, and
ultimately taxpayers, incurred significant costs for the program, including contract
transition and exit costs which continued to accrue to a total amount of more than £10
billion.
Since NPfIT was a public-sector program, there is a large amount of documentation

and press about the case available. If you are interested in reading more about it, I
have collected many of these documents here.
6
Timeline of Events
2002
> NPfIT starts with Richard Granger being the appointed NHS IT director.
2003
> BT awarded contract for the national data spine

> Local service provider 10 year contracts awarded (CSC for North West and West
Midland cluster; BT Capital Care Alliance for London cluster; Fujitsu for Southern
cluster; Accenture for North East and Eastern England clusters.
2004
> BT awarded N* (NHS broadband network) contract
2005
> NHS Connecting for Health (NHS CFH) set up to deliver NPfIT
> Contract reset 1 (BT) for “interim solutions” in London
2006
> Accenture withdraws as local service provider

> CSC awarded 9 year contract for Accenture’s former clusters
2007
> NPfIT Local Ownership Programme (devolves responsibility for local delivery of the
program from NHS CFH to groupings of strategic health authorities
> Replaces original five clusters with three program areas: Southern (local service
provider Fujitsu), London (local service provider BT), and North, Midlands and East
(local service provider CSC).
> Contract reset 2 (BT) for “best of breed” London solutions
2008
> Fujitsu contract for local service provider in Southern area terminated; legal dispute
continues
7
> Contract reset negotiations 3 (BT) for new delivery model in London
> Richard Granger, head of NHS CFH, leaves in January
> Gordon Hextall, acting head, leaves in April
> Christine Connelly and Martin Bellamy appointed to jointly lead NHS CFH in
September
2009
> BT awarded additional contract to take over eight trusts formerly with Fujitsu, plus
25 trusts for RiO and four additional acute trusts in Southern area.
> Other Southern trusts given choice of local service provider solution from BT or CSC
or from various suppliers in Additional Supply Capability and Capacity List (ASCC)
> Martin Bellamy, director of programmes and systems delivery, NHS CFH, resigns
> NHS CFH, headed by Christine Connelly, is integrated with Department of Health
Informatics Directorate
> Parliamentary announcement of contract negotiations with BT and CSC - seeking
NPfIT costs savings
2010
> New memorandum of agreement signed between BT and NHS CFH, including
reduced number of deployments in acute trusts in London
> Contract discussions with CSC continuing
> UK general election in May - new coalition government
2011
> The National Programme for IT has finally come to an end, although the bill for the
enormously expensive and controversial project will continue to be paid for years to
come.
> The deadline to exit NPfIT national contracts in the North, Midlands and East passed
on 7 July, marking the end of the final chapter of the £12.7 billion attempt to bring the
NHS into the digital age.
> Around £2.6 billion of actual benefits had been identified as of March 2011
8
What Went Wrong

In order to bring some light on all the things that went wrong I follow the structure of
Oliver Campion-Awwad, Alexander Hayton, Leila Smith and Mark Vuaran. You will find
their case history document in the directory mentioned above.
It identifies three main themes:
1) Rush
2) Design
3) Culture and skills
The rest of this article will look into these into more detail.
Rush
In their rush to reap the rewards of the program, politicians and program managers
raced headlong into policy-making, procurement and implementation processes that
allowed little time for consultation with key stakeholders and failed to deal with
confidentiality concerns. This resulted in:
> An unrealistic timetable
> No time to engage with users and privacy campaigners
> Inadequate preliminary work
> Failure to check progress against expectations
> Failure to test systems
Design
In an effort to reduce costs and ensure swift uptake at the local levels, the government
pursued an overambitious and unwieldy centralized model, without giving
consideration to how this would impact user satisfaction and confidentiality issues.
This resulted in:
9
> Failure to recognize the risks or limitations of big IT projects
> Failure to recognize that the longer the project takes, the more likely it is to be
overtaken by new technology
> Sheer ambition
> The project being too large for the leadership to manage competently
> Confidentiality issues
Culture and skills
The NPfIT lacked clear direction, project management and an exit strategy, meaning
that the inevitable setbacks of pursuing such an ambitious program quickly turned into
system-wide failures. Furthermore, the culture within the Department of Health and
government in general was not conducive to swift identification and rectification of
strategic or technical errors. This resulted in:
> A lack of clear leadership
> Not knowing, or continually changing, the aim of the project
> Not committing the necessary budget from the outset
> Not providing training
> A lack of concern for privacy issues
> No exit plans and no alternatives
> A lack of project management skills
> Treasury emphasis on price over quality
> IT suppliers that depend on lowballing for contracts and charge heavily for variations
to poorly written specifications
10
How NHS Could Have Done Things Differently

This case study contains several lessons useful for project managers, IT professionals,
and business leaders.
Understanding the problem
"Top-down" projects are much more likely to fail than "bottom-up" projects, and NPfIT
was top-down project par excellence. I identify a top-down project as one done for
political reasons: and this can be both genuinely Political with a capital P in the public
sector or a "vanity" or CEO-inspired project in the private sector. The history of public
sector ICT and outsourcing is littered with politically-inspired projects that failed.
The motivation to commence NPfIT came from Cabinet level and it's hard to argue
against the fact that many of its aims were entirely laudable. But there is a big gap
between laudability and deliverability. The decision to commence any project - let
alone one which will transform a fundamental building block of a nation's healthcare
system - must be made by the right people who really know about the issues involved.
It's unfortunate for civil servants and the departments they run that they have to carry
the can for projects devised by ministers that often only make sense on the political
drawing board and are almost impossible to translate into reality.
Stakeholder engagement
Rarely is a project ever just an IT project; generally it should be viewed as a broader

process to deliver business benefits.
It is a given of successful technology projects that there should be good consultation

with all stakeholders involved, including end-users. Ever since NPfIT began, there have
been concerns expressed by key stakeholders within the health system, especially
doctors and GPs, about the accessibility and utility of the planned system.
In particular, it was not clear even from the outset of NPfIT exactly what was going to
be delivered to the ultimate end-users. Add to that the entrenched interests in NHS
trusts about loss of control over their own systems and you have an inherently
suspicious, if not downright hostile, user base. Few projects can succeed over the
outright opposition of the proposed users.
11
Start slow in order to run fast later
One so-called innovation for which NPfIT was originally praised was the speed and
efficiency of its procurement and contracting process. That process has subsequently
become a mill-stone around the project's neck.
NPfIT rushed to award contracts in almost indecent speed with insufficient planning,
particularly for such a large contract. Contract scope was unclear and much work
needed to be done after contract award to agree key contract parameters such as
scope and deliverables. At the time, this was felt by many to be a sign of success and
the model for how future procurement should be done.
No-one could argue that there must always be a desire to procure and award contracts
as efficiently as possible. But this can't come at the sacrifice of agreeing all appropriate
contract terms up-front, rather than retrospectively once the contract has been put in
place. Nor is this a substitute for doing appropriate due diligence before the contract is
awarded and actually writing clear statements of work and requirements.
One of the lessons that should be learned is that projects will always run into trouble if
they try to complete the contractual paperwork before actually working out the scope
of what a project is about, what its deliverables will be and how they will be
implemented.
Balancing risk and reward
The NPfIT procurement model called for a drastic cut in timescales, with no negotiation
allowed, contracts offered on a "take-it-or-leave-it" basis and a very aggressive
approach to legal remedies against service providers. NPfIT and its advisers appeared
to forget the golden rule that these contracts involve a long-term relationship; so a
hyper-aggressive approach to supplier management is counter-productive.
One of the issues that bedevilled the project from the outset was the extent to which
the NPfIT was attempting to force service providers to accept onerous and one-sided
contracts. Negotiation was a dirty word and NPfIT used heavy-handed tactics to ram
through contract terms that were considerably harsher than had ever been seen within
a government (or even private sector) context before.
It is worth noting that some of these provisions (for example, in relation to what
happens on financial distress of a service provider or in relation to clawback of prepaid
milestone payments) have found their way into standard practice within government -
not always with entirely successful results.
12
The combination of the implemented payment provisions (under which service

providers had to do all the work upfront with no payments until successful delivery)
and the harsh termination and liability provisions, meant that the risks being absorbed
by service providers were extremely high.
While many may say that service providers make a healthy margin and, therefore,
ought to absorb risk, no service provider is a bottomless pit able to accept enormous
costs and risks of delivery, particularly where the customer is in a position of being
able to re-interpret and add requirements during the course of delivery. A number of
significant service providers have fallen by the wayside during the course of the NPfIT
project simply because of the difficulty of delivering what NPfIT wanted within its
timelines and risk profile.
Hopefully, one lesson government will learn from NPfIT is that there needs to be a
balance of risk and reward in negotiating contracts even for very large projects. It is
notable that the standard paradigm for public sector contract terms has moderated
significantly since NPfIT was initiated. Many might wish that there had been more
reasonable voices involved in the original project who could have argued for putting in
place a more moderate, deliverable contract that didn't force service providers to take
on so many contractual risks that their own internal business cases became unstable.
Multisourcing
NPfIT was certainly innovative in its structure of making different service providers
work together by awarding work in a series of lots. Part of the rationale for this
approach was that different regional service providers could be swapped in or out if
and when other regional service providers failed.
At least this has meant that, as some service providers have had their contracts
terminated over the past few years, there have been others prepared to pick up the
slack. But it does illustrate that anything other than a "one customer, one service
provider" structure is very difficult to operate.
This doesn't mean that it cannot be established in this way - as the current hype of
multisourcing contracts has shown. But it does mean that there needs to be much
more careful thought and planning in advance about how different service providers
will be incentivised to collaborate - not forced to co-operate almost against their will
and without some "script" or plan as to how disputes will be resolved.
13
Checks and balances
There is a tendency amongst people involved in a project which isn't going well to
adopt the "I've started so I'll finish" approach, circle the wagons and not step back and
take the inevitable decision at a more appropriate stage.
Good managers on technology projects are always asking themselves whether the
original course of action is correct and whether adjustments are required - and they
are prepared to take the ultimate decision at the right time and not delay the
inevitable.
NPfIT was run by a very strong project director with a powerful personality. Maybe
that was the best way to give the project a chance of success. Unfortunately, once the
project ran into trouble there much less room for manoeuvre and few supporters
outside the core team. Clear accountability is fine (indeed, essential) but accountability
needs to be in the right hands and with the right checks and balances. If the public
sector learns anything from NPfIT, let's hope that it will be how to identify and
implement those checks and balances from the outset.
Closing Thoughts
The Public Accounts Committee (PAC) calls the saga one of the “worst and most
expensive contracting fiascos” in public sector history, and adds: “Although the
Department told us that the National Programme had been dismantled, the
component programmes are all continuing, the existing contracts are being honoured
and significant costs are still being incurred. The only change from the National
Programme that the Department could tell us about was that new governance
arrangements were now in place.”
The estimated costs of the scheme rose from £6.4bn to £9.8bn, but ongoing costs
arising from legal battles and other issues will keep dragging this figure higher, the MP
said, especially the price of terminating the Fujitsu contract in the south of England,
and the ongoing costs of Lorenzo in the north, Midlands and east.
Committee member Richard Bacon MP said the fact that only 22 trusts are now
expected to take the Lorenzo system – despite the original contracts with CSC totalling
£3.1bn – is another indictment.
Let’s hope the whole public sector has learned something from this disaster. It is our
money that is wasted.
14
References
> Oliver Campion-Awwad, Alexander Hayton, Leila Smith and Mark Vuaran (2014). The
National Programme for IT in the NHS: A Case History. Available at:
https://drive.google.com/open?id=1xAo0k_xAHaphbsCJW0wzg3Npnd42OfMt
> BCS Health Informatics Forum Strategic Panel (2006). The Way Forward for NHS
Health Informatics. Available at:
https://drive.google.com/open?id=1Vy79BRjjF_TS2dPDbokxyD3IKntz_aIu
> Ross Anderson et al. (2010). The NHS’s National Programme for Information
Technology (NPfIT): A Dossier of Concerns. Available at:
https://drive.google.com/open?id=1AbVZ0KNxrSq9OFjE5LxBVZvd5QyH2KK9
15
Case Study 2: The Epic Meltdown of TSB Bank
“Terribly sorry that some people have had troubles, but migrating all our data onto a
new system was a massive exercise and it went ok for most of our customers. The
bank is now working and I have the information here to prove it! ” — Paul Pester, at
the time CEO TSB Bank
With clients locked out of their bank accounts, mortgage accounts vanishing, small
businesses reporting that they could not pay their staff and reports of debit cards
ceasing to work, the TSB Bank computer crisis of April 2018 has been one of the worst
in recent memory. The bank’s CEO, Paul Pester, admitted in public that the bank was
“on its knees” and that it faces a compensation bill likely to run to tens of millions of
pounds.
But let’s start from the beginning. First, we’ll examine the background of what led to
TSB’s ill-fated system migration. Then, we’ll look at what went wrong and how it could
have been prevented.
Timeline of Events
September 2013
When TSB split from Lloyds Banking Group (LBG) in September 2013, a move forced by
the EU as a condition of its taxpayer bailout in 2008, a clone of the original group’s
computer system was created and rented to TSB for £100m a year.
That banking system was a combination of many old systems for TSB, BOS, Halifax,
Cheltenham & Gloucester, and others that had resulted from the integration of HBOS
with Lloyds as a result of the banking crisis.
Under this arrangement, LBG held all the cards. It controlled the system and offered it
as a costly service to TSB when it was spun off from LBG.
March 2015
When the Spanish Banco Sabadell bought TSB for £1.7bn in March 2015, it put into
motion a plan it had successfully executed in the past for several other smaller banks it
had acquired: merge the bank’s IT systems with its own Proteo banking software and,
in doing so, save millions in IT costs.
16
Sabadell was warned in 2015 that its ambitious plan was high risk and that it was likely
to cost far more than the £450m Lloyds was contributing to the effort.
“It is not overly generous as a budget for that scale of migration,” John Harvie, a
director of the global consultancy firm Protiviti, told the Financial Times in July 2015.
But the Proteo system was designed in 2000 specifically to handle mergers such as that
of TSB into the Spanish group, and Sabadell pressed ahead.
Summer 2016
By the summer of 2016, work on developing the new system was meant to be well
underway and December 2017 was set as a hard-and-fast deadline for delivery.
The time period to develop the new system and migrate TSB over to it was just 18
months. TSB people were saying that Sabadell had done this many times in Spain. But
tiny Spanish local banks are not sprawling LBG legacy systems.
To make matters worse, the Sabadell development team did not have full control—and
therefore a full understanding—of the system they were trying to migrate client data
and systems from because LBG was still the supplier.
Autumn 2017
By the autumn the system was not ready. TSB announced a delay, blaming the
possibility of a UK interest rate rise—which did materialize—and the risk that the bank
might leave itself unable to offer mortgage quotes over a crucial weekend.
Sabadell pushed back the switchover to April 2018 to try to get the system working. It
was an expensive delay because the fees TSB had to pay to LBG to keep using the old IT
system were still clocking up: CEO Pester put the bill at £70m.
April 2018
On April 23, Sabadell announced that Proteo4UK—the name given to the TSB version
of the Spanish bank’s IT system—was complete, and that 5.4m clients had been
“successfully” migrated over to the new system.
Josep Oliu, the chairman of Sabadell, said: “With this migration, Sabadell has proven its
technological management capacity, not only in national migrations but also on an
international scale.”
17
The team behind the development were celebrating. In a LinkedIn post that has since
been removed, those involved in the migration were describing themselves as
“champions,” a “hell of a team,” and were pictured raising glasses of bubbly to cheers
of “TSB transfer done and dusted.”
However, only hours after the switch was flicked, systems crumpled and up to 1.9m
TSB clients who use internet and mobile banking were locked out.
Twitter had a field day as clients frustrated by the inability to access their accounts or
get through to the bank’s call centers started to vent their anger.
Clients reported receiving texts saying their cards had been used abroad; that they had
discovered thousands of pounds in their accounts they did not have; or that mortgage
accounts had vanished, multiplied or changed currency.
One bemused account holder showed his TSB banking app recording a direct debit paid
to Sky Digital 81 years from now. Some saw details of other people’s accounts, and
holidaymakers complained that they had been left unable to pay restaurant and hotel
bills.
TSB, to clients’ fury, at first insisted the problems were only intermittent. At 3:40 a.m.
on Wednesday, April 25, Pester tweeted that the system was “up and running,” only to
be forced to apologize the next day and admit it was actually only running at 50
percent capacity.
On Thursday he admitted the bank was on its knees, announced that he was personally
seizing control of the attempts to fix the problem from his Spanish masters, and had
hired a team from IBM to do the job. Sabadell said it would probably be next week
before normal service returned.
The financial ombudsman and the Financial Conduct Authority have launched
investigations. The bank has been forced to cancel all overdraft fees for April and raise
the interest rate it pays on its classic current account in a bid to stop disillusioned
clients from taking their business elsewhere.
The software Pester had boasted about in September of being 2,500 man-years in the
making, with more than 1,000 people involved, has been a client service disaster that
will cost the bank millions and tarnish its reputation for years.
18
The Basic Principles of a System Migration

The two main things to avoid in a system migration are an unplanned outage of the
service for users and loss of data, either in the sense that unauthorized users have
access to data, or in the sense that data is destroyed.
In most cases, outages cannot be justified during business hours, so migrations must
typically take place within the limited timeframe of a weekend. To be sure that a
migration over a weekend will run smoothly, it is normally necessary to perform one or
more trial migrations in non-production environments, that is, migrations to a copy of
the live system which is not used by or accessible to real users. The trial migration will
expose any problems with the migration process, and these problems can be fixed
without any risk of affecting the service to users.
Once the trial migration is complete, has been tested, and any problems with it have
been fixed, the live migration can be attempted. For a system of any complexity, the
go-live weekend must be carefully pre-planned hour by hour, ensuring that all the
correct people are available and know their roles.
As part of the plan, a rollback plan should be put in place. The rollback plan is a
planned, rapid way to return to the old system in case anything should go wrong
during the live migration. One hopes not to have to use it because the live migration
should not normally be attempted unless there has been a successful trial migration
and the team is confident that all the problems have been ironed out.
On the go-live weekend, the live system is taken offline, and a period of intense, often
round-the-clock, activity begins, following the previously made plan. At a certain point,
while there is still time to trigger the rollback plan, a meeting will be held to decide
whether to go live with the migration or not (a “go/no go” meeting).
If the migration work has gone well, and the migrated system is passing basic tests
(there is no time at that point for full testing; full testing should have been done on the
trial migration), the decision will be to go live. If not, the rollback plan will be triggered
and the system returned to its previous state, that which was obtained before the
go-live weekend.
If the task of migration is so great that it is difficult to fit it into a weekend, even with
very good planning and preparation, it may be necessary to break it into phases. The
data or applications are broken down into groups which are migrated separately.
19
This approach reduces the complexity of each group migration compared to one big
one, but it also has disadvantages. If the data or applications are interdependent, it
may cause performance issues or other technical problems if some are migrated while
others remain, especially if the source and destination are physically far apart.
A phased migration will also normally take longer than a single large migration, which
will add cost, and it will be necessary to run two data centers in parallel for an
extended period, which may add further cost. In TSB’s case, it may have been possible
to migrate the clients across in groups, but it is hard to be sure without knowing its
systems in detail.
Testing a System Migration

Migrations can be expensive because it can take a great deal of time to plan and
perform the trial migration(s). With complex migrations, several trial migrations may
be necessary before all the problems are ironed out. If the timing of the go-live
weekend is tight, which is very likely in a complex migration, it will be necessary to
stage some timed trial migrations—“dress rehearsals.” Dress rehearsals are to ensure
that all the activities required for the go-live can be performed within the timeframe of
a weekend.
Trial migrations should be tested. In other words, once a trial migration has been
performed, the migrated system, which will be hosted in a non-production
environment, should be tested. The larger and more complex the migration, the
greater the requirement for testing. Testing should include functional testing, user
acceptance testing and performance testing.
Functional testing of a migration is somewhat different from functional testing of a

newly developed piece of software. In a migration, the code itself may be unchanged,
and if so there is little value in testing code which is known to work. Instead, it is
important to focus the testing on the points of change between the source
environment and the target. The points of change typically include the interfaces
between each application and whatever other systems it connects to.
In a migration, there is often change in interface parameters used by one system to

connect to another, such as IP addresses, database connection strings, and security
credentials. The normal way to test the interfaces is to exercise whatever functionality
of the application uses the interfaces. Of course, if code changes are necessary as part
of a migration, the affected systems should be tested as new software.
In the case of TSB, the migration involved moving client bank accounts from one
20
banking system to another. Although both the source and target systems were mature
and well-tested, they had different code bases, and it is likely that the amount of
functional testing required would have approached that required for new software.
User acceptance testing is functional testing performed by users. Users know their
application well and therefore have an ability to spot errors quickly, or see problems
that IT professionals might miss. If users test a trial migration and express themselves
satisfied, it is a good sign, but not adequate on its own because, amongst other things,
a handful of user acceptance testers will not test performance.
Performance testing checks that the system will work fast enough to satisfy its
requirements. In a migration the normal requirement is for there to be little or no
performance degradation as a result of the migration. Performance testing is
expensive because it requires a full-size simulation of the systems under test, including
a full data set.
If the data is sensitive, and in TSB’s case it was, it will be necessary, at significant time
and cost, to protect the data by security measures as stringent as those protecting the
live data, and sometimes by anonymizing the data. In the case of TSB, the IBM inquiry
into what went wrong identified insufficient performance testing as one of the
problems.
What Went Wrong

Where did it go wrong for TSB? The bank was attempting a very complex operation.
There would have been a team of thousands drawn from internal staff, staff from IT
service companies, and independent contractors. Their activities would have had to be
carefully coordinated, so that they performed the complex set of tasks in the right
order to the right standard. Many of them would have been rare specialists. If one
such specialist is off sick, it can block the work of hundreds of others. One can imagine
that, as the project approached go-live, having been delayed several times before, the
trial migrations were largely successful but not perfect.
The senior TSB management would have been faced with a dilemma of whether to
accept the risks of doing the live migration without complete testing in the trials, or to
postpone go-live by several weeks and report to the board another slippage, and
several tens of millions of pounds of further cost overrun. They gambled and lost.
21
How TSB Could Have Done Things Differently

Firstly, a migration should have senior management backing. TSB clearly had it, but
with smaller migrations, it is not uncommon for the migration to be some way down
senior managers’ priorities. This can lead to system administrators or other actors,
whose reporting lines lead elsewhere from those doing the migration, frustrating key
parts of the migration because their managers are not ordering them or paying them
to cooperate.
Secondly, careful planning and control is essential. It hardly needs saying that it is not
possible to manage a complex migration without careful planning and those managing
the migration must have an appropriate level of experience and skill. In addition,
however, the planning must follow a sound basic approach that includes trial
migrations, testing, and rollback plans as described above. While the work is going on,
close control is important. Senior management must stay close to what is happening
on the ground and be able to react quickly, for example by fast-tracking authorizations,
if delays or blockages occur.
Thirdly, there must be a clear policy on risk, and the policy should be stuck to. What
criteria must be met for go-live? Once this has been determined, the amount of testing
required can be determined. If the tests are not passed, there must be the discipline
not to attempt the migration, even if it will cost much more.
Finally, in complex migrations, a phased approach should be considered.
Closing Thoughts
In the case of TSB Bank, the problems that occurred after the live migration were
either not spotted in testing, or they were spotted but the management decided to
accept the risk and go live anyway. If they were not spotted, it would indicate that
testing was not comprehensive enough—IBM specifically pointed to insufficient
performance testing. That could be due to a lack of experience among the key
managers. If the problems were spotted in testing, it implies weak go-live criteria
and/or an inappropriate risk policy. IBM also implied that TSB should have performed a
phased migration.
It may be that the public will never fully know what caused TSB’s migration to go
wrong, but it sounds like insufficient planning and testing were major factors. Sensitive
client data was put at risk, and clients suffered long unplanned outages, resulting in
CEO Paul Pester being summoned to the Treasury select committee and the Financial
Conduct Authority launching an investigation into the bank. Ultimately Pester lost his
22
job.
When migrating IT systems in the financial sector, cutting corners is dangerous.

Ultimately, TSB’s case goes to show that the consequences can be dire. For success,
one needs to follow some basic principles, use the right people, and be prepared to
allocate sufficient time and money to planning and testing. Only then can it be ensured
a successful system migration will take place.
References
> IBM CIO Leadership Office (2018). Update for TSB Board. Available at:
https://drive.google.com/open?id=14mO48R6KJo1zlu3jyeAtyZn2mQnxiARa
23
Case Study 3: How a Screwed-Up SAP

Implementation Almost Brought Down National
Grid
"We are aware of the challenges experienced at National Grid and we have worked
with the customer continuously." — SAP
National Grid USA (NGUSA), which is part of the UK-based National Grid Ltd., supplies
electricity and gas in Massachusetts, New York, and Rhode Island. It is one of the
largest investor-owned power distribution companies in the U.S. In October 2012 the
company faced a very difficult decision.
The SAP project that was three years in running and marred by delays and budget
overruns was scheduled to go live on November 5. Failure to go live meant a delay of
another five months, likely another $50 million in additional spending, and a trip back
to the Utilities Rate Commission to request approval to pay for the overruns.
At the same time, Hurricane Sandy was pounding up the East Coast. By mid-October,
forecasts for damage in NGUSA’s service area were extensive. For a utility company,
power restoration takes precedence over everything else after a hurricane.
NGUSA had to know they had a bumpy ride coming when they made the decision to go
live. What they clearly didn’t understand was just how bumpy the ride would be.
In the weeks that followed the go-live, while the NGUSA crews were working tirelessly
to restore power, the SAP project team was just beginning to understand the full
extent of the damage being caused because of the screwed-up SAP implementation.
The problems spanned many areas, including:
Payroll: The new SAP system miscalculated time, pay rates, and reimbursements, so
that employees were paid too little, too much, or nothing at all. Over-payments of
more than $6 million were made to employees and not recovered. There was $12
million paid in settlements to employees related to short pays and deductions. Delays
in the generation of W2s and other tax reporting occurred.
Vendor payments: Just two months after go-live, NGUSA had 15,000 vendor invoices
that they were unable to process for payment, inventory record-keeping was in
shambles, and vendors were being issued payments with the understanding that
24
reconciliation would take place later.
Financial reporting: Prior to go-live it took NGUSA four days to close its financial books.
Following go-live the close took 43 days. So bad was the financial reporting, NGUSA
lost its ability to access short-term borrowing financial vehicles based upon its inability
to provide satisfactory financial reports.
To deal with the many accounting, payroll, and supply-chain issues NGUSA was
grappling with, they launched a stabilization program. To support the program, 300
contractors were initially brought in to assist with payroll issues. A total of 450
contractors were eventually brought in to address payroll problems. Another 200
contractors were brought in to assist with supply-chain issues. And, another 200
contractors were brought in to support the financial close issues.
The first priority of the stabilization effort was to ensure that NGUSA could comply
with its obligations, including:
> Paying employees accurately and on time
> Paying vendors accurately and timely
> Providing legal, regulatory, and other reports to external stakeholders that are
accurate and timely
The team’s second stabilization priority was to enable NGUSA to be efficient and
self-sufficient in operating the SAP system and realize the benefits the system can
provide without significant reliance on external support.
The continuing effort to stabilize SAP was anticipated to be about $30 million per
month in September 2013.
The problems were so profound that the cleanup took more than two years to
complete with a calculated cost of $585 million, more than 150 percent of the cost of
implementation.
25
Timeline of Events
2007
The journey to the decision to go live on November 5 was not unlike that experienced
by other companies that have followed a similar path. In 2007, NGUSA had finalized a
major acquisition, making it one of the largest privately held power distribution
companies in the U.S.
This acquisition left the company with two sets of financial and operating systems.
Capturing the synergies of combining these systems and adopting new sets of business
processes were key components of the justification for the project. The project was
also viewed as a method to allow NGUSA to address significant audit deficiencies in its
financial business processes.
2008
The project to upgrade NGUSA's legacy systems – many of which were running on
Oracle – began in 2008.
2009
In mid-2009, NGUSA hired Deloitte as its systems integrator and set a project budget of
$290 million that was submitted and approved by the Utilities Rate Commission.
2010
Deloitte was initially employed as the lead implementation partner, project manager
and systems integrator, but in June 2010 it was replaced by Ernst and Young (EY) in the
first two roles, and by Wipro as systems integrator. The main reason for this switch
was to lower implementation costs.
2012
The program operated with a target go-live date of December 2011. This date was
later moved to July 2012, followed by October 2012, and then a November 2012 target
date. The final sanctioned estimate of the project was set at $383 million, nearly 30
percent beyond the original target budget that was approved by the board.
NGUSA continued to engage Wipro after the go live in making the necessary fixes to
the installed SAP system tolling their agreement (extending the statute of limitations
26
for filing suit). In many instances, functional and technical specifications had to be
completely rewritten and entire SAP modules had to be rebuilt or abandoned.
2017
On November 30, 2017, NGUSA filed a lawsuit against Wipro in the U.S. District Court
Eastern District of New York. The lawsuit notes that NGUSA was unable to file suit
against EY due to the language of their contract. The suit alleges that Wipro
fraudulently induced NGUSA into signing the original agreements. NGUSA claims that
Wipro misrepresented its SAP implementation capabilities, talent, and knowledge of
the U.S. utilities business operations and common practices.
As Wipro knew or should have known, it had neither the ability nor intent to assign
appropriately experienced and skilled consultants to the Project because... it in fact
had virtually no experience implementing an SAP platform for a U.S.-regulated utility. –
National Grid USA.
Besides this, the suit alleges that Wipro breached its contract with NGUSA by:
> Failing to prepare design documents and specifications to industry standards.
> Failing to prepare programming and configuration to industry standards.
> Failing to adequately test, detect, and inform of problems.
> Failing to advise that the system was not ready to go live.
> Breaching express and implied warranties by not providing consultants that were
consistent with a top 25 percent SAP implementation firm.
> Negligently misrepresenting itself for the same reasons identified in the first cause
for action.
> Violating New York’s General Business Law for deceptive practices.
NGUSA was seeking damages in the form of relief of all contractual obligations,
restitution of all amounts paid to Wipro, damages associated with a poor go-live,
punitive damages, and attorney's fees and costs associated with the lawsuit.
2018
27
On June 1, 2018, Wipro filed a motion to dismiss on three of the five causes for claims
that it fraudulently misrepresented its capabilities and of negligent misrepresentation.
In its response to the NGUSA RFP, Wipro claims that it identified that it had a
well-established SAP practice, installed SAP globally for utilities, and had a
long-running relationship with U.S. utilities. There was no explicit statement indicating
that Wipro had not completed implementations of SAP for U.S.-based utilities nor
were specific references provided in this regard.
Wipro also defended much of the language in the RFP response as common puffery,
implying that NGUSA had a basic responsibility to check references.
In August 2018 Wipro paid NGUSA $75 million to settle the lawsuit. Wipro states that
the settlement has been effected for an amount of US$75 million and is without
admission of liability or wrongdoing of any kind by the parties.
NGUSA has been a valued customer of Wipro for over a decade and both organizations
have had a mutually beneficial relationship over the years. We believe that this
settlement will be commercially beneficial for us and will help us remain focused on
growth. – Wipro
What Went Wrong

In my experience with such large projects, it is never one party that is responsible for
such a disaster by itself. Where were NGUSA’s project owners? Client project teams
have responsibility for signing off on requirements, designs, project strategies, and test
results. Did NGUSA provide Wipro with the appropriate access to expert personnel to
properly identify requirements? Did Wipro believe that they had accurately captured
all requirements based on NGUSA’s sign-off?
In July of 2014, the NorthStar Consulting Group presented their findings of a

comprehensive management and operations audit of the U.S. NGUSA companies
sponsored by the New York Public Service Commission. The 265-page report (see
references) covered a broad range of the company’s operations and governance.
Throughout the report, the impact of the failed go-live was noted as well as the
governance processes that led the company to determine that going live on November
5 was the best decision.
Below are some interesting observations noted in the report.
Design
28
The system only produced limited reports for management. Most managers have
received only highly summarized reports of the costs they are responsible for since the
go-live date. November 2013, eight months into the fiscal year, was the first time
managers received a detailed cost report that also contained their corresponding
budget figures. Some of the lack of reporting was a result of the system design, and
many reports that had been provided by the predecessor systems were not provided in
the design of the SAP.
Training
Another reason for the lack of information to managers is that the philosophy of
information access at the SAP system is that managers are expected to request
tailored information and reports from the system with the support of analysts from
Decision Support. The lack of staff with the high level of skills necessary to query the
data and produce reports for managers has greatly limited the success of this strategy.
Testing
Testing was conducted during each phase of development of the SAP system. One of
the lessons learned is that the testing was designed to determine where the system
did work rather than identifying the areas where it did not work. Another lesson is that
errors were found in the final test stages. Fixes were installed but there was no time
for retesting.
Complexity
Building an SAP system requires the development of a series of components commonly

referred to as RICEFWs (Reports, Interfaces, Conversions, Enhancements, Forms,
Workflows). NGUSA’s design had a total of 636 RICEFWs. As Exhibit IV-5 illustrates, this
was a large number for even a large power utility. The NGUSA system design was twice
as complex as NGUSA UK’s R1 implementation of SAP and three times as complex as
NGUSA UK’s R2 implementation.
Preparation
Pre-implementation, NGUSA did not benefit from the rest of the industry’s SAP lessons
learned. NGUSA did not use vendors with a strong track record of U.S. utility industry
experience in SAP platform implementation and to date has had almost no interface
with other U.S. utilities that have implemented SAP.
29
Transparency
While problems with system and company readiness were identified by particular
groups within NGUSA prior to implementation, that information was subsumed by a
push to go live. The overly optimistic risk scoring and executive expectations for the
project in its early stages continues with stabilization work.
Validation
During the initial SAP development process, there was minimal interaction with
operations personnel regarding desired information or reports. NGUSA implemented a
complex field time reporting system without investigating its feasibility given how work
is actually performed.
As part of the findings, NorthPoint documented NGUSA’s management observations as

to the root cause of the failed implementation. These included:
> Overly ambitious design
> Significantly underestimated scale of transformation needed
> Limited availability of internal personnel due to ambitious business agenda
> Multi-partner model did not deliver business benefits
> Lack of ownership of certain business processes
> Testing less effective than expected due to limited range of scenarios tested and
limited data availability
> Inadequate quality of data from legacy systems
> Too much focus on timeline and not enough focus on quality
> Training methods proved ineffective
Closing Thoughts
There are many checkpoints a project the magnitude of NGUSA’s SAP implementation
must pass to move forward, each requiring NGUSA to sign off on the quality of the
delivered product. There were many opportunities for NGUSA to identify poor-quality
30
talent on the part of Wipro and demand replacements.
The final decision to go live always rests with the client and unless Wipro was looking
to deceive NGUSA regarding the results of its testing, NGUSA is partly to blame.
And where was Ernst and Young? EY was providing project management oversight.
They clearly have an understanding of what it takes to put in a major SAP
implementation. How did they not see or anticipate the major problems that occurred,
and how did they fail to warn the NGUSA management team? Or did they?
Where was SAP? In their suit, NGUSA claims that Wipro developed an overly complex
system that relied on the development of new capabilities as opposed to using the
software as designed. NGUSA identified SAP as providing some level of oversight. Why
didn’t SAP point out these significant deviations from standard? Or did they?
Where were the auditors? Projects of this size and impact are often reviewed by both
internal and external auditing. Were project reviews performed? Were the appropriate
mitigations put in place?
While there are many questions left to be answered about the botched SAP
implementation that almost brought down National Grid, one thing is sure. When you
start a large project like the implementation of a SAP system, you have to take
responsibility and make sure that you as an organization are ready and committed to
it.
References
> UNITED STATES DISTRICT COURT EASTERN DISTRICT OF NEW YORK (2017). National
Grid USA Service Company Inc. VS Wipro Limited. Available at:
https://drive.google.com/open?id=1NCSdf0AjvYnT5jEuryGKTRPTmYcdl_6i
> Northstar Consulting Group (2014). A comprehensive management and operations

audit of National Grid USA’s new york gas companies. Available at:
https://drive.google.com/open?id=16N-RhbnCyYs411EtaHiDb7ySoFNlnRu9
31
Case Study 4: The $440 Million Software Error at

Knight Capital
“Frankly, I don’t see how the SEC can be possibly OK it.” — Thomas Joyce, at the time
CEO Knight Capital
Knight Capital Group was an American global financial services firm engaging in market
making, electronic execution, and institutional sales and trading. In 2012 Knight was
the largest trader in U.S. equities with a market share of around 17 percent on the
New York Stock Exchange (NYSE) as well as on the Nasdaq Stock Market. Knight’s
Electronic Trading Group (ETG) managed an average daily trading volume of more than
3.3 billion trades, trading over $21 billion … daily.
It took 17 years of dedicated work to build Knight Capital Group into one of the leading
trading houses on Wall Street. And it all nearly ended in less than one hour.
What happened to Knight on the morning of August 1, 2012, is every CEO’s nightmare:
A simple human error, easily spotted with hindsight but nearly impossible to predict in
advance, threatened to end the firm.
At Knight, some new trading software contained a flaw that became apparent only
after the software was activated when the New York Stock Exchange (NYSE) opened
that day. The errant software sent Knight on a buying spree, snapping up 150 different
stocks at a total cost of around $7 billion, all in the first hour of trading.
Under stock exchange rules, Knight would have been required to pay for those shares
three days later. However, there was no way it could pay, since the trades were
unintentional and had no source of funds behind them. The only alternatives were to
try to have the trades canceled, or to sell the newly acquired shares the same day.
Knight tried to get the trades canceled. Securities and Exchange Commission (SEC)
chairman Mary Schapiro refused to allow this for most of the stocks in question, and
this seems to have been the right decision. Rules were established after the “flash
crash” of May 2010 to govern when trades should be canceled. Knight’s buying binge
did not drive up the price of the purchased stocks by more than 30 percent, the
cancellation threshold, except for six stocks. Those transactions were reversed. In the
other cases, the trades stood.
32
This was very bad news for Knight but was only fair to its trading partners, who sold
their shares to Knight’s computers in good faith. Knight’s trades were not like those of
the flash crash, when stocks of some of the world’s largest companies suddenly began
trading for as little as a penny and no buyer could credibly claim the transaction price
reflected the correct market value.
Once it was clear that the trades would stand, Knight had no choice but to sell off the
stocks it had bought. Just as the morning’s buying rampage had driven up the price of
those shares, a massive sale into the market would likely have forced down the price,
possibly to a point so low that Knight would not have been able to cover the losses.
Goldman Sachs stepped in to buy Knight’s entire unwanted position at a price that cost
Knight $440 million – a staggering blow, but one the firm might be able to absorb. And
if Knight failed, the only injured party, apart from Knight’s shareholders (including
Goldman), would have been Goldman itself.
Disposing of the accidentally purchased shares was only the first step in Knight CEO
Thomas Joyce’s battle to save his company. The trades had sapped the firm’s capital,
which would have forced it to greatly cut back its business, or maybe to stop operating
altogether, without a cash infusion. And as word spread about the software debacle,
customers were liable to abandon the company if they did not trust its financial and
operational capacities.
A week later, Knight received a $400 million cash infusion from a group of investors,
and by the next summer, it was acquired by a rival, Getco LLC. This case study will
discuss the events leading up to this catastrophe, what went wrong, and how this
could be prevented.
Timeline of Events
Some of Knight’s biggest customers were the discount brokers and online brokerages
such as TD Ameritrade, E*Trade, Scottrade, and Vanguard. Knight also competed for
business with financial services giants like Citigroup, UBS, and Citadel. However, these
larger competitors could internalize increasingly larger amounts of trading away from
the public eye in their own exclusive markets or shared private markets, so-called dark
pools. Since 2008, the portion of all stock trades in the U.S. taking place away from
public markets has risen from 15 percent to more than 40 percent.
In October 2011, the NYSE proposed a dark pool of its own, called the Retail Liquidity
Program (RLP). The RLP would create a private market of traders within the NYSE that
could anonymously transact shares for fractions of pennies more or less than the
33
displayed bid and offer prices, respectively. The RLP was controversial even within
NYSE Euronext, the parent company of the NYSE; its CEO, Duncan Niederauer, had
written a public letter in the Financial Times criticizing dark pools for shifting “more
and more information… outside the public view and excluded from the price discovery
process.”
The SEC decision benefited large institutional investors who could now buy or sell large
blocks of shares with relative anonymity and without moving the public markets;
however, it came again at the expense of market makers. During the months of
debate, Joyce had not given the RLP much chance for approval, saying in one
interview, “Frankly, I don’t see how the SEC can be possibly OK it.” In early June 2012,
the NYSE received SEC approval of its RLP, and it quickly announced the RLP would go
live on August 1, 2012, giving market makers just over 30 days to prepare. Joyce
insisted on participating in the RLP because giving up the order flow without a fight
would have further dented profits in its best line of business.
What Went Wrong

With only a month between the RLP’s approval and its go-live, Knight’s software
development team worked feverishly to make the necessary changes to its trade
execution systems – including SMARS, its algorithmic, high-speed order router. SMARS
stands for Smart Market Access Routing System.
SMARS was able to execute thousands of orders per second and could compare prices
between dozens of different trading venues within fractions of a second.
A core feature of SMARS is to receive orders from other upstream components in

Knight’s trading platform (“parent” orders) and then, as needed based on the available
liquidity and price, sends one or more representative (“child”) orders to downstream,
external venues for execution.
Power Peg
The new RLP code in SMARS replaced some unused code in the relevant portion of the
order router; the old code previously had been used for an order algorithm called
“Power Peg,” which Knight had stopped using since 2003. Power Peg was a test
program that bought high and sold low; it was specifically designed to move stock
prices higher and lower in order to verify the behavior of its other proprietary trading
algorithms in a controlled environment. It was not to be used in the live, production
environment.
34
There were grave problems with Power Peg in the current context. First, the Power
Peg code remained present and executable at the time of the RLP deployment despite
its lack of use. Keeping such “dead code” is bad practice, but common in large software
systems maintained for years. Second, the new RLP code had repurposed a flag that
was formerly used to activate the Power Peg code; the intent was that when the flag
was set to “yes,” the new RLP component – not Power Peg – would be activated.
Such repurposing often creates confusion, had no substantial benefit, and was a major
mistake, as we shall see shortly.
Code refactoring
There had been substantial code refactorings in SMARS over the years without
thorough regression testing; in 2005, Knight changed the cumulative quantity function
that counted the number of shares of the parent order that had been executed and
filled to decide whether to route another child order. The cumulative quantity function
was now invoked earlier in the SMARS workflow, which in theory was a good idea to
prevent excess system activity; in practice, it was now disconnected from Power Peg
(which used to call it directly), could no longer throttle the algorithm when orders
were filled, and Knight never retested Power Peg after this change.
Manual deployment
In the week before go-live, a Knight engineer manually deployed the new RLP code in
SMARS to its eight servers. However, the engineer made a mistake and did not copy
the new code to one of the servers. Knight did not have a second engineer review the
deployment, and neither was there an automated system to alert anyone to the
discrepancy. Knight also had no written procedures requiring a supervisory review, all
facts we shall return to later.
The crash
On August 1, 8:01 a.m. EST, an internal system called BNET generated 97 email
messages that referenced SMARS and identified an error described as “Power Peg
disabled.” These obscure, internal messages were sent to Knight personnel, but their
channel was not designated for high-priority alerts and the staff generally did not
review them in real-time; however, they were the proverbial smoke of the smoldering
code and deployment bits about to burn, and it was a missed opportunity to identify
and fix the DevOps issue prior to market open.
At 9:30 a.m. EST, Knight began receiving RLP orders from broker-dealers, and SMARS
distributed the incoming work to its servers. The seven servers that had the new RLP
35
code processed the orders correctly. However, orders sent to the eighth server with
the defective Power Peg code activated by the repurposed flag soon triggered the fault
line of a financial tectonic plate. This server began to continuously send child orders for
each incoming parent order without regard to the number of confirmed executions
Knight had already received from other trading venues.
The results were immediately catastrophic. For the 212 incoming parent orders
processed by the defective Power Peg code, SMARS sent thousands of child orders per
second that would buy high and sell low, resulting in 4 million executions in 154 stocks
for more than 397 million shares in approximately 45 minutes. For 75 of these stocks,
Knight’s executions jostled prices more than 5% and comprised more than 20% of
trading volume; for 37 stocks, prices lurched more than 10% and Knight’s executions
constituted more than 50% of trading volume.
Following the flash crash of May 6, 2010, in which the Dow Jones Industrial Average
(DJIA) lost over 1000 points in minutes, the SEC announced several new rules to
regulate securities trading.
1) Circuit breakers were required to stop trading if the market experienced what was
labeled as “significant price fluctuations” of more than 10 percent during a five-minute
period.
2) The SEC required more specific conditions governing the cancellation of trades. For
events involving between five and 20 stocks, trades could be cancelled if they were at
least 10 percent away from the “reference price,” the last sale before pricing was
disrupted; for events involving more than 20 stocks, trades could be cancelled if they
deviated more than 30 percent from the reference price.
3) Securities Exchange Act Rule C.F.R 240.15c3–5 (“Rule”) went into effect, requiring
the exchanges and broker-dealers to implement risk management controls to ensure
the integrity of their systems as well as executive review and certification of the
controls.
Since the flash crash rules were designed for price swings, not trading volume, they did
not kick in as intended and stop trading because few of the stocks traded by Knight on
that fateful day exceeded the 10 percent price change threshold.
By 9:34 a.m., NYSE computer analysts noticed that market volumes were double the
normal level and traced the volume spike back to Knight. Niederauer tried calling
Joyce, but Joyce was still at home recovering from knee surgery.
36
The NYSE then alerted Knight’s chief information officer, who gathered the firm’s top
IT people; most trading shops would have flipped a kill switch in their algorithms or
would have simply shut down systems. However, Knight had no documented
procedures for incident response, again, another fact we shall return to later. So, it
continued to fumble in the dark for another 20 minutes, deciding next that the
problem was the new code.
Because the “old” version allegedly worked, Knight reverted back to the old code still
running on the eighth server and reinstalled it on the others. As it turned out, this was
the worst possible decision because all eight servers now had the defective Power Peg
code activated by the misappropriated RLP flag and executing without a throttle.
It was not until 9:58 a.m. that Knight engineers identified the root cause and shut
down SMARS on all the servers; however, the damage had been done. Knight had
executed over 4 million trades in 154 stocks totaling more than 397 million shares; it
assumed a net long position in 80 stocks of approximately $3.5 billion as well as a net
short position in 74 stocks of approximately $3.15 billion.
How Knight Capital could have done things differently

and business leaders. Knight could have prevented the failure and minimized the
damage with a variety of modern software development and operating practices
(DevOps). Below, I describe eight of these measures and how they could have made a
difference for Knight Capital.
Use of Version Control
Do not run dead code. Instead, always prune dead code and use version control
systems to track the changes. You should not re-purpose configuration flags; rather,
activate new features with new flags.
Version control is any kind of practice that tracks and provides control over changes to
source code. Teams can use version control software to maintain documentation and
configuration files as well as source code.
As teams design, develop, and deploy software, it is common for multiple versions of
the same software to be deployed in different sites and for the software's developers
to be working simultaneously on updates. Bugs or features of the software are often
only present in certain versions (because of the fixing of some problems and the
introduction of others as the program develops).
37
Therefore, for the purposes of locating and fixing bugs, it is vitally important to be able
to retrieve and run different versions of the software to determine in which version(s)
the problem occurs. It may also be necessary to develop two versions of the software
concurrently: for instance, where one version has bugs fixed, but no new features
(branch), while the other version is where new features are worked on (trunk).
Writing Unit Tests
The purpose of unit testing is not for finding bugs. It is a specification for the expected
behavior of the code under test. The code under test is the implementation for those
expected behaviors. So unit tests and the code under test are used to check the
correctness of each other and protect each other. Later, when someone changes the
code under test, and it changes the behavior that is expected by the original author,
the test will fail. If your code is covered by a reasonable amount of unit tests, you can
maintain the code without breaking the existing feature. That’s why Michael Feathers
defines legacy code in his seminal book "Working Effectively with Legacy Code" as code
without unit tests. Without unit tests your development efforts will be a major risk
every time you change your legacy code.
Code Reviews
Code review is a systematic examination (sometimes referred to as peer review) of

source code. It is intended to find mistakes overlooked in software development,
improving the overall quality of software. Reviews are done in various forms such as
pair programming, informal walkthroughs, and formal inspections.
Automated Tests and Test Automation
In the world of testing in general, and continuous integration and delivery in particular,
there are two types of automation:
1) Automated Tests
2) Test Automation
While it might just seem like two different ways to say the same thing, these terms
actually have very different meanings.
Automated tests are tests that can be run automatically, often developed in a
programming language. In this case, we talk about the individual test cases, either
unit-tests, integration/service, performance tests, end-2-end tests, or acceptance
38
tests. The latter is also known as specification by example.
Test automation is a broader concept and includes automated tests. From my

perspective, it should be about the full automation of test cycles, from check-in up to
deployment – also called continuous testing. Both automated testing and test
automation are important to continuous delivery, but it's really the latter that makes
continuous delivery of a high quality even possible.
Had Knight implemented automated tests and test automation for the new and
existing SMARS functionalities they would have caught the error before deploying it in
production.
Automated Deployment Process
It is not enough to build great software and test it; you also have to ensure it is
delivered to market correctly so that your customers get the value you are delivering
(and so you don’t bankrupt your company). The engineer(s) who deployed SMARS are
not solely to blame here – the process Knight had set up was not appropriate for the
risk they were exposed to. Additionally, their process (or lack thereof) was inherently
prone to error. Any time your deployment process relies on humans reading and
following instructions you are exposing yourself to risk. Humans make mistakes. The
mistakes could be in the instructions, in the interpretation of the instructions, or in the
execution of the instructions.
Deployments need to be automated and repeatable and as free from potential human
error as possible. Had Knight implemented an automated deployment system –
complete with configuration, deployment, and test automation – the error that caused
the nightmare would have been avoided.
Step-by-Step Deployment Process Guide
Anybody (even somebody who is usually not doing this) should be able to deploy on
production with this guide on the table. Of course, the more you go into the direction
of automated deployment, the smaller this guide becomes, because all documentation
of this process is coded in your automated processes. The probability of doing
something wrong with a step-by-step guide (or a checklist) is a multitude smaller as
without. This has been proven many times in the medical space.
Timeline
The timeline was another reason Knight failed to deliver the RLP solution. Knight’s IT
39
project managers and CIO should have pushed back on the hyper-aggressive delivery
schedule and countered its business leaders with an alternative phased schedule
instead of the Big Bang – pun intended. Thirty days to implement, test, and deploy
major changes to an algorithmic trading system that is used to make markets daily
worth billions of dollars is impulsive, naive, and reckless.
Risk Management
Risk management is a vital capability for a modern organization, especially for financial
services companies. The SEC’s report (see References) concluded: “Although
automated technology brings benefits to investors, including increased execution
speed and some decreased costs, automated trading also amplifies certain risks. As
market participants increasingly rely on computers to make order routing and
execution decisions, it is essential that compliance and risk management functions at
brokers or dealers keep pace… Good technology risk management practices include
quality assurance, continuous improvement, controlled user acceptance testing,
process measuring, management and control, regular and rigorous review for
compliance with applicable rules and regulations, an independent audit process,
technology governance that prevents software malfunctions, system errors and
failures, service outages, and when such issues arise, a prompt, effective, and
risk-mitigating response.”
While Knight had order controls in other systems, it did not compare orders exiting
SMARS with those that entered it. Knight’s primary risk monitoring tool, known as
“PMON,” is a post-execution position monitoring system. At the opening of the
market, senior Knight personnel observed a large volume of positions in a special
account called 33 that temporarily held multiple types of positions, including positions
resulting from executions that Knight received back from markets that its systems
could not match to the unfilled quantity of a parent order. There was a $2 million gross
limit to the 33 account, but it was not linked to any automated controls concerning
Knight’s overall financial exposure.
Furthermore, PMON relied entirely on human monitoring, did not generate automated
alerts, and did not highlight the display of account exposures based on whether a limit
had been exceeded. Moreover, Knight also had no circuit breakers, which is a standard
pattern and practice for financial services companies.
40
Closing Thoughts
Although Knight was one of the most experienced companies in automated trading at
the time (and the software that goes with it), it failed to implement many of the
standard DevOps best practices that could have prevented this disaster at any number
of intervals.
Knight Capital Group Holdings was eventually acquired by another market making rival,
Virtu LLC, in July 2017 for $1.4 billion. The silver lining to the story was that Knight was
not too big to fail, and the market handled the failure with a relatively organized
rescue without the help of taxpayers. However, a dark cloud remains: market data
suggests that 70 percent of U.S. equity trading is now executed by high-frequency
trading firms, and one can only wonder when, not if, the next flash crash will occur.
References
> SECURITIES EXCHANGE ACT OF 1934 Release No. 70694 (2013).ADMINISTRATIVE
PROCEEDING File No. 3-15570. Available at:
https://drive.google.com/open?id=1NlUEumXKvkXFdVm7wnIIBXGGLLzI2XoV
41
Case Study 5: Workday did not work for

Sacramento Public Schools
“While Workday and Sierra-Cedar got paid… they put the district right back where it
started with nothing to show after two years.” — SCUSD
California’s Sacramento City Unified School District (SCUSD) thought they were getting
a good deal when they hired software as a service (SaaS) provider Workday and their
service partner Sierra-Cedar for a program intended to improve the management of
the district’s finances, payroll, and human resources (HR).
Unfortunately, SCUSD’s high hopes of cutting costs and increasing efficiency in these
areas were more than dashed. They were completely destroyed. The school district
alleges that after two years of major financial investment in the project, they were left
with zero results. Not only that, but in a complaint filed in August of 2018, the district
claims that each company infringed their contracts, falsely presented themselves, and
defied California’s business and professions code. In addition, SCUSD alleges that
Sierra-Cedar committed fraud.
Sierra-Cedar boasts a workforce of approximately 900 employees and has been in

business since 1995. They purport to provide consulting services that pertain to
upgrading, strategizing and implementing technologies specific to certain industries.
Sierra-Cedar claims to hold “over two decades of enterprise applications experience

across a variety of industries including Higher Education and [the] Public Sector.”
According to SCUSD, Sierra-Cedar stated that they would provide staff with specific
expertise and experience in their area of the public sector, namely K–12 education. Yet
the district alleges that no such staff was provided. Rather, they state that they were
given consultants with no knowledge or work history in the subject.
Workday is 10 years younger than its codefendant, beginning as a start-up in 2005 with
the goal of selling HR and finance technology. The company specifically targets many
different industries, including education, specifically calling out to K–12 schools and
school districts. Workday claims that their software can be used for teacher
recruitment, to gain useful HR information, and to reduce overhead, among other
services. It is designed to manage payroll and expenses, track absences, and organize
job candidates.
42
According to SCUSD’s complaint, Workday has more than 8,000 employees and brings
in billions of dollars in revenue each year. Workday had its beginnings with private
companies, but soon branched into work with public entities like local governments.
In spite of Sierra-Cedar and Workday’s alleged failures to perform contractually

required work for the Sacramento schools, the companies raked in enormous amounts
of cash in the project. The school district claims that the contracts were written in such
a way that both companies were able to extract enormous fees whether or not they
performed up to par.
The district extended the project’s end date twice, presumably hoping to see an
improvement in results, but none came. Finally, with a cringe-worthy payroll testing
quality of, at best, a mere 70 percent, SCUSD threw in the towel. The project was never
implemented and the district lost a serious amount of money. As the district so sharply
stated:
“While Workday and Sierra-Cedar got paid… they put the district right back where it
started with nothing to show after two years.” – SCUSD
These harsh words showcase the depth of the issue at hand.
As a result of claimed damages and a demand for the recovery of fees paid, SCUSD is
suing for $5.2 million and they need that money badly. These days the district is in dire
straits, suffering the threat of insolvency. It may soon have no more funds left and risks
being taken over by the state.
The district claims that, should they be taken over, it will take 10 years to recover and
the process will be highly detrimental to the student body.
Timeline of Events
2013
SCUSD serves more than 43,000 students across 77 campuses and employs more than
4,200 individuals. The district operates with an annual budget of more than $500M in
annual spend. In 2013, they determined the district’s HR and ERP systems did not
position them to effectively move forward.
The district’s chief business officer (CBO), with the help of a consultant, set out to
identify possible replacement options. In early 2013, the district personnel engaged
with the Workday sales team and began to discuss their requirements. Over the course
43
of a number of months, through the exchange of phone calls and emails, it was
determined that Workday could meet the district’s requirements (even though
Workday had no K–12 implementations).
Workday presented CedarCrestone as a qualified implementation partner and the

project was initially presented to the board with a budget of $816K for the first year of
Workday cloud fees. The maximum estimate for the time and materials contract
presented by CedarCrestone was $3,871,000. The project was scheduled to run from
August 2014 through October 2015.
2014
In July 2014, Sierra-Cedar was formed by combining the operations of Sierra Systems
US and CedarCrestone.
In October 2014, the numbers were adjusted to $1.275M in annual fees to Workday,
and Sierra-Cedar’s anticipated fees were reduced to $3.098M. It is not clear if the
amount of Sierra-Cedar’s PO represented the full maximum estimate.
2015
In August 2015, the CBO left SCUSD. Around that same time, it was determined that
the targeted January 2016 go-live date would not be met. The school district claims
that they pressured Workday and Sierra-Cedar to provide a go-live date that would
absolutely be met. The parties agreed to a target go-live of July 2016.
2016
As the go-live date approached, the project team could not achieve payroll testing
quality of more than 70 percent. With more than $250M in annual payroll at risk,
SCUSD made the call that they would not go live.
Additional negotiations took place with Workday and Sierra-Cedar, in an attempt to

salvage the program. In November 2016, SCUSD decided to kill it.
2017
In July 2017, SCUSD named a new superintendent. And in August 2018, SCUSD filed a
suit in Sacramento Superior Court. The lawsuit alleges that Sierra-Cedar committed
fraud and that both Workday and Sierra-Cedar violated the business and professions
code, misrepresented themselves, and breached contracts they had with the district,
44
among other things.
What Went Wrong

According to the complaint filed by the school district, Workday and Sierra-Cedar had
agreed to provide a cloud-based software system to suit their needs. SCUSD chose this
particular software in hopes that it would improve many of their most important
operations. Yet, after spending millions on the system and related services, the district
claims they were left empty-handed.
No experience with K-12 schools
The district argues that Workday and Sierra-Cedar used SCUSD as a test experience
with the schools in order to “tout and enrich themselves as experts in elementary and
high school, or K-12, education technology ‘solutions.’” Most significantly, the district
argues for the return of the fees spent on what should have been functioning
technology which, they say, was agreed upon but never delivered.
They allege that Workday never suggested that their systems could not be used
effectively in K–12 schools. In fact, SCUSD claims that company representatives
presented the software as being fully capable of handling the CBO’s specific
requirements. In other words, SCUSD placed their trust in these companies to do as
they promised.
However, SCUSD now believes that they were duped. Workday’s intentions, rather
than to provide a cloud system to suit the needs of Sacramento’s schools, was,
according to the district, a means to enter into the K–12 sector with as little financial
risk as possible for what they refer to in their complaint as a “practice run.”
Furthermore, SCUSD believes that Sierra-Cedar was well aware of Workday’s
intentions.
The district claims that Sierra-Cedar was presented as the only option available for
project implementation and that they were, thus, qualified to do so. A series of
contracts were drawn up which the district assumed to be fair following the many
conversations SCUSD’s CBO had shared with Workday regarding the project’s
parameters. Workday, they believe, was under obligation to provide staff experienced
in the K–12 sector and that were capable of getting the job done. Sierra-Cedar vowed
to provide such staff for implementation.
In spite of their promises to provide experienced staff, Sierra-Cedar SCUSD the district
with a project manager with no experience in the K–12 sector. This was also allegedly
45
true of the additional staff they provided.
Due to the staff’s alleged lack of understanding and knowledge of the K–12 sector,
they had great difficulty in handling the schools’ data and organizing the technology to
suit the district’s needs. SCUSD claims their data mapping and conversion was regularly
late, incorrect, or only partially completed, which led to extra expenses and delays.
One-sided contracts
The financial side of the project was problematic. The district believes that the
companies wrote the contracts in such a way that all the financial risk fell on the
schools. Before the project was made live, Workday required the payment of two fees
of over $1.5 million. Training payments were requested to be made before training had
begun. In addition, the district was to pay an hourly rate to Sierra-Cedar.
Workday promised many things: regular reviews of Sierra-Cedar’s work, a subscription

to Workday’s software as a service (SaaS), training for school staff, and support
throughout the project. Various steps were agreed upon which would be applied in the
course of project implementation. It was understood that Workday would check up on
Sierra-Cedar and ensure they were performing in an agreed-upon manner. This did not
happen
Payroll accuracy
Another important aspect of Workday’s SaaS solution was that it was supposed to
handle the schools’ payroll. Yet, between January and June 2016, Sierra-Cedar was
unable to get the payroll accuracy to over 70%. Though Sierra-Cedar and Workday
assured the district that they could go ahead and implement payroll, the accuracy
numbers from the test were too low for the district to agree. Moreover, the district
claims that the companies would not assure them that payroll would be correct if they
did decide to implement it. Thus, the schools chose not to do so.
No usable outcome
By November 2016, SCUSD decided it was not possible to continue the project in spite
of investing millions of dollars in it. They claim that in 2016, the proposals that the
companies made were entirely insufficient and not possible to put into action. On
November 22, SCUSD notified Workday and Sierra-Cedar that they were putting an end
to their relationship.
The school district believes that the project was an enormous loss of funds and time
46
for them, while Workday and Sierra-Cedar exited the contract with both K–12
experience they could market to other customers and a gain of millions of dollars. In
other words, the district claims that the companies made significant money using
SCUSD as their K–12 guinea pig while enriching themselves.
In fact, on their website, Workday describes the company as a program which

“streamlines the ‘business’ of education” for K–12 school districts by providing
assistance in business planning, financial management, human capital management,
and prism analytics. SCUSD believes they were the testing ground for forming this
K-12–specific focus.
In the end, the school district sued the two companies for the following:
> Breach of Contract against Sierra-Cedar for not performing their contractual
obligations;
> Breach of Contract against Workday for not providing the experienced K–12 sector
experts they were promised, not offering sufficient check-up, and not fulfilling their
contractual duties;
> Breach of Covenant of Good Faith and Fair Dealing against Sierra-Cedar for failing to
truly attempt to make the project go live and instead giving insufficient fluff proposals;
> Breach of Covenant of Good Faith and Fair Dealing against Workday, for not offering
true and real support to successfully release the project on the go-live date;
> Fraud against Sierra-Cedar for agreeing to provide staff with K–12 experience and
expertise in the contract, then proceeding to do no such thing. In fact, SCUSD claims
that Sierra-Cedar knowingly used whatever staff they had available for the project and
took advantage of the opportunity by using the school district as a K¬¬–12 training
ground while gaining a substantial hourly rate. If true, Sierra-Cedar would have
knowingly misrepresented themselves. The school district claims that this
misrepresentation caused a breach of their rights;
> Negligent misrepresentation against Sierra-Cedar for stating that they would offer
experienced employees and then not doing so;
> Violation of the Business and Professions Code against Sierra-Cedar for unfair
competition through their alleged misrepresentation and for allegedly using SCUSD as
a K–12 experiment;
47
> Negligent misrepresentation against Workday for presenting Sierra-Cedar as being

capable of carrying through on the project; and
> Violation of the Business and Professions Code against Workday for unfair
competition through their claim that Sierra-Cedar employees were experienced and
could complete the work well.
In the end, Sacramento City Unified School District claims to have paid $3,240,955.45
in fees to Sierra-Cedar, a substantial sum for any school district. Moreover, they never
received the benefits which were promised by Workday representatives way back in
the early conversations with the school’s Chief Business Officer Forrest.
For the school district, the project with Sierra-Cedar and Workday was a disaster.
How SCUSD Could Have Done Things Differently

If SCUSD’s allegations are true, both Workday and Sierra-Cedar were deeply in the
wrong. But SCUSD also has to think hard about their own role in this project failure.
If you sign up a trusted advisor, make sure they are qualified.
The lawsuit claims that SCUSD used an experienced K–12 consultant to assist with the
ERP selection process. This program looks like a disaster from top to bottom. It is not
clear what experience the initial consultant had. Perhaps SCUSD is suing the wrong
party. Or maybe the advice was sound, and SCUSD just ignored it.
Get IT on board.
The lawsuit makes no mention of IT. The published board documentation makes no
reference to IT. Any well-qualified CIO would have likely stopped the idea of planning
to use software that had not had a thorough industry shakedown, particularly in the
public sector where budgets are tight and regulatory compliance requirements are
high.
Get a decent lawyer on board.
The suit claims that simple boilerplate contracts were used that offered very little
protection to SCUSD. They were largely relying on the “good word” of their suppliers.
Given this was a $5M spend that involved a system that controlled the payment of
more than $400M in salaries and benefits, you would have thought that somebody
from Legal would have been a little more forceful in protecting SCUSD.
48
Never be first … unless you are willing to have the project blow up in your face.
SCUSD could have and should have known that there were no Workday K–12
customers. A simple reference check would have told them that. Any company that
signs up to be the beta test for any software should do so with some form of significant
financial gain possible.
If you do want to be the first, make sure the vendor is paying their share.
If Workday and Sierra-Cedar wanted to penetrate the K–12 market so badly,

Sacramento could have likely gained major contract concessions to offset the risk they
were willing to take. Having an experienced negotiator on their side could have saved
SCUSD millions of dollars up front and likely increased the probability of success with
the vendors having more skin in the game.
Do quality checks.
As a part of the Workday implementation practice, quality checks are performed, and
the results are reported to the client. The lawsuit, while mentioning the quality checks
were to be performed, makes no mention of what was reported and what the issues
were that the team might be dealing with. It is entirely possible that Workday
identified issue(s) described in this article. And even if they didn’t, as the paying client,
you are always responsible for doing quality checks yourself as well. Never trust a third
party to sign off on quality.
Be ready for the project and engage yourself.
Given the apparent lack of participation of IT and Legal, and the apparent lack of
quality and detailed requirements documentation, it is highly probable that there was
insufficient participation throughout the program. If quantifiable participation was
called out as a part of the boilerplate contract, then this is going to be a heavy lift for
SCUSD’s legal counsel.
Closing Thoughts
The loss of millions of dollars down the drain for Sacramento’s public school district
doubtlessly affects teachers, students, staff, and parents. SCUSD had high hopes of
implementing a tech program that would save them time and money. In the end,
dollars and hours were simply wasted.
49
Yet the school district is not a wholly innocent victim here, even if the allegations prove
true.
Staff and school officials did not do their due diligence in checking up on the
company’s background in K–12 or looking into the project managers they had on staff.
Rather than simply accepting Workday’s word that they had experienced professionals
at Sierra-Cedar, SCUSD ought to have looked deeply into the matter.
The schools would have done well to request specific evidence of Workday’s SaaS
solution working successfully in other school districts. With a bill in the millions and
hours of invested time, it seems like the least they could have done.
The district should have more carefully checked the language in the contracts to
ensure that the companies could not have exited the project with the cash without
delivering a functional product.
Regardless of the Sacramento Superior Court’s final decision, it’s clear that both the
plaintiff and the defendants could have done better.
References
> Sacramento Superior Court (2018). Sacramento City Unified School District vs.
Workday Inc a Delaware Corporation. No. 2018-00238302. Available at:
https://drive.google.com/open?id=1r5ajcFjRZqaTmulL7b3cxOj_TiRwduer
> Meeting Minutes Sacramento City Unified School District Board of Education (2014).
Available at:
https://drive.google.com/open?id=1zOWuCrIJOsbVCPGy91-ySTJ6eJhaG8Co
50
Case Study 6: How Revlon Got Sued by Its Own

Shareholders Because of a Failed SAP
Implementation
“As a result of the poor preparation and planning of the implementation of the ERP
system, Revlon was unable to fulfill product shipments of approximately $64 million
of net sales," — Rosen Law Firm
Cosmetics company Revlon’s roots stretch back to its 1932 nail polish launch by
Charles Lachman and brothers Charles and Joseph Revson. Since its early days, much
has changed.
The company’s products have found their way into the everyday lives of individual
customers and onto the shelves (digital or physical) of major retailers across the world.
Though best known for their makeup, Revlon sells a startling range of cosmetics
including hair coloring kits, deodorants, and fragrances to both men and women
globally.
Last year on a call following the announcement of the company’s first-quarter results
for fiscal 2018, Revlon announced that its SAP ERP implementation was a disaster. This
came on the heels of delayed financial reporting because of said SAP implementation
failure.
The company’s stock fell 6.9 percent within 24 hours of reporting the news, which led
to investor lawsuits against the company. Not good, especially for a publicly traded and
well-known consumer product company.
Some four law firms have filed class action suits to date, based on the U.S. beauty
behemoth’s quarterly results where it confessed it was unable to fulfil orders to the
tune of US$64 million because of its inability to record and account for inventory.
“In early February [2018], we rolled out SAP for a large part of our North American
business to integrate planning, sourcing, manufacturing, distribution and finance…
However, we experienced issues during the SAP changeover that caused the plant to
ramp up capacity slower than anticipated,” Christopher Peterson, COO, told investors
on this call.
51
"As a result of the poor preparation and planning of the implementation of the ERP
system, Revlon was unable to fulfill product shipments of approximately $64 million of
net sales," claims a lawsuit filed in the U.S. District Court in New York by Rosen Law
Firm on behalf of an investor. The lawsuit was announced late last week and seeks
class action status.
Revlon is the latest in a string of recent SAP enterprise resource planning (ERP) failures.
Lidl, National Grid, and Haribo are just a few other companies that have experienced
such massive challenges.
ERP problems, including implementation failures, are common. But an ERP problem
that results in an investor lawsuit is rare. The more common disputes are between ERP
customers and the system implementers.
So what happened at Revlon?
Timeline of Events
2016
In June 2016, U.S. cosmetics company Revlon announced its intention to buy Elizabeth
Arden Inc. for US$870M. The acquisition was completed in September of the same
year.
Revlon then kicked off 2017 with a major restructure, cutting jobs and streamlining
operations as part of the acquisition and integration. This activity alone was projected
to cost between $65 million and $75 million over the coming four years. Revlon also
acquired the Cutex brands in the same year.
Elizabeth Arden was an early adopter of Oracle Fusion Applications according to a

Forrester report of 2013. In the 2008 annual report for Elizabeth Arden, the chairman,
president and CEO E. Scott Beattie was quoted as saying, “Our global efficiency
re-engineering project remains on track and we currently anticipate savings of
approximately $10 million to $12 million by the end of fiscal 2009 and additional
savings of approximately $13 million to $15 million by the end of fiscal 2010. A
significant component of our global efficiency re-engineering project is the
implementation of an Oracle financial accounting and order processing system. During
fiscal 2007, we successfully implemented the same Oracle solution in our Greater
China business.”
52
Revlon had previously chosen Microsoft Dynamics AX which was an equal success
story. Revlon ran 50 different business entities with a plan to collapse 21 separate ERP
systems into one. Dynamics AX was to be implemented at Revlon to act as one unified
organization under the leadership of David Giambruno, senior VP and CIO of Revlon
back in 2013.
Giambruno was quoted as being in favor of a “no customization rule” within Revlon.
The base implementation and the first company took a year to go live. The second
country went live in 14 days and the third country in less than a day! Giambruno left
Revlon in November 2013. Juan Figuereo, a new Revlon CFO, joined the company in
April 2016 only to retire just over a year later in mid 2017.
Fast forward to today, and it comes as no surprise that off the back of the acquisition,
Revlon considered migrating to something different. Colomer, the Barcelona-based
hair and beauty products firm that was acquired by Revlon in 2013, ran on SAP. The
subsidiary Red Door Spas, part of the Arden business, also ran on SAP. Colomer/Revlon
implemented HANA in April 2014. In the middle of the project, Colomer was acquired
by Revlon; this affected the project, in that the landscape had to be moved from Spain
to the USA.
When exactly the decision to switch to SAP was made, or why, I can’t say; however,
this decision was already on the cards per the fiscal year ending December 31, 2016
filing. Ahead of the implementation the filing even stated that
This system implementation subjects the Company to substantial costs, the majority
of which are capital expenditures, and inherent risks associated with migrating from
the Company's legacy systems. These costs and risks could include, but are not
limited to:
> inability to fill customer orders accurately or on a timely basis, or at all;
> inability to process payments to vendors accurately or in a timely manner;
> disruption of the Company's internal control structure;
> inability to fulfill the Company's SEC or other governmental reporting requirements
in a timely or accurate manner;
> inability to fulfill federal, state and local tax filing requirements in a timely or
accurate manner;
53
> increased demands on management and staff time to the detriment of other
corporate initiatives; and
> significant capital and operating expenditures.
If the Company is unable to successfully plan, design or implement this new SAP ERP
system, in whole or in part, or experience unanticipated difficulties or delays in doing
so, it could have a material adverse effect on the Company's business, prospects,
results of operations, financial condition and/or cash flows.
While statements like this are not entirely unusual, it is a red flag to investors, or
should be, indicating that the business anticipates problems and chooses to protect
itself accordingly with such caveats.
2018
Revlon said on its earnings conference call in March 2018 that while it remained on
track with the Elizabeth Arden integration activities, it was seeing lower sales volumes,
but that they expected the total estimated synergies to be impacted over the
multiyear period.
At the time the specifics of the implementation of the new SAP ERP system, which
went live at the beginning of February, was said to be overall on schedule in terms of
the implementation steps. "Production capabilities and inventory recovery has been
slower than expected, affecting our current order fulfilment levels."
Revlon said it had taken "immediate actions to address the situation, have
implemented a robust service recovery plan and have communicated with our key
customers." This suggests many temporary hands in warehouses to pick, pack and ship
as well optimize and cut backorders.
As Revlon COO Christopher Peterson stated in 2018, Revlon had implemented SAP for
a huge portion of the North American company for improved supply chain integration
to increase efficiency and improve working capital management, but the SAP
changeover resulted in an initial slowing of plant operations, disrupting the
manufacture of finished goods and fulfilling orders to large retailers.
Revlon attributed to the changeover a reduction of $20M in net earnings in one

quarter alone, accompanied by $10M in unplanned expenses including non-recurring
54
labor to improve customer support. At the time (2018), Revlon had implemented SAP
in 22 countries on the Revlon heritage side of the company. Apparently, the Arden
switch-out of JD Edwards had not even begun at that stage.
2019
A year later, in March 2019, CFO Victoria Dolan said Revlon had spent $32M in 2018 on
operating activities in comparison to 2017, taking the costs of the migration to $54M;
understandably, profits and stock prices dipped. Revlon reported increased losses. The
fiscal year 2018 (ended on December 31) closed with $294.2 million dollars in the red,
compared to $183.2 million-dollar losses registered in 2017.
Ironically the results were due to a drop in sales of all its business categories, except
Elizabeth Arden, partly caused by the breaks in service levels directly attributed to the
SAP implementation. Revlon also qualified the losses by saying that the rise in losses
was also related to the re-acquisition of some rights connected to the brand Elizabeth
Arden.
In March 2019, the United States Securities and Exchange Commission (SEC) made
public Revlon’s first-quarter 10-K filing, thus illustrating to the wider world the dire SAP
ERP issues in the Oxford plant and their wide-reaching effects on the business. Around
the time of the SEC’s release, the company’s chief financial officer made clear to the
public that the SAP ERP issues had been resolved and that the Oxford plant was up and
running as normal.
After the March 2019 financial statement stock prices improved.
Yet, in May of 2019, Revlon faced a stream of painful class action lawsuits from firms
Bragar, Eagel, & Squire; Rosen Law Firm; Wolf, Haldenstein, Adler, Freeman, & Herz;
and Zhang Investor Law. Ouch.
In the class action suit, specific reference was made to the Oxford center in North
Carolina where the business was supposedly unable to fulfil product imports of about
$64M worth of revenue.
What Went Wrong

As if the above weren’t enough Revlon also reported that:
> The company was unable to recover lost sales from the SAP failure
55
> Customer service levels were disrupted
> There were increased demands on its management team and staff, which
cannibalized focus on other corporate priorities
> It incurred significant capital and operating expenditures
> It experienced difficulty processing payments to vendors
> It was unable to fulfill federal, state and local reporting and filing requirements in a
timely or accurate manner
> It had greater than expected expedited shipping fees, resulting from the customer
firefighting stemming from the SAP failure
> It suffered from an “inability to fill customer orders accurately or on a timely basis, or
at all”
How Revlon Could Have Done Things Differently

and business leaders.
Incremental Software Implementation Strategy
Perhaps Revlon should have considered starting small. The disaster that was their SAP
ERP implementation in the Oxford plant might have been avoided if the company had
used a narrow roll-out strategy for the software.
When a company utilizes an incremental roll-out strategy the former system is slowly
replaced as the new SAP ERP is rolled in. The benefits of such a strategy are obvious: if
the new software is implemented piece by piece, potential problems can be sussed
out, understood, and handled on a small-scale without causing serious damage to the
company’s reputation, efficiency, or bottom line. Even if large-scale issues occur in the
future, the company will have experience getting such matters under control and will
be more likely able to snuff out the problem quickly.
Furthermore, employee training may be more gradual and thorough should the
company choose a roll-out rather than big-bang strategy. Opting for a big bang means
that they have selected to implement the new SAP ERP across the board
simultaneously.
56
Revlon could have chosen to implement a roll-out strategy on a less vital company
entity first, thus working out the kinks of the software-business partnership, before
gradually stepping into major company entities like the Oxford plant.
Risk Management Is Project Management for Adults
Revlon, like many companies implementing SAP S/4HANA and other ERP solutions,
didn’t seem to understand the risks associated with their transformations. Worse yet,
they must not have quantified or implemented effective strategies to mitigate those
risks.
In this case, Revlon experienced shipping delays and lost sales due to production
stoppages at its North Carolina plant – the location of the first phase of its SAP go-live.
Had Revlon understood, quantified, and mitigated the inherent ERP implementation
risks, it would have taken action to ensure that its go-live didn’t materially affect its
operations.
Had Revlon had a sufficient emergency plan in place, they might have amended the
Oxford disaster in a far shorter period of time, and thus caused less damage to the
company. How many of the issues which arose in 2018 could have been foreseen?
Could solutions have been found in advance of implementation?
If Revlon had better planned for such problems, perhaps the problems could have
been avoided in the first place or even simply nipped in the bud.
As Tim Lister says, risk management is project management for adults.
Organizational Implementation Readiness
Prior to implementing SAP, Revlon had some significant operational issues related to
its acquisition and integration of the Elizabeth Arden brand. As it outlined in its
financial filings, it was having trouble integrating the operations, processes, and
organizational structure with its newly acquired company. An SAP implementation is
likely to fail against this type of backdrop.
Instead of focusing on the futile attempt to roll out new SAP ERP software, the
company should have built a stronger business blueprint and foundation for the overall
transformation. This would have begun with defining how Elizabeth Arden would fit
57
into its overall operations, as well as defining clear business processes for the
implementation.
This SAP S/4HANA implementation readiness phase is a critical but often overlooked
ingredient to transformation success.
Closing Thoughts
While the circumstances surrounding the newest ERP lawsuit are unique, as are the
plaintiffs, the events that led to the failure of Revlon’s SAP system are all too familiar.
It may be tempting to blame the SAP software or whoever the system integrator was,
but Revlon and others should recognize that they ultimately have to own the results of
these sorts of transformations.
For starters, it does not seem as if Revlon understood either the size of the projects
nor any of the risks inherent in rolling out a new ERP software system. It doesn’t
appear as if the cosmetics giant developed any strategies to offset the possible risks.
Complicating matters, the company was encountering major problems integrating the
Elizabeth Arden brand into its business. Rather than focusing on getting the new ERP
system up and running, it should have first addressed the operational problems. Any
ERP implementation is likely to fail under these circumstances.
Furthermore, the company admitted in the SEC filing that it had “material weaknesses”
with its own internal controls caused by the ERP implementation. Having an effective
business plan before starting an ERP project can help companies to avoid this.
But without question, the largest problem for Revlon was that as a result of the ERP
train wreck the project resulted in a negative ROI stemming from production and
operational issues when Revlon brought the SAP system online.
Costly failures such as the one Revlon encountered can be avoided. Doing so requires
top management to own the project and develop a plan to smoothly meld the
technology into the business process, not the other way around.
References
> US SECURITIES AND EXCHANGE COMMISSION, Commission File Number: 1-11178
REVLON, INC., Form 10K
58
> US SECURITIES AND EXCHANGE COMMISSION, Commission File Number: 1-11178

REVLON, INC., Form 10Q
> CLASS ACTION COMPLAINT FOR VIOLATIONS OF THE FEDERAL SECURITIES LAWS,
UNITED STATES DISTRICT COURT EASTERN DISTRICT OF NEW YORK, Case
1:19-cv-02859, Document 1, Filed 05/14/19
> REVLON REPORTS SECOND QUARTER 2018 RESULTS
> REVLON REPORTS THIRD QUARTER 2018 RESULTS AND ANNOUNCES 2018
OPTIMIZATION PROGRAM
59
Case Study 7: Target’s $2.5 Billion Cross-Border

Expansion Mistake
“After extensive internal due diligence and research, paired with counsel from
outside experts, we fulfilled our obligation to do what was right for our company and
our shareholders, and made the decision to exit Canada, — Brian Cornell, Target’s
Chairman and CEO
Less than two years after entering Canada, Target shocked the retail world by pulling
out. After accumulating $2.5 billion in losses, the Minneapolis-based company shut
down all of its 133 Canadian locations and laid off 17,600 employees.
In a blog post, Brian Cornell, Target’s chairman and CEO, said the decision to exit
Canada was the best option available to the company. “After extensive internal due
diligence and research, paired with counsel from outside experts, we fulfilled our
obligation to do what was right for our company and our shareholders, and made the
decision to exit Canada,” he wrote.
Target is one of America's largest and most successful retailers. The 114-year-old
company that evolved out of the old Dayton-Hudson Corporation now has more than
1,800 retail locations.
What most Americans with three Target stores right in their neighborhood don’t
realize is that Target isn't a worldwide thing. Walmart, by contrast, operates over
11,000 stores in 28 countries. Walmart is a $465 billion company. Target is a $72 billion
company, certainly not small potatoes. But Target, it seems, wanted to be more like
Walmart.
Target was a careful, analytical and efficient organization with a highly admired
corporate culture. The corporation’s entry into Canada was uncharacteristically
bold—not just for Target, but for any retailer. Under Gregg Steinhafel (Target’s CEO at
the time), the company paid $1.8 billion for the leases to the entire Zellers department
store chain in 2011 and formulated a plan to open 124 locations by the end of 2013.
Not only that, but the chain expected to be profitable within its first year of
operations.
That should have been easy, right? After all, Americans and Canadians speak the same
language (ignoring the French-speaking Québécois) and most Americans somehow
seem to think of Canada as their 51st, more polite, colder state to the north.
60
But it's not that simple. Take these two factors as an example. Canada has a different
currency. Sure, it uses dollars, but at the time of this writing a Canadian dollar is worth
only 72 percent of an American dollar. That conversion rate is constantly fluctuating.
Also, Canada uses the metric system. To the people in the U.S., a 2-foot deep shelf is a
2-foot deep shelf. In Canada, that shelf is 60.96 centimeters.
You can already begin to see the IT challenges, can't you?
An inventory system that was set up to handle U.S. dollars would need to be updated
to handle Canadian dollars. If the system didn't already have a currency field, that
would need to be added throughout. Conversion methods would need to be added.
And, for an inventory management system that has to fill shelves, knowing the size of
product packaging would be important. Software that calculates area for placement
would have to be modified to handle multiple measurements and measurement
systems.
Add to that issues of sourcing of products and pricing. The products don't all just come
from the U.S. A box of small widgets in the U.S. might be 12 inches tall. But that same
product packaged for the Canadian market might only be 11 1/2 inches tall, or
whatever that might translate to in centimeters.
You get the idea. Internationalizing an IT system is a lot of work. For an IT system
tracking the amount of data that an enterprise the size of Target needs, you're talking
about a lot of development and customization.
But let’s start from the beginning.
Timeline of Events
2010
Everything started with Richard Baker, the executive chairman of Hudson’s Bay Co.
(HBC). Although Baker is a retail executive, he is, at heart, a real estate man. His
maternal grandfather started buying and selling real estate in New York City in 1932
and helped pioneer the concept of shopping malls. Baker’s greatest business insight
was to recognize the value of the property developed by both his grandfather and his
father. In the 1990s, he started selling some of it off to various companies, including
Walmart.
61
That relationship proved fortuitous in late 2010, when Walmart approached him and
offered to buy the Zellers chain from HBC. Baker realized there was more value to
Zellers’ real estate than to the operation itself, since Walmart had soundly beaten the
brand. An astute dealmaker, Baker and his team reached out to Target to stoke the
company’s interest.
It was an open secret that Target was interested in the Canadian market. But the
company had previously decided it wanted to grow as quickly as possible if it were to
enter Canada, rather than pursue a slow, piecemeal expansion. The challenge was in
acquiring enough real estate to make that possible. The Zellers sale provided just such
an opportunity. After Baker’s team let Target know Zellers was on the block—and
Walmart was interested—the American company acted quickly to finalize its own
offer.
2011
Walmart would eventually back out, but Target put down $1.8 billion. Steinhafel
bought everything, essentially committing the company to opening stores as quickly as
possible to avoid paying rent on stores that weren’t operational and leaving landlords
without anchor tenants. The price Steinhafel paid raised eyebrows. “When the
numbers got up as high as they did, we found that pretty surprising,” says Mark Foote,
the CEO of Zellers at the time.
Almost immediately after signing the deal, employees in Minneapolis were seconded
to work on the Canadian launch. It was considered a privilege to be recruited. “The
company was pouring in resources left, right and sideways, so it was palpably exciting
in Minneapolis,” says a former employee.
Early 2012
By early 2012, with the planned opening still a year away, the nerve center for the
Canadian launch had moved from Minneapolis to Mississauga, Ontario, and waves of
American expats settled up north. Hiring was a top priority.
Fall 2012
Strange things started happening in 2012, once ordering began for the pending launch.
Items with long lead times coming from overseas were stalled—products weren’t
fitting into shipping containers as expected, or tariff codes were missing or incomplete.
Merchandise that made it to a distribution center couldn’t be processed for shipping to
a store. Other items didn't fit properly onto store shelves. What initially appeared to
62
be isolated fires quickly became a raging inferno threatening to destroy the company’s
supply chain. It didn’t take long for Target to figure out the underlying cause of the
breakdown: The data contained within the company’s supply chain software, which
governs the movement of inventory, was riddled with flaws.
Thus, “data week” was held in the fall of 2012. Merchandisers essentially had to
confirm every data point for every product with their vendors. A buyer might have
1,500 products and 50 to 80 fields to check for each one. The more experienced
employees had the foresight to keep records of verified information (dubbed “sources
of truth”), which made the task a little easier. Others weren’t so lucky.
Complicating matters was the dummy information entered into the system when SAP
was set up. That dummy data was still there, confusing the system, and it had to be
expunged. “We actually sat there and went through every line of data manually,” says
a former employee. “It was terrible.” Target anticipated how awful it would be and
designed the week to help keep employees sane. To kick it off and rally spirits, a few
employees performed a hip-hop song-and-dance routine on the first day. Ice cream
and pizza flooded in to keep employees fueled up; some stayed at work well past
midnight that week, squinting at screens through bleary eyes.
February 2013
The grand opening of Target Canada was set to begin in one month, and they needed
to know whether the company was actually ready. In February 2013, about a dozen
senior-level employees gathered at the company’s Mississauga headquarters to offer
updates on the state of their departments. Tony Fisher, Target Canada’s president, was
holding these meetings every day as the launch date crept closer. The news was rarely
good.
The news at this particular meeting was no different. The company was having trouble
moving products from its cavernous distribution centers and onto store shelves, which
would leave Target outlets poorly stocked. The checkout system was glitchy and didn’t
process transactions properly. Worse, the technology governing inventory and sales
was new to the organization; no one seemed to fully understand how it all worked.
The 750 employees at the Mississauga head office had worked furiously for a year to
get up and running, and nerves were beginning to fray. Three test stores were slated to
open at the beginning of March, followed shortly by another 21. A decision had to be
made, and they made it. They opted to go ahead.
March 2013
63
On March 4, 2013, Tony Fisher led a gaggle of reporters through a new Target location
in Guelph, Ontario. The store officially opened the next day, along with two others in
the province.
The company had been teasing consumers for a year at this point, starting with a
pop-up shop in Toronto featuring designer Jason Wu. There had also been a
high-profile ad during the Academy Awards to hype the Canadian launch, and actors
Sarah Jessica Parker and Blake Lively were lined up to appear at the grand opening.
Workers were still stocking shelves at the time, and signs throughout the store read,
“We’re open (mostly).” The three Ontario stores were part of Target’s soft launch, and
the company explained in a press release that the goal was to use them to iron out
kinks and “determine operational readiness” before opening 21 more locations as part
of its official launch that month.
Fall 2013
In the fall of 2013, hundreds of Target Canada head office staff piled into the
auditorium at the Mississauga Living Arts Centre for a state-of-the-union address from
their leaders. The employees were weary and frustrated by this point. The bulk of the
124 stores had opened, and it was clear the launch had gone seriously awry.
Consumers were frustrated when confronted with empty shelves, and the media and
financial analysts were hammering the company for it. On stage, Fisher stated his
conviction that Target Canada was making progress and that 2014 would be a greatly
improved year.
A Q&A session followed; one employee bravely asked Fisher what he would do
differently if he could do the launch over again. A man in the front row stood up and
offered to field the question. Taking the microphone, Steinhafel, Target’s CEO, didn’t
hesitate with his answer: He would renegotiate the real estate deal that facilitated the
company coming to Canada in the first place.
February 2014
In February 2014 Target headquarters released its annual results, revealing a

US$941-million loss in Canada. The company attributed the shortfall to growing pains,
expansion costs and—because of all that excess inventory sitting in
warehouses—significant markdowns. “As we enter 2014 with a much cleaner
inventory position, the team’s number one operation focus is on in-stocks—ensuring
64
we have the right quantity of each item in the right place at the right time,” Steinhafel
said on the earnings call.
It was his last as Target CEO. A month prior, Target had disclosed a massive security
breach in which hackers stole the personal information of 70 million customers in the
U.S. Combined with the bleeding operations in Target Canada, Steinhafel’s position
was untenable, and he stepped down in May. (He walked away with US$61 million in
compensation.) Fisher, who was hand-picked by Steinhafel, left the company two
weeks later. Mark Schindele was his replacement.
June 2014
Target Canada released an apology on YouTube; the video featured employees and
executives reflecting on the challenges of the first year and confessing to their sins.
“Maybe we didn’t put our best foot forward when we entered into Canada,” said
Damien Liddle, the company’s senior corporate counsel. “Certainly we know we’ve
disappointed our Canadian guests.” The video was remarkably candid as far as
corporate mea culpas go but maintained an optimistic note. “We’re headed in the right
direction now,” said another employee in the video. “For sure.”
December 2014
Target employees were beginning to feel like there was a light at the end of the tunnel.
The company had a much better handle on its technology, its data and the supply
chain, and every day no longer felt like a crisis. Target Canada was at last transitioning
into a functional—almost normal—retailer. There were even big plans for 2015, such
as implementing online shopping at Target.ca.
January 2015
The news on January 15, 2015 was much different: Target Canada was filing for
bankruptcy protection. It had spent $7 billion on the expansion so far, and it didn’t
project turning a profit until at least 2021. Early that morning, Schindele’s direct
reports broke the news to their teams, who then informed their own departments. A
press release went out at 8 a.m. By then, the entire company knew.
All 133 Canadian stores closed by April.
65
What Went Wrong

Data quality
It didn’t take long for Target to figure out the underlying cause of the supply chain
breakdown in 2012: The data contained within the company’s supply chain software,
which governs the movement of inventory, was riddled with flaws. At the very start, an
untold number of mistakes were made, and the company spent months trying to
recover from them.
In order to stock products, the company had to enter information about each item into
SAP. There could be dozens of fields for a single product. For a single product, such as a
blender, there might be fields for the manufacturer, the model, the UPC, the
dimensions, the weight, how many can fit into a case for shipping and so on. Typically,
this information is retrieved from vendors before Target employees put it into SAP.
The system requires correct data to function properly and ensure products move as
anticipated.
A team assigned to investigate the problem discovered an astounding number of

errors. Product dimensions would be in inches, not centimeters or entered in the
wrong order: width by height by length, instead of, say, length by width by height.
Sometimes the wrong currency was used. Item descriptions were vague. Important
information was missing. There were myriad typos. “You name it, it was wrong,” says a
former employee. “It was a disaster.”
It was also something the company should have seen coming. The rush to launch
meant merchandisers were under pressure to enter information for roughly 75,000
different products into SAP according to a rigid implementation schedule. Getting the
details from suppliers largely fell on the young merchandising assistants. In the
industry, information from vendors is notoriously unreliable, but merchandising
assistants were often not experienced enough to challenge vendors on the accuracy of
the product information they provided. (The staff were also working against the
countdown to opening.)
“There was never any talk about accuracy,” says a former employee. “You had these
people we hired, straight out of school, pressured to do this insane amount of data
entry, and nobody told them it had to be right.” Worse, the company hadn’t built a
safety net into SAP at this point; the system couldn’t notify users about data entry
errors. The investigative team estimated information in the system was accurate about
30 percent of the time. In the U.S., it’s between 98 and 99 percent. (Accenture, which
Target hired as a consultant on SAP, said in a statement: “Accenture completed a
66
successful SAP implementation for Target in Canada. The project was reviewed
independently and such review concluded that there is no Accenture connection with
the issues you refer to.”)
Data loading
There was an entirely different process to ensure the correct data actually made it into
SAP. The employees in Mississauga couldn’t add it directly. Instead, the information
was sent to a Target office in India, where staff would load it into SAP. Extra
contractors had to be hired in India, too. “Sometimes even when we had the data
correct, it got mixed up by the contractors in Target India,” says a former employee.
(Another former employee disputes this: “Sometimes the quality of their work wasn’t
so great, but for the most part they did a good job.”) In any event, uploading took
longer than expected, and data week stretched into two. Periodic data blitzes in
individual departments became common into the following year.
Empty shelves
The foot traffic in the early days was higher than expected, which was encouraging, but
it didn’t take long for consumers to start complaining on social media about empty
shelves. “Target in Guelph, please stock up and fill the shelves,” wrote one aggrieved
shopper on Facebook. “How can I or anyone purchase if there is nothing left for me to
buy?” Target told the media that it was overwhelmed by demand and made
assurances that it was improving the accuracy of product deliveries.
The reality was that Target was still struggling with data quality problems that were
hampering the supply chain, and it didn’t have time to address the root causes before
opening another wave of stores. Problems multiplied, and the public mood continued
to turn against Target. Consumers soured on the brand when confronted with empty
shelves—the exact scenario some senior employees warned of earlier in the year.
Stock overflow
Ironically, even as consumers encountered barely stocked stores, Target’s distribution

centers were bursting with products. Target Canada had ordered way more stock than
it could actually sell. The company had purchased a sophisticated forecasting and
replenishment system made by a firm called JDA Software, but it wasn’t particularly
useful at the outset, requiring years of historical data to actually provide meaningful
sales forecasts. When the buying team was preparing for store openings, it instead
relied on wildly optimistic projections developed at U.S. headquarters.
67
According to someone with knowledge of the forecasting process in Minneapolis, the

company treated Canadian locations the same way they did operational stores in the
U.S. and not as newcomers that would have to draw consumers away from rival
retailers. Even if the stores were in out-of-the-way spots—and some of the locations in
the Zellers portfolio certainly were—the company assumed the strength of the Target
brand would lure customers. There was another element at play, too. “Once you
signed up to do 124 Zellers locations, it felt like there was a point where it’s like we
have to assume sales will be good,” says the former employee. “It’s very backwards.”
Distribution problems
The distribution depots were hampered by other factors caused by lingering data
problems and the learning curve associated with the new systems. Manhattan, the
company’s warehouse software, and SAP weren’t communicating properly.
Sometimes, the issues concerned dimensions and quantities. An employee at
headquarters might have ordered 1,000 toothbrushes and mistakenly entered into SAP
that the shipment would arrive in a case pack containing 10 boxes of 100 toothbrushes
each. But the shipment might actually be configured differently—four larger boxes of
250 toothbrushes, for example.
As a result, that shipment wouldn’t exist within the distribution center’s software and
couldn’t be processed. It would get set aside in what was designated as the “problem
area.” These sorts of hang-ups happen at any warehouse, but at Target Canada, they
happened with alarming frequency. Warehouse workers got so desperate to move
shipments they would sometimes slice open a crate that was supposed to contain, say,
a dozen boxes of paper towels but only had 10, stuff in two more boxes, tape it shut
and send it to a store that way.
Malfunctioning point-of-sale system
To add even more headaches, the point-of-sale system was malfunctioning. The
self-checkouts gave incorrect change. The cash terminals took unusually long to boot
up and sometimes froze. Items wouldn’t scan, or the POS returned the incorrect price.
Sometimes a transaction would appear to complete, and the customer would leave the
store—but the payment never actually went through. The POS package was purchased
from an Israeli company called Retalix, which worked closely with Target Canada to
address the issues.
Progress was maddeningly slow. In 2014, a Retalix team flew to Toronto to see
first-hand what Target was dealing with. After touring a store, one of the Retalix
executives remarked, “I don’t understand how you’re using this,” apparently baffled
68
the retailer managed to keep going with so many bugs. But Target didn’t have time to
find a new vendor and deploy another technology. “We were bound to this one bad
decision,” says a former employee. (Retalix was purchased in 2012 by NCR Corp., the
American global payment transaction firm.)
“When entering a new country, it is normal for retail software systems to require
updates to tailor the solution to market needs and processes,” NCR said in a statement
in response to questions about Target’s experience. “NCR was making progress to
customize the solution for the market and Target’s new operations until their decision
to exit the country.”
Gaming the system
A small group of employees also made an alarming discovery that helped explain why
certain items appeared to be in stock at headquarters but were actually missing from
stores. Within the chain’s replenishment system was a feature that notified the
distribution centers to ship more product when a store runs out. Some of the business
analysts responsible for this function, however, were turning it off—on purpose.
Target's business analysts (who were young and fresh out of school, remember) were
judged based on the percentage of their products that were in stock at any given time,
and a low percentage would result in a phone call from a vice-president demanding an
explanation.
But by flipping the auto-replenishment switch off, the system wouldn’t report an item
as out of stock, so the analyst’s numbers would look good on paper. “They figured out
how to game the system,” says a former employee. “They didn’t want to get in trouble
and they didn’t really understand the implications.” Two people involved in the
discovery allow that human error may have been a component, too. Like SAP, the
replenishment software was brand new to Target, and the company didn’t fully
understand how to use it.
When Schindele was told of the problem, he ordered the function to be fully activated,
which revealed for the first time the company’s pitifully low in-stock percentages.
From there, a team built a tool that reported when the system was turned on or off,
and determined whether there was a legitimate reason for it to be turned off, such as
if the item was seasonal. Access to the controls was taken away from the analysts,
depending on the product.
69
How Target Could Have Done Things Differently

The obvious recurring theme through all problems and wrong decisions with Target's
entry into Canada is the immense time pressure. From the very beginning, a clock was
ticking … and that clock was absurd. The company did everything it could to remove
barriers that might slow progress and to ensure decisions could be made quickly.
Timelines were hugely compressed. Building a new distribution center from scratch,
for example, might take a few years. Target was going to do it in less than two
years—and it planned to construct three of them. Introducing a new SAP system within
two years? Hiring and training 15,000 employees? They never had a chance.
But aside from the obvious there are some additional lessons learned in this story.
Technology
One of the most important decisions Target made concerned technology—the systems
that allow the company to order products from vendors, process goods through
warehouses and get them onto store shelves promptly. In the U.S., Target used custom
technology that had been fine-tuned over the years to meet its exacting needs, and
the corporation had developed a deep well of knowledge around how these systems
functioned. Target faced a choice: Was it better to extend that existing technology to
Canada or buy a completely new, off-the-shelf system?
Finding an answer was tricky. By using Target’s existing technology, employees in

Canada could draw on the large amount of expertise in the U.S. That plan had
shortcomings as well. The technology was not set up to deal with a foreign country,
and it would have to be customized to take into account the Canadian dollar and even
French-language characters. Those changes would take time—which Target did not
have. A ready-made solution could be implemented faster, even if the company had
little expertise in actually using it.
The team responsible for the decision went with SAP. Considered the gold standard in
retail, SAP is used by many companies around the world, from Indigo in Canada to
Denmark’s Dansk supermarket chain. It essentially serves as a retailer’s brain, storing
huge amounts of data related to every single product in stores. That data would be fed
by SAP into Target’s other crucial systems: software to forecast demand for products
and replenish stocks, and a separate program for managing the distribution centers.
After implementing SAP in Canada, Target wanted to eventually switch the U.S.
operations over as well, aligning the two countries and ensuring the entire company
benefited from the latest technology.
70
While SAP might be considered best-in-class, it’s an ornery, unforgiving beast. Sobeys
introduced a version of SAP in 1996 and abandoned the effort by 2000. (It wasn’t until
2004 that the grocery chain tried again.) Similarly, Loblaws started moving to SAP in
2007 and projected three to five years to get it done. The implementation took two
years longer than expected because of unreliable data in the system. Target was again
seeking to do the impossible: It was going to set up and run SAP in roughly two years.
The company wasn’t doing it alone, however, and hired Accenture (which also worked
on Loblaws’ integration) as the lead consultant on the project. Target believed the
problems other retailers faced were due to errors in data conversion. Those companies
were essentially taking information from their existing systems and translating it for
SAP, a messy process in which it’s easy to make mistakes. Target, on the other hand,
was starting fresh. There was no data to convert, only new information to input.
Training
Target has a unique, well-established corporate culture in the U.S., which the company
views as one of the reasons for its success, and leaders sought to replicate that
environment in Canada. Target describes itself as “fast, fun and friendly” to work for
and it’s a place where attitude and soft skills are of equal—if not more—importance to
experience. “Target’s motto was they could train you for the job, but they couldn’t
train culture,” says a former employee.
In the U.S., the company prides itself on its development programs for even junior
positions like business analysts, who help coordinate the flow of product, and
merchandising assistants, who work with buyers to choose which products to stock
and negotiate costs with vendors. Target typically recruits candidates for these
positions straight out of school and prepares them for a career in retail. That’s how
Tony Fisher got his start—he joined the company as an analyst in 1999, after he was
drafted by the Texas Rangers baseball organization and played for two years in the
minor leagues. Young employees receive months of instruction and are paired with a
mentor. Hiring for culture over experience works, essentially, because Target in the
U.S. provides ample training.
In Canada, the company succeeded in hiring people with the right personalities, but
young staff received only a few weeks of training, according to former employees who
worked at Target in both countries. The Canadian team lacked the institutional
knowledge and time to properly mentor the new hires. “Everyone was stretched thin.
We didn’t have the manpower to get everything done in the time frame that was laid
71
out,” says a former employee. Another was surprised to see how green his colleagues
were. “I was one of the older people there, and I was in my mid-30s,” he says.
Target Canada would eventually learn what happens when inexperienced employees
working under a tight timeline are expected to launch a retailer using technology that
nobody—not even at the U.S. headquarters—really understood.
Delay market entry
Fisher, 38 years old at the time, was regarded as a wunderkind who had quickly risen
through the ranks at Target’s American command post in Minneapolis, from a lowly
business analyst to leader of a team of 400 people across multiple divisions. Launching
the Target brand in a new country was his biggest task to date. The news he received
from his group that afternoon in February 2013 should have been worrying, but if he
was unnerved, Fisher didn’t let on. He listened patiently as two people in the room
strongly expressed reticence about opening stores on the existing timetable.
Their concern was that with severe supply chain problems and stores facing the
prospect of patchy or empty shelves, Target would blow its first date with Canadian
consumers. Still, neither one outright advocated that the company push back its plans.
“Nobody wanted to be the one to say, ‘This is a disaster,’” says a former employee. But
by highlighting the risks of opening now, the senior employees’ hope was that Fisher
would tell his boss back in Minneapolis, Target CEO Gregg Steinhafel, that they needed
more time.
The magnitude of what was at stake began weighing on some of those senior officials.
“I remember wanting to vomit,” recalls one participant. Nobody disagreed with the
negative assessment—everyone was well aware of Target’s operational problems—but
there was still a strong sense of optimism among the leaders, many of whom were U.S.
expats. The mentality, according to one former employee, was, “If there’s any team in
retail that can turn this thing around, it’s us.”
The group was riding a wave of momentum, in fact. They had overcome seemingly
endless hurdles and worked grueling hours to get to this point, and they knew there
were costs to delaying. The former employee says the meeting ultimately concerned
much more than when to open the first few stores; it was about the entirety of
Target’s Canadian launch. Postponement would mean pushing back even more store
openings. Everyone else in attendance expressed confidence in sticking to the
schedule, and by the time the meeting concluded, it was clear the doors would open as
promised. “That was the biggest mistake we could have made,” says the former
employee.
72
Closing Thoughts
To say that Target had unrealistic and highly optimistic growth plans for its entry into
Canada would be an understatement—in 2011 the company had plans to open 124
locations by the end of 2013 with the expectation that the Canadian company would
be profitable within its first year of operations. Not to mention that they also intended
to build three new distribution centers in less than two years—a feat that typically
takes a couple of years per center. The end result was that Target Canada filed for
bankruptcy, wasted billions of dollars, tarnished its reputation and left approximately
17,600 people without jobs. In most cases, this would have completely destroyed a
company’s chances of surviving and Target has been lucky to continue to see success
among its U.S. stores.
In a nutshell: Unmanageable deadlines and disastrous IT wrecked this top U.S.

retailer's attempt at international expansion. Just another proof that nowadays
technology can make or break an enterprise.
References
> Dahlhoff, D. (2015). Why Target’s Canadian Expansion Failed. [online] Harvard
Business Review. Available at: https://goo.gl/9JykFG
> McMahon, T. (2015). Missing the Mark: Some Five Reasons and Explanations Why
Target Failed in Canada. [online] The Globe and Mail. Available at: https://goo.gl/f6aSjf
> Northrup, L. (2016). Fifteen Things We Learned From Target’s Downfall in Canada.
[online] Consumerist. Available at: https://goo.gl/Z1asvc
> Randall, K. (2016). Target misread its Canadian shoppers and retail market: business
expert. [online] CBC News. Available at: https://goo.gl/QZRPGb
> Wahba, P. (2015). Why Target failed in Canada. [online] Fortune. Available at:
https://goo.gl/ryaetb
> Joe Castaldo (2015). The last days of Target. [online] Canadian Business. Available at:
https://www.canadianbusiness.com/the-last-days-of-target-canada/
73
Case Study 8: How Hertz Paid Accenture $32M for

a Website That Never Went Live
"Accenture never delivered a functional website or mobile app," - Hertz
Car rental giant Hertz is suing consultant mammoth Accenture over a website redesign
that ended in something that never saw daylight.
The U.S. corporation Hertz operates the car rental brands Hertz, Dollar, and Thrifty,
and has approximately 10,200 corporate and franchise locations throughout the world.
With the rapid growth of rideshare apps like Uber and Lift, increased competition, and
falling used car prices, Hertz has struggled with profitability over the last five years,
and the stock price has fallen 85 percent since then. The company has replaced its CEO
twice over the same period, most recently at the start of 2017.
Hertz hired Accenture in August 2016 to completely revamp its online presence. The
new site was due to go live in December 2017, but this had to be delayed to January
2018. A second delay put the new go-live date to April 2018, which was then also
missed.
As Hertz endured the delays, it realized that there was a nasty situation at hand: the
product and design apparently didn't do half of what was specified, and even that was
still not finished. "By that point, Hertz no longer had any confidence that Accenture
was capable of completing the project, and Hertz terminated Accenture," the car
rental company complained in a lawsuit against Accenture in New York in April this
year.
Hertz is suing for the $32 million it paid Accenture in fees, and it wants more millions
to cover the cost of fixing the mess. "Accenture never delivered a functional website or
mobile app," Hertz claims.
Timeline of Events
2015
Hertz hires Tyler Best as the new chief information officer (CIO) and creates a plan to
transform the Hertz IT environment, which has become very complex over the years.
74
According to a presentation at a MuleSoft conference in 2018 they had at the time

around 1,800 IT systems, 6 database vendors, and 30 rental processing systems. In
order to add a new product, Hertz needs to make 18 system changes.
The company launches a major end-to-end technology upgrade program that includes
outsourcing operations of its legacy systems to IBM, and designing and building a
cloud-based infrastructure for Hertz’s five core platforms: digital, CRM fleet
management and fleet accounting, reservations, and rentals. The overall cost of the
program is expected to be in excess of $400M.
First half of 2016
Hertz wants to redefine the customer experience of its market-leading brand of the
same name by creating a new, modern website and mobile applications.
The envisioned solution is intended to be readily extendable to other Hertz brands, like
Dollar and Thrifty.
Hertz engages Accenture to assist in validating its strategy and planning for the project.
The engagement is governed by a consulting services agreement between Hertz and
Accenture that has been in place since 2004.
The program appears to have adopted an “agile” methodology with the use of product
owners and the application of sprints to deliver iterations of the solution.
At the same time Hertz is implementing a large MuleSoft middleware and Oracle
enterprise resource planning (ERP) project to upgrade the firm’s transaction processing
capabilities.
Second half of 2016
Hertz requests proposals from several of the leading technology services providers to
implement the new solution.
They eventually select Accenture, relying on Accenture’s claimed world-class expertise

in website and mobile application development.
Hertz completes outsourcing many of its U.S. IT jobs to IBM in efforts to cut back on
office costs.
75
Accenture and Hertz engage in phase 1 of the project, producing a “solution blueprint”
that describes the functionality, business processes, technology, and security aspects
of the envisioned solution. Fees paid to Accenture for this phase total $7M.
First half of 2017
Hertz announces a new CEO and board member, Kathryn Marinello.
Accenture and Hertz enter into discussions about phase 2 of the project to design,
build, test, validate, and deploy the website, mobile applications, and other
deliverables.
During an investor presentation in May, Hertz commits to delivering the “modernized

e-commerce platform” by the end of 2017.
Accenture removes the product manager and a project architect from the project.
Second half of 2017
Accenture and Hertz sign a formal agreement for phase 2 with fees totaling $26M for
this portion of the project. The contract states that Accenture will provide “project
management” services, including Accenture’s obligation to “plan, control, and lead the
execution of Accenture’s scope of services.” The agreement includes language
regarding “a focused objective of launching the [website and mobile applications]
platforms and experience in December 2017.”
Soon after signing the agreement Accenture reports that it will not be able to meet the
December 2017 go-live date and requests a one-month extension to January 2018. Not
long afterward, the go-live date is further postponed until April 2018.
At the end of the year Hertz and Accenture sign a change request. According to
Accenture, this change order altered the party’s responsibilities under the phase 2
scope of work (SOW) and includes language in which both parties release any claims
“arising out of or related to the need to provide Services beyond” the project's
estimated December 2017 launch date, and agree that they will not bring any suit
related to delays in the project.
2018
76
Hertz pays Accenture for the work contracted under the SOW and the first change
request. A second change request is signed for Accenture to provide additional
services for Hertz for an agreed-upon amount of fees.
In April Hertz’s CIO Tyler Best steps down and receives a severance package that is
consistent with firing without cause. The CEO, Kathryn Marinello, takes over on an
interim basis.
In May Accenture is removed from the project.
In June Hertz hires a new vendor to complete the project. Hertz claims to have spent
an additional $10M in fees to correct or replace the work produced by Accenture.
On July 31 Hertz announces Opal Perry as the new CIO. She joins the company from
Allstate Insurance.
2019
On April 19, Hertz files a lawsuit against Accenture.
A spokesperson for Accenture tells The Register: "We believe the allegations in this
lawsuit are without merit, and we intend to defend our position. Because this is an
ongoing legal matter, we decline any further comment."
In May Accenture submits a statement that the damages should be limited to only
Accenture’s direct damages capped by Accenture fees and further that Hertz’s claims
are barred by the mutual release that both parties entered into as part of the first
change request that was signed for the phase 2 SOW.
Hertz responds that the releases agreed on expressly exclude breach of warranty.
According to the letter, the work, which consisted of redesigning the car rental
provider’s “website, apps, digital marketing and related services,” unfolded in two
phases, the first of which “proceeded relatively smoothly.” The letter acknowledges
that phase 2 included “some delays and setbacks” that required the parties to adjust
the statement of work twice as planned launch dates came and went.
The letter claims that Hertz and Accenture agreed on these changes to the contract
and that the client paid for the first extension period. When Accenture asked to be
paid for the second period, however, Hertz claimed it was not obligated to pay “due to
perceived deficiencies in Accenture’s work on earlier phases of the project.”
77
It followed by suing the firm to prevent further requests for payment, stating in its own
filing that Accenture “never delivered a functional website or mobile app.”
The letter continues, “Accenture intends to assert counterclaims, including for

payment of these past-due invoices.”
On June 7, the judge issues scheduling orders that sets the trial date for March of
2020.
On June 20, 2019, Hertz files an amended lawsuit against Accenture and lays out their
case on their claim of deceptive and unfair practices. In doing so, Hertz takes aim at
Accenture talent and trashes the team that had worked on the Hertz program.
What Went Wrong

Before we can understand what went wrong we have to understand what was planned
to be delivered. Hertz describes the five-layer digital platform architecture below in its
suit. They suggest that Accenture recommended this architecture.
Front End Layer – The presentation layer in which the screens and the interface were
presented to the users using the JavaScript framework Angular.
Content Management – Adobe Experience Manager (AEM) was the tool of choice to
update and revise the content that appears on the website.
Microservices Layer – Composed of code that performed various “services,” such as

searching for a Hertz location or for a type of vehicle, writing a review, sending an
email to a Hertz representative, etc.
Integration Layer – MuleSoft was the tool of choice to allow the front-end systems to
request and receive data from the back-end systems.
Back-End Systems – Hertz’s reservation systems, rewards systems, etc.
Now that we know the plan, we can have a look at the results.
No responsive design
One of the most staggering allegations in Hertz's suit is that Accenture didn't
incorporate a responsive design, by which web pages automatically resize to
78
accommodate the visitor's screen size, whether they are using a phone, tablet,
desktop, or laptop.
This has been a standard website practice for years, and is even included in the
contract that was signed. But somehow the team from Accenture decided that only
desktop and mobile versions were needed, according to Hertz.
When Hertz execs asked where the tablet version was, Accenture "demanded
hundreds of thousands of dollars in additional fees to deliver the promised
medium-sized layout."
No common core
The specification documents defined a common core of libraries to be "a fundamental

principle of the design" so that the company could share information and structures
across all its companies' websites and apps. And Accenture, well, completely ignored
that, according to Hertz.
"Accenture deliberately disregarded the extensibility requirement and wrote the code
so that it was specific to the Hertz brand in North America and could not be used for
the Hertz global brand or for the Dollar and Thrifty brands," the lawsuit alleged.
Vulnerable code
Hertz states the code that was written by Accenture was terrible and a security
nightmare waiting to happen.
"Accenture’s developers wrote the code for the customer-facing ecommerce website
in a way that created serious security vulnerabilities and performance problems," it
says before noting that "the defects in the front end development code were so
pervasive that all of Accenture’s work on that component had to be scrapped."
No experience with used technologies
In its revised suit, Hertz states that Accenture represented that they had “the best
talent in the world,” “the skills needed to win,” and that they would “put the right
team on the ground day one.”
Hertz claims that they were far from experts in these technologies and that Accenture
was deceptive in their marketing claims.
79
Where the previous point regarding vulnerable code shows that the front-end
developers were not familiar with and did not really understand Angular, the
Accenture developers were also inexperienced with the Adobe Experience Manager
(AEM) code.
The lawsuit complains that Accenture decided to use Adobe's AEM analytics but didn't
follow its archetype in either the coding or the file structure "which made the
application unreliable and difficult to maintain, as well as making future updates
challenging and inefficient."
On top of this Accenture apparently told Hertz that to speed up the production of the
website's content management system, it wanted to use something called "RAPID" —
and told Hertz it would have to buy licenses for it to do so. Hertz bought the licenses;
however, it turned out that Accenture didn't actually know how to use the technology
and the quick-fix took longer than it would have without it.
The lawsuit notes: "As Accenture's project leaders acknowledged, Accenture 'spent a
good deal of time fighting through integration of RAPID' into Hertz’s environment."
No testing and documentation
Accenture also failed to test the software, Hertz claims, and when it did do tests "they
were seriously inadequate, to the point of being misleading." It didn't do real-world
testing, we're told, and it didn’t do error handling.
Accenture’s developers also misrepresented the extent of their testing of the code by
commenting out portions of the code, so the code appeared to be working.
On top of that, despite having specifically requested that the consultants provide a
style guide in an interactive and updateable format — rather than a PDF — Accenture
kept providing the guide in PDF format only, Hertz complained.
When Hertz confronted the consultants about the PDF problem, guess what the
response was? Yep, it wanted "hundreds of thousands of dollars in additional fees" to
cover the cost.
A bad ending...
After missing the April deadline the team working on the project was pulled off by
Accenture "but their replacements did not have the same level of experience, and a
good deal of knowledge was lost in the transition," Hertz states.
80
Despite having missed the deadline by five months, with no completed solution and
slowed down by buggy code and an inexperienced team, Accenture tells Hertz it will
cost an additional $10M — on top of the $32M it had already been paid — to finish the
project.
How Hertz Could Have Done Things Differently

On July 15th, Accenture delivered their response to the initial Hertz lawsuit. These
responses are straight to the point, and they show a number of things that Hertz could
have done better.
The following represents a summary of Hertz’s claims (in bold) and Accenture’s
responses.
Accenture claimed they had “the best talent” and “the skills needed to win.”
> Hertz’s claims are made on marketing language. “Simply put, no reasonable person,
much less a Fortune 500 company planning a multi-million dollar redesign of its digital
platforms, could interpret a statement in a marketing presentation that a company had
‘the best talent’ and the ‘the skills you need to win’ as a factual assertion that
everyone who ever worked on the project would perform their work flawlessly.”
> Accenture limited its warranty to only what was defined within the contract and
expressly excluded all other conditions and representations.
Accenture said they had the best talent in the world and would provide their best
team from the start.
> Hertz provided no evidence that Accenture claimed to bring the best Angular JS
front-end and AEM back-end coders.
> Accenture had the right under the contract to determine all staffing, including to
replace staff at will, with no agreement as to the minimum levels of experience of any
particular staff member.
Accenture had the responsibility to deliver the product by December 2017.
> Accenture only had responsibilities to manage the portions of the project for the
Accenture scope of services.
81
> Hertz was responsible for implementing mid-tier and backend integrations, security,
and user acceptance testing.
> Hertz was responsible for managing all third parties.
> Hertz was responsible to provide all website content, without which the website
could not launch.
Accenture exhibited “extortionist-like” claims, asking for more money to complete

work.
> This was not a fixed-price contract and instead, Accenture was paid on a
time-and-materials basis. Therefore, there are no circumstances under which an
Accenture request for payment would be considered “extortionate.”
Accenture did not properly test the code or comment out sections of the code.
> Hertz did not provide specific examples of this situation as required by governing
law, so Accenture would have the ability to specially respond or defend itself. These
claims should be dismissed.
We are entitled to consequential damages as a result of the delay in the program and
additional costs to a third party.
> The consulting services agreement signed in 2004 between the companies barred
Accenture from being liable “for any consequential, incidental, indirect, special or
punitive damage, loss or expense (including but not limited to business interruption,
lost business, lost profits, or lost savings)."
Closing Thoughts
This is an incredibly important case for any company that will be engaging systems
integrators like Accenture, IBM, Deloitte, EY, and others. This case is an early
high-profile case in digital transformation using agile methodologies.
The information available in these suit filings provides insights into how these
integrators position themselves in contracts and marketing materials, and how they
behave when actively engaged with the buyer.
82
As with any lawsuit, there are at least two sides to every story. There are many
questions that will need to be answered to understand what really happened, but no
matter what the answers are, Hertz did not act as a responsible buyer.
A responsible buyer of third-party systems and systems development will have

excellent knowledge, understanding and experience in defining, planning, directing,
tracking, controlling and reporting systems development projects. They will know what
should be done, when, why and how.
You can delegate authority for doing the project management to the supplier but you
cannot delegate responsibility.
In a nutshell: Responsibility for the project — including responsibility for it failing —

always rests ultimately with the buyer.
References
> THE HERTZ CORPORATION vs ACCENTURE LLP Civil Action No. 19-3508
> Letter Accenture - Hertz May 17
> Letter Hertz - Accenture May 23
> Presentation Hertz - Mulesoft 2018
83
Case Study 9: The Payroll System That Cost

Queensland Health AU$1.25 Billion
"Arguably the worst failure of public administration in Australia’s history." -

Australian Premier Campbell Newman
The payroll system implementation disaster at Queensland Health in 2010 is said to be

the most spectacular technology project failure in the Southern Hemisphere and
arguably the worst failure of public administration in Australia’s history.
Queensland Health is the public sector healthcare provider for the Australian state of
Queensland, located in the country’s northeast. It provides dental, medical, and
aged-care facilities in Queensland, which has the most geographically dispersed
population of all Australian states.
Queensland Health needs to ensure that adequate healthcare services can be provided
in the most remote parts of the state, which has a population of 5.07 million across an
area of 1.85 million square kilometers. Every day, the organization provides hospital
services to approximately 40,000 people and is responsible for approximately 85,000
employees across 300 sites.
What began as an AU$6.19 million contract between the State of Queensland and IBM
Australia to replace Queensland Health’s aging payroll system eventually led to over
35,000 payroll mistakes and will ultimately cost taxpayers a whopping AU$1.25 billion,
which translates to approximately US$850 million.
When the system went live, a large number of Queensland Health employees,
including doctors and nurses, were either incorrectly paid or not paid at all. It led to
the resignation of the minister of health, industrial strike action and loss of staff
members to other employers.
Timeline of Events
2002
At this point in time Queensland Health used two systems for their payroll: Lattice and
the Environment for Scheduling Personnel (ESP) rostering engine. Lattice and ESP were
rolled out progressively over six years from 1996 to 2002.
84
Payroll departments were part of their respective districts. Processing of pay was
undertaken locally and there were close working relationships between line managers
and local payroll staff.
Whilst processing of pay occurred locally, the actual running of the pay was
undertaken centrally; essentially, a 'hub and spoke' model was used.
Lattice required a substantial amount of manual interventions to accommodate the

complex award and incentive structures within Queensland Health.
2003
In 2003 the Queensland State government formally established a government shared

services initiative (SSI) mandating that all state government departments, including
Queensland Health, replace their existing legacy payroll systems with a standardized
software solution that incorporates SAP HR and SAP Finance. The overarching
objective of the SSI was to consolidate technology and resources through delivering a
high-quality solution with standardized business processes.
The SSI was expected to deliver the following benefits: (1) increased opportunities
through enabling workforce mobility; (2) increased visibility into the cost of services;
(3) reduced data duplication through the consolidation of systems; (4) reduction in
costs associated with licensing agreements; (5) reduction of personnel; (6) achieve
economies of scale; (7) enable the government organizations to focus on their core
competencies, thus increasing the standard of service; and (8) consistency of HR and
finance information across all government agencies.
2005
In 2005, just three years after the progressive rollout was completed, Queensland
Health received notification from the Lattice system vendor, Talent2, that their existing
Lattice system was becoming obsolete and was no longer going to be supported, with
services and updates ceasing on June 30, 2008. As a result, Queensland Health needed
to consider replacing the obsolete Lattice system much sooner than it would have
liked.
SSI decided that the system for payroll would be SAP ECCS and Workbrain. Accordingly,
it was decided that Queensland Health would replace the Lattice I ESP system with SAP
ECC5 I Workbrain as part of the shared services initiative.
85
2007
As part of the SSI, the Queensland government established CorpTech within the
Queensland Treasury to oversee the standardized implementation across all state
government departments.
CorpTech was responsible for overseeing the consultant selection process (Request for
Information, Request for Proposals, and Invitation to Offer) and managing the
consultant organizations. CorpTech sent out a Request for Information (RFI) on July 2,
2007, and four consultant firms responded: Accenture, IBM, Logica, and SAP.
Afterwards, CorpTech requested detailed proposals from the aforementioned
consultant firms on July 25, 2007.
Prior to the RFI being issued, CorpTech had managed the implementation of SAP HR at
the Department of Housing, and SAP Finance at the Department of Justice. These
implementations proved to be quite costly as a substantial number of consultant firms
and private consultants had been utilized. Due to the large expense associated with
the multiple consultant firms, the consultant methodology for the SSI was changed to
the prime contractor model on August 16, 2007.
Subsequently, on September 12, 2007, CorpTech released an Invitation to Offer, and

IBM, Accenture, and Logica responded. Ultimately, on December 5, 2007, IBM officially
signed an AU$98 million contract to be the prime contractor of the SSI.
2008
Around October 2008, IBM had not achieved any of the "contracted performance
criteria." It had, however, had been paid AU$32 million of its AU$98 million contract.
The idea to build a payroll system across the entire Queensland public service was now
officially scrapped, but the decision was made to push ahead with a payroll system for
Queensland Health.
2009
To cater for Queensland Health's specific business needs, including the complex award
structure, retrospectivity, and concurrent employment, a significant number of
customizations were made to both Workbrain and SAP.
Payroll and user acceptance testing was performed in parallel over a series of stages
starting July 2009. The first test of the payroll compared the pays of only 10 percent of
86
employees from all employee groups when performed in the SAP HR and Workbrain
rostering solution as opposed to the legacy Lattice system, which resulted in an AU$1.2
million discrepancy in the biweekly payroll.
2010
A second payroll test occurred in February 2010, which only resulted in an AU$30,000
discrepancy; however, casuals and overtime claims were not tested. Queensland
Health accepted the inherent risks and opted to go-live without full testing of all the
functionalities of the system.
IBM continued to operate under varying scope and the state government kept signing
off on the change requests. The project documentation reveals that prior to the payroll
system going live, the project underwent four revised go-live dates and four separate
stages of change requests, often done at the last minute.
By the time the system finally went live in March of 2010, 20 months after the original
start date, the bill had already ballooned to AU$101 million.
Given the significant issues identified following the initial go-live, it was decided to
establish a payroll stabilization project specifically focused on stabilizing the new
payroll system. The four key focus areas for this project were: standardization and
improvement of district and division business processes; payroll processing; payroll
system performance; and support and communications for staff, line managers, and
other key stakeholders.
2012
The cost of going live with a premature system resulted in more than 35,000 payroll
mistakes, and by this point it had cost the state in excess of AU$400 million just to
operate the system. KPMG estimated that the cost of making the system function for
the next five years would be another AU$836 million.
2013
IBM should never have been appointed as the prime contractor in Queensland’s failed
health payroll project, according to the finding of the Commission of Inquiry that
investigated the bungled project. The inquiry, which ran for nearly three months at a
cost of AU$5 million, heard from some 50 witnesses, including former premier Anna
Bligh, former health minister Paul Lucas, and senior IBM executive Bill Doak.
87
The report, by Commissioner Richard Chesterman, was handed down on July 31 in the
Queensland Parliament by Premier Campbell Newman. The premier described the
project as "arguably the worst failure of public administration in Australia’s history."
The report was particularly damning of the procurement process that led to the
appointment of IBM, and decisions made by senior public servants and contractors
involved with the project.
The report also found the state government could not recover any funds from IBM for
the failure of the project, and that the decision to reach a negotiated settlement with
IBM rather than commencing legal proceedings was made without any proper analysis.
2016
The Queensland government was ordered to pay significant costs to IBM Australia
after failing in a bid to recoup losses from the health payroll debacle.
The Newman government launched legal action against IBM in 2013, arguing the
company had misrepresented its capability to deliver a new payroll system on time and
on budget.
But IBM challenged the lawsuit and pointed to a 2010 agreement that the company
said released it from the damages claim.
A trial was held in the Brisbane Supreme Court in 2016 with Justice Glenn Martin ruling
in favor of IBM.
What Went wrong
System availability
During the payroll cut-over period to the new system, there were significant issues
with the availability of the system to payroll staff which reduced the processing time
available. This created an initial backlog of payroll forms and unprocessed adjustments
for the period just prior to the go-live date that grew over subsequent pay periods.
It took approximately eight months to process the backlog of pay adjustments and
forms to return to previous business as usual (BAU) levels.
Performance issues
88
The degree of retrospectivity accommodated by the Queensland Health payroll

system, whereby staff could submit forms for work completed up to six years ago, was
creating significant payroll system performance issues.
System issues
As of 2 May 2012, there were 570 logged system issues, 76 of which were identified as
having the potential to impact on staff pay. System defect fixes and enhancements
were required to occur during designated 'major release' schedules, of which there
were three scheduled per annum. There were some delays in addressing specific
defects and issues due to the prioritization of other 'fixes' including the pay date
change, changes associated with enterprise bargaining changes, legislative compliance
changes, etc. There was a need to gain endorsement for an agreed longer-term
approach to implementing key system changes so that the release windows could be
utilized more effectively.
Overpayments and entitlements
As of May 2012, Queensland Health had overpaid staff AU$112.3 million, of which
AU$16.5 million had been repaid and AU$3.3 million was waived, leaving AU$9 million
outstanding. Queensland Health had an obligation under the Financial Accountability
Act 2009 to recover these amounts; however, there was a moratorium in place
preventing Queensland Health implementing Queensland Health-instigated
overpayment recovery.
Queensland Health was required to fund fringe benefit tax (FBT) liabilities associated
with overpayments, and this represented a significant additional cost burden to
Queensland Health. While the previously agreed overpayment moratorium was in
place, the amount increased by approximately AU$1.7 million every two weeks.
In addition to overpayments, the issue of employee leave and balances requires

further investigation and analysis. PwC has conducted a number of reviews into leave
balances and they have identified that up to 20,000 leave transactions are still
outstanding since the move from the previous Lattice Payroll system across to SAP.
New business processes
Part of the implementation of the new system was further standardization and
centralization of payroll processing, including the introduction of central processing
teams and a centralized pay run. As such, the key linkage between the districts and
89
their local payroll providers was severed — payroll staff were required to process
unfamiliar rosters for staff members across the state.
In addition, fundamental differences in how districts and line managers were providing
pay information and rosters were identified, with each district continuing to provide
the information in the format they had developed locally (this was a continuation of
what had occurred with the Lattice system; however, now the payroll officers
responsible for interpreting the pay information from the districts did not have the
local knowledge or relationships that had previously assisted with the interpretation
process).
How Queensland Health Could Have Done Things

Differently
The key findings from the auditor general's report were as follows:
Project governance
Project governance prior to go-live, including managing relationships with key

Stakeholders, was not effective in ensuring roles and responsibilities were clearly
articulated and in ensuring there was clear accountability for efficient and effective
implementation of the system.
The governance structure for the system implementation, as it related to CorpTech,

the prime contractor, and Queensland Health, was not clear, causing confusion over
the roles and responsibilities of the various parties.
Management of the project became even muddier after it commenced. Numerous

agencies and boards divided oversight and authority, causing significant confusion
which, in the end, rendered them all "ineffective in establishing a shared
understanding of stakeholder expectations in relation to the quality of project
deliverables":
> The Solution Design Authority (SDA) (which, during this period, transformed into the
program delivery office (PDO) of the state's CorpTech IT division);
> The Queensland Health Enterprise Solution Transition (Queensland HealthEST), the
state's information technology management program and acting project manager
(which inexplicably retitled the "Interim Solution" as the "Queensland Health
Implementation of Continuity" (Queensland HealthIC) — no confusion there!);
90
> The executive steering committee (ESC), which included personnel from CorpTech as
well as the shared services agency (SSA) and the Department of Education, Training
and the Arts (DETA), and
> The "Release Steering Committee," which answered to both the ESC and CorpTech
and counseled its chair regarding the development of the Interim Solution.
While there appeared to be lots of oversight of the program, Australia's

auditor-general reported that "it was not clear which Accountable Officer had
responsibility for the overall governance and successful completion of the whole
project."
Scope and requirements
There was inadequate documentation of business requirements at the

commencement of the project. The absence of a periodic review of the business needs
contributed to subsequent difficulties with system testing and the implementation of a
system which did not meet the needs of Queensland Health's operating environment.
The complexity of the project was immense and involved the management of over
24,000 differing combinations of wage payments and withholdings for over 80,000
workers and subcontractors spread over 13 contractors and multiple industrial
agreements. Because of the fear that the existing system was in imminent danger of
immediate failure, IBM agreed to take just seven months to develop and implement an
"Interim Solution" to tide the agency over until a full replacement became operational.
Within that seven months, only two weeks were set aside at the beginning of the
project to scope out the "critical business requirements" needed by the agency and the
digital solutions that would respond to those demands. Not surprisingly, the lack of
identifiable objectives was a significant cause of the project's abject bungling.
Testing
System and process testing prior to go-live had not identified a number of significant
implementation risks, and therefore, the extent of the potential impact on the
effective operation of the payroll system had not been fully understood and
quantified.
Neither system usability testing nor the validation of new processes in the business
environment were performed. As a result, Queensland Health had not determined
91
whether systems, processes, and infrastructure were in place for the effective
operation of the new system.
Business processes
A number of critical business readiness activities and practices were not fully
developed prior to the implementation of the new system. Several changes to payroll
administration practices including the reallocation of processing duties within payroll
were introduced at the same time as the release of the SAP and Workbrain systems.
Standardization
The implemented Workbrain (1,029 customizations) and SAP (1,507 customizations)

systems were heavily customized and are not operating optimally in the Queensland
Health environment. Customizations are costly to manage, increase risk and impact on
system performance and should be minimized where practical.
Supplier
Despite its affiliation with a global digital leader, this was IBM Australia's first attempt
at delivering a project of this size. That fact was not helpful considering that
Queensland Health was probably the most complex of the Australian agencies needing
the overhaul and was perhaps not the best candidate for IBM's first go.
Layer on top of those contributors the reality that the "Solution Design Authority," the
state agency with the responsibility to define and maintain the scope, architecture,
and design of the new system, was "passive, perhaps lazy" about communicating its
requirements for a payroll system. Before project development began, the Solution
Design Authority accepted IBM's "incomplete, ... unsatisfactory scope [of work]
documents" as-is and with no questions. The project was off to a horrible start.
Closing Thoughts
As a consequence of Queensland Health’s disastrous payroll implementation project,
the Queensland Government changed their Information Technology (IT) strategy and
governance procedures.
Furthermore, the Queensland Government IT audit specified a series of constraints

that must be placed on high-risk projects, which include the following:
1) project management personnel must be of the highest quality;
92
2) rigorous application of the Queensland Government project and program

methodology;
3) the project must be approved by numerous chains of command, and
4) a reporting regime is to be established to increase the visibility of the costs

associated with IT projects.
Being a state government department in the healthcare industry, Queensland Health is

by definition already riddled with substantial layers of bureaucracy, adding to the
complexity and increasing the difficulties associated with decision making, visibility,
and accountability.
And these reforms, whilst necessary, add additional layers to the governance process,
increasing both the number of bureaucratic decisions and the degree of red tape.
Consequently, both client and consultant organizations have been more cautious
throughout recent technology projects in the public sector due to the increase in
compliance which leads to project delays, and increased project costs.
Thus, aside from the financial and societal implications associated with the Queensland
Health payroll implementation failure, the failure has also had national, industry-wide
ramifications.
But do we really learn from these massive failures? Looking at the long list of big
technology projects gone wrong after this one, I doubt it.
Especially in the area of public services.
References
> Queensland Health Payroll System Commission of Inquiry Report (2013)
> KPMG Review of the Queensland Health Payroll System (2012)
> Rebekah Eden and Darshana Sedera, The Largest Admitted IT Project Failure in the
Southern Hemisphere: A Teaching Case (2014)
> Raul Manongdo, Queensland Health Payroll system – A case study on business
process management and application enterprise integration (2014)
93
94
Case Study 10: Leaseplan Paid $100 Million for a

SAP System That Never Went Live
"The system is not fit for purpose in the emerging digital world. The monolithic
nature of the SAP system hinders its ability to make incremental product and service
improvements at a time of accelerated technological change. As a consequence, the
system is being restructured." - LeasePlan CEO Tex Gunning
International automotive fleet management company LeasePlan has been hit by a

massive bill for a failed SAP implementation. LeasePlan has since dropped SAP to
pursue an alternative IT infrastructure, citing the “monolithic nature” of the ERP
system as being incompatible with its more agile needs – but not before sinking almost
€100 million into its SAP efforts.
Established in the mid-1960s, LeasePlan is one of the world’s leading fleet

management and driver mobility companies, with 1.8 million vehicles under
management in more than 30 countries, and approximately 6,600 employees. Their
core business involves managing the entire vehicle life-cycle for their clients, taking
care of everything from purchasing and maintenance to car remarketing.
Timeline of Events
2015
At this point of time LeasePlan has offices across 32 countries and is owned by a
consortium consisting of the Volkswagen Group (50%), Olayan Group (25%), and
Mubadala Development Company (25%).
Each location operates under the same brand, but each country uses its own model to
do so.
Alfonzo Venturi, CIO for LeasePlan Australia, explained that every country was basically
allowed to do its own thing; run its own IT, and manage its own activities.
"Yes, we all reported up, but we had autonomy to really do what we wanted to. If you
can imagine 32 entities with 35 leasing solutions -- so very expensive, with no agility at
all," he explained. "Any change would take a long time and it just wouldn't allow us to
do any of the new technology sharing that our customers are wanting today."
95
2016
In 2016, LeasePlan, headquartered in the Netherlands, picked up a new 100 percent

shareholder in LP Group BV, which represents a consortium that includes Dutch
pension fund service provider PGGM, Denmark-based pension fund ATP, GIC, Luxinva
SA, and investment funds managed by TDR Capital LLP.
"In all the due diligence prior to the acquisition, they saw that there was a major
opportunity in IT," according to Venturi. "I think they started realising that something
had to be done and they couldn't continue the way we were going."
This kicked off a large programme aimed at designing and building a Core Leasing
System (CLS) based on SAP’s enterprise resource planning technology which was used
at LeasePlan Australia since 2010. It was here the trouble began.
Venturi was asked to visit a number of European countries to perform a feasibility

study and assess the possibility of putting the Australian-based solution in place.
He conducted a proof of concept for three LeasePlan entities, which Venturi said ticked
all the boxes. Following the closing of the acquisition, the new owner wanted to see
this stretch to the rest of the world.
Venturi then performed a full assessment of each country and it was decided he would
lead the way with replacing the legacy systems in 15 countries in Europe -- as a first
step -- taking the baseline of what he did in Australia and filling in the country-specific
gaps.
India-based system integrator and SAP partner HCL Technologies has been brought in
as a strategic partner.
SAP will provide a host of solutions to LeasePlan “to support our main operations
end-to-end, achieve efficiency and straight-through processing”.
These are SAP Leasing at the back-end, Fiori for user experience (UX), Ariba for
procurement, Hybris for e-commerce and product content management, S/4HANA
(including for the new insurance business) and Business Objects for reporting and
analytics/business intelligence (BI).
96
The deployment will be done on the Amazon Web Services (AWS) cloud. The Australian
operations that already have SAP’s tech deployed on-premise on IBM’s servers will
continue to operate as is for the time-being.
The first country to go live is planned to be the head office in the Netherlands, slated
for October 2017.
This will be followed by the roll-out of a single instance of the system across 15
countries, mainly in Europe plus Australia (new SAP components, such as Hybris) and
New Zealand.
This phase is set to be completed by October 2020. There will be two main deployment
stages per year – in April and October. These will be “big bang” go-lives in the majority
of countries, with a couple of exceptions where the operations are deemed too big. In
these cases, go-lives will be phased by customer segment.
“We will go live with the minimal viable product, and expand functionality at a later
stage,” Venturi says.
The remaining countries will follow at later stages as well.
All legacy systems will be switched off.
2018
As the company’s new owners sought to drum up interest in shares ahead of a stock
market launch, LeasePlan announced the plans to use SAP to streamline and smooth
internal operations across all of LeasePlan’s country organisations, making the group
more efficient and lowering total costs of ownership.
When LeasePlan subsequently launched its initial public offering in October 2018, the
company repeatedly stated that it had high expectations of the benefits its new SAP
system would bring.
This was supposed to be a chief reason for investors to factor in a considerable

IT-driven efficiency in their valuation of the lease company. Not all investors however
fell for the charm offensive, however.
LeasePlan found investors were unwilling to pay what its shareholders had set as a
target, and in the same month it was announced, LeasePlan abruptly called off its
initial public offering, blaming the deterioration in “market conditions”.
97
2019
In the 12 months since the called off IPO, the worst possible scenario has unfolded for
LeasePlan. In September 2019, the company finally pulled the plug on the major SAP
project. A total investment of €98 million has been deemed unrecoverable and
subsequently impaired from the firm’s books.
CEO Tex Gunning in a recent meeting with shareholders said, “The system is not fit for
purpose in the emerging digital world. The monolithic nature of the SAP system
hinders its ability to make incremental product and service improvements at a time of
accelerated technological change. As a consequence, the system is being
restructured.”
Instead, LeasePlan will now press forward with work on an alternative IT solution,
which it has called a “Next Generation Digital Architecture”. Rather than placing its
faith in one supplier from now on, the car leasing company is opting for a combination
of best-of-breed third-party solutions combined with a deeper in-house involvement.
The company will now leverage leading off-the-shelf solutions for various modules
(e.g. contract management, insurance claims, predictive maintenance) and combine
them with existing LeasePlan best practices to form its new infrastructure.
This approach is, according to Gunning, better suited to the “digital revolution” taking
place in the global lease industry. It will allow for a more scalable and flexible IT
infrastructure, smoother product deployments and updates, and will enhance
integration with third-party systems to speed up innovation.
What Went Wrong

Since LeasePlan is a private company, and currently there is no public lawsuit going on,
it is hard to find information about what actually went wrong.
But in conducting research for this case study, I found a number of warning signs that
should have tipped off the company, its team, and its consultants that this was going
to be a very difficult project.
Here are a few of the warning signs, which were found in articles with various quotes
from LeasePlan executives touting the merits of the transformation:
> The company was consolidating 35 systems into one, which is typically a Herculean
task in and of itself.
98
> The company failed in a past SAP implementation at one of its affiliates, which took 4
years to implement and another 3 years to “settle down” after go-live.
> The company’s CIO touted its “extremely aggressive plan” for rolling out the new
technology, which is typically not a good thing.
> LeasePlan was pursuing a big bang rollout strategy for most countries it operated,
which creates a high-risk profile for most organizations.
> The team acknowledged that change management would be difficult, but that they
had it under control with a “comprehensive change management” plan.
How LeasePlan Could Have Done Things Differently

Consider your industry-specific needs and solutions
Fleet leasing is a fairly unique industry. Standard, off-the-shelf ERP systems are less
likely to address the variety of distinctive needs of companies in this space.
You may also be in a similar industry. For example, digital transformations are typically
more difficult for industries and companies such as these that are unique:
> Financial services

> Complex, engineer-to-order manufacturing
> Utilities and energy
> Any industry with field services
> Fast-changing industries
> Companies that are actively disrupting the status quo in their respective industries
Be skeptical of concepts like SAP’s Model Company, Oracle’s Unified Model, NetSuite’s
Suite Success and other ERP software industry hoaxes that may mislead you to
unrealistic expectations.
Industry best practices baked into software don’t exist. Neither do silver bullet
implementation solutions. It is important to plan and execute accordingly.
If you fit into any of the categories above, it is less likely that a single-vendor solution is
going to be the best fit for your strategic and operational needs. It is more likely that a
best of breed approach with multiple technologies is going to be a more viable option.
99
Deploy technology solutions that align with your business model
Digital transformations are too often misaligned with an evolving business model and
overarching corporate strategy. It is nearly impossible to succeed in this type of
situation.
Not only is LeasePlan in a unique industry, but it was also actively disrupting its
industry with new platforms such as CarNext.com and its Car-as-a-Service business
models.
This entrepreneurial spirit and new ways of doing business are typically not playing to
the strengths of S/4HANA. This misalignment likely created problems during the
transformation.
It is important that you understand your business model and define how your digital
transformation or project will support that bigger-picture strategy. If you don’t see or
don’t define the connection for the rest of your team, then you’re not ready to be
starting a large-scale transformation.
Be a responsible buyer of technology
Being a responsible buyer of technology and outsourced software development

services, and working well with suppliers during projects are crucial skills for any
organization.
Yet, the absence of those skills explains more project failures in third-party projects
than any other factor. You will find some prominent examples of these among these
and other project failure case studies.
Some may argue that suppliers should have all the skills required to make their
projects a success, but any company relying completely on the skills of a supplier is
making themselves dependant on good luck.
If you are not a ‘responsible buyer’ then you risk not spotting when the supplier and/or
the project is failing. Things will always start to drift and get off track. That’s a normal
part of complex business transformations like these. But the key is how we identify
risks, mitigate risks, and take action when warranted.
That could mean firing our systems integrator before we are $100M in the hole. It
could mean that we pivot on our strategy when we see that the original plan isn’t
100
working. Or it could mean that we divert resources from technical workstreams so we

can double-down on our organizational change management strategy.
Closing Thoughts
A responsible buyer of third-party systems and systems development will have
excellent knowledge, understanding and experience in defining, planning, directing,
tracking, controlling and reporting systems development projects. They will know what
should be done, when, why and how.
In many projects the supplier should be running the above-mentioned processes as

part of helping a buyer achieve their target business outcome (after all, the supplier is
expected to have done a great many projects of this type). However, this does not
mean that the supplier will, in fact, be doing all of those things.
That's why it is vital that the buyer themselves knows what needs to be done.
In most large technology projects, it is excellence in program and project management

that is the crucial factor in determining success — not knowledge of technology. This is
often true in situations when, for example, a project is being carried out across an
organization (especially a global organization); across a group of companies in
collaboration; or on behalf of a central marketplace and its participants (such as a
stock exchange).
In large business-critical projects neither the supplier nor the buyer should be doing
any aspect of the project in isolation, as doing so will increase the risk of failure. This
isn’t just a need for transparency, it is a need for active communication plus active
confirmation and verification that messages have been received and understood.
The following three excuses for total project failure will never work in court:
1) "I was drunk,"

2) "I thought the buyer or supplier knew what they were doing," and
3) "I thought the buyer or supplier was doing it, not me."
If you are the buyer and you do not have all the necessary skills and experience to be
able to define and control important projects (which is perfectly understandable as in
most companies they don’t happen very often), there is an easy fix for this problem:
Hire a very experienced interim executive to act on your behalf, even if the supplier
will still do most of the project management and other work. You can delegate
101
authority for doing the project management to the supplier but you cannot delegate
responsibility.
Responsibility for the project — including responsibility for it failing — always rests
ultimately with you, the buyer.
References
> Article Financial Times
> Article ZDNet
> Article FinTech Futures
> Article Consultancy UK
> LeasePlan Q2 2019 Results
102
Conclusion
I think most organizations feel that project sponsors, managers, and teams can reduce
project costs and duration by learning from past projects, by implementing past
successes, and by avoiding past failures.
But at the same time, many organizations have no standards for collecting, analyzing,
storing, disseminating, and reusing Lessons Learned. Consequently, they are losing
valuable knowledge gained during projects and between projects.
They seem to be able to learn the little lessons, like improving small aspects of
projects, but the big lessons seem to be relearned time and time again. Here is why:
No Priority
Projects are not making sufficient time for a Lessons Learned session and key people
(like the sponsor or main stakeholders) are not available for a Lessons Learned session.
Organizations have an ineffective lessons capture process
Lesson learning crucially needs a standard lessons reporting format and structure, an
effective approach to root cause analysis, a focus on lesson quality, openness and
honesty, and a validation process.
Project teams do not see the benefit of a Lessons Learned session
Lessons Learned captured on a project seldom benefit that project. They benefit future
projects. Often, the project sponsor and manager see capturing Lessons Learned as
simply another chore that provides his or her project with little value, especially if the
Lessons Learned procedure is complex, takes a fair amount of resources and time to
implement, and management has not provided adequate resources to perform the
work. The solution here is to have a simple procedure, ensure projects have the
resources and time to implement the procedure, and hold project managers
accountable for following the procedure.
Ineffective lessons dissemination process
The value of even well-crafted reports is often undermined because they are not
distributed effectively. Most dissemination is informal, and as a result development
and adoption of new practices is haphazard. Generally, project teams must actively
103
seek reports in order to obtain them. There is no trusted, accessible repository that
provides Lessons Learned information to project teams company-wide, although some
departments do have lessons repositories.
Lack of motivation to fix the issues
There is a reluctance to make big fixes if it's not what you are being rewarded for, a
reluctance to learn from other parts of the organization, and difficulties in deciding
which actions are valid.
Lack of dedicated resources
Commitment to learning is wasted if resources are not available to support the

process. Unfortunately, funds available to sustain corrective action, training, and
exercise programs are even leaner than those available for staff and equipment.
Lesson-learning and lesson management need to be resourced. Roles are needed to
support the process, such as those seen in the US Army and the RCAF, or in Shell.
Under-resourcing lesson-learning is a major reason why the process so often fails.
Lack of leadership involvement in and commitment to the learning process
This is the most critical barrier. An effective Lessons Learned process means having a
disciplined procedure that people are held accountable to follow. It means
encouraging openness about making mistakes or errors in judgment. It often means
cultural or organizational change, which does not come easily in most organizations. It
means leading by example. If management is unwilling to learn from their mistakes, it
is unlikely that the rest of the organization will be willing to admit to mistakes either. In
fact, management must reward people for being open and admitting to making
mistakes, bad decisions, judgment errors, etc. This, of course, flies in the face of many
corporate cultures.
Process change versus accountability
When something goes wrong on a project, there is someone accountable. One of the
biggest problems in implementing an effective Lessons Learned process is to separate
the “accountability” issue from the “process” issue. Accountability is important, but is
something to be dealt with by management. Lessons Learned must deal with the
process deficiency that caused the problem (e.g., inadequate procedure, too much of a
rush, inadequate training, poor communications, etc.). Once a Lessons Learned process
focuses on blame or finger-pointing, the process will soon fade into oblivion.
104
Not using Lessons Learned in the initiation and planning phases of new projects
You should ensure that projects in these stages incorporate Lessons Learned from prior
projects by making a Lessons Learned session mandatory.
Instead of leaning the little lessons, like improving small aspects of projects, I think it
would be very valuable to learn the big lessons, instead of relearning them time and
time again by making the same mistakes on similar projects.
There are many reasons why lesson learning is not working for most organizations.
Perhaps the underlying causes include organizations treating lesson learning as a
system, rather than as a product (i.e., a report with documented lessons), and a failure
to treat lesson learning with the urgency and importance that it deserves.
If learning lessons is important (and it usually is), then the process needs proper
attention, not lip service.
Newsletter
You can subscribe to my monthly newsletter here. You will receive articles I have
written, how to guides, and case studies around project management, program
management, and portfolio management.
Blog
With my blog “Doing the right projects and doing projects right” I want to provide you
with relevant, helpful content in all areas of project, program and portfolio
management that you can use to further your organizational and professional growth.
If you’re new to my site you best start here.
105
About the author

Hi, my name is Henrico Dolfing, and I help C-level
executives in the financial service industry with interim
management and recovering troubled technology
projects.
I have done project reviews, recoveries and advisory for

over a decade, and have worked in the trenches of
software development for more than 15 years.
Projects fail for a variety of reasons. Technology projects in particular have a low
success rate. Typically more than half of them are considered a failure. If your current
project is off-track, chances are I can bring the necessary knowledge and experience to
get the job done.
Troubled projects are never pretty. Often there isn’t time for guessing, just acting. I
have helped companies to save projects from the brink of failure. By combining my
deep technical knowledge with a proven process, I quickly address major problems and
bottlenecks, putting the project back on track.
I provide the leadership necessary to solve complicated problems. I am not afraid to

deliver bad news; I am not afraid of putting my reputation behind the decisions I make;
and most of all, I am responsive and sensitive to the intricacies of troubled projects.
I have a strong technical background as a software developer and solution architect,

including an M.Sc. degree in Computer Science and a B.Ec. in Economics.
My diverse international experience allows me to successfully collaborate with people

from all over the globe. Born in the Netherlands, I have lived in Germany, the USA, and
Switzerland. I have worked on both longer and shorter projects all through Europe, the
Channel Islands, the Caribbean, and North America.
I spend my spare time in the Swiss mountains enjoying alpine marathons, climbing,
downhill mountain biking, river rafting and leisurely hikes with my wife and toddler
son.
Connect with me on LinkedIn or via email:
LinkedIn: http://ch.linkedin.com/in/henricodolfing
106
Email: henrico.dolfing@data-solutions.ch
107

Free Gift - Project Failure Case Studies V1.8

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Free Gift - Project Failure Case Studies V1.8

Uploaded by

Copyright:

Available Formats

Project Failure Case Studies

Project Failure Case Studies

Lessons learned from other people’s mistakes

Copyright © 2020 Henrico Dolfing. All rights reserved.

Case Study 1: The £10 Billion IT Disaster at the NHS 6

Case Study 2: The Epic Meltdown of TSB Bank 16

Case Study 3: How a Screwed-Up SAP Implementation Almost Brought Down

Case Study 4: The $440 Million Software Error at Knight Capital 32

Case Study 7: Target’s $2.5 Billion Cross-Border Expansion Mistake 60

About the author 105

“A lesson learned is knowledge or understanding gained by experience. The

Personally I like the following definition:

“Generalizations based on evaluation experiences with projects, programs, or

This eBook is an ever-growing collection of such project failure case studies.

Case Study 1: The £10 Billion IT Disaster at the NHS

After a history marked by delays, stakeholder opposition and implementation issues,

Since NPfIT was a public-sector program, there is a large amount of documentation

>​ BT awarded contract for the national data spine

>​ BT awarded N* (NHS broadband network) contract

>​ Accenture withdraws as local service provider

What Went Wrong

It identifies three main themes:

3)​ Culture and skills

>​ An unrealistic timetable

>​ No time to engage with users and privacy campaigners

>​ Inadequate preliminary work

>​ Failure to check progress against expectations

>​ Failure to test systems

>​ Failure to recognize the risks or limitations of big IT projects

>​ Sheer ambition

>​ Confidentiality issues

Culture and skills

>​ A lack of clear leadership

>​ Not knowing, or continually changing, the aim of the project

>​ Not committing the necessary budget from the outset

>​ Not providing training

>​ A lack of concern for privacy issues

>​ No exit plans and no alternatives

>​ A lack of project management skills

>​ Treasury emphasis on price over quality

How NHS Could Have Done Things Differently

Understanding the problem

Rarely is a project ever just an IT project; generally it should be viewed as a broader

It is a given of successful technology projects that there should be good consultation

Start slow in order to run fast later

Balancing risk and reward

The combination of the implemented payment provisions (under which service

Checks and balances

Case Study 2: The Epic Meltdown of TSB Bank

The Basic Principles of a System Migration

Testing a System Migration

Functional testing of a migration is somewhat different from functional testing of a

In a migration, there is often change in interface parameters used by one system to

What Went Wrong

How TSB Could Have Done Things Differently

Finally, in complex migrations, a phased approach should be considered.

When migrating IT systems in the financial sector, cutting corners is dangerous.

Case Study 3: How a Screwed-Up SAP

reconciliation would take place later.

>​ Paying employees accurately and on time

>​ Paying vendors accurately and timely

>​ Failing to prepare design documents and specifications to industry standards.

>​ Failing to prepare programming and configuration to industry standards.

>​ Failing to adequately test, detect, and inform of problems.

> BT awarded contract for the national data spine

> BT awarded N* (NHS broadband network) contract

> Accenture withdraws as local service provider

3) Culture and skills

> An unrealistic timetable

> No time to engage with users and privacy campaigners

> Inadequate preliminary work

> Failure to check progress against expectations

> Failure to test systems

> Failure to recognize the risks or limitations of big IT projects

> Sheer ambition

> Confidentiality issues

> A lack of clear leadership

> Not knowing, or continually changing, the aim of the project

> Not committing the necessary budget from the outset

> Not providing training

> A lack of concern for privacy issues

> No exit plans and no alternatives

> A lack of project management skills

> Treasury emphasis on price over quality

> Paying employees accurately and on time

> Paying vendors accurately and timely

> Failing to prepare design documents and specifications to industry standards.

> Failing to prepare programming and configuration to industry standards.

> Failing to adequately test, detect, and inform of problems.

> Overly ambitious design

> Significantly underestimated scale of transformation needed

> Limited availability of internal personnel due to ambitious business agenda

> Multi-partner model did not deliver business benefits

> Lack of ownership of certain business processes

> Inadequate quality of data from legacy systems

> Training methods proved ineffective

> Northstar Consulting Group (2014). A comprehensive management and operations

> Customer service levels were disrupted

> It incurred significant capital and operating expenditures

> It experienced difficulty processing payments to vendors