Risks and Liabilities

of Computer-based
Systems
Overview

 Historical examples of software risks

 Implications of software complexity
 Risk assessment and management
Historical Examples

Software errors
Can KILL
Indirectly
Directly

Cost MONEY
Loss of equipment
Loss of business
Software Aids and Abets
Murder: 1992
A computer detected the tampering
 when it called a second computer to report
the incident, the first computer received a
busy signal and never called back
Radiation Machine Kills Four:
1985 to 1987
 Faulty software in a Therac-25
radiation-treatment machine resulted in
several cancer patients receiving lethal
overdoses of radiation
Radiation Machine Kills Four:
1985 to 1987
Four patients died
 Radiation Machine Kills Four:
1985 to 1987
 When their families sued, all the cases
were settled out of court
 There were several errors,
 among them the failure of the programmer
to detect a race condition (i.e.,
miscoordination between concurrent tasks)
 Radiation Machine Kills Four:
1985 to 1987
 Itwas found that found that accidents
occurred even after AECL thought it had
fixed particular bugs
 "A lesson to be learned from the Therac-25
story is that focusing on particular software
bugs is not the way to make a safe system”
 "The basic mistakes here involved poor
software-engineering practices and building
a machine that relies on the software for
safe operation”
Hyphen Costs $80 Million: 1962

A probe launched from Cape Canaveral

was set to go to Venus
 After takeoff, the unmanned rocket carrying
the probe went off course
 NASA had to blow up the rocket to avoid
endangering lives on earth
Hyphen Costs $80 Million: 1962
 NASA later attributed the error to a
faulty line of Fortran code
 “Somehow a hyphen had been dropped
from the guidance program loaded aboard
the computer, allowing the flawed signals to
command the rocket to veer left and nose
down
 ...Suffice it to say, the first U.S. attempt at
interplanetary flight failed for want of a
hyphen”
Hyphen Costs $80 Million: 1962

 Thevehicle cost more than $80 million,

prompting Arthur C. Clarke to refer to
the mission as “the most expensive
hyphen in history”
AT&T Long Distance Service
Fails: 1991
 Inthe summer of 1991, telephone
outages occurred in local telephone
systems in California and along the
Eastern seaboard
 These breakdowns were all the fault of
an error in signalling software
AT&T Long Distance Service
Fails: 1991
 Right before the outages
 DSC Communications introduced a bug
when it changed three lines of code in the
several-million-line signalling program
 After this tiny change, nobody thought it
necessary to retest the program
AT&T Long Distance Service
Fails: 1991
 These switching errors in AT&T's call-
handling computers caused the
company's long-distance network to go
down for nine hours
 The meltdown affected thousands of
services and was eventually traced to a
single faulty line of code
There’s a Hole in the Bucket

 Small systems
 …form part of larger systems
A fault within a small part could
result in a catastrophe later on
There’s a Hole in the Bucket

 Designers have an ethical

responsibility to design the best
system possible
Bugs
 Bugs exist because
 …humans aren't perfect
 Since humans design and program
hardware and software, mistakes are
inevitable
 That's what computer and software
vendors tell us, and it's partly true
 What they don't say is that software is
buggier than it has to be
Bugs

 This is how bugs are born

 Because time is money, especially in
the software industry
Bugs

A software or hardware company sees a

business opportunity and starts building
a product to take advantage of that
 Long before development is finished,
the company announces that the product
is on the way
Bugs

 Allthe while pressuring the

software engineers to add more and
more features
Bugs
 Shareholders and venture capitalists
clamour for quick delivery because
that's when the company will see the
biggest surge in sales
 Meanwhile, the quality-assurance
division has to battle for sufficient bug-
testing time
Bugs
 “Thesimple fact is that you get the most
revenues at the release of software,”
 “The faster you bring it out, the more
money you make. You can always fix it
later, when people howl. It's a fine line
when to release something, and the
industry accepts defects“
What Is Risk Assessment and
Management?
 Riskand uncertainty are fundamental
elements of modern life
 They are ever present in the actions of
human beings and they are frequently
magnified in large-scale technological
systems
 Risk and uncertainty must be managed
effectively to protect people from injury
and to permit the development of reliable,
high-quality products
What Is Risk Assessment and
Management?
 Risk is often defined as a measure of the
probability and severity of adverse
effects
What Is Risk Assessment and
Management?
 In risk assessment, the analyst often
attempts to answer the following set of
triplet questions
 What can go wrong?
 What is the likelihood that it would go
wrong?
 What are the consequences?
What Is Risk Assessment and
Management?
 Answers to these questions help risk
analysts identify, measure, quantify, and
evaluate risks and their consequences
and impacts
What Is Risk Assessment and
Management?
 Risk management builds on the risk
assessment process by seeking answers
to a second set of three questions
 What can be done?
 What options are available and what are
their associated trade-offs in terms of all
costs, benefits, and risks?
 What are the impacts of current
management decisions on future options?
What Is Risk Assessment and
Management?
 To be effective and meaningful, risk
management must be an integral part of
the overall management of a system
 This is particularly important in the
management of technological systems,
where the failure of the system can be
caused by the failure of the hardware,
the software, the organization, or the
humans