CTEC5163 (2021) Software Quality Assurance

CTEC5163
Software Quality Assurance and Testing
@CyberTechnologyMSc
1
Motivation
2
MODULE INTRODUCTIONS
That’s me
Dr Feng Chen
Senior Lecturer, Deputy Faculty Head of Research Students
Coordinator of Doctoral Training Programme in Cyber Technology
Head of Context Intelligence and Interaction Research Group (CIIRG)
De Montfort University
5.32 Gateway House, Leicester, LE1 9BH, England
Phone: +44(0)116 257 7860
Email: fengchen@dmu.ac.uk
4
Dr Francois Siewe
Reader in Computer Science

Head of Software Technology Research Laboratory (STRL)
Cyber Technology Institute (CTI)
School of Computer Science and Informatics
De Montfort University
The Gateway
Leicester LE1 9BH, UK
Room GH5.31, Gateway House
Tel: (44) 1162577938
Email: FSiewe@dmu.ac.uk
5
CYBER TECHNOLOGY @ DMU
Smart – Safe – Secure
Cyber Technologies
Software Technology Research Laboratory

&
Cyber Security Centre
@
De Montfort University in Leicester UK
Our mission
• We aspire to provide the full benefits to all of a smart,
safe, secure and resilient cyberspace.
• We are committed to a programme of sustained
excellence in research, training and post-graduate
education in Cyber Technologies and Software
Engineering.
• Our aim is the advancement of formal and practical
approaches to the development of distributed, smart
and context-aware systems that operate in real-time,
safety and security critical applications.
8
Three rules for this module
1. Don’t be afraid to ask — ever!

2. Enjoy your study and try to apply what you learnt.
3. Work with other students and staff to create some lasting value for
yourself, your employer and your tutors.
9
What am I learning here?
Software Quality Assurance and Testing
Module Schedule
11
Module Sessions
S1: Module S5: CMM S9: Project S13: Structure- Coursework Spec
Introductions Evaluation based Techniques
&Estimation
S2: Quality and S6: Software S10: Quality S14: Formal Coursework Spec
Quality Project Assurance Method &
Assurance Management Techniques software Testing
S3: ISO 9000 S7: Software S11: S15: Test

Quality Introduction to Management
Management Testing New Trend
S4: Quality in S8: Risk S12: Test S16: Course Recap

Software Management; Specification & Online test
Processes Design
12
Online Test
• You have 60min (40 questions) to complete this
test
• You must achieve 70% of the marks
• You must pass this before you can take the
academic coursework
• You can retake this up to 3 times
• It is “closed book”
• The Quiz at the end of each section prepare you
for this
13
Coursework
• If you are enrolled on the MSc CyberTech you will
have to complete a coursework
• On Thursday your Lecturer will give detailed
coursework instructions to you
• You can join in on Friday to discuss your approach
and timeline with your Lecturer
• You have three months to complete your
coursework
• It must be submitted before the deadline set in
the coursework
14
Aim
• The module aims to provide students with insight into the
effective testing methodologies for quality assurance in
software engineering process.
– It covers basic concepts, sound principles, best practices, and

rigorous approaches for testing software and assuring its quality.
– The emphasis is placed on the skills of problem formulation,

modelling and problem solving for building and delivering high
quality software.
Learning Outcomes
• On successful completion of this module a student
should be able to:
– O1. Understand software processes and software quality topics
deeply;
– O2. Apply software quality assurance activities, methods, and
techniques for an industrial project;
– O3. Analyse and evaluate software quality and it models, in terms
of change, configuration and quality management;
– O4. Understand and apply advanced knowledge of testing, testing
techniques and the role of formal methods in the testing process;
– O5. Design and perform effective software testing with tool support
in structured and managed ways;
– O6. Analyse or develop testing methods using new (e.g. formal
methods) and conventional software testing techniques.
Motivation
• Software Crisis
– Faulty software
– Delay in completion time
– Over budgeted
– Difficult to maintain software
How to develop quality software?

Some UK projects
• Home Office asylum seeker Casework system:
£80m; abandoned 2001 (Siemens).
• Pathway UK benefits identity card system:
abandoned 1999 with £1bn losses (ICLFujitsu).
• Datacentre for UK cabinet office: £85m, cancelled in
2004 (ITNET).
• MRC Laboratory of Molecular Biology, FAMIS
purchasing system: £55m; ‘operating failure’
(LogicaGMC).
• UK e-University: £62m; abandoned 2004 (Sun,
though only £20m of the full figure).
American experience
• 25% of development projects are abandoned
• probability of cancellation rises to 50% for largest
developments
• average project overshoots schedule by 50%
• 75% of systems are regarded as ‘operating failures’
U.S. Bureau of Labor Statistics, 1997
 IBM Consulting group estimates that 55% of large

distributed systems projects cost more than expected,
68% overrun their schedules, and 88% require redesign.
 Software errors cost the U.S. economy $60 billion
annually in rework, lost productivity and actual damages.
20 Famous Software Disasters
http://www.devtopics.com/20-famous-software-
disasters/
D1. Mariner Bugs Out (1962)
1. Cost: $18.5 million
2. Disaster: The Mariner 1 rocket with a space
probe headed for Venus diverted from its
intended flight path shortly after launch.
Mission Control destroyed the rocket 293
seconds after liftoff.
3. Cause: A programmer incorrectly
transcribed a handwritten formula into
computer code, missing a single superscript
bar. Without the smoothing function
indicated by the bar, the software treated
normal variations of velocity as if they were
serious, causing faulty corrections that sent
the rocket off course.
D2. Hartford Coliseum Collapse (1978)
1. Cost: $70 million, plus another $20 million
damage to the local economy
2. Disaster: Just hours after thousands of
fans had left the Hartford Coliseum, the
steel-latticed roof collapsed under the
weight of wet snow.
3. Cause: The programmer of the CAD
software used to design the coliseum
incorrectly assumed the steel roof supports
would only face pure compression. But when
one of the supports unexpectedly buckled
from the snow, it set off a chain reaction
that brought down the other roof sections
like dominoes.
D3. CIA Gives the Soviets Gas (1982)
1. Cost: Millions of dollars, significant damage
to Soviet economy
2. Disaster: Control software went haywire and
produced intense pressure in the Trans-
Siberian gas pipeline, resulting in the largest
man-made non-nuclear explosion in Earth’s
history.
3. Cause: CIA operatives allegedly planted a
bug in a Canadian computer system purchased
by the Soviets to control their gas pipelines.
The purchase was part of a strategic Soviet
plan to steal or covertly obtain sensitive U.S.
technology. When the CIA discovered the
purchase, they sabotaged the software so
that it would pass Soviet inspection but fail
in operation.
D4. World War III… Almost (1983)
1. Cost: Nearly all of humanity
2. Disaster: The Soviet early warning
system falsely indicated the United
States had launched five ballistic
missiles. Fortunately the Soviet duty
officer had a “funny feeling in my gut”
and reasoned if the U.S. was really
attacking they would launch more than
five missiles, so he reported the
apparent attack as a false alarm.
3. Cause: A bug in the Soviet software
failed to filter out false missile
detections caused by sunlight reflecting
off cloud-tops.
D5. Medical Machine Kills (1985)
1. Cost: Three people dead, three
people critically injured
2. Disaster: Canada’s Therac-25
radiation therapy machine
malfunctioned and delivered lethal
radiation doses to patients.
3. Cause: Because of a subtle bug
called a race condition, a technician
could accidentally configure Therac-
25 so the electron beam would fire
in high-power mode without the
proper patient shielding.
D6. Wall Street Crash (1987)
1. Cost: $500 billion in one day
2. Disaster: On “Black Monday” (October 19,
1987), the Dow Jones Industrial Average
plummeted 508 points, losing 22.6% of its
total value. The S&P 500 dropped 20.4%.
This was the greatest loss Wall Street ever
suffered in a single day.
3. Cause: A long bull market was halted by a
rash of SEC investigations of insider trading
and by other market forces. As investors
fled stocks in a mass exodus, computer
trading programs generated a flood of sell
orders, overwhelming the market, crashing
systems and leaving investors effectively
blind.
D7. AT&T Lines Go Dead (1990)
1. Cost: 75 million phone calls missed, 200
thousand airline reservations lost
2. Disaster: A single switch at one of AT&T’s
114 switching centers suffered a minor
mechanical problem and shut down the
center. When the center came back up, it
sent a message to other switching centers,
which in turn caused them to shut down and
brought down the entire AT&T network for 9
hours.
3. Cause: A single line of buggy code in a
complex software upgrade implemented to
speed up calling caused a ripple effect that
shut down the network.
D8. Patriot Fails Soldiers (1991)
1. Cost: 28 soldiers dead, 100 injured
2. Disaster: During the first Gulf War,
an American Patriot Missile system
in Saudi Arabia failed to intercept
an incoming Iraqi Scud missile. The
missile destroyed an American Army
barracks.
3. Cause: A software rounding error
incorrectly calculated the time,
causing the Patriot system to ignore
the incoming Scud missile.
D9. Pentium Fails Long Division (1993)
1. Cost: $475 million, corporate credibility
2. Disaster: Intel’s highly-promoted Pentium chip
occasionally made mistakes when dividing floating-
point numbers within a specific range. For
example, dividing 4195835.0/3145727.0 yielded
1.33374 instead of 1.33382, an error of 0.006%.
Although the bug affected few users, it become a
public relations nightmare. With an estimated 5
million defective chips in circulation, Intel
offered to replace Pentium chips only for
consumers who could prove they needed high
accuracy. Eventually Intel replaced the chips for
anyone who complained.
3. Cause: The divider in the Pentium floating point
unit had a flawed division table, missing about five
of a thousand entries and resulting in these
rounding errors.
D10. Ariane Rocket Goes Boom (1996)
1. Cost: $500 million
2. Disaster: Ariane 5, Europe’s newest
unmanned rocket, was intentionally destroyed
seconds after launch on its maiden flight.
Also destroyed was its cargo of four
scientific satellites to study how the Earth’s
magnetic field interacts with solar winds.
3. Cause: Shutdown occurred when the
guidance computer tried to convert the
sideways rocket velocity from 64-bits to a
16-bit format. The number was too big, and
an overflow error resulted. When the
guidance system shut down, control passed to
an identical redundant unit, which also failed
because it was running the same algorithm.
D11. Skynet Brings Judgement Day (1997)
1. Cost: 6 billion dead, near-total destruction of human
civilisation and animal ecosystems (fictional)
2. Disaster: Human operators attempt to shut off the
Skynet global computer network. Skynet responds by
firing U.S. nuclear missiles at Russia, initiating global
nuclear war on what became known as Judgement Day
(August 29, 1997).
3. Cause: Cyberdyne, the leading weapons manufacturer,
installed Skynet technology in all military hardware
including stealth bombers and missile defense systems.
The Skynet technology formed a seamless network and
effectively removed humans from strategic defense.
Eventually Skynet became sentient, was threatened
when the humans tried to take it offline, sought to
survive, and retaliated with nuclear war.
D12. Mars Climate Crasher (1998)
1. Cost: $125 million
2. Disaster: After a 286-day journey
from Earth, the Mars Climate Orbiter
fired its engines to push into orbit
around Mars. The engines fired, but
the spacecraft fell too far into the
planet’s atmosphere, likely causing it to
crash on Mars.
3. Cause: The software that controlled
the Orbiter thrusters used imperial
units (pounds of force), rather than
metric units (Newtons) as specified by
NASA.
D13. Disastrous Study (1999)
1. Cost: Scientific credibility
2. Disaster: In this ironic case,
software used to analyse disasters
had a disaster of its own. The New
England Journal of Medicine
reported increased suicide rates
after severe natural disasters.
Unfortunately, these results proved
to be incorrect.
3. Cause: A programming error caused
the number of suicides for one year
to be doubled, which was enough to
throw off the entire study.
D14. British Passports to Nowhere (1999)
1. Cost: £12.6 million, mass inconvenience
2. Disaster: The U.K. Passport Agency
implemented a new Siemens computer
system, which failed to issue passports on
time for a half million British citizens. The
Agency had to pay millions in compensation,
staff overtime and umbrellas for people
queuing in the rain for passports.
3. Cause: The Passport Agency rolled out its
new computer system without adequately
testing it or training its staff. At the same
time, a law change required all children under
16 traveling abroad to obtain a passport,
resulting in a huge spike in passport demand
that overwhelmed the buggy new computer
system. (more)
D15. Y2K (1999)
1. Cost: $500 billion
2. Disaster: One man’s disaster is another
man’s fortune, as demonstrated by the
infamous Y2K bug. Businesses spent billions
on programmers to fix a glitch in legacy
software. While no significant computer
failures occurred, preparation for the Y2K
bug had a significant cost and time impact on
all industries that use computer technology.
3. Cause: To save computer storage space,
legacy software often stored the year for
dates as two digit numbers, such as “99″ for
1999. The software also interpreted “00″ to
mean 1900 rather than 2000, so when the
year 2000 came along, bugs would result.
D16. Dot-Bomb Collapse (2000)
1. Cost: $5 trillion in market value, thousands
of companies failed
2. Disaster: A speculative bubble from 1995–
2001 fueled a rapid increase in venture
capital investments and stock market values
in the Internet and technology sectors. The
“dot-com bubble” began to collapse in early
2000, erasing trillions in stock market value,
wiping out thousands of companies and jobs,
and launching a global recession.
3. Cause: Companies and investors dismissed
standard business models, and instead
focused on increasing market share at the
expense of profits.
D17. Love Virus (2000)
1. Cost: $8.75 billion, millions of computers
infected, significant data loss
2. Disaster: The LoveLetter worm infected
millions of computers and caused more
damage than any other computer virus in
history. The worm deleted files, changed
home pages and messed with the Registry.
3. Cause: LoveLetter infected users via e-mail,
Internet chat and shared file systems. The
email had an executable file attachment and
subject line, “ILOVEYOU.” When the user
opened the attachment, the virus would
infect the user’s computer and send itself to
everyone in the address book.
D18. Cancer Treatment to Die For (2000)
1. Cost: Eight people dead, 20 critically
injured
2. Disaster: Radiation therapy software
by Multidata Systems International
miscalculated the proper dosage,
exposing patients to harmful and in
some cases fatal levels of radiation.
The physicians, who were legally
required to double-check the software’s
calculations, were indicted for murder.
3. Cause: The software calculated
radiation dosage based on the order in
which data was entered, sometimes
delivering a double dose of radiation.
D19. EDS Drops Child Support (2004)
1. Cost: £539 million and counting
2. Disaster: Business services giant EDS
developed a computer system for U.K.’s
Child Support Agency (CSA) that
accidentally overpaid 1.9 million people,
underpaid another 700,000, had £3.5
billion in uncollected child support
payments, a backlog of 239,000 cases,
36,000 new cases “stuck” in the system,
and still over 500 documented bugs.
3. Cause: EDS introduced a large, complex
IT system to the CSA while trying to
simultaneously restructure the agency.
D20. FBI’s Trilogy Terminated (2005)
1. Cost: $105 million, still no effective
case file solution
2. Disaster: The FBI scrapped its
computer systems overhaul after four
years of effort. The Virtual Case File
project was a massive, integrated
software system for agents to share
case files and other information.
3. Cause: Mismanagement, and an attempt
to build a long-term project on
technology that was outdated before
the project completed, resulted in a
complex and unusable system.
Latest: Boeing 737 Max 8
1. In October 2018, Lion Air Flight
610 crashed in Indonesia, killing
189 people.
2. In March 2019, Ethiopian Airlines
Flight 302 crashed in Bishoftu,
Ethiopia, killing 157 people.
3. Software problem; the automated
stall-prevention system was
malfunctioning.
SESSION 2: SOFTWARE QUALITY
AND QUALITY ASSURANCE
Overview
1. Quality Assurance
2. Software Quality
3. Quality factors and criteria
4. Quality Models
5. Software Quality Assurance
What is Quality?
• “degree of excellence”
• “fitness for purpose” (Juran)
• “zero defects” (Crosby)
• “all about reducing variation”(Deming)
ISO Definition – “ the totality of features and
characteristics of a product or service that bear
on its ability to satisfy specified or implied
needs”
Quality (Cont’d)
• Quality is not absolute
• Quality is multi-dimensional
• Quality is subject to constraints
• Quality is about acceptable compromises
• Quality criteria are not independent
What is Assurance?
• “promise, pledge, assertion made to
inspire confidence”
What is Quality Assurance?
• Quality assurance (QA) is a way of preventing
mistakes or defects in manufactured products
and avoiding problems when delivering
solutions or services to customers; which ISO
9000 defines as "part of quality
management focused on providing confidence
that quality requirements will be fulfilled".
– Objective: Assure conformance with requirements.
– Focus on fault prevention.
– Umbrella Activity.
– Applied over the whole process.
The Quality Revolution History (1)
• Started in Japan by Deming, Juran, and Ishikawa during
1940s
• In 1950s, Deming introduced statistical quality control to
Japanese engineers
• Statistical quality control (SQC) is a discipline based on
measurement and statistics
– SQC methods use seven basic quality management tool
• Pareto analysis, Trend Chart, Flow chart, Histogram,
Scatter diagram, Control chart, Cause and effect
diagram
• “Lean principle” was developed by Taiichi Ohno of Toyota
“A systematic approach to identifying and eliminating waste
through continuous improvement, flowing the product at
the pull of the customer in pursuit of perfection.”
The Quality Revolution History(2)
Figure 1.1: The Shewhart cycle

• Deming introduced Shewhart’s PDCA cycle to Japanese researchers
• It illustrate the activity sequence:
– Setting goals
– Assigning them to measurable milestones
– Assessing the progress against the milestones
– Take action to improve the process in the next cycle
49
The Quality Revolution History (3)
• In 1954, Juran spurred the move from SQC to TQC (Total Quality Control)
• Key Elements of TQC:
– Quality comes first, not short-term profits
– The customer comes first, not the producer
– Decisions are based on facts and data
– Management is participatory and respectful of all employees
– Management is driven by cross-functional committees
• An innovative methodology developed by Ishikawa called cause-and-effect
diagram
Figure 1.2: Ishikawa diagram

Overview
2. Software Quality
4. Quality Models
What is special about Software?
• Software is: Computer programs, procedures, and
possibly associated documentation and data
pertaining to the operation of a computer system.
[IEEE_Std_610.12-1990]
– Invisibility of the product
– Limited opportunities to detect defects (“bugs”)
– Often new demanding functionality has to be
realised
– Often software has to realize extraordinary high
complexity
Software Quality
• Five Views of Software Quality:
– Transcendental view
– User’s view
– Manufacturing view
– Product view
– Value-based view
• Software Quality in terms of quality factors and criteria
– A quality factor represents behavioral characteristic of a system
• Examples: correctness, reliability, efficiency, and testability
– A quality criterion is an attribute of a quality factor that is related to software development
• Example: modularity is an attribute of software architecture
• Quality Models: With models of quality we can articulate quality goals and progress to measuring
aspects of quality
– Gilb — “design by measurable objectives”
– Factor-Criteria-Metric model (Mc Call)
– Boehm’s Model
– COQUAMO (COnstructive QUAlity MOdel) (Kitchenham & Walker)
– ISO 9126 standard quality model
– Goal-Question-Metric Model
53
Transcendental view
• Quality is something that can be recognized
through experience, but not defined in some
tractable form.
• A good quality object stands out, and it is
easily recognized.
54
User view
• Quality concerns the extent to which a product
meets user needs and expectations.
• Is a product fit for use?
• This view is highly personalized.
– A product is of good quality if it satisfies a
large number of users.
– It is useful to identify the product attributes
which the users consider to be important.
• This view may encompass many subject
elements, such as usability, reliability, and
efficiency.
55
Manufacturing view
• This view has its genesis in the manufacturing industry – auto and
electronics.
• Key idea: Does a product satisfy the requirements?
– Any deviation from the requirements is seen as reducing the
quality of the product.
• The concept of process plays a key role.
• Products are manufactured “right the first time” so that the cost is
reduced
– Development cost
– Maintenance cost
• Conformance to requirements leads to uniformity in products.
• Some argue that such uniformity does not guarantee quality.
• Product quality can be incrementally improved by improving the
process.
– The CMM and ISO 9001 models are based on the manufacturing
view.
56
Product view
• Hypothesis: If a product is manufactured with
good internal properties, then it will have good
external properties.
• One can explore the causal relationship
between internal properties and external
qualities.
• Example: Modularity enables testability.
57
Value-based view
• This represents the merger of two concepts:
excellence and worth.
• Quality is a measure of excellence, and value is
a measure of worth.
• Central idea
– How much a customer is willing to pay for
a certain level of quality.
– Quality is meaningless if a product does not
make economic sense.
– The value-based view makes a trade-off
between cost and quality.
58
Quality Models
• Such general definitions of software quality
are not sufficient in practice
• Thus, software quality is described by specific
quality models
• Causal relationship from intangible quality
views to tangible software measures
• Standard Models:
– McCall
– ISO/IEC 9126
59
Reasons for quantitative view
• Reasons for developing a quantitative view of
quality
– Measurement allows us to establish
baselines for qualities.
– Measurement is key to process
improvement.
– The needs for improvements can be
investigated after performing measurements.
60
Measurement of user’s view
• User’s View:
– Reliability (number of failures over time)
– Availability
– Usability etc.
– Often directly measurable
• Apply Gilb’s technique.
– The quality concept is broken down into component parts
until each can be stated in terms of directly measurable
qualities.
– Example: Usability can be broken down into
• Learnability
• Understandability
• Operability
61
Measurement of manufacturer’s view
• Manufacturers are interested in defect count and rework cost.
• Defect count: The total number of defects detected during development and
operation.
– It is a measure of the quality of the work produced.
– One can analyze the defects as follows.
• For each defect, identify the development phase in which it was
introduced.
• Categorize the defects, say, based on modules.
• Normalize defect count by product size.
– Defect density: Number of defects per 1000 lines of code.
• Separate the defects found during operation from those found
during development.
• Rework cost: How much does it cost to fix the known defects?
– Development rework cost: This is the rework cost incurred before a
product is released. This is a measure of development efficiency.
– Operation rework cost: This is the rework cost incurred when a product
is in operation. This is a measure of the delivered quality.
62
Overview
2. Software Quality
4. Quality Models
Quality Factors
• Quality Factors(User)
– McCall, Richards, and Walters studied the concept of software
quality in terms of two key concepts as follows:
• quality factors, and
• quality criteria.
– A quality factor represents the behavioral characteristic of a
system.
• Examples: correctness, reliability, efficiency, testability,
portability, …
– A quality criterion (developer) is an attribute of a quality factor
that is related to software development.
• Example:
– Modularity is an attribute of the architecture of a
software system.
– A highly modular software allows designers to put
cohesive components in one module, thereby increasing
the maintainability of the system.
64
McCall’s Quality Factors
McCall et al. identified 11 quality factors
65
Categorization of McCall’s quality factors
• The 11 quality factors have been grouped into
three broad categories
– Product operation
– Product revision
– Product transition
66
Quality Criteria
• Quality Criteria (developer)
– A quality criterion is an attribute of a quality
factor that is related to software development.
– Example:
• Modularity is an attribute of the architecture
of a software system.
• A highly modular software allows designers
to put cohesive components in one module,
thereby increasing the maintainability of the
system.
67
McCall’s 23 Quality Criteria
68
Quality Factors and Criteria
• Relationship Between Quality Factors and Quality
Criteria
– Each quality factor is positively influenced by a set
of quality criteria, and the same quality criterion
impacts a number of quality factors.
• Example: Simplicity impacts reliability, usability,
and testability.
– If an effort is made to improve one quality factor,
another quality factor may be degraded.
• Portable code may be less efficient.
– Some quality factors positively impact others.
• An effort to improve the correctness of a system
will increase its reliability.
69
McCall’s Quality Factors and Criteria
70
The ISO 9126 Quality Characteristics
• The ISO 9126 document is the product of an international effort.
– ISO: International Organization for Standardization
• The document defines six broad quality characteristics.
• The document includes an example quality model that further
decomposes the quality characteristics into 20 subcharacteristics.
71
20 Quality Subcharacteristics
72
The ISO 9126 Quality Characteristics
73
Overview
2. Software Quality
4. Quality Models
Gilb approach
• Break high level concepts which cannot be measured
directly into lower level concepts which can.
• For example, usability:
– complexity of the interface (e.g. number of key-
strokes/dialogue-boxes required to perform simple
operations),
– amount of training required,
– response time,
– help and navigation aids available.
• This is a do-it-yourself approach to modelling quality
—analysis is tailored to the specific product.
75
McCall’s model (1977)
76
Boehm’s Model (1978)
77
COQUAMO (Kitchenham and Walker)
• The COnstructive QUAlity MOdel extended Gilb’s
approach and automated tool-support.
• It provides a set of Templates that specify for example:
– classification
– required level
– associated qualities
– definition
– measurement units
– measurement tool
– measurement conditions
– worst/planned/best/current
– justification and the consequences of failure.
78
ISO 9126 model
79
GQM model
80
Example
81
GQM Discussion
• Benefits :
– generates only those measures relevant to the goal
– Several measurements may be needed to answer a single
question.
– A single measurement may apply to more than one question.
– The goal provides the purpose for collecting the data.
• Not evident from the GQM
– The model needed to combine the measurement in a sensible
way so that the question can be answered.
– It must be supplemented by one or more models that express
the relationship among the metrics. (equation definition is not
clear)
• Disadvantages:
– Additional efforts to derive the goals and metrics
– Error prone compared to standard models
82
McCall’s model vs. ISO 9126 model
• McCall’s quality model vs. ISO 9126 model
– What is called quality factor in McCall’s model is called a quality subcharacteristic in the
ISO 9126 model.
– The following quality factors/characteristics are found in both the models.
• reliability, usability, efficiency, maintainability, and portability
– Differences between the two models
• The ISO 9126 model emphasizes on characteristics visible to the users, whereas the
McCall model considers internal qualities as well.
• In McCall’s model, one quality criterion can impact many quality factors, whereas in
the ISO 9126 model, one subcharacteristic impacts exactly one quality characteristic.
• A high-level quality factor, such as testability, in the McCall’s model is a low-level
subcharacteristic of maintainability in the ISO 9126 model.
• Concerns
– There is no consensus about what high-level quality factors are most important at the top
level. McCall, et al. define 11 high-level quality factors, whereas there are only six in the ISO
9126 document.
– There is no consensus regarding what is a top-level quality factor/ characteristic and what is a
more concrete quality criterion/ subcharacteristic.
83
4 parts of ISO9126
• quality model
• external metrics
• internal metrics
• quality in use metrics.
Metric
Seven Criteria for a Good Metric (Watts 1987)
• Objectivity – results free from subjectivity
• Reliability – results precise and repeatable
• Validity – measures the correct characteristics
• Standardisation – unambiguous and allows for
comparison
• Comparability – comparable with other
measures of the same criterion
• Economy – simpler and cheaper the better
• Usefulness – must address a need
Measurable Properties
Many metrics based on the same fundamental
measurable property. In practice, metrics are
often based on 7 distinct measurable properties
• Readability
• Error Prediction
• Error Detection
• Complexity
• Mean Time to Failure (MTTF)
• Modularity
• Testability
Examples of Metrics
• Readability as a Measure of Usability
– Fog Index (Gunning) for documentation
– Fog Index = 0.4a + b, Where “a” is number of words in a sentence and “b” is % of
words with more than 2 syllables
• Readability of Code as a Measure of Maintainability
– De Young and Kampen proposed measuring readability in terms of: Number of
statement lines; Average length of variable names; Total number of program
branches
• Error detection as a Measure of Correctness
– Remus and Zilles uses the number of detected errors and a measure of error
detection efficiency to predict the number of errors remaining undetected
– Assumes number of defects proportional to length of the program and that the
rate of error removal can be predicted
Overview
2. Software Quality
4. Quality Models
What is Software Quality Assurance (SQA)?
• “Set of systematic activities providing

evidence of the ability of the software
process to produce a software product that
is fit to use”
– G. Schulmeyer and J. McManus, Software
Quality Handbook, Prentice Hall, 1998.
What is SQA?
• Monitoring processes and products throughout the
software development lifecycle to ensure the quality of
the delivered product(s)
• Monitoring the processes
– Provides management with objective feedback
regarding process compliance to approved plans,
procedures, standards, and analyses
• Monitoring the products
– Focus on the quality of product within each phase of
the SDLC
• e.g., requirements, test plan, architecture, etc.
– Objective: identify and remove defects throughout
the lifecycle, as early as possible
Software Quality Assurance
• Software quality assurance [IEEE]:
– A planned and systematic pattern of all actions necessary to provide
adequate confidence that an item or product conforms to established
technical requirements.
– A set of activities designed to evaluate the process by which the
products are developed or manufactured.
• Software quality assurance is [Galin2004] :
– A systematic, planned set of actions necessary to provide
adequate confidence that the software development process or
the maintenance process of a software system product
conforms to established functional technical requirements as
well as with the managerial requirements of keeping the
schedule and operating within the budgetary confines.
• Software Quality assurance consists of the auditing and reporting
functions of management [Pressman2004]
91
Software Quality Assurance
• Conformance to software requirements is the
foundation from which software quality is measured.
• Specified standards are used to define the
development criteria that are used to guide the
manner in which software is engineered.
• Software must conform to implicit requirements
(ease of use, maintainability, reliability, etc.) as well
as its explicit requirements.
92
Quality of Software developed in-house & COTS
components
• SQA processes apply when integrating
purchased or customer-supplied software
products into the developed product
• Question. How do you determine the “quality”
of COTS components?
– Current research problem
Process Assessment
• Use of standards and process models has a positive
impact on the quality of the software product
– Disciplined, controlled development process
• Examples include:
– ISO 9001
– CMM
• CMU SEI, 5 levels
– SPICE
• Developing a standard for software process assessment
• ISO joint committee, Europe, Australia
– IEEE 1074, IEEE 12207, …
Product Assessment
• Reviews, inspections, walkthroughs

– Specialized techniques available:
• How to review/assess requirements,
architecture, detailed designs, code
• …
• Testing
• Simulation
• Prototyping
• Formal verification
– Model checking, theorem proving
Product Assessment
• Reviews, inspections, walkthroughs of

Plans, reports, models, standards
– Project management, quality assurance,
training, test plan(s)
– Requirements, analysis, architecture,
detailed design model, test cases
– Issue or problem reports
– Metric reports
– Traceability reports
– Documentation, coding standards
–…
Summary
• Five views of quality
– Transcendental view
– User view
– Manufacturing view
– Product view
– Value-based view
• Measure quality for three reasons
– Baseline
– Quality improvement
– Know the present level for future planning
• Measuring the user’s & manufacturer’s view
• McCall’s quality factors
– Quality factors (11): 3 classes (operation, revision, transition)
– Quality criteria (23)
• ISO 9126
– Six quality characteristics
– Twenty quality subcharacteristics
97
Session 2 Quiz
1. Ture or False?
Quality is absolute.
Quality is one-dimensional
Quality is subject to constraints
Quality is about acceptable compromises
Quality criteria are independent
2. TQC is an acronym for?
3. What are five views of software quality?
4. A quality factor represents _____________ characteristic of a system.
5. A quality criterion is an _____________ of a quality factor that is related to software development.
6. What are reasons for developing a quantitative view of quality?
7. Manufacturers are interested in _____________ and _____________ .
8. McCall, et al. define _____________ high-level quality factors, whereas there are only _____________ in the ISO 9126 document.
9 . McCall, et al. define _____________ quality criteria, whereas there are _____________ quality subcharacteristics in the ISO 9126
document.
Session 2 Quiz Answers
1. Ture or False?
Quality is absolute. F
Quality is one-dimensional F
Quality is subject to constraints
Quality is about acceptable compromises
Quality criteria are independent F
2. TQC is an acronym for? Total Quality Control
3. What are five views of software quality? Transcendental, User, Manufacturing, Product, Value-based
4. A quality factor represents _behavioral____ characteristic of a system.
5. A quality criterion is an ___attribute__________ of a quality factor that is related to software development.
6. What are reasons for developing a quantitative view of quality? baselines for qualities, process improvement, improvements
investigation
7. Manufacturers are interested in ___defect count_____ and _rework cost_____ .
8. McCall, et al. define ___11__________ high-level quality factors, whereas there are only _____6________ in the ISO 9126 document.
9 . McCall, et al. define _____23________ quality criteria, whereas there are ____20_________ quality subcharacteristics in the ISO
9126 document.
SESSION 3: ISO 9000
Overview
1. ISO 9000 Family of Standards
2. Twenty Quality Elements in ISO 9000
3. Characteristics of an ISO 9000 Quality System
4. ISO 9000-3
5. ISO 9000:2000
6. Tick It
Standards
• ISO 9000 family of standards
– Ensures you have a documented quality
system, but does not check the quality of
your products
– "Tick It" is specifically designed for IT
companies wishing to register under the
ISO 9000 series
ISO 9000 Family of Standards
• A series of international quality standards
developed by the International Organization for
Standardization
• Originally developed for two-party contractual
situations, mainly for the manufacturing
environment
• Applies to the quality management system and
the process used to produce a product
• Ensures that the process can consistently
produce products that meet the expectation of
the customers
(cont’d)
• Provides a framework for improving business
processes
• Does NOT provide for leading-edge quality, but
does provide a strong quality foundation upon
which a company can build
• Provide a generic model of the quality process;
must be instantiated for each organization
• Describe what, at the minimum, must be done;
does NOT specify how things are to be done
ISO 9000 and Quality Management
ISO9000
quality models
is instantiated as
Organization
Organization quality process
Quality manuals
For assessment
Is used to develop
Project 1 Project 2 Project 3 Project quality
Quality plan
Quality plan Quality plan management
supports
Guidelines for selection and use of
the ISO 9000 standards
A standard
for manu-
ISO9000-1 facturing
Standards used for

certification ISO9001 ISO9002 ISO9003
A guideline
for ISO 9001
for software
Guidelines to
standards ISO9004 ISO9004-2 ISO9000-3
(cont’d)
• ISO 9000-1 is a general guideline which
gives background information about the
family of standards
• ISO 9001, ISO 9002, and ISO 9003 are
standards in the family, containing
requirements on a supplier
(cont’d)
• ISO 9002 and ISO 9003 are subsets of
ISO 9001
– ISO 9002 applies when there is no design
– ISO 9003 applies when there is neither
design nor production
(cont’d)
• ISO 9004 is a comprehensive guideline to the
use of the ISO 9000 standards
• For software development, ISO 9001 is the
standard to use
• ISO 9000-3 is a guideline on how to use ISO
9001 for software development
• ISO 9004-2 is a guideline for the application
of ISO 9001 to the supply of services
(including computer centers and other
suppliers of data services)
Relationship of ISO 9000 standards
ISO9001
Design and Servicing
ISO9002
Production and
Installation
ISO9003
Final Inspection
and Testing
Overview of ISO 9001
• The first version of ISO 9001 was published
in 1987
• Versions of ISO standards are defined by the
year of publications (e.g. ISO 9001:1994)
• Since software production is largely a
question of design, ISO 9001 is the standard
to use
• Its title is “Quality systems – Model for quality
assurance in design, development,
production, installation, and servicing”
Overview of ISO 9001 (cont’d)
• ISO 9001 focuses on management
instead of products
• Two basic requirements of ISO 9001
– All operations influencing quality shall be
under control
– This control shall be visible (i.e. it requires
that plans, procedures, and organization
be documented, and important activities be
recorded)
Overview of ISO 9001 (cont’d)
• ISO 9001 expects a fairly strict organization,
where managers have the responsibility and
authority to control the work of their
subordinates (hence, self-organizing groups
are difficult to fit into ISO 9001)
• Because ISO 9001 is written for the
manufacturing industry, some interpretation is
required to apply it to software development
Software Development vs
Manufacturing
Manufacturing Customer Product
Production
Inspection
Install
Maintenance
Process requirements development & test & service
Design ISO 9003
ISO 9002
ISO 9001
Design Implementation
Software High- Low-

Customer Package Maintenance
Development requirements
level level Code Test
& install & service
Process design design
Application of ISO 9001 Standard to the Manufacturing and

Development Processes
Three Levels of Quality Assurance
• ISO 9001 Quality systems – Model for quality
assurance in design/development,
production, installation, and servicing
– If the software development organization designs
the product it develops, then ISO 9001 will apply
• ISO 9002 Quality systems – Model for quality
assurance in production and installation
– If the software development organization
implements products from a design that is
provided to it, then ISO 9002 will apply
Three Levels of Quality Assurance
(cont’d)
• ISO 9003 Quality systems – Model for
quality assurance in final inspection and
test
– If the organization is a test organization,
then ISO 9003 will apply
• Because ISO 9001 covers more
aspects of development, more elements
of the standard apply to ISO 9001 than
to ISO 9002 and ISO 9003
Manufacturing Industry vs
Software Industry
Manufacturing Software
Design Production Functionality

Manufacturing Industry Vs Software
Industry (cont’d)
• Manufacturing
– Design is a relatively minor activity (e.g. ball pens)
– Production cost for each manufactured item is
notable
• Software development
– Nearly 100% design
– Production cost for each copy of the software is
insignificant
– The functionality of software is orders of
magnitude greater than most manufactured items
Overview
4. ISO 9000-3
5. ISO 9000:2000
6. Tick It
Twenty Quality Elements in ISO 9000
• 1. Management responsibility
– You must clearly define the general responsibilities
of a company’s management, in terms of: (i)
quality policy, (ii) organization, and (iii)
management review
• 2. Quality system
– You must establish, document, implement, and
maintain a quality system that conforms with ISO
9000
(cont’d)
• 3. Contract review
– You must have procedures for ensuring that what
is expected from you is adequately defined and
documented and that you have the capability to
satisfy the requirements
• 4. Design control
– You must have procedures for controlling and
verifying the design output to ensure that specified
requirements will be met
(cont’d)
• 5. Document control
– You must have defined procedures to control all
documents, including review, approval, and
change, and to ensure that the right level of
information is available to the right people at the
right time
– You must also maintain a master list of current
documents
(cont’d)
• 6. Purchasing
– You must ensure that parts, obtained from
elsewhere, used in the product or in the
production of the product, meet their specified
requirements
• 7. Customer-supplied products
– You must have procedures for verification, safe
storage, and maintenance of products, or parts,
provided by the customer to be included in the
product
(cont’d)
• 8. Product identification and traceability
– Where appropriate, you must have procedures for
identifying and tracing the product during all
stages of production, delivery, and installation
• 9. Process control
– You must carry out production under controlled
conditions, including monitoring progress,
approval of processes and equipment, etc.
(cont’d)
• 10. Inspection and testing
– You must have procedures for all levels of
inspection and testing that you have identified as
being required
– You are also required to maintain records of test
activity
• 11. Inspection, measuring, and test
equipment
– You must control, calibrate, and maintain
inspection, measuring, and test equipment
(cont’d)
• 12. Inspection and test status
– You must be able to identify the test status of the
product throughout the process
• 13. Control of nonconforming products
– You must have procedures for controlling a
product that does not conform to its specified
requirements
(cont’d)
• 14. Corrective action
– You must have procedures for investigating the
causes for nonconforming products and ensuring
corrective actions to prevent recurrences
• 15. Handling, storage, packaging, and
delivery
– You must have a good system for storing and
controlling the various parts that will compose your
product during product development and through
product delivery
(cont’d)
• 16. Quality records
– You must identify and keep records to
demonstrate achievement of product quality and
effective operation of your quality system
• 17. Internal quality audits
– You must plan and carry out internal quality audits,
by qualified individuals, to verify you are doing
what you say you are doing and to determine the
effectiveness of your quality system
(cont’d)
• 18. Training
– You must identify the training needs of your
people, provide the required training, and keep
records of the training
• 19. Servicing
– You must have procedures for servicing your
product when this requirement is specified in the
contract
• 20. Statistical techniques
– You must show that any statistical techniques that
you use are correct
Overview
4. ISO 9000-3
5. ISO 9000:2000
6. Tick It
Characteristics of an ISO 9000
Quality System
• Quality objectives
– The company should have a quality policy that
states its quality goals and objectives and the
strategy it will use to achieve them
• Commitment, involvement, and attitude
– All employees and managers must be committed
to the quality objectives and involved in achieving
the objectives
Quality System (cont’d)
• Controlled
– Every aspect of what is done during the
development process must controlled
• Effective
– It is the means by which you measure whether
your quality system is really working for you
• Auditable
– ISO 9000 requires that systematic internal audits
of your quality system be conducted
Quality System (cont’d)
• Documented quality system
– Your quality system, including your processes and
procedures, should be documented to the extent
that, if you had to replace all of your employees,
you could do it and still continue your business
• Continual improvement
– ISO 9000 requires that your quality system be
continually monitored and reviewed for
weaknesses and that improvements be identified
and implemented
Satisfying ISO 9000
• Quality policy
– You must have a quality policy in written form
• Quality manager
– You must assign a management representative,
reporting at a high level, to be responsible for your
quality system and for assuring ISO 9000
conformance
• Quality manual
– ISO 9000 requires that your quality system be
documented
Satisfying ISO 9000 (cont’d)
• Documented processes and procedures
– You should document all procedures that would be
needed to continue your operation if all of your
people were replaced
• Project plan
– For software development, this means planning
the steps and activities that will be performed in
transforming the product requirements into a final
product
• Build plan
– It should specify what parts have to come together
to create the total product, in what order, when,
and it should specify their interdependencies
• Test plan
– Every project should have a test plan that is
established at the beginning of the project and
updated as the project progresses
• Service plan
– Every product should have a service plan stating
the planned maintenance activities that will be
performed after the product is delivered and who
will perform the activities
• Quality records
– Quality records are kept so that you can show that
you have done what you said you were going to
do
• Training records
– ISO 9000 requires that you are able to show that
you assign qualified people to various tasks and
that you identify and provide required training to
your employees
• Internal quality system audits
– Periodic planned internal audits of your quality
system should be conducted by qualified
personnel for the purpose of determining the
effectiveness of your quality system and ensuring
that planned activities and procedures are being
followed
• Library control system
– ISO 9000 requires proper and safe storage of the
parts being developed
– The library control system should also be used to
store and control project and quality system
documentation, including documented processes
and procedures
Essentials Vs Standards Elements
ISO 9000 Standards Elements
Essentials to conformance 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Quality objectives X X
Commitment, involvement, and
attitude X X
Controlled X X X X X X X X X
Effective X X X
Auditable X X X X X X X X X X
Documented quality system X X
Continual improvement X X
Quality policy X
Quality manager X X
Quality manual X X
Essentials Vs Standards Elements
(cont’d)
ISO 9000 Standards Elements
Essentials to conformance 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Documented procedures & processes X X X X X X X X X X X X X X X X
Project plan X X
Build plan X X X X X
Test plan X X X X X X
Service plan X X X X
Quality records X X X X X X X X X X X X X X X X
Training records X X X X X
Internal quality system audits X X X
Library control system X X X X X
Overview
4. ISO 9000-3
5. ISO 9000:2000
6. Tick It
ISO9000 Conforming Quality System for
Software Development
Quality System
Quality
Must be: Support items manual Project items
•Quality policy & •Requirements Must be controlled
•Documented objectives •Project plan Should demonstrated:
•Effective •Processes •Design output
•Control
•Controlled •Procedures •Test plan
•Internal quality system •Service plan •Effectiveness
•Continually Procedure
improved audits •Quality records •Auditability
•Library control system handbook •Build plan
Need to be: Personnel

•Committed •Employees Product items
Library Must be
•Management •Internally
central •Involved developed parts
•Quality manager •Controlled
system •Aware •Product
•Purchaser •Identifiable
•Responsible •Subcontractors documentation
•Included software •Traceable
•Subcontracted parts •Verified/validated
Introduction of ISO 9000-3
• ISO 9001 is generic and many IT
people find it difficult to interpret and
apply
• ISO 9000-3 is a set of guidelines that
helps interpret and apply ISO 9001 for
software development
• Since it is NOT a standard, companies
are still assessed against ISO 9001
Assumptions of ISO 9000-3
• Each development project is associated
with a life cycle with phases
• The software product produced is the
result of a contractual agreement
between a purchaser and a supplier
Overview of ISO 9000-3
• It consists of 22 clauses that do not
correspond directly with the 20 clauses of ISO
9001
• These 22 clauses are grouped into three
major sections:
– Section 4: Quality system – Framework
– Section 5: Quality system – Life cycle activities
– Section 6: Quality system – Supporting activities
Cross-reference ISO9000-3 to
ISO9001
Clause in ISO 9000-3 Clause in ISO 9001
4.1 Management responsibility 4.1
4.2 Quality system 4.2
4.3 Internal quality system audits 4.17
4.4 Corrective action 4.14

ISO9001 (cont’d)
5.2 Contract review 4.3
5.3 Purchaser’s requirements specification 4.3, 4.4
5.4 Development planning 4.4
5.5 Quality planning 4.2, 4.4
5.6 Design and implementation 4.4, 4.9, 4.13
5.7 Testing and validation 4.4, 4.10, 4.11, 4.13
5.8 Acceptance 4.10, 4.15
5.9 Replication, delivery, and installation 4.10, 4.13, 4.15
5.10 Maintenance 4.13, 4.19
ISO9001 (cont’d)
6.1 Configuration management 4.4, 4.5, 4.8, 4.12, 4.13
6.2 Document control 4.5
6.3 Quality records 4.16
6.4 Measurement 4.20
6.5 Rules, practices, and conventions 4.9, 4.11
6.6 Tools and techniques 4.9, 4.11
6.7 Purchasing 4.6
6.8 Included software product 4.7
6.9 Training 4.18
Why Comply with ISO 9001?
• Provide a foundation for a quality system
which is needed for quality software
• Increase productivity and reduce costs
because development is done right the first
time under control
• Ensure consistency of software quality
• Stay competitive by keeping up with market
standards
• Fulfil software contractual requirements
• Improve corporate image
Potential Problems of ISO 9001
• Creating rules and formality to fulfill ISO
9001:
– Too many rules result in bureaucracy
– Too few rules result in insufficient control
over quality Quality
productivity
Formality, paperwork
Overview
4. ISO 9000-3
5. ISO 9000:2000
6. Tick It
ISO 9000:2000 Software Quality Standard
• The international organization ISO has developed a series
of standards for quality assurance and quality management,
collectively known as the ISO 9000.
• The ISO 9000 standards are reviewed and updated once
every 5-8 years. The standards released in the year 2000 are
known as ISO 9000:2000.
• There are three components of the ISO 9000:2000 standard.
– ISO 9000: Fundamentals and vocabulary
– ISO 9001: Requirements
– ISO 9004: Guidelines for performance improvements
• Note: ISO 9002 and ISO 9003 were parts of ISO

9000:1994, but these are no more parts of ISO 9000:2000.
153
ISO 9000:2000 Fundamentals:
• This is based on eight principles.
– Principle 1: Customer focus
– Principle 2: Leadership
– Principle 3: Involvement of people
– Principle 4: Process approach
– Principle 5: System approach to management
– Principle 6: Continual improvement
– Principle 7: Factual approach to decision making
– Principle 8: Mutually beneficial supplier relationships
154
ISO 9001:2000 Requirements
• The five major parts of this document are as
follows.
– Part 4. Systemic requirements
– Part 5. Management requirements
– Part 6. Resource requirements
– Part 7. Realization requirements
– Part 8. Remedial requirements
155
• Part 4. Systemic requirements (partial)
– Document the organizational policies and goals. Publish a vision
document.
– Document all quality processes and their interrelationship.
– Implement a mechanism to approve documents before those are
distributed.
• Part 5: Management requirements (partial)
– Generate an awareness for quality to meet a variety of
requirements, such as customer, regulatory, and statutory.
– Focus on customers by identifying and meeting their
requirements in order to satisfy them.
– Develop a quality policy to meet the customer’s needs.
– Clearly define individual responsibilities and authorities
concerning the implementation of quality policies.
156
• Part 6. Resource requirements (partial)
– Identify and provide resources required to support the
organizational quality policy in order to realize the quality
objectives.
– Allocate quality personnel resource to projects.
– Put in place a mechanism to enhance the quality level of
personnel.
• Part 7: Realization requirements (partial)
– Develop a plan to realize a product from its requirements.
– Interact with customers to gather their requirements. Classify
those requirements.
– Review the requirements.
– Follow a defined purchasing process by evaluating potential
suppliers based on a number of factors, such as ability to meet
requirements and price.
157
• Part 8. Remedial requirements (partial)
– Measure and track the customer’s
satisfaction level.
– Perform internal audit.
• Example: Find out whether or not
personnel with adequate education,
experience, and skill have been assigned
to a project.
– Monitor processes by using a set of key
performance indicators.
158
Overview
4. ISO 9000-3
5. ISO 9000:2000
6. Tick It
TickIT Initiative
• A system for certifying software
development organizations to ISO 9001
• Led by the TickIT project office of the
UK Department of Trade and Industry,
and supported by the British Computer
Society
TickIT Initiative (cont’d)
• Objectives of TickIT:
– To ensure that the ISO 9000 series of
standards is applied appropriately to
software
– To ensure consistency of certification within
the IT industry
– To enable mutual recognition of registration
across the IT industry
• TickIT scheme requires auditors to use the
TickIT Guide (which is based on ISO 9000-3)
• The TickIT Guide tends to suggest more of
how to implement an ISO 9000 conforming
quality system than do the standards
• Under the TickIT scheme, auditors are
required to pass a rigid set of criteria to
become TickIT accredited
• TickIT auditors use ISO 9000-3 as a
guide to check the quality system
implemented in an organization
• If any discrepancy between the quality
system and ISO 9000-3 is found, then
these auditors will require explanations
as to how the standards are being
satisfied
Summary
• ISO 9000 provides an internationally mandated attempt to
define and provide for (software) product quality in the
customer-supplier relationship
• Three important things about ISO 9000:
– It is a tool for buyers as well as builders
– It is about what, not how
– It provides necessary, but not sufficient, direction
• ISO 9000:2000
– ISO 9000: Fundamentals
– ISO 9001: Requirements
– ISO 9004: Performance improvements
• TICKIT
Session 3 Quiz
1. Ture or False?
ISO 9000 family of standards does not check the quality of your products.
"Tick It" is specifically designed for IT companies wishing to register under the ISO 9000
series
ISO 9000 family of standards describes what, at the minimum, must be done; does NOT
specify how things are to be done
ISO 9000-1 is a general guideline
ISO 9001, ISO 9002, and ISO 9003 are standards in the family, containing requirements
on a supplier
ISO 9000-3 is a guideline on how to use ISO 9001 for software development
Self-organizing groups are difficult to fit into ISO 9001
ISO 9000:2000 is based on eight principles.
2. For software development, ____________ is the standard to use.

3. There are _____________ Quality Elements in ISO 9000.
SOFTWARE
SESSION 4: QUALITY IN SOFTWARE PROCESSES

Overview
1. Software Development Life-Cycle Models
2. Integration of software quality assurance into the life cycle
Software Development Life-Cycle
Models
• Software Development Life-Cycle Models
(SDLC) are:
– used to describe the process of developing
software
– sequences of activities
– Iterative SDLCs repetitively enter the same
phases
– objective is to incorporate best practises
– allow to control risks.
Software Development Life-Cycle
Models
• Mega-process: initiation, development,
maintenance, termination.
• Development process components: requirement,
specification, design, coding, testing, release.
• Process variations:
– waterfall development process
– iterative development process
– spiral development process
– lightweight/agile development processes and XP
(extreme programming)
– maintenance process too
– mixed/synthesized/customized processes
• QA important in all processes
requirements Software development life
gathering
cycle (ISO 12207)
specification
design
code
testing
implement
Waterfall Model
Requirements
Analysis
System
Design
Coding
Testing
Maintenance
Waterfall Model (cont’d)
• The Waterfall Model:
– The “traditional” SDLC
– developed/described by Royce in 1970.
– model is sequential
– phases are signed off, eg. once the
requirements phase is completed, the
process moves on to the Design phase.
– relies heavily on upfront planning
• classical
• one-shot approach
• effective control
• limited scope of iteration
• long cycle time
• not suitable for system of high
uncertainty
• QA throughout process
– defect prevention in early phases
– focused defect removal in testing phase
– defect containment in late phases
– phase transitions: inspection/review/etc.
V Model Maintenance
Requirements
User Acceptance Testing
Analysis
System Design System Testing
Unit and
Program Design
Integration Testing
Coding
V Model (cont’d)
• is an extension to the Waterfall model
• still a phased model
• includes Quality Assurance Techniques
• emphasis is on Validation Activities
• test-execution is delayed until the product is
developed
• test-plans and test-cases are designed in
concurrence
• Acceptance Tests can already be defined
based on the requirements.
Spiral Model (adapted from Boehm 1987)
Cumulative cost
Progress through steps
Determine objectives, Evaluate alternatives;
alternatives and Risk
identify and resolve
constraints analysis risks
Risk
analysis
Risk
analysis Prototype
Prototype
Prototype
Requirements Concept of Software System Detailed
plan operation requirements product design
design
Development Requirements
plan validation Coding
Integration and
Test plan Design
validation Unit
testing Develop and verify
Plan next phases Integration next-level product
Acceptance and Test
test
Spiral Model (cont’d)
The Spiral Model (Boehm 1988) is
• iterative development approach
• emphasises the involvement of stakeholders
• phases are similar to those in the waterfall model
• strong emphasis on the management of risk
• each iteration
– 1 identifies objectives
– 2 identifies alternatives
– 3 identifies constraints
– 4 followed by a risk-driven evaluation.
– 5 development and verification commences,
– 6 resulting in an increment of functionality.
– 7 plan for the next iteration of development.
• The spiral starts in the centre!
• Four major activities
– Planning
– Risk analysis
– Engineering
– Evaluation
Prototyping Model
YES
User
Build prototype
satisfaction
NO
User feedback
Prototyping Sequences
• Requirements gathering
• Quick design
• Prototype construction
• Customer evaluation
• Refinement
• Loop back to quick design for fine tuning
• Product engineering
Prototyping Model
• Goals
– meet (some) user requirements at an early
stage
– reduce risk and uncertainty
– verify a design or implementation approach
• Should always answer specific
questions; goals must be identified
Classification of Prototype
• Throw-away
– After users agree the requirements of the
system, the prototype will be discarded.
• Evolutionary
– Modifications are based on the existing
prototype.
• Incremental
– Functions will be arranged and built
accordingly.
Forms of Prototypes
• Mock-ups
• Simulated interaction
• Partial working model
Benefits of Prototyping
• Learning by doing
• Improved communication
• Improved user involvement
• Clarification of partially-known requirements
• Demonstration of the consistency and
completeness of a specification
• Reduced need for documentation
• Reduced maintenance costs
• Feature constraint
• Production of expected results for testing real
system
Drawbacks of Prototyping
• Users sometimes misunderstand the
role of the prototype
• Lack of project standards possible
• Lack of control
• Additional expense
• Machine efficiency
• Close proximity of developers
Phased Development
• Reduce cycle time
• Two parallel systems:
– operational system (Release n)
– development system (Release n+1)
• Two approaches
– incremental
– iterative
Incremental Model
• Break system into small components
• Implement and deliver small
components in sequence
• Every delivered component provides
extra functionality to user
Incremental Model (cont’d)
Arrange
Requirements Design and develop
requirements in
Analysis increment
increments
NO
Validate Integrate System

increment increment OK?
YES
Iterative Model
• Deliver full system in the beginning
• Enhance functionality in new releases
Iterative Model (cont’d)
Design system version n n = n+1

NO
Develop system version Validate system System

n version n complete
YES
Combined Incremental and Iterative
Model
• Every new release includes
– extra functionality
– enhancement of existing functionality
• Popularly used in software industry
Ranking the Increments
• Rank by value to cost ratio
• V = value to customer (score 1 -10)
• C = cost (score 1- 10)
• Value to cost ratio = V/C
Advantages of Phased Development
• Early feedback
• Less possible requirement changes
• Early benefits for users
• Improved cash flow
• Easier to control and manage
• Capture early market
• Facilitate early training
• Can be temporarily abandoned
• Increase job satisfaction
Disadvantages of Phased
Development
• ‘Software breakage’
• Reduced productivity
Agile Systems Development
Ancestry in Rapid Application Development
Held up as antithesis of traditional software development.
Focus on:
• Early delivery priority business requirements.
• Dealing with partial knowledge
• Reduced documentation
• Small groups of programmers
• Iteration
• Continuous testing
• Integral customer involvement
Agile Software Development
Approaches
• Crystal technologies
• Atern (formerly DSDM-Dynamic System
Development Method)
• Feature-driven development
• Scrum
• Extreme Programming (XP)
Core Values
• Individuals and interaction over
processes and tools
• Working together over comprehensive
documentation
• Customer collaboration over contract
negotiation
• Responding to change over following a
plan
Dynamic Systems Development
Method
• Attempt to develop industry-standard
RAD method
• Consortium first met in Jan 1994
• Founding members include
– IBM
– ICL
– LBMS
– British Airways
Dynamic Systems Development
Method
• The consortium’s mission: “ to develop
and continuously evolve a public domain
method for RAD”
• the published manual includes framework
for all project activities
• the method has 4 phases:-
– feasibility
– functional prototype iteration
– design prototype iteration
– implementation
Overview
1. Software Development Life-Cycle Models
2. Integration of software quality assurance into the life cycle
Integrate QA into Life Cycle Phases
• Phase entry and exit criteria - inputs

and outputs
• Quality Checkpoints
• Audits and reviews of products and
processes
• Timely management notification of
problems - Risk Management
Several Quality Assurance Activities
Activities (see quality control):
• Requirement specification review
– Approval of the requirement specification to proceed with the next phase
• Design inspection
– Detecting errors and derivations from standards
• Design review
– Approval of the design to proceed with the next phase
• Code inspection
– Detecting errors and derivations from standards
• Unit test
– Approval of the unit quality to proceed with the next phase
• Integration tests
– Approval of the integrated units to proceed with the system test
• Documentation review
– Approval of the documentation for operation
• System test
– Approval of the system for operation
• Operation phase detection
– Defects will be found during operation by the customer
QA and the Strategy Phase
• Develop the QA Plan and Procedures
• Create QA records
• Determine Metrics
• Review and Analyse Requirements
• Establish the Deliverable Review Process
• Project Standards and Procedures
– Shared components and their management
– Externally developed coding standards
– Internally developed standards and procedures
QA and the Analysis Phase
• Technical & QA Reviews and Audits
– Requirements Document
– Function Hierarchy/Process Flow Diagram
– Requirements Traceability Matrix (RTM)
– Logical Database Design
• Entity Relationship Diagram(s) (ERD)
• Data Dictionary
– Create Read Update Delete (CRUD) Matrix
QA and the Design Phase
• Technical and QA Reviews and Audits
of all Deliverables
– Physical Database Design
– Module Network (Menu) Hierarchy
– Module Specifications
– Prototypes
• User Interface
• Scenarios/walkthrough
QA and the Build and Test Phase
• Configuration Management (CM)
– Version/build control through software
– Control software and documentation
• Peer Reviews and Code Walkthroughs
• Prototypes - customer demos
– Review form and follow-up
• Unit Test - formal
• Problem Tracking
• Integration Test - partially automated
– Business scenarios via QA/Director
– Load Runner for load testing
• System Test
– Customer/client involvement
– Acceptance test
• Integrated Project Teams (IPT)
• Independent Validation and Verification (IV&V)
QA and the Deployment Phase
• Phased implementation (no “Big Bang”)
– By function/subsystem
– By organization/user group
• QA reviews and test procedures continued
• Expedite test and delivery of modified
code - fast turnaround required
• User training - review and test of training
materials
QA and the Maintenance Phase
• Continue with established QA and CM
procedures
– Action Items/User meetings
– MoSCoW evaluation and followup
– Problem Reports - user accessible
Collect Project Metrics
• Areas of greatest problems/defects
• Number/results of QA audits and reviews
• Test coverage and test results
• Problem Reports/Defects found - e.g., per
module, per subsystems, classification and
type, time taken to resolve
• Development Time - Estimated vs. Actual
• - SEI CMMI Level 4 Quantitatively
Managed
Process Improvement
• Take existing process
• Analyze step-by-step
• Modify to improve
– e.g., testing/QA/CM process
– unit testing - formalized
• Training - e.g., COEs, Test Writing,
Testing
• - SEI CMMI Level 5 Optimizing
Summary
• We looked at the historical evolution of life-cycle models
• We have seen a move from upfront planning towards iterative models
• SDLCs became more iterate to address the uncertainty of software
projects
• agile” characterises a group of methodologies, all based on iterative
lifecycle models, emphasise on the “people-aspect”.
• Scheduled audits & reviews of all project processes and deliverables
• Maintenance of QA records of reviews and audits
• Management notification of non-compliance with standards and
procedures, or of notable problems
• Managing the Requirement Traceability process
• Peer Reviews, code walkthroughs
• Including QA personnel in project and customer meetings
• Providing training in standards, testing, or other QA-related topics
• Independent Testing
Session 4 Quiz
1. Ture or False?
Software Development Life-Cycle Models (SDLC) are sequences of activities
Software Development Life-Cycle Models (SDLC) allow to control risks
The V-Model is an extension to the Waterfall model
The V-Model is still a phased model
In the Spiral Model, The spiral starts in the centre
SDLCs became more iterate to address the uncertainty of software projects
In requirement specification review, people approve the requirement
specification to proceed with the next phase
SESSION 5: CMM
Overview
1. Process based Quality
2. What is Capability Maturity Model (CMM)
3. Structure of CMM
4. Use of CMM
Process and Product Quality
• The quality of a developed product is influenced by the quality of
the production process.
• This is important in software development as some product
quality attributes are hard to assess.
• However, there is a very complex and poorly understood
relationship between software processes and product quality.
Process-based Quality
• There is a straightforward link between process and product in
manufactured goods.
• More complex for software because:
– The application of individual skills and experience is particularly important in
software development;
– External factors such as the novelty of an application or the need for an
accelerated development schedule may impair product quality.
• Care must be taken not to impose inappropriate process
standards - these could reduce rather than improve the product
quality.
Practical Process Quality
• Define process standards such as how reviews should be
conducted, configuration management, etc.
• Monitor the development process to ensure that standards are
being followed.
• Report on the process to project management and software
procurer.
• Don’t use inappropriate practices simply because standards
have been established.
What is a process? …
• It can be seen as a method that can be used to focus the efforts
of a development team towards a desired result
• A process integrates:
– People (with Knowledge, Skills, Training, Motivation)
– Tools and Equipment
– Procedures and Methods defining the relationship of tasks
• A good process will provide clear guidance, is disciplined and
constantly refined based on experience
• A process provides a framework to work in:
– Most developers take pride in their work and want to deliver quality
output
– Wrong tools that do not fit into the process can end up as shelfware
– Procedures and Methods that do not support people will cause
resistance – not ideal
• A good process resolves these issues and is flexible
• Processes are not there just to be followed, they are supposed
to help deliver a better product (at lower cost)
How is a process used?
• Help ensure production of high-quality User Needs
software matching the needs of the
end-users
• Enhance team productivity Process
• Make purchase/hiring/management decisions
• Control schedule and budget Software
• Put software development best practices System
in action
• A well understood “Process Model” is
used to communicate the details visually.
An Example Process – RUP
Development Process
Consists of
produce
Phases Artifacts
monitor
has
end with
focus Iterations Release
Engineers
Use
Workflows
The Rational Unified Process Model

RUP Overview Diagram
Immature Organizations
• Immature Organization:
– A defined/documented process may not exist
– If Processes exist they are improvised (as
required), not rigorously followed
– Managers react to crises only (fire fighting)
– Ad-hoc project planning (poorly documented)
– Schedules/budgets are rarely met (poor
estimation)
– Product quality is difficult to predict or judge
– Difficult to maintain the products in the long term
– Has a high turn-over of employees
Mature Organizations
• Mature Organization:
– Well-defined and well-followed processes that are updated
when necessary (Process changes are formal)
– Well-defined roles and responsibilities (Reduces confusion)
– Product and process quality are monitored
– Schedules are realistic (refined estimation process)
– Participants understand value of the process (Staff are fully
trained in the company process, expectations)
– The deliverables from these organizations take longer, but
the output is stable and predictable
– Long term costs are low
Process Management – A Premise
• “Improvements in process will improve quality”
– This has been proven to work when the process is tuned to
work with the people, tools, domain.
– The process has to be defined within the context of available
resources [people and money] as well as the deadline.
• Total Quality Management principles have been
shown to provide great benefits in manufacturing and
service industries
• Software is different, but some principles have been
shown to work – CMM was built on top of these
Overview
3. Structure of CMM
4. Use of CMM
CMM History
• In the 1980s, realization about the inability to
manage the software process
– Projects late, over budget, or plain failures
• 1986-1987: Software Engineering Institute
(SEI)
– Began developing a process maturity framework
• 1991: CMM-SW 1.0
• 1993: CMM-SW 1.1
What is CMM?…
• Capability Maturity Model (CMM) is a framework that describes the
key elements of an effective software process.
• A road-map for process improvement. It describes an evolutionary
improvement path from an ad hoc, immature process to a mature,
disciplined process.
• Defines the attributes of an organization with a good software
development process and a culture of continuous process
improvement.
• Expresses the maturity of a software process in terms of five levels.
(Staged Representation)
• A scale for measuring the current capabilities of the organization
and guidance in prioritizing the areas for improvement.
• Covers practices for:
– Planning
– Engineering
– Managing software development and maintenance.
• When followed, these key practices improve the ability of
organizations to meet goals for cost, schedule, functionality, and
product quality.
Definitions from the CMM
Specification
• We shall look at the definitions of:
– Capability Maturity Model (CMM)
– Software process
– Software process capability
– Software process performance
– Software process maturity
• All definitions are quoted from the SEI
CMM v1.1 Specifications.
CMM - Definition
• A description of the stages through which
software organizations evolve as they define,
implement, measure, control, and improve
their software processes
• A guide for selecting process improvement
strategies by facilitating:
– determination of current process capabilities
– identification of the issues most critical to software
quality and process improvement
CMM Assessment: Principles
• Quantitative management methods increases the
organization's capability to control the quality and
improve the productivity.
• Application of the five-level capability maturity model
that enables to evaluate the achievements and
determine the efforts needed to reach the next
capability.
• Generic process areas that define the “what” — not
“how” enables the model's applicability to a wide
range of implementation organizations:
– It allows use of any life cycle model.
– It allows use of any design methodology, development tool
and programming language.
– It does not specify any particular documentation standard.
Software Process
• A software process can be defined as a set of
activities, methods, practices, and transformations
that people use to develop and maintain software and
the associated products
– E.g., project plans, design documents, code, test cases, and
user manuals.
• As an organization matures, the software process
becomes better defined and more consistently
implemented throughout the organization.
Software Process Capability
• Software process capability describes the
range of expected results that can be
achieved by following a software process.
• The software process capability of an
organization provides one means of
predicting the most likely outcomes to be
expected from the next software project the
organization undertakes.
Software Process Performance
• Software process performance
represents the actual results achieved
by following a software process.
• Software process performance focuses
on the results achieved, while software
process capability focuses on results
expected.
Software Process Maturity
• Software process maturity is the extent to
which a specific process is explicitly defined,
managed, measured, controlled, and
effective.
• Maturity implies a potential for growth in
capability and indicates both the richness of
an organization's software process and the
consistency with which it is applied in projects
throughout the organization.
Overview
3. Structure of CMM
4. Use of CMM
Structure of CMM
• The CMM is composed of five maturity levels.
• Each maturity level is composed of several
key process areas (except Level 1).
• Each key process area is organized into five
sections called common features.
• The common features specify the key
practices that, when collectively addressed,
accomplish the goals of the key process area.
Structure of CMM
Goals
Maturity Level
indicate
contain achieve
Key PA Infrastructure/
Process Activities
organized by
Capability
Common
Features describe
address
Implementation contain
Key
Practices
Maturity Level
Level 1: Initial
Level 2: Repeatable
Level 3: Defined
Level 4: Managed
Level 5: Optimising
Key Process Area
• Each maturity level is composed of key process
areas.
• Each key process area identifies a cluster of related
activities that, when performed collectively, achieve a
set of goals considered important for establishing
process capability at that maturity level.
• The key process areas have been defined to reside
at a single maturity level.
• For example, one of the key process areas for Level
2 is Software Project Planning.
Level 1 - Key Process Areas
• None that can be observed
• Software configuration management
• Software quality assurance
• Software subcontract management
• Software project tracking and oversight
• Software project planning
• Requirements management
• Peer reviews
• Inter-group coordination
• Software product engineering
• Integrated software management
• Training program
• Organization process definition
• Organization process focus
• Quality management
• Process measurement
• Software quality management
• Quantitative process management
• Process change management
• Technology change management
• Defect prevention
Goals
• The goals:
– Summarize the key practices of a key process area
– Can be used to determine whether an organization or
project has effectively implemented the key process
area
– Signify the scope, boundaries, and intent of each key
process area
• E.g. : a goal from the Software Project Planning
key process area :
– "Software estimates are documented for use in
planning and tracking the software project."
Common Features
• The common features are attributes that indicate
whether the implementation and institutionalization of
a key process area is effective, repeatable, and
lasting.
• The key practices are divided among five Common
Features sections:
– Activities Performed (Describes Implementation Activities)
– Commitment to Perform (Organizational culture)
– Ability to Perform (Institutionalization factor)
– Measurement and Analysis (Organization culture)
– Verifying Implementation (Institutionalization factor).
Key Practices
• Each key process area is described in terms of key
practices that, when implemented, help to satisfy the
goals of that key process area.
• The key practices describe the infrastructure and
activities that contribute most to the effective
implementation and institutionalization of the key
process area.
• For example, one of the practices from the Software
Project Planning key process area is "The project's
software development plan is developed according to
a documented procedure."
Overview
3. Structure of CMM
4. Use of CMM
Uses of the CMMI
• Within an organization:
– assess the current capabilities of the software process,
– identify strengths and weaknesses,
– a first step in a software process improvement programme,
– use as Appraisal for Internal Process Improvement.
• By an external agency:
– assess the software process of the organization, perhaps on
behalf of a customer (e.g. defence procurement),
– Software Capability Evaluation (SCE).
1342 organizations [SEMA 2003]
Time Required to Progress
to the Next CMM Level
Project resources distribution
by CMM capability level
ISO 9001 Vs CMM
• Almost all concerns raised by ISO 9001 are
encompassed by CMM.
• ISO 9001 describes the minimum criteria for
adequate quality management systems - rather than
process improvement. CMM address process
improvement as well as provides a clear path to
achieving it.
• CMM provides more detailed guidance and greater
breadth provided to software organizations.
• Building competitive advantage should be focused on
improvement, not on achieving a score (which is the
primary focus of ISO 9001).
ISO 9001 Certification Vs CMM
Levels
• An ISO 9001-compliant organization would
not necessarily satisfy all of the level 2 key
process areas, it would satisfy most of the
level 2 goals and many of the level 3 goals
• It is possible (in theory) for a level 1
organization to receive ISO 9001 registration
• A level 3 organization would have little
difficulty in obtaining ISO 9001 certification
• A level 2 organization would have significant
advantages in obtaining certification.
Summary
• The quality of a developed product is influenced by the quality of
the production process. Improvements in process will improve
quality
• Quantitative management methods increases the organization's
capability to control the quality and improve the productivity.
• Software process performance focuses on the results achieved,
while software process capability focuses on results expected.
• CMM expresses the maturity of a software process in terms of
five levels
• Each maturity level is composed of several key process areas
(except Level 1). Each key process area is organized into five
sections called common features. The common features specify
the key practices that accomplish the goals of the key process
area.
• CMM address process improvement as well as provides a clear
path to achieving it.
Session 5 Quiz
1. Ture or False?
The quality of a developed product is influenced by the quality of the production
process.
Improvements in process will improve quality
Quantitative management methods increases the organization's capability to control the
quality and improve the productivity
CMM expresses the maturity of a software process in terms of five levels
Software process performance focuses on the results achieved, while software process
capability focuses on results expected.
Almost all concerns raised by ISO 9001 are encompassed by CMM.
CMM provides more detailed guidance and greater breadth provided to software
organizations.
CMM address process improvement as well as provides a clear path to achieving it.
2. ____________ describes the minimum criteria for adequate quality management systems -
rather than process improvement. ____________ address process improvement as well as
provides a clear path to achieving it.
SOFTWARE
SESSION 6: SOFTWARE PROJECT MANAGEMENT

Overview
1. What is project management?
2. Project plan and scheduling
Why is project management important?
• Money on ICT projects.
• Projects are not always successful.
• The reason for project shortcomings is
often the management of projects. – lack
of skills and proven approach to project
management and risk management.
What is software project
management?
• Understand what is meant by a project
• Understand what is meant by
management
What is a project?
• Dictionary: planned activity
• Key elements(routine ->uncertainty)
– non-routine
– planned
– specific objectives
– predetermined timespan
– for someone
– several specialisms
– several phases
– constrained resources
– large or complex
Nature of a Project
• Creates Change
• Has Composite Goals
– Human, Technical, Structural
• Is Unique
• Is limited in time and scope
• Involves a variety of resources
• May be large or complex
• May involve several phases.
Are software projects really different from
other projects?
Not really, but:
• invisibility
• Complexity
• conformity
• flexibility
all add to difficulties
Therefore…
• Greater chance of failure
• Greater Uncertainty
• Greater chance of being affected by

organisational change
• …
Common problems with software
projects
• Lack of quality standards and measures
• Lack of measurable milestones
• Difficult to make the progress visible
• Poor communications
• Poor documentation
• Frequent changes of requirements
• Over budget and late delivery of software
• …
What is Management?
Management may involve:
 planning — deciding what is to be done
 organising — making arrangement
 staffing — selecting the right people
 directing — giving instructions
 monitoring — checking on progress
 controlling — taking action to remedy hold-ups
 innovating — coming up with new solutions
 representing — liaising with users
Major issues of software project
management to be covered
• Software development models
• Software size and cost estimation
• Software project planning
• Software risk management
• Resource allocation
• Standards: ISO 9000 and CMM
• Performance tracking and reporting
• Software project configuration management
• Software project team management
What Management Practices
should be in place?
• Project Planning, Control and Tracking
Methods
• Software Quality Management
• Software Configuration Management
• Training
Major Processes in Developing a Software
Project
• Feasibility study
• Project planning
• Project execution
Project Activities
Is it worth doing?
How do we do it?
feasibility study
Do it!
planning
Project execution
 Feasibility study - decide if project is worth doing

 Plan how you are going do it,
 then do it
Feasibility Study
• Analyse the general requirements, costs
and the functionalities and services
provided by the system to be developed
• Aimed to determine whether a system
should be developed or not
• Can be viewed as a project itself
Contents of a feasibility study report
• Introduction: identifies what the document is;
• Description of current situation
• Problem description
• Proposed development
 business and financial aspects
 technical aspects
 organisational aspects
• Estimated costs
 development costs
 operational costs
• Envisaged benefits
• Recommendations
Overview
1. What is project management?
2. Project plan and scheduling
Project Plan
• Successful project are planned and
tracked.
• Traditional software development life-
cycles heavily rely on accurate plans
• More agile (iterative) approaches rely
on accurate short-term plans and
renegotiation of targets with the clients.
Project Planning Activities
• Identify project scope and objectives
• Identify project infrastructure
• Analyse project characteristics
• Identify project products and activities
• Estimate effort for each activity
• Identify activity risks
• Allocate resources
• Review and publicize plan
Activity based Scheduling
• Once we have a project plan (or, project
schedule), we need to schedule the
activities in a project taking into account
the resource constraints
Scheduling
• Once tasks (from the WBS) and size/effort
(from estimation) are known: then schedule
• Primary objectives
• Best time
• Least cost
• Least risk
• Secondary objectives
• Evaluation of schedule alternatives
• Effective use of resources
• Communications
Terminology
• Precedence:
• A task that must occur before another is said to have
precedence of the other
• Concurrence:
• Concurrent tasks are those that can occur at the same
time (in parallel)
• Leads & Lag Time
• Delays between activities
• Time required before or after a given task
Terminology
• Milestones
– Have a duration of zero
– Identify critical points in your schedule
– Shown as inverted triangle or a diamond
– Often used at “review” or “delivery” times
• Or at end or beginning of phases
• Ex: Software Requirements Review (SRR)
• Ex: User Sign-off
– Can be tied to contract terms
Terminology
Example
Milestones
Terminology
• Slack & Float
– Float & Slack: synonymous terms
– Free Slack
– Slack an activity has before it delays next task
– Total Slack
– Slack an activity has before delaying whole project
– Slack Time TS = TL – TE
• TE = earliest time an event can take place
• TL = latest date it can occur w/o extending project’s
completion date
Activity Float
• Time allowed for an activity to delay
• 3 different types:
– Total float (without affecting the completion of the
project)
= latest start date – earliest start date
– Free float (without affecting the next activity)
= earliest start date of next activity – latest end date
of previous activity
– Interfering float (= total float - free float)
Scheduling Techniques
• Simple sequencing
– Suitable for small projects
• Critical Path Method (CPM)
– Suitable for large software projects
– The most commonly used “networking”
technique
Simple sequencing
• A simple sequencing of the tasks and
the responsible personnel taken into
account of the resources
• Easily presented in a simple bar chart
• Suitable for allocating individuals to
particular tasks at an early stage
Scheduling Techniques
– Mathematical Analysis
• Network Diagrams
– PERT
– CPM
– GERT
– Bar Charts
• Milestone Chart
• Gantt Chart
Activity Network
4
B D
3
3
1 G
A 2
2 E
C
4
1 2 3 4 5 6 7 8 9 10
Bar chart - Technical Plan

Network Diagrams
• Developed in the 1950’s

• A graphical representation of the tasks
necessary to complete a project
• Visualizes the flow of tasks & relationships
Network Diagrams
• Two classic formats
– AOA: Activity on Arrow
– AON: Activity on Node
• Each task labeled with
• Identifier (usually a letter/code)
• Duration (in std. unit like days)
• There are other variations of labeling
• There is 1 start & 1 end event
• Time goes from left to right
Node Formats
The activities can be listed:-
Ref Activity Description Duration Dependencies
(days)
05 Produce stage plans 3
10 Produce requirements 15 05
20 Identify enhancements 5 10
30 Produce acceptance criteria 5 05
The activities can then be ordered on a PERT activity network:
30
05
10 20
Network Diagrams
• AOA consists of
• Circles representing Events
– Such as ‘start’ or ‘end’ of a given task
• Lines representing Tasks
– Thing being done ‘Build UI’
• a.k.a. Arrow Diagramming Method (ADM)
• AON
• Tasks on Nodes
– Nodes can be circles or rectangles (usually latter)
– Task information written on node
• Arrows are dependencies between tasks
• a.k.a. Precedence Diagramming Method (PDM)
Mathematical Analysis
• PERT
– Program Evaluation and Review Technique
• CPM
– Critical Path Method
• Sometimes treated synonymously
• All are models using network diagrams
MS-Project Example
Critical Path
• “The specific set of sequential tasks upon
which the project completion date depends”
– or “the longest full path”
• All projects have a Critical Path
• Accelerating non-critical tasks do not directly
shorten the schedule
Critical Path Method (CPM)
• Critical Path Method
– The process for determining and optimising the
critical path
• Non-CP tasks can start earlier or later w/o
impacting completion date
• Note: Critical Path may change to another as
you shorten the current
• Should be done in conjunction with the you &
the functional manager
Critical Path Method (cont’d)
• Primary objectives:
– Planning the project so that it can be
completed as quickly as possible
– Identifying those activities where their
delays is likely to affect the overall project
completion date
• Developed by Du Pont Chemical
Company and published in 1958
• Capture the activities and their inter-relationships
using a graph
– Lines are used to represent the activities
– Nodes are used to represent the start and stop of
activities
• Adding time dimension
– The forward pass
• calculate the earliest start dates of the activities
• To calculate the project completion date
– The backward pass
• calculate the latest start dates for activities
• identify the critical path from the graph
• Identifying critical path and critical

event
– Critical event: an event that has zero
slack
– Critical path: a path joining those
critical events
Software Project Management 304

4 Task Dependency Types
• Mandatory Dependencies
• “Hard logic” dependencies
• Nature of the work dictates an ordering
• Ex: Coding has to precede testing
• Ex: UI design precedes UI implementation
• Discretionary Dependencies
• “Soft logic” dependencies
• Determined by the project management team
• Process-driven
• Ex: Discretionary order of creating certain modules
• External Dependencies
• Outside of the project itself
• Ex: Release of 3rd party product; contract signoff
• Ex: stakeholders, suppliers, Y2K, year end
• Resource Dependencies
• Two task rely on the same resource
• Ex: You have only one DBA but multiple DB tasks
Task Dependency Relationships
• Finish-to-Start (FS)
– B cannot start till A finishes
– A: Construct fence; B: Paint Fence
• Start-to-Start (SS)
– B cannot start till A starts
– A: Pour foundation; B: Level concrete
• Finish-to-Finish (FF)
– B cannot finish till A finishes
– A: Add wiring; B: Inspect electrical
• Start-to-Finish (SF)
– B cannot finish till A starts (rare)
Example Step 1
Forward Pass
• To determine early start (ES) and early finish (EF) times for
each task
• Work from left to right
• Adding times in each path
• Rule: when several tasks converge, the ES for the next task
is the largest of preceding EF times
Example Step 2
Backward Pass
• To determine the last finish (LF) and last start (LS) times
• Start at the end node
• Compute the bottom pair of numbers
• Subtract duration from connecting node’s earliest start time
Example Step 3
Example Step 4
Example to construct a CPM
Id. Activity Name Duration (weeks) Precedents
A Hardware selection 7
B Software design 4
C Hardware Installation 6 A
D Coding 4 B
E Data Preparation 5 B
F User Documentation 9
G User Training 5 E,F
H System Installation 3 C,D
Example to construct a CPM (cont’d)
Event Number
3
Earliest start date 1 7 Latest start date
6
Slack
Example to construct a CPM (cont’d)
2
7 7
0 C=6
A=7
1 B=4 3 D=4 4 H=3 6

0 0 4 6 13 13 16 16
0 2 0 0
E=5
F=9 5 G=5
9 11
2
Significance of critical path
• During planning stage
– Shortening the critical path will reduce the
overall project duration
• During management stage
– Pay more attention to those activities which
fall in the critical path
Network Diagrams
• Advantages
– Show precedence well
– Reveal interdependencies not shown in other techniques
– Ability to calculate critical path
– Ability to perform “what if” exercises
• Disadvantages
– Default model assumes resources are unlimited
• You need to incorporate this yourself (Resource Dependencies)
when determining the “real” Critical Path
– Difficult to follow on large projects
PERT
• Program Evaluation and Review Technique
• Based on idea that estimates are uncertain
– Therefore uses duration ranges
– And the probability of falling to a given range
• Uses an “expected value” (or weighted
average) to determine durations
• Use the following methods to calculate the
expected durations, then use as input to your
network diagram
PERT
• Start with 3 estimates
– Optimistic
• Would likely occur 1 time in 20
– Most likely
• Modal value of the distribution
– Pessimistic
• Would be exceeded only one time in 20
PERT Formula
• Combined to estimate a task duration
PERT Formula
• Confidence Interval can be determined
• Based on a standard deviation of the
expected time
• Using a bell curve (normal distribution)
• For the whole critical path use

PERT Example
Description Planner 1 Planner 2
m 10d 10d
a 9d 9d
b 12d 20d
PERT time 10.16d 11.5d
Std. Dev. 0.5d 1.8d
• Confidence interval for P2 is 4 times wider than P1 for a given probability

• Ex: 68% probability of 9.7 to 11.7 days (P1) vs. 9.5-13.5 days (P2)
PERT
• Advantages
– Accounts for uncertainty
• Disadvantages
– Time and labor intensive
– Assumption of unlimited resources is big issue
– Lack of functional ownership of estimates
– Mostly only used on large, complex project
• Get PERT software to calculate it for you
CPM vs. PERT
• Both use Network Diagrams
• CPM: deterministic
• PERT: probabilistic
• CPM: one estimate, PERT, three estimates
• PERT is infrequently used
Gantt Chart
• Disadvantages
– Does not show interdependencies well
– Does not process uncertainty of a given activity (as does
PERT)
• Advantages
– Easily understood
– Easily created and maintained
• Note: Software now shows dependencies among tasks
in Gantt charts
– In the “old” days Gantt charts did not show these
dependencies, bar charts typically do not. Modern Gantt
charts do show them.
Reducing Project Duration
• How can you shorten the schedule?
• Via
– Reducing scope (or quality)
– Adding resources
– Concurrency (perform tasks in parallel)
– Substitution of activities
Compression Techniques
• Shorten the overall duration of the project
• Crashing
• Looks at cost and schedule tradeoffs
• Gain greatest compression with least cost
• Add resources to critical path tasks
• Limit or reduce requirements (scope)
• Changing the sequence of tasks
• Fast Tracking
• Overlapping of phases, activities or tasks that would otherwise be
sequential
• Involves some risk
• May cause rework
Summary
• Describe the major issues in software project management
• Understand the basic principles for planning and scheduling a
project
• Network Diagrams Techniques
Session 6 Quiz
1. Ture or False?
Successful project are planned and tracked.
Traditional software development life-cycles heavily rely on accurate plans.
More agile (iterative) approaches rely on accurate short-term plans and renegotiation of
targets with the clients.
The reason for project shortcomings is often the management of projects.
A project is limited in time and scope.
Critical Path Method (CPM) Suitable for large software projects.
All projects have a Critical Path.
Accelerating non-critical tasks do not directly shorten the schedule.
Critical Path may change to another as you shorten the current.
2. CPM is an acronym for?
3. PERT is an acronym for?

SOFTWARE
What is Software Quality Assurance?

SESSION 7: SOFTWARE QUALITY MANAGEMENT
Quality Management
Quality Management System [ISO 9000]:
The organizational structure, responsibilities,
procedures, processes and resources for
implementing quality management
• Concerned with ensuring that the required level of
quality is achieved in a software product.
• Involves defining appropriate quality standards and
procedures and ensuring that these are followed.
• Should aim to develop a ‘quality culture’ where
quality is seen as everyone’s responsibility.
[Sommerville2004]
331
Scope of Quality Management
• Quality management is particularly important for
large, complex systems. The quality
documentation is a record of progress and
supports continuity of development as the
development team changes.
• For smaller systems, quality management needs
less documentation and should focus on
establishing a quality culture.
[Sommerville2004]
332
Quality Management Activities
• Quality assurance
– Establish organisational procedures and standards for quality.
• Quality planning
– Select applicable procedures and standards for a particular project
and modify these as required.
• Quality control
– Ensure that procedures and standards are followed by the
software development team.
Quality management should be separate from project management to

ensure independence.
[Sommerville2004]
333
Quality Planning
• Quality planning is the process of assessing the
requirements of the procedure and of the product
and the context in which these must be observed.
• Quality assurance plan is the central aid for planning
and checking the quality assurance.
[Pressman2004]
334
Quality Assurance Plan
• A quality assurance plan sets out the desired
product qualities and how these are assessed and
defines the most significant quality attributes.
• The quality assurance plan should define the
quality assessment process.
• It should set out which organisational standards
should be applied and, where necessary, define
new standards to be used.
[Sommerville2004]
335
Quality Plan
• Quality planning should begin at an early stage in the
software process.
– The result of the quality planning process is the
project quality plan.
• The quality plan should be as short as possible.
– Due to the wide range of quality attributes, in
general it is not possible to handle all quality
attributes. Hence, the plan should identify among
all the potential quality attributes, the most critical
ones, and focus on how to achieve them.
– The quality plan should also define the quality
assessment process.
336
Quality Assurance Plan Structure
• Product introduction: description of the product, its intended
market and quality expectations.
• Product plans: critical release dates and responsibilities for the
product along with plans for distribution and product servicing.
• Process descriptions: development and service processes which
should be used for product development and management.
• Quality goals: quality goals and plans for the product, including
an identification and justification of critical product quality
attributes.
• Risk and risk management: key risks which might affect product
quality and the actions to address these risks.
[Sommerville2004]
337
SQA Plan
• Management section
– describes the place of SQA in the structure of the
organization
• Documentation section
– describes each work product produced as part of the
software process
• Standards, practices, and conventions section
– lists all applicable standards/practices applied during the
software process and any metrics to be collected as part of
the software engineering work
338
SQA Plan (Con’t)
• Reviews and audits section
– provides an overview of the approach used in the
reviews and audits to be conducted during the
project
• Test section
– references the test plan and procedure document
and defines test record keeping requirements
• Problem reporting and corrective action section
– defines procedures for reporting, tracking, and
resolving errors or defects, identifies organizational
responsibilities for these activities
• Other
– tools, SQA methods, change control, record keeping,
training, and risk management
339
SQA Plan Example
340
Quality Control
Quality Control [ISO 9000]:
The operational techniques and activities that are used
to fulfil requirements for quality
• Quality Control is the series of inspections, reviews and tests
used throughout the development cycle to ensure that each
work product meets the requirements placed upon
it.[Pressman2004]
• This involves checking the software development process to
ensure that procedures and standards are being followed.
• There are two approaches to quality control
– Quality reviews;
– Automated software assessment and software
measurement.
341
Quality Control
• Objective:
– minimize the produced defects
– increase the product quality
• Implementation approaches:
– Fully automated
– Entirely manual
– Combination of automated tools and human
interactions
342
Quality Control
Measures you can take to reduce the occurrence of
errors:
• creating a quality policy, and ensuring staff
understand the procedures they must follow (the
quality system)
• adopting an analysis and design method, and a
project management method
• implementing standards for software and
documents
• independent checking and testing
• using automated software testing tools
• monitoring errors (metrics), tracing their cause,
improving the processes that regularly cause errors
General recommendations
• Quality cannot be "added in" by the quality
control process. Good quality products are
made that way from the start, not improved by
testing at the end of a project
• Some individuals consistently produce work of
high quality (and others consistently poor
quality), so choose a good team
• Make everyone in the organisation responsible
for the quality of the final product(s). This is the
basis of "Total quality management" or TQM
Encourage a quality culture in the organisation
Quality Control & Assurance
• In the ISO 9000 standard, clause 3.2.10 defines Quality
Control as:
“A part of quality management focused on fulfilling
quality requirements”
Clause 3.2.11 defines Quality Assurance as:
“A part of quality management focused on providing
confidence that quality requirements will be fulfilled”
– QA and QC are closely related concepts, and are
both aspects of quality management
– Quality Assurance focuses on the process of
quality, while Quality Control focuses on the
quality of output.
345
Quality Control & Assurance
• Quality Assurance is process oriented and focuses on
defect prevention, while quality control is product
oriented and focuses on defect identification.
– QA is the process of managing for quality.
– QC is used to verify the quality of the output;
– QA: a Strategy of Prevention
– QC: a Strategy of Detection
– QA is a managerial tool
– QC is a corrective tool
– Verification is an example of QA
– Validation/Software Testing is an example of QC
346
Summary
• Quality Management Activities
– Quality assurance
– Quality planning
– Quality control
• Quality Assurance is process oriented and focuses on defect

prevention, while quality control is product oriented and focuses
on defect identification.
Session 7 Quiz
1. Ture or False?
Quality management is particularly important for large, complex systems.
For smaller systems, quality management needs less documentation and should focus
on establishing a quality culture.
Quality management should be separate from project management to ensure

independence.
QA and QC are closely related concepts, and are both aspects of quality management.
QA is a managerial tool, QC is a corrective tool.
2. Quality Management Activities include ____________, ____________, ____________.

SOFTWARE
SESSION 8: PROJECT RISK MANAGEMENT

Overview
1. What is Risk Management?
2. Risk Identification
3. Risk Estimation
4. Risk Engineering
Risk Management is a Core Activity
• Software Management is about risk
management
– Maintaining a tolerance for uncertainty and
learning;
– Expecting and reacting to changing context;
– Structuring project management around
risk management;
– Understanding the relationship of the
project to organisational risk (internal and
external).
What is Risk Management?
• Risk:
– Risks are potential problems that may affect
successful completion of a software project.
– Risks involve uncertainty and potential losses (The
possibility of suffering harm or loss, danger).
– Risk analysis and management are intended to help
a software team understand and manage
uncertainty during the development process.
Risk is not bad
[Van Scoy, Roger L. Software Development Risk:
Opportunity, Not Problem. Software Engineering Institute,
CMU/SEI-92-TR-30, ADA 258743, September 1992]
• Risk in itself is not bad; risk is essential to
progress, and failure is often a key part of
learning. But we must learn to balance the
possible negative consequences of risk against the
potential benefits of its associated opportunity."
Risk Strategies
• Reactive strategies
– very common, also known as fire fighting
– project team sets resources aside to deal with
problems
– team does nothing until a risk becomes a problem
• Proactive strategies
– risk management begins long before technical
work starts, risks are identified and prioritised by
importance
– team builds a plan to avoid risks if they can or to
minimize risks if they turn into problems
Nature of Project Risks
• Planning assumptions
• Estimation errors
• Eventualities
Planning Assumptions
• Why assumptions
– Uncertainties in early stage of the project
• Common assumption:
– “Everything will go smoothly”
• Environment is reliable and fixed
• Design will be perfect first time
• Coding will be ‘nearly perfect’
• Guidelines
– List all the assumptions
– Identify the effects of these assumptions on the
project if they are no longer valid
Estimation Errors
• Difficult to have accurate size or time estimations
– Lack of experience of similar tasks
– Lack of historical data
– Nature of the task
• Estimation can be improved by analyzing historic
data for similar tasks and similar projects
– Keep historic data of your estimation and the actual
performance
– Compare your estimation and the actual value
– Classify the tasks that are easy or difficult to give
accurate estimation
Eventualities
• Unexpected and unimaginable events
• Common unexpected events
– Hardware cannot be delivered on time
– Requirements specification needs to be
rewritten
– Staffing problem
A loss associated with the event
• Loss of time, quality, control,
understanding
– e.g. Requirements change
– Loss of key staff
– Optimistic Estimates
• Risk Impact (often expressed in £ or $?)
• Cause - Effect
Overview
3. Risk Estimation
4. Risk Engineering
Risk Identification
• Identify the hazards that might affect the
duration or resource costs of the project
Hazard  Problem  Risk
• A hazard is an event that might occur
and will create a problem for the
successful completion of the project, if it
does occur
Hazard, Problem, and Risk
• Hazard: Mary’s baby may be born early
• Problem: Modules P and Q will have no
coder
• Risk: Milestone 7 will be delayed, or
extra budget will be needed to hire
another coder
Type of risks
• Generic risk (common to all projects)
– Standard checklist can be modified based on the risk analysis of
previous projects
• Specific risk (only applies to individual projects)
– More difficult to find
– Need to involve project team members
– Need an environment that encourages risk assessment
• Involuntary (e.g. size of system)
• Voluntary (accepting tight delivery date)
• Known risks
– predictable from careful evaluation of current project plan and those
extrapolated from past project experience
• Unknown risks
– some problems will simply occur without warning
Software Risks
• Project risks
– threaten the project plan
• Technical risks
– threaten product quality and the timeliness of the
schedule
• Business risks
– threaten the viability of the software to be built
(market risks, strategic risks, management risks,
budget risks)
Risk Identification Methods
• Checklists (simply registering possible

risks)
• Decomposition (breaking down risks into

component risks)
• Assumption Analysis (question “givens”!)

Risk Identification (cont’d)
• Guideline
– Use checklist that lists the potential
hazards and their corresponding factors
– Maintain an updated checklist for future
projects
Common Risk Factors
• Application • Changeover
factors factors
• Staff factors • Supplier factors
• Project factors • Environment
• Hardware and factors
software factors • Health and safety
factors
Application Factors
• Nature of the application
– A data processing application or a life-
critical system (e.g. X-ray emission system)
• Expected size of the application
– The larger is the size, the higher is the
chance of errors, communication problems
and management problems
Staff Factors
• Experience and skills
• Appropriateness of experience
• Staff satisfaction
• Staff turn-over rates
Project Factors
• Project objectives:
– Ill defined
– Unclear to every team member and user
• Project methods:
– Ill specified methods
– Unstructured methods
Hardware and Software Factors
• New hardware
– Stability of the new hardware system
• Cross platform development
– Development platform is not the operation
platform
– Does the language used support cross
platform development?
Changeover Factors
• ‘All-in-one’ changeover
– The new system is put into operation
• Incremental or gradual changeover
– Adding new components to the system by
phases
• Parallel changeover
– Both the existing system and the new
system are used in parallel
Supplier Factors
• Late delivery of hardware
• Instability of hardware
• Late completion of building sites
Environment Factors
• Changes in environment such as
hardware platforms
• Changes in government policies
• Changes in business rules
• Restructuring of organizations
Health and Safety Factors
• Health and safety of staff and
environment
– Staff sickness, death, pregnancy etc.
– Any tragic accident to staff
Boehm’s Top Ten Risk Items
• Personnel shortfalls
• Unrealistic schedules and budgets
• Developing the wrong software
functions
• Developing the wrong user interface
• Gold plating
Boehm’s Top Ten Risk Items (cont’d)
• Continuing stream of requirements

changes
• Shortfalls in externally performed tasks
• Shortfalls in externally furnished
components
• Real-time performance shortfalls
• Straining computer science capabilities
Overview
3. Risk Estimation
4. Risk Engineering
The likelihood that the event will occur
• Estimate probability that event will occur

– e.g. New machine not delivered on time.
• Risk probability (percentage)

Risk Estimation
• Recall that
Hazard  Problem  Risk
• Risk estimation is to assess the
likelihood and impact of each hazard
• Risk exposure (risk value)
– It is the importance of the risk
Risk exposure = risk likelihood × risk
impact
Quantification
• Risk exposure =
– Risk impact x risk probability
e.g. impact £50k, probability 15%
exposure = £7.5k
Risk Estimation Techniques
• Risk likelihood: The probability that a hazard
is going to occur
– Rank from Low, Medium to High
– Rank from 1 (least likely) to 10 (most likely)
• Risk Impact: The effect of the problem
caused by the hazard
– Rank from 1 to 10
Risk Impact
• Risk components
– performance
– cost
– support
– schedule
• Risk impact
– negligible
– marginal
– critical
– catastrophic
Risk Table Construction
• List all risks in the first column of the table
• Classify each risk and enter the category label in
column two
• Determine a probability for each risk and enter it into
column three
• Enter the severity of each risk (negligible, marginal,
critical, catastrophic) in column four.
• Sort the table by probability and impact value
• Determine the criteria for deciding where the sorted
table will be divided into the first priority concerns and
the second priority concerns
• First priority concerns must be managed.
Risk Estimation (cont’d)
• Advantages
– The only way to compare or rank the risks
– To have a good quantitative estimate, the
extra effort can provide a better
understanding of the problem
• Disadvantages
– Estimation is difficult, subjective, time-
consuming and costly
Risk Evaluation
• Ranking the risks
• Determining the corresponding risk
reduction strategies
Ranking Risks
• Ranking the risks based on their risk
exposures
• Ranking shows the order of importance
• In practice, also consider factors like
– Confidence of the risk assessment
– Compound risk
– The number of risks
– Cost of action
Risk Prioritisation
• Risk Exposure
• Compound risk reduction
– A priority scheme enables you to devote your

resources only to the most threatening risks
Risk Reduction Leverage (RRL)
• RRL is used to determine whether it is
worthwhile to carry out the risk
reduction plan.
• The higher is the RRL value, the more
worthwhile is to carry out the risk
reduction plan.
RE before − RE after
RRL =
risk reduction cost
Risk Leverage
• (Risk exposure before reduction - risk
exposure after reduction) / (cost of risk
reduction)
e.g. Paying higher salaries to reduce risk of losing

key personnel, RE before = £100k, RE after
=£40k, reduction cost = £30k
Risk Leverage = (100-40)/30 = 2
(greater than 1.00, attractive proposition)
Overview
3. Risk Estimation
4. Risk Engineering
Boehm’s Risk Engineering
Risk
Engineering
Risk Risk
Analysis Management
Risk Risk Risk Risk Risk Risk Risk Risk

Identification Estimation Evaluation Planning Control Monitoring Directing Staffing
Risk Analysis
• Risk identification
• Risk projection
– impact of risks/likelihood of risk actually
happening
• Risk assessment
– what will change if risk becomes problem
Risk Analysis (Cont’d)
• Performance models (e.g. Monte Carlo
simulations)
• Cost models (e.g. budgets, spreadsheets)
• Network Analysis (e.g. PERT / z values,)
• Decision Analysis (e.g. risk attitude)
• Quality Analysis (formal quality
management approach)
Risk Management
• Risk planning
• Risk control
• Risk monitoring
• Risk directing
• Risk staffing
Risk Planning
• Making contingency plans
• Where appropriate, adding these plans
into the project’s overall task structure
Risk Management Planning
• Contingency Planning (what to do if risk

materialises)
• Risk element planning (micro level)
• Risk plan integration (holistic enterprise
risk management)
The degree to which we can change
the outcome
• Minimising effect of event occurring.

• Avoid the event occurring.
Risk Control
Risk Monitoring and Control
• Risk monitoring
– Determine who is responsible for monitoring
– How are risks monitored?
• Project tracking, resources, quality, etc
– Communicating the status of identified risks
• Reviews and Audits
• Once a risk is identified as occurring
– Communicate
– Take action
Risk Monitoring
• It is an ongoing activity throughout the
whole project to monitor
– the likelihood of a hazard; and
– the impact of the problem caused.
Risk Control
• Minimizing and reacting to problems
arising from risks throughout the project
Risk Directing and Staffing
• These concerns with the day-to-day
management of risk.
• Risk aversion strategies and problem
solving strategies frequently involve the
use of additional staff and this must be
planned for and should be considered.
Risk Resolution
• Risk mitigation
• Risk Monitoring and reporting
• Risk reassessment
Risk Mitigation Example
Risk: loss of key team members
• Determine causes of job turnover.
• Eliminate causes before project starts.
• After project starts, assume turnover is going to occur
and work to ensure continuity.
• Make sure teams are organized and distribute
information widely.
• Define documentation standards and be sure
documents are produced in a timely manner.
• Conduct peer review of all work.
• Define backup staff.
RMMM
• Risk mitigation
– proactive planning for risk avoidance
• Risk monitoring
– assessing whether predicted risks occur or not
– ensuring risk aversion steps are being properly applied
– collect information for future risk analysis
– determining which risks caused which problems
• Risk Management
– contingency planning
– actions to be taken in the event that mitigation steps
have failed and the risk has become a live problem
Risk Reduction
• Hazard Prevention
• Likelihood reduction
• Risk reduction leverage
– Avoid the risk

– Transfer the risk
– Assume the risk
Risk Reduction Strategies
• 5 different types in a generic sense
– Hazard prevention
– Likelihood reduction
– Risk avoidance
– Risk transfer
– Contingency planning
• Distinctions among them are fuzzy
Hazard prevention
• Prevent a hazard from occurring or
reduce its likelihood to an insignificant
level
– Lack of skilled staff can be prevented by
employing staff with appropriate skills
– Unclear requirements specification can be
prevented by using formal specification
techniques
Likelihood reduction
• Reduce the likelihood of an unavoidable
risk by prior planning
– Late change to the requirements
specification can be reduced by using
prototyping
Risk avoidance
• Some hazards cannot be avoided but
their risks may
– A project can be protected from the risk of
overrunning the schedule by increasing
duration estimates.
Risk transfer
• The impact of the risk can be
transferred away from the project by
contracting out or taking out insurance
– The risk of shortfalls in external supplied
components can be transferred away by
quality assurance procedures and
certification, and contractual agreements.
Contingency planning
• Contingency plans are needed to
reduce the impact of those risks that
cannot be avoided
– The impact of any unplanned absence of
programming staff can be minimized by
using agency programmers
Invoke a contingence plan
• When a quantitative risk indicator crosses
a predetermined threshold.
• Usually you can have different levels,
– At first levels the operative level take action,
– If can’t correct the situation then the
contingency plan will start.
Recovery from a crisis
• After a crisis certain actions are required:
– Reward personnel who have worked in
burnout mode,
– Reevaluating cost and schedule in light of the
drain on resources from managing the crisis.
Manage the crisis
• Despite a team’s best effort, the
contingence plan may fail, in which case
project enters crisis mode
Summary
• Software Management is about risk management.
• Risk Identification Methods
– Checklists (simply registering possible risks)
– Decomposition (breaking down risks into component risks)
– Assumption Analysis (question “givens”!)
• Risk estimation is to assess the likelihood and impact of each
hazard
• Risk Management
– Risk Planning
– Risk Control
– Risk Monitoring
– Risk Directing
– Risk Staffing
• Risk Reduction Strategies
– Hazard prevention
– Likelihood reduction
– Risk avoidance
– Risk transfer
– Contingency planning
Session 8 Quiz
1. Ture or False?
Software Management is about risk management.
Risks involve uncertainty and potential losses
Risk estimation is to assess the likelihood and impact of each hazard
Estimation can be improved by analyzing historic data for similar tasks and similar
projects.
Contingency plans are needed to reduce the impact of those risks that cannot be
avoided.
The contingence plan may fail, in which case project enters crisis mode.
2. Risk management includes ____________, ____________, ____________, ____________,

____________.
3. Risk Reduction Strategies are ____________, ____________, ____________,

____________, ____________.
SOFTWARE
SESSION 9: PROJECT EVALUATION AND ESTIMATION

Overview
1. Project evaluation
2. Project Estimation
3. Estimation Models
Project Evaluation
• A high level assessment of the project
– to see whether it is worthwhile to proceed
with the project
– to see whether the project will fit in the
strategic planning of the whole
organisation
Project Evaluation - Why
• Want to decide whether a project can
proceed before it is too late
• Want to decide which of the several
alternative projects has a better success
rate, a higher turnover, a higher ...
• Is it desirable to carry out the
development and operation of the
software system?
Project Evaluation - Who
• Senior management
• Project manager/coordinator
• Team leader
Project Evaluation - When
• Usually at the beginning of the project
Project Evaluation - What
• Strategic assessment
• Technical assessment
• Economic assessment
Strategic Assessment
• Used to assess whether a project fits in the
long-term goal of the organisation
• Usually carried out by senior management
• Needs a strategic plan that clearly defines the
objectives of the organization
• Evaluates individual projects against the
strategic plan or the overall business
objectives
Strategic Assessment (cont’d)
• Programme management
– suitable for projects developed for use in
the organization
• Portfolio management
– suitable for project developed for other
companies by software houses
SA – Programme Management
• Individual projects as components of a
programme within the organization
Programme as “a group of projects that are
managed in a coordinated way to gain
benefits that would not be possible were
the projects to be managed independently”
by D.C. Ferns
Journal of Project Management
Aug. 1991
SA – Programme Management Issues
• Objectives
– How does the project contribute to the long-term goal of the organisation?
– Will the product increase the market share? By how much?
• IS plan
– Does the product fit into the overall IS plan?
– How does the product relate to other existing systems?
• Organization structure
– How does the product affect the existing organizational structure? the existing
workflow? the overall business model?
• MIS
– What information does the product provide?
– To whom is the information provided?
– How does the product relate to other existing MISs?
• Personnel
– What are the staff implications?
– What are the impacts on the overall policy on staff development?
• Image
– How does the product affect the image of the organisation?
SA – Portfolio Management
• suitable for product developed by a
software company for an organisation
• may need to assess the product for the
client organisation
– Programme management issues apply
• need to carry out strategic assessment
for the providing software company
SA – Portfolio Management Issues
• Long-term goal of the software

company
• The effects of the project on the
portfolio of the company (synergies and
conflicts)
• Any added-value to the overall portfolio
of the company
Technical Assessment
• Functionality against hardware and
software
• The strategic IS plan of the organisation
• any constraints imposed by the IS plan
Economic Assessment
• Consider whether the project is the best
among other options
• Prioritise the projects so that the
resources can be allocated effectively if
several projects are underway
Project Evaluation - How
• Cost-benefit analysis
• Cash flow forecasting
• Cost-benefit evaluation techniques
• Risk analysis
EA – Cost-benefit Analysis
• A standard way to assess the economic
benefits
• Two steps
– Identify and estimate all the costs and
benefits of carrying out the project
– Express the costs and benefits in a
common unit for easy comparison (e.g. $)
EA – Cost-benefit Analysis (cont’d)
• Costs
– Development costs
– Setup costs
– Operational costs
• Benefits
– Direct benefits
– Assessable indirect benefits
– Intangible benefits
EA – Cash Flow Forecasting
• What?
– Estimation of the cash flow over time
• Why?
– An excess of estimated benefits over the
estimated costs is not sufficient
– Need detailed estimation of benefits and
costs versus time
EA – Cash Flow Forecasting (Cont’d)
Income
Expenditure
EA – Cash Flow Forecasting (Cont’d)
• Need to forecast the expenditure

and the income
• Accurate forecast is not easy
• Need to revise the forecast from
time to time
Cost-benefit Evaluation Techniques
• Net profit
= Total income – Total costs
• Payback period
= Time taken to break even
• Return on Investment (ROI)
average annual profit
= ×100%
total investment
Cost-benefit Evaluation Techniques
Example
Year Project 1 Project 2 Project 3 Project 4
0 -100,000 -1,000,000 -100,000 -120,000
1 10,000 200,000 30,000 30,000
2 10,000 200,000 30,000 30,000
3 20,000 200,000 30,000 30,000
4 20,000 200,000 20,000 25,000
5 100,000 350,000 20,000 50,000
Net Profit 60,000 150,000 30,000 45,000
Payback 5 5 4 5
ROI 12% 3% 6% 7.5%
Cost-benefit Evaluation Techniques –
NPV
Net present value (NPV)
• It is the sum of the present values of all future
amounts.
• Present value is the value which a future
amount is worth at present
• It takes into account the profitability of a project
and the timing of the cash flows
• Discount rate is the annual rate by which we
discount future earning
– e.g. If discount rate is 10% and the return of an
investment in a year is $110, the present value of
the investment is $100.
NPV (cont’d)
• Let n be the number of year and r be
the discount rate, the present value
(PV) is given by
value in year n
PV =
(1 + r ) n
NPV (cont’d)
• Issues in NPV
– Choosing an appropriate discount rate is difficult
– Ensuring that the rankings of projects are not
sensitive to small changes in discount rate
• Guidelines:
– Use the standard rate prescribed by the
organisation
– Use interest rate + premium rate
– Use a target rate of return
– Rank the projects using various discount rates
IRR
• Internal Rate of Return (IRR)
– The percentage discount rate that would
produce a NPV of zero
– A relative measure
IRR (cont’d)
Net Present Value($)
9000
6000
3000 Discount rate

(%)
0
8 9 10 11 12
-3000
IRR (cont’d)
• Advantages
– Convenient
• Directly comparable with rate of return on other
projects and with interest rates
– Useful
• Dismiss a project due to its small IRR value
• Indicate further precise evaluation of a project
– Supported by MS Excel and Lotus 1-2-3
Overview
Estimation
“The single most important task of a project: setting
realistic expectations.
Unrealistic expectations based on inaccurate estimates

are the single largest cause of software failure.”
• Why? – to define the project budget and to ‘refine’ the product to

realise the budget
• Who? – the manager…
• What? – size and cost
• When? – always
• How? – techniques and models
Different level of estimation
• Before decision to do a project
– The estimation is coarse
– The estimation is in high level terms
• Profit? Good to the organisation? etc.
• After decision to go ahead
– More detailed size and cost estimations
are required
Issues related to Estimation
• Difficult to make accurate estimation
• Better to have previous data and analyse the
actual values against their estimates so that you
know how accurate you are
• Even better to have previous data of the whole
organisation so that you know how accurate the
estimation method, if any, used within the
organisation is
• Use your estimation as a guide to manage your
project
• From time to time, you need to revise your
estimation based on the current status of the
project
Why is Estimating So Difficult?
• Think of some things you have to estimate: Are you
generally accurate? Are your estimates generally
optimistic?
• People often underestimate times and distances. Why is
this?
• Should we rely on people’s subjective estimates of
programming tasks?
• Often accurate estimates of the size of a software project
are so large as to be dismissed as unrealistic, or put the
client off. Is there a compromise possible?
Who should do the estimating?
• Development Team
• Consultant Project Estimator
• Project Manager
• Customer
• Steering Committee
What are we trying to estimate?
• Staff resources Analysis
– Programming
– Documentation
– Management
• Computer resources
– Terminals
– Disk space requirements
– CPU time
• Overheads
– Support
– Meetings
– Performance assessment
– Project control systems
Fundamental estimation questions
• How much effort is required to complete an
activity?
• How much calendar time is needed to
complete an activity?
• What is the total cost of an activity?
• Project estimation and scheduling are
interleaved management activities.
Costing and pricing
• Estimates are made to discover the cost, to
the developer, of producing a software
system.
• There is not a simple relationship between
the development cost and the price charged
to the customer.
• Broader organisational, economic, political
and business considerations influence the
price charged.
Estimation methods
• Each method has strengths and weaknesses.
• Estimation should be based on several
methods.
• If these do not return approximately the same
result, then you have insufficient information
available to make an estimate.
• Some action should be taken to find out more
in order to make more accurate estimates.
• Pricing to win is sometimes the only applicable
method.
Properties of an Estimating Method
• Gives early guidance to the size of a project with an
accuracy of +/- 30%,
• Should be easy to use,
• Should give consistent results,
• Uses agreed rules,
• Is supported by tools.
Estimation Approaches
• Expert judgement
– Ask the knowledgeable experts
• Estimation by analogy
– Use the data of a similar and completed
project
• Pricing to win
– Use the price that is low enough to win the
contract
Estimation Approaches (cont’d)
• Top-down
– An overall estimate is determined and then broken
down into each component task
• Bottom-up
– The estimates of each component task are
aggregated to form the overall estimate
• Algorithmic model
– Estimation is based on the characteristics of the
product and the development environment.
Expert judgment
• One or more experts in both software
development and the application domain use
their experience to predict software costs.
Process iterates until some consensus is
reached.
• Advantages: Relatively cheap estimation
method. Can be accurate if experts have
direct experience of similar systems
• Disadvantages: Very inaccurate if there are
no experts!
Estimation by analogy
• The cost of a project is computed by
comparing the project to a similar project in
the same application domain
• Advantages: May be accurate if project data
available and people/tools the same
• Disadvantages: Impossible if no comparable
project has been tackled. Needs
systematically maintained cost database
Cost Pricing to win
• The project costs whatever the customer has
to spend on it
• Advantages: You get the contract
• Disadvantages: The probability that the
customer gets the system he or she wants is
small. Costs do not accurately reflect the work
required.
• How do you know what customer has?
• Only a good strategy if you are willing to take a
serious loss to get a first customer, or if
Delivery of a radically reduced product is a real
option.
Cost Pricing to win(Cont’)
• This approach may seem unethical and un-
businesslike.
• However, when detailed information is
lacking it may be the only appropriate
strategy.
• The project cost is agreed on the basis of an
outline proposal and the development is
constrained by that cost.
• A detailed specification may be negotiated or
an evolutionary approach used for system
development.
Top-down and bottom-up estimation
• Any of these approaches may be used top-
down or bottom-up.
• Top-down
– Start at the system level and assess the overall
system functionality and how this is delivered
through sub-systems.
• Bottom-up
– Start at the component level and estimate the
effort required for each component. Add these
efforts to reach a final estimate.
Top-down estimation
• Usable without knowledge of the system
architecture and the components that might
be part of the system.
• Takes into account costs such as
integration, configuration management and
documentation.
• Can underestimate the cost of solving
difficult low-level technical problems.
Bottom-up estimation
• Usable when the architecture of the system is
known and components identified.
• This can be an accurate method if the system
has been designed in detail.
• It may underestimate the costs of system
level activities such as integration and
documentation.
Overview
Algorithmic cost modeling
• Cost is estimated as a mathematical function of
product, project and process attributes whose
values are estimated by project managers
• The function is derived from a study of historical
costing data
• Most commonly used product attribute for cost
estimation is LOC (code size)
• Most models are basically similar but with
different attribute values
Software Cost Estimation Models
Sizer Stage
NOT BASED ON LINES OF CODE
Productivity Based on lines Based on Based on etc.

Stage of code function points functional
primitives
Software productivity
• A measure of the rate at which individual
engineers involved in software development
produce software and associated
documentation.
• Not quality-oriented although quality
assurance is a factor in productivity
assessment.
• Essentially, we want to measure useful
functionality produced per time unit.
Productivity measures
• Size related measures based on some output
from the software process. This may be lines
of delivered source code, object code
instructions, etc.
• Function-related measures based on an
estimate of the functionality of the delivered
software. Function-points are the best known
of this type of measure.
Measurement problems
• Estimating the size of the measure (e.g. how
many function points).
• Estimating the total number of programmer
months that have elapsed.
• Estimating contractor productivity (e.g.
documentation team) and incorporating this
estimate in overall estimate.
Productivity estimates
• Real-time embedded systems, 40-160
LOC/P-month.
• Systems programs , 150-400 LOC/P-month.
• Commercial applications, 200-900
LOC/P-month.
• In object points, productivity has been
measured between 4 and 50 object
points/month depending on tool support and
developer capability.
Quality and productivity
• All metrics based on volume/unit time are
flawed because they do not take quality into
account.
• Productivity may generally be increased at the
cost of quality.
• It is not clear how productivity/quality metrics
are related.
• If requirements are constantly changing then an
approach based on counting lines of code is not
meaningful as the program itself is not static;
Size Estimation
• Problems related to size estimation
• Size Estimation Model
– Function Point Analysis (FPA)
Problems related to size estimation
• Nature of software
• Novel application of software
• Fast changing technology
• Lack of homogeneity of project
experience
• Subjective nature of estimation
• Political implications within the
organization
Function Point Analysis (FPA)
• Developed by A. Albrecht in IBM
• Aim: To estimate the LOC of a system
LOC of system
= FP of system × LOC-per-FP of the
language
Function Point Analysis (cont’d)
• Idea: Software system consists of five
major components (or, external user
types)
– External input types
– External output types
– Logical internal file types
– External interface file types
– External inquiry types
Function Point Analysis - Steps
• Identify each instance of each external
user type in the proposed system
• Classify each instance as having high,
medium or low complexity
• Assign the FP of each instance
• FP of the system = sum of FP of
individual components
Function Point Analysis
Number of FPs Complexity
External user type Low Average High
External input type 3 4 6
External output type 4 5 7
Logical internal file type 7 10 15
External interface file type 5 7 10
External inquiry type 3 4 6
Function Point Analysis - Example
• A component of an inventory system
consisting of ‘Add a record’, ‘Delete a record’,
‘Display a record’, ‘Edit a record’, and ‘Print a
record’ will have
– 3 external input types
– 1 external output type
– 1 external inquiry type
Then, assign FPs based on the complexity of
each type
Function Point Analysis-Issues
• The assignment of level of complexity is
rather subjective
• International FP User Group (IFPUG)
imposes rules on assigning the level of
complexity to individual external user
types
Object Point Analysis
• Similar to function point analysis
• Used on 4GL development projects
• Takes account of features that may be
more readily identifiable if the system is
built on high-level application building
tools
Object Point Analysis – Steps
• Identify the number of screens, reports and 3GL
components
• Classify each object as Simple, Medium and Difficult
• Assign the weight accordingly
• Calculate the total object points
Total OP = sum of individual OP × weighting
• Deduct the reused objects (r% reused)
NOP = OP × (1 – r%)
• Identify the productivity rate of both developer and
CASE
• Productivity rate = average of the two PRs
• Calculate the effort
Effort = NOP / Productivity Rate
Object Point Analysis – Screens
Number and source of data tables
Number of Total < 4 Total < 8 Total 8+

views (<2 server, (2-3 server, (>3 server,
contained <2 client) 3-5 client) >5 client)
<3 Simple Simple Medium
3–7 Simple Medium Difficult
8+ Medium Difficult Difficult

Object Point Analysis – Reports
Number and source of data tables
Number of Total < 4 Total < 8 Total 8+

sections (<2 server, (2-3 server, (>3 server,
contained <2 client) 3-5 client) >5 client)
<2 Simple Simple Medium
2 or 3 Simple Medium Difficult
>3 Medium Difficult Difficult

Object Point Analysis – Complexity
Weightings
Complexity
Type of object Simple Medium Difficult
Screen 1 2 3
Report 2 5 8
3GL component N/A N/A 10

Object Point Analysis – Productivity
Rate
Very Very
Low Nominal High
low High
Developer’s
experience 4 7 13 25 50
and capability
CASE
maturity and 4 7 13 25 50
capability
Object Point Analysis – Issues
• Adopted in Boehm’s COCOMO II in the
application composition stage
Constructive Cost Model II
(COCOMO II - Empirical Model )
• A parametric cost model
– Important aspects of software projects are
characterized by variables (or parameters)
– Once the value of the parameters are
determined, the cost can be computed
from an equation
• Recognises different approaches to
software development
– Prototyping, Incremental development etc.
COCOMO II
• A family of models
– Uses different models in 3 different stages of the project
• 3 stages: application composition, early design and post architecture
– Supports estimation early in the process
– Allows further detailed estimation after the system architecture has been defined
• The basic model equation
Effort = Constant × (Size)scale factor
× Effort Multiplier
– Effort in terms of person-months
– Constant: 2.45 in 1998
– Size: Estimated Size in KSLOC
– Scale Factor: combined process factors
– Effort Multiplier (EM): combined effort factors
COCOMO II (cont’d)
• Advantages • Disadvantages
– Good improvement – Still immature,
over COCOMO diverse projects in
– Good match for database
iterative – Hard to believe
development,
that it will be any
modern technology,
and management
more reliable than
process the original
COCOMO model
Summary
• Project Evaluation
– Strategic assessment
– Technical assessment
– Economic assessment
• Economic Assessment
– Cost-benefit analysis
– Cash flow forecasting
– Cost-benefit evaluation techniques
• Estimation Approaches
– Expert judgement
– Estimation by analogy
– Pricing to win
• Size Estimation
– Function Point Analysis (FPA)
– Object Point Analysis
Session 9 Quiz
1. Ture or False?
Project Evaluation is a high level assessment of the project.
Strategic Assessment Used to assess whether a project fits in the long-term goal of the
organisation.
Strategic Assessment usually carried out by senior management.
Unrealistic expectations based on inaccurate estimates are the single largest cause of
software failure.
It is difficult to make accurate estimation.
There is not a simple relationship between the development cost and the price charged
to the customer.
Most commonly used product attribute for cost estimation is LOC (code size).
Productivity may generally be increased at the cost of quality.
2. Strategic Assessment includes ____________, ____________.

SESSION 10: SOFTWARE QUALITY
ASSURANCE TECHNIQUES
Quality Assurance Techniques
Software Faults, Errors & Failures
1. Software Fault : A static defect (bug) in the
software.
2. Software Error or Mistake: An incorrect internal

state that is the manifestation of some faults.
3. Software Failure : External, incorrect behaviour

with respect to the requirements or other
description of the expected behaviour.
Example: Failure-Error-Fault
A Y2K issue
• Failure: person’s age appears as negative!
• Error: failed to account for dates beyond
20th century
• Fault: code for computing age yields
negative value if birthdate is in 20th
century and current date in 21st.
Code Example: Fault-Error-Failure
public static int numZero (int[] x) {
// Effects: if x == null throw NullPointerException
// else return the number of occurrences of 0 in x
int count = 0;
for (int i = 1; i < x.length; i++)
{
if (x[i] == 0)
{
count++;
}
}
return count;
}
• Fault: The fault in this program is that it starts looking for zeroes at index 1 instead
• of index 0, as is necessary for arrays in Java.
• Error: case1: numZero([2,7,0]), case2: numZero([0,7,2]) - error states
• Failure: Case 2: Output = 0
• No Failure: Case 1: Output = 1
Fault-Failure Model (RIP Model)
• Three conditions must be present for a
failure to be observed.
1. The location or locations in the program that
contain the fault must be reached (Reachability).
2. After executing the location, the state of the
program must be incorrect (Infection).
3. The infected state must propagate to cause some
output of the program to be incorrect
(Propagation).
Verification and Validation
• Verification
– Evaluation of software system that help in
determining whether the product of a given
development phase satisfy the requirements
established before the start of that phase
• Building the product correctly
• Validation
– Evaluation of software system that help in
determining whether the product meets its intended
use
• Building the correct product
501
Static and Dynamic Techniques
• With dynamic methods, software is executed using
a set of input values and its output is then
examined and compared to what is expected.
• With static methods, software work products are
examined manually, or with a set of tools, but not
executed. As a consequence, dynamic methods can
only be applied to software code.
• Dynamic and static methods are complementary
methods, as they tend to find different types of
defects effectively and efficiently.
Static Techniques
1. Static techniques is used to find errors
before software is actually executed.
2. The earlier we catch an error, the cheaper it
is, usually, to correct.
3. Some techniques are applied to
documentation (e.g. walkthroughs, reviews and
Inspections) and some are used to analyse the
physical code (e.g. compilers, data flow
analysers).
This is a huge subject and we can only hope to
give an introduction.
Review
1. Basically a review is any of a variety of activities involving
evaluation of technical matter by a group of people working
together.
2. The objective of any review is to obtain reliable information,
usually as to status and/or work quality.
3. There are many different types of reviews throughout the
development life cycle. This includes, requirements documents,
designs, database specifications, designs, data models, code,
test plans, test scripts, test documentation and so on.
4. Reviews are the only testing technique that is available to us in
the early stages of testing.
5. Reviews share similarities to test activities in that they must
be planned (what are we testing), what are the criteria for
success (expected results) and who will do the work
(responsibilities).
Why do peer reviews?
• To improve quality.
• Catches 80% of all errors if done properly.
• Catches both coding errors and design errors.
• Enforce the spirit of any organization
standards.
• Training and insurance.
505
Roles and responsibilities
1. The moderator (or review leader)
2. The author
3. The scribe (or recorder)
4. The reviewers (checkers or inspectors)
5. The manager
Review Guidelines
• Keep it short (< 30 minutes).
• Don’t schedule two in a row.
• Don’t review product fragments.
• Use standards to avoid style disagreements.
• Let the coordinator run the meeting and
maintain order.
507
Rules for Review
• A structured approach must be for the
review process.
• Be sure to know what is being reviewed
• Checklists must be used.
• Reports must be produced.
• Quality must be specified in terms of:
Adherence to standards; Reliability required;
Robustness required; Modularity; Accuracy;
Ability to handle errors.
Formality and Timing
• Formal review presentations
– resemble conference presentations.
• Informal presentations
– less detailed, but equally correct.
• Early
– tend to be informal
– may not have enough information
• Later
– tend to be more formal
– Feedback may come too late to avoid rework
509
When
• Analysis is complete.
• Design is complete.
• After first compilation.
• After first test run.
• After all test runs.
• Any time you complete an activity that
produce a complete work product.
510
Review procedure for documents
1. Establishing standards and format for document.
2. Check contents list.
3. Check appendix list.
4. Follow up references outside documents.
5. Cross check references inside document.
6. Check for correct and required outputs.
7. Review each screen layout against appropriate standard, data
dictionary, processing rules and files/data base access.
8. Review each report layout against appropriate standard, data
dictionary, processing rules and files/data base access.
9. Review comments and reports on work and reviews done prior to this
review.
10. Documents reviewed will range from whole reports such as
Specification of Requirements to pages from output that system is
to produce. All documents will need careful scrutiny.
Types of Review
1. Walk-through
2. Informal reviews
3. Technical reviews
4. Inspections
 All have their strengths and weaknesses

and their place in the project development
cycle.
Walkthrough
• A walkthrough is characterised by the author of the
document under review guiding the participants
through the document and his or her thought
processes, to achieve a common understanding and to
gather feedback.
• This is especially useful if people from outside the
software discipline are present, who are not used to,
or cannot easily understand software development
documents.
• The content of the document is explained step by
step by the author, to reach consensus on changes or
to gather information.
Walkthrough(Cont)
• Goals:
- to present the document to stakeholders both
within and outside the software discipline, in
order to gather information regarding the
topic under documentation;
- to explain (knowledge transfer) and evaluate
the contents of the document;
- to establish a common understanding of the
document;
- to examine and discuss the validity of proposed
solutions and the viability of alternatives,
establishing consensus.
Walkthrough(Cont)
• Key characteristics:
- The meeting is led by the authors; often a
separate scribe is present.
- Scenarios and dry runs may be used to
validate the content.
- Separate pre-meeting preparation for
reviewers is optional.
Technical Review
• A technical review is a discussion meeting that focuses on
achieving consensus about the technical content of a
document.
• Compared to inspections, technical reviews are less
formal and there is little or no focus on defect
identification on the basis of referenced documents,
intended readership and rules.
• During technical reviews defects are found by experts,
who focus on the content of the document.
• The experts that are needed for a technical review are,
for example, architects, chief designers and key users.
• In practice, technical reviews vary from quite informal to
very formal.
Technical Review(Cont)
• Goals:
- assess the value of technical concepts and
alternatives in the product and project
environment;
- establish consistency in the use and
representation of technical concepts;
- ensure, at an early stage, that technical
concepts are used correctly;
- inform participants of the technical
content of the document.
Technical Review(Cont)
- It is a documented defect-detection process that
involves peers and technical experts.
- It is often performed as a peer review without
management participation.
- Ideally it is led by a trained moderator, but possibly also
by a technical expert.
- A separate preparation is carried out during which the
product is examined and the defects are found.
- More formal characteristics such as the use of
checklists and a logging list or issue log are optional.
Inspection
• Inspection is the most formal review type.
• The document under inspection is prepared and checked
thoroughly by the reviewers before the meeting, comparing
the work product with its sources and other referenced
documents, and using rules and checklists.
• In the inspection meeting the defects found are logged and
any discussion is postponed until the discussion phase. This
makes the inspection meeting a very efficient meeting.
• The reason for carrying out inspections can be explained by
using Weinberg's concept of egoless engineering [Weinberg,
1971]. Since we tend not to see evidence that conflicts with
our strong beliefs, our ability to find errors in our own work is
impaired.
• Depending on the organisation and the objectives of a project,
inspections can be balanced to serve a number of goals.
Inspection(Cont)
• Goals:
- help the author to improve the quality of the document under
inspection;
- remove defects efficiently, as early as possible;
- improve product quality, by producing documents with a higher
level of quality;
- create a common understanding by exchanging information among
the inspection participants;
- train new employees in the organisation's development process;
- learn from defects found and improve processes in order to
prevent recurrence of similar defects;
- sample a few pages or sections from a larger document in order
to measure the typical quality of the document, leading to
improved work by individuals in the future, and to process
improvements.
Inspection(Cont)
- It is usually led by a trained moderator (certainly not by the author).
- It uses defined roles during the process.
- It involves peers to examine the product.
- Rules and checklists are used during the preparation phase.
- A separate preparation is carried out during which the product is
examined and the defects are found.
- The defects found are documented in a logging list or issue log.
- A formal follow-up is carried out by the moderator applying exit
criteria.
- Optionally, a causal analysis step is introduced to address process
improvement issues and learn from the defects found.
- Metrics are gathered and analysed to optimise the process.
Success factors for reviews
• Find a 'champion'
• Pick things that really count
• Explicitly plan and track review activities
• Train participants
• Manage people issues
• Follow the rules but keep it simple
• Continuously improve process and tools
• Report results
• Just do it!
Static Analysis
• Static analysis is an examination of requirements, design
and code that differs from more traditional dynamic
testing in a number of important ways:
 Static analysis is performed on requirements, design
or code without actually executing the software
artifact being examined.
 Static analysis is ideally performed before the types
of formal review.
 Static analysis is unrelated to dynamic properties of
the requirements, design and code, such as test
coverage.
 The goal of static analysis is to find defects, whether
or not they may cause failures. As with reviews, static
analysis finds defects rather than failures.
Value of Static Analysis
• early detection of defects prior to test execution;
• early warning about suspicious aspects of the code,
design or requirements;
• identification of defects not easily found in dynamic
testing;
• improved maintainability of code and design since
engineers work according to documented standards
and rules;
• prevention of defects, provided that engineers are
willing to learn from their errors and continuous
improvement is practised.
Summary
• Validation and Verification
• Types of Review
– Walk-through
– Informal reviews
– Technical reviews
– Inspections
Session 10 Quiz
1. Ture or False?
Verification is to build the product correctly.
Validation is to build the correct product
With dynamic methods, software is executed using a set of input values and its output is
then examined and compared to what is expected.
With static methods, software work products are examined manually, or with a set of
tools, but not executed.
Dynamic and static methods are complementary methods, as they tend to find different
types of defects effectively and efficiently.
Static techniques is used to find errors before software is actually executed.
The earlier we catch an error, the cheaper it is, usually, to correct.
Reviews are the only testing technique that is available to us in the early stages of
testing.
Inspection is the most formal review type.
Static analysis finds defects rather than failures.
2. Types of Review include ____________, ____________, ____________, ____________.

CTEC5163 (2021) Software Quality Assurance

Uploaded by

Copyright:

Available Formats

You might also like

CTEC5163 (2021) Software Quality Assurance

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CTEC5163 (2021) Software Quality Assurance

Uploaded by

Copyright:

Available Formats

CTEC5163

Software Quality Assurance and Testing

Reader in Computer Science

Software Technology Research Laboratory

1. Don’t be afraid to ask — ever!

S3: ISO 9000 S7: Software S11: S15: Test

S4: Quality in S8: Risk S12: Test S16: Course Recap

– It covers basic concepts, sound principles, best practices, and

– The emphasis is placed on the skills of problem formulation,

How to develop quality software?

 IBM Consulting group estimates that 55% of large

Figure 1.1: The Shewhart cycle

Figure 1.2: Ishikawa diagram

• “Set of systematic activities providing

• Reviews, inspections, walkthroughs

• Reviews, inspections, walkthroughs of

Standards used for

Design ISO 9003

Software High- Low-

Application of ISO 9001 Standard to the Manufacturing and

Design Production Functionality

Documented procedures & processes X X X X X X X X X X X X X X X X

Need to be: Personnel

4.1 Management responsibility 4.1

4.2 Quality system 4.2

4.3 Internal quality system audits 4.17

4.4 Corrective action 4.14

• Note: ISO 9002 and ISO 9003 were parts of ISO

2. For software development, ____________ is the standard to use.

SESSION 4: QUALITY IN SOFTWARE PROCESSES

System Design System Testing

Validate Integrate System

Design system version n n = n+1

Develop system version Validate system System

• Phase entry and exit criteria - inputs

The Rational Unified Process Model

SESSION 6: SOFTWARE PROJECT MANAGEMENT

• Greater chance of being affected by

 Feasibility study - decide if project is worth doing

Bar chart - Technical Plan

• Developed in the 1950’s

• Identifying critical path and critical

Software Project Management 304

1 B=4 3 D=4 4 H=3 6

• For the whole critical path use

• Confidence interval for P2 is 4 times wider than P1 for a given probability

2. CPM is an acronym for?

3. PERT is an acronym for?

What is Software Quality Assurance?

Quality management should be separate from project management to

• Quality Assurance is process oriented and focuses on defect

Quality management should be separate from project management to ensure

QA is a managerial tool, QC is a corrective tool.

2. Quality Management Activities include ____________, ____________, ____________.

SESSION 8: PROJECT RISK MANAGEMENT

• Checklists (simply registering possible

• Decomposition (breaking down risks into

• Assumption Analysis (question “givens”!)

2. Quality Management Activities include , , .

2. Risk management includes , , , ,

3. Risk Reduction Strategies are , , ,

2. Strategic Assessment includes , .