Software Quality Assurance

What is SW?
Software is:

Computer programs, procedures, and possibly associated documentation and data pertaining to the operation of a computer system.

What is special about Software?

Invisibility of the product Limited opportunities to detect defects

(bugs) Often new demanding functionality has to be realized Often software has to realize extraordinary high complexity

The uniqueness of software quality assurance

High product complexity
-Millions of software operation possibilities

Product visibility
-Software products are invisible

Opportunities to detect defects (bugs) are

limited to the product development phase

The uniqueness of SQA (cont.)

Typical causes of software errors

Faulty definition of requirements Client-developer communication failures Deliberate deviations from software requirements Logical design errors Coding errors Non-compliance with documentation and coding

instructions Shortcomings of the testing process User Interface and procedure errors Documentation errors
6

Client-developer communication failures

Misunderstanding of the clients requirement

document Miscommunications during client-developer meetings Misinterpretation of the clients requirement changes

Faulty definition of requirements

One of the main causes of software errors Erroneous definition of requirements Missing vital requirements Incomplete requirements Unnecessary requirements

Deliberate deviations from the software requirements

The developer reuses software modules from earlier

projects without sufficient analysis of the changes needed to fulfill the requirements Due to time/money pressures, the developer leaves parts of the required functions in an attempt to manage the pressure The developer makes unapproved improvements to the software without the clients knowledge

Logical design errors

Defining the software requirements by means

of erroneous algorithms Process definitions contain sequencing errors Erroneous definition of boundary conditions Omission of definitions concerning special cases (lack of error handling and special state handling)

10

Coding errors
Misunderstanding of the design documents,

linguistic errors, development tool errors, and so forth

11

Non-compliance with documentation and coding instructions

Non-compliance with coding/documentation

standards makes it harder to understand, review, test and maintain the software

12

Shortcomings of the testing process

Greater number of errors are left undetected

and uncorrected Incomplete test plans will leave many of the functions and states untested Failures to report and promptly correct found errors and faults Incomplete test due to time pressures

13

User Interface and Procedure errors

Misunderstanding in process implementation

and failure of communication with potential users on the details of planned solution and the steps required to perform the application as in SW systems processing is conducted in several steps

14

Documentation errors
Omission of software functions Errors in explanations and instructions List of non-existing SW functions that were

planned in early development stage but later dropped and the inclusion of SW functions which were dropped in the early stages have been included in the later stages Incorrect phrasing, negligence, incorrect versions
15

What is quality?
Quality, simplistically, means that a product should

meet its specification. This is problematical for software systems

There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.); Some quality requirements are difficult to specify in an unambiguous way; Software specifications are usually incomplete and often inconsistent.

16

Software Quality IEEE View

The degree to which a system, component, or

process meets specified requirements. The degree to which a system, component, or process meets customer or user needs or expectations.

17

Software Quality-Pressman2004
Conformance to explicitly stated functional

and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software

18

Software Quality-Glass1992
the degree of excellence of something. We

measure the excellence of software via a set of attributes

19

Software quality assurance definition

Definition: SQA is a systematic, planned set of

actions necessary to provide adequate confidence that the software development process or the maintenance process of a software system product conforms to established functional technical requirements as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines
--Daniel Galin

20

Software Quality Control

Quality Control
Set

of activities designed to evaluate the quality of a developed or manufactured product (IEEE 1991). With holding of any product not qualifying

21

21

The objectives of SQA activities

Software development (process oriented)
-To assure that the software will meet the functional technical requirements -To assure that the software will conform to managerial scheduling and budgetary requirements -To continuously improve the development process and SQA activities in order to improve the quality and at the same time reduce the cost

The same things apply to software maintenance

(product oriented)

22

Ishikawa
6 Features of quality work

Quality Policy

Company wide quality control Top management quality control audit Education and training Quality circles Application of statistical methods Nationwide quality control promotion (eg Japan)

A work is guided by quality policies. Must formulate quality requirements for projects. Use quality plans for projects. A quality manual is used to define a quality management system to implement quality policy
23

Juran
Strategy for achieving quality: Structured annual improvements in quality Massive quality-oriented training program Upper management must lead quality program Structured annual improvements in quality

Defects can be Worker-controllable or Management

controllable

Study symptoms of defects and failures Develop a theory on the causes of the symptoms Test the theory until the cause is known Stimulate remedial action by appropriate action.

The worker is responsible if:

The worker knows what he is to do The worker knows the result of his work The worker has the means to control the results
24

Juran

Quality training program includes

Universal sequence of events for improving quality and reducing quality-related costs (creation of beneficial change) Universal feedback loop (prevention & adverse change) Fundamentals of data collection and analysis

25

Demming
14 points

Create constancy of purpose towards improvement of product and service Adopt the new philosophy Cease dependence on inspection to achieve quality build quality in, in the first place End the practice of awarding business on the basis of price tag - get single supplier for any one item. Instead minimize total cost Improve constantly and forever the system of production and service to improve quality and productivity - this constantly decreases costs

26

Demming
14 points

Institute training on the job Institute leadership. The aim of supervision is to help people to do a better job Drive out fear, so everyone may work effectively for the company Break down the barriers between departments - work in teams Eliminate slogans, and targets for the workforce asking for zero-defects and new levels of productivity. They create adversarial relationships. The bulk of the causes of low quality and low productivity belong to the system

27

Demming (cont)

Eliminate work standards and management by objectives - substitute leadership Remove barriers that rob workers/managers of the right to pride of workmanship - abolish annual merit rating Institute a vigorous program of education and self improvement Put everybody in the company to work to accomplish the transformation - the transformation is everybody's job

28

Demming Cycle
A common approach to managing problems. 5 Step Repetitive Cycle:
1. 2. 3. 4. 5. Plan Do Check Analyse Act

In software can be incorporated in every

phase of development process as well as to manage the process itself.

29

Quality Software
The International standard for Software Product Evaluation :
ISO-9126 released in 1991 lists six key factors that are important in the production of quality software. They are:

Functionality Reliability Usability Efficiency Maintainability Portability

30

Key Ideas
Process

Generally accepted that the process employed in

the development of a product crucial to the quality of the product.
Constructive Principle

Quality must be built into the product from the

outset. It can not be added on later.
People

Above all else it is people that determine

whether or not a quality product is produced.
31

Enemies of Quality Software

Faith in new technologies, methods, etc. as a panacea (The Quick Fix)

Quality return proportional to effort spent achieving quality

Lack of commitment to quality at all levels of

an organisation

Quality systems and standards produced and ignored

Culture Failure to identify and manage the risks to

quality

32

Quality Factors
The reality of issues related to the various

attributes of software and its use and maintenance are classified into content groups called quality factors

33

Software quality factors

The need for comprehensive software quality

requirements Classification of requirements into software quality factors Product operation factors Product revision factors Product transition factors Alternative models of software quality factors Who is interested in defining quality requirements? Software compliance with quality factors
34

McCalls quality factors model

Software quality factors

Product operation factors Product revision factors Product transition factors
35

Software quality factors

McCalls quality factor model
11

Quality factors grouped into 3 categories

Product operation factors: Correctness, Reliability, Efficiency, Integrity, Usability Product revision factors: Maintainability, Flexibility, Testability Product transition factors: Portability, Reusability, Interoperability
36

Product operation factors

Correctness:
Defined

in a list of the software systems required output

The output mission (e.g. red alarms when temperature rises to 100 C) Required accuracy of the output (e.g. non-accurate output will not exceed 1%) Completeness of the output info (e.g. probability of missing data less than 1%) The up-to-dateness of the info (e.g. it will take no more than 1s for the information to be updated) The availability of the info (e.g. reaction time for queries will be less than 2s on average) The required standards and guidelines (the software and its docs must comply with the clients guidelines)

37

Product operation factors (cont.)

Reliability:
Determines

the maximum allowed software system failure rate Can focus on entire system or just on a separate function E.g. a heart-monitors failure rate must be less than 1:20 years
38

Product operation factors (cont.)

Efficiency:
-Focus is on hardware resources needed to perform the operations to fulfill the requirements (memory, storage, CPU speed, etc.)

Integrity:
-Requirements to prevent access to unauthorized persons -Rights management (e.g. limit the write permit to key personnel)
39

Product operation factors (cont.)

Usability:
Deals

with the scope of staff resources needed to train a new employee and to operate the software system E.g. training of a new employee to operate the system will take no more than 2 working days (16h)

40

Product revision factors

Maintainability:
Determines

the effort needed to identify the causes for software failures, to correct them, and to verify the success of the corrections Refers to the modular structure of the software as well as to the manuals and documentations
41

Product revision factors (cont.)

Flexibility: - The effort required to support adaptive maintenance activities - E.g. man-days required to adapt a software package to a variety of customers of the same trade Testability:
-

Testability requirements include automatic diagnostics checks and log files, etc. E.g. a standard test must be run every morning before the production begins

42

Product transition factors

Portability:
- Adaptation of the system to other environments of different hardwares, OS, etc.

Reusability:
- Mostly the developer himself will initiate the reusability requirement by recognizing the potential benefit of a reuse - Its expected to save development resources, shorten the development time and provide higher quality modules
43

Product transition factors (cont.)

Interoperability:
Focuses

on developing interfaces with other software systems or with other equipment firmwares E.g a laboratory equipment is required to process its results (output) according to a standard data structure, which the laboratory information system can then use as an input
44

Additional factors
No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Software quality factor

Correctness Reliability Efficiency Integrity Usability Maintainability Flexibility Testability Portability Reusability Interoperability Verifiability Expandability Safety Manageability Survivability

McCalls classic model + + + + + + + + + + +

Alternative factor models

Evans and Marciniak model Deutsch and Willis model

+ + + + + + + + + + + +

+ + + + + + + + + + + + + + +
45

45

Developers interested in the definition of quality requirements

Reusability Portability Verifiability

46

47

48

49

50

Overview of SQA system components

Pre-project components Software project life cycle components Infrastructure components for error prevention and improvements Management SQA components SQA standards, system certification and assessment components Organizing for SQA the human components
51

Pre-project SQA components

Contract reviews The agreed-upon functional specification, budget and time between developer and client for the development of a system For the review the following need detail examination Project proposal The contract Contract review process Clarification of customer requirements Review of project schedule and resource requirement estimates Evaluation of staff competency Evaluation of customer capability Evaluation of Risks
52

Pre-project SQA components

Development and quality plans

Schedule Required manpower and hardware resources Risk evaluation Organization issues

Team members Sub-contractors

Project methodology Software reuse plans

53

Pre-project SQA components

Project quality plan

Quality goals in measurable terms Criteria for starting and ending each project stage List of reviews, tests, and other scheduled V&V activities

54

Software project lifecycle SQA components

Development life cycle stage and operation Maintenance stage Main components Reviews Expert opinions Software testing Software maintenance components Assurance of the quality of external participants work

55

Infrastructure SQA components: prevention & improvement

Procedures and work instruction Templates and checklists Staff training, retraining and certification Preventive and corrective actions Configuration management Documentation control

56

56

Management SQA components

Project progress control Software quality metrics Software quality costs

57

57

SQA standards, assessments & certifications

Project process standards Quality management standards Objectives:

Utilization of international professional knowledge Improvement of coordination with other organizations quality systems Objective professional evaluation and measurement of the organizations SQA achievement

58

SQA organization the human components

Managements role in SQA The SQA unit SQA trustees SQA committees SQA forums

59

Contract review process & stages

Proposal draft review This stage reviews the final proposal draft and the documents on which it is based : customer documents and customer's detailed explanations of the requirements, resource and financial estimates, existing contracts with partners and subcultures, etc. Contract draft review

This stage reviews the contract draft on the basis of the proposal and the understandings reached during the subsequent negotiations.
60

Objectives of Proposal review

The objectives of the proposal draft review are to make sure that the

following activities have been completed satisfactorily: Customer requirements have been clarified and documented. Alternatives for carrying out the project have been examined. A formal aspects relationship with the customer has been defined.

Development risks have been identified. Resources and schedules for the project have been adequately estimated. Examine the companys capacity to perform the project. Examine the customers capacity to fulfill his commitments. Definition of partner and subcontractor participation. Definition and protection of Proprietary rights.

61

Objectives of Draft Contract review

The objectives of the contract draft review are

to guarantee satisfactory completion of the following activities.

No un-clarified issues remain in the contract draft. All understandings subsequent to the proposal are correctly documented. No changes, additions, or omissions are to be found.
62

Implementation of Contract Review - Identify the factors that affect the extent of the contract review
The efforts to be expended on the contract review

depend on the characteristics of the project. The most important factors are the project:

magnitude Complexity the staffs acquaintance with and experience in the project area, and the number of additional organizations carrying out the project (partners, subcontractors, and the customer).

63

Identify the difficulties in performing a major contract review.

The main difficulties are:

Pressures of the time Need to invest substantial professional working hours when the contract review them members is already occupied by other commitments.

64

Recommended avenues for implementing a major contract review

To conduct a proper major contract review, one should abide by the following guidelines:
The contract review should be part of the proposal

preparation schedule. The contract review should be carried out by a team. A contract review leader should be appointed.

65

Importance of carrying out a contract review for internal projects

The looser relationships maintained between

the internal customer and the internal developer increase the probability of project failure. This trend can be reduced by adequate procedures that will define the preparation and by applying the same guidelines used for external project contract review.

66

Explain the objectives of development and quality plans

The plan's objectives are to provide the basis

for: Scheduling development activities. Recruiting team members and allocating development risks. Resoling development risks. Implementing required SQA activates. Providing management with needed date for project control.
67

Elements of a development plan

Eleven types of elements constitute a development plan. 1. Project Products. 2. Project interfaces. 3. Project methodology and development tools. 4. Software development standards and procedures. 5. mapping of the development process. 6. Project milestones. 7. Project staff organization. 8. Required development facilities. 9. Development risks. 10. Control methodology. 11. Project cost estimate.

68

Project products
The development plan includes the following products: Design documents specifying dates of completion, indicating those items to be delivered to the customer (deliverables) Software products (specifying completion date and installation site) Training tasks (specifying dates, participants and sites.)

69

Project interfaces
Project interfaces include: Interfaces with existing software packages

(software interface) Interfaces with other software and/or hardware development teams that are working on the same system or project (i.e., cooperation and coordination links). Interfaces with existing hardware (hardware interface).

70

Project methodology and development tools to be applied at each phase of the project.

71

Software development standards and procedures.

A list should be prepared of the software development standards and procedures to be applied in the project.

72

REVIEWS

73

Reviews in SQA
Sometimes called Formal Technical Review

(FTR), Formal Design Review, Inspection, Walkthrough, Peer Review, etc. Originally developed by Michael Fagan in the 1970s (IBM) Meeting technique: based on teamwork The goal is to find errors from basically any written document (specification, code, etc.)

74

Goals of reviews
Basic idea is to remove the errors in the early part of

the project Project is divided into intermediate stages/phases (makes the progression of the project more visible) Some basic objectives:

Uncover errors in functions, logic or implementation To verify that the document (software code, specification, etc.) meets its requirements To ensure that the software has been represented according to the predefined standards To make projects more manageable

75

Objectives of reviews
Direct objectives
a. To detect analysis and design errors as well as subjects where corrections, changes and completions are required b.To identify new risks likely to affect the project. c.To locate deviations from templates, style procedures and conventions. d. To approve the analysis or design product. Approval allows the team to continue on to the next development phase.
76

Objectives of reviews
Indirect objectives
a. To provide an informal meeting place for exchange of professional knowledge about methods, tools and techniques. b. To record analysis and design errors that will serve as a basis for future corrective actions.

77

Types of Reviews
Formal design reviews (FDRs) Peer reviews (inspections and walkthroughs) Expert opinions

78

Common Formal Design Reviews

DPR Development Plan Review SRSR Software Requirement Specification Review PDR Preliminary Design Review DDR Detailed Design Review DBDR Data Base Design Review TPR Test Plan Review STPR Software Test Procedure Review VDR Version Description Review OMR Operator Manual Review SMR Support Manual Review TRR Test Readiness Review PRR Product Release Review IPR Installation Plan Review

79

Formal Design Reviews - Process

Participants review leader and team Preparation Session Post review tasks
80

Design Review Leader

Knowledge and experience in development of projects of the type reviewed. Preliminary acquaintance with the current project is not necessary Seniority at a level similar if not higher than that of the project leader A good relationship with the project leader and his team A position external the project team
81

Design Review Team

Seniority Customer representative Developing team representative Size between 3 to 5 members

82

Design Review Preparation

Seniority Customer representative Developing team representative Size between 3 to 5 members

83

Design Review Session

A short presentation of the design document. Comments made by members of the review team. Verification and validation of comments is discussed to determine the required action items (corrections, changes and additions). Decisions about the design product (document), which determines the project's progress: Full approval. Partial approval. Denial of approval.

84

13 Rules for Successful design reviews

Design Review Infrastructure
1. Develop checklists for common types of design docs. 2. Train senior professionals to serve as a reservoir for DR teams. 3. Periodically analyze past DR effectiveness. 4. Schedule the DRs as part of the project plan.

The Design Review Team

5. Review teams size should be limited, with 35 members being the optimum.
85

13 Rules for Successful design reviews

The Design Review Session 6. Discuss professional issues in a constructive way dont personalize the issues. 7. Keep to the review agenda. 8. Focus on detection of defects by verifying and validating the participants' comments. Dont discuss possible solutions. 9. In cases of disagreement about an error - end the debate by noting the issue - shift its discussion to another forum. 10. Properly document the discussed comments, and the results of their verification and validation. 11. The duration of a review should not exceed two hours.
86

13 Rules for Successful design reviews

Post-Review Activities
12.

Prepare the review report, including the action items Establish follow-up to ensure the satisfactory performance of all action items

13.

87

Post design review activities

a. Preparation of the DR report. The report's major sections: A summary of the review discussions. The decision about continuation of the project. A full list of the required action items corrections, changes and additions. For each action item, completion date and project team member responsible are listed. The name(s) of the review team member(s) assigned to follow up. b. Follow up performance of the corrections 88 and to

Sign of worthless FDR

Short report limited to approvals without list of

detected defects Report with identification of defects with any action suggested Report with identification of action items but without no follow-up member

89

Organizing the review meeting

The amount of material to be inspected must

be reasonable (not too much material) No unfinished material Participants must have enough time to get to know the material beforehand The amount of participants must be kept as low as possible (no unnecessary people)

90

The actual review meeting

Roles: presenter (usually the producer himself),

moderator/review leader, secretary/recorder. Focus is on finding the problems rather than solving them Review the product, not the producer. Limit the amount of debate The more new errors found the more successful the meeting is

91

After the review meeting

Accept the product (no modifications

necessary) Reject the product (severe errors) Accept the product provisionally (small errors, new meeting not necessary) All the findings and conclusions are noted to the record (important)

92

Reviews: pros/cons
Most of the errors are found early in the

project (saves money) Producer has a better understanding of the correctness of his work. All the problems arent found just by testing: errors in phase products, style errors, unnecessary code, etc. Problems: causes extra work in the early stages of the project, organizing the inspection might be problematic, attitudes, unprepareness to the meeting
93

FDR Vs Peer Review

FDR Members superior than the project team Authority with the team to even reject the artifact Peer Review The members can be of the project team They are just to detect the defects

94

Peer Reviews Inspections

More formal

Walkthroughs
Just focus on errors of

Checklists Defect type frequency table Review for artifact errors Look at development methods Comprehensive Infrastructure Periodic analysis Schedule analysis

artifact and comments

95

Peer Reviews
Participants Prep Session Post-review tasks Efficiency Coverage

96

Participants of Peer Reviews Inspections

Review leader The author Specialized Review leader The author Specialized

Walkthrus

professionals:

professionals:
Standards enforcer Maintenance expert User representative

Designer Coder or implementer Tester

97

COMPARISON OF THE REVIEW METHODOLOGIES

Formal Design Reviews Properties Main direct objectives Main indirect objectives Review leader (1) Detect errors (2) Identify new risks (3) Approve the design document Knowledge exchange (1) Detect errors (2) Identify deviations from standards (1) Knowledge exchange (2) Support corrective actions Trained moderator (peer) Detect errors Inspections Walkthroughs Knowledge exchange

Chief software engineer or senior staff member Top-level staff and customer representatives Yes

Coordinator (peer, the project leader on occasion) Peers Yes; usually as the reviews initiator

Participants Project leader participation Specialized professionals in the team

Peers Yes

(1) Designer (1) Standards enforcer (2) Coder or implementer (2) Maintenance expert (3) Tester (3) User representative

98

Properties Overview meeting Participants preparations Review session Follow-up of corrections Formal training of participants Participants use of checklists Error-related data collection

Design review No Yes - thorough Yes Yes No No

Inspection Yes Yes - thorough Yes Yes Yes Yes

Walkthrough No Yes - brief Yes No No No Not formally required

Not formally required Formally required

Review documentation Formal design review 1) Inspection session report findings report 2) Inspection session summary report

99

 Insufficient in-house professional capabilities in a specialized area. Temporary lack of in-house professionals for review team. Indecisiveness caused by major disagreements among the organizations senior professionals. In small organizations, where the number of suitable candidates for a review team is insufficient.

100

Software Testing

101

Black box testing - IEEE

1.

Testing that ignores the internal mechanism of the system or component and focuses solely on the outputs in response to selected inputs and execution conditions

2.

Testing conducted to evaluate the compliance of a system or component with specified functional requirements

102

Software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units or an entire software package are examined by running the programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.
103

Direct objectives a. To identify and reveal as many errors as possible in the tested software b. To bring the tested software, after correction of the identified errors and retesting, to an acceptable level of quality c. To perform the required tests efficiently and effectively, within the limits budgetary and scheduling limitation. Indirect objectives a. To compile a record of software errors for use in error prevention (by corrective and preventive actions)

104

 Incremental testing

strategies:

Bottom-up testing Top-down testing

Big bang testing

105

Stage 4
Integration B

M11 Integration c M9 Integration A M10

Stage 3

Stage 2

M8

Stage 1

M1

M2

M3

M4

M5

M6

M7

106

Integration D Integration C Integration B

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6

M1 M2 M8

Integration A

M11 M10

M9

M6

M7

M3

M4

M5
107

Top-down testing of module M8

Bottom-up testing of module M8

M9

Module tested in an earlier stage Module on test

Drive of M9 M8 M1
Module on test Modules tested in an earlier stage
108

M8 Stub of M1

Stub of M2

M2

Big bang approach

In Big bang approach every module is unit

tested in isolation from other module and then each modules are combined all at a once and tested

109

Black box testing

1.

2.

Testing that ignores the internal mechanism of the system or component and focuses solely on the outputs in response to selected inputs and execution conditions Testing conducted to evaluate the compliance of a system or component with specified functional requirements Testing that takes into account the internal mechanism of a system or component
110

White box testing

Test Classification - Operation

FACTOR Correctness

White-box

Black-box

Accuracy/Documentation/Reaction time Calculations / Software Qualification

Reliability Efficiency Integrity Usability

Training Operation

111

Test Classification - Revision

FACTOR

White-box

Black-box

Maintainability Flexibility Testability

112

Test Classification - Transition

FACTOR

White-box

Black-box

Reusability Portability Interoperability

Software Equipment

113

White box testing

Testing that takes into account the internal mechanism of a system or component IEEE

Data Processing & calculation correctness Software Qualification Maintenance Reusability testing

114

White box testing - coverage

Path coverage
Path coverage of a test is measured by the percentage of all

possible program paths included in planned testing.

Line coverage
Line coverage of a test is measured by the percentage of

program code lines included in planned testing.

115

Module program flow chart

1 Charge the minimal fare D > 1000 3 WT > 3 6 S >1 2 Distance 5 Waiting time 8 No.of suitcases 11 Regular client? 15 Yes D 1000 4 WT 3 7 S1

10

12

Yes

No 14 Night journey?

13

No 16
116

17 Print receipt.

Module program flow graph

1 2 3 6 5 8 9 11 12 4 7 10 14 13 16
117

R1 R2 R3

R6

R4

15 17

R5

Minimum paths for full line coverage

1 2 3 6 5 8 9 11 12 4 7 10 14 13 16
118

R1 R2 R3

R6

R4 15
17

R5

Differences in coverage
Path coverage
Simple module required 24 tests to cover all paths

Basic path coverage = Line coverage

Required only 3 tests to cover all lines Test cases required is 1:8

This ratio increases with complexity of code to be tested !

119

McCabes Cyclomatic Complexity

Measures complexity of program Determines maximum no. of independent path to achieve full line

coverage
Draw program flow graph Independent path = Any path on the flow graph that includes at

least one edge that is not included in any former independent path.

120

McCabes Cyclomatic Complexity

V(G) = R V(G) = E-N+2 V(G) = P+1 R no. of regions (enclosed areas) in the program flow graph.

Also, area around the graph is a region E no. of edges in the graph N no. of nodes in the graph P no. of decisions in the graph (nodes with more than one leaving edge

121

Module program flow graph

1 2 3 6 5 8 9 11 12 4 7 10 14 13 16
122

R1 R2 R3

R6

R4

15 17

R5

Control Flow Testing

Statement coverage: The test cases are generated

so that all the program statements are executed at least once. Decision coverage (branch coverage): The test cases are generated so that the program decisions take the value true or false. Condition coverage: The test cases are generated so that all the program conditions (predicates) that form the logical expression of the decision take the value true or false. Path coverage: Test cases are generated to execute all/some program paths.
123

 The testing process

Determining the test methodology phase Planning the tests Test design Test implementation
124

Determining the test methodology Planning the tests Designing the tests Performing the tests (implementation)
125

1. 2. 3. 4. 5. 6. 7. 8.

Endangers the safety of human beings Affects an essential organizational function with no system replacement capability available Affects functioning of firmware, causing malfunction of an entire system Affects an essential organizational function but a replacement is available Affects proper functioning of software packages for business applications Affects proper functioning of software packages for a private customer Affects functioning of a firmware application but without affecting the entire system. Inconveniences the user but does not prevent accomplishment of the systems capabilities
126

1. Financial losses * Damages paid for physical injuries * Damages paid to organizations for malfunctioning of software * Purchase cost reimbursed to customers * High maintenance expanses for repair of failed systems 2. Non-quantitative damages * Expected to affect future sales * Substantially reduced current sales

127

Determining the test methodology Planning the tests Designing the tests Performing the tests (implementation)
128

Determining the test methodology Planning the tests Designing the tests Performing the tests (implementation)
129

1. 2. 3. 4. 5.

What to test ? Which sources to use for test cases ? Who is to perform the test ? Where to perform the test ? When to terminat the test ?

130

Module/application issues
1. 2. 3. Magnitude Complexity and difficulty Percentage of original software (vs. percentage of reused software) Professional qualifications Experience with the module's specific subject matter. Availability of professional support (backup of knowledgeable and experience). Acquaintance with the programmer and the ability to evaluate his/her capabilities.
131

Programmer issues
4. 5. 6. 7.

1. 2. 3. 4. 5.

What to test ? Which sources to use for test cases ? Who is to perform the test ? Where to perform the test ? When to terminate the test ?

132

1
1.1 1.2

Scope of the tests

The software package to be tested (name, version and revision) The documents that provide the basis for the planned tests

2
2.1 2.2 2.3 2.4 2.5

Testing environment
Sites Required hardware and firmware configuration Participating organizations Manpower requirements Preparation and training required of the test team

133

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8

Tests details (for each test)

Test identification Test objective Cross-reference to the relevant design document and the requirement document Test class Test level (unit, integration or system tests) Test case requirements Special requirements (e.g., measurements of response times, security requirements) Data to be recorded

4
4.1 4.2 4.3 4.4

Test schedule (for each test or test group) including time estimates for:
Preparation Testing Error correction Regression tests

134

Determining the test methodology Planning the tests Designing the tests Performing the tests (implementation)
135

1 Scope of the tests 1.1 The software package to be tested (name, version and revision) 1.2 The documents providing the basis for the designed tests (name and version for each document) 2 Test environment (for each test) 2.1 Test identification (the test details are documented in the STP) 2.2 Detailed description of the operating system and hardware configuration and the required switch settings for the tests 2.3 Instructions for software loading 3. Testing process 3.1 Instructions for input, detailing every step of the input process 3.2 Data to be recorded during the tests 4. Test cases (for each case) 4.1 Test case identification details 4.2 Input data and system settings 4.3 Expected intermediate results (if applicable) 4.4 Expected results (numerical, message, activation of equipment, etc.) 5. Actions to be taken in case of program failure/cessation 6. Procedures to be applied according to the test results summary

136

The mapping of the development process.

a) b) c)

Mapping of the development process involves providing detailed definitions of each of the projects phases. These descriptions include definitions of inputs outputs, and the specific activates planned . Activity descriptions include: An estimate of the activitys duration. These estimates are highly dependent on the experience gained in previous projects. The logical sequence in which each activity is to be performed , including description of each activitys dependence on previously completed activates . The type of professional resources required and estimates of how much of these resources are necessary for each activity. Several methods are available for scheduling and graphically presenting the development process. One of the most commonly used methods is .

137

Project milestones.
For each milestone, its completion time and project products (documents and code) are to be defined.

138

Project staff organization and coordination with external participants

The organization plan comprises: Organizational structure: definition of project teams and their tasks, including teams comprised of a subcontractors temporary workers. Professional requirements: professional certification, experience in a specific programming language or development tool, experience with a specific software product and type , and so forth. Number of team members required for each period of time, according to a scheduled . It is expected that teams will commence their activities at different times and that their team size may very form one period to the next , depending on the planned activates. Names of team leaders and team members. Difficulties are expected to arise with respect to the long term assignment of staff members to teams because of unanticipated changes in their current assignments. Therefore, the names of staff are required to help keep track of their participation as team members. Coordination with external participants: subcontractors, partners and customers departments that participate in the project (see Chapter 12).

139

Development facilities.
Required development facilities include

hardware , software and hardware development tools , office space, and other items. For each facility, the period required for its use should be indicated on the timetable.

140

Development risks.
Development risks are inherent in any project. To understand their pervasiveness, and how they can be controlled, we should first define the concept. A development risk is a state of property of a development task or averment, which, if ignored, will increase the likelihood of project failure. Typical development risk are: Technological gaps-Lack of adequate and sufficient professional knowledge and experience to carry out the demands of the development contract. Staff shortages-Unanticipated shortfalls of professional staff. Interdependence of organizational elements-The likelihood that suppliers of specialized hardware or software subcontractors for example, will not fulfill their obligations on scheduled.

141

Control methods.
In order to control project implementation , the project manager and the department management apply a series of monitoring practices when preparing progress report and coordination meetings. A comprehensive discussion of project control methods is found in Chapter 19.

142

Project cost estimation.

Project cost estimates are based on proposal costs estimates, followed by a through review of their continued relevance based on updated human resource estimate, contracts negotiated with subcontractors and suppliers, and so forth. For instance, part of the project, planned to be carried out by an internal development team , needs to be performed by a subcontractor, due to unavailability of the team. A change of this nature is usually involved in a substantial additional budget.

143

Elements of a quality plan.

Five the elements constitute a quality plan. 1. Quality goals. 2. Planned review activates. 3. Planned software tests. 4. Planned acceptance tests for externally developed software. 5. Planned configuration management.

144

Major software risk items

Typical development risks are: Technological gaps-lack of adequate and sufficient professional knowledge. Staff shortages. Interdependence on other oqanizations:suppliers,subcontractors,etc.

145

Process of software risk management

The activates involved in risk management

include:

Planning Implementation monitoring of implementation

The pertinent planning activates are: identification of SRls evaluation of those SRls planning RMAs to resolve the SRls,
146

Benefits of preparing development and quality plans for small project

For small development projects (of not less that 15 man-days ),

preparation of development and quality plans is optional However, one should consider the substantial advantages obtained by the plans developer The main advantages of plan preparation are improvements in the developers The understanding of the task greater commitment to complete the project as planned the plan documents contribute to a better understanding between the developer and the customer easier and more effective project control.

147

Benefits of preparing development and quality plans for internal project

It is recommended that internal projects, undertaken on behalf of other departments and for development of software packages geared toward the market , be treated as regulate projects . This implies that full-scale development and quality plans are to be prepared. Their benefits include:
a)

b)

c)

The development department will avoid losses incurred by unrealistic timetables and budgets, as well as the consequent damage to other projects and to the firm's reputation. The internal customer will enjoy reduced risk of late completion and budget overruns in addition to and by improved project control and coordination with the developer. The firm will enjoy reduced risk of its software products late entry into the market, reduced risk of a decline in its reputation resulting form late supply, and reduced risk of budget overruns.

148

Measuring Quality?
Users view:

Reliability (number of failures over time) Availability Usability etc. Often directly measurable

Manufacturers view:

Defect counts Rework costs

149

Reliability
Reliability is the probability of a component, or system,

functioning correctly over a given period of time under a given set of operating conditions. Quantitative: Reliability R(t) is the probability that the system will conform to its specification throughout a period of duration t. Note that it is important If a system only needs to operate for ten hours at a time, then that is the reliability target

150

Availability
The availability of a system is the probability

that the system will be functioning correctly at any given time. Quantitative:

Availability A (or V) is the percentage of time for which the system will conform to its specification.

Literally, readiness for service

151

Safety
Safety is a property of a system that it will not

endanger human life or the environment.

A safety-related system (safety-critical

system) in one by which the safety of equipment or plant is assured.

152

Maintainability
Maintenance is the action taken to retain a

system in, or return a system to, its designed operating conditions. Maintainability is the ability of a system to be maintained (ability to undergo repairs and modifications).

Mean time to repair (MTTR)

Problem: Maintenance induced failures

153

Software Quality Assurance?

Requirements:

Completeness is hard to achieve (complexity) Design: design for reliability only in special cases design for manufacturability is not required design for maintainability in special cases Focal area of software quality assurance! Manufacturing Implementation: limited success with statistical process control (metrics) Operation: Fixing found bugs

154

Quality Perspectives & Expectations

Transcendental view: It is generally associated with some intangible properties that delight users Abstracts and intangible properties Product view: the focus is on inherent characteristics in the product itself in the hope that controlling these internal quality indicators will result in improved external product behavior (quality in use). User view: quality is fitness for purpose or meeting users needs. Manufacturing view: quality means conformance to process standards Value-based view (Economic): quality is the customers willingness to pay for a software

155

Roles & Responsibilities

Consumers internally Externally non-human or invisible users as other software, embedded hardware Operational environment Producers

development, management, maintenance, marketing, and service of software products third-party participants who may be involved

156

Quality expectations on the consumer side

Correctness Performing right functionality as per specifications Reliability

performs these specified functions correctly over repeated use or over a long period of time

For many users ease of use, or usability, graphical user interfaces (GUI), command interpreters often used in mainframes is primarily driven by the usability ease of installation - plug-and-play

157

Quality expectations on the producer side

adherence to pre-selected software process

and relevant standards, proper choice of software methodologies, languages, and tools usability and modifiability, maintainability for maintenance profitability and customer value for product marketing.

158

Quality Frameworks ISO-9126

Functionality: A set of attributes that bear on the existence of a

set of functions and their specified properties. The functions are those that satisfy stated or implied needs. Suitability Accuracy Interoperability Security Reliability: A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time. The Maturity Fault tolerance Recoverability

159

Quality Frameworks ISO-9126

Usability: A set of attributes that bear on the effort needed for

use, and on the individual assessment of such use, by a stated or implied set of users. Understandability Learnability Operability Efficiency: A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used, under stated conditions. Time behavior Resource behavior

160

Quality Frameworks ISO-9126

Maintainability: A set of attributes that bear on the effort needed

to make specified modifications. Analyzability Changeability Stability Testability Portability: A set of attributes that bear on the ability of software to be transferred from one environment to another Adaptability Installability Conformance Replaceability

161

Alternative frameworks and focus on correctness

CUPRIMDS (capability, usability,

performance, reliability, installation, maintenance, documentation, and service) CUPRIMDS is often used together with overall customer satisfaction (thus the acronym CUPRIMDSO) to characterize and measure software quality for IBMs software products.

162

 Web-based applications Primary quality attributes as

Reliability Usability security, Availability Scalability Maintainability time to market

163

the secondary quality attributes as

CORRECTNESS AND DEFECTS

Failure: The inability of a system or component to perform its required functions within specified performance requirements. refers to a behavioral deviation from the user requirement or the product specification Fault: An incorrect step, process, or data definition in a computer program refers to an underlying condition within a software that causes certain failure(s) to occur Error: A human action that produces an incorrect result. refers to a missing or incorrect human action resulting in certain fault(s) being injected .into a software
164

Fault/Failure state transitions

165

Fault Classification
Nature (Critical distinction):

random/hardware faults logical/systematic/design faults E.g., Degradation (wear-out) vs. design faults

Duration:

permanent, transient, intermittent localized, global

166

Extent:

Dealing With Faults

Fault Prevention: Development techniques that prevent the introduction of faults Fault Elimination: Development techniques that find and remove faults already introduced Fault Forecasting: Estimate current number, future incidence and likely consequences Fault Tolerance (not considered here): Execution-time techniques that cope with the effects of faults

167

Main causes for software faults:

Faulty requirements definition Client-developer communication failures Deliberate deviations from software requirements Logical design errors Coding errors Non-compliance with documentation and coding instructions Shortcomings of the testing process User interface and procedure errors Documentation errors

168

Scope of Quality Management

Quality management is particularly important

for large, complex systems. The quality documentation is a record of progress and supports continuity of development as the development team changes. For smaller systems, quality management needs less documentation and should focus on establishing a quality culture.

169

Quality Management and Software Development

170

Quality Software Engineering

Cost Schedule Functionality Stages Functional

the focus was on providing the automated functions to replace the focus was on introducing important features and new systhe focus was on reducing the price to stay competitive accompanied by the widespread use of personal computers. the focus was managing users quality expectations under the increased dependency on software and high cost or severe damages associated with software failures.
171

Schedule

Cost

Reliability

The SQA system

Goal is to minimize the number of software errors

and to achieve an acceptable level of software quality Can be divided into to six classes:

Pre-project components Components of project life cycle activities assessment Components of infrastructure error prevention and improvement Components of software quality management Components of standardization, certification, and SQA system assessment Organizing for SQA-the human components

172

Pre-project components
The schedule and budget as well as other

project commitments are adequately planned Must assure that development and quality plans are correctly determined

173

Components of project life cycle activities assessment

Two phases:
-Development life cycle (verification-validation-qualification, reviews, expert opinions, software testing) -Operation-maintenance stage (Special maintenance components and life cycle components for improving maintenance tasks)

Assuring the quality of parts made by

subcontractors and other external participants during development and maintenance phases

174

Components of infrastructure error prevention and improvement

Goals are to lower software fault rates and to

improve productivity Procedures and work instructions, staff training, configuration management, documentation control, etc. Applied throughout the entire organization

175

Components of software quality management

Major goals are to control development and

maintenance activities and early managerial support (minimize schedule and budget failures) Software quality metrics, quality cost, project progress control, etc.

176

Components of standardization, certification, and SQA system assessment

Implements international, professional and

managerial standards within the organization Quality management standards: focuses on what is required in regards of managerial quality system (e.g. ISO 9001, SEI CMM assessment standard) Project process standards: methodological guidelines (how) for the development team (e.g. IEEE 1012, ISO/IEC 12207)

177

Organizing for SQA the human component

The organizational base: managers, testing

personnel, SQA unit, etc. To develop and support the implementation of SQA components To detect deviations from SQA procedures and methodology To suggest improvements to SQA components

178