Software Quality Assurance: walkthroughs, inspections, formal verication, and testing

IEEE 610.12 denes the terms mistake, fault, failure, and error. The last three are generically indicated by the word defect. The main goal of Software Quality Assurance (SQA) is to eliminate defects from the software. For software, quality implies adherence to the specications, but is this enough? In a large enough company, it is the job of the SQA group to ensure that the product developed (by the developers) is correct. The term (independent) validation and verication, or IV&V is often used. Note the dierence between validation (Are we building the right product?) and verication (Are we building the product right?). Why independent? In a large enough development environment, managerial independence is fundamental. In a small group, the best approximation is to test other peoples design and code (this is also a good way to get an understanding of the entire project). Walkthroughs A walkthrough is performed by a team consisting of four to six people. For example, for the specication walkthrough: at least one person from the team that created the specications, the technical manager, a client representative, at least one person from the team for the next activity (design in this case), and a person from the SQA team (who chairs the meeting). The goal of the walkthrough is to nd and record faults, not to correct them (correction-bycommittee is not as eective, costs more, and might go overboard). There are two ways to run a walkthrough. In any case: Distribute the material well in advance of the meeting. Each reviewer reads it and prepares two lists: items that require explanations, and items that seem incorrect. then, at the meeting, if it is participant driven: Reviewers present their lists in turn. The representative of the team that created the document claries the items that require explanations, and either agrees that items are incorrect, or explains why they are not. or, if it is document driven: The representative of the team that created the document walks through the document. Each reviewer interrupts as needed to bring up an item of concern. It is essential to have an interactive meeting, and to keep ones ego in check. The manager responsible for employee evaluations (i.e., rai$e$) should not be present. Inspections Inspections are more formal than walkthroughs. They consists of ve steps: An overview is given by a member of the team that created the document. At the end, the document is distributed.

 In the preparation step, each participant studies the document, consider lists of faults discovered in similar previous inspections, and takes notes. Then, the actual inspection meeting begins, with a team member walking through the document (again, the goal is to uncover faults, not to x them). Within a day after the meeting, the moderator produces a written report. The rework is performed by the team that created the document, to take into account all the items in the report. Finally, in the follow-up, the moderator ensures that every item raised at the meeting has been resolved. In addition, xes must be checked to ensure that no new faults have been introduced. If more than 5% of the document has been reworked, a new full inspection of the entire document is called for. Note: it must be decided in advance how to measure this fraction. Inspections teams should have four members. For example, for a design inspection, there must be a moderator, a designer, a coder, and a tester. The moderator is the leader of the team. The roles of reader and recorder would be played by the designer and the coder or the tester in this case. Walkthrough and, even more so, inspections, have a good chance of nding faults early in the development. However, for large projects, they require that the work products be small enough (reviews should not last many hours) and, unless good documentation is available at all steps, they are not eective at discovering problems. Formal verication Used to prove the correctness of certain complex algorithms, the adherence of a design to the specication, or even of an implementation to a design or specication. Due to its high manpower costs and limited applicability, it is usually employed in some form for certain aspects of highlycritical systems, but less so in commercial software (although the state-of-the-art is continually improving). Testing Testing is performed on the actual code (e.g., unit testing). While by necessity incomplete (i.e., cannot be used to prove correctness), testing is a fact of life. It is very good practice to develop test cases along, or even before, specication, design, and implementation. Indeed, one view is that such test cases are part of the specication, design, and implementation documents. Then, large-scale integration testing is performed after coding (or at least a portion of it) has been completed. Finally, for one-of-a-kind custom software projects, acceptance testing is performed by the customer prior to agreeing that the project has been properly completed (and prior to payment in full).