Advanced Testing of Systems of Systems Volume 2 Practical Aspects 1St Edition Bernard Homes Full Chapter

Advanced Testing of
Systems-of-Systems, Volume 2 -
Practical Aspects 1st Edition Bernard
Homes
Visit to download the full and correct content document:
https://ebookmass.com/product/advanced-testing-of-systems-of-systems-volume-2-pr
actical-aspects-1st-edition-bernard-homes/
Advanced Testing of Systems-of-Systems 2
Advanced Testing of
Systems-of-Systems 2
Practical Aspects
Bernard Homès
First published 2022 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd John Wiley & Sons, Inc.

27-37 St George’s Road 111 River Street
London SW19 4EU Hoboken, NJ 07030
UK USA
www.iste.co.uk www.wiley.com
© ISTE Ltd 2022

The rights of Bernard Homès to be identified as the author of this work have been asserted by him in
accordance with the Copyright, Designs and Patents Act 1988.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the
author(s), contributor(s) or editor(s) and do not necessarily reflect the views of ISTE Group.
Library of Congress Control Number: 2022944148
British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library
ISBN 978-1-78630-750-7
Contents
Dedication and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Chapter 1. Test Project Management . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1. General principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1. Quality of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2. Completeness of deliveries . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3. Availability of test environments . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.4. Availability of test data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.5. Compliance of deliveries and schedules . . . . . . . . . . . . . . . . . . . 5
1.1.6. Coordinating and setting up environments . . . . . . . . . . . . . . . . . . 6
1.1.7. Validation of prerequisites – Test Readiness Review (TRR) . . . . . . . . 6
1.1.8. Delivery of datasets (TDS) . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.9. Go-NoGo decision – Test Review Board (TRB) . . . . . . . . . . . . . . . 7
1.1.10. Continuous delivery and deployment . . . . . . . . . . . . . . . . . . . . 8
1.2. Tracking test projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3. Risks and systems-of-systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4. Particularities related to SoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5. Particularities related to SoS methodologies . . . . . . . . . . . . . . . . . . . 11
1.5.1. Components definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5.2. Testing and quality assurance activities . . . . . . . . . . . . . . . . . . . . 12
1.6. Particularities related to teams . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Chapter 2. Testing Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1. Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2. Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1. Project WBS and planning . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
vi Advanced Testing of Systems-of-Systems 2
2.3. Control of test activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.4. Analyze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.6. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.7. Test execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.8. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.10. Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.11. Infrastructure management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.12. Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.13. Adapting processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.14. RACI matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.15. Automation of processes or tests . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.15.1. Automate or industrialize? . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.15.2. What to automate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.15.3. Selecting what to automate . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Chapter 3. Continuous Process Improvement . . . . . . . . . . . . . . . . . . . 37

3.1. Modeling improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.1.1. PDCA and IDEAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.1.2. CTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.1.3. SMART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2. Why and how to improve? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.3. Improvement methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3.1. External/internal referential . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4. Process quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.1. Fault seeding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.2. Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.3. A posteriori . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4.4. Avoiding introduction of defects . . . . . . . . . . . . . . . . . . . . . . . 47
3.5. Effectiveness of improvement activities . . . . . . . . . . . . . . . . . . . . . . 48
3.6. Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Chapter 4. Test, QA or IV&V Teams. . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.1. Need for a test team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.2. Characteristics of a good test team . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3. Ideal test team profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4. Team evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.1. Skills assessment table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.4.2. Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.4.3. Select, hire and retain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.5. Test manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Contents vii
4.5.1. Lead or direct? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.5.2. Evaluate and measure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.5.3. Recurring questions for test managers . . . . . . . . . . . . . . . . . . . . 62
4.6. Test analyst . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.7. Technical test analyst . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.8. Test automator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.9. Test technician . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.10. Choose our testers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.11. Training, certification or experience?. . . . . . . . . . . . . . . . . . . . . . . 67
4.12. Hire or subcontract? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.12.1. Effective subcontracting . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.13. Organization of multi-level test teams . . . . . . . . . . . . . . . . . . . . . . 68
4.13.1. Compliance, strategy and organization . . . . . . . . . . . . . . . . . . . 69
4.13.2. Unit test teams (UT/CT) . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.13.3. Integration testing team (IT) . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.13.4. System test team (SYST) . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.13.5. Acceptance testing team (UAT) . . . . . . . . . . . . . . . . . . . . . . . 71
4.13.6. Technical test teams (TT) . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.14. Insourcing and outsourcing challenges . . . . . . . . . . . . . . . . . . . . . . 72
4.14.1. Internalization and collocation . . . . . . . . . . . . . . . . . . . . . . . . 72
4.14.2. Near outsourcing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.14.3. Geographically distant outsourcing . . . . . . . . . . . . . . . . . . . . . 74
Chapter 5. Test Workload Estimation . . . . . . . . . . . . . . . . . . . . . . . . 75

5.1. Difficulty to estimate workload. . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2. Evaluation techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.1. Experience-based estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.2.2. Based on function points or TPA . . . . . . . . . . . . . . . . . . . . . . . 77
5.2.3. Requirements scope creep . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2.4. Estimations based on historical data. . . . . . . . . . . . . . . . . . . . . . 80
5.2.5. WBS or TBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.2.6. Agility, estimation and velocity . . . . . . . . . . . . . . . . . . . . . . . . 81
5.2.7. Retroplanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.2.8. Ratio between developers – testers . . . . . . . . . . . . . . . . . . . . . . 82
5.2.9. Elements influencing the estimate . . . . . . . . . . . . . . . . . . . . . . . 83
5.3. Test workload overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3.1. Workload assessment verification and validation . . . . . . . . . . . . . . 86
5.3.2. Some values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4. Understanding the test workload . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4.1. Component coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4.2. Feature coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.4.3. Technical coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
viii Advanced Testing of Systems-of-Systems 2
5.4.4. Test campaign preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

5.4.5. Running test campaigns . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.4.6. Defects management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.5. Defending our test workload estimate . . . . . . . . . . . . . . . . . . . . . . . 91
5.6. Multi-tasking and crunch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.7. Adapting and tracking the test workload. . . . . . . . . . . . . . . . . . . . . . 92
Chapter 6. Metrics, KPI and Measurements . . . . . . . . . . . . . . . . . . . . 95

6.1. Selecting metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6.2. Metrics precision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.2.1. Special case of the cost of defaults . . . . . . . . . . . . . . . . . . . . . . 97
6.2.2. Special case of defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.2.3. Accuracy or order of magnitude? . . . . . . . . . . . . . . . . . . . . . . . 98
6.2.4. Measurement frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2.5. Using metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2.6. Continuous improvement of metrics . . . . . . . . . . . . . . . . . . . . . 100
6.3. Product metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.3.1. FTR: first time right . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.3.2. Coverage rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.3.3. Code churn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.4. Process metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.4.1. Effectiveness metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.4.2. Efficiency metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.5. Definition of metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.5.1. Quality model metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.6. Validation of metrics and measures . . . . . . . . . . . . . . . . . . . . . . . . 110
6.6.1. Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.6.2. Historical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.6.3. Periodic improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.7. Measurement reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.7.1. Internal test reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.7.2. Reporting to the development team . . . . . . . . . . . . . . . . . . . . . . 114
6.7.3. Reporting to the management . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.7.4. Reporting to the clients or product owners . . . . . . . . . . . . . . . . . . 115
6.7.5. Reporting to the direction and upper management . . . . . . . . . . . . . . 116
Chapter 7. Requirements Management . . . . . . . . . . . . . . . . . . . . . . . 119

7.1. Requirements documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.2. Qualities of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.3. Good practices in requirements management . . . . . . . . . . . . . . . . . . . 122
7.3.1. Elicitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.3.2. Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Contents ix
7.3.3. Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

7.3.4. Approval and validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.3.5. Requirements management . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.3.6. Requirements and business knowledge management . . . . . . . . . . . . 125
7.3.7. Requirements and project management . . . . . . . . . . . . . . . . . . . . 125
7.4. Levels of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.5. Completeness of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.5.1. Management of TBDs and TBCs . . . . . . . . . . . . . . . . . . . . . . . 126
7.5.2. Avoiding incompleteness. . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.6. Requirements and agility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.7. Requirements issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Chapter 8. Defects Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

8.1. Defect management, MOA and MOE . . . . . . . . . . . . . . . . . . . . . . . 129
8.1.1. What is a defect? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
8.1.2. Defects and MOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.1.3. Defects and MOE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.2. Defect management workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.2.1. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.2.2. Simplify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
8.3. Triage meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
8.3.1. Priority and severity of defects . . . . . . . . . . . . . . . . . . . . . . . . 133
8.3.2. Defect detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
8.3.3. Correction and urgency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
8.3.4. Compliance with processes . . . . . . . . . . . . . . . . . . . . . . . . . . 136
8.4. Specificities of TDDs, ATDDs and BDDs . . . . . . . . . . . . . . . . . . . . 136
8.4.1. TDD: test-driven development. . . . . . . . . . . . . . . . . . . . . . . . . 136
8.4.2. ATDD and BDD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
8.5. Defects reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
8.5.1. Defects backlog management . . . . . . . . . . . . . . . . . . . . . . . . . 139
8.6. Other useful reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.7. Don’t forget minor defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Chapter 9. Configuration Management . . . . . . . . . . . . . . . . . . . . . . . 143

9.1. Why manage configuration? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
9.2. Impact of configuration management . . . . . . . . . . . . . . . . . . . . . . . 144
9.3. Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.4. Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.5. Organization and standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
9.6. Baseline or stages, branches and merges . . . . . . . . . . . . . . . . . . . . . 147
9.6.1. Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
9.6.2. Branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
x Advanced Testing of Systems-of-Systems 2
9.6.3. Merge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

9.7. Change control board (CCB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
9.8. Delivery frequencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
9.9. Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
9.10. Version management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
9.11. Delivery management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
9.11.1. Preparing for delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
9.11.2. Delivery validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
9.12. Configuration management and deployments . . . . . . . . . . . . . . . . . . 155
Chapter 10. Test Tools and Test Automation . . . . . . . . . . . . . . . . . . . 157

10.1. Objectives of test automation . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
10.1.1. Find more defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
10.1.2. Automating dynamic tests . . . . . . . . . . . . . . . . . . . . . . . . . . 159
10.1.3. Find all regressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
10.1.4. Run test campaigns faster . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.2. Test tool challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.2.1. Positioning test automation . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.2.2. Test process analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.2.3. Test tool integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.2.4. Qualification of tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
10.2.5. Synchronizing test cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
10.2.6. Managing test data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
10.2.7. Managing reporting (level of trust in test tools). . . . . . . . . . . . . . . 165
10.3. What to automate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
10.4. Test tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
10.4.1. Selecting tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
10.4.2. Computing the return on investment (ROI) . . . . . . . . . . . . . . . . . 169
10.4.3. Avoiding abandonment of tools and automation . . . . . . . . . . . . . . 169
10.5. Automated testing strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
10.6. Test automation challenge for SoS . . . . . . . . . . . . . . . . . . . . . . . . 171
10.6.1. Mastering test automation . . . . . . . . . . . . . . . . . . . . . . . . . . 171
10.6.2. Preparing test automation . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
10.6.3. Defect injection/fault seeding . . . . . . . . . . . . . . . . . . . . . . . . 173
10.7. Typology of test tools and their specific challenges . . . . . . . . . . . . . . . 174
10.7.1. Static test tools versus dynamic test tools . . . . . . . . . . . . . . . . . . 175
10.7.2. Data-driven testing (DDT) . . . . . . . . . . . . . . . . . . . . . . . . . . 176
10.7.3. Keyword-driven testing (KDT). . . . . . . . . . . . . . . . . . . . . . . . 176
10.7.4. Model-based testing (MBT) . . . . . . . . . . . . . . . . . . . . . . . . . 177
10.8. Automated regression testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.8.1. Regression tests in builds . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.8.2. Regression tests when environments change . . . . . . . . . . . . . . . . 179
Contents xi
10.8.3. Prevalidation regression tests, sanity checks and smoke tests . . . . . . . 179
10.8.4. What to automate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
10.8.5. Test frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
10.8.6. E2E test cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.8.7. Automated test case maintenance or not? . . . . . . . . . . . . . . . . . . 184
10.9. Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
10.9.1. Automated reporting for the test manager . . . . . . . . . . . . . . . . . . 186
Chapter 11. Standards and Regulations . . . . . . . . . . . . . . . . . . . . . . . 187

11.1. Definition of standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
11.2. Usefulness and interest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
11.3. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
11.4. Demonstration of compliance – IADT . . . . . . . . . . . . . . . . . . . . . . 190
11.5. Pseudo-standards and good practices . . . . . . . . . . . . . . . . . . . . . . . 191
11.6. Adapting standards to needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
11.7. Standards and procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
11.8. Internal and external coherence of standards. . . . . . . . . . . . . . . . . . . 192
Chapter 12. Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

12.1. Case study: improvement of an existing complex system . . . . . . . . . . . 195
12.1.1. Context and organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
12.1.2. Risks, characteristics and business domains . . . . . . . . . . . . . . . . 198
12.1.3. Approach and environment . . . . . . . . . . . . . . . . . . . . . . . . . . 200
12.1.4. Resources, tools and personnel . . . . . . . . . . . . . . . . . . . . . . . . 210
12.1.5. Deliverables, reporting and documentation . . . . . . . . . . . . . . . . . 212
12.1.6. Planning and progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
12.1.7. Logistics and campaigns . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
12.1.8. Test techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12.1.9. Conclusions and return on experience . . . . . . . . . . . . . . . . . . . . 218
Chapter 13. Future Testing Challenges . . . . . . . . . . . . . . . . . . . . . . . 223

13.1. Technical debt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
13.1.1. Origin of the technical debt . . . . . . . . . . . . . . . . . . . . . . . . . . 224
13.1.2. Technical debt elements . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
13.1.3. Measuring technical debt . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
13.1.4. Reducing technical debt. . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
13.2. Systems-of-systems specific challenges . . . . . . . . . . . . . . . . . . . . . 228
13.3. Correct project management. . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
13.4. DevOps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
13.4.1. DevOps ideals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
13.4.2. DevOps-specific challenges . . . . . . . . . . . . . . . . . . . . . . . . . 231
13.5. IoT (Internet of Things) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
xii Advanced Testing of Systems-of-Systems 2
13.6. Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

13.7. Services and microservices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
13.8. Containers, Docker, Kubernetes, etc. . . . . . . . . . . . . . . . . . . . . . . . 235
13.9. Artificial intelligence and machine learning (AI/ML). . . . . . . . . . . . . . 235
13.10. Multi-platforms, mobility and availability . . . . . . . . . . . . . . . . . . . 237
13.11. Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
13.12. Unknown dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
13.13. Automation of tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
13.13.1. Unrealistic expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
13.13.2. Difficult to reach ROI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
13.13.3. Implementation difficulties . . . . . . . . . . . . . . . . . . . . . . . . . 242
13.13.4. Think about maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . 243
13.13.5. Can you trust your tools and your results? . . . . . . . . . . . . . . . . . 244
13.14. Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
13.15. Blindness or cognitive dissonance . . . . . . . . . . . . . . . . . . . . . . . . 245
13.16. Four truths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
13.16.1. Importance of Individuals . . . . . . . . . . . . . . . . . . . . . . . . . . 247
13.16.2. Quality versus quantity . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
13.16.3. Training, experience and expertise . . . . . . . . . . . . . . . . . . . . . 248
13.16.4. Usefulness of certifications . . . . . . . . . . . . . . . . . . . . . . . . . 248
13.17. Need to anticipate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
13.18. Always reinvent yourself . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
13.19. Last but not least . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
Summary of Volume 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Dedication and Acknowledgments
Inspired by a dedication from Boris Beizer1, I dedicate these two books

to many very bad projects on software and systems-of-systems development
where I had the opportunity to – for a short time – act as a consultant. These taught
me multiple lessons on difficulties that these books try and identify and led me to
realize the need for this book. Their failure could have been prevented; may they rest
in peace.
I would also like to thank the many managers and colleagues I had the privilege
of meeting during my career. Some, too few, understood that quality is really
everyone’s business. We will lay a modest shroud over the others.
Finally, paraphrasing Isaac Newton, If I was able to reach this level of

knowledge, it is thanks to all the giants that were before me and on the shoulders of
which I could position myself. Among these giants, I would like to mention (in
alphabetical order) James Bach, Boris Beizer, Rex Black, Frederic Brooks, Hans
Buwalda, Ross Collard, Elfriede Dustin, Avner Engel, Tom Gilb, Eliahu Goldratt,
Dorothy Graham, Capers Jones, Paul Jorgensen, Cem Kaner, Brian Marick, Edward
Miller, John Musa, Glenford Myers, Bret Pettichord, Johanna Rothman, Gerald
Weinberg, James Whittaker and Karl Wiegers.
After 15 years in software development, I had the opportunity to focus on

software testing for over 25 years. Specialized in testing process improvements, I
founded and participated in the creation of multiple associations focused on software
testing: AST (Association of Software Tester), ISTQB (International Software
1 Beizer, B. (1990). Software Testing Techniques, 2nd edition. ITP Media.

xiv Advanced Testing of Systems-of-Systems 2
Testing Qualification Board), CFTL (Comité Français des Tests Logiciels, the
French Software Testing committee) and GASQ (Global Association for Software
Quality). I also dedicate these books to you, the reader, so that you can improve your
testing competencies.
Preface
Implementation
In the first part of these two books on systems-of-systems testing, we identified

the impacts of software development cycles, testing strategies and methodologies,
and we saw the benefit of using a quality referential and the importance of test
documentation and reporting. We have identified the impact of test levels and test
techniques, whether we are talking about static techniques or dynamic techniques.
We ended with an approach to test project management that allowed us to identify
that human actor and how their interactions are essential elements that must be
considered.
In this second part of the book on systems-of-systems testing, we will focus on

more practical aspects such as managing test projects, testing processes and how to
improve them continuously. We will see the additional but necessary processes such
as the management of requirements, defects and configurations, and we will also see
a case study allowing us to ask ourselves several useful questions. We will end with
a perilous prediction exercise by listing the challenges that tests will have to face in
the years to come.
August 2022
1
Test Project Management
We do not claim to replace the many contributions of illustrious authors on good

practices in project management. Standards such as PMBOK (PMI 2017) or CMMI
and methodologies such as ITIL and PRINCE2 comprehensively describe the tasks,
best practices and other activities recommended to properly manage projects. We
focus on certain points associated with the testing of software, components, products
and systems within systems-of-systems projects.
At the risk of writing a tautology, the purpose of project management is to

manage projects, that is, to define the tasks and actions necessary to achieve the
objectives of these projects. The purpose, the ultimate objective of the project, takes
precedence over any other aspect, even if the budgetary and time constraints are
significant. To limit the risks associated with systems-of-systems, the quality of the
deliverables is very important and therefore tests (verifications and validations that
the object of the project has been achieved) are necessary.
Project management must ensure that development methodologies are correctly

implemented (see Chapter 2) to avoid inconsistencies. Similarly, project
management must provide all stakeholders with an image of the risks and the
progress of the system-of-systems, its dependencies and the actions to be taken in
the short and medium term, in order to anticipate the potential hazards.
1.1. General principles
Management of test projects, whether on components, products, systems or

systems-of-systems, has a particularity that other projects do not have: they depend
– for their deadlines, scope and level of quality – on other parts of the projects: the
development phases. Requirements are often unstable, information arrives late,
deadlines are shorter because they depend on evolving developments and longer
2 Advanced Testing of Systems-of-Systems 2
deadlines, the scope initially considered increases, the level of quality of input data –
requirements, components to be tested, interfaces – is often of lower quality than
expected and the number of faults or anomalies is greater than anticipated. All of
these are under tighter budgetary and calendar constraints because, even if the
developments take longer than expected, the production launch date is rarely
postponed.
The methodologies offered by ITIL, PRINCE2, CMMI, etc. bring together a set
of good practices that can be adapted – or not – to our system-of-systems project.
CMMI, for example, does not have test-specific elements (only IVV), and it may be
necessary to supplement CMMI with test-specific tasks and actions as offered by
TMM and TMMI.
Let us see the elements specific to software testing projects.
1.1.1. Quality of requirements
Any development translates requirements (needs or business objectives) into a

component, product or system that will implement them. In an Agile environment,
requirements are defined in the form of User Stories, Features or Epics. The
requirements can be described in so-called specification documents (e.g. General
Specifications Document or Detailed Specifications Document). Requirements are
primarily functional – they describe expected functionality – but can be technical or
non-functional. We can classify the requirements according to the quality
characteristics they cover as proposed in Chapter 5 of Volume 1 (Homès 2022a).
Requirements are provided to development teams as well as test teams.

Production teams – design, development, etc. – use these requirements to develop
components, products or systems and may propose or request adaptations of these
requirements. Test teams use requirements to define, analyze and implement, or
even automate, test cases and test scenarios to validate these requirements. These
test teams must absolutely be informed – as soon as possible – of any change in the
requirements to proceed with the modifications of the tests.
The requirements must be SMART, that is:

– Specific: the requirements must be clear, there must be no ambiguity and the
requirements must be simple, consistent and with an appropriate level of detail.
– Measurable: it must be possible, when the component, product or system is
designed, to verify that the requirement has been met. This is directly necessary for
the design of tests and metrics to verify the extent to which requirements are met.
Test Project Management 3
– Achievable: the requirements must be able to be physically demonstrated

under given conditions. If the requirements are not achievable (e.g. the system will
have 100% reliability and 100% availability), the result will be that the component,
product or system will never be accepted or will be cost-prohibitive. Achievable
includes that the requirement can be developed in a specific time frame.
– Realistic: in the context of software development – and testing – is it possible
to achieve the requirement for the component, product or system, taking into
account the constraints in which the project is developed? We add to this aspect the
notion of time: are the requirements achievable in a realistic time?
– Traceable: requirements traceability is the ability to follow a requirement from
its design to its specification, its realization and its implementation to its test, as well
as in the other direction (from the test to the specification). This helps to understand
why a requirement was specified and to ensure that each requirement has been
correctly implemented.
1.1.2. Completeness of deliveries
The completeness of the software, components, products, equipment and systems

delivered for the tests is obviously essential. If the elements delivered are
incomplete, it will be necessary to come back to them to modify and complete them,
which will increase the risk of introducing anomalies.
This aspect of completeness is ambiguous in incremental and iterative

methodologies. On the one hand, it is recommended to deliver small increments, and
on the other hand, losses should be eliminated. Small increments imply partial
releases of functionality, thus generation of “losses” both regarding releases and
testing (e.g. regression testing) – in fact, all the expectations related to these multiple
releases and multiple test runs – to be performed on these components. Any
evolution within the framework of an iteration will lead to a modification in the
functionalities and therefore an evolution compared to the results executed during
the previous iterations.
1.1.3. Availability of test environments
The execution of the tests is carried out in different test environments according
to the test levels envisaged. It will therefore be necessary to ensure the availability
of environments for each level.
The test environment is not limited to a machine on which the software

component is executed. It also includes the settings necessary for the proper
execution of the component, the test data and other applications – in the appropriate
versions – with which the component interacts.
Test environments, as well as their data and the applications they interface with
must be properly synchronized with each other. This implies an up-to-date definition
of the versions of each system making up the system-of-systems and of the
interfaces and messages exchanged between them.
Automating backups and restores of test environments allows testers to self-

manage their environments so that they are not a burden on production systems
management teams.
In DevOps environments, it is recommended to enable automatic creation of

environments to test builds as they are created by developers. As proposed by Kim
et al. (2016), it is necessary to allow to recreate – automatically – the test
environments rather than trying to repair them. This automatic creation solution
ensures an identical test environment to the previous version, which will facilitate
regression testing.
1.1.4. Availability of test data
It is obvious that the input test data of a test case and the expected data at the
output of a test case are necessary, and it is also important to have a set of other data
that will be used for testing:
– data related to the users who will run the tests (e.g. authorization level,
hierarchical level, organization to which they are attached, etc.);
– information related to the test data used (e.g. technical characteristics,
composition, functionalities present, etc.) and which are grouped in legacy systems
interfaced with the system-of-systems under test;
– historical information allowing us to make proposals based on this historical
information (e.g. purchase suggestions based on previous purchases);
– information based on geographical positioning (e.g. GPS position), supply
times and consumption volumes to anticipate stock replenishment needs (e.g. need
to fill the fuel tank according to the way to drive and consume fuel, making it
possible to offer – depending on the route and GPS information – one or more
service stations nearby);
– etc.
The creation and provision of quality test data is necessary before any test
campaign. Designing and updating this data, ensuring that it is consistent, is
extremely important because it must – as far as possible – simulate the reality of the
exchanges and information of each of the systems of the system-of-systems to be
tested. We will therefore need to generate data from monitoring systems (from
sensors, via IoT systems) and ensure that their production respects the expected
constraints (e.g. every n seconds, in order to identify connection losses or deviations
from nominal operating ranges).
Test data should be realistic and consistent over time. That is, they must either
simulate a reference period and each of the campaigns must ensure that the systems
have modified their reference date (e.g. use a fixed range of hours and reset systems
at the beginning of this range) or be consistent with the time of execution of the test
campaign. This last solution requires generating the test data during the execution of
the test campaign, in order to verify the consistency of the data with respect to the
expected (e.g. identification of duplicate messages, sequencing of messages, etc.)
and therefore the proper functioning of the system-of-systems as a whole.
1.1.5. Compliance of deliveries and schedules
Development and construction projects are associated with often strict delivery
dates and schedules. The impact of a late delivery of a component generates
cascading effects impacting the delivery of the system and the system-of-systems.
Timely delivery, with the expected features and the desired level of quality, is
therefore very important. In some systems-of-systems, the completeness of the
functionalities and their level of quality are often more important than the respect of
the delivery date. In others, respecting the schedule is crucial in order to meet
imperatives (e.g. launch window for a rocket aiming for another planet).
Test projects depend on the delivery of requirements and components to be

tested within a specific schedule. Indeed, testers can only design tests based on the
requirements, user stories and features delivered to them and can only run tests on
the components, products and systems delivered to them in the appropriate test
environments (i.e. including the necessary data and systems). The timely delivery of
deliverables (contracts, requirements documents, specifications, features, user
stories, etc.) and components, products and systems in a usable state – that is, with
information or expected and working functionality – is crucial, or testers will not be
able to perform their tasks properly.
This involves close collaboration between test manager and project managers in
charge of the design and production of components, products or systems to be
tested, as well as managers in charge of test environments and the supply of test
data.
In the context of Agile and Lean methods, any delay in deliveries and any
non-compliance with schedules is a “loss of value” and should be eliminated. It is
however important to note that the principles of agility propose that it is the
development teams that define the scope of the functionalities to be delivered at each
iteration.
1.1.6. Coordinating and setting up environments
Depending on the test levels, environments will include more and more
components, products and systems that will need to coordinate to represent test
environments representative of real life. Each environment includes one or more
systems, components, products, as well as interfaces, ETLs and communication
equipment (wired, wireless, satellite, optical networks, etc.) of increasing
complexity. The design of these various environments quickly becomes a full-time
job, especially since it is necessary to ensure that all the versions of all the software
are correctly synchronized and that all the data, files, contents of databases and
interfaces are synchronized and validated in order to allow the correct execution of
the tests on this environment.
The activity of coordinating and setting up environments interacts strongly with

all the other projects participating in the realization of the system-of-systems. Some
test environments will only be able to simulate part of the target environment (e.g.
simulation of space vacuum and sunlight with no ability to simulate zero gravity),
and therefore there may be, for the same test level, several test execution campaigns,
each on different technical or functional domains.
1.1.7. Validation of prerequisites – Test Readiness Review (TRR)
Testing activities can start effectively and efficiently as soon as all their
prerequisites are present. Otherwise, the activities will have to stop and then start
again when the missing prerequisite is provided, etc. This generates significant
waste of time, not to mention everyone’s frustration. Before starting any test task,
we must make sure that all the prerequisites are present, or at the very least that they
will arrive on time with the desired level of quality. Among the prerequisites, we
have among others the requirements, the environment, the datasets, the component
to be tested, the test cases with the expected data, as well as the testers, the tools and
procedures for managing tests and anomalies, the KPIs and metrics allowing the
reporting of the progress of the tests, etc.
One solution to ensure the presence of the prerequisites is to set up a TRR (Test
Readiness Review) milestone, a review of the start of the tests. The purpose of this
milestone is to verify – depending on the test level and the types of test – whether or
not the prerequisites are present. If prerequisites are missing, it is up to the project
managers to decide whether or not to launch the test activity, taking into account the
identified risks.
In Agile methods, such a review can be informal and only apply to one user story
at a time, with the acronym DOR for definition of ready.
1.1.8. Delivery of datasets (TDS)
The delivery of test datasets (TDS) is not limited to the provision of files or
databases with information usable by the component, product or system. This also
includes – for the applications, components, products or systems with which the
component, product or system under test interacts – a check of the consistency and
synchronization of the data with each other. It will be necessary to ensure that the
interfaces are correctly described, defined and implemented.
Backup of datasets or automation of dataset generation processes may be

necessary to allow testers to generate the data they need themselves.
The design of coherent and complete datasets is a difficult task requiring a good
knowledge of the entire information system and the interfaces between the
component, product or system under test on the one hand and all the other systems
of the test environment on the other hand. Some components, products or systems
may be missing and replaced by “stubs” that will simulate the missing elements. In
this case, it is necessary to manage these “stubs” with the same rigor as if they were
real components (e.g. evolution of versions, data, etc.).
1.1.9. Go-NoGo decision – Test Review Board (TRB)
A Go-NoGo meeting is used to analyze the risks associated with moving to the
next step in a process of designing and deploying a component, product, system or
system-of-systems, and to decide whether to proceed to the next step.
This meeting is sometimes split into two reviews in time:

– A TRB (Test Review Board) meeting analyzes the results of the tests carried
out in the level and determines the actions according to these results. This technical
meeting ensures that the planned objectives have been achieved for the level.
– A management review to obtain – from the hierarchy, the other stakeholders,

the MOA and the customers – a decision (the “Go” or the “NoGo” decision)
accepted by all, with consideration of business risks, marketing, etc.
The Go-NoGo meeting includes representatives from all business stakeholders,

such as operations managers, deployment teams, production teams and marketing
teams.
In an Agile environment, the concept of Go-NoGo and TRB is detailed under the
concept of DOD (definition of done) for each of the design actions.
1.1.10. Continuous delivery and deployment
The concept of continuous integration and continuous delivery (CI/CD) is

interesting and deserves to be considered in systems-of-systems with preponderant
software. However, such concepts have particular constraints that we must study,
beyond the use of an Agile design methodology.
1.1.10.1. Continuous delivery

The continuous delivery practices mentioned in Kim et al. (2016) focus primarily
on the aspects of continuous delivery and deployment of software that depend on
automated testing performed to ensure developers have quick (immediate) feedback
on the defects, performance, security and usability concerns of the components put
in configuration. In addition, the principle is to have a limited number of
configuration branches.
In the context of systems-of-systems, where hardware components and

subsystems including software must be physically installed – and tested on physical
test benches – the ability to deliver daily and ensure the absence of regressions
becomes more complex, if not impossible, to implement. This is all the more true
since the systems-of-systems are not produced in large quantities and the
interactions are complex.
1.1.10.2. Continuous testing

On-demand execution of tests as part of continuous delivery is possible for unit
testing and static testing of code. Software integration testing could be considered,
but anything involving end-to-end (E2E) testing becomes more problematic because
installing the software on the hardware component should generate a change in the
configuration reference of the hardware component.
Among the elements to consider, we have an ambiguity of terminology: the term

ATDD (Acceptance Test-Driven Development) relates to the acceptance of the
software component alone, not its integration, nor the acceptance of the system-of-
system nor of the subsystem or equipment.
Another aspect to consider is the need for test automation and (1) the continued
increase in the number of tests to be executed, which will mean increasing test
execution time as well as (2) the need to ensure that the test classes in the software
(case of TDD and BDD) are correctly removed from the versions used in integration
tests and in system tests.
One of the temptations associated with testing in a CI/CD or DevOps

environment is to pool the tests of the various software components into a single test
batch for the release, instead of processing the tests separately for each component.
This solution makes it possible to pool the regression tests of software components,
but is a difficult practical problem for the qualification of systems-of-systems as
mentioned in Sacquet and Rochefolle (2016).
1.1.10.3. Continuous deployment

Continuous deployment depends on continuous delivery and therefore automated
validation of tests, and the presence of complete documentation – for component
usage and administration – as well as the ability to run end-to-end on an
environment representative of production.
According to Kim et al. (2016), in companies like Amazon and Google, the
majority of teams practice continuous delivery and some practice continuous
deployment. There is wide variation in how to perform continuous deployment.
1.2. Tracking test projects
Monitoring test projects requires monitoring the progress of each of the test
activities for each of the systems of the system-of-systems, as well as on each of the
test environments of each of the test levels of each of these systems. It is therefore
important that the progress information of each test level is aggregated and
summarized for each system and that the test progress information of each system is
aggregated at the system-of-systems level. This involves defining the elements that
must be measured (the progress), against which benchmark they must be measured
(the reference) and identifying the impacts (dependencies) that this can generate.
Reporting of similar indicators from each of the systems will facilitate
understanding. Automated information feedback will facilitate information retrieval.
1.3. Risks and systems-of-systems
Systems-of-systems projects are subject to more risk than other systems in that
they may inherit upstream-level risks and a process’s tolerance for risk may vary by
organization and the delivered product. In Figure 1.1, we can identify that the more
we advance in the design and production of components by the various
organizations, the risks will be added and the impact for organizations with a low
risk tolerance will be more strongly impacted than others.
Figure 1.1. Different risk tolerance
In Figure 1.2, we can identify that an organization will be impacted by all the
risks it can inherit from upstream organizations and that it will impose risks on all
downstream organizations.
Figure 1.2. Inherited and imposed risks
We realize that risk management in systems-of-systems is significantly more

complex than in the case of complex systems and may need to be managed at
multiple levels (e.g. interactions between teams, between managers of the project or
between the managers – or leaders – of the organizations).
1.4. Particularities related to SoS
According to Firesmith (2014), several pitfalls should be avoided in the context

of systems-of-systems, including:
– inadequate system-of-systems test planning;
– unclear responsibilities, including liability limits;
– inadequate resources dedicated to system-of-systems testing;
– lack of clear systems-of-systems planning;
– insufficient or inadequate systems-of-systems requirements;
– inadequate support of individual systems and projects;
– inadequate cross-project defect management.
To this we can add:

– different quality requirements according to the participants/co-contractors,
including regarding the interpretation of regulatory obligations;
– the needs to take into account long-term evolutions;
– the multiplicity of level versions (groupings of software working and delivered
together), multiple versions and environments;
– the fact that systems-of-systems are often unique developments.
1.5. Particularities related to SoS methodologies
Development methodologies generate different constraints and opportunities.

Sequential developments have demonstrated their effectiveness, but involve
constraints of rigidity and lack of responsiveness, if the contexts change. Agility
offers better responsiveness at the expense of a more restricted analysis phase and an
organization that does not guarantee that all the requirements will be developed. The
choice of a development methodology will imply adaptations during the
management of the project and during the testing of the components of the system-
of-systems.
Iterative methodologies involve rapid delivery of components or parts of

components, followed by refinement phases if necessary. That implies that:
– The planned functionalities are not fully provided before the last delivery of
the component. Validation by the business may be delayed until the final delivery of
the component. This reduces the time for detecting and correcting anomalies and
can impact the final delivery of the component, product or system, or even the
system-of-systems.
– Side effects may appear on other components, so it will be necessary to retest
all components each time a component update is delivered. This solution can be
limited to the components interacting directly with the modified component(s) or
extend to the entire system-of-systems, and it is recommended to automate it.
– The interfaces between components may not be developed simultaneously and
therefore that the tests of these interfaces may be delayed.
Sequential methodologies (e.g. V-cycle, Spiral, etc.) focus on a single delivery,

so any evolution – or need for clarification – of the requirement will have an impact
on lead time and workload, both in terms of development (redevelopment or
adaptation of components, products or systems) and in terms of testing (design and
execution of tests).
1.5.1. Components definition
Within the framework of sequential methodologies, the principle is to define the

components and deliver them finished and validated at the end of their design phase.
This involves a complete definition of each product or system component and the
interactions it has with other components, products or systems. These exhaustive
definitions will be used both for the design of the component, product or system and
for the design of the tests that will validate them.
1.5.2. Testing and quality assurance activities
It is not possible to envisage retesting all the combinations of data and actions of
the components of a level of a system-of-systems; this would generate a workload
disproportionate to the expected benefits. One solution is to verify that the design
and test processes have been correctly carried out, that the proofs of execution are
available and that the test activities – static and dynamic – have correctly covered
the objectives. These verification activities are the responsibility of the quality
assurance teams and are mainly based on available evidence (paper documentation,
execution logs, anomaly dashboards, etc.).
1.6. Particularities related to teams
In a test project, whether it is software testing or systems-of-systems testing, one

element to take into account is the management of team members, and their
relationships with each other, others and to the outside. This information is grouped
into what NASA calls CRM (Crew Resource Management). Developed in the
1970s–1980s, CRM is a mature discipline that applies to complex projects and is
ideal for decision-making processes in project management.
It is essential to:
– recognize the existence of a problem;
– define what the problem is;
– identify probable solutions;
– take the appropriate actions to implement a solution.
If CRM is mainly used where human error can have devastating effects, it is
important to take into account the lessons that CRM can bring us in the
implementation of decision-making processes. Contrary to a usual vision, people
with responsibilities (managers and decision-makers) or with the most experience
are sometimes blinded by their vision of a solution and do not take into account
alternative solutions. Among the points to keep in mind is communication between
the different members of the team, mutual respect – which will entail listening to the
information provided – and then measuring the results of the solutions implemented
in order to ensure their effectiveness. Team members can all communicate important
information that will help the project succeed.
The specialization of the members of the project team, the confidence that we
have in their skills and the confidence that they have in their experience, the
management methods and the constraints – contractual or otherwise – mean that the
decision-making method and the decisions made can be negatively impacted in the
absence of this CRM technique. This CRM technique has been successfully
implemented in aeronautics and space, and its lessons should be used successfully in
complex projects.
2
Testing Process
Test processes are nested within the set of processes of a system-of-systems.

More specifically, they prepare and provide evidence to substantiate compliance
with requirements and provide feedback to project management on the progress of
test process activities. If CMMI is used, other process areas than VER and VAL will
be involved: PPQA (Process and Product Quality Assurance), PMC (Project
Monitoring and Control), REQM (Requirements Management), CM (Configuration
Management), TS (Technical Solution), MA (Measurement and Analysis), etc.
These processes will all be involved to some degree in the testing processes.
Indeed, the test processes will decline the requirements, whether or not they are
defined in documents describing the conformity needs, and the way in which these
requirements will be demonstrated (type of IADT proofs), will split the types of
demonstration according to the levels test and integration (system, subsystem,
sub-subsystem, component, etc.) and static types (Analysis and Inspection for static
checks during design) or dynamic (demonstration and tests during the levels of
integration and testing of subsystems, systems and systems-of-systems). Similarly,
the activities of the test process will report information and progress metrics to
project management (CMMI PMC for Project Monitoring and Control process) and
will be impacted by the decisions descending from this management.
The processes described in this chapter apply to a test level and should be
repeated on each of the test levels, for each piece of software or containing software.
Any modification in the interfaces and/or the performance of a component
interacting with the component(s) under test will involve an analysis of the impacts
and, if necessary, an adaptation of the test activities (including about the test) and
evidence to be provided to show the conformity of the component (or system,
subsystem, equipment or software) to its requirements. Each test level should
coordinate with the other levels to limit the execution of tests on the same
requirements.
The ISO/IEC/IEEE29119-1 standard describes the following processes, grouping

these in organizational processes (in yellow), management processes (pink) and
dynamic processes (green).
Figure 2.1. Test processes. For a color version of this

figure, see www.iste.co.uk/homes/systems2.zip
Defined test processes are repeatable at each test level of a system-of-systems:

– the general organization of the test level;
– planning of level testing activities;
– control of test activities;
– analysis of needs, requirements and user stories to be tested;
– design of the test cases applicable to the level;
– implementation of test cases with automation and provision of test data;
– execution of designed and implemented test cases, including the management
of anomalies;
– evaluation of test execution results and exit criteria;
– reporting;
– closure of test activities, including feedback and continuous improvement
actions;
– infrastructure and environment management.
Testing Process 17
An additional process can be defined: the review process, which can be carried
out several times on a test level, on the one hand, on the input deliverables, and on
the other hand, on the deliverables produced by each of the processes of the level.
Review activities can occur within each defined test process.
The proposed test processes are applicable regardless of the development mode
(Agile or sequential). In the case of an Agile development mode, the testing
processes must be repeated for each sprint and for each level of integration in a
system-of-systems.
The processes must complement each other and – even if they may partially
overlap – it must be ensured that the processes are completed successfully.
2.1. Organization
Objectives:
– develop and manage organizational needs, in accordance with the company’s
test policy and the test strategies of higher levels;
– define the players at the level, their responsibilities and organizations;
– define deliverables and milestones;
– define quality targets (SLA, KPi, maximum failure rate, etc.);
– ensure that the objectives of the test strategy are addressed;
– define a standard RACI matrix.
Actor(s):
– CPI (R+A), CPU/CPO (I), developers (C+I);
– experienced “test manager” having a pilot role of the test project (R).
Prerequisites/inputs:
– calendar and budgetary constraints defined for the level;
– actors and subcontractors envisaged or selected;
– repository of lessons learned from previous projects.
Deliverables/outputs:
– organization of level tests;
– high-level WBS with the main tasks to be carried out;
– initial definition of test environments.
Entry criteria:
– beginning of the organization phase.
Exit criteria:
– approved organizational document (ideally a reduced number of pages).
Indicators:
1) efficiency: writing effort;
2) coverage: traceability to the quality characteristics identified in the project test
strategy.
Points of attention:
– ensure that the actors and meeting points (milestones and level of reporting)
are well defined.
2.2. Planning
Objective:
– plan test activities for the project, level, iteration or sprint considering existing
issues, risk levels, constraints and objectives for testing;
– define the tasks (durations, objectives, incoming and outgoing, responsibilities,
etc.) and sequencing;
– define the exit criteria (desired quality level) for the level;
– identify the prerequisites, resources (environment, personnel, tools, etc.)
necessary;
– define measurement indicators and frequencies, as well as reporting.
Actor(s):
– experienced testers “test manager”, having a role of manager of the test
project (R);
– testers (C+I).
Testing Process 19
– information on the volume, workload and deadlines of the project;
– information on available environments and interfaces;
– objectives and scope of testing activities.
2.2.1. Project WBS and planning
Objective:
– plan test activities for the project or level, iteration or sprint considering
existing status, risk levels, constraints and objectives for testing;
– define the tasks (durations, objectives, incoming and outgoing, responsibilities,
etc.) and sequencing;
– define the exit criteria (desired quality level) for the level;
– identify prerequisites, resources (environment, personnel, tools, etc.)
necessary;
– define measurement indicators and frequencies, as well as reporting.
Actor(s):
– experienced testers “test manager”, having a role of manager of the test project
(R);
– testers (C+I).
– REAL and project WBS defined in the investigation phase;
– lessons learned from previous projects (repository of lessons learned).
– master test plan, level test plan(s);
– level WBS (or TBS for Test Breakdown Structure), detailing – for the
applicable test level(s) – the tasks to be performed;
Entry criteria:
– start of the investigation phase.
Exit criteria:
– test plan approved, all sections of the test plan template are completed.
Indicators:
1) efficiency: writing effort vs. completeness and size of the deliverables
provided;
2) coverage: coverage of the quality characteristics selected in the project Test
Strategy.
– ensure that test data (for interface tests, environment settings, etc.) will be well
defined and provided in a timely manner;
– collect lessons learned from previous projects.
– master test plan, level test plan(s);
– level WBS, detailing – for the test level(s) – the tasks to be performed;
– detailed Gantt of test projects – each level – with dependencies;
Entry criteria:
– start of the investigation phase.
Exit criteria:
– approved test plan, all sections of the applicable test plan template are
completed.
Indicators:
1) efficiency: writing effort;
2) coverage: coverage of the quality characteristics selected in the project’s test
strategy.
Testing Process 21
– ensure that test data (for interface testing, environment settings, etc.) will be
well defined and provided in a timely manner.
2.3. Control of test activities
Objective:
– throughout the project: adapt the test plan, processes and actions, based on the
hazards and indicators reported by the test activities, so as to enable the project to
achieve its objectives;
– identify changes in risks, implement mitigation actions;
– provide periodic reporting to the CoPil and the CoSuiv;
– escalate issues if needed.
Actor(s):
– CPI (A+I), CPU/CPO (I), developers (I);
– test manager with a test project manager role (R);
– testers (C+I) [provide indicators];
– CoPil CoNext (I).
– risk analysis, level WBS, project and level test plan.
– periodic indicators and reporting for the CoPil and CoSuiv;
– updated risk analysis;
– modification of the test plan and/or activities to allow the achievement of the
“project” objectives.
Entry criteria:
– project WBS, level WBS.
Exit criteria:
– end of the project, including end of the software warranty period.
Indicators:
– dependent on testing activities.
2.4. Analyze
Objective:
– analyze the repository of information (requirements, user stories, etc. usable
for testing) to identify the test conditions to be covered and the test techniques to be
used. A risk or requirement can be covered by more than one test condition. A test
condition is something – a behavior or a combination of conditions – that may be
interesting or useful to test.
Actor(s):
– testers, test analysts, technical test analysts.
– initial definition of test environments;
– requirements and user stories (depending on the development method);
– acceptance criteria for (if available);
– analysis of prioritized project risks;
– level test plan with the characteristics to be covered, the level test environment.
– detailed definition of the level test environment;
– test file;
– prioritized test conditions;
– requirements/risks traceability matrix – test conditions.
Entry criteria:
– validated and prioritized requirements;
– risk analysis.
Exit criteria:
– each requirement is covered by the required number of test conditions
(depending on the RPN of the requirement).
Testing Process 23
Indicators:
1) Efficiency:
- number of prioritized test conditions designed,
- updated traceability matrix for extension to test conditions.
2) Coverage:
- percentage of requirements and/or risks covered by one or more test

conditions,
- for each requirement or user story analyzed, implementation of traceability to
the planned test conditions,
- percentage of requirements and/or risks (by risk level) covered by one or
more test conditions.
2.5. Design
Objective:
– convert test conditions into test cases and identify test data to be used to cover
the various combinations. A test condition can be converted into one or more test
cases.
Actor(s):
– testers, test technicians.
– prioritized test conditions;
– requirements/risks traceability matrix – test conditions.
– prioritized test cases, definition of test data for each test case (input and
expected);
– prioritized test procedures, taking into account the execution prerequisites;
– requirements/risks traceability matrix – test conditions – test cases.
Entry criteria:
– test conditions defined and prioritized;
– risk analysis.
Exit criteria:
– each test condition is covered by one or more test cases (according to the
RPN);
– partitions and typologies of test data defined for each test;
– defined test environments.
Indicators:
1) Efficiency:
- number of prioritized test cases designed,
- updated traceability matrix for extension to test cases.
2) Coverage:
- percentage of requirements and/or risks covered by one or more test cases
designed.
2.6. Implementation
Objective:
– finely describe – if necessary – the test cases;
– define the test data for each of the test cases generated by the test design
activity;
– automate the test cases that need to be;
– setting up test environments.
Actor(s):
– testers, test automators, data and systems administrators.
– prioritized test cases;
– risk analysis.
– automated or non-automated test scripts, test scenarios, test procedures;
Testing Process 25
– test data (input data and expected data for comparison);

– traceability matrix of requirements to risks – test conditions – test cases – test
data.
Entry criteria:
– prioritized test cases, defined with their data partitions.
Exit criteria:
– test data defined for each test;
– test environments defined, implemented and verified.
Indicators:
1) Efficiency:
- number of prioritized test cases designed with test data,
- updated traceability matrix for extension to test data,
- number of test environments defined, implemented and verified vs. number
of environments planned in the test strategy.
2) Coverage:
- percentage of test environments ready and delivered,
- coverage of requirements and/or risks by one or more test cases with data,
- coverage of requirements and/or risks by one or more automated test cases.
2.7. Test execution
Objective:
– execute the test cases (on the elements of the application to be tested) delivered
by the development;
– identify defects and write anomaly sheets;
– report monitoring and coverage information.
Actor(s):
– testers, test technicians.
– system to be tested is available and managed in delivery (configuration
management), accompanied by a delivery sheet.
– anomaly sheets filled in for any identified defect;
– test logs.
Entry criteria:
– testing environment and resources (including testing tools) available for the
level, and tested;
– anomaly management tool available and installed;
– test cases and test data available for the level;
– component or application to be tested available and managed in delivery
(configuration management);
– delivery sheet provided.
Exit criteria:
– coverage of all test cases for the level.
Indicators:
1) Efficiency:
- percentage of tests passed, skipped (not passed) and failed, by level of risk,
- percentage of test environment availability for test execution,
- test execution workload achieved vs. planned.
2) Coverage:
- percentage of requirements/risks tested with at least one remaining defect,
- percentage of requirements/risks tested without any defect remaining.
2.8. Evaluation
Objective:
– identify whether the test execution results show that the execution campaign
will be able to achieve the objectives;
Testing Process 27
– ensure that the acceptance criteria defined for the requirements or user stories
are met;
– if scope or quality changes impact testing, identify and select the mitigation
actions.
Actor(s):
– person responsible for project testing activities.
– definition of acceptance criteria;
– project load estimation data;
– actual usage data of project loads;
– progress data (coverage, deadlines, anomalies, etc.) of the project.
– progress graphs, identification of trends;
– progress comments (identification of causes and proposals for mitigation).
Entry criteria:
– start of the project.
Exit criteria:
– end of the duration of each test task and of the test campaign;
– complete coverage achieved for the features or components to be tested.
Indicators:
1) Efficiency:
- identify the workload used, the anomalies identified – including priority and
criticality – as well as the level of coverage achieved, compare against the objectives
defined in the planning part.
2) Coverage:
- all the activities planned for the test task or for the test campaign have been
carried out,
- to be defined based on the planned objectives and their achievement for each
requirement or user story.
Another random document with
no related content on Scribd:
time in the imagos of Pteronarcys (see p. 401). Although these
fossils are of such enormous antiquity, the tracheae can, M.
Brongniart says, be still perceived in these processes.
The Platypterides include also a considerable number of Insects of

large size, with four large equal wings, frequently spotted or
variegate. Some of these Insects were provided with expansions or
lobes on the sides of the prothorax (Fig. 213); these are looked on
as analogous to the expansions of meso- and meta-thorax, which
are supposed by some writers to have been the rudiments from
which wings were developed. These prothoracic wing-rudiments, if
such they be, are said to have a system of nervures similar to what
we find in true wings. The genus Lithomantis includes a Scotch
fossil, and has already been mentioned by us on p. 259.
Fig. 213.—Lithomantis carbonaria. Carboniferous strata of Commentry,

France. (After Brongniart.)
The third family of extinct carboniferous Neuroptera is the

Stenodictyopterides, in which Brongniart places the Dictyoneura of
Goldenberg, the North American Haplophlebium, and several genera
from Commentry. Some of them were very large Insects, with robust
bodies, and possessed wing-like expansions on the prothorax, and
lateral gill-like appendages on the sides of the abdomen.
It is worthy of note that though so large a number of carboniferous

Neuroptera have now been discovered, no larvae or immature forms
have been found.
We now pass to the consideration of the divisions of Neuroptera still

living.
Fam. I. Mallophaga—Bird-Lice or Biting Lice.
Small Insects, wingless, with large head; thorax usually of two,

rarely of one or three segments; prothorax always distinct; hind
body consisting of eight to ten segments, in addition to the
posterior two thoracic segments which usually are but little or not
at all separated from it. The metamorphosis is very slight. The
creatures live on the skins of birds or mammals, finding
nourishment in the epidermal products.
The whole of the Insects of this family live a parasitic, or rather

epizoic, life. They all creep about those parts that are near to the
skin, the feathers of birds or the hair of mammals; they rarely come
quite to the surface, so that they are not detected on a superficial
examination. It is curious that under these circumstances they
should exhibit so great a variety of form and of anatomical
characters as they do.
Fig. 214.—Trinoton luridum. Lives on the common duck and various

species of Anas. (After Giebel.)
They are very depressed, that is, flat, Insects, with a large head,
which exhibits a great variety of shape; frequently it is provided in
front of the antennae with some peculiar tubercles called trabeculae,
which in some cases are mobile. The antennae are never large,
frequently very small; they consist of from three to five joints, and are
sometimes concealed in a cavity on the under side of the head.
Fig. 215.—Under-surface of head of Lipeurus heterographus. (After
Grosse.) ol, Labium; md, mandible; mx, maxilla; ul, labium.
Fig. 216.—Under lip of Nirmus, A; and of Tetrophthalmus chilensis, B.

(After Grosse.) m, Mentum; g, ligula; pl, palp; pg, paraglossa; hy,
lingua.
The eyes are very rudimentary, and consist of only a small number
of isolated facets placed behind the antennae; sometimes they are
completely absent. The mouth parts are situated entirely on the
under-surface of the head and in a cavity. The upper lip is frequently
of remarkable form, as if it were a scraping instrument (ol, Fig. 215).
The mandibles are sharply toothed and apparently act as cutting
instruments. The maxillae have been described in the principal work
on the family[270] as possessing in some cases well-developed palpi.
According to Grosse[271] this is erroneous; the maxillae, he says, are
always destitute of palpi, and are of peculiar form, being each merely
a lobe of somewhat conical shape, furnished on one aspect with
hooks or setae. The under lip is peculiar, and apparently of very
different form in the two chief groups of Mallophaga. The large
mentum bears, in Liotheides (Fig. 216, B), on each side a four-
jointed palpus, the pair of palps being very widely separated; the
ligula is broad and undivided; on each side there is a paraglossa
bearing an oval process, and above this is a projection of the
hypopharynx. In Philopterides (Fig. 216, A) the palpi are absent, and
the parts of the lower lip are—with the exception of the paraglossae
—but little differentiated. The lingua (hypo-pharynx) in Mallophaga is
largely developed, and bears near the front a chitinous sclerite
corresponding with another placed in the epipharynx.
The prothorax in Mallophaga is a distinct division of the body even

when the meso- and meta-thorax appear to be part of the abdomen.
The mesothorax is frequently very small; it and the metathorax are
sometimes intimately connected. In other cases (Laemobothrium)
the metathorax appears to differ from the following abdominal
segment only by having the third pair of legs attached to it. In
Trinoton (Fig. 214) the three thoracic segments are well developed
and distinct. The abdominal segments visible, vary in number from
eight to ten; there is sometimes a difference according to sex, the
male having one segment taken into the interior in connexion with
the reproductive organs. The legs have short, broad coxae and small
tarsi of one or two joints; very rarely three joints are present; there
are either one or two claws; the legs with one claw being adapted for
clinging to or clutching hairs. The front pair of legs is used not for
locomotion so much as for grasping the food and bringing it within
the range of the mouth. No trace of wings has been detected in any
species.
Fig. 217.—Ganglia of nervous system of Lipeurus bacillus. (After

Giebel.) a, Cavity of head.
The nervous system has been examined by Giebel in Lipeurus

bacillus; there is a supra- and an infra-oesophageal ganglion, and
three thoracic, but no abdominal ganglia. The supra-oesophageal is
remarkably small, in fact not larger than the infra-oesophageal; it
consists evidently of two conjoined halves. The alimentary canal has
a slender, elongate oesophagus, dilated behind into a crop; this is
frequently received between two cornua formed by the anterior part
of the stomach, which, except for these, is simply tubular in form,
though somewhat narrower at the posterior extremity. In some forms
—Philopterides—the crop is of a very peculiar nature (Fig. 218),
forming an abrupt paunch separated from the stomach by the
posterior portion of the oesophagus. There are only four Malpighian
tubes; in some species the basal half of each tube is much dilated.
The two divisions of the intestine are short and are separated by the
intervention of a glandular girdle. Salivary glands exist; Giebel
figures what we may consider to be an enormous salivary reservoir
as existing in Menopon leucostomum.
The testes and ovaries are of a simple nature. The former consist of
two or three capsules, each having a terminal thread; the vasa
deferentia are tortuous and of variable length; they lead into the
anterior part of the ejaculatory duct, where also opens the elongate
duct proceeding from the bicapsular vesicula seminalis; these
structures have been figured by Grosse[272] as well as by Giebel.
The ovaries consist of three to five short egg-tubes on each side; the
two oviducts combine to form a short common duct with which there
is connected a receptaculum seminis.
Fig. 218.—Alimentary canal of Docophorus fuscicollis. (After Giebel.) a,

Oesophagus; b, paunch; a′, posterior division of oesophagus; c,
chylific ventricle or stomach; d, Malpighian tubes; e, small
intestine; f, glandular girdle; g, rectum.
The eggs of some Mallophaga have been figured by Melnikow;[273]

they possess at one extremity a cover with a multiple micropyle-
apparatus, and at the opposite pole are provided with seta-like
appendages. They are very like the eggs of the true lice, and are
said in some cases to be suspended by threads to the hairs or
feathers after the fashion of the eggs of Pediculi.
Little is known as to the development; the young are extremely like

the adult, and are thought to moult frequently; the duration of life is
quite unknown.
It has been stated by some writers that the mouth is truly of the
sucking kind, and that the Mallophaga feed on the blood of their
hosts. This is, however, erroneous; they eat the delicate portions of
the feathers of birds, and of mammals perhaps the young hair. Their
fertility is but small, and it is believed that in a state of nature they
are very rarely an annoyance to their hosts. The majority of the
known species live on birds; the forms that frequent mammals are
less varied and have been less studied; most of them have only one
claw to the feet (Fig. 220), while the greater portion of the avicolous
species have two claws.
Fig. 219.—Lipeurus ternatus, male; inhabits Sarcorhamphus papa.

(After Giebel.)
Fig. 220.—Trichodectes latus, male; inhabits the dog, Canis familiaris.

Most of the forms have the anterior legs small, and they are usually
drawn towards the mouth, owing, it is believed, to their being used
after the manner of hands to bring the food to the mouth; hence in
some of our figures (219, 220) the body looks as if it had only four
legs.
Very diverse statements have been made as to whether allied forms

of Mallophaga are found only on allied birds. It would appear that this
is the case only to a limited extent, as certain species are found on
quite a variety of birds; moreover, some birds harbour several
species of bird-lice, even five genera having been found, it is said,
on one species of bird. Docophorus icterodes has been recorded as
occurring on many kinds of ducks and geese; the swan, however,
harbours a distinct species, Docophorus cygni, and this is said to
have also been found on the bean-goose.
At least five species, belonging to three distinct genera, have been

found on the common fowl. The parasite most frequently met with on
this valuable creature is Menopon pallidum (Fig. 221), which is said
to have been figured by Redi two hundred years ago under the name
of Pulex capi. This species multiplies to a considerable extent; it is of
very active habits, and passes readily from one bird to another, so
that it is found on other species besides the domestic fowl. It is even
said that horses kept near hen-roosts have been seriously troubled
by Menopon pallidum, but it is suggested by Osborn that these
attacks may perhaps have been really due to itch-mites. There is,
however, no doubt that this species may infest poultry, especially if
sickly, to an enormous extent. The dust-baths in which poultry are so
fond of indulging are considered to be of great use in keeping down
the numbers of this Insect.
Fig. 221.—Menopon pallidum; inhabits the common fowl, Gallus
domesticus. (After Piaget.)
A table of the birds and mammals on which Mallophaga have been

found, together with the names of the latter, has been given by
Giebel.[274] The classification of the group, so far as the principal
divisions are concerned, by no means accords with the kind of
animals that serve as hosts, for the only two genera peculiar to
quadrupeds (Trichodectes, Fig. 220; and Gyropus) belong to the two
chief divisions of Mallophaga. The genus Menopon includes
numerous species found on birds, and three or four others peculiar
to mammals.
Two very natural divisions, Philopterides and Liotheides, were

adopted by Giebel and Nitzsch, but unfortunately the chief character
they made use of for diagnosing the two groups—the presence or
absence of maxillary palpi—was illusory. Apparently the labial palps
will serve the purpose of distinguishing the two divisions, they being
present in the Liotheides and absent in the Philopterides. A table of
the characters of the avicolous genera of these two groups is given
by Grosse.[275]
The Liotheides are more active Insects, and leave their host after its
death to seek another. But the Philopterides do not do so, and die in
about three days after the death of their host. Possibly Mallophaga
may be transferred from one bird to another by means of the
parasitic two-winged flies that infest birds. The writer has
recorded[276] a case in which a specimen of one of these bird-flies
captured on the wing was found to have some Mallophaga attached
to it.
We should perhaps point out that these Mallophaga, though called

bird-lice, have nothing to do with the true lice which are so frequently
found with them, and that live by sucking the blood of their hosts. It
would in fact be better to drop the name of bird-lice altogether, and
call the Mallophaga biting lice. Trichodectes latus, according to this
method, would be known as the biting louse of the dog, the true or
sucking louse of which animal is Haematopinus piliferus, and
belongs to the anoplurous division of Hemiptera.
Fam. II. Embiidae.
Elongate feeble Insects; with small prothorax, elongate meso-

and meta-thorax, which may either bear wing or be without
them. In the former case these organs are not caducous, are
delicately membranous, and all of one consistence, with three or
four indefinite longitudinal nervures and a few cross-veinlets.
The development is incompletely known. The individuals do not
form organised societies.
Fig. 222.—Oligotoma michaeli. (After M‘Lachlan.)
The Embiidae are one of the smallest families of Insects; not more
than twenty species are known from all parts of the world, and it is
probable that only a few hundred actually exist. They are small and
feeble Insects of unattractive appearance, and shrivel so much after
death as to render it difficult to ascertain their characters. They
require a warm climate. Hence it is not a matter for surprise that little
should be known about them.
Fig. 223.—Under-surface of Embia sp. Andalusia.
The simple antennae are formed of numerous joints, probably

varying in number from about fifteen to twenty-four. The mouth is
mandibulate. Chatin states[277] that the pieces homologous with
those of a maxilla can be detected in the mandible of Embia. The
labium is divided. The legs are inserted at the sides of the body, the
coxae are widely separated (Fig. 223), the hind pair being, however,
more approximate than the others. The abdomen is simple and
cylindrical, consisting of ten segments, the last of which bears a pair
of biarticulate cerci. In the male sex there is a slight asymmetry of
these cerci and of the terminal segment. The thorax is remarkable on
account of the equal development of the meso- and meta-thorax and
their elongation in comparison to the prothorax. When they bear
wings there is no modification or combination of the segments for the
purposes of flight, the condition of these parts being, even then, that
of wingless Insects; so that the Embiidae that have wings may be
described as apterous-like Insects provided with two pairs of
inefficient wings. The wings are inserted on a small space at the
front part of each of the segments to which they are attached. The
legs have three-jointed tarsi, and are destitute of a terminal
appendage between the claws.
Fig. 224.—Anterior wing of Oligotoma saundersii: A, the wing; B,

outline of the wing, showing nervures. (After Wood-Mason.) 1,
Costal; 2, subcostal; 3, radial; 4, discoidal; 5, anal nervure.
The wings in Embiidae are very peculiar; they are extremely flimsy,
and the nervures are ill-developed; stripes of a darker brownish
colour alternate with pallid spaces. We figure the anterior wing of
Oligotoma saundersii, after Wood-Mason; but should remark that the
neuration is really less definite than is shown in these figures; the
lower one represents Wood-Mason's interpretation of the nervures.
He considers[278] that the brown bands "mark the original courses of
veins which have long since disappeared." A similar view is taken by
Redtenbacher,[279] but at present it rests on no positive evidence.
One of the most curious features of the external structure is the

complex condition of the thoracic sternal sclerites. These are shown
in Fig. 223, representing the under-surface of an Embia of uncertain
species recently brought by Mr. Bateson from Andalusia.
According to Grassi[280] there are ten pairs of stigmata, two thoracic

and eight abdominal; these are connected by longitudinal and
transverse tracheae into a single system. The ganglia of the ventral
chain are, one suboesophageal, three thoracic, and seven
abdominal; these are segmentally placed, except that there is no
ganglion in the fifth abdominal segment. There is a stomato-gastric
system but no "sympathetic." Salivary glands are present. The
stomodaeal portions of the alimentary canal are remarkably
capacious; the stomach is elongate and slender, without diverticula;
the Malpighian tubes are elongate and slender; they vary in number
with the age of the individual, attaining that of twenty in the adult.
The ovaries are arranged somewhat after the fashion of those of
Japyx, there being in each five short egg-tubes, opening at equal
intervals into a straight duct. The testes are remarkably large; each
one consists of five masses of lobules, and has a large vesicula
seminalis, into the posterior part of which there open the ducts of two
accessory glands. The large joint of the front tarsus includes glands
whose secretion escapes by orifices at the tips of certain setae
interspersed between the short spines that are placed on the sole.
Species of this genus occur in the Mediterranean region, but their
characters have not yet been examined. Our information as to these
is chiefly to be found in Grassi's work. The two species studied by
him were wingless. They live under stones, where they spin webs by
means of the front feet, whose first joint is, as we have said,
enlarged and contains glands; the Insect uses the webs as a means
of support in progression, acting on them by means of papillae and a
comb-like structure placed on the four posterior feet.
Grassi informs us that these Insects are not uncommon under

stones in Catania; they require moisture as well as warmth, but not
too much; sometimes there is only one individual found under a
stone, at others eight or ten. In the winter and spring their galleries
are found on the surface of the earth, but in the hot months of
summer they secure the requisite amount of moisture by sinking
their galleries to the depth of ten or fifteen centimetres. Their food
consists chiefly of vegetable matter. They may be reared with ease
in glass vessels. Other species of the family attain wings; the details
of the process are not well known. Oligotoma michaeli (Fig. 222) was
discovered in a hothouse in London among some orchid roots
brought from India, and was found in more than one stage of
development; the young greatly resemble the adult, except in the
absence of wings. A nymph-form is described by M‘Lachlan[281] as
possessing wings of intermediate length, and Hagen has suggested
that this supposed nymph is really an adult female with short wings.
If this latter view be correct, nothing is known as to the mode of
development of wings in the family. It is still uncertain whether
female Embiidae ever possess wings. Wood-Mason and Grassi have
shown that there are wingless females in some species, and we
know that there are winged males in others, but what the usual
relation of the sexes may be in this respect is quite uncertain. These
Insects have been detected in various parts of the world. In the
Sandwich Islands Oligotoma insularis was discovered by the Rev. T.
Blackburn in the wood and thatch forming the roofs of natives'
houses. A species has been found in Prussian amber, and Grassi
thinks that Embia solieri—one of the Mediterranean species—is not
to be distinguished with certainty from the Insect found in amber.
Embiidae still remains one of the most enigmatic of the families of
Insects. Although Grassi's recent observations are of great value
from an anatomical point of view, they rather add to, than diminish,
the difficulties we encounter in endeavouring to understand the lives
of these obscure creatures. That Embiidae form webs has long been
known, and it was thought by some that the webs, like those of
spiders, might be of assistance in procuring food. We may, however,
infer from Grassi's observations that this is not the case, but that the
silken tunnels or galleries—as he calls them—serve chiefly as a
means of locomotion and protection, the feet of the Insects being
highly modified in conformity with this mode of life. Grassi seems to
be of opinion that the galleries are also useful in preserving a proper
degree of humidity round the Insects. We have already alluded to the
mystery that surrounds the mode of growth of their wings. Nearly all
that is known as to the Embiidae is contained in Grassi's paper, or is
referred to in Hagen's monograph of the family.[282]
Considerable difference of opinion has prevailed as to the allies of

these obscure Insects. It would seem that they are most nearly allied
to Termitidae and Psocidae. Grassi, however, considers these
affinities only remote, and suggests that Embiidae should form a
separate Order, to be placed in a super-Order Orthoptera, which
would include our Aptera, the two families mentioned above,
Mallophaga, Embiidae, and the ordinary Orthoptera. Brauer places
the family in his Orthoptera genuina.
CHAPTER XVI
NEUROPTERA CONTINUED—TERMITIDAE, TERMITES OR WHITE ANTS
Fam. III. Termitidae—White Ants, Termites.

Fig. 225.—Termes (Hodotermes) mossambicus. Winged adult. (After
Hagen.)
Each species is social, and consists of winged and wingless

individuals. The four wings are, in repose, laid flat on the back,
so that the upper one only is seen except just at the bases; they
are membranous and very elongate, so that they extend far
beyond the apex of the abdomen; the hind pair is remarkably
similar in size, form, and consistence to the front pair: near the
base of each wing there is a suture, or line of weakness, along
which the wings can be broken off, the stumps in that case
remaining as short horny flaps reposing on the back. Ligula
channelled but not divided into two parts. The wingless
individuals are very numerous, and have the head and thirteen
body segments distinct; the body is terminated by a pair of short
cerci. The metamorphosis is slight and gradual, and in some
individuals is dispensed with.
The term White Ants has been so long in use for the Termitidae that
it appears almost hopeless to replace it in popular use by another
word. It has, however, always given rise to a great deal of confusion
by leading people to suppose that white ants differ chiefly from
ordinary ants by their colour. This is a most erroneous idea. There
are scarcely any two divisions of Insects more different than the
white ants and the ordinary ants. The two groups have little in
common except that both have a social life, and that a very
interesting analogy exists between the forms of the workers and
soldiers of these two dissimilar Orders of Insects, giving rise to
numerous analogies of habits. The word Termites—pronounced as
two syllables—is a less objectionable name for these Insects than
white ants.
The integument in Termites is delicate, and the chitinous plates are

never very hard; frequently they are so slightly developed that the
creature appears to consist of a single membranous sac with
creases in it, the head alone being very distinct. The head is
exserted, frequently of large size, sometimes as large as all the rest
of the body together. Termites may be quite blind, or possess
facetted and simple eyes, the latter when present being two in
number and always accompanied by facetted eyes. The antennae
are simple, consisting of from nine to thirty-one joints, which differ
but little from one another; the number in each individual increases
as the development progresses. The parts of the mouth are large,
the ligula consists of one piece (Fig. 226, A), but often has the
appearance of being formed by two united pieces; on its extremity
are seated two pairs of lobes.
Fig. 226.—Termes bellicosus. Labium, A, maxilla, B, of winged adult;

lower face of each. (After Hagen.)
The head is articulated to the thorax by means of two very large

cervical sclerites on each side, placed at right angles to one another,
and visible on the under-surface. The prothorax is well developed
and distinct from the parts behind it. The pronotum, of variable form
and size, is very distinct in the perfect Insects; with it are connected
the largely developed pleura. The episternum is very peculiar,
consisting of an elongate chitinous slip on each side hanging
downwards, the two not quite meeting in the middle; they thus form
the margin of the very large anterior orifice, and are in contiguity with
the cervical sclerites; behind them are the very large epimera. The
prosternum appears to be usually entirely membranous; in some
cases the sclerite in it is small and delicate, and apparently differs
according to the species. The meso- and meta-thorax are sub-equal
in size; the mesosternum forms a peculiar, large, adpressed fold.
The metasternum is membranous, but is terminated behind by a
sclerite apparently of variable form. The hind body is voluminous,
simple in form, consisting of ten segments and bearing at the
extremity two short distant cerci of a variable number of joints. The
terminal ventral sclerites differ greatly in form according to the
species and sometimes according to the sex; there are sometimes, if
not always, present near the extremity two peculiar minute
biarticulate styles, called appendices anales. The coxae are all large,
free, and exserted; at the base of each is a transverse trochantin.
The femora are articulated with the trochanters, not with the coxae;
both femora and tibiae are slender, the tarsi small, four-jointed; the
terminal joint elongate.
Fig. 227.—Front tibia and tarsus of Calotermes rugosus larva, showing

auditory organ. × 90. (After F. Müller.)
It is now well established that Termites have a means of

communication by sounds. The individuals have a peculiar way of
jerking themselves, as has been frequently noticed by observers of
the Insects; these convulsive movements may possibly be
connected with the production of sound, which may perhaps be
evoked by contact between the back of the head and the pronotum;
the exact mode by which the sounds are produced is not, however,
known. The existence of an auditory organ in the front tibia has been
demonstrated by Fritz Müller,[283] and we reproduce (Fig. 227) one
of his figures. The structure seems to be in plan and position similar
to the ear of Locustidae, though much less perfect.
Fig. 228.—Wings of Termites: A, Termes lucifugus; B, Hodotermes
brunneicornis; C, Culotermes nodulosus. (After Redtenbacher: B
and C diagrammatic.) III, V, VII, homologous areas and nervures
according to Redtenbacher. 1, Costal; 2, subcostal; 3, median; 4,
submedian nervures according to Hagen.
The wings of Termitidae are not like those of any other Insects; their
neuration is very simple, but nevertheless the wings of the different
forms exhibit great differences in the extent to which they are made
up of the various fields. This is shown in Fig. 228, where the
homologous nervures are numbered according to the systems of
both Hagen and Redtenbacher. The area, VII, that forms the larger
part of the wing in C, corresponds to the small portion at the base of
the wing in B. The most remarkable feature of the wing is, however,
its division into two parts by a suture or line of weakness near the
base, as shown in Fig. 225. The wings are used only for a single
flight, and are then shed by detachment at this suture; the small
basal portion of each of the four wings is horny and remains
attached to the Insect, serving as a protection to the dorsal surface
of the thorax.
The nature of the suture that enables the Termites to cast their wings
with such ease after swarming is not yet understood. There are no
true transverse veinlets or nervules in Termites. Redtenbacher
suggests[284] that the transverse division of the wing at its base, as
shown in Fig. 225, along which the separation of the wing occurs at
its falling off, may have arisen from a coalescence of the subcostal
vein with the eighth concave vein of such a wing as that of Blattidae.
The same authority also informs us that the only point of
resemblance between the wings of Termitidae and those of Psocidae
is that both have an unusually small number of concave veins.
The information that exists as to the internal anatomy of Termites is
imperfect, and refers, moreover, to different species; it would appear
that considerable diversity exists in many respects, but on this point
it would be premature to generalise. What we know as to the
respiratory system is chiefly due to F. Müller.[285] The number of
spiracles is ten; Hagen says three thoracic and seven abdominal,
Müller two thoracic and eight abdominal. In fertile queens there
usually exist only six abdominal stigmata. There is good reason for
supposing that the respiratory system undergoes much change
correlative with the development of the individual; it has been
suggested that the supply of tracheae to the sexual organs is
deficient where there is arrest of development of the latter.
The alimentary canal is only of moderate length. Salivary glands

exist, as also do salivary reservoirs; these latter are large, in some
species remarkably so. The oesophagus is slender, but abruptly
enlarged behind to form a large crop; a proventriculus is apparently
either present or absent; the chylific ventricle, or stomach, is slender
and simple. The Malpighian tubules are very long; their number is
probably from four to eight in the adult, and in the earlier stages less.
Behind the tubes the alimentary canal forms a large paunch, and
after this there is a small intestine and rectum. The paunch is a
peculiar structure, and probably of great importance in the economy
of Termites.
These creatures emit minute quantities of a secretion that is

corrosive, and can act on metal and even glass;[286] its nature and
source are not understood. Hagen describes peculiar structures in
the rectum to which he is inclined[287] to ascribe the origin of this
substance, but this is very uncertain.
Fig. 229.—Head and alimentary canal of Termes lucifugus (nymph). a,
head; b, salivary glands; c, salivary receptacles; d, crop; e,
stomach; f, intestinal paunch; g, small, h, large intestine; i,
Malpighian tubes; k, extremity of body. (After Dufour.)
The brain is small; the infra-oesophageal ganglion is placed

immediately under the supra-oesophageal; there are three thoracic
and six abdominal ganglia. The nervous system apparently differs
but little in the various forms, or in the different stages of life, except
that in the fertile females the abdominal ganglia become so much
enlarged that they even exceed the brain in size.
The testes are unusually simple; each consists of eight capsules

opening into the vas deferens; the two vasa converge and are
continued as a short ejaculatory duct; at the point of convergence
there is a pair of curled vesiculae seminales.
The ovarian system is also simple; there is a variable number of

elongate egg-tubes, each of which opens separately into the oviduct;
the two ducts unite to form a short uterus, on which there is placed
first a spermatheca, and near the extremity a convolute tubular
sebific gland. The number of egg-tubes is subject to extraordinary
variation, according to the species, and according to the age of the
fertilised individual.
Social Life.—Termites live in communities that consist sometimes of

enormous numbers of individuals. The adult forms found in a
community are (1) workers; (2) soldiers; (3) winged males and
females; (4) some of these winged forms that have lost their wings.
Some species have no worker caste. The individuals of the third
category are only present for a few days and then leave the nest in
swarms. In addition to the adult individuals there are also present
various forms of young. The individuals that have lost their wings are
usually limited to a single pair, king and queen; there may be more
than one king and queen, but this is not usual. The king and queen
may be recognised by the stumps of their cast wings, which exist in
the form of small triangular pieces folded on the back of the thorax
(Fig. 235). The continuance of the community is effected entirely by
the royal pair; they are the centres of activity of the community,
which is thrown into disorder when anything happens to them.
Usually the pair are physically incapable of leaving the nest,
especially the queen, and frequently they are enclosed in a cell
which they cannot leave. In consequence of the disorganisation that
arises in the community in the absence of a royal pair, Termites keep
certain individuals in such a state of advancement that they can
rapidly be developed into royalties should occasion require it. These
reserve individuals are called complementary by Grassi; when they
become royalties they are usually immature as regards the condition
of the anterior parts of the body, and are then called by Grassi and
others neoteinic, as is more fully explained on p. 380.
Swarms.—As a result of the Termite economy large numbers of

superfluous individuals are frequently produced; these, in the winged
state, leave the community, forming swarms which are sometimes of
enormous extent, and are eagerly preyed on by a variety of animals
including even man. Hagen has given particulars[288] of a swarm of
T. flavipes in Massachusetts, where the Insects formed a dark cloud;
they were accompanied by no less than fifteen species of birds,
some of which so gorged themselves that they could not close their
beaks.
There is but little metamorphosis in Termitidae. Young Termites are

very soft; they have a thin skin, a disproportionately large head, and
are of a peculiar white colour as if filled with milk. This condition of
milkiness they retain, notwithstanding the changes of form that may

Advanced Testing of Systems of Systems Volume 2 Practical Aspects 1St Edition Bernard Homes Full Chapter

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Testing of Systems of Systems Volume 2 Practical Aspects 1St Edition Bernard Homes Full Chapter

Uploaded by

Copyright:

Available Formats

Advanced Testing of

ISTE Ltd John Wiley & Sons, Inc.

© ISTE Ltd 2022

Library of Congress Control Number: 2022944148

British Library Cataloguing-in-Publication Data

Dedication and Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Chapter 1. Test Project Management . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2. Testing Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3. Control of test activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Chapter 3. Continuous Process Improvement . . . . . . . . . . . . . . . . . . . 37

Chapter 4. Test, QA or IV&V Teams. . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.5.1. Lead or direct? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Chapter 5. Test Workload Estimation . . . . . . . . . . . . . . . . . . . . . . . . 75

5.4.4. Test campaign preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Chapter 6. Metrics, KPI and Measurements . . . . . . . . . . . . . . . . . . . . 95

Chapter 7. Requirements Management . . . . . . . . . . . . . . . . . . . . . . . 119

7.3.3. Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Chapter 8. Defects Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Chapter 9. Configuration Management . . . . . . . . . . . . . . . . . . . . . . . 143

9.6.3. Merge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

Chapter 10. Test Tools and Test Automation . . . . . . . . . . . . . . . . . . . 157

Chapter 11. Standards and Regulations . . . . . . . . . . . . . . . . . . . . . . . 187

Chapter 12. Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Chapter 13. Future Testing Challenges . . . . . . . . . . . . . . . . . . . . . . . 223

13.6. Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

Summary of Volume 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

Inspired by a dedication from Boris Beizer1, I dedicate these two books

Finally, paraphrasing Isaac Newton, If I was able to reach this level of

After 15 years in software development, I had the opportunity to focus on

1 Beizer, B. (1990). Software Testing Techniques, 2nd edition. ITP Media.

In the first part of these two books on systems-of-systems testing, we identified

In this second part of the book on systems-of-systems testing, we will focus on

Test Project Management

We do not claim to replace the many contributions of illustrious authors on good

At the risk of writing a tautology, the purpose of project management is to

Project management must ensure that development methodologies are correctly

1.1. General principles

Management of test projects, whether on components, products, systems or

Let us see the elements specific to software testing projects.

1.1.1. Quality of requirements

Any development translates requirements (needs or business objectives) into a

Requirements are provided to development teams as well as test teams.

The requirements must be SMART, that is:

– Achievable: the requirements must be able to be physically demonstrated

1.1.2. Completeness of deliveries

The completeness of the software, components, products, equipment and systems

This aspect of completeness is ambiguous in incremental and iterative

1.1.3. Availability of test environments

The test environment is not limited to a machine on which the software

Automating backups and restores of test environments allows testers to self-

In DevOps environments, it is recommended to enable automatic creation of

1.1.4. Availability of test data

1.1.5. Compliance of deliveries and schedules

Test projects depend on the delivery of requirements and components to be

1.1.6. Coordinating and setting up environments

The activity of coordinating and setting up environments interacts strongly with

1.1.7. Validation of prerequisites – Test Readiness Review (TRR)

1.1.8. Delivery of datasets (TDS)

Backup of datasets or automation of dataset generation processes may be

1.1.9. Go-NoGo decision – Test Review Board (TRB)

This meeting is sometimes split into two reviews in time:

– A management review to obtain – from the hierarchy, the other stakeholders,

The Go-NoGo meeting includes representatives from all business stakeholders,