DO Qualification Kit

Simulink® Test™
Tool Qualification Plan

R2017a

December 5, 2016 qualkitdo_sltest_tqp

How to Contact MathWorks
Latest news: www.mathworks.com

Sales and services: www.mathworks.com/sales_and_services

User community: www.mathworks.com/matlabcentral

Technical support: www.mathworks.com/support/contact_us

Phone: 508-647-7000
The MathWorks, Inc.
3 Apple Hill Drive
Natick, MA 01760-2098
DO Qualification Kit: Simulink® Test TM Tool Qualification Plan

© COPYRIGHT 2015–2017 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used or copied only under
the terms of the license agreement. No part of this manual may be photocopied or reproduced in any form without prior written
consent from The MathWorks, Inc.

FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the
federal government of the United States. By accepting delivery of the Program or Documentation, the government hereby agrees
that this software or documentation qualifies as commercial computer software or commercial computer software documentation
as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification,
reproduction, release, performance, display, and disclosure of the Program and Documentation by the federal government (or
other entity acquiring for or through the federal government) and shall supersede any conflicting contractual terms or conditions.
If this License fails to meet the government’s needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.

Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a
list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective
holders.

Patents
MathWorks products are protected by one or more U.S. patents. Please see www.mathworks.com/patents for more
information.

December 5, 2016 qualkitdo_sltest_tqp

Revision History
September 2015 New for DO Qualification Kit Version 3.0 (Applies to Release 2015b)

March 2016 Revised for DO Qualification Kit Version 3.1 (Applies to Release 2016a)

September 2016 Revised for DO Qualification Kit Version 3.2 (Applies to Release 2016b)

March 2017 Revised for DO Qualification Kit Version 3.3 (Applies to Release 2017a)

December 5, 2016 qualkitdo_sltest_tqp

December 5, 2016 qualkitdo_sltest_tqp
Contents
1 Introduction.......................................................................................................................................1-1

2 Tool Overview and Identification.....................................................................................................2-1

2.1 Simulink Test Product Description..........................................................................................2-2

2.2 Simulink Test Product Identifier..............................................................................................2-3

3 Tool Operational Requirements........................................................................................................3-1

4 Certification Considerations..............................................................................................................4-1

4.1 Requirements for Qualification................................................................................................4-2

4.2 Certification Credit...................................................................................................................4-3

5 Tool Development Life Cycle – Tool User.......................................................................................5-1

5.1 Planning....................................................................................................................................5-2

5.2 Requirements............................................................................................................................5-3

5.3 Verification...............................................................................................................................5-4

6 Additional Considerations.................................................................................................................6-1

6.1 Customer Bug Reporting..........................................................................................................6-2

6.2 Automation of Tool Qualification............................................................................................6-2

7 Tool Life Cycle Data.........................................................................................................................7-1

8 Schedule............................................................................................................................................8-1

December 5, 2016 qualkitdo_sltest_tqp v

December 5, 2016 qualkitdo_sltest_tqp vi
1 Introduction

This document comprises the Tool Qualification Plan (Reference DO-330 Section 10.1.2) for
the following capabilities of the Simulink® Test™ verification tool:

 Comparing simulation results or executable object code test results against expected results to
determine pass/fail simulation or test status.

 Generating reports that contain information on pass/fail simulation or test status, and
requirements tracing

This document is intended for use in the DO-178C and DO-330 tool qualification process for
Criteria 3 TQL-5 tools.

See also the DO Qualification Kit User's Guide.

2 Tool Overview and Identification
2.1 Simulink Test Product Description
Develop, manage, and execute simulation-based tests
Simulink Test provides tools for authoring, managing, and executing systematic, simulation-
based tests. You can create nonintrusive test harnesses to test models and subsystems. Simulink
Test includes a test sequence block that lets you construct complex test sequences and
assessments, and a test manager that lets you manage and execute tests. It enables functional,
baseline, equivalence, and back-to-back testing, including software-in-the-loop (SIL) and
processor-in-the-loop (PIL). You can generate reports, archive and review test results, rerun
failed tests, and debug the component or system under test.

The test harness in Simulink Test lets you test components without creating a separate test
model. You can apply pass and fail criteria that include absolute and relative tolerances, limits,
logical checks, and temporal conditions. Test execution can be automated or customized with
setup and cleanup scripts. Simulink Test stores test cases and their results, creating a repository
for reviewing and investigating failures. You can link requirements to a test case captured in
Microsoft® Word, IBM® Rational® DOORS®, and other documents (with Simulink® Verification
and Validation™).

Key Features

 Test harness for subsystem or model testing

 Test sequence block for running tests and assessments
 Pass-fail criteria, including tolerances, limits, and temporal conditions
 Baseline, equivalence, and back-to-back testing
 Setup and cleanup scripts for customizing test execution
 Test manager for authoring, executing, and organizing test cases and their results
 Automatic report generation to document test outcomes
2.2 Simulink Test Product Identifier

Software Tool Version (Release) Tool Vendor

Simulink Test Version 2.2 (R2017a) The MathWorks, Inc.

3 Apple Hill Drive
DO Qualification Kit Version 3.3 (R2017a) Natick, MA, 01760-2098 USA

December 5, 2016 qualkitdo_sltest_tqp 3

December 5, 2016 qualkitdo_sltest_tqp 4
3 Tool Operational Requirements

The Tool Operational Requirements for Simulink® Test™ are documented in:

Simulink Test Tool Operational Requirements

To access the tool operational requirements document, on the MATLAB® command line, type
qualkitdo to open the Artifacts Explorer. The document is in Simulink Test.
4 Certification Considerations

This section provides the certification considerations for the following capabilities of the
Simulink Test verification tool:

 Compare simulation results or executable object code test results against expected results to
determine pass/fail simulation or test status.

 Generate reports that contain information on pass/fail simulation status, and requirements
tracing.
4.1 Requirements for Qualification
To determine whether a tool must be qualified, the following questions based on guidance of
paragraph 12.2.1 of DO-178C are addressed:

Question Answer

Can the tool insert an error into the airborne software or fail to detect an existing Yes1
error in the software within the scope of its intended usage?
Will the output of the tool not be verified as specified in Section 6 of DO-178C, Yes
DO-278A, DO-331, DO-332 or DO-333?
Are processes of DO-178C, DO-278A, DO-331, DO-332 or DO-333 eliminated, Yes
reduced, or automated by the use of the tool and the output from the tool are used to
meet an objective or replace an objective of DO-178C, DO-278A, DO-331, DO-
332 or DO-333, Annex A or Annex C?
1
Simulink Test might fail to detect an error

Because the answers to the preceding questions are yes, Simulink Test shall be qualified.

To determine the type of qualification (Criteria 1, Criteria 2, or Criteria 3), the following
questions based on guidance of paragraph 12.2.2 of DO-178C about the tool are addressed:

Question Answer

1. Is the tool output part of the airborne software, such that the output can insert No
an error into the software?

2. Could the tool fail to detect an error in the airborne software and is the tool No
also used to justify the elimination or reduction of either of the following:

 Verification processes other than that automated by the tool.

 Development processes that could have an impact on the airborne

software.

3. Could the tool fail to detect an error in the airborne software? Yes
 Because the answer to the preceding first and second questions are no, Simulink Test can be
qualified as a Criteria 3 tool.

December 5, 2016 qualkitdo_sltest_tqp 3

4.2 Certification Credit
The following table shows the certification credit (see DO-331 Annex A or Annex C Objectives)
being sought for Simulink Test.

Certification Credit for Simulink Test with Respect to DO-331 Objectives

Annex A Objective DO-331 Software or Credit Taken

or C Reference Assurance
Table Levels
Table Simulation results are Section MB.6.8.3.2.c A, B, C, D Partial1 – Simulink Test is
MB.A-3 correct and discrepancies used to compare actual
explained. simulation results to
expected results;
discrepancies have to be
analyzed and explained
manually.

Table Simulation results are Section `MB.6.8.3.2.c A, B, C Partial1 – Simulink Test is

MB.A-4 correct and discrepancies used to compare actual
explained. simulation results to
expected results;
discrepancies have to be
analysed and explained
manually.

Table Simulation results are Section MB.6.8.3.2.c AL1, AL2, AL3, Partial1 – Simulink Test is
MB.C-3 correct and discrepancies AL4, AL5 used to compare actual
explained. simulation results to
expected results;
discrepancies have to be
analysed and explained
manually.

Table Simulation results are Section MB.6.8.3.2.c AL1, AL2, AL3, Partial1 – Simulink Test is
MB.С-4 correct and discrepancies AL4 used to compare actual
explained. simulation results to
expected results;
discrepancies have to be
analysed and explained

December 5, 2016 qualkitdo_sltest_tqp 4

 manually.

Notes:
1
This credit is taken only if the Simulink® and Stateflow® models are considered as a Design or Specification Models for
the project and simulation is used as a means of compliance for objectives 1, 2, 4, or 7 of Tables MB.A-3, MB.A-4,
MB.C-3 or MB.C-4.

December 5, 2016 qualkitdo_sltest_tqp 5

 The following table shows the certification credit, with respect to DO-178C or DO-278A
objectives, being sought for Simulink Test.

Certification Credit for Simulink Test with Respect to DO-178C or DO-278A

Objectives

Annex A Objective DO-178C Software or Credit Taken

Table DO-278A Assurance
Reference Levels
Table Executable object code Section 6.4.2.1 & A, B, C, D Partial1 – Simulink Test is used to
A-6 complies with high-level Section 6.4.3 compare actual executable object code
requirements. test results to expected results. The
expected results must be provided as
input to this tool.

Table Executable object code is Section 6.4.2.2 & A, B, C, D Partial1 – Simulink Test is used to
A-6 robust with high-level Section 6.4.3 compare actual executable object code
requirements. test results to expected results. The
expected results must be provided as
input to this tool.

Table Executable object code Section 6.4.2.1 & A, B, C Partial1 – Simulink Test is used to
A-6 complies with low-level Section 6.4.3 compare actual executable object code
requirements. test results to expected results. The
expected results must be provided as
input to this tool.

Table Executable object code is Section 6.4.2.2 & A, B, C Partial1 – Simulink Test is used to
A-6 robust with low-level Section 6.4.3 compare actual executable object code
requirements. test results to expected results. The
expected results must be provided as
input to this tool.

Table A-7 Test coverage of software Section 6.4.4.a
high-level requirements is
achieved.
A, B, C, D Partial1 – Simulink Test is used to
generate a report for traceability
between high-level requirements and
test. The links between high-level
requirements and test must be set
manually.

Table A-7 Test coverage of software Section 6.4.4.b A, B, C Partial1 – Simulink Test is used to
low-level requirements is generate a report for traceability
achieved. between high-level requirements and

December 5, 2016 qualkitdo_sltest_tqp 6

 test. The links between high-level
requirements and test must be set
manually.

Notes:
1
This credit is taken only if Simulink Test is used to automatically verify output from execution
of executable object code.

December 5, 2016 qualkitdo_sltest_tqp 7

5 Tool Development Life Cycle –
Tool User
5.1 Planning
The Plan for Software Aspects of Certification (PSAC) designates that the Simulink Test will be
qualified as a verification tool.

This document provides the Tool Qualification Plan for Simulink Test.
5.2 Requirements
 Tool Operational Requirements for Simulink Test are in:
Simulink Test Tool Operational Requirements
 To access the tool operational requirement document, on the MATLAB® command line, type
qualkitdo to open the Artifacts Explorer. The document is in Simulink Test.

 The applicant will:

- Review the Tool Operational Requirements for applicability to the project under
consideration.
- Configure the Tool Operational Requirements in a configuration management system.

 User information for Simulink Test is available in the Simulink Test User’s Guide, R2017a
and the Simulink Test Reference, R2017a.

 To access the documents, on the MATLAB command line, type qualkitdo to open the
Artifact Explorer. The documents are in Simulink Test.

 Instructions for installing the Simulink Test product are at the MathWorks Documentation
Center, R2017a:
 Installation

December 5, 2016 qualkitdo_sltest_tqp 3

5.3 Verification
Requirements-based test cases and procedures will be developed from:

Simulink Test Tool Operational Requirements

The test cases and procedures will be developed in the form of the Simulink Test test file, which
exercise the capabilities being qualified in Simulink Test.

The test cases and procedures are documented in:

Simulink Test Test Cases and Procedures

To access the documents, on the MATLAB command line, type qualkitdo to open the
Artifacts Explorer. The documents are in Simulink Test.

The applicant will:

 Review the test cases and procedures for applicability to the project under consideration.
 Configure the test cases and procedures in a configuration management system.
 Execute the test cases and procedures in the installed environment.

Executing the Simulink Test report listed in the following table generates tool verification
results in the specified test report.

Test File Summary Test Report

qualkitdo_sltest_rs.rpt qualkitdo_simulinktest_qualificationreport.html

The applicant will:

 Review the test results and confirm they are correct and discrepancies are explained.
 Configure the test results in a configuration management system.

December 5, 2016 qualkitdo_sltest_tqp 4

6 Additional Considerations
6.1 Customer Bug Reporting
MathWorks reports known critical bugs brought to its attention on its bug report system at
www.mathworks.com/support/bugreports. The bug reports are an integral part of the
documentation for each release.

The bug report system provides an interface for customers to view and submit bug reports. Users
can track the status of open bugs. Users can choose to receive notifications for new or updated
bug reports. The bug reports on this web site include internally and externally nominated bugs.
If applicable, bug reports include provisions for known workarounds or file replacements.
Customers can use the bug report mechanism to nominate bugs. These nominations are
processed and evaluated by The MathWorks, Inc. development organization

6.2 Automation of Tool Qualification

The DO Qualification Kit for Simulink Test includes the automated comparison of the generated
test reports to expected results (baseline reports). In order to provide assurance that this
capability operates correctly in the installed environment, the tests executing the automation
capability are included in the DO Qualification Kit. The test cases and procedures for the
automation capability are documented in

Simulink Test Test Cases and Procedures

7 Tool Life Cycle Data

The following table shows the life cycle data for the Simulink Test capability. The table maps
the documents and artifacts to DO-330 life cycle data items.

Simulink Test Life Cycle Data

Data Available/ DO-330 Documents/

Submit Reference Artifacts
Plan for Software Aspects of Submit Section <Insert PSAC or PSAA** reference here.>
Certification (PSAC) or 10.1.1

Plan for Software Aspects of

Approval (PSAA)

Tool Qualification Plan Submit* Section Simulink Test Tool Qualification Plan
10.1.2 (this document)

Tool Operational Available Section Simulink Test Tool Operational Requirements

Requirements 10.3.1

Simulink Test User’s Guide

Test Cases and Procedures Available Section Simulink Test Test Cases and Procedures
10.3.3

qualkitdo_sltest_rs.rpt
Test Results Available Section Simulink Test Test Results
10.3.4

qualkitdo_simulinktest_qualificatio
nreport.html
Software Accomplishment Submit Section <Insert reference to SAS** here.>
Summary (SAS) 10.1.16
 Tool Qualification Submit* Section <Insert reference to Tool Qualification
Accomplishment Summary 10.1.15 Accomplishment Summary** here.>

Notes:
* Optional for TQL-5 tool qualification
** To be created by applicant

The applicant must deliver data marked “Submit” to the certification authorities. Data marked
“Available” must be available at the applicant’s or tool vendor’s site for inspection by the
certification authorities.
8 Schedule

<Insert tool schedule in this section.>