Formal Methods

Design & Test

Yogananda Jeppu

Yogananda Jeppu – CO366 Formal Methods

Problem Statement

Yogananda Jeppu – CO366 Formal Methods 2

Type of testing
• Low level testing All tests should be requirements based.
• Functional testing
• White box testing Therefore requirements are the most
• Integration testing important thing in a system
development process and correct
• High Level Testing requirements more so.
• Hardware Software Integration Testing
• Black box testing
• System Level testing
• Iron bird testing
• Rig testing
• Flight trials
• Acceptance testing

Yogananda Jeppu – CO366 Formal Methods 3

Low level testing
• The developer may test the function for correctness.
• You can perhaps test the train model. Does the position change from 2200
to -1600 as the simulation is done. Do the Boolean flags Entered, In Zone,
Exit work properly.
• We look at code coverage metrics here during the testing.
• We need to ensure that the function is called properly.
• In aerospace we do robust testing. Variables are set to 10^38 the max
possible and the output checked. The smallest number is also checked. We
check for qNaN, sNAN and what happens if the variable is not a number.
• Check up on the Not A Number in the wiki (homework -
https://en.wikipedia.org/wiki/NaN)

Yogananda Jeppu – CO366 Formal Methods 4

Low level testing
• There are tools available that help you do the low level tests in the
industry. We need to ensure 100% code coverage at this low level.
Most of the industries work on such test cases.
• These tests are also termed as white box tests. This is because the
tester can see the code. He/She is testing the code directly against
the design requirements
• The design requirements are the equations, the finite state machines,
the truth table etc.
• The testing is also termed as verification – the process of building the
system right.

Yogananda Jeppu – CO366 Formal Methods 5

High level testing
• This is also termed as black box testing. These tests are based on the
requirements – the WHATs of the system.
• You will test the railway gate problem against the EARS requirements
that you have defined. Have you tested all requirements – is the
question you need to ask. So that leads to the question – are your
requirements complete?
• Formal methods address some of these questions.
• The high level testing is also called validation – are you building the
right system.
• In aircraft control system a plant model will also be used to test the
system. In our case we will call the train and gate model as plant
models.
Yogananda Jeppu – CO366 Formal Methods 6
Rig level tests
• Aircrafts are tested on ground rigs called iron bird

Yogananda Jeppu – CO366 Formal Methods 7

Rig level tests – A380
• Aircrafts are tested on ground rigs called iron bird

Yogananda Jeppu – CO366 Formal Methods 8

Code coverage metrics
• Statement Coverage
• Decision Coverage
• Condition Coverage
• Condition/Decision Coverage
• Modified Condition/Decision Coverage
• Multiple Condition Coverage

Yogananda Jeppu – CO366 Formal Methods 9

Statement coverage
• The measure reports on the percentage of executable statements of
the code exercised by a set of test cases. Also known as basic bloc
coverage or segment coverage.
This is just the flow of program. It should just
cover all the lines of code. This is easily achieved
if(A) then unless the variable A is never true. Then you need
F1(); to ask why are we having this logic there. This is
F2(); the advantage of code coverage.

Statement coverage does not report whether

Test Case: A=True
loops reach their termination condition - only
Statement Coverage whether the loop body was executed.
achieved

Yogananda Jeppu – CO366 Formal Methods 10

Decision coverage
• Decision coverage (also known as branch coverage, all-edges
coverage, basis path coverage, decision-decision-path testing) reports
whether Boolean expressions in control structures are evaluated to
both true and false values by the test cases.
if(A)
F1(); The if statement needs to be exercised for true
else and false. “A” could be (a > 10 && b < 20 && c ==
10) . We are only interested in the true and false
F2(); of the decision.

Test Cases for Decision Coverage: Decision coverage requires two test cases: one for
A=T a true outcome and another for a false outcome.
A=F

Yogananda Jeppu – CO366 Formal Methods 11

Condition coverage
• Condition coverage requires that each condition in a decision take on
all possible outcomes at least once, but does not require that the
decision take on all combinations.
if(A && B)
F1(); Just condition alone is not sufficient. We need
else decision coverage. Here the test case provides the
F2(); condition coverage but the F2() function is
exercised always by this test case.
Test Cases for Condition Coverage:
A=T, B=F
A=F, B=T

Yogananda Jeppu – CO366 Formal Methods 12

Condition/Decision coverage
• Condition/decision coverage combines the requirements for decision
coverage with those for condition coverage. That is, there must be
sufficient test cases to toggle the decision outcome between true and
false and to toggle each condition value between true and false.

if(A && B) This is a good coverage metric as the decision is

F1(); also covered and the conditions is also covered.
else This however does not indicate the correctness of
F2(); implementation. What if the conditional
statement was
Test Cases for Condition Coverage:
A=T, B=T
A=F, B=F if(A || B) ?

Yogananda Jeppu – CO366 Formal Methods 13

Modified Condition/Decision coverage
• The MC/DC criterion enhances the condition/decision coverage
criterion by requiring that each condition be shown to independently
affect the outcome of the decision.
• Every input in the logic should effect the outcome clearly.

A B Out
if(A && B)
F F F
F1();
else F T F
F2(); T F F
T T T

Yogananda Jeppu – CO366 Formal Methods 14

MC/DC Tests
• To test if (A or B)
A: T F F
B: F T F

• To test if (A and B)
A: F T T
B: T F T

• To test if (A xor B)
A: T T F
B: T F T

Yogananda Jeppu – CO366 Formal Methods 15

Test Case

A B C A xor B NOT(A xor B) C' O

0 0 0 0 1 1 1

0 0 1 0 1 0 0

0 1 0 1 0 1 0 2

0 1 1 1 0 0 0

1 0 0 1 0 1 0 3

1 0 1 1 0 0 0

1 1 0 0 1 1 1 1

1 1 1 0 1 0 0 4

Yogananda Jeppu – CO366 Formal Methods 16

Multiple Condition coverage
• Multiple condition coverage requires test cases that ensure each
possible combination of inputs to a decision is executed at least once;
that is, multiple condition coverage requires exhaustive testing of the
input combinations to a decision.
• For a decision with n inputs, multiple condition coverage requires 2^n
tests.
• This can be really huge. Its okay for small number of conditions in a
decision.

Yogananda Jeppu – CO366 Formal Methods 17

Assignment
• Look at the testing of RGCS. What have you covered in your tests?
• Make a MC/DC test case for this requirement
• While gate is opening, If gate Is not open after a duration of 12
seconds the RGCS shall sound an alarm

• If RGCS_gate_open AND NOT gate_open AND timer > 12 then

alarm = TRUE;
RGCS_gate_open is the gate open command to
Else open the gate, gate_open is the Boolean form
gate model, timer is the counter that you make
alarm = FALSE; for 12 sec

Yogananda Jeppu – CO366 Formal Methods 18

Summary
• We have looked at the Finite State machine
• We have looked at how the code can be done
• We have looked at the test coverage metrics

• The assignments should give you a good insight into the system
working

Yogananda Jeppu – CO366 Formal Methods 19