Simulation based verification:

coverage

Verification
coverage
estimates the
quality of
simulation

Digitaalsüsteemide verifitseerimise kursus 1

 Coverage metrics
• System can’t be simulated with all possible
stimuli
• Question: how many stimuli are needed?
• Coverage metrics used
• Coverage shows our confidence in how
thoroughly we verified

Digitaalsüsteemide verifitseerimise kursus 2

 Coverage metrics

• Three types:
– Code coverage
– Parameter coverage
– Functional coverage
• Also will talk about
– Domain coverage, cross coverage, stuck-at fault
coverage

Digitaalsüsteemide verifitseerimise kursus 3

Coverage metrics: code coverage

• Code coverage shows how thoroughly is the

code examined during simulation
 Based on implementation only
 Easy to compute

Digitaalsüsteemide verifitseerimise kursus 4

Coverage metrics: parameter coverage

• How well parameters and dimensions

covered in functional blocks
• Based on implementation, spec or both

Digitaalsüsteemide verifitseerimise kursus 5

 Coverage metrics: functional
coverage
• Based on behaviors derived from the spec
• Shows the number of covered behaviors
• Problem: how to formally generate the
behaviors and guarantee that the full
functionality is covered…

Digitaalsüsteemide verifitseerimise kursus 6

 Comparison: coverage metrics
• None of the three types is universally best
– Advantages of code coverage are simplicity and
formalisation
– Advantage of functional coverage is high
verification power
– Parameter coverage is a trade-off between the two
• Coverage metrics may be used for guiding test
generation
• To reach 100 % coverage is very much harder
than to reach 90 %...
Digitaalsüsteemide verifitseerimise kursus 7
 Code coverage: statement coverage

• All lines except

begin, end, else
• Example: a > x, x = y
• Statement coverage
8 out of 10 = 80%

Digitaalsüsteemide verifitseerimise kursus 8

Code coverage: block coverage

Digitaalsüsteemide verifitseerimise kursus 9

 Instrumented code
• Instrumented code applied in code
coverage analysis. Thus, simulation
slowed down considerably.
• Turn off the code already covered by
simulation
• In that case almost no extra time penalty
towards the end of simulation

Digitaalsüsteemide verifitseerimise kursus 10

Instrumented code

Digitaalsüsteemide verifitseerimise kursus 11

Code coverage: branch and path

• More stringent than

statement coverage
• Path coverage does
not scale

Digitaalsüsteemide verifitseerimise kursus 12

Code coverage: expression coverage
• Again, more stringent than statement coverage
• We analyze which expressions were evaluated
• Expressions divided into layers:
E = (x1x2 + x3x4)
layer 1: E = y1 + y2
layer 2: y1 = x1x2, y2 = x3x4
• With multi-layer expressions, either the top
layer or all layers analyzed

Digitaalsüsteemide verifitseerimise kursus 13

 Code coverage: expression coverage
• Minimum input tables applied:
f = (y ? x1 : x2) y x1 x2
x2
f = x1 & x2 x1 - 1 1 1
0
0 - 0 1 0
0 0 1 0 1
1 1 1 0 0 0

• Expression x1 & x2 and two

f = (x > y) x > y meaning
input stimuli: (x1=1,x2=0) and
0 0 x <= y (x1=1,x2=1)
1 1 x>y
• In that case the expression
coverage for & is 66.67%
Digitaalsüsteemide verifitseerimise kursus 14
Code coverage: state coverage

• Sequence
b,a,b,a,b,a,b,a,... gives
66.67% of state
coverage, as S1 and S2
traversed (2 out of 3).
• S4 is illegal state, thus
max state coverage is
75%!

Digitaalsüsteemide verifitseerimise kursus 15

Code coverage: transition coverage

• Sequence
b,a,b,a,b,a,b,a,... gives
40% of transition
coverage as 2 out of 5
transitions traversed.
• Transition S2/b not
defined!
• Calculated by transitions
specified (5) or all
possible transitions (6)

Digitaalsüsteemide verifitseerimise kursus 16

Code coverage: sequence coverage

• User can specify state sequences at the

basis of the coverage calculation
• Normally essential functionality or
corner cases specified
• Possible to keep track on which desired
(or undesired) sequences were simulated

Digitaalsüsteemide verifitseerimise kursus 17

Code coverage: toggle coverage

• Measures how many bits in the signals

changed their state between 0 and 1
• E.g. a signal for a 4-bit bus and values
0000 and 1101 would give a toggle
coverage of 75%

Digitaalsüsteemide verifitseerimise kursus 18

 Parameter coverage

Parameter Range
Server configuration {(S1,S2): S1, S2 passive or active}
Task processing time {3,...,6}
Length of the queue {(m1,m2,m3): 1  mi  8, 1  i  3 }

Digitaalsüsteemide verifitseerimise kursus 19

 Parameter coverage: example
Parameter Range
Server configuration {(S1,S2): S1, S2 passive or active}
Task processing time {3,...,6}
Length of the queue {(m1,m2,m3): 1  mi  8, 1  i  3 }

• If simulation provides 2 server configurations: (S1

active, S2 passive) ja (S1 active, S2 active); processing
times 3, 4 and 5; maximum que lengths (6,4,8), then ...
• ... Parameter coverages: server conf. 50 %, processing
times 75 %, queue lengths 75%,50%,100%

Digitaalsüsteemide verifitseerimise kursus 20

 Functional coverage
• Functional coverage based on behaviors
derived from the specification
• Functional coverage reports the ratio of
covered behaviors
• Problem: how to formally generate
behaviors and guarantee that the entire
functionality is covered

Digitaalsüsteemide verifitseerimise kursus 21

 Cross coverage

• Cartesian product of 2 or more coverages

• More stringent than using just a single
coverage metric
• Cartesian product of several coverages may
represent a huge space

Digitaalsüsteemide verifitseerimise kursus 22

 Domain coverage (boundary coverage)

• Every condition defines a partition of the input domain

• Small design error may slightly shift the partitioning

foo (in1, in2) out=1

if (in1 < in2) in2
out = 1;
else out=0
out = 0;
in1

• What if in1 < in2 should be rather in1 <= in2?

• What if the value of in1 or in2 is slightly wrong?

Digitaalsüsteemide verifitseerimise kursus 23

 Domain coverage: Setting up a test

• Testing with a value at the boundary and with a value

immediately off the boundary

foo (in1, in2) in2 out=1

if (in1 < in2)
out = 1; out=0
else 2
out = 0; 1

1 in1

•Test 1: in1, in2 = 1,1 – false, at the boundary

•Test 2: in1, in2 = 1,2 – true, immediately off the boundary

Digitaalsüsteemide verifitseerimise kursus 24

 Domain coverage: example
• Per each condition (predicate) two tests:
– True and immediately off the boundary
– False and immediately off the boundary
Test 1: in1, in2 = 0,0
a = in1 + in2; • c = 0, and a = 0
b = 0; c = 0; • false and at the boundary for c < a
while (c < a) • false and at the boundary for c < in2
c = c + in1;
if (c < in2) Test 2: in1, in2 = 1,0
out = a + b; • c = 0, and a = 1
else • true and at the boundary for c < a
out = a + c; • c set to 1
• false and at the boundary for c < in2

• 3 conditions out of 4 satisfied, 75% domain coverage

Digitaalsüsteemide verifitseerimise kursus 25

 Issues with domain coverage
• Non-linear boundaries
Partly linear boundaries: test each part separately

in2
(in2 > in1) and (in1 > 2)

2 in1
Higher order boundaries: no standard solutions

(in2 > in12) in2

in1

Digitaalsüsteemide verifitseerimise kursus 26

 Observation coverage
• Code coverage ignores the observability aspect
• It may happen that the covered code items have
no influence to any observable point
• It has been showed that 100% code coverage
may result in 60-80% observation coverage in
the worst case [Fallah, Devadas, Keutzer
DAC’98].
• Thus, we should consider propagating the bug
effect to an observable output
Digitaalsüsteemide verifitseerimise kursus 27
 Other metrics: stuck-at coverage

• Assume that a bug manifests itself as a

signal line being constantly stuck to 0 (or 1)
• Assune single faults, i.e. only one signal
line is faulty at a time
• Stuck-at coverage indicates how well we
have excercised the structure

Digitaalsüsteemide verifitseerimise kursus 28

 Issues with coverage
• None of the coverage metrics provides
100% of confidence in correct functionality
• E.g. 100 % stuck-at coverage test for OR
gate does not discover the design error
OR XOR
x1 x2 x1 V x2 x1 x2 x1  x2
0
1
1
0
1
1
= 0
1
1
0
1
1
0 0 0 0 0 0

Digitaalsüsteemide verifitseerimise kursus 29

 Issues with coverage
• Similar problem with expression coverage
• 100 % expression coverage for OR gate
does not discover the error OR XOR
x1 x2 x1 V x2 x1 x2 x1  x2
0
1
1
0
1
1
= 0
1
1
0
1
1
0 0 0 0 0 0

Thus, we need to consider a set of coverages!

Digitaalsüsteemide verifitseerimise kursus 30