Introduction to design verification

12/17/15 1
Agenda
 Why Verification?
 Verification Alternatives
 Languages for System Modeling and Verification
 Concluding Remarks

12/17/15 2
 Why Verification
 Goal of verification:
 Demonstrate functional correctness of a design
 Attempt to find design errors
 Attempt to show that design implements specification
 Importance of Verification
 Costs of design errors can be high
(think “Pentium Floating-Point Error” ~ $475M!)
 According to [1], verification consumes about 70-80% design effor
t in current systems design

[1] J. Bergeron, Writing Testbenches: Functional Verification of HDL Models Kluwer Acade
mic Publishers, 2000 .

12/17/15 3
 Verification – Reconvergence Model

 Verification checks a “transformation” for correctness

 RTL Design and Coding
 Synthesis
 Physical Design

 Reconvergence Model:
Transformation

Initial Transformation
Specification Result

Transformation

12/17/15 4
 Verification of RTL Design

 The Idea:
RTL Coding

Written
RTL Code
Specification

Verification

 How it works
RTL Coding

Interpretation
Written RTL Code
Specification

Verification

12/17/15 5
 RTL Synthesis flow

12/17/15 6
 Verification in the design cycle

12/17/15 7
Trend of Verification Effort in the Desig
n
 Verification portion of design increases to anywhere fr
om 50 to 80% of total development effort for the desi
gn.

Earlier
Code Verify (30 ~ 40%) Synthesis P&R

Now Code Verify (50 ~ 80%) Synthesis P&R

Verification methodology manual, 2008-

TransEDA

12/17/15 8
Percentage of Total Flaws
 About 50% of flaws are functional flaws.
 Need verification method to fix logical & functional flaws

From Mentor presentation material, 2003

12/17/15 9
Some more interesting stats ……..
 Another recent independent study showed that mor
e than half of all chips require one or more re-spin
s, and that functional errors were found in 74% of t
hese re-spins.
 With increasing chip complexity, this situation could
worsen.
 Who can afford that with >= 1M Dollar NRE cost?

12/17/15 10
Bug Fixing Cost in Time
 Cost of fixing a bug/problem increases as design progr
esses.
 Need verification method at early design stage

Cost of
Fixing
a Problem

Behaviora RTL Gate Device

12/17/15 l Design Level Production
11
Design Design
 Testbench: Basic building block
 A testbench is HDL code to verify a module
 Apply input vectors to module inputs
 Check module outputs
 Report errors to user
 Why use a testbench ?
 Portability - testbench will work on any HDL simulator
 Automatic checking - don't have to interpret waveform
 Expressability - can use the full semantics of HDL to:
 generate input vectors (possibly from input file)
 check output vectors
 control simulation

12/17/15 12
 Simple HDL Testbench

Testbench Module

Module Instance:
Device
Under
Verification
(DUV)

12/17/15 13
 Comparing approaches
 Visual inspection
 Only practical for small designs
 Automatic support: timing diagram editor
 Output comparison
 Effective when a good reference model is available
 Used by ASIC foundries - “Gold” vectors are legal definition o
f a “functional” chip
 Output checking
 Most difficult to code
 Mandatory for large designs

12/17/15 14
 Testbench Approaches -
Visual Inspection

Testbench File

Device
Stimulus Waveform Viewer
under
Generator OR
Verification
Text Output
(DUV)

12/17/15 15
 Testbench Approaches
Output Comparison

Testbench File
“Gold”
Vectors
Reference
Model

Output Error/Status
Comparator Messages
Device
Stimulus under
Generator Verification
(DUV)

12/17/15 16
 Testbench Approaches
Self-Checking
Testbench File
Input Signals
Output
Signals

Device Error/Status
Stimulus under Output
Checker Messages
Generator Verification
(DUV)

12/17/15 17
 What a testbench is supposed to do?
 Self-checking testbenches
 Identify important features
 Create conditions that test these features
 Check conditions
 Write message when error occurs
 “Insert” errors to demonstrate when self-check fails
 Test for varying values of all possible input values

12/17/15 18
Completion Metrics; How do we know wh
en the verification is done?
 Emotionally or Intuitively;
 Out of money? Exhausted?
 Competing product is there.
 Software people are happy with your hardware.
 There have been no bugs reported for two weeks.

 More rigorous criteria;

 All tests passed
 Functional Coverage
 Code Coverage
 Bug Rates have flattened toward bottom.

12/17/15 19
Agenda
 Why Verification ?
 Verification Alternatives
 Simulation
 Emulation
 Prototyping
 Formal verification
 Semi-Formal verification
 Languages for System Modeling and Verification
 Concluding Remarks

12/17/15 20
Overview of Verification Methodologi
es

p ro duct
to f inal
, c loser Prototyping
e ed
er sp
F a st Hardware
Emulation
Accelerated
Simulation
Simulation
Basic Semi-formal
verification Verification
tool Formal
Bigg Verification
e r cov
erag
e

12/17/15 21
 Software Simulation
 Dynamic verification method
 Bugs are found by running the design implementatio
n.
 Thoroughness depends on the test vector used.
 Some parts are tested repeatedly while other parts ar
e not even tested.
Other parts
Testbench DUV are not even
tested.
a = 1;
#20 b = 1; Some part of
the design is
$display (“status is = %d”,c);
tested
... repeatedly.

12/17/15 22
Cycle-Based Simulation
 Simulate the behavior of the design cycle-by-cycle.
 Cycle-accurate information is provided as a result of
simulation.
 Only signals at the flip-flop input are evaluated to be
stored, not internal signals of combinational logic.

Combinational Combinational
Combinational
logic logic
logic

12/17/15 23
Cycle-based vs. Event-driven

Cycle-based Event-driven
Timing resolution Clock cycle User-defined minimum
delay
Evaluation time Rising/falling/bot At the occurrence of
point h clock edges events
Evaluation node Every flip-flop At the output of every
boundary logic gate on the
event propagation
path
Simulation time Proportional to Proportional to the
the (number of number of events
cycles) times (circuit size* no. of
(C/L size * cycles*event density)
number of F/F’s)
12/17/15 24
Software Simulation
 Pros
 The design size is limited only by the computing re
source.
 Simulation can be started as soon as the RTL descri
ption is finished.
 Set-up cost is minimal.
 Cons
 Slow (~100 cycles/sec) ; Speed gap between
the speed of software simulation and real
silicon widens. (Simulation speed = size of
the circuit simulated / speed of the simulation
engine)
 The designer does not exactly know how
12/17/15 25
much percentage of the design have been
Emulation
 Imitating the function of another system to achieve th
e same results as the imitated system.
 Usually, the emulation hardware comprises an array o
f FPGA’s (or special-type processors) and interconnect
ion scheme among them.
 About 1000 times faster than simulation.

Prototyping
Emulation
Hardware
Accelerated
Simulation
Simulation
12/17/15 26
Emulation
 User logic design is mapped to emulation board with
multiple FPGA’s or special processors.
 The emulation board has external interconnection har
dware that emulates the pins of final chip.

Logic design Emulation hardware with multiple FPGAs

Design
&
mapping
>

& +

12/17/15 External pins

27
Prototyping
 Pros
 Higher (than emulation) clock rate (over 1M cycles
/sec) due to specific design of prototyping board.
 Components as well as the wiring can be customiz
ed for the corresponding application.
 Can be carried along. (Hardware Emulation? Forge
t it!)
 Cons
 Not flexible for design change
(Every new prototype requires a new board archite
cture. / Even a small change requires a new PCB.)

12/17/15 28
Overview of Verification Methodologi
es
 Formal verification
 Application of logical reasoning to the development of digita
l system
 Both design and its specification are described by a language
in which semantics are based on mathematical rigor.
 Semi-formal verification
 Combination of simulation and formal verification.
 Formal verification cannot fully cover large designs, and simu
lation can come to aid in verifying the large design.

Semi-formal Formal
Simulation
Verification Verification
More complete verification
12/17/15 29
Formal Verification
 Objective
 Check properties of model with all possible conditions
 Pros
 Assures 100% coverage.
 Fast.
 Cons
 Works only for small-size finite state systems.
 Uncomfortable due to culture difference (E.g., engineers are
not familiar with the use of temporal logic used for “property
” description in Model Checking)

12/17/15 30
Formal Verification : Equivalence Checker
 Equivalence checker compares the golden model with the refined
model.

?
=
Golden
Golden Refined
Refined
Model
Model Model
Model

 Functional representations are extracted from the designs and co

mpared mathematically.
 Pros
 Exhaustive design coverage
 Very fast
 Cons
 Memory explosion
 Tools such as LEC (Verplex), Formality (Synopsys), FormalPro (Me
ntor) supports Equivalence checking.
12/17/15 31
Semi-Formal Verification - Assertion
 Assertion-based verification (ABV)
 “Assertion” is a statement on the intended behavior of a desi
gn.
 The purpose of assertion is to ensure consistency between th
e designer’s intention and the implementation.
 Key features of assertions
 1. Error detection : If the assertion is violated, it is detected b
y the simulator.
 2. Error isolation : The signals related to the violated assertio
n are identified.
 3. Error notification : The source of error is reported to the u
ser.

12/17/15 32
Semi-Formal Verification - Assertion
 Example of assertion-based bug detection

Identify signals related to

the violated assertion
PCI
PCIDMA
DMAController
Controller
PCI
event devsel :
if (FRAME=0) [1..4]
“devsel” (DEVSEL=0)
assertion is assert(devsel);
violated!

Report to the
user!!
12/17/15 33
Semi-Formal Verification - Assertion
 Simulation Quality of assertion-based verification
Number of bugs found

Simulation with assertions

Efficiency of
assertion

Formal verification
Simulation

Time, Effort
Setup Describe
testbench assertions By IBM in “Computer-Aided Verification” 2000
12/17/15 34
Semi-Formal Verification - Coverage
 Coverage-directed verification
 Increase the probability of bug detection by checking the ‘qual
ity’ of stimulus
 Used as a guide for the generation of input stimulus

Test
TestPlan
Plan Random
Random
(Coverage Directives Test
(Coverage Test
Definition)
Definition)
Directives Test
Generator TestVectors
Vectors
Generator

Coverage
Coverage Coverage
Reports Coverage Simulation
Reports analysis Simulation
analysis

12/17/15 35
Semi-Formal Verification - Coverage
 Coverage metrics for coverage-directed verification
 Code-based metrics
 Line/code block coverage
 Branch/conditional coverage
 Path coverage
 Circuit structure based metrics
 Toggle coverage
 Register activity
 State-space based metrics
 Pair-arcs : usually covered by Line + condition coverage
 Functional coverage metrics
 % of specification items satisfied

12/17/15 36
Semi-Formal Verification - Coverage
 Coverage Checking tools
 VeriCover (Veritools)
 SureCov (Verisity)
 Coverscan (Cadence)
 HDLScore, VeriCov (Summit Design)
 HDLCover, VeriSure (TransEDA)
 Polaris (Synopsys)
 Covermeter (Synopsys)

12/17/15 37
Semi-Formal Verification
 Pros
 Designer can measure the coverage of the test environment
as the formal properties are checked during simulation.
 Cons
 The simulation speed is degraded as the properties are check
ed during simulation.
 Challenges
 There is no unified testbench description method.
 It is difficult to guide the direction of test vectors to increase
the coverage of the design.
 Development of more efficient coverage metric to represent
the behavior of the design.

12/17/15 38
Speed Comparison
Speed (Cycles/sec, log scale)

10MHz
1~10MHz

1MHz
500KHz

100kHz
100 kHz

10 kHz

100Hz 50-70Hz
0 kHz
Software Hardware- Hardware Prototyping Semi-formal
Simulation Accelerated emulation (Assertion-
Simulation (from Quickturn based
(from presentation) verification)
12/17/15 Quickturn/Dynalith 39
Presentation)
Design Complexity
Gate counts Comments

Simulation/Semi- Unlimited
formal verification

Emulation/Hardwar 1M~16M gates Depends on the

e-accelerated number of FPGA’s in
simulation the architecture
Prototyping 1M~5M gates Depends on the
components on the
board

Formal verification < 10K gates Limited to about 500

flip-flops due to state
explosion

12/17/15 40
Verification Time vs. Coverage
Coverage

Semi-formal

Emulation
/Accelerated simulation
Prototyping

Simulation

Redirection
of
testbench
constraints

Verification Time
12/17/15
Simulatio Semi-formalEmulation Prototypin 41
n setup setup setup g setup
Agenda
 Why Verification ?
 Verification Alternatives
 Languages for System Modeling and Verification
 System modeling languages
 Testbench automation & Assertion languages
 Concluding Remarks

12/17/15 42
Language Heritage for SoC Design
 New languages are developed to fill the productivity g
ap.

Language for JAVA

Software development
C++
Assembly C
SystemC
Language for
TestBuilder
Hardware test

Vera
SystemVerilog
Language for
Verilog
Hardware description

Schematic VHDL
12/17/15 43
Past present Future
System Description Languages Summar
y
Languge Pros Cons
C/C++ • Easy to write test • Unable
to handle some
vectors/environmen hardware
t environments.
HDL • Familiarity • Focuses on the lower-
(Verilog, • Easy to describe H/W level designs.
VHDL) designs • Improper for system
modeling.
SystemC • Easilyconnected to • Limited
tools
C/C++ codes. (simulation, synthesis,
• Easy to model system etc.)
behaviors.
SystemVerilo •Easy to learn for the • Few tools (simulation,
g HDL designers. synthesis, etc.)
•Easy to model system • Subset support
12/17/15 44
behaviors.
ASIC Verification Methods

Running Speed

100MHz Make it faster Real Silicon

Ideal Verification
10MHz Solution
Rapid Prototype
1MHz
Make it cheaper HW Emulator
100KHz
10KHz
1KHz HW Accelerator
100Hz
SW Simulator
10Hz

Investment
12/17/15 45
 Overall Functional Verification Flow

Architecture Define

Microcode RTL Description

Description (Verilog HDL)

Synthesis
Microcode
Verifier RTL Simulation
Gate Level
Simulation

For version
Hardware control
Emulation

12/17/15 Verification Completed 46

Concurrent Verification

Without Emulation Sequential

Verification
System
SW Design Code Debug
Integration

Hardware
HW Design Build Debug
Integration

CHIP Design Fab Debug

Back
annotation Time

With Emulation Concurrent

Verification
SW
Design Code

HW Design Build

CHIP Design Fab

Early to
HW integration Market!!
Final
HW emulation Chip & HW Debug
Integration Debug
Debug Sys integration
& SW Debug

12/17/15 Back annotation 47

Agenda
 Why Verification ?
 Verification Alternatives
 Languages for System Modeling and Verification
 Concluding Remarks

12/17/15 48
Concluding Remarks
 Verification is challenging; It needs strategy!
 Strategy is to apply each method when appropriate.
 Verify as early as possible; Catch the bug when it is small and still
isolated in a smaller region. (Don’t wait until it grows and kills yo
u.)
 1st step: Apply formal methods
 Static formal verification
 Assertion-based verification
 2nd step: Coverage driven verification aids in bringing closure
 Code and functional coverage , if not covered design doesn’t work !!
 Selecting the proper verification methodology
 3rd step: Emulate design
 Emulate IP operation in FPGA

12/17/15 49
Concluding Remarks
 Powerful debugging features handling both hardware part and s
oftware part are required.
 Language, Tool/Data Interfaces need standardization.
 DFV (Design for Verification) ; You lose in the beginning, but will
win later, like Design for Reuse.

12/17/15 50
 Thank You!!

12/17/15 51
 Questions ???

12/17/15 52
 Appendices

12/17/15 53
Debugging Design in the FPGA
 Embed logic analyzer with user design in EDIF format
 Logic to store pre-registered signals into the probing memory.
 Logic for trigger condition generation.
 Triggering condition is dynamically configured.
 Internal node extraction
 Sometimes the designer wants to watch internal nodes in the
design.
 Internal node probing Top block DUT
Built-In
enables this by Logic
wiring-out the internal Sub-block Analyzer
nodes to the boundary
of the DUT top block.
External
Internal node Dump
BILA, Dynalith Systems
Memory
12/17/15 54
RTL Debugging Feature
 Emulation is based on gate-level netlist.
 Gate-level netlist generated from the synthesis tools h
as too complex name styles difficult to trace manually.
 Techniques to resolve RTL symbol names from the gat
e-level symbol names and to provide debugging enviro
nment in RTL name spaces is required.
 Insert RTL instrumentation IP for debugging
 Design flow
 Read RTL design (Verilog, VHDL)
 Generate instrumented RTL design (spiced with triggering and d
ump logic)
 Synthesis
 Compile (mapping & PAR)
 DiaLite (Temento), Identify (Synplicity)

12/17/15 55
RTL Debugging Feature
 Instrumentation IPs for debugging logic blocks mappe
d into FPGAs.
 Trigger Structures of the RTL design
 Logic Equation Module
 History Register
 Transaction Register
 Random Generator
 Traffic Analyzer
 Instrumentation IPs are
interconnected to
support various
configurations.
Interconnection of instrumentation IPs

DiaLite from Temento

12/17/15 56
Connecting Actual Chip to the Simulator
 Building a correct and fast reference model for the hardware is v
ery difficult.
 Use the actual discrete chip
for the IP (or FPGA).

 Control the clock signal to the

actual chip (or FPGA), i.e,
slow down and
synchronize with the HW simulator
and SW debugger
in the host machine.

 Application
 FPGA prototyping
 HW/SW co-verification
 Silicon validation

12/17/15 57 from SimPOD

 Microprocessor Design Verification Methodolo
gy
more refined model
CPU
C Language HDL

Instruction
Micro-
Behavior RT-Level Gate-Level
architecture
In C in Verilog in Verilog
in C
(Polaris)

Virtual Chip FlexPC MCV Using

PLI

Real
Virtual PC in C language
Mother-board
(VPC)
H/W

Peripherals
12/17/15


MCV : Microcode Verifier
58
PLI : Programming Language
Interface
Prototyping
 Special (more dedicated and customized) hardware a
rchitecture made to fit a specific application.

Prototyping
Emulation
Hardware
Accelerated
Simulation
Simulation

12/17/15 59
A Prototyping Example
Switch board
 Prototype of 4-Port G
igabit Ethernet Switc
h
 Two Xilinx Virtex-E 20
00 FPGAs are on FPG
A board.
 Four FPGA boards are
used.
Xilinx FPGA
 Processor board cont
ains PCI bridge and M Processor board
PC860 microprocesso
PCI bridge
r.

MPC860
Courtesy of Paion, Inc. microprocessor
12/17/15 60