Verification Planning and Management Introduction

Plan of Attack
Peet James Senior Verification Consultant
academy@mentor.com www.verificationacademy.com

Verification Effort Summary

Why Plan?
Because verification takes more time & effort than design Because it takes a separate, informed, documented architecture, process and team Because ad-hoc verification will lead to minimal results and much wasted effort

Why is it Hard?
Because there is too much to do Because people resist change Because it more than just tools & training, it also takes deployment Because complex change takes time and happens in stages Because it is a steep learning curve

Verification Effort Summary

Its about:
Not re-inventing the wheel Making informed, strategic decisions early Avoiding pitfalls, and having to re-coding Finding and fixing problems early Coding for debug Choosing your verification battles Setting up for successful feedback & guidance Ramping up quickly Managing Change Organizing large quantities of data Defining & clarifying processes & problems Downsizing larger problems into smaller, doable problems

Engineering is about:

Verification Effort Summary

Its about:
Not re-inventing the wheel Making informed, strategic decisions early Avoiding pitfalls, and having to re-coding Finding and fixing problems early Coding for debug Choosing your verification battles Setting up for successful feedback & guidance Ramping up quickly Managing Change Organizing large quantities of data Defining & clarifying processes & problems Downsizing larger problems into smaller, doable problems

Engineering is about:

Bottomline: We need a PLAN OF ATTACK to verifying the most in the allotted time

Verification Plan of Attack

Plan

Big Picture
Architecting a verification solution 30,000 ft to 1,000 ft view

Verification Plan of Attack

Plan

Populate

Big Picture
Architecting a verification solution 30,000 ft to 1,000 ft view

Details
Requirements

1,000 ft to 1 ft view

Verification Plan of Attack

Plan

Populate

Pilot

Big Picture
Architecting a verification solution 30,000 ft to 1,000 ft view

Details
Requirements

Day to Day
Managing The entire Verification System Start To finish

1,000 ft to 1 ft view

Verification Planning: PLAN

Plan

Big Picture

Big Picture
Architecting a verification solution 30,000 ft to 1,000 ft view

Mission & Goals Strategizing Architecting Layers Phases What and How and in What order we will build the testbench?

Q&A

Verification Planning: PLAN

Plan

Big Picture

Main Areas of Focus

Big Picture
Architecting a verification solution 30,000 ft to 1,000 ft view

Mission & Goals Strategizing Architecting Layers Phases

Generation Checking: Scoreboard, Assertions Coverage Hardest part to automate ReUse Libraries: AVM, OVM

Q&A

Automation

What and How and in What order we will build the testbench?

Verification Planning: PLAN

Plan

Output:

Big Picture
Architecting a verification solution 30,000 ft to 1,000 ft view

VAD = Verification Architect Document VIDs = Verification Implementation Docs VI = Verification Infrastructure Deliverables:

VAD / VID
Verification Infrastructure

Basic Strategy / Approach Basic Structure Block Diagram List of Components Line & Effort Estimates that can lead to a schedule Basic OOP setup: UMLs & Layers Basic Build Order Phases & Milestones Basic ReUse

Verification Planning: POPULATE Populate

Requirements

Details
Requirements

Extract Refine Prioritize Track

1,000 ft to 1 ft view

Q&A: What & When

specifically will we?

Apply to the DUT Check in the scoreboard Check with assertions Cover

Verification Planning: POPULATE Populate

Requirements

Main Areas of Focus

Details
Requirements

Extract Refine Prioritize Track

1,000 ft to 1 ft view

Q&A: What & When

specifically will we?

Generation Checking: Scoreboard, Assertions Coverage Grow your own: ACCESS, SQL, QuestaVerification Mgmt with UCDB ReqTracer

Automation:

Apply to the DUT Check in the scoreboard Check with assertions Cover

Verification Planning: POPULATE

Extraction / Prioritizing / Implement

Populate

DR Extraction DR Prioritization Implementation VR Translation

DR Gen/Chk/Cov VR

Details
Requirements

1,000 ft to 1 ft view

VR Mapping:
VR DR VR VI

Design Requirement Verification Requirement

Verification Planning: POPULATE

Extraction / Prioritizing / Implement

DR DR DR DR DR DR

Populate

DR Extraction DR Prioritization Implementation VR Translation

DR Gen/Chk/Cov VR

Details
Requirements

DR

1,000 ft to 1 ft view

VR Mapping:
VR DR VR VI

Design Requirement Verification Requirement

Verification Planning: POPULATE

Extraction / Prioritizing / Implement

DR DR DR DR DR DR

Populate

DR Extraction DR Prioritization Implementation VR Translation

DR Gen/Chk/Cov VR VR

Details
Requirements

DR

1,000 ft to 1 ft view

VR

VR

VR Mapping:
VR DR VR VI

VR

VR

VR

Design Requirement Verification Requirement

VR

Checking: SB / Assert

VR

Generation

Cover

Verification Planning: POPULATE Populate

Details
Requirements

Outputs

Database: xls, UCDB, etc. Reports List of Generation Items Sequences / Scenarios List of Scoreboard Elements List of Assertions Areas List of Coverage Areas

Deliverables: General

1,000 ft to 1 ft view

Detailed Requirements Custom Docs

Verification Planning: PILOT

Pilot

Day to Day
Managing The entire Verification System Start To finish

Day to Day Management

Document Run & Control Analyze & Debug Measure

Control & keep track of everything Repeat the results? Know how we are doing? Know we are done?

Q&A: How do we?

Verification Planning: PILOT

Pilot

Day to Day
Managing The entire Verification System Start To finish

Day to Day Management

Main Areas of Focus

Document Run & Control Analyze & Debug Measure

Control & keep track of everything Repeat the results? Know how we are doing? Know we are done?

Q&A: How do we?

Staff & Schedule Rules of Engagement Run Management Triage Metrics Templates & Conventions Scripts Tools: Bugzilla, VRM, SVN, etc/

Automation

Verification Planning: PILOT

Pilot

Outputs

Day to Day
Managing The entire Verification System Start To finish

Docs: File Structure, Templates, Conventions Auto Docs: Schedule, Status Reports Script Code HDL, HVL Code Repeatability Regression Management Change & Release Management Data: Gather, Measure, Respond

Deliverables

Docs Code Scripts

Verification Plan of Attack Plan Big Picture Architecting a verification solution 30,000 ft to 1,000 ft view VAD / VID Populate Details Requirements Pilot Day to Day Managing The entire Verification System Start To finish Docs Code Scripts

1,000 ft to 1 ft view
Detailed Requirements

Verification Infrastructure

Custom Docs

Verification Architecture & Implementation Docs

Verification Flows
Plan

START

Populate

Pilot

END

Verification Flows
Plan

START

Populate

Pilot

END

START

Populate

Plan

Pilot

END

Verification Flows
Plan

START

Populate

Pilot

END

START

Populate

Plan

Pilot

END

START

Plan

Populate END
Pilot

Verification Flows

START

Plan

Populate

Pilot

END

Verification Plan of Attack: Methodologies

Plan Big Picture Architecting a verification solution 30,000 ft to 1,000 ft view VAD / VID Populate Details Requirements Pilot Day to Day Managing The entire Verification System Start To finish Docs Code Scripts

1,000 ft to 1 ft view
Detailed Requirements Custom Docs

Verification Infrastructure

Verification Plan of Attack: Methodologies

Plan Big Picture Architecting a verification solution Populate Details Requirements Pilot Day to Day Managing The entire Verification System Start To finish Docs Code Scripts

DDV=Document Driven Verification

30,000 ft to 1,000 ft view VAD / VID

1,000 ft to 1 ft view
Detailed Requirements

Verification Infrastructure

Custom Docs

Verification Plan of Attack: Methodologies

Plan Big Picture Architecting a verification solution Populate Details Requirements Pilot Day to Day Managing The entire Verification System Start To finish Docs Code Scripts

30,000 ft to 1,000 ft view VAD / VID

1,000 ft to 1 ft view
Detailed Requirements

Verification Infrastructure

Custom Docs

RDV=Requirement Driven Verification

DDV=Document Driven Verification

Verification Plan of Attack: Methodologies

Plan Big Picture Architecting a verification solution Populate Details Requirements Pilot Day to Day Managing The entire Verification System Start To finish Docs Code Scripts

30,000 ft to 1,000 ft view VAD / VID

1,000 ft to 1 ft view
Detailed Requirements

Verification Infrastructure

Custom Docs

RDV=Requirement Driven Verification

DDV=Document Driven Verification

CDV=Coverage Driven Verification

Verif Manager, UCDB

Verification Plan of Attack: Methodologies

Plan Big Picture Architecting a verification solution Populate Details Requirements Pilot
MDV=Metric Driven Verification

RDV=Requirement Driven Verification

DDV=Document Driven Verification

Verif Run Manager

30,000 ft to 1,000 ft view VAD / VID

1,000 ft to 1 ft view
Detailed Requirements

Docs Code Scripts

Verification Infrastructure

Custom Docs

CDV=Coverage Driven Verification

Verif Manager, UCDB

Triage Analylsis

Day to Day Managing The entire Verification System Start To finish

Plan

Verification Plan of Attack: Pilot Populate Methodologies

Details Day to Day
MDV=Metric Driven Verification

Big Picture

RDV=Requirement Driven Verification

DDV=Document Driven Verification

Architecting a verification solution 30,000 ft to 1,000 ft view

Requirements

Verif Run Manager

1,000 ft to 1 ft view

VAD / VID
Verification Infrastructure

Detailed Requirements Custom Docs

Docs Code Scripts

CDV=Coverage Driven Verification

Verif Manager, UCDB

Triage Analylsis

Managing The entire Verification System Start To finish

VPM Outline

Plan
DDV=Document Driven Verification

Populate

Pilot
MDV=Metric Driven Verification

RDV=Requirement Driven Verification CDV=Metric Driven Verification

Audit Strategize Architect VAD -Verification Architecture Doc VID Verification Implementation Doc Verification Infrastructure (VI)

Discussions

VPM Outline

Plan
DDV=Document Driven Verification

Populate

Pilot
MDV=Metric Driven Verification

RDV=Requirement Driven Verification CDV=Metric Driven Verification

Audit Strategize Architect VAD -Verification Architecture Doc VID Verification Implementation Doc Verification Infrastructure (VI)

Discussions

Discussions Requirements: Hunt, Refine, Prioritize, Track Design Requirements Verification Requirements Translation & Mapping

VPM Outline

Plan
DDV=Document Driven Verification

Populate

Pilot
MDV=Metric Driven Verification

RDV=Requirement Driven Verification CDV=Metric Driven Verification

Audit Strategize Architect VAD -Verification Architecture Doc VID Verification Implementation Doc Verification Infrastructure (VI)

Discussions

Discussions Requirements: Hunt, Refine, Prioritize, Track Design Requirements Verification Requirements Translation & Mapping

Discussions Staffing & Scheduling Verification Run Manager Triage & Bug Tracking Change Mgmt Metrics & Dashboards

VPM Deployment Paths - VPMlite

Plan
Why Plan? Why Verification is hard? How to do a audit? How to make a VAD? VAD Example

Populate
Requirements hunting overview Requirements aproaches Requirements example

Pilot
Scheduling overview Staffing overview Metric overview

Approach

Hit highlights only Introduction only All lecture, no application Half day to one day

VPM Deployment Paths - VPMlite

Plan
Why Plan? Why Verification is hard? How to do a audit? How to make a VAD? VAD Example

Populate
Requirements hunting overview Requirements aproaches Requirements example

Pilot
Scheduling overview Staffing overview Metric overview

Approach

Pros

Hit highlights only Introduction only All lecture, no application Half day to one day

Quick Cheap Great intro for further VPM deployment later Good for management Minimal knowledge transfer No direct application Minimal retention

Cons

VPM Deployment Paths - VPMlite

Plan
Why Plan? Why Verification is hard? How to do a audit? How to make a VAD? VAD Example

Populate
Requirements hunting overview Requirements aproaches Requirements example

Pilot
Scheduling overview Staffing overview Metric overview

Approach

Pros

Hit highlights only Introduction only All lecture, no application Half day to one day

Quick Cheap Great intro for further VPM deployment later Good for management Minimal knowledge transfer No direct application Minimal retention

Only Recommended as a teaser to doing a future, real VPM

Cons

VPM Deployment Paths - VPMfull

Plan:
Decide & Document

Build:
Write & Debug VI
VI SV Files

Run:
Find & Fix RTL Bugs

Cover:
Write, Analyze & Steer

Plan

DDV

VAD VIDs Schedule

CoVI Block Diagram mponent List Layers & Phases

TBAI ABV
RQMTs xls

Populate

RDV:

RDV:

RQMTs Database

Hunt, Extract, Refine & Prioritize Translate DRs to VRs & map to VI Staff & Schedule Rules of Engagement Run & Triage Strategy Metric Strategy

Traceability: Schema & Pipeline Link: To coverage & assertions

Pilot

CDV: Verif Manager, VRM, Triage & MDV 1 3 Months Months

UCDB

1 3 Weeks

Months Separate Consulting Modules: TBAI=OVM Testbench ABV = Assertion

Approach Deployment through out the entire project: start to finish Most lectures & topics Fully customized Timeframe: Months

VPM Deployment Paths - VPMfull

Plan:
Decide & Document

Build:
Write & Debug VI
VI SV Files

Run:
Find & Fix RTL Bugs

Cover:
Write, Analyze & Steer

Plan

DDV

VAD VIDs Schedule

CoVI Block Diagram mponent List Layers & Phases

TBAI ABV
RQMTs xls

Populate

RDV:

RDV:

RQMTs Database

Hunt, Extract, Refine & Prioritize Translate DRs to VRs & map to VI Staff & Schedule Rules of Engagement Run & Triage Strategy Metric Strategy

Traceability: Schema & Pipeline Link: To coverage & assertions

Pilot

CDV: Verif Manager, VRM, Triage & MDV 1 3 Months

UCDB

1 3 Weeks

Months

Months Separate Consulting Modules: TBAI=OVM Testbench ABV = Assertion

Approach Deployment through out the entire project: start to finish Most lectures & topics Fully customized Timeframe: Months

Pros
Full verification planning documents: VAD, VIDs & requirements database Maximum knowledge transfer & retention Maximum application to the project Cons: Expense

VPM Deployment Paths - VPMfull

Plan:
Decide & Document

Build:
Write & Debug VI
VI SV Files

Run:
Find & Fix RTL Bugs

Cover:
Write, Analyze & Steer

Plan

DDV

VAD VIDs Schedule

CoVI Block Diagram mponent List Layers & Phases

TBAI ABV
RQMTs xls

Populate

RDV:

RDV:

RQMTs Database

Hunt, Extract, Refine & Prioritize Translate DRs to VRs & map to VI Staff & Schedule Rules of Engagement Run & Triage Strategy Metric Strategy

Traceability: Schema & Pipeline Link: To coverage & assertions

Pilot

CDV: Verif Manager, VRM, Triage & MDV 1 3 Months

UCDB

1 3 Weeks

Months

Months Separate Consulting Modules: TBAI=OVM Testbench ABV = Assertion

Approach Deployment through out the entire project: start to finish Most lectures & topics Fully customized Timeframe: Months

Pros
Full verification planning documents: VAD, VIDs & requirements database Maximum knowledge transfer & retention Maximum application to the project Cons: Expense

Recommended for maximum strategic change

Plan:
Decide & Document

VPM Deployment Paths VPMtypical

Build: Run: Cover:
Write, Analyze & Steer Write & Debug VI Find & Fix RTL Bugs TBAI RDV: RQMTs xls ABV

Plan

DDV VAD

VIDs Schedule

VI Block Diagram Component List Layers & Phases

Populate

Pick DRs or VRs only Example subset Extracted

Pilot

CDV : MDV The Rest of the Project

Staff & Schedule Pick one issue VRM, Triage, MDV, etc

3 Days 2 Weeks

Approach Deployment at the start of a project & then at key times Key lectures & topics Customize for Low Hanging Fruit, best bang for the buck Timeframe: Month Overall

Separate Consulting Modules: TBAI=OVM Testbench ABV = Assertion

Plan:
Decide & Document

VPM Deployment Paths VPMtypical

Build: Run: Cover:
Write, Analyze & Steer Write & Debug VI Find & Fix RTL Bugs TBAI RDV: RQMTs xls ABV

Plan

DDV VAD

VIDs Schedule

VI Block Diagram Component List Layers & Phases

Populate

Pick DRs or VRs only Example subset Extracted

Pilot

CDV : MDV The Rest of the Project

Staff & Schedule Pick one issue VRM, Triage, MDV, etc

3 Days 2 Weeks

Approach Deployment at the start of a project & then at key times Key lectures & topics Customize for Low Hanging Fruit, best bang for the buck Timeframe: Month Overall

Pros

Strategic Verification planning to drive sane deployment Plot for key knowledge transfer & retention Plot for strategic application to the project Cons: Team left to self-guidance on some things

Separate Consulting Modules: TBAI=OVM Testbench ABV = Assertion

Plan:
Decide & Document

VPM Deployment Paths VPMtypical

Build: Run: Cover:
Write, Analyze & Steer Write & Debug VI Find & Fix RTL Bugs TBAI RDV: RQMTs xls ABV

Plan

DDV VAD

VIDs Schedule

VI Block Diagram Component List Layers & Phases

Populate

Pick DRs or VRs only Example subset Extracted

Pilot

CDV : MDV The Rest of the Project

Staff & Schedule Pick one issue VRM, Triage, MDV, etc

3 Days 2 Weeks

Approach Deployment at the start of a project & then at key times Key lectures & topics Customize for Low Hanging Fruit, best bang for the buck Timeframe: Month Overall

Pros

Best Overall Trade Off

Strategic Verification planning to drive sane deployment Plot for key knowledge transfer & retention Plot for strategic application to the project Cons: Team left to self-guidance on some things

Separate Consulting Modules: TBAI=OVM Testbench ABV = Assertion

Plan of Attack Summary

Customize your path between typical & full

Plan:
- Start with Planning & DDV up front: o An hour of up front planning will save days of rework later - Make a VAD & let it drive the building of a solid VI in logical layers and phases

Populate:
- Choose your RDV path over several projects o Choose either DRs only, VRs only or both - Choose your CDV integration

Pilot:
- Choose which piloting battles over several projects o Rules of Engagement, VRM, Triage, MDV

Verification Planning and Management Introduction

Plan of Attack
Peet James Senior Verification Consultant
academy@mentor.com www.verificationacademy.com