9-2

UT Austin, EE 382M-11

2/15/2017

Assertion-Based Verification Harry Foster Chief Scientist Verification

[email protected] | www.verificationacademy.com

Outline • How Verification is Done Today • What Makes Verification Difficult • Observability and Controllability Challenge • Assertion-Based Verification • Industry Case Studies • Conclusions

2

H Foster, EE 382M-11, Verification of Digital Systems, Spring 2017

© Mentor Graphics Corporation

© Mentor Graphics Corporation, all rights reserved.

1


2/15/2017

HOW VERIFICATION IS DONE TODAY

What is Verification?

Verification is a process of ensuring that a design implementation meets its specification.

4

H Foster, EE 382M, Verification of Digital Systems, Spring 2017



2


2/15/2017

Simulation-Based Techniques • Fundamental verification technique in use today • Generally scales well • Testing all possible states is generally incomplete Simulation Testbench Measure Coverage

Generate Stimulus

Design Model

Check Results

Assertions can be used to check results and measure coverage 5



Simulation Traversal Through the State Space

// SystemVerilog Assertion

initial state

6



property p_comp; @(posedge clk) E |-> (A==B); endproperty assert property (p_comp);


3


2/15/2017

Time Explosion Problem • How long would it take to exhaustively simulate this example?


A [31:0] 1000000011101011011011110111

E B [31:0] 101010001000110101110100101


264 vectors X 1 vector every micro-second = 584,941 years An extremely fast simulator by today’s standards! 7



Simulation and the Time Explosion Problem

264 vectors X 1 vector every micro-second = 584,941 years

8




4


2/15/2017

Formal-Based Techniques • Does not require a testbench or input stimulus! • Automatically uses algorithms to verify the functionality • Verification can be complete • Complements simulation-based techniques Formal Tool

Yes

Pass ?

Done

No

Design Model 9

Assertions



Conceptual Formal Tool

a

Tx

x y

Tx(a,x,y) // next state

z

10




5


2/15/2017

How is formal different than simulation? initial states



Very fast!

11



State Space Explosion • There are more states in today’s design than there are atoms in the universe!

How many states exist in a typical design today?

12




6


2/15/2017

WHAT MAKES VERIFICATION DIFFICULT

INDUSTRY DRIVERS Rising Design Complexity


7


2/15/2017

Avergage Number of IP Blocks

Rise in the Average Number of IP Blocks 90

Closing the Design Productivity Gap!

80 70 60 50 40 30 20 10 0 2007

2008

2009

2010

2011

2012

Avg. Number of 'Other' SIP Blocks

2013

2014

2015*

2016*

2017*

2018*

Avg. Number of CPU / DSP / Controllers

Avg. Number of Embedded Memory Blocks Source: Semico Research Corp. 15



Design Engineers are Being Productive 1,000,000,000,000,000

Transistors produced per electronic engineer more than 5-orders of magnitude since 1985

100,000,000,000,000 10,000,000,000,000 1,000,000,000,000

Quantity

100,000,000,000 10,000,000,000 1,000,000,000 100,000,000 10,000,000 1,000,000 100,000 10,000

Transistors Produced

Total Electronic Engineers

Source: Technology Research Group – EDA Database, 1986, EDA TAM, 1989 & Gartner/Dataquest 2005 Seat Count Report, Gary Smith EDA, 2013 Seat Count Analysis , VLSI Research, 2013 - Transistors Produced Analysis 16




8


2/15/2017

Design Engineers are Being Productive! Growth of Transistor Volume Leads to Sustained ~ 30% per Year Cost Reduction

1.E+03

Revenue/Transistor ($)

1.E+02 1.E+01 1.E+00 1.E-01 1.E-02 1.E-03 1.E-04 1.E-05 1.E-06 1.E-07 1.E-08

Semiconductor Learning Curve 1954 – 2012 … Adjusted for Inflation

1.E-09 1.E-10 1.0E+04

1.0E+06

1.0E+08

Source: VLSI Research, SIA, Federal Reserve Note: Revenue adjusted for Inflation… 1954-2012 17

1.0E+10

1.0E+12

1.0E+14

1.0E+16

1.0E+18

1.0E+20

Cumulative Transistors Shipped



We are Keeping Up with Design Complexity! Thanks to Automation and Reuse!

18




9


2/15/2017

Another View of Moore’s Law

1.00E-04

1.00E-04

1.00E-05

1.00E-05

1.00E-06

1.00E-06

1.00E-07

1.00E-07

1.00E-08

1.00E-08

1.00E-09 1.00E+13

1.00E+14

1.00E+15

1.00E+16

1.00E+17

1.00E+18

Cumulative Transistors Shipped

1.00E+19

IC Revenue/Transistor ($)

EDA Cost/Transistor ($)

EDA Cost per Transistor and Total IC Revenue per Transistor Both Decrease About 30% per Year

1.00E-09 1.00E+20

EDA Cost/transistor IC Revenue/transistor

Note: EDA Cost Consists of EDA License and Maintenance revenue adjusted for Inflation…

Source: SIA, VLSI Research, Federal Reserve 19



Demand for Design Engineers Grows Slowly 12

ASIC/IC Mean Peak Number of Engineers

CAGR Designers 3.6% 10

10.48 10.05 8

7.80

8.10

8.53

Design Engineers 6

4

2

0 2007

20

2010



2012

2014

2016


10


2/15/2017

But what about Verification Productivity? 12

ASIC/IC Mean Peak Number of Engineers

CAGR Designers 3.6% CAGR Verifiers 10.4%

11.6 11.0

10

10.48 10.05 8

8.53

8.10

7.80

8.4

7.6

Design Engineers 6

Verification Engineers 4

4.8

2

0 2007

21

2010

2012

2014

2016



INDUSTRY DRIVERS Rising Verification Complexity


11


2/15/2017

The Emergence of New Layers of Verification

Software Verification Layers

Security Domains Power Domains Clock Domains Functional

23



What Makes Verification Difficult? Channel Encoder

TX

Decoder

Data Link Layer

Compressed Audio

PHY

RX

Single, sequential data streams

Multiple, concurrent data streams

— Floating point unit

— Cross bar

— Graphics shading unit

— Bus traffic controller

— DSP convolution unit

— DMA controller

— MPEG decode

— Standard I/F (e.g., PCIe)

— ...

— ...

Sequential data streams 1x number of bugs

Concurrent data streams 5x number of bugs

-Ted Scardamalia, internal IBM study 24




12


2/15/2017

Directed-Test Approach

Imagine

verifying a car using a directed-test approach

—Requirement: Fuse will not blow under any normal operation —Scenario 1: accelerate to 37 mph, pop in the new Lady GaGa CD, and turn on the windshield wipers

25



A FEW WEEKS LATER…


13


2/15/2017

Directed-Test Approach

Imagine

verifying a car using a directed-test approach

—Requirement: Fuse will not blow under any normal operation —Scenario 714: accelerate to 48 mph, roll down the window, and turn on the left-turn signal

27



The Concurrency Challenge

A purely directed-test methodology does not scale — Imagine writing a directed test for this scenario! — Truly heroic effort—but not practical

28




14


2/15/2017

Finding Corner Case Bugs Due to Concurrency

Directed-test-based simulation finds the bugs you can think of…

Constrained-random simulation finds the bugs you never anticipated!

29



Concurrency is Complicated to Verify

Packet-Based Design

From Fabric

Tx

Transaction Layer Packet Reformater Retry Buffer

Arbiter

To PHY

Data Link Layer Packet Reformater

Rx From Rx Channel 30




15


2/15/2017

Adoption Trends in Verification Techniques Code coverage

Assertions

2007 2012 2014

Functional coverage

ConstrainedRandom Simulation 0%

10%

20%

30%

40%

50%

60%

70%

80%

Design Projects Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study 31



OBSERVABILITY & CONTROLLABILITY


16


2/15/2017

Fundamental Challenge of Verification

DUT

A

A

1. Activate 0010100101010001110101001110101010100000000011101011011011110111 A

Stimulus

A

3. Detect

2. Propagate

Checkers A

33

= Assertions



Observability vs. Controllability Test didn’t set up the condition to propagate the bug bug

A 0 1

0

0 1

1

Assertions improve observability and reduce the need to propagate bugs 34




17


2/15/2017

Poor Observability Misses Bugs • Code coverage measures controllability • 100% code coverage does not mean all bugs are detected [S. Devadas, A. Ghosh, and K. Keutzer. DAC 1996] • DAC paper study found cases where:

35

Code Coverage Achieved

% of covered lines observable

90% Covered

Only 54% Observable

100% Covered

Only 70% Observable



Assertions Improve Observability Testbench

= 36

Bugs missed due to poor observability

=

Reduce debugging up to 50% [CAV 2000, IBM FoCs paper] Bugs detected closer to their source due to improved observability




18


2/15/2017

2014 Where Verification Engineers Spend Their Time

Test Planning

37%

Testbench Development 3%

Creating Test and Running Simulation 24%

14%

Debug Other

22%

Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study 37



Designers Spend a Lot of Time in Verification & Debug

Mean time design engineer spends in design vs. verification

60%

Doing Design Doing Verification

53%

55%

54%

51%

53%

50%

46%

47%

49%

45%

47%

40%

2007

2010

2012

2014





19


2/15/2017

ASSERTION-BASED VERIFICATION

Assertion-Based Verification

“How can one check a large routine in the sense of making sure that it’s right? In order that the man who checks may not have too difficult a task, the programmer should make a number of definite assertions which can be checked individually, and from which the correctness of the whole program easily flows.” Alan Turing, 1949

40




20


2/15/2017

Property •

Property • a statement of design intent • used to specify behavior

Testbench test env

DUT

41



Assertion •


•

Assertion

Testbench test env

• A verification directive

A Trace from simulation

42



DUT


21


2/15/2017

High-Level Assertion •


•

Assertion

Testbench test env

• A verification directive

•

High-level • Architectural focused • Can be part of testbench

A Trace from simulation

43


DUT


Low-Level Assertion •


•

Assertion • A verification directive

•

High-level

RTL A

• Architectural focused • Can be part of testbench

•

Low-level • Implementation focused • Embedded in or bind to the RTL

44

A



// Assert that the FIFO controller // cannot overflow nor underflow


22


2/15/2017

How Assertions Are Used Today Testbench

State Search

RTL

improved bug rate

FPGA or Emulation

Formal Prop’s

Assertions passing tests

Formal Verification

Simulation

O/S Trials

[Foster, Larsen, Turpin - DVCon 2006]

45



Who should create the assertions? Verification Engineer

High-Level Assertions

46

Design Engineer

Low-Level Assertions

Requirement focused

Implementation focused

Black-box assertions

White-box assertions

Accounted for in testplan

Not accounted for in testplan

Compliance traceability

Improve observability

Create reusable ABV IP

Reduce debugging time




23


2/15/2017

Who should create high-level assertions? Verification Engineer


47

Design Engineer


Requirement focused












Who should create low-level assertions? Verification Engineer


48

Design Engineer


Requirement focused













24


2/15/2017

Specifying Design Intent Assertions allow us to specify design intent in a way that lends itself to automation

clk grant0

reset_n

Arbiter

grant1

req0 req1

// Assert that the grants for our simple arbiter are mutually exclusive

49



Identifying the Error Condition For our arbiter example, we can write a Boolean expression for the error condition, as follows:

clk grant0

reset_n

Arbiter

grant1

req0 req1

(grant0 & grant1) // error condition 50




25


2/15/2017

Checking the Error Condition before Assertions • Doesn’t lend itself to automation. module arbiter (clk, rst_n, req0, req1, grant0, grant1); ... always @(posedge clk or negedge rst_n) begin if (rst_n != 1’b0) if (grant0 & grant1)

Error Condition Boolean Expression

$display (“ERROR: Grants not mutex”); ...

endmodule

51



Assertion Language Adoption 80%

Design Projects

2007 World 70%

2012 World

60%

2014 World

50% 40% 30% 20% 10% 0% Accellera Open Verification Library (OVL)

SystemVerilog Assertions (SVA)

PSL

Assertion Languages and Libraries

Other

* Multiple answers possible





26


2/15/2017

IEEE 1800 SystemVerilog Mutex Example grant0 and grant1 must be mutually exclusive

clk

grant0

grant1 error

assert property ( @(posedge clk) disable iff (~rst_n) !(grant0 & grant1));

53



IEEE 1850 PSL Fair Arbiter Example grant0 and grant1 must be mutually exclusive

clk

grant0

grant1 error

assert always (!(grant1 & grant2) abort ~rst_n) @(posedge clk);

54




27


2/15/2017

Accellera OVL Memory Address Example grant0 and grant1 must be mutually exclusive

clk

grant0

grant1 error

ovl_never a_mutex (clk, rst_n, (grant1 & grant2));

55



INDUSTRY CASE STUDIES


28


2/15/2017

Published Data on Assertions Use Percentage bugs found by various techniques

Assertion Monitors Cache Coherency Checkers Register File Trace Compare Memory State Compare End-of-Run State Compare PC Trace Compare Self-Checking Test Simulation Output Inspection Simulation Hang Other

•

34% 9% 8% 7% 6% 4% 11% 7% 6% 8%

17% of bugs found by assertions on Cyrix M3(p1) project [Krolnik '98]

•

50% of bugs found by assertions on Cyrix M3(p2) project [Krolnik ‘98]

• Kantrowitz and Noack [DAC 1996]

85% of bugs found using over 4000 assertions on an HP server chipset project [Foster and Coelho HDLCon 2001]

Assertion Monitors

25%

Register Miscompare Simulation "No Progress” PC Miscompare Memory State Miscompare Manual Inspection Self-Checking Test Cache Coherency Check SAVES Check

22% 15% 14% 8% 6% 5% 3% 2%

•

Thousands of assertions in Intel Pentium project [Bentley 2001]

•

10,000 OVL assertion in Cisco project [Sean Smith 2002]

Taylor et al. [DAC 1998]

57



DAC 2008 Sun paper with lots of metrics Assertion-Based Verification of a 32 thread SPARC™ CMT Processor [Turumella, Sharma, DAC 2008]

Category

Unique

Instantiated

Low-Level

3912

132773

Interface

5004

44756

High-Level

1930

18618 Bugs Found by Type of Assertion

Low-level Interface High-level

58




29


2/15/2017

Significant reduction in debugging time Assertion-Based Verification of a 32 thread SPARC™ CMT Processor [Turumella, Sharma, DAC 2008]

Category

Unique

Instantiated

Low-Level

3912

132773

Interface

5004

44756

High-Level

1930

18618

Hours

Average Debug Time 16 14 12 10 8 6 4 2 0

>50% Sim + Assert Sim + None

Formal 59

Formal

85%

Sim + Assert

Sim + None



SUMMARY


30


2/15/2017

Assertion-Based Verification • The process of creating assertions forces the engineer to think. . . and in this incredible world of automation, there is no substitute for thinking.

61



Assertion-Based Verification Harry Foster Chief Scientist Verification

[email protected] | www.verificationacademy.com


31

9-2

Recommend Documents