Why Do We Need Code and Functional Coverage

In short: Because the tests are generated randomly.

When you have a test that can generate 1 out of 1000 different types of stimulus at any given moment, it is not safe to assume that permutation #379, which you consider to be necessary to confirm some vital functionality of the chip design, was in fact generated. Assuming all 1000 permutations of stimulus are equally likely to be generated, permutation #379 would have a 0.1% chance of ever being generated.

The diagram Random Testing Suffers From Lack of Observability depicts this circumstance: state space of a design (the big circle), random tests (the grey clouds), and particular locations in the state space of the design that are considered to be high-priority testing points (small greyed out dotted line circles).

Random Testing Suffers From Lack of Observability

These high priority test points are grey and dotted-line to indicate that the engineer does not know whether or not a given test has reached these locations in the state space. It also shows that there often is a number of these important locations in the state space that are in fact reached by tests, but it’s not known by the engineer as of yet.

In short there is a lack of observability of the test results.

Hence, if there are certain stimulus permutations that are considered to be essential to the thorough testing of the design, a mechanism must exist for the purpose of programmatically detecting when these essential stimulus permutations are created. Additionally, the engineer may want to know if the chip design responded in specific ways to this stimulus.

The key point from the diagram Random Testing Suffers From Lack of Observability is that the engineer cannot determine in fact which points in the design state space are hit and which are not hit by the tests, because the tests are random.

This issue is resolved by another key element of constrained random testing strategy: the functional coverage model. I explain what a functional coverage model is below.

But first, let’s focus first on what does tracking code coverage solve.

What does tracking code coverage solve

Code coverage is a metric that measures what parts of the source code, which represents the executable model your chip design, were executed as a consequence of simulating that model’s response to test stimulus.[¹]

The diagram What Code Coverage Does illustrates what code coverage does to help us gain test observability.

What Code Coverage Does

The figure contains the code section of a basic ALUand two tests that were run on this ALU. One test drives the ALU to perform an ADD. Another test drives the ALU to perform a SHIFTRIGHT. Code coverage indicates which lines of code were executed as a consequence of running these tests. It does so in a manner that a human can see as illustrated in the diagram above. The lines of code having been executed are in green boxes, and the lines that were not executed are in grey boxes.

But yet there remains certain questions left unanswered by code coverage:

• Did we ever run a test that commands the ALU to perform SUBTRACT where the result ended up being 0?
• Did we ever run a test that commands the ALU to perform MULTIPLY where operand2 and operand1 are odd numbers, and are not equal to each other?
• Did we ever run a test that commands the ALU to perform ADD where the result is an even number?

The functional coverage model makes determining this in the context of randomly generated tests feasible.

What does functional coverage solve that code coverage does not solve

Code coverage is meant to provide metrics on the level of execution with respect to the structure of the chip design as represented by the HDL source code. But it cannot be used to easily identify critical scenarios.

By critical scenarios I mean a series of events over time in response to certain stimulus patterns, or a combination of such events occurring in response to stimulus. The source code does not make such critical scenarios obvious; rather, these events are usually described at levels of abstraction not clearly represented in the source code. Often these are best identified in a well written and carefully reviewed specification or specifications.[²]

Covergroups are the mechanism used to detect these critical scenarios.

You can think of the covergroup as an inspector that is monitoring various parts of the chip design and that samples these criitical scenarios as they occur in the design while the design responds to stimulus. The nature of these critical scenarios is baked into how the covergroup is defined.

Taking the example of the ALU, here is how a covergroup could be related the ALU design iteself:

Relationship Between The ALU and its Covergroup

So as you can see, the covergroup is watching the input ports of the ALU (operand1, operand2, and command) and the output of the ALU (result).

Below is how you could implement the covergroup in SystemVerilog. We are taking the set of scenarios mentioned at the end of the previous section (What does tracking code coverage solve) and turn them into a SystemVerilog covergroup:

Please refer to the IEEE Standard for SystemVerilog for more details of how this works.

[¹]: Viraj Athavale, Sai Ma, Samuel Hertz, and Shobha Vasudevan. 2014. Code Coverage of Assertions Using RTL Source Code Analysis. In Proceedings of the 51st Annual Design Automation Conference (DAC ’14). Association for Computing Machinery, New York, NY, USA, 1–6. DOI:https://doi.org/10.1145/2593069.2593108

[²]: Nathan Kitchen and Andreas Kuehlmann. 2007. Stimulus generation for constrained random simulation. In Proceedings of the 2007 IEEE/ACM international conference on Computer-aided design (ICCAD ’07). IEEE Press, 258–265.