Revisiting System / Scenario / Use-Case Coverage

It’s been more than a year since we presented the Breker view of system coverage in detail, so it’s time to revisit the topic. We first defined the notion of system coverage as measuring which realistic, system-level application scenarios have been exercised using the existing test cases. We then demonstrated how our graph-based scenario models are ideally suited to capture system coverage metrics and fine-tune them using graph constraints if needed.

More recently, we noted that the term “use cases” has become more widespread and introduced the example of a digital camera SoC to show the types of use cases that should be exercised. The measurement for this exercise is also system coverage, so the bottom line is that all these terms are really talking about the same thing. Using a regular expression, we might say:

[application|realistic] (scenario|use-case) coverage = system coverage

One of the prompts to revisit the topic of system coverage is a recent three–part roundtable moderated by Brian Bailey on the SemiconductorEngineering site. Many aspects of coverage were addressed over the three articles, but the second one included two questions specifically about system coverage. We’ve been included in several of Brian’s panels and roundtables in the past, but we didn’t happen to participate in this particular one.

So let’s have some fun. To bring out a few points not addressed in the article, we’ll pretend that we attended and provided our answers to Brian’s questions:

SE: In the past, coverage was well contained in that it talked about functionality at the block level. Today, it includes the system level, but new markets such as automotive are redefining what coverage means. How do we get our hands around the problem of defining when you can tape out a chip?

Breker: First of all, we fully endorse all traditional metrics, including code coverage, assertion coverage, and functional coverage of both the design and the testbench. Many of these are measured at the block level, while others are also applicable for the full chip. However, these metrics must be complemented by system coverage that shows whether all the realistic scenarios for the design have been verified. This means that the coverage has been exercised by self-checking test cases. Coverage without results checking is meaningless.

The best way to specify system coverage is with a graph-based scenario model. The graph shows all the possible use cases for the chip, including how multiple IP blocks in a single SoC can be strung together for application-level scenarios. These represent how the chip is actually used by the end consumer. You want to be sure, for example, that a smartphone user can make a phone call while not corrupting a Web upload or interrupting email download.

It is easy to gather node and path system coverage metrics from a scenario model, since these derive directly from the graph representation. Breker’s tools can automatically generate test cases that exercise every path in the graph and run in any target platform, from simulation to silicon. If certain paths are illegal according to the chip specification, a design or verification engineer can easily add graph constraints to prohibit them. The generated test cases will ignore these paths, and they will be excluded from the system coverage metrics.

SE: If we know at the system level that it is impossible to cover everything, then how do you prioritize and make sure you have considered the most important cases?

That is a very good point. For an SoC, the total number of legal paths in a scenario may be huge. Our tools can certainly generate all the test cases, but the time to actually run them may be prohibitive. This is another reason why graph-based scenario models work so well. A verification engineer can easily specify which particular nodes and paths must be hit by the test cases, and specify cross-coverage metrics so that all interesting combinations of branches in the graph are also exercised.

We hope that this refresher on our view of system coverage has been helpful. “Effortless system coverage reflecting end-use applications” was one of our top 5 gifts to verification engineers in 2013 and “Integration of system coverage with other coverage metrics” was on our 2014 list. Our customers have found this feature one of the key advantages of our verification approach. If you’d like to share in the benefits as well, just let us know.

Tom A.

The truth is out there … sometimes it’s in a blog.

Tags: Accellera, application, Breker, functional verification, graph, graph-based, node coverage, path coverage, portable stimulus, PSWG, realistic use case, scenario model, simulation, SoC verification, system coverage, test case generator, Universal Verification Methodology, use-case coverage, uvm

This entry was posted on Wednesday, January 6th, 2016 at 6:17 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.