Industry Struggles with System Coverage

DVCon is a great place to talk to design and verification engineers. As the Accellera Portable Stimulus Standard (PSS) gets closer to reality, we were able to share with them during the conference the progress made and the ways in which it may impact their task. Most of them are as excited about PSS as we are. While we have been working in this field for more than a decade and have received a lot of feedback, there are now many more people becoming aware of it and the potential that it has. This provides us with the opportunity to learn as well.

Panels at DVCon provide a forum for discussions that relate more to tools and flows than industry trends or visionary talks. While the standard defines how something is written, it does not define the methodologies that will go along with it or how users will choose to introduce tools into their flows. Plus, there are always transition periods and this is an important consideration for new users.

This year, Breker participated in a panel titled, “Help! System Coverage is a Big Data Problem!” Yes, the title was a little provocative because it is not clear, even after the panel, if system coverage is a Big Data problem or just a big problem with lots of data! I think it tends to be more the latter, but some users may find interesting ways to apply machine learning techniques to the problems associated with coverage.

Let me back up a little and summarize Breker’s position on that panel. Verification engineers have used implementation coverage as a proxy for intent coverage for the past 20 years. Functional coverage attempts to show that important aspects of a design have been exercised but there is no direct correspondence between that and use cases that really need to be verified. We also have to rethink what closure means at the system level. At Breker, we represent systems as graphs of interconnected use cases and those graphs show all of the possible paths through the design. When you see it visually, you quickly realize that you are dealing with very big numbers.

A graph for a unit could have 10^30 paths and a graph for a system could be 10^300 paths. How many simulations can you run? Perhaps 10^5 or 10^6. How many in emulation? Perhaps 10^8. How many in silicon? Perhaps 10^12. All told, we have 10^12 samples and a space of 10^300. Can someone compute the coverage?

The task at hand is to prove that chips have no problems in hardware or software based on experiments with a small number of samples. You have to identify the important tests to run because we cannot exhaustively cover the graph. This is different from block-level verification. Closure does not equate to 100%. In fact, closure may correspond to a coverage of 0.001%. Picking the right testcases is what it is all about.

PSS contains an equivalent of functional coverage that may be important for engineers initially adding PSS into their existing SystemVerilog/UVM flow. At Breker, we do not believe that this is the best way to define system-level closure. Users often tell us how difficult it is becoming to define good coverage models using existing notions.

Verification is a risk analysis problem. We have to decide how much verification we can afford versus the risk of having something go wrong. This is different for different types of products. Acceptable risk levels for a toy are different from those necessary for safety-critical applications, such as a car.

The beauty of graph-based models is that we can view the entire space.  Instead of guessing, we can reason and decide. Do any of us know the one correct way to define system-level coverage? Breker has some ideas and, in our tools, we allow users to use the mechanisms defined in the standard that are portable. Our tools provide some additional ways to define and analyze coverage that are unique capabilities. We also allow users to access the coverage database so that they can define their own notions of coverage and potentially apply machine learning to the coverage data.

At the end of the day, system level coverage closure has to be about intelligently assessing the risk of a bug escape. Our goal is to provide tools with the maximum flexibility to enable users to get their job done efficiently and effectively. We continue to learn by having these discussions at conferences such as DVCon and, over time, tools will evolve to provide the features with the most value. Some of these may even be included in later version of the standard. Together we can do this.

Tags: functional coverage, machine learning, PSS, system coverage, SystemVerilog, uvm

Category: Knowledge Depot

This entry was posted on Thursday, March 15th, 2018 at 3:58 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.