Part 3: Formal Verification Program Leader: Finishing Well and Learning from the Experience

In this 3-part blog, I’ve been examining the emerging role of the Formal Verification (FV) Program Leader, an individual who plays a critical part in adoption of FV as part of an organization’s verification strategy, and its deployment on real projects.   The FV Program Leader’s importance and value stems not from him or her being an expert in FV technology, techniques or tools, nor from being the direct manager of the FV engineers in the organization, although he or she may also be one or both of these things in addition to being the FV Program Leader.  Rather, it comes from the FV Program Leader being an advocate, evangelist, facilitator and coordinator for the organization’s efforts to understand, adopt and utilize formal verification.

In part one of this discussion, I listed the six primary aspects of the FV Program Leader’s role: Organization, Training and Upskilling, Test Planning, Progress Metrics, Sign-off Process, and Post Mortem Analysis. In parts one and two, I talked in detail about the first four of these roles.  In this, the third and final installment, I’ll discuss the last two: achieving final sign-off via formal verification and learning from the experience via post-mortem analysis.

Sign-off Process: Sign-off flows are, of course, very familiar to design and verification teams, who are accustomed to using them to sign-off various aspects of a design, such as timing via static timing analysis, functionality via simulation, netlists via RTL-to-gate equivalence checking, and final tape-out databases via LVS and DRC checking.  Sign-off typically involves a checklist of gating criteria that must be reviewed and approved by a committee of stakeholders in the process. The sign-off process helps to determine when a given stage of the design flow is complete and enforces a minimum standard of quality control.

The FV Program Leader must establish a standardized FV sign-off process for their organization. This helps to guide the FV team towards a common set of agreed upon goals that will give each project the greatest chance of success. Implementing a sign-off flow also has the side benefit that the FV process gains the appropriate level of visibility as a gating item in the schedule and takes its rightful place as a legitimate contributor to the overall success of design projects.

The first step in developing the sign-off process is to establish the checklist of exit criteria. This is a list of goals that must be achieved before declaring that formal verification is complete. Examples of checklist criteria are:

• completion of a review of checkers and constraints and gaining agreement that they are correct and complete, including that there are no unintended over-constraints
• validation of constraints through cross-checks with simulation and/or formal testing of neighboring blocks and at higher levels of the design hierarchy
• analysis of the required proof depth showing that this depth is sufficient to provide adequate verification coverage on the design achievement of goals for the metrics used to measure progress, as discussed in part two of this blog, including documenting the coverage results and any waived coverage holes
• creation of a regression environment to perform periodic formal runs to ensure no bugs in the design or holes in the formal testbenches creep in after they’ve been deemed complete

The sign-off process should also prescribe who the members of the review committee must be. The review committee is typically composed of the project team members that have a stake in the functional design and verification of the block being formally verified. This may include management such as the project manager, the design manager and the verification manager. It will also include project engineers such as the architect, the RTL designer and the simulation verifier responsible for the block.

The FV Program Leader should work with the FV team to create a clear, compelling presentation of the data which satisfies the sign-off criteria and present it to the review committee for formal sign-off.

Post-mortem Analysis: With any engineering project, there is much to be gained from a post-mortem analysis. However, this is especially true for FV projects, since many organizations are still early in their FV adoption process. At this critical stage, there is both much uncharted territory for the organization, and, often, skeptics who need to be convinced of the value and efficacy of FV. Hence, the main goals of this analysis are to identify two key things: lessons learned of what worked and what didn’t work, that will be carried forward to future projects, and the advantage gained from using FV versus other verification strategies.

In a post-mortem review, the FV Program Leader and FV team will typically work together to create a final report that summarizes each aspect of the FV effort on the project. The report documents what the original goals and expectations were, and how the project performed relative to them.  It’s Important to highlight the key benefits of FV, including its ability to exhaustively verify complex logic. To do that, it is helpful to have a list of all bugs found by FV, highlighting ones that would be very difficult to uncover via simulation. At the same time, it might be useful to study the bugs found by traditional methods on blocks that were not chosen to be formally verified, and to discuss whether such blocks could have benefitted from formal verification.

FV engineers use advanced techniques, such as abstractions, to overcome complexity barriers. Each major complexity issue that the project encountered, particularly the unexpected ones, and the method used to resolve it, should be reviewed.  This will further convey the benefits of exploiting formal methodology to achieve deeper analysis of complex designs.

Finally, as with any project post mortem, the original schedule should be compared to what was achieved, and any substantial slips should be evaluated and understood. This might also be a good time to review the sequence of the work done with formal verification – for example, it could have been better (or worse) to delay finding bugs due to CRC errors until a later stage in the verification process.

The post-mortem analysis will often result in some useful feedback to modify and improve the five areas I have previously described in this series of blog posts. As the value of formal verification is better understood, it will lead to organizational changes where more resources are shifted towards formal. Lessons learned may influence the training of new formal engineers. Results achieved may change the way that the organization performs its planning process or the metrics used for measuring progress or the checklist criteria used for sign-off.

The role of the Formal Verification Program Leader is a multi-faceted one.  In this three-part blog, I’ve discussed the six biggest and most important aspects of the role.  There are, of course, as many variations on this theme as there are companies pursuing a formal verification agenda. So, above all else, an effective FV Program Leader must be flexible.  The role of Formal Verification Program Leader, while not for everyone, is a critical one for companies making an investment in formal. It’s highly visible and ready-made for a capable individual who is a good communicator, adept at navigating organizational politics, able to quickly grasp and manage tradeoffs, and a strong process leader. Could it be a position in your future career path?

This entry was posted on Thursday, December 29th, 2016 at 4:16 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.