DVCon 2013: Best Practices in Verification Planning

Sometimes magic happens at panel discussions at technical conferences, and that was the case mid-day on Wednesday at DVCon in San Jose this week, where the conversation was lively, entertaining and informative on the pedestrian, albeit foundational, topic of “Best Practices in Verification Planning.”

Ironically, the hour-long conversation did not appear to be planned at all, but to be organic and spontaneous. The Cadence-sponsored lunch and panel discussion, moderated by Cadence’s John Brennan, included Verilab’s Jason Sprott, Cadence’s Mike Stellfox, ParadigmWorks’ Ambar Sarkar, Maxim’s Neyaz Khan, Oski Technology’s Vigyan Singhal, and Xilinx’ Meirav Nitzan. The panelists began with an overview of their experiences.

ParadigmWork’s Ambar Sarkar recalled: I’ve seen hundreds of projects, both good and bad, and what works is having a plan – a live document that really works. What also works is to bring the architects into that plan. Finally, there’s got to be a way to close the loop to be sure that the plan is maintained. At that point, we can be pretty sure that things will go well.

Verilab’s Jason Sprott was next: Our clients often look for tools solutions, but often it’s really a people issue. The things that make [projects] difficult are not things that can be structurally analyzed. It’s ruthless prioritization that is the key. That’s the most important aspect to planning, to prioritize!

When we make a verification plan, we try to concentrate on the business goals, to focus on those things first and foremost. Then the other thing is the statistics, the real ‘gold dust’ of planning. If you have experience from previous projects, these stats can make your present plan be more accurate and allow you to schedule time for the problems that may crop up. These are not things don’t influence the product, but the plan [is crucial nonetheless].

Oski Technology’s Vigyan Singhal continued: We do a fair bit of planning [and know] what’s really important is unifying a formal verification plan with your simulation plan. Any plan we create has to be [go across] all the various methodologies, including experience from previous projects. And to the extent you can measure coverage metrics from formal verification and simulation, they [need to be] in the same cockpit to decide.

Cadence’s Mike Stellfox offered: From an EDA point of view and my experience over the last 10 years, we’ve been helping clients go from directed test generation to [more structured verification]. UVM has codified that, but equally if not more important is, how do you move from verification planning and traditional test planning to a planning approach that lends itself to coverage-driven verification.

We work with the human factor, moving from traditional metrics to the newer metrics in a way that fits management’s reporting requirements. If the metrics change, the way you plan also has to change. We’re still seeing customers move forward here, plus Excel and Word are still being used a lot. We’ve also worked a lot with our enterprise management tools, a kind of VIP that should be part of the whole process.

Maxim’s Neyaz Khan added: In my 20-plus years, I’ve seen places where [people have] put together plans that are so over-engineered they’re not usable at all. [What we want is a] plan where everything comes together. It’s the verification guy’s job to [make that happen], to bring everybody else’s plan together. Traditionally, for instance, RTL verification was good enough, but now there’s low-power that people are worried about. How does that effect everything else that’s been verified?

Maxim is a mixed-signal company, so a plan is what brings everything together for us. Analog design is very different from digital. The analog guy is worried about parametrics, but the digital guy is worried about coverage. So we need [to consider] how do you bring coverage in analog and digital? How are you going to build your advanced verification environment?

We also believe a plan needs to be executable. It’s a living document, where if anything changes everybody gets affected.

Xilinx’ Meirav Nitzan concluded the first round: Xilinx always thought the direct test approach was best, but my team has tried to move from that to verification planning.

First you have to get management to understand the value of an executable plan that’s feature-driven, and you need to make people understand that a check mark in the verification plan means it’s actually been done. Once you understand that, it’s easier to correct. It’s also important to get the designers and Marketing to sign off on the plan, because that’s when questions come up that we previously hadn’t thought of.

John Brennan asked the group to lay out their biggest verification planning challenges.

Meirav Nitzan responded: For us, it’s how far should we go in the planning? The tendency is to think in terms of test cases, but the challenge is in having management understand the need for planning, and to allocate the necessary time to build a plan.

Ambar Sarkar agreed: Yes, and it’s important to get Marketing involved. You want a plan to nail down a specification from the Marketing team. I have faced that problem from my clients. One of problems with plans made today, they say here’s the feature and here’s what we want, but you need to not write a really long test, or detailed test sequences, etc. You need to leave those details out until the specifications have been worked out for you.

Also, if you have ever built a house or supervised a renovation, this [conversation about planning] is déjà vu all over again. ECOs, architects, it’s the same with a house remodel.

Jason Sprott added: You need not to get down into the details, but instead decide what, not how, it is going to be verified – not the details of each individual feature.

Ambar Sarkar reiterated: If you don’t do that, some junior guy will see a feature and want to check it across everything. Again, [it’s like the remodel]. You need to gather the ideas, develop the initial specification, and then work with the general construction guy to actually implement the design.

Vigyan Singhal added: You can spend a lot of time for known knowns versus known unknowns – what are your plans for that, and for the unknown unknowns? Things you might be able to predict from previous projects: You have to give planning weight to those things.

Neyaz Khan warned: There is always a tendency to over-engineer a plan and create something that’s unachievable. For example, cross coverage – everybody wants to cross all of this and all of that.

The other major problem, your coverage is always expanding. Suddenly, you come in and see that the coverage is down from your estimate. Somebody may have forgotten to add the assertions to the plan, and now that coverage has reduced. We’re working in a mixed-signal environment, but the stakeholders differ [and use different techniques]. I’ve even seen where people take a picture of the white board and try to use that in the plan.

The question is always, How do you extract what you need? Remember that specs change all the time. Last-minute customer changes mean that suddenly the plan I put together must be revised, or even thrown out, so I have to start all over again.

Mike Stellfox enumerated: These are the three biggest challenges.

One – How do you capture the plan, especially avoiding too much detail too early on? We spend a lot of time and have developed methodology around the project. There’s a process by which you can capture that information.

Two – Engineers don’t like documentation, paper plans especially. You capture a plan, but then it gets stale. Or you write a plan, but now you are downstream [on the project] and it’s no longer being verified against the original specs.

Three – How do you track your progress? How do you track that you’re meeting your milestones?

You’ll hear a lot of people say, we don’t have time to capture the plan. We don’t have the time, even though the planning tools looks great. But we know that you’ll waste a lot more time if you don’t set up a plan with milestones which you can track. We set a lot of time aside at outset to establish the plan, but we have overall savings [as a result]. Looking at their coverage and analyzing it in a specific way [based on the plan], that’s the experience we’ve seen with any customer who’s adopted a good coverage plan.

Jason Sprott noted: Sometimes it’s hard to answer a customer’s question, ‘How do I benefit from the planning.?’ We say it’s hard to say how many weeks, months or bugs it saves, but if you don’t plan, you’ll never be able to capture the spec or look at things to say: If I had left out this feature, what would I do now?

Ambar Sarkar recounted: Usually when people hire us, they put us into the fire. I ask where is the plan, and find sometimes that it’s good and sometimes not. I don’t want to say what the customer’s going to save in time by [establishing a plan], but it helps the customer narrow and specify the scope of what’s being done. I never think it’s about saving time. What we want is [for the outcome] to be more predictable.

Jason Sprott advised: You should never underestimate the value of planning. The specifications we receive might be woeful, but they can be salvaged by planning!

Vigyan Singhal agreed: The planning will save time in the long run. Look at our verification challenge that we ran at DAC last year. We spent an enormous amount of time at the outset to do the planning, [and it paid off].

Neyaz Khan added: Planning also let’s you know what your coverage should, or will be.

Meirav Nitzan continued: If we were just working on certain features, the tendency was to implement it all. Now with a plan, you see that you only have 5 days, for instance, until the deadline and there are features you absolutely have to do [to get there].

Jason Sprott repeated: It’s about ruthless prioritization. What you don’t need, put it on the back burner!

Neyaz Khan added: And you need to know what stimulus is optimal to get you from where you’re coming from to get where you want to go. You have to have a plan!

John Brennan moved on: Switching to the human factor in all of this, how do you effectively solicit input from people who don’t want to be part of planning process?

Neyaz Khan answered: We have a mandatory deep dive. All designers, verification, software guys, everybody who has a stake in the chip. We also invite the product definer, and we invite the test engineer, particularly in the case of analog.

At the end of the day, I’m not interested in the micro-architecture, or how you implemented the analog. I’m only interested in how the chip works. Every time we walk away from one of these deep dive sessions, some things have been identified that need to be changed or eliminated. We save a lot of time in the end by doing this.

John Brennan asked: Is there a methodology for good planning?

Mike Stellfox said: Yes, there’s a methodology when we work with customers, when we help to capture a feature-based plan. The approach we always start with is a peer-review process. Verification is also part of the planning process. The specifications can be very misinterpreted, or the intent of the architecture. Even before you start any executable work, this is crucial.

You may not hear from absolutely everybody, but just capturing the features and how to do the verification – this is really key. The what needs to be verified, versus the how.

Jason Sprott added: These are the people who care whether a feature is important or not. If the guys in the room want to decide if a feature is really important, it can help reduce the scope. Deep dive is an excellent way to start, but you have to have subsequent smaller meeting. And you’ve got to give everybody involved time to know it’s coming. [You can’t call the meeting on a moment’s notice.] You have to give them time to prepare.

Ambar Sarkar laughed: I worry the most when someone [in the meeting says], ‘Oh, don’t worry about it.’ Those are always the things that will get you later on in the project!

Vigyan Singhal advised: Things change for reasons that are hard to predict. Maybe you change the plan, and maybe you don’t. But when you’re doing the post mortem, where a project ended up is part of the historical data that will help you plan for the next time. Recording and validating what happened [is critical].

Jason Sprott lamented: It’s a shame when the knowledge is lost at the end of the day.

A question from the floor: It’s hard to see what’s missing in the plan if there’s already a lot loaded into it.

Ambar Sarkar responded: A plan is never complete. Even at the last moment, you’ll see that something’s missing. So your good-enough criteria has to be rigorous!

Vigyan Singhal emphasized: The plan is actually never good enough!

Another question from the floor: It’s important to see what’s missing from the plan, otherwise somebody will say everything in the plan in verified.

Ambar Sarkar agreed: Plan completion is important, and input from the other teams involved.

Neyaz Khan suggested: Look at your plan. Does Feature X have coverage? Does Feature Y have assertions? Once you think it’s all done and you run regressions, you’ll still find holes but you’ll establish a priority in the process.

Mike Stellfox added: The plan needs to be hierarchical to see if you have everything. In terms of methodology of planning, there’s a peer review process. But in terms of tooling, there have to be ways to review details of the specifications. You should have a plan that’s a living thing.

A third question from the floor: Between project planning and Microsoft Project, it’s often the case that the manager can only read the Microsoft plan, which is always a problem.

Jason Sprott advised: Perhaps you should try Agile-based planning. I’ve found Agile to be very useful, in the right hands. It has has number of advantages from a project planning point of view. It’s much more up-to-date and it’s all web based. It can be a good thing, but you have to train the managers to understated the data.

A fourth question from the floor prompted laughter: I’m with Microsoft and I want to say that Microsoft supports Agile.

Ambar Sarkar warned: And it doesn’t do for people to be reviewing Agile results in the morning, and again at night, spending hours on the process. The team has a different attitude about what’s useful, [compared to Management], so it’s important to customize the reporting to understand what is useful for you.

Jason Sprott wrapped up with a clarification: One big thing that never works is when management doesn’t like the plan. Another thing that doesn’t work is just relying on the tools. But the Agile mindset is not just about the tools – it’s about know what communication is truly useful!

*******************

Tags: Ambar Sarkar, Cadence, DVCon 2013, Jason Sprott, John Brennan, Maxim, Meirav Nitzan, Mike Stellfox, Neyaz Khan, Oski Technology, ParadigmWorks, Verification Planning, Verilab, Vigyan Singhal, Xilinx

This entry was posted on Thursday, February 28th, 2013 at 6:34 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.