Are Verification and Validation Different? Does Anyone Care?

For the most part, the terms “verification” and “validation” are used interchangeably in the electronics industry. However, there are many who argue that these are distinct activities in the development of SoC s and systems, performed at different times in the schedule and usually by different groups of engineers. We refer to ourselves as “The SoC Verification Company” and this is a deliberate choice we made. So we thought that it would be useful to define the two terms as we see them and talk about the similarities and  differences.

This post was inspired by an article from 2010 that our CFO and co-founder Maheen Hamid discovered recently. It opens with the “usual definitions” as follows:

• “Validation: Are we building the right system?”
• “Verification: Are we building the system right?”

This seems like a good place to start the discussion.

So the basic difference is that validation determines whether the system definition (specification) meets its intended purpose and usage. Verification, in contrast, determines whether an implementation of the system satisfies the definition/specification. Consider, for example, a cell phone designed for children. The validation team might check that the specified size fits comfortably in a child’s hand and that the specified video and audio quality metrics align with the capabilities of the average child.

The verification team would check during the design phase that each component complies with the specification. Of course, there is also a third team that checks during manufacturing that each instance (or representative samples) coming off the production line meets the specification as well. The article includes a very interesting Venn Diagram showing how various tasks can be grouped into verification, validation, or a combination of the two.

So let’s bear this discussion in mind as we take a look at the world of Breker and our customers. They design some of the biggest and most complex SoCs and systems in the world. A detailed specification, usually written by a combination of product marketing and system architects, is a must. Validation that this specification meets the needs of the market is primarily performed early and late in the project. Focus groups with target customers help guide the product requirements and the specification.

An architecture team typically models the design at a high level to ascertain basic functionality, estimate power consumption, and measure expected performance. This may involve running production code, which provides true validation of the hardware-software system. If the tests are hand-written, or automatically generated with Breker’s technology, one might argue that the validation covers the hardware but not the complete system.

Once a physical implementation of the design is available, for example mapped into an in-circuit emulator (ICE) or an FPGA prototype, further validation of the SoC may be possible in the context of the target system. Again, if production software is used, one can argue that the validation is more complete than with manual or generated tests. After fabrication, early systems will undergo additional validation in the lab, during Alpha test in-house, and during Beta test at customer sites.

In between, there is a whole lotta verification goin’ on. From individual IP blocks through the full SoC, simulations are run using manual or generated tests. Formal analysis and assertions may be used to verify specific functionality. Static analysis is often used to find a wide range of problems that would cause the design to vary from the specification. Some aspects of  simulation acceleration and ICE are also more about verification than validation.

By the earlier definitions and this discussion, it is clear that Breker is mostly about SoC verification, just as our company descriptor claims. Since our whole verification approach is built on realistic use cases that stress the limits of the hardware design more than production code ever would, we are arguably validating the hardware design as well as verifying it. We tend not to say this because it may confuse people who consider only full-system hardware-software validation.

Since we generate tests on “bare metal” to have more control over the hardware, we’re not directly participating in full-system validation. However, it’s not uncommon for the scenario model graphs that our customers build to call routines and sometimes even full drivers for some of the IP blocks. In this case we are providing a level of system validation by running some portions of production code within our generated test cases on the hardware design.

The bottom line is that not too many people care which word is used when, but many people care that the full range of testing and analysis spanning both terms is performed before an SoC is fabricated and the system is shipped. Do you have thoughts on this topic? Please use the comment section. We’d love to hear from you!

The truth is out there … sometimes it’s in a blog.

Tom A.

Tags: Breker, EDA, formal analysis, functional verification, graph, graph-based, portable stimulus, scenario model, simulation, SoC verification, software-driven verification, static analysis, test generator, Universal Verification Methodology, uvm, validation, VIP

This entry was posted on Tuesday, August 25th, 2015 at 5:47 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.