What Makes a Good Code Coverage Tool for HLS?
High-Level Synthesis (HLS) is now a reality. Technology companies all around the world are successfully deploying HLS in their flows to reduce the time to market and to cut down verification costs. The HLS generated RTL needs to fit into the well-established corporate verification flows. An important aspect of the verification flow is to close code coverage on the RTL. Traditionally, RTL designers have been using a mix of statement coverage, branch coverage, expression coverage and functional coverage to guarantee the “goodness” of the RTL. To achieve all of this on machine generated RTL is indeed a tall order.
HLS users would love to have a flow where they close code coverage once on their high level source code and use the same tests on the HLS-generated RTL to get close to 100% code coverage. That way, they can leverage the advantages of gcc simulation which is not only free but about 100X faster than RTL simulation. But one cannot use conventional software coverage tools to achieve this as the requirements for closing coverage on the high level models are very different.
Let’s look at the tool requirements below.
1. Should understand the hardware intent
HLS is all about exploring various architectures. The coverage tool should understand the hardware intent of the designer and report the coverage numbers specific to the particular architecture. For example, pipelining a loop or unrolling it can lead to two very different RTLs but coverage tools like gcov will report the same code coverage for the loop in both the architectures. When a loop is unrolled, the coverage for each iteration of the loop is important, which is not the case if the loop is pipelined. Thus, gcov and other software coverage tools will lead to poor correlation between code coverage of the C++ source and the generated RTL. The coverage tool for HLS should not only understand the loop behavior but also provide a convenient way for the designer to specify the behavior.
2. Should report the coverage for each function call separately
In HLS, each call to an inlined function contributes to the logic independently of other calls. Thus, having a consolidated report for such functions across all calls is generally conservative and can result in a poorly covered RTL even for fully covered C++ code. A coverage tool for HLS should report the coverage for each function call separately unless the function is intended to be synthesized as a separate hierarchy, in which case, the consolidated report is fine. Further, for each function, it would be best to present both a call specific report as well as a consolidated report.
3. Should report coverage for not-so-obvious branches in C++
There are many constructs in C++ where the branching is not so obvious but it translates to MUXs in the RTL. Some examples of such constructs are:
a. Variable array access
b. Variable bit-select or bit-range access
c. Virtual functions
d. Function pointers
Conventional coverage tools do not treat these constructs differently. For example, a variable array access on a particular line is reported as fully covered even if all the elements of an array are not accessed. A variable array access will translate to a MUX in the RTL and the uncovered indices may show up as coverage holes on the MUX lines in the RTL leading to poor RTL coverage in spite of good C++ code coverage. An ideal coverage tool should be able to recognize the implicit branches in the high level model and report it fully covered only if all the branches have been covered.
4. Should support wide variety of coverage metrics
Most of the software coverage tools support only statement and branch expression. This is generally not enough to certify the quality of the high level models. Ideally, the coverage tool for HLS should support:
a. Functional Coverage – allows the user to track important values, set of values or boundary conditions
b. Expression Coverage – coverage for all valid combinations of expressions involving short-circuit evaluation
c. Toggle Coverage – coverage for each bit of a multi-bit variable
These metrics are already familiar to RTL designers and need to be available in software coverage tools so that the designers can sign-off the high level models with high confidence.
5. Differentiate between dead code and uncovered code
Coverage tools report uncovered parts of code as a coverage hole. Since these tools are simulation based, they cannot differentiate between “true” uncovered code and dead code. The designer may waste a lot of his time trying to cover some code which is really dead. This is where the ability to differentiate between dead code and uncovered code is really useful for the designer and typically requires some kind of integration of the coverage tool with a formal verification based tool. The formal tool should be able to report dead code as well as be able to generate test vectors for portions of code which were uncovered by simulation.
6. Should be language neutral
The coverage tool should support both C++ and SystemC thereby giving the designers the flexibility to design in the language of their choice.
These requirements may appear to be daunting but are extremely critical for signing-off the HLS model and to achieve a good correlation between the HLS model and the generated RTL. Catapult Coverage, a unique offering from Mentor, is a significant step in the direction of meeting this goal. It understands the hardware intent of the designer, supports a wide variety of coverage metrics, is language neutral and therefore an ideal choice for closing coverage on HLS models.