with the UVVM Requirements Traceability Matrix
Espen Tallaksen, CEO, EmLogic
For FPGA engineers, requirements tracking often feels like a balancing act between precision and practicality.
Specifications may be vague, inconsistent, or scattered across multiple tools or documents, making it difficult to ensure complete coverage. Verification teams struggle with inefficient one-to-one mapping between requirements and test cases. For mission-critical and safety-related projects, regulations add another layer of complexity, mandating strict traceability and verification in predefined test cases. As a result, the process quickly becomes time-consuming, error-prone, and hard to maintain across large as well as small projects.
So, what is requirements tracking and how can it be made simpler? At its most basic, it’s the process of ensuring that all requirements in a specification have been properly verified. A prerequisite is, of course, to have clearly specified requirements. For a very simple motor controller, some of the requirements could be as shown in figure 1. There would normally be a lot more but reducing it to five makes it easier to explain. This example illustrates four normal operation requirements and one for what should happen in the event of an illegal module setup:
Fig 1. Requirements
Requirement specification
Once you have the requirements, the tracking part starts to get more complicated the further you go into the process. For example, the requirements must be specified somewhere, like Excel, Word, Jira, etc, and should be properly labelled in order to easily recognize and reference the various requirements. There is no standard way of doing this labelling, and companies do it in many different ways, from long prosaic names like Req_acceleration_max, via various types of abbreviations like Req_Acc_3, to pure numeric labels like R_14. There is no right or wrong here, but some notations and approaches make the development process a bit easier, especially for the requirements definition phase.
Examples of this could be to use names like R_UART_TX_2_STOP_BITS or R_UART_TX_5, rather than just R_251, but there could be good reasons for most approaches. The important point is to have a well-defined system that suits your needs. Let us assume that the motor controller requirements are defined as in figure 2:
Fig 2. Requirements list
The Requirements Traceability Matrix
There are two ways of seeing the tracking of requirements vs the test case in which they are verified. Forward traceability shows which test cases verify each requirement. Backward traceability shows which requirements are verified by each test case. The combination of these yields the Requirements Traceability Matrix (RTM). This could be shown as a real matrix, as indicated in figure 3, but very often it is just shown as two separate tables showing forward and backward traceability respectively.
Fig 3. Requirements Traceability Matrix
A common but very inefficient approach
Quite a few projects apply a one-to-one approach to Requirement vs Test, where they run a dedicated test case per requirement. This way, they can get the total overview and generate a report from the test case label (or name or summary) from all test cases executed. So 100 test cases are needed for 100 requirements. This is often extremely inefficient. In the case of our four normal operation motor requirements, we would need to run four test cases as shown in figure 4. But this is a major waste of time when we could have run the tests in series as shown in figure 5.
Fig 4. Inefficient test case execution
Fig 5. Efficient test case execution
Predefined requirement vs Test case relation
Very often, it’s not important in which test case a requirement is verified, as long as it is verified, but for mission-critical and safety-related projects, regulations come into play. A good example is DO-254 from the Radio Technical Commission for Aeronautics (RTCA), which defines the design assurance process for airborne electronic hardware such as FPGAs and ASICs. Under such regulations, each requirement must be verified in the test case specified in the verification plan. Verification of a requirement in a non-specified test case will be ignored and not count towards compliance. Specifying up-front in which test case a requirement should be verified, prior to implementing the testbenches and test cases, is an old-fashioned and rather inefficient approach to requirements tracking. This probably made sense when requirements tracking was a manual job, but with modern automated tools for requirements tracking, this is no longer the case. With such tools, you can easily generate an RTM along with other useful reports. However, there are still many applications and projects, including all of those which are mission-critical or safety-related, where it is mandatory to specify in which test case a requirement must be executed. This must be supported by the development process and preferably also by the tools. For such an approach, the requirement list given at the start must be extended with the required test case, as shown in figure 6:
Fig 5. Requirement list with test cases
Compound requirements
Very often, the tracking requirements from sales, higher-level system requirements or other sources are vague or a combination of multiple, more or less related requirements. A requirement may sometimes just refer to a complete protocol like UART or AXI-stream, or a table of various modes. This is quite common but it can be difficult to ensure we’ve covered all relevant functionality in our test cases. So how can this be handled?
From general feedback, just remembering to check it all seems to be the most common approach, but it would be better to modify the Requirement Specification to extend or split the initial requirements. This would mean modifying the potentially already approved Requirement Specification document, however, which could be bad for both manpower usage, schedule and deliveries.
It would be far more efficient to add corresponding sub-requirements in the requirements tracking tool rather than the Requirement Specification. However, to achieve this, we need support for sub-requirement handling in the tool itself.
Other important aspects
So far, we’ve discussed standard simple scenarios, but there are other aspects that often apply:
- A requirement must be tested in one specific test case, but also tested in other test cases, with or without errors
- A requirement must be tested in multiple specified test cases
- A requirement may be tested in only one out of multiple specified test cases
- Reuse of verified modules and testbenches from a previous project may fail compliance if requirement labels or groupings differ in the new project. For example, even if the same UART design and testbench are reused, differing requirement labels or groupings mean the verification results no longer map correctly to the new specification, and compliance is lost.
All of this could be handled manually, but that would be time-consuming, error-prone and boring.
Step in UVVM
This is where the Universal VHDL Verification Methodology (UVVM) can help. The free and open‑source framework designed to make FPGA and ASIC verification more structured, efficient, and reusable is already used by 27 percent of FPGA engineers and provides a standardized architecture with ready‑made verification components, utilities, and libraries that simplify the creation of testbenches.
First launched in 2015, UVVM has evolved over time in collaboration with major partners like the European Space Agency (ESA). One of our first projects with ESA was to enhance the functionality of UVVM yet further with essential improvements that introduced specification coverage, or requirements tracking, to the project. Importantly, the functionality is fully compliant with the needs of mission-critical ESA projects. Very soon after the release, we got feedback from users that they had successfully used the UVVM Specification Coverage capability for various safety applications including DO-254. It meets all of the requirements and provides a ready-made roadmap to handle and unify every step of the requirements tracking journey in one place.
Getting started with UVVM Specification Coverage
It’s easy to introduce requirements tracking to your workflow with UVVM, and starts with four steps:
- Make a CSV file with labelled requirements – with or without test cases specified. Often this can be generated automatically from Word, Excel or Jira.
- Implement your testbench and test case(s) as before with the following commands written inside the test cases:
- initialize_req_cov() at the start of your test case
- tick_off_req_cov()for each requirement you have verified
- finalize_req_cov()at the end of your test case
- Run (simulate) your test case(s). This will generate a partial coverage file per test case
- Run a simple Python script to accumulate the coverage from all partial coverage files (Not strictly needed if there is only one test case)
This will generate all of the report files needed.
The Requirements Traceability Matrix
Following the workflow for the motor controller, let’s assume that in test case TC_B, where we’re checking for illegal module setup (Motor_R5), we also check requirement Motor_R2. This would yield the partial coverage files shown in figures 7 and 8:
Then we could run the Python script (run_spec_cov.py) to accumulate the result over all the partial coverage files (here, only two) to generate the RTM files. The following two files would normally be the most interesting and yield a complete RTM via the forward and backward traceability files, as shown in figures 9 and 10.
Taking it one step further
UVVM can also generate additional files like:
*.req_compliance_extended.csv which is similar to the minimal above, but includes all test cases where the relevant requirement has been verified.
*.req_non_compliance.csv which lists all requirements (if any) that are non-compliant or not tested, and the reasons for any NON_COMPLIANCE.
*.warnings.csv which shows issues that the user should be aware of.
The example above showed a situation where all test cases pass. If a test case does not pass, the partial coverage file will show it as FAIL rather than PASS, and the coverage summary files will show NON_COMPLIANT rather than COMPLIANT, with the second parameter showing a reference to the non-compliance file rather than the test case. The result of these reports is a very good overview of the RTM, the status and what potentially needs to be fixed.
Summary
It is unfortunately quite common during FPGA development to tick off somewhere, at some time, that a particular requirement has been tested, often just as a mental exercise. It is always better to use a written, repeatable and automated approach to guarantee requirements tracking has been conducted in a demonstrable and auditable manner. The UVVM Specification Coverage capability significantly simplifies such an approach.
