LIVE WEBINAR: FPGA Verification with VHDL (Four Part Webinar Series)
Part 3: OSVVM's Test Reports and Simulator Independent Scripting (EU)
Jim Lewis, VHDL User, Designer, Verification Engineer, Trainer, OSVVM developer, and IEEE VHDL Chair
Thursday, June 16, 2022
3:00 PM - 4:00 PM (CEST)
|
Abstract:
According to the 2020 Wilson Verification Survey FPGA verification engineers spend 46% of their time debugging. As a result, we need good scripting to simplify running tests and good reports to simplify debug and help find problems quickly.
Scripting can be complicated no matter what language – particularly with EDA tools that need to stay rooted in one directory while it is advantageous to co-locate the user scripts with the verification IP they support. As a result, the scripts must manage the file source locations relative to the simulator directory. Further complicating the scripts is that each simulator API has a different way of specifying commands and command options. No wonder it is frustrating and messy.
With OSVVM scripting, it is ok to hate TCL as it is unlikely you will use it directly. OSVVM creates a procedure-based API on top of TCL. User scripts are based on the OSVVM simple simulator command API that uses paths that are relative to the script's location. The messy TCL stuff is handled internally by the OSVVM command API. The result is that scripts include just a little more than a source file list. Generally, the most TCL that user scripts need is a simple if statement – but even this is rare (and there are examples in the OSVVM library).
Not meaning to name drop, but OSVVM scripting supports Aldec's Active-HDL and Riviera-PRO, Siemens' ModelSim and QuestaSim, GHDL, Synopsys' VCS, and Cadence's Xcelium.
With respect to reports, when we run a set of tests, we need to be able to assess whether all test cases passed or quickly identify which test cases failed. Once we have determined which test case failed, we need to have detailed information for each test case in a separate report that helps reveal the source of the issue.
OSVVM's test reporting capability adds another reason as to why OSVVM should be your VHDL Verification Methodology. Our test reporting includes:
- An HTML Build Summary Report for human inspection that summarizes the completion status of each test case in a test suite
- A JUnit XML Build Summary Report for use with continuous integration (CI/CD) tools.
- A separate HTML Test Case Detailed report for each test case with Alert, Functional Coverage, and Scoreboard reports.
- An HTML based simulator transcript/log files (simulator output)
- A text-based test case transcript file (from OSVVM's TranscriptOpen)
- Links to tool generated code coverage reports
Why do we go to all this trouble? When OSVVM runs its full regression suite, our build includes 22 test suites with a total of 524 test cases. The log file is 170K lines long. Without good tools we could not easily run our regressions and quickly assess whether it passed or failed.
How well does OSVVM work with continuous integration tools? OSVVM uses our scripts and JUnit XML output when running our verification component regression suite on GitHub using GHDL. See https://github.com/OSVVM/OsvvmLibraries/actions.
About OSVVM
OSVVM is an advanced verification methodology that defines a VHDL verification framework, verification utility library, verification component library, and a scripting flow that simplifies your FPGA or ASIC verification project from start to finish. Using these libraries, you can create a simple, readable, and powerful testbench that is suitable for either a simple FPGA block or a complex ASIC.
OSVVM is developed by the same VHDL experts who have helped develop VHDL standards. We have used our expert VHDL skills to create advanced verification capabilities that:
- Are simple to use and work like built-in language features.
- Maximize reuse and reduce project schedule.
- Improve readability and reviewability by the whole team including software and system engineers.
- Facilitate debug with HTML based test suite and test case reporting.
- Support continuous integration (CI/CD) with JUnit XML test suite reporting.
- Provide buzz word features including Constrained Random, Functional Coverage, Scoreboards, FIFOs, Memory Models, error logging and reporting, and message filtering.
- Rival the verification capabilities of SystemVerilog + UVM.
OSVVM is a competitive solution with SystemVerilog + UVM for FPGA Verification. World-wide, 18% of the FPGA market uses OSVVM [1]. In Europe, OSVVM (with 34%) leads SystemVerilog+UVM (with 26%). Based on the growth in our training, we expect to see improved numbers in the next survey.
Pre-Webinar Homework
Want to try out OSVVM before the webinar? See our OSVVM Script User Guide to run our demos. See: https://github.com/osvvm/OSVVM-Scripts#readme. If you prefer pdf (it has section numbers) see: https://github.com/OSVVM/Documentation/blob/main/Script_user_guide.pdf.
Webinar Duration:
- 50 min presentation/live demo
- 10 min Q&A
Presenter Bio:
Jim Lewis is an innovator and leader in the VHDL community. He has 30 plus years of design and teaching experience. He is the Chair of the IEEE 1076 VHDL Standards Working Group. He is a co-founder of the Open Source VHDL Verification Methodology (OSVVM) and the chief architect of the packages and methodology. He is an expert VHDL trainer for SynthWorks Design Inc. In his design practice, he has created designs for print servers, IMA E1/T1 networking, fighter jets, video phones, and space craft.
