Skip to content

Code and functional coverage with VCS - #1918

Draft
davidharrishmc wants to merge 7 commits into
openhwfoundation:mainfrom
davidharrishmc:dh/vcs-coverage
Draft

davidharrishmc wants to merge 7 commits into
openhwfoundation:mainfrom
davidharrishmc:dh/vcs-coverage

Conversation

@davidharrishmc

@davidharrishmc davidharrishmc commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Adds code and functional coverage on Synopsys VCS. wsim --sim vcs --ccov/--fcov (with lockstep for --ccov) writes per-run .vdb databases under sim/vcs, make -C sim VcsCodeCoverage/VcsFunctCoverage merge them with urg, and regression-wally --coveragesim vcs runs the same coverage regressions as Questa. Code coverage uses line, cond (observability), branch and fsm on the core; functional coverage compiles the same riscv-arch-test covergroups. Rather than keep a second exclusion list, sim/vcs/exclusions2el translates coverage-exclusions-rv64gc.do and the RTL // coverage off pragmas into a URG exclusion file. It logs the exclusions VCS has no exact counterpart for (43 of 299, mostly where Questa excludes one row of a vector VCS scores as a whole), and, like Questa, fails on an exclusion that names a line with nothing to exclude.

Closes #1179.

🤖 Generated with Claude Code

davidharrishmc and others added 4 commits October 2, 2026 11:18
wsim accepted --ccov and --fcov only with Questa.  run_vcs now compiles for code coverage with line, cond,
branch and fsm metrics on testbench.dut.core (coverage.hier), the counterparts of Questa's +cover=sbecf:
condition coverage in its default mode counts the 1-bit bitwise operators Wally's logic is written with
(allops would add a condition per bit of every vector operation), and -cm_cond obs counts a subexpression
only where it reaches the output, as Questa's focused expression coverage does.  --fcov compiles in the
riscv-arch-test covergroups the same way wally-compile.do does, with riscv-arch-test's RVVI trace in
place of ImperasDV's.

Each simulation writes its own database, sim/vcs/ucdb/<cfg>_<name>.vdb or fcov_ucdb/<cfg>_<name>.vdb,
named like Questa's .ucdb, so that runs can go in parallel and be merged or discarded one at a time.  A
code coverage database starts as a copy of the coverage model the compile wrote, so it can be merged on
its own.  Functional coverage runs with -vcs_assert off, as riscv-arch-test's own VCS flow does, because
VCS W-2024.09 otherwise stops with an internal assertion while sampling some covergroups (I_beq_cg).
wsim keys the VCS compile on the covergroups, as it does for Questa, and recompiles when run_vcs or
coverage.hier changes.  Lockstep with ImperasDV works with --ccov.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Harris <David_Harris@hmc.edu>
…ions

VcsCodeCoverage and VcsFunctCoverage are the urg counterparts of QuestaCodeCoverage and
QuestaFunctCoverage: they merge the per-run .vdb databases and write text reports, a per-instance
summary for code coverage (rv64gc_coverage_hierarchical.rpt, with covered/total counts) and the total
covergroup score for functional coverage.

URG takes exclusions only as an exclusion file whose entries name each object by its id and checksum,
which only urg itself can produce.  Rather than keep a second list of exclusions, vcs/exclusions2el
translates the ones Questa uses: it replays coverage-exclusions-rv64gc.do with tclsh to get the
coverage exclude commands, scans the RTL for // coverage off pragmas (which VCS ignores), and picks the
matching objects out of urg's full exclusion dump of the merged database, using urg's text report for
the source line of each branch condition.  Statements, branch arms, whole conditions, FSM states and
transitions, and expression and condition rows all have a counterpart.  A row maps to the vector of the
subexpression that has the row's input as an operand; for an & or | operand at its non-controlling
value that vector is shared with the other operands, so it is excluded only when all of their rows are.
The exclusions are applied to the merged database, as they now are for Questa, and the translation log
(vcs/cov/exclusions-rv64gc.log) says for each exclusion what it became or why VCS has no counterpart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Harris <David_Harris@hmc.edu>
--coveragesim {questa,vcs} picks the simulator for --ccov and --fcov; Questa stays the default.  With
vcs the same tests run (code coverage one ELF per simulation in lockstep with ImperasDV, functional
coverage in chunks of 16 ELFs), each writing a .vdb under sim/vcs, and VcsCodeCoverage or
VcsFunctCoverage merges and reports them.  Reuse of up-to-date code coverage, removal of databases no
selected test writes, and the check for runs that left no database work the same way; a .vdb counts as
complete only once the simulation has added its results, since run_vcs creates it before the run, and
the tool versions and run scripts that invalidate reuse are VCS's.  A dry run no longer deletes the
coverage databases of an earlier run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Harris <David_Harris@hmc.edu>
…erage

VCS counted the statements, branches and conditions behind a constant condition, such as the XLEN=32
arm of if (P.XLEN == 64) in csrc or ~P.ZICCLSM_SUPPORTED in trap, which Questa's optimizer removes before
it counts coverage, so they showed up as holes no test can fill.  -cm_noconst leaves them out.  On
rv64i/I, priv/Sm and tests/coverage, the uncovered priv lines drop from 37 to 1, as in Questa.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Harris <David_Harris@hmc.edu>
@davidharrishmc
davidharrishmc marked this pull request as draft October 2, 2026 22:11
Take main's final wsim/regression-wally/run_vcs (commands as argument lists
without a shell, coverage per simulation with a record of its ELFs) and put
the VCS coverage back on top:
- wsim: --ccov/--fcov accepted with --sim vcs; the VCS build hash includes
  fcovKey, and run_vcs and coverage.hier count as compile inputs.
- run_vcs: coverage compile/run options, fcov sources and per-run .vdb as
  argument lists; with --lockstep --fcov, leave ImperasDV's rvviTrace.sv out
  of main's glob of the RVVI sources; per-build compile log name.
- regression-wally: --coveragesim; databases under sim/<coveragesim>/ (.vdb
  for VCS), coverageMtime/removeCoverage for .vdb directories and their
  .elfs records, the VCS scripts and urg version among the coverage inputs,
  a dry run discards no coverage, and VcsCodeCoverage/VcsFunctCoverage.
- sim/Makefile: keep the VCS targets; main's removal of the riscv-dv
  targets stands.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: David Harris <David_Harris@hmc.edu>
@rosethompson
rosethompson marked this pull request as ready for review October 13, 2026 20:02
@rosethompson
rosethompson marked this pull request as draft October 13, 2026 20:03
Keep main's coverage32Dir next to the --coveragesim comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Harris <David_Harris@hmc.edu>
@davidharrishmc

Copy link
Copy Markdown
Contributor Author

Review notes: one fix before taking this out of draft. Questa, Verilator and non-coverage VCS runs are unaffected.

  • Over-exclusion on VCS. sim/questa/coverage-exclusions-rv64gc.do:71-72 uses -linerange … -code b -allfalse, which on Questa excludes only the implicit-else branch of the ifs at src/fpu/fctrl.sv:155,165. sim/vcs/exclusions2el:673-682 ignores -code and -allfalse and treats -linerange without -item as all of s/b/c/e on the line, so it also excludes the reachable fmv true arms and line 165's condition vectors, and logs both as PORTED. Honour -code/-allfalse, or report the form as NOT PORTED, and audit the other options the parser drops.
  • No staleness check on VCS. Questa fails on an exclusion that matches nothing (the vish-4014 grep), but exclusions2el turns "nothing on line L" into N/A (:489, :720) and VcsCodeCoverage (sim/Makefile:81-92) never fails on NOT PORTED or N/A, so a drifted exclusion passes silently on a VCS-only machine.
  • exclusions2el:825 runs tclsh with PATH=/usr/bin:/bin, and the install script doesn't install tclsh. shutil.which on the caller's PATH would be more portable.
  • The "41 of 296" and "scores match Questa" figures predate main's later exclusion and RTL changes.
  • Main's --ccov currently fails at the exclusion step: coverage-exclusions-rv64gc.do:488,490 still look up ~ITLBMissF & … in ifu.sv, which ifu: don't fetch while an ITLB A-update walk is pending #1934 renamed to ~ITLBMissOrUpdateRawF (ifu.sv:289-290). Because exclusions2el evaluates the same file, the VCS flow hits it too until main is fixed.

🤖 Generated with Claude Code

-code b -allfalse now excludes only the all-false branch (the implicit else or
missing case default) instead of every statement, branch arm and condition on
the line; -code limits -linerange to the named kinds, and any other option is
reported NOT PORTED instead of being ignored.

An exclusion that matches nothing in VCS and whose source line holds nothing of
the kind it excludes is reported STALE and makes the script, and so
VcsCodeCoverage, fail, as Questa does for an exclusion that matches nothing.

tclsh is found on the caller's PATH, and the package install script installs tcl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@davidharrishmc

Copy link
Copy Markdown
Contributor Author

Review items addressed in 527d837:

  • -code b -allfalse now excludes only the all-false branch (the implicit else, or a case's missing default). On fctrl.sv:155,165 that is one VCS vector each, where it used to be 13 objects including the reachable fmv arms. -code limits -linerange to the kinds it names, and any option the script doesn't translate is now logged NOT PORTED rather than silently dropped. The current .do file only uses -scope, -linerange, -item, -code, -allfalse, -fec*row, -ftrans and -fstate.
  • New STALE status: an exclusion that matches nothing in VCS and whose source line holds nothing of the kind it excludes fails exclusions2el, and so VcsCodeCoverage, matching Questa's behaviour on an exclusion that matches nothing.
  • tclsh is looked up on the caller's PATH, and wally-package-install.sh now installs tcl.
  • -vcs_assert off turns off only VCS's own internal checks: a failing immediate assert still reports under it.
  • The description now gives the current translation counts (43 of 299 not or partly ported).

The stale ~ITLBMissF patterns in coverage-exclusions-rv64gc.do are being fixed in a separate PR against main.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add coverage support to VCS

2 participants