testsuite: WAST runner assertion counts are non-deterministic (~18/run) — can't gate conformance on the number

[avrabe]

Two full runs of the IDENTICAL tree report different WAST assertion totals: 65776 pass vs 65794 pass (~18 vary run-to-run); total executed assertions also drift (66325 vs 66307). So before/after assertion-count comparison is unreliable — a real change's delta (+2 for #359) is smaller than the ±18 run-to-run noise, which can mask or fake a regression.

The FILE-level result is stable: Files: N pass/M fail and the failing-file SET reproduce. The #149 no-regression check had to be done by diffing failing-file sets (comm), not counts:

files failing AFTER but not BASE: (none)              # no regression
files failing BASE but not AFTER: type-subtyping.wast # the fix

Impact: blocks gating conformance in CI on the assertion count (not reproducible); slowed #149 verification (multiple ~20-min full runs + set-diffing).

Suspected cause: order/state-dependence (a module failing instantiation changes how many downstream assert_returns execute) and/or runner parallelism.

Asks: (1) make the executed-assertion count deterministic (fixed order, isolated per-module state), and/or (2) emit a stable machine-readable per-file report (--wast-report JSON) so CI gates on the file-level set/counts (already stable). Surfaced while landing #359 (GC call_indirect subtyping fix, #149).

[avrabe]

Root-cause progress (narrowing) + the pragmatic path.

Ruled out by inspection:

• File discovery is sorted (collect_wast_files → files.sort(), wast.rs:1706) — file order is deterministic, not the cause.
• The runner's modules/instance_ids/precise_types HashMaps are keyed-lookup only (never iterated) — not the cause.
• module.exports/imports use ordered DirectMap/BoundedMap, so engine export/import iteration is deterministic.

Narrowed to: the GC type path. Every file whose result varied run-to-run is GC/cast-related (br_on_cast*, ref_cast*, exact*, custom-descriptors, return_call_ref). GC subtype results flow from type canonicalization (type_canonical_ids) and the subtype walker. A non-deterministic structure there (a std HashMap/HashSet iterated to assign canonical ids or resolve a "matching" type, subject to Rust's per-process random hash seed) would yield different canonical ids → different subtype outcomes → ±N assertions per run, concentrated in GC files. Pinning the exact site needs empirical localization (run the GC subset twice, diff per-file counts, trace the varying file's path) — a focused debug session, not inspection.

Important nuance for the CI ask: the file-level result is far more stable than the assertion count, but not perfectly (one base run reported 19 failing files, another 18 — at least partly a measurement artifact). So the robust CI gate should use the runner's own structured --wast-report output and gate on the failing-file SET vs a checked-in baseline (fail on a NEW failing file = regression; tolerate the known set), NOT on the assertion count. That's gateable today on the stable signal, independent of fixing the assertion-count determinism.

Recommended sequencing: (1) ship a CI conformance gate on the file-set baseline (scheduled weekly + workflow_dispatch, like the security-audit cron — a ~20-min full run is too heavy per-PR) so regressions are caught now; (2) the assertion-count determinism fix (this issue) as a focused debug session that localizes the GC-path non-determinism. (1) doesn't block on (2).

[avrabe]

RESOLVED — this was a false alarm. Not runtime non-determinism; a submodule-version confound. Systematic localization found it:

• Same-worktree runs are deterministic: the cast cluster gave 242 pass / 413 fail on two consecutive runs (identical), and a full-suite before/after on the SAME worktree (testsuite @7e0b83a) gave identical non-type-subtyping results (65679 passing both runs, byte-for-byte).
• The apparent ±18-assertion "non-determinism" came from comparing runs in two different git worktrees: the main worktree has external/testsuite checked out at 7e0b83a, but the recorded gitlink is c337f0d (the submodule is modified, M). A fresh git worktree (e.g. an isolation-mode subagent) checks out the gitlink c337f0d — a different test-suite version → different files/assertions → the count "varied" between worktrees, never within one.

So the runner is deterministic. The real defect is that external/testsuite is unpinned: the working tree (7e0b83a, the commit the docs + all our measurements use) doesn't match the committed gitlink (c337f0d). Any fresh checkout / CI / worktree silently tests a different suite.

Fix (trivial): pin the submodule — commit the gitlink so it matches the intended/tested commit (7e0b83a per architecture-components.yaml's verified baseline), i.e. git add external/testsuite && git commit. Then every checkout/CI/worktree runs the same suite → the conformance number is reproducible.

Consequence — the CI gate is unblocked. My earlier conclusion here ("must make the runner deterministic before gating") was based on the confounded data and is wrong: the number is already reproducible same-commit. Once the submodule is pinned, a CI conformance gate (run the suite, diff the failing-file set vs a checked-in baseline, fail on a NEW file) is straightforward — no determinism work needed. Recommend closing this as "not a bug (submodule confound)" once the pin lands; tracking the pin + CI gate separately.

(Also: this confound meant the original #359 no-regression check compared mismatched testsuites; re-verified correctly same-worktree same-testsuite — clean +2 pass, zero regression. #359 stands.)

[avrabe]

Resolved by #361 (merged, main bc3854c). Not a runner bug — external/testsuite was unpinned (gitlink ≠ checkout), so different worktrees/CI ran different suites; same-commit runs are deterministic. #361 pins the gitlink to latest (193e551, == checkout, clean) so conformance is reproducible, and adds a monthly re-pin workflow (.github/workflows/testsuite-repin.yml) to keep it pinned-but-current. New reproducible baseline: 263/281 files, 65980 assertions pass / 529 fail. Closing as fixed (submodule confound).