tests(exit_parity): single return-only fixture can't catch the throw/panic/exit divergences its doc claims to guard (masks #779)

[dekobon]

Summary

tests/exit_cross_language_parity.rs documents itself as the guardian of cross-language exit-metric parity, but its single fixture exercises only return — the one exit construct counted identically in every language — so it is structurally incapable of detecting the throw/raise/panic/error/exit/yield divergences that are the real cross-language inconsistencies (e.g. #779). The module doc's claims overstate what the test guards.

Location

• tests/exit_cross_language_parity.rs:8-22 (module doc)
• tests/exit_cross_language_parity.rs:38-205 (the single early_exit_in_while_loop_parity test)

Evidence

The module doc claims comprehensive cross-language coverage:

//! Per lesson 11 ... the same logical construct must produce the same
//! metric value across every language we support ...
//!
//! ... Every supported language tested below reports `exit_max() == 1`.

Two problems make this misleading:

1. "every language we support" / "Every supported language tested below" is false.
   src/metrics/nexits.rs implements Exit for 23 languages (Python, Mozjs,
   Javascript, Typescript, Tsx, Cpp, Mozcpp, C, Objc, Java, Groovy, Rust, Csharp,
   Go, Perl, Kotlin, Lua, Bash, Tcl, Irules, Php, Ruby, Elixir). The test covers
   only 12 (Rust, Cpp, Java, Javascript, Typescript, Csharp, Php, Python, Bash,
   Kotlin, Groovy, Go). Lua, Ruby, Elixir, Perl, Tcl, Irules, Objc, C, Mozjs,
   Mozcpp, and Tsx are absent — including the two languages named in the open
   divergence metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 (Lua is omitted entirely; Go is present but see point 2).

2. The fixture only exercises return, which is uniform everywhere. The body
   is a while/if-guarded single return. Every language's Exit impl counts
   its return node, so this path is trivially in parity. The genuine
   cross-language exit divergences live in the non-return constructs:

   • metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779: Go panic(...) and Lua error(...)/os.exit(...) are uncounted,
     while Python raise, PHP throw/exit, Bash exit, Elixir
     exit/raise, Csharp/Kotlin throw, JS-family throw, and generator
     yield all ARE counted (src/metrics/nexits.rs:166-382).

   Because the fixture never uses any of these, the test passes identically
   whether or not metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 is fixed. It cannot catch metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779, metrics(npa): PHP counts enum cases as attributes; Java/Kotlin/Rust/C# do not (cross-language inconsistency) #781, or metrics(loc): multi-line string interior rows counted as PLOC in Python but as blank in Perl (cross-language inconsistency) #778-style
   divergence — yet its doc tells a reader those are guarded.

The sibling tests/cyclomatic_cross_language_parity.rs covers a broader set
(including Tcl, Irules, Objc) and multiple fixture shapes, underscoring that
the exit-parity test's single trivial fixture is the gap, not a deliberate
minimalism.

Expected Behavior

Either the doc should be scoped down to what the test actually proves ("the
return exit construct is in parity across these 12 languages"), or — better —
the test should add fixtures that exercise the divergent exit constructs
(throw/raise/panic/error/os.exit/yield) across the languages that
support them, so the suite actually guards the cross-language exit-parity
contract it claims to.

Actual Behavior

The doc claims the test guards exit parity across "every supported language,"
but the test only proves return-parity across 12 of 23 languages and is blind
to the exact divergences (#779) it implies it protects against. A reviewer
trusting this file would wrongly believe Go-panic / Lua-error parity is covered.

Impact

False confidence in cross-language exit-metric parity. A regression that breaks
non-return exit counting in any language — or a new language wired up with an
incomplete Exit set — sails past this suite while the module doc asserts it is
covered. This is the test-quality masking gap behind #779 remaining open without
a guarding test.

---

Resolution (commit 6a671493)

Fixed (preferred option): tests/exit_cross_language_parity.rs now
exercises the divergent exit constructs. Each language fixture has two
counted exits — a return plus its abrupt-exit construct
(throw/raise/panic/error/process-exit), or two returns for
return-only languages — asserting nexits_sum() == 2 per language.
Coverage extended from 12 to all 23 languages with an Exit impl. The
module doc was rewritten to match. Test-via-revert confirmed the new
fixtures catch dropped Go-panic/Lua-error/Kotlin-throw counting
(the #779 masking gap). See the resolution comment for details.

[dekobon]

General Plan for Fix

Root Cause

The test was written to demonstrate return-parity (the lesson-11 pattern) but
its module doc was phrased as if it guards all exit constructs across all
languages. The fixture deliberately uses only return so Java's wrapping class
does not skew exit_max, but return is the single exit kind every language
counts identically — so the test exercises the one path that is never the source
of cross-language disagreement. The divergent constructs (throw/raise/
panic/error/os.exit/yield, per src/metrics/nexits.rs:166-382) are the
ones that actually differ (#779), and none are tested.

Implementation Steps

1. Minimum (doc-only): Rescope the module doc so it claims only what the
   fixture proves — return-construct parity across the 12 listed languages —
   and drop "every supported language." This removes the false-confidence claim
   immediately.
2. Preferred (close the masking gap): Add fixtures that exercise the
   divergent exit constructs across the languages that support them, asserting
   the documented cross-language contract. Suggested groupings:
   • throw/raise parity: Python raise, Csharp/Java/Kotlin/JS-family
     throw, PHP throw, Elixir raise — all expect a counted exit.
   • process-exit parity: Bash exit, PHP exit, Elixir exit, and the
     metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 targets (Go panic, Lua error/os.exit). This fixture is the
     direct regression guard for metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 — it should be marked as such and is
     expected to FAIL until metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 is fixed (gate it behind metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 or add it in
     the same PR that fixes metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779).
3. Add the absent languages (Lua, Ruby, Elixir, Perl, Tcl, Irules, Objc, C,
   Mozjs, Mozcpp, Tsx) to at least the existing return-parity assertion so the
   "every language" intent is actually met.

Tests to Add/Update

• Rescope or split early_exit_in_while_loop_parity's doc + assertions.
• New throw_raise_parity test across the throw/raise languages.
• New process_exit_parity test (the metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 guard) across Bash/PHP/Elixir/Go/Lua.

Verification

• cargo test -p big-code-analysis --test exit_cross_language_parity passes
  (with the metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 guard either fixed or explicitly gated).
• cargo clippy -p big-code-analysis clean.
• Test-via-revert: confirm the new throw/exit fixtures FAIL against current
  src/metrics/nexits.rs for Go/Lua (proving they would have caught metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779),
  then pass after metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779's fix.

[dekobon]

Resolution (fixed on worktree-agent-a94387632262ba815, commit 6a671493)

Adopted the preferred option from the plan — closed the masking gap
rather than only rescoping the doc.

tests/exit_cross_language_parity.rs no longer uses a single
return-only fixture. Each language now gets a fixture that exercises
two counted exits: a plain return plus the language's
abrupt-exit construct, asserting nexits_sum() == 2 with a per-language
label:

• return + throw: Cpp, Mozcpp, Objc (@throw), Java, Groovy,
  Csharp, Kotlin, Javascript, Mozjs, Typescript, Tsx, Php.
• return + raise: Python.
• return + process-exit: Bash (exit).
• return + builtin abrupt-exit call (the metrics(nexits): Go panic and Lua error/os.exit uncounted as exits (cross-language inconsistency) #779 targets): Go (panic),
  Lua (error).
• return + ?: Rust.
• abrupt-exit only (no return node): Elixir (raise + throw).
• return-only languages (two returns): C, Perl, Ruby, Tcl, Irules.

Coverage now spans all 23 languages with an Exit impl in
src/metrics/nexits.rs (previously 12) — the absent set (Lua, Ruby,
Elixir, Perl, Tcl, Irules, Objc, C, Mozjs, Mozcpp, Tsx) is added.

The module doc was rewritten to state exactly what the test proves and
why a return-only fixture is insufficient.

Test-via-revert verification (per .claude/rules/testing.md):
temporarily dropping the abrupt-exit counting in nexits.rs made the
matching assertion fail — Kotlin throw → kotlin fails at 1 vs 2; Go
panic (#779 guard) → go fails at 1 vs 2; Lua error similarly. The
production code was restored to pristine (git diff empty) and the test
passes.

Note: #779 is already CLOSED/fixed, so Go panic / Lua error are
counted today — the new fixtures are the regression guard that was
missing when #779 was filed.

cargo fmt --check + cargo clippy --workspace --all-targets -D warnings clean. Test-only change: no metric behaviour change, no
snapshot/integration-submodule drift. Branch held local per policy.