A5: enforce absolute tolerance floors on near-zero reference fields

[pdlourenco]

Stacked on #210 (A3). Base is claude/a3-nan-sentinels. Retarget to main once #210 merges; the diff here is A5 only.

Summary

Implements plan item A5 of the review-v2 remediation plan (docs/plans/2026-10-04-review-v2-remediation-plan.md). It also covers the W-tolerance part of A9.

The defect: standing rule 1 (every field has an absolute floor) was a review convention only. Per the analysis (§4.1 rule 1, F8b), 78 of 113 output fields had no atol, 8 of them held exact or ~1e-16 zeros, and 6 relied on the default rtol.

The gate:

• The rule (testdata/refAtolViolations.m): a field holding an exact zero, or whose smallest finite magnitude is below 1e-12 of its largest, must carry <key>_atol. Keys resolve as in the validator: Response_real/_imag → Response_*.
• Enforcement:
  • generate_reference.m refuses to write such a vector.
  • The Octave validator fails it.
  • The Python consumer mirrors the rule (_atol_violations), with a unit test and a per-vector check.
  • Non-numeric output fields are an explicit error in both.

The floors added, at the cross-engine residual scale the sibling vectors already use:

• Response_atol 1e-12: siso_bt, siso_bt_large_M, siso_btfdr, freqmap_bt.
• internals: W_atol 1e-15 (also the plan A9 min-deps 1-ULP fix), Phi_xz_imag_atol 1e-14, DFT_imag_atol 1e-12.
• spectrogram: Frequency_atol 1e-14, for the 0 Hz bin.

Default-tolerance audit:

• Explicit Frequency_rel 1e-12 for the five vectors still on the default, as every other frequency grid already has.
• model_order.n_rel 0: the order is an integer, so the comparison is exact, matching Python's ==.

Rule 4 (payloads change only by regeneration):

• The tolerance blocks are edited directly, with the generator's matching entries in the same commit.
• A local regeneration reproduces all 32 tolerance blocks byte for byte; it was then discarded.
• No payload byte changes.

Spec sections touched

None. This is no algorithmic change; it touches the reference-vector tolerances and the consumers.

For the maintainer

• CONTRIBUTING.md rule 1 still says every field carries an atol; the gate enforces it for near-zero fields only. I left the governance wording alone. Options: reword rule 1 to the enforced form, or keep the stronger aspiration and add floors to the rest later.
• Hand-off to A4: the new Phi_xz_imag_atol, DFT_imag_atol and the five Frequency_rel entries are read by the Octave validator only today. Python compares Phi_* as complex using _real_rel, and doesn't compare DFT_* or those Frequency fields at all (analysis F2/F4). A4's structural loop picks them up.
• The 1e-12 threshold is the plan's own A5 wording, so I recorded no ADR.

Type of change

• Docs / tooling / CI only (vector tolerances, generator and consumer checks)

Checks

• Tests written against the rule, not against another port's output.
• Pre-push review: 5 findings.
  • Fixed in 658bda3:
    • the first floors were rtol × max|field|, which loosened mid-range entries (about 16× on a 0.52 Response_real entry) — replaced with residual-scale floors;
    • the README wording;
    • silent skip of non-numeric fields.
  • Noted above: the CONTRIBUTING wording; the A4 hand-off.
• Local CI: green.
  • Python: 484 passed, 4 skipped.
  • validate_reference.m: 32/32 under both Octave 11.3 and MATLAB R2024a.
  • Mutation check: dropping W_atol from reference_internals.json fails the vector in both consumers.
  • generate_reference.m runs with the gate passing. No matlab/ changes.

🤖 Generated with Claude Code

[pdlourenco]

Review of PR #211 at 658bda3: A5, absolute tolerance floors (stacked on #210)

Posted as a comment review because this session's GitHub identity is the PR author. Verdict: request changes. The floors themselves are sound and no comparison becomes vacuous. Before merge:

• the gate needs a test of its own failing branch (1);
• the clamp-capable NoiseSpectrum fields need floors (2);
• the MATLAB, Octave and validator jobs must actually run on this head (4). None has run yet.

The merge path after #210 is your call (Merge notes).

Method. Two reviewers with fresh context each took one lens: (1) the gate and the floor values, and (2) the 10 changed vectors, provenance and CI. An independent verifier then tried to refute each finding. A separate agent trial-merged all five open PRs and simulated the post-merge bot regeneration.

The work included:

• the Octave 8.4 validator and generator on this head;
• the full Python suite;
• per-field residual measurements against the R2025a bytes in both engines;
• mutations of the gate, the floors and the values;
• a Python 3.10 / numpy 1.22 / scipy 1.8 venv for the A9 W case.

Disclosure: this session wrote the remediation plan (item A5); it did not write this PR.

Should-fix (before merge)

1. Nothing tests the MATLAB/Octave gate itself (testdata/refAtolViolations.m).
   • It is the only implementation behind both the generator's refusal (generate_reference.m:960-966) and the Octave structural check (validate_reference.m:69-78), and every committed vector passes it.
   • Mutation: change line 39 from [key '_atol'] to [key '_rel'], then strip the floors from 6 vectors. The validator still reports 32 passed, 0 failed.
   • Python's TestToleranceFloors tests only the Python copy and has no NaN/Inf case. Without the non-finite filter, [nan, 0, 1] is no longer flagged, and nothing notices.
   • Fix: add a selfTestAtolFloors() beside selfTestNonFinite() (validate_reference.m:39), with these cases:
     • the Python fixture;
     • [NaN 0 1] and [Inf 1e-20 1], both flagged;
     • all-NaN, skipped;
     • a ragged cell holding a 0, flagged;
     • a char field, raising refAtolViolations:unsupported.
   • Add the NaN/Inf and ragged cases to the Python unit test too.
2. NoiseSpectrum can be exactly 0 by spec, and the gate then stops the generator. SPEC §2.7 clamps Φ̂_v to 0.
   • Under Octave's rng(42) stream, siso_bt's NoiseSpectrum has two exact zeros, and writeJSON errors on vector 1. The base branch's generator ran all 32 vectors under the same Octave.
   • That breaks the cloud workflow plan §10.1 relies on ("generator logic can be developed in CLOUD"). The same trip could hit R2025a after any change that lands a frequency on the clamp.
   • testdata/README.md:116 already says to floor every field that can hold a zero. Yet 9 NoiseSpectrum-carrying vectors have no NoiseSpectrum_atol; only btfdr_vecres and timeseries_etfe carry one.
   • Recommended: add NoiseSpectrum_atol (1e-12, as the siblings use, citing §2.7) to the clamp-capable vectors, in the generator and the committed blocks per rule 4. The time-series vectors return Φ̂_y and can be left out.
   • Either way, name the engine in the PR body's "generate_reference.m runs with the gate passing".
3. The README's "measured cross-engine residual scale" isn't backed by a recorded measurement, and "a floor binds only near zero" isn't true for W (testdata/README.md:55-58, commit 658bda3).
   • Measured here, residuals are ≤ ~1e-14 in Octave and ≤ 5.4e-15 in Python. So the 1e-12 floors sit about two orders above the residual, not at it. That is fine as headroom, but it should be described accurately.
   • W_atol = W_rel = 1e-15 deliberately also loosens W's mid-range entries (20 of 21), absorbing the A9 1-ULP misses. This is correct; the README just doesn't say so.
   • Fix: reword along these lines: "floors sit about two orders above the observed cross-engine residual (≤ ~1e-14 for the spectral fields) and well below rtol·|typical entry|, so they bind only at near-zero entries. The exception is W_atol, which also covers W's mid-range 1-ULP differences."
4. No MATLAB, Octave or validator job has run on this head. Only the push-triggered Python 3.10–3.14 jobs and Python lint ran, because the base isn't main.
   • Retargeting alone won't start them: the edited event isn't in any workflow's trigger types.
   • None of those jobs is a required check, so even once they run they won't block the merge.
   • The MATLAB job's generator step is the only place the new writeJSON gate runs under R2025a; plan §10.3 row A5 asks for that run.
   • Before merge: after retargeting and resolving the conflicts below, check by hand that these are green:
     • MATLAB tests & examples, including "Generate cross-language reference data";
     • Octave tests & examples;
     • both validators.
   • Say in the PR body that these have to be checked by hand.

Nits

5. A NaN, Inf or negative floor passes the gate. Both gates check only that <key>_atol exists, and in Octave a NaN atol makes every comparison pass. Python still fails the value test. The hole is in tolerance consumption and predates this PR. Validating each _rel/_atol as a finite, non-negative scalar in compareOutputs and _tol would close it; it could be a follow-up with A4.
6. Python's non-numeric error is a side effect of the ragged-list fallback. The message names a character, and a numeric string is accepted. Octave raises refAtolViolations:unsupported. Raise explicitly in _atol_violations.
7. The complex-key resolution is written in three places: refAtolViolations.m:20-24, compareOutputs and _atol_violations. A drift would fail closed. Still, add a pointer comment in compareOutputs, or factor out a shared helper when A4 generalises the complex keys.
8. testdata/README.md:120-121 is narrower than CONTRIBUTING rule 4. It still limits hand-edited tolerance blocks to "the same PR that adds the vector". Rule 4 allows any edit made with matching generator entries, which is what this PR relies on. Reconcile the two.
9. PR body: add the note testdata/README.md:39-41 asks for: the canonical R2025a round trip after merge re-stamps provenance on these 10 vectors, which still say aee812a, and their tolerance blocks already match the generator.

Merge notes (cross-PR): the 10 vector edits will conflict

• Why it conflicts: every push to main re-stamps provenance in all 32 single-line vectors. So once #210 merges (or anything else does), each of the 10 vectors edited here conflicts as a whole file. GitHub will refuse the merge until that is resolved.
• The options, your choice:
  • (a) Merge #211 into #210's branch first, then merge #210. That gives one regeneration and no JSON conflict, but it widens #210's scope.
  • (b) Recommended: merge #210 and wait for the bot commit. Then retarget #211 and merge origin/main into it (or git rebase --onto origin/main bf71afe if #210 was squash-merged).
    • Resolve each of the 10 files by taking main's bytes and re-inserting only this PR's flat "tolerance":{…} object. The other agent checked this mechanically: 10/10 files stay valid JSON, with main's input, output and provenance and this PR's tolerance. Key order and 1E-xx formatting already match the generator.
    • Don't hand-merge payloads or regenerate under Octave, and don't merge #211 while a bot regeneration job is still running.
• #209: also conflicts with this PR in one hunk of test_cross_validation.py. The resolution is in the #209 review.

Verified

• Gate rule: it matches the plan's A5 text (an exact zero, or min < 1e-12·max over finite magnitudes). It is keyed on what the comparator actually reads (Response_real/_imag → Response_atol).
• Coverage:
  • The base had 8 near-zero fields with no floor and 6 fields on the default rtol. This PR covers all 14. After it: 0 near-zero fields without a floor and 0 on the default rtol, out of 113 fields.
  • All 8 F8b / §4.1 rule-1 fields are floored.
• The 10 JSON files: only the tolerance object changed, as pure insertions. Payload and provenance are byte-identical, and the payload bytes are the canonical R2025a bytes from bot commit 7848f20. No locally generated bytes were committed. Rule 4 permits the hand edit.
• Generator matches the committed blocks: the generator's tolerance structs reproduce all 32 committed tolerance blocks (keys, order and values). A canonical regeneration therefore yields the same keys.
• No vacuous floor: each new atol is ≤ 4e-13 of max|field|, and only the zero or noise entries fall inside it.
• Threshold margin: every unfloored field has min/max ≥ 1.3e-8, so R2025a ULP drift can't flip the 1e-12 gate on main.
• W_atol: backed by a measurement. Base fails test_hann_window on numpy 1.22 / scipy 1.8 at W entries 0.0245 and 0.0545; this head passes 137/137.
• Mutations:
  • Dropping a floor fails both consumers.
  • Perturbing each floored entry by 10× its floor fails the Octave validator for all 7 fields.
  • Python checks W and Frequency per part. Response is checked as a complex modulus (|G| ≈ 0.5 at Nyquist), so it isn't vacuous. Phi_xz and DFT are Octave-only until A4, as the hand-off note says.
• Exactness: n_rel 0 is exact in both consumers. Frequency_rel tightens from 1e-6 to 1e-12 and still passes, at 2.2e-16.
• Tests: Octave validator 32/32. Python 484 passed, 4 skipped, matching the PR. ruff clean.

---

Generated by Claude Code