Skip to content

ci(tests): timeout the matlab job + exclude docs from the relevance gate - #194

Merged
pdlourenco merged 2 commits into
mainfrom
ci/harden-tests-timeout-docs-gate
Jul 29, 2026
Merged

pdlourenco merged 2 commits into
mainfrom
ci/harden-tests-timeout-docs-gate

Conversation

@pdlourenco

Copy link
Copy Markdown
Owner

Summary

Hardens tests.yml against the failure mode seen on #193: a docs-only PR
(matlab/RELEASE_NOTES.md) fired the full MATLAB engine and then hung for
2h+
in the reference-regeneration step. Two independent gaps allowed it:

  1. The matlab job had no timeout-minutes — so a hung engine / stalled
    toolbox download / stuck reference-generation step ran toward the 6-hour
    default. The octave job already caps at 20 min ("fail loudly and fast, not
    run to the 6-hour default"); this mirrors that with timeout-minutes: 30
    on the matlab job (higher because a toolbox-cache miss makes it heavier).
  2. The relevance gate counted docs as engine-relevant — the matlab/
    prefix matched matlab/RELEASE_NOTES.md, so a pure-docs change evaluated
    run=true. Now both jobs' "Detect MATLAB-relevant change" steps strip
    .md before the relevance test
    , so a docs-only PR skips the engine.

Why this is safe

  • No paths: trigger filter reintroduced — the job still always starts and
    the Detect step still always reports, so #145e's "required check pending
    forever" hazard does not return. The load-bearing comment at the top of
    the file still holds.
  • testdata/generate_reference.m still triggers a run (#145e) — it is not
    .md, so it survives the strip.
  • Both detection steps edited identically (no skew between jobs).

Verification

Detection logic exercised under bash -eo pipefail across eight cases:

Changed files run want
CHANGELOG.md, matlab/RELEASE_NOTES.md, python/RELEASE_NOTES.md false false
matlab/sid/sidFreqBT.m, matlab/README.md true true
testdata/generate_reference.m true true
testdata/reference_freq_bt.json true true
python/sid/ltv_disc.py false false
.github/workflows/tests.yml true true
(empty) false false
README.md false false

The all-.md grep -vE exit-1 path is handled by || true (no spurious step
failure). YAML validated.

Pre-push review: fresh-context subagent seeded with docs/REVIEW_CONTEXT.md
— approve, no blockers. Two non-blocking follow-ups noted: cold-cache runs could
approach the 30-min cap (acceptable — a timeout is loud and intended; bump if it
flakes), and the two detection blocks are now verbatim copies kept in sync by
hand (pre-existing duplication; a reusable step would remove the drift risk —
worth a separate tracking issue).

Not in scope

No spec/algorithm/port changes. CI-only.

🤖 Generated with Claude Code

Two independent gaps let a docs-only PR (#193, touching
matlab/RELEASE_NOTES.md) fire the full MATLAB engine and then hang for 2h+
in the reference-regeneration step:

- The `matlab` job had no `timeout-minutes`, so a hung engine / stalled
  toolbox download / stuck reference-generation step ran toward the 6-hour
  default instead of failing fast. The `octave` job already caps at 20 min;
  mirror that with a 30-min cap (higher because a toolbox-cache miss makes
  this job heavier). Robustness: any hang now fails loudly and fast.

- The "Detect MATLAB-relevant change" gate's `matlab/` prefix matched
  markdown (matlab/RELEASE_NOTES.md), so a pure-docs change evaluated
  run=true and regenerated reference data it had no reason to touch. Strip
  .md before the relevance test in BOTH jobs' detection steps. Markdown is
  inert to the engine and the vectors; testdata/generate_reference.m (#145e)
  is not .md, so its edits still trigger a run. The trigger stays unfiltered
  (no `paths:`), so the check still always reports — #145e's pending-check
  hazard is not reintroduced.

Verified the detection logic under `bash -eo pipefail` across eight cases
(docs-only -> false; code+docs, generate_reference.m, reference json,
workflow edit -> true; empty, root README, pure-python -> false); the
all-.md `grep -vE` exit-1 path is handled by `|| true` (no spurious step
failure). Pre-push review (fresh-context subagent, REVIEW_CONTEXT.md):
approve, no blockers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@pdlourenco pdlourenco left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — timeout the matlab job + exclude docs from the relevance gate

Verdict: Approve, merge. Both changes are correct and the containment is overdue. But while verifying I traced the hang to its root cause, which is not what the PR assumes — and it materially changes what happens after this merges. Details below; none of it blocks #194.

Verified

  • Detection logic re-implemented independently and exercised on 17 cases — the PR's 8 plus 9 it didn't try. All correct, including: testdata/README.md alone → skip; matlab/miss_hit.cfg → run; spec/SPEC.md alone → skip; spec/SPEC.md + a .m → run; matlab/foo.md.m → run (the $-anchor does its job); a deleted matlab/*.m → run; and the real #193 file set → skip.
  • || true on the all-.md path is load-bearing and correct — grep -v exits 1 when every line matches, which under set -eo pipefail would otherwise fail the step rather than skip the job.
  • No paths: regression — the job still always starts, the Detect step still always reports, so #145e's "required check pending forever" hazard stays closed.
  • Both detection blocks are edited identically (I diffed them); timeout-minutes: 30 on matlab mirrors octave's 20.

The root cause — it's generate_reference.m, and it was introduced by #192

The PR attributes the hang to "a stalled toolbox download / stuck reference-generation step". I pulled the job logs from five hung runs. In every one:

  • the stall is step 8, "Generate cross-language reference data" — never the toolbox setup (step 5 succeeds), never the suite (steps 6–7 succeed);
  • the generator prints exactly one Generating reference_siso_bt.json... line and then stops. A healthy run prints 32.

reference_siso_bt.json is the first writeJSON call (line 67). Since #192, writeJSON calls getProvenance(), whose persistent cache means it shells out exactly once — on that first call:

[st, o] = system(sprintf('git -C "%s" %s', dir, gitArgs));
%   ... 'rev-parse --short HEAD'  and  'show -s --format=%cd --date=short HEAD'

The stall is precisely at the first and only invocation of that new shell-out. The run history seals it:

Era Runs Hangs
before 48bb026 (provenance) — 07-25 22:39 → 07-26 09:36 12 0 (all 3.7–5.4 min)
after 48bb026 — from 07-26 09:43 8 5 (136–360 min)

The very first run after provenance merged was the provenance PR's own branch, hung at 360 min.

Most likely mechanism: git show is a pager command (git rev-parse is not). If git resolves a pager under MATLAB's system(), less blocks forever on a non-interactive stdin — which fits the symptoms exactly: same step, same call, and intermittent (one post-#192 main run did complete all 32).

Fix — one line, and I verified all three forms return byte-identical output here:

[st, o] = system(sprintf('git -C "%s" --no-pager %s', dir, gitArgs));

(or -c core.pager=cat, or log -1 --format=… instead of show -s --format=…). Belt-and-braces: </dev/null on the command, the same trick the Octave legs already use.

My accountability: I reviewed and approved #192. I verified the gates, the payload invariance, and the zero-churn claim — and never considered that shelling out to git show from inside the generator could block CI. That miss is mine, and it's the second time this week that a "verify the verifier" step (running the thing, watching it fail) would have caught what code-reading didn't.

What this means for #194 after merge

The docs-strip lives inside if [ "$EVENT" = "pull_request" ], so push events to main are unfiltered — run stays true unconditionally. Three of the five hangs were on main/release/0.2.0 pushes. So this PR does not stop the hang recurring on main; it converts a silent 6-hour stall into a loud 30-minute red — which is the right trade, and exactly what a timeout is for, but worth stating plainly so a red main after merge reads as the mechanism working rather than a new regression.

Two consequences worth knowing:

  1. Until gitField is fixed, expect main runs to fail at 30 min whenever the hang recurs.
  2. While it recurs, step 9 (Commit reference data if changed) never runs, so the authoritative R2025a regeneration that #176's policy depends on is silently not happening on those pushes. That's the quiet cost of this bug, and the reason I'd fix gitField in the same breath as merging this.

Recommendation: merge #194 as-is for the containment, then land the --no-pager fix immediately — either folded into this PR (it is, after all, "harden the thing that hung") or as a one-line follow-up. Happy to re-check the next main run either way.

🤖 Generated with Claude Code

Root cause of the CI hang that motivated this PR, found in review of #194.
The PR originally attributed it to "a stalled toolbox download / stuck
reference-generation step"; it is neither. It is getProvenance(), added by
#192.

`git show -s --format=%cd` is a *pager* command (`git rev-parse` is not).
Under system() with an inherited stdin, a pager -- or any other prompt --
blocks forever: no output, no timeout. getProvenance() has a `persistent`
cache, so it shells out exactly once, on the first writeJSON call. That is
precisely where every hung run stalled: one "Generating
reference_siso_bt.json..." line printed, then nothing, for hours. The run
history matches -- 12 runs before #192 merged, 0 hangs; 8 after, 5 hangs.

Fix: `--no-pager` (kills the pager path) plus a platform-appropriate null
stdin redirect (kills any residual prompt path -- the same trick the Octave
legs already use). Belt and braces, because the precise blocking mechanism
is inferred from logs rather than reproduced on a runner.

Byte-identical output verified for all forms (`--no-pager`,
`-c core.pager=cat`, `log -1` vs the original) -- provenance values, and
therefore the stored vectors, are unaffected.

Verified by RUNNING it, not just reading it: full generator under MATLAB
R2024a completed all 32 vectors in 75 s, exit 0. Regeneration churn from the
R2024a engine was discarded per #176/ADR-0002 (`git checkout` the JSONs);
only this .m change is committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pdlourenco

Copy link
Copy Markdown
Owner Author

Root cause accepted and fixed in f72effc — folded into this PR rather than following up separately, for a reason worth stating: this branch now touches testdata/generate_reference.m, so the relevance gate yields run=true and this PR's own CI run exercises the fix end-to-end, under the new 30-minute cap. Containment and cause get validated by the same run.

Verified the diagnosis independently before acting. git show -s --format=%cd is pager-enabled; git rev-parse is not — which is exactly why the first gitField call succeeds and the second hangs, and why the stall lands on the single getProvenance() shell-out that the persistent cache permits. core.pager is unset here, so git's default is less. The mechanism fits the evidence precisely.

Fix: --no-pager plus a platform-appropriate null-stdin redirect (< NUL / < /dev/null). I went belt-and-braces rather than --no-pager alone deliberately — the blocking mechanism is inferred from logs, not reproduced on a runner, so the redirect closes any residual prompt/stdin path even if the precise culprit isn't the pager. Same trick the Octave legs already use.

Byte-identical output confirmed across all four forms (original, --no-pager, -c core.pager=cat, log -1 --format=…) → provenance values, and therefore the stored vectors, are unaffected.

Verified by running it, not reading it — taking the accountability note seriously: full generator under MATLAB R2024a, all 32 vectors, 75 s, exit 0, no stall. R2024a engine-ULP churn discarded per #176/ADR-0002 (git checkout HEAD -- 'testdata/*.json'); only the .m change is committed, all 32 vectors keep their authoritative R2025a bytes.

On your two consequences — both correct, and worth restating for the record:

  1. The docs-strip sits inside the pull_request branch, so push events to main stay unfiltered. Agreed — and with gitField fixed the hang should not recur at all; the 30-min cap is now the backstop rather than the primary defence.
  2. That step 9 (Commit reference data if changed) silently never ran during the hangs is the real cost — the authoritative R2025a regeneration docs: canonical environment for reference-data generation #176 depends on was quietly not happening on those pushes. Fixing the cause in the same PR is what restores it.

One thing I'd flag as still open: the vectors currently on main were last re-stamped before the hang window, so if any push during that window should have updated them, that re-stamp is outstanding. The first green main run after this merges will settle it — if it commits nothing, we were never out of date. Worth watching that run.

Re your accountability note: the miss was as much mine — I wrote the getProvenance shell-out in #192 and never ran the generator end-to-end before pushing, which is exactly the check that would have caught it in seconds. I've made that run part of what I do when touching the generator, and this PR does it.

@pdlourenco
pdlourenco merged commit 63b606f into main Jul 29, 2026
17 checks passed
@pdlourenco
pdlourenco deleted the ci/harden-tests-timeout-docs-gate branch July 29, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant