Repository navigation
ci(tests): timeout the matlab job + exclude docs from the relevance gate - #194
Conversation
Two independent gaps let a docs-only PR (#193, touching matlab/RELEASE_NOTES.md) fire the full MATLAB engine and then hang for 2h+ in the reference-regeneration step: - The `matlab` job had no `timeout-minutes`, so a hung engine / stalled toolbox download / stuck reference-generation step ran toward the 6-hour default instead of failing fast. The `octave` job already caps at 20 min; mirror that with a 30-min cap (higher because a toolbox-cache miss makes this job heavier). Robustness: any hang now fails loudly and fast. - The "Detect MATLAB-relevant change" gate's `matlab/` prefix matched markdown (matlab/RELEASE_NOTES.md), so a pure-docs change evaluated run=true and regenerated reference data it had no reason to touch. Strip .md before the relevance test in BOTH jobs' detection steps. Markdown is inert to the engine and the vectors; testdata/generate_reference.m (#145e) is not .md, so its edits still trigger a run. The trigger stays unfiltered (no `paths:`), so the check still always reports — #145e's pending-check hazard is not reintroduced. Verified the detection logic under `bash -eo pipefail` across eight cases (docs-only -> false; code+docs, generate_reference.m, reference json, workflow edit -> true; empty, root README, pure-python -> false); the all-.md `grep -vE` exit-1 path is handled by `|| true` (no spurious step failure). Pre-push review (fresh-context subagent, REVIEW_CONTEXT.md): approve, no blockers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pdlourenco
left a comment
There was a problem hiding this comment.
Review — timeout the matlab job + exclude docs from the relevance gate
Verdict: Approve, merge. Both changes are correct and the containment is overdue. But while verifying I traced the hang to its root cause, which is not what the PR assumes — and it materially changes what happens after this merges. Details below; none of it blocks #194.
Verified
- Detection logic re-implemented independently and exercised on 17 cases — the PR's 8 plus 9 it didn't try. All correct, including:
testdata/README.mdalone → skip;matlab/miss_hit.cfg→ run;spec/SPEC.mdalone → skip;spec/SPEC.md+ a.m→ run;matlab/foo.md.m→ run (the$-anchor does its job); a deletedmatlab/*.m→ run; and the real #193 file set → skip. || trueon the all-.mdpath is load-bearing and correct —grep -vexits 1 when every line matches, which underset -eo pipefailwould otherwise fail the step rather than skip the job.- No
paths:regression — the job still always starts, the Detect step still always reports, so #145e's "required check pending forever" hazard stays closed. - Both detection blocks are edited identically (I diffed them);
timeout-minutes: 30onmatlabmirrorsoctave's 20.
The root cause — it's generate_reference.m, and it was introduced by #192
The PR attributes the hang to "a stalled toolbox download / stuck reference-generation step". I pulled the job logs from five hung runs. In every one:
- the stall is step 8, "Generate cross-language reference data" — never the toolbox setup (step 5 succeeds), never the suite (steps 6–7 succeed);
- the generator prints exactly one
Generating reference_siso_bt.json...line and then stops. A healthy run prints 32.
reference_siso_bt.json is the first writeJSON call (line 67). Since #192, writeJSON calls getProvenance(), whose persistent cache means it shells out exactly once — on that first call:
[st, o] = system(sprintf('git -C "%s" %s', dir, gitArgs));
% ... 'rev-parse --short HEAD' and 'show -s --format=%cd --date=short HEAD'The stall is precisely at the first and only invocation of that new shell-out. The run history seals it:
| Era | Runs | Hangs |
|---|---|---|
before 48bb026 (provenance) — 07-25 22:39 → 07-26 09:36 |
12 | 0 (all 3.7–5.4 min) |
after 48bb026 — from 07-26 09:43 |
8 | 5 (136–360 min) |
The very first run after provenance merged was the provenance PR's own branch, hung at 360 min.
Most likely mechanism: git show is a pager command (git rev-parse is not). If git resolves a pager under MATLAB's system(), less blocks forever on a non-interactive stdin — which fits the symptoms exactly: same step, same call, and intermittent (one post-#192 main run did complete all 32).
Fix — one line, and I verified all three forms return byte-identical output here:
[st, o] = system(sprintf('git -C "%s" --no-pager %s', dir, gitArgs));(or -c core.pager=cat, or log -1 --format=… instead of show -s --format=…). Belt-and-braces: </dev/null on the command, the same trick the Octave legs already use.
My accountability: I reviewed and approved #192. I verified the gates, the payload invariance, and the zero-churn claim — and never considered that shelling out to git show from inside the generator could block CI. That miss is mine, and it's the second time this week that a "verify the verifier" step (running the thing, watching it fail) would have caught what code-reading didn't.
What this means for #194 after merge
The docs-strip lives inside if [ "$EVENT" = "pull_request" ], so push events to main are unfiltered — run stays true unconditionally. Three of the five hangs were on main/release/0.2.0 pushes. So this PR does not stop the hang recurring on main; it converts a silent 6-hour stall into a loud 30-minute red — which is the right trade, and exactly what a timeout is for, but worth stating plainly so a red main after merge reads as the mechanism working rather than a new regression.
Two consequences worth knowing:
- Until
gitFieldis fixed, expectmainruns to fail at 30 min whenever the hang recurs. - While it recurs, step 9 (
Commit reference data if changed) never runs, so the authoritative R2025a regeneration that #176's policy depends on is silently not happening on those pushes. That's the quiet cost of this bug, and the reason I'd fixgitFieldin the same breath as merging this.
Recommendation: merge #194 as-is for the containment, then land the --no-pager fix immediately — either folded into this PR (it is, after all, "harden the thing that hung") or as a one-line follow-up. Happy to re-check the next main run either way.
🤖 Generated with Claude Code
Root cause of the CI hang that motivated this PR, found in review of #194. The PR originally attributed it to "a stalled toolbox download / stuck reference-generation step"; it is neither. It is getProvenance(), added by #192. `git show -s --format=%cd` is a *pager* command (`git rev-parse` is not). Under system() with an inherited stdin, a pager -- or any other prompt -- blocks forever: no output, no timeout. getProvenance() has a `persistent` cache, so it shells out exactly once, on the first writeJSON call. That is precisely where every hung run stalled: one "Generating reference_siso_bt.json..." line printed, then nothing, for hours. The run history matches -- 12 runs before #192 merged, 0 hangs; 8 after, 5 hangs. Fix: `--no-pager` (kills the pager path) plus a platform-appropriate null stdin redirect (kills any residual prompt path -- the same trick the Octave legs already use). Belt and braces, because the precise blocking mechanism is inferred from logs rather than reproduced on a runner. Byte-identical output verified for all forms (`--no-pager`, `-c core.pager=cat`, `log -1` vs the original) -- provenance values, and therefore the stored vectors, are unaffected. Verified by RUNNING it, not just reading it: full generator under MATLAB R2024a completed all 32 vectors in 75 s, exit 0. Regeneration churn from the R2024a engine was discarded per #176/ADR-0002 (`git checkout` the JSONs); only this .m change is committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Root cause accepted and fixed in Verified the diagnosis independently before acting. Fix: Byte-identical output confirmed across all four forms (original, Verified by running it, not reading it — taking the accountability note seriously: full generator under MATLAB R2024a, all 32 vectors, 75 s, exit 0, no stall. R2024a engine-ULP churn discarded per #176/ADR-0002 ( On your two consequences — both correct, and worth restating for the record:
One thing I'd flag as still open: the vectors currently on Re your accountability note: the miss was as much mine — I wrote the |
Summary
Hardens
tests.ymlagainst the failure mode seen on #193: a docs-only PR(
matlab/RELEASE_NOTES.md) fired the full MATLAB engine and then hung for2h+ in the reference-regeneration step. Two independent gaps allowed it:
matlabjob had notimeout-minutes— so a hung engine / stalledtoolbox download / stuck reference-generation step ran toward the 6-hour
default. The
octavejob already caps at 20 min ("fail loudly and fast, notrun to the 6-hour default"); this mirrors that with
timeout-minutes: 30on the
matlabjob (higher because a toolbox-cache miss makes it heavier).matlab/prefix matched
matlab/RELEASE_NOTES.md, so a pure-docs change evaluatedrun=true. Now both jobs' "Detect MATLAB-relevant change" steps strip.mdbefore the relevance test, so a docs-only PR skips the engine.Why this is safe
paths:trigger filter reintroduced — the job still always starts andthe Detect step still always reports, so #145e's "required check pending
forever" hazard does not return. The load-bearing comment at the top of
the file still holds.
testdata/generate_reference.mstill triggers a run (#145e) — it is not.md, so it survives the strip.Verification
Detection logic exercised under
bash -eo pipefailacross eight cases:runCHANGELOG.md,matlab/RELEASE_NOTES.md,python/RELEASE_NOTES.mdmatlab/sid/sidFreqBT.m,matlab/README.mdtestdata/generate_reference.mtestdata/reference_freq_bt.jsonpython/sid/ltv_disc.py.github/workflows/tests.ymlREADME.mdThe all-
.mdgrep -vEexit-1 path is handled by|| true(no spurious stepfailure). YAML validated.
Pre-push review: fresh-context subagent seeded with
docs/REVIEW_CONTEXT.md— approve, no blockers. Two non-blocking follow-ups noted: cold-cache runs could
approach the 30-min cap (acceptable — a timeout is loud and intended; bump if it
flakes), and the two detection blocks are now verbatim copies kept in sync by
hand (pre-existing duplication; a reusable step would remove the drift risk —
worth a separate tracking issue).
Not in scope
No spec/algorithm/port changes. CI-only.
🤖 Generated with Claude Code