Skip to content

ci(conformance): keep the Maestro version probe's failure evidence - #4503

Merged
thymikee merged 2 commits into
mainfrom
fix/4455-maestro-version-probe-diagnostics
Nov 13, 2026
Merged

thymikee merged 2 commits into
mainfrom
fix/4455-maestro-version-probe-diagnostics

Conversation

@thymikee

@thymikee thymikee commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

The Maestro pin check hid stderr and masked CLI failures. The workflow now probes the CLI through one script that retains exit/signal/spawn diagnostics and each stream’s last 8 KiB. Only a successful CLI reporting the pinned version passes; no retry, fallback or pin change.

Version matching scans the full stdout stream independently of diagnostic retention. This preserves versions before long trailing notices and prevents truncation from turning 12.5.1 into the pinned 2.5.1. Matching state and diagnostics remain bounded.

5 touched files. No scope expansion. Closes #4455.

Validation

Head: 94f427458ddb05f56f8bcee4f387e03aad34a989.

  • Frozen install/build, full-repository formatting, focused lint and all 77 conformance tests passed locally; 14 exercise the probe.
  • Original base efe6bb9: nonzero CLI exit with matching stdout was accepted. Worker head bae08d4 rejects it and retains stderr.
  • On bae08d4, truncating 12.5.1 falsely passed; a valid pin before oversized trailing notices falsely failed. Both controls pass at this head. Oversized numeric versions also remain bounded and fail closed.
  • Exact affected validation: intermediate 8f42d7 failed scanner complexity; replaced with explicit state transitions. Current-head rerun was cancelled while waiting for live host-lock owner78515; no checks executed. CI runs the Fallow and conformance gates. Pre-existing untracked website output was preserved.
  • CI at this head: 24 checks passed, two skipped; maintainer found no code blockers. No live conformance or release dispatch. A hung CLI still inherits the workflow’s existing job timeout; this change bounds retained diagnostics, not execution time.

…4455)

The layer-3 pin check parsed `maestro --version` inside a pipeline that sent
stderr to /dev/null and masked the pipeline exit with `|| true`. Its whole
failure record was `expected=<pin> actual=`, so the November 9 run could not
tell a CLI that never started from one that printed nothing parseable.

Run the probe through scripts/maestro-conformance/verify-cli-version.mjs. It
records the exit status, signal, spawn error and both streams, retains each
stream's last 8 KiB newest-first, and reports one verdict: command failure,
missing or unparseable version, mismatch, or match. The pin invariant is
unchanged and still fails closed; a probe that did not complete cleanly is
never accepted on the strength of its stdout.

The fixture-executable suite covers a matching version inside notices, a wrong
version, empty and unparseable stdout, a nonzero exit with matching-looking
stdout, an absent executable, and bounded oversized output.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.25 MB 6.25 MB +56 B
Package (unpacked) 6.25 MB 6.25 MB +56 B
Package (download) 1.90 MB 1.90 MB +14 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 21.3 ms 22.3 ms +1.0 ms
CLI --help 43.4 ms 43.0 ms -0.4 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread scripts/maestro-conformance/README.md Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 1 still open · 1 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
scripts/maestro-conformance/verify-cli-version.mjs:103 Probe has no deadline; a hung maestro --version stalls the 90-min job

Reviewed with this repository's own Sentinel configuration (efe6bb9eece4).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of bae08d40 · 13.11.2026 10:09 UTC

Apex Sentinel · ✅ Approve

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
scripts/maestro-conformance/verify-cli-version.mjs:57 Probe has no deadline; a hung maestro --version stalls the 90-min job
scripts/maestro-conformance/verify-cli-version.test.mjs:24 Scratch dirs leak when the suite runs outside scripts/node-test-tmpdir.ts

Reviewed with this repository's own Sentinel configuration (efe6bb9eece4).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on bae08d4. The verdict and the full list are in the summary comment.

@thymikee

Copy link
Copy Markdown
Member Author

No code blockers found at 94f4274: version matching reads the full stdout stream independently of the retained diagnostic tail, fixing the earlier version-truncation finding. CI has 24 successful checks and two skipped checks; the scheduled Conformance Differential itself was not run. There are no conflicts.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 13, 2026
@thymikee
thymikee merged commit 80075b6 into main Nov 13, 2026
26 checks passed
@thymikee
thymikee deleted the fix/4455-maestro-version-probe-diagnostics branch November 13, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conformance Differential: retain Maestro version-probe failures instead of only expected/actual mismatch

1 participant