Repository navigation
ci(conformance): keep the Maestro version probe's failure evidence - #4503
Conversation
…4455) The layer-3 pin check parsed `maestro --version` inside a pipeline that sent stderr to /dev/null and masked the pipeline exit with `|| true`. Its whole failure record was `expected=<pin> actual=`, so the November 9 run could not tell a CLI that never started from one that printed nothing parseable. Run the probe through scripts/maestro-conformance/verify-cli-version.mjs. It records the exit status, signal, spawn error and both streams, retains each stream's last 8 KiB newest-first, and reports one verdict: command failure, missing or unparseable version, mismatch, or match. The pin invariant is unchanged and still fails closed; a probe that did not complete cleanly is never accepted on the strength of its stdout. The fixture-executable suite covers a matching version inside notices, a wrong version, empty and unparseable stdout, a nonzero exit with matching-looking stdout, an absent executable, and bounded oversized output.
Size Report
Startup median (7 runs, lower is better):
|
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Apex Sentinel · ✅ ApproveSince the last review: 0 new · 1 still open · 1 no longer raised 1 finding(s): 1 minor. 🟡 Minor (1)
Reviewed with this repository's own Sentinel configuration ( 💬 Talking to SentinelIn a finding's thread, reply with:
On this conversation, comment Earlier reviewsReview of
|
| Where | Finding |
|---|---|
scripts/maestro-conformance/verify-cli-version.mjs:57 |
Probe has no deadline; a hung maestro --version stalls the 90-min job |
scripts/maestro-conformance/verify-cli-version.test.mjs:24 |
Scratch dirs leak when the suite runs outside scripts/node-test-tmpdir.ts |
Reviewed with this repository's own Sentinel configuration (efe6bb9eece4).
There was a problem hiding this comment.
Apex Sentinel: 2 new findings on bae08d4. The verdict and the full list are in the summary comment.
|
No code blockers found at 94f4274: version matching reads the full stdout stream independently of the retained diagnostic tail, fixing the earlier version-truncation finding. CI has 24 successful checks and two skipped checks; the scheduled Conformance Differential itself was not run. There are no conflicts. |
Summary
The Maestro pin check hid stderr and masked CLI failures. The workflow now probes the CLI through one script that retains exit/signal/spawn diagnostics and each stream’s last 8 KiB. Only a successful CLI reporting the pinned version passes; no retry, fallback or pin change.
Version matching scans the full stdout stream independently of diagnostic retention. This preserves versions before long trailing notices and prevents truncation from turning
12.5.1into the pinned2.5.1. Matching state and diagnostics remain bounded.5 touched files. No scope expansion. Closes #4455.
Validation
Head:
94f427458ddb05f56f8bcee4f387e03aad34a989.efe6bb9: nonzero CLI exit with matching stdout was accepted. Worker headbae08d4rejects it and retains stderr.bae08d4, truncating12.5.1falsely passed; a valid pin before oversized trailing notices falsely failed. Both controls pass at this head. Oversized numeric versions also remain bounded and fail closed.8f42d7failed scanner complexity; replaced with explicit state transitions. Current-head rerun was cancelled while waiting for live host-lock owner78515; no checks executed. CI runs the Fallow and conformance gates. Pre-existing untracked website output was preserved.