Skip to content

feat(scripts): end every result run with a verdict and sharpen remedies - #324

Merged
possiblyneal merged 2 commits into
mainfrom
possiblyneal/script-output-verdict
Nov 18, 2026
Merged

possiblyneal merged 2 commits into
mainfrom
possiblyneal/script-output-verdict

Conversation

@possiblyneal

@possiblyneal possiblyneal commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Intent

From an operator-requested review of past session transcripts: 5,231 script runs and hook firings, checked to see whether the scripts work and where they could be widened.

The scripts' verdicts were sound, but agents misread them. They cut output short with tail or grep, which lost the exit status and the unavailable rows. Several remedies also pointed the wrong way. After this change:

  • every Result-table command ends with <cmd> exited <status>: <count per state>;
  • a Generated-By trailer must be shaped as a model ID;
  • adr-index run by hand does not break an && chain;
  • doctor's remedy works under a global core.hooksPath;
  • payload-parity prints the mirroring cp, or git rm for a deletion, when only one copy of a pair changed;
  • the Go audit names the release that clears a failure found only in the local standard library;
  • scripts/clean moves its "Add project-specific cleanup" hint from every run's output into a comment, and scripts/detect's usage says probe lists which checks exist.

Read first

  • scripts/libs/result.sh:99: result_verdict, and result_exit_verdict with its guarded EXIT trap. Every Result-table command inherits it; summarize calls result_verdict for the command it ran.
  • scripts/attribute-commit:163: the --check branch, wired as the attribution-check commit-msg hook in both .pre-commit-config.yaml copies.
  • scripts/libs/detect.sh:677: _govulncheck_module and _govulncheck_stdlib_note.
  • scripts/payload-parity:88: staged_edit and mirror_command, which prints git rm when the edited copy is a staged deletion.
  • scripts/doctor:95: the remedy under core.hooksPath.
  • Mechanical: every apps/github-repository-template/src/base-repo/ file under scripts/ and .github/ is a byte copy of the root's (payload-parity passes). apps/repo-builder/src/scripts/retrofit.py's hook probe accepts any Generated-By once the Co-Authored-By is gone.

Decisions

  • The verdict comes from an EXIT trap in result.sh, not from tally. tally is also called mid-script (for example in security before the audit), so it cannot be the last line. The trap is installed only when no EXIT trap exists, so it never clobbers a script's or the test harness's cleanup. The suites print their own totals instead.
  • --check accepts any vendor's ID shape (^[A-Za-z0-9][A-Za-z0-9._:@/-]*$), not a list of Claude models. A list would refuse the next model, and the shape admits qwen3-coder:30b, ...@20250805 and provider paths while refusing spaces and [1m]. It checks shape, not that the model exists.
  • A missing trailer is left to commitlint's trailer-exists, which skips merge, revert and fixup subjects on its default ignore list; --check refusing it too broke a person's local git merge.
  • With no model known, the Co-Authored-By fallback slugs the display name rather than dropping it, so a commit made without the commit-model hook still lands, with a searchable trailer.
  • The Go audit still fails on stdlib-only findings. The local binary really carries them, so the note explains the failure rather than excusing it.
  • No commitlint ignores for Merge #N: title. commitlint's ignores are functions, which YAML cannot hold, and a JS config would bring back the package.json the YAML exists to avoid. The default-ignored Merge <what> into <branch> shape is documented and tested instead.
  • SKIP=no-commit-to-branch and shellcheck -x are unchanged. scripts/check already sets the skip. Measured, -x costs nothing: 53 files took 16.43s serially without it and 16.45s with it. The real cost is about 0.8s per script that sources detect.sh, and parallelism already brings the hook to 8.5s.

Out of scope

  • Having doctor flag node_modules that is out of date with package-lock.json, from the same review: not requested.
  • Hiding passing rows unless --verbose is given: not requested.

Verified

  • scripts/check → scripts/check exited 0: 0 FAIL, 0 unavailable, 5 not-applicable, 26 pass
  • scripts/tests/capabilities-test → 81 passed, with new cases for a stdlib-only Go failure (note printed) and a module failure (no note).
  • scripts/tests/commitlint-test → 7 passed; Merge feature/x into main is accepted and Merge #12: title refused.
  • scripts/tests/attribute-commit-test → 33 passed (vendor IDs, a trailer-less merge, no end hyphen); scripts/tests/payload-parity-test → 22 (a deletion mirrors as git rm); scripts/tests/precommit-hooks-test → 10 (the hooksPath remedy now runs from a path with a space); scripts/tests/result-gate-test → 12; scripts/tests/summarize-test → 10; scripts/tests/adr-index-test → 17.
  • repo-builder -k Hooks → 11 passed.
  • A real govulncheck run under go1.24.4 printed fixed by go1.25.8. Rerunning with GOTOOLCHAIN=go1.25.8+auto exited 0 with No vulnerabilities found.
  • doctor's printed hooksPath remedy, run in a scratch clone, installed every hook, and a commit through them succeeded.

Not verified

  • BSD/macOS sed for the fallback's y/// lowercasing and the sort -V in the Go note. Both ran only on GNU coreutils.
  • CI itself: everything above ran locally.
  • The doctor remedy on macOS, and with a local, rather than global, core.hooksPath.

Merge danger

Door: two-way. A revert restores every script, and nothing is stored or migrated.

Blast radius:

  • Every commit in this repository and in generated repositories once they update: the new attribution-check refuses a Generated-By that is not a model ID. Hand-typed display names such as Claude Opus 5.5 stop passing.
  • Every script that prints a Result table: its last line is now the verdict.
  • scripts/summarize's parser.
  • repo-builder's retrofit hook probe.

neal added 2 commits November 18, 2026 15:52
A review of past sessions found agents piping check output through
tail or grep, losing the exit status and missing unavailable rows.
Every command printing a Result table now ends with one verdict line,
naming its exit status and a count per state, from libs/result.sh's
EXIT trap; summarize ends the same way.

The same review found remedies that misled. Generated-By trailers were
typed in a dozen spellings, so attribute-commit --check now refuses
one that is not a model ID at commit-msg, and the Co-Authored-By
fallback spells the display name as an ID. adr-index exits 0 when its
rewrite runs by hand, failing only under pre-commit. doctor's remedy
under a global core.hooksPath now installs the hooks, where pre-commit
install refuses. payload-parity prints the cp that mirrors a pair when
exactly one copy changed. The Go audit names the Go release that
clears a failure found only in the local standard library, which CI's
stable Go never reports.

commitlint's ignores are functions a YAML config cannot hold, so a
hand-written merge takes the default-ignored "Merge <what> into
<branch>" shape, now documented and tested.

Generated-By: claude-opus-5-5
attribute-commit --check refused a merge, revert or fixup commit with no
trailer, which commitlint's trailer-exists skips; a missing trailer is now
left to commitlint, and --check refuses only a malformed value. Its ID
shape admits other vendors' IDs (qwen3-coder:30b, ...@20250805, a provider
path, uppercase) and still refuses spaces and a [suffix]. The display-name
fallback trims a hyphen left at either end by stray spacing.

payload-parity mirrored a staged deletion with a cp from the deleted copy;
it now prints the git rm that mirrors it. doctor's hooksPath remedy quotes
the shim paths, so it runs from a clone whose path holds a space.

The verdict line is formatted once, by result_verdict in libs/result.sh,
which summarize now calls for the command it ran. The retrofit hook probe
accepts any Generated-By once the Co-Authored-By is gone, since a session
with a known model writes the model rather than the slug.

Generated-By: claude-opus-5-5
@possiblyneal possiblyneal added the code-reviewed Multi-axis code review passed label Nov 18, 2026
@possiblyneal

Copy link
Copy Markdown
Owner Author

Code review: standard scan, clean.

Axes: Standards+Spec, Correctness/Logic

Reviewed-Head: 18b1b06

@possiblyneal
possiblyneal merged commit 6ccb86b into main Nov 18, 2026
7 checks passed
@possiblyneal
possiblyneal deleted the possiblyneal/script-output-verdict branch November 18, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

code-reviewed Multi-axis code review passed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant