docs: ADR 0002 (CI-attested verdict), the Slice 4a plan, and a Spec Kit scoreboard correction#105
Conversation
…RPs rename Spec Kit ships a first-party workflow engine our fixture never reached. At the pinned commit 3f7392a, src/specify_cli/workflows/base.py:28 defines a RunStatus enum (created/running/paused/completed/failed/aborted) that engine.py:484 atomically persists to .specify/workflows/runs/RUN_ID/state.json alongside an append-only log.jsonl. A machine reading that file can tell a failed run from a completed one. That falsifies the generalization this board drew in two places: the Spec Kit row's "no machine-readable done flag", and the field-summary's claim that "failed, and here is the machine-readable reason" is never expressible. Both are corrected in place and dated, with the narrower finding stated instead: what state.json gives you is a status a run writes about itself, not evidence a third party can re-check that the completed was earned. The correction is also a live demonstration of the method's limit — a fixture built from documentation cannot see machinery the documentation does not surface. The row now says so and invites the same treatment for any other row. Also refreshes star counts (608,137 combined, was 599,034 on 2026-07-09) and notes that Wirasm/PRPs-agentic-eng now resolves to Wirasm/prp.
Discharges the decision ADR 0001 deferred in its Non-goals ("Those require
separate decisions"). ADR 0001 remains Accepted and unamended; its
non-equivalence statement is restated here as a standing limit.
The seam: the kernel disposes on contents, the CI lane notarizes context,
neither is the other. The kernel may hash. It must never sign, never establish
authenticity, and never read an environment variable.
Decisions:
- loop verdict emits a loop-engineer/verdict@1 PREDICATE BODY, not an in-toto
Statement. actions/attest builds the Statement from subject + predicate-type
+ predicate; handing it a full Statement would nest a Statement inside a
predicate and write a malformed attestation to a permanent public log.
- Subject is the chain head alone. The source commit stays out of the predicate:
the kernel reads no environment today, and the Fulcio cert already binds the
source repository digest more strongly than the kernel could assert it.
- predicateType is urn:loop-engineer:verdict:1 — no vendor host, no org, no repo
name, because the URI is immutable in a public log and a rename must not be
able to orphan it.
- Authenticity (gh attestation verify) and semantic agreement (loop verdict
--compare) are separate checks; --compare reports signature_checked: false on
every path and no flag may flip it.
- Ships as two slices: 4a emission, 4b consumption. 4b changes what the hard
gate accepts and earns its own review.
- Human review required on loop/**, action.yml, and .github/workflows/**.
Rejected: SLSA VSA (its verificationResult is a PASSED/FAILED bit and
verifiedLevels is required — it would collapse seven typed terminals and imply
a build-level evaluation we did not perform), in-toto SVR (pre-1.0, stringly
typed properties), in-toto test-result, in-kernel cryptography, local DSSE/HMAC,
attesting on pull requests, and a multi-subject Statement.
Standing limits are stated plainly rather than footnoted, including the one the
adversarial review corrected: opt-in anchor auto-resolution detects an
unattested rewrite for exactly one run. An actor who can land a commit lets CI
mint a genuine attestation over the rewritten chain and rolls the anchor
forward. Opt-in changes consent, not that property.
The Slice 4a plan lands alongside: nine TDD tasks covering the schema, the
projection, the verified-evidence digest set, CLI wiring, the boundary made
mechanical, the action step, the repo's own attestation job, normative §23, and
the governance change. Five open items are flagged for resolution during
execution rather than guessed here.
Baseline measured for the plan at HEAD 0025acc in a fresh worktree:
1305 passed / 18 skipped with pyyaml+jsonschema+pytest.
There was a problem hiding this comment.
Pull request overview
This PR updates documentation to (1) correct the harness scoreboard’s Spec Kit fairness note and star counts, (2) add ADR 0002 defining the CI-attested verdict boundary, and (3) add the Slice 4a implementation plan for verdict@1 emission.
Changes:
- Added a detailed Slice 4a implementation plan for emitting a
loop-engineer/verdict@1predicate and optionally attesting it in CI. - Corrected the scoreboard’s Spec Kit row narrative and updated star counts / repo link rename.
- Added ADR 0002 documenting the architectural boundary: kernel emits predicate only; CI lane attests.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| docs/superpowers/plans/2026-07-25-slice4a-verdict-emission.md | New Slice 4a plan describing schema/module/CLI/action/workflow/tests and constraints for verdict emission + attestation. |
| docs/gap-reports/scoreboard.md | Scoreboard corrections (Spec Kit fairness note), star count refresh, and PRPs repo rename reference. |
| docs/adr/0002-ci-attested-verdict.md | New ADR establishing the CI-attested verdict design and kernel/CI responsibility split. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "issue_codes": { | ||
| "type": "array", | ||
| "items": { "type": "string", "pattern": "^[a-z0-9_]{1,64}$" } | ||
| }, |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08d5df8c54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| path: examples/flaky-test-triage | ||
| attest: "true" |
There was a problem hiding this comment.
In the proposed attest.yml workflow, the selected examples/flaky-test-triage fixture has no .loop/events.db; I checked the tracked fixture tree, and loop doctor consequently reports event_store.present: false. The action's empty-head guard therefore skips actions/attest, while the workflow remains green and merely records an empty attestation URL, so the repository's advertised post-merge attestation job never creates an attestation. Add a tracked chained fixture (or construct one in the job) and assert that the attestation output is non-empty.
Useful? React with 👍 / 👎.
| store = report.get("event_store") or {} | ||
| chain = store.get("chain") or {} | ||
| head = chain.get("head") or {} |
There was a problem hiding this comment.
For a finished workspace without .loop/events.db, doctor_report() returns event_store: {"present": false} without failing, and this fallback converts that result into an empty store before returning a successful predicate with a fabricated workspace-name run_id and null chain head. That contradicts the accepted ADR's explicit requirement that loop verdict refuse this case with a typed error and permits emitting a verdict that cannot be tied to any chain. Check present/readability and raise VerdictError before constructing the document.
Useful? React with 👍 / 👎.
|
|
||
| - [ ] **Step 3: Wire the CLI** | ||
|
|
||
| In `loop/__main__.py`, add `"verdict"` to both `_COMMANDS` (line 15) and `_READ_COMMANDS` (line 19), add it to the `_USAGE` alternation (line 21), and add a `_HELP` usage line beside `metrics` plus a command description: |
There was a problem hiding this comment.
The CLI instructions add verdict to _COMMANDS and _READ_COMMANDS but omit the mode-parsing command set at loop/__main__.py:244. Consequently, for the documented loop verdict --mode strict <workspace> form, --mode remains in argv, becomes Path("--mode"), and the command exits with a nonexistent-target error instead of running in strict mode. Include verdict in that parsing set and cover the flag-bearing invocation in the CLI tests.
Useful? React with 👍 / 👎.
| if [ -z "${ACTIONS_ID_TOKEN_REQUEST_URL:-}" ]; then | ||
| echo "::error::attest: true requires the calling job to declare" \ | ||
| "'permissions: { id-token: write, attestations: write }'." \ | ||
| "A composite action cannot declare its own permissions." |
There was a problem hiding this comment.
When the calling job grants id-token: write but omits attestations: write, ACTIONS_ID_TOKEN_REQUEST_URL is still populated, so this check passes and actions/attest surfaces the raw authorization failure that the ADR explicitly requires the composite action to replace with a legible message. Add a permission-aware preflight for the attestation write operation, or stop claiming this condition validates both permissions.
Useful? React with 👍 / 👎.
Docs only. No code, no schema, no version bump. Three things, in descending urgency.
1. The scoreboard correction (launch-gating)
Spec Kit can express a typed failed run, and the board said it couldn't.
At our pinned commit
3f7392a,src/specify_cli/workflows/base.py:28defines:workflows/engine.pyatomically persists it to.specify/workflows/runs/<run_id>/state.json(line 484) alongside an append-onlylog.jsonl(line 560), transitioning toFAILEDon step failure andPAUSEDoninterruption. A machine reading that file can tell a failed run from a completed
one. Verified against the GitHub API at the pinned SHA, not against
main.That falsifies the generalization in two places:
machine-readable reason' never is"
Both corrected in place and dated. The narrower finding is what survives: what
state.jsongives you is a status a run writes about itself, not evidence athird party can re-check that the
completedwas earned. That distinction isthe whole subject of the board, and it's worth stating precisely instead of
loudly.
The row also now says the quiet part: a fixture built from documentation cannot
see machinery the documentation does not surface. This is that failure mode,
caught on our own board, and the row invites the same treatment for any other.
The
0/7column is unchanged and still true — it measures the contract's sevencanonical terminals landing on disk in a portable form, which is a defined and
checkable count, not "can this harness say the word failed".
Also in this commit: star counts refreshed (608,137 combined, was 599,034 on
2026-07-09) and
Wirasm/PRPs-agentic-eng→Wirasm/prp. All nine reposre-verified live: none archived, all pinned SHAs still resolve.
Why this is urgent: the Show HN draft publicly commits to this fairness note
existing. Posting before this lands means a commenter can disprove the post with
one grep of the linked artifact.
2. ADR 0002 — CI-attested verdict
Discharges the decision ADR 0001 deferred ("Those require separate
decisions"). ADR 0001 remains Accepted and unamended — a non-goal that
explicitly defers to a later decision is discharged by making it, not reversed
by it. The three slice plans whose program table says "reverses ADR 0001
non-goal" need the same correction; not done here to keep this PR docs-scoped.
The governing seam: the kernel disposes on contents; the CI lane notarizes
context. The kernel may hash — never sign, never establish authenticity, never
read an environment variable.
Four decisions came from the operator; three came from an adversarial review
that returned FLAWED on the first design and was right to:
actions/attestbuildsthe in-toto Statement itself. Passing it a complete Statement as
predicate-pathnests a Statement inside a predicate — a malformedattestation, written permanently to Rekor. This also turns out to be the
cleaner purity line.
grep -rn "environ\|getenv" --include=*.py loop/returns zero matches today. ReadingGITHUB_SHAwouldmake
verdict.pythe first environment-reading module in the kernel and put avendor identifier in the portable layer. The Fulcio certificate already binds
the source repository digest more strongly than we could assert it.
An actor who can land a commit lets CI run once, which mints a genuine
attestation over the rewritten chain, and the next run resolves forward to it.
The first draft claimed opt-in avoided a self-satisfied gate; it doesn't —
opt-in changes consent, not the property. Stated plainly in Standing Limits.
Rejected with reasons: SLSA VSA, in-toto SVR, in-toto test-result, in-kernel
cryptography, local DSSE/HMAC, attesting on PRs, multi-subject Statements, and
both a
github.comand an owned-domain predicateType URI.Five open verification items are listed rather than guessed — notably that the
chain head is a SHA-256 over a synthesized event preimage, not over retrievable
bytes, so the
sha256DigestSet key licenses an inference that is false here.3. Slice 4a implementation plan
Nine TDD tasks: schema + module, the doctor/chain/terminal projection, the
verified-evidence digest set, CLI wiring, the boundary made mechanical, the
action step, the repo's own attestation job, normative §23, and the governance
change. Emission only —
--compare, anchor resolution, and signer-trust policyare 4b and the plan says so in an explicit out-of-scope section.
Baseline measured for the plan at HEAD
0025accin a fresh worktree:1305 passed / 18 skipped with pyyaml+jsonschema+pytest. The plan pins the
measurement trap that cost a run:
uv run --project <worktree>installs thewheel, materializes
loop/_bundle/, and makestest_resources.py::test_repo_checkout_resolves_to_repo_dirsfail correctly.Verification
scripts/self_eval.py→ 13/13, soreference_filenamesis still 8 and §23will be an append, not a new file
resolution
Operator actions this PR does not perform
main protectioncurrentlyrequires a PR with 0 approvals, so the
CODEOWNERSfile Slice 4a adds wouldenforce nothing on its own. Requires "Require review from Code Owners" plus 1
required approval, set in Settings → Rules. Until then decision 6 is documented
but not in force.
roadmap/launch/;its remaining checklist items are re-running the try-it block immediately
before posting, and the timing call.