Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4c62031
test(codeql): reproduce rerun dead-end after pre-runner cancellation
seonghobae Sep 5, 2026
fabc998
fix(codeql): recover reruns with no dispatch verdict
seonghobae Sep 5, 2026
e47de34
docs(codeql): trace pre-runner rerun recovery evidence
seonghobae Sep 5, 2026
db15e4c
chore(codeql): restack rerun recovery on protected main
seonghobae Sep 5, 2026
9691960
chore(codeql): adopt latest verification guidance without force restack
seonghobae Sep 5, 2026
67a6bc9
docs(codeql): align ADR with evidence-driven rerun recovery
seonghobae Sep 5, 2026
510e337
merge(main): adopt concurrent CI governance deltas
seonghobae Sep 5, 2026
0599331
chore(codeql): adopt latest protected main
seonghobae Sep 5, 2026
2d4624a
chore(codeql): adopt protected main into rerun recovery
seonghobae Sep 5, 2026
e72ae30
chore(codeql): adopt current protected main
seonghobae Sep 6, 2026
acfa17e
test(codeql): require complete status pagination before redispatch
seonghobae Sep 6, 2026
7628274
fix(codeql): inspect complete status history before redispatch
seonghobae Sep 6, 2026
622ea74
docs(codeql): record complete status-history recovery guard
seonghobae Sep 6, 2026
bf732f9
test(codeql): exercise paginated empty-verdict recovery
seonghobae Sep 6, 2026
eb9d4d2
chore(codeql): merge current main into PR 1902
seonghobae Sep 6, 2026
4bf80b9
test(codeql): align verdict fixtures with paginated status reads
seonghobae Sep 6, 2026
951d0ec
fix(scheduler): require explicit open live PR identity
seonghobae Sep 6, 2026
9bfe575
fix(scheduler): bind Strix reruns to verified job identity
seonghobae Sep 6, 2026
fe64f24
test(ci): cover remaining Strix rerun identity rejections
seonghobae Sep 6, 2026
3549de5
chore(codeql): isolate rerun recovery on current protected main
seonghobae Sep 6, 2026
82ca0b8
fix(codeql): require preserved SARIF before status
seonghobae Sep 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions .github/workflows/codeql-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@
# codeql-scan-dispatch.yml (which runs natively, unrestricted, in
# ContextualWisdomLab/.github). The shard then fails intentionally to release
# its runner; the handler publishes codeql-dispatch/<language> and reruns only
# that exact failed job. On rerun the shard reads the terminal status once.
# that exact failed job. On rerun the shard consumes an authenticated terminal
# status when one exists. If earlier attempts never reached dispatch and no
# authenticated verdict exists, the rerun may dispatch the same exact shard;
# the central target/PR/language concurrency lane bounds duplicate recovery.
# Design:
# docs/adr/0025-codeql-required-workflow-dispatch-architecture.md. The
# merge-preview scan (analyze-merge) is required nowhere (PR #1766) and was
Expand Down Expand Up @@ -158,9 +161,11 @@ jobs:
steps:
- name: Request current-head CodeQL scan dispatch
# Each shard dispatches only its own language and passes its exact
# run/job identity. The shard intentionally fails after dispatch so
# its runner is released; the trusted handler later reruns that one
# failed job after publishing a terminal current-head verdict.
# run, job, language, and head identity. The shard intentionally fails
# after dispatch so its runner is released; the trusted handler later
# reruns that exact failed job after publishing a terminal verdict.
# Rerun attempt count is not a dispatch receipt: earlier attempts can
# be cancelled before runner assignment and execute zero steps.
id: dispatch
if: needs.detect-languages.outputs.code == 'true'
env:
Expand Down Expand Up @@ -196,10 +201,10 @@ jobs:
exit 0
fi

statuses="$(gh api "repos/${TARGET_REPOSITORY}/commits/${PR_HEAD_SHA}/statuses")"
statuses="$(gh api --paginate --slurp "repos/${TARGET_REPOSITORY}/commits/${PR_HEAD_SHA}/statuses?per_page=100")"
verdict_state="$(printf '%s' "$statuses" | jq -r --arg ctx "codeql-dispatch/${LANGUAGE}" '
[
.[]
.[][]
| select(.context == $ctx)
| select(
(.creator.login // "" | ascii_downcase) as $creator
Expand All @@ -215,10 +220,6 @@ jobs:
exit 0
;;
esac
if [ "$RUN_ATTEMPT" != "1" ]; then
echo "::error::Exact CodeQL job was rerun without an authenticated terminal verdict."
exit 1
fi
if ! [[ "$REQUIRED_RUN_ID" =~ ^[1-9][0-9]*$ ]] ||
! [[ "$REQUIRED_JOB_ID" =~ ^[1-9][0-9]*$ ]]; then
echo "::error::CodeQL dispatch requires canonical current run and job ids."
Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/codeql-scan-dispatch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -397,11 +397,13 @@ jobs:
run: python3 "$RUNNER_TEMP/codeql_sarif_gate.py" codeql-results-dispatch

- name: Preserve CodeQL SARIF evidence
id: sarif_upload
if: always() && hashFiles('codeql-results-dispatch/**/*.sarif') != ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: codeql-dispatch-${{ matrix.language }}-${{ github.run_id }}-${{ github.run_attempt }}
path: codeql-results-dispatch
if-no-files-found: error
retention-days: 7

- name: Publish CodeQL dispatch status
Expand All @@ -416,8 +418,13 @@ jobs:
HEAD_SHA: ${{ needs.validate-dispatch.outputs.head_sha }}
LANGUAGE: ${{ matrix.language }}
GATE_OUTCOME: ${{ steps.gate.outcome }}
SARIF_UPLOAD_OUTCOME: ${{ steps.sarif_upload.outcome }}
run: |
set -euo pipefail
if [ "${SARIF_UPLOAD_OUTCOME:-}" != "success" ]; then
echo "::error::CodeQL SARIF evidence was not preserved; terminal status publication and exact-job wake are blocked."
exit 1
fi
case "$GATE_OUTCOME" in
success)
state="success"
Expand Down
67 changes: 58 additions & 9 deletions docs/adr/0025-codeql-required-workflow-dispatch-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,14 +96,18 @@ codeql-pr.yml (required workflow, runs in target repo context)
does) before dispatching.
analyze-head (matrix) -- SAME REQUIRED-CHECK NAME:
"CodeQL compatibility analysis (${{ matrix.language }})".
No codeql-action reference. On attempt one it
dispatches its exact run id, job id, language,
and head, then fails intentionally to release
the runner. The trusted handler publishes the
terminal status and reruns only that failed
job. On attempt two the shard reads the
authenticated current-head status once and
reflects it as this job's own exit code.
No codeql-action reference. Each invocation
first consumes a trusted terminal
codeql-dispatch/<language> status for the exact
current head when one exists. Otherwise it
validates the exact required run/job/language
identity, dispatches that one shard, records
verdict=pending, and fails intentionally to
release the runner. The trusted handler later
publishes the terminal status and reruns only
that failed job. A later run_attempt is not
treated as proof that any earlier attempt
reached the dispatch step.

.github/workflows/codeql-scan-dispatch.yml (NEW, runs natively in .github,
NOT admitted through the ruleset, so codeql-action is unrestricted here)
Expand Down Expand Up @@ -170,6 +174,36 @@ dispatch was rejected because the handler validates one shard and wakes one
exact required job per run; changing that contract would enlarge the security
and recovery surface without solving another observed need.

### Rerun recovery is evidence-driven, not attempt-driven

`github.run_attempt` is execution metadata. It is not an authenticated receipt
that a previous attempt reached `Request current-head CodeQL scan dispatch`.
The concrete counterexample is
`ContextualWisdomLab/accounting-information-platform#49@065f9ab7038bf35db4ef129827de6ab8ee6a1038`,
required CodeQL run `33890965185`: attempts 1 and 2 were cancelled before
runner assignment (`runner_id=0`, `steps=[]`). Attempt 3 finally ran, found no
trusted terminal `codeql-dispatch/actions` or `codeql-dispatch/python` status,
and the former `RUN_ATTEMPT != 1` guard rejected both shards before dispatch.
The unchanged consumer head was therefore unable to recover after capacity
returned.

The required workflow must instead use authenticated evidence. For the exact
live PR head and language shard, a terminal status created by the expected
central identity is consumed. If no such terminal verdict exists, the shard
re-validates its run/job/head identity and may dispatch again regardless of the
numeric attempt. The central target/repository/PR/language concurrency key
bounds duplicate recovery; the handler independently re-validates live PR and
wake identity before it publishes a verdict or reruns the exact job. Missing
evidence remains fail closed: redispatch produces `verdict=pending`, never a
synthetic success.

A manually requested rerun can arrive while an earlier native dispatch is still
queued but has not published a terminal status. In that case the existing
concurrency lane may replace work for the same exact logical shard. This is a
bounded restart risk, not a reason to restore attempt-number inference. If
observed churn becomes material, the successor design must add an authenticated
pending/dispatch-receipt state keyed to the same exact identity.

## Scope decision: `analyze-merge` is dropped, not migrated

`analyze-merge` ("CodeQL merge preview") is confirmed, per PR #1766's own
Expand Down Expand Up @@ -230,6 +264,10 @@ blocker for this one.
documented, evidently deliberate platform limitation
("CodeQL requires configuration at the repository level"), not a bug
report candidate.
- **Use run-attempt number as a dispatch receipt:** rejected after the AIP #49
reproduction. Earlier attempts can be cancelled before any step executes,
so an attempt number cannot prove that a native scan was requested. Only
authenticated exact-head status/receipt evidence may suppress redispatch.

## Risks and effects

Expand All @@ -250,6 +288,10 @@ blocker for this one.
assert zero matches, as a permanent contract test) — re-adding it with
the bug still present would recreate the exact org-wide 100%-startup_failure
incident this ADR exists to prevent.
- Rerun recovery can replace an already queued same-shard native dispatch when
no terminal status exists yet. The concurrency key keeps that restart within
the exact repository/PR/language identity. If this causes material churn,
add an authenticated pending receipt rather than trusting run-attempt order.

## Follow-up

Expand All @@ -261,7 +303,14 @@ blocker for this one.
ADR is the record).
4. Add a permanent contract test asserting no `codeql-action` reference
exists anywhere in `codeql-pr.yml`.
5. Only then, re-add `.github/workflows/codeql-pr.yml` to ruleset `18156473`'s
5. Keep the rerun-recovery regression that executes the production dispatch
shell with a later `run_attempt`, no trusted terminal verdict, and exact
run/job/head identity; it must reach bounded dispatch with `verdict=pending`.
6. After protected integration, rerun the unchanged AIP #49 head and require
real authenticated terminal `codeql-dispatch/actions` and
`codeql-dispatch/python` verdicts before treating the owner repair as
effective for that consumer.
7. Only then, re-add `.github/workflows/codeql-pr.yml` to ruleset `18156473`'s
required `workflows` list (admin:org PUT, same mechanism used to remove
it) and verify a real PR observes a successful, correctly-named required
check before declaring this ADR's status Accepted.
74 changes: 74 additions & 0 deletions docs/doctoring/codeql-rerun-pre-runner-cancellation-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# CodeQL rerun recovery after pre-runner cancellation

## Problem and exact evidence

The required `CodeQL PR` workflow used `github.run_attempt != 1` as if it proved that an earlier attempt had successfully dispatched the native CodeQL scan. That inference is false when an earlier attempt is cancelled before runner assignment.

`ContextualWisdomLab/accounting-information-platform` PR #49 provides the concrete reproduction on exact head `065f9ab7038bf35db4ef129827de6ab8ee6a1038`, workflow run `33890965185`.

- Attempt 1 `Detect CodeQL languages` job `101082241642` ended `cancelled` with `runner_id=0` and `steps=[]`; its downstream compatibility job was also cancelled without execution.
- Attempt 2 `Detect CodeQL languages` job `101128192785` ended the same way: `cancelled`, `runner_id=0`, `steps=[]`; the downstream compatibility job again never executed.
- Attempt 3 finally obtained runners. The `actions` shard job `101220582725` and `python` shard job `101220582747` reached `Request current-head CodeQL scan dispatch`, found no authenticated `codeql-dispatch/<language>` terminal status, then failed solely because `RUN_ATTEMPT=3`.
- The target exact head had no `codeql-dispatch/actions` or `codeql-dispatch/python` commit status. Thus the attempt number did not identify a prior dispatch receipt or a terminal scan verdict.

This leaves an unchanged PR head permanently unable to obtain the required CodeQL result even after runner capacity recovers.

## Chosen repair

Keep the existing trust sequence:

1. re-read the live pull request and reject closed or moved heads;
2. read only `codeql-dispatch/<language>` statuses created by the expected `opencode-agent` identity;
3. if an authenticated terminal status exists, reflect it without dispatching;
4. otherwise validate the exact required run/job identity, obtain the OIDC-bound app token, and dispatch the exact repository/PR/head/language shard.

Remove the `RUN_ATTEMPT != 1` veto. A rerun attempt number is execution metadata, not evidence that the dispatch step ever ran. The native handler already serializes the same target-repository / pull-request / language tuple and re-validates live PR and wake identity before publishing a verdict or rerunning the exact required job.

This does not convert a missing CodeQL verdict to success. The required shard still fails with `verdict=pending` after dispatch and becomes successful only when the trusted handler publishes an authenticated terminal `success` status and reruns the exact job. A forged status, stale head, failed/error verdict, unavailable OIDC/app token, malformed run/job identity, or absent dispatch receipt remains fail closed.

## Follow-up review: complete status-history authority

Current-head review on `e72ae30e3e989396b8cfdd1d850f7db1f45c6a7e` found a second defect in the same evidence boundary. `GET /commits/{sha}/statuses` was read without pagination. Treating an empty default response page as proof that no authenticated terminal `codeql-dispatch/<language>` verdict exists is unsafe on a commit with enough status history to push an older trusted verdict to a later page. The recovery path could then redispatch even though terminal authority already existed.

The rejected alternatives are increasing an assumed first-page size without pagination, trusting the combined commit-status summary, or restoring `RUN_ATTEMPT` inference. None proves absence of the exact creator-bound language status across the complete history.

RED `acfa17e84f1ef6a0da5b93c642fcdf0d67d1d814` extends the focused contract to require a paginated, slurped status lookup and page-flattening before absence can authorize redispatch. Minimal repair `7628274f3e146e32fba124fe3e21e1fef8b107b3` changes only that read boundary: `gh api --paginate --slurp .../statuses?per_page=100` collects every page, and the existing trusted-context/creator filter runs across `.[][]`. Live PR/head validation, OIDC/app-token exchange, exact run/job/language binding, pending fail-closed behavior, handler validation and concurrency are unchanged.

The security effect is narrower than “more reliable pagination”: **absence is now established over the complete status population before dispatch authority is exercised**. An authenticated terminal status on any page therefore prevents a redundant redispatch. If GitHub changes the status API representation, the focused regression must fail rather than silently fall back to first-page semantics.

## Executable regression

`tests/test_codeql_pr_rerun_recovery_contract.py` executes the production `Request current-head CodeQL scan dispatch` Bash block with:

- `RUN_ATTEMPT=3`;
- the same live target head;
- no authenticated CodeQL status;
- mocked OIDC and app-token exchange boundaries; and
- an exact run/job/language wake identity matching the accounting-platform reproduction.

The test requires the step to publish `verdict=pending` and to emit a `codeql-scan` repository-dispatch payload bound to `ContextualWisdomLab/accounting-information-platform`, PR #49, the exact head, run `33890965185`, job `101220582747`, and `python`. The companion status-history contract requires `--paginate --slurp`, an explicit `per_page=100`, and page flattening before the trusted verdict filter.

Before the production change, the original regression exits at the attempt-number guard before OIDC or dispatch. Before the pagination repair, the status-history contract fails because the production read asks only for the default first page. After both repairs, the same shell block reaches the bounded dispatch path only when the complete authenticated status history contains no terminal verdict.

## Risks, rollback, and acceptance

A manually requested rerun while a prior native dispatch is still queued but has not yet published a terminal status may replace work in the existing central target/PR/language concurrency lane. This is bounded to the same exact logical shard and does not broaden repository, head, language, credential, or merge authority. If live evidence shows harmful restart churn, the successor design should add an authenticated dispatch-receipt/pending state rather than restoring attempt-number inference.

Pagination adds API reads proportional to commit-status history, bounded at 100 statuses per page. That cost is accepted because a false “verdict absent” decision authorizes external dispatch; status absence therefore requires complete evidence rather than a first-page heuristic.

Rollback is not `RUN_ATTEMPT != 1` and not a non-paginated status read; either recreates a proven dead end or an incomplete-authority check. A valid replacement must distinguish “prior dispatch accepted” from “prior attempt never executed” using authenticated complete-history evidence and retain exact-head fail-closed semantics.

GREEN requires all of the following on one unchanged successor head:

- the focused rerun-recovery and complete-status-history regressions pass;
- the existing `test_codeql_pr_workflow_contract.py` suite remains green;
- the complete central test, 100% coverage, docstring, workflow syntax, security and review gates pass;
- after protected integration, the unchanged accounting-platform PR #49 head is rerun and obtains a real authenticated terminal CodeQL verdict without provider/model or leaf-repository workaround.

## References

GitHub. (2026). *Re-running workflows and jobs*. GitHub Docs. https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs

GitHub. (2026). *REST API endpoints for workflow runs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-runs

GitHub. (2026). *REST API endpoints for commit statuses*. GitHub Docs. https://docs.github.com/en/rest/commits/statuses
9 changes: 9 additions & 0 deletions docs/doctoring/codeql-sarif-publication-boundary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# CodeQL SARIF publication boundary

The central CodeQL dispatch handler publishes a terminal commit status only after the same matrix shard has successfully preserved its SARIF artifact. A successful finding gate without durable evidence is not a successful scan contract: upload failure, a skipped upload, cancellation, or a missing outcome fails closed before any status credential is used and therefore before the exact required job can be woken.

`actions/upload-artifact` owns the evidence boundary. The upload step has a stable step identifier and rejects an empty artifact input. The status-publication step consumes that step's outcome and accepts only `success`; it does not infer preservation from a generated local file or from the SARIF gate result. The gate result continues to determine whether preserved evidence represents a passing or failing security verdict.

Executable regression coverage runs the real publication shell against a fixture-backed GitHub API. The success control permits one exact-head status post. Upload outcomes `failure`, `skipped`, `cancelled`, and empty each exit before a post, preventing a false terminal success and the downstream exact-job rerun.

This source repair does not change repository-dispatch actor authorization or cross-repository credential authority. Those remain separate configuration and GitHub App permission boundaries tracked in ContextualWisdomLab/.github issue #1929.
Loading
Loading