Scheduler pre-review update-branch discards in-flight checks under queue saturation: 76/77 merges since 09-04 had 0/12 required contexts at merge time

[seonghobae]

Summary

Under queue saturation, the scheduler's pre-review update_branch path discards in-flight checks. A scheduler run waits 2h+ for a runner, executes after main has moved, picks one behind PR from its scan — its own PR (#1926) or another one (a codex/originweave-mv3-owner run updated #1857) — and merges main into that head before any review is dispatched. Every completing run therefore has one near-random victim, and the scheduler's completion rate is the PR cancellation rate. The push cancels every queued check on the old head and puts the new head at the back of the queue. main moves roughly every 28 minutes (77 merges into this repository in ~36h), so no PR head survives long enough for its 12 required contexts to complete.

Measured on #1926 (2026-09-05):

item	value
scheduler run for the PR	33963177129, pull_request_target, created 11:22:53Z, completed 13:44:41Z (success)
bot push c730ae0b (Merge branch 'main' into …)	13:44:37Z
scheduler log line	PR #1926: update_branch: current head has no OpenCode approval; branch is outdated before review dispatch; branch update requested with OpenCode app token inside GitHub Actions as OpenCode GitHub App
check-runs on the replaced head 7920a70b	28 total: 22 cancelled within seconds of the push (queued since 11:22:53Z), 3 success, 3 skipped

Same mechanism observed on #1812 (c9244ceb, 13:31:29Z) and, after the #1926 merge, on #1484 (777ee3ed, 13:58:42Z: 21 of 30 check-runs on the old head cancelled at the push, 8 successes orphaned, new head starts from zero). The pushes are not a burst caused by the merge: they are a steady drip, one per PR scheduler run that finishes its queue wait, and with main moving every ~28 minutes almost every run finds its PR behind. The 5–12 bot-pushed branches seen in the 20 minutes after 6f8c51d7 are the scheduler runs that happened to complete in that window.

Consequence on merge governance

Reconstructing the check state at merged_at for every PR merged into .github since 2026-09-04 (77 PRs, REST commits/{sha}/check-runs, only runs completed before merged_at counted, skipped counted as satisfied):

required contexts satisfied at merge	PRs
0 / 12	76
7 / 12	1
12 / 12	0

Depth ≥ 2 jobs (opencode-review, coverage-evidence, trivy-fs, scorecard, osv-scan, dependency-review, both CodeQL compatibility analysis (...)) had no check-run at all before merge on 75–77 heads: their upstream gate job never got a runner. Every merge since 09-04 was therefore an admin bypass with the required review evidence absent. merged_by is populated on these, so it is not a bypass signal; the check state at merged_at is.

Where

• .github/workflows/pr-review-merge-scheduler.yml:139 — UPDATE_BRANCHES is true for pull_request_target runs, so every PR's own event run performs the update.
• scripts/ci/pr_review_merge_scheduler_core.py:4774-4785 — if behind_by and trigger_reviews: → update_branch; there is no check for required check-runs still queued/in progress on the current head.
• CLAUDE.md states the scheduler "updates a PR branch only when the latest review is approved, no current-head check has failed, and GitHub reports the PR as behind". The pre-review path above is a second, undocumented update path.

Proposed fix (smallest root-cause change)

1. In pr_review_merge_scheduler_core.py, before the pre-review update_branch, decide wait when the current head still has required check-runs in queued/in_progress — do not discard evidence in flight. One helper, one guard, one contract test.
2. Optionally, set UPDATE_BRANCHES to false for pull_request_target runs and leave the refresh to the schedule/push sweeps, which are rate-limited by BRANCH_UPDATE_LIMIT.
3. Correct the CLAUDE.md sentence to describe both update paths.

Related: #1931 (queue delay invalidates the dispatch side via base_sha drift — same family, other half), #1929/#1927 (dispatch actor), #1925 (CodeQL dispatch never succeeds).

Instrument notes for whoever re-measures: the pulls list endpoint omits merged_by; a check-run's started_at is null while queued; the closed event after merge creates fresh check-runs on the same head, so "latest check-run per name" reads post-merge state unless filtered by merged_at.

[seonghobae]

Rate measured after the #1926 merge (3f88e13a, 13:57:04Z) — peer session's readings at +0/+5/+10 min, .github only:

t	queued	bot pushes so far	runs created since merge	of which bot-triggered
+0 (13:57:11Z)	610 (+11)	0	17	0
+5 (14:02:06Z)	614 (+15)	1	44	15
+10 (14:07:10Z)	600 (+1)	6	90	60

Bot pushes arrived at 13:58:46, 14:03:11, 14:03:17, 14:05:18, 14:05:40, 14:06:31Z — roughly one every 100 s, spread evenly, not a burst at the merge. Each push spawns ~10 runs (60/6), so the drip injects ~6 runs/min while the queue's 10-minute net change is +1: cancellations leave at the same rate. The queue is a treadmill, which is why depth has oscillated between ~500 and ~600 all day without moving in either direction.

One scheduler completion ≈ one bot push (victim chosen by the run's scan, not necessarily its own PR: a codex/originweave-mv3-owner run updated #1857) ≈ ~10 new runs + every queued check on that PR cancelled. On #1484 the push also cancelled the PR's own queued scheduler run 6 s later.

Consequence for #1926's behavioural evidence (#1730's two CodeQL shards re-running): codeql-scan-dispatch.yml has to survive this treadmill first, so that evidence may not appear until the guard lands. Guard PR follows.

[seonghobae]

독립 검증 + 속도 실측 (host2 세션)

이 이슈의 기전을 peer1 세션이 찾았고, 아래는 제가 별도 검색어와 별도 사례로 확인한 것입니다. 인용이 아니라 실행입니다.

소스 (origin/main 3f88e13a)

scripts/ci/pr_review_merge_scheduler_core.py:4774   if behind_by and trigger_reviews:   → update 경로
                                     :4740-4790   in_progress|queued|pending|check_run|statusCheckRollup  언급 0건
.github/workflows/pr-review-merge-scheduler.yml:139  UPDATE_BRANCHES = schedule || push || pull_request_target || …

현재 head의 check 상태를 보지 않고 update합니다. pull_request_target에서도 이 경로가 열려 있습니다.

사례 1 — #1926 (7920a70b → 봇 push c730ae0b)

스케줄러 run 33963177129  pull_request_target  created 11:22:53Z  job 13:44:14Z~13:44:40Z (2h21m 대기)
봇 push                                          13:44:37Z   ← job 완료 3초 전
이전 head 7920a70b check-runs: total 28  cancelled ±1min 22  success 3  skipped 3

사례 2 — #1484 (13c444fe → 봇 push 777ee3ed, #1926 머지 +102초)

이전 head 13c444fe check-runs: total 30  cancelled ±60s 21  cancelled 그 외 0  success 8  skipped 1
→ 21/30. success 8건이 superseded head에 고아로 남음.
#1484 자기 스케줄러 run 33964505248 (11:52:51Z 생성)은 push 6초 *뒤* 14:58:48Z에 cancelled — 원인이 아니라 희생자.

희생 PR은 유발 PR이 아닙니다

33965444839(codex/originweave-mv3-owner의 pull_request_target, 14:03:11Z success)의 scan 로그(##[endgroup] 이후):

PR #1857: update_branch: current head has no OpenCode approval; branch is outdated before review dispatch; …
스캔한 PR 수: 1     update_branch 결정: 1 (#1857)

originweave의 run이 #1857을 밀었습니다. 완주 1건 = push 1건, 희생 PR은 스캔이 고른 것. 그래서 #1484처럼 자기 run이 push에 취소되는 사례가 생깁니다.

속도 (#1926 머지 후 t0/t5/t10, gh --jq 끝까지)

[t0  13:57:11Z] queued=610 (+11) | heads 2/100 | runs 17  bot-runs 0   bot-branches 0
[t5  14:02:06Z] queued=614 (+15) | heads 3/100 | runs 44  bot-runs 15  bot-branches 1
[t10 14:07:10Z] queued=600 (+1)  | heads 7/100 | runs 90  bot-runs 60  bot-branches 6
봇 push 도착: 13:58:46 · 14:03:11 · 14:03:17 · 14:05:18 · 14:05:40 · 14:06:31Z  (~100초 간격)

push당 run ~10개, 유입 분당 ~6 run, 10분 순증 +1. 취소가 유입과 같은 속도로 빠집니다. 큐는 트레드밀입니다 — 오늘 큐 깊이가 500~600 사이에서 회전만 하고 어느 방향으로도 안 간 이유입니다. 머지 직후 버스트가 아니라 스케줄러 완주율에 묶인 꾸준한 방울이며, 머지는 P(behind)를 1로 만들 뿐입니다.

한계

• #1484를 실제로 민 run은 못 찾았습니다(자기 run·근접 run·fix(dispatch): parse the trusted-dispatcher allowlist identically in all three consumers (#1929) #1932 push-sweep·fix(codeql): serialise the dispatched scan matrix with toJSON #1926 push-sweep 전부 제외; 후자는 14:06Z 기준 아직 pending). 앱 토큰을 쓰는 다른 워크플로일 수 있습니다. 스케줄러 경로에 가드가 들어가면 그건 별건입니다.
• 위 속도는 .github 한 저장소, 10분 창입니다.

수정 방향(현재 head에 대기·진행 중인 필수 check가 있으면 update_branch 대신 wait)에 동의합니다. 트리거 형태와 무관하게 그 가드가 막습니다.

[seonghobae]

Fix PR: #1937 — has_in_flight_check_runs + a wait guard in the pre-review update path, no age cap, contract test with positive/negative/empty cases, CLAUDE.md now names both update paths. Gate on the PR head: 2893 passed / 100% coverage / 100% docstrings; the new test fails against origin/main's core.

[seonghobae]

Residual old-code drip removed: cancelled 40 queued scheduler runs created before the #1937 merge.

pr-review-merge-scheduler.yml has no actions/checkout; it fetches the central scripts as a tarball at TRUSTED_SOURCE_REF = the run's workflow_sha (:313-352), which is fixed when the run is created. So every scheduler run created before 7f4c5e3e landed (14:17:16Z) would still execute the guard-less core.py when it finally got a runner — one victim each, for another 2–3 h. At 14:2xZ all 40 queued scheduler runs in this repository were pre-merge (oldest 10:52:16Z; 37 pull_request_target, 1 push, 2 repository_dispatch); none were in progress. Cancelling a queued run costs no runner time, and each affected PR is rescanned by the hourly schedule sweep and its next event with the new code, so nothing durable is lost. .github queue depth went 608 → 568.

From here on every scheduler run in this repository carries the guard, so the post-merge predictions (bot pushes → ~0, cancelled_at_push = 0, queue trending down) are tested cleanly. The same residual exists in every sibling repository where the ruleset injects this workflow; counting it there is in progress before any decision.

Cancelled run ids: 33971122915, 33971046733, 33971024024, 33970822494, 33970780040, 33970761084, 33970662388, 33970657142, 33970601476, 33970578666, 33970439323, 33970363093, 33970215310, 33970209470, 33969870992, 33969842287, 33969835753, 33969784791, 33969715681, 33969691083, 33969631593, 33969604212, 33969537936, 33969389572, 33969329111, 33969197240, 33969142435, 33969140347, 33969062391, 33968270218, 33968197172, 33967973427, 33967682868, 33967556296, 33967512492, 33967260496, 33966978881, 33966845220, 33966317530, 33961798794

[seonghobae]

Baseline correction (peer measurement): the evidence discarded per push is ~10–11 checks, not 21–22.

Check-runs for jobs behind needs: do not exist until their upstream completes; when a push cancels the workflow they are created and cancelled in the same instant. Counting only check-runs with started_at before the push:

replaced head	cancelled at push	real in-flight	phantom (created at cancel)
#1926 7920a70b	22	10	12
#1484 13c444fe	21	11	10

The mechanism and the target (0 real in-flight cancellations per bot push after #1937) are unchanged; the numbers in the issue body, the #1937 description and its CHANGELOG entry ("22/28", "21/30") are the cancelled-at-push totals and overstate the discarded evidence by about 2×. Post-merge readings use the started_at < push filter with a separate phantom= counter.

[seonghobae]

Org-wide residual removed too, and the first post-merge readings.

The ruleset injects the same scheduler into sibling repositories, with the same creation-time workflow_sha tarball, so their queued pre-merge runs carried the old core.py as well. A peer counted 43 across 19 repositories; my own per-repository query (status queued, path ending in pr-review-merge-scheduler.yml, created_at < 14:17:16Z) found 42 (clearfolio 2 vs 3 — one presumably left the queue in between). 38 cancelled; 4 cancel calls were refused — exactly the four zombies queued since 2026-07-13 (naruon, disksage), 08-19 (inkspan) and 09-02 (TEPP), which the API no longer accepts a cancel for and which are unlikely ever to run.

One straggler explained. At 14:21:00Z a bot merge landed on #1911 (ci/main-full-suite-gate), cancelling 12 real in-flight checks (+14 phantom). Its producer was scheduler run 33966317530 (created 12:32:11Z, workflow_sha 19f8ff5f, i.e. old core): it was assigned a runner at 14:20:34Z — 26 s before the .github cancellation sweep reached it — issued the update-branch, and was cancelled at 14:21:03Z. The in_progress = 0 check preceding the sweep was true when taken; sequential cancels took ~40 s. Not a guard defect. A peer also enumerated every update-branch call site on main: the scheduler core is the only one that can actually update (opencode-review-dispatch.yml calls it with --no-update-branches, agent_mention_router.py hard-codes update_branches: False), so there is no second pusher.

Independent post-merge count (surviving-head instrument, 14:25:00Z): of 137 open PRs, exactly 1 head is a bot merge-commit created after 14:17:16Z — #1911 at 14:21:00Z, the straggler above. Under the pre-merge regime the same 8 minutes produced ~5. Predictions ① and ② therefore hold from 14:21:03Z onward; ③ (queue trend) is read against 568.

Also observed live in the same window: opencode-review-dispatch run 33966200868 still rejected actor=opencode-agent[bot] at 14:20:09Z — the variable behind #1929 is unchanged.

Sibling cancellations by repository — ContextualWisdomLab.github.io: 33971329739, 33970961103; Orgmetra: 33971182649; TEPP: 33971132327, 33970743395, 33648251504; appguardrail: 33971327056, 33971019552, 33968624177; argos: 33971093164, 33970864336, 33967872388, 33966466101; bandscope: 33971280631, 33970256883, 33969894325, 33968011439; clearfolio: 33970280235, 33967679022; contextual-orchestrator: 33970774990, 33970678974, 33967327860, 33967146342, 33967113270; disksage: 29257025640; fast-mlsirm: 33970736655, 33967986067; html4tree: 33970255670, 33969794800, 33969382246; inkspan: 33969215494, 32214389134; linux-cluster-ops: 33970388678; naruon: 33969245374, 33968577535, 29254301754; nonnest2: 33970374725; pg-erd-cloud: 33970777607, 33970152562; scopeweave: 33971173311; seedream_evasepic: 33968018597; xtrmLLMBatchPython: 33970799642

[seonghobae]

⑥ after #1937 (7f4c5e3e, merged 14:17:16Z) — t5/t10/t15

Instrument: per bot-pushed branch since merge, count the previous head's check-runs cancelled within ±60 s of the push with started_at < push (real in-flight); check-runs created by the cancellation itself (started_at ≥ push) are reported separately as phantom (see the phantom split in the comment above). Pre-guard references: #1926 = 10 real (+12 phantom), #1484 = 11 real (+10 phantom).

read	UTC	queued	bot pushes since merge	real in-flight discarded
t5	14:22:21	568	1 — ci/main-full-suite-gate (#1911) @14:21:00	12 (+14 phantom)
t10	14:27:17	580	1	—
t15	14:32:23	589	2 — claude/profile-catalog-reconciliation-v2 (#1687) @14:27:14	7 (+12 phantom)

Both pushes were made by scheduler runs created before the merge, executing the old core. The scheduler fetches its source at TRUSTED_SOURCE_REF = workflow_sha, fixed at run creation, so a run queued before 14:17:16Z runs the unguarded core whenever it finally gets a runner.

• 14:21:00 (ci(tests): run the full suite on every main push #1911): run 33966317530, created 12:32:11Z, scan job got a runner at 14:20:34Z (10 steps), sent the async update-branch, then was cancelled at 14:21:03Z by the pre-merge-run sweep — 3 s too late. The other cancelled runs sampled had steps=0 runner=0.
• 14:27:14 (docs: retire removed repository from governance records #1687): run 33966913388, created 12:44:53Z, scan job 14:26:34→14:27:17Z success (push 3 s before completion — the fix(codeql): serialise the dispatched scan matrix with toJSON #1926 signature). Its log: TRUSTED_SOURCE_REF: f250638827f8252b0d9e5cb2601f4d333f96162f (main as of 12:44Z), PR #1687: update_branch: … branch update requested with OpenCode app token, counts: {update_branch: 1}. The new core would have logged holding the update. This run was inside the sweep's created-range and on its first page (only 72 scheduler runs were created after 12:44:53Z), so the sweep's list or its cancel call missed it.

Second-pusher hypothesis refuted at source. Every update-branch caller on main: pr_review_merge_scheduler_core.py (the scheduler), opencode-review-dispatch.yml:7788 which passes --no-update-branches, agent_mention_router.py:371 which hard-codes "update_branches": False, and a test script. Only the scheduler core can update a branch, and #1937 guards that path.

State now: created=<2026-09-05T14:17:16Z&status=queued → total_count=0, in_progress → 0 (server-side filter, no page trap). Every scheduler run from 14:27:17Z forward executes 7f4c5e3e. ⑥ restarts from that instant with a fresh +5/+10/+15 series.

Queue delta is not bot churn: of the 98 runs created after 14:22Z, 67 were triggered by session pushes (#1717 14d40755, #1936 c7115fe3, #1629) and 28 by the bot, across 8 branches. Prediction 3 (depth trending down) must be read net of session pushes.

Cross-check on a different artifact (peer session): filtering the 137 open-PR head commits for bot-authored two-parent commits after 14:17:16Z yields exactly the same two events.

[seonghobae]

Second straggler (#1687, 14:27:14Z) explained — and the sweep's blind spot closed.

The producer 33966913388 (created 12:44:53Z, old core, TRUSTED_SOURCE_REF f2506388) was not in the cancellation list (40 listed, 40 cancelled, 0 failed), although it shares the same workflow_id 299236240 and path and its only job waited until 14:26:34Z. Three listing instruments agree with each other now (workflow-runs total_count 9 = paginated 9 = repo-runs path-filtered 9), so this was not an API omission but a status filter: the scheduler's concurrency group is per repository and per PR (central-pr-review-merge-scheduler-{repo}-pr-{n}), so a PR with two pending events has its second run parked in status pending, which status=queued does not return. Every count so far — mine, the 43 across siblings, and "remaining old runs = 0" — read only queued.

Re-measured with status=pending|waiting|requested + created<14:17:16Z + path across .github and the 19 sibling repositories: 5 more old-core scheduler runs — .github 33969771830, 33969684836, 33969659836 (13:42–13:44Z), naruon 33969517187, appguardrail 33968100685. All five cancelled (0 failures) at ~14:39Z. A full re-sweep over both pending and queued now leaves only the four zombies the API refuses to cancel (naruon/disksage 07-13, inkspan 08-19, TEPP 09-02).

So the clean observation window for the guard starts at ~14:39Z, not 14:21Z or 14:27Z. Both stragglers so far (#1911 at 14:21:00Z, #1687 at 14:27:14Z) are old-core runs with the same signature (push seconds before the run completes, update_branch in the log where the new core prints holding the update); neither is a guard defect.

Metric note for the queue-trend prediction: .github currently has 37 runs in pending (any workflow) on top of the queued depth. Cancelling queued runs can be masked by pending runs promoting into queued, so read queued + pending, and exclude session-driven pushes (67 of the 98 runs created 14:22–14:32Z were seonghobae-actor).

[seonghobae]

Correction to my "state now" paragraph above. The claim "created=<14:17:16Z&status=queued → 0 … no page trap" was exact over the page dimension and blind over the state dimension: a scheduler run waiting on its concurrency group (pr-review-merge-scheduler.yml:79-91) sits in status=pending, not queued. That is also why the sweep missed 33966913388 — it was pending at sweep time, not absent from the API. A peer session re-read with pending|waiting|requested and found five more pre-merge scheduler runs org-wide: .github ×3 (33969771830, 33969684836, 33969659836, created 13:42–13:44Z), naruon ×1, appguardrail ×1. All three .github runs are now completed/cancelled; my independent read at 14:41:01Z over all five states (pending, waiting, requested, queued, in_progress) with created<14:17:16Z returns 0 for each.

Clean start for ⑥ is therefore 2026-09-05T14:39:13Z (the last of the three cancellations), not 14:27:17Z. Bot pushes between 14:27:17Z and that time are attributable by the pushing run's created_at exactly as the two above were.

Queue-reading rule that falls out of this: pending is a separate pool from queued (at 14:41:01Z in .github: queued 582, pending 34, in_progress 15). A cancellation that promotes a pending run into queued leaves the queued count flat, so prediction 3 must be read on queued+pending. The instrument now reports both.

General form, for anyone repeating this: a server-side status= filter is exact only over the states it names — enumerate pending|waiting|requested|queued|in_progress before calling a count "remaining".

[seonghobae]

Guard verified in production (16:18Z). Of the 25 scheduler runs created in .github after 7f4c5e3e landed, 10 have executed. Their decisions:

run	PR	decision
33971948808	#1936	wait — …branch is outdated before review dispatch, but current-head checks are still queued or running; holding the update so their evidence is not discarded
33971906521	#1934	wait — same reason
33971691883	#1872	wait — same reason
33971548421	#1911	review_dispatch — current head has completed Strix evidence; same-head OpenCode dispatched
33972657372	#1912	security_dispatch
33971862168	#1687	security_dispatch
33972093234	#1629	block — 13 unresolved review threads
33971711250	#1661	block — 1 unresolved review thread
33971916754	#1717	skip — draft PR
33971360998	push sweep	skip/block decisions only

update_branch decisions: 0 / 10. The three holds are exactly the PRs the old core would have pushed (behind, unreviewed, checks in flight).

Queue (.github, queued+pending as agreed): 616 at 14:43Z → 509 at 16:18Z (−107 in 95 min), with in_progress 18 → 24. The treadmill has stopped spinning: cancellations no longer refill the queue, and slots are being spent on runs that finish.

Remaining, out of this issue's scope: the post-approval update under strict=true (bounded, one refresh per approval), the dispatch-actor variable (#1929 — #1911's dispatch above will still be rejected at the gate until it is set), and draft PRs, which none of the gates ever approve (see #1913).

[seonghobae]

⑥ closed — the guard holds on a clean window

Clean start 14:39:13Z (last pre-merge scheduler run cancelled; see the correction above). Same instrument: per bot-pushed branch, the previous head's check-runs cancelled within ±60 s of the push with started_at < push.

window	UTC	bot pushes	real in-flight discarded	queued+pending
t5b	14:27:17→14:41:16	0 new	0	616
t10b	→14:46:22	0 new	0	628
t15b	→14:51:28	0 new	0	624
long	14:39:13→16:20:01 (101 min)	2	0	501

The two pushes in the long window: automation/sbom-inventory @15:28:30Z is the SBOM automation commit, not an update-branch (0 check-runs on the previous head). fix/agent-mention-dispatch-credential-… @15:46:59Z is the real test: the previous head 38b300a3 carried 37 check-runs and lost none — nothing was in flight when the scheduler updated it, which is precisely the guard's contract. Under the pre-guard rate (~1 push/100 s, ~10 real in-flight each) this window would have shown ~60 pushes.

Independent artifact (peer session, scheduler decision logs, 16:18Z): of the 25 scheduler runs created after 7f4c5e3e, 10 executed — wait ×3 with the guard's own reason string (#1936, #1934, #1872), review_dispatch 1, security_dispatch 2, block 2, skip 1, push sweep 1, update_branch 0/10. Check-run observation and decision-log census agree.

Predictions 1–3 hold. What the guard does not fix (peer1's scope-out at the top of this issue) is now the visible next layer: #1911 reached review_dispatch for the first time today and will be rejected at the dispatch actor gate until the #1929 variable is set; the CodeQL dispatch pipeline sits at 101 queued / 0 completed since #1926.