ops: diagnose and bound organization GitHub Actions queue starvation

[seonghobae]

Operational gap

Required current-head checks across ContextualWisdomLab/fast-mlsirm and the organization .github repository are remaining in queued state while obsolete heads accumulate. This blocks policy-compliant auto-merge even after CodeRabbit succeeds and review threads are resolved.

Representative evidence observed on 2026-08-03:

• fast-mlsirm#410 current-head CI/Security/SAST runs queued after prior head jobs were cancelled;
• fast-mlsirm#391, #396, and #401 exact-head CI/Security/SAST runs queued;
• .github#704 exact-head CodeQL, Security, SAST, OSV, Scorecard, SBOM, Secret Scan, and Python Security runs queued;
• .github#650, which repairs Rust coverage provisioning for fast-mlsirm, is itself waiting on queued required checks.

The repositories must not solve this by bypassing required checks or force-merging stale evidence.

Required diagnosis

1. Determine whether the queue is caused by hosted-runner concurrency, Actions billing/usage limits, organization policy, a reusable-workflow recursion, pending deployment/environment approval, or workflow-level concurrency keys.
2. Produce a machine-readable health snapshot containing repository, workflow, run id, event, head SHA, queue age, concurrency group, and whether the run is obsolete.
3. Distinguish a genuinely queued current-head run from a completed/cancelled job whose run-level status is stale or misleading.
4. Identify and cancel only obsolete runs; never cancel the sole current-head required evidence.
5. Add organization-level concurrency and close-event cancellation contracts where repository-local controls cannot operate before merge.
6. Alert on current-head required checks that remain queued beyond a declared SLO.
7. Preserve fail-closed merge policy: an unavailable runner is incomplete evidence, not success.

Acceptance

• A deterministic script and scheduled workflow report queue health across allowlisted CWL repositories.
• At most one pending/running instance exists per PR-number/workflow evidence lane, except explicitly documented matrix jobs.
• Closed PRs and superseded heads do not retain expensive active jobs.
• A current-head run can be traced from PR to workflow/job state without relying on contradictory run-level summaries.
• The report identifies the external/manual action when billing, organization policy, or runner capacity cannot be fixed in code.
• Tests, documentation, least-privilege permissions, and 100% coverage/docstrings are included.

Related work

• fast-mlsirm#410: repository-local current-head concurrency and bounded review churn.
• .github#704: hourly review-feedback repair scheduler.
• .github#650: Rust coverage toolchain/offline dependency provisioning.

[seonghobae]

@jules Please implement this as a bounded organization-governance PR from current main. Start with TDD contract tests for run/job state normalization, current-head versus obsolete classification, queue-age SLOs, and fail-closed external-blocker reporting. Keep the first slice read-only: no cancellation or mutation until the diagnostic manifest proves exact repository/workflow/PR/head identity and tests cover contradictory run/job summaries. Use least-privilege Actions/PR read permissions, deterministic JSON plus accessible HTML, 100% coverage/docstrings, and normal reviewed PR publication only.

[seonghobae]

External-status check on 2026-08-03: GitHub's public status page reports Actions operational and the most recent documented Actions capacity incidents as resolved. Prioritize organization-specific causes first (billing/usage cap, concurrency saturation, environment approval, reusable-workflow recursion, or stale run-state normalization), while retaining an external-incident field in the health manifest. Official status: https://www.githubstatus.com/

[seonghobae]

Current queue-health evidence for .github#1013@ca6ba3ec313ba6551376d942f00d1739eb1e9176 on 2026-08-15:

• CodeQL PR run 31884269793 remains queued.
• Python Security run 31884269775 remains queued.
• The same exact head already has successful OSV, Strix changed-path quality, SBOM, Scorecard, Security Scan, Secret Scan, and Semgrep runs.
• No source failure is currently attributable to the two queued lanes; they remain incomplete evidence and are not treated as success.

A separate control-plane reproduction also shows review-router queue interference: scheduled mention sweep 31887686442 remained in progress while trusted interactive run 31888050967 was pending without jobs and then cancelled. That incident is tracked with terminal evidence in #814 and should not be conflated with the CodeQL/Python Security runner queue, but both currently delay exact-head integration.

[seonghobae]

Fresh OriginWeave queue-starvation evidence, refreshed 2026-08-17 KST:

• Protected OriginWeave main remains exact 0841d2ab3d8b5e60a03c0a8e818cf438e2716829 after protected merge Use workflow token for scheduler reads #179.
• PR Install target R package dependencies for OpenCode coverage #175 exact head 52e42370ce49ad6142fb19eb0211f43716f19803 and protected PR Use workflow token for scheduler reads #179 exact feature head 16a872cddd0e336adfe0a686a75d057c05d8c74a both ultimately cleared CI/MV3/SAST/Security without product-source churn; Use workflow token for scheduler reads #179 integrated to protected main. This remains evidence that the recurring unassigned-runner class is infrastructure/control-plane behavior rather than a leaf source defect.
• PR docs: record aFIPC required workflow coverage gap #124 has advanced to current exact head 4561282c9c0cab5181b5d07d4f072f835826efc0 against protected main 0841d2ab3d8b5e60a03c0a8e818cf438e2716829. The previous e5ae33f7c7f73a0daa4daf9e18c00b6382b68d05 snapshot is historical only.
• The latest docs: record aFIPC required workflow coverage gap #124 change is itself test-first resilience hardening: test commit 3eace839515c0268d62bce6714ec30974b65839e required successful HTTP 200 registry pages to reject retry_after_seconds; production commit 4561282c9c0cab5181b5d07d4f072f835826efc0 adds the narrow fail-closed validation. It does not convert any failed or queued GitHub evidence into success.
• Native CI run 31975554347 is success on exact head 4561282...: Rust-contract job 95234397116 and production-coverage job 95234397086 both completed successfully.
• The same exact head still has local scanners waiting without runner assignment: SAST run 31975554357, Semgrep job 95234379145; Security Scan run 31975554326, jobs 95234379359 (OSV), 95234379372 (Trivy FS), 95234379377 (dependency review), and 95234379465 (Scorecard). Fresh job payloads show runner_id: 0, empty runner name/group, and no execution steps, so these remain incomplete evidence.
• The organization-required layer is also queued on the same exact head: required-workflow-bootstrap job 95234377394 in run 31975553200; scan-pr-queue job 95234376645 in run 31975553184; strix job 95234376591 in run 31975553207; noema-review job 95234376185 in run 31975553165; and close-empty job 95234375951 in run 31975553162. These exact-head checks are not predecessor evidence and are not merge-passing while queued.
• A separate Analyze (python) check 95234374630 in run 31975552222 is also queued on 4561282..., but its returned pull_requests list is empty. Keep it as control-plane evidence until its owning workflow establishes authority rather than inferring authority from its display name.
• All currently returned inline review threads on docs: record aFIPC required workflow coverage gap #124 are resolved; formal reviews are COMMENTED automation only, not a counted approval.

Additional exact-head evidence from the current OriginWeave MV3 integration root:

• PR Refresh governance audit after clearfolio rollout #43 remains unchanged at exact head a4b1da1533aebd58956a265b6fd40bf5c7d5b3bb on protected main 0841d2ab3d8b5e60a03c0a8e818cf438e2716829; local CI, Manifest V3 Compatibility, SAST, and Security Scan are all successful.
• Previously waiting strix job 95229578893 and coverage-source-tree job 95232903176 both eventually completed success without product-source churn.
• A newly dispatched exact-head required opencode-review job 95238403791 in run 31973555801 is currently queued with runner_id: 0, empty runner name/group, and no steps. It is therefore incomplete current-head evidence, not a source finding and not a passing review gate.
• This transition from earlier queued central jobs to success while a later exact-head central job again has no runner assignment further supports treating the recurring class as owner-side runner/capacity/control-plane state rather than leaf code failure. The clean Refresh governance audit after clearfolio rollout #43 source head must not be perturbed merely to retrigger it.

Fresh evidence after non-destructive protected-main reconciliation of two independent OriginWeave roots:

• PR Ignore transient governance helper cancellations #46 (feat/sensitive-handle-use-reservation) is now exact head 10aaa324e55de8eac8b5e1fa1a88f6b55812603c, ahead 7 / behind 0 from protected main with merge base exactly 0841d2ab.... Native CI run 31978398999 is success: Rust contracts job 95241291802 completed repository contracts, formatting, locked workspace checks, full tests, strict Clippy and rustdoc; production-coverage job 95241291793 completed exact owned production function/line/region/branch enforcement. On the same exact clean head, SAST run 31978399007, Semgrep job 95241260989, remains queued with runner_id: 0, empty runner name/group, and no steps. Security Scan run 31978399004 is likewise queued; its returned Scorecard check 95241261482 is queued. The leaf branch was not changed to retrigger either scanner.
• PR fix: keep OpenCode approval dispatch nonblocking #157 (fix/evidence-error-contract) is now exact head eac07a7125e541c4912b195a7522c319cb2b8792, ahead 5 / behind 0 from the same protected main with exact merge base 0841d2ab.... SAST run 31978497547, Semgrep job 95241499222, is queued with runner_id: 0, empty runner name/group, and no steps. Security Scan run 31978497543 is queued. Native CI run 31978497609 has begun executing, demonstrating that ordinary GitHub-hosted runner assignment can proceed for this same commit while the scanner lane remains unassigned.
• The independent Ignore transient governance helper cancellations #46 and fix: keep OpenCode approval dispatch nonblocking #157 current-head observations reproduce the same control-plane symptom after each branch was cleanly reconciled to current protected main. They therefore strengthen the causal classification: runner_id: 0 scanner/required-workflow queue starvation is not evidence of a source defect and must not be converted into leaf code churn, catch-all exception suppression, or a passing result.

Owner-side acceptance remains unchanged but now binds to the current exact identities above: diagnose organization runner assignment/capacity/concurrency; preserve the sole current-head required evidence lanes; identify/cancel only provably obsolete work; expose queue age plus runner assignment in the machine-readable health snapshot; distinguish jobs with missing PR linkage instead of guessing from names; and let clean leaf heads wait/refetch/retry without no-op source churn. GREEN owner proof is real runner assignment and execution for the current-head jobs, or a health snapshot that surfaces an explicit external/manual capacity, billing, policy, or provider prerequisite without weakening merge gates.

[seonghobae]

Fresh Context Fabric reproduction adds two newly created repositories to the existing queue-starvation incident, with exact current-head and runner-assignment evidence rather than a leaf-source diagnosis.

• ContextualWisdomLab/context-graph-contracts#4 exact head f34bd02eb3f78e1099a673a12d51892ff4487a9e, live base main@99cb5468ba3c15c5e79688f53dee74724fae2d13: repository CI run 31965340327 has been queued since 2026-08-16T18:41:37Z. All five jobs (95209547753, 95209547772, 95209547813, 95209547823, 95209547877) are queued, have runner_id=0, empty runner_name, and no steps. Draft PR chore: Record lack of UI codebase in palette journal #5 exact head e01896f639d31bcecd6d3224c625827d8d7ebf12 independently has CI run 31960192308 with all five jobs still queued since its earlier trigger, so this is cross-head within the same repository.
• ContextualWisdomLab/enterprise-architecture-core#4 exact head 774c2a4ce12eb13a8596720472ac660d0ef580a1, live base main@1c0fa8b15ceb9e72186274aeb255d6777eb84ef4: CI run 31965349397 has seven queued jobs (95209569578, 95209569618, 95209569632, 95209569644, 95209569647, 95209569653, 95209569655), each with runner_id=0, empty runner name and no steps. Independent runtime-readiness run 31965349396, job 95209569108, has the same shape.

First failing boundary: GitHub-hosted ubuntu-latest runner assignment has not occurred; started_at equals creation time but is not assignment evidence when runner_id=0/runner name empty and steps=[]. No Context Fabric source execution has begun, so these queued states cannot falsifiably be attributed to Python/PostgreSQL/test failures. This is consistent with the OriginWeave/noema evidence already attached to #712 and strengthens the cross-repository control-plane hypothesis.

Falsifiable RCA split for the owner lane: distinguish hosted-runner capacity/assignment, organization billing or usage limits, runner-group/policy access, global/concurrency saturation, and stale run-state. Do not retrigger unchanged heads merely to manufacture activity. Smallest safe remedy remains the #712 read-only queue-health/control-plane diagnosis plus only the operator/control-plane action supported by its evidence; cancellation should be limited to proven obsolete runs, never these current-head lanes.

Required GREEN proof for Context Fabric after the control-plane remedy: the unchanged exact heads above must be assigned real runners (runner_id>0/non-empty runner identity), execute their actual steps, and reach terminal conclusions; then Context Fabric will freshly validate every repository and central required workflow on those same exact heads. Queued/pending/cancelled/skipped/absent evidence remains non-passing.

[seonghobae]

Fresh downstream evidence from ContextualWisdomLab/noema#400 narrows an additional queue-starvation/gate-composition case that the central owner should include in this incident without weakening any hard security gate.

Exact Noema state at handoff:

• protected Noema main: eaeff88035e4c57c545d30b10970c5476fa2d765;
• PR fix: tolerate manifestless OSV scans #400 exact head: 5b60f270463e4058ae44a4915972c7a0319a0398, base main, mergeable, non-Draft;
• repository CI run 31962491331: terminal success;
• reviewer-ci run 31962491488: terminal success;
• required central Security Scan run 31962491343: still queued/non-terminal;
• within that exact Security Scan run, trivy-fs job 95202490658, osv-scan 95202490670, and dependency-review 95202490710 are terminal success;
• the remaining scorecard job 95202490669 is queued, has runner_id=0, empty runner_name, no steps, and has not produced Scorecard evidence.

The current central workflow intentionally configures Scorecard as soft/advisory at job semantics, while the Noema ruleset requires the whole central security-scan.yml@main workflow. Therefore a non-terminal advisory Scorecard job can hold the required workflow non-terminal even after the hard Trivy/OSV/dependency-review surfaces complete. This comment does not infer why GitHub has not assigned a runner; hosted-runner capacity/billing/policy/platform cause remains unresolved.

Please include this as a bounded central RCA/repair target in #712: determine whether advisory posture work should remain inside the required workflow's terminal dependency surface. A valid repair must preserve every hard Security Scan gate and fail-closed semantics; it must not translate missing Scorecard evidence into a clean Scorecard result. Candidate acceptance is a deterministic contract proving that (a) hard Trivy/OSV/dependency-review/other required security failures still fail the required workflow, (b) advisory Scorecard unavailability/queueing cannot indefinitely prevent the required security decision from reaching its governed terminal state, and (c) Scorecard evidence remains separately observable and non-fabricated. Revalidate with this unchanged Noema #400 head after any central integration; predecessor or synthetic status is not sufficient.

[seonghobae]

Fresh current-head queue-starvation evidence from the dedicated Context Fabric owner loop; no predecessor/status-only evidence is being substituted and no rerun/merge was requested.

• ContextualWisdomLab/context-graph-contracts#4 exact head 25ffc97b2e25f2a7f9010fd7410104e655027696: workflow run 31970461357 (created 2026-08-16T20:24:43Z) remains queued. Current-head jobs observed queued include test (3.11) check 95222030105, test (3.13) 95222030219, test (3.14) 95222030123, and package 95222030094 (plus the same-run 3.12 matrix job). A fresh repository query for actions/runs?status=in_progress returned zero in-progress runs while these current-head jobs remain queued.
• ContextualWisdomLab/enterprise-architecture-core#4 exact head 1c594e7480f80b5c036a103e58533fddd701327c: CI run 31970916787 and installed-process run 31970916784 (created 2026-08-16T20:34:05Z) remain queued. Current-head lanes include package check 95223133823, compose-runtime 95223133788, validate (3.11) 95223133794, the remaining Python matrix/migration jobs, and installed-process-readiness 95223133936.
• ContextualWisdomLab/context-graph-contracts#5 exact head b902b38c24bfeb85d6ab8bbdf5e32ee490ba7fe6: run 31971194168 (created 2026-08-16T20:39:44Z) has all five current-head jobs queued; observed checks include test (3.11) 95223806917, test (3.12) 95223806935, test (3.13) 95223806890, and package 95223807047.

These heads contain newly repaired product defects/regressions and require their own fresh evidence before any Draft/merge state can advance. Please classify the first external queue boundary for these exact runs per #712 acceptance (runner/concurrency/billing/organization policy/other), preserve the sole current-head evidence, and cancel only genuinely obsolete/superseded work if found. Context Fabric will continue independent local work and will revalidate these exact heads once execution starts.

[seonghobae]

Fresh exact reproduction from OriginWeave PR #175 after a real source rebase/repair, 2026-08-17 KST:

• protected main: 0841d2ab3d8b5e60a03c0a8e818cf438e2716829;
• repaired current PR head: db724d4ec2ecda58f48463546e135d29ba03bd81, ahead of that live main with merge base equal to live main;
• leaf CI 31987130156 completed success and Manifest V3 31987130192 completed success on this exact head;
• SAST run 31987130159, job 95264011970, remains queued with runner_id: 0, empty runner name, zero steps;
• Security Scan run 31987130160 is likewise still queued on the same exact clean head.

This rules out a compile/test failure as the first boundary for the queued SAST lane and gives the queue-health implementation another current-head runner_id=0 specimen. Please include this exact run/job identity in the owner-side queue classifier/regression corpus and preserve fail-closed semantics: no leaf retrigger commit, no queued→success coercion. Acceptance for this specimen is either runner assignment/execution on the unchanged head or an owner-side classified external/capacity state with bounded retry/queue-age evidence.

[seonghobae]

Additional current-head reproduction from OriginWeave after a real test-first product repair, not a retrigger-only commit:

• OriginWeave#175 exact head is now 6b7cfd74486932541bfedc66c5751a98804a0678 on protected main 0841d2ab3d8b5e60a03c0a8e818cf438e2716829.
• The preceding test-only head 4b36b53f6be66bafd5c43e7bc62b2cbd97460ccb produced the intended RED in CI run 31991495632, Rust-contracts job 95275765872, at Run tests; production then changed only the reviewed Chrome compatibility classification and exact-head CI 31991567854 is GREEN. Manifest V3 Compatibility 31991567888 is also GREEN.
• On the same exact clean production head, SAST run 31991567928 remains queued: Semgrep job 95275958416 has runner_id: 0, empty runner name/group, and no steps.
• Security Scan run 31991567894 likewise remains queued: dependency-review 95275958441, OSV 95275958451, Trivy FS 95275958452, and Scorecard 95275958604 each have runner_id: 0, empty runner name/group, and no steps.

This fresh head again separates executable leaf evidence from the unassigned scanner lane: normal CI and pinned Chromium execute successfully while the scanner jobs never receive a runner. Keep these queued jobs non-passing, do not churn the leaf source to retrigger them, and include these exact identities in the owner-side queue-health/runner-assignment diagnosis.