chore(ci): prove and prevent intermittent GitHub Actions runner-assignment stalls

[seonghobae]

Problem

Noema must distinguish GitHub Actions runner assignment from a workflow/job that is merely queued. started_at, workflow-level state, a predecessor attempt, a run-wide union of jobs, JavaScript adapter function arity, or non-canonical/invisible runner_name bytes are not runner-assignment authority by themselves.

This issue separates repository-owned evidence classification from organization/platform RCA. A repository diagnostic may classify assignment evidence; it cannot repair GitHub-hosted capacity, billing, enterprise policy, runner-group access, or a GitHub incident, and it cannot substitute for required CI/security/review conclusions.

Current implementation state — 2026-09-07 KST

• Protected Noema truth remains main@e26d771470a4ece873c367b40b3cd6cb03ac7de3, the GitHub-verified normal merge commit for fix(trust): rebind audited central workflow source #527.
• Repository-wide semantic reviewer prerequisite fix(reviewer): fail closed on empty CodeGraph semantics #546 and runner-assignment semantic-evidence lane fix(naming): make runner-assignment evidence semantic #533 are already protected truth. Their repository contracts remain the authority for distinguishing assignment from queue timestamps/state.
• Protected central .github/main is now ad0779bee66624c3997947d7691f4b0dbb973be1. PR fix(trust): roll audited central workflow source to 49eb9e7 #554 exact 888828dc35a6a45400c3e7fc5784ec7968fa2ba9 owns the narrow fail-closed OIDC job_workflow_sha consumer roll-forward; provider/retry/logging/security implementation remains with central/contextual-orchestrator owners.
• Fresh current-generation fix(trust): roll audited central workflow source to 49eb9e7 #554 application CI 34046378057, job 101522050369, is queued with steps=[], runner_id=0, empty runner_name, and runner_group_id=0, while both created_at and started_at are 2026-09-06T16:43:53Z. This is direct current evidence that started_at is not assignment authority.
• Fresh current-generation fix(trust): roll audited central workflow source to 49eb9e7 #554 Security Scan 34046378099, scope job 101522051034, shows the same pre-assignment state: queued, steps=[], runner_id=0, empty runner identity, with started_at == created_at. The sibling gitleaks job is skipped and therefore is not an assigned-runner control.
• Earlier fix(trust): roll audited central workflow source to 49eb9e7 #554 generations remain dated evidence only; do not splice their jobs into the current attempt. The current 888828dc... generation has not yet produced a positive assigned control.
• Current owner RCA remains ContextualWisdomLab/.github#712; Noema leaf reruns, no-op commits, gate weakening, or runner-selector churn remain unacceptable substitutes.

Repository-owned acceptance

• Preserve required CI/reviewer/security workflows; no synthetic replacement status.
• Keep environment/dependency waits distinct from runner-assignment stalls.
• Preserve bounded true-stall classification for a queued run/job with no assignment evidence after grace.
• Prevent retained GitHub diagnostics from exposing active delegated credentials.
• Stop treating timestamps/workflow-level state alone as assignment authority.
• Bind fetched run id, exact positive run_attempt, exact current head and exact-attempt job pages; retain run_attempt in runner evidence.
• Require the public evaluator itself to reject invalid/missing run_attempt rather than relying solely on the collector.
• Treat run-wide filter=all as non-authoritative for current-attempt identity.
• Stop treating JavaScript function arity as proof of attempt-scoped reader authority.
• Use the hardened owner-only delegated-token capability-file boundary and reject non-canonical path/token authority.
• Reject ASCII-control-only and Unicode Control/Format-only runner_name evidence when no positive runner ID exists.
• Reject raw runner-name identity containing embedded Unicode Control/Format characters rather than deleting those bytes and promoting the normalized result to assignment authority.
• Keep fix(actions): bind runner evidence to current run attempt #502 closed only after exact ancestry/preservation proved fix(governance): preserve delegated capability path authority #503 is its strict successor; fix(egress): bound anonymous GitHub API authority #500/fix(governance): preserve delegated capability path authority #503 remain integrated on protected lineage.
• Finish fix(reviewer): fail closed on empty CodeGraph semantics #546 on an unchanged exact head with terminal current reviewer/CI/Security/image evidence and zero valid unresolved findings; it is now protected truth.
• Finish fix(naming): make runner-assignment evidence semantic #533 only after the repaired reviewer contract produced current-head semantic evidence; it is now protected truth.
• From protected main, exercise operations:runner-assignment against both assigned and genuinely unassigned/stale evidence and retain the bounded receipt. Current exact unassigned specimens include fix(trust): roll audited central workflow source to 49eb9e7 #554 jobs 101522050369 and 101522051034; a positive assigned control must come from the same bounded evidence procedure rather than a predecessor attempt.

Organization/platform RCA

• Obtain authorized Actions enablement/hosted-runner policy history if needed.
• Check billing/usage restrictions where applicable.
• Check runner-group/access and organization concurrency for affected periods.
• Check enterprise policy/IP restrictions where applicable.
• Compare affected timestamps with GitHub platform incident evidence where available.
• Identify a falsifiable external cause or retain it explicitly unresolved when evidence remains insufficient.

Guardrails

Do not merge with queued/pending/failed applicable checks or non-authoritative scanner evidence. Do not change runs-on, remove checks, synthesize statuses, weaken security/review gates, normalize hostile runner-name bytes into trusted authority, duplicate an already-owned dependency-root repair in a later leaf, reopen a proven superseded predecessor without new contrary evidence, repeatedly rerun an unchanged starved head, or reuse predecessor-attempt jobs to hide allocation latency. Keep runner assignment, check conclusion, scanner evidence, formal review, merge, release, deployment, and production evidence separate.

Related: #5, #27, #29, #96, #500, #502, #503, #527, #533, #542, #546, #548, #554; ContextualWisdomLab/.github#712.

[seonghobae]

2026-08-03 follow-up

The latest PR #28 head is edc693bc5cf3e43d010c528d8438a34ca672d10e. GitHub reports the PR as mergeable and CodeRabbit is successful, but all current-head GitHub-hosted jobs are still waiting for runner assignment:

• ci run 30795894312 → job 91629278475 (verify) remains queued; no execution steps or logs
• reviewer-ci run 30795894333 → job 91629278294 (reviewer) remains queued; no execution steps or logs
• Security Scan run 30795894282 → jobs 91629281319 (osv-scan), 91629281336 (dependency-review), 91629281381 (trivy-fs), and 91629281469 (scorecard) remain queued; only cancel-closed-pr-runs is skipped

GitHub's public status page currently reports Actions as operational and does not show an active Actions incident for this period. That does not prove the repository configuration root cause, but it makes organization/repository billing, Actions enablement, hosted-runner policy, runner-group access, concurrency, or enterprise restrictions the next checks rather than weakening workflow definitions.

The PR will remain unmerged until these exact current-head jobs start and reach successful terminal conclusions. Repeated commits have reproduced the same queue condition, so no further no-op retrigger commits should be used as remediation.

[seonghobae]

2026-08-03 runner recovery and current-head state

Runner assignment partially recovered on PR #28 head 94c7d93a56bf74a5e855d22cd4ba4d19fc1128a3:

• reviewer-ci run 30796803567 completed successfully.
• Security Scan run 30796803596 completed successfully.
• ci run 30796803509 started and produced logs; it failed on three malformed Vitest parameter rows rather than on runner infrastructure.

The test table was corrected in ee776fc2f0f27abffb9a91596da7ea4ee5e7f2bc, the feature branch was fast-forwarded through the current main merge result so it is no longer behind, and the changelog trace was added. Current PR head is b77263b5c7b127ba2e266141c1315f16ec2ec0b3.

Current-head runs are again queued without runner assignment at this inspection:

• ci run 30798515233
• reviewer-ci run 30798515122
• Security Scan run 30798515064

This narrows the incident from a permanent repository disablement to intermittent or delayed GitHub-hosted runner assignment across independent workflows. The merge remains blocked until these exact current-head jobs execute and reach successful terminal conclusions. No workflow or gate has been weakened.

[seonghobae]

2026-08-03 recovery evidence

PR #28 head b77263b5c7b127ba2e266141c1315f16ec2ec0b3 received runner assignment and all current-head workflows completed successfully:

• ci run 30798515233 / job verify: success, including npm run release:verify.
• reviewer-ci run 30798515122 / job reviewer: success, including the 100% line+branch coverage and 100% docstring gates.
• Security Scan run 30798515064: scorecard, osv-scan, trivy-fs, and dependency-review all succeeded.
• CodeRabbit current-head status: success.
• Unresolved review threads: 0.

PR #28 was then SHA-bound squash-merged as 82feeb3200d4e0c318545d1f896f6e00784d229e.

The immediate merge blocker has recovered, but this issue remains open because the underlying organization-level runner-assignment cause has not been established and the acceptance criteria require reliable recurrence evidence rather than one recovered head. Do not weaken or remove the workflows.

[seonghobae]

Additional recurrence evidence

PR #31 head 2fc98d606e7e9ae5287cb04c960ff09b3c683c2b also received normal GitHub-hosted runner assignment:

• ci run 30803093531: success (release verify succeeded).
• reviewer-ci run 30803093522: success (100% line+branch and docstring gates succeeded).
• Security Scan run 30803093570: scorecard, osv-scan, trivy-fs, and dependency-review all succeeded.

PR #31 was SHA-bound squash-merged as e7e6dbac826a77260a32ec3df3285fe7892f3885. This is a second consecutive recovered head, but the original organization-level queuing root cause remains undocumented, so the issue remains open pending configuration/billing/concurrency evidence.

[seonghobae]

Additional current-head evidence (2026-08-04): PR #59 head 677180bc5ff599ae579784eea4bb6dca2a688060 created the three required pull-request workflows, but all remain queued without runner assignment or logs at inspection time:

• ci run 30876945749, job 91890238070 (verify): queued
• reviewer-ci run 30876945759, job 91890238015 (reviewer): queued
• Security Scan run 30876945723: scorecard job 91890238333, dependency-review 91890238353, osv-scan 91890238390, and trivy-fs 91890238404 queued

The PR is not being merged and no required context is being bypassed. This reproduces the cross-workflow runner-assignment condition on a new exact head after the earlier recoveries.

[seonghobae]

2026-08-06 additional runner-assignment evidence

PR #65 exact head 9e03b17d0bdda75809f394aaafb0b44cf1267a51 received normal GitHub-hosted runner assignment across all required workflows:

• ci run 31031918931: completed successfully.
• reviewer-ci run 31031918958: completed successfully, including 331 tests and the documented 100% production statement, branch, and public-docstring gates.
• Security Scan run 31031918997: completed successfully.

This is further evidence that the earlier cross-workflow queue condition is not currently reproducing. The issue remains open because the original organization-level cause and configuration/billing/concurrency evidence are still not established; no workflow, runner label, or required gate was weakened.

[seonghobae]

2026-08-06 runner-assignment recovery evidence

PR #65 exact head 44e3cb839a9769e6ad795e4e16f4015b1548f506 received normal GitHub-hosted runner assignment and terminal results for every current-head workflow:

• ci run 31065763303, job 92503029261: completed successfully; release verification executed.
• reviewer-ci run 31065763329, job 92503029119: completed successfully; 332 tests passed, production statement/branch coverage remained 100%, and public docstring coverage remained 100%.
• Security Scan run 31065763285: dependency-review job 92503029324, OSV job 92503029351, Trivy job 92503029360, and Scorecard job 92503029399 all completed successfully. Conditional fallback/report steps were skipped only where their primary path succeeded; the closed-PR cancellation job was expectedly skipped because PR feat(sandbox): add quarantined patch validation #65 is open.

This demonstrates current recovery for one integrated exact head and no missing job logs. It does not establish the original organization-level root cause or long-term reliability, so #30 should remain open until Actions/billing/runner-group/concurrency policy is audited and the remediation/root-cause evidence required by the acceptance criteria is attached. No workflow or required gate was weakened.

[seonghobae]

2026-08-07 recurrence evidence: PR event has no observable Actions runs

PR #69 exact head 9abe3be2220361cb7cb289325f3f03e6e26bf739 is open against main, but the commit-scoped Actions query currently returns no pull-request-triggered workflow runs at all for this exact head. This differs from the earlier queued-run symptom: there are no current-head ci, reviewer-ci, or Security Scan run objects to inspect, so there is nothing safe to rerun and no terminal check evidence to treat as success.

The repository's ci workflow remains configured for pull_request; no workflow gate is being weakened or replaced. PR #69 therefore remains blocked on missing exact-head CI/security/coverage evidence. No no-op retrigger commit or repair workflow will be used. The current PR body records the missing-run state explicitly.

This comment is evidence only; it does not establish whether the cause is Actions event suppression, repository/organization policy, App/token event semantics, or runner infrastructure. #30 should remain open until the relevant Actions/billing/policy/event-delivery configuration can be audited with authoritative evidence.

[seonghobae]

2026-08-07 runner/event recovery evidence

PR #69 has since advanced to exact head eaf712097d11a830667971095f9dcf985cc2978f. Commit-scoped pull-request workflow discovery now returns terminal runs instead of the prior no-run state:

• ci run 31136829151 / verify: completed successfully; release:verify executed, 68 test files / 764 tests passed, production statement/branch/function/line coverage were 100%, and npm reported zero vulnerabilities.
• reviewer-ci run 31136829154 / reviewer: completed successfully, including its 100% line+branch coverage and 100% public-docstring gates.
• Security Scan run 31136829152: Scorecard, dependency-review, Trivy filesystem, and OSV jobs all reached successful terminal conclusions.

This documents recovery from the missing-run recurrence but does not establish the original Actions/event-delivery/runner-assignment root cause or satisfy the configuration/billing/concurrency audit acceptance criteria. #30 therefore remains open. No runner label, required check, workflow gate, or protection has been weakened.

[seonghobae]

2026-08-09 fresh queue evidence: current PR #71 exact head 768882e173de23f521f6ca40534e0ee390c85c8b has three independently created PR workflow runs whose executable jobs have not received a runner: ci run 31310403045 → job 93237244647 (verify) is queued with no steps/logs; reviewer-ci run 31310403046 → job 93237244482 (reviewer) is queued with no steps/logs; central Security Scan run 31310403060 → scorecard, dependency-review, trivy-fs, and osv-scan jobs are all queued with no steps/logs (only the unrelated cancel-closed-pr-runs condition is terminal-skipped). Separately, PR #67's targeted patch-validator-image retry is also still queued, while older #76 exact-head runs completed successfully. This supports an intermittent/shared runner-assignment or organization capacity/policy boundary rather than a #71 source failure: the queued jobs have not begun source checkout. Repository source mutation or repeated commits cannot repair this boundary. Acceptance still requires checking organization/repository Actions enablement, billing/spending/usage, hosted-runner/runnerset access and queue/concurrency state as this issue lists. Pending/queued evidence remains non-passing; no required workflow was weakened or bypassed.

[seonghobae]

Additional current feasibility evidence: GitHub's official status page presently reports Actions as operational and no current platform-wide incident (https://www.githubstatus.com/). Combined with #71's independently queued ci, reviewer-ci, and Security Scan jobs having no runner-assigned steps/logs, while older #76 jobs completed, this makes a current global GitHub Actions outage a weaker hypothesis. The highest-value remaining probes are organization/repository-specific Actions enablement, hosted-runner eligibility, billing/spending/usage hold, runner-group access, and shared concurrency/queue saturation. This remains an inference, not a proven root cause, until those control-plane settings are inspected.