From cc7128c3b0fa12fa7a7367a9b27e08c935fa5878 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 17:10:24 +0900 Subject: [PATCH 1/5] docs(gap-baseline): measure PR job fan-out, disprove the workflow-waste hypothesis Static analysis of all 35 workflows (matrix expansion + three-valued if: evaluation, evaluator checked against 7 real conditions first): one PR synchronize dispatches 33 job slots, not the 36 a naive count reports. Every candidate reduction was checked and rejected on evidence: 12 slots are required status-check contexts (live branch protection), all 12 push triggers are already restricted to main/master/develop so no PR-branch duplication exists, every small quality-CI workflow already carries paths filters, and adding paths-ignore to sast-semgrep/python-security is explicitly forbidden by tests/test_docs_only_pr_runner_admission.py. The changed-scope gate jobs are documented deliberate design, not waste. Conclusion: queue depth is 76-repo ruleset fan-out (where on: filters are discarded) against a shared concurrency ceiling -- observed in_progress 0-1 for 3+ hours while queued climbed 372->424 with GitHub reporting all systems operational. Further .github YAML optimisation is not a productive lever; the open question is org-level Actions capacity, readable only with admin:org. Co-Authored-By: Claude Opus 5 --- docs/product-technical-gap-baseline.md | 83 ++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 100db3b4de..6b706b5136 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2805,6 +2805,89 @@ Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s **A concrete, lower-effort path to direction (1) already exists, half-built, sitting unmerged.** `contextual-orchestrator#911` ("Persist bounded model-group routing observations," open, not merged, `Devin`-reviewed, `2537 passed` at its own head) adds exactly the persistence primitive direction (1) calls for: a new `routing_observation_store.py` (`RoutingObservationStore`, a time-windowed SQLite-backed store keyed by `member_id`/`success`/`latency_seconds`) that lets `ModelGroupRouter`'s ledger survive across separate gateway processes within a configured wall-clock window (`--routing-observation-window-seconds`/`--state-db`). This is real, tested, already-built cross-process persistence infrastructure. **But it persists the wrong ledger for this specific gap**: `#911` wires the store into `_group_router`/`_quality_router` (the *ranking* ledgers `ModelGroupRouter.member_score`/`ranked_member_ids` read) — and, per the entry above, ranking is only ever consulted for candidates that share a `group_name`, which the free-tier pool's discovered agents never have. The mechanism that *is* consulted regardless of grouping — the circuit breaker (`self._circuit`, `_record_failure`/`_circuit_open`) — is not touched by `#911` at all; it stays exactly as in-memory and per-process as before. The concrete, low-risk next step once `#911` lands is to route `self._circuit`'s reads/writes through the same `RoutingObservationStore` mechanism `#911` already built and tested, using its existing `circuit_reset_seconds` (30s) as the natural replay window — reusing tested infrastructure for a second ledger, not building a new persistence layer from scratch. Not attempted in this pass: `#911` itself is unmerged and could still change shape before landing, and building on top of an unmerged PR risks needing a full rebase; flagging this connection (also left as a comment on `#911` itself) so whoever picks up either PR next has the concrete linkage. +## Actions queue depth: measured, and not caused by `.github` workflow waste — 2026-09-05 + +**Why this was measured.** The standing directive has repeatedly asked to find and +remove workflows that trigger unnecessarily ("쓸데 없이 Trigger 되는 workflow가 있는 거 +같은데요. 왜 각 모든 단계마다 Trigger 되고 있죠?"), on the hypothesis that gratuitous +triggering inside this repository is what fills the organization's Actions +concurrency ceiling. This entry reports a direct measurement of that hypothesis. +**It does not hold.** No recoverable waste was found; the queue depth is fan-out +arithmetic against a capacity ceiling. Recording the negative result with its +evidence so no future session re-runs this same search from scratch. + +**Method.** A static analyser walked all 35 workflow files, resolved each `on:` +block, expanded statically-enumerable matrices (each combination consumes its own +runner slot), and evaluated each job's `if:` with a three-valued +(true/false/unknown) evaluator — counting a job as *running* unless its condition +is provably false, so the number never under-reports. The evaluator was checked +against seven real conditions lifted from `noema-review.yml`, `opencode-review.yml`, +`strix.yml`, and `repository-metadata-reconcile.yml` before its output was trusted. + +**Result: one `pull_request` synchronize dispatches 33 job slots** across 14 +PR-triggered workflows. A naive job count says 36; three jobs +(`noema-review.yml:cancel-closed-pr-runs`, `strix.yml:publish-manual-pr-evidence-status`, +`repository-metadata-reconcile.yml:apply`) are provably skipped for a synchronize +event and consume nothing. Largest contributors: `opencode-review.yml` and +`security-scan.yml` at 6 each, `strix.yml` at 4, `noema-review.yml`, +`python-security.yml` at 3. + +**Every candidate lever was checked and rejected on evidence:** + +- **Required status-check contexts — 12 of the 33 slots.** Live branch protection + on `main` requires `Detect CodeQL languages`, `CodeQL compatibility analysis + (actions)`, `CodeQL compatibility analysis (python)`, `scan-pr-queue`, + `dependency-review`, `osv-scan`, `trivy-fs`, `scorecard`, `noema-review`, + `required-workflow-bootstrap`, `coverage-evidence`, and `opencode-review`. + Removing, renaming, or path-filtering any of these leaves its context Pending + forever and blocks every merge — the failure mode + `docs/doctoring/required-workflow-path-filter-boundary.md` already documents. +- **Push-trigger duplication — hypothesis disproven.** Every one of the 12 + push-triggered workflows restricts to `branches: [main, master, develop]`. No + push workflow fires on a PR feature branch, so there is no PR/push double-run on + the same commit. The 20 push-triggered job slots apply only after merge. +- **Trigger-level path filters — already applied wherever they are legal.** All + five small quality-CI workflows (`agent-mention-router`, `agent-review-runtime`, + `javascript-coverage`, `trusted-uv-materializer`, `repository-metadata-reconcile`) + plus `cloudflare-dns.yml` already carry `paths:` filters. The workflows that + remain unfiltered are unfiltered *deliberately*. +- **Adding `paths-ignore` to `sast-semgrep.yml` / `python-security.yml` — actively + forbidden.** These looked like the one remaining lever (5 slots, no required + context among their job names). `tests/test_docs_only_pr_runner_admission.py` + pins the opposite as a contract: trigger-level filters on these are a no-go + because org ruleset `18156473` discards every `on:` filter in target + repositories, and `.github`'s own classic branch protection would strand a + filtered required context as Pending. The safe mechanism is the job-level + `changed-scope` gate these workflows already use. +- **The `changed-scope` / `admit-current-head` gate jobs — deliberate, not waste.** + Roughly a quarter of the slots are small gate jobs whose only output is an `if:` + input for downstream jobs. That pattern is the documented, contract-tested + workaround for required-workflow semantics (`CLAUDE.md`: "skip at job level via a + `changed-scope` gate job instead, and always keep one job with no output-dependent + `if:` so the run concludes `success` rather than `skipped`"). Collapsing them + would trade a small slot saving for the exact Pending-forever breakage above. + +**What actually explains the queue depth.** The organization holds **76 +repositories**, and required-workflow ruleset `18156473` runs this repository's +central review and security workflows inside each target repository's context — +where, per the boundary document above, their `on:` filters are discarded +entirely. Per-PR cost therefore lands near the unfiltered ceiling in every sibling +repository simultaneously, multiplied by however many PRs are open across those 76 +repositories, against one shared concurrency ceiling. Observed the same day: +`in_progress` sat at 0–1 for over three hours while `queued` climbed 372 → 424, +with `githubstatus.com` reporting "All Systems Operational" (so not a platform +incident). That combination — deep queue, near-zero execution, healthy platform — +is what a hard concurrency or spending ceiling looks like, not what workflow waste +looks like. + +**Consequence for future work.** Optimising `.github`'s workflow YAML further is +not a productive lever; the measured floor is essentially the current 33, and the +remaining slots are load-bearing. The open question is org-level Actions capacity +(concurrency limits, spending caps), readable only with an `admin:org`-scoped +credential that no agent session holds — `gh api orgs/ContextualWisdomLab/rulesets/18156473` +and the org Actions settings both return 404/scope errors from a session token. +That is an owner action, and it is the single highest-value unblock available. + ## Item 41: CodeQL PR `startup_failure` blocking merges org-wide — dispatch-safe re-admission in progress **2026-09-04 correction.** The emergency ruleset removal below fixed the old From 7c36682c502340d25bee42ecd049627c97620c60 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 17:16:33 +0900 Subject: [PATCH 2/5] docs(gap-baseline): correct the "no recoverable waste" claim with a Codex-found exception An independent Codex audit, run against the same repo under the same constraints but without knowledge of the first pass's conclusion, found one genuine redundant slot the first pass missed: opencode-review.yml's coverage-source-tree is a full runner slot whose whole body is one echo, and which exists only to carry needs/if: into the required coverage-evidence context (itself another echo-only job). Recorded, not shipped: it is 1 slot of 33 (3%) in a queue measured 424 deep against near-zero execution, and collapsing it edits the job graph of a required review workflow the ruleset injects into 76 repositories. The capacity-bound conclusion is unchanged. The first pass classified both jobs as load-bearing gate jobs by pattern- matching the documented changed-scope design without opening them to confirm they do any work -- kept as a method note, since two agents re-reading one agent's evidence is not corroboration. Co-Authored-By: Claude Opus 5 --- docs/product-technical-gap-baseline.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 6b706b5136..a7155bb430 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2880,6 +2880,32 @@ incident). That combination — deep queue, near-zero execution, healthy platfor is what a hard concurrency or spending ceiling looks like, not what workflow waste looks like. +**Correction, from an independent Codex audit of the same question.** The claim above that +no recoverable waste exists is *slightly* too strong, and the exception is worth recording +precisely. `opencode-review.yml:coverage-source-tree` is a full runner slot whose entire body is +one `echo` ("PR-head source and coverage execution are delegated to the authenticated +default-branch OpenCode review dispatch"). It holds no other purpose than carrying +`needs: [required-workflow-bootstrap, admit-current-head]` plus `if: admitted == 'true'` into the +required `coverage-evidence` context, which is itself another single-`echo` job. Moving that +`needs`/`if:` pair onto `coverage-evidence` directly would preserve every required context and the +same skip propagation while reclaiming one slot; the documented "keep one job with no +output-dependent `if:`" invariant is carried by `required-workflow-bootstrap`, not by these two, +so it would survive. + +**It is recorded, not shipped, and deliberately so.** The saving is one slot out of 33 (3%) in a +queue measured at 424 deep against near-zero execution — it cannot move the outcome. Against that, +the change edits the job graph of a required review workflow that the org ruleset injects into 76 +repositories, and would require updating the structural contract tests that pin it. Spending +org-wide review-governance risk to reclaim 3% of a capacity-bound queue is the wrong trade. The +finding is logged here so a future session with a real reason to touch `opencode-review.yml` can +fold it in as a free side effect rather than rediscovering it. + +Method note worth keeping: this exception was found by a second auditor (Codex) run against the +same repository with the same constraints but no knowledge of the first analysis's conclusion. +Two agents re-reading one agent's evidence is not corroboration; an independent vantage point is. +The first pass had classified both jobs as load-bearing gate jobs by pattern-matching the +documented `changed-scope` design, without opening them to confirm they do any work. + **Consequence for future work.** Optimising `.github`'s workflow YAML further is not a productive lever; the measured floor is essentially the current 33, and the remaining slots are load-bearing. The open question is org-level Actions capacity From f7f480a820b1a8278eb4537943b0fcf830b9e97c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 17:38:18 +0900 Subject: [PATCH 3/5] docs(gap-baseline): correct the "3% not worth shipping" call -- wrong dimension This entry counted runner slots and concluded reclaiming 1 of 33 could not matter in a capacity-bound queue. Wrong. The two echo-only jobs sit in series on the review critical path, and a job is not created until its needs: complete, so each link waits out the whole queue again. Measured on naruon#1528: 9h40m and 13h01m of queue wait for 4s and 5s of runtime, ~22h41m total holding the real review behind it, on every PR across 76 repositories. Slot count made that look like 3%; critical-path latency is the dimension that governs how long a PR actually takes to clear. Fix shipped as #1910 (serial depth 5 -> 3, queue waits 4 -> 2). Also records the two non-obvious safety conditions: coverage-evidence had no if: of its own and relied on transitive skipping, and opencode-review-dispatch .yml has same-named jobs whose edge is a real artifact dependency -- job names are unique only within a workflow file. Co-Authored-By: Claude Opus 5 --- docs/product-technical-gap-baseline.md | 31 ++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index a7155bb430..931d4d517d 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2906,6 +2906,37 @@ Two agents re-reading one agent's evidence is not corroboration; an independent The first pass had classified both jobs as load-bearing gate jobs by pattern-matching the documented `changed-scope` design, without opening them to confirm they do any work. +**Second correction: the "recorded, not shipped" judgement above was wrong, and the metric was +measuring the wrong dimension.** This entry counted *runner slots* and concluded that reclaiming one +of 33 (3%) could not matter in a capacity-bound queue. That reasoning does not survive contact with +the data. The two echo-only jobs are not merely one wasted slot each — they sit **in series** on the +review critical path (`required-workflow-bootstrap` → `admit-current-head` → `coverage-source-tree` → +`coverage-evidence` → `opencode-review-target`), and a job is not created until its `needs:` complete, +so under a saturated queue every link waits out the entire queue again. Measured on `naruon#1528` +(run [`33581213805`](https://github.com/ContextualWisdomLab/naruon/actions/runs/33581213805)), where +each job's `created_at` equals the previous job's `completed_at` — which is what establishes the +mechanism rather than merely suggesting it: + +| job | queue wait | runtime | +| --- | --- | --- | +| `required-workflow-bootstrap` | 7h57m | 4s | +| `coverage-source-tree` | 9h40m | 4s | +| `coverage-evidence` | 13h01m | 5s | +| `opencode-review` | 12h13m | — | + +Roughly **22h41m of queue time to print two sentences**, holding the real review behind it, on every +PR in all 76 repositories the ruleset injects this workflow into. Slot count made that look like 3%; +critical-path latency is the dimension that actually governs how long a PR takes to clear. The fix +(depend both context holders on `admit-current-head` directly, dropping serial depth 5 → 3 and queue +waits 4 → 2) is `ContextualWisdomLab/.github#1910`. + +Two safety conditions that were not obvious and are worth carrying forward: `coverage-evidence` had +no `if:` of its own and relied entirely on *transitive* skipping through `coverage-source-tree`, so +parallelising without restating its admission gate would have run a required context on unadmitted +heads; and `opencode-review-dispatch.yml` contains jobs of the **same two names** whose edge is a +real artifact data dependency (one uploads the materialized PR merge tree, the other downloads it), +so the change must never be mirrored there. Job names are unique only within a workflow file. + **Consequence for future work.** Optimising `.github`'s workflow YAML further is not a productive lever; the measured floor is essentially the current 33, and the remaining slots are load-bearing. The open question is org-level Actions capacity From 841e87fc7c7147589b8f8d91ca4b5854a60c37fd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 7 Sep 2026 11:58:43 +0900 Subject: [PATCH 4/5] docs(gap): narrow Actions fan-out conclusions to measured scope --- docs/product-technical-gap-baseline.md | 32 ++++++++++++++++---------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 931d4d517d..981371d5c3 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2805,16 +2805,19 @@ Sequence across the day: 649.5s → 1332.6s → 1462.9s → 2161.9s → 2296.6s **A concrete, lower-effort path to direction (1) already exists, half-built, sitting unmerged.** `contextual-orchestrator#911` ("Persist bounded model-group routing observations," open, not merged, `Devin`-reviewed, `2537 passed` at its own head) adds exactly the persistence primitive direction (1) calls for: a new `routing_observation_store.py` (`RoutingObservationStore`, a time-windowed SQLite-backed store keyed by `member_id`/`success`/`latency_seconds`) that lets `ModelGroupRouter`'s ledger survive across separate gateway processes within a configured wall-clock window (`--routing-observation-window-seconds`/`--state-db`). This is real, tested, already-built cross-process persistence infrastructure. **But it persists the wrong ledger for this specific gap**: `#911` wires the store into `_group_router`/`_quality_router` (the *ranking* ledgers `ModelGroupRouter.member_score`/`ranked_member_ids` read) — and, per the entry above, ranking is only ever consulted for candidates that share a `group_name`, which the free-tier pool's discovered agents never have. The mechanism that *is* consulted regardless of grouping — the circuit breaker (`self._circuit`, `_record_failure`/`_circuit_open`) — is not touched by `#911` at all; it stays exactly as in-memory and per-process as before. The concrete, low-risk next step once `#911` lands is to route `self._circuit`'s reads/writes through the same `RoutingObservationStore` mechanism `#911` already built and tested, using its existing `circuit_reset_seconds` (30s) as the natural replay window — reusing tested infrastructure for a second ledger, not building a new persistence layer from scratch. Not attempted in this pass: `#911` itself is unmerged and could still change shape before landing, and building on top of an unmerged PR risks needing a full rebase; flagging this connection (also left as a comment on `#911` itself) so whoever picks up either PR next has the concrete linkage. -## Actions queue depth: measured, and not caused by `.github` workflow waste — 2026-09-05 +## Actions queue depth: static admission ceiling and observed queue evidence — 2026-09-05 **Why this was measured.** The standing directive has repeatedly asked to find and remove workflows that trigger unnecessarily ("쓸데 없이 Trigger 되는 workflow가 있는 거 같은데요. 왜 각 모든 단계마다 Trigger 되고 있죠?"), on the hypothesis that gratuitous triggering inside this repository is what fills the organization's Actions -concurrency ceiling. This entry reports a direct measurement of that hypothesis. -**It does not hold.** No recoverable waste was found; the queue depth is fan-out -arithmetic against a capacity ceiling. Recording the negative result with its -evidence so no future session re-runs this same search from scratch. +concurrency ceiling. This entry directly measures the modeled PR-trigger fan-out +under the conditions below, not actual concurrent runner occupancy. Its original +global conclusion was too broad: [#1900](https://github.com/ContextualWisdomLab/.github/pull/1900) +later demonstrated recoverable serial runner-held wait in a separate workflow +path. The static analysis remains evidence for this modeled trigger set, but it is +not an exhaustive proof that no workflow waste exists and must not be read as a +measured lower bound. **Method.** A static analyser walked all 35 workflow files, resolved each `on:` block, expanded statically-enumerable matrices (each combination consumes its own @@ -2937,13 +2940,18 @@ heads; and `opencode-review-dispatch.yml` contains jobs of the **same two names* real artifact data dependency (one uploads the materialized PR merge tree, the other downloads it), so the change must never be mirrored there. Job names are unique only within a workflow file. -**Consequence for future work.** Optimising `.github`'s workflow YAML further is -not a productive lever; the measured floor is essentially the current 33, and the -remaining slots are load-bearing. The open question is org-level Actions capacity -(concurrency limits, spending caps), readable only with an `admin:org`-scoped -credential that no agent session holds — `gh api orgs/ContextualWisdomLab/rulesets/18156473` -and the org Actions settings both return 404/scope errors from a session token. -That is an owner action, and it is the single highest-value unblock available. +**Consequence for future work.** The current 33 is a conservative static +admission ceiling/estimate for the modeled trigger set, not observed concurrent +runner occupancy and not a measured floor. [#1900](https://github.com/ContextualWisdomLab/.github/pull/1900) +removed a verified 56-second runner-held wait and 105 lines from one workflow path; +its exact branch evidence reports 470 CI tests, 3 behavior tests, and official +`actionlint` passing, but that repair is not yet protected-main evidence. Required- +context-preserving structural waste removal and org-level capacity investigation +therefore remain parallel levers. The latter (concurrency limits and spending caps) +requires an `admin:org`-scoped credential that no agent session holds — +`gh api orgs/ContextualWisdomLab/rulesets/18156473` and the org Actions settings +both return 404/scope errors from a session token — and remains an owner action, +without excluding further evidence-backed workflow repair. ## Item 41: CodeQL PR `startup_failure` blocking merges org-wide — dispatch-safe re-admission in progress From 1f48d1635cffd24542c461395b5eb3f9a23265a3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 7 Sep 2026 11:59:55 +0900 Subject: [PATCH 5/5] docs(gap): bound static fan-out estimator --- docs/product-technical-gap-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 981371d5c3..305e8469ad 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -2823,7 +2823,7 @@ measured lower bound. block, expanded statically-enumerable matrices (each combination consumes its own runner slot), and evaluated each job's `if:` with a three-valued (true/false/unknown) evaluator — counting a job as *running* unless its condition -is provably false, so the number never under-reports. The evaluator was checked +is provably false, so the static model does not omit a potentially admitted job\namong the paths it can enumerate. The evaluator was checked against seven real conditions lifted from `noema-review.yml`, `opencode-review.yml`, `strix.yml`, and `repository-metadata-reconcile.yml` before its output was trusted.