docs(doctoring): record the org's Actions plan concurrency ceiling as root cause #1754
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| # Doctoring record: the org's GitHub Actions concurrency ceiling is a plan-level quota, not a workflow defect (2026-09-03) | ||
|
|
||
| - **Date:** 2026-09-03 | ||
| - **Subject:** two peer sessions independently observed the org's GitHub Actions run queue growing rather | ||
| than shrinking this week and, in that tick, proposed auditing/consolidating/centralizing workflow files | ||
| across the org as the fix. Before either session sank time into that plan, this root cause needed a | ||
| durable record: the actual bottleneck this session identified is a **plan-level concurrent-job quota**, | ||
| not workflow duplication, and consolidating workflow files cannot lift it. | ||
| - **Decision record:** none in `docs/adr/` — this is a diagnostic/root-cause finding for the org owner's | ||
| awareness and eventual plan-tier decision, not an architecture decision this repository can make. | ||
| - **PR:** see the PR that carries this commit. | ||
|
|
||
| ## Primary evidence | ||
|
|
||
| The user directly reported, and shared a screenshot of, the organization's GitHub Actions usage view | ||
| earlier in this session showing **58-60 of a 60 concurrent-job plan limit in use**. That is the primary | ||
| source for the specific ceiling figure in this record. The raw screenshot itself is not reproducible from | ||
| this doc (it was shared inline in conversation, not committed to the repository), so the number here is | ||
| reported as the user stated it, not independently re-derived pixel-for-pixel — flagged explicitly so a | ||
| reader can tell primary-source-observed-directly-by-the-user apart from what this session could verify | ||
| itself via the API (below). GitHub does not expose an org's concurrent-job plan ceiling through the | ||
| standard REST API available to this session (it is a billing/plan-settings value, visible only in the | ||
| org's own Settings → Actions/Billing UI) — confirming the exact number and its precise scope (whether it | ||
| counts standard-runner jobs only, whether larger/self-hosted runners have a separate pool, which plan tier | ||
| the org is on) requires the org owner to check that page directly; this record does not claim to have | ||
| re-verified those specifics independently. | ||
|
|
||
| ## Corroborating evidence (live, reproducible, gathered for this record) | ||
|
|
||
| A live sample taken 2026-09-03 across three of the org's most CI-active repositories, using: | ||
|
|
||
| ```bash | ||
| gh api "repos/ContextualWisdomLab/<repo>/actions/runs?status=in_progress&per_page=1" --jq '.total_count' | ||
| gh api "repos/ContextualWisdomLab/<repo>/actions/runs?status=queued&per_page=1" --jq '.total_count' | ||
| ``` | ||
|
Comment on lines
+33
to
+35
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- repository guidance ---'
find /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478 -maxdepth 2 -type f -name '*.md' -print
printf '%s\n' '--- target document ---'
cat -n docs/doctoring/actions-plan-concurrency-ceiling-20260903.md
printf '%s\n' '--- relevant architecture/learnings/guidance excerpts ---'
for f in /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/{contextual_wisdom,architecture,learnings}/*.md; do
[ -f "$f" ] || continue
printf '\n### %s\n' "$f"
sed -n '1,220p' "$f"
doneRepository: ContextualWisdomLab/.github Length of output: 10424 🏁 Script executed: #!/bin/bash
set -eu
for f in \
/tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/conventions/docs.md \
/tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/conventions/docs-doctoring.md \
/tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/conventions/repo-wide.md \
/tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/learnings/*.md
do
[ -f "$f" ] || continue
printf '\n### %s\n' "$f"
cat "$f"
doneRepository: ContextualWisdomLab/.github Length of output: 1727 🏁 Script executed: #!/bin/bash
set -eu
cat -n docs/CWL-MASTER-CONTEXT.mdRepository: ContextualWisdomLab/.github Length of output: 38230 🌐 Web query:
💡 Result: The GitHub REST API provides specific endpoints to retrieve information about workflow runs and their associated jobs, including counts and concurrency management. Workflow Runs and Jobs To retrieve workflow runs, use the List workflow runs for a repository endpoint (GET /repos/{owner}/{repo}/actions/runs) [1][2]. The response body includes a total_count field, which returns the total number of workflow runs for the repository [1][2]. To list jobs for a specific workflow run, use the List jobs for a workflow run endpoint (GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs) [3][4]. This response also includes a total_count field representing the number of jobs in that specific run [5]. Concurrent Jobs Limits GitHub imposes limits on the number of concurrent jobs that can run on GitHub-hosted runners, which vary by plan [6][7]: - Free: 20 concurrent jobs [6][7]. - Pro: 40 concurrent jobs [6][7]. - Team: 60 concurrent jobs [6][7]. - Enterprise: 500 concurrent jobs [6][7]. Larger runners have higher limits (e.g., 1,000 concurrent jobs for Team and Enterprise plans) [6][7]. These limits can sometimes be increased by contacting GitHub Support [6][7]. Managing Concurrency You can manage the execution of concurrent jobs within your own workflows using the concurrency key [8][9]. By defining a concurrency group, you can ensure that only a single job or workflow run with that key executes at a time [8][9]. When configured: - By default (single), only one job or workflow run can be pending, and new runs will cancel existing pending ones [8][9]. - You can optionally set queue: max to allow up to 100 jobs or workflow runs to wait in the queue before new ones are canceled [8]. Citations:
워크플로 실행 수를 동시 job 수로 해석하지 마세요.
표의 열을 🤖 Prompt for AI AgentsSource: MCP tools |
||
|
|
||
| | Repository | `in_progress` | `queued` | | ||
| |---|---|---| | ||
| | `.github` | 5 | 1,877 | | ||
| | `contextual-orchestrator` | 0 | 727 | | ||
| | `naruon` | 5 | 416 | | ||
| | **Total (3-repo sample)** | **10** | **3,020** | | ||
|
Comment on lines
+30
to
+42
Contributor
There was a problem hiding this comment. 🔍 Run counts obscure job utilization The commands count workflow runs, while the asserted ceiling limits jobs. Multi-job runs and unrelated queue states weaken this sample's root-cause evidence. 
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| This is a deliberately small sample, not a full 63-repo census — an attempted full sweep across every | ||
| non-archived, non-fork repository (the same corpus as the 2026-09-02 workflow-duplication audit) hung | ||
| indefinitely on this run and was aborted; a post-hoc `gh api rate_limit` check immediately after showed | ||
| 5,000/5,000 REST calls remaining, so the hang was not caused by hitting the org's shared REST rate limit | ||
| (consistent with this session's standing practice of preferring REST over GraphQL to avoid that limit) — | ||
|
Comment on lines
+46
to
+48
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- repository knowledge scopes ---'
head -5 /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/*/*.md 2>/dev/null || true
printf '%s\n' '--- target document ---'
sed -n '35,120p' docs/doctoring/actions-plan-concurrency-ceiling-20260903.md
printf '%s\n' '--- referenced investigation ---'
sed -n '100,155p' docs/doctoring/org-queue-sweep-rate-limit-investigation-20260902.mdRepository: ContextualWisdomLab/.github Length of output: 12057 🌐 Web query:
💡 Result: GitHub enforces two primary types of API rate limits to maintain service availability: primary rate limits and secondary rate limits [1][2]. Primary Rate Limits These are the standard, documented limits based on authentication methods and organization plans [1]: - Unauthenticated requests are limited to 60 requests per hour [1]. - Authenticated requests generally have a limit of 5,000 requests per hour [1]. - GitHub Enterprise Cloud organizations may grant apps acting on your behalf higher limits (e.g., 15,000 requests per hour) [1]. - Exceeding these limits results in a 403 or 429 status code, with an x-ratelimit-remaining header value of 0 [1][3]. You must wait until the time specified in the x-ratelimit-reset header (in UTC epoch seconds) to retry [3][4]. Secondary Rate Limits Secondary rate limits are used to prevent abuse and ensure fair resource usage [2][5]. These are subject to change without notice and can include [2]: - Concurrent Requests: A maximum of 100 concurrent requests are allowed across the REST and GraphQL APIs [2]. - Endpoint Rate: Limits on requests to a single endpoint per minute (e.g., 900 points per minute for REST API) [2]. - CPU Time: A limit of 90 seconds of CPU time per 60 seconds of real time [2]. - Content Creation: Limits on how much content (e.g., issues, comments) can be created in a short period (generally 80 requests/minute or 500 requests/hour) [2]. - OAuth Token Requests: A limit of 2,000 requests per hour for GitHub Apps and OAuth apps [2]. If you exceed a secondary rate limit, you receive a 403 or 429 response [3][4]. If a Retry-After header is provided, respect the indicated duration; otherwise, use an exponential backoff strategy [3][4]. The /rate_limit Endpoint You can check your current primary rate limit status at any time by calling the GET /rate_limit endpoint [6][7]. - This request does not count against your primary rate limit, though it may count against secondary rate limits [1][4]. - The response categorizes your limits by resource (e.g., core, search, graphql), providing the limit, remaining, used, and reset values for each [6][7]. - Official documentation recommends using the rate limit response headers (provided with every API call) whenever possible instead of polling the /rate_limit endpoint [1][4]. Citations:
🤖 Prompt for AI AgentsSource: MCP tools |
||
| its actual cause is undetermined and not investigated further here, since the 3-repo sample already | ||
| establishes the pattern this record needs. | ||
|
|
||
| The pattern itself is the useful signal: single-digit `in_progress` counts (5, 0, 5) against | ||
| quadruple-digit `queued` counts (1,877; 727; 416) in the same moment, across independently-owned | ||
| repositories, each triggering its own workflows on its own schedule. That shape — many jobs queued, | ||
| very few ever concurrently running — is exactly what a hard, roughly-constant, **org-wide** (not | ||
| per-repository) concurrent-job ceiling produces, and is hard to explain by per-repository causes alone | ||
| (each repository's own workflow volume, trigger frequency, and CI design differ substantially). It is | ||
| consistent with, though does not by itself prove, the specific 58-60/60 figure from the primary evidence | ||
| above. | ||
|
|
||
| ## Relationship to other queue-related findings already in this repository | ||
|
|
||
| This is not the first queue-depth observation recorded here, and this finding does not supersede or | ||
| contradict the earlier ones — they describe different, plausibly-compounding causes: | ||
|
|
||
| - `docs/product-technical-gap-baseline.md`'s 2026-08-31 entry (chained required-workflow poller removal) | ||
| cites "53 concurrent Actions runs and a growing runner queue" as the trigger for removing roughly eleven | ||
| runner-hours of polling per PR — a real, already-fixed contributor to total load, but framed as a | ||
| mechanism-level fix (reduce runner-hours consumed per PR), not a claim about the plan's own ceiling. | ||
| - The later `ubuntu-latest` starved-floating-image finding (same file, referencing 822 queued Actions runs | ||
| observed at merge time) diagnosed a *scheduling* problem — GitHub-hosted runners requesting the floating | ||
| `ubuntu-latest` label sitting `queued` with no runner assignment for hours even when capacity should have | ||
| been available, fixed by pinning off the floating label. That is a distinct failure mode from a hard | ||
| concurrency quota: a starved image can leave slots idle *despite* available capacity, whereas a plan | ||
| ceiling caps how many jobs can ever run concurrently even with perfect scheduling. Both can be true at | ||
| once and both can slow the same queue; neither finding invalidates the other. | ||
| - A separate, still-unmerged-as-of-this-writing finding (`project_strix_concurrency_starvation_unfixed` in | ||
| this session's own working notes) identifies that `strix.yml`'s concurrency group is scoped per-repository | ||
| rather than per-PR, which starves cross-PR Strix evidence specifically — again a distinct, compounding | ||
| mechanism, not the same thing as the org-wide plan ceiling this record documents. | ||
|
Comment on lines
+77
to
+80
Contributor
There was a problem hiding this comment. 🔍 Private notes replace durable evidence The Strix claim cites
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| ## Implication for workflow-consolidation proposals | ||
|
|
||
| Consolidating or centralizing workflow files — the idea both peer sessions were independently converging | ||
| on this tick as *the* fix for the growing queue — is real hygiene and can reduce the *total number of | ||
| runs triggered* (fewer redundant CI paths competing for the same slots), which helps the queue drain | ||
| somewhat faster once jobs are submitted. It does **not** change how many jobs GitHub will run concurrently | ||
| for this organization at once: that number is set by the plan tier, not by how many `.yml` files exist or | ||
| how many of them are centralized versus per-repository. A large cross-repo consolidation-and-deletion | ||
|
Comment on lines
+87
to
+89
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- applicable repository guidance ---'
find /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478 -maxdepth 2 -type f -name '*.md' -print
printf '%s\n' '--- target document ---'
sed -n '70,110p' docs/doctoring/actions-plan-concurrency-ceiling-20260903.mdRepository: ContextualWisdomLab/.github Length of output: 4523 🏁 Script executed: printf '%s\n' '--- documentation conventions ---'
cat /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/conventions/docs.md
cat /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478/conventions/docs-doctoring.md
printf '%s\n' '--- target context and audit data ---'
sed -n '1,105p' docs/doctoring/actions-plan-concurrency-ceiling-20260903.mdRepository: ContextualWisdomLab/.github Length of output: 8651 🌐 Web query:
💡 Result: GitHub Actions concurrency limits depend on the type of runner used, as outlined below: Standard GitHub-hosted runners Concurrency for standard GitHub-hosted runners is governed by your GitHub plan [1][2]. Limits apply to the total number of concurrent jobs, with specific sub-limits for macOS and GPU-powered runners [1][3]: - Free plan: 20 total concurrent jobs (5 macOS) [1] - Pro plan: 40 total concurrent jobs (5 macOS) [1] - Team plan: 60 total concurrent jobs (5 macOS) [1] - Enterprise plan: 500 total concurrent jobs (50 macOS) [1] Note that the maximum concurrent macOS jobs limit is shared across both standard GitHub-hosted runners and GitHub-hosted larger runners [1][3]. Larger runners Larger runners allow for higher concurrency, typically up to 1,000 concurrent jobs for Linux CPU runners, though exact limits vary by runner type [1][3]. Concurrency for larger runners is configured when setting up the runner pool in the "Capacity" section, where you can define a "Maximum concurrency" value to suit your scaling needs [4][5]. Self-hosted runners Self-hosted runners do not have plan-level concurrency limits enforced by GitHub [6]. Instead, their concurrency is governed entirely by the capacity you provide [6]. - For static fleets, concurrency is limited by the number of registered machines [6]. - For autoscaled fleets (such as those managed by Actions Runner Controller), concurrency is determined by your autoscaling configuration (e.g., the max runners setting on a scale set) [6][7]. Support requests GitHub Support can increase standard job concurrency limits for GitHub Actions [1][3]. To request an increase, you must submit a support ticket through the GitHub Support portal [1][3]. Similarly, if your usage requires adjustments to larger runner capacities or specific configuration needs, contact GitHub Support for assistance [8][5]. Citations:
🏁 Script executed: printf '%s\n' '--- master documentation context ---'
if [ -f docs/CWL-MASTER-CONTEXT.md ]; then cat docs/CWL-MASTER-CONTEXT.md; else find . -name 'CWL-MASTER-CONTEXT.md' -print; fi
printf '%s\n' '--- relevant learnings and architecture ---'
find /tmp/coderabbit-repo-knowledge/contextualwisdomlab-github-bad40478 -maxdepth 2 -type f \( -path '*/learnings/*' -o -path '*/architecture/*' \) -print -exec cat {} \;Repository: ContextualWisdomLab/.github Length of output: 36701 🌐 Web query:
💡 Result: GitHub-hosted larger runners have a default maximum concurrency limit of 1,000 concurrent jobs for both Team and Enterprise plans [1]. While this is the standard limit for larger runners, specific limits for macOS and GPU jobs differ and are shared across standard and larger runners [1]. If your organization requires a higher concurrency limit than the default, you can request an increase by submitting a ticket through GitHub Support [1]. For standard GitHub-hosted runners, the concurrency limits are plan-dependent: - Free: 20 concurrent jobs [1] - Pro: 40 concurrent jobs [1] - Team: 60 concurrent jobs [1] - Enterprise: 500 concurrent jobs [1] When requesting an increase for larger runners, it is also important to note that if you are using private IP scaling for virtual network (VNet) injection, you must maintain a 30% buffer to accommodate the maximum job concurrency you anticipate [1]. Citations: 🌐 Web query:
💡 Result: GitHub Actions does not impose a hard, platform-enforced "concurrency limit" on the number of jobs that can run on self-hosted runners [1][2]. Unlike GitHub-hosted larger runners, which allow you to explicitly configure a "Maximum concurrency" setting in the runner group settings, self-hosted runner capacity is determined entirely by the hardware, infrastructure, and management strategy you provide [3][4][2]. Key points regarding self-hosted runner concurrency: 1. Capacity is hardware-dependent: Because you manage the underlying infrastructure, the number of concurrent jobs a self-hosted runner can handle is limited only by the CPU, RAM, and disk resources of the machine or cluster you deploy [2]. 2. Multiple runners per machine: You can configure multiple runner applications on a single physical or virtual machine to process jobs in parallel [2]. Each runner application typically processes one job at a time. 3. Autoscaling: GitHub does not provide native autoscaling for self-hosted runners [2][5]. To achieve dynamic concurrency (scaling the number of runners up or down based on the queue size), you must implement your own autoscaling solution, such as using runner controller tools (e.g., Actions Runner Controller on Kubernetes) [5]. 4. Concurrency vs. Queuing: While there is no global limit on how many jobs your self-hosted runners can process simultaneously, you can use the Citations:
runner pool별 동시 작업 한도를 구분하세요. 87-89행의 plan-level 한도 설명은 standard GitHub-hosted runner에만 적용하세요. Larger runner는 별도 🤖 Prompt for AI AgentsSource: MCP tools |
||
| effort undertaken on the theory that it would resolve the backlog would be solving the wrong layer of the | ||
| problem, at real cost (each deletion needs branch-protection `required_status_checks` re-verified per | ||
| repo, and any repo-specific `with:` tuning preserved or intentionally dropped). | ||
|
|
||
| ## Recommendation | ||
|
|
||
| This is a plan/billing decision, not a code change either agent session can make: raising the concurrent-job | ||
| ceiling (a higher GitHub plan tier, purchasing additional included concurrency, or provisioning | ||
| self-hosted/larger runners with their own separate capacity pool) is the org owner's call to make with the | ||
| actual billing page in front of them, not something to infer further from repository-side evidence. | ||
| Workflow consolidation remains worth pursuing for its own, independent hygiene reasons (see | ||
| `docs/doctoring/ci-workflow-duplication-audit-20260902.md` for what is and is not already duplicated | ||
| org-wide) — but should not be scoped or prioritized as *the* fix for the current backlog growth. | ||
|
|
||
| ## Audit trail | ||
|
|
||
| - User-reported screenshot of the organization's Actions usage view, shared earlier in this session | ||
| (primary source for the 58-60/60 figure; not independently re-verifiable from this record alone). | ||
| - Live `gh api` sample gathered 2026-09-03 for this record (table above); `gh api rate_limit` confirmed | ||
| 5,000/5,000 REST calls remaining immediately after the aborted full-org sweep, ruling out rate-limiting | ||
| as the sweep's failure cause. | ||
| - `docs/product-technical-gap-baseline.md` — the 2026-08-31 chained-poller-removal entry and the | ||
| `ubuntu-latest` starved-image entry, both cross-referenced above. | ||
| - `docs/doctoring/ci-workflow-duplication-audit-20260902.md` — the org-wide workflow-duplication sweep this | ||
| record's "Implication" section points back to. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
증거 수집일을 현재 날짜와 일치시키세요.
현재 검토 기준일은 2026년 9월 2일입니다. 문서는 표본이 2026-09-03에 수집됐다고 기록합니다. 2026-09-03은 미래 날짜입니다. 실제 실행일을 기록하거나, 2026년 9월 3일 이후의 커밋으로 이 기록을 이동하세요.
Date, 본문,Audit trail의 날짜도 같은 값으로 맞추세요.🤖 Prompt for AI Agents