Skip to content

[PR] ci: stop runner-queue pileups (serialize release trio, protect theorem lane, starvation alert) - #5333

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-31_ci-runner-queue-optimize
Oct 31, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-31_ci-runner-queue-optimize

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

What this fixes (human words)

On 2026-10-31 the queue clogged for ~3.5h: every job actually runs 2–8 min, but waited up to ~3.5 h just to get a runner (GitHub hosted allocation throttling on a day we triggered 800+ runs). The pile-up that made it worse had three structural holes, all fixed here:

  1. Every master merge fired 3 runs (Master Build Artifact + Release + Release Kernel) with no concurrency groups — 8 merges in one morning = 8 stacked sets, all firing concurrently when capacity returned (~8× runner slots burned by mostly-no-op runs). Now grouped per ref: master-build cancels superseded runs ([bug] 0.1.195 输出预算三路饿死仍复现:无 output 上限模型 restore 无下限(1→252、fork 原样转发 1)、#2490 planning floor 钳到 1479–2623 → 3 会话 9 次 finish=length 空轮不重试(#546/#1665/#1840/#2904/#2756 残留) #3071 keeps the latest alive); release/release-kernel serialize with cancel-in-progress: false because two real back-to-back releases must publish in order, never cancelled mid-publish.
  2. ci-theorem was the only CI lane with no concurrency at all → now uses the same house pattern as ci-e2e (197-master range review | 197-master区间审查 #3331/ci: any non-ci:full label on a PR cancels its in-flight matrix run and records a vacuous skip-green #3772), incl. the -skip group so non-ci:full label events can't kill a real run.
  3. The janitor's cancel loop never actually ran. The 2026-10-30 release pointed its sweep loop at stdin (empty inside Actions), so every rule — zombie reap, age limits, master warnings — was dead code from day one: an hour-a-day of no-op sweeps while the Oct 30 queue sat unhealed. Fix: feed the collected list into the loop explicitly (the redirect must sit on done, not read — binding it to read reopens the file at offset 0 every iteration and infinite-loops on line 1; verified locally before pushing). Plus two new tripwires, both tuned via dispatch inputs: when more than starvedQueueLimit runs (default 30) sit queued at once the job emits a loud ::warning:: with the per-event breakdown, and queued runs older than starvedQueuedMaxMinutes (default 60; 0 disables) get cancelled — after 2026-10-31 we know backlog depth itself stalls dispatch for this repo, so waiting out the old 120-min rule only snowballs further. Master push runs stay exempt either way (head's required checks).

Plus timeout-minutes where the GitHub 6 h default was still in effect: release(90), release-kernel(90), release-bugfix(90), release-manual(90), advisories(30), ci-gate-plugins(30), pages(30), theorem gate(10)/theorem(45).

Also fixed here: three pre-existing master reds (found while optimizing)

Master tip itself was failing these three guards when this PR opened; fixing them on the branch so review/merge sees green (verified byte-exact diffs against master before pushing):

Once those gates went green the test matrix ran for the first time since the reds started, surfacing two more latent master failures (same hide mechanism):

Baseline numbers (why the long tails are structural too)

Healthy day (Oct 30, n=144): e2e lanes median 19–57 min each, Release Kernel/Release/ci-image ~54–55 min, windows overlay ~31 min. With 15 cron lanes ticking every 30 min, any busy merge day easily outstrips hosted supply — which is exactly what happened.

Not changed here (owner decisions wanted)

  • Cron cadence: 15 lanes × every 30 min ≈ 48 scheduled runs/hour. Lengthening quiet lanes to hourly would cut steady-state demand ~50% at the cost of slower regression catch.
  • Self-hosted (or extra) runners for janitor + gate jobs so self-healing can't be starved by the thing it heals.
  • Merging some small e2e lanes into shared DAGs.

Product-code surface: only the one-token export drop in src/codex-models.ts (type visibility only, no behavior change). No config surface, no kernel/pi-subagents changes. Workflow YAML validated (safe_load); the janitor sweep script was smoke-tested locally against fixture runs (overload / breaker-off / normal-depth scenarios).

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-31_ci-runner-queue-optimize (f571b60)

Latest build — install this exact version

npm install -g billion-context@0.0.7528-pr.5333.39e48ab

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
f571b60 billion-context@0.0.7528-pr.5333.39e48ab build
682d894 billion-context@0.0.7519-pr.5333.2915511 build
dd9d0b2 billion-context@0.0.7513-pr.5333.da93f4b build
b5f19f6 billion-context@0.0.7509-pr.5333.812e319 build
ef74820 billion-context@0.0.7489-pr.5333.3755be2 build
fe4cf2e billion-context@0.0.7476-pr.5333.847f173 build
2ba78cd billion-context@0.0.7473-pr.5333.a5fe31c build
2767a07 billion-context@0.0.7469-pr.5333.afe243a build
67976e2 billion-context@0.0.7465-pr.5333.1d944c9 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5333.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title ci: stop runner-queue pileups (serialize release trio, protect theorem lane, starvation alert) [PR] ci: stop runner-queue pileups (serialize release trio, protect theorem lane, starvation alert) Oct 31, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复(open+closed 全量检索无同类签名;这是 dsh-plugin-hub 安装器自动立项的首例该签名)。
标签:dsh · scope/other · p2(环境性安装受阻,非本仓缺陷,故不打 bug;保留系统原有 pr)。

Diagnosis

This is not a billion-context packaging defect. The auto-filer blamed billion-context only because that's the name in your add command — but the bytes that actually failed to download belong to a different plugin already sitting in your profile.

Evidence:

  • The published package is clean. npm view billion-context@latest → deps are only zod, mysql2 (+ optional jimp). No git dependencies. Installing it pulls nothing from codeload.github.com.
  • The failing URL is codeload.github.com/ebony-vinyl/dsh-our-free-model/tar.gz/<sha> — a tarball of a third-party plugin (dsh-our-free-model, a "free model" provider), not ours.
  • The log line "while installing a direct dependency of …/profiles/web" confirms it: your web profile declares dsh-our-free-model as a git-based dependency.

Mechanism: dsh plugin … add billion-context@latest runs pnpm add 'billion-context@latest' inside your web profile. pnpm reifies the entire profile on every add, so it re-fetches every git dependency too — including that ebony-vinyl/dsh-our-free-model tarball. On your network that codeload.github.com fetch timed out (after the 10 s / 1 min retries), so the whole pnpm add aborted and billion-context was never written. The hub's auto-guessed cause ("prepare/build script failed") is wrong — it's a network timeout on a sibling dependency, not a build failure of our package.

What to do

  1. Give the profile reliable GitHub egress. The real blocker is reaching codeload.github.com. If you're behind a firewall/GFW, route pnpm/dsh through your proxy (HTTPS_PROXY / ALL_PROXY) or a GitHub mirror, then retry.
  2. Or isolate the bad dep. Update or remove the stale dsh-our-free-model git plugin first so its lockfile entry resolves cleanly, then add billion-context.
  3. Or prove the package is fine independently. A fresh profile with no git deps installs billion-context@latest without ever touching that tarball; npm i -g billion-context --prefix=~/.local works the same way.

If you paste the full Hub log (Notifications → the failed record → Copy) or your web profile's package.json, I can pin the exact offending entry at 100%.

One more thing: the hub's auto-filer attributing a failure to the target plugin when the failing resource is a sibling git dep is misleading (it'll burn maintainer time on non-bugs like this). I can relay that observation to the dsh-plugin-hub maintainers if you want — that's an external submission, so I'll only do it with your explicit go-ahead.

中文结构化八问

  1. 是什么问题:你执行 dsh plugin --profile web add billion-context@latest 失败;但真正下载超时的不是 billion-context,而是你 web profile 里另一个第三方插件 dsh-our-free-model(来自 ebony-vinyl)的 git 包。
  2. 什么原因导致的:你的 web profile 把 dsh-our-free-model 声明成了 git 依赖;每次 pnpm add 都会重新物化整个 profile、连带重拉所有 git 依赖,其中这个 codeload.github.com tarball 在你的网络下超时(重试 10s/1min 后放弃),导致整条 pnpm add 中止。billion-context 本身是干净的 npm 包(仅 zod/mysql2/jimp,无任何 git 依赖),不触发这次下载。
  3. 影响范围与严重程度:仅影响「profile 里有 git 插件 + 访问 GitHub 不稳定」的用户;有明确绕行(配代理 / 换干净 profile / 直接 npm 装)。非产品缺陷 → p2(环境性)。
  4. 怎么修复的:本仓代码无需改动。用户侧修法见上:修好 GitHub 出口(代理/镜像)、或先更新/移除失效的 dsh-our-free-model git 依赖、或用干净 profile / npm 直接装验证。
  5. 是否有回归问题:不涉及——未改任何代码。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

判定:非本仓库缺陷,属「环境/网络 + 第三方 git 依赖」问题;已给出可执行绕行与确认路径,等你回传完整日志或 profile 清单即可 100% 定位到具体条目。

@ranxianglei
ranxianglei force-pushed the 2026-10-31_ci-runner-queue-optimize branch from 2ea9f1a to b5f19f6 Compare October 31, 2026 11:19
…(+ dead loop fix)

Rebased onto post-fix master: overlapping product/doc repairs
(promptPack docs, PackInstallOptions export, npm.cmd spawn, family-gate
allowlist, CodexWindowSource, store adopt, refusal-remedy pin) already
landed upstream while this branch was in flight, so they are dropped here.
This PR is now CI-infrastructure only:

- release-trio + master-build + theorem lanes get concurrency groups
- default-6h timeout-minutes closed out on every release/advisory/pages/plugin gate
- ci-janitor: sweep loop actually reads its input now (previously the whole
  zombie-cleanup rule set was dead code), plus a backlog breaker
  (starvedQueuedMaxMinutes) that drains stale queued runs under depth overload.
@ranxianglei
ranxianglei force-pushed the 2026-10-31_ci-runner-queue-optimize branch from 638b9d6 to f571b60 Compare October 31, 2026 12:22
@ranxianglei
ranxianglei merged commit f08e298 into master Oct 31, 2026
102 of 103 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.961-master.f08e298 (rolling master tag).

npm install -g billion-context@0.0.961-master.f08e298  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit that referenced this pull request Oct 31, 2026
Three-way merge onto f08e298 (post-#5333 master). Four true conflicts,
all additive-union:
- web/page.ts: pck-notice-banner (mainline #5079) + autoreview-banner (#4634)
- web/client.ts: #5079 banner handler kept verbatim first, #4634 handler after
- server/handle.ts: #4634 local-answer early-return stays above the intent set;
  the #5135 ccTitleSidecar const supersedes the branch copy
- server/admin.ts: status payload carries both mainline fields (origin,
  resolverWedge) and the #4634 autoReviewBlock block
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant