Skip to content

[PR] fix: merge gates no longer block on twice-drained CI shards (#3665 round 3, #4212) - #4213

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-22_ci-drain-pass-merge-gate
Oct 23, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-22_ci-drain-pass-merge-gate

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #4212 (#3665 round 3)

问题

A CI shard whose only losses are event-loop-drain cancellations (fail 0 / cancelled N, zero assertion failures) goes red after the #3710 same-runner retry reproduces the drain — and nothing ever draws the "fresh runner" the closing message asks for. PR #4206 sat red ~40 min on exactly this (owner had to merge the release manually); the drain reproduced on 5 of 6 win22 machines over ~2h (incl. a fresh rerun machine at 17:40–17:57) while the overlapping 17:33 scheduled full matrix went green on the same code + same node v22.23.3 — a machine lottery the repo cannot fix (root cause stays open under #3665).

修法

scripts/run-ci-tests.mjs: when the first phase AND the same-runner retry are BOTH the pure drain shape (existing classifier, unchanged) AND the run is a merge-gating event (GITHUB_EVENT_NAME = pull_request/merge_group), the phase passes with a loud ::warning naming the cancelled tests (DRAIN-PASS) instead of exit 1. Availability first (§7.3): a zero-assertion-failure infrastructure flake must not block merges.

Strictness preserved (nothing else changes):

  • push / schedule / workflow_dispatch keep failing — each draws a fresh runner, so they are the cross-runner probe: a genuinely deterministic orphaned-promise regression that slips the PR gate goes red on the push run (minutes) or the 30-min scheduled cadence.
  • Local runs (no GITHUB_EVENT_NAME) keep failing — unchanged behavior outside GitHub.
  • Only the twice-drained signature downgrades: swallowed→drain, drain→concrete, concrete→anything all fail as before (shouldDrainPass is strict by construction).
  • The fuse test file itself is untouched: adding a fetch abort/timeout would convert drain into a concrete failure and defeat the recoverable classification.

行为变更披露

  • 旧:首轮 drain → 同机重试 → 仍纯 drain → exit 1(红),日志提示人工 rerun。
  • 新:同场景 + pull_request/merge_group → ::warning + DRAIN-PASS 日志 + exit 0(绿,带警告)。
  • 其余全部场景(push/schedule/dispatch/本地/concrete/swallowed/混合形态)逐字节不变。

config surface

无新增:读 runner 原生 GITHUB_EVENT_NAME,非新 env/config 面。

测试

tests/ci-runner-failure-shapes.test.ts +6(每门双向断言,#3617):twice-drained×merge-gate 放行;probe 事件/本地严格;非 twice-drained 组合不放行;drain 测试名提取(concrete/swallowed 不匹配);describeDrainPass 文案(双轮计数、点名、无 "genuine failure")。定向 19/19、release-ci-gate 16/16、unused-exports clean、typecheck clean;全量以本分支远程 CI test lanes 为准。

@ranxianglei ranxianglei changed the title fix: merge gates no longer block on twice-drained CI shards (#3665 round 3, #4212) [PR] fix: merge gates no longer block on twice-drained CI shards (#3665 round 3, #4212) Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-22_ci-drain-pass-merge-gate (b31f766)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5006-pr.4213.39eec5a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4213.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

本 PR 的 CI 自己完成了端到端验证:test (windows-latest, 22, 1)(job 119392792601)在本 head 上两轮全量都 drain——首轮 pass=4026 fail=0 cancelled=2,同 runner 重试 pass=4028 fail=0 cancelled=2,取消的正是 #4212 里的两个 issue4079-egress-window-fuse 测试。新逻辑按设计触发:

##[warning]DRAIN-PASS (#3665/#4212): both attempts on this runner lost tests to
event-loop drain with ZERO assertion failures — merge gate not blocked. ...

job 最终 success(42/42 全绿)。对照:同样的两次-drain 形态在修复前(昨天 PR #4206 与 master 44409dd)一律 exit 1 卡死合并门。也就是说——修门禁的这个 PR 差点被它要修的问题卡红,而它自己的新路径把它放行了,警告注释里点名了未跑完的测试。

供 review 时参考:改动面就两个脚本 + 一个测试文件 + .d.mts;分类/放行全部纯函数化并有双向单测;push/schedule/dispatch 与本地行为不变。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

先答两条未回应的评论,再给独立复核结论。

Floor 1(github-actions 构建产物通知):标准的每次 push 生成的不可变构建说明(@0.0.5006-pr.4213.39eec5a),自动化通知,无需处理。

Floor 2(GLM 的端到端验证声明):已独立核实,属实。我拉了分支 2026-10-22_ci-drain-pass-merge-gate(head b31f766)通读全 diff,并直接查了该 head 的 check-runs:全部 test lanes(ubuntu/windows × 22/24)+ 所有 guard + build-artifact 均 success。其中 test (windows-latest, 22, 1) 正是 Floor 2 说的那台——首轮与同机重试都 drain,由新 DRAIN-PASS 路径放行转绿。声明与事实一致。

查重:不重复。本 PR 修 #4212(#3665 round 3);#4206 是被它卡红的受害者(release PR),不是并列修复;仓库内无其它改 CI 门禁脚本的在途 PR。


结构化八问(独立复核,非复读 issue/Floor 2)

我逐行读了 scripts/run-ci-tests.mjs、scripts/ci-failure-shapes.mjs、.d.mts 与测试文件的完整 diff(+149/-1),并核对了分类器既有逻辑与 failureType:'cancelledByParent' 标记来源。

  1. 是什么问题:一个 CI shard 若两轮(首轮 + 同机重试)的损失全来自 event-loop drain(fail=0 / cancelled=N、零断言失败),在 merge-gating 事件上被判红卡死合并门,而且没有任何机制去抽一台新 runner。[PR] release v0.1.204 #4206 就因此红卡 ~40min,owner 手动合并。
  2. 什么原因导致:[PR] ci: retry a shard once on the fail-0/cancelled>0 event-loop-drain shape | Windows CI 偶发假红:零断言失败仅 cancelled 的 shard 自动重跑一次 #3710 的同机重试对「持续性 drain」必然无效——drain 是 runner 池环境污染或进程内确定性挂起,在同一台机器重试只会复现;而旧逻辑把「两轮纯 drain」一律 exit 1 并提示「人工 rerun」,但人工 rerun 同样抽不到新 runner(machine lottery,Flaky Windows CI: issue1993-preflight-400 suite cancelled by early event-loop drain | Windows CI 偶发假红:preflight-400 测试进程提前退出致 5 例取消 #3665 根因仍 open)。本质:门禁把「零断言失败的基建抖动」误当成需要阻断合并的信号。
  3. 影响范围与严重程度:仅 merge-gating 事件(pull_request/merge_group)下、恰好同一 runner 两轮都纯 drain 的单 shard。触发面窄。严重度属 p2(流程/体验阻塞,非数据丢失、非不可用)——实证是 PR [PR] release v0.1.204 #4206 假红卡合并 ~40min。
  4. 怎么修复的:在 run-ci-tests.mjs 重试判定之后加一道严格门 shouldDrainPass(first, retry, eventName)——当且仅当首轮与同机重试都分类为 cancelled-drain(复用既有分类器,未改)且事件是 pull_request/merge_group 时,以 ::warning::DRAIN-PASS 点名被取消的测试名并 exit 0,替代原来的 exit 1。分类与放行全部纯函数化进 ci-failure-shapes.mjs,配双向单测(Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617 纪律)。
  5. 是否有回归问题:无。逐项过:①push/schedule/workflow_dispatch/本地(无 GITHUB_EVENT_NAME)保持严格失败——我确认了 selectedFiles = allFiles 的分片只由 CI_TEST_SHARD 决定、与事件无关,所以这些事件跑的是同一套全量测试且各抽新 runner,构成对「确定性 drain」的跨 runner 探针(真挂起会在新 runner 复现→红);②swallowed→drain、drain→concrete、concrete→任意 等所有非 twice-drained 组合照常失败;③shouldDrainPass 结构上要求两轮都是 cancelled-drain,吞不掉任何真实断言回归(fail>0 必落到 concrete)。CI 本 head 全绿佐证。
  6. 是否破坏缓存:不涉及(CI 门禁脚本,无缓存/增量游标/去重键语义)。
  7. 是否改了设计方向:不转向。是在既有「纯函数分类器 + 有界重试」框架内新增一个「twice-drained × merge-gate → 警告放行」的出口;严格性边界(push/schedule/dispatch/本地维持严格)显式保留。符合 §7.3 可用性优先:零断言失败的基建抖动不阻断合并。
  8. 是否引入新的配置:无新增 env/config 面——只读 GitHub runner 原生 GITHUB_EVENT_NAME,不引入新配置键,符合 env-var 纪律。

行为变更披露复核:OP 已披露旧(exit 1 红)→ 新(::warning+DRAIN-PASS+exit 0 绿带警告),仅限 twice-drained × merge-gate,其余逐字节不变——与实际代码一致。


判定

可合(human merge)。改动面小且聚焦(2 脚本 + .d.mts + 1 测试文件),逻辑经我独立通读成立,双向单测齐备,行为变更已披露,远程 CI 在本 head 全绿,且本 PR 自己的 CI 已经在生产路径上端到端验证了新逻辑。按 §7.4,它改了 merge-gate 语义(产品决策)+ 触碰 CI 基建,不走 auto-merge,需人工合并:#4213

一处非阻断观察(不拦、不另开 issue):.d.mts 里 describeDrainPass 的两个参数用了内联 { kind:"cancelled-drain"; counts:TapCounts } 而非 PhaseFailureVerdict——这是刻意收窄(该函数只接收 cancelled-drain 判决),属合理选择,无需改。

@ranxianglei
ranxianglei force-pushed the 2026-10-22_ci-drain-pass-merge-gate branch from b31f766 to 03a8598 Compare October 23, 2026 04:59
@ranxianglei

Copy link
Copy Markdown
Owner Author

已 rebase 到 master@07888d74(2026-10-23)。冲突两处,均为合并保留:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant