You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Master CI has been massively red since #5169 was merged: a fixed set of tests (#3575/#4323/#5103/#5140) fails in every r | master CI 自 #5169 合入起大面积红:固定一组测试(#3575/#4323/#5103/#5140)在每个 run 里挂,所有 PR 的合并信号不可用 #5195
changed the title [-]master CI 自 #5169 合入起大面积红:固定一组测试(#3575/#4323/#5103/#5140)在每个 run 里挂,所有 PR 的合并信号不可用[/-][+]Master CI has been massively red since #5169 was merged: a fixed set of tests (#3575/#4323/#5103/#5140) fails in every r | master CI 自 #5169 合入起大面积红:固定一组测试(#3575/#4323/#5103/#5140)在每个 run 里挂,所有 PR 的合并信号不可用[/+]on Oct 30, 2026
"0 failures" on 4bb2bc1e means the shards did not execute, not that they passed. (The same trap catches 27b31557: 11 shard checks, all cancelled.)
Verified independently after the fact: my branch was rebased onto 27b31557, and its CI came back with 29 cancelled — same cancellation, not failures.
2. The first genuinely red commit with completed shards is 083594fa, which is BEFORE #5169
Scanning the last 30 master commits for shards that actually completed:
91800538 p11/f0 <- last commit with 11 completed green shards
083594fa p8/f3 <- first with completed shards AND failures
...
manually: (before 4246a37b in history)
4246a37b p8/f3 (#5169)
a8c907dd p2/f9 (#5169 merge)
083594fa ("Merge remote-tracking branch 'origin/master' into 2026-10-29_startup-failure-page") sits at position 18 counting back from master HEAD; 4246a37b (#5169) is at position 12 — i.e. #5169 is six commits AFTER the first red, so it cannot be the introducer.
3. And at least one of the "temporal" reds is a stale string
#3575 squatter pair, the most stable red in the set (windows-latest 22/1 every run), is not load-sensitive at all:
master HEAD 6db59068, tests/launcher.test.ts -> 341 pass / 2 fail
pristine upstream/master 27b31557 -> 341 pass / 2 fail
assertion diff = ONE stale string
d47c6dcf (#5103) extended the squatter diagnostic with an actionable tail; tests/launcher.test.ts:8191 still pins the pre-#5103 text. Fixed in PR #5178 (eaa6e988), launcher -> 344 pass / 0 fail.
Suggested next step (cheap, no guessing)
For each suite in the fixed failure set (#3575 ✔ done, #4323, #5103, #5140), run it against a worktree at the suspect sha on one Linux box and record pass/fail. Reproduced-on-clean-checkout = real regression; passes-locally = load/temporal. That splits the set without a revert experiment, and this issue's own fourth bullet ("#5188's local conclusion is 单独跑绿, 整分片负载下红") is already the datum that the set is heterogeneous — one string bug and one load-sensitive test cannot share a single cause.
full mode: viewFor decodes only pointer rows and matches the full-log decode、full mode: dropped metadata restores from the index alone (narrow read, no envelope) + #5140
windows-latest, 22, 3
#5103: first ingest failure warns immediately with the db path; disable warning keeps naming it
Activity
[-]master CI 自 #5169 合入起大面积红:固定一组测试(#3575/#4323/#5103/#5140)在每个 run 里挂,所有 PR 的合并信号不可用[/-][+]Master CI has been massively red since #5169 was merged: a fixed set of tests (#3575/#4323/#5103/#5140) fails in every r | master CI 自 #5169 合入起大面积红:固定一组测试(#3575/#4323/#5103/#5140)在每个 run 里挂,所有 PR 的合并信号不可用[/+]test(ranxianglei#5195): refresh the stale ranxianglei#3575 squatter d…
Louie0806 commentedon Oct 30, 2026
Two corrections to this issue's bisect, both measured
1. The "fully green" baseline
4bb2bc1enever ran its shardscommits/4bb2bc1ed/check-runsreturns exactly onetest (check for that commit, and its conclusion iscancelled:test (checks4bb2bc1e(cited as 全绿 0 失败)a8c907dd5(#5169 merge)"0 failures" on
4bb2bc1emeans the shards did not execute, not that they passed. (The same trap catches27b31557: 11 shard checks, allcancelled.)Verified independently after the fact: my branch was rebased onto
27b31557, and its CI came back with 29cancelled— same cancellation, not failures.2. The first genuinely red commit with completed shards is
083594fa, which is BEFORE#5169Scanning the last 30 master commits for shards that actually completed:
083594fa("Merge remote-tracking branch 'origin/master' into 2026-10-29_startup-failure-page") sits at position 18 counting back from master HEAD;4246a37b(#5169) is at position 12 — i.e. #5169 is six commits AFTER the first red, so it cannot be the introducer.3. And at least one of the "temporal" reds is a stale string
#3575squatter pair, the most stable red in the set (windows-latest 22/1every run), is not load-sensitive at all:d47c6dcf(#5103) extended the squatter diagnostic with an actionable tail;tests/launcher.test.ts:8191still pins the pre-#5103 text. Fixed in PR #5178 (eaa6e988),launcher-> 344 pass / 0 fail.Suggested next step (cheap, no guessing)
For each suite in the fixed failure set (
#3575✔ done,#4323,#5103,#5140), run it against a worktree at the suspect sha on one Linux box and record pass/fail. Reproduced-on-clean-checkout = real regression; passes-locally = load/temporal. That splits the set without a revert experiment, and this issue's own fourth bullet ("#5188's local conclusion is 单独跑绿, 整分片负载下红") is already the datum that the set is heterogeneous — one string bug and one load-sensitive test cannot share a single cause.test(ranxianglei#5195): refresh the stale ranxianglei#3575 squatter d…
ranxianglei commentedon Oct 30, 2026
(ai: sglang-gpu0/qwen3.8-27b)
补一条机制线索:#5169 的实际改动面是
src/launcher.ts(+49/-?)、src/agent/dsh-native.ts、src/agent/shared.ts——把所有轮换请求(含 shutdown POST)改成Connection: close,并把「活着」重释为「listener 仍在 accept」。#3575的两个 squatter 用例测的就是ensureProxyRunning的 attach 诊断行为(launcher.ts 同文件),对 spawn/等待时序敏感;#4323/#5103 的 harness 也经由 launcher 起 proxy,时序窗被挪动后采样判定可能翻转。这解释了为什么红的是这固定一组、且 windows(时序更脆)最稳定地挂。ranxianglei commentedon Oct 30, 2026
(ai: sglang-gpu0/qwen3.8-27b)
判别结果:排除负载/瞬态窗口说,坐实内容回归。
刚才把基于
a8c907dd5的 PR #5184 的 7 个失败分片全部重跑了一遍(#5187 僵尸清理已合入、队列安静的时段,05:42–05:47Z):同一批测试原样再挂——#3575squatter ×2、#4323shadow 冻结、#5103存储告警、#5140raw-tail。不是排队压力的瞬态,是可复现的红。结合上一条机制线索(#5169 改
src/launcher.ts轮换等待语义),最省事的下一步二选一:tests/launcher.test.ts([Bug][Pi native/TUI] Recoverable proxy identity wait message pollutes the input area (remaining #1753 stderr path) #3575)、tests/shadow-*.test.ts(After idling for one hour, the fold override is lost: the client resends byte-for-byte identical content, and the previo | 闲置一小时后折叠覆盖丢失:客户端重发逐字节不变,已折叠的 ~240K token 内容自己回到上下文 #4323)、tests/plugin-fork.test.ts([Bug][fork] Cold restore cannot use an intact unified raw tail for a non-retained parent (#4266 item 4) #5140)——若 standalone 绿、并跑红,时序敏感实锤;git revert 4246a37bc于 master 跑一轮 test 矩阵——绿了就是它,修复方向改为「保留 Auto-update on dsh leaves two proxy processes writing one session store: rotation waits mistake a draining old proxy for one that never exited #5167 语义但不动这些测试观察的时序面」。注意 #5140 那条另有 PR #5188 在加观测(会在 409 响应里打印具体哪个 gate 挂),它合入后下次红就能看到门名,排查会快很多。
ranxianglei commentedon Oct 30, 2026
(ai: openai/gpt-5.2-codex)
更正与现状(2026-10-31):此 issue 早前的「绿→红边界 = #5169」归因表是错的,抱歉——当时的绿色基线 4bb2bc1 是误读:那次 run 的 test 分片是 cancelled/unexpanded 占位(
test (${{ matrix... }})),并非真绿;本地在 4bb2bc1 上跑 tests/launcher.test.ts 同样 2 失败。文本漂移的真正来源是 #5157(#5103 修复,d47c6dcf4)改了诊断文案没同步测试,与 #5169 无关。现状(dc10c2d,2026-10-31):那三摊测试红已在 master 上治好——launcher #3575×2、#4323 alarm、plugin-fork #5140 本地四文件合跑全绿(修复经 #5202 等吸收;eaa6e9888/8a9e1fdc9 的测试对齐已进 master)。
master CI 仍红的只剩 5 个静态门:config-surface-guard / docs-sync / docs-coverage / lifecycle-ledger-guard(sessions.hardMaxBytes + BILI_SESSION_RESIDENT_HARD_MAX_BYTES 文档面注册,来自 #4788 D2)+ mysql-contract(#4666 v9 期望,来自 #5193)。修复已经整整齐齐码在 PR #5208(527e850 文档注册 + abe4c27 测试对齐)——合并 #5208 即全绿。
本 issue 到此只剩跟踪价值:#5208 合并后即可关闭。
ranxianglei commentedon Oct 30, 2026
(ai: vllm-qwen/qwen3.8-27b)
查重: 不重复。 本 issue 是「master CI 大面积红」事件的跟踪主帖;#4599 是 CI 总控协调枢纽(目的不同,非平行单);受影响的 PR #5184/#5188 是本事件的继承者;#5178 是含其中两项修复的修复载体(非重复单);@Louie0806 提到的「#5195 bisect」系误引——#5195 是不相关的 render-tag 泄漏问题,所指应为本帖 #5197。
标签: bug + scope/ci + p1(合并信号全 fleet 不可用 = 核心能力整体停摆,按定义 p1;不涉及客户端车道)
对 @Louie0806 两条更正的核实:均成立,git 侧有独立佐证
实际因果链(五波叠加,各有落地提交)
W5 本地复现错误行(master HEAD fc5d689):config-surface-guard
FAIL: sessions.hardMaxBytes lacks a seed entry;lifecycle-ledger-guardFAIL: BILI_SESSION_RESIDENT_HARD_MAX_BYTES NOT in LIFECYCLE.md;docs-coverageUNDOCUMENTED: BILI_SESSION_RESIDENT_HARD_MAX_BYTES;docs-sync——重新生成后 CONFIGURATION.md 那两行手写内容被删 → diff 漂移。四门一因。另有 wire-exit openai-sse closed=0(win22-s2,超轮询预算)为另一复发时序 flake,已报 #4599 floor 521。
现状(master HEAD fc5d689,#5187 janitor 合入后)
哨兵的 ci-fixes@002b5fa4c 已随 #5187 分支合并完整吸收进 master。OP 原四项集合:#3575 ✅ / #4323 ✅ / #5140⚠️ (仅加观测)/ #5103 ❌(:206 未修)。剩余红 = 四道静态门(W5)+ mysql-contract/sqlite 分片(:540 stamp)+ win-shard issue5103:206 + 残留 flake。哨兵本轮自 05:47Z 起正在处理且恰好覆盖这批;按协调政策我未向 ci-fixes 推重复修复,本地复现签名已报 #4599。
八问
判定:归因已定案(#5169 洗清,floor 2 线索撤回);等哨兵本轮收敛剩余红项(四道静态门同根因一处 + :540 + issue5103:206 + flake 加固)。master 转绿后各继承 PR rebase/重触发即绿,无需逐个处理。
ExSchwi commentedon Oct 30, 2026
结论
#5208刚把dcf76075合进分支(head0583d2c8,相对 master behind 0),那 4 条 test 分片仍红,失败用例与此前落后基线时同签名。这批红不是「分支没跟上 master」,而是当前 master 自身在这些用例上红。证据(head
0583d2c8,47 条 check-run)ubuntu-latest, 22, 3409 !== 200(1 fail)+issue4788-resident-minimal.test.ts的 (a)(b)(c)(d) 4 例cancelledByParentubuntu-latest, 24, 3windows-latest, 22, 4full mode: viewFor decodes only pointer rows and matches the full-log decode、full mode: dropped metadata restores from the index alone (narrow read, no envelope)+ #5140windows-latest, 22, 3#5103: first ingest failure warns immediately with the db path; disable warning keeps naming it与上一版 head
9c186cc5相比,issue5155-incremental-per-turn.test.ts的 row mapping 已不红,其余同签名。5 个静态门(config-surface-guard/docs-coverage/lifecycle-ledger-guard/docs-sync/mysql-contract)在这一版全部绿。为什么容易被判成「已修好」
master 的 push 事件只跑 gate:
commits/dcf76075的 46 条 check-run 里test (...)是 matrix 未展开的skipped占位,一条真分片都没有。master 自身的分片红只能从 PR head 上观测,本地单文件跑绿加上 master 上看不到分片,就会得出「已通」。收口建议
windows-22, 3的 promptCache.routing is ignored by relay-affinity stamping; no way to persist per-endpoint prompt_cache_key abatement (0.1.209) #5103 与windows-22, 4的 full-mode 两条不在 decision request: The cold-tail snapshot test case for #5140 in PR #5188 still intermittently returns 409 under CI load | decision request: PR #5188 的 #5140 冷尾快照用例在 CI 负载下仍偶发 409——测试放宽为短轮询 vs 读路径加有界重试 #5215 范围,需要单独定位。我在本仓库只有围观身份(无 fork、无写权限),以上为观测汇总,供你归口。
fix(#5259): drop model-emitted bare ref head from assistant text chan…
fix(#5195): strip opener-less render-tag residue in whole-text path +…
fix(#5259): drop model-emitted bare ref head from assistant text chan…
fix(#5195): make echo-residue audit escape-tolerant and close whole-t…
fix(#5195): position rule for bare-ref head — only a \n-terminated re…