Repository navigation
[PR] fix: stop the three CI-shard flake families (#3769 pair + dsh attach-mode) | 修掉三个 CI 分片抖动家族 - #4058
Closed
ranxianglei wants to merge 2 commits into
Closed
[PR] fix: stop the three CI-shard flake families (#3769 pair + dsh attach-mode) | 修掉三个 CI 分片抖动家族#4058ranxianglei wants to merge 2 commits into
ranxianglei wants to merge 2 commits into
Conversation
added 2 commits
October 21, 2026 05:26
Contributor
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4841-pr.4058.5fdf6c9This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr4058.tgz
npm install -g packageInstall instructions are refreshed on each push. |
This was referenced Oct 20, 2026
Owner
Author
|
(ai: vllm-qwen/qwen3.8-27b) Frequency update from PR #4057's CI (branch 2026-10-21_snapshot-sig-covered-bind, base bc4a3f8):
Net effect: since ~16:47Z today, every full-regression run (master included) has been red on this family — no PR can go green until it is fixed. Recommend treating this as a CI blocker; fastest unblock is either a deterministic repro under shard concurrency or running these two tests sequentially/off-shard. |
This was referenced Oct 20, 2026
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4049
查重: 不重复(本 PR 即 #4049 的修复;#4049 本身是三个同症状子缺陷的归并条目,来源链见该 issue)
kernel-change: 不涉及(未触碰 kernel/**)
config surface: 不涉及(无新增/改名/改语义的配置项、env、CLI flag)
What breaks what (one line each)
tests/issue3769-pooled-bare-load.test.ts+tests/issue3769-upgrade-compat.test.ts(paired red across shards) — root cause proven, not guessed: when a freshSessionStorecannot open its unified sidecar (index.db) at construction — any CANTOPEN under shared-runner load (fd limits, lock contention while a sibling store holds the dir, Windows Defender, ENOSPC) —UnifiedShadow.maybeOpendegrades to legacy-only withshadow === undefined, and the disk-fallback seed latch inSessionStore.loadEnvelopewas gated on that live shadow object (if (!this.diskFilesSeeded && this.shadow)). Result: the envelope walk never seeds, every bare-idloadSyncover a pooled-path record (openai/<host>_<hash>.json) returns null forever, and the two tests' "record hydrates" asserts fail. Deterministic local repro (makeindex.dba directory → sqlite "unable to open database file" → same silent-null signature as CI, incl. the missing CANTOPEN warn in CI logs being lost to async stderr on process exit). Fix: seed the walk whenever the store is enabled, independent of sidecar health — the envelope walk is the last-resort read path and must outlive sidecar failure. New regression test pins the degraded-store variant.tests/dsh-native.test.ts"bug: dsh-native — attach 目标瞬时不可达 → 回退 spawn 第二实例:routed baseURL 下工具/模型通道必然脑裂(工具 404 → preflight 502 → 会话不可自愈) #1365 … waits the target back" (cancelledByParent) — root cause proven by instrumentation: the test calledrespawn()while the apply-time verify flight was still in flight;singleFlightmade it JOIN that stale flight, whose pre-death "attach anyway" verdict settled instantly — before the 100 ms re-listen timer. The test then didclearTimeout(upTimer); await up;— cancellingup's only resolver before awaiting it ⇒ dangling promise, drained event loop ⇒ node:test reportsPromise resolution is still pending but the event loop has already resolved. Fast machines hit the join window deterministically; loaded CI lanes shift it either way (explains ok-elsewhere vs 1-of-6 shards red). Fix (test-only): settle the apply-time chains before simulating death (_settleNativeForTest()), andawait upbefore clearing its timer. Sibling bug: dsh-native — attach 目标瞬时不可达 → 回退 spawn 第二实例:routed baseURL 下工具/模型通道必然脑裂(工具 404 → preflight 502 → 会话不可自愈) #1365 tests audited: their shapes are timing-deterministic, no change needed.Behavior change disclosure (old → new)
loadSyncover pooled-path records — old: permanent silent null (session looks nonexistent until a full boot walk); new: hydrated from the on-disk envelope via a one-shot bounded directory walk. This IS the fix; no other read path changes. Enabled+healthy stores behave exactly as before (the gate only adds seeding whereshadowis undefined).respawn()joining an in-flight verify flight can inherit a verdict computed before the observed runtime death; production self-heals on the next retry tick (Windows/dsh 宿主重启:代理就绪超出 SPAWN_WAIT_MS(20s) 导致 bootstrap 失败,本进程内无重试 → /bili/origin 恒为 null、设置面板永久"未绑定"、整场会话直连(压缩失效) #2187 cadence). Changing single-flight join semantics is load-bearing attach-recovery territory — left to a dedicated decision if you want it addressed.Verification (targeted local; full suite = remote CI)
tests/issue3769-pooled-bare-load.test.ts2/2 pass;tests/issue3769-upgrade-compat.test.ts1/1 pass;tests/dsh-native.test.ts62/62 pass, 0 cancelled (flake test duration 47 ms → ~290 ms: it now genuinely waits for recovery); flake test ×20 standalone runs all green.tests/proxy-persist.test.tsreproduce byte-for-byte on stashed master too — environment artifact, CI-green).npm run typecheckclean.