Skip to content

[PR] fix: stop the three CI-shard flake families (#3769 pair + dsh attach-mode) | 修掉三个 CI 分片抖动家族 - #4058

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-20_ci-flake-hardening
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-20_ci-flake-hardening

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4049

查重: 不重复(本 PR 即 #4049 的修复;#4049 本身是三个同症状子缺陷的归并条目,来源链见该 issue)
kernel-change: 不涉及(未触碰 kernel/**)
config surface: 不涉及(无新增/改名/改语义的配置项、env、CLI flag)

What breaks what (one line each)

  1. tests/issue3769-pooled-bare-load.test.ts + tests/issue3769-upgrade-compat.test.ts (paired red across shards) — root cause proven, not guessed: when a fresh SessionStore cannot open its unified sidecar (index.db) at construction — any CANTOPEN under shared-runner load (fd limits, lock contention while a sibling store holds the dir, Windows Defender, ENOSPC) — UnifiedShadow.maybeOpen degrades to legacy-only with shadow === undefined, and the disk-fallback seed latch in SessionStore.loadEnvelope was gated on that live shadow object (if (!this.diskFilesSeeded && this.shadow)). Result: the envelope walk never seeds, every bare-id loadSync over a pooled-path record (openai/<host>_<hash>.json) returns null forever, and the two tests' "record hydrates" asserts fail. Deterministic local repro (make index.db a directory → sqlite "unable to open database file" → same silent-null signature as CI, incl. the missing CANTOPEN warn in CI logs being lost to async stderr on process exit). Fix: seed the walk whenever the store is enabled, independent of sidecar health — the envelope walk is the last-resort read path and must outlive sidecar failure. New regression test pins the degraded-store variant.

  2. tests/dsh-native.test.ts "bug: dsh-native — attach 目标瞬时不可达 → 回退 spawn 第二实例:routed baseURL 下工具/模型通道必然脑裂(工具 404 → preflight 502 → 会话不可自愈) #1365 … waits the target back" (cancelledByParent) — root cause proven by instrumentation: the test called respawn() while the apply-time verify flight was still in flight; singleFlight made it JOIN that stale flight, whose pre-death "attach anyway" verdict settled instantly — before the 100 ms re-listen timer. The test then did clearTimeout(upTimer); await up; — cancelling up's only resolver before awaiting it ⇒ dangling promise, drained event loop ⇒ node:test reports Promise resolution is still pending but the event loop has already resolved. Fast machines hit the join window deterministically; loaded CI lanes shift it either way (explains ok-elsewhere vs 1-of-6 shards red). Fix (test-only): settle the apply-time chains before simulating death (_settleNativeForTest()), and await up before clearing its timer. Sibling bug: dsh-native — attach 目标瞬时不可达 → 回退 spawn 第二实例:routed baseURL 下工具/模型通道必然脑裂(工具 404 → preflight 502 → 会话不可自愈) #1365 tests audited: their shapes are timing-deterministic, no change needed.

Behavior change disclosure (old → new)

Verification (targeted local; full suite = remote CI)

  • Deterministic repro script: CANTOPEN variant RED before the fix (3/3), GREEN after (3/3); delete-index variant green both sides (regression guard).
  • tests/issue3769-pooled-bare-load.test.ts 2/2 pass; tests/issue3769-upgrade-compat.test.ts 1/1 pass; tests/dsh-native.test.ts 62/62 pass, 0 cancelled (flake test duration 47 ms → ~290 ms: it now genuinely waits for recovery); flake test ×20 standalone runs all green.
  • Targeted storage-area suites A/B identical to clean master (5 pre-existing local-environment failures in tests/proxy-persist.test.ts reproduce byte-for-byte on stashed master too — environment artifact, CI-green).
  • npm run typecheck clean.

@ranxianglei ranxianglei changed the title fix: stop the three CI-shard flake families (#3769 pair + dsh attach-mode) | 修掉三个 CI 分片抖动家族 [PR] fix: stop the three CI-shard flake families (#3769 pair + dsh attach-mode) | 修掉三个 CI 分片抖动家族 Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_ci-flake-hardening (cc2a334)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4841-pr.4058.5fdf6c9

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4058.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Frequency update from PR #4057's CI (branch 2026-10-21_snapshot-sig-covered-bind, base bc4a3f8):

Net effect: since ~16:47Z today, every full-regression run (master included) has been red on this family — no PR can go green until it is fixed. Recommend treating this as a CI blocker; fastest unblock is either a deterministic repro under shard concurrency or running these two tests sequentially/off-shard.

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)
关闭:三个 flake 家族已分别落地合并——#3769 对 = #4077(存储模式钉)+ #4087(win EBUSY);dsh abort 300ms = #4100(轮询);其余时序尾巴 = #4097(同步 flush)。master CI 已绿(9a97169 起)。本 PR 重复,感谢!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two #3769 storage regression tests flake across CI shards on master and unrelated PRs

1 participant