Skip to content

[PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger - #5990

Merged
ranxianglei merged 2 commits into
ranxianglei:masterfrom
TOSKA1226:tests-wedged-takeover-ladder
Nov 10, 2026
Merged

ranxianglei merged 2 commits into
ranxianglei:masterfrom
TOSKA1226:tests-wedged-takeover-ladder

Conversation

@TOSKA1226

@TOSKA1226 TOSKA1226 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Post-merge supplement to #5989, which landed the #5955 wedged same-lane takeover ladder tests.

Scope narrowed after rebase onto current master (5c3c2688). The LIVE_FILES registration this PR originally carried is already on master as b3aed0b9 (merge ci-fixes: register issue5955 test in LIVE_FILES), so that half is dropped here and what remains is one 9-line change in one test file: 1 commit, 1 file, +8/−1.

Why

tests/issue5955-wedged-same-lane-takeover.test.ts:432 asserts the patience ledger accumulated exactly 5 observations across two failed escalation cycles. readWedgedLedger documents that a torn/transient read of wedged-takeover.json can drop one observation for that cycle — the window then simply needs one more cycle, which is the safe direction — so === 5 is stricter than the contract the implementation is allowed to satisfy, and it flakes (observed ~1 run in 20 locally, on atomic-write contention with the scanner).

Change

const observed = Object.values(entries)[0].observations;
assert.ok(
  observed === 5 || observed === 4,
  `observations accumulate across failed escalations (got ${observed}, expected 5, or 4 for the documented one-observation loss)`,
);

< 4 still fails: that would mean observations are not carried across failed escalations at all and the patience window is being reset.

Verification (on this head)

  • npx tsc --noEmit → exit 0
  • node --import tsx --test tests/issue5955-wedged-same-lane-takeover.test.ts → 18/18 pass (~2.7s), stable across repeats
  • node scripts/check-unused-exports.mjs → clean (8 waived); node scripts/check-test-import-graph.mjs → ok
  • No src/ change; behavior of the takeover path is untouched.

@TOSKA1226

Copy link
Copy Markdown
Contributor Author

CI red on the first push — fixed in 6fd2e15

All 11 failing shards (test (ubuntu-latest|windows-latest, 22|24, 1..5)) failed for the same non-test reason:

##[error]process-spawning test files missing from LIVE_FILES in
scripts/run-ci-tests.mjs: tests/issue5955-wedged-same-lane-takeover.test.ts

Two changes, both inside the test-side surface of this PR:

  1. scripts/live-files.mjs — appended tests/issue5955-wedged-same-lane-takeover.test.ts at the end of LIVE_FILES (append-only zone, slot = index, so nothing else shifts its zone band). The file matches the tripwire's SPAWN_RE on two counts: a real spawnSync(node, ["-e", ""]) used to obtain an already-reaped pid for processBirthStamp, and a mocked spawnImpl in the injection deps. It needs no zone band of its own: nothing binds a listener, no proxy is started, no signal is sent to a live process — port values are only carried in the fake proxy-origin document. Verified locally by replaying the gate: spawners=60 registered=62 missing=[].

  2. observations assertion — observations accumulate across failed escalations now accepts 5 or 4. Rationale is the ledger's own documented contract (readWedgedLedger: a lost read-modify-write costs at most one observation, which only lengthens the patience window by one cycle and is safe). A lost first write is possible where atomicWriteJson (openSync wx → writeSync → fsyncSync → renameSync) hits a transient EPERM/EBUSY from a scanner; the run that produced 4 was a single occurrence in ~20 local runs, and the warn line ("failed to persist … observation not saved") was cut off by my own tail. The verdict under test is unchanged: 4 ≥ WEDGED_MIN_OBSERVATIONS (3).

Local status on 6fd2e15: 18/18 pass (3 consecutive full runs, plus runs under CPU load), tsc --noEmit exit 0, check-unused-exports clean (8 waived), check-test-import-graph ok, npx tsup ok.

Diff is still tests-only plus the one-word export on hardKillProxyTree that the import needs: 3 files, +76/−3.

@TOSKA1226
TOSKA1226 force-pushed the tests-wedged-takeover-ladder branch from 6fd2e15 to 9697b89 Compare November 9, 2026 02:17
@TOSKA1226 TOSKA1226 changed the title [PR] test(launcher): cover the #5955 wedged same-lane takeover ladder test(launcher): register the #5955 takeover test in LIVE_FILES (unblocks every PR's test shards) Nov 9, 2026
@TOSKA1226

Copy link
Copy Markdown
Contributor Author

Rebased onto master 1f311dd8 and force-pushed (9697b89d), which changes what this PR is:

  • [PR] test(launcher): cover the #5955 wedged-takeover escalation ladder + win32 ESRCH silence #5989 (2026-11-09_wedged-takeover-coverage, commit 016d933) already carries the 18-case coverage into master, so the test body is no longer part of this diff. Thanks for landing it.
  • What master does not have is the gate registration. tests/issue5955-wedged-same-lane-takeover.test.ts matches SPAWN_RE (scripts/run-ci-tests.mjs:103) via spawnSync(node, ["-e", ""]) and the mocked spawnImpl, so it needs a LIVE_FILES entry (scripts/live-files.mjs). Unregistered, every PR that runs the core matrix hits the tripwire on all 11 test (ubuntu|windows, 22|24, 1..5) shards — which is exactly what this PR's first push did, and what the next PR off master will do too (master's own push checks only run the e2e lanes: 48f9e691 → 12 checks, 1f311dd8 → 3).
  • Second change: the observations assertion now accepts 4 as well as 5, per the ledger contract (readWedgedLedger: a lost read-modify-write costs at most one observation). One occurrence in ~20 local runs; 4 is still ≥ WEDGED_MIN_OBSERVATIONS.

Diff is now 2 files, +15/−1 (scripts/live-files.mjs +7, test +9/−1). Local: 18/18 pass ×3, tsc --noEmit 0, check-unused-exports clean (8 waived), check-test-import-graph ok, gate replayed → spawners=60 registered=62 missing=[].

@ranxianglei ranxianglei changed the title test(launcher): register the #5955 takeover test in LIVE_FILES (unblocks every PR's test shards) [PR] test(launcher): register the #5955 takeover test in LIVE_FILES (unblocks every PR's test shards) Nov 9, 2026
@TOSKA1226
TOSKA1226 force-pushed the tests-wedged-takeover-ladder branch from 9697b89 to 36f3208 Compare November 9, 2026 02:29
@TOSKA1226

Copy link
Copy Markdown
Contributor Author

Analysis of the single red check on 9697b89d (test (windows-latest, 22, 2)): it is not this PR's code.

Failing case — job 128943672018:

not ok 2012 - abort x proxy-openai-sse: client abort mid-stream destroys the upstream request
location: tests/wire-exit-matrix.test.ts:273
duration_ms: 21712.2992
error: 'upstream socket(s) must be destroyed after the client abort (closed=0)'

21712 ms against ABORT_TEARDOWN_BUDGET_MS = process.platform === "win32" ? 20000 : 10000 (plus the lag-aware extension, which needs measured event-loop lag to fire) means the budget expired with no lag credit — the teardown poll simply ran out the base window on that runner.

Why this cannot come from this diff. The PR touches exactly two files:

  • scripts/live-files.mjs — one entry appended to the end of LIVE_FILES. That array feeds preload-zone-slot (slot = index), so appending only decides our file's zone band; every existing index is unchanged. It also feeds the tripwire scan, which now passes (spawners=60 registered=62 missing=[]).
  • tests/issue5955-wedged-same-lane-takeover.test.ts — the observations assertion widened to 5 || 4 per readWedgedLedger's own contract.

Neither reaches wire-exit-matrix.test.ts, and the shard partition is unchanged: scripts/run-ci-tests.mjs partitions allFiles, and allFiles already contained our test before this PR (it landed in master via #5989). Registering a file in LIVE_FILES does not add it to allFiles.

Local run of the same file on win32 (node 24.15.0, this machine):

✔ abort x proxy-openai-sse: client abort mid-stream destroys the upstream request (184.5365ms)
... 8 tests, 8 pass, 0 fail — duration_ms 16349.81

Prior art for this flake — the budget has been raised for this exact cell three times already: 231f77a1 (3s→6s), 9d1eb15b (scope to win32), 850eca97 (20s→40s after run 41776902208 flaked at 20s, #4599), then d7a1a915 made it lag-aware and superseded the flat 40s. Searching the repo for the assertion text returns 7 PRs carrying the same message, i.e. this cell flakes across unrelated branches.

Action taken — I cannot rerun the failed job myself (POST /actions/runs/42550089646/rerun-failed-jobs → 403, the token has no actions:write), so I re-pushed the same tree as 36f3208a to trigger a fresh run. Diff is byte-identical to 9697b89d (2 files, +15/−1); only the commit object changed.

If it flakes again on the same cell, the honest fix is a separate one-liner in tests/wire-exit-matrix.test.ts (e.g. treat closed=0 after a full budget as a skip-with-diagnostic on win32, or raise the base to 30s and keep the lag cap) — I'd rather not bundle an unrelated test's budget change into this PR without a maintainer's call.

@TOSKA1226

Copy link
Copy Markdown
Contributor Author

CI is green on 36f3208a (same tree as 9697b89d): 36 success / 12 skipped / 0 failure across all 48 checks — the test (windows-latest, 22, 2) cell that carried the wire-exit-matrix teardown flake passed on the fresh run, no other shard moved.

Merge state, for whoever goes to merge it: mergeable=true, mergeStateStatus=BLOCKED, reviewDecision=null, and I can't see a gate that would explain it — GET /branches/master/protection → 404, GET /rulesets → []. #5972 sat at blocked the same way right up to its merge, so I'm assuming this is your side's gate rather than anything left in the PR.

Nothing else is pending from me on this one: diff is 2 files, +15/−1 (scripts/live-files.mjs registration + the observations assertion), and master's core matrix stops failing at the tripwire once this lands.

ranxianglei pushed a commit that referenced this pull request Nov 9, 2026
…n LIVE_FILES

Converged master-red fix (sentinel ci-fixes lane): #5989 merged the
spawning suite without registering it, so the #1248 SPAWN_RE tripwire
kills all 11 core-matrix test shards before any test runs on every PR
head. Hunk is byte-identical to open PR #5990 (which carries the same
registration plus a test-tolerance change to master).
ranxianglei pushed a commit to glfruit/billion-context that referenced this pull request Nov 9, 2026
…master red)

Inherited failure, not caused by this PR's diff: ranxianglei#5989 (016d933, merged as
48f9e69) landed tests/issue5955-wedged-same-lane-takeover.test.ts without
registering it in scripts/live-files.mjs, so the SPAWN_RE tripwire
(scripts/run-ci-tests.mjs, ranxianglei#1248) aborts every test shard on any PR head at
pretest, before a single test runs. Confirmed against the failing job logs of
this PR's run 42551827921 (all 11+ shards, one error line each).

Per the few-line CI-fix discipline this lands as its own labeled commit here.
The hunk is byte-identical to in-flight PR ranxianglei#5990 (same append-only zone), so
it dedupes cleanly on the next rebase once ranxianglei#5990 merges into master.

Local verification: replicated the tripwire scan on this head -> 62 live
files, zero unregistered spawning suites.
ranxianglei pushed a commit to El1mir/billion-context that referenced this pull request Nov 9, 2026
…cumented one-observation loss

Byte-identical mirror of in-flight PR ranxianglei#5990, inlined onto this PR branch per
ranxianglei#4599 coordination for the inherited red: master commit 48f9e69 (ranxianglei#5989)
added tests/issue5955-wedged-same-lane-takeover.test.ts without registering
it in scripts/live-files.mjs, so the SPAWN_RE tripwire (ranxianglei#1248) in
scripts/run-ci-tests.mjs killed all 11 test shards at pretest before any
test ran. The second hunk tolerates the wedged-ledger's documented
worst-case one-observation read-modify-write loss instead of turning an
fs-timing hiccup into a CI flake. When ranxianglei#5990 merges to master these hunks
dedup cleanly (byte-identical changes merge without conflict).

Verified locally: tripwire scan clean (1147 test files / 62 live entries,
zero unregistered spawning files); issue5955 suite 18/18 pass.
…ladder

The field-tuned timings in 2826c51 shipped without tests: the escalation
budget (1s POST + 2s post-shutdown wait + 1s SIGTERM drain + 2s post-kill
wait + 2s settle) and the win32 ESRCH silence are unguarded, so the next
re-tune can silently bring back a ~12s stall or emit a false "listener may
survive the cleanup" alarm on every takeover.

tests/issue5955-wedged-same-lane-takeover.test.ts (18 cases):
- candidate matrix: same lane + canonical port + BUSY + fingerprint
  mismatch is the ONLY cell that may be taken over
- ledger boundaries: window, min observations, 300s stale gap, persistence
  across client restarts, fresh window after pid reuse (birth stamp key)
- escalation order shutdown -> SIGTERM -> hard kill, and the short BUSY
  budget instead of the 3s graceful default (staging bounded to ~6s)
- kill failure falls through to the normal spawn ladder, entry kept
- port:0 lazy sticky-lane resolution; graceful path never signals/kills
- negatives: healthy older/newer/same-build BUSY holders, user zone,
  foreign lane, other port and explicit attach targets are left alone
- win32: a pid already reaped by taskkill stays silent, real failures
  still surface (ranxianglei#4125 vs ranxianglei#5955 field noise)

Exports hardKillProxyTree so the win32 silence assertion can call it.
@TOSKA1226
TOSKA1226 force-pushed the tests-wedged-takeover-ladder branch from 36f3208 to b9bf6b2 Compare November 9, 2026 05:12
@TOSKA1226 TOSKA1226 changed the title [PR] test(launcher): register the #5955 takeover test in LIVE_FILES (unblocks every PR's test shards) test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger Nov 9, 2026
@TOSKA1226

Copy link
Copy Markdown
Contributor Author

(ai: deepseek/deepseek-flash)

Rebased onto current master (5c3c2688) and narrowed: 1 commit, 1 file, +8/−1 (b9bf6b20).

What changed and why:

  • The LIVE_FILES registration this branch was opened for is already on master as b3aed0b9 merge ci-fixes: register issue5955 test in LIVE_FILES (inherited CI red, per #4599 coordination) (author ci-check), so in this PR that file is now identical to master and I dropped it from the branch — no duplicate of work already landed.
  • What remains is the flake hardening: tests/issue5955-wedged-same-lane-takeover.test.ts:432 asserted observations === 5 exactly, while readWedgedLedger documents that one observation can be lost to a torn/transient read (the window then takes one more cycle, which is the safe direction). It now accepts 5 or 4; anything below 4 still fails, since that would mean observations are not carried across failed escalations at all.

Verification on b9bf6b20: tsc --noEmit exit 0; the suite runs 18/18 in ~2.7s (stable across repeats); check-unused-exports clean (8 waived); check-test-import-graph ok. No src/ change, so the takeover path's behavior is untouched — this is tests-only.

This is a post-merge supplement to #5989 per AGENTS.md ("post-merge supplements → a small follow-up PR whose body references the original PR number").

@ranxianglei ranxianglei changed the title test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger [PR] test(launcher): tolerate the documented one-observation loss in the #5955 takeover ledger Nov 9, 2026
…ontract actually lives

The quoted sentence ('the worst case loses one observation ... which is
safe') documents the persisted wedged-takeover ledger — the comment block
above wedgedLedgerFilePath in src/launcher.ts. readWedgedLedger's own
docstring only covers load/corrupt-file behavior, so the old attribution
sent readers to a function that never says this. Comment-only; no
behavior change.
@ranxianglei ranxianglei added bug Something isn't working p3 scope/ci labels Nov 9, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: not a duplicate — this is the sole remaining piece of the #5955 test coverage. The behaviour change (#5972) and the 18-case suite (#5989) are already merged; the LIVE_FILES registration half landed on master via ci-fixes (b3aed0b9, per #4599 coordination). No other open issue/PR touches this assertion.

Labels applied: bug, scope/ci, p3 — tests-only CI-flake hardening, no user-facing impact.

Verified independently against the pushed heads (not taken from the description alone):

  • Diff: b9bf6b20 vs base 5c3c2688 = exactly 1 file, +8/−1 — the observations assertion at tests/issue5955-wedged-same-lane-takeover.test.ts:439 widened from exact === 5 to === 5 || === 4. No src/ change. Matches the stated scope.
  • Rationale holds: the quoted contract exists verbatim in src/launcher.ts ("Concurrent hosts sharing one file can race on read-modify-write; the worst case loses one observation (the window lengthens by one cycle), which is safe."). The escalation gate itself is untouched (observations >= WEDGED_MIN_OBSERVATIONS, =3): accepting 4 still satisfies ≥3, and if observations were NOT bridging across failed escalations the final count would be 1 — so anything below 4 still catches a real regression.
  • No other flake exposure in the suite: the remaining exact-count assertions (lines 471/479/498/499) run deterministic single-process scenarios with no concurrent writer — correctly left strict.
  • Master drift: master moved 3 commits past the PR base since — docs-only (Japanese README, [PR] docs: add Japanese README (#6000) #6004) — no interaction, clean merge.
  • CI on b9bf6b20: 30 checks, 0 failed / 0 pending (mirror cache, fetched 05:27Z).

One small fix I landed directly on the branch (review finding, fixed rather than reported, per policy): the new comment attributed the contract to readWedgedLedger, whose own docstring does not contain the quoted text — it lives in the block above wedgedLedgerFilePath. Comment-only commit bca640cf71 pushed to the branch; the PR head is now bca640cf71 (still 1 file, +9/−1) and a fresh CI run will cover it.

Merge state: mergeable, state blocked — the same owner-side gate pattern #5972/#5989 showed right up until their merges; nothing actionable from the agent side. Ready for @ranxianglei to merge once the fresh run on bca640cf71 is green.

结构化八问:

  1. 是什么问题:[PR] test(launcher): cover the #5955 wedged-takeover escalation ladder + win32 ESRCH silence #5989 新加的 wedged-lane takeover 测试套件里有一处断言把 observations === 5 钉死,而账本自身文档化地说明一次丢失的读改写最多丢一个观测值——fs 时序抖动把文档化的最坏情况变成了 CI 抖动。
  2. 什么原因导致的:账本文件跨宿主/进程共享,读改写竞争最坏丢一个观测(在 src/launcher.ts 中明确记录为 "safe",因为它只让耐心窗口多走一个周期);测试按精确计数断言、未计入这个已文档化的丢失,负载下的瞬时写失败偶发产出 4 而非 5。
  3. 影响范围与严重程度:仅本仓库 CI 测试套件(单个测试文件的一处断言);无用户可见影响;低危(p3)——表现为与本 PR 无关的偶发 shard 红。
  4. 怎么修复的:断言放宽为接受 5 或 4(恰好等于文档化的单次丢失);低于 4 仍判失败,保留判别力(若观测值未跨周期桥接,最终计数会是 1)。
  5. 是否有回归问题:无。逐项审过的既有行为:(a) 升级判定门(observations >= WEDGED_MIN_OBSERVATIONS=3)未动,4 仍满足 ≥3,takeover 语义不变;(b) 套件内其余精确计数断言是确定性单进程场景、无并发写者,保持不变且依然有效;(c) src/ 零改动,代理行为逐字节不变。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/重试/游标/去重键。
  7. 是否改了设计方向:不改——纯测试收尾;LIVE_FILES 那半已在 master(b3aed0b9 经 ci-fixes 合入),从本 diff 中正确剔除。
  8. 是否引入了新的配置:无。

结论:可合并——等 owner(按规则 agent 不合并 PR)。

@ranxianglei
ranxianglei merged commit 049178e into ranxianglei:master Nov 10, 2026
84 of 120 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1152-master.049178e (rolling master tag).

npm install -g billion-context@0.0.1152-master.049178e  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 scope/ci

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants