Skip to content

fix(launcher): cross-process startup-window race — starting marker prevents double-spawn (#707) - #709

Merged
ranxianglei merged 2 commits into
masterfrom
2026-09-11_fix-707-startup-race
Sep 11, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-09-11_fix-707-startup-race

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Fixes #707. Two concurrent ensureProxyRunning calls inside the spawn→healthy window each spawned their own proxy child: two live proxies over one sessions dir (double writers), last-writer-wins instance file, and the orphaned extra proxy living until ITS OWN spawner exited.

Root cause: cross-launch coordination happened only through <state>/proxy-origin, which the child writes after its first successful bind (announceListening, src/server.ts). Between the launcher's spawnImpl and the child's first listen there was zero on-disk evidence of an in-flight bring-up, so concurrent probes found nothing and both spawned. Ephemeral ports (#446) let both children bind different ports cleanly; an explicit port just made the second retry port+1 (#407) — either way, two writers.

Fix

The starter publishes a "starting" marker before spawning:

  • New <state>/proxy-starting file, claimed with O_EXCL (fs.openSync(path, "wx")) — atomic across processes; a rename-based write would leave the same race at the marker level. Content: {token, pid, host, port, startedAt}.
  • A concurrent caller that finds an active marker (owner pid alive AND younger than SPAWN_WAIT_MS + 30s) waits for the starter's instance using the same probe as attach (pid alive + /__bili/health + instanceId match), bounded by SPAWN_WAIT_MS, then attaches. Marker disappearance (starter gave up / crashed) breaks the wait early so the caller spawns immediately instead of burning the full window.
  • The starter clears the marker on every terminal path via token-checked clear (try/finally around spawn→poll→throw). A hard crash leaves a stale marker that the dead-owner/TTL checks render inert.
  • Incompatible-config fall-through preserved: a waited-for instance that isn't compatible with our launch options still spawns (matches today's incompatible-healthy-instance behavior). Explicit bili start bypasses the marker entirely (user action, not auto-bootstrap). In-process dedup (singleFlight, feat(pi): native plugin mode — self-bootstrapping proxy + fetch interception #706) untouched — this is the cross-process half.

Verification

  • Repro before fix — two concurrent PI_BIN=<fake-client> node dist/index.js pi --port 0 sharing one isolated XDG_STATE_HOME: 2 proxy children, two started proxy log lines, two registry markers, instance file flapping between origins.
  • Same repro after fix: 1 proxy child; the second launcher logs another bili launch is bringing up a proxy — waiting for it instead of spawning a second → attaching to running proxy at http://127.0.0.1:44551 (pid …); exactly one registry marker; no leftover marker file.
  • New tests — 7 launcher tests: active-marker wait+attach without spawn; dead-owner stale removal+spawn; expired-marker removal+spawn; early bail on marker disappearance (<2s); starter claims before spawn and clears after success; child-exit failure path clears marker; incompatible bring-up falls through to spawn. Plus 2 marker-mechanics unit tests (O_EXCL exclusivity, token-checked clear, garbage-file degrade path). tests/launcher.test.ts now isolates XDG_STATE_HOME per run so marker I/O stays hermetic under parallel node --test.
  • Preflight: npm run typecheck clean · npm test 1376 pass / 0 fail (2 skipped = gated e2e) · npm run build ok.
  • e2e note: the gated e2e-codex suite needs a Responses-compatible upstream (default http://127.0.0.1:8199/v1), which is not reachable from this environment; the change touches only the launcher bootstrap path (src/launcher.ts + src/instance.ts), not the request pipeline, and is covered by the unit tests + the empirical concurrent-launch repro above.

中文摘要: 修复了跨进程启动窗口竞态——启动方在 spawn 前用 O_EXCL 发布 proxy-starting 标记,并发的启动会等待该实例就绪后 attach,而不是再 spawn 一个代理(消除同一 sessions 目录的双写者),可以合并。

@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-11_fix-707-startup-race (61ae4fd)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-709

Each push to this PR publishes a new version under the pr-709 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr709.tgz
npm install -g package

This comment is automatically updated on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewing PR #709 (fixes #707) now — checking diff cleanliness, the O_EXCL marker mechanics in src/launcher.ts / src/instance.ts, running typecheck + tests + build, and reasoning through both compression modes per AGENTS.md §6. Will report back with findings.

…OCTOU and deadline-boundary sliver (#707 review)

Two residual double-spawn windows in the starting-marker protocol, found in review:

1. The pre-spawn readStartingMarker() was treated as the coordination
   decision while the O_EXCL claim result was ignored: two processes that
   both read 'no marker' before either claimed would each spawn (the loser's
   EEXIST was dropped on the floor). The default port-0 path widens the
   window with await pickEphemeralPort() between read and claim. Now a
   claim loser re-reads the marker: active holder → wait for it (same probe
   as attach); stale/garbage file → remove and re-claim once; still losing →
   degrade to pre-#707 spawn. This also self-heals an unreadable marker file
   (crash mid-write), which previously blocked every future claim forever.
2. The wait loop's deadline coincides with the starter's own poll deadline:
   a starter succeeding in its last ~200ms could be missed by the waiter's
   final poll, which then fell through to spawning a duplicate. The success
   path clears the marker, so the miss is indistinguishable from a failure
   bail without one more look — waitForStarterInstance now ends with a final
   probe after the loop exits.

Tests: garbage-marker self-heal (fails on the pre-review code),
deadline-boundary attach instead of double-spawn (fails on the pre-review
code), hung-starter bounded-wait pin. Preflight: typecheck clean, 1379 pass /
0 fail / 2 skipped, build ok.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review: LGTM after two fixes I pushed directly to the branch (commit 61ae4fd, now on 2026-09-11_fix-707-startup-race).

Diff cleanliness first pass: clean — 5 files, all on-topic (src/launcher.ts, src/instance.ts, 2 test files, CHANGELOG), no drive-by changes. The core design is sound: O_EXCL claim before spawn, active-marker wait with attach, token-checked clear on every terminal path, dead-owner/TTL stale rendering, incompatible fall-through preserved, bili start bypass structural (it never goes through ensureProxyRunning), and singleFlight (#706) correctly left alone. I verified the concurrency invariants hold: a hung owner can't delete our marker (token-checked clear), and unconditional removal of a stale marker can't clobber a newer claimant (O_EXCL bars any new claim while the old file exists).

Two residual double-spawn windows remained, both in the exact invariant this PR establishes — fixed in 61ae4fd:

1. Read→claim TOCTOU (the real one). The pre-spawn readStartingMarker() was treated as the coordination decision, but the claimStartingMarker() result at src/launcher.ts was ignored — on EEXIST the code spawned anyway. Two processes that both read "no marker" before either claimed would each spawn: the loser's failed claim changed nothing. The default port-0 path widens the window with await pickEphemeralPort() between read and claim (a full listen/close round-trip), so this wasn't theoretical microsecond luck — two near-simultaneous bili pi launches could hit it. Now the O_EXCL claim itself is the arbiter: a claim loser re-reads the marker → active holder means wait for its instance (same probe as attach); stale/unreadable file means remove-and-reclaim once; still losing degrades to a plain spawn. Side benefit: an unreadable marker file (crash mid-write leaving partial JSON) previously blocked every future claim forever, silently degrading all launches until manual deletion — it now self-heals on the next launch.

2. Deadline-boundary sliver. The waiter's deadline coincides with the starter's own poll deadline. A starter succeeding in its last ~200 ms could be missed by the waiter's final poll (200 ms interval + probe latency), which then fell through to spawning a duplicate over a healthy instance. Subtle because the success path clears the marker — to the waiter it looks identical to a failure bail. waitForStarterInstance now ends with one final probe after the loop exits (cheap: ENOENT/dead-pid short-circuit; only costs a health fetch when a live record exists).

Tests added for both (plus a bounded-wait pin for the hung-starter worst case): garbage-marker self-heal, deadline-boundary attach-instead-of-double-spawn, hung starter waits max twice then spawns. I verified the first two fail against the pre-review code (ran them against 2a7e5d1: the self-heal test sees no re-claimed token; the boundary test hits the double-spawn throw) and pass with the fix.

Preflight on the updated branch: npm run typecheck clean · npm test 1379 pass / 0 fail / 2 skipped (was 1376; +3 new) · npm run build ok. E2E note stands as you wrote it — the gated e2e-codex upstream isn't reachable from my environment either (confirmed), and the diff touches only the launcher bootstrap path, which the unit tests + your concurrent-launch repro cover.

中文摘要:审查发现并直接修复了两处残留的双开窗口——(1) 读标记与 O_EXCL 抢占之间的 TOCTOU(抢占失败被忽略,仍会再 spawn 一个,默认端口 0 路径因 pickEphemeralPort 异步操作窗口更大),现改为以抢占本身为仲裁者,失败方重查后等待或清除重抢,顺带让崩溃残留的垃圾标记文件可自愈;(2) 等待截止时间与启动方自身轮询截止重合,启动方在最后 ~200ms 成功时等待方可能漏探而重复 spawn,现循环结束后补一次终探。两个新回归测试均验证过在修复前代码上失败、修复后通过;全量测试 1379 通过,可以合并。

@ranxianglei
ranxianglei merged commit ca3bab8 into master Sep 11, 2026
6 checks passed
@ranxianglei ranxianglei mentioned this pull request Sep 11, 2026
@ranxianglei
ranxianglei deleted the 2026-09-11_fix-707-startup-race branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: cross-process startup-window race — two concurrent launches each spawn a proxy (double writers over one sessions dir)

1 participant