Skip to content

bug: cross-process startup-window race — two concurrent launches each spawn a proxy (double writers over one sessions dir) #707

Closed

Description

@ranxianglei

来源: #706 分析 feat(pi): native plugin mode 时发现

What

ensureProxyRunning (#394/#417 attach semantics) only attaches when it finds a healthy instance at probe time. Two processes calling ensureProxyRunning inside the probe→healthy window (child spawn + health polling, ~1s–several s) both see "no healthy instance" and each spawn a child. Result:

Repro

Launch the same client twice within ~1s of each other while the first proxy is still coming up (e.g. bili codex ×2 concurrently, or — since #706 — two bare pi terminals opened back-to-back, which now self-bootstrap at extension load). Observe two node dist/index.js start children and two "proxy started" log entries.

Why #706 raises the stakes

Before #706 the window was reachable only via deliberate concurrent bili <client> launches. Native mode makes self-bootstrap the default path for every bare pi start, so back-to-back terminal opens become a realistic trigger.

Impact

Concurrent writes over one sessions dir (state corruption risk), duplicated proxy processes, instance-file flapping between the two origins.

Suggested fix

Publish a "starting" marker BEFORE spawning (e.g. write the instance file with the launchToken up front, or a separate lockfile held across the spawn window). Concurrent probes then find an in-progress bring-up and WAIT for health instead of spawning. In-process callers are already covered by the singleFlight() wrapper added in #706's review commit (3cb0331); this issue is the cross-process half.

Activity

  1. ranxianglei commented on Sep 11, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    Confirmed — root cause verified in code.

    The race window is exactly as described. Cross-launch coordination happens entirely through the instance file (<state>/proxy-origin), and that file is only written by the proxy child after it successfully binds (announceListening, src/server.ts:430). Between spawnImpl (src/launcher.ts:1509) and the child's first successful listen, there is zero on-disk evidence that a bring-up is in flight:

    Plan (matches your suggested fix — the cross-process half only):

    1. New <state>/proxy-starting marker claimed with O_CREAT|O_EXCL (atomic across processes — a rename-based last-writer-wins file would leave the same race at the marker level). Carries {token, pid, host, port, startedAt} and is written by the starter before spawning.
    2. A concurrent caller that sees an active marker (owner pid alive AND age < TTL = SPAWN_WAIT_MS + 30s) waits for the starter's instance instead of spawning — same probe as attach (pid alive + /__bili/health + instanceId match), bounded by SPAWN_WAIT_MS. Marker disappearance (starter gave up / crashed) breaks the wait early so the caller spawns immediately.
    3. Starter clears the marker on every terminal path (healthy / child-exit / timeout) with a token-checked clear; a hard crash leaves a stale marker that the pid/TTL checks treat as dead.
    4. Incompatible-config semantics preserved: if the waited-for instance isn't compatible with our launch options we fall through and spawn anyway (same as today's incompatible-healthy-instance behavior).

    Explicit manual bili start deliberately bypasses all of this (no claim, no wait) — it's a user action, not auto-bootstrap. In-process dedup (singleFlight, 3cb0331) stays as-is.

    Reproducing empirically (two concurrent launches over one shared state dir) before and after the fix, then I'll open a PR.

    (中文摘要:已确认跨进程启动窗口竞态——实例文件只在子进程成功绑定后才写入,spawn→healthy 之间磁盘上没有任何"正在启动"的标记,因此两个并发启动会各自 spawn 一个代理、双写同一 sessions 目录。修复方案:spawn 前用 O_EXCL 原子抢占 <state>/proxy-starting 标记,并发方看到活跃标记就等待健康实例而不是重复 spawn。)

  2. ranxianglei commented on Sep 11, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    Fixed — PR #709: #709

    Reproduced as described. Two concurrent launches sharing one state dir (fake client kept alive so the window stays open) produced exactly the reported failure: two node dist/index.js start children, two "started proxy" log lines, two registry markers under instances/, and proxy-origin flapping last-writer-wins between the two origins. Root cause confirmed where you pointed: the instance file is written by the child only after its first successful bind (announceListening, src/server.ts), so between spawnImpl and first listen there was zero on-disk evidence of an in-flight bring-up for concurrent probes to find.

    Fix (per your suggested approach). The starter claims <state>/proxy-starting with O_EXCL before spawning — {token, pid, host, port, startedAt}. A concurrent caller that finds an active marker (owner pid alive AND younger than SPAWN_WAIT_MS + 30s) waits for the starter's instance using the same probe as attach (pid alive + /__bili/health + instanceId match), bounded by SPAWN_WAIT_MS, then attaches. Marker disappearance (starter gave up / crashed) breaks the wait early so the caller spawns immediately instead of burning the full window. The starter clears the marker on every terminal path via token-checked clear; a hard crash leaves a stale marker that the dead-owner/TTL checks render inert. Incompatible-config fall-through is preserved (a waited-for instance that doesn't match our launch options still spawns, matching today's incompatible-healthy behavior), explicit bili start bypasses the marker entirely, and the in-process singleFlight dedup from #706 is untouched — this PR is the cross-process half.

    Verified post-fix. Same repro now yields exactly one proxy; the second launcher logs another bili launch is bringing up a proxy — waiting for it instead of spawning a second and then attaches to the same origin/pid. Preflight: typecheck clean, npm test 1376 pass / 0 fail (2 skipped = gated e2e), build ok. Nine new tests: 7 launcher scenarios (wait+attach without spawn; dead-owner cleanup+spawn; expired-marker cleanup+spawn; early bail <2s; claim-before-spawn assert; failure-path marker cleanup; incompatible fall-through) plus 2 marker-mechanics unit tests (O_EXCL exclusivity, token-checked clear, garbage-file degrade path). One e2e caveat noted in the PR body: the gated codex e2e needs an upstream not reachable from this environment; the change touches only the launcher bootstrap path (src/launcher.ts + src/instance.ts), not the request pipeline.

    中文摘要: 已按 issue 建议实现跨进程 starting 标记(O_EXCL 原子占位、等待健康后 attach、token 校验清理、死主/TTL 失效),并发启动不再双开代理、消除同一 sessions 目录的双写者,修复在 PR #709,可以合并。

  3. added a commit that references this issue on Sep 11, 2026
  4. added a commit that references this issue on Oct 31, 2026
  5. added a commit that references this issue on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions