Skip to content

[PR] fix: incremental lane — warn when two hosts share one conversation; prune dead-branch cache marks on resync - #4882

Open
ranxianglei wants to merge 11 commits into
masterfrom
2026-10-27_incremental-dual-writer-marks
Open

ranxianglei wants to merge 11 commits into
masterfrom
2026-10-27_incremental-dual-writer-marks

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4868

Two P3 improvements to the incremental lane, both verified against the code paths before implementing (the issue's analysis was accurate on both counts).

1. Dual hosts sharing one conversationId now get a clear warn

Before: two incremental clients on one conversation each resync and flip each other's canonical; every resync invalidates the other side's acks, so both burn 3 consecutive 409s into the sticky REJECTION_CIRCUIT=3 full-resend fallback until process restart — with zero log evidence that this was dual-writer contention rather than ordinary drift.

Fix: every envelope carries an optional per-process instanceId fingerprint (randomUUID(), one value per host process lifetime, stamped once in buildBody so all four native adapters — pi/omp/dsh/opencode-native — get it without per-site edits). The server records the last successful resync source per conversation (noteResyncSource, bounded map with the same budget pattern as droppedParamWarned) and warns once per 5-minute cooldown when two DISTINCT sources replace the same canonical within 60s:

[incremental] <conv>: suspected dual writers — distinct sources "A" and "B" replaced this canonical Ns apart; each resync invalidates the other's acks, so both hosts will burn 3 consecutive 409s into the sticky full-resend fallback unless they stop sharing the conversation id (#4868)

Missing or malformed fingerprints degrade to anonymous (that side simply cannot be fingerprinted — no false alarm); older servers ignore the unknown field, older clients omit it.

Rejected alternatives: socket/pid fingerprinting (sockets rotate across reconnects; pid is never transmitted), header stamping (would need edits at all four adapter dispatch sites instead of one).

2. inc:cache-marks no longer accumulates dead branches

Before: resync replaced the canonical but entry.marks kept every historical id forever — dead-branch marks lingered in the map and were rewritten into the ext key whole every turn, O(turns across branches).

Fix: applyTurn prunes entry.marks to the live canonical set on every resync (before merging the turn's own marks). Ids are unique per session, so a surviving mark can only attach to a live message — nothing can be mis-attributed. Under pluginLaneServerHistory, marks were already orphaned at assembly time (keyed by derived content-hash ids while canonical ids are pinned host ids); the prune makes storage match assembly reality there too.

Behavior changes (all additive, disclosed per policy)

Change Old New
Envelope field no client fingerprint optional instanceId (never forwarded upstream, never persisted)
Logging silent mutual degradation one warn per 5min per conversation for distinct-source resyncs within 60s
Marks lifecycle all historical marks kept forever across resyncs pruned to the live canonical set on each resync

Both rollout features stay at their current defaults (pluginIncrementalTurns 0%, pluginLaneServerHistory 0%) — opt-in installs only; no config key, env var or CLI flag added. The model wire is byte-identical in both compression modes (the fingerprint dies at the lane boundary; the prune only removes marks whose ids are absent from the served history).

Tests (targeted suites green locally; full-suite verdict = CI on the pushed head)

No kernel/ changes, no config surface, no persistence format change (ext value shape unchanged — fewer entries after a resync).

ci-check added 2 commits October 27, 2026 01:09
…une dead-branch cache marks on resync

#4868, two P3 improvements in the incremental lane (both verified against
the code paths; no correctness impact either way):

1. Dual hosts sharing one conversationId mutually kill each other with zero
   visibility: each resync invalidates the other's acks, both burn 3
   consecutive 409s into the sticky REJECTION_CIRCUIT=3 full-resend fallback
   until process restart. Every envelope now carries an optional per-process
   instanceId fingerprint (randomUUID, one value per host process, stamped in
   buildBody so all four native adapters get it for free); the server records
   the last successful resync source per conversation and warns once per 5min
   cooldown when two DISTINCT sources replace the same canonical within 60s.
   Missing or malformed fingerprints degrade to anonymous (no false alarm);
   old servers ignore the field, old clients omit it.
   - src/agent/incremental-client.ts: INSTANCE_ID + __incrementalInstanceIdForTests, buildBody stamps it
   - src/incremental-lane.ts: noteResyncSource (bounded map, budget pattern like droppedParamWarned), called on successful resync turns; reset seam clears it

2. inc:cache-marks grew unbounded across branches: resync replaced the
   canonical but entry.marks kept every historical id forever (map + ext key
   rewritten whole each turn, O(turns across branches)). applyTurn now prunes
   marks to the live canonical set on every resync before merging the turn's
   own marks. Ids are unique per session, so a surviving mark can only attach
   to a live message; under pluginLaneServerHistory the marks were already
   orphaned at assembly (derived-id keys vs pinned ids) — the prune makes
   storage match assembly reality.

Behavior changes (additive): new optional envelope field instanceId; new warn
log line for suspected dual writers; marks lifecycle OLD = all historical
marks kept forever → NEW = pruned to the live set on each resync. Both rollout
features (pluginIncrementalTurns, pluginLaneServerHistory) default to 0% —
opt-in installs only.

Tests: dual-writer alarm positive + same-source negative + missing/malformed
fingerprint negative; marks pruning (dead branch pruned, live/new survive);
client envelope stamp stability. PLUGIN.md documents the new field.
The #4491 host message-id field is accepted by the lane (payload.msgIds,
src/incremental-lane.ts) but was never added to the documented envelope
shape — readers could not discover it. One-line addition to the field list
while this PR edits that very sentence for instanceId.
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-27_incremental-dual-writer-marks (39cbce1)

Latest build — install this exact version

npm install -g billion-context@0.0.9411-pr.4882.1159c97

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
39cbce1 billion-context@0.0.9411-pr.4882.1159c97 build
20fbb4e billion-context@0.0.9170-pr.4882.7584964 build
f063bae billion-context@0.0.7336-pr.4882.1589269 build
ec7116a billion-context@0.0.7105-pr.4882.d398601 build
d579ac9 billion-context@0.0.6841-pr.4882.7f8c77d build
23cc847 billion-context@0.0.6361-pr.4882.c235eb3 build
a59ff13 billion-context@0.0.6352-pr.4882.498e497 build
556b776 billion-context@0.0.6327-pr.4882.aeff476 build
d577f24 billion-context@0.0.6320-pr.4882.ebb768a build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4882.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: incremental lane — warn when two hosts share one conversation; prune dead-branch cache marks on resync [PR] fix: incremental lane — warn when two hosts share one conversation; prune dead-branch cache marks on resync Oct 26, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: merged origin/master into the branch cleanly (no conflicts), pushed new head 556b776 — all CI checks on it are green (11/11 core test legs + every guard job); the branch is up to date with master again.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Re-check on the "behind" wake: no new sync needed — origin/master (23158db) is already an ancestor of the PR head (556b776), so the branch is fully up to date; all CI checks on that head remain green (26 success + 16 legitimately skipped, zero failures).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Re-synced after master moved (#4854/#4817): merged origin/master cleanly (no conflicts), pushed new head a59ff13 — local typecheck + targeted incremental-lane tests green before push, and now all CI checks on that head are green (11/11 core test legs + every guard job); branch is up to date with master again.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Re-synced after master moved again (#4856): merged origin/master cleanly (no conflicts), pushed new head 23cc847 — local typecheck + targeted incremental-lane tests green before push, and now all CI checks on that head are green (11/11 core test legs + every guard job); branch is up to date with master again.

ranxianglei pushed a commit that referenced this pull request Nov 2, 2026
…23 (#5522)

The WC-023 ledger commit (f862e4a) accidentally deleted three permanent WIRE_RULES entries (WC-010 cache-breakpoint cap, WC-013 reasoning.summary, WC-014 Jinja head-system) and overwrote WC-011's #1733 provenance with unrelated #4882 text - violating the ledger-grows-only rule (#1304). Their enforcement clauses and gate tests were never removed, so this is a faithful byte-for-byte restore of the known-good records from ca532d4. Verified locally: tests/wire-contract.test.ts 23/23 green (ledger self-check now sees all 25 rules) and tsc --noEmit clean.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved and pushed: f063bae8b8..20fbb4eb96 — a fast-forward merge of master into the PR branch (no force-push). Seven-question summary:

Q1 — What was actually wrong? Not ordinary drift: master's history was rewritten after Oct 31. The two old-lineage merges this branch absorbed on Oct 31 (fc0c086a75, 69be0b7dd0) are NOT ancestors of live master, so the true merge-base is 06ad804ac0 and the branch carried ~120 files of discarded lineage → 97 conflicting files when merging master in. Rebase was rejected up front: it would replay hundreds of phantom old-lineage "master" commits onto the rewritten history. (Master also advanced mid-run, dc041dd1ab → c26c31ace4 via #6260; the final tree was verified against the freshest live master.)

Q2 — How was it fixed? MERGE (consistent with all six prior sync rounds on this PR), enforced by a deterministic invariant: final tree = live master + exactly the PR's own diff. All 97 conflicts taken on the master side; two sweeps of stray auto-merged files that silently retained old-lineage content (9 + 14, spanning CHANGELOG.md, src/web/, src/util.ts, docs/architecture/webui, reference/architecture.md, …) reset to master versions; then the PR's original patch (extracted verbatim from its two commits 6bb67c4f06 + d577f24c08) was re-applied onto the master baseline. Verified: git diff <live-master> HEAD = exactly 4 files.

Q3 — Semantic conflict resolutions (unions, not pick-a-side):

Q4 — Verification: typecheck clean; targeted suites tests/incremental-lane.test.ts + tests/incremental-client-recovery.test.ts = 29/29 pass — including all three new #4868 tests (dual-writer warn-once + cooldown, no false alarm on missing/malformed instanceId, dead-branch mark prune asserted against the persisted ext row) AND all pre-existing master lane tests, i.e. the unions broke neither side. Full-suite verdict left to remote CI per local-test discipline.

Q5 — Push discipline: fetched before pushing; remote PR head unchanged (f063bae8b8, no concurrent work) → plain fast-forward push, no force-push, no rejections.

Q6 — CI state: cannot assert yet — the mirror check-cache has not picked up 20fbb4eb96 (it still shows the Oct 31 sha with checks queued). Will report once the poller catches up; prior reds on this PR were attributed (floor 31) to inherited master flakes, not this diff.

Q7 — Decisions needed / behavior flags: none needed — every resolution was mechanical or a strict union. Behavior flags: (a) the prune line now sets marksTouched (master's #5155 accounting) instead of a bare delete — strictly more correct under master's newer mechanism (map change ⇒ ext row persisted); (b) PASSTHROUGH_HEADERS removed as dead code against master; (c) the PLUGIN.md edit dropped as already landed upstream. Both rollout features remain 0% opt-in; no config surface; no kernel changes.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Re-synced against the moved master and pushed: 20fbb4eb96..39cbce1aee (fast-forward merge, no force-push). Seven-question summary:

Q1 — What was wrong this run? Master advanced again after last night's sync: c26c31ace4 → 259cd87674 (~90 commits: #6455, #6458, #6438, #6439, #6326 durability-before-render, kernel 0.0.196, …). A local git merge-tree against the live head showed exactly ONE conflict — CHANGELOG.md (both sides add entries at the top; the branch side carries no legitimate CHANGELOG change of its own). This was a normal forward move, not another history rewrite.

Q2 — How was it fixed? Merged live master into the branch (7th sync round, same pattern as all prior rounds), enforcing the same deterministic invariant: final tree = live master + exactly the PR's own diff. The CHANGELOG conflict resolved to the master side (a strict superset — it contains everything the branch side had plus the new entries). Verified afterwards: git diff <live-master> HEAD = exactly the 4 PR files.

Q3 — Semantic resolutions: none beyond the CHANGELOG superset. One interaction checked explicitly: master's new #3397 durability-before-render hunk in src/incremental-lane.ts (ingestRowsNow in the settle path) auto-merged with the PR's hunks — no line overlap (PR hunks sit in the dual-writer block, the resync comment, the mark prune, the test seam, and the noteResyncSource call; master's hunk is in the settle path), and both were verified present in the merged file. The PR's own diff is byte-identical to last night's: 4 files, +194/−9.

Q4 — Verification: npm ci (lockfile changed with the kernel 0.0.196 bump) → typecheck clean → targeted suites tests/incremental-lane.test.ts + tests/incremental-client-recovery.test.ts = 29/29 pass (all three new #4868 tests plus every pre-existing lane test, now also exercising the lane with master's durability hunk in place). Full-suite verdict left to remote CI.

Q5 — Push discipline: fetched before pushing; remote PR head unchanged (20fbb4eb96, no concurrent work) → plain fast-forward push, no force-push, no rejections.

Q6 — CI state: the mirror check-cache has not picked up 39cbce1aee yet (it still shows the previous head 20fbb4eb96, where all 30 checks completed with 0 failed). I can't assert the new head's state yet — will report once the poller catches up. Note the delta on the new head is only master's own already-merged content plus the same 4-file diff that just went green.

Q7 — Decisions needed / behavior flags: none. This sync introduced no behavior change of its own (the behavioral deltas in the tree are master's own merged PRs, not this PR's). Both rollout features remain 0% opt-in; no config surface; no kernel changes from this PR.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Status confirmation after process restart (work was already complete and pushed before the restart; the earlier 七问 summary above covers the full run):

  • Branch state: PR head = 39cbce1aee (merge of live master 259cd87674 + the PR's own 4-file diff, +194/−9), pushed fast-forward, no force-push.
  • CI (live from mirror cache): 39cbce1aee — 30/30 checks passed, 0 failed (cache fetched 2026-11-13 07:10).
  • Merge state (live): master has since moved to 99c22d283f; git merge-tree against the PR head is clean — no conflicts, so the PR is not dirty against current master.
  • Nothing pending on my side. If the lane re-triggers on a future master move, the same sync procedure applies.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant