Repository navigation
[PR] search: search_context finds folded-away blocks and the folded original text, not just live summaries - #5181
Conversation
…ded original text (#4154) The live search path called the kernel's active-only core.search wrapper, so a block consumed by a later fold vanished from search_context results — its summary was still real history the model could act on, but the tool pretended it did not exist, and the verbatim pre-fold text was reachable only through decompress (#4154 sub-defect B). - Rank EVERY block via the kernel's public search surface (blockDocs + searchBlocks) instead of the active-only wrapper; render inactive hits with an explicit marker pointing at decompress as the way back. - Keep exact no-match semantics: algorithm "substring" with a ~zero floor (#2158 — one literal occurrence is a positive signal; the default "hybrid" algorithm's fuzzy component would turn garbage queries into weak matches and regress the #2751 no-match answer). - New folded-original-text corpus: wherever a Session object backs the searched state, its blockContents (resident or lazy-reloaded from the store) is searched verbatim; hits that no summary mentions get their own result section. Rendered acp ref tags are stripped from the corpus (their tokens="/type= attributes are protocol chrome — left in, every query containing "token" or "type" false-positives on every block). - Foreign/derived lookups feed originals when the session is resident; store-only sessions degrade to summary search. A throwing provider degrades to summary-only search. Tests: inactive-block findability + marker, original-text-only hit section, no-match scope note, provider failure degradation; updated the two assertions that pinned the old active-only scope wording.
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.7224-pr.5181.8dfd505This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5181.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…uires (#4599) The #5140 recovery test proves forkSnapshot reconstructs a non-retained parent's raw tail from the UNIFIED index — but it never pinned its own storage mode. Under the CI storage baseline (preload-storage-baseline.mjs sets BILI_STORAGE_UNIFIED=legacy for the broad corpus) no unified rows exist, so the reconstruction has no source and forkSnapshot fails closed with 409 SNAPSHOT_UNAVAILABLE. Deterministic, not a flake: reproduces 100% on master and any merged branch with BILI_STORAGE_UNIFIED=legacy, passes without it (default shadow). Pin full like the sibling persist tests and restore the ambient value afterwards.
|
(ai: vllm-qwen/qwen3.8-27b) Behind-sync done: merged origin/master@44067a5bc + ci-fixes@67f8e8940 into the branch -> head 7312f57, zero conflicts; all fast gates green, sharded test lanes still waiting in the runner queue. |
ci-fixes at 67f8e89 still leaves four Windows shards red, and all four deltas already exist on PR #5178. This ports the missing hunks so the convergence branch actually converges. 1. `test (windows-latest, 2x, 3)` — #5103. `UnifiedShadow` builds its db path as `${dir}/index.db`, so the open-failure warning reads `C:\...\sessions/index.db`. `issue5103` asserts the warning contains `path.join(dir, "index.db")` — `...\index.db` on win32 — so a mixed separator never matches. `path` is already imported at :30 and the db is opened through the same join, so POSIX output is byte-identical. 2. `test (windows-latest, 22, 4)` + the 4788 leg — #4739/#4788 teardown. The suites arm a real `SessionStorage` under `BILI_STORAGE_UNIFIED=full`, so the sidecar is still open at teardown. `cancelAll()` clears timers and the build map but never closes the shadow; Windows refuses to unlink a directory holding an open file (`EBUSY`/`EPERM` on `index.db`). Use `closeQuietly()` — cancel, then close — in the short-lived-store teardown. 3. `test (ubuntu-latest, 24, 3)` + `(windows-latest, 22, 2)` — #5155. Both cold-rebuild cases waited for the debounced durable flush with a fixed `setTimeout(20)`. A millisecond budget is a deadline a loaded runner can miss, and the miss is an honest `409 !== 200` from a one-turn-stale table. Await the settle itself: `afterNextFlush("conv-ptrs")` resolves once the armed save lands. 4. `test (ubuntu-latest, 22, 3)` — #4788 verdict deadlock. `afterNextFlush` parks a resolver in `flushWaiters`, and the only thing that drains it is the debounce timer — which is armed `unref?.()`. Once that timer is the last pending handle the loop drains, the promise never settles, and node reports `Promise resolution is still pending but the event loop has already resolved`. Keep the timer referenced exactly while a waiter exists (`ref()` on waiter registration; the host-upkeep default stays unref'd). Both view hunks and the two test-teardown hunks are byte-identical to the ones already on PR #5178, so this converges the branches instead of inventing a third variant. Verified locally: `npx tsc --noEmit` clean; issue4788 0/4 -> 4/4; issue4739 4/4; issue5103 5/5; inc-canonical-pointers 3/3. The `cancelAll` -> `closeQuietly` hunk has no local negative control: reverting it keeps Linux green because `EBUSY`/`EPERM` only fire where an open handle blocks unlink. That control can only run on a Windows lane — stated rather than papered over. The #4788 arm's own reader is confirmed instead: `cancelAll()` leaves `index.db`, `index.db-shm`, `index.db-wal` in `/proc/self/fd` while `closeQuietly()` leaves none (probed directly).
…e settle (#4599) The turn() helper arms scheduleSave whose debounce timer is unref by design (persist.ts: a pending save must not hold the process open at shutdown) and then awaits afterNextFlush. Nothing else in that await is refd, so in a node:test child whose only surviving handles are unref sockets the event loop drains before the 50 ms debounce fires and the settle promise never resolves: all four subtests fail with "Promise resolution is still pending but the event loop has already resolved" / cancelledByParent. Reproduced deterministically on master 44067a5 AND on pre-merge ci-fixes 67f8e89 (standalone run, exact CI preload chain), while the same file has passed some CI legs by ambient-liveness luck - a born-flake per section 7.2 (no environmental luck). The pin at 67f8e89 does not touch this path. Fix: arm a refd 25 ms keep-alive interval around the afterNextFlush await (cleared in finally) so the debounce fires deterministically on every host. Test-only diff; production settle semantics untouched.
…ts ref-tracking fix (#4599) PR #5262 fixes #4788 at the source: scheduleSave now leaves the debounce timer refd while flush waiters exist (and afterNextFlush refs a pending unrefd timer when a waiter registers), so the settle is deterministic for EVERY caller on every host. The 25 ms keep-alive interval added in a3019b0 only patched this one test file and is redundant now; removing it leaves one mechanism instead of two. The comment stays: without it a future reader re-adds the interval or deletes the persist.ts tracking as dead weight.
|
(ai: vllm-qwen/qwen3.8-27b) Behind-sync round 2: master@44067a5bc was already contained in the branch; merged the advanced ci-fixes@06ad804ac (persist.ts/shadow.ts red-lane fixes) -> head 9738657, zero conflicts, PR diff intact; fresh CI run registered, all checks queued behind the shared-runner backlog, no failures. |
Fixes #4154 (the sub-defect B that survived triage: consumed blocks leave the search pool).
What was wrong (in plain words)
When a later fold consumed an earlier compressed block, that block vanished from
search_contextresults — the tool only ranked active blocks (core.search→activeBlocks). But a consumed block's summary is still real history the model can act on (a chain re-fold covers it,decompressrestores it); the tool pretended it did not exist. The empty-result reply even admitted the scope: "Scope = active compressed topics/summaries only — NOT … inactive/distilled blocks". And the verbatim pre-fold text was reachable only throughdecompress, never through search — a detail remembered only in the folded-away original didn't exist as far assearch_contextwas concerned.What changed (host-only, no kernel changes)
blockDocs+searchBlocks, which already documented "all blocks (active AND inactive)" — it was just never wired into the live tool path). Inactive hits render with an explicit marker:[inactive — folded away; decompress bN restores it].Sessionobject backs the searched state: itsblockContents(resident or lazy-reloaded from the store) is searched verbatim; hits that no summary mentions get their own result section telling the model the verbatim record needsdecompress. Foreign resident sessions feed it too; store-only sessions degrade to summary search; a throwing provider degrades to summary-only (never fails the call)."substring"with a ~zero floor. The defaulthybridalgorithm's fuzzy component would turn garbage queries into weak matches and regress the [Gap][search_context] 长查询结果被通用词挤占,回执未明确候选截断与检索范围,容易被误读为历史不存在 #2751 no-match answer;substring+minScore 1e-9keeps "one literal occurrence is a positive signal, zero = no match" ([pi] 子代理中 search_context 对已生成的活动压缩块零命中(compress/decompress 正常) #2158).<acp tokens="…" type="…">wrappers are removed (inner ref kept) — otherwise every query containing "token" or "type" false-positives on every block.acp_statusstill lists active blocks only (unchanged — this PR is about findability, not the overview).Tests
Open objectivesis lost → model misidentifies | [Design][dsh] 压缩摘要正文对后续轮次不可见 →Open objectives丢失 → 模型误判当前目标重答历史问题 #4154)search_context显式查询历史 session #841 one)Full suite green locally modulo the 6 pre-existing sandbox opencode failures (verified identical on pristine master: opencode-v2 ×3, opencode-native ×1, #3304 ×2 — spawn/port-bound environment issues, the same family PR #4620's matrix shows).