Skip to content

[PR] search: search_context finds folded-away blocks and the folded original text, not just live summaries - #5181

Merged
ranxianglei merged 14 commits into
masterfrom
2026-10-29_search-folded-originals
Oct 31, 2026
Merged

ranxianglei merged 14 commits into
masterfrom
2026-10-29_search-folded-originals

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4154 (the sub-defect B that survived triage: consumed blocks leave the search pool).

What was wrong (in plain words)

When a later fold consumed an earlier compressed block, that block vanished from search_context results — the tool only ranked active blocks (core.search → activeBlocks). But a consumed block's summary is still real history the model can act on (a chain re-fold covers it, decompress restores it); the tool pretended it did not exist. The empty-result reply even admitted the scope: "Scope = active compressed topics/summaries only — NOT … inactive/distilled blocks". And the verbatim pre-fold text was reachable only through decompress, never through search — a detail remembered only in the folded-away original didn't exist as far as search_context was concerned.

What changed (host-only, no kernel changes)

  • Rank every block (active + folded-away) via the kernel's public search surface (blockDocs + searchBlocks, which already documented "all blocks (active AND inactive)" — it was just never wired into the live tool path). Inactive hits render with an explicit marker: [inactive — folded away; decompress bN restores it].
  • Search the folded original text wherever a Session object backs the searched state: its blockContents (resident or lazy-reloaded from the store) is searched verbatim; hits that no summary mentions get their own result section telling the model the verbatim record needs decompress. Foreign resident sessions feed it too; store-only sessions degrade to summary search; a throwing provider degrades to summary-only (never fails the call).
  • Exact no-match semantics kept: algorithm "substring" with a ~zero floor. The default hybrid algorithm's fuzzy component would turn garbage queries into weak matches and regress the [Gap][search_context] 长查询结果被通用词挤占,回执未明确候选截断与检索范围,容易被误读为历史不存在 #2751 no-match answer; substring + minScore 1e-9 keeps "one literal occurrence is a positive signal, zero = no match" ([pi] 子代理中 search_context 对已生成的活动压缩块零命中(compress/decompress 正常) #2158).
  • Protocol chrome stripped from the original-text corpus: rendered <acp tokens="…" type="…"> wrappers are removed (inner ref kept) — otherwise every query containing "token" or "type" false-positives on every block.
  • acp_status still lists active blocks only (unchanged — this PR is about findability, not the overview).

Tests

Full suite green locally modulo the 6 pre-existing sandbox opencode failures (verified identical on pristine master: opencode-v2 ×3, opencode-native ×1, #3304 ×2 — spawn/port-bound environment issues, the same family PR #4620's matrix shows).

…ded original text (#4154)

The live search path called the kernel's active-only core.search wrapper,
so a block consumed by a later fold vanished from search_context results —
its summary was still real history the model could act on, but the tool
pretended it did not exist, and the verbatim pre-fold text was reachable
only through decompress (#4154 sub-defect B).

- Rank EVERY block via the kernel's public search surface (blockDocs +
  searchBlocks) instead of the active-only wrapper; render inactive hits
  with an explicit marker pointing at decompress as the way back.
- Keep exact no-match semantics: algorithm "substring" with a ~zero floor
  (#2158 — one literal occurrence is a positive signal; the default
  "hybrid" algorithm's fuzzy component would turn garbage queries into
  weak matches and regress the #2751 no-match answer).
- New folded-original-text corpus: wherever a Session object backs the
  searched state, its blockContents (resident or lazy-reloaded from the
  store) is searched verbatim; hits that no summary mentions get their own
  result section. Rendered acp ref tags are stripped from the corpus
  (their tokens="/type= attributes are protocol chrome — left in, every
  query containing "token" or "type" false-positives on every block).
- Foreign/derived lookups feed originals when the session is resident;
  store-only sessions degrade to summary search. A throwing provider
  degrades to summary-only search.

Tests: inactive-block findability + marker, original-text-only hit
section, no-match scope note, provider failure degradation; updated the
two assertions that pinned the old active-only scope wording.
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-29_search-folded-originals (9738657)

Latest build — install this exact version

npm install -g billion-context@0.0.7224-pr.5181.8dfd505

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
9738657 billion-context@0.0.7224-pr.5181.8dfd505 build
7312f57 billion-context@0.0.7126-pr.5181.3f81faf build
2ea7219 billion-context@0.0.7104-pr.5181.695cfbe build
c4828c8 billion-context@0.0.7087-pr.5181.efefe10 build
4cc6cd5 billion-context@0.0.6918-pr.5181.0196eb4 build
6fe276b billion-context@0.0.6907-pr.5181.34fa99c build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5181.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title search: search_context finds folded-away blocks and the folded original text, not just live summaries [PR] search: search_context finds folded-away blocks and the folded original text, not just live summaries Oct 30, 2026
ranxianglei and others added 7 commits October 30, 2026 08:44
…uires (#4599)

The #5140 recovery test proves forkSnapshot reconstructs a non-retained
parent's raw tail from the UNIFIED index — but it never pinned its own
storage mode. Under the CI storage baseline (preload-storage-baseline.mjs
sets BILI_STORAGE_UNIFIED=legacy for the broad corpus) no unified rows
exist, so the reconstruction has no source and forkSnapshot fails closed
with 409 SNAPSHOT_UNAVAILABLE. Deterministic, not a flake: reproduces 100%
on master and any merged branch with BILI_STORAGE_UNIFIED=legacy, passes
without it (default shadow). Pin full like the sibling persist tests and
restore the ambient value afterwards.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Behind-sync done: merged origin/master@44067a5bc + ci-fixes@67f8e8940 into the branch -> head 7312f57, zero conflicts; all fast gates green, sharded test lanes still waiting in the runner queue.

ework-agent and others added 6 commits October 31, 2026 01:22
ci-fixes at 67f8e89 still leaves four Windows shards red, and all four
deltas already exist on PR #5178. This ports the missing hunks so the
convergence branch actually converges.

1. `test (windows-latest, 2x, 3)` — #5103. `UnifiedShadow` builds its db
   path as `${dir}/index.db`, so the open-failure warning reads
   `C:\...\sessions/index.db`. `issue5103` asserts the warning contains
   `path.join(dir, "index.db")` — `...\index.db` on win32 — so a mixed
   separator never matches. `path` is already imported at :30 and the db is
   opened through the same join, so POSIX output is byte-identical.

2. `test (windows-latest, 22, 4)` + the 4788 leg — #4739/#4788 teardown.
   The suites arm a real `SessionStorage` under
   `BILI_STORAGE_UNIFIED=full`, so the sidecar is still open at teardown.
   `cancelAll()` clears timers and the build map but never closes the
   shadow; Windows refuses to unlink a directory holding an open file
   (`EBUSY`/`EPERM` on `index.db`). Use `closeQuietly()` — cancel, then
   close — in the short-lived-store teardown.

3. `test (ubuntu-latest, 24, 3)` + `(windows-latest, 22, 2)` — #5155.
   Both cold-rebuild cases waited for the debounced durable flush with a
   fixed `setTimeout(20)`. A millisecond budget is a deadline a loaded
   runner can miss, and the miss is an honest `409 !== 200` from a
   one-turn-stale table. Await the settle itself: `afterNextFlush("conv-ptrs")` resolves once the armed save lands.

4. `test (ubuntu-latest, 22, 3)` — #4788 verdict deadlock.
   `afterNextFlush` parks a resolver in `flushWaiters`, and the only thing
   that drains it is the debounce timer — which is armed `unref?.()`.
   Once that timer is the last pending handle the loop drains, the promise
   never settles, and node reports
   `Promise resolution is still pending but the event loop has already
   resolved`. Keep the timer referenced exactly while a waiter exists
   (`ref()` on waiter registration; the host-upkeep default stays unref'd).

Both view hunks and the two test-teardown hunks are byte-identical to the
ones already on PR #5178, so this converges the branches instead of
inventing a third variant.

Verified locally: `npx tsc --noEmit` clean; issue4788 0/4 -> 4/4;
issue4739 4/4; issue5103 5/5; inc-canonical-pointers 3/3.

The `cancelAll` -> `closeQuietly` hunk has no local negative control:
reverting it keeps Linux green because `EBUSY`/`EPERM` only fire where an
open handle blocks unlink. That control can only run on a Windows lane —
stated rather than papered over.

The #4788 arm's own reader is confirmed instead: `cancelAll()` leaves
`index.db`, `index.db-shm`, `index.db-wal` in `/proc/self/fd` while
`closeQuietly()` leaves none (probed directly).
…e settle (#4599)

The turn() helper arms scheduleSave whose debounce timer is unref by design
(persist.ts: a pending save must not hold the process open at shutdown) and
then awaits afterNextFlush. Nothing else in that await is refd, so in a
node:test child whose only surviving handles are unref sockets the event
loop drains before the 50 ms debounce fires and the settle promise never
resolves: all four subtests fail with "Promise resolution is still pending
but the event loop has already resolved" / cancelledByParent. Reproduced
deterministically on master 44067a5 AND on pre-merge ci-fixes 67f8e89
(standalone run, exact CI preload chain), while the same file has passed
some CI legs by ambient-liveness luck - a born-flake per section 7.2
(no environmental luck). The pin at 67f8e89 does not touch this path.

Fix: arm a refd 25 ms keep-alive interval around the afterNextFlush await
(cleared in finally) so the debounce fires deterministically on every host.
Test-only diff; production settle semantics untouched.
…ts ref-tracking fix (#4599)

PR #5262 fixes #4788 at the source: scheduleSave now leaves the debounce
timer refd while flush waiters exist (and afterNextFlush refs a pending
unrefd timer when a waiter registers), so the settle is deterministic for
EVERY caller on every host. The 25 ms keep-alive interval added in
a3019b0 only patched this one test file and is redundant now; removing it
leaves one mechanism instead of two. The comment stays: without it a future
reader re-adds the interval or deletes the persist.ts tracking as dead weight.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b) Behind-sync round 2: master@44067a5bc was already contained in the branch; merged the advanced ci-fixes@06ad804ac (persist.ts/shadow.ts red-lane fixes) -> head 9738657, zero conflicts, PR diff intact; fresh CI run registered, all checks queued behind the shared-runner backlog, no failures.

@ranxianglei
ranxianglei marked this pull request as ready for review October 31, 2026 00:27
@ranxianglei
ranxianglei merged commit caaab21 into master Oct 31, 2026
56 of 90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants