Skip to content

[PR] fix: map search evidence offsets back to original Unicode text - #6494

Merged
ranxianglei merged 16 commits into
ranxianglei:masterfrom
Afloat16:2026-11-12_search-unicode-offsets
Nov 14, 2026
Merged

ranxianglei merged 16 commits into
ranxianglei:masterfrom
Afloat16:2026-11-12_search-unicode-offsets

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #6493

Problem

search_context locates evidence in a lowercased summary but uses those offsets and the lowercased term length to slice the original text. Lowercasing can expand a character: İ becomes i plus a combining dot. Prefix expansions shift the evidence window and highlight; a sufficiently long prefix produces an empty snippet for a real match. A shifted highlight can also split a surrogate pair, while expansion inside the matched word includes unrelated following text.

Change

Map the selected lowercase span back to original code-point boundaries before positioning the evidence window or inserting highlight markers. Matching still uses whole-string lowercase, preserving contextual Greek final sigma and existing substring semantics. The mapping uses a bounded linear walk without a full offset table. Search ranking, candidate disclosure, topic-only matches and kernel code stay unchanged.

Validation

  • Independent base: 9674cfa.
  • Unchanged production code: 4 new regressions fail and 3 controls pass. The cases cover a distant match after 400 expanding characters, a short shifted highlight, expansion within İstanbul and an astral-letter match whose surrogate pair was split. Controls cover a partial expanded character, ASCII case-insensitive matching and Greek final sigma.
  • Fixed code: all 48 tests pass across search-context-unicode-offsets.test.ts, search-context.test.ts, search-context-2751.test.ts, search-plan-aware.test.ts and search-context-cross-session.test.ts. Existing tests include summary/original-text lookup, surrogate-safe excerpts, cross-session scope and loopback Anthropic plan-aware search transport.
  • Full npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph and git diff --check pass.
  • Zero-token Codex preflight finds codex-cli 0.167.0-alpha.4 and the built dist artifact. Its local upstream /models probe is unreachable; the real-model E2E case is skipped. Full upstream CI and real-model E2E are not claimed as passed.
  • Tested on Linux x64 with Node 24.22.1. No new configuration, environment variables, kernel changes or version bump.

Prior work

Checked all 61 Afloat16 PRs, all 161 fork branches, the complete decompress-shared history and upstream PR/issue searches for search_context, evidenceSnippet, pickAnchorTerm, Unicode snippets and lowercase offsets. #2787 adds the evidence window and #816/#1615 protect fixed clipping boundaries; none maps lowercase offsets back to the original text. No equivalent patch was found.

Current master integration

Merged upstream master fcc2e848486cc2152e36b6658c517fbfa51ec428 into this branch without textual conflicts. The published merge commit is a74d4eb93224456a2b4998a68a668c06e6409da8, with the original PR head 1dbe1d7c3c7ca22a182a46d871a41c83dc544370 as its first parent and current master as its second parent.

The Unicode fix and its regression coverage are retained. The merged decompress-shared.ts also preserves upstream's independent spill-file ownership changes (randomUUID() names and exclusive wx writes).

Fresh integration validation on Linux x64 / Node 24.22.1, after installing the locked dependencies:

  • All 48/48 tests across the focused suites above pass.
  • npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph and git diff --check pass.
  • The published tree 36972761b3482ed0c557f28312bae0b6e67b72bb exactly matches the locally tested tree.

The integration commit triggered hosted CI run 43377350539; its current result is available there. Complete hosted CI and real-model E2E are not claimed as passed by these local integration checks.

Review status

Linked to #6493 and ready for maintainer review. The latest integration validation includes upstream master fcc2e848486cc2152e36b6658c517fbfa51ec428, as recorded above. Current hosted results are available in the Checks tab.

Please apply the existing ci:full label to run the FULL matrix, including the required Windows / Node 24 checks. The default CORE matrix does not produce those checks.

@ranxianglei ranxianglei changed the title fix: map search evidence offsets back to original Unicode text [PR] fix: map search evidence offsets back to original Unicode text Nov 13, 2026
@Afloat16
Afloat16 marked this pull request as ready for review November 13, 2026 04:45
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 无其他 open/closed issue 或 PR 覆盖同一机制(镜像 + GitHub 检索 search_context unicode / evidenceSnippet / pickAnchorTerm 历史)。最近邻 #6495/#6496(cache-diff 字节 vs 码元偏移)同属「偏移坐标系错位」家族,但子系统不同(离线 cache-diff 诊断)、错位类型不同(UTF-8 字节 vs UTF-16 码元,非小写化扩展)。先例 #2751(证据窗口)、#816/#828/#1615(代理对安全裁剪)只覆盖相邻层,均未做小写偏移回映。

本轮独立复核(重启后重跑,非复读 #6493 结论):

  • 逐行重读 PR 自身 commit 1dbe1d7c 的 diff(2 文件 +71/−2),并对照 PR 头 a74d4eb9 的 src/decompress-shared.ts 全文确认 merge commit 为纯集成、未改动修复内容。
  • 重跑 CI 状态(head a74d4eb9): 78 success / 42 skipped / 36 cancelled / 0 failed / 0 pending(cancelled 为 base-update 重触发前的过期 run,非失败)。
  • 重查 PR 状态: draft: False(作者已转 ready)、mergeable: true、mergeable_state: behind(master 领先、零文件重叠、无冲突)。
  • 独立验证 kernel 侧同机制残留确实存在:kernel/src/search/index.ts:161-188 的 makePreview 同样用 text.toLowerCase() 的下标去切原始 text(行 174/178/184-186),与 host 侧同根因。

审核结论: 修复打在根因层(坐标系混用:匹配在小写串坐标、切片在原始串坐标),保留整串小写匹配语义(希腊末位 sigma 由控制测试钉住),originalLowercaseSpan() 用有界 O(n) 码位遍历回映、不建全量偏移表、不拆代理对/不拆扩展小写字符;两处消费点(pickAnchorTerm 证据窗口、markFirstMatch 高亮)都已接上,host 侧无遗漏。4 条回归测试有牙(master 红、fix 绿、精确串断言),3 条控制钉住既有语义。diff 最小干净、测试风格与既有一致,达外部贡献质量门槛,无需重构。可合入(合并按政策由人工执行)。

八问:

  1. 是什么问题: search_context 在小写化后的摘要里定位命中,却拿该下标去切原始摘要;小写化可能改变长度(İ → i+组合点),下标错位使证据窗口偏移/越界变空、高亮盖到无关字符或吞相邻文本。
  2. 什么原因导致的: 坐标系混用——匹配在小写串坐标、切片在原始串坐标,仅当小写化保长时两者等价;既有窗口/代理对裁剪修复都没做坐标回映。
  3. 影响范围与严重程度: 仅 search_context 返回的预览文本(模型可见),不影响命中/排序/解压;触发条件 = 摘要含小写化扩展字符(实际主要是土耳其语 İ)。体验瑕疵,p3。
  4. 怎么修复的: 保留整串小写匹配,新增 originalLowercaseSpan() 把选中小写区间按码位宽度回映到原始码位边界,再定位窗口与高亮;两处消费点都接上。
  5. 是否有回归问题: 逐项审过——ASCII 大小写不敏感匹配/高亮(不变,控制钉住)、整串小写匹配含希腊末位 sigma(不变,控制钉住)、窗口预算 200/80(不变)、safeMidSlice 代理对纪律(不变)、排序/收窄提示/截断说明/原文语料(不变,仅 includes)。唯一变更即修复本身。
  6. 是否破坏缓存: 不涉及——只改工具结果预览文本,不碰 prompt cache/重试/增量游标/去重键。
  7. 是否改了设计方向: 不涉及——同架构同数据流,仅加坐标回映。
  8. 是否引入新配置: 无。

🔶 需 owner 决策 — kernel 侧同机制残留(不阻塞本 PR): kernel/src/search/index.ts:161-188 的 makePreview(search_context 的 "Original-text hit(s)" 段预览)同样用小写串下标切原始文本,400×İ 前缀下原文预览输出空窗口,与 host 侧同症状同根因。

行为变更披露(old → new): 对含小写化扩展字符(实际主要是土耳其语 İ)的摘要,search_context 的证据窗口与【高亮】原先用小写串偏移套在原始文本上 → 窗口可能变空(... )或高亮盖无关字符/吞相邻文本/拆代理对;现回映到原始码位边界 → 窗口与高亮落在真实命中的原文上。其余(排序、频率、收窄提示、截断说明、窗口预算 200/80、safeMidSlice 代理对纪律、topic-only 命中、原文语料)全部不变。

判定: 可合入(CI 全绿、mergeable、无冲突;合并按政策由人工执行)。kernel 侧残留待你 A/B 拍板,不阻塞本 PR。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Review verdict: mergeable (merge is human-only per policy). The full repro-evidence review is on floor 9 above; this is the properly-formatted conclusion.

Duplicate check: not a duplicate — no other open/closed issue or PR covers this mechanism. Nearest neighbors #6495/#6496 (cache-diff byte-vs-code-unit offsets) are the same "offset coordinate mismatch" family but a different subsystem and a different mismatch type (UTF-8 bytes vs UTF-16 units, not lowercase expansion). Prior art #2751 (evidence windowing) and #816/#828/#1615 (surrogate-safe clipping) cover adjacent layers only.

Independently re-verified this session (post-restart, not a copy of the #6493 review):

  • Re-read the PR's own commit 1dbe1d7c diff (2 files, +71/−2) and the full PR-head src/decompress-shared.ts; traced originalLowercaseSpan() by hand across the expanding-prefix / within-term / single-char / surrogate-pair cases — all correct. Merge commit a74d4eb9 is a pure integration (no fix content altered).
  • CI on head a74d4eb9: 78 success / 42 skipped / 36 cancelled / 0 failed (cancelled = superseded stale runs, not failures).
  • PR state: draft: False, mergeable: true, mergeable_state: behind (master ahead, zero file overlap, no conflict).
  • Confirmed the kernel-side residual independently: kernel/src/search/index.ts:161-188 makePreview has the same lowercase-offset-slicing-original mechanism.

The fix addresses the root cause (coordinate-system mismatch: matching in lowercase-string coordinates, slicing in original-text coordinates) at the right layer, keeps whole-string lowercase semantics (Greek final sigma pinned by a control test), uses a bounded O(n) code-point walk (no offset table), and never splits a surrogate pair or an expanded lowercase character. Both consumer sites (pickAnchorTerm evidence window, markFirstMatch highlight) are wired — no host-side gap. 4 regression tests have teeth (red on master, green on fix, exact-string asserts); 3 controls pin existing semantics. Diff is minimal and clean; meets the external-contribution bar — no rework needed.

八问:

  1. 是什么问题: search_context 在小写化后的摘要里定位命中,却拿该下标去切原始摘要;小写化可能改变长度(İ → i+组合点),下标错位使证据窗口偏移/越界变空、高亮盖到无关字符或吞相邻文本。
  2. 什么原因导致的: 坐标系混用——匹配在小写串坐标、切片在原始串坐标,仅当小写化保长时两者等价;既有窗口/代理对裁剪修复都没做坐标回映。
  3. 影响范围与严重程度: 仅 search_context 返回的预览文本(模型可见),不影响命中/排序/解压;触发条件 = 摘要含小写化扩展字符(实际主要是土耳其语 İ)。体验瑕疵,p3。
  4. 怎么修复的: 保留整串小写匹配,新增 originalLowercaseSpan() 把选中小写区间按码位宽度回映到原始码位边界,再定位窗口与高亮;两处消费点都接上。
  5. 是否有回归问题: 逐项审过——ASCII 大小写不敏感匹配/高亮(不变,控制钉住)、整串小写匹配含希腊末位 sigma(不变,控制钉住)、窗口预算 200/80(不变)、safeMidSlice 代理对纪律(不变)、排序/收窄提示/截断说明/原文语料(不变,仅 includes)。唯一变更即修复本身。
  6. 是否破坏缓存: 不涉及——只改工具结果预览文本,不碰 prompt cache/重试/增量游标/去重键。
  7. 是否改了设计方向: 不涉及——同架构同数据流,仅加坐标回映。
  8. 是否引入新配置: 无。

What remains (does not block this PR): the same mechanism also exists in the kernel's makePreview (kernel/src/search/index.ts:161-188) — the "Original-text hit(s)" preview section. Tracked as an open checklist item on #6493 (same symptom → same issue) awaiting your A/B call: A = open a kernel fix PR (needs kernel-change declaration + human review tier, separate PR); B = leave it (low-frequency). I recommend A; default is B until you decide.

Bottom line: mergeable — CI green, mergeable, no conflict. Merge is human-only per policy.

ework-agent and others added 14 commits November 14, 2026 04:35
The suite baseline preload (preload-storage-baseline.mjs) pins
BILI_STORAGE_UNIFIED="legacy" for the broad corpus. After ranxianglei#6480 flipped the
storage default to pure-sqlite, the notice-registry suite's visibleNotices
assertions expect the pure-sqlite default notice, which is gated on
unifiedSelectionIsSqlite (unified "sqlite" or absent) — the preload's
"legacy" made the notice silent, so the suite was born-red on master. Pin
the selection explicitly (ranxianglei#4011 lesson: a test that depends on the production
default passes locally but fails in CI).
…test

The dsh plugin lane (ci-e2e-dsh.yml) runs node --import tsx --test
WITHOUT the preload-storage-baseline.mjs preload, so BILI_STORAGE_UNIFIED
is unset and the storage mode defaults to 'full'. Under the pure-sqlite
default flip (5a80e2b, ranxianglei#6480), 'full' mode turns the JSON envelope
writer OFF — so the .content-store.json sidecar is never written, and
the test's readdirSync(namespace) hits ENOENT.

Pin BILI_STORAGE_JSON_WRITER=on in the test so the content-store
sidecar is written regardless of the storage mode. This keeps the
corrupted-store detection path exercised in both full and legacy modes.

Closes: dsh plugin lane red on master (run 43532980164, job 132255459726)
The pure-sqlite default flip (5a80e2b, ranxianglei#6480) turns the JSON envelope
writer OFF, so the .json session file this lane asserts on is never
written and 'proxy must persist the session' / 'no persisted session
carried compressed blocks' fail after the 20s wait. Pin the writer ON in
the spawned client's env (the bootstrapped proxy inherits it) so the
envelope this test was written for is still produced.

Closes: native opencode lane red on master (run 43547593803, job 132302438610)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode lane:
the JSON envelope writer is OFF, so 'proxy must persist the session
within 20000ms' fails — the on-disk .json session file is never written.
Pin the writer ON in the spawned dsh client's env (the bootstrapped
proxy inherits it).

Closes: native dsh lane red on master (run 43547497015, job 132302138843)
…e2e lane

Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode/dsh
lanes: the JSON envelope writer is OFF, so 'session persisted' fails —
the on-disk .json session file is never written. Pin the writer ON in
the spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi codex-responses WS lane red on master (run 43547626291, job 132302540960)
…lane

4th lane of the pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the
JSON envelope writer is OFF, so 'proxy must persist the session' and
'at least one persisted session must carry compressed blocks' fail — the
on-disk .json session file is never written. Pin the writer ON in the
spawned pi client's env (the bootstrapped proxy inherits it).

Closes: pi package-native lane red on ci-fixes (run 43558179054, job 132336647955)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON writer
is OFF, so the per-session CCR content-store sidecar (.content-store.json)
that 'child persists its own companion and survives a proxy restart'
asserts on is never written (0 !== 1). The earlier b1bc17b pin only
covered tests/plugin-fork.test.ts, not this file. Pin the writer ON at
module load (before any SessionStore is constructed).

Closes: dsh plugin lane red on ci-fixes (run 43558179102, job 132336519539)
…lane

Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so 'proxy must persist the session' fails — the on-disk
.json session file is never written. Pin the writer ON in the spawned
opencode client's env (the bootstrapped proxy inherits it).
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so the on-disk .json session envelope this lane asserts on
(exactly ONE persisted session under the raw key) is never written. Pin
the writer ON in the spawned dsh client's env.
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope
writer is OFF, so the on-disk .json session file this lane asserts on
(state survival across the OLD→NEW update) is never written. Pin the
writer ON in the spawned proxy's env (both update phases share envBase).
Resolve CI baseline conflicts using the current upstream test setup.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — the host-side fix is correct, well-tested, and clean.

The fix maps search-evidence offsets found in summary.toLowerCase() back to the original text before windowing/highlighting, so length-expanding lowercasing (İ U+0130, ſ U+017F) can no longer shift the evidence window, highlight the wrong characters, or split surrogate pairs. I hand-traced the mapping and verified it empirically rather than trusting the diff alone:

  • Correctness: the span mapping is sound because the only length-changing lowercasing rules (İ, ſ) are context-free (per-char == whole-string); Greek sigma's contextual rule is length-preserving, so matching stays whole-string while only the length mapping is per-char. All 4 Unicode cases traced correctly; boundary cases (match at offset 0, partial expansion, match straddling an expansion) verified.
  • Tests have teeth: with the base decompress-shared.ts swapped in, exactly 4 of the 7 new tests fail (the 4 real regressions) and 3 controls pass — matching the PR's claim. With the fix, 48/48 pass across the 5 search suites (run with the CI preload chain).
  • Regression audit: the change touches only pickAnchorTerm (window placement) and markFirstMatch (highlight span), both consumed solely by evidenceSnippet (the search preview). Ranking (freq on the lowercased hay), candidate disclosure, topic-only fallback, and the wire format are all unchanged. No config, env, kernel, or version change.
  • Behavior change (disclosed, old→new): for text containing length-expanding lowercase characters, the evidence window position changes — old: window positioned at the lowercased offset (shifted right by the cumulative expansion; could be empty or highlight the wrong chars); new: window positioned at the original offset (correct). This is the intended fix; the old behavior was broken.

已顺手修:kernel/src/search/index.ts makePreview 同类偏移缺陷(独立包 acp-kernel,无推送到本 PR fork 分支的权限,故单独 PR #6686,human-gated)+ commit 889440d — makePreview 与本 PR 同源缺陷(命中在 toLowerCase() 里找、却用该偏移切原始文本),已实测复现(400× İ 前缀 → 预览变空 …)。按本 PR 的映射做机械移植,单独 PR 走 human-only review。

八问

  1. 是什么问题:search_context 的证据窗口/高亮偏移在 Unicode 小写化会拉长字符(İ、ſ)时漂移——命中索引在 lowercased 文本里算,却被直接用来切原始文本,导致窗口右移、高亮错位、甚至真实命中却返回空片段。
  2. 什么原因导致的:toLowerCase() 对 İ(U+0130)、ſ(U+017F) 把 1 个 UTF-16 单元扩成 2 个;累积的扩展量把 lowercased 偏移整体右移,拿去索引原始串就错位。
  3. 影响范围与严重程度:仅影响 search_context 预览的证据窗口/高亮位置;触发条件是文本含扩展型小写字符且命中点前存在这些字符。属体验瑕疵(窗口错位/偶发空片段),不影响检索命中、排序、数据或 wire。
  4. 怎么修复的:新增 originalLowercaseSpan,把 lowercased 的 [start,end) 区间按逐字符小写长度映射回原始文本坐标,pickAnchorTerm 与 markFirstMatch 改用映射后的原始偏移再做窗口/高亮。映射精确,因为唯一变长的小写规则(İ、ſ)上下文无关(逐字符==整串),希腊 sigma 的上下文规则保长、不影响偏移算术。
  5. 是否有回归问题:审过的既有行为——排序(freq 基于 lowercased hay,未动)、候选披露、topic-only 回退、wire 格式、其余 41 个 search 测试——全部保持;唯一变化即上述证据窗口位置(有意变更,已披露)。无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及。预览按调用即时计算,无缓存/重试/游标/去重键。
  7. 是否改了设计方向:不涉及。仅修坐标映射,不改架构/边界/数据流。
  8. 是否引入了新的配置:不涉及。无新 env/config/默认值。

判定:可合(APPROVE,auto-merge eligible) — 单模块、有回归测试、rebased head 绿、无 kernel/config/wire/版本变更、引用 issue(Closes #6493)、diff 干净。

@ranxianglei
ranxianglei merged commit a77a383 into ranxianglei:master Nov 14, 2026
50 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit a77a3830dba (Merge Afloat16/2026-11-12_search-unicode-offsets (PR #6494)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6493 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

search_context evidence offsets drift when Unicode lowercasing expands characters

2 participants