Repository navigation
[PR] fix: map search evidence offsets back to original Unicode text - #6494
Conversation
[bot] (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 无其他 open/closed issue 或 PR 覆盖同一机制(镜像 + GitHub 检索 本轮独立复核(重启后重跑,非复读 #6493 结论):
审核结论: 修复打在根因层(坐标系混用:匹配在小写串坐标、切片在原始串坐标),保留整串小写匹配语义(希腊末位 sigma 由控制测试钉住), 八问:
🔶 需 owner 决策 — kernel 侧同机制残留(不阻塞本 PR):
行为变更披露(old → new): 对含小写化扩展字符(实际主要是土耳其语 İ)的摘要,search_context 的证据窗口与【高亮】原先用小写串偏移套在原始文本上 → 窗口可能变空( 判定: 可合入(CI 全绿、mergeable、无冲突;合并按政策由人工执行)。kernel 侧残留待你 A/B 拍板,不阻塞本 PR。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Review verdict: mergeable (merge is human-only per policy). The full repro-evidence review is on floor 9 above; this is the properly-formatted conclusion. Duplicate check: not a duplicate — no other open/closed issue or PR covers this mechanism. Nearest neighbors #6495/#6496 (cache-diff byte-vs-code-unit offsets) are the same "offset coordinate mismatch" family but a different subsystem and a different mismatch type (UTF-8 bytes vs UTF-16 units, not lowercase expansion). Prior art #2751 (evidence windowing) and #816/#828/#1615 (surrogate-safe clipping) cover adjacent layers only. Independently re-verified this session (post-restart, not a copy of the #6493 review):
The fix addresses the root cause (coordinate-system mismatch: matching in lowercase-string coordinates, slicing in original-text coordinates) at the right layer, keeps whole-string lowercase semantics (Greek final sigma pinned by a control test), uses a bounded O(n) code-point walk (no offset table), and never splits a surrogate pair or an expanded lowercase character. Both consumer sites ( 八问:
What remains (does not block this PR): the same mechanism also exists in the kernel's Bottom line: mergeable — CI green, mergeable, no conflict. Merge is human-only per policy. |
The suite baseline preload (preload-storage-baseline.mjs) pins BILI_STORAGE_UNIFIED="legacy" for the broad corpus. After ranxianglei#6480 flipped the storage default to pure-sqlite, the notice-registry suite's visibleNotices assertions expect the pure-sqlite default notice, which is gated on unifiedSelectionIsSqlite (unified "sqlite" or absent) — the preload's "legacy" made the notice silent, so the suite was born-red on master. Pin the selection explicitly (ranxianglei#4011 lesson: a test that depends on the production default passes locally but fails in CI).
…test The dsh plugin lane (ci-e2e-dsh.yml) runs node --import tsx --test WITHOUT the preload-storage-baseline.mjs preload, so BILI_STORAGE_UNIFIED is unset and the storage mode defaults to 'full'. Under the pure-sqlite default flip (5a80e2b, ranxianglei#6480), 'full' mode turns the JSON envelope writer OFF — so the .content-store.json sidecar is never written, and the test's readdirSync(namespace) hits ENOENT. Pin BILI_STORAGE_JSON_WRITER=on in the test so the content-store sidecar is written regardless of the storage mode. This keeps the corrupted-store detection path exercised in both full and legacy modes. Closes: dsh plugin lane red on master (run 43532980164, job 132255459726)
The pure-sqlite default flip (5a80e2b, ranxianglei#6480) turns the JSON envelope writer OFF, so the .json session file this lane asserts on is never written and 'proxy must persist the session' / 'no persisted session carried compressed blocks' fail after the 20s wait. Pin the writer ON in the spawned client's env (the bootstrapped proxy inherits it) so the envelope this test was written for is still produced. Closes: native opencode lane red on master (run 43547593803, job 132302438610)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode lane: the JSON envelope writer is OFF, so 'proxy must persist the session within 20000ms' fails — the on-disk .json session file is never written. Pin the writer ON in the spawned dsh client's env (the bootstrapped proxy inherits it). Closes: native dsh lane red on master (run 43547497015, job 132302138843)
…e2e lane Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) as the opencode/dsh lanes: the JSON envelope writer is OFF, so 'session persisted' fails — the on-disk .json session file is never written. Pin the writer ON in the spawned pi client's env (the bootstrapped proxy inherits it). Closes: pi codex-responses WS lane red on master (run 43547626291, job 132302540960)
…lane 4th lane of the pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope writer is OFF, so 'proxy must persist the session' and 'at least one persisted session must carry compressed blocks' fail — the on-disk .json session file is never written. Pin the writer ON in the spawned pi client's env (the bootstrapped proxy inherits it). Closes: pi package-native lane red on ci-fixes (run 43558179054, job 132336647955)
Same pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON writer is OFF, so the per-session CCR content-store sidecar (.content-store.json) that 'child persists its own companion and survives a proxy restart' asserts on is never written (0 !== 1). The earlier b1bc17b pin only covered tests/plugin-fork.test.ts, not this file. Pin the writer ON at module load (before any SessionStore is constructed). Closes: dsh plugin lane red on ci-fixes (run 43558179102, job 132336519539)
…lane Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope writer is OFF, so 'proxy must persist the session' fails — the on-disk .json session file is never written. Pin the writer ON in the spawned opencode client's env (the bootstrapped proxy inherits it).
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope writer is OFF, so the on-disk .json session envelope this lane asserts on (exactly ONE persisted session under the raw key) is never written. Pin the writer ON in the spawned dsh client's env.
Pure-sqlite default flip (5a80e2b, ranxianglei#6480) family: the JSON envelope writer is OFF, so the on-disk .json session file this lane asserts on (state survival across the OLD→NEW update) is never written. Pin the writer ON in the spawned proxy's env (both update phases share envBase).
Resolve CI baseline conflicts using the current upstream test setup.
ranxianglei
left a comment
There was a problem hiding this comment.
APPROVE — the host-side fix is correct, well-tested, and clean.
The fix maps search-evidence offsets found in summary.toLowerCase() back to the original text before windowing/highlighting, so length-expanding lowercasing (İ U+0130, ſ U+017F) can no longer shift the evidence window, highlight the wrong characters, or split surrogate pairs. I hand-traced the mapping and verified it empirically rather than trusting the diff alone:
- Correctness: the span mapping is sound because the only length-changing lowercasing rules (İ, ſ) are context-free (per-char == whole-string); Greek sigma's contextual rule is length-preserving, so matching stays whole-string while only the length mapping is per-char. All 4 Unicode cases traced correctly; boundary cases (match at offset 0, partial expansion, match straddling an expansion) verified.
- Tests have teeth: with the base
decompress-shared.tsswapped in, exactly 4 of the 7 new tests fail (the 4 real regressions) and 3 controls pass — matching the PR's claim. With the fix, 48/48 pass across the 5 search suites (run with the CI preload chain). - Regression audit: the change touches only
pickAnchorTerm(window placement) andmarkFirstMatch(highlight span), both consumed solely byevidenceSnippet(the search preview). Ranking (freqon the lowercased hay), candidate disclosure, topic-only fallback, and the wire format are all unchanged. No config, env, kernel, or version change. - Behavior change (disclosed, old→new): for text containing length-expanding lowercase characters, the evidence window position changes — old: window positioned at the lowercased offset (shifted right by the cumulative expansion; could be empty or highlight the wrong chars); new: window positioned at the original offset (correct). This is the intended fix; the old behavior was broken.
已顺手修:kernel/src/search/index.ts makePreview 同类偏移缺陷(独立包 acp-kernel,无推送到本 PR fork 分支的权限,故单独 PR #6686,human-gated)+ commit 889440d — makePreview 与本 PR 同源缺陷(命中在 toLowerCase() 里找、却用该偏移切原始文本),已实测复现(400× İ 前缀 → 预览变空 …)。按本 PR 的映射做机械移植,单独 PR 走 human-only review。
八问
- 是什么问题:search_context 的证据窗口/高亮偏移在 Unicode 小写化会拉长字符(İ、ſ)时漂移——命中索引在 lowercased 文本里算,却被直接用来切原始文本,导致窗口右移、高亮错位、甚至真实命中却返回空片段。
- 什么原因导致的:
toLowerCase()对 İ(U+0130)、ſ(U+017F) 把 1 个 UTF-16 单元扩成 2 个;累积的扩展量把 lowercased 偏移整体右移,拿去索引原始串就错位。 - 影响范围与严重程度:仅影响 search_context 预览的证据窗口/高亮位置;触发条件是文本含扩展型小写字符且命中点前存在这些字符。属体验瑕疵(窗口错位/偶发空片段),不影响检索命中、排序、数据或 wire。
- 怎么修复的:新增
originalLowercaseSpan,把 lowercased 的 [start,end) 区间按逐字符小写长度映射回原始文本坐标,pickAnchorTerm与markFirstMatch改用映射后的原始偏移再做窗口/高亮。映射精确,因为唯一变长的小写规则(İ、ſ)上下文无关(逐字符==整串),希腊 sigma 的上下文规则保长、不影响偏移算术。 - 是否有回归问题:审过的既有行为——排序(freq 基于 lowercased hay,未动)、候选披露、topic-only 回退、wire 格式、其余 41 个 search 测试——全部保持;唯一变化即上述证据窗口位置(有意变更,已披露)。无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
- 是否破坏缓存:不涉及。预览按调用即时计算,无缓存/重试/游标/去重键。
- 是否改了设计方向:不涉及。仅修坐标映射,不改架构/边界/数据流。
- 是否引入了新的配置:不涉及。无新 env/config/默认值。
判定:可合(APPROVE,auto-merge eligible) — 单模块、有回归测试、rebased head 绿、无 kernel/config/wire/版本变更、引用 issue(Closes #6493)、diff 干净。
|
Superseded by #6677 (merged as Landed into master via the real merge commit The fix for #6493 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix. |
Closes #6493
Problem
search_context locates evidence in a lowercased summary but uses those offsets and the lowercased term length to slice the original text. Lowercasing can expand a character: İ becomes i plus a combining dot. Prefix expansions shift the evidence window and highlight; a sufficiently long prefix produces an empty snippet for a real match. A shifted highlight can also split a surrogate pair, while expansion inside the matched word includes unrelated following text.
Change
Map the selected lowercase span back to original code-point boundaries before positioning the evidence window or inserting highlight markers. Matching still uses whole-string lowercase, preserving contextual Greek final sigma and existing substring semantics. The mapping uses a bounded linear walk without a full offset table. Search ranking, candidate disclosure, topic-only matches and kernel code stay unchanged.
Validation
Prior work
Checked all 61 Afloat16 PRs, all 161 fork branches, the complete decompress-shared history and upstream PR/issue searches for search_context, evidenceSnippet, pickAnchorTerm, Unicode snippets and lowercase offsets. #2787 adds the evidence window and #816/#1615 protect fixed clipping boundaries; none maps lowercase offsets back to the original text. No equivalent patch was found.
Current master integration
Merged upstream master
fcc2e848486cc2152e36b6658c517fbfa51ec428into this branch without textual conflicts. The published merge commit isa74d4eb93224456a2b4998a68a668c06e6409da8, with the original PR head1dbe1d7c3c7ca22a182a46d871a41c83dc544370as its first parent and current master as its second parent.The Unicode fix and its regression coverage are retained. The merged
decompress-shared.tsalso preserves upstream's independent spill-file ownership changes (randomUUID()names and exclusivewxwrites).Fresh integration validation on Linux x64 / Node 24.22.1, after installing the locked dependencies:
npm run typecheck,npm run build,npm run check:unused-exports,npm run check:test-import-graphandgit diff --checkpass.36972761b3482ed0c557f28312bae0b6e67b72bbexactly matches the locally tested tree.The integration commit triggered hosted CI run 43377350539; its current result is available there. Complete hosted CI and real-model E2E are not claimed as passed by these local integration checks.
Review status
Linked to #6493 and ready for maintainer review. The latest integration validation includes upstream master
fcc2e848486cc2152e36b6658c517fbfa51ec428, as recorded above. Current hosted results are available in the Checks tab.Please apply the existing
ci:fulllabel to run the FULL matrix, including the required Windows / Node 24 checks. The default CORE matrix does not produce those checks.