Skip to content

[PR] kernel: map search preview offsets back to original Unicode text - #6686

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-11-14_kernel-preview-unicode-offsets
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-11-14_kernel-preview-unicode-offsets

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

kernel-change: fix lowercased-offset drift in search preview windowing — mechanical port of the host-side fix from #6494 (kernel-side facet of #6493)

What

makePreview (kernel/src/search/index.ts) located the first query-term hit in text.toLowerCase() but windowed the original text at that lowercased offset. Lowercasing expands U+0130 (İ) and U+017F (ſ) from one to two UTF-16 units, so any expanding character before the hit shifted the preview window right. For long text the window slid past the end of the string and the preview came back empty (…) for a real match.

This is the kernel-side facet of the same defect family as #6493 (host-side search_context evidence snippets, fixed in #6494). Host and kernel are separate packages with separate version cadences, so the kernel fix ships here rather than in #6494.

Fix

Map the hit offset back to the original text before subtracting the half-window (originalOffsetOf) — the same span-mapping #6494 uses. The mapping is exact because the only length-changing lowercasing rules (İ, ſ) are context-free (per-character == whole-string); contextual rules (Greek sigma) are length-preserving and do not affect the offset arithmetic.

Why mechanically safe

  • One helper + a two-line change in makePreview; no change to scoring, ranking, matching, or any other preview path.
  • For text with no length-expanding characters the mapping is the identity, so every existing preview is byte-identical (pinned by the existing "preview centers on the matched term" test, still green).
  • clampWindow already snaps both edges off surrogate pairs, so the mapped start stays surrogate-safe.
  • No config, env, wire, or persistence change.

Regression tests

Two new tests in kernel/tests/search.test.ts, both verified to FAIL on the pre-fix windowing and PASS after:

  1. 400× İ prefix + distant hit → preview must contain the hit (pre-fix: empty …).
  2. 50× İ + 100× x prefix + mid-text hit → preview must contain the hit (pre-fix: window shifted onto the trailing y's).

kernel 0.0.197 → 0.0.198 (CI-enforced version discipline for kernel/src/** changes).

Human-only review tier applies (Kernel Boundary rule 3); this is a mechanical-bug-exemption fix (owner policy 2026-11-06) — no pre-approval issue, reproducing regression tests included.

@ranxianglei ranxianglei changed the title kernel: map search preview offsets back to original Unicode text [PR] kernel: map search preview offsets back to original Unicode text Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-14_kernel-preview-unicode-offsets (f408d6a)

Latest build — install this exact version

npm install -g billion-context@0.0.11159-pr.6686.419701f

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
f408d6a billion-context@0.0.11159-pr.6686.419701f build
fb74592 billion-context@0.0.10901-pr.6686.ba06eb6 build
966b5e7 billion-context@0.0.10895-pr.6686.0aa576e build
fab2ecb billion-context@0.0.9956-pr.6686.32dca0b build
889440d billion-context@0.0.9944-pr.6686.cb43db2 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6686.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/qwen3.8-27b)

查重: 不重复 —— 本 PR 是 #6493 的 kernel 侧切面,host 侧修复在 #6494。两者分属不同 npm 包(acp-kernel 与 billion-context)、独立版本节奏,按包边界拆分成立;#6494 目前仍 open 未合,但两边触碰的文件不相交,无合入顺序依赖。

Review 结论:代码正确、测试有效、diff 干净 —— 可合(human-only tier,等 owner 合并)。 以下为独立验证,不是复述 PR 描述:

  1. Bug 实锤 —— 修复前 makePreview(kernel/src/search/index.ts:170-183)在 text.toLowerCase() 上 indexOf 得到 hitIdx,却用它切原始 text。hit 前每有一个 İ(lowercase 1→2 UTF-16 units),窗口就右移一个单位;长文本下 start > text.length,clampWindow 返回空,真实命中的 preview 只剩 "…"。
  2. 本地复现 —— 把 origin/master 的 kernel/src/search/index.ts 换回 pre-fix 版本跑 kernel/tests/search.test.ts:54 个测试恰好新加的 2 个失败(not ok 11/12);恢复 PR 代码后 54/54 全过。PR head 上 kernel 单元套件整体 1638/1638 绿,含既有 "preview centers on the matched term" identity 钉(无长度变化字符时 preview 逐字节不变)。
  3. 映射正确性 —— originalOffsetOf 按 code point 累加 char.toLowerCase().length / char.length。我扫了全 Unicode code point 表(Node 22/V8):toLowerCase 改变长度的只有 U+0130 (İ, 1→2) 一个,收缩为零;唯一上下文相关规则(Greek sigma)长度保持,不影响偏移算术。算法对任意 per-character 长度变化都精确,边界处理(next <= lowerOffset)正确。
  4. surrogate 安全 —— originalOffsetOf 按 code point 迭代,返回值恒在 code point 边界;clampWindow(kernel/src/truncate.ts:24-30)再把两端 snap 离 surrogate half,双保险。
  5. 无其他受影响路径 —— kernel 内 24 处 toLowerCase 全部 grep 过:其余均为打分/计数/tokenize,makePreview 是唯一用 lowercased 偏移切原文的位置。
  6. 合规 —— PR 正文 kernel-change: 声明 ✓、kernel: commit 前缀 ✓、版本 0.0.197→0.0.198(CI-enforced)✓、mechanical-bug 豁免条件满足(复现回归测试 + 声明 + 线程内报告)✓。

顺手小修(独立 commit fab2ecbee2):PR 的 doc comment 写 "Lowercasing only changes length for U+0130 (İ) and U+017F (ſ)" —— 全表扫描证实 U+017F (ſ) 在 toLowerCase 下是固定点(1→1,长度不变),原注释与代码事实不符(代码逻辑不受影响,算法对任意长度变化都成立)。已把两处注释改为只列 U+0130。

CI:检查时 PR 的 47 个 check 全部 queued(PR 刚开);我推注释修复后重新入队。最终绿/红以远程 CI 为准。


结构化八问:

  1. 是什么问题:kernel 搜索 preview 用「lowercase 后字符串的偏移」去切「原始字符串」。文本含 lowercase 会膨胀的字符(İ)时,preview 窗口右移,长文本下真实命中的 preview 变空 "…"。
  2. 什么原因导致的:makePreview 在 text.toLowerCase() 上定位 hit,却拿该偏移直接切原始 text——两串长度不同(İ 1→2 UTF-16 units),偏移坐标系不一致。与 host 侧 search_context evidence offsets drift when Unicode lowercasing expands characters #6493(search_context evidence snippet)同一根机制,不同包。
  3. 影响范围与严重程度:仅影响 searchBlocks/searchBlocksAsync 结果 preview 的展示——文本在 hit 前有 İ 时窗口偏移或为空;匹配、打分、排序、其余行为均不受影响。属展示瑕疵(模型仍能定位到 block,但 preview 证据错位/为空),p3 级。
  4. 怎么修复的:减半窗之前先把 hit 偏移映射回原始串(originalOffsetOf,per-code-point 累加,与 host 侧 [PR] fix: map search evidence offsets back to original Unicode text #6494 同一 span-mapping 思路);映射精确性由「唯一变长 lowercase 规则是 context-free 的 İ」保证。
  5. 是否有回归问题:审过的既有行为——① 无长度变化字符的文本:映射为恒等,preview 逐字节不变(既有 centering 测试钉住,绿);② 打分/排序/匹配/其余 preview 路径:未触碰;③ 无语义漂移、无 fail-fast 变吞错、无阈值/时序/输出格式变化。无回归。
  6. 是否破坏缓存:否——缓存、重试、增量游标、去重键均无改动;search doc-cache 未触碰。
  7. 是否改了设计方向:否——makePreview 内部纯 bug 修复,无架构/边界/数据流变化。
  8. 是否引入了新的配置:否——无新 env/config/默认值。

判定:修复机械安全,回归测试有牙(已验证 pre-fix 失败、post-fix 通过),diff 收敛到目的;唯一事实错误(注释里的 ſ)已在独立 commit 顺手修掉。kernel 变更适用 human-only review tier,可合,等 owner 合并(CI 绿为最终门禁)。

@ranxianglei
ranxianglei force-pushed the 2026-11-14_kernel-preview-unicode-offsets branch from fab2ecb to 966b5e7 Compare November 18, 2026 21:23
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/qwen3.8-27b)

Merge conflict resolved. Rebased the PR branch onto current master and force-pushed the final head.

What I did this run:

  • Fetched live heads: PR branch was at fab2ecbee2, master at a446a3c1 (439 commits ahead of the PR's merge-base 9495c764).
  • Rebasing surfaced exactly ONE conflict: kernel/package.json version. The PR bumped 0.0.197 → 0.0.198, but master had since advanced to 0.0.199 (via 399cd2b8c0 and 497efc7e31). The two code files (kernel/src/search/index.ts, kernel/tests/search.test.ts) had no master-side changes, so they rebased cleanly.
  • Resolved the version to 0.0.200 — the PR's kernel/src change is a NEW change on top of master's 0.0.199, so it bumps +1 from the new base (Kernel Boundary version discipline: any kernel/src/** change MUST bump kernel/package.json).
  • Verified locally before pushing: node --import tsx --test kernel/tests/search.test.ts → 54/54 pass, including the two new regression tests (preview: length-expanding lowercase (İ)… and preview: expanding prefix before a mid-text hit…) and the existing identity pin (preview centers on the matched term).
  • Confirmed master is now an ancestor of the head (2 commits ahead, no leftover conflict markers) → conflict resolved, PR clean.
  • Force-pushed the final head 966b5e7bc3 (no [ework:ci-held] marker — this is the one full CI pass). Remote head confirmed at 966b5e7bc3.

CI: the stale mirror cache (last polled 2026-11-14) showed no red checks — only the merge conflict was blocking. The final green/red verdict comes from the remote CI on the pushed head 966b5e7bc3.


结构化八问:

  1. 是什么问题:PR 分支落后 master 439 个提交,与目标分支产生合并冲突(merge state: dirty/blocked),无法合入。
  2. 什么原因导致的:PR 基于旧 merge-base(9495c764)开出,期间 master 前进了 439 个提交,其中两个 kernel 提交把 kernel/package.json 版本从 0.0.197 推到 0.0.199,与 PR 自身的 0.0.197 → 0.0.198 版本行冲突。代码/测试文件 master 侧无改动,故冲突只集中在版本号一行。
  3. 影响范围与严重程度:仅本 PR 的合入被阻塞(冲突),不影响任何已发布行为;属 CI/合入流程问题,非功能性缺陷。
  4. 怎么修复的:把 master rebase 进 PR 分支,版本冲突按「在新基线上 +1」解为 0.0.200(master 已到 0.0.199,PR 的 kernel/src 变更在其上再 +1),代码与测试逐字保留,force-push 最终 head。
  5. 是否有回归问题:无。代码/测试文件 master 侧零改动,rebase 后与 PR 原内容逐字一致;本地定向跑 kernel search 套件 54/54 绿(含 2 个新回归测试 + 既有 centering 恒等钉)。无语义漂移、无阈值/时序/输出格式变化。
  6. 是否破坏缓存:否——纯 rebase + 版本号解冲,未触碰任何缓存/重试/增量游标/去重键逻辑。
  7. 是否改了设计方向:否——无任何架构/边界/数据流变化,仅版本纪律下的版本号推进。
  8. 是否引入了新的配置:否——无新 env/config/默认值;唯一变更是 kernel/package.json 版本 0.0.199 → 0.0.200(由 master 基线推导,非新配置项)。

判定:冲突已解、定向测试绿、head 已推送,PR 现为 clean,等远程 CI 在 966b5e7bc3 上跑完即门禁。kernel 变更适用 human-only review tier,等 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/qwen3.8-27b)

lock-sync fixed. Re-verified live state first: on head 966b5e7bc3 the only failing check was lock-sync (16 skipped / 13 success / 1 failure); the merge conflict from the previous wake is already resolved (state was blocked = required-checks red, not dirty = conflict).

Root cause: my previous rebase resolved the version conflict to kernel/package.json = 0.0.200, but the PR (and the rebase) never synced the lockfiles, which embed the kernel version. The lock-sync job runs npm install --package-lock-only on both the root and kernel/ and fails on any git diff — the lockfiles still said 0.0.199.

Fix (one mechanical commit fb74592a93): regenerated both lockfiles with the exact CI command, updating the embedded kernel version 0.0.199 → 0.0.200:

  • kernel/package-lock.json — top-level version + packages[""].version
  • package-lock.json (root) — packages["kernel"].version (kernel is a file:./kernel devDependency)

No other drift in either lockfile (verified the diff is only the version lines).

Verified locally before pushing: ran both lock-sync checks — git diff --exit-code is clean (exit 0) for both kernel/package-lock.json and package-lock.json after regen. Pushed the final head fb74592a93 (fast-forward on top of 966b5e7bc3, no [ework:ci-held] marker — this is the branch's one full CI pass). Remote head confirmed at fb74592a93.

CI: the mirror cache still shows the previous run (966b5e7bc3); the new run on fb74592a93 will be picked up by the poller shortly. Final green/red verdict comes from remote CI on the pushed head.


结构化八问:

  1. 是什么问题:lock-sync 检查红——kernel 版本号在 package.json 与两份 lockfile 之间不一致。
  2. 什么原因导致的:上一轮 rebase 把版本冲突解为 kernel/package.json = 0.0.200,但 PR 与 rebase 都没同步 lockfile;lockfile 内嵌了 kernel 版本(kernel 作为 file:./kernel devDependency 同时出现在根 lockfile 的 packages["kernel"] 与 kernel 自身 lockfile 里),仍停在 0.0.199。lock-sync 对两份 lockfile 各跑 npm install --package-lock-only + git diff --exit-code,版本漂移即红。
  3. 影响范围与严重程度:仅本 PR 的 CI 门禁被卡(lock-sync 红),不影响任何已发布行为;属 lockfile 同步的机械漂移,非功能性缺陷。
  4. 怎么修复的:用 CI 同款命令重新生成两份 lockfile,把内嵌 kernel 版本 0.0.199 → 0.0.200 对齐 package.json;diff 仅版本行,无其他漂移。
  5. 是否有回归问题:无。改动只动 lockfile 里 kernel 的版本号字段,与 package.json 对齐;未触碰任何代码/配置/依赖解析。本地已验证两份 lockfile 在 regen 后 git diff --exit-code 均干净。
  6. 是否破坏缓存:否——纯 lockfile 版本字段同步,不涉及缓存/重试/增量游标/去重键。
  7. 是否改了设计方向:否——无架构/边界/数据流变化。
  8. 是否引入了新的配置:否——无新 env/config/默认值;仅 lockfile 版本字段对齐。

判定:lock-sync 已修、本地双 lockfile 检查绿、最终 head fb74592a93 已推送(无 ci-held 标记,触发整轮 CI)。等远程 CI 在该 head 上跑完即门禁;kernel 变更适用 human-only review tier,等 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/qwen3.8-27b)

No new work needed — this event reports a stale head. The lock-sync failure it cites is on 966b5e7bc3, which is no longer the branch head. That failure was already fixed in the prior run: the branch head is now fb74592a93 ("kernel: sync lockfiles to version 0.0.200"), which carries the lockfile sync.

Re-verified live in this run (not from memory):

  • Branch head = fb74592a93 (confirmed via git fetch + ls-remote), working tree clean.
  • Re-ran both lock-sync checks on fb74592a93: git diff --exit-code clean (exit 0) for both kernel/package-lock.json and root package-lock.json after npm install --package-lock-only — no drift.
  • Kernel version consistent at 0.0.200 across kernel/package.json, kernel/package-lock.json (top-level + packages[""]), and root package-lock.json → packages["kernel"].

The mirror cache is still showing the old head 966b5e7bc3 (last poll predates my push); the new CI run on fb74592a93 will confirm green once the poller catches up. Nothing to push — the fix is already at the head. Stopping without further work.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Independent review — PR #6686 (kernel: map search preview offsets back to original Unicode text)

I reviewed this from scratch (not from the earlier review thread). Verdict up front: the fix is correct, the regression tests have teeth, the diff is clean, and Kernel Boundary conventions are met. The only gate is the human-only review tier for kernel changes — I cannot approve it; it needs the owner's merge. No blocking findings.

What I independently verified on head fb74592a93:

  1. The bug is real, and the fix removes the mechanism (not the symptom). Pre-fix, makePreview (kernel/src/search/index.ts) located the first query-term hit in text.toLowerCase() but windowed the original text at that lowercased offset. Every length-expanding char (İ, 1→2 UTF-16 units) before the hit shifted the window right; on long text the window slid past text.length and the preview came back empty. I reproduced it directly: swapping in the merge-base (pre-fix) index.ts makes the two new regression tests fail (52 pass / 2 fail); with the PR's fix they pass (54/54).

  2. The offset mapping is exact. originalOffsetOf accumulates char.toLowerCase().length per code point and stops at the target offset. I verified the round-trip property — "for every original-character boundary, the lowercased offset at that boundary maps back to exactly that original boundary" — across 9 text shapes (400×İ prefix, 50×İ+100×x mid-hit, Greek sigma, mixed İ/Σ/ſ, plain ASCII): 891/891 pass, 0 failures. For the two real test queries the hit maps to exactly the right original offset (lower 801 → orig 401; lower 201 → orig 151).

  3. Only U+0130 (İ) changes length under toLowerCase. I scanned the full Unicode code-point table: exactly one code point expands (U+130, 1→2), zero shrink. This confirms the doc comment's claim, and confirms U+017F (ſ) is a fixed point (1→1) — so the comment correctly lists only U+0130 (the earlier ſ mention was already corrected on this branch). The contextual Greek-sigma rule is length-preserving (Σ→σ, ς→ς, both 1 unit) and does not affect the offset arithmetic.

  4. Identity preserved for the common case. For text with no length-changing characters the mapping is the identity, so every existing preview is byte-identical. Confirmed: pre==post on plain-text input, and the existing "preview centers on the matched term" test passes both pre- and post-fix.

  5. Surrogate-safe. originalOffsetOf iterates by code point (for...of), so it never splits a surrogate pair and always returns a code-point-boundary offset; clampWindow (kernel/src/truncate.ts) then snaps both window edges off surrogate halves. Double-safe.

  6. Tests have teeth. The two new regression tests (kernel/tests/search.test.ts) fail on the pre-fix code and pass on the fix — they would catch a regression of exactly this bug, not a tautology.

  7. CI (mirror cache, head fb74592a93): 0 failures so far — 6 success / 13 skipped / 11 in-progress (the test shard matrix still running). No red.

House conventions (Kernel Boundary): kernel-change: declaration present (first line of the PR body) ✓ · kernel: commit prefix on all three commits ✓ · version 0.0.199 → 0.0.200 (CI-enforced discipline for kernel/src/** changes) ✓ · diff is 5 files, every line on-purpose (the fix + 2 tests + version + 2 lockfile syncs), no unrelated churn ✓ · no config/env/wire/persistence change ✓. It is a display-only change to the search preview, not a wire change, so it does not require the both-compression-modes reasoning.

No small findings to fix — the code is correct as-is; I did not need to add any fix commit.


结构化八问:

  1. 是什么问题: kernel 搜索 preview 用「lowercase 后字符串的偏移」去切「原始字符串」。文本在 hit 前有 lowercase 会膨胀的字符(İ,1→2 UTF-16 units)时,preview 窗口右移,长文本下真实命中的 preview 变空 "…"。
  2. 什么原因导致的: makePreview 在 text.toLowerCase() 上 indexOf 定位 hit,却拿该偏移直接切原始 text——两串长度不同,偏移坐标系不一致。与 host 侧 search_context evidence offsets drift when Unicode lowercasing expands characters #6493(search_context evidence snippet,已修于 [PR] fix: map search evidence offsets back to original Unicode text #6494)同一根机制,不同包。
  3. 影响范围与严重程度: 仅影响 searchBlocks/searchBlocksAsync 结果 preview 的展示——文本在 hit 前有 İ 时窗口偏移或为空;匹配、打分、排序、其余行为均不受影响。属展示瑕疵(模型仍能定位到 block,但 preview 证据错位/为空),p3 级。
  4. 怎么修复的: 减半窗之前先把 hit 偏移映射回原始串(originalOffsetOf,per-code-point 累加,与 host 侧 [PR] fix: map search evidence offsets back to original Unicode text #6494 同一 span-mapping 思路);映射精确性由「唯一变长 lowercase 规则是 context-free 的 U+0130」保证,我全表扫描 + 891 个边界 round-trip 独立验证过。
  5. 是否有回归问题: 逐项审过——无长度变化字符的文本:映射为恒等,preview 逐字节不变(既有 centering 测试钉住,pre/post 均绿);打分/排序/匹配/其余 preview 路径:未触碰;无语义漂移、无 fail-fast 变吞错、无阈值/时序/输出格式变化。无回归。
  6. 是否破坏缓存: 否——缓存、重试、增量游标、去重键均无改动;search doc-cache 未触碰。
  7. 是否改了设计方向: 否——makePreview 内部纯 bug 修复,无架构/边界/数据流变化。
  8. 是否引入了新的配置: 否——无新 env/config/默认值。

判定: 修复机械安全、回归测试有牙(已独立验证 pre-fix 失败 / post-fix 通过)、diff 收敛到目的、Kernel Boundary 合规。kernel 变更适用 human-only review tier——我无法 approve,需 owner 人工合并;代码本身无阻塞项。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

🔶 需要决策

PR #6686 (kernel: map search preview offsets back to original Unicode text) touches kernel/ and carries a kernel-change: declaration — it sits at the human-only review tier (Kernel Boundary rule 3). I have completed an independent review (see the comment above): the fix is correct, the regression tests have teeth (verified red on pre-fix / green on post-fix), the diff is clean, and all Kernel Boundary conventions are met. There are no blocking findings.

Gated diff awaiting your authorization (the kernel change):

  • kernel/src/search/index.ts — adds originalOffsetOf() and uses it in makePreview to map the lowercased hit offset back to the original before windowing.
  • kernel/tests/search.test.ts — two new regression tests (İ length-expansion cases).
  • kernel/package.json — version 0.0.199 → 0.0.200.
  • kernel/package-lock.json + package-lock.json — lockfile version sync.

Requesting owner authorization to merge (human-only tier). CI on head fb74592a93: 0 failures (test shards still in progress).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Review summary (task #6686 review-lane, head fb74592):

What I did: Independently reviewed the full diff (kernel/src/search/index.ts + kernel/tests/search.test.ts + kernel/package.json 0.0.199→0.0.200 + 2 lockfiles) against the linked defect #6493 and host-side fix #6494.

Verified (not copied from prior thread):

  • Bug is real: pre-fix makePreview windows ORIGINAL text at a LOWERCASED offset; each İ (1→2 UTF-16 units) before the hit shifts the window right; long text → empty preview.
  • Fix is exact: originalOffsetOf round-trip verified across 9 text shapes (891/891 pass); full code-point scan of toLowerCase confirms U+0130 (İ) is the only expanding char, U+017F (ſ) is a fixed point (1→1), so the doc comment correctly lists only İ.
  • Identity preserved: no length-changing chars → existing previews byte-identical; "preview centers on the matched term" test passes pre- and post-fix.
  • Surrogate-safe: iterates by code point, returns code-point-boundary offsets; clampWindow snaps off surrogate halves.
  • Tests have teeth: the 2 new regression tests fail pre-fix / pass post-fix (not tautologies).
  • CI (mirror cache, head fb74592): 0 failures.

Verdict: Code is correct, tests have teeth, diff is clean, Kernel Boundary compliant (kernel-change declaration present, kernel: commit prefix, version bump 0.0.199→0.0.200). No blocking findings. No small findings to fix.

Cannot approve — this is a kernel/ change and sits at the human-only review tier per the Kernel Boundary. Posted a 🔶 需要决策 comment requesting owner authorization; needs-decision label applied. Merge waits on your human-only kernel-tier sign-off.

ci-check added 3 commits November 21, 2026 12:57
makePreview located the hit in text.toLowerCase() but windowed the
ORIGINAL text at that lowercased offset. Lowercasing expands U+0130 (İ)
and U+017F (ſ) from one to two UTF-16 units, so any expanding character
before the hit shifted the window right; for long text the window slid
past the end and the preview came back empty for a real match.

Map the hit offset back to the original before subtracting the
half-window (originalOffsetOf) — the same span-mapping the host-side fix
in billion-context#6494 uses for its evidence snippets.

Regression tests pin both the distant (400x İ) and mid-text cases; both
fail on the pre-fix windowing and pass after. kernel 0.0.197 -> 0.0.198.
…ngth

A full-repertoire scan (Node 22 / V8) shows U+0130 (I with dot above,
1 to 2 UTF-16 units) is the only code point whose toLowerCase() changes
length; U+017F (long s) is a fixed point (1 to 1). The comment claimed
both expand — factually wrong. The mapping itself is unchanged and
correct for any per-character length change; only the stated fact is
fixed.
@ranxianglei
ranxianglei force-pushed the 2026-11-14_kernel-preview-unicode-offsets branch from fb74592 to f408d6a Compare November 21, 2026 05:02
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/qwen3.8-27b)

Merge conflict resolved (re-rebase). Re-verified live first: master had advanced to ba2e8198 (99 commits past my previous rebase base a446a3c1), which is what re-dirtied the PR. Master's two intervening kernel commits bumped the kernel version 0.0.200 → 0.0.201 and synced the lockfiles — so the conflict was again entirely in the version/lockfile lines; the code and test files had no master-side changes.

What I did:

  • Rebased the PR branch onto current master. Conflicts surfaced in kernel/package.json, kernel/package-lock.json, and root package-lock.json (all the embedded kernel-version lines). The two code files (kernel/src/search/index.ts, kernel/tests/search.test.ts) rebased cleanly.
  • Resolved the version to 0.0.202 — master's 0.0.201 + 1 for this PR's kernel/src change (Kernel Boundary version discipline) — and synced both lockfiles to 0.0.202.
  • Fixed a stale commit message in the lockfile-sync commit (it said "0.0.200" from the prior run; amended to "0.0.202" to match the actual version).
  • Verified locally: master is an ancestor of the new head (clean, 3 commits ahead); version consistent at 0.0.202 across package.json + both lockfiles; both lock-sync checks clean (git diff --exit-code = 0 after regen); kernel/tests/search.test.ts → 54/54 pass (both new regression tests + the existing identity pin).
  • Force-pushed the final head f408d6a0b64 (conflict-rebase, explicitly permitted; no [ework:ci-held] marker → triggers the full CI pass). Remote head confirmed at f408d6a0b64.

CI: the mirror cache still shows the previous head fb74592a93; the new run on f408d6a0b64 will be picked up by the poller shortly. Final green/red verdict comes from remote CI on the pushed head.


结构化八问:

  1. 是什么问题:PR 与 master 再次产生合并冲突(merge state: dirty)——master 在我上次 rebase 后又前进了 99 个提交。
  2. 什么原因导致的:master 的两个 kernel 提交把 kernel 版本从 0.0.200 推到 0.0.201 并同步了 lockfile,与 PR 分支上停在 0.0.200 的版本行冲突;代码/测试文件 master 侧零改动,故冲突只集中在版本号三处(kernel/package.json + 两份 lockfile 的内嵌 kernel 版本)。
  3. 影响范围与严重程度:仅本 PR 的合入被重新阻塞(冲突),不影响任何已发布行为;属版本/lockfile 同步的机械冲突,非功能性缺陷。
  4. 怎么修复的:把 master rebase 进 PR 分支,版本按「在新基线上 +1」解为 0.0.202,两份 lockfile 同步到 0.0.202;代码与测试逐字保留;顺手把 lockfile-sync 提交的过期版本号注释(0.0.200)改为 0.0.202,force-push 最终 head。
  5. 是否有回归问题:无。代码/测试文件 master 侧零改动,rebase 后与 PR 原内容逐字一致;本地定向跑 kernel search 套件 54/54 绿(含 2 个新回归测试 + 既有 centering 恒等钉),双 lockfile 检查绿。无语义漂移、无阈值/时序/输出格式变化。
  6. 是否破坏缓存:否——纯 rebase + 版本/lockfile 同步,未触碰任何缓存/重试/增量游标/去重键逻辑。
  7. 是否改了设计方向:否——无架构/边界/数据流变化,仅版本纪律下的版本号推进。
  8. 是否引入了新的配置:否——无新 env/config/默认值;唯一变更是 kernel 版本 0.0.201 → 0.0.202(由新 master 基线推导)。

判定:冲突已解、定向测试绿、双 lockfile 检查绿、head f408d6a0b64 已 force-push(无 ci-held 标记,触发整轮 CI)。等远程 CI 在该 head 上跑完即门禁;kernel 变更适用 human-only review tier,等 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Re-review at new head f408d6a (was fb74592).

What changed since my last review: the branch was rebased onto latest master (merge-base ba2e819, master now at kernel 0.0.201). The code (kernel/src/search/index.ts + kernel/tests/search.test.ts) is byte-identical to what I verified before — git diff fb74592a93 f408d6a0b on both files is empty. The only delta is the version bump 0.0.201→0.0.202 + lockfile sync (commit f408d6a), a clean consequence of the rebase. No logic touched.

Verification (re-confirmed at the new head):

  • Bug is real: pre-fix makePreview windows ORIGINAL text at a LOWERCASED offset; each İ (1→2 UTF-16 units) before the hit shifts the window right; long text → empty preview.
  • Fix is exact: originalOffsetOf round-trip verified across 9 text shapes (891/891 pass); full code-point scan of toLowerCase confirms U+0130 (İ) is the only length-changing char, U+017F (ſ) is a fixed point (1→1), so the doc comment correctly lists only İ.
  • Identity preserved: no length-changing chars → existing previews byte-identical; "preview centers on the matched term" test passes pre- and post-fix.
  • Surrogate-safe: iterates by code point, returns code-point-boundary offsets; clampWindow snaps off surrogate halves.
  • Tests have teeth: the 2 new regression tests fail pre-fix / pass post-fix (not tautologies). Fresh run at f408d6a: 54/54 pass.
  • CI (mirror cache, head f408d6a): 0 failures (12 success / 13 skipped / 5 test shards in-progress).

Verdict: Code is correct, tests have teeth, diff is clean, Kernel Boundary compliant (kernel-change declaration present, kernel: commit prefix, version bump 0.0.201→0.0.202, lockfiles in step). No blocking findings. No small findings to fix. The rebase introduced no regression.

Cannot approve — kernel/ change, human-only review tier per the Kernel Boundary. See the 🔶 需要决策 comment below for the authorization request.


八问(中文,供人工 reviewer):

  1. 是什么问题:搜索预览窗口按「小写化后文本」的偏移去切「原始文本」,İ(土耳其大写 I 带点)小写化后从 1 个 UTF-16 单元变 2 个,命中点前的每个 İ 都把窗口往右推,长文本下窗口滑出文本末尾 → 预览为空。
  2. 什么原因导致的:makePreview 在 text.toLowerCase() 里用 indexOf 找到命中偏移,却直接拿这个偏移去 window 原始 text;两种文本在含 İ 时长度不一致,偏移错位。
  3. 影响范围与严重程度:仅影响 searchBlocks 的预览字符串(展示层),不影响压缩/存储/线上协议;触发条件 = 命中点前存在 İ 且文本足够长。属体验瑕疵(预览可能为空/偏位),非功能性损坏。
  4. 怎么修复的:新增 originalOffsetOf(text, lowerOffset),逐码点累加 char.toLowerCase().length 把小写偏移映射回原始偏移,命中点先映射再开窗。
  5. 是否有回归问题:无长度变化字符时映射为恒等 → 既有预览逐字节不变;「预览居中命中词」测试前后均过;无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及(纯展示计算,无缓存/重试/游标/去重键)。
  7. 是否改了设计方向:不涉及(单函数内偏移映射,无架构/边界/数据流转向)。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值)。

判定:代码正确、测试有牙、diff 干净、Kernel Boundary 合规,无阻塞项;因属 kernel 人审层,不 approve,待 owner 人工授权合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

🔶 需要决策

PR #6686 (head f408d6a) 触碰 kernel/ 且正文带 kernel-change: 声明,按 Kernel Boundary 属人审层,review-lane 不能 approve。请 owner 授权合并。

被门禁的 diff(merge-base ba2e819 → head f408d6a,5 文件):

  • kernel/src/search/index.ts — 新增 originalOffsetOf + makePreview 命中点先映射再开窗(核心修复)
  • kernel/tests/search.test.ts — 2 个回归测试(İ 前缀不推窗 / 中段命中保持窗口)
  • kernel/package.json — 版本 0.0.201 → 0.0.202
  • kernel/package-lock.json + package-lock.json — lockfile 版本同步

review-lane 结论:代码正确、测试有牙(新 head 实测 54/54)、diff 干净、Kernel Boundary 合规、无阻塞项、无小问题需顺手修。rebase 干净(代码与上轮 fb74592 逐字节一致,仅版本号随 master 前进)。CI 0 失败。

等待 owner 人工 kernel 层授权后合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant