Skip to content

[PR] fix: normalize plan text once per source - #7162

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_normalize-plan-text-once
Nov 20, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_normalize-plan-text-once

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #7160.

Problem

With plan-aware search enabled, planTokenize() lowercases its entire input on every token match, including the final failed match. A Chinese plan produces a token for nearly every character, so an unchanged source is repeatedly rescanned and normalized. This adds synchronous CPU work to the retrieval tool even within the existing 4,000-character source cap.

Change

Lowercase each source once, after stripping ACP markup and before iterating its tokens. Token order, stopwords, character limits, weights, result ranking and model-visible search text are preserved. The production change is two lines; the added script makes the affected search workflow and its plain-search control reproducible.

Validation

Base: 548e396022b9391e28a3c9061560be297779b8e8, package 0.1.216; Node.js 24.22.1, Linux CPU.

  • Relevant existing suites on both base and changed source: 20 passed, 0 skipped. They exercise latest-plan selection, custom protected paths, reranking/steering, own versus foreign sessions, and real Anthropic proxy requests with plan-aware search enabled and disabled.
  • 256 deterministic differential cases preserve the exact ordered term/weight sequence. Inputs include Latin/CJK mixtures, case-changing Unicode, upper- and lowercase ACP markup, stopwords, empty input, custom tools and lengths immediately around the 4,000-character cap.
  • All 19 benchmark workload output digests match across all six base/changed processes, including the complete returned search text.
  • npm run typecheck, npm run build, npm run check:test-import-graph and git diff --check passed. The build's import-annotation and external-dependency guards also pass.
  • Independent review checked the unchanged strip/lowercase order, regex state, term ordering and benchmark's actual plan-aware invocation. Full platform/shard testing remains the pushed-head CI verdict.
node --import ./tests/preload-test-env.mjs --import tsx --test \
  tests/search-plan-aware.test.ts tests/protected-tool-paths.test.ts
node --import ./tests/preload-test-env.mjs --import tsx \
  scripts/bench-plan-aware-search.ts

Measured search workflow

The script creates six real compressed blocks, calls executeSearchContext({ query: "token", limit: 5 }, ...), and supplies a valid TodoWrite snapshot plus the latest user turn. It tests Latin, Chinese and mixed text at 1,000, 2,000 and 4,000 characters per source, and includes extraction-only and plain-search controls.

The table reports milliseconds, as medians of three process medians. Base and changed processes alternate; each workload has three warmups and 15 measured calls. Output hashing and equality assertions are outside each timed interval.

Workload (4,000 characters per source) Base ms Changed ms
Chinese plan-aware search 42.111 0.860
Mixed Chinese/Latin plan-aware search 18.018 0.485
Latin plan-aware search 0.313 0.253
Chinese plan extraction 42.703 0.799
Mixed Chinese/Latin plan extraction 16.384 0.367
Latin plan extraction 0.166 0.179
Plain search, no plan reranking 0.025 0.039

These are bounded synthetic workloads on a shared CPU. The unchanged plain-search control and short Latin extraction show timing variation; this does not establish a speedup for all queries. The large reduction applies to plans with many tokens. Plan-aware search remains disabled by default.

To reproduce against the unchanged tokenizer, save the base source beside the current module so relative imports stay identical, then use the script's optional module argument:

git show 548e396022b9391e28a3c9061560be297779b8e8:src/decompress-shared.ts \
  > src/decompress-shared.plan-token-baseline.ts
node --import ./tests/preload-test-env.mjs --import tsx \
  scripts/bench-plan-aware-search.ts ../src/decompress-shared.plan-token-baseline.ts

The temporary base copy is not part of this PR.

Prior work

Checked all 258 prior fork heads, four source-file versions and all 51 upstream revisions of this file. All 32 revisions containing this tokenizer repeat the normalization. Related #4325 fixes Han matching; #5048 changes plan relevance. Neither removes this repeated full-text work.

Reuse the stripped, lowercased plan string across token matches instead of normalizing it for every token. Preserve extraction and search results while avoiding repeated full-text scans on CJK and mixed-language plans.

Add a reproducible benchmark using actual compressed blocks and the plan-aware search entry point.

Fixes ranxianglei#7160
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plan-aware search repeatedly lowercases the entire plan for every token

2 participants