Repository navigation
[PR] fix: normalize plan text once per source - #7162
Merged
ranxianglei merged 1 commit intoNov 20, 2026
Merged
Conversation
Reuse the stripped, lowercased plan string across token matches instead of normalizing it for every token. Preserve extraction and search results while avoiding repeated full-text scans on CJK and mixed-language plans. Add a reproducible benchmark using actual compressed blocks and the plan-aware search entry point. Fixes ranxianglei#7160
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7160.
Problem
With plan-aware search enabled,
planTokenize()lowercases its entire input on every token match, including the final failed match. A Chinese plan produces a token for nearly every character, so an unchanged source is repeatedly rescanned and normalized. This adds synchronous CPU work to the retrieval tool even within the existing 4,000-character source cap.Change
Lowercase each source once, after stripping ACP markup and before iterating its tokens. Token order, stopwords, character limits, weights, result ranking and model-visible search text are preserved. The production change is two lines; the added script makes the affected search workflow and its plain-search control reproducible.
Validation
Base:
548e396022b9391e28a3c9061560be297779b8e8, package0.1.216; Node.js24.22.1, Linux CPU.npm run typecheck,npm run build,npm run check:test-import-graphandgit diff --checkpassed. The build's import-annotation and external-dependency guards also pass.Measured search workflow
The script creates six real compressed blocks, calls
executeSearchContext({ query: "token", limit: 5 }, ...), and supplies a valid TodoWrite snapshot plus the latest user turn. It tests Latin, Chinese and mixed text at 1,000, 2,000 and 4,000 characters per source, and includes extraction-only and plain-search controls.The table reports milliseconds, as medians of three process medians. Base and changed processes alternate; each workload has three warmups and 15 measured calls. Output hashing and equality assertions are outside each timed interval.
These are bounded synthetic workloads on a shared CPU. The unchanged plain-search control and short Latin extraction show timing variation; this does not establish a speedup for all queries. The large reduction applies to plans with many tokens. Plan-aware search remains disabled by default.
To reproduce against the unchanged tokenizer, save the base source beside the current module so relative imports stay identical, then use the script's optional module argument:
git show 548e396022b9391e28a3c9061560be297779b8e8:src/decompress-shared.ts \ > src/decompress-shared.plan-token-baseline.ts node --import ./tests/preload-test-env.mjs --import tsx \ scripts/bench-plan-aware-search.ts ../src/decompress-shared.plan-token-baseline.tsThe temporary base copy is not part of this PR.
Prior work
Checked all 258 prior fork heads, four source-file versions and all 51 upstream revisions of this file. All 32 revisions containing this tokenizer repeat the normalization. Related #4325 fixes Han matching; #5048 changes plan relevance. Neither removes this repeated full-text work.