Deferred (decision 2026-09-21). Not from the jgrep comparison's priority list, but jgrep's --chunks/--overlap pair surfaced it.
Problem
MultiChunkClassifier cuts chunks at hard boundaries. A sentence straddling a cut is fragmented in both neighbouring chunks, so two votes get degraded by one bad cut.
Fix
Chunk N+1 starts N tokens before the end of chunk N, so any boundary sentence appears intact in at least one chunk.
chunk_text_tokens(text, max_tokens, overlap) already accepts an overlap parameter (default 0) — only CLI exposure is missing:
--overlap N in advanced options, default 20 tokens at the default 200-token chunk size (10%)
- pass through
MultiChunkClassifier → chunk_text_tokens
Sizing rationale (measured reasoning)
- Call count is unchanged in fixed mode (N chunks = N calls) and tends to decrease in adaptive mode (chunk 1 = title/abstract stays intact, more often decisive).
- Only tokens/call grow: overhead = overlap × (chunks − 1). At 200-token chunks, 8 chunks: 10 tokens → ~4%; 20 tokens → ~8%.
- 5% (10 tokens) is below one sentence and barely protects the intended case; 10% (20 tokens ≈ one sentence) is the minimum useful. Express in tokens, not percent, since
--chunk-tokens is user-tunable.
Acceptance
--overlap in --help-all, threaded into both fixed and adaptive multi-chunk paths
- test: a sentence spanning a chunk boundary is whole in at least one chunk
- test: overlap does not change chunk call count in fixed mode
🤖 Generated with Claude Code
Deferred (decision 2026-09-21). Not from the jgrep comparison's priority list, but jgrep's
--chunks/--overlappair surfaced it.Problem
MultiChunkClassifiercuts chunks at hard boundaries. A sentence straddling a cut is fragmented in both neighbouring chunks, so two votes get degraded by one bad cut.Fix
Chunk N+1 starts N tokens before the end of chunk N, so any boundary sentence appears intact in at least one chunk.
chunk_text_tokens(text, max_tokens, overlap)already accepts anoverlapparameter (default 0) — only CLI exposure is missing:--overlap Nin advanced options, default 20 tokens at the default 200-token chunk size (10%)MultiChunkClassifier→chunk_text_tokensSizing rationale (measured reasoning)
--chunk-tokensis user-tunable.Acceptance
--overlapin--help-all, threaded into both fixed and adaptive multi-chunk paths🤖 Generated with Claude Code