Track B (perf): memoize single-token detokenization on the generation hot path

[konjoinfinity]

Summary

First item of Track B — perf quick wins from the PR #57 roadmap. The batch scheduler decoded each generated token individually (tokenizer.decode([next_id])) at three hot-path sites. That call is deterministic per token id, so frequent ids (common words, punctuation, whitespace) were re-detokenized on every occurrence — within a response and across concurrent requests.

This adds TokenDecodeCache, a bounded id → text LUT that decodes each distinct id once.

Why it's safe

Strictly behaviour-preserving. Each id still maps to exactly the string an isolated tokenizer.decode([id]) produces — the same call the hot path already made — so generated output is byte-identical; only the redundant repeat work is removed. Uses is not None (not truthiness) so an id that legitimately decodes to "" is still cached rather than re-decoded.

Changes

• New squish/serving/token_decode_cache.py — TokenDecodeCache(tokenizer, max_entries=65_536) with decode(), clear(), size; bounded so a pathological stream of distinct ids can't grow it without limit.
• scheduler.py — construct one cache per scheduler; route the three per-token decode sites through it.
• +6 unit tests (tests/serving/test_token_decode_cache.py): equivalence with direct decode, memoization (one decode per id), empty-string caching, bounded size, clear(), validation.
• Bumped the three module-census assertions 103 → 104 for the new module.

Validation

• CI=1 full suite: 4020 passed, 127 skipped, ruff check clean.
• The perf gain itself is realized on the Apple-Silicon generation path and isn't measured by CI — correctness is fully covered here; throughput impact wants a real device benchmark (noted for follow-up).

Scope note

Wired into BatchScheduler (the continuous-generation hot path) only. The scattered server.py single-token decode sites can adopt the same cache as a follow-up; kept out of this PR to keep the change small and reviewable.

🤖 Generated with Claude Code

---

Generated by Claude Code