perf: lazy-load ai-tokenizer

[onedotmint]

Summary

Lazy-load ai-tokenizer on the first estimateTokens() call instead of importing it during plugin startup.

As described in #286, the static imports caused the Claude tokenizer vocabulary to be included in the eagerly loaded bundle even when token estimation had not been used yet.

This change:

loads the tokenizer with createRequire(import.meta.url) only when needed
caches the tokenizer instance after the first load
preserves the existing synchronous estimateTokens() API and encode(text, "all") behavior
Bundle impact

For the Pi plugin, the eagerly imported shared chunk decreased from:

4,634,679 B → 2,136,654 B

ai-tokenizer inputs in the Bun metafile decreased from:

4 inputs / 2,413,204 B → 0 inputs / 0 B

The tokenizer is also no longer included in the OpenCode eager bundle.

Validation

estimate-tokens.test.ts: 4/4 passing
plugin typecheck: passing
Pi plugin typecheck: passing
plugin build: passing
Pi plugin build: passing
verified via Bun metafile that ai-tokenizer is no longer part of the eager bundle
verified runtime resolution from a packed Pi installation
confirmed token counts remain equivalent to the existing Claude tokenizer implementation, including special-token-like input

---

Thanks to the author of #286 for the detailed investigation, proposed approach, and benchmark. This PR implements and validates that approach against the current master.

Fixes #286

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Lazy-loads ai-tokenizer on first estimateTokens() call to keep the Claude vocabulary out of the eager bundle and cut startup size. Token counts and the synchronous API remain unchanged.

• Refactors
  • Replace static import with createRequire(import.meta.url) lazy load and cache the tokenizer after first use.
  • Preserve estimateTokens() sync API and encode(text, "all") behavior.
  • Bundle impact: Pi shared chunk 4,634,679 B → 2,136,654 B; Bun metafile ai-tokenizer inputs 4 (2,413,204 B) → 0.

^{Written for commit d989231. Summary will update on new commits.}

Greptile Summary

The PR defers loading ai-tokenizer until the first non-empty token estimate while retaining the synchronous API and caching the initialized tokenizer.

• Replaces eager tokenizer imports with createRequire(import.meta.url) calls whose non-literal specifiers avoid eager Bun bundling.
• Supports either default or named Tokenizer exports.
• Reuses the initialized Claude tokenizer for subsequent estimates.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness, compatibility, or security failures identified.

The affected packages retain ai-tokenizer as a runtime dependency, target Node-compatible environments, and preserve the existing synchronous encoding behavior while changing only when tokenizer initialization occurs.

Important Files Changed

Filename	Overview
packages/plugin/src/hooks/magic-context/read-session-formatting.ts	Lazily resolves and caches the Claude tokenizer without changing token-estimation semantics; no actionable defect was established.

_{Reviews (1): Last reviewed commit: "perf: lazy-load ai-tokenizer" | Re-trigger Greptile}

[Copilot]

Pull request overview

This PR reduces plugin cold-start and eager bundle size by lazy-loading ai-tokenizer only on the first estimateTokens() invocation, caching the tokenizer instance afterward while keeping the existing synchronous estimateTokens() API and encode(text, "all") behavior.

Changes:

• Replace static ai-tokenizer imports with a createRequire(import.meta.url)-based lazy loader.
• Cache the tokenizer after first load to avoid repeated resolution and initialization.
• Route estimateTokens() through getTokenizer() to preserve behavior while deferring the heavy dependency.

Suppressed comments (1)

packages/plugin/src/hooks/magic-context/read-session-formatting.ts:122

• Type-only imports (import type …) can’t be referenced via typeof … type queries. The current InstanceType<typeof TokenizerType>, the module-shape typing in tokenizerModule, and as typeof ClaudeEncodingType will fail TypeScript checking. Prefer defining constructor/module types via typeof import("…") and use TokenizerType directly as the cached instance type.

const lazyRequire = createRequire(import.meta.url);
let cachedTokenizer: InstanceType<typeof TokenizerType> | null = null;

function getTokenizer(): InstanceType<typeof TokenizerType> {
    if (cachedTokenizer) return cachedTokenizer;

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cubic-dev-ai]

No issues found across 1 file

_{Re-trigger cubic}

[cubic-dev-ai]

No issues found across 1 file

_{Re-trigger cubic}

[onedotmint]

I checked the failing lint job. The three errors are pre-existing issues in compress-cues.ts and rust-mode-transform.ts; this PR only changes read-session-formatting.ts, and there are no lint diagnostics against the lazy-loading change itself.

I’ve left the PR unchanged to avoid mixing unrelated formatting fixes into this PR.