token efficiency: lazy tool registration + tool-result trimming policy #211

Summary

Feature-research ring 2 (2026-10-06, validated: all tools register upfront in the system prompt; per-tool output caps exist but there is no cross-turn trimming/compaction policy). Original levers (from Cursor's Sept-2026 token-efficiency work): (1) register tool schemas lazily/progressively; (2) a trimming policy for old tool RESULTS in re-sent history. Design settled by a grilling session on 2026-10-09 after a fact-finding pass over the code; the decisions below supersede the original sketch.

Facts that drove the decisions (develop @ 4c90467, v0.8.0)

• The wire body is a pure function of (system prompt, tool specs, entries) (OpenAiProvider::request_body); history is append-only and nothing is reshuffled per step; specs are sorted by name and the system prompt is pinned in the session header. No provider cache markers exist; caching relies on the gateway prefix cache.
• Archived GLM traces: first-step prompt ≈ 3.65k tokens (system + tool specs with the 18-tool cora MCP preset), history grows only ≈ 40–100 tokens per step in short missions, cache hit ≈ 96–98% on later steps. cached_tokens is already in the usage ledger but is not reported anywhere.
• mission.rs::tokens_used() sums prompt_tokens + completion_tokens over every step, i.e. the re-sent (mostly cached) prefix is charged against --max-tokens every step: the issue is as much budget ACCOUNTING as real cost.
• No tokenizer and no local token estimator exist; mock providers emit no usage. UsageRecord.usage is a raw provider Value, so additive keys need no storage format change.
• Largest history bodies: read_file (up to 1 MiB, no head cap, unlike git/web at 20k chars), MCP results (256 KiB), --skill bodies in the system prompt, load_skill results (16 KiB).
• An unknown tool call aborts the turn (settles to Final); changing the tools array mid-session breaks the prefix cache for the whole history.

Decisions

1. Scope / phasing: measure first. Lazy tool registration is OUT of scope (deferred with reasons: it trades a small saving — the prefix is already ~96% cached — for risk to two invariants: cache-prefix stability and the unknown-tool safety behavior; it would also need a new meta-tool). It can be re-opened if the numbers justify it.
2. Success measure: reference missions (deterministic mock missions with large tool results and many steps) with a loose CI upper bound; live missions are judged by the real usage report. The CI metric is serialized request CHARACTERS (tokenizer-free), see 4.
3. History trimming is deferred until the instrumentation shows history is a material share on long missions. Design is ready if needed: wire-only (in request_body, never touching the write-once log), applied once at a fixed age, deterministic function of the entry sequence only (no clock/process state), only for tool results that can be re-fetched deterministically (read_file, git, web_fetch), replaced by a stub inside the existing fence with the intent input kept visible; run_command/job output never trimmed; a no-op below its threshold so the Tier-1 wire golden is unaffected.
4. Metric + storage: per step, compute the serialized request characters split into system, tools, history and store them under a namespaced key tole_wire inside the existing UsageRecord.usage object (provider values untouched, no storage format change).
5. Report: derive from the ledger at read time — tole status and GET /sessions/{id}/status show, for ANY session: total prompt/completion/cached tokens, cache-hit rate, the system+tools vs history character split at the first and last step, and the step count. The fact/mission register gains an additive cached_tokens. Check that consumers of the status JSON (mobile app, docs/mobile-control.md) tolerate new fields.
6. Budget accounting: unchanged for now (only cached_tokens is REPORTED). Whether --max-tokens should count non-cached tokens only is a follow-up decision once the report shows the size of the gap (it changes the meaning of a user-visible flag and depends on providers reporting cached_tokens).
7. read_file contract (separate PR): head cap of 20,000 characters (like git/web) with optional offset and limit in characters (UTF-8 boundary-safe); the result carries truncated, total_chars and next_offset so the model can continue reading.

Staging

1. PR 1 — tole_wire per step + report in tole status/REST + cached_tokens in fact/mission (no wire change, no behavior change).
2. PR 2 — reference mock missions + loose CI bound on request characters (depends on PR 1).
3. PR 3 — read_file cap with offset/limit (independent; needs a test that a mission can still read a large file).
4. Follow-ups, only if the numbers justify them: history trimming (3 above), non-cached budget accounting (6 above), lazy registration (out of scope for now).

Acceptance

• PR 1: per-step wire split stored; tole status + REST status show tokens, cache-hit rate and the split for any session; additive fields only
• PR 2: reference missions with a CI bound that fails on a deliberate prompt-size regression
• PR 3: read_file capped with a continuation contract and a large-file mission test
• Docs (README/mobile-control status fields) + CHANGELOG same-PR; no change to the wire golden or to durable entry shapes