Warn before sending into an expired prompt cache (no TTL tracking across providers)

[delef]

Problem

Prompt caching cuts the price of a turn by an order of magnitude, but only while the cache is still warm — and warmth is a clock nobody in this plugin is watching.

Claude. markMessageCache (src/translate/anthropic.ts:213-220) marks up to MESSAGE_CACHE_BREAKPOINTS (3) breakpoints in the message history, plus one more for system+tools (toAnthropicSystem, line 242) — all { type: 'ephemeral' }. That's Anthropic's default 5-minute TTL, refreshed on every cache hit (docs: "the cache is refreshed for no additional cost each time the cached content is used") — an idle clock, not a fixed age since creation. Nothing sends ttl: '1h' or the extended-cache-ttl-2025-04-11 beta flag (CLAUDE_BETA_FLAGS at claude.ts:96 is just CLAUDE_BETA_FALLBACK, no cache-ttl string in it anywhere) — and this one is worth flagging as more than a staleness-detection gap: Claude Code's own docs (How Claude Code uses prompt caching) state that on a Claude subscription within plan usage, the real Claude Code CLI requests the one-hour TTL by default for the main conversation, dropping to five minutes only past plan usage. Since this plugin authenticates as Claude Code and serves the same subscription plans, sending only the bare 5-minute default is this codebase diverging from genuine Claude Code traffic shape, not a conservative default — see the "Adopt Claude's 1-hour TTL" item below.

Codex. The Responses backend applies its own automatic idle-based caching, and the TTL is not one flat number: OpenAI's docs put it at ~5-10 minutes of inactivity for older models (the in_memory default), a sliding 30 minutes for GPT-5.6-and-later ("30 minutes after its most recent write or reuse"), and there's a separate 24h-retention tier available as an explicit opt-in that this codebase never requests. On top of that, this route talks to the separate, undocumented chatgpt.com/backend-api/codex/responses backend rather than the documented platform Responses API, so even picking the right number off that table is an extrapolation, not a confirmed fact for this specific endpoint. The one cache-relevant thing this codebase does send is prompt_cache_key: String(sessionId) (codex.ts:573) — OpenAI's docs describe this as a best-effort routing hint toward the same cache-holding machine, explicitly not a pin or a guarantee — and nothing tracks whether that key's cache is actually still warm.

Grok. xAI documents automatic prompt caching as available on every Grok model — this is a confirmed, existing behavior, not the implicit maybe the rest of this section originally assumed. What's actually missing is narrower and more concrete than "unknown caching": xAI's own best-practices guide says to send either an x-grok-conv-id header or a prompt_cache_key, specifically to route repeat requests to the same cache-holding machine — and grok.ts's request (grok.ts:777-794, fully enumerated: model, instructions?, input, tools?, tool_choice, parallel_tool_calls, max_output_tokens?, reasoning?, store, stream) sends neither. So Grok is the one route with no cache-affinity hint at all (Claude has its breakpoints, Codex has prompt_cache_key), which plausibly suppresses its own hit rate independent of any TTL question. The plugin does already have a way to notice a hit once one happens: grok.ts streams through the exact same streamResponses → mapResponsesUsage path as Codex (translate/responses.ts:229-238, shared by every Responses-wire adapter — Codex, Grok, and Copilot's Responses wire alike), which unconditionally maps input_tokens_details.cached_tokens to cacheReadTokens whenever a response carries it. Nobody has confirmed empirically whether Grok's cached_tokens is ever non-zero as this codebase calls it today, and there's no test asserting either way. Split out as #49 — sending the missing hint is a wire-shape change to a working route (its own review/revert unit) and a genuine prerequisite: without it, any cached_tokens reading measures shard-routing luck, not TTL.

Nobody tracks elapsed idle time for cache purposes. No provider or pool code keys anything off how long it's been since a session's last turn (confirmed by grep for lastRequestAt/lastActivity/idleSince/similar across src/ — no hits). The one sliding idle-timestamp-with-TTL that does exist in the tree is wired to something else: Copilot's reasoning-replay store keeps a per-(account, session, model) capture time (copilot.ts:616-621) against a REPLAY_TTL_MS = 30 * 60_000 (copilot.ts:645) that's refreshed on every hit (copilot.ts:861-862) — an idle clock, not an age clock, scoped to reasoning-block replay rather than cache warmth. It's a working precedent for exactly the mechanism this issue wants, just pointed at the wrong thing. The closest-sounding thing, streamIdleTimeoutMs (claude.ts:415, codex.ts:473, grok.ts:551, copilot.ts:599), is a mid-stream network watchdog with nothing to do with inter-turn cache lifetime — worth flagging because its default (DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000, index.ts:107) happens to be the same 5 minutes as Claude's cache TTL, which is exactly the kind of name collision a cache-staleness warning needs to stay clear of. Even the pool's own sticky routing is only spatially aware, not temporally: pool.ts's sticky map (line 65) remembers which account served a session specifically so "prompt caches survive" (the comment at pool.ts:6), but never when it last served — a session idle for an hour is still pinned to the same member, well past the point its Claude cache actually expired.

The pieces to notice staleness are half-built: the plugin already reads the post-hoc cache-hit accounting once a response comes back — cacheReadTokens / cacheWriteTokens from Anthropic's cache_read_input_tokens / cache_creation_input_tokens (translate/anthropic.ts:399-403), and cached_tokens from the shared Responses path (translate/responses.ts:230) and OpenAI-style chat completions (translate/chat-completions.ts:177). But that's retrospective — it feeds the harness's own token totals, nothing in src/client/ reads it, and nothing uses it to predict the next message's cost before it's sent.

Impact

A user who steps away mid-conversation for ten minutes and comes back has no signal that the next message will reprocess the entire prompt from scratch at full price — on Claude that's the difference between a cache-read discount and a full-price prefill, drawn from the same session/weekly quota window whose exhaustion #27/#28 handle. Burning through it needlessly here just gets a user to that wall sooner. The cost is invisible until the usage numbers move.

Not a duplicate of #39 / #41. Both of those (open) PRs surface how much of the rate-limit window (5h/weekly quota) is left — a completely different clock from the prompt-cache TTL (~5 minutes) this issue is about. Neither reads or displays anything cache-related today.

Related but distinct from #17 / #21 / #24. #21 and #24 were closed by #25 (which fixed the underlying "caching not wired up at all" problem); #17 is the same problem but was closed three days earlier, alongside an unmerged attempt (#18), before #25 existed. This issue assumes caching now works (it does, since #25) and is about the next gap: knowing when it's gone cold.

Blocked, for the Grok row only, on #49 — sending Grok's missing prompt_cache_key so a hit becomes measurable instead of shard-routing luck.

Proposed direction

Staleness has two structurally different triggers, and they don't deserve equal treatment:

1. Deterministic (primary). This plugin builds every request, so it has perfect knowledge of whether the model, tool set, or system prompt changed since a session's last turn — a certain cache miss, per both Anthropic's and OpenAI's own docs, with zero false positives and no TTL number required. Identical across all four providers.
2. Idle-time (secondary, layered on top). The clock this issue originally proposed — inherently a guess, and only as good as each provider's TTL confidence.

Of (1), only a model switch is knowable before send (the client already polls the selected model); an effort switch is only knowable once the request is built, so it can explain a slow turn after the fact but can't warn ahead of it.

The mechanism, one shared module (src/providers/cache-warmth.ts) rather than a rate-limit.ts-style per-provider reader: no provider discloses a cache TTL on the wire the way all three disclose a rate-limit reset, so a reader function would just be a constant in a trenchcoat. What is already normalized per-provider is the post-hoc cache-hit accounting this codebase already reads (cacheReadTokens/cacheWriteTokens — see above), so the shared piece wraps that, and the per-provider piece shrinks to a small declarative confidence table instead of a function:

Provider	Lifetime	Warns at	Basis
claude	sliding, 5 min	after adopting the 1h TTL below: 60 min	Documented, and matches Claude Code's own subscription default (see above)
codex	sliding, up to 30 min	30 min idle	OpenAI documents 30 min (GPT-5.6+) / 5-10 min (older) for the platform Responses API; this route talks to a separate, undocumented ChatGPT backend, so treat as inferred, not confirmed
grok	unknown	never	Blocked on #49 — no honest number exists until the routing hint lands and a hit becomes measurable
copilot	unknown	never	No published lifetime; its existing 30-min REPLAY_TTL_MS (copilot.ts:645) is reasoning-replay, a different clock — must not be reused or conflated

A provider with no upper bound (unknown) never warns — a hedge ("caching may have gone cold") is worse than silence, because it's unactionable and trains the user to ignore the signal that is trustworthy.

• Key: (provider, account, session, model, effort) — not just (provider, account, session). Anthropic's and Claude Code's own docs are explicit that each model, and each effort level within a model, keeps its own independent cache ("switching models recomputes the entire request… changing effort mid-session recomputes the entire request… each has its own cache"). A key without model+effort can only remember the single most recent combination, so ping-ponging between two models (or two efforts) that are each individually still within TTL would misreport the return trip as cold. Tool/system-prompt changes stay outside the key, as a deterministic invalidation event layered on top of whichever (model, effort) bucket is active — they aren't parallel cache lines the way model/effort are, they're a break in the one active line.
• Adopt Claude's 1-hour TTL (cache_control: { type: 'ephemeral', ttl: '1h' } + the extended-cache-ttl-2025-04-11 beta flag) for the main-conversation cache breakpoints. This is not the cost/compatibility trade-off feat(claude): cache the prompt prefix and the conversation tail #25 originally framed it as: it's closing a gap where this plugin's traffic diverges from the real Claude Code CLI's own default subscription behavior (see above), at no cost outside a subscription's normal included usage. The beta flag string is stable and widely used by other client integrations as of writing.
• Codex: no wire change. Leave prompt_cache_retention unset — recent openai/codex reports (August 2026: #39392, #39397) show it actively breaking requests on models that don't support it, and it's unconfirmed whether the ChatGPT subscription backend this route talks to honors it at all. Just use 30 minutes as the working idle-TTL assumption above — no request shape changes, no new failure surface.
• Surface: conversation.composer.dock, passive only for v1 — no send-blocking gate. Verified against the shipped @deepseek-ai/dsh-client-ui-conversation package: composer.dock's own contract is explicitly for "an ambient readout about the conversation… anything the user must click belongs in the tool row instead," which fits a silent warning line better than conversation.input.right (a slot for interactive controls — this plugin's own SpeedSelect already sits there). Render nothing when warm; one line when cold, using the provider's own last-reported cachedTokens as a replayed fact, not a cost prediction.
  • A blocking Yes/No confirmation (the interaction Claude Code's own CLI uses for /model switches while the cache is still warm) was considered and explicitly deferred. The platform has no plugin-facing "soft confirm, allow through" primitive: ask_user_question/the approval composer-takeover is model/tool-call-initiated only, and the one plugin-facing primitive that exists, ctx.conversation.blocks (ComposerBlocks), is a hard block — an inert textarea with a reason, not a dismissible dialog. A v2 gate is buildable on top of it (hold the block, add a "Send anyway" control in input.right that clears it) but is a materially different, stateful feature; out of scope for v1.
  • Note this needs no dependency on feat: show subscription usage in composer stats strip #39 or Composer usage meter, Claude plan tier, and Retry-After handling #41 — both are open, unmerged, and feat: show subscription usage in composer stats strip #39 is already stale against main (reads a providers[id]?.loggedIn field the multi-account refactor removed).

Out of scope

• Estimating Grok's or Copilot's cache TTL — no upper bound is known for either, so no honest warning is possible (Grok specifically blocked on grok: send prompt_cache_key so cache hits aren't left to shard-routing luck #49).
• Adopting Codex's prompt_cache_retention: '24h' tier — real breakage reports on unsupported models, unconfirmed backend support; not needed since the idle-TTL layer above doesn't require it.
• The blocking Yes/No send-confirmation gate (v2, see above) — a stateful feature change, not a warning.
• Learning a TTL from observed hit/miss pairs, or persisting warmth state across a restart — after a restart, "never warmed" is the honest state and renders as silence, same as today.
• Pool-routing integration (pool.ts's sticky selection ignoring warmth) — a real gap (an hour-idle session stays pinned to a member whose cache is provably gone) but a routing-behavior change, a different blast radius from showing a warning. The tracker is designed so pool.ts could consult it later without rework.
• Byte-exact cost prediction — a rough "cache likely cold" plus the last known cachedTokens figure is enough to change behavior.

Verification

• Unit test: a session whose last request under the same (model, effort) is older than the provider's TTL reports cold; within TTL reports warm; a switch to a different (model, effort) combo reports cold via the deterministic path regardless of the idle clock, and reports warm again if that combo's own entry is still within its own TTL.
• Per-provider lifetime table covered the same way feat: wait out a closed rate-limit window on every route #28's test/rate-limit.spec.ts tables each provider's reset parsing.
• A live idle-then-send Claude request confirming cache_read_input_tokens actually drops to 0 at (or near) the boundary — checked once against the current 5-minute default, and again after the 1-hour TTL lands, since this specific OAuth endpoint (not the documented platform API) is the one thing here that's never been directly measured.
• A live Codex request past 30 minutes idle confirming cached_tokens actually drops to 0 at that boundary on the chatgpt.com/backend-api/codex/responses backend specifically.
• Grok verification is entirely grok: send prompt_cache_key so cache hits aren't left to shard-routing luck #49's: a non-zero cached_tokens after the routing hint lands is the acceptance test for that issue, and the prerequisite for ever filling in Grok's row here.

[delef]

@V1ki — the "Proposed direction" above is now a full design, not just a problem report, and it makes a few calls worth your sign-off before I start implementing:

• Treating this as two separate triggers (deterministic content-change vs. idle-TTL), with the deterministic one as the primary mechanism and idle-TTL layered on top, rather than one flat idle clock.
• Adopting Claude's 1-hour cache TTL as part of this work, since it turns out to be Claude Code's own default on a subscription — not just a staleness-detection feature, an actual behavior change.
• Deliberately not touching Codex's prompt_cache_retention (real breakage reports upstream on unsupported models) — just using 30 min as a TTL assumption, no wire change.
• Splitting Grok's missing prompt_cache_key into its own issue (grok: send prompt_cache_key so cache hits aren't left to shard-routing luck #49) as a prerequisite, rather than folding it in here.
• conversation.composer.dock as the surface, passive-only for v1 — explicitly not a send-blocking confirmation (that would need a stateful mechanism the platform doesn't cleanly expose to plugins today; noted as a possible v2).

Would appreciate a look before this turns into a PR, especially on the 1-hour TTL adoption (it's the one item here with real behavior impact beyond "add a warning") and the composer.dock placement relative to #39/#41.

[delef]

@V1ki This is an important feature, particularly for LLMs with long context and a limited cache window. A single query that misses the cache can consume a lot of tokens. Claude Code already warns about this and advises starting a new session. What do you think about this?

[V1ki]

I've been thinking about this for quite a while. The 1-hour cache is attractive, but its cache-write cost is substantially higher, so I'm hesitant to enable it unconditionally.

Could you help confirm one detail about mixed-TTL behavior?

Suppose a prompt prefix was initially cached using the default 5-minute TTL. Before that entry expires, the next request sends the same prefix but marks the breakpoint with ttl: '1h':

• Can that request still read/hit the existing 5-minute cache entry?
• If it hits, will Anthropic also perform (and charge for) a cache write to promote or extend that entry to the 1-hour TTL?
• More generally, are 5-minute and 1-hour cache entries interoperable for an identical prefix, or are they treated as separate entries that require a rewrite?

If a request asking for the 1-hour TTL can reuse an existing 5-minute entry, then perhaps we could use the 1-hour TTL only in selected scenarios rather than enabling it for every request. That might preserve most of the benefit while avoiding the higher write cost everywhere.