Skip to content

feat: expose cache creation tokens by TTL to cost expressions - #2677

Open
lspataroG wants to merge 1 commit into
theagentrouter:mainfrom
lspataroG:cache-creation-ttl-cost
Open

lspataroG wants to merge 1 commit into
theagentrouter:mainfrom
lspataroG:cache-creation-ttl-cost

Conversation

@lspataroG

Copy link
Copy Markdown

Description

Creating a cache entry is not priced at a single rate. Anthropic charges twice the
base input rate for an entry that lives an hour, against 1.25x for one that lives
five minutes. Responses already report which was used, in usage.cache_creation,
but only the combined cache_creation_input_tokens was read — so a
costExpression cannot tell the two apart and under-charges the longer-lived
writes. On cache-heavy clients those writes can be the largest single item on a
bill, so the gap is not a rounding difference.

This adds cache_creation_5m_input_tokens and cache_creation_1h_input_tokens
as CEL variables, populated from usage.cache_creation on both the streaming and
non-streaming Anthropic paths. They can be used instead of the combined figure:

cache_creation_5m_input_tokens * 275u + cache_creation_1h_input_tokens * 440u

Nothing changes for existing users: the combined field keeps its meaning, and the
two new variables are 0 when a backend does not report the breakdown.

Related Issues/PRs (if applicable)

None.

Special notes for reviewers (if applicable)

  • IsZero() and Override() were extended to cover the new fields, for
    consistency with the existing ones.
  • Where a response reports cache_creation with zero values, the fields are
    marked as set rather than absent — present-and-zero is a different statement
    from not reported. Two existing test fixtures include that block, so their
    expectations were updated.
  • Verified against live Vertex AI responses for both TTLs, streaming and not.
    make precommit and go test ./internal/... pass.
  • Written with AI assistance; I have reviewed and tested the change.

@lspataroG
lspataroG requested a review from a team as a code owner September 11, 2026 10:04
@netlify

netlify Bot commented Sep 11, 2026

Copy link
Copy Markdown

Deploy Preview for theagentrouter canceled.

Name Link
🔨 Latest commit bd0198f
🔍 Latest deploy log https://app.netlify.com/projects/theagentrouter/deploys/6aa3d31e823e6e0008fcb873

Anthropic charges different rates to create a cache entry depending on how
long it lives: 2x the base input rate for one hour, against 1.25x for five
minutes. Responses report the split in usage.cache_creation, but only the
combined cache_creation_input_tokens was read, so a cost expression could not
tell the two apart and under-charged long-lived cache writes.

Adds cache_creation_5m_input_tokens and cache_creation_1h_input_tokens as CEL
variables, populated from usage.cache_creation on both the streaming and
non-streaming Anthropic paths. Both are 0 when a backend does not report the
breakdown, so existing expressions are unaffected.

Signed-off-by: lspataroG <lspataro@google.com>
@lspataroG
lspataroG force-pushed the cache-creation-ttl-cost branch from 3fb053c to bd0198f Compare September 11, 2026 10:08
@lspataroG

Copy link
Copy Markdown
Author

@theagentrouter/agent-router-maintainers — could someone approve the workflow runs on this PR? Build and Test, Precommit Check and CodeQL are all sitting in action_required, so no CI has actually run yet.

For the record, the red Netlify check is expected, not a failure: the deploy status is literally "Deploy Preview canceled". The last push only touched Go files, so nothing under the site/ base directory changed since the preview built for the previous commit, and Netlify skips the build in that case. The docs change in this PR was already rendered in that earlier preview.

Locally make precommit and go test ./internal/... pass. Happy to rebase or split anything out if that helps review.

@missBerg missBerg added enhancement New feature or request area/quota Rate limiting, quota, token/cost accounting and attribution area/translation Provider/endpoint coverage and schema translation (incl. fidelity bugs) labels Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/quota Rate limiting, quota, token/cost accounting and attribution area/translation Provider/endpoint coverage and schema translation (incl. fidelity bugs) enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants