Skip to content

translator: support field cache_write_tokens for openai - #2675

Merged
aabchoo merged 6 commits into
theagentrouter:mainfrom
aabchoo:aaron/update-openai-translation-cache
Sep 14, 2026
Merged

aabchoo merged 6 commits into
theagentrouter:mainfrom
aabchoo:aaron/update-openai-translation-cache

Conversation

@aabchoo

@aabchoo aabchoo commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Description

Adding an additional field cache_write_tokens for openai translation and marking cache_creation_tokens as deprecated in v1.3.0. This fixes cache writes not being accounted for for gpt-5.6+ models.

Openai translated fields returns cache_write_tokens as part of the usage response:

  "usage": {
    "prompt_tokens": 1116,
    "completion_tokens": 278,
    "total_tokens": 1394,
    "prompt_tokens_details": {
      "cached_tokens": 0,
      "cache_creation_tokens": 1113, # same
      "cache_write_tokens": 1113,
    },
}

https://developers.openai.com/api/docs/guides/prompt-caching/diagnostics#confirm-the-improvement

Updating the translation layer to parse the cache_write_tokens from/to openai as well. Keeping the legacy field in temporarily but will move to remove it in a later release. Cache write tokens created by openai are currently ignored.

This means the response of openai translation will be cache_write_tokens in addition to cache_creation_tokens.

Related Issues/PRs (if applicable)

n/a

Signed-off-by: achoo30 <achoo30@bloomberg.net>
Signed-off-by: achoo30 <achoo30@bloomberg.net>
@netlify

netlify Bot commented Sep 10, 2026

Copy link
Copy Markdown

Deploy Preview for theagentrouter canceled.

Name Link
🔨 Latest commit e662a27
🔍 Latest deploy log https://app.netlify.com/projects/theagentrouter/deploys/6aa816478dad3c0008b61875

@aabchoo aabchoo changed the title translation: support latest cache write field cache_write_tokens for openai translator: support latest cache write field cache_write_tokens for openai Sep 10, 2026
@aabchoo aabchoo changed the title translator: support latest cache write field cache_write_tokens for openai translator: support field cache_write_tokens for openai Sep 10, 2026
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.58333% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/translator/openai_completions.go 50.00% 2 Missing ⚠️
internal/translator/openai_openai.go 50.00% 2 Missing ⚠️
internal/translator/openai_awsbedrock.go 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@aabchoo
aabchoo marked this pull request as ready for review September 10, 2026 19:56
@aabchoo
aabchoo requested a review from a team as a code owner September 10, 2026 19:56
CachedTokens int `json:"cached_tokens,omitzero"`
// Tokens written to the cache.
CacheCreationTokens int `json:"cache_creation_input_tokens,omitzero"`
CacheWriteTokens int `json:"cache_write_tokens,omitzero"`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i understand we are renaming because openai recently added the cache-write field
but since this will remove a field from the response users receives, i think we should do the field-deletion over atleast 2 minor releases
for this release we can have both CacheCreationTokens and CacheWriteTokens fields in response, note CacheWriteTokens as deprecated in upcoming release and remove it 2 versions down the line

@aabchoo
aabchoo enabled auto-merge (squash) September 14, 2026 16:15
@aabchoo
aabchoo merged commit f6518da into theagentrouter:main Sep 14, 2026
63 of 65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants