Repository navigation
fix(telemetry): avoid counting routed cache tokens twice - #5286
Merged
gabsprogrammer merged 2 commits intoSep 15, 2026
Merged
gabsprogrammer merged 2 commits into
gabsprogrammer merged 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and change
Routed requests report inclusive OpenAI/LiteLLM prompt counts alongside cache counters. Token Spy sums disjoint input/output/cache categories: 20 prompt tokens (8 read + 4 write) and 5 output tokens therefore became 37 instead of 25.
Partition the reported input count when model-router and the LiteLLM callback construct their metadata event. Cache counters cannot exceed the available prompt total. Router streaming keeps aggregating inclusive snapshots first, then partitions once, including when cache details arrive late. Provider responses remain unchanged. Document the routed contract and wire the callback suite into Linux CI.
The inclusive contract is described in the OpenAI prompt caching guide. I also inspected the pinned LiteLLM 1.90.7 wheel: its Anthropic transformation adds cache reads/writes into prompt_tokens before creating normalized Usage.
Validation
Related work and limits
Live overlap review: #2519 targets the separate Token Spy OpenAI proxy parser on main; routed metadata bypasses that parser. #4888 adds LiteLLM cache-creation field extraction. This PR keeps that mapping unchanged and partitions whatever counters the producer has extracted, so its later field addition can use the same partitioning. These related changes should be reviewed together; this does not claim to implement #4888.
Historical stored rows are not rewritten. Currently unrecognized cache fields remain in the inclusive input remainder until their producer mapping is added. Tests use local fixtures, not live provider traffic or the full installed LiteLLM runtime. Independent human review and live telemetry qualification remain pending.
AI-assisted implementation and validation by Codex; ready for review is not approval to merge.