Skip to content

fix(telemetry): avoid counting routed cache tokens twice - #5286

Merged
gabsprogrammer merged 2 commits into
Osmantic:public-betafrom
0xacee:fix/telemetry-split-inclusive-cache-usage
Sep 15, 2026
Merged

gabsprogrammer merged 2 commits into
Osmantic:public-betafrom
0xacee:fix/telemetry-split-inclusive-cache-usage

Conversation

@0xacee

@0xacee 0xacee commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem and change

Routed requests report inclusive OpenAI/LiteLLM prompt counts alongside cache counters. Token Spy sums disjoint input/output/cache categories: 20 prompt tokens (8 read + 4 write) and 5 output tokens therefore became 37 instead of 25.

Partition the reported input count when model-router and the LiteLLM callback construct their metadata event. Cache counters cannot exceed the available prompt total. Router streaming keeps aggregating inclusive snapshots first, then partitions once, including when cache details arrive late. Provider responses remain unchanged. Document the routed contract and wire the callback suite into Linux CI.

The inclusive contract is described in the OpenAI prompt caching guide. I also inspected the pinned LiteLLM 1.90.7 wheel: its Anthropic transformation adds cache reads/writes into prompt_tokens before creating normalized Usage.

Validation

  • Fresh upstream/public-beta base; 22 new boundary cases: 18 failed / 4 passed before the fix.
  • Full model-router and LiteLLM suites: 118 passed. ASGI upstream fixtures cover Chat, Responses, no cache, all cache, mixed categories and inconsistent counters; streaming tests cover early/late cache snapshots.
  • Callback tests enqueue through async_log_success_event, validate the actual routed event, persist it to temporary SQLite and assert the report total is 25.
  • Ruff comparison found no added diagnostics (72 pre-existing router findings); callback files are clean. git diff --check passed. Existing framework deprecation warnings remain.

Related work and limits

Live overlap review: #2519 targets the separate Token Spy OpenAI proxy parser on main; routed metadata bypasses that parser. #4888 adds LiteLLM cache-creation field extraction. This PR keeps that mapping unchanged and partitions whatever counters the producer has extracted, so its later field addition can use the same partitioning. These related changes should be reviewed together; this does not claim to implement #4888.

Historical stored rows are not rewritten. Currently unrecognized cache fields remain in the inclusive input remainder until their producer mapping is added. Tests use local fixtures, not live provider traffic or the full installed LiteLLM runtime. Independent human review and live telemetry qualification remain pending.

AI-assisted implementation and validation by Codex; ready for review is not approval to merge.

@gabsprogrammer
gabsprogrammer merged commit 391a967 into Osmantic:public-beta Sep 15, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants