fix(aio): stop over-billing cache-heavy gemini generations

[fivestarspicy]

Problem

• Teams using Gemini context caching see reported input cost far above what the provider actually charges, on every generation where the cached prompt is most of the input.
• Gemini reports cache tokens as a subset of input tokens. Under explicit context caching, cached_content_token_count is counted when the cache is created and prompt_token_count is counted per request, so the two counts land a few percent apart.
• fix(aio): stop negative AI costs when cache reads exceed input tokens #74183 added a fallback that reads any input < cache_read + cache_write as proof of exclusive accounting.
• That fallback flips these events to exclusive, so the full input bills at the prompt rate and the cache bills again at the cache rate.
• Gemini prices cache reads at a tenth of the prompt rate, so a nearly fully cached prompt bills several times over.
• Any inclusive-reporting provider that breaks the invariant slightly hits this, not only Gemini.

Changes

• The fallback now requires the cache pool to exceed input by more than 1.5x before resolving exclusive.
• Genuine exclusive reporting sends only the incremental turn as input, so it overshoots by orders of magnitude and stays well clear of the threshold.
• clampTextTokens now clamps the residual text pool to zero on every path, not only when modality tokens are present.
• The clamp is what stops an inclusive event with a small overshoot from billing a negative residual, which is the outcome fix(aio): stop negative AI costs when cache reads exceed input tokens #74183 set out to remove.

I picked 1.5x by measuring both populations on production generations. Reporting slop on inclusive providers clusters within roughly 8% over input. Genuine exclusive reporting sits about three orders of magnitude higher. The threshold sits in the empty band between them, so neither population is near it.

Note

An exclusive-reporting integration whose cache pool sits between 1x and 1.5x of input now resolves inclusive and is slightly underbilled. #74183 already documented and accepted that direction of error. The reverse direction is the expensive one, and this PR closes it.

How did you test this code?

• Ran npx jest src/ingestion/pipelines/ai/costs/ src/ingestion/pipelines/ai/process-ai-event.test.ts. All suites pass.
• New resolver case: an inclusive Gemini event whose cache pool sits just over input must stay inclusive. Removing the ratio makes it fail.
• New cost case: the same event bills the cache at the cache rate with the residual clamped to zero. This fails if the flip returns, and fails differently if the clamp is dropped.
• Updated handles negative token counts gracefully. It asserted a negative cost for negative input tokens, which is the behavior this PR removes. It now asserts zero.
• Existing fallback cases were left alone and still pass, because their overshoot ratios sit far above the threshold.
• Not run: the full nodejs suite, and any live ingestion. The change is confined to pure functions covered by the suites above.

Automatic notifications

• Publish to changelog?

Docs update

None. products/ai_observability/skills/exploring-llm-costs/references/cache-accounting.md describes ingestion as auto-detecting the value, which stays true.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

I (the PostHog Slack app, running Claude) investigated this from a Slack thread on Gemini cost reconciliation, and a PostHog engineer directed the fix. I did not set an assignee because I could not verify their GitHub handle in session.

Skills invoked: /writing-code-comments, /writing-tests, /writing-pr-descriptions.

Before settling on the threshold I checked two other explanations and ruled both out. I traced the Gemini path in posthog-python and confirmed it copies both counts off a single usage_metadata object with no arithmetic, so the skew is not a mapping bug. I then found a real asymmetry in merge_usage_stats, where cumulative mode overwrites input_tokens on every chunk but only writes cache_read_input_tokens when it is above zero, which would let the two counts come from different chunks. That path needs streaming, and the affected calls are non-streaming, so it is not the cause here.

I also considered setting $ai_cache_reporting_exclusive explicitly in the Gemini wrapper. That is correct and worth doing separately, but it only reaches updated SDKs and does not cover OTel or manual capture, so it does not recover the affected population on its own.

Public artifact: no customer or session material reached this PR. The token values in the new tests are invented to exercise the ratio boundary.

---

Created with PostHog from a Slack thread

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[fivestarspicy]

Closing in favor of posthog-python#860, which declares Gemini's cache accounting model in the SDK rather than inferring it in ingestion.

The approach here layered a tolerance on top of the existing inference, which meant a threshold constant tuned against observed token shapes. Declaring the accounting model at the source removes the need to infer it at all for providers we know.

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs