Skip to content

Haiku 5.5's prompt-length pricing tiers aren't modelled — long chats under-estimate by up to 5× #1003

Open

Description

@edspencer

Follow-up from #1002, which adds Claude Haiku 5.5 to the model catalog.

The gap

Haiku 5.5 is the only model in the catalog priced by prompt length rather than at a flat per-token rate (pricing docs):

Prompt size Input Output 5m cache write Cache read
≤ 100,000 tokens $0.10 $0.50 $0.125 $0.01
> 100,000 tokens $0.50 $2.50 $0.625 $0.05

A request's prompt length counts all of its input tokens, including cache reads and writes. Each request is priced on its own, so a request over the threshold pays the higher rate even when most of its prompt was a cache hit.

estimateCostUsd in packages/server/src/models.ts prices a chat's cumulative TokenTotals, which no longer carry per-request prompt lengths. It therefore can't tell the tiers apart. #1002 pins Haiku 5.5 at the base tier, which is exact for short chats and under-estimates a long one by up to 5×. A test in test/unit/usage.test.ts pins this so a change is deliberate rather than accidental.

Why it's fixable

The tier is decidable where the totals are built. readSessionTokenUsageFile (packages/server/src/usage.ts) already computes each turn's prompt length — it's the same input + cacheCreation + cacheRead sum it uses for contextTokens — so each turn could be bucketed into a tier at read time.

Sketch, not a decision

One shape that fits the existing seams:

  • ModelPricing gains an optional longContext?: { thresholdTokens, inputPer1M, outputPer1M }.
  • TokenTotals splits into a base TokenCounts plus an optional longContext?: TokenCounts holding the over-threshold portion.
  • usage.ts picks the bucket per turn; estimateCostUsd prices each at its own rate.
  • mergeByModelInto in subagents.ts folds the nested counts too, so sub-agent roll-ups stay correct.

The cost is that TokenTotals stops being four flat numbers, which several call sites construct literally. Worth weighing against how much Haiku 5.5 actually gets used for long chats before building it.

Filed for a design decision before implementation rather than folding it into #1002.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions