Skip to content

fix(pricing): normalize Bedrock dot-form model names and add opus-4-7#17

Open
estelledc wants to merge 1 commit into
hmenzagh:mainfrom
estelledc:fix/pricing-bedrock-formats-and-opus-47
Open

fix(pricing): normalize Bedrock dot-form model names and add opus-4-7#17
estelledc wants to merge 1 commit into
hmenzagh:mainfrom
estelledc:fix/pricing-bedrock-formats-and-opus-47

Conversation

@estelledc
Copy link
Copy Markdown

Summary

Fixes ~56% cost under-counting on Bedrock-proxy users (and a smaller hit on direct-API Opus 4.7 users). Independent of #16 — that one fixes token counts, this one fixes the per-token rates.

Two bugs converging

PRICING_TABLE uses dash-form patterns (opus-4-6, haiku-4-5). Two cases never match anything in the table:

  1. Bedrock proxy emits dot form. aws.claude-opus-4.7 does not contain opus-4-6 or opus-4-7 (or anything else dash-separated in the table). Every Bedrock call falls to FALLBACK.
  2. opus-4-7 is missing from the table. Even direct-API users on the newest Opus fall to FALLBACK.

FALLBACK_PRICING was hard-coded Sonnet ($3/$15/$0.30). So:

Model Old behavior Should be Effect
aws.claude-opus-4.7 Sonnet $3/$15 Opus $5/$25 under by ~5/3
aws.claude-opus-4.6 Sonnet Opus under by ~5/3
aws.claude-haiku-4.5 Sonnet Haiku $1/$5 over by 3× (smaller volume)
aws.claude-sonnet-4.5 Sonnet Sonnet accidentally correct
claude-opus-4-7 (direct API) Sonnet Opus under by ~5/3

Net on a real Bedrock-proxy archive (28k events, ~12.7k unique messages, 8 active days):

total cost
Pre-fix CCMeter $1,718
Post-fix CCMeter (this PR, simulated) $3,007
Reference (CCCostMonitor / LiteLLM) $3,007

(Token totals are unaffected; this PR only changes per-token rates. PR #16 separately fixes token counts on the same proxy environment.)

Fix

model_pricing now does a two-stage lookup:

  1. Normalize the input — lowercase, replace . with - — then substring-match against PRICING_TABLE. A single dash-form table now covers both name styles.
  2. Family fallback on miss — if the normalized string contains opus, haiku, or sonnet, use the latest known pricing for that family. Truly unknown strings still default to Sonnet, preserving prior behavior for non-family inputs.

Adds opus-4-7 to the explicit table.

The family-default change is the load-bearing piece: it means a future Opus release we don't yet know about silently bills at Opus rates instead of Sonnet rates. Under-billing Opus traffic was the larger risk; over-billing a hypothetical future Sonnet (which would still match sonnet substring anyway) is not.

Cache schema bump

Bumped CURRENT_SCHEMA_VERSION 2 → 3. Existing v2 caches on user disks were populated by the buggy pricing and would otherwise freeze the under-counted cost in place. The existing "Cache rebuilt" banner handles the one-time recompute UX with no manual intervention.

Note for the maintainer: PR #16 also bumps schema to v3 for an independent reason (token-count fix). If both PRs land, the second-merged one needs a trivial bump to v4 (or we collapse both bumps into one v3 with a combined comment). The bump itself is just a one-line change in src/data/cache.rs.

Tests

12 new tests in src/data/models.rs::tests covering:

  • Direct-API dash form (Opus 4-6, 4-7, Sonnet, Haiku 4-5, legacy Opus 4-1, 3-5-haiku)
  • Bedrock dot form (Opus 4-7, 4-6, Haiku, Sonnet)
  • Family fallback for unknown versions (claude-opus-9-9 → Opus, not Sonnet)
  • Case-insensitive lookup (uppercase ARN-style ids)
  • Truly-unknown strings still default to Sonnet (preserves prior contract)

All 64 prior tests still pass. The 2 pre-existing rate_limits failures on main are unrelated to this change.

Test plan

  • cargo test models:: → 15 pass (3 baseline + 12 new)
  • cargo test → 64 pass (same 2 pre-existing rate_limits failures as on main)
  • Real-data sanity: post-fix simulated cost on a 28k-event Bedrock archive matches the LiteLLM-pricing reference exactly
  • Reviewer to confirm direct-API users on Sonnet/Haiku see no regression (covered by prices_direct_sonnet_dash and prices_direct_haiku_4_5_dash)

🤖 Generated with Claude Code

Two bugs in model_pricing converged into massive cost under-counting for
Bedrock-proxy users:

1. PRICING_TABLE patterns are dash-form ("opus-4-6"), but Bedrock emits
   dot-form model strings ("aws.claude-opus-4.7"). String contains never
   matched, so every Bedrock call hit the FALLBACK.
2. PRICING_TABLE is missing an "opus-4-7" entry. Direct-API users on the
   newest Opus also hit the FALLBACK.

The FALLBACK was Sonnet pricing ($3/$15) — so:
- aws.claude-opus-4.7  → billed as Sonnet (under by ~5/3)
- aws.claude-opus-4.6  → billed as Sonnet (under by ~5/3)
- aws.claude-haiku-4.5 → billed as Sonnet (over by 3×; smaller volume)
- claude-opus-4-7      → billed as Sonnet (under by ~5/3)

Fix:
- Normalize lower-case + dots-to-dashes before lookup, so a single
  dash-form table covers both name styles.
- Add opus-4-7 entry alongside opus-4-6 / opus-4-5.
- Family fallback (opus/sonnet/haiku → latest known pricing for that
  family) instead of defaulting every miss to Sonnet, so a future
  unknown Opus release does not silently mis-bill at Sonnet rates.

Bumps cache schema to v3 — existing v2 caches froze the under-counted
cost in place; the existing "Cache rebuilt" banner handles the
one-time rebuild on first launch.

Tests: 12 new in src/data/models.rs covering direct-API dash form,
Bedrock dot form, family fallback, case-insensitive lookup, and the
truly-unknown-string default. All 64 prior tests still pass (the 2
pre-existing rate_limits failures on main are unrelated).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant