Skip to content

fix(pricing): general vendor-rename normalizer + fuzzy-pricing UI disclosure - #194

Merged
pitimon merged 1 commit into
mainfrom
fix/193-192-vendor-rename-normalizer-and-fuzzy-disclosure
Aug 22, 2026
Merged

fix(pricing): general vendor-rename normalizer + fuzzy-pricing UI disclosure#194
pitimon merged 1 commit into
mainfrom
fix/193-192-vendor-rename-normalizer-and-fuzzy-disclosure

Conversation

@pitimon

@pitimon pitimon commented Aug 22, 2026

Copy link
Copy Markdown
Owner

PR Goal (one sentence)

Close two gaps a cross-verification review found in #187/#188: the fable/mythos fix was a per-name patch with no general handling and a bare-substring collision risk (#193), and fuzzy-matched prices are computed but never disclosed to the user (#192).

Scope

  • CLI (src/)
  • Dashboard (dashboard/)
  • Docs / CI / config

Checklist

  • npm run ci:local passes, or focused test commands are listed below with rationale
  • New dashboard/user-facing strings go through dashboard/src/content/copy.csv (usage.overview.provider_fuzzy_pricing)
  • Commits follow conventional style (feat: / fix: / refactor: / docs: / chore: / test: / ci:)
  • PR description explains why, not just what
  • Version bump files are in lockstep when release-bound — deferred to a follow-up chore(release) PR per the established pattern (chore(release): 0.39.53 #189 after fix(pricing): resolve fable/mythos canary variants via fuzzy match #188)
  • npm publish state is recorded when release-bound — same, deferred to the release PR

Why

A cross-verification review of #187/#188 (17-question pre-ship checklist) flagged three gaps after independently reproducing the original bug fix against the published npm artifacts and 76 real production queue rows. #191 needs external billing data and is tracked separately. This PR resolves the two gaps that are resolvable from source:

#193 — no general normalizer, and a real collision risk in the narrow fix

#188s fix added {match:"fable", ref:"claude-fable-5"} / {match:"mythos", ref:"claude-mythos-5"} as bare curated-fuzzy substring rules. Two problems verified independently against the published pricing seed:

  1. No structural fix. Any future auto-pilot/canary rename of an already-priced model (e.g. claude-auto-pilot-opus-4-8-v1-canary) reproduces the exact Claude auto-pilot/canary model ids (fable/mythos) bill $0 — fuzzy pricing match misses non-contiguous substrings #187 bug and needs another one-line curated patch. Verified: before this PR, that hypothetical id resolves miss; after, it resolves litellm:exact via the new denoise retry with no curated entry added.
  2. Real collision risk, not hypothetical. claude-mythos-preview is a genuine LiteLLM-priced model in the published seed (generated_at: 2026-08-22T15:00:43.511Z) that contains the bare substring "mythos". It currently survives because litellm:exact runs before curated:fuzzy in the resolve order, but the bare rule was one design change away from silently mispricing it (or any future model with the same shape and no exact entry) at the Opus-tier Fable/Mythos rate. Narrowed the rules to "claude-fable" / "claude-mythos" to remove that risk while keeping the Claude auto-pilot/canary model ids (fable/mythos) bill $0 — fuzzy pricing match misses non-contiguous substrings #187 fix intact.

Fix: matcher.js gains a stripVendorRenameNoise() denoise retry — strips -auto-pilot- infix and trailing -vN-canary/-canary, then re-runs the full 7-tier lookup ladder exactly once, only after every tier has already missed on the raw id (so a genuine exact match on the noisy raw id is never shadowed).

#192 — fuzzy pricing computed but never shown to the user

model-breakdown.ts already computes fuzzyPricingModels per provider and plan-value.ts already uses it to mark the plan-value comparison as a "floor," but ProviderBreakdownCard.jsx only ever rendered missingPricingModels ("N pricing missing"). A user looking at a provider total like $3,164.69 (the real post-#188 total for claude-auto-pilot-fable-v1-canary across 76 production rows) had no way to tell it apart from an exactly-priced total.

Fix: wire the existing fuzzyPricingModels into the same card, mirroring the missing-pricing pattern exactly — new copy key usage.overview.provider_fuzzy_pricing ("{{count}} pricing estimated").

Codex Context (required when requesting @codex review)

  • Delta since last Codex review: first review on this branch; builds on merged fix(pricing): resolve fable/mythos canary variants via fuzzy match #188/chore(release): 0.39.53 #189/chore(pricing): record 0.39.53 publish snapshot #190.
  • Intended behavior / invariants: the denoise retry never runs before the 7 existing tiers, never runs twice (_denoised guard), never fires when stripping is a no-op, and never shadows a real exact match on the raw (noisy) id. The narrowed fuzzy rules still resolve every id fix(pricing): resolve fable/mythos canary variants via fuzzy match #188 fixed; they no longer match models that merely contain "fable"/"mythos" without the "claude-" prefix.
  • Edge cases covered: raw-id exact match precedence over denoise retry; no-noise strings pass through unchanged; empty/null input; bare-substring false-positive (some-vendor-fable-mini) now correctly misses; claude-mythos-preview (real seed collision) still resolves via litellm:exact, unaffected.
  • Tests run (command + result):
    • node --test test/pricing.test.js → 39/39 pass (5 new: stripVendorRenameNoise unit test, denoise-retry-resolves-without-curated-rule, denoise-retry-does-not-shadow-raw-exact-match, denoise-retry-no-loop-on-clean-string, fuzzy-rule-scoped-not-bare-substring)
    • npm --prefix dashboard run test -- ProviderBreakdownCard → 10/10 pass (2 new: shows-fuzzy-caveat, does-not-show-fuzzy-caveat-when-exact)
    • npm run ci:local → full pass: dashboard build + 320/320 dashboard tests, 1059/1059 product-suite tests, all validators (copy, ui-hardcode, guardrails, curated-expiry, outbound, openwiki)
  • Known gaps / out of scope: Verify claude-fable-5 pinned price ($10/$50) against actual Anthropic billing for canary route #191 (verify the $10/$50 Fable 5 rate against a real invoice) is out of scope — needs external billing access not available in this environment, tracked separately and left open.

Risk Layer Trigger (if any)

  • Public exposure / share links / unauthenticated access
  • Auth/session/token handling
  • Cross-endpoint invariants or shared logic
  • External gateway / environment constraints

(No risk-layer trigger — pure pricing-lookup logic + a read-only UI disclosure, no I/O/auth/exposure surface touched.)

Regression Test Gate

Most likely regression surface

Verification method (choose at least one)

Uncovered scope

…closure

Two follow-ups from the #187/#188 cross-verification review.

1. matcher.js: add a scoped denoise retry (issue #193). #188 fixed the
   fable/mythos canary regression with a per-name curated fuzzy rule,
   but left the structural gap open — any future auto-pilot/canary
   rename of an already-priced model would reproduce the same bug and
   require another one-line patch. The denoise retry strips the
   '-auto-pilot-' infix and trailing '-vN-canary'/'-canary' suffix and
   retries the full lookup ladder exactly once, only after every other
   tier has missed on the raw id, so a genuine exact match on the raw
   id (however unlikely) is never shadowed.

   Also narrows the #188 fuzzy rules from bare 'fable'/'mythos'
   substrings to 'claude-fable'/'claude-mythos'. Verified against the
   published pricing seed that a real collision risk existed:
   claude-mythos-preview is a genuine LiteLLM-priced model that
   happens to contain the substring 'mythos' — the bare rule would
   have silently mispriced any future model in the same shape that
   lacks its own exact LiteLLM/curated entry.

2. dashboard: wire the already-computed fuzzyPricingModels (from
   model-breakdown.ts) into ProviderBreakdownCard (issue #192). The
   provider card already discloses '{{count}} pricing missing' for
   unpriced models; it now also discloses '{{count}} pricing
   estimated' when any model resolved via substring/fuzzy match
   rather than an exact price, so a user sees when a total like
   $3,164.69 is a curated estimate, not a reconciled invoice figure.

Verified: node --test test/pricing.test.js (39/39), dashboard
ProviderBreakdownCard suite (10/10, incl. 2 new), npm run ci:local
full gate (dashboard 320/320, product suite 1059/1059, all
validators) — all green.

Addresses #193, #192.
@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@pitimon
pitimon merged commit 88385b6 into main Aug 22, 2026
2 checks passed
@pitimon
pitimon deleted the fix/193-192-vendor-rename-normalizer-and-fuzzy-disclosure branch August 22, 2026 15:27
@pitimon pitimon mentioned this pull request Aug 22, 2026
11 tasks
pitimon added a commit that referenced this pull request Aug 22, 2026
Version bump for the vendor-rename normalizer + fuzzy-pricing UI
disclosure fix (#193, #192; merged in #194). Updates the
version-lockstep test's hardcoded package.json expectation in step.

Co-authored-by: itarun.p <itarun.p@somapait.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant