Skip to content

feat(models): Phase 3 — pricing axis to D2' (authority, model) exact-lookup contract - #4629

Draft
wpfleger96 wants to merge 4 commits into
duncan/databricks-model-label-registryfrom
hayt/usage-v2-pricing-axis
Draft

feat(models): Phase 3 — pricing axis to D2' (authority, model) exact-lookup contract#4629
wpfleger96 wants to merge 4 commits into
duncan/databricks-model-label-registryfrom
hayt/usage-v2-pricing-axis

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

Implements the D2' pricing contract from the Usage v2 plan: an optional generated pricing table/API keyed on exact (billing-authority domain, billable-model-string) pairs. No family inference, no prefix stripping, no case normalization. Billing authority is a distinct namespace from the runtime transport Provider enum; match is exact or nothing.

Manifest (scripts/model-capabilities.json)

Replaced pricing_rules (17 prefix-match rules) + pricing_exact_records (3 records) with a flat pricing_records list of 20 exact records, each containing:

  • authority — registered bare-host billing-authority token (api.anthropic.com, api.openai.com)
  • model — exact model string returned by the provider API in its response body
  • usd_per_mtok — input, output, cache_read, cache_write (null fields = price unpublished, never zero)
  • _source — models.dev entry + retrieval date (2026-08-03, SHA 6fd293ae)

Semantics: current list prices (ccusage-aligned). cache_write scoped to default ephemeral class; other classes/TTLs unknown.

Authorities included: api.anthropic.com (11 models) and api.openai.com (9 models). Databricks routes and custom/unknown endpoints are not represented. The publisher (P2 scope) must omit PricingIdentity for those routes; manifest lookup returns null/None.

Generator (scripts/generate-model-capabilities.mjs)

Replaced all old pricing machinery with strict validation + exact-emit:

Validation (hard errors on non-conforming records — no silent skips):

  • authority must be an exact registered token from the closed set (api.anthropic.com, api.openai.com, openrouter.ai); path-bearing, scheme-prefixed, or unknown values rejected
  • model must be nonempty; control/NUL/quote/backslash characters rejected
  • usd_per_mtok.input and .output: finite nonneg required; both cache_read and cache_write are required fields (absent member is hard error; use null for unpublished prices)
  • _source provenance field required and nonempty
  • Duplicate detection on exact emitted (authority, model) pairs

Generation: no toLowerCase() anywhere — records are emitted verbatim. PRICING_TABLE keys on ${authority}\0${model} (exact strings, no transforms).

modelPricing.ts is standalone — no import from modelCapabilities.ts.

Generated API

generated_model_capabilities.rs:

pub fn lookup_pricing(authority: &str, model: &str) -> Option<ModelPricing>

Exact match — no to_ascii_lowercase(), no fallback. Returns None for unknown authority, unknown model, or noncanonical casing.

modelPricing.ts (new file):

export function lookupModelPricing(authority: string, model: string): ModelPricing | null

Same semantics: exact Map lookup, null means price unknown, never treat as zero.

CI (.github/workflows/ci.yml)

Replaced the two-step regenerate-then-diff check with a single step:

- name: Check generated artifacts are up to date
  run: |
    if ! node scripts/generate-model-capabilities.mjs --check; then
      echo ""
      echo "ERROR: Generated model-capability files are stale."
      echo "Run: node scripts/generate-model-capabilities.mjs"
      echo "Then commit the regenerated files."
      exit 1
    fi

The --check mode validates all three generated artifacts (Rust capabilities, TS capabilities, TS pricing) without writing. It exits 1 with CHECK FAILED on any stale or missing file and self-maintains if a fourth artifact is added.

Tests

Rust pricing tests (23):

  • Anthropic and OpenAI model rates
  • Null guards: unknown authority, custom base URL, unknown model, empty authority/model, both empty, wrong authority for known model
  • Exact-match guards: uppercase authority → None; mixed-case authority → None; mixed-case model → None (no case folding)
  • Databricks guards: workspace URL authority → None; "databricks" authority → None

Manifest validator mutation tests (21 new, 45 total in scripts/test-manifest-validator.mjs):

  • Missing authority, unregistered/path-bearing/scheme authority, uppercase authority (case collision)
  • Empty model, unsafe chars in model (double-quote, backslash, NUL, control character)
  • Missing usd_per_mtok.output, negative rates, string cache rate, negative cache_read
  • Deleted cache_read member, deleted cache_write member (absent member rejected; use null for unpublished)
  • Missing/empty _source provenance
  • Duplicate (authority, model) pair

Gate

Check Result
cargo test -p buzz-agent pricing 23 passed, 0 failed
cargo test -p buzz-agent (full) 452 passed, 0 failed
node scripts/generate-model-capabilities.mjs --check OK: Rust + TS + TS Pricing
node --test scripts/test-manifest-validator.mjs 45 passed, 0 failed
just desktop-check Clean
git diff --check vs base Clean
modelCapabilities.ts lines 809 ≤ 1000 ✓
modelPricing.ts lines 172 ✓

Related

Stacks on #3603 (duncan/databricks-model-label-registry) — when that merges, this PR retargets to main and gets rebased.

Related: #4632 (P1 — NIP-AM normative amendment, defines authority token registry), #4000 (P2+P4a — cache-category repair, aggregation, Usage v2 backend), #4001 (P5 — Usage v2 UI). P4b (cost computation in #4000) consumes lookup_pricing(authority, model) from this PR after it lands on main.

wpfleger96 and others added 3 commits August 3, 2026 16:01
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

* origin/main: (40 commits)
  fix(mobile): recover stale relay sessions (#4372)
  chore(release): release Buzz Desktop version 0.5.4 (#4562)
  test(mobile): assert follow boundary semantics (#4559)
  docs(release): align desktop handoff instructions (#3988)
  fix: report agent usage per provider round, not once per turn (#4545)
  fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382)
  feat(desktop): improve channel template discovery (#4549)
  fix(desktop): save key backups to authorized path (#4022)
  Add channel activity hover menu (#3935)
  feat(desktop): show saved Run on settings when editing an agent (#4539)
  fix(desktop): disambiguate provider API key labels and annotate mint key (#4406)
  fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140)
  fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580)
  Polish mobile composer and messaging UI (#3918)
  ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524)
  fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411)
  fix(mobile): recover and pace live subscriptions (#3053)
  feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395)
  fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392)
  fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778)
  ...
…t apparatus (#4589)

## Summary

Retires the old hand-table authorities and transitional verification
scaffolding from the model-capability manifest arc. All production
routes now run exclusively through the generated interpreters introduced
in Phase 1 ([#3821](#3821)) and wired
in Phase 2 ([#3958](#3958)).

Stack: [#3821](#3821) →
[#3958](#3958) → this PR
Base: [#3603](#3603)

## What the manifest system is now

**Source of truth:** `scripts/model-capabilities.json`
**Generator:** `scripts/generate-model-capabilities.mjs` — emits Rust
and TS interpreters only (coverage JSON output removed)
**Generated interpreters:**
`crates/buzz-agent/src/generated_model_capabilities.rs` (+ normative
tests), `desktop/src/features/agents/ui/modelCapabilities.ts`
**Label registry:** `generate-databricks-model-names.py` →
`databricks_model_names.rs` / `databricksModelNames.ts`
**Permanent gates:** `scripts/normative-corpus.json` +
`scripts/run-corpus.mjs` (both-interpreter equivalence, 51 vectors),
`scripts/test-manifest-validator.mjs` (schema, 24 cases), regen-diff job
inside `ci.yml`
**One doc:** `scripts/MODEL_CAPABILITIES.md`

## Deleted

**Old hand-table authorities (production):**
- `getProviderEffortConfig_oldHandTable()` and all supporting helpers
from `desktop/src/features/agents/ui/buzzAgentConfig.ts`
- `normalize_effort_for_openai_route()`,
`_old_anthropic_thinking_config_for_databricks_v2()`, test-only
re-export wrappers from `crates/buzz-agent/src/config.rs`
- `strip_catalog_prefix()`, `anthropic_thinking_config()`,
`anthropic_model_supports_xhigh()`, `clamp_adaptive_effort()`,
`anthropic_efforts_for_model()`, `is_manual_budget_model()`,
`is_adaptive_thinking_model()`, `gpt5_token_matches()`,
`gpt5_base_matches()`, `openai_efforts_for_model()` from
`crates/buzz-agent/src/config.rs` — all superseded by generated
interpreter
- Old DBv2 body-level tests, `_OLD_DATABRICKS_V2_*` constants,
`model_name_segments()`, `_old_databricks_v2_route_for_model()`, all
Phase-2 behavioral differential test functions from
`crates/buzz-agent/src/llm.rs`

**Transitional scaffolding:**
- `scripts/run-differential.mjs` — old-vs-new JS differential harness
- `scripts/run-mutation-evidence.mjs` — one-time mutation evidence
runner
- `desktop/src/features/agents/ui/effortTable.fixture.json` — Phase-2
TS/Rust sync fixture
- `desktop/src/features/agents/ui/effortTable.fixture.test.mjs` —
fixture sync guard
- `.github/workflows/model-capability-regen-diff.yml` — standalone
workflow (steps folded into `ci.yml`)

**One-time evidence and generated snapshots:**
- `scripts/MUTATION_EVIDENCE.md`,
`scripts/MODEL_CAPABILITIES_SCHEMA.md`,
`scripts/MODELS_DEV_RECONCILIATION.md` — consolidated into
`scripts/MODEL_CAPABILITIES.md`
- `scripts/generated-model-capabilities-coverage.json` — full-table
snapshot (generator no longer emits it)
- `scripts/catalog-sample-fixture.json` — models.dev snapshot used only
by the deleted differential harness

## Verification

- `cargo test -p buzz-agent --lib` with `RUSTFLAGS="-D warnings"`:
**337/337** (clean — no dead_code warnings)
- `node --experimental-strip-types scripts/run-corpus.mjs`: **51/51**
- `node scripts/generate-model-capabilities.mjs` + regen diff: **clean
(exit 0)**
- `node --test scripts/test-manifest-validator.mjs`: **24/24**
- `just clippy`: zero warnings, zero errors

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Delete scripts/MODEL_CAPABILITIES.md and update the layer-3 note in
generated_model_capabilities_tests.rs to be self-contained, dropping
the now-dangling file reference.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner August 3, 2026 22:26
@wpfleger96
wpfleger96 marked this pull request as draft August 3, 2026 22:32
@wpfleger96
wpfleger96 force-pushed the hayt/usage-v2-pricing-axis branch from 4f6c083 to 9ef6623 Compare August 3, 2026 23:15
@wpfleger96 wpfleger96 changed the title feat(models): Phase 3 — add pricing axis to model-capabilities manifest feat(models): Phase 3 — pricing axis to D2' (authority, model) exact-lookup contract Aug 3, 2026
@wpfleger96
wpfleger96 force-pushed the hayt/usage-v2-pricing-axis branch 2 times, most recently from bd68126 to b636a69 Compare August 4, 2026 00:37
…lookup contract

Replaces lookup_pricing(raw_model_id) + normalize_for_pricing + 17 prefix rules
with an exact (billing-authority domain, billable-model-string) lookup. Billing
authority is a distinct namespace from the runtime transport Provider enum.
Match is exact: no case normalization, no prefix matching, no inference.

Manifest (scripts/model-capabilities.json):
- Removed pricing_rules (17 prefix-match rules), pricing_exact_records (3 records)
- Added flat pricing_records list of 20 exact records, each with authority + model +
  usd_per_mtok (input/output/cache_read/cache_write) + _source provenance
- Authorities: api.anthropic.com, api.openai.com — registered bare-host tokens per
  D2'/NIP-AM; not transport Provider enum values
- Semantics: current list prices (ccusage-aligned); cache_write scoped to ephemeral;
  other classes unknown

Generator (scripts/generate-model-capabilities.mjs):
- Removed all old pricing machinery (pricingNormalize, normalizeVersionSeparators,
  pricingRules, pricingExactRecords, resolvePricing, all TS rule-resolver emission)
- Strict pricing-record validation: registered authority (closed set: api.anthropic.com,
  api.openai.com, openrouter.ai); nonempty verbatim model; no control/NUL/quote/backslash
  in key fields; finite nonnegative input/output; both cache_read and cache_write required
  (absent member is hard error; use null for unpublished); _source provenance required;
  duplicate detection on exact emitted identities
- byAuthority grouping uses exact record values — no toLowerCase() anywhere in
  pricing generation path
- REGISTERED_AUTHORITIES set; hasUnsafeKeyChars() guard
- TS PRICING_TABLE: key is ${authority}\0${model} (exact strings, no transforms)
- lookupModelPricing(authority, model) performs exact Map lookup (no normalization)
- modelPricing.ts is standalone (no import from modelCapabilities)

Generated artifacts:
- generated_model_capabilities.rs: lookup_pricing(authority: &str, model: &str)
  performs exact match — no to_ascii_lowercase() calls, no fallback
- modelPricing.ts: exact PRICING_TABLE + lookupModelPricing(authority, model)

CI (.github/workflows/ci.yml):
- Replaced two-step "regenerate + git diff" check with a single
  node scripts/generate-model-capabilities.mjs --check step, which validates
  all three generated artifacts (Rust capabilities, TS capabilities, TS pricing)
  and exits 1 if any are stale. Self-maintains if a fourth artifact is added.

Tests (generated_model_capabilities_tests.rs, 23 pricing tests):
- Anthropic models with correct rates (claude-fable-5, claude-opus-5, claude-sonnet-5,
  claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5)
- OpenAI models incl. gpt-5.6-luna different tier, gpt-5.5 null cache_write, gpt-5-pro
  null cache fields
- Null guards: unknown authority, custom base URL, unknown model, empty authority,
  empty model, both empty, wrong authority for known model
- Exact-match guards: uppercase authority -> None, mixed-case authority -> None,
  mixed-case model -> None (no case folding)
- Databricks guards: workspace URL authority -> None, 'databricks' authority -> None

Manifest validator tests (scripts/test-manifest-validator.mjs, 45 tests total):
- 21 new pricing-record mutation tests covering: missing authority, unregistered/path-
  bearing/scheme authority, empty model, unsafe chars (double-quote, backslash, NUL,
  control), missing output, negative rates, invalid cache rate, deleted cache_read,
  deleted cache_write, missing/empty provenance, duplicate records, uppercase authority

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the hayt/usage-v2-pricing-axis branch from b636a69 to 4855d95 Compare August 4, 2026 14:59
@wpfleger96
wpfleger96 force-pushed the duncan/databricks-model-label-registry branch from 0189510 to 15aa135 Compare August 4, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant