feat(models): Phase 3 — pricing axis to D2' (authority, model) exact-lookup contract - #4629
Draft
wpfleger96 wants to merge 4 commits into
Draft
feat(models): Phase 3 — pricing axis to D2' (authority, model) exact-lookup contract#4629wpfleger96 wants to merge 4 commits into
wpfleger96 wants to merge 4 commits into
Conversation
Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com> * origin/main: (40 commits) fix(mobile): recover stale relay sessions (#4372) chore(release): release Buzz Desktop version 0.5.4 (#4562) test(mobile): assert follow boundary semantics (#4559) docs(release): align desktop handoff instructions (#3988) fix: report agent usage per provider round, not once per turn (#4545) fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382) feat(desktop): improve channel template discovery (#4549) fix(desktop): save key backups to authorized path (#4022) Add channel activity hover menu (#3935) feat(desktop): show saved Run on settings when editing an agent (#4539) fix(desktop): disambiguate provider API key labels and annotate mint key (#4406) fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140) fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580) Polish mobile composer and messaging UI (#3918) ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524) fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411) fix(mobile): recover and pace live subscriptions (#3053) feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395) fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392) fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778) ...
…t apparatus (#4589) ## Summary Retires the old hand-table authorities and transitional verification scaffolding from the model-capability manifest arc. All production routes now run exclusively through the generated interpreters introduced in Phase 1 ([#3821](#3821)) and wired in Phase 2 ([#3958](#3958)). Stack: [#3821](#3821) → [#3958](#3958) → this PR Base: [#3603](#3603) ## What the manifest system is now **Source of truth:** `scripts/model-capabilities.json` **Generator:** `scripts/generate-model-capabilities.mjs` — emits Rust and TS interpreters only (coverage JSON output removed) **Generated interpreters:** `crates/buzz-agent/src/generated_model_capabilities.rs` (+ normative tests), `desktop/src/features/agents/ui/modelCapabilities.ts` **Label registry:** `generate-databricks-model-names.py` → `databricks_model_names.rs` / `databricksModelNames.ts` **Permanent gates:** `scripts/normative-corpus.json` + `scripts/run-corpus.mjs` (both-interpreter equivalence, 51 vectors), `scripts/test-manifest-validator.mjs` (schema, 24 cases), regen-diff job inside `ci.yml` **One doc:** `scripts/MODEL_CAPABILITIES.md` ## Deleted **Old hand-table authorities (production):** - `getProviderEffortConfig_oldHandTable()` and all supporting helpers from `desktop/src/features/agents/ui/buzzAgentConfig.ts` - `normalize_effort_for_openai_route()`, `_old_anthropic_thinking_config_for_databricks_v2()`, test-only re-export wrappers from `crates/buzz-agent/src/config.rs` - `strip_catalog_prefix()`, `anthropic_thinking_config()`, `anthropic_model_supports_xhigh()`, `clamp_adaptive_effort()`, `anthropic_efforts_for_model()`, `is_manual_budget_model()`, `is_adaptive_thinking_model()`, `gpt5_token_matches()`, `gpt5_base_matches()`, `openai_efforts_for_model()` from `crates/buzz-agent/src/config.rs` — all superseded by generated interpreter - Old DBv2 body-level tests, `_OLD_DATABRICKS_V2_*` constants, `model_name_segments()`, `_old_databricks_v2_route_for_model()`, all Phase-2 behavioral differential test functions from `crates/buzz-agent/src/llm.rs` **Transitional scaffolding:** - `scripts/run-differential.mjs` — old-vs-new JS differential harness - `scripts/run-mutation-evidence.mjs` — one-time mutation evidence runner - `desktop/src/features/agents/ui/effortTable.fixture.json` — Phase-2 TS/Rust sync fixture - `desktop/src/features/agents/ui/effortTable.fixture.test.mjs` — fixture sync guard - `.github/workflows/model-capability-regen-diff.yml` — standalone workflow (steps folded into `ci.yml`) **One-time evidence and generated snapshots:** - `scripts/MUTATION_EVIDENCE.md`, `scripts/MODEL_CAPABILITIES_SCHEMA.md`, `scripts/MODELS_DEV_RECONCILIATION.md` — consolidated into `scripts/MODEL_CAPABILITIES.md` - `scripts/generated-model-capabilities-coverage.json` — full-table snapshot (generator no longer emits it) - `scripts/catalog-sample-fixture.json` — models.dev snapshot used only by the deleted differential harness ## Verification - `cargo test -p buzz-agent --lib` with `RUSTFLAGS="-D warnings"`: **337/337** (clean — no dead_code warnings) - `node --experimental-strip-types scripts/run-corpus.mjs`: **51/51** - `node scripts/generate-model-capabilities.mjs` + regen diff: **clean (exit 0)** - `node --test scripts/test-manifest-validator.mjs`: **24/24** - `just clippy`: zero warnings, zero errors --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Delete scripts/MODEL_CAPABILITIES.md and update the layer-3 note in generated_model_capabilities_tests.rs to be self-contained, dropping the now-dangling file reference. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
marked this pull request as draft
August 3, 2026 22:32
wpfleger96
force-pushed
the
hayt/usage-v2-pricing-axis
branch
from
August 3, 2026 23:15
4f6c083 to
9ef6623
Compare
wpfleger96
force-pushed
the
hayt/usage-v2-pricing-axis
branch
2 times, most recently
from
August 4, 2026 00:37
bd68126 to
b636a69
Compare
…lookup contract
Replaces lookup_pricing(raw_model_id) + normalize_for_pricing + 17 prefix rules
with an exact (billing-authority domain, billable-model-string) lookup. Billing
authority is a distinct namespace from the runtime transport Provider enum.
Match is exact: no case normalization, no prefix matching, no inference.
Manifest (scripts/model-capabilities.json):
- Removed pricing_rules (17 prefix-match rules), pricing_exact_records (3 records)
- Added flat pricing_records list of 20 exact records, each with authority + model +
usd_per_mtok (input/output/cache_read/cache_write) + _source provenance
- Authorities: api.anthropic.com, api.openai.com — registered bare-host tokens per
D2'/NIP-AM; not transport Provider enum values
- Semantics: current list prices (ccusage-aligned); cache_write scoped to ephemeral;
other classes unknown
Generator (scripts/generate-model-capabilities.mjs):
- Removed all old pricing machinery (pricingNormalize, normalizeVersionSeparators,
pricingRules, pricingExactRecords, resolvePricing, all TS rule-resolver emission)
- Strict pricing-record validation: registered authority (closed set: api.anthropic.com,
api.openai.com, openrouter.ai); nonempty verbatim model; no control/NUL/quote/backslash
in key fields; finite nonnegative input/output; both cache_read and cache_write required
(absent member is hard error; use null for unpublished); _source provenance required;
duplicate detection on exact emitted identities
- byAuthority grouping uses exact record values — no toLowerCase() anywhere in
pricing generation path
- REGISTERED_AUTHORITIES set; hasUnsafeKeyChars() guard
- TS PRICING_TABLE: key is ${authority}\0${model} (exact strings, no transforms)
- lookupModelPricing(authority, model) performs exact Map lookup (no normalization)
- modelPricing.ts is standalone (no import from modelCapabilities)
Generated artifacts:
- generated_model_capabilities.rs: lookup_pricing(authority: &str, model: &str)
performs exact match — no to_ascii_lowercase() calls, no fallback
- modelPricing.ts: exact PRICING_TABLE + lookupModelPricing(authority, model)
CI (.github/workflows/ci.yml):
- Replaced two-step "regenerate + git diff" check with a single
node scripts/generate-model-capabilities.mjs --check step, which validates
all three generated artifacts (Rust capabilities, TS capabilities, TS pricing)
and exits 1 if any are stale. Self-maintains if a fourth artifact is added.
Tests (generated_model_capabilities_tests.rs, 23 pricing tests):
- Anthropic models with correct rates (claude-fable-5, claude-opus-5, claude-sonnet-5,
claude-opus-4-8, claude-sonnet-4-6, claude-haiku-4-5)
- OpenAI models incl. gpt-5.6-luna different tier, gpt-5.5 null cache_write, gpt-5-pro
null cache fields
- Null guards: unknown authority, custom base URL, unknown model, empty authority,
empty model, both empty, wrong authority for known model
- Exact-match guards: uppercase authority -> None, mixed-case authority -> None,
mixed-case model -> None (no case folding)
- Databricks guards: workspace URL authority -> None, 'databricks' authority -> None
Manifest validator tests (scripts/test-manifest-validator.mjs, 45 tests total):
- 21 new pricing-record mutation tests covering: missing authority, unregistered/path-
bearing/scheme authority, empty model, unsafe chars (double-quote, backslash, NUL,
control), missing output, negative rates, invalid cache rate, deleted cache_read,
deleted cache_write, missing/empty provenance, duplicate records, uppercase authority
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
force-pushed
the
hayt/usage-v2-pricing-axis
branch
from
August 4, 2026 14:59
b636a69 to
4855d95
Compare
wpfleger96
force-pushed
the
duncan/databricks-model-label-registry
branch
from
August 4, 2026 21:16
0189510 to
15aa135
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the D2' pricing contract from the Usage v2 plan: an optional generated pricing table/API keyed on exact
(billing-authority domain, billable-model-string)pairs. No family inference, no prefix stripping, no case normalization. Billing authority is a distinct namespace from the runtime transportProviderenum; match is exact or nothing.Manifest (
scripts/model-capabilities.json)Replaced
pricing_rules(17 prefix-match rules) +pricing_exact_records(3 records) with a flatpricing_recordslist of 20 exact records, each containing:authority— registered bare-host billing-authority token (api.anthropic.com,api.openai.com)model— exact model string returned by the provider API in its response bodyusd_per_mtok— input, output, cache_read, cache_write (nullfields = price unpublished, never zero)_source— models.dev entry + retrieval date (2026-08-03, SHA6fd293ae)Semantics: current list prices (ccusage-aligned).
cache_writescoped to defaultephemeralclass; other classes/TTLs unknown.Authorities included:
api.anthropic.com(11 models) andapi.openai.com(9 models). Databricks routes and custom/unknown endpoints are not represented. The publisher (P2 scope) must omitPricingIdentityfor those routes; manifest lookup returnsnull/None.Generator (
scripts/generate-model-capabilities.mjs)Replaced all old pricing machinery with strict validation + exact-emit:
Validation (hard errors on non-conforming records — no silent skips):
authoritymust be an exact registered token from the closed set (api.anthropic.com,api.openai.com,openrouter.ai); path-bearing, scheme-prefixed, or unknown values rejectedmodelmust be nonempty; control/NUL/quote/backslash characters rejectedusd_per_mtok.inputand.output: finite nonneg required; bothcache_readandcache_writeare required fields (absent member is hard error; usenullfor unpublished prices)_sourceprovenance field required and nonempty(authority, model)pairsGeneration: no
toLowerCase()anywhere — records are emitted verbatim.PRICING_TABLEkeys on${authority}\0${model}(exact strings, no transforms).modelPricing.tsis standalone — no import frommodelCapabilities.ts.Generated API
generated_model_capabilities.rs:Exact match — no
to_ascii_lowercase(), no fallback. ReturnsNonefor unknown authority, unknown model, or noncanonical casing.modelPricing.ts(new file):Same semantics: exact Map lookup,
nullmeans price unknown, never treat as zero.CI (
.github/workflows/ci.yml)Replaced the two-step regenerate-then-diff check with a single step:
The
--checkmode validates all three generated artifacts (Rust capabilities, TS capabilities, TS pricing) without writing. It exits 1 withCHECK FAILEDon any stale or missing file and self-maintains if a fourth artifact is added.Tests
Rust pricing tests (23):
None; mixed-case authority →None; mixed-case model →None(no case folding)None;"databricks"authority →NoneManifest validator mutation tests (21 new, 45 total in
scripts/test-manifest-validator.mjs):usd_per_mtok.output, negative rates, string cache rate, negative cache_readcache_readmember, deletedcache_writemember (absent member rejected; usenullfor unpublished)_sourceprovenance(authority, model)pairGate
cargo test -p buzz-agent pricingcargo test -p buzz-agent(full)node scripts/generate-model-capabilities.mjs --checknode --test scripts/test-manifest-validator.mjsjust desktop-checkgit diff --checkvs basemodelCapabilities.tslinesmodelPricing.tslinesRelated
Stacks on #3603 (
duncan/databricks-model-label-registry) — when that merges, this PR retargets to main and gets rebased.Related: #4632 (P1 — NIP-AM normative amendment, defines authority token registry), #4000 (P2+P4a — cache-category repair, aggregation, Usage v2 backend), #4001 (P5 — Usage v2 UI). P4b (cost computation in #4000) consumes
lookup_pricing(authority, model)from this PR after it lands on main.