Skip to content

feat(models): manifest-driven model capabilities and curated labels - #3603

Open
wpfleger96 wants to merge 23 commits into
mainfrom
duncan/databricks-model-label-registry
Open

feat(models): manifest-driven model capabilities and curated labels#3603
wpfleger96 wants to merge 23 commits into
mainfrom
duncan/databricks-model-label-registry

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

This PR establishes a single hand-curated model-capability manifest (scripts/model-capabilities.json) as the source of truth for model metadata — display labels, supported reasoning-effort sets, default efforts, thinking modes, and Databricks v2 wire routes. A generator emits equivalent Rust and TypeScript interpreters from it, production resolution in both languages runs on the generated interpreters, and the superseded hand-maintained tables are deleted. It also ships the curated Databricks label registry and the frontend resolveModelLabel() contract.

Closes #3586. Landed as three sub-PRs into this branch: #3821 (manifest + generator), #3958 (production integration), #4589 (hand-table retirement).

Capability manifest and generated interpreters

scripts/generate-model-capabilities.mjs reads the manifest and emits two committed artifacts from one source of truth:

Artifact Consumer
crates/buzz-agent/src/generated_model_capabilities.rs Rust interpreter
desktop/src/features/agents/ui/modelCapabilities.ts TS interpreter

The generator validates every interpolated manifest string through a shared requireSafeString() boundary (rejects quotes, backslashes, control chars), enforces the manifest schema (41 negative tests in scripts/test-manifest-validator.mjs, covering effort canonicality, post-inheritance record consistency, integer match priorities, and normalized duplicate keys), and supports --check for byte-clean verification.

Production integration

Rust (buzz-agent): reasoning-effort normalization for OpenAI and legacy Databricks flows through normalize_effort_for_provider(), backed by the generated interpreter; anthropic_body() is provider-aware and Adaptive thinking clamps against the generated supported_efforts; Databricks v2 request routing reads the manifest's databricks_v2_wire_route instead of a hand-maintained segment classifier. The superseded effort/thinking hand tables and model-family classifiers in config.rs and llm.rs are deleted — the generated interpreter is the only authority.

Desktop: getProviderEffortConfig() resolves through the generated modelCapabilities.ts; canonicalizeProvider() (trim/lowercase plus the databricks-v2 and openai-compat aliases) is applied before every provider-scoped lookup, and the provider is threaded through all label/effort surfaces so a supplied provider can never fall through to unscoped registry lookups. The old hand-table effort path and its fixture are deleted.

Drift protection

The model-capabilities job in .github/workflows/ci.yml gates every PR:

  • regenerates both artifacts and fails on any byte difference, so the committed interpreters cannot drift from the manifest;
  • runs the shared 80-vector normative corpus (scripts/normative-corpus.json) against both interpreters — scripts/run-corpus.mjs for TS, generated Rust tests for Rust — every vector pins all six capability axes (label, thinking mode, effort set, default effort, wire route, normalization policy), and both runners reject sparse vectors;
  • runs the schema-negative validator suite and cargo test -p buzz-agent --lib (the only CI gate exercising the buzz-agent unit suite; just test-unit excludes that crate).

Model display labels

The manifest's databricks_v2 exact records carry a registry_label axis with models.dev-verbatim display names for the ~30 managed Databricks endpoints. The generator derives a flat DATABRICKS_MODEL_NAMES registry from those records and emits it into both artifacts (Rust static slice, TS Map), so the two languages cannot drift.

Display rule: a model's display name comes from an exact/curated record only — family rules carry no labels, so a family-matched ID with no curated record resolves registryLabel: null and displays its raw ID (or its upstream discovery name). Endpoint IDs like databricks-gpt-5-5 resolve to human-readable labels (GPT-5.5); unknown or custom workspace endpoints pass through as their raw ID unchanged, so a custom endpoint like databricks-team-2025-01 can never be given an invented name.

Rust label resolution

databricks_model_name(id) in crates/buzz-agent/src/catalog.rs reads the generated registry and is applied at every ModelEntry construction path — v1 and v2 discovery parsing, the authenticated-empty-catalog slate, and the configured-model fallback — so AgentModelInfo.name reaches the frontend already curated. Mirrors the existing openai_model_display_name precedent; no new IPC. DATABRICKS_V2_KNOWN_MODELS is likewise a re-export of the generated constant.

Frontend label resolution

resolveModelLabel(id, discoveredName?, provider?) in desktop/src/features/agents/lib/formatAgentModelLabel.ts owns the label contract with three-tier precedence: nonblank discovered/API name, then registry lookup by ID (provider-qualified exact record when a provider is supplied; unscoped registry only on the providerless path), then raw ID. Every model-label surface routes through it — ModelPicker, usePersonaModelDiscovery, agentCardModelLabel, AgentConfigFields, ManagedAgentRow, UserProfilePopover. formatAgentModelLabel() wraps the resolver so a null or empty ID renders Auto.

Tests

  • Shared corpus 80/80 in both interpreters (full six-axis expectations per vector); schema-negative suite 41/41; cargo test -p buzz-agent --lib covers the corpus harness and generated-interpreter tests as a CI merge gate.
  • Compact seam tests guard the live production boundaries: startup validation of pure-Anthropic effort values through Config::validate(), Anthropic-route effort normalization (none/minimal omitted from the wire), and the unsupported-effort clamp through anthropic_body() (a no-xhigh model record clamps xhigh to high).
  • Corpus vectors pin the label contract: provider-qualified curated IDs resolve their exact-record labels (including the family-masquerade defect shape, databricks-gpt-5-mini → GPT-5 Mini), and a family-matched ID with no curated record pins registry_label: null; provider-alias tests pin openai-compat/databricks-v2 canonicalization in the desktop surfaces.
  • usePersonaModelDiscovery.test.mjs pins discovery-row label precedence; Rust unit tests in catalog.rs cover known IDs, custom endpoints, and the known-models fallback slate.

Docs and build infra

desktop/src/features/agents/AGENTS.md documents label precedence and the regeneration workflow (edit scripts/model-capabilities.json, run node scripts/generate-model-capabilities.mjs); the generator and CI job emit the regeneration command on any stale-artifact failure. desktop/scripts/check-file-sizes.mjs gains a fileOverrides mechanism because the generated modelCapabilities.ts exceeds the 1000-line ceiling intended for hand-authored files.

npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
feat(catalog): replace Databricks model-label tokenizer with models.d…
144ef4d 
…ev registry lookup

The frontend heuristic in #3586 title-cased every word and converted adjacent
digit pairs to decimals for any string starting with 'databricks-'. This
invented false labels for custom workspace endpoints: 'databricks-team-2025-01'
→ 'Team 2025.01', 'databricks-finance-2025-01-30' → 'Finance 2025.01 30'.

Replace with a lookup-and-pass-through approach drawn from the same design
goose uses in production:

- Generator script (scripts/generate-databricks-model-names.py) fetches
  https://models.dev/api.json and emits a sorted Rust static slice of
  (id, name) pairs under crates/buzz-agent/src/databricks_model_names.rs.
  Refresh by rerunning the script and committing the diff.

- catalog.rs consults the table via databricks_model_name(id): known managed
  endpoints get curated names (e.g. 'databricks-gpt-5-5' → 'GPT-5.5'),
  unknown/custom endpoints return their raw ID unchanged. Applied to all
  ModelEntry construction paths: v1/v2 discovery, the empty-list fallback,
  and discovery_failure_fallback.

- Frontend: a matching TS registry (desktop/src/features/agents/lib/
  databricksModelNames.ts) covers persisted raw IDs on cards, rows, and
  popovers that render before discovery data is available. formatAgentModelLabel
  and formatDefaultModelLabel use the registry; no heuristic string mangling
  exists anywhere in the TS layer.

- AGENTS.md documents the three-tier label precedence: API/runtime name first,
  table-backed fallback second, raw ID last.

Supersedes #3586 (kennylopez-model-display-labels).

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner July 29, 2026 19:04
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 3 commits July 29, 2026 15:41
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(catalog): address pass-1 review blockers on Databricks model-labe…
3855850 
…l PR

- Generator emits both Rust and TypeScript registries in one invocation;
  both are formatted (cargo fmt, biome) so regenerate is byte-for-byte stable
- TS registry uses Map<string, string> — eliminates Object.prototype key hole
  where 'constructor', '__proto__', 'toString' resolved through Object.prototype
  instead of passing through as raw IDs
- Centralize three-tier resolver (resolveModelLabel) in formatAgentModelLabel.ts;
  apply to ModelPicker trigger + unsupported-switching state, usePersonaModelDiscovery
  default rows, and AgentConfigFields defaultModelLabel — all surfaces now show
  curated names where available instead of raw endpoint IDs
- lib.rs module ordering fixed; just fmt-check green
- Tests: prototype-key cases (constructor, __proto__, toString, hasOwnProperty),
  resolver precedence (discovered name wins, blank falls through), DATABRICKS_MODEL_NAMES
  Map type assertion, Rust/TS parity spot-check; 3794 TS tests pass, 0 fail
- AGENTS.md refresh instructions updated: generator emits both files, resolveModelLabel
  is the single frontend resolver entry point; curl --fail + --max-time 30, explicit
  JSON shape validation, key/value character validation added to generator

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(catalog): route all model rows through the shared label resolver
be7d66f 
Two discovered-row callsites still formatted labels inline, so a known
Databricks endpoint returned by discovery without a name rendered as its
raw ID instead of the curated registry name — the exact inconsistency the
shared resolver exists to prevent.

The generator also trusted the models.dev payload shape: a non-object
model value silently emitted `id -> id`, baking a wrong label into both
committed artifacts where no test could distinguish it from a genuine
pass-through. Validation now rejects that instead. Registry parity is
pinned across the whole table rather than five sampled rows, so a
half-committed regenerate cannot pass.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
chore: merge origin/main into databricks model-label registry
485b362 
Pre-push branch-skew guard blocked the push: main changed crates/buzz-agent/src/lib.rs and desktop/src/features/agents/AGENTS.md, both touched by this branch, so local checks had run on a tree CI would never test. Both auto-merged without conflict.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 changed the title feat(catalog): replace Databricks model-label tokenizer with models.dev registry lookup feat(catalog): resolve Databricks labels from models.dev Jul 30, 2026
@wpfleger96 npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7
feat(agent): Phase 1 — model-capability manifest, generator, and test…
4d47f48 
… oracle (#3821)

## What this does

Introduces the model-capability manifest infrastructure (Phase 1 of the
Model-Capability Manifest plan v4, Thufir-approved 9/9/9). No consumer
cutover — `config.rs`, `llm.rs`, `catalog.rs`, and `buzzAgentConfig.ts`
are unchanged. Phase 2 wires them.

**Single source of truth** replaces hand-mirrored metadata across four
files:

```
scripts/model-capabilities.json         → hand-curated manifest
scripts/generate-model-capabilities.mjs → emits Rust + TS artifacts
crates/buzz-agent/src/generated_model_capabilities.rs
desktop/src/features/agents/ui/modelCapabilities.ts
```

## Resolver contract (plan v4 §Resolver contract)

Total function `resolve(provider, raw_model_id) → CapabilityResult`.
Three ordered steps:

1. Provider-qualified raw exact lookup — key is `(provider,
raw_model_id)`, matched before any prefix stripping. A prefixed alias
never inherits an exact record.
2. Provider-scoped ordered family rules — on normalized
(prefix-stripped) alias, by `match_priority` desc.
3. Per-axis provider fallback — `blank` vs `concrete_unknown`, per
provider.

Result is complete — every axis populated, runtime consumers never
compose fields.

## Boundaries the manifest does NOT own

- Transport for pure OpenAI, legacy Databricks, OpenRouter:
`OpenAiApi`/`openai_request()` remain authoritative.
`databricks_v2_wire_route` is DBv2-only (all other providers emit
`not-applicable`).
- Final display labels: `resolveModelLabel()` three-tier precedence
unchanged. `registry_label` feeds only the static registry tier.
- `llm.rs` scope: only `databricks_v2_route_for_model` (Phase 2).

## Test oracle (three independent layers)

1. Generated full-table coverage —
`scripts/generated-model-capabilities-coverage.json`: every manifest
entry + provider fallbacks.
2. Hand-authored normative corpus — `scripts/normative-corpus.json` (44
vectors): Anthropic manual-budget/adaptive families, OpenAI gpt-5
adversarial boundary cases, DBv2 segment-routing collision tests, P2-A
resolver-contract vectors, P2-B blank/concrete-unknown per provider.
Runs against JS resolver (`run-corpus.mjs`) and mirrored in Rust
(`generated_model_capabilities_tests.rs`).
3. Schema-negative tests — `scripts/test-manifest-validator.mjs` (17
tests): every validator rule has a failing-input test.

## Reconciliation table

`scripts/MODELS_DEV_RECONCILIATION.md` — all models.dev divergences
dispositioned. `databricks-gpt-5-4-mini` and `databricks-gpt-5-4-nano`
adopt models.dev `[low,medium,high]` (family rule adds `none+xhigh` the
endpoint doesn't advertise).

## CI

`.github/workflows/model-capability-regen-diff.yml`: triggers on
manifest/generator/artifact changes; regenerates and fails if stale;
runs JS corpus + schema-negative tests.

## Acceptance criteria (plan v4 Phase 1)

- Byte-clean regen: `node scripts/generate-model-capabilities.mjs
--check` passes
- Rust compiles: `cargo check -p buzz-agent`
- TS typechecks: `pnpm tsc --noEmit --strict`
- 44/44 normative corpus vectors pass (JS interpreter)
- 41/41 Rust corpus tests pass
- 17/17 schema-negative tests pass (every validator rule)
- Reconciliation table complete with doc citations
- No consumer changes (config.rs, llm.rs, catalog.rs, buzzAgentConfig.ts
untouched)

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
github-advanced-security[bot]
github-advanced-security AI found potential problems Jul 31, 2026
View reviewed changes
Comment thread .github/workflows/model-capability-regen-diff.yml Fixed
@wpfleger96 wpfleger96 changed the title feat(catalog): resolve Databricks labels from models.dev feat(models): manifest-driven model capabilities with generated Rust/TS interpreters and curated Databricks labels Jul 31, 2026
@wpfleger96 wpfleger96 changed the title feat(models): manifest-driven model capabilities with generated Rust/TS interpreters and curated Databricks labels feat(models): manifest-driven model capabilities and curated labels Jul 31, 2026
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7
Merge remote-tracking branch 'origin/main' into duncan/databricks-mod…
847f2c5 
…el-label-registry-sync

* origin/main: (39 commits)
  feat(relay): accept kind:30621 multi-repo projects at ingest (#3171)
  fix(release): preserve main in desktop PR body (#3979)
  chore(release): release Buzz Desktop version 0.5.3 (#3972)
  fix(release): require exact-head approval for desktop tags (#3973)
  fix(release): make desktop tagging squash-safe (#3965)
  Revert "chore(release): release Buzz Desktop version 0.5.3" (#3960)
  docs(nips): add single-coordinate manual-unread override layer and verification model to NIP-RS (#2864)
  chore(release): release Buzz Desktop version 0.5.3
  fix(release): make immutable desktop release operable (#3943)
  feat(desktop): import local Pocket voices (#3259)
  fix(desktop): open profiles from avatars (#3751)
  refactor(voice): extract reusable Pocket primitives + Pocket voice settings (relands #2467 + #3208) (#3910)
  docs: add VISION_REMOTE_AGENTS.md (#3924)
  feat(desktop): auto-enable huddle transcription for agents (#3180)
  feat(agent): optional reply guard reminds a silent turn to publish (#3763)
  feat(desktop): upgrade Pocket TTS model (#3266)
  feat(desktop): delete a message by clearing its edit to empty (#3813)
  feat(relay): raise hosted community limit to five (#3829)
  feat(desktop): locally stored NIP-49 encrypted key backup (#2937)
  fix(catalog): update Amp tagline (#3806)
  ...

Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
@mikececco mikececco mentioned this pull request Aug 3, 2026
Open
@wpfleger96 npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw
feat(agent): Phase 2 — wire Rust and TS consumers to generated model-…
cc00060 
…capabilities module (#3958)

## What

Phase 2 consumer cutover targeting the
`duncan/databricks-model-label-registry` umbrella branch. Wires
`crates/**` and `desktop/**` consumers to the generated capability
module introduced in Phase 1 (#3821), while keeping old and new paths
both live for differential testing. Phase 3 removes the old paths.

## Commits (boundary-separated)

### feat(agent): Phase 2a — wire Rust consumers to generated capability
module (`crates/**`, `scripts/**`)

- `catalog.rs`: `DATABRICKS_V2_KNOWN_MODELS` re-exported from the
generated module — single source of truth.
- `llm.rs`: `databricks_v2_route_for_model` delegates to
`resolve_model_capabilities("databricks_v2", model)`. Old segment-based
classifier preserved as `#[cfg(test)] _old_*` for the differential
harness. New `databricks_v2_route_differential_old_vs_new` test confirms
100% agreement on all 20 route vectors.
- `config.rs`: new `effort_table_fixture_differential_old_vs_new` test
runs `resolve_model_capabilities` over the 36-entry
`effortTable.fixture.json` and asserts old/new agree modulo a doc-cited
allowlist (4 F1 corrections).
- `scripts/run-differential.mjs`: JS differential harness over
effortTable fixture + normative corpus + catalog-sample fixture. 85
checks, 0 unexpected divergences (5 allowlisted: 4 F1 corrections +
goose-opus-5 anthropic route correction).
- `scripts/MODELS_DEV_RECONCILIATION.md`: deferred MINOR from Phase 1 —
8 trailing-double-space line breaks replaced with `<br>`.

### feat(desktop): Phase 2b — cut TS consumers to generated
model-capabilities module (`desktop/**`)

- `buzzAgentConfig.ts`: adds
`getProviderEffortConfigFromManifest(provider, model?)` — thin wrapper
over `resolveModelCapabilities()` from `modelCapabilities.ts`. Maps
`supportedEfforts → validValues` and `defaultEffort → defaultValue`
(null preserved for manual-budget/Inherit). Old
`getProviderEffortConfig()` and all hand-tables stay live for the
differential harness; Phase 3 retires them.
- `formatAgentModelLabel.ts`: registry-label lookup re-pointed from
hand-maintained `databricksModelNames.ts` import to generated
`DATABRICKS_MODEL_NAMES` exported from `modelCapabilities.ts`. Same Map
shape, identical contents, behavior unchanged.

### fix(scripts): add ts-esm-loader and fix allowlist coverage in
run-differential (`scripts/**`)

- `scripts/ts-esm-loader.mjs`: minimal ESM custom loader that resolves
extensionless relative TS imports. Required because Phase 2b's
`buzzAgentConfig.ts` imports `modelCapabilities` without `.ts` extension
— which Node's `--experimental-strip-types` runner cannot resolve
without a hook.
- `scripts/run-differential.mjs`: shebang updated to self-bootstrap with
the loader; fixes the `totalAllowlisted` counter (was declared but never
incremented — always printed `0 allowlisted`). Replaced with per-axis
hit tracking: reports exercised slot count (`N/total`) in summary; fails
with `STALE_ALLOWLIST` if any declared entry fires zero divergences,
preventing stale entries from silently masking future regressions.

## Verification

- `cargo test -p buzz-agent --lib`: 426/426
- Corpus: 45/45 · schema-negative: 24/24 · `--check` byte-clean
- Differential: 85 checks, 0 unexpected divergences, 6/6 allowlist slots
exercised
- Desktop: 3847/3847 · typecheck clean · biome clean
- Mobile: 1019 pass, 1 skipped — same 5 flaky tests in
`mobile/test/features/channels/` that reproduce at the umbrella base;
zero mobile files in this branch range
- `git diff --check`: clean

## What Remains (Phase 3)

Remove old hand-maintained paths: `_old_*` functions in
`llm.rs`/`config.rs`, old `getProviderEffortConfig` tables in
`buzzAgentConfig.ts`, old `databricksModelNames.ts` import in
`formatAgentModelLabel.ts`, old `databricks_model_names.rs` module.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz>
Signed-off-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: npub1g8493u0xfsjrvflg4n08ezd7vec99mnwzlv0qgwpr9d7gvjwhuzqx59rhw <41ea58f1e64c243627e8acde7c89be667052ee6e17d8f021c1195be4324ebf04@buzz.block.builderlab.xyz>
github-advanced-security[bot]
github-advanced-security AI found potential problems Aug 3, 2026
View reviewed changes
Comment thread .github/workflows/model-capability-regen-diff.yml Fixed
@wpfleger96 wpfleger96 mentioned this pull request Aug 3, 2026
Merged
@wpfleger96
chore: sync umbrella with origin/main (Cargo.lock + ci.yml updates)
b0b5718 
* origin/main: (40 commits)
  fix(mobile): recover stale relay sessions (#4372)
  chore(release): release Buzz Desktop version 0.5.4 (#4562)
  test(mobile): assert follow boundary semantics (#4559)
  docs(release): align desktop handoff instructions (#3988)
  fix: report agent usage per provider round, not once per turn (#4545)
  fix(desktop): harden Windows installs against Defender block and orphaned Node (#4382)
  feat(desktop): improve channel template discovery (#4549)
  fix(desktop): save key backups to authorized path (#4022)
  Add channel activity hover menu (#3935)
  feat(desktop): show saved Run on settings when editing an agent (#4539)
  fix(desktop): disambiguate provider API key labels and annotate mint key (#4406)
  fix(desktop): make OpenAI key re-enterable after first save in card mint dialog (#4140)
  fix(config-bridge): add harness-definition env tier and fix equal-value model override (#3580)
  Polish mobile composer and messaging UI (#3918)
  ci(linux): enable mesh-llm feature in Linux release and canary builds (#4524)
  fix(desktop): stop the create-agent provider config probe from erasing keystrokes (#4411)
  fix(mobile): recover and pace live subscriptions (#3053)
  feat(acp): deliver system prompt via _meta.systemPrompt for claude-agent-acp (#4395)
  fix(security): bump nostr crates for RUSTSEC-2026-0225..0232 + default sprig image to published digest (#4392)
  fix(desktop): back/forward via keyboard chords, mouse X1/X2 buttons, and swipe gestures (#3778)
  ...

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the duncan/databricks-model-label-registry branch from cc00060 to 4d47f48 Compare August 3, 2026 20:04
@wpfleger96 npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7
refactor(models): Phase 3 — retire old hand tables and shrink manifes…
126647e 
…t apparatus (#4589)

## Summary

Retires the old hand-table authorities and transitional verification
scaffolding from the model-capability manifest arc. All production
routes now run exclusively through the generated interpreters introduced
in Phase 1 ([#3821](#3821)) and wired
in Phase 2 ([#3958](#3958)).

Stack: [#3821](#3821) →
[#3958](#3958) → this PR
Base: [#3603](#3603)

## What the manifest system is now

**Source of truth:** `scripts/model-capabilities.json`
**Generator:** `scripts/generate-model-capabilities.mjs` — emits Rust
and TS interpreters only (coverage JSON output removed)
**Generated interpreters:**
`crates/buzz-agent/src/generated_model_capabilities.rs` (+ normative
tests), `desktop/src/features/agents/ui/modelCapabilities.ts`
**Label registry:** `generate-databricks-model-names.py` →
`databricks_model_names.rs` / `databricksModelNames.ts`
**Permanent gates:** `scripts/normative-corpus.json` +
`scripts/run-corpus.mjs` (both-interpreter equivalence, 51 vectors),
`scripts/test-manifest-validator.mjs` (schema, 24 cases), regen-diff job
inside `ci.yml`
**One doc:** `scripts/MODEL_CAPABILITIES.md`

## Deleted

**Old hand-table authorities (production):**
- `getProviderEffortConfig_oldHandTable()` and all supporting helpers
from `desktop/src/features/agents/ui/buzzAgentConfig.ts`
- `normalize_effort_for_openai_route()`,
`_old_anthropic_thinking_config_for_databricks_v2()`, test-only
re-export wrappers from `crates/buzz-agent/src/config.rs`
- `strip_catalog_prefix()`, `anthropic_thinking_config()`,
`anthropic_model_supports_xhigh()`, `clamp_adaptive_effort()`,
`anthropic_efforts_for_model()`, `is_manual_budget_model()`,
`is_adaptive_thinking_model()`, `gpt5_token_matches()`,
`gpt5_base_matches()`, `openai_efforts_for_model()` from
`crates/buzz-agent/src/config.rs` — all superseded by generated
interpreter
- Old DBv2 body-level tests, `_OLD_DATABRICKS_V2_*` constants,
`model_name_segments()`, `_old_databricks_v2_route_for_model()`, all
Phase-2 behavioral differential test functions from
`crates/buzz-agent/src/llm.rs`

**Transitional scaffolding:**
- `scripts/run-differential.mjs` — old-vs-new JS differential harness
- `scripts/run-mutation-evidence.mjs` — one-time mutation evidence
runner
- `desktop/src/features/agents/ui/effortTable.fixture.json` — Phase-2
TS/Rust sync fixture
- `desktop/src/features/agents/ui/effortTable.fixture.test.mjs` —
fixture sync guard
- `.github/workflows/model-capability-regen-diff.yml` — standalone
workflow (steps folded into `ci.yml`)

**One-time evidence and generated snapshots:**
- `scripts/MUTATION_EVIDENCE.md`,
`scripts/MODEL_CAPABILITIES_SCHEMA.md`,
`scripts/MODELS_DEV_RECONCILIATION.md` — consolidated into
`scripts/MODEL_CAPABILITIES.md`
- `scripts/generated-model-capabilities-coverage.json` — full-table
snapshot (generator no longer emits it)
- `scripts/catalog-sample-fixture.json` — models.dev snapshot used only
by the deleted differential harness

## Verification

- `cargo test -p buzz-agent --lib` with `RUSTFLAGS="-D warnings"`:
**337/337** (clean — no dead_code warnings)
- `node --experimental-strip-types scripts/run-corpus.mjs`: **51/51**
- `node scripts/generate-model-capabilities.mjs` + regen diff: **clean
(exit 0)**
- `node --test scripts/test-manifest-validator.mjs`: **24/24**
- `just clippy`: zero warnings, zero errors

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
wpfleger96 added a commit that referenced this pull request Aug 3, 2026
@wpfleger96 npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7
refactor(models): Phase 3 — retire old hand tables and shrink manifes…
899f55c 
…t apparatus (#4589)

## Summary

Retires the old hand-table authorities and transitional verification
scaffolding from the model-capability manifest arc. All production
routes now run exclusively through the generated interpreters introduced
in Phase 1 ([#3821](#3821)) and wired
in Phase 2 ([#3958](#3958)).

Stack: [#3821](#3821) →
[#3958](#3958) → this PR
Base: [#3603](#3603)

## What the manifest system is now

**Source of truth:** `scripts/model-capabilities.json`
**Generator:** `scripts/generate-model-capabilities.mjs` — emits Rust
and TS interpreters only (coverage JSON output removed)
**Generated interpreters:**
`crates/buzz-agent/src/generated_model_capabilities.rs` (+ normative
tests), `desktop/src/features/agents/ui/modelCapabilities.ts`
**Label registry:** `generate-databricks-model-names.py` →
`databricks_model_names.rs` / `databricksModelNames.ts`
**Permanent gates:** `scripts/normative-corpus.json` +
`scripts/run-corpus.mjs` (both-interpreter equivalence, 51 vectors),
`scripts/test-manifest-validator.mjs` (schema, 24 cases), regen-diff job
inside `ci.yml`
**One doc:** `scripts/MODEL_CAPABILITIES.md`

## Deleted

**Old hand-table authorities (production):**
- `getProviderEffortConfig_oldHandTable()` and all supporting helpers
from `desktop/src/features/agents/ui/buzzAgentConfig.ts`
- `normalize_effort_for_openai_route()`,
`_old_anthropic_thinking_config_for_databricks_v2()`, test-only
re-export wrappers from `crates/buzz-agent/src/config.rs`
- `strip_catalog_prefix()`, `anthropic_thinking_config()`,
`anthropic_model_supports_xhigh()`, `clamp_adaptive_effort()`,
`anthropic_efforts_for_model()`, `is_manual_budget_model()`,
`is_adaptive_thinking_model()`, `gpt5_token_matches()`,
`gpt5_base_matches()`, `openai_efforts_for_model()` from
`crates/buzz-agent/src/config.rs` — all superseded by generated
interpreter
- Old DBv2 body-level tests, `_OLD_DATABRICKS_V2_*` constants,
`model_name_segments()`, `_old_databricks_v2_route_for_model()`, all
Phase-2 behavioral differential test functions from
`crates/buzz-agent/src/llm.rs`

**Transitional scaffolding:**
- `scripts/run-differential.mjs` — old-vs-new JS differential harness
- `scripts/run-mutation-evidence.mjs` — one-time mutation evidence
runner
- `desktop/src/features/agents/ui/effortTable.fixture.json` — Phase-2
TS/Rust sync fixture
- `desktop/src/features/agents/ui/effortTable.fixture.test.mjs` —
fixture sync guard
- `.github/workflows/model-capability-regen-diff.yml` — standalone
workflow (steps folded into `ci.yml`)

**One-time evidence and generated snapshots:**
- `scripts/MUTATION_EVIDENCE.md`,
`scripts/MODEL_CAPABILITIES_SCHEMA.md`,
`scripts/MODELS_DEV_RECONCILIATION.md` — consolidated into
`scripts/MODEL_CAPABILITIES.md`
- `scripts/generated-model-capabilities-coverage.json` — full-table
snapshot (generator no longer emits it)
- `scripts/catalog-sample-fixture.json` — models.dev snapshot used only
by the deleted differential harness

## Verification

- `cargo test -p buzz-agent --lib` with `RUSTFLAGS="-D warnings"`:
**337/337** (clean — no dead_code warnings)
- `node --experimental-strip-types scripts/run-corpus.mjs`: **51/51**
- `node scripts/generate-model-capabilities.mjs` + regen diff: **clean
(exit 0)**
- `node --test scripts/test-manifest-validator.mjs`: **24/24**
- `just clippy`: zero warnings, zero errors

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Co-authored-by: npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
docs(models): remove MODEL_CAPABILITIES.md reference doc
94ef5fe 
Delete scripts/MODEL_CAPABILITIES.md and update the layer-3 note in
generated_model_capabilities_tests.rs to be self-contained, dropping
the now-dangling file reference.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 mentioned this pull request Aug 3, 2026
Draft
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 5 commits August 4, 2026 17:15
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(models): address Kalvin-review: gpt-segment/left-boundary/luna-te…
dd6e3a9 
…rra/validator

CRITICAL: Fix Rust gpt5-base short-version guard to mirror TS regex semantics.
The old 4-char window check diverged from TS /^\d{1,3}(?:[^a-z\d]|$)/i for
inputs like gpt-5-10-preview (10 digits). Replaced with find(non-digit) approach
that exactly matches the TS digit-run semantics. Add left-boundary guards (start
or preceded by '-'/'.') to both gpt5_token_matches_rs and gpt5_base_matches_rs
to prevent sgpt-model-class false-positives.

IMPORTANT: Add exact_records validation to generator — enum checks for all
optional override axes, non-empty override check, canonical-order enforcement,
default_effort-in-override check, match_priority non-negative-int check
(injected into source comments, treated as injection surface). Add 9 schema-
negative tests in test-manifest-validator.mjs.

IMPORTANT: Restrict DBv2 gpt segment rule from starts_with("gpt") to exact
segment match ("gpt" or "gpt5"). Raise priority 5->6 to restore old dual-marker
OpenAI-before-Claude contract. Add left-boundary guards to token helpers. Add
collision-negative corpus vectors: gptoss-model, gptj-6b, gpt-neox,
customgpt-5-5-endpoint.

MINOR: Restore .trim() on model string before resolveModelCapabilities call in
buzzAgentConfig.ts. Make exact-record lookup case-insensitive (lowercase keys at
build + lookup). Extend run-corpus.mjs to compare all 6 axes including
normalization_policy and registry_label. Add positive corpus vectors: opus-5
rule, dbv2 gpt-segment, sol/luna/terra, gpt-5-4-nano exact record, openrouter/
unknown/legacy-databricks fallbacks, case-insensitive lookup vectors. Fix Rust
corpus test harness provider lowercasing to handle vectors like provider=OpenAI.

HYGIENE: Fix "Generated 3 files" -> "Generated 2 files" message. Remove dead
$schema pointer from manifest. Add module doc to generated_model_capabilities_tests.rs
noting hand-maintained status. Switch PROVIDER_ALIASES from plain object to Map
in formatAgentModelLabel.ts and run-corpus.mjs. Align CI node-version to 24.
Tighten python SAFE_NAME_RE from * to + to reject empty display names.

luna/terra: models.dev confirms databricks-gpt-5-6-luna and -terra advertise
[low,medium,high] (differs from sol's [low,medium,high,max]). Added two exact
records with supported_efforts=[low,medium,high], default_effort=medium.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
Merge remote-tracking branch 'origin/main' into duncan/model-manifest…
aa962f1 
…-phase2-integration

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(models): close round-2 gaps: 6-axis Rust harness + gpt-neox honesty
bca1d5d 
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(models): round-3 corrective pass — boundary strip, gpt-version-se…
b6162be 
…gment, 6-axis corpus, validation symmetry, prototype-leak

- stripCatalogPrefix boundary-aligned: customgpt/sgpt/mygpt no longer false-strip
  to gpt-* forms; boundary check requires position 0 or non-alphanumeric precursor
- New match_kind gpt-version-segment: gpt-neox-20b routes mlflow-chat (was residual
  collision); gpt segment match requires next segment to start with digit or be
  dashless gpt5 form
- PROVIDER_FALLBACKS Record→Map: constructor/__proto__ prototype-key leak closed;
  resolveModelCapabilities/getProviderEffortConfig return complete records for all
  provider strings
- Corpus 69→77 vectors, all 6-axis mandatory; JS and Rust runners hard-fail on missing
  axes; prototype-key vectors added; boundary-negative vectors added (sgpt-5-5,
  mygpt-5, customgpt-5-5-endpoint all route mlflow-chat); gpt-neox-20b pinned as
  mlflow-chat (corrected behavior, not residual collision)
- Manifest validator 33→42 tests: family match_priority integer guard, lowercase
  duplicate-key detection, post-inheritance materialized default∈supported check,
  assertEfforts shared helper covering family/fallback/exact
- Clippy: map_or(false)→is_some_and in emitter template; fmt clean
- Desktop regression tests: constructor/__proto__ prototype-key stability

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
style(models): biome format buzzAgentConfig.test.mjs prototype-key re…
7862cd3 
…gression tests

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the duncan/databricks-model-label-registry branch from c1f7517 to 8016fe9 Compare August 4, 2026 19:36
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 2 commits August 4, 2026 19:52
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(models): type DEFAULT_FALLBACK to satisfy CapabilityResult (TS2322)
46f0d44 
Emitter generated `DEFAULT_FALLBACK` as an untyped object literal; its
`as const` effort arrays and widened `defaultEffort` union didn't satisfy
`CapabilityResult`, failing tsc with TS2322 at line 861. Add explicit type
annotation matching the PROVIDER_FALLBACKS Map value type so the `??`
fallback path type-checks. Regen TS artifact only (Rust unaffected).

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(test): correct stale gpt5-4 family default comment (none → medium)
15aa135 
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the duncan/databricks-model-label-registry branch from 0189510 to 15aa135 Compare August 4, 2026 21:16
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 6 commits August 4, 2026 18:11
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
feat(manifest): consolidate label authority — fold registry_labels in…
0c02ca7 
…to exact records

Implements Thufir pass-1 finding 5 and Will's display ruling (a): a model's
display name comes from an exact/curated record only; family labels never
masquerade as model names.

Changes:
- Fold all 30 registry_labels entries into databricks_v2 exact records;
  reconcile GPT-5.4 mini/nano casing to models.dev verbatim (lowercase).
- Remove registry_label from all 16 family rules — family-matched models
  resolve registryLabel: null (display falls to raw ID or discovery name).
- Emit DATABRICKS_MODEL_NAMES from exact records (provider=databricks_v2 +
  registry_label), replacing the registryLabelsArr source.
- Point catalog.rs at generated_model_capabilities::DATABRICKS_MODEL_NAMES.
- Delete python chain: generate-databricks-model-names.py,
  databricks_model_names.rs (+ lib.rs decl), databricksModelNames.ts,
  databricksModelNames.test.mjs.
- Update desktop/src/features/agents/AGENTS.md: manifest is the label
  authority; document display rule (a) and single-chain refresh.
- Corpus: 80 vectors (3 added for gpt-5-mini/nano exact-label pins and one
  family-matched-no-exact → null); 37 existing vectors updated to null for
  family-matched labels; 3 casing vectors updated to lowercase.
- Validator: replace 3 stale registry_labels schema-negative tests with
  2 exact_record tests (duplicate key rejection, empty label rejection).
- File-size ratchet: add fileOverrides parameter to runFileSizeCheck so the
  generated modelCapabilities.ts can have a tighter per-file ceiling (1200)
  rather than the hand-authored 1000-line default.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
chore: merge origin/main — resolve catalog.rs conflict (auth semantic…
f24e30c 
…s + curated labels)

Main's #4008 reauthentication loop (discover_databricks_models_with_token_source),
authenticated_empty_v2_catalog(), configured_model_fallback(), and removal of
discovery_failure_fallback() all land. Our labeling wins at every ModelEntry
construction site: databricks_model_name() applied in parse_v1_endpoints,
parse_v2_endpoints_page, and authenticated_empty_v2_catalog (curated label +
suffix). DATABRICKS_V2_KNOWN_MODELS stays as the generated-constant re-export.

lib.rs auto-merged cleanly: resolve_models_catalog now returns Result (main's
shape); discovery_failure_fallback callers replaced by configured_model_fallback.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
chore(manifest): remove dead registry_labels key
77c24ef 
The key was an empty array with zero references in the generator or
validator after the consolidation round folded all labels into exact
records. Regen is byte-clean before and after the deletion.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(buzz-agent): curate display name in configured_model_fallback
0e6aacf 
Route the display name through databricks_model_name() so that when
Databricks discovery fails the picker shows the curated label (e.g.
GPT-5.5) rather than the raw model ID — consistent with every other
ModelEntry construction site in catalog.rs.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
fix(manifest): enforce registry_label exclusion on family rules; test…
1e9df55 
… fileOverrides boundary

Rule (a) is now structurally unexpressible: the generator rejects any
family rule carrying registry_label (exit non-zero), removes the dead
reads from resolveFamilyRules() and both emitters, and adds a
schema-negative test to catch reintroduction.

The fileOverrides integration test is replaced with a real temporary-
repository runFileSizeCheck() call: proves the named path passes under
its raised ceiling, proves a second governed path at the same size still
fails (no leak), and restores process.exitCode and console.error in
cleanup for test isolation.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 @wpfleger96
test(file-size): fix fileOverrides integration test under GITHUB_ACTIONS
817b887 
Add a second empty commit to the temp repo before staging test files so
HEAD^1 resolves in CI (where resolveBaseRef returns HEAD^1 as the base).
Remove now-unused spawnSync import.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@wpfleger96 @github-advanced-security