Skip to content

fix(generate): provenance names the model that ran, not the slot's - #181

Merged
tobert merged 1 commit into
mainfrom
fix/generate-op-provenance
Sep 3, 2026
Merged

fix(generate): provenance names the model that ran, not the slot's#181
tobert merged 1 commit into
mainfrom
fix/generate-op-provenance

Conversation

@tobert

@tobert tobert commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Found live tonight, running the pre-tag checks against the shipped binary. A generate
with op: "flux-dev" wrote this beside the artifact:

{ "prompt": "", "model": "bfl/flux-2-pro", "cast": "flux", "tool": "generate" }

The op resolved correctly and the call really did go to /v1/flux-dev
(bfl.rs:191request.op.as_deref().unwrap_or(slot_model), with tests already
covering it). Only the record was wrong. So the one file an operator reads months
later, to decide whether an artifact can be trusted or reproduced, named a model that
never ran — and at BFL's per-request pricing, a different cost.

The rule was already written down, and enforced one door over

generate refuses fields.model with this reasoning (server/mod.rs:3173-3177):

Reserved keys: recorded provenance must describe the request that actually ran. […]
fields.model would reroute the call while the sidecar records the slot's model — so
both are refused loudly.

op reroutes exactly the same way and walked straight past that guard. This closes the
hole op opened, with the same principle the existing guard states.

Fixed where the asymmetry lives

Not by special-casing BFL in the handler. MediaModel::effective_model defaults to
None"the cast's image slot already names it" — which is true for every provider
whose ops are routes over one model: Stability's edit/inpaint, the Images API's
edits. BFL overrides it, because its ops are model ids (flux-dev, flux-2-pro).

That is the same fail-closed shape as accepts_inputs and accepts_ops sitting beside
it: a provider added later that reroutes and forgets to say so gets the safe answer — a
slot ref — rather than a confidently wrong model.

MediaArm::recorded_model recomposes onto the slot's backend so the record keeps its
backend/model shape, splitting on the first / only, since a model id can contain
more (crusoe/deepseek-ai/Deepseek-V4-Flash).

All five places the handler names a model now name the one that ran: the sidecar, the
footer the caller reads, the span (cost attribution), and the deferred lane's job label
and handle message.

Two tests, teeth checked on both

test proves fails when
bfl::…::the_recorded_model_is_the_op_that_ran_not_the_slot BFL resolves the op as the model the override is dropped
server::…::generate_records_the_model_an_op_rerouted_to_not_the_slot the handler carries it to what an operator reads the handler reverts to slot_ref

The first carries a control arm for the no-op case, so an impl that always answered
"flux-dev" would fail it. Both teeth were run, not assumed.

Suite 1331 passed / 0 failed, clippy clean.

Scope notes

No changelog entry. BFL and op are both new in this same unreleased section, so this
bug has never been in a release — the feature bullet describes the behavior that ships.

Not fixed here, deliberately: a Stability op names a route, not a model, so the
slot ref stays the honest answer there. Recording the route as well would genuinely
improve the sidecar, but that is an added field rather than a correction, and it is not
what tonight found.

Also noticed, not addressed: the CLI's generate has no --op flag, so this path is
reachable only over MCP. Worth its own look, separately.

🤖 Generated with Claude Code

Found live, tonight, doing the pre-tag checks on the shipped binary: a
`generate` with `op: "flux-dev"` wrote `"model": "bfl/flux-2-pro"` into the
artifact's sidecar. The op resolved correctly and the call really did go to
`/v1/flux-dev` (`bfl.rs:191`); only the record was wrong. So the one file an
operator reads months later to decide whether an artifact can be trusted or
reproduced named a model that never ran — and at BFL's per-request pricing, a
different cost.

The rule was already written down, and already enforced one door over.
`generate` refuses `fields.model` because it "would reroute the call while the
sidecar records the slot's model" — recorded provenance must describe the request
that ran. `op` reroutes exactly the same way and walked straight past that guard.

Fixed where the asymmetry actually lives rather than special-casing BFL in the
handler. `MediaModel::effective_model` defaults to `None`, meaning "the slot
already names it" — true for every provider whose ops are routes over one model,
which is Stability's `edit/inpaint` and the Images API's `edits`. BFL overrides
it, because its ops *are* model ids. Same fail-closed shape as `accepts_inputs`
and `accepts_ops` beside it: a provider that reroutes and stays silent gets the
safe answer, and the cost of forgetting is a slot ref rather than a wrong model.

`MediaArm::recorded_model` recomposes the id onto the slot's backend so the
record keeps the `backend/model` shape, splitting on the first `/` only — a model
id can contain more (`crusoe/deepseek-ai/Deepseek-V4-Flash`).

All five places the handler names a model now name the one that ran: the sidecar,
the footer the caller reads, the span (cost attribution), and the deferred lane's
job label and handle message.

Two tests, each with its teeth checked. `bfl::tests::the_recorded_model_is_the_op_that_ran_not_the_slot`
pins the resolution and carries a control arm for the no-`op` case, so an impl
that always answered "flux-dev" would fail it; drop the BFL override and it
fails. `server::tests::generate_records_the_model_an_op_rerouted_to_not_the_slot`
pins the wiring through to what an operator reads; revert the handler to
`slot_ref` and it fails.

No changelog entry: BFL and `op` are both new in this same unreleased section, so
this bug has never been in a release and the feature bullet describes the
behavior that ships.

Not fixed here, and deliberately: a Stability `op` names a route, not a model, so
the slot ref stays the honest answer there. Recording the route as well would be
a real improvement to the sidecar, but it is an added field rather than a
correction, and it is not what tonight found.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tobert
tobert merged commit 925e595 into main Sep 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant