fix(generate): provenance names the model that ran, not the slot's - #181
Merged
Conversation
Found live, tonight, doing the pre-tag checks on the shipped binary: a `generate` with `op: "flux-dev"` wrote `"model": "bfl/flux-2-pro"` into the artifact's sidecar. The op resolved correctly and the call really did go to `/v1/flux-dev` (`bfl.rs:191`); only the record was wrong. So the one file an operator reads months later to decide whether an artifact can be trusted or reproduced named a model that never ran — and at BFL's per-request pricing, a different cost. The rule was already written down, and already enforced one door over. `generate` refuses `fields.model` because it "would reroute the call while the sidecar records the slot's model" — recorded provenance must describe the request that ran. `op` reroutes exactly the same way and walked straight past that guard. Fixed where the asymmetry actually lives rather than special-casing BFL in the handler. `MediaModel::effective_model` defaults to `None`, meaning "the slot already names it" — true for every provider whose ops are routes over one model, which is Stability's `edit/inpaint` and the Images API's `edits`. BFL overrides it, because its ops *are* model ids. Same fail-closed shape as `accepts_inputs` and `accepts_ops` beside it: a provider that reroutes and stays silent gets the safe answer, and the cost of forgetting is a slot ref rather than a wrong model. `MediaArm::recorded_model` recomposes the id onto the slot's backend so the record keeps the `backend/model` shape, splitting on the first `/` only — a model id can contain more (`crusoe/deepseek-ai/Deepseek-V4-Flash`). All five places the handler names a model now name the one that ran: the sidecar, the footer the caller reads, the span (cost attribution), and the deferred lane's job label and handle message. Two tests, each with its teeth checked. `bfl::tests::the_recorded_model_is_the_op_that_ran_not_the_slot` pins the resolution and carries a control arm for the no-`op` case, so an impl that always answered "flux-dev" would fail it; drop the BFL override and it fails. `server::tests::generate_records_the_model_an_op_rerouted_to_not_the_slot` pins the wiring through to what an operator reads; revert the handler to `slot_ref` and it fails. No changelog entry: BFL and `op` are both new in this same unreleased section, so this bug has never been in a release and the feature bullet describes the behavior that ships. Not fixed here, and deliberately: a Stability `op` names a route, not a model, so the slot ref stays the honest answer there. Recording the route as well would be a real improvement to the sidecar, but it is an added field rather than a correction, and it is not what tonight found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found live tonight, running the pre-tag checks against the shipped binary. A
generatewith
op: "flux-dev"wrote this beside the artifact:{ "prompt": "…", "model": "bfl/flux-2-pro", "cast": "flux", "tool": "generate" }The op resolved correctly and the call really did go to
/v1/flux-dev(
bfl.rs:191—request.op.as_deref().unwrap_or(slot_model), with tests alreadycovering it). Only the record was wrong. So the one file an operator reads months
later, to decide whether an artifact can be trusted or reproduced, named a model that
never ran — and at BFL's per-request pricing, a different cost.
The rule was already written down, and enforced one door over
generaterefusesfields.modelwith this reasoning (server/mod.rs:3173-3177):opreroutes exactly the same way and walked straight past that guard. This closes thehole
opopened, with the same principle the existing guard states.Fixed where the asymmetry lives
Not by special-casing BFL in the handler.
MediaModel::effective_modeldefaults toNone— "the cast'simageslot already names it" — which is true for every providerwhose ops are routes over one model: Stability's
edit/inpaint, the Images API'sedits. BFL overrides it, because its ops are model ids (flux-dev,flux-2-pro).That is the same fail-closed shape as
accepts_inputsandaccepts_opssitting besideit: a provider added later that reroutes and forgets to say so gets the safe answer — a
slot ref — rather than a confidently wrong model.
MediaArm::recorded_modelrecomposes onto the slot's backend so the record keeps itsbackend/modelshape, splitting on the first/only, since a model id can containmore (
crusoe/deepseek-ai/Deepseek-V4-Flash).All five places the handler names a model now name the one that ran: the sidecar, the
footer the caller reads, the span (cost attribution), and the deferred lane's job label
and handle message.
Two tests, teeth checked on both
bfl::…::the_recorded_model_is_the_op_that_ran_not_the_slotserver::…::generate_records_the_model_an_op_rerouted_to_not_the_slotslot_refThe first carries a control arm for the no-
opcase, so an impl that always answered"flux-dev"would fail it. Both teeth were run, not assumed.Suite 1331 passed / 0 failed, clippy clean.
Scope notes
No changelog entry. BFL and
opare both new in this same unreleased section, so thisbug has never been in a release — the feature bullet describes the behavior that ships.
Not fixed here, deliberately: a Stability
opnames a route, not a model, so theslot ref stays the honest answer there. Recording the route as well would genuinely
improve the sidecar, but that is an added field rather than a correction, and it is not
what tonight found.
Also noticed, not addressed: the CLI's
generatehas no--opflag, so this path isreachable only over MCP. Worth its own look, separately.
🤖 Generated with Claude Code