Context
engine/filetools.go's read_file tool now returns a message.Blob for an
image file (PR adding read_file image support). Every transcoder already
maps an image/* Blob to a real wire image block when the target model
supports it. A model with no vision capability instead rejects the
request or silently mishandles the block.
What is missing
There is no per-model vision-capability signal anywhere in the codebase
today:
- No embedded models.dev catalog snapshot exists (AGENTS.md's
"Architecture" section describes one as a design goal, but no such
package or embedded data currently exists — confirmed by a repo-wide
search for "models.dev", "modelsdev", "ModelInfo", "Capabilit*", and
"Vision" prior to this PR).
- provider.Request carries no capability flag a caller could set from
such a catalog (Effort and SessionKey are the only per-request
provider-behavior fields, and both document that the engine cannot
know per-model support from the ref alone — see provider/provider.go's
Request.Effort doc comment).
Proposed follow-up
- Build (or embed) a models.dev catalog snapshot with a vision/modality
flag per model, exposed to the engine.
- Add a provider.Request-level flag (mirroring how Effort/SessionKey
already flow from the engine to each adapter) that the engine sets
from that catalog per request.
- At transcode time, each adapter (or a shared pre-transcode pass,
following imageclamp's shape) replaces an image Blob with a Text
placeholder ("[image omitted: model has no vision capability]") when
the flag says the target model has no vision support. This MUST be
transcode-only (throwaway request), never a mutation of stored
history — see AGENTS.md's live-vs-transcode invariant.
Why deferred, not built now
Forcing this without a real capability source would mean inventing an ad
hoc, likely-wrong static model list inside this PR, which does not meet
the same evidence bar as imageclamp's provider-documented caps. Filed
per this PR's own instructions rather than forcing an ugly seam.
Context
engine/filetools.go's read_file tool now returns a message.Blob for an
image file (PR adding read_file image support). Every transcoder already
maps an image/* Blob to a real wire image block when the target model
supports it. A model with no vision capability instead rejects the
request or silently mishandles the block.
What is missing
There is no per-model vision-capability signal anywhere in the codebase
today:
"Architecture" section describes one as a design goal, but no such
package or embedded data currently exists — confirmed by a repo-wide
search for "models.dev", "modelsdev", "ModelInfo", "Capabilit*", and
"Vision" prior to this PR).
such a catalog (Effort and SessionKey are the only per-request
provider-behavior fields, and both document that the engine cannot
know per-model support from the ref alone — see provider/provider.go's
Request.Effort doc comment).
Proposed follow-up
flag per model, exposed to the engine.
already flow from the engine to each adapter) that the engine sets
from that catalog per request.
following imageclamp's shape) replaces an image Blob with a Text
placeholder ("[image omitted: model has no vision capability]") when
the flag says the target model has no vision support. This MUST be
transcode-only (throwaway request), never a mutation of stored
history — see AGENTS.md's live-vs-transcode invariant.
Why deferred, not built now
Forcing this without a real capability source would mean inventing an ad
hoc, likely-wrong static model list inside this PR, which does not meet
the same evidence bar as imageclamp's provider-documented caps. Filed
per this PR's own instructions rather than forcing an ugly seam.