Repository navigation
feat(delegate): dispatch the vision call on model.api (openai-responses, google-generative-ai, anthropic-messages) - #27
Open
quantmind-br wants to merge 5 commits into
Conversation
callVisionModel always POSTed {baseUrl}/chat/completions, whatever the
model's api. A model registered as google-generative-ai or
anthropic-messages therefore hit a route that does not exist on its
gateway and failed with an opaque 404 (vision_call_error).
Move the per-API request/response shapes into lib/wire.ts and dispatch
on model.api:
- openai-completions: unchanged request
- google-generative-ai: {baseUrl}/models/{id}:generateContent, x-goog-api-key,
inlineData, thinkingConfig.thinkingLevel (honours thinkingLevelMap)
- anthropic-messages: {baseUrl}/v1/messages, x-api-key, base64 image block,
extended-thinking budget; temperature is never sent
Any other api is rejected before the network, and before the image is read,
with an actionable unsupported_api error (primary and fallback). An empty
response now reports the block/finish reason.
autoDetectDefaults picked the first vision-capable model of any provider, sorted by name, so a provider such as an anthropic-messages or openai-responses one could win and fail on the first image. Only consider models whose api the vision call can speak.
…ones The /vision model picker listed every image-capable model. Restrict it to models on a supported api, and warn when /vision model or the settings panel sets a registered model whose api the vision call cannot speak.
A model registered as openai-responses was rejected with unsupported_api,
so a Responses-only gateway could not be the vision model. Add the
Responses shape to lib/wire.ts next to the other three APIs:
- POST {baseUrl}/responses with input_image / input_text parts
- system prompt goes in `instructions`
- reasoning is the nested `reasoning.effort` (only for reasoning models,
and never when the level is off)
- `temperature` is never sent: reasoning models reject it
- response text is the output_text parts of message items; reasoning
items are skipped, and an incomplete response reports its reason
Auto-detect and the /vision model picker now include openai-responses
models. README documents the new row.
quantmind-br
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
callVisionModelalways POSTs{baseUrl}/chat/completions, whatever the model's registeredapi. A vision model registered asgoogle-generative-ai,anthropic-messagesoropenai-responsestherefore hits a route that does not exist on its gateway and fails with an opaque 404 (vision_call_error).autoDetectDefaults()and the/vision modelpicker made it worse: they offer every image-capable model, so a model on an API the call cannot speak can be auto-picked and only fails on the first image.Fix
The per-API request/response shapes live in a new
lib/wire.ts, andcallVisionModeldispatches onmodel.api:apiopenai-completionsPOST {baseUrl}/chat/completions, unchangedAuthorization: Bearerreasoning_effortopenai-responsesPOST {baseUrl}/responses,input_image/input_textparts, system prompt asinstructionsAuthorization: Bearerreasoning.effort;temperaturenever sentgoogle-generative-aiPOST {baseUrl}/models/{id}:generateContent,inlineDatax-goog-api-keythinkingConfig.thinkingLevel, honoursthinkingLevelMapanthropic-messagesPOST {baseUrl}/v1/messages, base64imageblock, top-levelsystemx-api-key+anthropic-versiontemperaturenever sentoutput_textparts ofmessageitems.reasoningitems are skipped. An incomplete response reportsincomplete_details.reason(e.g.max_output_tokens).google-vertex,bedrock-converse-stream,openai-codex-responses, …) are rejected before the network call and before the image is read, with an actionableunsupported_apierror. This applies to both the primary and the fallback model.autoDetectDefaults()and/vision modelonly consider models on a supported API. The settings panel and/vision modelwarn when the configured model uses an unsupported one.Relation to open PRs
lib/wire.ts. feat: Add openai-responses-api support and filter out unsupported models. #22 predateslib/wire.tsand conflicts with it, so it should be closed in favour of this PR.Testing
pnpm typecheck: cleanpnpm test:run: 385/385 pass (+5 vs. the previous commit)tests/wire.test.ts: Responses request URL, headers,input/instructions/reasoningshape, notemperature; Responses text extraction incl. skipped reasoning/refusal parts and incomplete reasontests/delegate.test.ts: end-to-endopenai-responsesprimary →/responses; unsupported-API tests now usebedrock-converse-streamtests/defaults.test.ts:openai-responsesmodels stay eligible for auto-detectopenai-responsesmodelgpt-5.6-luna,reasoning: low, with a system prompt: HTTP 200, description returned throughcallVisionModel.