An empty AI completion was returned as a successful answer - #147
Merged
Conversation
`completeWithOpenAICompat` ended with `content ?? ''`, so a response carrying
`content: null` came back as the empty string — a SUCCESS. Three things
followed from that, none of them visible:
- `withProviderFallback` called `recordAIHealthSuccess()`, so /api/health
reported the AI chain `ok` on a request that produced nothing
- the chain never fell through, so the second provider was never asked —
the fallback existed and did not engage on the one failure it could not see
- the caller got '' and had to invent its own meaning for it
The trigger is ordinary, not exotic: a reasoning model spends the whole
`max_tokens` budget on reasoning tokens and stops with
`finish_reason: "length"` before emitting any content. Observed on an
OpenRouter auto-routed model at a small budget. The app's real budgets are far
from that today — but which model sits behind a provider is not ours to pin
forever, and this repo has already been bitten by model ids rotating
underneath it.
An empty completion is now an AIProviderError with status 502, chosen so
`shouldTryNextProvider` falls through on `>= 500`: an unusable response from
one vendor should be retried at the next, which is the whole point of having a
chain. `finish_reason` rides on the error body so the log says WHY it was
empty, following the existing rule that the body is for logs and never for a
browser.
This is the shape the codebase keeps meeting — absence read as an answer. The
health signal saying `ok` while returning nothing is the same failure as a
green gate that tests nothing.
Mutation-proven: deleting the guard fails six tests.
Verified with `SESSION_SECRET=… npm run build` (exit 0), the gate npm run
verify does not cover.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cd183M6472xBgTKWA2is6h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
completeWithOpenAICompatended with:So a response carrying
content: nullcame back as the empty string — a success. Three things followed, none of them visible:withProviderFallbackcalledrecordAIHealthSuccess(), so/api/healthreported the AI chainokon a request that produced nothing.''and had to invent its own meaning for it.How it happens
Not exotic: a reasoning model spends the entire
max_tokensbudget on reasoning tokens and stops withfinish_reason: "length"before emitting any content. Observed on an OpenRouter auto-routed model at a small budget.The app's real budgets (2000 for form fill) are far from that today — but which model sits behind a provider is not ours to pin forever, and this repo has already been bitten by model ids rotating underneath it.
Fix
An empty completion is now an
AIProviderErrorwith status 502, chosen deliberately:shouldTryNextProviderfalls through on>= 500, so an unusable response from one vendor gets retried at the next — which is the entire point of having a chain.finish_reasonrides on the error body so the log says why it was empty, following the existing rule that the body is for logs and never for a browser.Why it matters beyond this line
This is the shape this codebase keeps meeting: absence read as an answer. A health signal reporting
okwhile returning nothing is the same failure as a green gate that tests nothing — the instrument agrees with you and measures the wrong thing.Mutation-proven: deleting the guard fails six tests.
npm run verify200 suites / 3524 passed, and verified with a realnext build(exit 0) — the gateverifyexplicitly does not cover, now runnable per #146.🤖 Generated with Claude Code
https://claude.ai/code/session_01Cd183M6472xBgTKWA2is6h