Skip to content

An empty AI completion was returned as a successful answer - #147

Merged
github-actions[bot] merged 1 commit into
masterfrom
fix/empty-ai-completion
Sep 1, 2026
Merged

An empty AI completion was returned as a successful answer#147
github-actions[bot] merged 1 commit into
masterfrom
fix/empty-ai-completion

Conversation

@catomean

@catomean catomean commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

completeWithOpenAICompat ended with:

return body.choices?.[0]?.message?.content ?? ''

So a response carrying content: null came back as the empty string — a success. Three things followed, none of them visible:

  • withProviderFallback called recordAIHealthSuccess(), so /api/health reported the AI chain ok on a request that produced nothing.
  • The chain never fell through, so the second provider was never asked. The fallback existed and did not engage on the one failure it couldn't see.
  • The caller received '' and had to invent its own meaning for it.

How it happens

Not exotic: a reasoning model spends the entire max_tokens budget on reasoning tokens and stops with finish_reason: "length" before emitting any content. Observed on an OpenRouter auto-routed model at a small budget.

The app's real budgets (2000 for form fill) are far from that today — but which model sits behind a provider is not ours to pin forever, and this repo has already been bitten by model ids rotating underneath it.

Fix

An empty completion is now an AIProviderError with status 502, chosen deliberately: shouldTryNextProvider falls through on >= 500, so an unusable response from one vendor gets retried at the next — which is the entire point of having a chain.

finish_reason rides on the error body so the log says why it was empty, following the existing rule that the body is for logs and never for a browser.

Why it matters beyond this line

This is the shape this codebase keeps meeting: absence read as an answer. A health signal reporting ok while returning nothing is the same failure as a green gate that tests nothing — the instrument agrees with you and measures the wrong thing.


Mutation-proven: deleting the guard fails six tests. npm run verify 200 suites / 3524 passed, and verified with a real next build (exit 0) — the gate verify explicitly does not cover, now runnable per #146.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cd183M6472xBgTKWA2is6h

`completeWithOpenAICompat` ended with `content ?? ''`, so a response carrying
`content: null` came back as the empty string — a SUCCESS. Three things
followed from that, none of them visible:

  - `withProviderFallback` called `recordAIHealthSuccess()`, so /api/health
    reported the AI chain `ok` on a request that produced nothing
  - the chain never fell through, so the second provider was never asked —
    the fallback existed and did not engage on the one failure it could not see
  - the caller got '' and had to invent its own meaning for it

The trigger is ordinary, not exotic: a reasoning model spends the whole
`max_tokens` budget on reasoning tokens and stops with
`finish_reason: "length"` before emitting any content. Observed on an
OpenRouter auto-routed model at a small budget. The app's real budgets are far
from that today — but which model sits behind a provider is not ours to pin
forever, and this repo has already been bitten by model ids rotating
underneath it.

An empty completion is now an AIProviderError with status 502, chosen so
`shouldTryNextProvider` falls through on `>= 500`: an unusable response from
one vendor should be retried at the next, which is the whole point of having a
chain. `finish_reason` rides on the error body so the log says WHY it was
empty, following the existing rule that the body is for logs and never for a
browser.

This is the shape the codebase keeps meeting — absence read as an answer. The
health signal saying `ok` while returning nothing is the same failure as a
green gate that tests nothing.

Mutation-proven: deleting the guard fails six tests.
Verified with `SESSION_SECRET=… npm run build` (exit 0), the gate npm run
verify does not cover.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cd183M6472xBgTKWA2is6h
@github-actions
github-actions Bot merged commit fbeeace into master Sep 1, 2026
4 checks passed
@github-actions
github-actions Bot deleted the fix/empty-ai-completion branch September 1, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant