fix(ai): a total LLM outage was reporting itself as success - #151
Merged
Conversation
Found by driving the Lex demo in production: the chat answered "I'm sorry,
I encountered an error processing your request." Behind it, every AI feature
on the site is down and has been invisible.
The box has an invalid Groq key, no other provider key, and no Ollama:
Groq API error: 401 (model openai/gpt-oss-120b) {"code":"invalid_api_key"}
Groq API error: 401 (model openai/gpt-oss-20b) {"code":"invalid_api_key"}
Every endpoint answered HTTP 200 anyway:
/api/demo/chat {"success":true,"data":{"response":""}}
/api/quick-chat {"success":true,"data":{"response":"I'm having a moment..."}}
/api/professional-chat same
/api/health {"success":true,"data":{"status":"healthy"}}
Green everywhere, product dead. demo/chat was the worst of the three: its
catch returns `content: context`, and a bot demo has no context, so a failed
generation became a successful empty string.
- demo/chat throws LLMUnavailableError when there is nothing to fall back
on; the knowledge-base demo still serves its retrieved passage
- quick-chat and professional-chat return 503 LLM_UNAVAILABLE instead of
apologising with 200
- lib/llm-health.ts records what generations actually did; /api/health
reports it and returns 503 once the chain is consistently failing, so
"healthy" stops meaning "the database answered"
tests/__tests__/api/professional-chat.test.ts asserted the bug -- 200, the
word "moment", provider 'fallback'. It now asserts 503 LLM_UNAVAILABLE. A
test that pins broken behaviour is worse than no test.
Never-twice: tests/__tests__/lib/llm-health.test.ts covers the tracker's
state machine and asserts at source level that no chat route answers an LLM
failure with success, and that health consults the LLM chain.
This makes the outage visible. It does NOT fix it -- that needs a valid
GROQ_API_KEY (or OPENROUTER_API_KEY) in /opt/botsmann/shared/.env on the box.
Until then these endpoints will correctly return 503.
verify: format, lint, typecheck, 268 tests, build -- all green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVwg8DKHQktxJuHeLM3xpG
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 28, 2026
…#153) #151 made /api/health return 503 whenever the LLM chain was down. That is right for alerting and wrong for liveness: restarting botsmann does not repair an expired Groq key, so the only effect of failing there is to get a healthy process killed, and to fail deploy gates on a problem no deploy caused. That is the failure class that teaches people to bypass the gate. GET /api/health liveness -- 200 while the process serves and the database answers. The body still carries status: healthy | degraded | down and the full llm block, so the outage stays visible. GET /api/health?strict=1 readiness -- 503 once the chain is consistently failing. Point alerting here. 'unknown' (nothing has exercised the chain since this process started) is not evidence of a problem, so it reports healthy rather than degraded. The deploy gate was never at risk -- selfhost-deploy.yml curls the homepage, not this endpoint -- but the shape was wrong regardless, and the guidance in CLAUDE.md does point a post-deploy check at /api/health. The guard now asserts the 503-on-LLM path stays behind `strict`, so this cannot quietly become unconditional again. verify: format, lint, typecheck, 269 tests, build -- all green. Claude-Session: https://claude.ai/code/session_01HVwg8DKHQktxJuHeLM3xpG Co-authored-by: Mao Nakamoto <41178744+maonakamoto@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by driving the Lex demo in production. The chat answered "I'm sorry, I encountered an error processing your request." Behind it: every AI feature on the site is down, and has been invisible.
The outage
The box has an invalid Groq key, no other provider key, and no Ollama:
Every endpoint answered HTTP 200 anyway
Green everywhere, product dead.
demo/chatwas the worst of the three: its catch returnscontent: context, and a bot demo has no context — so a failed generation became a successful empty string.What changed
demo/chatthrowsLLMUnavailableErrorwhen there's nothing to fall back on; the knowledge-base demo still serves its retrieved passagequick-chatandprofessional-chatreturn 503LLM_UNAVAILABLEinstead of apologising with 200lib/llm-health.tsrecords what generations actually did./api/healthreports it and returns 503 once the chain is consistently failing — so "healthy" stops meaning "the database answered"A test was pinning the bug
professional-chat.test.tsasserted200, the word"moment", andprovider: 'fallback'. It now asserts503 LLM_UNAVAILABLE. A test that pins broken behaviour is worse than no test.Never-twice
tests/__tests__/lib/llm-health.test.tscovers the tracker's state machine and asserts at source level that no chat route answers an LLM failure with success, and that health consults the LLM chain.The actual fix needs a valid
GROQ_API_KEY(orOPENROUTER_API_KEY) in/opt/botsmann/shared/.envon the box. Until then these endpoints will correctly return 503 and/api/healthwill reportdegraded/down.Verify
format:check,lint,typecheck, 268 tests,build— all green.🤖 Generated with Claude Code
https://claude.ai/code/session_01HVwg8DKHQktxJuHeLM3xpG