fix: guard chaos injection behind CHAOS_ENABLED env var in /suggest endpoint#199
Open
devin-ai-integration[bot] wants to merge 1 commit into
Open
fix: guard chaos injection behind CHAOS_ENABLED env var in /suggest endpoint#199devin-ai-integration[bot] wants to merge 1 commit into
devin-ai-integration[bot] wants to merge 1 commit into
Conversation
…ndpoint The /suggest endpoint had a chaos-injected ranking-score enrichment path that was unconditionally triggered by the Redis flag chaos:search-service:suggest_500. This path calls service.suggest() which returns list[str], then tries to sort by s['_rankingScore'] — causing TypeError/KeyError and returning 500s. Guard the chaos path behind CHAOS_ENABLED=true so it cannot fire in production. The normal suggest path (which works correctly) is now always used unless chaos mode is explicitly enabled.
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the high 5xx error rate on search-service caused by the
/suggestendpoint's chaos-injected ranking-score enrichment path.Root cause: The
/suggestendpoint had a chaos code path triggered unconditionally by the Redis flagchaos:search-service:suggest_500. This path:service.suggest(prefix)which returnslist[str](plain strings)s["_rankingScore"]on those strings →TypeError/KeyError[{}]for empty results, which also raisesKeyErrorFix: Guard the chaos injection behind a
CHAOS_ENABLEDenvironment variable (defaulting tofalse). The broken ranking path can no longer fire in production. The normal suggest path — which correctly returns suggestions without ranking enrichment — is always used unless chaos mode is explicitly enabled.Review & Testing Checklist for Human
CHAOS_ENABLEDis not set totruein any production deployment config (Helm values, env vars, k8s manifests)/suggestdrops after deployment by checking Grafana dashboard/suggestendpoint manually:curl /api/v1/search/suggest?q=testshould return 200 with suggestionsNotes
CHAOS_ENABLED=trueto re-enable itSearchSuggestHighErrorRateLink to Devin session: https://partner-workshops.devinenterprise.com/sessions/249344b7740f4bc59aee1c352ed685a2