Skip to content

Fix Google API key invalidation classification - #39

Merged
mitchins merged 4 commits into
mainfrom
codex/fix-google-key-invalidation
Jul 23, 2026
Merged

mitchins merged 4 commits into
mainfrom
codex/fix-google-key-invalidation

Conversation

@mitchins

@mitchins mitchins commented Jul 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • prevent generic Google 400/403/401 responses from quarantining API keys
  • classify SDK and image errors from structured Google fields; retain explicit RPD quotaId handling and RPM cooldowns
  • keep configured keys intact and persist invalid-key attribution/recovery in Redis
  • return bounded Google errors to clients and store only classification fields

Incident

A generic 400 INVALID_ARGUMENT matched a broad fallback indicator and invalidated keys fleet-wide.

Verification

  • uv run pytest -q tests/unit (1011 passed, 1 skipped)
  • uv run ruff check smolrouter/google_genai_provider.py smolrouter/redis_backend.py smolrouter/database.py smolrouter/mediator.py tests/unit/test_google_genai_helpers.py tests/unit/test_new_integrations.py tests/unit/test_redis_backend.py
  • adversarial Codex Sol final review: APPROVE

Summary by CodeRabbit

  • Bug Fixes

    • Improved Google upstream error classification using structured evidence from Google payloads (including quota ID and retry delay).
    • More reliably routes invalid keys vs quota exhaustion vs temporary failures, with evidence-driven cooldown timing.
    • Enhanced API-key invalidation/recovery to store and restore provider-scoped metadata (reason, optional status code, optional request ID) while keeping quota state consistent (including reset timing).
    • Refined request error handling for more accurate status/error-type responses.
  • Tests

    • Expanded unit and integration coverage for Google evidence parsing, quota/key handling, invalid-key recovery audits, and bounded evidence recording.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1162b7fa-e4f6-45ad-b397-dcec13363fb3

📥 Commits

Reviewing files that changed from the base of the PR and between babf8ad and a5f7470.

📒 Files selected for processing (4)
  • smolrouter/google_genai_provider.py
  • smolrouter/redis_backend.py
  • tests/unit/test_new_integrations.py
  • tests/unit/test_redis_backend.py
🚧 Files skipped from review as they are similar to previous changes (4)
  • tests/unit/test_redis_backend.py
  • tests/unit/test_new_integrations.py
  • smolrouter/redis_backend.py
  • smolrouter/google_genai_provider.py

📝 Walkthrough

Walkthrough

Google GenAI failures are parsed into structured evidence for request errors, quota classification, cooldowns, and API-key invalidation. Redis stores provider-scoped invalid-key metadata and recovery audits, while quota APIs and tests carry the expanded metadata.

Changes

Google error evidence and classification

Layer / File(s) Summary
Structured error parsing and classification
smolrouter/google_genai_provider.py, tests/unit/test_google_genai_helpers.py, tests/unit/test_new_integrations.py
Google failures expose typed evidence containing status, reason, quota identifier, and retry delay; invalid-key, quota, image-error, and bounded recording paths consume that evidence.
Typed request error propagation
smolrouter/google_genai_provider.py, smolrouter/mediator.py
Request errors carry status and error type fields through provider and mediator handling.
Provider-scoped quota invalidation and recovery
smolrouter/redis_backend.py, tests/unit/test_redis_backend.py
Redis stores invalid-key metadata, normalizes daily reset fields, preserves provider-specific quota state during recovery, and records bounded JSON audit entries.
Quota API metadata forwarding
smolrouter/database.py, tests/unit/test_database.py
Quota invalidation forwards reason, status code, and request ID to Redis, with delegation and return-value coverage.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: improving Google API key invalidation classification.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-google-key-invalidation

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.82%. Comparing base (de3c662) to head (a5f7470).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #39      +/-   ##
==========================================
+ Coverage   76.35%   76.82%   +0.46%     
==========================================
  Files          38       38              
  Lines       10736    10902     +166     
  Branches     1653     1685      +32     
==========================================
+ Hits         8198     8375     +177     
+ Misses       1938     1930       -8     
+ Partials      600      597       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mitchins
mitchins marked this pull request as ready for review July 23, 2026 02:19

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: babf8adb68

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +3357 to +3358
evidence = self._extract_google_error_evidence(error)
status_code = evidence.status_code

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore status fallback for unstructured Google errors

When the lower layer raises a non-Google exception that only carries the HTTP status in its text (for example a wrapper/transport RuntimeError("429 quota"), which _extract_status_code_from_exception() still explicitly supports), this now leaves status_code as None. The generic branch then records bounded evidence without the 429/400/403 signal, so quota/invalid-key state is not updated, and the mediator sees the new generic Google GenAI request failed. message and returns a 500 instead of the upstream status. Falling back to _extract_status_code_from_exception(error) when structured evidence has no status preserves the previous behavior for these unstructured errors.

Useful? React with 👍 / 👎.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
smolrouter/redis_backend.py (1)

1613-1625: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Recovery audit list grows unbounded.

recover_invalid lpushes to google_invalid_key_recovery_audit:{provider} but never trims it, so the list grows without bound as recoveries accumulate. Consider capping it (e.g. pipe.ltrim(audit_key, 0, MAX_AUDIT-1) in the same transaction) or setting a TTL.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@smolrouter/redis_backend.py` around lines 1613 - 1625, Update
recover_invalid’s audit-list transaction after the audit_key lpush to bound
google_invalid_key_recovery_audit:{provider}, using the existing audit-limit
constant if available and trimming it to retain only the newest entries. Keep
the trim in the same pipeline/transaction so each recovery cannot leave the list
unbounded.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@smolrouter/redis_backend.py`:
- Around line 1613-1625: Update recover_invalid’s audit-list transaction after
the audit_key lpush to bound google_invalid_key_recovery_audit:{provider}, using
the existing audit-limit constant if available and trimming it to retain only
the newest entries. Keep the trim in the same pipeline/transaction so each
recovery cannot leave the list unbounded.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 795d75d2-6965-4faf-a505-f925b0770363

📥 Commits

Reviewing files that changed from the base of the PR and between de3c662 and babf8ad.

📒 Files selected for processing (8)
  • smolrouter/database.py
  • smolrouter/google_genai_provider.py
  • smolrouter/mediator.py
  • smolrouter/redis_backend.py
  • tests/unit/test_database.py
  • tests/unit/test_google_genai_helpers.py
  • tests/unit/test_new_integrations.py
  • tests/unit/test_redis_backend.py

@sonarqubecloud

Copy link
Copy Markdown

@mitchins
mitchins merged commit 7568c8f into main Jul 23, 2026
12 checks passed
@mitchins
mitchins deleted the codex/fix-google-key-invalidation branch July 23, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant