-
Notifications
You must be signed in to change notification settings - Fork 270
fix(logging): stop writing the API key and DB password to the log stream #79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
e84457d
e503700
33a631d
fb5c4ec
17c1216
46181b2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,6 +31,7 @@ | |
| from app.protocols.sse import AdapterError | ||
| from app.quality_scores import resolve_model_metrics | ||
| from app.schemas import ChatCompletionRequest | ||
| from packages.auth.spend import budget_exceeded, get_lifetime_spend_microcents | ||
| from packages.auth.types import KeyContext | ||
| from packages.db.models.request_log import RequestLog | ||
| from packages.litellm_adapter.catalog import CATALOG, CATALOG_BY_ID | ||
|
|
@@ -306,6 +307,34 @@ async def execute_chat( | |
| detail=f"Model '{body.model}' is not allowed for this API key", | ||
| ) | ||
|
|
||
| # Budget enforcement: `budget_limit_cents` is a lifetime cap on this | ||
| # key's total spend (sum of `cost_microcents` for all non-deleted | ||
| # request-log rows, including streaming 499/503 rows that already | ||
| # incurred provider billing). Checked before any routing, resolution, | ||
| # or cache work so an exhausted key costs the operator nothing — no | ||
| # upstream attempt, no cache fill. | ||
| # | ||
| # NOTE: This is a best-effort soft limit, not a hard atomic cap. | ||
| # Spend is read before the request and the current request's cost is | ||
| # only written after the response/stream completes. N concurrent | ||
| # requests from the same budgeted key all observe the same | ||
| # pre-request total and may all pass the check, exceeding the cap by | ||
| # up to N× per-request cost in a burst. A hard cap would require a | ||
| # reservation/claim or row-level lock before dispatch; the current | ||
| # design trades strictness for simplicity and avoids holding a DB | ||
| # transaction across the upstream call. See spend.py for aggregation | ||
| # semantics. | ||
| if kc.budget_limit_cents is not None: | ||
| spend = await get_lifetime_spend_microcents(db, str(kc.key_id)) | ||
| if budget_exceeded(spend, kc.budget_limit_cents): | ||
| raise HTTPException( | ||
| status_code=429, | ||
| detail=( | ||
| "API key budget exhausted " | ||
| f"({spend} of {kc.budget_limit_cents * 10_000} microcents spent)." | ||
| ), | ||
| ) | ||
|
|
||
| client = await router_cache.get_router(db) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 P1 Make the budget check atomic (reserve/claim or lock) so concurrent requests cannot overshoot the cap The new enforcement is a plain check-then-act with no compare-and-swap: get_lifetime_spend_microcents(db, key_id) reads the sum of committed request-log rows, and only after the upstream call completes is the current request's cost written to requests_log (blocking path lines 746-750; streaming path _finalize lines 567-577, in a separate session). The check and the accounting write share no transaction, no lock, and no reservation. N concurrent requests from the same budgeted key all read the same pre-request total, all pass budget_exceeded(), all hit the provider, and the sum of their post-hoc cost rows exceeds budget_limit_cents by up to N× per-request cost. Since the API presents the check as "API key budget exhausted" (a hard lifetime cap on the operator's spend), the operator is billed beyond the limit they set — an accounted quantity limited wrongly. The code comment documents this as an accepted soft limit, which is why confidence is medium, but the consequence (unbounded-in-N overspend against a stated cap) is real and is exactly the read-modify-write-without-CAS pattern. Fix: reserve the expected cost before dispatch (e.g. an atomic pending-spend row per request, or SELECT ... FOR UPDATE on the api_keys row for the duration of the check+reserve, reconciled after completion) so the cap cannot be exceeded by concurrent admission. |
||
| raw_strategy = getattr(client, "strategy", None) | ||
| strategy = raw_strategy if isinstance(raw_strategy, str) and raw_strategy else "balanced" | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 P2 Make budget enforcement atomic against concurrent requests
The budget check in chat.py (lines 275-284) reads the key's historical spend and then serves the request; the current request's cost is only written to requests_log afterwards (after the response for non-stream, at stream end via _finalize for streaming). There is no atomic claim, reservation, or post-completion clamp. N concurrent requests from the same budgeted key all observe the same pre-budget total and all pass
budget_exceeded, so the lifetime cap can be exceeded by up to N× the per-request cost in a single burst. The commit's stated guarantee ("an exhausted key costs the operator nothing — no upstream attempt") holds only for sequentially-arriving requests; under concurrency the cap is soft. If the intended contract is a hard cap, this needs an atomic pre-check (e.g. reserve/claim the request's projected cost before dispatching) or a documented soft-limit semantics.