Problem
The PostgreSQL-backed token-bucket rate limiters perform their decrement as a
non-atomic SELECT tokens ... ; UPDATE tokens = tokens - cost ... guarded only by
a per-process lock. Under simultaneous cross-node bursts for the same key, two nodes
can both read tokens >= cost and both decrement, allowing a small bounded
overshoot beyond the configured rate.
This affects both PG-backed token buckets:
This is an accepted limitation today (the overshoot is bounded by concurrency, and it
is a massive improvement over the pre-#1332 configured x workers x nodes behavior).
The #1332 docstrings were made honest about it ("bounded ... small transient overshoot
possible under simultaneous cross-node bursts"). Filing so strict bounding can be done
deliberately if ever required.
Proposed fix
Replace the SELECT-then-UPDATE in TokenBucketManager._pg_consume with a single
atomic conditional decrement, e.g.:
UPDATE <table> SET tokens = tokens - %s, last_access = %s
WHERE <key_column> = %s AND tokens >= %s
RETURNING tokens
- Row present + sufficient tokens -> one row returned, decremented atomically.
- Insufficient tokens -> zero rows -> deny.
- Combine with the existing refill computation (may need a refill-then-decrement in one
statement or a short transaction with SELECT ... FOR UPDATE).
Applies to BOTH limiters (shared TokenBucketManager), so it is a single change with
broad blast radius -> requires the full auth + admission test suites, and a real-PG
concurrency test proving no overshoot under simultaneous cross-node consumption.
Acceptance criteria
Severity
priority-4. Bounded, non-corrupting, accepted today; strict bounding is a hardening.
Source: code review of #1332 (non-blocking note #2).
Problem
The PostgreSQL-backed token-bucket rate limiters perform their decrement as a
non-atomic
SELECT tokens ... ; UPDATE tokens = tokens - cost ...guarded only bya per-process lock. Under simultaneous cross-node bursts for the same key, two nodes
can both read
tokens >= costand both decrement, allowing a small boundedovershoot beyond the configured rate.
This affects both PG-backed token buckets:
PerConsumerRateLimiter->consumer_rate_limit_state(added in feat: request admission control / backpressure middleware (opt-in) #1332).token_bucket_state(pre-existing; migration 024's"atomic UPDATE with RETURNING" comment notwithstanding, the code has always been
SELECT-then-UPDATE).
This is an accepted limitation today (the overshoot is bounded by concurrency, and it
is a massive improvement over the pre-#1332
configured x workers x nodesbehavior).The #1332 docstrings were made honest about it ("bounded ... small transient overshoot
possible under simultaneous cross-node bursts"). Filing so strict bounding can be done
deliberately if ever required.
Proposed fix
Replace the SELECT-then-UPDATE in
TokenBucketManager._pg_consumewith a singleatomic conditional decrement, e.g.:
statement or a short transaction with
SELECT ... FOR UPDATE).Applies to BOTH limiters (shared
TokenBucketManager), so it is a single change withbroad blast radius -> requires the full auth + admission test suites, and a real-PG
concurrency test proving no overshoot under simultaneous cross-node consumption.
Acceptance criteria
_pg_consumedecrement is atomic (single conditionalUPDATE ... RETURNINGorSELECT FOR UPDATEtransaction); no SELECT-then-UPDATE race window.nodes never exceed the configured capacity (zero overshoot).
Severity
priority-4. Bounded, non-corrupting, accepted today; strict bounding is a hardening.
Source: code review of #1332 (non-blocking note #2).