Skip to content

Add per-account rate limiting with GCRA#121

Open
ravitri wants to merge 9 commits into
openshift-online:mainfrom
ravitri:ratelimit
Open

Add per-account rate limiting with GCRA#121
ravitri wants to merge 9 commits into
openshift-online:mainfrom
ravitri:ratelimit

Conversation

@ravitri

@ravitri ravitri commented Jul 10, 2026

Copy link
Copy Markdown

Description

Adds per-account rate limiting to the platform API using the Generic Cell Rate Algorithm (GCRA). Requests are tracked per AWS account ID, extracted from the X-Amz-Account-Id header injected by API Gateway via SigV4 signing.

Commit 1: Rate limiting middleware with Redis and in-memory backends

  • GCRA-based middleware at pkg/ratelimit/ with two backends:
    • Redis (go-redis/redis_rate) for production with Valkey/ElastiCache
    • In-memory for local development and CLI testing
  • YAML config with per-route overrides, exempt accounts, and default rate/burst/window
  • Fail-open when Redis is unreachable — requests allowed, failure_mode_allowed metric emitted
  • Prometheus counter ratelimit_requests_total with labels: method, path, result (ok | over_limit | failure_mode_allowed)
  • Response headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After (on 429)
  • RATE_LIMIT_TEST_MODE=true sets low defaults (rate=3, burst=6, window=1s)
  • Unit tests with miniredis covering middleware, GCRA algorithm, metrics, concurrency, fail-open, route overrides

Commit 2: E2E tests for rate limiting

  • 6 E2E tests behind Ginkgo label ratelimit: headers, 429 triggering, 429 response body, exempt accounts, per-account isolation, window reset
  • Concurrent goroutines with start-gate channel pattern to overwhelm GCRA burst (sequential HTTP requests over the network are too slow to outpace token replenishment)
  • Auto-discovers server rate from X-RateLimit-Limit response header; RATE_LIMIT_TEST_MODE=true on the test runner assumes rate=3
  • Run with: make test-e2e-api E2E_LABEL_FILTER=ratelimit

Commit 3: Documentation

  • docs/api/rate-limit.md — design doc covering algorithm, headers, 429 format, config, fail-open, metrics
  • docs/testing/rate-limit.md — three-layer testing strategy (unit/GCRA/E2E), scenario coverage matrix, environment variables
  • CLAUDE.md updated with rate limiting section

Test plan

  • make test — unit tests pass (miniredis, GCRA algorithm, config parsing)
  • RATE_LIMIT_TEST_MODE=true make test-e2e-api E2E_LABEL_FILTER=ratelimit — E2E tests pass against deployed API with Valkey
  • Verify 429 response with curl against a running instance with RATE_LIMIT_TEST_MODE=true
  • Confirm Prometheus metrics at /metrics endpoint show ratelimit_requests_total counter

🤖 Assisted by Claude Code

Summary by CodeRabbit

  • New Features
    • Added optional per-account API rate limiting with route-specific rules, configurable defaults, exempt accounts, and standard X-RateLimit-* response headers.
    • Over-limit requests return 429 with Retry-After, plus Prometheus metrics, with fail-open behavior when the rate-limit backend is unreachable.
  • Documentation
    • Added comprehensive rate-limiting and rate-limit testing docs, including how to run rate-limit E2E tests.
  • Tests
    • Added unit, middleware, and rate-limit E2E coverage for concurrency, recovery/reset timing, exemptions, and per-account isolation.
  • Chores
    • Updated Make targets for label-filtered rate-limit E2E runs and updated git ignore to exclude the rate-limit YAML config.

ravitri and others added 3 commits July 10, 2026 13:05
Implements per-account GCRA rate limiting for the platform API:

- Redis-backed limiter (go-redis/redis_rate) for production with Valkey/ElastiCache
- In-memory GCRA limiter for local development and CLI test mode
- Configurable per-route overrides, exempt accounts, and fail-open on backend errors
- Prometheus counter (ratelimit_requests_total) with ok/over_limit/failure_mode_allowed labels
- RATE_LIMIT_TEST_MODE=true for low-rate testing (rate=3, burst=6, window=1s)
- Unit tests with miniredis covering middleware, GCRA algorithm, metrics, concurrency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
E2E test suite that runs against a deployed API with real Valkey backend.
Uses concurrent goroutines with a start-gate channel pattern to overwhelm
GCRA burst before token replenishment.

Tests: rate limit headers, 429 triggering, 429 response body, exempt
accounts, per-account isolation, and window reset.

Auto-discovers the server's rate from X-RateLimit-Limit response header;
RATE_LIMIT_TEST_MODE=true on the test runner assumes rate=3.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- API design doc: GCRA algorithm, headers, 429 response format, exempt
  accounts, fail-open behavior, Prometheus metrics
- Testing doc: three-layer strategy (unit/GCRA/E2E), scenario coverage
  matrix, environment variables, running commands

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds configurable per-account API rate limiting with Redis/Valkey or in-memory GCRA, HTTP middleware enforcement, server integration, metrics and headers, E2E coverage, and operational documentation.

Changes

Rate Limiting

Layer / File(s) Summary
Configuration and limiter backends
pkg/config/config.go, pkg/ratelimit/config.go, pkg/ratelimit/local_rate_limiter.go, pkg/ratelimit/match.go, pkg/ratelimit/*_test.go, go.mod
Defines configuration, defaults, validation, route matching, Redis and in-memory limiter implementations, and unit coverage.
HTTP middleware enforcement
pkg/ratelimit/middleware.go, pkg/ratelimit/middleware_test.go
Applies account- and route-specific limits, emits metrics and headers, handles backend errors, and returns structured 429 responses.
Runtime configuration and server wiring
cmd/rosa-regional-platform-api/main.go, pkg/server/server.go
Reads environment settings, selects the limiter backend, installs middleware, and closes Redis during shutdown.
E2E validation and documentation
test/e2e-api/ratelimit_e2e_test.go, docs/api/rate-limit.md, docs/testing/rate-limit.md, Makefile, CLAUDE.md, .gitignore
Adds E2E scenarios, filtered test commands, rate-limit documentation, architecture notes, and the rate-limits.yaml ignore rule.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Middleware
  participant RateLimiter
  participant APIHandler
  Client->>Middleware: Send account-scoped request
  Middleware->>RateLimiter: Check account, method, route, and limit
  RateLimiter-->>Middleware: Return allow, deny, or backend error
  Middleware->>APIHandler: Forward allowed request
  Middleware-->>Client: Return headers or JSON 429
Loading

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error Middleware logs account_id, and startup logs endpoint/redis_addr/service URLs; these can expose customer IDs and internal hostnames. Redact or remove those fields from logs; use opaque identifiers/counts instead and avoid emitting backend hostnames, URLs, or account IDs at INFO/WARN.
Docstring Coverage ⚠️ Warning Docstring coverage is 3.77% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Ai-Attribution ⚠️ Warning AI use is declared, but commits use Co-Authored-By only; no Assisted-by/Generated-by trailers were found. Replace AI attribution with Red Hat trailers (Assisted-by or Generated-by) and remove Co-Authored-By from AI-assisted commits.
✅ Passed checks (8 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: per-account GCRA-based rate limiting.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No-Weak-Crypto ✅ Passed No weak crypto, custom crypto, or insecure secret/token compares were added; the new TLS client config sets MinVersion TLS1.2 and no InsecureSkipVerify.
Container-Privileges ✅ Passed No changed manifest or repo YAML/Dockerfile sets privileged, hostPID/Network/IPC, SYS_ADMIN, allowPrivilegeEscalation, or root-only settings.
No-Hardcoded-Secrets ✅ Passed Scanned the touched Go/docs/test files; found no hardcoded secrets, embedded-credential URLs, or long base64 literals. Only benign config/default strings.
No-Injection-Vectors ✅ Passed Changed files use yaml.Unmarshal and config parsing only; no SQL concatenation, shell/exec, pickle, or dangerous HTML injection patterns were found.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 15

🧹 Nitpick comments (2)
CLAUDE.md (1)

28-57: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Update the canonical source instead of editing only the generated file.

CLAUDE.md says it is auto-generated from AGENTS.md, but all new rate-limit guidance is added here. Regeneration can silently discard these changes; add the content to AGENTS.md and regenerate CLAUDE.md.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CLAUDE.md` around lines 28 - 57, Move the new rate-limiting guidance from
CLAUDE.md into the canonical AGENTS.md source, preserving the Architecture and
Rate Limiting sections and their details. Then regenerate CLAUDE.md using the
repository’s documented generation process and verify the generated file
contains the transferred content.
pkg/ratelimit/config_test.go (1)

1-365: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use Ginkgo/Gomega for this test suite.

This new *_test.go file uses testing.T assertions instead of the repository’s required Ginkgo/Gomega style. Migrate these cases so make test runs a consistent test framework. As per coding guidelines, “**/*_test.go: Use Ginkgo/Gomega for tests.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/ratelimit/config_test.go` around lines 1 - 365, The ratelimit test suite
uses testing.T instead of the repository-required Ginkgo/Gomega framework.
Migrate all tests in config_test.go, including
TestLoadConfig_*、TestNewDefaultConfig_*、and TestConfig_IsExempt, into Ginkgo
Describe/Context/It blocks with Gomega assertions, preserving the existing
scenarios and coverage so they run through the repository’s Ginkgo test setup.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/rosa-regional-platform-api/main.go`:
- Around line 181-182: Remove the Redis/Valkey endpoint from the rate-limit
test-mode logs in the relevant logging branches, including the log statements
near the rate-limit backend selection. Log only the selected backend identifier,
such as “redis” or “in_memory”, and retain the rate configuration if needed
without including cfg.RateLimit.RedisAddr.
- Around line 197-205: Handle conversion errors for RATE_LIMIT_DEFAULT_RATE,
RATE_LIMIT_DEFAULT_BURST, and RATE_LIMIT_DEFAULT_WINDOW in the
environment-loading logic, rather than discarding strconv.Atoi errors. Return a
descriptive error identifying the malformed variable and stop configuration
initialization when conversion fails.

In `@docs/api/rate-limit.md`:
- Around line 114-116: Add language identifiers to the fenced code blocks
containing the `ratelimit_requests_total` metric and directory-tree examples,
using `text` or another appropriate language so markdownlint passes.

In `@docs/testing/rate-limit.md`:
- Around line 128-138: Align the test-mode backend semantics across
docs/testing/rate-limit.md and docs/api/rate-limit.md. Update the test-mode
documentation to state that RATE_LIMIT_TEST_MODE changes the defaults, while
backend selection still depends on whether REDIS_ENDPOINT is set; ensure both
documents consistently describe in-memory GCRA versus real Valkey behavior.

In `@go.mod`:
- Around line 88-89: Upgrade the aligned go.opentelemetry.io/otel and
go.opentelemetry.io/otel/trace dependencies from v1.39.0 to v1.41.0 or later in
go.mod, and update the dependency lock data consistently.

In `@pkg/ratelimit/config.go`:
- Around line 39-43: Prevent integer overflow when deriving the default burst in
the config normalization paths, including the logic around cfg.Default and the
additional rate-limit configurations. Before computing rate * 2, validate that
the rate is no greater than MaxInt/2; reject or return a configuration error
otherwise, and add boundary tests covering MaxInt/2 and MaxInt/2+1.

In `@pkg/ratelimit/local_rate_limiter_test.go`:
- Around line 69-110: Update the Allow calls in TestLocalLimiter_IsolatesKeys,
TestLocalLimiter_ResetAfterIsPositive, and
TestLocalLimiter_DeniedResultHasZeroRemaining to check and assert returned
errors before inspecting results. Replace discarded errors with explicit test
failures so unexpected setup or request errors are reported clearly instead of
allowing assertions on invalid results.

In `@pkg/ratelimit/middleware_test.go`:
- Around line 22-28: Update the cleanup handlers in setupTest and the other two
Redis client cleanup locations to capture the error returned by rdb.Close() and
report non-nil errors with t.Errorf, ensuring no Redis cleanup error is ignored.

In `@pkg/ratelimit/middleware.go`:
- Around line 123-130: Handle the error returned by json.NewEncoder(w).Encode in
Limiter.writeRateLimitError instead of discarding it; capture the error and
report it through the middleware’s established logging or error-handling
mechanism, while preserving the existing HTTP response behavior.
- Around line 71-95: Sanitize rate-limit warning logs to avoid exposing tenant
identifiers, raw resource paths, or backend details. In the failure branch of
the rate-limit middleware, replace the logged key and raw error with a fixed
failure category and log the matched route pattern instead of request-specific
identifiers; in the “rate limit exceeded” branch, remove account_id and
r.URL.Path while retaining only non-sensitive method, route pattern, rate, and
retry metadata.

In `@pkg/server/server.go`:
- Around line 70-90: Update the rate-limit configuration logic in the server
initialization block to preserve YAML routes and exempt accounts: when a config
file is configured, call ratelimit.LoadConfig first, then overwrite only the
non-zero RATE_LIMIT_DEFAULT_* fields; use ratelimit.NewDefaultConfig only when
no config file is configured. Keep logging consistent with the resulting source
and ensure route and exemption data from YAML remains intact.

In `@test/e2e-api/ratelimit_e2e_test.go`:
- Around line 88-89: Redact sensitive values in all GinkgoWriter logs within the
rate-limit E2E test, including the messages around the baseURL/accountID
diagnostic and the additionally referenced ranges. Use safe redacted
representations for account IDs and potentially internal URLs while preserving
useful logging context; update the relevant logging statements without exposing
raw os.Getenv("E2E_BASE_URL") or accountID values.
- Around line 55-90: Rate-limit E2E specs share a persistent account and bucket,
making results dependent on test order and prior runs. Update the Rate Limiting
suite setup and each spec to use isolated account IDs/namespaces or explicitly
reset and wait for rate-limit state before execution; do not rely on Ordered
alone to provide isolation.
- Around line 110-112: The rate-limit tests incorrectly derive burst capacity as
rateLimitRate*2, despite burst being independently configurable. Update the
affected tests and request-count calculations to discover the actual burst value
from configuration or headers, or explicitly use a test route/configuration with
a known burst; ensure all 429 assertions exceed that real burst.
- Around line 205-217: In the rate-limit response handling within
concurrentRequests, stop ignoring strconv.Atoi errors: require X-RateLimit-Reset
to be present and assert that parsing succeeds, failing the test immediately for
a missing or malformed header instead of defaulting resetSeconds to 2.

---

Nitpick comments:
In `@CLAUDE.md`:
- Around line 28-57: Move the new rate-limiting guidance from CLAUDE.md into the
canonical AGENTS.md source, preserving the Architecture and Rate Limiting
sections and their details. Then regenerate CLAUDE.md using the repository’s
documented generation process and verify the generated file contains the
transferred content.

In `@pkg/ratelimit/config_test.go`:
- Around line 1-365: The ratelimit test suite uses testing.T instead of the
repository-required Ginkgo/Gomega framework. Migrate all tests in
config_test.go, including TestLoadConfig_*、TestNewDefaultConfig_*、and
TestConfig_IsExempt, into Ginkgo Describe/Context/It blocks with Gomega
assertions, preserving the existing scenarios and coverage so they run through
the repository’s Ginkgo test setup.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3e2b02a5-32a2-473f-bfb8-c943692b840d

📥 Commits

Reviewing files that changed from the base of the PR and between 4521cd6 and 2b90020.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (18)
  • .gitignore
  • CLAUDE.md
  • Makefile
  • cmd/rosa-regional-platform-api/main.go
  • docs/api/rate-limit.md
  • docs/testing/rate-limit.md
  • go.mod
  • pkg/config/config.go
  • pkg/ratelimit/config.go
  • pkg/ratelimit/config_test.go
  • pkg/ratelimit/local_rate_limiter.go
  • pkg/ratelimit/local_rate_limiter_test.go
  • pkg/ratelimit/match.go
  • pkg/ratelimit/match_test.go
  • pkg/ratelimit/middleware.go
  • pkg/ratelimit/middleware_test.go
  • pkg/server/server.go
  • test/e2e-api/ratelimit_e2e_test.go

Comment thread cmd/rosa-regional-platform-api/main.go
Comment thread cmd/rosa-regional-platform-api/main.go
Comment thread docs/api/rate-limit.md
Comment on lines +114 to +116
```
ratelimit_requests_total{method, path, result}
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add language identifiers to fenced code blocks.

Use text (or another appropriate language) for the metric and directory-tree examples so markdownlint passes.

Also applies to: 143-149

🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 114-114: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/api/rate-limit.md` around lines 114 - 116, Add language identifiers to
the fenced code blocks containing the `ratelimit_requests_total` metric and
directory-tree examples, using `text` or another appropriate language so
markdownlint passes.

Source: Linters/SAST tools

Comment on lines +128 to +138
## Test Mode Configuration (`RATE_LIMIT_TEST_MODE`)

`RATE_LIMIT_TEST_MODE=true` sets low rate limit defaults for testability. The backend depends on whether `REDIS_ENDPOINT` is also set:

| `RATE_LIMIT_TEST_MODE` | `REDIS_ENDPOINT` | Backend | Use case |
|---|---|---|---|
| `true` | unset | In-memory GCRA | CLI / local development |
| `true` | set | Real Valkey | E2E testing in deployed environments |
| `false` | set | Real Valkey | Production |

Test mode defaults: `rate=3`, `burst=6`, `window=1s`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Clarify the backend semantics of test mode across the documentation.

This table says RATE_LIMIT_TEST_MODE=true uses in-memory mode only when REDIS_ENDPOINT is unset, while docs/api/rate-limit.md describes test mode as in-memory/no Redis. Document that test mode changes defaults and backend selection still depends on REDIS_ENDPOINT, or make both documents describe the same behavior.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/testing/rate-limit.md` around lines 128 - 138, Align the test-mode
backend semantics across docs/testing/rate-limit.md and docs/api/rate-limit.md.
Update the test-mode documentation to state that RATE_LIMIT_TEST_MODE changes
the defaults, while backend selection still depends on whether REDIS_ENDPOINT is
set; ensure both documents consistently describe in-memory GCRA versus real
Valkey behavior.

Comment thread go.mod Outdated
Comment thread pkg/server/server.go Outdated
Comment thread test/e2e-api/ratelimit_e2e_test.go
Comment thread test/e2e-api/ratelimit_e2e_test.go
Comment thread test/e2e-api/ratelimit_e2e_test.go Outdated
Comment thread test/e2e-api/ratelimit_e2e_test.go Outdated
Comment thread pkg/ratelimit/config.go Outdated
Comment thread pkg/ratelimit/config.go
Comment thread pkg/ratelimit/config.go Outdated
@openshift-ci

openshift-ci Bot commented Jul 13, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ravitri
Once this PR has been reviewed and has the lgtm label, please ask for approval from psav. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ravitri

ravitri commented Jul 13, 2026

Copy link
Copy Markdown
Author

/label review-ready

@slopezz

slopezz commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

PR #121 Review: Add per-account rate limiting with GCRA (API)

PR: #121
Author: @ravitri (5 commits, latest: 861e5a8 Jul 13 — "Address PR review feedback")
Reviewer: @slopezz (design doc author, SRE)
Date: 2026-07-14
Status: Changes requested (by @psav, Jul 10 — addressed in latest commit)


Overall Assessment

Solid GCRA implementation that aligns with the design doc. Pete's feedback (failOverTimeout rename, dead config, field documentation) has been addressed in the latest commit. From an SRE perspective, simplification (remove in-memory backend) and security hardening (TLS) are the remaining must-fix items.


What's Good

  • Clean package structure: pkg/ratelimit/ — config, middleware, backends, route matching — well organized
  • GCRA via go-redis/redis_rate: Correct library choice, Lua script handles atomic check-and-set with automatic key TTL (SET key value EX ceil(reset_after) — keys self-expire, no memory leak)
  • Fail-open with metrics: When Redis is unreachable, requests pass through and failure_mode_allowed counter increments — exactly as designed
  • Prometheus metrics: ratelimit_requests_total with method, path, result labels — no account_id in labels (avoids cardinality explosion)
  • Response headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After on 429 — RFC-compliant
  • E2E tests with concurrent goroutines: Start-gate channel pattern to overwhelm burst — correct approach for GCRA testing
  • ConfigMap-driven YAML config: Hot-reload via hash annotation, no image rebuild for limit changes
  • Pete's feedback addressed (commit 861e5a8): failOverTimeout renamed, lint fixes, YAML config preserved

Must-Fix Before Merge

1. OpenTelemetry CVE (GHSA-mh2q-q3fh-2475)

go.opentelemetry.io/otel v1.39.0 is affected by CVE-2026-29181 (DoS via multi-value baggage headers). Fixed in v1.41.0.

Action: go get go.opentelemetry.io/otel@v1.41.0 go.opentelemetry.io/otel/trace@v1.41.0

2. Valkey TLS in client — must match RDS standard

The Go code connecting to Valkey must use TLS when REDIS_ENDPOINT is set. We mandate TLS for RDS (parameter_group forces SSL). Same standard for Valkey:

rdb := redis.NewClient(&redis.Options{
    Addr:      cfg.RedisAddr,
    TLSConfig: &tls.Config{MinVersion: tls.VersionTLS12},
})

Overhead is <1ms. We store account IDs. FedRAMP SC-8 requires encryption in transit. Customer-managed KMS key for encryption at rest (infra PR handles this).

Action: Add TLSConfig when connecting to deployed Redis. No TLS needed for miniredis in tests.

3. Consider removing in-memory backend — simplify to single code path

The in-memory GCRA implementation is nice (works locally without infra), but it adds maintenance cost:

  • Extra library dependency (local_rate_limiter.go + tests)
  • Backend selection logic (env var matrix)
  • Two code paths to maintain and test
  • The enable_rate_limit_redis boolean in infra that propagates through Terraform/Helm/ApplicationSet

A simpler approach: shared Redis/Valkey among all pods, fail-open on timeout with metric. This is how ROSA v1 works with Limitador — one backend, one code path. Less code, fewer libs to maintain.

For unit tests: miniredis is already used (stays). For local dev: docker run redis or just let it fail-open (requests pass, metric emits — effectively the same behavior as in-memory but with zero extra code).

RATE_LIMIT_TEST_MODE would only control rate defaults (rate=3, burst=6, window=1s), not backend selection.

Recommendation: Consider removing in-memory backend in favor of simpler single-path architecture. Less to maintain, less to break.


Should-Fix

4. E2E tests: verify per-account isolation with distinct account IDs

Critical test gap: do different account IDs have independent rate-limit buckets? If the key format accidentally doesn't namespace per account, one customer's traffic would consume another's quota.

The E2E should:

  • Use at least 2 different (fake) account IDs via X-Amz-Account-Id header
  • Exhaust the limit for account A (get 429)
  • Verify account B still gets 200 (independent bucket)
  • Use random/UUID account IDs per test run to avoid flakiness from leftover Valkey state

Note: ephemeral E2E env already deploys Valkey — this runs against real infra as production would.

5. Window field validation

The GCRA emission interval is calculated as period / rate. If Window (period) is 0, this results in a zero-division. If negative, it produces a negative emission interval which breaks the GCRA algorithm (negative burst offset, inverted allow/deny logic).

Currently, a misconfigured YAML like window: 0 or window: -1 would be loaded without error and produce undefined behavior at runtime — either a panic (zero division) or silently allowing all requests (infinite rate).

Action: Add validation in LoadConfig() / NewDefaultConfig():

  • Reject Window <= 0 with a descriptive error at config load time
  • Same for Rate <= 0 (zero rate = no token replenishment = permanent block after burst)
  • Same for Burst < 0 (negative burst makes no sense)

This catches operator mistakes in the ConfigMap before they reach production. Pete's request for field documentation (comments on struct fields) was addressed in latest commit 861e5a8.

6. Redis timeout default 50ms — lower to 10-20ms

Default redisTimeout: 50 (ms) in Go code. For same-region single-node Valkey (typical latency <1ms p50, <5ms p99), 10-20ms is more appropriate. If Valkey has a problem, you don't want every API request paying 50ms of latency waiting before failing open. Fail fast, let the failure_mode_allowed metric tell you something is wrong.


Dismiss

7. CodeRabbit No-Sensitive-Data-In-Logs — dismiss both findings

CodeRabbit flagged redis_addr in startup logs and account_id in 429 denial logs. Both follow existing patterns in this codebase:

  • DynamoDB endpoints, Maestro URLs, Hyperfleet URLs, Cedar endpoints are all logged the same way
  • Account_id in denial logs is the required investigation path when RateLimitHighDenialRatio fires (account_id is intentionally NOT in metric labels to avoid cardinality)

Override the pre-merge check with this justification.


Non-blocking

8. Test framework mismatch

config_test.go uses testing.T while the rest of the repo uses Ginkgo/Gomega. Both work fine with go test — it's a style inconsistency, not a functional issue. Can be migrated in a follow-up if the team wants uniformity.

9. CLAUDE.md vs AGENTS.md

Rate-limiting docs were added to CLAUDE.md but this file is auto-generated from AGENTS.md. Next regeneration will overwrite the additions. Move content to AGENTS.md as the canonical source.

10. strconv.Atoi errors silently ignored

When RATE_LIMIT_DEFAULT_RATE, RATE_LIMIT_DEFAULT_BURST, or RATE_LIMIT_DEFAULT_WINDOW env vars contain non-numeric values (e.g., a typo like "2O" instead of "20"), the parse error is discarded and the hardcoded default is used silently. Low-risk (Helm templates these), but a log.Warn would save debugging time.


Verified — No Action Needed

  • Redis key TTL: go-redis/redis_rate Lua script sets automatic TTL via SET key value EX ceil(reset_after). Keys auto-expire. No memory growth risk. (Verified against kuadrant/limitador.)
  • Pete's feedback (commit 861e5a8): failOverTimeout renamed to redisTimeout ✅, dead config moved to Config level ✅, YAML config preserved ✅

Summary of Actions

# Item Priority Action
1 OpenTelemetry CVE Must-fix Bump to v1.41.0
2 Valkey TLS in client Must-fix Add TLSConfig when REDIS_ENDPOINT set
3 Consider removing in-memory backend Recommended Simpler: single backend, less libs, less code to maintain
4 E2E: per-account isolation Should-fix Use 2+ distinct account IDs, verify independent buckets
5 Window field validation Should-fix Reject <=0 at config load to avoid zero-division/infinite-rate
6 Redis timeout default 50ms Should-fix Lower to 10-20ms — don't add 50ms latency per request if Valkey is degraded
7 CodeRabbit log findings Dismiss Both follow existing patterns (DynamoDB, Maestro, etc.)
8 Test framework mismatch Non-blocking config_test.go uses testing.T, repo convention is Ginkgo
9 CLAUDE.md vs AGENTS.md Non-blocking Rate-limit docs in wrong source file, will be overwritten
10 strconv.Atoi errors silent Non-blocking Malformed env vars silently use defaults, add log.Warn

Verdict

Excellent work — really well done. The code is clean, well-structured, and shows a strong understanding of the GCRA algorithm and production-grade middleware design. Test coverage is thorough (unit + E2E with concurrent patterns), Pete's feedback was addressed quickly, and the implementation follows the design doc closely. This is exactly the quality bar we want.

Approve with minor changes on items 1-2 (CVE bump, TLS). Item 3 (in-memory removal) is a recommendation for keeping simpler code. Items 4-5-6 (E2E isolation, validation, timeout) are should-fix within this PR. Item 7 is a CodeRabbit dismiss. Items 8-9-10 are non-blocking — can be fixed or not, no impact. Great job.

@typeid

typeid commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Removing review-ready as a review has been submitted by @slopezz -please re-add when this is ready again.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/ratelimit/config.go`:
- Around line 60-62: Validate RedisTimeout in the configuration normalization
logic before it is converted to time.Duration, rejecting values above the
maximum safe millisecond duration instead of allowing overflow. Preserve the
existing defaulting for non-positive values, and add a boundary test covering
the maximum accepted value and the first overflowing value.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d461639a-848a-40c7-af8a-6c3e4347d9ea

📥 Commits

Reviewing files that changed from the base of the PR and between 861e5a8 and 47d4fee.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (5)
  • docs/api/rate-limit.md
  • go.mod
  • pkg/ratelimit/config.go
  • pkg/ratelimit/middleware_test.go
  • pkg/server/server.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • docs/api/rate-limit.md
  • pkg/ratelimit/middleware_test.go
  • pkg/server/server.go

Comment thread pkg/ratelimit/config.go
Comment on lines +60 to +62
if cfg.RedisTimeout <= 0 {
cfg.RedisTimeout = 10
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

rg -n -C 4 'RedisTimeout|WithTimeout' pkg/ratelimit
rg -n -C 3 'redisTimeout|RedisTimeout' pkg/ratelimit/config_test.go pkg/ratelimit/middleware_test.go

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 7601


🏁 Script executed:

#!/bin/bash
set -eu

sed -n '1,140p' pkg/ratelimit/config.go
printf '\n----\n'
sed -n '1,180p' pkg/ratelimit/middleware.go

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 6860


🏁 Script executed:

#!/bin/bash
set -eu

rg -n -C 4 'type RateLimiter|func .*Allow\(|context\.DeadlineExceeded|DeadlineExceeded|context.Canceled|fail-open|failure_mode_allowed' pkg/ratelimit

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 4965


🏁 Script executed:

#!/bin/bash
set -eu

rg -n -C 3 'RedisTimeout|redisTimeout|MaxDuration|overflow|boundary' pkg/ratelimit

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 4388


🏁 Script executed:

#!/bin/bash
set -eu

cat <<'EOF' >/tmp/timeout_overflow.go
package main

import (
	"fmt"
	"time"
)

func main() {
	vals := []int{
		10,
		2147483647,
		9223372036854,
		9223372036855,
		9223372036854775,
	}
	for _, v := range vals {
		d := time.Duration(v) * time.Millisecond
		fmt.Printf("%d => duration=%d ns, repr=%s, expiredNow=%v\n", v, int64(d), d, time.Now().Add(d).Before(time.Now()))
	}
}
EOF

go run /tmp/timeout_overflow.go

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 573


Bound RedisTimeout before converting it to time.Duration. A large YAML value can wrap time.Duration(l.config.RedisTimeout) * time.Millisecond negative, which makes the context expire immediately and pushes Redis errors into the fail-open path. Reject values above the max safe millisecond duration and add a boundary test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/ratelimit/config.go` around lines 60 - 62, Validate RedisTimeout in the
configuration normalization logic before it is converted to time.Duration,
rejecting values above the maximum safe millisecond duration instead of allowing
overflow. Preserve the existing defaulting for non-positive values, and add a
boundary test covering the maximum accepted value and the first overflowing
value.

Source: Path instructions

@ravitri
ravitri requested a review from psav July 17, 2026 05:24
@ravitri

ravitri commented Jul 17, 2026

Copy link
Copy Markdown
Author

/label review-ready

@openshift-ci

openshift-ci Bot commented Jul 17, 2026

Copy link
Copy Markdown

@ravitri: The label(s) /label review-ready cannot be applied. These labels are supported: acknowledge-critical-fixes-only, platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, ux-approved, no-qe, rebase/manual, cluster-config-api-changed, run-integration-tests, verified, ready-for-human-review, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/skip-dependent-bug-check, jira/valid-bug, ok-to-test, stability-fix-approved, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

Details

In response to this:

/label review-ready

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants