Skip to content

fix(cache-proxy): scope cache keys by SigV4 credential and lock down relay paths - #1156

Merged
fuziontech merged 2 commits into
mainfrom
sec/cache-proxy-tenant-isolation
Sep 4, 2026
Merged

fix(cache-proxy): scope cache keys by SigV4 credential and lock down relay paths#1156
fuziontech merged 2 commits into
mainfrom
sec/cache-proxy-tenant-isolation

Conversation

@fuziontech

Copy link
Copy Markdown
Member

Summary

Security fix for two findings in the cache-proxy DaemonSet, an unauthenticated forward proxy reachable cluster-wide via hostPort:

  • Cross-tenant cache reads (High). Cache keys were sha256(url | range) — no credential material. A hit (local NVMe or a peer node) is served with zero authorization. Managed-warehouse tenants share a bucket with per-org path prefixes, so a tenant requesting another org's object URL was served the warm entry despite holding credentials S3 would reject. Cache keys now mix in the request's tenant scope — the SigV4 access key ID, unique per issued STS credential set — so a cross-tenant request misses and goes to origin, where S3 authorization applies. Unsigned requests share one namespace (correct for public objects). Peer traffic carries opaque keys, so the peer protocol is unchanged.
  • Open relay / SSRF (High). CONNECT dialed any host:port and the plain-HTTP forward path forwarded to any absolute URL, from the node's network context (bypassing pod egress policy). CONNECT is now restricted to port 443, refuses loopback/link-local/unspecified IP literals, and honors the new CONNECT_ALLOWED_SUFFIXES list when set. The plain-HTTP forward path now forwards only to hosts matching CACHE_HOST_SUFFIXES when that list is configured (signed S3 traffic matches by definition; the MW chart sets amazonaws.com). Legacy no-suffix mode is unchanged.

Changes

  • cmd/cache-proxy/scope.go (new): TenantScope(r) extracts the access key ID from the SigV4 Authorization header (an identifier, not a secret — the secret key and signature never enter the key).
  • CacheKey(scope, url, range) and BlockKey(scope, url, idx, size) take the scope; hash input uses a NUL separator so the scope boundary is unambiguous. All call sites threaded; peer protocol untouched.
  • handleConnect: connectRefusalReason gates the target before dialing.
  • New forward wrapper gates plain-HTTP forwarding on CACHE_HOST_SUFFIXES.
  • main.go wires CONNECT_ALLOWED_SUFFIXES.

Behavior changes / risks

  • All existing cache entries become unreachable after rollout (new key space) and age out via LRU — a cold-cache roll, brief origin-traffic spike expected.
  • Tenant cache sharing disappears even within an org when credentials rotate (scope = access key ID). Cache hit rate may drop; the block-summaries/peer machinery is unaffected.
  • Plain-HTTP reads of non-S3 hosts (read_csv('http://...')) are refused when CACHE_HOST_SUFFIXES is set; external HTTPS reads still work via CONNECT:443.

Testing

  • New tests: two access key IDs → independent namespaces (B misses A's warm entry, origin-authorizes); unsigned requests share; CONNECT to :8816 / link-local / non-443 → 403; :443 hostname allowed; plain-HTTP non-suffix host → 403.
  • go test ./cmd/cache-proxy/... green (note: the suite's disk-capacity tests need >5% free disk; verified on tmpfs).

Rollback

Reverting the image pin restores the old key space; no persistent state to migrate.

…relay paths

Security fix for two findings in the unauthenticated forward proxy:

(A) Cache keys mixed no credential material, so a warm entry (local or
from a peer) was served with zero authorization. In the managed-warehouse
topology tenants share a bucket with per-org path prefixes, so one tenant
could read another tenant's objects from a warm cache. The tenant scope —
the SigV4 access key ID from the Authorization header, unique per issued
STS credential set — is now part of every CacheKey and BlockKey hash
input (scope + "\x00" + url + ...). A cross-tenant request misses and
goes to the origin, where S3 authorization applies. Unsigned requests
share the empty-scope namespace, which is correct for public objects.
Peer traffic carries only opaque keys, so the peer protocol is unchanged.

(B) handleConnect dialed any host:port and the plain-HTTP forward path
forwarded to any absolute URL — open relay and SSRF primitives reachable
cluster-wide via the hostPort. CONNECT now allows only port 443, refuses
loopback/link-local/unspecified IP literals, and honors the new
CONNECT_ALLOWED_SUFFIXES hostname list when configured. The plain-HTTP
forward path forwards only to hosts matching CACHE_HOST_SUFFIXES when
that list is configured (signed S3 traffic matches by definition);
legacy no-suffix mode stays unrestricted for backward compatibility.

Co-authored-by: Shelley <shelley@exe.dev>
@fuziontech
fuziontech requested a review from a team September 3, 2026 23:38
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 1 6 0
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +10 / -0
  • Assertions: +44 / -1
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

@fuziontech
fuziontech merged commit ccbff7a into main Sep 4, 2026
35 checks passed
@fuziontech
fuziontech deleted the sec/cache-proxy-tenant-isolation branch September 4, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant