Skip to content

fix(cache-proxy): scope cache keys by SigV4 credential and lock down relay paths #1156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fuziontech merged 2 commits into from
Sep 4, 2026
+673 −82
Merged

fix(cache-proxy): scope cache keys by SigV4 credential and lock down relay paths #1156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
321665d
fix(cache-proxy): scope cache keys by SigV4 credential and lock down …
fuziontech Sep 3, 2026
da5747a
test(cache-proxy): appease staticcheck SA4000 in tenant-scope key test
fuziontech Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions cmd/cache-proxy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ available, and forwards cache misses to origin object storage.
| `CACHE_MAX_CONCURRENT_PEER_PROBES` | `64` | Per-pod non-blocking cap on active summary-mode `/cache/has` HTTP requests and sockets. It is not a process goroutine limit. When exhausted, confirmations are skipped and the request fetches origin. `CACHE_MAX_PEER_PROBES_IN_FLIGHT` is a deprecated alias. |
| `CACHE_PROXY_ID` | pod name, node name, then hostname | Stable opaque receiver identity used for deterministic peer-summary selection; it must not be a customer or object identifier. |
| `HEALTH_ADDR` | `:8082` | Health and Prometheus metrics listener. |
| `CACHE_HOST_SUFFIXES` | empty | Empty means all `GET` hosts are cacheable. Otherwise, cache only hosts containing one of the comma-separated suffixes. |
| `CACHE_HOST_SUFFIXES` | empty | Empty means all `GET` hosts are cacheable. Otherwise, cache only hosts containing one of the comma-separated suffixes. When non-empty, the plain-HTTP forward path also refuses targets outside the list with `403`; when empty (legacy mode), forwarding stays unrestricted. |
| `CONNECT_ALLOWED_SUFFIXES` | empty | Optional hostname substrings a `CONNECT` target must contain. Empty means any hostname on port 443 is allowed. The port-443 and local-IP-refusal rules always apply. |
| `CACHE_BLOCK_MODE` | `off` | `on` enables block-aligned caching; any other value (including unset) keeps the legacy exact-range path. See [Block-aligned mode](#block-aligned-mode). |
| `CACHE_BLOCK_SIZE_BYTES` | `8388608` (8 MiB) | Fixed block size for block-aligned mode. Ignored when block mode is off. |
| `CACHE_BLOCK_MAX_SPAN_BLOCKS` | `8` | Max blocks coalesced into one origin range fetch. Ignored when block mode is off. |
Expand Down Expand Up @@ -91,9 +92,40 @@ All lookup state comes from the in-memory index under the cache mutex, not
from filesystem stats, so `/cache/has` and eviction/size accounting can never
disagree about whether an entry exists.

## Security boundaries

The proxy binds a hostPort and serves unauthenticated requests from any pod in
the cluster. Two boundaries keep that safe in the managed-warehouse topology,
where tenants share a bucket with per-org path prefixes:

- **Cache keys are tenant-scoped.** The scope is the SigV4 access key ID from
the request's `Authorization` header
(`AWS4-HMAC-SHA256 Credential=<ACCESS_KEY_ID>/...`). STS access key IDs are
unique per issued credential set, so the scope separates tenants. The legacy
key is `sha256(scope + "\x00" + url + "|" + range)` and the block key is
`sha256(scope + "\x00" + url + "|blk|" + idx + "|" + blockSize)`. A warm
entry for one tenant — local or from a peer — is never served to a request
signed by another tenant; it simply misses and goes to the origin, where
S3 authorization applies. Requests without a parseable SigV4 header share
the empty-scope namespace, which is correct for public objects. The access
key ID is an identifier, not a secret; the secret key and the signature
never enter the key. Peer traffic carries only these opaque keys, so the
peer protocol is unchanged. Changing the key format invalidates all
existing on-disk entries: they become unreachable and age out by LRU.
- **The relay surface is locked down.** `CONNECT` only dials port 443, refuses
loopback, link-local, and unspecified IP literals, and optionally requires
the hostname to contain a `CONNECT_ALLOWED_SUFFIXES` entry. Port 443 for
arbitrary hostnames stays allowed because DuckDB reads external HTTPS
sources through the proxy while `http_proxy` is set globally. The
plain-HTTP forward path (non-`GET`, passthrough, non-cache hosts) forwards
only to hosts matching `CACHE_HOST_SUFFIXES` when that list is configured;
signed S3 traffic matches by definition. With no suffixes configured the
forward path stays unrestricted for backward compatibility.

## Block-aligned mode

The legacy cache key is `sha256(url|range)` — an exact match on the client's
The legacy cache key is `sha256(scope + "\x00" + url + "|" + range)` — an
exact match on the client's
`Range` header. DuckDB's Parquet reader rarely issues the same byte range
twice, even across repeat runs of the same query: footer probes, row-group
reads, and column-chunk reads all drift by a few bytes depending on prior
Expand All @@ -104,7 +136,8 @@ miss rate on a workload that should have been fully warm.

Block-aligned mode fixes this by keying the cache on fixed-size blocks of the
underlying object instead of the client's exact range. The key is
`sha256(url|blk|idx|blockSize)`, where `idx` is the block index (`start /
`sha256(scope + "\x00" + url + "|blk|" + idx + "|" + blockSize)`, where `idx`
is the block index (`start /
blockSize`) and `blockSize` is part of the key so a config change can't serve
a wrong-sized entry — old-size entries just become unreachable and age out
normally. A request is served by locating the blocks its range overlaps
Expand Down
29 changes: 16 additions & 13 deletions cmd/cache-proxy/block_serve.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,14 @@ func writeRangeNotSatisfiable(w http.ResponseWriter, objectSize int64) {
}

// fetchOriginSpan fetches blocks [firstIdx, lastIdx] of r.URL in ONE origin
// range GET and commits each block to the store under its BlockKey. Rewriting
// range GET and commits each block to the store under its BlockKey. scope is
// the tenant scope from TenantScope and is part of every block key. Rewriting
// the Range header is legal: DuckDB httpfs signs only
// host;x-amz-content-sha256;x-amz-date (see forwardUncached), so Range is not
// covered by the SigV4 signature. Content-Range is validated before any block
// is committed, and each selected block must contain exactly the advertised
// number of bytes.
func (p *CacheProxy) fetchOriginSpan(r *http.Request, blockSize, firstIdx, lastIdx int64) error {
func (p *CacheProxy) fetchOriginSpan(r *http.Request, blockSize, firstIdx, lastIdx int64, scope string) error {
timeout := p.originTimeout
if timeout <= 0 {
timeout = defaultOriginTimeout
Expand Down Expand Up @@ -160,7 +161,7 @@ func (p *CacheProxy) fetchOriginSpan(r *http.Request, blockSize, firstIdx, lastI
remaining := expectedBodySize
for idx := firstIdx; idx <= lastIdx && remaining > 0; idx++ {
blockBytes := min(blockSize, remaining)
size, err := p.store.PutStream(BlockKey(r.URL.String(), idx, blockSize), &exactLengthReader{
size, err := p.store.PutStream(BlockKey(scope, r.URL.String(), idx, blockSize), &exactLengthReader{
r: resp.Body,
remaining: blockBytes,
})
Expand Down Expand Up @@ -190,9 +191,11 @@ func (p *CacheProxy) blockPresent(key string) bool {
// serveBlockAligned serves a cacheable GET whose Range is an absolute
// bytes=start-end pair from block-aligned cache entries: local disk, then
// peers, then coalesced origin fetches for contiguous missing runs (chunked
// at maxSpanBlocks per origin request). Returns false when the request shape
// is not block-servable; the caller then runs the legacy exact-range path.
func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, rangeHeader string) bool {
// at maxSpanBlocks per origin request). scope is the tenant scope from
// TenantScope and is part of every block key. Returns false when the request
// shape is not block-servable; the caller then runs the legacy exact-range
// path.
func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, rangeHeader, scope string) bool {
requestStart := time.Now()
var peerDur, s3Dur, writeDur time.Duration

Expand Down Expand Up @@ -251,15 +254,15 @@ func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, r
// if only lo keyed the call, the loser would adopt the winner's
// (shorter) fetch result while believing its own longer span was
// covered — silently leaving trailing blocks unfetched.
flightKey := fmt.Sprintf("%s|%d", BlockKey(urlStr, lo, p.blockSize), hi)
flightKey := fmt.Sprintf("%s|%d", BlockKey(scope, urlStr, lo, p.blockSize), hi)
_, err := p.flights.Do(flightKey, func() (fetchResult, error) {
fetchStart := time.Now()
// Retry transient origin failures inside the flight so every
// waiter on this key benefits, and a brief origin blip is
// absorbed here instead of reaching DuckDB as a 502.
_, fetchSpan := proxyTracer.Start(r.Context(), "cache.origin_span_fetch")
fetchErr := p.retryOriginFetch(r, fetchSpan, func() error {
return p.fetchOriginSpan(r, p.blockSize, lo, hi)
return p.fetchOriginSpan(r, p.blockSize, lo, hi, scope)
})
fetchSpan.End()
s3Dur += time.Since(fetchStart)
Expand Down Expand Up @@ -300,7 +303,7 @@ func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, r
return true
}
for idx := firstIdx; idx <= lastIdx; idx++ {
key := BlockKey(urlStr, idx, p.blockSize)
key := BlockKey(scope, urlStr, idx, p.blockSize)
if p.blockPresent(key) {
if !flushRun(idx - 1) {
return true // error already written
Expand Down Expand Up @@ -376,7 +379,7 @@ func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, r
fetchStart := time.Now()
_, fetchSpan := proxyTracer.Start(r.Context(), "cache.origin_span_refetch")
err := p.retryOriginFetch(r, fetchSpan, func() error {
return p.fetchOriginSpan(r, p.blockSize, lo, runEnd)
return p.fetchOriginSpan(r, p.blockSize, lo, runEnd, scope)
})
fetchSpan.End()
s3Dur += time.Since(fetchStart)
Expand All @@ -388,7 +391,7 @@ func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, r
nOrigin += runEnd - lo + 1
}
for idx := firstIdx; idx <= lastIdx; idx++ {
if p.blockPresent(BlockKey(urlStr, idx, p.blockSize)) {
if p.blockPresent(BlockKey(scope, urlStr, idx, p.blockSize)) {
reverify(idx - 1)
continue
}
Expand All @@ -398,7 +401,7 @@ func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, r
}
reverify(lastIdx)
for idx := firstIdx; idx <= lastIdx; idx++ {
if !p.blockPresent(BlockKey(urlStr, idx, p.blockSize)) {
if !p.blockPresent(BlockKey(scope, urlStr, idx, p.blockSize)) {
slog.Error("Block still missing after presence re-fetch; failing closed.",
"url", urlStr, "block", idx)
http.Error(w, "block cache entry missing after re-fetch", http.StatusBadGateway)
Expand All @@ -425,7 +428,7 @@ func (p *CacheProxy) serveBlockAligned(w http.ResponseWriter, r *http.Request, r
}
}
for idx := firstIdx; idx <= lastIdx; idx++ {
reader, size, ok := p.store.openFile(BlockKey(urlStr, idx, p.blockSize))
reader, size, ok := p.store.openFile(BlockKey(scope, urlStr, idx, p.blockSize))
if !ok {
closeOpened()
blockFallbackTotal.WithLabelValues("entry_vanished").Inc()
Expand Down
Loading
Loading