Chained caches compound independent hit-rate rolls without key or tier semantics

[carlulsoe]

What happened

Chaining several generic caches produces a large, mechanically compounded improvement even when the caches have no distinct tier or workload semantics.

The cache behavior performs a fresh independent roll() < hitRate check at every cache and does not use the request key. Consequently, three caches configured for an 80% hit rate behave as three independent miss filters.

I reproduced this directly against dc9c1a07e573da5441c12e1a205ffe8678504dd5, constructing the complete topology before starting the engine so the separate client-generator bug does not affect the result. Averaged over 290 one-second samples:

Client      50.06 RPS
Cache 1     50.06 RPS
Cache 2     10.22 RPS
Cache 3      2.08 RPS
Database     0.36 RPS

This matches 50 * 0.2 * 0.2 * 0.2 = 0.4 RPS, but that result is misleading for generic caches over the same workload. A request reaches cache 2 precisely because it missed cache 1; the caches' hits are not normally independent discounts. The conditional hit rate depends on key popularity, tier capacity, content overlap, admission and eviction policy, and whether these represent browser, edge, regional, or application caches.

I would expect one of the following:

• cache tiers with explicit, topology-aware semantics over a shared key distribution;
• a conditional hit-rate model for downstream cache tiers;
• topology validation that warns about repeated generic caches; or
• at minimum, a visible assumption that cache hit rates are independent per-hop probabilities and arbitrary stacking can produce unrealistic results.

The current behavior makes it possible to improve a design by adding generic caches indefinitely, teaching the arithmetic of independent filters rather than the tradeoffs of cache hierarchies.

Preset

Custom system

Load

50 RPS on the only client

Browser and OS

Observed in the browser; reproduced independently in the pure simulation engine on Linux.

Anything else

This is separate from the client-generator duplication problem: the cache measurement above starts the engine with the complete topology and never calls setTopology during the run.

The screenshot URL is temporary and expires after 30 days.

[xevrion]

You are right, and thanks for the measured reproduction against a pinned commit.

Confirmed in the code: behaviour.ts:334 and behaviour-edge.ts:58 both do a bare ctx.roll() < hitRate and never look at req.key. So the caches are independent coin flips and chaining them multiplies, exactly as your numbers show. The engine already draws a key per request and other kinds use it (behaviour-data.ts shards on req.key), so the machinery for a key-aware cache is there and the cache simply does not use it.

On the four options you list, my instinct is the conditional hit rate over topology validation, because a warning tells someone their design is wrong without teaching them why, and the whole point here is the why. But I would rather not rush the model. A cache whose hit rate depends on key popularity and tier capacity is a real modelling decision and getting it half right would be worse than the current honest simplification.

So for now I am treating the last option as the immediate fix: say plainly in the docs and the inspector that hit rates are independent per-hop probabilities. Leaving this open as the discussion for the real model.

[csy20]

Taking the immediate fix you named: say in the inspector and the glossary that hit rates are independent per-hop probabilities.

Not touching the engine. behaviour.ts and behaviour-edge.ts still do ctx.roll() < hitRate and ignore req.key. The real key/tier model stays for the discussion.

What I will change:

• Cache (and CDN, if it still reads as a single-hop offload) blurb in Inspector.tsx
• A hint on the hit-rate control so stacking caches is not read as a hierarchy
• cache and hit-rate in src/content/glossary.ts

Copy only. I will open a PR against this issue.

[xevrion]

Yes, go ahead. That is exactly the scope I wanted, and leaving the engine alone is the right call.

Worth also covering the CDN blurb while you are in there, since it reads as a single-hop offload and has the same independent-roll behaviour.