feat(moe): --moe-collect-stats, so expert-cache behaviour is measurable

[vcruz305]

The offload cache already collects everything you need to know whether the expert cache is
doing a good job, but none of it is reachable. decode_miss_stats,
decode_miss_stats_per_layer and decode_routing_stats are all implemented and none of
them has a caller. collect_stats exists on EngineConfig but has no CLI flag, and
collect_decode_freq has no flag at all. So the numbers are there and nobody can see them.

This adds --moe-collect-stats and wires the readout into the decode loop.

Both collectors sit behind the one flag on purpose. The miss rate alone tells you how often
you fetch, but not whether the fetch was avoidable. decode_routing_stats gives you an
oracle hit rate, which is the best any policy holding that many slots could do on the
routing you actually observed. That second number is what tells you whether eviction policy
is worth touching.

What it looks like on Ornith-1.5-35B-A3B IQ3_S, 8GB 4060, 2267 slots:

MoE cache (256 decode steps): miss_rate=0.259, active/layer=8.0, missing/layer=2.1
MoE cache worst layers: L0=0.498, L1=0.446, L2=0.441, L3=0.361, L4=0.326
MoE routing: oracle_hit=0.764 (realized 0.741), slots/layer=56.7, working_set=173.1/243,
             experts_for_90pct=92.3, norm_entropy=0.813

Realized 0.741 against an oracle of 0.764 says LRU is basically at the ceiling the routing
allows on this model, so the remaining misses are a capacity problem and not a policy
problem. Normalised entropy of 0.81 says the same thing from the other side, routing is
close to diffuse and there is not much skew for a frequency policy to exploit. I went
looking for this because of the cache policy discussion in #174, and I would rather people
be able to check it on their own model than take my word for it.

It also picks up batching effects. Same server at 4 concurrent requests:

MoE cache (256 decode steps): miss_rate=0.255, active/layer=19.7, missing/layer=5.0

active/layer goes from 8.0 to 19.7, so 4 tokens that could have touched 32 experts only
touch 19.7 distinct ones, and fetches per layer go 1.8 to 5.0 for 4x the tokens. That is
about 30% less PCIe traffic per token.

Worth being careful about what that buys, since it is less than the traffic saving suggests.
On the server's own decode counter the aggregate goes 45.8 to 53.4 tok/s median, so 1.16x,
against the roughly 1.4x you would predict if decode were purely PCIe-bound. So decode is
only partly bandwidth-bound at this batch size and there is a compute component that grows
with it. Being able to see both the traffic and the throughput is the point of the flag.

Notes on the implementation:

• The counters are captured into the decode CUDA graph, so this has to be a startup flag
  and cannot be toggled per request. It costs a little decode throughput, which is why it
  is off by default.
• Reading the counters needs a host sync, so the report fires every 256 decode steps rather
  than every step.
• reset_stats runs per window for the miss counters, but the routing histogram is left
  alone, since the oracle bound wants the whole run's distribution rather than one window's.
• Layers with zero steps in a window are excluded from the worst-layer ranking so they do
  not show up as perfect.

Tests cover the flag being registered and defaulting off, the report formatting, the hybrid
fetch/cpu split only appearing for the hybrid target, the idle window emitting nothing and
leaving the counters alone, and the zero-step layer exclusion.

[gdevenyi]

Tested on a Qwen3.8-Flash-Next deployment (512 experts top-10, 48 MoE layers, 8984 GPU slots = 36.6% residency, KV 262k). Works, cost is within noise.

	main 86214a9	+ this PR, --moe-collect-stats on
single-stream decode	58.6 tok/s	58.0 tok/s
8 concurrent, aggregate	132.8 tok/s	133.9 tok/s

Output during the single-stream phase:

MoE cache (256 decode steps): miss_rate=0.057, active/layer=10.0, missing/layer=0.6
MoE cache (256 decode steps): miss_rate=0.078, active/layer=10.0, missing/layer=0.8
MoE cache (256 decode steps): miss_rate=0.055, active/layer=10.0, missing/layer=0.6

Two notes. (1) Only the first line type appeared in this ~1.2k-token session; the worst layers / MoE routing: oracle_hit lines from the description never printed, so if those are gated on a longer window that is worth saying in the help text. (2) The number itself is useful: a 6-8% per-layer miss rate at 37% residency on a 512-expert top-10 model is far below the 63% a uniform router would give, which tells me straight away that the PCIe gather I see in traces (23% of the step) is a few hot experts churning, not capacity. That is exactly the question the flag is for.

Measured on a 2x RTX 6000 Ada (48 GiB, sm_89, PCIe Gen4, no NVLink) / 2x Xeon Gold 6526Y / 503 GiB box, CUDA 13.3, torch 2.11+cu130, sgl_kernel 0.4.5, model RadixArk/Qwen3.8-Flash-Next-NVFP4 (qwen4_exp, NVFP4 experts + bf16 dense). The PR cherry-picks cleanly onto main 86214a9; both runs also carry a local fix so --moe-cache-auto honours --num-tokens (it OOMs otherwise at this KV size). One GPU, one run each, same flags:

ft serve --model <RadixArk snapshot> --moe-backend offload --ple-backend pinned --num-tokens 262144 \
  --memory-ratio 0.94 --moe-prefill-hit-d2d --max-running-requests 8 --cuda-graph-max-bs 8

Single-stream = median of three 64-vs-256-token completion pairs, aggregate = 8 concurrent 256-token completions, TTFT on a ~1k-token prompt. Run-to-run spread of the baseline on this box is about +-4% single-stream.

---

Update 2026-09-05, merged into the deploy branch. Tried on 2 x RTX 6000 Ada (sm_89) serving Qwen3.8-Flash-Next (RadixArk NVFP4) at TP=2, offload backend, fp8 KV pool of 8 x 262,144 tokens, merged onto my deploy branch (main af71ba4 + #385/#386/#389/#392/#354 and ten other open PRs), tests run on the box, then put in production.

The readout on this box, single-stream decode at 70.9% residency (17,423 of 24,576 slots resident, the rest of the VRAM holding the eight-context fp8 KV pool):

MoE cache (256 decode steps): miss_rate=0.003, active/layer=10.0, missing/layer=0.0
MoE cache worst layers: L15=0.006, L16=0.006, L14=0.006, L17=0.005, L27=0.005
MoE routing: oracle_hit=0.999 (realized 0.997), slots/layer=363.0, working_set=337.2/439, experts_for_90pct=122.6, norm_entropy=0.757

That is the number I was missing: it says why decode here is flat from 100% residency down to 70.9% (LRU sits at the oracle bound, the working set is a third of the slots). Merged clean, carried in production behind the flag.