Correct the memory hierarchy: the earlier table measured the thread count, not the part

[AndreSlavescu]

The hierarchy constants I added a few commits ago were wrong, and an occupancy probe written to explain
something else is what showed it. That table put the fast level's capacity at 2 MB with a fourfold cliff
past it. There is no cliff at 2 MB.

Every size in it was measured at 64 threadgroups — enough to saturate DRAM at 128 MB, and starving an
8 MB working set by a factor of twelve:

working set	64 groups	512 groups
8 MB	196	2403
16 MB	160	1910
32 MB	131	447

So the "levels" between 4 and 32 MB were the thread count, not the memory system.

Corrected, sweeping occupancy per size and keeping the best

working set	best groups	GB/s	thread-count effect
4 MB	256	2435	13.1x
8 MB	512	2431	15.7x
16 MB	1024	2453	19.7x
32 MB	1024	1795	15.3x
64 MB	1024	448	3.8x
256 MB	1024	156	1.4x

Capacity moves from 2 MB to at least 16 MB, and the far side becomes a ramp rather than a cliff.

The DRAM constant survives, and that needed checking

The corrected sweep reads 156 GB/s at 256 MB against a recorded STREAMING_READ_GBPS = 120.6. A
constant that low would have reopened every decode conclusion built on it — MLX at ~120 would be at 77% of
the limit, not 100%.

At a 1 GB working set bandwidth converges to 125–131 GB/s whatever the occupancy (256 → 2048
groups). So 120.6 is right for true streaming, and the 156 was partial residency. The constant stands and
the decode roofline conclusions hold.

Residency is necessary, not sufficient

Also recorded: threadgroups needed before a resident set reads at resident speed — 256 at 4 MB rising to
1024 at 16 MB.

With a caveat that matters more than the numbers: not yet shown to be actionable for real kernels. The
affine decode matmul launches N / block_n groups — 35 to 280 at N=8960, an eightfold range straddling
those thresholds — and its configs measure within 3% of each other. Whatever binds that kernel, it isn't
this.

One test changed rather than made to pass

It asserted the drop past resident capacity was at least threefold, which encoded the cliff the corrected
measurement removed — it was testing an artefact of the harness. It now asserts the span and the ordering,
which is what survives. A second test pins that residency alone doesn't deliver resident bandwidth.

687 pass. Lint and vulture clean.

🤖 Generated with Claude Code