perf(metal): raise the command-buffer input budget for decode only

[inureyes]

Summary

On M1-M4 mlxcel raises MLX_MAX_OPS_PER_BUFFER to 1000 (#353) but leaves MLX's other commit trigger, the input budget MLX_MAX_MB_PER_BUFFER, at 40-50. That budget counts input elements (data_size() >> 20), not bytes, and decode reads the whole weight set every token, so a 4-bit 7B model splits each token into about 23 command buffers and the GPU idles about 40 us at every boundary (Metal System Trace: 89.5% busy).

Why decode only

Raising the budget globally to 1000 roughly doubles prefill peak memory on long prompts (2048 tokens: Qwen2.5 7B 6.0 to 12.8 GB, Qwen3-30B-A3B 19.8 to 36.2 GB), because a buffer's activations stay alive until it completes. It also hurts a synchronous decode step, which then cannot start on the GPU until the whole step is encoded: server decode for ignore_eos requests went from a steady 98-102 tok/s to 68-102. Device latches the budget once, so:

• a mlx/backend/metal/device.cpp overlay adds a process-wide override that needs_commit reads (three marked hunks to re-apply on pin bumps);
• DecodeCommandBufferBudget raises it around pipelined decode only: the four generate loops (not under MLXCEL_FORCE_SYNC) and the server's lookahead tick and prime. Prefill and synchronous decode steps stay on the device default;
• MLXCEL_DECODE_MB_PER_BUFFER tunes or disables it, and an explicit MLX_MAX_MB_PER_BUFFER still pins both phases. The hardware gate matches the op cap (M1-M4).

Measurements (M1 Ultra, 500-token prompt, 128 tokens)

Checkpoint	Decode, default	Decode, 1000
command-r7b 4-bit	101.4-103.3	109.4-110.3
Llama 3.1 8B 4-bit	99.0-99.2	103.4-105.1
Qwen2.5 7B 4-bit	98.2-100.1	106.4-108.1
Gemma 3n E4B 4-bit	59.7-60.6	62.1-62.4
Gemma 3 4B 4-bit	95.6-97.3	96.3-96.9
Granite 4.0 H Tiny 4-bit	103.3-104.0	113.4-113.8
Qwen3-30B-A3B 4-bit	72.9-76.0	90.0-90.4
Mixtral 8x7B 4-bit	51.0-52.1	61.9-62.8
Llama 3.1 8B bf16	34.2-34.7	40.1-40.4

With the switch as shipped, command-r7b, same binary on and off: CLI harness 104.2-104.4 to 110.1-111.2 tok/s; server lookahead median 110.0 to 118.5 (ten requests, two starts); server synchronous decode unchanged (100.6-101.1 off, 100.5-102.1 on); pp2048 peak memory unchanged (Qwen2.5 6.01 vs 6.07 GB, Qwen3-30B-A3B 19.75 vs 19.75 GB). Full record: docs/benchmark_results/metal-mb-per-buffer-m1ultra-2026-09-21.md.

Validation

Unit tests cover the overlay round trip, guard nesting and restore, and the resolution precedence. The full workspace gate (make verify-test, 11,411 passed), clippy and fmt pass on the branch this was developed on; on this branch, clippy, fmt and the new tests. Command-buffer boundaries do not change arithmetic, and greedy output for three prompts at 200 tokens is identical to main.

Not measured: M2-M4 other than Ultra, M5 (kept on MLX defaults), speculative decode loops (not wired yet), decode beyond 2048 tokens of context.

[inureyes]

The WebUI installed artifact failure is the WebUI Activity performance gate on the GB10 runner (lablup-dgxspark21) returning status: investigate: median decode degradation 2.1%, baseline CV 1.7%, paired range -3.4% to +6.5%. The same gate failed the same way on the main push run 35590282982 on the same runner (median -1.1%, baseline CV 5.8%, paired range -4.2% to +7.5%), so it flaps without this change. This PR is also inert off Apple Silicon: decode_mb_per_buffer() resolves to None there, so DecodeCommandBufferBudget never touches the override. Re-running the failed job once after the run completes.

[inureyes]

The re-run failed the same gate again, just over the 2% target: two-visible median decode degradation 2.14% (baseline CV 0.72%, paired range 0 to +5.9%), after 2.12% (CV 1.75%) on the first run. For comparison, the webui-activity-performance-Linux evidence from the four most recent passing runs of this job reports 1.30%, -0.80%, 1.67% and 0.80%, and the failing main push run 35590282982 reported -1.11% with CV 5.85%.

I could not find a path by which this PR affects that measurement on the GB10 runner. Off Apple Silicon decode_mb_per_buffer() resolves to None once (cached), so each decode tick only constructs and drops an inactive guard; the device.cpp overlay is copied but not compiled without the Metal backend; the bridge setter is an empty stub. I am not re-running it again. If the gate needs to be green before merge, a comparison run of main on the same runner would settle whether 2.1% is this runner's current level.