Add generated quant-kernel JIT and accelerate Gemma4, BGE-M3, and GLiNER2

[timkaye11]

Summary

This PR adds an evidence-gated GPU kernel pipeline and applies it across Gemma4, BGE-M3, and GLiNER2.

• Adds shared descriptor/IR-based kernel planning with backend-specific Metal and CUDA lowering, checked-in artifacts, route telemetry, conformance evidence, guarded runtime JIT modes, caching, and safe fallback.
• Expands Gemma4 QAT with generated GPU routes, batching, long-context KV fixes, MTP/speculative decoding, refreshed GGUF compatibility, and CUDA/Metal release gates.
• Adds antfly inference chat: model aliases and pulling, streamed multi-turn generation, context trimming, slash commands, interruption, and per-turn statistics.
• Accelerates BGE-M3 Metal inference with resident encoder/projection execution, packed QKV, fused FFN/attention paths, and dense-f16 mirrors.
• Accelerates GLiNER2 Metal inference with resident DeBERTa slots, tuned disentangled attention, device-side concatenation, and a resident span head.
• Refreshes inference OpenAPI surfaces and the Go, Python, and TypeScript SDKs.

Benchmarks

Local, unattested Apple measurements use ReleaseFast builds. Decode measurements exclude prefill.

Workload	Antfly	Comparator	Result
Gemma4 E4B Metal, 256-token decode	24.87 tok/s plain; 31.27 tok/s MTP k=1	llama.cpp b8990: 28.69 tok/s	MTP +25.7% vs plain and +9.0% vs llama.cpp
Gemma4 E4B Metal, canonical 2K+300	15.86 tok/s	pinned llama.cpp b10182: 28.93 tok/s	Correct, with remaining long-context optimization headroom
Gemma4 E2B Metal sampled chat	52.2 tok/s	previous path: 5.1 tok/s	~10.2× improvement; greedy remained 80.6 tok/s
Gemma4 E2B CUDA, L4, 2,051+300	740.8 ms TTFT; 110.2 tok/s decode	llama.cpp: 329.5 ms; 114.4 tok/s	Decode within 4%; total-latency ratio 1.173
BGE-M3 Metal, real-text batch 16	12.52 embeddings/s, ~857 input tok/s	native CPU was 27× slower at batch 1	Cosine 0.9973 vs llama.cpp on the same GGUF

The short Gemma4 MTP result uses qualified experimental opt-ins; the canonical long-output and CUDA measurements identify remaining prefill and long-context headroom rather than claiming universal superiority.

GLiNER2 shape	Metal	Native CPU	Fastino MPS
215 tokens, batch 16	1248.8 ms	1662.6 ms	1240.8 ms
430 tokens, batch 16	2375.3 ms	3389.7 ms	3384.3 ms

GLiNER2 Metal improved approximately 6.9× over its pre-optimization batch-16 baseline and is 33–43% faster than native CPU at these shapes. Entity sets matched native exactly at batches 1 and 16; one batch-8 run produced three additional near-threshold entities with ≤0.3% score-sum drift.

Validation

• A 2,003-token Gemma4 prompt produced exact target/live/MTP agreement for 64 generated tokens and natural-EOS agreement at 128 tokens.
• Generated Gemma4 Metal smoke recorded 343 generated quant dispatches—342 Q4_0 and one Q8_0—with zero missing artifacts, wiring failures, or decode fallbacks.
• Metal long-output contracts passed 42/42 cases; A/B benchmark contracts passed 13/13.
• The Metal inference suite passed 2,548 tests with 14 skipped and zero failures.
• CUDA validation includes balanced paired L4 benchmarks, generated-artifact freshness, route telemetry, batching, quality, and release-contract gates.
• CI covers Zig base/E2E builds, generated-source checks, Go/Python/TypeScript SDKs, OpenAPI generation, and reproducible release packaging.

[ajroetker]

Seems like we probably want this to be an e2e test or not in ci