Repository navigation
Conversation
…aarch64 The plugin runs on this host: aleph-alpha-inference 1.0.0 via vLLM 0.31.0+cpu, TP=4, bf16 — greedy decode of the 8-prompt battery reproduced, primary goldens committed (tests/vllm/models/kolibri1_goldens_primary.json) with the generator and smoke test. The record's GPU-lease framing had the blocker wrong: the fp8 checkpoint is unusable on CPU outright (no fp8 ScaledMM kernel in vLLM's CPU backend), so the bf16 dequant was always mandatory; and the first bf16 conversion shipped an internally contradictory index (fp8 scale siblings left in place) that routed vLLM's MoE to its fp8 path — fixed by e7bba53. Also recorded: the pre-existing kolibri1_goldens.json is a pure-torch TRANSCRIPTION of the plugin's math, and the primary oracle disagrees with it on 6/8 first tokens and 3/8 full sequences — gates scored against it are calibrated to a proxy. The schema decision (generated-ids across all three implementations vs logits export) is flagged, not decided. Facts and recipe from the mistral.rs Kolibri-1 CPU work (PR 2488), handed over via /tmp/for_maki.md. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…100 rows The primary's own per-position distributions over the 8-prompt battery (202 positions, top-100 token/logprob pairs + the emitted token's logprob, teacher-forced via prompt_logprobs=100), with the generator. This is the adjudication artifact for the CPU numerics decision (localai-org#3438): the primary's top1-top2 margin is <0.05 nats at only 1.5% of positions (median 3.9), and its top6-top7 margin is <0.05 at 16.3% — and with these rows our own chains' positions can be scored against the primary's distributions directly, per numerics mode. Facts from the same run: vLLM CPU on this host (Neoverse-class, bf16+dotprod+i8mm in cpuinfo) is BUILT with BFDOT (-march=armv8.2-a+bf16+dotprod+fp16, ARM_BF16_SUPPORT) — the primary oracle's own runtime is bf16-dot class. BFMMLA is not in vLLM's march flags (armv8.2 pin). fp8-on-CPU: the ScaledMM dispatch covers floating types only — hard absence, not a build flag; the host has no fp8 silicon either. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-opening of the earlier pull request (closed during the merge run while its base moved) with its head rebased onto current main (it had been closed during the merge run while its base moved; the two commits are the same records, rebased).
The oracle record correction:
gateable = nocited a GPU lease. The plugin actually runs on THIS host — no GPU: aleph-alpha-inference 1.0.0 (049a6a7bd240) via vLLM 0.31.0+cpu, aarch64, TP=4, bf16; greedy decode of the 8-prompt battery reproduced. The real blockers were: (1) the fp8 checkpoint is unusable on CPU outright (vLLM's CPU backend has no fp8 ScaledMM kernel, and the host has no fp8 silicon) — the bf16 dequant was always mandatory; (2) the first bf16 conversion shipped an internally contradictory index (fp8 scale siblings left in place) that routed vLLM's MoE to its fp8 path — fixed by the landed e7bba53 (row/kolibri-bf16-reference-fix). The full venv/build/serve recipe (torch==2.13.0+cpu from the PyTorch cpu wheel — the C++ extension links torch's ABI; VLLM_TARGET_DEVICE=cpu source build, ~6 min at -j64; TP=4 required because bf16 at 151 GB exceeds one 63.5 GiB node; gpu_memory_utilization is a memory FRACTION, re-tune per run) is in the record.The numerics-class finding for the #3438 decision (verified by disassembling vllm/_C.abi3.so): vLLM CPU on this host runs the bf16 linear layers through BFMMLA (11,328 instances, kernels
bf16fp32_mmla_8x3/8x12/6x4, bf16 x bf16 -> f32 accumulate), runtime-selected off HWCAP2_BF16 — no build-flag change needed. So the primary oracle's instruction-level class (bf16 multiply, f32 accumulate) is reachable in our C++ GEMM by dispatch. Caveat recorded: matching the primary BIT-EXACTLY would require matching oneDNN's tile packing order (the kernel symbol names name it); near-tie-level agreement needs only the class.Artifacts: tests/vllm/models/kolibri1_primary_topk_rows.json (the primary's per-position top-100 distributions over the 8-prompt battery — 202 positions, teacher-forced via prompt_logprobs=100; the adjudication artifact for the CPU numerics decision #3438) with the generator (scripts/gen-kolibri1-primary-topk-rows.py), the primary goldens (tests/vllm/models/kolibri1_goldens_primary.json) with the generator, and the smoke test. The pre-existing kolibri1_goldens.json is recorded as a pure-torch TRANSCRIPTION of the plugin's math — the primary disagrees with it on 6/8 first tokens and 3/8 sequences; gates scored against it are calibrated to a proxy (schema decision flagged in the record, not decided).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]