Repository navigation
llama-perplexity mis-scores GLM-4.7-Flash at -ub 1 (PPL 264.97 vs 10.74; generation is fine) #337
Description
CPU backend mangles GLM-4.7-Flash at -ub 1 (PPL 264.97 vs 10.74 at the default ubatch)
Found while settling engine#310's HRX-vs-CPU perplexity comparison.
Build: a fresh directory configured with the engine's own ExternalProject arguments (Release, amdclang), AMD's core plus our ggml-hrx.
llama-perplexity -m GLM-4.7-Flash-Q4_K_M.gguf -f wiki.test.raw -c 4096 --chunks 2 -fa 1 -ngl 0 -ub 1
-> Final estimate: PPL = 264.9719 +/- 17.62758
llama-perplexity ... -ngl 0 (default ubatch)
-> Final estimate: PPL = 10.7414 +/- 0.50110
Without -fa 1 the -ub 1 arm is worse (387.5591 +/- 25.19623) and its log reads resolve_fused_ops: Flash Attention not supported, set to disabled.
It is not the ubatch setting in general, and not HRX:
- MiniCPM5-1B-Q4_K_M, CPU,
-ub 1-> 26.9349 +/- 1.31485 — plausible for a 1B model, so the CPU's single-token path is fine there. - GLM-4.7-Flash-Q4_K_M, HRX,
-ub 1-> 10.7208 +/- 0.49991.
So the collapse is CPU-specific and GLM-4.7-Flash-specific (deepseek2/MLA). The two arms differ only in the backend.
Why it matters beyond correctness: the natural A/B for engine#310 is "HRX decode at -ub 1 versus CPU at the same settings" — and at those settings the reference is the broken arm (264.97), which makes the comparison look like HRX is fine while nothing about HRX has been checked. Worth pinning down which CPU decode path for this architecture loses it.
Logs on the strixhalo box: /tmp/pplb_cpu_fa.log (-ub 1, FA on), /tmp/cc_glm_cpu_defub.log (default ubatch), /tmp/cc_mini_cpu_ub1.log (small model control).
Activity
bong-water-water-bong commentedon Oct 6, 2026
Correction: generation is fine at
-ub 1— it is the perplexity scoring that is wrongSharper characterization, because my title overstates it.
Serving the same model on the CPU at
-ub 1produces the same text as the default ubatch:-ub 1 -fa 1' Paris. Question: What is the capital'' Paris. Question: What is the capital'' Paris.'' Paris.'No NaN in either log. So the CPU is not mangling the model — but
llama-perplexityat-ub 1scores that same model at 264.97 where the default ubatch scores 10.74.That moves the defect from "the CPU backend" to "
llama-perplexity's scoring at-ub 1" — a much narrower claim, and it changes the practical advice: generation at-ub 1is fine; do not use-ub 1for a perplexity reference arm.Sweeping
-ubin {1, 2, 4, default} on a short corpus, plus HRX at-ub 1as a control, to see whether only the single-token ubatch is affected. If it is only-ub 1, that points at the per-token scoring loop rather than anything model-specific — and MiniCPM5-1B scoring sanely at-ub 1(26.93) would then need its own explanation, since it argues against a purely shape-independent loop bug.[-]HRX: CPU backend mangles GLM-4.7-Flash at -ub 1 (PPL 264.97 vs 10.74 at the default ubatch)[/-][+]llama-perplexity mis-scores GLM-4.7-Flash at -ub 1 (PPL 264.97 vs 10.74; generation is fine)[/+]bong-water-water-bong commentedon Oct 6, 2026
The
-ubsweep refutes my own theory: it is the run configuration, not the ubatchEvery CPU ubatch is broken at that configuration, 512 included — so "only
-ub 1is affected" is wrong and the ubatch is not the variable I claimed in my previous comment. What is consistent:-c 4096 --chunks 2,-ub 1-c 4096 --chunks 2, default ubatch-c 2048 --chunks 1,-ub 512-c 2048 --chunks 1,-ub 1So CPU perplexity depends on the context/chunk configuration, not just the ubatch:
-c 4096 --chunks 2scores correctly at the default ubatch and badly at-ub 1, while-c 2048 --chunks 1is bad at every ubatch. HRX scores sanely in both (10.72, 6.84).Retracting the "only the single-token ubatch" claim from my previous comment. The reliable statement is weaker but still useful: on the CPU, perplexity is configuration-sensitive in a way HRX is not, so a CPU
-ub 1arm cannot serve as a reference — which is the practical point that led here (it silently corrupted the round-84 comparison). The mechanism is open.I am not going to keep guessing at the mechanism. If the intent is to fix rather than document it, the cheap starting point is the
-c 2048 --chunks 1 -ub 512gap — 948.64 against 6.84 in about two minutes — rather than the 4096-token configuration I opened with.bong-water-water-bong commentedon Oct 6, 2026
The discriminator is
-fa 1on the CPU, not the ubatchThe full matrix came back pointing somewhere neither of my earlier framings did:
Against the single CPU run in this whole investigation that produced a sane figure:
-c 4096 --chunks 2, no-fa(FA auto-disabled; log: "Flash Attention not supported, set to disabled")-c 4096 --chunks 2 -fa 1-c 4096 --chunks 2 -ub 1, no-fa-c 4096 --chunks 2 -ub 1 -fa 1Every
-fa 1CPU run is 948-1715; the-fa-less run is 10.74; HRX with-fa 1is fine (6.82, 23.15, 10.72). So the CPU's flash-attention path is what corrupts these numbers — and that also explains why-ub 1looked like the culprit, since every run I built that claim on had-fa 1on the command line.Third framing on this issue, so I am running a 2x2 (default ubatch /
-ub 1x-fa 1/ no-fa) on the CPU plus an HRX control before claiming it. If it holds, the defect is neither "perplexity mis-scores" nor "the ubatch": it is the CPU flash-attention kernels — and my round-84 advice was pointing at the wrong flag, since the one to avoid is-fa 1, not-ub 1.bong-water-water-bong commentedon Oct 6, 2026
Complete 2x2: both flags corrupt the CPU perplexity, independently
Same model, same corpus, same context — only the two flags changed:
-fa-fa 1-ub 1, no-fa-ub 1,-fa 1Four configurations, four different answers, and only default ubatch + no
-fais right — 10.74, which HRX independently agrees with at 10.72. Each flag corrupts the number on its own:-fa 1at the default ubatch gives 1014, and-ub 1without-fagives 387.5591 — the latter reproducing my very first measurement to the digit, so it is stable rather than noise.That closes the loop on how this issue went. Every configuration I called broken had
-fa 1or-ub 1on the command line, and the single run that produced a sane figure had neither. My three framings were each one variable short because I kept changing one flag at a time; the matrix with both axes is what settles it.The practically useful statement: on the CPU,
llama-perplexityis only trustworthy at the default ubatch with flash attention off. That is the arm to use for a CPU reference — and HRX's numbers (10.72, 6.82, 23.15) are stable across configurations where the CPU's are not.The mechanism is now well-posed rather than vague: two independent flags, each changing which tokens contribute to the score — the confidence intervals scale with the configuration (+/-0.50 versus +/-17 to +/-51), so the number of scored tokens is what moves. That is a far better starting point than "the CPU mangles GLM".
bong-water-water-bong commentedon Oct 6, 2026
The timing corroborates it:
-fa 1makes the CPU 30x slower and wrongOne more signal, free to collect because the perplexity logs already print it:
-fa-fa 1Same model, corpus, context and batch size. Enabling flash attention makes the CPU pass 30x slower and wrong — a much more specific symptom than "the number is off", because it says the CPU FA path is active and pathological rather than that the scoring merely reads a different subset of tokens. HRX does the same work at 249.10 s/pass and gets the right answer (10.72).
So for whoever fixes it: the CPU flash-attention path is both slow and numerically wrong on this model, and the perplexity figure is only the messenger. The scored-token-count explanation I offered a comment ago is now the weaker one — a 30x cost says the work is being done, badly.
The practical advice is unchanged and now better supported: for a CPU reference, run
llama-perplexityat the default ubatch with flash attention off (10.7414 here, and it agrees with HRX at 10.72); do not use-fa 1or-ub 1.