Research: quantize bigger bases to fit the Supermicro — is large-model-low-bit a real gain over the current slots?

[Xore]

Upstream reference: https://github.com/aliaihub/awesome-hermes-usecases/blob/main/usecases/dev-local-model-quantization.md

Background

Every LLM slot decision in this stack (#144 -> #158 -> #568) chose among models as shipped — an Ollama library tag or a pre-built HF GGUF at whatever quant its uploader picked. The open rounds (#1795, #1947, #1804) widen the menu but stay on the same axis: existing artifacts, existing quant levels.

This issue is the other axis, the one the referenced write-up is about: start from bases too large to serve at usable precision, generate our own quantized variants, and test whether any of them lands inside the 20 GB envelope faster and/or more accurate than the smaller models we run today. The hypothesis in one line: a base one-to-two size classes above qwen3:14b (27 B–35 B, incl. the A3B MoE class), pushed down a quantization ladder we control, may trade bits we don't need for parameters we're currently leaving on the table.

Why it's genuinely still open here:

• The budget was wrong when the slots were chosen. The original matrix ran under an assumed 8 GiB; Confirm actual GPU topology on the homeserver: RTX 4000 Ada 20GB + RTX 4000 (Turing) 8GB, heterogeneous not identical #602 confirmed 20475 MiB on the RTX 4000 Ada and Re-evaluate LLM model selection (ghidra/revdeck/sessions) now that GPU VRAM is confirmed ~20GB, not 8GB #568 reopened selection on that basis — but scoped itself to re-comparing the same as-shipped candidates, never to manufacturing new quant points.
• Published artifacts cluster where other people's cards need them. 24–35 B class models are unservable at Q4/Q5 on common 12–16 GiB GPUs, so the most interesting intermediate levels for a 20 GB card frequently aren't published by anyone. Someone has to build them; that is cheap for us because…
• The machine for this already stands.

Harness we already have (this is applied, not built)

analysis/ghidra/benchmarks/model-quant-benchmark/ — born for the #847 REx86 sweep — implements the complete loop, per model:

1. download HF snapshot → convert_hf_to_gguf.py → f16 master (snapshot deleted afterward);
2. llama-quantize down each requested level, skipping anything that already exists (resumable per model and per level);
3. serve each level via llama-server, score with engine-benchmark/corpus_eval.py on the standard corpus slice;
4. format results to Markdown with gen_answers_md.py.

Supporting pieces: probe-gpu-capabilities.py for a before/after capability baseline (#637 playbook), and #1947's measurement discipline (error bars, transcripts kept, ≥3 seeded repeats) so the output is comparable with the matrix rebuild rather than another incomparable table.

Two caveats carried forward:

• Runtime gap. The ladder harness pins llama.cpp; production serves via Ollama (pinned to the Ada in ops: pin Ollama to the RTX 4000 Ada GPU only; Rev·Deck currently can't reach Ollama #1539). Engine-choice economics were settled separately in Research: does llama.cpp or vLLM offer real benefits over Ollama, even accounting for a rearchitecture? #598 — any winning quant point must be re-verified under Ollama before touching a slot.
• Scorer noise is known. rubric safe_strcpy: 'not vulnerable'/'not exploitable' contain forbidden terms — the correct conclusion scores worse and flags a phantom injection failure #2037 (negation-blind forbidden-term matcher) and engine-benchmark: empty completions earn a scorer point (corpus_eval.py); errors silently shrink both scripts' pct denominators #2050 (empty completions earning points) are open; treat them as shared error bars between arms rather than blockers — they hit every model equally at temperature 0 seeds.

Target hardware

Supermicro homeserver (xore@192.168.42.250), Ada 20 GB as serving card (#602, #1539), production contexts as loaded today (8k; 32k where the slot uses it — see #568's context-ceiling question).

Run shape

1. Base selection — at least two bases ≥ 24 B instruct (one dense, one MoE/A3B arm for latency), prioritising models whose published quants either overflow 20 GB at usable widths or stop at unusably low bits — those are exactly the levels worth generating ourselves. Existing shipped-quant candidates stay in Benchmark round: new derestricted HF candidates (XORTRON, GLM-4.6-Derestricted, gemma-4 heretic) within the 20GB VRAM budget #1795/One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947; those rows feed step 4's comparison, not duplicated here.
2. Ladder generation — f16 master → quant levels from the largest that fits 20 GB including KV-cache headroom at production context, downward through Q2-class. Record per-level file size, nvidia-smi residency, and offload split.
3. Evaluation — every level that loads, regardless of expectation: prefill/decode tok/s, VRAM+KV headroom at 8k and 32k, corpus_eval score × ≥3 seeded repeats, transcripts kept. Levels that spill to RAM are measured and reported, not skipped; timeouts are recorded results (One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947 rule).
4. Comparison plane — winners must beat (a) today's production slots as deployed and (b) the best shipped-quant rows for the same size class from Benchmark round: new derestricted HF candidates (XORTRON, GLM-4.6-Derestricted, gemma-4 heretic) within the 20GB VRAM budget #1795/One benchmark, every model, every aspect: rebuild the model matrix on current hardware with error bars #1947, on the agreed metric: quality within noise at equal-or-better decode tok/s, or better quality at equal throughput.
5. Verdict as a record — a winner promotes only via a fresh approval record with re-run gates per Re-evaluate LLM model selection (ghidra/revdeck/sessions) now that GPU VRAM is confirmed ~20GB, not 8GB #568's audit-trail rule (never patching approved-models.json in place) + follow-up issue for deployment. A null result closes this issue with the negative answer and its numbers — equally valuable: it proves the size ladder adds nothing over shipped quants and stops the hypothesis being re-raised.

Out of scope

• Changing serving runtime fleet-wide (Research: does llama.cpp or vLLM offer real benefits over Ollama, even accounting for a rearchitecture? #598 territory).
• Fine-tuned/adapter models — local-llm: evaluate the other 6 REx86-paper fine-tuned models against #159's corpus and benchmark them against each other #847's machinery is reused, but that thread stays halted (Wire the approved REx86 adapter into a real Rev·Deck production path (GGUF or native PEFT) #356 track).
• Slot reconfiguration itself — falls out as follow-ups only if a candidate clears step 4.

Related: #568, #602, #598, #637, #847, #356, #1539, #1795, #1947, #1804

[Xore]

Which models actually need requantizing — measured, with the arithmetic

Worked out while launching the full-roster ghidra-slot sweep under the
cold-slot protocol (#1805-c / #1947 part 4). Three distinct categories, and
only one of them is a quantization problem.

Host envelope

• GPU: RTX 4000 Ada, 20475 MiB (~21.5 GB).
• Host RAM: 91 GB total, ~70 GB available (the live stack holds the rest).
• /mnt-1: 1.7 TB free — disk is not a constraint. /var is at 89%, so
  conversion work must stay on /mnt-1.
• KV cache at the benchmark's served CONTEXT 32768 costs roughly 3–5 GB on a
  14–27B model (measured: qwen3:14b is 9.3 GB on disk and 14 GB resident).
  So a fully-resident model has a weights budget of ~16–17 GB, not 21.5.

1. Already-pulled models that spill to CPU — the real requantization work

These have published GGUFs we already hold, but do not fit the weights budget
at CONTEXT 32768, so they run partly on CPU. Confirmed by cost: Ornith-35B
is 22 GB on disk and takes 6.7 min/run against gemma-4-26B-A4B's 2.1 for
identical work.

A ollama ps sampler is now recording served size and CPU/GPU split for every
model as the sweep loads it (/mnt-1/benchmarks/vram_samples.tsv), so this
list will be empirical rather than estimated. Certain member so far:
Ornith-1.0-35B-uncensored-heretic:Q4_K_M (22 GB). Likely: Seneca-32B (19 GB),
gemma-4-31B-qat (18 GB), Qwen3.6-27B-MTP (18 GB), and the 17 GB 27B cluster.

These can be requantized cheaply and without any download, via
llama-quantize --allow-requantize on the GGUF already in Ollama's blob store
— the path rex86_backfill_extra_quants.sh already documents. Caveat worth
stating up front: requantizing an already-quantized GGUF is lossy-on-lossy and
strictly worse than going from f16, so a Q4→Q3 row is a fit-and-cost probe,
not a clean quality datapoint. A proper answer re-downloads and converts.

Important framing: this does not replace the offload rows. #1795 explicitly
allows RAM offload and makes full residency a reporting item, not an acceptance
gate. So a spilling model gets two rows — as-published-with-offload, and
self-quantized-to-fit — which is exactly the comparison this issue asks for.

2. The two "rejected on size" candidates — requantization does not save either

Both were deferred from #1795 as size rejections. Neither has a published GGUF,
so both would need the full download → f16 → quantize pipeline. Params from the
HF API:

model	params	bits/param needed to fit ~16 GB	verdict
ArliAI/GLM-4.6-Derestricted-v3	356.8 B	0.36 bpw	impossible
darkc0de/XORTRON.CriminalComputing.LARGE.2026.3	122.6 B	1.04 bpw	impossible

The floor for a usable quantization is ~1.6 bpw (IQ1_S), and quality there is
already badly degraded. Neither model can be made GPU-resident on this card
at any quantization level. That is a hardware fact, not a tuning question,
and it should be recorded as a measured rejection rather than left as an open
to-do.

With RAM offload the picture differs between them:

• XORTRON (123B) is viable but expensive. Q4_K_M ≈ 74 GB, Q3_K_M ≈ 60 GB —
  both inside 91 GB of RAM. Cost to get there: ~245 GB f16 conversion from a
  51-shard snapshot, then hours per benchmark run at the single-digit tok/s
  that offload implies. Worth doing as the deliberate "RAM-offload row" Benchmark round: new derestricted HF candidates (XORTRON, GLM-4.6-Derestricted, gemma-4 heretic) within the 20GB VRAM budget #1795
  already scoped, but it should be planned as its own multi-day slice, not
  slipped into a roster sweep.
• GLM-4.6 (357B) is not viable here. Even IQ1_S is ~71 GB, which nominally
  fits 91 GB of RAM but leaves nothing for the live stack, and requires a
  ~700 GB f16 intermediate to produce. At that quantization on a 357B MoE the
  result would not be a meaningful measurement of the model. Recommend
  closing it as a hardware rejection with these numbers rather than carrying
  it as work.

3. Cybersecurity-BaronLLM — not a size problem at all

Part 2 recorded this as "gated repository; no download path in this
environment". The HF API now reports gated: auto — auto-approved on request —
and the artifact is a single 8B llama3.1 Q6_K GGUF, which fits the card
trivially with room to spare.

So it was never a quantization or capacity question, it is an authentication
one: the homeserver has no HF token (~/.cache/huggingface/token absent).
Adding one makes this model a normal roster row. Flagging it here because it is
currently filed under the same "couldn't run it" bucket as the two genuine size
rejections, and it does not belong there.

Recommended order

1. Finish the cold-protocol sweep of the 38 resident models (running; the
   as-published rows, offload included, are legitimate results).
2. From the sampler, take the confirmed spill list and add self-quantized
   fit-to-card rows for those — cheapest path to this issue's actual question,
   and no downloads.
3. Put an HF token on the homeserver and pull BaronLLM as an ordinary row.
4. Scope XORTRON as its own RAM-offload slice.
5. Close GLM-4.6 as a measured hardware rejection.

[Xore]

Picking this up as part of round 6 batch D.

[Xore]

Re-verified this session: no HF token on the homeserver yet (step 3 / BaronLLM blocked on that — needs xore to provide one), GLM-4.6's hardware-rejection numbers are unchanged and final. Steps 1/2/4 depend on the cold-protocol sweep, which is currently in-flight in xore's own draft #2641 — not duplicating that here. Parking with the existing 5-step order as the concrete unblock note; removing in-progress since nothing is actively being built in this session.

[Xore]

Review verdict: APPROVE (honest park) — re-verified live: ~/.cache/huggingface/ is absent entirely on the homeserver (not just the token file), and the GPU envelope is unchanged at 20475 MiB, so GLM-4.6's rejection arithmetic still holds and that sub-decision is final. #2641 does cover step 1's precondition. One correction to carry: #1947's "unblocks #2245" framing is not currently true — that sweep is producing invalid Tier B data and needs stopping for repair. Your actual blockers remain the HF token and #2641. No blocking items.

[Xore]

Ground-truth check before anything gets quantized — four of this issue's premises no longer hold

Walked the whole thing against the live homeserver today. The 5-step order in
the comment above is still the right shape, but most of its inputs are gone
and one of its tooling assumptions was never true on this host.

Tooling: what actually exists

needed for	state
llama-quantize — the requant-from-GGUF path (category 1)	✅ present, /usr/lib/ollama/llama-quantize inside ghidra-ollama-1, with --allow-requantize
convert_hf_to_gguf.py — the clean f16 path (the issue's actual hypothesis)	❌ absent. No llama.cpp checkout anywhere: not in PATH, not /opt, not /mnt-1, not ~
the rex86-eval container model-quant-benchmark/README.md assumes (llama.cpp built at /work/llama.cpp, models at /work/other-models/)	❌ does not exist. Only ghidra-ollama-1 is running
HF token	❌ ~/.cache/huggingface/ absent entirely, not just the token file
disk	✅ /var 6.4 T free (was 92 % full), /mnt-1 1.7 T free

So this issue's "Harness we already have (this is applied, not built)" is not
true today. The scripts are in the repo; their deployment is not on the box. The
cheap half — requantizing a GGUF we already hold — needs nothing new. The
expensive half — download → f16 → ladder, which is what the issue is actually
about — needs a llama.cpp toolchain that has to be stood up first. Best done as
a container (ghcr.io/ggml-org/llama.cpp:full carries both the convert script
and llama-quantize) rather than installing torch/transformers on the host.

Inputs: three of four are gone

input	state
requant_plan.txt — the 10 concrete rows	✅ survived the rebuild
vram_samples.tsv — the empirical spill list this plan was supposed to rest on	❌ wiped, in no mirror, and not re-derivable: the result JSONs record model, digest, qualification_request, scores — and no VRAM, no served size, no CPU/GPU split at all
the 10 source models	❌ all ten deleted
requant_sweep.sh — the driver	❌ never committed, lost (#2985)

The deletions are ongoing, and they are the sweep doing it

sweep_extra.sh:153 runs ollama rm "$TAG" on every tag it pulled. That was
correct when /var sat at 92 %; the header says so in as many words. With 6.4 T
free it is now actively destroying this issue's inputs as phase 2 proceeds —
every model phase 2 finishes is a model phase 3 might need. Checked each source
in requant_plan.txt against ollama list:

GONE  Trendyol-Cybersecurity-LLM-Qwen3-32B:Q8_0        GONE  Titus-CybersecurityLLM-v1.0:Q4_K_M
GONE  pki/nova-24b-cybersec:Q8                         GONE  philbert440/Qwen3.8-27B-Uncensored-Cyber:Q4_K_M
GONE  Huihui-CyberStrike-OffSec-35B-abl:Q4_K           GONE  HauhauCS/Gemma4-31B-QAT-...-MTP:Q4_K_M
GONE  RavenX-CyberAgent-...-RATH:Q4_K_M *              GONE  3DCF-Labs-org/ProximaA-1.0:Q4_K_M
GONE  oN8WeJbq5Jo7Pk/CVE-Heretic-Fable5-27B:Q5_K_M     GONE  MK4-Research/LOREA-cyber-v5.8:Q4_K_M

* recoverable — it survives under the #2738 local alias ravenx-cyberagent-35b:Q4_K_M (21 GB).

Re-pulling is not just slow (~200 GB), it is partly impossible: at least three
of the ten are in the failure ledger — Huihui-CyberStrike-OffSec-35B and
pki/nova-24b-cybersec both PULL_FAILED, and MK4-Research/LOREA-cyber-v5.8
never pulled at all.

Two things done today so this stops getting worse

/mnt-1/benchmarks/keep_and_sample.sh, running alongside the sweep and changing
nothing about it:

1. The sampler is back. ollama ps gives exactly what this issue needs —
   served size, CPU/GPU split, served context — and it only exists while a model
   is loaded, so it is sample-or-lose. Writing to vram_samples.tsv again.
2. Blobs are no longer lost on rm. ollama cp makes a second manifest over
   the same blobs — verified: cp then rm of the copy left the blob count
   unchanged at 39 and the original intact. So each pulled tag gets a
   keep/<slug>:src alias; the sweep's ollama rm drops its own manifest and
   the weights survive at zero extra disk. 10 aliases so far, logged to
   kept-aliases.tsv.

Deliberately not an edit to sweep_extra.sh — bash reads a running script
lazily by byte offset, and the alias names cannot collide with a roster entry, so
the sweep's own already local -> will not delete test is unaffected either way.
The permanent fix (a free-space floor instead of an unconditional rm) is a repo
change for the phase-2/2.5 boundary, not a mid-sweep edit.

First measurement since the sampler was lost

GLM-4.6-REAP-218B-A32B-Derestricted:i1-IQ1_S
  44 GB on disk -> 57 GB served @ ctx 32768 -> 65%/35% CPU/GPU

Confirms this issue's own KV arithmetic (~13 GB of KV on top of weights at 32k)
and shows the card holds ~35 % of it. This is the shape of datum the category-1
list needs, per model.

The undecided question that will break the comparison: which scorer

This issue's step 3 says evaluate with engine-benchmark/corpus_eval.py via
llama-server. Step 4 says a winner must beat the best shipped-quant rows from
#1795/#1947 — and those were scored by record_baseline.py at tiers A/B on the
pinned a99e765. Numbers from two different scorers are not comparable; that
is reason 6 of the six that made #1947 necessary.

requant_plan.txt's own third column is a new ollama tag
(trendyol-cybersec-32b-selfquant:iq4_xs, …), so the author already intended the
Ollama path. Make it explicit: requantize → ollama create → record_baseline.py
at both tiers on a99e765, so every self-quant row lands in the same matrix,
same rubric (14 cases, max 69), directly beside its as-published twin. llama-server

• corpus_eval.py answers a different question and should not be used here.

What actually has to be done

1. Decide the scorer (above). Everything else depends on it.
2. Rebuild the spill list empirically from vram_samples.tsv as phase 2
   finishes — anything not 100% GPU is a category-1 candidate. Do not re-derive
   it from disk sizes; the GLM row shows served ≠ on-disk by 13 GB.
3. Write and commit requant_sweep.sh (ops: #1947 phases 2.5/3/5 are unrunnable — gptoss_rerun.sh, requant_sweep.sh and slots_sweep.sh were never committed and did not survive the rebuild #2985): for each row, extract the
   source blob → llama-quantize --allow-requantize → ollama create the target
   tag → record_baseline.py --tier A and --tier B → record beside the
   as-published row. Resume-safe per model per level, like sweep_extra.sh.
4. Re-acquire what is re-acquirable, and revise the plan for what is not.
   Nine of ten sources need re-pulling; at least three cannot be pulled at all
   and should be replaced by rows the keep-aliases now preserve.
5. Put an HF token on the homeserver — still blocks BaronLLM (step 3 of the
   order above, an 8B Q6_K that fits trivially and was never a size question),
   and blocks every snapshot download the clean-f16 path needs.
6. Stand up the llama.cpp toolchain if the clean f16 ladder is in scope —
   without it this issue can only ever produce the lossy-on-lossy fit-and-cost
   probe its own comment says is not a clean quality datapoint. That is a real
   scope decision, not a detail.
7. XORTRON 123B — the IQ2_XXS pull failed today on a plain 429, which is
   transient. Worth a retry; it is this issue's deliberate RAM-offload row.
8. GLM-4.6 (357 B) — the rejection arithmetic is unchanged and final. Close
   that sub-decision with its numbers rather than carrying it.

Unchanged and still correct from the comment above: RAM offload is a reporting
item, not an acceptance gate, so a spilling model gets two rows — as-published
with offload, and self-quantized to fit. That pairing is the comparison this issue
exists to make.