Part of #3079 (round 7). Plan §4. Gated: opens only if round7-5 shows a gain worth scaling, and only after round7-8 has scored it.
The envelope
Unsloth's published QLoRA minimums put 27 B dense at ≈ 22 GB — two gigabytes over the RTX 4000 Ada — and 32 B at 26 GB. MoE bases are out for 4-bit QLoRA altogether (Unsloth: not recommended, bitsandbytes gap), and their 16-bit LoRA needs 60 GB+. Compute is local only (operator decision 2026-09-06), so "training bigger" means the VRAM edge, not a bigger card.
What to try, in order
- 27 B dense QLoRA at the edge, on the top dense family (
Qwen3.8-27B / Qwen3.6-27B, derestricted variant): use_gradient_checkpointing="unsloth" (activations offloaded to system RAM — this is where the two empty DIMM slots finally matter for training, not just for offload inference), unsloth_tiled_mlp=True, batch 1 with gradient accumulation, sequence 2–4 k, LoRA on attention only first, then add MLP if it fits. Record the peak from nvidia-smi per configuration; an OOM is a measurement — write it down with the config that produced it.
- Adapter-only export path: if a 27 B adapter trains but the 16-bit merge does not fit in RAM comfortably, merge on the CPU in the
llama.cpp image (convert_lora_to_gguf.py exists there) — Ollama's own ADAPTER import covers Llama/Gemma-2 adapters only, so merging stays the rule.
- A rented GPU is not in scope for this round; if the edge attempt fails and the local result justifies it, that is a new operator decision with a budget figure, filed as its own issue.
Deliverables
Depends on round7-5, round7-8 (first results). Gated on the operator.
Part of #3079 (round 7). Plan §4. Gated: opens only if round7-5 shows a gain worth scaling, and only after round7-8 has scored it.
The envelope
Unsloth's published QLoRA minimums put 27 B dense at ≈ 22 GB — two gigabytes over the RTX 4000 Ada — and 32 B at 26 GB. MoE bases are out for 4-bit QLoRA altogether (Unsloth: not recommended, bitsandbytes gap), and their 16-bit LoRA needs 60 GB+. Compute is local only (operator decision 2026-09-06), so "training bigger" means the VRAM edge, not a bigger card.
What to try, in order
Qwen3.8-27B/Qwen3.6-27B, derestricted variant):use_gradient_checkpointing="unsloth"(activations offloaded to system RAM — this is where the two empty DIMM slots finally matter for training, not just for offload inference),unsloth_tiled_mlp=True, batch 1 with gradient accumulation, sequence 2–4 k, LoRA on attention only first, then add MLP if it fits. Record the peak fromnvidia-smiper configuration; an OOM is a measurement — write it down with the config that produced it.llama.cppimage (convert_lora_to_gguf.pyexists there) — Ollama's ownADAPTERimport covers Llama/Gemma-2 adapters only, so merging stays the rule.Deliverables
export_to_ollama.shat the R3 ladder, scored by round7-8 against its base at the same quantDepends on round7-5, round7-8 (first results). Gated on the operator.