Skip to content

round7-5: SFT QLoRA of the slot students (Qwen3-14B, gpt-oss-20b, Gemma-4-12B) on corpus v1 — per-slot and multi-slot adapters, exported at equal bits #3084

Description

@Xore

Part of #3079 (round 7). Plan §4 and §7, experiments T2 (per-slot) and T3 (multi-slot). GPU work; queues behind the cold re-run.

Students, and why these

The 20 GB envelope decides. Unsloth's published minimums: QLoRA 4-bit 14 B ≈ 8.5 GB, 27 B ≈ 22 GB, 32 B ≈ 26 GB; 16-bit LoRA 7 B ≈ 19 GB. MoE bases in 4-bit QLoRA are not recommended by Unsloth (bitsandbytes gap) and their 16-bit LoRA needs 60 GB+ — so gemma-4-26B-A4B, Ornith-1.0-35B (qwen3_5_moe) and Qwen3.6-35B-A3B are not trainable on this card; they are round7-7's requant targets instead. The one MoE that does fit is gpt-oss-20b through its native MXFP4 path (~14 GB).

student why fit
Qwen/Qwen3-14B the incumbent in all three slots (qwen3:14b, 62 / 60) — a fine-tune of the exact production weights is the cleanest possible A/B ≈ 8.5 GB QLoRA
gpt-oss-20b the only 20 B-class model trainable here; also the #2279 harmony-shape family that scored 0 as shipped — worth knowing whether training fixes it ≈ 14 GB MXFP4
Gemma-4-12B the OBLITERATED variant is joint top of the matrix at 64 / 66 from a 12 B — the best small base; pick the derestricted variant as the start point (refusals on malware evidence are a failure mode here) and record which ≈ 8 GB QLoRA

Use Unsloth's dynamic 4-bit variants (unsloth/<model>-unsloth-bnb-4bit) where published — they exist precisely to lose less in QLoRA.

Runs

  • T2: one adapter per slot per student — ghidra (Tier A + Tier B evidence shapes, S2 + S3), sessions (S1), revdeck (S3 + S4) — train_on_responses_only, r = 16–32, α = 2r, lr 2e-4, 1–3 epochs, packing, seq 8 k (32 k for the ghidra adapter with unsloth_tiled_mlp=True), use_gradient_checkpointing="unsloth", fixed seed; early stop on corpus v1's dev slice by schema-valid rate and dev rubric, not by loss
  • T3: one multi-slot adapter per student on the union — production runs one model per slot today, but a single model serving all three is a real deployment option and costs nothing extra to measure
  • T2′: the same as T2 starting from round7-4's CPT checkpoint, for the students where CPT finished
  • every run exported through export_to_ollama.sh at Q8_0, Q6_K, Q4_K_M and an imatrix IQ4_XS (round7-7's calibration set) so the fine-tune is compared with its base at equal bits, never confounded with quantisation

Deliverables

  • train.py + configs under analysis/ghidra/training/, run cards under docs/benchmarks/training/runs/ (loss, dev metrics, VRAM peak, wall-clock, tokens/s)
  • adapters + merged 16-bit under /mnt-1/training/runs/t2-*/, mirrored; optional private HF adapter repo (operator decision)
  • every artefact registered in round7-8's row list with its base-at-same-quant control
  • a Qwen3 note in the run card: thinking is off in the production contract (thinking: false); train non-thinking data with enable_thinking=False in the template and verify the exported Modelfile carries the same template — the Bump esbuild from 0.25.12 to 0.28.1 in /dashboard/frontend in the frontend-compatible group #1 cause of degraded exports per Unsloth's docs

Depends on round7-1, round7-2 green, round7-3. Feeds round7-6 and round7-8.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestllmLLM analysis workermlML worker and GPU scoring

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions