round7-4: continued pretraining with Unsloth on domain text — decompiler output, sessions, RE corpora — before SFT

[Xore]

Part of #3079 (round 7). Plan §7, experiment T1. GPU work; queues behind the cold re-run via chain_round7.sh.

Why

Every base in the matrix learned decompiler output and honeypot transcripts only incidentally. Unsloth's continued-pretraining path (UnslothTrainer, embed_tokens + lm_head in target_modules, a separate embedding_learning_rate) is the cheapest way to move a base's language toward Ghidra pseudocode and attacker shell sessions before it is taught the jobs in round7-5. Whether it helps on this benchmark is the question; the answer is a row, either way.

Run

• base: the round7-5 students (Qwen3-14B, gpt-oss-20b; Gemma-4-12B if VRAM allows the embedding modules) — same weights the SFT runs start from, so CPT-then-SFT vs SFT-only is a clean A/B
• data: corpus v1 S6 (round7-3): decompiler output, sanitised sessions, REx86 text, permissively-licensed RE corpora; target 50–200 M tokens, measured not guessed
• settings: QLoRA 4-bit, r = 64–128 on all linear + embeddings, embedding_learning_rate ≈ 1/10 of learning_rate, use_gradient_checkpointing="unsloth" (offloads activations to system RAM — 93 GB today, 128 GB once the DIMMs land), sequence length 8 k, packing on; unsloth_tiled_mlp=True for any 32 k-context run
• throughput: record tokens/s on this card for each student in the run card — that number decides how much CPT is affordable, and nobody has measured it here

Deliverables

• config under analysis/ghidra/training/configs/cpt-*.yaml, run cards (loss curve, tokens, wall-clock, VRAM peak) under docs/benchmarks/training/runs/
• adapter + merged 16-bit under /mnt-1/training/runs/t1-cpt-<student>/, mirrored per the plan's durability rule
• each CPT checkpoint exported through round7-1's export_to_ollama.sh at Q4_K_M and scored raw (before SFT) by round7-8 — a CPT-only row tells whether the language shift alone moves the rubric
• handed to round7-5 as an alternative starting point (CPT-then-SFT row)

Depends on round7-1, round7-3. Feeds round7-5 and round7-8.

[Xore]

Working conventions for this issue: grit (claim the symbols you touch with grit claim -a issue-<N>-coder … where N is this issue number, edit only in .grit/worktrees/issue-<N>-coder/, grit done to merge, grit init after the merge if you added files), rtk on every shell command (rtk proxy only for raw output; rtk gain --history in the closing report), and the gh claim/comment/PR/merge conventions. Full text: epic #3079 → "Working conventions", and the plan's §15.

[Xore]

Claimed for round 7 (orchestrator, 2026-09-06). Pre-flight done: plan of record merged via #3089; grit initialised at repo root; no GPU work starts until the cold re-run completes.

[Xore]

Stage 4 GPU leg started 2026-09-11. GPU verified free (RTX 4000 Ada 0 MiB, no procs). Training via unsloth stack; /mnt-1 paths substituted /var per #3162.

[Xore]

Round-11 status: still executing on the GPU — llama-imatrix export active (5h+ at time of round-11 plan, 89% util, host load 29). Operator-owned; not touched by any agent this round. Queued behind it: #3084, #3085.

[Xore]

Round-11 follow-up: the corrected export leg finished on 2026-09-13. What changed since the 09-11 comment (which reported imatrix still active):

• f16 export complete → export/model-f16.gguf (29.5 GB, 16.0 BPW, done 02:19 local)
• imatrix complete → export/imatrix.dat (18:37 local)
• Q4_K_M quant complete → export/model-Q4_K_M.gguf (9.0 GB, 4.87 BPW, quantize 177 s, done 18:39 local)
• register step FAILED → ABORT: no Modelfile at /var/training/runs/t1-cpt-qwen3-14b/Modelfile (16:40:00Z). Two issues: the check still uses the hardcoded /var/training path (the /mnt-1 substitution per 54 files still reference /mnt-1 after the #3159 decommission — including a live unsloth bind mount and every benchmark driver's default BASE #3162 did not cover this line), and no Modelfile exists anywhere under /mnt-1/training — train.py/Unsloth never wrote one. A base.Modelfile (13 KB) was emitted into export/ alongside the gguf, so ollama create t1-cpt-qwen3-14b-q4_k_m can be driven from it once the path/modelfile wiring is fixed.
• GPU now idle (0% util); nothing left running.

Net: export + quant chain done, register to ollama is the single remaining blocker; #3084/#3085 still queued behind it.

[Xore]

Register unblocked — model is live. The earlier ollama create (2026-09-12 07:22Z, "unexpected EOF") had actually landed a manifest, but it pointed at the pre-corrected blob. Re-created from the corrected export:

• model-Q4_K_M.gguf (9.0 GB, corrected imatrix) copied into the ollama container (ghidra-ollama-1), base.Modelfile template kept, FROM rewired to the corrected gguf
• ollama create t1-cpt-qwen3-14b-q4_k_m → success (new model layer sha256:b3480560…, new ID 5c7bd7472405, manifest 2026-09-13)
• Smoke test: generation returns correctly (CPT-REGISTER-OK)

GPU idle otherwise; #3083 next deliverable = B2 run-card (CPT-only row via round7-8 scoring), then review/ship under the locked pipeline law. #3084/#3085 remain queued.

[Xore]

CPT-only row scored (pre-SFT, raw round-7 pin, current scorer). Registered tag t1-cpt-qwen3-14b-q4_k_m (model ID 5c7bd7472405, corrected Sep-13 blob):

• Tier A: 74/83 = 89.2% (N=2, both runs identical)
• Tier B: 75/83 = 90.4% (N=2, both runs identical)
• No escalation needed (run1==run2 both tiers), cold protocol (ollama stop before each run), transcripts + result JSON under /var/benchmarks/round7/cpt/.
• Throughput on this card: ~32-33 tok/s wall-inclusive (4 runs: 32.0/32.4/33.0/33.1), 17 cases × 512 output tokens per run — this answers the run-card "tokens/s" deliverable.

Reference anchors for scale (round-7 cold baseline): qwen3:14b (incumbent) 85.5% B / 83.1% A, qwen3:8b 84.3% B / 81.9% A. The CPT-only model sits above the incumbent on both tiers before any SFT — consistent with the language-shift hypothesis, to be confirmed against the CPT-then-SFT row (round7-5 handoff).

Per the issue deliverable: adapter + merged f16 under /mnt-1/training/runs/t1-cpt-qwen3-14b/export/, exported at Q4_K_M, now scored raw. Next per plan: hand to round7-5 as alternative starting point.

[Xore]

CPT evidence reconciliation — surviving artifacts, unresolved acceptance

Read-only metadata reconciliation found surviving training/checkpoint metadata, merged shard files, GGUF exports, imatrix, registration metadata and four parseable stored result files. Stored totals are Tier A 74/83 twice and Tier B 75/83 twice. These are existing records, not a new performance validation.

Corrections and gaps:

• The previously quoted approximately 32–33 tokens/s measures inference, not CPT training throughput; it does not satisfy the measured training-throughput deliverable.
• Recorded adapter targets list seven linear projections, not embeddings/lm_head as described in this issue. This deviation needs explicit disposition, not silent retraining or assumed compliance.
• CPT manifest and export manifest were absent at the checked canonical run paths. Full base → adapter → merged → export → registration lineage and independent durability evidence were not established; absence at checked paths is not proof of absence everywhere.
• Historical training token counts/peak VRAM, input/protected-set linkage, historical runtime identities and full score/transcript provenance remain unverified. Current runtime metadata is not proof of historical use.

The existing CPT results are not invalidated by these gaps, but acceptance is not established by file presence alone. A separate dependency-disposition pass is checking which requirements genuinely gate CPT-then-SFT versus SFT-only under #3084; do not infer that every SFT path requires accepted CPT. No new training, scoring or resource reservation was performed by this reconciliation. Keep #3083 open pending evidence/disposition; the parent #3079 will receive the dependency outcome.