v2-i1: Qwen3-0.6B controlled comparison + DeepSeek judge (ADR 0050) - #4
Merged
Conversation
Add configs/qwen3-0.6b-lora-r16.yaml as a copy of the 1.7B config with only the three model-identity fields changed: base model id (Qwen/Qwen3-0.6B), hub_model_id (committed-qwen3-0.6b-lora), and run_name. The recipe (LoRA rank/alpha, lr, sequence length, epochs, batch size, grad-accum) is left byte-identical so the v1->v2-i1 comparison is attributable to base-model capacity alone, per ADR 0049. Parameterize scripts/train.slurm to take the config path as its first arg, defaulting to the 0.6B config, so the same script reproduces either run. Gate 0 verified: Qwen3-0.6B and Qwen3-1.7B tokenizers are identical (vocab_equal and ids_equal both True), so committed-train is reused unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
The first 0.6B run crashed because train.py hardcoded resume_from_checkpoint=True while the 0.6B config's output_dir still pointed at outputs/qwen3-1.7b-lora-r16, so the trainer tried to resume 1.7B checkpoints into the 0.6B model. - configs/qwen3-0.6b-lora-r16.yaml: output_dir -> outputs/qwen3-0.6b-lora-r16, and add io.resume_from_checkpoint (default false). - train.py: read io.resume_from_checkpoint (default False via .get, so the frozen 1.7B config is unaffected and also starts clean). A fresh run now starts from scratch; an interrupted run can resume by setting the flag true. W&B project is unchanged: train.py still pins WANDB_PROJECT=committed before the wandb integration loads, matching the slurm export. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
Phase 3 prep. merge_adapter.py: base -> Qwen/Qwen3-0.6B, adapter ->
committed-qwen3-0.6b-lora, output under gitignored models/.
merge_convert_quantize.py was a non-functional scaffold (every stage raised
NotImplementedError). Implemented it as a runnable merge -> convert -> quantize
pipeline that shells out to a llama.cpp checkout (convert_hf_to_gguf.py +
llama-quantize), defaulting to the 0.6B adapter and Q4_K_M. A --no-adapter mode
converts the bare base model for the eval baseline arm.
Produced and pushed (private) marzoukbaig14/committed-gguf-0.6b:
committed-0.6b-{finetuned,baseline}-Q4_K_M.gguf. Sanity generation through the
frozen engine + GBNF grammar yields valid Conventional Commits from both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
…tial) Generation (both arms, 442 each over the frozen strata ids) complete. Gemini judge: fine-tune arm fully judged (442/442, $0.8182), report written. Baseline arm reached only 35/442 before Gemini's daily request quota was exhausted; the harness stopped gracefully and the log is resumable. Committing completed evidence now (ephemeral sandbox); the baseline judge resumes once quota resets. Eval generation reproduced v1's no-guard path (the looks_like_diff guard added later to engine.generate() rejects 1 of the 442 real rows, id 350); bypassed in the eval driver only, engine.py unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
Mirror the v1 .gitignore allowlist convention (ADR 0046) for the 0.6B candidates and judge logs. Fine-tune judge log is complete (442); baseline judge log is the resumable partial (35/442, pending Gemini daily-quota reset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
Gemini judge credits depleted mid-eval (prepaid balance, not a rate cap) and the account has a $10 minimum top-up. Switch the v2-i1 judge to DeepSeek (pay-as-you-go, ~10-50x cheaper, OpenAI-compatible). - src/committed/eval/judge_deepseek.py: new backend, model deepseek-chat, mirrors judge_gemini.py exactly (same JudgeResult schema, same JSONL log contract, same resumable/graceful-stop behavior; BalanceExhausted is the 402 analogue of DailyQuotaExceeded). Rubric (judge_prompt.py) imported UNCHANGED; a JSON-format envelope is added in-backend for DeepSeek's schemaless json_object mode, with validation + retry. - run_eval.py: additive --backend deepseek (choices/import/rpm); metrics, composite, sample, prompt, grammar all untouched. - ADR 0050: records the switch, the comparability cost (v1 Gemini numbers become a superseded reference; all four arms re-judged on DeepSeek for one-judge consistency), and the human-validation re-check plan. Regenerated DECISION_LOG.md + decision-tree.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
First arm of the DeepSeek re-judge (ADR 0050) complete: 0.6B baseline, 442/442, $0.32. type_correctness pass 0.296, conjunctive (deploy) 0.101 — the feat-collapse confirmed by the judge. Remaining arms (0.6B fine-tune, 1.7B baseline+fine-tune) still judging in the background; their evidence lands as each completes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
0.6B fine-tune re-judged on DeepSeek (442/442, $0.31). DeepSeek-judged 0.6B lift baseline->fine-tune: type-correct pass 0.296->0.726, prefix acc 0.154->0.601 (above the 0.489 floor), graded 0.777->2.094 — the feat-collapse rescue, judge-confirmed. judge_deepseek.py robustness: at temperature=0 a malformed-JSON generation repeated identically on every retry and one bad example crashed a whole arm (hit on the 1.7B baseline). Fix: escalate temperature on JSON-parse retries so a resample can differ, strip stray code fences, and skip a persistently-bad example (logged) instead of aborting the batch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
All four arms now DeepSeek-judged (442 each, 0 skipped). Adds the 1.7B baseline (+ human validation) and 1.7B fine-tune evidence, and the comparison report docs/eval/FINDINGS_v2_i1.md. Headline: the fine-tune rescues 0.6B feat-collapse decisively (type_correctness 0.296->0.726; prefix acc 0.154->0.601, above the 0.489 floor; feat-share of generations 86.7%->9.7%). The 0.6B fine-tune lands within ~4-5 pts of the 1.7B fine-tune on type/faithfulness/completeness/prefix and near-identical graded mean (2.094 vs 2.139); the capacity gap is concentrated in specificity (0.545 vs 0.667) and the conjunctive all-pass it caps. DeepSeek judge vs human (n=50): agreement >= Gemini on every axis (type/faith/compl kappa 0.56-0.61; specificity weakest at 0.34). Total judge cost ~$1.2 est. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
… var) Drop the unused `openai.APIError` import (F401) and the unused binding in `except (RateLimitError, APIConnectionError) as e` (F841). No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
v2 iteration 1 (ADR 0049): re-run the v1 fine-tune + eval pipeline with exactly one variable changed — base model 1.7B → 0.6B. Recipe, dataset, grammar, prompt, 442-row strata sample, harness, and composite all frozen.
configs/qwen3-0.6b-lora-r16.yaml— copy of the 1.7B config, only the three model-identity fields changed (base id, hub_model_id, run_name); recipe byte-identical.scripts/train.slurmparameterized (default 0.6B).merge_adapter.pyrepointed to 0.6B;merge_convert_quantize.py(was a non-functional stub) implemented into a real merge→convert→quantize pipeline. Producedcommitted-gguf-0.6b(Q4_K_M).train.py— checkpoint resume made config-driven (default off) after the first run crashed trying to resume 1.7B checkpoints into the 0.6B model.Judge switch — ADR 0050
Gemini's prepaid credits depleted mid-eval, so the judge was switched to DeepSeek (
deepseek-chat, OpenAI-compatible, pay-as-you-go). New backendjudge_deepseek.pyreuses the frozen rubric unchanged;run_eval.pygains an additive--backend deepseek. To stay consistent under one judge, all four arms were re-judged on DeepSeek (0.6B + the v1 1.7B candidate files); the old Gemini numbers are a superseded reference. Full findings indocs/eval/FINDINGS_v2_i1.md, raw evidence inanalysis/results/(ADR 0046).Headline result
The fine-tune rescues 0.6B feat-collapse decisively (type-correctness 0.296 → 0.726; prefix accuracy 0.154 → 0.601, above the 0.489 floor; feat-share of generations 86.7% → 9.7%). The 0.6B fine-tune lands within ~4–5 pts of the 1.7B fine-tune on type/faithfulness/completeness/prefix and near-identical graded mean (2.094 vs 2.139); the capacity gap concentrates in specificity (0.545 vs 0.667). DeepSeek judge vs human (n=50): agreement ≥ Gemini on every axis. Total judge cost ~$1.2 est.
🤖 Generated with Claude Code
Generated by Claude Code