docs(benchmarks): round 7 plan — train and requantise with Unsloth, serve with Ollama, score on a fresh three-slot pin - #3089
Merged
Merged
Conversation
…erve with Ollama, score on a fresh three-slot pin Plan of record for epic #3079 (children #3080–#3088): the measured state of the #1947 sweep on 2026-09-06 (phase 2 settled, phase 3 ladder scoring, cold re-run armed), why the current matrix cannot show a training gain (sixteen models within one point at 63–64/69; self-quant Q3/Q4/Q5 all at 62), what Unsloth and Ollama each do and do not do, the 20 GB training envelope, the train → merge → GGUF → imatrix → quantise → ollama create → score → promote pipeline with the container that owns each step, the data rules (test set off limits, captured data never leaves the host, local vs OpenRouter teachers), the experiment ladder T0–T6 / R1–R3 with a named control per row, the round-7 benchmark design (17-case /79 pin, pooled claims, all three slots, cold protocol), and the GPU sequencing behind the cold re-run. Cross-links the resume plan to its successor.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
…ith grit, rtk and gh grit is the mandatory function-level coordination layer for parallel work in this repository and rtk fronts every shell command; the plan named only the gh conventions. §15 carries the grit claim → edit-in-worktree → done workflow (one agent name per subagent, serialized done, grit init after merges, absolute paths inside grit worktrees), the rtk rules, the unchanged gh conventions and the two-agent cap.
9 tasks
This was referenced Sep 6, 2026
Xore
changed the title
Xore
changed the title
Xore
added a commit
that referenced
this pull request
Sep 6, 2026
…the round-7 plan citations round7_build_corpus.sh hardcoded both LAN resolvers on its `docker run` line; one of them is the home-server address scripts/check-public-leaks.py bans from this repo, so the public-leak gate and its #2285 regression test both failed. The resolver pair is read from the host's own /etc/resolv.conf instead (the benchmark host's list is exactly those two nodes, and install-homeserver.sh asserts the first entry is one of them), loopback stubs filtered out, with a RESOLVERS="ip ip" override and a loud abort when nothing usable is found. Doc-path lint (#2458): - the round-7 hermes-init doc cited PR #3089's branch as if it were a path; the plan is merged, so it now cites the merged path alone; - the plan's grit note used `../x` / `../requirements.txt` as generic examples of relative-path dependencies, not repo citations -- rewritten as `../<crate>` / `../<file>` placeholders, which the lint skips by design; - analysis/ghidra/training and its TOOLCHAIN.md are the training tree #3080 creates, untracked by design until then, so they take allowlist entries.
Xore
added a commit
that referenced
this pull request
Sep 6, 2026
…seline, and the orchestrator handoff (#3091) * feat(benchmarks): round 7 drivers — pin, corpus, cache, smoke, launch; fold the #1947 cold re-run into the round-7 pin Operator decision 2026-09-06: the a99e765 cold re-run (2–4 GPU-days on a 14-case rubric whose top is saturated) is folded into round 7, which needed the same roster re-scored on the 17-case pin anyway. Stopped after 13 models (kept in 1947cold/, marked ABORTED), then prepared and launched the round-7 cold baseline on pin 32dbdeb the same afternoon. New, all with operational copies in /mnt-1/benchmarks/: - round7_prep_pin.sh second clone APIARY-round7, detached at the pin; the a99e765 clone is never touched - round7_build_corpus.sh 17-case corpus rebuilt in the provenance container through ci_verify.sh (manifest byte-identical, 280 semantic executions, 0 failed) -> corpus-round7/ - round7_cache.sh 17-case Tier B cache in its own directory - round7_smoke.sh one Tier A + one Tier B run before any launch (measured: qwen2.5:7b 70/83 A, 69/83 B) - round7_coldrun.sh coldrun.sh for the new pin: PIN guard, 17-entry cache guard, results round7/, roster models_round7.txt; self-matching process guard fixed - round7_launch.sh refuses without a Tier B smoke on this pin (17 cases, max 83, score > 0); starts the sampler - round7_prepare.sh the three prep steps in order, resume-safe sweep_extra.sh gains GHIDRA_CACHE and OPERATOR env overrides; defaults are the a99e765 sweep's, unchanged. The rubric maximum on this pin is 83, not the 79 the resume plan guessed (required_groups + 1 per case) — corrected in the plan, which also gains §0 (what changed after it was written) and the runbook for the running sweep, plus an init.md-style handoff for the orchestrator. * fix(benchmarks): derive round-7 container DNS from the host, and fix the round-7 plan citations round7_build_corpus.sh hardcoded both LAN resolvers on its `docker run` line; one of them is the home-server address scripts/check-public-leaks.py bans from this repo, so the public-leak gate and its #2285 regression test both failed. The resolver pair is read from the host's own /etc/resolv.conf instead (the benchmark host's list is exactly those two nodes, and install-homeserver.sh asserts the first entry is one of them), loopback stubs filtered out, with a RESOLVERS="ip ip" override and a loud abort when nothing usable is found. Doc-path lint (#2458): - the round-7 hermes-init doc cited PR #3089's branch as if it were a path; the plan is merged, so it now cites the merged path alone; - the plan's grit note used `../x` / `../requirements.txt` as generic examples of relative-path dependencies, not repo citations -- rewritten as `../<crate>` / `../<file>` placeholders, which the lint skips by design; - analysis/ghidra/training and its TOOLCHAIN.md are the training tree #3080 creates, untracked by design until then, so they take allowlist entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Plan of record for #3079 (round 7), children #3080–#3088. Docs only:
docs/benchmarks/plans/2026-09-06-round7-unsloth-train-requant-ollama.md(§1–§15) and the cross-link from the 2026-09-05 resume plan.The drivers, the §0 status block, the /83 correction and the orchestrator handoff landed separately in the follow-up PR from
feat/round7-drivers.