From 4aa5445fdb8b1498fe9bf437963ea3763cddd5e8 Mon Sep 17 00:00:00 2001 From: Tremayne Timms Date: Tue, 1 Sep 2026 10:28:15 -0500 Subject: [PATCH] roadmap: queue best-in-class 16 GB agentic coder plan Adds a dated PLAN section: honest current standing (52.0% vs Devstral Small 56.4% same scaffold, within n=50 noise; #1 on eval rigor; real-world unproven), diagnosis (KV-budget ceiling, not capability), and a cheapest-first ladder -- validate open context-managed/chat-template levers, benchmark breadth (full SWE-bench Verified, Aider polyglot, Terminal-Bench), 2-3 week dogfood, a controlled cross-model table, a base decision gate (keep KAT vs NVFP4 Devstral Small 2512), then SFT/GRPO on agentic traces as last resort. Not started -- queued behind current project. --- ROADMAP.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index eef1a96..e956786 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -449,6 +449,95 @@ revisit the two SM120-specific decode-throughput fixes noted in `docs/optimization_research_2026-08-21.md` §1 — those didn't help the A16 checkpoint but weren't tested against W4A4's different kernel path. +## PLAN (2026-09-01): path to best-in-class 16 GB agentic coder — queued, not started + +Goal, stated plainly: the best agentic coding model that fits a single 16 GB +consumer card, measured on benchmarks **and** real-world use. Queued behind the +current unrelated project; nothing below has been run. + +**Where this stands today (honest, so the plan targets the real gap):** + +- SWE-bench Verified, the one apples-to-apples comparison that exists: 52.0% + (26/50, `kat_overrides_sota.yaml`, official grading) vs. Devstral Small 2512's + 56.4% under the same `mini-swe-agent` bash-only scaffold. Behind by 4.4 points + on n=50 — inside the ~+/-14-point binomial CI at that sample size, so "trails + by a margin within noise," not a settled rank. +- Pure code-gen: HumanEval+ 90.9% / MBPP+ 89.9% (shipped RTN A16) — top-tier for + anything in 16 GB. +- Documentation / eval rigor: no other 16 GB-class coding model publishes + SWE-bench + HumanEval+ + MBPP+ with confidence intervals and failure-mode + breakdowns. This is the current defensible claim. +- Real-world use: **no evidence collected.** One HF thank-you comment. No + multi-benchmark table, no dogfood log. Cannot currently claim it. + +**Diagnosis (from the 2026-08-22 RESULT above):** the gap is not capability — +81.25% resolve-of-completed. 17/50 die at `ContextWindowExceeded` on the 49K +ceiling before submitting a patch. That is a KV-budget ceiling forced by the +16 GB card, not a reasoning deficit. + +**Already done — do not re-tread as "new levers":** fp8 KV cache is on +(`run_pilot_all.sh` / `serve_kat.sh`, `--kv-cache-dtype fp8`). The 5,000-char +observation truncation from `kat_overrides_context_managed.yaml` is already +adopted into the SOTA config. `presence_penalty`/`top_k` was full-pilot tested +and regressed (RESULT 2026-08-23). + +### Ladder, cheapest first — each gate is "re-measure before proceeding" + +1. **Close the open cheap levers.** Full-pilot validate + `kat_overrides_context_managed.yaml` (`max_tokens` 3072->1024 + step_limit 30) + against the 52.0% baseline — never actually run, only reasoned about; its own + header flags the -truncation risk, so it needs the measurement, not + more argument. Also the two Longer-horizon items below: the Qwen-Sharp chat + template (one-file swap, targets the token tax directly) and, more + speculatively, FreeToken expert-offload (could serve a lighter-pruned or + unpruned base in 16 GB, skipping the REAP accuracy tax entirely). +2. **Benchmark breadth.** The 50-instance SWE-bench subset is too thin to rank + on. Add: full SWE-bench Verified (500); a fresher / less-contaminated set + (SWE-bench Live or Multimodal); the **Aider polyglot leaderboard** (the + community's trusted coding-assistant benchmark — measures edit-format + reliability, a real-world failure mode); **Terminal-Bench** for agentic + tool-use. Same scaffold, same hardware, same session for every entry. +3. **Real-world evidence — the missing half.** Serve the checkpoint through the + vLLM OpenAI endpoint and use it as the daily coding assistant for 2-3 weeks + in a real agent harness (aider / opencode / Claude-Code-style). Log every + failure with the task shape. This produces honest card content and a signal + SWE-bench cannot give. +4. **Head-to-head table — this IS the differentiator.** KAT-Coder REAP-50 vs. + Devstral Small 2512 vs. Qwen3-Coder-30B-A3B (4-bit), identical scaffold, + identical GPU, same session, >=200 SWE-bench instances + the Aider set. + Nobody else in this class publishes a controlled cross-model table. This is + what turns "trails by noise" into a number. +5. **Base decision gate.** After 1-4: + - Clears 56.4% same-scaffold -> commit to KAT REAP-50, ship the updated card. + - Doesn't -> quantize **Devstral Small 2512** to NVFP4 with this repo's eval + discipline. Apache-2.0, already fits 16 GB, already beats the bar, and + nobody has published a rigorously-evaluated NVFP4 of it. Faster route to + "best-documented 16 GB agentic coder" than grinding this base. KAT REAP-50 + stays as the "only evaluated 16 GB KAT-Coder" niche entry it already is. + (Distinct from the Ornith-1.5-35B-A3B swap, tracked in the private + `sota-ornith-build` branch.) +6. **Training levers — only if 1-5 leave a gap worth closing.** + - SFT on successful agentic traces: self-distill from the 26 SWE-bench wins + plus a teacher model's `mini-swe-agent` runs. Targets patch-format + reliability and completion rate. Moderate cost. + - GRPO on agentic coding with a test-pass reward: real project. Long + trajectory rollouts on a 17.5B MoE inside 16 GB are far more expensive than + a 4B GRPO run. Last resort for the final point or two. + +### Claim discipline + +Until the step-4 table exists, the only defensible public claim is: *"competitive +with the best 16 GB-class agentic coders on SWE-bench Verified under an identical +local scaffold; the only one in the class with published multi-benchmark results +and confidence intervals."* Not "best," not "top N in the world" — those are +unfalsifiable without a ranked field and cost the credibility the eval rigor +buys. + +### Timeline + +One base, done properly — cheap levers + 4-5 benchmarks + real-world writeup + +the cross-model table — is a ~3-4 week slow-lane effort, not a weekend. + ## Longer-horizon / not scheduled - **Candidate performance levers surfaced 2026-08-22, not yet tested against