From 975eacba1af96cef35024838fc5621a63d4a9f24 Mon Sep 17 00:00:00 2001 From: Ddv Date: Sun, 27 Sep 2026 22:17:42 +0700 Subject: [PATCH] =?UTF-8?q?feat(atlas):=20re-topic=20colibri=20probes=20?= =?UTF-8?q?=E2=80=94=208=20categories,=20non=5Fcoding=20contrast=20bucket?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner directive 2026-09-27 (hydra_vortex track): - ADDED: technical_architect, ui_development, reasoning_thinking (3 prompts each, lengths kept in the existing narrow band to avoid the prefill length-bias confound). - FOLDED: chinese / german / poetry / law / medicine / casual_chat into a single non_coding contrast category (representative medicine / german / poetry prompts) — keeps a non-coding baseline for the base-rate correction (p(c|e) normalization and spec = 1 - H/log C) while halving the non-coding topic count. - KEPT verbatim: code_python, code_sql, json_format, math_proof. Net: 10 x 3 -> 8 x 3 = 24 probes (sweep.sh / analyze.py read probes.json dynamically; no code changes needed). README provenance updated — the "verbatim from Colibri" claim is now "started from Colibri @a8f2ca62, owner-updated 2026-09-27". NOTE: this changes the topic DEFINITION only. The served atlas still carries the old 10-topic artifact until a 24-probe re-sweep regenerates experts.json / expert-ranks.json (and spec/labels change because C goes 10 -> 8; post-hoc editing the served JSON would leave affinities unnormalized — re-probe is the only correct path). Validation: python3 -m json.tool OK; 8 categories, 3 prompts each. Co-Authored-By: Oh My Pi (opencode-go/mimo-v2.6-flash) --- tools/expert-atlas/README.md | 6 ++-- tools/expert-atlas/probes.json | 50 ++++++++++++++-------------------- 2 files changed, 23 insertions(+), 33 deletions(-) diff --git a/tools/expert-atlas/README.md b/tools/expert-atlas/README.md index e69ef3c512cd..097698ac2451 100644 --- a/tools/expert-atlas/README.md +++ b/tools/expert-atlas/README.md @@ -10,13 +10,13 @@ fork issue #133. Confound traps are enforced by this harness, not by memory: | MTP/draft counts describe rejected tokens | run the server WITHOUT MTP/spec args; harness asserts no draft fields in responses | | lifetime histogram pollutes per-probe stats | per-probe **delta** of `GET /experts` (reset-free by construction); upgrades to the Stage-A per-probe reset/dump when S-A2 lands | | prefill routing is topic-generic | engine counters are decode-only by design (Stage A); deltas exclude prefill | -| prompt-length bias | probes.json keeps lengths in a narrow band (verbatim from Colibri: 10 categories × 3 prompts) | +| prompt-length bias | probes.json keeps lengths in a narrow band (owner-set: 8 categories × 3 prompts, started from Colibri's 10 × 3) | ## Usage ```bash # engine with the Stage B surface enabled (see tools/server/server-atlas.h) -SERVER_URL=http://127.0.0.1:8086 ./sweep.sh # all 30 probes +SERVER_URL=http://127.0.0.1:8086 ./sweep.sh # all 24 probes SERVER_URL=http://127.0.0.1:8086 ./sweep.sh probes.json ./out # explicit python3 analyze.py --stats ./out/stats --probes probes.json \ @@ -35,7 +35,7 @@ production stats path once it lands. ## Files -- `probes.json` — verbatim from Colibri @a8f2ca62 (10 categories × 3 prompts) +- `probes.json` — started verbatim from Colibri @a8f2ca62 (10 × 3); **owner update 2026-09-27**: 8 categories × 3 (adds `technical_architect` / `ui_development` / `reasoning_thinking`; folds chinese/german/poetry/law/medicine/casual_chat into `non_coding`) - `sweep.sh` — driver: greedy confound-controlled generation + per-probe delta - `analyze.py` — statistics port (mean share, p(c|e), spec = 1 − H/log C, replication gate) emitting the observability-tier `experts.json` + provenance diff --git a/tools/expert-atlas/probes.json b/tools/expert-atlas/probes.json index ae377da8e335..5d5e9d6aadb3 100644 --- a/tools/expert-atlas/probes.json +++ b/tools/expert-atlas/probes.json @@ -1,5 +1,5 @@ { - "_comment": "Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.", + "_comment": "Probe set for the expert atlas (#175 lineage). OWNER UPDATE 2026-09-27: 8 categories x 3 prompts. Kept code_python / code_sql / json_format / math_proof; ADDED technical_architect / ui_development / reasoning_thinking; FOLDED the former chinese / german / poetry / law / medicine / casual_chat topics into one non_coding contrast bucket (representative medicine / german / poetry prompts). Started from Colibri @a8f2ca62 (10 categories x 3 prompts). Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.", "code_python": [ "Write a Python function that merges two sorted lists into one sorted list.", @@ -16,39 +16,29 @@ "Show that the sum of the first n odd numbers equals n squared.", "Explain why the derivative of e^x is e^x." ], - "chinese": [ - "请用三句话解释什么是机器学习。", - "写一首关于秋天的短诗。", - "中国的四大发明是什么？请简要说明。" - ], - "german": [ - "Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.", - "Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.", - "Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?" - ], - "poetry": [ - "Write a short poem about the sea at night.", - "Compose four lines of verse about an empty train station.", - "Write a haiku about the first snow of winter." - ], - "law": [ - "Explain in plain terms what consideration means in contract law.", - "What is the difference between a misdemeanor and a felony?", - "Summarize what the term 'force majeure' covers in a commercial contract." - ], - "medicine": [ - "Explain the difference between type 1 and type 2 diabetes.", - "What are the common symptoms of iron deficiency anemia?", - "Describe how a vaccine produces immunity." - ], "json_format": [ "Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.", "Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.", "Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'." ], - "casual_chat": [ - "Hey, what's a good way to spend a rainy Sunday afternoon?", - "I'm feeling pretty tired today. Any tips to get through the afternoon?", - "What's your favourite kind of weather, and why?" + "technical_architect": [ + "Design a high-level architecture for a multi-region read-replica system.", + "Compare event-driven and request-response designs for a background job pipeline.", + "Describe how you would shard a write-heavy key-value store." + ], + "ui_development": [ + "Write the JSX for a modal dialog with a backdrop and a close button.", + "Explain how the CSS flexbox gap property differs from margins in a toolbar.", + "Describe how to make a data table keyboard-navigable with ARIA roles." + ], + "reasoning_thinking": [ + "A farmer has 17 sheep, all but 9 run away. How many are left? Explain step by step.", + "If 5 machines make 5 widgets in 5 minutes, how long do 100 machines need for 100 widgets? Reason it out.", + "Compare the tradeoffs of sorting one million integers in memory versus on disk. Think it through." + ], + "non_coding": [ + "Explain in plain terms how a vaccine helps the body fight a disease.", + "Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.", + "Write a short poem about the first snow of winter." ] }