Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions tools/expert-atlas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ fork issue #133. Confound traps are enforced by this harness, not by memory:
| MTP/draft counts describe rejected tokens | run the server WITHOUT MTP/spec args; harness asserts no draft fields in responses |
| lifetime histogram pollutes per-probe stats | per-probe **delta** of `GET /experts` (reset-free by construction); upgrades to the Stage-A per-probe reset/dump when S-A2 lands |
| prefill routing is topic-generic | engine counters are decode-only by design (Stage A); deltas exclude prefill |
| prompt-length bias | probes.json keeps lengths in a narrow band (verbatim from Colibri: 10 categories × 3 prompts) |
| prompt-length bias | probes.json keeps lengths in a narrow band (owner-set: 8 categories × 3 prompts, started from Colibri's 10 × 3) |

## Usage

```bash
# engine with the Stage B surface enabled (see tools/server/server-atlas.h)
SERVER_URL=http://127.0.0.1:8086 ./sweep.sh # all 30 probes
SERVER_URL=http://127.0.0.1:8086 ./sweep.sh # all 24 probes
SERVER_URL=http://127.0.0.1:8086 ./sweep.sh probes.json ./out # explicit

python3 analyze.py --stats ./out/stats --probes probes.json \
Expand All @@ -35,7 +35,7 @@ production stats path once it lands.

## Files

- `probes.json` — verbatim from Colibri @a8f2ca62 (10 categories × 3 prompts)
- `probes.json` — started verbatim from Colibri @a8f2ca62 (10 × 3); **owner update 2026-09-27**: 8 categories × 3 (adds `technical_architect` / `ui_development` / `reasoning_thinking`; folds chinese/german/poetry/law/medicine/casual_chat into `non_coding`)
- `sweep.sh` — driver: greedy confound-controlled generation + per-probe delta
- `analyze.py` — statistics port (mean share, p(c|e), spec = 1 − H/log C,
replication gate) emitting the observability-tier `experts.json` + provenance
Expand Down
50 changes: 20 additions & 30 deletions tools/expert-atlas/probes.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"_comment": "Probe set for the GLM-5.2 Expert Atlas (#175). 10 categories x 3 prompts. Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.",
"_comment": "Probe set for the expert atlas (#175 lineage). OWNER UPDATE 2026-09-27: 8 categories x 3 prompts. Kept code_python / code_sql / json_format / math_proof; ADDED technical_architect / ui_development / reasoning_thinking; FOLDED the former chinese / german / poetry / law / medicine / casual_chat topics into one non_coding contrast bucket (representative medicine / german / poetry prompts). Started from Colibri @a8f2ca62 (10 categories x 3 prompts). Prompts are deliberately varied in phrasing within a category so the affinity vector reflects the TOPIC, not one prompt's surface form. Lengths are kept within a narrow band across categories: prefill routes the prompt tokens too, so a long prompt in one category would inflate its counts.",

"code_python": [
"Write a Python function that merges two sorted lists into one sorted list.",
Expand All @@ -16,39 +16,29 @@
"Show that the sum of the first n odd numbers equals n squared.",
"Explain why the derivative of e^x is e^x."
],
"chinese": [
"请用三句话解释什么是机器学习。",
"写一首关于秋天的短诗。",
"中国的四大发明是什么?请简要说明。"
],
"german": [
"Erkläre in drei Sätzen, wie ein Verbrennungsmotor funktioniert.",
"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.",
"Was ist der Unterschied zwischen Dativ und Akkusativ im Deutschen?"
],
"poetry": [
"Write a short poem about the sea at night.",
"Compose four lines of verse about an empty train station.",
"Write a haiku about the first snow of winter."
],
"law": [
"Explain in plain terms what consideration means in contract law.",
"What is the difference between a misdemeanor and a felony?",
"Summarize what the term 'force majeure' covers in a commercial contract."
],
"medicine": [
"Explain the difference between type 1 and type 2 diabetes.",
"What are the common symptoms of iron deficiency anemia?",
"Describe how a vaccine produces immunity."
],
"json_format": [
"Return a JSON object with keys name, age, and email for a fictional user. Output only JSON.",
"Convert this to JSON: name Alice, roles admin and editor, active true. Output only JSON.",
"Write a JSON schema for an object with a required string field 'id' and an optional integer 'count'."
],
"casual_chat": [
"Hey, what's a good way to spend a rainy Sunday afternoon?",
"I'm feeling pretty tired today. Any tips to get through the afternoon?",
"What's your favourite kind of weather, and why?"
"technical_architect": [
"Design a high-level architecture for a multi-region read-replica system.",
"Compare event-driven and request-response designs for a background job pipeline.",
"Describe how you would shard a write-heavy key-value store."
],
"ui_development": [
"Write the JSX for a modal dialog with a backdrop and a close button.",
"Explain how the CSS flexbox gap property differs from margins in a toolbar.",
"Describe how to make a data table keyboard-navigable with ARIA roles."
],
"reasoning_thinking": [
"A farmer has 17 sheep, all but 9 run away. How many are left? Explain step by step.",
"If 5 machines make 5 widgets in 5 minutes, how long do 100 machines need for 100 widgets? Reason it out.",
"Compare the tradeoffs of sorting one million integers in memory versus on disk. Think it through."
],
"non_coding": [
"Explain in plain terms how a vaccine helps the body fight a disease.",
"Schreibe eine kurze formelle E-Mail, in der du einen Termin absagst.",
"Write a short poem about the first snow of winter."
]
}