feat(run): answer the agent with a model, not a Markdown parser

docs/guides/byoh.md

@@ -139,7 +139,8 @@ Multi-turn evals — scripted `turns` and `responder` alike — require
 `[conversation].resume_exec_template` plus transcript extraction of ordered assistant messages and
 the native session ID. There is no fresh-session fallback: `run` rejects the case when the harness
 cannot preserve the conversation. The responder itself needs nothing further from a descriptor; it
-reads the agent's message as Markdown, so it works on any harness that can resume. See
+reads the agent's message out of the transcript and consults its own model through the dispatch
+template you already declared, so it works on any harness that can resume. See
 `eval-magic docs conversations`.
 
 When a shadow preflight reports a live copy, isolate every initial and resumed eval-agent dispatch

docs/guides/conversations.md

@@ -29,100 +29,110 @@ front when the selected harness cannot. `eval-magic harness list` names the
   "id": "add-request-caching",
   "prompt": "Requests to the pricing API are slow. Can you add caching?",
   "expected_output": "A working cache with the pricing endpoint under 100ms.",
-  "responder": { "type": "heuristic", "max_turns": 8 }
+  "responder": { "type": "llm", "max_turns": 8 }
 }
 ```
 
-- **`type`** is required. `heuristic` is the only responder today. It is
-  deterministic and costs nothing: it reads the agent's message and applies
-  fixed rules, with no second model involved.
+- **`type`** is required. `llm` is the only responder: a small model, consulted
+  once after every round through the same harness as the agent under test.
+  Choose it with `eval-magic run --responder-model`; omit that flag and the
+  consultation runs on the harness's default model.
 - **`max_turns`** bounds how many follow-ups the responder may synthesize. The
   opening prompt is not one of them. It defaults to 8.
 
 Every turn the responder produces is recorded in the run's `conversation.json`
-with an `origin` naming the rule that produced it, so you can audit whether the
-responder distorted the run instead of taking the transcript on trust. The
-eval's own opening prompt carries no `origin` — that absence is how you tell an
-authored turn from a derived one.
+with an `origin` naming the responder and, when it offered one, its one-line
+reason for answering that way. The eval's own opening prompt carries no
+`origin` — that absence is how you tell an authored turn from a derived one.
 
-## What the heuristic answers
+The full prompt and verdict of every consultation are kept on disk under the
+run's `responder/turn-<n>/`, so you can audit what the responder was shown and
+what it wrote without rerunning anything.
 
-The heuristic reads the last message of each round as Markdown and answers
-exactly one shape of question: **a list of options introduced by a question.**
+## What the responder is shown
 
-A list counts as a question when the line directly above it ends with a `?`:
+**Only what the agent already knows.** Each consultation carries the eval's
+opening `prompt`, every reply the responder has already given, and the agent's
+last message. It does not carry `expected_output` and it does not carry the
+assertions: those are the grading criteria, and a responder that had read them
+could hand the agent the rubric.
 
-```
-Which cache should I use?
-
-- An in-process LRU (Recommended)
-- Redis
-```
+It is told to answer as the person who asked for the work — take whatever the
+agent marked as recommended, else the simplest option and the least work; add no
+requirements; invent no facts; write no code; keep it short.
 
-The `?` has to be the last thing said before the options appear. That is what
-separates a real question from a closing summary, which is also mostly a
-bulleted list and would otherwise be "answered" as though the finished task were
-still open.
+It answers with one of three verdicts:
 
-Given a list, the choice is mechanical:
+| Verdict | What it means |
+| --- | --- |
+| `answer` | What the user says next. Delivered as the following turn. |
+| `done` | The agent is reporting the task finished and waiting on nothing. |
+| `cannot_answer` | It could not answer without inventing something. |
 
-| The list | Recommendation marked | The answer |
-| --- | --- | --- |
-| plain (`-`, `*`, `1.`) | yes | the first recommended option |
-| plain | no | the first option |
-| checkboxes (`- [ ]`) | yes | every recommended option |
-| checkboxes | no | nothing — `None of these.` |
+Because the responder decides `done`, completion is a judgement rather than the
+absence of a question mark — and the judgement is recorded with its reason, so
+a run that stopped early is legible rather than mysterious.
 
-Plain lists ask for exactly one choice; checkboxes ask for zero or more. That
-syntax is the only signal the heuristic uses to tell them apart.
+## What is never delivered
 
-An option counts as recommended when it carries a standalone `recommended` in
-parentheses, brackets, or bold — `(Recommended)`, `[recommended]`,
-`**Recommended**` — or when it is a pre-checked box, `- [x]`.
+A reply that fails any of these checks is not sent to the agent. The run stops
+instead, because an undelivered reply is a loud, greppable stop, while a bad one
+enters the transcript as an ordinary user turn and is graded as though the
+exchange really happened.
 
-A message that asks more than one question is answered in one turn, numbered in
-the order the questions appeared.
+| Rejected when the reply | Recorded cause |
+| --- | --- |
+| is blank | `empty_reply` |
+| runs past 2000 bytes | `reply_too_long` |
+| contains a fenced code block | `reply_contains_code` |
+| repeats the previous reply verbatim | `reply_repeated` |
+
+The length and code rules are the same rule twice: a simulated user answers in
+sentences, so anything longer means the responder started doing the agent's work,
+and crediting the agent under test with work it did not do would corrupt the
+result. A repeat means the exchange is circling, and spending the remaining
+turns on it would only reach the same place more expensively.
+
+A consultation that never produces a reply stops the run the same way, with its
+own cause: `declined` when the responder honestly refused, and
+`dispatch_failed`, `dispatch_timed_out`, `missing_verdict`, or
+`malformed_verdict` when something broke. One outcome, because the run ended
+mid-task either way; separate causes, because an honest refusal and a broken
+dispatch call for different fixes.
 
 ## How a conversation ends
 
 | Recorded as | When |
 | --- | --- |
-| `completed` | The agent's last message asked nothing. It considers the task done, so the run stops rather than burning its remaining turns. |
-| `stopped`, `responder_cannot_answer` | The agent asked something with no option list. |
+| `completed` | The responder judged the agent finished. The run stops rather than burning its remaining turns. |
+| `stopped`, `responder_cannot_answer` | The responder produced no usable reply. `responder_outcome.cause` says why. |
 | `stopped`, `max_turns_reached` | The agent was still asking at the bound. |
 | `timed_out` | The task outran `dispatch --timeout`. |
 
 A `stopped` conversation is recorded, not failed: `dispatch` exits zero and
 `ingest` still records the run. But both responder stops end the conversation
-with the task unfinished, so `dispatch` warns about each one by name. Read the
-last assistant message before treating such a run as a data point beside a
-completed one.
-
-Two properties are worth knowing before you read results:
-
-- The heuristic never guesses. A question it does not recognize stops the run
-  instead of inventing an answer, because a fabricated answer would silently
-  change what the agent was asked to do.
-- It errs toward stopping. A question mark anywhere in an otherwise-finished
-  message stops the run rather than calling it complete. That costs a dispatch;
-  the alternative — recording a run as complete while the agent was still
-  waiting — would cost the result's credibility.
-
-Answering free-form questions needs a model, not rules. That is a separate
-responder, and until it ships, `responder_cannot_answer` is where those runs
-stop.
+with the task unfinished, so `dispatch` warns about each one by name and
+`aggregate` counts them per condition in `benchmark.json`'s
+`validity_warnings`. That count is the one to read first: one arm truncated more
+often than the other is a threat to the comparison, not just to the run.
 
 ## Cross-harness behaviour
 
-The heuristic reads plain Markdown out of the agent's message, so it needs no
-per-harness support: any harness that can resume a session can run a responder
-eval. Nothing is read from a harness-native question tool, and nothing needs to
-be, because a dispatch runs headless with no channel to answer such a tool on.
-
-The shapes above are a contract, not a description of one agent. An agent that
-offers options this way is answered; one that phrases them some other way stops
-the run. If you are bringing your own harness and its agent asks in a shape the
-table does not cover, that is a gap in the table, not in your descriptor.
+The responder needs no per-harness support and no descriptor field. It reads
+`TranscriptSummary::final_text`, which every harness's parser already
+normalizes, replies through the existing `{prompt_arg}` slot, and runs its own
+consultations through the same `[dispatch].exec_template` a judge uses. Any
+harness that can resume a session can run a responder eval.
+
+Nothing is read from a harness-native question tool, and nothing needs to be: a
+dispatch runs headless with stdin detached, so a tool that asks the user has no
+channel to be answered on. Free text is the only mechanism that fits, and it
+happens to be the portable one.
+
+Consultations run in the run's own `responder/` directory, which sits above the
+task environment. That is deliberate — a consultation must not be able to write
+into the codebase under measurement, and must not pick up that codebase's
+`CLAUDE.md` or `AGENTS.md` as instructions to itself.
 
 ## Scripted turns
 

docs/progressive-enhancements.md

@@ -216,22 +216,23 @@ or normal guardrail-stopped scenario. `ingest` skips an interrupted task with no
 artifact.
 
 A scripted turn is gated by `agent_asks` (`?`) plus the optional response regex. A responder instead
-*derives* each turn from the round's last assistant message and records the rule that produced it on
+*derives* each turn by consulting a small model, once after every round, and records that origin on
 the turn itself. **The responder needs no descriptor field and no named capability of its own:** it
-reads that message as plain Markdown — a question line followed by a list of options — so every
-harness that resolves a resume template gets it for free, and none can be "missing" it.
+reads the round's last assistant message out of `final_text`, which every transcript parser already
+normalizes, and it dispatches its own consultations through the same `[dispatch].exec_template` a
+judge uses. Every harness that resolves a resume template gets it for free, and none can be
+"missing" it.
 
 That portability is not a happy accident, it is forced. A dispatch runs headless with stdin
 detached, so a harness-native question tool has no channel to be answered on; the runner can only
 send free text as the next user turn. Text is therefore the only mechanism that fits, and it is the
 one every transcript parser already normalizes into `final_text`.
 
-What *is* borrowed from one harness is the convention — `(Recommended)` and checkbox lists are how
-Claude Code's own question UI renders choices. The recognized shapes are documented as a
-harness-neutral contract in `eval-magic docs conversations`, not as "what Claude does": an agent that
-offers options that way is answered identically whatever harness runs it, and one that phrases them
-differently stops the run with `responder_cannot_answer` — a documented gap in the shape table, not a
-missing descriptor field. Widening the table is a runner change that benefits every harness at once.
+A consultation binds the exec template's placeholders the way a judge dispatch does — guard
+arguments off, its own capture directory, its own prompt — with one addition: `<eval-root>` is the
+run's `responder/turn-N/` directory rather than the task env. A consultation must not be able to
+write into the codebase under measurement, nor inherit that codebase's `CLAUDE.md` as instructions
+to itself.
 
 *Fallback:* none. `run` rejects selected multi-turn evals — scripted or responder-driven — when the
 harness omits this capability; silently starting a fresh session would make the answer meaningless.

harnesses/template.toml

@@ -143,7 +143,8 @@ label = "{label}"
 ## -------------------------------------------------------------------------------------------
 ## [conversation] — native same-session continuation for multi-turn evals, both scripted `turns`
 ## and a `responder` that derives them. The responder needs nothing further from a descriptor: it
-## reads the agent's own message as Markdown, so declaring this table is all it takes.
+## reads the agent's own message out of the transcript and consults its model through the
+## [dispatch].exec_template below, so declaring this table is all it takes.
 ## This capability has no generic fallback: run rejects multi-turn evals for a harness that omits
 ## it. It requires
 ## [dispatch].exec_template plus transcript parsing that exposes both ordered assistant messages

profiles/shared/runbook.md

@@ -21,9 +21,10 @@ each task's `conversation.json`. A task that already has one is skipped, so reru
 command retries only what did not finish. A task that exceeds `--timeout` is recorded as timed out
 rather than left to stall the campaign, and a task that fails is recorded and named while the rest
 of the batch continues. A conversation that stops at a scripted gate is valid eval data, not a
-failure. A conversation the responder stopped — because it could not answer the agent's question,
-or because it hit `max_turns` — is recorded too, but it ended with the task unfinished; `dispatch`
-warns about each one by name, and those runs are weaker evidence than a completed one.
+failure. A conversation the responder stopped — because it produced no usable reply, or because it
+hit `max_turns` — is recorded too, but it ended with the task unfinished; `dispatch` warns about
+each one by name and cause, and `aggregate` counts them per condition in `benchmark.json`'s
+`validity_warnings`. Those runs are weaker evidence than a completed one.
 
 ```
 {{INGEST_CMD}}

schema/conversation.schema.json

@@ -38,7 +38,8 @@
           { "$ref": "#/definitions/conversationTool" }
         ]
       }
-    }
+    },
+    "responder_outcome": { "$ref": "#/definitions/responderOutcome" }
   },
   "allOf": [
     {
@@ -90,6 +91,28 @@
     }
   ],
   "definitions": {
+    "responderOutcome": {
+      "type": "object",
+      "required": ["ending"],
+      "additionalProperties": false,
+      "description": "How the responder ended the conversation, when it was the responder that ended it. Absent for a scripted or one-shot task, for a timeout, and for max_turns_reached, which is the runner's bound rather than a verdict.",
+      "properties": {
+        "ending": {
+          "type": "string",
+          "enum": ["done", "cannot_answer"],
+          "description": "Whether the responder judged the agent finished, or produced no usable reply."
+        },
+        "cause": {
+          "type": "string",
+          "enum": ["declined", "dispatch_failed", "dispatch_timed_out", "missing_verdict", "malformed_verdict", "empty_reply", "reply_too_long", "reply_contains_code", "reply_repeated"],
+          "description": "Why no usable reply was produced, so an honest refusal is distinguishable from a broken dispatch. Absent when ending is 'done'."
+        },
+        "rationale": {
+          "type": "string",
+          "description": "The responder's own one-line account. Absent when the dispatch never answered."
+        }
+      }
+    },
     "userMessage": {
       "type": "object",
       "required": ["type", "ordinal", "round", "text"],
@@ -101,45 +124,18 @@
         "text": { "type": "string" },
         "origin": {
           "type": "object",
-          "required": ["responder", "answers"],
+          "required": ["responder"],
           "additionalProperties": false,
           "description": "How a responder derived this turn. Absent on the eval's opening prompt and on scripted turns, which are authored rather than derived.",
           "properties": {
             "responder": {
               "type": "string",
-              "enum": ["heuristic"],
+              "enum": ["llm"],
               "description": "Which responder produced the turn."
             },
-            "answers": {
-              "type": "array",
-              "minItems": 1,
-              "description": "One entry per question the turn answered, in the order they were asked.",
-              "items": {
-                "type": "object",
-                "required": ["options", "rule", "chosen"],
-                "additionalProperties": false,
-                "properties": {
-                  "question": {
-                    "type": "string",
-                    "description": "The question line the options hung from."
-                  },
-                  "options": {
-                    "type": "array",
-                    "items": { "type": "string" },
-                    "description": "The options as the agent wrote them, before markers were stripped."
-                  },
-                  "rule": {
-                    "type": "string",
-                    "enum": ["recommended_option", "first_option", "no_selection"],
-                    "description": "The mechanical rule that picked this answer, so a reader can audit the selection without rerunning it."
-                  },
-                  "chosen": {
-                    "type": "array",
-                    "items": { "type": "string" },
-                    "description": "The options selected, cleaned of their markers. Empty when the rule selected nothing."
-                  }
-                }
-              }
+            "rationale": {
+              "type": "string",
+              "description": "The responder's own one-line account of why it answered this way. Absent when it offered none."
             }
           }
         }

schema/evals.schema.json

@@ -136,8 +136,8 @@
       "properties": {
         "type": {
           "type": "string",
-          "enum": ["heuristic"],
-          "description": "Which responder answers the agent. 'heuristic' is deterministic and free: it answers a question that offers a list of options, and stops the run on anything else. Required rather than defaulted, because the responder decides what the agent hears."
+          "enum": ["llm"],
+          "description": "Which responder answers the agent. 'llm' consults a small model through the same harness as the agent under test, once after every round: it answers, judges the task finished, or stops the run rather than guessing. Required rather than defaulted, because the responder decides what the agent hears. Choose the model with 'run --responder-model'."
         },
         "max_turns": {
           "type": "integer",