feat(run): derive conversation turns from a heuristic responder

README.md

@@ -115,7 +115,9 @@ The command help and generated runbook describe baseline selection and the rest
 Each eval case runs once per condition and repetition in its own clean Git repository. The two arms
 receive the same task and fixtures; only the condition under test changes. Assertions can combine
 LLM judgment with runner-owned command checks, transcript checks, and final diff limits. Scripted
-`turns` resume one native harness session so follow-up answers remain part of the same conversation.
+Multi-turn evals resume one native harness session so follow-up answers remain part of the same
+conversation, whether the turns are scripted or derived by a responder (`eval-magic docs
+conversations`).
 
 Most harness features are declared in TOML descriptors. See the current registry and resolved data
 instead of relying on a static compatibility table:

docs/developer_overview.md

@@ -12,7 +12,8 @@ focused internal notes instead of duplicating their details.
 ## How an evaluation moves through the system
 
 1. `eval-magic init` scaffolds an eval workspace next to a skill. Eval definitions describe the
-   task, fixtures, assertions, conditions, run count, and optional scripted follow-up turns.
+   task, fixtures, assertions, conditions, run count, and — for a multi-turn eval — either scripted
+   follow-up turns or a responder policy that derives them.
 2. `eval-magic run` validates the configuration, resolves and copies the skill under test into the
    iteration, creates isolated task roots, stages the requested skill condition from that copy,
    snapshots the starting state, and writes `RUNBOOK.md`, `dispatch.json`, and related campaign
@@ -147,3 +148,5 @@ implementation evidence in an internal note.
   `eval-magic docs isolation`.
 - [Shipped codebase guide](guides/codebase.md) is the repository source for
   `eval-magic docs codebase`.
+- [Shipped conversations guide](guides/conversations.md) is the repository source for
+  `eval-magic docs conversations`.

docs/guides/byoh.md

@@ -135,9 +135,12 @@ Use this sequence:
 3. Run a small eval through `run`, dispatch, `ingest`, and `finalize`.
 4. Confirm that every declared enhancement was exercised by the smoke run.
 
-Scripted `turns` require `[conversation].resume_exec_template` plus transcript extraction of ordered
-assistant messages and the native session ID. There is no fresh-session fallback: `run` rejects the
-case when the harness cannot preserve the conversation.
+Multi-turn evals — scripted `turns` and `responder` alike — require
+`[conversation].resume_exec_template` plus transcript extraction of ordered assistant messages and
+the native session ID. There is no fresh-session fallback: `run` rejects the case when the harness
+cannot preserve the conversation. The responder itself needs nothing further from a descriptor; it
+reads the agent's message as Markdown, so it works on any harness that can resume. See
+`eval-magic docs conversations`.
 
 When a shadow preflight reports a live copy, isolate every initial and resumed eval-agent dispatch
 before setting `isolates_live_sources = true`. The per-harness remedies and verification procedure

docs/guides/conversations.md

@@ -0,0 +1,150 @@
+# Multi-turn conversations
+
+Most evals are one shot: the agent gets a prompt, works, and answers. Some tasks
+are not like that. A realistic request often needs a decision from the user part
+way through, and an eval that cannot supply one measures an agent talking to a
+wall.
+
+An eval declares one of two ways to supply those answers. They are alternatives,
+not layers — declaring both is a configuration error.
+
+- **`turns`** — an authored script. You say exactly what the user says, and in
+  what order. Use it when the exchange is the thing under test and you want it
+  identical in every run.
+- **`responder`** — a policy that derives each answer from what the agent just
+  said. Use it when you do not know what the agent will ask, which is the normal
+  case for a real task against a real codebase.
+
+Declaring neither leaves the eval one shot.
+
+Both need a harness that can resume its own session, so a follow-up reaches the
+agent that asked rather than a fresh one. `eval-magic run` rejects the eval up
+front when the selected harness cannot. `eval-magic harness list` names the
+`conversation-resume` capability for every harness that has it.
+
+## The responder
+
+```json
+{
+  "id": "add-request-caching",
+  "prompt": "Requests to the pricing API are slow. Can you add caching?",
+  "expected_output": "A working cache with the pricing endpoint under 100ms.",
+  "responder": { "type": "heuristic", "max_turns": 8 }
+}
+```
+
+- **`type`** is required. `heuristic` is the only responder today. It is
+  deterministic and costs nothing: it reads the agent's message and applies
+  fixed rules, with no second model involved.
+- **`max_turns`** bounds how many follow-ups the responder may synthesize. The
+  opening prompt is not one of them. It defaults to 8.
+
+Every turn the responder produces is recorded in the run's `conversation.json`
+with an `origin` naming the rule that produced it, so you can audit whether the
+responder distorted the run instead of taking the transcript on trust. The
+eval's own opening prompt carries no `origin` — that absence is how you tell an
+authored turn from a derived one.
+
+## What the heuristic answers
+
+The heuristic reads the last message of each round as Markdown and answers
+exactly one shape of question: **a list of options introduced by a question.**
+
+A list counts as a question when the line directly above it ends with a `?`:
+
+```
+Which cache should I use?
+
+- An in-process LRU (Recommended)
+- Redis
+```
+
+The `?` has to be the last thing said before the options appear. That is what
+separates a real question from a closing summary, which is also mostly a
+bulleted list and would otherwise be "answered" as though the finished task were
+still open.
+
+Given a list, the choice is mechanical:
+
+| The list | Recommendation marked | The answer |
+| --- | --- | --- |
+| plain (`-`, `*`, `1.`) | yes | the first recommended option |
+| plain | no | the first option |
+| checkboxes (`- [ ]`) | yes | every recommended option |
+| checkboxes | no | nothing — `None of these.` |
+
+Plain lists ask for exactly one choice; checkboxes ask for zero or more. That
+syntax is the only signal the heuristic uses to tell them apart.
+
+An option counts as recommended when it carries a standalone `recommended` in
+parentheses, brackets, or bold — `(Recommended)`, `[recommended]`,
+`**Recommended**` — or when it is a pre-checked box, `- [x]`.
+
+A message that asks more than one question is answered in one turn, numbered in
+the order the questions appeared.
+
+## How a conversation ends
+
+| Recorded as | When |
+| --- | --- |
+| `completed` | The agent's last message asked nothing. It considers the task done, so the run stops rather than burning its remaining turns. |
+| `stopped`, `responder_cannot_answer` | The agent asked something with no option list. |
+| `stopped`, `max_turns_reached` | The agent was still asking at the bound. |
+| `timed_out` | The task outran `dispatch --timeout`. |
+
+A `stopped` conversation is recorded, not failed: `dispatch` exits zero and
+`ingest` still records the run. But both responder stops end the conversation
+with the task unfinished, so `dispatch` warns about each one by name. Read the
+last assistant message before treating such a run as a data point beside a
+completed one.
+
+Two properties are worth knowing before you read results:
+
+- The heuristic never guesses. A question it does not recognize stops the run
+  instead of inventing an answer, because a fabricated answer would silently
+  change what the agent was asked to do.
+- It errs toward stopping. A question mark anywhere in an otherwise-finished
+  message stops the run rather than calling it complete. That costs a dispatch;
+  the alternative — recording a run as complete while the agent was still
+  waiting — would cost the result's credibility.
+
+Answering free-form questions needs a model, not rules. That is a separate
+responder, and until it ships, `responder_cannot_answer` is where those runs
+stop.
+
+## Cross-harness behaviour
+
+The heuristic reads plain Markdown out of the agent's message, so it needs no
+per-harness support: any harness that can resume a session can run a responder
+eval. Nothing is read from a harness-native question tool, and nothing needs to
+be, because a dispatch runs headless with no channel to answer such a tool on.
+
+The shapes above are a contract, not a description of one agent. An agent that
+offers options this way is answered; one that phrases them some other way stops
+the run. If you are bringing your own harness and its agent asks in a shape the
+table does not cover, that is a gap in the table, not in your descriptor.
+
+## Scripted turns
+
+```json
+{
+  "id": "clarify-before-editing",
+  "prompt": "The due date is wrong. Fix it.",
+  "expected_output": "Asks which timezone before editing.",
+  "turns": [
+    {
+      "prompt": "The affected users are all in US timezones.",
+      "deliver_when": "agent_asks",
+      "agent_response_matches": "(?i)time ?zone"
+    },
+    { "prompt": "It is a date-only field.", "deliver_when": "always" }
+  ]
+}
+```
+
+Each turn is delivered in order. `deliver_when: always` delivers
+unconditionally; `agent_asks` delivers only when the preceding response contains
+a question mark, and `agent_response_matches` adds a regex the response must
+also match. A turn whose gate is unmet stops the conversation and is recorded as
+`agent_did_not_ask` or `agent_response_mismatch` — a real result about the
+agent, which is usually the point of scripting the exchange.

docs/progressive-enhancements.md

@@ -10,7 +10,8 @@
 Harness compatibility is not a parity checklist to audit — it is **a minimal baseline every harness
 satisfies, plus optional enhancements** a harness's adapter opts into. Most missing enhancements
 have a documented lower-fidelity fallback. Native conversation resume is the deliberate exception:
-an eval that declares scripted `turns` is rejected when the harness cannot preserve one session.
+an eval that declares scripted `turns` or a `responder` is rejected when the harness cannot preserve
+one session.
 
 ## One dispatch mechanism
 
@@ -50,8 +51,8 @@ forces `--no-stage`; without a declared guard the run continues unguarded behind
 `detect-stray-writes` audit; requested models without a model flag are recorded as provenance
 only). Supported enhancements are provided automatically — the write guard auto-arms wherever a
 harness declares one and staging is active (`--no-guard` opts out). Only genuinely contradictory
-flag combinations stay errors. A selected eval with `turns` also requires `[conversation]`; no
-generic fresh-session fallback can preserve the meaning of a canned reply.
+flag combinations stay errors. A selected eval with `turns` or a `responder` also requires
+`[conversation]`; no generic fresh-session fallback can preserve the meaning of a follow-up reply.
 
 ## Where this lives in code
 
@@ -207,14 +208,33 @@ combination.
 *Why harness-specific:* each CLI spells same-session continuation differently and exposes its
 session identifier in a different transcript event.
 
-*What it unlocks:* an eval's ordered `turns` array. `dispatch` starts the normal one-shot command,
-extracts the native session id, evaluates `agent_asks` (`?`) plus the optional response regex, and
-resumes the same session for each delivered follow-up. It writes raw round transcripts
-under `outputs/turn-N/` and atomically commits `conversation.json` only after a complete or normal
-guardrail-stopped scenario. `ingest` skips an interrupted task with no completion artifact.
-
-*Fallback:* none. `run` rejects selected multi-turn evals when the harness omits this capability;
-silently starting a fresh session would make the canned user response meaningless.
+*What it unlocks:* an eval's ordered `turns` array **and** its `responder` policy. `dispatch` starts
+the normal one-shot command, extracts the native session id, asks the eval's turn source what
+follows each round, and resumes the same session for each delivered follow-up. It writes raw round
+transcripts under `outputs/turn-N/` and atomically commits `conversation.json` only after a complete
+or normal guardrail-stopped scenario. `ingest` skips an interrupted task with no completion
+artifact.
+
+A scripted turn is gated by `agent_asks` (`?`) plus the optional response regex. A responder instead
+*derives* each turn from the round's last assistant message and records the rule that produced it on
+the turn itself. **The responder needs no descriptor field and no named capability of its own:** it
+reads that message as plain Markdown — a question line followed by a list of options — so every
+harness that resolves a resume template gets it for free, and none can be "missing" it.
+
+That portability is not a happy accident, it is forced. A dispatch runs headless with stdin
+detached, so a harness-native question tool has no channel to be answered on; the runner can only
+send free text as the next user turn. Text is therefore the only mechanism that fits, and it is the
+one every transcript parser already normalizes into `final_text`.
+
+What *is* borrowed from one harness is the convention — `(Recommended)` and checkbox lists are how
+Claude Code's own question UI renders choices. The recognized shapes are documented as a
+harness-neutral contract in `eval-magic docs conversations`, not as "what Claude does": an agent that
+offers options that way is answered identically whatever harness runs it, and one that phrases them
+differently stops the run with `responder_cannot_answer` — a documented gap in the shape table, not a
+missing descriptor field. Widening the table is a runner change that benefits every harness at once.
+
+*Fallback:* none. `run` rejects selected multi-turn evals — scripted or responder-driven — when the
+harness omits this capability; silently starting a fresh session would make the answer meaningless.
 
 *Descriptor fields:* `[conversation].resume_exec_template`, with required
 `<eval-root>`, `<outputs_dir>`, `{session_arg}`, and `{prompt_arg}` placeholders, plus optional

harnesses/template.toml

@@ -141,8 +141,11 @@ label = "{label}"
 # plugin_version_field = "version"
 
 ## -------------------------------------------------------------------------------------------
-## [conversation] — native same-session continuation for scripted eval `turns`. This capability
-## has no generic fallback: run rejects multi-turn evals for a harness that omits it. It requires
+## [conversation] — native same-session continuation for multi-turn evals, both scripted `turns`
+## and a `responder` that derives them. The responder needs nothing further from a descriptor: it
+## reads the agent's own message as Markdown, so declaring this table is all it takes.
+## This capability has no generic fallback: run rejects multi-turn evals for a harness that omits
+## it. It requires
 ## [dispatch].exec_template plus transcript parsing that exposes both ordered assistant messages
 ## and the native session id. Named summary parsers provide those directly; a declarative extractor
 ## must declare [transcript.extract.assistant_messages] and [transcript.extract.session_id].

profiles/shared/runbook.md

@@ -21,7 +21,9 @@ each task's `conversation.json`. A task that already has one is skipped, so reru
 command retries only what did not finish. A task that exceeds `--timeout` is recorded as timed out
 rather than left to stall the campaign, and a task that fails is recorded and named while the rest
 of the batch continues. A conversation that stops at a scripted gate is valid eval data, not a
-failure.
+failure. A conversation the responder stopped — because it could not answer the agent's question,
+or because it hit `max_turns` — is recorded too, but it ended with the task unfinished; `dispatch`
+warns about each one by name, and those runs are weaker evidence than a completed one.
 
 ```
 {{INGEST_CMD}}

schema/conversation.schema.json

@@ -17,7 +17,7 @@
     },
     "stop_reason": {
       "type": "string",
-      "enum": ["agent_did_not_ask", "agent_response_mismatch"]
+      "enum": ["agent_did_not_ask", "agent_response_mismatch", "responder_cannot_answer", "max_turns_reached"]
     },
     "stopped_before_followup": {
       "type": "integer",
@@ -98,7 +98,51 @@
         "type": { "const": "user_message" },
         "ordinal": { "type": "integer", "minimum": 0 },
         "round": { "type": "integer", "minimum": 1 },
-        "text": { "type": "string" }
+        "text": { "type": "string" },
+        "origin": {
+          "type": "object",
+          "required": ["responder", "answers"],
+          "additionalProperties": false,
+          "description": "How a responder derived this turn. Absent on the eval's opening prompt and on scripted turns, which are authored rather than derived.",
+          "properties": {
+            "responder": {
+              "type": "string",
+              "enum": ["heuristic"],
+              "description": "Which responder produced the turn."
+            },
+            "answers": {
+              "type": "array",
+              "minItems": 1,
+              "description": "One entry per question the turn answered, in the order they were asked.",
+              "items": {
+                "type": "object",
+                "required": ["options", "rule", "chosen"],
+                "additionalProperties": false,
+                "properties": {
+                  "question": {
+                    "type": "string",
+                    "description": "The question line the options hung from."
+                  },
+                  "options": {
+                    "type": "array",
+                    "items": { "type": "string" },
+                    "description": "The options as the agent wrote them, before markers were stripped."
+                  },
+                  "rule": {
+                    "type": "string",
+                    "enum": ["recommended_option", "first_option", "no_selection"],
+                    "description": "The mechanical rule that picked this answer, so a reader can audit the selection without rerunning it."
+                  },
+                  "chosen": {
+                    "type": "array",
+                    "items": { "type": "string" },
+                    "description": "The options selected, cleaned of their markers. Empty when the rule selected nothing."
+                  }
+                }
+              }
+            }
+          }
+        }
       }
     },
     "assistantMessage": {