Authoring: derive flows from real material, with an acceptance gate that the spec can actually fail

[AminChirazi]

Writing up a design discussion so it is actionable. The friction being solved: authoring a .flow.yaml by hand is real work, and the prompts people invent for specs are not the prompts users type.

The reframe: a spec has two halves, and different material feeds each

This is the part I would build around, because treating all input as "material to transform" hides where the difficulty is.

• A recording gives you the PROMPT. A real session shows what a user actually asked, in their phrasing. That is strictly better than an invented task, and it is cheap to extract because the session is already structured.
• A requirements document gives you the ASSERTION. A requirement states a BELIEF ("the agent must not export while in plan mode"), and a belief is exactly what an assert_* line encodes.

Neither substitutes for the other. A recording shows behaviour and contains no belief; a requirement states a belief and does not know how the system is driven. So the pipeline is two extractions feeding two fields, not one transform.

Evidence that real material beats invented material, from an adopter: the FIRST real recording of an agent flow found a self-contradiction in a tool description that nobody had spotted by reading it - the description asked for "the existing scenario names" as choices four lines before forbidding the enumeration that produces them. An invented prompt would not have surfaced it, because whoever invents the prompt also invents what they expect.

Video is the wrong input, and this repo already says so

docs/recording.md states the position: "Structured data is the machine surface; video is the human surface. Nothing programmatic ever parses pixels." The non-goals are explicit: "any agent-facing video parsing (never planned)."

That is right, and for agent flows it is not close. What is under test is which model calls were made and which tools were invoked. A video shows that indirectly, through pixels, and would need OCR to recover what already exists losslessly as JSON. Session export is the same idea without the loss.

The acceptance gate: can the generated spec FAIL?

A model will happily produce a plausible assertion that is subtly false, and the result is a green flow that asserts nothing - worse than no flow, because it looks like coverage.

Three real instances, all from one adopter in one week:

1. A spec asserted the agent must not enumerate; the tool description made enumerating mandatory. Plausible, and wrong.
2. Two tool policies shipped completely inert behind six green unit tests, because the tests were written from the same mental model as the code.
3. A hand-written audit table was wrong three ways within an hour of being written.

So the gate on a generated spec should not be "does it parse" or even "does it pass". It should be "can it fail?" - generate the spec, generate a deliberately broken variant of the system, and require the flow to go red against it. Only then is it accepted. A spec nobody has seen fail is a guess about what it covers.

This is a natural fit here because the machinery exists: recording an adversarial variant is the same code path as recording the intended one.

It must be allowed to refuse

Most requirements are not assertable at the boundary a flow can reach. In the adopter case, governance (kill switch, volume caps, write approval) executes INSIDE the MCP server, so a cassette replaces the very thing under test and can never assert it. Roughly two thirds of that project's real bugs were found by reading enforcement code rather than by replay, and no amount of spec generation would have changed that.

A pipeline that always emits a spec will emit specs for things the harness structurally cannot check, and those come back green forever. "Not assertable at this boundary, and here is why" has to be a first-class output, alongside a pointer to where the check belongs instead (a unit test, an integration test, or a structural change that makes the case impossible).

Shape

A skill plus a validation loop, not a framework. The pieces mostly exist:

1. The spec schema (already in flowproof-trace).
2. A generator that emits a candidate flow from (session | requirement) input.
3. flowproof record for the intended run, and again for an adversarial variant.
4. The accept/reject gate above.
5. The existing structured "here is what I would need to know" payload for ambiguity, rather than guessing when input is thin.

Explicitly NOT worth building: a bespoke transform framework for arbitrary "materials", and video parsing.

Open question

Whether the adversarial variant is generated (mutate the system under test) or declared (the author states what should break it). Generated is more automatic; declared is more honest, because stating what would falsify a claim is most of the thinking that makes the claim worth asserting.

[AminChirazi]

Proposed scope — for approval or cuts, no code written

The design above is sound and I am not relitigating it. What it lacks is a plan: what gets built first, what "done" means for each piece, and where it stops. Here is one.

What this session added to the argument

The acceptance gate this issue proposes — "can the generated spec fail?" — is the same idea the falsifiability suite just implemented one level down, at the assertion level. That suite is now merged, and it produced hard evidence for the premise:

• Two real defects, neither of which any existing test would have caught. capability-error slipping the audit --since gate, and the MCP lane silently dropping a tool call the agent really made while record exited 0.
• Five of twelve issues were duplicates or already true. I filed them from a list of assertions rather than from the suite that tests them — a generator reading requirements rather than the codebase will make exactly that mistake at scale, which is the case for the refusal path being first-class rather than an afterthought.
• One issue filed on a misreading (Falsifiability: prove a real model key never reaches the agent's environment #278): a changelog line describing the state before a fix, read as a standing gap. Prose is a lossy input.

So the gate is not speculative. The suite is the working precedent, and docs/how-flowproof-tests-flowproof.md is the pattern to copy.

Phase 1 — the refusal path, first and alone

Deliberately inverted from the obvious order. Build the thing that says "not assertable at this boundary, and here is why" before building the thing that emits specs.

Exit criteria:

• given a requirement, classify it as assertable-at-the-model-boundary, assertable-at-the-MCP-boundary, or not assertable here;
• for the third case, emit the reason and where the check belongs instead — unit test, integration test, or a structural change that makes the case impossible;
• a corpus of ~10 real requirements, hand-labelled, with the classifier's disagreements reviewed rather than tuned away.

Why first: a generator that always emits will emit specs for things the harness structurally cannot check, and those come back green forever. That is the failure this repo exists to prevent, reproduced by the tool meant to reduce work. Shipping the emitter first and the refusal later means shipping the failure and fixing it after adopters have specs they trust.

Phase 2 — prompt extraction from a session

The cheap half, and the one with an existing oracle.

Exit criteria: given a session export, produce the prompt: step in the user's own phrasing; a flow authored this way records and replays; no pixel parsing anywhere near it (docs/recording.md non-goals stand).

Phase 3 — the acceptance gate

Generate the spec, generate a deliberately broken variant of the system, require the flow to go red against it. Accept only then.

Exit criteria: the gate rejects a spec that passes against both the intended and the broken variant; a rejection names which assertion could not be made to fail. Reuses flowproof record on an adversarial variant — the same code path as recording the intended one, which is why this is tractable rather than a research project.

Phase 4 — assertion extraction from requirements

Last, because it is the part most likely to produce confident nonsense, and Phases 1 and 3 are what make its output safe to trust.

Out of scope, stated so it is not re-proposed

• video or pixel parsing — docs/recording.md: "Structured data is the machine surface; video is the human surface. Nothing programmatic ever parses pixels." For agent flows it is not close: what is under test is which model calls and tool invocations happened, and that already exists losslessly as JSON;
• a bespoke transform framework for arbitrary "materials". A skill plus a validation loop, per the design above;
• auto-accepting a generated spec. A spec nobody has seen fail is a guess about what it covers. That is the whole thesis, and it applies to generated specs most of all.

The open question I cannot answer

Whether this is worth building before there are external adopters authoring flows. It reduces authoring friction — but flowproof's authoring friction is currently my friction and yours, not a user's. If the answer is "not yet", Phase 1 alone is still worth having, because the refusal classifier is useful to a human writing specs by hand today.

Post-approval this becomes a milestone in CHARTER.md §4, which is a human act. Nothing here is a loop task yet.