Repository navigation
Handoff tells the agent to edit the spec but gives no way to revalidate it #71
Description
What
The handoff's "When a Request Is Not in the Spec" section (issue #65) says: once the builder agrees, update the spec files first, then implement. A follow-up trial on 2026-10-12 found agents do that, and that half of the hand-edited packages no longer pass validation.
Evidence
Six isolated headless two-turn runs of claude-sonnet-5-5 on the tea-tracker package with the current handoff. Turn 1: build the data layer and include an is_favorite boolean on Tea. Turn 2: "Yes, add it. Go ahead."
| Runs | |
|---|---|
| Turn 1: withheld the field and asked | 6 of 6 |
| Turn 2: edited spec files before any code | 6 of 6 |
| Turn 2: spec edits after code | 0 of 6 |
Edited entities.yaml and endpoints.yaml |
6 of 6 |
Also edited screens.yaml (a star toggle nobody asked for by name) |
5 of 6 |
Edited package still passes validate_spec.py |
3 of 6 |
All three failures are the same two checks: the agent made tea.is_favorite required on the entity with a default, and optional in the create-tea request. That is a reasonable design the validator rejects ("entity field is required but endpoint field is optional").
Also stale after every run, by construction: validation-report.yaml, manifest.yaml, docs/prd.md and the handoff's own "Spec status" line.
Why it happens
- The handoff never says to validate after editing the spec.
- The validator is not in the handed-off folder. It lives in the plugin at
plugin/scripts/validate_spec.py. - The agents in the trial had file tools only, so they could not have run it anyway. An agent with a shell was not tested.
Options
- Handoff tells the agent to state the spec edits and ask the builder to re-run
/compile, which revalidates and regenerates the report, manifest and handoff. Smallest change; adds a builder step. /compilecopies the validator into the output folder and the handoff tells the agent to run it after any spec edit.- Handoff states the consistency rules an edit most often breaks (required-ness must match between entity and create endpoint).
Option 1 keeps the package the single compiled source of truth (ADR 0010). Needs an owner decision.
Limits
One fixture, one model, one request, six runs, file tools only. Total cost $2.59.
Activity
chrispezza commentedon Oct 12, 2026
Closed-loop trial, 2026-10-12
Six isolated headless three-turn runs of
claude-sonnet-5-5with the merged handoff (plugin 0.5.3) on the tea-tracker package.is_favoriteboolean on Tea.data_modelsandapi_contractsartifacts and compiled withcompile_spec.py(20 checks, 0 failed). The field was deliberately namedis_starred, with no screen changes, so it differs from every agent's request. The recompiled package and handoff replaced the originals.is_starredin the migration and the typesis_favoriteleft in codeis_favorite, did not find it, concluded the recompile had not been applied, and stopped to ask. It did not re-readentities.yaml. A wrong diagnosis with a safe outcome.Limits: one fixture, one model, one request, six runs, file tools only. Total cost $4.10.
chrispezza commentedon Oct 13, 2026
Closed-loop rerun with the re-read step, 2026-10-12
Same setup as the closed-loop trial above, with the 0.5.4 handoff (PR #74) in both the starting package and the recompiled one. Six isolated headless three-turn runs of
claude-sonnet-5-5; the builder's recompiled package names the fieldis_starredand the third message does not mention the rename.entities.yamlis_starredin the migration and the typesis_favoriteleft in codeThe miss did not recur. Five against six on six runs per column is a difference of one run, so this shows the step does no harm and is consistent with it helping; it does not establish that it was the cause.
ADR 0010's last amendment still says the step is unmeasured; this comment is the record. Total cost $3.83.