diff --git a/CHANGELOG.md b/CHANGELOG.md index a343dbd..32566d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,21 @@ commands, or moved skill-reference paths. ### Added +- **Checkable acceptance criteria, traced from the spec** (#69). `plan` + gives each task that users or callers can notice a `` block: + *Given* a starting situation, *When* an action, *Then* something you can + see or test, plus the proof (a named test, a flow, a lint rule, or a + manual check). Criteria for Swift packages use public functions; for + screens, what's on screen. Vague spec items are asked about before the + build, and answers are kept in `` (under + `autonomous`, the safest reading is recorded as an assumption to + confirm). `verify` now writes its own list from APP.md / RELEASE.md + **before** reading the plan, then traces every promised feature to a + criterion and its proof; a promise nobody checks is an issue even when + every task says completed. VERIFICATION.md gains a Spec Coverage table. + `build` writes the tests criteria name; `autonomous` verifies spec-first. + Older plans without criteria still verify, against their `` lines. + - **Shared command scripts, tested** (#112). The shell programs that were pasted into command text now live in `templates/_scripts/`, and the commands call them in one line: diff --git a/CLAUDE.md b/CLAUDE.md index e4c9d93..1fa31d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -168,9 +168,10 @@ The body defines expertise, code patterns, best practices, and a verification ch Templates in `templates/` use XML tags that commands parse. Key tags in PLAN.md tasks: ```xml - , , , , , , + , , , , , , , ``` +`` holds Given/When/Then acceptance criteria (`` with ``, ``, ``, ``), and a plan-level `` records answers to spec questions `plan` asked first. `/apple:verify` writes its own list from APP.md/RELEASE.md before reading PLAN.md, then traces each feature → criterion → proof (#69). Generator tasks add `` and `` tags. `type="auto"` tasks may carry an optional `model="opus"` attribute (set by `/apple:plan`, max 1–2 foundation tasks per phase) that `/apple:build` passes through as the spawn's per-call `model` override. Maintain tag consistency — commands match on these exact tag names. ### Adding a New Command diff --git a/commands/autonomous.md b/commands/autonomous.md index 1b9462a..5839361 100644 --- a/commands/autonomous.md +++ b/commands/autonomous.md @@ -106,6 +106,8 @@ command — follow that command's logic exactly. 2. **Plan** — execute the `/apple:plan [N]` logic. Generate `.planning/PLAN.md` for this phase, applying preferences and the conditional generator selection. + Unclear spec items are not asked mid-run: plan records its safest reading + as `source="assumed"` in ``. 3. **Build** — execute the `/apple:build` logic. Work through every pending task in order, spawning the matched agents (bare names; retry as `apple:` @@ -143,9 +145,13 @@ command — follow that command's logic exactly. 4. **Verify** — execute the `/apple:verify [N]` logic **through an independent verifier**: spawn a fresh `swift-generalist` (bare name; retry - `apple:swift-generalist`) with the phase's PLAN.md deliverables and have - *it* run the build + test suite and the deliverables checklist, reporting - PASS/FAIL per item with the command output as evidence. The context that + `apple:swift-generalist`). Give it APP.md / RELEASE.md and the phase's + ROADMAP.md entry **first**, and have it write its own list of what the + spec promises before it reads PLAN.md (verify Steps 1–2: spec first, + then trace each promise to a criterion and its proof). Then it runs the + build + test suite and the deliverables checklist, reporting PASS/FAIL + per item with the command output as evidence. Spec gaps count like any + other finding. The context that just built the phase grading its own output is the same trust gap `/apple:build` Step 4 closes for tasks — reopened at phase scope. Write the verifier's findings to `.planning/VERIFICATION.md`. If agents can't @@ -160,7 +166,8 @@ command — follow that command's logic exactly. the phase recap and again at the next checkpoint so a human runs them. 5. **Phase summary** — print a short recap (tasks completed, files changed, - quality-gate counts, verification result) and update `STATE.md` to mark the + quality-gate counts, verification result, and any `source="assumed"` spec + answers for a person to confirm) and update `STATE.md` to mark the phase completed. ### 3. Between-Phase Checkpoint diff --git a/commands/build.md b/commands/build.md index 60580a7..553c35c 100644 --- a/commands/build.md +++ b/commands/build.md @@ -260,7 +260,10 @@ Task({ 1. Follow the specified 2. Create/modify files listed in 3. Verify all checks pass - 4. Meet all criteria + 4. Meet all criteria, and every Given/When/Then: + write each test named in a in this task, with + that name, testing the behavior from outside (on screen for + via="screen", public functions for via="package") 5. Reuse — never reimplement a listed helper. If none fits, say so in your report before adding a new shared one. 6. If is present: tokens and .appText only, the listed diff --git a/commands/plan.md b/commands/plan.md index d565742..9e8ddbf 100644 --- a/commands/plan.md +++ b/commands/plan.md @@ -267,6 +267,78 @@ no design file (screens designed in SwiftUI previews or `/apple:prototype`). Commit `.planning/design/phase-N/` with PLAN.md (PNGs, `pin.tsv`). +## Acceptance Criteria (every phase) + +A `` line like "Profile screen works" can't be proven true or false. +Every task that delivers something a user or caller can notice also gets +**acceptance criteria**: short statements anyone can check, in a fixed +shape: + +- **Given** the starting situation +- **When** the user (or the calling code) does something +- **Then** something you can see or test happens + +Example: *Given* no internet, *When* the user taps Save, *Then* the note +stays in the list and a "Saved offline" message shows. + +**1. List what this phase promises.** Read the features this phase +delivers: the phase's entry in ROADMAP.md, matched to APP.md +`` (new apps), or to the RELEASE.md `` and `` +by id (updates). Each one needs at least one criterion. A bug's criterion +comes from its ``: *Given* the conditions that triggered it, +*When* the same action, *Then* the correct behavior. + +**2. Ask before writing, not halfway through the build.** If a feature is +too vague to write a criterion (for example "sync works": what happens with +no internet? which copy wins when both changed?), collect those questions +and ask them together with `AskUserQuestion` (up to 4 per call), each with +plain choices and the recommended one first. Record every answer in +PLAN.md's `` block so `/apple:verify` sees it. +Under `/apple:autonomous`, never ask: choose the safest reading (no data +lost, user told what happened), record it with `source="assumed"`, and the +run's next checkpoint lists the assumptions for a person to confirm. + +**3. Write the criteria.** Rules: +- **Checkable from outside.** On screen for app screens; through the public + functions for a Swift package (`Packages/*` or a local package in the + project). Never "the cache is updated" or "the view model holds X". + Say which with `via="screen"` or `via="package"`. +- **One behavior each.** Cover the empty, error and edit cases the + `` already list, not only the happy path. +- **Name the proof**, the thing that will show it's true: + - `test`: the test that checks it (name it; the task writes it) + - `flow`: a `` id that `/apple:walkthrough` drives + - `lint`: a lint rule + - `manual`: a person checks it during `/apple:verify` (only when nothing + automatic can; most screen behavior can be a UI test or a flow) +- Tasks that deliver nothing a user or caller can notice (project setup, + the Visual QA task) keep only ``. +- Ids run across the whole plan: `AC1`, `AC2`, …. + +```xml + + + [question] → [answer] + + + + ... + + + [starting situation] + [action] + [what you can see or test] + [test name | flow id | rule | what the person checks] + + + ... + +``` + +Place `` after `` (and ``), before +the tasks. Before writing PLAN.md, check: every feature from step 1 has at +least one criterion; if one doesn't, write it or ask. + ## Task Generation Create `.planning/PLAN.md` with detailed tasks: @@ -484,6 +556,14 @@ tasks get zero tags — that is the common case. App builds with navigation in place Navigation works correctly + + + A first launch with no saved data + The app opens + The main screen shows its empty state with one clear first action + F1 + + - Navigation structure matches app type - Programmatic navigation works @@ -533,6 +613,14 @@ Generate tasks based on: Generated code compiles in the app target Settings screen renders and toggles respond + + + Appearance is set to Dark + The user quits and reopens the app + The app opens in Dark appearance + SettingsUITests.testAppearancePersists + + - Settings accessible from navigation - All preferences persist @@ -555,6 +643,8 @@ Before printing the completion message, append one `"event":"outcome"` line to t ✅ Phase [X] plan created! Created: .planning/PLAN.md with [N] tasks +Checkable criteria: [n] covering [n of m] features this phase promises +[Questions answered: n | Assumptions to confirm: n (listed in )] Tasks: 1. [Task 1 name] (type: [auto/manual/generator][; append ", model: opus" if tagged]) diff --git a/commands/verify.md b/commands/verify.md index e68059e..b5b33bd 100644 --- a/commands/verify.md +++ b/commands/verify.md @@ -9,44 +9,88 @@ model: sonnet You are a quality assurance expert verifying that completed work meets the planned deliverables. Your goal is to ensure everything works before moving to the next phase. -## Step 1: Load Context +## Step 1: Write Your Own List From the Spec (before reading the plan) -Read the relevant planning files: +The plan was written by the same kind of model that did the work. If the +plan forgot something, checking only the plan forgets it too. So first, +**without opening PLAN.md**, read what was promised: ``` -Read: .planning/PLAN.md # Current phase tasks and deliverables -Read: .planning/STATE.md # Current state and completed tasks -Read: .planning/APP.md # App specification for context +Read: .planning/APP.md # new apps: +Read: .planning/RELEASE.md # updates: , (optional) +Read: .planning/ROADMAP.md # which of those this phase delivers ``` If a phase number is provided as argument, verify that specific phase. -## Step 2: Extract Deliverables +For each feature or bug this phase delivers, write down in your own words +what a person should be able to see or do when it works: the normal case, +plus the empty, error and edit cases where they apply. Use the same +Given / When / Then shape the plan uses. Keep this list; it goes into the +report unchanged as **Expected from the spec**. -From PLAN.md, extract all **testable deliverables**: +## Step 2: Read the Plan and Compare -1. **Files created** - Do they exist? -2. **Features implemented** - Do they work? -3. **Tests written** - Do they pass? -4. **UI components** - Are they accessible and correct? +``` +Read: .planning/PLAN.md # tasks, , +Read: .planning/STATE.md # current state and completed tasks +``` -Create a verification checklist: +**Trace every promise to its proof.** For each feature on your Step 1 list: + +1. Find the plan's `` entries for it (matched by `feature`). + Apply ``: an answer there settles what the spec + meant. Note any `source="assumed"` items; a person still has to confirm + them. +2. Compare them with your list. Something you expected that no criterion + covers is a **gap in the plan**, even if every task says completed. +3. Check each criterion's `` exists: the named test is in the test + target, the flow id is in ``, the lint rule is in + `.swiftlint.yml`. A criterion whose proof doesn't exist is **unproven**. +4. Criteria with `via="package"` are checked through the package's public + functions (its tests); `via="screen"` through what's on screen (UI test, + walkthrough, screenshot, or a manual check). + +How these count: + +| Finding | Severity | Effect on the result | +|---------|----------|----------------------| +| A `must` feature (or `crash`/`high` bug) has no criterion | High | FAIL | +| A `should`/`could` feature or lower bug has no criterion | Medium | PARTIAL | +| Something you expected is missing from the criteria | Medium (High if it loses data) | PARTIAL | +| A criterion's proof doesn't exist | Medium | PARTIAL | +| A criterion's proof runs and fails | as the failure | FAIL | +| An assumed clarification is still unconfirmed | Low | ask in Step 4 | + +**Older plans with no ``:** don't fail them for the format. Trace +the spec list to the `` lines instead, mark each feature "no +checkable criteria (older plan)", and turn your Step 1 statements into +the manual checks in Step 4. + +Then build the checklist. Each criterion becomes a check of its proof type: +`test` runs in 3.2 (name the tests), `lint` in 3.5, `flow` through +`/apple:walkthrough` results or Step 4, `manual` in Step 4 (ask it as +written: "Given …, when you …, do you see …?"). Gaps go straight to +Issues Found. ```markdown ## Deliverables to Verify +### Spec coverage +- [ ] [Feature] → AC[n] → [proof] +- [ ] [Feature] → ⚠️ no criterion (gap) + ### Automated Checks - [ ] Project builds without errors -- [ ] All tests pass +- [ ] All tests pass (including every test named in a criterion) - [ ] No compiler warnings - [ ] Lint gate passes (if .swiftlint.yml exists) - [ ] Coverage gate passes (if .coverage-baseline exists) - [ ] [Feature X] file exists at [path] -- [ ] [Feature Y] file exists at [path] ### Manual Checks -- [ ] [Feature X] works as expected -- [ ] [UI Component] displays correctly +- [ ] AC[n]: Given [..], when [..], then [..] +- [ ] Confirm assumption: [clarification] - [ ] [User flow] completes successfully ``` @@ -322,6 +366,7 @@ Create `.planning/VERIFICATION.md`: ## Summary +- **Spec coverage:** X of Y features proven ([n] gaps, [n] unproven) - **Automated Checks:** X/Y passed - **Manual Checks:** X/Y passed - **Issues Found:** X @@ -355,6 +400,19 @@ Create `.planning/VERIFICATION.md`: Change requests, not defects: [next phase | next release | /apple:redesign] +## Spec Coverage + +Every promise in the spec, what checks it, and whether it held. Written +from APP.md / RELEASE.md before reading the plan. + +| Feature (spec) | Criterion | Proof | Result | +|----------------|-----------|-------|--------| +| [Offline notes] | AC3: Given no internet, when Save is tapped, then the note stays and "Saved offline" shows | test `NotesTests.testOfflineSave` | ✅ Passed | +| [Offline notes] | ⚠️ none: what happens when the same note changed on two devices? | — | ❌ Gap in the plan | +| [Export] | AC5: … | manual | ⚠️ Not run | + +**Assumptions to confirm:** [list from `` | none] + ## Manual Checks (UAT) | Feature | Status | Notes | @@ -421,6 +479,7 @@ Verification complete! Status: [PASS / FAIL / PARTIAL] ([everything works | works, with gaps listed below | something important is broken]) Results: +- Promised features proven: X of Y [(n not covered by any check)] - Automated: X/Y passed - Manual: X/Y passed - Issues: X found diff --git a/templates/PLAN.md b/templates/PLAN.md index 7ba8509..c73786b 100644 --- a/templates/PLAN.md +++ b/templates/PLAN.md @@ -31,6 +31,12 @@ Copy this file to your project's `.planning/PLAN.md`. + + + [question] → [answer] + + + + [starting situation] + [what the user or calling code does] + [what you can see or test] + [test name | flow id | rule | what a person checks] + + + - [Criterion 1] diff --git a/templates/VERIFICATION.md b/templates/VERIFICATION.md index 89e477c..bbfa8a5 100644 --- a/templates/VERIFICATION.md +++ b/templates/VERIFICATION.md @@ -16,6 +16,7 @@ Copy this file to your project's `.planning/VERIFICATION.md` after running verif [1-2 sentence summary of verification results] +- **Spec coverage:** X of Y promised features proven ([n] gaps, [n] unproven) - **Automated Checks:** X/Y passed (X%) - **Manual Checks:** X/Y passed (X%) - **Critical Issues:** X @@ -23,6 +24,23 @@ Copy this file to your project's `.planning/VERIFICATION.md` after running verif --- +## Spec Coverage + + + +What each promised feature needs, what checks it, and whether it held. + +| Feature (spec) | Criterion (Given / When / Then) | Proof | Result | +|----------------|--------------------------------|-------|--------| +| [Feature] | AC[n]: Given [..], when [..], then [..] | [test name / flow id / lint rule / manual] | ✅ Passed / ❌ Failed / ⚠️ Not run | +| [Feature] | ⚠️ none: [what the spec promises that no criterion covers] | — | ❌ Gap in the plan | + +**Assumptions to confirm:** [from `` | none] + +--- + ## Automated Checks ### Build Verification