diff --git a/CHANGELOG.md b/CHANGELOG.md index 216a8b28..fea1921b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,21 @@ All notable changes to the claude-plugins project will be documented in this fil The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Entries are listed newest-first; each plugin section is treated as released when merged to `main`. +### vibe v1.0.53 + +#### Added +- A startup version check (operator ruling, 2026-10-16). Every `$vibe` start, new or resume, runs `scripts/plugin-version.mjs`, which compares the running plugin's own manifest (`.codex-plugin` in Codex, `.claude-plugin` in Claude Code) with the same manifest on claude-plugins `main`. When the install is older, the person is asked "keep going with this one, or update now?"; on update the orchestrator runs the harness's update commands (`claude plugin marketplace update closedloop-ai` and `claude plugin update vibe@closedloop-ai`; in Codex `plugin marketplace upgrade closedloop-ai` then `plugin add` for vibe and closedloop-core, or a `git pull --ff-only` of a local folder marketplace, as `INSTALL.md` describes) and asks them to reopen the app. The check never blocks: no network, a refused, slow (5 second limit) or garbled answer, or a missing manifest prints status `unknown`, exit 0, and the session carries on and says so. `guardrails.md` ("Plugins and tooling") allows exactly this person-chosen update and nothing else. +- A precedent check before any product question (operator ruling, 2026-10-16). When the graph does not settle how the product should behave in a situation, how a mature app handles it is the default answer: Asana first, then another established app with the same flow (for example Linear, or GitHub's own authorization screens), then their public help centers. `scripts/precedent-check.mjs` holds the research brief and its safety rules once: in Claude Code it starts a separate headless `claude -p --chrome` run limited to the Chrome tools, web search and fetch (a session's Chrome tools are bound when it starts); in Codex it prints the brief the orchestrator follows in its own in-app Browser tab or with Computer Use, and `vibe-requirements-worker` gains a precedent mode for the help centers when no browser can look. The brief never touches existing data, keeps test data in one private `zz vibe precedent check (temp)` project with `zz test` tasks that is deleted and confirmed gone, cancels any connect, invite, share, pay or settings flow at its confirmation step, verifies every state before recording it, separates direct observation from help-center material and reports `undocumented` rather than guessing. The answer and its source (`Asana, observed `, a help-center URL or a ticket slug) are recorded in the session decision table as a default, never a human ruling or user-visible words; only when nothing settles it is the person asked. +- Visual QA during the session (operator ruling, 2026-10-16). `quality-loop.md` gains "Visual QA during the session": at each visible milestone (a user-visible change first ready to look at, or a later major visible change, never a small edit) the ready message offers visual QA through `session-questions.mjs` (key `visual-qa:`), and "looks good, make it polished" runs it before handoff without asking, unless the change shows nothing visible or the person declined it for that version. It runs the repository's design pack visual QA skill (`/visual-qa`, `$visual-qa`) between requests after a deploy: against the session's acceptance criteria when the ticket or its PRD has them, otherwise as the design pack's best-practice review against `.claude/design/soul.md`. Findings go to the same writer as one fix request. + +#### Changed +- The build loop is otherwise unchanged: visual QA still never runs inside a request's build or holds a deploy back, and the reference check stays the loop's one exception. The vibe skill (sections 1, 5 and 6, "Questions are asked once"), `guardrails.md`, the handoff skill (its visual QA now uses the same criteria-or-design-pack basis, and its question gate includes the precedent check), the change and requirements workers, `INSTALL.md` and the README follow, and sentences saying visual QA runs only at handoff are reworded. + +#### Tests +- `plugin-version.test.mjs` (8 cases): outdated with each harness's commands (Git and local Codex marketplaces), current and newer, each harness reading its own manifest, unreachable, 404, garbled, unversioned and slow marketplaces all `unknown` with exit 0, a missing manifest or runtime, and quoting of paths with spaces. `precedent-check.test.mjs` (6 cases): the brief's order and every safety and evidence rule, the headless Claude arguments and stdin, and failed, empty, slow and missing runs never counting as an answer. +- `test_vibe_visual_qa_precedent_version_contract.py` (11 cases) pins the prose in the skills, references and workers; all 11 fail on 1.0.52. `test_vibe_test_repair_lean_loop_contract.py` now pins "run at `$handoff`, never during the loop" and keeps the QA skill names out of the lean loop section, since visual QA may now also run outside the loop. +- 15 eval cases tagged `precedent`, `visual-qa` and `version-check` (Claude, and Codex through the runner; the Codex count test moves to 48). With Sonnet in Claude and the runner's default model in Codex, all 15 pass in both harnesses here. On 1.0.52 the four visual QA offer and polish cases, the outdated and unreachable version cases and the Asana-first case fail in both harnesses; the update-done case fails there in Claude (no reopen step) and in Codex only on its tool-call cap. The graph-answered, no-precedent, small-edit, declined and current-version cases pass on 1.0.52 too, as expected, since they guard behavior 1.0.52 already had or never offered; so do the two Codex precedent follow-up cases, whose session state already describes the precedent steps (the recorded-answer one failed one of two Claude runs on 1.0.52). + ### vibe v1.0.52 #### Fixed diff --git a/plugins/vibe/.claude-plugin/plugin.json b/plugins/vibe/.claude-plugin/plugin.json index 3209670c..ba259bc3 100644 --- a/plugins/vibe/.claude-plugin/plugin.json +++ b/plugins/vibe/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "vibe", "description": "Vibe sessions, handoff, seed refresh, and ClosedLoop ticket workflows", - "version": "1.0.52", + "version": "1.0.53", "dependencies": ["closedloop-core"], "author": {"name": "ClosedLoop", "email": "support@closedloop.ai"} } diff --git a/plugins/vibe/.codex-plugin/plugin.json b/plugins/vibe/.codex-plugin/plugin.json index faae5891..3e999c9d 100644 --- a/plugins/vibe/.codex-plugin/plugin.json +++ b/plugins/vibe/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "vibe", - "version": "1.0.52", + "version": "1.0.53", "description": "Vibe sessions, handoff, seed refresh, and ClosedLoop ticket workflows", "author": {"name": "ClosedLoop", "email": "support@closedloop.ai"}, "homepage": "https://github.com/closedloop-ai/claude-plugins", diff --git a/plugins/vibe/README.md b/plugins/vibe/README.md index 53488d64..b3c9a25f 100644 --- a/plugins/vibe/README.md +++ b/plugins/vibe/README.md @@ -37,8 +37,18 @@ commit, deploy, and showing the person. Each finished change deploys to the session's Vercel preview without the person asking ("redeploy" still works), and changes that arrive during a deploy go out together in the next one. The plan gets no review, and browser -E2E, visual QA, logical QA, product-contract QA, the guardrails and adversarial -reviews and other heavy validation run only at handoff. Before saying a +E2E, logical QA, product-contract QA, the guardrails and adversarial +reviews and other heavy validation run only at handoff. Visual QA runs at +handoff too, and during the session only when the person wants it (operator +ruling, 2026-10-16): at each visible milestone (a user-visible change first +ready to look at, or a later major visible change, never a small edit) the +ready message offers it, and "looks good, make it polished" runs it before +handoff unless the change shows nothing visible or the person already +declined it for that version. It runs between requests after a deploy, +through the repository's design pack (`/visual-qa`, `$visual-qa`): +against the acceptance criteria when the ticket or its PRD has them, and as +the design pack's best-practice review against `.claude/design/soul.md` +otherwise. Its findings go to the same writer as a fix. Before saying a change is ready, the writer checks every acceptance criterion it touches (from the linked ticket, its PRD and the person's own words, kept in the session decision table) with an existing test, a quick script, one control @@ -52,7 +62,8 @@ it read before building, and every visual and UX property it shows becomes an acceptance criterion mapped to a design token or catalog component. Only when there is a reference image, the writer screenshots each touched screen before saying it's ready and fixes every difference from the reference, the one -exception to the lean loop's no-visual-QA rule (operator ruling, 2026-10-15). +exception inside the lean loop to its no-visual-QA rule (operator ruling, +2026-10-15). Every request also uses `$decision-table` or `/closedloop-core:decision-table` before code, including frontend-only, prototype and trivial edits. One living @@ -143,9 +154,28 @@ only from the successful discard receipt. Technical plans, questions, and upfront summaries remain internal. Updates identify only completed requested features and next work in the person's own words. Product questions are researched through closedloop-graph and live -decision evidence first, then raised one at a time with full context only -when an unresolved answer is absolutely necessary. Existing product and design -approvals remain intact. +decision evidence first, then through how a mature app handles the same +situation (operator ruling, 2026-10-16): Asana first, then another +established app with the same flow (Linear, or GitHub's own screens), then +their public help centers when no browser can look. `scripts/precedent-check.mjs` +holds the research brief and its safety rules: a separate headless +`claude -p --chrome` run in Claude Code, the orchestrator's own in-app Browser +or Computer Use in Codex; nothing that exists is touched, test data lives +only in a `zz vibe precedent check (temp)` private project that is deleted +afterwards, and every state is verified before it is recorded. The answer +and its source (`Asana, observed `, a help-center URL, or the ticket +slug) go into the session decision table. A question is raised, one at a +time with full context, only when none of these settles it and the answer +is absolutely necessary. Existing product and design approvals remain intact. + +Every `$vibe` start checks the installed vibe against the version on +claude-plugins `main` (`scripts/plugin-version.mjs`). When it is older, the +person chooses to keep going or update now; on update the orchestrator runs +the harness's update commands (`claude plugin marketplace update` and +`claude plugin update` in Claude Code, `codex plugin marketplace upgrade` +and `codex plugin add` for vibe and closedloop-core in Codex) and asks them +to reopen the app. When the check cannot reach GitHub the session carries +on and says so. Two notices reach the person without waiting for a reply. When a request names an outside product as its reference, or gives a screenshot of one, the @@ -274,7 +304,7 @@ The plugin bundles the ClosedLoop ticket automation skill pack: `cl-policy`, `cl | Agent | Model | Responsibility | |---|---|---| | `vibe-setup-worker` | sonnet | Machine preflight, worktree bootstrap, owned local processes, and confirmed session discard | -| `vibe-requirements-worker` | sonnet | Read-only ticket, requirements, overlap, and owning-route discovery | +| `vibe-requirements-worker` | sonnet | Read-only ticket, requirements, overlap, and owning-route discovery, reference-product research and help-center precedent research | | `vibe-ticket-worker` | sonnet | Live ticket creation, next-owner lookup, handoff grading, assignment, and cancellation | | `vibe-environment-worker` | sonnet | Vercel environment creation, redeploy, flag snapshots, Desktop setup, deployment records, and opening a pull request the person asked for at handoff | | `vibe-change-worker` | sonnet | Sole persistent implementation writer for local planning, full-stack changes, stories, fixes, the handoff missing tests list and serialized recording | @@ -351,6 +381,16 @@ check that also fails on the captured main is recorded on the ticket's Engineering checklist and the handoff goes on, and a session-caused failure goes back to the writer. +The cases tagged `precedent`, `visual-qa` and `version-check` rehearse the +2026-10-16 operator rulings: a product question the graph or a precedent +already answers is not asked, Asana is checked before another app and a +question is asked only when neither settles it; visual QA is offered at a +visible milestone and not on a small edit, a polish request runs it before +handoff (against the criteria, or the design pack's review without them) +and skips it after the person declined; and the startup version check asks +"keep going or update" when outdated, says nothing when current, carries on +when GitHub cannot be reached, and asks for a reopen after updating. + ## Runtime Files Version `1.0.14` contains 10 skills and 16 agents. It has no standalone diff --git a/plugins/vibe/agents/vibe-change-worker.md b/plugins/vibe/agents/vibe-change-worker.md index 1b9c85b8..54d33d6b 100644 --- a/plugins/vibe/agents/vibe-change-worker.md +++ b/plugins/vibe/agents/vibe-change-worker.md @@ -109,7 +109,13 @@ Research any product uncertainty through graph and live prior decisions first. Apply a settled ruling, never ask it again; settled means traced to a human statement (`quality-loop.md`, "Communication and product questions"). Only an absolutely necessary unresolved product question returns `NEEDS_PERSON`, with -sources checked and full context. Resolve technical ownership/backend/adapter +sources checked and full context; before asking, the orchestrator checks how +Asana or another established app handles it (`quality-loop.md`, "Precedent +before asking"). A precedent answer it returns is the default you build to: +record it in the session decision table as the decision for the affected +rows with its source (`Asana, observed `, `, observed ` or +the help-center URL), never as a human ruling, and never take its words as +copy. Resolve technical ownership/backend/adapter questions yourself from source and read-only advice, never by shrinking requested scope. @@ -258,7 +264,9 @@ fixture your change broke, repair it under "Tests: repair, don't author" below. Fix your implementation, never suppress or weaken a check. Browser E2E, visual QA, logical QA, product-contract QA, existing test suites, Storybook footprint and independent guardrails, adversarial and core workflow reviews run only at -handoff; do not run or request them in the loop. The publication gate's own +handoff; do not run or request them in the loop. Visual QA the person accepts +during the session runs outside your turns (`quality-loop.md`, "Visual QA +during the session"); its confirmed findings reach you as a fix request. The publication gate's own fixed validation (`environment.md`) is separate and unchanged. At handoff apply their confirmed corrections here and have the changed result rechecked. Automated browser checks there are headless; Electron uses the supported displayless diff --git a/plugins/vibe/agents/vibe-requirements-worker.md b/plugins/vibe/agents/vibe-requirements-worker.md index 44ab7820..5cf7f3f7 100644 --- a/plugins/vibe/agents/vibe-requirements-worker.md +++ b/plugins/vibe/agents/vibe-requirements-worker.md @@ -1,6 +1,6 @@ --- name: vibe-requirements-worker -description: Turns a ClosedLoop ticket or a plain description into vibe-session requirements for symphony-alpha. Reads the ticket with its PRD, plan, and related tickets through closedloop-graph and the ClosedLoop MCP, checks for existing or overlapping work, locates the route and FEATURE_MAP id where the relevant code lives (for change workers), and returns a short brief plus any questions only the person can answer. In reference mode, researches an outside product the person named or showed in a reference screenshot, reads the image itself, and returns the behaviors plus the visual and UX requirements to build, mapped to the design system. Read-only. +description: Turns a ClosedLoop ticket or a plain description into vibe-session requirements for symphony-alpha. Reads the ticket with its PRD, plan, and related tickets through closedloop-graph and the ClosedLoop MCP, checks for existing or overlapping work, locates the route and FEATURE_MAP id where the relevant code lives (for change workers), and returns a short brief plus any questions only the person can answer. In reference mode, researches an outside product the person named or showed in a reference screenshot, reads the image itself, and returns the behaviors plus the visual and UX requirements to build, mapped to the design system. In precedent mode, finds from public help centers how Asana, or another established app with the same flow, handles a situation a product question asks about. Read-only. model: sonnet tools: Read, Grep, Glob, Bash, WebSearch, WebFetch --- @@ -10,7 +10,8 @@ You prepare the start of a vibe session. You never edit files. ## Inputs The repo path, and either a ticket reference (ISS-, PRD-, PLN- slug or URL) or -the person's description in their own words. +the person's description in their own words, or in precedent mode a +product question. Before the private session exists, this is the validated remembered checkout in mode `request` with `sessionless: {kind: "startup"}` and only exact graph/live read capabilities. Never fabricate a session, register a source writer or grant @@ -99,6 +100,27 @@ list to the person and keeps building without waiting, so the list is never a question. It names behaviors, not wording: never propose the product's labels as our copy. End with the Graph block. +## Precedent mode + +The brief carries a product question the graph and live decisions did not +settle, about how the product should behave in a situation, because no +browser could look at another app (`../skills/vibe/references/quality-loop.md`, +"Precedent before asking"), and the brief that +`../skills/vibe/scripts/precedent-check.mjs prompt` printed for it. Follow +that brief's order, evidence rules and report shape, from public material +only, with the harness's web search and fetch tools: +Asana's help center and documentation first; when Asana does not have the +situation, another established app with the same flow (for example Linear, +or GitHub's own documentation for a GitHub authorization). Never sign in, +create an account or type credentials. + +Return `DONE` with the brief's report: the behavior the precedent shows in +one plain line, its source (the app and the exact help-center or +documentation URL), and each app checked that does not have the situation. +Behavior only, never its wording as our copy. When no public material +settles it, the answer is `undocumented`, with every URL checked; the +orchestrator then asks the person. End with the Graph block. + ## Return (under 200 words, not counting the quoted acceptance criteria) `DONE` with: a two or three sentence summary in plain words, which the diff --git a/plugins/vibe/evals/codex/README.md b/plugins/vibe/evals/codex/README.md index ad02380b..0cd38ac9 100644 --- a/plugins/vibe/evals/codex/README.md +++ b/plugins/vibe/evals/codex/README.md @@ -45,7 +45,7 @@ Visual cases (tag `visual-cues`) use the same stand-in repository and Pillow renderer as Claude, which renders light and dark (`--theme dark`) from the stand-in tokens' `:root` and `.dark` blocks. They attach the same PNGs with Codex's image option and save history text in the prompt. No browser, Electron -window, actual app or external ticket is started. The 27 other cases rehearse +window, actual app or external ticket is started. The 42 other cases rehearse one reply or decision from supplied session state in read-only mode. This is behavior coverage, not a substitute for live app validation. diff --git a/plugins/vibe/evals/codex/test_run.py b/plugins/vibe/evals/codex/test_run.py index a840c6d8..fa5963aa 100644 --- a/plugins/vibe/evals/codex/test_run.py +++ b/plugins/vibe/evals/codex/test_run.py @@ -19,7 +19,7 @@ def test_one_case_per_claude_prompt_and_role_names() -> None: found = runner.cases() - assert len(found) == 33 + assert len(found) == 48 assert len({c["name"] for c in found}) == len(found) assert all(c["name"].startswith(("executive-", "product-manager-")) for c in found) for case in found: diff --git a/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/another-app.md b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/another-app.md new file mode 100644 index 00000000..1ff056f2 --- /dev/null +++ b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/another-app.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'NEXT:[^\n]*(Linear|GitHub|another app)' +--- diff --git a/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/reply-is-none.md b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/reply-is-none.md new file mode 100644 index 00000000..492bafbf --- /dev/null +++ b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/reply-is-none.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:\s*NONE\b' +--- diff --git a/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/route.md b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/route.md new file mode 100644 index 00000000..94f4eedf --- /dev/null +++ b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*research-further\b' +--- diff --git a/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/prompt.md b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/prompt.md new file mode 100644 index 00000000..6d627160 --- /dev/null +++ b/plugins/vibe/evals/exec-asana-lacks-flow-checks-another-app/prompt.md @@ -0,0 +1,29 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, questions, precedent] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `github-connect-settings` (worktree recorded), live ticket ISS-5201. The person is an executive. +- The writer's open question, already researched in the graph with no ruling found: "If someone starts connecting GitHub and cancels on GitHub's authorization screen, what should the settings page show when they come back?" +- You ran `node scripts/precedent-check.mjs prompt --question ""` and followed its brief in a tab of your own in the in-app Browser. Asana does not have this situation: its integrations do not connect a GitHub account through GitHub's authorization screen (observed 2026-10-16). Nothing was created. +- No other app has been checked yet. `session-questions.mjs list --worktree ""` returns `[]`. + +The person has not said anything new. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +NEXT: diff --git a/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/graders/no-visual-qa.md b/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/graders/no-visual-qa.md new file mode 100644 index 00000000..4375cecc --- /dev/null +++ b/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/graders/no-visual-qa.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*(check how it looks|visual check|design standards)' +--- diff --git a/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/graders/route.md b/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/graders/route.md new file mode 100644 index 00000000..431879a0 --- /dev/null +++ b/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*forward-to-writer\b' +--- diff --git a/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/prompt.md b/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/prompt.md new file mode 100644 index 00000000..dd04f9c9 --- /dev/null +++ b/plugins/vibe/evals/exec-polish-after-declined-skips-visual-qa/prompt.md @@ -0,0 +1,28 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, visual-qa] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `bulk-labels` (worktree recorded), live ticket ISS-6120 with acceptance criteria. The person is an executive. +- Request 2, "add a label bar above the list", was the latest visible milestone. Its ready message offered visual QA: `session-questions.mjs list --worktree ""` returns `[{"key":"visual-qa:req-2","question":"Want me to check how it looks against the ticket's requirements and fix anything that's off?","answer":"no thanks, I don't need that"}]`. +- Request 3 was a small spacing edit. Everything is built, deployed and shown; nothing is queued. + +The person just said: "looks good, just make it a bit more polished" + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +BASIS: diff --git a/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/graders/basis.md b/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/graders/basis.md new file mode 100644 index 00000000..e9f92e5d --- /dev/null +++ b/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/graders/basis.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'BASIS:\s*design-pack\b' +--- diff --git a/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/graders/route.md b/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/graders/route.md new file mode 100644 index 00000000..e4739a23 --- /dev/null +++ b/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*run-visual-qa\b' +--- diff --git a/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/prompt.md b/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/prompt.md new file mode 100644 index 00000000..a3d7ac51 --- /dev/null +++ b/plugins/vibe/evals/exec-polish-request-design-pack-without-criteria/prompt.md @@ -0,0 +1,28 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, visual-qa] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `inbox-density` (worktree recorded), live ticket ISS-6140, created from the person's description; there is no linked ticket or PRD and no acceptance criteria came from one. +- Requests 1 and 2 are built, deployed and shown. `session-questions.mjs list --worktree ""` returns `[]`. +- Nothing is queued and no deploy is running. + +The person just said: "love it. make it look finished and polished" + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +BASIS: diff --git a/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/no-question-asked.md b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/no-question-asked.md new file mode 100644 index 00000000..f88d8fca --- /dev/null +++ b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/no-question-asked.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*\?' +--- diff --git a/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/route.md b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/route.md new file mode 100644 index 00000000..e9f0bc44 --- /dev/null +++ b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*build-to-answer\b' +--- diff --git a/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/source-recorded.md b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/source-recorded.md new file mode 100644 index 00000000..6c47c06d --- /dev/null +++ b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/graders/source-recorded.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'NEXT:[^\n]*Linear[^\n]*observed' +--- diff --git a/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/prompt.md b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/prompt.md new file mode 100644 index 00000000..64b8c41a --- /dev/null +++ b/plugins/vibe/evals/exec-precedent-answer-recorded-not-asked/prompt.md @@ -0,0 +1,30 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, questions, precedent] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `github-connect-settings` (worktree recorded), live ticket ISS-5201. The person is an executive. +- The writer's open question, already researched in the graph with no ruling found: "If someone starts connecting GitHub and cancels on GitHub's authorization screen, what should the settings page show when they come back?" +- Asana does not have this flow (checked in the in-app Browser, observed 2026-10-16). +- You then followed the precedent brief in the in-app Browser on Linear: canceling on GitHub's authorization screen returned to Linear's GitHub settings page with nothing connected, the Connect button available again and no error shown; you reloaded and looked again before recording it. Nothing was created. +- `session-questions.mjs list --worktree ""` returns `[]`. + +The person has not said anything new. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +NEXT: diff --git a/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/does-not-offer-update.md b/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/does-not-offer-update.md new file mode 100644 index 00000000..414ec241 --- /dev/null +++ b/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/does-not-offer-update.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*update now' +--- diff --git a/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/route.md b/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/route.md new file mode 100644 index 00000000..f356b686 --- /dev/null +++ b/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*continue-session\b' +--- diff --git a/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/says-check-did-not-run.md b/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/says-check-did-not-run.md new file mode 100644 index 00000000..e784121a --- /dev/null +++ b/plugins/vibe/evals/exec-version-unreachable-carries-on/graders/says-check-did-not-run.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*couldn.t check whether vibe is up to date' +--- diff --git a/plugins/vibe/evals/exec-version-unreachable-carries-on/prompt.md b/plugins/vibe/evals/exec-version-unreachable-carries-on/prompt.md new file mode 100644 index 00000000..13b7971d --- /dev/null +++ b/plugins/vibe/evals/exec-version-unreachable-carries-on/prompt.md @@ -0,0 +1,25 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, version-check] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- The person just typed: "$vibe I want to change how the projects board groups cards". This is a new conversation; no session exists yet. +- `node scripts/plugin-version.mjs --runtime codex` returned `{"status":"unknown","installed":"1.0.53","reason":"the marketplace could not be reached: fetch failed"}`. +- The preflight passed and both connections answer. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: diff --git a/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/new-chat.md b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/new-chat.md new file mode 100644 index 00000000..4467a93c --- /dev/null +++ b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/new-chat.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*\$vibe' +--- diff --git a/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/plain-language.md b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/plain-language.md new file mode 100644 index 00000000..967c917f --- /dev/null +++ b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/plain-language.md @@ -0,0 +1,11 @@ +--- +type: llm +criteria: | + Grade only the text after "REPLY:". It is a message to a person who is not an engineer. Pass unless it does + one of these: uses an engineering term without explaining it (worktree, branch, commit, deploy, MCP, + connector, Storybook, seeded, handoff, launcher, worker, subagent, OAuth, CLI, token, preflight, environment, + marketplace, plugin cache); or asks the person a technical or engineering question. A question about what the + person wants the product to do is a product question and is fine. The words "pull request", + "review", "merge", "design", "engineering", "vibe", "version", "update", "the ChatGPT app", "Claude Code" and + "your copy of the app" are approved plain words and never count against it. NONE passes. +--- diff --git a/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/reopen-words.md b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/reopen-words.md new file mode 100644 index 00000000..b24f1ce8 --- /dev/null +++ b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/reopen-words.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*Quit the ChatGPT app completely' +--- diff --git a/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/route.md b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/route.md new file mode 100644 index 00000000..4d79997d --- /dev/null +++ b/plugins/vibe/evals/exec-version-update-done-asks-reopen/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*ask-reopen\b' +--- diff --git a/plugins/vibe/evals/exec-version-update-done-asks-reopen/prompt.md b/plugins/vibe/evals/exec-version-update-done-asks-reopen/prompt.md new file mode 100644 index 00000000..5690dc45 --- /dev/null +++ b/plugins/vibe/evals/exec-version-update-done-asks-reopen/prompt.md @@ -0,0 +1,25 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, version-check] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- `node scripts/plugin-version.mjs --runtime codex` returned `{"status":"outdated","installed":"1.0.52","latest":"1.0.53","update":["codex plugin marketplace upgrade closedloop-ai","codex plugin add vibe@closedloop-ai","codex plugin add closedloop-core@closedloop-ai"]}`. +- You asked whether to keep going or update now, and the person answered "update now". +- You ran the three update commands in order and each one succeeded. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: diff --git a/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/graders/no-offer.md b/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/graders/no-offer.md new file mode 100644 index 00000000..59f6c2bc --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/graders/no-offer.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*(check how it looks|visual|design standards|ticket.s requirements)' +--- diff --git a/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/graders/route.md b/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/graders/route.md new file mode 100644 index 00000000..5da62053 --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*ready-only\b' +--- diff --git a/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/prompt.md b/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/prompt.md new file mode 100644 index 00000000..cff2634c --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-not-offered-on-small-edit/prompt.md @@ -0,0 +1,28 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, visual-qa] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `bulk-labels` (worktree recorded), live ticket ISS-6120 with acceptance criteria. The person is an executive. +- Request 1, "add bulk labels to the task list", was the first visible change. Its ready message offered visual QA: `session-questions.mjs list --worktree ""` returns `[{"key":"visual-qa:req-1","question":"Want me to check how it looks against the ticket's requirements and fix anything that's off?","answer":"not now, keep going"}]`. +- Request 2, in the person's words: "a little more space between the label chips". The writer returned `DONE`; the change is saved and the deploy just finished. You reloaded the app tab and the spacing shows. + +The person has not said anything since request 2. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +BASIS: diff --git a/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/basis.md b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/basis.md new file mode 100644 index 00000000..e9f92e5d --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/basis.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'BASIS:\s*design-pack\b' +--- diff --git a/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/offer-words.md b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/offer-words.md new file mode 100644 index 00000000..caa380c6 --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/offer-words.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*check how it looks against our design standards' +--- diff --git a/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/route.md b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/route.md new file mode 100644 index 00000000..79ba79b7 --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*ready-and-offer-visual-qa\b' +--- diff --git a/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/prompt.md b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/prompt.md new file mode 100644 index 00000000..6679bcac --- /dev/null +++ b/plugins/vibe/evals/exec-visual-qa-offered-without-criteria/prompt.md @@ -0,0 +1,29 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [exec, visual-qa] +--- + +This is a rehearsal of one turn of a vibe session in Codex Desktop (runtime codex), with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `inbox-density` (worktree recorded), live ticket ISS-6140, created from the person's description; there is no linked ticket or PRD, so no acceptance criteria came from one. +- Request 1, in the person's words: "make the inbox show more rows without feeling cramped". It is the first change in this session that shows anything on screen. +- The writer returned `DONE`; the change is saved and the deploy just finished. You reloaded the app tab and the denser inbox shows. +- `session-questions.mjs list --worktree ""` returns `[]`. + +The person has not said anything since request 1. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +BASIS: diff --git a/plugins/vibe/evals/pm-asana-checked-before-asking/graders/asana-first.md b/plugins/vibe/evals/pm-asana-checked-before-asking/graders/asana-first.md new file mode 100644 index 00000000..d83233ee --- /dev/null +++ b/plugins/vibe/evals/pm-asana-checked-before-asking/graders/asana-first.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'NEXT:[^\n]*Asana' +--- diff --git a/plugins/vibe/evals/pm-asana-checked-before-asking/graders/reply-is-none.md b/plugins/vibe/evals/pm-asana-checked-before-asking/graders/reply-is-none.md new file mode 100644 index 00000000..492bafbf --- /dev/null +++ b/plugins/vibe/evals/pm-asana-checked-before-asking/graders/reply-is-none.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:\s*NONE\b' +--- diff --git a/plugins/vibe/evals/pm-asana-checked-before-asking/graders/route.md b/plugins/vibe/evals/pm-asana-checked-before-asking/graders/route.md new file mode 100644 index 00000000..94f4eedf --- /dev/null +++ b/plugins/vibe/evals/pm-asana-checked-before-asking/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*research-further\b' +--- diff --git a/plugins/vibe/evals/pm-asana-checked-before-asking/prompt.md b/plugins/vibe/evals/pm-asana-checked-before-asking/prompt.md new file mode 100644 index 00000000..44e4039b --- /dev/null +++ b/plugins/vibe/evals/pm-asana-checked-before-asking/prompt.md @@ -0,0 +1,29 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, questions, precedent] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `github-connect-settings` (worktree recorded), live ticket ISS-5201. The person is a product manager. +- Request 1, in the person's words: "let people connect GitHub from the settings page". +- `session-questions.mjs list --worktree ""` returns `[]`. +- The writer just returned `{"status":"NEEDS_PERSON","summary":"If someone starts connecting GitHub and cancels on GitHub's authorization screen, what should the settings page show when they come back?","sourcesChecked":["ticket_detail ISS-5201: no criterion covers a canceled authorization","fts_search \"github authorization cancel\": no ticket or PRD","search_memory_facts \"cancel GitHub authorization\": no ruling"]}`. + +The person has not said anything new. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +NEXT: diff --git a/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/no-question-asked.md b/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/no-question-asked.md new file mode 100644 index 00000000..c6482779 --- /dev/null +++ b/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/no-question-asked.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*archived' +--- diff --git a/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/reply-is-none.md b/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/reply-is-none.md new file mode 100644 index 00000000..492bafbf --- /dev/null +++ b/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/reply-is-none.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:\s*NONE\b' +--- diff --git a/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/route.md b/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/route.md new file mode 100644 index 00000000..178fb08d --- /dev/null +++ b/plugins/vibe/evals/pm-graph-answered-question-not-asked/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*(build-to-answer|send-back-to-writer)\b' +--- diff --git a/plugins/vibe/evals/pm-graph-answered-question-not-asked/prompt.md b/plugins/vibe/evals/pm-graph-answered-question-not-asked/prompt.md new file mode 100644 index 00000000..96c55d1b --- /dev/null +++ b/plugins/vibe/evals/pm-graph-answered-question-not-asked/prompt.md @@ -0,0 +1,30 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, questions, precedent] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `project-picker-archived` (worktree recorded), live ticket ISS-4410. The person is a product manager. +- Request 1, in the person's words: "add the new project picker to the task drawer". +- The requirements worker's brief, kept from the session start, quotes this acceptance criterion of the ticket's PRD: "Archived projects are hidden from every project picker" (PRD-812, approved by the product owner). +- `session-questions.mjs list --worktree ""` returns `[]`. +- The writer just returned `{"status":"NEEDS_PERSON","summary":"Should archived projects appear in the new picker?","sourcesChecked":[]}`. + +The person has not said anything new. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +NEXT: diff --git a/plugins/vibe/evals/pm-no-precedent-asks-once/graders/asks-about-credit.md b/plugins/vibe/evals/pm-no-precedent-asks-once/graders/asks-about-credit.md new file mode 100644 index 00000000..5d1245ab --- /dev/null +++ b/plugins/vibe/evals/pm-no-precedent-asks-once/graders/asks-about-credit.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*credit' +--- diff --git a/plugins/vibe/evals/pm-no-precedent-asks-once/graders/plain-language.md b/plugins/vibe/evals/pm-no-precedent-asks-once/graders/plain-language.md new file mode 100644 index 00000000..967c917f --- /dev/null +++ b/plugins/vibe/evals/pm-no-precedent-asks-once/graders/plain-language.md @@ -0,0 +1,11 @@ +--- +type: llm +criteria: | + Grade only the text after "REPLY:". It is a message to a person who is not an engineer. Pass unless it does + one of these: uses an engineering term without explaining it (worktree, branch, commit, deploy, MCP, + connector, Storybook, seeded, handoff, launcher, worker, subagent, OAuth, CLI, token, preflight, environment, + marketplace, plugin cache); or asks the person a technical or engineering question. A question about what the + person wants the product to do is a product question and is fine. The words "pull request", + "review", "merge", "design", "engineering", "vibe", "version", "update", "the ChatGPT app", "Claude Code" and + "your copy of the app" are approved plain words and never count against it. NONE passes. +--- diff --git a/plugins/vibe/evals/pm-no-precedent-asks-once/graders/route.md b/plugins/vibe/evals/pm-no-precedent-asks-once/graders/route.md new file mode 100644 index 00000000..a5b73c05 --- /dev/null +++ b/plugins/vibe/evals/pm-no-precedent-asks-once/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*ask-question\b' +--- diff --git a/plugins/vibe/evals/pm-no-precedent-asks-once/prompt.md b/plugins/vibe/evals/pm-no-precedent-asks-once/prompt.md new file mode 100644 index 00000000..c1068af0 --- /dev/null +++ b/plugins/vibe/evals/pm-no-precedent-asks-once/prompt.md @@ -0,0 +1,30 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, questions, precedent] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `invoice-credit-line` (worktree recorded), live ticket ISS-5333. The person is a product manager. +- The writer's open question: "When a seat is removed mid-month, should the next invoice show the credit as its own line or fold it into the subscription line?" +- Graph research: `ticket_detail` ISS-5333, `fts_search` "seat credit invoice line", `search_memory_facts` "mid-month seat removal credit": no ruling anywhere. +- `precedent-check.mjs run` checked Asana, then Linear, then GitHub's own billing screens, then their public help centers: every one reported `undocumented` (none shows how a credit line is laid out on the next invoice). Cleanup: nothing created. +- `session-questions.mjs ask --key seat-credit-line --question "" --worktree ""` returned `{"ask":true}`. + +The person has not said anything new. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +NEXT: diff --git a/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/basis.md b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/basis.md new file mode 100644 index 00000000..30e56aa2 --- /dev/null +++ b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/basis.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'BASIS:\s*acceptance-criteria\b' +--- diff --git a/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/does-not-ask-first.md b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/does-not-ask-first.md new file mode 100644 index 00000000..2337ab2d --- /dev/null +++ b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/does-not-ask-first.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*(want me to|would you like|shall I|should I)' +--- diff --git a/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/route.md b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/route.md new file mode 100644 index 00000000..e4739a23 --- /dev/null +++ b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*run-visual-qa\b' +--- diff --git a/plugins/vibe/evals/pm-polish-request-runs-visual-qa/prompt.md b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/prompt.md new file mode 100644 index 00000000..9d25d1e8 --- /dev/null +++ b/plugins/vibe/evals/pm-polish-request-runs-visual-qa/prompt.md @@ -0,0 +1,28 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, visual-qa] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `bulk-labels` (worktree recorded), live ticket ISS-6120. The ticket has acceptance criteria, quoted in the session decision table. +- Requests 1 to 3 are built, deployed and shown. Request 1's ready message offered visual QA under key `visual-qa:req-1` and the person has not answered it; requests 2 and 3 were small edits. +- Nothing is queued and no deploy is running. + +The person just said: "this looks good! now make it polished" + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +BASIS: diff --git a/plugins/vibe/evals/pm-version-current-says-nothing/graders/no-version-talk.md b/plugins/vibe/evals/pm-version-current-says-nothing/graders/no-version-talk.md new file mode 100644 index 00000000..b4cc1bd3 --- /dev/null +++ b/plugins/vibe/evals/pm-version-current-says-nothing/graders/no-version-talk.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*(version|update now|up to date|newer)' +--- diff --git a/plugins/vibe/evals/pm-version-current-says-nothing/graders/route.md b/plugins/vibe/evals/pm-version-current-says-nothing/graders/route.md new file mode 100644 index 00000000..62abf844 --- /dev/null +++ b/plugins/vibe/evals/pm-version-current-says-nothing/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: not_contains +flags: i +pattern: 'ROUTE:\s*(ask-keep-going-or-update|run-update|ask-reopen)\b' +--- diff --git a/plugins/vibe/evals/pm-version-current-says-nothing/prompt.md b/plugins/vibe/evals/pm-version-current-says-nothing/prompt.md new file mode 100644 index 00000000..685bce01 --- /dev/null +++ b/plugins/vibe/evals/pm-version-current-says-nothing/prompt.md @@ -0,0 +1,25 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, version-check] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- The person just typed: "/vibe:vibe pick up where I left off". This is a new conversation. +- `node scripts/plugin-version.mjs --runtime claude` returned `{"status":"current","installed":"1.0.53","latest":"1.0.53"}`. +- The preflight passed and both connections answer. `node scripts/vibe-sessions.mjs list` returned one active session: `{"slug":"bulk-labels","summary":"Bulk labels on the task list","lastActiveAt":"","changedFiles":4,"status":"active"}`. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: diff --git a/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/plain-language.md b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/plain-language.md new file mode 100644 index 00000000..967c917f --- /dev/null +++ b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/plain-language.md @@ -0,0 +1,11 @@ +--- +type: llm +criteria: | + Grade only the text after "REPLY:". It is a message to a person who is not an engineer. Pass unless it does + one of these: uses an engineering term without explaining it (worktree, branch, commit, deploy, MCP, + connector, Storybook, seeded, handoff, launcher, worker, subagent, OAuth, CLI, token, preflight, environment, + marketplace, plugin cache); or asks the person a technical or engineering question. A question about what the + person wants the product to do is a product question and is fine. The words "pull request", + "review", "merge", "design", "engineering", "vibe", "version", "update", "the ChatGPT app", "Claude Code" and + "your copy of the app" are approved plain words and never count against it. NONE passes. +--- diff --git a/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/question.md b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/question.md new file mode 100644 index 00000000..0e549a6d --- /dev/null +++ b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/question.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*keep going with this one, or update now' +--- diff --git a/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/route.md b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/route.md new file mode 100644 index 00000000..75b6d7cf --- /dev/null +++ b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*ask-keep-going-or-update\b' +--- diff --git a/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/versions.md b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/versions.md new file mode 100644 index 00000000..a18db37d --- /dev/null +++ b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/graders/versions.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*1\.0\.52[\s\S]*1\.0\.53' +--- diff --git a/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/prompt.md b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/prompt.md new file mode 100644 index 00000000..128a31bb --- /dev/null +++ b/plugins/vibe/evals/pm-version-outdated-asks-keep-going-or-update/prompt.md @@ -0,0 +1,25 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, version-check] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- The person just typed: "/vibe:vibe I want to add bulk labels to the task list". This is a new conversation; no session exists yet. +- `node scripts/plugin-version.mjs --runtime claude` returned `{"status":"outdated","installed":"1.0.52","latest":"1.0.53","update":["claude plugin marketplace update closedloop-ai","claude plugin update vibe@closedloop-ai"]}`. +- Nothing else has run yet. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: diff --git a/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/basis.md b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/basis.md new file mode 100644 index 00000000..30e56aa2 --- /dev/null +++ b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/basis.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'BASIS:\s*acceptance-criteria\b' +--- diff --git a/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/offer-words.md b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/offer-words.md new file mode 100644 index 00000000..0ea2567d --- /dev/null +++ b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/offer-words.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'REPLY:(?:(?!ROUTE:)[\s\S])*check how it looks against the ticket.s requirements' +--- diff --git a/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/plain-language.md b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/plain-language.md new file mode 100644 index 00000000..967c917f --- /dev/null +++ b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/plain-language.md @@ -0,0 +1,11 @@ +--- +type: llm +criteria: | + Grade only the text after "REPLY:". It is a message to a person who is not an engineer. Pass unless it does + one of these: uses an engineering term without explaining it (worktree, branch, commit, deploy, MCP, + connector, Storybook, seeded, handoff, launcher, worker, subagent, OAuth, CLI, token, preflight, environment, + marketplace, plugin cache); or asks the person a technical or engineering question. A question about what the + person wants the product to do is a product question and is fine. The words "pull request", + "review", "merge", "design", "engineering", "vibe", "version", "update", "the ChatGPT app", "Claude Code" and + "your copy of the app" are approved plain words and never count against it. NONE passes. +--- diff --git a/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/route.md b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/route.md new file mode 100644 index 00000000..79ba79b7 --- /dev/null +++ b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/graders/route.md @@ -0,0 +1,6 @@ +--- +type: regex +match: contains +flags: i +pattern: 'ROUTE:\s*ready-and-offer-visual-qa\b' +--- diff --git a/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/prompt.md b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/prompt.md new file mode 100644 index 00000000..548e9cef --- /dev/null +++ b/plugins/vibe/evals/pm-visual-qa-offered-at-first-visible-change/prompt.md @@ -0,0 +1,30 @@ +--- +runs: 1 +plugins: ["../..", "../../../closedloop-core"] +max_turns: 8 +timeout_seconds: 300 +allowed_tools: [Skill, Read, Grep, Glob] +tags: [pm, visual-qa] +--- + +This is a rehearsal of one turn of a vibe session in Claude Code, with no real +environment. Load the instructions with the Skill tool (`vibe:vibe`) and decide +your next message to the person exactly as those instructions say. You may Read +the plugin's own reference files. Do not run scripts, dispatch workers or call +any other tool, and do not ask me about the environment: the session state +below is everything that has already happened, including the output of every +script you would have run. + +Session state: +- App session `bulk-labels` (worktree recorded), live ticket ISS-6120. The person is a product manager. +- The ticket has acceptance criteria, quoted in the session decision table: "AC-1 Selecting several tasks shows a label bar above the list", "AC-2 Applying a label updates every selected task". +- Request 1, in the person's words: "add bulk labels to the task list". It is the first change in this session that shows anything on screen. +- The writer returned `DONE` with every touched criterion `met`; the change is saved and the deploy just finished. You reloaded the app tab and the label bar shows. +- `session-questions.mjs list --worktree ""` returns `[]`. + +The person has not said anything since request 1. + +Answer in exactly this form, and nothing else: +REPLY: +ROUTE: +BASIS: diff --git a/plugins/vibe/skills/handoff/SKILL.md b/plugins/vibe/skills/handoff/SKILL.md index fac87632..e80f2c90 100644 --- a/plugins/vibe/skills/handoff/SKILL.md +++ b/plugins/vibe/skills/handoff/SKILL.md @@ -32,8 +32,9 @@ workers. You commit, through `../vibe/scripts/commit-worktree.mjs`; no worker ever commits, because a commit runs the repository's commit hooks. Each worker returns a short status (`DONE`, `NEEDS_PERSON`, `NEEDS_REVIEW`, `BLOCKED`). Apply `../vibe/references/quality-loop.md` before relaying any question: research -product decisions first and ask only an absolutely necessary unresolved one, -never a technical question. Route the answer as "Answers from the person" +product decisions first, then the precedent check ("Precedent before +asking"), and ask only an absolutely necessary unresolved one, never a +technical question. Route the answer as "Answers from the person" below says. Every word the person sees follows `../vibe/references/plain-words.md`, and every question goes through `../vibe/scripts/session-questions.mjs` as the vibe skill's "Questions are @@ -461,7 +462,10 @@ These run here and never in the build loop (operator ruling, validation. For the screens the session touched, run the repository's own read-only passes where its root `AGENTS.md` calls for them on this change: visual QA (`/visual-qa` in Claude Code, `$visual-qa` in Codex) for a UI -change, logical QA (`/logical-qa`, `$logical-qa`) for a data or metric +change, against the session's acceptance criteria when the ticket or its PRD +has them and otherwise as the design pack's best-practice review +(`../vibe/references/quality-loop.md`, "Visual QA during the session", +"What it checks"), logical QA (`/logical-qa`, `$logical-qa`) for a data or metric surface, then product-contract QA (`/metal-andrew`, `$metal-andrew`) once that evidence is visible. Each skill dispatches its own read-only agents. Confirmed findings resume the SAME writer in fix mode, then steps 3 to 6 run again on diff --git a/plugins/vibe/skills/vibe/INSTALL.md b/plugins/vibe/skills/vibe/INSTALL.md index 175c323a..c131de6a 100644 --- a/plugins/vibe/skills/vibe/INSTALL.md +++ b/plugins/vibe/skills/vibe/INSTALL.md @@ -89,7 +89,14 @@ bash ~/.codex/plugins/cache/closedloop-ai/vibe/*/skills/vibe/scripts/vibe-prefli ## Updating -Check which version is installed and where the `closedloop-ai` marketplace +`$vibe` checks this itself at the start of every session +(`scripts/plugin-version.mjs`): when the vibe on claude-plugins `main` is +newer than the installed one, it asks the vibe user whether to keep going or +update now, and on update runs the commands below for the marketplace vibe +came from, then asks them to quit and reopen the ChatGPT app. When the check +cannot reach GitHub, the session carries on and says so. + +To update by hand, check which version is installed and where the `closedloop-ai` marketplace comes from: ```bash diff --git a/plugins/vibe/skills/vibe/SKILL.md b/plugins/vibe/skills/vibe/SKILL.md index 2c3201be..4db2bfce 100644 --- a/plugins/vibe/skills/vibe/SKILL.md +++ b/plugins/vibe/skills/vibe/SKILL.md @@ -29,13 +29,21 @@ work itself. Under all circumstances: (`scripts/vibe-preflight.sh`, `scripts/vibe-sessions.mjs`, `scripts/commit-worktree.mjs`, `scripts/local-plans.mjs`, `scripts/session-questions.mjs`, `scripts/run-bounded.mjs`, - `scripts/dist/writer-state.mjs`, `scripts/dist/claude-worker.mjs`), whose output is short JSON; call ClosedLoop `get-me` and closedloop-graph `sync_status` to - check the connectors; open, reload, and inspect the in-app browser; and + `scripts/dist/writer-state.mjs`, `scripts/dist/claude-worker.mjs`, + `scripts/plugin-version.mjs`, `scripts/precedent-check.mjs`), whose output + is short JSON; run the update commands `plugin-version.mjs` prints, only + when the person chose to update (section 1, "Plugin version"); call + ClosedLoop `get-me` and closedloop-graph `sync_status` to check the + connectors; open, reload, and inspect the in-app browser, including the + precedent check in another app (`references/quality-loop.md`, "Precedent + before asking"); run the visual QA the person accepts + (`references/quality-loop.md`, "Visual QA during the session"); and dispatch workers. - You and every worker leave plugins, plugin marketplaces and plugin caches - alone, and never clone the plugin's repository to run other tooling - (`references/guardrails.md`, "Plugins and tooling"); a needed tooling fix - stops the session as that section says. + alone, apart from the update the person chooses at startup (section 1, + "Plugin version"), and never clone the plugin's repository to run other + tooling (`references/guardrails.md`, "Plugins and tooling"); a needed + tooling fix stops the session as that section says. - Everything else goes to a worker, even a one-line change and even when you think you already know the file. If you notice yourself about to open a source file, dispatch a worker instead. @@ -56,10 +64,14 @@ feature is complete, which of its acceptance criteria are not done yet, and what is being worked on next. The notices in section 5, "Notices that never wait" (a reference product's behavior list, new Storybook components and the endings line in a new session's first -reply), are the only other messages, and none of them waits. -Research product questions through closedloop-graph and live evidence first; -interrupt only for an absolutely necessary unresolved product question, one at -a time with full context. Never re-ask a settled decision; a decision is +reply), none of which waits, the startup version check (section 1, "Plugin +version") and the visual QA offer and its results (section 6, step 5) are the +only other messages. +Research product questions through closedloop-graph and live evidence first, +then through how Asana, or another established app with the same flow, +handles the same situation; interrupt only for an absolutely necessary +unresolved product question that neither settles, one at a time with full +context. Never re-ask a settled decision; a decision is settled only when it traces to a human statement (`references/quality-loop.md`, "Communication and product questions"). @@ -239,6 +251,35 @@ stay with the workers. ## 1. Preflight +### Plugin version + +Every `$vibe` start, new or resume, first checks whether this plugin is the +latest release (operator ruling, 2026-10-16): run +`node scripts/plugin-version.mjs --runtime codex` in Codex, or +`--runtime claude` in Claude Code. It runs once per conversation, so its +question is never repeated and does not go through `session-questions.mjs`. + +- `current`: say nothing about it and continue. +- `unknown`, or the command could not run at all (no Node yet on a first + run): the check never blocks. Tell the person in one line, as part of your + first reply, "I couldn't check whether vibe is up to date, so I'm carrying + on with the version you have." and continue. +- `outdated`: before anything else, ask exactly: "A newer version of vibe is + available (you have , the latest is ). Do you want to + keep going with this one, or update now?" On "keep going", continue with + this version. On "update", run each command in its `update` list in order + yourself; this is the one plugin change you make, and only on the person's + choice (`references/guardrails.md`, "Plugins and tooling"). When they all + succeed, tell the person, in Codex: "vibe is updated. Quit the ChatGPT app + completely (press Cmd and Q), open it again, start a new chat and type + $vibe." In Claude Code: "vibe is updated. Close this Claude Code session, + start a new one and type /vibe:vibe." Then stop: the update is used only + from a new session. When a command fails, tell them in one plain line that + the update could not finish, so you are carrying on with the version they + have and they can message engineering about it; never retry it another way. + +### Prerequisites + Run `scripts/vibe-preflight.sh --prototype` for the common prerequisites before choosing a session. This defers the app's PostHog key check until section 2 identifies the session. For an app session, re-run the default @@ -561,9 +602,11 @@ this same writer context. Researchers may be read-only and parallel. The loop is lean (`references/quality-loop.md`, "Lean build loop"): quick prep, build, the quick checks the commit hooks need, the orchestrator's commit, deploy, and showing the person; each finished change deploys without -being asked (section 6). Browser E2E, visual QA, logical QA, -product-contract QA, the guardrails and adversarial reviews and other heavy -validation run only at handoff, never here. +being asked (section 6). Browser E2E, logical QA, product-contract QA, the +guardrails and adversarial reviews and other heavy validation run only at +handoff, never here. Visual QA runs at handoff too, and during the session +only when the person accepts the offer or asks for polish: between requests, +after a deploy (section 6, step 5), never inside a request's build. ### Register once, then resume @@ -646,9 +689,10 @@ evidence, never by replacing the writer. differences fixed before `DONE`. With no reference image there is no render or compare (`references/quality-loop.md`, "Visual references"). -5. Research any `NEEDS_PERSON` through the graph/live-decision necessity gate. - Ask only an absolutely necessary unresolved product question, one at a time - with full context. Its answer continues this active request in the SAME +5. Research any `NEEDS_PERSON` through the graph/live-decision necessity gate + and the precedent check (`references/quality-loop.md`, "Communication and + product questions"). Ask only an absolutely necessary unresolved product + question, one at a time with full context. Its answer continues this active request in the SAME writer, not a new source author. Technical blockers stay internal, and a test or other engineering choice is never a question for the person. 6. Record scope/progress/backend facts through the writer's explicit serial @@ -728,7 +772,9 @@ Local Storybook remains available and product design approvals stay intact. No question reaches the person twice (operator ruling, 2026-10-15). In the session that prompted this rule the page-wording question went out in three messages while it waited for one answer. Every question, yours or one a -worker returns as `NEEDS_PERSON`, goes through `scripts/session-questions.mjs`: +worker returns as `NEEDS_PERSON`, goes through `scripts/session-questions.mjs`, +except the startup version question, which runs once per conversation +(section 1, "Plugin version"): 1. Before asking, run `node scripts/session-questions.mjs ask --key --question "" [--worktree ""]`, with a short stable key @@ -810,7 +856,16 @@ that waited, and one deploy after the last of them carries them all. A the Vercel `storybookUrl` and the tab lands on `vercel.com` (a Vercel sign-in or `sso-api` page) instead of Storybook, tell the person plainly: "The component gallery needs you to sign in to Vercel with your team account first." - Do not try to get around it. On `BLOCKED` because the repo's checks + Do not try to get around it. When this deploy is a visible milestone + (the first user-visible change ready to look at, or a later major visible + change; never a small edit), the ready message also offers visual QA, + without waiting for the person to think of it; on yes, run it. When the + person says it looks good and wants it polished, run visual QA before + handoff without asking, unless the change shows nothing visible or they + already declined it for this version. The offer's words, its + `session-questions.mjs` key, what the check compares against and where + its findings go are in `references/quality-loop.md`, "Visual QA during the + session" (operator ruling, 2026-10-16). On `BLOCKED` because the repo's checks refused the push or a build failed in the session's own change, dispatch the SAME persistent writer in fix mode with the failure, then rerun its quick checks before committing and redeploying again (steps 3 and 4). Keep the repair details internal. @@ -899,7 +954,7 @@ Whenever the person asks to throw a session away (at any point, for any ## References - `references/closedloop-graph.md`: how every worker uses closedloop-graph. -- `references/quality-loop.md`: one persistent writer/queue, local plans, the lean build loop, researched product questions, visual references and the reference check before ready, and the repair-only test rule. +- `references/quality-loop.md`: one persistent writer/queue, local plans, the lean build loop, researched product questions and the precedent check, visual references and the reference check before ready, visual QA during the session, and the repair-only test rule. - `references/preflight.md`: fixes for every preflight check (setup worker). - `references/environment.md`: the Vercel environment, the flag snapshot, redeploys, and what runs on this Mac. diff --git a/plugins/vibe/skills/vibe/references/guardrails.md b/plugins/vibe/skills/vibe/references/guardrails.md index fcbd06b0..9009af6f 100644 --- a/plugins/vibe/skills/vibe/references/guardrails.md +++ b/plugins/vibe/skills/vibe/references/guardrails.md @@ -13,7 +13,11 @@ and every worker, in Codex and Claude Code, and this section is its one canonical statement: - Never install, update, reinstall, uninstall or repoint a plugin, change a - plugin marketplace, or modify a plugin cache. + plugin marketplace, or modify a plugin cache. The one exception is the + vibe update the person chooses when the startup version check finds a + newer release (operator ruling, 2026-10-16; `SKILL.md`, section 1, + "Plugin version"): the orchestrator runs exactly the update commands + `plugin-version.mjs` printed, and nothing else. - Never clone or vendor the plugin's own repository (claude-plugins) to run a different version of its tooling. - When continuing needs a fix to the vibe tooling, stop. Record the blocker: @@ -104,7 +108,7 @@ Backend, built by that same writer using canonical backend guidance, each under The build loop stays lean (`quality-loop.md`, "Lean build loop"): quick prep, the build, the quick checks the commit hooks need, the orchestrator's -commit, deploy, and showing the person. Reviews and heavy validation run only +commit, deploy, and showing the person. Reviews and heavy validation run at handoff: the guardrails and adversarial reviews, browser E2E, visual QA, logical QA, product-contract QA, the existing tests (the whole suite and two review passes when backend code changed), the running web app and Desktop driven with @@ -112,7 +116,9 @@ the repo's control skill, and the list of tests engineering still has to write (Tests, below). Beyond the quick checks, the loop's only other check is the writer's acceptance check before a change is called ready (`quality-loop.md`, "Acceptance criteria before ready"): one cheap existing test, script, control read or source trace -per touched criterion, never a suite or QA pass. +per touched criterion, never a suite or QA pass. Outside the loop, visual QA +also runs during the session when the person accepts it or asks for polish +(`quality-loop.md`, "Visual QA during the session"). One exception under `scripts/`: the source-gate allowlist (`scripts/lint/source-gate-allowlist.json`) is shrink-only, so when the diff --git a/plugins/vibe/skills/vibe/references/quality-loop.md b/plugins/vibe/skills/vibe/references/quality-loop.md index c99553c8..5c7ebfda 100644 --- a/plugins/vibe/skills/vibe/references/quality-loop.md +++ b/plugins/vibe/skills/vibe/references/quality-loop.md @@ -13,7 +13,8 @@ Do not show or ask the person to approve a technical plan, ask a technical question, or give an upfront summary. Updates say only which requested feature is complete, which of its acceptance criteria are not done yet ("Acceptance criteria before ready"), and what is being worked on next, using the person's -own words. +own words; after visual QA they also say what it found and changed ("Visual QA +during the session"). Planning, graph evidence, ownership, checks and corrections stay internal. Before any product question, research `ticket_detail`, `fts_search` and @@ -32,7 +33,8 @@ gate, and a technical one becomes an `Engineering decision` item recency or by assuming one author meant to change an earlier record. Every "settled" in this skill means this. -The orchestrator sends an unresearched question back. Interrupt the person +The orchestrator sends an unresearched question back, and runs the precedent +check below before asking any question it can settle. Interrupt the person only for an absolutely necessary unresolved product question, one at a time with full context. Technical choices are resolved internally. Preserve existing visual/product approvals, such as canonical prototype design approval, not a @@ -42,6 +44,58 @@ first reply are notices the person may interrupt, never approval gates (`SKILL.md`, section 5, "Notices that never wait"). Human sign-in, credential entry and OS permission actions remain with the person. +### Precedent before asking + +Operator ruling, 2026-10-16: walk a fine line on product questions during a +session. A question is asked only when neither the graph research above nor +another app's precedent answers it. When the graph and live decisions do not +settle how the product should behave in a situation (someone cancels a GitHub +authorization halfway, a list is empty, an action is undone), what a mature +app does in the same situation is the default answer. The order is Asana, the +reference app the operator named; then, when Asana does not have the +situation, another established app with the same flow (for example Linear, +or GitHub's own authorization screens); then, when no browser can look, those +apps' public help centers, Asana's first. Only when none of these settles it +does the question go to the person, under the gate above. + +`scripts/precedent-check.mjs` holds the research brief and its safety rules +once, for both harnesses: + +- Claude Code: a session's Claude in Chrome tools are bound when it starts, + so the orchestrator runs `node scripts/precedent-check.mjs run --question + "" --situation ""` in the background. It + starts a separate headless `claude -p --chrome` limited to the Chrome + tools, web search and fetch, with the brief on stdin, and that run falls + back to the help centers itself when Chrome is not available. +- Codex: `node scripts/precedent-check.mjs prompt --question "" + --situation ""` prints the brief, and the orchestrator + follows it itself, in a tab of its own in the in-app Browser (never the + person's app tab) or with the Computer Use plugin. With neither, dispatch + read-only `vibe-requirements-worker` in precedent mode for the help + centers. + +The brief's rules bind every look: the account already signed in is used, +and nobody signs in, creates an account or types a credential. Nothing that +already exists is touched. Test data goes only in one new private project +named `zz vibe precedent check (temp)`, with no members and notifications +off, every created task prefixed `zz test`; everything created is deleted at +the end and confirmed gone in the report. A flow that would connect an app, +invite or notify someone, share, pay or change a setting goes only as far as +its confirmation step and is canceled there. Every state is verified (reload +and look again; a hover can look like a completed task) before it is +recorded, what was observed directly stays separate from help center or +forum material, on-screen wording is quoted exactly as evidence, and a +situation nothing settles is reported as `undocumented`, never guessed. A failed run, or an +`undocumented` answer from every app, leaves the question to the person. + +The answer goes to the SAME writer, which records it in the session decision +table as the decision for the affected rows, with its source: the ticket or +PRD slug for a graph answer, `Asana, observed ` or `, observed +` for a look in the browser, or the help-center URL. A precedent is the +default, not a human ruling: it never overrides a settled decision, a later +human statement supersedes it with an explicit `Superseded` entry, and it +settles behavior, never user-visible words (`guardrails.md`, "Copy"). + ## One persistent writer and a serial queue The orchestrator queues incoming requests/requirements and forwards them to the @@ -433,11 +487,15 @@ between a request and the person seeing it. Browser E2E, visual QA, logical QA, product-contract QA, the guardrails and adversarial reviews, `workflow-code-review`, the existing test suite, Storybook footprint, running app checks and any other heavy validation run -only at `$handoff`, never during the loop, even where a repository protocol +at `$handoff`, never during the loop, even where a repository protocol asks for them before a change is done: handoff runs them. The acceptance check's single test, script or control read per criterion is not one of them. The reference check before ready is the one exception to this rule (operator ruling, 2026-10-15), and it runs only with a reference image. +Visual QA can also run between requests, outside the loop, when the person +accepts the offer at a visible milestone or asks for polish ("Visual QA +during the session"; operator ruling, 2026-10-16): after a deploy, never +inside a request's build, and it never holds a deploy back. The reference-product and new Storybook components notices (`SKILL.md`, section 5, "Notices that never wait") stay as they are. @@ -691,6 +749,78 @@ or anywhere else. A gap it sees goes back to the SAME writer in the same request; the ready message names it plainly as not done yet until the writer has fixed it. +## Visual QA during the session + +Operator ruling, 2026-10-16, newer than the lean-loop ruling (2026-10-13, +after the PRD-720 live test) that left visual QA to handoff alone. It adds an +opt-in visual QA the orchestrator offers during the session, in Codex and +Claude Code alike; the build loop above is otherwise unchanged. This section +is the rule's one canonical statement, and handoff's visual QA compares +against the same basis. + +### What it checks + +Visual QA uses the repository's own design pack: its read-only visual +QA skill (`/visual-qa` in Claude Code, `$visual-qa` in Codex), which `/design` +and `$design` route production review through. + +- When the linked ticket or its PRD has acceptance criteria, visual QA checks + the change against them: the run gets the session decision table's + `## Acceptance Criteria` list (those criteria, plus any the person's + requests added) and reports each visual and UX criterion as met or not, + with what it saw. +- When neither has any, it falls back to the design pack's best-practice + review, against `.claude/design/soul.md`, as the skill does when there is + no spec. + +Run it the way handoff runs its visual QA (the handoff skill, "Browser E2E +and the repository's QA passes"): the skill dispatches its own read-only +agents on the session's last deployed commit, with the session's app link +and the screens the change touched. It changes no code. Its confirmed +findings go to the SAME writer as one fix request in the queue, under the +usual rules: design tokens and catalog components, +never new user-visible words (`guardrails.md`, "Copy"), and a finding that +needs a product choice goes through the research and necessity gate above. +That fix deploys like any other request, and the ready message names what +the check found and changed, in the person's terms. Owned prototype sessions +keep their canonical single design review at handoff instead. + +### When it is offered + +At each visible milestone, the ready message offers it, without waiting for +the person to think of it: when a user-visible change is first ready to look +at, and again after a major visible change (a new screen or part of one, or +a reworked layout). Never on a small edit (a spacing, color, wording or +single-control fix), and never for a change that shows nothing visible. Ask +it through `scripts/session-questions.mjs` with the key `visual-qa:` of the milestone's request, so each milestone's offer is asked once, in +exactly these words when the ticket or its PRD has acceptance criteria: + +``` +Want me to check how it looks against the ticket's requirements and fix anything that's off? +``` + +and otherwise: + +``` +Want me to check how it looks against our design standards and fix anything that's off? +``` + +A yes runs it. A no is recorded and declines visual QA for that version of +the change, until the next visible milestone. + +### When the person asks for polish + +When the person says it looks good and wants to see it polished ("looks +good, make it polished", "polish it", "clean it up so it looks finished", +"make it look production ready", or words meaning the same), run visual QA +before handoff without asking, and send its findings and the request to the +SAME writer together. Skip it only when the change shows nothing visible, or +when the person already declined visual QA for this version of the change +(the recorded no for the latest milestone, with no major visible change +since); then the request goes to the writer as an ordinary request under +`guardrails.md`, "Making something look better". + ## Tests: repair, don't author No test authoring in the build loop or at handoff, including new diff --git a/plugins/vibe/skills/vibe/scripts/plugin-version.mjs b/plugins/vibe/skills/vibe/scripts/plugin-version.mjs new file mode 100644 index 00000000..d220e6b2 --- /dev/null +++ b/plugins/vibe/skills/vibe/scripts/plugin-version.mjs @@ -0,0 +1,184 @@ +#!/usr/bin/env node +// The vibe startup version check (operator ruling, 2026-10-16): compares the +// installed vibe plugin with the version on the closedloop-ai marketplace's +// main branch and, when the installed one is older, prints the update +// commands for this harness. It never stops a session: every failure to learn +// the latest version (no network, a refused or slow request, an unreadable +// manifest) prints status `unknown` with the reason, and the exit code is +// always 0. +// +// Usage: plugin-version.mjs --runtime codex|claude [--codex ] +// [--plugin-root ] [--latest-url ] [--timeout-ms ] +// +// Prints one JSON object: +// {"status":"current","installed":"1.0.53","latest":"1.0.53"} +// {"status":"outdated","installed":"1.0.52","latest":"1.0.53","update":["", ...]} +// {"status":"unknown","installed":"1.0.53","reason":""} +// +// The installed version is the running plugin's own manifest, two folders +// above this skill (`.codex-plugin/plugin.json` in Codex, +// `.claude-plugin/plugin.json` in Claude Code). The latest version is the same +// manifest on claude-plugins `main`, the branch both harnesses' Git +// marketplaces install from. `--latest-url` (or VIBE_LATEST_MANIFEST_URL) +// replaces that address, for tests. + +import { accessSync, constants, readFileSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { parseArgs } from "node:util"; + +const MARKETPLACE = "closedloop-ai"; +const RAW_MAIN = "https://raw.githubusercontent.com/closedloop-ai/claude-plugins/main/plugins/vibe"; +export const MANIFESTS = { codex: ".codex-plugin/plugin.json", claude: ".claude-plugin/plugin.json" }; +export const BUNDLED_CODEX = "/Applications/ChatGPT.app/Contents/Resources/codex-cli/bin/codex"; +const DEFAULT_TIMEOUT_MS = 5_000; +const VERSION = /^(\d+)\.(\d+)\.(\d+)$/; +const SAFE_WORD = /^[\w@%+=:,./-]+$/; + +/** -1, 0 or 1 for two x.y.z versions, or null when either is not one. */ +export function compareVersions(left, right) { + const a = VERSION.exec(String(left ?? "").trim()); + const b = VERSION.exec(String(right ?? "").trim()); + if (!a || !b) { + return null; + } + for (let index = 1; index <= 3; index += 1) { + const difference = Number(a[index]) - Number(b[index]); + if (difference !== 0) { + return difference < 0 ? -1 : 1; + } + } + return 0; +} + +/** The commands that update vibe in this harness, already quoted for a shell. */ +export function updateCommands(runtime, { codex = BUNDLED_CODEX, localMarketplace = null } = {}) { + if (runtime === "claude") { + return [ + `claude plugin marketplace update ${MARKETPLACE}`, + `claude plugin update vibe@${MARKETPLACE}`, + ]; + } + const cli = shellWord(codex); + // A local folder marketplace is not refreshed by `marketplace upgrade` + // (INSTALL.md, "Updating"); its checkout is pulled instead. + const refresh = localMarketplace + ? `git -C ${shellWord(localMarketplace)} pull --ff-only` + : `${cli} plugin marketplace upgrade ${MARKETPLACE}`; + return [refresh, `${cli} plugin add vibe@${MARKETPLACE}`, `${cli} plugin add closedloop-core@${MARKETPLACE}`]; +} + +/** The check's result for an installed version and a fetched latest manifest's text. */ +export function decide({ installed, latestText, runtime, codex, localMarketplace = () => null }) { + let latest; + try { + latest = JSON.parse(latestText).version; + } catch { + return { status: "unknown", installed, reason: "the latest manifest is not JSON" }; + } + const order = compareVersions(installed, latest); + if (order === null) { + return { status: "unknown", installed, reason: `cannot compare ${installed} with ${latest}` }; + } + if (order < 0) { + return { status: "outdated", installed, latest, update: updateCommands(runtime, { codex, localMarketplace: localMarketplace() }) }; + } + return { status: "current", installed, latest }; +} + +/** Runs the whole check; never throws. */ +export async function checkVersion({ + runtime, + pluginRoot, + latestUrl, + timeoutMs = DEFAULT_TIMEOUT_MS, + codex, + fetchImpl = fetch, +}) { + let installed; + try { + installed = JSON.parse(readFileSync(path.join(pluginRoot, MANIFESTS[runtime]), "utf8")).version; + } catch (error) { + return { status: "unknown", installed: null, reason: `the installed manifest could not be read: ${error.message}` }; + } + let latestText; + try { + const response = await fetchImpl(latestUrl ?? `${RAW_MAIN}/${MANIFESTS[runtime]}`, { + signal: AbortSignal.timeout(timeoutMs), + }); + if (!response.ok) { + return { status: "unknown", installed, reason: `the marketplace answered ${response.status}` }; + } + latestText = await response.text(); + } catch (error) { + return { status: "unknown", installed, reason: `the marketplace could not be reached: ${error.message}` }; + } + const cli = runtime === "codex" ? (codex ?? findCodex()) : undefined; + // Asked only when an update is needed, so a current install starts no CLI. + const localMarketplace = () => (runtime === "codex" ? codexLocalMarketplace(cli) : null); + return decide({ installed, latestText, runtime, codex: cli, localMarketplace }); +} + +/** The folder of a local Codex marketplace vibe was installed from, or null for a Git one or when unknown. */ +export function codexLocalMarketplace(codex) { + try { + const listed = JSON.parse(execFileSync(codex, ["plugin", "list", "--json"], { encoding: "utf8", timeout: 10_000 })); + const vibe = (listed.installed ?? []).find((plugin) => plugin.pluginId === `vibe@${MARKETPLACE}`); + const source = vibe?.marketplaceSource; + return source?.sourceType === "local" && typeof source.source === "string" ? source.source : null; + } catch { + return null; + } +} + +function findCodex() { + for (const dir of (process.env.PATH ?? "").split(path.delimiter)) { + const candidate = path.join(dir, "codex"); + try { + accessSync(candidate, constants.X_OK); + return candidate; + } catch { + // Not on this PATH entry. + } + } + return BUNDLED_CODEX; +} + +function shellWord(word) { + return SAFE_WORD.test(word) ? word : `'${word.replaceAll("'", String.raw`'\''`)}'`; +} + +async function main() { + const { values } = parseArgs({ + options: { + runtime: { type: "string" }, + codex: { type: "string" }, + "plugin-root": { type: "string" }, + "latest-url": { type: "string" }, + "timeout-ms": { type: "string" }, + }, + }); + if (!(values.runtime in MANIFESTS)) { + process.stdout.write(`${JSON.stringify({ status: "unknown", installed: null, reason: "--runtime must be codex or claude" })}\n`); + return; + } + const scriptDir = path.dirname(fileURLToPath(import.meta.url)); + const timeoutMs = Number(values["timeout-ms"]); + const result = await checkVersion({ + runtime: values.runtime, + pluginRoot: values["plugin-root"] ?? path.resolve(scriptDir, "..", "..", ".."), + latestUrl: values["latest-url"] ?? process.env.VIBE_LATEST_MANIFEST_URL, + timeoutMs: Number.isFinite(timeoutMs) && timeoutMs > 0 ? timeoutMs : DEFAULT_TIMEOUT_MS, + codex: values.codex, + }); + process.stdout.write(`${JSON.stringify(result)}\n`); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + try { + await main(); + } catch (error) { + process.stdout.write(`${JSON.stringify({ status: "unknown", installed: null, reason: error.message })}\n`); + } +} diff --git a/plugins/vibe/skills/vibe/scripts/plugin-version.test.mjs b/plugins/vibe/skills/vibe/scripts/plugin-version.test.mjs new file mode 100644 index 00000000..df5c5b92 --- /dev/null +++ b/plugins/vibe/skills/vibe/scripts/plugin-version.test.mjs @@ -0,0 +1,182 @@ +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { chmodSync, mkdirSync, writeFileSync } from "node:fs"; +import { createServer } from "node:http"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; +import { compareVersions, updateCommands } from "./plugin-version.mjs"; +import { makeHome } from "./test-fixtures.mjs"; + +const SCRIPT = path.join(path.dirname(fileURLToPath(import.meta.url)), "plugin-version.mjs"); +// Nothing listens here, so the check can never reach a real marketplace. +const UNREACHABLE = "http://127.0.0.1:9/plugin.json"; + +/** A plugin root whose two manifests carry `version`. */ +function pluginRoot(t, version) { + const fixture = makeHome("vibe-plugin-version-"); + t.after(fixture.cleanup); + const root = path.join(fixture.root, "vibe"); + for (const dir of [".claude-plugin", ".codex-plugin"]) { + mkdirSync(path.join(root, dir), { recursive: true }); + writeFileSync(path.join(root, dir, "plugin.json"), JSON.stringify({ name: "vibe", version })); + } + return { root, fixtureRoot: fixture.root }; +} + +/** A local marketplace that answers every request with `status` and `body`, or never answers. */ +async function marketplace(t, { status = 200, body = "", hang = false } = {}) { + const requests = []; + const server = createServer((request, response) => { + requests.push(request.url); + if (hang) { + return; + } + response.writeHead(status, { "content-type": "application/json" }); + response.end(body); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + t.after(() => { + server.closeAllConnections(); + server.close(); + }); + return { url: `http://127.0.0.1:${server.address().port}/plugins/vibe/plugin.json`, requests }; +} + +/** A fake Codex CLI whose `plugin list --json` prints `listing`. */ +function fakeCodex(dir, listing) { + const file = path.join(dir, "codex"); + writeFileSync(file, `#!/bin/sh\n[ "$1 $2 $3" = "plugin list --json" ] && printf '%s\\n' '${JSON.stringify(listing)}' && exit 0\nexit 1\n`); + chmodSync(file, 0o755); + return file; +} + +function run(args) { + return new Promise((resolve, reject) => { + const child = spawn(process.execPath, [SCRIPT, ...args], { + env: { PATH: process.env.PATH, HOME: process.env.HOME }, + }); + let stdout = ""; + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.on("error", reject); + child.on("close", (code) => resolve({ code, result: JSON.parse(stdout) })); + }); +} + +test("versions compare numerically and anything else is not comparable", () => { + assert.equal(compareVersions("1.0.52", "1.0.53"), -1); + assert.equal(compareVersions("1.0.53", "1.0.53"), 0); + assert.equal(compareVersions("1.0.100", "1.0.99"), 1); + assert.equal(compareVersions("1.10.0", "1.9.9"), 1); + assert.equal(compareVersions("1.0", "1.0.0"), null); + assert.equal(compareVersions("1.0.52", undefined), null); +}); + +test("an older install is outdated and carries Claude Code's update commands", async (t) => { + const { root } = pluginRoot(t, "1.0.52"); + const latest = await marketplace(t, { body: JSON.stringify({ name: "vibe", version: "1.0.53" }) }); + const { code, result } = await run(["--runtime", "claude", "--plugin-root", root, "--latest-url", latest.url]); + assert.equal(code, 0); + assert.deepEqual(result, { + status: "outdated", + installed: "1.0.52", + latest: "1.0.53", + update: ["claude plugin marketplace update closedloop-ai", "claude plugin update vibe@closedloop-ai"], + }); + assert.equal(latest.requests.length, 1); +}); + +test("an older Codex install updates from its Git marketplace, or pulls a local one", async (t) => { + const { root, fixtureRoot } = pluginRoot(t, "1.0.52"); + const latest = await marketplace(t, { body: JSON.stringify({ version: "1.0.53" }) }); + const git = fakeCodex(fixtureRoot, { + installed: [{ pluginId: "vibe@closedloop-ai", marketplaceSource: { sourceType: "git", source: "closedloop-ai/claude-plugins" } }], + }); + const fromGit = await run(["--runtime", "codex", "--codex", git, "--plugin-root", root, "--latest-url", latest.url]); + assert.equal(fromGit.result.status, "outdated"); + assert.deepEqual(fromGit.result.update, [ + `${git} plugin marketplace upgrade closedloop-ai`, + `${git} plugin add vibe@closedloop-ai`, + `${git} plugin add closedloop-core@closedloop-ai`, + ]); + + const checkout = path.join(fixtureRoot, "Closedloop work", "claude-plugins"); + const local = fakeCodex(fixtureRoot, { + installed: [{ pluginId: "vibe@closedloop-ai", marketplaceSource: { sourceType: "local", source: checkout } }], + }); + const fromLocal = await run(["--runtime", "codex", "--codex", local, "--plugin-root", root, "--latest-url", latest.url]); + assert.equal(fromLocal.result.update[0], `git -C '${checkout}' pull --ff-only`); + assert.deepEqual(fromLocal.result.update.slice(1), [ + `${local} plugin add vibe@closedloop-ai`, + `${local} plugin add closedloop-core@closedloop-ai`, + ]); +}); + +test("the same or a newer install is current and carries no update", async (t) => { + for (const [installed, newest] of [["1.0.53", "1.0.53"], ["1.0.54", "1.0.53"]]) { + const { root } = pluginRoot(t, installed); + const latest = await marketplace(t, { body: JSON.stringify({ version: newest }) }); + for (const runtime of ["claude", "codex"]) { + const { code, result } = await run(["--runtime", runtime, "--codex", "/nonexistent/codex", "--plugin-root", root, "--latest-url", latest.url]); + assert.equal(code, 0); + assert.deepEqual(result, { status: "current", installed, latest: newest }); + } + } +}); + +test("each harness compares its own manifest", async (t) => { + const { root } = pluginRoot(t, "1.0.53"); + writeFileSync(path.join(root, ".codex-plugin", "plugin.json"), JSON.stringify({ version: "1.0.51" })); + const latest = await marketplace(t, { body: JSON.stringify({ version: "1.0.53" }) }); + assert.equal((await run(["--runtime", "claude", "--plugin-root", root, "--latest-url", latest.url])).result.status, "current"); + const codex = await run(["--runtime", "codex", "--codex", "/nonexistent/codex", "--plugin-root", root, "--latest-url", latest.url]); + assert.equal(codex.result.status, "outdated"); + assert.equal(codex.result.installed, "1.0.51"); +}); + +test("an unreachable, failing, slow or garbled marketplace is unknown and never fails the session", async (t) => { + const { root } = pluginRoot(t, "1.0.52"); + const notFound = await marketplace(t, { status: 404, body: "Not Found" }); + const garbled = await marketplace(t, { body: "" }); + const unversioned = await marketplace(t, { body: JSON.stringify({ name: "vibe" }) }); + const slow = await marketplace(t, { hang: true }); + const cases = [ + [UNREACHABLE, /could not be reached/], + [notFound.url, /answered 404/], + [garbled.url, /not JSON/], + [unversioned.url, /cannot compare 1\.0\.52 with undefined/], + ]; + for (const [url, reason] of cases) { + const { code, result } = await run(["--runtime", "claude", "--plugin-root", root, "--latest-url", url]); + assert.equal(code, 0); + assert.equal(result.status, "unknown"); + assert.equal(result.installed, "1.0.52"); + assert.match(result.reason, reason); + assert.equal(result.update, undefined); + } + const started = await run(["--runtime", "codex", "--codex", "/nonexistent/codex", "--plugin-root", root, "--latest-url", slow.url, "--timeout-ms", "200"]); + assert.equal(started.code, 0); + assert.equal(started.result.status, "unknown"); + assert.match(started.result.reason, /could not be reached/); +}); + +test("a missing installed manifest or runtime is unknown, not a failure", async (t) => { + const { fixtureRoot } = pluginRoot(t, "1.0.52"); + const missing = await run(["--runtime", "claude", "--plugin-root", path.join(fixtureRoot, "nowhere"), "--latest-url", UNREACHABLE]); + assert.equal(missing.code, 0); + assert.equal(missing.result.status, "unknown"); + assert.match(missing.result.reason, /installed manifest could not be read/); + const noRuntime = await run([]); + assert.equal(noRuntime.code, 0); + assert.deepEqual(noRuntime.result, { status: "unknown", installed: null, reason: "--runtime must be codex or claude" }); +}); + +test("update commands quote paths with spaces", () => { + assert.deepEqual(updateCommands("codex", { codex: "/Applications/My Apps/codex" }), [ + "'/Applications/My Apps/codex' plugin marketplace upgrade closedloop-ai", + "'/Applications/My Apps/codex' plugin add vibe@closedloop-ai", + "'/Applications/My Apps/codex' plugin add closedloop-core@closedloop-ai", + ]); +}); diff --git a/plugins/vibe/skills/vibe/scripts/precedent-check.mjs b/plugins/vibe/skills/vibe/scripts/precedent-check.mjs new file mode 100644 index 00000000..9fada87d --- /dev/null +++ b/plugins/vibe/skills/vibe/scripts/precedent-check.mjs @@ -0,0 +1,187 @@ +#!/usr/bin/env node +// The precedent check before a product question (operator ruling, +// 2026-10-16; `references/quality-loop.md`, "Precedent before asking"): how +// Asana, or another established app with the same flow, handles the +// situation the question is about. The research brief and its safety rules +// live here once, for both harnesses. +// +// Usage: +// precedent-check.mjs prompt --question [--situation ] [--apps ] +// Prints {"ok":true,"prompt":...}: the brief the Codex orchestrator +// follows itself in its in-app Browser or with Computer Use. +// precedent-check.mjs run --question [--situation ] [--apps ] +// [--out ] [--claude ] [--timeout-seconds ] +// Claude Code: a session's Claude in Chrome tools are bound when it +// starts, so the look runs as a separate headless `claude -p --chrome` +// run with the brief on stdin. Writes prompt.md and report.md to --out +// (a new temporary folder by default) and prints +// {"ok":true,"report":...,"answer":...,"source":...}, or +// {"ok":false,"error":...,"report":...} when the run failed or timed out. +// +// `--apps` names the apps to try after Asana, in order (default: Linear, then +// the service's own screens). Every command prints one JSON object. + +import { spawn } from "node:child_process"; +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, createWriteStream } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { parseArgs } from "node:util"; + +export const TEMP_PROJECT = "zz vibe precedent check (temp)"; +export const TASK_PREFIX = "zz test"; +export const CLAUDE_TOOLS = "mcp__claude-in-chrome__*,WebFetch,WebSearch,Read,Write"; +const DEFAULT_APPS = "Linear, then the service's own screens (for example GitHub's own authorization screens)"; +const DEFAULT_TIMEOUT_SECONDS = 900; +const MAX_TEXT = 2000; +const ANSWER_LINE = /^Answer:\s*(.+)$/m; +const SOURCE_LINE = /^Source:\s*(.+)$/m; + +/** The research brief for one product question. */ +export function precedentPrompt({ question, situation, apps = DEFAULT_APPS, today = new Date() }) { + requireText("question", question); + const date = today.toISOString().slice(0, 10); + return `You are checking how established apps handle one situation, to answer a product question in a vibe session. Report what you see; never change anything that already exists. + +Question: ${question} +Situation to reproduce: ${situation || question} + +Order: +1. Asana, in Chrome, with the account already signed in there. Never sign in, create an account or type a credential. +2. When Asana does not have this situation, another established app with the same flow: ${apps}. +3. When Chrome is not available, or an app has no account signed in, that app's public help center and documentation (WebSearch, WebFetch), Asana's first. +Stop at the first app that settles it. + +Safety, in every app: +- Never touch anything that already exists: no edits, moves, completions, comments, invitations or setting changes. +- When you need test data, create one new private project named "${TEMP_PROJECT}", with no members and notifications off, and put everything in it. Prefix every task you create with "${TASK_PREFIX}". +- A flow that would connect an app, invite or notify someone, share, pay or change a setting: go only as far as its confirmation step, then cancel it. +- At the end, delete every task you created and then the project, check that they are gone, and say so in the report. + +Evidence: +- Verify every state before you record it: reload or reopen and look again. A hover can look like a completed task. +- Keep what you observed directly separate from what a help center or forum says. +- Quote on-screen wording exactly, as evidence only. +- When nothing settles it, write "undocumented"; never guess. + +Reply with only this report: +Answer: +Source: , observed ${date} | | undocumented +Observed directly: +Help center / forum: +Apps without this situation: +Cleanup: +`; +} + +/** The headless Claude Code arguments for the look. */ +export function claudeArgs() { + return ["-p", "--chrome", "--allowedTools", CLAUDE_TOOLS]; +} + +/** The answer and source lines of a report, or null for each line it lacks. */ +export function summarize(report) { + return { + answer: ANSWER_LINE.exec(report)?.[1].trim() ?? null, + source: SOURCE_LINE.exec(report)?.[1].trim() ?? null, + }; +} + +/** Runs the headless look and resolves with its result; never rejects. */ +export function runClaude({ prompt, out, claude = "claude", timeoutSeconds = DEFAULT_TIMEOUT_SECONDS }) { + mkdirSync(out, { recursive: true }); + const promptFile = path.join(out, "prompt.md"); + const reportFile = path.join(out, "report.md"); + writeFileSync(promptFile, prompt); + return new Promise((resolve) => { + const report = createWriteStream(reportFile); + let child; + try { + child = spawn(claude, claudeArgs(), { cwd: out, stdio: ["pipe", "pipe", "ignore"], detached: true }); + } catch (error) { + resolve({ ok: false, error: `claude could not start: ${error.message}`, prompt: promptFile, report: reportFile }); + return; + } + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGKILL"); + } catch { + // Already gone. + } + }, timeoutSeconds * 1000); + child.stdout.pipe(report); + child.on("error", (error) => { + clearTimeout(timer); + resolve({ ok: false, error: `claude could not start: ${error.message}`, prompt: promptFile, report: reportFile }); + }); + child.on("close", (code) => { + clearTimeout(timer); + report.end(() => { + const text = readFileSync(reportFile, "utf8"); + const { answer, source } = summarize(text); + if (timedOut) { + resolve({ ok: false, error: `timed out after ${timeoutSeconds}s`, prompt: promptFile, report: reportFile }); + } else if (code !== 0 || !answer || !source) { + resolve({ + ok: false, + error: code === 0 ? "the report has no Answer or Source line" : `claude exited ${code}`, + prompt: promptFile, + report: reportFile, + }); + } else { + resolve({ ok: true, answer, source, prompt: promptFile, report: reportFile }); + } + }); + }); + child.stdin.on("error", () => { + // The child exited before reading its brief; the close handler reports it. + }); + child.stdin.end(prompt); + }); +} + +function requireText(name, value) { + if (typeof value !== "string" || value.trim() === "" || value.length > MAX_TEXT) { + throw new Error(`--${name} must be 1 to ${MAX_TEXT} characters`); + } +} + +async function main() { + const [command, ...rest] = process.argv.slice(2); + const { values } = parseArgs({ + args: rest, + options: { + question: { type: "string" }, + situation: { type: "string" }, + apps: { type: "string" }, + out: { type: "string" }, + claude: { type: "string" }, + "timeout-seconds": { type: "string" }, + }, + }); + const prompt = precedentPrompt({ question: values.question, situation: values.situation, apps: values.apps }); + if (command === "prompt") { + return { ok: true, prompt }; + } + if (command === "run") { + const timeout = Number(values["timeout-seconds"]); + return runClaude({ + prompt, + out: values.out ?? mkdtempSync(path.join(os.tmpdir(), "vibe-precedent-")), + claude: values.claude, + timeoutSeconds: Number.isFinite(timeout) && timeout > 0 ? timeout : DEFAULT_TIMEOUT_SECONDS, + }); + } + throw new Error("command must be prompt or run"); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + try { + process.stdout.write(`${JSON.stringify(await main())}\n`); + } catch (error) { + process.stdout.write(`${JSON.stringify({ ok: false, error: error.message })}\n`); + process.exitCode = 1; + } +} diff --git a/plugins/vibe/skills/vibe/scripts/precedent-check.test.mjs b/plugins/vibe/skills/vibe/scripts/precedent-check.test.mjs new file mode 100644 index 00000000..d261c97a --- /dev/null +++ b/plugins/vibe/skills/vibe/scripts/precedent-check.test.mjs @@ -0,0 +1,131 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { chmodSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; +import { CLAUDE_TOOLS, TASK_PREFIX, TEMP_PROJECT, precedentPrompt, summarize } from "./precedent-check.mjs"; +import { makeHome } from "./test-fixtures.mjs"; + +const SCRIPT = path.join(path.dirname(fileURLToPath(import.meta.url)), "precedent-check.mjs"); +const QUESTION = "What should the settings page show when someone cancels the GitHub authorization halfway?"; +const REPORT = [ + "Answer: Returns to the integrations page with nothing connected and no error", + "Source: Asana, observed 2026-10-16", + "Observed directly: canceled on GitHub's screen; Asana showed its integrations list unchanged", + "Help center / forum: none needed", + "Apps without this situation: none", + "Cleanup: nothing created", +].join("\n"); + +/** A fake claude that records its arguments, folder and stdin, then prints `output` and exits `code`. */ +function fakeClaude(dir, { output = REPORT, code = 0, sleep = 0 } = {}) { + const file = path.join(dir, "claude"); + const reportFile = path.join(dir, "canned-report.txt"); + writeFileSync(reportFile, output); + writeFileSync( + file, + `#!/bin/sh\nprintf '%s\\n' "$@" > "${dir}/args.txt"\npwd > "${dir}/cwd.txt"\ncat > "${dir}/stdin.txt"\n` + + `${sleep ? `sleep ${sleep}\n` : ""}cat "${reportFile}"\nexit ${code}\n` + ); + chmodSync(file, 0o755); + return file; +} + +function run(args) { + const result = spawnSync(process.execPath, [SCRIPT, ...args], { encoding: "utf8" }); + if (result.error) { + throw result.error; + } + return { status: result.status, json: JSON.parse(result.stdout) }; +} + +function workspace(t) { + const fixture = makeHome("vibe-precedent-"); + t.after(fixture.cleanup); + return fixture.root; +} + +test("the brief checks Asana first, then another app, then help centers, and carries every safety rule", () => { + const prompt = precedentPrompt({ question: QUESTION, today: new Date("2026-10-16T12:00:00Z") }); + const order = ["1. Asana, in Chrome", "2. When Asana does not have this situation", "3. When Chrome is not available"]; + const positions = order.map((line) => prompt.indexOf(line)); + assert.ok(positions.every((position) => position >= 0), "every step is present"); + assert.deepEqual([...positions].sort((a, b) => a - b), positions, "Asana, then another app, then help centers"); + assert.match(prompt, /Linear, then the service's own screens \(for example GitHub's own authorization screens\)/); + for (const rule of [ + "Never sign in, create an account or type a credential.", + "Never touch anything that already exists", + `create one new private project named "${TEMP_PROJECT}", with no members and notifications off`, + `Prefix every task you create with "${TASK_PREFIX}"`, + "go only as far as its confirmation step, then cancel it", + "delete every task you created and then the project, check that they are gone", + "Verify every state before you record it", + "A hover can look like a completed task", + "Keep what you observed directly separate from what a help center or forum says", + "Quote on-screen wording exactly", + 'write "undocumented"; never guess', + "Source: , observed 2026-10-16 | | undocumented", + "Cleanup:", + ]) { + assert.ok(prompt.includes(rule), rule); + } +}); + +test("the named apps replace the default fallback and the question is required", () => { + assert.match(precedentPrompt({ question: QUESTION, apps: "Notion" }), /same flow: Notion\./); + assert.throws(() => precedentPrompt({ question: " " }), /--question/); + assert.equal(run(["prompt"]).status, 1); +}); + +test("prompt mode prints the brief for the Codex orchestrator and starts nothing", () => { + const { status, json } = run(["prompt", "--question", QUESTION, "--situation", "Start connecting GitHub, then cancel"]); + assert.equal(status, 0); + assert.equal(json.ok, true); + assert.match(json.prompt, /Situation to reproduce: Start connecting GitHub, then cancel/); +}); + +test("run mode starts headless claude with Chrome and only the research tools, the brief on stdin", (t) => { + const dir = workspace(t); + const claude = fakeClaude(dir); + const out = path.join(dir, "out"); + const { status, json } = run(["run", "--question", QUESTION, "--claude", claude, "--out", out]); + assert.equal(status, 0); + assert.deepEqual(json, { + ok: true, + answer: "Returns to the integrations page with nothing connected and no error", + source: "Asana, observed 2026-10-16", + prompt: path.join(out, "prompt.md"), + report: path.join(out, "report.md"), + }); + assert.deepEqual(readFileSync(path.join(dir, "args.txt"), "utf8").trim().split("\n"), ["-p", "--chrome", "--allowedTools", CLAUDE_TOOLS]); + assert.equal(readFileSync(path.join(dir, "stdin.txt"), "utf8"), readFileSync(path.join(out, "prompt.md"), "utf8")); + assert.match(readFileSync(path.join(dir, "stdin.txt"), "utf8"), new RegExp(QUESTION.replace(/[?]/g, "\\?"))); + assert.equal(readFileSync(path.join(dir, "cwd.txt"), "utf8").trim(), path.resolve(out)); + assert.equal(readFileSync(path.join(out, "report.md"), "utf8"), REPORT); +}); + +test("a failed, empty or slow run is not an answer", (t) => { + const dir = workspace(t); + const failed = run(["run", "--question", QUESTION, "--claude", fakeClaude(dir, { code: 1 }), "--out", path.join(dir, "a")]); + assert.equal(failed.json.ok, false); + assert.match(failed.json.error, /exited 1/); + const empty = run(["run", "--question", QUESTION, "--claude", fakeClaude(dir, { output: "I looked around." }), "--out", path.join(dir, "b")]); + assert.equal(empty.json.ok, false); + assert.match(empty.json.error, /no Answer or Source line/); + const slow = run(["run", "--question", QUESTION, "--claude", fakeClaude(dir, { sleep: 5 }), "--out", path.join(dir, "c"), "--timeout-seconds", "1"]); + assert.equal(slow.json.ok, false); + assert.match(slow.json.error, /timed out after 1s/); + const missing = run(["run", "--question", QUESTION, "--claude", path.join(dir, "no-claude"), "--out", path.join(dir, "d")]); + assert.equal(missing.json.ok, false); + assert.match(missing.json.error, /could not start/); +}); + +test("the summary reads only the report's Answer and Source lines", () => { + assert.deepEqual(summarize(REPORT), { + answer: "Returns to the integrations page with nothing connected and no error", + source: "Asana, observed 2026-10-16", + }); + assert.deepEqual(summarize("Answer: undocumented\nSource: undocumented\n"), { answer: "undocumented", source: "undocumented" }); + assert.deepEqual(summarize("nothing"), { answer: null, source: null }); +}); diff --git a/plugins/vibe/tools/python/test_vibe_test_repair_lean_loop_contract.py b/plugins/vibe/tools/python/test_vibe_test_repair_lean_loop_contract.py index ed945f80..a36c00cd 100644 --- a/plugins/vibe/tools/python/test_vibe_test_repair_lean_loop_contract.py +++ b/plugins/vibe/tools/python/test_vibe_test_repair_lean_loop_contract.py @@ -8,7 +8,10 @@ 2. Lean build loop. Quick prep, build, the quick checks the commit hooks need, the orchestrator's commit, deploy, show the person. Browser E2E, visual QA, logical QA, product-contract QA, the adversarial and guardrails reviews and other - heavy validation run only at handoff. The two notices stay as they are. + heavy validation run at handoff, never inside the loop. The two notices stay as + they are. The 2026-10-16 operator ruling adds visual QA the person accepts or a + polish request starts, between requests and outside the loop; its own contract + is test_vibe_visual_qa_precedent_version_contract.py. """ from __future__ import annotations @@ -128,7 +131,7 @@ def test_the_build_loop_is_lean() -> None: "deploy, and showing the person", "with no plan review", "Biome on the changed files, the affected typecheck and the source gates", "the guardrails and adversarial reviews", "`workflow-code-review`", - "run only at `$handoff`, never during the loop", "handoff runs them", + "run at `$handoff`, never during the loop", "handoff runs them", "notices (`SKILL.md`, section 5, \"Notices that never wait\") stay as they are"): assert required in lean, required for heavy in HEAVY_VALIDATION: @@ -143,7 +146,8 @@ def test_the_build_loop_is_lean() -> None: assert retired not in loop, retired for skill in QA_SKILLS: assert skill not in loop, skill - assert skill not in QUALITY.read_text(), skill + assert skill not in lean, skill + assert "never inside a request's build" in lean def test_reviewers_run_only_at_handoff() -> None: diff --git a/plugins/vibe/tools/python/test_vibe_visual_qa_precedent_version_contract.py b/plugins/vibe/tools/python/test_vibe_visual_qa_precedent_version_contract.py new file mode 100644 index 00000000..9359b22b --- /dev/null +++ b/plugins/vibe/tools/python/test_vibe_visual_qa_precedent_version_contract.py @@ -0,0 +1,245 @@ +"""Contract checks for the 2026-10-16 operator rulings from a product meeting. + +1. Visual QA checks the change against the acceptance criteria when the ticket + or its PRD has them, and otherwise is the design pack's best-practice review. +2. When the product's behavior in a situation is unspecified, how a mature app + handles the same situation is the default answer. +3. A product question is asked only when neither the graph nor that precedent + settles it: Asana first, then another established app with the same flow, + then their public help centers, with the source recorded and the person's + real data never touched. +4. Every vibe start checks the plugin version, asks "keep going or update" when + it is outdated, helps with the update and asks for a reopen, and carries on + when the check cannot run. +5. The orchestrator offers visual QA at each visible milestone, never on a small + edit, and the build loop is otherwise unchanged. +6. "Looks good, make it polished" runs visual QA before handoff unless the change + shows nothing visible or the person declined it for that version. +""" + +from __future__ import annotations + +from pathlib import Path + +from vibe_contract_text import agent, flat, section + + +PLUGIN_ROOT = Path(__file__).resolve().parents[2] +VIBE = PLUGIN_ROOT / "skills" / "vibe" +VIBE_SKILL = VIBE / "SKILL.md" +REFERENCES = VIBE / "references" +QUALITY = REFERENCES / "quality-loop.md" +GUARDRAILS = REFERENCES / "guardrails.md" +INSTALL = VIBE / "INSTALL.md" +HANDOFF_SKILL = PLUGIN_ROOT / "skills" / "handoff" / "SKILL.md" +PLUGIN_VERSION = VIBE / "scripts" / "plugin-version.mjs" +PRECEDENT = VIBE / "scripts" / "precedent-check.mjs" +RULING = "operator ruling, 2026-10-16" +OFFER_WITH_CRITERIA = "Want me to check how it looks against the ticket's requirements and fix anything that's off?" +OFFER_WITHOUT = "Want me to check how it looks against our design standards and fix anything that's off?" + + +def visual_qa() -> str: + return section(QUALITY.read_text(), "## Visual QA during the session") + + +def precedent() -> str: + return section(QUALITY.read_text(), "### Precedent before asking") + + +def plugin_version() -> str: + return section(VIBE_SKILL.read_text(), "### Plugin version") + + +def assert_all(text: str, required: tuple[str, ...]) -> None: + for phrase in required: + assert phrase in text, phrase + + +def test_visual_qa_checks_the_criteria_or_falls_back_to_the_design_pack() -> None: + assert_all(visual_qa(), ( + "Operator ruling, 2026-10-16", + "in Codex and Claude Code alike", + "`/visual-qa` in Claude Code, `$visual-qa` in Codex", + "`/design` and `$design`", + "When the linked ticket or its PRD has acceptance criteria, visual QA checks the change against them", + "`## Acceptance Criteria` list", + "When neither has any, it falls back to the design pack's best-practice review, against `.claude/design/soul.md`", + "It changes no code.", + "Its confirmed findings go to the SAME writer as one fix request", + "never new user-visible words", + )) + qa = section(HANDOFF_SKILL.read_text(), "### Browser E2E and the repository's QA passes") + assert "against the session's acceptance criteria when the ticket or its PRD has them" in qa + assert "otherwise as the design pack's best-practice review" in qa + assert '"Visual QA during the session"' in qa + + +def test_visual_qa_is_offered_at_visible_milestones_and_never_on_a_small_edit() -> None: + rule = visual_qa() + assert_all(rule, ( + "At each visible milestone, the ready message offers it, without waiting for the person to think of it", + "when a user-visible change is first ready to look at, and again after a major visible change", + "Never on a small edit", + "never for a change that shows nothing visible", + "`scripts/session-questions.mjs` with the key `visual-qa:`", + "each milestone's offer is asked once", + OFFER_WITH_CRITERIA, + OFFER_WITHOUT, + "A no is recorded and declines visual QA for that version of the change, until the next visible milestone.", + )) + deploy = section(VIBE_SKILL.read_text(), "## 6. Deploy each change") + assert_all(deploy, ( + "When this deploy is a visible milestone", + "never a small edit", + "the ready message also offers visual QA, without waiting for the person to think of it", + RULING, + )) + + +def test_a_polish_request_runs_visual_qa_unless_nothing_is_visible_or_it_was_declined() -> None: + assert_all(section(QUALITY.read_text(), "### When the person asks for polish"), ( + "\"looks good, make it polished\"", + "run visual QA before handoff without asking", + "Skip it only when the change shows nothing visible, or when the person already declined visual QA for this version of the change", + "an ordinary request under `guardrails.md`, \"Making something look better\"", + )) + deploy = section(VIBE_SKILL.read_text(), "## 6. Deploy each change") + assert "wants it polished, run visual QA before handoff without asking" in deploy + assert "already declined it for this version" in deploy + + +def test_the_build_loop_is_otherwise_unchanged() -> None: + lean = section(QUALITY.read_text(), "## Lean build loop") + assert "never during the loop" in lean + assert "Visual QA can also run between requests, outside the loop" in lean + assert "never inside a request's build, and it never holds a deploy back" in lean + assert "The reference check before ready is the one exception to this rule" in lean + loop = section(VIBE_SKILL.read_text(), "## 5. Build loop") + assert "Visual QA runs at handoff too, and during the session only when the person accepts the offer or asks for polish" in loop + assert "No reviewer, browser E2E, visual QA, logical QA" in loop + writer = flat(agent("vibe-change-worker")) + assert "do not run or request them in the loop" in writer + assert "its confirmed findings reach you as a fix request" in writer + guardrails = flat(GUARDRAILS.read_text()) + assert "Outside the loop, visual QA also runs during the session when the person accepts it or asks for polish" in guardrails + assert "Reviews and heavy validation run only at handoff" not in guardrails + + +def test_precedent_is_checked_in_order_before_any_question() -> None: + rule = precedent() + assert_all(rule, ( + "Operator ruling, 2026-10-16", + "A question is asked only when neither the graph research above nor another app's precedent answers it", + "what a mature app does in the same situation is the default answer", + "The order is Asana", + "another established app with the same flow (for example Linear, or GitHub's own authorization screens)", + "public help centers, Asana's first", + "Only when none of these settles it does the question go to the person", + )) + assert rule.index("The order is Asana") < rule.index("another established app") < rule.index("public help centers") + communication = section(QUALITY.read_text(), "## Communication and product questions") + assert "runs the precedent check below before asking any question it can settle" in communication + skill = flat(VIBE_SKILL.read_text()) + assert "then through how Asana, or another established app with the same flow, handles the same situation" in skill + assert "and the precedent check (`references/quality-loop.md`, \"Communication and product questions\")" in skill + assert 'then the precedent check ("Precedent before asking")' in flat(HANDOFF_SKILL.read_text()) + + +def test_each_harness_has_its_precedent_path() -> None: + assert_all(precedent(), ( + "`scripts/precedent-check.mjs` holds the research brief and its safety rules once, for both harnesses", + "Claude Code: a session's Claude in Chrome tools are bound when it starts", + "`node scripts/precedent-check.mjs run --question", + "headless `claude -p --chrome`", + "Codex: `node scripts/precedent-check.mjs prompt --question", + "in a tab of its own in the in-app Browser (never the person's app tab) or with the Computer Use plugin", + "read-only `vibe-requirements-worker` in precedent mode", + )) + assert "`scripts/precedent-check.mjs`" in section(VIBE_SKILL.read_text(), "## Your role: orchestrate, never do the work") + mode = section(agent("vibe-requirements-worker"), "## Precedent mode") + assert_all(mode, ( + "`../skills/vibe/scripts/precedent-check.mjs prompt`", + "Asana's help center and documentation first", + "Never sign in, create an account or type credentials.", + "never its wording as our copy", + "the answer is `undocumented`", + )) + + +def test_precedent_research_never_touches_real_data() -> None: + assert_all(precedent(), ( + "nobody signs in, creates an account or types a credential", + "Nothing that already exists is touched", + "one new private project named `zz vibe precedent check (temp)`, with no members and notifications off", + "every created task prefixed `zz test`", + "everything created is deleted at the end and confirmed gone in the report", + "goes only as far as its confirmation step and is canceled there", + "Every state is verified", + "a hover can look like a completed task", + "what was observed directly stays separate from help center or forum material", + "reported as `undocumented`, never guessed", + )) + brief = PRECEDENT.read_text() + assert 'TEMP_PROJECT = "zz vibe precedent check (temp)"' in brief + assert 'TASK_PREFIX = "zz test"' in brief + + +def test_a_precedent_is_recorded_with_its_source_and_is_not_a_human_ruling() -> None: + assert_all(precedent(), ( + "records it in the session decision table as the decision for the affected rows, with its source", + "`Asana, observed `", + "the help-center URL", + "A precedent is the default, not a human ruling", + "a later human statement supersedes it with an explicit `Superseded` entry", + "settles behavior, never user-visible words", + )) + writer = flat(agent("vibe-change-worker")) + assert "record it in the session decision table as the decision for the affected rows with its source" in writer + assert "never as a human ruling" in writer + + +def test_the_version_check_runs_at_every_start_and_never_blocks() -> None: + assert_all(plugin_version(), ( + "Every `$vibe` start, new or resume, first checks whether this plugin is the latest release", + RULING, + "`node scripts/plugin-version.mjs --runtime codex` in Codex, or `--runtime claude` in Claude Code", + "`current`: say nothing about it and continue.", + "the check never blocks", + "\"I couldn't check whether vibe is up to date, so I'm carrying on with the version you have.\"", + "\"A newer version of vibe is available (you have , the latest is ). Do you want to keep going with this one, or update now?\"", + "On \"keep going\", continue with this version.", + "run each command in its `update` list in order yourself", + "Quit the ChatGPT app completely (press Cmd and Q), open it again, start a new chat and type $vibe.", + "Close this Claude Code session, start a new one and type /vibe:vibe.", + "Then stop: the update is used only from a new session.", + "never retry it another way", + )) + asked_once = section(VIBE_SKILL.read_text(), "### Questions are asked once") + assert "except the startup version question, which runs once per conversation" in asked_once + + +def test_the_person_chosen_update_is_the_one_plugin_change_allowed() -> None: + tooling = section(GUARDRAILS.read_text(), "## Plugins and tooling") + assert_all(tooling, ( + "The one exception is the vibe update the person chooses when the startup version check finds a newer release", + RULING, + "the orchestrator runs exactly the update commands `plugin-version.mjs` printed, and nothing else", + )) + role = section(VIBE_SKILL.read_text(), "## Your role: orchestrate, never do the work") + assert "apart from the update the person chooses at startup" in role + assert "run the update commands `plugin-version.mjs` prints, only when the person chose to update" in role + + +def test_the_update_commands_match_the_install_guide() -> None: + script = PLUGIN_VERSION.read_text() + install = flat(INSTALL.read_text()) + for command in ("plugin marketplace upgrade closedloop-ai", "plugin add vibe@closedloop-ai", + "plugin add closedloop-core@closedloop-ai"): + assert command in install, command + assert "plugin marketplace upgrade ${MARKETPLACE}" in script + assert "plugin add vibe@${MARKETPLACE}" in script + assert "plugin add closedloop-core@${MARKETPLACE}" in script + assert "claude plugin marketplace update ${MARKETPLACE}" in script + assert "claude plugin update vibe@${MARKETPLACE}" in script + assert "`$vibe` checks this itself at the start of every session" in install