Modernize for the Claude 5 era: model/effort policy, workflow layer, refreshed reviewers, test harness - #4
Modernize for the Claude 5 era: model/effort policy, workflow layer, refreshed reviewers, test harness#4dennisonbertram wants to merge 6 commits into
Conversation
…refreshed external reviewers, test harness - Pin model AND reasoning effort on all 14 agents: Fable coordinator (thin judgment-only control plane), Opus only where reasoning is the deliverable (reviewer @ xhigh, intent-validator), Sonnet 5 for all execution roles, Haiku for mechanical I/O. Policy + traps documented in README. - Add workflows/ (coord-implement, coord-review, coord-verify-product): deterministic fan-out scripts that batch the delegate/integrate, review, and test phases into one structured result each — schema-validated worker output, JS-level TDD-evidence gate, in-workflow artifact recording, and tiered adversarial review verification (root-cause consolidation, Opus xhigh only for critical/high). - Refresh external second-opinion models: gpt-5.4 -> gpt-5.5, gemini-3.1 -> gemini-3.1-pro-preview, with GLM-5.2 / DeepSeek V4 Pro documented as open cheap alternatives; schema field names are now version-neutral (external_code_review / external_visual_review / external_ux_review, each recording the model that ran). This is a breaking schema change. - Add test-harness/: fixture repo with planted defects, task contracts, token/cost metrics collector, and measured RESULTS.md. Live runs verified the TDD gates from git history and drove the review optimization (per-run cost $22.96 -> $11.37; Opus spend -84%). - install.sh now installs workflows into the project's .claude/workflows/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 836efb5ed2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const RESULT_SCHEMA = { | ||
| type: 'object', | ||
| additionalProperties: true, | ||
| required: ['task_id', 'task_type', 'status', 'scope_completed', 'files_changed', 'summary'], |
There was a problem hiding this comment.
When coord-implement delegates to the existing worker agents, their mandatory per-agent schemas do not include status and several also do not include summary, while the canonical schemas disallow extra fields. This workflow requires those fields before accepting a result, so a worker that follows its agent contract gets rejected/retried, while a worker that satisfies the workflow writes .coord/tasks artifacts that no longer validate with bin/coord-validate. This breaks the advertised schema-validated implement path for normal worker results; update the worker schemas/prompts or validate against the canonical per-agent schemas before recording.
Useful? React with 👍 / 👎.
| if (!c[k] || !c[k].hash || c[k].hash.trim() === '') { | ||
| return `missing audit-trail ${k}-commit hash (required for TDD task type '${task.type}')` |
There was a problem hiding this comment.
For projects that are not git repositories, worker.md and schemas/worker-output.schema.json explicitly allow each audit-trail entry to be { "status": "n/a — no git" } while still reporting the TDD test evidence. This gate only accepts a non-empty .hash, so those legitimate no-git feature/bugfix results are marked failed after retry even though they conform to the worker contract. Accept the documented n/a — no git form when the evidence fields are present.
Useful? React with 👍 / 👎.
| COVERAGE, | ||
| `Read the actual files/diffs — do not review from the description alone.`, | ||
| ].filter(Boolean).join('\n'), | ||
| { agentType: 'reviewer', model: 'sonnet', effort: 'high', schema: FINDINGS_SCHEMA, label: `find:${d.key}`, phase: 'Find' }, |
There was a problem hiding this comment.
These dimension finder calls use the reviewer agent, whose instructions unconditionally require running the external llm review pass on every review invocation. In reviews where external is enabled, the correctness/security/concurrency/tests finders therefore each attempt the external pass before the dedicated find:external call, and verifier calls repeat the same requirement for every cluster, multiplying cost/time and failure points. Add a workflow-specific exemption or a non-external reviewer mode for internal finder/verifier passes.
Useful? React with 👍 / 👎.
… the Claude Code plugin - pi/ is a self-contained Pi Package (npm/git-shareable): a coordinator.ts extension registering coord_implement / coord_review / coord_verify tools that spawn `pi --mode json -p --no-session` worker processes per agent role, with git-worktree isolation + serial merge-back, fenced-JSON result contracts with a deterministic TDD-evidence gate (one retry), root-cause consolidation + severity-tiered adversarial review verification, and .coord/ artifact recording done in TypeScript (zero tokens). - 11 agents in Pi frontmatter format (worker/refactor/test/investigation, planner, reviewer @ opus:xhigh, three testers, briefer, learning-extractor); coordinator and scribe roles are deliberately dropped (your session is the coordinator; recording is deterministic code). /coord-* prompt templates drive the phase discipline incl. in-session intent validation. - Installation is clarified everywhere: the Claude flavor installs via claude plugin / install.sh into ~/.claude; the Pi flavor installs ONLY via `pi install /path/to/repo/pi [-l]` (Pi's package manager). Comparison table in pi/README.md and the main README. - Verified: extension parses (esbuild), `pi install` + `pi list` succeed in a scratch project, and RPC get_commands proves the extension and all six prompt templates register (tool registration precedes the /coord-agents command in the load path). Not yet exercised against a live provider — flagged as shakedown status in pi/README.md. - Also adds docs/pi-workflow-delivery.md (the research behind the design). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Second commit adds pi-coordinator ( |
…ion git gate Both delivery modes exercised against a live provider (openai/gpt-4o via project .pi/agents overrides): - Headless (--mode json -p): session model called coord_implement, worker child fixed the planted median bug in a worktree, gate passed on REAL commits, merge landed, artifacts recorded. Verified independently. - Interactive TUI in tmux: /coord-agents roster (project overrides win), natural-language prompt triggered coord_verify, system-tester child ran the suite, PASS rendered live. Run 1 caught a real defect: a worker fabricated success (invented commit hashes + fake test output) and the shape-only gate let it merge a no-op branch. New verifyGitClaims(): claimed hashes must exist, be reachable from branch HEAD, be three distinct commits, and the branch must be ahead of base — fabrication errors feed the single retry. Also: thinking-level tier suffix only applied to models that declare one; stale worktree cleanup; results in test-harness/RESULTS-pi.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
… state - Coordinator authors inline Workflow scripts per phase; drop shipped named workflows (coord-implement/review/verify-product) and bin/coord-validate — output schemas now validate at the tool layer via the Workflow schema option - Coordinator tools: + SendMessage, Task tools (live visibility only), Skill (loads /efficient-fable at startup), Read (spot-verification) - Planner Sonnet -> Opus; external models via codex MCP: GPT-5.6 terra (medium) for well-specified implementation, GPT-5.6 Sol (xhigh) reviewer second opinion; drop GPT-5.5/Gemini llm-CLI passes - State trims: remove milestones/M-XXX.json and execution-brief.md, merge current-intent + command-intent into docs/context/intent.md, save review artifacts only when findings triggered re-delegation - Fix stale patterns: fork is subagent_type "fork"; background is the spawn default, intent-validator needs run_in_background: false Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S3Z8Q7P8bQwVLLNUrxqKGV
- Update plugin description, reviewer/tester examples, and output schemas to codex-MCP models (gpt-5.6-sol); drop llm-CLI/Gemini/GLM external-pass fields from ui/ux tester agents and schemas - Rewrite test-harness README and tasks.json for inline-authored implement workflow; fix setup-fixture.sh copying deleted workflows/; mark RESULTS.md as a historical record of the removed named workflows - intent-validator schema: command-intent.md -> intent.md - pi/README comparison table no longer claims install.sh ships workflows; fix stale comment in pi coordinator extension - Remove stale 2026-07-modernization.pdf duplicate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S3Z8Q7P8bQwVLLNUrxqKGV
- Planner: size tasks to one focused pass — one behavior, exact file paths, concrete verification; split anything needing a paragraph - Coordinator: intake and plan phases run inline checklists (placeholder scan, ambiguity, scope, coverage) instead of extra agent round-trips; task-size guidance reconciled to one-coherent-behavior-per-task Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S3Z8Q7P8bQwVLLNUrxqKGV
647e3e8 to
f6197c3
Compare
1 participant
Summary
xhigh) and intent-validator, Sonnet 5 for every execution role with per-role effort, Haiku for mechanical I/O. Rationale + known traps documented in README → Model & Effort Policy.workflows/): three deterministic scripts batching the fan-out phases —coord-implement(route by type, worktree isolation, schema-validated output, JS-level TDD-evidence gate, in-workflow artifact recording),coord-review(coverage-first finders → root-cause consolidation → tiered adversarial verification),coord-verify-product(parallel testers). The coordinator sees one structured result per phase instead of N round trips in its Fable context.gemini-3.1-pro-preview, plus GLM-5.2 / DeepSeek V4 Pro documented as open cheap alternatives (code-review role only — no vision). Schema fields are now version-neutral (external_code_reviewetc., each recording themodelthat ran) so future model bumps aren't breaking changes. The rename itself is a one-time breaking schema change.test-harness/): fixture repo with planted defects, task contracts, and a token/cost metrics collector — plusRESULTS.mdfrom live runs.Measured results (live harness runs, API-equivalent estimates, n=1 each)
docs/2026-07-modernization.md(+ PDF).Test plan
test-harness/RESULTS.md)install.shsyntax-checked; installs workflows into.claude/workflows/🤖 Generated with Claude Code