Builder calibration: five bounded deliveries each for Cursor, Muse, Antigravity, and Devin

[jeffhuber]

Goal

Collect enough comparable delivery evidence to evaluate Cursor, Muse, Antigravity, and Devin as Code Mower builders without manufacturing low-value production changes.

Sampling target

Complete five bounded, real issues per provider. Balance each provider's five samples across the same task classes where practical:

1. focused test or fixture hardening;
2. small CLI text/JSON consistency change;
3. documentation or adoption-flow correction;
4. privacy/redaction guardrail;
5. provider/configuration validation.

Each issue should normally fit below 300 changed lines and finish within one working session. Existing real backlog is preferred; do not create duplicate implementations merely to complete a quota.

Controlled protocol

• Freeze the task contract and acceptance criteria before selecting the builder.
• Use a fresh branch/worktree and explicit builder identity.
• Do not reveal peer-review findings until the builder declares its first implementation complete.
• Require established Codex and Claude review, excluding the author lane.
• Record all audit findings with one disposition: accepted-fixed, owner-decided, false-positive, duplicate, or infrastructure.
• Record start-to-PR, PR-to-merge, total elapsed time, interventions, CI retries, audit/fix rounds, accepted severity, reported tokens/cost, and post-merge health.
• Treat unavailable cost as unavailable, never zero.
• Upload metadata-only evidence after dry-run inspection. Never upload source, diffs, transcripts, issue bodies, raw stdout/stderr, auth output, local paths, or secrets.

Builder targets

• Cursor: 3 / 5; sample 4 active on Provider registry: separate Cursor Cloud Agent and Grok Bot identities #717 / PR Provider registry: separate Cursor Cloud Agent and Grok Bot identities #722
• Muse: 4 / 5
• Antigravity: 3 / 5
• Devin: 0 / 5

Completed samples

• Cursor: v1.0.4: audit installation and documentation flow end to end #657 / PR docs: audit v1.0.3 installation and documentation flow #662; Release qualification: local adoption-result contract and runner #669 / PR Release qualification: local adoption-result contract and runner #672; Preserve audit verdicts when metadata findings omit source lines #681 / PR Preserve audit verdicts when metadata findings omit source lines #715.
• Muse: v1.0.4: tighten offline tests and adoption diagnostics #656 / PR Tighten offline tests and adoption diagnostics #660; Guard write-capable pull_request_target jobs from fork PRs #664 / PR Guard write workflows from fork pull requests #665; Make green gate auto-merge robust across GitHub merge-state transitions #666 / PR fix: make green gate auto-merge resilient #667; Release campaign: preserve retry chronology and upload terminal failure evidence #711 / PR Release campaign: preserve retry chronology and upload terminal failure evidence #716.
• Antigravity: Harden generated PR workflows around DISPATCH_TOKEN #654 / PR Harden generated DISPATCH_TOKEN workflows #661; Release campaign: deterministic supported runtime and truthful adapter readiness #710 / PR Enforce deterministic Python 3.12+ runtime and adoption readiness (#710) #719; Antigravity adapter: isolate every campaign in a new project #721 / PR Antigravity campaign argv isolation and doctor readiness capability check (#721) #724.

Infrastructure evidence that does not count as a completed sample

• Release campaign: discover state consistently across worktrees #712: Devin received one explicit paid/opt-in work order but did not acknowledge within the bounded wait. No automatic retry was sent and no delivery sample is claimed.
• Release campaign: discover state consistently across worktrees #712 / PR Discover release campaigns across worktrees #720: Grok built the initial cross-worktree discovery change. The current Grok-to-Cursor alias misrouted its blocked-audit fix round to Cursor; that Cursor correction is retained as provider-routing evidence for Provider registry: separate Cursor Cloud Agent and Grok Bot identities #717, not counted as a comparable Cursor delivery sample.

Remaining samples must come from real backlog. Do not create low-value or duplicate changes solely to complete the target.

Antigravity and Devin currently require explicit manual adapter sessions. Their dispatch/setup time must be recorded as orchestration overhead rather than omitted. The experiment should inform whether first-class adapters are worth implementing.

Decision output

After 20 completed samples, publish a provider scorecard with confidence intervals and a promotion recommendation for each role. Do not rank providers solely by raw BLOCKED rate because reviewer assignment and task mix can confound it.

[jeffhuber]

Muse sample 1 / 5 completed through #656 and PR #660.

• Task class: package runtime and adoption diagnostics.
• Initial authoring: 390.7 seconds.
• Fix round: 56.0 seconds.
• Orchestration interventions: 1 CLI invocation correction; the failed pre-model invocation is retained locally as experiment evidence.
• Finding dispositions: 1 accepted-fixed orchestration finding (duplicate text rendering); 0 peer-audit findings; 0 false positives; 0 infrastructure findings after authoring began.
• Review: Codex PASS, Claude PASS, Gitar approved.
• CI: Python 3.12, 3.13, 3.14, package, static, privacy, and smoke checks passed.
• Merge: Code Mower gate green; merged at the final audited head.
• Cost/model: unavailable, not zero.
• Cloud: one metadata-only builder_run accepted; no source, diff, transcript, stdout/stderr, auth output, local path, or secret uploaded.

Progress: Muse 1 / 5; Cursor 0 / 5; Antigravity 0 / 5; Devin 0 / 5.

[jeffhuber]

Antigravity calibration sample 1/5 completed on #654 / PR #661.

• Task class: workflow security and regression tests.
• Builder: Antigravity CLI; model and tool version unavailable rather than inferred.
• Final head: a6d0460.
• Builder fix rounds: 0.
• Orchestrator interventions: 2 setup corrections after authoring (rebase to current main; audit checkout topology correction).
• Verification: 863 unit tests, focused tests, Ruff, compileall, privacy scan, Python 3.12/3.13/3.14 package checks.
• Reviews: Codex PASS in 39s; Claude PASS in 64s; Gitar approved. No findings.
• Disposition: known-clean pass; merged as 1004101.
• CodeMower.com: one builder_run and two reviewer_run events accepted; metadata only, no reports or source content.

Calibration progress: Cursor 0/5 (first sample active on #657), Muse 1/5, Antigravity 1/5, Devin 0/5.

[jeffhuber]

Cursor calibration sample 1/5 completed on #657 / PR #662.

• Builder: Cursor hosted agent (cursor_cloud_agent)
• Final builder head: 4838487e088fdb97aff64ad83668ca74db088c6d
• Merge: a2267db718893671e28eb985b748b8ecaea858cf
• Recorded elapsed time: 1,259 seconds from dispatch to final candidate
• Human owner interventions: 0
• Orchestrator interventions: 2 accepted fix rounds before merge-authority audit

Finding dispositions:

1. The first candidate crossed issue ownership boundaries by changing the package/release version and release files, while failing 14 release-hygiene tests. Disposition: accepted and fixed; the final PR contains only the installation/documentation audit artifact.
2. The second candidate contained incorrect provenance, incomplete scope, an unsupported exhaustive-link claim, overstated CLI verification, a generic root artifact name, and three whitespace errors. Disposition: accepted and fixed; the final artifact records the correct base commit, bounded scope, reproducible evidence, and lives under docs/.
3. Final Codex merge-authority audit: PASS in 68 seconds, no findings.
4. Final Claude merge-authority audit: PASS in 14 seconds, no findings.
5. Gitar: approved; informational corroboration.

Final verification: 863 tests passed, Ruff passed, compileall passed, privacy scan passed, git diff --check passed, 172 local Markdown links/anchors checked with zero broken, and all remote package checks passed on Python 3.12, 3.13, and 3.14.

CodeMower.com metadata-only uploads:

• Builder run: 103b30cb-f1ce-4941-a534-892f88979c45
• Reviewer runs: 709df20c-0cf1-4c16-ad99-20a9fc239709

No source, diffs, transcripts, report text, raw stdout/stderr, auth output, or secrets were uploaded.

[jeffhuber]

v1.0.4 calibration checkpoint:

• Cursor: 1/5 completed through v1.0.4: audit installation and documentation flow end to end #657 / PR docs: audit v1.0.3 installation and documentation flow #662
• Muse: 1/5 completed through v1.0.4: tighten offline tests and adoption diagnostics #656 / PR Tighten offline tests and adoption diagnostics #660
• Antigravity: 1/5 completed through Harden generated PR workflows around DISPATCH_TOKEN #654 / PR Harden generated DISPATCH_TOKEN workflows #661
• Devin: 0/5

Each completed sample used established Codex and Claude peer reviewers, retained explicit finding dispositions, passed the final-head gate, and uploaded metadata-only builder/reviewer evidence to CodeMower.com. Gitar provided informational corroboration.

The experiment remains open. I am deliberately not creating synthetic low-value PRs to reach the counts: that would make the providers look comparable while weakening the evidence. Remaining samples should come from bounded real backlog items with similar task classes, verification, final-head audits, and explicit accepted/fixed, false-positive, owner-decision, or infrastructure dispositions. v1.0.4 itself is complete and does not depend on reaching 5/5.

[jeffhuber]

Muse calibration sample 2/5 completed on #664 / PR #665.

• Task class: workflow security, generated-template consistency, and regression tests.
• Builder: Muse CLI 1.0.2, model muse-spark-1.3-contributor.
• Final head: 9d4184976e4cbdbc54da4905c99eb6a04815f9f1.
• Merge: 904971d3129abf84bd879452ea115bc79fa7f5d6.
• Builder fix rounds: 1.
• Orchestrator interventions: 1 pre-model checkout-topology correction; 1 accepted peer-finding fix.
• First implementation verification: 868 tests, Ruff, privacy, compileall, and generated-workflow actionlint passed on the host. Fifteen Board socket tests failed only in Muse's restricted sandbox and passed on the host, recorded as infrastructure evidence.
• Finding disposition: Codex P1 codex:0369ee4b72460290810c accepted-fixed because active checked-in workflows initially lacked the new fork guard. The final patch protects templates, packaged mirrors, fallback content, and active workflows.
• Final review: Codex PASS in 54 seconds; Claude PASS in 56 seconds; Gitar approved.
• Final verification: 869 unit tests, 6 focused tests, Ruff, generated-workflow actionlint, and diff checks passed.
• Owner interventions: 0. The orchestrator manually merged after the green gate because GitHub rejected the gate's auto-merge mutation during a transient UNSTABLE state; that automation race is tracked separately and is not attributed to Muse.
• Cost: unavailable, not zero.
• CodeMower.com: one metadata-only builder event and four reviewer events accepted. No source, diffs, transcripts, report text, raw stdout/stderr, auth output, local paths, or secrets were uploaded.

Progress: Cursor 1/5; Muse 2/5; Antigravity 1/5 with sample 2 active on dashboard #154; Devin 0/5.

[jeffhuber]

Muse calibration sample 3/5 completed on #666 / PR #667.

• Task class: gate reliability and exact-head auto-merge behavior.
• Builder: Muse CLI 1.0.2, model muse-spark-1.3-contributor.
• Final head: 63a888acc8dbd1cde4f1dd9944eb61b475240447.
• Merge: 9ef39b00fd17d99724ab0af26e65877440353306.
• Builder elapsed time: approximately 720 seconds.
• Builder fix rounds and owner interventions: 0 during implementation.
• Verification: 879 unit tests, focused gate tests, Ruff, compileall, privacy scan, easy-mode smoke, generated-workflow actionlint, and diff checks passed.
• Review: Codex PASS in 43 seconds; Claude first returned UNKNOWN after 116 seconds when structured-output retries were exhausted, then PASS on the same final head in 176 seconds; Gitar approved.
• Finding dispositions: 0 accepted code findings; 1 Claude infrastructure retry; 0 owner decisions or false positives.
• Merge behavior: the old default-branch workflow reproduced the known auto-merge failure while the gate stayed green. The PR was merged under standing owner authority after both final-head audits passed. The newly merged workflow is available for the next PR to exercise.
• CodeMower.com reviewer upload: 96fd9373-bdbd-4992-82c0-22b549720625 (3 events).
• CodeMower.com builder upload: 1964be88-7907-4b7c-8701-ed1f098774cc (1 event).
• Uploads contained metadata only and zero report text; no source, diffs, prompts, transcripts, issue bodies, raw stdout/stderr, auth output, local paths, or secrets were uploaded.

Progress: Cursor 1/5, Muse 3/5, Antigravity 1/5, Devin 0/5: 5/20 complete. Cursor sample 2 is now active on #669 under release-qualification epic #668.