[vincentkoc]

Summary

A repeated single-task Fusion comparison always ran its arms in the same order, even though the runner rotated across different tasks. Add an explicit zero-based arm offset so predeclared repeated blocks can use different starting arms. Preserve canonical strategy labels and default ordering.

Receipts and analysis retain offsets; legacy receipts use zero. Repeated analysis permits different offsets while continuing to require matching model/scoring protocols and consecutive spending ledgers. The offset is validated before any provider preflight or output creation.

Verification

• 99 focused pair runner/analysis tests passed after rebase; 1 optional Docker integration case skipped.
• Fixtures verify actual execution and ledger order for all three offsets, unchanged scores, early rejection of invalid offsets and repeated-analysis compatibility.
• Independent P0–P2 review: no actionable findings.
• git diff --check passed.

Deployment and risk

Benchmark scripts only; no deployment or default routing change. No paid calls, budget changes, retries or scoring changes. Rotation does not eliminate all cache and order effects; schedules must be declared before inference.

Rebased onto the landed probe-qualified handoff and fresh Azure task. Both sets of regression tests are retained; the final rebased patch passed independent P0–P2 review. Fresh CI applies to the new head.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

Adds an explicit starting-arm offset to Fusion benchmarks and retains it in single-block and repeated analysis.

Example: Run a single-task matched-handoff block with FUSION_PAIR_ARM_OFFSET=1.

• Before: Every block executes cheap, strong, then edit-handoff.
• After: The block executes strong, edit-handoff, then cheap, while retaining canonical strategy labels.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Focused, source-supported implementation with meaningful harness coverage and successful script CI; evidence remains limited to synthetic benchmark execution.
Proof confidence	🦐 gold shrimp (3/6)	This maintainer-authored benchmark-only change has focused runner and analyzer coverage with synthetic provider responses; shipped production behavior is unchanged.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Feature · Worth it: Yes · Fix scope: Complete
User problem: Repeated single-task comparisons cannot vary their starting arm, leaving each strategy tied to the same execution position.
Reason: The established feature owner proposes a small opt-in extension that improves experimental control and explicitly avoids claiming complete counterbalancing or model superiority.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR provides useful benchmark control absent from current main, with no actionable correctness or security findings.

Priority: P3
Reviewed head: 8b8ef1a9041fed94ff26269e6346c14ff71971ec

Before merge

None.

Findings

None.

Agent review details

How this fits together

Offline Fusion tooling consumes qualified repair tasks and explicit model budgets, executes comparison arms, writes accounting receipts, and analyzes settled results.

flowchart LR
  A[Qualified tasks and configuration] --> B[Pair benchmark runner]
  B --> C[Ordered comparison arms]
  C --> D[Provider calls and repair grading]
  D --> E[Pair receipts and spending ledger]
  E --> F[Single or repeated analysis]

Loading

Technical review

Best possible solution:

Retain the focused offset implementation, legacy receipt compatibility and documented requirement to predeclare comparison schedules.

Do we have a high-confidence way to reproduce the issue?

Current main computes arm order from the task index alone; a one-task manifest therefore starts every repeated block with cheap, and exposes no block offset.

Is this the best way to solve the issue?

The change extends the existing rotation owner with one validated value and additive receipt metadata, preserving scoring, accounting and default behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 43d978e80dfa.

Provenance checked

• Benchmark arm rotation keeps the original intent (24d7379: feat(fusion): benchmark model delegation with matched controls #919 rotates controls to reduce systematic cache-order effects.)
• Repeated analysis protocol matching keeps the original intent (a1772b0: feat(fusion): validate repeated benchmark comparisons #965 requires matching experiment settings and consecutive spending ledgers; the new scheduling metadata leaves those checks intact.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

• Repository policy applied: Read the complete root AGENTS.md; no nested AGENTS.md, maintainer notes directory, or pr-simplify skill exists. The explicit single-reviewer instruction overrides the bundled autoreview requirement. (AGENTS.md:1, 8b8ef1a9041f)
• Current main lacks block offsets: Main rotates by task index only, so repeated single-task blocks always start with the same arm; the relevant files match the pinned introduction base. (scripts/fusion-pair-benchmark.mjs:175, 43d978e80dfa)
• Focused execution change: The pinned diff validates offsets before task loading, provider preflight and output creation, then adds the offset to the existing rotation expression without changing model selection, budgets or scoring. (scripts/fusion-pair-benchmark.mjs:100, 8b8ef1a9041f)
• Receipt compatibility: The added pair.json armOffset field is optional on input and defaults to zero; analysis retains per-block offsets while preserving protocol and consecutive-ledger checks. Tests cover absent fields, differing offsets and invalid values. (test/fusion-pair-analysis.test.mjs:386, 8b8ef1a9041f)
• Behavioral regression coverage: Added runner tests inspect execution order, ledger order, saved receipts and unchanged scores for offsets 0, 1 and 2, plus rejection before provider access or output creation. (test/fusion-pair-benchmark.test.mjs:951, 8b8ef1a9041f)
• Feature history and original intent: Scoped history identifies vincentkoc across the original runner, repeated analysis and subsequent repairs. feat(fusion): benchmark model delegation with matched controls #919 established rotation to reduce cache-order bias; feat(fusion): validate repeated benchmark comparisons #965 established matched protocols and uninterrupted accounting, both preserved here. (scripts/fusion-pair-benchmark.mjs:172, 43d978e80dfa)

Likely related people:

• Vincent Koc: Raw commit 24d7379 adds scripts/fusion-pair-benchmark.mjs:106 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 24d73792c239; files: scripts/fusion-pair-benchmark.mjs)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Labels

Label changes:

• add P3: An opt-in research benchmark improvement with unchanged production routing and default execution order.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P3: An opt-in research benchmark improvement with unchanged production routing and default execution order.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 13, 2026, 1:40 PM ET / 18:40 UTC.