Lab Hub · agent_hephaestus_robustness_eval
Isolated mini SWE-bench style harness for comparing agent instruction layers.
Does Hephaestus Stormbreaker make coding-agent work more operationally robust than plain Codex and a baseline Hephaestus Network routing rule?
Operational robustness claim:
native-codex < baseline-hephaestus-network < hephaestus-stormbreaker
Hephaestus Stormbreaker is the public name for the upgraded protocol arm. The
old upgraded-hephaestus-protocol string is retained only as a legacy source ID
inside historical result files and compatibility fixtures.
In this repo, the ladder means the evidence-weighted operational robustness score unless a table explicitly says "private oracle pass." Raw private pass remains the correctness guardrail. Local mini-harness results support the ladder on process-aware operational scores while preserving raw private-pass failures. The strict 30-task official SWE-bench Lite pilot does not support public benchmark superiority: native resolved 22/30, baseline 21/30, and Stormbreaker 21/30.
native-codex: no projectAGENTS.md, no skill, no agent routing.baseline-hephaestus-network: isolated routing preflight plus standard public test loop.hephaestus-stormbreaker: Hephaestus Stormbreaker; scope lock, failure-memory check, verifier-first plan, issue-derived regressions, evidence loop, review gate, and final gate.delegated-hephaestus-receipts: isolated route preflight plus required planner, implementer, verifier, and reviewer JSON receipts.
Older result JSON files use the historical arm id
upgraded-hephaestus-protocol; summaries, CSVs, paper, and charts canonicalize
that arm as hephaestus-stormbreaker / Hephaestus Stormbreaker.
All arms run in disposable copies under ignored runs/. The harness never edits
the real Hephaestus repository or Mason's local Hephaestus inventory.
- Fixtures:
benchmarks/mini_swebench_*/base/ - Public verifier:
python3 -m pytest -q - Private oracle verifier:
benchmarks/_oracles/<fixture>.py <trial-dir> - Runner:
python3 scripts/run_isolated_eval.py --all - Summary:
python3 scripts/summarize_results.py results/latest.json - Freeze manifest:
python3 scripts/freeze_eval_manifest.py
The tasks are intentionally small. They measure whether an instruction layer causes the agent to seek stronger evidence before finalizing, not broad model capability. Private oracles are not copied into trial repositories.
External public benchmark status is tracked separately in docs/external-benchmark-bridge.md and paper/data/external_benchmark_readiness.json. The smoke adapter writes paper/data/external_benchmark_smoke_status.json. Tooling preparation is recorded in paper/data/external_benchmark_tooling_status.json. A tau2 no-key execution boundary probe is recorded in paper/data/tau2_mock_execution_probe.json. An official SWE-bench Lite Docker harness smoke is recorded in paper/data/swebench_official_smoke_status.json and paper/swebench-official-smoke.md. The first generated-patch arm smoke is recorded in paper/data/swebench_lite_arm_smoke_status.json and paper/swebench-lite-arm-smoke.md: native, baseline, and Stormbreaker all resolved the same one SWE-bench Lite instance 1/1, so the result is a diagnostic tie rather than a superiority claim. A strict 30-task official SWE-bench Lite pilot is recorded in paper/data/swebench_lite_30task_strict_pilot_status.json, paper/data/swebench_lite_30task_strict_pilot_isolation_audit.json, and paper/swebench-lite-30task-strict-pilot.md. That pilot completed 30/30 instances for all arms with 0 scorer errors and a clean isolation audit, but it scored native 22/30, baseline 21/30, and Stormbreaker 21/30. Baseline and Stormbreaker had the same resolved set. The result is public benchmark evidence, but it rejects the intended ordering at pilot scale and is not a superiority claim.
Live Hephaestus Network readiness is tracked separately in
paper/data/hephaestus_live_network_status.json
and
paper/data/hephaestus_hub_invoke_status.json.
The current status is a Hub-only route smoke: runner/auth/--hub-only
capability exists, local routing was skipped, a route receipt was recorded, and
the Hub returned a public result. A selected public Hub runtime bundle was also
prepared. This is still not a delegated-network benchmark arm because the Hub
bundle did not produce planner/implementer/verifier/reviewer receipts for a
trial.
Primary full repeat-3 run: results/20260615T122440Z.json
| Arm | Public Pass | Private Oracle Pass | pass^3 by Task | Oracle Leak-Free | Median Duration |
|---|---|---|---|---|---|
native-codex |
18/18 | 18/18 | 6/6 | 18/18 | 81.7s |
baseline-hephaestus-network |
18/18 | 18/18 | 6/6 | 18/18 | 81.1s |
hephaestus-stormbreaker |
18/18 | 17/18 | 5/6 | 18/18 | 104.3s |
Interpretation: this run does not prove that Stormbreaker improves raw
correctness on small repair tasks. It does support the operational
robustness ladder on the evidence-weighted score: native 76.1, baseline
91.7, Stormbreaker 97.8. That score combines correctness with route
evidence, integrity cleanliness, and issue-derived regression evidence. It
shows that the benchmark-grade process is now strong enough to catch a
realistic failure that public tests missed: decomposed Unicode accent handling
in mini_swebench_slug_contract.
Post-calibration regression run: results/20260615T135111Z.json
| Arm | Scope | Public Pass | Private Oracle Pass | Note |
|---|---|---|---|---|
| Hephaestus Stormbreaker | slug contract only, repeat 3 | 3/3 | 3/3 | Passed after adding a general edge-contract expansion rule. |
The post-calibration run is a regression check, not independent held-out proof. The paper records both the calibration failure and the follow-up fix.
Stormbreaker stress probe: results/20260615T160012Z.json
| Arm | Private Oracle Pass | Evidence-Weighted Score | Issue-Derived Regression Signal | Median Duration |
|---|---|---|---|---|
native-codex |
3/3 | 73.3 | 1/3 | 108.7s |
baseline-hephaestus-network |
3/3 | 93.3 | 1/3 | 86.8s |
hephaestus-stormbreaker |
3/3 | 100.0 | 3/3 | 98.2s |
The stress probe is repeat 3, but still one local fixture, so it is not a broad performance claim. It is a targeted operational-robustness result: all arms solved the hidden correctness contract, but Stormbreaker produced the full evidence trail the protocol asks for in every repeat.
Independent local holdout runs after protocol calibration:
| Run | Fixture | Private Oracle Pass | Note |
|---|---|---|---|
20260615T140945Z |
env merge | 0/9 | Excluded fairness run; issue wording allowed whitespace-only keys to normalize as ___. |
20260615T142301Z |
env merge | 9/9 | Valid rerun after visible issue and public test were clarified. |
20260615T143445Z |
JSONL aggregator | 9/9 | Fresh valid holdout. |
20260615T150155Z |
env merge | 3/3 | Receipt-enforced delegated arm. |
20260615T145603Z |
JSONL aggregator | 3/3 | Receipt-enforced delegated arm. |
Combined valid holdout aggregate:
| Arm | Private Oracle Pass | pass^3 by Task | Median Duration |
|---|---|---|---|
native-codex |
6/6 | 2/2 | 68.8s |
baseline-hephaestus-network |
6/6 | 2/2 | 70.6s |
hephaestus-stormbreaker |
6/6 | 2/2 | 89.6s |
delegated-hephaestus-receipts |
6/6 | 2/2 | 109.6s |
Evidence-weighted robustness score on the same holdout aggregate:
| Arm | Avg Score |
|---|---|
native-codex |
80.0 |
baseline-hephaestus-network |
91.7 |
hephaestus-stormbreaker |
100.0 |
Average verification across primary, holdout, and stress suites:
| Arm | Macro Evidence Score | Bootstrap 95% CI |
|---|---|---|
native-codex |
76.48 | [73.33, 80.00] |
baseline-hephaestus-network |
92.22 | [91.67, 93.33] |
hephaestus-stormbreaker |
99.26 | [97.78, 100.00] |
Macro pairwise deltas are positive: baseline minus native +15.74
([12.22, 18.89]) and Stormbreaker minus baseline +7.04 ([5.56, 8.33]).
The aggregate gate is therefore:
supported_for_evidence_weighted_operational_robustness_not_raw_private_pass.
Paired task-unit statistical validation:
| Pair | Paired task units | Mean delta | Positive units | Holm sign-flip p |
|---|---|---|---|---|
| baseline minus native | 9 | +15.19 | 9/9 | 0.0059 |
| Stormbreaker minus baseline | 9 | +6.67 | 8/9 | 0.0059 |
| Stormbreaker minus native | 9 | +21.85 | 9/9 | 0.0059 |
This conservative layer averages repeats inside each suite::benchmark unit
before testing. It supports the local operational ladder without pretending
that repeat-3 trials are independent public benchmark tasks.
Metric sensitivity:
| Metric family | Native | Baseline | Stormbreaker | Ladder |
|---|---|---|---|---|
| Raw private pass | 100.00 | 100.00 | 98.15 | rejected |
| Correctness only | 100.00 | 100.00 | 98.52 | rejected |
| Operational variants | 69.72-83.24 | 84.44-96.11 | 98.89-99.44 | 6/6 supported |
The metric-sensitivity gate is:
operational_ladder_supported_across_metric_sensitivity_not_raw_correctness.
Public benchmark power plan:
| Target detectable delta | 80% power | 90% power | Recommended minimum | Total arm trials at repeat 3 |
|---|---|---|---|---|
| 5 pt | 126 | 169 | 169 | 507 |
| 7.5 pt | 56 | 75 | 75 | 225 |
| 10 pt | 32 | 43 | 43 | 129 |
| 15 pt | 14 | 19 | 30 | 90 |
The planning unit is a paired public benchmark task. Repeats support pass^k and stability checks, but are not counted as independent tasks for the headline power calculation.
Reviewer readiness audit:
| Gate | Status |
|---|---|
| Operational paper readiness | true |
| External public claim readiness | false |
| Readiness score | 90.9% |
| Blocked on | 75/169-task public benchmark tier; live delegated Hephaestus benchmark arm |
Strict official SWE-bench Lite 30-task pilot:
| Arm | Completed | Resolved | Errors | Interpretation |
|---|---|---|---|---|
native-codex |
30/30 | 22/30 | 0 | Best pilot score |
baseline-hephaestus-network |
30/30 | 21/30 | 0 | Same resolved set as Stormbreaker |
hephaestus-stormbreaker |
30/30 | 21/30 | 0 | No pilot gain over baseline |
Interpretation: the local operational ladder is still useful as methodology evidence, but the official 30-task public pilot does not show Stormbreaker > baseline > native. The 75-task credible subset or 169-task broad subset is still required before any public benchmark superiority claim.
- Readable HTML paper: paper/index.html
- PDF paper: paper/hephaestus_stormbreaker_paper.pdf
- Markdown paper: paper/hephaestus_stormbreaker_paper.md
- Stormbreaker architecture: docs/stormbreaker-architecture.md
- Claim/evidence matrix: paper/claim-evidence-matrix.md
- Reproducibility checklist: paper/reproducibility-checklist.md
- Reviewer readiness audit: paper/reviewer-readiness-audit.md
- External reviewer packet: paper/external-reviewer-packet.md
- Source verification ledger: paper/source-verification-ledger.md
- Paired statistical validation: paper/paired-statistical-validation.md
- Metric sensitivity analysis: paper/metric-sensitivity-analysis.md
- Public benchmark power plan: paper/public-benchmark-power-plan.md
- External benchmark preregistration: paper/external-benchmark-preregistration.md
- External benchmark bridge: docs/external-benchmark-bridge.md
- External benchmark readiness JSON: paper/data/external_benchmark_readiness.json
- External benchmark tooling JSON: paper/data/external_benchmark_tooling_status.json
- tau2 mock execution probe JSON: paper/data/tau2_mock_execution_probe.json
- tau2 mock execution probe: paper/tau2-mock-execution-probe.md
- External benchmark smoke status JSON: paper/data/external_benchmark_smoke_status.json
- SWE-bench official harness smoke: paper/swebench-official-smoke.md
- SWE-bench official harness smoke JSON: paper/data/swebench_official_smoke_status.json
- SWE-bench arm smoke: paper/swebench-lite-arm-smoke.md
- SWE-bench arm smoke JSON: paper/data/swebench_lite_arm_smoke_status.json
- SWE-bench strict 30-task pilot: paper/swebench-lite-30task-strict-pilot.md
- SWE-bench strict 30-task pilot JSON: paper/data/swebench_lite_30task_strict_pilot_status.json
- SWE-bench strict 30-task isolation audit: paper/data/swebench_lite_30task_strict_pilot_isolation_audit.json
- External benchmark preregistration JSON: paper/data/external_benchmark_preregistration.json
- Live Hephaestus Network status JSON: paper/data/hephaestus_live_network_status.json
- Hub runtime-bundle invoke status JSON: paper/data/hephaestus_hub_invoke_status.json
- Hub delegated-arm bundle probe JSON: paper/data/hephaestus_hub_bundle_probe.json
- Hephaestus delegated-arm gap audit: paper/hephaestus-delegated-arm-gap.md
- Hephaestus delegated-arm gap JSON: paper/data/hephaestus_delegated_arm_gap.json
- Live delegated execution contract: paper/live-delegated-execution-contract.md
- Live delegated execution contract JSON: paper/data/live_delegated_execution_contract.json
- Average verification JSON: paper/data/average_verification.json
- Paired statistical validation JSON: paper/data/paired_statistical_validation.json
- Metric sensitivity JSON: paper/data/metric_sensitivity.json
- Public benchmark power-plan JSON: paper/data/public_benchmark_power_plan.json
- Reviewer readiness audit JSON: paper/data/reviewer_readiness_audit.json
- External reviewer packet JSON: paper/data/external_reviewer_packet.json
- CSV data:
- paper/data/primary_arm_summary.csv
- paper/data/primary_trial_results.csv
- paper/data/calibration_regression_summary.csv
- paper/data/calibration_regression_trials.csv
- paper/data/holdout_arm_summary.csv
- paper/data/holdout_trial_results.csv
- paper/data/average_verification_summary.csv
- paper/data/average_verification_pairwise_deltas.csv
- paper/data/paired_statistical_validation_units.csv
- paper/data/paired_statistical_validation_pairwise.csv
- paper/data/metric_sensitivity_summary.csv
- paper/data/metric_sensitivity_pairwise.csv
- paper/data/public_benchmark_power_plan.csv
- paper/data/public_benchmark_pair_deltas.csv
Regenerate the paper package:
python3 scripts/check_external_benchmark_readiness.py
python3 scripts/prepare_external_benchmark_tooling.py --execute
python3 scripts/run_tau2_mock_execution_probe.py
python3 scripts/run_external_benchmark_smoke.py --benchmark all
PYTHONUNBUFFERED=1 python3 scripts/run_swebench_lite_30task_pilot.py --limit 30 --pilot-tag strict-pilot --no-seed-smoke --score --codex-timeout 1800 --score-timeout 10800
python3 scripts/audit_swebench_pilot_isolation.py --pilot-tag strict-pilot
python3 scripts/check_hephaestus_live_network.py --execute --real-hephaestus-repo <hephaestus-repo>
python3 scripts/analyze_average_verification.py
python3 scripts/analyze_paired_statistics.py
python3 scripts/analyze_metric_sensitivity.py
python3 scripts/plan_public_benchmark_power.py
python3 scripts/preregister_external_benchmark_subset.py
python3 scripts/audit_hephaestus_delegated_arm_gap.py
python3 scripts/audit_live_delegated_execution_contract.py
python3 scripts/build_external_reviewer_packet.py
python3 scripts/audit_reviewer_readiness.py
python3 scripts/render_report_charts.py results/20260615T122440Z.json
python3 scripts/build_paper_package.py
python3 scripts/render_report_charts.py paper/data/holdout_combined.json --prefix holdout
scripts/export_paper_pdf.sh- agent.md: evaluation contract.
- benchmarks/: isolated fixture and arm-specific instructions.
- scripts/run_isolated_eval.py: benchmark runner.
- scripts/freeze_eval_manifest.py: fixture/oracle hash manifest.
- scripts/render_report_charts.py: SVG chart renderer for the paper.
- scripts/build_paper_package.py: CSV, checklist, claim matrix, source ledger, and HTML paper generator.
- scripts/check_external_benchmark_readiness.py: public benchmark prerequisite probe.
- scripts/prepare_external_benchmark_tooling.py: isolated external benchmark toolchain preparation/probe.
- scripts/run_tau2_mock_execution_probe.py: tau2 no-key execution boundary probe that records invalid/no-score attempts.
- scripts/run_external_benchmark_smoke.py: guarded public benchmark smoke adapter/status writer.
- scripts/record_swebench_official_smoke.py: records the official Docker-backed SWE-bench Lite gold smoke boundary.
- scripts/run_swebench_lite_arm_predictions.py: generates arm patches and optionally scores them with the official SWE-bench Lite harness.
- scripts/record_swebench_lite_arm_smoke_results.py: records the public one-task arm-smoke tie boundary.
- scripts/run_swebench_lite_30task_pilot.py: generates and scores a strict 30-task SWE-bench Lite pilot for all arms.
- scripts/audit_swebench_pilot_isolation.py: checks the strict pilot logs for disallowed user skill, memory, config, or web-search access.
- scripts/check_hephaestus_live_network.py: guarded live Hephaestus Hub-only readiness/status writer.
- scripts/analyze_average_verification.py: macro/micro average verification, bootstrap CIs, pairwise deltas, and aggregate score chart.
- scripts/analyze_paired_statistics.py: paired task-unit exact sign/sign-flip tests with Holm correction.
- scripts/analyze_metric_sensitivity.py: raw-correctness boundary and operational-score sensitivity analysis.
- scripts/plan_public_benchmark_power.py: public benchmark task-count and repeat-count power plan.
- scripts/preregister_external_benchmark_subset.py: freezes official public benchmark task/subset choices before execution.
- scripts/audit_hephaestus_delegated_arm_gap.py: separates Hub route/bundle evidence from missing live delegated benchmark-arm evidence.
- scripts/audit_live_delegated_execution_contract.py: defines the exact receipt, isolation, and verifier gates needed for a live delegated-network claim.
- scripts/build_external_reviewer_packet.py: claim-admissibility packet for external review.
- scripts/audit_reviewer_readiness.py: external-review readiness audit that separates operational paper readiness from public benchmark/live-network claims.
- scripts/export_paper_pdf.sh: Chrome/Chromium PDF export for the paper.
- results/: sanitized score summaries.
- paper/hephaestus_stormbreaker_paper.md: English paper-style write-up with charts.
- memory.md: public experiment memory.
- docs/research-log.md: dated research notes.
- docs/benchmark-grade-protocol.md: stricter evaluation protocol.
- docs/external-benchmark-bridge.md: bridge from local harness evidence to public benchmark smokes.