Skip to content

benchmarks.md

Latest commit

 

History

History
280 lines (215 loc) · 13.7 KB

benchmarks.md

File metadata and controls

280 lines (215 loc) · 13.7 KB
Raw

End-to-end benchmark

The benchmark compares actual Codex runs from independent historical workspaces. Hidden executable tests and a changed-path boundary decide correctness. The capture path then reports provider usage, transport savings, tools, and Hpatch delivery from observed traffic rather than production counters or hypothetical baselines.

Requirements

The runner checks its host dependencies. You need Docker Compose, Codex authentication at $CODEX_AUTH_PATH or $CODEX_HOME/auth.json, and the task's local source repository under benchmarks/repos/.

Run it

Default A/B preset: one attempt per arm, gpt-6-astra at medium effort. A is stock passthrough/native; B is Hpatch + CTP/2. Mentor Handoff and issue reporting are disabled in both arms. The default task is etcd-range-stream.

Default paired run:

bash benchmarks/bench.sh

For a matching Sol A/B pair, run MODEL=gpt-5.6-sol bash benchmarks/bench.sh.

One stock control attempt, with no Hpatch router or treatment attempt:

MODEL=gpt-5.6-sol BENCHMARK_MODE=control-only bash benchmarks/bench.sh

control-only requires one repetition and issue reporting disabled. Its report validates only stock capture and usage; it contains no A/B delta or Hpatch delivery section.

For local preparation and hidden-grader qualification without any model calls:

MODEL=gpt-5.6-sol BENCHMARK_MODE=control-only BENCHMARK_PREPARE_ONLY=true \
  bash benchmarks/bench.sh

Preparation builds the image, prepares dependencies, and proves that the historical base fails and oracle passes the hidden grader. It does not run an agent. The runner prints phase elapsed times separately from agent wall time; a later measured invocation repeats preparation in its own isolated workspace and qualifies router/network isolation before invoking Codex.

One Hpatch attempt against a matching published control:

MODEL=gpt-5.6-sol BENCHMARK_MODE=hpatch-only REPETITIONS=1 bash benchmarks/bench.sh

For another model, also set CONTROL_BASELINE_DIR to a published control with matching model, effort, task-content fingerprint, and instruction evidence. The fingerprint covers the manifest, visible task prompt, and hidden graders. Results without it, or from a different task contract, cannot be imported; collect a fresh control instead. Keep task files unchanged during a run: content checks before agent launch and around grading fail a run if those files change.

One diagnostic run without a control:

BENCHMARK_MODE=hpatch-diagnostic REPETITIONS=1 bash benchmarks/bench.sh

Native Hpatch versus CTP/2:

TASK_ID=batch-diagnostic-collapse BENCHMARK_MODE=ctp-only \
  BENCHMARK_REPORT_ISSUES=false REPETITIONS=4 bash benchmarks/bench.sh

Hpatch versus Mentor Handoff:

MODEL=gpt-5.6-luna REASONING_EFFORT=xhigh REPETITIONS=2 \
  BENCHMARK_MODE=mentor-handoff BENCHMARK_REPORT_ISSUES=false \
  bash benchmarks/bench.sh

To keep Astra as the main agent and enable CTP/2 alongside Mentor Handoff:

MENTOR_PARENT_MODEL=gpt-6-astra MENTOR_MODEL_PROTOCOL=ctp2 \
  MODEL=gpt-5.6-luna REPETITIONS=2 BENCHMARK_MODE=mentor-handoff \
  BENCHMARK_REPORT_ISSUES=false bash benchmarks/bench.sh

In Mentor mode, MODEL selects the requested child model (Luna or Terra), while MENTOR_PARENT_MODEL selects the main agent (default Sol, high effort). The router's default Sol/high-to-child handoff schedule is unchanged by the main model. MENTOR_MODEL_PROTOCOL selects native (default) or ctp2 for both arms; only the treatment enables Mentor Handoff. Two repetitions produce four graded attempts. The summary names the main model separately from the child and mentor models, and checks required CTP/2 compression in both CTP/2 arms.

Exhaustive commentary coverage across diagnostic, native/CTP, and Mentor Handoff arms:

bash benchmarks/run-commentary-coverage.sh

The commentary suite runs one repetition in each mode and continues through all three modes so one failed arm does not hide later evidence. It intentionally permits repeated edits because stale-target recovery is part of the task. MODEL may select gpt-5.6-luna or gpt-5.6-terra, and REASONING_EFFORT overrides the default medium effort.

Coverage includes Hpatch apply and recovery, optional issue reporting, Bash and POSIX runtime publications, provider-owned exec invocation, Code Mode runtime publication, subagent start and response projection, and terminal token telemetry. Runtime publications and exec invocation are proven by successful command markers because Codex JSONL does not reliably retain their user-only messages. Host-only continuation events are outside this retained event boundary.

MODEL, REASONING_EFFORT, and paired-mode REPETITIONS override the defaults. CTP/2 and Mentor Handoff disable issue reporting so the reporting tool does not confound either treatment.

One-listener topology

Each measured arm runs one hpatch-router process with one listener:

Codex ──HTTP──> hpatch-router ──HTTP──> provider
                 │         │
                 └─ in-process capturer

The capturer wraps the existing Responses handler and provider transport. It is a Go subpackage, not a service. It opens no port. The router's same listener exposes:

POST /v1/responses
GET  /v1/models
GET  /api/metrics
GET  /                 # human-readable view of /api/metrics

The Compose file therefore contains control and hpatch router services but no front or back capturer services. Each agent joins only its router's internal network. Each router also joins the egress network and talks directly to the provider. The runner supplies --capture-output so each router appends its own sanitized evidence file.

What differs between arms

Paired control uses passthrough mode and the pinned stock instructions. The paired treatment enables CTP/2 as well as Hpatch. Hpatch mode replaces the supported Code Mode editing owner with Hpatch and shell while preserving unrelated tools. Each arm gets a separate workspace and alternates execution order across repetitions.

CTP-only uses Hpatch in both arms; only the model protocol and owning guidance differ. Mentor Handoff uses Hpatch in both arms; only the treatment router enables its bounded subagent model schedule. Parent and child traffic remains visible through actual model names in capture exchanges.

Capture and metrics

capturer records schema-5 JSONL at both boundaries without storing credentials, prompts, instructions, tool arguments, command output, response text, diagnostics, scripts, reports, or patches. Records contain sizes, token estimates, status, duration, provider usage, tool identities, and sanitized delivery kinds or diagnostic codes. Correlation is process-private Go context; no private header crosses the network.

Each response boundary retains at most 8 MiB for parsing while the complete stream remains forwarded and byte-counted. Overflow is incomplete evidence. Diagnostic capture accepts only stable allowlisted reason codes from a complete router-owned envelope; arbitrary text(...) content is discarded.

GET /api/metrics returns hpatch.capture.metrics.v3. Schema-5 capture records and metrics v3 exclude router-generated commentary from model-origin output, while transport still includes it. Streamed output is rebuilt from finalized items when the terminal array is empty, absent, null, or contains only generated commentary. Genuine model commentary is retained, including text that resembles token telemetry. Payload estimates are not billed output-token counts.

Older records cannot be repaired from retained counters, since raw output items are not saved. Their functional grading and provider usage remain historical evidence, but current comparison validation rejects their output-accounting version. Fresh captures are required.

It is authoritative for:

  • logical requests and provider retry attempts;
  • provider input, cached input, uncached input, output, and reasoning tokens;
  • cold/new and eligible-prefix cache attribution between logical requests from each final attempt;
  • client and provider payload bytes and GPT-5 token estimates;
  • signed protocol input and output savings;
  • provider-emitted and client-delivered tool shapes;
  • correlated Hpatch calls, corrections, deliveries, rejections, diagnostics, and carrier savings;
  • actual provider model for every attempt, including attempts without usage; and
  • capture completeness, dropped-detail, and write health.

Cumulative totals cover the router lifetime. Detailed exchanges retain the latest 4,096 requests; if that window fills, totals remain complete but dropped-detail health invalidates benchmark use.

The report validator requires both fresh arms in paired, CTP/2, and Mentor modes. It reconciles raw records, exchanges, aggregate usage, capture health, each measured root thread's provider usage and configured model, and Mentor child lineage and model schedules. It rejects partial or unproved evidence. Configured CTP/2 compression requirements use signed snapshot savings and retain a failed value in the summary before the run exits nonzero. The report formats snapshot values rather than calculating another notion of gain. This means retries remain retries, negative expansion stays visible, and Hpatch is compared with the native carrier actually delivered to Codex.

Artifacts

A retained run includes:

results.jsonl
summary.md
benchmark-config.json
control-metrics.json                 # when a fresh baseline arm ran
hpatch-metrics.json
captures/control.jsonl               # when a fresh baseline arm ran
captures/hpatch.jsonl
control-router.log                   # when applicable
hpatch-router.log
artifacts/                            # per-attempt result, events, patch, and grader evidence
agent-issue-reports.jsonl             # when issue reporting collected records

An opt-in commentary task also writes commentary-coverage.json beside each attempt's result.json. Functional correctness remains in the hidden grader record; commentary coverage is a separate result field derived from retained assistant messages, successful command markers, and completed item types in Codex events.

Mentor Handoff renames the treatment snapshot and log to hpatch-mentor-metrics.json and hpatch-mentor-router.log. Child event and content-free lineage proof artifacts remain under the attempt directory. Summary output intentionally omits request, session, thread, call, and capture identities.

Fixed local CTP replay

Codec replay measures compression of reconstructed requests from real Codex sessions. It does not call a model or measure task success, provider billing, response quality, or task completion time. Use the paired task modes above for those end-to-end questions.

Freeze a private sample once, then reuse it for comparisons. The destination must be a new directory outside the repository. It contains raw conversations, including any sensitive text they contain: do not commit, upload, or share it. Set corpus_root to an absolute private path:

corpus_root=/absolute/private/ctp-replay
HPATCH_CTP_REPLAY_FREEZE="$corpus_root" \
  go test ./internal/router -run '^TestFreezeCTPReplayCorpus$' -count=1 -v

HPATCH_CTP_REPLAY_MANIFEST="$corpus_root/manifest.json" \
  go test ./internal/router -run '^$' -bench '^BenchmarkCTPCorpusReplay$' -benchtime=1x -count=1 -v

Freezing scans sessions and archived_sessions under $CODEX_HOME, or ~/.codex when unset. Eligibility requires a completed session with stock editing and execution guidance, no Hpatch guidance, and a custom exec tool call. The active CODEX_THREAD_ID is excluded. When several eligible rollouts share a logical session identity, a seeded hash selects one rollout for that session. Logical session identities are then ranked by a seeded hash; up to 50 are copied without altering their bytes. Neither selection uses token counts or compression results. All eligible logical sessions are included when fewer than 50 exist, with one rollout per session.

The private manifest records selection criteria, seed, eligible session and rollout counts, ordered session and rollout identities, byte lengths, and content hashes. Replay reports the manifest hash and verifies the frozen files before using them. Missing or changed files fail rather than silently replacing the sample from live history. Preserve the corpus and record the code revision alongside results; use a different new destination when deliberately selecting another sample. Without an explicit manifest, replay benchmarks skip.

This is a sample of eligible local stock-execution sessions, not all Codex workloads. Hash-based selection avoids choosing sessions because they compress well, but does not remove that coverage limit. Report both the sample definition and manifest identity with compression measurements. Synthetic test inputs cover replay mechanics only and are not efficiency evidence.

Validate reporting

bash benchmarks/commentary_coverage_test.sh
bash benchmarks/expected_final_response_test.sh
bash benchmarks/report_test.sh
bash benchmarks/control_only_test.sh
bash benchmarks/task_contract_test.sh

The commentary fixture covers profile selection, operation and collaboration messages, successful command markers, event minimums, missing evidence, malformed event streams, and unsupported modes. The expected-response fixture proves router token telemetry remains auxiliary while later ordinary assistant text remains authoritative. The reporting fixture covers every report mode and falsifies capture health, aggregate usage, baseline presence and schema, configured provider models, Mentor lineage, and required CTP compression. Go tests under capturer/ prove retry correlation, privacy, streaming, gzip, multiline SSE, and bounded detail with complete cumulative totals.