feat(benchmark): connect native Codex Goal to the real runtime

[huangruiteng]

What changed

• Move the native Codex Goal transaction into the installed benchmark toolkit.
• Add a real stdio JSON-RPC transport for codex app-server, including Goal attachment, terminal-event correlation, post-turn readback, process cleanup, and public-safe receipts.
• Keep the research import as a thin compatibility facade and add a runnable DeepSWE example with a no-model preflight mode.
• Cover the pure transaction, real subprocess wiring, event-id reconciliation, compact evidence, and the runnable example.

Why

The existing research helper defined only an in-process transport protocol. A benchmark adapter still had to implement app-server process wiring and could easily grow a second copy of the Goal state machine. This change makes the installed runtime the single implementation: adapters retain their runner-specific isolation and environment bridge while importing the Goal transaction.

Impact and boundaries

This is a benchmark helper/runtime seam. It does not change benchmark scoring, task semantics, verifier behavior, permissions, submission behavior, or launch any benchmark job. The receipt continues to exclude raw objectives, task instructions, assistant messages, tool events, credentials, and local paths.

Validation

• python3 -m pytest -q benchmark/tests/test_native_codex_goal.py tests/capabilities/test_benchmark_toolkit.py — 23 passed.
• Focused Ruff lint and format checks — passed.
• Live codex-cli 0.144.1 app-server preflight in an isolated temporary Codex home — initialize, thread creation, Goal set/get passed without starting a model turn.
• Downstream adapter import/parity suite — 11 passed; its runner-owned isolation and bridge behavior remain in place.
• loopx canary premerge --from-git-diff — diff, compile, 9 catalog canaries, 8 risk-profile smokes, and public-boundary scan all passed. The generic benchmark_sensitive manual hold remains; owner authorization covers this helper-only seam, and no score, task, verifier, permission, or job-launch surface changed.

[huangruiteng]

Pre-merge validation receipt:

• Changed surfaces: installed benchmark helper runtime, native Goal stdio transport, thin research facade, runnable DeepSWE example, focused tests, and public docs.
• Focused checks: 23 public tests passed; Ruff lint/format passed; a live Codex app-server Goal-attachment preflight passed without a model turn; 11 downstream import/parity tests passed.
• Risk gate: loopx canary premerge --from-git-diff passed diff hygiene, Python compile, 9/9 catalog canaries, 8/8 risk-profile smokes, and 1/1 public-boundary scan.
• Failures/skips: none.
• Manual hold: benchmark_sensitive, because benchmark paths changed. Owner authorization permits self-merge for this helper/runtime seam. Coverage is sufficient because the change does not alter scoring, task or verifier semantics, permission boundaries, submission behavior, or launch a benchmark job; a real consumer now imports the same runtime and retains its runner-owned isolation and bridge tests.

[huangruiteng]

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确评审头（Exact Head）: 3271@5a84e0c149c1a245f74f97c5c584a12df6b52c8b
Base: main | 类型: native Codex Goal 接入真实运行时（8 files, +~1400）
作者: huangruiteng（owner）

---

详细中文评审

动机

上一轮 benchmark reset 后，研究助手只有 in-process 传输协议，benchmark adapter 仍需自己实现 app-server 进程接线，容易复制第二份 Goal 状态机。本 PR 把 native Codex Goal 事务搬进已安装的 benchmark_toolkit，并提供真实 stdio JSON-RPC 传输与可运行 DeepSWE 示例。

改动思路

单一实现收口：loopx/capabilities/benchmark_toolkit/native_codex_goal.py 成为唯一 Goal 事务实现（initialize → thread/start → goal/set → goal/get → turn/start，终端事件关联、post-turn readback、进程清理、compact receipt）；benchmark/native_codex_goal.py 降级为兼容 import facade，防止研究树再长第二份实现；adapter 保留 runner 专属隔离/环境桥，只 import 已安装运行时。

具体改动

• 安装面：benchmark_toolkit/native_codex_goal.py（真实 stdio JSON-RPC 传输 + 进程生命周期 + 事件归因 + 公共安全 receipt）。
• 研究面：benchmark/native_codex_goal.py 改 facade；新增 deepswe/run_native_codex_goal.py（--preflight-only 无模型预检 + 完整 turn 示例，objective/task 走文件避免命令历史泄漏）。
• 文档：benchmark/deepswe README 与 toolkit README 更新接入契约与 authority 边界。
• 测试：benchmark/tests/test_native_codex_goal.py（纯事务、真实 subprocess wiring、event-id 对账、compact evidence、可运行示例）。

关键内容讲解

1. 单实现原则：研究 facade 只是 re-export，运行时代码只能 import toolkit；杜绝双状态机漂移。
2. 职责边界：已安装运行时不拥有 evaluator 访问、任务命令桥、凭据策略或评分权；这些仍是 runner 职责。
3. 公共安全：receipt 不含原始 objective/task/assistant 消息/tool 事件/凭据/本地路径；示例用文件输入避免命令历史泄露。

对主干的风险

无阻断项。benchmark helper/runtime seam：不改变评分、任务语义、verifier、权限、提交行为，也不启动 benchmark job。CI：Sign-off、dependency-review SUCCESS；build ×2、pytest IN_PROGRESS（无失败）。canary 的 benchmark_sensitive manual hold 属 owner 授权项（PR 自述已覆盖该 helper-only seam）。

验证矩阵（exact head 实测）

场景	结果
pytest benchmark/tests/test_native_codex_goal.py tests/capabilities/test_benchmark_toolkit.py	23 passed
parity/import 相关 4 个套件	70 passed
作者 live app-server preflight（codex-cli 0.144.1，隔离 Codex home）	PASS（initialize/thread/Goal 无模型 turn）
canary premerge（作者）	diff/compile/catalog 9/9/risk 8/8/boundary 全过
CI：Sign-off / dependency-review	SUCCESS
build ×2 / pytest（CI）	IN_PROGRESS（评审时刻）

我的整体评价

方向正确：把“runner 专属”与“控制面可复用”的边界再推进一步——Goal 事务成为已安装单实现，研究树只剩薄 facade，真实传输与可运行示例补齐了接线证据。结论：APPROVE（author-owned PR，以 COMMENTED 记录批准结论；合并前确认 CI 绿并由 owner 处理 benchmark_sensitive hold）。

---

English Verdict

Verdict: APPROVE (author-owned PR; recorded as COMMENTED because GitHub blocks formal self-approval). The change makes the installed benchmark_toolkit.native_codex_goal the single Goal-transaction implementation (real stdio JSON-RPC transport, process lifecycle, event correlation, post-turn readback, compact public-safe receipts), turns the research-tree module into a thin compatibility facade, and adds a runnable DeepSWE example with a no-model preflight. 23 native/toolkit tests and 70 parity tests pass at the exact head; Sign-off and dependency-review are green, CI build/pytest were in progress, and no scoring/task/verifier/permission/submission surface or benchmark job is affected.