feat(benchmark): verify formally installed native Goal treatments

[huangruiteng]

What changed

• add a benchmark-toolkit helper that creates an isolated LoopX profile through the shipped scripts/install-local.sh path;
• verify that the profile uses a release-snapshot CLI, exact clean source revision, fixed-installer skill readback, and the codex-app-ssh doctor surface;
• render the real thin visible-Goal body through that installed CLI, require the codex_app_ssh_goal profile and interface budget, and bind isolated runs to their explicit runtime registry;
• optionally require the native Goal runtime to discover named skills through the real app-server skills/list call before it creates a thread;
• add a no-model end-to-end smoke and document the three independent treatment inputs: Goal prompt, installed skills, and installed CLI.

Why

A benchmark adapter can otherwise generate a valid Goal prompt and invoke a LoopX CLI while starting Codex with an empty or unrelated CODEX_HOME. That looks like a LoopX treatment at the runner boundary, but the model cannot discover the product's skills. Filesystem existence checks are insufficient because Codex's actual skill catalog is the execution boundary.

This change makes that mismatch fail before turn/start and gives adapters a reusable formal-install and prompt-rendering path instead of hand-copying skill files or supplying an unproven prompt.

Boundaries

The helper grants no model credential, network, evaluator, upload, submission, scoring, or benchmark-launch authority. Receipts omit task text, skill bodies, credentials, and local paths. This PR does not change benchmark scoring or task semantics and does not launch a benchmark job.

Validation

• python3 -m pytest -q benchmark/tests — 23 passed
• python3 -m pytest -q tests/test_skill_delivery_parity.py tests/test_slash_command_install.py tests/test_doctor_install_freshness.py — 63 passed
• Ruff, formatting check, py_compile, and git diff --check passed
• python3 examples/benchmark-native-goal-installed-profile-smoke.py --require-app-server passed from the clean committed head: formal install ready, real 3,274-character Goal prompt bound to the installed CLI and case registry, all required skills discovered, Goal active, no model turn started
• scripts/loopx canary premerge --from-git-diff: 18/18 selected checks passed, including the public/private boundary scan; it reported the expected benchmark_sensitive manual hold, so maintainer review remains explicit before merge

[huangruiteng]

Exact-head self-review and validation

Reviewed exact head c577292ad0fbc9437ede16a5e1231631ffbb9978.

Changed surfaces

• benchmark-toolkit runtime/API: isolated formal installation profile and opt-in app-server skill discovery;
• compatibility facade and package exports;
• durable unit tests plus a no-model formal-install/app-server smoke;
• public benchmark and DeepSWE integration documentation.

Review result

No actionable findings remain. The 409-line profile module is cohesive around one shipped call site: formal installer orchestration, immutable identity/readback checks, and a path-free receipt. It adds no speculative provider layer. The Goal runtime remains backward compatible because skill discovery is opt-in, and discovery fails before thread/model creation.

Checks

• 21 benchmark tests passed.
• 63 installer, skill-delivery, slash-install, and doctor contract tests passed.
• Ruff, formatting, py_compile, and diff hygiene passed.
• Strict clean-head smoke passed on this exact head: formal release install, six required skills discovered through real skills/list, Goal active, no turn/start.
• Risk canary executed all 18 selected checks with zero failures or skips; public/private boundary scan passed.

Holds and coverage rationale

The canary reported one explicit benchmark_sensitive manual hold. It is acknowledged here and covered by exact-head maintainer review. This PR does not change task semantics, scoring, verifier behavior, permissions, submission/leaderboard behavior, or launch a benchmark job. Coverage includes both the fixed-installer contracts and the real Codex discovery boundary, so it is sufficient for this narrow reusable benchmark seam. Remote required checks must still pass before merge.

[huangruiteng]

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确评审头（Exact Head）: 3275@c577292ad0fbc9437ede16a5e1231631ffbb9978
Base: main | 类型: formal native Goal treatment 安装校验（10 files, +1059）
作者: huangruiteng（owner）

---

详细中文评审

动机

benchmark adapter 可能生成合法 Goal prompt 并调用 LoopX CLI，却用空/无关的 CODEX_HOME 启动 Codex——从 runner 边界看像 LoopX treatment，但模型实际发现不了产品 skill；文件存在性检查不足以代表 Codex 的真实 skill 目录（执行边界）。本 PR 让该错配在 turn/start 前失败，并提供可复用的正式安装路径。

改动思路

新增 benchmark_toolkit/native_codex_profile.py：通过 scripts/install-local.sh 创建隔离 LoopX profile，并校验 release-snapshot CLI、exact clean source revision、fixed-installer skill readback、codex-app-ssh doctor surface；native Goal runtime 可选地在建 thread 前调用真实 app-server skills/list 确认命名 skill 可发现；新增 no-model 端到端 smoke 与三个独立 treatment 输入（Goal prompt / installed skills / installed CLI）文档。

具体改动

• loopx/capabilities/benchmark_toolkit/native_codex_profile.py：正式安装 + 四类校验。
• native_codex_goal.py：skills/list 发现门（可选，before thread）。
• benchmark/tests/test_native_codex_profile.py + 既有 native goal 测试扩展；examples/benchmark-native-goal-installed-profile-smoke.py（--require-app-server）。
• 文档：benchmark/deepswe/README 与 toolkit README 明确三项独立 treatment 输入。

关键内容讲解

1. 执行边界即真相：以 app-server skills/list 为准，而不是文件系统存在性；发现失败在 turn/start 前 fail-closed。
2. 正式安装复用：adapter 不再手抄 skill 文件；profile 创建走 install-local.sh 并校验版本/修订/doctor。
3. 边界不扩张：不授予模型凭据/网络/evaluator/上传/提交/评分/benchmark-launch 权；receipt 不含任务文本/skill 正文/凭据/本地路径。

对主干的风险

无阻断项。不改变评分/任务语义/权限/提交行为，不启动 benchmark job。CI：Sign-off、dependency-review、build ×1 SUCCESS；build ×1、pytest IN_PROGRESS（无失败）。canary 的 benchmark_sensitive manual hold 属 owner 决策项。

验证矩阵（exact head 实测/CI）

场景	结果
python -m pytest -q benchmark/tests	21 passed
test_skill_delivery_parity / test_slash_command_install / test_doctor_install_freshness	63 passed
作者 no-model e2e smoke（--require-app-server）	PASS（正式安装就绪、skills 全部发现、Goal active、无模型 turn）
canary premerge（作者）	18/18 selected checks 通过（含 public/private boundary）
Sign-off / dependency-review / build	SUCCESS
pytest / build（CI）	IN_PROGRESS（评审时刻，无失败）

我的整体评价

方向正确：把“treatment 真实性”提升到执行边界（skills/list），并给 adapter 一条可复用的正式安装/校验路径；测试与 smoke 覆盖充分，边界不扩张。结论：APPROVE（author-owned PR，以 COMMENTED 记录批准结论；合并前确认 CI 绿并由 owner 处理 benchmark_sensitive hold）。

---

English Verdict

Verdict: APPROVE (author-owned PR; recorded as COMMENTED because GitHub blocks formal self-approval). The benchmark toolkit now provides a formal isolated-install helper (install-local.sh path) that verifies release-snapshot CLI, exact source revision, installer skill readback, and the codex-app-ssh doctor surface, and the native Goal runtime can fail closed before turn/start when the real app-server skills/list cannot discover the named skills. 84 focused tests pass at the exact head; Sign-off, dependency-review, and one build are green, with pytest/another build in progress. No scoring, task, permission, submission, or job-launch surface changes; the benchmark_sensitive owner hold remains.

[huangruiteng]

Exact-head self-review and validation

This review supersedes the earlier exact-head note after the prompt-binding follow-up. Reviewed exact head 5e6a1240e4aa3dc6eb18be8db452da54b143c7ec.

Changed surfaces

• benchmark-toolkit runtime/API: isolated formal installation profile, installed-CLI Goal prompt rendering, and opt-in app-server skill discovery;
• compatibility facade and package exports;
• durable unit tests plus a no-model formal-install/app-server smoke;
• public benchmark and DeepSWE integration documentation.

Review result

No actionable findings remain. The profile module is cohesive around one shipped call path: formal installer orchestration, installed-CLI prompt rendering, immutable identity/readback checks, and path-free receipts. It adds no speculative provider layer. The Goal runtime remains backward compatible because skill discovery is opt-in, and discovery fails before thread/model creation.

Checks

• 23 benchmark tests passed.
• 63 installer, skill-delivery, slash-install, and doctor contract tests passed.
• Ruff, formatting, py_compile, and diff hygiene passed.
• Strict clean-head smoke passed on this exact head: formal release install, real 3,274-character Goal prompt bound to the installed CLI and explicit case registry, six required skills discovered through real skills/list, Goal active, no turn/start.
• Risk canary executed all 18 selected checks with zero failures or skips; public/private boundary scan passed.

Holds and coverage rationale

The canary reported one explicit benchmark_sensitive manual hold. It is acknowledged here and covered by exact-head maintainer review. This PR does not change task semantics, scoring, verifier behavior, permissions, submission/leaderboard behavior, or launch a benchmark job. Coverage includes both the fixed-installer contracts and the real Codex discovery boundary, so it is sufficient for this narrow reusable benchmark seam. Remote required checks must still pass before merge.

[huangruiteng]

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确评审头（Exact Head）: 3275@5e6a1240e4aa3dc6eb18be8db452da54b143c7ec
Base: main | 类型: native Goal treatment 安装校验扩展（8 files, +422/-38）
作者: huangruiteng（owner）

---

详细中文评审

动机

在上一已批准头之上，作者把 Goal prompt 与已安装 profile 绑定：不仅校验安装/技能发现，还让生成的 Goal prompt 直接由 profile CLI（heartbeat-prompt）在 profile 环境（CODEX_HOME/PATH/HOME）中渲染，确保 treatment 的 CLI 与 skill 目录与 runner 实际启动环境一致。

改动思路

native_codex_profile.py 新增 NativeCodexGoalPrompt、native_codex_profile_environment、render_native_codex_goal_prompt（versioned loopx_native_codex_goal_prompt_v0）：用 profile 的 CLI + 环境渲染真实 visible-Goal body，校验 project 目录/ids/timeout 并 fail-closed；研究 facade 同步导出；smoke 扩展覆盖完整安装路径。

具体改动

• native_codex_profile.py（+189）：prompt 渲染 + profile 环境构造 + 校验。
• benchmark/native_codex_goal.py / __init__.py：facade 导出。
• benchmark/tests/test_native_codex_profile.py（+89）：新增 prompt 绑定测试。
• examples/benchmark-native-goal-installed-profile-smoke.py（+124/-…）：完整安装路径 smoke。
• 文档：deepswe/toolkit README 更新完整 installed Goal 路径说明。

关键内容讲解

1. 环境即真相：渲染用 CODEX_HOME/PATH/HOME 来自安装的 profile，而非当前 shell 环境；heartbeat-prompt 从 profile CLI 调用。
2. fail-closed：project 非目录、ids 空、timeout 非正均抛错。
3. 版本化：prompt schema loopx_native_codex_goal_prompt_v0，receipt 保持 compact/public-safe。

对主干的风险

无阻断项。不改变评分/任务/权限/提交行为，不启动 benchmark job；receipt 不含任务文本/skill 正文/凭据/本地路径。CI：Sign-off、dependency-review SUCCESS；build ×2、pytest IN_PROGRESS（无失败）。

验证矩阵（exact head 实测/CI）

场景	结果
python -m pytest -q benchmark/tests	23 passed
test_skill_delivery_parity / test_slash_command_install / test_doctor_install_freshness	63 passed
作者 no-model e2e smoke	PASS（完整安装路径）
Sign-off / dependency-review	SUCCESS
build ×2 / pytest（CI）	IN_PROGRESS（评审时刻，无失败）

我的整体评价

扩展方向一致且闭环：prompt、CLI、skill 目录全部绑定已安装 profile 环境，fail-closed 校验与测试覆盖到位。结论：APPROVE（author-owned PR，以 COMMENTED 记录批准结论；合并前确认 CI 绿并由 owner 处理 benchmark_sensitive hold）。

---

English Verdict

Verdict: APPROVE (author-owned PR; recorded as COMMENTED because GitHub blocks formal self-approval). The extension binds Goal prompt rendering to the installed profile environment (CODEX_HOME/PATH/HOME, profile CLI), with versioned schema and fail-closed validation; 86 focused tests pass at the exact head and the no-model end-to-end smoke covers the complete installed path. Sign-off and dependency-review are green, CI build/pytest were in progress with no failures, and the benchmark_sensitive owner hold remains.