Repository navigation
chore(benchmark): archive FrontierCS-2 single-host compatibility sources - #7954
Conversation
Signed-off-by: song <liusongstep@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max
没有发现阻塞该归档切片的缺陷。APPROVE;完整 premerge 的既有失败和 benchmark-sensitive merge hold 仍保留。
动机 / Motivation
检查历史实验的维护者,需要看到当时实际使用的客户端、桥接与裁判包装源码,才能理解已经发布的观察来自什么执行边界。 此前这五个文件只在实验工作区和独立发布中,仓库读者缺少固定版本、依赖说明与可运行的身份检查。 本 PR 把原始源码作为不可变历史快照保留,并提供只读校验;本轮验证了五份文件与固定 HF 发布的逐字节一致性和异常文件拒绝。 本次只解决来源可检查性,不交付当前 runner、可移植沙箱、正式 Driver/Management 部署或实验收益资格。
改动思路 / Approach
复用既有 benchmark archive placement,不增加 capability、产品导入或活跃 CI benchmark。当前边界合理:冻结树只保存原始字节,说明与校验器放在外部,后续运行修复另立版本。这样既保留历史失败,又避免为当前产品长期维护第二套 runner。不保存源码会使历史检查继续依赖外部工作区;把它直接接入产品则会引入已知 controller、token 恢复及沙箱边界问题。
因此选择保留这个显式历史兼容边界,而不是重构冻结源码。只读检查器是局部 helper,负责文件集合、内容身份、执行位和语法;它不拥有评分、运行、权限或发布决策。哈希一致只证明与已发布快照一致,不证明实验当时的运行、分数或效果可信。需要重跑的人仍必须在新的隔离工作区准备外部 controller、数据与授权,不能把归档检查当成执行许可。
具体改动 / Changes
固定 head f3aee4586d87b678581a22d23b61fba7e49d4672,实际 merge base 71fe25b809fb9ea1ef72a02f5be3c1b833ff1b73。完整变更 9 个文件、+1085/-0:五个冻结文件、77 行检查器、外部版本说明、README 登记和仅针对冻结树的 LF 属性。loopx/、apps/、packages/、工作流及打包配置未改;在产品、测试、CI 中没有归档导入或自动执行引用。
关键代码讲解
check_archive对固定五文件集合检查 hash、非执行模式和语法;拒绝文件级 symlink、额外源码和缺失源码。它只 compile 四个 Python 模块和命名 bootstrap,不 import 或执行归档程序。main将错误作为 JSON/exit 1 返回,成功也明确保留 benchmark_executed=false、standalone_runnable=false、runtime_validation=not_performed。调用者能区分身份检查和运行验证。Trial.request与NNP_BOOTSTRAP属于被冻结的历史实现:bridge 的内存 token/idempotency 不能靠 persist 恢复,bootstrap 依赖 Linux/Docker 的特定安全状态。版本说明没有把这些未提供的 controller 边界冒充为当前可用能力。
规格 spec_ref benchmark/README.md#archive-placement,spec_revision 71fe25b809fb9ea1ef72a02f5be3c1b833ff1b73,按变更前规范的四项条件映射:条件 1 public-safe 源码 implemented,本轮逐文件审查及边界扫描通过;条件 2 外部 provenance、依赖与已知缺陷说明 implemented;条件 3 内容和模式只读检查 implemented,有独立 mutation 对照;条件 4 惰性归档与另版本修复 implemented,无产品/活跃 CI 消费者。规范明确排除当前运行资格,不另加整场实验或父级 roadmap 验收。
对主干的风险 / Risks
独立下载 HF 固定版本的 protocol_snapshot 后,五份文件的全部 42085 字节与当前 Git 文件一致,Git 模式均为 100644。未扩称已复核外部发布中的全部 148 文件、评分、处理忠实度或原始运行。外部发布与 Git 相同仍不是独立历史执行证明。
| 本轮实际检查 | 结果 |
|---|---|
| 五文件身份、四模块语法、一个嵌入 bootstrap | PASS;未执行归档模块 |
| 改字节、额外/缺失源码、文件 symlink、执行位变化、非字节码 cache | PASS:六类 mutation 均 exit 1 |
| 原始副本、纯 pyc cache、显式 root locator symlink | PASS;root locator 可由调用者选择,内部文件约束仍执行 |
| 每次检查前后文件内容对照 | PASS:校验器不写源码 |
| base/head capability list 与 issue-fix help | PASS:完整输出逐字节一致 |
| Ruff、diff/编译/双语 RFC/public-boundary、完整 premerge 的其他选择项 | PASS;full gate 仍 FAIL,见下文 |
standard premerge 的 5 个 direct 检查通过,19 项选中检查中 issue-fix-pr-lifecycle-smoke.py 在读取 gate_item["evidence"] 时失败,benchmark-sensitive hold 另存。相同命令在固定 base/head 独立重跑,完整 stderr 仅替换 checkout 前缀后 511 字节完全相同;整个产品源码与该失败路径无差异,归档身份、拒绝路径和产品入口另有独立通过证据。它是本轮验证的 pre_existing_unrelated,仍为 failed,不能称完整 gate 全绿,也不要求本 PR 修复无关产品代码。
没有加载归档为新 runner,因而没有新增自动启用、quota、Todo、凭据或 actor 生命周期语义。冻结协议值属于历史外部兼容输入,未注册为通用 LoopX 义务。最强残余风险是读者把可检查源码误认作可复现实验;版本说明保留 integrity=false、fidelity pending、countability=false 和 controller 缺失,检查输出也明确不证明 runtime。后续修复须新版本和独立真实路径验证。
我的整体评价 / Verdict
APPROVE,problem_context 为 goal_achieved,仅指这次不可变归档的可检查性;long_horizon 和 user_experience 均 preserved,当前工作路径没有被新门禁或人工步骤阻断。 架构选择 retain:既有归档规范已给出精确兼容边界,冻结树内重构会破坏本次 provenance;bounded refactor 在该树中 unnecessary,新的只读校验保持局部职责。 已检查的是 CLI、Git 字节/模式、目录及文档,没有渲染界面变更,visual_review 不适用。 原生重跑、模型执行、controller 安全和历史效果资格未验证,也不属于本次接受范围。
本次批准不撤销失败观察,不提供合并或 benchmark 执行权限。 精确范围 CQR 也保留 required premerge failure,decision 为 fail;未把它改成 optional 来通过 strict receipt。完整 premerge 与 benchmark-sensitive 人工 hold 继续阻止把这条 review 当作 ready-to-merge 证明;后续由维护者处理现有门禁,不改写历史源码或实验结果。
English verdict: APPROVE — the five-file immutable archive matches the pinned published snapshot, rejects identity/mode mutations, and stays outside product execution. The reproduced pre-existing premerge failure and benchmark merge hold remain separate and unresolved.
The native client, local HTTP bridge, Docker transport and verifier wrapper behind a terminal
batch_local_countingpilot were retained only in the experiment workspace. This PR preserves their exact five-file source snapshot and provides a read-only identity check, dependencies and known limitations so the published artifacts can be inspected.Scope and provenance
benchmark/frontiercs2-single-host-v1/under the existing immutable archive policy. The four Python files and original image manifest keep their original bytes and non-executable Git modes; LF checkout attributes preserve those identities.The separate HF release contains 63 train evaluations, four final evaluations, 63 distinct arm/archive pairs, all 112 purpose-scoped stored-payload records and 686 numeric usage observations. All 148 files were downloaded at revision
7adf7b7e6828895dee4393506af80afecc843d37and matched against local SHA-256 digests; the downloaded relationship validator also passed. Task/patch attribution and licenses are included there. Raw experiment records and candidate outputs are kept out of this Git PR.The evaluated LoopX revision is
e9eda527afd5fcb262ee7c199e1822b1a936cc24; the task export pinsd247b3f15b73805e455ff93c6d043d84603ec341. This single-host, one-task/one-run-per-arm pilot used a five-hour cap and earlier final selections. The original board's integrity and score-countability flags remain false, and LoopX treatment fidelity remains pending. Publication does not qualify a matched benefit, official Driver/Management deployment or leaderboard result.Validation
issue-fix-pr-lifecycle-smoke.pyraisesKeyError: evidence. The same command on immutable main baseline71fe25b809fb9ea1ef72a02f5be3c1b833ff1b73fails at the same line with byte-identical stderr after replacing only the checkout prefix. The maintainability diagnostic also reproduces with identical output on that baseline; runtime, application, package and original smoke source bytes are unchanged. The full gate remains failed, with the separate benchmark-sensitive maintainer hold retained.loopx/; it therefore does not cover thisbenchmark/archive. The inherited Frontier client schema/status vocabulary remains local to the frozen compatibility source, with no new LoopX vocabulary registration. The standard full-tree semantic check passed.Bounded refactoring is unnecessary inside an immutable historical snapshot: changing the original adapters would invalidate provenance. Later execution repairs require a separately versioned implementation and real-path qualification. No graphical UI changes or model/scorer rerun is part of this PR. Maintainer review and merge authority remain separate.