Repository navigation
chore(benchmark): compose terminal reconciliation and validation fixes - #8237
huangruiteng wants to merge 11 commits into
Conversation
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com> Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…gebench-r35-signed-candidate-20261107 Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…gebench-r35-signed-candidate-20261107 Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…gebench-r35-signed-candidate-20261107 Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max
[P1] 修复集成提交里同次试验的回执恢复。 r35 原样继承 #8236 的兜底写入。最终回执在替换前失败,或已替换成功但报告失败时,兜底都会移除 managed_run 与求解器摘要。两次隔离复现中,provider 终态与评分排空证据均已落盘,重放仍返回 2、ready=false,原槽位持续占用。请在原子回执 owner 保留有效的已提交快照,并使替换前失败能从同次尝试的保留证据恢复。 兜底写入 · 源 PR 评审
动机 / Motivation
管理 EdgeBench 并发批次的人,需要在试验结束后确认结果并回收原名额,也需要用现有验证流程检查这份集成提交。
r34 需要分别确认看板和槽位。r35 将二者接到同一次试验的证据上,且修复了引导输出和首次连接验证。保存终态回执时若出错,原槽位仍可能卡住。
当前集成提交通过正常的终态读回和原槽位释放。最终回执在替换前或替换后报错时,重放都无法恢复绑定。
验证覆盖原生 CLI、隔离的 provider 证据与本地账本。真实模型求解、评分和实验净收益仍由独立试验验收。
改动思路 / Approach
分开保留运行器证据和控制器的有序写入。引导压缩、Yardstick 类型及首次连接验证可分别用现有 owner 核对。
本集成 head 仍需修复原子回执的重放边界,再用同一次尝试验证释放。保留已核验的三项验证改动。
候选相对 r34 包含八个带 DCO 的提交。与 #8236 的完整 benchmark diff 一致,另有七条路径承载引导预算、Yardstick 类型和首次连接 smoke 的修复。求解结束、评分排空、看板确认和槽位释放继续各自读回。 终态计划
具体改动 / Changes
依据改动前的 loopx/capabilities/benchmark_toolkit/README.md,版本 c4536ec439fa7480bbfc76d7e26a96ac956fbf2f。runtime-continuity 的同次尝试恢复仍有缺口。concurrency-envelope 的看板先写入、读回再释放原槽位,健康路径通过。score-countability 沿用现有评分边界。 既有工具契约
关键代码讲解
run.main调用retain_managed_solver保存最终回执。错误处理会进入普通回执写入,替换已经提交或仍有恢复价值的绑定。 运行入口publish_managed_terminal用绑定、求解器摘要和评分排空记录生成证明。reconcile_benchmark_terminal在锁内先读回看板终态,再释放并读回原槽位。 证明生成 · 终态控制器render_start_goal_guided_markdown移除重复的命令标签,保留命令、顺序、目的和原有读回要求。Yardstick 只补现有参数与返回类型。 引导输出- 首次连接的
run_cli用已有测试 reader 读取同一次捕获的决策,继续执行原断言并清理测试运行时。新增维护说明也将该做法限定在测试 reader。 原生策略读回 · 测试维护说明
完整 diff 为 21 个文件,新增 1,497 行、删除 69 行。终态证据与有序写入沿用 #8236,验证改动有独立的 caller 和反例。
对主干的风险 / Risks
当前 head 的独立故障探针得到 2 失败 / 1 通过。使用真实 CLI、固定 SForge 版本 476b34d17e759a7dbcf9ec36a12ff42b757b462a 的原生文件写入、实际准入和账本,求解与评分结果来自确定性测试数据。健康对照读回看板终态并释放原槽位。33 项 Python 管理路径、19 项 TypeScript、187 项引导和 Yardstick 测试通过。基线与 head 各 75 项账本回归通过,默认运行器和账本的 10,381 字节归一化结果完全相同。配置的 mypy 检查通过 17 个文件。
语义与 CI 对齐
配对引导输出去掉末尾换行后的字符数为 small 3456→3328、crowded 3457→3329、multi_agent 3456→3328。Markdown 只删除重复标签,JSON 决策内容等价。分别去掉禁止用摘录验收的条款后,三种场景都无法通过语义检查。首次连接 smoke 在 r34 把导航当策略并触发 KeyError: should_run,当前 head 从原捕获读回并通过。同一 r34 base 的 quick premerge 六项直接检查与四项选中检查通过,保留 benchmark 人工门。
请在更新后的 head 用 pytest test_managed_final_receipt_replay.py 复验替换前和替换后的故障,恢复存储后从原试验重放并确认只释放原槽位。当前远端 CI 未核验。
我的整体评价 / Verdict
引导预算和首次连接验证的修复可独立复核。集成 head 的终态回执仍会丢失恢复锚点,阻断同一次尝试的重放。合入前要让两种故障都能从原证据恢复,并核对求解和准入各只有一次。真实模型评分、已安装版本和用户运行未核验。
English verdict: REQUEST_CHANGES
| try: | ||
| solver_retained = retain_managed_solver(trial, managed_binding, managed_launch, receipt=receipt, result=result) | ||
| except (OSError, TypeError, ValueError) as error: | ||
| receipt_path.write_text(json.dumps(receipt, indent=2)) |
There was a problem hiding this comment.
[P1] 这份集成 head 仍会在最终回执写入失败时覆盖同次试验的绑定。实际 CLI 在原子替换前及替换成功后分别注入错误,均丢失 managed_run 和求解摘要,重放返回 2,原槽位持续占用。请在原子回执 owner 保留已落盘的绑定,并验证替换前失败可从同次尝试的原生证据恢复,求解和准入各只发生一次。
Native workers can finish while their original grading tail is still pending. This integration successor composes terminal reconciliation with Yardstick, guided-output and validation fixes. It preserves the frozen r34 source branch and adds selected-base propagation so the premerge parent and CLI-budget child compare the same revisions.
Base:
codex/edgebench-integration-r34-20261107atc4536ec439fa7480bbfc76d7e26a96ac956fbf2f.Current head:
96a303532f166c4045730f41fda4daa1d68fddec.Included sources:
cd2f27317f57caeadd19db6ced3fc8bc0cd19d45: reconcile retained solver/grading terminal evidence, write and read back the same run row, then release and read back the original reservation generation. Managed launch binding is opt-in.b1428c3acc12307a27b54c2ab9181d501be78cf1: complete the existing Yardstick transport/path types without changing decisions or defaults.a4e723b5ca156240e169cf50a40c5c34cc93aaa8: remove repeated decorative labels from guided steps while preserving commands, purposes, ordering and stop obligations; retain old/new semantic recognition.db1f0c5e0749bf1bc071857a07f6bda8055c916a: use the existing current-capture reader and isolated runtime cleanup in the first-connect smoke, preserving its connection and first-Todo assertions.65296441836a7ce4b6bad55ea8627c20707b684e: pass the normalized premerge base only to the existing CLI-budget child. Standalone execution, neighboring checks and failed-child behavior remain unchanged.79665b97a0a453380da02e15b1b62eee895147dfenters through fix(canary): forward premerge base to CLI budget checks #8242's main ancestry: read retained project results without replaying work. Its affected Python and TypeScript conversation paths were tested on the combined checkout.All 11 commits introduced over r34 have DCO sign-offs. The local integration capability records the five exact source heads above; #8234 is their already-merged ancestry. Later main changes are not implied to be included.
Validation on this combined head:
manual_review_requiredfor benchmark-sensitive changes, with no failed executed checks. This does not grant main merge authority or certify every remote check.Earlier evidence remains scoped to its tested revisions: 17 actual terminal-reconciliation CLI/temp-ledger tests passed; the Yardstick follow-up passed 23 tests; the guided-step follow-up passed 258 tests with 1 skip and 1 reproduced pre-existing host-list assertion failure; the first-connect real CLI smoke passed on
95f3c152. Those checks were not silently relabeled as a fresh standard-tier aggregate on this head.The earlier standard premerge on
cc26e5beretained two failures (the unintended CLI differential base and the obsolete first-connect reader). Their repairs are included above. #8242's reported remote Ruff F841 is byte-identical in its selected main base and head; scoped checks passed, while the complete CI aggregate is not claimed green.The source fix #8242 has an independent exact-head approval review, published as COMMENTED for the author-owned PR. Its reviewed head is unchanged.
Exact-head integration review/readiness, maintainer merge authorization and live worker adoption remain separate. This PR update does not migrate active scorers, alter an experiment's frozen source, or claim a measured score, token or fee improvement.
Main delivery
The terminal reconciliation proposal now targets
mainin its original component PR. Follow the existing terminal receipt recovery finding there. The independent validation and Chat changes are represented in their own mainline PRs. #8236 · #8202 · #8193 · #8222 · #8234 · #8242