Skip to content

chore(benchmark): compose terminal reconciliation and validation fixes - #8237

Closed
huangruiteng wants to merge 11 commits into
codex/edgebench-integration-r34-20261107from
codex/edgebench-r35-signed-candidate-20261107
Closed

huangruiteng wants to merge 11 commits into
codex/edgebench-integration-r34-20261107from
codex/edgebench-r35-signed-candidate-20261107

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Native workers can finish while their original grading tail is still pending. This integration successor composes terminal reconciliation with Yardstick, guided-output and validation fixes. It preserves the frozen r34 source branch and adds selected-base propagation so the premerge parent and CLI-budget child compare the same revisions.

Base: codex/edgebench-integration-r34-20261107 at c4536ec439fa7480bbfc76d7e26a96ac956fbf2f.
Current head: 96a303532f166c4045730f41fda4daa1d68fddec.

Included sources:

All 11 commits introduced over r34 have DCO sign-offs. The local integration capability records the five exact source heads above; #8234 is their already-merged ancestry. Later main changes are not implied to be included.

Validation on this combined head:

  • Quick premerge against the exact r34 base, with both CLI-budget comparison environment variables unset: all 6 direct and 4 selected checks passed. This includes the actual differential budget script and the semantic-vocabulary, maintainability and public/private boundary checks.
  • 183 Python tests passed across repository-root/canary and retained Codex history/conversation bindings.
  • 61 TypeScript conversation-binding tests passed; TypeScript typecheck passed.
  • The quick premerge exit remains manual_review_required for benchmark-sensitive changes, with no failed executed checks. This does not grant main merge authority or certify every remote check.

Earlier evidence remains scoped to its tested revisions: 17 actual terminal-reconciliation CLI/temp-ledger tests passed; the Yardstick follow-up passed 23 tests; the guided-step follow-up passed 258 tests with 1 skip and 1 reproduced pre-existing host-list assertion failure; the first-connect real CLI smoke passed on 95f3c152. Those checks were not silently relabeled as a fresh standard-tier aggregate on this head.

The earlier standard premerge on cc26e5be retained two failures (the unintended CLI differential base and the obsolete first-connect reader). Their repairs are included above. #8242's reported remote Ruff F841 is byte-identical in its selected main base and head; scoped checks passed, while the complete CI aggregate is not claimed green.

The source fix #8242 has an independent exact-head approval review, published as COMMENTED for the author-owned PR. Its reviewed head is unchanged.

Exact-head integration review/readiness, maintainer merge authorization and live worker adoption remain separate. This PR update does not migrate active scorers, alter an experiment's frozen source, or claim a measured score, token or fee improvement.

Main delivery

The terminal reconciliation proposal now targets main in its original component PR. Follow the existing terminal receipt recovery finding there. The independent validation and Chat changes are represented in their own mainline PRs. #8236 · #8202 · #8193 · #8222 · #8234 · #8242

cocolord and others added 8 commits November 7, 2026 07:22
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
)

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…gebench-r35-signed-candidate-20261107

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…gebench-r35-signed-candidate-20261107

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…gebench-r35-signed-candidate-20261107

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

[P1] 修复集成提交里同次试验的回执恢复。 r35 原样继承 #8236 的兜底写入。最终回执在替换前失败,或已替换成功但报告失败时,兜底都会移除 managed_run 与求解器摘要。两次隔离复现中,provider 终态与评分排空证据均已落盘,重放仍返回 2、ready=false,原槽位持续占用。请在原子回执 owner 保留有效的已提交快照,并使替换前失败能从同次尝试的保留证据恢复。 兜底写入 · 源 PR 评审

动机 / Motivation

管理 EdgeBench 并发批次的人,需要在试验结束后确认结果并回收原名额,也需要用现有验证流程检查这份集成提交。
r34 需要分别确认看板和槽位。r35 将二者接到同一次试验的证据上,且修复了引导输出和首次连接验证。保存终态回执时若出错,原槽位仍可能卡住。
当前集成提交通过正常的终态读回和原槽位释放。最终回执在替换前或替换后报错时,重放都无法恢复绑定。
验证覆盖原生 CLI、隔离的 provider 证据与本地账本。真实模型求解、评分和实验净收益仍由独立试验验收。

改动思路 / Approach

分开保留运行器证据和控制器的有序写入。引导压缩、Yardstick 类型及首次连接验证可分别用现有 owner 核对。
本集成 head 仍需修复原子回执的重放边界,再用同一次尝试验证释放。保留已核验的三项验证改动。
候选相对 r34 包含八个带 DCO 的提交。与 #8236 的完整 benchmark diff 一致,另有七条路径承载引导预算、Yardstick 类型和首次连接 smoke 的修复。求解结束、评分排空、看板确认和槽位释放继续各自读回。 终态计划

具体改动 / Changes

依据改动前的 loopx/capabilities/benchmark_toolkit/README.md,版本 c4536ec439fa7480bbfc76d7e26a96ac956fbf2f。runtime-continuity 的同次尝试恢复仍有缺口。concurrency-envelope 的看板先写入、读回再释放原槽位,健康路径通过。score-countability 沿用现有评分边界。 既有工具契约

关键代码讲解

  • run.main 调用 retain_managed_solver 保存最终回执。错误处理会进入普通回执写入,替换已经提交或仍有恢复价值的绑定。 运行入口
  • publish_managed_terminal 用绑定、求解器摘要和评分排空记录生成证明。reconcile_benchmark_terminal 在锁内先读回看板终态,再释放并读回原槽位。 证明生成 · 终态控制器
  • render_start_goal_guided_markdown 移除重复的命令标签,保留命令、顺序、目的和原有读回要求。Yardstick 只补现有参数与返回类型。 引导输出
  • 首次连接的 run_cli 用已有测试 reader 读取同一次捕获的决策,继续执行原断言并清理测试运行时。新增维护说明也将该做法限定在测试 reader。 原生策略读回 · 测试维护说明

完整 diff 为 21 个文件,新增 1,497 行、删除 69 行。终态证据与有序写入沿用 #8236,验证改动有独立的 caller 和反例。

对主干的风险 / Risks

当前 head 的独立故障探针得到 2 失败 / 1 通过。使用真实 CLI、固定 SForge 版本 476b34d17e759a7dbcf9ec36a12ff42b757b462a 的原生文件写入、实际准入和账本,求解与评分结果来自确定性测试数据。健康对照读回看板终态并释放原槽位。33 项 Python 管理路径、19 项 TypeScript、187 项引导和 Yardstick 测试通过。基线与 head 各 75 项账本回归通过,默认运行器和账本的 10,381 字节归一化结果完全相同。配置的 mypy 检查通过 17 个文件。

语义与 CI 对齐

配对引导输出去掉末尾换行后的字符数为 small 3456→3328、crowded 3457→3329、multi_agent 3456→3328。Markdown 只删除重复标签,JSON 决策内容等价。分别去掉禁止用摘录验收的条款后,三种场景都无法通过语义检查。首次连接 smoke 在 r34 把导航当策略并触发 KeyError: should_run,当前 head 从原捕获读回并通过。同一 r34 base 的 quick premerge 六项直接检查与四项选中检查通过,保留 benchmark 人工门。

请在更新后的 head 用 pytest test_managed_final_receipt_replay.py 复验替换前和替换后的故障,恢复存储后从原试验重放并确认只释放原槽位。当前远端 CI 未核验。

我的整体评价 / Verdict

引导预算和首次连接验证的修复可独立复核。集成 head 的终态回执仍会丢失恢复锚点,阻断同一次尝试的重放。合入前要让两种故障都能从原证据恢复,并核对求解和准入各只有一次。真实模型评分、已安装版本和用户运行未核验。

English verdict: REQUEST_CHANGES

try:
solver_retained = retain_managed_solver(trial, managed_binding, managed_launch, receipt=receipt, result=result)
except (OSError, TypeError, ValueError) as error:
receipt_path.write_text(json.dumps(receipt, indent=2))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 这份集成 head 仍会在最终回执写入失败时覆盖同次试验的绑定。实际 CLI 在原子替换前及替换成功后分别注入错误,均丢失 managed_run 和求解摘要,重放返回 2,原槽位持续占用。请在原子回执 owner 保留已落盘的绑定,并验证替换前失败可从同次尝试的原生证据恢复,求解和准入各只发生一次。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants