Repository navigation
fix(benchmark): cap restored phase at audited remaining allowance - #8205
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6-astra; OpenAI; runtime_reported; ultra
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
受影响的是从早期 checkpoint 接续有限预算研究的 benchmark 操作者。快照尚余两小时,但后续已花掉一小时;原入口仍发两小时,新入口可把新接收端限制为核算的一小时。仅在外面加一个定时终止进程并不足够:worker 和原生注册仍可能看到不同截止,重试也可能再次获得旧额度。本次可观察的改进是,接收端和原生注册共用缩短后的截止,恢复原请求不会重新计时。
我将这个 PR 判断为有独立价值的增量:它交付预算缩减,不证明旧 checkpoint 包含最新工作,也不授予额外模型时间、评分机会或实验启动权限。完整研究状态交接和原 session 后续工作仍需逐个接收端独立验证。该边界应保留,不能因时间修好了就宣布完整恢复成功。
改动思路 / Approach
改动留在既有原生恢复适配层,复用首次激活记录和单一截止,没有新增预算控制器。调用方把已审计的剩余时间传给原恢复入口;控制器取“捕获余量加独立授权扩展”与该上限的较小值,然后走原来的持久激活、原生准入及 worker 接收链。
我比较了不改、外部超时和这次设计:不改无法表达更短余额,外部超时无法让所有接收端读同一截止;新建控制器则会复制既有恢复权威。因此这个数值参数和原激活绑定是足够小且完整的边界。它是专用 provider 的 Python 传输,不需要另建通用 TypeScript 预算 owner。首次激活之后必须保留相同 cap,包括比原捕获余量更大的 cap;恢复时不能重新解释调用方意图。
本 PR 只交付可验证的预算缩减与恢复不补时;完整状态交接仍独立验收。
具体改动 / Changes
关键代码讲解
_restore在读取控制器和原生准入前拒绝布尔、负数、非整数及不可精确表示的上限;有 cap 时必须有捕获的 Host 时钟。通过封存校验后计算较小余额,并继续沿用工作预留时间限制,零额度不会创建可运行接收端。_activation_epoch保持原有实现:先持久化首次激活的绝对截止,再联系原生服务。新 cap 只作为显式提供时的绑定字段;遗漏时不改变旧记录形状。恢复校验精确绑定,不重新从当前时间计算额度。- runner 的 warm activation 把公开参数透传给该控制器,再把同一 epoch 交给原生注册与 WarmStart。运行回执也仅在明确提供 cap 时投影该字段;冷启动误用在创建实验尝试之前拒绝。
我按 benchmark/README.md 在基线 3a0c9608cc068cebcdeadf45217be8b5e4196341 的既有承诺核对:captured-allowance 的默认捕获余量保持;activation-bound-recovery 的首次激活和不补时成立;native-owned-opportunities 的次数、冷却及分数仍归原生 owner;research-state-boundary 的完整研究状态资格在本 PR 范围外,没有借 cap 放宽。
对主干的风险 / Risks
最强反例是原生激活已经提交、响应却丢失,稍后 recover 重新发放一份短额度。我独立运行真实原生 HTTP 服务与 typed artifact 封存/恢复链,现有测试加两个补充反例共 139 通过、3 个可选 Docker 资格测试跳过;另一个基线/新版差分及补充反例批次 3 通过。基线模块取自上述不可变提交,固定同一个捕获点和时钟,只归一化新 run、目标路径和 token/relay 身份,比较完整绑定、结果和截止相等。
额外核对了短额度到期后的恢复:它拒绝重新准入,原 phase 和激活回执字节不变。将 cap 在公开控制器 CLI 路径故意丢掉时,600 秒的独立期限断言确实失败,暴露约 7200 秒的旧余量;恢复正确代码后同一断言通过。Ruff、diff hygiene 通过;没有修改 RFC 配对文件。
默认与显式路径均被覆盖。CLI help/README 增加发现入口不会启用 cap,也没有改 solver 提示、Goal quota 或评分协议。不能把本次 synthetic grading 和 HTTP 验证解释成完整真实模型恢复;跳过的 Docker 资格及 latest-state 交接仍需原恢复流程独立处理。
语义与 CI 对齐
这是 provider 局部的整数上限和原持久绑定扩展,没有新增共享状态词汇。公开语义是“只缩短、首次绑定、恢复不补时”,并明确排除完整状态资格,命名与实际权威相符。远端 CI 当前 kernel-static-checks 及下游汇总门为红,若干后续套件跳过;本次未归因,不将它们冒称旧问题。DCO 和 dependency-review 成功。上述精确 head 的独立本地验证已覆盖本次改动的不变量,审查结论与远端 CI/合并门分开保留。
我的整体评价 / Verdict
在 229dae520c9cf10736c2b347fa81dc36dbfebcd1 上没有发现阻塞问题,支持这项有边界的改动。它改善长程运行的有限预算一致性,也让操作者在既有命令中表达已知余额,不增加额外确认或人工同步两个截止。默认行为、历史回执和恢复错误方向均有实证;完整状态恢复及收益没有被夸大。
本次没有渲染页面变化,检查范围是操作 CLI、provider 传输及文档,无需 UI 截图;未验证的视觉范围为无。继续保留单一激活 owner,现阶段不需要新 schema 版本或独立 watchdog。此结论是审查通过,不授予合并或实验启动权限。
English verdict: APPROVE — exact head 229dae5. Real native HTTP, typed artifact transport, omitted-cap baseline parity, expiry and lost-response recovery pass; the deliberate dropped-cap mutation fails as expected. Complete research-state/model recovery remains outside this bounded change.
A checkpoint can retain more time than the study now has left. Restoring it currently republishes that older allowance; an outer timeout cannot make the receiving model and native registration see the correct deadline.
Add an explicit remaining-time cap to the existing native restore controller and runner. The first activation uses
min(captured remaining + explicitly authorized extension, cap)and binds the cap to recovery. Omitting it preserves existing activation receipts and behavior. Accepted submission counts, cooldown, history and scoring stay with the native owner. A cap does not certify uncaptured work or a stale checkpoint as a lossless continuation.This stays in the existing specialized Python provider transport; it adds no generic control-plane decision owner or product execution toggle. The adjacent improvement keeps both deadline consumers on the controller's single persisted epoch. Public operator usage and limits are documented in English and Chinese.
Validation: Linux production controller CLI with real native HTTP and typed artifact transport: 137 tests passed, 3 optional Docker qualification tests skipped. Synthetic grading is used in this suite; no live model or research-state recovery is claimed. macOS selected tests: 87 passed, 53 unavailable-provider skips. A deliberately dropped cap failed the public-CLI deadline oracle (7200 seconds instead of 600); corrected code passed. Covers lost-response readback, identical-cap recovery, changed-cap refusal, no extension from a larger cap, zero/reserve refusal, malformed values, and omitted-cap compatibility. Ruff, diff checks and bilingual RFC check pass. Complete stale-worker handoff remains a separate gap.
The local risk-based premerge run completed all 19 selected checks with zero validation failures. It returned exit 1 solely for the expected
benchmark_sensitivemanual-review hold; this is not merge readiness or merge authority. Independent exact-head review is published. The integration build also completed 140 tests with 3 optional Docker skips through the real native HTTP and typed restore transport.