Repository navigation
Resume continued benchmark phases after a proven no-host budget stop - #8151
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: actor_kind=model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max.
Request changes conclusion (author-owned PR; GitHub blocks formal self-review)
当前 exact head b8cee72004360c0833eb44e33de665b66474d69b 有 1 个阻塞问题:更晚阶段可以绕过中间阶段的不可重试 Host 失败。
动机 / Motivation
运行多阶段 Harbor 或恢复 SForge 试验的操作员,会在上一阶段耗尽时间后需要继续原任务。
旧路径会重放已记录的预算停止;新路径按后续阶段的 deadline 选择新 transport state,首次续接能够继续原生会话,但更晚的续接会绕过中间阶段的失败。
可观察目标是显式续接恢复有用执行,同时保持原失败、重试次数、会话和 Core admission 的约束;当前 head 尚未满足失败保持这一部分。
本 PR 不应改变任务评分、总预算、模型或账号,也不应把本地 transport 续接变成 Core 完成或恢复授权。
改动思路 / Approach
phase wrapper 留在研究 transport,复用原 shell scheduler 的 argv、锁和 Host retry owner,边界本身合适;当前资格判定始终读取最早 state,未覆盖之后产生的持久失败。
本次边界是显式阶段续接的 transport 选择与回读;不新增 Core retry policy、评分器或模型执行。
这里的 transport state 是 shell 调度器保存的本地调用记录,不是 Goal 的完成状态。显式续接不宜重建 Goal、清空旧 attempt 或把 benchmark 时间规则塞进通用 Core。现有单消费者锁、独立 IO witness(启动前写入的持久见证)以及 Host 失败策略应继续约束后续调用。当前实现保存了文件,却没有让最新失败继续参与下一阶段的选择。
| 场景 | 当前观察 |
|---|---|
| 初次预算停止后续接 | 能启动并保留原 native session |
| 同一 deadline 下重启失败阶段 | 原失败保持,Host 调用次数仍为 1 |
| 再换一个后续 deadline | 另选空 state,Host 调用次数变成 2,违反失败保持 |
具体改动 / Changes
审查基准为已验证 main:faab5f5c291c5359216a0f944fe63fcfbaacdedc。完整差异为 5 个文件、+253/-2:3 个 production 文件 +113/-2、测试 +131、文档 +9,未采用旧 review 结论。接受依据为 benchmark/runtime/RUNTIME.md,spec_revision faab5f5c291c5359216a0f944fe63fcfbaacdedc;其中 Results, timeout and recovery 对原 Turn/session 与既有恢复 owner 的要求仍由未改动的 worker 和 Core 路径承担,定向真实进程用例验证了 session/Goal bytes 保持。另核对了同一基准的 Shell worker Host failures:更换 state 路径不能成为硬失败恢复手段。
关键代码讲解
resume_state_file在原 state 的锁内检查 stopped/75、witness 及 pre-dispatch receipt;它只读取传入的根 state。然后按新 deadline 生成 sibling,遗漏实际最近阶段的 started/failed attempt。worker_command保持普通入口 argv,只在 explicit continuation 时进入 wrapper;main再 exec 原 shell scheduler。base/head 五种普通模式的 argv 完全相同,未发现 feature-off 路径漂移。Harbor.run对每个后续阶段仍传同一个_SCHEDULER_STATE;SForge warm restore 也使用固定入口。因而第三阶段不会自然接续第二阶段的 transport record。Core admission 没有被修改,但它也不能替代 Host attempt 的失败保持。
对主干的风险 / Risks
[P1] 续接时必须检查最近实际阶段的失败。 我用真实 shell Host 子进程复现:第一阶段保留合法预算 stop;第二阶段记录不可重试 quota failure;同阶段重试正确返回 hold,调用次数为 1;第三阶段仅改变 finite deadline,便选择新 state 并再次调用 Host,次数变为 2。原 stop 和第二阶段失败文件 bytes 均未改变。既有新测试只证明同一个 deadline 会复用失败文件,因此全部通过仍会漏掉这个反例。
续接必须定位并校验实际最近的 predecessor attempt、IO witness 和对应 worker receipt;中间阶段失败、started 或证据未知时保留其恢复路径。 最小回归矩阵应覆盖三阶段 hard failure、unknown、started/interrupted,并同时证明连续两个合法 budget-only stop 仍能前进;不能只把所有续接关闭。
语义与 CI 对齐
这是本地派生 transport 身份,未新增公共 actor 或 Core vocabulary;问题在于选错 predecessor 后绕过既有 typed failure 约束。--wake-host-failure 的硬失败保持是强制规则,不是建议。修复应保持 Core Stop、同 session、重试次数和原证据的既有语义。
验证:PASS 32 项 phase resume/budget 定向用例、Ruff、普通五模式 base/head argv 对照、5 项 native direct checks。首次 premerge 的 vocabulary 检查因未安装 TypeScript 失败;补齐 npm 依赖后原检查 PASS,首次失败记录保留。完整 benchmark 集为 892 PASS / 670 SKIP / 10 FAIL;同样 10 项在 immutable base/head 重放均失败:5 项在未修改的裸测试 import 中缺少 test_loopx_turn_codex_cli,5 项在调用 worker_command 前访问不存在的 should_run。这些既有失败不解释或抵消本次独立反例。原 maintainability debt 也在 base/head 同样失败,未更改限额。
当前强制 no-failure-reset oracle FAIL,strict change-quality 不具备 passing qualification;首次 premerge 的整体失败和 manual hold 均保留。没有查询或等待远端 CI,也没有启动真实 benchmark、付费模型、安装或升级。
我的整体评价 / Verdict
phase wrapper 留在研究 transport,复用原 shell scheduler 的 argv、锁和 Host retry owner,边界本身合适;当前资格判定始终读取最早 state,未覆盖之后产生的持久失败。
交付判定为 not_yet_proven:首次续接的有用增量已证明,整个阶段链的失败约束则发生回归。long_horizon 与 user_experience 均为 regression;保留文件不等于保留其执行约束。当前 wrapper 的体量与研究边界相称,不需要扩成新 Core 状态机;必须先修复最近 predecessor 的资格判定,再重审新 head。
Inspected benchmark scheduler, Harbor/SForge command builders and runtime documentation; no rendered product presentation changes.
Real Harbor backend, live provider allowance and model outcomes were not executed.
English verdict: REQUEST_CHANGES — a later phase can bypass the immediately preceding nonretryable Host failure; preserve that attempt's recovery before dispatch.
| return str(path) | ||
| # Only the immutable phase clock names the new state. Todo/reset-token | ||
| # changes cannot clear a failure already recorded in this new phase. | ||
| suffix = hashlib.sha256(float(phase_deadline).hex().encode()).hexdigest()[:20] |
There was a problem hiding this comment.
[P1] 这里始终从最早的 stopped state 派生新 deadline 文件,没有校验中间实际阶段。已用真实 shell Host 子进程复现:phase1 预算停止,phase2 quota_exhausted;同 deadline 重试保持 1 次调用,phase3 换 deadline 后新 state 又启动 Host,调用数变 2。原 stop、失败文件仍在,所以保留 bytes 不等于保留失败约束。请绑定并校验最近 predecessor 的 attempt/witness/receipt;hard failure、unknown、started/interrupted 不得借新路径绕过。增加三阶段负例,并保留连续合法预算停止仍可续接的正例。
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Self-review after the P1 repair — exact head 1f92f13f66364502cd440f983d6ff955107adcb5.
- Phase continuation now follows explicit
resume_fromlinks from the root state and locks existing phase records while selecting the predecessor. - If the latest predecessor is failed, unknown, started, interrupted, unreadable, or otherwise not a pre-dispatch budget stop, that same state is returned so the original scheduler failure hold remains authoritative.
- A new phase is created only from the actual latest predecessor with a matching no-host receipt; two consecutive valid budget stops can still advance.
- Added a three-stage regression matrix for hard failure, unknown, started, unreadable, and interrupted attempts, plus a positive consecutive-budget-stop case.
Verification at the exact head: 18 passed in test_scheduler_phase_resume.py; phase-budget suite 20 passed with the previously disclosed intermittent session-identity failure; Ruff and diff checks pass. Current main was merged with DCO.
huangruiteng
left a comment
There was a problem hiding this comment.
Independent approval of exact head 1f92f13 after latest-phase failure preservation.
An explicitly continued benchmark could immediately replay its previous phase’s budget-stop receipt instead of invoking the retained native conversation. Merely extending its deadline did not resume work.
Harbor follow-up phases and SForge warm restores now reuse the existing shell scheduler with a phase-specific transport state only when the retained worker receipt proves budget exhaustion before native dispatch. The finite deadline identifies that phase, so retries retain its failures. The original stop, independent IO witness, Goal/Agent, Todos, native session and score history remain intact. Interrupted native work, unknown evidence, provider failures, other stops and terminal LoopX decisions retain their original handling.
Validation: the 12 new tests passed, including a real worker subprocess and SQLite continuation of the same native conversation; the characterization first proves that changing the clock alone still replays the old stop. The broader pinned-provider run had 539 passes and one failure in the existing 3-second input-backpressure/session-identity case. A targeted recheck reproduced that failure; it remains unresolved. Ruff and diff checks passed. This is a completed narrow recovery proposal, with that validation limitation recorded; no merge or full-duration result is claimed.
Placement: this composes the existing research runtime’s transport and host-attempt validator. It adds no product retry policy, authority vocabulary, checkpoint cadence or native scoring behavior. The separate short-budget reserve fix is compatible but independent.