fix(turn): fence remote execution and terminal writeback

[AetherX-Technologies]

Summary

• add durable fencing tokens and an append-only journal across every LoopX Turn transaction phase
• make CLI, SkillsBench, remote/adaptive hosts, quota, scheduler, and todo writeback replay-safe
• preserve task leases until the complete terminal writeback commits, retain released tombstones, and permit only owner/key-matched terminal replay
• extract state-refresh recording projections to keep the control-plane maintainability ratchet within budget

Validation

• 321 focused control-plane and Turn tests passed
• fake-host walkthrough passed for preview, commit/replay, and recovery after partial writeback
• mypy: 16 source files, no issues
• maintainability ratchet: passed
• premerge: 9/9 catalog canaries, 8/8 risk-profile smokes, Python compile, diff hygiene, and public-boundary scan passed

Review hold

The premerge gate reports manual_review_required only because the diff touches the benchmark-sensitive SkillsBench adapter. This PR must receive maintainer review and must not be self-merged.

[huangruiteng]

PR #3074 exact-head 评审（head 70c535f8d2ce09465807a937f13b7179879a057e）

详细中文评审

动机

LoopX Turn 事务此前只有单文件 JSON journal 覆盖与简单写回：进程崩溃/远端 host 回放时，writeback、spend、scheduler 等不可逆副作用缺少"一次且仅一次"的栅栏保证，任务租约也可能在终态写回提交前被释放，导致回放重复执行副作用或丢失 fencing 单调性。本 PR 把 Turn 运行改造为"租约 fencing + append-only 校验 journal + 副作用 exactly-once"的封闭运行时，并让 CLI、SkillsBench/远端 host、quota、scheduler、todo 写回全部走同一套可回放契约。

改动思路

四条主线：

1. 租约 fencing：任务租约新增确定性 fencing_token（goal/todo/owner/idempotency_key/acquired_at/generation 的 sha256）；release 不再删文件而是写 released tombstone（version+1、保留 generation），后续 acquire 从 tombstone 递增 generation，保证跨代单调；require_current/effect_guard 在每次副作用前后校验。
2. Append-only journal：每 Turn 一个 JSONL 事件流（turn-journals/<digest>.jsonl），每条事件带规范哈希、phase_key 唯一性、schema/字段/大小/行数强校验，追加时先校验租约 fence；投影文件原子重建。状态从事件流重建，回放只依赖权威日志。
3. Fenced runtime：fenced_runtime.py 把每个不可逆副作用（durable_writeback / quota_spend / scheduler_apply）包进 _guarded_effect：记录 phase_intent → 再次校验 fence → effect_guard 内执行 → 校验最新 fence；租约或 journal 不变量破坏一律 fail-closed（failed_closed，带失败阶段）。
4. Exactly-once 副作用：turn_effect.py 定义 turn_effect_key + 输入哈希；state-refresh 与 quota spend 在各自 run index 上做 effect 记录去重（同 key 同 hash 幂等回放，同 key 异 hash 冲突报错）；完成写回不再立即释放租约（release_task_lease_on_commit=False），由驱动在终态提交后 release(final_fence)。

具体改动

• turn_driver/journal.py（+459）：事件 schema v1、build_turn_event/_validate_event/_load_turn_events_unlocked（哈希+字段+phase_key 唯一性+截断检测）、append_turn_event（租约锁内校验 fence 后追加+fsync+投影）、rebuild_turn_projection、LocalTurnJournalStore。
• turn_driver/lease.py（+217）：TurnFence/TurnLeaseAuthority 协议、TurnLeaseController（acquire/renew/require_current/release/heartbeat/effect_guard）、心跳续租线程。
• turn_driver/fenced_runtime.py（+1070）：load_turn_state（事件流重建）、_guarded_effect、_execute_fenced（各阶段：host→validation→settlement→scheduler→committed；失败/重试/fault-injection）、run_fenced_loopx_turn_once（acquire→heartbeat→release，失败 fail-closed）。
• turn_driver/executor.py：旧 staged 逻辑迁入 fenced_runtime，run_loopx_turn_once 变成薄适配；transaction.py 新增 TurnEffectEnvelope（phase_key 规范、fencing token 格式校验）与 FAILED_CLOSED 结果类型；settlement.py 接受 idempotent 提交。
• work_items/task_lease.py：tombstone release、task_lease_fencing_generation/token、_require_task_lease_fence_unlocked/require_task_lease_fence/hold_task_lease_fence、可重入 hold_task_lease_lock、terminal_replay_key 的 done-todo owner 放行（仅匹配 completion_turn_key）。
• turn_effect.py + state_refresh_effect.py + state_refresh_recording.py + slot_accounting.py：effect key/input hash、refresh/spend 的 exactly-once 封装（run index 锁内查重、幂等回放返回原 receipt）。
• CLI：turn.py 增加隐藏 journal-read/journal-append；refresh-state/quota spend-slot 增加隐藏 --turn-effect-key/--turn-fence-* 并经 hold_cli_turn_effect_fence 执行；turn run-once --execute 构造 TurnLeaseController（idempotency turn:<turn_key>、terminal_replay_key=turn_key）。
• benchmark_adapters/skillsbench_turn_runtime.py：SkillsBenchTurnLeaseController（scored-workspace 远端租约）+ SkillsBenchTurnJournalStore（远端 journal-read/append）+ 租约/journal 失败分类（todo_lease_conflict/stale_fencing_token/lease_not_active/lease_cas_mismatch/journal_invariant_failed）+ 副作用 fence 参数透传。
• turn_envelope.py/driver.py：selected_todo 投影 required_write_scopes、adaptive primary_todo、签名覆盖率 V2/V3 与差分迁移表。
• todos.py：complete_goal_todo 使用 hold_task_lease_mutation_locks，支持 release_task_lease_on_commit=False。
• 测试/smoke：新增 tests/test_loopx_turn_* 覆盖 journal/fenced runtime/transaction、task-lease tombstone/fencing、state-refresh effect、fake-host walkthrough。

对主干的风险

• 风险点是"副作用幂等性依赖 effect key + input hash 的一致性"：key 派生自 turn:<turn_key>，输入哈希覆盖 refresh/spend 的完整参数；若未来某个调用漏传 effect key，会退回非幂等旧路径（_call_compatible 兼容旧回调）。这是设计上有意的兼容缝，建议后续把 Turn 路径的所有 refresh/spend 调用收敛到必传 effect key（P2 观察，不阻塞）。
• SkillsBenchTurnLeaseController.effect_guard 本地为空实现，理由远端 CLI 自身持锁；若 scored-workspace 是旧版 CLI（无 fence 参数），远端命令会因未知参数 fail-closed，不会静默降级——可接受（P2 观察）。
• 评审时该分支尚无 CI checks（PR 很新），321 测试/预合并结果来自本地验证；需 CI 在分支上确认，尤其它触碰 benchmark-sensitive SkillsBench 适配器（PR 自述 manual_review_required、禁止 self-merge——与我们的评审流程一致，本评审即 maintainer review，合入仍由 owner 执行）。
• 其余（append-only 哈希 journal、tombstone generation、fail-closed、fault injection 测试）风险低。

我的整体评价

这是对 Turn 事务正确性的一次系统级收口：租约 fencing 贯穿宿主/远端/benchmark 三端，journal 从"覆盖写"升级为"带哈希校验的 append-only 权威日志"，三个不可逆副作用都有 exactly-once 幂等回放，所有破坏不变量路径都 fail-closed 并保留失败阶段。本地验证充分：299 项聚焦测试、fake-host walkthrough（preview/commit/replay/部分写恢复）、maintainability ratchet 与公共边界扫描均通过。无阻塞项，整体评价 APPROVE；按 PR 自己的 hold 要求，合入前需 owner 在 CI 绿后执行（不 self-merge）。

验证

• exact head 70c535f8d2ce09465807a937f13b7179879a057e（评审时 head 未变）：
  • 299 项聚焦测试通过（turn driver/executor/journal/transaction、skillsbench turn runtime、turn envelope、task lease、quota slot accounting、todo mutation authority、orchestration admission；Python 3.12）
  • examples/loopx-turn-fake-host-walkthrough-smoke.py：通过
  • examples/control_plane/control-plane-maintainability-ratchet-smoke.py：通过（0 unreviewed/stale/magnitude）
  • loopx check 公共边界扫描：6 个新增/核心文件 clean
  • CI：评审时该分支尚无已上报 checks（需 owner 在 CI 绿后合入）

---

Review: APPROVE

Head: 70c535f8d2ce09465807a937f13b7179879a057e

Verdict: APPROVE — a systematic closure of Turn transaction correctness: lease fencing (deterministic tokens + released tombstones with monotonic generation), append-only hash-verified journal, exactly-once replay for state-refresh/quota-spend effects, fail-closed invariants, and end-to-end CLI/SkillsBench/remote-host integration. 299 focused tests plus the fake-host walkthrough, maintainability ratchet, and public-boundary scan pass at head.

Key finding: No blockers. Two P2 observations: keep routing all Turn-path refresh/spend calls through the mandatory effect key (the legacy _call_compatible seam is intentional but should shrink), and note that SkillsBenchTurnLeaseController.effect_guard relies on the remote CLI holding its own fence — older scored-workspace CLIs fail closed rather than degrade silently. CI had not yet reported on this branch at review time; per the PR's own hold, merge must wait for green CI and owner action (no self-merge).

Validation: exact-head 299 focused tests, fake-host walkthrough, maintainability ratchet, and public-boundary scan all pass.

[huangruiteng]

Thanks for the detailed summary and the review hold. The fencing/journal direction looks sound, and this PR is already partially on the effect-program abstraction: build_loopx_turn_transaction_plan constructs a typed SettlementPlan (validation -> durable_writeback -> quota_spend) whose steps all reference $.identity.effect_id as the idempotency key, and fenced_runtime consumes interpret_turn_result_packet + settlement_result_payload. That is exactly the "second real caller" the effect-program RFC (docs/architecture/rfcs/agent-loop-effect-interpreter-v0.md and the .zh-CN mirror, M7.0/M7.2) was waiting for.

Non-blocking suggestions to converge the remaining parallel machinery onto that same abstraction:

1. One identity lineage for journal + settlement
Today there are two idempotency systems: settlement effect_id, and turn_effect_key/effect_input_hash (turn_effect.py), wired as turn_effect_key=effect.phase_key in cli_commands/turn.py:661/727. Suggest folding the turn effect key into SettlementIdentity (optional field) or EffectRequest.context, and deriving phase_key from effect_id. Owner/key-matched terminal replay then becomes a pure function of one identity instead of a convention kept in sync between two stores.

2. Derive phase vocabulary from the program
TRANSACTION_PHASES/TURN_EFFECT_PHASES and TurnEffectEnvelope.phase_key are a parallel vocabulary to SettlementPlan.steps. Suggest building the transaction phases from one EffectProgram (settlement steps plus host_execute/typed_result/scheduler_apply/scheduler_ack), so journal event_types use SettlementStepKind-based tokens (phase_intent/effect_result/replay_decision) and _record_intent/_unique_phase_key/load_turn_state become generic trace/replay logic instead of turn-specific helpers.

3. Replay authorization as a read-only lens
Add interpret_turn_journal(packet) mirroring interpret_quota_should_run_packet: input = journal trace + replay request, output = typed decision covering owner/key match, tombstone retention, and phase-order legality. This keeps the M6 "read-only packet interpreters" discipline and gives the replay tests a stable pure surface.

4. Keep lease as a handler seam
TurnLeaseAuthority is already the right protocol. If fenced_runtime stays a handler set (local task_lease plus remote SkillsBench adapter implement it), the benchmark-sensitive SkillsBench diff shrinks to adapter-only code and the review hold becomes easier to clear.

5. M7.1 parity fixtures
The fake-host walkthrough is good; add independent characterization fixtures for partial execution, retry, cancellation, permission denial, budget rejection, and owner/key-mismatched replay (per RFC M7.1), asserted from intended invariants rather than implementation output.

Does this require a generic abstraction enhancement?
A small, read-only one - and this PR is the justification, not the risk:

• SettlementIdentity: optional turn_effect_key (or an idempotency_keys tuple) with backward-compatible serialization.
• effect_program.py: a helper projecting SettlementPlan (+ host phases) into one EffectProgram (extend effect_program_from_ordered_steps).
• interpret_turn_journal(...): a new read-only interpreter alongside interpret_quota_should_run_packet / interpret_turn_result_packet.

Explicitly out of scope for this PR: a shared EffectInterpreter protocol or generic executor (RFC R4/M7.3 defers it until both the Codex App path and this isolated turn driver consume the same plan/receipt algebra). Keeping journal/lease as handlers now avoids the speculative framework the project explicitly warns against.

Happy to prepare a follow-up diff for items 1-3 if useful.

[huangruiteng]

Requesting changes per the design feedback in #3074 (comment). Key asks: (1) unify the turn effect key into the settlement identity so journal + settlement share one idempotency lineage; (2) derive transaction phase vocabulary from the SettlementPlan/EffectProgram instead of a parallel phase list; (3) express replay authorization as a read-only interpret_turn_journal lens; (4) keep lease/fencing behind the TurnLeaseAuthority handler seam so the SkillsBench diff stays adapter-only; (5) add M7.1 parity fixtures for partial execution, retry, cancellation, permission denial, budget rejection, and replay mismatch. No objection to the fencing/journal semantics themselves; these are convergence asks before merge.

[huangruiteng]

1

[huangruiteng]

详细中文评审（重审确认）

精确评审头： 3074@70c535f8d2ce09465807a937f13b7179879a057e

动机

按全队列重审要求复核本 PR（Turn fencing：37 个文件，覆盖 fenced_runtime/journal/lease/settlement/transaction）。exact head 自 08-11 的 CHANGES_REQUESTED 后未更新（updatedAt 2026-08-11T14:29:19Z），阻断项仍待作者修复。

改动思路

PR 为 LoopX Turn 增加真实 fencing：fenced runtime、journal、lease、settlement 与 transaction，方向正确，08-11 早间曾 APPROVE，随后因设计反馈转为 CHANGES_REQUESTED。

具体改动（关键内容讲解）

• 阻断项 1（design）：turn effect key 需统一进 settlement identity（08-11 14:28 CR 的 key ask），当前 head 未体现修复。
• 阻断项 2（设计反馈）：CR 引用 #3074 评论中的其余设计要求（effect identity 一致性），需作者逐项回应。
• head 未变：自 CR 后无新 commit，阻断项未解决。

对主干的风险

fencing 语义若不一致会导致回放/结算错位，影响 quota 与 spend 正确性；属高影响控制面改动，需设计共识后再合。

我的整体评价

REQUEST_CHANGES（确认既有结论）。 修复要求：按 08-11 CR 的 key ask 统一 turn effect key 与 settlement identity，并回复其余设计反馈；修复后需 exact-head 复审。

---

English Verdict

REQUEST_CHANGES — exact head 70c535f8d2ce09465807a937f13b7179879a057e.

Confirms the existing CHANGES_REQUESTED: the turn effect key must be unified into the settlement identity (and the referenced design feedback addressed). No new commits since 08-11; blockers remain. Re-review after the author updates the head.