Refactor/event driven control plane - #3200
Conversation
|
感谢把方向和实现一次性展开。事件事实、显式 Task lifecycle、lease/retry、Goal acceptance 与 closure 分离，这些方向值得保留；但当前提交更像一个并行控制面原型，不是可直接合入的迁移 PR（+12,685/-97，旧路径基本未删除）。我建议先不要合并或直接 retarget，而是从最新 我本地做了以下验证：
合并前至少有这些阻断项：
建议拆分与合并节奏：
另外，当前 head 基于 |
|
补充一个更明确的取舍，尤其是 Policy 与现有 Effect Program 的关系。 先按当前真实价值拆分
硬约束：Policy 不得成为第二套 Effect/Settlement 引擎主线已经有通用 Effect Program 与 settlement algebra：有 ordered steps、failure short-circuit、receipt accumulation、effect identity，以及 scheduler-outside-settlement。新 Policy 层应成为它的纯决策前端/编译阶段，而不是并列的执行系统： 这里值得形成一个通用抽象，但抽象点应是 “typed verdict → effect program”，不是
第一版无需做庞大的通用 policy framework。建议只增加最小 typed contracts，例如 建议把 #3200 停在设计/试验参考，不继续叠代码；从最新
这样 Policy 的收益是统一决策形状与组合律，Effect Program 的收益是统一执行与结算语义；两者互补，同时避免多一套 queue/event/checkpoint/worker truth。 |
huangruiteng
left a comment
huangruiteng
left a comment
There was a problem hiding this comment.
PR #3200 全量双语评审 — Refactor/event driven control plane
精确评审头（Exact Head）: 3200@f6549a9e9f1f21cb1600de2765b461840c26a196
Base: frontend-control-plane-im-prototype-rfc（owner 已两次要求改为 main，见下）
规模: 67 files, +12685/-97；production 41、test/example 22、build config 1、public docs 1、other 2；无 CI checks rollup
详细中文评审
动机
PR 将 LoopX 控制平面从「心跳轮询 → 多层检查 → 运行时 → 分散投影」重构为「事件 → 状态投影 → 统一策略 → 任务队列 → worker 租约执行 → 事件」的事件驱动 Task OS 模型。动机是真实且可复现的：quota/capability/scope/scheduler 决策分散在多层，状态同时存在于多个投影与 ledger，任务缺少统一的 claim/lease/retry/崩溃恢复生命周期，goal 闭包依赖单个 todo 的 no_followup 而缺少验收证据。PR 明确引用了 website1 颜色会话的具体故障（已闭 goal 被复用污染、完成 todo 不写事件导致 ready=6 永远无法闭包），这些是真实的自动化卡死问题，而不是演示性的重构理由。就近小修不足以解决：这些是横切架构问题，继续在心跳层打补丁只会增加更多层；因此「事件驱动 + 显式生命周期」的整体方向成立。
改动思路
架构主线清晰：Event=事实、Projection=当前状态、Policy=约束、Scheduler=调度权威、Lease=执行权威、Worker=执行、Acceptance=验证、Closure=终止。正路径设计为：todo add/complete 桥接 rollout 事件 → dispatch 从 handoff gate 重算 READY 后继 → 入队（幂等）→ worker claim（带 lease）→ resident 执行门（command + prefix + guard）→ finalize 把任务标记完成 → Goal Acceptance 验证 → goal_closure_ready + goal_closed 原子闭包 → registry 状态同步 closed。负路径也有明确设计：终态 todo 永不重新入队；lease 过期 → pending（僵尸恢复）；retry_wait 到期 → pending；goal_acceptance_pending 阻止无 criteria 的 tick 静默闭包；执行门不满足绝不自动执行。接入面包括：quota live_decision 附加统一 policy_decision、Claude/OpenCode/Pi 的 run gate 优先读 policy_decision、bootstrap/start-goal 增加 closed-goal 拦截、todos.py 增加事件桥、registry 增加 sync_registry_goal_closed、前端投影展示调度/策略摘要。
设计整体自洽，但有两处「设计说得很安全、实现没有兑现」的关键矛盾：worker 前缀允许列表在实现中退化为首 token 匹配；能力绑定任务的 capability 字段在入队时被丢弃，导致 fail-closed 只对测试中手工构造的队列条目生效。此外「opt-in / 默认不变」的文档与默认开启的实现互相矛盾（详见风险 2/4）。
具体改动
按面分类：
- 调度核心：
scheduler/event_driven_dispatch.py(768)、scheduler/resident.py(811)、scheduler/task_lifecycle.py(677)、scheduler/merge.py(327)。 - 统一策略：
policy/decision.py、decision_events.py、engine.py（run/wait/deny + ALLOW/DENY/DEFER/RETRY/BLOCK/CANCEL/ESCALATE，序列化 Decision，transition-only 决策事件）。 - Goal 生命周期：
goals/goal_acceptance.py（grep 独立验证 + absence 语义）、goals/goal_closure.py（RUN/WAIT/CLOSE 三态）、goals/goal_channel_projection.py。 - 运行时：
runtime/checkpoint.py、runtime/replay.py、quota/cost_projection.py、quota/live_decision.py。 - 能力桥接：
capabilities_bridge.py(504)（token 归一化、P1 资格、P2 registry 驱动 CLI 注册、P3 事件/钩子 hub）。 - 可观测：
status/control_plane_observability.py(295) +status_markdown渲染。 - CLI/集成：
starter_scheduler新增 dispatch/resident/merge 三命令、project_lifecycle新增goal-closure、cli.py改为 registry 驱动注册；chat_agent、Claude goal-mode、OpenCode/Pi goal loops、todos.py、bootstrap_command_pack.py、registry.py、rollout_event_log.py均有桥接改动。 - 测试：
tests/control_plane/新增 22 个文件约 5,200 行，含调度、策略、检查点/重放、验收/闭包、能力桥、可观测、todo 变更权威。
关键代码讲解
1. loopx/control_plane/scheduler/resident.py::_command_matches_worker_prefix（P1 安全回归）
该函数声称「Mirrors the original scheduler executor gate」，但与原实现不同：它只比较命令的第一个 token（first == prefix.split(None, 1)[0]），而不是完整前缀 token 序列。实测：
_command_matches_worker_prefix("git push --force origin main", ["git status"])→True_command_matches_worker_prefix("python3 -c 'import os; os.system(\"id\")'", ["python3 examples/x.py"])→True- 原
codex_cli_scheduler._command_matches_allowed_prefix使用command_parts[: len(prefix_parts)] == prefix_parts，多 token 前缀是完整匹配的。
后果：--worker-command-prefix "git status" 会授权任意 git … 命令（包括 push/reset/checkout --force），python3 examples/foo.py 会授权任意 python3 -c …。这是本 PR 核心卖点「显式 worker 执行安全门」的实质性削弱。现有测试只用单 token 前缀（sed），未覆盖多 token 场景。
2. loopx/control_plane/scheduler/event_driven_dispatch.py::enqueue_tasks（P1 能力门失效）
队列条目只写入 schema_version/goal_id/todo_id/status/enqueued_at/enqueued_by，丢弃 required_capabilities 与 capability_binding_ref。claim_next_eligible_task 对无能力字段的条目返回任意 worker 可 claim。实测：经 build_event_driven_dispatch 入队的能力绑定任务（required_capabilities=["issue_fix"] + capability_binding_ref="issue-fix:feasibility_v0"），一个无任何能力声明的 worker 也能成功 claim。测试 test_event_driven_dispatch.py:629/666/747 是入队后手工修改 entries 再断言，绕过了真实入队路径，因此迁移笔记中「带 binding 的任务 fail-closed」在端到端 CLI 路径上不成立。
3. loopx/control_plane/new_architecture.py::master_switch_enabled + scheduler/event_driven_dispatch.py::event_driven_dispatch_enabled（P1 默认行为与文档矛盾）
env 未设置时 master_switch_enabled() 返回 True，dispatch/event-source/merge 全部默认开启；live_decision._attach_unified_policy_decision 默认把 policy_decision 附加到每个 quota should-run 并写 policy_decision 审计事件；OpenCode/Pi goal loop 默认 fire-and-forget 调用 dispatch（claim 任务、写 task_ready/enqueued/dispatched）。而 CLI help、模块 docstring 与 PR 正文反复声称「Opt-in … disabled by default / default unchanged」。test_policy_pilot_wiring.py 的 docstring 写「default behavior is unchanged (no policy_decision key…)」，测试断言却要求默认附加 policy_decision——同一文件自相矛盾，说明披露面没有收敛。
4. loopx/control_plane/quota/live_decision.py::_attach_unified_policy_decision（P1 行为变更未披露）
默认开启后，每次 loopx quota should-run / turn 都会：(a) 在 payload 增加 policy_decision（JSON schema 变化）；(b) 向目标 goal 的 rollout event log 写 policy_decision 事件（持久化写）；(c) 下游 goal-mode/statusline/opencode/pi 的 run gate 改为优先消费该字段。这属于会改变现有自动化与状态量的默认行为变更，未在 PR 正文/help/迁移笔记中统一披露，也未说明旧/新默认与关闭方式的一致性。
5. loopx/todos.py::add_goal_todo / complete_goal_todo（P2，并入披露项）
两个核心命令现在无条件追加 todo_add/todo_complete rollout 事件（只要 runtime root 可解析且非 dry-run），即使 LOOPX_NEW_ARCHITECTURE=0 也写。这与「master switch 关闭即恢复 legacy 路径」的说法不符；同时 update_goal_todo/supersede_goal_todo 未桥接事件，rollout 投影对 reopen/unblock 等更新会过期（只有 add/complete 两态）。
正向路径
以「新 goal 首个任务」为例：todo add 写 markdown + todo_add 事件 → codex-cli-local-scheduler-dispatch --completed-todo-id … 写 todo_complete 事件并同步内存状态 → advance_ready_todo_ids 从 handoff gate/自由 advancement todo 算出 READY → 去重入队 → claim_next_task 加 lease 后 claim → resident 模式经 execute_claimed_task（command+prefix+guard 全过）执行 → finalize_resident_execution 写 task_completed → evaluate_goal_acceptance 验证 → maybe_close_goal 原子发 goal_closure_ready+goal_closed → sync_registry_goal_closed 同步 registry。该路径的单元/集成测试覆盖充分（1204 个 control-plane 测试通过）。
负向路径
- 能力不匹配：设计意图是
eligible_bridgedfail-closed；实际端到端因入队丢字段而失效（风险 3）。 - 多 token 前缀：设计意图是白名单精确匹配；实际首 token 即放行（风险 2）。
- 验收未满足：
goal_acceptance_pending会阻止无 criteria 的后续 tick 静默闭包，这是实现正确的亮点；但 resident 模式 grep 证据无base_dir/pattern，自报ok=True即可满足（风险 6）。 - 执行门跳过：gate-skip 的任务在 finalize 被
transient=False标记为 failed，任务永久卡住（风险 7）。
对主干的风险
阻断项（P1）：
- 基线分支不匹配：PR 仍指向
frontend-control-plane-im-prototype-rfc，但改动是生产 runtime/CLI/quota/todos，owner 已两次 CHANGES_REQUESTED（「这个 PR 似乎应该合往前端那个分支」→「又看了下，没改前端，应该合往 main」）。合入前必须重定向到 main（或作者明确说明该分支作为新架构主线）。 - worker 前缀允许列表安全回归（见关键代码 1）：多 token 前缀退化为首 token，任意同首 token 命令可执行。最小修复：移植原
command_parts[:len(prefix_parts)] == prefix_parts逻辑，并加负例测试（git push --force对git status前缀必须拒绝；python3 -c …对python3 examples/x.py必须拒绝）。 - 能力匹配未端到端接线（见关键代码 2）：入队丢失能力字段，能力绑定任务可被任意 worker claim，capability 边界形同虚设。最小修复：
enqueue_tasks/队列条目携带required_capabilities+capability_binding_ref（含 task_ready/task_enqueued 事件的可审计字段），并新增「经build_event_driven_dispatch入队 → 无能力 worker 必须被拒绝」的端到端测试，替换手工改 entries 的测试。 - 默认开启与文档/披露矛盾（见关键代码 3/4/5）：要么把默认改为真正 opt-in（与 help/docstring 一致），要么在 PR 正文、CLI help、迁移笔记统一披露默认开启、旧/新默认、关闭方式（
LOOPX_NEW_ARCHITECTURE=0），并让todos.py事件桥与决策记录跟随 master switch；同时修正test_policy_pilot_wiring.py自相矛盾的 docstring。
非阻断（P2）：
- M6 maintainability ratchet 仍为红：head 上
test_m6_maintainability_ratchet_has_no_unreviewed_debt失败，新增 unreviewed 债务dependency_debt: event_driven_dispatch -> capabilities.catalog（control_plane 外向依赖）。base 分支本就红（chat_actions.py1590 行超 1500 上限，PR 已通过 baseline 修复），但 PR 同时引入了新的未审依赖。需要添加 reviewed exception（稳定 finding id + reason + retirement plan），或把build_capability_registry()改为注入式 seam，消除 control_plane 对 capabilities 包的 import。 - resident 验收证据不对称：dispatch 路径传
--project做独立 grep 验证；resident CLI 不传 project/base_dir，且 evidence 解析split("=", 2)后ok=True无条件，无 regex/absence 语义。目标可以在 resident 模式靠自报证据闭包。应与 dispatch 共用同一套验收解析与验证路径。 - 安全跳过被永久失败：
finalize_resident_execution对 gate-skip（未执行）任务调用fail_task(transient=False)，任务进入 failed 且不会自动重试，只能手工 requeue；安全门的「跳过」不应消耗任务失败预算。应保持 pending/重新入队，仅对真正尝试且失败的任务标记失败。 - 次要契约/披露漂移：
verify_criteriondocstring 承诺「无明确证据要求时有 manual/snapshot 兜底」但代码未实现；replay_audit_record声称「no task contents」却内嵌完整 checkpoint state snapshot；frontstage 投影goal_channel_projection._compact_quota暴露scheduler_reset_token（控制 token，建议仅保留 rrule/cadence 标量）；--dry-run参数在goal-closure中只回显不参与逻辑。
其余残留风险：12,685 行新增且无 status-check rollup，依赖本地 pytest 自证；队列 claim 为 FIFO，Decision.priority 未接入队列排序；OpenCode/Pi 循环 fire-and-forget dispatch 只 claim 不 complete，lease 过期后会产生重复 claim/执行风险（在原型分支可接受，合 main 前需明确 worker 归属）；迁移笔记自认 90+ 处硬编码钩子未迁移，属阶段边界。
验证矩阵（exact head 实测）：
| 场景 | 命令 | 状态 | 结果 |
|---|---|---|---|
| control-plane 全量 | pytest tests/control_plane |
PASS | 1204 passed |
| M6 门 | pytest tests/control_plane/test_m6_quality_gates.py |
FAIL | 1 failed（unreviewed dependency_debt） |
| Claude policy | pytest tests/test_claude_goal_policy.py |
PASS | 3 passed |
| OpenCode/Pi runtime | node --test tests/opencode_goal_bridge_runtime.test.mjs tests/pi_goal_loop_runtime.test.mjs |
PASS | 56 passed |
| CLI 注册 smoke | loopx codex-cli-local-scheduler-{dispatch,resident,merge} --help |
PASS | 三命令可注册 |
| 前缀门负例 | 聚焦探针 | FAIL（期望拒绝） | git push --force 通过 git status 前缀 |
| 能力传播端到端 | 聚焦探针 | FAIL（期望拒绝） | 无能力 worker 可 claim 绑定任务 |
| 默认开关 | 聚焦探针 | 与文档矛盾 | env 未设时 dispatch/event_source/merge=True |
| base 对照 | M6 on ece816a2 |
FAIL（pre-existing） | chat_actions 1590>1500；head 变为新 dependency 债务 |
我的整体评价
这是一次方向正确、测试投入很大的架构重构：typed lifecycle/Decision、幂等事件、原子 checkpoint、闭环 acceptance→closure 的设计都明显优于现状，1204 个 control-plane 测试 + 56 个 JS 测试通过，代码组织也遵守了 bounded context。但本 PR 的核心卖点是「显式 worker 安全门 + capability fail-closed」，而这两条链路在实现层面都失效（前缀首 token 匹配、入队丢能力字段），且「opt-in」文档与默认开启的实现矛盾、基线仍指向 owner 明确要求改掉的原型分支。code_volume 判定：necessary（重构规模与面匹配），但需要补丁级修整并补齐 M6 未审债务。建议：先重定向 base 到 main，修复 3 个 P1（前缀门、能力传播、默认/披露），再补 5-8 的 P2，重新在 exact head 上跑一遍上述验证矩阵后再合并。当前结论：REQUEST_CHANGES。
English Verdict
Verdict: REQUEST_CHANGES
Exact head: 3200@f6549a9e9f1f21cb1600de2765b461840c26a196
Key findings:
- The PR still targets the
frontend-control-plane-im-prototype-rfcbranch although the owner twice requested a retarget tomain; the diff contains production runtime/CLI/quota/todos changes. - Worker execution prefix allow-list regression: multi-token prefixes match on the first token only, so
--worker-command-prefix "git status"permitsgit push --force, andpython3 examples/x.pypermitspython3 -c .... The original matcher compares the full prefix token sequence. - Capability matching is not wired end-to-end:
enqueue_tasksdropsrequired_capabilities/capability_binding_ref, so capability-bound tasks are claimable by any worker via the real CLI path; tests mutate entries manually after enqueue and do not cover the real path. - The new architecture is ON by default (master switch) while the PR body, CLI help and docstrings claim opt-in / disabled by default / unchanged:
quota should-rungains and recordspolicy_decisionby default, OpenCode/Pi loops dispatch by default, andtodo add/todo completewrite rollout events even with the master switch off.
Validation: At the exact head, pytest tests/control_plane = 1204 passed / 1 failed (M6 maintainability ratchet: new unreviewed event_driven_dispatch -> capabilities.catalog dependency debt; base was already red on chat_actions), tests/test_claude_goal_policy.py = 3 passed, JS runtime tests = 56 passed, CLI registration smoke passed; focused probes reproduced the prefix-gate bypass and the capability-drop claim; default flags confirmed ON with no env.
Minimum repair: retarget to main; port the original full-token prefix matcher and add negative tests; carry capability fields on queue entries and add an end-to-end claim-rejection test; align default-on behavior with documentation (or make it truly opt-in) and update the contradictory pilot-wiring test docstring; then re-run the validation matrix on the new exact head.
huangruiteng
added
the
direction/operator-surface-im
2 participants
refactor: migrate control plane to event-driven task scheduling
Summary
This PR refactors the LoopX Control Plane from the legacy heartbeat-driven execution model to an event-driven task scheduling model.
The core change is:
The new architecture treats Task, Event, Scheduler, Lease, and Worker as first-class concepts, while keeping Goal Acceptance and Goal Closure as explicit control-plane stages.
Why
The legacy Control Plane relies heavily on heartbeat-driven polling.
Each heartbeat wakes the system and re-evaluates multiple independent layers:
This creates several structural problems:
execution is driven by periodic polling rather than state changes
quota, capability, scope, and scheduler decisions are distributed across multiple layers
decision values have accumulated into many different states/actions
state is spread across multiple projections and ledgers
scheduler has limited task lifecycle management
there is no unified claim/lease/retry lifecycle
crash recovery and zombie execution recovery are difficult to reason about
observability is distributed across multiple sources
The refactor moves the control plane toward an event-driven Task OS model.
New Runtime Model
The heartbeat is no longer the primary execution driver. It is reduced to an event source/timer mechanism where periodic observation is actually required.
Key Changes
1. Event-driven task dispatch
Introduce:
scheduler/event_driven_dispatch.pyscheduler/resident.pyTask readiness produces events which are placed into the Task Queue.
The scheduler consumes ready work incrementally instead of repeatedly scanning the entire control plane on every heartbeat.
Idle systems can remain idle instead of repeatedly waking and recomputing the entire execution state.
2. Task lifecycle with lease and retry
Introduce:
scheduler/task_lifecycle.pyTasks now have an explicit lifecycle:
Failure and recovery paths are explicitly modeled:
Workers claim tasks using:
lease_untilattemptclaimed_bycapability requirements
idempotent task identity
This provides a foundation for crash recovery, retry, duplicate prevention, and multi-worker execution.
3. Unified Policy Engine
Introduce:
policy/decision.pypolicy/engine.pypolicy/decision_events.pyThe legacy quota/capability/scope decisions are consolidated behind:
The primary decision space becomes:
while richer action semantics are represented separately:
Decisions are also serializable through
Decision.to_dict()/from_dict()for event/audit integration.4. Event Store as source of truth
The new architecture treats append-only events as the authoritative record.
State is derived through projections rather than maintained as multiple independent sources of truth.
Relevant changes include:
append-only rollout events
state projection
checkpoint/replay support
generation-aware task identity
Task identities use generation information such as:
to reduce replay and duplicate-execution collisions.
5. Explicit worker execution safety gates
Resident workers do not automatically execute arbitrary claimed work.
Execution requires the expected control-plane gates:
If the required execution gates are not satisfied, the task is not automatically executed.
This makes the execution boundary explicit rather than relying on implicit routing through heartbeat/runtime layers.
6. Resident scheduler reconciliation
The resident scheduler performs reconciliation before normal dispatch.
This covers:
expired leases
zombie task recovery
retry advancement
queue state reconciliation
The scheduler therefore has a deterministic recovery path rather than depending on another heartbeat cycle to eventually rediscover stale work.
7. Goal Acceptance and Closure
Introduce:
goals/goal_acceptance.pygoals/goal_closure.pyTask completion is intentionally kept separate from Goal completion.
The control plane can therefore distinguish:
Goal closure is evaluated after acceptance and remaining-work checks rather than being inferred from an individual Todo completion.
8. Control-plane observability
Introduce:
status/control_plane_observability.pyA unified read-only snapshot exposes six major categories:
This is also exposed through:
so the runtime state can be inspected without modifying control-plane state.
Architecture Comparison
Conceptual Simplification
The refactor also reduces the number of concepts directly exposed to the scheduling layer.
The legacy model contains many independent concepts around:
The new model consolidates these around:
The goal is not to remove domain semantics, but to establish clearer ownership boundaries between:
state
policy
scheduling
execution
acceptance
closure
Tests
New and updated tests cover:
task lifecycle
lease expiration
retry behavior
event-driven dispatch
resident scheduler reconciliation
policy decisions
policy integration
rich decision serialization
goal acceptance
goal closure
heartbeat event source
checkpoint/replay
cost projection
capability bridge
control-plane observability
CLI integration
Core validation:
Scope
This PR focuses on:
and the integration points required to move the existing runtime toward the new control-plane model.
The intent is to establish the new scheduling/control-plane primitives while preserving the existing Goal/Agent integration points where migration is still required.
Review Focus
The main areas for review are:
Event Store / projection ownership
Task lifecycle and lease semantics
PolicyEngine decision semantics
Resident scheduler reconciliation
Worker execution safety gates
Goal acceptance / closure semantics
Compatibility between the new event-driven path and existing runtime integrations
Design Principle
The architectural direction can be summarized as:
The intended end state is: