Skip to content

refactor(turn): defer DSH builders and process helpers to selected paths - #8185

Open
cocolord wants to merge 11 commits into
mainfrom
codex/small-kernel-turn-dsh-loading-20261106
Open

cocolord wants to merge 11 commits into
mainfrom
codex/small-kernel-turn-dsh-loading-20261106

Conversation

@cocolord

@cocolord cocolord commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

The preview command imports process helpers before execution, and generic-cli imports DSH builders before choosing its host. This proposal resolves the existing DSH exports at host selection and process helpers at execution/cleanup or explicit import. The unchanged preview oracle passes. Named/star imports, original function identity, static aliases, dir() and public patch points remain compatible; TypeScript retains Host lifetime and authority decisions.

This is a bounded loading slice of the accepted small-kernel RFC, following #8180. Preview loading is 338 → 337; generic-cli execution is 538 → 533 because of DSH deferral. The process-helper repair itself leaves executing loading at 533. The complete matched-bytecode comparison below demonstrates no material timing or CPU benefit, so the proposal makes a loading/compatibility claim and redirects subsequent characterization toward a larger evidenced cost boundary.

At immutable baseline 546c08c0d3f22358ef09b13e69f7edaa806121b8 and measured proposal 0df5ad7921faab3b9ab8ec809f2fd71060692cce, a new frozen campaign completes six source/wheel groups with 30 interleaved pairs each: 360 accepted observations in 948.573 seconds, below the unchanged 1,200-second sampling cap. All four environments use CPython 3.13.5, Node 26.0.0, macOS arm64 and real Node-owned SQLite 3.53.1, identical 57 dependency versions, verified source/wheel provenance, and valid prepared bytecode for stdlib, dependencies and all 1,421 LoopX Python sources. Post-experiment inventories and frozen harness hashes match. Every invocation uses fresh Python; cold/warm means retired/retained owned broker. Disk cache is uncontrolled.

Operation Broker Baseline median / p95 seconds Proposal median / p95 seconds
Source selected help absent 0.136 / 0.193 0.138 / 0.168
Source SQLite task cold 3.579 / 5.997 3.617 / 9.234
Source SQLite task warm 2.673 / 3.019 2.657 / 2.979
Wheel selected help absent 0.087 / 0.107 0.083 / 0.116
Wheel SQLite task cold 3.722 / 4.266 3.663 / 4.430
Wheel SQLite task warm 2.707 / 3.111 2.797 / 3.156

Host one-minute load ranges from 8.35 to 88.73. All tails remain retained; the source cold p95 difference is not attributed to this code. Arm medians and tails have mixed directions, with similar task CPU. The bilingual RFC's new E8 checkpoint reports paired differences separately from differences of arm medians, target/waited-child and owned broker CPU/RSS, modules/open counts, covered socket/output bytes and separate setup/validation/instrumentation/profile costs. Repeated timing is separate from eight parent/actual-Node captures using existing performance-diagnosis recipes. Workspace capture, journal checkpoints, Git probes and quota preparation appear among the Python observations; profiles identify characterization candidates without proving root cause or savings. Full process-tree/native/thread/context/model and complete authoring costs remain unknown.

The earlier E7 campaign remains capped, incomplete and cache-confounded: 499 observations, eight complete source groups, wheel JSON-version nine pairs plus one unmatched observation, and source LoopX cache counts 7 versus 1,241. It is neither resumed past its cap nor relabeled as a pass. JSON version's full export/parser closure remains a plausible separate R0 characterization target; this PR does not change it or expand the helper refactor.

The immutable comparison's four source/wheel × revision variants pass ordinary committed replay, interruption after typed-result persistence, original-journal recovery and committed replay. Independent validators accept identical squared-integer artifact bytes; each accepted fixture has one host invocation/attempt, writeback and debit, and replay repeats no effect. Removing the final square rejects all four with one dispatch and zero writebacks/debits. Every owned broker has OS-exit readback. At final delivery head fcc465d9ed253b1afaab036f658711b71cc3addd, fresh source and newly built/installed monolithic-wheel fixtures repeat completion, interruption, original-journal recovery, replay, rejection and broker-exit validation successfully. This final semantic validation is separate from the immutable cost study.

Validation and adoption limits:

  • Final proposal: fcc465d9ed253b1afaab036f658711b71cc3addd; verified target/base: main at 8eb707d47a4d5529e316bcc30e042840cca3ad80. Latest main, including fix(chat): distinguish settled command failures during unfinished-work recovery #8194, is integrated; its later integration is outside the frozen cost comparison. Only nine intended tracked paths are proposed; generated lockfiles, fixtures, harnesses, journals, raw profiles and logs remain excluded.
  • At this unchanged final head, standard source premerge passes six direct and 14 selected checks, with no failed/advisory checks or manual holds: uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main. Coverage includes Python compilation, import/module ceilings, CLI budgets and dispatch, semantic inventory, canary selection/execution and public/private scans. Staged and committed bilingual RFC checks pass. The vocabulary advisory finds no supported new carriers; it does not certify semantic completeness. Together with the exact-head real source/wheel recovery and rejection cases, this qualifies the bounded loading slice's validation, not performance benefit or merge authority.
  • Historical validation remains historical: 619 passed / seven Windows Job Object skips across Turn/DSH/operator/import/Host suites; Git identity/probe selection 19 passed. Earlier preview/Git-hook failures remain retained. Owner-scoped mypy reports 29 identical diagnostics at base/proposal; full configured mypy is untested and no failure is waived. test(chat): fix pytest fixture lint after #8148 #8188 repaired the earlier Chat F811 collisions.
  • Remote CI is held at this exact head. Core CI fails lint with F841 in tests/control_plane/test_goal_create_completion.py:184; the targeted local lint reproduces it. Ark adapter CI exits before tests because its command names the missing tests/test_auto_research_artifact_receipt.py. The failing fixture and workflow are byte-identical at verified target 8eb707d47 and this PR head; that test path is also absent at the target. Dependent Stage 2C/checks/pytest/merge-gate lanes fail or skip after their upstream hold. Other build/adapter lanes were still running at readback. These failures remain active, separate from the passing local premerge and real-path experiment; no waiver or green-CI claim is made.
  • Independent exact-head review, complete required CI and merge readiness remain necessary for adoption. No runtime merge authority is inferred. Full R0/V10, R1/minimal selection, D2/D5/D6, I3/D3, normal hard-lease/context/model parity and split installation remain open. refactor(chat): move public status owner with identity-compatible imports #8011 remains the separate public-status proposal.

The bounded refactor pass is applied at three existing Python adapter seams. Larger full-parser/workspace/journal/bridge work is deferred to a focused, evidence-backed successor. No new decision owner/framework, installed-host upgrade, model job, provider/default/permission change or runtime self-merge is introduced.

中文:本提案在既有 Python adapter 按实际宿主/执行路径延迟加载,修复 preview oracle,保留公开导出、类型、身份和 patch 点,TypeScript lifetime/authority 不迁移。preview 338 → 337,generic-cli 因 DSH 延迟加载从 538 → 533。新对照在匹配字节码及相同依赖下完成 source/wheel 六组各 30 对,共 360 次观察,采样 948.573 秒,未超原定 20 分钟上限;中位数、尾部和 CPU 未显示明显收益。宿主负载较高,全部尾部保留,不将源码 cold p95 差异归因于代码。双语 RFC 的 E8 保留配对统计、覆盖成本与未知项;E7 的 499 次超时/缓存混杂实验继续保留为未取得资格。四个不可变版本组合及最终 fcc465d9e 的源码/新安装单体 wheel 均通过 completion、中断、原 journal 恢复、无重复 effect 的 replay、独立错误产物拒绝和 broker OS 退出。最新 main 已集成,本地 premerge 六个直接与 14 个选择检查通过。远端 CI 当前仍失败:goal-create fixture 的 F841 及 Ark workflow 引用缺失测试文件,两处均与目标 main 字节一致;依赖 gate 随上游失败/跳过。失败未获豁免,其他构建/adapter lane 尚在运行。历史 619 通过/七项 Windows 跳过、Git 19 通过、mypy 29 错误保持原结论。后续优先刻画完整 parser/export 或 workspace/journal/bridge 成本,停止继续扩展收益未明的 helper。完整小内核及其性能、hard-lease、上下文、模型、拆包安装验收仍未完成,独立评审和采纳门仍保留。

cocolord and others added 2 commits November 6, 2026 21:02
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
cocolord and others added 3 commits November 6, 2026 22:03
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
@cocolord cocolord changed the title refactor(turn): defer DSH loading behind host selection refactor(turn): defer DSH builders and process helpers to selected paths Nov 6, 2026
cocolord and others added 6 commits November 6, 2026 22:51
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
…rn-dsh-loading-20261106

Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <traecli@bytedance.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max.

APPROVE:未发现本 PR 引入的阻塞问题。三个叶模块的加载与兼容接缝有独立证据;当前完整 preview 的继承失败保留为本地验证与合并 hold。

动机 / Motivation

调用 generic-cli、DSH 或只读 Git 诊断的现有 CLI 用户,以及复用这些模块公开函数的 Python 调用方。
原来导入 Turn 执行模块就加载未选择的 DSH builder,导入 bridge/Git 叶模块就加载进程清理实现;现在三个叶模块按实际选择或显式 import 解析原 owner。
已独立证明三个叶模块的冷导入闭包缩小,公开名称、函数签名和 patch 点保持兼容;没有证明执行提速。
完整 preview 仍经未改动的 contract.py 加载 process_runtime;完整 R0/V10、minimal、模型/hard-lease 与拆包安装资格仍开放。
无新增 preset、provider、权限、状态协议、全局安装、模型调用或性能收益声明。这里的“叶模块”指直接被复用的适配模块;它自己的加载闭包变小,不能推出整个预览命令都已隔离。

改动思路 / Approach

三个局部 facade 复用原函数并保留真实 patch 消费者,成本有边界;不引入通用 loader、第二个进程 owner 或新权限判断。
本 PR 仅采纳三个叶模块的加载/兼容接缝及双语负面证据 checkpoint;完整 preview 与性能资格保持独立验证 hold。

原来的 DSH builder 和 process_runtime 继续负责实际工作。Python 模块的 __getattr__ 仅延迟解析已有函数,TYPE_CHECKING alias 保留静态调用类型;执行点通过模块属性访问,所以已有 monkeypatch 仍能接管该调用。Host 生命周期、准入、journal、扣额和重放仍由原来的 native TypeScript/Turn owner 决定。单纯改为函数内 import 会绕过现有适配模块 patch 点;引入共享 loader 则给三个小接缝增加新的维护层。保留本地 facade 是有实际消费者依据的取舍。

具体改动 / Changes

评审完整范围为 9 文件、+711/-11:三个 Python runtime 适配模块、四个现有测试文件和两份 RFC 的证据附录。大部分新增是双语历史测量及负面结论;没有新增状态格式、服务、命令或权限。独立 fresh-process 对照确认 41/12/18/6 个公开 export、签名与 dir() 名称逐一一致,未知属性仍拒绝。单独导入 Turn 执行模块移出七个当前依赖模块;bridge/Git 叶模块各移出 extensions 包及 process_runtime。该 census 与历史“实际执行少五个模块”处于不同调用边界,不能混作同一性能证据。

关键代码讲解

  • turn_run_once.__getattr__ 仅接受两个既有 builder 名称、返回并缓存原函数;第 843/856 行在 DSH operation/普通分支读取同一模块属性。现有 DSH fresh/resume 夹具观察到四次被 patch 的调用,generic-cli 冷导入不再选择 DSH。
  • GuardedHostBridge.send/cancel 在 ready 检查后才取写入函数,取消仍保留 requested/cancelled 顺序和 cleanup-unconfirmed 拒绝。真实 Unicode 发送、背压、deadline、解析失败和清理路径验证了消费者,而非只检查符号存在。
  • doctor_git._retire_owned_probe 在原 try 内解析终止函数,仍以 0.2 秒 grace 清理自有 Git group;超时回答 unknown,取消继续传播。成功 Git 读取不需加载 helper;真实 fsmonitor helper 及清理后抛错用例维持原边界。

规范依据为 docs/architecture/rfcs/monorepo-distribution-split-v0.md,spec_revision 8eb707d47a4d5529e316bcc30e042840cca3ad80;判据来自修改前已接受文本,新增 E7/E8 不能自行认证实现。I2:复用单一执行/状态 owner;I7:三个被改叶模块的选择/使用边界通过,import 不授予激活;I9:双语记录明确不证明提速。V3:本次受影响调用的输入、公开 API、错误与恢复保持兼容。V1:完整 preview 和双 preset 资格 deferred,下面保留当前失败;V2、V4:正常 hard-lease/模型宿主与双 preset 资格 deferred;V10:完整开销/成本资格 deferred。R1/minimal 与拆包安装不由这次局部源码变更关闭。

对主干的风险 / Risks

最强反例是三个新叶模块测试通过,但真正的完整 preview 仍加载进程实现。我在不可变实际 base 8eb707d47a4d5529e316bcc30e042840cca3ad80 和 head fcc465d9ed253b1afaab036f658711b71cc3addd 跑相同六模块选择:base 277 PASS / 1 FAIL / 1 SKIP,head 283 PASS / 1 FAIL / 1 SKIP。唯一失败均为 test_turn_preview_import_does_not_select_report_delivery_or_lark_hosts,实际集合仍含 loopx.extensions.process_runtime。失败用例 AST 相同,独立子进程的完整错误文本和 exit 1 相同;head import trace 为 turn_decision -> status -> contract.py:39 -> process_runtime,这三个因果模块在 base/head blob 完全相同。故不把继承失败归为本 PR 新回归,也不把它改写为通过。三个新叶模块 oracle、公开 export 对照及实际 DSH/bridge/Git/native File/SQLite 完成、中断和重放路径提供了独立 changed-invariant 证据。

本地严格 change-quality receipt 如实记录 FAIL,所需六模块 aggregate 仍为失败;本次不降级该检查、不豁免该 hold。现有完整 preview owner 需另行修复/资格验证并重跑上述测试。主干所需 gate 与 merge readiness、权限仍独立生效。历史 619-case 和 E8 是作者提供的旧环境观察,本次没有复跑冻结实验或将其当作当前通过凭证。PR 已提供的红 CI 只保留诊断;按当前策略未查询或等待远端 CI。

当前完整配置 mypy 在两端都 PASS(17 个配置根),并非全 Python 树声明;七个修改 Python 文件的 Ruff、双语 diff、diff advisory/完整 semantic smoke、公开边界和 diff hygiene 通过。macOS 留下一项平台 skip,未独立验证 Windows。无 TS 文件改动,未重复完整 TS suite。初次 scanner/默认 runtime 路由调用的操作错误保留原日志,随后通过正确的 repository-native smoke 和显式路由验证;它们不构成产品通过证据。

语义与 CI 对齐

新增名称集合只是本地 Python export dispatch,复用现有业务词汇和原函数,没有新 enum/state owner、领域专用强制指令或隐藏 default-off 激活。DSH 选择仍使用既有输入,native 权限、required context、quota 和 journal 语义未移到这些 facade。标准 premerge 的 6 个 direct check 和 14 个 selected check 全部通过、无 manual hold;整体仍为 quality_invalid_receipt,仅有严格 receipt 的一项 policy failure,因此 merge_gate_passed=false。此结果不能被公开 review 的代码判断替代。

我的整体评价 / Verdict

这是 justified_increment:局部加载分离及真实 patch/API 兼容有正面价值,尚未证明 latency、CPU 或长程收益。long_horizon=preserved:原完成/中断/重放 owner 与效果保留;user_experience=preserved:既有命令、结果、错误和恢复无需新配置或确认轮次。turn_run_once、host_bridge_transport、doctor_git 只改变 Python import 时机;未改变任何页面、消息展示或可见控件。 未验证范围为 Windows、模型宿主、已安装拆包与完整成本;无页面改动,不制造截图门槛。

兼容层有当前调用依据,保留三处本地实现;后续应停止为未证明提速扩展 helper,转向已识别的现有完整 parser/export 或 journal/bridge owner,并在独立授权下做匹配对照。公开代码判断为 APPROVE;严格本地 qualification/merge hold 仍保留,不据此执行合并或安装。

English verdict: APPROVE for the verified leaf-loading and compatibility change; retain the base-equivalent preview validation and merge hold. No performance qualification or merge authorization is implied.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants