Skip to content

test: isolate machine CLI smokes from model context transport - #8119

Merged
huangruiteng merged 1 commit into
mainfrom
codex/machine-smoke-context-20261105
Nov 6, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/machine-smoke-context-20261105

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Two machine CLI smokes inherited LOOPX_CONTEXT_DELIVERY=on and parsed a model context-navigation response as inline Core JSON. Required premerge then failed on missing autonomous_replan_obligation and interaction_contract fields before exercising its intended assertions.

Use the existing LOOPX_CONTEXT_DELIVERY=off contract only around each machine call. In-process calls restore the parent environment; subprocess calls receive a copied environment. All original smoke assertions and timeouts remain intact. Production transport defaults and Core admission policy are unchanged.

  • Basis: reproduced engineering-validation defect; intended base main at 3c7e0935fd6697291c5c432e5267816b4a667cfa.
  • Observable result: the new regression fails in all 18 parent-on cases before the fix, then passes across all 36 on/off cases. Both previously failing required smokes pass in full.

Author Declaration

  • Written by: model_agent, OpenAI Codex (GPT-6 family).
  • Implemented against: the existing machine CLI output contract and docs/development/good-smokes.md at the base revision above. This PR's reproduced defect is the scope basis; no new RFC or policy is introduced.
Criterion Disposition Path / evidence
Preserve independent original semantic assertions implemented Both existing smoke consumers; six complete smoke executions
Preserve parent transport setting and real Core gates implemented tests/control_plane/test_canary_cli_transport.py: File/SQLite ready, user-gate and zero-quota cases, parent on/off, three call paths
Reuse existing fixture authority and transport controls implemented Existing canonical fixture import and per-call environment override

Scope And Continuation

Complete within the machine-consumer repair scope. The two existing helpers receive the same narrow isolation; no new abstraction, persistent setting, production owner or UI is added. The existing canary CLI test module supplies the regression. Adjacent cleanup is unnecessary for this boundary.

Independent exact-head review is pending. The optional raw maintainability ratchet still fails on existing debt, with identical base/candidate output and no magnitude regression. Native premerge classifies that failure as inherited advisory debt; the separate change-quality receipt remains fail because independent exact-scope baseline attribution is not qualified. This proposal does not waive that hold or claim merge readiness.

Validation

  • Tested revision: 3247dc442231b063426a215d068a50402bb0a9bb (validation ran on the identical pre-commit source bytes).
  • Run state: finished. Input classes: synthetic.
Check kind Result Evidence / limitation
regression_parity passed On base 3c7e0935 with new regression and original helpers: 18 failed / 24 passed; after isolation: 42 passed. All 18 failures are the parent-on transport mismatch.
real_entrypoint passed Six full original smoke runs: parent on/off, autonomous in-process / subprocess, and refresh-state write-correctness wrapper. Original assertions retained.
real_backend passed Isolated real File and SQLite canonical authority fixtures. Ready permits work; scoped user gate and zero quota still deny it. Parent context mode survives each call.
static passed Configured Ruff and mypy (17 configured source files), all three changed Python files, whitespace, bilingual RFC gate and semantic-vocabulary checks. No paired RFC changed.
integration passed Standard diff-selected canary premerge: five direct checks, nine selected checks, zero blocking failures and one inherited ratchet advisory. Both original required failures are restored.
static failed Raw control-plane-maintainability-ratchet-smoke.py: existing oversized interaction-contract function on both base and candidate; two reviewed compatibility facades, zero magnitude regressions. Failure is retained.
manual blocked Exact-scope change-quality receipt cqr_a5e13762a3685e87395a records the optional failed validator without an independently qualified attribution; independent review/qualification remains.

Coverage is limited to the changed machine-consumer boundary. The full Python suite, installed App/Host, live model, Windows and Lark journeys were not run; no production or installed behavior qualification is claimed.

Frontend / Visual Evidence

UI impact: none. Before/after screenshots and viewport review: N/A; only engineering smoke/test calls change. Source data: synthetic fixtures.

Type of Change

  • Bug fix
  • Test update

LoopX Area

  • Build, packaging, installer, or CI
  • Host or runtime integration

Technical Direction

Engineering caller validation within the existing roadmap S8/S12 boundary; no migration or provider-promotion claim.

Shared-authority RFC fixture impact

N/A: no production state schema, authority routing or migration behavior changes. The File/SQLite fixtures exercise existing contracts.

Boundary Checklist

  • Diff and body contain no private operating state, credentials, raw traces, internal links or local paths.
  • Existing maintainer-owned benchmark work is not duplicated.
  • Scope is the reproduced machine-consumer defect.
  • No paired RFC changed; bilingual check passed.
  • UI impact is none.
  • Commit includes DCO sign-off.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

审查对象:8119@3247dc442231b063426a215d068a50402bb0a9bb。未发现阻塞问题;以下结论只覆盖机器 CLI smoke 的隔离修复。

动机 / Motivation

启用上下文呈现的开发环境会把 CLI 输出交给 Agent 阅读,但两条既有 smoke 需要完整的机器 JSON。原有读者直接索引 Core 字段,遇到导航封装便在真正的重规划、共享运行时断言之前失败。这个故障会制造误报和重复排查,也容易诱使操作者关闭全局偏好。独立运行原始命令确认:不可变基线在父环境 on 时两条都失败,off 时两条都通过;当前 head 在两种父环境下均通过。

正常状态允许交付,用户 gate 与零额度都继续拒绝;每格还检查父环境没有被覆盖。

没有使用真实用户 Goal、安装宿主配置、调用付费模型或运行冻结 benchmark。

改动思路 / Approach

保留原有测试入口和语义断言,把完整 JSON 的要求放回每次机器调用。进程内调用临时覆盖环境,退出作用域后恢复;子进程使用复制的环境字典,覆盖值只属于该次调用。父进程的交互偏好与生产上下文默认值保持原样,无需增加操作者确认、重新填写参数或先修改设备设置。

这里复用原 CLI、原 File/SQLite authority fixture 和原 transport 测试文件。通用新包装层会增加导入与归属成本;全局 off 会扩大作用范围,另造 smoke 则重复原有断言。当前三文件边界能直接修复已复现的调用者故障,不需要先完成相邻生产功能或上层长程目标。

相同原始 smoke 命令 基线 当前 head
父环境 on,两条 smoke 2 个失败,读不到预期 Core 字段 2 个通过
父环境 off,两条 smoke 2 个通过 2 个通过

表中是本次实际运行结果;它不代表模型采纳、长期收益或真实费用对照。

具体改动 / Changes

基线为 3c7e0935fd6697291c5c432e5267816b4a667cfa,完整差异为 3 文件 +47/-2,没有生产运行时、新命令或持久协议变化。规范依据 docs/reference/required-work-context.md,spec_revision 3c7e093:机器读者需要解析 Core JSON 时,应逐次显式关闭呈现,保留原 registry、Goal、Agent 和 Turn。docs/development/good-smokes.md 的独立语义断言与复用既有 fixture 要求同样适用。

  • machine-core-reader:implemented。正常和错误调用都采用局部环境隔离;原参数、超时和返回码检查没有被删去。完整原始 smoke 的 base/head、on/off 对照验证了实际消费者。
  • durable-smoke-boundary:implemented。在原 transport 测试文件中复用 authority fixture,检查 3 个消费者、2 个父环境、2 个 provider、3 种 gate,共 36 格;先前六项 wire/capture 测试继续保留。

autonomous run_cli 的进程内分支使用 patch.dict 限定覆盖寿命,子进程分支传入复制的环境;对应错误 helper 同步处理。两者仍要求原来的正确或非零退出码,JSON 解析与后续重规划断言继续执行,避免用“存在封装”代替原本的产品语义。

shared-runtime run_cli 保留既有 PYTHONPATH、共享 runtime 路由和调用工作目录,只在原子进程环境中加入 off;失败 helper 也保持原来的非零与 JSON 检查。新增矩阵 实际加载原 smoke helper,覆盖进程内、autonomous 子进程、shared-runtime 子进程,而不是再实现一套运输逻辑。正常状态允许交付,用户 gate 与零额度都继续拒绝;每格还检查父环境没有被覆盖。

覆盖扫描检查了这两个 helper、既有 canary_harness 与原 transport 测试,未发现重复的消费者/gate 矩阵。原 smoke 分别为 456/661 行;后者超过指导性的大小信号,但此次只加两个环境项,没有增加一个大型临时构造器。保留两个独立语义边界是有依据的,本 PR 无需顺带做全脚本重构。

对主干的风险 / Risks

最强反例是“测试变绿,但真实用户 gate 被绕过,或者 off 泄漏到父进程”。36 格矩阵针对这些方向分别检查,完整原始 smoke 也在两种父环境下运行,原重规划和共享投影断言没有简化。当前 42 项 transport 测试通过,改动 Python Ruff、编译、差异和公共边界检查通过。standard premerge 在补齐所需 npm 依赖后返回 merge_gate_passed=true;最初缺依赖引起的语义检查失败保留为验证环境问题,修复后重跑通过。

原始 maintainability ratchet 仍为 FAIL:独立 base/head 同一命令的完整输出逐字节相同,唯一未审阅项位于未修改的 interaction_contract::_admitted_interaction_next_cli_actions,没有 magnitude regression。native premerge 将其保留为 inherited advisory;本次 CQR 以精确范围、相同工作负载及环境摘要记录独立归因,原始失败没有改写为通过,也没有调低阈值。当前 CQR 已验证有效。

父环境隔离不会证明实际交互 Agent 在上下文 on 时正确阅读;这需要相邻生产 transport 的证据,不能由机器 smoke 的 off 结果替代。按照当前 no-wait 策略未查询、轮询或等待远端 CI。没有使用真实用户 Goal、安装宿主配置、调用付费模型或运行冻结 benchmark。

我的整体评价 / Verdict

该 PR 完成了一个已复现且可独立验收的工程修复:让原有语义检查在上下文开启的环境里继续运行,同时保留真实拒绝与调用者偏好。它复用既有 owner 和 fixture,没有增加另一套判定状态,也不需要另开后续任务证明这次窄修复。仓库经验 #5944 关于“第一个有用结果”的建议在这里体现为原始 smoke 完整结束,经验投递本身没有被当作结论或效用证明。

视觉审查不适用:三个改动文件只涉及 Python 测试运输和断言,没有呈现、CSS、文案或 UI 调用路径变化。已安装 App、模型输入保留、真实 provider 与上层长程效果均未由此验证。批准结论与合并权限、GitHub 原始 reviewDecision 及独立合并准备检查分开;本轮没有执行合并。

English verdict: APPROVE this exact head. The original machine-reader regressions are repaired without weakening gate assertions or changing the parent preference. The unchanged raw ratchet failure remains explicitly attributed; installed-host adoption and long-horizon benefit are unverified.

@huangruiteng
huangruiteng merged commit 586ae28 into main Nov 6, 2026
25 of 37 checks passed
@huangruiteng
huangruiteng deleted the codex/machine-smoke-context-20261105 branch November 6, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants