Skip to content

feat(research): carry scoped yardstick decisions across work turns - #8202

Merged
huangruiteng merged 3 commits into
mainfrom
codex/yardstick-current-work-20261107
Nov 7, 2026
Merged

huangruiteng merged 3 commits into
mainfrom
codex/yardstick-current-work-20261107

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

An agent can discover that its local screening metric disagrees with the accepted outcome, yet lose the scope and reason for that decision on the next ordinary work Turn. This implements the opt-in M1 slice of the target/yardstick RFC: capture a current target, explicitly scoped authoritative and proxy measurements, counterevidence and the next bounded method through the existing Explore result attachment. Subsequent work on the same Todo receives a short cue and a command to read the complete basis.

The record is an agent declaration, not acceptance, permission or a new execution gate. An uncalibrated proxy cannot become an independent veto over an informative authorized check. Existing correctness, authorization and budget constraints remain binding.

  • Intended base: main.
  • No separate planner, journal, per-Turn form or mandatory replan.
  • Existing settlement, Explore history, evidence references and Todo identity own the continuation.

Author Declaration

Written by: model_agent, OpenAI Codex, with a separate configuration implementation agent.

Implemented against

Specification: Joint Target, Yardstick and Stride Decisions, sections 5, 7–11; this PR refines both language versions and records the M1 implementation boundary.

Criterion Disposition Implementation / decisive evidence
One typed owner; ordinary Turn and existing replan compatibility implemented work_items/yardstick_decision.ts; existing attachment and Turn context; source CLI journey
Same-work binding, replay and delayed-result continuity implemented settlement-minted binding, predecessor validation, existing append CAS, committed-time ordering; typed counterexamples and SQLite interruption/replay
Independent default-off setting and historical readability implemented existing cadence configuration, sparse API, CLI and capability editor; off-path and inheritance checks
Short ordinary cue and reachable full evidence implemented current same-Todo decision summary; existing explore turn-context --result-node detail read
CLI/App/Lark readback implemented within M1 shared normalization; packaged history/configuration journeys; Lark projection dry run
Actual model adoption and outcome/cost advantage deferred M2/M3 remain unqualified; deterministic followthrough proves attribution only

Scope And Continuation

This is complete within the bounded M1 proposal. It delivers a reversible source and packaged journey; it does not claim improved scores or actual model adoption. M2's provider-specific fidelity qualification and M3's matched outcome/cost study remain in the existing RFC.

Enable Review measurement criteria / 复核衡量标准 in the existing Goal capability editor, or set the optional execution_profile.yardstick_enabled. It is off by default, independently inherits the machine setting, and a Goal's explicit false overrides inheritance. Enabling it does not enable Explore or start tools. Disable it to stop new captures and runtime cues; history stays readable, and already committed attachments can finish replay. An ordinary Turn may omit a new attachment without erasing its current decision.

A bounded refactor extracts the existing pure Lark record mapping into explore_record_values.py, keeping transport and old record values unchanged.

Validation

Current follow-up head: b1428c3acc12307a27b54c2ab9181d501be78cf1. Configured mypy exposed two missing Python transport annotations: the JSON-decoded typed Yardstick object and the optional runtime path. This follow-up supplies those annotations without changing the TS decision owner, validation, cache, configuration precedence or replay behavior. Configured mypy now passes all 17 source files; 23 existing real CLI/cadence tests, changed-file Ruff, diff hygiene and the staged bilingual check pass. Independent review confirmed the transport types match their existing owners. The broader validation below was run on the prior stated revision; it is retained, not relabeled as rerun. Remote CI and exact-head merge readiness remain separate.

  • Tested revision: 385263c2af586636d8caa319ae8c85e8039289d4.
  • Run state: finished.
  • Input classes: synthetic, public_fixture.
Check kind Result Evidence / limitation
real_backend / real_entrypoint passed Isolated SQLite, real CLI settlement → ordinary guard → referenced decision → concrete synthetic action/result hash → subsequent measurement and history; 106 focused Python tests after independent review fixes
regression_parity passed Disabled capture rejected before primary commit; disabled cue absent; historical read and exact replay retained; committed attachment recovers after injected append failure, a newer same-Todo target and disable; one snapshot binds predecessor validation to append CAS
unit passed 132 configuration tests; independent inheritance, explicit false, clear and sparse API behavior
unit / static passed 51 typed tests and control-plane typecheck; after short-cue refinement, 21 targeted typed tests rerun; after replay/CAS corrections, all 6 owner tests, 4 real CLI cases and the independent reviewer's 2 original counterexamples pass
real_backend / packaged passed Existing packaged research-history browser scenario backed by isolated SQLite/HTTP; mixed statuses, independent questions, unavailable source, expanded detail, narrow viewport and navigation
real_backend / packaged passed Existing configuration browser scenario with real configuration backend; inheritance, explicit false/true, invalid preview, write/readback failure and recovery
integration passed Packaged dashboard build, configuration frontend smoke, existing Explore Lark result-layer dry run
static passed Changed Python lint; bilingual RFC diff; exact-head quick premerge including maintainability, vocabulary, CLI output budget and public boundary
manual not_run Live Lark publication, installed-host rollout and real-model adoption/outcome study; no active runtime or live user data used

Earlier premerge failures exposed module growth and a Python reverse import; these were resolved through the pure mapping extraction and configuration-owned resolver. A later whitespace failure was corrected; the committed head passes. Independent exact-head review then reproduced a validation/CAS snapshot gap and delayed historical capture rejection. The updated head uses one validated snapshot for append CAS and restores only strictly older exact committed replays without changing the current target; both original counterexamples now pass. No threshold waiver or baseline adjustment was added.

Coverage limits: the synthetic actor establishes a real persisted attribution chain, not causal quality improvement. The optional vocabulary extends the existing Explore attachment; older callers that omit it retain the existing path. No new provider or authority-store routing is introduced.

Frontend / Visual Evidence

New optional readback and configuration affordances; before: no work-yardstick control or typed finding detail. Existing finding content remains primary, and the detail is collapsed by default.

All captures use public-safe synthetic data and the packaged dashboard at 1b13f97377bc1e844b1182cea93131b33b30222d. The subsequent replay/CAS fix changes no UI, configuration or presentation code; the captured product views are unchanged at the tested head. The history scenario uses its real isolated owning backend; the configuration scenario uses its real configuration backend with unrelated workspace data synthesized. No installed-host rollout is claimed.

First viewport, including populated mixed-state history:

Research history first viewport

Expanded decision, desktop:

Scoped target and yardstick desktop readback

Expanded decision, narrow:

Scoped target and yardstick narrow readback

Existing configuration journey, explicit off, desktop and narrow:

Configuration explicit off desktop

Configuration explicit off narrow

Attention review: the current finding and evidence remain the first screen; provenance, metric scope and next action are available in one disclosure. Long references wrap on narrow screens. Existing unavailable/error states and configuration readback recovery remain visible. The change adds no hero, grouping-only card, raw color, motion or duplicate settings page.

Type / Area

  • New feature
  • Documentation update
  • Test update
  • Control plane
  • Capability or extension
  • Public docs or presentation surface

Shared-authority RFC fixture impact: N/A; no promotion, new provider or authority routing change.

Boundary Checklist

  • No private state, credentials, raw traces/verifier output, internal links or local machine paths in the diff, body or attachments.
  • Scoped to the existing target/yardstick M1 request, without duplicate benchmark work.
  • Both RFC languages updated; exact-head bilingual check passed.
  • Public-safe first viewport and desktop/narrow visual evidence included.
  • DCO sign-off present.
  • Independent review and merge authorization remain separate; this PR is not self-merged.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
loopx-agent previously approved these changes Nov 6, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6-astra · OpenAI · runtime_reported · reasoning_effort=ultra

动机

这项改动面向正在同一个 Todo 内反复试验、比较结果的 Agent,以及需要复查这些选择的用户。过去,Explore 能保留发现、测量和执行记录,Stride 能在正式 replan 时记录决策,但普通工作轮次缺少一个明确的入口,把“现在追求什么、用哪个指标判断、为什么改变下一步”接到后续工作中。

一个具体例子是:本地筛选分数越来越好,真正的验收结果却没有改善。旧路径可以保存这个反例,但下一轮未必能辨认它针对哪个当前方法;新路径把反例、当前目标和有成本边界的下一步选择附在已有结果写回中,再向同一任务投影简短提示。已验证的改进是这些信息可以进入普通 Turn、持久化、恢复并关联后续候选产物;尚未证明真实模型会采用它,更未证明最终分数或总成本改善。

本 PR 不增加独立规划器、每轮必填表或新的验收权限,也不把代理指标视为已验证的真实指标。它是一个有实际调用链的阶段性交付;后续仍需真实模型采用、代理指标可靠性及等条件效果/完整成本研究。

独立评审基准是 target-yardstick-stride-decisions-v0.md,固定版本 5a58635028fbef1316dc66531daf5fdb655e9b13,而非本 PR 改写后的验收文字。C1、C8、C9、C10、C11、C12 的本阶段路径已核验;C7 的“旧提交恢复不覆盖当前选择”也覆盖。C2、C3、C4、C5、C6 的完整目标/代理校准政策、C7 的更完整异步评测适用性,以及 C13 的真实采用和 cue/no-cue 效果比较仍属后续资格,不能从本次通过的传输测试推导出来。

改动思路

实现复用 Explore 的结果附件、提交身份、事件存储和重放,不另存一份可编辑的规划日志。TypeScript owner 校验声明、同一 Todo 的前驱和指标身份;Python 继续承担现有持久化与调用适配。新声明是 Agent 的方法意图,不能从已有数值测量可靠推断;当前提示和历史回读则从该声明派生。

触发 当前行为 边界
普通写回有实质方法变化 附带可选目标/指标/下一步声明 绑定来自原提交的 Goal、Agent、Todo、Turn
下一次同任务工作 默认读简短提示、反例引用及详情命令 完整指标和关联记录按需读取;不另触发 replan
关闭新能力 不提供新提示,也不接收新的能力声明 已提交历史仍可读,精确重放仍可补完
验证后发生并发图写入 拒绝用失效快照追加,保留已提交主结果 使用原身份恢复,不重复扣费或执行实验
旧提交晚于新选择补写 仅严格更早的已提交结果可补历史 当前选择仍按原提交时间确定

这个边界适合保留在现有 typed owner 和 Explore 写回链内。若仅增加提示语,无法稳定关联来源和恢复;若加入第二个规划器或日志,则增加同步和权限成本。当前 PR 的有用范围是“可选、可恢复、可回读的同任务选择”,不是完成整个 Yardstick 研究计划。

具体改动

  1. normalizeWorkYardstickDecision / validateWorkYardstickContinuation 约束字段、引用、指标方向与作用域。反例必须属于本 finding,代理与真实评估声明不能完全相同;外部附件不能自带提交绑定或重放时间。authoritative 是声明的评价来源,不是系统颁发的验收证明。
  2. prepare_result_attachment / deliver_result_attachment 复用现有主提交和可恢复后置关联。独立审阅在旧 head 发现的两个问题已修复:前驱校验、演化计划与 append CAS 现在共享一次快照;已提交但未关联的旧结果在新目标出现后仍能补入历史。等时、较新或非精确重放的失效前驱仍被拒绝。
  3. projectExploreTurnContext 只投影归当前 Agent 所有的未完成任务。普通入口保留简短提示和可达详情;已有完整证据在截断显示前参与定位。后续执行关联记录候选 hash、执行方法和测量,不把“提示出现过”冒充“采取了行动”。
  4. apply_todo_replan_cadence_machine_default 保持 Yardstick 与已有 cadence/Stride 配置独立:默认关闭,Goal 显式 false 覆盖设备 true,清除覆盖后恢复继承。App 复用现有能力配置入口;CLI 与配置 API 使用同一解析规则。
  5. WorkYardstickReadback、历史 decoder 和 Lark 纯字段映射读取同一标准化记录。详情默认折叠,来源、方法和声明性质仍可检查。Lark 拆分的是既有字段映射,没有新增自动外发权限。两种语言的 RFC 均说明本阶段与未来研究的界线。

我的独立执行覆盖如下,均针对 385263c2af586636d8caa319ae8c85e8039289d4:

  • PASS:真实隔离 SQLite CLI、既有 Explore 写回和 Turn context 共 106 个用例;包含普通 guard → 当前提示 → 合成候选产物与 hash → 下一次测量/历史回读。合成 actor 只证明关联和恢复,不证明模型采用或实际研究收益。
  • PASS:52 个 typed owner、context、writeback 和历史用例,以及 103 个配置/API/cadence 用例。
  • PASS:我独立构造的两个并发/迟到提交反例,在原 head 均失败,在当前 head 均通过;不是仅复用作者新增断言。
  • PASS:主干与当前 head 的三组同输入默认关闭对照,整个公开 Explore 结果包相等;仅规范化临时路径、时钟与由路径派生的不透明身份,未规范化提示、错误、状态、准入或副作用。
  • PASS:公开调用路径上的冗余注释、12 条较新无关发现、分页边界、损坏日志恢复、完成任务退休检查;无关显示变化不会抹掉当前反例,损坏源被拒绝而非冒充空结果。
  • PASS:双语 RFC、diff whitespace;独立检查了完整 diff 和生产调用者。作者的精确 head quick premerge、packaged HTTP/SQLite browser 和构建证据用作补充,不替代上述独立验证。

对主干的风险

这是默认关闭的行为扩展。已做 base/head 对照,未发现关闭时新增决策要求、调用、调度或扣费变化。启用后仍会增加当前工作声明和上下文内容;完整指标按需读取可以限制重复信息,但真实模型开销、长期累积成本及收益尚未量化。不要把本 PR 的通过当成广泛启用或效果推广资格。

恢复方面,提交后关联失败仍由原写回重放负责;关闭功能不应丢弃已提交历史,降级二进制前仍需完成新格式的待交付写入。拒绝较新失效前驱是有意的并发保护,不应通过任意重放时间绕过。默认关闭、旧配置以及旧 finding 的读取均有覆盖。

当前 GitHub CI 并非全绿。kernel-static-checks 在 tests/control_plane/test_goal_create_completion.py:184 报 F841;我用相同 Ruff 命令在固定 base 与当前 head 得到同一失败签名,且本 PR 未修改该文件或 lint 配置。其后若干工作流被跳过或失败,另有构建待完成;这些保持为单独的合并资格状态,不伪称全套 CI 通过,也不要求本 PR 顺便修改无关测试。

未执行真实模型、现有运行实例热更新、在线 Lark 外发或安装版验收。历史和配置截图来自前一提交;后续提交仅修写回/typed owner/测试/RFC,已比较确认可见代码未变,未把旧截图改称新构建。

我的整体评价

APPROVE,作为可选同任务决策传输与恢复能力的阶段性交付。完整 diff 未发现仍未解决的阻断项;此前两项恢复问题已通过独立反例确认修复。这个结论不包含 merge 授权、不覆盖待完成的 CI,也不证明 LoopX 的模型效果提升。

视觉上已人工检查含 32 条记录、20 个 finding 和混合状态的完整页面、桌面展开、390px 窄屏展开以及配置写后回读失败状态。默认视图保留发现内容的优先级,完整指标、成本和来源在展开后可达;窄屏按纵向重排并换行。复用截图的可见代码未变。未验证范围是安装版和真实用户阅读效果;配置失败截图中已有顶层错误与侧栏叠放的表现不作为本 PR 新引入的回归。真实模型采用、C13 对照和后续 M2/M3 的效果/完整成本资格仍待验证。

English verdict: APPROVE — head 385263c2af586636d8caa319ae8c85e8039289d4. Independent real-path tests resolve both reproduced recovery races and preserve default-off behavior. Model adoption and outcome benefit remain unqualified; CI and merge readiness are separate holds.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@huangruiteng
huangruiteng merged commit fa266a2 into main Nov 7, 2026
27 of 33 checks passed
@huangruiteng
huangruiteng deleted the codex/yardstick-current-work-20261107 branch November 7, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants