Skip to content

fix(explore): retain paired candidate measurements in one finding - #8207

Merged
huangruiteng merged 1 commit into
mainfrom
codex/explore-candidate-measurements-20261107
Nov 7, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/explore-candidate-measurements-20261107

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Recording a matched baseline and candidate in one Explore finding failed with measurement metric/scope is duplicated, even when the candidates had distinct identities and result receipts. The ingestion check reused the comparison-group key, so valid paired evidence could only be recorded after splitting it or dropping an observation.

This fixes within-finding uniqueness to include candidate_ref, while keeping the comparison group independent of candidate identity. Both observations survive the ordinary refresh-state writeback, next-turn feedback and existing research reader. Repeated observations for the same candidate/metric/conditions still reject; repeated evaluations belong in separate findings.

  • Outcome basis: reproduced failure of the existing quantitative-findings contract.
  • Intended base: main; no runtime deployment or experiment changes.
  • Owner: the existing TypeScript Explore measurement codec. Python transport, schemas, settings, comparison grouping, acceptance and execution authority are unchanged.

Author Declaration

  • Written by: model_agent — OpenAI GPT-6 Astra.
  • Specification: loopx/capabilities/explore/README.md#quantitative-findings at base 22b0f6c25a0450a66bd93f05954808dc0a5800ed, with the paired-observation clarification in this PR.
Criterion Disposition Path Evidence
Candidate identity describes the evaluated artifact; input/evaluator/scope describe comparison conditions implemented explore_measurements.ts Distinct candidates form one comparison group; unchanged candidate duplicates still reject
Explicit candidate/result provenance and execution binding implemented Existing normalization and attachment owners Missing candidate, undeclared result, wrong input and cross-execution receipts remain invalid
Immutable capture, replay and next-turn evidence implemented Existing refresh-state and result readers File/SQLite write, pre-commit rejection, exact replay and changed-retry refusal
Historical best is evidence, not retained-solution or acceptance authority implemented Existing feedback projection and documentation Higher/lower directions, regression, unknown values, reordered pairs and changed conditions

Within one finding, existing array order remains declaration order: latest is the last matching observation and prior best includes earlier declared observations. It does not assert execution chronology or the currently retained candidate. No candidate-less measurement format previously existed; missing candidate refs remain rejected, and old unmeasured findings remain readable.

Scope And Continuation

Complete within this slice. The same-domain improvement separates observation identity from comparison identity without introducing another measurement owner or planner. No UI source, controls, Lark delivery behavior or default-off behavior changes. Existing saved-result/history readers retain both candidate/result associations.

Subsequent model adoption, decision quality and end-to-end score/cost benefit require separate observation after integration; this PR claims removal of a demonstrated recording failure only. It does not add a final-artifact or incumbent-retention gate.

Validation

  • Tested revision: 7d16deaaff619b63c7be881b48f48c1818a6db84.
  • Run state: finished.
  • Input classes: synthetic. The motivating failure was inspected privately; no raw experiment data or local state is published.
Check kind Result Evidence / limitation
regression_parity passed Both new focused TypeScript cases fail against the original codec. Candidate-aware normalization passes without relaxing same-candidate or execution/source checks
unit passed 50 TypeScript tests across measurements, writeback and turn-context
real_entrypoint / real_backend passed 45 Python tests across measurements and history compatibility, including real File/SQLite CLI refresh, replay, next-turn and HTTP readback
regression_parity passed 11 existing invalid/off CLI entry cases, including rejection before primary commit and omitted-attachment behavior
real_entrypoint / real_backend passed Final-head Linux File/SQLite paired CLI regression: 2 passed with supported Node.js on PATH. An initial invocation stopped before product entry because Node.js was absent from PATH; no product or test changes were needed
real_entrypoint / real_backend passed Packaged research-measurements browser scenario uses the actual history HTTP/SQLite owner; both candidate sources remain accessible at desktop and narrow widths
static passed Control-plane TypeScript check, changed-file Ruff, semantic diff advisory, diff checks and quick premerge; advisory has no new supported vocabulary carriers and is not a proof of unchanged semantics
integration not_run Full repository suite, live model benefit and deployment; independent review and CI remain separate

The focused suites ran against the same changed-path bytes before the unrelated main update; the paired CLI regression and packaged browser scenario were repeated on the final head. Runtime/backend failures, unmeasured quantities and historical receipts keep their existing distinctions.

Visual Evidence

The existing research reader layout is unchanged. These captures use the packaged application at 7d16deaaff619b63c7be881b48f48c1818a6db84 and isolated synthetic history through the real HTTP/SQLite owner. They demonstrate retained values and reachable source associations, not model adoption or evaluation benefit. No private trajectory or benchmark output is included.

Unchanged reader viewport:

Existing measured-result reader

Paired candidate source details, desktop:

Paired candidates and separate result sources, desktop

Paired candidate readout and source disclosure, narrow:

Paired candidate readout, narrow

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent — OpenAI gpt-6-astra — runtime_reported — ultra

动机 / Motivation

面向把基线与候选放在同一份结果中比较的 Agent,以及通过研究历程复查两者的用户。同条件下基线得 9 分、候选得 6 分。旧版把它们当作重复指标,整次写回在提交前失败;新版一次保存两条观测及各自来源,下一轮仍显示最佳 9 分和候选退步 3 分。独立验证证明配对结果可以保存、原样重放并在后续上下文和 HTTP 历史中读回;同候选重复和不匹配来源继续拒绝。本 PR 不改变评分器、比较条件、执行权限或验收标准,不声称模型已采用这些信息或最终成绩提高。

改动思路 / Approach

去重属于既有 TypeScript 测量编解码器的正确性约束;比较、持久化和读取继续复用各自现有 owner,无新增策略、存储或审批步骤。当前边界完整交付同条件双候选写入和读取,真实模型决策与最终效果仍需后续实验。实现把“是否同一次重复观测”与“能否放进同一个比较组”分开:前者要加候选标识,后者继续只看题目/输入/评测协议等条件。要求调用方把配对结果拆开只是绕过错误;新增一套评分或持久化机制则超出所需。现有普通写回在前置校验成功后才提交,之后的精确重放不会再次记账,也没有额外用户确认步骤。

下面是我通过真实 File/SQLite CLI 独立复现的差异,基线为 22b0f6c25a0450a66bd93f05954808dc0a5800ed,head 为 7d16deaaff619b63c7be881b48f48c1818a6db84:

输入 基线 当前 head
同条件、不同候选,9 与 6 提交前拒绝重复 一次保存两条;best=9,latest=6,improvement=-3
同一配对倒序声明 提交前拒绝重复 best仍为9;latest=9,improvement=+3
同一候选,换结果引用 提交前拒绝 仍拒绝且没有写入
改比较范围 分组保留 仍分组,单组不制造增益
旧无测量 / Explore关闭且无附件 正常写回并可重放 相同;关闭时graph为null

具体改动 / Changes

共五个文件:生产代码仅修改去重键和诊断,并补充排序含义;其余是已有能力文档、真实写回测试和现有 packaged 场景。没有新增 schema、CLI 参数、状态日志或 UI 源代码。评审依据为 loopx/capabilities/explore/README.md#quantitative-findings,固定版本 22b0f6c25a0450a66bd93f05954808dc0a5800ed。原文要求按候选分别记录,本 PR 对“一份配对结果”的允许是有意扩展;我没有拿改后的说明反向证明旧版本已支持它。

对应原契约段落的评审锚点:quantitative-findings 的保存及下一轮读取、comparison-conditions 的严格同条件比较、source-binding 的输入/结果/执行来源一致性、history-and-authority 的历史及权限边界均通过。这里的锚点是段落索引,不是新加的产品规则。

关键代码讲解

  1. normalizeExploreMeasurements 在已有比较键外加 candidate_ref,故两个候选能通过;同一候选即使换收据、改数值或加空白仍不能重复。八条上限、完整字段、显式 null、有限数值和执行来源校验均保留。
  2. projectExploreMeasurementFeedback 算法没有改为按候选分组。它仍遍历完整匹配来源后才交给分页展示;配对内部的最后声明项是 latest,best 仍由方向和数值决定。倒序后的正增量不能解释成真实执行顺序上的进步,更不能说明当前保留了最佳代码。
  3. test_paired_cli_refresh_rejects_duplicates_then_replays_and_reads_both_candidates 走真实准入、写回、重复/冲突、后续上下文及 HTTP 历史,而非另造写入器。我独立运行了这条所在的 83 个生产用例,并另写同一套输入在基线/head 的两种真实后端执行。

对主干的风险 / Risks

最危险的回归是把“候选可区分”误写成“比较组也按候选分开”,或让换一个收据成为同候选重复的绕过方式。独立对照确认两者没有发生。来源不属于 finding、输入不匹配和执行结果绑定仍拒绝;合法配对在关闭 Explore 时仍在提交前被挡住,只是错误变成更准确的未启用提示。旧无 measurements 记录与普通关闭路径不增加字段、提示、写入或审批轮次。

需要保留一个升级边界:旧版本严格 reader 会拒绝新版配对事件,所以写入新格式后不能宣称任意降级仍可透明读取;应保持该数据的读写 owner 同步升级,不能为了回滚静默删掉历史。该变化复用既有字段和 typed owner,不通过字符串猜测任务状态,也不把证据读取变成执行、评分或验收权限。

语义与 CI 对齐

PASS:独立 83 个生产写回/测量测试、34 个 TypeScript 用例、两版本各 18 个 File/SQLite 真实 CLI 对照、TypeScript 类型检查、改动 Python 的 Ruff、双语 RFC 差异检查及 diff 检查。最初私有评审用例把关闭后的 graph=null 当成对象,两个版本均在测试自己的读取行失败;修正测试为明确断言 null 后,两版本各两项重跑通过,产品源码未变。此诊断保留,未作为产品修复或收益计算。依当前 packet 不查询或等待 CI。全仓库、安装版、真实模型效果未声称通过。

我的整体评价 / Verdict

本片段解决了可复现的记录失败,长程价值在于比较证据能跨普通写回、重复请求和下一轮保留下来;用户体验改善在于原本一份结果不再需要人为拆分。它没有证明模型会因此换策略或提高最终分数。已检查该精确头的合成 packaged 截图:主值与未测量状态可辨认,展开后两组候选及结果引用分别保留,窄屏长引用换行;UI 源文件未变。未验证安装版、真实模型采用或真实用户理解;现有展开详情偏长,但本补丁未增加重复说明或改变布局。生产改动集中在已有 owner,没有更便宜且等价的独立机制需要加入;旧可读格式保留,新配对数据的降级边界已明确。无阻塞发现;批准此精确 head 的代码评审,合并、安装和实验准入仍分别处理。

English verdict: APPROVE 7d16dea. Paired candidate capture works through real File/SQLite writeback and readback; provenance, replay, default-off isolation and best-versus-latest semantics remain intact. Model benefit and deployment are not claimed.

@huangruiteng
huangruiteng merged commit b9faf8d into main Nov 7, 2026
23 of 29 checks passed
@huangruiteng
huangruiteng deleted the codex/explore-candidate-measurements-20261107 branch November 7, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants