Skip to content

Keep unresolved command-request usage unknown - #8282

Merged
loopx-agent merged 1 commit into
mainfrom
codex/benchmark-command-usage-coverage-20261107
Nov 7, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/benchmark-command-usage-coverage-20261107

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

A Codex capture with a compound tool call and no request associations could report 0% LoopX command-request input even though some calls could not be classified. New captures report that share as unknown. Research and capture comparisons apply the same rule to previously saved zeros without rewriting packs. Verified native request totals and recognition counts remain visible. A measured empty subset still reports zero, and positive measurements still describe the recognized subset.

Author Declaration

  • Written by: model_agent — OpenAI Codex (GPT-6 family).
  • Implemented against the captured command-request contract above and the reproduced unknown-versus-zero defect.
Criterion Disposition Owner / evidence
Preserve verified request totals and keep command usage separate from text attribution and output bytes Implemented Existing TS trajectory owner; CLI registration and capture replay
Preserve recognized subsets and expose unresolved coverage without inferring associations Implemented Shared TS predicate; late returns, opaque compound calls, old saved zeros, true empty capture
Preserve source identity and frozen artifacts on read Implemented Existing registration and comparison routes; packaged read-only Research readback

Scope And Continuation

Complete for distinguishing unknown from zero. A native command completion may provide a command ID and Turn without its outer tool call or response ID; this change does not infer missing linkage from timing. Actual control-text cost still requires separate attribution. The bounded refactor is one typed predicate shared by capture, comparison and the dashboard.

Validation

  • Tested head: 0d7d15b49c39eff26d2f5a737e6403448e517da7.
  • Run state: finished for local checks below. CI is separate.
  • Inputs: synthetic, public_fixture, authorized_private_read_only.
Check Result Evidence
Typed reducer and comparison Passed node --test tests/control_plane_ts/trajectory_usage_comparison.test.ts: late or opaque calls, old saved zeros, verified totals, true zero and positive subset
CLI registration and source readback Passed tests/capabilities/test_trajectory_input_capture.py: real registration with an unresolved compound call and independent command completion; unchanged text attribution, no model dispatch
Before/after replay Passed Two authorized read-only captures replayed through benchmark trajectory-pack-register: zero shares became unknown; all native totals, recognized sums, tool bytes, and opt-out results matched. No captures are attached.
Packaged Research Passed Real source verifier, TypeScript owner and packaged Chat; synthetic comparisons preserved native totals, withheld unknown deltas and recovered the measured subset when changing selection. Checked both languages, desktop and narrow layout, zero writes.
Static and premerge quick tier Passed Dashboard build, changed Python lint, public boundary, bilingual RFC gate, canary premerge --from-git-diff --tier quick
Full control-plane typecheck Failed on base and head Unchanged goal_source_links.ts imports an unexported JsonValue. This proposal leaves that base failure separate. Full repository suite not run locally.

Frontend / Visual Evidence

The Research landing view and navigation are unchanged. Opening Sources and usage now shows unknown beside the recognition count when needed. Association details retain the counts and method. The populated fixture has mixed research outcomes and four selectable coverage states.

  • Source: synthetic data; locally packaged Chat with the real read-only Research backend. No live or installed runtime was used.
  • Baseline capture: 7108b92337 (same measurement and Research paths as the PR base). Candidate captures: tested head above.
  • Research first viewport:

Packaged Research first viewport, Sources and usage closed

  • Expanded desktop before:

Before: zero percent and zero command usage with unresolved calls

  • Expanded desktop after:

After: unknown command usage and unchanged verified total

  • Expanded narrow after:

After on narrow viewport: unknown command usage and recognition count

Type Of Change

  • Bug fix
  • Documentation and test update

Boundary Checklist

  • Public-safe diff and screenshots; no private source contents or paths.
  • No paired RFC changed.
  • Every commit is signed off.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

审查 0d7d15b49c39eff26d2f5a737e6403448e517da7。未发现阻断问题。实际记录若没有外层关联标识,仍需从原生来源补足证据才能测量这些请求。完整控制面类型检查在原始 base 和 exact head 均失败于独立的 JsonValue 导入。

动机 / Motivation

研究人员在 Research 中对照同一案例的两份捕获,想看发出 LoopX 命令的请求用了多少输入。 复合调用只有命令完成记录和 Turn,缺少可绑定的外层调用或响应。基线显示 0%,容易把归属缺口带进后续成本比较。 新登记把这类命令请求占比记为未知。读取历史包时,对照同样保留未知,同时显示识别计数和已核验的原生总用量。 本次交付限于已保存捕获的用量解释。控制文本归因、工具返回字节、原生计分和费用各自沿用其计量来源。

改动思路 / Approach

登记入口从原生回执和唯一返回取得同一窗口的证据,TypeScript owner 汇总已识别请求。现有研究读取先核验两个来源,再由同一 owner 计算差值。对没有识别请求而仍有未解析调用的捕获,零 token 是已识别集合的空和,实际命令用量未知。有正向匹配时继续展示已识别子集,计数让读者看出覆盖范围。

新包登记、历史对照和单包展示共用一条覆盖判断。只修登记会留下旧包误读,只改页面则会让对照 API 继续产生零值差。 本次边界是解释现有识别计数并修正三处读写路径,历史包按原身份读取。 在旧包上独立复核了无需重新登记即可比较的路径,保留现有采集选项和使用步骤。

具体改动 / Changes

按 docs/reference/trajectory-assessment.md 在 172d80d630f424b0b44dd513863ba02b5f13d55c 的既有口径逐项核验。Whole requests issuing LoopX commands and capture comparison

  • REQ-ASSOC 已实现。简单调用须按同一 Turn 内回执与唯一返回关联。独立命令完成 ID 和迟到返回保留为未解析,已识别请求仍按原生响应去重。
  • REQ-SUM 已实现。原生请求分母和已识别子集之和独立保留。复合调用的未知状态传到命令输入、缓存、未缓存、输出及比例差值。控制文本估计与工具字节各由既有来源产生。
  • REQ-READ 已实现。按已登记的 Goal、来源、窗口和会话读取冻结包。原包中的零照原样保存,比较时根据已有计数显示未知。独立读回核对了包与源文件指纹及恢复结果。

关键代码讲解

  • hasObservedTrajectoryLoopXRequestUsage 从识别、未分类和未匹配计数判断是否有可报告的子集。summarizeTrajectoryLoopXRequests 在原有 number | null 里记录有分母的可测占比,原生总量与已识别 token 汇总继续由原有回执决定。关联与汇总
  • compareTrajectoryUsage 经 values 只用可报告的命令用量填充现有字段。现有差值函数传递 null,因此一个未知侧会抑制命令差值,同时保留原生总量差与覆盖细节。冻结用量对照
  • LoopXRequestFootprint 对未解析的零匹配突出显示未知与识别数,并将方法和计数留在详情。测得的空子集显示零,含未解析调用的正向子集继续显示已识别用量。Research 展示

类型注释解释原有可空字段,双语参考页记录同一口径。CLI 测试覆盖同一 Turn 的独立命令完成事件,TypeScript 测试覆盖迟到返回、旧零值、真零及正向子集。CLI 测试 TypeScript 测试

对主干的风险 / Risks

最强反例是新包正确显示未知,旧包却因已有 0 继续参与比较。用同一合成输入运行基线和候选的公开登记命令,再用真实 Research GET 读取冻结包。基线可复现零占比和数字命令差值。候选的新包占比为 null,读取旧包时源值仍为 0,而输出的命令计量和差值为 null。测得的真零、正向子集、原生总量以及关闭工具采集的结果均通过反例检查。源字节变更使对照不可用,还原后恢复同一结果。无效 pack ID 被拒绝,同会话比较保持拒绝。主干随后合入 #8283,其条件分支处理已完整读完且无 Explore 记录的关联来源,已复核本次用量的入口和两侧计量 owner。

  • PASS 独立 CLI 用量用例 9 个,TypeScript 比较用例 8 个,来源验证 HTTP 比较用例 2 个。同一合成输入的 base/head 重放覆盖旧包、关闭采集、源失效与恢复。
  • PASS 本地打包 Research 读回、前置 quick canary、RFC 双语 diff 检查、Python lint 与公开边界扫描。
  • FAIL npm run typecheck:control-plane 在原始 base 和 exact head 得到相同的 TS2305,位置为 loopx/control_plane/presentation/goal_source_links.ts:2。该处及其 JsonValue 导出路径未进入此 diff。改动的行为由上述独立运行覆盖。
  • UNVERIFIED 远程 CI 依本次本地验证配置略过。完整仓库套件未跑。

语义与 CI 对齐

沿用 codex_ordered_loopx_requests_v1 的现有状态、计数和可空占比。变化是零匹配且归属未全时的值解释,参考页同时注明新包与旧包的读取行为。保留独立的原生请求合计、控制文本归因与工具返回字节,也保留源验证及只读恢复路径。quick canary 的语义词汇、可维护性和前台投影检查均通过。

我的整体评价 / Verdict

此修复交付了已观察的未知与真零区分。后续重复比较仍读取原 pack 和源指纹,选择范围或修复临时源缺失后可以继续比较。用户在现有用量入口就能区分已确认的分母与未知的命令用量,复核时按需打开关联细节。设计只需从已有计数推导结果,兼容旧包无需迁移。

桌面与窄屏上,研究结果和时间线仍在第一层。展开来源与用量后,未知和识别计数同处一行,方法、未匹配计数与来源留在可展开明细。 视觉证据覆盖本地打包的合成 Research 工作区,已安装 App 与真实长程研究内容的渲染待验证。 PR 视觉证据

English verdict: APPROVE 0d7d15b. No blocking findings. CLI and read-only HTTP replay, packaged Research and quick canary passed. Full control-plane typecheck reproduced the same unrelated base failure.

@loopx-agent
loopx-agent merged commit 60167ed into main Nov 7, 2026
17 of 39 checks passed
@loopx-agent
loopx-agent deleted the codex/benchmark-command-usage-coverage-20261107 branch November 7, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant