Repository navigation
feat(benchmark): waive small terminal gaps with explicit duration policy - #8258
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | reasoning_effort=max
Request changes conclusion (author-owned PR; GitHub blocks formal self-review)
[P2] 限定 JSON 报告输出的策略字段。 输入包含有效的 makeup_tolerance_ms 和额外的 operator_annotation 时,两版 CLI 都能读入。当前提交把整个 duration_policy 写进报告,连这项未声明的输入注记也带入了 JSON。去掉注记后,报告的时长判定和比分完全相同。JSON 只投影通过校验的 target_ms、allow_goal_completion 和 makeup_tolerance_ms。 用真实 CLI 验证带额外字段的输入仍能生成报告,同时 JSON 只含这三项。 报告生成
动机 / Motivation
维护实验报告的人需要从已经结束的运行中挑出确实要补时的结果。 例如目标 18 小时、已验证运行 17 小时 45 分钟时,旧报告把它列为待补时。明确允许 15 分钟容差后,报告把这条记录标为 within_tolerance。 独立 CLI 复测保留 900000 毫秒的原缺口,超出一毫秒仍待补时。 这项策略只影响报告中的时长资格。运行是否结束、实际分数及实验完整性仍由原系统判定。 当前实现把未声明的策略字段一并写入 JSON,下面的 P2 给出了可复现输入。
改动思路 / Approach
操作员在现有 JSON 输入里声明容差。报告继续对同一运行的区间求并集,并保留原有的提前完成、运行中和未知时长分支。容差在算出真实缺口后才参与判定。这样可以复用一处时长规则,也让操作员在生成报告后核对阈值和实际差值。保留现有报告的时长计算和比较规则,在生成 JSON 时限定策略字段。 时长计算
具体改动 / Changes
关键代码讲解
- validateInput 只接受小于目标的非负安全整数。省略时按零处理,八种非法输入均在写文件前退出。 输入检查
- durationCoverage 已结束运行的已知缺口小于或等于容差时,标为 within_tolerance。重叠片段只记一次。空区间仍是零,缺少区间仍是未知。 覆盖判定
- buildReport 单独计算时长资格。合成数据中,符合资格的共同题从 2 道变成 3 道,描述性共同题、均分、排除原因和待重跑计数保持一致。Markdown 同时显示阈值。 比较
按 b9faf8d 的 docs/reference/benchmark-campaign-report.md 核对 observed-intervals、strict-default、audited-early-finish 和 comparison-authority。原区间、默认要求、经审计的提前完成和比分来源均通过对照。容差是这份提案明确增加的报告规则。 基准契约
对主干的风险 / Risks
两组各 15 道合成题走了实际 Node CLI。省略容差的 JSON、Markdown、CSV 和 HTML 在基准与本提交逐字节一致。取消、未知和不可计数记录的原有约束保留。引用的集成提交里,原生执行承诺仍以剩余时间为零判定。离目标差一秒返回 false,达到目标返回 true,未闭合或校验不符仍为未知。该复测使用隔离的执行记录。 集成调用
语义与 CI 对齐
within_tolerance 是报告内的状态值,独立于原生运行和计分。24 项现有测试、改动范围的 TypeScript 检查和快速 premerge 通过。主干 b9b2501 与本提交合并预检无内容冲突。严格 CQR 因新增的 JSON 输出问题未通过。按当前策略,托管 CI 保持未核验。 现有测试
我的整体评价 / Verdict
保留报告内的容差规则。JSON 只投影通过校验的 target_ms、allow_goal_completion 和 makeup_tolerance_ms。 我在隔离副本验证了该修复,额外注记消失,其余 JSON 内容一致,Markdown、CSV 和 HTML 也逐字节相同。检查的是 CLI 生成的 JSON 和 Markdown 文本,现有 HTML 图表没有改动。实际操作员采用和节省的重跑成本仍需实测。请在修改后的提交复跑正例和额外字段反例。
English verdict: REQUEST_CHANGES. Emit only the three validated policy fields in the JSON report. The explicit duration tolerance, strict defaults, and native execution boundary passed independent source probes.
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Self-review after fixes — exact head 49c6885d8cf702423f4cfc4043b327b75b1b27e1.
- Closed the P2: JSON now emits only validated
target_ms,allow_goal_completion, andmakeup_tolerance_ms; extra input fields are not leaked. - Added a real CLI regression with
duration_policy.operator_annotationand a full deep-equality assertion on the projected policy. - Integrated current
mainwithout conflicts.
Verification at the exact head: all 24 campaign-report tests pass, including the extended-field CLI case and inclusive tolerance boundary; diff hygiene passes; commits carry DCO. No blocking finding remains for this exact-head merge readiness.
huangruiteng
left a comment
There was a problem hiding this comment.
Independent approval of exact head 49c6885 after projecting only validated duration-policy fields.
Terminal benchmark runs with tiny verified duration gaps currently enter the same makeup queue as runs missing hours. This adds an explicit
duration_policy.makeup_tolerance_msto the existing campaign report reducer. Within-tolerance terminal runs retain their measured active time and exact gap, and receive a separatewithin_tolerancedisposition. The default remains zero; planned/running and unknown-duration cases keep their existing behavior. No runtime clock, stop condition, native scoring, or integrity authority changes.The JSON projection retains the selected tolerance and Markdown identifies the waiver. The same projection can drive operator makeup selection; score/countability and cancellation exclusions still apply independently. The new disposition stays local to the existing report vocabulary, rather than creating a control-plane state. The adjacent refactoring pass reuses the existing interval-union and qualification owner; no parallel reducer was added.
Validation: all 24 campaign-report tests pass, including real CLI output, interval overlap, inclusive tolerance boundary, unknown/zero duration, active/planned runs, invalid policies and qualification exclusions. Diff check, bilingual RFC check, semantic diff advisory and exact-path public boundary scan pass. The advisory cannot detect dynamic string vocabulary, which was reviewed explicitly. This is a report/selection policy, not evidence of improved task scores or authorization to launch runs.
The same change is composed into the research integration branch at
b9ccb2610a51eeab6fae59f48c0d53e742c93411. This PR remains based onmain; the broader research integration history is excluded.