Skip to content

[12 points] Focused agent-weakness study for issue #455 - #559

Closed
JunkaiWang-TheoPhy wants to merge 1 commit into
ktwu01:mainfrom
JunkaiWang-TheoPhy:codex/issue-455-clean
Closed

JunkaiWang-TheoPhy wants to merge 1 commit into
ktwu01:mainfrom
JunkaiWang-TheoPhy:codex/issue-455-clean

Conversation

@JunkaiWang-TheoPhy

@JunkaiWang-TheoPhy JunkaiWang-TheoPhy commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

中文说明

改了什么

为什么改

这个 PR 将 issue #455 的研究证据、分析代码和报告内容放在一个聚焦的变更中,避免把历史 review 修复和其他 issue 的 report changes 混在一起。

重点审核

  1. evidence YAML 的来源锚点与 benchmark-family 去重;
  2. blinded secondary review 与 agreement 计算;
  3. 报告中 8 个 demonstrated families、6/8 state-control、2/8 decision-execution 的表述;
  4. next-draft PDF 是否与 YAML 和分析输出一致。

验证

  • Ruff check / format check
  • normalize-external、classify、build-data-release
  • 完整测试:1293 passed
  • focused tests:67 passed
  • PDF 重新生成并检查

Closes #455

The clean PR isolates the issue ktwu01#455 evidence package from the merged report history: validated YAML coding data, blinded review artifacts, deterministic analyzer outputs, a concise report subsection, and a regenerated next-draft PDF.\n\nConstraint: The study must remain separate from the frozen v0.9.0 core and include exact evidence anchors.\nRejected: Carry the long review-fix history into the replacement PR | it obscures the review surface.\nConfidence: high\nScope-risk: moderate\nDirective: Keep this PR focused on issue ktwu01#455 and do not fold unrelated report work into it.\nTested: Ruff, full generators, full pytest (1293 passed), focused tests (67 passed), regenerated PDF.\nNot-tested: GitHub CI on the new branch.\n
@ktwu01

ktwu01 commented Sep 21, 2026

Copy link
Copy Markdown
Owner

Nice evidence discipline: every row carries a source anchor, a counter-reading
and a counter-example location, and the 8+1+1 split reconciles.

The main thing this needs is a rebase, and it is not a defect in your work.
The branch edits scripts/build_technical_report.py and
scripts/build_system_evaluation.py, and neither file exists on current
main: the report has moved to the docs/technical-report/latex submodule.
The documented workflow here is also the superseded v0.9.0 one, where current
rules require frozen v0.11.0 inputs. The green CI on this PR ran against the
older base, so it does not tell us whether the study integrates with the
current report build. Rebasing onto main first will show what actually needs
rewiring.

Two smaller things:

  1. data/agent_weakness_evidence.yml has radar_record_id: null on the
    swe_science_misguided_exploration and prbench_end_to_end_gap rows. Every
    other row links to a corpus record. swe_bench_science is already
    registered on main (data/model_cards.yml:1073), so the first should be
    linkable now without waiting on anything. Either link them or mark the gap
    explicitly so it reads as a known absence.
  2. The null-ID validator does substring matching and skips validation entirely
    when the index is absent. docs/query-surfaces.md makes QueryService the
    single source of truth for lookups; routing the check through it would make
    a silent skip impossible.

Also please state the exact PDF regeneration command in the PR body; the
next-draft PDF lands as a binary diff.


Verdict: Comment. Review produced by an agent (Claude), reading the diff, the repo rule files and the cited sources, then re-checked against an independent adversarial pass. Posted as a comment rather than a blocking review.

@ktwu01 ktwu01 closed this Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[12 points] Paper study: identify agent weaknesses and measurement gaps / Agent 弱点研究

2 participants