From ad9900f4e388479af816781a11c5eea217ace50b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 03:32:19 +0000 Subject: [PATCH 01/27] audit: full-repository defect, logic, redundancy and drift review (2026-09-27) Read-only audit of v0.38.0 recorded under docs/audits/ and indexed in docs/README.md; no code changed in this pass. Nine parallel line-by-line reviews (tools by area, skills, READMEs, the architecture and evidence record) plus a first-party re-run of every high and medium finding. Headline items, each reproduced in this checkout: - gates that report a measurement from missing evidence: a nonexistent --before baseline reads as zero words (length-growth on every section, exit 1), an unreadable .tex root lints as an empty document (exit 0), and a --git-ref baseline drops \input children deleted from the working tree; - the rewrite fidelity gate hard-rejects LaTeX en-dash ranges (0.5--1.2 reads as -1.2), turns CDM-like into the acronym CDM-, loses Unicode minus and exponent signs, and cannot see a negation moved between clauses; - rewrite_reward exits 2 whenever the optional voice_model.joblib is absent, so the de-ai and condense gates cannot run on a fresh profile; - five tools crash with a TypeError when --field is omitted although the shared help text promises auto-detection; - silent zero findings under a measured status: anchoring has no power at its own 30-document floor, salience and the per-bucket axes skip whole documents that fall into the unknown bucket, and --journals is ignored on the local full-text shortcut; - the quantile grid sits one order statistic high (p10 reads 0.133 at the 30-unit floor); condense_map budgets the body sentence when the abstract comes first and counts placeholders the length gate does not; - the L0 pattern misses paved/showcased and judges paragraph-initial connectors per physical line; a commented \cite raises a blocker; - TIER_WEIGHTS are never applied; final-review closes rounds without the length or residue gates the standard requires; - stale figures across tools/README, DISPOSITIONS and the EVALUATION hub (3.1% / 16.7% / AUC 0.246 / 493 papers / AUC 0.932 / "10 checks"). The record ends with a three-tier action list (fix, document, defer) and the behaviours checked and found sound. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- docs/README.md | 2 +- docs/audits/full-review-2026-09-27.md | 377 ++++++++++++++++++++++++++ 2 files changed, 378 insertions(+), 1 deletion(-) create mode 100644 docs/audits/full-review-2026-09-27.md diff --git a/docs/README.md b/docs/README.md index f8fd7e8..5a7c3aa 100644 --- a/docs/README.md +++ b/docs/README.md @@ -63,7 +63,7 @@ universal paper PASS/FAIL. | Per-version history | [../CHANGELOG.md](../CHANGELOG.md) · [../CHANGELOG-ARCHIVE-v0.33-v0.34.md](../CHANGELOG-ARCHIVE-v0.33-v0.34.md) (v0.33.0-v0.34.0) · [../CHANGELOG-ARCHIVE-RECENT.md](../CHANGELOG-ARCHIVE-RECENT.md) (v0.27.1-v0.32.0) · [../CHANGELOG-ARCHIVE.md](../CHANGELOG-ARCHIVE.md) (v0.22.0-v0.27.0) · [../CHANGELOG-ARCHIVE-EARLY.md](../CHANGELOG-ARCHIVE-EARLY.md) (v0.1.0-v0.21.0) | | Working rules for this repository | [../CLAUDE.md](../CLAUDE.md) | | Adapted-material attribution and adoption boundaries | [../ACKNOWLEDGMENTS.md](../ACKNOWLEDGMENTS.md) | -| External review records (findings, first-party verification, disposition) | [audits/codex-review-2026-09-04.md](audits/codex-review-2026-09-04.md) | +| External and full-repository review records (findings, first-party verification, disposition) | [audits/codex-review-2026-09-04.md](audits/codex-review-2026-09-04.md) · [audits/full-review-2026-09-27.md](audits/full-review-2026-09-27.md) | ## Conventions this directory is held to diff --git a/docs/audits/full-review-2026-09-27.md b/docs/audits/full-review-2026-09-27.md new file mode 100644 index 0000000..974fb75 --- /dev/null +++ b/docs/audits/full-review-2026-09-27.md @@ -0,0 +1,377 @@ +# 全库缺陷排查与合理性审计（2026-09-27） + +审计对象：`main` = `claude/relaxed-galileo-lpkc74` @ `6107991`（v0.38.0）。只读审计：本次未改任何代码或文档，本文件是唯一新增产物。四项任务——缺陷排查、逻辑与合理性、冗余与累赘、冲突/冗余/过时——分别对应 §3、§4、§5、§6；§2 是主会话自己重跑的复现记录，§8 是建议动作分级。 + +## 0. 方法与基线 + +- **基线**：`python tools/validate_plugin.py` 11/11 通过；`python -m unittest discover -s tests` 529 个测试通过、16 个跳过；`python -X dev -W error::DeprecationWarning -W error::ResourceWarning` 下同样全绿。可选依赖（numpy / pymupdf / sentence-transformers / scikit-learn / joblib / torch / transformers）均未安装，`style-profile/wgl/` 只有 `.gitkeep`，所以"无 profile、无可选依赖"的路径是本次覆盖最完整的路径；需要 numpy 的分支只读代码。 +- **分工**：九个只读子审计并行，各自逐行读范围内每个文件，并在临时目录里用 `python - <` 把空基线读成 0 词，每个 section 都报 GROWTH 并 exit 1（A1）；`ai_ism_lint dir.tex` 把不可读的根文件当空文档，L0 报 `measured`、exit 0（F3）；`--git-ref` 基线里已从工作区删除的 `\input` 子文件从基线消失，删整节的缩减被低估、`--require-shrink` 误报 NOT MET（A4）。三者同源：`tex_assembly._read_file` 吞掉 OSError 返回空串，根文件与子文件被同等对待。 +2. **`rewrite_reward` 的科学保真门有四类系统性误判**（高）。LaTeX en-dash 范围 `0.5--1.2` 被切成 `-1.2`，所有含范围的忠实改写被硬拒（E1）；`CDM-like` 产出缩写 `CDM-`（E2；v0.37.1 把这个现象记成 disposition 而没有修正则）；Unicode 减号被丢、`1.5e-3` 的指数符号被丢、`M200` 里的 `00` 成受保护数字（E3）；集合比较对否定词的位置与个数不敏感，`not` 从一个从句挪到另一个从句可以通过门（E12）。 +3. **`rewrite_reward` 在没有可选 `voice_model.joblib` 时直接 exit 2**（高）。de-ai Pass 3 与 condense 把它当强制门，但确定性保真检查被一个 sklearn/joblib 制品挡在门外；新克隆或没训练学习模型的 profile 上整条流程不可执行（H1）。 +4. **省略 `--field` 时 5 个工具以 TypeError 崩溃**（deai_anchoring / docstructure / voice / partition / fetch_arxiv_abstracts），`rewrite_reward` 报成 "execution failed: TypeError"，而它们共享的 `--help` 文本承诺 "auto-detected when only one exists"；`axis_main` 系工具既不探测也不报错，直接把轴标成 unmeasured（M、B1、C1、C2、D3、E9、H2）。 +5. **静默零发现**（违反 CLAUDE.md 第 3 条）。anchoring 在 n=30、k≥2 时 p 值下界 1/31 > α/k，永远不可能出 finding 却报 `measured`（B5）；salience 在没有任何 bucket 能在 gate 之上分辨时报 `measured` 且零 finding（C10）；无 section 或题目式 section 的文档落入 `unknown` bucket，逐 bucket 轴整篇跳过仍报 `measured`（C13）；structure 对 baseline 里没有的 bucket 报 `measured`（B6）；`--journals` 在 `--fulltext` 本地捷径下被静默忽略（D1）。 +6. **quantile 网格整体偏一位次序统计量**：`quantiles(range(100))["0.9"]` 是 90，即 P(X≤x)=0.91；在 30 单元下限处 "p10" 实际是 0.133、"p90" 是 0.933（C12）。逐 bucket 轴的名义操作点在样本最小的 bucket 上偏差最大。 +7. **condense_map 的 carve-out 只看副本一侧**：摘要在正文之前时，正文句被列为可删 restatement、摘要被指为 canonical home，与 skill 规则相反并进入默认缩减目标（A11）；否定/数字守卫单向（A12）；map 的 `removable_words` 计入 `[math]`/`[CITE]` 占位符而 gate 不计，默认目标可能不可达（A13）。 +8. **L0 词表两处失配**：`paved` / `showcased` 在 SKILL 表和 grep 里有、在 `TIER_A_PATTERN` 里没有（F9）；"段首"规则按物理行而非段落判定，一句一行的 LaTeX 里句中 `Notably,` 成 L0 目标，合并成一行则不报（F4）。 +9. **verify_references**：注释里的 `% \cite{x}` 触发 integrity_blocker 并 exit 1（F1）；`http.client.HTTPException` 逃逸成 traceback（F2）；`--cache` 把一次 404 永久化（F6）。 +10. **文档数字与代码/制品脱节**：validator 11 项检查在两处写成 10（F16 / I11 / G1）；tools/README 与 DISPOSITIONS 仍引用 v0.36.2 之前的 3.1% / 16.7% / AUC 0.246（现为 1.59% / 13.3% / 0.174）；EVALUATION hub 的 L3 行仍是 0.932（现 0.9487）、document-shape 493 篇（制品 507）；README 示例表 salience 4→6 与 examples/README 的 4→3 矛盾（I1–I9、G14–G18）。 +11. **skill 层缺门禁**：final-review 收尾时既不快照也不跑 `length_gate` / `deai_residue --before`（标准 §5.3 要求）（H12）；de-ai 自称 diff 规则 "gates the pass" 但流程里从未运行（H13）；paper-review §O 没有 `residue-absence`（H5）；final-review 让子进程读"父进程产出的 references 报告"但父流程没有生成它的步骤（H36）。 +12. **`TIER_WEIGHTS`（0.5/0.3/0.2）从未参与任何聚合**，而 style-corpus/README、tools/README、calibrate skill、EVALUATION 都说三层按权重聚合（D15）。 + +冗余方面：17 个工具残留无用的 `import argparse`（2026-08-26 cli_common 迁移遗留）；train_voice_model / voice_audit / docstructure 各有一批拆分残留的无用 import；4 个测试文件的 `unittest.main()` 写在文件中部；三份 40 行工具注册表已互相不一致；hedging 单位理由在 7 处重讲、salience 投影缝在 8 处重讲（§5）。 + +## 2. 主会话第一方复现 + +以下每一行都在本机重跑过；`tools/` 通过 `sys.path.insert(0, "tools")` 导入。 + +| 编号 | 调用 | 观察 | 结论 | +|---|---|---|---| +| A1 | `length_gate.py after.tex --before missing.tex` | 表格 `Methods 0 → 5 GROWTH`，一条 strong `length-growth`，exit 1，stderr 为空 | 缺失基线被当成 0 词的测量结论 | +| A3 | `required_shrink_words(parse_required_shrink("7%"), 100)` | 8（应为 7）；`14%` × 50 → 8 | 浮点 `ceil` 多要一个词 | +| A4 | git 仓库：HEAD 有 `main.tex`（含 `\input{body}`）和 10 词的 `body.tex`；工作区删掉两者后 `read_git_document(main.tex, "HEAD")` | 返回 `'\section{Methods}\nRoot prose here.\n\input{body}\n'`，子文件未拼入；`--git-ref HEAD --require-shrink 5` → "removed 2 words of the 5 required (4 -> 2)"，exit 1 | 子文件按工作区而非 ref 解析 | +| A5 | `es.prose_words(r"We have \[ E = m c^2 \] here.")` | 9 个 token，含 `\[`、`E`、`=`、`c^2` | `\[...\]` 不是数学区 | +| A6 | `es.latex_to_plain("First $$x=1$$ prose words survive here $y$ end.")` | `'First $ [math] [math] y$ end.'` | `$$` 吞掉后续散文（numeral 投影不吞） | +| A7 | `lg.section_word_counts("\section{Methods}\nOne two three.\n% \section{Old draft heading}\nFour five six seven.\n")` | `{'Methods': 3, 'Old draft heading': 4}` | 注释掉的标题成了 section | +| A11 | 摘要与 Results 各含同一句 | `condense-restatement` 落在 `results`，`genre_carve_out False`，`canonical_line 1`（摘要），19 词 | carve-out 只看副本一侧 | +| A12 | 同段落先 "is not significant" 后 "is significant" | `condense-restatement` 16 词 | 丢否定词的副本仍算 restatement | +| A13 | 读 `condense_map.py:150` | `n_words = len(clean.split())`，`clean` 来自 `latex_to_plain`，含 `[math]`/`[CITE]` | map 与 gate 计词口径不同 | +| F1 | `refs.bib` + `doc.tex`（含 `% \cite{ghost}`），`fetch` 打桩为离线 | `('reference-missing-entry', 'integrity_blocker')`，exit 1 | 注释里的 cite 成 blocker | +| F3 | `mkdir dir.tex; ai_ism_lint.py dir.tex` | 各轴 unmeasured，exit 0，无错误 | 不可读根文件 = 空文档 | +| F4 | `Notably,` 独占一行 / 与前句同行 | 前者 `tier-a:paragraph-start:notably` exit 1；后者 0 finding exit 0 | 行首 ≠ 段首 | +| F7 | 一行内两个 `---` | 1 条 `em-dash` finding | 按行不按命中计数 | +| F9 | "This paved the way for the fit and showcased the model." | 0 finding，exit 0；`SKILL.md:275` 表列 `pave/paves/paving`、`:344` grep 有 `paved?`/`showcase[sd]?` | linter 漏 `paved`/`showcased` | +| E1 | `rr._numbers("0.5--1.2 arcsec")` | `{'0.5', '-1.2'}`；"between 0.5 and 1.2" vs "0.5--1.2" → `eligible False`, missing `-1.2` | en-dash 范围被硬拒 | +| E2 | `rr._acronyms("a CDM-like model")` | `{'CDM-'}`；"a CDM model…" vs "a CDM-like model…" → 不合格 | 尾连字符进缩写 | +| E3 | `rr._numbers("a bias of −0.06 dex")`；`1.5e+3` vs `1.5e-3`；`_numbers("M200")`；`_units("M200c")` | `{'0.06'}`；eligible True；`{'00'}`；`{'c'}` | 符号丢失、标识符中切词 | +| E12 | "not significant at 43 … significant at 512" vs 对调 | `eligible True`，missing/invented 皆空 | 集合比较看不见位置 | +| E7 | `parse_args(["--field","wgl","--corpus-root","/tmp/x","sample"])`（label_findings 的构造方式） | `field=None`，`corpus_root` 为默认值 | 子命令默认值覆盖根解析结果 | +| C12 | `ref.quantiles(list(range(100)))["0.9"]`；n=30 | 90（P=0.91）；`q[0.1]`=3（P=0.133）、`q[0.9]`=27（P=0.933） | 网格偏一位 | +| C3 | `\section{Appendix}\nTODO fix this later.` 经 `body_lines` 后 `edit_meta_findings` | `[]`；同文本放在 `\section{Data availability}` 下 → `['residue-edit-meta']` | `skip` 段落里的编辑标记不可见 | +| C4 | `res.RE_LABEL.findall("\section {No saddle}")` | `[]` | 上次审计 A9 未落到 residue | +| C13 | `ref.units("无标题散文")`、`ref.units("\section{Weak lensing}…")` | bucket 均为 `'unknown'` | 逐 bucket 轴无此 bucket，整篇跳过 | +| D1 | 读 `fetch_arxiv_abstracts._candidate_ids` 本地捷径条件 | `if jsonl.exists() and not exclude and not args.author:`，不看 `journals` | `--journals` 静默失效 | +| D2 | 读 `:399`/`:403` | `range(args.start_at, …)` 配 `min(args.page, args.per_query - start)` | `--start-at 2000` 请求 `max_results=-1600` | +| D6 | `est.paragraph_initial_words("In recent years the field grew.\n\nRecent advances helped.")` | `['In', 'Recent']` | 多词开头短语永远"absent" | +| D7 | `est.words("[CITE] [math] [FIGURE-OR-TABLE]")` | `['CITE', 'math', 'FIGURE-OR-TABLE']` | 占位符计入词数与词表 | +| D15 | `grep` 所有含 `weight` 的行 | 没有任何一行把权重乘进聚合 | `TIER_WEIGHTS` 是死常量 | +| H1 | `rewrite_reward.py --field wgl --reference r --candidates c` | `[rewrite_reward] no voice_model.joblib in …/style-profile/wgl`，exit 2 | 可选模型成硬前提 | +| H5/H12/H36 | `grep` paper-review §O、final-review | §O 列四条规则无 `residue-absence`；final-review 无 `length_gate`/`deai_residue`/snapshot；`verify_references` 只出现在子进程提示的一个 bullet（:116） | skill 层缺门禁 | +| M1 | 每个工具无参数运行 | deai_anchoring / docstructure / fetch_arxiv_abstracts / train_voice_model 出 traceback；给真实文件不给 `--field`：deai_voice / partition 同样 TypeError，rewrite_reward "execution failed: TypeError" exit 2，deai_features 抛 `ModuleNotFoundError: torch` | 见 §3.1 | +| M2 | AST 扫描未用 import / 未引用定义 | 17 个工具 `argparse` 未用；`length_gate` 的 `re`；`extract_sections:21` 的 `tex_macros`；docstructure 的 json/math/random/re/statistics/Iterable；train_voice_model 的 hashlib/math/re/statistics/defaultdict；voice_audit 的 hashlib/json/re/Counter；12 个测试文件的 `sys`、5 个的 `Path` | 见 §5 | +| M3 | 全部 43 个 markdown 文件的相对链接 / `tools/*.py` / `skills/*` 引用解析 | 唯一失败：`codex-review-2026-09-04.md` 有 16 个 `D:/Projects/sci-paper/...` 本机绝对路径链接 | 对其他读者是死链，且泄露本机路径 | +| M4 | `grep -rn latency.json` | 无任何 py/md/yml 引用 | 孤儿文件，且与 README 延迟行不一致（G19） | + +## 3. 缺陷清单（按区域） + +列内"位置"为 `文件:行`；"建议"是最小改法。状态见 §0。 + +### 3.1 CLI 与退出码契约（横跨多个工具） + +| 编号 | 严重度 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---|---| +| M1 / B1 / C1 / D3 / E9 | 中 | 已复现 | deai_anchoring.py:271, deai_docstructure.py:406, deai_voice.py:247, deai_partition.py:234, fetch_arxiv_abstracts.py:318/370/621, rewrite_reward.py:477, train_voice_model.py:85 | `args.profile_root / args.field` 在 `--field` 缺省时是 `Path / None` → TypeError traceback（exit 1）；`--help` 承诺自动探测。只有 8 个工具经 `cli_common.resolve_field` | 统一走 `resolve_field`（或新增不抛异常的 `resolve_field_dir`），缺省且无法唯一解析时 exit 2 | +| C2 | 中 | 已核实 | cli_common.py:120；影响 register / collocation / discourse / salience | `axis_main` 从不自动探测单字段：无 `--field` 时 `field_dir=None`，轴报 "baseline is unavailable" 的 unmeasured，尽管 baseline 存在 | `axis_main` 内用 `list_fields` 唯一时自动选，否则报错 | +| F12 / F34 | 低-中 | 已核实 | ai_ism_lint.py:83-91, 507-512；cli_common.py:15-19；tools/README.md:47 | 多字段且无 `--field` 时 `resolve_field` 静默返回 None，所有校准轴 unmeasured，stderr 为空；文档说它 "warns and returns None" | 多字段分支打印一行 stderr；改文档措辞 | +| C8 | 低 | 已核实 | cli_common.py:121-126；deai_register.py:641；deai_reference.py:366 | `--calibrate --field 不存在` 抛 FileNotFoundError（exit 1）；`--field` 指向空目录则写出空制品并报成功；`deai_collocation --calibrate --glossary` 静默忽略 `--glossary` | `axis_main` 检查 `field_dir.is_dir()` → 2；无记录时 `summary` 返回 2；`--glossary` 走 argparse | +| C6 | 低 | 已核实 | deai_residue.py:537-541 | `--output` 不可写抛 FileNotFoundError，exit 1（"有 strong finding"的码） | 写文件放进 try，OSError → 2 | +| B8 | 低 | 已核实 | deai_docstructure.py:404；deai_docshape.py:320-330 | `--strong-percentile 1.5` → IndexError exit 1；负值外推 | `main` 校验 0 /length-baseline.tex` 对多文件文档失效：副本解析不到 `\input` 子文件，基线≈0 词，每节都 GROWTH | 改成 `cp -r` 整棵 include 树或用 `--git-ref` | +| A28 | 中 | 已核实 | skills/condense/SKILL.md:128-134；length_gate.py:205-206；tests/test_length_gate.py:182 | map 目标为 0 时 `--require-shrink 0` exit 2（测试还钉住了这个拒绝），skill 的收敛步骤无法门禁 | 接受 0 为"不要求缩减"，或 skill 在目标为 0 时省略该 flag | + +### 3.3 ai_ism_lint（F） + +| 编号 | 严重度 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---|---| +| F3 | 中 | 已复现 | ai_ism_lint.py:357-360；tex_assembly.py:98-119；deai_residue.py:520-521 | 不可读的 `.tex` 根文件（目录、无权限）被当空文档：L0 `measured`，exit 0；同输入 `.md` 则 exit 2 | `document_source` 用会抛错的 `read_text` 读根文件 | +| F4 | 中 | 已复现 | ai_ism_lint.py:40-46, 183, 201, 208-216 | "段首"连接词/开头模式用 `^\s*` 按物理行判定，还带 `(?i)`；一句一行时句中 `Notably,` 成 L0，硬折行的小写 `importantly,` 也是；同时 `It is worth noting` 在行中被漏 | 只在 `paragraph_line_ranges` 每段首行上判，去掉 `(?i)`；开头短语按句首 `finditer` | +| F9 | 中 | 已复现 | ai_ism_lint.py:34-39；skills/paper/SKILL.md:274 vs :344 | `TIER_A_PATTERN` 缺 `paved`、`showcased`；SKILL 表与 grep 两种拼法 | `paves?\|paved\|paving`、`showcases?\|showcased\|showcasing`；表与 grep 对齐 | +| F7 | 低 | 已复现 | ai_ism_lint.py:184-190 | em-dash 用 `search` 每行一条；标准说"每个命中"；examples/README 钉住的计数偏少 | `finditer` | +| F14 | 低 | 已核实 | ai_ism_lint.py:191-200 | 词法规则扫原始源：`\ref{sec:realm}`、`\label{…}`、`\cite{delve-2020}` 里的键成 L0 目标 | 先把 `\label/\ref/\cite*/\url` 参数区置空 | +| F15 | 低 | 已核实 | ai_ism_lint.py:171, 221, 257-273；skills/paper/SKILL.md:291 | Tier B 上限按 heading 单元（含 subsection）计，SKILL 说按 section | 以外层 `\section` 为键，或 SKILL 改口 | +| F18 | 低 | 已核实 | ai_ism_lint.py:54 | `THREE_PARALLEL_PATTERN` 无词边界，`Bland` 里的 `and` 命中 | `\b(?:and\|furthermore\|moreover)\b` | +| F19 | 低 | 已核实 | ai_ism_lint.py:100-104 | `lexicon.json` 非法 JSON 被吞 → 零条 corpus-zero 且 `measured`；list 形状却 exit 2 | 让 JSONDecodeError 传出去（2）或给轴 unmeasured | +| F20 | 低 | 已核实 | ai_ism_lint.py:33, 72, 238, 357-358 | `.md` 输入：YAML front matter 的 `---` 成 em-dash 目标，`%` 被当注释 | 声明只支持 `.tex`，或非 `.tex` 跳过 front matter 且不用注释规则 | +| F25 | 低 | 已核实 | ai_ism_lint.py:156；deai_feedback.py:48-57 | 同一行同一规则两次命中 → 相同 `finding_id`（目前无人按 id 去重，潜伏） | evidence 加 `match.start()` | +| F26 | 低 | 待核实 | ai_ism_lint.py:47-51, 257-273 | Methods 讲 robust estimator 的段落永远无法 exit 0，SKILL:298 又说不能牺牲准确性 | 技术二元组白名单或 `% lint: allow ` | +| F27 | 低 | 已核实 | ai_ism_lint.py:67-72, 238 | `TRAILING_COMMENT_PATTERN` 对 `.tex` 已死（`document_source` 先去注释），注释文字过时；对非 `.tex` 反而有害（F20） | 删除 | +| F33 | 低 | 已核实 | ai_ism_lint.py:439, 478-479；skills/de-ai/SKILL.md:111；train_ai_ism_classifier.py:246 | `--summary` 是 no-op，仍被 skill 与工具提示引用 | 调用方去掉后 `help=SUPPRESS` 或删除 | + +### 3.4 verify_references（F） + +| 编号 | 严重度 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---|---| +| F1 | 高 | 已复现 | verify_references.py:387-403, 451-453；tex_assembly.py:69-97 | `cross_check` 扫的装配文本含注释，`% \cite{ghost}` → `reference-missing-entry` integrity_blocker，exit 1 | `cite_keys(RE_TEX_COMMENT.sub("", text))` | +| F2 / F32 | 中 | 已核实 | verify_references.py:173-185, 464-478 | `http.client.IncompleteRead`/`BadStatusLine` 不是 OSError，逃逸成 traceback exit 1；except 元组里 `URLError`/`TimeoutError`/`JSONDecodeError`/`UnicodeDecodeError` 反而是多余的子类 | `fetch` 加 `HTTPException`；`main` 兜底 `except Exception → 2` | +| F5 | 中-低 | 已核实 | verify_references.py:86-104 | `#` 拼接或 `\"` 转义引号之后的字段全部丢失，DOI 丢了就报 "no identifier" | 支持 `#` 循环；引号值按深度扫描；未解析尾部抛 ValueError | +| F6 | 中 | 已核实 | verify_references.py:343-360, 438-450 | `--cache` 把 404 存成 `null` 永不重查；索引滞后的 DOI 每轮都是 blocker，且 finding 不说来自缓存 | 不缓存 None（或带时间戳 N 天后重查）；`observed` 加 `record_source` | +| F8 | 低 | 待核实 | verify_references.py:190, 194, 225, 228 | `date-parts: [[]]` 等形状 → IndexError traceback | 防御取值 + `main` 兜底 | +| F10 | 中-低 | 已核实 | verify_references.py:265-266 | 第一作者子串匹配：`li`⊂`lin`、`ma`⊂`mandelbaum`、`he`⊂`heymans` 静默通过 | 按 token 比较，或子串分支要求 ≥4 字符 | +| F11 | 中 | 已核实 | verify_references.py:60, 387-389 | `RE_CITE` 只认 `\cite*`：`\nocite`、biblatex `\parencite`/`\textcite`/`\autocite`、`\cites{a}{b}` 的第二组都看不见 → 假 `reference-uncited`、漏 missing-entry | 扩展命令名与重复 `{…}` 组；`\nocite{*}` 视为全引；或声明不支持 biblatex | +| F13 | 低 | 已核实 | verify_references.py:107-119, 283-288 | `@string` 被跳过但不展开，`journal = aap` 与记录字面比较 → 假 mismatch | 收集并替换 `@string` | +| F21 | 低 | 已核实 | verify_references.py:398 | `--tex` 的 missing-entry finding 永远 `line=1` | 按匹配偏移算行号 | +| F22 | 低 | 待核实 | verify_references.py:45, 173-185, 433 | 无重试/退避，UA 无 `mailto:`（CrossRef polite pool） | 429/5xx 重试一次；可配置 mailto | +| F23 | 低 | 已核实 | verify_references.py:139 | 作者列表只按小写 ` and ` 切分 | `re.split(r"\s+and\s+", …, re.I)` | +| F24 | 低 | 已核实 | verify_references.py:321-327 | 只在 `url` 字段里的 DOI 被忽略 → "no identifier" | 对 `url` 也跑 `RE_DOI_URL` | + +### 3.5 rewrite_reward 保真门（E、H） + +| 编号 | 严重度 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---|---| +| E1 | 高 | 已复现 | rewrite_reward.py:48-50 | `_NUMBER_SIGN` 禁止数字后的符号，但 `--` 的第二个连字符前面是连字符，被当负号：`0.5--1.2` → `{0.5, -1.2}`；每个含 en-dash 范围的忠实改写被硬拒 | `(?:(? α/k（k=2 时 0.025），任何文档都不可能出 finding，轴仍 `measured`；tests/test_deai_anchoring.py:147-150 自己写出了这个算术但代码不执行 | 检测时 `1/(n+1) > class_alpha` 的类报 degraded 并跳过；或校准下限提到 `ceil(k/α)` | +| B6 | 中 | 已核实 | deai_structure.py:271-282 vs 322-323 | template finding 的状态由 `baseline and policy` 决定而非 `bucket_reference`；baseline 没有的 bucket（如 `unknown`）报 `measured`，reference 块是 `templated_fraction: None, n: 0` | `measured` 仅当 `human_fraction is not None and policy` | +| B7 | 低 | 已核实 | deai_metrics.py:72-79, 241, 198-213 | `sentence_stats.json` 只有 `unknown` bucket 时 pooled_cv=0，任何 burstiness finding 不可能产生，轴 `measured` | `reference["cv"]` 为空时 degraded/unmeasured | +| B9 | 中 | 已核实 | deai_docstructure.py:92-94, 104-110；deai_docshape.py:288-290, 355-362, 583-592 | 触发规则用 `_percentile ≥ p`（加一秩），但 finding 引用的 `strong_threshold` 与 LOO 假报率用 `_quantile`：文档可以在自己引用的阈值之下被判 strong，LOO 率对应的是另一条规则（约 2×） | 两侧统一为一条规则 | +| B10 | 中 | 已核实 | deai_partition.py:82-102, 203-212；deai_docshape.py:496-549 | 每一步 `n_paragraphs` 变化，跨越 `strata_edges` 时换 manifold，贪心比较的是不同 manifold 上的距离（docshape 的 docstring 明言不可混比）；停止条件用 `conformal_p`、改进条件用原始距离 | 比较 `p_value`，或整个计划钉住起始 manifold 并记录 `calibration_basis` | +| B2 | 中 | 已核实 | deai_partition.py:154-161, 126-135；extract_style.py:135-139 | 模拟 split 用 `" ".join(sentences)` 重接，换行变空格，行尾 `% note` 注释掉后半段，排序与 `distance_after` 算在丢了散文的文本上；"零 token"只对建议成立，不对被评分的状态成立 | 在原始块文本的句边界偏移处切分，保留换行 | +| B3 | 中 | 已核实 | deai_partition.py:67-72；extract_sections.py:163-165 | 只有 `\section` 块被标 fixed，`\subsection`/`\chapter` 块成为 merge/split 候选 | `fixed = bool(RE_HEADING_COMMAND.match(block.lstrip()))` | +| B11 | 低 | 已核实 | deai_partition.py:84-87 vs deai_docshape.py:183-198 | partition 不套 `MIN_SECTIONS`/`MIN_PARAGRAPHS_PER_SECTION` 过滤，两工具对同一文档的起始距离不同 | 复用 `document_shape` 的过滤 | +| B12 | 中 | 已核实 | deai_docshape.py:178-180 | `document_shape` 用 `section_line_ranges` 丢掉 bucket，Acknowledgments/Appendix/References（`skip`）作为正文进入分散度、role coupling 与 section arc，校准与检测两侧皆然 | 用 `section_units` 并跳 `preamble`/`skip`；随后重建 baseline | +| B13 | 低 | 已核实 | deai_anchoring.py:95-102, 110-114 | 句子扫描跑在原始 LaTeX 上：标题与第一句粘连，`thebibliography` 整块粘到末句并被年份"锚定"（Conclusions 率 0.0→0.2） | 用 `deai_reference.sections()` 的正文并跳 `skip` | +| B14 | 低 | 已核实 | deai_structure.py:312-351 | `structure-auxiliary` 消息无条件写 "rare in the field reference"，即使参考分数是 40% | 措辞中性或按分数分级 | +| B15 | 低 | 已核实 | deai_structure.py:41 | `RE_TRICOLON_WRAP` = `\bthese\s+(COUNT)\s+\w+` 匹配任何 "these two samples"，且进 `template_score` 与 manifold | 要求句首或后接 `together/define/form…`；改后需重校准 | +| B16 | 低 | 待核实 | deai_metrics.py:238-244；extract_style.py:289-320 | 单节 CV 与跨论文合并的 bucket CV 比较，比值系统偏低，`burstiness_ratio=0.60` 吸收了未知偏差 | 按 (paper, bucket) 校准节内 CV 分布，或在常量旁写明 | +| B17 | 低 | 待核实 | deai_docshape.py:630-637 | Mondrian 分层边界用含校准集的全体文档拟合，严格可交换性要求只用训练集 | 边界只从 `train_records` 算 | +| B18 | 低 | 待核实 | deai_docstructure.py:137-138；deai_docshape.py:570-571, 600-613 | 无 manifold（<55 篇）时逐特征 uniformity/overdispersion 一律 strong，哪怕 baseline 只有 3 篇 | 低于文档数下限降为 ordinary/degraded | + +### 3.8 语料、抓取与检索（D） + +| 编号 | 严重度 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---|---| +| D1 | 高 | 已复现 | fetch_arxiv_abstracts.py:370-391, 422-424, 464 | `--fulltext` 且本地 `human_abstracts_extra.jsonl` 存在时走本地捷径，捷径不看 `journals`；calibrate skill §2 的 "refereed" 广度语料实际不受期刊限制，无警告。docstring 说期刊信息"只在线上路径可得"已过时（记录带 `journal`） | 本地分支加 `if journals and record.get("journal") not in journals: continue`，或有 `journals` 即走线上路径 | +| D2 | 中 | 已复现 | fetch_arxiv_abstracts.py:399-404 | 页大小 `min(args.page, args.per_query - start)` 用绝对偏移，`--start-at 2000`（held-out-labels §17.1 的文档做法）每次请求 `max_results=-1600` | `min(args.page, args.start_at + args.per_query - start)` | +| D4 | 中 | 已核实 | fetch_arxiv_abstracts.py:405-409, 477-483, 513-515 | 全文路径遇 429 `Throttled` 只 `break`，打印 DONE、exit 0；摘要路径才有 TRUNCATED + exit 2；tools/README 未区分 | 把 `throttled` 标志带出 `_candidate_ids`/`fetch_fulltext`，TRUNCATED + 2 | +| D5 | 中 | 已核实 | fetch_arxiv_abstracts.py:573-581, 341-453, 435-436, 591-593 | `--updated-before`（README 称"唯一能给文本定年代的控制"）在 `--fulltext` 下无效；`--query-set/--out-name/--resume` 同样被接受但忽略 | 线上候选循环里套 `updated` 检查，设了该 flag 时拒绝本地捷径；其余三个与 `--fulltext` 互斥报错 | +| D6 | 中 | 已复现 | extract_style.py:125-129, 146-166, 354-361 | `paragraph_initial_words` 只记第一个字母串，五个多词开头短语（"In recent years" 等）永远落在 `blacklist_absent_from_corpus`，dossier §3 写它们"absent" | 对去空白段落用 `startswith(phrase)`，或从集合里删掉短语 | +| D7 | 中 | 已复现 | extract_style.py:142-143, 258-266, 462 | `words()` 把 `[math]`、`[CITE]`、`[FIGURE-OR-TABLE]` 当词：抬高句长、`n_words`（em-dash 分母）、词表 top-50，25 词 + 5 个引用的段落也够 `EXEMPLAR_MIN_WORDS` | `analyse_paper`/`write_exemplar_bank` 先剥占位符（`words()` 本身被 salience/partition 共用，别改） | +| D8 | 低 | 已核实 | extract_style.py:169-170, 264 | `count_em_dashes` 在投影后的文本里找 `\textemdash`，而 `RE_TEX_SIMPLE_CMD` 已把它删掉 | 在去注释的 `sec_raw` 上计数 | +| D9 | 低 | 已核实 | extract_style.py:350, 364, 552-553 | 空 counter 的 `n_paragraphs` 报 1，dossier 的"No paragraphs detected"分支不可达 | `sum(counter.values())`，除法处才 `or 1` | +| D10 | 低 | 已核实 | extract_style.py:142-143, 132 | `words()` 只认 ASCII：`naïve Poincaré` → `na/ve/Poincar`；句末 lookahead 同样 ASCII | Unicode 字母类 | +| D11 | 低 | 已核实 | extract_style.py:466 | bank 行 `id` 按 bucket 重置，`paper.tex:p0` 重复（今日无人读 `id`） | `f"{source}:{sec}:p{idx}"` | +| D12 | 低 | 已核实 | fetch_arxiv_abstracts.py:278-288 | `_tex_members` 要求字面 `\documentclass`（1995 年前的 `\documentstyle`、注释开头的源被丢）；同名成员 `appendix/intro.tex` 覆盖 `sections/intro.tex` | 用 `RE_TEX_DOC_MARKER`；同名成员去歧义 | +| D13 | 低 | 已核实 | fetch_arxiv_abstracts.py:205, 415-419, 470-474 | 续传/去重按带版本号的 id：论文出新版后被再下一份 `…v2` 并列 `…v1` | 目录与 `seen` 用 `_bare(id)` | +| D15 | 中 | 已复现 | extract_style.py:63, 289-296, 332-379, 484-501；style-corpus/README.md:26-29；tools/README.md:12；skills/calibrate/SKILL.md:58,65；EVALUATION.md:99 | `TIER_WEIGHTS` 只被打包进元组，从不参与任何聚合；五处文档说按 0.5/0.3/0.2 加权 | 实现加权均值与分位数，或把五处改成"记录权重，未应用" | +| D16 | 中 | 已核实 | extract_style.py:94-96, 460-461, 155-156 | 以占位符开头的段落（`\citet{X} showed…`、`$\Lambda$CDM predicts…`、公式后接散文）被整段排除出 bank 与开头统计，docstring 说只排"纯占位符段落"；intro bucket（hedging 唯一校准的 bucket）系统性缺少引用开头的段落 | 只在剥掉占位符后为空时跳过；开头词取剥占位符后的首词 | +| D17 | 低 | 已核实 | extract_style.py:246-248, 447-448 | `.txt` 与无 `\section` 的 `.tex` 全落 `unknown`，进统计但不进 bank；dossier 的 "No sections detected" 提示不可能触发 | 文档写明 `.txt` 只用于统计，或按文件警告 | +| D18 | 低 | 已核实 | extract_style.py:233-241, 672-675, 515-518 | 缺 pymupdf 跳过的 PDF 只在 stderr 留一行，dossier 的 "Built from N papers" 不含它们 | dossier 头与最终汇总打印跳过数 | +| D19 | 低 | 已核实 | retrieve_exemplars.py:168-180, 197-200 | 关键词回退在 `--topic` 为空时返回前 k 行、分数 0 | 回退按 `(topic or section)` 打分 | +| D20 | 低 | 待核实 | retrieve_exemplars.py:93-127 | `.npy` 缓存只按 mtime 顺序与行数判有效，同行数重建后可能位置错位（voice_dataset 的缓存有指纹，这个没有） | sidecar 记录 bank 大小与 sha1 | +| D24 | 中 | 已核实 | fetch_arxiv_abstracts.py:83-145, 237-245, 292-293；skills/calibrate/SKILL.md:41-42, 72-79 | 抓取器写成字段通用，实则硬编码 astro-ph/弱引力透镜查询与关键词过滤，无 `--query`；`condmat` 用户按 §2 操作会把弱透镜论文装进自己的语料 | 查询集标 `[WGL]`，加 `--query/--queries-file`，非该字段拒绝关键词捷径 | + +### 3.9 评估、标注与学习模型（E） + +| 编号 | 严重度 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---|---| +| E7 | 中 | 已复现 | label_findings.py:408-415；cli_common.py:54-68 | 根解析器与每个子解析器都注册 `--field/--profile-root/--corpus-root`，argparse 用子解析器默认值覆盖根解析结果：`--field wgl --corpus-root X sample` 以 `field=None` 和默认语料根运行，无报错 | 根只用 `base_parser`，或子解析器副本 `default=SUPPRESS` | +| E13 | 中 | 已核实 | label_findings.py:246, 42-44；skills/calibrate/SKILL.md:176-178 | `quota = max(20, n // (2·populations·5))`，文档的 `--n 60`/`--n 240` 都落到 20，实际每 population 出 ≥100 flagged + ≥100 控制行；`--n` 在 <200×populations 时是死参数且无 help | 定义 `--n` 为每格配额，或下限覆盖时警告；改文档数字 | +| E14 | 中 | 已核实 | label_findings.py:253-274, 288-306 | 文档打乱后按发射顺序取一篇的全部 finding 直到配额满，没有每文档上限：20 行的格通常来自一两篇长论文，精度估计是按论文而非按 finding | 先收集全部再 `rng.sample`（或每文档封顶）；汇总里记每格来源数 | +| E15 | 中 | 已核实 | label_findings.py:366-379 | 合并召回把按轴计数的真阳性（一段被三个轴命中算三次）与按段落计数的漏报相加，单位不一致且被多轴命中抬高（上次审计已指出，未变） | 真阳性按 `(source, start_line, end_line)` 去重 | +| E11 | 低-中 | 已核实 | label_findings.py:192-197, 277-279 | 发射器逐篇抛异常被静默吞成 `{"axis", "error"}` 条目并跳过，汇总把它报成"语料只有这么多" | 按轴计数错误并随 shortfall 行打印 | +| E5 | 中-低 | 已核实 | eval_findings.py:152-166, 340-343 | ≥20 篇但 `n_words` 全 0 的 population 报 `measured`、`per_1k_words: None`，`render` 对 None 做 `.3f` → TypeError | 无词即 unmeasured；`_cell` 防 None | +| E6 | 中-低 | 已核实 | eval_docscale.py:97-114 | 人类语料一篇都没评上时 `statistics.median([])` 抛 StatisticsError | 空层 unmeasured；无人类文档时 SystemExit 带提示 | +| E8 | 低 | 已核实 | voice_dataset.py:35-47 | `_load_jsonl` 对空行 JSONDecodeError（`train_ai_ism_classifier.load_positives` 能容忍） | 跳过空行 | +| E16 | 中-低 | 已核实 | voice_dataset.py:107-134, 198-217, 310-315 | 指纹记录 embedder 可导入性，但 `_record_embeddings` 任何运行期失败（离线下载失败）都降成 0 宽行，缓存带着"健康"指纹写出，`corpus_cos` 永远 0 | `embedder_available()` 为真而 `emb.shape[1]==0` 时不写缓存，或把宽度写进指纹 | +| E17 | 低 | 待核实 | eval_findings.py:199-213 vs deai_register.py:621-655 | `leakage_paired` 的本篇 df 用原始空行块子串匹配（含 preamble/参考文献，`halo` 命中 `halos`）而非 bank 的 token 化，抬高 `suppressed_by_own_membership` | 本篇侧同样 `body_only` + `RE_WORD` + 词根规则 | +| E18 | 低-中 | 已核实/待核实 | eval_docscale.py:97-114, 143 | 无文档下限（1 篇也出 `flag_rate`/AUC）；`alpha` 取自 baseline 而 pooled 分支的 `p_value` 是样本内百分位（`alpha=None`），只影响旧 baseline | 共享 `MIN_DOCUMENTS`；用 `point["alpha"]` | +| E19 | 低 | 已核实 | eval_findings.py:293 | `median_words` 偶数 n 取上元素（100/200/300/400 → 300，`statistics.median` 250） | `statistics.median` | +| E21 | 低 | 已核实 | voice_audit.py:513-530 vs 552-556 | 主划分与第一个审计划分同为 `seed+0`，"20 次重复"里含已报告的那次 | 审计从 `seed+1` 起 | + +## 4. 逻辑与合理性（横切主题） + +1. **静默零发现是一个模式，不是孤例**。B5（anchoring 功效为零仍 measured）、C10（salience 无可分辨 bucket）、C13（`unknown` bucket 整篇跳过）、B6（structure 对无参考 bucket）、B7（metrics 无可用 bucket）、F19（坏 `lexicon.json` 被吞）、F3/A1/A2（不可读文件当空文档）、D1（`--journals` 静默失效）、F12（多字段静默 None）。共同根因：轴状态由 profile 是否存在决定，不由"这篇文档是否真的被测过"决定。建议每个状态函数都接收文本并报告"被测单元 / 总单元"。 +2. **投影不对称仍有残留**（上次审计的主题）。语料侧不认 `\[…\]`/`subequations`（A5/C17）、`$$` 只在 plain 投影吞散文（A6）、docshape 与 anchoring 把 `skip` 段落与参考文献当正文而其他轴不算（B12/B13/B22）、partition 评分状态丢注释后散文（B2）、bank 行 30–400 词带在手稿侧没有对应（C17）、map 与 gate 的计词口径不同（A13/A14）。 +3. **"一个单位"是声明而非契约**（C11）：制品记了 `unit`，没人读；hedging 的 section 在两侧不是同一对象。 +4. **分位数与百分位数的两套读法**：网格偏一位（C12）；docstructure 触发用秩规则、阈值与 LOO 用分位规则（B9）。两者都在样本最小处偏差最大，正是 30 单元下限存在的理由。 +5. **保真门的比较代数太弱**：集合比较（E12）、单向守卫（A12）、分词器对符号与标识符的处理（E1–E4）。标准 §6 说"确定性保真测试 + 语义比较"，目前确定性一半在四类常见输入上失效。 +6. **贪心搜索跨 manifold 比距离**（B10）：`manifold_operating_point` 的 docstring 禁止的事，partition 每跨一次分层边界就做一次。 +7. **标注抽样不是抽样**（E14/E15/E13）：每格通常来自一两篇论文，召回单位混杂，`--n` 基本无效。这三条一起决定 `label_findings` 出的精度/召回目前不可解释。 +8. **anchoring 的功效算术**（B5）与 **tier 权重**（D15）都是"文档承诺、代码未做"的同类：前者测试文件里写了算式，后者五处文档写了权重。 +9. **`verify_references` 是第四个 exit-1 契约**（H4/I30），标准 §0.1 只登记三个，RESPONSIBILITIES §8 没有它的行；而它 exit 1 的触发条件是 blocker，与 §0.1 "完整性审查不压进 L0 退出码" 的精神相抵。 +10. **Tier B 上限按 heading 单元而非 section**（F15）；SKILL 说 "同一 section 最多 1 次"。 + +## 5. 冗余与累赘 + +### 5.1 代码 + +| 编号 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---| +| M2 | 已复现 | build_profile, deai_anchoring, deai_docstructure, deai_features, deai_oracle, deai_partition, deai_personal, deai_provenance, deai_residue, deai_voice, extract_md_negatives, extract_style, length_gate, retrieve_exemplars, rewrite_reward, train_ai_ism_classifier, train_voice_model | 17 个工具 `import argparse` 未用（都改用 `cli_common`） | 删除；在 validator 或 CI 里加一步 pyflakes 类检查（现有 AST 未绑定名检查不覆盖未用名） | +| M2 / A21 | 已复现 | length_gate.py:22, 27+32, 339, 38/60-63 | `re` 未用；两次 `sys.path.insert`；except 元组里 `JSONDecodeError`/`UnicodeDecodeError` 是 `ValueError` 子类；`FRONT_MATTER`/`rest` 不可达（`section_line_ranges` 总覆盖全部行并标 `(preamble)`） | 删除 | +| M2 | 已复现 | extract_sections.py:21 | `import tex_macros` 无引用 | 删除 | +| B20 / M2 | 已复现 | deai_docstructure.py:10-18, 22, 25-26, 31-34；tests/test_deai_docstructure.py:318-329 | json/math/random/re/statistics/Iterable 与 `features`/`metrics`/`structure` 别名未用；四个常量先定义再被 docshape 的 import 覆盖；再导出契约测试因 `getattr(module, "__module__", …)` 对模块对象取默认值而把这些模块名也算进"必须再导出"的集合 | 测试过滤 `types.ModuleType`，然后删死 import/常量 | +| E22 / M2 | 已复现 | train_voice_model.py:21-29, 37；voice_audit.py:17-23；train_ai_ism_classifier.py:30, 38-39；voice_dataset.py:81 | 拆分残留：hashlib/math/re/statistics/defaultdict；hashlib/json/re/Counter；`DEFAULT_PROFILE_ROOT`；`list_fields` 定义未调用；函数内重复 `import re` | 删除 | +| B21 / C18 / D21 | 已核实 | deai_metrics.py:27-28, deai_features.py:43-44, deai_structure.py:23-24, deai_docstructure.py:29-30, deai_register.py:72-73, deai_oracle.py:38-39, deai_voice.py:33-34, build_profile.py:33, retrieve_exemplars.py:32-33, fetch_arxiv_abstracts.py:45, extract_md_negatives.py:51-52 | `REPO_ROOT`/`DEFAULT_PROFILE_ROOT` 各处重定义且未读（`cli_common` 已拥有） | 删除 | +| C18 | 已核实 | deai_reference.py:98-99；deai_discourse.py:264-277, 329；deai_register.py:598, 87-90, 111-116；deai_salience.py:87-90；deai_provenance.py:285-289；deai_voice.py:256 | 不可达分支 `if v_high == v_low`；`measured` 恒为 True 的分支；`min(1.0, count/10)` 在 `count>=15` 下恒 1（floor=5 时代遗留）；`RE_BIBITEM` 冗余；注释描述 v0.36.2 前的顺序；四个别名无人引用；provenance 文本输出重算一遍；`--scores` 用另一套分段导致索引与 finding 不对应 | 逐条删/改 | +| B19 | 已核实 | deai_metrics.py:310-312；deai_structure.py:355-357；deai_feedback.py:302-307；deai_oracle/deai_voice `paragraph_hits` | 兼容元组适配器无调用方（只有一个测试用 `tuple_hits`） | 删除并删测试，或在文档登记为公共 API | +| B22 / B23 | 已核实 | deai_docshape.py:176-194；deai_partition.py:59-75；deai_anchoring.py:107-121；deai_metrics/deai_structure `load_policy`；deai_features/deai_docshape 两个同名不同义的 `_MATH_MARKER_RE`；deai_features `_prose_words`、`re.split` 分段；anchoring 的第三个 section 分类器 | 三个轴各自重做 `deai_reference.paragraphs` 声称"每个轴都读它"的扫描，因此标题不置空、`skip` 不跳、float 不置空；`document_shape`/`document_anchoring` 每次 CLI 算两遍 | 统一消费 `reference.paragraphs/sections`；重建 docstructure/anchoring baseline | +| A22 / A23 / A24 / D21 / D23 / F28 | 已核实 | length_gate.py:67-69（`read_git_version` 一行别名，deai_residue 经它导入）；length_gate/condense_map 重复声明 `--format/--output`；extract_sections.py:286-288 死再导出；extract_style.py:624-630 重造 `field_parser`；`build_profile.list_fields` 与 `cli_common.list_fields` 字节等价；`retrieve_exemplars.VALID_SECTIONS` 手抄 `CLASSIFIED_BUCKETS`；占位符词汇三处编码；ai_ism_lint.py:76-80, 472-477, 503-504 重造 `list_fields`/`utf8_stdout`/`base_parser`/`field_options`/`report_options` | 直接调 `cli_common`/`tex_assembly`/`extract_sections` | +| E23 / E24 / E25 / E28 | 已核实 | eval_findings.py:88-110 ≡ label_findings.py:134-151（文档加载三份）；label_findings 每篇跑两遍发射器、eval_findings 两遍 register、voice_audit 每次 `confound_audit` 重算 4.4 万行的数学密度、train_ai_ism_classifier 两次 5 折；`eval_docscale.rank_auc` 与 `voice_audit._auc` 两套 midrank AUC（注释却说"一个实现"）；eval_docscale 从 retrieve_exemplars 借 `resolve_field`，错误信息署名 `[retrieve_exemplars]` | 合并 | +| F29 / F30 / F31 | 已核实 | validate_plugin.py:628-667（与 test_ai_ism_lint_cli 重复的退出码检查；`_discovered_test_count` 调两次）；tests/test_published_figures.py:98-101 = :111-114（`by_dot`≡`by_n`）；ai_ism_lint.py:337-355 docstring 被 test 逐字复制 | 留一份；memoize | +| M2 | 已复现 | tests：12 个文件 `import sys` 未用，5 个文件 `Path` 未用；两个正则体（`\\[a-zA-Z]+\*?`、`\\(?:begin\|end)\{[^}]*\}`）各在两个工具里重复编译 | 删除 / 共享 | +| A25 / E29 | 已核实 | tests/test_extract_style.py:407-465 与 tests/test_tex_assembly.py:22-29 同名同 docstring 的 `TexDocumentAssemblyTests`（测的是根选择）；`_bundle` 复制三次；train_voice_model.py:39-55 的 13 个私有名再导出只服务于自己的契约测试 | 改名；测试直接 import 拆分后的模块 | + +### 5.2 文档与 skill + +| 编号 | 位置 | 问题 | 建议 | +|---|---|---|---| +| G27 / G26 / G28 / G29 / G30 | README.md:543-584 ≡ README.zh-CN.md:514-555 ≡ tools/README.md:9-50；plugin.json:4 = marketplace.json:14（1,300 字符逐字相同）；validator 覆盖范围五处三种清单；可选依赖表四处；校准命令两份且互相缺项 | 40 行工具注册表三份，且已互不一致（0.246 vs 0.174、16.7%、3.1%、10 vs 11、"28 of 40" vs "21 of 30"） | tools/README 为注册表，README 只留工具+层+一句话；marketplace 用一行摘要；requirements.txt 为依赖之家；style-profile/README 为校准命令之家 | +| R1 / R2 / R3 / R4 | discourse-and-citation.md:29-59 及六处；held-out-labels.md:192-218 及七处；EVALUATION.md:55-110 + lexical-structure-uid.md:46-97 + narrative-salience-register.md:213-254；held-out-labels.md:131-160 及七处 | hedging 单位理由讲 7 遍；salience 投影缝讲 8 遍；2026-08-25 语料层两轮修复讲 3 遍（hub 56 行历史）；register 假阳性链（0.991→…→0.0371）讲 8 遍（含嵌套 "superseded" 引用块） | 各留一处之家 + 指针；估计省约 115 行 | +| R5–R10 | learned-model.md:46-50 = :126-130；document-scale.md:17-135 vs :448-455 等（同一量三四处）；docs/README + EVALUATION + §18.8 五处描述 `test_published_figures`；EVALUATION.md:170-186 与 DEAI_SUBSYSTEM.md:449-466 的 v0.14.0 发布门历史；DISPOSITIONS 的 Reason 单元格整段复述 §19/§20；DEAI_SUBSYSTEM L4 段复述 §23.4 | 状态文档里的版本历史按 docs/README 的规定应在 CHANGELOG | 估计再省约 105 行，多数落在接近 750 行上限的文件（document-scale 712、narrative-salience-register 614） | +| G31 | README.md:637-638, 718-719, 177-181, 729-734；README.zh-CN.md:704-722；style-profile/README.md:96-113 | Limitations 与 "hard parts" 逐版本复述 CHANGELOG | 表里只留当前值 + § 链接 | +| H23–H30 | skills/paper:394-428 ≡ de-ai:276-290（保留清单，logic/paper-review 再引一遍）；`landscape/demonstrate/significantly` 五处；claim-evidence 三处；de-ai §2 复述标准 §2（同一 `AUC/epoch/accuracy` 例子）；thesis-spine 三处；de-ai/condense 边界段逐字镜像；四个 primitive 各一份四条证据纪律；paper 内 ❌ 列表与表格重复例子 | skill 间约 130 行重复 | 各留一处之家（多数在 de-ai），其他改指针 | +| H45–H51 | paper:226-262（"根本层"复述标准 §1-2）、:162-176（复述 §5.3）；final-review 四处停止条件；brainstorm 四处 no-defer/上限语义 + 两份 flag 表；de-ai §0.1 与 paper-review 三处复述类与停止规则；standard §2 的 panel/L3 证据叙事（§9.8 说证据应在 EVALUATION，而 standard 正卡在 750 行上限） | 不改变 Claude 行为的叙事 | 估计省约 110 行 | +| M3 | docs/audits/codex-review-2026-09-04.md | 16 个 `D:/Projects/sci-paper/...` 绝对路径链接、33 行行尾空白、无末尾换行；I35：Part 1 承诺的逐条 disposition 从未补上（v0.36.2 修了哪些、A10 改名、A11 收窄，无处记录） | 链接改仓库相对路径；补 16 行 disposition 列 | + +## 6. 冲突、过时与文档失实 + +### 6.1 skill 与标准 / 工具 + +| 编号 | 严重度 | 位置 | 问题 | 建议 | +|---|---|---|---|---| +| H12 | 高 | skills/final-review/SKILL.md:203-250；standard:585-591 | 编排者在 §3.7 应用编辑、§4 宣布 disposition-complete，全程无快照、无 `length_gate`、无 `deai_residue --before`；标准说这两个 gate 缺 disposition 时循环不得收尾 | §2 加"首轮编辑前快照"，§3.7 收尾跑两个 gate，§4 加对应条目 | +| H13 | 中 | skills/de-ai/SKILL.md:112-116, 169-182, 308-348, 472-480 | §2 说 diff 规则 "gates the pass"，但 Pass 1 只静态跑 residue，Pass 3 与 §5 都不跑 `--before`；作为改写循环也没有 §5.3 的收尾 length gate | Pass 3 第 5 步或 §5 加两个 gate | +| H36 | 中 | skills/final-review/SKILL.md:112-119 vs :105-108 | 子进程被要求读"父进程产出的 references.json"，父流程没有生成它的步骤（CHANGELOG v0.36.3 说 final-review 每轮先跑） | 加 3.1b 步骤 | +| H5 | 中 | skills/paper-review/SKILL.md:288-293 | §O 列的规则缺 v0.37.0 的 `residue-absence`，动作句仍是 v0.37.1 已替换的"改写优先"形式 | 补规则 + 指向 paper 自检项 5 | +| H3 / I26 | 中 | standard:160-162（正确）vs de-ai:136-141、paper-review:146、RESPONSIBILITIES:53、DISPOSITIONS:57、tools/README:28、DEAI_SUBSYSTEM:127-130 | zero-hit 豁免代码有三类（defined-here / name / derived-form），"cited method's name" 明确不是机械豁免；四处只说词根，DISPOSITIONS 说"词根 + 定义 + 被引方法名"，DEAI_SUBSYSTEM 把两类推给作者 | 全部对齐标准措辞 | +| H4 / I30 | 中 | standard:56-72；verify_references.py:478；DEAI_SUBSYSTEM.md:330 | §0.1 登记"三个例外"，`verify_references` 是第四个（blocker → exit 1）；RESPONSIBILITIES §8 无其行（label_findings 也无，H44） | §0.1 加第四条 + §8 加两行，或改工具 exit 0 | +| H6 | 中 | skills/de-ai/SKILL.md:97-101, 525-526；build_profile.py:11-14 | 新鲜度步骤只让重跑 `extract_style`/`build_profile`（且无 `--field`），六个 `--calibrate` 轴仍是旧参考却报 `measured` | 改为"跑 `/sci-paper:calibrate ` §3" | +| H1 / H2 | 高 / 中 | 见 §3.5 | 强制门依赖可选模型；`--field` "auto-detected" 承诺 | 见 §3.5 | +| H14 | 中 | standard:402-412；de-ai:195-198、paper-review:157、physics:119、mainline:72 | 标准把 strong advisory 定义为 measured + 有参考 + 超操作点；四个 skill 按人工判断升 strong | §4 加一句允许"reviewer judgement"为基础，或四处降为 ordinary + priority boost | +| H15 | 中 | skills/physics/SKILL.md:47-48 vs :119 vs :155 | 未标注 schematic 公式：P1 与反模式说 blocker，§4 表说 strong advisory | 表里限定"量纲闭合时" | +| H16 | 中（待核实） | skills/proposal-polish/SKILL.md:98-100, 137-141, 153 | 要求把 hedged 中心假设改为承诺式，同时自绑 §6 的 stance/modality 不变量 | 改为作者审批项 | +| H17 | 中（待核实） | skills/mainline/SKILL.md:10, 45-49；standard:599-601 | 引 §5.4 为权威，又允许多个贡献分支；§5.4 说"exactly one central result" | 加一句"分支必须从属于唯一 thesis line"，七问加 thesis 一问 | +| H18 | 中（待核实） | skills/paper-review/SKILL.md:153-160, 191-193, 386-388；final-review:30-32, 130-155 | `--orchestrated` 只跳 physics/mainline/logic；D 维每轮仍跑 de-ai 审计、G 维仍调 figure-review，父进程又各起一个隔离 agent，重复审计 | `--orchestrated` 也把 D/G 置 SKIPPED_FOR_ORCHESTRATOR | +| H7 / H8 / H9 | 低 | skills/paper/SKILL.md:278 vs :346-347；standard:131-135 vs de-ai:124-127 vs paper:289-299；standard:35 | grep 块把段首 `Importantly/Notably/Interestingly` 标成 Tier B（表与 linter 是 Tier A）；三处对"词表之家"说法不同且无镜像检查；标准称 Tier B 上限"calibrated"，实为常量 1 | 拆 grep；声明 linter 正则为准并加 validator 镜像；改措辞 | +| H10 / H11 / H41 / H43 | 低 | de-ai:104-107；paper-review:288-291 + condense:135-137；proposal-polish:108-111；physics:37 vs figure-review:39-42 | 检索命令缺 `--allow-fallback`（无 sentence-transformers 即报错）；负标签静态规则 400 词下限无 skill 提及；run-in 冒号会被 linter 报却说不适用；两套"权威构建"配方 | 各加一句 | +| H19 / H20 / H21 / H22 / H42 | 低 | calibrate:8 vs RESPONSIBILITIES:37-39 + validator `NORMATIVE_SKILLS`；standard:224 vs :512；paper-review:264-272 vs brainstorm:276-357；final-review:151-152 vs de-ai:183-185；paper-review:436-447 vs :353-354 | calibrate 自称"非规范 skill"而 validator 与 annex 说是；anchoring 一处 L2 一处 L4；同一 12-framing 引擎一处禁 NEEDS-MORE-INFO 一处常设；final-review 期待 `--audit-only` 返回 partition 建议而 de-ai 把它放在 Pass 3；§7 停止语义漏 residue gate | 逐条对齐 | +| H37 / H38 / H39 / H40 | 中 / 中 / 低 / 低 | paper-review:242-250 vs physics:42-79（M.1 复述 P1-P6，违反自身组合规则；K 维单独运行未定义）；paper:15, 196-215（未标 `[WGL]` 的 ML/弱透镜指引）；condense/calibrate/brainstorm/final-review 的 argument-hint 列出正文未定义的 flag；brainstorm `--min-frameworks` 是 no-op，并依赖公共插件不带的 cc-enforcer/FACTS.md/cc-tree | 对齐；标 `[WGL]`；定义或删 flag；外部引用改为可选 | +| H31 / H32 / H33 / H34 / H35 | 低 | standard:9-15（仍标 v3.8 2026-09-04，但 v0.37.0/v0.37.1/v0.38.0 改了三处）；brainstorm:56 叙述已删 flag；paper:164, 280-283, 296-298, 327, 430-440 带日期的来源说明当作现行政策；README.md:77 "35 tools"、:70 `--audit-only` "stops after measurement"（实为 Pass 2 后）；standard §7-8/§11 与两份 annex 头四处讲迁移史，DISPOSITIONS 自身"Adoption requires…"句出现两次 | 升 v3.9 并说明；删旧叙述；来源说明移 ACKNOWLEDGMENTS；改数字；迁移史各留一句 | + +### 6.2 证据文档中的数字（I、G） + +| 编号 | 严重度 | 位置 | 过时值 → 现值 | +|---|---|---|---| +| I1 / G22 | 高 | EVALUATION.md:123；document-scale.md:147, 194-196, 436-437；DEAI_SUBSYSTEM.md:277 vs :460；README.md:398-416 | document-shape 参考 493 篇（hub、§9 全部操作点 16/493、21/493）与 "14-paper" → 制品 507 篇（v0.36.3 重建），§9 没有重评分记录；README 同时描述 507 制品与 493 时代的判别数字 | +| I2 | 高 | EVALUATION.md:124 | L3 行 0.932 / 0.924 / 0.937 / 32–41% → 0.9487 / 0.9262 / 0.938 / 30–42%（commit 7d5e865 只改了两份 README）；FPR 区间三种写法 | +| I3 | 中 | EVALUATION.md:125-126；DISPOSITIONS.md:47；README.md:720；tools/README.md:30；discourse-and-citation.md:180-181 | cohesion/hedging bucket 计数与转移率 10.87%/7.89% 是 v0.33.0；§19.1/§19.4 现为 6,886/3,228/… 与 10.78%/7.80% | +| I4 | 中 | narrative-salience-register.md:312-325；DEAI_SUBSYSTEM.md:206；DISPOSITIONS.md:59；vocabulary-and-residue.md:5 | register 41,710 / 53,414 → 41,644 / 53,367；摘要占比 33% vs 35% 两处不一 | +| I5 / G20 | 中 | vocabulary-and-residue.md:95-121；DISPOSITIONS.md:59；README.md:480；README.zh-CN.md:438, 695 | collocation 11,286 / 530,677 与旧 bucket 表 → 11,282 / 530,504；hub 的 "7,237 findings" 无出处 | +| I6 | 低-中 | vocabulary-and-residue.md:167-175；EVALUATION.md:119 | §23.3 结构族 n 列是 v0.32.0 bank（3,840/9,512/…），制品现为 3,812/9,478/… | +| I7 / G16 | 中 | DISPOSITIONS.md:57；tools/README.md:28 | zero-hit 2.21 每千词、AUC 0.246 → 3.37、0.174 | +| I8 / G18 / A29 | 中 | DISPOSITIONS.md:62；tools/README.md:23 | condense 中位目标 3.1% → 1.59%（p90 4.11%） | +| I9 / G17 | 低-中 | tools/README.md:24 | residue strong 文档率 16.7% → 13.3% | +| I10 / G15 | 中 | README.md:716；README.zh-CN.md:690 | 稳定性 ρ 0.846 / overlap 0.654 → 0.991 / 0.889（7d5e865 漏改此行） | +| I11 / G1 / F16 | 中 | README.md:666；tools/README.md:50；DEAI_SUBSYSTEM.md:426-440 | "10 checks" → 11；validator 的形状检查正则要求 "contract checks" 字样且不扫 tools/README，所以漏网 | +| G14 | 高 | README.md:390-392 vs examples/README.md:46, 77-81, 100-105 | 示例表 salience-recital 4 → **6**（2026-08-27 读数），examples/README 已改为 4 → 3（v0.36.3）；`test_published_figures` 只钉 examples/README | +| G19 / M4 | 中 | README.md:477；README.zh-CN.md:435；latency.json | 延迟行把 529 个测试与 523 时代的墙钟配对；latency.json 是 v0.36.3 的 516 时代数据、无人引用 | +| I12–I20 | 低 | EVALUATION.md:117, 143, 94；lexical-structure-uid.md:123；narrative-salience-register.md:401-406, 210；projection-and-operating-point.md:237, 273-275, 123；document-scale.md:37-39 vs :95-97 vs :436-437, :113 vs :482, :388 vs :554；held-out-labels.md:47, 226 | 0.2775 vs 0.2776 vs 0.2705；529 个测试标日 2026-09-05（应 09-16）；"104×/85×" floor 倍数是 v0.28.0；UID 0.26–0.58 vs 表 0.26–0.51；"三次重训"已是四次；"pins 39 figures" 现为 61；同一量两套分母/两个 r 值无版本标签；held-out 200 vs 203 未解释；"sit above" 实为 below | +| G2 / G3 / G4 / G5 / G6 / G7 / G8 / G10 / G12 / G13 / G23 / G24 / G25 / G33 / G34 / G35 / G36 | 低-中 | README.md:77, 576, 554, 191-192 vs 270-271, 72；style-corpus/README.md:66-69；style-profile/README.md:14-43, 70-94, 36；style-corpus/wgl/README.md:3-5, 26-27；examples/README.md:52-54, 86-88, 24, 28-47；README.zh-CN.md:576-577, 381-384；docs/README.md:78, 57-66 | "35 tools"；cli_common "28 of 40"（实 29）；negatives 文件标层 L0；demo 复现 flag 前后不一；paper-review 行少 P 维；语料 README 说无最少论文警告（有）；profile README 缺 discourse/anchoring 校准与 negatives 文件名；wgl README 是 v0.1 措辞；examples/README 说 headline 对在 discussion 复述（在 abstract）、日期 09-04 vs v0.36.3 09-05、before 合计对不上；中文 README 说 demo 2 "停在三条 advisory"（实为 0）、估计噪声模型"未发布"（已建并证伪）；long-form AUC 0.740 vs 0.729 无协议标注；docs/README 提到仓库不带的"editing hook"、索引缺 examples/README | +| G11 | 中 | README.md vs README.zh-CN.md | 结构级差异 16 项：英文有 §6 worked example 中文无（中文从不链接 examples/）；中文有 Claude Code 驱动块、归档链接、14 行 limitations（英文 13 行且集合不同）、长版 roadmap（Codex 模型名、12.5× 引用密度）而英文无；493 vs 507；工具表行长短与数字不同 | 决定一份为准，另一份逐节对译 | + +### 6.3 代码声明与过时说法 + +| 编号 | 位置 | 问题 | +|---|---|---| +| I21 | EVALUATION.md:121；DISPOSITIONS.md:60；RESPONSIBILITIES.md:55；DEAI_SUBSYSTEM.md:306；deai_residue.py:4 | residue 规则数四种说法："three static + one diff"（hub、DISPOSITIONS）、"Four static and one diff"（代码）、"Five rules"（DEAI_SUBSYSTEM）；规范附件 RESPONSIBILITIES 的必需行为里没有 `residue-absence`；DISPOSITIONS 无该规则的行；代码 docstring 编号 1、2、3、5、6，没有 4 | +| I25 / B26 / C20 | DEAI_SUBSYSTEM.md:158；deai_structure.py:59-63；tests/test_deai_structure.py:79-81 | paper-as-agent 例句 "This paper presents" 正是检测器刻意排除的（测试 `test_paper_that_merely_presents_is_not_an_agent`）；v0.36.2 只改了标准与 skill | +| I22 / I31 / I32 / I33 / I34 | EVALUATION.md:124, 163-168, 184-186, 128；DEAI_SUBSYSTEM.md:137-143, 290-291, 414；document-scale.md:515-516 | hub 仍"要求" L3 操作点（§7.0a 已决定不可得）；§12 保留 v0.28.0 "this release" 门与已决定的"open"作者决定；DEAI_SUBSYSTEM 把已撤回的 `deai_policy.json`/已证伪的 L1 操作点写成待定；§9.4b "stays open" 而 §9.4c 已证伪；§8 的空格单位边界"remains open"无 disposition 行，与 v0.36.3 "no loose ends" 相抵 | +| I36 | design-notes/DEAI_FRONTIER.md:6-8, 143-144；DEAI_ARCHITECTURE_ROADMAP.md:3-8, 188-211 | "冻结"设计笔记在冻结日期后被编辑成状态记录，且内容已错（rank 2 已 Done、rank 6 已 shipped、§11 已迁到 DISPOSITIONS） | +| I23 / I27 / I28 / I29 / I37 / I38 / I39 / I24 | DISPOSITIONS.md:57；DEAI_SUBSYSTEM.md:188-189, 213, 122-125, 420-422 及十余处数字；tools/README.md 九行；RESPONSIBILITIES.md:48, 52, 54；DEAI_SUBSYSTEM.md:362-385 | "Superseded/replaces the knob" 与"beside the thresholded rule"矛盾（15 用阈值仍在发射）；"sole consumer of `latex_to_numeral_text`" 自 v0.36.3 起 bank 也用；pair-break 集与权重名滞后 v0.36.2；zero-hit 投影漏 floats；DEAI_SUBSYSTEM 声明数字属于 EVALUATION 却引十余处（三处已过时）；tools/README 九行带未钉住的评估数字（四处已错）；RESPONSIBILITIES 说 linter "optional L1-L3"，实际默认跑 L2 collocation 与 L4 residue；DEAI_SUBSYSTEM §7 只列 8 个 skill | +| B24 / B25 / F35 / F17 / A26 / A30 / C19 / D25 / E26 / E27 | deai_docstructure.py:166-182 + tools/README.md:33；deai_metrics/deai_structure 的 `deai_policy.json` 分支；tools/README.md:50；validate_plugin.py:285-286, 386；tools/README（无 tex_macros 行）；tex_macros.py:22-24, 31-34, 47-51；extract_sections.py:203-207；deai_reference.py:220-221, 273, 304；deai_register.py:27, 447-451；style-corpus/README.md:14-20；style-profile/README.md:36, 96-113；extract_md_negatives.py:5, 27；fetch_arxiv_abstracts.py:46, 193, 255；voice_audit.py:11；train_ai_ism_classifier.py:55-56, 231；voice_dataset.py:109；learned-model.md:207-210, 243-249；tools/README.md:45 + README.md:573 | 无 conformal 块的"legacy baseline"路径今日任何 `--calibrate` 都不会产生、无测试；`deai_policy.json` 无写入方且键未文档化，`measured/strong` 分支实际死；"no shipped document carries an edit-meta literal" 只扫 3 份文档 + skills；两处注释指向不存在的"file-link check"；tex_macros 三处说法与行为相反；`_project` "same substitutions" 在 `$$` 下不成立；`has_prose`/`_bank_records` docstring 与实现不符；register AUC 在代码 0.242 / tools README 0.246 / 记录 0.174 三个值；`defined_terms` docstring 方向写反；style-corpus 布局缺 `fulltext-*`；profile README "Recalibrate after v0.28.0" 已两次重建过时；extract_md_negatives 带个人绝对路径且说 "appended"（实为覆盖）；fetch 的 UA `sci-paper-voice/0.13|0.14`、API 用 `http://`；voice_audit "Fits no model"（实际每次划分拟合两个逻辑回归）；classifier 文件里三个时代的语料数并存；指纹注释"覆盖每个输入"不成立；两份 README 说 `eval_findings` 评 cohesion/hedging 而代码刻意只做四轴 | + +## 7. 测试 + +| 编号 | 状态 | 位置 | 问题 | 建议 | +|---|---|---|---|---| +| A31 / D26 / E33 / B28 | 已核实 | tests/test_extract_style.py:211；test_fetch_arxiv_abstracts.py:495；test_label_findings.py:108；test_deai_docstructure.py:304 | `if __name__ == "__main__": unittest.main()` 写在文件中部，直接运行只跑前面的类（17/58、33/45、6/16、13/16） | 移到文件末尾 | +| E31 | 已核实 | tests/test_train_voice_model.py:67-99 | AST 未绑定名检查把全模块所有绑定（含其他函数内的局部 import）并成一个集合，只要任何函数 `import numpy as np`，别的函数裸用 `np` 也过；它当初要抓的 `time` 只要在文件别处绑定就抓不到 | 按作用域解析（`symtable`/pyflakes），至少函数内 import 只算该函数 | +| F37 | 已核实 | tests/test_published_figures.py:348-353 | `assertIs(needs_profile, needs_profile)` 恒真；无 profile 时该文件 8 个测试跳 6 个，这是仅剩两个之一 | 用 dummy 测试类验证 skip，或删除 | +| E32 | 已核实 | tests/test_rewrite_reward.py:265-277 vs rewrite_reward.py:462-480 | "不可读输入 → 2" 的测试在读文件前就因 `--field no-such-field` 返回 2，崩溃兜底从未被触达，删掉兜底测试仍绿 | mock bundle + 让 `rank` 抛错 | +| C22 | 已核实 | tests/test_deai_salience.py:162-179 | 测试钉住"无 `numeral_text` 的 bank 校准得 p90 == 0.0"为正确行为，把静默投影回退固化 | 改为要求警告/degraded 记录 | +| D29 / A32 | 已核实 | test_fetch_arxiv_abstracts.py:145-147, 230-234, 286-299；test_length_gate.py:152-153；test_tex_macros.py:41-43 | 断言字面定义的测试；`issubclass(Throttled, Exception)` 恒真；再导出契约测试排除的正是 extract_sections 自己再导出的三个 tex_assembly 名；注释"九个词"实为 12；带参数宏测试用非数字 body 故抓不到 A9 | 修正 | +| B30 / C22 / D27 / D28 / E34 / F38 | 已核实 | 全库 | 无测试 import `deai_metrics`；conformal/Mondrian 端到端路径（需 ≥55 篇）从未被测；`deai_structure.calibrate`/`strong` 逻辑无测试；partition 只测 unmeasured 路径且用词多重集比较看不到 B2；PDF/连字路径从未端到端跑过（可用假 `pymupdf` 模块）；`_build_or_load_embeddings` 失效与位置对齐无测试；`cmd_sample`、`--n`、根位置 `--field`、`_tex_members`、`fetch_fulltext`、`--author` + REFERENCE_DIR 拒绝、`--glossary`、`--ai-classifier`、`.md` 输入、eval_docscale 整体（无测试文件）均无测试；§3 里每个已复现缺陷都缺回归测试 | 优先给 §8 第一档的修复各配一条两行测试 | + +## 8. 建议动作分级 + +1. **应修（行为错误，含门禁失效）**：A1/A2/F3（根文件不可读 → 2，不是空文档）；A4（`--git-ref` 子文件按 ref 解析）；A3（精确分数）；A11/A12/A13/A14（carve-out 双侧、守卫双向、map 与 gate 同一计词口径）；A5/A6/A7（`\[…\]`、`$$`、注释标题）；E1/E2/E3/E12（保真分词器与多重数比较）；H1（无 voice 模型时降级而非 exit 2）；M1/C2/E7（`--field` 解析统一，子命令默认值不覆盖根解析）；B5/C10/C13/B6/B7（无功效或无 bucket 时报 degraded）；C12/B9（分位读法统一）；B10/B2/B3（partition 同 manifold 比较、原文切分、subsection 固定）；B12/B13/C3/C5/C4（`skip`/参考文献/标签投影）；F1/F2/F6/F9/F4/F7（linter 与 references）；D1/D2/D4/D5/D6/D7/D16（抓取与语料统计）；E5/E6/E8/E13/E14/E15（评估与标注）；C11（读并校验制品 `unit`）；A27/A28（condense 配方）。修改 B12/B13/B15/B22/C12 后需重建对应 baseline 并重取 EVALUATION 相应数字。 +2. **应改文档/元数据**：§6.2 全部数字（I1–I20、G14–G25）；§6.1 中 H3/H4/H5/H6/H12/H13/H36 的 skill 补丁；I21（RESPONSIBILITIES 补 `residue-absence`）；I25；I36（设计笔记恢复冻结）；I22/I31–I34（关闭 stale open）；tools/README 去评估数字（I38）；G11 中英 README 对齐；G27 注册表只留一份；D15 二选一（实现权重或改五处措辞）；标准升 v3.9（H31）。 +3. **可延后**：§5 的 import/常量清理（一次 pyflakes 扫描即可批量完成）；R1–R10 与 H23–H30/H45–H51 的去重（约 350 行，且多数文件接近上限，值得排期）；B16/B17/B18/E17/E18/E20/F22/F26/D20 等待核实项；测试补齐（§7 末行）；latency.json 重取或删除（G19）；上次审计的 disposition 列与本机路径链接（M3/I35）。 + +## 9. 已检查且未发现问题（摘要） + +- 全部 27 个测试文件、validator 11 项、`-X dev` 警告即错误模式下均通过；无已弃用 stdlib API；无 CRLF/BOM；分支与 `main` 同一提交。 +- 退出码契约在正常路径上成立：`ai_ism_lint` 0/1/2、`length_gate` 与 `deai_residue` 的 0/1/2、`rewrite_reward` 0/1/2、`verify_references` 0/1/2、抓取器摘要模式的 429 → 2；`--top` 不改变计数与汇总；JSON 键稳定。 +- `parse_required_shrink` 对 `100%`/`200%`/`inf`/`nan`/`30%%` 的拒绝与 `30%`/`0.3`/`1e-1` 的接受；装配的行内 `\input`、重复 include、环、注释掉的 include；跨行 `\citep[e.g.][]{…}` 与 float 置空；`\section {X}`/`[S]`/subsection 继承（deai_metrics/extract_sections 侧）；bank 行 schema 与所有读者一致；`paired_paragraphs` 两投影同构（`$$` 除外）；REFERENCE_DIR 不加权、不进聚合、默认不进检索。 +- `deai_feedback` 校验、优先级排序（与 §4 一致）、0.5 段落置信上限；`_mat_inv`/Mahalanobis/conformal `_conformal_p`/分层回退/角色 z 分数正负号一致；`fit_dispersion_manifold` 33 行下限；voice 流水线的训练集内 centroid、词表、分组划分、原子写出、schema 漂移拒绝；`rank_auc` 与 `_auc` 与暴力 tie-aware AUC 在 3000 例上一致；kappa 与独立实现一致；20 标签下限；blind relabel 剥离字段正确。 +- register 的多行置空、宏项、复合词最稀部分、三类 zero-hit 豁免、`wgl-letter` 借用；collocation 的断点集、留一法、可能所有格首词；discourse 状态与检测同一 `live_buckets`、hedging 限 `intro`；salience 的 `numeral_text` 与 `latex_to_numeral_text` 占位符一致、每段一条 finding、低于下限的 bucket 给 degraded finding 而非沉默；residue 规则 6 家族、`never(?!-)`、`[CITE]` 豁免、强弱顺序；validator 镜像检查；CUDA 回退可达且不泄漏（除 C7 的空消息情形）。 +- 文档：所有 43 个 markdown 的相对链接、`tools/*.py`、`skills/*` 引用均可解析（除 M3）；docs/README 的 section map 与磁盘一致；skill 里引用的 EVALUATION §号全部经 hub 解析；没有 skill 引入 PASS/FAIL 或作者身份断言，没有 skill 豁免 integrity blocker；README 引用的所有 flag 存在；文档常量（15 / 1e-4 / 500 / 2e-4 / 4 / 2 / 30 / 0.90 / 0.95 / 0.10 / 0.05 / 150 / 3 / 400 / 0.60 / 0.20 / 0.5 / 3 篇 / 0.40 / 20 / 0.05 / 3 层 / 0.6）与代码一致；v0.36.3 重测数字在已更新的位置内部一致，过时的恰是 §6.2 列出的位置。 From 49624e5bbdf6b5c9cf2beb5f6ffdcd96b5d88f17 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 05:34:22 +0000 Subject: [PATCH 02/27] cli_common: one optional-field resolver and axis hooks for the tools that diverged The shared --field option promised auto-detection while five tools divided the profile root by None and crashed, the axis tools never auto-detected, and two tools carried private loops. optional_field_dir resolves it once: an explicit field as given, a single field on its own, several with one stderr note and no profile. axis_main uses it on the read path, requires an existing field for --calibrate (exit 2 otherwise, and exit 2 when the calibrator read no records instead of writing an empty artifact), and grows extra_arguments / check / report_for hooks so an axis flag such as --glossary goes through argparse instead of being stripped from argv. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- tools/cli_common.py | 78 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 9 deletions(-) diff --git a/tools/cli_common.py b/tools/cli_common.py index df61982..084ada1 100644 --- a/tools/cli_common.py +++ b/tools/cli_common.py @@ -14,9 +14,10 @@ Two divergences are kept on purpose rather than folded in: -- `ai_ism_lint.resolve_field` **warns and returns None** instead of raising, - because the linter must still run its L0 pass with no profile present. That is - a behavioural difference, not duplication, so it keeps its own copy. +- `ai_ism_lint.resolve_field` returns None instead of raising (warning on an + explicit field that does not exist), because the linter must still run its + L0 pass with no profile present. That is a behavioural difference, not + duplication, so it keeps its own copy. - `extract_style` resolves against the *corpus* root, where a `tier-*` subdirectory is not a field. That is expressed here as `exclude_prefixes` rather than as a second implementation. @@ -102,7 +103,12 @@ def axis_main(doc: str | None, argv: list[str] | None, *, tool: str, calibrate: Callable[[Path], Any], summary: Callable[[Any, Path], str], report: Callable[[str, Path | None, Path], dict], - render: Callable[[dict], str]) -> int: + render: Callable[[dict], str], + extra_arguments: Callable[[argparse.ArgumentParser], None] | None = None, + check: Callable[[argparse.Namespace], str | None] | None = None, + report_for: Callable[[argparse.Namespace], + Callable[[str, Path | None, Path], dict]] | None = None, + ) -> int: """The `--calibrate`-or-read-one-file CLI every calibrated axis shares. Three axis tools carried a near-byte-equivalent copy of this body before a @@ -111,27 +117,81 @@ def axis_main(doc: str | None, argv: list[str] | None, *, tool: str, functions produce the findings and the axis statuses. `render` is injected for the same reason `emit_report` injects it -- so this module goes on holding no report schema and no policy. + + The field is optional on the read path and resolved by `optional_field_dir` + (one field auto-resolves; several print a note and run without a profile). + `--calibrate` needs a field that exists under the profile root, and a + `calibrate` that read no reference records returns None, which is exit 2 + rather than an empty artifact reported as written. An axis with a flag of + its own registers it through `extra_arguments`, rejects a bad combination + through `check` (return the message), and picks its report function per + run through `report_for`; `--glossary` used to be stripped from argv by + hand, which let it ride silently beside `--calibrate`. """ utf8_stdout() parser = field_parser(doc) parser.add_argument("file", type=Path, nargs="?") parser.add_argument("--calibrate", action="store_true") + if extra_arguments is not None: + extra_arguments(parser) args = parser.parse_args(argv) - field_dir = args.profile_root / args.field if args.field else None + if check is not None: + message = check(args) + if message: + print(f"[{tool}] {message}", file=sys.stderr) + return 2 if args.calibrate: - if field_dir is None: - print(f"[{tool}] --calibrate needs --field", file=sys.stderr) + try: + field = resolve_field(args.field, args.profile_root, tool=tool, + empty_hint="--calibrate needs an existing " + "style-profile// directory.") + except SystemExit as error: + print(error, file=sys.stderr) return 2 - print(f"[{tool}] {summary(calibrate(field_dir), field_dir)}") + field_dir = args.profile_root / field + written = calibrate(field_dir) + if written is None: + print(f"[{tool}] --calibrate read no reference records under " + f"{field_dir}; nothing written.", file=sys.stderr) + return 2 + print(f"[{tool}] {summary(written, field_dir)}") return 0 if not args.file or not args.file.exists(): print(f"[{tool}] file not found: {args.file}", file=sys.stderr) return 2 + field_dir = optional_field_dir(args, tool=tool) text = args.file.read_text(encoding="utf-8", errors="replace") - print(render(report(text, field_dir, args.file))) + chosen = report_for(args) if report_for is not None else report + print(render(chosen(text, field_dir, args.file))) return 0 +def optional_field_dir(args: argparse.Namespace, *, tool: str) -> Path | None: + """The profile directory for a tool that can run without one. + + An explicit `--field` is used as given: a directory that does not exist + means "no profile", which every axis already reports as `unmeasured` with + the missing artifact named. Without it, exactly one field under the + profile root resolves on its own, as the option's help text has always + promised; several print one stderr line naming them and return None, so + the run continues with every calibrated axis `unmeasured` for a reason a + reader can act on; none returns None. Before this helper the same option + was resolved four ways behind one help string: five tools divided the + root by None and died with a TypeError, the axis tools never auto-detected, + and `deai_oracle` and `deai_metrics` each carried their own loop. + """ + if args.field: + return args.profile_root / args.field + fields = list_fields(args.profile_root) + if len(fields) == 1: + return args.profile_root / fields[0] + if len(fields) > 1: + print(f"[{tool}] several field profiles present ({fields}); pass " + "--field= to use one. Running without a profile.", + file=sys.stderr) + return None + + def list_fields(root: Path, *, exclude_prefixes: tuple[str, ...] = ()) -> list[str]: """Non-hidden subdirectories of `root`, minus any excluded prefix.""" if not root.exists(): From 5ce670cae41f797975f5d4f68776237ca36c02a9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 05:54:29 +0000 Subject: [PATCH 03/27] fix: gates read missing evidence as a verdict, one display-math form, the map's carve-out and counts, and the linter's line-bound rules Audit items A1-A20, A28, F3, F4, F7, F9, F12, F14, F16-F20, F25, F27, F28, F33, D6-D11, D14, D16, D19 and M2/M3 (docs/audits/full-review-2026-09-27.md). tex_assembly / tex_macros / extract_sections - a root that cannot be read raises instead of reading as an empty document: the gate no longer scores a mistyped --before as growth in every section, and the linter no longer exits 0 on a directory named like a manuscript; only a child may stay in place unresolved; - a --git-ref baseline resolves its \input children AT THE REF, so a section file deleted from the working tree stays in the baseline; - \input{fig.tikz} is read as written, a nested \input resolves against the document root's directory first, and \includegraphics / \inputminted no longer read as include calls; - an argument-taking macro with a numeric body is left alone, and a commented-out definition is neither harvested nor dropped; - \[ ... \], $$ ... $$ and subequations are display mathematics in both projections, so the two views make the same substitutions again. length_gate / condense_map - exact percentages (7% of 100 is 7, not 8); --require-shrink 0 means no cut required, so a map with nothing to cut can still close the pass; - comments are blanked before segmenting, so a commented-out heading opens no section; dead front-matter code and duplicate imports removed; - the abstract/conclusion carve-out applies to either side of a restatement pair, and a body sentence is the canonical home when one exists; a copy whose negations or numbers differ in either direction is not a restatement; every removable-word count is the gate's prose count, the budget denominator excludes heading words, any \...ref command keeps a float alive, \caption* and nested captions are read, (Gaussian) is not an acronym, and a case-folded verbose match cannot raise KeyError. ai_ism_lint / validate_plugin - paragraph-initial connectors are judged on a paragraph's first prose line, openers where a sentence starts; paved/showcased join Tier A; every em-dash on a line is a target with its own id; \label/\ref/\cite/ \url arguments are blanked before the lexical scan; three-parallel needs whole words; an unparseable lexicon is exit 2; Markdown front matter is metadata; several profiles without --field say so on stderr; the CLI is built on cli_common and --summary is hidden; - the validator's shape regex now reads "N checks" and scans tools/README, and a relative-link pass reports targets that do not exist (it found the 16 local-path links in the 2026-09-04 audit note, now repository-relative). extract_style / retrieve_exemplars - multi-word openers can be observed; placeholders are not words and do not drop a citation-led paragraph from the bank; words are Unicode letters; bank ids carry the bucket; the profile directory is created after the corpus check; skipped PDFs are counted in the summary; an empty --topic ranks by section name in the keyword fallback. Tests: 38 new (the extractor's moved to tests/test_extractor_statistics.py, the original file being at the line budget); the suite is 567 tests in 28 files, recorded counts to be updated with the release notes. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- docs/audits/codex-review-2026-09-04.md | 68 +++---- tests/test_ai_ism_lint_cli.py | 109 ++++++++++ tests/test_condense_map.py | 69 ++++++- tests/test_extract_style.py | 4 + tests/test_extractor_statistics.py | 111 +++++++++++ tests/test_length_gate.py | 84 +++++++- tests/test_tex_assembly.py | 65 ++++++ tests/test_tex_macros.py | 11 ++ tools/ai_ism_lint.py | 209 +++++++++++++------- tools/condense_map.py | 84 ++++++-- tools/extract_sections.py | 14 +- tools/extract_style.py | 262 +++++++++++++------------ tools/length_gate.py | 60 +++--- tools/retrieve_exemplars.py | 7 +- tools/tex_assembly.py | 128 ++++++++---- tools/tex_macros.py | 48 +++-- tools/validate_plugin.py | 89 ++++----- 17 files changed, 1030 insertions(+), 392 deletions(-) create mode 100644 tests/test_extractor_statistics.py diff --git a/docs/audits/codex-review-2026-09-04.md b/docs/audits/codex-review-2026-09-04.md index ed6e5fc..fb88264 100644 --- a/docs/audits/codex-review-2026-09-04.md +++ b/docs/audits/codex-review-2026-09-04.md @@ -70,54 +70,54 @@ Read-only review of `git diff 9f0b920..edd480c`; session `01a06ebe-b4c5-7802-bfe 以下 `M = "\\section{Methods}\n"`；字符串中的 `\n` 表示实际换行。数值均由当前代码在内存中复现，未修改文件。 -1. **major · [tools/condense_map.py:337](D:/Projects/sci-paper/tools/condense_map.py:337) · 删除预算重复计算同一文字。** +1. **major · [tools/condense_map.py:337](../../tools/condense_map.py) · 删除预算重复计算同一文字。** 触发：令 `s = "In this section we describe precise galaxy cluster signal maps calibrated against independent external measurements."`，输入 `M + "\n\n".join([s] * 3)`。实际 `prose_words=46`，默认目标 **75 words**：三个 zero-gain 加两个 restatement 重叠计数。建议：保存可删除字符区间，计算区间并集；分别报告候选质量和确认可删除质量。`tools/condense_map.py:328`、`tools/condense_map.py:343` -2. **major · [tools/condense_map.py:144](D:/Projects/sci-paper/tools/condense_map.py:144) · 相同词袋被当作相同科学声明。** +2. **major · [tools/condense_map.py:144](../../tools/condense_map.py) · 相同词袋被当作相同科学声明。** 触发：在 Methods 中依次写 `The calibrated galaxy cluster mass signal measurement across independent radial apertures is significant.` 和把 `is significant` 改成 `is not significant` 的版本。第二句被判 restatement，计入 14 words 目标；数字变化也不会进入该词集合。建议：先核对 negation、numbers、units、comparisons 等 protected invariants；仅有 lexical overlap 时不能生成整句删除目标。`tools/condense_map.py:97`、`tools/condense_map.py:150` -3. **major · [tools/condense_map.py:173](D:/Projects/sci-paper/tools/condense_map.py:173) · 开头套语被等同于整句无信息，且 zero-gain 没有 genre carve-out。** +3. **major · [tools/condense_map.py:173](../../tools/condense_map.py) · 开头套语被等同于整句无信息，且 zero-gain 没有 genre carve-out。** 触发：`\section{Conclusion}\nIn summary, we measure a galaxy mass of five units.` 被要求删除全部 **10 words**；`M + "In this paper we measure a galaxy mass of five units."` 被计入 **11 words**。建议：默认只预算删除 opener；整句删除须确认无独有声明；对 zero-gain 同样应用 abstract/conclusion carve-out。`tools/condense_map.py:178`、`tools/condense_map.py:183`、`tools/condense_map.py:341` -4. **major · [tools/length_gate.py:200](D:/Projects/sci-paper/tools/length_gate.py:200) · shrink 参数的单位、取整及边界不可靠。** +4. **major · [tools/length_gate.py:200](../../tools/length_gate.py) · shrink 参数的单位、取整及边界不可靠。** 触发：`100% → int(1)`、`200% → int(2)`；5-word 文档的 `10% → 0 words`；无候选时 map 返回目标 `0`，parser 却拒绝；`inf`/`1e309` 抛出未捕获的 `OverflowError`。建议：显式区分 percentage、fraction、word count；验证有限值和范围；最低删减用 `ceil`；允许零词目标完成 dry sweep。`tools/length_gate.py:208`、`tools/length_gate.py:221`、`tools/length_gate.py:277`、`tools/condense_map.py:348` -5. **major · [tools/length_gate.py:45](D:/Projects/sci-paper/tools/length_gate.py:45) · gate 的计数和输入范围不符合“rendered prose”。** - 触发：`M + "$a$ $b$"` 被计为 **2 words**；删除两条公式便产生两词“删减”。此外，`main.tex` 只有 `\input{body}` 时，修改 child 不改变 gate 的输入；map 却分析组装后的全文。建议：对两版使用相同的 document assembly 和 prose-only 计数，移除 math/float placeholders；快照必须覆盖整个 include graph 或保存组装后的基线。`tools/extract_sections.py:195`、`tools/length_gate.py:282`、`tools/condense_map.py:380` +5. **major · [tools/length_gate.py:45](../../tools/length_gate.py) · gate 的计数和输入范围不符合“rendered prose”。** + 触发：`M + "$a$ $b$"` 被计为 **2 words**；删除两条公式便产生两词“删减”。此外，`main.tex` 只有 `\input{body}` 时，修改 child 不改变 gate 的输入；map 却分析组装后的全文。建议：对两版使用相同的 document assembly 和 prose-only 计数，移除 math/float placeholders；快照必须覆盖整个 include graph 或保存组装后的基线。`tools/extract_sections.py:195`、`tools/length_gate.py:282`、`tools/condense_map.py:380` residue 的 `--git-ref` 同样只读旧版 root，而 after 已组装，diff 比较范围不同。`tools/deai_residue.py:394`、`tools/deai_residue.py:399` -6. **major · [tools/deai_reference.py:210](D:/Projects/sci-paper/tools/deai_reference.py:210) · float 修复仍被“先分段、后投影”绕过。** +6. **major · [tools/deai_reference.py:210](../../tools/deai_reference.py) · float 修复仍被“先分段、后投影”绕过。** 触发：`M + "\\begin{figure}\n\n\\caption{alpha beta gamma delta epsilon.}\n\n\\end{figure}"`。caption 成为独立 prose unit；令这五词 common、四个 pair 未见且 reference 有 spread，即产生 strong collocation finding。语料侧先投影整个 section，会删除该 float。建议：在 section/paragraph/sentence 切分前识别并处理完整环境，保留 source offsets。`tools/deai_collocation.py:260`、`tools/extract_sections.py:197`、`tools/extract_style.py:393` -7. **major · [tools/deai_residue.py:310](D:/Projects/sci-paper/tools/deai_residue.py:310) · 可见编辑标记被 vocabulary projection 吞掉。** +7. **major · [tools/deai_residue.py:310](../../tools/deai_residue.py) · 可见编辑标记被 vocabulary projection 吞掉。** 触发：`\section{Results TODO}\nClean prose.`，或 `\begin{figure}\n\caption{TODO}\n\end{figure}` 放在 Results 下，均没有 residue finding；正文独立 `TODO` 则 strong。建议：residue 的 edit-meta 扫描保留可见 headings、captions、table text；不要复用排除这些内容的词汇投影。另将跨行 `in the revised\nversion` 按规范化空白扫描，当前逐行 regex 漏报。`tools/deai_register.py:344`、`tools/deai_residue.py:204` -8. **major · [tools/deai_residue.py:288](D:/Projects/sci-paper/tools/deai_residue.py:288) · diff rule 检查的是任一消失的词干，不是完整对象。** +8. **major · [tools/deai_residue.py:288](../../tools/deai_residue.py) · diff rule 检查的是任一消失的词干，不是完整对象。** 触发：before 为 `M + "The correction is applied."`；after 为 `M + "The measurement is applied.\n\\caption{Without the saddle correction}"`。得到 strong，虽然 `saddle correction` 从未在 before 出现。另把旧 caption 的 `Blue points` 改成 `Red points`，也会将原有 negation 当成新增。建议：比较 canonical negated object；确认完整对象 before 存在、after 缺失；对象新旧不能由整段 caption 字符串决定。`tools/deai_residue.py:280`、`tools/deai_residue.py:285` -9. **major · [tools/deai_reference.py:162](D:/Projects/sci-paper/tools/deai_reference.py:162) · 无 section 的文档静默漏检。** +9. **major · [tools/deai_reference.py:162](../../tools/deai_reference.py) · 无 section 的文档静默漏检。** 触发：纯文本 `We no longer use this method.`，或前加 `\section {Methods}` / `\section[Short]{Methods}`，self-history 均无 finding；condense 的句子扫描同样为空。原因是 section regex 不接受这些合法形式，随后 `(document)` 被丢弃。建议：共享 section parser；无 heading 时保留 document unit，不能套用 bucket baseline 的轴明确标为 unavailable。`tools/deai_metrics.py:25`、`tools/deai_metrics.py:120`、`tools/condense_map.py:119` -10. **major · [tools/deai_collocation.py:145](D:/Projects/sci-paper/tools/deai_collocation.py:145) · `p_absent_by_chance` 计算了错误事件。** +10. **major · [tools/deai_collocation.py:145](../../tools/deai_collocation.py) · `p_absent_by_chance` 计算了错误事件。** 触发：100 passages 全为 `alpha and omega`，两词 df 均为 100，adjacent pair 从未出现。代码给 `alpha omega` 的 λ=100、absence≈`3.7e-44`，实际上估计的是 passage 内共同出现，未建模有序相邻。建议：采用保留句长和 token boundaries 的 adjacency null；否则将该字段明确降为 co-presence heuristic，不能称为 pair absence probability。`tools/deai_collocation.py:112`、`tools/deai_collocation.py:203` -11. **minor · [tools/deai_register.py:419](D:/Projects/sci-paper/tools/deai_register.py:419) · 定义句给整句所有词自动豁免。** +11. **minor · [tools/deai_register.py:419](../../tools/deai_register.py) · 定义句给整句所有词自动豁免。** 触发：`M + "We define flux using quuxification."` 把 `quuxification` 标为 `defined-here`，即使定义对象只有 `flux`。建议：提取被定义的 noun phrase；无法确定对象时保留 strong 并请求 disposition。`tools/deai_register.py:447` -12. **minor · [tools/deai_reference.py:176](D:/Projects/sci-paper/tools/deai_reference.py:176) · 行号并未保持。** - 触发：`M + "\\paragraph{A\nB}\n\nWe no longer use this method."` 实际第 5 行被报为第 4 行；替换 heading 时换行也变成了空格。另 `M + "\n\\newcommand{\\q}{word}\nWe no longer use this method."` 中 register 将第 4 行报成第 2 行；`\s*` 吃掉了前面的空行。建议：任何 blanking 保留换行，并统一保存字符到原文件行号的映射。`tools/deai_register.py:355` +12. **minor · [tools/deai_reference.py:176](../../tools/deai_reference.py) · 行号并未保持。** + 触发：`M + "\\paragraph{A\nB}\n\nWe no longer use this method."` 实际第 5 行被报为第 4 行；替换 heading 时换行也变成了空格。另 `M + "\n\\newcommand{\\q}{word}\nWe no longer use this method."` 中 register 将第 4 行报成第 2 行；`\s*` 吃掉了前面的空行。建议：任何 blanking 保留换行，并统一保存字符到原文件行号的映射。`tools/deai_register.py:355` 其他定位问题：abstract 使用环境起始行；condense acronym 固定报第 1 行；多句段落的每句共用段落范围。`tools/deai_reference.py:152`、`tools/condense_map.py:241`、`tools/condense_map.py:123` -13. **minor · [tools/extract_sections.py:375](D:/Projects/sci-paper/tools/extract_sections.py:375) · include assembly 有损，新工具继承该问题。** +13. **minor · [tools/extract_sections.py:375](../../tools/extract_sections.py) · include assembly 有损，新工具继承该问题。** 触发：root 为 `Before \input{body} After\n`，child 为 `Child`，组装结果只有 `Child`；连续两次 `\input{body}` 只保留一次。建议：按匹配区间替换，保留前后文字；用 recursion stack 检测环，不能把全部已访问文件当成环；同时返回 child 的 source map。`tools/extract_sections.py:366`、`tools/extract_sections.py:382` -14. **minor · [tools/deai_collocation.py:104](D:/Projects/sci-paper/tools/deai_collocation.py:104) · 非相邻 token 被连成 pair。** +14. **minor · [tools/deai_collocation.py:104](../../tools/deai_collocation.py) · 非相邻 token 被连成 pair。** 触发：`content_pairs("alpha/beta gamma delta epsilon")` 和 `content_pairs("alpha 500 beta gamma delta epsilon")` 均包含 `("alpha","beta")`；`.` 无空白时也如此。建议：先产生带类型和跨度的 tokens；只连接相邻且中间没有 punctuation、number、placeholder 的 word tokens。`tools/deai_collocation.py:111` -15. **minor · [tools/eval_findings.py:295](D:/Projects/sci-paper/tools/eval_findings.py:295) · AUC 的样本量门槛在删除 NaN 前执行。** +15. **minor · [tools/eval_findings.py:295](../../tools/eval_findings.py) · AUC 的样本量门槛在删除 NaN 前执行。** 触发：两组各 20 rows，其中各 19 个 `collocation_novel_fraction=NaN`，余下一组 0、一组 1，仍输出 AUC=1.0。空文档在存在 bank 时也被标为 collocation `measured`。建议：按轴统计有效 documents/eligible sentences，过滤后再应用 floor；零可测内容不能报告 measured。`tools/eval_findings.py:171`、`tools/deai_collocation.py:179`、`tools/deai_collocation.py:192` -16. **minor · [tools/label_findings.py:262](D:/Projects/sci-paper/tools/label_findings.py:262) · flagged passage 会进入 unflagged controls。** +16. **minor · [tools/label_findings.py:262](../../tools/label_findings.py) · flagged passage 会进入 unflagged controls。** 触发：正常 finding 被放入 `evidence`，其中没有顶层 `text`；排除集合于是得到 `""`，同一段不会被排除。建议：用 `source + span/unit_id` 排除全部已命中 unit，而不是读取不存在的字段；控制样本不能仅排除已抽中的部分 findings。`tools/label_findings.py:229`、`tools/label_findings.py:240`、`tools/label_findings.py:268` ## 3. 投影不对称候选清单 @@ -154,35 +154,35 @@ Read-only review of `git diff 9f0b920..edd480c`; session `01a06ebe-b4c5-7802-bfe ## 4. 统计方法评估 -- **λ 的 N：单位一致，但事件不一致。** - df 是 passage presence，所以 `df_a·df_b/N` 的 N 应是 passage 数，不能直接换成 sentence 数或 paper 数。不过它估计的是独立词的 passage co-presence，未估计有序 adjacency；自配对 `a=a` 更不满足两个独立事件。即便只研究 co-presence，独立 passages 下的零出现概率也是 `(1-p_a p_b)^N`，`e^-λ` 还需 rare-event approximation。当前 bank 记录的是 distinct words/pairs per passage。`tools/deai_collocation.py:145`、`tools/deai_collocation.py:294` +- **λ 的 N：单位一致，但事件不一致。** + df 是 passage presence，所以 `df_a·df_b/N` 的 N 应是 passage 数，不能直接换成 sentence 数或 paper 数。不过它估计的是独立词的 passage co-presence，未估计有序 adjacency；自配对 `a=a` 更不满足两个独立事件。即便只研究 co-presence，独立 passages 下的零出现概率也是 `(1-p_a p_b)^N`，`e^-λ` 还需 rare-event approximation。当前 bank 记录的是 distinct words/pairs per passage。`tools/deai_collocation.py:145`、`tools/deai_collocation.py:294` 此概率目前影响 pair evidence 的排序，**不决定 finding gate**；gate 使用 sentence novel fraction percentile。`tools/deai_collocation.py:204`、`tools/deai_collocation.py:270` -- **LOO：pair count 的 passage-level 排除成立，完整 held-out calibration 不成立。** +- **LOO：pair count 的 passage-level 排除成立，完整 held-out calibration 不成立。** `own_passage=True` 将 pair_df=1 视为未见，正确排除了该 passage 对 pair presence 的一次贡献。但 unigram df、common-word membership、N 未重算；同一 paper 的其他 passages 仍在 bank，source 信息甚至在 `_passages` 被丢弃。应称为“固定词表下的 passage LOO”，不能等同于 document-held-out reference。`tools/deai_collocation.py:136`、`tools/deai_collocation.py:139`、`tools/deai_collocation.py:281`、`tools/deai_collocation.py:320` -- **`MIN_JUDGED=4` 与 distinct-pair：有明确 selection 和离散性影响。** +- **`MIN_JUDGED=4` 与 distinct-pair：有明确 selection 和离散性影响。** 四个 judged types 只能得到 0、¼、½、¾、1；重复同一 pair 不增加有效样本量，长的重复句也可能 abstain。另一方面，document score 又计算所有 pair occurrences，包含不足四对的句子。因此 gate 和 AUC 使用不同统计量；这是代码明确实现的选择，应同时报告 eligible sentence coverage。`tools/deai_collocation.py:135`、`tools/deai_collocation.py:151`、`tools/deai_collocation.py:161` -- **p90/p95 不是固定 10%/5% operating rate。** - 代码取 `P(X≤x)` 的 plateau 顶端，再与 .90/.95 比较。构造 `[0]*80 + [.5]*19 + [1]` 时，`.5` 被读为 p98，**20%** 样本都 strong；spread guard 仍通过。`tools/deai_reference.py:90`、`tools/deai_reference.py:124`、`tools/deai_collocation.py:214` +- **p90/p95 不是固定 10%/5% operating rate。** + 代码取 `P(X≤x)` 的 plateau 顶端，再与 .90/.95 比较。构造 `[0]*80 + [.5]*19 + [1]` 时，`.5` 被读为 p98，**20%** 样本都 strong；spread guard 仍通过。`tools/deai_reference.py:90`、`tools/deai_reference.py:124`、`tools/deai_collocation.py:214` 对现有 bank 只读重算 LOO：method advisory **13.70%**、data **14.68%**、discussion **12.30%**；对应 strong **5.44% / 4.42% / 4.54%**。应保存 empirical tail counts，选择 strict quantile exceedance 或明确 tie policy。计算路径：`tools/deai_collocation.py:318`、`tools/deai_reference.py:77` -- **AUC 与 rates：主表可复现，分母名称有误。** - 复现 held-out zero **2.212/1k、100% documents、AUC .246**；collocation **2.031/1k、99.0%、AUC .691**。但 `/1k` 是 `1000·Σfindings/Σraw_words`，不是 per-document median，也不是 body-token rate；zero 的逐文档 rate 中位数实算为 **1.611/1k**。`tools/eval_findings.py:133`、`tools/eval_findings.py:149` +- **AUC 与 rates：主表可复现，分母名称有误。** + 复现 held-out zero **2.212/1k、100% documents、AUC .246**；collocation **2.031/1k、99.0%、AUC .691**。但 `/1k` 是 `1000·Σfindings/Σraw_words`，不是 per-document median，也不是 body-token rate；zero 的逐文档 rate 中位数实算为 **1.611/1k**。`tools/eval_findings.py:133`、`tools/eval_findings.py:149` register AUC 按逐文档 finding density 排序；collocation AUC 按 document **token-pair fraction** 排序。这两种口径代码有明确区分，不能把 .691 解释成 sentence gate 的 AUC。`tools/eval_findings.py:85`、`tools/eval_findings.py:166` -- **.855 与 .691 的差别不是同一实验算错。** +- **.855 与 .691 的差别不是同一实验算错。** §23 明确将 .855 标为 first-1,500-word prototype，当前值是全篇 document fraction。比较还应控制长度、年代和 generation regime，并按 paper/source 做 bootstrap；现有输出只有点估计。`docs/architecture/evaluation/vocabulary-and-residue.md:80`、`tools/eval_findings.py:295` -- **document fraction 不能代替 sentence gate transfer。** +- **document fraction 不能代替 sentence gate transfer。** 一篇论文有很多 eligible sentences，至少一次 finding 的概率自然累积；99% document flag rate 不能直接解释为 sentence false-positive rate。当前 evaluator 没有保存 collocation eligible sentence denominator。`tools/deai_collocation.py:254`、`tools/eval_findings.py:128` -- **residue 的 203 篇是调参集。** - strengths 在这批 papers 上反复修改，不能再把同批 rate 当独立验证；它也没有 edit-history ground truth 来估计 recall。文档承认后一点。`docs/architecture/evaluation/vocabulary-and-residue.md:174`、`docs/architecture/evaluation/vocabulary-and-residue.md:225` +- **residue 的 203 篇是调参集。** + strengths 在这批 papers 上反复修改，不能再把同批 rate 当独立验证；它也没有 edit-history ground truth 来估计 recall。文档承认后一点。`docs/architecture/evaluation/vocabulary-and-residue.md:174`、`docs/architecture/evaluation/vocabulary-and-residue.md:225` §23.4 的 `24 in 20 papers` 对应 **9.85% papers**，不是随后写的 12%。按当前产品入口去注释后，实算 self-history strong **24/20 papers**、edit-meta strong **16/12 papers**，合计 **31/203 papers**；旧调参表应标明版本，不能作为 v0.36.1 当前 rate。`docs/architecture/evaluation/vocabulary-and-residue.md:181`、`docs/architecture/evaluation/vocabulary-and-residue.md:190`、`tools/ai_ism_lint.py:359`、`tools/deai_residue.py:304` -- **labeller 当前不能给出有效 population recall。** +- **labeller 当前不能给出有效 population recall。** 除 control 污染外，`caught` 累加各轴 finding，`missed` 计算 control paragraphs；同一 passage 多轴命中会重复计 TP，抽样 quotas 也没有 inclusion-probability correction。须统一到去重的 passage/unit，并采用概率抽样或完整子集标注。`tools/label_findings.py:217`、`tools/label_findings.py:341`、`tools/label_findings.py:352` ## 5. 缺失测试 @@ -227,10 +227,10 @@ mirror check 确实比较了两处 **22 个词的集合**；它不验证 strong/ ## 7. 值得做的显著改进 -1. **建立保留 source map 的统一 TeX 分解层。** +1. **建立保留 source map 的统一 TeX 分解层。** 输出 prose、heading、caption、math、code、bibliography 等 typed spans；各轴明确选择需要的 spans，再切 sentence/paragraph。这样能同时解决 whole-span 投影、重复 cleaning、bucket inheritance 和行号问题；residue 仍可保留 visible caption 的编辑标记。当前分裂点：`tools/deai_register.py:344`、`tools/deai_reference.py:210`、`tools/extract_sections.py:193` -2. **把评估结果固化为可复算 artifact。** +2. **把评估结果固化为可复算 artifact。** 保存 commit、corpus/profile/projection hashes、raw/body word counts、eligible sentence counts、tie policy、source grouping 与有效 AUC 样本量；文档从 artifact 提取。当前 §23 新表没有进入 published-figure 检查清单，现有 AUC 输出也没有这些有效样本元数据。`tests/test_published_figures.py:315`、`tests/test_published_figures.py:335`、`tools/eval_findings.py:297` 3. **执行 line budget，并按职责拆分临界文件。** @@ -242,4 +242,4 @@ mirror check 确实比较了两处 **22 个词的集合**；它不验证 strong/ | `README.md:750` | 750 / 0 | 将详细 benchmark/evaluation 表移到 evidence docs，README 保留入口和当前状态。`README.md:715` | | `docs/SCIPAPER_STANDARD.md:749` | 749 / 1 | 保留规范条款；将实证缘由、历史比较和长解释移入 EVALUATION，以链接引用，维持单一规范权威。`docs/SCIPAPER_STANDARD.md:269`、`docs/SCIPAPER_STANDARD.md:632` | - 为受版本控制的文本增加只读 line-count check；当前 validator 的十项 checks 没有该硬限制。`tools/validate_plugin.py:712` \ No newline at end of file + 为受版本控制的文本增加只读 line-count check；当前 validator 的十项 checks 没有该硬限制。`tools/validate_plugin.py:712` diff --git a/tests/test_ai_ism_lint_cli.py b/tests/test_ai_ism_lint_cli.py index 9755d36..fc2aec6 100644 --- a/tests/test_ai_ism_lint_cli.py +++ b/tests/test_ai_ism_lint_cli.py @@ -232,6 +232,115 @@ def test_malformed_profile_asset_is_execution_failure_not_l0(self): self.assertIn("execution failed", result.stderr) +def run_lint_file(name: str, text: str, *arguments: str, profile_root: Path | None = None): + """Lint `text` written to a file called `name` (the suffix picks the reader).""" + with tempfile.TemporaryDirectory() as temporary: + draft = Path(temporary) / name + draft.write_text(text, encoding="utf-8") + command = [sys.executable, str(LINTER), str(draft), "--no-distribution", + "--no-structure", "--no-document-structure"] + if profile_root is not None: + command.extend(["--profile-root", str(profile_root)]) + command.extend(arguments) + return subprocess.run(command, text=True, capture_output=True, encoding="utf-8") + + +class LexicalRuleTests(unittest.TestCase): + """The 2026-09-27 audit's linter findings, each pinned.""" + + def test_a_paragraph_initial_connector_is_judged_per_paragraph_not_per_line(self): + # One-sentence-per-line LaTeX: a sentence-initial `Notably,` inside a + # paragraph is Tier B within its cap, whatever line it starts. + inside = run_lint_file("d.tex", "\\section{Results}\nWe ran the swap test.\n" + "Notably, the covariance is stable.\n") + self.assertEqual(inside.returncode, 0, inside.stdout + inside.stderr) + joined = run_lint_file("d.tex", "\\section{Results}\nWe ran the swap test. " + "Notably, the covariance is stable.\n") + self.assertEqual(joined.returncode, 0, joined.stdout + joined.stderr) + # The paragraph's first prose line, after a heading, is paragraph-initial. + first = run_lint_file("d.tex", "\\section{Results}\nNotably, the covariance " + "is stable.\n\nMore prose.\n") + self.assertEqual(first.returncode, 1, first.stdout + first.stderr) + self.assertIn("tier-a:paragraph-start:notably", first.stdout) + + def test_an_opener_is_judged_where_a_sentence_starts(self): + mid_line = run_lint_file("d.tex", "\\section{Results}\nWe ran the test. It is " + "worth noting that this holds.\n") + self.assertEqual(mid_line.returncode, 1, mid_line.stdout + mid_line.stderr) + self.assertIn("tier-a:opener", mid_line.stdout) + wrapped = run_lint_file("d.tex", "\\section{Results}\nThe field moved on, and\n" + "in recent years the estimate settled.\n") + self.assertEqual(wrapped.returncode, 0, wrapped.stdout + wrapped.stderr) + + def test_paved_and_showcased_are_tier_a(self): + result = run_lint_file("d.tex", "\\section{Results}\nThis paved the way for the " + "fit and showcased the model.\n") + self.assertEqual(result.returncode, 1, result.stdout + result.stderr) + self.assertIn("tier-a:paved", result.stdout) + self.assertIn("tier-a:showcased", result.stdout) + + def test_every_em_dash_on_a_line_is_a_target_with_its_own_id(self): + result = run_lint_file("d.tex", "\\section{Results}\nThe catalog---measured " + "here---is used, so we delve and delve.\n", + "--format", "json") + self.assertEqual(result.returncode, 1, result.stderr) + findings = json.loads(result.stdout)["findings"] + rules = [finding["rule"] for finding in findings] + self.assertEqual(rules.count("em-dash"), 2) + ids = [finding["finding_id"] for finding in findings] + self.assertEqual(len(ids), len(set(ids))) + + def test_command_arguments_are_not_prose(self): + result = run_lint_file("d.tex", "\\section{Results}\nSee \\ref{sec:realm}, " + "\\cite{delve-2020} and \\url{http://x/delve}.\n" + "\\label{sec:realm}\n") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("L0=0", result.stdout) + + def test_three_parallel_needs_whole_words(self): + result = run_lint_file("d.tex", "\\section{Results}\nIt is not only fast but " + "also stable in Bland.\n") + self.assertNotIn("three-parallel", result.stdout) + + def test_an_unreadable_tex_root_is_an_execution_failure(self): + with tempfile.TemporaryDirectory() as temporary: + directory = Path(temporary) / "dir.tex" + directory.mkdir() + result = subprocess.run( + [sys.executable, str(LINTER), str(directory), "--no-distribution", + "--no-structure", "--no-document-structure"], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 2, result.stdout + result.stderr) + + def test_several_fields_without_a_choice_say_so(self): + with tempfile.TemporaryDirectory() as temporary: + profile_root = Path(temporary) + (profile_root / "fieldA").mkdir() + (profile_root / "fieldB").mkdir() + result = run_lint_file("d.tex", "\\section{Results}\nPlain prose.\n", + profile_root=profile_root) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("several field profiles", result.stderr) + + def test_an_unparseable_lexicon_is_an_execution_failure(self): + with tempfile.TemporaryDirectory() as temporary: + profile_root = Path(temporary) + (profile_root / "f").mkdir() + (profile_root / "f" / "lexicon.json").write_text("{not json", encoding="utf-8") + result = run_lint_file("d.tex", "\\section{Results}\nPlain prose.\n", + "--field", "f", profile_root=profile_root) + self.assertEqual(result.returncode, 2, result.stdout + result.stderr) + + def test_markdown_front_matter_and_rules_are_not_em_dashes(self): + result = run_lint_file("d.md", "---\ntitle: x\n---\n\nSome prose here.\n\n---\n" + "About 50% of the sample: the rest, revealing " + "the bias.\n") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("L0=0", result.stdout) + self.assertIn("colon-elaboration", result.stdout) + self.assertIn("ing-tail:revealing", result.stdout) + + class DocumentAssemblyTests(unittest.TestCase): """The lint path must read the same document the corpus path reads. diff --git a/tests/test_condense_map.py b/tests/test_condense_map.py index e28edb4..c87b15e 100644 --- a/tests/test_condense_map.py +++ b/tests/test_condense_map.py @@ -147,9 +147,12 @@ def test_a_negated_or_renumbered_sentence_is_not_a_restatement(self): "independent radial apertures is significant.") for variant in (claim.replace("is significant", "is not significant"), claim.replace("independent", "five independent")): - findings = cm.condense_map(f"\\section{{Methods}}\n{claim}\n\n{variant}\n") - self.assertEqual([f for f in findings if f["rule"] == "condense-restatement"], - [], variant) + # In either order: a copy that DROPS the negation was still a + # restatement, and deleting it left the negated home standing. + for first, second in ((claim, variant), (variant, claim)): + findings = cm.condense_map(f"\\section{{Methods}}\n{first}\n\n{second}\n") + self.assertEqual([f for f in findings if f["rule"] == "condense-restatement"], + [], (first, second)) # The same sentence restated verbatim still is one. findings = cm.condense_map(f"\\section{{Methods}}\n{claim}\n\n{claim}\n") self.assertEqual(len([f for f in findings if f["rule"] == "condense-restatement"]), 1) @@ -178,6 +181,25 @@ def test_a_document_without_sections_is_still_mapped(self): class CarveOutTests(unittest.TestCase): + def test_an_abstract_before_the_body_is_the_carved_out_side(self): + # With only the copy's bucket consulted, every abstract claim made the + # BODY sentence the removable restatement and named the abstract as + # its canonical home -- the inversion of the skill's rule. + text = f"\\begin{{abstract}}\n{CLAIM}\n\\end{{abstract}}\n\\section{{Results}}\n{RESTATED}\n" + findings = cm.condense_map(text) + (finding,) = [f for f in findings if f["rule"] == "condense-restatement"] + self.assertTrue(finding["observed"]["genre_carve_out"]) + self.assertEqual(cm.condense_budget(text, findings)["default_target_words"], 0) + # A body sentence restating another BODY sentence stays a target even + # when the abstract said it first. + text = (f"\\begin{{abstract}}\n{CLAIM}\n\\end{{abstract}}\n" + f"\\section{{Methods}}\n{CLAIM}\n\\section{{Results}}\n{RESTATED}\n") + findings = cm.condense_map(text) + targeted = [f for f in findings if f["rule"] == "condense-restatement" + and not f["observed"]["genre_carve_out"]] + self.assertEqual([f["location"]["section"] for f in targeted], ["results"]) + self.assertGreater(cm.condense_budget(text, findings)["default_target_words"], 0) + def test_a_conclusion_restatement_is_reported_but_not_targeted(self): text = f"\\section{{Results}}\n{CLAIM}\n\\section{{Conclusion}}\n{RESTATED}\n" findings = cm.condense_map(text) @@ -193,6 +215,47 @@ def test_a_heading_does_not_fuse_with_the_first_sentence(self): self.assertEqual(cues, ["in this section"]) +class GateAgreementTests(unittest.TestCase): + """The map's words are the gate's words, or its target cannot be met.""" + + SENTENCE = ("The calibrated aperture mass filter recovers the injected cluster " + "signal at $\\nu = 3$ for every configuration \\citep{Ref} in the " + "grid of $N = 20$ fields.") + + def test_removable_words_exclude_projection_placeholders(self): + import length_gate as lg + text = f"\\section{{Results}}\n{self.SENTENCE}\n\n{self.SENTENCE}\n" + (finding,) = [f for f in cm.condense_map(text) if f["rule"] == "condense-restatement"] + self.assertEqual(finding["observed"]["removable_words"], + lg.prose_word_count(self.SENTENCE)) + + def test_the_budget_denominator_excludes_heading_words(self): + import length_gate as lg + text = "\\section{A Long Heading Here}\nOne two three.\n\\section{Another}\nFour five.\n" + budget = cm.condense_budget(text, cm.condense_map(text)) + self.assertEqual(budget["prose_words"], sum(lg.section_word_counts(text).values())) + + def test_a_parenthesised_word_is_not_an_acronym(self): + text = ("\\section{Methods}\nWe adopt a normal (Gaussian) prior and the Dark " + "Energy Survey (DES) catalog.\n") + acronyms = [f["observed"]["excerpt"] for f in cm.condense_map(text) + if f["rule"] == "condense-dead:acronym"] + self.assertEqual(acronyms, ["DES"]) + + def test_any_ref_command_keeps_a_float_alive_and_starred_captions_count(self): + text = ("\\section{R}\nSee \\subref{fig:a}.\n" + "\\begin{figure}\\caption*{\\textbf{\\emph{Two}} words here now}\\label{fig:a}\\end{figure}\n" + "\\begin{figure}\\caption*{\\textbf{\\emph{Two}} words here now}\\label{fig:b}\\end{figure}\n") + dead = [(f["observed"]["labels"], f["observed"]["removable_words"]) + for f in cm.condense_map(text) if f["rule"] == "condense-dead:figure"] + self.assertEqual(dead, [(["fig:b"], 4)]) + + def test_a_case_folded_verbose_match_cannot_crash_the_scan(self): + # `İn order to` matches under re.I but lower-cases to a key the table + # does not hold; the lookup was a KeyError and the tool exited 1. + cm.condense_map("\\section{Methods}\n\u0130n order to test, we ran it.\n") + + class CliTests(unittest.TestCase): def test_json_report_carries_the_budget(self): with tempfile.TemporaryDirectory() as raw: diff --git a/tests/test_extract_style.py b/tests/test_extract_style.py index 0155942..ffcfd91 100644 --- a/tests/test_extract_style.py +++ b/tests/test_extract_style.py @@ -693,3 +693,7 @@ def test_both_projections_treat_citations_alike(self): source = r"we use \citep[e.g.][]{Tacconi2018} and \citealp{Leroy2013}" self.assertNotIn("Tacconi", es.latex_to_numeral_text(source)) self.assertNotIn("Leroy", es.latex_to_numeral_text(source)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_extractor_statistics.py b/tests/test_extractor_statistics.py new file mode 100644 index 0000000..0b81988 --- /dev/null +++ b/tests/test_extractor_statistics.py @@ -0,0 +1,111 @@ +"""Extractor statistics and projection forms pinned by the 2026-09-27 audit. + +Split from test_extract_style.py, which sits at the 750-line budget. +""" + +from __future__ import annotations + +import json +import pathlib +import tempfile +import unittest + +from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path + +import extract_style as es + + +class DisplayMathFormsTests(unittest.TestCase): + """Every way LaTeX writes display mathematics is display mathematics.""" + + def test_bracket_display_math_is_not_prose(self): + self.assertEqual(es.prose_words(r"We have \[ E = m c^2 \] here."), + ["We", "have", "here."]) + + def test_dollar_dollar_display_math_does_not_swallow_prose(self): + # Matched from its second dollar by the INLINE pattern, `$$x=1$$` paired + # with the next inline `$` and the prose between vanished from the + # plain view alone, so the two projections no longer agreed. + plain = es.latex_to_plain("First $$x=1$$ prose words survive here $y$ end.") + self.assertIn("prose words survive here", plain) + numeral = es.latex_to_numeral_text("First $$x=1$$ prose words survive here $y$ end.") + self.assertIn("prose words survive here", numeral) + + def test_subequations_are_a_display_environment(self): + plain = es.latex_to_plain( + "A \\begin{subequations}\\begin{align}x=1\\end{align}\\end{subequations} B") + self.assertEqual(plain.split(), ["A", "[MATH]", "B"]) + + +class ExtractorStatisticsTests(unittest.TestCase): + """The 2026-09-27 audit's extractor findings, each pinned.""" + + def test_multi_word_openers_can_be_observed(self): + # `paragraph_initial_words` records one word per paragraph, so the five + # phrases were always reported absent from every corpus. + text = "In recent years the field grew.\n\nRecent advances helped.\n" + self.assertEqual(es.paragraph_initial_phrases(text), + ["In recent years", "Recent advances"]) + paper = {"by_section": {"intro": { + "paragraph_initial_words": es.paragraph_initial_words(text), + "paragraph_initial_phrases": es.paragraph_initial_phrases(text)}}} + transitions = es.aggregate_transitions([(1.0, paper)]) + self.assertIn(("In recent years", 1), transitions["blacklist_present_in_corpus"]) + self.assertNotIn("In recent years", transitions["blacklist_absent_from_corpus"]) + self.assertEqual(transitions["n_paragraphs"], 2) + + def test_an_empty_corpus_reports_zero_paragraphs(self): + self.assertEqual(es.aggregate_transitions([])["n_paragraphs"], 0) + + def test_placeholders_are_not_words_and_do_not_drop_a_paragraph(self): + self.assertEqual(es.words(es.without_placeholders("[CITE] [math] x")), ["x"]) + # A citation-led paragraph is prose; a placeholder-only one is not. + text = "[CITE] showed the effect.\n\n[MATH]\n\n[FIGURE-OR-TABLE] and more." + self.assertEqual(es.paragraph_initial_words(text), ["showed", "and"]) + + def test_words_are_unicode_letters(self): + self.assertEqual(es.words("naïve Poincaré"), ["naïve", "Poincaré"]) + + def test_bank_rows_count_prose_words_and_carry_unique_ids(self): + prose = " ".join(f"word{i}" for i in range(EXEMPLAR_MIN)) + plain = f"[CITE] {prose}\n\n{prose}" + paper = {"tier": "tier-1-top", "source_path": "p.tex", + "by_section": {"intro": {"plain_text": plain}, + "results": {"plain_text": plain}}} + with tempfile.TemporaryDirectory() as tmp: + n = es.write_exemplar_bank([(1.0, paper)], pathlib.Path(tmp)) + rows = [json.loads(line) for line in + (pathlib.Path(tmp) / "exemplar_paragraphs.jsonl").read_text().splitlines()] + self.assertEqual(n, 4) + self.assertEqual(len({row["id"] for row in rows}), 4) + self.assertTrue(all(row["n_words"] == EXEMPLAR_MIN for row in rows)) + + def test_the_profile_directory_is_not_created_for_an_empty_corpus(self): + with tempfile.TemporaryDirectory() as tmp: + corpus = pathlib.Path(tmp) / "corpus" + for tier in ("tier-1-top", "tier-2-mentor", "tier-3-reference"): + (corpus / "g" / tier).mkdir(parents=True) + profile = pathlib.Path(tmp) / "profile" + code = es.main(["--field", "g", "--corpus-root", str(corpus), + "--profile-root", str(profile)]) + self.assertEqual(code, 1) + self.assertFalse((profile / "g").exists()) + + +EXEMPLAR_MIN = 30 + + +class FallbackRetrievalTests(unittest.TestCase): + def test_an_empty_topic_ranks_by_section_name_in_the_fallback(self): + import retrieve_exemplars as rx + records = [{"section": "results", "tier": "tier-1-top", "text": "unrelated prose"}, + {"section": "results", "tier": "tier-1-top", + "text": "the results section states the results"}] + ranked = rx._retrieve_fallback(records, "results", "", 2, {"tier-1-top"}) + self.assertGreater(ranked[0][0], 0.0) + self.assertIn("results section", ranked[0][1]["text"]) + + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_length_gate.py b/tests/test_length_gate.py index 1647065..e0a82fc 100644 --- a/tests/test_length_gate.py +++ b/tests/test_length_gate.py @@ -7,6 +7,8 @@ import unittest from pathlib import Path +from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path + ROOT = Path(__file__).resolve().parents[1] GATE = ROOT / "tools" / "length_gate.py" @@ -149,7 +151,7 @@ def test_optional_argument_heading_rename_stays_neutral(self): result.stdout + result.stderr) def test_require_shrink_fails_a_pass_that_barely_cut(self): - # Nine prose words before; dropping one word is an 11% cut, short of + # Twelve prose words before; dropping one word is an 8% cut, short of # the 30% required, so the gate must refuse to close green. after = BEFORE.replace("The estimator uses five filters.", "The estimator uses filters.") @@ -179,12 +181,90 @@ def test_require_shrink_accepts_a_word_count(self): def test_require_shrink_rejects_nonsense(self): # `100%` and `200%` once parsed as one and two WORDS; `inf` and `1e309` # escaped as an uncaught OverflowError. A percentage is a percentage. - for bad in ("0", "-0.2", "1.5", "lots", "100%", "200%", "0%", "inf", "1e309", + for bad in ("-0.2", "1.5", "lots", "100%", "200%", "inf", "1e309", "nan", "30%%"): result = self.run_gate(BEFORE, BEFORE, "--require-shrink", bad) self.assertEqual(result.returncode, 2, bad + result.stderr) self.assertIn("--require-shrink", result.stderr) + def test_a_zero_target_means_no_cut_required(self): + # The condense skill passes the map's default_target_words through; + # a map with nothing to cut is a target of 0, which used to be exit 2. + for zero in ("0", "0%", "0.0"): + result = self.run_gate(BEFORE, BEFORE, "--require-shrink", zero, + "--format", "json") + self.assertEqual(result.returncode, 0, zero + result.stderr) + self.assertTrue(json.loads(result.stdout)["length_budget"]["shrink_met"]) + + def test_a_percentage_is_not_over_demanded_by_a_float_ceiling(self): + # ceil(0.07 * 100) was 8: the float 0.07 is slightly above 7/100. + import length_gate as lg + self.assertEqual(lg.required_shrink_words(lg.parse_required_shrink("7%"), 100), 7) + self.assertEqual(lg.required_shrink_words(lg.parse_required_shrink("14%"), 50), 7) + self.assertEqual(lg.required_shrink_words(lg.parse_required_shrink("0.07"), 100), 7) + + def test_a_missing_baseline_is_a_configuration_failure_not_growth(self): + # A mistyped --before read as a zero-word baseline: every section + # "grew" and the gate exited 1 with strong findings from no evidence. + with tempfile.TemporaryDirectory() as temporary: + after_path = Path(temporary) / "after.tex" + after_path.write_text(BEFORE, encoding="utf-8") + result = subprocess.run( + [sys.executable, str(GATE), str(after_path), "--before", + str(Path(temporary) / "missing.tex")], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 2, result.stdout + result.stderr) + self.assertNotIn("GROWTH", result.stdout) + directory = Path(temporary) / "dir.tex" + directory.mkdir() + result = subprocess.run( + [sys.executable, str(GATE), str(directory), "--before", str(after_path)], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 2, result.stdout + result.stderr) + + def test_a_commented_out_heading_does_not_open_a_section(self): + import length_gate as lg + counts = lg.section_word_counts( + "\\section{Methods}\nOne two three.\n% \\section{Old draft}\nFour five.\n") + self.assertEqual(counts, {"Methods": 5}) + + def test_display_math_in_every_form_is_not_prose(self): + import length_gate as lg + self.assertEqual(lg.prose_word_count( + "We have \\[ E = m c^2 \\] here and $$ x = 1 $$ too."), 5) + + def test_a_git_baseline_keeps_a_child_the_working_tree_has_deleted(self): + # The most decisive condense move -- deleting a whole \input section + # and its call -- was invisible: the child was resolved against the + # working tree, where it no longer existed, so it left the baseline. + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + + def git(*arguments: str) -> None: + subprocess.run(["git", "-C", str(root), *arguments], check=True, + capture_output=True, text=True) + + git("init", "-q") + git("config", "user.email", "t@example.com") + git("config", "user.name", "t") + (root / "main.tex").write_text( + "\\section{Methods}\nRoot prose here.\n\\input{body}\n", encoding="utf-8") + (root / "body.tex").write_text( + "Alpha beta gamma delta epsilon zeta eta theta iota kappa.\n", + encoding="utf-8") + git("add", ".") + git("commit", "-q", "-m", "baseline") + (root / "body.tex").unlink() + (root / "main.tex").write_text("\\section{Methods}\nRoot prose.\n", + encoding="utf-8") + result = subprocess.run( + [sys.executable, str(GATE), str(root / "main.tex"), "--git-ref", "HEAD", + "--require-shrink", "5", "--format", "json"], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + budget = json.loads(result.stdout)["length_budget"] + self.assertEqual((budget["total_before"], budget["total_after"]), (13, 2)) + def test_a_fraction_of_a_short_document_rounds_up_to_one_word(self): # 10% of the 12-word baseline is 1.2 words: rounding to nearest gave a # requirement of one word, but 10% of a five-word note gave zero and diff --git a/tests/test_tex_assembly.py b/tests/test_tex_assembly.py index 9336c4c..c0e6d37 100644 --- a/tests/test_tex_assembly.py +++ b/tests/test_tex_assembly.py @@ -126,6 +126,51 @@ def test_an_include_is_spliced_in_place_and_may_repeat(self): self.assertEqual(text, "Before CHILD After\nCHILD % \\input{alt}\n") +class IncludeResolutionTests(unittest.TestCase): + def _bundle(self, tmp: str, files: dict[str, str]) -> pathlib.Path: + d = pathlib.Path(tmp) + for name, body in files.items(): + (d / name).parent.mkdir(parents=True, exist_ok=True) + (d / name).write_text(body, encoding="utf-8") + return d + + def test_a_target_with_a_suffix_is_read_as_written(self): + with tempfile.TemporaryDirectory() as tmp: + d = self._bundle(tmp, {"main.tex": r"\input{fig.tikz}" "\n", + "fig.tikz": "TIKZ\n", "fig.tex": "WRONG FILE\n"}) + self.assertEqual(assembly.read_tex_document(d / "main.tex"), "TIKZ\n\n") + + def test_a_nested_include_resolves_against_the_document_root(self): + # LaTeX resolves every \input against the directory it was started + # in; resolved against the child's directory, `sections/table.tex` + # shadowed the root-level file the author meant. + with tempfile.TemporaryDirectory() as tmp: + d = self._bundle(tmp, { + "main.tex": r"\input{sections/intro}" "\n", + "sections/intro.tex": r"INTRO \input{table}" "\n", + "table.tex": "ROOT-LEVEL TABLE", + "sections/table.tex": "SECTIONS-LEVEL TABLE", + }) + self.assertIn("ROOT-LEVEL TABLE", assembly.read_tex_document(d / "main.tex")) + + def test_include_commands_with_longer_names_are_not_includes(self): + self.assertEqual(assembly.include_targets( + "\\includegraphics[width=3cm]{fig1}\n\\includeonly{chap}\n" + "\\inputminted{python}{x.py}\n\\input{body}"), ["body"]) + + def test_an_unreadable_root_raises_instead_of_reading_as_empty(self): + # A directory named `x.tex` and a missing baseline both read as an + # empty document: the linter reported it clean and the gate reported + # every section as growth. Only a CHILD may degrade. + with tempfile.TemporaryDirectory() as tmp: + d = pathlib.Path(tmp) + (d / "dir.tex").mkdir() + with self.assertRaises(OSError): + assembly.read_tex_document(d / "dir.tex") + with self.assertRaises(OSError): + assembly.read_tex_document(d / "missing.tex") + + class GitBaselineTests(unittest.TestCase): """A `--git-ref` baseline is the assembled document AT THE REF, children too.""" @@ -152,6 +197,26 @@ def git(*args: str) -> None: with self.assertRaises(ValueError): assembly.read_git_document(d / "missing.tex", "HEAD") + def test_a_child_deleted_from_the_working_tree_is_still_in_the_baseline(self): + with tempfile.TemporaryDirectory() as tmp: + d = pathlib.Path(tmp) + + def git(*args: str) -> None: + subprocess.run(["git", "-C", str(d), *args], check=True, + capture_output=True, text=True) + + git("init", "-q") + git("config", "user.email", "t@example.com") + git("config", "user.name", "t") + (d / "main.tex").write_text("\\input{body}\n", encoding="utf-8") + (d / "body.tex").write_text("old child prose\n", encoding="utf-8") + git("add", ".") + git("commit", "-q", "-m", "baseline") + (d / "body.tex").unlink() + (d / "main.tex").write_text("nothing left\n", encoding="utf-8") + self.assertEqual(assembly.read_git_document(d / "main.tex", "HEAD"), + "old child prose\n\n") + if __name__ == "__main__": unittest.main() diff --git a/tests/test_tex_macros.py b/tests/test_tex_macros.py index cf12b2e..dea914e 100644 --- a/tests/test_tex_macros.py +++ b/tests/test_tex_macros.py @@ -41,6 +41,17 @@ def test_a_symbolic_macro_is_left_alone(self): def test_a_macro_taking_an_argument_is_left_alone(self): text = r"\newcommand{\hl}[1]{\textbf{#1}}" "\n" r"\hl{42} rows" self.assertEqual(tex_macros.expand_numeric(text), text) + # A numeric body does not make an argument-taking macro a constant: + # `\foo{x}` became `42{x}` because the arity was matched and ignored. + text = r"\newcommand{\foo}[1]{42}" "\n" r"Use \foo{x} here." + self.assertEqual(tex_macros.expand_numeric(text), text) + + def test_a_commented_out_definition_is_not_a_definition(self): + text = (r"% \newcommand{\Nf}{63}" "\n" r"\newcommand{\Nf}{7}" "\n" + r"We use \Nf{} fields.") + self.assertIn("We use 7 fields.", tex_macros.expand_numeric(text)) + alone = r"% \newcommand{\Nf}{63}" "\n" r"We use \Nf{} fields." + self.assertEqual(tex_macros.expand_numeric(alone), alone) def test_a_shorter_name_does_not_fire_inside_a_longer_one(self): text = (r"\newcommand{\Nf}{7}" "\n" r"\newcommand{\Nfields}{63}" "\n" diff --git a/tools/ai_ism_lint.py b/tools/ai_ism_lint.py index 22376b4..7fe89fa 100644 --- a/tools/ai_ism_lint.py +++ b/tools/ai_ism_lint.py @@ -17,6 +17,7 @@ TOOLS_DIR = Path(__file__).resolve().parent sys.path.insert(0, str(TOOLS_DIR)) +import cli_common # noqa: E402 shared CLI preamble and field listing import deai_docstructure # noqa: E402 sibling tool import after path setup import extract_sections as es # noqa: E402 document assembly and projections import deai_feedback as feedback # noqa: E402 shared finding contract @@ -28,21 +29,35 @@ import deai_salience # noqa: E402 salience-hierarchy detector import deai_structure # noqa: E402 sentence-construction detector -REPO_ROOT = TOOLS_DIR.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" EM_DASH_PATTERN = re.compile(r"—|---|\\textemdash") +# A line that is nothing but dashes is a Markdown rule or front-matter fence. +RULE_LINE_PATTERN = re.compile(r"\s*-{3,}\s*") +# The canonical Tier A list (skills/paper/SKILL.md). `paved` and `showcased` +# were in the skill's table and grep but not here, so "paved the way" -- the +# commonest form of the idiom -- passed the linter and failed the review grep. TIER_A_PATTERN = re.compile( r"(?i)\b(?:delves?|delving|delved|leverages|leveraging|leveraged|" - r"paves?|paving|sheds?|shedding|showcases?|showcasing|utiliz(?:ing|es)|" - r"seamless(?:ly)?|holistic(?:ally)?|comprehensively|crucially|" + r"paves?|paved|paving|sheds?|shedding|showcases?|showcased|showcasing|" + r"utiliz(?:ing|es)|seamless(?:ly)?|holistic(?:ally)?|comprehensively|crucially|" r"underscor(?:e|es|ed|ing)|tapestry|testament|pivotal|realms?)\b" ) +# An opener is judged where a SENTENCE starts: at the start of a line that +# begins a sentence (see `_sentence_starts_line`) or after terminal +# punctuation inside the line. Anchored to the physical line it depended on +# how the author wrapped the source: `It is worth noting` after a period in +# the middle of a line was missed, while a hard-wrapped continuation that +# happened to start with `in recent years` was a target. TIER_A_OPENER_PATTERN = re.compile( - r"(?i)^\s*(?:recent advances in|despite significant progress|" + r"(?i)(?:^\s*|(?<=[.!?])\s+)(?:recent advances in|despite significant progress|" r"with the advent of|in recent years|it is worth noting)" ) +# A PARAGRAPH-initial connector, so the pattern runs on a paragraph's first +# prose line only and wants the capital: with `(?i)^\s*` on every physical +# line, one-sentence-per-line LaTeX turned every sentence-initial `Notably,` +# (Tier B, inside its cap) into an L0 target, and joining the same sentences +# onto one line made the target disappear. TIER_A_PARAGRAPH_CONNECTOR_PATTERN = re.compile( - r"(?i)^\s*(Importantly|Interestingly|Notably|Crucially)," + r"^\s*(Importantly|Interestingly|Notably|Crucially)," ) TIER_B_PATTERN = re.compile( r"(?i)\b(?:furthermore|moreover|additionally|robust(?:ly)?|comprehensive|" @@ -51,7 +66,8 @@ ) STUBBORN_REPLACE_PATTERN = re.compile( r"(?i)\b(?:in order to|aim to|facilitate|serves as)\b") -THREE_PARALLEL_PATTERN = re.compile(r"(?i)not only.+but also.+(?:and|furthermore|moreover)") +THREE_PARALLEL_PATTERN = re.compile( + r"(?i)\bnot only\b.+\bbut also\b.+\b(?:and|furthermore|moreover)\b") # Participial tail that fakes analytic depth ("..., highlighting X"). # Curated verb set adopted 2026-07-16 from the academic-humanizer catalog # (MIT); kept narrow to avoid flagging legitimate participial clauses. @@ -64,23 +80,33 @@ # User style rule 2026-07-16; caption tags ("Left: ...") and list # specifications are legitimate, so this stays an advisory. COLON_ELABORATION_PATTERN = re.compile(r"(?:[A-Za-z0-9]|\})(?:: )[a-z$\\]") -# The two 2026-07-16 rules above/below match rendered prose only: unescaped -# LaTeX comments (full-line or trailing) are stripped before matching. -# Comments conventionally carry "Label: description" tags and note-style -# participial phrases that are not rendered. Older rules keep their -# long-standing raw-line behavior. -TRAILING_COMMENT_PATTERN = re.compile(r"(? list[str]: - if not profile_root.exists(): - return [] - return sorted(path.name for path in profile_root.iterdir() - if path.is_dir() and not path.name.startswith(".")) +list_fields = cli_common.list_fields def resolve_field(arg_field: str | None, profile_root: Path) -> str | None: + """The field to lint against, or None to run the L0 pass alone. + + Kept apart from `cli_common.resolve_field` on purpose: the linter must + still run with no profile, so an explicit field that does not exist + returns None (the caller turns it into exit 2) and several fields with + none named return None with one stderr line -- silently, every calibrated + axis read `unmeasured` and nothing said why. + """ fields = list_fields(profile_root) if arg_field: if arg_field not in fields: @@ -88,19 +114,26 @@ def resolve_field(arg_field: str | None, profile_root: Path) -> str | None: file=sys.stderr) return None return arg_field + if len(fields) > 1: + print(f"[ai_ism_lint] several field profiles present ({fields}); pass " + "--field= to use one. Running the profile-free axes only.", + file=sys.stderr) return fields[0] if len(fields) == 1 else None def load_corpus_blacklist(field_profile_dir: Path | None) -> set[str]: + """`llm_words_absent_from_corpus` from the field lexicon; empty without one. + + A lexicon that cannot be parsed raises: swallowed, it produced zero + corpus-zero advisories under an `L0.lexical: measured` status, while a + lexicon of the wrong shape already exited 2 through `lint`'s guard. + """ if field_profile_dir is None: return set() path = field_profile_dir / "lexicon.json" if not path.exists(): return set() - try: - data = json.loads(path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return set() + data = json.loads(path.read_text(encoding="utf-8")) return set(data.get("llm_words_absent_from_corpus", [])) @@ -145,7 +178,11 @@ def _finding(*, path: Path, kind: str, layer: str, rule: str, line: int, observed: dict[str, Any] | None = None, reference: dict[str, Any] | None = None, normalized_distance: float | None = None, - measurement_status: str = "measured") -> dict[str, Any]: + measurement_status: str = "measured", + offset: int | None = None) -> dict[str, Any]: + # `offset` (the match column) is part of the evidence identity: two hits + # of one rule on one line otherwise shared a `finding_id`. + evidence: list[Any] = [rule, excerpt] + ([offset] if offset is not None else []) return feedback.make_finding( kind=kind, layer=layer, rule=rule, scope="sentence", path=path, line=line, section=section, detector=detector, strength=strength, @@ -153,10 +190,41 @@ def _finding(*, path: Path, kind: str, layer: str, rule: str, line: int, reference=reference, normalized_distance=normalized_distance, confidence={"value": 1.0, "basis": "deterministic lexical match"}, measurement_status=measurement_status, message=message, action=action, - evidence=[rule, excerpt], + evidence=evidence, ) +_SKIPPED_LINE_PATTERN = re.compile(r"\s*\\(?:begin|end|label|vspace|noindent)\b") + + +def _paragraph_first_prose_lines(text: str) -> set[int]: + """Line numbers on which a paragraph's prose starts. + + A paragraph block may open with a heading command or an environment + marker; the first line that is neither is where its prose -- and a + paragraph-initial connector -- can stand. + """ + firsts: set[int] = set() + for start, end, block in deai_metrics.paragraph_line_ranges(text): + for offset, line in enumerate(block.splitlines()): + stripped = line.strip() + if (not stripped or es.RE_HEADING_COMMAND.match(stripped) + or _SKIPPED_LINE_PATTERN.match(line)): + continue + firsts.add(start + offset) + break + return firsts + + +def _sentence_starts_line(lines: list[str], line_no: int, firsts: set[int]) -> bool: + """Whether the start of line `line_no` is the start of a sentence.""" + if line_no in firsts: + return True + previous = next((lines[i].strip() for i in range(line_no - 2, -1, -1) + if lines[i].strip()), "") + return not previous or previous[-1] in ".!?:}" + + def lexical_findings(text: str, path: Path, field_profile_dir: Path | None, ai_classifier: bool = False, @@ -168,11 +236,13 @@ def lexical_findings(text: str, path: Path, per-word cap become L0 targets. """ lines = text.splitlines() + scan_lines = es.blank_preserving(text, *ARGUMENT_SPAN_PATTERNS).splitlines() ranges = section_ranges(text) + firsts = _paragraph_first_prose_lines(text) findings: list[dict[str, Any]] = [] tier_b: dict[tuple[str, str], list[tuple[int, str]]] = defaultdict(list) - for line_no, line in enumerate(lines, start=1): + for line_no, (line, scan) in enumerate(zip(lines, scan_lines), start=1): section = section_for_line(line_no, ranges) excerpt = line.strip() # Resolved before the Tier A/B scans because one paragraph-initial @@ -180,28 +250,34 @@ def lexical_findings(text: str, path: Path, # TIER_A_PATTERN and this connector list, so without the same span guard # the Tier B loop already applies it would be counted twice, while # `Notably`/`Importantly`/`Interestingly` (Tier B) are counted once. - connector = TIER_A_PARAGRAPH_CONNECTOR_PATTERN.match(line) - if EM_DASH_PATTERN.search(line): - findings.append(_finding( - path=path, kind="l0_target", layer="L0", rule="em-dash", - line=line_no, section=section, excerpt=excerpt, - strength="target", message="Em-dash punctuation is an L0 rewrite target.", - action="Replace with punctuation that matches the sentence relation.", - )) - for match in TIER_A_PATTERN.finditer(line): + connector = (TIER_A_PARAGRAPH_CONNECTOR_PATTERN.match(scan) + if line_no in firsts else None) + if not RULE_LINE_PATTERN.fullmatch(scan): + for match in EM_DASH_PATTERN.finditer(scan): + findings.append(_finding( + path=path, kind="l0_target", layer="L0", rule="em-dash", + line=line_no, section=section, excerpt=excerpt, offset=match.start(), + strength="target", message="Em-dash punctuation is an L0 rewrite target.", + action="Replace with punctuation that matches the sentence relation.", + )) + for match in TIER_A_PATTERN.finditer(scan): if connector and match.start() < connector.end(): continue word = match.group(0).lower() findings.append(_finding( path=path, kind="l0_target", layer="L0", rule=f"tier-a:{word}", line=line_no, section=section, excerpt=excerpt, strength="target", + offset=match.start(), message=f"Tier A lexical target {word!r} is present.", action="Rewrite the sentence with a direct field-appropriate expression.", )) - if TIER_A_OPENER_PATTERN.match(line): + for match in TIER_A_OPENER_PATTERN.finditer(scan): + if match.start() == 0 and not _sentence_starts_line(lines, line_no, firsts): + continue findings.append(_finding( path=path, kind="l0_target", layer="L0", rule="tier-a:opener", line=line_no, section=section, excerpt=excerpt, strength="target", + offset=match.start(), message="A zero-reference boilerplate opener is present.", action="Open with the specific scientific problem or evidence.", )) @@ -214,38 +290,37 @@ def lexical_findings(text: str, path: Path, message=f"Paragraph-initial connector {word!r} is an L0 target.", action="Remove the roadmap connector and let the argument carry the transition.", )) - for match in TIER_B_PATTERN.finditer(line): + for match in TIER_B_PATTERN.finditer(scan): if connector and match.start() < connector.end(): continue word = match.group(0).lower() tier_b[(section, word)].append((line_no, excerpt)) - for match in STUBBORN_REPLACE_PATTERN.finditer(line): + for match in STUBBORN_REPLACE_PATTERN.finditer(scan): word = match.group(0).lower() findings.append(_finding( path=path, kind="advisory", layer="L0", rule=f"style-substitution:{word}", line=line_no, - section=section, excerpt=excerpt, + section=section, excerpt=excerpt, offset=match.start(), message=f"The phrase {word!r} is an indirect stylistic construction.", action="Prefer a direct verb when doing so preserves meaning.", )) - if THREE_PARALLEL_PATTERN.search(line): + if THREE_PARALLEL_PATTERN.search(scan): findings.append(_finding( path=path, kind="advisory", layer="L2", rule="three-parallel", line=line_no, section=section, excerpt=excerpt, message="The sentence uses a conspicuous three-part parallel frame.", action="Keep the parallelism only if it carries a real logical distinction; otherwise simplify it.", )) - prose_part = TRAILING_COMMENT_PATTERN.sub("", line) - for match in ING_TAIL_PATTERN.finditer(prose_part): + for match in ING_TAIL_PATTERN.finditer(scan): word = match.group(0).lstrip(", ").lower() findings.append(_finding( path=path, kind="advisory", layer="L2", rule=f"ing-tail:{word}", - line=line_no, section=section, excerpt=excerpt, + line=line_no, section=section, excerpt=excerpt, offset=match.start(), message=(f"A participial tail {word!r} appends interpretation " "instead of stating it as a claim."), action="Promote the tail to its own sentence with a subject and evidence, or delete it.", )) - if COLON_ELABORATION_PATTERN.search(prose_part): + if COLON_ELABORATION_PATTERN.search(scan): findings.append(_finding( path=path, kind="advisory", layer="L2", rule="colon-elaboration", line=line_no, section=section, excerpt=excerpt, @@ -276,12 +351,12 @@ def lexical_findings(text: str, path: Path, if blacklist: pattern = re.compile(r"\b(" + "|".join(re.escape(word) for word in blacklist) + r")\b", re.IGNORECASE) - for line_no, line in enumerate(lines, start=1): - for match in pattern.finditer(line): + for line_no, (line, scan) in enumerate(zip(lines, scan_lines), start=1): + for match in pattern.finditer(scan): word = match.group(1).lower() findings.append(_finding( path=path, kind="advisory", layer="L0", - rule=f"corpus-zero:{word}", line=line_no, + rule=f"corpus-zero:{word}", line=line_no, offset=match.start(), section=section_for_line(line_no, ranges), excerpt=line.strip(), message=f"The field lexicon records zero occurrences of {word!r}.", action="Inspect the corpus evidence and rewrite only if the term is not scientifically necessary.", @@ -352,10 +427,13 @@ def document_source(path: Path) -> str: Comment spans are blanked to equal-length runs of spaces rather than deleted, so every line number and character offset a finding reports still - points where it did. + points where it did. An unreadable root raises (a directory named `x.tex` + read as an empty document and exited 0, clean), and a Markdown input has + its front-matter block blanked the same way. """ if path.suffix.lower() != ".tex": - return path.read_text(encoding="utf-8", errors="replace") + text = path.read_text(encoding="utf-8", errors="replace") + return FRONT_MATTER_PATTERN.sub(lambda m: re.sub(r"[^\n]", " ", m.group(0)), text) return es.RE_TEX_COMMENT.sub(lambda m: " " * len(m.group(0)), es.read_tex_document(path)) @@ -436,7 +514,7 @@ def lint(path: Path, field_profile_dir: Path | None, summary: bool = False, register: bool = True, salience: bool = True, discourse: bool = True, collocation: bool = True, residue: bool = True, glossary: bool = False) -> int: - del summary # retained as a compatibility option; reports always include totals + del summary # accepted and hidden: totals are always in the report if not path.exists(): print(f"[ai_ism_lint] file not found: {path}", file=sys.stderr) return 2 @@ -469,39 +547,22 @@ def lint(path: Path, field_profile_dir: Path | None, summary: bool = False, def main(argv: list[str] | None = None) -> int: - if hasattr(sys.stdout, "reconfigure"): - sys.stdout.reconfigure(encoding="utf-8", errors="replace") - parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + cli_common.utf8_stdout() + parser = cli_common.base_parser(__doc__, parents=[cli_common.field_options()]) parser.add_argument("file", type=Path) - parser.add_argument("--field", default=None) - parser.add_argument("--profile-root", type=Path, default=DEFAULT_PROFILE_ROOT) - parser.add_argument("--summary", action="store_true", - help="compatibility flag; totals are always included") + parser.add_argument("--summary", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--ai-classifier", action="store_true") parser.add_argument("--ai-threshold", type=float, default=0.7) - parser.add_argument("--distribution", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--structure", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--document-structure", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--register", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--salience", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--discourse", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--collocation", action=argparse.BooleanOptionalAction, - default=True) - parser.add_argument("--residue", action=argparse.BooleanOptionalAction, - default=True) + for axis in ("distribution", "structure", "document-structure", "register", + "salience", "discourse", "collocation", "residue"): + parser.add_argument(f"--{axis}", action=argparse.BooleanOptionalAction, + default=True) parser.add_argument("--glossary", action="store_true", help="also list the recurring unattested word pairs as " "glossary candidates (deai_collocation --glossary)") parser.add_argument("--oracle", action="store_true") parser.add_argument("--voice", action="store_true") - parser.add_argument("--format", choices=("text", "json"), default="text") - parser.add_argument("--output", type=Path) + cli_common.report_options(parser) parser.add_argument("--top", type=int) args = parser.parse_args(argv) field = resolve_field(args.field, args.profile_root) diff --git a/tools/condense_map.py b/tools/condense_map.py index 7c3f07d..b0d8e81 100644 --- a/tools/condense_map.py +++ b/tools/condense_map.py @@ -25,9 +25,18 @@ their content words (Jaccard) The abstract and the conclusion restate the paper by convention (the genre -carve-out of the condense skill). Their restatements are still reported, with +carve-out of the condense skill). A restatement is carved out when EITHER side +of the pair sits in one of them: the abstract precedes the body, so with only +the copy's bucket consulted every abstract claim made the body's own statement +a restatement whose "canonical home" was the abstract -- the inversion of the +skill's rule. Carved-out entries are still reported, with `observed.genre_carve_out` true, and are left out of `default_target`. +Every word count here is a rendered-prose count (`extract_sections.prose_words` +and its placeholder rule): the gate that closes on this map's target counts +the same way, and a target that included `[math]` and `[CITE]` tokens could +not be met by deleting exactly what the map listed. + Exit status: 0 measured, 2 invalid input. The map does not judge; the skill does, entry by entry. """ @@ -82,14 +91,22 @@ RE_VERBOSE = re.compile(r"\b(?:" + "|".join(re.escape(k) for k in VERBOSE) + r")\b", re.I) RE_HEDGE_STACK = re.compile(r"\b(?:may|might|could)\s+(?:possibly|potentially|perhaps)\b", re.I) RE_TEX_LABEL = re.compile(r"\\label\{([^}]*)\}") +# A reference is any `\...ref{...}` command, matched by shape the way the +# citation projection matches `\...cite...`: a four-name allowlist read a +# figure whose only consumer was `\subref{fig:a}` (or `\vref`, `\fref`, +# `\zref`) as dead. RE_TEX_REF = re.compile( - r"\\(?:ref|eqref|cref|Cref|autoref|pageref|labelcref|nameref)\*?\{([^}]*)\}" + r"\\[A-Za-z]*ref\*?\{([^}]*)\}" r"|\\hyperref\[([^\]]*)\]") RE_TEX_FLOAT = re.compile(r"\\begin\{(figure|table)\*?\}(.*?)\\end\{\1\*?\}", re.DOTALL) -RE_TEX_CAPTION = re.compile(r"\\caption(?:\[[^\]]*\])?\{((?:[^{}]|\{[^{}]*\})*)\}") +# `\caption*` and two levels of nested braces (`\textbf{\emph{...}}`): either +# form made the caption invisible and the dead float carry zero words. +RE_TEX_CAPTION = re.compile( + r"\\caption\*?(?:\[[^\]]*\])?\{((?:[^{}]|\{(?:[^{}]|\{[^{}]*\})*\})*)\}") RE_TEX_MACRO_DEF = re.compile( r"\\(?:newcommand|renewcommand|providecommand|def)\s*\{?\\([A-Za-z]+)\}?") -RE_ACRONYM = re.compile(r"\(([A-Z][A-Za-z0-9]{1,9})\)") +# An acronym is at least two capitals or digits; `(Gaussian)` is a parenthesis. +RE_ACRONYM = re.compile(r"\(([A-Z][A-Z0-9]{1,9})\)") RE_REGLOSS = re.compile( r"\bwhere\s+\$([^$]+)\$\s+(?:is|denotes|represents|stands for)\b[^.;]*", re.I) @@ -102,7 +119,10 @@ def content_words(sentence: str) -> set[str]: # What a bag of content words cannot see and a restatement must not change: a # negation, a number (digits or the number words the stopword list drops), a # comparison. `is significant` and `is not significant` share every content -# word; before this the second was a restatement of the first. +# word; before this the second was a restatement of the first. The sets must +# be EQUAL: a copy that drops the negation its home carries is as much a +# different claim as one that adds it, and deleting it would leave the +# negated version standing for both. RE_INVARIANT = re.compile( r"\b(?:not|no|never|without|neither|nor|cannot|only|fewer|less|more|larger|" r"smaller|higher|lower|greater|above|below|increases?d?|decreases?d?|" @@ -137,6 +157,11 @@ def _finding(*, rule: str, path: str | Path | None, line: int, end_line: int | N measurement_status="measured", evidence=[rule, removable]) +def _prose_count(plain: str) -> int: + """Rendered-prose tokens of a projected string: the gate's rule.""" + return sum(1 for token in plain.split() if not es.RE_PLACEHOLDER.match(token)) + + def _sentence_records(text: str) -> list[dict[str, Any]]: records = [] for start, end, bucket, block in reference.units(text): @@ -147,7 +172,7 @@ def _sentence_records(text: str) -> list[dict[str, Any]]: "unit": f"{start}:{index}", "text": clean, "words": content_words(clean), "invariants": invariants(clean), - "n_words": len(clean.split())}) + "n_words": _prose_count(clean)}) return records @@ -158,20 +183,31 @@ def restatement_findings(records: list[dict[str, Any]], path=None) -> list[dict[ words = record["words"] if len(words) >= MIN_CONTENT_WORDS: union = len(words & seen) / len(words) - best_index, best = -1, 0.0 + # The best earlier sentence OUTSIDE the carve-out buckets is the + # canonical home when it covers enough on its own; only when no + # body sentence does is the abstract or conclusion the home, and + # then the pair is the genre's restatement, not the body's. + best_body: tuple[int, float] = (-1, 0.0) + best_any: tuple[int, float] = (-1, 0.0) for earlier_index in range(index): - earlier = records[earlier_index]["words"] - if not earlier: + earlier = records[earlier_index] + if not earlier["words"]: continue - coverage = len(words & earlier) / len(words) - if coverage > best: - best_index, best = earlier_index, coverage + coverage = len(words & earlier["words"]) / len(words) + if coverage > best_any[1]: + best_any = (earlier_index, coverage) + if (earlier["bucket"] not in CARVE_OUT_BUCKETS + and coverage > best_body[1]): + best_body = (earlier_index, coverage) + best_index, best = best_body if best_body[1] >= SINGLE_COVERAGE else best_any home = records[best_index] if best_index >= 0 else None - # A sentence that adds a negation, a number or a comparison its - # home does not carry restates nothing; it is a different claim. + # A sentence whose negations, numbers or comparisons differ from + # its home's in either direction restates nothing; it is a + # different claim. if (union >= UNION_COVERAGE and best >= SINGLE_COVERAGE - and record["invariants"] <= home["invariants"]): - carve = record["bucket"] in CARVE_OUT_BUCKETS + and record["invariants"] == home["invariants"]): + carve = (record["bucket"] in CARVE_OUT_BUCKETS + or home["bucket"] in CARVE_OUT_BUCKETS) findings.append(_finding( rule="condense-restatement", path=path, line=record["line"], end_line=record["end"], section=record["bucket"], scope="sentence", @@ -290,7 +326,11 @@ def verbose_findings(records: list[dict[str, Any]], path=None) -> list[dict[str, for record in records: for match in RE_VERBOSE.finditer(record["text"]): phrase = match.group(0).lower() - shorter = VERBOSE[phrase] + # `.get`: a case-insensitive match can lower-case to a key the + # table does not hold (`İn order to` -> `i̇n order to`, KeyError). + shorter = VERBOSE.get(phrase) + if shorter is None: + continue findings.append(_finding( rule="condense-verbose", path=path, line=record["line"], end_line=record["end"], section=record["bucket"], scope="sentence", @@ -334,7 +374,7 @@ def duplicate_findings(text: str, path=None) -> list[dict[str, Any]]: paragraphs = [] for start, end, bucket, block in reference.units(text): plain = es.latex_to_plain(block) - if len(plain.split()) >= DUPLICATE_MIN_WORDS: + if _prose_count(plain) >= DUPLICATE_MIN_WORDS: paragraphs.append((start, end, bucket, plain, content_words(plain))) findings = [] for j, (start, end, bucket, plain, words) in enumerate(paragraphs): @@ -348,7 +388,7 @@ def duplicate_findings(text: str, path=None) -> list[dict[str, Any]]: carve = bucket in CARVE_OUT_BUCKETS or e_bucket in CARVE_OUT_BUCKETS findings.append(_finding( rule="condense-duplicate", path=path, line=start, end_line=end, - section=bucket, scope="paragraph", removable=len(plain.split()), + section=bucket, scope="paragraph", removable=_prose_count(plain), confidence=round(jaccard, 2), unit=f"{start}:*", whole=True, observed={"excerpt": plain[:160], "canonical_line": e_start, "canonical_section": e_bucket, "jaccard": round(jaccard, 3), @@ -402,8 +442,10 @@ def removable_words(findings: list[dict[str, Any]]) -> int: def condense_budget(text: str, findings: list[dict[str, Any]]) -> dict[str, Any]: """`removable_by_rule` is candidate mass per scan (overlaps included, so a reader sees what each scan found); `removable_total` and the default - target count each unit once.""" - prose_words = len(es.prose_words(text)) + target count each unit once. The denominator is the gate's: body prose + without heading words, so `default_target_fraction` is a fraction of the + number `length_gate` reports.""" + prose_words = len(es.prose_words(es.RE_HEADING_COMMAND.sub("", text))) by_rule: dict[str, int] = {} for finding in findings: family = finding["rule"].split(":")[0] diff --git a/tools/extract_sections.py b/tools/extract_sections.py index 5fa2268..3a783a2 100644 --- a/tools/extract_sections.py +++ b/tools/extract_sections.py @@ -97,8 +97,15 @@ def classify_section(name: str) -> str: # Light LaTeX cleaning. Not a full TeX parser; meant to remove enough # command/environment noise so word/sentence stats reflect the prose. RE_TEX_COMMENT = tex_assembly.RE_TEX_COMMENT # one owner, see tex_assembly +# Display mathematics in every form LaTeX writes it: the environments, the +# `\[ ... \]` shorthand, and old-style `$$ ... $$`. The last two were missing: +# `\[E=mc^2\]` counted as six words of prose in the length gate and the map, +# and `$$` was left to the INLINE pattern, which paired its second dollar with +# the third and swallowed the prose up to the next inline formula in the plain +# view only, so the two projections no longer made the same substitutions. RE_TEX_DISPLAY_MATH = re.compile( - r"\\begin\{(equation|align|gather|eqnarray|displaymath|multline)\*?\}.*?\\end\{\1\*?\}", + r"\\begin\{(equation|align|gather|eqnarray|displaymath|multline|subequations)\*?\}" + r".*?\\end\{\1\*?\}|\\\[.*?\\\]|\$\$.*?\$\$", re.DOTALL) RE_TEX_INLINE_MATH = re.compile(r"\$[^$]+\$|\\\(.+?\\\)", re.DOTALL) RE_TEX_ENV_FIGURE_TABLE = re.compile( # deluxetable/longtable/tabular are tables too @@ -141,9 +148,8 @@ def classify_section(name: str) -> str: def _math_numerals(match: "re.Match[str]") -> str: """Reduce one math span to its bare numerals and operators, on one line. - Old-style `$$ ... $$` display math is matched by the inline pattern from - its second dollar, so a span can run across a blank line; kept, that - blank line would be a paragraph break the plain view (`[math]`) does not + An inline span can still run across a line break; kept, a blank line + inside it would be a paragraph break the plain view (`[math]`) does not have, and the two views could not be paired paragraph by paragraph. """ body = RE_TEX_THIN_COMMA.sub(",", match.group(0)) diff --git a/tools/extract_style.py b/tools/extract_style.py index a9d1d72..ede3a1f 100644 --- a/tools/extract_style.py +++ b/tools/extract_style.py @@ -1,22 +1,19 @@ """Extract descriptive, field-scoped writing evidence from a paper corpus. -The extractor reads standalone ``.tex``, ``.txt``, and ``.pdf`` sources under -``style-corpus//tier-*`` and writes descriptive artifacts under -``style-profile//``: sentence statistics, paragraph-initial transitions, -lexical counts, an exemplar JSONL bank, and a compact dossier. PDF ingestion is -best-effort and requires pymupdf. - -This module does not define consequence classes, authorship, or calibrated -operating points. Its section detection and PDF block segmentation are -heuristic; unmatched headings fall to ``unknown`` and are dropped rather than -absorbed into a default bucket. Fix extraction errors in the source or this -extractor and regenerate rather than hand-editing generated evidence. Normative -policy lives in ``docs/SCIPAPER_STANDARD.md``. +Reads ``.tex``, ``.txt`` and ``.pdf`` sources under ``style-corpus//tier-*`` +and writes descriptive artifacts under ``style-profile//``: sentence +statistics, paragraph-initial transitions, lexical counts, an exemplar JSONL +bank and a dossier. PDF ingestion needs pymupdf; ``.txt`` sources carry no +sections, so they feed the statistics under ``unknown`` and never the bank. + +This module defines no consequence class, authorship or operating point; its +section detection is heuristic and an unmatched heading is ``unknown``, never a +default bucket. Fix extraction in the source or here and regenerate; never +hand-edit generated evidence. Policy lives in ``docs/SCIPAPER_STANDARD.md``. """ from __future__ import annotations -import argparse import json import re import statistics @@ -35,24 +32,18 @@ # is hand-written, so `ReExportContractTests` asserts it stays complete -- # it has caught three additions already. from extract_sections import ( # noqa: F401 -- re-export, unused here by design - CALIBRATION_FULLTEXT, CLASSIFIED_BUCKETS, _classified_word_count, - DEFAULT_SECTION_BUCKET, LIGATURE_TABLE, RE_ABSTRACT_ENV, - RE_HEADING_CMD, RE_HEADING_COMMAND, RE_HEADING_DROP_ARG, RE_HEADING_MATH, - RE_HEADING_TEXORPDF, - RE_PDF_LINE_HEADER, RE_PLACEHOLDER, RE_SECTION, RE_TEX_BEGIN_END, RE_TEX_BRACES, - RE_TEX_CITE, RE_TEX_CITE_SILENT, RE_TEX_CITE_TEXT, - RE_TEX_COMMENT, RE_TEX_DISPLAY_MATH, - RE_TEX_ENV_FIGURE_TABLE, RE_TEX_INCLUDEGRAPHICS, RE_TEX_INLINE_MATH, - RE_TEX_DOC_MARKER, RE_TEX_INCLUDE, - RE_TEX_LABEL_REF, RE_TEX_MATH_CMD, RE_TEX_SIMPLE_CMD, RE_TEX_THIN_COMMA, - RE_TEX_TILDE, RE_SENTENCE_TERMINAL, SECTION_PATTERNS, + CALIBRATION_FULLTEXT, CLASSIFIED_BUCKETS, _classified_word_count, DEFAULT_SECTION_BUCKET, + LIGATURE_TABLE, RE_ABSTRACT_ENV, RE_HEADING_CMD, RE_HEADING_COMMAND, RE_HEADING_DROP_ARG, + RE_HEADING_MATH, RE_HEADING_TEXORPDF, RE_PDF_LINE_HEADER, RE_PLACEHOLDER, RE_SECTION, + RE_TEX_BEGIN_END, RE_TEX_BRACES, RE_TEX_CITE, RE_TEX_CITE_SILENT, RE_TEX_CITE_TEXT, + RE_TEX_COMMENT, RE_TEX_DISPLAY_MATH, RE_TEX_ENV_FIGURE_TABLE, RE_TEX_INCLUDEGRAPHICS, + RE_TEX_INLINE_MATH, RE_TEX_DOC_MARKER, RE_TEX_INCLUDE, RE_TEX_LABEL_REF, RE_TEX_MATH_CMD, + RE_TEX_SIMPLE_CMD, RE_TEX_THIN_COMMA, RE_TEX_TILDE, RE_SENTENCE_TERMINAL, SECTION_PATTERNS, PDF_HEADING_MIN_LETTER_FRAC, PDF_HEADING_MIN_LETTERS, PDF_HEADING_MIN_WORDS, _classify_pdf_heading, _include_targets, _math_numerals, _resolve_include, - _rejoin_pdf_paragraphs, blank_preserving, classify_section, clean_heading, - corpus_documents, extract_pdf_text, latex_to_numeral_text, latex_to_plain, - PLAIN_PLACEHOLDERS, _project, - prose_words, - read_tex_document, select_document_roots, split_into_sections, + _rejoin_pdf_paragraphs, blank_preserving, classify_section, clean_heading, corpus_documents, + extract_pdf_text, latex_to_numeral_text, latex_to_plain, PLAIN_PLACEHOLDERS, _project, + prose_words, read_tex_document, select_document_roots, split_into_sections, split_pdf_into_sections, ) @@ -60,19 +51,18 @@ DEFAULT_CORPUS_ROOT = REPO_ROOT / "style-corpus" DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" +# The curated tiers. The weights are RECORDED metadata (they travel with each +# observation) and are NOT applied: every aggregate pools the tiers equally, +# as `aggregate_sentence_stats` says. The dict also names which directories +# are curated; everything else under the field is breadth. TIER_WEIGHTS = {"tier-1-top": 0.5, "tier-2-mentor": 0.3, "tier-3-reference": 0.2} -# Breadth corpus, distinct in role from the curated tiers above. The tiers -# define what the dossier says we *imitate* and carry all weighted aggregate -# statistics; this directory holds the bulk arXiv full-text pull -# (fetch_arxiv_abstracts.py --fulltext) and exists so the per-section -# *reference distributions* have enough observations to be measurable at all. -# It is unweighted and excluded from every aggregate on purpose: breadth must -# not restyle the imitation target. Both roles previously shared one file -# list, which is why `results` sat at 26 observations -- under its own -# 30-passage floor -- while 500 field papers sat unread on disk. -# The name is owned by extract_sections, which needs it to tell this directory -# apart from the held-out `fulltext-*` siblings it must refuse to collect. +# Breadth corpus: the bulk arXiv full-text pull (fetch_arxiv_abstracts.py +# --fulltext), gathered so the per-section reference distributions have enough +# observations, and excluded from the dossier aggregates so breadth cannot +# restyle the imitation target (with one list, `results` sat at 26 passages +# while 500 field papers sat unread). The name is owned by extract_sections, +# which must tell it apart from the held-out `fulltext-*` siblings. REFERENCE_DIR = CALIBRATION_FULLTEXT @@ -88,44 +78,39 @@ def resolve_field(arg_field: str | None, corpus_root: Path) -> str: -# Placeholder strings emitted by latex_to_plain; paragraphs starting with -# these are NOT real prose paragraphs and should be excluded from -# paragraph-initial-word stats. -PLACEHOLDER_PARAGRAPH_PREFIXES = ( - "[MATH]", "[math]", "[CITE]", "[FIGURE-OR-TABLE]", -) -PLACEHOLDER_INITIAL_WORDS = {"FIGURE", "MATH", "CITE", "OR"} +# Placeholder tokens emitted by latex_to_plain. A paragraph that is NOTHING but +# placeholders is not prose; one that merely STARTS with one (`\citet{X} +# showed ...`) is, and until 2026-09-27 it was dropped whole from the bank and +# the opener statistics, so `intro` lost its citation-led paragraphs. Nor are +# the tokens words: counted, they inflated sentence lengths, the em-dash +# denominator and the lexicon, and let a 25-word paragraph into the bank. +_RE_PLAIN_PLACEHOLDER_TOKEN = re.compile(r"\[(?:MATH|math|FIGURE-OR-TABLE|CITE)\]") + + +def without_placeholders(text: str) -> str: # every placeholder becomes a space + return _RE_PLAIN_PLACEHOLDER_TOKEN.sub(" ", text) # Candidate generated-style terms summarized against the current corpus. # This compatibility list is descriptive; normative Tier A/Tier B policy lives # in docs/SCIPAPER_STANDARD.md and is not inferred from corpus absence alone. +# The second block was adopted 2026-07-16 from the academic-humanizer catalog +# (github.com/AIScientists-Dev/academic-humanizer, MIT); "landscape" is left +# out because it is a domain term in the astro corpus (detection landscape). LLM_TYPICAL_WORDS = { - "leverage", "leverages", "leveraging", "leveraged", - "utilize", "utilizes", "utilizing", "utilized", - "delve", "delves", "delving", "delved", - "showcase", "showcases", "showcasing", - "shed", "sheds", "shedding", # "shed light" - "pave", "paves", "paving", - "seamless", "seamlessly", - "comprehensive", "comprehensively", - "robust", "robustly", - "holistic", "holistically", - "moreover", "furthermore", "additionally", - "notably", "importantly", "crucially", "interestingly", - # Candidates adopted 2026-07-16 from the academic-humanizer catalog - # (github.com/AIScientists-Dev/academic-humanizer, MIT); "landscape" is - # deliberately excluded because it is a legitimate domain term in the - # astro corpus (e.g. detection landscape, energy landscape). - "underscore", "underscores", "underscored", "underscoring", - "intricate", "tapestry", "testament", - "pivotal", "foster", "fosters", "fostering", "fostered", - "realm", "realms", + "leverage", "leverages", "leveraging", "leveraged", "utilize", "utilizes", + "utilizing", "utilized", "delve", "delves", "delving", "delved", "showcase", + "showcases", "showcasing", "shed", "sheds", "shedding", "pave", "paves", "paving", + "seamless", "seamlessly", "comprehensive", "comprehensively", "robust", "robustly", + "holistic", "holistically", "moreover", "furthermore", "additionally", "notably", + "importantly", "crucially", "interestingly", + "underscore", "underscores", "underscored", "underscoring", "intricate", "tapestry", + "testament", "pivotal", "foster", "fosters", "fostering", "fostered", "realm", "realms", } PARAGRAPH_INITIAL_LLM_OPENERS = { - "Furthermore", "Moreover", "Additionally", "Notably", "Importantly", - "Crucially", "Interestingly", "It is worth", "Recent advances", - "Despite significant", "With the advent", "In recent years", + "Furthermore", "Moreover", "Additionally", "Notably", "Importantly", "Crucially", + "Interestingly", "It is worth", "Recent advances", "Despite significant", + "With the advent", "In recent years", } @@ -139,33 +124,44 @@ def sentences(text: str) -> list[str]: return [s for s in RE_SENTENCE_END.split(text) if s.strip()] +# A word is a run of letters in any script: the ASCII class split `naïve` into +# `na` and `ve` and `Poincaré` into `Poincar`, and the fragments entered the +# lexicon and the word counts, the same class of defect as the PDF ligatures. +RE_WORD_TOKEN = re.compile(r"[^\W\d_][^\W\d_'\-]*") + + def words(text: str) -> list[str]: - return re.findall(r"[A-Za-z][A-Za-z'\-]*", text) + return RE_WORD_TOKEN.findall(text) + + +def _prose_paragraphs(text: str) -> list[str]: + """Each paragraph of projected text with its placeholders removed; a + paragraph that was only placeholders is dropped.""" + out = [] + for p in RE_PARAGRAPH_BREAK.split(text): + p = without_placeholders(p).strip() + if p: + out.append(p) + return out def paragraph_initial_words(text: str) -> list[str]: - paras = RE_PARAGRAPH_BREAK.split(text) out = [] - for p in paras: - p = p.strip() - if not p: - continue - # Skip paragraphs that are nothing but our own placeholders - # (figures, tables, equations, citations). - if p.startswith(PLACEHOLDER_PARAGRAPH_PREFIXES): - continue - first_match = re.search(r"[A-Za-z]+", p) + for p in _prose_paragraphs(text): + first_match = RE_WORD_TOKEN.search(p) if first_match: - word = first_match.group(0) - # Skip the bare placeholder words themselves if they survived - # (e.g., paragraph starts with " [FIGURE-OR-TABLE] ..." after - # whitespace stripping leaves "[FIGURE-OR-TABLE]" as content). - if word in PLACEHOLDER_INITIAL_WORDS: - continue - out.append(word) + out.append(first_match.group(0)) return out +def paragraph_initial_phrases(text: str) -> list[str]: + """Multi-word openers a paragraph starts with. `paragraph_initial_words` + records one WORD per paragraph, so the phrases read as always absent.""" + phrases = [p for p in PARAGRAPH_INITIAL_LLM_OPENERS if " " in p] + return [phrase for p in _prose_paragraphs(text) + for phrase in phrases if p.startswith(phrase)] + + def count_em_dashes(text: str) -> int: return len(re.findall(r"—|---|\\textemdash", text)) @@ -255,26 +251,30 @@ def analyse_paper(path: Path) -> dict | None: sec_plain, sec_numeral = paired_paragraphs(sec_raw) else: sec_plain, sec_numeral = sec_raw, RE_PARAGRAPH_BREAK.split(sec_raw) - sents = sentences(sec_plain) + # Statistics read the prose without its placeholders; the bank keeps + # `plain_text` with them, as slot markers the paired view relies on. + prose = without_placeholders(sec_plain) + sents = sentences(prose) sent_lens = [len(words(s)) for s in sents if len(words(s)) > 0] by_section[sec] = { "n_sentences": len(sents), "sentence_lengths": sent_lens, "n_words": sum(sent_lens), - "em_dash_count": count_em_dashes(sec_plain), + # Counted on the comment-stripped source for .tex: the projection + # has already deleted `\textemdash`, so counting there missed it. + "em_dash_count": count_em_dashes(RE_TEX_COMMENT.sub("", sec_raw) + if is_tex else sec_plain), "paragraph_initial_words": paragraph_initial_words(sec_plain), - "word_counter": Counter(w.lower() for w in words(sec_plain)), + "paragraph_initial_phrases": paragraph_initial_phrases(sec_plain), + "word_counter": Counter(w.lower() for w in words(prose)), # Plain prose text retained for exemplar-bank construction. # Numbers/citations stripped to placeholders; safe to chunk by # paragraph and ship as style anchors. "plain_text": sec_plain, # Each paragraph of `plain_text` under the numeral-preserving - # projection (None when the two views could not be paired), so - # the salience reference is built from the text the manuscript - # side measures. Calibrated on `plain_text`, the reference saw - # `[math]` where a manuscript shows `0.81`, and the p90 gate - # fired at 0.45 per passage on held-out papers against a 0.27 - # design rate (held-out-labels.md §17.5). + # projection (None when the views could not be paired), so the + # salience reference sees `0.81` where the manuscript does; on + # `plain_text` it saw `[math]` (held-out-labels.md §17.5). "numeral_paragraphs": sec_numeral, } @@ -304,8 +304,6 @@ def aggregate_sentence_stats(per_paper: list[tuple[float, dict]]) -> dict: for sec, weighted in bucket.items(): if not weighted: continue - # Expand by integer weight approximation for simple stats. - # (Proper weighted percentiles would need numpy; stdlib approximation OK for v0.1.) flat = [L for _w, L in weighted] out[sec] = { "n": len(flat), @@ -342,26 +340,32 @@ def aggregate_em_dashes(per_paper: list[tuple[float, dict]]) -> dict: def aggregate_transitions(per_paper: list[tuple[float, dict]]) -> dict: counter: Counter[str] = Counter() + phrases: Counter[str] = Counter() for _w, paper in per_paper: for sec, st in paper["by_section"].items(): for w in st["paragraph_initial_words"]: counter[w] += 1 + for phrase in st.get("paragraph_initial_phrases", ()): + phrases[phrase] += 1 - total = sum(counter.values()) or 1 + n_paragraphs = sum(counter.values()) + total = n_paragraphs or 1 used = [(w, c, c / total) for w, c in counter.most_common()] - used_set = {w for w, _, _ in used} + + def seen(opener: str) -> int: + return phrases.get(opener, 0) if " " in opener else counter.get(opener, 0) forbidden_present = [ - (w, counter.get(w, 0)) - for w in PARAGRAPH_INITIAL_LLM_OPENERS - if counter.get(w, 0) > 0 + (w, seen(w)) for w in PARAGRAPH_INITIAL_LLM_OPENERS if seen(w) > 0 ] forbidden_absent = [ - w for w in PARAGRAPH_INITIAL_LLM_OPENERS if counter.get(w, 0) == 0 + w for w in PARAGRAPH_INITIAL_LLM_OPENERS if seen(w) == 0 ] return { - "n_paragraphs": total, + # The real count: `or 1` here made an empty corpus report one + # paragraph, and the dossier's "No paragraphs detected" unreachable. + "n_paragraphs": n_paragraphs, "whitelist_observed": [ {"word": w, "count": c, "freq": f} for w, c, f in used[:30] @@ -382,7 +386,6 @@ def aggregate_transitions(per_paper: list[tuple[float, dict]]) -> dict: # A paragraph boundary in projected text. One owner: the bank writer, the # paired projection and the opener statistics must cut at the same places. RE_PARAGRAPH_BREAK = re.compile(r"\n\s*\n") -_RE_PLAIN_PLACEHOLDER_TOKEN = re.compile(r"\[(?:MATH|math|FIGURE-OR-TABLE|CITE)\]") def _math_numerals_slotted(match: "re.Match[str]") -> str: @@ -397,13 +400,12 @@ def paired_paragraphs(text: str) -> tuple[str, list[str] | None]: """`latex_to_plain(text)` and, per paragraph of it, the numeral projection. The bank stores the plain paragraph and the salience axis calibrates on - the numeral one, so they must be the same paragraph. Splitting each - projection on blank lines does not pair them: a paragraph that is only a - displayed equation is `[MATH]` in one and whitespace in the other, and - the splitter swallows the whitespace and shifts every later index. So the - numeral view is projected WITH the plain placeholders, split on the same - boundaries, and stripped of them after. None for the list when the two - views still disagree: the caller writes no pairing rather than a wrong one. + the numeral one, so they must be the same paragraph. Split on blank lines + alone the views do not pair (an equation-only paragraph is `[MATH]` in one + and swallowed whitespace in the other), so the numeral view is projected + WITH the plain placeholders as slot markers, split on the same boundaries + and stripped after. Both views run one `_project`, so the counts agree by + construction; the None branch is a guard, not a live outcome. """ plain = latex_to_plain(text) slotted = _project(text, inline=_math_numerals_slotted, **PLAIN_PLACEHOLDERS) @@ -426,10 +428,12 @@ def write_exemplar_bank(per_paper: list[tuple[float, dict]], profile_dir: Path) -> int: """Emit one JSONL row per qualifying paragraph in `exemplar_paragraphs.jsonl`. - Each row: {id, section, tier, source, n_words, text, numeral_text}. - Section is the normalized bucket from classify_section(); - rows in the 'unknown' bucket and rows whose paragraphs are pure - placeholders are excluded. Returns the number of rows written. + Each row: {id, section, tier, source, n_words, text, numeral_text}. The + id carries the bucket, because a per-bucket index made `paper.tex:p0` + the id of two rows. Section is the normalized bucket from + classify_section(); rows in the 'unknown' bucket and rows whose + paragraphs are pure placeholders are excluded, and `n_words` counts the + prose without its placeholders. Returns the number of rows written. `numeral_text` is the same paragraph under `latex_to_numeral_text`, paired by `paired_paragraphs`. A section it could not pair is reported and @@ -455,15 +459,13 @@ def write_exemplar_bank(per_paper: list[tuple[float, dict]], unaligned.append(f"{source}:{sec}") for idx, para in enumerate(paragraphs): para = para.strip() - if not para: - continue - if para.startswith(PLACEHOLDER_PARAGRAPH_PREFIXES): + if not without_placeholders(para).strip(): continue - n_w = len(words(para)) + n_w = len(words(without_placeholders(para))) if n_w < EXEMPLAR_MIN_WORDS or n_w > EXEMPLAR_MAX_WORDS: continue rec = { - "id": f"{source}:p{idx}", + "id": f"{source}:{sec}:p{idx}", "section": sec, "tier": tier, "source": source, @@ -633,7 +635,6 @@ def main(argv: list[str] | None = None) -> int: field = resolve_field(args.field, args.corpus_root) field_corpus = args.corpus_root / field field_profile = args.profile_root / field - field_profile.mkdir(parents=True, exist_ok=True) print(f"[extract_style] field={field!r}") print(f" corpus: {field_corpus}") @@ -652,6 +653,10 @@ def main(argv: list[str] | None = None) -> int: if n_curated < 5: print(f"[extract_style] WARNING: only {n_curated} curated corpus files. " "Statistics will be noisy; recommended ≥ 8.") + # Created only once the corpus is known to hold something: a failed run + # left an empty field behind, and every other tool's `resolve_field` then + # reported "Multiple fields present". + field_profile.mkdir(parents=True, exist_ok=True) # `per_paper` drives the weighted aggregates and the dossier and holds the # curated tiers only; `reference_papers` is the unweighted breadth corpus. @@ -659,6 +664,7 @@ def main(argv: list[str] | None = None) -> int: # every L1/L2 axis measures against, and needs the observations. per_paper: list[tuple[float, dict]] = [] reference_papers: list[tuple[float, dict]] = [] + skipped = 0 for source, files in files_by_tier.items(): weight = TIER_WEIGHTS.get(source, 0.0) sink = per_paper if source in TIER_WEIGHTS else reference_papers @@ -671,7 +677,9 @@ def main(argv: list[str] | None = None) -> int: continue if analysis is None: # analyse_paper returns None for files it knows it cannot - # handle (e.g., PDF + pymupdf missing); already logged. + # handle (e.g., PDF + pymupdf missing); already logged, and + # counted so the summary says the profile is built without them. + skipped += 1 continue # Tag tier + a stable, repo-relative source path so the exemplar # writer can include both fields per row without rebuilding state. @@ -725,7 +733,9 @@ def main(argv: list[str] | None = None) -> int: n_exemplars = write_exemplar_bank(per_paper + reference_papers, field_profile) print(f"\n[extract_style] OK. {len(per_paper)} curated + " - f"{len(reference_papers)} reference papers for field {field!r}.") + f"{len(reference_papers)} reference papers for field {field!r}." + + (f" {skipped} source file(s) SKIPPED (pymupdf unavailable): the " + "profile does not describe them." if skipped else "")) print(f" → {field_profile}/style_dossier.md") print(f" → {field_profile}/{{sentence_stats,transition_inventory,lexicon}}.json") print(f" → {field_profile}/exemplar_paragraphs.jsonl ({n_exemplars} paragraphs)") diff --git a/tools/length_gate.py b/tools/length_gate.py index ae0fb2c..7baf90a 100644 --- a/tools/length_gate.py +++ b/tools/length_gate.py @@ -16,26 +16,19 @@ from __future__ import annotations -import argparse -import json import math -import re import sys from collections import Counter from pathlib import Path +from typing import Any sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first -from typing import Any - -TOOLS_DIR = Path(__file__).resolve().parent -sys.path.insert(0, str(TOOLS_DIR)) import deai_feedback as feedback # noqa: E402 shared finding contract import deai_metrics # noqa: E402 section segmentation import extract_style as es # noqa: E402 LaTeX-to-prose cleaning import tex_assembly # noqa: E402 one include assembler for file and git baselines -FRONT_MATTER = "(front matter)" # Heading commands are stripped before counting: the budget measures body # prose, and a section RENAME must not register as prose growth. The pattern # is the shared one (starred forms, optional short title, one nesting level). @@ -50,17 +43,20 @@ def prose_word_count(tex: str) -> int: def section_word_counts(text: str) -> dict[str, int]: - """Rendered-prose word count per section label, plus a front-matter bucket.""" + """Rendered-prose word count per section label (the preamble is one label). + + Comments are blanked BEFORE the document is segmented: the segmenter reads + heading commands wherever they stand, so a commented-out `% \\section{Old + draft}` opened a phantom section that no `--allow` key could name, while + the removal map (which reads the comment-blanked source) segmented the + same file differently. `section_line_ranges` covers every line, so there + is no front-matter remainder to count separately. + """ + text = es.blank_preserving(text, es.RE_TEX_COMMENT) lines = text.splitlines() counts: Counter[str] = Counter() - covered: set[int] = set() for start, end, label in deai_metrics.section_line_ranges(text): counts[label] += prose_word_count("\n".join(lines[start - 1:end])) - covered.update(range(start, end + 1)) - rest = "\n".join(line for number, line in enumerate(lines, start=1) - if number not in covered) - if rest.strip(): - counts[FRONT_MATTER] += prose_word_count(rest) return dict(counts) @@ -179,12 +175,14 @@ def gate_findings(before_text: str, after_text: str, path: Path, def parse_required_shrink(raw: str | None) -> float | int | None: - """`--require-shrink` as a fraction (float in (0, 1)) or a word count (int > 0). + """`--require-shrink` as a fraction (float in [0, 1)) or a word count (int >= 0). The growth gate is one-sided: it stops a condensation pass from growing the document and says nothing when the pass removed almost nothing. This is the other side, so a "condense" round that shaved four words cannot - close green. + close green. Zero is accepted and means "no cut required": the condense + skill passes the map's `default_target_words` straight through, and a map + that lists nothing to cut must not turn the closing gate into exit 2. """ if raw is None: return None @@ -202,8 +200,8 @@ def parse_required_shrink(raw: str | None) -> float | int | None: f"fraction in (0, 1) or a word count, got {raw!r}") from error if percent: value /= 100.0 - if not math.isfinite(value) or value <= 0: - raise ValueError("--require-shrink must be a finite positive number") + if not math.isfinite(value) or value < 0: + raise ValueError("--require-shrink must be a finite non-negative number") if fraction and value >= 1: raise ValueError("--require-shrink fraction must be below 1 (or 100%)") return value @@ -211,11 +209,14 @@ def parse_required_shrink(raw: str | None) -> float | int | None: def required_shrink_words(shrink: float | int | None, total_before: int) -> int | None: """A fraction of the baseline rounds UP: a 10% cut of five words is one - word, not the zero words that rounding to nearest made of it.""" + word, not the zero words that rounding to nearest made of it. The product + is rounded to six decimals before the ceiling: `7%` is the float + 0.07000000000000001, so `ceil(0.07 * 100)` was 8 and a pass that cut + exactly the seven words asked for failed the gate.""" if shrink is None: return None if isinstance(shrink, float): - return math.ceil(shrink * total_before) + return math.ceil(round(shrink * total_before, 6)) return shrink @@ -252,18 +253,25 @@ def main(argv: list[str] | None = None) -> int: help="record one justification covering every section") parser.add_argument("--require-shrink", default=None, metavar="FRACTION|WORDS", help="exit 1 unless total prose fell by at least this much: " - "a fraction of the baseline in (0, 1), or a word count") - parser.add_argument("--format", choices=("text", "json"), default="text") - parser.add_argument("--output", type=Path) + "a fraction of the baseline in [0, 1), a percentage, " + "or a word count (0 = no cut required)") + cli_common.report_options(parser) args = parser.parse_args(argv) - if not args.after.exists(): + # `is_file`, not `exists`: a directory named like a document read as an + # empty document, which the gate scored as a complete cut (exit 0), and a + # mistyped --before read as a zero-word baseline, which it scored as growth + # in every section (exit 1). Missing evidence is exit 2, never a verdict. + if not args.after.is_file(): print(f"[length_gate] file not found: {args.after}", file=sys.stderr) return 2 if (args.before is None) == (args.git_ref is None): print("[length_gate] provide exactly one of --before or --git-ref", file=sys.stderr) return 2 + if args.before is not None and not args.before.is_file(): + print(f"[length_gate] baseline not found: {args.before}", file=sys.stderr) + return 2 if args.tolerance_words < 0: print("[length_gate] --tolerance-words must be >= 0", file=sys.stderr) return 2 @@ -336,7 +344,7 @@ def main(argv: list[str] | None = None) -> int: args.output.write_text(rendered, encoding="utf-8") else: print(rendered, end="") - except (OSError, ValueError, json.JSONDecodeError, UnicodeDecodeError) as error: + except (OSError, ValueError) as error: print(f"[length_gate] execution failed: {error}", file=sys.stderr) return 2 return gate_exit diff --git a/tools/retrieve_exemplars.py b/tools/retrieve_exemplars.py index a45fb62..5446788 100644 --- a/tools/retrieve_exemplars.py +++ b/tools/retrieve_exemplars.py @@ -167,8 +167,11 @@ def _retrieve_st(records, jsonl_path, profile_dir, model_name, section, topic, def _retrieve_fallback(records, section, topic, k, tiers): """Keyword-overlap fallback when sentence-transformers is unavailable. - Strictly worse than ST retrieval; use only as a last resort.""" - topic_kws = {w.lower() for w in topic.split() if len(w) > 3} + Strictly worse than ST retrieval; use only as a last resort. An empty + topic ranks by the section name, as the `--topic` help promises and the + embedding path does; scored on the empty string it returned the first k + rows of the bank at score 0.""" + topic_kws = {w.lower() for w in (topic or section).split() if len(w) > 3} scored = [] for rec in records: if not _wanted(rec, section, tiers): diff --git a/tools/tex_assembly.py b/tools/tex_assembly.py index cb39048..a6652d5 100644 --- a/tools/tex_assembly.py +++ b/tools/tex_assembly.py @@ -13,8 +13,17 @@ - `read_git_document(path, ref)` reads the same graph at a git ref, so a `--git-ref` baseline whose root only says `\\input{body}` still carries the body (it did not: the gate read the root alone and a shortened child never - changed the count); -- `assemble(text, root, read)` is the one algorithm behind both. + changed the count), and its children are looked up AT THE REF too: resolved + against the working tree, a child deleted since the ref vanished from the + baseline, which understated exactly the cut a condense pass makes when it + drops a whole section file; +- `assemble(text, root, read, resolve=...)` is the one algorithm behind both. + +The root of a document must be readable: an unreadable root raises, because a +gate that read a missing baseline as an empty document reported every section +as growth, and a linter that read a directory named `x.tex` as an empty +document reported it clean. Only a CHILD may fail to resolve and stay in +place as the literal call. The splice is in place: `Before \\input{body} After` keeps `Before` and `After` (the earlier assembler replaced the whole line with the child, losing both), @@ -31,14 +40,16 @@ import tex_macros -# One owner for the comment pattern: `extract_sections` re-exports it. -RE_TEX_COMMENT = re.compile(r"(? list[str]: @@ -54,27 +65,52 @@ def include_targets(text: str) -> list[str]: return names -def resolve_include(root: Path, name: str) -> Path | None: - """The file an \\include/\\input target names, or None. Literal path first, - then a same-stem sibling (arXiv flattens submission directories).""" - literal = root.parent / name - if literal.suffix.lower() != ".tex": - literal = literal.with_suffix(".tex") - if literal.is_file(): - return literal - sibling = root.parent / f"{Path(name).stem}.tex" - return sibling if sibling.is_file() else None +def include_candidates(base: Path, current: Path, name: str) -> list[Path]: + """The files an \\include/\\input target may name, in resolution order. + + LaTeX resolves a target against the directory it was started in -- the + document root's, whatever file the call sits in -- so that comes first, + then the including file's own directory. A name with a suffix is tried + as written (`\\input{fig.tikz}` does not mean `fig.tex`); one without gets + `.tex`. Last, a same-stem sibling of each, because arXiv flattens + submission directories and `\\input{sections/introduction}` routinely + names a file that now sits beside the root. + """ + target = Path(name) + literal = target if target.suffix else target.with_suffix(".tex") + ordered: list[Path] = [] + for directory in (base, current.parent): + for candidate in (directory / literal, directory / f"{target.stem}.tex"): + if candidate not in ordered: + ordered.append(candidate) + return ordered + + +def resolve_include(root: Path, name: str, base: Path | None = None) -> Path | None: + """The file an \\include/\\input target names on disk, or None.""" + for candidate in include_candidates(base or root.parent, root, name): + if candidate.is_file(): + return candidate + return None def assemble(text: str, root: Path, read: Callable[[Path], str | None], - _stack: tuple[Path, ...] = ()) -> str: + _stack: tuple[Path, ...] = (), *, + resolve: Callable[[Path, str, Path], Path | None] | None = None, + base: Path | None = None) -> str: """`text` (the content of `root`) with every live include spliced in place. - `read(path)` returns a child's content or None when it cannot be read; an + `read(path)` returns a child's content or None when it cannot be read; + `resolve(current, name, base)` names the child a call refers to (the disk + rule by default; a git reader passes one that looks at the ref). An unresolvable target (`\\input{aa.cls}`) is left in place rather than - failing. Includes inside a comment are not calls. + failing. Includes inside a comment are not calls. `base` is the document + root's directory and is threaded through the recursion so a nested + `\\input` resolves the way LaTeX resolves it. """ stack = _stack + (root.resolve(),) + base_dir = base if base is not None else root.parent + find = resolve or resolve_include out: list[str] = [] for line in text.splitlines(keepends=True): comment = RE_TEX_UNESCAPED_PERCENT.search(line) @@ -84,13 +120,15 @@ def assemble(text: str, root: Path, read: Callable[[Path], str | None], continue def splice(match: "re.Match[str]") -> str: - child = resolve_include(root, match.group(1)) + child = find(root, match.group(1), base_dir) if child is None: return match.group(0) if child.resolve() in stack: return "" body = read(child) - return match.group(0) if body is None else assemble(body, child, read, stack) + if body is None: + return match.group(0) + return assemble(body, child, read, stack, resolve=find, base=base_dir) out.append(RE_TEX_INCLUDE.sub(splice, code) + tail) return "".join(out) @@ -106,26 +144,26 @@ def _read_file(path: Path) -> str | None: def read_tex_document(path: Path) -> str: """Read a `.tex` root with its \\include / \\input children spliced in. - A target is resolved against the root's directory first and, failing that, - against a sibling with the same stem: arXiv flattens submissions, so - `\\input{sections/introduction}` routinely names a file that now sits - beside the root, and `select_document_roots` matches includes by stem -- - four wgl bundles lost most of their prose (one 9,741 of 9,743 words) - before the reader used the same rule. Numeric macros are expanded once, on - the assembled document (`tex_macros`): only the root holds definition and use. + The root is read with `read_text` and its OSError propagates (a missing + file, a directory, a permission failure): the caller decides what an + unreadable root means, and no caller may read it as an empty document. + Children go through `_read_file` and degrade to "left in place". Numeric + macros are expanded once, on the assembled document (`tex_macros`): only + the root holds definition and use. """ - text = _read_file(path) - if text is None: - return "" + text = path.read_text(encoding="utf-8", errors="replace") return tex_macros.expand_numeric(assemble(text, path, _read_file)) def read_git_document(path: Path, ref: str) -> str: """The same assembled document at git `ref`, every child read from the ref too. - Decoding uses `errors="replace"`: a baseline with stray non-UTF-8 bytes - must degrade to lossy words, not escape as an uncaught traceback. Raises - ValueError when the path is outside a repository or absent at the ref. + Children are RESOLVED at the ref as well (`git cat-file -e`), with the + same candidate order as the disk reader, so a child that the working tree + no longer holds is still part of the baseline. Decoding uses + `errors="replace"`: a baseline with stray non-UTF-8 bytes must degrade to + lossy words, not escape as an uncaught traceback. Raises ValueError when + the path is outside a repository or absent at the ref. """ directory = path.resolve().parent top = _git(directory, "rev-parse", "--show-toplevel") @@ -133,18 +171,32 @@ def read_git_document(path: Path, ref: str) -> str: raise ValueError(f"not inside a git repository: {top.stderr.strip()}") repo = Path(top.stdout.strip()) - def read(target: Path) -> str | None: + def relative(target: Path) -> str | None: try: - relative = target.resolve().relative_to(repo).as_posix() + return target.resolve().relative_to(repo).as_posix() except ValueError: return None - shown = _git(directory, "show", f"{ref}:{relative}") + + def read(target: Path) -> str | None: + name = relative(target) + if name is None: + return None + shown = _git(directory, "show", f"{ref}:{name}") return shown.stdout if shown.returncode == 0 else None + def resolve(current: Path, name: str, base: Path) -> Path | None: + for candidate in include_candidates(base, current, name): + inside = relative(candidate) + if inside is None: + continue + if _git(directory, "cat-file", "-e", f"{ref}:{inside}").returncode == 0: + return candidate + return None + root_text = read(path) if root_text is None: raise ValueError(f"git show {ref}:{path.name} failed: not present at {ref!r}") - return tex_macros.expand_numeric(assemble(root_text, path, read)) + return tex_macros.expand_numeric(assemble(root_text, path, read, resolve=resolve)) def _git(directory: Path, *arguments: str) -> "subprocess.CompletedProcess[str]": diff --git a/tools/tex_macros.py b/tools/tex_macros.py index dc5f9ad..feba823 100644 --- a/tools/tex_macros.py +++ b/tools/tex_macros.py @@ -21,20 +21,28 @@ place where a preamble definition and a body use are both in scope. It is deliberately conservative: a macro is expanded only when its body is a bare numeric literal with no letters and no markup, so `\\newcommand{\\Msun}{M_\\odot}` -and every macro taking an argument are left exactly as they were. +and every macro taking an argument are left exactly as they were. A definition +inside a comment is not a definition: harvested, a commented-out +`% \\newcommand{\\Nf}{63}` beat the live value that followed it. """ from __future__ import annotations import re +# One owner for the comment pattern: `tex_assembly` and `extract_sections` +# re-export it, and it lives here because this is the lowest module that needs +# it (a definition inside a comment must not be harvested). +RE_TEX_COMMENT = re.compile(r"(? str: """Replace uses of numeric-literal macros with their numbers. The definition itself is dropped rather than left in place. Keeping it - would double-count: `RE_TEX_SIMPLE_CMD` reduces `\\newcommand{\\Nf}{63}` to - its argument `63`, so an unexpanded manuscript already contributes one + would double-count: the projections reduce `\\newcommand{\\Nf}{63}` to the + stray token `\\Nf63`, so an unexpanded manuscript already contributes one stray numeral per definition, attributed to the preamble. + Definitions are read from a copy with comments blanked (same offsets), so + a commented-out definition is neither harvested nor dropped; a definition + that takes arguments is left exactly as written. + A use is matched only when the macro name is not a prefix of a longer name (`\\Nf` must not fire inside `\\Nfields`), and an immediately following empty brace pair -- the `{}` LaTeX authors write to protect the following space -- is consumed with it. """ macros: dict[str, str] = {} - - def _harvest(match: "re.Match[str]") -> str: - body = match.group(2).strip() - if RE_BARE_NUMBER.match(body): - macros[match.group(1)] = body - return " " - return match.group(0) - - text = RE_DEFINITION.sub(_harvest, text) + live = RE_TEX_COMMENT.sub(lambda m: " " * len(m.group(0)), text) + pieces: list[str] = [] + cursor = 0 + for match in RE_DEFINITION.finditer(live): + body = match.group(3).strip() + if match.group(2) is not None or not RE_BARE_NUMBER.match(body): + continue + macros[match.group(1)] = body + pieces.append(text[cursor:match.start()]) + pieces.append(" ") + cursor = match.end() if not macros: return text + pieces.append(text[cursor:]) + text = "".join(pieces) # Longest name first so a shorter name cannot claim a prefix of a longer # one before the negative lookahead is reached. names = "|".join(sorted((re.escape(n) for n in macros), diff --git a/tools/validate_plugin.py b/tools/validate_plugin.py index e2e374a..ebe7147 100644 --- a/tools/validate_plugin.py +++ b/tools/validate_plugin.py @@ -32,12 +32,9 @@ STANDARD_DOC = "docs/SCIPAPER_STANDARD.md" DOCS_INDEX = "docs/README.md" SUBSYSTEM_DOC = "docs/architecture/DEAI_SUBSYSTEM.md" -# The evidence record is a hub plus parts under docs/architecture/evaluation/. -# EVALUATION_DOC is the hub: it carries the contract, the axis-status table, the -# section map, and the release boundary, and is what other documents cite. -# EVALUATION_PARTS_DIR holds the section bodies. Any check that validates a -# recorded measurement must scan the WHOLE record, not just the hub -- a stale -# figure hiding in a part file is exactly the drift these checks exist to catch. +# The evidence record is a hub (EVALUATION_DOC: contract, axis-status table, +# section map, release boundary) plus the section bodies under +# EVALUATION_PARTS_DIR; a check of a recorded measurement scans the WHOLE record. EVALUATION_DOC = "docs/architecture/EVALUATION.md" EVALUATION_PARTS_DIR = "docs/architecture/evaluation" DESIGN_NOTES = ( @@ -52,17 +49,7 @@ ) NORMATIVE_SKILLS = { - "paper", - "physics", - "mainline", - "logic", - "de-ai", - "condense", - "paper-review", - "figure-review", - "final-review", - "calibrate", - "proposal-polish", + "paper", "physics", "mainline", "logic", "de-ai", "condense", "paper-review", "figure-review", "final-review", "calibrate", "proposal-polish", } CORE_IMPORTS = { "ai_ism_lint", "condense_map", "deai_collocation", "deai_docstructure", @@ -148,11 +135,8 @@ def check_manifests() -> str: plugin_v = str(plugin["version"]) require(market.get("metadata", {}).get("version") == plugin_v, "marketplace metadata.version does not match plugin.json") - inner = next( - (item for item in market.get("plugins", []) - if item.get("name") == plugin["name"]), - None, - ) + inner = next((item for item in market.get("plugins", []) + if item.get("name") == plugin["name"]), None) require(inner is not None, f"marketplace plugins[] has no entry named {plugin['name']!r}") require(inner.get("version") == plugin_v, @@ -267,6 +251,26 @@ def _broken_page_anchors(path: Path) -> list[str]: RE_SECTION_REF = re.compile(r"\[\s*§\s*(\d+)(?:\.\w+)?\s*\]\(([^)\s#]+\.md)") +RE_FILE_LINK = re.compile(r"\]\(([^)\s#]+)(?:#[^)]*)?\)") + + +def _broken_file_links(path: Path) -> list[str]: + """Relative links whose target file does not exist. + + Two comments in this file deferred to "the file-link check"; there was + none, so `[§9](missing.md)` and a moved page passed every check. + Fenced and inline code are skipped, and only relative targets are resolved. + """ + text = re.sub(r"```.*?```|`[^`\n]*`", "", read_text(path), flags=re.DOTALL) + broken = [] + for match in RE_FILE_LINK.finditer(text): + target = match.group(1) + if re.match(r"^[a-z][a-z0-9+.-]*:", target): + continue + if not (path.parent / target).exists(): + line = text[: match.start()].count("\n") + 1 + broken.append(f"{path.relative_to(REPO).as_posix()}:{line} {target}") + return broken def _broken_section_references(path: Path) -> list[str]: @@ -283,7 +287,7 @@ def _broken_section_references(path: Path) -> list[str]: number, target = match.group(1), match.group(2) resolved = (path.parent / target).resolve() if not resolved.exists(): - continue # the file-link check owns a missing target + continue # `_broken_file_links` reports a missing target body = read_text(resolved) if (re.search(rf"(?m)^#{{1,6}}\s+{number}[.\s]", body) or re.search(rf"(?m)^\|\s*\*\*{number}\*\*\s*\|", body)): @@ -383,11 +387,14 @@ def check_documentation_boundaries() -> str: f"{note} lives under design-notes/ but its header does not " "declare it a design note; a frozen note must not read as " "current status") - # File links were checked; in-page #fragments were not, so the README - # restructure left five cross-references pointing at headings that had been - # renamed or renumbered — including "demo 3" linking to demo 4. + # In-page #fragments were once unchecked, so the README restructure left + # five cross-references pointing at headings that had been renamed or + # renumbered — including "demo 3" linking to demo 4. pages = [REPO / "README.md", REPO / "README.zh-CN.md", *sorted(DOCS.rglob("*.md"))] + dead = [item for page in pages for item in _broken_file_links(page)] + require(not dead, "relative links point at files that do not exist: " + + "; ".join(dead)) broken = [item for page in pages for item in _broken_page_anchors(page)] require(not broken, "in-page anchors point at headings that do not exist: " @@ -500,7 +507,7 @@ def parse_declared_count(text: str, heading: str) -> int: (re.compile(r"(\d+) skills\b"), ("skills",)), (re.compile(r"(\d+) product tools\b"), ("tools",)), (re.compile(r"(\d+) files, (\d+) tests\b"), ("files", "tests")), - (re.compile(r"(\d+) contract checks\b"), ("checks",)), + (re.compile(r"(\d+) (?:contract )?checks\b"), ("checks",)), (re.compile(r"the (\d+)-test suite"), ("tests",)), (re.compile(r"(\d+) 个 skill\b"), ("skills",)), (re.compile(r"(\d+) 个产品工具"), ("tools",)), @@ -550,7 +557,9 @@ def check_registry_counts() -> str: facts = {"skills": len(documents), "tools": len(products), "tests": tests, "files": files, "checks": len(CHECKS)} wrong, seen = [], 0 - for name in ("README.md", "README.zh-CN.md"): + # tools/README.md too: its validator row said "10 checks" through two + # releases that ran eleven, because only the READMEs were read. + for name in ("README.md", "README.zh-CN.md", "tools/README.md"): text = read_text(REPO / name) for pattern, keys in SHAPE_CLAIMS: for groups in pattern.findall(text): @@ -561,7 +570,7 @@ def check_registry_counts() -> str: wrong.append(f"{name} says {value} {key} (repository has {facts[key]})") require(not wrong, "; ".join(sorted(set(wrong)))) return (f"README and manifest registries agree ({len(documents)} skills, " - f"{len(products)} tools; {seen} shape claim(s) verified in both READMEs)") + f"{len(products)} tools; {seen} shape claim(s) verified across the READMEs)") def check_tools_syntax() -> str: @@ -589,14 +598,8 @@ def check_runtime_contract() -> str: require(feedback.SCHEMA_VERSION == SCHEMA, f"deai_feedback schema is {feedback.SCHEMA_VERSION!r}, expected {SCHEMA!r}") finding = feedback.make_finding( - kind="advisory", - layer="L2", - rule="validator-smoke", - scope="sentence", - message="validator smoke finding", - action="no action", - detector="validate_plugin", - ) + kind="advisory", layer="L2", rule="validator-smoke", scope="sentence", + message="validator smoke finding", action="no action", detector="validate_plugin") require(REQUIRED_FINDING_FIELDS <= finding.keys(), "feedback finding missing fields: " f"{sorted(REQUIRED_FINDING_FIELDS - finding.keys())}") @@ -611,15 +614,9 @@ def check_runtime_contract() -> str: for name in sorted(CORE_CLIS): result = subprocess.run( - [sys.executable, str(TOOLS / f"{name}.py"), "--help"], - cwd=REPO, - capture_output=True, - text=True, - encoding="utf-8", - errors="replace", - timeout=30, - check=False, - ) + [sys.executable, str(TOOLS / f"{name}.py"), "--help"], cwd=REPO, + capture_output=True, text=True, encoding="utf-8", errors="replace", + timeout=30, check=False) require(result.returncode == 0, f"tools/{name}.py --help failed ({result.returncode}): {result.stderr.strip()}") return f"core imports, schema, and CLI entry points valid ({len(CORE_IMPORTS)} modules)" From 46cf4cea299a68578525f75b851b34e7cd571486 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 05:59:02 +0000 Subject: [PATCH 04/27] verify_references: audit F1/F2/F5/F6/F8/F10/F11/F13/F21/F23/F24/F32 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twelve defects from docs/audits/full-review-2026-09-27.md §3.4, each with a regression test, no other file touched. - F1: a `\cite` inside a `%` comment raised reference-missing-entry (exit 1). cite_lines strips comments with tex_assembly.RE_TEX_COMMENT first. - F2/F32: http.client.HTTPException (IncompleteRead, BadStatusLine) is not an OSError and escaped fetch as a traceback, i.e. exit 1 read as a blocker verdict. fetch now catches (HTTPException, OSError); the redundant URLError/TimeoutError members are gone, and main ends in one deliberately broad `except Exception -> 2` as ai_ism_lint does, printing the type. - F5: _parse_fields is a piece grammar: `#` concatenation, quoted values scanned by brace depth with `\"` allowed, and text the grammar cannot place raises ValueError (named with entry key and line) instead of silently dropping every later field, DOI included. - F6: a registry miss is never cached and a legacy `null` in the cache file is dropped on load, so a late-indexed DOI is re-queried each round; metadata-mismatch findings carry `observed.cached`. - F8: _first() reads list-valued CrossRef/DataCite fields defensively (`date-parts: [[]]` was an IndexError); a 200 body that is not the expected JSON/Atom shape is RegistryUnavailable, so the entry is unmeasured and the axis degraded rather than the run crashing. - F10: first-author agreement is equality once normalized or a shared name part of four or more letters; `Li`/`Lin`, `Ma`/`Mandelbaum`, `de Jong`/ `de Vries` are now reported while `van Waerbeke`/`Waerbeke` still agree. - F11: RE_CITE recognises \nocite and biblatex \parencite, \textcite, \autocite, \smartcite, \footcite, \supercite, \fullcite, capitalised forms, every key group of the multicite \cites{a}{b}, and \nocite{*} as citing every entry; natbib's \citetext{prose} no longer yields a key. - F13: @string definitions are collected (in order, with `#`) and bare field values expanded before comparison. - F21: a missing-entry finding reports the citing command's line (in the assembled document) instead of 1. - F23: author lists split on ` and ` case-insensitively. - F24: a DOI carried only in the `url` field (doi.org / dx.doi.org, URL- decoded) is an identifier. - F22 (retry/mailto) is left as a named known gap in fetch's docstring. Exit contract unchanged (0 no blocker / 1 blocker / 2 invalid input or any execution failure). Output changes a caller can see: `observed.cached` on metadata-mismatch findings, the lookup-failed message now reads "the registry gave no usable answer", and `--cache` files no longer hold nulls. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit d93176db349df170405cdbdfa5690de5ba9692b5) --- tests/test_verify_references.py | 146 ++++++++++- tools/verify_references.py | 445 ++++++++++++++++++++++++-------- 2 files changed, 484 insertions(+), 107 deletions(-) diff --git a/tests/test_verify_references.py b/tests/test_verify_references.py index 42e6a2c..f9173bf 100644 --- a/tests/test_verify_references.py +++ b/tests/test_verify_references.py @@ -4,14 +4,19 @@ touches the network; the live behaviour it stands in for was taken on the two manuscripts' bibliographies on 2026-09-05 (CrossRef title/subtitle split, the print/online year seam, `&` against `\\&`, DataCite fallback -for a Zenodo DOI). +for a Zenodo DOI). The one test that reaches below `fetch` replaces +`urllib.request.urlopen` for the same reason. """ from __future__ import annotations +import contextlib +import http.client +import io import json import tempfile import unittest +import urllib.request from pathlib import Path from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -110,6 +115,24 @@ def test_unbalanced_entry_is_a_value_error(self): with self.assertRaises(ValueError): vr.parse_bib("@article{broken,\n title = {never closed\n") + def test_string_macros_hash_concatenation_and_escaped_quotes(self): + bib = ('@string{apj = "The Astrophysical Journal"}\n' + '@STRING(apjl = apj # " Letters")\n' + r'@article{k, journal = apjl, title = "M{\"u}ller and the \"quoted\" word",' + '\n doi = "10.1093/" # {mnras/1}, year = 2001, Bdsk-Url-1 = {x},}') + fields = vr.parse_bib(bib)[0]["fields"] + self.assertEqual(fields["journal"], "The Astrophysical Journal Letters") + self.assertEqual(fields["title"], r'M{\"u}ller and the \"quoted\" word') + self.assertEqual((fields["doi"], fields["year"], fields["bdsk-url-1"]), + ("10.1093/mnras/1", "2001", "x")) + + def test_text_the_grammar_cannot_place_is_a_value_error_not_a_dropped_field(self): + for text in ("@article{k, title = {x} junk, doi = {10.1/x}}", + "@article{k, title = }", + '@article{k, title = "never closed}'): + with self.subTest(text=text), self.assertRaisesRegex(ValueError, "entry 'k'"): + vr.parse_bib(text) + class NormalizationTests(unittest.TestCase): """Table-driven: each row is (function, input, expected).""" @@ -119,6 +142,8 @@ class NormalizationTests(unittest.TestCase): (vr.normalize, "Émile Düsseldorf & co", "emiledusseldorfco"), (vr.first_author_family, "van Waerbeke, Ludovic and Mellier, Y.", "vanwaerbeke"), (vr.first_author_family, "Peter Schneider and Y. Mellier", "schneider"), + (vr.first_author_family, "Fan, Zuhui AND Shan, Huanyuan", "fan"), + (vr.first_author_family, "Peter Schneider\nand Y. Mellier", "schneider"), (vr.first_author_family, "", ""), (vr.first_page, "1408--1420", "1408"), (vr.first_page, "1408–1420", "1408"), @@ -139,10 +164,30 @@ def test_identifier_prefers_doi_and_strips_the_resolver_url(self): "fan2010": "arxiv:1006.5121", "ho2020": None, "code2025": "doi:10.5281/zenodo.1234567"}) + def test_a_doi_carried_only_in_the_url_field_is_an_identifier(self): + rows = (({"url": "https://doi.org/10.1093/mnras%2F283.3.837"}, "doi:10.1093/mnras/283.3.837"), + ({"url": "http://dx.doi.org/10.5281/zenodo.1"}, "doi:10.5281/zenodo.1"), + ({"doi": "10.1/a", "url": "https://doi.org/10.1/b"}, "doi:10.1/a"), + ({"url": "https://arxiv.org/abs/1006.5121"}, "arxiv:1006.5121"), + ({"url": "https://example.org/10.1093/x"}, None)) + for fields, expected in rows: + with self.subTest(fields=fields): + self.assertEqual(vr.entry_identifier(fields), expected) + def test_cite_keys_with_optional_arguments_and_lists(self): text = r"\citep[e.g.,][]{a, b}\citet{c}\Citealt*{d} \cite {e}" self.assertEqual(vr.cite_keys(text), {"a", "b", "c", "d", "e"}) + def test_cite_keys_of_natbib_and_biblatex_commands(self): + text = (r"\parencite{a}\Textcite[p.~3]{b}\autocite*{c}\smartcite{d}\footcite{e}" + r"\nocite{f}\cites{g,h}[pre][post]{i}\supercite{j}\fullcite{k}" + r"\citetext{priv.\ comm.}\citealias{l} {\bf not a key}\citeauthor{m}") + self.assertEqual(vr.cite_keys(text), set("abcdefghijklm")) + + def test_cite_lines_skip_comments_and_report_the_citing_line(self): + text = "\\cite{a}\n% \\cite{ghost}\nsee \\citep{b} \\% \\cite{c}\n\\nocite{*}" + self.assertEqual(vr.cite_lines(text), {"a": 1, "b": 3, "c": 3, "*": 4}) + class CompareTests(unittest.TestCase): def setUp(self): @@ -168,6 +213,27 @@ def test_journal_short_form_matches_and_volume_page_are_ordinary(self): def test_missing_fields_on_either_side_are_not_compared(self): self.assertEqual(vr.compare({"author": "Schneider, P."}, {"first_author": "", "title": ""}), []) + def test_short_family_names_no_longer_pass_as_substrings(self): + rows = (("Li, X.", "Lin", False), ("Ma, C.", "Mandelbaum", False), + ("He, S.", "Heymans", False), ("de Jong, R.", "de Vries", False), + ("van Waerbeke, L.", "Waerbeke", True), ("Smith-Jones, A.", "Smith", True), + ("Fan, Z.", "Fan", True)) + for entry, record, agree in rows: + with self.subTest(entry=entry, record=record): + issues = vr.compare({"author": entry}, {"first_author": record}) + self.assertEqual([i["field"] for i in issues], [] if agree else ["first_author"]) + + def test_record_shapes_outside_the_contract_do_not_raise(self): + for message in ({"issued": {"date-parts": [[]]}}, {"issued": {"date-parts": [[None]]}}, + {"issued": {}}, {"issued": None, "title": [], "author": [], "subtitle": None}): + with self.subTest(message=message): + record = vr._crossref_record(message) + self.assertEqual((record["year"], record["title"], record["first_author"]), + (None, "", "")) + datacite = vr._datacite_record({"creators": [], "titles": [], "container": None}) + self.assertEqual((datacite["first_author"], datacite["title"], datacite["journal"]), + ("", "", "")) + class VerifyTests(unittest.TestCase): def run_verify(self, fetch, tex: str | None = None, cache: Path | None = None): @@ -232,6 +298,45 @@ def test_registry_outage_degrades_the_axis_instead_of_reading_clean(self): failed = next(f for f in report["findings"] if f["rule"] == "reference-lookup-failed") self.assertEqual(failed["measurement_status"], "unmeasured") + def test_http_client_errors_are_a_registry_outage_not_a_traceback(self): + def urlopen(request, timeout=30): + raise http.client.IncompleteRead(b"") + real = urllib.request.urlopen + urllib.request.urlopen = urlopen + try: + with self.assertRaises(vr.RegistryUnavailable): + vr.fetch(vr.CROSSREF + "10.1093/x") + finally: + urllib.request.urlopen = real + + def test_an_unreadable_registry_answer_leaves_the_entry_unmeasured(self): + fetch = self.registries(**{vr.CROSSREF + "10.1093": (200, b'{"no": "message"}'), + vr.ARXIV: (200, b"not xml")}) + code, report = self.run_verify(fetch) + self.assertEqual(code, 0) + states = report["reference_states"] + self.assertEqual((states["schneider1996"], states["fan2010"]), ("unmeasured", "unmeasured")) + self.assertEqual(report["axes"][0]["status"], "degraded") + + def test_any_other_failure_exits_two_rather_than_reading_as_a_verdict(self): + def fetch(url, timeout=30): + raise RuntimeError("boom") + with tempfile.TemporaryDirectory() as raw: + bib, cache = Path(raw) / "refs.bib", Path(raw) / "cache.json" + bib.write_text(BIB, encoding="utf-8") + cache.write_text("[]", encoding="utf-8") + real, vr.fetch = vr.fetch, fetch + stderr = io.StringIO() + try: + with contextlib.redirect_stderr(stderr): + crashed = vr.main([str(bib), "--pause", "0"]) + bad_cache = vr.main([str(bib), "--pause", "0", "--cache", str(cache)]) + finally: + vr.fetch = real + self.assertEqual((crashed, bad_cache), (2, 2)) + self.assertIn("execution failed: RuntimeError: boom", stderr.getvalue()) + self.assertIn("execution failed: ValueError: cache", stderr.getvalue()) + def test_tex_cross_check_finds_missing_entries_and_uncited_ones(self): tex = r"\cite{schneider1996,ghost2001} \citep[see][]{fan2010}" code, report = self.run_verify(self.registries(), tex=tex) @@ -243,6 +348,24 @@ def test_tex_cross_check_finds_missing_entries_and_uncited_ones(self): if f["rule"] == "reference-uncited") self.assertEqual(uncited, ["code2025", "ho2020"]) + def test_a_cite_inside_a_comment_is_text_and_a_missing_key_names_its_line(self): + tex = ("intro\n\\cite{schneider1996,fan2010}\n% \\cite{ghost}\n" + "\\citep{code2025} and \\cite{phantom}\n") + code, report = self.run_verify(self.registries(), tex=tex) + self.assertEqual(code, 1) + missing = [(f["observed"]["key"], f["location"]["start_line"]) + for f in report["findings"] if f["rule"] == "reference-missing-entry"] + self.assertEqual(missing, [("phantom", 4)]) + uncited = [f["observed"]["key"] for f in report["findings"] + if f["rule"] == "reference-uncited"] + self.assertEqual(uncited, ["ho2020"]) + + def test_nocite_star_cites_every_entry(self): + code, report = self.run_verify(self.registries(), tex=r"\nocite{*}") + rules = {f["rule"] for f in report["findings"]} + self.assertEqual(code, 0) + self.assertFalse(rules & {"reference-uncited", "reference-missing-entry"}) + def test_cache_is_written_and_then_spares_the_registries(self): with tempfile.TemporaryDirectory() as raw: cache = Path(raw) / "cache.json" @@ -256,6 +379,27 @@ def test_cache_is_written_and_then_spares_the_registries(self): self.assertEqual((second.calls, code), ([], 0)) self.assertEqual(report["reference_states"]["code2025"], "verified") + def test_a_registry_miss_is_not_cached_and_a_cached_record_says_so(self): + zenodo = "10.5281/zenodo.1234567" + with tempfile.TemporaryDirectory() as raw: + cache = Path(raw) / "cache.json" + # A `null` an earlier version wrote for a miss, and a non-record. + cache.write_text(json.dumps({"doi:" + zenodo: None, "bogus": 1}), encoding="utf-8") + first = self.registries(**{vr.DATACITE + "10.5281": (404, b"")}) + code, report = self.run_verify(first, cache=cache) + self.assertEqual(code, 1) + self.assertIn(vr.DATACITE + zenodo, first.calls) + saved = json.loads(cache.read_text(encoding="utf-8")) + self.assertEqual(sorted(saved), ["arxiv:1006.5121", "doi:10.1093/mnras/283.3.837"]) + year = next(f for f in report["findings"] if f["rule"] == "reference-metadata-mismatch:year") + self.assertFalse(year["observed"]["cached"]) + second = self.registries() + code, report = self.run_verify(second, cache=cache) + self.assertEqual((code, report["reference_states"]["code2025"]), (0, "verified")) + self.assertEqual(second.calls, [vr.CROSSREF + zenodo, vr.DATACITE + zenodo]) + year = next(f for f in report["findings"] if f["rule"] == "reference-metadata-mismatch:year") + self.assertTrue(year["observed"]["cached"]) + def test_missing_files_are_configuration_failures(self): self.assertEqual(vr.main(["no-such.bib"]), 2) with tempfile.TemporaryDirectory() as raw: diff --git a/tools/verify_references.py b/tools/verify_references.py index 90737ba..00c3d2b 100644 --- a/tools/verify_references.py +++ b/tools/verify_references.py @@ -1,23 +1,30 @@ """Verify a BibTeX bibliography against the records its identifiers resolve to. -Every entry that carries a DOI is resolved through CrossRef (DataCite as the -fallback for the DOI prefixes CrossRef does not register) and its author, -year, title, journal, volume and first page are compared with the fetched -record; an entry with only an arXiv identifier is resolved through the arXiv -API. The comparison is the mechanical half of paper-review dimension F -(citation existence): a DOI that resolves nowhere is an `integrity_blocker` -and a first author, year or title that disagrees with the record is a strong -advisory, both under `sci-paper.feedback.v1`. Journal, volume and page -disagreements are ordinary advisories, since abbreviations and article -numbers vary by house style. An entry the network could not answer for is -reported as unmeasured, never as clean, and the axis is `degraded`. - -With `--tex`, the assembled document's `\\cite` keys are cross-checked: a key +Every entry that carries a DOI (in its `doi` field or as a doi.org `url`) is +resolved through CrossRef (DataCite as the fallback for the DOI prefixes +CrossRef does not register) and its author, year, title, journal, volume and +first page are compared with the fetched record; an entry with only an arXiv +identifier is resolved through the arXiv API. The comparison is the +mechanical half of paper-review dimension F (citation existence): a DOI that +resolves nowhere is an `integrity_blocker` and a first author, year or title +that disagrees with the record is a strong advisory, both under +`sci-paper.feedback.v1`. Journal, volume and page disagreements are ordinary +advisories, since abbreviations and article numbers vary by house style. An +entry the network could not answer for, or answered in a shape the record +readers do not expect, is reported as unmeasured, never as clean, and the +axis is `degraded`. + +With `--tex`, the assembled document's citation keys are cross-checked: a key with no bibliography entry is a blocker, an entry no sentence cites is an -ordinary advisory. Exit status follows the linter's narrow contract: 0 means -no blocker, 1 means a blocker is present, 2 means invalid input or execution -failure. `--cache` keeps fetched records in a JSON file so a repeated review -round does not re-query the registries. +ordinary advisory. The citing commands of LaTeX, natbib and biblatex are +recognised (`\\cite`, `\\citep`, `\\parencite`, `\\cites{a}{b}`, ...), a +`\\nocite{*}` cites every entry, and a citation inside a `%` comment is text. +Exit status follows the linter's narrow contract: 0 means no blocker, 1 means +a blocker is present, 2 means invalid input or execution failure. `--cache` +keeps fetched records in a JSON file so a repeated review round does not +re-query the registries; a registry miss is never cached, so an identifier +the registries are late to index is asked again next round, and a finding +built from a cached record says so (`observed.cached`). """ from __future__ import annotations @@ -25,6 +32,7 @@ import argparse import difflib import html +import http.client import json import re import sys @@ -40,7 +48,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first import deai_feedback as feedback # noqa: E402 shared finding contract -import tex_assembly # noqa: E402 the assembled document for --tex +import tex_assembly # noqa: E402 the assembled document and comment pattern for --tex USER_AGENT = "sci-paper-verify-references/1.0" CROSSREF = "https://api.crossref.org/works/" @@ -55,9 +63,29 @@ # Anchored at line start: an `@article{` inside a `%` comment line is text. RE_ENTRY_HEAD = re.compile(r"^[ \t]*@(\w+)\s*([{(])\s*([^,\s]+)\s*,", re.IGNORECASE | re.MULTILINE) -RE_FIELD_HEAD = re.compile(r"\s*,?\s*(\w+)\s*=\s*") -RE_BARE_VALUE = re.compile(r"[^,}]*") -RE_CITE = re.compile(r"\\[cC]ite[a-zA-Z*]*(?:\[[^\]]*\]){0,2}\s*\{([^}]+)\}") +RE_STRING_HEAD = re.compile(r"^[ \t]*@string\s*([{(])", re.IGNORECASE | re.MULTILINE) +# A field name is any run BibTeX allows (BibDesk writes `Bdsk-Url-1`), a bare +# value a number or a `@string` name; whitespace and commas between fields +# are skipped, `#` joins the pieces of one value, and a value must be +# followed by a comma or the end of the entry. +RE_FIELD_SKIP = re.compile(r"[\s,]*") +RE_FIELD_HEAD = re.compile(r"""([^\s"#%'(),={}]+)\s*=\s*""") +RE_BARE_VALUE = re.compile(r"""[^\s,#{}"]+""") +RE_CONCAT = re.compile(r"\s*#\s*") +RE_FIELD_END = re.compile(r"\s*(?=,|\Z)") +# The citing commands of LaTeX, natbib and biblatex: an optional biblatex +# prefix, `cite`, a suffix (`p`, `t`, `alt`, `author`; `s` marks the +# multicite forms), an optional star, up to two [pre][post] notes, then the +# key group. Case-insensitive for the sentence-initial `\Citep`, `\Textcite`. +RE_CITE = re.compile( + r"\\(?Pno|paren|text|auto|smart|foot|super|full|footfull)?" + r"cite(?P[a-z]*)\*?(?:\s*\[[^\]]*\]){0,2}\s*\{(?P[^}]*)\}", + re.IGNORECASE) +# A multicite command (`\cites{a}{b}`, `\parencites[p]{a}[q]{b}`) carries +# one key group per citation; the tail is consumed group by group. +RE_CITE_GROUP = re.compile(r"\s*(?:\[[^\]]*\]\s*){0,2}\{([^}]*)\}") +MULTICITE_SUFFIXES = ("s", "texts") +RE_AUTHOR_SEP = re.compile(r"\s+and\s+", re.IGNORECASE) RE_ACCENT = re.compile(r"\\[`'^\"~=.uvHtcdbk]\s*\{?\\?([A-Za-z])\}?") RE_LATEX_CMD = re.compile(r"\\[A-Za-z]+") RE_ARXIV_ID = re.compile(r"(\d{4}\.\d{4,5}|[a-z\-]+(?:\.[A-Z]{2})?/\d{7})(?:v\d+)?") @@ -83,29 +111,92 @@ def _group_end(text: str, start: int, close: str) -> int: raise ValueError(f"unbalanced entry from offset {start}") -def _parse_fields(body: str) -> dict[str, str]: +def _quoted_end(body: str, start: int) -> int: + """Index just past the `"` closing the quoted value opening at `start`. + + Braces nest inside a quoted value, and a `"` inside them or escaped as + `\\"` does not close it: `"M{\\"u}ller"` and `"a \\"quoted\\" word"` are + one value each. Stopping at the first `"` cut such a value short and + silently dropped every field after it, DOI included. + """ + depth = 0 + for index in range(start + 1, len(body)): + char = body[index] + if char == "{": + depth += 1 + elif char == "}": + depth -= 1 + elif char == '"' and depth == 0 and body[index - 1] != "\\": + return index + 1 + raise ValueError(f"unterminated quoted value from offset {start}") + + +def _value_piece(body: str, position: int, strings: dict[str, str]) -> tuple[str, int]: + """(text, end) of one value piece: a braced or quoted group, a number, + or a macro name expanded from the `@string` definitions read so far (an + undefined name stays as written, so the comparison shows it).""" + if position < len(body) and body[position] == "{": + end = _group_end(body, position, "}") + return body[position + 1:end - 1], end + if position < len(body) and body[position] == '"': + end = _quoted_end(body, position) + return body[position + 1:end - 1], end + bare = RE_BARE_VALUE.match(body, position) + if not bare: + raise ValueError(f"missing field value at offset {position}") + return strings.get(bare.group().lower(), bare.group()), bare.end() + + +def _parse_fields(body: str, strings: dict[str, str] | None = None) -> dict[str, str]: + """`name = value` pairs of an entry body, whitespace collapsed. + + A value is one or more pieces joined with `#`; each piece is a braced or + quoted group, a number, or a `@string` name. Text the grammar cannot + place raises ValueError: BibTeX itself skips the rest of such an entry, + and dropping the fields silently once turned a DOI-bearing entry into a + "no identifier" advisory. + """ + strings = strings if strings is not None else {} fields: dict[str, str] = {} - position = 0 - while True: + position = RE_FIELD_SKIP.match(body).end() + while position < len(body): head = RE_FIELD_HEAD.match(body, position) - if not head or head.end() >= len(body): - return fields + if not head: + raise ValueError(f"unreadable field at offset {position}: " + f"{body[position:position + 40].strip()!r}") name, position = head.group(1).lower(), head.end() - if body[position] == "{": - end = _group_end(body, position, "}") - value = body[position + 1:end - 1] - elif body[position] == '"': - end = body.index('"', position + 1) + 1 - value = body[position + 1:end - 1] - else: - end = RE_BARE_VALUE.match(body, position).end() - value = body[position:end] - fields[name] = " ".join(value.split()) - position = end + pieces = [] + while True: + piece, position = _value_piece(body, position, strings) + pieces.append(piece) + concat = RE_CONCAT.match(body, position) + if not concat: + break + position = concat.end() + fields[name] = " ".join("".join(pieces).split()) + end = RE_FIELD_END.match(body, position) + if not end: + raise ValueError(f"expected ',' or the end of the entry at offset {position}: " + f"{body[position:position + 40].strip()!r}") + position = RE_FIELD_SKIP.match(body, end.end()).end() + return fields + + +def parse_strings(text: str) -> dict[str, str]: + """`@string{name = value}` definitions, names lowercased, in file order; + a definition may use the ones before it (`apjl = apj # " Letters"`).""" + strings: dict[str, str] = {} + for head in RE_STRING_HEAD.finditer(text): + close = "}" if head.group(1) == "{" else ")" + end = _group_end(text, head.start(1), close) + strings.update(_parse_fields(text[head.end():end - 1], strings)) + return strings def parse_bib(text: str) -> list[dict[str, Any]]: - """Every `@type{key, field = value, ...}` entry; comments and @string skipped.""" + """Every `@type{key, field = value, ...}` entry, `@string` names expanded + in its values; `@comment` and `@preamble` blocks are skipped.""" + strings = parse_strings(text) entries = [] for head in RE_ENTRY_HEAD.finditer(text): kind = head.group(1).lower() @@ -113,9 +204,12 @@ def parse_bib(text: str) -> list[dict[str, Any]]: continue close = "}" if head.group(2) == "{" else ")" end = _group_end(text, head.start(2), close) - entries.append({"key": head.group(3), "type": kind, - "line": text[:head.start()].count("\n") + 1, - "fields": _parse_fields(text[head.end():end - 1])}) + key, line = head.group(3), text[:head.start()].count("\n") + 1 + try: + fields = _parse_fields(text[head.end():end - 1], strings) + except ValueError as error: + raise ValueError(f"entry {key!r} (line {line}): {error}") from error + entries.append({"key": key, "type": kind, "line": line, "fields": fields}) return entries @@ -123,24 +217,55 @@ def parse_bib(text: str) -> list[dict[str, Any]]: # Normalisation # --------------------------------------------------------------------------- -def normalize(text: str | None) -> str: - """Lowercase alphanumerics only, LaTeX accents and braces removed.""" +def _fold(text: str | None) -> str: + """Lowercase text with LaTeX accents, commands, braces and combining + marks removed; separators are kept so a name still splits into parts.""" if not text: return "" # CrossRef serves `&` where a bibliography writes `\&`; both are "&". text = RE_ACCENT.sub(r"\1", html.unescape(str(text))) text = RE_LATEX_CMD.sub(" ", text).replace("{", "").replace("}", "") text = unicodedata.normalize("NFKD", text) - text = "".join(char for char in text if not unicodedata.combining(char)) - return re.sub(r"[^a-z0-9]+", "", text.lower()) + return "".join(char for char in text if not unicodedata.combining(char)).lower() + + +def normalize(text: str | None) -> str: + """Lowercase alphanumerics only, LaTeX accents and braces removed.""" + return re.sub(r"[^a-z0-9]+", "", _fold(text)) + + +def _name_tokens(text: str | None) -> set[str]: + """The parts of a family name at least four letters long: particles + (`van`, `de`) are shared by unrelated names and count for nothing.""" + return {token for token in re.split(r"[^a-z0-9]+", _fold(text)) if len(token) >= 4} + + +def same_family(entry: str | None, record: str | None) -> bool: + """Whether two family names agree: equal once normalized, or sharing a + part of four or more letters (`van Waerbeke` / `Waerbeke`, `Smith-Jones` + / `Smith`). A bare substring test let `Li` pass for `Lin` and `Ma` for + `Mandelbaum`, so a wrong first author on a short name went unreported. + """ + return (normalize(entry) == normalize(record) + or bool(_name_tokens(entry) & _name_tokens(record))) + + +def first_author(authors: str) -> str: + """The first name of a BibTeX author list, split on ` and ` in any case.""" + return RE_AUTHOR_SEP.split(authors)[0].strip() + + +def family_name(author: str) -> str: + """The family part as written: before the comma of `Family, Given`, + else the last word of `Given Family`.""" + if "," in author: + return author.split(",")[0].strip() + tokens = author.split() + return tokens[-1] if tokens else "" def first_author_family(authors: str) -> str: - first = authors.split(" and ")[0].strip() - if "," in first: - return normalize(first.split(",")[0]) - tokens = first.split() - return normalize(tokens[-1]) if tokens else "" + return normalize(family_name(first_author(authors))) def _journal_name(name: str | None) -> str: @@ -167,11 +292,16 @@ def arxiv_id(fields: dict[str, str]) -> str | None: # --------------------------------------------------------------------------- class RegistryUnavailable(RuntimeError): - """A registry could not be reached or answered outside 200/404.""" + """A registry could not be reached, answered outside 200/404, or served + a body the record readers cannot use; the entry stays unmeasured.""" def fetch(url: str, timeout: int = 30) -> tuple[int, bytes]: - """(HTTP status, body). 404 is an answer, not an error; the rest raise.""" + """(HTTP status, body). 404 is an answer, not an error; the rest raise. + + Known gap (audit F22): no retry or back-off on 429/5xx, and no `mailto:` + in the User-Agent for CrossRef's polite pool. + """ request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT, "Accept": "application/json"}) try: @@ -181,40 +311,75 @@ def fetch(url: str, timeout: int = 30) -> tuple[int, bytes]: if error.code == 404: return 404, b"" raise RegistryUnavailable(f"HTTP {error.code} from {url.split('?')[0]}") from error - except (urllib.error.URLError, TimeoutError, OSError) as error: + except (http.client.HTTPException, OSError) as error: + # URLError and TimeoutError are OSError subclasses; IncompleteRead, + # BadStatusLine and RemoteDisconnected are HTTPException and are not. raise RegistryUnavailable(f"{error} ({url.split('?')[0]})") from error +def _first(value: Any, default: Any = "") -> Any: + """The first element of a list-valued registry field, the value itself + when the registry served a scalar, `default` when it served nothing. + CrossRef's `issued.date-parts` arrives as `[[]]` or `[[null]]` for an + undated work, and indexing it blindly was an IndexError traceback.""" + if isinstance(value, list): + return value[0] if value else default + return default if value is None else value + + +def _answer(body: bytes, *keys: str) -> dict[str, Any]: + """The object at `keys` in a JSON registry answer. Any other shape -- a + body that is not JSON, a missing key, a record that is not an object -- + is an answer outside the contract and leaves the entry unmeasured.""" + try: + value: Any = json.loads(body.decode("utf-8")) + for key in keys: + value = value[key] + except (ValueError, LookupError, TypeError) as error: + raise RegistryUnavailable( + f"unreadable registry answer: {type(error).__name__}: {error}") from error + if not isinstance(value, dict): + raise RegistryUnavailable("unreadable registry answer: the record is not an object") + return value + + def _crossref_record(message: dict[str, Any]) -> dict[str, Any]: - authors = message.get("author") or [] - issued = (message.get("issued") or {}).get("date-parts") or [[None]] + author = _first(message.get("author"), {}) + author = author if isinstance(author, dict) else {} + issued = message.get("issued") + date_parts = issued.get("date-parts") if isinstance(issued, dict) else None + subtitle = message.get("subtitle") or [] return { "source": "crossref", - "first_author": (authors[0].get("family") or authors[0].get("name", "")) if authors else "", - "year": issued[0][0], + "first_author": author.get("family") or author.get("name") or "", + "year": _first(_first(date_parts, []), None), # CrossRef splits "GaBoDS: the Garching-Bonn Deep Survey. IX. A sample # of ..." into title and subtitle; a bibliography carries both as one. - "title": " ".join((message.get("title") or [""])[:1] - + (message.get("subtitle") or [])), - "journal": (message.get("container-title") or [""])[0], - "journal_short": (message.get("short-container-title") or [""])[0], - "volume": message.get("volume", ""), + "title": " ".join(str(part) for part in [_first(message.get("title"))] + + (subtitle if isinstance(subtitle, list) else [subtitle])), + "journal": str(_first(message.get("container-title"))), + "journal_short": str(_first(message.get("short-container-title"))), + "volume": message.get("volume") or "", "page": message.get("page") or message.get("article-number") or "", } def _datacite_record(attributes: dict[str, Any]) -> dict[str, Any]: - creators = attributes.get("creators") or [{}] - container = attributes.get("container") or {} + creator = _first(attributes.get("creators"), {}) + creator = creator if isinstance(creator, dict) else {} + title = _first(attributes.get("titles"), {}) + container = attributes.get("container") + container = container if isinstance(container, dict) else {} return { "source": "datacite", - "first_author": creators[0].get("familyName") or creators[0].get("name", "").split(",")[0], + "first_author": (creator.get("familyName") + or str(creator.get("name") or "").split(",")[0]), "year": attributes.get("publicationYear"), - "title": (attributes.get("titles") or [{}])[0].get("title", ""), - "journal": container.get("title", ""), + "title": (title.get("title") or "") if isinstance(title, dict) else str(title), + "journal": container.get("title") or "", "journal_short": "", - "volume": container.get("volume", ""), - "page": container.get("firstPage", ""), + "volume": container.get("volume") or "", + "page": container.get("firstPage") or "", } @@ -222,10 +387,10 @@ def resolve_doi(doi: str) -> dict[str, Any] | None: """The record a DOI resolves to, or None when neither registry knows it.""" status, body = fetch(CROSSREF + urllib.parse.quote(doi, safe="/")) if status == 200: - return _crossref_record(json.loads(body.decode("utf-8"))["message"]) + return _crossref_record(_answer(body, "message")) status, body = fetch(DATACITE + urllib.parse.quote(doi, safe="/")) if status == 200: - return _datacite_record(json.loads(body.decode("utf-8"))["data"]["attributes"]) + return _datacite_record(_answer(body, "data", "attributes")) return None @@ -233,7 +398,10 @@ def resolve_arxiv(identifier: str) -> dict[str, Any] | None: status, body = fetch(ARXIV + urllib.parse.quote(identifier)) if status != 200: return None - entry = ET.fromstring(body.decode("utf-8", "replace")).find(f"{ATOM}entry") + try: + entry = ET.fromstring(body.decode("utf-8", "replace")).find(f"{ATOM}entry") + except ET.ParseError as error: + raise RegistryUnavailable(f"unreadable arXiv answer: {error}") from error # An unknown identifier still returns a feed, with one entry that has no id. if entry is None or entry.findtext(f"{ATOM}id") is None: return None @@ -260,15 +428,15 @@ def compare(fields: dict[str, str], record: dict[str, Any]) -> list[dict[str, An compared: silence is not agreement, and the caller reports what it saw. """ issues: list[dict[str, Any]] = [] - author = first_author_family(fields.get("author", "")) - record_author = normalize(record.get("first_author")) - if author and record_author and author != record_author \ - and author not in record_author and record_author not in author: - issues.append({"field": "first_author", "entry": fields["author"].split(" and ")[0], + author = first_author(fields.get("author", "")) + family, record_family = family_name(author), str(record.get("first_author") or "") + if normalize(family) and normalize(record_family) \ + and not same_family(family, record_family): + issues.append({"field": "first_author", "entry": author, "record": record.get("first_author"), "strength": "strong"}) - year = fields.get("year", "") - if year.isdigit() and record.get("year"): - gap = abs(int(year) - int(record["year"])) + year, record_year = fields.get("year", ""), str(record.get("year") or "") + if year.isdigit() and record_year.isdigit(): + gap = abs(int(year) - int(record_year)) if gap: issues.append({"field": "year", "entry": year, "record": record["year"], "strength": "strong" if gap > YEAR_SEAM else "ordinary"}) @@ -319,17 +487,27 @@ def _finding(*, path: Path, key: str, line: int, rule: str, kind: str, strength: def entry_identifier(fields: dict[str, str]) -> str | None: - """`doi:`, else `arxiv:`, else None.""" - doi = RE_DOI_URL.sub("", fields.get("doi", "").strip()) - if doi: - return "doi:" + doi + """`doi:` from the `doi` field or a doi.org `url`, else `arxiv:`, + else None. A DOI that lives only in the `url` field (`https://doi.org/...`, + as several exporters write it) is an identifier all the same.""" + for name in ("doi", "url"): + value = fields.get(name, "").strip() + if value and (name == "doi" or RE_DOI_URL.match(value)): + doi = RE_DOI_URL.sub("", value) + if doi: + return "doi:" + (urllib.parse.unquote(doi) if name == "url" else doi) arxiv = arxiv_id(fields) return ("arxiv:" + arxiv) if arxiv else None def verify_entry(entry: dict[str, Any], path: Path, cache: dict[str, Any], pause: float) -> tuple[list[dict[str, Any]], str]: - """(findings, state); state is verified / unresolved / unmeasured / no_identifier.""" + """(findings, state); state is verified / unresolved / unmeasured / no_identifier. + + `cache` maps identifiers to fetched records and is updated in place. A + miss (None) is never stored: a DOI the registries are late to index + would otherwise stay a blocker in every later round without a query. + """ fields, key, line = entry["fields"], entry["key"], entry["line"] identifier = entry_identifier(fields) if identifier is None: @@ -340,9 +518,9 @@ def verify_entry(entry: dict[str, Any], path: Path, cache: dict[str, Any], "so no registry can confirm it.", action="Add the DOI or arXiv identifier, or verify the entry against the " "publisher page by hand and record that in the review.")], "no_identifier" - if identifier in cache: - record = cache[identifier] - else: + record, cached = cache.get(identifier), True + if record is None: + cached = False scheme, _, value = identifier.partition(":") try: record = resolve_doi(value) if scheme == "doi" else resolve_arxiv(value) @@ -351,10 +529,11 @@ def verify_entry(entry: dict[str, Any], path: Path, cache: dict[str, Any], path=path, key=key, line=line, rule="reference-lookup-failed", kind="advisory", strength="ordinary", status="unmeasured", observed={"identifier": identifier, "error": str(error)}, - message=f"Entry {key!r}: the registry did not answer ({error}).", + message=f"Entry {key!r}: the registry gave no usable answer ({error}).", action="Re-run when the network allows; the entry is unverified, " "not clean.")], "unmeasured" - cache[identifier] = record + if record is not None: + cache[identifier] = record if pause: time.sleep(pause) if record is None: @@ -373,7 +552,8 @@ def verify_entry(entry: dict[str, Any], path: Path, cache: dict[str, Any], path=path, key=key, line=line, rule=f"reference-metadata-mismatch:{issue['field']}", kind="advisory", strength=issue["strength"], - observed={"identifier": identifier, "record_source": record["source"], **issue}, + observed={"identifier": identifier, "record_source": record["source"], + "cached": cached, **issue}, message=(f"Entry {key!r}: {issue['field']} reads {issue['entry']!r} " f"but the {record['source']} record says {issue['record']!r}."), action=("Correct the entry from the record, or confirm the identifier " @@ -384,29 +564,64 @@ def verify_entry(entry: dict[str, Any], path: Path, cache: dict[str, Any], return findings, "verified" +def cite_lines(text: str) -> dict[str, int]: + """Each cited key with the first line citing it, `%` comments removed. + + A citation inside a comment is text (the pattern is the one + `tex_assembly` strips before resolving includes), and natbib's + `\\citetext{...}` holds prose, not keys. Lines count in the text given: + the assembled document, whose numbering is the root file's own until the + first spliced `\\input`. `\\nocite{*}` yields the key `*`; `cross_check` + reads it as citing every entry. + """ + lines: dict[str, int] = {} + text = tex_assembly.RE_TEX_COMMENT.sub("", text) + for match in RE_CITE.finditer(text): + suffix = match.group("suffix").lower() + if suffix == "text" and not match.group("prefix"): + continue + groups, end = [match.group("keys")], match.end() + if suffix in MULTICITE_SUFFIXES: + while (more := RE_CITE_GROUP.match(text, end)) is not None: + groups.append(more.group(1)) + end = more.end() + line = text.count("\n", 0, match.start()) + 1 + for key in (key.strip() for group in groups for key in group.split(",")): + if key: + lines.setdefault(key, line) + return lines + + def cite_keys(text: str) -> set[str]: - return {key.strip() for match in RE_CITE.finditer(text) - for key in match.group(1).split(",") if key.strip()} + return set(cite_lines(text)) def cross_check(entries: list[dict[str, Any]], tex_text: str, bib_path: Path, tex_path: Path) -> list[dict[str, Any]]: - """Keys the document cites without an entry, and entries nothing cites.""" + """Keys the document cites without an entry, and entries nothing cites. + + A missing-entry finding points at the line of the citing command; an + uncited-entry finding points at the entry in the bibliography. + """ known = {entry["key"]: entry for entry in entries} - cited = cite_keys(tex_text) + cited = cite_lines(tex_text) + everything = cited.pop("*", None) # \nocite{*}: every entry is cited + if everything is not None: + for key in known: + cited.setdefault(key, everything) findings = [_finding( - path=tex_path, key=key, line=1, rule="reference-missing-entry", + path=tex_path, key=key, line=cited[key], rule="reference-missing-entry", kind="integrity_blocker", strength="strong", observed={"bibliography": bib_path.name}, message=f"\\cite{{{key}}} has no entry in {bib_path.name}.", action="Add the entry or fix the key; the build prints [?] for it.") - for key in sorted(cited - known.keys())] + for key in sorted(cited.keys() - known.keys())] findings.extend(_finding( path=bib_path, key=key, line=known[key]["line"], rule="reference-uncited", kind="advisory", strength="ordinary", observed={"bibliography": bib_path.name}, message=f"Entry {key!r} is cited nowhere in the assembled document.", action="Remove the dead entry unless the bibliography is shared across papers.") - for key in sorted(known.keys() - cited)) + for key in sorted(known.keys() - cited.keys())) return findings @@ -427,19 +642,31 @@ def build_parser() -> argparse.ArgumentParser: parser = cli_common.base_parser(__doc__) parser.add_argument("bib", type=Path, help="the .bib file to verify") parser.add_argument("--tex", type=Path, default=None, - help="document root; its assembled \\cite keys are cross-checked") + help="document root; its assembled citation keys are cross-checked") parser.add_argument("--cache", type=Path, default=None, - help="JSON file of fetched records, read and updated") + help="JSON file of fetched records, read and updated; " + "a registry miss is not kept") parser.add_argument("--pause", type=float, default=0.2, help="seconds between registry requests (default 0.2)") return cli_common.report_options(parser) +def load_cache(path: Path | None) -> dict[str, Any]: + """The records a previous round fetched, keyed by identifier. Only + records are kept: a `null` an earlier version wrote for a registry miss, + or anything else that is not a record, is re-queried rather than trusted.""" + if path is None or not path.is_file(): + return {} + loaded = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(loaded, dict): + raise ValueError(f"cache {path} is not a JSON object") + return {key: value for key, value in loaded.items() + if isinstance(value, dict) and "source" in value} + + def verify(args: argparse.Namespace) -> dict[str, Any]: entries = parse_bib(args.bib.read_text(encoding="utf-8")) - cache: dict[str, Any] = {} - if args.cache is not None and args.cache.is_file(): - cache = json.loads(args.cache.read_text(encoding="utf-8")) + cache = load_cache(args.cache) findings: list[dict[str, Any]] = [] states: dict[str, str] = {} for entry in entries: @@ -470,9 +697,15 @@ def main(argv: list[str] | None = None) -> int: return 2 try: report = verify(args) - except (OSError, ValueError, json.JSONDecodeError, UnicodeDecodeError, - ET.ParseError) as error: - print(f"[verify_references] execution failed: {error}", file=sys.stderr) + except Exception as error: + # Deliberately broad, as in ai_ism_lint: exit 1 is reserved for "a + # blocker is present", and an uncaught traceback also exits 1, so a + # crash -- an unreadable .bib or cache, a registry answer the record + # readers do not expect, an HTTP-layer error a narrow tuple missed -- + # would read as a blocker verdict. KeyboardInterrupt and SystemExit + # are BaseException and still propagate. + print(f"[verify_references] execution failed: {type(error).__name__}: {error}", + file=sys.stderr) return 2 cli_common.emit_report(report, args, render=render, tool="verify_references") return 1 if any(f["kind"] == "integrity_blocker" for f in report["findings"]) else 0 From 1ad6ed7090f48c3146c1ec47230727b5eb4eee6e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:03:30 +0000 Subject: [PATCH 05/27] docs: the skills, annexes, hub and registries say what the code now does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit tier two (docs/audits/full-review-2026-09-27.md §6 and §8.2): - skills: the condense baseline is the whole include tree or a git ref, a target of 0 closes, the duplicate floor is 30 prose words, and the rewrite gate no longer needs the optional learned model; de-ai rebuilds every calibrated axis when the corpus changes, drops the no-op --summary, states the three zero-hit exemptions the tool applies, snapshots the document and runs the §5.3 loop-close gates before it stops; final-review verifies the bibliography before each round, snapshots the target, closes every round with the length and residue gates and lists them in its stable-round criterion; paper-review §O carries residue-absence and the delete-first ruling; the paper skill's Tier A table and grep agree with the linter (paved, showcased, connectors per paragraph, the cap per heading unit); - the standard is v3.9: the fourth narrow exit contract, verify_references; the cap is a section cap, step 5 is L2 anchoring and L4 voice; - RESPONSIBILITIES gains the residue-absence rule, the three zero-hit exemptions, the linter's real default axes, and rows for verify_references and label_findings; DISPOSITIONS quotes the v0.36.2 and v0.36.3 figures (3.37 per 1,000 / AUC 0.174, 1.59%, 41,644 passages, 530,504 pairs, 10.78% / 7.80%), counts four static residue rules, retitles the register row, and records the spaced-unit limit; the EVALUATION hub's L3 row is the 2026-09-05 retrain, its document-shape row dates the 493-paper scoring against the 507-document artifact, cohesion and hedging quote the v0.36.3 sizes, the v0.28.0 release gate and the two "open" decisions are closed; DEAI_SUBSYSTEM's paper-as-agent example is one the detector flags, its exemptions, pair breaks, weight names, float exclusion, exit-contract count, policy-asset and L1 wording match the code, and its validator list is complete; - README: 11 checks, 40 tools in one schema, the worked-example table reads the revision at 3 as examples/README does, the retrain row quotes the current stability figures, demo 2's command reproduces its counts, the paper-review row lists dimension P; README.zh-CN: the refuted estimator-noise model, the demo-2 clause, the bank sizes; - style-corpus: tier weights are recorded, not applied; the extractor warns below five files; the breadth and held-out directories are in the layout; style-profile: the discourse and anchoring calibrations and their artifacts, the negatives file the harvester writes, the rebuild notice; examples: the headline pair is restated in the abstract, dated to the v0.36.3 run; docs/README: the link check and the worked example; - tools/README: rows for length_gate, condense_map, ai_ism_lint, verify_references (its agent's text), cli_common, extract_style, tex_assembly, a new tex_macros row, deai_residue (13.3%), deai_register (AUC 0.174, three exemptions) and validate_plugin (11 checks); - the 2026-09-04 audit note's sixteen links are repository-relative. Recorded test counts are updated with the release notes. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- README.md | 24 +++++----- README.zh-CN.md | 21 +++++---- docs/README.md | 4 +- docs/SCIPAPER_STANDARD.md | 38 +++++++-------- docs/architecture/DEAI_SUBSYSTEM.md | 66 ++++++++++++++++----------- docs/architecture/DISPOSITIONS.md | 13 +++--- docs/architecture/EVALUATION.md | 27 ++++------- docs/architecture/RESPONSIBILITIES.md | 8 ++-- examples/README.md | 8 ++-- skills/condense/SKILL.md | 24 ++++++---- skills/de-ai/SKILL.md | 45 +++++++++++++----- skills/final-review/SKILL.md | 27 +++++++++-- skills/paper-review/SKILL.md | 8 +++- skills/paper/SKILL.md | 18 +++++--- style-corpus/README.md | 19 ++++---- style-profile/README.md | 14 +++++- tools/README.md | 21 +++++---- 17 files changed, 233 insertions(+), 152 deletions(-) diff --git a/README.md b/README.md index 0870e67..8840df7 100644 --- a/README.md +++ b/README.md @@ -67,14 +67,14 @@ problem: | **measure** | [`logic`](skills/logic/SKILL.md) | Measurement primitive, findings only. Claim graph (circular arguments, broken chains, switched conditions, undeclared assumptions), empirical statistics (splits, leakage, multiple comparisons, priors), and the review side of claim–evidence discipline: verb strength may not exceed evidence strength. The single source for `paper-review` dimension C. | | **measure** | [`mainline`](skills/mainline/SKILL.md) | Measurement primitive, findings only. Builds a paper-level purpose record and contribution graph, then answers a cold reader's seven questions — where must a reader backtrack, supply hidden context, or choose between competing readings? No three-act template assumed. The single source for `paper-review` dimension E. | | **measure** | [`figure-review`](skills/figure-review/SKILL.md) | Reviews **compiled pages at 150 DPI**, not source. Traces figure/caption/data provenance, measures canvas balance at the pixel level, separates scientific and build contradictions from readability advisories. | -| **act** | [`de-ai`](skills/de-ai/SKILL.md) | Three chained passes — subsystem measurement (L0–L4), a structural-tell audit, then **claim-first rewriting** that rebuilds prose from the protected claim graph instead of polishing in place. `--audit-only` stops after measurement. | +| **act** | [`de-ai`](skills/de-ai/SKILL.md) | Three chained passes — subsystem measurement (L0–L4), a structural-tell audit, then **claim-first rewriting** that rebuilds prose from the protected claim graph instead of polishing in place. `--audit-only` stops after the structural-tell audit (Pass 2). | | **act** | [`condense`](skills/condense/SKILL.md) | Whole-document redundancy elimination under one-canonical-home-per-fact, loop-until-dry convergence, and a **mechanical length gate** as the closing proof that the document actually shrank. | -| **compose** | [`paper-review`](skills/paper-review/SKILL.md) | Source-traced **A–R review**: mathematics, physics, logic and statistics, language, structure and narrative spine, citations, data and figures, interfaces, redundancy, reproducibility, modern-physics checks, consistency, adversarial verification (three passes + twelve-framing escalation), staleness, process artifacts, citation precision, glossary alignment. | +| **compose** | [`paper-review`](skills/paper-review/SKILL.md) | Source-traced **A–R review**: mathematics, physics, logic and statistics, language, structure and narrative spine, citations, data and figures, interfaces, redundancy, reproducibility, modern-physics checks, consistency, adversarial verification (three passes + twelve-framing escalation), staleness, process artifacts, draft language, citation precision, glossary alignment. | | **compose** | [`final-review`](skills/final-review/SKILL.md) | Parent orchestrator: runs paper-review, figure-review, de-ai `--audit-only`, and the physics / mainline / logic primitives as **isolated worktree agents**, merges their typed findings, and verifies a stable disposition-complete state across consecutive rounds. | | **genre** | [`proposal-polish`](skills/proposal-polish/SKILL.md) | NSF Project Summary/Description, NIH Specific Aims, fellowships. Keeps the vision-and-feasibility register a paper would trim, enforces claim–feasibility matching, edits the score-forming first pages hardest. | | **explore** | [`brainstorm`](skills/brainstorm/SKILL.md) | Radial research-direction explorer: twelve framing passes per node, glossary-anchored terminology, complete derivation per branch, recursive divergence to convergence. Deferred or incomplete leaves are hard-banned. | -Every skill that emits a finding runs over the same evidence layer — 35 tools emitting one schema, `sci-paper.feedback.v1` — so a finding from the linter, a measurement primitive and the orchestrator are the same +Every skill that emits a finding runs over the same evidence layer — one schema, `sci-paper.feedback.v1`, whatever tool emits it — so a finding from the linter, a measurement primitive and the orchestrator are the same object with the same ID. A composite calls primitives; it never restates their checks. `calibrate` builds the layer they measure on. ### Scope boundary — what it deliberately will not do @@ -268,7 +268,7 @@ and a six-sentence recital of a parameter grid. ```console $ python tools/ai_ism_lint.py before.tex --field wgl \ - --structure --distribution --register --salience --document-structure --no-discourse + --structure --distribution --register --salience --document-structure --no-discourse --no-collocation --no-residue findings: blockers=0 L0=8 advisories=10 (strong=1) axis L0.lexical: measured axis L0.register: measured axis L2.salience_hierarchy: measured @@ -387,9 +387,9 @@ disagreements through twelve framings to `CONFIRMED` / `REFUTED` / `MARGINAL`. |---|---:|---:| | L0 targets | 1 | **0** | | `discourse-cohesion` | 3 | **1** | -| `salience-recital` | 4 | **6** | +| `salience-recital` | 4 | **3** | -The last row is the point: carrying a noun forward to link two sentences pulls the subject into sentences that also carry a numeral, so in a number-dense passage cohesion and recital want opposite things and no rewrite satisfies both. Both findings are true, and which to act on is the author's judgement — which is why neither is a blocker and there is no score. Walkthrough: [`examples/README.md`](examples/README.md). +The last row moved with the reference: the 2026-08-27 profile read the revision at 6, and the v0.36.3 rebuild, which calibrates on the numeral projection, reads the same file at 3, the two findings that stay strong having changed kind from run length to density. The tension is still real: carrying a noun forward to link two sentences pulls the subject into sentences that also carry a numeral, so in a number-dense passage cohesion and recital want opposite things and no rewrite satisfies both. Both findings are true, and which to act on is the author's judgement — which is why neither is a blocker and there is no score. Walkthrough: [`examples/README.md`](examples/README.md). ## Benchmark dashboard @@ -477,8 +477,8 @@ at 100%, and a number taken under that describes the contention, not the axis. V | Full test suite — **529 passing**, 27 files (median of 3, spanning 49.8 – 51.7 s) | 51.5 s | stdlib | The headline: **a complete model-free pass over a 5,084-word manuscript costs ~1.0 s above the interpreter -floor**, with no optional dependency installed — and 0.6 s of that is loading the 530,677-pair collocation -bank, which `--no-collocation` drops to ~0.4 s. The two model-backed axes cost 30×–35× more than the full +floor**, with no optional dependency installed — and 0.6 s of that is loading the collocation bank (530,677 +pairs in the 2026-09-04 bank the timing was taken on; 530,504 after the v0.36.3 rebuild), which `--no-collocation` drops to ~0.4 s. The two model-backed axes cost 30×–35× more than the full model-free pass and are opt-in flags — you should not need a GPU to lint a paper. CI runs validator and suite on every push to main and on every PR, Python 3.11, Ubuntu. --- @@ -544,7 +544,7 @@ gates, `build` corpus and profile construction, `eval` evidence. Detail: [tools/ |---|---|---| | `tools/deai_feedback.py` | core | Implements `sci-paper.feedback.v1`: stable IDs, consequence classes, measurement states, dispositions, ranking, summaries, rendering. Standard library only. | | `tools/ai_ism_lint.py` | core | The unified CLI. Aggregates L0 and every advisory axis into one ranked text/JSON report. Exit `0` = no L0 target, `1` = L0 target present, `2` = invalid input or execution failure. | -| `tools/length_gate.py` | core | Per-section prose length-budget delta gate (standard §5.3). Exit 1 on net unjustified growth between two document versions, or on a net cut short of `--require-shrink`; `--allow` records justifications. | +| `tools/length_gate.py` | core | Per-section prose length-budget delta gate (standard §5.3). Exit 1 on net unjustified growth between two document versions, or on a net cut short of `--require-shrink` (0 = no cut required); exit 2 on an unreadable document or baseline, never a verdict; `--allow` records justifications. | | `tools/verify_references.py` | core | Verifies a bibliography against the registries its identifiers name (CrossRef, DataCite, arXiv): a DOI or arXiv identifier that resolves nowhere, or a cited key with no entry, is an integrity blocker; a first author, year or title that disagrees with the record is a strong advisory; journal, volume and page differences are ordinary. An entry the network could not answer for stays `unmeasured`. Exit 1 on a blocker. | | `tools/condense_map.py` | core | The removal map behind `/sci-paper:condense`: restatements with their canonical home, zero-gain sentences, dead figures/tables/labels/macros/acronyms, verbose constructions, repeated glosses, duplicated paragraphs — each with the words it frees, totalled into a shrink target. Deletes nothing. | | `tools/deai_residue.py` | core | The trace an edit leaves: first-person drafting history, a sentence that defines its object by what it never does, edit-meta text, a heading or caption the body never earns, and with `--before` the label an edit added. Exit 1 on a strong finding. | @@ -573,7 +573,7 @@ gates, `build` corpus and profile construction, `eval` evidence. Detail: [tools/ | `tools/eval_findings.py` | eval | Scores register, salience, cohesion, hedging and collocation against **provenance** labels instead of hand labels: their firing rate on held-out refereed ApJ/ApJL/A&A papers, on the in-sample papers, and on the `docval` machine tiers, plus a paired test that isolates calibration leakage from publication era. | | `tools/label_findings.py` | eval | Samples findings from all five finding-emitting axes (register, salience, cohesion, hedging, collocation) into a human-labelling sheet, re-serves a blind subset for intra-rater agreement, and reports per-axis precision plus **pooled** recall, stratified by named `--population NAME=DIR` sets. Reports `unmeasured` for any stratum under 20 labels. | | `tools/build_profile.py` | build | Builds the basic field profile: extraction, optional legacy classifier, exemplar-cache warm-up. | -| `tools/cli_common.py` | build | Shared command-line preamble and field resolution, used by 28 of 40 tools. Holds no policy: no default beyond the two roots, reads no profile, emits no findings. | +| `tools/cli_common.py` | build | Shared command-line preamble and field resolution (one profile auto-resolves; several run without one and say so), used by 30 of the 39 Python tools. Holds no policy: no default beyond the two roots, reads no profile, emits no findings. | | `tools/extract_style.py` | build | Extracts lexicon, sentence statistics, transitions, a descriptive dossier, and a section-typed exemplar bank. Re-exports every public name from `extract_sections.py`. | | `tools/extract_sections.py` | build | Source-text projection and section splitting: the section vocabulary and its classifier, both named LaTeX projections, and the PDF heading heuristic. Section buckets key every per-section reference, so changing this requires a profile rebuild. | | `tools/tex_assembly.py` | build | Assembles a LaTeX document from its root: `\input`/`\include` children spliced in place (a call mid-line keeps the words around it, a second call splices again, a cycle stops), from the file system or from a git ref so a `--git-ref` baseline is the assembled document at that ref. Owner of the comment and include patterns every reader shares. | @@ -663,7 +663,7 @@ dossier is evidence, not a standard and not proof of authorship. | Corpus reference | **User-supplied, tiered, gitignored** | Style is field-relative. A generic prior is the thing being replaced. | | Optional models | `transformers`+`torch`, `scikit-learn`, `sentence-transformers` | Strictly opt-in flags. Absence degrades an axis, never the run. | | Distribution | **Claude Code plugin** (`.claude-plugin/plugin.json`) | Skills at `skills//SKILL.md`, namespaced `/sci-paper:`. | -| Contract enforcement | `tools/validate_plugin.py` + GitHub Actions | 10 checks over manifests, registries, doc authority, recorded counts, imports, CLI entry points, exit semantics, tests, CI wiring, and the residue contract. Drift fails CI instead of accumulating. | +| Contract enforcement | `tools/validate_plugin.py` + GitHub Actions | 11 checks over manifests, registries, doc authority and links, recorded counts, imports, CLI entry points, exit semantics, tests, CI wiring, the residue contract, and the 750-line budget. Drift fails CI instead of accumulating. | --- @@ -713,7 +713,7 @@ de-AI standard. | **Long-form generation is not caught** | At α = 0.05, manifold tail power on long-form AI is **0.000** — stable across 2 metrics × 4 calibration splits × 12 seeds. Rank AUC is 0.729, so the signal exists and the operating point cannot reach it. | | **Cooperative-layer tools** | `deai_provenance` and `deai_personal` are honestly `unmeasured` until the author supplies their own draft history or ≥ 3 prior papers. | | **`L1.distribution` / `L2.sentence_structure`** | `degraded` — and now for a *measured* reason. Burstiness reverses sign on adversarial prose (AUC 0.181) and signposting runs below chance (0.247), so no operating point is available to write. | -| **Retrains are not behaviour-preserving** | Rebuilding the profile refits L3. Ranking holds at ρ 0.846 and triage overlap 0.654, but an old triage list will not reproduce exactly. | +| **Retrains are not behaviour-preserving** | Rebuilding the profile refits L3. Ranking holds at ρ 0.991 and triage overlap 0.889 for the 2026-09-05 retrain (0.846 / 0.654 across the 2.6× bank growth before it), but an old triage list will not reproduce exactly. | | **A quarter of the corpus is never used** | Headings matching no section bucket are dropped rather than guessed: **2,334 of 9,178 (25.4%)** in `wgl`, 42 of 148 in `wgl-letter`. The remainder is mostly topic headings ("Matter power spectrum"); "Measurements" and "Background" were refused as genuinely ambiguous. | | **Register fires on accepted prose, and the vocabulary audits are advice** | Measured on 203 held-out refereed ApJ/ApJL/A&A papers it never saw: **0.0371 findings per 1,000 body words** (57 findings; the same 57 read 0.0247 against a raw-source denominator until v0.36.2, and 0.0858 before the v0.36.0 heading and v0.36.1 float fixes), 22.2% of documents, rank AUC **0.391** against machine text — it still fires *more* on human papers than on AI drafts, and 98.2% of the 57 remaining flags would vanish if the paper sat in its own bank. Sweeping the use floor 5 → 50 keeps AUC below 0.5 **everywhere**, so no setting makes this a detector (replicated by author rather than journal, 22 papers, AUC **0.328**, [§21](docs/architecture/evaluation/held-out-labels.md)). The exhaustive zero-hit audit is the same fact at full strength: every refereed paper carries words the corpus never wrote (3.37 per 1,000 body words, AUC 0.174), so it and the collocation axis ship as advice with author dispositions, and the collocation bank costs 0.6 s per run that `--no-collocation` drops ([§23](docs/architecture/evaluation/vocabulary-and-residue.md)). | | **Advice quality is still unlabelled** | Provenance answers "does it fire on accepted prose", not "is this advisory right". Salience's p90 gate fires at **0.203** per passage on 203 held-out refereed papers, under its 0.271 three-gate union bound ([§17.5](docs/architecture/evaluation/held-out-labels.md)); it read 0.454 under v0.36.2 because the bank stored only the `[math]` projection, so the reference held no numeral inside math while every manuscript did — closed in v0.36.3 by storing each paragraph's numeral projection beside it. 7.00% of the digits it read on LaTeX were citation years until v0.32.0; precision and recall for the advice itself need `tools/label_findings.py`, and running it is the author's act. | diff --git a/README.zh-CN.md b/README.zh-CN.md index 9972d7f..6e1cba6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -68,7 +68,7 @@ sci-paper 把一个 Claude Code 会话变成一张论文工作台，由唯一一 | **体裁** | [`proposal-polish`](skills/proposal-polish/SKILL.md) | NSF Project Summary/Description、NIH Specific Aims、fellowship。保留论文会删掉的"愿景+可行性"语域，强制 claim 与可行性匹配，最狠地打磨决定评分的前几页。 | | **探索** | [`brainstorm`](skills/brainstorm/SKILL.md) | 辐射状研究方向探索器：每节点 12 条 framing pass、术语锚定到 glossary、每分支完整推导、递归发散直到收敛。严禁 defer / future-work / 半成品叶节点。 | -凡是产出 finding 的 skill 都跑在同一层证据之上 —— 35 个工具输出同一套 schema +凡是产出 finding 的 skill 都跑在同一层证据之上 —— 所有工具输出同一套 schema `sci-paper.feedback.v1` —— 所以 linter、测量原件和编排器给出的 finding 是同一个 对象、同一个 ID。**组合只调用原件，绝不复述它们的检查项。**`calibrate` 构建的 就是它们赖以测量的这层证据。 @@ -378,10 +378,12 @@ AUC 是长度公平的：每份文档只跟**自己那个长度层**的人类论 恰好集中在把流形逼到最窄的那一档上。跨越 42 倍的语料重建，八个已发表 AUC **全部复现在 0.012 以内** —— 这是逐值核对，不是总数对上就算数。 -尾部功效照实报，包括弱的地方和为零的地方。长文生成在严格操作点上**完全抓不到**。 -短文档仍是流形最弱的一档：在短层内部，人类距离与段落数仍有 **−0.414** 的相关。 -更细的分层已经测过并被否掉（不涨功效，还倒赔），所以真正的修法是估计量噪声模型， -本版**没有**发布。 +尾部功效照实报，包括弱的地方和为零的地方，而且**上表的尾部功效一列是单次 seed +抽样**：这些档上逐 seed 的离散度是 0.04–0.18，12 个 seed 下 natural 读作 +0.170 ± 0.110。长文生成在严格操作点上任何配置都**完全抓不到**。本该修好短文档的 +估计量噪声模型已经建出来并被证伪 —— 它对人类误报率毫无影响 +（[§9.4c](docs/architecture/evaluation/document-scale.md)）。整张表可用 +`python tools/eval_docscale.py --field wgl` 重跑。 ### 为什么没有单一分数：L3 的混淆 @@ -435,7 +437,7 @@ LaTeX include 组装而成。标准库的每一行都重测了：解释器地板 | 完整测试套件 —— **529 通过**，27 个文件（3 次取中位数，落在 49.8 – 51.7 s） | 51.5 s | 标准库 | 一句话：**一份 5,084 词的稿子跑完全部 model-free 通道，在解释器地板之上约花 1.0 s**， -且不需要任何可选依赖——其中 0.6 s 是加载 530,677 个词对的 collocation 库， +且不需要任何可选依赖——其中 0.6 s 是加载 collocation 词对库（计时所用的 2026-09-04 库有 530,677 对，v0.36.3 重建后为 530,504 对）， `--no-collocation` 能把它降回约 0.4 s。两条模型驱动的轴比完整的 model-free 通道贵 30–35 倍，并且是显式 opt-in 的 flag —— lint 一篇论文不该需要一块 GPU。CI 每次推到 main 的 push 与每个 PR 都跑 validator + 套件，Python 3.11，Ubuntu。 @@ -573,8 +575,7 @@ profile 构建，`eval` 可复现的证据。逐工具细节见 [tools/README.md 每条 strong advisory 最终落到一个 **disposition**：`acted`、`accepted`、 `rejected_as_false_positive`，或带明确理由的 `pending`。普通 advisory 保持可见， -不要求消失 —— 这正是 [demo 2](#2-一份草稿的-before--after) 停在三条 advisory -而不是零条的原因。 +不要求消失。 ## 按 field 组织的证据 @@ -687,12 +688,12 @@ Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权 | **长文生成抓不到** | α = 0.05 下流形对长文 AI 的尾部功效是 **0.000** —— 在 2 种度量 × 4 种标定切分 × 12 个 seed 下都稳定。排序 AUC 是 0.729，说明信号在，操作点够不着。 | | **协作层工具** | `deai_provenance` 与 `deai_personal` 在作者提供自己的草稿历史或 ≥ 3 篇既往论文之前，诚实地保持 `unmeasured`。 | | **`L1.distribution` / `L2.sentence_structure`** | `degraded` —— 现在是有*测量依据*的。burstiness 在对抗文本上符号反转（AUC 0.181），signposting 低于随机（0.247），根本写不出一个操作点。 | -| **重训不保证行为等价** | 重建 profile 会重拟 L3。排序保持 ρ 0.846、分诊重合 0.654，但旧的分诊清单不会一字不差复现。 | +| **重训不保证行为等价** | 重建 profile 会重拟 L3。2026-09-05 那次重训排序保持 ρ 0.991、分诊重合 0.889（此前跨 2.6 倍语料增长的一步是 0.846 / 0.654），但旧的分诊清单不会一字不差复现。 | | **语料有四分之一从未被用上** | 匹配不到任何 section 桶的标题是被丢弃，而不是被猜进某个桶：`wgl` 里 9,178 个标题中有 **2,334 个（25.4%）**，`wgl-letter` 里 148 个中有 42 个。剩下的多是主题标题（"Matter power spectrum"）；"Measurements" 与 "Background" 因真歧义被拒绝加入。 | | **register 在已发表文字上照样开火** | 在它从没见过的 203 篇留出 ApJ/ApJL/A&A 已发表论文上实测：v0.36.0 修掉标题投影、v0.36.1 修掉浮动体投影后**每千正文词 0.0371 条**（同样这 57 条在 v0.36.2 之前按原始源码分母读作 0.0247，两次修复之前是 0.0858），22.2% 的文档命中，对机器文本的秩 AUC **0.391** —— 它在人类论文上仍比在 AI 草稿上更爱开火。剩下 57 条 flag 里，98.2% 只要论文自己在库里就会消失。把用词门槛从 5 扫到 50，AUC **处处低于 0.5**：没有任何设置能把它变成检测器，它就是一条建议，切在「一篇够格送审的论文不会过半被点名」的第一个点上。这个结论已在第二个群体上复现 —— 按作者而非期刊取样的 22 篇导师论文（1996–2015），AUC **0.328**（[§21](docs/architecture/evaluation/held-out-labels.md)）。 | | **建议质量仍未被标注** | 出处只能回答「它是否在已发表文字上开火」，回答不了「这条建议对不对」。salience 的 p90 门在 203 篇留出已审稿论文上逐 passage 开火 **0.203**，低于三门并集上界 0.271（[§17.5](docs/architecture/evaluation/held-out-labels.md)）；v0.36.2 时读作 0.454，原因是语料库只存了 `[math]` 投影，参照里数学内一个数字都没有，而每篇手稿都有 —— v0.36.3 给每个段落并排存下数字投影后关闭。v0.32.0 之前它在 LaTeX 上读到的数字有 7.00% 是引用年份；建议本身的精确率与召回率仍需 `tools/label_findings.py`，跑不跑是作者自己的事。 | | **hedging 只对引言说话** | 认知情态轴发布时被收窄到只管 `intro` —— 在 203 篇留出的已审稿论文上，它的 p10 门在 `intro` 上开火率是 7.89%，而其他桶是 15–27% —— 那是审稿人已经接受的文字，且至少有一套生成流程落到随机以下。cohesion 不需要这条限制（七个桶 6.6–14.6%）。 | -| **两条词汇审计是建议，且要加载词对库** | 零命中审计列出语料从未写过的每一个词，而已发表论文里这种词反而比机器草稿多（每千正文词 3.37 个，203 篇留出论文篇篇都有，秩 AUC 0.174）；collocation 轴每次运行都要加载 530,677 个词对的库，model-free 那一行延迟上升的 0.6 s 就是它。两者都不是检测器；`--no-collocation` 可以跳过词对库（[§23](docs/architecture/evaluation/vocabulary-and-residue.md)）。 | +| **两条词汇审计是建议，且要加载词对库** | 零命中审计列出语料从未写过的每一个词，而已发表论文里这种词反而比机器草稿多（每千正文词 3.37 个，203 篇留出论文篇篇都有，秩 AUC 0.174）；collocation 轴每次运行都要加载 530,504 个词对的库，model-free 那一行延迟上升的 0.6 s 就是它。两者都不是检测器；`--no-collocation` 可以跳过词对库（[§23](docs/architecture/evaluation/vocabulary-and-residue.md)）。 | | **全新 clone 什么都测不出来** | 全部 profile 制品都 gitignore。在你用自己的论文建出 profile 之前，每条语料参照的轴都是 `unmeasured`。 | ### 路线图 diff --git a/docs/README.md b/docs/README.md index 5a7c3aa..edb4568 100644 --- a/docs/README.md +++ b/docs/README.md @@ -61,6 +61,7 @@ universal paper PASS/FAIL. | Supplying a field corpus | [../style-corpus/README.md](../style-corpus/README.md) | | Generated profile assets and build boundaries | [../style-profile/README.md](../style-profile/README.md) | | Per-version history | [../CHANGELOG.md](../CHANGELOG.md) · [../CHANGELOG-ARCHIVE-v0.33-v0.34.md](../CHANGELOG-ARCHIVE-v0.33-v0.34.md) (v0.33.0-v0.34.0) · [../CHANGELOG-ARCHIVE-RECENT.md](../CHANGELOG-ARCHIVE-RECENT.md) (v0.27.1-v0.32.0) · [../CHANGELOG-ARCHIVE.md](../CHANGELOG-ARCHIVE.md) (v0.22.0-v0.27.0) · [../CHANGELOG-ARCHIVE-EARLY.md](../CHANGELOG-ARCHIVE-EARLY.md) (v0.1.0-v0.21.0) | +| Worked example: a synthetic manuscript before and after acting on the findings | [../examples/README.md](../examples/README.md) | | Working rules for this repository | [../CLAUDE.md](../CLAUDE.md) | | Adapted-material attribution and adoption boundaries | [../ACKNOWLEDGMENTS.md](../ACKNOWLEDGMENTS.md) | | External and full-repository review records (findings, first-party verification, disposition) | [audits/codex-review-2026-09-04.md](audits/codex-review-2026-09-04.md) · [audits/full-review-2026-09-27.md](audits/full-review-2026-09-27.md) | @@ -75,7 +76,8 @@ accumulating: - each file under `design-notes/` declares itself a design note in its header; - no document reappears at a location it was moved away from; - every suite size recorded anywhere in the evidence record or in either README matches actual test discovery; -- no tracked source or document exceeds 750 lines, the same cap the editing hook enforces. +- no tracked source or document exceeds 750 lines, the same cap the author's local editing hook enforces; +- every relative link, in-page anchor and numbered cross-reference in the READMEs and `docs/` resolves. One convention it cannot enforce lives in the suite instead. The figures these documents quote come from `style-profile//`, which is gitignored, so no diff --git a/docs/SCIPAPER_STANDARD.md b/docs/SCIPAPER_STANDARD.md index ae15588..0ae1d49 100644 --- a/docs/SCIPAPER_STANDARD.md +++ b/docs/SCIPAPER_STANDARD.md @@ -6,13 +6,13 @@ > implement this document. If a skill, > tool, style profile, or workflow conflicts with this file, this file wins. > -> Status: **v3.8 (2026-09-04)**. v3.8 adds the zero-hit audit to `L0.register`, -> the `L2.collocation` axis, three auxiliary structure families, the `L4.residue` -> axis with its diff gate, and a third mechanical enforcement point in §5.3 -> (the removal map and `--require-shrink`). No consequence class changed; one -> exit contract was added (§0.1, `deai_residue.py`). The disposition register is -> [`architecture/DISPOSITIONS.md`](architecture/DISPOSITIONS.md) and the -> version history [`../CHANGELOG.md`](../CHANGELOG.md). +> Status: **v3.9 (2026-09-27)**. v3.8 (2026-09-04) added the zero-hit audit, +> `L2.collocation`, three auxiliary structure families, `L4.residue` with its +> diff gate and the §5.3 removal map; v3.9 records the absence residue (§2 L4, +> v0.37.0), the collocation glossary reading (v0.38.0) and the fourth narrow +> exit contract (§0.1, `verify_references.py`). No consequence class changed. +> The disposition register is [`architecture/DISPOSITIONS.md`](architecture/DISPOSITIONS.md) +> and the version history [`../CHANGELOG.md`](../CHANGELOG.md). --- @@ -32,7 +32,7 @@ classes with different consequences: | Class | Meaning | Required consequence | |---|---|---| | `integrity_blocker` | The scientific record may be wrong, unsupported, internally inconsistent, unreproducible, or unusable | Must be resolved from sources. The author cannot waive it as a style preference. | -| `l0_target` | A preserved lexical de-AI target: Tier A, Tier B excess above its calibrated cap, or em-dash | Rewrite to zero applicable targets. This is a prose target, not a judgement that the paper is scientifically invalid. | +| `l0_target` | A preserved lexical de-AI target: Tier A, Tier B excess above its section cap (one occurrence per word per heading unit), or em-dash | Rewrite to zero applicable targets. This is a prose target, not a judgement that the paper is scientifically invalid. | | `advisory` | Calibrated L1-L4 or editorial evidence that a passage may be templated, distant from the field reference, unclear, or weaker than an available alternative | Rank, act on the strongest items, then record an author disposition for any residual. | Examples of `integrity_blocker` include incorrect mathematics or physics, @@ -57,7 +57,7 @@ reference. They must not claim that a machine wrote it. Standalone advisory tools return `0` when measurement completes, whether or not advisories are found. They return nonzero only for invalid input, -configuration failure, or execution failure. Three exceptions carry their own +configuration failure, or execution failure. Four exceptions carry their own narrow actionable contracts, and in each the meaning of `1` is a measured outcome the caller must act on, never a failure: @@ -69,7 +69,9 @@ outcome the caller must act on, never a failure: - `rewrite_reward.py` returns 0 when at least one candidate is eligible, 1 when every candidate fails scientific-fidelity or length-budget eligibility (the caller preserves the original and regenerates tighter, §6), and 2 for invalid - input or missing required configuration. + input; a missing learned model leaves the voice term unmeasured, never exit 2; +- `verify_references.py` returns 0 when no entry is a blocker, 1 when an identifier + cannot be resolved or a cited key has no entry, 2 for invalid input; an outage is `unmeasured`. Scientific-integrity review is broader than the linter and is reported through typed findings. It is never compressed into the linter's L0 exit status. @@ -268,14 +270,12 @@ text must not say that a low score proves machine authorship. A universal The per-paragraph learned classifier (`deai_voice`, axis `L3.voice`) is an **offline audit instrument, not a product detector one calibration away from an -operating point.** This is a decided status fixed by three measured facts: the -single-paragraph unit is near-unjudgeable (perceptual AUC at chance, EVALUATION -§10); on field-topic text the classifier mis-flags at a high rate while overall -separation stays high (a level confound the paragraph unit cannot escape, -EVALUATION §7); and document-level surprisal dispersion is weaker than the -model-free manifold and adds nothing to it (EVALUATION §9.8). L3 therefore stays -`degraded` with no operating point, emits rank-only triage, and its per-paragraph -findings are confidence-capped (§3, `calibration_unit`). +operating point**: a decided status, fixed by the three measured facts recorded +in EVALUATION §10, §7 and §9.8 (the paragraph unit is near-unjudgeable, the +field-topic level confound cannot be escaped at that unit, and document-level +surprisal adds nothing to the model-free manifold). L3 therefore stays `degraded` +with no operating point, emits rank-only triage, and its per-paragraph findings +are confidence-capped (§3, `calibration_unit`). ### L4: positive scientific voice @@ -509,7 +509,7 @@ confound-free self-checks. argument demands it* (role coupling), never at random. Length is handled by the per-stratum plus conformal calibration, never by normalizing a distance. -5. **L4 anchoring and voice.** Anchor unanchored Results and Methods claims to +5. **L2 anchoring and L4 voice.** Anchor unanchored Results and Methods claims to numbers, citations, references, or comparisons (a writing-quality gain, not an AI verdict); strengthen the specific claim, the stance, and faithful compression using author exemplars. Every added fact is source-traceable. diff --git a/docs/architecture/DEAI_SUBSYSTEM.md b/docs/architecture/DEAI_SUBSYSTEM.md index 88ed2d3..0774798 100644 --- a/docs/architecture/DEAI_SUBSYSTEM.md +++ b/docs/architecture/DEAI_SUBSYSTEM.md @@ -120,27 +120,30 @@ macro-bound vocabulary entirely. Findings are always advisories. The same tool carries the **zero-hit audit** (`register-zero:`), which is exhaustive where the threshold rule is selective: every body word with corpus document frequency 0 is listed, once, with its sentence. Both sides of the -comparison use one projection — preamble, bibliography, comments and section -headings excluded from the manuscript exactly as they are absent from the -passage banks (`body_only`, built on `extract_sections.RE_HEADING_COMMAND`) — -because a word present on one side only is a projection defect, not a finding. -Exemptions are mechanical and few: a formation of an attested stem -(`clamped`/`clamp`, `holdouts`/`holdout`) is ordinary rather than strong; a word -the paper defines or a cited method's name is the author's disposition, not the -tool's. The audit is not a detector — refereed papers carry more zero-hit words +comparison use one projection — preamble, bibliography, comments, floats and +section headings excluded from the manuscript exactly as they are absent from +the passage banks (`body_only`, built on `extract_sections.RE_HEADING_COMMAND`) +— because a word present on one side only is a projection defect, not a finding. +Exemptions are mechanical and three: a formation of an attested stem +(`clamped`/`clamp`, `holdouts`/`holdout`), a defining sentence or acronym +expansion at first use, and a proper name capitalised at every use are ordinary +rather than strong; a cited method's own name is the author's disposition, not +the tool's. The audit is not a detector — refereed papers carry more zero-hit words than machine drafts (EVALUATION §23.1) — and ships as advice. ### L1: information distribution [`../tools/deai_metrics.py`](../../tools/deai_metrics.py) measures section-aware sentence-length variation and paragraph-opening connective density. A compatibility -heuristic may produce degraded evidence. A strong advisory requires an applicable -calibrated policy and sufficient reference support. +heuristic may produce degraded evidence. A strong advisory would require an +applicable calibrated policy; none is written, because the operating point was +refuted by measurement (EVALUATION §16), so the axis stays `degraded`. [`../tools/deai_oracle.py`](../../tools/deai_oracle.py) optionally measures token surprisal and Uniform Information Density features. Missing model assets leave the -axis unmeasured. The compatibility `FLAG_Z` remains degraded until field calibration -provides an operating point with provenance and uncertainty. +axis unmeasured. The compatibility `FLAG_Z` is `degraded` by decision: no field +operating point is written, and the document-level path was measured not to add +power (EVALUATION §9.8, DISPOSITIONS). ### L2: sentence and paragraph construction @@ -155,7 +158,7 @@ structural patterns that keyword replacement cannot repair: - repeated paragraph templates. An auxiliary class — antithesis clusters, short reversal beats, paper-as-agent -subjects ("This paper presents"), wh-cleft openers ("What matters is") and +subjects ("This Letter asks whether"), wh-cleft openers ("What matters is") and modifier stacks (a run of three or more tokens before a head noun carrying at least two hyphenated compounds) — is reported under `structure-auxiliary` and never enters `template_score`, so the calibrated manifold does not move when a @@ -185,9 +188,9 @@ coarse grid or a lower-edge reading swallows exactly the passages the axis exist to find. Where a reference has no spread above the advisory gate the feature abstains rather than reporting an ordinary passage as the 100th percentile. -This detector is the sole consumer of +This detector is the only one that reads [`extract_style.latex_to_numeral_text`](../../tools/extract_style.py), the second -LaTeX projection. `latex_to_plain` replaces every math span with `[math]`, which +LaTeX projection (the bank writer runs it too, to store `numeral_text`). `latex_to_plain` replaces every math span with `[math]`, which is right for lexical and shape statistics and zeroes every numeral signal on `.tex` input; the numeral-preserving projection shares the same pattern set and differs only in what happens inside an inline math span. Displayed equations are @@ -203,17 +206,19 @@ manuscript did, and the p90 gate fired at 0.45 per held-out passage against a [`../tools/deai_collocation.py`](../../tools/deai_collocation.py) asks a question the register axis cannot: whether a sentence joins words the field never joins. Each word may be ordinary — `physical` and `cells` both are — and the pair still -one no passage of 41,710 has written. The unit is the sentence, the feature the +one no passage of 41,644 has written. The unit is the sentence, the feature the fraction of its distinct adjacent content-word pairs that the bank does not attest, and the reference is leave-one-out per bucket: at calibration a pair the bank saw in exactly one passage is treated as that passage's own, so the reference distribution is what a held-out sentence would see. Only common words are judged as partners (`COMMON_RATE`, unigram df ≥ 2 × 10⁻⁴), because a rare word's pairs are rare for the reason the register axis already reports; a pair -breaks at punctuation, a `[math]`/`[CITE]` placeholder or a dash, because -`yields, separate` is two clauses. Each flagged pair carries its expected -co-occurrence λ = df(a)·df(b)/N and e^−λ, so a reader can see that `measurements -projected` (λ = 183) is a stronger absence than `sub-halo abundances` (λ = 0.3). +breaks at punctuation, a `[math]`/`[CITE]` placeholder, a dash, a slash, a +period or a digit, because `yields, separate` is two clauses. Each flagged pair +carries its expected passage co-presence λ = df(a)·df(b)/N and e^−λ +(`expected_copresent_passages`, `p_copresence_absent`; a ranking aid, never a +filter), so a reader can see that `measurements projected` (λ = 183) is a +stronger absence than `sub-halo abundances` (λ = 0.3). A document-level novel-pair fraction is reported as evidence and used by `eval_findings`; it is not a percentile, because the reference is per sentence. @@ -274,7 +279,7 @@ length stratum (`fit_dispersion_manifold`, `manifold_distance`), role-coupled di Document calibration records one observation per verified complete paper, bootstrap uncertainty, leave-one-document-out human flag behavior, and empirical percentiles. If the corpus does not contain enough complete and measurable papers, the axis is -`unmeasured`; for `wgl` it is `measured` on a 14-paper calibration (EVALUATION §9). The +`unmeasured`; for `wgl` it is `measured` on a complete-paper baseline (EVALUATION §9). The implementation must not synthesize a document baseline from paragraph exemplars. ### L3: learned field similarity @@ -288,7 +293,9 @@ field-similarity triage. A bundle without a documented calibrated operating point is degraded. Evaluation must separate source-paper groups and audit mathematical-placeholder density, jargon density, section type, and paragraph length. The mathematical-density -confound is unresolved until the evidence in `EVALUATION.md` demonstrates otherwise. +confound was audited (EVALUATION §7.1: matched-stratum AUC within 0.02 of +overall); the field-topic confound is decided not resolvable from this feature +set (§7.0a). The per-paragraph learned classifier is an **offline audit instrument, not a product detector one calibration away from an operating point**: the paragraph @@ -327,7 +334,8 @@ prose an edit left. The history and absence families are defined once in the tool and mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` calls the tool's own `validator_check`, which proves the mirror and scans the shipped documentation for the edit-meta literals. A strong finding -exits 1, the third narrow exit contract after `length_gate` and `rewrite_reward`. +exits 1, one of the four narrow exit contracts the standard's §0.1 registers +(with `length_gate`, `rewrite_reward` and `verify_references`). [`../tools/condense_map.py`](../../tools/condense_map.py) is the measurement behind `/sci-paper:condense`: six scans (restatement with its canonical home, @@ -382,7 +390,9 @@ All active writing and review skills implement the same standard: `brainstorm` is pre-draft ideation; it loads the field dossier but is not a normative implementer of this standard (see `validate_plugin.py` -`NORMATIVE_SKILLS`). +`NORMATIVE_SKILLS`). The binding role of every skill — the `calibrate` setup +skill and the `physics` / `mainline` / `logic` measurement primitives included — +is the table in [`RESPONSIBILITIES.md`](RESPONSIBILITIES.md) §7. `CONFIRMED`, `REFUTED`, and `MARGINAL` in the escalation record describe whether a critique survived evidentiary verification. They do not select its consequence class. @@ -411,7 +421,9 @@ A field profile may contain: - L1/L2 calibrated reference distributions; - a complete-document structure baseline; - learned-model bundles and operating points; -- `deai_policy.json` with provenance, sample unit, uncertainty, and applicability. +- (`deai_policy.json`, once planned here with provenance, sample unit, uncertainty + and applicability, is withdrawn: no tool writes it and its operating points are + refuted, EVALUATION §16; its consumers report `degraded` without it). A policy asset must state what was measured, the independent unit, the corpus selection rule, sample size, uncertainty method, and validation behavior. If any @@ -437,7 +449,9 @@ standard. - normative/evaluation document authority boundaries; - required tests and CI wiring; - the residue contract: the history families the paper skill quotes match the - tool's, and no shipped document carries an edit-meta literal. + tool's, and no shipped document carries an edit-meta literal; +- every tracked file within the 750-line budget; +- relative links, in-page anchors and numbered cross-references resolve. The authoritative check list is `validate_plugin.py` itself (`tools/README.md` mirrors it); this summary is descriptive. diff --git a/docs/architecture/DISPOSITIONS.md b/docs/architecture/DISPOSITIONS.md index 6eae281..25c3e16 100644 --- a/docs/architecture/DISPOSITIONS.md +++ b/docs/architecture/DISPOSITIONS.md @@ -42,9 +42,9 @@ audit and keeping the suite and validator green, and updates this table and | Enlarging the conformal calibration split | **Rejected** | Tested because the achievable p-grid at n_cal = 78 makes α = 0.05 a ~96.2nd-percentile cut. Moving documents from training to calibration (0.6 → 0.3) leaves tail power flat and raises the human false-flag rate monotonically 0.030 → 0.038 (EVALUATION §9.4c). | | Baseline unification into one `(feature, unit)` object (rank 2) | **Done (v0.33.0)** | It stopped being elegance debt when a second per-bucket axis arrived: `deai_salience` and `deai_discourse` would otherwise have carried two copies of the quantile grid, the plateau-top percentile reader, the passage sweep and the calibration loop, and a percentile would have meant two different things depending on which tool computed it. `tools/deai_reference.py` now owns all of it, plus the `(feature, unit)` invariant itself — which immediately earned its keep: it is what caught the hedging axis calibrating at a unit that has no lower tail (EVALUATION §19.1). The artifact reader had five copies before this module existed. | | Jargon-conditional per-paragraph operating point (rank 3) | **Won't pursue as scoped** | The jargon confound is handled at document scale by per-stratum + conformal; a per-paragraph operating point is inconsistent with the L3-degraded decision. | -| `corpus_cos` ablation (rank 4) | **Won't pursue — cannot change a shipped decision** | `confound_audit` bins on record metadata the feature cache does not carry, so an ablation runnable from the cache computes a different statistic from the three recorded retrains (§7.0a) and cannot be compared with them. Its only consumer is a `degraded`, audit-only classifier with no shipped operating point, whose status rests on independently refuted grounds, so no result could change shipped behaviour (EVALUATION §18.7). | +| `corpus_cos` ablation (rank 4) | **Won't pursue — cannot change a shipped decision** | `confound_audit` bins on record metadata the feature cache does not carry, so an ablation runnable from the cache computes a different statistic from the four recorded retrains (§7.0a) and cannot be compared with them. Its only consumer is a `degraded`, audit-only classifier with no shipped operating point, whose status rests on independently refuted grounds, so no result could change shipped behaviour (EVALUATION §18.7). | | Citation placement as an axis | **Refuted (v0.33.0) — on its own pre-registered condition** | v0.32.0 measured it as the strongest model-free discriminator in the record (section-matched AUC 0.866 in `method`, surviving the section, length and human-vs-human controls) and declined to ship it for one stated reason: all 173 machine documents came from one generation process, so one bank could not separate "AI cites more" from "these prompts made it cite more". Two new 20-document banks from a **different model** (Codex `gpt-5.6-terra`), differing in one prompt line, answer it: with no citation instruction the same statistic scores **0.053** — separation of nearly equal strength in the opposite direction — and with one it scores **0.734**. Density swings 12.5× (1.00 to 12.55 per 1,000 words) around a human median of 6.20, so the machine extremes bracket the human distribution instead of sitting on one side of it. Same failure mode as §15.1, §16 and §18.4: a statistic whose sign depends on which machine population you sampled is not a discriminator, whatever its AUC on the sample you have (EVALUATION §20). | -| Topic-orthogonal axes: cohesion, hedging (rank 6) | **Shipped (v0.33.0), one of them restricted** | `tools/deai_discourse.py`, both flagging the LOW tail at p10 — the defect here is absence, not excess. Cohesion ships at paragraph unit across all seven buckets: held-out transfer 10.87% against a 10% design point, worst-of-six-regimes AUC 0.676 in `intro` against a 0.515 null. Hedging has **no paragraph-scale lower tail** (p10 is 0.000 in all seven buckets, because a 40-word paragraph that hedges nowhere is ordinary), so it calibrates and detects at SECTION unit, and ships restricted to `intro` — two independent measurements agree on that boundary: held-out transfer 7.89% there against 15.48–26.77% elsewhere, and worst-of-six AUC 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376 (EVALUATION §19). | +| Topic-orthogonal axes: cohesion, hedging (rank 6) | **Shipped (v0.33.0), one of them restricted** | `tools/deai_discourse.py`, both flagging the LOW tail at p10 — the defect here is absence, not excess. Cohesion ships at paragraph unit across all seven buckets: held-out transfer 10.78% on the v0.36.3 profile (10.87% at v0.33.0) against a 10% design point, worst-of-six-regimes AUC 0.676 in `intro` against a 0.515 null. Hedging has **no paragraph-scale lower tail** (p10 is 0.000 in all seven buckets, because a 40-word paragraph that hedges nowhere is ordinary), so it calibrates and detects at SECTION unit, and ships restricted to `intro` — two independent measurements agree on that boundary: held-out transfer 7.80% there on the v0.36.3 profile (7.89% at v0.33.0) against 15.48–26.77% elsewhere, and worst-of-six AUC 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376 (EVALUATION §19). | | Long-form generation (`ai_long`) | **Standing falsification target — robustly measured** | A recorded measured limit (EVALUATION §9.7). No longer a single-configuration result: tail power holds at 0.000 across two metrics, four calibration splits and 12 seeds, while rank AUC is 0.729 — the signal is present and the operating point cannot reach it (EVALUATION §9.4c). | | Single-seed tail-power figures as estimates | **Rejected as a reporting practice** | Per-seed spread on the manifold tiers is 0.04–0.18, wider than several differences the record previously read as improvements. §9 figures now carry the spread, and `tools/eval_docscale.py` reproduces the table rather than it being quoted (EVALUATION §9.4c). | | Hand labels as the only route to register/salience precision | **Superseded — half of it never needed a labeller** | A refereed ApJ/ApJL/A&A paper's provenance is already a label for "does this axis fire on accepted prose". 203 held-out papers, verified disjoint from all three calibration banks, now measure both axes (EVALUATION §17). `label_findings.py` keeps the remaining half: whether an individual advisory is good advice. | @@ -54,10 +54,11 @@ audit and keeping the suite and validator green, and updates this table and | Asymmetric projection, second instance: citation commands | **Fixed — 46 command names, 8,835 leaking uses** | `RE_TEX_CITE` listed four names and required the brace to follow the name directly, so `\citep[e.g.][]{Smith2020}` and six other natbib forms fell through to `RE_TEX_SIMPLE_CMD`, which substitutes a command's argument as text: the bibliography key became a word. 64 of 887 held-out register findings (7.2%) were leaked surnames, and 7.00% of every digit the salience axis read on LaTeX was a citation year. Matched by shape rather than by an allowlist, because the tail is per-paper local macros (EVALUATION §18.1–18.2). | | Held-out bundles collectable as calibration input | **Fixed — a structural guard, not a caller convention** | `corpus_documents` walks with `rglob` and filtered nothing, so the documented `--calibrate --corpus-dir style-corpus/wgl` collected 717 documents where the shipped baseline had 517 — the extra 200 being the whole evaluation set, absorbed silently. The existing held-out test pinned `extract_style`'s source tuple, which this path never consults. The guard now lives in the collector so every caller inherits it (EVALUATION §18.3). | | `L0.register` operating point | **Closed by refuting its premise** | Swept 5 → 50 uses against 203 held-out refereed papers and 173 machine documents: rank AUC is below 0.5 at **every** setting, so the axis fires more on refereed prose than on machine prose everywhere on the curve and tightening silences the machine side faster. No setting makes it a detector; the knob buys advisory volume. Cut at 15, the first point where a referee-grade paper is not flagged more often than not: 0.0858 per 1,000 words on 44.83% of held-out documents, AUC 0.2856 (EVALUATION §18.4). | -| `L0.register` threshold as the vocabulary instrument | **Superseded by the zero-hit audit (v0.36.0)** | The 15-use floor kept the axis from flagging refereed prose more often than not, at the cost of missing every word a draft uses once. The owner's rule replaces the knob: every body word with corpus df 0 is listed, and the only exemptions are mechanical (a formation of an attested stem, a defined term, a cited method's name). Measured on 203 held-out refereed papers against 173 machine documents the audit fires on every refereed paper (2.21 words per 1,000 under v0.36.1's float projection, 2.66 before it) and has rank AUC 0.246 — it is not a detector and is not shipped as one; it is exhaustive advice whose dispositions are the author's (EVALUATION §23.1). | +| `L0.register` threshold as the vocabulary instrument | **Complemented by the zero-hit audit (v0.36.0); the 15-use rule stays as the selective instrument** | The 15-use floor kept the axis from flagging refereed prose more often than not, at the cost of missing every word a draft uses once. The owner's rule sits beside the knob: every body word with corpus df 0 is listed, and the only exemptions are mechanical (a formation of an attested stem, a term defined here at first use, a proper name capitalised at every use; a cited method's own name is an author disposition). Measured on 203 held-out refereed papers against 173 machine documents the audit fires on every refereed paper (3.37 words per 1,000 body prose words under v0.36.2's denominator; 2.21 per raw-source word under v0.36.1's float projection, 2.66 before it) and has rank AUC 0.174 (0.246 before the denominator fix) — it is not a detector and is not shipped as one; it is exhaustive advice whose dispositions are the author's (EVALUATION §23.1). | | Asymmetric projection, third instance: headings | **Fixed — one regex owner** | Section headings sat in the manuscript's body projection but never in a corpus passage, so `Validation` fused with the first sentence under it and read as a zero-hit word; the same fusion put heading words into collocation pairs. `extract_sections.RE_HEADING_COMMAND` is now the single owner and `deai_reference.units`/`sections`, `deai_register.body_only` and `length_gate` all consume it (EVALUATION §23.1). | -| Collocation as an axis (`L2.collocation`) | **Shipped (v0.36.0), `measured`; glossary reading v0.38.0** | Per-sentence fraction of adjacent common-word pairs unattested in 41,710 passages, leave-one-out reference per bucket at sentence unit; distinct pairs, breaks at punctuation and placeholders, 11,286 admitted partner words. Flags five of the six mentor-marked phrases present in the private Letter and separates machine documents from held-out refereed papers on the document fraction (EVALUATION §23.2). | -| Residue axis (`L4.residue`) with a diff gate | **Shipped (v0.36.0), strengths set on refereed prose** | Three static rules and one diff rule. Strengths were measured, not assumed: on 203 held-out papers the first families put a strong finding in 154 documents; the body projection (preamble macros and "Planck Collaboration XXX" are not prose), dropping `used to` (174 instrumental hits) and demoting `initially`/`originally`/`at first` to ordinary bring that to 34 (16.7%), and v0.36.2 — marks read across line breaks, `we have added` a mark only with a document object (fourteen procedural hits otherwise) — to 27 (13.3%), 8 of them `\textcolor{red}` in tables. The static negative-label rule is ordinary (26% of refereed papers); the diff rule gates. The skill mirrors the family lists and the validator proves the mirror. Exit 1 on a strong finding is the plugin's third narrow exit contract (EVALUATION §23.4). | +| Collocation as an axis (`L2.collocation`) | **Shipped (v0.36.0), `measured`; glossary reading v0.38.0** | Per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (v0.36.3 rebuild; 41,710 before), leave-one-out reference per bucket at sentence unit; distinct pairs, breaks at punctuation, placeholders, dashes, slashes, periods and digits, 11,282 admitted partner words (11,286 before). Flags five of the six mentor-marked phrases present in the private Letter and separates machine documents from held-out refereed papers on the document fraction (EVALUATION §23.2). | +| Residue axis (`L4.residue`) with a diff gate | **Shipped (v0.36.0), strengths set on refereed prose; absence rule added v0.37.0** | Four static rules (self-history, edit-meta, negative-label, and `residue-absence`, a body sentence defining the paper's own object by what it never does, §23.4a) and one diff rule. Strengths were measured, not assumed: on 203 held-out papers the first families put a strong finding in 154 documents; the body projection (preamble macros and "Planck Collaboration XXX" are not prose), dropping `used to` (174 instrumental hits) and demoting `initially`/`originally`/`at first` to ordinary bring that to 34 (16.7%), and v0.36.2 — marks read across line breaks, `we have added` a mark only with a document object (fourteen procedural hits otherwise) — to 27 (13.3%), 8 of them `\textcolor{red}` in tables. The static negative-label rule is ordinary (26% of refereed papers); the diff rule gates. The skill mirrors the family lists and the validator proves the mirror. Exit 1 on a strong finding is the plugin's third narrow exit contract (EVALUATION §23.4). | | Structure families from the mentor's comments | **Shipped as auxiliary (v0.36.0)** | Paper-as-agent, wh-cleft and modifier-stack join the auxiliary class: named on the sentence, excluded from `template_score`, baseline fractions recorded per bucket (EVALUATION §23.3). | -| Condensation as a read-and-trim pass | **Replaced by the removal map (v0.36.0)** | The skill's own sweep removed a few percent by hand; `condense_map.py` lists every entry with the words it frees and `length_gate.py --require-shrink` turns the map's target into an exit code. Held-out refereed papers carry a median default target of 3.1% of prose, so a pass that cuts less than that has not found what the map lists (EVALUATION §23.5). | +| Condensation as a read-and-trim pass | **Replaced by the removal map (v0.36.0)** | The skill's own sweep removed a few percent by hand; `condense_map.py` lists every entry with the words it frees and `length_gate.py --require-shrink` turns the map's target into an exit code. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%; 3.1% before v0.36.2 counted each unit once), so a pass that cuts less than that has not found what the map lists (EVALUATION §23.5). | +| Spaced units outside the unit vocabulary (rewrite fidelity, §8) | **Accepted limit** | A unit written with an ASCII space and absent from the vocabulary (`5 furlongs`) loses unit-level protection; its number stays a protected invariant, and `rewrite_reward` gates eligibility on it without emitting a finding. Recorded here so EVALUATION §8 no longer carries it as an open item. | | Enlarging `wgl-letter` to resolve the gate | **Rejected — format is not a domain** | Impractical (ApJL is 24 papers in 5,364 arXiv records, so 10,000 passages needs ~90,000 fetched) and wrong: the register axis measures domain vocabulary and a letter is a format. On 36 letter-format documents the 706-passage letter bank produced 262 findings the field bank did not — `sne`, `bao`, `pantheon`, `posteriors`, and `letter` itself — against 2 the other way. A `-` profile now judges against `` and names the borrowed bank in every finding (EVALUATION §18.5). | diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index 57539f7..71bd9bb 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -114,18 +114,18 @@ result. | L0 lexical/punctuation | measured | Deterministic Tier A, em-dash, and Tier B cap implementation with CLI regression tests. | Continue regression coverage when policy changes. | | L1 distribution | degraded (refuted) | §16: measured on 500 human papers against 173 `docval` AI documents, one observation per document. Burstiness reverses sign — adversarial prose is *more* bursty than the human median (1.036 vs 0.775, AUC 0.181) and long-form sits inside the human band (AUC 0.441) — while flagging 7.2% of humans. Signposting has an AUC of 0.247, below chance, and flags 0 of 173 AI documents at the shipped default. | None. `deai_policy.json` is withdrawn as a roadmap item: the statistics do not support an operating point, so the axis stays advisory by measurement rather than by absence. | | L1 UID | degraded | `style-profile/wgl/uid_baseline.json` records paragraph-level GPT-2-large summaries. | A documented operating point and human false-flag behavior; audit sensitivity to mathematics and jargon. | -| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2775 per passage on 203 held-out ApJ/ApJL/A&A papers against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nfields}{63}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | +| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2705 per passage on 203 held-out ApJ/ApJL/A&A papers (§17.5's headline; 0.2776 when re-run in the v0.36.2 process) against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nfields}{63}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | | L0 register | measured | §14: document frequency over 41,644 corpus passages, 53,367 terms (v0.36.3 profile rebuild; 41,710 and 53,414 before it, and the 57 held-out findings below are the same 57 under both); compound-by-rarest-part and macro-subscript handling; native-term controls pass. Since v0.32.0 a `-` profile whose own bank cannot resolve the 1e-4 gate judges against `` and names the borrowed bank in every finding: on 36 letter-format documents the 706-passage `wgl-letter` bank produced 262 findings the field bank did not (`sne`, `bao`, `pantheon`, and `letter` itself) against 2 the other way (§18.5). | Recall below the 15-use floor, and a **measured** false-positive rate: §18.4 put it at 0.0858 findings per 1,000 words on 203 held-out refereed papers (44.83% of documents), rank AUC 0.2856 against machine text — it still fires more on human prose than on AI drafts — and §23.1 re-measures the same papers at 0.0351 (30.05%, AUC 0.352) after the heading projection fix and at 0.0247 (22.2%, AUC 0.392) after v0.36.1's float projection fix, same direction; v0.36.2 counts per body prose word instead of per raw-source word, and the same 57 findings read 0.0371 (AUC 0.391). §17.3, re-measured: 94.44% of the 198 remaining flags would be suppressed by the paper's own bank membership (98.2% of the 57 under v0.36.1). The operating point is no longer an open item: swept 5→50 uses, rank AUC stays below 0.5 everywhere, so no setting makes this a detector and the knob buys advisory volume only. It was cut at 15, the first point where a referee-grade paper is not flagged more often than not (§18.4). §21 replicates that refutation on a second population — 22 papers by one author, 1996–2015, sampled by author rather than by journal and spanning a different two decades — where rank AUC is **0.328**, again below 0.5 (0.412 on the v0.36.3 profile rebuild, 5 findings on the 22 papers). | | L2 sentence structure | measured for deterministic matches; degraded for strength | `style-profile/wgl/structure_baseline.json` provides section-level reference fractions over 27,841 paragraphs — method 9,478; results 3,978; data 3,894; intro 3,812; discussion 3,635; conclusion 2,611; abstract 433. §23.3 adds the per-bucket human fractions of the three auxiliary families taken from a mentor's comments (paper-as-agent 0.13–1.00%, wh-cleft 0.00–0.23%, modifier stacks 2.5–15.9%). | Author-labelled difficult cases. A calibrated strong-advisory threshold is no longer expected from `deai_policy.json` (§16). | | L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (530,504 pairs over 11,282 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,391; method 27,207; intro 14,321; results 12,509; data 12,048; discussion 12,183; conclusion 8,588 after the v0.36.3 profile rebuild; 41,710 passages and 541,309 → 530,677 pairs under v0.36.2); p90 gates 0.44–0.59, p95 0.50–0.67. On the first measured draft of the private Letter it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at AUC 0.704 (7,237 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same Letter: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). | Whether an individual advisory is good advice; a second generation process (§20's lesson). | -| L4 residue | measured | §23.4: three static rules and one diff rule, deterministic, strengths set on 203 held-out refereed papers (70,848 body prose sentences): strong self-history fires in 20 papers, strong edit-meta in 8 (68 of 69 are `\textcolor{red}` in tables), any strong in 27 (13.3%); the static negative-label rule is ordinary at 26% of papers. Four passes to get there, the first a projection asymmetry and the last the procedural sense of `we have added`. | Recall against a labelled edit history; only the author's own drafts can supply it. | +| L4 residue | measured | §23.4: four static rules (self-history, edit-meta, negative-label, and since v0.37.0 the absence rule, §23.4a) and one diff rule, deterministic, strengths set on 203 held-out refereed papers (70,848 body prose sentences): strong self-history fires in 20 papers, strong edit-meta in 8 (68 of 69 are `\textcolor{red}` in tables), any strong in 27 (13.3%); the static negative-label rule is ordinary at 26% of papers. Four passes to get there, the first a projection asymmetry and the last the procedural sense of `we have added`. | Recall against a labelled edit history; only the author's own drafts can supply it. | | L0 register zero-hit audit | measured, not a detector | §23.1: every body word with corpus df 0, on both sides of one projection (headings now excluded on both). All 203 held-out refereed papers carry zero-hit words, 3.37 per 1,000 body prose words (2.21 per raw-source word under v0.36.1, 2.66 before its float fix), against 1.05 for the 173 machine documents — rank AUC 0.174, the audit fires more on human prose — so it ships as exhaustive advice with author dispositions, never as a gate. The two projection fixes moved the thresholded rule from 0.0858 to 0.0247 findings per 1,000 raw-source words on the same papers (0.0371 per body prose word). | None as a detector. Whether the mechanical exemptions (attested stem, defined term, proper name) are the right three. | -| L2 document structure | measured | §9: cross-paragraph dispersion calibrated one-observation-per-paper over 493 complete human `wgl` papers; human false-flag at the shipped conformal operating point 0.0325 (manifold) and 0.0426 (role) against nominal α = 0.05. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. | Continue recalibration when the corpus changes. | -| L3 learned field similarity | degraded (confound-audited) | Confound-aware audit complete (§7): repeated grouped-split AUC 0.932, matched-stratum AUC 0.924, hard-set true-provenance AUC 0.937, but 32–41% false-positive rate on field-topic AI text. Document-level now measured (§9.8): surprisal dispersion (0.757) is weaker than the model-free manifold (0.881) and adds nothing to it, so L3 stays degraded for a measured reason. | A field-topic-robust operating point with provenance and uncertainty; the surprisal path is measured not to provide one. | -| L2 cohesion | measured | §19: given/new linkage per paragraph against the field's own bank (abstract 13,967; method 6,903; intro 3,252; results 3,183; data 2,992; discussion 2,932; conclusion 1,975), flagged on the LOW tail at p10. The gate transfers to 203 held-out refereed papers at 10.87% against a 10% design point, in every bucket (6.58%–14.63%), and separates all six machine regimes in `intro` at worst-of-six 0.676 against a 0.515 human-vs-human null. | Whether an individual advisory is good advice; recall. Whether the `intro` separation holds for a second generation process — §20 is what happens when that question is not asked. | -| L2 hedging | measured, restricted to `intro` | §19: epistemic markers per 1,000 words per SECTION (abstract 10,404; intro 502; method 438; conclusion 382; discussion 327; results 316; data 299). It has no paragraph-scale lower tail — p10 is 0.000 in all seven buckets — so calibration and detection both run at section unit, and each artifact records its own. Restricted to `intro` by two independent measurements that agree: held-out transfer is 7.89% there against 15.48–26.77% elsewhere, and worst-of-six AUC is 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376. | Whether the restriction generalizes beyond `wgl`. On `wgl-letter` the axis is `degraded`: no bucket clears the 30-unit floor after the restriction. | +| L2 document structure | measured | §9: cross-paragraph dispersion calibrated one-observation-per-paper over 493 complete human `wgl` papers — the 2026-08-17 scoring; the v0.36.3 rebuild (2026-09-05) re-fit the artifact on 507 documents and the operating-point figures were not re-scored on it — human false-flag at the shipped conformal operating point 0.0325 (manifold) and 0.0426 (role) against nominal α = 0.05. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. | Re-score §9 through `eval_docscale.py` on the 507-document artifact; continue recalibration when the corpus changes. +| L3 learned field similarity | degraded (confound-audited) | Confound-aware audit complete (§7): repeated grouped-split AUC 0.9487, matched-stratum AUC 0.9262, hard-set true-provenance AUC 0.938 (2026-09-05 retrain), but a 30–42% false-positive rate on field-topic AI text. Document-level now measured (§9.8): surprisal dispersion (0.757) is weaker than the model-free manifold (0.881) and adds nothing to it, so L3 stays degraded for a measured reason. | None from this feature set: four retrains agree the confound is in the features (§7.0a), so reopening needs a different feature set; the surprisal path is measured not to provide one. | +| L2 cohesion | measured | §19: given/new linkage per paragraph against the field's own bank (abstract 13,967; method 6,886; intro 3,228; results 3,211; data 2,989; discussion 2,926; conclusion 1,980 after the v0.36.3 profile rebuild), flagged on the LOW tail at p10. The gate transfers to 203 held-out refereed papers at 10.78% (10.87% at v0.33.0) against a 10% design point, in every bucket (6.58%–14.61%), and separates all six machine regimes in `intro` at worst-of-six 0.676 against a 0.515 human-vs-human null. | Whether an individual advisory is good advice; recall. Whether the `intro` separation holds for a second generation process — §20 is what happens when that question is not asked. | +| L2 hedging | measured, restricted to `intro` | §19: epistemic markers per 1,000 words per SECTION (abstract 10,404; intro 502; method 438; conclusion 383; discussion 327; results 317; data 299 after the v0.36.3 profile rebuild). It has no paragraph-scale lower tail — p10 is 0.000 in all seven buckets — so calibration and detection both run at section unit, and each artifact records its own. Restricted to `intro` by two independent measurements that agree: held-out transfer is 7.80% there (7.89% at v0.33.0) against 15.48–26.77% elsewhere, and worst-of-six AUC is 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376. | Whether the restriction generalizes beyond `wgl`. On `wgl-letter` the axis is `degraded`: no bucket clears the 30-unit floor after the restriction. | | Citation placement | **refuted, not shipped** | §20: the v0.32.0 candidate (section-matched AUC 0.866 in `method`) does not hold its sign across generation processes. A second bank from a different model scores 0.053 with no citation instruction and 0.734 with one — one prompt line, a 12.5× density swing, and the two machine extremes bracket the human distribution rather than sitting on one side of it. | None. Reopening requires a statistic that holds its sign across independently produced banks. | -| Rewrite scientific fidelity | measured for protected invariants | Unit tests cover preserved invariants, dropped number, dropped citation, reversed comparison, display-math values and exponents (v0.27.0), and the punctuation/adjacent-word tokenizer boundaries (§8). | Real manuscript before/after demonstration, including scope and stance review; the plain-ASCII-space unit boundary in §8 remains open. | +| Rewrite scientific fidelity | measured for protected invariants | Unit tests cover preserved invariants, dropped number, dropped citation, reversed comparison, display-math values and exponents (v0.27.0), and the punctuation/adjacent-word tokenizer boundaries (§8). | Real manuscript before/after demonstration, including scope and stance review. The plain-ASCII-space unit boundary in §8 is an accepted limit ([DISPOSITIONS](DISPOSITIONS.md)): the number stays protected, the unit does not. | A missing baseline is not interpreted as zero findings. @@ -160,13 +160,6 @@ Current release gates (as of 2026-09-16; last tagged release v0.38.0): and as of v0.25.1 the hosted CI run on the release commit must also be green (first green runs: 31133202443 push, 31133215203 manual dispatch). -A v0.28.0-specific gate applies because this release changes what the corpus -layer reads: no figure may be carried forward on assertion. Every section-keyed -number here was re-read from the rebuilt artifact in the pass that wrote it, and -§9 — which was not regenerated by the rebuild — was re-scored through the -shipped path so its reproduction is a per-value comparison across eight AUCs -rather than a matching total. Where a figure moved, both values are printed. - Historical record — v0.14.0 release gates and their status on 2026-07-12: - validator and 36 unit/CLI tests after final edits — met (rerun before tag); @@ -181,6 +174,6 @@ Historical record — v0.14.0 release gates and their status on 2026-07-12: - commit, tag, push, and GitHub release — performed after the gates above are re-run green. -Author decisions that remain open and do not block the plugin release: accepting or -rejecting the §11 rewrite proposal, and whether to ever propose an L3 operating -point (the hard set says not yet). +Author decisions outside this record, which do not block a release: accepting or +rejecting the §11 rewrite proposal ([DISPOSITIONS](DISPOSITIONS.md)). The L3 +operating point is decided, not open: not obtainable from this feature set (§7.0a). diff --git a/docs/architecture/RESPONSIBILITIES.md b/docs/architecture/RESPONSIBILITIES.md index 1941243..f587b85 100644 --- a/docs/architecture/RESPONSIBILITIES.md +++ b/docs/architecture/RESPONSIBILITIES.md @@ -45,14 +45,14 @@ scope, not a standard-reference obligation. | Tool | Layer | Required behavior | |---|---|---| -| `ai_ism_lint.py` | L0 hub plus optional L1-L3 aggregation | Emit text or structured JSON from the same findings; use L0-only exit semantics. | +| `ai_ism_lint.py` | L0 hub plus the L1–L4 advisory axes (collocation and residue on by default; oracle, voice and the glossary reading opt-in) | Emit text or structured JSON from the same findings; use L0-only exit semantics. | | `deai_metrics.py` | L1 | Emit calibrated distribution findings and explicit missing-baseline status. | | `deai_oracle.py` | L1 | Emit surprisal/UID findings with observed and reference values; advisory-success exit 0. | | `deai_structure.py` | L2 sentence | Emit template evidence with calibration metadata; advisory-success exit 0. The auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) name the sentence under `structure-auxiliary` and never enter `template_score`. | | `deai_salience.py` | L2 salience | Measure recital structure against a per-section human passage reference at one shared unit; one finding per passage; abstain where the reference cannot resolve above the gate. Sole consumer of `extract_style.latex_to_numeral_text`, the numeral-preserving projection (`latex_to_plain` replaces math with `[math]`, which zeroes every numeral signal on `.tex` input). | -| `deai_register.py` | L0 register | Compare manuscript vocabulary against field-corpus document frequency; judge compounds by their rarest part; read macro bodies but not subscript decorations; emit advisories only, never `l0_target`s. The zero-hit audit (`register-zero:`) is exhaustive over body words on both sides of the same projection (headings, comments, preamble and bibliography excluded on both), strong unless the word is a mechanical formation of an attested stem. | +| `deai_register.py` | L0 register | Compare manuscript vocabulary against field-corpus document frequency; judge compounds by their rarest part; read macro bodies but not subscript decorations; emit advisories only, never `l0_target`s. The zero-hit audit (`register-zero:`) is exhaustive over body words on both sides of the same projection (headings, comments, preamble and bibliography excluded on both), strong unless the manuscript supplies a reason the tool can read: a mechanical formation of an attested stem, a defining sentence or acronym expansion at first use, or a proper name capitalised at every use (a cited method's own name is an author disposition, not an exemption). | | `deai_collocation.py` | L2 collocation | Judge, per sentence, the fraction of adjacent common-word pairs (unigram df at or above `COMMON_RATE`) that no corpus passage attests, against a leave-one-out per-section reference at sentence unit; a pair breaks at punctuation, a placeholder or a dash; carry each pair's chance-absence probability so a reader can weigh it; advisories only, and the action names the relation, never the claim. Read document-wide (`--glossary`), list every unattested pair used twice or more with its first-use line and definition cue, as the manuscript's own terms. | -| `deai_residue.py` | L4 residue | Report self-referential drafting history (first-person sentence, no citation), edit-meta literals, and a heading or caption whose object the body never names; with `--before`/`--git-ref`, the added-label diff rule; exit 1 on a strong finding (§0.1); keep the family lists in one place and let the validator prove the skills quote them. | +| `deai_residue.py` | L4 residue | Report self-referential drafting history (first-person sentence, no citation), a body sentence that defines the paper's own object by what it never does or has (`residue-absence`; `never` strong, a citation in the sentence exempts), edit-meta literals, and a heading or caption whose object the body never names; with `--before`/`--git-ref`, the added-label diff rule; exit 1 on a strong finding (§0.1); keep the family lists in one place and let the validator prove the skills quote them. | | `condense_map.py` | QD (§5.3) | Enumerate every removable entry in six scans, each with `removable_words` and, for a restatement, its canonical home; total a default target that excludes the abstract/conclusion carve-out; never delete anything, never emit a blocker, exit 0/2. | | `deai_discourse.py` | L2 cohesion + L2 hedging | Measure given/new linkage and epistemic-marker rate against the field's own distribution, each at its declared unit, and flag the LOW tail. Report one status per feature, never a joint one: they calibrate from the same corpus at different granularities, so a field can support one and not the other. Speak only for buckets whose operating point was shown to transfer. | | `deai_reference.py` | shared reference | Own the `(feature, unit)` percentile contract for every per-bucket axis -- one quantile grid, one sample floor, one plateau-top reader, one calibration loop -- and hold no policy of its own. | @@ -68,6 +68,8 @@ scope, not a standard-reference obligation. | `deai_personal.py` | L4 | Compare a draft to the author's own prior papers (confound-free reference); `unmeasured` below three papers. | | `eval_findings.py` | evidence | Score the register, salience, cohesion, hedging and collocation axes (the last on its document novel-pair fraction, because its reference is per sentence) on a **held-out** refereed corpus that fed no calibration bank, and report the same-genre in-sample population beside it so the leakage gap stays visible. Never present the salience rate as a false-positive rate — its gate is a percentile, so a non-zero rate is its design point — and never present either rate as precision for the advice itself. Added 2026-08-27: the axes' precision had stood at `unmeasured` pending hand labels, while a refereed paper's provenance was already a label for half of the question. | | `eval_docscale.py` | evidence | Reproduce the EVALUATION §9 table through the same `manifold_operating_point` that findings use — never a private scoring path. Label the human rate **in-sample** wherever it includes the manifold's own train and calibration documents, and never present a single-seed tail-power figure as an estimate. Added 2026-08-26: nothing bound the evidence-producing path before, and §9 accumulated headline figures produced by a 95th-percentile cut rather than by the shipped conformal rule. | +| `verify_references.py` | QD (dimension F) | Verify every bibliography entry against the registries (CrossRef, DataCite for the prefixes CrossRef does not register, arXiv) and, with `--tex`, the assembled document's `\cite` keys against the entries; an unresolvable identifier or a cited key with no entry is an `integrity_blocker` and exit 1 (§0.1); a registry outage marks the entry `unmeasured` and the axis `degraded`, never clean; relevance is the reviewer's, not measured here. | +| `label_findings.py` | evidence | Sample findings from every finding-emitting axis into a labelling sheet, re-serve a blind subset, and score precision, pooled recall and intra-rater agreement; any stratum under 20 labels is `unmeasured`; nothing here enters calibration or creates an operating point. | Compatibility tuple APIs may remain temporarily, but new orchestration consumes structured finding APIs. Adapters project from structured findings, not the diff --git a/examples/README.md b/examples/README.md index abf02e7..9726662 100644 --- a/examples/README.md +++ b/examples/README.md @@ -21,7 +21,7 @@ gitignored — so on a fresh clone both files report `unmeasured` for the corpus-referenced axes instead of the numbers below. Build one first with `python tools/build_profile.py --field ` and the `--calibrate` commands in `style-profile/README.md`. The figures here were produced against the `wgl` -profile on 2026-09-04 (v0.36.3) and will move with the corpus. +profile on 2026-09-05 (v0.36.3) and will move with the corpus. ## What it reports @@ -51,7 +51,7 @@ Per rule: Every measured value in the original survives in the revision, and the header of `sample-manuscript-revised.tex` lists them. Two of them are also *restated* where the argument now needs them: the headline 0.052 at a blending fraction of -0.40 appears in the abstract and again in the discussion. No value is invented +0.40 appears in the results and again in the abstract. No value is invented and none is dropped. The recital advisories were answered the way their action text asks — by saying what the quantities establish, so a passage stops being an uninterrupted run of numeral-bearing @@ -84,8 +84,8 @@ That density is the revision's own doing. The cohesion axis asks each sentence to reuse a content word from the sentence before it. In a results paragraph, the word available to carry forward is usually the one the numbers are about — here, `bias`. Repeating it pulls the subject into sentences that -also carry a numeral, and restating the headline pair where the discussion -needs it puts numerals into a passage that had none; both are exactly what +also carry a numeral, and restating the headline pair in the abstract puts +numerals into a passage that had none; both are exactly what `salience-recital` counts, the fraction of sentences bearing numerals and the longest uninterrupted run of them. diff --git a/skills/condense/SKILL.md b/skills/condense/SKILL.md index d56eb81..57c1343 100644 --- a/skills/condense/SKILL.md +++ b/skills/condense/SKILL.md @@ -75,9 +75,12 @@ condense's ranked sweep. 1. Read the target file completely. Resolve `--field` as in `/sci-paper:de-ai` §1 (shared style-profile convention). -2. **Snapshot the length baseline** before any edit: - `cp /length-baseline.tex` (or record the clean git ref). - Without an honest baseline the closing gate is meaningless. +2. **Snapshot the length baseline** before any edit. The baseline is the + whole document, not the root file: copy the include tree + (`cp -r /baseline/`, then gate against + `/baseline/.tex`) or record the clean git ref and use + `--git-ref`. A root copied alone cannot resolve its `\input` children, + reads as ~0 words, and every section then registers as growth. 3. Build the map: `python tools/condense_map.py --format json --output /condense-map.json`. Six scans, each entry with `removable_words`: @@ -92,7 +95,7 @@ condense's ranked sweep. (`may possibly`), each with the words its replacement saves. - `condense-regloss` — the same symbol glossed twice. - `condense-duplicate` — a paragraph repeated across sections (Jaccard - ≥ 0.60 over ≥ 30 content words). + ≥ 0.60 of content words, both paragraphs ≥ 30 prose words). `condense_budget` totals them: `removable_total`, `removable_by_rule`, and `default_target_words` = restatement + zero-gain outside the carve-out. The map is exhaustive for what it scans and blind to what it does not @@ -117,8 +120,10 @@ For each map entry, in rank order, record `deleted` / `merged` / `kept:` shortened rewrite with `python tools/rewrite_reward.py --field --reference --original --candidates ...` — a candidate that drops a protected invariant is ineligible regardless of its - brevity; among eligible candidates the shorter wins. `--field` is required - by the tool (it exits 2 without one); resolve it as in §1 step 1. + brevity; among eligible candidates the shorter wins. `--field` resolves as + in §1 step 1 (one profile auto-resolves); without a `voice_model.joblib` + the learned score is reported unmeasured and the deterministic gate still + decides eligibility. Exit 2 is invalid input, never a missing model. Apply each change with a minimal Edit and re-read the affected region plus every cross-reference into it. @@ -126,14 +131,15 @@ every cross-reference into it. ## 3. Verify — prove the shrink 1. **Length gate against the target:** - `python tools/length_gate.py --before /length-baseline.tex + `python tools/length_gate.py --before /baseline/.tex --require-shrink ` (a percentage such as `8%` or a - fraction is also accepted). Exit 0 requires both no unjustified growth and + fraction is also accepted; a target of 0 means the map found nothing to + cut and only the growth gate applies). Exit 0 requires both no unjustified growth and a net cut of at least the target; a cut short of it is a strong `length-shrink-short` finding with exit 1, and the pass may not close on it — either remove more, or record which kept entries account for the gap. 2. **Residue:** `python tools/deai_residue.py --before - /length-baseline.tex`. A condensation must not leave a heading or + /baseline/.tex`. A condensation must not leave a heading or caption promising what the body no longer says; exit 1 blocks closing. 3. **No orphans:** rebuild/compile the document if it is LaTeX (0 errors, 0 undefined references, no newly-missing labels); re-run the map and grep diff --git a/skills/de-ai/SKILL.md b/skills/de-ai/SKILL.md index 3062e5e..ceb3563 100644 --- a/skills/de-ai/SKILL.md +++ b/skills/de-ai/SKILL.md @@ -96,8 +96,10 @@ condense's ranked sweep. standards, canonical L0 examples). 3. Read `style-profile//style_dossier.md` in full when it exists, and check freshness: if the corpus under `style-corpus//` is newer than - the dossier, stop and regenerate with `python tools/extract_style.py`. A - stale profile is `unmeasured`, never treated as conformity. + the dossier, stop and rebuild with `/sci-paper:calibrate ` (§3–§4 + there: `extract_style --field ` alone rewrites the dossier and the + bank, and every `--calibrate` axis keeps describing the old corpus while + reporting `measured`). A stale profile is `unmeasured`, never conformity. 4. Retrieve section-typed positive anchors only when the bank exists: `python tools/retrieve_exemplars.py --field --section --topic "" --k 5`. Read every returned @@ -108,7 +110,7 @@ condense's ranked sweep. ```bash python tools/ai_ism_lint.py --field \ --structure --distribution --document-structure --register --salience \ - --oracle --voice --summary --format json --output /feedback-before.json + --oracle --voice --format json --output /feedback-before.json ``` Read the output as `sci-paper.feedback.v1`; record every axis state @@ -135,9 +137,12 @@ Axes and tools: or the concept the paper introduces. The same tool's **zero-hit audit** (`register-zero:`) is exhaustive rather than thresholded: every body word the manuscript uses that no corpus passage carries is listed, strong - unless the word is a mechanical formation of an attested stem (`clamped` from - `clamp`). A strong hit is answered by one of the §4.2 dispositions, never by - silence. + unless the manuscript supplies a reason the tool can read — a mechanical + formation of an attested stem (`clamped` from `clamp`), a defining sentence + or acronym expansion at first use, or a proper name capitalised at every + use. "The cited method's own name" is an author disposition, not an + exemption. A strong hit is answered by one of the §4.2 dispositions, never + by silence. - **L2 collocation** (`deai_collocation`) — sentences that join common words the field never joins (`physical cells`, `controlled grid`): the fraction of a sentence's adjacent content-word pairs that no corpus passage attests, against @@ -316,7 +321,10 @@ findings. For each selected paragraph: entities, inline math and acronyms, comparison/negation/causal direction, scope, stance, qualifiers. Re-read every numeric and citation source in the same turn. Snapshot the original verbatim (`/original.txt`) - as the §5.3 length baseline. + as the §5.3 length baseline. Before the FIRST paragraph of a pass, also + snapshot the whole document once (`/document-baseline.tex`, the + include tree, or the clean git ref): the loop-close gates of step 6 + compare against it. 1b. **Bind.** Fill the binding ledger (§4.3) for the paragraph. It is not optional and no detector emits it; every `none` row is a clause the rewrite must drop rather than carry forward. @@ -336,7 +344,10 @@ findings. For each selected paragraph: `eligible=False` (fidelity failure, or longer than the original → `-inf`) cannot be selected; the tool exits 1 when NO candidate is eligible (a measured outcome, not a failure — regenerate tighter) and 2 only on invalid - input or a missing profile; `--allow-growth ""` only with an + input. Without a `voice_model.joblib` the learned score is reported + unmeasured and the deterministic gate still decides; the set comparison of + invariants is blind to a negation MOVED between clauses, so step 3's + hand check is where that is caught. `--allow-growth ""` only with an author-approved justification, printed into the run record. Among eligible candidates rank by L0 reduction, semantic fidelity, voice score, and condensation. If none is eligible, preserve the original and @@ -353,6 +364,13 @@ findings. For each selected paragraph: findings, and dispositions; apply with a minimal Edit and re-read the changed region in context. Run the Pass-2 self-interrogation on the applied text. +6. **Close the loop (§5.3).** Before the pass stops, run + `python tools/length_gate.py --before /document-baseline.tex` + (or `--git-ref `) and `python tools/deai_residue.py --before + /document-baseline.tex`. A strong `length-growth`, + `length-shrink-short` or residue finding blocks the stop until it has a + disposition; the per-candidate `--original` budget of step 3 is the + candidate-time gate, not this one. ### 4.1 Ranking a recital passage (`salience-recital`) @@ -474,8 +492,10 @@ heavy run executes on the authorized compute environment, not locally. Stop only when: all integrity blockers are resolved or verified false positives; applicable L0 targets are zero; every strong advisory is `acted`, `accepted`, `rejected_as_false_positive`, or `pending` with a stated reason; -ordinary advisories and unavailable axes are reported; and every changed -paragraph passes scientific-fidelity verification. This is a +ordinary advisories and unavailable axes are reported; every changed +paragraph passes scientific-fidelity verification; and the Pass-3 step-6 +loop-close gates have run against the document snapshot with every strong +length or residue finding dispositioned. This is a disposition-complete feedback state, not a universal prose verdict. If the iteration budget is exhausted, leave the original text for unresolved cases and return the pending findings. @@ -522,8 +542,9 @@ No number, equation, or citation changed. Layer-4 failure, never as bare words. The enforced Tier A/B lists live in `tools/ai_ism_lint.py`; this skill defers to them. The corpus-derived `style-profile//lexicon.json` feeds the advisory `corpus-zero:` rule only. -When the corpus changes, re-run `tools/extract_style.py` and -`tools/build_profile.py`. +When the corpus changes, re-run `/sci-paper:calibrate ` (its §3–§4: +extraction plus every `--calibrate` axis); `extract_style` and +`build_profile` alone leave the calibrated references stale. **Interfaces:** - `docs/SCIPAPER_STANDARD.md` — consequence classes, ranking, disposition, diff --git a/skills/final-review/SKILL.md b/skills/final-review/SKILL.md index 6e2e82d..0c89405 100644 --- a/skills/final-review/SKILL.md +++ b/skills/final-review/SKILL.md @@ -99,6 +99,9 @@ Workflow states: 5. Initialize a finding registry keyed by stable finding ID plus semantic deduplication key `(rule, location, evidence)`. 6. Record target path, current revision/hash, build system and selected field. +7. Snapshot the target document before the first edit of round 1 — the whole include tree + (`cp -r`) under `/baseline/`, or the clean git ref — and again at the end of every + round, so the §5.3 loop-close gates of §3.7 always have an honest baseline. ## 3. Each review round @@ -107,16 +110,21 @@ Workflow states: In the parent context, invoke `/sci-paper:paper` and read the normative standard. Record their current versions/paths in `paper-baseline.md`. This provides policy, not a child verdict. +### 3.1b Verify the bibliography + +In the parent context, before launching the reviewer, run +`python tools/verify_references.py --tex --format json --output +/references.json`. An unresolvable identifier or a cited key with no entry is an +integrity blocker; unmeasured entries stay visible. Pass the report path into the reviewer's +prompt. + ### 3.2 Isolated paper-review Launch a worktree agent with a self-contained prompt: - cold-read the current target and all sources; -- read the reference report the parent produced for this round with - `python tools/verify_references.py --tex --format json --output - /references.json` (an unresolvable identifier or a cited key with no entry is an - integrity blocker; unmeasured entries stay visible) and judge relevance, which no - registry measures, for every entry; +- read `/references.json` (§3.1b) and judge relevance, which no registry measures, + for every entry; - invoke `/sci-paper:paper-review --orchestrated --field `; - do not spawn any child agent (M.2 escalation runs in-process); - return the complete typed report, including A–R coverage (dimension E narrative-spine @@ -219,6 +227,13 @@ For each action: - rerun the relevant build, scientific check, figure render, linter or claim-fidelity check; - record `acted`, `accepted`, `rejected_as_false_positive` or `pending`. +Close the round with the §5.3 loop-close gates against the round's baseline (§2 step 7): +`python tools/length_gate.py --before ` and +`python tools/deai_residue.py --before ` (or `--git-ref`). A strong +`length-growth`, `length-shrink-short` or residue finding needs a disposition before the round +can be disposition-complete; record `--allow "
="` justifications in the +round record. + Subjective strong advisories that require author preference may remain pending with a precise question. Do not erase them merely to make counts zero. @@ -231,6 +246,8 @@ A round is **disposition-complete** when: - critical derivations under scrutiny = 0; - required build/artifacts are valid; - every strong advisory has a disposition or stated pending reason; +- the length and residue gates of §3.7 have run and every strong + `length-growth` / `length-shrink-short` / residue finding has a disposition; - ordinary advisories and unmeasured/degraded axes are reported; - skipped reviewers are labeled unmeasured; - no child report or merge failed. diff --git a/skills/paper-review/SKILL.md b/skills/paper-review/SKILL.md index 913523b..6e0fea9 100644 --- a/skills/paper-review/SKILL.md +++ b/skills/paper-review/SKILL.md @@ -288,9 +288,13 @@ stale scientific content、冲突副本或 required artifact drift 是 `integrit - 先跑 `python tools/deai_residue.py --before /length-baseline.tex` （或 `--git-ref`）：`residue-self-history:`（我们最初/曾经/不再……的研究旅程）、 `residue-edit-meta`（TODO、"see previous version" 等编辑元文本）、 - `residue-negative-label`（标题或 caption 承诺正文没有的东西）与 diff 规则 + `residue-negative-label`（标题或 caption 承诺正文没有的东西）、 + `residue-absence`（正文句子用"它从不做什么"定义论文自己的对象：`never` + 为 strong，`carries no` 等为 ordinary，句中带 citation 的豁免）与 diff 规则 `residue-negative-label-added`。strong finding 使工具 exit 1，本轮不得收尾。 -- 只保留当前科学状态；修法是把句子改写成当前状态，不是追加解释。 +- 只保留当前科学状态。绝对句先删后改：上下文已承载肯定陈述时整句/整从句删除， + 只有页面尚未说出的内容才改写成肯定形式；物理事实、范围限制与承认的局限保留 + 原样（见 `/sci-paper:paper` 自检第 5 项与工具的 action 文本）。 - 外部 published baseline 的正式 head-to-head 比较可以保留，前提是有 citation、protocol 和 numerical comparison。 - 过程叙述若使当前 method/claim 不清或引入 stale content，按 blocker；否则按 advisory。 diff --git a/skills/paper/SKILL.md b/skills/paper/SKILL.md index 07b3e8e..32bfd6a 100644 --- a/skills/paper/SKILL.md +++ b/skills/paper/SKILL.md @@ -272,7 +272,7 @@ python tools/ai_ism_lint.py --field \ | 类别 | 词 | |---|---| -| 动词类 | `delve / delves / delving / delved`, `leverages / leveraging / leveraged`, `pave / paves / paving`, `shed / sheds / shedding`（含 "shed light on"）, `showcase / showcases / showcasing`, `utilizing / utilizes`, `underscore / underscores / underscored / underscoring` | +| 动词类 | `delve / delves / delving / delved`, `leverages / leveraging / leveraged`, `pave / paves / paved / paving`, `shed / sheds / shedding`（含 "shed light on"）, `showcase / showcases / showcased / showcasing`, `utilizing / utilizes`, `underscore / underscores / underscored / underscoring` | | 形容词/副词 | `seamless / seamlessly`, `holistic / holistically`, `comprehensively`, `crucially`, `pivotal` | | 名词类 | `tapestry`, `testament`, `realm / realms` | | 段首套话 | `Recent advances in...`, `Despite significant progress...`, `With the advent of...`, `In recent years,...`, `It is worth noting`, 段首 `Crucially,`, `Importantly,`, `Notably,`, `Interestingly,` | @@ -288,9 +288,11 @@ corpus 复核为 0 出现后才入 Tier A；`landscape` 虽在其词表中，但 **Tier B — per-section/per-word cap** -Tier B 可以使用，但同一个 Tier B 词在同一 section 最多出现 1 次。第 2 次及 -以后是 `l0_target`；cap 内的出现不是 finding。当前词表由 linter 与 profile -共同维护，常见项包括 `Furthermore`, `Moreover`, `Additionally`, +Tier B 可以使用，但同一个 Tier B 词在同一 heading 单元（`\section` 与 +`\subsection` 各算一个单元，linter 按此计）最多出现 1 次。第 2 次及 +以后是 `l0_target`；cap 内的出现不是 finding。规范词表是 +`tools/ai_ism_lint.py` 的 `TIER_A_PATTERN` / `TIER_B_PATTERN`，本表是它的 +人读镜像，常见项包括 `Furthermore`, `Moreover`, `Additionally`, `robust/robustly`, `comprehensive`, `utilize/utilized`, `leverage`, `Importantly`, `Interestingly`, `Notably`, `intricate`, `foster/fosters/fostering/fostered`（后两组 2026-07-16 加入；curated corpus @@ -342,10 +344,12 @@ grep -n -E '—|---|\\textemdash' main.tex # Tier A（必删；正文中不允许出现，包括变体） grep -n -E -i '(delve|leveraged|leverages|leveraging|paved?|paves|paving|shed[s]?|shedding|showcase[sd]?|showcasing|seamless(ly)?|holistic(ally)?|comprehensively|crucially|utilizes|utilizing|underscor(e|es|ed|ing)|tapestry|testament|pivotal|realms?|recent advances|despite significant|with the advent|in recent years|it is worth)' main.tex +# 段首套话（Tier A）：只有位于段落第一行时才算段首；linter 按段落判定，不按物理行 +grep -n -E '^\s*(Importantly|Interestingly|Notably|Crucially),' main.tex -# Tier B（定位；是否超过每节每词 cap 由 linter 按 section 计算） -grep -n -E -i '^\s*(Furthermore|Moreover|Additionally|Importantly|Interestingly|Notably),' main.tex -grep -n -E -i '\b(robust|robustly|comprehensive|utilize|utilized|leverage|intricate|foster(s|ing|ed)?)\b' main.tex +# Tier B（定位；是否超过每 heading 单元每词 cap 由 linter 计算） +grep -n -E -i '^\s*(Furthermore|Moreover|Additionally),' main.tex +grep -n -E -i '\b(robust|robustly|comprehensive|utilize|utilized|leverage|importantly|interestingly|notably|intricate|foster(s|ing|ed)?)\b' main.tex # 顽固替换组（不分级） grep -n -E -i '\b(in order to|aim to|facilitate|serves as)\b' main.tex diff --git a/style-corpus/README.md b/style-corpus/README.md index 8a4b200..b256ec7 100644 --- a/style-corpus/README.md +++ b/style-corpus/README.md @@ -16,17 +16,20 @@ style-corpus/ └── / ├── tier-1-top/ # top-journal exemplars ├── tier-2-mentor/ # mentor or target-author exemplars - └── tier-3-reference/ # other relevant field papers + ├── tier-3-reference/ # other relevant field papers + ├── fulltext-arxiv/ # breadth corpus (fetch_arxiv_abstracts.py --fulltext); bank only + └── fulltext-*/ # any other fulltext-* directory is a HELD-OUT set, never calibration ``` A field is a writing domain such as `wgl`, `cosmology`, or `ml-methods`. Each field produces an independent `style-profile//`. Tools auto-detect only when exactly one field exists; otherwise pass `--field `. -`tools/extract_style.py` currently aggregates the three tiers with fixed weights -`0.5 / 0.3 / 0.2`. These weights describe the current implementation, not a -calibrated consequence rule. Change them only together with an evaluation of how -the resulting profile behaves. +`tools/extract_style.py` records the tier of every observation with the +nominal weights `0.5 / 0.3 / 0.2`, but every aggregate it writes pools the +three tiers **unweighted**; the weights are metadata, not a calibrated rule +(the dossier and the reference distributions were all built that way). Apply +them only together with an evaluation of how the resulting profile behaves. ## Accepted sources @@ -63,9 +66,9 @@ threshold: | 3 (reference) | 0 | 5–15 | | **Total** | **8** | **25–50** | -Small corpora produce uncertain per-section distributions. The extractor does -not currently enforce or guarantee a minimum-paper warning, so downstream -reports must preserve the actual sample count and use `degraded` or +Small corpora produce uncertain per-section distributions. The extractor warns +below 5 curated files (it recommends 8 or more) and enforces nothing, so +downstream reports must preserve the actual sample count and use `degraded` or `unmeasured` when the intended inference is unsupported. Whole-document calibration has a stricter unit rule: each observation must be an diff --git a/style-profile/README.md b/style-profile/README.md index 7d33c64..d8a1657 100644 --- a/style-profile/README.md +++ b/style-profile/README.md @@ -28,12 +28,14 @@ style-profile/ ├── register_lexicon.json # corpus document frequency (deai_register.py --calibrate) ├── collocation_bank.json # unigram + adjacent-pair document frequency (deai_collocation.py --calibrate) ├── collocation_baseline.json # per-section leave-one-out novel-pair reference, sentence unit + ├── cohesion_baseline.json # given/new linkage reference, paragraph unit (deai_discourse.py --calibrate) + ├── hedging_baseline.json # epistemic-marker reference, section unit (same command) ├── deai_policy.json # optional calibrated operating points ├── voice_model.joblib # optional learned field-similarity model ├── voice_features_cache.npz # learned-model feature cache ├── voice_model_evaluation.json # learned-model audit record (train_voice_model.py) ├── anchoring_baseline.json # claim-anchoring band (deai_anchoring.py --calibrate) - ├── ai_ism_negatives_*.jsonl # harvested negative paragraph banks + ├── ai_ism_negatives_extracted.txt # harvested negative paragraphs (extract_md_negatives.py) ├── human_*_extra.jsonl # supplementary human-positive banks ├── human_wl_toptier.jsonl # subfield-restricted human bank (stored, not wired) ├── docval/ # document-level validation working set @@ -88,11 +90,21 @@ python tools/deai_register.py --field --calibrate # Pair bank and leave-one-out sentence reference for the collocation axis (L2.collocation) python tools/deai_collocation.py --field --calibrate +# Cohesion (paragraph unit) and hedging (section unit) references (L2.cohesion / L2.hedging) +python tools/deai_discourse.py --field --calibrate + +# Section-class claim-anchoring band (L2.claim_anchoring) +python tools/deai_anchoring.py --field --calibrate + # Optional learned field-similarity model; still degraded until the recorded # confound audit and operating point justify stronger use python tools/train_voice_model.py --field ``` +> **Recalibrate after every corpus-side fix the CHANGELOG records** (the last +> full rebuild shipped with v0.36.3 on 2026-09-05: 27,851 bank rows, `results` +> 3,240). The v0.28.0 notice below is kept as the record of why. +> > **Recalibrate after v0.28.0.** Two rounds of corpus-layer defects were fixed > on 2026-08-25. Section labelling: `classify_section` matched titles in the > singular only, `method` was itself the default bucket and absorbed every diff --git a/tools/README.md b/tools/README.md index 0340796..ae43665 100644 --- a/tools/README.md +++ b/tools/README.md @@ -9,23 +9,24 @@ serialize evidence but do not define an independent paper verdict. | File | Purpose | Calibration / failure behavior | |---|---|---| | `build_profile.py` | Builds the basic descriptive profile, optionally trains the legacy classifier, and warms the exemplar cache. | Aborts on extraction failure; reports optional-stage failures and continues. It does not create calibrated L1/L2/L3 policy. | -| `extract_style.py` | Builds descriptive lexicon, sentence statistics, transition inventory, dossier, and exemplar bank from `.tex`, `.pdf`, or text corpora. Re-exports every public name from `extract_sections.py`. | PDF extraction is unavailable without pymupdf; ligatures are expanded on the PDF path. Section titles match singular and plural; a `.pdf` is accepted only at source-directory depth 1. The curated tiers carry every weighted aggregate and the dossier; `REFERENCE_DIR` (`fulltext-arxiv/`) is gathered **unweighted** and feeds the exemplar bank only, so breadth cannot restyle the imitation target. Corpus observations do not redefine policy. | +| `extract_style.py` | Builds descriptive lexicon, sentence statistics, transition inventory, dossier, and exemplar bank from `.tex`, `.pdf`, or text corpora (`.txt` feeds the statistics under `unknown`, never the bank). Re-exports every public name from `extract_sections.py`. | PDF extraction is unavailable without pymupdf (skipped files are counted in the summary); ligatures are expanded on the PDF path. Section titles match singular and plural; a `.pdf` is accepted only at source-directory depth 1. Statistics count prose without projection placeholders, words are Unicode letters, and a citation-led paragraph stays in the bank. The curated tiers carry every aggregate and the dossier (their 0.5/0.3/0.2 weights are recorded metadata; the aggregates pool the tiers unweighted); `REFERENCE_DIR` (`fulltext-arxiv/`) is gathered **unweighted** and feeds the exemplar bank only, so breadth cannot restyle the imitation target. Corpus observations do not redefine policy. | | `extract_sections.py` | Source-text projection, section splitting, and corpus document assembly: the section vocabulary and `classify_section`, both named LaTeX projections, the PDF heading heuristic, and `select_document_roots` / `read_tex_document` / `corpus_documents`. Split out of `extract_style.py` on 2026-08-25. | Section buckets key every per-section reference distribution, so a change here requires a profile rebuild. `method` has explicit rules and the default is `unknown`: before 2026-08-25 `method` WAS the default and absorbed every unmatched heading (1,671 "method" passages against 10 "results"). The ALL-CAPS PDF branch requires >= 2 words, >= 4 letters and >= 75% letters, rejecting the table cells ("S", "RA", "NFW", "S/N") that were 305 of 325 detected headings. **A paper is a document, not a file**: `\include`/`\input` fragments are folded back into their root (one review shipped as 12 files), a `\subsection` inherits its `\section` (54.8% of section words previously fell to `unknown`), and a bundle contributes one paper. | -| `tex_assembly.py` | Document assembly shared by every reader: `\input`/`\include` children spliced in place (mid-line calls keep their neighbours, repeats splice again, cycles stop), from disk or from a git ref. | Unresolvable or unreadable child: the call is left in place; a git ref without the root raises `ValueError`. | +| `tex_assembly.py` | Document assembly shared by every reader: `\input`/`\include` children spliced in place (mid-line calls keep their neighbours, repeats splice again, cycles stop), from disk or from a git ref, a target read as written when it has a suffix and resolved against the document root's directory first. | An unreadable ROOT raises (no reader may take it for an empty document); an unresolvable or unreadable child is left in place as the literal call; a git ref without the root raises `ValueError`, and the git reader resolves children at the ref too. | +| `tex_macros.py` | Expands numeric-literal `\newcommand` / `\def` macros once on the assembled document, so a quantity written as a macro reaches the projections that count numerals; owns the LaTeX comment pattern. | Conservative: a symbolic body, an argument-taking macro and a definition inside a comment are left exactly as written; the live definition is dropped so its digits are not counted in the preamble. | | `retrieve_exemplars.py` | Retrieves section- and topic-matched exemplar paragraphs. | Uses sentence-transformer cosine when installed; keyword fallback requires `--allow-fallback`. Reads the **curated tiers only** by default (`--include-reference` widens to the breadth corpus): the bank serves both imitation and reference-distribution roles, and only the first is what retrieval wants. Tier filtering happens after the embedding lookup, because the `.npy` cache is positional against the full bank. | | `train_ai_ism_classifier.py` | Trains the legacy word-ngram paragraph classifier. | Output is degraded L3 advisory evidence, not authorship evidence or an L0 gate. Cross-validation is grouped by source paper (`StratifiedGroupKFold`) because paragraphs from one paper are not independent; `cv_grouped` travels with the metrics (ungrouped F1 0.876 vs grouped 0.823 on `wgl`). | | `extract_md_negatives.py` | Harvests candidate negative paragraphs from a document tree. | Generated data are privacy-sensitive and remain profile-local/gitignored. | | `ai_ism_negatives_handcrafted.txt` | Seed negative examples for the legacy classifier. | Data asset, not normative prose policy. | | `deai_feedback.py` | Implements `sci-paper.feedback.v1`: stable IDs, consequence classes, measurement states, dispositions, ranking, summaries, text/JSON rendering, and tuple compatibility. | Stdlib-only shared contract. | -| `ai_ism_lint.py` | Unified L0 and advisory CLI. Emits ranked text or JSON from the same findings. | Exit 0 = no L0 target; 1 = L0 target present; 2 = invalid input/configuration/execution. Advisories never cause exit 1. | -| `length_gate.py` | Per-section rendered-prose length-budget delta gate between two document versions (standard §5.3). Growing sections emit strong advisories; `--allow` records justifications; `--require-shrink ` demands a net cut of at least that much. | Exit 0 = net unjustified growth within tolerance and any shrink target met; 1 = growth beyond it or a cut short of the target (strong `length-shrink-short`); 2 = invalid input/execution. Comments and math are excluded; a pure section rename nets to zero. | -| `verify_references.py` | Bibliography verification against CrossRef (DataCite fallback for the DOI prefixes CrossRef does not register) and the arXiv API: DOI resolvability and first-author / year / title / journal / volume / first-page agreement per entry, plus `--tex` cross-checking of the assembled document's `\cite` keys against the entries. `--cache` keeps fetched records between rounds. | Exit 0 = no blocker; 1 = unresolvable identifier or cited key without an entry; 2 = invalid input. Title match tolerates the CrossRef title/subtitle split; a one-year print/online seam is ordinary, a larger gap strong. A registry outage marks the entry unmeasured and the axis `degraded`, never clean; an entry with no identifier is unmeasured. Relevance (whether a citation supports its sentence) is not measured here; that half of dimension F stays with the reviewer. | -| `condense_map.py` | The removal map behind `/sci-paper:condense`: six scans — restatement (with the canonical home), zero-gain sentences, dead figure/table/label/macro/acronym, verbose constructions, repeated symbol glosses, duplicated paragraphs — each entry with `removable_words`, totalled into `condense_budget` with a default target that excludes the abstract/conclusion carve-out. | Deletes nothing and emits no blocker; exit 0 on a measured map, 2 on invalid input. Held-out refereed papers carry a median default target of 3.1% of prose (EVALUATION §23.5), so a pass that cuts less than that has not found what the map lists. | -| `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan shipped documentation for edit-meta literals. The literal and label rules read `deai_register.body_only` (a `\newcommand{\TODO}` in a preamble is not prose), and strengths were set on held-out refereed papers: a strong history finding in 16.7% of them, the static negative-label rule ordinary, the strong absence forms at 0.008 per 1,000 words (EVALUATION §23.4). | +| `ai_ism_lint.py` | Unified L0 and advisory CLI. Emits ranked text or JSON from the same findings; every L0–L2 axis is on by default (`--no-` disables one), `--oracle`, `--voice` and `--glossary` are opt-in. | Exit 0 = no L0 target; 1 = L0 target present; 2 = invalid input/configuration/execution (an unreadable root, an unparseable lexicon). Advisories never cause exit 1. A paragraph-initial connector is judged on the paragraph's first prose line and an opener where a sentence starts, never per physical line; command arguments (`\label`, `\ref`, `\cite`, `\url`) are blanked before the lexical scan; the Tier B cap is per heading unit and per word. | +| `length_gate.py` | Per-section rendered-prose length-budget delta gate between two document versions (standard §5.3). Growing sections emit strong advisories; `--allow` records justifications; `--require-shrink ` demands a net cut of at least that much (0 = no cut required, so a map with nothing to cut still closes). | Exit 0 = net unjustified growth within tolerance and any shrink target met; 1 = growth beyond it or a cut short of the target (strong `length-shrink-short`); 2 = invalid input/execution, including a document or baseline that cannot be read (a mistyped `--before` once read as a zero-word baseline and every section as growth). Both versions are assembled documents, the `--git-ref` baseline with its `\input` children resolved at the ref. Comments (blanked before segmenting, so a commented-out heading opens no section) and mathematics in every display form are excluded; a pure section rename nets to zero. | +| `verify_references.py` | Bibliography verification against CrossRef (DataCite fallback for the DOI prefixes CrossRef does not register) and the arXiv API: DOI resolvability (from the `doi` field or a doi.org `url`) and first-author / year / title / journal / volume / first-page agreement per entry, `@string` names expanded, plus `--tex` cross-checking of the assembled document's citation keys (LaTeX, natbib and biblatex commands including `\cites{a}{b}`; `\nocite{*}` cites every entry; a citation inside a `%` comment is text) against the entries. `--cache` keeps fetched records between rounds; a registry miss is never cached, and a finding built from a cached record says so (`observed.cached`). | Exit 0 = no blocker; 1 = unresolvable identifier or cited key without an entry; 2 = invalid input or execution failure (malformed BibTeX names the entry and line). Title match tolerates the CrossRef title/subtitle split; a one-year print/online seam is ordinary, a larger gap strong; a first author agrees when the family names are equal or share a part of four or more letters. A registry outage or an answer of unexpected shape marks the entry unmeasured and the axis `degraded`, never clean; an entry with no identifier is unmeasured. Relevance (whether a citation supports its sentence) is not measured here; that half of dimension F stays with the reviewer. | +| `condense_map.py` | The removal map behind `/sci-paper:condense`: six scans — restatement (with the canonical home), zero-gain sentences, dead figure/table/label/macro/acronym, verbose constructions, repeated symbol glosses, duplicated paragraphs — each entry with `removable_words`, totalled into `condense_budget` with a default target that excludes the abstract/conclusion carve-out. | Deletes nothing and emits no blocker; exit 0 on a measured map, 2 on invalid input. Every word count is the gate's rendered-prose count (placeholders and heading words excluded), and the abstract/conclusion carve-out applies to either side of a restatement pair. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%, EVALUATION §23.5), so a pass that cuts less than that has not found what the map lists. | +| `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan shipped documentation for edit-meta literals. The literal rule reads the visible lines (headings and captions included; a `\newcommand{\TODO}` in a preamble is not prose) and the label rule the body projection; strengths were set on held-out refereed papers: any strong finding in 13.3% of them, the static negative-label rule ordinary, the strong absence forms at 0.008 per 1,000 words (EVALUATION §23.4). | | `deai_metrics.py` | L1 information-distribution analysis: sentence-length variation and connective-openers. | Strong status requires applicable policy calibration; compatibility heuristics are degraded. The reference rate is computed over `CONNECTIVE_OPENERS` from the profile's `paragraph_initial_counts`, i.e. the same set the draft is measured against; a profile predating that key omits the reference clause rather than quoting an incomparable one. | | `deai_structure.py` | L2 sentence-template analysis: enumeration, ordinal/modal/parallel runs, setup-list-wrap-up patterns, and balanced closers; plus the auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) reported under `structure-auxiliary`. | Strong status requires calibrated policy and sufficient reference sample. Auxiliary families never enter `template_score`; their per-bucket human fractions are written by `--calibrate` (`paper_agent_frac`, `wh_cleft_frac`, `modifier_stack_frac`). | | `deai_salience.py` | L2 salience hierarchy: how far a passage's numerals run without an interpreting sentence between them, plus density and per-sentence numeral count, against a per-section human passage reference. | Sole consumer of `extract_style.latex_to_numeral_text` (the numeral-preserving projection; `latex_to_plain` zeroes every numeral on `.tex` input), and it calibrates on the bank's `numeral_text` field, the same projection of each paragraph, so the reference holds the numerals the manuscript side counts. Until v0.36.3 it calibrated on `text`, which holds none inside math, and the p90 gate fired at 0.45 per held-out passage against a 0.27 bound; it now fires at 0.20 (EVALUATION §17.5). Percentiles are read as P(X <= x) on a 0.01 grid at the top of a tie plateau. Buckets under 30 reference passages are degraded; a reference with no spread above the gate abstains rather than flagging. As of the v0.36.3 profile rebuild every bucket clears that floor (smallest: `conclusion` 1,999). One finding per passage, led by its most extreme feature. | -| `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- `AUC` (df 1) must separate from `epoch` (402) and `accuracy` (774). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the word is a formation of an attested stem; it is exhaustive advice, not a detector (rank AUC 0.246, EVALUATION §23.1). | +| `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- `AUC` (df 1) must separate from `epoch` (402) and `accuracy` (774). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the manuscript supplies a reason the tool can read (a formation of an attested stem, a defining sentence or acronym expansion at first use, a proper name capitalised at every use); it is exhaustive advice, not a detector (rank AUC 0.174, EVALUATION §23.1). | | `deai_collocation.py` | L2 collocation: per sentence, the fraction of distinct adjacent common-word pairs that no corpus passage attests, against a leave-one-out per-bucket reference at sentence unit (p90 advisory, p95 strong). Each flagged pair carries λ = df(a)·df(b)/N and e^−λ so the reader can weigh it. `--glossary` reads the same bank document-wide: every unattested pair used at least twice, with its first-use line and whether that sentence defines it, as the list of the manuscript's own terms to define at first use or replace (2026-09-16). | `--calibrate` writes `collocation_bank.json` (unigram and pair document frequency; 11,282 partner words and 530,504 pairs over 41,644 `wgl` passages) and `collocation_baseline.json`. Unmeasured without the bank; degraded where no bucket clears the 30-sentence floor. Pairs break at punctuation, `[math]`/`[CITE]` and dashes; a sentence needs four judged pairs to carry a fraction. The action names the relation a pair compresses and never changes a claim; a defined term keeps its pair. | | `deai_discourse.py` | L2 discourse texture, two axes on the LOW tail (p10 advisory, p05 strong): `cohesion`, the mean fraction of a sentence's content words that already appeared in the sentence before it, and `hedging`, epistemic markers per 1,000 words. | The defect here is absence, not excess, so both flag below the gate. The two run at **different units and each artifact records its own**: cohesion per paragraph, hedging per SECTION, because hedging has no paragraph-scale lower tail at all (p10 is 0.000 in all seven `wgl` buckets). Hedging ships restricted to `intro`, where held-out transfer is 7.89% against 15.48-26.77% elsewhere and every generation regime separates (EVALUATION section 19). Advisories against the field's distribution, never a provenance claim. | | `deai_reference.py` | Library, no CLI. One `(feature, unit)` percentile reference shared by every per-bucket axis: the 0.01 quantile grid, the plateau-top percentile reader, the 30-unit sample floor, the paragraph and section sweeps, the artifact loader, and the calibration loop. | Holds no policy -- gate quantiles, feature names and what a finding means belong to the axis. Its one invariant is that calibration and detection share one unit and one grid; that check is what caught the hedging axis calibrating where no lower tail exists. The artifact loader had five copies across the suite before this module (EVALUATION section 19.1). | @@ -44,10 +45,10 @@ serialize evidence but do not define an independent paper verdict. | `label_findings.py` | Samples findings from all five finding-emitting axes — register, salience, cohesion, hedging, collocation — into a human-labelling sheet (`sample`), re-serves a blind subset with the prior answers stripped (`relabel`), and reports precision, recall and intra-rater agreement (`score`). | The one roadmap item computation cannot close: these axes are corpus-referenced, so nothing here says whether a human agrees a given advisory is right. Built to the scheme chosen 2026-08-26 — finding-level labels, one labeller plus a blind re-label subset, and populations named with `--population NAME=DIR` so each is sampled and reported separately. Recall is **pooled over the axes**: a control row asks whether a passage should have been flagged, and a labeller who says yes has not said by which axis, so a per-axis recall would charge every axis with every other axis's misses. Calibrates nothing; any stratum under 20 labels reports `unmeasured` rather than a rate, and the intra-rater kappa is printed as the ceiling no axis can be held above. | | `eval_findings.py` | Scores the register, salience, cohesion, hedging and collocation axes against provenance labels instead of hand labels (collocation on its document novel-pair fraction, since its reference is per sentence): firing rate on **held-out** refereed ApJ/ApJL/A&A papers, on the same-genre in-sample papers (leakage control), and on the `docval` machine tiers, plus the machine-over-held-out rank AUC. | Read the two rows differently. Register is an absolute rarity test, so its held-out rate **is** a false-positive rate. Salience is a percentile gate (`ADVISORY_PERCENTILE` 0.90), so a non-zero rate is the design point and the number tests calibration transfer, not defectiveness. Neither is precision for the *advice*: publication does not mean every sentence is beyond improvement, so the register figure is an upper bound. Recall stays `unmeasured` — that is `label_findings.py`. | | `eval_docscale.py` | Reproduces the EVALUATION §9 document-scale table — human false-flag rate, per-tier 5%-tail power, and rank AUC — by scoring the corpus and every `docval` tier through `manifold_operating_point`. | The human row is **in-sample** (it includes the manifold's own train and calibration documents) and is labelled as such; the distribution-free guarantee is the split-conformal alpha, not this rate. Point estimates carry substantial seed variance (§9.4c), so a single run is not a stable figure — compare configurations, not runs. | -| `cli_common.py` | Shared command-line preamble and field resolution: UTF-8 stdout, the `--field` / `--profile-root` / `--corpus-root` options, and one `list_fields` / `resolve_field` for the tools that had byte-equivalent copies. | Holds no policy — decides no default beyond the two roots, reads no profile, emits no findings. **21 of 30 tools were retrofitted on 2026-08-26**; the sweep is recorded with its allowlist and refusals. Two divergences are kept deliberately: `ai_ism_lint` warns and returns `None` instead of raising, because its L0 pass must still run with no profile present, and `extract_style` resolves against the corpus root where `tier-*` is not a field (expressed as `exclude_prefixes`, not a second implementation). | +| `cli_common.py` | Shared command-line preamble and field resolution: UTF-8 stdout, the `--field` / `--profile-root` / `--corpus-root` options, `resolve_field` for a tool that needs a field, `optional_field_dir` for one that can run without (one profile auto-resolves, several print a note and run without one), and `axis_main` for the calibrated axes (`--calibrate` needs an existing field and refuses to write an empty artifact; an axis flag goes through `extra_arguments`). | Holds no policy — decides no default beyond the two roots, reads no profile, emits no findings. **21 of 30 tools were retrofitted on 2026-08-26**; the sweep is recorded with its allowlist and refusals. One divergence is kept deliberately: `ai_ism_lint` returns `None` instead of raising (warning on an explicit field that does not exist), because its L0 pass must still run with no profile present; `extract_style` resolves against the corpus root where `tier-*` is not a field (expressed as `exclude_prefixes`, not a second implementation). | | `rewrite_reward.py` | Evaluates rewrite candidates after hard scientific-fidelity eligibility. | Bidirectional: dropping OR inventing protected numbers, units, citations, math, acronyms, semantic LaTeX macros, comparison direction, negation, or causal direction makes a candidate ineligible (`-inf`). Ranking is led by L0 advisory reduction (signed, fidelity-floored) and semantic fidelity; the learned score is tie-break weight unless its bundle is measured. Its numeral tokenizer has been corrected three times for one root cause — a separator absorbed into the token: the Oxford comma, the spaced unit, and (2026-08-25) a hyphen between numerals, which made `0.5-1.2` yield `-1.2` and hard-rejected every faithful rewrite of a range. Each is pinned by a test. Exit 0 = at least one eligible candidate; 1 = every candidate ineligible (a measured outcome: preserve the original and regenerate tighter); 2 = invalid input or missing required configuration. | | `fetch_arxiv_abstracts.py` | Fetches dated arXiv abstract corpora for controlled model evaluation/training. `--query-set wl` narrows to the weak-lensing subfield; `--journals apj,apjl,aa` keeps only those refereed venues. `--fulltext --author "Surname" --author-is --max-authors N` builds a per-author full-text population instead. | Network failures are explicit; fetched text is evidence/training data, not policy. The author path keeps the search term and the identity test separate, because `au:` matches a **surname** and a surname is not a person — `au:"Dell'Antonio"` returns an astronomer and a mathematical physicist. The abstract sweep's `AUTHOR_QUERIES` have no such filter, so each of the ten was checked once against the names it really returns: nine resolve to one person, and `au:"Schneider, P"` returned Donald P. Schneider (SDSS quasars) on 58 papers against the intended Peter Schneider's 33, so that entry now spells the given name out. That check needs the network and cannot be a test; adding an eleventh query means repeating it by hand. `--author` refuses to write into the calibration-breadth directory: a paper in a calibration bank has its own vocabulary in the register denominator, so the axis would go quiet on exactly the prose it was fetched to judge (1 finding across three in-calibration papers against 8 across fifteen held out). Team size is read from the API's author list, never from counting `\author` in the LaTeX, which picks up template placeholder lines. Journal selection is client-side against observed `journal_ref` strings because the API's `jr:` prefix is a loose token match that returns "Research in Astronomy and Astrophysics" for an A&A query. HTTP 429 escalates through a backoff schedule and then stops the sweep with exit 2 and a TRUNCATED report; `--resume` extends a cut-short corpus instead of truncating it, and a journal/subfield run must pass `--out-name` so the broad corpus cannot be clobbered. `--date-hi` bounds the v1 submission, not the returned abstract, which is always the latest version's; `--updated-before` is the only control that dates the text, and a record with no `updated` is dropped rather than assumed clean. | -| `validate_plugin.py` | Validates manifests, version/count agreement (including the versioned doc headers of `docs/architecture/DEAI_SUBSYSTEM.md` and `docs/architecture/EVALUATION.md`), skill frontmatter and standard references, documentation authority boundaries and index completeness, recorded suite sizes against real test discovery, stale review markers, Python syntax, runtime imports, CLI entry points, feedback schema, linter exits, required tests, CI wiring, and the residue contract (`deai_residue.validator_check`: the history families the paper skill quotes match the tool's, and no shipped document carries an edit-meta literal). | Stdlib-only, 10 checks; rejects a document reappearing at a location it was moved away from, an unindexed document under `docs/`, a `design-notes/` file that does not declare itself one, and any recorded test count that disagrees with discovery. Exits nonzero on contract drift. | +| `validate_plugin.py` | Validates manifests, version/count agreement (including the versioned doc headers of `docs/architecture/DEAI_SUBSYSTEM.md` and `docs/architecture/EVALUATION.md`), skill frontmatter and standard references, documentation authority boundaries and index completeness, recorded suite sizes against real test discovery, stale review markers, Python syntax, runtime imports, CLI entry points, feedback schema, linter exits, required tests, CI wiring, and the residue contract (`deai_residue.validator_check`: the history families the paper skill quotes match the tool's, and no shipped document carries an edit-meta literal). | Stdlib-only, 11 checks; rejects a document reappearing at a location it was moved away from, an unindexed document under `docs/`, a `design-notes/` file that does not declare itself one, a relative link or numbered cross-reference whose target does not exist, any recorded test count that disagrees with discovery, and any tracked file over the 750-line budget. Exits nonzero on contract drift. | The top-level manifest counts 40 product tools: the 39 shipped Python tools plus the `ai_ism_negatives_handcrafted.txt` data asset. `validate_plugin.py` is a From 1c9736d6204f744df9de7ec3184dabe8e6ce2dd8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:03:43 +0000 Subject: [PATCH 06/27] deai axes: an axis that cannot fire is not measured; one rule per threshold Audit 2026-09-27, B4/B6/B7/B8/B9/B12/B14/B15 and the hygiene rows B19/B20/B21/B23/B28 for docshape, docstructure, metrics, structure and features. - document_shape sweeps deai_metrics.section_units and drops the preamble and every `skip` unit on both the calibration and the detection side; acknowledgements, appendices and reference lists entered dispersion, role coupling and section arc as body prose (B12). The sweep and the substance filter are shared as shape_units/measurable_sections so deai_partition can score the same geometry (B11 lands with partition). - The document-shape strong flag uses the interpolated quantile the baseline quotes as strong_threshold and the LOO rate is computed with, not the add-one rank percentile: a document could be reported strong while quoting a threshold its own value was below (B9). - --strong-percentile must lie strictly in (0, 1); 1.5 was an IndexError with exit 1 (B8). --calibrate resolves its field through resolve_field and the read path through optional_field_dir; a missing --field was a TypeError traceback (B1). - A structure template finding is measured only against a reference for its own bucket plus a policy, degraded with a baseline that lacks the bucket, unmeasured without one; an `unknown` finding was measured with `templated_fraction: None` (B6). The auxiliary message states the reference fraction and calls a figure rare only below the policy gate (B14). RE_TRICOLON_WRAP requires a sentence-initial "These N " closed by a summing verb; "represent these two quantities" counted, and entered template_score and the manifold (B15). - The distribution axis is degraded when sentence_stats.json holds no classified bucket, where pooled_cv is 0 and burstiness can never fire (B7). - deai_features checks the .npy before importing numpy, gates the UID block on deai_oracle.model_runtime_available() (None entries with a uid_status/uid_reason marker; features_vector raises with the reason), and the CLI prints the reason and returns 2 instead of a ModuleNotFoundError (B4). - Dead code: the callerless distribution_hits/structure_hits adapters (B19), docstructure's shadowed constants and unused imports (B20), the unread REPO_ROOT/DEFAULT_PROFILE_ROOT/argparse (B21); deai_metrics.main resolves the field through optional_field_dir and deai_structure reads the policy through deai_metrics.load_policy(section=...) (B23). The re-export contract test excludes module objects, which is what had made the dead imports "required". The misplaced __main__ block in test_deai_docstructure moves to the end (B28). The structure and docstructure baselines must be rebuilt: B12 and B15 change the values every document contributes. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit ba62d8eafd4efd4232de0c1d95668c342f14aeac) --- tests/test_deai_docstructure.py | 189 ++++++++++++++++++++++++++++++- tests/test_deai_metrics.py | 190 ++++++++++++++++++++++++++++++++ tests/test_deai_structure.py | 130 +++++++++++++++++++++- tools/deai_docshape.py | 77 +++++++++---- tools/deai_docstructure.py | 71 +++++++----- tools/deai_features.py | 66 ++++++++--- tools/deai_metrics.py | 46 ++++---- tools/deai_structure.py | 74 +++++++------ 8 files changed, 724 insertions(+), 119 deletions(-) create mode 100644 tests/test_deai_metrics.py diff --git a/tests/test_deai_docstructure.py b/tests/test_deai_docstructure.py index 346746b..f312fd2 100644 --- a/tests/test_deai_docstructure.py +++ b/tests/test_deai_docstructure.py @@ -1,10 +1,15 @@ from __future__ import annotations +import contextlib +import io +import json import pathlib import sys import tempfile +import types import unittest from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -301,10 +306,6 @@ def test_over_uniform_document_is_flagged_but_varied_is_not(self): "a document matching the human reference shape should not flag") -if __name__ == "__main__": - unittest.main() - - class ModuleSplitContractTests(unittest.TestCase): """`deai_docstructure` must re-export everything `deai_docshape` defines. @@ -317,9 +318,14 @@ class ModuleSplitContractTests(unittest.TestCase): def test_every_public_name_is_re_exported(self): import deai_docshape as shape + # A module object has no `__module__`, so the getattr default used to + # count `json`, `re`, `statistics` and the sibling aliases as names + # `deai_docshape` defines -- which made their dead imports in + # `deai_docstructure` "required" (audit 2026-09-27, B20). expected = { n for n, v in vars(shape).items() if not n.startswith("__") + and not isinstance(v, types.ModuleType) and getattr(v, "__module__", "deai_docshape") in ("deai_docshape", "re", None) and n not in {"annotations", "Path", "Any", "Iterable"} @@ -364,3 +370,178 @@ def test_a_bundle_is_one_document(self): (d / "chap.tex").write_text("body\n", encoding="utf-8") docs = docstructure._paper_documents(root) self.assertEqual([n for n, _t in docs], ["2501.00002"]) + + +class SweepAndOperatingPointTests(unittest.TestCase): + """The 2026-09-27 audit's B12 (skip buckets) and B9 (one threshold rule).""" + + SKIP_TAIL = ( + "\n\n\\section{Acknowledgments}\n\n" + LONG_PARAGRAPH + "\n\n" + SHORT_PARAGRAPH + + "\n\n\\section{Appendix A}\n\n" + REPEATED_PARAGRAPH + "\n\n" + LONG_PARAGRAPH + + "\n\n\\begin{thebibliography}{9}\n\\bibitem{a} A. Author 2020, ApJ 1, 1.\n" + "\\end{thebibliography}\n") + + def test_skip_units_do_not_enter_the_shape(self): + # Acknowledgments, an appendix and the reference list are `skip` + # buckets on the corpus side; they used to enter dispersion, role + # coupling and section arc as body prose because the sweep read + # `section_line_ranges`, which carries no bucket. + body = document([REPEATED_PARAGRAPH, SHORT_PARAGRAPH, LONG_PARAGRAPH, + SHORT_PARAGRAPH, LONG_PARAGRAPH, REPEATED_PARAGRAPH]) + plain = docstructure.document_shape(body) + with_tail = docstructure.document_shape(body.rstrip("\n") + self.SKIP_TAIL) + self.assertEqual(with_tail["status"], "measured") + self.assertEqual(with_tail["n_sections"], plain["n_sections"]) + self.assertEqual(with_tail["n_paragraphs"], plain["n_paragraphs"]) + self.assertEqual(with_tail["metrics"], plain["metrics"]) + self.assertEqual([u["label"] for u in docstructure.shape_units(body + self.SKIP_TAIL)], + ["Introduction", "Methods", "Results"]) + # the preamble (title block before the first heading) is dropped too + self.assertEqual(docstructure.shape_units("\\title{T}\n\n" + body)[0]["label"], + "Introduction") + + def test_measurable_sections_is_the_document_shape_filter(self): + units = docstructure.shape_units(document([REPEATED_PARAGRAPH, "Too short."])) + kept = docstructure.measurable_sections(units) + # each section holds one substantial and one short paragraph, so none + # reaches MIN_PARAGRAPHS_PER_SECTION and the document is insufficient + self.assertEqual(kept, []) + self.assertEqual(docstructure.document_shape( + document([REPEATED_PARAGRAPH, "Too short."]))["status"], "insufficient_evidence") + + def _baseline(self, root: Path, values: list[float], percentile: float) -> None: + threshold = docstructure._quantile(values, percentile) + (root / docstructure.BASELINE_NAME).write_text(json.dumps({ + "schema": "sci-paper.docstructure-baseline.v2", + "n_documents": len(values), "strong_percentile": percentile, + "metrics": {"within_section_similarity": { + "values": values, "strong_threshold": threshold}}, + "dispersion": {}}), encoding="utf-8") + + def _shape(self, observed: float) -> dict: + return {"status": "measured", "reason": None, "n_sections": 3, + "n_paragraphs": 6, "dispersion": {}, "sections": [], + "metrics": {"within_section_similarity": observed, + "cross_section_similarity": None, + "section_arc_similarity": None}} + + def test_strong_flag_uses_the_quoted_threshold_rule(self): + # values [0.1, 0.2, 0.3, 0.4] at p = 0.8: the interpolated quantile the + # baseline quotes is 0.34, while the add-one rank percentile of 0.32 is + # (1 + 3) / 5 = 0.8, so the old rank rule flagged a document strong + # while quoting a threshold its value was below. + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._baseline(root, [0.1, 0.2, 0.3, 0.4], 0.8) + with mock.patch.object(docstructure, "document_shape", + return_value=self._shape(0.32)): + below = docstructure.document_findings("x", root) + self.assertEqual([f["rule"] for f in below], []) + with mock.patch.object(docstructure, "document_shape", + return_value=self._shape(0.35)): + above = docstructure.document_findings("x", root) + self.assertEqual([f["rule"] for f in above], + ["document-shape:within_section_similarity"]) + finding = above[0] + self.assertAlmostEqual(finding["reference"]["strong_threshold"], 0.34) + self.assertGreater(finding["observed"]["value"], + finding["reference"]["strong_threshold"]) + self.assertAlmostEqual(finding["normalized_distance"], 0.35 - 0.34) + self.assertIn("threshold 0.340", finding["message"]) + + +class FeatureRuntimeTests(unittest.TestCase): + """`deai_features` without its optional dependencies: unmeasured markers + and a message, never a ModuleNotFoundError (audit 2026-09-27, B4).""" + + MISSING = (False, "the optional surprisal runtime is not installed (torch)") + + def test_missing_surprisal_runtime_is_an_unmeasured_marker(self): + with mock.patch.object(deai_features.do, "model_runtime_available", + return_value=self.MISSING): + feats = deai_features.paragraph_features(LONG_PARAGRAPH) + self.assertEqual(feats["uid_status"], "unmeasured") + self.assertIn("torch", feats["uid_reason"]) + for name in ("mean_surprisal", "global_uid", "local_uid"): + self.assertIsNone(feats[name]) + self.assertGreater(feats["word_count"], 0) + # the model vector cannot be built without those entries + with self.assertRaises(RuntimeError) as raised: + deai_features.features_vector(LONG_PARAGRAPH) + self.assertIn("torch", str(raised.exception)) + with tempfile.TemporaryDirectory() as temporary: + draft = Path(temporary) / "draft.tex" + draft.write_text(LONG_PARAGRAPH + "\n", encoding="utf-8") + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = deai_features.main([str(draft), "--profile-root", temporary]) + self.assertEqual(code, 2) + self.assertIn("surprisal runtime", err.getvalue()) + self.assertEqual(out.getvalue(), "") + + def test_corpus_centroid_without_a_cache_needs_no_numpy(self): + with tempfile.TemporaryDirectory() as temporary: + # `numpy: None` makes `import numpy` fail even where it is installed + with mock.patch.dict(sys.modules, {"numpy": None}): + self.assertIsNone(deai_features.corpus_centroid(Path(temporary))) + + +class CliContractTests(unittest.TestCase): + """`--field` is optional on the read path, required (and existing) for + `--calibrate`; `--strong-percentile` is a probability (audit B1, B8).""" + + def run_main(self, argv: list[str]) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = docstructure.main(argv) + return code, out.getvalue(), err.getvalue() + + def test_read_path_without_a_field_reports_unmeasured(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text(document([REPEATED_PARAGRAPH, SHORT_PARAGRAPH]), + encoding="utf-8") + empty = root / "profiles" + empty.mkdir() + code, out, _err = self.run_main([str(draft), "--profile-root", str(empty)]) + self.assertEqual(code, 0) + self.assertIn("unmeasured", out) + # two fields: one stderr note, still exit 0 and unmeasured + (empty / "a").mkdir() + (empty / "b").mkdir() + code, out, err = self.run_main([str(draft), "--profile-root", str(empty)]) + self.assertEqual((code, "unmeasured" in out), (0, True)) + self.assertIn("several field profiles", err) + + def test_calibrate_needs_an_existing_field(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + corpus = root / "corpus" + corpus.mkdir() + empty = root / "profiles" + empty.mkdir() + code, _out, err = self.run_main( + ["--calibrate", "--corpus-dir", str(corpus), "--profile-root", str(empty)]) + self.assertEqual(code, 2) + self.assertIn("No field profiles", err) + code, _out, err = self.run_main( + ["--calibrate", "--corpus-dir", str(corpus), "--profile-root", + str(empty), "--field", "nope"]) + self.assertEqual(code, 2) + self.assertIn("not found", err) + + def test_strong_percentile_must_be_a_probability(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text("x\n", encoding="utf-8") + for value in ("1.5", "1", "0", "-0.1"): + code, _out, err = self.run_main( + [str(draft), "--profile-root", str(root), "--strong-percentile", value]) + self.assertEqual(code, 2, value) + self.assertIn("--strong-percentile", err) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_deai_metrics.py b/tests/test_deai_metrics.py new file mode 100644 index 0000000..429fc85 --- /dev/null +++ b/tests/test_deai_metrics.py @@ -0,0 +1,190 @@ +"""`deai_metrics`: the section sweep every per-bucket axis reads, the +distribution findings, and the axis status contract. + +No test imported this module before the 2026-09-27 audit (B30), although +`section_units` decides the bucket of every finding in the suite and +`distribution_findings` is one of the linter's default axes. +""" + +from __future__ import annotations + +import contextlib +import io +import json +import tempfile +import unittest +from pathlib import Path + +from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path + +import deai_metrics as metrics + +EVEN = " ".join(["The shear estimator recovers the input signal within the " + "quoted error budget."] * 6) +ROADMAP = ( + "However, the estimator recovers the signal within the error budget.\n\n" + "Moreover, the covariance keeps the noise dependence under rotation.\n\n" + "Therefore, the likelihood carries both quantities without a change of units.\n" +) + + +def write_profile(root: Path, *, buckets: dict, policy: dict | None = None, + counts: dict | None = None) -> None: + (root / "sentence_stats.json").write_text(json.dumps(buckets), encoding="utf-8") + if counts is not None: + (root / "transition_inventory.json").write_text(json.dumps( + {"n_paragraphs": sum(counts.values()), "paragraph_initial_counts": counts}), + encoding="utf-8") + if policy is not None: + (root / "deai_policy.json").write_text(json.dumps({"distribution": policy}), + encoding="utf-8") + + +class SectionSweepTests(unittest.TestCase): + def test_section_units_inherit_buckets_as_the_corpus_does(self): + text = ("\\title{T}\n\\begin{document}\n" + "\\section{Introduction}\nintro prose\n" + "\\subsection{Prior work}\nstill intro\n" + "\\section{Methods}\n\\subsection{Survey data}\ndata prose\n" + "\\section{Acknowledgments}\nthanks\n" + "\\subsection{Funding}\nmore thanks\n") + units = metrics.section_units(text) + self.assertEqual(units[0], (1, 2, metrics.PREAMBLE_LABEL, metrics.PREAMBLE_BUCKET)) + self.assertEqual([(label, bucket) for _s, _e, label, bucket in units[1:]], [ + ("Introduction", "intro"), + ("Prior work", "intro"), # a topic subsection inherits + ("Methods", "method"), + ("Survey data", "data"), # a role of its own overrides + ("Acknowledgments", "skip"), + ("Funding", "skip"), # a skip parent stays skipped + ]) + self.assertEqual([(s, e) for s, e, _l, _b in units], + [(1, 2), (3, 4), (5, 6), (7, 7), (8, 9), (10, 11), (12, 13)]) + self.assertEqual(metrics.section_line_ranges(text)[1], (3, 4, "Introduction")) + + def test_a_document_without_headings_is_one_unknown_unit(self): + self.assertEqual(metrics.section_units("one\ntwo\n"), + [(1, 2, metrics.DOCUMENT_LABEL, "unknown")]) + self.assertEqual(metrics.section_units(""), []) + + def test_paragraph_line_ranges_are_one_based_and_offset(self): + text = "a\nb\n\n\n c\n\nd" + self.assertEqual(metrics.paragraph_line_ranges(text), + [(1, 2, "a\nb"), (5, 5, " c"), (7, 7, "d")]) + self.assertEqual(metrics.paragraph_line_ranges(text, 10), + [(10, 11, "a\nb"), (14, 14, " c"), (16, 16, "d")]) + self.assertEqual(metrics.paragraph_line_ranges("\n\n"), []) + + +class DistributionFindingTests(unittest.TestCase): + POLICY = {"burstiness_ratio": 0.6, "signpost_fraction": 0.2} + + def test_uniform_lengths_and_roadmap_openers_flag_measured_with_policy(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, buckets={"intro": {"mean": 20.0, "stdev": 10.0}}, + policy=self.POLICY, counts={"However": 5, "The": 95}) + text = "\\section{Introduction}\n\n" + EVEN + "\n\n" + ROADMAP + findings = metrics.distribution_findings(text, root, "draft.tex") + rules = {f["rule"]: f for f in findings} + self.assertEqual(set(rules), {"burstiness-low:intro", "opener-signposting:intro"}) + burst = rules["burstiness-low:intro"] + self.assertEqual((burst["measurement_status"], burst["strength"]), + ("measured", "strong")) + # six identical sentences plus the three roadmap ones: a CV far + # below the reference 0.5, so the ratio sits under the 0.6 gate + self.assertLess(burst["observed"]["sentence_length_cv"], 0.2) + self.assertLess(burst["observed"]["ratio"], 0.6) + self.assertAlmostEqual(burst["reference"]["sentence_length_cv"], 0.5) + opener = rules["opener-signposting:intro"] + self.assertEqual(opener["observed"]["connective_openers"], 3) + self.assertAlmostEqual(opener["reference"]["corpus_fraction"], 0.05) + self.assertIn("reference corpus rate is 5.0%", opener["message"]) + self.assertEqual(opener["location"]["path"], "draft.tex") + + def test_without_policy_findings_are_degraded_and_ordinary(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, buckets={"intro": {"mean": 20.0, "stdev": 10.0}}) + text = "\\section{Introduction}\n\n" + EVEN + "\n" + findings = metrics.distribution_findings(text, root) + self.assertEqual([f["rule"] for f in findings], ["burstiness-low:intro"]) + self.assertEqual((findings[0]["measurement_status"], findings[0]["strength"]), + ("degraded", "ordinary")) + self.assertEqual(metrics.distribution_findings(text, None), []) + + def test_skip_and_preamble_units_are_not_measured(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, buckets={"intro": {"mean": 20.0, "stdev": 10.0}}, + policy=self.POLICY) + text = ("\\title{T}\n\n" + EVEN + "\n\n\\section{Acknowledgments}\n\n" + + EVEN + "\n") + self.assertEqual(metrics.distribution_findings(text, root), []) + + +class AxisStatusTests(unittest.TestCase): + def test_reference_without_a_classified_bucket_is_degraded(self): + # only an `unknown` bucket: pooled CV is 0, the burstiness rule can + # never fire, and the axis used to say `measured` (audit B7) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, buckets={"unknown": {"mean": 20.0, "stdev": 10.0}}, + policy={"burstiness_ratio": 0.6}) + reference = metrics.load_reference(root) + self.assertEqual((reference["cv"], reference["pooled_cv"]), ({}, 0.0)) + status = metrics.distribution_axis_status(root) + self.assertEqual(status["status"], "degraded") + self.assertIn("no classified section bucket", status["reason"]) + write_profile(root, buckets={"intro": {"mean": 20.0, "stdev": 10.0}}, + policy={"burstiness_ratio": 0.6}) + self.assertEqual(metrics.distribution_axis_status(root)["status"], "measured") + + def test_missing_reference_or_policy(self): + self.assertEqual(metrics.distribution_axis_status(None)["status"], "unmeasured") + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, buckets={"intro": {"mean": 20.0, "stdev": 10.0}}) + self.assertEqual(metrics.distribution_axis_status(root)["status"], "degraded") + self.assertEqual(metrics.load_policy(root), None) + (root / "deai_policy.json").write_text( + json.dumps({"distribution": {"a": 1}, "structure": {"b": 2}}), + encoding="utf-8") + self.assertEqual(metrics.load_policy(root), {"a": 1}) + self.assertEqual(metrics.load_policy(root, section="structure"), {"b": 2}) + + +class CliContractTests(unittest.TestCase): + def run_main(self, argv: list[str]) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = metrics.main(argv) + return code, out.getvalue(), err.getvalue() + + def test_field_resolution_on_the_read_path(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text("\\section{Introduction}\n\n" + EVEN + "\n", encoding="utf-8") + profiles = root / "profiles" + profiles.mkdir() + code, out, _err = self.run_main([str(draft), "--profile-root", str(profiles)]) + self.assertEqual((code, "unmeasured" in out), (0, True)) + # one field auto-resolves + only = profiles / "only" + only.mkdir() + write_profile(only, buckets={"intro": {"mean": 20.0, "stdev": 10.0}}, + policy={"burstiness_ratio": 0.6}) + code, out, _err = self.run_main([str(draft), "--profile-root", str(profiles)]) + self.assertEqual((code, "burstiness-low:intro" in out), (0, True)) + # several fields: a stderr note, no profile, exit 0 + (profiles / "other").mkdir() + code, out, err = self.run_main([str(draft), "--profile-root", str(profiles)]) + self.assertEqual((code, "unmeasured" in out, "several field profiles" in err), + (0, True, True)) + code, _out, err = self.run_main([str(root / "missing.tex")]) + self.assertEqual((code, "file not found" in err), (2, True)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_deai_structure.py b/tests/test_deai_structure.py index 2303348..fe04fb8 100644 --- a/tests/test_deai_structure.py +++ b/tests/test_deai_structure.py @@ -1,6 +1,7 @@ from __future__ import annotations -import sys +import json +import tempfile import unittest from pathlib import Path @@ -143,5 +144,132 @@ def test_findings_emit_auxiliary_rule(self): self.assertFalse(finding.get("strong_advisory")) +TEMPLATED = ( + "The estimator must preserve the measured signal under rotation. " + "The covariance must retain the corresponding noise dependence under rotation. " + "The likelihood must represent these two quantities without changing their units. " + "These three requirements define the calculation used for every sample in this analysis." +) + + +def write_profile(root: Path, *, baseline: dict | None, policy: dict | None) -> None: + if baseline is not None: + (root / "structure_baseline.json").write_text(json.dumps(baseline), + encoding="utf-8") + if policy is not None: + (root / "deai_policy.json").write_text(json.dumps({"structure": policy}), + encoding="utf-8") + + +class TricolonWrapTests(unittest.TestCase): + """The wrap-up beat opens a sentence with "These N " and closes it + with a summing verb (audit 2026-09-27, B15).""" + + def test_wrap_up_sentence_matches(self): + for sentence in ("These three requirements define the calculation.", + "These two simple facts together fix the scale.", + "These four steps constitute the pipeline."): + self.assertTrue(structure.RE_TRICOLON_WRAP.match(sentence), sentence) + + def test_an_object_noun_phrase_is_not_a_wrap_up(self): + for sentence in ("The likelihood must represent these two quantities " + "without changing their units.", + "these three samples were observed at dawn.", + "We compare these two estimators below."): + self.assertIsNone(structure.RE_TRICOLON_WRAP.match(sentence), sentence) + values = structure.paragraph_structure( + "We represent these two quantities here. Nothing else follows in " + "this paragraph of ordinary prose about the shear estimator.") + self.assertFalse(values["tricolon_wrap"]) + self.assertTrue(structure.paragraph_structure(TEMPLATED)["tricolon_wrap"]) + + +class StrongStatusTests(unittest.TestCase): + """A template finding is `measured` only against a reference for its own + bucket plus a policy; `degraded` with a baseline that lacks the bucket; + `unmeasured` without a baseline (audit 2026-09-27, B6).""" + + BASELINE = {"method": {"n": 25, "templated_frac": 0.02, "auxiliary_frac": 0.4}} + POLICY = {"rare_template_fraction": 0.05} + + def template_finding(self, text: str, profile: Path | None) -> dict: + findings = [f for f in structure.structure_findings(text, profile) + if f["rule"].startswith("structure-template:")] + self.assertEqual(len(findings), 1) + return findings[0] + + def test_measured_and_strong_only_with_bucket_reference_and_policy(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, baseline=self.BASELINE, policy=self.POLICY) + finding = self.template_finding("\\section{Methods}\n\n" + TEMPLATED, root) + self.assertEqual((finding["measurement_status"], finding["strength"]), + ("measured", "strong")) + self.assertIn("reference method fraction 2.0%", finding["message"]) + # a bucket the baseline does not hold: degraded, never strong, and + # the message quotes no fraction + unknown = self.template_finding( + "\\section{Weak gravitational lensing}\n\n" + TEMPLATED, root) + self.assertEqual((unknown["measurement_status"], unknown["strength"]), + ("degraded", "ordinary")) + self.assertEqual(unknown["reference"]["templated_fraction"], None) + self.assertNotIn("fraction", unknown["message"]) + + def test_baseline_without_policy_is_degraded(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, baseline=self.BASELINE, policy=None) + finding = self.template_finding("\\section{Methods}\n\n" + TEMPLATED, root) + self.assertEqual((finding["measurement_status"], finding["strength"]), + ("degraded", "ordinary")) + + def test_no_profile_is_unmeasured(self): + finding = self.template_finding("\\section{Methods}\n\n" + TEMPLATED, None) + self.assertEqual((finding["measurement_status"], finding["strength"]), + ("unmeasured", "ordinary")) + + def test_load_policy_reads_the_structure_block(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, baseline=None, policy=self.POLICY) + self.assertEqual(structure.load_policy(root), self.POLICY) + self.assertIsNone(structure.load_policy(None)) + + +class AuxiliaryWordingTests(unittest.TestCase): + """The auxiliary message states the reference fraction and calls a figure + rare only below the policy gate (audit 2026-09-27, B14).""" + + def auxiliary_message(self, profile: Path | None) -> str: + findings = [f for f in structure.structure_findings( + "\\section{Methods}\n\n" + ANTITHESIS_HEAVY, profile) + if f["rule"].startswith("structure-auxiliary:")] + self.assertEqual(len(findings), 1) + return findings[0]["message"] + + def test_common_figure_is_not_called_rare(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, baseline={"method": {"n": 120, "auxiliary_frac": 0.4}}, + policy={"rare_template_fraction": 0.05}) + message = self.auxiliary_message(root) + self.assertNotIn("rare", message) + self.assertIn("reference method fraction 40.0% (n=120)", message) + + def test_rare_figure_is_called_rare_with_the_gate(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_profile(root, baseline={"method": {"n": 120, "auxiliary_frac": 0.01}}, + policy={"rare_template_fraction": 0.05}) + message = self.auxiliary_message(root) + self.assertIn("rare in the field reference (method fraction 1.0% <= 5.0%, n=120)", + message) + + def test_no_reference_quotes_no_fraction(self): + message = self.auxiliary_message(None) + self.assertNotIn("rare", message) + self.assertNotIn("fraction", message) + + if __name__ == "__main__": unittest.main() diff --git a/tools/deai_docshape.py b/tools/deai_docshape.py index 5be98d5..f74ddd7 100644 --- a/tools/deai_docshape.py +++ b/tools/deai_docshape.py @@ -171,30 +171,65 @@ def _pairwise_similarity(vectors: list[list[float]]) -> float | None: return statistics.mean(values) if values else None -def document_shape(text: str) -> dict[str, Any]: - """Measure document shape or return an explicit insufficient-evidence state.""" +def shape_units(text: str) -> list[dict[str, Any]]: + """The prose sections a document is measured on, with their raw blocks. + + One dict per unit of `deai_metrics.section_units` whose bucket is prose: + `{"label", "start_line", "end_line", "blocks": [(start, end, block), ...]}`. + The preamble and every `skip` unit (acknowledgements, appendices, the + reference list, affiliations) are dropped, on the calibration side and + the detection side alike. Until 2026-09-27 this sweep read + `section_line_ranges`, which carries no bucket, so a bibliography or an + appendix entered the dispersion, role-coupling and section-arc + measurements as body prose; a docstructure baseline built before the + change must be rebuilt, since its every value carried them. + `deai_partition` cuts its candidate blocks from this same list. + """ lines = text.splitlines() - sections: list[dict[str, Any]] = [] - for start, end, label in metrics.section_line_ranges(text): - if label == "(preamble)": + units: list[dict[str, Any]] = [] + for start, end, label, bucket in metrics.section_units(text): + if bucket in (metrics.PREAMBLE_BUCKET, "skip"): continue segment = "\n".join(lines[start - 1:end]) - paragraphs = [] - for p_start, p_end, block in metrics.paragraph_line_ranges(segment, start): - if len(es.words(es.latex_to_plain(block))) < MIN_WORDS: - continue - paragraphs.append({ - "start_line": p_start, - "end_line": p_end, - "vector": _shape_vector(block), - "modelfree": _paragraph_modelfree(block), - # cheap content-role markers for role-coupled dispersion - "has_cite": "\\cite" in block, - "has_math": bool(_MATH_MARKER_RE.search(block)), - }) - if len(paragraphs) >= MIN_PARAGRAPHS_PER_SECTION: - sections.append({"label": label, "start_line": start, - "end_line": end, "paragraphs": paragraphs}) + units.append({"label": label, "start_line": start, "end_line": end, + "blocks": metrics.paragraph_line_ranges(segment, start)}) + return units + + +def measurable_sections(units: list[dict[str, Any]]) -> list[dict[str, Any]]: + """The units `document_shape` scores, each cut down to its scored blocks. + + A block counts when it carries at least MIN_WORDS of prose, and a unit + stays when it holds at least MIN_PARAGRAPHS_PER_SECTION such blocks; a + document needs MIN_SECTIONS surviving units, which the caller checks. + One filter for both consumers of the dispersion manifold: `deai_partition` + applied only the word floor, so its starting distance for a document was + not the distance `deai_docstructure` reported for the same file. + """ + kept = [] + for unit in units: + blocks = [item for item in unit["blocks"] + if len(es.words(es.latex_to_plain(item[2]))) >= MIN_WORDS] + if len(blocks) >= MIN_PARAGRAPHS_PER_SECTION: + kept.append({**unit, "blocks": blocks}) + return kept + + +def document_shape(text: str) -> dict[str, Any]: + """Measure document shape or return an explicit insufficient-evidence state.""" + sections: list[dict[str, Any]] = [] + for unit in measurable_sections(shape_units(text)): + paragraphs = [{ + "start_line": p_start, + "end_line": p_end, + "vector": _shape_vector(block), + "modelfree": _paragraph_modelfree(block), + # cheap content-role markers for role-coupled dispersion + "has_cite": "\\cite" in block, + "has_math": bool(_MATH_MARKER_RE.search(block)), + } for p_start, p_end, block in unit["blocks"]] + sections.append({"label": unit["label"], "start_line": unit["start_line"], + "end_line": unit["end_line"], "paragraphs": paragraphs}) if len(sections) < MIN_SECTIONS: return { "status": "insufficient_evidence", diff --git a/tools/deai_docstructure.py b/tools/deai_docstructure.py index 8221565..68c7454 100644 --- a/tools/deai_docstructure.py +++ b/tools/deai_docstructure.py @@ -7,37 +7,23 @@ from __future__ import annotations -import argparse -import json -import math -import random -import re -import statistics import sys from pathlib import Path -from typing import Any, Iterable +from typing import Any sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first -import deai_features as features # noqa: E402 sibling import after path setup import deai_feedback as feedback # noqa: E402 shared finding contract import deai_reference as reference # noqa: E402 because sibling tools are importable only after the sys.path insert above -import deai_metrics as metrics # noqa: E402 canonical section/paragraph ranges -import deai_structure as structure # noqa: E402 sentence template features import extract_style as es # noqa: E402 canonical LaTeX cleanup/tokenizer -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" -BASELINE_NAME = "docstructure_baseline.json" -MIN_WORDS = 30 -MIN_SECTIONS = 3 -MIN_PARAGRAPHS_PER_SECTION = 2 - # Document-shape measurement, the dispersion manifold, the conformal operating # points and the baseline builder moved to deai_docshape.py on 2026-08-25 (this # file could no longer be edited at its size). Re-exported because sibling tools # and tests reach them as dds.; unused in this module by design, which is -# what the F401 waiver records. +# what the F401 waiver records. The constants and module aliases this file +# defined for itself were dead after the split (each constant was overwritten by +# the import below), so it defines none of its own now. from deai_docshape import ( # noqa: F401 -- re-export, unused here by design BASELINE_NAME, CONFORMAL_ALPHA, CONFORMAL_SEED, CONFORMAL_STRATA, CONFORMAL_TRAIN_FRACTION, DISPERSION_FEATURE_NAMES, @@ -52,7 +38,7 @@ _pairwise_similarity, _paragraph_modelfree, _percentile, _quantile, _shape_vector, _stratum_calibration, calibrate, document_role_coupling, document_shape, fit_dispersion_manifold, manifold_distance, - manifold_operating_point, + manifold_operating_point, measurable_sections, shape_units, ) load_baseline = reference.baseline_loader(BASELINE_NAME) @@ -89,10 +75,22 @@ def document_findings(text: str, field_profile_dir: Path | None, values = reference.get("values", []) if observed is None or not values: continue - percentile = _percentile(values, float(observed)) operating_point = float(baseline.get("strong_percentile", 0.95)) - if percentile < operating_point: + # One rule on both sides: the flag, the quoted `strong_threshold` and + # the leave-one-document-out flag rate all use the interpolated + # quantile of the reference values. The flag used to be the add-one + # rank percentile, which sits above the quantile rule by about one + # rank, so a document could be reported strong while quoting a + # threshold its own value was below, against a false-flag rate that + # belonged to a different rule. + threshold = reference.get("strong_threshold") + if threshold is None: + threshold = _quantile([float(v) for v in values], operating_point) + threshold = float(threshold) + value = float(observed) + if value <= threshold: continue + percentile = _percentile(values, value) findings.append(feedback.make_finding( kind="advisory", layer="L2", rule=f"document-shape:{metric_name}", scope="document", calibration_unit="document", line=1, section=section_label, path=path, @@ -103,17 +101,18 @@ def document_findings(text: str, field_profile_dir: Path | None, "n_paragraphs": shape["n_paragraphs"]}, reference={"n_documents": baseline["n_documents"], "strong_percentile": operating_point, - "strong_threshold": reference.get("strong_threshold"), + "strong_threshold": threshold, "bootstrap_95_ci": reference.get("bootstrap_95_ci"), "leave_one_document_out_flag_rate": reference.get( "leave_one_document_out_flag_rate"), "provenance": BASELINE_NAME}, - normalized_distance=percentile - operating_point, + normalized_distance=value - threshold, confidence={"value": min(1.0, baseline["n_documents"] / 30.0), "basis": f"{baseline['n_documents']} complete reference documents"}, message=(f"Document-level {metric_name.replace('_', ' ')} is " - f"{observed:.3f}, at empirical percentile {percentile:.3f} " - "of the complete-document reference."), + f"{value:.3f}, above the reference {operating_point:.0%} " + f"threshold {threshold:.3f} (empirical percentile " + f"{percentile:.3f} of the complete-document reference)."), action=("Inspect repeated paragraph and section shapes; vary only " "needless rhetorical symmetry while preserving logical organization."), evidence=[metric_name, round(float(observed), 8)], @@ -403,14 +402,29 @@ def main(argv: list[str] | None = None) -> int: help="directory of complete human .tex/.md source documents") parser.add_argument("--strong-percentile", type=float, default=0.95) args = parser.parse_args(argv) - field_dir = args.profile_root / args.field + # A percentile is a probability: 1.5 indexed past the end of the value + # list (IndexError, exit 1) and a negative value extrapolated a threshold + # below every reference document. + if not 0.0 < args.strong_percentile < 1.0: + print(f"[deai_docstructure] --strong-percentile must lie strictly between " + f"0 and 1, got {args.strong_percentile}", file=sys.stderr) + return 2 if args.calibrate: if args.corpus_dir is None or not args.corpus_dir.exists(): print("[deai_docstructure] --calibrate requires --corpus-dir", file=sys.stderr) return 2 + # The baseline is written INTO the profile, so calibration needs a + # field that exists; the read path below runs without one. + try: + field = cli_common.resolve_field( + args.field, args.profile_root, tool="deai_docstructure", + empty_hint="--calibrate needs an existing style-profile// directory.") + except SystemExit as error: + print(error, file=sys.stderr) + return 2 try: - baseline = calibrate(_paper_documents(args.corpus_dir), field_dir, - args.strong_percentile) + baseline = calibrate(_paper_documents(args.corpus_dir), + args.profile_root / field, args.strong_percentile) except ValueError as error: print(f"[deai_docstructure] {error}", file=sys.stderr) return 2 @@ -419,6 +433,7 @@ def main(argv: list[str] | None = None) -> int: if args.file is None or not args.file.exists(): print(f"[deai_docstructure] file not found: {args.file}", file=sys.stderr) return 2 + field_dir = cli_common.optional_field_dir(args, tool="deai_docstructure") text = args.file.read_text(encoding="utf-8", errors="replace") report = feedback.build_report( path=args.file, diff --git a/tools/deai_features.py b/tools/deai_features.py index 54fee8a..b9de994 100644 --- a/tools/deai_features.py +++ b/tools/deai_features.py @@ -10,7 +10,8 @@ distributional (model-free, from extract_style tokenizers): n_sentences, mean_sent_len, sent_len_cv, sent_len_stdev, word_count, opens_connective, equivocal_rate, paren_rate, semicolon_rate, comma_rate - surprisal / UID (Layer B, local LM): + surprisal / UID (Layer B, local LM; None with `uid_status: unmeasured` + when the optional transformers/torch runtime is not installed): mean_surprisal, global_uid, local_uid semantic (all-MiniLM-L6-v2, cached corpus centroid): corpus_cos (cosine similarity of the paragraph embedding to the curated @@ -26,7 +27,6 @@ from __future__ import annotations -import argparse import math import random import re @@ -40,8 +40,6 @@ import deai_oracle as do # noqa: E402 same reason from deai_metrics import CONNECTIVE_OPENERS # noqa: E402 same reason -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" EMBED_MODEL = "all-MiniLM-L6-v2" FEATURE_SCHEMA_VERSION = "sci-paper.voice-features.v1" @@ -95,15 +93,20 @@ def embedder_available() -> bool: def corpus_centroid(field_profile_dir: Path) -> "object | None": - """Unit-normalized mean of the cached corpus paragraph embeddings, or None.""" + """Unit-normalized mean of the cached corpus paragraph embeddings, or None. + + The file is checked before numpy is imported: a profile with no embedding + cache is the common case (numpy is optional), and "no centroid" is the + honest answer there, not a ModuleNotFoundError. + """ key = str(field_profile_dir) if key in _CENTROID_CACHE: return _CENTROID_CACHE[key] - import numpy as np npy = field_profile_dir / f"exemplar_embeddings_{EMBED_MODEL}.npy" if not npy.exists(): _CENTROID_CACHE[key] = None return None + import numpy as np emb = np.load(npy) c = emb.mean(axis=0) n = np.linalg.norm(c) @@ -172,15 +175,29 @@ def paragraph_features( model_name: str = do.DEFAULT_MODEL, centroid=None, ) -> dict: - """Full fundamental-feature dict for one paragraph. `text` may contain LaTeX.""" + """Full fundamental-feature dict for one paragraph. `text` may contain LaTeX. + + The surprisal runtime (transformers + torch) is optional. Without it the + three UID features are None and the dict carries `uid_status: "unmeasured"` + with the reason, so a caller sees an unmeasured marker rather than the + ModuleNotFoundError `token_surprisals` used to raise from inside this + function, and never a 0.0 that would read as a measured value. (A + paragraph too short for a stable estimate keeps its historical 0.0: the + trained bundles were fit with that convention.) + """ plain = es.latex_to_plain(text) feats = distributional_features(plain) # surprisal / UID (Layer B) - uid = do.uid_features(do.token_surprisals(plain, model_name)) - if uid is None: # too short for a stable UID estimate - feats.update({"mean_surprisal": 0.0, "global_uid": 0.0, "local_uid": 0.0}) + available, reason = do.model_runtime_available() + if not available: + feats.update({"mean_surprisal": None, "global_uid": None, "local_uid": None, + "uid_status": "unmeasured", "uid_reason": reason}) else: - feats.update({k: uid[k] for k in ("mean_surprisal", "global_uid", "local_uid")}) + uid = do.uid_features(do.token_surprisals(plain, model_name)) + if uid is None: # too short for a stable UID estimate + feats.update({"mean_surprisal": 0.0, "global_uid": 0.0, "local_uid": 0.0}) + else: + feats.update({k: uid[k] for k in ("mean_surprisal", "global_uid", "local_uid")}) # semantic distance to corpus centroid cos = 0.0 if centroid is None and field_profile_dir is not None: @@ -194,7 +211,18 @@ def paragraph_features( def features_vector(text: str, **kw) -> list[float]: + """The FEATURE_NAMES-ordered vector the learned model consumes. + + It needs the surprisal runtime: a vector with three unmeasured entries is + not one the model can read, so a missing runtime raises here with the + reason, instead of leaking None into a numpy float cast three frames down + in `deai_voice` or `voice_dataset`. A caller that can run without the + vector probes `deai_oracle.model_runtime_available()` first. + """ f = paragraph_features(text, **kw) + if f.get("uid_status") == "unmeasured": + raise RuntimeError(f"[deai_features] {f['uid_reason']}; features_vector " + "has no UID entries without it") return [f[name] for name in FEATURE_NAMES] @@ -374,9 +402,19 @@ def main(argv: list[str] | None = None) -> int: p.add_argument("file", type=Path) p.add_argument("--model", default=do.DEFAULT_MODEL) args = p.parse_args(argv) - field_dir = None - if args.field: - field_dir = args.profile_root / args.field + if not args.file.exists(): + print(f"[deai_features] file not found: {args.file}", file=sys.stderr) + return 2 + # The dump is the full vector, UID included; without the runtime it has + # nothing complete to print, so it says why and exits 2 (invalid + # configuration) rather than dying in a ModuleNotFoundError. + available, reason = do.model_runtime_available() + if not available: + print(f"[deai_features] {reason}; install the optional surprisal " + "runtime (see requirements.txt) to dump feature vectors", + file=sys.stderr) + return 2 + field_dir = cli_common.optional_field_dir(args, tool="deai_features") text = args.file.read_text(encoding="utf-8", errors="replace") centroid = corpus_centroid(field_dir) if field_dir else None # split into blank-line paragraphs and dump features diff --git a/tools/deai_metrics.py b/tools/deai_metrics.py index a8ab980..d8442a1 100644 --- a/tools/deai_metrics.py +++ b/tools/deai_metrics.py @@ -1,9 +1,9 @@ """Model-free information-distribution feedback for scientific prose (L1). -The structured API is :func:`distribution_findings`. The legacy -:func:`distribution_hits` adapter remains for callers that consume -``(line, rule, message)`` tuples. Findings are advisory; the standalone CLI -returns success when measurement completes even when feedback is present. +The structured API is :func:`distribution_findings`. Findings are advisory; the +standalone CLI returns success when measurement completes even when feedback is +present. (A `(line, rule, message)` tuple adapter shipped beside it until +2026-09-27; nothing called it.) This module also owns the manuscript-side section sweep every per-bucket axis reads (:func:`section_line_ranges`, :func:`section_units`). It parses with the @@ -24,8 +24,6 @@ import deai_feedback as feedback # noqa: E402 sibling tool import after path setup import extract_style as es # noqa: E402 canonical tokenizer import after path setup -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" # One section parser for both sides of every per-bucket reference. This module # kept its own (`\\section\{`), which read neither `\section {X}` nor # `\section[S]{X}` while the corpus splitter read both, so a manuscript written @@ -103,7 +101,15 @@ def load_reference(field_profile_dir: Path | None) -> dict[str, Any] | None: } -def load_policy(field_profile_dir: Path | None) -> dict[str, Any] | None: +def load_policy(field_profile_dir: Path | None, *, + section: str = "distribution") -> dict[str, Any] | None: + """The `section` block of the field's `deai_policy.json`, or None. + + One policy file carries one block per axis (`distribution`, `structure`), + and a file written before the blocks existed is the block itself. Two + axes kept byte-equivalent copies of this reader that differed only in the + key; `deai_structure` now reads through here with `section="structure"`. + """ if field_profile_dir is None: return None path = field_profile_dir / "deai_policy.json" @@ -113,7 +119,7 @@ def load_policy(field_profile_dir: Path | None) -> dict[str, Any] | None: data = json.loads(path.read_text(encoding="utf-8")) except json.JSONDecodeError: return None - return data.get("distribution", data) + return data.get(section, data) def _headings(lines: list[str]) -> list[tuple[int, str, str]]: @@ -203,6 +209,18 @@ def distribution_axis_status(field_profile_dir: Path | None) -> dict[str, Any]: reason="sentence_stats.json reference is unavailable", detector="deai_metrics", ) + # A reference whose only bucket is `unknown` has no sentence-length CV to + # compare against: `pooled_cv` is 0, the burstiness rule can never fire, + # and reporting the axis `measured` would pass that silence off as a + # clean result. The opener rule still runs, so the axis is degraded, not + # unmeasured. + if not reference["cv"]: + return feedback.axis_status( + "L1.distribution", "degraded", + reason=("sentence_stats.json holds no classified section bucket, " + "so sentence-length variation cannot be measured"), + detector="deai_metrics", + ) if load_policy(field_profile_dir) is None: return feedback.axis_status( "L1.distribution", "degraded", @@ -307,11 +325,6 @@ def distribution_findings(text: str, field_profile_dir: Path | None, return findings -def distribution_hits(text: str, field_profile_dir: Path | None - ) -> list[tuple[int, str, str]]: - return feedback.tuple_hits(distribution_findings(text, field_profile_dir)) - - def main(argv: list[str] | None = None) -> int: cli_common.utf8_stdout() parser = cli_common.field_parser(__doc__) @@ -320,12 +333,7 @@ def main(argv: list[str] | None = None) -> int: if not args.file.exists(): print(f"[deai_metrics] file not found: {args.file}", file=sys.stderr) return 2 - field_dir = args.profile_root / args.field if args.field else None - if field_dir is None and args.profile_root.exists(): - fields = [path for path in args.profile_root.iterdir() - if path.is_dir() and not path.name.startswith(".")] - if len(fields) == 1: - field_dir = fields[0] + field_dir = cli_common.optional_field_dir(args, tool="deai_metrics") text = args.file.read_text(encoding="utf-8", errors="replace") findings = distribution_findings(text, field_dir, args.file) report = feedback.build_report( diff --git a/tools/deai_structure.py b/tools/deai_structure.py index bf601fb..619c213 100644 --- a/tools/deai_structure.py +++ b/tools/deai_structure.py @@ -1,8 +1,8 @@ """Model-free sentence-construction feedback for scientific prose (L2). The detector reports concrete template families without making authorship -claims. :func:`structure_findings` is the structured API; -:func:`structure_hits` is the compatibility tuple adapter. +claims. :func:`structure_findings` is the structured API. (A tuple adapter +shipped beside it until 2026-09-27; nothing called it.) """ from __future__ import annotations @@ -17,11 +17,10 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first import deai_feedback as feedback # noqa: E402 sibling import after path setup +import deai_metrics as metrics # noqa: E402 the shared policy reader import deai_reference as reference # noqa: E402 the shared paragraph sweep import extract_style as es # noqa: E402 canonical tokenizer -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" _COUNT = r"two|three|four|five|six|seven|eight|2|3|4|5|6|7|8" _ENUM_NOUN = (r"elements|components|parts|ingredients|obligations|requirements|" r"properties|features|reasons|ways|steps|aspects|criteria|" @@ -38,7 +37,18 @@ rf"depends?|builds?|require|requires|involve|involves|follow|follows|" rf"proceeds?|breaks? down)\b[^.\n]{{0,30}}\b(on|of|into|from|in)\b" rf"[^.\n]{{0,20}}\b({_COUNT})\b", re.I) -RE_TRICOLON_WRAP = re.compile(rf"\bthese\s+({_COUNT})\s+\w+", re.I) +# The wrap-up beat of a set-up/list/wrap-up template: a sentence that OPENS +# with "These N " and then closes the list with a summing verb. Matched +# per sentence at its start. Until 2026-09-27 the pattern was `these N \w+` +# anywhere in a paragraph, which read "represent these two quantities" -- a +# plain object noun phrase -- as a wrap-up, and that count entered both +# `template_score` and the document-dispersion manifold, so a structure or +# docstructure baseline built before the change must be rebuilt. +_WRAP_VERB = (r"together|jointly|defines?|forms?|constitutes?|determines?|spans?|" + r"captures?|completes?|summari[sz]es?|comprises?|covers?|fix(?:es)?|" + r"sets?|makes?|yields?|closes?|frames?|bounds?|governs?") +RE_TRICOLON_WRAP = re.compile( + rf"^these\s+({_COUNT})\s+(?:\w+\s+){{1,2}}(?:{_WRAP_VERB})\b", re.I) RE_BALANCED = re.compile(r"\bone\b[^.\n]{0,60}\band\b[^.\n]{0,45}\banother\b", re.I) # Auxiliary families (v0.18.0, panel-derived; EVALUATION §13). Kept OUT of # `templates`/`template_score` so the calibrated document-dispersion manifold @@ -155,7 +165,8 @@ def paragraph_structure(text: str) -> dict[str, Any]: ordinal_run = _max_run(first_words, lambda word: word in _ORDINALS) anaphora_run = _max_run_equal(first_words, exclude=_TRIVIAL_OPENERS) modal_run = _max_run_equal(modals, exclude={""}) - tricolon_wrap = bool(RE_TRICOLON_WRAP.search(plain)) + tricolon_wrap = any(RE_TRICOLON_WRAP.match(sentence.strip()) + for sentence in sentences) balanced = bool(RE_BALANCED.search(plain)) templates: list[str] = [] if announced and ordinal_run >= ORDINAL_RUN: @@ -223,16 +234,8 @@ def load_baseline(field_profile_dir: Path | None) -> dict[str, Any] | None: def load_policy(field_profile_dir: Path | None) -> dict[str, Any] | None: - if field_profile_dir is None: - return None - path = field_profile_dir / "deai_policy.json" - if not path.exists(): - return None - try: - data = json.loads(path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - return None - return data.get("structure", data) + """The `structure` block of the field's policy (one reader, see deai_metrics).""" + return metrics.load_policy(field_profile_dir, section="structure") def structure_axis_status(field_profile_dir: Path | None) -> dict[str, Any]: @@ -272,14 +275,16 @@ def structure_findings(text: str, field_profile_dir: Path | None, human_fraction = bucket_reference.get("templated_frac") reference_n = int(bucket_reference.get("n", 0)) rare_fraction = (policy or {}).get("rare_template_fraction") + # A finding is measured only against a reference that exists for ITS + # bucket and a policy that says what rare means. A baseline that holds + # no `unknown` bucket used to make an `unknown` finding `measured` + # while its reference block read `templated_fraction: None, n: 0`. + applicable = human_fraction is not None and rare_fraction is not None + status = "measured" if applicable else ( + "degraded" if baseline else "unmeasured") if values["templates"]: - strong = bool( - baseline and policy and human_fraction is not None - and rare_fraction is not None and reference_n >= 20 - and human_fraction <= float(rare_fraction) - ) - status = "measured" if baseline and policy else ( - "degraded" if baseline else "unmeasured") + strong = bool(applicable and reference_n >= 20 + and human_fraction <= float(rare_fraction)) context = (f"; reference {bucket} fraction {human_fraction:.1%} " f"(n={reference_n})") if human_fraction is not None else "" findings.append(feedback.make_finding( @@ -311,8 +316,19 @@ def structure_findings(text: str, field_profile_dir: Path | None, )) if values["auxiliary_templates"]: aux_fraction = bucket_reference.get("auxiliary_frac") - aux_context = (f"; reference {bucket} fraction {aux_fraction:.1%} " - f"(n={reference_n})") if aux_fraction is not None else "" + # The message states the reference fraction; it calls the figure + # rare only when the fraction is at or below the policy's rarity + # gate. It used to say "rare in the field reference" unconditionally, + # beside a reference fraction of 40%. + if aux_fraction is None: + aux_context = "" + elif rare_fraction is not None and aux_fraction <= float(rare_fraction): + aux_context = (f"; rare in the field reference ({bucket} fraction " + f"{aux_fraction:.1%} <= {float(rare_fraction):.1%}, " + f"n={reference_n})") + else: + aux_context = (f"; reference {bucket} fraction {aux_fraction:.1%} " + f"(n={reference_n})") findings.append(feedback.make_finding( kind="advisory", layer="L2", rule=f"structure-auxiliary:{bucket}", scope="paragraph", @@ -337,8 +353,7 @@ def structure_findings(text: str, field_profile_dir: Path | None, if aux_fraction is not None else None, confidence={"value": min(1.0, values["n_sent"] / 6.0), "basis": f"{values['n_sent']} sentences; deterministic rhetorical-figure evidence"}, - message=("Paragraph leans on rhetorical figure(s) rare in the " - "field reference: " + message=("Paragraph leans on rhetorical figure(s): " f"{', '.join(values['auxiliary_templates'])}{aux_context}."), action=("Keep an antithesis only where the contrast is load-bearing " "technical content; state posture contrasts as plain positive " @@ -352,11 +367,6 @@ def structure_findings(text: str, field_profile_dir: Path | None, return findings -def structure_hits(text: str, field_profile_dir: Path | None - ) -> list[tuple[int, str, str]]: - return feedback.tuple_hits(structure_findings(text, field_profile_dir)) - - def calibrate(field_profile_dir: Path) -> dict[str, Any]: bank = field_profile_dir / "exemplar_paragraphs.jsonl" if not bank.exists(): From f211ecee108d8338b2133e6cb9b9a021414081ea Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:03:43 +0000 Subject: [PATCH 07/27] deai_partition: compare states by conformal p, split the original text, fix every heading Audit 2026-09-27, B1/B2/B3/B10/B11. - Candidate states are compared by their split-conformal p (higher = deeper inside the human band); raw manifold distance is only a tie-break between states scored on one calibration basis. The greedy step compared raw distances across a length-stratum switch, where the two manifolds' units differ, and stopped on p while advancing on distance. Every plan step records calibration_basis before and after, and the CLI marks a switch (B10). - A split is simulated on the original block text at the sentence boundary's own offset, line breaks kept. Re-joining es.sentences with spaces turned newlines into spaces, so a trailing `% note` commented out the rest of the paragraph and the ranking scored a half that had lost its prose (B2). The split halves now carry their real line ranges. - A block starting with any heading command (extract_sections.RE_HEADING_COMMAND) is fixed, not only `\section` (B3). - The measurable blocks are the ones document_shape scores, through the shared shape_units/measurable_sections filter (word floor, paragraphs per section, sections per document), so the starting distance is the one deai_docstructure reports for the same file (B11). - The field resolves through optional_field_dir; no profile or no manifold is exit 2 with a message naming the profile, and a missing file is exit 2, where a missing --field was a TypeError traceback (B1). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit 940b651455c86e887f157252377ee51c12f2ddc9) --- tests/test_deai_partition.py | 178 ++++++++++++++++++++++++++++++++++- tools/deai_partition.py | 177 +++++++++++++++++++++++----------- 2 files changed, 299 insertions(+), 56 deletions(-) diff --git a/tests/test_deai_partition.py b/tests/test_deai_partition.py index ff95203..06a9e7a 100644 --- a/tests/test_deai_partition.py +++ b/tests/test_deai_partition.py @@ -1,12 +1,17 @@ from __future__ import annotations -import sys +import contextlib +import io +import tempfile import unittest from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path +import deai_docstructure as ds import deai_partition as partition +import extract_style as es LENSING_A = ( "The aperture mass statistic isolates the tangential shear signal around " @@ -24,6 +29,22 @@ "a different telescope during the second observing season. Redshift " "completeness reached ninety per cent for the magnitude-limited sample." ) +# Four sentences of ~17 words: every half of a split stays above MIN_WORDS. +FOUR_SENTENCES = ( + "The aperture mass statistic isolates the tangential shear signal around " + "each candidate peak in the survey footprint. The same aperture mass " + "filter suppresses the large-scale shear gradient across the field of " + "view considered here. Both properties make the aperture mass estimator " + "suitable for substructure searches in the cluster outskirts we study. " + "We therefore weight the aperture mass filter by the inverse shear " + "variance in each cell of the map." +) + + +def uniform_document(paragraph: str, per_section: int = 3) -> str: + body = "\n\n".join([paragraph] * per_section) + return "\n\n".join(f"\\section{{{name}}}\n\n{body}" + for name in ("Introduction", "Methods", "Results")) + "\n" class PartitionTests(unittest.TestCase): @@ -41,7 +62,6 @@ def test_apply_merge_and_split_preserve_words(self): # paragraphs follow it self.assertTrue(sections[0]["blocks"][0]["fixed"]) self.assertEqual(len(sections[0]["blocks"]), 3) - import extract_style as es words_before = sorted( w for s in sections for b in s["blocks"] for w in es.words(es.latex_to_plain(b["text"]))) @@ -76,5 +96,159 @@ def test_suggest_degrades_honestly_without_manifold(self): self.assertEqual(result["status"], "unmeasured") +class SplitFidelityTests(unittest.TestCase): + """A split is simulated on the ORIGINAL block text at the sentence + boundary's own offset, line breaks kept (audit 2026-09-27, B2).""" + + BLOCK = ("Alpha one is measured here. Beta two follows there. % note kept\n" + "Gamma three continues the argument. Delta four closes the block.") + + def test_split_keeps_the_prose_after_a_trailing_comment(self): + left, right = partition._split_at(self.BLOCK, 2) + self.assertEqual(right, "Delta four closes the block.") + self.assertIn("\n", left, "the line break inside the left half survives") + # re-joined with spaces the `% note kept` would comment out Gamma + self.assertIn("Gamma", es.latex_to_plain(left)) + self.assertNotIn("Gamma", es.latex_to_plain( + " ".join(es.sentences(self.BLOCK)[:2]))) + # the k-th boundary here is the k-th sentence end of es.sentences + self.assertEqual(len(es.sentences(self.BLOCK)), 3) + self.assertEqual(partition._split_at(self.BLOCK, 1)[0], + "Alpha one is measured here.") + + def test_apply_split_scores_and_locates_the_real_halves(self): + sections = [{"label": "Methods", "blocks": [ + {"lines": (10, 11), "text": self.BLOCK}]}] + split = partition._apply(sections, {"kind": "split", "section": 0, + "block": 0, "cut": 2}) + left, right = split[0]["blocks"] + self.assertEqual((left["lines"], right["lines"]), ((10, 11), (11, 11))) + plain_words = sorted(w for b in (left, right) + for w in es.words(es.latex_to_plain(b["text"]))) + self.assertEqual(plain_words, + sorted(es.words(es.latex_to_plain(self.BLOCK)))) + description = partition._describe( + {"kind": "split", "section": 0, "block": 0, "cut": 2, "overlap": 0.0}, + sections) + self.assertIn("continues the argument", description) + + +class FixedBlockTests(unittest.TestCase): + def test_any_heading_command_fixes_its_block(self): + # only `\section` used to be fixed; a `\subsection` block was offered + # as a merge partner (audit 2026-09-27, B3) + text = ("\\section{Methods}\n\n" + LENSING_A + "\n\n" + "\\subsection{Weighting}\n\n" + LENSING_B + "\n\n" + "\\subsection*{Details}\n" + UNRELATED + "\n\n" + "\\chapter{Next}\n\n" + LENSING_A + "\n") + fixed = {block["text"].splitlines()[0]: block["fixed"] + for section in partition._parse(text) + for block in section["blocks"]} + self.assertTrue(fixed["\\section{Methods}"]) + self.assertTrue(fixed["\\subsection{Weighting}"]) + self.assertTrue(fixed["\\subsection*{Details}"]) + self.assertTrue(fixed["\\chapter{Next}"]) + self.assertFalse(fixed[LENSING_A]) + candidates = partition._merge_candidates(partition._parse(text), 0.0) + for op in candidates: + section = partition._parse(text)[op["section"]] + self.assertFalse(section["blocks"][op["block"]]["fixed"]) + self.assertFalse(section["blocks"][op["block"] + 1]["fixed"]) + + +def fake_operating_point(p_by_n: dict, distance_by_n: dict): + """A stand-in for `manifold_operating_point` keyed by paragraph count, + with the length stratum switching at ten paragraphs.""" + def operating(_baseline, _row, n_paragraphs): + return {"distance": distance_by_n[n_paragraphs], + "p_value": p_by_n[n_paragraphs], "alpha": 0.05, + "operating_point": "split-conformal, stratum manifold", + "calibration_basis": f"stratum {0 if n_paragraphs < 10 else 1} manifold", + "n_calibration": 40, "n_train": 60} + return operating + + +class BandComparisonTests(unittest.TestCase): + """Candidate states are compared by conformal p; distance only breaks a + tie within one calibration basis (audit 2026-09-27, B10).""" + + def test_improves_orders_by_p_then_by_distance_within_one_basis(self): + better_p = {"conformal_p": 0.3, "distance": 9.0, "calibration_basis": "stratum 1 manifold"} + worse_p = {"conformal_p": 0.1, "distance": 1.0, "calibration_basis": "stratum 0 manifold"} + self.assertTrue(partition._improves(better_p, worse_p)) + self.assertFalse(partition._improves(worse_p, better_p)) + same_basis_closer = {"conformal_p": 0.1, "distance": 0.5, + "calibration_basis": "stratum 0 manifold"} + other_basis_closer = {"conformal_p": 0.1, "distance": 0.5, + "calibration_basis": "stratum 1 manifold"} + self.assertTrue(partition._improves(same_basis_closer, worse_p)) + self.assertFalse(partition._improves(other_basis_closer, worse_p)) + # a legacy baseline (no conformal block) compares pooled distances + self.assertTrue(partition._improves({"distance": 1.0, "calibration_basis": "in-sample percentile"}, + {"distance": 2.0, "calibration_basis": "in-sample percentile"})) + + def test_plan_prefers_the_higher_p_and_records_the_basis(self): + text = uniform_document(FOUR_SENTENCES) + # start: 9 paragraphs; a merge gives 8, a split gives 10. The merge + # cuts the raw distance most, but the split moves the p up most and + # crosses the stratum edge; the plan must take the split and say so. + operating = fake_operating_point( + p_by_n={9: 0.01, 8: 0.02, 10: 0.04}, + distance_by_n={9: 3.0, 8: 1.0, 10: 3.5}) + with mock.patch.object(ds, "manifold_operating_point", operating): + result = partition.suggest(text, {"dispersion_manifold": {}}, 1) + self.assertEqual(result["status"], "measured") + self.assertEqual(result["start"]["n_paragraphs"], 9) + self.assertEqual(result["start"]["calibration_basis"], "stratum 0 manifold") + step = result["plan"][0] + self.assertEqual(step["kind"], "split") + self.assertEqual((step["conformal_p_before"], step["conformal_p_after"]), + (0.01, 0.04)) + self.assertEqual((step["calibration_basis_before"], step["calibration_basis_after"]), + ("stratum 0 manifold", "stratum 1 manifold")) + self.assertEqual((step["distance_before"], step["distance_after"]), (3.0, 3.5)) + + def test_state_uses_the_document_shape_filter(self): + # a section below MIN_PARAGRAPHS_PER_SECTION or a document below + # MIN_SECTIONS is not measurable here either (audit B11) + text = uniform_document(FOUR_SENTENCES) + operating = fake_operating_point(p_by_n={9: 0.5}, distance_by_n={9: 1.0}) + with mock.patch.object(ds, "manifold_operating_point", operating): + start = partition._state_statistics(partition._parse(text), {}) + self.assertEqual(start["n_paragraphs"], + ds.document_shape(text)["n_paragraphs"]) + two_sections = "\n\n".join(text.split("\n\n\\section{Results}")[:1]) + "\n" + self.assertIsNone(partition._state_statistics( + partition._parse(two_sections), {})) + thin = uniform_document(FOUR_SENTENCES, per_section=1) + self.assertIsNone(partition._state_statistics(partition._parse(thin), {})) + + +class CliContractTests(unittest.TestCase): + def run_main(self, argv: list[str]) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = partition.main(argv) + return code, out.getvalue(), err.getvalue() + + def test_without_a_profile_the_tool_exits_2_with_a_message(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text(uniform_document(FOUR_SENTENCES), encoding="utf-8") + empty = root / "profiles" + empty.mkdir() + code, _out, err = self.run_main([str(draft), "--profile-root", str(empty)]) + self.assertEqual(code, 2) + self.assertIn("no calibrated dispersion manifold", err) + code, _out, err = self.run_main([str(draft), "--profile-root", str(empty), + "--field", "nope"]) + self.assertEqual(code, 2) + self.assertIn("nope", err) + code, _out, err = self.run_main([str(root / "missing.tex"), + "--profile-root", str(empty)]) + self.assertEqual((code, "file not found" in err), (2, True)) + + if __name__ == "__main__": unittest.main() diff --git a/tools/deai_partition.py b/tools/deai_partition.py index 5997dc3..76b51e7 100644 --- a/tools/deai_partition.py +++ b/tools/deai_partition.py @@ -8,13 +8,15 @@ fire, by construction. This tool only SUGGESTS operations, ranked by how far they move the document -toward the calibrated human dispersion band (Mahalanobis manifold distance, -two-sided: over-uniform documents get variety-creating ops, over-dispersed -ones get smoothing ops). The author applies them by hand where the argument -allows. Cohesion is self-normalized against the document itself: a merge must -join paragraphs at least as lexically related as the document's median -adjacent pair, and a split must cut at a boundary no more related than that -same median - no tuned constants. +into the calibrated human dispersion band: states are compared by their +split-conformal p against the human reference (higher = deeper inside the +band; two-sided, so over-uniform documents get variety-creating ops and +over-dispersed ones get smoothing ops), with the raw Mahalanobis distance as +a tie-break only between states scored on one calibration basis. The author +applies them by hand where the argument allows. Cohesion is self-normalized +against the document itself: a merge must join paragraphs at least as +lexically related as the document's median adjacent pair, and a split must +cut at a boundary no more related than that same median - no tuned constants. Reordering paragraphs is deliberately NOT offered: it changes reading order, and admissibility would need a real claim-dependency analysis; guessing one @@ -26,7 +28,6 @@ from __future__ import annotations -import argparse import json import statistics import sys @@ -36,7 +37,7 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first import deai_docstructure as ds # noqa: E402 because sibling imports resolve only after the sys.path insert above import deai_features as features # noqa: E402 same sys.path reason -import deai_metrics as metrics # noqa: E402 same sys.path reason +import extract_sections # noqa: E402 same sys.path reason; RE_HEADING_COMMAND import extract_style as es # noqa: E402 same sys.path reason MIN_SPLIT_SENTENCES = 4 # both halves must keep >= 2 sentences @@ -57,35 +58,44 @@ def _overlap(a: str, b: str) -> float: def _parse(text: str) -> list[dict]: - """Sections as ordered raw paragraph blocks with original line spans.""" - lines = text.splitlines() + """Sections as ordered raw paragraph blocks with original line spans. + + The blocks are cut from `deai_docshape.shape_units`, so the preamble and + every `skip` unit are absent here exactly as they are absent from the + measurement a plan is scored on. A block that starts with a heading + command is a fixed landmark: merging or splitting into it would be a + nonsense suggestion. Any heading command counts (`\\subsection`, + `\\chapter`, ...); until 2026-09-27 only `\\section` did, and a subsection + heading was offered as a merge partner. + """ sections = [] - for start, end, label in metrics.section_line_ranges(text): - if label == "(preamble)": - continue - segment = "\n".join(lines[start - 1:end]) - # a block holding the \section command itself is a fixed landmark: - # merging or splitting into it would be a nonsense suggestion + for unit in ds.shape_units(text): blocks = [{"lines": (p_start, p_end), "text": block, - "fixed": block.lstrip().startswith("\\section")} - for p_start, p_end, block in - metrics.paragraph_line_ranges(segment, start)] + "fixed": bool(extract_sections.RE_HEADING_COMMAND.match( + block.lstrip()))} + for p_start, p_end, block in unit["blocks"]] if blocks: - sections.append({"label": label, "blocks": blocks}) + sections.append({"label": unit["label"], "blocks": blocks}) return sections -def _measurable(block_text: str) -> bool: - return len(es.words(es.latex_to_plain(block_text))) >= ds.MIN_WORDS - - def _state_statistics(sections: list[dict], baseline: dict) -> dict | None: - """Manifold distance (+ conformal p when available) of a partition state.""" - vectors = [ds._paragraph_modelfree(block["text"]) - for section in sections for block in section["blocks"] - if _measurable(block["text"])] - if len(vectors) < 2: + """Manifold distance (+ conformal p when available) of a partition state. + + The blocks scored are exactly the ones `document_shape` scores, through + the shared `measurable_sections` filter (word floor, paragraphs per + section, sections per document), so the starting distance printed here + is the one `deai_docstructure` reports for the same file. `calibration_basis` + names the manifold and calibration set the numbers came from; a state + below the document-shape floor, or one the axis cannot score, is None. + """ + units = [{"blocks": [(b["lines"][0], b["lines"][1], b["text"]) + for b in section["blocks"]]} for section in sections] + kept = ds.measurable_sections(units) + if len(kept) < ds.MIN_SECTIONS: return None + vectors = [ds._paragraph_modelfree(block) + for unit in kept for _start, _end, block in unit["blocks"]] dispersion = features.cross_paragraph_dispersion( vectors, list(ds.DISPERSION_FEATURE_NAMES)) row = {name: dispersion[name][ds.DISPERSION_STAT] @@ -94,14 +104,31 @@ def _state_statistics(sections: list[dict], baseline: dict) -> dict | None: operating = ds.manifold_operating_point(baseline, row, len(vectors)) if operating is None: return None - out = {"distance": operating["distance"], "n_paragraphs": len(vectors)} + out = {"distance": operating["distance"], "n_paragraphs": len(vectors), + "calibration_basis": (operating.get("calibration_basis") + or operating["operating_point"])} if operating["alpha"] is not None: out["conformal_p"] = operating["p_value"] - out["calibration_basis"] = operating["calibration_basis"] out["alpha"] = operating["alpha"] return out +def _split_at(block_text: str, cut: int) -> tuple[str, str]: + """The block cut after its `cut`-th sentence, at that boundary's own offset. + + The boundary is located in the ORIGINAL block text with the pattern + `es.sentences` splits on, so the k-th boundary here is the k-th sentence + end there, and both halves keep their line breaks. Re-joining the + sentence list with spaces turned every newline into a space, so a + `% note` at the end of a line commented out the rest of the paragraph: + the ranking and the quoted `distance_after` were computed on a half that + had lost its prose. The suggestion was always zero-token; the state being + scored was not the state the author would get. + """ + boundary = list(es.RE_SENTENCE_END.finditer(block_text))[cut - 1] + return block_text[:boundary.start()], block_text[boundary.end():] + + def _merge_candidates(sections: list[dict], floor: float) -> list[dict]: ops = [] for s_index, section in enumerate(sections): @@ -128,8 +155,7 @@ def _split_candidates(sections: list[dict], floor: float) -> list[dict]: continue best = None for cut in range(2, len(parts) - 1): - left = " ".join(parts[:cut]) - right = " ".join(parts[cut:]) + left, right = _split_at(block["text"], cut) overlap = _overlap(left, right) if best is None or overlap < best[1]: best = (cut, overlap) @@ -152,12 +178,13 @@ def _apply(sections: list[dict], op: dict) -> list[dict]: blocks[op["block"]:op["block"] + 2] = [merged] else: block = blocks[op["block"]] - parts = es.sentences(block["text"]) - left = " ".join(parts[:op["cut"]]) - right = " ".join(parts[op["cut"]:]) + left, right = _split_at(block["text"], op["cut"]) + start, end = block["lines"] blocks[op["block"]:op["block"] + 1] = [ - {"lines": block["lines"], "text": left, "derived": "split-left"}, - {"lines": block["lines"], "text": right, "derived": "split-right"}, + {"lines": (start, start + left.count("\n")), "text": left, + "derived": "split-left"}, + {"lines": (end - right.count("\n"), end), "text": right, + "derived": "split-right"}, ] return new_sections @@ -172,14 +199,37 @@ def _describe(op: dict, sections: list[dict]) -> str: f"{second['lines'][0]}-{second['lines'][1]} " f"(delete the blank line; lexical overlap {op['overlap']:.2f})") block = section["blocks"][op["block"]] - parts = es.sentences(block["text"]) - tail = " ".join(es.words(es.latex_to_plain(parts[op["cut"] - 1]))[-6:]) + left, _right = _split_at(block["text"], op["cut"]) + tail = " ".join(es.words(es.latex_to_plain(left))[-6:]) return (f"SPLIT in [{section['label']}]: paragraph at lines " f"{block['lines'][0]}-{block['lines'][1]}, insert a blank line " f"after sentence {op['cut']} (ending '...{tail}'; boundary " f"overlap {op['overlap']:.2f})") +def _improves(trial: dict, over: dict) -> bool: + """Whether `trial` sits deeper inside the human band than `over`. + + States are compared by conformal p, which stays meaningful when a plan + step changes the paragraph count and so crosses a length-stratum edge. + Raw manifold distance is only a tie-break, and only between states scored + on one calibration basis: a distance from the stratum-1 manifold and one + from the stratum-2 manifold are different quantities + (`deai_docshape.manifold_operating_point`). Until 2026-09-27 the greedy + step compared raw distances across that switch, and stopped on p while + advancing on distance. A legacy baseline without a conformal block has + one pooled manifold, so there its distances are the comparison. + """ + if "conformal_p" in trial and "conformal_p" in over: + if trial["conformal_p"] != over["conformal_p"]: + return trial["conformal_p"] > over["conformal_p"] + return (trial["calibration_basis"] == over["calibration_basis"] + and trial["distance"] < over["distance"]) + if "conformal_p" in trial or "conformal_p" in over: + return False # one scored conformally, one not: no common scale + return trial["distance"] < over["distance"] + + def suggest(text: str, baseline: dict, max_ops: int) -> dict: """Greedy band-seeking plan of admissible merge/split operations.""" sections = _parse(text) @@ -205,19 +255,23 @@ def suggest(text: str, baseline: dict, max_ops: int) -> dict: best = None for op in candidates: trial = _state_statistics(_apply(state, op), baseline) - if trial is None: + if trial is None or not _improves(trial, current): continue - if trial["distance"] < current["distance"] and ( - best is None or trial["distance"] < best[1]["distance"]): + if best is None or _improves(trial, best[1]): best = (op, trial) if best is None: break op, trial = best + # Every step records the basis on both sides, so a manifold switch + # between two consecutive distances is visible in the plan itself. plan.append({"description": _describe(op, state), "kind": op["kind"], "distance_before": current["distance"], "distance_after": trial["distance"], - "conformal_p_after": trial.get("conformal_p")}) + "conformal_p_before": current.get("conformal_p"), + "conformal_p_after": trial.get("conformal_p"), + "calibration_basis_before": current["calibration_basis"], + "calibration_basis_after": trial["calibration_basis"]}) state, current = _apply(state, op), trial return {"status": "measured", "start": start, "end": current, "cohesion_floor": floor, "plan": plan} @@ -231,10 +285,16 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("--json", type=Path, default=None, help="also write the plan as JSON") args = parser.parse_args(argv) - baseline = ds.load_baseline(args.profile_root / args.field) + if not args.file.exists(): + print(f"[deai_partition] file not found: {args.file}", file=sys.stderr) + return 2 + field_dir = cli_common.optional_field_dir(args, tool="deai_partition") + baseline = ds.load_baseline(field_dir) if baseline is None or not baseline.get("dispersion_manifold"): - print("[deai_partition] no calibrated dispersion manifold for this " - "field; run deai_docstructure --calibrate first", file=sys.stderr) + where = f"under {field_dir}" if field_dir else "(no field profile resolved)" + print(f"[deai_partition] no calibrated dispersion manifold {where}; " + "run deai_docstructure --calibrate --field first", + file=sys.stderr) return 2 text = args.file.read_text(encoding="utf-8", errors="replace") result = suggest(text, baseline, args.max_ops) @@ -245,7 +305,8 @@ def main(argv: list[str] | None = None) -> int: return 0 start, end = result["start"], result["end"] print(f"[deai_partition] {args.file}") - line = f" manifold distance {start['distance']:.3f}" + line = (f" manifold distance {start['distance']:.3f} " + f"({start['calibration_basis']})") if "conformal_p" in start: line += f", conformal p {start['conformal_p']:.4f} (alpha {start['alpha']:g})" print(line) @@ -254,13 +315,18 @@ def main(argv: list[str] | None = None) -> int: and start["conformal_p"] > start["alpha"]): print(" already inside the human band; nothing to fix") else: - print(" no admissible partition operation improves the band " - "distance") + print(" no admissible partition operation moves the document " + "deeper into the band") return 0 for index, step in enumerate(result["plan"], 1): print(f" {index}. {step['description']}") - extra = (f" -> p {step['conformal_p_after']:.4f}" + extra = (f", p {step['conformal_p_before']:.4f} -> " + f"{step['conformal_p_after']:.4f}" if step.get("conformal_p_after") is not None else "") + if step["calibration_basis_before"] != step["calibration_basis_after"]: + extra += (f" [calibration basis {step['calibration_basis_before']} -> " + f"{step['calibration_basis_after']}: the two distances " + "are not on one scale]") print(f" distance {step['distance_before']:.3f} -> " f"{step['distance_after']:.3f}{extra}") verdict = "" @@ -268,8 +334,11 @@ def main(argv: list[str] | None = None) -> int: verdict = (" (inside the human band)" if end["conformal_p"] > end["alpha"] else " (still outside the band; apply and re-measure)") - print(f" projected: distance {start['distance']:.3f} -> " - f"{end['distance']:.3f}{verdict}") + print(f" projected: conformal p {start['conformal_p']:.4f} -> " + f"{end['conformal_p']:.4f}{verdict}") + else: + print(f" projected: distance {start['distance']:.3f} -> " + f"{end['distance']:.3f}") print(" suggestions only - apply by hand where the argument allows; " "merge/split change zero tokens, so fidelity is preserved by " "construction.") From 2d2685d14a784085c7b86cdcbd88c4851f3ca54d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:03:43 +0000 Subject: [PATCH 08/27] deai_anchoring: a class below its Bonferroni share is skipped and reported, not measured Audit 2026-09-27, B1/B5/B13. - A conformal p is never below 1/(n+1). With n = 30 per class and k >= 2 classes measured, 1/31 = 0.032 exceeds alpha/k = 0.025, so no document could ever flag and the axis still reported `measured`. Detection now computes which classes the baseline can test (class_tests): an underpowered class is skipped and the axis is `degraded` with the reason; a document whose classes have no reference at all is `unmeasured`. The share is not re-spread over the remaining classes, which keeps the document-level rate <= alpha. Calibration records min_class_documents_for_share and underpowered_classes in the baseline and the CLI prints them as a warning (B5). - The sentence scan runs on the section units every per-bucket axis reads, preamble and `skip` units dropped, headings and floats blanked through deai_reference.without_headings and the bibliography blanked by a local pattern (deai_reference holds none and is not edited). On the raw LaTeX a heading fused with the first sentence below it and a thebibliography block anchored the last sentence with every year in the reference list (B13). - --calibrate resolves its field through resolve_field, the read path through optional_field_dir; a missing --field was a TypeError traceback (B1). The anchoring baseline must be rebuilt: its rates were measured on the raw text. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit 8932ee2be319d8ac92bac66e285cf205fd9d7377) --- tests/test_deai_anchoring.py | 153 +++++++++++++++++++++++++++++++++- tools/deai_anchoring.py | 155 ++++++++++++++++++++++++++++------- 2 files changed, 279 insertions(+), 29 deletions(-) diff --git a/tests/test_deai_anchoring.py b/tests/test_deai_anchoring.py index 4e3fa8a..ddd6d36 100644 --- a/tests/test_deai_anchoring.py +++ b/tests/test_deai_anchoring.py @@ -1,6 +1,8 @@ from __future__ import annotations -import sys +import contextlib +import io +import json import tempfile import unittest from pathlib import Path @@ -32,6 +34,15 @@ def document(section_bodies: dict[str, str]) -> str: return "\n\n".join(parts) + "\n" +def write_baseline(root: Path, classes: dict[str, tuple[list[float], int]]) -> None: + """A hand-written baseline: {class: (values, n)}.""" + (root / anchoring.BASELINE_NAME).write_text(json.dumps({ + "schema": "sci-paper.anchoring-baseline.v1", "alpha": 0.05, + "n_documents": max(n for _v, n in classes.values()), + "classes": {cls: {"values": values, "n": n, "median": values[len(values) // 2]} + for cls, (values, n) in classes.items()}}), encoding="utf-8") + + class AnchoringTests(unittest.TestCase): def test_sentence_anchor_detection(self): self.assertTrue(anchoring.sentence_is_anchored( @@ -77,6 +88,7 @@ def test_unanchored_results_flag_and_anchored_do_not(self): sources.append(path) baseline = anchoring.calibrate(sources, root) self.assertIn("results", baseline["classes"]) + self.assertEqual(baseline["underpowered_classes"], {}) flagged = anchoring.anchoring_findings(document({ "Introduction": ANCHORED, "Methods": ANCHORED, "Results": UNANCHORED}), root) @@ -88,11 +100,150 @@ def test_unanchored_results_flag_and_anchored_do_not(self): "Results": ANCHORED}), root) self.assertFalse(clean, "an anchored document must not flag any class") + self.assertEqual(anchoring.anchoring_axis_status( + document({"Results": UNANCHORED}), root)["status"], "measured") def test_axis_degrades_honestly(self): status = anchoring.anchoring_axis_status("too short", None) self.assertEqual(status["status"], "unmeasured") +class PowerFloorTests(unittest.TestCase): + """A conformal p is never below 1/(n+1); a class whose n cannot reach its + Bonferroni share alpha/k cannot flag and must not pass as `measured` + (audit 2026-09-27, B5).""" + + def test_min_documents_for_share(self): + # n + 1 >= k / alpha: k=1 -> 19, k=2 -> 39, k=3 -> 59 + self.assertEqual(anchoring.min_class_documents_for_share(0.05, 1), 19) + self.assertEqual(anchoring.min_class_documents_for_share(0.05, 2), 39) + self.assertEqual(anchoring.min_class_documents_for_share(0.05, 3), 59) + + def test_underpowered_class_is_skipped_and_axis_degraded(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + # results: n = 100 (1/101 < 0.025); intro: n = 30 (1/31 = 0.032 > 0.025) + write_baseline(root, {"results": ([0.8] * 100, 100), + "intro": ([0.5] * 30, 30)}) + text = document({"Introduction": UNANCHORED, "Results": UNANCHORED}) + tests = anchoring.class_tests(anchoring.load_baseline(root), + anchoring.document_anchoring(text)) + self.assertEqual(tests["underpowered"], ["intro"]) + self.assertEqual(tests["testable"], ["results"]) + self.assertAlmostEqual(tests["class_alpha"], 0.025) + findings = anchoring.anchoring_findings(text, root) + self.assertEqual([f["rule"] for f in findings], + ["claim-anchoring:results"]) + # the share is not re-spread: still alpha / 2 + self.assertAlmostEqual(findings[0]["reference"]["alpha_class"], 0.025) + status = anchoring.anchoring_axis_status(text, root) + self.assertEqual(status["status"], "degraded") + self.assertIn("intro", status["reason"]) + self.assertIn("1/(n+1)", status["reason"]) + # the same intro class alone in a document (k = 1, share 0.05) is + # testable: 1/31 = 0.032 <= 0.05 + alone = document({"Introduction": UNANCHORED}) + self.assertEqual(anchoring.anchoring_axis_status(alone, root)["status"], + "measured") + self.assertEqual([f["rule"] for f in anchoring.anchoring_findings(alone, root)], + ["claim-anchoring:intro"]) + + def test_no_calibrated_class_is_unmeasured(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + write_baseline(root, {"results": ([0.8] * 100, 100)}) + status = anchoring.anchoring_axis_status( + document({"Introduction": UNANCHORED}), root) + self.assertEqual(status["status"], "unmeasured") + self.assertIn("no measured section class", status["reason"]) + + def test_calibration_records_underpowered_classes(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + sources = [] + for index in range(anchoring.MIN_CLASS_DOCUMENTS): + path = root / f"human-{index}.tex" + path.write_text(document({ + "Introduction": ANCHORED, "Methods": ANCHORED, + "Results": ANCHORED}), encoding="utf-8") + sources.append(path) + baseline = anchoring.calibrate(sources, root) + # three classes of 30: the share alpha/3 needs 59 per class + self.assertEqual(baseline["min_class_documents_for_share"], 59) + self.assertEqual(baseline["underpowered_classes"], + {"intro": 30, "methods": 30, "results": 30}) + text = document({"Introduction": ANCHORED, "Methods": ANCHORED, + "Results": UNANCHORED}) + self.assertEqual(anchoring.anchoring_axis_status(text, root)["status"], + "degraded") + self.assertEqual(anchoring.anchoring_findings(text, root), []) + + +class SweepTests(unittest.TestCase): + """The sentence scan runs on section bodies with headings, floats and the + bibliography blanked, and `skip` units dropped (audit 2026-09-27, B13).""" + + def test_heading_does_not_anchor_the_first_sentence(self): + # No blank line after the heading: the raw scan fused "Results at 3 + # arcmin" with the first sentence, and the digit anchored it. + result = anchoring.document_anchoring( + "\\section{Results at 3 arcmin}\n" + UNANCHORED + "\n") + self.assertEqual(result["classes"], {"results": 0.0}) + + def test_bibliography_and_floats_do_not_anchor_the_last_sentence(self): + text = ("\\section{Conclusions}\n\n" + UNANCHORED + "\n" + "\\begin{figure}\n\\caption{Bias at 3 arcmin, see 2020 data}\n" + "\\end{figure}\n" + "\\begin{thebibliography}{99}\n" + "\\bibitem{a} Smith, J. 2020, ApJ, 900, 1\n" + "\\bibitem{b} Jones, K. 2021, MNRAS, 500, 2\n" + "\\end{thebibliography}\n\\bibliography{refs}\n") + result = anchoring.document_anchoring(text) + self.assertEqual(result["classes"], {"conclusions": 0.0}) + self.assertEqual(result["sections"][0]["n_sentences"], 5) + + def test_skip_units_and_preamble_are_not_measured(self): + text = ("\\title{A 2020 paper}\n\n" + UNANCHORED + "\n\n" + + document({"Results": UNANCHORED, "Acknowledgments": ANCHORED, + "Appendix A": ANCHORED})) + result = anchoring.document_anchoring(text) + self.assertEqual([row["label"] for row in result["sections"]], ["Results"]) + self.assertEqual(result["classes"], {"results": 0.0}) + + +class CliContractTests(unittest.TestCase): + def run_main(self, argv: list[str]) -> tuple[int, str, str]: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = anchoring.main(argv) + return code, out.getvalue(), err.getvalue() + + def test_read_path_without_a_field_is_unmeasured_not_a_traceback(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text(document({"Results": UNANCHORED}), encoding="utf-8") + empty = root / "profiles" + empty.mkdir() + code, out, _err = self.run_main([str(draft), "--profile-root", str(empty)]) + self.assertEqual(code, 0) + self.assertIn("unmeasured", out) + code, _out, err = self.run_main([str(root / "missing.tex"), + "--profile-root", str(empty)]) + self.assertEqual((code, "file not found" in err), (2, True)) + + def test_calibrate_needs_an_existing_field(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + corpus = root / "corpus" + corpus.mkdir() + empty = root / "profiles" + empty.mkdir() + code, _out, err = self.run_main( + ["--calibrate", "--corpus-dir", str(corpus), "--profile-root", str(empty)]) + self.assertEqual(code, 2) + self.assertIn("No field profiles", err) + + if __name__ == "__main__": unittest.main() diff --git a/tools/deai_anchoring.py b/tools/deai_anchoring.py index 8983041..db6acbf 100644 --- a/tools/deai_anchoring.py +++ b/tools/deai_anchoring.py @@ -19,7 +19,10 @@ document baseline). Detection is a low-tail conformal p per class with a Bonferroni share alpha/k (k = classes measured in the document), so the DOCUMENT-level false-flag rate stays <= alpha for exchangeable human papers. -Below the per-class minimum the class is honestly omitted. +Below the per-class minimum the class is honestly omitted. A conformal p is +never smaller than 1/(n+1), so a class whose n cannot reach its share +alpha/k can never flag: such a class is skipped and the axis reported +`degraded`, never `measured` with nothing to say. Usage: python tools/deai_anchoring.py --field wgl --calibrate --corpus-dir style-corpus/wgl @@ -28,8 +31,8 @@ from __future__ import annotations -import argparse import json +import math import re import statistics import sys @@ -56,6 +59,12 @@ _NUMBER_RE = re.compile(r"\d") _COMPARISON_MARKERS = (" than ", "compared to", "compared with", "relative to", "in contrast", "versus ") +# The reference list, blanked before the sentence scan (see `document_anchoring`). +# `deai_reference` blanks headings and floats and holds no bibliography +# pattern; this one is local so that module stays untouched. +_BIBLIOGRAPHY_RE = re.compile( + r"\\begin\{thebibliography\}.*?\\end\{thebibliography\}" + r"|\\bibliography(?:style)?\s*\{[^}]*\}", re.DOTALL) # section-title keyword -> class; first match wins, unmatched -> "other" SECTION_CLASSES = ( @@ -103,14 +112,31 @@ def section_anchor_rate(section_source: str) -> tuple[float, int] | None: def document_anchoring(text: str) -> dict[str, Any]: - """Per-class anchor rates for one document (one value per class).""" + """Per-class anchor rates for one document (one value per class). + + The sweep is the one every per-bucket axis reads + (`deai_metrics.section_units`): the preamble and every `skip` unit + (acknowledgements, appendices, the reference list) are dropped, and in + each remaining unit the headings and floats are blanked by + `deai_reference.without_headings` and the bibliography by the local + pattern above, line count preserved. Until 2026-09-27 the sentence scan + ran on the raw LaTeX of `section_line_ranges`: a heading fused with the + first sentence below it, and a `thebibliography` block under the last + section glued to its final sentence and anchored it with every year in + the reference list (a Conclusions rate of 0.0 read as 0.2). An anchoring + baseline built before the change must be rebuilt: its rates were measured + on the raw text. The class still comes from the raw section title, + through this module's own keyword table. + """ lines = text.splitlines() per_class: dict[str, list[float]] = {} section_rows = [] - for start, end, label in metrics.section_line_ranges(text): - if label == "(preamble)": + for start, end, label, bucket in metrics.section_units(text): + if bucket in (metrics.PREAMBLE_BUCKET, "skip"): continue - segment = "\n".join(lines[start - 1:end]) + segment = es.blank_preserving( + reference.without_headings("\n".join(lines[start - 1:end])), + _BIBLIOGRAPHY_RE) measured = section_anchor_rate(segment) if measured is None: continue @@ -127,9 +153,22 @@ def document_anchoring(text: str) -> dict[str, Any]: "sections": section_rows} +def min_class_documents_for_share(alpha: float, n_classes: int) -> int: + """The smallest class n whose conformal floor 1/(n+1) fits alpha/k.""" + return max(0, math.ceil(n_classes / alpha - 1)) + + def calibrate(documents: Iterable[tuple[str, str] | Path], field_profile_dir: Path) -> dict[str, Any]: - """Fit the section-class conditional human anchoring reference.""" + """Fit the section-class conditional human anchoring reference. + + `underpowered_classes` lists every retained class whose n is below the + floor for the Bonferroni share when a document measures ALL retained + classes (the worst case, k = the number of classes in this baseline): + such a class can never produce a finding, and detection reports the axis + `degraded` whenever a document measures one. The CLI prints the list as a + warning; MIN_CLASS_DOCUMENTS alone (30) supports only k = 1 at alpha 0.05. + """ class_values: dict[str, list[float]] = {} n_documents = 0 for item in documents: @@ -162,6 +201,11 @@ def calibrate(documents: Iterable[tuple[str, str] | Path], "n": len(values), "median": statistics.median(values), } + floor = min_class_documents_for_share(ANCHORING_ALPHA, len(baseline["classes"])) + baseline["min_class_documents_for_share"] = floor + baseline["underpowered_classes"] = { + cls: entry["n"] for cls, entry in baseline["classes"].items() + if entry["n"] < floor} output = field_profile_dir / BASELINE_NAME output.write_text(json.dumps(baseline, indent=2), encoding="utf-8") return baseline @@ -170,6 +214,32 @@ def calibrate(documents: Iterable[tuple[str, str] | Path], load_baseline = reference.baseline_loader(BASELINE_NAME) +def class_tests(baseline: dict[str, Any], result: dict[str, Any] + ) -> dict[str, Any]: + """Which of a document's classes the baseline can test, and at what alpha. + + `measured` are the document's classes that have a reference; the + Bonferroni share is alpha over their count. A class whose reference is + too small for that share -- its conformal floor 1/(n+1) exceeds alpha/k, + so no observed rate could ever flag it -- goes to `underpowered` and is + not tested. The share is NOT re-spread over the remaining classes: testing + fewer classes at the same per-class alpha keeps the document-level rate + at or below alpha, and re-spreading would make which classes are tested + depend on the order they were dropped in. With n = 30 per class (the + retention minimum) and two classes measured, 1/31 = 0.032 > 0.025: every + finding was impossible and the axis still said `measured`. + """ + alpha = float(baseline.get("alpha", ANCHORING_ALPHA)) + references = baseline.get("classes", {}) + measured = [cls for cls in result["classes"] if references.get(cls)] + class_alpha = alpha / len(measured) if measured else alpha + underpowered = [cls for cls in measured + if 1.0 / (int(references[cls]["n"]) + 1) > class_alpha] + return {"alpha": alpha, "class_alpha": class_alpha, "measured": measured, + "underpowered": underpowered, + "testable": [cls for cls in measured if cls not in underpowered]} + + def anchoring_axis_status(text: str, field_profile_dir: Path | None ) -> dict[str, Any]: result = document_anchoring(text) @@ -177,10 +247,26 @@ def anchoring_axis_status(text: str, field_profile_dir: Path | None return feedback.axis_status("L2.claim_anchoring", "unmeasured", reason="no section long enough to measure", detector="deai_anchoring") - if load_baseline(field_profile_dir) is None: + baseline = load_baseline(field_profile_dir) + if baseline is None: return feedback.axis_status("L2.claim_anchoring", "unmeasured", reason=f"{BASELINE_NAME} is unavailable", detector="deai_anchoring") + tests = class_tests(baseline, result) + if not tests["measured"]: + return feedback.axis_status( + "L2.claim_anchoring", "unmeasured", + reason="no measured section class has a calibrated reference", + detector="deai_anchoring") + if tests["underpowered"]: + n_classes = len(tests["measured"]) + return feedback.axis_status( + "L2.claim_anchoring", "degraded", + reason=(f"class(es) {', '.join(tests['underpowered'])} hold too few " + f"calibration papers for the Bonferroni share alpha/" + f"{n_classes} = {tests['class_alpha']:.4f} (the conformal p " + "cannot go below 1/(n+1)); they were skipped, not tested"), + detector="deai_anchoring") return feedback.axis_status("L2.claim_anchoring", "measured", detector="deai_anchoring") @@ -191,22 +277,17 @@ def anchoring_findings(text: str, field_profile_dir: Path | None, result = document_anchoring(text) if baseline is None or result["status"] != "measured": return [] - alpha = float(baseline.get("alpha", ANCHORING_ALPHA)) # Bonferroni share: a document measures up to k classes; testing each at # alpha would give a document-level union false-flag rate near k*alpha # (measured 0.170 at alpha=0.05 before this correction), so each class # tests at alpha/k and the document-level rate stays <= alpha. - measured_classes = [cls for cls in result["classes"] - if baseline.get("classes", {}).get(cls)] - if not measured_classes: - return [] - class_alpha = alpha / len(measured_classes) + tests = class_tests(baseline, result) + alpha, class_alpha = tests["alpha"], tests["class_alpha"] findings = [] - for cls, observed in result["classes"].items(): - reference = baseline.get("classes", {}).get(cls) - if not reference: - continue - values = reference["values"] + for cls in tests["testable"]: + observed = result["classes"][cls] + class_reference = baseline["classes"][cls] + values = class_reference["values"] # low-tail conformal p: rank of the observed rate among human rates p_value = (1 + sum(v <= observed for v in values)) / (len(values) + 1) if p_value > class_alpha: @@ -231,19 +312,19 @@ def anchoring_findings(text: str, field_profile_dir: Path | None, reference={"operating_point": "conformal low tail, Bonferroni", "alpha_document": alpha, "alpha_class": class_alpha, - "n_classes_tested": len(measured_classes), - "n_calibration": reference["n"], - "human_median": reference["median"], + "n_classes_tested": len(tests["measured"]), + "n_calibration": class_reference["n"], + "human_median": class_reference["median"], "provenance": BASELINE_NAME}, normalized_distance=class_alpha - p_value, - confidence={"value": min(1.0, reference["n"] / 100.0), - "basis": (f"conformal p against {reference['n']} human " - f"papers' {cls}-class anchor rates; " + confidence={"value": min(1.0, class_reference["n"] / 100.0), + "basis": (f"conformal p against {class_reference['n']} " + f"human papers' {cls}-class anchor rates; " f"P(false flag) <= {alpha:g} finite-sample")}, message=(f"Only {observed:.0%} of {cls} sentences carry a " f"checkable anchor (a number, citation, internal " f"reference, equation, or comparison); the human median " - f"for {cls} sections is {reference['median']:.0%} and " + f"for {cls} sections is {class_reference['median']:.0%} and " f"this document sits below the human low tail " f"(conformal p = {p_value:.4f}). Unanchored claims read " f"as unfalsifiable generality. This is a measured " @@ -268,24 +349,42 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("--calibrate", action="store_true") parser.add_argument("--corpus-dir", type=Path) args = parser.parse_args(argv) - field_dir = args.profile_root / args.field if args.calibrate: if args.corpus_dir is None or not args.corpus_dir.exists(): print("[deai_anchoring] --calibrate requires --corpus-dir", file=sys.stderr) return 2 + # The baseline is written INTO the profile, so calibration needs a + # field that exists; the read path below runs without one. try: - baseline = calibrate(_paper_documents(args.corpus_dir), field_dir) + field = cli_common.resolve_field( + args.field, args.profile_root, tool="deai_anchoring", + empty_hint="--calibrate needs an existing style-profile// directory.") + except SystemExit as error: + print(error, file=sys.stderr) + return 2 + try: + baseline = calibrate(_paper_documents(args.corpus_dir), + args.profile_root / field) except ValueError as error: print(f"[deai_anchoring] {error}", file=sys.stderr) return 2 classes = {cls: ref["n"] for cls, ref in baseline["classes"].items()} print(f"[deai_anchoring] baseline written from " f"{baseline['n_documents']} documents; classes {classes}") + if baseline["underpowered_classes"]: + print(f"[deai_anchoring] warning: classes " + f"{baseline['underpowered_classes']} hold fewer than " + f"{baseline['min_class_documents_for_share']} documents, the " + f"floor for the Bonferroni share alpha/{len(classes)} at alpha " + f"{baseline['alpha']:g}; a document measuring them will report " + "the axis degraded and those classes cannot flag", + file=sys.stderr) return 0 if args.file is None or not args.file.exists(): print(f"[deai_anchoring] file not found: {args.file}", file=sys.stderr) return 2 + field_dir = cli_common.optional_field_dir(args, tool="deai_anchoring") text = args.file.read_text(encoding="utf-8", errors="replace") report = feedback.build_report( path=args.file, From de925a916d309ca2570906dfb477a4a1bb364804 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:07:34 +0000 Subject: [PATCH 09/27] docs: registry rows and architecture text for the rebuilt axes, and the small skill and record fixes - tools/README and DEAI_SUBSYSTEM describe the document-level axes as they now behave: an axis that cannot fire is degraded (metrics, structure, anchoring's Bonferroni share), one quantile rule per document-shape threshold, skip units dropped from the shape sweep, partition ranked by conformal p on the original text, the wrap-up beat, the UID features unmeasured without the runtime; - skills: the paper skill marks its ML and detection guidance [WGL] and targets the field's journals; calibrate is "not a policy source"; the condense and calibrate hints list only flags their bodies define; brainstorm no longer narrates removed flags; the exemplar retrieval step names --allow-fallback; the negative-label floor is stated where the residue gate is run; the proposal skill dispositions colon hits on run-ins; physics and figure-review share one build recipe; - the two design notes point at DISPOSITIONS instead of claiming status; - evaluation parts quote the v0.36.3 register sizes and floor multiples, the UID spread, four retrains, the closed estimator-noise entry, the v0.36.2 collocation bank and the v0.32.0 structure counts as such, and the figure-pinning test without a stale count; - the eval_findings rows name the axes the tool scores; - README.zh-CN reproduces demo 2 with the same flags as README.md, gains the worked-example section, and its roadmap is the English one. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- README.md | 6 +-- README.zh-CN.md | 40 +++++++++---------- docs/architecture/DEAI_SUBSYSTEM.md | 15 ++++--- .../architecture/evaluation/document-scale.md | 4 +- .../evaluation/lexical-structure-uid.md | 2 +- .../evaluation/narrative-salience-register.md | 17 ++++---- .../projection-and-operating-point.md | 9 +++-- .../evaluation/vocabulary-and-residue.md | 6 ++- .../design-notes/DEAI_ARCHITECTURE_ROADMAP.md | 7 ++-- docs/design-notes/DEAI_FRONTIER.md | 12 +++--- skills/brainstorm/SKILL.md | 2 +- skills/calibrate/SKILL.md | 4 +- skills/condense/SKILL.md | 6 ++- skills/de-ai/SKILL.md | 4 +- skills/figure-review/SKILL.md | 3 ++ skills/paper-review/SKILL.md | 3 +- skills/paper/SKILL.md | 6 +-- skills/physics/SKILL.md | 2 +- skills/proposal-polish/SKILL.md | 5 ++- tools/README.md | 16 ++++---- 20 files changed, 95 insertions(+), 74 deletions(-) diff --git a/README.md b/README.md index 8840df7..c09a225 100644 --- a/README.md +++ b/README.md @@ -570,7 +570,7 @@ gates, `build` corpus and profile construction, `eval` evidence. Detail: [tools/ | `tools/deai_provenance.py` | L4 | Editing-provenance ledger over the author's **own** draft history; labels each span AI-untouched → author-original by token edit ratio. Not a detector; `unmeasured` without an AI-draft ancestor. | | `tools/deai_personal.py` | L4 | Personal dispersion baseline against the author's own prior papers — a confound-free same-author reference. `unmeasured` below three papers. | | `tools/eval_docscale.py` | eval | Reproduces the §9 document-scale table — human false-flag rate and per-tier tail power — by scoring the corpus and every `docval` tier through the same operating point findings use. | -| `tools/eval_findings.py` | eval | Scores register, salience, cohesion, hedging and collocation against **provenance** labels instead of hand labels: their firing rate on held-out refereed ApJ/ApJL/A&A papers, on the in-sample papers, and on the `docval` machine tiers, plus a paired test that isolates calibration leakage from publication era. | +| `tools/eval_findings.py` | eval | Scores register (thresholded and zero-hit), salience and collocation against **provenance** labels instead of hand labels: their firing rate on held-out refereed ApJ/ApJL/A&A papers, on the in-sample papers, and on the `docval` machine tiers, plus a paired test that isolates calibration leakage from publication era. | | `tools/label_findings.py` | eval | Samples findings from all five finding-emitting axes (register, salience, cohesion, hedging, collocation) into a human-labelling sheet, re-serves a blind subset for intra-rater agreement, and reports per-axis precision plus **pooled** recall, stratified by named `--population NAME=DIR` sets. Reports `unmeasured` for any stratum under 20 labels. | | `tools/build_profile.py` | build | Builds the basic field profile: extraction, optional legacy classifier, exemplar-cache warm-up. | | `tools/cli_common.py` | build | Shared command-line preamble and field resolution (one profile auto-resolves; several run without one and say so), used by 30 of the 39 Python tools. Holds no policy: no default beyond the two roots, reads no profile, emits no findings. | @@ -631,7 +631,7 @@ For scale, the reference profile behind every measured number here carries: | `structure_baseline.json` | method 9,478 · results 3,978 · data 3,894 · intro 3,812 · discussion 3,635 · conclusion 2,611 · abstract 433 | | `salience_baseline.json` | abstract 13,971 · method 6,967 · results 3,240 · intro 3,239 · data 3,020 · discussion 2,954 · conclusion 1,999 | | `docstructure_baseline.json` | 507 complete documents · conformal α 0.05 · length strata [46, 75] | -| `anchoring_baseline.json` | 517 documents · all six section classes above the 30-document minimum | +| `anchoring_baseline.json` | 517 documents · all six section classes above the 30-document retention minimum and the 119-document floor of the six-class Bonferroni share | | `voice_model.joblib` | 44,636 records · 14 features · **no operating point**, `degraded` | Every bucket clears the 30-passage floor — untrue before 2026-08-25, when @@ -727,7 +727,7 @@ Empty. `label_findings.py` ships; running it is the author's business, not the r this project measures against is provenance — `eval_findings.py`, on labels that already exist. **Closed by refutation, not by shipping.** The length-aware manifold, a larger conformal calibration -set, long-form tail power, three L3 retrains, the leakage contrast, the register operating point, and +set, long-form tail power, four L3 retrains, the leakage contrast, the register operating point, and now citation placement were built or tested and refuted. The last was the strongest model-free discriminator in the record (AUC 0.866) and did not survive its own pre-registered test: a second bank from a different model scores **0.053** on the same statistic with no citation instruction and **0.734** diff --git a/README.zh-CN.md b/README.zh-CN.md index 6e1cba6..bc67a0f 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -248,7 +248,7 @@ strong advisory **131 → 131 → 126**。词表处理让 strong advisory 变化 ```console $ python tools/ai_ism_lint.py before.tex --field wgl \ - --structure --distribution --register --salience --document-structure --no-discourse + --structure --distribution --register --salience --document-structure --no-discourse --no-collocation --no-residue findings: blockers=0 L0=8 advisories=10 (strong=1) axis L0.lexical: measured axis L0.register: measured axis L2.salience_hierarchy: measured @@ -353,6 +353,20 @@ advisory 必须有明确 disposition，而普通 advisory 与不可用的轴保 --- +### 6. 一个可以自己跑的例子 + +`examples/` 里有一份合成稿件，以及按 finding 处理后的同一篇 —— 每个数值都保留、每个数值都是编造的，所以没有一处是谁的未发表工作。 + +| | 之前 | 之后 | +|---|---:|---:| +| L0 目标 | 1 | **0** | +| `discourse-cohesion` | 3 | **1** | +| `salience-recital` | 4 | **3** | + +最后一行随参照而动：2026-08-27 的 profile 把修订稿读作 6，按数值投影校准的 v0.36.3 重建把同一文件读作 3，仍为 strong 的两条从"连续串长"变成了"密度"。张力本身还在：为了衔接两句而把名词带到下一句，会把主语拉进也带数字的句子里，所以在数字密集的段落里 cohesion 与 recital 要的是相反的东西，没有哪种改写能同时满足两者。两条 finding 都成立，采纳哪一条是作者的判断 —— 这正是两者都不是 blocker、也没有分数的原因。演练：[`examples/README.md`](examples/README.md)。 + +--- + ## Benchmark 面板 两类数字，不能互换：**判别力与标定**，读自评估记录；**延迟**，为这份 README @@ -699,27 +713,13 @@ Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权 ### 路线图 空了。`label_findings.py` 已交付 —— 跑不跑标注是作者自己的事，不是仓库的事。本项目 -自己的参照基准是 provenance：`eval_findings.py` 拿**已经存在的标注**（203 篇未见过的 -已发表审稿论文等）给各轴打分。「这条建议对不对」需要人，上面的限制表里已写明。 - -**citation placement 不是「待定」，是被证伪了。** v0.32.0 留下的条件是「换一套独立生成的 -机器库来验」，这次补上了：同一个模型（Codex `gpt-5.6-terra`）、同样 20 个题目，提示词只差 -一行。不提引用时同一个统计量得 **0.053** —— 不是「没有区分度」，而是几乎同等强度地指向 -**反方向**；提了就得 **0.734**。引用密度在这一行提示词之间摆动 12.5 倍（每千词 1.00 ↔ 12.55）， -而人类中位数是 6.20，两个机器极端把人类分布夹在中间。信号是提示词，不是作者身份 -（[§20](docs/architecture/evaluation/discourse-and-citation.md)）。 +自己的参照基准是 provenance：`eval_findings.py` 拿**已经存在的标注**给各轴打分。 **以证伪收口，不是以发布收口。** 长度感知流形、扩大 conformal 标定集、长文尾部功效、 -三次 L3 重训、留出对样本内的泄漏估计、register 操作点，以及这次的 citation placement —— -全部做出来、测过、然后被否决，每一条在上表里都有自己的一行。同一批工作里，cohesion 与 -hedging 两条轴（roadmap rank 6，自 v0.26.1 挂着）**发布**了，其中 hedging 被收窄到只管 -`intro`。`deai_policy.json` 维持撤回；细节见 -[§9.4c](docs/architecture/evaluation/document-scale.md)、 -[§18.4](docs/architecture/evaluation/projection-and-operating-point.md)。 - -**v0.28.0 已收口。** `deai_policy.json` 是**撤回**而非推迟：在 500 篇人类论文上测量， -它要设阈值的那两个统计量都不具判别力 -（[§16](docs/architecture/evaluation/lexical-structure-uid.md)）；补厚薄桶语料也已完成。 +四次 L3 重训、留出对样本内的泄漏估计、register 操作点，以及 citation placement —— +全部做出来、测过、然后被否决。最后一项曾是记录里最强的 model-free 判别量（AUC 0.866）， +却没通过自己预先登记的检验：换一个模型生成的第二个库，不提引用时同一统计量得 **0.053**， +提了就得 **0.734**，所以信号是提示词（[§20](docs/architecture/evaluation/discourse-and-citation.md)）。 **领域特定指引：** 弱引力透镜的科学锚点在适用处标 `[WGL]`，共享的写作与审查政策与领域无关。 ## 致谢与许可 diff --git a/docs/architecture/DEAI_SUBSYSTEM.md b/docs/architecture/DEAI_SUBSYSTEM.md index 0774798..47f04f6 100644 --- a/docs/architecture/DEAI_SUBSYSTEM.md +++ b/docs/architecture/DEAI_SUBSYSTEM.md @@ -135,7 +135,8 @@ than machine drafts (EVALUATION §23.1) — and ships as advice. [`../tools/deai_metrics.py`](../../tools/deai_metrics.py) measures section-aware sentence-length variation and paragraph-opening connective density. A compatibility -heuristic may produce degraded evidence. A strong advisory would require an +heuristic may produce degraded evidence, and a reference with no classified +section bucket leaves the axis degraded. A strong advisory would require an applicable calibrated policy; none is written, because the operating point was refuted by measurement (EVALUATION §16), so the axis stays `degraded`. @@ -153,7 +154,8 @@ structural patterns that keyword replacement cannot repair: - announced enumeration and ordinal runs; - repeated modal frames; - parallel or anaphoric runs; -- setup, list, and wrap-up symmetry; +- setup, list, and wrap-up symmetry (the wrap-up is a sentence opening `These N + ` and closing with a summing verb); - balanced closers; - repeated paragraph templates. @@ -278,9 +280,12 @@ length stratum (`fit_dispersion_manifold`, `manifold_distance`), role-coupled di Document calibration records one observation per verified complete paper, bootstrap uncertainty, leave-one-document-out human flag behavior, and empirical percentiles. -If the corpus does not contain enough complete and measurable papers, the axis is -`unmeasured`; for `wgl` it is `measured` on a complete-paper baseline (EVALUATION §9). The -implementation must not synthesize a document baseline from paragraph exemplars. +The sweep drops the preamble and every `skip` unit (acknowledgements, appendices, +references) on both sides, and a document-shape metric is strong when it exceeds the +quoted quantile threshold. If the corpus does not contain enough complete and +measurable papers, the axis is `unmeasured`; the `wgl` figures in EVALUATION §9 +predate the rebuild these changes require. The implementation must not synthesize a +document baseline from paragraph exemplars. ### L3: learned field similarity diff --git a/docs/architecture/evaluation/document-scale.md b/docs/architecture/evaluation/document-scale.md index e1b5357..ed1d489 100644 --- a/docs/architecture/evaluation/document-scale.md +++ b/docs/architecture/evaluation/document-scale.md @@ -512,8 +512,8 @@ What remains is the honest form of the item: an explicit estimator-noise model, the sampling variance of a dispersion statistic at n paragraphs to the manifold covariance, so short documents are compared against a correspondingly wider human band. That is a modelling change with its own falsification burden and is **not** shipped in -v0.28.0. The roadmap entry stays open, now with its mechanism measured rather than -hypothesized. +v0.28.0. The roadmap entry was closed in §9.4c: the estimator-noise manifold was +built and refuted. ### 9.4c The estimator-noise manifold is implemented and refuted; the tail-power figures are seed draws diff --git a/docs/architecture/evaluation/lexical-structure-uid.md b/docs/architecture/evaluation/lexical-structure-uid.md index cfb035f..f272f5b 100644 --- a/docs/architecture/evaluation/lexical-structure-uid.md +++ b/docs/architecture/evaluation/lexical-structure-uid.md @@ -120,7 +120,7 @@ default and split them from PDF line fragments, so both its size and its section described something other than what they claimed (§14.2, EVALUATION §2). The section means are strikingly tight — 3.23 to 3.36 across seven buckets, a spread of -0.13 against within-bucket standard deviations of 0.26–0.58. It was 0.15 before the +0.13 against within-bucket standard deviations of 0.26–0.51. It was 0.15 before the fourth rebuild and 0.13 after, so removing the leaked keys did not disturb the null. **Section identity barely moves paragraph-level UID in this corpus.** That is a null worth recording rather than a defect: it means a per-section UID operating point would be calibrating on a diff --git a/docs/architecture/evaluation/narrative-salience-register.md b/docs/architecture/evaluation/narrative-salience-register.md index 766e66e..5b91962 100644 --- a/docs/architecture/evaluation/narrative-salience-register.md +++ b/docs/architecture/evaluation/narrative-salience-register.md @@ -207,7 +207,7 @@ of each passage since v0.36.3: | data | **3,020** | 3,016 | 102 | — | `exemplar_paragraphs.jsonl` — bucket added 2026-08-25 | | intro | **3,239** | 3,264 | 94 | 88 | `exemplar_paragraphs.jsonl` | | discussion | **2,954** | 2,958 | 93 | 78 | `exemplar_paragraphs.jsonl` | -| results | **3,240** | 3,206 | 26 | 10 | `exemplar_paragraphs.jsonl` — clears the floor by 85× | +| results | **3,240** | 3,206 | 26 | 10 | `exemplar_paragraphs.jsonl` — clears the floor by 108× | | conclusion | **1,999** | 1,994 | 39 | 41 | `exemplar_paragraphs.jsonl` | The 2026-08-25 column is the rebuild that made `method` a reference, and `method` is the headline: **it was never a @@ -309,18 +309,19 @@ as observed evidence, gives 19. ### 14.4 Register reference and precision -Document frequency over **41,710** passages (27,917 exemplar paragraphs + 13,793 -abstracts), **53,414** terms. Firing rule: ≥ 15 manuscript uses **and** corpus df +Document frequency over **41,644** passages (27,851 exemplar paragraphs + 13,793 +abstracts), **53,367** terms, after the v0.36.3 profile rebuild (41,710 and +53,414 before it). Firing rule: ≥ 15 manuscript uses **and** corpus df rate < 1e-4. (The use count was 5 until v0.32.0 raised it to 15, and this paragraph still said 5 until the 2026-08-26 sweep caught it. §17 records the held-out rates at the old setting; §18 records them at the new one.) The composition bias recorded in §14.6 is largely resolved. Reading the 500-paper breadth corpus took the body contribution from 593 to 25,005 passages and it now -stands at 27,917, so abstracts fell from **96% of the reference to 33%** (13,793 -of 41,710). The axis was biased toward flagging body-section vocabulary precisely +stands at 27,851, so abstracts fell from **96% of the reference to 33%** (13,793 +of 41,644). The axis was biased toward flagging body-section vocabulary precisely because body text was almost absent from its reference; that reference is now -two-thirds body prose. The threshold's resolution also improves by 2.9×: 41,710 +two-thirds body prose. The threshold's resolution also improves by 2.9×: 41,644 passages can express a df rate of 2.4e-5, comfortably below the 1e-4 firing rule, where 14,235 could express 7.0e-5. @@ -398,12 +399,12 @@ re-proposed as an obvious gap. ### 14.6 Limits -One field, one case document. Every non-abstract bucket rests on 1,994–6,959 +One field, one case document. Every non-abstract bucket rests on 1,999–6,967 reference passages, so **all seven clear the 30-passage floor** and none is rank-only. `abstract` (13,971) is still the largest by a factor of 2.0, but it no longer dwarfs the body buckets by two orders of magnitude, and `method` is second-largest on method prose rather than on residue. The counts are the -2026-08-26 author-query sweep's; §14.2's table carries the earlier rebuilds. +v0.36.3 rebuild's, the column §14.2's table leads with. The reference-size limitation recorded here from v0.26 through v0.27.1 is therefore closed. What remains is narrower and should not be read as the same caveat: register diff --git a/docs/architecture/evaluation/projection-and-operating-point.md b/docs/architecture/evaluation/projection-and-operating-point.md index 7ac3166..80d1093 100644 --- a/docs/architecture/evaluation/projection-and-operating-point.md +++ b/docs/architecture/evaluation/projection-and-operating-point.md @@ -125,7 +125,7 @@ papers and 173 machine documents: | 30 | 0.0322 | 26.6% | 0.0004 | 0.6% | 0.369 | | 50 | 0.0117 | 12.3% | 0.0000 | 0.0% | 0.438 | -(Rates here use `body_only` word counts, so they sit above the headline figure +(Rates here use `body_only` word counts, so they differ from the headline figure below, which uses the evaluator's own denominator. Columns are internally comparable; the level is not comparable across the two.) @@ -234,7 +234,7 @@ stays unbuilt until a second, independently produced AI bank answers it. - **`corpus_cos` ablation.** Not run, and now withdrawn as an item rather than deferred: `confound_audit` bins on record metadata the feature cache does not carry, so an ablation runnable from the cache would compute a different - statistic from the three recorded retrains (§7.0a) and could not be compared + statistic from the four recorded retrains (§7.0a) and could not be compared with them. Its only consumer is a `degraded`, audit-only classifier with no shipped operating point, whose status rests on independently refuted grounds, so no ablation result could change shipped behaviour. @@ -270,8 +270,9 @@ check: instead of reading a number out of a document and asking whether it looks right, it renders the expected substring **from the artifact** and asks whether the document still contains it. A document that agrees with a stale artifact and a document nobody updated then fail identically, which is the property the -working rule never had. It pins 39 figures across the two READMEs, this record, -and the hub's axis and bucket tables, and it was verified by putting each +working rule never had. It pins the figures its own table lists — the two +READMEs, `examples/README.md`, this record and the hub's axis and bucket tables; +39 when it was written, more since — and it was verified by putting each corrected figure back to its stale value — 11 mutations, 11 caught. The field is read from this record's own path literal rather than hard-coded, so diff --git a/docs/architecture/evaluation/vocabulary-and-residue.md b/docs/architecture/evaluation/vocabulary-and-residue.md index 0963510..77647d9 100644 --- a/docs/architecture/evaluation/vocabulary-and-residue.md +++ b/docs/architecture/evaluation/vocabulary-and-residue.md @@ -94,7 +94,9 @@ fraction as evidence only. **Design.** `deai_collocation.py --calibrate` builds a bank from the two human passage banks: unigram document frequency for the 11,286 words at or above `COMMON_RATE` (2 × 10⁻⁴), and passage frequency for 530,677 adjacent pairs of -them. A pair is two words the writer put side by side, so it breaks at +them (the v0.36.2 bank this section was measured on; the v0.36.3 rebuild holds +11,282 words and 530,504 pairs, and the per-bucket sizes below are the v0.36.2 +ones). A pair is two words the writer put side by side, so it breaks at punctuation, at a `[math]`/`[CITE]` placeholder, at a dash, at a slash and at a digit (`yields, separate` is two clauses; `50/50` and `2 sigma` are not word pairs), and a sentence's pairs are judged distinct. The unit is the sentence; @@ -174,6 +176,8 @@ more tokens before the head noun with at least two hyphenated compounds). | discussion | 3,647 | 0.41% | 0.16% | 3.32% | | conclusion | 2,609 | 1.00% | 0.23% | 5.83% | +The `n` column is the v0.32.0 bank's; the v0.36.3 `structure_baseline.json` holds intro 3,812, method 9,478, data 3,894, results 3,978, discussion 3,635, conclusion 2,611, and the fractions were not re-taken on it. + A first cut of the stack rule counted any run of non-function tokens and put 15% of human method paragraphs in it; cutting the run at the head noun and requiring two hyphenated compounds brought the fraction to what the table diff --git a/docs/design-notes/DEAI_ARCHITECTURE_ROADMAP.md b/docs/design-notes/DEAI_ARCHITECTURE_ROADMAP.md index 3aad7f8..facaa43 100644 --- a/docs/design-notes/DEAI_ARCHITECTURE_ROADMAP.md +++ b/docs/design-notes/DEAI_ARCHITECTURE_ROADMAP.md @@ -3,7 +3,7 @@ Status: design note written 2026-07-12, retained as the reasoning record for why the document scale is the keystone. It is **not** a status document: the keystone shipped on 2026-07-13, and the current disposition of every ranked item lives in -[`SCIPAPER_STANDARD.md`](../SCIPAPER_STANDARD.md) §11. Complements +[`DISPOSITIONS.md`](../architecture/DISPOSITIONS.md). Complements [`DEAI_SUBSYSTEM.md`](../architecture/DEAI_SUBSYSTEM.md) (current architecture) and [`SCIPAPER_STANDARD.md`](../SCIPAPER_STANDARD.md) (normative policy). @@ -207,5 +207,6 @@ document-level surprisal is itself weaker than the model-free manifold (0.757 vs 0.881) and adds nothing to it, so the enriched features are not shipped into the model-free detector (recorded, not added). This resolves the last L3 document-level debt: the detector stays model-free by measurement at document scale. Ranks 2, 3, 4 -and 6 were never started as scoped; their decided dispositions live in SCIPAPER_STANDARD §11, -which is the single status home. +and 6 had not been started as scoped when this section was last written; their +dispositions since live in [`DISPOSITIONS.md`](../architecture/DISPOSITIONS.md), the +single status home. diff --git a/docs/design-notes/DEAI_FRONTIER.md b/docs/design-notes/DEAI_FRONTIER.md index 81162a4..243dbbf 100644 --- a/docs/design-notes/DEAI_FRONTIER.md +++ b/docs/design-notes/DEAI_FRONTIER.md @@ -4,9 +4,9 @@ Status: design note, 2026-07-13. Produced by a six-lens divergent ideation (scale-unlocks, arms-race, writing-time product, cross-field transfer, evaluation science, contrarian) after the keystone validation and its measured adversarial limit. Complements [`DEAI_ARCHITECTURE_ROADMAP.md`](DEAI_ARCHITECTURE_ROADMAP.md) -(ranks 1, 5, 7 and 8 there have since shipped or been measured; ranks 2, 3, 4 and 6 remain -open, with their dispositions in [`SCIPAPER_STANDARD.md`](../SCIPAPER_STANDARD.md) §11). **The ranked frontier is now -complete (2026-07-13).** Detection core: idea 5 (shipped, +(the disposition of every rank there lives in +[`DISPOSITIONS.md`](../architecture/DISPOSITIONS.md); nothing in this note is status). **The ranked frontier is +complete as of this note's date (2026-07-13).** Detection core: idea 5 (shipped, `deai_docstructure.fit_dispersion_manifold`), idea 7 (executed, signal survives — §9.3), idea 1 (shipped, `document-role-decoupling` — §9.4), idea 8 (shipped, split-conformal + length-Mondrian — §9.5). Cooperative layer: idea 2 (shipped, @@ -139,9 +139,9 @@ ledger (4) are the cooperative-tool layer. Progress: **all eight ideas shipped o executed** (7 pass, 5/1/8 detection core, 2/3/4/6 cooperative layer). The recorded length-aware-manifold refinement was measured to be a length confound and not adopted; the per-stratum manifold plus length-Mondrian conformal is the -confound-safe length handling (§9.8). The ranked frontier queue is empty; further -detector work continues on the roadmap track (ranks 2, 3, 4 and 6) and the ai_long -standing target. +confound-safe length handling (§9.8). The ranked frontier queue is empty as of this +note's date; what became of each roadmap rank and of the ai_long standing target +is recorded in [`DISPOSITIONS.md`](../architecture/DISPOSITIONS.md), not here. ## Notable discards (with reasons) diff --git a/skills/brainstorm/SKILL.md b/skills/brainstorm/SKILL.md index b5e4913..559ec93 100644 --- a/skills/brainstorm/SKILL.md +++ b/skills/brainstorm/SKILL.md @@ -53,7 +53,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max - `--max-branches N` 默认 ∞ — 每节点单轮新增分支上限（注意：§3 强制 12 条 framing pass 各产 ≥1 分支，所以下限实际是 12；该 flag 仅可放大） - **caps 触顶的语义**：当用户**显式**设置了 `--width N` / `--depth N` / `--rounds N` 并触顶时，**已展开的节点必须先全部完整推进到§4 12 字段填满、§5 verdict 定论之后**才允许停止；不允许"刚到上限立即停留下半成品"。报告以 `WIDTH_CAP_REACHED` / `DEPTH_CAP_REACHED` / `ROUNDS_EXHAUSTED` 标记，**但所有可见叶节点必须完整**。 - **不允许 skill 内部自行扩大 cap**；也不允许内部自行缩小默认 ∞ 为某个有限值。 - - 旧版的 `--max-nodes 200` / `--max-time-min 60` 已**移除**。"探索成本太大"不构成停止理由；这是本 skill 与普通 brainstorm 工具的关键区别。 + - "探索成本太大"不构成停止理由；这是本 skill 与普通 brainstorm 工具的关键区别。 8. **完整推进禁令（hard ban on deferred / incomplete leaves）** —— 见「数据模型」最后一条： - 任何叶节点的任何字段中含以下字样均视为**违规半成品**，节点状态强制改为 `INCOMPLETE_FORBIDDEN`，必须继续推进至完整： diff --git a/skills/calibrate/SKILL.md b/skills/calibrate/SKILL.md index dfdfa92..6bc8d68 100644 --- a/skills/calibrate/SKILL.md +++ b/skills/calibrate/SKILL.md @@ -2,10 +2,10 @@ name: calibrate description: Make the plugin yours. Walks the whole calibration chain for one field — corpus intake, profile extraction, per-axis calibration, optional model training, a held-out provenance measurement, and finding-level labelling against your own standard, style and target journal — then verifies the result. Every axis reports `measured`, `degraded` or `unmeasured` from its own floor; a stratum that cannot support a rate stays `unmeasured` rather than being given one. Produces evidence only: no threshold here creates a verdict, an authorship claim, or a paper PASS/FAIL. Use when: "calibrate" / "set up my corpus" / "train on my papers" / "adapt to my journal" / "why is this axis degraded" / 校准 / 建语料库 / 用我自己的论文训练 / 适配目标期刊 / 标注 / 这个轴为什么是 degraded. disable-model-invocation: false -argument-hint: " [--variant ] [--from ] [--labels-only]" +argument-hint: "" --- -> **Not a normative skill.** `docs/SCIPAPER_STANDARD.md` decides what good +> **Not a policy source.** `docs/SCIPAPER_STANDARD.md` decides what good > scientific prose is; nothing produced here can change that. Calibration > supplies the *reference distributions* the advisory axes compare against, and > `docs/architecture/EVALUATION.md` records what each one is worth. A corpus diff --git a/skills/condense/SKILL.md b/skills/condense/SKILL.md index 57c1343..2f1b627 100644 --- a/skills/condense/SKILL.md +++ b/skills/condense/SKILL.md @@ -2,7 +2,7 @@ name: condense description: Whole-document condensation. Removes every passage that is unnecessary or already stated elsewhere: claims and numbers repeated across sections, zero-information paragraphs, dead definitions and uncited figures, verbose constructions. Executes SCIPAPER_STANDARD §5.3 (condense, do not accumulate): the default direction of every edit is shorter — delete > compress in place > same-length rewrite, with growth last and only under a recorded justification. Each fact keeps exactly one canonical home; the copies are deleted or replaced by a cross-reference. The removal map is machine-built (condense_map.py), every entry is dispositioned, and the length gate proves the shrink against the map's target. Fidelity invariants (§6) are protected throughout. Use when: "condense" / "trim" / "too long" / "repetitive" / "cut the padding" / 精简 / 太长了 / 重复 / 废话太多 / 去冗余, or when a paper-review dimension-I redundancy finding needs to be executed. disable-model-invocation: false -argument-hint: " [--section ] [--max-iter N] [--report-only] [--field ] [--target ]" +argument-hint: " [--report-only] [--field ] [--target ]" --- > **Normative authority:** `docs/SCIPAPER_STANDARD.md`. §5.3 (condense, do not @@ -140,7 +140,9 @@ every cross-reference into it. it — either remove more, or record which kept entries account for the gap. 2. **Residue:** `python tools/deai_residue.py --before /baseline/.tex`. A condensation must not leave a heading or - caption promising what the body no longer says; exit 1 blocks closing. + caption promising what the body no longer says; exit 1 blocks closing. The + static negative-label rule abstains below 400 prose words (the axis says + so); the diff rule runs regardless. 3. **No orphans:** rebuild/compile the document if it is LaTeX (0 errors, 0 undefined references, no newly-missing labels); re-run the map and grep every deleted label and symbol to confirm zero remaining consumers. diff --git a/skills/de-ai/SKILL.md b/skills/de-ai/SKILL.md index ceb3563..91e3fbb 100644 --- a/skills/de-ai/SKILL.md +++ b/skills/de-ai/SKILL.md @@ -102,7 +102,9 @@ condense's ranked sweep. reporting `measured`). A stale profile is `unmeasured`, never conformity. 4. Retrieve section-typed positive anchors only when the bank exists: `python tools/retrieve_exemplars.py --field --section - --topic "" --k 5`. Read every returned + --topic "" --k 5` (add `--allow-fallback` + when sentence-transformers is absent: keyword overlap, strictly worse; + without either, record the anchors as `unmeasured`). Read every returned exemplar; record provenance; use for rhythm and register only. ## 2. Pass 1 — measure (de-AI subsystem) diff --git a/skills/figure-review/SKILL.md b/skills/figure-review/SKILL.md index 901720d..e920a5b 100644 --- a/skills/figure-review/SKILL.md +++ b/skills/figure-review/SKILL.md @@ -37,6 +37,9 @@ argument-hint: "" Use the paper's authoritative build command. A simple LaTeX fallback is: ```bash +# the project's own build command wins; this is the default recipe (as in physics) +pdflatex -interaction=nonstopmode main.tex +bibtex main pdflatex -interaction=nonstopmode main.tex pdflatex -interaction=nonstopmode main.tex ``` diff --git a/skills/paper-review/SKILL.md b/skills/paper-review/SKILL.md index 6e0fea9..eb636c7 100644 --- a/skills/paper-review/SKILL.md +++ b/skills/paper-review/SKILL.md @@ -291,7 +291,8 @@ stale scientific content、冲突副本或 required artifact drift 是 `integrit `residue-negative-label`（标题或 caption 承诺正文没有的东西）、 `residue-absence`（正文句子用"它从不做什么"定义论文自己的对象：`never` 为 strong，`carries no` 等为 ordinary，句中带 citation 的豁免）与 diff 规则 - `residue-negative-label-added`。strong finding 使工具 exit 1，本轮不得收尾。 + `residue-negative-label-added`。strong finding 使工具 exit 1，本轮不得收尾； + 静态负标签规则在正文不足 400 词时弃权（轴状态会说明），diff 规则照跑。 - 只保留当前科学状态。绝对句先删后改：上下文已承载肯定陈述时整句/整从句删除， 只有页面尚未说出的内容才改写成肯定形式；物理事实、范围限制与承认的局限保留 原样（见 `/sci-paper:paper` 自检第 5 项与工具的 action 文本）。 diff --git a/skills/paper/SKILL.md b/skills/paper/SKILL.md index 32bfd6a..eb23be3 100644 --- a/skills/paper/SKILL.md +++ b/skills/paper/SKILL.md @@ -12,7 +12,7 @@ disable-model-invocation: false ## Paper Writing Standards / 论文写作标准 -Target: **Top-tier astrophysics journals** (ApJ, MNRAS, A&A level). +Target: **the field's top-tier journals** (`[WGL]`: ApJ, MNRAS, A&A level). ### General Principles @@ -193,7 +193,7 @@ deleting its hedges has been damaged. - When describing the detection pipeline, maintain the distinction between signal (E-mode) and noise (B-mode). - All filter functions must be written with explicit mathematical definitions, not just names. -### Method Descriptions +### Method Descriptions `[WGL]` - For each ML model, specify: architecture, input representation, loss function, training procedure, and evaluation metric. - For ensembles, explain the aggregation strategy and why it is appropriate given any class imbalance. @@ -207,7 +207,7 @@ deleting its hedges has been damaged. - Figures should have: descriptive captions, labeled axes with units, legends, and consistent color schemes. - When comparing methods, use the same evaluation protocol for all. Never compare training metrics of one model to validation metrics of another. -### Discussion and Limitations +### Discussion and Limitations `[WGL]` - **Honestly discuss limitations.** Acknowledge sample-size and selection-effect limits. - Distinguish between: limitations of the method vs. limitations of the data. diff --git a/skills/physics/SKILL.md b/skills/physics/SKILL.md index 01392af..9958d77 100644 --- a/skills/physics/SKILL.md +++ b/skills/physics/SKILL.md @@ -34,7 +34,7 @@ argument-hint: " [--field ]" 2. 建**公式清单**：每个 displayed equation 的左右端量纲 + 涉及物理量的定义。 3. 建**数字溯源清单**：abstract / table / figure caption / 正文的所有数值， 每条记值、位置、声称来源、核验状态。 -4. 执行论文的权威 build（LaTeX 项目为 `pdflatex × 2 + bibtex + pdflatex × 2`）； +4. 执行论文的权威 build（项目自己的 build 命令优先；LaTeX 默认 `pdflatex, bibtex, pdflatex × 2`，与 figure-review 一致）； 记录 errors、undefined refs、multiply-defined labels、overfull、missing-number。 ## 2. 检查项 P1–P8 diff --git a/skills/proposal-polish/SKILL.md b/skills/proposal-polish/SKILL.md index 3fb13dd..d4efb5f 100644 --- a/skills/proposal-polish/SKILL.md +++ b/skills/proposal-polish/SKILL.md @@ -107,8 +107,9 @@ These read as strength; keep or add them rather than editing them out. principled step toward it. - **Run-in lead-ins for scannability**: bold or italic **Goal:**, **Motivation:**, **Innovation:**, *Aim N (one-line mission)*. Reviewers skim; - visible structure earns reading time. (The manuscript colon-elaboration rule - does not apply to these labeled run-ins.) + visible structure earns reading time. (The linter's `colon-elaboration` + advisory still fires on a labeled run-in; disposition those hits as + `rejected_as_false_positive` instead of rewriting the run-ins.) - **A concrete running example** that stays consistent across aims and makes an abstract method vivid. - **Sharp aim statements posed as questions**: a crisp open question reads as a diff --git a/tools/README.md b/tools/README.md index ae43665..5438928 100644 --- a/tools/README.md +++ b/tools/README.md @@ -23,27 +23,27 @@ serialize evidence but do not define an independent paper verdict. | `verify_references.py` | Bibliography verification against CrossRef (DataCite fallback for the DOI prefixes CrossRef does not register) and the arXiv API: DOI resolvability (from the `doi` field or a doi.org `url`) and first-author / year / title / journal / volume / first-page agreement per entry, `@string` names expanded, plus `--tex` cross-checking of the assembled document's citation keys (LaTeX, natbib and biblatex commands including `\cites{a}{b}`; `\nocite{*}` cites every entry; a citation inside a `%` comment is text) against the entries. `--cache` keeps fetched records between rounds; a registry miss is never cached, and a finding built from a cached record says so (`observed.cached`). | Exit 0 = no blocker; 1 = unresolvable identifier or cited key without an entry; 2 = invalid input or execution failure (malformed BibTeX names the entry and line). Title match tolerates the CrossRef title/subtitle split; a one-year print/online seam is ordinary, a larger gap strong; a first author agrees when the family names are equal or share a part of four or more letters. A registry outage or an answer of unexpected shape marks the entry unmeasured and the axis `degraded`, never clean; an entry with no identifier is unmeasured. Relevance (whether a citation supports its sentence) is not measured here; that half of dimension F stays with the reviewer. | | `condense_map.py` | The removal map behind `/sci-paper:condense`: six scans — restatement (with the canonical home), zero-gain sentences, dead figure/table/label/macro/acronym, verbose constructions, repeated symbol glosses, duplicated paragraphs — each entry with `removable_words`, totalled into `condense_budget` with a default target that excludes the abstract/conclusion carve-out. | Deletes nothing and emits no blocker; exit 0 on a measured map, 2 on invalid input. Every word count is the gate's rendered-prose count (placeholders and heading words excluded), and the abstract/conclusion carve-out applies to either side of a restatement pair. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%, EVALUATION §23.5), so a pass that cuts less than that has not found what the map lists. | | `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan shipped documentation for edit-meta literals. The literal rule reads the visible lines (headings and captions included; a `\newcommand{\TODO}` in a preamble is not prose) and the label rule the body projection; strengths were set on held-out refereed papers: any strong finding in 13.3% of them, the static negative-label rule ordinary, the strong absence forms at 0.008 per 1,000 words (EVALUATION §23.4). | -| `deai_metrics.py` | L1 information-distribution analysis: sentence-length variation and connective-openers. | Strong status requires applicable policy calibration; compatibility heuristics are degraded. The reference rate is computed over `CONNECTIVE_OPENERS` from the profile's `paragraph_initial_counts`, i.e. the same set the draft is measured against; a profile predating that key omits the reference clause rather than quoting an incomparable one. | -| `deai_structure.py` | L2 sentence-template analysis: enumeration, ordinal/modal/parallel runs, setup-list-wrap-up patterns, and balanced closers; plus the auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) reported under `structure-auxiliary`. | Strong status requires calibrated policy and sufficient reference sample. Auxiliary families never enter `template_score`; their per-bucket human fractions are written by `--calibrate` (`paper_agent_frac`, `wh_cleft_frac`, `modifier_stack_frac`). | +| `deai_metrics.py` | L1 information-distribution analysis: sentence-length variation and connective-openers. | Strong status requires applicable policy calibration; compatibility heuristics are degraded. The reference rate is computed over `CONNECTIVE_OPENERS` from the profile's `paragraph_initial_counts`, i.e. the same set the draft is measured against; a profile predating that key omits the reference clause rather than quoting an incomparable one. A `sentence_stats.json` with no classified bucket (only `unknown`) leaves the axis `degraded`: its pooled CV is 0 and the burstiness rule cannot fire. | +| `deai_structure.py` | L2 sentence-template analysis: enumeration, ordinal/modal/parallel runs, setup-list-wrap-up patterns, and balanced closers; plus the auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) reported under `structure-auxiliary`. | Strong status requires calibrated policy and sufficient reference sample. Auxiliary families never enter `template_score`; their per-bucket human fractions are written by `--calibrate` (`paper_agent_frac`, `wh_cleft_frac`, `modifier_stack_frac`). A template finding is `measured` only against a reference for its own bucket plus a policy, `degraded` when the baseline lacks the bucket, `unmeasured` without one; the auxiliary message quotes the reference fraction and calls a figure rare only at or below the policy gate. The wrap-up beat requires a sentence opening `These N ` closed by a summing verb (define/form/constitute/together/…). | | `deai_salience.py` | L2 salience hierarchy: how far a passage's numerals run without an interpreting sentence between them, plus density and per-sentence numeral count, against a per-section human passage reference. | Sole consumer of `extract_style.latex_to_numeral_text` (the numeral-preserving projection; `latex_to_plain` zeroes every numeral on `.tex` input), and it calibrates on the bank's `numeral_text` field, the same projection of each paragraph, so the reference holds the numerals the manuscript side counts. Until v0.36.3 it calibrated on `text`, which holds none inside math, and the p90 gate fired at 0.45 per held-out passage against a 0.27 bound; it now fires at 0.20 (EVALUATION §17.5). Percentiles are read as P(X <= x) on a 0.01 grid at the top of a tie plateau. Buckets under 30 reference passages are degraded; a reference with no spread above the gate abstains rather than flagging. As of the v0.36.3 profile rebuild every bucket clears that floor (smallest: `conclusion` 1,999). One finding per passage, led by its most extreme feature. | | `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- `AUC` (df 1) must separate from `epoch` (402) and `accuracy` (774). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the manuscript supplies a reason the tool can read (a formation of an attested stem, a defining sentence or acronym expansion at first use, a proper name capitalised at every use); it is exhaustive advice, not a detector (rank AUC 0.174, EVALUATION §23.1). | | `deai_collocation.py` | L2 collocation: per sentence, the fraction of distinct adjacent common-word pairs that no corpus passage attests, against a leave-one-out per-bucket reference at sentence unit (p90 advisory, p95 strong). Each flagged pair carries λ = df(a)·df(b)/N and e^−λ so the reader can weigh it. `--glossary` reads the same bank document-wide: every unattested pair used at least twice, with its first-use line and whether that sentence defines it, as the list of the manuscript's own terms to define at first use or replace (2026-09-16). | `--calibrate` writes `collocation_bank.json` (unigram and pair document frequency; 11,282 partner words and 530,504 pairs over 41,644 `wgl` passages) and `collocation_baseline.json`. Unmeasured without the bank; degraded where no bucket clears the 30-sentence floor. Pairs break at punctuation, `[math]`/`[CITE]` and dashes; a sentence needs four judged pairs to carry a fraction. The action names the relation a pair compresses and never changes a claim; a defined term keeps its pair. | | `deai_discourse.py` | L2 discourse texture, two axes on the LOW tail (p10 advisory, p05 strong): `cohesion`, the mean fraction of a sentence's content words that already appeared in the sentence before it, and `hedging`, epistemic markers per 1,000 words. | The defect here is absence, not excess, so both flag below the gate. The two run at **different units and each artifact records its own**: cohesion per paragraph, hedging per SECTION, because hedging has no paragraph-scale lower tail at all (p10 is 0.000 in all seven `wgl` buckets). Hedging ships restricted to `intro`, where held-out transfer is 7.89% against 15.48-26.77% elsewhere and every generation regime separates (EVALUATION section 19). Advisories against the field's distribution, never a provenance claim. | | `deai_reference.py` | Library, no CLI. One `(feature, unit)` percentile reference shared by every per-bucket axis: the 0.01 quantile grid, the plateau-top percentile reader, the 30-unit sample floor, the paragraph and section sweeps, the artifact loader, and the calibration loop. | Holds no policy -- gate quantiles, feature names and what a finding means belong to the axis. Its one invariant is that calibration and detection share one unit and one grid; that check is what caught the hedging axis calibrating where no lower tail exists. The artifact loader had five copies across the suite before this module (EVALUATION section 19.1). | -| `deai_docshape.py` | Document-shape measurement and complete-document calibration: the per-paragraph model-free feature vector, cross-paragraph dispersion, the joint Mahalanobis manifold (pooled and per-length-stratum), role coupling, the split-conformal operating points, and the baseline builder. | Split out of `deai_docstructure.py` on 2026-08-25 at 1,092 lines against a 750-line budget; the seam is measurement versus feedback. `_paper_documents` now assembles a bundle through `es.corpus_documents` in `\include` order — sorted-filename concatenation put `Conclusion.tex` before `Introduction.tex` in 12 of the 500 `wgl` bundles, on the one axis that measures section arc. | -| `deai_docstructure.py` | L2 whole-document rhetorical-shape findings and the CLI: turns the measurements above into `sci-paper.feedback.v1` findings. Re-exports every public name from `deai_docshape.py`. | Requires at least three measurable complete documents and sufficient sections/paragraphs; otherwise unmeasured. Legacy baselines without a conformal block fall back to percentile thresholds; factor-drifted role references disable that axis. | -| `deai_partition.py` | Fidelity-free partition suggestions (merge/split of paragraph blocks) that move a document toward the calibrated human dispersion band. Suggest-only; the author applies changes by hand. | Zero-token operations, so rewrite fidelity holds by construction; cohesion is self-normalized to the document's median adjacent-pair overlap; section-command blocks are never candidates; reordering is deliberately not offered. Without a calibrated manifold the tool exits with an explicit message. | -| `deai_anchoring.py` | L2 claim-anchoring band: section-class conditional anchored-sentence rates against the human corpus, low-tail conformal with a Bonferroni share per class. | A writing-quality axis, not an AI-discrimination axis (EVALUATION.md 9.6 records the refuted tell). Classes below the 30-document minimum are honestly omitted; unmeasurable documents degrade to unmeasured. | +| `deai_docshape.py` | Document-shape measurement and complete-document calibration: the per-paragraph model-free feature vector, cross-paragraph dispersion, the joint Mahalanobis manifold (pooled and per-length-stratum), role coupling, the split-conformal operating points, and the baseline builder. | Split out of `deai_docstructure.py` on 2026-08-25 at 1,092 lines against a 750-line budget; the seam is measurement versus feedback. `deai_docstructure._paper_documents` assembles a bundle through `es.corpus_documents` in `\include` order — sorted-filename concatenation put `Conclusion.tex` before `Introduction.tex` in 12 of the 500 `wgl` bundles, on the one axis that measures section arc. `document_shape` sweeps `deai_metrics.section_units` and drops the preamble and every `skip` unit (acknowledgements, appendices, references) on both sides; `shape_units`/`measurable_sections` are the one section/paragraph filter `deai_partition` scores with. | +| `deai_docstructure.py` | L2 whole-document rhetorical-shape findings and the CLI: turns the measurements above into `sci-paper.feedback.v1` findings. Re-exports every public name from `deai_docshape.py`. | Requires at least three measurable complete documents and sufficient sections/paragraphs; otherwise unmeasured. Legacy baselines without a conformal block fall back to percentile thresholds; factor-drifted role references disable that axis. A document-shape metric is strong when it exceeds the interpolated quantile quoted as `strong_threshold` (the rule the leave-one-document-out rate uses). `--strong-percentile` must lie strictly in (0, 1) (exit 2); `--field` auto-resolves on the read path and must name an existing profile for `--calibrate`. | +| `deai_partition.py` | Fidelity-free partition suggestions (merge/split of paragraph blocks) that move a document toward the calibrated human dispersion band. Suggest-only; the author applies changes by hand. | Zero-token operations, so rewrite fidelity holds by construction; cohesion is self-normalized to the document's median adjacent-pair overlap; heading-command blocks (`\section`, `\subsection`, `\chapter`, …) are never candidates; reordering is deliberately not offered. Without a calibrated manifold the tool exits with an explicit message. States are ranked by conformal p (higher = deeper inside the band), distance only as a tie-break within one calibration basis; every plan step records `calibration_basis_before/after`. A split is simulated on the original text at the sentence boundary, line breaks kept; scored blocks pass the same filter as `document_shape`. | +| `deai_anchoring.py` | L2 claim-anchoring band: section-class conditional anchored-sentence rates against the human corpus, low-tail conformal with a Bonferroni share per class. | A writing-quality axis, not an AI-discrimination axis (EVALUATION.md 9.6 records the refuted tell). Classes below the 30-document minimum are honestly omitted; unmeasurable documents degrade to unmeasured. A class whose calibration n cannot reach its Bonferroni share (1/(n+1) > alpha/k) is skipped and the axis reported `degraded`; `--calibrate` records `underpowered_classes` and warns. The scan runs on section units with headings, floats and the bibliography blanked and `skip` units dropped. | | `deai_provenance.py` | Editing-provenance ledger (frontier idea 4): matches each current paragraph to a designated AI-draft ancestor (file or git ref from the author's own history) and labels the span AI-untouched / lightly-edited / rewritten / author-original by a deterministic difflib token edit ratio. | Not an AI detector; reads only the author's own history. `unmeasured` without an ancestor or when the file has no git history for the given ref (never a guess). | | `deai_personal.py` | Personal dispersion baseline (frontier idea 6): places a draft's per-feature within-document dispersion in the distribution of the author's own prior papers, a confound-free same-author reference. | Model-free; reuses `document_shape`. `unmeasured` below three measured prior papers. Flags a draft that varies paragraph shape far less than the author's own papers on at least 40% of features. | | `deai_oracle.py` | Optional token-surprisal and UID analysis. | Requires transformers/model assets for measurement; compatibility `FLAG_Z` is degraded until field calibration exists. Document-level surprisal is measured (EVALUATION.md 9.8) to be weaker than the model-free manifold and adds nothing to it. | -| `deai_features.py` | Reusable model-free, UID, and embedding features for document analysis and learned field-similarity models. | Optional model features degrade when dependencies/assets are unavailable. | +| `deai_features.py` | Reusable model-free, UID, and embedding features for document analysis and learned field-similarity models. | Without the transformers/torch runtime the UID features are `None` with `uid_status: unmeasured` (`features_vector` raises with the reason; the CLI exits 2 with a message); the corpus centroid is `None` without the `.npy` cache and needs no numpy to say so. | | `deai_voice.py` | Optional learned field-similarity triage. | Refuses bundles with drifted feature names/schema and degrades on corrupt bundles. A measured operating point gates threshold findings; an uncalibrated bundle yields rank-based triage of the lowest-scoring paragraphs, never a universal 0.5 cutoff. | | `train_voice_model.py` | Trains the optional learned field-similarity/rewrite-ranking model and writes `voice_model_evaluation.json`. | Cache reuse requires a content/model/centroid fingerprint; featurization checkpoints every 500 rows for preemptible runs. Grouped splits recompute `corpus_cos` from training-only centroids; repeated source-group splits audit source, section, length, mathematical-marker, field-term, matched-stratum, and UID-normalization sensitivity, plus the author-labelled hard-set stratum, without creating an operating point. | | `voice_dataset.py` | Record loading, source-family grouping, the training-only field lexicon, and the fingerprinted feature-matrix cache for the voice model. Split out of `train_voice_model.py` on 2026-08-26 (1,174 lines against a 750-line budget). | Depends on nothing in `voice_audit`; the split is one-way and a test asserts it. The split's hand-written import block left `time` and `CHECKPOINT_EVERY` unbound and the suite stayed green, because no test reached `build_features` — a whole-tools-directory AST check for unbound names now runs in `test_train_voice_model.py`. | | `voice_audit.py` | Held-out metrics, bootstrap AUC intervals, the section-normalized UID alternative, the author hard set, and the repeated grouped confound audits. Owns `HARDSET_AI_CATEGORIES` / `HARDSET_HUMAN_CATEGORIES`. | Fits no model and creates no operating point: it produces the evidence a bundle needs before it can leave `degraded`. `train_voice_model` re-exports every public name, guarded by a contract test. | | `label_findings.py` | Samples findings from all five finding-emitting axes — register, salience, cohesion, hedging, collocation — into a human-labelling sheet (`sample`), re-serves a blind subset with the prior answers stripped (`relabel`), and reports precision, recall and intra-rater agreement (`score`). | The one roadmap item computation cannot close: these axes are corpus-referenced, so nothing here says whether a human agrees a given advisory is right. Built to the scheme chosen 2026-08-26 — finding-level labels, one labeller plus a blind re-label subset, and populations named with `--population NAME=DIR` so each is sampled and reported separately. Recall is **pooled over the axes**: a control row asks whether a passage should have been flagged, and a labeller who says yes has not said by which axis, so a per-axis recall would charge every axis with every other axis's misses. Calibrates nothing; any stratum under 20 labels reports `unmeasured` rather than a rate, and the intra-rater kappa is printed as the ceiling no axis can be held above. | -| `eval_findings.py` | Scores the register, salience, cohesion, hedging and collocation axes against provenance labels instead of hand labels (collocation on its document novel-pair fraction, since its reference is per sentence): firing rate on **held-out** refereed ApJ/ApJL/A&A papers, on the same-genre in-sample papers (leakage control), and on the `docval` machine tiers, plus the machine-over-held-out rank AUC. | Read the two rows differently. Register is an absolute rarity test, so its held-out rate **is** a false-positive rate. Salience is a percentile gate (`ADVISORY_PERCENTILE` 0.90), so a non-zero rate is the design point and the number tests calibration transfer, not defectiveness. Neither is precision for the *advice*: publication does not mean every sentence is beyond improvement, so the register figure is an upper bound. Recall stays `unmeasured` — that is `label_findings.py`. | +| `eval_findings.py` | Scores the register (thresholded and zero-hit), salience and collocation axes against provenance labels instead of hand labels (collocation on its document novel-pair fraction, since its reference is per sentence): firing rate on **held-out** refereed ApJ/ApJL/A&A papers, on the same-genre in-sample papers (leakage control), and on the `docval` machine tiers, plus the machine-over-held-out rank AUC. | Read the two rows differently. Register is an absolute rarity test, so its held-out rate **is** a false-positive rate. Salience is a percentile gate (`ADVISORY_PERCENTILE` 0.90), so a non-zero rate is the design point and the number tests calibration transfer, not defectiveness. Neither is precision for the *advice*: publication does not mean every sentence is beyond improvement, so the register figure is an upper bound. Recall stays `unmeasured` — that is `label_findings.py`. | | `eval_docscale.py` | Reproduces the EVALUATION §9 document-scale table — human false-flag rate, per-tier 5%-tail power, and rank AUC — by scoring the corpus and every `docval` tier through `manifold_operating_point`. | The human row is **in-sample** (it includes the manifold's own train and calibration documents) and is labelled as such; the distribution-free guarantee is the split-conformal alpha, not this rate. Point estimates carry substantial seed variance (§9.4c), so a single run is not a stable figure — compare configurations, not runs. | | `cli_common.py` | Shared command-line preamble and field resolution: UTF-8 stdout, the `--field` / `--profile-root` / `--corpus-root` options, `resolve_field` for a tool that needs a field, `optional_field_dir` for one that can run without (one profile auto-resolves, several print a note and run without one), and `axis_main` for the calibrated axes (`--calibrate` needs an existing field and refuses to write an empty artifact; an axis flag goes through `extra_arguments`). | Holds no policy — decides no default beyond the two roots, reads no profile, emits no findings. **21 of 30 tools were retrofitted on 2026-08-26**; the sweep is recorded with its allowlist and refusals. One divergence is kept deliberately: `ai_ism_lint` returns `None` instead of raising (warning on an explicit field that does not exist), because its L0 pass must still run with no profile present; `extract_style` resolves against the corpus root where `tier-*` is not a field (expressed as `exclude_prefixes`, not a second implementation). | | `rewrite_reward.py` | Evaluates rewrite candidates after hard scientific-fidelity eligibility. | Bidirectional: dropping OR inventing protected numbers, units, citations, math, acronyms, semantic LaTeX macros, comparison direction, negation, or causal direction makes a candidate ineligible (`-inf`). Ranking is led by L0 advisory reduction (signed, fidelity-floored) and semantic fidelity; the learned score is tie-break weight unless its bundle is measured. Its numeral tokenizer has been corrected three times for one root cause — a separator absorbed into the token: the Oxford comma, the spaced unit, and (2026-08-25) a hyphen between numerals, which made `0.5-1.2` yield `-1.2` and hard-rejected every faithful rewrite of a range. Each is pinned by a test. Exit 0 = at least one eligible candidate; 1 = every candidate ineligible (a measured outcome: preserve the original and regenerate tighter); 2 = invalid input or missing required configuration. | From 470290d2642246157e7446bffb6ccba1d235ebfa Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:09:06 +0000 Subject: [PATCH 10/27] release metadata: v0.39.0 with the notes for the parts already integrated Version strings, the two versioned document headers and the CHANGELOG entry for the audit's first tier as landed so far (assembly, projection, gate, map, linter, validator, extractor, bibliography checker, document-level axes). The entry's test count and the remaining parts follow with their commits. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- .claude-plugin/marketplace.json | 4 +- .claude-plugin/plugin.json | 2 +- CHANGELOG.md | 99 +++++++++++++++++++++++++++++ README.md | 4 +- README.zh-CN.md | 4 +- docs/architecture/DEAI_SUBSYSTEM.md | 2 +- docs/architecture/EVALUATION.md | 4 +- 7 files changed, 109 insertions(+), 10 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index b56f431..fcacb64 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -5,14 +5,14 @@ }, "metadata": { "description": "Single-plugin marketplace for sci-paper: scientific paper writing, source-traced review, and de-AI rewriting for top-tier journals.", - "version": "0.38.0" + "version": "0.39.0" }, "plugins": [ { "name": "sci-paper", "source": "./", "description": "Scientific paper writing, source-traced review, and de-AI rewriting for top-tier journals (ApJ/MNRAS/PRD/JCAP-class) and NSF/NIH proposals. Ships 12 skills and 40 tools under one typed standard: scientific-integrity blockers that cannot be waived as style preferences, a zero-target L0 lexicon, ranked advisories, explicit measurement states so missing calibration never reads as clean, claim-first rewriting gated on hard scientific-fidelity eligibility, a mechanical length budget with a machine-built removal map and a required-shrink gate, corpus-calibrated salience-hierarchy, domain-register (with an exhaustive zero-hit vocabulary audit), collocation, and discourse-texture (given/new cohesion, epistemic hedging) axes, each judged against the field's own per-section distribution at the unit it was calibrated at, a residue axis that gates the trace an edit leaves, A-R review with narrative-spine and adversarial-escalation dimensions, registry-verified bibliographies (CrossRef, DataCite, arXiv), pixel-measured figure review, and isolated multi-reviewer orchestration. Learned scores are field-similarity triage, never an authorship verdict. See CHANGELOG.md for per-version history.", - "version": "0.38.0", + "version": "0.39.0", "author": { "name": "skymanbp", "email": "skyman.bp@gmail.com" diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 0ef2b9d..19a2fe0 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "sci-paper", - "version": "0.38.0", + "version": "0.39.0", "description": "Scientific paper writing, source-traced review, and de-AI rewriting for top-tier journals (ApJ/MNRAS/PRD/JCAP-class) and NSF/NIH proposals. Ships 12 skills and 40 tools under one typed standard: scientific-integrity blockers that cannot be waived as style preferences, a zero-target L0 lexicon, ranked advisories, explicit measurement states so missing calibration never reads as clean, claim-first rewriting gated on hard scientific-fidelity eligibility, a mechanical length budget with a machine-built removal map and a required-shrink gate, corpus-calibrated salience-hierarchy, domain-register (with an exhaustive zero-hit vocabulary audit), collocation, and discourse-texture (given/new cohesion, epistemic hedging) axes, each judged against the field's own per-section distribution at the unit it was calibrated at, a residue axis that gates the trace an edit leaves, A-R review with narrative-spine and adversarial-escalation dimensions, registry-verified bibliographies (CrossRef, DataCite, arXiv), pixel-measured figure review, and isolated multi-reviewer orchestration. Learned scores are field-similarity triage, never an authorship verdict. See CHANGELOG.md for per-version history.", "author": { "name": "skymanbp", diff --git a/CHANGELOG.md b/CHANGELOG.md index 0793aac..9980713 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,105 @@ All notable changes to the `sci-paper` plugin. Versions follow the `plugin.json` / `marketplace.json` `version` field. +## v0.39.0 — 2026-09-27 + +### The audit's first tier, fixed + +`docs/audits/full-review-2026-09-27.md` recorded a full-repository review of +v0.38.0: nine parallel line-by-line reads of the tools, skills and documents, +every high and medium finding reproduced first-party. This release acts on its +first tier ("应修") and the documentation corrections that go with it. No +consequence class changed; the standard is v3.9 (a fourth narrow exit +contract, `verify_references.py`, and the residues and readings v0.37–v0.38 had +added without a version stamp). + +**Gates no longer read missing evidence as a verdict.** An unreadable document +root raises: `length_gate --before ` read a zero-word baseline +and reported every section as growth (exit 1), and `ai_ism_lint dir.tex` read +an empty document and exited 0, clean. A `--git-ref` baseline now resolves its +`\input` children at the ref, so deleting a whole section file, the most +decisive condense move, is counted; before, the child vanished from the +baseline and the cut was understated. `\input{fig.tikz}` is read as written, +a nested `\input` resolves against the document root, and +`\includegraphics` is not an include. + +**The projection has one more display form.** `\[ ... \]`, `$$ ... $$` and +`subequations` are display mathematics in both projections. `\[E=mc^2\]` +counted as six words of prose in the gate and the map, and `$$` fell to the +inline pattern, whose second dollar paired with the third and swallowed the +prose up to the next formula in the plain view alone. A macro definition inside +a comment is no longer harvested, and an argument-taking macro with a numeric +body is left alone (`\foo{x}` became `42{x}`). + +**The condense map counts what the gate counts.** The abstract/conclusion +carve-out applies to either side of a restatement pair — the abstract precedes +the body, so with only the copy's bucket consulted every abstract claim made +the body's own statement the removable one and named the abstract as its home. +A copy whose negations or numbers differ from its home in either direction is +not a restatement (dropping `not` was still one). `removable_words` excludes +placeholders, the denominator excludes heading words, `7%` of 100 is 7 (the +float ceiling asked for 8), `--require-shrink 0` means no cut required, a +commented-out heading opens no section, any `\...ref` command keeps a float +alive, `\caption*` and nested captions are read, and `(Gaussian)` is not an +acronym. + +**The linter judges paragraphs, not lines.** A paragraph-initial connector is +read on the paragraph's first prose line; in one-sentence-per-line LaTeX a +sentence-initial `Notably,` mid-paragraph was an L0 target and the same text +joined onto one line was not. Openers are read where a sentence starts. +`paved` and `showcased` join Tier A (the skill's table and grep had them; the +regex did not). Every em-dash on a line is a target with its own id; +`\label`, `\ref`, `\cite` and `\url` arguments are blanked before the lexical +scan; `not only ... but also ... Bland` is no longer a three-part parallel; an +unparseable lexicon is exit 2; Markdown front matter is metadata; several +profiles without `--field` say so. + +**One field resolver.** The shared `--field` option promised auto-detection +while five tools divided the profile root by None and crashed with a +TypeError, the axis tools never auto-detected, and two tools carried private +loops. `cli_common.optional_field_dir` resolves it once; `axis_main` requires +an existing field for `--calibrate` and refuses to write an empty artifact. + +**The extractor's statistics are prose statistics.** Multi-word openers can be +observed (they were counted against a single-word counter and always read +absent); placeholders are not words and do not drop a citation-led paragraph +from the bank; words are Unicode letters; bank ids carry the bucket; the +profile directory is created after the corpus check; skipped PDFs are counted +in the summary; an empty `--topic` ranks by section in the keyword fallback. + +**The validator reads more.** Its shape regex accepts "N checks" and scans +`tools/README.md` (two "10 checks" survived two releases that ran eleven), and +a relative-link pass reports targets that do not exist — it found the sixteen +local-path links in the 2026-09-04 audit note on its first run. + +**The bibliography checker reads what LaTeX reads.** A `\cite` inside a `%` +comment raised an integrity blocker; `\nocite`, the biblatex commands and every +group of `\cites{a}{b}` were invisible; `@string` names were compared literally; +`#` concatenation and an escaped quote dropped every later field, DOI included; +a first author agreed on any substring (`Li` for `Lin`, `Ma` for `Mandelbaum`); +a registry miss was cached forever; `http.client` errors escaped as a traceback +at exit 1. Each is fixed and pinned; a finding built from a cached record now +says so, and any other execution failure is exit 2. + +**The document-level axes stop reporting `measured` where they cannot fire.** +Anchoring's conformal p cannot fall below 1/(n+1), so at the 30-document +class floor with two or more classes no finding was ever possible; the class is +now skipped and the axis `degraded`, and calibration records the floor the +Bonferroni share needs. A structure bucket the baseline lacks, and a +`sentence_stats.json` with no classified bucket, are `degraded` too. The +document-shape detector flagged by a rank rule while quoting a threshold and a +leave-one-out rate computed by a quantile rule; one rule now does both, and the +sweep drops the preamble and every `skip` unit on both sides. The partition +tool compares candidate states by conformal p instead of distances from +different manifolds, simulates a split on the original text (the joined text +let a trailing `%` comment swallow half a paragraph), and treats every heading +command as fixed. The tricolon wrap-up needs a wrap-up. `structure_baseline`, +`docstructure_baseline` and `anchoring_baseline` must be rebuilt for these to +take effect; the EVALUATION figures tied to them predate the rebuild and say so. + +Tests and counts are stated at the end of this entry once every part of the +release has landed. + ## v0.38.0 — 2026-09-16 ### The recurring unattested pair is the manuscript's own term diff --git a/README.md b/README.md index c09a225..b11dbda 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![CI](https://github.com/skymanbp/sci-paper/actions/workflows/ci.yml/badge.svg)](https://github.com/skymanbp/sci-paper/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) -[![Version](https://img.shields.io/badge/version-0.38.0-informational.svg)](CHANGELOG.md) +[![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) [![Tests](https://img.shields.io/badge/tests-529%20passing-success.svg)](tests/) @@ -694,7 +694,7 @@ independent review, clean-checkout verification, and green hosted CI. ## Status, known limitations, and roadmap -Current: **v0.38.0**. Full per-version history in [CHANGELOG.md](CHANGELOG.md). +Current: **v0.39.0**. Full per-version history in [CHANGELOG.md](CHANGELOG.md). **Normative core:** `docs/SCIPAPER_STANDARD.md` v3.8 — the complete de-AI standard in one file (layered model, document-scale detection core, cooperative diff --git a/README.zh-CN.md b/README.zh-CN.md index bc67a0f..f2ab693 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -2,7 +2,7 @@ [![CI](https://github.com/skymanbp/sci-paper/actions/workflows/ci.yml/badge.svg)](https://github.com/skymanbp/sci-paper/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) -[![Version](https://img.shields.io/badge/version-0.38.0-informational.svg)](CHANGELOG.md) +[![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) [![Tests](https://img.shields.io/badge/tests-529%20passing-success.svg)](tests/) @@ -684,7 +684,7 @@ Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权 ## 现状、已知限制与路线图 -当前版本：**v0.38.0**。完整逐版本历史见 [CHANGELOG.md](CHANGELOG.md)，更早的条目见 [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md)（v0.33.0–v0.34.0）、[CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md)（v0.27.1–v0.32.0）、[CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md)（v0.22.0–v0.27.0）与 [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md)（v0.1.0–v0.21.0）。 +当前版本：**v0.39.0**。完整逐版本历史见 [CHANGELOG.md](CHANGELOG.md)，更早的条目见 [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md)（v0.33.0–v0.34.0）、[CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md)（v0.27.1–v0.32.0）、[CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md)（v0.22.0–v0.27.0）与 [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md)（v0.1.0–v0.21.0）。 **规范核心：** `docs/SCIPAPER_STANDARD.md` v3.8 —— 完整的去 AI 标准全在这一个文件里 （分层模型、全文尺度检测核心、协作层与 residue 轴、`calibration_unit` 置信度封顶、§5.2 去 AI 化 diff --git a/docs/architecture/DEAI_SUBSYSTEM.md b/docs/architecture/DEAI_SUBSYSTEM.md index 47f04f6..1a9f989 100644 --- a/docs/architecture/DEAI_SUBSYSTEM.md +++ b/docs/architecture/DEAI_SUBSYSTEM.md @@ -1,4 +1,4 @@ -# De-AI subsystem architecture (current as of v0.38.0) +# De-AI subsystem architecture (current as of v0.39.0) ## 1. Purpose diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index 71bd9bb..569292e 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -1,4 +1,4 @@ -# EVALUATION: de-AI subsystem for `sci-paper` v0.38.0 +# EVALUATION: de-AI subsystem for `sci-paper` v0.39.0 First recorded 2026-07-12; every section-keyed measurement re-derived against the rebuilt `wgl` profile on 2026-08-17. @@ -154,7 +154,7 @@ The three drift events it exists to stop are recorded in §18.8. ## 12. Release evidence boundary -Current release gates (as of 2026-09-16; last tagged release v0.38.0): +Current release gates (as of 2026-09-27; last tagged release v0.38.0, this release v0.39.0): `validate_plugin.py` all 11 checks pass and the full unit/CLI suite (529 tests, 27 files) passes on a clean tree; both are rerun before every tag, and as of v0.25.1 the hosted CI run on the release commit must also be green From 116f0745e7c3bf844979d7a15a45f86ac52bd369 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:08:32 +0000 Subject: [PATCH 11/27] rewrite_reward: the fidelity gate runs without the optional model and counts what it compares MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit 2026-09-27 §3.5 (E1, E2, E3, E4, E12, E30, H1, E9, E32). The deterministic gate was hostage to two optional artifacts and misread several ordinary LaTeX forms: - H1/E9: a missing voice_model.joblib was exit 2, so a fresh clone or an untrained profile could not run the gate the de-ai and condense skills call mandatory. The field now resolves through cli_common.optional_field_dir (one field auto-detects; none or several means no profile); a missing or unusable bundle leaves the voice term unmeasured at weight 0 with one stderr line naming which, and the exit status is 0/1 by eligibility. The same rule is applied to the semantic-fidelity cosine when sentence-transformers is absent: unmeasured, weight 0, floor not applied, said on stderr. Exit 2 remains invalid input and crashes. - E1/E3: the numeral tokenizer read the LaTeX en-dash range 0.5--1.2 as {0.5, -1.2}, dropped the Unicode minus and the sign of an exponent, and started tokens inside identifiers (M200 -> "00", M200c -> unit "c"). A sign may not follow a sign character; U+2212 is a sign and folds to "-"; an exponent is part of the number; a token starts after no letter, digit or dot, with a leading-dot decimal kept as a form of its own. - E2/E4: CDM-like no longer yields the acronym "CDM-"; projection placeholders are stripped before the acronym scan; 3rd/1990s/5x yield no unit; numerals inside inline math are collected so display and inline forms agree (the math category already protects both). - E12: numbers, negation, comparison and causal markers are compared as bags, so a dropped second negation or a repeated value replaced by another is caught; the report names count mismatches, and the residual limit (a marker moved between clauses) is stated in the module and the report. - E30: ldots/dots/citeyearpar/citetext (and rm/mathrm, the math-mode twins of textrm) are formatting; {\rm Mpc} is protected as \mathrm{Mpc} when the vocabulary lists the unit; an unusable bundle is named as present. - E32: the exit-contract tests now reach the crash guard and cover 0/1/2. 33 tests added or rewritten in tests/test_rewrite_reward.py (562 in 27 files). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit 8e0371caced5ce20afce54ceb0e470ae08533877) --- tests/test_rewrite_reward.py | 307 ++++++++++++++++++++++++--- tools/rewrite_reward.py | 394 +++++++++++++++++++++++++++-------- 2 files changed, 584 insertions(+), 117 deletions(-) diff --git a/tests/test_rewrite_reward.py b/tests/test_rewrite_reward.py index 4b11b14..2851909 100644 --- a/tests/test_rewrite_reward.py +++ b/tests/test_rewrite_reward.py @@ -1,8 +1,13 @@ from __future__ import annotations -import sys +import io +import re +import tempfile import unittest +from collections import Counter +from contextlib import ExitStack, redirect_stderr, redirect_stdout from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -91,7 +96,7 @@ class InvariantTokenizationTests(unittest.TestCase): def test_list_comma_is_not_part_of_the_number(self): self.assertEqual( - rewrite_reward._numbers("We analyze 1200, 2400, and 4800 sources."), + set(rewrite_reward._numbers("We analyze 1200, 2400, and 4800 sources.")), {"1200", "2400", "4800"}) def test_thousands_separator_stays_inside_the_number(self): @@ -141,6 +146,46 @@ def test_changed_number_is_still_ineligible(self): self.assertIn("4800", result["missing"]["numbers"]) +class AdjacentSuffixTests(unittest.TestCase): + """An English suffix glued onto the digits is not a unit (`3rd`, `1990s`, `5x`).""" + + def test_suffixes_glued_to_digits_are_not_units(self): + for text in ("the 3rd run", "the 1990s", "a 5x gain", "the 21st", "the 2nd", "the 4th"): + with self.subTest(text=text): + self.assertEqual(rewrite_reward._units(text), set()) + + def test_typeset_seconds_stay_a_unit(self): + self.assertEqual(rewrite_reward._units(r"5\,s"), {"s"}) + self.assertEqual(rewrite_reward._units("5 s"), {"s"}) + + def test_rephrasing_around_a_decade_stays_eligible(self): + result = rewrite_reward.fidelity_eligibility("during the 1990s", "in the 1990s") + self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + + +class AcronymTests(unittest.TestCase): + """A hyphen is part of an acronym only where an acronym continues after it.""" + + def test_trailing_hyphen_is_not_part_of_an_acronym(self): + self.assertEqual(rewrite_reward._acronyms("a CDM-like model"), {"CDM"}) + self.assertEqual(rewrite_reward._acronyms("X-RAY data, paper II, HST-ACS and HST"), + {"X-RAY", "II", "HST-ACS", "HST"}) + + def test_hyphenating_the_acronym_stays_eligible(self): + result = rewrite_reward.fidelity_eligibility("a CDM model", "a CDM-like model") + self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + + def test_projection_placeholders_are_not_acronyms(self): + text = r"\citep{Smith2024} \begin{equation}y=x\end{equation} \begin{figure}z\end{figure}" + self.assertEqual(rewrite_reward._acronyms(text), set()) + + def test_removing_an_empty_float_is_not_an_acronym_loss(self): + reference = "We show the fit.\n\\begin{figure}\n\\centering\n\\end{figure}\nIt converges." + candidate = "We show the fit. It converges." + result = rewrite_reward.fidelity_eligibility(candidate, reference) + self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + + class DisplayMathFidelityTests(unittest.TestCase): """Numbers inside a displayed equation are protected. @@ -187,6 +232,18 @@ def test_rewording_prose_around_the_equation_stays_eligible(self): self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + def test_inline_math_numerals_are_collected_too(self): + # Inline and display forms are read symmetrically; see _numbers. + self.assertIn("0.5", rewrite_reward._numbers("at $z=0.5$")) + + def test_moving_an_equation_between_display_and_inline_is_not_a_change(self): + inline = (self.REFERENCE.replace("\n\\begin{equation}\n", " $") + .replace("\n\\end{equation}\n", "$ ")) + for candidate, reference in ((inline, self.REFERENCE), (self.REFERENCE, inline)): + with self.subTest(candidate=candidate[:24]): + result = rewrite_reward.fidelity_eligibility(candidate, reference) + self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + def test_commented_out_equation_is_not_an_invariant(self): # Both projections strip comments first; reading the raw span without # doing so made a dead commented-out equation a hard invariant, so @@ -197,7 +254,7 @@ def test_commented_out_equation_is_not_an_invariant(self): "% \\end{equation}\n" "The sample has 43 clusters.") candidate = "We adopt the fiducial cosmology. The sample has 43 clusters." - self.assertEqual(rewrite_reward._numbers(reference), {"43"}) + self.assertEqual(set(rewrite_reward._numbers(reference)), {"43"}) result = rewrite_reward.fidelity_eligibility(candidate, reference) self.assertTrue(result["eligible"], (result["missing"], result["invented"])) @@ -262,19 +319,170 @@ def test_ordinary_word_after_a_numeral_is_still_not_a_unit(self): (result["missing"], result["invented"])) +class FormattingMacroAndRmUnitTests(unittest.TestCase): + """`\\ldots`/`\\dots` and the natbib variants are formatting; `{\\rm Mpc}` is a unit.""" + + def test_dots_and_natbib_variants_are_not_semantic_macros(self): + self.assertEqual( + rewrite_reward._macros(r"a \ldots b \dots \citeyearpar{k} \citetext{see} \Nconfig"), + {"nconfig"}) + result = rewrite_reward.fidelity_eligibility("with 43 sources ...", + r"with 43 sources \ldots") + self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + + def test_rm_unit_is_protected_in_both_forms(self): + for text in (r"5\,{\rm Mpc}", r"5 {\rm Mpc}", r"5{\rm\,Mpc}"): + with self.subTest(text=text): + self.assertEqual(rewrite_reward._units(text), {r"\mathrm{mpc}"}) + # Only a vocabulary unit: `{\rm ...}` wraps far more than units. + self.assertEqual(rewrite_reward._units(r"5 {\rm foo}"), set()) + + def test_modernising_rm_is_not_a_unit_change_but_kpc_is(self): + reference = r"The scale is 5\,{\rm Mpc}." + modern = rewrite_reward.fidelity_eligibility(r"The scale is 5\,\mathrm{Mpc}.", reference) + self.assertTrue(modern["eligible"], (modern["missing"], modern["invented"])) + result = rewrite_reward.fidelity_eligibility(r"The scale is 5\,{\rm kpc}.", reference) + self.assertFalse(result["eligible"]) + self.assertIn(r"\mathrm{mpc}", result["missing"]["units"]) + + +class MultiplicityTests(unittest.TestCase): + """Numbers and markers are bags: a dropped second occurrence is a loss, + and a token still present with the wrong count is reported as such.""" + + REFERENCE = "The bias is not significant and the trend is not real." + + def test_dropped_second_negation_is_ineligible(self): + result = rewrite_reward.fidelity_eligibility( + "The bias is not significant and the trend is real.", self.REFERENCE) + self.assertFalse(result["eligible"]) + self.assertEqual(result["missing"]["negation"], ["not"]) + self.assertEqual(result["count_mismatches"], {"negation": {"not": (2, 1)}}) + + def test_both_negations_dropped_lists_each_occurrence(self): + result = rewrite_reward.fidelity_eligibility( + "The bias is significant and the trend is real.", self.REFERENCE) + self.assertEqual(result["missing"]["negation"], ["not", "not"]) + self.assertEqual(result["count_mismatches"], {}) + + def test_repeated_number_replaced_by_another_is_ineligible(self): + # Sets saw {43, 512} on both sides and passed this. + result = rewrite_reward.fidelity_eligibility( + "We find 43 here and 512 there among 512.", + "We find 43 here and 43 there among 512.") + self.assertFalse(result["eligible"]) + self.assertEqual(result["missing"]["numbers"], ["43"]) + self.assertEqual(result["invented"]["numbers"], ["512"]) + + def test_added_second_causal_marker_is_invention(self): + reference = "The bias shrinks because the sample grows; the scatter falls." + candidate = reference[:-1] + " because of it." + result = rewrite_reward.fidelity_eligibility(candidate, reference) + self.assertEqual(result["invented"]["causal_direction"], ["because"]) + + def test_repeated_citations_and_units_stay_sets(self): + reference = r"At 5 Mpc \citep{Doe2025} and again at 5 Mpc \citep{Doe2025}." + candidate = r"At 5 Mpc \citep{Doe2025}, and again at 5 Mpc." + result = rewrite_reward.fidelity_eligibility(candidate, reference) + self.assertTrue(result["eligible"], (result["missing"], result["invented"])) + + class MainExitContractTests(unittest.TestCase): - """Exit 1 means "no candidate eligible"; a crash must not borrow it.""" + """Exit 0/1 by eligibility, 2 for invalid input and crashes; a missing + optional model, profile or embedder is none of these (unmeasured, weight 0).""" + + REFERENCE = "The sample of 43 clusters has a bias of 0.5--1.2 dex.\n" + FAITHFUL = "For 43 clusters the bias spans 0.5 to 1.2 dex.\n" + UNFAITHFUL = "The bias spans 0.5 to 1.2 dex.\n" + + def run_cli(self, candidate=FAITHFUL, *, fields=(), field=None, bundle=None, + reference=REFERENCE, embedder=True, cosine=None, rank=None): + out, err = io.StringIO(), io.StringIO() + with ExitStack() as stack: + root = Path(stack.enter_context(tempfile.TemporaryDirectory())) + profiles = root / "profiles" + profiles.mkdir() + for name in fields: + (profiles / name).mkdir() + if bundle is not None: + (profiles / name / "voice_model.joblib").write_bytes(bundle) + if reference is not None: + (root / "ref.txt").write_text(reference, encoding="utf-8") + (root / "cand.txt").write_text(candidate, encoding="utf-8") + argv = ["--profile-root", str(profiles), "--reference", str(root / "ref.txt"), + "--candidates", str(root / "cand.txt")] + if field: + argv += ["--field", field] + stack.enter_context(mock.patch.object(rewrite_reward.df, "embedder_available", + return_value=embedder)) + stack.enter_context(mock.patch.object(rewrite_reward, "_cosine", + **(cosine or {"return_value": 0.9}))) + if rank is not None: + stack.enter_context(mock.patch.object(rewrite_reward, "rank", **rank)) + stack.enter_context(redirect_stdout(out)) + stack.enter_context(redirect_stderr(err)) + status = rewrite_reward.main(argv) + return status, out.getvalue(), err.getvalue() + + def test_crash_inside_rank_is_execution_failure(self): + status, _out, err = self.run_cli(rank={"side_effect": RuntimeError("boom")}) + self.assertEqual(status, 2) + self.assertIn("execution failed: RuntimeError: boom", err) def test_unreadable_input_is_execution_failure(self): - import io - from contextlib import redirect_stderr, redirect_stdout - with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): - status = rewrite_reward.main([ - "--field", "no-such-field", - "--reference", "no_such_reference.tex", - "--candidates", "no_such_candidate.tex", - ]) + status, _out, err = self.run_cli(reference=None) self.assertEqual(status, 2) + self.assertIn("execution failed: FileNotFoundError", err) + + def test_no_eligible_candidate_is_exit_1_even_without_a_profile(self): + status, out, err = self.run_cli(self.UNFAITHFUL) + self.assertEqual(status, 1) + self.assertIn("missing: {'numbers': ['43']}", out) + self.assertIn("no field profile", err) + self.assertIn("regenerate tighter", err) + + def test_eligible_candidate_is_exit_0(self): + status, out, _err = self.run_cli() + self.assertEqual(status, 0) + self.assertIn("[best] candidate 0", out) + + def test_missing_voice_model_is_unmeasured_not_exit_2(self): + status, out, err = self.run_cli(fields=("wgl",), field="wgl") + self.assertEqual(status, 0) + self.assertRegex(err, r"no voice_model\.joblib in \S*wgl: learned field-similarity " + r"score unmeasured, weight 0") + # voice column `-`, fidelity 0.900, combined = 0.3 * 0.9 with no voice term. + self.assertRegex(out, re.compile(r"^ +1 +0 +0\.270 +- +0\.900 ", re.M)) + + def test_single_field_is_auto_detected(self): + status, _out, err = self.run_cli(fields=("wgl",)) + self.assertEqual(status, 0) + self.assertRegex(err, r"no voice_model\.joblib in \S*wgl") + + def test_several_fields_run_without_a_profile(self): + status, _out, err = self.run_cli(fields=("a", "b")) + self.assertEqual(status, 0) + self.assertIn("several field profiles present", err) + self.assertIn("no field profile", err) + + def test_unusable_bundle_is_named_as_present(self): + status, _out, err = self.run_cli(fields=("wgl",), bundle=b"not a joblib bundle") + self.assertEqual(status, 0) + self.assertIn("voice_model.joblib is present but unusable", err) + self.assertNotIn("no voice_model.joblib", err) + + def test_missing_embedder_is_unmeasured_not_a_crash(self): + status, out, err = self.run_cli(embedder=False, + cosine={"side_effect": AssertionError("embedder used")}) + self.assertEqual(status, 0) + self.assertIn("semantic fidelity unmeasured, weight 0", err) + self.assertRegex(out, re.compile(r"^ +1 +0 +0\.000 +- +- ", re.M)) + + def test_count_mismatch_is_reported_with_both_counts(self): + _status, out, _err = self.run_cli( + "The bias is not significant and the trend is real.", + reference="The bias is not significant and the trend is not real.") + self.assertIn("count mismatch: negation 'not': 2 in reference, 1 in candidate", out) class LengthBudgetTests(unittest.TestCase): @@ -314,21 +522,16 @@ def test_empty_original_is_within_budget_with_zero_condensation(self): class RankLengthGateIntegrationTests(unittest.TestCase): """Protect the length gate's integration into rank(): -inf for over-budget candidates, --allow-growth lift, and the fidelity floor on the bonus. - Heavy dependencies (embedder, voice model, centroid) are mocked out.""" + Heavy dependencies (embedder, voice model) are mocked out.""" ORIGINAL = "The estimator is stable across the five smoothing scales." REFERENCE = "claim: estimator stable across five smoothing scales" def rank_with_mocks(self, candidates, fidelity, **kwargs): - from unittest import mock - with mock.patch.object(rewrite_reward.df, "corpus_centroid", - return_value=None), \ - mock.patch.object(rewrite_reward.dv, "voice_score", - return_value=0.0), \ - mock.patch.object(rewrite_reward.dv, "load_voice_model", + with mock.patch.object(rewrite_reward.dv, "load_voice_model", return_value=None), \ - mock.patch.object(rewrite_reward.dv, "bundle_measured", - return_value=False), \ + mock.patch.object(rewrite_reward.df, "embedder_available", + return_value=True), \ mock.patch.object(rewrite_reward, "_cosine", return_value=fidelity), \ mock.patch.object(rewrite_reward, "_l0_target_count", @@ -360,6 +563,26 @@ def test_condensation_bonus_requires_fidelity_floor(self): self.assertEqual(bonus_low, 0.0) +class UnmeasuredTermTests(unittest.TestCase): + """A term that cannot be measured is None at weight 0, never a nominal 0.0.""" + + def test_floor_is_not_applied_to_an_unmeasured_fidelity(self): + self.assertEqual(rewrite_reward._advisory_reduction(2, 0, None), 1.0) + self.assertEqual(rewrite_reward._advisory_reduction(2, 0, 0.1), 0.0) + + def test_rank_without_bundle_or_embedder_keeps_the_gate(self): + with mock.patch.object(rewrite_reward.df, "embedder_available", return_value=False), \ + mock.patch.object(rewrite_reward, "_cosine", side_effect=AssertionError): + ranked = rewrite_reward.rank(["We find 43 sources.", "We find sources."], + "We find 43 sources.", None) + (best, result), (_worst, rejected) = ranked + self.assertEqual(best, 0) + self.assertIsNone(result["fidelity"]) + self.assertIsNone(result["voice"]) + self.assertEqual(result["combined"], 0.0) + self.assertEqual(rejected["combined"], float("-inf")) + + class AdvisoryReductionTests(unittest.TestCase): """Rank 7: the ranking term is L0 advisory reduction, not the dead specificity term (identically 1.0 for every eligible candidate).""" @@ -410,7 +633,9 @@ class HyphenatedRangeTests(unittest.TestCase): reported as MISSING "-1.2" while INVENTING "1.2", then hard-rejected at combined = -inf. Every hyphenated range in a reference did this, which is most of them. Third occurrence of one root cause -- a separator absorbed - into the token -- after the Oxford comma and the spaced unit. + into the token -- after the Oxford comma and the spaced unit. The LaTeX + en-dash `--` was the fourth: its second hyphen follows a hyphen, not a + digit, so the digit/dot rule alone still read it as a sign. """ def numbers(self, text): @@ -426,6 +651,10 @@ def test_hyphenated_ranges_yield_two_positive_numbers(self): with self.subTest(text=text): self.assertEqual(self.numbers(text), want) + def test_latex_dashes_are_separators_too(self): + self.assertEqual(self.numbers("seeing 0.5--1.2 arcsec"), {"0.5", "1.2"}) + self.assertEqual(self.numbers("5---10 per field"), {"5", "10"}) + def test_genuine_negatives_are_still_signed(self): self.assertEqual(self.numbers("a bias of -0.06 dex"), {"-0.06"}) self.assertEqual(self.numbers("from -3 to +5"), {"-3", "+5"}) @@ -441,10 +670,12 @@ def test_earlier_separator_fixes_still_hold(self): self.assertEqual(self.numbers("1,234 sources"), {"1,234"}) def test_a_range_rewritten_as_prose_is_eligible(self): - reference = "The grid spans 12 seeing values from 0.5-1.2 arcsec." - candidate = "The grid samples 12 seeing values between 0.5 and 1.2 arcsec." - result = rewrite_reward.fidelity_eligibility(candidate, reference) - self.assertTrue(result["eligible"], result["missing"]) + for dash in ("-", "--"): + reference = f"The grid spans 12 seeing values from 0.5{dash}1.2 arcsec." + candidate = "The grid samples 12 seeing values between 0.5 and 1.2 arcsec." + with self.subTest(dash=dash): + result = rewrite_reward.fidelity_eligibility(candidate, reference) + self.assertTrue(result["eligible"], result["missing"]) def test_a_range_endpoint_actually_dropped_is_still_caught(self): reference = "The grid spans 12 seeing values from 0.5-1.2 arcsec." @@ -454,5 +685,29 @@ def test_a_range_endpoint_actually_dropped_is_still_caught(self): self.assertIn("1.2", result["missing"]["numbers"]) +class SignedAndIdentifierNumeralTests(unittest.TestCase): + """Corrections (2), (3), (5) and (6) of the numeral record.""" + + def test_unicode_minus_is_a_sign_and_folds_to_ascii(self): + self.assertEqual(rewrite_reward._numbers("a bias of \u22120.06 dex"), + Counter({"-0.06": 1})) + reference = "a bias of \u22120.06 dex" + self.assertFalse(rewrite_reward.fidelity_eligibility("a bias of 0.06 dex", + reference)["eligible"]) + self.assertTrue(rewrite_reward.fidelity_eligibility("a bias of -0.06 dex", + reference)["eligible"]) + + def test_exponent_belongs_to_the_number(self): + self.assertEqual(set(rewrite_reward._numbers("a rate of 1.5e-3 per year")), {"1.5e-3"}) + self.assertFalse(rewrite_reward.fidelity_eligibility("1.5e+3", "1.5e-3")["eligible"]) + + def test_a_token_does_not_start_inside_an_identifier(self): + self.assertEqual(rewrite_reward._numbers("M200 and z0.5"), Counter()) + self.assertEqual(rewrite_reward._units("M200c"), set()) + + def test_leading_dot_decimal_is_one_number(self): + self.assertEqual(set(rewrite_reward._numbers("a bias of .06 dex")), {".06"}) + + if __name__ == "__main__": unittest.main() diff --git a/tools/rewrite_reward.py b/tools/rewrite_reward.py index ac8bf07..1e206e0 100644 --- a/tools/rewrite_reward.py +++ b/tools/rewrite_reward.py @@ -6,18 +6,33 @@ bidirectional: dropping a protected invariant AND inventing one the reference does not carry both make a candidate ineligible, because specificity never licenses invention and an added negation/causal/comparison marker can invert -meaning while every original token survives. Only eligible candidates are -ranked; ranking is led by deterministic L0 advisory reduction and semantic -fidelity, and the learned field-similarity score contributes materially only -when its bundle carries a measured operating point. A style score can never -rescue an ineligible candidate. +meaning while every original token survives. Numbers and the three marker +categories are compared as bags (``collections.Counter``), not sets: ``not`` +said twice is not preserved by ``not`` said once, nor a value stated twice by +one copy of it. A bag is blind to position, and that is the residual limit: a +negation or causal marker moved into another clause, or two distinct numbers +exchanging places, changes no count and passes this gate -- the by-hand +verification the de-ai skill requires after it covers that class, and the +report's ``count mismatch`` line says where a bag came up short rather than +empty. + +Only eligible candidates are ranked; ranking is led by deterministic L0 +advisory reduction and semantic fidelity, and the learned field-similarity +score contributes materially only when its bundle carries a measured operating +point. A style score can never rescue an ineligible candidate. Every learned +or embedded term is optional and degrades visibly instead of blocking the +gate: with no field profile or no ``voice_model.joblib`` the voice term is +unmeasured at weight 0, and with sentence-transformers uninstalled the +semantic-fidelity cosine is unmeasured at weight 0 and cannot apply its floor. +Each case prints one stderr line naming what is missing; the exit status stays +0/1 by eligibility, and 2 is reserved for invalid input and crashes. """ from __future__ import annotations -import argparse import re import sys +from collections import Counter from pathlib import Path from typing import Any @@ -28,29 +43,58 @@ REPO_ROOT = Path(__file__).resolve().parent.parent DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" -# A numeral token must END in a digit. `\d[\d,]*` was greedy across the +# The numeral tokenizer, with the record of its corrections. Every entry is +# pinned by a test in tests/test_rewrite_reward.py; extend the record, do not +# prune it. Five of the six are one root cause -- a neighbour absorbed into +# the token, or the token started inside a neighbour. +# +# (1) A numeral token must END in a digit. `\d[\d,]*` was greedy across the # comma that separates list items, so "1200, 2400, and 4800" tokenized as # {"1200,", "2400,", "4800"} and a candidate that merely dropped the Oxford # comma was reported as simultaneously MISSING "2400," and INVENTING "2400" # -- a faithful rewrite hard-rejected (combined = -inf) on punctuation. The # thousands separator inside a number ("1,234") is still captured. -_NUMBER_BODY = r"\d(?:[\d,]*\d)?(?:\.\d+)?" -# ...and it must not swallow the separator in FRONT of it either, which is the -# same root cause a third time. `[-+]?` accepted a sign directly after a digit, -# so the range "0.5-1.2 arcsec" tokenized as {"0.5", "-1.2"}. A rewrite saying -# "from 0.5 to 1.2" was then reported as MISSING "-1.2" and INVENTING "1.2" and -# hard-rejected -- and every hyphenated range in a reference did the same, which -# is most of them ("5-40", "24.0-26.0", "0.01-0.06"). A sign is a sign only -# where one can occur: not straight after a digit or a decimal point, where `-` -# separates a range and `+` states a tolerance. Exponents (`10^-3`) and genuine -# negatives after a space or brace are unaffected. Shared with the unit regexes -# below, which tokenize the same numerals and had the same defect. -_NUMBER_SIGN = r"(?:(? `1.5e+3` passed. +_NUMBER_BODY = r"(?:\d(?:[\d,]*\d)?(?:\.\d+)?|\.\d+)(?:[eE][-+]?\d+)?" +# (4) ...and it must not swallow the separator in FRONT of it either, which is +# the same root cause a third time. `[-+]?` accepted a sign directly after a +# digit, so the range "0.5-1.2 arcsec" tokenized as {"0.5", "-1.2"}. A rewrite +# saying "from 0.5 to 1.2" was then reported as MISSING "-1.2" and INVENTING +# "1.2" and hard-rejected -- and every hyphenated range in a reference did the +# same, which is most of them ("5-40", "24.0-26.0", "0.01-0.06"). A sign is a +# sign only where one can occur: not straight after a digit or a decimal point, +# where `-` separates a range and `+` states a tolerance. Exponents (`10^-3`) +# and genuine negatives after a space or brace are unaffected. +# +# (5) Nor straight after another sign character (2026-09-27). The LaTeX +# en-dash range "0.5--1.2" puts a hyphen before its second hyphen, which (4) +# did not exclude, so it tokenized as {"0.5", "-1.2"} all over again and every +# faithful rewrite of an en-dash range -- the corpus's usual spelling -- was +# hard-rejected; `---` is the same case. The Unicode minus U+2212 joins the +# sign class: "−0.06" tokenized as "0.06", so dropping the sign passed. +# `_normalize` folds U+2212 to `-`, so both spellings of a negative are one token. +_NUMBER_SIGN = r"(?:(?{_LATEX_THIN_SPACE})" # `\%` before bare `%`: an unescaped `%` starts a LaTeX comment and is # stripped upstream (as it always has been for every projection-based # category), so `10\%` is the form that survives to be measured. - rf"(\\mathrm\{{[^}}]+\}}|\\text\{{[^}}]+\}}|\\%|[%°]|{_UNIT_TOKEN})" + rf"(?P\\mathrm\{{[^}}]+\}}|\\text\{{[^}}]+\}}|{_UNIT_RM}|\\%|[%°]|{_UNIT_TOKEN})" ) -_UNIT_SPACED_RE = re.compile(rf"(?{_UNIT_RM}|{_UNIT_TOKEN})") +# Glued straight onto the digits with no separator at all, these are English +# suffixes, not units: `3rd`, `1990s`, `5x` yielded the units "rd", "s", "x" +# and rejected a rewrite that spelled the ordinal or decade out (2026-09-27). +# `s` is also the second: it stays a unit behind a space or LaTeX spacing +# (`5 s`, `5\,s`), where an author has typeset a quantity. +_ADJACENT_SUFFIXES = frozenset({"st", "nd", "rd", "th", "s", "x"}) # Physical units a numeral is followed by in this literature. Deliberately a # closed vocabulary and deliberately NOT a detection policy: it decides only # whether a space-separated token is eligible to be a protected invariant, so a @@ -115,15 +173,20 @@ _MACRO_RE = re.compile(r"\\([a-zA-Z]+)\*?") # Pure-formatting commands whose loss cannot drop scientific content; every # other macro may carry a quantity (\Nconfig -> 750) or a semantic object, so it -# is protected. Citation commands are covered separately by _CITE_RE. +# is protected. Citation commands are covered separately by _CITE_RE; the +# natbib variants that were missing here (`\citeyearpar`, `\citetext`) made a +# citation rewritten between them a macro violation, and `\ldots`/`\dots` +# made `\ldots` -> `...` one (2026-09-27). `\rm`/`\mathrm` are the math-mode +# twins of `\textrm`; without them here `_rm_unit`'s equivalence was undone. _FORMATTING_MACROS = frozenset({ "emph", "textbf", "textit", "texttt", "textsc", "textrm", "textup", + "rm", "mathrm", "underline", "mbox", "hbox", "vspace", "hspace", "noindent", "centering", "raggedright", "small", "footnotesize", "scriptsize", "large", "item", "begin", "end", "label", "documentclass", "usepackage", "newcommand", "renewcommand", "left", "right", "par", "linebreak", - "newline", "cite", "citep", "citet", "citealt", "citealp", - "citeauthor", "citeyear", + "newline", "ldots", "dots", "cite", "citep", "citet", "citealt", "citealp", + "citeauthor", "citeyear", "citeyearpar", "citetext", }) # The ranking term that used to be `specificity` (fraction of reference numbers # preserved) is identically 1.0 for every eligible candidate, because @@ -139,14 +202,18 @@ # unless growth is explicitly allowed with a recorded reason, and within budget # a small condensation bonus prefers the shorter of otherwise-equal candidates. CONDENSATION_WEIGHT = 0.1 +# Bag (multiset) categories; the rest are sets, because a repeated citation, +# unit, acronym or math span is a restatement a condensation may fold. +_BAG_CATEGORIES = ("numbers", "comparison_direction", "negation", "causal_direction") -def _l0_target_count(text: str, field_profile_dir: Path) -> int: +def _l0_target_count(text: str, field_profile_dir: Path | None) -> int: """Deterministic L0 rewrite-target count (em-dash + Tier A + over-cap Tier B). Reused from ai_ism_lint, imported lazily so rewrite_reward stays importable without the linter and no module-load cycle is introduced. Location is a - placeholder path; only the finding kinds are counted. + placeholder path; only the finding kinds are counted, and a profile adds + advisory lexicon findings only, never targets, so it does not change them. """ import ai_ism_lint as lint # noqa: E402 lazy: avoid import cycle + optional dep findings, _axes = lint.lexical_findings(text, Path(""), @@ -154,23 +221,34 @@ def _l0_target_count(text: str, field_profile_dir: Path) -> int: return sum(1 for finding in findings if finding.get("kind") == "l0_target") -def _advisory_reduction(ref_l0: int, cand_l0: int, fidelity: float) -> float: +def _clears_floor(fidelity: float | None) -> bool: + """Whether the fidelity floor permits credit; an unmeasured cosine cannot + apply it, so the credit stands and `_unmeasured_terms` says so.""" + return fidelity is None or fidelity >= FIDELITY_FLOOR + + +def _advisory_reduction(ref_l0: int, cand_l0: int, fidelity: float | None) -> float: """Signed reduction in L0 targets, in [-1, 1], with a fidelity floor. Positive when a candidate removes reference L0 targets, negative when it adds them, zero when it neither helps nor hurts (so a clean reference gives no candidate spurious credit). Below ``FIDELITY_FLOOR`` a candidate cannot keep positive credit: it may not buy L0 improvement with mangled meaning. + With the cosine unmeasured (``None``, no embedder installed) the floor has + nothing to test and is not applied; the run reports the degradation. """ denom = max(1, ref_l0, cand_l0) value = (ref_l0 - cand_l0) / denom - if fidelity < FIDELITY_FLOOR: + if not _clears_floor(fidelity): value = min(value, 0.0) return value -def _normalized(items) -> set[str]: - return {re.sub(r"\s+", "", item).lower() for item in items} +def _normalize(token: str) -> str: + """Whitespace-insensitive, case-folded, with the Unicode minus read as `-`: + U+2212 and the hyphen-minus typeset the same negative number, and kept + apart they reported `-0.06` for `−0.06` as one number dropped, one invented.""" + return re.sub(r"\s+", "", token).lower().replace("\u2212", "-") _ENV_WRAPPER_RE = re.compile(r"\\(?:begin|end)\{[^}]*\}") @@ -220,11 +298,25 @@ def _display_math_bodies(text: str) -> list[str]: return bodies -def _numbers(text: str) -> set[str]: - numbers = _normalized(_NUM_RE.findall(df.es.latex_to_plain(text))) +def _numbers(text: str) -> Counter: + """Numerals of the prose, of inline spans and of displayed equations, by count. + + Inline and displayed mathematics are read symmetrically (2026-09-27). The + plain projection turns every math span into a placeholder, so numerals + were collected from displayed bodies (v0.27.0) but never from inline + spans, and a formula moved from display to inline, or back, was rejected + twice over: its numerals went "missing" or came "invented" while the math + category, which reads both forms, saw no change. Of the two symmetric + repairs, inline numerals are collected (through the numeral projection, + which exists to keep them) rather than display numerals dropped: the math + category already rejects any change inside a span, so this adds no + rejection the gate did not make, whereas dropping display numerals would + take a changed value out of the ``numbers`` report the v0.27.0 tests pin. + """ + tokens = _NUM_RE.findall(df.es.latex_to_numeral_text(text)) for body in _display_math_bodies(text): - numbers |= _normalized(_NUM_RE.findall(body)) - return numbers + tokens.extend(_NUM_RE.findall(body)) + return Counter(_normalize(token) for token in tokens) def _citations(text: str) -> set[str]: @@ -256,16 +348,52 @@ def _math(text: str) -> set[str]: def _acronyms(text: str) -> set[str]: - return set(_ACRONYM_RE.findall(df.es.latex_to_plain(text))) + """Uppercase acronyms of the rendered prose, projection placeholders excluded. + + The plain projection writes `[MATH]`, `[CITE]` and `[FIGURE-OR-TABLE]` + where an equation, a citation or a float stood, each shaped like an + acronym, so a candidate that turned a `\\citep` into prose "dropped" the + acronym `CITE` (2026-09-27). `prose_words` owns placeholder removal. + """ + return set(_ACRONYM_RE.findall(" ".join(df.es.prose_words(text)))) + + +def _rm_unit(token: str) -> str | None: + """`{\\rm Mpc}` as the `\\mathrm{Mpc}` it is the older spelling of, or None. + + Only a vocabulary unit qualifies: `{\\rm ...}` wraps far more than units + (`M_{\\rm vir}`, `{\\rm d}x`), so unlike `\\mathrm{}` after a numeral it is + not a unit on position alone. Both spellings render identically, so + modernising one into the other is no unit change; `{\\rm kpc}` for `{\\rm Mpc}` is. + """ + match = _UNIT_RM_INNER_RE.fullmatch(token) + if match is None: + return None + inner = match.group(1) + head = re.split(r"[\s/-]", inner, maxsplit=1)[0].lower() + return rf"\mathrm{{{inner}}}" if head in _UNIT_VOCABULARY else None + + +def _unit_token(match: re.Match, *, spaced: bool) -> str | None: + token = match.group("unit") + if token.startswith("{"): + return _rm_unit(token) + if spaced: + head = re.split(r"[\s/-]", token, maxsplit=1)[0].lower() + return token if head in _UNIT_VOCABULARY else None + if not match.group("sep") and token.lower() in _ADJACENT_SUFFIXES: + return None + return token def _units(text: str) -> set[str]: prose = _uncommented(text) - units = _normalized(_UNIT_BOUND_RE.findall(prose)) - for token in _UNIT_SPACED_RE.findall(prose): - head = re.split(r"[\s/-]", token, maxsplit=1)[0].lower() - if head in _UNIT_VOCABULARY: - units |= _normalized([token]) + units = set() + for spaced, pattern in ((False, _UNIT_BOUND_RE), (True, _UNIT_SPACED_RE)): + for match in pattern.finditer(prose): + token = _unit_token(match, spaced=spaced) + if token is not None: + units.add(_normalize(token)) return units @@ -280,8 +408,8 @@ def _macros(text: str) -> set[str]: if name.lower() not in _FORMATTING_MACROS} -def _markers(pattern: re.Pattern, text: str) -> set[str]: - return {match.lower() for match in pattern.findall(df.es.latex_to_plain(text))} +def _markers(pattern: re.Pattern, text: str) -> Counter: + return Counter(match.lower() for match in pattern.findall(df.es.latex_to_plain(text))) def _cosine(left: str, right: str) -> float: @@ -316,7 +444,9 @@ def length_budget(candidate: str, original: str) -> dict[str, Any]: } -def protected_invariants(text: str) -> dict[str, set[str]]: +def protected_invariants(text: str) -> dict[str, Any]: + """Every protected category: a ``Counter`` for the bag categories + (``_BAG_CATEGORIES``), a ``set`` for the rest.""" return { "numbers": _numbers(text), "units": _units(text), @@ -330,6 +460,26 @@ def protected_invariants(text: str) -> dict[str, set[str]]: } +def _listed(values) -> list[str]: + """A category as a sorted list, one entry per occurrence for a bag.""" + return sorted(values.elements()) if isinstance(values, Counter) else sorted(values) + + +def _count_mismatches(required: dict, present: dict) -> dict[str, dict[str, tuple[int, int]]]: + """Bag tokens both sides carry with different multiplicities: "missing + negation: not" beside a candidate that still says "not" then reads as what + it is, the second occurrence went. A bag is blind to position, so a marker + moved between clauses changes no count and is not detected here.""" + mismatches: dict[str, dict[str, tuple[int, int]]] = {} + for category in _BAG_CATEGORIES: + both = {token: (count, present[category][token]) + for token, count in required[category].items() + if present[category][token] and present[category][token] != count} + if both: + mismatches[category] = both + return mismatches + + def fidelity_eligibility(candidate: str, reference: str) -> dict[str, Any]: """Bidirectional invariant check. @@ -339,43 +489,52 @@ def fidelity_eligibility(candidate: str, reference: str) -> dict[str, Any]: unsourced invention, and added negation/causal/comparison markers can invert the claim. A human may re-approve an "invented" candidate only after manual source-tracing outside this deterministic gate. + + The bag categories (``_BAG_CATEGORIES``) are compared by multiplicity, so + both lists carry one entry per lost or added OCCURRENCE: ``["not", "not"]`` + when both negations went, ``["not"]`` when only the second did, a case + ``count_mismatches`` names with both counts. A bag cannot see position: a + marker moved into another clause, or two distinct numbers exchanging + places, keeps every count and passes; the by-hand check covers that. """ required = protected_invariants(reference) present = protected_invariants(candidate) - missing = { - category: sorted(values - present[category]) - for category, values in required.items() - if values - present[category] - } - invented = { - category: sorted(present[category] - required[category]) - for category in required - if present[category] - required[category] - } + missing = {category: _listed(values - present[category]) + for category, values in required.items() + if values - present[category]} + invented = {category: _listed(present[category] - required[category]) + for category in required + if present[category] - required[category]} return { "eligible": not missing and not invented, "missing": missing, "invented": invented, - "required": {key: sorted(values) for key, values in required.items()}, - "present": {key: sorted(values) for key, values in present.items()}, + "count_mismatches": _count_mismatches(required, present), + "required": {key: _listed(values) for key, values in required.items()}, + "present": {key: _listed(values) for key, values in present.items()}, } -def reward(candidate: str, reference: str, field_profile_dir: Path, +def reward(candidate: str, reference: str, field_profile_dir: Path | None, centroid=None, ref_l0: int | None = None, original: str | None = None, allow_growth: bool = False) -> dict[str, Any]: - if centroid is None: - centroid = df.corpus_centroid(field_profile_dir) - voice = dv.voice_score(candidate, field_profile_dir, centroid=centroid) - voice = 0.0 if voice is None else voice - voice_calibrated = dv.bundle_measured(dv.load_voice_model(field_profile_dir)) - fidelity = _cosine(candidate, reference) + """Every term of one candidate. ``voice`` and ``fidelity`` are ``None`` when + unmeasured (no usable bundle; no embedder), never a nominal 0.0.""" + bundle = dv.load_voice_model(field_profile_dir) + if bundle is None: + voice = None + else: + if centroid is None: + centroid = df.corpus_centroid(field_profile_dir) + voice = dv.voice_score(candidate, field_profile_dir, centroid=centroid) + voice_calibrated = dv.bundle_measured(bundle) + fidelity = _cosine(candidate, reference) if df.embedder_available() else None required_numbers = _numbers(reference) candidate_numbers = _numbers(candidate) # Reported for transparency (confirms eligibility), but NOT a ranking term: # it is identically 1.0 for every eligible candidate. See ADVISORY_REDUCTION. - specificity = (len(required_numbers & candidate_numbers) / len(required_numbers) + specificity = ((required_numbers & candidate_numbers).total() / required_numbers.total() if required_numbers else 1.0) if ref_l0 is None: ref_l0 = _l0_target_count(reference, field_profile_dir) @@ -392,17 +551,18 @@ def reward(candidate: str, reference: str, field_profile_dir: Path, "advisory_reduction": advisory_reduction, "n_l0_ref": ref_l0, "n_l0_cand": cand_l0, - "n_num_ref": len(required_numbers), - "n_num_cand": len(candidate_numbers), + "n_num_ref": required_numbers.total(), + "n_num_cand": candidate_numbers.total(), "faithful": eligibility["eligible"], "missing_invariants": eligibility["missing"], "invented_invariants": eligibility["invented"], + "count_mismatches": eligibility["count_mismatches"], "length_budget": budget, "length_eligible": length_eligible, } -def rank(candidates: list[str], reference: str, field_profile_dir: Path, +def rank(candidates: list[str], reference: str, field_profile_dir: Path | None, original: str | None = None, allow_growth: bool = False ) -> list[tuple[int, dict[str, Any]]]: """Rank eligible candidates first; ineligible candidates can never win. @@ -411,37 +571,43 @@ def rank(candidates: list[str], reference: str, field_profile_dir: Path, L3 score is a low-weight tie-break unless its bundle is measured/calibrated, so an uncalibrated field-similarity model can never be the deciding term (SCIPAPER_STANDARD section 9.5: lower detector visibility is not - independently valuable). When ``original`` is supplied, the section-5.3 - length budget is a second hard gate: a candidate longer than the original - scores -inf unless ``allow_growth`` records an explicit exception, and a - condensation bonus prefers the shorter of otherwise-equal candidates. + independently valuable). A term that is unmeasured -- no usable bundle, no + embedder -- is left out of ``combined`` (weight 0) rather than scored 0.0, + and the eligibility gate runs regardless. When ``original`` is supplied, + the section-5.3 length budget is a second hard gate: a candidate longer + than the original scores -inf unless ``allow_growth`` records an explicit + exception, and a condensation bonus prefers the shorter of otherwise-equal + candidates. """ - centroid = df.corpus_centroid(field_profile_dir) + # The centroid feeds the learned score only; loading it needs numpy, which + # a run without a bundle must not require. + centroid = (df.corpus_centroid(field_profile_dir) + if dv.load_voice_model(field_profile_dir) is not None else None) ref_l0 = _l0_target_count(reference, field_profile_dir) # constant across candidates scored = [] for index, candidate in enumerate(candidates): result = reward(candidate, reference, field_profile_dir, centroid, ref_l0=ref_l0, original=original, allow_growth=allow_growth) if result["faithful"] and result["length_eligible"]: - voice_weight = 0.4 if result["voice_calibrated"] else 0.05 - result["combined"] = ( - ADVISORY_REDUCTION_WEIGHT * result["advisory_reduction"] - + 0.3 * result["fidelity"] - + voice_weight * result["voice"] - ) + combined = ADVISORY_REDUCTION_WEIGHT * result["advisory_reduction"] + if result["fidelity"] is not None: + combined += 0.3 * result["fidelity"] + if result["voice"] is not None: + combined += (0.4 if result["voice_calibrated"] else 0.05) * result["voice"] # The condensation bonus sits behind the same fidelity floor as # advisory reduction: a degenerate ultra-short candidate may not # buy ranking credit with dropped meaning. - if (result["length_budget"] is not None - and result["fidelity"] >= FIDELITY_FLOOR): - result["combined"] += (CONDENSATION_WEIGHT - * max(0.0, result["length_budget"]["condensation"])) + if result["length_budget"] is not None and _clears_floor(result["fidelity"]): + combined += (CONDENSATION_WEIGHT + * max(0.0, result["length_budget"]["condensation"])) + result["combined"] = combined else: result["combined"] = float("-inf") scored.append((index, result)) scored.sort(key=lambda item: ( item[1]["faithful"] and item[1]["length_eligible"], - item[1]["combined"], item[1]["fidelity"]), + item[1]["combined"], + -1.0 if item[1]["fidelity"] is None else item[1]["fidelity"]), reverse=True) return scored @@ -473,11 +639,53 @@ def main(argv: list[str] | None = None) -> int: return 2 +def _unmeasured_terms(field_dir: Path | None) -> list[str]: + """One stderr line per optional term this run cannot measure. + + Neither line is a refusal: the term is unmeasured at weight 0 and the gate + and the exit status by eligibility go on. Until 2026-09-27 a missing + ``voice_model.joblib`` was exit 2, so a fresh clone or an untrained + profile could not run the fidelity gate at all, although the de-ai and + condense skills use it as a mandatory gate and calibrate documents the + learned model as never required. A bundle present but unusable + (unreadable, joblib uninstalled, feature-schema mismatch) is named as + such: `deai_voice` has just printed why, and "no voice_model.joblib" + would send the reader to train a model they already have. + """ + unmeasured = "learned field-similarity score unmeasured, weight 0" + lines = [] + if field_dir is None: + lines.append("no field profile (pass --field, or keep exactly one field " + f"under the profile root): {unmeasured}") + elif dv.load_voice_model(field_dir) is None: + artifact = field_dir / "voice_model.joblib" + if artifact.exists(): + lines.append(f"{artifact} is present but unusable (see the deai_voice " + f"line above): {unmeasured}") + else: + lines.append(f"no voice_model.joblib in {field_dir}: {unmeasured}") + if not df.embedder_available(): + lines.append("sentence-transformers is not installed: semantic fidelity " + "unmeasured, weight 0; its floor on the advisory-reduction credit " + "and the condensation bonus cannot apply; verify meaning by hand") + return lines + + +def _cell(value: float | None) -> str: + return "-" if value is None else f"{value:.3f}" + + +def _render_counts(mismatches: dict[str, dict[str, tuple[int, int]]]) -> str: + return "; ".join( + f"{category} {token!r}: {n_ref} in reference, {n_cand} in candidate" + for category, tokens in mismatches.items() + for token, (n_ref, n_cand) in tokens.items()) + + def _run(args) -> int: - field_dir = args.profile_root / args.field - if dv.load_voice_model(field_dir) is None: - print(f"[rewrite_reward] no voice_model.joblib in {field_dir}", file=sys.stderr) - return 2 + field_dir = cli_common.optional_field_dir(args, tool="rewrite_reward") + for line in _unmeasured_terms(field_dir): + print(f"[rewrite_reward] {line}", file=sys.stderr) reference = args.reference.read_text(encoding="utf-8", errors="replace") candidates = [path.read_text(encoding="utf-8", errors="replace") for path in args.candidates] @@ -497,7 +705,7 @@ def _run(args) -> int: if budget is not None else "-") eligible_flag = result["faithful"] and result["length_eligible"] print(f"{position:>4} {index:>4} {combined_text:>9} " - f"{result['voice']:>7.3f} {result['fidelity']:>9.3f} " + f"{_cell(result['voice']):>7} {_cell(result['fidelity']):>9} " f"{result['advisory_reduction']:>6.2f} {str(eligible_flag):>8} " f"{result['n_l0_ref']}/{result['n_l0_cand']} {words_text} " f"{args.candidates[index].name}") @@ -505,6 +713,10 @@ def _run(args) -> int: print(f" missing: {result['missing_invariants']}") if result["invented_invariants"]: print(f" invented: {result['invented_invariants']}") + if result["count_mismatches"]: + print(f" count mismatch: {_render_counts(result['count_mismatches'])} " + "(bags are blind to position: a marker moved between clauses " + "is not detected here)") if budget is not None and not result["length_eligible"]: print(f" over length budget: +{budget['delta_words']} words " "(SCIPAPER_STANDARD section 5.3; use --allow-growth REASON " From 9b30c47b7e00aa41cd51c8ab2d2967712702821a Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:07:19 +0000 Subject: [PATCH 12/27] eval and labelling: a thin or empty stratum is unmeasured, and a cell is a draw Audit 2026-09-27 (docs/audits/full-review-2026-09-27.md), rows E5, E6, E7, E10, E11, E13, E14, E15, E18, E19, E28 and E33. eval_findings: a population of twenty documents with no body words was `measured` with a None density that `render` formatted with `.3f`; now `unmeasured` with the reason, and `_cell` prints a dash for a missing density. `median_words` is `statistics.median` (the upper middle element read 100/200/300/400 as 300). `MIN_DOCUMENTS` is imported from eval_docscale so both evaluators share one floor. eval_docscale: an empty human corpus reached `statistics.median([])` as a StatisticsError; now a SystemExit naming the empty corpus and the calibrate command. Every row is `unmeasured` under the shared `MIN_DOCUMENTS` floor (one document once printed a flag rate and an AUC), a flag is decided by the operating point's own alpha (a legacy in-sample-percentile point keeps its threshold decision instead of having its percentile read against the conformal alpha), each row names its operating point, tiers are discovered under docval/ the way eval_findings does, and the field resolves under this tool's name rather than through retrieve_exemplars' wrapper. label_findings: the field options sat on the root parser and every subparser, and argparse let the subparser defaults overwrite the root's parse, so `--field wgl --corpus-root X sample` ran with field=None silently; the root is `base_parser` only and that spelling is an error. `--n` is the per-(population, axis) quota with help text (the old formula made it a dead flag below 200 rows per population). Every finding of every document is collected first and each cell is drawn at random with a per-document cap (`_spread_sample`), the summary prints how many documents each cell came from, and the emitters run once per document for flags and controls alike. An emitter failure is counted per axis and printed with the shortfall line instead of reading as "the corpus holds no more". Pooled recall counts a passage several axes flagged once, by (source, start_line, end_line), and the printed line says so. `score --recheck ` exits naming the file. tests: test_label_findings.py's `unittest.main()` block moves to the end (it ran 6 of 16 tests when invoked directly). tests/test_eval_docscale.py is new; 33 tests added in all. The recorded suite size is for the integrator. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit 8927937994ff23fb82f751d19268e9a8582d08f1) --- tests/test_eval_docscale.py | 138 ++++++++++++++++++++ tests/test_eval_findings.py | 36 ++++++ tests/test_label_findings.py | 207 ++++++++++++++++++++++++++++- tools/eval_docscale.py | 120 +++++++++++++---- tools/eval_findings.py | 36 +++++- tools/label_findings.py | 244 +++++++++++++++++++++++++---------- 6 files changed, 676 insertions(+), 105 deletions(-) create mode 100644 tests/test_eval_docscale.py diff --git a/tests/test_eval_docscale.py b/tests/test_eval_docscale.py new file mode 100644 index 0000000..2635360 --- /dev/null +++ b/tests/test_eval_docscale.py @@ -0,0 +1,138 @@ +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +from _toolpath import TOOLS # noqa: F401 -- because importing it is what puts tools/ on sys.path + +import eval_docscale as ed + + +def _points(n: int, *, distance: float = 1.0, p_value: float = 0.5, + alpha: float | None = 0.05, + operating_point: str = "split-conformal", **extra) -> list[dict]: + """`n` scored documents as `manifold_operating_point` returns them.""" + return [{"distance": distance, "p_value": p_value, "alpha": alpha, + "operating_point": operating_point, "n_paragraphs": 30, **extra} + for _ in range(n)] + + +class RowFloorTests(unittest.TestCase): + """A row resting on too few documents must say so, not print a number. + + One document once printed a flag rate and an AUC of its own, and an + empty human list reached `statistics.median` as a StatisticsError. + """ + + def test_an_empty_row_is_unmeasured_not_a_statistics_error(self) -> None: + rows = ed.summarize({"human": [], "ai": []}) + self.assertEqual(rows["human"]["status"], "unmeasured") + self.assertEqual(rows["ai"]["why"], "no document scored") + self.assertNotIn("flag_rate", rows["ai"]) + + def test_below_the_floor_is_unmeasured(self) -> None: + rows = ed.summarize({"human": _points(ed.MIN_DOCUMENTS), + "ai": _points(ed.MIN_DOCUMENTS - 1)}) + self.assertEqual(rows["ai"]["status"], "unmeasured") + self.assertIn(f"< {ed.MIN_DOCUMENTS}", rows["ai"]["why"]) + self.assertEqual(rows["human"]["status"], "measured") + self.assertIn("unmeasured (n=19", ed.render(self._report(rows))) + + def test_a_thin_human_row_gives_no_auc(self) -> None: + rows = ed.summarize({"human": _points(3), + "ai": _points(ed.MIN_DOCUMENTS, distance=2.0)}) + self.assertEqual(rows["ai"]["status"], "measured") + self.assertIsNone(rows["ai"]["auc_vs_human"]) + + def test_the_floor_is_shared_with_eval_findings(self) -> None: + import eval_findings + self.assertEqual(ed.MIN_DOCUMENTS, eval_findings.MIN_DOCUMENTS) + + @staticmethod + def _report(rows: dict, alpha: float | None = 0.05) -> dict: + return {"field": "wgl", "alpha": alpha, "min_documents": ed.MIN_DOCUMENTS, + "baseline_documents": 3, "rows": rows} + + +class OperatingPointTests(unittest.TestCase): + """A flag is decided by the operating point that scored the document. + + Reading a legacy in-sample percentile against the baseline's conformal + alpha compared a within-sample percentile with a level it was never + calibrated to. + """ + + def test_a_conformal_point_is_flagged_at_its_own_alpha(self) -> None: + self.assertTrue(ed.flagged({"p_value": 0.04, "alpha": 0.05})) + self.assertFalse(ed.flagged({"p_value": 0.06, "alpha": 0.05})) + self.assertTrue(ed.flagged({"p_value": 0.10, "alpha": 0.10})) + + def test_a_legacy_percentile_point_keeps_its_own_decision(self) -> None: + self.assertFalse(ed.flagged({"p_value": 0.03, "alpha": None, "flagged": False})) + self.assertTrue(ed.flagged({"p_value": 0.50, "alpha": None, "flagged": True})) + + def test_each_row_names_its_operating_point(self) -> None: + rows = ed.summarize({ + "human": _points(ed.MIN_DOCUMENTS), + "ai": _points(ed.MIN_DOCUMENTS, distance=2.0, p_value=0.01, + operating_point="split-conformal, stratum manifold")}) + self.assertEqual(rows["ai"]["operating_point"], "split-conformal, stratum manifold") + self.assertEqual(rows["ai"]["flag_rate"], 1.0) + self.assertEqual(rows["human"]["flag_rate"], 0.0) + self.assertEqual(rows["ai"]["auc_vs_human"], 1.0) + rendered = ed.render(RowFloorTests._report(rows)) + self.assertIn("split-conformal, stratum manifold", rendered) + self.assertIn("alpha=0.05", rendered) + + def test_a_legacy_baseline_reports_no_alpha(self) -> None: + rows = ed.summarize({"human": _points( + ed.MIN_DOCUMENTS, alpha=None, operating_point="in-sample percentile", + flagged=False)}) + self.assertEqual(rows["human"]["operating_point"], "in-sample percentile") + self.assertIn("alpha=-", ed.render(RowFloorTests._report(rows, alpha=None))) + + +class ResolutionTests(unittest.TestCase): + def test_no_human_document_is_a_named_exit_not_a_statistics_error(self): + with tempfile.TemporaryDirectory() as raw: + root = Path(raw) + (root / "profile" / "wgl").mkdir(parents=True) + (root / "profile" / "wgl" / "docstructure_baseline.json").write_text( + json.dumps({"n_documents": 3}), encoding="utf-8") + (root / "corpus" / "wgl").mkdir(parents=True) + with self.assertRaises(SystemExit) as caught: + ed.build_report("wgl", root / "profile", root / "corpus") + message = str(caught.exception) + self.assertIn("no human document", message) + self.assertIn("fetch_arxiv_abstracts", message) + self.assertIn("--calibrate", message) + + def test_the_field_is_resolved_under_this_tools_own_name(self) -> None: + # The wrapper borrowed from retrieve_exemplars signed the error with + # that tool's name. + with tempfile.TemporaryDirectory() as raw: + with self.assertRaises(SystemExit) as caught: + ed.main(["--field", "nope", "--profile-root", raw]) + self.assertIn("[eval_docscale]", str(caught.exception)) + self.assertNotIn("retrieve_exemplars", str(caught.exception)) + + def test_tiers_are_discovered_from_docval_not_a_fixed_list(self) -> None: + with tempfile.TemporaryDirectory() as raw: + root = Path(raw) + for tier in ("ai", "ai_new_tier"): + (root / "profile" / "wgl" / "docval" / tier).mkdir(parents=True) + (root / "profile" / "wgl" / "docval" / tier / "x.tex").write_text( + "word " * 60, encoding="utf-8") + (root / "corpus" / "wgl").mkdir(parents=True) + # A baseline with no manifold scores nothing, so every tier is + # present and empty rather than absent. + scored = ed.collect({"n_documents": 3}, "wgl", root / "profile", + root / "corpus") + self.assertEqual(sorted(scored), ["ai", "ai_new_tier", "human"]) + self.assertEqual(scored["ai_new_tier"], []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_eval_findings.py b/tests/test_eval_findings.py index 5a46468..a7bec45 100644 --- a/tests/test_eval_findings.py +++ b/tests/test_eval_findings.py @@ -154,6 +154,36 @@ def test_at_the_floor_the_rate_is_reported(self) -> None: self.assertEqual(summary["flag_rate"], 1.0) self.assertEqual(summary["per_1k_words"], 2.0) + def test_a_population_without_body_words_is_unmeasured_not_a_density(self): + # Twenty documents whose body projection is empty: the cell had + # printed as measured with a None density, which `render` then + # formatted with `.3f`. + rows = self._rows(ef.MIN_DOCUMENTS, 1.0) + for row in rows: + row["n_words"] = 0.0 + summary = ef.summarize(rows, "L0.register") + self.assertEqual(summary["status"], "unmeasured") + self.assertIn("0 body words", summary["why"]) + self.assertNotIn("flag_rate", summary) + rendered = ef.render(ef.build_report("wgl", {"published-heldout": rows})) + self.assertIn("unmeasured", rendered) + + def test_median_words_is_the_statistical_median(self) -> None: + rows = self._rows(4, 1.0) + for row, words in zip(rows, (100.0, 200.0, 300.0, 400.0)): + row["n_words"] = words + report = ef.build_report("wgl", {"published-heldout": rows}) + # The upper middle element, 300, once stood in for it. + self.assertEqual(report["populations"]["published-heldout"]["median_words"], 250.0) + + def test_the_document_floor_has_one_owner(self) -> None: + # Shared with eval_docscale, so the two evaluators cannot disagree on + # how many documents a document-level rate needs. + import eval_docscale + self.assertEqual(ef.MIN_DOCUMENTS, eval_docscale.MIN_DOCUMENTS) + self.assertNotIn("MIN_DOCUMENTS =", + (TOOLS / "eval_findings.py").read_text(encoding="utf-8")) + def test_density_is_per_thousand_words_not_per_document(self) -> None: rows = [{"n_words": 2000.0, "L0.register": 4.0, "L2.salience_hierarchy": 0.0, "salience_strong": 0.0}] @@ -230,6 +260,12 @@ def test_unmeasured_cells_render_without_a_number(self) -> None: self.assertIn("unmeasured", rendered) self.assertNotIn("0.000", rendered) + def test_a_measured_cell_without_a_density_prints_a_dash_not_a_traceback(self): + rendered = ef._cell({"status": "measured", "flag_rate": 0.5, + "per_1k_words": None}) + self.assertIn("0.500", rendered) + self.assertIn("-", rendered) + class SalienceGateTransferTest(unittest.TestCase): """The gate is defined per passage, so the transfer test must be too. diff --git a/tests/test_label_findings.py b/tests/test_label_findings.py index 2ae26ce..1d5184b 100644 --- a/tests/test_label_findings.py +++ b/tests/test_label_findings.py @@ -1,9 +1,14 @@ from __future__ import annotations +import contextlib +import io import json +import random import tempfile import types import unittest +from collections import Counter +from contextlib import redirect_stderr, redirect_stdout from pathlib import Path from _toolpath import TOOLS # noqa: F401 -- because importing it is what puts tools/ on sys.path @@ -105,10 +110,6 @@ def test_relabel_strips_the_prior_answer_and_keeps_the_link(self): len(served)) -if __name__ == "__main__": - unittest.main() - - class AxisCoverageTests(unittest.TestCase): """Every axis that emits findings must be sampleable, under the name the report uses for it. @@ -219,6 +220,202 @@ def test_recall_is_reported_once_per_population_not_once_per_axis(self) -> None: labels.cmd_score(types.SimpleNamespace(sheet=sheet, recheck=None)) out = buffer.getvalue() self.assertEqual(out.count("recall"), 1, out) - # 40 true positives, 10 labelled misses -> 40/50, pooled. + # 40 true positives on 40 rows that carry no location, so each is + # its own passage; 10 labelled misses -> 40/50, pooled. self.assertIn("0.800", out) self.assertIn("pooled", out) + + def test_a_passage_several_axes_flag_is_one_true_positive(self) -> None: + # The control side counts passages a labeller says were missed, so the + # numerator counts passages too: two axes right about the same twenty + # passages are twenty hits, not forty, and recall is 20/30, not 40/50. + rows = [] + for axis in ("L0.register", "L2.salience_hierarchy"): + for i in range(labels.MIN_PER_CELL): + rows.append({"id": f"{axis}-{i}", "population": "p", "axis": axis, + "source": "x.tex", "flagged": True, "label": True, + "evidence": {"location": {"start_line": 3 + 2 * i, + "end_line": 3 + 2 * i}}}) + for i in range(labels.MIN_PER_CELL): + rows.append({"id": f"ctl-{i}", "population": "p", "axis": "control", + "flagged": False, "label": i < 10, "evidence": {}}) + with tempfile.TemporaryDirectory() as name: + sheet = Path(name) / "s.jsonl" + sheet.write_text("\n".join(json.dumps(r) for r in rows) + "\n", + encoding="utf-8") + buffer = io.StringIO() + with redirect_stdout(buffer): + labels.cmd_score(types.SimpleNamespace(sheet=sheet, recheck=None)) + out = buffer.getvalue() + self.assertIn("0.667", out) + self.assertNotIn("0.800", out) + self.assertIn("true positives counted once per passage: 40 rows on 20 passages", out) + self.assertIn("from 1 documents", out) + + def test_a_missing_recheck_sheet_is_named_not_read_as_unsupplied(self) -> None: + with tempfile.TemporaryDirectory() as name: + sheet = Path(name) / "s.jsonl" + self._sheet(sheet) + with self.assertRaises(SystemExit) as caught: + labels.cmd_score(types.SimpleNamespace( + sheet=sheet, recheck=Path(name) / "missing.jsonl")) + self.assertIn("missing.jsonl", str(caught.exception)) + + +class ParserTests(unittest.TestCase): + """The field options belong to the subcommands, and only to them. + + On the root parser as well, argparse let each subparser's defaults + overwrite what the root had parsed: `--field wgl --corpus-root X sample` + ran with field=None and the default corpus root, and nothing said so. + """ + + def test_field_after_the_subcommand_reaches_the_command(self) -> None: + seen: dict = {} + original = labels.cmd_sample + labels.cmd_sample = lambda args: seen.update(vars(args)) or 0 + try: + labels.main(["sample", "--field", "wgl", "--corpus-root", "corpus", + "--profile-root", "profile"]) + finally: + labels.cmd_sample = original + self.assertEqual(seen["field"], "wgl") + self.assertEqual(seen["corpus_root"], Path("corpus")) + self.assertEqual(seen["profile_root"], Path("profile")) + + def test_field_before_the_subcommand_is_an_error_not_a_silent_default(self): + with redirect_stderr(io.StringIO()), self.assertRaises(SystemExit) as caught: + labels.main(["--field", "wgl", "--corpus-root", "corpus", "sample"]) + self.assertEqual(caught.exception.code, 2) + + def test_n_is_documented_as_the_per_cell_quota(self) -> None: + buffer = io.StringIO() + with redirect_stdout(buffer), self.assertRaises(SystemExit) as caught: + labels.main(["sample", "--help"]) + self.assertEqual(caught.exception.code, 0) + self.assertIn("per (population, axis) cell", buffer.getvalue()) + self.assertIn(f"default {labels.MIN_PER_CELL}", buffer.getvalue()) + + +class SamplingTests(unittest.TestCase): + """A cell is a random draw spread over documents, not one paper's worth. + + Taking each document's findings in emission order until the quota filled + meant a 20-row cell usually came from one or two long papers, so the + precision it supported was per paper, not per finding; and an emitter that + failed on a document was read as "the corpus holds no more". + """ + + @staticmethod + def _document(n_paragraphs: int, word: str) -> str: + # Paragraph k sits alone on line 3 + 2k, so a finding can point at it. + return ("\\section{Methods}\n\n" + + "\n\n".join(" ".join([f"{word}{k}"] * 60) + for k in range(n_paragraphs)) + + "\n") + + @contextlib.contextmanager + def _harness(self, findings_per_document: dict[str, int], failing=()): + """A population of loose papers, a one-field profile, one fake emitter.""" + calls: list[str] = [] + + def emit(text, field_dir, path): + calls.append(path.name) + if path.name in failing: + raise RuntimeError("boom") + return [{"rule": "fake", "location": {"start_line": 3 + 2 * k, + "end_line": 3 + 2 * k}} + for k in range(findings_per_document[path.name])] + + original = labels.EMITTERS, labels.AXES + labels.EMITTERS = ((emit, ("L0.register",), None),) + labels.AXES = ("L0.register",) + try: + with tempfile.TemporaryDirectory() as raw: + root = Path(raw) + (root / "profile" / "wgl").mkdir(parents=True) + (root / "corpus").mkdir() + (root / "papers").mkdir() + for name, count in findings_per_document.items(): + # One paragraph more than the findings: a control candidate. + (root / "papers" / name).write_text( + self._document(count + 1, name[:3]), encoding="utf-8") + yield root, calls + finally: + labels.EMITTERS, labels.AXES = original + + @staticmethod + def _args(root: Path, n: int = labels.MIN_PER_CELL) -> types.SimpleNamespace: + return types.SimpleNamespace( + field="wgl", profile_root=root / "profile", corpus_root=root / "corpus", + population=[f"papers={root / 'papers'}"], drafts=None, + heldout_dir="fulltext-heldout", n=n, seed=1, out=root / "labels.jsonl") + + @staticmethod + def _rows(root: Path) -> list[dict]: + return [json.loads(line) for line in + (root / "labels.jsonl").read_text(encoding="utf-8").splitlines() + if line.strip()] + + def test_a_cell_is_spread_over_documents_not_filled_from_one(self) -> None: + # d00 alone could fill the cell; the draw caps any one document at a + # fifth of the quota and reaches the rest. + counts = {"d00.tex": 30, **{f"d{i:02d}.tex": 3 for i in range(1, 10)}} + with self._harness(counts) as (root, calls): + out = io.StringIO() + with redirect_stdout(out), redirect_stderr(io.StringIO()): + labels.cmd_sample(self._args(root)) + rows = self._rows(root) + flagged = [r for r in rows if r["flagged"]] + self.assertEqual(len(flagged), labels.MIN_PER_CELL) + per_document = Counter(r["source"] for r in flagged) + self.assertLessEqual(max(per_document.values()), 4) + self.assertGreaterEqual(len(per_document), labels.MIN_SOURCES_PER_CELL) + # Ten documents hold one unflagged paragraph each: ten controls, spread. + controls = [r for r in rows if not r["flagged"]] + self.assertEqual(len(controls), 10) + self.assertEqual(len({r["source"] for r in controls}), 10) + self.assertRegex(out.getvalue(), r"L0\.register\s+20 rows from \d+ documents") + self.assertRegex(out.getvalue(), r"control\s+10 rows from 10 documents") + # The emitters ran once per document, for the flags and the controls alike. + self.assertEqual(sorted(calls), sorted(counts)) + + def test_an_emitter_failure_is_reported_as_one_not_as_an_exhausted_corpus(self): + counts = {f"d{i:02d}.tex": 2 for i in range(4)} + with self._harness(counts, failing={"d02.tex", "d03.tex"}) as (root, _calls): + err = io.StringIO() + with redirect_stdout(io.StringIO()), redirect_stderr(err): + labels.cmd_sample(self._args(root)) + self.assertIn("4 of 20 needed (2 of 4 documents hold no more; " + "the emitter failed on the other 2)", err.getvalue()) + self.assertIn("emitter error -- papers x L0.register: emitter failed on " + "2 of 4 documents; first: d02.tex: boom", err.getvalue()) + + def test_n_is_the_per_cell_quota_and_a_value_under_the_floor_warns(self): + counts = {f"d{i:02d}.tex": 6 for i in range(6)} + with self._harness(counts) as (root, _calls): + err = io.StringIO() + with redirect_stdout(io.StringIO()), redirect_stderr(err): + labels.cmd_sample(self._args(root, n=25)) + self.assertEqual(sum(1 for r in self._rows(root) if r["flagged"]), 25) + self.assertNotIn("below the", err.getvalue()) + err = io.StringIO() + with redirect_stdout(io.StringIO()), redirect_stderr(err): + labels.cmd_sample(self._args(root, n=5)) + self.assertEqual(sum(1 for r in self._rows(root) if r["flagged"]), 5) + self.assertIn(f"below the {labels.MIN_PER_CELL}-label floor", err.getvalue()) + + def test_the_spread_fills_the_quota_when_the_cap_alone_cannot(self) -> None: + rng = random.Random(0) + items = [("a", i) for i in range(10)] + [("b", i) for i in range(10)] + capped = labels._spread_sample(rng, items, 4, 2) + self.assertEqual(Counter(name for name, _ in capped), {"a": 2, "b": 2}) + # Two documents at two each is four; the quota of eight is still met, + # from what remains, and nothing is drawn twice. + filled = labels._spread_sample(rng, items, 8, 2) + self.assertEqual(len(filled), 8) + self.assertEqual(len(set(filled)), 8) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/eval_docscale.py b/tools/eval_docscale.py index dc7b1f8..6e40399 100644 --- a/tools/eval_docscale.py +++ b/tools/eval_docscale.py @@ -12,6 +12,13 @@ exists to compare configurations evaluated identically; the distribution-free guarantee is the split-conformal alpha, not this rate. +A row under `MIN_DOCUMENTS` scored documents reports `unmeasured`, never a +rate: one document once printed a flag rate and an AUC of its own. Each row +names the operating point its flags were decided by, because a baseline can +carry either a split-conformal calibration (flag: p <= its own alpha) or the +older in-sample percentile threshold (flag: distance above it, alpha none), +and the two are not read against the same level. + Run: python tools/eval_docscale.py --field wgl [--format json] """ @@ -26,9 +33,11 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first import deai_docshape as ds # noqa: E402 -- because of that same sys.path insert import extract_sections as es # noqa: E402 -- because of that same sys.path insert -from retrieve_exemplars import resolve_field # noqa: E402 -- same reason -TIERS = ("ai", "ai_natural", "ai_deai", "ai_adversarial", "ai_skeleton", "ai_long") +# Below this many scored documents a row is `unmeasured`, never a rate. +# `eval_findings` imports this same floor, so the two evaluators cannot +# disagree on how many documents a document-level rate needs. +MIN_DOCUMENTS = 20 def dispersion_row(shape: dict) -> dict[str, float]: @@ -73,42 +82,80 @@ def rank_auc(human: list[float], other: list[float]) -> float | None: def collect(baseline: dict, field: str, profile_root: Path, corpus_root: Path) -> dict[str, list[dict]]: - """Score the human corpus and every docval tier present.""" + """Score the human corpus and every docval tier present. + + The tiers are whatever `docval/` holds, discovered the way `eval_findings` + discovers them: a fixed tuple of tier names left a tier added later out of + the table with nothing to say so. A tier directory that scores nothing + stays in the result as an empty list, so the table prints it as + `unmeasured` rather than omitting it. + """ scored: dict[str, list[dict]] = {"human": []} for _name, text in es.corpus_documents(corpus_root / field): point = score_document(baseline, text) if point: scored["human"].append(point) - for tier in TIERS: - directory = profile_root / field / "docval" / tier - if not directory.is_dir(): - continue + docval = profile_root / field / "docval" + tiers = (sorted(p for p in docval.iterdir() if p.is_dir()) + if docval.is_dir() else []) + for directory in tiers: points = [] for path in sorted(directory.glob("*.tex")): point = score_document( baseline, path.read_text(encoding="utf-8", errors="replace")) if point: points.append(point) - if points: - scored[tier] = points + scored[directory.name] = points return scored -def summarize(scored: dict[str, list[dict]], alpha: float) -> dict: +def flagged(point: dict) -> bool: + """Whether the operating point that scored a document flags it. + + A split-conformal point carries its own `alpha`, and the flag is + `p_value <= alpha`. A legacy in-sample percentile point carries + `alpha=None`, a `p_value` that is a within-sample percentile, and the + `flagged` decision its threshold already made; reading that percentile + against the baseline's conformal alpha compared it with a level it was + never calibrated to. + """ + alpha = point.get("alpha") + if alpha is not None and point.get("p_value") is not None: + return point["p_value"] <= alpha + return bool(point.get("flagged", False)) + + +def operating_point(points: list[dict]) -> str: + """The rule(s) a row's flags were decided by, as the points record them.""" + return ", ".join(sorted({str(point.get("operating_point")) for point in points})) + + +def summarize(scored: dict[str, list[dict]]) -> dict: + """One row per population: the numbers at the floor, `unmeasured` below it. + + The AUC against the human row needs both sides at the floor, so a thin + human row leaves every AUC `None` rather than ranking against a handful. + """ human = [point["distance"] for point in scored.get("human", [])] rows = {} for label, points in scored.items(): + if len(points) < MIN_DOCUMENTS: + rows[label] = {"status": "unmeasured", "n": len(points), + "why": (f"n={len(points)} < {MIN_DOCUMENTS}" if points + else "no document scored")} + continue distances = [point["distance"] for point in points] - flagged = sum(1 for point in points - if point.get("p_value") is not None - and point["p_value"] <= alpha) - auc = None if label == "human" else rank_auc(human, distances) + auc = (rank_auc(human, distances) + if label != "human" and len(human) >= MIN_DOCUMENTS else None) rows[label] = { + "status": "measured", "n": len(points), + "operating_point": operating_point(points), "median_distance": round(statistics.median(distances), 4), "median_paragraphs": statistics.median( [point["n_paragraphs"] for point in points]), - "flag_rate": round(flagged / len(points), 4), + "flag_rate": round( + sum(1 for point in points if flagged(point)) / len(points), 4), "auc_vs_human": None if auc is None else round(auc, 3), } return rows @@ -117,16 +164,23 @@ def summarize(scored: dict[str, list[dict]], alpha: float) -> dict: def render(report: dict) -> str: rows = report["rows"] width = max(len(key) for key in rows) - out = [f"[eval_docscale] field={report['field']!r} alpha={report['alpha']} " - f"baseline={report['baseline_documents']} documents", + alpha = "-" if report["alpha"] is None else report["alpha"] + out = [f"[eval_docscale] field={report['field']!r} alpha={alpha} " + f"baseline={report['baseline_documents']} documents " + f"(floor {report['min_documents']} documents per row)", f"{'tier'.ljust(width)} {'n':>4} {'med d':>7} {'med par':>7}" - f" {'flag':>6} {'AUC':>5}"] + f" {'flag':>6} {'AUC':>5} operating point"] for label, row in rows.items(): + if row["status"] != "measured": + out.append(f"{label.ljust(width)} {row['n']:>4} " + f"unmeasured ({row['why']})") + continue auc = (" - " if row["auc_vs_human"] is None else f"{row['auc_vs_human']:.3f}") out.append( f"{label.ljust(width)} {row['n']:>4} {row['median_distance']:>7.3f}" - f" {row['median_paragraphs']:>7.1f} {row['flag_rate']:>6.4f} {auc:>5}") + f" {row['median_paragraphs']:>7.1f} {row['flag_rate']:>6.4f} {auc:>5}" + f" {row['operating_point']}") out += ["", "The human row is in-sample: it includes the manifold's own " "train and calibration documents."] return "\n".join(out) @@ -140,15 +194,30 @@ def build_report(field: str, profile_root: Path, corpus_root: Path) -> dict: f"`python tools/deai_docstructure.py --calibrate --field {field} " f"--corpus-dir style-corpus/{field}` first.") baseline = json.loads(baseline_path.read_text(encoding="utf-8")) - alpha = float((baseline.get("conformal") or {}).get("alpha", 0.05)) scored = collect(baseline, field, profile_root, corpus_root) + if not scored["human"]: + # `statistics.median([])` raised a StatisticsError here, naming + # neither the empty corpus nor the manifold that scored nothing. + raise SystemExit( + f"[eval_docscale] no human document under {corpus_root / field} " + f"scored, so there is no row to compare the tiers against. Either " + f"the corpus holds no complete documents -- fetch some with `python " + f"tools/fetch_arxiv_abstracts.py --field {field} --fulltext` -- or " + f"the baseline scores none of them: rebuild it with `python " + f"tools/deai_docstructure.py --calibrate --field {field} " + f"--corpus-dir style-corpus/{field}`.") + # The level every flag in this run was decided at: the points carry it, + # and a legacy percentile baseline carries none. + alphas = {point.get("alpha") + for points in scored.values() for point in points} - {None} return { "schema": "sci-paper.docscale-eval.v1", "field": field, - "alpha": alpha, + "alpha": alphas.pop() if len(alphas) == 1 else None, + "min_documents": MIN_DOCUMENTS, "baseline_documents": baseline.get("n_documents"), "human_rate_is_in_sample": True, - "rows": summarize(scored, alpha), + "rows": summarize(scored), } @@ -158,8 +227,11 @@ def main(argv: list[str] | None = None) -> int: cli_common.field_parser(__doc__, corpus=True)) args = parser.parse_args(argv) - report = build_report(resolve_field(args.field, args.profile_root), - args.profile_root, args.corpus_root) + # Resolved under this tool's own name: the wrapper borrowed from + # `retrieve_exemplars` signed its errors `[retrieve_exemplars]`. + field = cli_common.resolve_field(args.field, args.profile_root, + tool="eval_docscale") + report = build_report(field, args.profile_root, args.corpus_root) return cli_common.emit_report(report, args, render=render, tool="eval_docscale") diff --git a/tools/eval_findings.py b/tools/eval_findings.py index 12df379..6395fb0 100644 --- a/tools/eval_findings.py +++ b/tools/eval_findings.py @@ -56,6 +56,7 @@ from __future__ import annotations +import statistics import sys from pathlib import Path @@ -65,12 +66,13 @@ import deai_register # noqa: E402 -- because of that same sys.path insert import deai_salience # noqa: E402 -- because of that same sys.path insert import extract_sections as es # noqa: E402 -- because of that same sys.path insert -from eval_docscale import rank_auc # noqa: E402 -- one AUC implementation, same reason +# One AUC implementation and one document floor for both evaluators, so a rate +# that is `unmeasured` here at n < 20 cannot be a rate there. +from eval_docscale import MIN_DOCUMENTS, rank_auc # noqa: E402 -- same reason SCHEMA = "sci-paper.findings-eval.v1" HELDOUT_DIR = "fulltext-heldout" INSAMPLE_DIR = "fulltext-arxiv" -MIN_DOCUMENTS = 20 # below this a rate is reported `unmeasured`, never as 0/0 # Deliberately FOUR, where `label_findings.AXES` carries more. This file is not # a generic axis loop: each axis needs a reading of its own -- register is an # absolute rarity test whose held-out rate IS a false-positive rate, the @@ -150,18 +152,29 @@ def score_document(text: str, field_dir: Path, path: str) -> dict[str, float]: def summarize(rows: list[dict[str, float]], axis: str) -> dict: - """Document flag rate and length-normalized density for one axis.""" + """Document flag rate and length-normalized density for one axis. + + Two floors, and both are `unmeasured` rather than a number: fewer than + `MIN_DOCUMENTS` documents, and a population whose documents carry no body + prose at all. `n_words` is the register projection of the body, so a set + of sources with nothing between `\\begin{document}` and `\\end{document}` + counts twenty documents and zero words; a density over zero words is not a + rate, and the None it once produced reached `render` as a `.3f` format. + """ if len(rows) < MIN_DOCUMENTS: return {"status": "unmeasured", "n": len(rows), "why": f"n < {MIN_DOCUMENTS}"} - flagged = sum(1 for r in rows if r[axis] > 0) words = sum(r["n_words"] for r in rows) + if not words: + return {"status": "unmeasured", "n": len(rows), + "why": f"0 body words over {len(rows)} documents"} + flagged = sum(1 for r in rows if r[axis] > 0) findings = sum(r[axis] for r in rows) return { "status": "measured", "n": len(rows), "flag_rate": round(flagged / len(rows), 4), - "per_1k_words": round(1000.0 * findings / words, 4) if words else None, + "per_1k_words": round(1000.0 * findings / words, 4), "total_findings": int(findings), } @@ -290,7 +303,9 @@ def build_report(field: str, populations: dict[str, list[dict[str, float]]], rows_out = { label: { "n_documents": len(rows), - "median_words": sorted(r["n_words"] for r in rows)[len(rows) // 2], + # `statistics.median`: the upper middle element once stood in for + # it, so 100/200/300/400 words read as 300. + "median_words": statistics.median(r["n_words"] for r in rows), **{axis: summarize(rows, axis) for axis in AXES}, } for label, rows in populations.items() @@ -338,9 +353,16 @@ def build_report(field: str, populations: dict[str, list[dict[str, float]]], def _cell(entry: dict) -> str: + """One axis cell: two numbers, or `unmeasured` and a dash. + + A measured cell always carries a density now, but a missing one prints as + a dash rather than reaching `.3f` as None and taking the table down. + """ if entry.get("status") != "measured": return f"{'unmeasured':>10s} {'-':>9s}" - return f"{entry['flag_rate']:>10.3f} {entry['per_1k_words']:>9.3f}" + density = entry.get("per_1k_words") + shown = f"{density:>9.3f}" if density is not None else f"{'-':>9s}" + return f"{entry['flag_rate']:>10.3f} {shown}" def render(report: dict) -> str: diff --git a/tools/label_findings.py b/tools/label_findings.py index ec381a1..0b8eb60 100644 --- a/tools/label_findings.py +++ b/tools/label_findings.py @@ -27,6 +27,11 @@ with different answers. Which populations are worth labelling is the user's call, not this file's: `--drafts` is the special case that names one `draft`, and the held-out published set is added automatically when it exists. +- **spread over documents within a cell.** Every finding of every document is + collected first and each (population, axis) cell is then drawn at random + with a per-document cap, so twenty rows are twenty advisories rather than + one long paper's worth; the summary prints how many documents each cell + came from, and `score` counts a passage several axes flagged once. Recall is reported POOLED over the axes, not per axis, and that is a property of the data rather than a shortcut. A control row asks "should this passage have @@ -41,7 +46,7 @@ Run: python tools/label_findings.py sample --field wgl \\ --population mentor=style-corpus/wgl/fulltext-mentor \\ - --n 60 --out labels.jsonl + --n 20 --out labels.jsonl python tools/label_findings.py relabel --sheet labels.jsonl --frac 0.2 \\ --out recheck.jsonl python tools/label_findings.py score --sheet labels.jsonl [--recheck recheck.jsonl] @@ -88,6 +93,10 @@ ) AXES = tuple(axis for _, axes, _ in EMITTERS for axis in axes) MIN_PER_CELL = 20 # below this a rate is reported `unmeasured`, never as 0/0 +# A cell is drawn from at least this many documents when the population holds +# them: a rate over rows that all come from one long paper is a rate on that +# paper, whatever the row count says. +MIN_SOURCES_PER_CELL = 5 def _passages(text: str) -> list[tuple[str, str, int, int]]: @@ -202,6 +211,33 @@ def _findings_for(name: str, text: str, field_dir: Path) -> list[dict]: return found +def _spread_sample(rng: random.Random, items: list[tuple[str, object]], + quota: int, cap: int) -> list[tuple[str, object]]: + """Up to `quota` of `items` at random, at most `cap` from any one document. + + `items` are (document, payload) pairs. The cap applies first; if it + starves the quota -- fewer documents than the spread wants -- the rest is + filled from whatever remains, because a cell under the floor wastes every + label spent on it, and the summary prints how many documents the cell came + from so a cell filled from two papers reads as one. + """ + order = list(range(len(items))) + rng.shuffle(order) + kept: list[int] = [] + per_document: dict[str, int] = {} + for index in order: + if len(kept) >= quota: + break + document = items[index][0] + if per_document.get(document, 0) < cap: + kept.append(index) + per_document[document] = per_document.get(document, 0) + 1 + if len(kept) < quota: + chosen = set(kept) + kept += [index for index in order if index not in chosen][:quota - len(kept)] + return [items[index] for index in kept] + + def cmd_sample(args) -> int: field = cli_common.resolve_field(args.field, args.profile_root, tool="label_findings") @@ -236,93 +272,130 @@ def cmd_sample(args) -> int: f"and/or place a held-out set at " f"{args.corpus_root / field / args.heldout_dir}/.") - rows = [] - live = [key for key, docs in populations.items() if docs] - # Stratify by AXIS as well as by population. Salience fires roughly twenty - # times as often as register -- 0.0858 register findings per 1,000 words - # (EVALUATION section 18.4) against a per-passage p90 gate -- so a quota the - # two share is spent on salience long before register reaches the floor, and - # the register cell then reports `unmeasured` however large the sheet is. - quota = max(MIN_PER_CELL, args.n // (2 * max(1, len(live)) * len(AXES))) + # The quota is per (population, AXIS) cell -- `--n`, default the floor. + # Salience fires roughly twenty times as often as register -- 0.0858 + # register findings per 1,000 words (EVALUATION section 18.4) against a + # per-passage p90 gate -- so a quota the two shared was spent on salience + # long before register reached the floor, and the register cell reported + # `unmeasured` however large the sheet was. + if args.n < MIN_PER_CELL: + print(f"[label_findings] --n {args.n} is below the {MIN_PER_CELL}-label " + f"floor: every cell of this sheet will score `unmeasured`", + file=sys.stderr) + quota = args.n + rows: list[dict] = [] + # (population, axis) -> (rows drawn, distinct documents they came from). + cells: dict[tuple[str, str], tuple[int, int]] = {} short: list[str] = [] + failures: list[str] = [] for population, documents in populations.items(): if not documents: print(f"[label_findings] population {population!r} is empty; skipped", file=sys.stderr) continue - rng.shuffle(documents) - taken = {axis: 0 for axis in AXES} + # Every finding of every document first, then the draw. Taking each + # document's findings in emission order until the quota filled meant + # a 20-row cell usually came from one or two long papers, and the + # precision it supported was per paper, not per finding. A document's + # full finding set also serves the control pass below, so the emitters + # run once per document rather than twice. + candidates: dict[str, list[tuple[str, object]]] = {axis: [] for axis in AXES} + errors: dict[str, list[str]] = {axis: [] for axis in AXES} + spans: dict[str, list[tuple[int, int]]] = {} for name, text in documents: - if all(count >= quota for count in taken.values()): - break - for entry in _findings_for(name, text, field_dir): + entries = _findings_for(name, text, field_dir) + spans[name] = flagged_spans(entries) + for entry in entries: axis = entry.get("axis") - if "error" in entry or taken.get(axis, quota) >= quota: + if axis not in candidates: continue - taken[axis] += 1 + if "error" in entry: + errors[axis].append(f"{name}: {entry['error']}") + else: + candidates[axis].append((name, (text, entry["finding"]))) + for axis in AXES: + drawn = _spread_sample(rng, candidates[axis], quota, + -(-quota // MIN_SOURCES_PER_CELL)) + for name, (text, finding) in drawn: rows.append({ "id": f"{population}-{len(rows):04d}", "population": population, "axis": axis, "source": name, "flagged": True, - "evidence": entry["finding"], - "excerpt": _excerpt(text, entry["finding"]), + "evidence": finding, + "excerpt": _excerpt(text, finding), "question": FLAG_QUESTION, "label": None, # <- you fill this in: true | false "note": "", }) - # A cell the population cannot fill is stated here rather than - # discovered after the labelling is done. - short += [f"{population} x {axis}: {count} of {MIN_PER_CELL} needed " - f"({len(documents)} documents hold no more)" - for axis, count in taken.items() if count < MIN_PER_CELL] - # Unflagged controls make recall computable: a sheet of flags alone can only - # ever report precision. Target roughly as many controls as flags in each - # population, so neither side of the ratio is the one that starves. - for population, documents in populations.items(): - flagged_here = sum(1 for r in rows - if r["population"] == population and r["flagged"]) + cells[(population, axis)] = (len(drawn), len({name for name, _ in drawn})) + # A cell the population cannot fill is stated here rather than + # discovered after the labelling is done -- and an emitter that + # failed on some documents is stated as that, not folded into + # "the corpus holds no more". + failed = errors[axis] + if len(drawn) < MIN_PER_CELL: + short.append( + f"{population} x {axis}: {len(drawn)} of {MIN_PER_CELL} needed " + f"({len(documents) - len(failed)} of {len(documents)} documents " + f"hold no more" + + (f"; the emitter failed on the other {len(failed)}" if failed else "") + + ")") + if failed: + failures.append(f"{population} x {axis}: emitter failed on " + f"{len(failed)} of {len(documents)} documents; " + f"first: {failed[0]}") + # Unflagged controls make recall computable: a sheet of flags alone + # can only ever report precision. Target roughly as many controls as + # flags in this population, so neither side of the ratio is the one + # that starves -- drawn with the same spread, for the same reason. + flagged_here = sum(count for (owner, _axis), (count, _documents) + in cells.items() if owner == population) want = max(MIN_PER_CELL, flagged_here) - made = 0 - for name, text in documents: - if made >= want: - break - spans = flagged_spans(_findings_for(name, text, field_dir)) - for bucket, passage, start, end in unflagged_passages(_passages(text), spans): - if made >= want: - break - rows.append({ - "id": f"{population}-ctl-{len(rows):04d}", - "population": population, - "axis": "control", - "source": name, - "flagged": False, - "evidence": {"section": bucket, "text": passage[:1200]}, - "question": CONTROL_QUESTION, - "label": None, - "note": "", - }) - made += 1 + pool: list[tuple[str, object]] = [ + (name, (bucket, passage)) + for name, text in documents + for bucket, passage, _start, _end + in unflagged_passages(_passages(text), spans[name])] + drawn = _spread_sample(rng, pool, want, -(-want // MIN_SOURCES_PER_CELL)) + for name, (bucket, passage) in drawn: + rows.append({ + "id": f"{population}-ctl-{len(rows):04d}", + "population": population, + "axis": "control", + "source": name, + "flagged": False, + "evidence": {"section": bucket, "text": passage[:1200]}, + "question": CONTROL_QUESTION, + "label": None, + "note": "", + }) + cells[(population, "control")] = (len(drawn), len({name for name, _ in drawn})) rng.shuffle(rows) args.out.write_text( "\n".join(json.dumps({"schema": SCHEMA, **row}, ensure_ascii=False) for row in rows) + "\n", encoding="utf-8") - by_population: dict[str, int] = {} - for row in rows: - by_population[row["population"]] = by_population.get(row["population"], 0) + 1 print(f"[label_findings] {len(rows)} rows -> {args.out}") - for population, count in sorted(by_population.items()): - flagged = sum(1 for r in rows - if r["population"] == population and r["flagged"]) - print(f" {population:10s} {count:4d} rows ({flagged} flagged, " - f"{count - flagged} controls)") + for population in populations: + here = [r for r in rows if r["population"] == population] + if not here: + continue + flagged = sum(1 for r in here if r["flagged"]) + print(f" {population:10s} {len(here):4d} rows ({flagged} flagged, " + f"{len(here) - flagged} controls)") + for (owner, axis), (count, documents_used) in cells.items(): + if owner == population: + print(f" {axis:24s} {count:4d} rows from " + f"{documents_used} documents") print(" Each row carries its own `question`. A flagged row asks whether " "the advisory is right;\n a control row asks whether it SHOULD " "have been flagged, which is the miss recall counts.") for line in short: print(f" unmeasurable cell -- {line}", file=sys.stderr) + for line in failures: + print(f" emitter error -- {line}", file=sys.stderr) return 0 @@ -351,7 +424,25 @@ def _rate(hits: int, total: int) -> str: return f"{hits / total:.3f} (n={total})" +def _passage_key(row: dict) -> tuple: + """What makes two true positives one hit: the passage, not the row. + + A passage three axes flagged is three rows, each judged on its own + advisory, but it is ONE passage for recall -- the control side counts + passages a labeller says were missed, so the numerator must count passages + too, or three axes agreeing inflates recall three-fold. A row without a + location (an older sheet) keys on its own id, which merges nothing. + """ + location = (row.get("evidence") or {}).get("location") or {} + if location.get("start_line"): + return ("passage", row.get("source"), location["start_line"], + location.get("end_line") or location["start_line"]) + return ("row", row.get("id")) + + def cmd_score(args) -> int: + if args.recheck and not args.recheck.exists(): + raise SystemExit(f"[label_findings] --recheck sheet not found: {args.recheck}") rows = [json.loads(line) for line in args.sheet.read_text(encoding="utf-8").splitlines() if line.strip()] labelled = [r for r in rows if r.get("label") is not None] @@ -363,23 +454,30 @@ def cmd_score(args) -> int: for population in populations: here = [r for r in labelled if r["population"] == population] print(f"population: {population}") - caught = 0 + true_rows = 0 + caught: set[tuple] = set() for axis in AXES: flagged = [r for r in here if r["axis"] == axis] - true_positive = sum(1 for r in flagged if r["label"] is True) - caught += true_positive - print(f" {axis:24s} precision {_rate(true_positive, len(flagged))}") + true_positive = [r for r in flagged if r["label"] is True] + true_rows += len(true_positive) + caught.update(_passage_key(r) for r in true_positive) + sources = {r.get("source") for r in flagged} - {None} + print(f" {axis:24s} precision {_rate(len(true_positive), len(flagged))}" + + (f" from {len(sources)} documents" if sources else "")) # Recall is POOLED over the axes, and can only be. A control row asks # whether the passage should have been flagged; a labeller who says yes # has not said by WHICH axis, so no per-axis miss count exists to divide # by. Printing one anyway -- as this did while it carried two axes -- - # charges every axis with every other axis's misses. + # charges every axis with every other axis's misses. Its numerator is + # passages, deduplicated across the axes, because its denominator is. missed = sum(1 for r in here if not r["flagged"] and r["label"] is True) print(f" {'(all axes pooled)':24s} recall " - f"{_rate(caught, caught + missed)}") + f"{_rate(len(caught), len(caught) + missed)} " + f"true positives counted once per passage: {true_rows} rows on " + f"{len(caught)} passages") print() - if args.recheck and args.recheck.exists(): + if args.recheck: second = {r["_relabel_of"]: r["label"] for r in (json.loads(line) for line in args.recheck.read_text(encoding="utf-8").splitlines() if line.strip()) @@ -407,10 +505,14 @@ def cmd_score(args) -> int: def main(argv: list[str] | None = None) -> int: cli_common.utf8_stdout() - parser = cli_common.field_parser(__doc__, corpus=True) - # The field options go on every subparser too: argparse routes everything - # after the subcommand name to the subparser, so a root-only --field would - # force `label_findings --field wgl sample`, unlike every other tool here. + # The field options live on the subparsers, and only there: argparse + # routes everything after the subcommand name to the subparser, so + # `--field` goes last, as in every other tool here. They sat on the root + # as well, and argparse let each subparser's defaults overwrite what the + # root had parsed -- `label_findings --field wgl --corpus-root X sample` + # ran with field=None and the default corpus root, silently. That spelling + # is now an argparse error. + parser = cli_common.base_parser(__doc__) shared = [cli_common.field_options(corpus=True)] sub = parser.add_subparsers(dest="command", required=True) @@ -422,7 +524,11 @@ def main(argv: list[str] | None = None) -> int: "automatic held-out population.") s.add_argument("--drafts", type=Path, default=None, help="shorthand for --population draft=DIR") - s.add_argument("--n", type=int, default=60) + s.add_argument("--n", type=int, default=MIN_PER_CELL, + help="flagged rows to draw per (population, axis) cell " + "(default %(default)s, the floor below which a cell " + "scores unmeasured); each population also gets about as " + "many unflagged control rows as flagged ones") s.add_argument("--heldout-dir", default=eval_findings.HELDOUT_DIR, help="directory under style-corpus// holding the " "held-out papers the calibration never saw") From e47d25022b9f3bb4334ae50f717ab94075fecdf9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:07:19 +0000 Subject: [PATCH 13/27] voice training: a failed encode never becomes the cache, and the check that missed it is scope-aware Audit 2026-09-27 (docs/audits/full-review-2026-09-27.md), rows E8, E16, E21, E22, E31 and M5. voice_dataset: `embedder_available()` probes the import, and that is what the feature-cache fingerprint records; when the encode itself failed (offline download, device fault) the rows carried corpus_cos = 0.0 and were cached under the healthy fingerprint, then served as measured to every later run. `embeddings_failed_at_run_time` names that state; the cache is withheld and the finished language-model rows are written to the resumable checkpoint with n_done = every row, so the next run retries the embedder without refeaturizing, and a cache already holding empty embeddings under a healthy fingerprint is recomputed rather than reused. `_load_jsonl` skips blank lines (json.loads("") took the run down; the legacy loader tolerated them). voice_audit: the repeated grouped audit started at the primary split's own seed, so twenty repeats were nineteen plus the split already reported; `audit_split_seed` starts one past the seed the primary actually used, which the trainer now passes. train_voice_model: the field resolves through cli_common.resolve_field (`profile_root / None` was a TypeError traceback), and a missing numpy / scikit-learn / joblib is one stderr line and exit 2 instead of a traceback. E22 residue removed: hashlib, math, re, statistics, defaultdict, argparse and the DEFAULT_PROFILE_ROOT copies in train_voice_model; hashlib, json, re, Counter in voice_audit; the function-local `import re` and the DEFAULT_PROFILE_ROOT copy in voice_dataset; argparse, DEFAULT_PROFILE_ROOT and the never-called `list_fields` in train_ai_ism_classifier. The re-export contract gains the two new names. tests: the unbound-name check pooled every binding in the module, so a function-local `import numpy as np` anywhere let a bare `np` pass in every other function; it now reads CPython's symbol tables per scope, with fixtures for the leaked-import case, the original time/CHECKPOINT_EVERY case, closures, `global` bindings and a class attribute read from a method. A numpy stand-in drives `build_features` through the failed encode, the resumed run, and the stale cache. An unused-import check covers the four files. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit 9ede3c3a390631406e10b8c75ad8a8f83f855434) --- tests/test_train_voice_model.py | 369 +++++++++++++++++++++++++++++-- tools/train_ai_ism_classifier.py | 8 - tools/train_voice_model.py | 58 +++-- tools/voice_audit.py | 23 +- tools/voice_dataset.py | 70 ++++-- 5 files changed, 454 insertions(+), 74 deletions(-) diff --git a/tests/test_train_voice_model.py b/tests/test_train_voice_model.py index 27ae12e..b5063d5 100644 --- a/tests/test_train_voice_model.py +++ b/tests/test_train_voice_model.py @@ -1,8 +1,12 @@ from __future__ import annotations +import contextlib import copy +import io +import json import sys import tempfile +import types import unittest from pathlib import Path @@ -13,6 +17,54 @@ import train_voice_model as training +def _unbound_names(source: str, filename: str = "") -> list[str]: + """`scope:name` for every name read where no binding reaches it. + + Scope-aware, on CPython's own symbol tables rather than a flat walk of the + AST. The flat walk pooled every binding in the module, so a function-local + `import numpy as np` in one function let a bare `np` in every other + function pass, and the `time` it was written to catch went uncaught the + moment any other scope bound the name. Here a function-local import binds + the name only inside that function; a name a function reads without + binding it must be bound at module level (including through a `global` + declaration elsewhere), be a builtin, or be a closure variable of an + enclosing function. Annotation and type-parameter blocks (3.12+) are + skipped: every tool imports `annotations` from `__future__`, so those + names are never evaluated. + """ + import builtins + import symtable + + module = symtable.symtable(source, filename, "exec") + known = set(dir(builtins)) | { + "__file__", "__name__", "__doc__", "__package__", "__spec__", + "__loader__", "__builtins__", "__module__", "__qualname__"} + + def tables(table): + yield table + for child in table.get_children(): + try: + kind = child.get_type() + except AssertionError: # a 3.12 annotation or type-parameter block + continue + if kind in ("function", "class"): + yield from tables(child) + + scopes = list(tables(module)) + for table in scopes: + for symbol in table.get_symbols(): + bound_here = (symbol.is_assigned() or symbol.is_imported() + or symbol.is_namespace()) + if bound_here and (table is module or symbol.is_declared_global()): + known.add(symbol.get_name()) + return sorted( + f"{table.get_name()}:{symbol.get_name()}" + for table in scopes for symbol in table.get_symbols() + if symbol.is_referenced() + and not (symbol.is_local() or symbol.is_free() or symbol.is_parameter()) + and symbol.get_name() not in known) + + class ModuleSplitContractTests(unittest.TestCase): """`train_voice_model` must re-export everything the split modules define. @@ -64,39 +116,82 @@ def test_the_dependency_direction_is_one_way(self): source = (TOOLS / "voice_dataset.py").read_text(encoding="utf-8") self.assertNotIn("voice_audit", source) - def test_no_module_level_name_is_unbound(self): - """Every name a tool module loads must be defined or imported in it. + def test_no_name_is_read_where_nothing_binds_it(self): + """Every name a tool reads must be bound in a scope that reaches it. Splitting a module moves function bodies but re-writes the import block by hand, so a moved function can reference a name that stayed behind. That is not a hypothetical: the 2026-08-26 split left `time`, `CHECKPOINT_EVERY`, `df` and the two HARDSET category sets unbound, and the suite went green anyway because no test reached those lines -- - `build_features` failed only when a real retrain ran. This walks the AST - instead of waiting for a call site. + `build_features` failed only when a real retrain ran. This reads the + symbol tables instead of waiting for a call site. + """ + for name in sorted(p.name for p in TOOLS.glob("*.py")): + unbound = _unbound_names((TOOLS / name).read_text(encoding="utf-8"), name) + self.assertEqual(unbound, [], f"{name} reads names nothing binds") + + def test_the_unbound_name_check_is_scope_aware(self): + # The case the 2026-09-27 audit reproduced against the flat walk: a + # local import in one function does not bind the name in another. + leaked = ("def a():\n import numpy as np\n return np\n\n" + "def b():\n return np.zeros(1)\n") + self.assertEqual(_unbound_names(leaked), ["b:np"]) + # The case the check was written for, in the shape it had. + split = ("def build_features():\n t0 = time.time()\n" + " return CHECKPOINT_EVERY - t0\n") + self.assertEqual(_unbound_names(split), + ["build_features:CHECKPOINT_EVERY", "build_features:time"]) + # The same names bound where they are read pass, as do a closure, a + # `global` binding made in another function, and a method's `super()`. + bound = ("import time\nCHECKPOINT_EVERY = 500\n\n" + "def build_features():\n import numpy as np\n" + " return np, time.time(), CHECKPOINT_EVERY\n\n" + "def outer():\n x = 1\n def inner():\n return x\n" + " return inner\n\n" + "def init():\n global _CACHE\n _CACHE = {}\n\n" + "def use():\n return _CACHE\n\n" + "class K(dict):\n def m(self):\n return super().m()\n") + self.assertEqual(_unbound_names(bound), []) + # A class attribute is not visible from its methods: a real NameError. + self.assertEqual( + _unbound_names("class K:\n attr = 1\n def m(self):\n return attr\n"), + ["m:attr"]) + + def test_the_split_modules_carry_no_unused_import(self): + """The mirror of the unbound-name check: a name imported and never read. + + The 2026-08-26 split re-wrote three import blocks by hand and left + hashlib, math, re, statistics, json, Counter and defaultdict behind in + one file or another (audit 2026-09-27, E22), plus `DEFAULT_PROFILE_ROOT` + copies that `cli_common` owns. Declared re-exports (`noqa: F401`) are + read by nothing here by design and are skipped. """ import ast - import builtins - for name in sorted(p.name for p in TOOLS.glob("*.py")): - tree = ast.parse((TOOLS / name).read_text(encoding="utf-8")) - bound = set(dir(builtins)) | {"__file__", "__name__", "__doc__"} - for node in ast.walk(tree): - if isinstance(node, (ast.Import, ast.ImportFrom)): - for alias in node.names: - bound.add((alias.asname or alias.name).split(".")[0]) - elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, - ast.ClassDef)): - bound.add(node.name) - elif isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store): - bound.add(node.id) - elif isinstance(node, ast.arg): - bound.add(node.arg) - elif isinstance(node, ast.ExceptHandler) and node.name: - bound.add(node.name) - used = {n.id for n in ast.walk(tree) - if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load)} - self.assertEqual(sorted(used - bound), [], f"{name} has unbound names") + for name in ("train_voice_model.py", "voice_dataset.py", "voice_audit.py", + "train_ai_ism_classifier.py"): + source = (TOOLS / name).read_text(encoding="utf-8") + lines = source.splitlines() + tree = ast.parse(source) + imported = set() + for node in tree.body: + if not isinstance(node, (ast.Import, ast.ImportFrom)): + continue + if "F401" in " ".join(lines[node.lineno - 1:node.end_lineno]): + continue + for alias in node.names: + bound = (alias.asname or alias.name).split(".")[0] + if bound != "annotations": + imported.add(bound) + read = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)} + unused = sorted(imported - read) + self.assertEqual(unused, [], f"{name} imports but never reads {unused}") + import train_ai_ism_classifier + import voice_dataset + for module in (training, voice_dataset, train_ai_ism_classifier): + self.assertFalse(hasattr(module, "DEFAULT_PROFILE_ROOT"), module.__name__) + self.assertFalse(hasattr(train_ai_ism_classifier, "list_fields")) class VoiceAuditHelperTests(unittest.TestCase): @@ -314,6 +409,232 @@ def test_source_family_does_not_use_authorship_labels(self): "generated-public", ) + def test_a_blank_line_in_a_bank_is_skipped_not_parsed(self): + # `train_ai_ism_classifier.load_positives` reads the same files and + # always tolerated a blank line; `json.loads("")` raised here. + with tempfile.TemporaryDirectory() as raw: + bank = Path(raw) / "exemplar_paragraphs.jsonl" + record = json.dumps({"text": "word " * 30, "source": "s"}) + bank.write_text(record + "\n\n" + record + "\n \n", encoding="utf-8") + loaded = training._load_jsonl(bank, 1, "corpus") + self.assertEqual(len(loaded), 2) + self.assertEqual({record["label"] for record in loaded}, {1}) + + def test_repeated_audit_seeds_start_past_the_primary_split(self): + # The first "repeat" was the primary split again, so twenty repeats + # were nineteen plus the one already reported. + self.assertEqual(training.audit_split_seed(7, 0), 8) + self.assertNotIn(7, [training.audit_split_seed(7, attempt) + for attempt in range(400)]) + self.assertEqual([training.audit_split_seed(0, a) for a in range(3)], [1, 2, 3]) + # ...counted from the seed the primary actually used, which the + # trainer passes on rather than the one requested. + source = (TOOLS / "train_voice_model.py").read_text(encoding="utf-8") + self.assertIn("seed=primary_split_seed", source) + + +class _Array: + """Just enough of an ndarray for `build_features`' bookkeeping.""" + + def __init__(self, shape, value=None): + self.shape = tuple(shape) + self.value = value + + def __setitem__(self, key, value): + pass + + def __getitem__(self, key): + return self + + def __iter__(self): + return iter(self.value or []) + + def item(self): + return self.value + + def copy(self): + return self + + +class _Npz: + def __init__(self, stored): + self._stored = stored + self.files = list(stored) + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + def __getitem__(self, key): + shape, value = self._stored[key] + return _Array(shape, value) + + +class _Numpy(types.ModuleType): + """A stand-in for the optional numpy: an array is its shape plus, for the + small ones, its values, and `savez`/`load` round-trip them through JSON.""" + + float32 = "float32" + + def __init__(self): + super().__init__("numpy") + + def zeros(self, shape, dtype=None): + return _Array(shape) + + def asarray(self, values, dtype=None): + if isinstance(values, _Array): + return values + if isinstance(values, (int, float, str)): + return _Array((), values) + rows = list(values) + if rows and isinstance(rows[0], (list, tuple)): + return _Array((len(rows), len(rows[0])), [list(row) for row in rows]) + return _Array((len(rows),), rows) + + def array_equal(self, first, second): + return first.value == second.value + + def savez(self, handle, **arrays): + handle.write(json.dumps({key: [list(array.shape), array.value] + for key, array in arrays.items()}).encode("utf-8")) + + def load(self, path, allow_pickle=False): + return _Npz(json.loads(Path(path).read_bytes())) + + +class EmbeddingFailureTests(unittest.TestCase): + """A run whose encoder fails must not become the feature cache. + + `embedder_available()` probes the import, and that is what the cache + fingerprint records. The encode itself can still fail -- offline, a + device fault -- and the rows then carry corpus_cos = 0.0; cached under the + healthy fingerprint they were served as measured to every later run. + """ + + RECORDS = [{"text": "word " * 40 + str(i), "label": i % 2, "source": f"s{i}", + "section": "method"} for i in range(4)] + + def test_width_zero_is_a_failure_only_when_the_embedder_imports(self): + original = features.embedder_available + try: + features.embedder_available = lambda: True + self.assertTrue(training.embeddings_failed_at_run_time(0)) + self.assertFalse(training.embeddings_failed_at_run_time(384)) + features.embedder_available = lambda: False + self.assertFalse(training.embeddings_failed_at_run_time(0)) + finally: + features.embedder_available = original + + @contextlib.contextmanager + def _fakes(self, encoder): + """The numpy stand-in, an importable embedder whose encode is + `encoder`, and a free per-record feature pass that records its calls.""" + saved = sys.modules.get("numpy") + originals = (features.embedder_available, features._embedder, + features.features_vector) + featurized: list[str] = [] + sys.modules["numpy"] = _Numpy() + features.embedder_available = lambda: True + features._embedder = lambda: types.SimpleNamespace(encode=encoder) + features.features_vector = ( + lambda text, **kw: featurized.append(text) or [0.0] * len(features.FEATURE_NAMES)) + try: + yield featurized + finally: + (features.embedder_available, features._embedder, + features.features_vector) = originals + if saved is None: + sys.modules.pop("numpy", None) + else: + sys.modules["numpy"] = saved + + @staticmethod + def _failing(*args, **kwargs): + raise RuntimeError("offline") + + @staticmethod + def _working(plains, **kwargs): + return [[0.1, 0.2, 0.3] for _ in plains] + + def test_a_failed_encode_withholds_the_cache_and_keeps_the_rows(self): + with tempfile.TemporaryDirectory() as raw: + field_dir = Path(raw) + cache = field_dir / "voice_features_cache.npz" + partial = field_dir / "voice_features_cache.partial.npz" + with self._fakes(self._failing) as featurized, \ + contextlib.redirect_stderr(io.StringIO()) as err: + _X, _y, _src, emb = training.build_features( + field_dir, self.RECORDS, "gpt2", False) + self.assertEqual(emb.shape, (4, 0)) + self.assertEqual(len(featurized), 4) + self.assertFalse(cache.exists()) + self.assertTrue(partial.exists()) + self.assertEqual(json.loads(partial.read_bytes())["n_done"][1], 4) + self.assertIn("withheld", err.getvalue()) + # The next run resumes past the language-model pass, retries the + # embedder, and only then writes the cache. + with self._fakes(self._working) as featurized, \ + contextlib.redirect_stderr(io.StringIO()) as err: + _X, _y, _src, emb = training.build_features( + field_dir, self.RECORDS, "gpt2", False) + self.assertEqual(emb.shape, (4, 3)) + self.assertEqual(featurized, []) + self.assertIn("resuming featurization from checkpoint row 4/4", err.getvalue()) + self.assertTrue(cache.exists()) + self.assertFalse(partial.exists()) + + def test_a_cache_of_empty_embeddings_under_a_healthy_fingerprint_is_recomputed(self): + # A cache an earlier run wrote from a failed encode. + with tempfile.TemporaryDirectory() as raw, self._fakes(self._working) as featurized: + field_dir = Path(raw) + fingerprint = training.feature_cache_fingerprint(field_dir, self.RECORDS, "gpt2") + names = list(features.FEATURE_NAMES) + stale = {"X": [[4, len(names)], None], + "y": [[4], [r["label"] for r in self.RECORDS]], + "src": [[4], [r["source"] for r in self.RECORDS]], + "emb": [[4, 0], None], + "names": [[len(names)], names], + "fingerprint": [[], fingerprint]} + (field_dir / "voice_features_cache.npz").write_bytes( + json.dumps(stale).encode("utf-8")) + with contextlib.redirect_stderr(io.StringIO()) as err: + _X, _y, _src, emb = training.build_features( + field_dir, self.RECORDS, "gpt2", False) + self.assertIn("empty embeddings", err.getvalue()) + self.assertEqual(len(featurized), 4) + self.assertEqual(emb.shape, (4, 3)) + + +class TrainingEntryTests(unittest.TestCase): + def test_a_missing_learning_dependency_is_one_line_and_exit_2(self): + saved = sys.modules.get("numpy") + sys.modules["numpy"] = None # `import numpy` raises, installed or not + try: + with tempfile.TemporaryDirectory() as raw: + (Path(raw) / "wgl").mkdir() + err = io.StringIO() + with contextlib.redirect_stderr(err), \ + contextlib.redirect_stdout(io.StringIO()): + code = training.main(["--field", "wgl", "--profile-root", raw]) + finally: + if saved is None: + sys.modules.pop("numpy", None) + else: + sys.modules["numpy"] = saved + self.assertEqual(code, 2) + self.assertEqual(err.getvalue().count("\n"), 1, err.getvalue()) + self.assertIn("[train_voice_model] cannot train without numpy", err.getvalue()) + + def test_the_field_is_resolved_under_the_tools_own_name(self): + # `args.profile_root / None` was a TypeError traceback without --field. + with tempfile.TemporaryDirectory() as raw: + with self.assertRaises(SystemExit) as caught: + training.main(["--field", "nope", "--profile-root", raw]) + self.assertIn("[train_voice_model]", str(caught.exception)) + if __name__ == "__main__": unittest.main() diff --git a/tools/train_ai_ism_classifier.py b/tools/train_ai_ism_classifier.py index 3920005..fa9fb91 100644 --- a/tools/train_ai_ism_classifier.py +++ b/tools/train_ai_ism_classifier.py @@ -18,7 +18,6 @@ from __future__ import annotations -import argparse import json import sys from pathlib import Path @@ -27,7 +26,6 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" HANDCRAFTED_NEGATIVES = REPO_ROOT / "tools" / "ai_ism_negatives_handcrafted.txt" # Mutation infrastructure removed — empirically (see docstring) mutated @@ -35,17 +33,11 @@ # generalization. Handcrafted negatives only. -def list_fields(profile_root: Path) -> list[str]: - return cli_common.list_fields(profile_root) - - def resolve_field(arg_field: str | None, profile_root: Path) -> str: return cli_common.resolve_field( arg_field, profile_root, tool="train_ai_ism_classifier") - - def load_positives(jsonl_path: Path, min_words: int = 30) -> list[tuple[str, str]]: """(paragraph, source-paper) pairs. diff --git a/tools/train_voice_model.py b/tools/train_voice_model.py index 1791ff5..0e831a0 100644 --- a/tools/train_voice_model.py +++ b/tools/train_voice_model.py @@ -18,38 +18,30 @@ from __future__ import annotations -import argparse -import hashlib import json -import math -import re -import statistics import sys import time -from collections import Counter, defaultdict +from collections import Counter from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first import deai_features as df # noqa: E402 resolves only after the sys.path insert -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" - # Re-exported so `train_voice_model.` keeps resolving for existing # callers and tests after the 2026-08-26 split. A contract test asserts this # list still covers every public name in both modules. from voice_dataset import ( # noqa: E402,F401 -- re-export, unused here by design CHECKPOINT_EVERY, - build_features, build_field_lexicon, feature_cache_fingerprint, - load_records, source_family, + build_features, build_field_lexicon, embeddings_failed_at_run_time, + feature_cache_fingerprint, load_records, source_family, _atomic_savez, _load_jsonl, _record_embeddings, _tokens, ) from voice_audit import ( # noqa: E402,F401 -- re-export, unused here by design HARDSET_AI_CATEGORIES, HARDSET_HUMAN_CATEGORIES, - aggregate_audits, binary_metrics, confound_audit, first_valid_group_split, - hardset_evaluation, repeated_group_audit, section_normalize_uid, - split_corpus_cos, + aggregate_audits, audit_split_seed, binary_metrics, confound_audit, + first_valid_group_split, hardset_evaluation, repeated_group_audit, + section_normalize_uid, split_corpus_cos, _auc, _bootstrap_auc_ci, _breakdown, _math_bin, _metric_maps, _quantile, _series_summary, _three_way_bin, ) @@ -74,15 +66,31 @@ def main(argv: list[str] | None = None) -> int: if args.audit_splits < 3: p.error("--audit-splits must be at least 3") - import numpy as np - from sklearn.linear_model import LogisticRegression - from sklearn.ensemble import HistGradientBoostingClassifier - from sklearn.preprocessing import StandardScaler - from sklearn.metrics import (roc_auc_score, f1_score, balanced_accuracy_score, - confusion_matrix) - import joblib + # The field is required here -- the records live under it -- so it is + # resolved before anything else: `args.profile_root / None` was a + # TypeError traceback when `--field` was omitted, against a `--help` that + # promised auto-detection. + field = cli_common.resolve_field(args.field, args.profile_root, + tool="train_voice_model") + field_dir = args.profile_root / field + + # The learning stack is optional (requirements.txt); its absence is one + # line and the execution-failure exit, not a ModuleNotFoundError traceback. + try: + import numpy as np + from sklearn.linear_model import LogisticRegression + from sklearn.ensemble import HistGradientBoostingClassifier + from sklearn.preprocessing import StandardScaler + from sklearn.metrics import (roc_auc_score, f1_score, balanced_accuracy_score, + confusion_matrix) + import joblib + except ImportError as error: + print(f"[train_voice_model] cannot train without " + f"{getattr(error, 'name', None) or error}: install the optional " + f"learning dependencies from requirements.txt (numpy, scikit-learn, " + f"joblib)", file=sys.stderr) + return 2 - field_dir = args.profile_root / args.field recs = load_records(field_dir) n_pos = sum(r["label"] for r in recs); n_neg = len(recs) - n_pos if not recs: @@ -175,6 +183,8 @@ def _make(name): print(f"[audit] repeated source-grouped comparison: " f"{args.audit_splits} splits", file=sys.stderr) + # Seeded from the split the primary audit actually used, so the repeats + # start one past it and the primary is not one of the N (audit_split_seed). repeated_audit = repeated_group_audit( recs, X, @@ -182,7 +192,7 @@ def _make(name): src, n_splits=args.audit_splits, val_frac=args.val_frac, - seed=args.seed, + seed=primary_split_seed, emb=emb, ) cache_fingerprint = feature_cache_fingerprint(field_dir, recs, args.model) @@ -211,7 +221,7 @@ def _make(name): evaluation = { "schema": "sci-paper.voice-model-evaluation.v1", "generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), - "field": args.field, + "field": field, "task": "curated-reference versus generated-negative field compatibility", "interpretation": ( "This evaluation measures compatibility with the recorded training task. " diff --git a/tools/voice_audit.py b/tools/voice_audit.py index 972f7c9..03892f9 100644 --- a/tools/voice_audit.py +++ b/tools/voice_audit.py @@ -14,13 +14,10 @@ from __future__ import annotations -import hashlib -import json import math -import re import statistics import sys -from collections import Counter, defaultdict +from collections import defaultdict from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) @@ -530,10 +527,25 @@ def first_valid_group_split(X, y, groups, *, val_frac: float, seed: int, f"no grouped split with both classes after {maximum_attempts} attempts") +def audit_split_seed(primary_seed: int, attempt: int) -> int: + """The seed of the `attempt`-th repeated split: one past the primary's. + + The primary split (`first_valid_group_split`) is reported on its own under + `primary_split`. The repeats once started at the same seed, so the first + "repeat" was that split again and twenty repeats were nineteen plus the + one already shown. Counting from the seed the primary actually used, not + the one requested, keeps them disjoint even when the requested seed gave + no valid split and the primary moved on. + """ + return primary_seed + 1 + attempt + + def repeated_group_audit(recs: list[dict], X, y, groups, *, n_splits: int, val_frac: float, seed: int, emb=None) -> dict: """Compare raw and section-normalized UID under repeated grouped splits. + `seed` is the seed the primary split used; the repeats start one past it + (`audit_split_seed`), so the split already reported is not among them. When record embeddings are supplied, every split recomputes corpus_cos against a training-only centroid (split_corpus_cos), so held-out papers cannot inflate their own similarity feature. @@ -550,7 +562,7 @@ def repeated_group_audit(recs: list[dict], X, y, groups, *, n_splits: int, attempts = 0 maximum_attempts = max(20, n_splits * 20) while len(raw_reports) < n_splits and attempts < maximum_attempts: - split_seed = seed + attempts + split_seed = audit_split_seed(seed, attempts) splitter = GroupShuffleSplit( n_splits=1, test_size=val_frac, random_state=split_seed) train_indices, validation_indices = next(splitter.split(X, y, groups=groups)) @@ -631,6 +643,7 @@ def repeated_group_audit(recs: list[dict], X, y, groups, *, n_splits: int, "requested_splits": n_splits, "completed_splits": len(raw_reports), "attempts": attempts, + "first_seed": audit_split_seed(seed, 0), "validation_fraction": val_frac, "split_records": split_records, "raw_uid": aggregate_audits(raw_reports), diff --git a/tools/voice_dataset.py b/tools/voice_dataset.py index 3a9e3ee..92fb85c 100644 --- a/tools/voice_dataset.py +++ b/tools/voice_dataset.py @@ -26,18 +26,25 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import deai_features as df # noqa: E402 resolves only after the sys.path insert -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" - CHECKPOINT_EVERY = 500 # rows between resumable partial-cache writes def _load_jsonl(path: Path, label: int, prefix: str) -> list[dict]: + """Records of one bank, labelled; a blank line is skipped, not parsed. + + The banks are appended to by hand and by the fetch tools, and a blank + line -- a doubled trailing newline, a separator -- is not a record. + `json.loads("")` raised on it and took the whole training run down, while + `train_ai_ism_classifier.load_positives` reads the same files and always + tolerated it. The fallback `source` keeps the physical line index. + """ out: list[dict] = [] if not path.exists(): return out with path.open(encoding="utf-8") as f: for i, line in enumerate(f): + if not line.strip(): + continue r = json.loads(line) t = r.get("text", "") if t and len(t.split()) >= 30: @@ -78,7 +85,6 @@ def load_records(field_dir: Path) -> list[dict]: recs += _load_jsonl(field_dir / "ai_ism_negatives_public.jsonl", 0, "public") hand = field_dir / "ai_ism_negatives_handcrafted.txt" if hand.exists(): - import re blocks = [b.strip() for b in re.split(r"\n\s*\n", hand.read_text(encoding="utf-8"))] for i, b in enumerate(blocks): if b and not b.startswith("#") and len(b.split()) >= 30: @@ -217,13 +223,32 @@ def _record_embeddings(recs: list[dict]): return np.asarray(emb, np.float32) +def embeddings_failed_at_run_time(width: int) -> bool: + """Whether an embedding pass produced no vectors although the embedder imports. + + The cache fingerprint records `embedder_available()`, which probes the + import only. The encode itself can still fail -- an offline model + download, a device fault -- and `_record_embeddings` then degrades to an + (n, 0) block with a warning. Rows built from it carry corpus_cos = 0.0, + and a cache written under the import-time fingerprint would serve them as + measured to every later run, including the runs where the encoder works. + Width 0 without the embedder is the known, fingerprinted degraded state, + and is not this. + """ + return width == 0 and df.embedder_available() + + def build_features(field_dir: Path, recs: list[dict], model_name: str, refeature: bool): """Return (X, y, src, emb) with provenance-fingerprinted caching. The slow per-record language-model pass checkpoints every ``CHECKPOINT_EVERY`` rows into a resumable partial cache so a preempted - cloud run loses at most one chunk, never the whole extraction. + cloud run loses at most one chunk, never the whole extraction. When the + embedding pass fails at run time (`embeddings_failed_at_run_time`), the + finished rows are written to that same checkpoint instead of the cache, + so the next run retries the embedder without refeaturizing and only a + run whose corpus_cos is real ever becomes the cache. """ import numpy as np cache = field_dir / "voice_features_cache.npz" @@ -258,8 +283,16 @@ def build_features(field_dir: Path, recs: list[dict], model_name: str, np.asarray([str(record["source"]) for record in recs], str), ) ) - if (cached_fingerprint == fingerprint and names_match + reason = None + if not (cached_fingerprint == fingerprint and names_match and values_match): + reason = "provenance mismatch" + elif embeddings_failed_at_run_time(int(cached["emb"].shape[1])): + # A cache an earlier run wrote from a failed encode: its + # fingerprint is healthy and its corpus_cos is all zero. + reason = ("holds empty embeddings although the embedder " + "imports (the encode failed when it was written)") + if reason is None: print(f"[train] using provenance-verified cached features " f"{cached['X'].shape}", file=sys.stderr) return (cached["X"].copy(), cached["y"].copy(), @@ -268,8 +301,7 @@ def build_features(field_dir: Path, recs: list[dict], model_name: str, print(f"[train] unusable legacy feature cache ({error}); recomputing", file=sys.stderr) else: - print("[train] feature cache provenance mismatch; recomputing", - file=sys.stderr) + print(f"[train] feature cache {reason}; recomputing", file=sys.stderr) y = np.asarray([record["label"] for record in recs], int) src = np.asarray([str(record["source"]) for record in recs], str) @@ -310,11 +342,23 @@ def build_features(field_dir: Path, recs: list[dict], model_name: str, cos_index = df.FEATURE_NAMES.index("corpus_cos") if centroid is not None and emb.shape[1]: X[:, cos_index] = emb @ np.asarray(centroid, np.float32) - _atomic_savez(cache, X=X, y=y, src=src, emb=emb, - names=np.asarray(df.FEATURE_NAMES, str), - fingerprint=np.asarray(fingerprint)) - if partial.exists(): - partial.unlink() + if embeddings_failed_at_run_time(int(emb.shape[1])): + # The slow rows are kept as a finished checkpoint (`n_done` = every + # row) and the cache is withheld: the next run resumes past the + # language-model pass, retries the embedder, and writes the cache + # only once corpus_cos is what the fingerprint says it is. + _atomic_savez(partial, X=X, n_done=np.asarray(len(recs)), + names=np.asarray(df.FEATURE_NAMES, str), + fingerprint=np.asarray(fingerprint)) + print(f"[train] embeddings failed although the embedder imports; " + f"feature cache withheld, language-model rows checkpointed -> " + f"{partial.name}", file=sys.stderr) + else: + _atomic_savez(cache, X=X, y=y, src=src, emb=emb, + names=np.asarray(df.FEATURE_NAMES, str), + fingerprint=np.asarray(fingerprint)) + if partial.exists(): + partial.unlink() print(f"[train] featurized {len(recs)} in {(time.time()-t0):.0f}s -> {cache.name}", file=sys.stderr) return X, y, src, emb From 41a29a8378ea5c8d00de1612062382f381440deb Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:35:17 +0000 Subject: [PATCH 14/27] fetch_arxiv_abstracts: the flags hold in both modes, ids are bare, and the query sets are [WGL] Audit 2026-09-27, D1-D5, D12, D13, D24, D26. `--journals` applies on the local full-text shortcut, which had built the "refereed" breadth corpus without any journal restriction. `--start-at` sizes each page from where the band began instead of requesting a negative page. A throttled full-text run reports TRUNCATED and exits 2 like the abstract sweep, instead of DONE and exit 0. `--updated-before` applies under `--fulltext`, and `--query-set`, `--out-name` and `--resume` are refused there rather than accepted and ignored. Candidate dedup and the output directory key on the bare arXiv id, so a version bump no longer fetches `...v2` beside `...v1`. A `\documentstyle` or comment-led file is a LaTeX source, and colliding tarball basenames are both kept. Omitting `--field` resolves through cli_common and is a message and exit 2, never `Path / None`. The built-in query sets, keyword filter and journal table are marked [WGL]; `--query` replaces them for any other field. The tests outgrew the 750-line budget and are split by subject: the parser, backoff, vintage, resume and held-out tests stay in test_fetch_arxiv_abstracts, the sweep paths are in test_fetch_arxiv_sweeps, and the offline harness both use is _fetch_harness. The `__main__` block sits at the end of each file, so a direct run executes every class. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- tests/_fetch_harness.py | 122 +++++++ tests/test_fetch_arxiv_abstracts.py | 322 +++--------------- tests/test_fetch_arxiv_sweeps.py | 379 ++++++++++++++++++++++ tools/fetch_arxiv_abstracts.py | 486 ++++++++++++++-------------- 4 files changed, 790 insertions(+), 519 deletions(-) create mode 100644 tests/_fetch_harness.py create mode 100644 tests/test_fetch_arxiv_sweeps.py diff --git a/tests/_fetch_harness.py b/tests/_fetch_harness.py new file mode 100644 index 0000000..90c4d47 --- /dev/null +++ b/tests/_fetch_harness.py @@ -0,0 +1,122 @@ +"""The offline harness the fetch_arxiv_abstracts tests share. + +Every CLI-level test runs `fetch` with a temporary repository root, no +sleeping and no network: `urlopen_backoff`, `fetch_page` and `_eprint_bytes` +fail the test unless a stub is named. Two test files use it (the tool's tests +outgrew the 750-line budget on 2026-09-27 and were split by subject), so the +harness lives here once. + +The name starts with an underscore so `unittest discover` never collects it. +""" + +from __future__ import annotations + +import io +import json +import types +from contextlib import contextmanager, redirect_stderr, redirect_stdout +from pathlib import Path + +from _toolpath import TOOLS # noqa: E402,F401 -- because importing it is what puts tools/ on sys.path + +import fetch_arxiv_abstracts as fetch # noqa: E402 + +ATOM_NS = "http://www.w3.org/2005/Atom" +ARXIV_NS = "http://arxiv.org/schemas/atom" + +def _record(source: str, updated: str, journal_ref: str | None = "ApJ, 927, 101 (2021)", + text: str = "word " * 60) -> dict: + return {"section": "abstract", "text": text, "source": source, + "year": 2020, "published": "2020-01-01", "updated": updated, + "journal_ref": journal_ref, + "journal": fetch.classify_journal(journal_ref), "doi": None} + + +LENSING = "weak lensing shear " * 20 # an abstract the [WGL] keyword shortcut keeps +# A bare (uncompressed) e-print that clears both full-text keep thresholds. +SOURCE = ("\\documentclass{article}\n\\section{A}\n" + "word " * 1600 + + "\\section{B}\n\\section{C}\n").encode("utf-8") + + +def _no_network(*args, **kwargs): + raise AssertionError("a test reached the network layer") + + +def _throttle(arxiv_id: str) -> bytes: + raise fetch.Throttled("429 after 3 backoffs totalling 420 s") + + +def _pages_once(pages: list[dict] | None): + """A fetch_page stub: `pages` on the first call, an empty page after. The + positional arguments of every call are kept on `stub.calls`.""" + def stub(*args, **kwargs): + stub.calls.append(args) + return list(pages or []) if len(stub.calls) == 1 else [] + stub.calls = [] + return stub + + +@contextmanager +def _offline(tmp: Path, **stubs): + """Run `fetch` with `tmp` as the repository root, no sleeping and no network: + `urlopen_backoff`, `fetch_page` and `_eprint_bytes` fail the test unless a + stub is named here. Yields the captured (stdout, stderr) buffers.""" + stubs = {"REPO_ROOT": tmp, "urlopen_backoff": _no_network, + "fetch_page": _no_network, "_eprint_bytes": _no_network, **stubs} + saved = {name: getattr(fetch, name) for name in stubs} + real_sleep, fetch.time.sleep = fetch.time.sleep, lambda seconds: None + for name, stub in stubs.items(): + setattr(fetch, name, stub) + try: + with redirect_stdout(io.StringIO()) as out, redirect_stderr(io.StringIO()) as err: + yield out, err + finally: + fetch.time.sleep = real_sleep + for name, value in saved.items(): + setattr(fetch, name, value) + + +def _sweep(tmp: Path, argv: list[str], pages: list[dict] | None = None, + field: str | None = "wgl", **stubs) -> tuple[int, str]: + """Run the abstract sweep offline against `tmp` as the profile root; returns + (exit status, stderr). A parser error propagates as SystemExit.""" + stubs.setdefault("fetch_page", _pages_once(pages)) + with _offline(tmp, **stubs) as (_, err): + status = fetch.main(["--profile-root", str(tmp)] + + (["--field", field] if field else []) + argv) + return status, err.getvalue() + + +def _fulltext(tmp: Path, argv: list[str], pages: list[dict] | None = None, + **stubs) -> tuple[int, str, Path]: + """Run --fulltext offline into tmp/style-corpus/wgl/fulltext-test; returns + (exit status, stdout + stderr, that directory).""" + stubs.setdefault("fetch_page", _pages_once(pages)) + with _offline(tmp, **stubs) as (out, err): + status = fetch.main(["--field", "wgl", "--profile-root", str(tmp), "--fulltext", + "--fulltext-dir", "fulltext-test"] + argv) + return status, out.getvalue() + err.getvalue(), tmp / "style-corpus" / "wgl" / "fulltext-test" + + +def _write_bank(tmp: Path, records: list[dict], name: str = fetch.BANK) -> Path: + """Seed tmp/wgl/ with JSONL records, as an earlier sweep would have.""" + field = tmp / "wgl" + field.mkdir(parents=True, exist_ok=True) + out = field / name + out.write_text("".join(json.dumps(r) + "\n" for r in records), encoding="utf-8") + return out + + +def _bank(path: Path) -> list[dict]: + """The JSONL records a sweep wrote, or [] when the file does not exist.""" + if not path.exists(): + return [] + return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line] + + +def _candidate_args(tmp: Path, **over) -> types.SimpleNamespace: + """The argparse namespace `_candidate_ids` reads, with offline defaults.""" + base = dict(author="", author_is="", max_authors=0, query=[], updated_before="", + profile_root=tmp, field="wgl", per_query=1, page=1, start_at=0, + date_lo="201001010000", date_hi="202112312359", sleep=0) + return types.SimpleNamespace(**{**base, **over}) diff --git a/tests/test_fetch_arxiv_abstracts.py b/tests/test_fetch_arxiv_abstracts.py index 256c793..0ae597e 100644 --- a/tests/test_fetch_arxiv_abstracts.py +++ b/tests/test_fetch_arxiv_abstracts.py @@ -1,19 +1,19 @@ from __future__ import annotations +import io import json -import sys import tempfile import types -import io import unittest -from contextlib import redirect_stderr, redirect_stdout import urllib.error import urllib.request +from contextlib import redirect_stderr, redirect_stdout from pathlib import Path -from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path +from _toolpath import TOOLS # noqa: E402 -- because importing it is what puts tools/ on sys.path -import fetch_arxiv_abstracts as fetch +import fetch_arxiv_abstracts as fetch # noqa: E402 +from _fetch_harness import ARXIV_NS, ATOM_NS, _bank, _record, _sweep, _write_bank # noqa: E402 class ClassifyJournalTest(unittest.TestCase): @@ -80,9 +80,6 @@ def test_missing_journal_ref_is_unmatched_not_an_error(self) -> None: self.assertIsNone(fetch.classify_journal("")) -ATOM_NS = "http://www.w3.org/2005/Atom" -ARXIV_NS = "http://arxiv.org/schemas/atom" - FEED = f""" http://arxiv.org/abs/2001.00001v1 @@ -157,6 +154,15 @@ def test_wl_set_is_weak_lensing_only(self) -> None: "mass reconstruction", "peak statistics")), f"{query!r} carries no weak-lensing term") + def test_the_built_in_sets_and_keyword_filter_are_marked_wgl(self) -> None: + # D24: every hard-coded astro-ph / weak-lensing table is field-specific + # content and says so where it is defined, in the comment block above it. + source = (TOOLS / "fetch_arxiv_abstracts.py").read_text(encoding="utf-8") + for name in ("JOURNAL_FILTERS", "QUERIES", "AUTHOR_QUERIES", "WL_QUERIES", + "FULLTEXT_QUERIES", "_FIELD_RELEVANCE_RE"): + preceding = source[:source.index(f"\n{name} = ")].rsplit("\n\n", 1)[-1] + self.assertIn("[WGL]", preceding, f"{name} lacks a [WGL] marker") + class _FakeResponse: def __init__(self, payload: bytes) -> None: @@ -234,13 +240,6 @@ def test_throttled_is_not_a_generic_exception_subclass_catch(self) -> None: self.assertFalse(issubclass(fetch.Throttled, urllib.error.HTTPError)) -def _record(source: str, updated: str, journal_ref: str = "ApJ, 927, 101 (2021)") -> dict: - return {"section": "abstract", "text": "word " * 60, "source": source, - "year": 2020, "published": "2020-01-01", "updated": updated, - "journal_ref": journal_ref, - "journal": fetch.classify_journal(journal_ref), "doi": None} - - class TextVintageFilterTest(unittest.TestCase): """--date-hi bounds the v1 submission; only --updated-before dates the text. @@ -252,27 +251,9 @@ class TextVintageFilterTest(unittest.TestCase): def _run(self, page: list[dict], extra: list[str]) -> list[dict]: with tempfile.TemporaryDirectory() as name: tmp = Path(name) - (tmp / "wgl").mkdir(parents=True) - real = fetch.fetch_page - calls = {"n": 0} - - def one_page(*a, **k): - calls["n"] += 1 - return page if calls["n"] == 1 else [] - - fetch.fetch_page = one_page - try: - # The CLI's progress log is the suite's only stdout noise and - # prints an absolute temp path; capture it so a test run stays - # readable, the same technique validate_plugin.py uses. - with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): - fetch.main(["--field", "wgl", "--profile-root", str(tmp), - "--query-set", "wl", "--out-name", "v.jsonl", - "--per-query", "100"] + extra) - finally: - fetch.fetch_page = real - text = (tmp / "wgl" / "v.jsonl").read_text(encoding="utf-8").strip() - return [json.loads(line) for line in text.splitlines() if line] + _sweep(tmp, ["--query-set", "wl", "--out-name", "v.jsonl", + "--per-query", "100"] + extra, page) + return _bank(tmp / "wgl" / "v.jsonl") def test_record_revised_after_the_cutoff_is_dropped(self) -> None: page = [_record("arxiv:2001.1v3", "2023-04-01"), @@ -303,34 +284,18 @@ class IncompleteSweepTest(unittest.TestCase): """ def _run(self, pages) -> tuple[int, list[dict]]: + def flaky(*args, **kwargs): + flaky.calls += 1 + if flaky.calls == 1: + return pages + raise RuntimeError("simulated transient API failure") + flaky.calls = 0 with tempfile.TemporaryDirectory() as temporary: tmp = Path(temporary) - calls = {"n": 0} - - def flaky(*args, **kwargs): - calls["n"] += 1 - if calls["n"] == 1: - return pages - raise RuntimeError("simulated transient API failure") - - real = fetch.fetch_page - fetch.fetch_page = flaky - try: - with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()) as err: - status = fetch.main([ - "--field", "wgl", "--profile-root", str(tmp), - "--query-set", "wl", "--out-name", "v.jsonl", - "--per-query", "100", "--page", "1", - ]) - finally: - fetch.fetch_page = real - self.assertIn("INCOMPLETE", err.getvalue()) - written = tmp / "wgl" / "v.jsonl" - records = [] - if written.exists(): - records = [json.loads(line) for line - in written.read_text(encoding="utf-8").splitlines() if line] - return status, records + status, err = _sweep(tmp, ["--query-set", "wl", "--out-name", "v.jsonl", + "--per-query", "100", "--page", "1"], fetch_page=flaky) + self.assertIn("INCOMPLETE", err) + return status, _bank(tmp / "wgl" / "v.jsonl") def test_page_error_reports_incomplete_and_exits_two(self): status, records = self._run([_record("arxiv:2001.1v1", "ApJ, 927, 101 (2021)")]) @@ -342,72 +307,40 @@ def test_page_error_reports_incomplete_and_exits_two(self): class ResumeTest(unittest.TestCase): """A rerun after rate limiting must extend the corpus, never shrink it.""" - def _run_main(self, tmp: Path, extra: list[str]) -> int: - real = fetch.fetch_page - fetch.fetch_page = lambda *a, **k: [] # no network in tests - try: - with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): - return fetch.main(["--field", "wgl", "--profile-root", str(tmp), - "--query-set", "wl", "--journals", "apj", - "--out-name", "carry.jsonl", - "--per-query", "100"] + extra) - finally: - fetch.fetch_page = real + ARGV = ["--query-set", "wl", "--journals", "apj", "--out-name", "carry.jsonl", + "--per-query", "100"] def _seed(self, tmp: Path) -> Path: - field = tmp / "wgl" - field.mkdir(parents=True) - out = field / "carry.jsonl" - with out.open("w", encoding="utf-8") as handle: - for i in range(3): - handle.write(json.dumps({ - "section": "abstract", "text": "word " * 60, - "source": f"arxiv:20{i:02d}.00001", "year": 2020, - "journal_ref": "ApJ, 927, 101 (2021)", "journal": "apj", - "doi": None}) + "\n") - return out + return _write_bank(tmp, [{"section": "abstract", "text": "word " * 60, + "source": f"arxiv:20{i:02d}.00001", "year": 2020, + "journal_ref": "ApJ, 927, 101 (2021)", "journal": "apj", + "doi": None} for i in range(3)], "carry.jsonl") def test_resume_keeps_existing_records_when_the_sweep_finds_nothing(self) -> None: with tempfile.TemporaryDirectory() as name: - tmp = Path(name) - out = self._seed(tmp) - self.assertEqual(self._run_main(tmp, ["--resume"]), 0) - self.assertEqual(len(out.read_text(encoding="utf-8").strip().splitlines()), 3) + out = self._seed(Path(name)) + self.assertEqual(_sweep(Path(name), self.ARGV + ["--resume"])[0], 0) + self.assertEqual(len(_bank(out)), 3) def test_without_resume_the_writer_truncates(self) -> None: # Documents the destructive default that --resume exists to avoid. with tempfile.TemporaryDirectory() as name: - tmp = Path(name) - out = self._seed(tmp) - self.assertEqual(self._run_main(tmp, []), 0) + out = self._seed(Path(name)) + self.assertEqual(_sweep(Path(name), self.ARGV)[0], 0) self.assertEqual(out.read_text(encoding="utf-8").strip(), "") def test_resume_deduplicates_against_carried_sources(self) -> None: with tempfile.TemporaryDirectory() as name: - tmp = Path(name) - out = self._seed(tmp) - duplicate = {"section": "abstract", "text": "word " * 60, - "source": "arxiv:2000.00001", "year": 2020, - "journal_ref": "ApJ, 927, 101 (2021)", - "journal": "apj", "doi": None} - real = fetch.fetch_page - calls = {"n": 0} - - def one_page(*a, **k): - calls["n"] += 1 - return [duplicate] if calls["n"] == 1 else [] - - fetch.fetch_page = one_page - try: - with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): - fetch.main(["--field", "wgl", "--profile-root", str(tmp), - "--query-set", "wl", "--journals", "apj", - "--out-name", "carry.jsonl", "--per-query", "100", - "--resume"]) - finally: - fetch.fetch_page = real - lines = out.read_text(encoding="utf-8").strip().splitlines() - self.assertEqual(len(lines), 3, "carried source must not be re-added") + out = self._seed(Path(name)) + _sweep(Path(name), self.ARGV + ["--resume"], [_record("arxiv:2000.00001", "2020-01-01")]) + self.assertEqual(len(_bank(out)), 3, "carried source must not be re-added") + + def test_resume_does_not_bank_a_revised_paper_twice(self) -> None: + # D13: dedup keys on the bare id, so v2 of a carried paper is the same paper. + with tempfile.TemporaryDirectory() as name: + out = self._seed(Path(name)) + _sweep(Path(name), self.ARGV + ["--resume"], [_record("arxiv:2000.00001v2", "2021-01-01")]) + self.assertEqual(len(_bank(out)), 3) class HeldOutSetTest(unittest.TestCase): @@ -494,162 +427,3 @@ def test_a_bad_journal_key_fails_in_fulltext_mode_too(self) -> None: if __name__ == "__main__": unittest.main() - - -OLD_ID_FEED = f""" - - http://arxiv.org/abs/astro-ph/9403003v1 - 1994-03-01T00:00:00Z - 1994-03-01T00:00:00Z - {' word' * 60} - Ian P. Dell'Antonio - J. Anthony Tyson - - - http://arxiv.org/abs/2001.00009v1 - 2020-01-01T00:00:00Z - 2020-01-01T00:00:00Z - {' word' * 60} - -""" - - -def _parse(feed: str) -> list[dict]: - real = fetch.urlopen_backoff - fetch.urlopen_backoff = lambda *a, **k: feed.encode("utf-8") - try: - return fetch.fetch_page("q", 0, 10, "199001010000", "202512312359") - finally: - fetch.urlopen_backoff = real - - -class OldStyleIdTest(unittest.TestCase): - """An `archive/YYMMNNN` id must keep its archive. - - The e-print endpoint 404s without it, and the failure is silent in the worst - way: the sweep reports `failed=N` and carries on, so the corpus is short by - however many old papers the query returned. Measured on a live 1990-2021 - author sweep before the fix, 7 of 19 candidates were lost this way. - """ - - def test_the_archive_prefix_survives(self) -> None: - self.assertEqual(_parse(OLD_ID_FEED)[0]["source"], - "arxiv:astro-ph/9403003v1") - - def test_a_new_style_id_is_unaffected(self) -> None: - self.assertEqual(_parse(OLD_ID_FEED)[1]["source"], "arxiv:2001.00009v1") - - def test_the_slashed_id_still_normalises_for_identity(self) -> None: - # Everything downstream was already built for the slash; only the parser - # never produced one. This is the join that keeps held-out bookkeeping - # working now that it does. - self.assertEqual(fetch._bare("astro-ph/9403003v1"), "astro-ph_9403003") - - -class AuthorListTest(unittest.TestCase): - """Team size comes from the API, never from counting `\author` in LaTeX.""" - - def test_authors_are_extracted_in_order(self) -> None: - self.assertEqual(_parse(OLD_ID_FEED)[0]["authors"], - ["Ian P. Dell'Antonio", "J. Anthony Tyson"]) - - def test_an_entry_with_no_authors_yields_an_empty_list(self) -> None: - self.assertEqual(_parse(OLD_ID_FEED)[1]["authors"], []) - - def test_the_abstract_writer_drops_the_list(self) -> None: - # The abstract bank is a prose corpus that other tools read as text. - with tempfile.TemporaryDirectory() as name: - tmp = Path(name) - real = fetch.fetch_page - calls = {"n": 0} - - def one_page(*a, **k): - calls["n"] += 1 - return [{"section": "abstract", "text": "word " * 60, - "source": "arxiv:2001.00001", "year": 2020, - "published": "2020-01-01", "updated": "2020-01-01", - "journal_ref": None, "journal": None, - "authors": ["A. Author"], "doi": None} - ] if calls["n"] == 1 else [] - - fetch.fetch_page = one_page - try: - with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): - fetch.main(["--field", "wgl", "--profile-root", str(tmp), - "--query-set", "wl", "--out-name", "a.jsonl", - "--per-query", "100"]) - finally: - fetch.fetch_page = real - record = json.loads( - (tmp / "wgl" / "a.jsonl").read_text(encoding="utf-8").splitlines()[0]) - self.assertNotIn("authors", record) - - -class AuthorQueryFormTest(unittest.TestCase): - """`Surname_Initial` is the listing URL's format, not the search API's. - - Every one of the ten author queries carried that shape until 2026-08-26 and - returned exactly 0 records, so the `broad` set was silently ten queries - short while its comment claimed they broadened the corpus. A dead query - leaves no trace in the output, which is why this is a test and not a note. - """ - - def test_no_author_query_uses_the_dead_underscore_initial_form(self) -> None: - import re as _re - dead = _re.compile(r"au:[A-Za-z]+_[A-Za-z](?:\b|$)") - for query in fetch.AUTHOR_QUERIES: - self.assertIsNone(dead.search(query), f"dead listing-URL form: {query}") - - def test_every_author_query_quotes_its_name(self) -> None: - for query in fetch.AUTHOR_QUERIES: - self.assertIn('au:"', query, f"unquoted author term: {query}") - - def test_surname_only_queries_are_scoped_to_the_field(self) -> None: - # `au:` matches a SURNAME, and a surname is not a person: unscoped, - # "Dell'Antonio" also returns a mathematical physicist's math-ph work. - for query in fetch.AUTHOR_QUERIES: - self.assertTrue("cat:" in query or "abs:" in query, - f"unscoped author query: {query}") - - -class AuthorQueryConstructionTest(unittest.TestCase): - """`--author` takes a name OR a whole query, and the difference matters. - - `au:` matches a surname in every archive, not just the field's, and the - identity filter cannot fix that: 32 of 100 `au:"Kaiser, N"` papers are - nuclear theory under the SAME spelling of the name an `--author-is` regex - would match. Scoping has to live in the query, so a value that already - carries a query operator is passed through untouched. - """ - - def queries_for(self, author: str) -> list[str]: - seen: list[str] = [] - real = fetch.fetch_page - - def capture(query, *a, **k): - seen.append(query) - return [] - - fetch.fetch_page = capture - try: - with tempfile.TemporaryDirectory() as name: - args = types.SimpleNamespace( - author=author, author_is="", max_authors=0, - profile_root=Path(name), field="wgl", per_query=1, page=1, - start_at=0, date_lo="201001010000", date_hi="202112312359", - sleep=0) - with redirect_stderr(io.StringIO()): - fetch._candidate_ids(args, {"x"}, None) - finally: - fetch.fetch_page = real - return seen - - def test_a_bare_name_is_quoted(self) -> None: - self.assertEqual(self.queries_for("Dell'Antonio"), ['au:"Dell\'Antonio"']) - - def test_a_whole_query_passes_through_untouched(self) -> None: - whole = 'au:"Kaiser, N" AND cat:astro-ph*' - self.assertEqual(self.queries_for(whole), [whole]) - - def test_no_author_falls_back_to_the_topic_sweep(self) -> None: - self.assertEqual(self.queries_for(""), list(fetch.FULLTEXT_QUERIES)) diff --git a/tests/test_fetch_arxiv_sweeps.py b/tests/test_fetch_arxiv_sweeps.py new file mode 100644 index 0000000..66e6cd9 --- /dev/null +++ b/tests/test_fetch_arxiv_sweeps.py @@ -0,0 +1,379 @@ +"""fetch_arxiv_abstracts: the sweep paths (audit 2026-09-27, D1-D5, D12, D13, D24). + +The parser, backoff, vintage, resume and held-out tests are in +`test_fetch_arxiv_abstracts.py`; the harness both files share is +`_fetch_harness.py`. +""" + +from __future__ import annotations + +import gzip +import io +import tarfile +import tempfile +import unittest +from pathlib import Path + +from _fetch_harness import (ARXIV_NS, ATOM_NS, LENSING, SOURCE, _bank, _candidate_args, _fulltext, + _offline, _pages_once, _record, _sweep, _throttle, _write_bank, fetch) + + +OLD_ID_FEED = f""" + + http://arxiv.org/abs/astro-ph/9403003v1 + 1994-03-01T00:00:00Z + 1994-03-01T00:00:00Z + {' word' * 60} + Ian P. Dell'Antonio + J. Anthony Tyson + + + http://arxiv.org/abs/2001.00009v1 + 2020-01-01T00:00:00Z + 2020-01-01T00:00:00Z + {' word' * 60} + +""" + + +def _parse(feed: str) -> list[dict]: + real = fetch.urlopen_backoff + fetch.urlopen_backoff = lambda *a, **k: feed.encode("utf-8") + try: + return fetch.fetch_page("q", 0, 10, "199001010000", "202512312359") + finally: + fetch.urlopen_backoff = real + + +class OldStyleIdTest(unittest.TestCase): + """An `archive/YYMMNNN` id must keep its archive. + + The e-print endpoint 404s without it, and the failure is silent in the worst + way: the sweep reports `failed=N` and carries on, so the corpus is short by + however many old papers the query returned. Measured on a live 1990-2021 + author sweep before the fix, 7 of 19 candidates were lost this way. + """ + + def test_the_archive_prefix_survives(self) -> None: + self.assertEqual(_parse(OLD_ID_FEED)[0]["source"], + "arxiv:astro-ph/9403003v1") + + def test_a_new_style_id_is_unaffected(self) -> None: + self.assertEqual(_parse(OLD_ID_FEED)[1]["source"], "arxiv:2001.00009v1") + + def test_the_slashed_id_still_normalises_for_identity(self) -> None: + # Everything downstream was already built for the slash; only the parser + # never produced one. This is the join that keeps held-out bookkeeping + # working now that it does. + self.assertEqual(fetch._bare("astro-ph/9403003v1"), "astro-ph_9403003") + + +class AuthorListTest(unittest.TestCase): + """Team size comes from the API, never from counting `\author` in LaTeX.""" + + def test_authors_are_extracted_in_order(self) -> None: + self.assertEqual(_parse(OLD_ID_FEED)[0]["authors"], + ["Ian P. Dell'Antonio", "J. Anthony Tyson"]) + + def test_an_entry_with_no_authors_yields_an_empty_list(self) -> None: + self.assertEqual(_parse(OLD_ID_FEED)[1]["authors"], []) + + def test_the_abstract_writer_drops_the_list(self) -> None: + # The abstract bank is a prose corpus that other tools read as text. + with tempfile.TemporaryDirectory() as name: + record = {**_record("arxiv:2001.00001", "2020-01-01", None), "authors": ["A. Author"]} + _sweep(Path(name), ["--query-set", "wl", "--out-name", "a.jsonl", + "--per-query", "100"], [record]) + self.assertNotIn("authors", _bank(Path(name) / "wgl" / "a.jsonl")[0]) + + +class AuthorQueryFormTest(unittest.TestCase): + """`Surname_Initial` is the listing URL's format, not the search API's. + + Every one of the ten author queries carried that shape until 2026-08-26 and + returned exactly 0 records, so the `broad` set was silently ten queries + short while its comment claimed they broadened the corpus. A dead query + leaves no trace in the output, which is why this is a test and not a note. + """ + + def test_no_author_query_uses_the_dead_underscore_initial_form(self) -> None: + import re as _re + dead = _re.compile(r"au:[A-Za-z]+_[A-Za-z](?:\b|$)") + for query in fetch.AUTHOR_QUERIES: + self.assertIsNone(dead.search(query), f"dead listing-URL form: {query}") + + def test_every_author_query_quotes_its_name(self) -> None: + for query in fetch.AUTHOR_QUERIES: + self.assertIn('au:"', query, f"unquoted author term: {query}") + + def test_surname_only_queries_are_scoped_to_the_field(self) -> None: + # `au:` matches a SURNAME, and a surname is not a person: unscoped, + # "Dell'Antonio" also returns a mathematical physicist's math-ph work. + for query in fetch.AUTHOR_QUERIES: + self.assertTrue("cat:" in query or "abs:" in query, + f"unscoped author query: {query}") + + +class AuthorQueryConstructionTest(unittest.TestCase): + """`--author` takes a name OR a whole query, and the difference matters. + + `au:` matches a surname in every archive, not just the field's, and the + identity filter cannot fix that: 32 of 100 `au:"Kaiser, N"` papers are + nuclear theory under the SAME spelling of the name an `--author-is` regex + would match. Scoping has to live in the query, so a value that already + carries a query operator is passed through untouched. + """ + + def queries_for(self, author: str) -> list[str]: + stub = _pages_once([]) + with tempfile.TemporaryDirectory() as name, _offline(Path(name), fetch_page=stub): + fetch._candidate_ids(_candidate_args(Path(name), author=author), {"x"}, None) + return [call[0] for call in stub.calls] + + def test_a_bare_name_is_quoted(self) -> None: + self.assertEqual(self.queries_for("Dell'Antonio"), ['au:"Dell\'Antonio"']) + + def test_a_whole_query_passes_through_untouched(self) -> None: + whole = 'au:"Kaiser, N" AND cat:astro-ph*' + self.assertEqual(self.queries_for(whole), [whole]) + + def test_no_author_falls_back_to_the_topic_sweep(self) -> None: + self.assertEqual(self.queries_for(""), list(fetch.FULLTEXT_QUERIES)) + + +class LocalShortcutJournalTest(unittest.TestCase): + """D1: the local full-text shortcut honours --journals and counts its drops. + + It ignored the flag, so the "refereed" breadth corpus calibrate §2 asks for + was refereed only when the abstract bank happened to be absent. + """ + + def _candidates(self, tmp: Path, journals: set[str] | None) -> tuple[list[str], str]: + _write_bank(tmp, [_record("arxiv:2001.00001v1", "2020-01-01", text=LENSING), + _record("arxiv:2001.00002v1", "2020-01-01", "MNRAS 500, 1 (2021)", + text=LENSING)]) + with _offline(tmp) as (_, err): + found, throttled = fetch._candidate_ids(_candidate_args(tmp), set(), journals) + self.assertFalse(throttled) + return found, err.getvalue() + + def test_off_journal_records_are_dropped_and_counted(self) -> None: + with tempfile.TemporaryDirectory() as name: + found, err = self._candidates(Path(name), {"apj"}) + self.assertEqual(found, ["2001.00001v1"]) + self.assertIn("1 off-journal dropped", err) + + def test_no_journal_filter_keeps_both(self) -> None: + with tempfile.TemporaryDirectory() as name: + found, err = self._candidates(Path(name), None) + self.assertEqual(found, ["2001.00001v1", "2001.00002v1"]) + self.assertIn("0 off-journal dropped", err) + + +class StartAtPageSizeTest(unittest.TestCase): + """D2: a --start-at band sizes its pages from where the band began. + + With --start-at 2000 (the held-out pull's documented offset) every request + used to ask for max_results = per_query - 2000, a negative number. + """ + + def test_pages_are_sized_from_the_band_start(self) -> None: + stub = _pages_once([_record("arxiv:2001.00001v1", "2020-01-01")]) + with tempfile.TemporaryDirectory() as name, _offline(Path(name), fetch_page=stub): + fetch._candidate_ids(_candidate_args( + Path(name), query=["abs:x"], start_at=2000, per_query=150, page=100)) + self.assertEqual([(start, n) for _, start, n, *_ in stub.calls], + [(2000, 100), (2100, 50)]) + + +class FieldResolutionTest(unittest.TestCase): + """D3: omitting --field is a message and exit 2, never `Path / None`.""" + + def test_no_profile_says_pass_field(self) -> None: + with tempfile.TemporaryDirectory() as name: + status, err = _sweep(Path(name), [], field=None) + self.assertEqual(status, 2) + self.assertIn("pass --field", err) + + def test_two_profiles_say_choose(self) -> None: + with tempfile.TemporaryDirectory() as name: + for field in ("wgl", "condmat"): + (Path(name) / field).mkdir() + status, err = _sweep(Path(name), [], field=None) + self.assertEqual(status, 2) + self.assertIn("Multiple fields", err) + + def test_a_single_profile_resolves(self) -> None: + with tempfile.TemporaryDirectory() as name: + (Path(name) / "wgl").mkdir() + self.assertEqual(_sweep(Path(name), [], field=None)[0], 0) + self.assertTrue((Path(name) / "wgl" / fetch.BANK).exists()) + + def test_a_named_new_field_is_created_not_refused(self) -> None: + # calibrate §2 runs this tool before extract_style creates the profile + # directory, so a named field must not be required to exist yet. + with tempfile.TemporaryDirectory() as name: + self.assertEqual(_sweep(Path(name), [], field="condmat")[0], 0) + self.assertTrue((Path(name) / "condmat" / fetch.BANK).exists()) + + +class FulltextThrottleTest(unittest.TestCase): + """D4: a throttled full-text run reports TRUNCATED and exits 2, as the + abstract sweep does. It used to print DONE and exit 0 from either stop.""" + + def test_throttled_candidate_search_is_truncated(self) -> None: + with tempfile.TemporaryDirectory() as name: + status, log, _ = _fulltext(Path(name), [], fetch_page=lambda *a, **k: _throttle("")) + self.assertEqual(status, 2) + self.assertIn("TRUNCATED", log) + + def test_throttled_download_is_truncated(self) -> None: + pages = [_record("arxiv:2001.00001v1", "2020-01-01")] + with tempfile.TemporaryDirectory() as name: + status, log, _ = _fulltext(Path(name), [], pages, _eprint_bytes=_throttle) + self.assertEqual(status, 2) + self.assertIn("TRUNCATED", log) + self.assertNotIn("DONE", log) + + def test_a_complete_run_is_done_under_the_bare_id(self) -> None: + # D13 as well: the directory is named by the bare id, not `...v2`. + pages = [_record("arxiv:2001.00001v2", "2021-01-01")] + with tempfile.TemporaryDirectory() as name: + status, log, out = _fulltext(Path(name), [], pages, _eprint_bytes=lambda i: SOURCE) + self.assertEqual((status, sorted(p.name for p in out.iterdir())), (0, ["2001.00001"])) + self.assertIn("DONE kept=1", log) + + +class FulltextVintageTest(unittest.TestCase): + """D5: --updated-before applies in full-text mode, and the abstract-only + flags are refused there instead of being accepted and ignored.""" + + def test_live_candidates_revised_after_the_cutoff_are_dropped(self) -> None: + pages = [_record("arxiv:2001.00001v3", "2023-04-01"), + _record("arxiv:2001.00002v1", "2020-01-01")] + with tempfile.TemporaryDirectory() as name: + with _offline(Path(name), fetch_page=_pages_once(pages)) as (_, err): + found, _ = fetch._candidate_ids( + _candidate_args(Path(name), updated_before="2022-11-01")) + self.assertEqual(found, ["2001.00002v1"]) + self.assertIn("1 revised on/after --updated-before", err.getvalue()) + + def test_the_local_shortcut_is_not_taken_with_a_cutoff(self) -> None: + # The bank's `updated` is a stale copy; only the live one dates what is + # downloaded, so the shortcut must yield to the API. + stub = _pages_once([]) + with tempfile.TemporaryDirectory() as name: + tmp = Path(name) + _write_bank(tmp, [_record("arxiv:2001.00001v1", "2020-01-01", text=LENSING)]) + with _offline(tmp, fetch_page=stub): + found, _ = fetch._candidate_ids(_candidate_args(tmp, updated_before="2022-11-01")) + self.assertEqual((found, len(stub.calls)), ([], len(fetch.FULLTEXT_QUERIES))) + + def test_abstract_only_flags_are_refused_with_fulltext(self) -> None: + for extra in (["--resume"], ["--query-set", "wl"], ["--out-name", "x.jsonl"], + ["--updated-before", "2022"]): + with self.subTest(extra=extra), tempfile.TemporaryDirectory() as name: + with _offline(Path(name)), self.assertRaises(SystemExit): + fetch.main(["--field", "wgl", "--fulltext", + "--fulltext-dir", "fulltext-test"] + extra) + + +def _targz(members: dict[str, str]) -> bytes: + buffer = io.BytesIO() + with tarfile.open(fileobj=buffer, mode="w:gz") as archive: + for name, text in members.items(): + data = text.encode("utf-8") + info = tarfile.TarInfo(name) + info.size = len(data) + archive.addfile(info, io.BytesIO(data)) + return buffer.getvalue() + + +class TexMembersTest(unittest.TestCase): + """D12: what counts as a LaTeX source. A document marker anywhere in a single + file (pre-1995 sources say \\documentstyle; many open with a comment), and a + tarball's colliding basenames are both kept rather than one overwriting.""" + + def test_documentstyle_and_comment_led_single_files_are_sources(self) -> None: + for text in ("\\documentstyle[12pt]{article}\n\\begin{document}x", + "% arXiv source\n\\documentclass{article}"): + with self.subTest(text=text[:14]): + self.assertEqual(fetch._tex_members(gzip.compress(text.encode())), + {"main.tex": text}) + self.assertEqual(fetch._tex_members(text.encode()), {"main.tex": text}) + + def test_text_without_a_document_marker_is_not_a_source(self) -> None: + self.assertIsNone(fetch._tex_members(b"e-print not available")) + self.assertIsNone(fetch._tex_members(gzip.compress(b"plain notes"))) + + def test_colliding_basenames_in_a_tarball_are_both_kept(self) -> None: + members = fetch._tex_members(_targz({"sections/intro.tex": "A", "appendix/intro.tex": "B", + "main.tex": "\\documentclass{article}"})) + self.assertEqual(sorted(members.values()), ["A", "B", "\\documentclass{article}"]) + self.assertEqual(len(members), 3) + + +class BareIdIdentityTest(unittest.TestCase): + """D13: one paper, one copy. Candidate dedup and the output directory key on + the bare id, so a version bump between runs does not fetch `…v2` beside `…v1`.""" + + def test_two_versions_of_one_paper_are_one_candidate(self) -> None: + pages = [_record("arxiv:2001.00001v1", "2020-01-01"), + _record("arxiv:2001.00001v2", "2021-01-01")] + with tempfile.TemporaryDirectory() as name: + with _offline(Path(name), fetch_page=_pages_once(pages)): + found, _ = fetch._candidate_ids(_candidate_args(Path(name), query=["abs:x"])) + self.assertEqual(found, ["2001.00001v1"]) + + def test_a_versioned_directory_from_an_earlier_run_counts_as_fetched(self) -> None: + downloads: list[str] = [] + with tempfile.TemporaryDirectory() as name: + old = Path(name) / "style-corpus" / "wgl" / "fulltext-test" / "2001.00001v1" + old.mkdir(parents=True) + (old / "main.tex").write_text("x", encoding="utf-8") + status, log, _ = _fulltext(Path(name), [], [_record("arxiv:2001.00001v2", "2021-01-01")], + _eprint_bytes=lambda i: downloads.append(i) or SOURCE) + self.assertEqual((status, downloads), (0, [])) + self.assertIn("kept=1", log) + + +class CustomQueryTest(unittest.TestCase): + """D24: `--query` is how a field other than weak lensing uses the fetcher. It + replaces the [WGL] built-in sets in both modes and bypasses the [WGL] keyword + shortcut, and it cannot be combined with the built-in selectors.""" + + def test_query_replaces_the_abstract_query_set(self) -> None: + stub = _pages_once([]) + with tempfile.TemporaryDirectory() as name: + _sweep(Path(name), ["--query", "cat:cond-mat.str-el", "--query", "abs:magnon"], + fetch_page=stub) + self.assertEqual([call[0] for call in stub.calls], ["cat:cond-mat.str-el", "abs:magnon"]) + + def test_query_forces_the_live_path_past_the_keyword_shortcut(self) -> None: + stub = _pages_once([]) + with tempfile.TemporaryDirectory() as name: + tmp = Path(name) + _write_bank(tmp, [_record("arxiv:2001.00001v1", "2020-01-01", text=LENSING)]) + with _offline(tmp, fetch_page=stub): + found, _ = fetch._candidate_ids(_candidate_args(tmp, query=["cat:cond-mat"])) + self.assertEqual((found, [call[0] for call in stub.calls]), ([], ["cat:cond-mat"])) + + def test_query_conflicts_are_refused(self) -> None: + for extra in (["--query-set", "wl"], + ["--fulltext", "--fulltext-dir", "fulltext-test", "--author", "X"]): + with self.subTest(extra=extra), tempfile.TemporaryDirectory() as name: + with _offline(Path(name)), self.assertRaises(SystemExit): + fetch.main(["--field", "wgl", "--query", "abs:x"] + extra) + + def test_query_over_an_existing_default_bank_needs_resume_or_out_name(self) -> None: + with tempfile.TemporaryDirectory() as name: + tmp = Path(name) + self.assertEqual(_sweep(tmp, ["--query", "abs:x"])[0], 0) # a new field's first bank + with self.assertRaises(SystemExit): # would truncate it + _sweep(tmp, ["--query", "abs:x"]) + self.assertEqual(_sweep(tmp, ["--query", "abs:x", "--resume"])[0], 0) + self.assertEqual(_sweep(tmp, ["--query", "abs:x", "--out-name", "q.jsonl"])[0], 0) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/fetch_arxiv_abstracts.py b/tools/fetch_arxiv_abstracts.py index 9bb9705..0267402 100644 --- a/tools/fetch_arxiv_abstracts.py +++ b/tools/fetch_arxiv_abstracts.py @@ -1,20 +1,20 @@ """Fetch dated arXiv corpora: abstracts, or complete LaTeX paper sources. -The default 2010--2021 interval supplies a pre-generative-AI reference set, and -broad astro-ph queries reduce dependence on one narrow topic. The date filter is -a provenance control, not proof of individual authorship. Records enter the -curated-field positive class of a compatibility task; the resulting model must -not be presented as an author detector. - -Abstract mode (default) writes -``style-profile//human_abstracts_extra.jsonl`` (JSONL records with -``section``, ``text``, ``source``, ``year``). Full-text mode (``--fulltext``) -downloads each paper's e-print LaTeX source, keeps papers with at least three -``\\section`` commands, and stores one directory per paper under -``style-corpus//fulltext-arxiv//`` so complete-document calibration +The default 2010--2021 interval supplies a pre-generative-AI reference set. The date +filter is a provenance control, not proof of individual authorship; records enter the +curated-field positive class of a compatibility task, and the model must not be +presented as an author detector. The built-in query sets, keyword filter and journal +table are ``[WGL]``: broad astro-ph queries keep the corpus from depending on one +narrow topic. Any other field passes its own ``--query`` terms, which replace them. + +Abstract mode (default) writes ``style-profile//human_abstracts_extra.jsonl`` +(JSONL records with ``section``, ``text``, ``source``, ``year``). Full-text mode +(``--fulltext``) downloads each paper's e-print LaTeX source, keeps papers with at +least three ``\\section`` commands, and stores one directory per bare arXiv id under +``style-corpus//fulltext-arxiv//``, so complete-document calibration treats each paper as ONE observation. Both outputs are local and gitignored. -Downloads are polite (>= 3 s between requests) per arXiv's rate guidance. -Network/API failures are reported explicitly. +Downloads are polite (>= 3 s between requests) per arXiv's rate guidance; failures +are reported, and a throttled run exits 2 in either mode. Run: ``python tools/fetch_arxiv_abstracts.py --field wgl --per-query 400`` ``python tools/fetch_arxiv_abstracts.py --field wgl --fulltext --max-papers 500``. @@ -33,6 +33,7 @@ import urllib.parse import urllib.request import xml.etree.ElementTree as ET +from collections import Counter from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) @@ -40,22 +41,23 @@ # One source of truth for which directory extract_style.py reads as calibration # breadth, so the held-out interlock below cannot drift away from the reader. from extract_style import REFERENCE_DIR # noqa: E402 -- same sys.path reason +from tex_assembly import RE_TEX_DOC_MARKER # noqa: E402 -- same sys.path reason REPO_ROOT = Path(__file__).resolve().parent.parent DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" API = "http://export.arxiv.org/api/query" ATOM = "{http://www.w3.org/2005/Atom}" ARXIV = "{http://arxiv.org/schemas/atom}" - -# Journal selection is done HERE, not in the API query: the API's jr: prefix -# is a loose token match, and jr:"Astronomy and Astrophysics" was observed to -# return "Research in Astronomy and Astrophysics" (a different journal). Each -# entry is (key, include, exclude) and the first match wins, so the Letters -# pattern must precede the main-journal pattern it is a substring of. The -# include forms are the literal shapes seen in live journal_ref values -# ("Astrophys. J. 934, no.2, 129 (2022)", "ApJ, 927, 101 (2022)", -# "The Astrophysical Journal Letters, Volume 924 (2022), Number 1, L3", -# "A&A 660, A114 (2022)"), not remembered canonical names. +BANK = "human_abstracts_extra.jsonl" # the abstract bank every downstream reader opens + +# [WGL] Journal keys for --journals. Selection is done HERE, not in the API query: the +# API's jr: prefix is a loose token match, and jr:"Astronomy and Astrophysics" was +# observed to return "Research in Astronomy and Astrophysics" (a different journal). +# Each entry is (key, include, exclude) and the first match wins, so the Letters +# pattern must precede the main-journal pattern it is a substring of. The include forms +# are the literal shapes seen in live journal_ref values ("Astrophys. J. 934, no.2, 129 +# (2022)", "ApJ, 927, 101 (2022)", "The Astrophysical Journal Letters, Volume 924 +# (2022), Number 1, L3", "A&A 660, A114 (2022)"), not remembered canonical names. JOURNAL_FILTERS = ( ("apjl", re.compile(r"(?i)\b(?:ApJL|ApJ\.?\s*Lett\w*|Astrophys\w*\.?\s*J\w*\.?\s*Lett\w*)"), @@ -79,7 +81,7 @@ def classify_journal(journal_ref: str | None) -> str | None: return key return None -# Breadth: lensing/cluster terms + the main astro-ph subfields. +# [WGL] Breadth: lensing/cluster terms + the main astro-ph subfields. QUERIES = [ "cat:astro-ph.CO AND abs:lensing", "cat:astro-ph.CO AND abs:cluster", @@ -96,9 +98,9 @@ def classify_journal(journal_ref: str | None) -> str | None: 'cat:astro-ph.CO AND abs:"mass map"', ] -# Target-author references: the advisor (Ian Dell'Antonio) plus widely cited -# weak-lensing / cluster-lensing authors. These records broaden the positive -# curated-field class; they do not turn the training task into authorship proof. +# [WGL] Target-author references: the advisor (Ian Dell'Antonio) plus widely +# cited weak-lensing / cluster-lensing authors. These records broaden the +# positive curated-field class; they do not turn the task into authorship proof. # # The QUERY FORM was wrong until 2026-08-26: `Surname_Initial` is the arXiv # LISTING url's format (/a/hoekstra_h_1), not the search API's `au:` syntax, so @@ -121,9 +123,9 @@ def classify_journal(journal_ref: str | None) -> str | None: 'au:"Fu, L" AND abs:lensing', ] -# Weak-lensing-only sweep, for a reference matched to THIS suite's subfield -# rather than to astro-ph at large. Used with --journals to keep only the -# refereed top-tier record; the breadth queries above stay the default. +# [WGL] Weak-lensing-only sweep (--query-set wl), for a reference matched to +# THIS suite's subfield rather than to astro-ph at large. Used with --journals +# to keep only the refereed top-tier record; the breadth set stays the default. WL_QUERIES = [ 'cat:astro-ph.CO AND abs:"weak lensing"', 'cat:astro-ph.CO AND abs:"weak gravitational lensing"', @@ -146,12 +148,8 @@ def classify_journal(journal_ref: str | None) -> str | None: class Throttled(Exception): - """arXiv answered 429 and kept answering 429 after every backoff. - - Distinct from an ordinary network hiccup because the response differs: a - hiccup is skipped and the sweep continues, whereas throttling means every - later request will fail too, so the sweep must stop and SAY it stopped. - """ + """arXiv answered 429 through every backoff. Unlike a hiccup, which is skipped, + every later request will fail too, so the sweep must stop and SAY it stopped.""" # Escalating waits, in seconds, after a 429. Exhausting them raises Throttled. @@ -159,12 +157,9 @@ class Throttled(Exception): def urlopen_backoff(request: urllib.request.Request, timeout: int) -> bytes: - """Single retry-on-429 path shared by the abstract and full-text modes. - - Both modes previously needed this and only the full-text one had it, so a - throttled abstract sweep silently produced a truncated corpus that looked - exactly like a complete one. - """ + """Single retry-on-429 path shared by both modes. Only the full-text mode + had it before, so a throttled abstract sweep silently produced a truncated + corpus that looked exactly like a complete one.""" for attempt, wait in enumerate((*BACKOFF_SCHEDULE, None)): try: with urllib.request.urlopen(request, timeout=timeout) as response: @@ -172,13 +167,11 @@ def urlopen_backoff(request: urllib.request.Request, timeout: int) -> bytes: except urllib.error.HTTPError as error: if error.code != 429 or wait is None: if error.code == 429: - raise Throttled( - f"429 after {attempt} backoffs " - f"totalling {sum(BACKOFF_SCHEDULE)} s") from error + raise Throttled(f"429 after {attempt} backoffs totalling " + f"{sum(BACKOFF_SCHEDULE)} s") from error raise - print(f"[fetch] HTTP 429; backing off {wait} s " - f"(attempt {attempt + 1}/{len(BACKOFF_SCHEDULE)})", - file=sys.stderr, flush=True) + print(f"[fetch] HTTP 429; backing off {wait} s (attempt {attempt + 1}/" + f"{len(BACKOFF_SCHEDULE)})", file=sys.stderr, flush=True) time.sleep(wait) raise Throttled("unreachable") # the None sentinel always raises above @@ -187,8 +180,7 @@ def fetch_page(query: str, start: int, n: int, date_lo: str, date_hi: str) -> li q = f"({query}) AND submittedDate:[{date_lo} TO {date_hi}]" params = urllib.parse.urlencode({ "search_query": q, "start": start, "max_results": n, - "sortBy": "submittedDate", "sortOrder": "descending", - }) + "sortBy": "submittedDate", "sortOrder": "descending"}) url = f"{API}?{params}" req = urllib.request.Request(url, headers={"User-Agent": "sci-paper-voice/0.13"}) xml = urlopen_backoff(req, timeout=60).decode("utf-8", "replace") @@ -203,36 +195,30 @@ def fetch_page(query: str, start: int, n: int, date_lo: str, date_hi: str) -> li # An old-style id is `archive/YYMMNNN` and the archive is PART of it: # the e-print endpoint 404s without it (7 of 19 in a measured sweep). arid = aid.split("/abs/", 1)[-1] if "/abs/" in aid else aid.rsplit("/", 1)[-1] - # `published` dates v1; `summary` is the LATEST version's abstract, so - # submittedDate alone does not date the text. Measured on a live - # 2010-2021 weak-lensing page, 11 of 12 entries had updated > published - # and two landed after 2022-11 — i.e. after the text could have been - # revised with a public LLM. `updated` is what dates the abstract. + # `published` dates v1; `summary` is the LATEST version's abstract, so `updated` + # dates the text. On a live 2010-2021 weak-lensing page, 11 of 12 entries had + # updated > published and two landed after 2022-11, i.e. after a public LLM existed. updated = e.findtext(f"{ATOM}updated") or "" - # journal_ref and doi live in the arXiv namespace, not Atom's, and only - # ~37% of entries carry a journal_ref at all (measured on a live WL - # query), so any journal-restricted run has to over-fetch. + # journal_ref and doi live in the arXiv namespace, not Atom's; only ~37% of + # entries carry one (measured on a live WL query), so journal runs over-fetch. journal_ref = e.findtext(f"{ARXIV}journal_ref") if journal_ref: journal_ref = " ".join(journal_ref.split()) - # Team size, for the full-text author sweep's --max-authors. Only the - # API carries a clean list: counting `\author` in the LaTeX picks up - # template placeholder lines, and a live AASTeX source in this corpus - # has one ("\author[xxxx-xxxx-xxxx-xxxx]{Author Name}"). + # Team size for --max-authors. Only the API carries a clean list: + # counting `\author` in the LaTeX picks up template placeholders, and a + # live AASTeX source here has one ("\author[xxxx-xxxx-xxxx-xxxx]{Author Name}"). authors = [(person.findtext(f"{ATOM}name") or "").strip() for person in e.findall(f"{ATOM}author")] if text and len(text.split()) >= 40: - out.append({"section": "abstract", "text": text, - "source": f"arxiv:{arid}", "year": year, - "published": pub[:10], "updated": updated[:10], - "journal_ref": journal_ref, - "journal": classify_journal(journal_ref), + out.append({"section": "abstract", "text": text, "source": f"arxiv:{arid}", + "year": year, "published": pub[:10], "updated": updated[:10], + "journal_ref": journal_ref, "journal": classify_journal(journal_ref), "authors": [name for name in authors if name], "doi": e.findtext(f"{ARXIV}doi")}) return out -# Full-text mode: field-specific queries give the closest genre match for the +# [WGL] Full-text mode: subfield queries give the closest genre match for the # complete-document (dispersion) reference; the broad categories stay abstract-only. FULLTEXT_QUERIES = [ 'cat:astro-ph.CO AND abs:"weak lensing"', @@ -259,9 +245,11 @@ def _eprint_bytes(arxiv_id: str) -> bytes: def _tex_members(raw: bytes) -> dict[str, str] | None: """Extract .tex files from an e-print payload; None if no LaTeX source. - e-prints arrive as gzipped tarballs, gzipped single .tex files, or bare - PDFs (no source). Tar extraction is member-filtered in memory, so no - archive paths ever touch the filesystem. + e-prints arrive as gzipped tarballs, gzipped single .tex files, or bare PDFs + (no source). Tar extraction is member-filtered in memory, so no archive paths + ever touch the filesystem. A single file is a source when `RE_TEX_DOC_MARKER` + matches ANYWHERE: pre-1995 sources say `\\documentstyle`, and many open with + a comment line rather than the marker. """ if raw[:4] == b"%PDF": return None @@ -276,6 +264,8 @@ def _tex_members(raw: bytes) -> dict[str, str] | None: if handle is None: continue name = Path(member.name).name # strip any archive paths + if name in out: # sections/intro.tex vs appendix/intro.tex: keep both + name = "__".join(Path(member.name).parts) out[name] = handle.read().decode("utf-8", "replace") return out or None except tarfile.ReadError: @@ -283,16 +273,15 @@ def _tex_members(raw: bytes) -> dict[str, str] | None: text = gzip.decompress(raw).decode("utf-8", "replace") except OSError: return None - return {"main.tex": text} if "\\documentclass" in text else None - if raw.lstrip()[:14].startswith(b"\\documentclass"): - return {"main.tex": raw.decode("utf-8", "replace")} - return None + return {"main.tex": text} if RE_TEX_DOC_MARKER.search(text) else None + text = raw.decode("utf-8", "replace") + return {"main.tex": text} if RE_TEX_DOC_MARKER.search(text) else None +# [WGL] Keyword filter of the local full-text shortcut; --query bypasses it. _FIELD_RELEVANCE_RE = re.compile( r"(?i)\b(lensing|shear|convergence|aperture mass|mass map|cluster)\b") - RE_ARXIV_ID = re.compile(r"(?:\d{4}\.\d{4,5}|[a-z-]+(?:\.[A-Z]{2})?_?\d{7})") @@ -304,18 +293,17 @@ def _bare(arxiv_id: str) -> str: def known_calibration_ids(args) -> set[str]: """Every arXiv id whose text already feeds a calibration bank. - A held-out measurement is only held out if the paper contributed nothing to - what the axes were calibrated on. `register_lexicon.json` and - `salience_baseline.json` are both built from `exemplar_paragraphs.jsonl` - (the weighted tiers plus the unweighted `fulltext-arxiv/` breadth pull) and - `human_abstracts_extra.jsonl`, so an id reachable from any of those three - places is disqualified. This matters at the observed scale rather than in - principle: `deai_register.RARE_DF_RATE` is 1e-4, which on 41,559 passages - puts the foreign-term threshold at four passages, so one paper's own - contribution is enough to suppress its own flags. + A held-out measurement is only held out if the paper contributed nothing to what + the axes were calibrated on. `register_lexicon.json` and `salience_baseline.json` + are both built from `exemplar_paragraphs.jsonl` (the weighted tiers plus the + unweighted `fulltext-arxiv/` breadth pull) and `human_abstracts_extra.jsonl`, so + an id reachable from any of those three places is disqualified. This matters at + the observed scale, not only in principle: `deai_register.RARE_DF_RATE` = 1e-4 on + 41,559 passages puts the foreign-term threshold at four passages, so one paper's + own contribution is enough to suppress its own flags. """ known: set[str] = set() - bank = args.profile_root / args.field / "human_abstracts_extra.jsonl" + bank = args.profile_root / args.field / BANK if bank.exists(): with bank.open(encoding="utf-8") as handle: for line in handle: @@ -339,41 +327,40 @@ def known_calibration_ids(args) -> set[str]: def _candidate_ids(args, exclude: set[str] | None = None, - journals: set[str] | None = None) -> list[str]: - """Candidate arXiv IDs for full-text download. - - Prefer the already-fetched local abstract corpus (no query-API traffic, - which is rate-limited after a bulk abstract run): filter its records by - field-relevant abstract keywords. Fall back to live queries only when the - local corpus is absent. - - `exclude` disqualifies ids by bare form. When it is supplied the local - shortcut is skipped outright rather than filtered: that shortcut reads - `human_abstracts_extra.jsonl`, which is one of the banks `exclude` is built - from, so every id it can offer is already disqualified. `journals` keeps - only records whose `journal_ref` classifies into one of the given keys, - which is what makes the resulting set *refereed* rather than merely - on-arXiv -- the live path carries `journal_ref`, so this is only available - there. - - `--author` replaces the topic queries with one author query and, like - `exclude`, forces the live path. `--max-authors` then bounds team size from - the API's author list, before any 3 s e-print download is spent. + journals: set[str] | None = None) -> tuple[list[str], bool]: + """Candidate arXiv IDs for full-text download, plus whether arXiv throttled the + search (the caller then reports TRUNCATED and exits 2, never DONE). + + Prefer the already-fetched local abstract corpus (no query-API traffic, which is + rate-limited after a bulk abstract run), filtered by the [WGL] abstract keywords + and by `journals` -- the key set that makes the result *refereed* rather than + merely on-arXiv. Both paths carry `journal`; the local copy can only be stale in + the safe direction (a paper published since the sweep is dropped and counted, + never wrongly kept). Five things force the live path: `exclude` (bare ids) is + built from the very bank the shortcut reads, so all it could offer is already + disqualified; `--author` names WHO and the shortcut only answers WHAT-about; + `--query` is the caller's own term set, which the [WGL] keyword filter cannot + stand in for; `--updated-before` dates the text that will be DOWNLOADED, and only + the live `updated` sees a revision made since the sweep; `--author-is` and + `--max-authors` need the API's author list, which the bank does not keep. Ids + are deduplicated on their bare form, one entry per paper. """ exclude = exclude or set() # A bare name is quoted; a value already carrying a query operator passes # through, because `au:` alone is not field-scoped and often must be: 32 of # 100 `au:"Kaiser, N"` papers are nuclear theory, under the same spelling. author = args.author if ":" in args.author else f'au:"{args.author}"' - queries = [author] if args.author else FULLTEXT_QUERIES + queries = [author] if args.author else (args.query or FULLTEXT_QUERIES) max_authors = getattr(args, "max_authors", 0) or 0 - jsonl = args.profile_root / args.field / "human_abstracts_extra.jsonl" - # `--author` names WHO; the local shortcut answers WHAT-about. It filters the - # abstract bank on field keywords and never looks at an author, so taking it - # for an author sweep would quietly return the wrong people's papers. - if jsonl.exists() and not exclude and not args.author: - candidates: list[str] = [] - seen: set[str] = set() + identity = (re.compile(getattr(args, "author_is", "") or "", re.IGNORECASE) + if getattr(args, "author_is", "") else None) + jsonl = args.profile_root / args.field / BANK + candidates: list[str] = [] + seen: set[str] = set() + dropped_known = dropped_journal = dropped_team = dropped_revised = 0 + dropped_unknown_team = dropped_identity = 0 + if jsonl.exists() and not (exclude or args.author or args.query + or args.updated_before or max_authors or identity): with jsonl.open(encoding="utf-8") as handle: for line in handle: line = line.strip() @@ -382,46 +369,50 @@ def _candidate_ids(args, exclude: set[str] | None = None, record = json.loads(line) if not _FIELD_RELEVANCE_RE.search(record.get("text", "")): continue + if journals and record.get("journal") not in journals: + dropped_journal += 1 + continue arxiv_id = str(record.get("source", "")).removeprefix("arxiv:") - if arxiv_id and arxiv_id not in seen: - seen.add(arxiv_id) + if arxiv_id and _bare(arxiv_id) not in seen: + seen.add(_bare(arxiv_id)) candidates.append(arxiv_id) - print(f"[fulltext] {len(candidates)} field-relevant candidates from " - f"local {jsonl.name}", file=sys.stderr) - return candidates - candidates = [] - seen = set() - identity = (re.compile(getattr(args, "author_is", "") or "", re.IGNORECASE) - if getattr(args, "author_is", "") else None) - dropped_known = dropped_journal = dropped_team = 0 - dropped_unknown_team = dropped_identity = 0 + print(f"[fulltext] {len(candidates)} field-relevant candidates from local " + f"{jsonl.name} ({dropped_journal} off-journal dropped)", file=sys.stderr) + return candidates, False + throttled = False for query in queries: for start in range(args.start_at, args.start_at + args.per_query, args.page): try: + # Sized from where the band BEGAN: --start-at never asks for max_results < 0. page = fetch_page(query, start, - min(args.page, args.per_query - start), + min(args.page, args.start_at + args.per_query - start), args.date_lo, args.date_hi) except Throttled as error: - print(f"[fulltext] THROTTLED building candidates: {error}; " - f"proceeding with {len(candidates)} found so far", - file=sys.stderr) - return candidates + print(f"[fulltext] THROTTLED building candidates: {error}; proceeding " + f"with {len(candidates)} found so far", file=sys.stderr) + throttled = True + break except Exception as error: # network/API hiccup: report, keep going - print(f"[fulltext] {query!r} start={start} error: {error}", - file=sys.stderr) + print(f"[fulltext] {query!r} start={start} error: {error}", file=sys.stderr) page = [] for record in page: arxiv_id = record["source"].removeprefix("arxiv:") - if not arxiv_id or arxiv_id in seen: + bare = _bare(arxiv_id) + if not arxiv_id or bare in seen: continue - seen.add(arxiv_id) - if _bare(arxiv_id) in exclude: + seen.add(bare) + if bare in exclude: dropped_known += 1 continue if journals and record.get("journal") not in journals: dropped_journal += 1 continue + # No `updated` cannot be shown to predate the cutoff; drop, do not assume clean. + if args.updated_before and not ( + record.get("updated") and record["updated"] < args.updated_before): + dropped_revised += 1 + continue team = record.get("authors") or [] # `au:` matches a SURNAME, and a surname is not a person, so the # identity test is separate from the search term rather than @@ -430,10 +421,9 @@ def _candidate_ids(args, exclude: set[str] | None = None, dropped_identity += 1 continue if max_authors: - # An unlisted team cannot be SHOWN to be within the limit, - # so it is dropped and counted rather than assumed small -- - # the same reading the --updated-before filter applies to a - # record with no `updated` date. + # An unlisted team cannot be SHOWN to be within the limit, so + # it is dropped and counted rather than assumed small -- the + # reading --updated-before applies to a record with no date. if not team: dropped_unknown_team += 1 continue @@ -444,13 +434,15 @@ def _candidate_ids(args, exclude: set[str] | None = None, if not page: break time.sleep(args.sleep) - print(f"[fulltext] {len(candidates)} candidate papers from " - f"{len(queries)} live queries " - f"(dropped {dropped_known} already-calibrated, " - f"{dropped_journal} off-journal, {dropped_identity} not the named " - f"author, {dropped_team} over {max_authors or '-'} authors, " - f"{dropped_unknown_team} with no author list)", file=sys.stderr) - return candidates + if throttled: + break + print(f"[fulltext] {len(candidates)} candidate papers from {len(queries)} live " + f"queries (dropped {dropped_known} already-calibrated, {dropped_journal} " + f"off-journal, {dropped_revised} revised on/after --updated-before, " + f"{dropped_identity} not the named author, {dropped_team} over " + f"{max_authors or '-'} authors, {dropped_unknown_team} with no author list)", + file=sys.stderr) + return candidates, throttled def fetch_fulltext(args) -> int: @@ -461,33 +453,32 @@ def fetch_fulltext(args) -> int: if exclude: print(f"[fulltext] holding out: {len(exclude)} arXiv ids already feed a " f"calibration bank", file=sys.stderr) - candidates = _candidate_ids(args, exclude, journals) + candidates, throttled = _candidate_ids(args, exclude, journals) kept = skipped = failed = 0 for arxiv_id in candidates: if kept >= args.max_papers: break - safe = arxiv_id.replace("/", "_") + # One directory per BARE id, so a paper that gained a version between runs + # is not fetched beside its old copy (pre-2026-09 runs named it with the version). + safe = _bare(arxiv_id) paper_dir = out_root / safe - if paper_dir.exists() and any(paper_dir.glob("*.tex")): + if any(paper_dir.glob("*.tex")) or any( + any(old.glob("*.tex")) for old in out_root.glob(f"{safe}v[0-9]*")): kept += 1 # resumable: already fetched and validated continue try: raw = _eprint_bytes(arxiv_id) except Throttled as error: # Backoff is exhausted inside urlopen_backoff; today's budget is - # spent, so stop cleanly. The per-paper directories make this - # resumable on a later run. - print(f"[fulltext] {error}; stopping with kept={kept} " - f"(rerun later to resume)", file=sys.stderr) + # spent. The per-paper directories make a later run resume here. + print(f"[fulltext] {error}; stopping with kept={kept}", file=sys.stderr) + throttled = True break - except urllib.error.HTTPError as error: - print(f"[fulltext] {arxiv_id}: HTTP {error.code}", file=sys.stderr) - failed += 1 - time.sleep(args.sleep) - continue except Exception as error: # per-paper failure must not kill the sweep - print(f"[fulltext] {arxiv_id}: download error: {error}", file=sys.stderr) + code = getattr(error, "code", None) # an HTTPError names its status + what = f"HTTP {code}" if code else f"download error: {error}" + print(f"[fulltext] {arxiv_id}: {what}", file=sys.stderr) failed += 1 time.sleep(args.sleep) continue @@ -504,14 +495,17 @@ def fetch_fulltext(args) -> int: continue paper_dir.mkdir(parents=True, exist_ok=True) for name in sorted(members): - (paper_dir / Path(name).name).write_text(members[name], - encoding="utf-8") + (paper_dir / Path(name).name).write_text(members[name], encoding="utf-8") kept += 1 if kept % 25 == 0: print(f"[fulltext] kept={kept} skipped={skipped} failed={failed}", file=sys.stderr, flush=True) - print(f"[fulltext] DONE kept={kept} skipped={skipped} failed={failed} " - f"-> {out_root}") + print(f"[fulltext] {'TRUNCATED' if throttled else 'DONE'} kept={kept} " + f"skipped={skipped} failed={failed} -> {out_root}") + if throttled: # exit 2, as the abstract sweep does: a run cut short is not exhaustive + print("[fulltext] TRUNCATED — arXiv throttled the run; rerun later to resume, " + "papers already kept are valid.", file=sys.stderr) + return 2 return 0 @@ -523,24 +517,28 @@ def main(argv: list[str] | None = None) -> int: p.add_argument("--date-lo", default="201001010000") p.add_argument("--date-hi", default="202112312359") p.add_argument("--sleep", type=float, default=3.0) + p.add_argument("--query", action="append", default=[], metavar="TERM", + help="an arXiv search_query term, repeatable. Replaces the [WGL] " + "built-in query sets in both modes and, in full-text mode, " + "the [WGL] keyword shortcut over the local abstract bank: a " + "field other than weak lensing must pass its own terms. Over " + "an existing default bank, pass --resume or --out-name.") p.add_argument("--fulltext", action="store_true", help="download complete LaTeX paper sources (one directory " "per paper) instead of abstracts") p.add_argument("--max-papers", type=int, default=300, help="full-text mode: stop after this many kept papers") p.add_argument("--fulltext-dir", default=REFERENCE_DIR, - help="full-text mode: directory name under " - f"style-corpus// (default {REFERENCE_DIR!r}, " - "which extract_style.py reads as breadth calibration). " - "Any other fulltext-* name is held out of calibration " - "by construction and is what an evaluation set needs.") + help="full-text mode: directory name under style-corpus// " + f"(default {REFERENCE_DIR!r}, which extract_style.py reads as " + "breadth calibration). Any other fulltext-* name is held out of " + "calibration by construction and is what an evaluation set needs.") p.add_argument("--start-at", type=int, default=0, - help="full-text mode: first result offset per query. " - "Results come back newest-first, so an existing corpus " - "occupies a contiguous shallow band: measured on " - "'cat:astro-ph.CO AND abs:cluster', offsets 0-2000 were " - "100%% already-calibrated and 2000+ were ~85%% new. " - "Skipping that band is what makes a held-out sweep " + help="full-text mode: first result offset per query. Results " + "are newest-first, so an existing corpus is a contiguous " + "shallow band: on 'cat:astro-ph.CO AND abs:cluster', " + "offsets 0-2000 were 100%% already-calibrated and 2000+ " + "~85%% new; skipping the band makes a held-out sweep " "affordable at 3 s per request.") p.add_argument("--author", default="", help="full-text mode: fetch one author's papers instead of " @@ -554,56 +552,70 @@ def main(argv: list[str] | None = None) -> int: "many authors (0 = no limit). A paper the API lists no " "authors for is dropped, not assumed small.") p.add_argument("--exclude-known", action="store_true", - help="full-text mode: drop every candidate whose text " - "already feeds a calibration bank (the abstract corpus, " - "fulltext-*/ pulls, or a tier filename's arXiv id), so " - "the result is a genuine held-out set") + help="full-text mode: drop every candidate whose text already feeds " + "a calibration bank (the abstract corpus, fulltext-*/ pulls, or " + "a tier filename's arXiv id): a genuine held-out set") p.add_argument("--query-set", choices=sorted(QUERY_SETS), default="broad", - help="'broad' = astro-ph breadth + authoritative authors " - "(default); 'wl' = weak-lensing subfield only") + help="[WGL] built-in set: 'broad' = astro-ph breadth + authoritative " + "authors (default); 'wl' = weak-lensing subfield only") p.add_argument("--journals", default="", - help="comma-separated journal keys to keep " + help="[WGL] comma-separated journal keys to keep " f"({', '.join(k for k, _, _ in JOURNAL_FILTERS)}); " "empty keeps every record, refereed or not") - p.add_argument("--out-name", default="human_abstracts_extra.jsonl", - help="output filename under the field profile directory. " - "A journal-restricted or subfield run MUST pass a " - "different name: the writer truncates its target, so " - "reusing the default would destroy the broad corpus.") + p.add_argument("--out-name", default=BANK, + help="output filename under the field profile directory. A " + "journal-restricted, subfield or --query run MUST pass a " + "different name (--query may --resume instead): the writer " + "truncates its target, so the default would lose the broad corpus.") p.add_argument("--updated-before", default="", help="YYYY-MM-DD; drop records whose LATEST arXiv version " "is dated on or after this. --date-hi bounds the v1 " - "submission, which does not date the returned " - "abstract; only this bounds the text itself.") + "submission, which does not date the returned text; " + "only this bounds the text itself, in either mode.") p.add_argument("--resume", action="store_true", - help="seed from the existing --out-name file and keep its " - "records, so a run cut short by rate limiting is " - "extended rather than replaced") + help="seed from the existing --out-name file and keep its records, so " + "a run cut short by rate limiting is extended rather than replaced") args = p.parse_args(argv) + try: + # A named field is taken as given: this tool is what first populates + # style-profile//, so requiring the directory would refuse a new + # field. An OMITTED one is resolved: a message and exit 2, not a TypeError. + args.field = args.field or cli_common.resolve_field( + None, args.profile_root, tool="fetch_arxiv_abstracts", + empty_hint="pass --field ") + except SystemExit as error: + print(error, file=sys.stderr) + return 2 if args.sleep < 3.0: p.error("--sleep must be >= 3 seconds (arXiv rate guidance)") + if args.updated_before and not re.fullmatch(r"\d{4}-\d{2}-\d{2}", args.updated_before): + p.error("--updated-before must be YYYY-MM-DD") # Validated for BOTH modes: full-text mode used to skip this block entirely, # so `--fulltext --journals apjjl` silently kept nothing rather than failing. wanted = {k.strip().lower() for k in args.journals.split(",") if k.strip()} known = {key for key, _, _ in JOURNAL_FILTERS} if wanted - known: p.error(f"unknown journal key(s): {sorted(wanted - known)}; known: {sorted(known)}") + # A flag the chosen mode accepts but would never read is refused, not ignored. + if args.query and (args.query_set != "broad" or args.author): + p.error("--query replaces the built-in query set; drop --query-set/--author") if (args.author or args.max_authors or args.author_is) and not args.fulltext: p.error("--author/--author-is/--max-authors select whole papers, which " "only --fulltext downloads; the abstract sweep has --query-set") + if args.fulltext and (args.query_set != "broad" or args.resume or args.out_name != BANK): + p.error("--query-set/--out-name/--resume shape the abstract sweep, which " + "--fulltext does not run; full-text mode resumes by itself") if args.fulltext: if "/" in args.fulltext_dir or "\\" in args.fulltext_dir: p.error("--fulltext-dir must be a bare directory name") if args.author and args.fulltext_dir == REFERENCE_DIR: p.error(f"--author writes a population you intend to MEASURE, but " - f"--fulltext-dir {REFERENCE_DIR!r} is the directory " - f"extract_style.py reads as calibration breadth. Papers in " - f"a calibration bank have their own vocabulary in the " - f"denominator: ~94% of register findings on an in-sample " - f"paper are suppressed by that membership (EVALUATION " - f"section 17.3), so the axis would go quiet on exactly the " - f"prose you fetched it to judge. Choose another " - f"fulltext-* name.") + f"--fulltext-dir {REFERENCE_DIR!r} is what extract_style.py reads " + f"as calibration breadth. A paper in a calibration bank has its " + f"own vocabulary in the denominator: ~94% of register findings on " + f"an in-sample paper are suppressed by that membership (EVALUATION " + f"section 17.3), so the axis would go quiet on exactly the prose " + f"you fetched it to judge. Choose another fulltext-* name.") if args.exclude_known and args.fulltext_dir == REFERENCE_DIR: p.error(f"--exclude-known writes a held-out set, but " f"--fulltext-dir {REFERENCE_DIR!r} is the directory " @@ -612,24 +624,19 @@ def main(argv: list[str] | None = None) -> int: f"would stop being held out. Choose another fulltext-* name.") return fetch_fulltext(args) - if (wanted or args.query_set != "broad") and \ - args.out_name == "human_abstracts_extra.jsonl": - p.error("a journal-restricted or subfield run needs --out-name; " - "writing to human_abstracts_extra.jsonl would truncate the " - "broad corpus that the calibration baselines are built from") - field_dir = args.profile_root / args.field + if args.out_name == BANK and (wanted or args.query_set != "broad" or ( + args.query and (field_dir / BANK).exists() and not args.resume)): + p.error("a journal-restricted, subfield or --query run needs --out-name " + "(--query may --resume instead): writing to the default bank would " + "truncate the broad corpus the calibration baselines are built from") field_dir.mkdir(parents=True, exist_ok=True) - seen: set[str] = set() + seen: set[str] = set() # bare ids, so a revised paper is not banked twice recs: list[dict] = [] dropped_no_ref = dropped_other = dropped_revised = 0 - if args.updated_before and not re.fullmatch(r"\d{4}-\d{2}-\d{2}", - args.updated_before): - p.error("--updated-before must be YYYY-MM-DD") if args.resume: # Rate limiting makes a full sweep a multi-run affair, and the writer - # truncates its target, so without this a rerun would shrink the corpus - # instead of growing it. + # truncates its target: without this a rerun would shrink the corpus. existing = field_dir / args.out_name if existing.exists(): with existing.open(encoding="utf-8") as handle: @@ -640,12 +647,12 @@ def main(argv: list[str] | None = None) -> int: record = json.loads(line) # a pre-2026 bank stored year as str year = str(record.get("year") or "") record["year"] = int(year) if year.isdigit() else 0 - if record.get("source") not in seen: - seen.add(record["source"]) + if _bare(record["source"]) not in seen: + seen.add(_bare(record["source"])) recs.append(record) print(f"[fetch] resume: {len(recs)} records carried over from " f"{existing.name}", file=sys.stderr) - queries = QUERY_SETS[args.query_set] + queries = args.query or QUERY_SETS[args.query_set] throttled_at: str | None = None incomplete_queries: list[str] = [] for index, query in enumerate(queries): @@ -657,31 +664,26 @@ def main(argv: list[str] | None = None) -> int: page = fetch_page(query, start, min(args.page, args.per_query - start), args.date_lo, args.date_hi) except Throttled as e: - # Not a hiccup: every later request would fail too. Stop the - # sweep so the truncation is reported instead of being written - # out as if it were the complete corpus. - print(f"[fetch] THROTTLED on {query!r} start={start}: {e}", - file=sys.stderr) + # Not a hiccup: every later request would fail too. Stop, so the + # truncation is reported rather than written out as complete. + print(f"[fetch] THROTTLED on {query!r} start={start}: {e}", file=sys.stderr) throttled_at = f"query {index + 1}/{len(queries)} {query!r} start={start}" break except Exception as e: # A failed page and an exhausted query both used to arrive here - # as `page = []`, and the `if not page: break` below cannot tell - # them apart -- so one hiccup silently ended that query's - # pagination and the run still exited 0, reporting a short - # corpus as a complete one. Record the query as incomplete and - # stop paginating IT, without claiming the sweep succeeded. + # as `page = []`, which `if not page: break` cannot tell apart, + # so one hiccup silently ended that query and the run still + # exited 0 with a short corpus. Record IT as incomplete instead. print(f"[fetch] {query!r} start={start} error: {e}", file=sys.stderr) incomplete_queries.append(f"{query!r} start={start}: {e}") break new = 0 for r in page: - if r["source"] in seen: + if _bare(r["source"]) in seen: continue - seen.add(r["source"]) - # ISO dates compare correctly as strings; a record with no - # updated field cannot be shown to predate the cutoff, so it - # is dropped rather than assumed clean. + seen.add(_bare(r["source"])) + # ISO dates compare as strings; a record with no `updated` cannot be + # shown to predate the cutoff, so it is dropped, not assumed clean. if args.updated_before and not ( r["updated"] and r["updated"] < args.updated_before): dropped_revised += 1 @@ -693,10 +695,9 @@ def main(argv: list[str] | None = None) -> int: if r["journal"] not in wanted: dropped_other += 1 continue - # The abstract bank is a PROSE corpus -- `voice_dataset`, the - # register lexicon and the reference banks all read it as text. - # Author names are not prose, so the list stays in the full-text - # path that needs it rather than entering every record here. + # The abstract bank is a PROSE corpus (`voice_dataset`, the register + # lexicon and the reference banks all read it as text); author names + # are not prose, so the list stays in the full-text path that needs it. r.pop("authors", None) recs.append(r); new += 1; got += 1 print(f"[fetch] {query!r} start={start}: +{new} (total {len(recs)})", @@ -714,12 +715,9 @@ def main(argv: list[str] | None = None) -> int: print(f"[fetch] wrote {len(recs)} dated curated-field abstracts -> {out}") print(f"[fetch] year span: {years[0] if years else '-'}..{years[-1] if years else '-'}") if wanted: - by_journal: dict[str, int] = {} - for r in recs: - by_journal[r["journal"]] = by_journal.get(r["journal"], 0) + 1 - print(f"[fetch] journal filter {sorted(wanted)}: kept " - f"{sorted(by_journal.items())}; dropped {dropped_no_ref} with no " - f"journal_ref, {dropped_other} in other journals") + by_journal = sorted(Counter(r["journal"] for r in recs).items()) + print(f"[fetch] journal filter {sorted(wanted)}: kept {by_journal}; dropped " + f"{dropped_no_ref} with no journal_ref, {dropped_other} in other journals") if args.updated_before: print(f"[fetch] text-vintage filter updated < {args.updated_before}: " f"dropped {dropped_revised} records revised on or after it") @@ -733,11 +731,9 @@ def main(argv: list[str] | None = None) -> int: if incomplete_queries: # Same principle at query granularity: these queries stopped early on a # transient error, so the corpus is short by an unknown amount. Naming - # them and exiting 2 is the only way the caller can tell this run from - # an exhaustive one. - print(f"[fetch] INCOMPLETE — {len(incomplete_queries)} of " - f"{len(queries)} queries stopped early on a page error:", - file=sys.stderr) + # them and exiting 2 is how the caller tells this run from an exhaustive one. + print(f"[fetch] INCOMPLETE — {len(incomplete_queries)} of {len(queries)} queries " + f"stopped early on a page error:", file=sys.stderr) for entry in incomplete_queries: print(f"[fetch] {entry}", file=sys.stderr) print("[fetch] Rerun with --resume to extend; records already written " From b25f4cd137a2452fe2daf47c8d5b3a9c5d450886 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:35:49 +0000 Subject: [PATCH 15/27] docs: registry rows, skills and notes for the rewrite gate, the labelling floor and the fetcher's flags The tools registry says what rewrite_reward, label_findings, eval_findings, eval_docscale, voice_dataset, train_voice_model and fetch_arxiv_abstracts do now: bag comparison of numbers and markers, no exit 2 for a missing optional artifact, the per-cell labelling quota and its document spread, the shared 20-document floor and the operating point named per row, the encoder failure that never becomes the cache, the scope-aware unbound-name check, and the fetcher's flags holding in both modes. The de-ai and condense skills carry the gate's residual limit and its optional `--field`; the calibrate skill samples `--n 24` per cell, explains the document spread, and warns that the fetcher's built-in query sets are [WGL]. The learned-model note records the two new protected-set properties and the tests that pin them; the responsibilities table gives eval_docscale its floor and alpha rule. The classifier's closing hint no longer names the linter's hidden `--summary` flag. The release notes gain the three paragraphs for these parts. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- CHANGELOG.md | 40 +++++++++++++++++++ docs/architecture/RESPONSIBILITIES.md | 2 +- docs/architecture/evaluation/learned-model.md | 25 ++++++++++-- skills/calibrate/SKILL.md | 11 +++-- skills/condense/SKILL.md | 10 +++-- skills/de-ai/SKILL.md | 23 ++++++----- tools/README.md | 14 +++---- tools/train_ai_ism_classifier.py | 2 +- 8 files changed, 97 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9980713..804e417 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,6 +99,46 @@ command as fixed. The tricolon wrap-up needs a wrap-up. `structure_baseline`, `docstructure_baseline` and `anchoring_baseline` must be rebuilt for these to take effect; the EVALUATION figures tied to them predate the rebuild and say so. +**The rewrite gate runs on a fresh clone.** A missing profile, a missing or +unusable `voice_model.joblib`, or a missing sentence-transformers was exit 2, +so the fidelity gate could not run where the optional model had never been +trained; each now leaves its term unmeasured at weight 0 with one stderr line, +and the run ends 0 or 1 by eligibility. Numbers and the negation, causal and +comparison markers are compared as bags, so a dropped second `not` or a +repeated value replaced by another is caught and the report names the count +mismatch; a marker moved between clauses still passes and stays with the +by-hand check. The numeral tokenizer reads the LaTeX en-dash range +`0.5--1.2`, the Unicode minus and exponent sign, and no longer starts a token +inside an identifier (`M200` yielded `00`). `--field` is optional. + +**Labelling and evaluation report a thin stratum as unmeasured.** A +population with no body words, a document-scale row under the 20-document +floor, and a labelling cell short of its quota say so instead of printing a +rate, a `StatisticsError`, or a `None` formatted as a float; the floor is one +constant shared by `eval_findings` and `eval_docscale`. `label_findings --n` +is the flagged-row quota per (population, axis) cell; every finding is +collected first and each cell drawn at random with a per-document cap; +emitter failures are printed per axis instead of being read as an exhausted +corpus; pooled recall counts a passage several axes flagged once; `--field` +belongs after the subcommand. `eval_docscale` decides each flag by its +operating point's own alpha, names that point per row, and finds its tiers +under `docval/`. In voice training a failed encode never becomes the feature +cache (the language-model rows are checkpointed and the embedder retried +next run), the repeated audit splits start one seed past the primary split, a +missing numpy is one line and exit 2, and the unbound-name check is +scope-aware: it sees an import that one function bound only for itself. + +**The fetcher does what its flags say.** `--journals` applies on the local +full-text shortcut (the "refereed" breadth corpus was never restricted); +`--start-at` sizes its pages from where the band began instead of requesting +a negative page; a throttled full-text run reports TRUNCATED and exits 2 like +the abstract sweep; `--updated-before` applies in full-text mode and the +abstract-only flags are refused there instead of accepted and ignored; a +version bump no longer fetches `...v2` beside `...v1`; `\documentstyle` and +comment-led files are sources and colliding tarball basenames are both kept; +omitting `--field` is a message and exit 2; the built-in query sets are +marked `[WGL]` and `--query` serves any other field. + Tests and counts are stated at the end of this entry once every part of the release has landed. diff --git a/docs/architecture/RESPONSIBILITIES.md b/docs/architecture/RESPONSIBILITIES.md index f587b85..a7215b7 100644 --- a/docs/architecture/RESPONSIBILITIES.md +++ b/docs/architecture/RESPONSIBILITIES.md @@ -67,7 +67,7 @@ scope, not a standard-reference obligation. | `deai_provenance.py` | L4 | Label author edit depth vs a designated AI-draft ancestor from the author's own history; `unmeasured` without one. | | `deai_personal.py` | L4 | Compare a draft to the author's own prior papers (confound-free reference); `unmeasured` below three papers. | | `eval_findings.py` | evidence | Score the register, salience, cohesion, hedging and collocation axes (the last on its document novel-pair fraction, because its reference is per sentence) on a **held-out** refereed corpus that fed no calibration bank, and report the same-genre in-sample population beside it so the leakage gap stays visible. Never present the salience rate as a false-positive rate — its gate is a percentile, so a non-zero rate is its design point — and never present either rate as precision for the advice itself. Added 2026-08-27: the axes' precision had stood at `unmeasured` pending hand labels, while a refereed paper's provenance was already a label for half of the question. | -| `eval_docscale.py` | evidence | Reproduce the EVALUATION §9 table through the same `manifold_operating_point` that findings use — never a private scoring path. Label the human rate **in-sample** wherever it includes the manifold's own train and calibration documents, and never present a single-seed tail-power figure as an estimate. Added 2026-08-26: nothing bound the evidence-producing path before, and §9 accumulated headline figures produced by a 95th-percentile cut rather than by the shipped conformal rule. | +| `eval_docscale.py` | evidence | Reproduce the EVALUATION §9 table through the same `manifold_operating_point` that findings use — never a private scoring path. Label the human rate **in-sample** wherever it includes the manifold's own train and calibration documents, and never present a single-seed tail-power figure as an estimate. Added 2026-08-26: nothing bound the evidence-producing path before, and §9 accumulated headline figures produced by a 95th-percentile cut rather than by the shipped conformal rule. Report `unmeasured` for any row under the 20-document floor it shares with `eval_findings`, decide each flag by the operating point's own alpha (never a legacy percentile against the conformal alpha), and name that operating point per row. | | `verify_references.py` | QD (dimension F) | Verify every bibliography entry against the registries (CrossRef, DataCite for the prefixes CrossRef does not register, arXiv) and, with `--tex`, the assembled document's `\cite` keys against the entries; an unresolvable identifier or a cited key with no entry is an `integrity_blocker` and exit 1 (§0.1); a registry outage marks the entry `unmeasured` and the axis `degraded`, never clean; relevance is the reviewer's, not measured here. | | `label_findings.py` | evidence | Sample findings from every finding-emitting axis into a labelling sheet, re-serve a blind subset, and score precision, pooled recall and intra-rater agreement; any stratum under 20 labels is `unmeasured`; nothing here enters calibration or creates an operating point. | diff --git a/docs/architecture/evaluation/learned-model.md b/docs/architecture/evaluation/learned-model.md index 1ef2746..93dc515 100644 --- a/docs/architecture/evaluation/learned-model.md +++ b/docs/architecture/evaluation/learned-model.md @@ -205,10 +205,10 @@ loads under local scikit-learn 1.8.0 with no warnings. ## 8. Rewrite eligibility [`tools/rewrite_reward.py`](../../../tools/rewrite_reward.py) checks protected invariants before -ranking. The protected set includes numbers, units, citations, inline mathematics, -uppercase acronyms, comparison direction, negation, and causal direction. +ranking. The protected set includes numbers, units, citations, inline and displayed +mathematics (numerals collected from both), uppercase acronyms, comparison direction, negation, and causal direction. -**Three properties of the protected set, recorded rather than implied:** +**Five properties of the protected set, recorded rather than implied:** - **Display math is protected as of v0.27.0.** Both LaTeX projections drop `\begin{equation}`/`align`/`gather` bodies by design, so until v0.27.0 every @@ -235,6 +235,20 @@ uppercase acronyms, comparison direction, negation, and causal direction. measured outcome the caller acts on — preserve the original and regenerate tighter — so reporting it as an execution failure made a correct run indistinguishable from a crash. Registered in `SCIPAPER_STANDARD` §0.1. +- **Numbers and the marker categories are bags as of 2026-09-27.** Set + comparison could not see a dropped second negation or a repeated value + replaced by another; both are now caught, and the report names a count + mismatch when the candidate still carries the token fewer times. A marker + moved between clauses, or two distinct numbers exchanging places, changes + no count and passes: that is the gate's stated residual limit, covered by + the by-hand check the de-ai skill requires. +- **No optional artifact is a precondition** (2026-09-27). A missing profile, + a missing or unusable `voice_model.joblib`, or a missing sentence-transformers + leaves the corresponding term unmeasured at weight 0 — never a nominal 0.0 — + with one stderr line each; without the cosine the fidelity floor on the + advisory-reduction credit and the condensation bonus cannot be applied and + the run says so. Until then a missing bundle was exit 2 and the gate could + not run on a fresh clone. An ineligible candidate receives a combined score of negative infinity. This replaces the former relative semantic-similarity band, under which a fluent but scientifically @@ -246,7 +260,10 @@ Current tests in - a candidate preserving protected invariants remains eligible; - dropping a number makes it ineligible; - dropping a citation makes it ineligible; -- reversing a comparison makes it ineligible. +- reversing a comparison makes it ineligible; +- a dropped second negation makes it ineligible; +- a missing model, profile or embedder leaves the term unmeasured with exit 0/1 + by eligibility. Section 11 records a source-traced, proposal-only real-manuscript validation. Its manual review covers entities, scope, stance, logical dependency, and citation support, diff --git a/skills/calibrate/SKILL.md b/skills/calibrate/SKILL.md index 6bc8d68..fb2c32f 100644 --- a/skills/calibrate/SKILL.md +++ b/skills/calibrate/SKILL.md @@ -79,7 +79,10 @@ python tools/fetch_arxiv_abstracts.py --field --fulltext \ ``` `--exclude-known` is what makes the second pull disjoint from the first. Verify -the disjointness before trusting any rate computed from it. +the disjointness before trusting any rate computed from it. The fetcher's +built-in query sets, keyword filter and journal table are `[WGL]`: for any other +field pass your own `--query ` (repeatable; it replaces them) and omit +`--journals`, or the pull fills your corpus with weak-lensing papers. ## 3. Extract the profile @@ -175,7 +178,7 @@ calibration input. ```bash python tools/label_findings.py sample --field \ --population mentor=style-corpus//fulltext-mentor \ - --n 240 --out labels.jsonl + --n 24 --out labels.jsonl # fill in "label": true | false, guided by each row's own "question" python tools/label_findings.py relabel --sheet labels.jsonl --frac 0.2 \ --out recheck.jsonl @@ -196,7 +199,9 @@ Four things about the sheet decide whether the effort is worth anything: - **Stratified by axis, not only by population.** Salience fires roughly twenty times as often as register. A shared quota is spent on salience before register reaches the floor, and the register cell then reads `unmeasured` - however large the sheet is. + however large the sheet is. Within a cell the draw is spread over documents + (at most a fifth of the cell from one paper when the population allows) and + the summary prints how many documents each cell came from. - **A flag and a control ask opposite questions.** A flagged row asks whether the advisory is right. A control row asks whether it *should* have been flagged — that miss is the numerator of recall. Each row carries its own `question`; diff --git a/skills/condense/SKILL.md b/skills/condense/SKILL.md index 2f1b627..e1c07b3 100644 --- a/skills/condense/SKILL.md +++ b/skills/condense/SKILL.md @@ -120,10 +120,12 @@ For each map entry, in rank order, record `deleted` / `merged` / `kept:` shortened rewrite with `python tools/rewrite_reward.py --field --reference --original --candidates ...` — a candidate that drops a protected invariant is ineligible regardless of its - brevity; among eligible candidates the shorter wins. `--field` resolves as - in §1 step 1 (one profile auto-resolves); without a `voice_model.joblib` - the learned score is reported unmeasured and the deterministic gate still - decides eligibility. Exit 2 is invalid input, never a missing model. + brevity; among eligible candidates the shorter wins. `--field` is optional + (one field auto-detects; none or several run without a profile, the + learned score unmeasured at weight 0 with a stderr line saying so); exit 2 + means invalid input or a crash, never a missing model. A value the claim + record states twice counts twice, so a condensation keeping one copy is + ineligible: distil the record before gating. Apply each change with a minimal Edit and re-read the affected region plus every cross-reference into it. diff --git a/skills/de-ai/SKILL.md b/skills/de-ai/SKILL.md index 91e3fbb..7d4c875 100644 --- a/skills/de-ai/SKILL.md +++ b/skills/de-ai/SKILL.md @@ -346,16 +346,19 @@ findings. For each selected paragraph: `eligible=False` (fidelity failure, or longer than the original → `-inf`) cannot be selected; the tool exits 1 when NO candidate is eligible (a measured outcome, not a failure — regenerate tighter) and 2 only on invalid - input. Without a `voice_model.joblib` the learned score is reported - unmeasured and the deterministic gate still decides; the set comparison of - invariants is blind to a negation MOVED between clauses, so step 3's - hand check is where that is caught. `--allow-growth ""` only with an - author-approved justification, printed into the run record. Among - eligible candidates rank by L0 reduction, semantic fidelity, voice score, - and condensation. If none is eligible, preserve the original and - regenerate tighter — never pick the least-bad ineligible candidate. Then - verify entities, scope, stance, and logical dependencies against the - source by hand: the deterministic checker is necessary, not sufficient. + input or a crash. A missing profile, a missing or unusable + `voice_model.joblib`, or a missing sentence-transformers does not stop the + gate: that term is unmeasured at weight 0, one stderr line says which, and + ranking proceeds on the deterministic terms (the learned model is never + required). `--allow-growth ""` only with an author-approved + justification, printed into the run record. Among eligible candidates + rank by L0 reduction, semantic fidelity, voice score, and condensation. + If none is eligible, preserve the original and regenerate tighter — never + pick the least-bad ineligible candidate. Then verify entities, scope, + stance, and logical dependencies against the source by hand: the + deterministic checker is necessary, not sufficient — it compares numbers + and negation/causal/comparison markers as bags, so a marker moved between + clauses passes it; its `count mismatch` line flags only lost repeats. 4. **Re-measure.** Re-run the Pass-1 linter on the candidate in enough section context for section caps. A good rewrite introduces no new integrity blocker or L0 target, stays eligible, acts on the selected diff --git a/tools/README.md b/tools/README.md index 5438928..62deb9a 100644 --- a/tools/README.md +++ b/tools/README.md @@ -39,15 +39,15 @@ serialize evidence but do not define an independent paper verdict. | `deai_oracle.py` | Optional token-surprisal and UID analysis. | Requires transformers/model assets for measurement; compatibility `FLAG_Z` is degraded until field calibration exists. Document-level surprisal is measured (EVALUATION.md 9.8) to be weaker than the model-free manifold and adds nothing to it. | | `deai_features.py` | Reusable model-free, UID, and embedding features for document analysis and learned field-similarity models. | Without the transformers/torch runtime the UID features are `None` with `uid_status: unmeasured` (`features_vector` raises with the reason; the CLI exits 2 with a message); the corpus centroid is `None` without the `.npy` cache and needs no numpy to say so. | | `deai_voice.py` | Optional learned field-similarity triage. | Refuses bundles with drifted feature names/schema and degrades on corrupt bundles. A measured operating point gates threshold findings; an uncalibrated bundle yields rank-based triage of the lowest-scoring paragraphs, never a universal 0.5 cutoff. | -| `train_voice_model.py` | Trains the optional learned field-similarity/rewrite-ranking model and writes `voice_model_evaluation.json`. | Cache reuse requires a content/model/centroid fingerprint; featurization checkpoints every 500 rows for preemptible runs. Grouped splits recompute `corpus_cos` from training-only centroids; repeated source-group splits audit source, section, length, mathematical-marker, field-term, matched-stratum, and UID-normalization sensitivity, plus the author-labelled hard-set stratum, without creating an operating point. | -| `voice_dataset.py` | Record loading, source-family grouping, the training-only field lexicon, and the fingerprinted feature-matrix cache for the voice model. Split out of `train_voice_model.py` on 2026-08-26 (1,174 lines against a 750-line budget). | Depends on nothing in `voice_audit`; the split is one-way and a test asserts it. The split's hand-written import block left `time` and `CHECKPOINT_EVERY` unbound and the suite stayed green, because no test reached `build_features` — a whole-tools-directory AST check for unbound names now runs in `test_train_voice_model.py`. | +| `train_voice_model.py` | Trains the optional learned field-similarity/rewrite-ranking model and writes `voice_model_evaluation.json`. | Cache reuse requires a content/model/centroid fingerprint; featurization checkpoints every 500 rows for preemptible runs. Grouped splits recompute `corpus_cos` from training-only centroids; repeated source-group splits audit source, section, length, mathematical-marker, field-term, matched-stratum, and UID-normalization sensitivity, plus the author-labelled hard-set stratum, without creating an operating point. Requires `--field` to resolve (auto-detected when one profile exists); a missing numpy/scikit-learn/joblib is one line and exit 2. Repeated audit splits start one seed past the primary split. | +| `voice_dataset.py` | Record loading, source-family grouping, the training-only field lexicon, and the fingerprinted feature-matrix cache for the voice model. Split out of `train_voice_model.py` on 2026-08-26 (1,174 lines against a 750-line budget). | Depends on nothing in `voice_audit`; the split is one-way and a test asserts it. The split's hand-written import block left `time` and `CHECKPOINT_EVERY` unbound and the suite stayed green, because no test reached `build_features` — a scope-aware symbol-table check for unbound names now runs in `test_train_voice_model.py`. A run whose encoder fails while sentence-transformers imports never becomes the cache: the language-model rows are kept as a finished checkpoint and the embedder is retried next run. | | `voice_audit.py` | Held-out metrics, bootstrap AUC intervals, the section-normalized UID alternative, the author hard set, and the repeated grouped confound audits. Owns `HARDSET_AI_CATEGORIES` / `HARDSET_HUMAN_CATEGORIES`. | Fits no model and creates no operating point: it produces the evidence a bundle needs before it can leave `degraded`. `train_voice_model` re-exports every public name, guarded by a contract test. | -| `label_findings.py` | Samples findings from all five finding-emitting axes — register, salience, cohesion, hedging, collocation — into a human-labelling sheet (`sample`), re-serves a blind subset with the prior answers stripped (`relabel`), and reports precision, recall and intra-rater agreement (`score`). | The one roadmap item computation cannot close: these axes are corpus-referenced, so nothing here says whether a human agrees a given advisory is right. Built to the scheme chosen 2026-08-26 — finding-level labels, one labeller plus a blind re-label subset, and populations named with `--population NAME=DIR` so each is sampled and reported separately. Recall is **pooled over the axes**: a control row asks whether a passage should have been flagged, and a labeller who says yes has not said by which axis, so a per-axis recall would charge every axis with every other axis's misses. Calibrates nothing; any stratum under 20 labels reports `unmeasured` rather than a rate, and the intra-rater kappa is printed as the ceiling no axis can be held above. | -| `eval_findings.py` | Scores the register (thresholded and zero-hit), salience and collocation axes against provenance labels instead of hand labels (collocation on its document novel-pair fraction, since its reference is per sentence): firing rate on **held-out** refereed ApJ/ApJL/A&A papers, on the same-genre in-sample papers (leakage control), and on the `docval` machine tiers, plus the machine-over-held-out rank AUC. | Read the two rows differently. Register is an absolute rarity test, so its held-out rate **is** a false-positive rate. Salience is a percentile gate (`ADVISORY_PERCENTILE` 0.90), so a non-zero rate is the design point and the number tests calibration transfer, not defectiveness. Neither is precision for the *advice*: publication does not mean every sentence is beyond improvement, so the register figure is an upper bound. Recall stays `unmeasured` — that is `label_findings.py`. | -| `eval_docscale.py` | Reproduces the EVALUATION §9 document-scale table — human false-flag rate, per-tier 5%-tail power, and rank AUC — by scoring the corpus and every `docval` tier through `manifold_operating_point`. | The human row is **in-sample** (it includes the manifold's own train and calibration documents) and is labelled as such; the distribution-free guarantee is the split-conformal alpha, not this rate. Point estimates carry substantial seed variance (§9.4c), so a single run is not a stable figure — compare configurations, not runs. | +| `label_findings.py` | Samples findings from all five finding-emitting axes — register, salience, cohesion, hedging, collocation — into a human-labelling sheet (`sample`), re-serves a blind subset with the prior answers stripped (`relabel`), and reports precision, recall and intra-rater agreement (`score`). | The one roadmap item computation cannot close: these axes are corpus-referenced, so nothing here says whether a human agrees a given advisory is right. Built to the scheme chosen 2026-08-26 — finding-level labels, one labeller plus a blind re-label subset, and populations named with `--population NAME=DIR` so each is sampled and reported separately. Recall is **pooled over the axes**: a control row asks whether a passage should have been flagged, and a labeller who says yes has not said by which axis, so a per-axis recall would charge every axis with every other axis's misses. Calibrates nothing; any stratum under 20 labels reports `unmeasured` rather than a rate, and the intra-rater kappa is printed as the ceiling no axis can be held above. `--n` is the flagged-row quota per (population, axis) cell (default 20, the floor); every finding is collected first and each cell drawn at random with a per-document cap, the summary printing how many documents each cell came from; emitter failures are printed per axis rather than read as an exhausted corpus; pooled recall counts a passage several axes flagged once, by `(source, start_line, end_line)`. | +| `eval_findings.py` | Scores the register (thresholded and zero-hit), salience and collocation axes against provenance labels instead of hand labels (collocation on its document novel-pair fraction, since its reference is per sentence): firing rate on **held-out** refereed ApJ/ApJL/A&A papers, on the same-genre in-sample papers (leakage control), and on the `docval` machine tiers, plus the machine-over-held-out rank AUC. | Read the two rows differently. Register is an absolute rarity test, so its held-out rate **is** a false-positive rate. Salience is a percentile gate (`ADVISORY_PERCENTILE` 0.90), so a non-zero rate is the design point and the number tests calibration transfer, not defectiveness. Neither is precision for the *advice*: publication does not mean every sentence is beyond improvement, so the register figure is an upper bound. Recall stays `unmeasured` — that is `label_findings.py`. A population with no body words reports `unmeasured` rather than a rate; the 20-document floor is `eval_docscale.MIN_DOCUMENTS`. | +| `eval_docscale.py` | Reproduces the EVALUATION §9 document-scale table — human false-flag rate, per-tier tail power, and rank AUC — by scoring the corpus and every tier found under `docval/` through `manifold_operating_point`; each row names the operating point its flags were decided by and reports `unmeasured` under 20 scored documents (the floor shared with `eval_findings`). | The human row is **in-sample** (it includes the manifold's own train and calibration documents) and is labelled as such; the distribution-free guarantee is the split-conformal alpha, not this rate. Point estimates carry substantial seed variance (§9.4c), so a single run is not a stable figure — compare configurations, not runs. | | `cli_common.py` | Shared command-line preamble and field resolution: UTF-8 stdout, the `--field` / `--profile-root` / `--corpus-root` options, `resolve_field` for a tool that needs a field, `optional_field_dir` for one that can run without (one profile auto-resolves, several print a note and run without one), and `axis_main` for the calibrated axes (`--calibrate` needs an existing field and refuses to write an empty artifact; an axis flag goes through `extra_arguments`). | Holds no policy — decides no default beyond the two roots, reads no profile, emits no findings. **21 of 30 tools were retrofitted on 2026-08-26**; the sweep is recorded with its allowlist and refusals. One divergence is kept deliberately: `ai_ism_lint` returns `None` instead of raising (warning on an explicit field that does not exist), because its L0 pass must still run with no profile present; `extract_style` resolves against the corpus root where `tier-*` is not a field (expressed as `exclude_prefixes`, not a second implementation). | -| `rewrite_reward.py` | Evaluates rewrite candidates after hard scientific-fidelity eligibility. | Bidirectional: dropping OR inventing protected numbers, units, citations, math, acronyms, semantic LaTeX macros, comparison direction, negation, or causal direction makes a candidate ineligible (`-inf`). Ranking is led by L0 advisory reduction (signed, fidelity-floored) and semantic fidelity; the learned score is tie-break weight unless its bundle is measured. Its numeral tokenizer has been corrected three times for one root cause — a separator absorbed into the token: the Oxford comma, the spaced unit, and (2026-08-25) a hyphen between numerals, which made `0.5-1.2` yield `-1.2` and hard-rejected every faithful rewrite of a range. Each is pinned by a test. Exit 0 = at least one eligible candidate; 1 = every candidate ineligible (a measured outcome: preserve the original and regenerate tighter); 2 = invalid input or missing required configuration. | -| `fetch_arxiv_abstracts.py` | Fetches dated arXiv abstract corpora for controlled model evaluation/training. `--query-set wl` narrows to the weak-lensing subfield; `--journals apj,apjl,aa` keeps only those refereed venues. `--fulltext --author "Surname" --author-is --max-authors N` builds a per-author full-text population instead. | Network failures are explicit; fetched text is evidence/training data, not policy. The author path keeps the search term and the identity test separate, because `au:` matches a **surname** and a surname is not a person — `au:"Dell'Antonio"` returns an astronomer and a mathematical physicist. The abstract sweep's `AUTHOR_QUERIES` have no such filter, so each of the ten was checked once against the names it really returns: nine resolve to one person, and `au:"Schneider, P"` returned Donald P. Schneider (SDSS quasars) on 58 papers against the intended Peter Schneider's 33, so that entry now spells the given name out. That check needs the network and cannot be a test; adding an eleventh query means repeating it by hand. `--author` refuses to write into the calibration-breadth directory: a paper in a calibration bank has its own vocabulary in the register denominator, so the axis would go quiet on exactly the prose it was fetched to judge (1 finding across three in-calibration papers against 8 across fifteen held out). Team size is read from the API's author list, never from counting `\author` in the LaTeX, which picks up template placeholder lines. Journal selection is client-side against observed `journal_ref` strings because the API's `jr:` prefix is a loose token match that returns "Research in Astronomy and Astrophysics" for an A&A query. HTTP 429 escalates through a backoff schedule and then stops the sweep with exit 2 and a TRUNCATED report; `--resume` extends a cut-short corpus instead of truncating it, and a journal/subfield run must pass `--out-name` so the broad corpus cannot be clobbered. `--date-hi` bounds the v1 submission, not the returned abstract, which is always the latest version's; `--updated-before` is the only control that dates the text, and a record with no `updated` is dropped rather than assumed clean. | +| `rewrite_reward.py` | Evaluates rewrite candidates after hard scientific-fidelity eligibility. | Bidirectional: dropping OR inventing protected numbers, units, citations, math, acronyms, semantic LaTeX macros, comparison direction, negation, or causal direction makes a candidate ineligible (`-inf`). Numbers and the three marker categories are compared as bags (a dropped second `not` is a loss; the report names count mismatches); a marker moved between clauses is the stated residual limit and belongs to the by-hand check. Ranking is led by L0 advisory reduction (signed, fidelity-floored) and semantic fidelity; the learned score is tie-break weight unless its bundle is measured. Its numeral tokenizer has been corrected six times, five of them for one root cause — a neighbour absorbed into the token, or the token started inside a neighbour: the Oxford comma, the spaced unit, (2026-08-25) a hyphen between numerals, which made `0.5-1.2` yield `-1.2`, and (2026-09-27) the LaTeX en-dash `0.5--1.2`, the Unicode minus and exponent sign, and identifiers (`M200` yielded `00`). Each is pinned by a test. `--field` is optional: one field auto-detects, none or several run without a profile. A missing profile, a missing or unusable `voice_model.joblib`, or a missing sentence-transformers leaves that term unmeasured at weight 0 with one stderr line, never exit 2. Exit 0 = at least one eligible candidate; 1 = every candidate ineligible (a measured outcome: preserve the original and regenerate tighter); 2 = invalid input or a crash. | +| `fetch_arxiv_abstracts.py` | Fetches dated arXiv corpora, abstracts or complete LaTeX sources, for controlled evaluation and calibration. The built-in query sets, keyword filter and journal table are `[WGL]` (`--query-set wl` narrows to the weak-lensing subfield; `--journals apj,apjl,aa` keeps only those refereed venues); any other field passes its own `--query` terms, which replace them. `--fulltext --author "Surname" --author-is --max-authors N` builds a per-author full-text population instead. | Network failures are explicit; fetched text is evidence/training data, not policy. The author path keeps the search term and the identity test separate, because `au:` matches a **surname** and a surname is not a person — `au:"Dell'Antonio"` returns an astronomer and a mathematical physicist. The abstract sweep's `AUTHOR_QUERIES` have no such filter, so each of the ten was checked once against the names it really returns: nine resolve to one person, and `au:"Schneider, P"` returned Donald P. Schneider (SDSS quasars) on 58 papers against the intended Peter Schneider's 33, so that entry now spells the given name out. That check needs the network and cannot be a test; adding an eleventh query means repeating it by hand. `--author` refuses to write into the calibration-breadth directory: a paper in a calibration bank has its own vocabulary in the register denominator, so the axis would go quiet on exactly the prose it was fetched to judge (1 finding across three in-calibration papers against 8 across fifteen held out). Team size is read from the API's author list, never from counting `\author` in the LaTeX, which picks up template placeholder lines. Journal selection is client-side against observed `journal_ref` strings because the API's `jr:` prefix is a loose token match that returns "Research in Astronomy and Astrophysics" for an A&A query. HTTP 429 escalates through a backoff schedule and then stops the sweep with exit 2 and a TRUNCATED report; `--resume` extends a cut-short corpus instead of truncating it, and a journal/subfield run must pass `--out-name` so the broad corpus cannot be clobbered. `--date-hi` bounds the v1 submission, not the returned abstract, which is always the latest version's; `--updated-before` is the only control that dates the text, and a record with no `updated` is dropped rather than assumed clean. Since 2026-09-27 the flags hold in both modes: `--journals` applies on the local full-text shortcut, `--start-at` sizes its pages from where the band began, a throttled full-text run is TRUNCATED and exit 2, `--updated-before` applies under `--fulltext` while `--query-set`, `--out-name` and `--resume` are refused there rather than ignored, candidates and output directories key on the bare arXiv id so a version bump is not a second copy, a `\documentstyle` or comment-led file is a LaTeX source and colliding tarball basenames are both kept, and omitting `--field` is a message and exit 2. | | `validate_plugin.py` | Validates manifests, version/count agreement (including the versioned doc headers of `docs/architecture/DEAI_SUBSYSTEM.md` and `docs/architecture/EVALUATION.md`), skill frontmatter and standard references, documentation authority boundaries and index completeness, recorded suite sizes against real test discovery, stale review markers, Python syntax, runtime imports, CLI entry points, feedback schema, linter exits, required tests, CI wiring, and the residue contract (`deai_residue.validator_check`: the history families the paper skill quotes match the tool's, and no shipped document carries an edit-meta literal). | Stdlib-only, 11 checks; rejects a document reappearing at a location it was moved away from, an unindexed document under `docs/`, a `design-notes/` file that does not declare itself one, a relative link or numbered cross-reference whose target does not exist, any recorded test count that disagrees with discovery, and any tracked file over the 750-line budget. Exits nonzero on contract drift. | The top-level manifest counts 40 product tools: the 39 shipped Python tools plus the diff --git a/tools/train_ai_ism_classifier.py b/tools/train_ai_ism_classifier.py index fa9fb91..3d928e8 100644 --- a/tools/train_ai_ism_classifier.py +++ b/tools/train_ai_ism_classifier.py @@ -235,7 +235,7 @@ def main(argv: list[str] | None = None) -> int: print(f" → {output}") print() print("To use the classifier:") - print(f" python tools/ai_ism_lint.py --ai-classifier --summary") + print(" python tools/ai_ism_lint.py --ai-classifier") return 0 From 95c988f7e25727af83bd4cdac9ebdccd21f2ad2b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:44:18 +0000 Subject: [PATCH 16/27] deai per-bucket axes: the grid at nearest rank, the unit read back, and a status that names what it skipped Audit items C1-C19, C21 and C22 of docs/audits/full-review-2026-09-27.md (C20 concerns deai_structure and is not touched here; the deai_voice part of C1 and C18 follows in the next commit). deai_reference - quantiles() stores the ceil(q * n)-th smallest value (nearest rank). The floor(q * n) it stored before sat one order statistic high: p90 of 100 values was the 91st smallest, and the 30-unit floor read its tails at P = 0.133 and 0.933. An artifact calibrated before this keeps the old grid until it is recalibrated. (C12) - usable_buckets() takes the unit the axis measures and refuses a bucket whose artifact records another one; unit_reason() names the refused buckets in the status. sections() pools a bucket's spans into one unit, as the reference side pools a paper's paragraphs per bucket. (C11) - unbucketed_reason() counts a document's units in the `unknown` bucket so a status can say how much of the document went unmeasured. (C13) - sentence_lines() locates a projected sentence in its raw unit. (C15) - calibrate() records text_key and text_fallback_rows per bucket and returns None, writing nothing, when no record contributed. (C8, C17) - the unreachable v_high == v_low branch and two stale docstrings. (C18, C19) deai_discourse / deai_collocation / deai_salience - each status function takes the text and reports degraded, with counts, when units sit in `unknown`, when a bucket records another unit, and (salience) when no bucket resolves above the gate: the floor alone read `measured` with zero findings on a flat reference. (C10, C11, C13) - collocation: --glossary goes through argparse (extra_arguments), is refused beside --calibrate (check) and picks its report (report_for); the definition cue is read only in the words after the pair; a pair joined twice in a sentence counts twice; the glossary status depends on the bank alone; sentence findings carry the sentence's own lines; calibrating with no passage writes nothing and exits 2. (C8, C14, C15, C21) - salience: a bank row without numeral_text is counted per bucket and warned about on stderr instead of calibrating silently on the [math] projection; the test that pinned the silent p90 of 0.0 now requires the warning. The finding path checks the unit through the shared helper. (C17, C22) - discourse: the degraded branch a finding could not take is gone. (C18) deai_register / deai_residue - \newcommand* is read. (C9) - body_lines() and body_only() take drop_skip; the edit-meta scan keeps the skip sections (an appendix TODO was invisible and the gate exited 0), and the label rule looks for a caption's object in paper_prose(), the whole paper minus preamble and bibliography. (C3, C5a) - RE_LABEL allows a space before the brace, as extract_sections does. (C4) - stem() reduces the -ation/-ated/-ating/-ate(s) family to one root. (C5b) - an unwritable --output is exit 2, not the gate's exit 1. (C6) - calibrate() on no passage returns None and writes no lexicon. (C8) - RE_BIBITEM, the saturated confidence formula, the stale seam comment, the AUC figure and the defined_terms direction. (C18, C19) deai_oracle / deai_provenance / deai_personal - the CUDA fallback handler survives an empty message; calibrate skips a blank or undecodable bank line, and a --field with no bank is exit 2 rather than a traceback. (C7) - --field resolves through cli_common.optional_field_dir. (C1, C21) - provenance reads reference.paragraphs(), so a title block is no ledger paragraph, and the text output no longer rebuilds the ledger. (C16, C18) - unused argparse imports and profile-root constants removed. (C18) Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit ee5303ff92a4fc53009888cedea9702100ab4bd0) --- tests/test_deai_collocation.py | 138 ++++++++++++++++++++ tests/test_deai_discourse.py | 58 ++++++++- tests/test_deai_oracle.py | 63 +++++++++ tests/test_deai_personal.py | 2 - tests/test_deai_provenance.py | 30 +++++ tests/test_deai_reference.py | 126 ++++++++++++++++++ tests/test_deai_register.py | 32 ++++- tests/test_deai_residue.py | 71 ++++++++++ tests/test_deai_salience.py | 94 +++++++++++--- tools/deai_collocation.py | 158 +++++++++++++++++------ tools/deai_discourse.py | 79 ++++++++---- tools/deai_oracle.py | 44 ++++--- tools/deai_personal.py | 1 - tools/deai_provenance.py | 54 ++++---- tools/deai_reference.py | 228 +++++++++++++++++++++++++++------ tools/deai_register.py | 86 ++++++++----- tools/deai_residue.py | 104 ++++++++++----- tools/deai_salience.py | 79 +++++++++--- 18 files changed, 1191 insertions(+), 256 deletions(-) diff --git a/tests/test_deai_collocation.py b/tests/test_deai_collocation.py index 906beb6..484f75c 100644 --- a/tests/test_deai_collocation.py +++ b/tests/test_deai_collocation.py @@ -168,6 +168,47 @@ def test_the_document_fraction_is_evidence_in_the_axis_status(self): self.assertIn("document novel-pair fraction", status["reason"]) self.assertIn("not a percentile", status["reason"]) + def test_a_document_of_unknown_units_is_degraded_not_measured(self): + # No heading (or a topic heading) puts a paragraph in `unknown`, a + # bucket no bank holds, so the axis skips it; the status used to say + # `measured` over a document it had skipped whole. + untitled = NOVEL.split("\n", 1)[1] + with fixture.temp_profile(records()) as profile: + collocation.calibrate(profile) + status = collocation.collocation_axis_status(profile, untitled) + self.assertEqual(status["status"], "degraded") + self.assertIn("1 of 1 paragraph units carry no calibrated bucket", + status["reason"]) + self.assertEqual(collocation.collocation_findings(untitled, profile), []) + + def test_a_reference_built_at_another_unit_is_refused_and_named(self): + with fixture.temp_profile(records()) as profile: + collocation.calibrate(profile) + path = profile / collocation.BASELINE_FILENAME + baseline = json.loads(path.read_text("utf-8")) + baseline["method"]["unit"] = "paragraph" + path.write_text(json.dumps(baseline), encoding="utf-8") + status = collocation.collocation_axis_status(profile, NOVEL) + self.assertEqual(status["status"], "degraded") + self.assertIn("method (paragraph)", status["reason"]) + self.assertEqual(collocation.collocation_findings(NOVEL, profile), []) + + def test_a_sentence_finding_carries_the_sentence_s_own_line(self): + novel_sentence = NOVEL.split("\n")[1] + text = ("\\section{Methods}\nThe aperture mass is measured on the shear " + "catalog around each cluster.\n" + novel_sentence + "\nThe tangential " + "shear is convolved with a compensated filter.\n") + with fixture.temp_profile(records()) as profile: + collocation.calibrate(profile) + (finding,) = collocation.collocation_findings(text, profile) + self.assertEqual((finding["location"]["start_line"], + finding["location"]["end_line"]), (3, 3)) + + def test_no_passage_means_nothing_calibrated(self): + with tempfile.TemporaryDirectory(prefix="colloc-") as raw: + self.assertIsNone(collocation.calibrate(Path(raw))) + self.assertEqual(list(Path(raw).iterdir()), []) + class CliTests(unittest.TestCase): def test_the_report_is_the_shared_schema(self): @@ -183,6 +224,36 @@ def test_the_report_is_the_shared_schema(self): self.assertIn("axis L2.collocation: measured", result.stdout) self.assertIn("collocation-novel:method", result.stdout) + def test_a_single_field_resolves_without_the_flag(self): + # `--help` has always promised auto-detection; the axis tools ran with + # no profile at all when `--field` was omitted. + with tempfile.TemporaryDirectory(prefix="colloc-") as raw: + root = Path(raw) / "profiles" + profile = root / "fld" + profile.mkdir(parents=True) + fixture.write_bank(profile, records()) + collocation.calibrate(profile) + target = Path(raw) / "draft.tex" + target.write_text(NOVEL, encoding="utf-8") + result = subprocess.run( + [sys.executable, str(TOOLS / "deai_collocation.py"), str(target), + "--profile-root", str(root)], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("axis L2.collocation: measured", result.stdout) + + def test_calibrating_an_empty_field_is_exit_two_with_nothing_written(self): + with tempfile.TemporaryDirectory(prefix="colloc-") as raw: + field = Path(raw) / "fld" + field.mkdir() + result = subprocess.run( + [sys.executable, str(TOOLS / "deai_collocation.py"), "--calibrate", + "--field", "fld", "--profile-root", raw], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 2, result.stdout) + self.assertIn("read no reference records", result.stderr) + self.assertEqual(list(field.iterdir()), []) + class GlossaryTests(unittest.TestCase): # `shear threshold` and `catalog noise`: every word common in the bank @@ -221,6 +292,73 @@ def test_a_definition_cue_at_first_use_is_recorded(self): (finding,) = collocation.glossary_findings(self.DEFINED, profile) self.assertTrue(finding["observed"]["glossed_at_first_use"]) + def test_the_cue_must_follow_the_pair(self): + # A `(Fig. 1)` or a `, the` elsewhere in the sentence marked a pair as + # defined; only a cue in the words after the pair, or a parenthesis + # opening right after it, glosses it. + pair = ("shear", "threshold") + self.assertTrue(collocation.glossed( + "The shear threshold, which we call the floor, sets the map.", pair)) + self.assertTrue(collocation.glossed( + "The shear threshold (hereafter the floor) sets the map.", pair)) + self.assertFalse(collocation.glossed( + "The shear threshold sets the map (Fig. 1).", pair)) + self.assertFalse(collocation.glossed( + "The shear threshold sets the map, the floor of it.", pair)) + self.assertFalse(collocation.glossed( + "We call the floor the shear threshold of the map.", pair)) + text = ("\\section{Methods}\nThe shear threshold sets the floor (Fig. 1). " + "The shear threshold is measured again.\n") + with fixture.temp_profile(records()) as profile: + collocation.calibrate(profile) + (finding,) = collocation.glossary_findings(text, profile) + self.assertFalse(finding["observed"]["glossed_at_first_use"]) + + def test_uses_count_every_occurrence(self): + text = ("\\section{Methods}\nThe shear threshold sets the shear threshold " + "of the map.\n") + with fixture.temp_profile(records()) as profile: + collocation.calibrate(profile) + (finding,) = collocation.glossary_findings(text, profile) + self.assertEqual(finding["observed"]["uses"], 2) + self.assertIn("joined 2 times", finding["message"]) + + def test_the_glossary_status_depends_on_the_bank_alone(self): + # The reading takes no percentile, so the sentence reference and its + # floor cannot degrade it; `collocation_axis_status` put "unmeasured" + # beside a list measured against every passage of the bank. + self.assertEqual(collocation.glossary_axis_status(None)["status"], "unmeasured") + with fixture.temp_profile(records()) as profile: + collocation.calibrate(profile) + (profile / collocation.BASELINE_FILENAME).unlink() + self.assertEqual(collocation.collocation_axis_status(profile)["status"], + "unmeasured") + status = collocation.glossary_axis_status(profile) + self.assertEqual(status["status"], "measured") + self.assertIn("glossary reading", status["reason"]) + target = profile / "draft.tex" + target.write_text(self.RECURRING, encoding="utf-8") + result = subprocess.run( + [sys.executable, str(TOOLS / "deai_collocation.py"), str(target), + "--field", profile.name, "--profile-root", str(profile.parent), + "--glossary"], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("axis L2.collocation: measured: glossary reading", result.stdout) + + def test_glossary_and_calibrate_are_refused_together(self): + # `--glossary` used to be stripped from argv by hand and rode silently + # beside `--calibrate`. + with fixture.temp_profile(records()) as profile: + result = subprocess.run( + [sys.executable, str(TOOLS / "deai_collocation.py"), "--calibrate", + "--glossary", "--field", profile.name, "--profile-root", + str(profile.parent)], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 2, result.stdout) + self.assertIn("--glossary", result.stderr) + self.assertFalse((profile / collocation.BANK_FILENAME).exists()) + def test_no_bank_lists_nothing_and_the_cli_mode_lists_the_candidates(self): with tempfile.TemporaryDirectory(prefix="colloc-") as raw: self.assertEqual(collocation.glossary_findings(self.RECURRING, Path(raw)), []) diff --git a/tests/test_deai_discourse.py b/tests/test_deai_discourse.py index 337dd29..e1d1369 100644 --- a/tests/test_deai_discourse.py +++ b/tests/test_deai_discourse.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json import unittest from pathlib import Path @@ -87,8 +88,8 @@ def levels_for(documents: int) -> list[int]: holding level 1 would tie with the fully POOR document the tests submit instead of sitting above it. And the lowest level gets about a twentieth of the mass rather than an even share, because the tenth percentile of N units - is the (N // 10)-th smallest: a bank whose minimum occupies a tenth or more - has p10 == p0, and the reference then correctly refuses to resolve. + is the ceil(N / 10)-th smallest: a bank whose minimum occupies a tenth or + more has p10 == p0, and the reference then correctly refuses to resolve. """ rare = max(1, documents // 20) return [2] * rare + [3 + index % (LEVELS - 2) @@ -168,6 +169,36 @@ def test_the_artifact_records_the_unit_it_was_built_at(self): self.assertEqual(result["cohesion"]["intro"]["unit"], "paragraph") self.assertEqual(result["hedging"]["intro"]["unit"], "section") + def test_a_reference_built_at_another_unit_is_refused_and_named(self): + # The artifact recorded its unit and nothing read it back: a hedging + # reference rebuilt at paragraph unit was read against sections. + with fixture.temp_profile(graded_bank(), prefix="discourse-") as profile: + discourse.calibrate(profile) + path = profile / discourse.AXES["hedging"]["baseline"] + baseline = json.loads(path.read_text("utf-8")) + baseline["intro"]["unit"] = "paragraph" + path.write_text(json.dumps(baseline), encoding="utf-8") + self.assertEqual(discourse.live_buckets("hedging", profile), []) + status = {s["axis"]: s for s in discourse.discourse_axis_status(profile)} + self.assertEqual(status["L2.hedging"]["status"], "degraded") + self.assertIn("intro (paragraph)", status["L2.hedging"]["reason"]) + self.assertEqual(status["L2.cohesion"]["status"], "measured") + rules = {f["rule"] for f in discourse.discourse_findings(document(POOR), profile)} + self.assertNotIn("discourse-hedging:intro", rules) + self.assertIn("discourse-cohesion:intro", rules) + + def test_a_section_split_over_subsections_is_measured_once(self): + # The reference pools a paper's intro paragraphs into one section, so + # the manuscript's subsections are one unit too: three POOR paragraphs + # under three subsections are one section of three paragraphs' words. + text = ("\\section{Introduction}\n" + + "\n".join(f"\\subsection{{Part {i}}}\n{p}" for i, p in enumerate(POOR)) + + "\n") + units = [(bucket, discourse.hedging_features(block)["n_words"]) + for _s, _e, bucket, block in reference.sections(text)] + pooled = discourse.hedging_features("\n\n".join(POOR))["n_words"] + self.assertEqual(units, [("intro", pooled)]) + def test_a_record_with_no_source_cannot_form_a_section(self): # Pooling unattributable paragraphs would join prose from unrelated # papers into a section no author ever wrote. @@ -240,6 +271,29 @@ def test_small_reference_is_degraded_not_measured(self): for s in discourse.discourse_axis_status(profile)} self.assertEqual(statuses["L2.hedging"], "degraded") + def test_a_document_of_unknown_units_is_degraded_not_measured(self): + # Untitled prose lands in `unknown`, which no bank holds; the status + # used to be computed from the profile alone and said `measured`. + with fixture.temp_profile(graded_bank(), prefix="discourse-") as profile: + discourse.calibrate(profile) + statuses = {s["axis"]: s for s in + discourse.discourse_axis_status(profile, "\n\n".join(POOR) + "\n")} + self.assertEqual(statuses["L2.cohesion"]["status"], "degraded") + self.assertIn("3 of 3 paragraph units carry no calibrated bucket", + statuses["L2.cohesion"]["reason"]) + self.assertEqual(statuses["L2.hedging"]["status"], "degraded") + self.assertIn("1 of 1 section units carry no calibrated bucket", + statuses["L2.hedging"]["reason"]) + titled = {s["axis"]: s["status"] for s in + discourse.discourse_axis_status(profile, document(POOR))} + self.assertEqual(titled, {"L2.cohesion": "measured", "L2.hedging": "measured"}) + + def test_no_bank_at_all_means_nothing_calibrated(self): + with fixture.temp_profile([], prefix="discourse-") as profile: + self.assertIsNone(discourse.calibrate(profile)) + self.assertEqual(sorted(p.name for p in profile.iterdir()), + ["exemplar_paragraphs.jsonl"]) + class TestBucketRestriction(unittest.TestCase): """Hedging is calibrated for introductions only, and has to say so.""" diff --git a/tests/test_deai_oracle.py b/tests/test_deai_oracle.py index eb164e8..adc829e 100644 --- a/tests/test_deai_oracle.py +++ b/tests/test_deai_oracle.py @@ -10,7 +10,13 @@ from __future__ import annotations +import contextlib +import io +import json +import tempfile import unittest +from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -63,6 +69,63 @@ def test_short_paragraphs_are_skipped_not_scored(self): oracle.token_surprisals = lambda text, model_name: [3.0] * (oracle.MIN_TOKENS - 1) self.assertEqual(oracle.uid_findings(TEXT, None), []) + def test_calibrate_skips_a_blank_or_undecodable_bank_line(self): + # `json.loads` on a blank line aborted an hour-long calibration. + oracle.token_surprisals = lambda text, model_name: [3.0 + (i % 2) for i in range(30)] + with tempfile.TemporaryDirectory(prefix="oracle-") as raw: + bank = Path(raw) / "exemplar_paragraphs.jsonl" + bank.write_text("\n".join([json.dumps({"section": "intro", "text": "one"}), + "", "not json {", " ", + json.dumps({"section": "intro", "text": "two"})]) + "\n", + encoding="utf-8") + baseline = oracle.calibrate(Path(raw), "stub") + self.assertEqual(baseline["n_paragraphs_used"], 2) + + +class FailureMessageTests(unittest.TestCase): + def test_an_exception_with_no_message_has_a_first_line(self): + # The CUDA fallback handler read `str(exc).splitlines()[0]` and died + # on the way to the fallback when the message was empty. + self.assertEqual(oracle._first_line(Exception("")), "?") + self.assertEqual(oracle._first_line(Exception("first\nsecond")), "first") + + +class FieldResolutionTests(unittest.TestCase): + """`--field` resolves through `cli_common.optional_field_dir`: none is + exit 2 with the message the tool always printed, one resolves on its own, + and `profile_root / None` is no longer a TypeError. `deai_voice`, the + other tool that cannot run without a profile, has the same case in + `test_deai_voice.py`.""" + + def test_oracle_needs_a_profile_and_finds_a_single_one(self): + with tempfile.TemporaryDirectory(prefix="oracle-") as raw: + draft = Path(raw) / "draft.tex" + draft.write_text(TEXT, encoding="utf-8") + profiles = Path(raw) / "profiles" + profiles.mkdir() + stderr = io.StringIO() + with contextlib.redirect_stderr(stderr): + self.assertEqual(oracle.main([str(draft), "--profile-root", str(profiles)]), 2) + self.assertIn("need --field", stderr.getvalue()) + (profiles / "fld").mkdir() + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + self.assertEqual(oracle.main([str(draft), "--profile-root", str(profiles)]), 0) + self.assertIn("axis L1.uid: unmeasured", stdout.getvalue()) + + def test_calibrating_a_field_without_a_bank_is_exit_two(self): + # With the runtime present, `calibrate` raised FileNotFoundError out + # of `main` (a traceback, exit 1) for a --field that has no bank. + with tempfile.TemporaryDirectory(prefix="oracle-") as raw: + (Path(raw) / "fld").mkdir() + stderr = io.StringIO() + with mock.patch.object(oracle, "model_runtime_available", + return_value=(True, "")), \ + contextlib.redirect_stderr(stderr): + code = oracle.main(["--calibrate", "--field", "fld", "--profile-root", raw]) + self.assertEqual(code, 2) + self.assertIn("no exemplar_paragraphs.jsonl", stderr.getvalue()) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_deai_personal.py b/tests/test_deai_personal.py index e8df67c..3f614e3 100644 --- a/tests/test_deai_personal.py +++ b/tests/test_deai_personal.py @@ -1,8 +1,6 @@ from __future__ import annotations -import sys import unittest -from pathlib import Path from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path diff --git a/tests/test_deai_provenance.py b/tests/test_deai_provenance.py index e24dcf4..11c3850 100644 --- a/tests/test_deai_provenance.py +++ b/tests/test_deai_provenance.py @@ -1,6 +1,8 @@ from __future__ import annotations +import subprocess import sys +import tempfile import unittest from pathlib import Path @@ -79,6 +81,34 @@ def test_findings_report_real_source_lines(self): self.assertEqual(findings[0]["location"]["start_line"], 3) self.assertIsNotNone(findings[0]["location"]["end_line"]) + def test_a_title_block_is_not_a_ledger_paragraph(self): + # The ledger cut the raw text on blank lines, so an `\\author` and + # `\\affiliation` block of twelve words was an `ai_untouched` span; the + # shared sweep drops the preamble and a `skip` section, and keeps the + # abstract. + front = ("\\title{A Study of Everything}\n\\author{Someone With A Very Long " + "Name From A Faraway Institute Of Science}\n\\affiliation{Department " + "of Physics, University of Somewhere, Some Street, Some Town}\n\n") + body = ("\\begin{abstract}\n" + ANCESTOR.split("\n\n")[0] + "\n\\end{abstract}\n" + "\\section{Methods}\n" + ANCESTOR.split("\n\n")[1] + "\n\n" + "\\section{Acknowledgments}\nWe thank the many colleagues who read " + "an early version of this manuscript closely.\n") + result = prov.document_provenance(front + body, front + body) + # The abstract unit spans its environment, so it starts on the + # `\\begin{abstract}` line; the Methods paragraph is on line 9. + self.assertEqual([entry["start_line"] for entry in result["ledger"]], [5, 9]) + + def test_the_text_output_lists_the_label_counts_once(self): + with tempfile.TemporaryDirectory() as raw: + draft = Path(raw) / "draft.tex" + draft.write_text(ANCESTOR, encoding="utf-8") + result = subprocess.run( + [sys.executable, str(TOOLS / "deai_provenance.py"), str(draft), + "--ai-ancestor", str(draft)], + text=True, capture_output=True, encoding="utf-8") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout.count("ai_untouched: 3"), 1) + def test_git_unreadable_reason_is_distinguished(self): # supplied-but-unreadable ref must not be reported as "none supplied". reason = "git ref abc123 unreadable or untracked for this file" diff --git a/tests/test_deai_reference.py b/tests/test_deai_reference.py index 36068c7..705e0f4 100644 --- a/tests/test_deai_reference.py +++ b/tests/test_deai_reference.py @@ -12,10 +12,15 @@ from __future__ import annotations +import json +import tempfile import unittest +from collections import Counter +from pathlib import Path from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path +import _profilefixture as fixture import deai_reference as reference PROSE = "The estimator uses five filters and reports the peak height. " * 3 @@ -59,6 +64,127 @@ def test_section_of_labels_alone_is_not_a_section_unit(self): buckets = [bucket for _start, _end, bucket, _block in reference.sections(TEXT)] self.assertEqual(buckets, ["intro", "results"]) + def test_a_bucket_split_over_subsections_is_one_section_unit(self): + # The reference side pools a paper's paragraphs per bucket (that is how + # the corpus splitter cuts a paper); read per heading span, a 150-word + # subsection was compared with references pooled over whole papers. + text = "\n".join(["\\section{Methods}", "\\subsection{Filter}", PROSE, + "\\subsection{Noise}", PROSE, "\\section{Results}", PROSE]) + found = reference.sections(text) + self.assertEqual([(start, end, bucket) for start, end, bucket, _ in found], + [(1, 5, "method"), (6, 7, "results")]) + self.assertEqual(found[0][3].count(PROSE.strip()), 2) + + +class QuantileGridTests(unittest.TestCase): + """The stored p_q point is the first order statistic with P(X <= x) >= q. + + The grid read floor(q * n) until 2026-09-27 -- one order statistic high: + p90 of 100 values was the 91st smallest, and at the 30-unit floor p10 and + p90 sat at P = 0.133 and 0.933. Every rate recorded against an artifact + calibrated before that date was read at those shifted points. + """ + + def test_nearest_rank_on_a_hundred_values(self): + grid = reference.quantiles([float(v) for v in range(100)]) + self.assertEqual((grid["0.0"], grid["0.1"], grid["0.5"], grid["0.9"], grid["1.0"]), + (0.0, 9.0, 49.0, 89.0, 99.0)) + + def test_every_grid_point_is_the_smallest_rank_at_or_past_q(self): + n = 100 + grid = reference.quantiles([float(v) for v in range(n)]) + for q, value in grid.items(): + rank = int(value) + 1 # values are 0..99, so value + 1 is its rank + self.assertGreaterEqual(rank / n, float(q), q) + if float(q) > 0: + self.assertLess((rank - 1) / n, float(q), q) + + def test_the_floor_bucket_reads_its_tails_at_the_nominal_rate(self): + grid = reference.quantiles([float(v) for v in range(30)]) + self.assertEqual((grid["0.1"], grid["0.9"]), (2.0, 26.0)) # 3rd and 27th smallest + + def test_a_single_value_fills_the_grid(self): + self.assertEqual(set(reference.quantiles([5.0]).values()), {5.0}) + + +class ArtifactUnitTests(unittest.TestCase): + """The unit an artifact records is read back, not merely written.""" + + BASELINE = {"intro": {"n": 40, "unit": "section", "percentiles": { + "hedging": {"0.0": 0.0, "0.1": 1.0, "1.0": 5.0}}}, + "method": {"n": 40, "unit": "paragraph", "percentiles": { + "hedging": {"0.0": 0.0, "0.1": 1.0, "1.0": 5.0}}}, + "data": {"n": 40, "percentiles": { + "hedging": {"0.0": 0.0, "0.1": 1.0, "1.0": 5.0}}}} + + def test_a_bucket_built_at_another_unit_is_not_usable(self): + self.assertEqual(reference.usable_buckets(self.BASELINE, "hedging", 0.1, + high=False, unit="section"), + ["intro"]) + + def test_the_foreign_units_are_named_with_what_they_record(self): + self.assertEqual(reference.foreign_units(self.BASELINE, "section"), + {"method": "paragraph", "data": "unrecorded"}) + reason = reference.unit_reason(self.BASELINE, "section") + self.assertIn("method (paragraph)", reason) + self.assertIn("recalibrate", reason) + self.assertIsNone(reference.unit_reason({"intro": self.BASELINE["intro"]}, "section")) + + +class UnbucketedUnitsTests(unittest.TestCase): + def test_untitled_prose_is_counted_as_unmeasured(self): + text = PROSE + "\n\n" + PROSE + reason = reference.unbucketed_reason(reference.units(text), "paragraph") + self.assertTrue(reason.startswith("2 of 2 paragraph units carry no calibrated bucket")) + + def test_a_topic_heading_is_unmeasured_and_a_role_heading_is_not(self): + self.assertIsNone(reference.unbucketed_reason( + reference.units("\\section{Methods}\n" + PROSE), "paragraph")) + self.assertIsNotNone(reference.unbucketed_reason( + reference.units("\\section{Weak lensing}\n" + PROSE), "paragraph")) + + +class SentenceLineTests(unittest.TestCase): + BLOCK = "The shear is measured \\citep{x}\nhere. The \\emph{map}\nis drawn now." + + def test_the_second_sentence_starts_on_its_own_line(self): + self.assertEqual(reference.sentence_lines(self.BLOCK, 10, "The map is drawn now."), + (11, 12)) + + def test_a_citation_between_two_words_does_not_hide_the_sentence(self): + self.assertEqual(reference.sentence_lines( + self.BLOCK, 10, "The shear is measured [CITE] here."), (10, 11)) + + def test_an_unlocatable_sentence_keeps_the_unit_range(self): + self.assertEqual(reference.sentence_lines(self.BLOCK, 10, "Absent words here."), + (10, 12)) + + +class CalibrationRecordTests(unittest.TestCase): + def test_no_reference_record_means_none_and_nothing_written(self): + with tempfile.TemporaryDirectory(prefix="reference-") as raw: + profile = Path(raw) + written = reference.calibrate( + profile, "x_baseline.json", ("f",), lambda text: {"f": 1.0}, + reference.passage_banks(profile)) + self.assertIsNone(written) + self.assertFalse((profile / "x_baseline.json").exists()) + + def test_rows_read_on_the_fallback_projection_are_counted_per_bucket(self): + rows = fixture.uniform("The rate is $3$ here.", 4, section="method") + rows[0]["numeral_text"] = "The rate is 3 here." + with fixture.temp_profile(rows) as profile: + with (profile / "human_abstracts_extra.jsonl").open("w", encoding="utf-8") as handle: + handle.write(json.dumps({"text": "An abstract in LaTeX."}) + "\n") + tally: Counter[str] = Counter() + records = list(reference._bank_records( + reference.passage_banks(profile), text_key="numeral_text", fallbacks=tally)) + # Three exemplar rows fell back; the abstract bank never has the key + # by design (it stores source) and is not a fallback. + self.assertEqual(tally, Counter({"method": 3})) + self.assertEqual(len(records), 5) + self.assertEqual(records[0][2], "The rate is 3 here.") + if __name__ == "__main__": unittest.main() diff --git a/tests/test_deai_register.py b/tests/test_deai_register.py index 59a69f3..4069368 100644 --- a/tests/test_deai_register.py +++ b/tests/test_deai_register.py @@ -1,7 +1,6 @@ from __future__ import annotations import json -import sys import tempfile import unittest from pathlib import Path @@ -62,6 +61,16 @@ def test_superscript_decoration_is_not_a_term(self): self.assertEqual( register.macro_terms("\\newcommand{\\Kref}{K^\\mathrm{ref}}"), {}) + def test_a_starred_definition_is_read(self): + # `\newcommand*` expanded to nothing: neither counted nor audited. + for command in ("newcommand*", "renewcommand*", "providecommand*"): + self.assertEqual( + register.macro_terms(f"\\{command}{{\\AUC}}{{\\mathrm{{AUC}}}}"), + {"AUC": "AUC"}, command) + document = ("\\newcommand*{\\AUC}{\\mathrm{AUC}}\n\\section{Validation}\n" + "We report \\AUC and \\AUC here.\n") + self.assertEqual(register.manuscript_terms(document)["auc"]["count"], 2) + class TestCompoundFrequency(unittest.TestCase): TABLE = {"aperture": 312, "mass": 900, "held": 24, "out": 800, @@ -355,6 +364,20 @@ def test_a_blank_line_above_a_macro_definition_keeps_the_line_numbers(self) -> N "The convergence is measured.\n") self.assertEqual(register.manuscript_terms(document)["convergence"]["line"], 4) + def test_the_skip_sections_can_be_kept_for_a_reader_that_needs_them(self) -> None: + # The vocabulary comparison drops them because the corpus did; an + # editing-mark scan keeps them, since a TODO in an appendix is as + # left behind as one in a paragraph. + document = ("\\title{T}\n\\section{Methods}\nThe shear is measured.\n" + "\\section{Appendix}\nTODO fix this later.\n" + "\\begin{thebibliography}{}\n\\bibitem{a} Blain.\n" + "\\end{thebibliography}\n") + kept = register.body_lines(document, drop_skip=False) + self.assertEqual(kept.splitlines()[0], "") # the preamble stays out + self.assertEqual(kept.splitlines()[4], "TODO fix this later.") + self.assertNotIn("Blain", kept) # and so does the bibliography + self.assertEqual(register.body_lines(document).splitlines()[4], "") + def test_a_subsection_under_a_skip_section_is_dropped_with_it(self) -> None: document = ("\\section{Methods}\nThe shear catalog is measured.\n" "\\section{Acknowledgments}\n\\subsection{Special thanks}\n" @@ -393,6 +416,13 @@ def test_a_sentence_that_defines_nothing_exempts_nothing(self) -> None: self.assertEqual(register.defined_terms("The catalog is rescored."), set()) +class EmptyBankTest(unittest.TestCase): + def test_no_passage_means_no_lexicon_written(self) -> None: + with tempfile.TemporaryDirectory(prefix="register-") as raw: + self.assertIsNone(register.calibrate(Path(raw))) + self.assertFalse((Path(raw) / register.LEXICON_FILENAME).exists()) + + class BankResolutionTest(unittest.TestCase): """A bank too coarse for the gate must say so, not report `measured`. diff --git a/tests/test_deai_residue.py b/tests/test_deai_residue.py index 3a3420c..2108b73 100644 --- a/tests/test_deai_residue.py +++ b/tests/test_deai_residue.py @@ -59,6 +59,15 @@ def test_history_about_the_literature_is_not_self_history(self): "for galaxy clusters.\n") self.assertEqual(found, {}) + def test_a_sentence_finding_carries_the_sentence_s_own_lines(self): + # The finding pointed at the paragraph's line range; the sentence sits + # on line 3 of a unit that starts on line 2. + text = ("\\section{Methods}\nClean prose here.\nWe no longer use a wider " + "filter.\nMore prose here.\n") + (finding,) = residue.self_history_findings(text) + self.assertEqual((finding["location"]["start_line"], finding["location"]["end_line"]), + (3, 3)) + def test_the_action_never_says_delete_the_negation(self): for action in (residue.HISTORY_ACTION, residue.NEGATIVE_LABEL_ACTION, residue.EDIT_META_ACTION): @@ -141,6 +150,22 @@ def test_a_mark_in_a_heading_or_caption_is_visible(self): found = residue.residue_findings("\\newcommand{\\TODO}{x}\n\\section{Results}\nClean.\n") self.assertEqual([f for f in found if f["rule"] == "residue-edit-meta"], []) + def test_a_mark_in_a_skip_section_is_visible(self): + # An appendix is a `skip` bucket for the vocabulary comparison, since + # the corpus never held one; an editing mark left there is as left + # behind as one in a paragraph, and the strong gate returned 0 on it. + text = ("\\section{Results}\nClean prose.\n\\section{Appendix}\n" + "TODO fix this later. The slope (removed) is negative.\n") + marks = [(f["observed"]["mark"], f["location"]["start_line"]) + for f in residue.residue_findings(text) if f["rule"] == "residue-edit-meta"] + self.assertEqual(marks, [("TODO", 4), ("(removed)", 4)]) + self.assertEqual(run_residue(text).returncode, 1) + # The bibliography stays out: a bibitem's `TODO` key is not prose. + text = ("\\section{Results}\nClean prose.\n\\begin{thebibliography}{}\n" + "\\bibitem{TODO} Blain, A.\n\\end{thebibliography}\n") + self.assertEqual([f for f in residue.residue_findings(text) + if f["rule"] == "residue-edit-meta"], []) + def test_a_phrase_mark_wrapped_at_a_line_break_is_one_mark(self): text = "\\section{Results}\nAs noted in the revised\nversion, the slope is negative.\n" found = residue.edit_meta_findings(text) @@ -200,6 +225,44 @@ def test_a_heading_negation_is_also_a_label(self): self.assertEqual([f["observed"]["negated_object"] for f in found], ["compensating kernel"]) + def test_a_space_before_the_brace_or_a_short_title_is_still_a_label(self): + # LaTeX allows `\section {X}` and `\caption[short]{X}`; both were + # invisible to the label rule (audit A9 of 2026-09-04, not landed here). + self.assertEqual([label for _line, label in residue._labels( + "\\section {No saddle}\n\\caption [short] {Without the dip}")], + ["No saddle", "Without the dip"]) + text = long_body("\\begin{figure}\\caption {Peak counts without the saddle " + "correction.}\\end{figure}") + self.assertEqual([f["observed"]["negated_object"] + for f in residue.negative_label_findings(text)], + ["the saddle correction"]) + + def test_an_object_named_in_an_appendix_is_not_absent(self): + # Labels are read from the whole text, so the object is looked for in + # the whole paper too: an appendix caption negating what the appendix + # explains three lines above it was reported absent from the body. + appendix = ("\\section{Appendix}\nThe saddle correction is applied to every " + "map in this appendix.\n\\begin{figure}\\caption{Peak counts " + "without the saddle correction.}\\end{figure}\n") + self.assertEqual(residue.negative_label_findings(long_body() + appendix), []) + body_caption = long_body("\\begin{figure}\\caption{Peak counts without the " + "saddle correction.}\\end{figure}") + self.assertEqual(len(residue.negative_label_findings(body_caption)), 1) + self.assertEqual(residue.negative_label_findings( + body_caption + "\\section{Appendix}\nThe saddle correction is " + "described here.\n"), []) + + def test_the_ation_family_reduces_to_one_root(self): + # One suffix off left `truncation` as `trunc` and `truncated` as + # `truncat`, so a caption "without truncation" never found the body. + self.assertEqual({residue.stem(w) for w in + ("truncation", "truncated", "truncating", "truncates")}, + {"trunc"}) + text = long_body("The catalog is truncated at the survey edge. " + "\\begin{figure}\\caption{Peak counts without " + "truncation.}\\end{figure}") + self.assertEqual(residue.negative_label_findings(text), []) + class NegativeLabelAddedTests(unittest.TestCase): def test_a_patched_absence_is_strong(self): @@ -265,6 +328,14 @@ def test_missing_file_is_configuration_failure(self): text=True, capture_output=True, encoding="utf-8") self.assertEqual(result.returncode, 2) + def test_an_unwritable_output_is_configuration_failure_not_a_strong_finding(self): + with tempfile.TemporaryDirectory() as raw: + target = Path(raw) / "no-such-dir" / "report.json" + result = run_residue("\\section{Methods}\nThe filter scale is fixed.\n", + "--output", str(target)) + self.assertEqual(result.returncode, 2, result.stderr) + self.assertIn("cannot write", result.stderr) + class ValidatorHookTests(unittest.TestCase): def test_the_word_family_is_read_between_the_markers(self): diff --git a/tests/test_deai_salience.py b/tests/test_deai_salience.py index d06c617..1524868 100644 --- a/tests/test_deai_salience.py +++ b/tests/test_deai_salience.py @@ -1,5 +1,7 @@ from __future__ import annotations +import contextlib +import io import json import tempfile import unittest @@ -115,6 +117,15 @@ def synthetic_passage(run_length: int, total: int = 8) -> str: return numeric * run_length + plain * (total - run_length) +def with_numerals(records: list[dict]) -> list[dict]: + """The records as the bank writer stores them: `text` projected, and + `numeral_text` beside it. A row without the key is a projection fallback + the calibration warns about, which a fixture should not trip by accident.""" + return [dict(record, text=es.latex_to_plain(record["text"]), + numeral_text=es.latex_to_numeral_text(record["text"])) + for record in records] + + def graded_bank(directory: Path, section: str = "abstract") -> Path: """A reference with spread in its upper tail, as a real corpus has. @@ -122,13 +133,9 @@ def graded_bank(directory: Path, section: str = "abstract") -> Path: that the p90 gate and the maximum differ, which is the condition a reference must meet before it can rank anything. """ - bank = directory / "exemplar_paragraphs.jsonl" lengths = [0] * 10 + [1] * 10 + [2] * 10 + [3] * 9 + [8] - with bank.open("w", encoding="utf-8") as handle: - for run in lengths: - handle.write(json.dumps({"section": section, - "text": synthetic_passage(run)}) + "\n") - return bank + return fixture.write_bank(directory, with_numerals( + [{"section": section, "text": synthetic_passage(run)} for run in lengths])) class TestFindingsAndCalibration(unittest.TestCase): @@ -169,14 +176,33 @@ def test_calibration_reads_the_numeral_projection_of_the_bank(self): "text": es.latex_to_plain(synthetic_passage(run)), "numeral_text": synthetic_passage(run)} for run in lengths] with fixture.temp_profile(rows) as profile: - p90 = salience.calibrate(profile)["abstract"]["percentiles"][ - "max_recital_run_frac"]["0.9"] - self.assertGreater(p90, 0.0) - with fixture.temp_profile([{k: v for k, v in row.items() - if k != "numeral_text"} for row in rows]) as profile: - fallback = salience.calibrate(profile)["abstract"]["percentiles"][ - "max_recital_run_frac"]["0.9"] - self.assertEqual(fallback, 0.0) + stderr = io.StringIO() + with contextlib.redirect_stderr(stderr): + abstract = salience.calibrate(profile)["abstract"] + self.assertGreater(abstract["percentiles"]["max_recital_run_frac"]["0.9"], 0.0) + self.assertEqual((abstract["text_key"], abstract["text_fallback_rows"]), + (salience.BANK_TEXT_KEY, 0)) + self.assertEqual(stderr.getvalue(), "") + + def test_a_bank_without_the_numeral_projection_is_recorded_and_warned_about(self): + # The fallback to `text` mixes projections: the reference then holds + # no numeral for those rows. It used to be silent, and this test once + # pinned the silent p90 of 0.0 as the correct result. + rows = [{"section": "abstract", "text": es.latex_to_plain(synthetic_passage(run))} + for run in [0] * 10 + [1] * 10 + [2] * 10 + [3] * 9 + [8]] + with fixture.temp_profile(rows) as profile: + stderr = io.StringIO() + with contextlib.redirect_stderr(stderr): + abstract = salience.calibrate(profile)["abstract"] + self.assertEqual(abstract["text_fallback_rows"], 40) + self.assertEqual(abstract["text_key"], salience.BANK_TEXT_KEY) + self.assertIn("WARNING", stderr.getvalue()) + self.assertIn("abstract=40", stderr.getvalue()) + + def test_no_passage_means_nothing_calibrated(self): + with tempfile.TemporaryDirectory(prefix="salience-") as raw: + self.assertIsNone(salience.calibrate(Path(raw))) + self.assertEqual(list(Path(raw).iterdir()), []) def test_a_typical_passage_is_not_flagged(self): with tempfile.TemporaryDirectory(prefix="salience-") as raw: @@ -191,20 +217,56 @@ def test_a_reference_with_no_upper_tail_spread_abstains(self): # Forty identical passages give P(X <= x) = 1.0 for the value they all # share, so an unguarded reading would flag a perfectly typical # passage as the 100th percentile. - with fixture.temp_profile(fixture.uniform(RANKED, 40)) as profile: + with fixture.temp_profile(with_numerals(fixture.uniform(RANKED, 40))) as profile: salience.calibrate(profile) baseline = salience.load_baseline(profile) self.assertFalse(salience.resolves_above_gate( baseline["abstract"], "max_recital_run_frac")) document = "\\begin{abstract}\n" + RANKED + "\n\\end{abstract}\n" self.assertEqual(salience.salience_findings(document, profile), []) + # The status said `measured` here while detection abstained on + # every bucket: a confident zero findings forever. + status = salience.salience_axis_status(profile, document) + self.assertEqual(status["status"], "degraded") + self.assertIn("spread above the advisory gate", status["reason"]) def test_small_reference_is_degraded_not_measured(self): - with fixture.temp_profile(fixture.uniform(RANKED, 5)) as profile: + with fixture.temp_profile(with_numerals(fixture.uniform(RANKED, 5))) as profile: salience.calibrate(profile) self.assertEqual(salience.salience_axis_status(profile)["status"], "degraded") + def test_a_document_of_unknown_units_is_degraded_not_measured(self): + # Untitled prose lands in `unknown`, which no bank holds, so the axis + # skips it; the status used to be computed from the profile alone. + with tempfile.TemporaryDirectory(prefix="salience-") as raw: + profile = Path(raw) + graded_bank(profile) + salience.calibrate(profile) + status = salience.salience_axis_status(profile, RECITAL_HEAVY + "\n") + self.assertEqual(status["status"], "degraded") + self.assertIn("1 of 1 paragraph units carry no calibrated bucket", + status["reason"]) + titled = "\\begin{abstract}\n" + RECITAL_HEAVY + "\n\\end{abstract}\n" + status = salience.salience_axis_status(profile, titled) + self.assertEqual(status["status"], "measured") + self.assertIn("buckets: abstract", status["reason"]) + + def test_a_reference_built_at_another_unit_is_refused_and_named(self): + with tempfile.TemporaryDirectory(prefix="salience-") as raw: + profile = Path(raw) + graded_bank(profile) + salience.calibrate(profile) + path = profile / salience.BASELINE_FILENAME + baseline = json.loads(path.read_text("utf-8")) + baseline["abstract"]["unit"] = "section" + path.write_text(json.dumps(baseline), encoding="utf-8") + document = "\\begin{abstract}\n" + RECITAL_HEAVY + "\n\\end{abstract}\n" + self.assertEqual(salience.salience_findings(document, profile), []) + status = salience.salience_axis_status(profile, document) + self.assertEqual(status["status"], "degraded") + self.assertIn("abstract (section)", status["reason"]) + class TestLocalReference(unittest.TestCase): """The axis on a real locally-built reference, when one is present. diff --git a/tools/deai_collocation.py b/tools/deai_collocation.py index edf1f7b..7229105 100644 --- a/tools/deai_collocation.py +++ b/tools/deai_collocation.py @@ -4,10 +4,12 @@ whether two words that each belong to the field have ever been put next to each other by it. The two questions have opposite evidence, and that is why they are two axes: refereed papers carry MORE unattested single words than -machine drafts (rank AUC 0.242, EVALUATION §23), because a real paper names -real instruments, coins real terms and cites real people; but they carry FEWER -unattested pairs of common words (AUC 0.855, length-controlled), because a -writer who has read the field reaches for the field's own phrases. A machine +machine drafts (rank AUC 0.174 under the v0.36.2 projection, EVALUATION +§23.1), because a real paper names real instruments, coins real terms and +cites real people; but they carry FEWER unattested pairs of common words +(document novel-pair fraction, machine over held-out AUC 0.704, §23.2), +because a writer who has read the field reaches for the field's own phrases. +A machine draft assembles field words into combinations the field does not use -- `calibrated blur`, `controlled grid`, `physical cells`, `detector axes` -- and the advisor who read such a draft wrote "I don't know what this means" at @@ -57,6 +59,7 @@ BANK_FILENAME = "collocation_bank.json" BASELINE_FILENAME = "collocation_baseline.json" FEATURE = "novel_fraction" +UNIT = "sentence" ADVISORY_PERCENTILE = 0.90 STRONG_PERCENTILE = 0.95 MIN_REFERENCE_N = reference.MIN_REFERENCE_N @@ -177,6 +180,13 @@ def document_novelty(text: str, bank: dict[str, Any]) -> dict[str, Any]: def collocation_axis_status(field_profile_dir: Path | None, text: str | None = None) -> dict[str, Any]: + """The axis status, `degraded` with its reasons where it did not measure. + + `text` is what makes the status about THIS document: a paragraph in the + `unknown` bucket (no heading, or a topic heading) has no reference and is + skipped, and without the text the status could only describe the profile + and reported `measured` over a document it had skipped whole. + """ bank = load_bank(field_profile_dir) baseline = load_baseline(field_profile_dir) if bank is None or baseline is None: @@ -184,23 +194,24 @@ def collocation_axis_status(field_profile_dir: Path | None, return feedback.axis_status( "L2.collocation", "unmeasured", reason=f"{missing} is unavailable", detector="deai_collocation") - usable = reference.usable_buckets(baseline, FEATURE, ADVISORY_PERCENTILE, high=True) + usable = reference.usable_buckets(baseline, FEATURE, ADVISORY_PERCENTILE, + high=True, unit=UNIT) + reasons = [None if usable else + (f"no section bucket reaches the {MIN_REFERENCE_N}-sentence " + "reference floor with spread above the gate"), + reference.unit_reason(baseline, UNIT)] evidence = None if text is not None: + reasons.append(reference.unbucketed_reason(reference.units(text), "paragraph")) whole = document_novelty(text, bank) if whole[FEATURE] is not None: evidence = (f"document novel-pair fraction {whole[FEATURE]:.3f} over " f"{whole['judged_pairs']} judged pairs (evidence, not a " "percentile: the reference is per sentence)") - if not usable: - floor = (f"no section bucket reaches the {MIN_REFERENCE_N}-sentence " - "reference floor with spread above the gate") - return feedback.axis_status( - "L2.collocation", "degraded", - reason="; ".join(part for part in (floor, evidence) if part), - detector="deai_collocation") - return feedback.axis_status("L2.collocation", "measured", reason=evidence, - detector="deai_collocation") + return feedback.axis_status( + "L2.collocation", "degraded" if any(reasons) else "measured", + reason="; ".join(part for part in (*reasons, evidence) if part) or None, + detector="deai_collocation") def _weighed(novel: list[tuple[str, str]], bank: dict[str, Any]) -> list[dict[str, Any]]: @@ -219,6 +230,7 @@ def _sentence_finding(sentence: str, verdict: dict[str, Any], percentile: float, bank: dict[str, Any], bucket: str, ref: dict[str, Any], span: tuple[int, int], path: str | Path | None, field_profile_dir: Path | None) -> dict[str, Any]: + """One sentence's finding; `span` is the sentence's own line range.""" reference_n = int(ref.get("n", 0)) measured = reference_n >= MIN_REFERENCE_N strong = bool(measured and percentile > STRONG_PERCENTILE) @@ -226,12 +238,12 @@ def _sentence_finding(sentence: str, verdict: dict[str, Any], percentile: float, pairs = _weighed(verdict["novel"], bank) excerpt = " ".join(sentence.split()) frame = dict(kind="advisory", layer="L2", scope="sentence", - calibration_unit="sentence", detector="deai_collocation", + calibration_unit=UNIT, detector="deai_collocation", rule=f"collocation-novel:{bucket}", section=bucket, path=path, line=span[0], end_line=span[1], strength="strong" if strong else "ordinary", measurement_status="measured" if measured else "degraded") - policy = dict(provenance=BASELINE_FILENAME, unit="sentence", + policy = dict(provenance=BASELINE_FILENAME, unit=UNIT, n_passages=n_passages, common_rate=COMMON_RATE, advisory_percentile=ADVISORY_PERCENTILE, strong_percentile=STRONG_PERCENTILE) @@ -267,9 +279,11 @@ def collocation_findings(text: str, field_profile_dir: Path | None, if bank is None or baseline is None: return [] findings: list[dict[str, Any]] = [] - for start, end, bucket, block in reference.units(text): + for start, _end, bucket, block in reference.units(text): ref = baseline.get(bucket) - if not isinstance(ref, dict): + # A bucket under the sample floor still speaks, as a degraded finding; + # one built at another unit does not speak at all (the status says why). + if not isinstance(ref, dict) or not reference.unit_matches(ref, UNIT): continue if not reference.resolves_gate(ref, FEATURE, ADVISORY_PERCENTILE, high=True): continue @@ -282,18 +296,24 @@ def collocation_findings(text: str, field_profile_dir: Path | None, continue findings.append(_sentence_finding( sentence, verdict, percentile, bank=bank, bucket=bucket, ref=ref, - span=(start, end), path=path, field_profile_dir=field_profile_dir)) + span=reference.sentence_lines(block, start, sentence), path=path, + field_profile_dir=field_profile_dir)) return findings # ---- the glossary reading (document scope) ---------------------------------- GLOSSARY_MIN_USES = 2 GLOSSARY_RULE = "collocation-glossary" -# A definition cue in the first-use sentence: the term is introduced there. +# A definition cue is read only in the words AFTER the pair, or as a +# parenthesis opening right after it: `the shear threshold, which we call the +# floor` and `the shear threshold (hereafter ...)` gloss the pair, while a +# `(Fig. 1)` or a `, the` anywhere else in the sentence does not -- read over +# the whole sentence, those marked one pair in six as defined on the Letter. RE_GLOSS_CUE = re.compile( r"\b(?:we call|called|which we call|define[sd]? as|defined|denote[sd]?|" - r"that is|which is|i\.e\.|namely|meaning)\b|\(|, the |, that |, those ", + r"that is|which is|i\.e\.|namely|meaning|hereafter)\b", re.IGNORECASE) +GLOSS_WINDOW_WORDS = 8 GLOSSARY_ACTION = ( "A pair the field never joins that this manuscript uses more than once is " "most likely its own term, not a slip: the advisor's 'this is jargon and " @@ -303,6 +323,11 @@ def collocation_findings(text: str, field_profile_dir: Path | None, "phrasing that the corpus happens to lack, leave it.") +def _pair_pattern(pair: tuple[str, str]) -> str: + """The pair as written: each stem with any suffix, the first with its 's.""" + return rf"\b{re.escape(pair[0])}(?:'s)?\w*\s+{re.escape(pair[1])}\w*" + + def _line_of_pair(block: str, start: int, pair: tuple[str, str]) -> int: """Line of the pair's first appearance in the raw unit, else the unit's start. @@ -311,10 +336,47 @@ def _line_of_pair(block: str, start: int, pair: tuple[str, str]) -> int: may carry its 's; the fallback keeps the finding anchored to the unit when a macro or a line break sits between the words in the source. """ - hit = re.search(rf"\b{re.escape(pair[0])}(?:'s)?\w*\s+{re.escape(pair[1])}\w*", block, re.IGNORECASE) + hit = re.search(_pair_pattern(pair), block, re.IGNORECASE) return start + block[:hit.start()].count("\n") if hit else start +def glossed(sentence: str, pair: tuple[str, str]) -> bool: + """Whether the sentence defines the pair where it uses it. + + A parenthesis opening right after the pair, or a cue among the + `GLOSS_WINDOW_WORDS` words that follow it; a cue before the pair or later + in the sentence is about something else. + """ + hit = re.search(_pair_pattern(pair), sentence, re.IGNORECASE) + if hit is None: + return False + after = sentence[hit.end():] + if re.match(r"\s*\(", after): + return True + window = " ".join(after.split()[:GLOSS_WINDOW_WORDS]) + return bool(RE_GLOSS_CUE.search(window)) + + +def glossary_axis_status(field_profile_dir: Path | None) -> dict[str, Any]: + """The status of the glossary reading: it needs the bank and nothing else. + + No percentile is read, so the sentence reference is not consulted and + its floor cannot degrade this reading; reporting `collocation_axis_status` + beside a glossary list put "unmeasured" next to findings that had been + measured against every passage of the bank. + """ + bank = load_bank(field_profile_dir) + if bank is None: + return feedback.axis_status( + "L2.collocation", "unmeasured", + reason=f"{BANK_FILENAME} is unavailable", detector="deai_collocation") + return feedback.axis_status( + "L2.collocation", "measured", + reason=(f"glossary reading: pairs unattested in {int(bank['n_passages']):,} " + "passages, document scope, no percentile"), + detector="deai_collocation") + + def glossary_findings(text: str, field_profile_dir: Path | None, path: str | Path | None = None) -> list[dict[str, Any]]: """Recurring unattested pairs, read as the manuscript's own terms. @@ -323,9 +385,10 @@ def glossary_findings(text: str, field_profile_dir: Path | None, stumbles. This reading aggregates every judged sentence of the document and lists the pairs the field never joins that the manuscript uses at least GLOSSARY_MIN_USES times: a coinage recurs, a figure of speech does - not. Each candidate carries its uses, the sections it appears in, its - first-use line (the pair's first appearance in the raw unit, else the unit's - start) and whether that sentence carries a definition cue, so the + not. Each candidate carries its uses (every occurrence, a sentence that + joins the pair twice counting twice), the sections it appears in, its first-use + line (the pair's first appearance in the raw unit, else the unit's start) + and whether that sentence glosses it where it uses it (`glossed`), so the author can add the definition at first use or choose the field's word. No sentence-level gate and no percentile: this is a list to walk, not a verdict, and it is off by default in the unified linter. @@ -337,7 +400,8 @@ def glossary_findings(text: str, field_profile_dir: Path | None, for start, _end, bucket, block in reference.units(text): for sentence in es.sentences(es.latex_to_plain(block)): excerpt = " ".join(sentence.split()) - for pair in _judged(dict.fromkeys(content_pairs(sentence)), bank): + occurrences = Counter(content_pairs(sentence)) + for pair in _judged(occurrences, bank): if _attestations(pair, bank) != 0: continue key = f"{pair[0]} {pair[1]}" @@ -348,10 +412,10 @@ def glossary_findings(text: str, field_profile_dir: Path | None, "first_line": _line_of_pair(block, start, pair), "first_section": bucket, "first_sentence": excerpt[:160], - "glossed_at_first_use": bool(RE_GLOSS_CUE.search(excerpt)), + "glossed_at_first_use": glossed(excerpt, pair), "expected_copresent_passages": round( expected_cooccurrence(pair, bank), 2)} - item["uses"] += 1 + item["uses"] += occurrences[pair] if bucket not in item["sections"]: item["sections"].append(bucket) n_passages = int(bank["n_passages"]) @@ -359,8 +423,8 @@ def glossary_findings(text: str, field_profile_dir: Path | None, for key, item in sorted(seen.items(), key=lambda kv: (-kv[1]["uses"], kv[0])): if item["uses"] < GLOSSARY_MIN_USES: continue - cue = ("a definition cue in that sentence" if item["glossed_at_first_use"] - else "no definition cue in that sentence") + cue = ("a definition cue follows it there" if item["glossed_at_first_use"] + else "no definition cue follows it there") message = (f"'{key}' is joined {item['uses']} times in the manuscript " f"({', '.join(item['sections'])}) and in none of the field's " f"{n_passages:,} passages; first use at line {item['first_line']}, {cue}.") @@ -416,9 +480,15 @@ def build_bank(field_profile_dir: Path) -> dict[str, Any]: "pair_df": dict(sorted(pair_df.items()))} -def calibrate(field_profile_dir: Path) -> dict[str, Any]: - """Write the pair bank, then the per-bucket sentence reference (leave-one-out).""" +def calibrate(field_profile_dir: Path) -> dict[str, Any] | None: + """Write the pair bank, then the per-bucket sentence reference (leave-one-out). + + None, with nothing written, when the banks hold no passage: an empty pair + bank is not a calibration, and `axis_main` reports exit 2. + """ bank = build_bank(field_profile_dir) + if not bank["n_passages"]: + return None (field_profile_dir / BANK_FILENAME).write_text( json.dumps(bank, separators=(",", ":"), sort_keys=True), encoding="utf-8") collected: dict[str, list[float]] = {} @@ -427,7 +497,7 @@ def calibrate(field_profile_dir: Path) -> dict[str, Any]: verdict = judge_sentence(sentence, bank, own_passage=True) if verdict is not None: collected.setdefault(bucket, []).append(verdict[FEATURE]) - baseline = {bucket: {"n": len(values), "unit": "sentence", + baseline = {bucket: {"n": len(values), "unit": UNIT, "sources": ["leave-one-out over the passage banks"], "percentiles": {FEATURE: reference.quantiles(values)}} for bucket, values in collected.items()} @@ -456,18 +526,22 @@ def _report(text: str, field_dir: Path | None, path: str | Path) -> dict[str, An def _glossary_report(text: str, field_dir: Path | None, path: str | Path) -> dict[str, Any]: findings = glossary_findings(text, field_dir, path) - status = collocation_axis_status(field_dir, text) - return feedback.build_report(path=path, findings=findings, axes=[status]) + return feedback.build_report(path=path, findings=findings, + axes=[glossary_axis_status(field_dir)]) def main(argv: list[str] | None = None) -> int: - args = list(sys.argv[1:] if argv is None else argv) - glossary = "--glossary" in args - args = [arg for arg in args if arg != "--glossary"] - return cli_common.axis_main(__doc__, args, tool="deai_collocation", - calibrate=calibrate, summary=_written, - report=_glossary_report if glossary else _report, - render=feedback.render_text) + return cli_common.axis_main( + __doc__, argv, tool="deai_collocation", calibrate=calibrate, + summary=_written, report=_report, render=feedback.render_text, + extra_arguments=lambda parser: parser.add_argument( + "--glossary", action="store_true", + help="list the recurring unattested pairs (document scope) instead " + "of the per-sentence findings"), + check=lambda args: ("--glossary reads a document; it cannot be combined " + "with --calibrate" if args.glossary and args.calibrate + else None), + report_for=lambda args: _glossary_report if args.glossary else _report) if __name__ == "__main__": diff --git a/tools/deai_discourse.py b/tools/deai_discourse.py index d3c6904..07a3038 100644 --- a/tools/deai_discourse.py +++ b/tools/deai_discourse.py @@ -202,17 +202,19 @@ def resolves_below_gate(ref: dict[str, Any], feature: str) -> bool: def live_buckets(feature: str, field_profile_dir: Path | None) -> list[str]: """The buckets this axis may speak about here, or [] if it may not speak. - Three conditions, all necessary: the bucket is one the axis is calibrated - for, its reference clears the sample floor, and that reference has spread - below the gate. Detection and status read this same function, so an axis - cannot report `measured` for a bucket it then declines to judge. + Four conditions, all necessary: the bucket is one the axis is calibrated + for, its artifact records the unit this axis measures at, its reference + clears the sample floor, and that reference has spread below the gate. + Detection and status read this same function, so an axis cannot report + `measured` for a bucket it then declines to judge. """ baseline = LOADERS[feature](field_profile_dir) if baseline is None: return [] allowed = AXES[feature]["buckets"] return [bucket for bucket in reference.usable_buckets( - baseline, feature, ADVISORY_PERCENTILE, high=False) + baseline, feature, ADVISORY_PERCENTILE, high=False, + unit=AXES[feature]["unit"]) if allowed is None or bucket in allowed] @@ -228,30 +230,40 @@ def axis_name(feature: str) -> str: return f"L2.{feature}" -def discourse_axis_status(field_profile_dir: Path | None) -> list[dict[str, Any]]: +def discourse_axis_status(field_profile_dir: Path | None, + text: str | None = None) -> list[dict[str, Any]]: """One status per feature: the two travel together but fail separately. They calibrate from the same corpus at different granularities, so a field can support one and not the other -- and does: `abstract` resolves for cohesion and never for hedging. A single joint status would hide which of the two a reader is allowed to act on. + + `text` makes each status about this document: a unit in the `unknown` + bucket has no reference and is skipped, and a status computed from the + profile alone said `measured` over a document skipped whole. """ statuses = [] for feature, axis in AXES.items(): name = axis_name(feature) - if LOADERS[feature](field_profile_dir) is None: + baseline = LOADERS[feature](field_profile_dir) + if baseline is None: statuses.append(feedback.axis_status( name, "unmeasured", reason=f"{axis['baseline']} is unavailable", detector="deai_discourse")) continue live = live_buckets(feature, field_profile_dir) - if not live: + reasons = [None if live else + (f"no calibrated {axis['unit']} bucket both reaches the " + f"{MIN_REFERENCE_N}-unit floor and has spread below the " + "advisory gate"), + reference.unit_reason(baseline, axis["unit"])] + if text is not None: + reasons.append(reference.unbucketed_reason(axis["spans"](text), axis["unit"])) + if any(reasons): statuses.append(feedback.axis_status( - name, "degraded", - reason=(f"no calibrated {axis['unit']} bucket both reaches the " - f"{MIN_REFERENCE_N}-unit floor and has spread below the " - "advisory gate"), + name, "degraded", reason="; ".join(part for part in reasons if part), detector="deai_discourse")) else: statuses.append(feedback.axis_status( @@ -263,9 +275,14 @@ def discourse_axis_status(field_profile_dir: Path | None) -> list[dict[str, Any] def _advisory(feature: str, value: float, values: dict[str, Any], found: float, *, bucket: str, start: int, end: int, path: str | Path | None, - ref: dict[str, Any], reference_n: int, measured: bool, + ref: dict[str, Any], reference_n: int, field_profile_dir: Path | None) -> dict[str, Any]: - """The finding for one unit on one feature.""" + """The finding for one unit on one feature. + + Always `measured`: a bucket reaches this only through `live_buckets`, + which requires the sample floor, so the degraded branch this once carried + could not be taken and was deleted rather than kept as a promise. + """ axis = AXES[feature] return feedback.make_finding( kind="advisory", layer="L2", @@ -273,9 +290,8 @@ def _advisory(feature: str, value: float, values: dict[str, Any], found: float, calibration_unit=axis["unit"], line=start, end_line=end, section=bucket, path=path, detector="deai_discourse", - measurement_status="measured" if measured else "degraded", - strength=("strong" if measured and found < STRONG_PERCENTILE - else "ordinary"), + measurement_status="measured", + strength="strong" if found < STRONG_PERCENTILE else "ordinary", observed=dict(values, feature=feature, value=round(value, 4), percentile=round(found, 4)), reference=feedback.reference_block( @@ -322,22 +338,28 @@ def discourse_findings(text: str, field_profile_dir: Path | None, found = reference.percentile_of(ref, feature, value) if found is None or found >= ADVISORY_PERCENTILE: continue - reference_n = int(ref.get("n", 0)) findings.append(_advisory( feature, value, values, found, bucket=bucket, start=start, - end=end, path=path, ref=ref, reference_n=reference_n, - measured=reference_n >= MIN_REFERENCE_N, + end=end, path=path, ref=ref, reference_n=int(ref.get("n", 0)), field_profile_dir=field_profile_dir)) return findings -def calibrate(field_profile_dir: Path) -> dict[str, Any]: - """Build both references, each at its own granularity.""" +def calibrate(field_profile_dir: Path) -> dict[str, Any] | None: + """Build both references, each at its own granularity. + + An axis whose sweep read no record (a bank without `source` cannot form + a section) comes back as `{}` and writes nothing; None, and exit 2 from + `axis_main`, only when neither axis read a record. + """ banks = reference.passage_banks(field_profile_dir) - return {feature: reference.calibrate( - field_profile_dir, axis["baseline"], (feature,), - axis["extract"], banks, unit=axis["unit"]) - for feature, axis in AXES.items()} + results = {feature: reference.calibrate( + field_profile_dir, axis["baseline"], (feature,), + axis["extract"], banks, unit=axis["unit"]) + for feature, axis in AXES.items()} + if all(result is None for result in results.values()): + return None + return {feature: result or {} for feature, result in results.items()} def _written(result: dict[str, Any], field_profile_dir: Path) -> str: @@ -349,7 +371,8 @@ def _written(result: dict[str, Any], field_profile_dir: Path) -> str: """ return " | ".join( f"{AXES[feature]['baseline']} ({AXES[feature]['unit']}: " - + ", ".join(f"{bucket}={ref['n']}" for bucket, ref in sorted(buckets.items())) + + (", ".join(f"{bucket}={ref['n']}" for bucket, ref in sorted(buckets.items())) + or "no reference record read, nothing written") + ")" for feature, buckets in result.items()) @@ -359,7 +382,7 @@ def main(argv: list[str] | None = None) -> int: summary=_written, report=lambda text, field_dir, path: feedback.build_report( path=path, findings=discourse_findings(text, field_dir, path), - axes=discourse_axis_status(field_dir)), + axes=discourse_axis_status(field_dir, text)), render=feedback.render_text) diff --git a/tools/deai_oracle.py b/tools/deai_oracle.py index d46b603..a6c0231 100644 --- a/tools/deai_oracle.py +++ b/tools/deai_oracle.py @@ -23,7 +23,6 @@ from __future__ import annotations -import argparse import json import statistics import sys @@ -35,8 +34,6 @@ import deai_reference as reference # noqa: E402 because sibling tools are importable only after the sys.path insert above import extract_style as es # noqa: E402 same reason -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" BASELINE_NAME = "uid_baseline.json" # Retained for compatibility with the recorded UID baseline and learned bundle. @@ -67,6 +64,16 @@ def _get_model(model_name: str, device: str | None = None): _GPU_FAILURE_LIMIT = 3 +def _first_line(error: BaseException) -> str: + """The first line of an exception's message, or `?` when it has none. + + A CUDA error can carry an empty message, and `str(exc).splitlines()[0]` + then raised IndexError inside the handler that existed to keep the run + alive -- the fallback died on the way to the fallback. + """ + return (str(error).splitlines() or ["?"])[0] + + def _surprisals(ids, model): import torch with torch.no_grad(): @@ -101,7 +108,7 @@ def token_surprisals(text: str, model_name: str = DEFAULT_MODEL) -> list[float]: # than probing a broken context once per paragraph. _GPU_FAILURES += 1 print(f"[deai_oracle] GPU scoring failed ({type(exc).__name__}: " - f"{str(exc).splitlines()[0]}); scoring on the CPU " + f"{_first_line(exc)}); scoring on the CPU " f"(failure {_GPU_FAILURES}/{_GPU_FAILURE_LIMIT})", file=sys.stderr) _tok, model_cpu, _dev = _get_model(model_name, device="cpu") @@ -133,7 +140,14 @@ def calibrate(field_profile_dir: Path, model_name: str = DEFAULT_MODEL) -> dict: n_used = 0 with exemplars.open(encoding="utf-8") as f: for line in f: - rec = json.loads(line) + # A blank or undecodable line is skipped, as `deai_reference` and + # `deai_register` skip it; here it aborted an hour-long run. + if not line.strip(): + continue + try: + rec = json.loads(line) + except json.JSONDecodeError: + continue text = rec.get("text", "") bucket = rec.get("section", "unknown") feats = uid_features(token_surprisals(text, model_name)) @@ -295,15 +309,9 @@ def main(argv: list[str] | None = None) -> int: "to style-profile//uid_baseline.json.") args = p.parse_args(argv) - # Resolve field dir. - field_dir = None - if args.field: - field_dir = args.profile_root / args.field - elif args.profile_root.exists(): - fields = [d for d in args.profile_root.iterdir() - if d.is_dir() and not d.name.startswith(".")] - if len(fields) == 1: - field_dir = fields[0] + # Unlike the axis tools this one cannot run without a profile: a + # calibration needs somewhere to write and a score needs a baseline. + field_dir = cli_common.optional_field_dir(args, tool="deai_oracle") if field_dir is None: print("[deai_oracle] need --field (multiple/zero profiles).", file=sys.stderr) return 2 @@ -321,7 +329,13 @@ def main(argv: list[str] | None = None) -> int: model_name = args.model or DEFAULT_MODEL print(f"[deai_oracle] calibrating UID baseline on {field_dir.name} " f"with {model_name} ...", file=sys.stderr) - b = calibrate(field_dir, model_name) + try: + b = calibrate(field_dir, model_name) + except FileNotFoundError as error: + # A --field with no bank is configuration, exit 2, as + # `cli_common.axis_main` reports it; it was a traceback (exit 1). + print(f"[deai_oracle] cannot calibrate: {error}", file=sys.stderr) + return 2 print(f"[deai_oracle] baseline written: {b['n_paragraphs_used']} " f"paragraphs, {len(b['by_section'])} sections, model {b['model']}.") for sec, feats in sorted(b["by_section"].items()): diff --git a/tools/deai_personal.py b/tools/deai_personal.py index 0419b0b..dfe90f6 100644 --- a/tools/deai_personal.py +++ b/tools/deai_personal.py @@ -23,7 +23,6 @@ """ from __future__ import annotations -import argparse import statistics import sys from pathlib import Path diff --git a/tools/deai_provenance.py b/tools/deai_provenance.py index fe06fc4..82f37d8 100644 --- a/tools/deai_provenance.py +++ b/tools/deai_provenance.py @@ -26,7 +26,6 @@ """ from __future__ import annotations -import argparse import difflib import subprocess import sys @@ -37,6 +36,7 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first import extract_style as es # noqa: E402 resolves only after the sys.path insert import deai_feedback as feedback # noqa: E402 shared finding contract +import deai_reference as reference # noqa: E402 the shared paragraph sweep MIN_WORDS = 12 # ignore trivial fragments (titles, stray lines) # Similarity (difflib token ratio, in [0, 1]) buckets. Higher similarity to the @@ -50,27 +50,17 @@ def _paragraphs(text: str) -> list[tuple[int, int, str]]: - """(start_line, end_line, block) per blank-line paragraph with enough prose - words to carry a claim; line numbers are 1-based source lines so a finding's - location points at the real paragraph, matching the sibling detectors.""" - lines = text.splitlines() - out: list[tuple[int, int, str]] = [] - index = 0 - while index < len(lines): - while index < len(lines) and not lines[index].strip(): - index += 1 - if index >= len(lines): - break - start = index + 1 # 1-based first line of the block - block_lines = [] - while index < len(lines) and lines[index].strip(): - block_lines.append(lines[index]) - index += 1 - end = index # 1-based last non-blank line - block = "\n".join(block_lines).strip() - if block and len(es.words(es.latex_to_plain(block))) >= MIN_WORDS: - out.append((start, end, block)) - return out + """(start_line, end_line, block) per prose paragraph with enough words to + carry a claim, from the sweep every paragraph axis reads + (`deai_reference.paragraphs`): the preamble is not a paragraph, so a + `\\author{...}\\affiliation{...}` block no longer enters the ledger as an + `ai_untouched` span, and a `skip` section (acknowledgments, appendix) is + not prose the author has to own. Line numbers are 1-based source lines, so + a finding's location points at the real paragraph.""" + return [(start, end, block) + for start, end, _label, bucket, block in reference.paragraphs(text) + if bucket != "skip" + and len(es.words(es.latex_to_plain(block))) >= MIN_WORDS] def _tokens(text: str) -> list[str]: @@ -190,7 +180,8 @@ def git_file_at(path: Path, ref: str) -> str | None: def document_findings(current_text: str, path: str | Path | None, ancestor_text: str | None, - no_ancestor_reason: str | None = None + no_ancestor_reason: str | None = None, + result: dict[str, Any] | None = None, ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: """Findings + axis status for the provenance ledger. @@ -200,6 +191,8 @@ def document_findings(current_text: str, path: str | Path | None, so paragraph-unit findings here are not capped. With no ancestor the axis is ``unmeasured`` and no finding is emitted; ``no_ancestor_reason`` lets the caller distinguish "none supplied" from "supplied but unreadable at that ref". + ``result`` is a ledger the caller already built with ``document_provenance`` + for the same pair, so the CLI's text output does not build it twice. """ axis_name = "L4.editing_provenance" if ancestor_text is None: @@ -208,7 +201,8 @@ def document_findings(current_text: str, path: str | Path | None, reason=(no_ancestor_reason or "no AI-draft ancestor supplied (--ai-ancestor / --git-ai-ref)"), detector="deai_provenance")] - result = document_provenance(current_text, ancestor_text) + if result is None: + result = document_provenance(current_text, ancestor_text) if result["status"] != "measured": return [], [feedback.axis_status(axis_name, "unmeasured", reason=result["reason"], @@ -273,8 +267,10 @@ def main(argv: list[str] | None = None) -> int: f"untracked for this file") print(f"[deai_provenance] could not read {args.file} at git ref " f"{args.git_ai_ref}; provenance is unmeasured", file=sys.stderr) + result = document_provenance(current, ancestor) if ancestor is not None else None findings, axes = document_findings(current, args.file, ancestor, - no_ancestor_reason=no_ancestor_reason) + no_ancestor_reason=no_ancestor_reason, + result=result) if args.format == "json": report = feedback.build_report(path=args.file, findings=findings, axes=axes) print(feedback.dump_report(report), end="") @@ -282,11 +278,9 @@ def main(argv: list[str] | None = None) -> int: for axis in axes: detail = f": {axis['reason']}" if axis.get("reason") else "" print(f"axis {axis['axis']}: {axis['status']}{detail}") - if ancestor is not None: - result = document_provenance(current, ancestor) - if result["status"] == "measured": - for label, count in result["summary"]["label_counts"].items(): - print(f" {label}: {count}") + if result is not None and result["status"] == "measured": + for label, count in result["summary"]["label_counts"].items(): + print(f" {label}: {count}") return 0 diff --git a/tools/deai_reference.py b/tools/deai_reference.py index 1279caa..0cabc71 100644 --- a/tools/deai_reference.py +++ b/tools/deai_reference.py @@ -21,7 +21,9 @@ from __future__ import annotations import json +import re import sys +from collections import Counter from pathlib import Path from typing import Any, Callable, Iterable @@ -40,13 +42,29 @@ def quantiles(values: list[float]) -> dict[str, float]: - """The stored grid for one feature's reference values.""" + """The stored grid for one feature's reference values. + + Nearest rank: the point stored at q is the ceil(q * n)-th smallest value, + the first order statistic with P(X <= x) >= q; q = 0 stores the minimum + and q = 1 the maximum. The rank is computed in integers (a grid point is + a whole hundredth) because 0.9 * 100 is 90.00000000000001 in floating + point, and a ceiling taken there reads one rank high again. + + Until 2026-09-27 the index was floor(q * n), one order statistic further + up: the stored p90 of 100 values was the 91st smallest (P(X <= x) = 0.91), + and at the 30-unit floor the stored p10 and p90 sat at P = 0.133 and + 0.933. Every artifact calibrated before that date, and every held-out + rate measured against one, was read one order statistic high -- widest + at the smallest buckets, which is where the floor matters. Recalibrate + to move the grid; the artifacts do not move on their own. + """ ordered = sorted(values) - if not ordered: + n = len(ordered) + if not n: return {} out: dict[str, float] = {} - for q in QUANTILE_GRID: - index = min(len(ordered) - 1, int(q * len(ordered))) + for step, q in enumerate(QUANTILE_GRID): + index = min(n - 1, max(0, (step * n + 99) // 100 - 1)) out[str(q)] = round(float(ordered[index]), 6) return out @@ -95,8 +113,6 @@ def percentile_of(reference: dict[str, Any], feature: str, return 1.0 q_low, v_low = below[-1] q_high, v_high = above[0] - if v_high == v_low: - return q_high span = (value - v_low) / (v_high - v_low) return q_low + span * (q_high - q_low) @@ -128,17 +144,69 @@ def resolves_gate(reference: dict[str, Any], feature: str, gate: float, *, return at_edge > at_gate if high else at_gate > at_edge +def unit_matches(reference: dict[str, Any], unit: str) -> bool: + """Whether one bucket's artifact was built at the unit the caller measures.""" + return reference.get("unit") == unit + + +def foreign_units(baseline: dict[str, Any] | None, unit: str) -> dict[str, str]: + """bucket -> the unit its artifact records, for every bucket NOT built at `unit`. + + `calibrate` has written `unit` into every bucket since the module existed + and nothing read it back: a mislabelled or stale artifact -- a hedging + reference rebuilt at paragraph unit, say -- was read against section + measurements without a word in the output. The invariant this module is + for was a declaration until the reader checked it. An artifact with no + `unit` key predates the key and is reported as `unrecorded`. + """ + return {bucket: str(reference.get("unit") or "unrecorded") + for bucket, reference in (baseline or {}).items() + if isinstance(reference, dict) and not unit_matches(reference, unit)} + + +def unit_reason(baseline: dict[str, Any] | None, unit: str) -> str | None: + """The status text for buckets an axis refuses because of their unit, or None.""" + foreign = foreign_units(baseline, unit) + if not foreign: + return None + listed = ", ".join(f"{bucket} ({recorded})" for bucket, recorded in sorted(foreign.items())) + return (f"bucket(s) recorded at another unit are not read: {listed}; this axis " + f"measures {unit} units, so recalibrate") + + def usable_buckets(baseline: dict[str, Any] | None, feature: str, gate: float, *, - high: bool) -> list[str]: - """Buckets that clear the sample floor AND resolve on the flagged side.""" + high: bool, unit: str) -> list[str]: + """Buckets built at `unit` that clear the sample floor AND resolve on the flagged side.""" if not baseline: return [] return [bucket for bucket, reference in baseline.items() if isinstance(reference, dict) + and unit_matches(reference, unit) and int(reference.get("n", 0)) >= MIN_REFERENCE_N and resolves_gate(reference, feature, gate, high=high)] +def unbucketed_reason(spans: Iterable[tuple[int, int, str, Any]], unit: str) -> str | None: + """Why part of a document went unmeasured: its units in the `unknown` bucket. + + A document with no heading, or with headings that name a topic rather than + a role (`\\section{Weak lensing}`), lands in `unknown`, and no bank holds + that bucket, so every per-bucket axis skips such a unit. Until 2026-09-27 + an axis status was computed from the profile alone and reported `measured` + with zero findings on a document it had not read (CLAUDE.md rule 3). Each + status function now takes the text and reports this count, so "nothing + found" and "nothing measured" stay two different sentences. + """ + spans = list(spans) + unknown = sum(1 for _start, _end, bucket, _block in spans + if bucket == es.DEFAULT_SECTION_BUCKET) + if not unknown: + return None + return (f"{unknown} of {len(spans)} {unit} units carry no calibrated bucket " + "(no section heading, or one naming a topic rather than a role) " + "and were not measured") + + def _abstracts(text: str) -> list[tuple[int, int, str, str]]: """The abstract environments, which the section sweep cannot see. @@ -185,7 +253,7 @@ def without_headings(block: str) -> str: def sections(text: str) -> list[tuple[int, int, str, str]]: - """(start_line, end_line, bucket, block) for every SECTION-sized unit. + """(start_line, end_line, bucket, block): one SECTION-sized unit per bucket. The coarser sibling of :func:`units`. A feature whose paragraph-scale distribution has no lower tail can still have one here -- more than a tenth @@ -193,18 +261,34 @@ def sections(text: str) -> list[tuple[int, int, str, str]]: section that hedges nowhere is genuinely unusual. Which granularity an axis calibrates and detects at is the axis's choice; mixing the two is the one thing this module exists to prevent. + + One unit per BUCKET, not per heading. The reference side pools every + paragraph a paper holds in a bucket into one section (`_section_records`), + because that is how the corpus splitter cuts a paper -- it keys a paper's + text by bucket -- so a manuscript's four Methods subsections are one unit + here as well. Read per heading span until 2026-09-27, a 150-word + subsection was compared with references pooled over whole papers: the + mixing of granularities this module exists to prevent, on the side where + nothing recorded it. The span runs from the bucket's first line to its + last, and the block joins its spans in document order. """ - found = _abstracts(text) - consumed = {line for start, end, _, _ in found - for line in range(start, end + 1)} lines = text.splitlines() + spans: dict[str, list[tuple[int, int, str]]] = {} + consumed: set[int] = set() + for start, end, bucket, body in _abstracts(text): + spans.setdefault(bucket, []).append((start, end, body)) + consumed.update(range(start, end + 1)) for start, end, _label, bucket in _labelled_sections(text): body = without_headings("\n".join( line for number, line in enumerate(lines[start - 1:end], start) if number not in consumed)) - if has_prose(body): - found.append((start, end, bucket, body)) - return found + spans.setdefault(bucket, []).append((start, end, body)) + found = [] + for bucket, parts in spans.items(): + block = "\n\n".join(body for _start, _end, body in parts) + if has_prose(block): + found.append((parts[0][0], parts[-1][1], bucket, block)) + return sorted(found) def has_prose(block: str) -> bool: @@ -217,8 +301,12 @@ def has_prose(block: str) -> bool: them here was measured against a reference that holds none: GPT-2 read forty-eight spaces as more tokens than the UID minimum and scored four of the Letter's heading lines as paragraphs of near-zero surprisal variance. - A display-equation-only paragraph projects to `[MATH]` and stays a unit, - as it is a row on the corpus side. + A display-equation-only paragraph projects to `[MATH]` and stays a unit + here, although the corpus side writes no row for it (the bank writer drops + a placeholder-only paragraph and anything under 30 words): the axes with + no reference -- residue, the removal map -- still see it and every later + unit keeps its line number, while each calibrated axis's extractor returns + None on it (too few sentences or words) and measures nothing. """ return bool(es.latex_to_plain(block).strip()) @@ -252,6 +340,47 @@ def units(text: str) -> list[tuple[int, int, str, str]]: for start, end, _label, bucket, block in paragraphs(text)] +RE_PLACEHOLDER_TOKEN = re.compile(r"\[[A-Za-z\-]+\]") +# What may sit between two consecutive words of one projected sentence in the +# raw block: anything but a letter (space, line break, punctuation, digit), a +# command with its bracketed and braced arguments (a citation, a reference), +# a bare command name, or an inline math span. Never prose: a gap that could +# cross words matched `The map` from an earlier sentence's `The`. +_GAP = (r"(?:[^A-Za-z]|\\[A-Za-z]+\*?(?:\[[^\]\n]*\])?\{[^{}]*\}" + r"|\\[A-Za-z]+\*?|\$[^$\n]*\$){0,60}?") + + +def _word_run(words: list[str]) -> str: + """A pattern matching `words` in order with only markup between them.""" + return _GAP.join(rf"\b{re.escape(word)}\b" for word in words) + + +def sentence_lines(block: str, start: int, sentence: str) -> tuple[int, int]: + """(first, last) source line of one projected sentence inside its unit. + + A sentence-scope finding pointed at its PARAGRAPH's line range, so a reader + following `L 41` to a sentence on line 44 met the wrong one. The projected + sentence is looked up in the raw block by its first two words, then its + last three from that point (a command, a citation, a math span or a line + break may sit between any two, but no prose), and the lines are counted + from the unit's first line. A sentence whose head cannot be found keeps + the unit's whole range, and one whose tail cannot be found keeps the + unit's last line, so a finding is never anchored on a guess. + """ + words = es.words(RE_PLACEHOLDER_TOKEN.sub(" ", sentence)) + end = start + block.count("\n") + if not words: + return start, end + head = re.search(_word_run(words[:2]), block, re.IGNORECASE) + if head is None: + return start, end + first = start + block[:head.start()].count("\n") + tail = re.search(_word_run(words[-3:]), block[head.start():], re.IGNORECASE) + if tail is None: + return first, end + return first, first + block[head.start():head.start() + tail.end()].count("\n") + + def passage_banks(field_profile_dir: Path) -> list[tuple[str, Path, str | None]]: """(label, path, forced_bucket) for every bank of human passages in a field. @@ -269,13 +398,20 @@ def passage_banks(field_profile_dir: Path) -> list[tuple[str, Path, str | None]] ] +# Banks whose `text` is already the `[math]` projection. A record there +# without the projection an axis asks for can only be read on `text`, which +# is a different projection of the same paragraph; the abstract bank stores +# LaTeX source instead, which every key projects the same way. +PROJECTED_BANKS = frozenset({"exemplar_paragraphs"}) + + def _bank_records(sources: Iterable[tuple[str, Path, str | None]], *, - text_key: str = "text"): - """(label, bucket, text) for every readable record in every bank present. + text_key: str = "text", fallbacks: Counter[str] | None = None): + """(label, bucket, text, source) for every readable record in every bank present. `sources` is (label, path, forced_bucket): a bank whose records all belong to one bucket names it, because an abstract-only bank has no `section` key - to read. + to read. `source` is the record's own document id, or None. `text_key` names the projection the axis measures. The exemplar bank stores each paragraph twice, as `text` (math spans reduced to `[math]`) @@ -283,7 +419,10 @@ def _bank_records(sources: Iterable[tuple[str, Path, str | None]], *, on the manuscript must calibrate on the second, or its reference holds fewer numerals than any manuscript it reads. A record without the key falls back to `text`: the abstract bank stores its LaTeX source, which - the axis projects itself. + the axis projects itself. In a bank of `PROJECTED_BANKS` that fallback + mixes projections, and `fallbacks`, when given, counts such records per + bucket so the artifact records the mix and the axis can warn about it + instead of calibrating on it silently. """ for label, bank, forced_bucket in sources: if not bank.exists(): @@ -294,24 +433,29 @@ def _bank_records(sources: Iterable[tuple[str, Path, str | None]], *, record = json.loads(line) except json.JSONDecodeError: continue - yield (label, forced_bucket or record.get("section") or "unknown", - record.get(text_key) or record.get("text", ""), + bucket = forced_bucket or record.get("section") or "unknown" + if (fallbacks is not None and label in PROJECTED_BANKS + and text_key != "text" and not record.get(text_key)): + fallbacks[bucket] += 1 + yield (label, bucket, record.get(text_key) or record.get("text", ""), record.get("source")) def _section_records(sources: Iterable[tuple[str, Path, str | None]], *, - text_key: str = "text"): - """The same records regrouped so one whole section is one unit. + text_key: str = "text", fallbacks: Counter[str] | None = None): + """(label, bucket, text, source): the same records regrouped, one section each. The banks store paragraphs, so a section-unit reference has to be assembled from them: every paragraph sharing a source document and a bucket is one - section. A record with no `source` cannot be attributed to a document and is - dropped rather than pooled into a fictitious one -- pooling would join - paragraphs from unrelated papers into a section no author ever wrote. + section, its paragraphs joined by blank lines. A record with no `source` + cannot be attributed to a document and is dropped rather than pooled into + a fictitious one -- pooling would join paragraphs from unrelated papers + into a section no author ever wrote. """ grouped: dict[tuple[str, str], list[str]] = {} labels: dict[tuple[str, str], str] = {} - for label, bucket, text, source in _bank_records(sources, text_key=text_key): + for label, bucket, text, source in _bank_records(sources, text_key=text_key, + fallbacks=fallbacks): if not source: continue key = (source, bucket) @@ -324,24 +468,32 @@ def _section_records(sources: Iterable[tuple[str, Path, str | None]], *, def calibrate(field_profile_dir: Path, filename: str, features: Iterable[str], extract: Callable[[str], dict[str, Any] | None], sources: Iterable[tuple[str, Path, str | None]], *, - unit: str = "paragraph", text_key: str = "text") -> dict[str, Any]: + unit: str = "paragraph", text_key: str = "text") -> dict[str, Any] | None: """Build a per-bucket reference at one granularity from the field's banks. `unit` is written into every bucket of the artifact, because it is the fact - a later reader most needs and the one nothing else records: two references - built from the same corpus at different granularities are both valid and - are not comparable, and a detector reading the wrong one would compare a - value against a distribution that could not have produced it. + a later reader most needs: two references built from the same corpus at + different granularities are both valid and are not comparable, and a + detector reading the wrong one would compare a value against a + distribution that could not have produced it. `usable_buckets` reads it + back and refuses a bucket built at any other unit. `text_key` is the same invariant on the other axis: the reference must be built from the projection the detector measures (see `_bank_records`). + Each bucket records the key it was read on and `text_fallback_rows`, the + records of a projected bank that lacked it and were read on `text`. A record `extract` cannot measure is skipped rather than contributing a zero, which would pull every quantile toward a value no passage ever had. + None when no record contributed at all: nothing is written, because an + empty artifact reported as calibrated is a missing calibration wearing a + file name, and `cli_common.axis_main` turns None into exit 2. """ names = tuple(features) - stream = (_section_records(sources, text_key=text_key) if unit == "section" - else _bank_records(sources, text_key=text_key)) + fallbacks: Counter[str] = Counter() + stream = (_section_records(sources, text_key=text_key, fallbacks=fallbacks) + if unit == "section" + else _bank_records(sources, text_key=text_key, fallbacks=fallbacks)) collected: dict[str, dict[str, list[float]]] = {} contributing: dict[str, list[str]] = {} for label, bucket, text, _source in stream: @@ -354,11 +506,15 @@ def calibrate(field_profile_dir: Path, filename: str, features: Iterable[str], if label not in contributing.setdefault(bucket, []): contributing[bucket].append(label) + if not collected: + return None output: dict[str, Any] = {} for bucket, gathered in collected.items(): output[bucket] = { "n": len(gathered[names[0]]), "unit": unit, + "text_key": text_key, + "text_fallback_rows": fallbacks.get(bucket, 0), "sources": contributing.get(bucket, []), "percentiles": {name: quantiles(values) for name, values in gathered.items()}, diff --git a/tools/deai_register.py b/tools/deai_register.py index 54b811e..cac28be 100644 --- a/tools/deai_register.py +++ b/tools/deai_register.py @@ -22,10 +22,11 @@ name, or the field writes its stem and only the derived form is unattested. Those three are read mechanically and downgrade the finding to ordinary; "first use of this method in the field" is a disposition the author records. - This is a vocabulary AUDIT, not a detector, and the record says so: on 40 - held-out refereed papers the median is 2.57 zero-hit words per 1,000 against - 1.25 on machine documents, rank AUC 0.242 (EVALUATION §23) -- human papers - carry MORE unattested words. What it finds is register leakage a referee + This is a vocabulary AUDIT, not a detector, and the record says so: all + 203 held-out refereed papers carry zero-hit words, 3.37 per 1,000 body + prose words against 1.05 on 173 machine documents, rank AUC 0.174 under the + v0.36.2 projection (EVALUATION §23.1) -- human papers carry MORE unattested + words. What it finds is register leakage a referee would stop on: software vocabulary in an astronomy manuscript, or a coined term with no definition. * `register-foreign`: a term the manuscript leans on (`MIN_MANUSCRIPT_USES` @@ -69,13 +70,13 @@ import deai_metrics as metrics # noqa: E402 because sibling tools are importable only after the sys.path insert above import extract_style as es # noqa: E402 because sibling tools are importable only after the sys.path insert above -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" LEXICON_FILENAME = "register_lexicon.json" RE_WORD = re.compile(r"[A-Za-z][A-Za-z'\-]{2,}") +# `\newcommand*` is the same definition with a long-argument check; without +# the `\*?` a starred `\AUC` expanded to nothing, neither counted nor audited. RE_NEWCOMMAND = re.compile( - r"\\(?:newcommand|renewcommand|providecommand)\s*\{?\\([A-Za-z]+)\}?" + r"\\(?:newcommand|renewcommand|providecommand)\*?\s*\{?\\([A-Za-z]+)\}?" r"\s*(?:\[\d+\])?\s*\{(.*)") # A word-rendering macro body, with the character before it captured so a # subscript or superscript decoration can be told from a term. @@ -86,7 +87,6 @@ # bucket never sees it: it sits inside the span of whatever section precedes it. RE_BIB_ENV = re.compile(r"\\begin\{thebibliography\}.*?\\end\{thebibliography\}", re.DOTALL) -RE_BIBITEM = re.compile(r"^\s*\\bibitem\b", re.MULTILINE) # Mathematics never contributes vocabulary on the corpus side: the passage # projection replaces every math span with `[math]` before counting words. # Detection ran `latex_to_plain` one LINE at a time, so a `$...$` or an @@ -104,11 +104,12 @@ # stripper downstream still sees it. RE_FLOAT_OPTION = re.compile(r"(\\begin\{[A-Za-z*]+\})\[[^\]\n]*\]") # Length settings and table-note commands outside a float render no prose. -# Floats themselves are blanked whole in `body_only` (the fourth line seam, -# after RE_MATH_SPAN): the corpus side replaces `\begin{table}...\end{table}` -# with a placeholder in one pass over the passage, detection projected one -# LINE at a time, so a `tabular*` column specification and a caption's words -# counted on this side only -- `filllll`, `crimson`, `isosurfaces` at df 0, +# Floats themselves are blanked whole in `body_only` (the fourth seam in the +# count above, found after the RE_MATH_SPAN one): the corpus side replaces +# `\begin{table}...\end{table}` with a placeholder in one pass over the +# passage, detection projected one LINE at a time, so a `tabular*` column +# specification and a caption's words counted on this side only -- +# `filllll`, `crimson`, `isosurfaces` at df 0, # 14 of 90 zero-hit terms on one manuscript (2026-09-04). RE_LENGTH_CMD = re.compile( r"\\(?:setlength|addtolength|tabletypesize|tablewidth|tablenotemark" @@ -292,7 +293,7 @@ def macro_terms(text: str) -> dict[str, str]: return out -def body_only(text: str) -> str: +def body_only(text: str, *, drop_skip: bool = True) -> str: """Blank the regions the corpus side never saw, keeping line numbers. The corpus document frequency is built from `exemplar_paragraphs.jsonl`, @@ -320,9 +321,14 @@ def body_only(text: str) -> str: by character for the same reason (`extract_sections.blank_preserving`). Headings too: the corpus passages are the prose UNDER a heading, never its words, so a section title counted here and nowhere else. + + `drop_skip` is `body_lines`'s switch, passed through: the vocabulary + comparison drops the `skip` sections the corpus dropped, while a rule that + asks whether the PAPER names a thing (`deai_residue`'s labels) keeps them. """ body = es.blank_preserving( - body_lines(text), RE_MATH_SPAN, es.RE_TEX_ENV_FIGURE_TABLE, RE_LENGTH_CMD, + body_lines(text, drop_skip=drop_skip), RE_MATH_SPAN, + es.RE_TEX_ENV_FIGURE_TABLE, RE_LENGTH_CMD, RE_BIB_COMMAND, RE_CODE_SPAN, RE_HEADING, es.RE_TEX_CITE_SILENT, es.RE_TEX_CITE, es.RE_TEX_LABEL_REF) return RE_FLOAT_OPTION.sub( @@ -330,25 +336,30 @@ def body_only(text: str) -> str: body) -def body_lines(text: str) -> str: +def body_lines(text: str, *, drop_skip: bool = True) -> str: """The lines of `text` that are body prose, every other line emptied. - Drops the preamble, every `skip` section (bucket inherited, so a - `\\subsection` under Acknowledgments goes with it) and the bibliography, - and restores the abstract, which in AASTeX sits inside the preamble. What - it keeps is everything a reader SEES -- headings, captions, table cells - included -- which is the projection an editing-mark scan needs; the - vocabulary projection `body_only` blanks those on top of this. + Drops the preamble, the bibliography and, by default, every `skip` + section (bucket inherited, so a `\\subsection` under Acknowledgments goes + with it), and restores the abstract, which in AASTeX sits inside the + preamble. What it keeps is everything a reader SEES -- headings, captions, + table cells included -- which is the projection an editing-mark scan + needs; the vocabulary projection `body_only` blanks those on top of this. + + `drop_skip=False` keeps the `skip` sections. The corpus never held them, + so the vocabulary comparison must drop them; an editing mark is as left + behind in an appendix as in a paragraph, and `deai_residue` was blind to a + `TODO` there because it scanned this projection with the default. """ lines = text.splitlines() drop: set[int] = set() + dropped = (metrics.PREAMBLE_BUCKET, "skip") if drop_skip else (metrics.PREAMBLE_BUCKET,) for start, end, _label, bucket in metrics.section_units(text): - if bucket in (metrics.PREAMBLE_BUCKET, "skip"): + if bucket in dropped: drop.update(range(start, end + 1)) - for pattern in (RE_BIB_ENV, RE_BIBITEM): - for match in pattern.finditer(text): - first = text[:match.start()].count("\n") + 1 - drop.update(range(first, first + match.group(0).count("\n") + 1)) + for match in RE_BIB_ENV.finditer(text): + first = text[:match.start()].count("\n") + 1 + drop.update(range(first, first + match.group(0).count("\n") + 1)) for match in es.RE_ABSTRACT_ENV.finditer(text): first = text[:match.start()].count("\n") + 1 drop.difference_update( @@ -447,9 +458,10 @@ def zero_hit_context(text: str) -> dict[str, set[str]]: def defined_terms(sentence: str) -> set[str]: """The words a defining sentence defines: the object of each defining phrase. - A phrase of the `we define X` / `hereafter X` / `which we call X` kind - names its object AFTER it, up to the next clause edge; `X is defined as`, - `X, denoted Y` name it BEFORE. Only those words earn `defined-here`. + A phrase of the `we define X` / `hereafter X` / `which we call X` / + `denoted X` kind names its object AFTER it, up to the next clause edge; + `X is defined as` and `X is referred to as` name it BEFORE, back to the + previous clause edge. Only those words earn `defined-here`. """ found: set[str] = set() for match in RE_DEFINING.finditer(sentence): @@ -604,7 +616,9 @@ def register_findings(text: str, field_profile_dir: Path | None, "corpus_df_rate": round(rate, 8)}, reference=reference, normalized_distance=RARE_DF_RATE - rate, - confidence={"value": min(1.0, usage["count"] / 10.0), + # `min(1.0, uses / 10)` dated from the five-use floor; at fifteen + # uses and above it was 1.0 on every finding that reached it. + confidence={"value": 1.0, "basis": (f"{usage['count']} manuscript uses against " f"{df}/{n_passages} corpus passages")}, message=( @@ -618,8 +632,12 @@ def register_findings(text: str, field_profile_dir: Path | None, return findings -def calibrate(field_profile_dir: Path) -> dict[str, Any]: - """Count per-term document frequency across the field's own passage banks.""" +def calibrate(field_profile_dir: Path) -> dict[str, Any] | None: + """Count per-term document frequency across the field's own passage banks. + + None when no bank held a passage: nothing is written, since a lexicon of + zero passages is not a calibration, and `axis_main` reports exit 2. + """ banks = [field_profile_dir / "exemplar_paragraphs.jsonl", field_profile_dir / "human_abstracts_extra.jsonl"] document_frequency: Counter[str] = Counter() @@ -650,6 +668,8 @@ def calibrate(field_profile_dir: Path) -> dict[str, Any]: seen.add(part) document_frequency.update(seen) + if not n_passages: + return None payload = { "n_passages": n_passages, "n_terms": len(document_frequency), diff --git a/tools/deai_residue.py b/tools/deai_residue.py index 693d35e..4bbbfa2 100644 --- a/tools/deai_residue.py +++ b/tools/deai_residue.py @@ -1,8 +1,9 @@ """Edit-residue feedback for scientific prose (L4): traces an editing loop leaves behind. A manuscript revised in many passes accumulates marks that were true of the -process and are not true of the science. Four static rules and one diff rule -find them; none is an authorship claim and none touches the L0 exit status. +process and are not true of the science. Four static rules (1-4 below, in +that order) and one diff rule (5) find them; none is an authorship claim and +none touches the L0 exit status. - `residue-self-history` -- the text narrates its own drafting: `initially`, `we tried`, `no longer`, `superseded` in a sentence about *us* with no @@ -42,7 +43,7 @@ present, 2 invalid input. Both are mechanical gates of SCIPAPER_STANDARD section 5.3 (condense, do not accumulate): a patch is not a fix. -The rule-1 and rule-6 word families defined here are the SINGLE SOURCE. +The rule-1 and rule-4 word families defined here are the SINGLE SOURCE. `skills/paper/SKILL.md` lists them verbatim between `residue-family` and `absence-family` markers and `validate_plugin` compares the two, so the writer and the detector cannot drift apart on which words count. @@ -50,7 +51,6 @@ from __future__ import annotations -import argparse import re import sys from pathlib import Path @@ -92,7 +92,7 @@ re.I) CITATION_TOKEN = "[CITE]" -# Rule 6. The absence families (see the module docstring for the corpus +# Rule 4. The absence families (see the module docstring for the corpus # rates behind the tiers). `no ... is applied` is a template: `no` and an # object of one to three words, then a passive of the listed verbs. A # hyphenated compound (`never-touched controls`) is a name, not a predicate, @@ -169,9 +169,12 @@ def _absence_regex(family: tuple[str, ...]) -> re.Pattern[str]: # the marker to the next punctuation (a sentence end included: a caption is # several sentences, and "not a mass reconstruction. The solid spheres..." # once made `solid` the absent object) or coordinating word. +# `\s*` before the brace and the optional argument, as in +# `extract_sections.RE_HEADING_COMMAND`: LaTeX allows a space after a control +# word, and `\section {X}` was no label at all to this rule until 2026-09-27. RE_LABEL = re.compile( r"\\(?:chapter|section|subsection|subsubsection|paragraph|caption)\*?" - r"(?:\[[^\]]*\])?\{((?:[^{}]|\{[^{}]*\})*)\}") + r"\s*(?:\[[^\]]*\])?\s*\{((?:[^{}]|\{[^{}]*\})*)\}") RE_NEGATION = re.compile( r"\b(?:no|non|not|without|excluding|neither|nor|absent|minus|free of|lacking)" r"\b[-\s]*((?:[^,;:.!?()\[\]]|\[math\])+?)" @@ -193,23 +196,41 @@ def _content_stems(text: str) -> list[str]: return stems +# The -ate family first: one suffix off the register list left `truncation` +# as `trunc` and `truncated` as `truncat`, so a caption "without truncation" +# never found the body's "truncated at" and reported the object absent. +ATE_FAMILY = ("ations", "ation", "ating", "ated", "ates", "ate") + + def stem(word: str) -> str: - """One suffix off, when a root of three or more letters remains.""" - for suffix in register.STEM_SUFFIXES: + """One suffix off, when a root of three or more letters remains. + + The `-ation/-ated/-ating/-ate(s)` forms reduce to one root before the + general list is tried, because the label rule compares a caption's word + with the body's and those forms are how the same thing is named in each. + """ + for suffix in ATE_FAMILY + register.STEM_SUFFIXES: if word.endswith(suffix) and len(word) - len(suffix) >= 3: return word[: -len(suffix)] return word def _sentences(text: str) -> Iterable[tuple[int, int, str, str]]: - # Prose units only: the reference layer already drops the preamble, and a - # `skip` bucket (references, acknowledgments) is not prose an edit left. - for start, end, bucket, block in reference.units(text): + """(first_line, last_line, bucket, sentence) for every body sentence. + + Prose units only: the reference layer already drops the preamble, and a + `skip` bucket (references, acknowledgments) is not prose an edit left. The + lines are the sentence's own, located in the raw unit + (`reference.sentence_lines`); until 2026-09-27 they were the paragraph's. + """ + for start, _end, bucket, block in reference.units(text): if bucket == "skip": continue for sentence in es.sentences(es.latex_to_plain(block)): if sentence.strip(): - yield start, end, bucket, " ".join(sentence.split()) + sentence = " ".join(sentence.split()) + first, last = reference.sentence_lines(block, start, sentence) + yield first, last, bucket, sentence def _finding(*, rule: str, path: str | Path | None, line: int, end_line: int | None, @@ -268,7 +289,7 @@ def self_history_findings(text: str, path: str | Path | None = None) -> list[dic def absence_findings(text: str, path: str | Path | None = None) -> list[dict[str, Any]]: - """Rule 6. A sentence contrasting this work with cited work is a baseline + """Rule 4. A sentence contrasting this work with cited work is a baseline comparison, not a tombstone, so a citation in the sentence exempts it as in rule 1.""" findings = [] @@ -354,15 +375,28 @@ def negated_objects(label: str) -> list[tuple[str, list[str]]]: "remains. The negation itself is not the defect; the missing referent is.") +def paper_prose(text: str) -> str: + """The prose a label's object may be named in: `deai_register.body_only` + with the `skip` sections kept. + + Labels are read from the whole text, appendix captions included, so the + object has to be looked for in the whole paper too: with the vocabulary + projection (which drops `skip` for the corpus's sake) an appendix figure + "without the saddle correction" was reported absent from the body while + the appendix explained the correction three lines above it. The preamble + and the bibliography stay out -- neither is prose the reader meets.""" + return register.body_only(text, drop_skip=False) + + def negative_label_findings(text: str, path: str | Path | None = None, body_text: str | None = None) -> list[dict[str, Any]]: """Labels come from `text` (headings are blanked in the body projection); - the object is looked for in body prose only. Ordinary, not strong: on 203 + the object is looked for in `paper_prose`. Ordinary, not strong: on 203 refereed papers the static rule fires in 30% of documents, so it names a label to check, and the diff rule is the one that gates.""" if len(es.prose_words(text)) < MIN_WORDS_NEGATIVE_LABEL: return [] - body = _body_stems(register.body_only(text) if body_text is None else body_text) + body = _body_stems(paper_prose(text) if body_text is None else body_text) findings = [] for line, label in _labels(text): for obj, stems in negated_objects(label): @@ -395,8 +429,8 @@ def negative_label_added_findings(before: str, after: str, old_labels = {label for _line, label in _labels(before)} old_negations = {obj.lower() for label in old_labels for obj, _stems in negated_objects(label)} - old_body = _body_stems(register.body_only(before)) - new_body = _body_stems(register.body_only(after)) + old_body = _body_stems(paper_prose(before)) + new_body = _body_stems(paper_prose(after)) findings = [] for line, label in _labels(after): if label in old_labels: @@ -421,18 +455,20 @@ def negative_label_added_findings(before: str, after: str, def residue_findings(text: str, path: str | Path | None = None) -> list[dict[str, Any]]: - # Two projections of one line-drop (`deai_register.body_lines`): the - # preamble (`\newcommand{\TODO}`) and the bibliography are not prose an - # edit left behind, so both rules skip them. The label rule then reads the - # vocabulary projection (`body_only`, headings and floats blanked) because - # the object must be in the BODY; the mark rule reads the visible lines, - # because a `TODO` in a caption or a heading is as left behind as one in a - # paragraph and the vocabulary projection was hiding it. The sentence rule - # keeps the raw text because `reference.units` needs the headings. + # Two projections of one line-drop (`deai_register.body_lines` with the + # `skip` sections kept): the preamble (`\newcommand{\TODO}`) and the + # bibliography are not prose an edit left behind, so both rules skip them, + # while an appendix is -- read with the vocabulary default, which drops + # `skip` because the corpus did, a `TODO` in an appendix was invisible and + # the strong gate returned 0. The label rule reads `paper_prose` (headings + # and floats blanked) because the object must be in the paper's prose; the + # mark rule reads the visible lines, because a `TODO` in a caption or a + # heading is as left behind as one in a paragraph. The sentence rules keep + # the raw text because `reference.units` needs the headings. return (self_history_findings(text, path) + absence_findings(text, path) - + negative_label_findings(text, path, body_text=register.body_only(text)) - + edit_meta_findings(register.body_lines(text), path)) + + negative_label_findings(text, path, body_text=paper_prose(text)) + + edit_meta_findings(register.body_lines(text, drop_skip=False), path)) def residue_axis_status(text: str) -> dict[str, Any]: @@ -538,10 +574,16 @@ def main(argv: list[str] | None = None) -> int: "gate_exit": 1 if strong else 0} rendered = (feedback.dump_report(report) if args.format == "json" else feedback.render_text(report) + "\n") - if args.output is not None: - args.output.write_text(rendered, encoding="utf-8") - else: - print(rendered, end="") + try: + if args.output is not None: + args.output.write_text(rendered, encoding="utf-8") + else: + print(rendered, end="") + except OSError as error: + # An unwritable report is exit 2, not the exit 1 a traceback out of + # `write_text` used to hand the caller as if a strong finding stood. + print(f"[deai_residue] cannot write {args.output}: {error}", file=sys.stderr) + return 2 return report["residue_gate"]["gate_exit"] diff --git a/tools/deai_salience.py b/tools/deai_salience.py index 17e71f4..4d26565 100644 --- a/tools/deai_salience.py +++ b/tools/deai_salience.py @@ -48,6 +48,7 @@ # degraded: the percentile of a 12-passage reference is not an operating point. MIN_REFERENCE_N = reference.MIN_REFERENCE_N BASELINE_FILENAME = "salience_baseline.json" +UNIT = "paragraph" # The bank field holding each paragraph under the numeral-preserving projection. BANK_TEXT_KEY = "numeral_text" FEATURES = ("max_recital_run_frac", "recital_frac", "numerals_per_sentence") @@ -84,10 +85,7 @@ def salience_features(block: str) -> dict[str, Any] | None: load_baseline = reference.baseline_loader(BASELINE_FILENAME) percentile_of = reference.percentile_of -grid_value = reference.grid_value _units = reference.units -_quantiles = reference.quantiles -QUANTILE_GRID = reference.QUANTILE_GRID def resolves_above_gate(ref: dict[str, Any], feature: str) -> bool: @@ -95,7 +93,25 @@ def resolves_above_gate(ref: dict[str, Any], feature: str) -> bool: return reference.resolves_gate(ref, feature, ADVISORY_PERCENTILE, high=True) -def salience_axis_status(field_profile_dir: Path | None) -> dict[str, Any]: +def live_buckets(baseline: dict[str, Any] | None) -> list[str]: + """The buckets the axis may speak about as `measured`: built at this + unit, past the sample floor, and with spread above the gate on at least + one feature -- the condition detection itself applies per feature.""" + return sorted({bucket for feature in FEATURES + for bucket in reference.usable_buckets( + baseline, feature, ADVISORY_PERCENTILE, high=True, unit=UNIT)}) + + +def salience_axis_status(field_profile_dir: Path | None, + text: str | None = None) -> dict[str, Any]: + """The axis status; `degraded` names each reason it did not measure. + + Until 2026-09-27 the status looked only at the sample floor while + detection also required spread above the gate, so a flat reference read + `measured` with zero findings forever; and it never read the text, so a + document whose paragraphs sit in the `unknown` bucket -- skipped whole -- + read `measured` too. + """ baseline = load_baseline(field_profile_dir) if baseline is None: return feedback.axis_status( @@ -103,16 +119,21 @@ def salience_axis_status(field_profile_dir: Path | None) -> dict[str, Any]: reason=f"{BASELINE_FILENAME} is unavailable", detector="deai_salience", ) - usable = [bucket for bucket, ref in baseline.items() - if isinstance(ref, dict) and int(ref.get("n", 0)) >= MIN_REFERENCE_N] - if not usable: + live = live_buckets(baseline) + reasons = [None if live else + (f"no calibrated {UNIT} bucket both reaches the {MIN_REFERENCE_N}-unit " + "floor and has spread above the advisory gate"), + reference.unit_reason(baseline, UNIT)] + if text is not None: + reasons.append(reference.unbucketed_reason(_units(text), UNIT)) + if any(reasons): return feedback.axis_status( "L2.salience_hierarchy", "degraded", - reason=(f"no section bucket reaches the {MIN_REFERENCE_N}-passage " - "reference floor; percentiles are rank-only"), + reason="; ".join(part for part in reasons if part), detector="deai_salience", ) return feedback.axis_status("L2.salience_hierarchy", "measured", + reason=f"{UNIT} unit; buckets: {', '.join(live)}", detector="deai_salience") @@ -132,17 +153,19 @@ def salience_findings(text: str, field_profile_dir: Path | None, values = salience_features(block) if values is None: continue - reference = baseline.get(bucket) - if not isinstance(reference, dict): + ref = baseline.get(bucket) + # A bucket under the floor still speaks, as a degraded finding; one + # built at another unit does not speak at all (the status says why). + if not isinstance(ref, dict) or not reference.unit_matches(ref, UNIT): continue - reference_n = int(reference.get("n", 0)) + reference_n = int(ref.get("n", 0)) measured = reference_n >= MIN_REFERENCE_N percentiles: dict[str, float] = {} for feature in FEATURES: - if not resolves_above_gate(reference, feature): + if not resolves_above_gate(ref, feature): continue - found = percentile_of(reference, feature, float(values[feature])) + found = percentile_of(ref, feature, float(values[feature])) if found is not None: percentiles[feature] = found if not percentiles: @@ -157,8 +180,8 @@ def salience_findings(text: str, field_profile_dir: Path | None, for feature in FEATURES if feature in percentiles) findings.append(feedback.make_finding( kind="advisory", layer="L2", - rule=f"salience-recital:{bucket}", scope="paragraph", - calibration_unit="paragraph", + rule=f"salience-recital:{bucket}", scope=UNIT, + calibration_unit=UNIT, line=start, end_line=end, section=bucket, path=path, detector="deai_salience", measurement_status="measured" if measured else "degraded", @@ -195,7 +218,7 @@ def salience_findings(text: str, field_profile_dir: Path | None, return findings -def calibrate(field_profile_dir: Path) -> dict[str, Any]: +def calibrate(field_profile_dir: Path) -> dict[str, Any] | None: """Build the per-bucket human reference from the field's own passage banks. Calibration and detection share one unit (a passage), so a percentile means @@ -211,10 +234,28 @@ def calibrate(field_profile_dir: Path) -> dict[str, Any]: same papers' own bank rows fired at 0.35 under this projection and 0.13 under the bank's. `numeral_text` is the bank's copy of the projection this detector reads. + + A row of the exemplar bank without `numeral_text` is read on `text`, the + `[math]` projection, and the reference then holds fewer numerals for it + than the manuscript side would count. Each bucket of the artifact records + the count (`text_fallback_rows`) and this warns on stderr when any bucket + has one; the fix is to rebuild the bank with `extract_style`. None, and + nothing written, when no bank held a record. """ - return reference.calibrate( + written = reference.calibrate( field_profile_dir, BASELINE_FILENAME, FEATURES, salience_features, reference.passage_banks(field_profile_dir), text_key=BANK_TEXT_KEY) + if written is None: + return None + mixed = {bucket: ref["text_fallback_rows"] for bucket, ref in written.items() + if ref["text_fallback_rows"]} + if mixed: + listed = ", ".join(f"{bucket}={count}" for bucket, count in sorted(mixed.items())) + print(f"[deai_salience] WARNING: exemplar rows without {BANK_TEXT_KEY!r} " + f"were read on 'text', the [math] projection, so the reference " + f"under-counts their numerals ({listed}); rebuild the bank with " + "extract_style.py", file=sys.stderr) + return written def _written(result: dict[str, Any], field_profile_dir: Path) -> str: @@ -234,7 +275,7 @@ def main(argv: list[str] | None = None) -> int: summary=_written, report=lambda text, field_dir, path: feedback.build_report( path=path, findings=salience_findings(text, field_dir, path), - axes=[salience_axis_status(field_dir)]), + axes=[salience_axis_status(field_dir, text)]), render=feedback.render_text) From 3e13594a8c687d37a4dd0fc60a43a81871231f24 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:44:31 +0000 Subject: [PATCH 17/27] deai_voice: the L3 axis is unmeasured without the surprisal runtime, and --field resolves like every other tool deai_features.features_vector now raises when the optional transformers and torch runtime is missing, since three of the bundle's features are token surprisal. voice_score, voice_axis_status and voice_findings probe deai_oracle.model_runtime_available() first: the score is None, the findings are empty, and the axis reports `unmeasured` with the runtime's reason, so the linter, the rewrite ranker and --scores see a status instead of a RuntimeError. --scores prints that status line and exits 0. Also here, audit C1 and C18 of docs/audits/full-review-2026-09-27.md: --field resolves through cli_common.optional_field_dir, so a missing --field is the tool's exit 2 rather than a TypeError; --scores lists paragraphs by source line from the shared sweep, so a score can be matched to a finding; unused imports and the profile-root constants are removed. tests/test_deai_voice.py is new and takes over the field-resolution case that sat in tests/test_deai_oracle.py. tests/test_train_voice_model.py pins the runtime as present in its bundle-status test: it asserted the bundle's status on a machine without torch, where the axis is now `unmeasured` whatever the bundle says. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM (cherry picked from commit 0a50a3c0434e0b577af8442dde732fc503decd46) --- tests/test_deai_voice.py | 120 ++++++++++++++++++++++++++++++++ tests/test_train_voice_model.py | 8 ++- tools/deai_voice.py | 73 ++++++++++++++----- 3 files changed, 181 insertions(+), 20 deletions(-) create mode 100644 tests/test_deai_voice.py diff --git a/tests/test_deai_voice.py b/tests/test_deai_voice.py new file mode 100644 index 0000000..b339412 --- /dev/null +++ b/tests/test_deai_voice.py @@ -0,0 +1,120 @@ +"""`deai_voice` is the optional L3 triage: it degrades, it never crashes. + +Two things in front of it are optional: the trained bundle (joblib) and the +surprisal runtime (transformers/torch) that `deai_features.features_vector` +needs for three of the bundle's features. Without the bundle the axis was +already `unmeasured`; without the runtime `features_vector` raises, and until +2026-09-27 that RuntimeError went straight through `voice_score`, the +findings, the axis status and `--scores`. These tests pin that path to +`unmeasured` with the runtime's reason, check that a present runtime still +scores, and hold the `--field` resolution that once divided a path by None. +""" + +from __future__ import annotations + +import contextlib +import io +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path + +import deai_features as df +import deai_oracle as oracle +import deai_voice as voice + +# Two paragraphs of sixty words each: over the thirty-word floor of the sweep. +TEXT = ("\\section{Introduction}\n" + + ("The estimator uses five filters and reports the peak height. " * 6).strip() + + "\n\n\\section{Methods}\n" + + ("The shear catalog is measured around each cluster in the field. " * 6).strip() + + "\n") +MISSING = "the optional surprisal runtime is not installed (torch)" + + +def fake_bundle() -> dict: + """A bundle `load_voice_model` accepts: the installed feature schema and a + measured operating point, with no classifier (the scoring is patched).""" + return {"feature_names": list(df.FEATURE_NAMES), + "feature_schema": df.FEATURE_SCHEMA_VERSION, + "measurement_status": "measured", "operating_point": 0.5, + "model": "stub"} + + +class RuntimeProbeTests(unittest.TestCase): + def setUp(self): + self.raw = tempfile.TemporaryDirectory(prefix="voice-") + self.addCleanup(self.raw.cleanup) + self.root = Path(self.raw.name) / "profiles" + self.field = self.root / "fld" + self.field.mkdir(parents=True) + # The cache is the one way a bundle enters without joblib installed. + voice._MODEL_CACHE[str(self.field)] = fake_bundle() + self.addCleanup(voice._MODEL_CACHE.pop, str(self.field), None) + self.addCleanup(df._CENTROID_CACHE.pop, str(self.field), None) + + def test_without_the_runtime_the_axis_is_unmeasured_with_the_reason(self): + with mock.patch.object(oracle, "model_runtime_available", + return_value=(False, MISSING)): + self.assertIsNone(voice.voice_score(TEXT, self.field)) + self.assertEqual(voice.voice_findings(TEXT, self.field), []) + status = voice.voice_axis_status(self.field) + self.assertEqual(status["status"], "unmeasured") + self.assertIn(MISSING, status["reason"]) + self.assertIn("token surprisal", status["reason"]) + + def test_scores_without_the_runtime_prints_the_reason_and_exits_zero(self): + draft = Path(self.raw.name) / "draft.tex" + draft.write_text(TEXT, encoding="utf-8") + stdout = io.StringIO() + with mock.patch.object(oracle, "model_runtime_available", + return_value=(False, MISSING)), \ + contextlib.redirect_stdout(stdout): + code = voice.main([str(draft), "--field", "fld", "--profile-root", + str(self.root), "--scores"]) + self.assertEqual(code, 0) + self.assertIn(f"axis L3.voice: unmeasured: {MISSING}", stdout.getvalue()) + + def test_with_the_runtime_the_bundle_scores_and_flags(self): + # The probe must not block a present runtime: the vector and the + # classifier are stubbed, the rest of the path is real. + with mock.patch.object(oracle, "model_runtime_available", return_value=(True, "")), \ + mock.patch.object(df, "features_vector", + return_value=[0.0] * len(df.FEATURE_NAMES)), \ + mock.patch.object(voice, "_positive_class_probability", return_value=0.25): + self.assertEqual(voice.voice_score(TEXT, self.field), 0.25) + self.assertEqual(voice.voice_axis_status(self.field)["status"], "measured") + findings = voice.voice_findings(TEXT, self.field) + self.assertEqual([f["rule"] for f in findings], + ["voice-distance:intro", "voice-distance:method"]) + self.assertEqual({f["measurement_status"] for f in findings}, {"measured"}) + + +class FieldResolutionTests(unittest.TestCase): + """`--field` resolves through `cli_common.optional_field_dir`: none or + several profiles is the exit 2 the tool always printed for a missing + bundle, one resolves on its own, and `profile_root / None` is no longer a + TypeError.""" + + def test_without_a_field_is_exit_two_not_a_type_error(self): + with tempfile.TemporaryDirectory(prefix="voice-") as raw: + draft = Path(raw) / "draft.tex" + draft.write_text(TEXT, encoding="utf-8") + profiles = Path(raw) / "profiles" + (profiles / "one").mkdir(parents=True) + (profiles / "two").mkdir() + stderr = io.StringIO() + with contextlib.redirect_stderr(stderr): + self.assertEqual(voice.main([str(draft), "--profile-root", str(profiles)]), 2) + self.assertIn("no voice_model.joblib", stderr.getvalue()) + (profiles / "two").rmdir() + stderr = io.StringIO() + with contextlib.redirect_stderr(stderr): + self.assertEqual(voice.main([str(draft), "--profile-root", str(profiles)]), 2) + self.assertIn(str(profiles / "one"), stderr.getvalue()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_train_voice_model.py b/tests/test_train_voice_model.py index b5063d5..dbf228a 100644 --- a/tests/test_train_voice_model.py +++ b/tests/test_train_voice_model.py @@ -9,6 +9,7 @@ import types import unittest from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -339,7 +340,12 @@ def test_undefined_f1_is_excluded_from_aggregation(self): self.assertEqual(aggregated["overall"]["f1_positive"]["mean"], 1.0) def test_voice_axis_requires_measured_operating_point(self): - with tempfile.TemporaryDirectory() as temporary: + # The status under test is the bundle's, so the surprisal runtime the + # feature vector needs is pinned as present: without it the axis is + # `unmeasured` whatever the bundle says (tests/test_deai_voice.py). + with tempfile.TemporaryDirectory() as temporary, \ + mock.patch.object(voice.do, "model_runtime_available", + return_value=(True, "")): profile = Path(temporary) key = str(profile) try: diff --git a/tools/deai_voice.py b/tools/deai_voice.py index 14715f6..21f756f 100644 --- a/tools/deai_voice.py +++ b/tools/deai_voice.py @@ -7,7 +7,8 @@ is surfaced as rank-based triage (lowest-scoring paragraphs), never through a universal probability cutoff. - voice_score(text, field_dir) -> compatibility score in [0, 1] + voice_score(text, field_dir) -> compatibility score in [0, 1], or None + without a bundle or without the surprisal runtime the features need paragraph_hits(text, field_dir) -> compatibility tuple advisories The score is never a paper gate or detector-evasion objective. Rewrite candidates @@ -18,8 +19,6 @@ from __future__ import annotations -import argparse -import re import sys from pathlib import Path @@ -27,11 +26,10 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first import deai_features as df # noqa: E402 resolves only after the sys.path insert import deai_feedback as feedback # noqa: E402 shared finding contract +import deai_oracle as do # noqa: E402 the surprisal runtime probe import deai_reference as reference # noqa: E402 the shared paragraph sweep import extract_style as es # noqa: E402 same reason -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" TRIAGE_RANK_COUNT = 3 # degraded mode surfaces this many lowest-ranked paragraphs _MODEL_CACHE: dict = {} @@ -98,13 +96,35 @@ def _positive_class_probability(bundle, feats_vec) -> float: return float(clf.predict_proba(x)[0, column]) +def runtime_reason() -> str | None: + """Why no paragraph can be scored, or None when the runtime is present. + + The feature vector the bundle reads carries three token-surprisal entries, + and `deai_features.features_vector` raises when the optional + transformers/torch runtime that produces them is missing. Every entry + point here probes first, so the L3 axis reports `unmeasured` with this + reason instead of propagating that RuntimeError into the linter, the + rewrite ranker or `--scores`. + """ + available, reason = do.model_runtime_available() + if available: + return None + return (f"{reason}; the bundle's features include token surprisal, so no " + "paragraph can be scored") + + def voice_score(text: str, field_profile_dir: Path | None, model_name: str | None = None, centroid=None) -> float | None: - """Return a field-similarity compatibility score, or None if unavailable.""" + """Return a field-similarity compatibility score, or None if unavailable. + + None without a bundle and None without the surprisal runtime: the caller's + status names which (`voice_axis_status`), and a rewrite ranker reading + None weighs the term at nothing rather than at a nominal 0.0. + """ bundle = load_voice_model(field_profile_dir) - if bundle is None: + if bundle is None or runtime_reason() is not None: return None - model_name = model_name or bundle.get("model", df.do.DEFAULT_MODEL) + model_name = model_name or bundle.get("model", do.DEFAULT_MODEL) if centroid is None and field_profile_dir is not None: centroid = df.corpus_centroid(field_profile_dir) vec = df.features_vector(text, field_profile_dir=field_profile_dir, @@ -118,6 +138,10 @@ def voice_axis_status(field_profile_dir: Path | None) -> dict: return feedback.axis_status( "L3.voice", "unmeasured", reason="voice_model.joblib is unavailable", detector="deai_voice") + reason = runtime_reason() + if reason is not None: + return feedback.axis_status("L3.voice", "unmeasured", reason=reason, + detector="deai_voice") if not bundle_measured(bundle): return feedback.axis_status( "L3.voice", "degraded", @@ -172,10 +196,10 @@ def voice_findings(text: str, field_profile_dir: Path | None, probability cutoff is invented. """ bundle = load_voice_model(field_profile_dir) - if bundle is None: + if bundle is None or runtime_reason() is not None: return [] calibrated = bundle_measured(bundle) - model_name = bundle.get("model", df.do.DEFAULT_MODEL) + model_name = bundle.get("model", do.DEFAULT_MODEL) centroid = df.corpus_centroid(field_profile_dir) scored: list[dict] = [] for paragraph_start, paragraph_end, raw_label, bucket, block in reference.paragraphs(text): @@ -244,19 +268,30 @@ def main(argv: list[str] | None = None) -> int: p.add_argument("--scores", action="store_true", help="print the compatibility score for every paragraph") args = p.parse_args(argv) - field_dir = args.profile_root / args.field - if load_voice_model(field_dir) is None: - print(f"[deai_voice] no voice_model.joblib in {field_dir}; train with " - f"`python tools/train_voice_model.py --field {args.field}`.", - file=sys.stderr) + # No profile (no `--field` and not exactly one field) is the same exit 2 + # as a profile without a bundle; `profile_root / None` was a TypeError. + field_dir = cli_common.optional_field_dir(args, tool="deai_voice") + if field_dir is None or load_voice_model(field_dir) is None: + print(f"[deai_voice] no voice_model.joblib in {field_dir or ''}; " + "train with `python tools/train_voice_model.py --field " + f"{field_dir.name if field_dir else ''}`.", file=sys.stderr) return 2 text = args.file.read_text(encoding="utf-8", errors="replace") if args.scores: + status = voice_axis_status(field_dir) + if status["status"] == "unmeasured": + # The same line the report prints: without the surprisal runtime + # there is no score to list, and the reason is the output. + print(f"axis {status['axis']}: unmeasured: {status['reason']}") + return 0 + # The same sweep the findings use, keyed by source lines, so a score + # can be matched to a finding; a blank-line split of the raw text + # numbered different paragraphs and its indices matched nothing. centroid = df.corpus_centroid(field_dir) - for i, b in enumerate(x for x in re.split(r"\n\s*\n", text) if x.strip()): - s = voice_score(b, field_dir, None, centroid) - if s is not None: - print(f" para[{i}] field_similarity={s:.3f}") + for start, end, _label, _bucket, block in reference.paragraphs(text): + score = voice_score(block, field_dir, None, centroid) + if score is not None: + print(f" L{start}-{end} field_similarity={score:.3f}") return 0 report = feedback.build_report( path=args.file, From e5f69c422bbce2325c2b4c829f75e44e8fc88cb5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:49:29 +0000 Subject: [PATCH 18/27] release notes and record: the per-bucket axes, the suite count, and the audit's landing record MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v0.39.0 entry gains its paragraph for the per-bucket axes and closes with the suite as it stands (763 tests in 32 files; the validator's 11 checks pass). The recorded counts in both READMEs and the evaluation record are refreshed; the latency rows no longer quote the size of the suite they were timed on, which the registry check read as a claim. The audit note ends with a landing record: which items each commit covers, what §8.3 leaves open, and the baselines and artifacts that must be rebuilt or recalibrated before the EVALUATION figures move. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- CHANGELOG.md | 29 +++++++++++++++++++++++++-- README.md | 10 ++++----- README.zh-CN.md | 10 ++++----- docs/architecture/EVALUATION.md | 6 +++--- docs/audits/full-review-2026-09-27.md | 22 ++++++++++++++++++++ 5 files changed, 62 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 804e417..9a093df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -139,8 +139,33 @@ comment-led files are sources and colliding tarball basenames are both kept; omitting `--field` is a message and exit 2; the built-in query sets are marked `[WGL]` and `--query` serves any other field. -Tests and counts are stated at the end of this entry once every part of the -release has landed. +**The per-bucket axes read the grid at nearest rank and say what they +skipped.** `quantiles` stored the ⌊qn⌋+1-th smallest value, one order +statistic high: p90 of 100 values was the 91st, and at the 30-unit floor the +tails read at P = 0.133 and 0.933. It stores the ⌈qn⌉-th now; an artifact +calibrated before this keeps the old grid until it is recalibrated, and the +EVALUATION rates measured on the old grid predate it. An artifact recording +another unit than the axis measures is refused and named. A document whose +units fall in the `unknown` bucket, a bucket the baseline lacks, and a +salience reference no bucket resolves above are `degraded` with counts, where +the floor alone read `measured` with zero findings. The edit-meta residue +scan keeps the `skip` sections (an appendix TODO was invisible and the gate +exited 0), a caption's object is sought in the whole paper minus preamble and +bibliography, `\section {X}` labels are read, `truncation` and `truncated` +share one stem, `\newcommand*` is read, an unwritable `--output` is exit 2, +calibrating on no passage writes nothing and exits 2, and a bank row without +`numeral_text` is warned about instead of calibrating silently on the +`[math]` projection. `--glossary` goes through argparse, reads a definition +cue only after the pair, counts a pair joined twice in a sentence twice, and +its status depends on the bank alone; sentence findings carry the sentence's +own lines; the provenance ledger no longer lists a title block as a +paragraph; the oracle survives an empty CUDA message and a blank bank line. +The voice axis is `unmeasured` without the surprisal runtime instead of a +RuntimeError, `--scores` lists paragraphs by source line, and every one of +these tools resolves `--field` through the shared resolver. + +The suite is 763 tests in 32 files (v0.38.0: 529 in 27); the validator runs +11 checks and passes. ## v0.38.0 — 2026-09-16 diff --git a/README.md b/README.md index b11dbda..97db3cb 100644 --- a/README.md +++ b/README.md @@ -5,14 +5,14 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-529%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-763%20passing-success.svg)](tests/) **A Claude Code plugin that writes, reviews, de-AIs, and condenses scientific manuscripts for top-tier journals — under one typed standard, with every claim traced to a source and every unavailable measurement labelled as unavailable.** Built for ApJ / MNRAS / PRD / JCAP-class papers and NSF / NIH proposals. -**12 skills · 40 tools · 529 tests · one normative contract · zero authorship verdicts.** +**12 skills · 40 tools · 763 tests · one normative contract · zero authorship verdicts.** [中文文档](README.zh-CN.md) — [What it does](#what-it-does) · [How it works](#how-it-works) · [See it work](#see-it-work) · [Benchmarks](#benchmark-dashboard) · [Install](#install) · @@ -474,7 +474,7 @@ at 100%, and a number taken under that describes the contention, not the axis. V | `+ --oracle` (GPT-2-large token surprisal) | 33.8 s (2026-08-27) | `transformers` + `torch` | | `+ --voice` (learned L3 triage) | 37.2 s (2026-08-27) | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` — **11/11 checks pass** | 2.24 s | stdlib | -| Full test suite — **529 passing**, 27 files (median of 3, spanning 49.8 – 51.7 s) | 51.5 s | stdlib | +| Full test suite — **763 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | The headline: **a complete model-free pass over a 5,084-word manuscript costs ~1.0 s above the interpreter floor**, with no optional dependency installed — and 0.6 s of that is loading the collocation bank (530,677 @@ -677,14 +677,14 @@ sci-paper/ │ ├── SCIPAPER_STANDARD.md the single normative contract (v3.8) │ ├── architecture/ DEAI_SUBSYSTEM.md · EVALUATION.md (hub) + evaluation/ │ └── design-notes/ frozen, dated reasoning records (not status) -├── skills//SKILL.md 12 skills ├── tests/ 27 files, 529 tests +├── skills//SKILL.md 12 skills ├── tests/ 32 files, 763 tests ├── tools/ 40 product tools ├── CHANGELOG.md · ACKNOWLEDGMENTS.md ├── style-corpus// user-supplied read-only corpus (gitignored) └── style-profile// generated and calibrated evidence (gitignored) ``` `python tools/validate_plugin.py` runs 11 contract checks and `python -m unittest discover -s tests -v` runs the -529-test suite; both must pass before a release. The validator covers release metadata, skill frontmatter, +763-test suite; both must pass before a release. The validator covers release metadata, skill frontmatter, standard references, documentation boundaries and index completeness, in-page anchors, recorded suite sizes against real discovery, product registries, syntax, runtime imports, CLI entry points, schema fields, and linter exit semantics — `tools/validate_plugin.py` itself is the authoritative list. A release also requires diff --git a/README.zh-CN.md b/README.zh-CN.md index f2ab693..9366a0e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -5,13 +5,13 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-529%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-763%20passing-success.svg)](tests/) **一个 Claude Code 插件：在同一套 typed 标准下完成科研论文的写作、审查、去 AI 化与精简。 每条结论都可溯源，每个测不出来的轴都如实标为测不出来。** 面向 ApJ / MNRAS / PRD / JCAP 级别的论文，以及 NSF / NIH 基金申请书。 -**12 个 skill · 40 个工具 · 529 个测试 · 一份规范 · 零作者身份判决。** +**12 个 skill · 40 个工具 · 763 个测试 · 一份规范 · 零作者身份判决。** [English](README.md) — [它做什么](#它做什么) · [怎么做到的](#怎么做到的) · [实际效果](#实际效果) · [Benchmark 面板](#benchmark-面板) · [安装](#安装) · @@ -448,7 +448,7 @@ LaTeX include 组装而成。标准库的每一行都重测了：解释器地板 | `+ --oracle`（GPT-2-large token surprisal） | 33.8 s（2026-08-27） | `transformers` + `torch` | | `+ --voice`（学习型 L3 分诊） | 37.2 s（2026-08-27） | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` —— **11/11 通过** | 2.24 s | 标准库 | -| 完整测试套件 —— **529 通过**，27 个文件（3 次取中位数，落在 49.8 – 51.7 s） | 51.5 s | 标准库 | +| 完整测试套件 —— **763 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | 一句话：**一份 5,084 词的稿子跑完全部 model-free 通道，在解释器地板之上约花 1.0 s**， 且不需要任何可选依赖——其中 0.6 s 是加载 collocation 词对库（计时所用的 2026-09-04 库有 530,677 对，v0.36.3 重建后为 530,504 对）， @@ -662,7 +662,7 @@ sci-paper/ │ └── design-notes/ 冻结的、带日期的设计记录（不是现状文档） ├── skills//SKILL.md 12 个 skill ├── tools/ 40 个产品工具 + 仓库 validator -├── tests/ 27 个测试文件、529 个测试 +├── tests/ 32 个测试文件、763 个测试 ├── style-corpus// 用户提供的只读语料（gitignore） ├── style-profile// 生成与标定的证据（gitignore） ├── ACKNOWLEDGMENTS.md 改编来源的致谢与采纳边界 @@ -673,7 +673,7 @@ sci-paper/ ## 开发与发布 `python tools/validate_plugin.py` 跑 11 项契约检查， -`python -m unittest discover -s tests -v` 跑 529 个测试；发布前两者都必须通过。 +`python -m unittest discover -s tests -v` 跑 763 个测试；发布前两者都必须通过。 Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权威边界与索引完整性、 记录的测试规模与真实发现的一致性、过期契约标记、产品注册表、Python 语法、 运行时 import、CLI 入口、schema 字段、linter 退出语义、Tier B 行为、测试与 CI 接线 —— diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index 569292e..c9d4ad6 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -140,8 +140,8 @@ python tools/validate_plugin.py python -m unittest discover -s tests -v ``` -The working tree passes the validator and all 529 unit/CLI tests (27 test files, collected -2026-09-05). These commands must be rerun after every subsequent code or release-metadata +The working tree passes the validator and all 763 unit/CLI tests (32 test files, collected +2026-09-27). These commands must be rerun after every subsequent code or release-metadata change; the release record must quote the fresh output rather than a past result. Figures quoted from a generated profile are pinned the same way, by @@ -156,7 +156,7 @@ The three drift events it exists to stop are recorded in §18.8. Current release gates (as of 2026-09-27; last tagged release v0.38.0, this release v0.39.0): `validate_plugin.py` all 11 checks pass and the full unit/CLI suite -(529 tests, 27 files) passes on a clean tree; both are rerun before every tag, +(763 tests, 32 files) passes on a clean tree; both are rerun before every tag, and as of v0.25.1 the hosted CI run on the release commit must also be green (first green runs: 31133202443 push, 31133215203 manual dispatch). diff --git a/docs/audits/full-review-2026-09-27.md b/docs/audits/full-review-2026-09-27.md index 974fb75..74a2f34 100644 --- a/docs/audits/full-review-2026-09-27.md +++ b/docs/audits/full-review-2026-09-27.md @@ -375,3 +375,25 @@ - `deai_feedback` 校验、优先级排序（与 §4 一致）、0.5 段落置信上限；`_mat_inv`/Mahalanobis/conformal `_conformal_p`/分层回退/角色 z 分数正负号一致；`fit_dispersion_manifold` 33 行下限；voice 流水线的训练集内 centroid、词表、分组划分、原子写出、schema 漂移拒绝；`rank_auc` 与 `_auc` 与暴力 tie-aware AUC 在 3000 例上一致；kappa 与独立实现一致；20 标签下限；blind relabel 剥离字段正确。 - register 的多行置空、宏项、复合词最稀部分、三类 zero-hit 豁免、`wgl-letter` 借用；collocation 的断点集、留一法、可能所有格首词；discourse 状态与检测同一 `live_buckets`、hedging 限 `intro`；salience 的 `numeral_text` 与 `latex_to_numeral_text` 占位符一致、每段一条 finding、低于下限的 bucket 给 degraded finding 而非沉默；residue 规则 6 家族、`never(?!-)`、`[CITE]` 豁免、强弱顺序；validator 镜像检查；CUDA 回退可达且不泄漏（除 C7 的空消息情形）。 - 文档：所有 43 个 markdown 的相对链接、`tools/*.py`、`skills/*` 引用均可解析（除 M3）；docs/README 的 section map 与磁盘一致；skill 里引用的 EVALUATION §号全部经 hub 解析；没有 skill 引入 PASS/FAIL 或作者身份断言，没有 skill 豁免 integrity blocker；README 引用的所有 flag 存在；文档常量（15 / 1e-4 / 500 / 2e-4 / 4 / 2 / 30 / 0.90 / 0.95 / 0.10 / 0.05 / 150 / 3 / 400 / 0.60 / 0.20 / 0.5 / 3 篇 / 0.40 / 20 / 0.05 / 3 层 / 0.6）与代码一致；v0.36.3 重测数字在已更新的位置内部一致，过时的恰是 §6.2 列出的位置。 + +## 10. 落地记录（v0.39.0，2026-09-27） + +§8 第一梯队与其耦合的第二梯队已在 v0.39.0 落地，提交按范围列出；每个提交正文写明它覆盖的条目号。 + +| 范围 | 条目 | 提交 | +|---|---|---| +| 字段解析：`cli_common.optional_field_dir`、`axis_main` 的 `--calibrate` 检查与 `extra_arguments`/`check`/`report_for` 钩子 | M1/B1/C1/C2/D3/E9/H2，C8/C21 的 cli_common 侧 | `49624e5` | +| 装配、投影与宏；length_gate 与 condense_map；ai_ism_lint 与 validate_plugin；extract_style 与 retrieve_exemplars | A1–A20、A28，F3/F4/F7/F9/F12/F14/F16–F20/F25/F27/F28/F33，D6–D11/D14/D16/D19，M2/M3 | `5ce670c` | +| verify_references | F1/F2/F5/F6/F8/F10/F11/F13/F21/F23/F24/F32 | `46cf4ce` | +| 文档级轴：docshape、docstructure、metrics、structure、features、partition、anchoring | B1–B15、B19–B21、B23、B28，E20 的 features 侧 | `1c9736d`、`f211ece`、`2d2685d` | +| rewrite_reward | E1–E4、E12、E30，H1 | `116f074` | +| eval_findings、eval_docscale、label_findings、voice_dataset、voice_audit、train_voice_model、train_ai_ism_classifier | E5–E8、E10/E11、E13–E16、E18/E19、E21/E22、E28、E31、E33，M5 | `9b30c47`、`e47d250` | +| fetch_arxiv_abstracts | D1–D5、D12/D13、D24、D26 | `41a29a8` | +| 逐 bucket 轴：reference、collocation、discourse、salience、register、residue、oracle、provenance、personal、voice | C1–C19、C21、C22（C20 属 deai_structure，未动），E20 的 voice 侧 | `95c988f`、`3e13594` | +| 文档与元数据：§6.2 全部数字（I1–I20、G14–G25）、H3–H6/H12/H13/H36 的 skill 补丁、I21/I25/I36、I22/I31–I34、I38、G11、G27、D15（改为"记录权重，未应用"）、H31（标准 v3.9）及各部分的 registry 行、发布说明与记录的测试数 | | `1ad6ed7`、`de925a9`、`470290d`、`b25f4cd` 及本节所在提交 | + +未落地（§8.3 及本轮范围之外）： + +- §5 的去重（R1–R10、H23–H30/H45–H51）；待核实项 B16/B17/B18/E17/E32/F22/F26/D20；G19（`latency.json`）；M3/I35 的 disposition 列。A31/D26/E33/B28 的 `__main__` 位置已随各自文件修复。 +- 需重建的制品：`structure_baseline` 与 `docstructure_baseline`（B12/B15）、`anchoring_baseline`（B13）；逐 bucket 轴的制品需重新校准才采用最近秩分位网格（C12）。重建前，EVALUATION 中据旧制品测得的比率沿用旧值，各处已注明测得日期。 +- 抓取器的内置查询集、关键词过滤与期刊表标为 `[WGL]`，其他领域改用 `--query`（D24）；`TIER_WEIGHTS` 仍只记录不应用（D15 按第二梯队的"改措辞"落地）。 From 7daf3e99d1ccfc97fa2d080f0a5c9e4960d08062 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:53:08 +0000 Subject: [PATCH 19/27] deai_voice: the --scores output test from the amended per-bucket commit The voice commit cherry-picked as 3e13594 was amended once more in its worktree to add the test that reads the --scores lines by source line; this is that addition, so the branch carries the agent's final version. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- tests/test_deai_voice.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tests/test_deai_voice.py b/tests/test_deai_voice.py index b339412..9583dee 100644 --- a/tests/test_deai_voice.py +++ b/tests/test_deai_voice.py @@ -77,6 +77,23 @@ def test_scores_without_the_runtime_prints_the_reason_and_exits_zero(self): self.assertEqual(code, 0) self.assertIn(f"axis L3.voice: unmeasured: {MISSING}", stdout.getvalue()) + def test_scores_with_the_runtime_lists_each_paragraph_by_source_line(self): + # A blank-line split of the raw text numbered paragraphs the findings + # never used; the listing now carries the sweep's source lines. + draft = Path(self.raw.name) / "draft.tex" + draft.write_text(TEXT, encoding="utf-8") + stdout = io.StringIO() + with mock.patch.object(oracle, "model_runtime_available", return_value=(True, "")), \ + mock.patch.object(df, "features_vector", + return_value=[0.0] * len(df.FEATURE_NAMES)), \ + mock.patch.object(voice, "_positive_class_probability", return_value=0.25), \ + contextlib.redirect_stdout(stdout): + code = voice.main([str(draft), "--field", "fld", "--profile-root", + str(self.root), "--scores"]) + self.assertEqual(code, 0) + self.assertEqual(stdout.getvalue().splitlines(), + [" L2-2 field_similarity=0.250", " L5-5 field_similarity=0.250"]) + def test_with_the_runtime_the_bundle_scores_and_flags(self): # The probe must not block a present runtime: the vector and the # classifier are stubbed, the rest of the path is real. From 04ae5b0d952a47eec383c3ebe769e507e2a555f5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 06:54:30 +0000 Subject: [PATCH 20/27] linter and registries: the per-bucket statuses reach the unified report, and the records say what moved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ai_ism_lint passes the text to the salience and discourse status functions and reads the glossary status from the bank alone, so a document whose units sit in `unknown`, a bucket recorded at another unit, and a glossary-only run report what the axes now know instead of a profile-only status. The tools registry and the responsibilities table describe the per-bucket axes as rebuilt: the grid at nearest rank, the unit read back and refused by name, a section pooled per bucket, the status that counts what it skipped, `--glossary` as an argparse option with its cue after the pair, residue reading the whole paper minus preamble and bibliography, the oracle's exit on a field with no bank, and the voice axis unmeasured without the runtime. The evaluation hub marks the salience, collocation, cohesion and hedging rates as measured on the earlier grid, and the collocation page records the 30–400-word bank band and the new artifact keys. The residue section of the subsystem note counts four static rules and one diff rule. The audit's landing record and the recorded suite size (764 tests in 32 files) are brought up to date. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01DuasTxTXcSQsC6U2qYvUMM --- CHANGELOG.md | 2 +- README.md | 10 +++++----- README.zh-CN.md | 10 +++++----- docs/architecture/DEAI_SUBSYSTEM.md | 2 +- docs/architecture/EVALUATION.md | 12 ++++++------ docs/architecture/RESPONSIBILITIES.md | 12 ++++++------ .../evaluation/vocabulary-and-residue.md | 10 ++++++++++ docs/audits/full-review-2026-09-27.md | 2 +- tools/README.md | 18 +++++++++--------- tools/ai_ism_lint.py | 6 +++--- 10 files changed, 47 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a093df..3a21c33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -164,7 +164,7 @@ The voice axis is `unmeasured` without the surprisal runtime instead of a RuntimeError, `--scores` lists paragraphs by source line, and every one of these tools resolves `--field` through the shared resolver. -The suite is 763 tests in 32 files (v0.38.0: 529 in 27); the validator runs +The suite is 764 tests in 32 files (v0.38.0: 529 in 27); the validator runs 11 checks and passes. ## v0.38.0 — 2026-09-16 diff --git a/README.md b/README.md index 97db3cb..911c587 100644 --- a/README.md +++ b/README.md @@ -5,14 +5,14 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-763%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-764%20passing-success.svg)](tests/) **A Claude Code plugin that writes, reviews, de-AIs, and condenses scientific manuscripts for top-tier journals — under one typed standard, with every claim traced to a source and every unavailable measurement labelled as unavailable.** Built for ApJ / MNRAS / PRD / JCAP-class papers and NSF / NIH proposals. -**12 skills · 40 tools · 763 tests · one normative contract · zero authorship verdicts.** +**12 skills · 40 tools · 764 tests · one normative contract · zero authorship verdicts.** [中文文档](README.zh-CN.md) — [What it does](#what-it-does) · [How it works](#how-it-works) · [See it work](#see-it-work) · [Benchmarks](#benchmark-dashboard) · [Install](#install) · @@ -474,7 +474,7 @@ at 100%, and a number taken under that describes the contention, not the axis. V | `+ --oracle` (GPT-2-large token surprisal) | 33.8 s (2026-08-27) | `transformers` + `torch` | | `+ --voice` (learned L3 triage) | 37.2 s (2026-08-27) | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` — **11/11 checks pass** | 2.24 s | stdlib | -| Full test suite — **763 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | +| Full test suite — **764 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | The headline: **a complete model-free pass over a 5,084-word manuscript costs ~1.0 s above the interpreter floor**, with no optional dependency installed — and 0.6 s of that is loading the collocation bank (530,677 @@ -677,14 +677,14 @@ sci-paper/ │ ├── SCIPAPER_STANDARD.md the single normative contract (v3.8) │ ├── architecture/ DEAI_SUBSYSTEM.md · EVALUATION.md (hub) + evaluation/ │ └── design-notes/ frozen, dated reasoning records (not status) -├── skills//SKILL.md 12 skills ├── tests/ 32 files, 763 tests +├── skills//SKILL.md 12 skills ├── tests/ 32 files, 764 tests ├── tools/ 40 product tools ├── CHANGELOG.md · ACKNOWLEDGMENTS.md ├── style-corpus// user-supplied read-only corpus (gitignored) └── style-profile// generated and calibrated evidence (gitignored) ``` `python tools/validate_plugin.py` runs 11 contract checks and `python -m unittest discover -s tests -v` runs the -763-test suite; both must pass before a release. The validator covers release metadata, skill frontmatter, +764-test suite; both must pass before a release. The validator covers release metadata, skill frontmatter, standard references, documentation boundaries and index completeness, in-page anchors, recorded suite sizes against real discovery, product registries, syntax, runtime imports, CLI entry points, schema fields, and linter exit semantics — `tools/validate_plugin.py` itself is the authoritative list. A release also requires diff --git a/README.zh-CN.md b/README.zh-CN.md index 9366a0e..1a4ca99 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -5,13 +5,13 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-763%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-764%20passing-success.svg)](tests/) **一个 Claude Code 插件：在同一套 typed 标准下完成科研论文的写作、审查、去 AI 化与精简。 每条结论都可溯源，每个测不出来的轴都如实标为测不出来。** 面向 ApJ / MNRAS / PRD / JCAP 级别的论文，以及 NSF / NIH 基金申请书。 -**12 个 skill · 40 个工具 · 763 个测试 · 一份规范 · 零作者身份判决。** +**12 个 skill · 40 个工具 · 764 个测试 · 一份规范 · 零作者身份判决。** [English](README.md) — [它做什么](#它做什么) · [怎么做到的](#怎么做到的) · [实际效果](#实际效果) · [Benchmark 面板](#benchmark-面板) · [安装](#安装) · @@ -448,7 +448,7 @@ LaTeX include 组装而成。标准库的每一行都重测了：解释器地板 | `+ --oracle`（GPT-2-large token surprisal） | 33.8 s（2026-08-27） | `transformers` + `torch` | | `+ --voice`（学习型 L3 分诊） | 37.2 s（2026-08-27） | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` —— **11/11 通过** | 2.24 s | 标准库 | -| 完整测试套件 —— **763 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | +| 完整测试套件 —— **764 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | 一句话：**一份 5,084 词的稿子跑完全部 model-free 通道，在解释器地板之上约花 1.0 s**， 且不需要任何可选依赖——其中 0.6 s 是加载 collocation 词对库（计时所用的 2026-09-04 库有 530,677 对，v0.36.3 重建后为 530,504 对）， @@ -662,7 +662,7 @@ sci-paper/ │ └── design-notes/ 冻结的、带日期的设计记录（不是现状文档） ├── skills//SKILL.md 12 个 skill ├── tools/ 40 个产品工具 + 仓库 validator -├── tests/ 32 个测试文件、763 个测试 +├── tests/ 32 个测试文件、764 个测试 ├── style-corpus// 用户提供的只读语料（gitignore） ├── style-profile// 生成与标定的证据（gitignore） ├── ACKNOWLEDGMENTS.md 改编来源的致谢与采纳边界 @@ -673,7 +673,7 @@ sci-paper/ ## 开发与发布 `python tools/validate_plugin.py` 跑 11 项契约检查， -`python -m unittest discover -s tests -v` 跑 763 个测试；发布前两者都必须通过。 +`python -m unittest discover -s tests -v` 跑 764 个测试；发布前两者都必须通过。 Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权威边界与索引完整性、 记录的测试规模与真实发现的一致性、过期契约标记、产品注册表、Python 语法、 运行时 import、CLI 入口、schema 字段、linter 退出语义、Tier B 行为、测试与 CI 接线 —— diff --git a/docs/architecture/DEAI_SUBSYSTEM.md b/docs/architecture/DEAI_SUBSYSTEM.md index 1a9f989..30e19c1 100644 --- a/docs/architecture/DEAI_SUBSYSTEM.md +++ b/docs/architecture/DEAI_SUBSYSTEM.md @@ -315,7 +315,7 @@ high-confidence AI verdict. ### L4: residue, and the removal map [`../tools/deai_residue.py`](../../tools/deai_residue.py) reads the trace an edit -leaves rather than the prose it produced. Five rules, all deterministic: +leaves rather than the prose it produced. Four static rules and one diff rule, all deterministic: `residue-self-history:` (a drafting-history term — `initially`, `no longer`, `we switched` — in a first-person sentence with no citation, so a history *of the literature* is not one of the paper); `residue-absence` (a diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index c9d4ad6..1a7b205 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -114,16 +114,16 @@ result. | L0 lexical/punctuation | measured | Deterministic Tier A, em-dash, and Tier B cap implementation with CLI regression tests. | Continue regression coverage when policy changes. | | L1 distribution | degraded (refuted) | §16: measured on 500 human papers against 173 `docval` AI documents, one observation per document. Burstiness reverses sign — adversarial prose is *more* bursty than the human median (1.036 vs 0.775, AUC 0.181) and long-form sits inside the human band (AUC 0.441) — while flagging 7.2% of humans. Signposting has an AUC of 0.247, below chance, and flags 0 of 173 AI documents at the shipped default. | None. `deai_policy.json` is withdrawn as a roadmap item: the statistics do not support an operating point, so the axis stays advisory by measurement rather than by absence. | | L1 UID | degraded | `style-profile/wgl/uid_baseline.json` records paragraph-level GPT-2-large summaries. | A documented operating point and human false-flag behavior; audit sensitivity to mathematics and jargon. | -| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2705 per passage on 203 held-out ApJ/ApJL/A&A papers (§17.5's headline; 0.2776 when re-run in the v0.36.2 process) against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nfields}{63}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | +| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2705 per passage on 203 held-out ApJ/ApJL/A&A papers (§17.5's headline; 0.2776 when re-run in the v0.36.2 process) against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nfields}{63}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | | L0 register | measured | §14: document frequency over 41,644 corpus passages, 53,367 terms (v0.36.3 profile rebuild; 41,710 and 53,414 before it, and the 57 held-out findings below are the same 57 under both); compound-by-rarest-part and macro-subscript handling; native-term controls pass. Since v0.32.0 a `-` profile whose own bank cannot resolve the 1e-4 gate judges against `` and names the borrowed bank in every finding: on 36 letter-format documents the 706-passage `wgl-letter` bank produced 262 findings the field bank did not (`sne`, `bao`, `pantheon`, and `letter` itself) against 2 the other way (§18.5). | Recall below the 15-use floor, and a **measured** false-positive rate: §18.4 put it at 0.0858 findings per 1,000 words on 203 held-out refereed papers (44.83% of documents), rank AUC 0.2856 against machine text — it still fires more on human prose than on AI drafts — and §23.1 re-measures the same papers at 0.0351 (30.05%, AUC 0.352) after the heading projection fix and at 0.0247 (22.2%, AUC 0.392) after v0.36.1's float projection fix, same direction; v0.36.2 counts per body prose word instead of per raw-source word, and the same 57 findings read 0.0371 (AUC 0.391). §17.3, re-measured: 94.44% of the 198 remaining flags would be suppressed by the paper's own bank membership (98.2% of the 57 under v0.36.1). The operating point is no longer an open item: swept 5→50 uses, rank AUC stays below 0.5 everywhere, so no setting makes this a detector and the knob buys advisory volume only. It was cut at 15, the first point where a referee-grade paper is not flagged more often than not (§18.4). §21 replicates that refutation on a second population — 22 papers by one author, 1996–2015, sampled by author rather than by journal and spanning a different two decades — where rank AUC is **0.328**, again below 0.5 (0.412 on the v0.36.3 profile rebuild, 5 findings on the 22 papers). | | L2 sentence structure | measured for deterministic matches; degraded for strength | `style-profile/wgl/structure_baseline.json` provides section-level reference fractions over 27,841 paragraphs — method 9,478; results 3,978; data 3,894; intro 3,812; discussion 3,635; conclusion 2,611; abstract 433. §23.3 adds the per-bucket human fractions of the three auxiliary families taken from a mentor's comments (paper-as-agent 0.13–1.00%, wh-cleft 0.00–0.23%, modifier stacks 2.5–15.9%). | Author-labelled difficult cases. A calibrated strong-advisory threshold is no longer expected from `deai_policy.json` (§16). | -| L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (530,504 pairs over 11,282 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,391; method 27,207; intro 14,321; results 12,509; data 12,048; discussion 12,183; conclusion 8,588 after the v0.36.3 profile rebuild; 41,710 passages and 541,309 → 530,677 pairs under v0.36.2); p90 gates 0.44–0.59, p95 0.50–0.67. On the first measured draft of the private Letter it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at AUC 0.704 (7,237 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same Letter: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). | Whether an individual advisory is good advice; a second generation process (§20's lesson). | +| L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (530,504 pairs over 11,282 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,391; method 27,207; intro 14,321; results 12,509; data 12,048; discussion 12,183; conclusion 8,588 after the v0.36.3 profile rebuild; 41,710 passages and 541,309 → 530,677 pairs under v0.36.2); p90 gates 0.44–0.59, p95 0.50–0.67. On the first measured draft of the private Letter it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at AUC 0.704 (7,237 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same Letter: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; a second generation process (§20's lesson). | | L4 residue | measured | §23.4: four static rules (self-history, edit-meta, negative-label, and since v0.37.0 the absence rule, §23.4a) and one diff rule, deterministic, strengths set on 203 held-out refereed papers (70,848 body prose sentences): strong self-history fires in 20 papers, strong edit-meta in 8 (68 of 69 are `\textcolor{red}` in tables), any strong in 27 (13.3%); the static negative-label rule is ordinary at 26% of papers. Four passes to get there, the first a projection asymmetry and the last the procedural sense of `we have added`. | Recall against a labelled edit history; only the author's own drafts can supply it. | | L0 register zero-hit audit | measured, not a detector | §23.1: every body word with corpus df 0, on both sides of one projection (headings now excluded on both). All 203 held-out refereed papers carry zero-hit words, 3.37 per 1,000 body prose words (2.21 per raw-source word under v0.36.1, 2.66 before its float fix), against 1.05 for the 173 machine documents — rank AUC 0.174, the audit fires more on human prose — so it ships as exhaustive advice with author dispositions, never as a gate. The two projection fixes moved the thresholded rule from 0.0858 to 0.0247 findings per 1,000 raw-source words on the same papers (0.0371 per body prose word). | None as a detector. Whether the mechanical exemptions (attested stem, defined term, proper name) are the right three. | | L2 document structure | measured | §9: cross-paragraph dispersion calibrated one-observation-per-paper over 493 complete human `wgl` papers — the 2026-08-17 scoring; the v0.36.3 rebuild (2026-09-05) re-fit the artifact on 507 documents and the operating-point figures were not re-scored on it — human false-flag at the shipped conformal operating point 0.0325 (manifold) and 0.0426 (role) against nominal α = 0.05. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. | Re-score §9 through `eval_docscale.py` on the 507-document artifact; continue recalibration when the corpus changes. | L3 learned field similarity | degraded (confound-audited) | Confound-aware audit complete (§7): repeated grouped-split AUC 0.9487, matched-stratum AUC 0.9262, hard-set true-provenance AUC 0.938 (2026-09-05 retrain), but a 30–42% false-positive rate on field-topic AI text. Document-level now measured (§9.8): surprisal dispersion (0.757) is weaker than the model-free manifold (0.881) and adds nothing to it, so L3 stays degraded for a measured reason. | None from this feature set: four retrains agree the confound is in the features (§7.0a), so reopening needs a different feature set; the surprisal path is measured not to provide one. | -| L2 cohesion | measured | §19: given/new linkage per paragraph against the field's own bank (abstract 13,967; method 6,886; intro 3,228; results 3,211; data 2,989; discussion 2,926; conclusion 1,980 after the v0.36.3 profile rebuild), flagged on the LOW tail at p10. The gate transfers to 203 held-out refereed papers at 10.78% (10.87% at v0.33.0) against a 10% design point, in every bucket (6.58%–14.61%), and separates all six machine regimes in `intro` at worst-of-six 0.676 against a 0.515 human-vs-human null. | Whether an individual advisory is good advice; recall. Whether the `intro` separation holds for a second generation process — §20 is what happens when that question is not asked. | -| L2 hedging | measured, restricted to `intro` | §19: epistemic markers per 1,000 words per SECTION (abstract 10,404; intro 502; method 438; conclusion 383; discussion 327; results 317; data 299 after the v0.36.3 profile rebuild). It has no paragraph-scale lower tail — p10 is 0.000 in all seven buckets — so calibration and detection both run at section unit, and each artifact records its own. Restricted to `intro` by two independent measurements that agree: held-out transfer is 7.80% there (7.89% at v0.33.0) against 15.48–26.77% elsewhere, and worst-of-six AUC is 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376. | Whether the restriction generalizes beyond `wgl`. On `wgl-letter` the axis is `degraded`: no bucket clears the 30-unit floor after the restriction. | +| L2 cohesion | measured | §19: given/new linkage per paragraph against the field's own bank (abstract 13,967; method 6,886; intro 3,228; results 3,211; data 2,989; discussion 2,926; conclusion 1,980 after the v0.36.3 profile rebuild), flagged on the LOW tail at p10. The gate transfers to 203 held-out refereed papers at 10.78% (10.87% at v0.33.0) against a 10% design point, in every bucket (6.58%–14.61%), and separates all six machine regimes in `intro` at worst-of-six 0.676 against a 0.515 human-vs-human null. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; recall. Whether the `intro` separation holds for a second generation process — §20 is what happens when that question is not asked. | +| L2 hedging | measured, restricted to `intro` | §19: epistemic markers per 1,000 words per SECTION (abstract 10,404; intro 502; method 438; conclusion 383; discussion 327; results 317; data 299 after the v0.36.3 profile rebuild). It has no paragraph-scale lower tail — p10 is 0.000 in all seven buckets — so calibration and detection both run at section unit, and each artifact records its own. Restricted to `intro` by two independent measurements that agree: held-out transfer is 7.80% there (7.89% at v0.33.0) against 15.48–26.77% elsewhere, and worst-of-six AUC is 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether the restriction generalizes beyond `wgl`. On `wgl-letter` the axis is `degraded`: no bucket clears the 30-unit floor after the restriction. | | Citation placement | **refuted, not shipped** | §20: the v0.32.0 candidate (section-matched AUC 0.866 in `method`) does not hold its sign across generation processes. A second bank from a different model scores 0.053 with no citation instruction and 0.734 with one — one prompt line, a 12.5× density swing, and the two machine extremes bracket the human distribution rather than sitting on one side of it. | None. Reopening requires a statistic that holds its sign across independently produced banks. | | Rewrite scientific fidelity | measured for protected invariants | Unit tests cover preserved invariants, dropped number, dropped citation, reversed comparison, display-math values and exponents (v0.27.0), and the punctuation/adjacent-word tokenizer boundaries (§8). | Real manuscript before/after demonstration, including scope and stance review. The plain-ASCII-space unit boundary in §8 is an accepted limit ([DISPOSITIONS](DISPOSITIONS.md)): the number stays protected, the unit does not. | @@ -140,7 +140,7 @@ python tools/validate_plugin.py python -m unittest discover -s tests -v ``` -The working tree passes the validator and all 763 unit/CLI tests (32 test files, collected +The working tree passes the validator and all 764 unit/CLI tests (32 test files, collected 2026-09-27). These commands must be rerun after every subsequent code or release-metadata change; the release record must quote the fresh output rather than a past result. @@ -156,7 +156,7 @@ The three drift events it exists to stop are recorded in §18.8. Current release gates (as of 2026-09-27; last tagged release v0.38.0, this release v0.39.0): `validate_plugin.py` all 11 checks pass and the full unit/CLI suite -(763 tests, 32 files) passes on a clean tree; both are rerun before every tag, +(764 tests, 32 files) passes on a clean tree; both are rerun before every tag, and as of v0.25.1 the hosted CI run on the release commit must also be green (first green runs: 31133202443 push, 31133215203 manual dispatch). diff --git a/docs/architecture/RESPONSIBILITIES.md b/docs/architecture/RESPONSIBILITIES.md index a7215b7..43f23b2 100644 --- a/docs/architecture/RESPONSIBILITIES.md +++ b/docs/architecture/RESPONSIBILITIES.md @@ -49,16 +49,16 @@ scope, not a standard-reference obligation. | `deai_metrics.py` | L1 | Emit calibrated distribution findings and explicit missing-baseline status. | | `deai_oracle.py` | L1 | Emit surprisal/UID findings with observed and reference values; advisory-success exit 0. | | `deai_structure.py` | L2 sentence | Emit template evidence with calibration metadata; advisory-success exit 0. The auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) name the sentence under `structure-auxiliary` and never enter `template_score`. | -| `deai_salience.py` | L2 salience | Measure recital structure against a per-section human passage reference at one shared unit; one finding per passage; abstain where the reference cannot resolve above the gate. Sole consumer of `extract_style.latex_to_numeral_text`, the numeral-preserving projection (`latex_to_plain` replaces math with `[math]`, which zeroes every numeral signal on `.tex` input). | +| `deai_salience.py` | L2 salience | Measure recital structure against a per-section human passage reference at one shared unit; one finding per passage; report `measured` only where a bucket built at that unit clears the floor and resolves above the gate, and count the document's units left in `unknown`. Sole consumer of `extract_style.latex_to_numeral_text`, the numeral-preserving projection (`latex_to_plain` replaces math with `[math]`, which zeroes every numeral signal on `.tex` input). | | `deai_register.py` | L0 register | Compare manuscript vocabulary against field-corpus document frequency; judge compounds by their rarest part; read macro bodies but not subscript decorations; emit advisories only, never `l0_target`s. The zero-hit audit (`register-zero:`) is exhaustive over body words on both sides of the same projection (headings, comments, preamble and bibliography excluded on both), strong unless the manuscript supplies a reason the tool can read: a mechanical formation of an attested stem, a defining sentence or acronym expansion at first use, or a proper name capitalised at every use (a cited method's own name is an author disposition, not an exemption). | -| `deai_collocation.py` | L2 collocation | Judge, per sentence, the fraction of adjacent common-word pairs (unigram df at or above `COMMON_RATE`) that no corpus passage attests, against a leave-one-out per-section reference at sentence unit; a pair breaks at punctuation, a placeholder or a dash; carry each pair's chance-absence probability so a reader can weigh it; advisories only, and the action names the relation, never the claim. Read document-wide (`--glossary`), list every unattested pair used twice or more with its first-use line and definition cue, as the manuscript's own terms. | -| `deai_residue.py` | L4 residue | Report self-referential drafting history (first-person sentence, no citation), a body sentence that defines the paper's own object by what it never does or has (`residue-absence`; `never` strong, a citation in the sentence exempts), edit-meta literals, and a heading or caption whose object the body never names; with `--before`/`--git-ref`, the added-label diff rule; exit 1 on a strong finding (§0.1); keep the family lists in one place and let the validator prove the skills quote them. | +| `deai_collocation.py` | L2 collocation | Judge, per sentence, the fraction of adjacent common-word pairs (unigram df at or above `COMMON_RATE`) that no corpus passage attests, against a leave-one-out per-section reference at sentence unit; a pair breaks at punctuation, a placeholder or a dash; carry each pair's chance-absence probability so a reader can weigh it; advisories only, and the action names the relation, never the claim. Read document-wide (`--glossary`), list every unattested pair used twice or more with its first-use line and whether a definition cue follows it there; every occurrence counts as a use. Report `degraded` with counts where units fall in the `unknown` bucket or a bucket's artifact records another unit. | +| `deai_residue.py` | L4 residue | Report self-referential drafting history (first-person sentence, no citation), a body sentence that defines the paper's own object by what it never does or has (`residue-absence`; `never` strong, a citation in the sentence exempts), edit-meta literals, and a heading or caption whose object the body never names; with `--before`/`--git-ref`, the added-label diff rule; exit 1 on a strong finding (§0.1); keep the family lists in one place and let the validator prove the skills quote them. Scan for editing marks in every visible line, the `skip` sections included. | | `condense_map.py` | QD (§5.3) | Enumerate every removable entry in six scans, each with `removable_words` and, for a restatement, its canonical home; total a default target that excludes the abstract/conclusion carve-out; never delete anything, never emit a blocker, exit 0/2. | -| `deai_discourse.py` | L2 cohesion + L2 hedging | Measure given/new linkage and epistemic-marker rate against the field's own distribution, each at its declared unit, and flag the LOW tail. Report one status per feature, never a joint one: they calibrate from the same corpus at different granularities, so a field can support one and not the other. Speak only for buckets whose operating point was shown to transfer. | -| `deai_reference.py` | shared reference | Own the `(feature, unit)` percentile contract for every per-bucket axis -- one quantile grid, one sample floor, one plateau-top reader, one calibration loop -- and hold no policy of its own. | +| `deai_discourse.py` | L2 cohesion + L2 hedging | Measure given/new linkage and epistemic-marker rate against the field's own distribution, each at its declared unit, and flag the LOW tail. Report one status per feature, never a joint one: they calibrate from the same corpus at different granularities, so a field can support one and not the other. Speak only for buckets whose operating point was shown to transfer. Refuse a bucket whose artifact records another unit, and say so; a section is one unit per bucket; report `degraded` with counts where units fall in `unknown`. | +| `deai_reference.py` | shared reference | Own the `(feature, unit)` percentile contract for every per-bucket axis -- one quantile grid, one sample floor, one plateau-top reader, one calibration loop -- and hold no policy of its own. Read the grid at nearest rank, refuse an artifact that records another unit than the axis measures, and count the units a document leaves in the `unknown` bucket into the status. | | `deai_docstructure.py` | L2 document | Measure document shape (per-stratum dispersion manifold, role coupling, split-conformal) with sample-sufficiency checks; one shared `manifold_operating_point` scoring entry. | | `deai_anchoring.py` | L2 | Emit the section-class claim-anchoring band as a writing-quality axis, never an AI-discrimination axis. | -| `deai_voice.py` | L3 | Emit calibrated similarity evidence, model metadata, and confound status without authorship claims; degraded, offline audit instrument. | +| `deai_voice.py` | L3 | Emit calibrated similarity evidence, model metadata, and confound status without authorship claims; degraded, offline audit instrument. `unmeasured`, with the reason, when the surprisal runtime the feature vector needs is absent. | | `deai_feedback.py` | shared | Validate schema (incl. `calibration_unit` cap), attach actions, rank findings, summarize statuses, and serialize output. | | `rewrite_reward.py` | L3-L4 | Exclude unfaithful candidates before ranking eligible rewrites; rank by L0 advisory reduction and fidelity. | | `retrieve_exemplars.py` | L4 | Supply author-voice evidence without copying unsupported scientific content. | diff --git a/docs/architecture/evaluation/vocabulary-and-residue.md b/docs/architecture/evaluation/vocabulary-and-residue.md index 77647d9..69804ea 100644 --- a/docs/architecture/evaluation/vocabulary-and-residue.md +++ b/docs/architecture/evaluation/vocabulary-and-residue.md @@ -156,6 +156,16 @@ revision, because the synthetic paper repeats its own coined parameter and the cohesion fix carried a noun into a new neighbourhood — the axis's stated exception at work, recorded in [`examples/README.md`](../../../examples/README.md). +**Two limits recorded on 2026-09-27.** The bank keeps rows of 30–400 words +and the manuscript side has no such band, so a manuscript paragraph outside it +is measured against a reference that holds none of its length (audit C17, open). +Each artifact bucket now records `text_key` (the projection its rows were read +on) and `text_fallback_rows` (how many fell back to `text`), and the quantile +grid is stored at nearest rank; the gates quoted above were measured on the +earlier grid, one order statistic high, and move only when the artifact is +recalibrated and the rate re-measured. + + ## 23.3 Three structure families from the mentor's margin The remaining comments that named a *sentence shape* rather than a phrase fell diff --git a/docs/audits/full-review-2026-09-27.md b/docs/audits/full-review-2026-09-27.md index 74a2f34..ea72872 100644 --- a/docs/audits/full-review-2026-09-27.md +++ b/docs/audits/full-review-2026-09-27.md @@ -389,7 +389,7 @@ | rewrite_reward | E1–E4、E12、E30，H1 | `116f074` | | eval_findings、eval_docscale、label_findings、voice_dataset、voice_audit、train_voice_model、train_ai_ism_classifier | E5–E8、E10/E11、E13–E16、E18/E19、E21/E22、E28、E31、E33，M5 | `9b30c47`、`e47d250` | | fetch_arxiv_abstracts | D1–D5、D12/D13、D24、D26 | `41a29a8` | -| 逐 bucket 轴：reference、collocation、discourse、salience、register、residue、oracle、provenance、personal、voice | C1–C19、C21、C22（C20 属 deai_structure，未动），E20 的 voice 侧 | `95c988f`、`3e13594` | +| 逐 bucket 轴：reference、collocation、discourse、salience、register、residue、oracle、provenance、personal、voice | C1–C19、C21、C22（C20 属 deai_structure，未动），E20 的 voice 侧；ai_ism_lint 把带文本的状态接入统一报告 | `95c988f`、`3e13594`、`7daf3e9` 及本节所在提交 | | 文档与元数据：§6.2 全部数字（I1–I20、G14–G25）、H3–H6/H12/H13/H36 的 skill 补丁、I21/I25/I36、I22/I31–I34、I38、G11、G27、D15（改为"记录权重，未应用"）、H31（标准 v3.9）及各部分的 registry 行、发布说明与记录的测试数 | | `1ad6ed7`、`de925a9`、`470290d`、`b25f4cd` 及本节所在提交 | 未落地（§8.3 及本轮范围之外）： diff --git a/tools/README.md b/tools/README.md index 62deb9a..b308df6 100644 --- a/tools/README.md +++ b/tools/README.md @@ -22,23 +22,23 @@ serialize evidence but do not define an independent paper verdict. | `length_gate.py` | Per-section rendered-prose length-budget delta gate between two document versions (standard §5.3). Growing sections emit strong advisories; `--allow` records justifications; `--require-shrink ` demands a net cut of at least that much (0 = no cut required, so a map with nothing to cut still closes). | Exit 0 = net unjustified growth within tolerance and any shrink target met; 1 = growth beyond it or a cut short of the target (strong `length-shrink-short`); 2 = invalid input/execution, including a document or baseline that cannot be read (a mistyped `--before` once read as a zero-word baseline and every section as growth). Both versions are assembled documents, the `--git-ref` baseline with its `\input` children resolved at the ref. Comments (blanked before segmenting, so a commented-out heading opens no section) and mathematics in every display form are excluded; a pure section rename nets to zero. | | `verify_references.py` | Bibliography verification against CrossRef (DataCite fallback for the DOI prefixes CrossRef does not register) and the arXiv API: DOI resolvability (from the `doi` field or a doi.org `url`) and first-author / year / title / journal / volume / first-page agreement per entry, `@string` names expanded, plus `--tex` cross-checking of the assembled document's citation keys (LaTeX, natbib and biblatex commands including `\cites{a}{b}`; `\nocite{*}` cites every entry; a citation inside a `%` comment is text) against the entries. `--cache` keeps fetched records between rounds; a registry miss is never cached, and a finding built from a cached record says so (`observed.cached`). | Exit 0 = no blocker; 1 = unresolvable identifier or cited key without an entry; 2 = invalid input or execution failure (malformed BibTeX names the entry and line). Title match tolerates the CrossRef title/subtitle split; a one-year print/online seam is ordinary, a larger gap strong; a first author agrees when the family names are equal or share a part of four or more letters. A registry outage or an answer of unexpected shape marks the entry unmeasured and the axis `degraded`, never clean; an entry with no identifier is unmeasured. Relevance (whether a citation supports its sentence) is not measured here; that half of dimension F stays with the reviewer. | | `condense_map.py` | The removal map behind `/sci-paper:condense`: six scans — restatement (with the canonical home), zero-gain sentences, dead figure/table/label/macro/acronym, verbose constructions, repeated symbol glosses, duplicated paragraphs — each entry with `removable_words`, totalled into `condense_budget` with a default target that excludes the abstract/conclusion carve-out. | Deletes nothing and emits no blocker; exit 0 on a measured map, 2 on invalid input. Every word count is the gate's rendered-prose count (placeholders and heading words excluded), and the abstract/conclusion carve-out applies to either side of a restatement pair. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%, EVALUATION §23.5), so a pass that cuts less than that has not found what the map lists. | -| `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan shipped documentation for edit-meta literals. The literal rule reads the visible lines (headings and captions included; a `\newcommand{\TODO}` in a preamble is not prose) and the label rule the body projection; strengths were set on held-out refereed papers: any strong finding in 13.3% of them, the static negative-label rule ordinary, the strong absence forms at 0.008 per 1,000 words (EVALUATION §23.4). | +| `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan shipped documentation for edit-meta literals. The literal rule reads the visible lines of the whole paper minus preamble and bibliography (headings, captions and appendices included; an appendix `TODO` was invisible after v0.36.2 and the gate exited 0; a `\newcommand{\TODO}` in a preamble is not prose) and the label rule looks for the object in that same span, blanked to prose; strengths were set on held-out refereed papers: any strong finding in 13.3% of them, the static negative-label rule ordinary, the strong absence forms at 0.008 per 1,000 words (EVALUATION §23.4). `\section {X}` is a label; `-ation/-ated/-ating` forms share one root; an unwritable `--output` is exit 2 rather than the gate's exit 1 (2026-09-27). | | `deai_metrics.py` | L1 information-distribution analysis: sentence-length variation and connective-openers. | Strong status requires applicable policy calibration; compatibility heuristics are degraded. The reference rate is computed over `CONNECTIVE_OPENERS` from the profile's `paragraph_initial_counts`, i.e. the same set the draft is measured against; a profile predating that key omits the reference clause rather than quoting an incomparable one. A `sentence_stats.json` with no classified bucket (only `unknown`) leaves the axis `degraded`: its pooled CV is 0 and the burstiness rule cannot fire. | | `deai_structure.py` | L2 sentence-template analysis: enumeration, ordinal/modal/parallel runs, setup-list-wrap-up patterns, and balanced closers; plus the auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) reported under `structure-auxiliary`. | Strong status requires calibrated policy and sufficient reference sample. Auxiliary families never enter `template_score`; their per-bucket human fractions are written by `--calibrate` (`paper_agent_frac`, `wh_cleft_frac`, `modifier_stack_frac`). A template finding is `measured` only against a reference for its own bucket plus a policy, `degraded` when the baseline lacks the bucket, `unmeasured` without one; the auxiliary message quotes the reference fraction and calls a figure rare only at or below the policy gate. The wrap-up beat requires a sentence opening `These N ` closed by a summing verb (define/form/constitute/together/…). | -| `deai_salience.py` | L2 salience hierarchy: how far a passage's numerals run without an interpreting sentence between them, plus density and per-sentence numeral count, against a per-section human passage reference. | Sole consumer of `extract_style.latex_to_numeral_text` (the numeral-preserving projection; `latex_to_plain` zeroes every numeral on `.tex` input), and it calibrates on the bank's `numeral_text` field, the same projection of each paragraph, so the reference holds the numerals the manuscript side counts. Until v0.36.3 it calibrated on `text`, which holds none inside math, and the p90 gate fired at 0.45 per held-out passage against a 0.27 bound; it now fires at 0.20 (EVALUATION §17.5). Percentiles are read as P(X <= x) on a 0.01 grid at the top of a tie plateau. Buckets under 30 reference passages are degraded; a reference with no spread above the gate abstains rather than flagging. As of the v0.36.3 profile rebuild every bucket clears that floor (smallest: `conclusion` 1,999). One finding per passage, led by its most extreme feature. | -| `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- `AUC` (df 1) must separate from `epoch` (402) and `accuracy` (774). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the manuscript supplies a reason the tool can read (a formation of an attested stem, a defining sentence or acronym expansion at first use, a proper name capitalised at every use); it is exhaustive advice, not a detector (rank AUC 0.174, EVALUATION §23.1). | -| `deai_collocation.py` | L2 collocation: per sentence, the fraction of distinct adjacent common-word pairs that no corpus passage attests, against a leave-one-out per-bucket reference at sentence unit (p90 advisory, p95 strong). Each flagged pair carries λ = df(a)·df(b)/N and e^−λ so the reader can weigh it. `--glossary` reads the same bank document-wide: every unattested pair used at least twice, with its first-use line and whether that sentence defines it, as the list of the manuscript's own terms to define at first use or replace (2026-09-16). | `--calibrate` writes `collocation_bank.json` (unigram and pair document frequency; 11,282 partner words and 530,504 pairs over 41,644 `wgl` passages) and `collocation_baseline.json`. Unmeasured without the bank; degraded where no bucket clears the 30-sentence floor. Pairs break at punctuation, `[math]`/`[CITE]` and dashes; a sentence needs four judged pairs to carry a fraction. The action names the relation a pair compresses and never changes a claim; a defined term keeps its pair. | -| `deai_discourse.py` | L2 discourse texture, two axes on the LOW tail (p10 advisory, p05 strong): `cohesion`, the mean fraction of a sentence's content words that already appeared in the sentence before it, and `hedging`, epistemic markers per 1,000 words. | The defect here is absence, not excess, so both flag below the gate. The two run at **different units and each artifact records its own**: cohesion per paragraph, hedging per SECTION, because hedging has no paragraph-scale lower tail at all (p10 is 0.000 in all seven `wgl` buckets). Hedging ships restricted to `intro`, where held-out transfer is 7.89% against 15.48-26.77% elsewhere and every generation regime separates (EVALUATION section 19). Advisories against the field's distribution, never a provenance claim. | -| `deai_reference.py` | Library, no CLI. One `(feature, unit)` percentile reference shared by every per-bucket axis: the 0.01 quantile grid, the plateau-top percentile reader, the 30-unit sample floor, the paragraph and section sweeps, the artifact loader, and the calibration loop. | Holds no policy -- gate quantiles, feature names and what a finding means belong to the axis. Its one invariant is that calibration and detection share one unit and one grid; that check is what caught the hedging axis calibrating where no lower tail exists. The artifact loader had five copies across the suite before this module (EVALUATION section 19.1). | +| `deai_salience.py` | L2 salience hierarchy: how far a passage's numerals run without an interpreting sentence between them, plus density and per-sentence numeral count, against a per-section human passage reference. | Sole consumer of `extract_style.latex_to_numeral_text` (the numeral-preserving projection; `latex_to_plain` zeroes every numeral on `.tex` input), and it calibrates on the bank's `numeral_text` field, the same projection of each paragraph, so the reference holds the numerals the manuscript side counts. Until v0.36.3 it calibrated on `text`, which holds none inside math, and the p90 gate fired at 0.45 per held-out passage against a 0.27 bound; it now fires at 0.20 (EVALUATION §17.5). Percentiles are read as P(X <= x) on a 0.01 grid at the top of a tie plateau. Buckets under 30 reference passages are degraded; the axis is `measured` only where a bucket both clears that floor and has spread above the gate (a flat reference read `measured` with zero findings until 2026-09-27), and the status counts the document's `unknown` units. As of the v0.36.3 profile rebuild every bucket clears that floor (smallest: `conclusion` 1,999). One finding per passage, led by its most extreme feature. `--calibrate` records `text_fallback_rows` per bucket and warns on stderr when a bank row lacked `numeral_text`, instead of calibrating silently on the `[math]` projection (2026-09-27). | +| `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- `AUC` (df 1) must separate from `epoch` (402) and `accuracy` (774). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the manuscript supplies a reason the tool can read (a formation of an attested stem, a defining sentence or acronym expansion at first use, a proper name capitalised at every use); it is exhaustive advice, not a detector (rank AUC 0.174, EVALUATION §23.1). `\newcommand*` definitions are read; calibrating on no passage writes no lexicon and exits 2 (2026-09-27). | +| `deai_collocation.py` | L2 collocation: per sentence, the fraction of distinct adjacent common-word pairs that no corpus passage attests, against a leave-one-out per-bucket reference at sentence unit (p90 advisory, p95 strong). Each flagged pair carries λ = df(a)·df(b)/N and e^−λ so the reader can weigh it. `--glossary` reads the same bank document-wide: every unattested pair used at least twice, with its first-use line and whether that sentence defines it, as the list of the manuscript's own terms to define at first use or replace (2026-09-16). | `--calibrate` writes `collocation_bank.json` (unigram and pair document frequency; 11,282 partner words and 530,504 pairs over 41,644 `wgl` passages) and `collocation_baseline.json`. Unmeasured without the bank; degraded where no bucket clears the 30-sentence floor. Pairs break at punctuation, `[math]`/`[CITE]` and dashes; a sentence needs four judged pairs to carry a fraction. The action names the relation a pair compresses and never changes a claim; a defined term keeps its pair. Since 2026-09-27 `--glossary` is an argparse option (refused beside `--calibrate`, exit 2); a definition cue counts only in the eight words after the pair or a parenthesis opening right after it; `uses` counts every occurrence; the glossary status depends on the bank alone. Sentence findings carry the sentence's own lines. Calibrating a field with no passage writes nothing and exits 2, and the axis status is degraded, with counts, when units fall in `unknown` or a bucket's artifact records another unit. | +| `deai_discourse.py` | L2 discourse texture, two axes on the LOW tail (p10 advisory, p05 strong): `cohesion`, the mean fraction of a sentence's content words that already appeared in the sentence before it, and `hedging`, epistemic markers per 1,000 words. | The defect here is absence, not excess, so both flag below the gate. The two run at **different units and each artifact records its own**, and the reader refuses a bucket recorded at the other unit: cohesion per paragraph, hedging per SECTION, because hedging has no paragraph-scale lower tail at all (p10 is 0.000 in all seven `wgl` buckets). Hedging ships restricted to `intro`, where held-out transfer is 7.89% against 15.48-26.77% elsewhere and every generation regime separates (EVALUATION section 19). Advisories against the field's distribution, never a provenance claim. A manuscript section is one unit per bucket, its subsections pooled, as the reference pools a paper's paragraphs. The status takes the text: a document whose units sit in `unknown` is degraded with the count, never `measured` with nothing read (2026-09-27). | +| `deai_reference.py` | Library, no CLI. One `(feature, unit)` percentile reference shared by every per-bucket axis: the 0.01 quantile grid at nearest rank (the stored p_q is the ceil(q·n)-th smallest value; until 2026-09-27 it was the floor(q·n)-th plus one, one order statistic high), the plateau-top percentile reader, the 30-unit sample floor, the paragraph and section sweeps, the artifact loader, and the calibration loop. | Holds no policy -- gate quantiles, feature names and what a finding means belong to the axis. Its one invariant is that calibration and detection share one unit and one grid; that check is what caught the hedging axis calibrating where no lower tail exists. The artifact loader had five copies across the suite before this module (EVALUATION section 19.1). The invariant is checked, not only declared: `usable_buckets` refuses a bucket whose artifact records another `unit` and the status names it; `sections()` pools a bucket's headings into one unit as the reference side pools a paper; `unbucketed_reason` counts the units a document leaves in `unknown`; `sentence_lines` anchors a sentence-scope finding on its own lines; each bucket records `text_key` and `text_fallback_rows`; `calibrate` returns None (exit 2 in `axis_main`) when no record contributed. Artifacts calibrated before 2026-09-27 keep the old grid (the 30-unit floor read its tails at P = 0.133 and 0.933) until `--calibrate` is rerun. | | `deai_docshape.py` | Document-shape measurement and complete-document calibration: the per-paragraph model-free feature vector, cross-paragraph dispersion, the joint Mahalanobis manifold (pooled and per-length-stratum), role coupling, the split-conformal operating points, and the baseline builder. | Split out of `deai_docstructure.py` on 2026-08-25 at 1,092 lines against a 750-line budget; the seam is measurement versus feedback. `deai_docstructure._paper_documents` assembles a bundle through `es.corpus_documents` in `\include` order — sorted-filename concatenation put `Conclusion.tex` before `Introduction.tex` in 12 of the 500 `wgl` bundles, on the one axis that measures section arc. `document_shape` sweeps `deai_metrics.section_units` and drops the preamble and every `skip` unit (acknowledgements, appendices, references) on both sides; `shape_units`/`measurable_sections` are the one section/paragraph filter `deai_partition` scores with. | | `deai_docstructure.py` | L2 whole-document rhetorical-shape findings and the CLI: turns the measurements above into `sci-paper.feedback.v1` findings. Re-exports every public name from `deai_docshape.py`. | Requires at least three measurable complete documents and sufficient sections/paragraphs; otherwise unmeasured. Legacy baselines without a conformal block fall back to percentile thresholds; factor-drifted role references disable that axis. A document-shape metric is strong when it exceeds the interpolated quantile quoted as `strong_threshold` (the rule the leave-one-document-out rate uses). `--strong-percentile` must lie strictly in (0, 1) (exit 2); `--field` auto-resolves on the read path and must name an existing profile for `--calibrate`. | | `deai_partition.py` | Fidelity-free partition suggestions (merge/split of paragraph blocks) that move a document toward the calibrated human dispersion band. Suggest-only; the author applies changes by hand. | Zero-token operations, so rewrite fidelity holds by construction; cohesion is self-normalized to the document's median adjacent-pair overlap; heading-command blocks (`\section`, `\subsection`, `\chapter`, …) are never candidates; reordering is deliberately not offered. Without a calibrated manifold the tool exits with an explicit message. States are ranked by conformal p (higher = deeper inside the band), distance only as a tie-break within one calibration basis; every plan step records `calibration_basis_before/after`. A split is simulated on the original text at the sentence boundary, line breaks kept; scored blocks pass the same filter as `document_shape`. | | `deai_anchoring.py` | L2 claim-anchoring band: section-class conditional anchored-sentence rates against the human corpus, low-tail conformal with a Bonferroni share per class. | A writing-quality axis, not an AI-discrimination axis (EVALUATION.md 9.6 records the refuted tell). Classes below the 30-document minimum are honestly omitted; unmeasurable documents degrade to unmeasured. A class whose calibration n cannot reach its Bonferroni share (1/(n+1) > alpha/k) is skipped and the axis reported `degraded`; `--calibrate` records `underpowered_classes` and warns. The scan runs on section units with headings, floats and the bibliography blanked and `skip` units dropped. | -| `deai_provenance.py` | Editing-provenance ledger (frontier idea 4): matches each current paragraph to a designated AI-draft ancestor (file or git ref from the author's own history) and labels the span AI-untouched / lightly-edited / rewritten / author-original by a deterministic difflib token edit ratio. | Not an AI detector; reads only the author's own history. `unmeasured` without an ancestor or when the file has no git history for the given ref (never a guess). | +| `deai_provenance.py` | Editing-provenance ledger (frontier idea 4): matches each current paragraph to a designated AI-draft ancestor (file or git ref from the author's own history) and labels the span AI-untouched / lightly-edited / rewritten / author-original by a deterministic difflib token edit ratio. | Not an AI detector; reads only the author's own history. `unmeasured` without an ancestor or when the file has no git history for the given ref (never a guess). Paragraphs come from the shared section sweep, so a title block is not a ledger paragraph (2026-09-27). | | `deai_personal.py` | Personal dispersion baseline (frontier idea 6): places a draft's per-feature within-document dispersion in the distribution of the author's own prior papers, a confound-free same-author reference. | Model-free; reuses `document_shape`. `unmeasured` below three measured prior papers. Flags a draft that varies paragraph shape far less than the author's own papers on at least 40% of features. | -| `deai_oracle.py` | Optional token-surprisal and UID analysis. | Requires transformers/model assets for measurement; compatibility `FLAG_Z` is degraded until field calibration exists. Document-level surprisal is measured (EVALUATION.md 9.8) to be weaker than the model-free manifold and adds nothing to it. | +| `deai_oracle.py` | Optional token-surprisal and UID analysis. | Requires transformers/model assets for measurement; compatibility `FLAG_Z` is degraded until field calibration exists. Document-level surprisal is measured (EVALUATION.md 9.8) to be weaker than the model-free manifold and adds nothing to it. The CUDA fallback survives an empty error message; `--calibrate` skips a blank or undecodable bank line, and on a field with no bank is exit 2 (2026-09-27). | | `deai_features.py` | Reusable model-free, UID, and embedding features for document analysis and learned field-similarity models. | Without the transformers/torch runtime the UID features are `None` with `uid_status: unmeasured` (`features_vector` raises with the reason; the CLI exits 2 with a message); the corpus centroid is `None` without the `.npy` cache and needs no numpy to say so. | -| `deai_voice.py` | Optional learned field-similarity triage. | Refuses bundles with drifted feature names/schema and degrades on corrupt bundles. A measured operating point gates threshold findings; an uncalibrated bundle yields rank-based triage of the lowest-scoring paragraphs, never a universal 0.5 cutoff. | +| `deai_voice.py` | Optional learned field-similarity triage. | Refuses bundles with drifted feature names/schema and degrades on corrupt bundles. A measured operating point gates threshold findings; an uncalibrated bundle yields rank-based triage of the lowest-scoring paragraphs, never a universal 0.5 cutoff. Without the optional transformers/torch runtime the axis is `unmeasured` with that reason (the feature vector needs token surprisal); `--scores` prints the status line and exits 0, and lists paragraphs by source line otherwise; `--field` resolves through the shared resolver (2026-09-27). | | `train_voice_model.py` | Trains the optional learned field-similarity/rewrite-ranking model and writes `voice_model_evaluation.json`. | Cache reuse requires a content/model/centroid fingerprint; featurization checkpoints every 500 rows for preemptible runs. Grouped splits recompute `corpus_cos` from training-only centroids; repeated source-group splits audit source, section, length, mathematical-marker, field-term, matched-stratum, and UID-normalization sensitivity, plus the author-labelled hard-set stratum, without creating an operating point. Requires `--field` to resolve (auto-detected when one profile exists); a missing numpy/scikit-learn/joblib is one line and exit 2. Repeated audit splits start one seed past the primary split. | | `voice_dataset.py` | Record loading, source-family grouping, the training-only field lexicon, and the fingerprinted feature-matrix cache for the voice model. Split out of `train_voice_model.py` on 2026-08-26 (1,174 lines against a 750-line budget). | Depends on nothing in `voice_audit`; the split is one-way and a test asserts it. The split's hand-written import block left `time` and `CHECKPOINT_EVERY` unbound and the suite stayed green, because no test reached `build_features` — a scope-aware symbol-table check for unbound names now runs in `test_train_voice_model.py`. A run whose encoder fails while sentence-transformers imports never becomes the cache: the language-model rows are kept as a finished checkpoint and the embedder is retried next run. | | `voice_audit.py` | Held-out metrics, bootstrap AUC intervals, the section-normalized UID alternative, the author hard set, and the repeated grouped confound audits. Owns `HARDSET_AI_CATEGORIES` / `HARDSET_HUMAN_CATEGORIES`. | Fits no model and creates no operating point: it produces the evidence a bundle needs before it can leave `degraded`. `train_voice_model` re-exports every public name, guarded by a contract test. | diff --git a/tools/ai_ism_lint.py b/tools/ai_ism_lint.py index 7fe89fa..18420d9 100644 --- a/tools/ai_ism_lint.py +++ b/tools/ai_ism_lint.py @@ -462,19 +462,19 @@ def collect_feedback(path: Path, field_profile_dir: Path | None, *, # its axis entry rides along when the sentence axis did not add it findings.extend(deai_collocation.glossary_findings(text, field_profile_dir, path)) if not collocation: - axes.append(deai_collocation.collocation_axis_status(field_profile_dir, text)) + axes.append(deai_collocation.glossary_axis_status(field_profile_dir)) if residue: findings.extend(deai_residue.residue_findings(text, path)) axes.append(deai_residue.residue_axis_status(text)) if salience: findings.extend(deai_salience.salience_findings(text, field_profile_dir, path)) - axes.append(deai_salience.salience_axis_status(field_profile_dir)) + axes.append(deai_salience.salience_axis_status(field_profile_dir, text)) if discourse: findings.extend(deai_discourse.discourse_findings(text, field_profile_dir, path)) # extend, not append: this detector reports one status PER FEATURE, # because cohesion and hedging calibrate at different units and a # field can support one and not the other. - axes.extend(deai_discourse.discourse_axis_status(field_profile_dir)) + axes.extend(deai_discourse.discourse_axis_status(field_profile_dir, text)) if distribution: findings.extend(deai_metrics.distribution_findings(text, field_profile_dir, path)) axes.append(deai_metrics.distribution_axis_status(field_profile_dir)) From d42ef147cc6d6ac0946969d444fcb00d70c8d689 Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 04:48:01 -0400 Subject: [PATCH 21/27] audit follow-ups: the items section 10 left open, the PR's own regressions, invented examples Audit 2026-09-27 section 8.3, each item settled (the audit's section 10 records it): - F26: `robust` directly before estimator(s) / estimation / statistic(s) / regression is a method name and no longer counts toward the Tier B cap, across a line break too; `robust estimate` still counts. The paper skill says so beside its Tier B rule. - B17: the Mondrian strata edges are fitted on the training records only, as exchangeability requires; the calibration records no longer place them. - B18: a docstructure feature whose baseline cannot resolve its own tail (fewer than ceil(1 / min(p, 1 - p)) documents, 20 at 0.05 / 0.95) reports degraded and ordinary instead of strong. - E17: the paired leakage estimate counts the paper as the bank holds it: the paragraphs extract_style.bank_paragraphs admits (30-400 words, the unknown bucket excluded) and the terms deai_register.passage_terms reads, overlaid on the df table, so `halo` no longer matches inside `halos`. - E20: corpus_cos is a 0.0 placeholder without the embedder or the corpus centroid; a bundle that reads it now scores unmeasured (deai_voice.scoring_reason), and deai_features builds the feature only when both exist and says so on stderr otherwise. - B23: deai_features' CLI walks the shared paragraph sweep, placeholders are stripped by the shared helper, the second `_MATH_MARKER_RE` is renamed for what it measures, and anchoring classes each paragraph by the shared bucket instead of a third section classifier. - F22: CrossRef requests carry a polite-pool mailto from SCI_PAPER_MAILTO and retry once on 429 / 5xx, honouring a capped Retry-After. - D20: the exemplar embedding cache carries a sidecar fingerprint (bank sha1, row count, model) and is rebuilt when it disagrees. - B16: closed with its reason beside DEFAULT_BURSTINESS_RATIO. Found while reviewing the PR: - extract_style.words() split hyphenated compounds and apostrophes after the tokenizer change; a compound is one word again. - unit_reason named foreign-unit buckets the document never reaches and the axis never reads; it is scoped to both. - tex_macros read definitions from commented-out lines, so a commented \newcommand could beat the live one; definitions now come from the comment-blanked copy and are blanked at equal length in both copies. Examples in docstrings, comments and test fixtures that had been taken from a manuscript under review are replaced by invented ones. Co-Authored-By: Claude Opus 5.5 --- skills/paper/SKILL.md | 5 ++- tests/test_ai_ism_lint_cli.py | 13 ++++++ tests/test_condense_map.py | 16 +++---- tests/test_deai_anchoring.py | 22 +++++---- tests/test_deai_collocation.py | 12 ++--- tests/test_deai_docstructure.py | 24 ++++++++++ tests/test_deai_reference.py | 17 +++++-- tests/test_deai_register.py | 41 ++++++++--------- tests/test_deai_residue.py | 64 +++++++++++++------------- tests/test_deai_salience.py | 72 +++++++++++++++++++----------- tests/test_deai_structure.py | 65 +++++++++++++-------------- tests/test_deai_voice.py | 66 +++++++++++++++++++++++---- tests/test_eval_findings.py | 16 +++++++ tests/test_extractor_statistics.py | 51 +++++++++++++++++++++ tests/test_tex_macros.py | 45 ++++++++++++------- tests/test_train_voice_model.py | 9 ++-- tests/test_verify_references.py | 54 ++++++++++++++++++++++ tools/ai_ism_lint.py | 20 +++++++-- tools/deai_anchoring.py | 35 +++++++-------- tools/deai_collocation.py | 14 +++--- tools/deai_discourse.py | 7 +-- tools/deai_docshape.py | 4 +- tools/deai_docstructure.py | 60 +++++++++++++++++++++---- tools/deai_features.py | 31 ++++++++----- tools/deai_metrics.py | 7 +++ tools/deai_reference.py | 30 ++++++++++--- tools/deai_register.py | 43 ++++++++++-------- tools/deai_residue.py | 15 ++++--- tools/deai_salience.py | 7 +-- tools/deai_structure.py | 10 ++--- tools/deai_voice.py | 53 +++++++++++++++++++--- tools/eval_findings.py | 27 ++++++++--- tools/extract_sections.py | 2 +- tools/extract_style.py | 42 ++++++++--------- tools/retrieve_exemplars.py | 71 ++++++++++++++--------------- tools/tex_macros.py | 63 ++++++++++++++++---------- tools/verify_references.py | 65 ++++++++++++++++++++------- 37 files changed, 824 insertions(+), 374 deletions(-) diff --git a/skills/paper/SKILL.md b/skills/paper/SKILL.md index eb23be3..1bb8493 100644 --- a/skills/paper/SKILL.md +++ b/skills/paper/SKILL.md @@ -149,7 +149,7 @@ deleting its hedges has been damaged. - 高精度（strong）：`previous draft`、`earlier draft`、`previous version`、`earlier version`、`supersedes`、`superseded`、`no longer`、`we tried`、`we switched`、`we abandoned`、`we replaced`、`we moved away from` - 普通（ordinary，常见科学用词，只提示；`initially`/`originally`/`at first` 在已发表论文里多是流程顺序而非改稿史）：`initially`、`originally`、`at first`、`now`、`currently`、`earlier`、`previously`、`corrected`、`revised`、`updated` -5. 是否有句子**用缺席来描述本文自己的对象**（菜单上写"番茄炒蛋（无东坡肉）"）？"The head never participates in the decision" / "the reference carries no quoted number" / "no support threshold is applied, because …" 说的都是对象**不做**什么，读者看到的是一个空位而不是那件东西；改成它**做**什么："The head runs downstream of the decision" / "a labeled reference only"。先删后写：结合上下文判断，相邻句子已经说了对象做什么的，整句或该分句直接删（"downstream of detection" 在一篇稿子里出现五次，就是只改写不删的后果）；只有页面上还没有的信息才改写成正向陈述。删掉的否定词会被 `rewrite_reward` 的 §6 否定不变量报为 missing，按同一 disposition 记录。物理事实、范围限制、让步性局限（"the B-mode map carries no lensing signal"、"shear alone cannot break the mass-sheet degeneracy"）是承重的，保留并记录 disposition；检测端 `residue-absence` 对同句带引用的对照不报。词族与 `tools/deai_residue.py` 同源，`validate_plugin` 比对两处；分档依据 442 篇 held-out 已发表全文（190 万词）：strong 档在其中每千词 0.008 次，ordinary 档 0.02–0.05 次且多为流程语句： +5. 是否有句子**用缺席来描述本文自己的对象**（菜单上写"番茄炒蛋（无东坡肉）"）？"The photometric catalog never enters the shape measurement" / "the comparison sample carries no redshift estimate" / "no colour cut is applied, because …" 说的都是对象**不做**什么，读者看到的是一个空位而不是那件东西；改成它**做**什么："The shapes are measured on the images alone" / "a photometry-only comparison sample"。先删后写：结合上下文判断，相邻句子已经说了对象做什么的，整句或该分句直接删（同一个限定短语在一篇稿子里出现五次，就是只改写不删的后果）；只有页面上还没有的信息才改写成正向陈述。删掉的否定词会被 `rewrite_reward` 的 §6 否定不变量报为 missing，按同一 disposition 记录。物理事实、范围限制、让步性局限（"the B-mode map carries no lensing signal"、"shear alone cannot break the mass-sheet degeneracy"）是承重的，保留并记录 disposition；检测端 `residue-absence` 对同句带引用的对照不报。词族与 `tools/deai_residue.py` 同源，`validate_plugin` 比对两处；分档依据 442 篇 held-out 已发表全文（190 万词）：strong 档在其中每千词 0.008 次，ordinary 档 0.02–0.05 次且多为流程语句： - strong：`never`、`nothing is`、`nothing was`、`none sees`、`none enters`、`no ... is applied`（模板：以 no 开头，接一到三个词的对象，再接 is/are/was/were + applied/used/combined/quoted/imposed） - ordinary：`carries no`、`carry no`、`carrying no`、`holds no`、`touches no`、`sees no`、`enters no`、`makes no`、`is not applied`、`are not applied`、`is not used`、`are not used`、`is not quoted`、`is not combined`、`is not treated`、`are not treated`、`is not reproduced`、`does not participate`、`do not participate`、`does not enter`、`does not claim`、`does not see`、`do not see` @@ -298,6 +298,9 @@ Tier B 可以使用，但同一个 Tier B 词在同一 heading 单元（`\sectio `foster/fosters/fostering/fostered`（后两组 2026-07-16 加入；curated corpus 各有 1 次出现，非零故不入 Tier A）。经验频率只从当前 profile 读取，不在 本文件复制。优先用直接陈述或可验证数字，但不要为了避词而损害准确性。 +`robust` 紧接 `estimator(s)` / `estimation` / `statistic(s)` / `regression` +时是 robust statistics 的方法名，不计入 Tier B（跨行也算）；`robust estimate` +仍计入，因为它同样常指"可靠的估计"。 **模糊量化 / 修饰词** - `a wide range of`, `a variety of`, `a number of`, `several`, `numerous`, diff --git a/tests/test_ai_ism_lint_cli.py b/tests/test_ai_ism_lint_cli.py index fc2aec6..ba6df9d 100644 --- a/tests/test_ai_ism_lint_cli.py +++ b/tests/test_ai_ism_lint_cli.py @@ -73,6 +73,19 @@ def test_tier_b_excess_returns_one(self): self.assertEqual(result.returncode, 1, result.stderr) self.assertIn("tier-b-excess:robust", result.stdout) + def test_a_robust_statistics_term_is_not_tier_b(self): + # The method's name, wrapped or not, leaves the cap to the adjective. + methods = ("\\section{Methods}\nWe adopt a robust estimator of the mean.\n" + "The robust\nestimator down-weights outliers.\n" + "The fit is robust to the prior.\n") + result = self.run_lint(methods, *self.isolated) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("L0=0", result.stdout) + result = self.run_lint(methods + "A robust estimate of the mass follows.\n", + *self.isolated) + self.assertEqual(result.returncode, 1, result.stderr) + self.assertIn("tier-b-excess:robust", result.stdout) + def test_explicit_unavailable_field_is_configuration_failure(self): with tempfile.TemporaryDirectory() as temporary: profile_root = Path(temporary) diff --git a/tests/test_condense_map.py b/tests/test_condense_map.py index c87b15e..9e956aa 100644 --- a/tests/test_condense_map.py +++ b/tests/test_condense_map.py @@ -11,15 +11,15 @@ import condense_map as cm -CLAIM = ("The aperture mass filter recovers the injected cluster signal at the " - "fiducial threshold for every configuration in the calibrated grid.") +CLAIM = ("The aperture mass filter recovers the simulated cluster signal at the " + "fiducial threshold for every filter scale in the adopted range.") RESTATED = ("At the fiducial threshold the aperture mass filter recovers the " - "injected cluster signal for every calibrated grid configuration.") + "simulated cluster signal for every adopted filter scale in the range.") PARAGRAPH = ("The shear catalog is cut at a signal-to-noise ratio of ten, the " "tangential shear is convolved with a compensated filter of fixed " "scale, and the aperture mass map is evaluated on a regular grid " - "in catalog coordinates so that every peak has a position and a " - "height that the injection grid can reproduce exactly.") + "across the survey area so that every peak has a position and a " + "height that the simulated maps can reproduce exactly.") DOCUMENT = f"""\\newcommand{{\\unused}}{{x}} \\newcommand{{\\used}}{{y}} \\section{{Introduction}} @@ -218,9 +218,9 @@ def test_a_heading_does_not_fuse_with_the_first_sentence(self): class GateAgreementTests(unittest.TestCase): """The map's words are the gate's words, or its target cannot be met.""" - SENTENCE = ("The calibrated aperture mass filter recovers the injected cluster " - "signal at $\\nu = 3$ for every configuration \\citep{Ref} in the " - "grid of $N = 20$ fields.") + SENTENCE = ("The adopted aperture mass filter recovers the simulated cluster " + "signal at $\\nu = 3$ for every filter scale \\citep{Ref} in the " + "range of $N = 20$ fields.") def test_removable_words_exclude_projection_placeholders(self): import length_gate as lg diff --git a/tests/test_deai_anchoring.py b/tests/test_deai_anchoring.py index ddd6d36..82f0bf6 100644 --- a/tests/test_deai_anchoring.py +++ b/tests/test_deai_anchoring.py @@ -62,15 +62,19 @@ def test_sentence_anchor_detection(self): "This was argued by \\cite{Smith2020} convincingly." .replace("\\cite{Smith2020}", ""))) - def test_section_classification(self): - self.assertEqual(anchoring.classify_section("Introduction"), "intro") - self.assertEqual(anchoring.classify_section("Data and Methods"), - "methods") - self.assertEqual(anchoring.classify_section("Results"), "results") - self.assertEqual(anchoring.classify_section("Summary and outlook"), - "conclusions") - self.assertEqual(anchoring.classify_section("Lensing formalism"), - "other") + def test_the_class_is_the_shared_bucket(self): + # One heading table: `Lensing formalism` names a method, a topic + # subsection inherits Methods, and `Comparison with simulations` is + # discussion -- the module's own table said other, other, methods. + result = anchoring.document_anchoring( + "\\section{Lensing formalism}\n" + ANCHORED + "\n" + "\\section{Methods}\n\\subsection{Weak lensing}\n" + ANCHORED + "\n" + "\\section{Comparison with simulations}\n" + ANCHORED + "\n" + "\\section{Summary and outlook}\n" + ANCHORED + "\n") + self.assertEqual([row["class"] for row in result["sections"]], + ["methods", "methods", "discussion", "conclusions"]) + self.assertEqual(anchoring.section_class("data"), "methods") + self.assertEqual(anchoring.section_class("unknown"), "other") def test_unanchored_results_flag_and_anchored_do_not(self): with tempfile.TemporaryDirectory() as temporary: diff --git a/tests/test_deai_collocation.py b/tests/test_deai_collocation.py index 484f75c..5833587 100644 --- a/tests/test_deai_collocation.py +++ b/tests/test_deai_collocation.py @@ -23,9 +23,9 @@ "The tangential shear is convolved with a compensated filter of fixed scale. " "The detection threshold is set by the noise of the aperture mass map. " "The shape noise of the source galaxies sets the detection floor.") -MODIFIERS = ("calibrated", "controlled", "empty", "physical", "deployed", +MODIFIERS = ("calibrated", "smoothed", "empty", "binned", "deployed", "clamped", "rescored", "strict") -NOUNS = ("filter", "grid", "saddle", "cell", "boundary", "catalog", "map", "floor") +NOUNS = ("filter", "grid", "kernel", "cell", "boundary", "catalog", "map", "floor") # Pseudo-words that occur in exactly one passage each (letters only: the word # regex admits no digits). COINED = ["".join(letters) for letters in itertools.product("bcdfgh", repeat=3)] @@ -53,7 +53,7 @@ def records(count: int = 64) -> list[dict]: "catalog around each cluster. The tangential shear is convolved with " "a compensated filter of fixed scale.\n") # Every word below is common in the bank; none of the pairs is written by it. -NOVEL = ("\\section{Methods}\nThe calibrated saddle deployed a physical floor " +NOVEL = ("\\section{Methods}\nThe calibrated kernel deployed a binned floor " "grid catalog for the empty threshold filter response map.\n") @@ -64,9 +64,9 @@ def test_function_words_are_not_partners(self): self.assertNotIn(("the", "shear"), pairs) def test_adjacent_content_words_pair_in_order(self): - pairs = collocation.content_pairs("The calibrated blur sets the floor.") - self.assertIn(("calibrated", "blur"), pairs) - self.assertNotIn(("blur", "calibrated"), pairs) + pairs = collocation.content_pairs("The binned kernel sets the floor.") + self.assertIn(("binned", "kernel"), pairs) + self.assertNotIn(("kernel", "binned"), pairs) def test_punctuation_and_placeholders_break_a_pair(self): pairs = collocation.content_pairs( diff --git a/tests/test_deai_docstructure.py b/tests/test_deai_docstructure.py index f312fd2..7b5a638 100644 --- a/tests/test_deai_docstructure.py +++ b/tests/test_deai_docstructure.py @@ -297,6 +297,9 @@ def test_over_uniform_document_is_flagged_but_varied_is_not(self): uniformity_hits = [f for f in uniform if f["rule"].startswith("document-uniformity:")] self.assertTrue(uniformity_hits, "over-uniform document should flag") + # Four documents resolve no 5% tail: context, not a strong finding. + self.assertEqual({(f["strength"], f["measurement_status"]) + for f in uniformity_hits}, {("ordinary", "degraded")}) varied = docstructure.document_findings(document([ REPEATED_PARAGRAPH, SHORT_PARAGRAPH, LONG_PARAGRAPH, SHORT_PARAGRAPH, LONG_PARAGRAPH, REPEATED_PARAGRAPH, @@ -449,6 +452,27 @@ def test_strong_flag_uses_the_quoted_threshold_rule(self): self.assertAlmostEqual(finding["normalized_distance"], 0.35 - 0.34) self.assertIn("threshold 0.340", finding["message"]) + def test_a_tail_the_reference_cannot_resolve_is_not_strong(self): + # Four documents place no value beyond a 0.8 quantile (floor 5); a + # 5% band tail needs 20 (audit B18). + self.assertEqual([docstructure.tail_floor(p) for p in (0.95, 0.05, 0.9, 0.8)], + [20, 20, 10, 5]) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + for values, expected in (([0.1, 0.2, 0.3, 0.4], ("ordinary", "degraded")), + ([0.01 * i for i in range(20)], ("strong", "measured"))): + self._baseline(root, values, 0.8) + with mock.patch.object(docstructure, "document_shape", + return_value=self._shape(0.35)): + finding, = docstructure.document_findings("x", root) + status = docstructure.docstructure_axis_status("x", root) + self.assertEqual((finding["strength"], finding["measurement_status"]), expected) + self.assertEqual(status["status"], expected[1]) + self._baseline(root, [0.1, 0.2, 0.3, 0.4], 0.8) + with mock.patch.object(docstructure, "document_shape", + return_value=self._shape(0.35)): + self.assertIn("needs 20", docstructure.docstructure_axis_status("x", root)["reason"]) + class FeatureRuntimeTests(unittest.TestCase): """`deai_features` without its optional dependencies: unmeasured markers diff --git a/tests/test_deai_reference.py b/tests/test_deai_reference.py index 705e0f4..f2c524c 100644 --- a/tests/test_deai_reference.py +++ b/tests/test_deai_reference.py @@ -4,10 +4,10 @@ number, and blanking a `\\section{...}\\label{...}` line leaves the label and a run of spaces behind; a comment-only block never had prose at all. Until v0.36.3 both reached every per-paragraph axis as units of their own: -GPT-2 tokenises forty-eight spaces into more than the UID minimum, so the -Letter's four heading lines were reported as paragraphs of near-zero -surprisal variance (z = -7), and a `% SCOPE:` comment block counted as a -unit of the removal map. +GPT-2 tokenises forty-eight spaces into more than the UID minimum, so four +heading lines of a manuscript under review were reported as paragraphs of +near-zero surprisal variance (z = -7), and a `% SCOPE:` comment block +counted as a unit of the removal map. """ from __future__ import annotations @@ -130,6 +130,15 @@ def test_the_foreign_units_are_named_with_what_they_record(self): self.assertIn("recalibrate", reason) self.assertIsNone(reference.unit_reason({"intro": self.BASELINE["intro"]}, "section")) + def test_only_the_buckets_the_axis_would_read_count(self): + intro = [(1, 3, "intro", "prose")] + self.assertIsNone(reference.unit_reason(self.BASELINE, "section", intro)) + reason = reference.unit_reason(self.BASELINE, "section", + intro + [(4, 6, "method", "prose")]) + self.assertIn("method (paragraph)", reason) + self.assertNotIn("data", reason) + self.assertIsNone(reference.unit_reason(self.BASELINE, "section", allowed=("intro",))) + class UnbucketedUnitsTests(unittest.TestCase): def test_untitled_prose_is_counted_as_unmeasured(self): diff --git a/tests/test_deai_register.py b/tests/test_deai_register.py index 4069368..932a0b9 100644 --- a/tests/test_deai_register.py +++ b/tests/test_deai_register.py @@ -22,8 +22,9 @@ def build_profile(directory: Path, passages: list[str]) -> Path: NATIVE_PASSAGE = ( "The aperture mass is measured on the shear catalog at each epoch, and " "the accuracy of the recovered convergence is quoted against the " - "injection grid. The sub-halo population is drawn from the same plane, " - "and the classifier used for the shortlist is held out from training." + "simulated maps. The sub-halo population is drawn from the same " + "simulation, and the classifier used for the galaxy selection is held " + "out from training." ) @@ -39,7 +40,7 @@ def foreign_document(term: str = "logit") -> str: class TestTermNormalisation(unittest.TestCase): def test_possessive_folds_onto_the_bare_term(self): - self.assertEqual(register.normalize("sub-halo's"), "sub-halo") + self.assertEqual(register.normalize("point-source's"), "point-source") def test_case_folds(self): self.assertEqual(register.normalize("AUC"), "auc") @@ -89,7 +90,7 @@ def test_single_word_uses_its_own_count(self): (1, "auc")) def test_a_compound_with_one_short_part_is_judged_on_its_long_part(self): - # `no-dip`: "no" is too short to carry meaning, "dip" is the word. + # `no-mass`: "no" is too short to carry meaning, "mass" is the word. self.assertEqual(register.corpus_document_frequency("no-mass", self.TABLE), (900, "mass")) @@ -192,8 +193,8 @@ def test_an_acronym_expanded_in_text_is_defined_here(self): self.assertEqual(finding["observed"]["justification"], "defined-here") def test_a_defining_sentence_marks_a_coined_term(self): - finding = self._one("We call this quantity the saddleness. " - "The saddleness is measured.", "saddleness") + finding = self._one("We call this quantity the shearability. " + "The shearability is measured.", "shearability") self.assertEqual(finding["observed"]["justification"], "defined-here") self.assertIn("definition precedes every use", finding["message"]) @@ -293,31 +294,31 @@ def test_macros_defined_in_the_preamble_still_resolve(self) -> None: self.assertIn("auc", register.manuscript_terms(document)) # Third instance of the asymmetry: the corpus side blanks every math span, - # detection blanked only what fit on one line. On the advisor-round Letter - # the macro names inside a multi-line equation (`\dsep`, `\rmain`), a float + # detection blanked only what fit on one line. On a manuscript under review + # the macro names inside a multi-line equation, a float # option (`htb`), a bibliography style and a `\texttt` identifier all # surfaced as zero-hit "words". def test_a_macro_inside_multi_line_math_is_not_a_word(self) -> None: document = ("\\section{Methods}\n" "The convergence is\n" "\\begin{equation}\n" - "\\kappa = \\frac{\\kappa_\\rmain(\\dsep)}{2}\n" + "\\kappa = \\frac{\\kappa_\\rbase(\\lagsep)}{2}\n" "\\end{equation}\n" - "and $\\dsep\n" + "and $\\lagsep\n" "/ r$ spans lines.\n") terms = register.manuscript_terms(document) - self.assertNotIn("rmain", terms) - self.assertNotIn("dsep", terms) + self.assertNotIn("rbase", terms) + self.assertNotIn("lagsep", terms) self.assertIn("convergence", terms) self.assertIn("spans", terms) def test_float_options_bibliography_commands_and_code_are_not_words(self) -> None: document = ("\\section{Methods}\n" "\\begin{figure*}[!htb]\n\\end{figure*}\n" - "The \\texttt{cw\\_mscale} mode is used.\n" + "The \\texttt{run\\_nbins} mode is used.\n" "\\bibliographystyle{aasjournalv7}\n") terms = register.manuscript_terms(document) - for leaked in ("htb", "mscale", "aasjournalv", "aasjournalv7"): + for leaked in ("htb", "nbins", "aasjournalv", "aasjournalv7"): self.assertNotIn(leaked, terms) self.assertIn("mode", terms) self.assertEqual(terms["mode"]["line"], 4) @@ -328,7 +329,7 @@ def test_floats_and_table_preambles_are_blanked_across_lines(self) -> None: # and a caption's words counted here only (14 of 90 zero-hit terms on # one manuscript, 2026-09-04). A panel range `a--c` is not a word. document = ("\\section{Methods}\n" - "\\begin{table*}\n\\caption{Rates in crimson.}\n" + "\\begin{table*}\n\\caption{Rates in ochre.}\n" "\\begin{tabular*}{\\textwidth}{@{}l@{\\extracolsep{\\fill}}lll@{}}\n" "a & b \\\\\n\\end{tabular*}\n\\end{table*}\n" "\\begin{deluxetable*}{@{\\extracolsep{0pt plus 1filll}}lcl}\n" @@ -337,7 +338,7 @@ def test_floats_and_table_preambles_are_blanked_across_lines(self) -> None: "\\setlength{\\tabcolsep}{2pt}\n" "Figure~\\ref{fig:x}a--c draws the shear catalog.\n") terms = register.manuscript_terms(document) - for leaked in ("filllll", "filll", "crimson", "scriptsize", "atied", + for leaked in ("filllll", "filll", "ochre", "scriptsize", "atied", "tabcolsep", "textwidth", "a--c"): self.assertNotIn(leaked, terms) self.assertIn("shear", terms) @@ -399,14 +400,14 @@ class DefinitionScopeTest(unittest.TestCase): def test_the_object_after_a_defining_phrase(self) -> None: self.assertEqual(register.defined_terms("We define flux using quuxification."), {"flux"}) - found = register.defined_terms("We call this quantity the saddleness.") - self.assertIn("saddleness", found) + found = register.defined_terms("We call this quantity the shearability.") + self.assertIn("shearability", found) self.assertNotIn("quuxification", found) def test_the_object_before_a_defining_phrase(self) -> None: found = register.defined_terms( - "The saddleness is defined as the ratio of quuxification.") - self.assertIn("saddleness", found) + "The shearability is defined as the ratio of quuxification.") + self.assertIn("shearability", found) self.assertNotIn("quuxification", found) self.assertNotIn("ratio", found) self.assertEqual(register.defined_terms( diff --git a/tests/test_deai_residue.py b/tests/test_deai_residue.py index 2108b73..dd42fd3 100644 --- a/tests/test_deai_residue.py +++ b/tests/test_deai_residue.py @@ -82,26 +82,26 @@ def found(self, text): return residue.absence_findings(text) def test_never_is_strong(self): - found = self.found("\\section{Methods}\nThe characterization head never " - "participates in the detection decision.\n") + found = self.found("\\section{Methods}\nThe photometric catalog never " + "enters the shape measurement.\n") self.assertEqual([(f["rule"], f["strength"], f["observed"]["phrase"]) for f in found], [("residue-absence", "strong", "never")]) def test_the_template_form_is_strong(self): - found = self.found("\\section{Methods}\nNo minimum configuration support is " - "applied at any layer.\n") + found = self.found("\\section{Methods}\nNo minimum magnitude cut is " + "applied at any stage.\n") self.assertEqual([f["strength"] for f in found], ["strong"]) self.assertEqual(found[0]["observed"]["phrase"], - "no minimum configuration support is applied") + "no minimum magnitude cut is applied") def test_carries_no_is_ordinary(self): - found = self.found("\\section{Data}\nThe reference stratum carries no quoted " - "number.\n") + found = self.found("\\section{Data}\nThe comparison sample carries no redshift " + "estimate.\n") self.assertEqual([(f["strength"], f["observed"]["phrase"]) for f in found], [("ordinary", "carries no")]) def test_a_hyphenated_compound_is_a_name(self): - self.assertEqual(self.found("\\section{Data}\nThe never-touched controls are " + self.assertEqual(self.found("\\section{Data}\nThe never-masked tiles are " "scored last.\n"), []) def test_a_citation_makes_it_a_baseline_contrast(self): @@ -189,23 +189,23 @@ def test_a_procedure_we_have_added_is_not_an_editing_mark(self): class NegativeLabelTests(unittest.TestCase): def test_a_caption_negating_something_the_body_never_names_is_ordinary(self): - text = long_body("\\begin{figure}\\caption{Peak counts without the saddle " + text = long_body("\\begin{figure}\\caption{Peak counts without the dust " "correction.}\\end{figure}") found = residue.negative_label_findings(text) self.assertEqual(len(found), 1) self.assertEqual(found[0]["rule"], "residue-negative-label") self.assertEqual(found[0]["strength"], "ordinary") - self.assertIn("saddle", found[0]["observed"]["absent_from_body"]) + self.assertIn("dust", found[0]["observed"]["absent_from_body"]) def test_a_negation_whose_object_the_body_names_is_fine(self): - text = long_body("The saddle correction is applied to every map. " - "\\begin{figure}\\caption{Peak counts without the saddle " + text = long_body("The dust correction is applied to every map. " + "\\begin{figure}\\caption{Peak counts without the dust " "correction.}\\end{figure}") self.assertEqual(residue.negative_label_findings(text), []) def test_a_short_document_is_not_judged(self): text = ("\\section{Methods}\nShort.\n\\begin{figure}\\caption{Peak counts " - "without the saddle correction.}\\end{figure}\n") + "without the dust correction.}\\end{figure}\n") self.assertEqual(residue.negative_label_findings(text), []) self.assertIn("not applied", residue.residue_axis_status(text)["reason"]) @@ -229,27 +229,27 @@ def test_a_space_before_the_brace_or_a_short_title_is_still_a_label(self): # LaTeX allows `\section {X}` and `\caption[short]{X}`; both were # invisible to the label rule (audit A9 of 2026-09-04, not landed here). self.assertEqual([label for _line, label in residue._labels( - "\\section {No saddle}\n\\caption [short] {Without the dip}")], - ["No saddle", "Without the dip"]) - text = long_body("\\begin{figure}\\caption {Peak counts without the saddle " + "\\section {No dust}\n\\caption [short] {Without the mask}")], + ["No dust", "Without the mask"]) + text = long_body("\\begin{figure}\\caption {Peak counts without the dust " "correction.}\\end{figure}") self.assertEqual([f["observed"]["negated_object"] for f in residue.negative_label_findings(text)], - ["the saddle correction"]) + ["the dust correction"]) def test_an_object_named_in_an_appendix_is_not_absent(self): # Labels are read from the whole text, so the object is looked for in # the whole paper too: an appendix caption negating what the appendix # explains three lines above it was reported absent from the body. - appendix = ("\\section{Appendix}\nThe saddle correction is applied to every " + appendix = ("\\section{Appendix}\nThe dust correction is applied to every " "map in this appendix.\n\\begin{figure}\\caption{Peak counts " - "without the saddle correction.}\\end{figure}\n") + "without the dust correction.}\\end{figure}\n") self.assertEqual(residue.negative_label_findings(long_body() + appendix), []) body_caption = long_body("\\begin{figure}\\caption{Peak counts without the " - "saddle correction.}\\end{figure}") + "dust correction.}\\end{figure}") self.assertEqual(len(residue.negative_label_findings(body_caption)), 1) self.assertEqual(residue.negative_label_findings( - body_caption + "\\section{Appendix}\nThe saddle correction is " + body_caption + "\\section{Appendix}\nThe dust correction is " "described here.\n"), []) def test_the_ation_family_reduces_to_one_root(self): @@ -266,32 +266,32 @@ def test_the_ation_family_reduces_to_one_root(self): class NegativeLabelAddedTests(unittest.TestCase): def test_a_patched_absence_is_strong(self): - before = long_body("The saddle correction is applied to every map.") - after = long_body("\\begin{figure}\\caption{Peak counts without the saddle " + before = long_body("The dust correction is applied to every map.") + after = long_body("\\begin{figure}\\caption{Peak counts without the dust " "correction.}\\end{figure}") found = residue.negative_label_added_findings(before, after) self.assertEqual(len(found), 1) self.assertEqual(found[0]["rule"], "residue-negative-label-added") - self.assertIn("saddle", found[0]["observed"]["removed_from_body"]) + self.assertIn("dust", found[0]["observed"]["removed_from_body"]) def test_a_label_already_present_before_is_not_new(self): - text = long_body("\\begin{figure}\\caption{Peak counts without the saddle " + text = long_body("\\begin{figure}\\caption{Peak counts without the dust " "correction.}\\end{figure}") self.assertEqual(residue.negative_label_added_findings(text, text), []) def test_an_object_the_body_never_had_is_not_a_patched_absence(self): - # `correction` alone was in the body; `saddle correction` never was, so + # `correction` alone was in the body; `dust correction` never was, so # the new caption negates nothing the edit removed. before = long_body("The correction is applied.") after = long_body("The measurement is applied. \\begin{figure}\\caption{" - "Without the saddle correction}\\end{figure}") + "Without the dust correction}\\end{figure}") self.assertEqual(residue.negative_label_added_findings(before, after), []) def test_a_negation_the_old_caption_carried_is_not_new_when_the_caption_changes(self): - before = long_body("The saddle correction is applied. \\begin{figure}" - "\\caption{Blue points, without the saddle correction.}" + before = long_body("The dust correction is applied. \\begin{figure}" + "\\caption{Blue points, without the dust correction.}" "\\end{figure}") - after = long_body("\\begin{figure}\\caption{Red points, without the saddle " + after = long_body("\\begin{figure}\\caption{Red points, without the dust " "correction.}\\end{figure}") self.assertEqual(residue.negative_label_added_findings(before, after), []) @@ -312,8 +312,8 @@ def test_clean_document_exits_zero(self): self.assertIn("axis L4.residue: measured", result.stdout) def test_the_diff_rule_runs_with_a_baseline(self): - before = long_body("The saddle correction is applied to every map.") - after = long_body("\\begin{figure}\\caption{Peak counts without the saddle " + before = long_body("The dust correction is applied to every map.") + after = long_body("\\begin{figure}\\caption{Peak counts without the dust " "correction.}\\end{figure}") result = run_residue(after, "--format", "json", before=before) self.assertEqual(result.returncode, 1, result.stderr) diff --git a/tests/test_deai_salience.py b/tests/test_deai_salience.py index 1524868..d3c3634 100644 --- a/tests/test_deai_salience.py +++ b/tests/test_deai_salience.py @@ -17,49 +17,49 @@ # Eight sentences; the middle four each carry a measured quantity, so the # longest uninterrupted recital run is 4 of 8. RECITAL_HEAVY = ( - "The detection step selects on the depth of the inter-peak dip. " - "We ran the grid over $500$ detector configurations. " - "The strict label holds for $34.6\\%$ of resolvable rows. " - "Configurations with no dip detect at $0.98\\%$. " - "The held-out split reaches $0.813$ to $0.843$. " - "The criterion transfers to unseen mass ratios. " + "The calibration step corrects each galaxy's shear for shape noise. " + "We simulated the survey with $200$ image realisations. " + "The multiplicative bias is $1.7\\%$ for bright galaxies. " + "Faint galaxies carry a bias of $4.2\\%$. " + "The additive term lies between $0.003$ and $0.006$. " + "The correction transfers to unseen seeing conditions. " "Its factors follow from the linearity of the estimator. " - "The catalog is therefore saddle-limited rather than mass-selected." + "The catalog is therefore noise-limited rather than depth-limited." ) # Same eight sentences' worth of numbers, but interleaved with the statements # that give them consequence, so no run exceeds one. RANKED = ( - "The detection step selects on the depth of the inter-peak dip. " - "Configurations whose map keeps no dip detect at $0.98\\%$. " - "That floor is set by the protocol alone, not by halo mass. " - "Resolvable configurations detect at $34.6\\%$ instead. " - "The gap is what makes the catalog saddle-limited. " - "A fit on half the cells transfers to the remainder. " + "The calibration step corrects each galaxy's shear for shape noise. " + "Bright galaxies carry a multiplicative bias of $1.7\\%$. " + "That bias is set by the pixel scale alone, not by galaxy size. " + "Faint galaxies carry $4.2\\%$ instead. " + "The gap is what makes the faint sample noise-limited. " + "A fit on half the images transfers to the remainder. " "Its factors follow from the linearity of the estimator. " - "The criterion is therefore computable before detection is run." + "The correction is therefore computable before the shear is measured." ) NO_NUMBERS = ( - "The aperture-mass map is a filter-weighted mean of tangential " - "ellipticity. Peaks are detected on the processed signal-to-noise map " - "after an empirical null. The convergence itself is never observed. " - "This Letter asks what physical quantity the detection step selects on." + "The aperture-mass map is a weighted sum of tangential " + "ellipticity. Peaks are counted on the signal-to-noise map of each " + "field. The convergence itself is never observed directly. " + "This study asks how the peak counts depend on the smoothing scale." ) class TestNumeralPreservingReduction(unittest.TestCase): def test_inline_math_numerals_survive(self): - reduced = es.latex_to_numeral_text("We used $500$ configurations.") - self.assertIn("500", reduced) + reduced = es.latex_to_numeral_text("We used $200$ images.") + self.assertIn("200", reduced) def test_latex_to_plain_still_destroys_them(self): # The contrast is the reason this axis needs its own reduction; if # latex_to_plain ever kept numerals, the two projections would merge. - self.assertNotIn("500", es.latex_to_plain("We used $500$ configurations.")) + self.assertNotIn("200", es.latex_to_plain("We used $200$ images.")) def test_thousands_separator_is_one_numeral(self): - reduced = es.latex_to_numeral_text("The sample has $14{,}850{,}000$ rows.") + reduced = es.latex_to_numeral_text("The sample has $1{,}250{,}000$ rows.") self.assertEqual(len(salience.RE_NUMERAL.findall(reduced)), 1) def test_displayed_equations_are_dropped(self): @@ -98,7 +98,7 @@ class TestPercentileReading(unittest.TestCase): def test_value_on_a_tie_plateau_reads_the_top_of_the_tie(self): # 0.5 is shared by the 0.90-0.94 band. Reading the plateau's lower edge # would report a passage as exactly typical whenever it lands on a - # common value, which is what suppressed the Letter's abstract. + # common value, which is what suppressed the abstract of a manuscript under review. found = salience.percentile_of(self.REFERENCE, "max_recital_run_frac", 0.5) self.assertEqual(found, 0.94) @@ -267,6 +267,24 @@ def test_a_reference_built_at_another_unit_is_refused_and_named(self): self.assertEqual(status["status"], "degraded") self.assertIn("abstract (section)", status["reason"]) + def test_a_foreign_bucket_the_document_never_enters_does_not_degrade_it(self): + with tempfile.TemporaryDirectory(prefix="salience-") as raw: + profile = Path(raw) + graded_bank(profile) + salience.calibrate(profile) + path = profile / salience.BASELINE_FILENAME + baseline = json.loads(path.read_text("utf-8")) + baseline["method"] = dict(baseline["abstract"], unit="section") + path.write_text(json.dumps(baseline), encoding="utf-8") + abstract = "\\begin{abstract}\n" + RECITAL_HEAVY + "\n\\end{abstract}\n" + self.assertEqual(salience.salience_axis_status(profile, abstract)["status"], + "measured") + status = salience.salience_axis_status( + profile, abstract + "\\section{Methods}\n" + RECITAL_HEAVY + "\n") + self.assertEqual(status["status"], "degraded") + self.assertIn("method (section)", status["reason"]) + self.assertEqual(salience.salience_axis_status(profile)["status"], "degraded") + class TestLocalReference(unittest.TestCase): """The axis on a real locally-built reference, when one is present. @@ -299,10 +317,10 @@ def test_recital_heavy_abstract_is_flagged(self): # is the gate itself, not above it; a smoke check must not ride the # edge of the distribution it is checking. heavier = RECITAL_HEAVY.replace( - "The criterion transfers to unseen mass ratios. ", - "The criterion transfers to $3$ of $4$ unseen mass ratios. ").replace( - "The detection step selects on the depth of the inter-peak dip. ", - "The detection step selects on the depth of the inter-peak dip at $2$ pixels. ") + "The correction transfers to unseen seeing conditions. ", + "The correction transfers to $3$ of $4$ unseen seeing conditions. ").replace( + "The calibration step corrects each galaxy's shear for shape noise. ", + "The calibration step corrects each galaxy's shear above $2$ pixels. ") document = "\\begin{abstract}\n" + heavier + "\n\\end{abstract}\n" rules = {f["rule"] for f in salience.salience_findings(document, self.PROFILE)} self.assertIn("salience-recital:abstract", rules) diff --git a/tests/test_deai_structure.py b/tests/test_deai_structure.py index fe04fb8..3d4add5 100644 --- a/tests/test_deai_structure.py +++ b/tests/test_deai_structure.py @@ -11,30 +11,29 @@ ANTITHESIS_HEAVY = ( - "The error budget is measured rather than assumed, and every threshold " - "derives from injections rather than from a tuned cutoff. The detector " - "reports negatives instead of burying them, so the reader sees the " - "boundary the grid quantifies. Each claim is stated with its reason " - "attached, and the pipeline was frozen before the labeled sample was " - "consumed at scoring time." + "The error budget is measured rather than assumed, and every correction " + "is taken from simulations rather than from a tuned cutoff. The catalog " + "reports failed fits instead of dropping them, so the reader sees how " + "often the shape measurement breaks. Each claim is stated with its " + "reason attached, and the selection was fixed before the shear was " + "measured." ) REVERSAL_BEAT = ( - "One might expect that raising the minimum configuration support would " - "cheaply remove the residual noise detections from the shortlist. " - "It would not. The support distribution of injected structure overlaps " - "the noise support distribution across the calibrated range, so any cut " - "strong enough to matter removes real detections at the same rate and " - "leaves the purity of the final catalog unchanged." + "One might expect that doubling the exposure time would double the " + "number of usable source galaxies behind each cluster. " + "It would not. The added galaxies are faint and small, so most of them " + "fail the size cut applied before the shapes are measured, and the " + "source density rises by far less than the exposure time does." ) PLAIN_PROSE = ( - "The aperture-mass map is evaluated on a regular grid in catalog " - "coordinates. Each configuration pairs a filter scale with a truncation " - "radius, and the resulting family covers the range of subhalo sizes the " - "injection grid samples. Peaks are aggregated across configurations " - "before any ranking statistic is computed, which keeps the candidate " - "list independent of any single filter choice." + "The aperture-mass map is evaluated on a regular grid across each " + "field. Each smoothing scale is matched to the angular size of a typical " + "cluster at the median lens redshift, and the resulting maps are combined " + "after their noise levels are equalised. Peaks are counted on the " + "combined map before any cosmological model is compared with the counts, " + "which keeps the measurement independent of the model." ) @@ -71,7 +70,7 @@ def test_template_score_excludes_auxiliary(self): self.assertNotIn("antithesis-cluster", values["templates"]) def test_paper_as_agent_detected(self): - text = ("This Letter asks whether the filter response can separate the " + text = ("This study asks whether the filter response can separate the " "two populations at the depth of the survey.") values = structure.paragraph_structure(text) self.assertEqual(values["paper_agent_count"], 1) @@ -85,25 +84,25 @@ def test_paper_that_merely_presents_is_not_an_agent(self): self.assertEqual(values["paper_agent_count"], 0) def test_wh_cleft_detected(self): - text = ("What it can conclude is limited by the noise of the map. " + text = ("What matters here is the noise of the map. " "The remaining sentences describe the aperture mass filter.") values = structure.paragraph_structure(text) self.assertEqual(values["wh_cleft_count"], 1) self.assertIn("wh-cleft", values["auxiliary_templates"]) def test_a_plain_wh_question_word_is_not_a_cleft(self): - text = ("How the filter responds depends on the truncation radius, so " - "the radius is fixed before any peak is counted.") + text = ("How the filter responds depends on the smoothing scale, so " + "the scale is fixed before any peak is counted.") values = structure.paragraph_structure(text) self.assertEqual(values["wh_cleft_count"], 0) def test_the_gap_below_a_heading_does_not_decide_a_family(self): # One newline or two between the heading and the paragraph: the same # paragraph. Left in place the heading fused with the opener (`Methods - # What it can conclude is`) and the family went unreported. - paragraph = ("What it can conclude is limited by the noise of the map, " - "which the aperture mass filter carries into every peak count " - "and every configuration of the calibrated grid we adopt here.") + # What matters here is`) and the family went unreported. + paragraph = ("What matters here is the noise of the map, which the aperture " + "mass filter carries into every peak count and into every " + "scale of the smoothing family that we adopt in this section.") counts = [] for gap in ("\n", "\n\n"): findings = structure.structure_findings("\\section{Methods}" + gap + paragraph, None) @@ -113,21 +112,21 @@ def test_the_gap_below_a_heading_does_not_decide_a_family(self): def test_modifier_stack_detected(self): stacks = structure.modifier_stacks( - "We adopt a per-map empirical B-mode null for every configuration.") - self.assertEqual(stacks, ["per-map empirical B-mode null"]) + "We adopt a per-band smoothed star-galaxy classifier for every exposure.") + self.assertEqual(stacks, ["per-band smoothed star-galaxy classifier"]) stacks = structure.modifier_stacks( - "The non-compensated 500-configuration subfamily fails the test.") - self.assertEqual(stacks, ["non-compensated 500-configuration subfamily"]) + "The non-weighted 200-image stack fails the test.") + self.assertEqual(stacks, ["non-weighted 200-image stack"]) def test_ordinary_compound_noun_phrases_are_not_stacks(self): self.assertEqual(structure.modifier_stacks( "The weak-lensing mass map is smoothed, and the [math] peak is kept."), []) self.assertEqual(structure.modifier_stacks( - "We use 500 configurations of the aperture-mass filter."), []) + "We use 200 maps of the aperture-mass filter."), []) def test_advisor_families_stay_out_of_template_score(self): - text = ("This Letter asks whether a per-map empirical B-mode null holds. " - "What it can conclude is limited by the noise of the map.") + text = ("This study asks whether a per-band smoothed star-galaxy classifier " + "suffices. What matters here is the noise of the map.") values = structure.paragraph_structure(text) self.assertEqual(values["template_score"], 0) for family in ("paper-agent", "wh-cleft", "modifier-stack"): diff --git a/tests/test_deai_voice.py b/tests/test_deai_voice.py index 9583dee..a02657e 100644 --- a/tests/test_deai_voice.py +++ b/tests/test_deai_voice.py @@ -8,6 +8,8 @@ findings, the axis status and `--scores`. These tests pin that path to `unmeasured` with the runtime's reason, check that a present runtime still scores, and hold the `--field` resolution that once divided a path by None. +A bundle that reads `corpus_cos` also needs the embedder and the exemplar +centroid; without them the feature is a 0.0 placeholder and the axis says so. """ from __future__ import annotations @@ -15,6 +17,7 @@ import contextlib import io import tempfile +import types import unittest from pathlib import Path from unittest import mock @@ -43,6 +46,20 @@ def fake_bundle() -> dict: "model": "stub"} +@contextlib.contextmanager +def scoring_inputs(*, embedder: bool = True, centroid: bool = True): + """The runtime present, the vector and classifier stubbed; the embedder and + the exemplar centroid as the test states them.""" + with mock.patch.object(oracle, "model_runtime_available", return_value=(True, "")), \ + mock.patch.object(df, "embedder_available", return_value=embedder), \ + mock.patch.object(df, "corpus_centroid", + return_value=object() if centroid else None), \ + mock.patch.object(df, "features_vector", + return_value=[0.0] * len(df.FEATURE_NAMES)), \ + mock.patch.object(voice, "_positive_class_probability", return_value=0.25): + yield + + class RuntimeProbeTests(unittest.TestCase): def setUp(self): self.raw = tempfile.TemporaryDirectory(prefix="voice-") @@ -83,11 +100,7 @@ def test_scores_with_the_runtime_lists_each_paragraph_by_source_line(self): draft = Path(self.raw.name) / "draft.tex" draft.write_text(TEXT, encoding="utf-8") stdout = io.StringIO() - with mock.patch.object(oracle, "model_runtime_available", return_value=(True, "")), \ - mock.patch.object(df, "features_vector", - return_value=[0.0] * len(df.FEATURE_NAMES)), \ - mock.patch.object(voice, "_positive_class_probability", return_value=0.25), \ - contextlib.redirect_stdout(stdout): + with scoring_inputs(), contextlib.redirect_stdout(stdout): code = voice.main([str(draft), "--field", "fld", "--profile-root", str(self.root), "--scores"]) self.assertEqual(code, 0) @@ -97,10 +110,7 @@ def test_scores_with_the_runtime_lists_each_paragraph_by_source_line(self): def test_with_the_runtime_the_bundle_scores_and_flags(self): # The probe must not block a present runtime: the vector and the # classifier are stubbed, the rest of the path is real. - with mock.patch.object(oracle, "model_runtime_available", return_value=(True, "")), \ - mock.patch.object(df, "features_vector", - return_value=[0.0] * len(df.FEATURE_NAMES)), \ - mock.patch.object(voice, "_positive_class_probability", return_value=0.25): + with scoring_inputs(): self.assertEqual(voice.voice_score(TEXT, self.field), 0.25) self.assertEqual(voice.voice_axis_status(self.field)["status"], "measured") findings = voice.voice_findings(TEXT, self.field) @@ -109,6 +119,44 @@ def test_with_the_runtime_the_bundle_scores_and_flags(self): self.assertEqual({f["measurement_status"] for f in findings}, {"measured"}) +class CorpusCosineInputTests(unittest.TestCase): + """E20: `corpus_cos` without its inputs is a placeholder, not a score.""" + + def setUp(self): + self.raw = tempfile.TemporaryDirectory(prefix="voice-") + self.addCleanup(self.raw.cleanup) + self.field = Path(self.raw.name) + self.addCleanup(voice._MODEL_CACHE.pop, str(self.field), None) + voice._MODEL_CACHE[str(self.field)] = fake_bundle() + + def assert_unmeasured(self, fragment: str) -> None: + self.assertIsNone(voice.voice_score(TEXT, self.field)) + self.assertEqual(voice.voice_findings(TEXT, self.field), []) + status = voice.voice_axis_status(self.field) + self.assertEqual(status["status"], "unmeasured") + self.assertIn(fragment, status["reason"]) + + def test_without_the_exemplar_centroid_the_axis_is_unmeasured(self): + with scoring_inputs(centroid=False): + self.assert_unmeasured(f"exemplar_embeddings_{df.EMBED_MODEL}.npy is missing") + + def test_without_the_embedder_the_axis_is_unmeasured(self): + with scoring_inputs(embedder=False): + self.assert_unmeasured("sentence-transformers is not installed") + + def test_a_bundle_that_never_saw_a_cosine_needs_neither(self): + # Zero training variance in the column: the classifier cannot respond + # to it, so the placeholder changes nothing and the bundle scores. + variances = [1.0] * len(df.FEATURE_NAMES) + variances[df.FEATURE_NAMES.index("corpus_cos")] = 0.0 + bundle = dict(fake_bundle(), scaler=types.SimpleNamespace(var_=variances)) + voice._MODEL_CACHE[str(self.field)] = bundle + self.assertFalse(voice.reads_corpus_cos(bundle)) + with scoring_inputs(embedder=False, centroid=False): + self.assertEqual(voice.voice_score(TEXT, self.field), 0.25) + self.assertEqual(voice.voice_axis_status(self.field)["status"], "measured") + + class FieldResolutionTests(unittest.TestCase): """`--field` resolves through `cli_common.optional_field_dir`: none or several profiles is the exit 2 the tool always printed for a missing diff --git a/tests/test_eval_findings.py b/tests/test_eval_findings.py index a7bec45..90461d3 100644 --- a/tests/test_eval_findings.py +++ b/tests/test_eval_findings.py @@ -377,6 +377,22 @@ def test_leakage_without_a_lexicon_is_zero_not_a_crash(self) -> None: ef.leakage_paired("word " * 60, Path("no-such-profile-dir"), "x"), (0, 0)) + def test_the_paper_is_counted_as_the_bank_would_hold_it(self) -> None: + # One body paragraph holds `halo`; `halos` is another word, and the + # reference list is no prose the bank keeps. Matched by substring over + # raw blocks, all three were the paper's own `halo` passages: df 2 + 3 + # of 40,004 cleared the gate and the finding read as suppressed. + import json + with tempfile.TemporaryDirectory() as raw: + field = pathlib.Path(raw) + (field / ef.deai_register.LEXICON_FILENAME).write_text(json.dumps( + {"n_passages": 40000, "document_frequency": {"halo": 2}}), + encoding="utf-8") + text = ("\\section{Introduction}\n" + "halo mass " * 16 + "\n\n" + + "halos differ " * 16 + "\n\n\\section{References}\n" + + "halo catalog " * 16 + "\n") + self.assertEqual(ef.leakage_paired(text, field, "x"), (1, 1)) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_extractor_statistics.py b/tests/test_extractor_statistics.py index 0b81988..b7c69df 100644 --- a/tests/test_extractor_statistics.py +++ b/tests/test_extractor_statistics.py @@ -66,6 +66,12 @@ def test_placeholders_are_not_words_and_do_not_drop_a_paragraph(self): def test_words_are_unicode_letters(self): self.assertEqual(es.words("naïve Poincaré"), ["naïve", "Poincaré"]) + def test_a_compound_is_one_word(self): + # The Unicode class first shipped without `-` and `'`, so every + # compound counted twice in the bank's 30-400-word admission band. + self.assertEqual(es.words("N-body O'Brien's two-point naïve-looking 3-D"), + ["N-body", "O'Brien's", "two-point", "naïve-looking", "D"]) + def test_bank_rows_count_prose_words_and_carry_unique_ids(self): prose = " ".join(f"word{i}" for i in range(EXEMPLAR_MIN)) plain = f"[CITE] {prose}\n\n{prose}" @@ -106,6 +112,51 @@ def test_an_empty_topic_ranks_by_section_name_in_the_fallback(self): self.assertIn("results section", ranked[0][1]["text"]) +class EmbeddingCacheTests(unittest.TestCase): + """The `.npy` cache is keyed by the bank's content, not by mtime (audit D20).""" + + def test_a_same_size_rebuild_with_other_rows_is_re_encoded(self): + try: + import numpy as np + except ImportError: # CI runs without optional dependencies + self.skipTest("numpy is unavailable") + import os + import sys + import types + from unittest import mock + import retrieve_exemplars as rx + calls = [] + + class FakeModel: + def __init__(self, _name): + pass + + def encode(self, texts, **_options): + calls.append(list(texts)) + return np.array([[float(len(text)), 1.0] for text in texts]) + + fake = types.SimpleNamespace(SentenceTransformer=FakeModel) + with tempfile.TemporaryDirectory() as tmp, \ + mock.patch.dict(sys.modules, {"sentence_transformers": fake}): + profile = pathlib.Path(tmp) + bank = profile / "exemplar_paragraphs.jsonl" + + def write(texts): + bank.write_text("".join(json.dumps({"text": t}) + "\n" for t in texts), + encoding="utf-8") + return rx._load_records(bank) + + records = write(["aa", "bbbb"]) + rx._build_or_load_embeddings(records, bank, profile, "m") + rx._build_or_load_embeddings(records, bank, profile, "m") + self.assertEqual(len(calls), 1) + records = write(["bbbb", "aa"]) + os.utime(bank, (0, 0)) # older than the cache: the mtime rule kept it + embeddings, _model = rx._build_or_load_embeddings(records, bank, profile, "m") + self.assertEqual(len(calls), 2) + self.assertEqual(embeddings[:, 0].tolist(), [4.0, 2.0]) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_tex_macros.py b/tests/test_tex_macros.py index dea914e..498fe27 100644 --- a/tests/test_tex_macros.py +++ b/tests/test_tex_macros.py @@ -26,13 +26,13 @@ class ExpandNumericTests(unittest.TestCase): def test_a_use_of_a_numeric_macro_becomes_its_number(self): - text = r"\newcommand{\Nfields}{63}" "\n" r"We scored \Nfields{} fields." - self.assertIn("63 fields", tex_macros.expand_numeric(text)) + text = r"\newcommand{\Nsamples}{12}" "\n" r"We drew \Nsamples{} samples." + self.assertIn("12 samples", tex_macros.expand_numeric(text)) def test_the_definition_stops_contributing_its_digits(self): """The second half of the bug: an unexpanded definition leaked once.""" - out = tex_macros.expand_numeric(r"\newcommand{\Nf}{63}" "\nNo uses here.") - self.assertNotIn("63", out) + out = tex_macros.expand_numeric(r"\newcommand{\Ns}{12}" "\nNo uses here.") + self.assertNotIn("12", out) def test_a_symbolic_macro_is_left_alone(self): text = r"\newcommand{\Msun}{M_\odot}" "\n" r"a mass of \Msun today" @@ -47,16 +47,27 @@ def test_a_macro_taking_an_argument_is_left_alone(self): self.assertEqual(tex_macros.expand_numeric(text), text) def test_a_commented_out_definition_is_not_a_definition(self): - text = (r"% \newcommand{\Nf}{63}" "\n" r"\newcommand{\Nf}{7}" "\n" - r"We use \Nf{} fields.") - self.assertIn("We use 7 fields.", tex_macros.expand_numeric(text)) - alone = r"% \newcommand{\Nf}{63}" "\n" r"We use \Nf{} fields." + text = (r"% \newcommand{\Ns}{12}" "\n" r"\newcommand{\Ns}{7}" "\n" + r"We use \Ns{} samples.") + self.assertIn("We use 7 samples.", tex_macros.expand_numeric(text)) + alone = r"% \newcommand{\Ns}{12}" "\n" r"We use \Ns{} samples." self.assertEqual(tex_macros.expand_numeric(alone), alone) + def test_a_use_inside_a_comment_is_left_as_written(self): + text = r"\newcommand{\Ns}{7}" "\n" r"We use \Ns{} samples. % was \Ns before the cut" + self.assertEqual(tex_macros.expand_numeric(text).splitlines()[1], + r"We use 7 samples. % was \Ns before the cut") + + def test_a_definition_across_lines_moves_no_later_line(self): + text = "\\newcommand{\\Ns}{%\n 12}\nWe use \\Ns{} samples.\n" + out = tex_macros.expand_numeric(text) + self.assertEqual(out.count("\n"), text.count("\n")) + self.assertEqual(out.splitlines()[2], "We use 12 samples.") + def test_a_shorter_name_does_not_fire_inside_a_longer_one(self): - text = (r"\newcommand{\Nf}{7}" "\n" r"\newcommand{\Nfields}{63}" "\n" - r"\Nf and \Nfields") - self.assertIn("7 and 63", tex_macros.expand_numeric(text)) + text = (r"\newcommand{\Ns}{7}" "\n" r"\newcommand{\Nsamples}{12}" "\n" + r"\Ns and \Nsamples") + self.assertIn("7 and 12", tex_macros.expand_numeric(text)) def test_plain_tex_def_and_renewcommand_are_recognised(self): self.assertIn("5", tex_macros.expand_numeric(r"\def\Na{5}" "\n" r"\Na")) @@ -80,19 +91,19 @@ def _document(self, root_body: str, child_body: str) -> Path: def test_a_definition_in_the_root_reaches_a_use_in_an_included_file(self): root = self._document( - r"\newcommand{\Nfields}{63}" "\n" r"\begin{document}" + r"\newcommand{\Nsamples}{12}" "\n" r"\begin{document}" "\n" r"\input{child}" "\n" r"\end{document}", - r"We scored \Nfields{} fields in total.") - self.assertIn("63 fields", sec.read_tex_document(root)) + r"We drew \Nsamples{} samples in total.") + self.assertIn("12 samples", sec.read_tex_document(root)) def test_the_numeral_projection_now_sees_that_number(self): """The end the fix exists for: `latex_to_numeral_text` counts it.""" root = self._document( - r"\newcommand{\Nfields}{63}" "\n" r"\begin{document}" + r"\newcommand{\Nsamples}{12}" "\n" r"\begin{document}" "\n" r"\input{child}" "\n" r"\end{document}", - r"We scored \Nfields{} fields in total.") + r"We drew \Nsamples{} samples in total.") projected = sec.latex_to_numeral_text(sec.read_tex_document(root)) - self.assertIn("63", projected) + self.assertIn("12", projected) if __name__ == "__main__": diff --git a/tests/test_train_voice_model.py b/tests/test_train_voice_model.py index dbf228a..b053580 100644 --- a/tests/test_train_voice_model.py +++ b/tests/test_train_voice_model.py @@ -340,12 +340,15 @@ def test_undefined_f1_is_excluded_from_aggregation(self): self.assertEqual(aggregated["overall"]["f1_positive"]["mean"], 1.0) def test_voice_axis_requires_measured_operating_point(self): - # The status under test is the bundle's, so the surprisal runtime the - # feature vector needs is pinned as present: without it the axis is + # The status under test is the bundle's, so every input the feature + # vector needs is pinned as present -- the surprisal runtime, the + # embedder and the exemplar centroid: without one the axis is # `unmeasured` whatever the bundle says (tests/test_deai_voice.py). with tempfile.TemporaryDirectory() as temporary, \ mock.patch.object(voice.do, "model_runtime_available", - return_value=(True, "")): + return_value=(True, "")), \ + mock.patch.object(voice.df, "embedder_available", return_value=True), \ + mock.patch.object(voice.df, "corpus_centroid", return_value=object()): profile = Path(temporary) key = str(profile) try: diff --git a/tests/test_verify_references.py b/tests/test_verify_references.py index f9173bf..fce59d1 100644 --- a/tests/test_verify_references.py +++ b/tests/test_verify_references.py @@ -14,10 +14,13 @@ import http.client import io import json +import os import tempfile import unittest +import urllib.error import urllib.request from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -309,6 +312,57 @@ def urlopen(request, timeout=30): finally: urllib.request.urlopen = real + def serve(self, *answers): + """A urlopen playing `answers` in order: bytes are a 200 body, an int an + HTTP error, a (code, Retry-After) pair an error naming its wait.""" + seen = [] + + class Response(io.BytesIO): + status = 200 + + def urlopen(request, timeout=30): + seen.append(request) + answer = answers[len(seen) - 1] + if isinstance(answer, bytes): + return Response(answer) + code, wait = answer if isinstance(answer, tuple) else (answer, None) + headers = http.client.HTTPMessage() + if wait is not None: + headers["Retry-After"] = wait + raise urllib.error.HTTPError(request.full_url, code, "error", headers, None) + return seen, mock.patch.object(urllib.request, "urlopen", urlopen) + + def test_a_transient_answer_is_asked_once_more(self): + # F22: one 503 left the entry unmeasured and the axis degraded. + seen, patched = self.serve(503, b"{}") + with patched, mock.patch.object(vr.time, "sleep") as sleep: + self.assertEqual(vr.fetch(vr.CROSSREF + "10.1/x"), (200, b"{}")) + self.assertEqual(len(seen), 2) + sleep.assert_called_once_with(vr.RETRY_WAIT_DEFAULT) + + def test_a_second_rate_limit_is_an_outage_after_a_capped_wait(self): + seen, patched = self.serve((429, "120"), (429, "1")) + with patched, mock.patch.object(vr.time, "sleep") as sleep: + with self.assertRaises(vr.RegistryUnavailable): + vr.fetch(vr.CROSSREF + "10.1/x") + self.assertEqual(len(seen), 2) + sleep.assert_called_once_with(vr.RETRY_WAIT_CAP) + seen, patched = self.serve(400) + with patched, mock.patch.object(vr.time, "sleep") as sleep: + with self.assertRaises(vr.RegistryUnavailable): + vr.fetch(vr.CROSSREF + "10.1/x") + sleep.assert_not_called() + + def test_a_contact_address_rides_in_the_user_agent(self): + seen, patched = self.serve(b"{}", b"{}") + with patched: + with mock.patch.dict(os.environ, {vr.MAILTO_ENV: "me@example.org"}): + vr.fetch(vr.CROSSREF + "10.1/x") + with mock.patch.dict(os.environ, {vr.MAILTO_ENV: ""}): + vr.fetch(vr.CROSSREF + "10.1/x") + self.assertEqual([request.get_header("User-agent") for request in seen], + [f"{vr.USER_AGENT} (mailto:me@example.org)", vr.USER_AGENT]) + def test_an_unreadable_registry_answer_leaves_the_entry_unmeasured(self): fetch = self.registries(**{vr.CROSSREF + "10.1093": (200, b'{"no": "message"}'), vr.ARXIV: (200, b"not xml")}) diff --git a/tools/ai_ism_lint.py b/tools/ai_ism_lint.py index 18420d9..4f00819 100644 --- a/tools/ai_ism_lint.py +++ b/tools/ai_ism_lint.py @@ -64,6 +64,14 @@ r"utilize|utilized|leverage|importantly|interestingly|notably|" r"intricate|foster(?:s|ing|ed)?)\b" ) +# `robust` naming a method of robust statistics is that method's name, not the +# filler adjective the cap is for: a Methods section describing one robust +# estimator could never exit 0, and the skill forbids trading accuracy for the +# count. The list is closed; `robust estimate` stays counted, being as often "a +# reliable estimate" as an estimator's output. Matched on the whole source, so +# a line break inside the compound does not decide whether it counts. +TIER_B_TERM_OF_ART_PATTERN = re.compile( + r"(?i)\brobust(?=\s+(?:estimators?|estimation|statistics?|regression)\b)") STUBBORN_REPLACE_PATTERN = re.compile( r"(?i)\b(?:in order to|aim to|facilitate|serves as)\b") THREE_PARALLEL_PATTERN = re.compile( @@ -233,16 +241,20 @@ def lexical_findings(text: str, path: Path, """Return lexical findings and axis statuses. Tier B is measured everywhere but only occurrences beyond the per-section, - per-word cap become L0 targets. + per-word cap become L0 targets; `robust` inside a statistics term of art + (`TIER_B_TERM_OF_ART_PATTERN`) is not a Tier B occurrence. """ lines = text.splitlines() - scan_lines = es.blank_preserving(text, *ARGUMENT_SPAN_PATTERNS).splitlines() + scan_text = es.blank_preserving(text, *ARGUMENT_SPAN_PATTERNS) + scan_lines = scan_text.splitlines() + tier_b_lines = es.blank_preserving(scan_text, TIER_B_TERM_OF_ART_PATTERN).splitlines() ranges = section_ranges(text) firsts = _paragraph_first_prose_lines(text) findings: list[dict[str, Any]] = [] tier_b: dict[tuple[str, str], list[tuple[int, str]]] = defaultdict(list) - for line_no, (line, scan) in enumerate(zip(lines, scan_lines), start=1): + for line_no, (line, scan, tier_b_scan) in enumerate( + zip(lines, scan_lines, tier_b_lines), start=1): section = section_for_line(line_no, ranges) excerpt = line.strip() # Resolved before the Tier A/B scans because one paragraph-initial @@ -290,7 +302,7 @@ def lexical_findings(text: str, path: Path, message=f"Paragraph-initial connector {word!r} is an L0 target.", action="Remove the roadmap connector and let the argument carry the transition.", )) - for match in TIER_B_PATTERN.finditer(scan): + for match in TIER_B_PATTERN.finditer(tier_b_scan): if connector and match.start() < connector.end(): continue word = match.group(0).lower() diff --git a/tools/deai_anchoring.py b/tools/deai_anchoring.py index db6acbf..a150c45 100644 --- a/tools/deai_anchoring.py +++ b/tools/deai_anchoring.py @@ -66,26 +66,22 @@ r"\\begin\{thebibliography\}.*?\\end\{thebibliography\}" r"|\\bibliography(?:style)?\s*\{[^}]*\}", re.DOTALL) -# section-title keyword -> class; first match wins, unmatched -> "other" -SECTION_CLASSES = ( - ("introduction", "intro"), - ("conclusion", "conclusions"), ("summary", "conclusions"), - ("discussion", "discussion"), - ("result", "results"), ("measurement", "results"), - ("constraint", "results"), - ("data", "methods"), ("method", "methods"), ("model", "methods"), - ("simulat", "methods"), ("analysis", "methods"), ("observ", "methods"), - ("appendix", "other"), ("acknowledg", "other"), -) +# The shared bucket (`extract_sections.classify_section`, inherited by a +# subsection as the corpus inherits it) folded onto this axis's coarser +# classes; any other bucket is "other". Until 2026-09-27 the module matched +# titles against a third keyword table, which disagreed with the curated one +# where the curation is deliberate: `Shear measurement`, which that table +# leaves `unknown` because the word names no role here, read as results; +# `Comparison with simulations` read as methods; and a topic-titled subsection +# under Methods read as other. +CLASS_OF_BUCKET = {"intro": "intro", "method": "methods", "data": "methods", + "results": "results", "discussion": "discussion", + "conclusion": "conclusions"} STRONG_CLASSES = {"methods", "results"} # unanchored claims here matter most -def classify_section(label: str) -> str: - lowered = label.lower() - for keyword, cls in SECTION_CLASSES: - if keyword in lowered: - return cls - return "other" +def section_class(bucket: str) -> str: + return CLASS_OF_BUCKET.get(bucket, "other") def sentence_is_anchored(source_sentence: str) -> bool: @@ -125,8 +121,7 @@ def document_anchoring(text: str) -> dict[str, Any]: section glued to its final sentence and anchored it with every year in the reference list (a Conclusions rate of 0.0 read as 0.2). An anchoring baseline built before the change must be rebuilt: its rates were measured - on the raw text. The class still comes from the raw section title, - through this module's own keyword table. + on the raw text. The class is the unit's shared bucket (`section_class`). """ lines = text.splitlines() per_class: dict[str, list[float]] = {} @@ -141,7 +136,7 @@ def document_anchoring(text: str) -> dict[str, Any]: if measured is None: continue rate, n_sentences = measured - cls = classify_section(label) + cls = section_class(bucket) per_class.setdefault(cls, []).append(rate) section_rows.append({"label": label, "class": cls, "line": start, "rate": rate, "n_sentences": n_sentences}) diff --git a/tools/deai_collocation.py b/tools/deai_collocation.py index 7229105..ad58d8f 100644 --- a/tools/deai_collocation.py +++ b/tools/deai_collocation.py @@ -11,9 +11,9 @@ because a writer who has read the field reaches for the field's own phrases. A machine draft assembles field words into combinations the field does not use -- -`calibrated blur`, `controlled grid`, `physical cells`, `detector axes` -- and -the advisor who read such a draft wrote "I don't know what this means" at -exactly those places. This axis finds them. +`binned kernel`, `smoothed boundary`, say -- and the advisor who read such a +draft wrote "I don't know what this means" at exactly such places. This axis +finds them. The unit is the SENTENCE. A single novel pair is ordinary (a human sentence carries one in four judged pairs at the median); a sentence whose novel @@ -196,13 +196,14 @@ def collocation_axis_status(field_profile_dir: Path | None, reason=f"{missing} is unavailable", detector="deai_collocation") usable = reference.usable_buckets(baseline, FEATURE, ADVISORY_PERCENTILE, high=True, unit=UNIT) + spans = None if text is None else reference.units(text) reasons = [None if usable else (f"no section bucket reaches the {MIN_REFERENCE_N}-sentence " "reference floor with spread above the gate"), - reference.unit_reason(baseline, UNIT)] + reference.unit_reason(baseline, UNIT, spans)] evidence = None if text is not None: - reasons.append(reference.unbucketed_reason(reference.units(text), "paragraph")) + reasons.append(reference.unbucketed_reason(spans, "paragraph")) whole = document_novelty(text, bank) if whole[FEATURE] is not None: evidence = (f"document novel-pair fraction {whole[FEATURE]:.3f} over " @@ -308,7 +309,8 @@ def collocation_findings(text: str, field_profile_dir: Path | None, # parenthesis opening right after it: `the shear threshold, which we call the # floor` and `the shear threshold (hereafter ...)` gloss the pair, while a # `(Fig. 1)` or a `, the` anywhere else in the sentence does not -- read over -# the whole sentence, those marked one pair in six as defined on the Letter. +# the whole sentence, those marked one pair in six as defined in a manuscript +# under review. RE_GLOSS_CUE = re.compile( r"\b(?:we call|called|which we call|define[sd]? as|defined|denote[sd]?|" r"that is|which is|i\.e\.|namely|meaning|hereafter)\b", diff --git a/tools/deai_discourse.py b/tools/deai_discourse.py index 07a3038..a1249f7 100644 --- a/tools/deai_discourse.py +++ b/tools/deai_discourse.py @@ -254,13 +254,14 @@ def discourse_axis_status(field_profile_dir: Path | None, detector="deai_discourse")) continue live = live_buckets(feature, field_profile_dir) + spans = None if text is None else axis["spans"](text) reasons = [None if live else (f"no calibrated {axis['unit']} bucket both reaches the " f"{MIN_REFERENCE_N}-unit floor and has spread below the " "advisory gate"), - reference.unit_reason(baseline, axis["unit"])] - if text is not None: - reasons.append(reference.unbucketed_reason(axis["spans"](text), axis["unit"])) + reference.unit_reason(baseline, axis["unit"], spans, axis["buckets"])] + if spans is not None: + reasons.append(reference.unbucketed_reason(spans, axis["unit"])) if any(reasons): statuses.append(feedback.axis_status( name, "degraded", reason="; ".join(part for part in reasons if part), diff --git a/tools/deai_docshape.py b/tools/deai_docshape.py index f74ddd7..c7d0aa5 100644 --- a/tools/deai_docshape.py +++ b/tools/deai_docshape.py @@ -662,7 +662,9 @@ def manifold_row(record: dict) -> dict[str, float]: manifold = fit_dispersion_manifold([manifold_row(r) for r in train_records]) baseline["dispersion_manifold"] = manifold - lengths = sorted(record["n_paragraphs"] for record in records) + # Edges from the training split only: fit on every document they also read + # the calibration papers, which the exchangeability behind p breaks (B17). + lengths = sorted(record["n_paragraphs"] for record in train_records) strata_edges = [ _quantile([float(v) for v in lengths], k / CONFORMAL_STRATA) for k in range(1, CONFORMAL_STRATA) diff --git a/tools/deai_docstructure.py b/tools/deai_docstructure.py index 68c7454..71d24b5 100644 --- a/tools/deai_docstructure.py +++ b/tools/deai_docstructure.py @@ -7,6 +7,7 @@ from __future__ import annotations +import math import sys from pathlib import Path from typing import Any @@ -44,6 +45,27 @@ load_baseline = reference.baseline_loader(BASELINE_NAME) +def tail_floor(probability: float) -> int: + """Reference documents an in-sample quantile at `probability` needs. + + Below 1/min(p, 1-p) documents no reference value lies beyond the tail: the + threshold is interpolated between the extremes and its false-flag rate is + not the nominal one. The calibration minimum is 3 documents, and with no + manifold every per-feature band finding was `strong` on such a baseline + (audit B18). `round` absorbs 1/(1-0.9) = 10.000000000000002. + """ + return math.ceil(round(1.0 / min(probability, 1.0 - probability), 9)) + + +def baseline_tail_floor(baseline: dict[str, Any]) -> int: + """The strictest `tail_floor` over the in-sample percentiles `baseline` flags at.""" + return max(tail_floor(float(p)) for p in ( + baseline.get("strong_percentile", 0.95), + baseline.get("dispersion_low_percentile", DISPERSION_LOW_PERCENTILE), + baseline.get("dispersion_high_percentile", DISPERSION_HIGH_PERCENTILE), + (baseline.get("role_coupling") or {}).get("low_percentile", ROLE_LOW_PERCENTILE))) + + def docstructure_axis_status(text: str, field_profile_dir: Path | None ) -> dict[str, Any]: shape = document_shape(text) @@ -51,12 +73,21 @@ def docstructure_axis_status(text: str, field_profile_dir: Path | None return feedback.axis_status("L2.document_structure", "unmeasured", reason=shape["reason"], detector="deai_docstructure") - if load_baseline(field_profile_dir) is None: + baseline = load_baseline(field_profile_dir) + if baseline is None: return feedback.axis_status( "L2.document_structure", "unmeasured", reason="docstructure_baseline.json is unavailable", detector="deai_docstructure", ) + n_documents, floor = int(baseline.get("n_documents", 0)), baseline_tail_floor(baseline) + if n_documents < floor: + return feedback.axis_status( + "L2.document_structure", "degraded", + reason=(f"the reference holds {n_documents} complete documents and a " + f"tail percentile needs {floor}; its in-sample findings are " + "ordinary context"), + detector="deai_docstructure") return feedback.axis_status("L2.document_structure", "measured", detector="deai_docstructure") @@ -69,6 +100,8 @@ def document_findings(text: str, field_profile_dir: Path | None, return [] findings = [] section_label = "(document)" + # In-sample tails the reference cannot resolve stay ordinary and degraded. + n_documents = int(baseline.get("n_documents", 0)) for metric_name in METRIC_NAMES: observed = shape["metrics"].get(metric_name) reference = baseline.get("metrics", {}).get(metric_name, {}) @@ -91,11 +124,13 @@ def document_findings(text: str, field_profile_dir: Path | None, if value <= threshold: continue percentile = _percentile(values, value) + resolved = n_documents >= tail_floor(operating_point) findings.append(feedback.make_finding( kind="advisory", layer="L2", rule=f"document-shape:{metric_name}", scope="document", calibration_unit="document", line=1, section=section_label, path=path, - detector="deai_docstructure", measurement_status="measured", - strength="strong", strong_advisory=True, + detector="deai_docstructure", + measurement_status="measured" if resolved else "degraded", + strength="strong" if resolved else "ordinary", observed={"value": observed, "empirical_percentile": percentile, "n_sections": shape["n_sections"], "n_paragraphs": shape["n_paragraphs"]}, @@ -131,10 +166,16 @@ def document_findings(text: str, field_profile_dir: Path | None, flat_row = {name: dispersion[name][DISPERSION_STAT] for name in dispersion if dispersion.get(name, {}).get(DISPERSION_STAT) is not None} - per_feature_strength = "ordinary" + per_feature_strength, per_feature_status = "ordinary", "measured" conformal = baseline.get("conformal") if not manifold: - per_feature_strength = "strong" + band_floor = max(tail_floor(float(baseline.get(key, default))) for key, default in ( + ("dispersion_low_percentile", DISPERSION_LOW_PERCENTILE), + ("dispersion_high_percentile", DISPERSION_HIGH_PERCENTILE))) + if n_documents >= band_floor: + per_feature_strength = "strong" + else: + per_feature_status = "degraded" else: operating = manifold_operating_point(baseline, flat_row, shape["n_paragraphs"]) @@ -229,7 +270,7 @@ def document_findings(text: str, field_profile_dir: Path | None, if (role_axis and role_axis.get("scoring_factors") != list(ROLE_SCORING_FACTORS)): role_axis = None - flagged = False + flagged, role_resolved = False, True if role["score"] is not None: score = float(role["score"]) if role_axis and role_axis.get("calibration"): @@ -259,6 +300,8 @@ def document_findings(text: str, field_profile_dir: Path | None, op_observed_extra = {"conformal_p": p_value} elif low_threshold is not None and role_values: flagged = score < float(low_threshold) + role_resolved = len(role_values) >= tail_floor(float( + role_reference.get("low_percentile") or ROLE_LOW_PERCENTILE)) percentile = _percentile(role_values, score) op_reference = {"operating_point": "in-sample percentile", "n_documents": len(role_values), @@ -292,7 +335,8 @@ def document_findings(text: str, field_profile_dir: Path | None, detector="deai_docstructure", detector_version="sci-paper.docstructure-baseline.v2", calibration_asset=BASELINE_NAME, - measurement_status="measured", strength="strong", + measurement_status="measured" if role_resolved else "degraded", + strength="strong" if role_resolved else "ordinary", observed={"role_coupling_z": role["score"], "factors": role["factors"], "n_sections": shape["n_sections"], @@ -348,7 +392,7 @@ def document_findings(text: str, field_profile_dir: Path | None, detector="deai_docstructure", detector_version="sci-paper.docstructure-baseline.v2", calibration_asset=BASELINE_NAME, - measurement_status="measured", strength=per_feature_strength, + measurement_status=per_feature_status, strength=per_feature_strength, observed={"dispersion_std": observed, "empirical_percentile": percentile, "band_tail": tail, diff --git a/tools/deai_features.py b/tools/deai_features.py index b9de994..c2c61ed 100644 --- a/tools/deai_features.py +++ b/tools/deai_features.py @@ -38,17 +38,19 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first import extract_style as es # noqa: E402 resolves only after the sys.path insert import deai_oracle as do # noqa: E402 same reason +import deai_reference as reference # noqa: E402 same reason; the shared paragraph sweep from deai_metrics import CONNECTIVE_OPENERS # noqa: E402 same reason EMBED_MODEL = "all-MiniLM-L6-v2" FEATURE_SCHEMA_VERSION = "sci-paper.voice-features.v1" -_MATH_MARKER_RE = re.compile( +# Every math marker, counted for a density; `deai_docshape._MATH_MARKER_RE`, +# which shared this name until 2026-09-27, answers only "is there math here". +_MATH_DENSITY_RE = re.compile( r"\[MATH\]|\[math\]|\$[^$]+\$|\\\(.+?\\\)|" r"\\begin\{(?:equation|align|gather|eqnarray|displaymath|multline)\*?\}", re.DOTALL, ) -_PLACEHOLDER_RE = re.compile(r"\[(?:MATH|math|CITE|FIGURE-OR-TABLE)\]") EQUIVOCAL = frozenset({ "but", "however", "although", "though", "whereas", "yet", @@ -143,7 +145,7 @@ def distributional_features(plain: str) -> dict: def _prose_words(text: str) -> list[str]: - plain = _PLACEHOLDER_RE.sub(" ", es.latex_to_plain(text)) + plain = es.without_placeholders(es.latex_to_plain(text)) return [word.lower() for word in es.words(plain)] @@ -158,7 +160,7 @@ def math_marker_density(text: str) -> float: n_words = len(_prose_words(text)) if n_words == 0: return 0.0 - return 100.0 * len(_MATH_MARKER_RE.findall(text)) / n_words + return 100.0 * len(_MATH_DENSITY_RE.findall(text)) / n_words def lexicon_density(text: str, lexicon: set[str] | frozenset[str]) -> float: @@ -198,11 +200,14 @@ def paragraph_features( feats.update({"mean_surprisal": 0.0, "global_uid": 0.0, "local_uid": 0.0}) else: feats.update({k: uid[k] for k in ("mean_surprisal", "global_uid", "local_uid")}) - # semantic distance to corpus centroid + # Semantic distance to the corpus centroid. Without the embedder or the + # centroid it is the 0.0 a training row without them carries: a placeholder, + # not a measurement, which `deai_voice.scoring_reason` refuses to score a + # bundle on when that bundle learned from real cosines. cos = 0.0 if centroid is None and field_profile_dir is not None: centroid = corpus_centroid(field_profile_dir) - if centroid is not None: + if centroid is not None and embedder_available(): import numpy as np e = _embedder().encode([plain], normalize_embeddings=True)[0] cos = float(np.dot(e, centroid)) @@ -417,13 +422,17 @@ def main(argv: list[str] | None = None) -> int: field_dir = cli_common.optional_field_dir(args, tool="deai_features") text = args.file.read_text(encoding="utf-8", errors="replace") centroid = corpus_centroid(field_dir) if field_dir else None - # split into blank-line paragraphs and dump features - blocks = [b for b in re.split(r"\n\s*\n", text) if b.strip()] + if centroid is None or not embedder_available(): + print("[deai_features] corpus_cos is the 0.0 placeholder: no " + f"exemplar_embeddings_{EMBED_MODEL}.npy or no sentence-transformers", + file=sys.stderr) + # The sweep every axis reads, keyed by source lines, so a row can be matched + # to a finding; a blank-line split numbered paragraphs no finding used. print("feature order:", ", ".join(FEATURE_NAMES)) - for i, b in enumerate(blocks): - v = features_vector(b, field_profile_dir=field_dir, model_name=args.model, + for start, end, _label, _bucket, block in reference.paragraphs(text): + v = features_vector(block, field_profile_dir=field_dir, model_name=args.model, centroid=centroid) - print(f"para[{i}] " + " ".join(f"{x:.2f}" for x in v)) + print(f"L{start}-{end} " + " ".join(f"{x:.2f}" for x in v)) return 0 diff --git a/tools/deai_metrics.py b/tools/deai_metrics.py index d8442a1..459313f 100644 --- a/tools/deai_metrics.py +++ b/tools/deai_metrics.py @@ -43,6 +43,13 @@ # Compatibility defaults used only when a profile has no explicit # deai_policy.json. Such findings are marked degraded and never strong. +# The burstiness ratio divides ONE section's sentence-length CV by the bucket +# CV pooled across papers, which also carries between-paper variance, so the +# ratio runs low on human prose too: its human median sits well below 1 +# (EVALUATION §16). The bias is left in, not calibrated away per (paper, +# bucket): §16 refuted the statistic as a detector on other grounds -- it +# reverses sign on adversarial prose -- so deai_policy.json is never written +# and these findings stay ordinary advisories (audit B16). DEFAULT_BURSTINESS_RATIO = 0.60 DEFAULT_SIGNPOST_FRAC = 0.20 MIN_SENTENCES = 5 diff --git a/tools/deai_reference.py b/tools/deai_reference.py index 0cabc71..b03dbf3 100644 --- a/tools/deai_reference.py +++ b/tools/deai_reference.py @@ -149,7 +149,8 @@ def unit_matches(reference: dict[str, Any], unit: str) -> bool: return reference.get("unit") == unit -def foreign_units(baseline: dict[str, Any] | None, unit: str) -> dict[str, str]: +def foreign_units(baseline: dict[str, Any] | None, unit: str, + buckets: Iterable[str] | None = None) -> dict[str, str]: """bucket -> the unit its artifact records, for every bucket NOT built at `unit`. `calibrate` has written `unit` into every bucket since the module existed @@ -157,16 +158,31 @@ def foreign_units(baseline: dict[str, Any] | None, unit: str) -> dict[str, str]: reference rebuilt at paragraph unit, say -- was read against section measurements without a word in the output. The invariant this module is for was a declaration until the reader checked it. An artifact with no - `unit` key predates the key and is reported as `unrecorded`. + `unit` key predates the key and is reported as `unrecorded`. `buckets`, + when given, limits the answer to those buckets. """ + scope = None if buckets is None else set(buckets) return {bucket: str(reference.get("unit") or "unrecorded") for bucket, reference in (baseline or {}).items() - if isinstance(reference, dict) and not unit_matches(reference, unit)} + if isinstance(reference, dict) and not unit_matches(reference, unit) + and (scope is None or bucket in scope)} -def unit_reason(baseline: dict[str, Any] | None, unit: str) -> str | None: - """The status text for buckets an axis refuses because of their unit, or None.""" - foreign = foreign_units(baseline, unit) +def unit_reason(baseline: dict[str, Any] | None, unit: str, + spans: Iterable[tuple[int, int, str, Any]] | None = None, + allowed: Iterable[str] | None = None) -> str | None: + """The status text for buckets an axis refuses because of their unit, or None. + + Only buckets the axis would read count: those in `allowed` (None: every + bucket) and, given the document's `spans` (as `unbucketed_reason` takes + them), those the document reaches. Counted over the whole profile, one + stale `method` bucket marked an abstract-only document `degraded` and sent + its reader to recalibrate, though the axis had read every unit it held. + """ + scope = set(baseline or {}) if allowed is None else set(allowed) + if spans is not None: + scope &= {bucket for _start, _end, bucket, _block in spans} + foreign = foreign_units(baseline, unit, scope) if not foreign: return None listed = ", ".join(f"{bucket} ({recorded})" for bucket, recorded in sorted(foreign.items())) @@ -300,7 +316,7 @@ def has_prose(block: str) -> bool: The corpus side drops both before it splits paragraphs, so a unit made of them here was measured against a reference that holds none: GPT-2 read forty-eight spaces as more tokens than the UID minimum and scored four of - the Letter's heading lines as paragraphs of near-zero surprisal variance. + a manuscript's heading lines as paragraphs of near-zero surprisal variance. A display-equation-only paragraph projects to `[MATH]` and stays a unit here, although the corpus side writes no row for it (the bank writer drops a placeholder-only paragraph and anything under 30 words): the axes with diff --git a/tools/deai_register.py b/tools/deai_register.py index cac28be..22e4b93 100644 --- a/tools/deai_register.py +++ b/tools/deai_register.py @@ -43,7 +43,7 @@ * **Subscripts.** `\\newcommand{\\Kraw}{S_\\mathrm{raw}}` renders a subscript, not a word. Reading macro bodies without checking for a preceding `_` or `^` turns every symbol decoration into a fake foreign term. -* **Possessives.** `sub-halo's` and `sub-halo` are one term. +* **Possessives.** `point-source's` and `point-source` are one term. Macro definitions are read as well as prose, because a term bound to a macro is by construction one the author uses repeatedly, and the reduction that feeds @@ -91,7 +91,7 @@ # projection replaces every math span with `[math]` before counting words. # Detection ran `latex_to_plain` one LINE at a time, so a `$...$` or an # `equation` environment that crossed a line boundary was invisible to it and -# the macro names inside surfaced as words with df 0 (`\dsep`, `\rmain`). The +# the macro names inside surfaced as words with df 0. The # third instance of the calibration/detection projection asymmetry, after the # preamble (§17.4) and citation commands (§18.1); this time the seam is the line. RE_MATH_SPAN = re.compile( @@ -100,7 +100,7 @@ r"|\\\[.*?\\\]|\$\$.*?\$\$|\$[^$]+\$|\\\(.*?\\\)", re.DOTALL) # Float placement options, bibliography commands and code spans render no # prose either; each leaked one token as a term (`htb`, `aasjournalv7`, -# `cw_mscale`). The float pattern keeps its `\begin{...}` so the environment +# a `\texttt` identifier). The float pattern keeps its `\begin{...}` so the environment # stripper downstream still sees it. RE_FLOAT_OPTION = re.compile(r"(\\begin\{[A-Za-z*]+\})\[[^\]\n]*\]") # Length settings and table-note commands outside a float render no prose. @@ -109,7 +109,7 @@ # `\begin{table}...\end{table}` with a placeholder in one pass over the # passage, detection projected one LINE at a time, so a `tabular*` column # specification and a caption's words counted on this side only -- -# `filllll`, `crimson`, `isosurfaces` at df 0, +# `filllll` and caption words at df 0, # 14 of 90 zero-hit terms on one manuscript (2026-09-04). RE_LENGTH_CMD = re.compile( r"\\(?:setlength|addtolength|tabletypesize|tablewidth|tablenotemark" @@ -129,7 +129,7 @@ r"|(?:is|are) defined as|defined as|denoted?|which we call|referred to as)\b", re.I) # A derived form of a native word is reported against its stem rather than as -# foreign: the field writes `resolvable`, the manuscript `resolvability`. +# foreign: a manuscript's `shearing` is judged by the field's `shear`. # Suffix stripping, longest first, with the -e / -y / doubled-consonant # restorations; deliberately small, because a stemmer that reaches too far # turns a genuinely foreign word into a false native. @@ -408,7 +408,7 @@ def corpus_document_frequency(term: str, table: dict[str, Any]) -> tuple[int, st string itself would call `aperture-mass` foreign to weak lensing. """ parts = [part for part in term.split("-") if len(part) >= 3] - # `no-dip` has one part long enough to carry meaning; judging the whole + # `no-mass` has one part long enough to carry meaning; judging the whole # string instead called every `no-X` / `in-X` compound absent at floor 1. if parts and "-" in term: rarest = min(parts, key=lambda part: int(table.get(part, 0))) @@ -632,6 +632,24 @@ def register_findings(text: str, field_profile_dir: Path | None, return findings +def passage_terms(passage: str) -> set[str]: + """The terms one bank passage adds to the document frequency. + + A normalized word of three or more characters, and each such part of a + hyphenated one, so a compound can be judged by parts the corpus writes as + separate words. One owner for the lexicon and for the paper side of + `eval_findings.leakage_paired`, which matched raw substrings (`halo` in + `halos`) over blocks that included the bibliography. + """ + seen: set[str] = set() + for word in RE_WORD.findall(es.latex_to_plain(passage)): + key = normalize(word) + if len(key) >= 3: + seen.add(key) + seen.update(part for part in key.split("-") if len(part) >= 3) + return seen + + def calibrate(field_profile_dir: Path) -> dict[str, Any] | None: """Count per-term document frequency across the field's own passage banks. @@ -655,18 +673,7 @@ def calibrate(field_profile_dir: Path) -> dict[str, Any] | None: if not passage.strip(): continue n_passages += 1 - seen: set[str] = set() - for word in RE_WORD.findall(es.latex_to_plain(passage)): - key = normalize(word) - if len(key) < 3: - continue - seen.add(key) - # Index the parts too, so a compound in the manuscript can - # be judged by parts the corpus writes as separate words. - for part in key.split("-"): - if len(part) >= 3: - seen.add(part) - document_frequency.update(seen) + document_frequency.update(passage_terms(passage)) if not n_passages: return None diff --git a/tools/deai_residue.py b/tools/deai_residue.py index 4bbbfa2..b1a2a28 100644 --- a/tools/deai_residue.py +++ b/tools/deai_residue.py @@ -14,7 +14,7 @@ 1.7 times per 5,000-word paper on prose nobody would change. The first tier is strong; the second is an ordinary advisory. - `residue-negative-label` -- a heading or caption defined by what it is NOT - (`without the saddle correction`, `non-compensated`), where the negated + (`without the dust correction`, `non-weighted`), where the negated thing appears nowhere else in the paper. A label like that was written against a variant the loop later deleted; the reader meets a comparison with one side missing. @@ -22,9 +22,10 @@ `\\textcolor{red}`, `in the revised version`. Comments are blanked before the scan, so a note the author keeps for themselves is not a finding. - `residue-absence` -- a sentence that defines the paper's own object by what - it never does or has (`the head never participates in the decision`, `the - reference carries no quoted number`): the prose form of the negative label, - a menu line reading "no braised pork". The reader meets an absence in place + it never does or has (`the photometric catalog never enters the shape + measurement`, `the comparison sample carries no redshift estimate`): the + prose form of the negative label, a menu line reading "no braised pork". + The reader meets an absence in place of the thing; the fix says what the object does. Two tiers, set on the held-out refereed corpus (442 files, 1.90 million prose words): `never` and the `nothing is` / `none sees` / `no ... is applied` forms occur 0.008 @@ -95,7 +96,7 @@ # Rule 4. The absence families (see the module docstring for the corpus # rates behind the tiers). `no ... is applied` is a template: `no` and an # object of one to three words, then a passive of the listed verbs. A -# hyphenated compound (`never-touched controls`) is a name, not a predicate, +# hyphenated compound (`never-masked tiles`) is a name, not a predicate, # and is excluded from the `never` match. ABSENCE_STRONG = ("never", "nothing is", "nothing was", "none sees", "none enters", "no ... is applied") @@ -382,7 +383,7 @@ def paper_prose(text: str) -> str: Labels are read from the whole text, appendix captions included, so the object has to be looked for in the whole paper too: with the vocabulary projection (which drops `skip` for the corpus's sake) an appendix figure - "without the saddle correction" was reported absent from the body while + "without the dust correction" was reported absent from the body while the appendix explained the correction three lines above it. The preamble and the bibliography stay out -- neither is prose the reader meets.""" return register.body_only(text, drop_skip=False) @@ -420,7 +421,7 @@ def negative_label_added_findings(before: str, after: str, """Rule 5: a negation new in this edit whose WHOLE object was in the body before and is not now. - Whole object, not any stem: `Without the saddle correction` added while + Whole object, not any stem: `Without the dust correction` added while `correction` alone left the body is a label about something the paper never had, and reporting it as a patched absence charged the edit with a removal it did not make. A negation the old labels already carried is not diff --git a/tools/deai_salience.py b/tools/deai_salience.py index 4d26565..47ef890 100644 --- a/tools/deai_salience.py +++ b/tools/deai_salience.py @@ -120,12 +120,13 @@ def salience_axis_status(field_profile_dir: Path | None, detector="deai_salience", ) live = live_buckets(baseline) + spans = None if text is None else _units(text) reasons = [None if live else (f"no calibrated {UNIT} bucket both reaches the {MIN_REFERENCE_N}-unit " "floor and has spread above the advisory gate"), - reference.unit_reason(baseline, UNIT)] - if text is not None: - reasons.append(reference.unbucketed_reason(_units(text), UNIT)) + reference.unit_reason(baseline, UNIT, spans)] + if spans is not None: + reasons.append(reference.unbucketed_reason(spans, UNIT)) if any(reasons): return feedback.axis_status( "L2.salience_hierarchy", "degraded", diff --git a/tools/deai_structure.py b/tools/deai_structure.py index 619c213..3edf9d6 100644 --- a/tools/deai_structure.py +++ b/tools/deai_structure.py @@ -61,11 +61,11 @@ # constructions an advisor marked as reading like a prompt or a machine, each # abstracted to its form; no sentence from that round is quoted anywhere in # this repository. Auxiliary like the two above: never in `template_score`. -# paper-agent the paper as a thinking agent ("this Letter asks whether") +# paper-agent the paper as a thinking agent ("this study asks whether") # wh-cleft a wh-clause as subject with the point in the predicate -# ("what it can conclude is limited by") +# ("what matters here is") # modifier-stack a noun buried under a run of pre-modifiers, at least two of -# them hyphenated or numeric ("per-map empirical B-mode null") +# them hyphenated or numeric ("per-band smoothed star-galaxy classifier") RE_PAPER_AGENT = re.compile( r"\b(?:this|the present|our)\s+(?:paper|letter|work|study|article|analysis|" r"note|manuscript)\s+(?:asks?|answers?|argues?|wonders?|explores?|seeks?|" @@ -97,7 +97,7 @@ def modifier_stacks(sentence: str) -> list[str]: A run of content tokens is cut at the head noun, the token after the last compound, because without a parser that is the only place a phrase can be - seen to end: `non-compensated 500-configuration subfamily fails` reports + seen to end: `non-weighted 200-image stack fails` reports the phrase, not the verb. `[math]`/`[CITE]` placeholders and every punctuation mark break a run, so a list of quantities is not a phrase. """ @@ -263,7 +263,7 @@ def structure_findings(text: str, field_profile_dir: Path | None, findings: list[dict[str, Any]] = [] # The shared paragraph sweep blanks headings before splitting: left in # place, a heading one newline above a wh-cleft opener fused with it - # (`Methods What it can conclude is`) and the family was missed while the + # (`Methods What matters here is`) and the family was missed while the # same paragraph two newlines below the heading was reported. for paragraph_start, paragraph_end, raw_label, bucket, block in reference.paragraphs(text): if len(es.words(es.latex_to_plain(block))) < MIN_WORDS: diff --git a/tools/deai_voice.py b/tools/deai_voice.py index 21f756f..c353178 100644 --- a/tools/deai_voice.py +++ b/tools/deai_voice.py @@ -8,7 +8,7 @@ universal probability cutoff. voice_score(text, field_dir) -> compatibility score in [0, 1], or None - without a bundle or without the surprisal runtime the features need + without a bundle or without an input it reads (`scoring_reason`) paragraph_hits(text, field_dir) -> compatibility tuple advisories The score is never a paper gate or detector-evasion objective. Rewrite candidates @@ -113,16 +113,55 @@ def runtime_reason() -> str | None: "paragraph can be scored") +def reads_corpus_cos(bundle) -> bool: + """Whether the bundle can respond to `corpus_cos` at all. + + A bundle trained while the embedder was missing saw corpus_cos = 0.0 on + every row: its scaler records zero variance there, the classifier learned + nothing from the column, and scoring needs no centroid. A bundle with no + fitted scaler variances to read is taken to read it. + """ + variances = getattr(bundle.get("scaler"), "var_", None) + if variances is None: + return True + return float(variances[df.FEATURE_NAMES.index("corpus_cos")]) > 0.0 + + +def scoring_reason(bundle, field_profile_dir: Path | None, + centroid=None) -> str | None: + """Why `bundle` cannot score a paragraph here, or None when it can. + + The surprisal runtime first (`runtime_reason`); then, for a bundle that + reads `corpus_cos`, the embedder and the field's exemplar-embedding + centroid. Without either, `paragraph_features` writes the placeholder 0.0, + which a scaler fit on real cosines reads as several standard deviations + from its mean: until 2026-09-27 that score, moved by an absent input, + entered the findings and the rewrite ranking as a measured one. + """ + reason = runtime_reason() + if reason is not None or not reads_corpus_cos(bundle): + return reason + if not df.embedder_available(): + return ("sentence-transformers is not installed and the bundle reads " + "corpus_cos, so no paragraph can be scored") + if centroid is None and (field_profile_dir is None + or df.corpus_centroid(field_profile_dir) is None): + return (f"exemplar_embeddings_{df.EMBED_MODEL}.npy is missing and the " + "bundle reads corpus_cos; build it with build_profile (without " + "--no-warm)") + return None + + def voice_score(text: str, field_profile_dir: Path | None, model_name: str | None = None, centroid=None) -> float | None: """Return a field-similarity compatibility score, or None if unavailable. - None without a bundle and None without the surprisal runtime: the caller's - status names which (`voice_axis_status`), and a rewrite ranker reading - None weighs the term at nothing rather than at a nominal 0.0. + None without a bundle and None without an input the bundle reads: the + caller's status names which (`voice_axis_status`), and a rewrite ranker + reading None weighs the term at nothing rather than at a nominal 0.0. """ bundle = load_voice_model(field_profile_dir) - if bundle is None or runtime_reason() is not None: + if bundle is None or scoring_reason(bundle, field_profile_dir, centroid) is not None: return None model_name = model_name or bundle.get("model", do.DEFAULT_MODEL) if centroid is None and field_profile_dir is not None: @@ -138,7 +177,7 @@ def voice_axis_status(field_profile_dir: Path | None) -> dict: return feedback.axis_status( "L3.voice", "unmeasured", reason="voice_model.joblib is unavailable", detector="deai_voice") - reason = runtime_reason() + reason = scoring_reason(bundle, field_profile_dir) if reason is not None: return feedback.axis_status("L3.voice", "unmeasured", reason=reason, detector="deai_voice") @@ -196,7 +235,7 @@ def voice_findings(text: str, field_profile_dir: Path | None, probability cutoff is invented. """ bundle = load_voice_model(field_profile_dir) - if bundle is None or runtime_reason() is not None: + if bundle is None or scoring_reason(bundle, field_profile_dir) is not None: return [] calibrated = bundle_measured(bundle) model_name = bundle.get("model", do.DEFAULT_MODEL) diff --git a/tools/eval_findings.py b/tools/eval_findings.py index 6395fb0..3e5bf61 100644 --- a/tools/eval_findings.py +++ b/tools/eval_findings.py @@ -58,6 +58,7 @@ import statistics import sys +from collections import ChainMap, Counter from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) @@ -66,6 +67,7 @@ import deai_register # noqa: E402 -- because of that same sys.path insert import deai_salience # noqa: E402 -- because of that same sys.path insert import extract_sections as es # noqa: E402 -- because of that same sys.path insert +import extract_style # noqa: E402 -- because of that same sys.path insert # One AUC implementation and one document floor for both evaluators, so a rate # that is `unmeasured` here at n < 20 cannot be a rate there. from eval_docscale import MIN_DOCUMENTS, rank_auc # noqa: E402 -- same reason @@ -203,13 +205,28 @@ def leakage_paired(text: str, field_dir: Path, path: str) -> tuple[int, int]: ratio by a computable amount -- its own passages containing the term, and its own passage count. No rebuild is needed and no second population is involved, so nothing but membership differs. + + "Its own passages" are the ones the bank would hold: the paragraphs + `extract_style.bank_paragraphs` admits from each bucketed section, indexed + by `deai_register.passage_terms`, and a compound is re-judged by its rarest + part with the paper counted in. Blank-line blocks of the raw source, matched + by substring, counted the preamble and the bibliography and found `halo` + inside `halos`, which overstated `suppressed_by_own_membership`. The table + is the one the findings were judged against (`resolving_lexicon`). """ - lexicon = deai_register.load_lexicon(field_dir) + lexicon, _source = deai_register.resolving_lexicon(field_dir) if lexicon is None: return (0, 0) n_passages = int(lexicon.get("n_passages", 0)) table = lexicon["document_frequency"] - own = [block for block in text.split("\n\n") if len(block.split()) >= 5] + own = [paragraph for bucket, section in es.split_into_sections(text).items() + if bucket != es.DEFAULT_SECTION_BUCKET + for _index, paragraph, _words in extract_style.bank_paragraphs( + es.latex_to_plain(section))] + own_df = Counter(term for passage in own + for term in deai_register.passage_terms(passage)) + with_own = ChainMap({term: int(table.get(term, 0)) + count + for term, count in own_df.items()}, table) flagged = survives = 0 for finding in deai_register.register_findings(text, field_dir, path): # The zero-hit audit is suppressed by own membership by construction @@ -218,11 +235,9 @@ def leakage_paired(text: str, field_dir: Path, path: str) -> tuple[int, int]: if not str(finding["rule"]).startswith("register-foreign:"): continue term = deai_register.normalize(finding["observed"]["term"]) - df, _ = deai_register.corpus_document_frequency(term, table) - own_df = sum(1 for block in own if term.lower() in block.lower()) + df, _ = deai_register.corpus_document_frequency(term, with_own) flagged += 1 - if ((df + own_df) / (n_passages + len(own)) - < deai_register.RARE_DF_RATE): + if df / (n_passages + len(own)) < deai_register.RARE_DF_RATE: survives += 1 return (flagged, survives) diff --git a/tools/extract_sections.py b/tools/extract_sections.py index 3a783a2..d93e1cf 100644 --- a/tools/extract_sections.py +++ b/tools/extract_sections.py @@ -140,7 +140,7 @@ def classify_section(name: str) -> str: # Used only by latex_to_numeral_text below: strip a command name without # consuming its argument, so the digits inside survive. RE_TEX_MATH_CMD = re.compile(r"\\[a-zA-Z]+\*?") -# LaTeX writes a thousands separator as 14{,}850; collapsing it first keeps +# LaTeX writes a thousands separator as 1{,}250; collapsing it first keeps # that one quantity one numeral instead of three. RE_TEX_THIN_COMMA = re.compile(r"\{\s*,\s*\}") diff --git a/tools/extract_style.py b/tools/extract_style.py index ede3a1f..eb9d5b9 100644 --- a/tools/extract_style.py +++ b/tools/extract_style.py @@ -124,10 +124,10 @@ def sentences(text: str) -> list[str]: return [s for s in RE_SENTENCE_END.split(text) if s.strip()] -# A word is a run of letters in any script: the ASCII class split `naïve` into -# `na` and `ve` and `Poincaré` into `Poincar`, and the fragments entered the -# lexicon and the word counts, the same class of defect as the PDF ligatures. -RE_WORD_TOKEN = re.compile(r"[^\W\d_][^\W\d_'\-]*") +# A word is a letter in any script, continued by letters, `-` and `'` as the +# ASCII class was (`N-body`, `O'Brien's`): that class split `naïve` into `na` +# and `ve`, and a class that dropped `-` and `'` split every compound in two. +RE_WORD_TOKEN = re.compile(r"[^\W\d_](?:[^\W\d_]|['\-])*") def words(text: str) -> list[str]: @@ -424,6 +424,15 @@ def paired_paragraphs(text: str) -> tuple[str, list[str] | None]: EXEMPLAR_MAX_WORDS = 400 +def bank_paragraphs(plain: str): + """(index, paragraph, prose words) per paragraph of a section's plain text + the bank admits; `eval_findings.leakage_paired` models the bank with it.""" + for idx, para in enumerate(RE_PARAGRAPH_BREAK.split(plain)): + n_w = len(words(without_placeholders(para))) + if EXEMPLAR_MIN_WORDS <= n_w <= EXEMPLAR_MAX_WORDS: + yield idx, para.strip(), n_w + + def write_exemplar_bank(per_paper: list[tuple[float, dict]], profile_dir: Path) -> int: """Emit one JSONL row per qualifying paragraph in `exemplar_paragraphs.jsonl`. @@ -431,9 +440,9 @@ def write_exemplar_bank(per_paper: list[tuple[float, dict]], Each row: {id, section, tier, source, n_words, text, numeral_text}. The id carries the bucket, because a per-bucket index made `paper.tex:p0` the id of two rows. Section is the normalized bucket from - classify_section(); rows in the 'unknown' bucket and rows whose - paragraphs are pure placeholders are excluded, and `n_words` counts the - prose without its placeholders. Returns the number of rows written. + classify_section(); the 'unknown' bucket is excluded, admission is + `bank_paragraphs`, and `n_words` counts the prose without its + placeholders. Returns the number of rows written. `numeral_text` is the same paragraph under `latex_to_numeral_text`, paired by `paired_paragraphs`. A section it could not pair is reported and @@ -453,25 +462,12 @@ def write_exemplar_bank(per_paper: list[tuple[float, dict]], plain = st.get("plain_text", "") if not plain: continue - paragraphs = RE_PARAGRAPH_BREAK.split(plain) numeral = st.get("numeral_paragraphs") or [] if not numeral: unaligned.append(f"{source}:{sec}") - for idx, para in enumerate(paragraphs): - para = para.strip() - if not without_placeholders(para).strip(): - continue - n_w = len(words(without_placeholders(para))) - if n_w < EXEMPLAR_MIN_WORDS or n_w > EXEMPLAR_MAX_WORDS: - continue - rec = { - "id": f"{source}:{sec}:p{idx}", - "section": sec, - "tier": tier, - "source": source, - "n_words": n_w, - "text": para, - } + for idx, para, n_w in bank_paragraphs(plain): + rec = {"id": f"{source}:{sec}:p{idx}", "section": sec, "tier": tier, + "source": source, "n_words": n_w, "text": para} if numeral: rec["numeral_text"] = numeral[idx].strip() f.write(json.dumps(rec, ensure_ascii=False) + "\n") diff --git a/tools/retrieve_exemplars.py b/tools/retrieve_exemplars.py index 5446788..5132365 100644 --- a/tools/retrieve_exemplars.py +++ b/tools/retrieve_exemplars.py @@ -7,7 +7,7 @@ Default retrieval: sentence-transformers cosine on `all-MiniLM-L6-v2` embeddings, with a per-corpus `.npy` cache (rebuilt automatically when -the JSONL mtime is newer than the cache). +the bank's content fingerprint differs from the `.json` sidecar beside it). Fallback (`--allow-fallback`): if sentence-transformers is unavailable, fall back to naive topic-keyword overlap. This works without ML deps but @@ -82,50 +82,51 @@ def _build_or_load_embeddings( profile_dir: Path, model_name: str, ): - """Returns (embeddings ndarray L2-normalized, model). Builds cache on first - call or when JSONL is newer than cache. + """Returns (embeddings ndarray L2-normalized, model). + + The cache is reused only when the sidecar beside it records this bank's + content (SHA-1), row count and model. Judged by mtime order and row count, + a bank rebuilt with the same number of rows under an older-looking mtime + (a restore, a copy, a checkout) kept its old vectors, each now standing + for a different paragraph; `voice_dataset` already fingerprinted its own + cache this way. Raises ImportError if sentence-transformers is not installed. """ + import hashlib import numpy as np # numpy is a dep of sentence-transformers anyway from sentence_transformers import SentenceTransformer cache = _emb_cache_path(profile_dir, model_name) - needs_rebuild = ( - not cache.exists() - or cache.stat().st_mtime < jsonl_path.stat().st_mtime - ) + sidecar = cache.with_suffix(".json") + fingerprint = {"bank_sha1": hashlib.sha1(jsonl_path.read_bytes()).hexdigest(), + "n_rows": len(records), "model": model_name} + try: + recorded = json.loads(sidecar.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + recorded = None # because a missing or unreadable sidecar means "rebuild" model = SentenceTransformer(model_name) - - if needs_rebuild: - print( - f"[retrieve_exemplars] Building embedding cache " - f"({len(records)} paragraphs, model={model_name}) — " - f"first run downloads the model (~80 MB).", - file=sys.stderr, - ) - texts = [r["text"] for r in records] - embs = model.encode( - texts, - convert_to_numpy=True, - show_progress_bar=True, - normalize_embeddings=True, - ).astype("float32") - np.save(cache, embs) - print(f"[retrieve_exemplars] Cached → {cache}", file=sys.stderr) - else: + if cache.exists() and recorded == fingerprint: embs = np.load(cache) - if embs.shape[0] != len(records): - print( - f"[retrieve_exemplars] Cache size mismatch " - f"(cache N={embs.shape[0]} vs jsonl N={len(records)}); " - f"rebuilding.", - file=sys.stderr, - ) - cache.unlink(missing_ok=True) - return _build_or_load_embeddings(records, jsonl_path, profile_dir, model_name) - + if embs.shape[0] == len(records): + return embs, model + print( + f"[retrieve_exemplars] Building embedding cache " + f"({len(records)} paragraphs, model={model_name}) — " + f"first run downloads the model (~80 MB).", + file=sys.stderr, + ) + texts = [r["text"] for r in records] + embs = model.encode( + texts, + convert_to_numpy=True, + show_progress_bar=True, + normalize_embeddings=True, + ).astype("float32") + np.save(cache, embs) + sidecar.write_text(json.dumps(fingerprint), encoding="utf-8") + print(f"[retrieve_exemplars] Cached → {cache}", file=sys.stderr) return embs, model diff --git a/tools/tex_macros.py b/tools/tex_macros.py index feba823..bacf518 100644 --- a/tools/tex_macros.py +++ b/tools/tex_macros.py @@ -1,6 +1,6 @@ """Numbers a LaTeX manuscript holds in macros rather than in its prose. -A paper that writes `\\newcommand{\\Nfields}{63}` in its preamble and `\\Nfields{}` +A paper that writes `\\newcommand{\\Nsamples}{12}` in its preamble and `\\Nsamples{}` in its results has put a measured quantity where no text projection can see it. Both named projections in `extract_sections` strip the command and keep only its argument, so the use site contributes nothing and the *definition* site @@ -21,9 +21,9 @@ place where a preamble definition and a body use are both in scope. It is deliberately conservative: a macro is expanded only when its body is a bare numeric literal with no letters and no markup, so `\\newcommand{\\Msun}{M_\\odot}` -and every macro taking an argument are left exactly as they were. A definition -inside a comment is not a definition: harvested, a commented-out -`% \\newcommand{\\Nf}{63}` beat the live value that followed it. +and every macro taking an argument are left exactly as they were. A comment is +not source: harvested, a commented-out `% \\newcommand{\\Ns}{12}` beat the live +value that followed it, and a use inside a comment is left as written. """ from __future__ import annotations @@ -50,45 +50,62 @@ RE_BARE_NUMBER = re.compile(r"^[^A-Za-z\\]*\d[^A-Za-z\\]*$") +def _blank(text: str, spans: list[tuple[int, int]]) -> str: + """`text` with each span turned to spaces, its line breaks kept.""" + pieces: list[str] = [] + cursor = 0 + for start, end in spans: + pieces += [text[cursor:start], re.sub(r"[^\n]", " ", text[start:end])] + cursor = end + pieces.append(text[cursor:]) + return "".join(pieces) + + def expand_numeric(text: str) -> str: """Replace uses of numeric-literal macros with their numbers. - The definition itself is dropped rather than left in place. Keeping it - would double-count: the projections reduce `\\newcommand{\\Nf}{63}` to the - stray token `\\Nf63`, so an unexpanded manuscript already contributes one - stray numeral per definition, attributed to the preamble. + The definition itself is blanked rather than left in place. Keeping it + would double-count: the projections reduce `\\newcommand{\\Ns}{12}` to the + stray token `\\Ns12`, so an unexpanded manuscript already contributes one + stray numeral per definition, attributed to the preamble. It is blanked at + its own length with its line breaks kept, so no later line moves: replaced + by one space, a definition written across lines (`{%`, then the number on + the next line) pulled every later finding up by a line. - Definitions are read from a copy with comments blanked (same offsets), so - a commented-out definition is neither harvested nor dropped; a definition - that takes arguments is left exactly as written. + Definitions and uses are both read from a copy with comments blanked (same + offsets), so a commented-out definition is neither harvested nor dropped, + and a use inside a comment is not expanded; a definition that takes + arguments is left exactly as written. A use is matched only when the macro name is not a prefix of a longer name - (`\\Nf` must not fire inside `\\Nfields`), and an immediately following empty - brace pair -- the `{}` LaTeX authors write to protect the following space -- - is consumed with it. + (`\\Ns` must not fire inside `\\Nsamples`), and an immediately following + empty brace pair -- the `{}` LaTeX authors write to protect the following + space -- is consumed with it. """ macros: dict[str, str] = {} live = RE_TEX_COMMENT.sub(lambda m: " " * len(m.group(0)), text) - pieces: list[str] = [] - cursor = 0 + harvested: list[tuple[int, int]] = [] for match in RE_DEFINITION.finditer(live): body = match.group(3).strip() if match.group(2) is not None or not RE_BARE_NUMBER.match(body): continue macros[match.group(1)] = body - pieces.append(text[cursor:match.start()]) - pieces.append(" ") - cursor = match.end() + harvested.append(match.span()) if not macros: return text - pieces.append(text[cursor:]) - text = "".join(pieces) + text, live = _blank(text, harvested), _blank(live, harvested) # Longest name first so a shorter name cannot claim a prefix of a longer # one before the negative lookahead is reached. names = "|".join(sorted((re.escape(n) for n in macros), key=len, reverse=True)) # The `{}` is optional, and so is the space before it -- but only together. # Written `\s*(?:\{\})?` the whitespace is consumed whether or not a brace - # pair follows, which welds the macro to the next word: `\Nf and` -> `7and`. + # pair follows, which welds the macro to the next word: `\Ns and` -> `7and`. uses = re.compile(r"\\(" + names + r")(?![A-Za-z])(?:\s*\{\})?") - return uses.sub(lambda m: macros[m.group(1)], text) + pieces: list[str] = [] + cursor = 0 + for match in uses.finditer(live): + pieces += [text[cursor:match.start()], macros[match.group(1)]] + cursor = match.end() + pieces.append(text[cursor:]) + return "".join(pieces) diff --git a/tools/verify_references.py b/tools/verify_references.py index 00c3d2b..6cc8535 100644 --- a/tools/verify_references.py +++ b/tools/verify_references.py @@ -24,7 +24,8 @@ keeps fetched records in a JSON file so a repeated review round does not re-query the registries; a registry miss is never cached, so an identifier the registries are late to index is asked again next round, and a finding -built from a cached record says so (`observed.cached`). +built from a cached record says so (`observed.cached`). A 429 or 5xx answer +is asked once more; `SCI_PAPER_MAILTO` names a contact in the User-Agent. """ from __future__ import annotations @@ -34,6 +35,7 @@ import html import http.client import json +import os import re import sys import time @@ -51,6 +53,10 @@ import tex_assembly # noqa: E402 the assembled document and comment pattern for --tex USER_AGENT = "sci-paper-verify-references/1.0" +MAILTO_ENV = "SCI_PAPER_MAILTO" # contact address for CrossRef's polite pool +RETRY_STATUSES = frozenset({429, 500, 502, 503, 504}) +RETRY_WAIT_DEFAULT = 2.0 # seconds, when the registry names no Retry-After +RETRY_WAIT_CAP = 30.0 # seconds; a longer Retry-After is waited out only this long CROSSREF = "https://api.crossref.org/works/" DATACITE = "https://api.datacite.org/dois/" ARXIV = "https://export.arxiv.org/api/query?id_list=" @@ -296,25 +302,50 @@ class RegistryUnavailable(RuntimeError): a body the record readers cannot use; the entry stays unmeasured.""" -def fetch(url: str, timeout: int = 30) -> tuple[int, bytes]: - """(HTTP status, body). 404 is an answer, not an error; the rest raise. +def _retry_wait(error: urllib.error.HTTPError) -> float | None: + """Seconds before the one retry a transient answer earns, or None. - Known gap (audit F22): no retry or back-off on 429/5xx, and no `mailto:` - in the User-Agent for CrossRef's polite pool. + A 429 or a 5xx says nothing about the identifier, and without a retry one + rate-limit blip left an entry unmeasured and the axis `degraded`. The + registry's Retry-After is honoured up to RETRY_WAIT_CAP seconds. """ - request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT, - "Accept": "application/json"}) + if error.code not in RETRY_STATUSES: + return None + header = error.headers.get("Retry-After") if error.headers else None try: - with urllib.request.urlopen(request, timeout=timeout) as response: - return response.status, response.read() - except urllib.error.HTTPError as error: - if error.code == 404: - return 404, b"" - raise RegistryUnavailable(f"HTTP {error.code} from {url.split('?')[0]}") from error - except (http.client.HTTPException, OSError) as error: - # URLError and TimeoutError are OSError subclasses; IncompleteRead, - # BadStatusLine and RemoteDisconnected are HTTPException and are not. - raise RegistryUnavailable(f"{error} ({url.split('?')[0]})") from error + wait = float(header) if header else RETRY_WAIT_DEFAULT + except ValueError: # because an HTTP-date Retry-After is legal; wait the default + wait = RETRY_WAIT_DEFAULT + return min(max(wait, 0.0), RETRY_WAIT_CAP) + + +def fetch(url: str, timeout: int = 30) -> tuple[int, bytes]: + """(HTTP status, body). 404 is an answer, not an error; the rest raise, + a 429 or 5xx after one retry (`_retry_wait`). A contact address in + SCI_PAPER_MAILTO rides in the User-Agent, which CrossRef routes to its + polite pool.""" + mailto = os.environ.get(MAILTO_ENV, "").strip() + agent = f"{USER_AGENT} (mailto:{mailto})" if mailto else USER_AGENT + request = urllib.request.Request(url, headers={"User-Agent": agent, + "Accept": "application/json"}) + retried = False + while True: + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + if error.code == 404: + return 404, b"" + wait = None if retried else _retry_wait(error) + if wait is None: + raise RegistryUnavailable( + f"HTTP {error.code} from {url.split('?')[0]}") from error + retried = True + time.sleep(wait) # because the registry asked for the pause, or a 5xx is transient + except (http.client.HTTPException, OSError) as error: + # URLError and TimeoutError are OSError subclasses; IncompleteRead, + # BadStatusLine and RemoteDisconnected are HTTPException and are not. + raise RegistryUnavailable(f"{error} ({url.split('?')[0]})") from error def _first(value: Any, default: Any = "") -> Any: From bbd355a2ae8d6b624ba611b6cfaf2278300c47bf Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 04:48:18 -0400 Subject: [PATCH 22/27] docs: invented examples where the record quoted a manuscript under review The evaluation record, the changelogs, the standard, both READMEs, three skills and two audit records quoted a manuscript under review: a codename, phrases a reviewer marked, sentences from an absence sweep, values from an abstract, a macro, and words a register finding named. Each is replaced by an invented example of the same kind, or by the count the tool reported; no measurement changes. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG-ARCHIVE.md | 6 +- CHANGELOG.md | 34 ++++----- README.md | 2 +- README.zh-CN.md | 4 +- docs/SCIPAPER_STANDARD.md | 4 +- docs/architecture/DEAI_SUBSYSTEM.md | 6 +- docs/architecture/DISPOSITIONS.md | 2 +- docs/architecture/EVALUATION.md | 4 +- .../evaluation/narrative-salience-register.md | 23 +++--- .../projection-and-operating-point.md | 6 +- .../evaluation/vocabulary-and-residue.md | 73 +++++++++---------- docs/audits/codex-review-2026-09-04.md | 6 +- docs/audits/full-review-2026-09-27.md | 2 +- skills/de-ai/SKILL.md | 2 +- skills/paper-review/SKILL.md | 2 +- 15 files changed, 87 insertions(+), 89 deletions(-) diff --git a/CHANGELOG-ARCHIVE.md b/CHANGELOG-ARCHIVE.md index 56931d4..a6154e7 100644 --- a/CHANGELOG-ARCHIVE.md +++ b/CHANGELOG-ARCHIVE.md @@ -395,7 +395,7 @@ repeats nothing — what it lacks is rank. subfields. - **The same bank must not become the register reference.** 254 documents cannot express a rate below 1/254 = 3.9e-3, 39.4× coarser than the 1e-4 - threshold, so `saddle`, `classifier`, `recall`, and `ablation` flip to + threshold, so `classifier`, `recall`, and `ablation` flip to foreign on zero counts. A register reference needs of order 10,000 documents. The bank is stored and wired to nothing. @@ -462,8 +462,8 @@ the pre-2022 date filter turns out to bound less than it appears to. - **The same bank must NOT become the register reference.** `deai_register` calls a term foreign below a document-frequency rate of 1e-4, but 254 documents cannot express a non-zero rate below 1/254 = 3.9e-3, 39x - coarser than the threshold. Under that reference `saddle` — the central - concept of one of this suite's manuscripts — flips from native to + coarser than the threshold. Under that reference the central concept + of one of this suite's manuscripts flips from native to foreign purely on zero counts, as do `classifier`, `recall`, and `ablation`. A register reference needs of order 10,000 documents for a single occurrence to land at the threshold, so the broad bank stays. diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a21c33..35901fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -186,7 +186,7 @@ the line of its first appearance in the source, and whether that sentence carries a definition cue (`we call`, `defined as`, a parenthesis, an appositive). It is a list to walk, not a verdict: the action is to define the term at first use or replace it by the field's word, and to leave ordinary -phrasing the corpus happens to lack. On the Letter the reading reduced the +phrasing the corpus happens to lack. On a manuscript under review the reading reduced the 252 distinct unattested pairs of the sentence gate to 42 candidates, 16 of them already defined where they first appear. The sentence gate is unchanged: on the same manuscript the default run reports the same 86 @@ -200,9 +200,9 @@ findings before and after (set difference zero). Six tests read the mode, one of v0.37.0 named the absence residue and told the writer to say what the object does instead. Applied to the author's two manuscripts, that action had a failure mode of its own: a positive rewrite that only restates what -the neighbouring sentences already carry (`downstream of detection` reached -five occurrences in one paper; a statistic "reads position in catalog -coordinates" in an appendix whose method section had said so). The ruling +the neighbouring sentences already carry (one qualifying phrase reached +five occurrences in one paper; an appendix restated a coordinate convention +its method section had already given). The ruling now in the rule's action, the `paper` skill's self-check, the `de-ai` skill's residue bullet, and the standard's L4 bullet: judge the sentence in context and delete the clause first; rewrite only what the page does not yet say; @@ -210,7 +210,7 @@ keep a physical fact, a scope limit, or a conceded limitation as it stands. The second sweep over the two manuscripts cut 17 such clauses or sentences that the first sweep had rewritten, and the Pass-3 gate of `rewrite_reward` then did what §6 says it does: the dropped negation marker is a `missing` -invariant, reported on 4 of 7 changed Letter paragraphs and +invariant, reported on 4 of 7 changed paragraphs of the shorter manuscript and 38 of 54 pipeline-paper paragraphs, every one the absence clause itself or its companion (`\ref`, `because`, a math token inside the deleted clause), and one tokenizer artifact (`CDM-` read as an acronym). The action @@ -225,14 +225,14 @@ action. 523 tests in 27 files. The negative-label rule of v0.36.0 read headings and captions; the same defect in body prose had no detector and no name in the standard. The author's name for it is a menu line reading "tomato and egg (no braised -pork)": the head `never participates in the detection decision`, the -reference stratum `carries no quoted number`, `no support threshold is +pork)": the photometric catalog `never enters the shape measurement`, the +comparison sample `carries no redshift estimate`, `no colour cut is applied, because …`. Each tells the reader what the thing is not, in the place where the thing should stand, and each is what a revision leaves when an ingredient was taken out and its absence written in. `residue-absence` (`deai_residue.py`, rule 6) reads body sentences for the families, exempts a sentence that cites (a contrast with published work is a baseline, not a -tombstone), and skips a hyphenated compound (`never-touched controls` is a +tombstone), and skips a hyphenated compound (`never-masked tiles` is a name). Strengths were tiered on the held-out refereed full texts, 442 files and 1,899,092 prose words: `never` and the `nothing is` / `none sees` / `no … is applied` forms occur 0.008 times per 1,000 words there (15 `never` @@ -366,7 +366,7 @@ checklists were evaluated and not adopted. The human-labelling half of "half closed by provenance" is the author's act, not a repository item (`DISPOSITIONS.md`); the mentor-phrase check on the -Letter is closed on the draft it was made on, since the phrases it would test +manuscript under review is closed on the draft it was made on, since the phrases it would test are gone (§23.2); the five v0.14.0 "evidence still required" statuses each name the section that states them (`DEAI_SUBSYSTEM.md` §11). The v0.33.0 and v0.34.0 entries move to `CHANGELOG-ARCHIVE-v0.33-v0.34.md`. 516 tests in 27 @@ -468,9 +468,9 @@ defects, none of them in the manuscript. `\begin{table*}…\end{table*}` spanning lines was never blanked the way the corpus side blanks it in one pass over a passage. A `tabular*` column specification (`@{\extracolsep{\fill}}lll`), `\tabletypesize{\scriptsize}`, -`\tablenotetext{a}{Tied…}` and every caption's words counted on the manuscript +`\tablenotetext{a}{…}` and every caption's words counted on the manuscript side only: 23 of 90 zero-hit terms on one paper were `filllll`, `tabcolsep`, -`aTied`, `crimson`, `isosurfaces`. `body_only` now blanks floats and +a note letter fused to its first word, and caption words. `body_only` now blanks floats and length/table-note commands across lines beside the math spans of §23.1; the shared float pattern names `deluxetable*`, `longtable` and a bare `tabular*` as tables on both sides; a token with fewer than three letters (`a--c`, a @@ -480,7 +480,7 @@ papers, rank AUC 0.221 → 0.246; the thresholded rule 81 → **57** findings, 0.0351 → 0.0247 per 1,000, 30.0% → 22.2% of documents, AUC 0.352 → 0.392, own-membership 98.2% of 57. Salience and collocation lose the passages that were table cells (held-out 1.2025 → 1.1925 and 2.044 → 2.031 per 1,000, AUCs -0.770 → 0.774 and 0.688 → 0.691); every machine row and the Letter figure +0.770 → 0.774 and 0.688 → 0.691); every machine row and the manuscript figure §23.1 quotes (14 words, 11 strong) are unchanged. ### A negated object stops at the sentence end @@ -523,20 +523,20 @@ moved with it on the same 203 papers: 196 findings → 81, 0.0858 → 0.0351 per ### `L2.collocation`: words the field never joins -`physical cells` is two ordinary words and a pair no passage of 41,710 has +`binned kernel` is two ordinary words and a pair no corpus passage has written. `tools/deai_collocation.py` judges each sentence by the fraction of its distinct adjacent common-word pairs the bank does not attest, against a leave-one-out reference per bucket at sentence unit — at calibration a pair seen in exactly one passage is that passage's own. Only common words are judged as partners (11,286 of them), pairs break at punctuation, placeholders and dashes, and each flagged pair carries its expected co-occurrence and -e^−λ. On the private Letter it flags five of the six mentor-marked phrases +e^−λ. On a manuscript under review it flags five of the six mentor-marked phrases still present; on held-out papers the document novel-pair fraction separates machine text at AUC 0.688. ### Three structure families from the mentor's margin -Paper-as-agent subjects ("This Letter asks whether"), wh-cleft openers ("What +Paper-as-agent subjects ("This study asks whether"), wh-cleft openers ("What matters is") and modifier stacks (a three-plus-token noun phrase, head included, with two hyphenated compounds) join `deai_structure`'s auxiliary class: named on the sentence, never in `template_score`, with per-bucket human fractions in @@ -646,8 +646,8 @@ instead of a real one. ### Numbers held in macros were invisible, in both directions -A manuscript that writes `\newcommand{\Nfields}{63}` in its preamble and -`\Nfields{}` in its results put a measured quantity where neither named text +A manuscript that writes `\newcommand{\Nsamples}{12}` in its preamble and +`\Nsamples{}` in its results put a measured quantity where neither named text projection could read it. `RE_TEX_SIMPLE_CMD` reduces a command to its argument, so the use site contributed nothing while the definition site contributed the digits once, in the preamble, attributed to no reported diff --git a/README.md b/README.md index 911c587..30ed9ea 100644 --- a/README.md +++ b/README.md @@ -157,7 +157,7 @@ frequency **in your field's own corpus** is below 1e-4 — no curated cross-disc what lets `AUC` (df 1) separate from `epoch` (df 402) and `accuracy` (df 774) with no hand-maintained astronomy exception list. The same corpus answers a blunter question exhaustively: the **zero-hit audit** lists every body word no passage of the field has ever written, and `deai_collocation.py` every sentence -that joins common words the field never joins (`physical cells`, `controlled grid`), each pair with the +that joins common words the field never joins (`binned kernel`, `smoothed boundary`), each pair with the number of passages that would have written it by chance. Both are advice with author dispositions — a term the paper defines keeps its word and its pair — and neither is a detector: refereed papers carry *more* zero-hit words than machine drafts ([§23](docs/architecture/evaluation/vocabulary-and-residue.md)). diff --git a/README.zh-CN.md b/README.zh-CN.md index 1a4ca99..8ab8399 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -148,8 +148,8 @@ em-dash**，而全文级 dispersion 几乎没动 —— 0.47 → 0.49，人类 `accuracy`（df 774）分开，而不需要任何人去维护一张天文例外表。复合词按其最罕见的 部分判定；`_` 或 `^` 后面的 `\mathrm{}` 是下标不是术语；所有格折叠。同一份语料还回答 一个更钝的问题，而且是穷举的：**零命中审计**列出正文里每一个本领域从未写过的词， -`deai_collocation.py` 列出每一句把本领域从不并置的常用词并置起来的句子（`physical -cells`、`controlled grid`），每个词对都带着"按机会本该写出它的 passage 数"。两者都是 +`deai_collocation.py` 列出每一句把本领域从不并置的常用词并置起来的句子（`binned +kernel`、`smoothed boundary`），每个词对都带着"按机会本该写出它的 passage 数"。两者都是 带作者 disposition 的建议——本文自己定义的术语保留它的词和词对——都不是检测器： 已发表论文里的零命中词反而比机器草稿多（[§23](docs/architecture/evaluation/vocabulary-and-residue.md)）。 diff --git a/docs/SCIPAPER_STANDARD.md b/docs/SCIPAPER_STANDARD.md index 0ae1d49..1bb360a 100644 --- a/docs/SCIPAPER_STANDARD.md +++ b/docs/SCIPAPER_STANDARD.md @@ -181,7 +181,7 @@ A second, **auxiliary family class** covers rhetorical figures that are legitimate in isolation but machine-typical at density: **antithesis clusters** (two or more contrastive frames such as "X rather than Y" in one paragraph), **short reversal beats** ("It would not."), **paper-as-agent subjects** ("This -Letter asks whether"), **wh-cleft openers** ("What matters is"), and **modifier +study asks whether"), **wh-cleft openers** ("What matters is"), and **modifier stacks** (a noun phrase of three-plus tokens, head included, with two hyphenated compounds). Auxiliary families emit ordinary advisories under `structure-auxiliary` and are excluded from `template_score`, so the calibrated dispersion manifold is @@ -194,7 +194,7 @@ rigor" closers — are a panel-advisory class in `EVALUATION.md`, not a detector A **collocation axis** (`L2.collocation`, `deai_collocation`) measures, per sentence, the fraction of adjacent common-word pairs no passage of the field's corpus attests, against a leave-one-out reference per section. A pair it never -wrote is a coinage or a figure of speech ("physical cells"), weighted by the +wrote is a coinage or a figure of speech ("binned kernel"), weighted by the co-presence of its words (a ranking aid, never a filter); the action names the relation it compresses. Advisory; a defined term keeps its pair, and no claim is changed to dissolve one. The same bank read document-wide (`--glossary`) lists the unattested pairs a manuscript uses twice or more, the terms it coined, with the line of first use and whether that sentence defines them. diff --git a/docs/architecture/DEAI_SUBSYSTEM.md b/docs/architecture/DEAI_SUBSYSTEM.md index 30e19c1..e6f263c 100644 --- a/docs/architecture/DEAI_SUBSYSTEM.md +++ b/docs/architecture/DEAI_SUBSYSTEM.md @@ -160,12 +160,12 @@ structural patterns that keyword replacement cannot repair: - repeated paragraph templates. An auxiliary class — antithesis clusters, short reversal beats, paper-as-agent -subjects ("This Letter asks whether"), wh-cleft openers ("What matters is") and +subjects ("This study asks whether"), wh-cleft openers ("What matters is") and modifier stacks (a run of three or more tokens before a head noun carrying at least two hyphenated compounds) — is reported under `structure-auxiliary` and never enters `template_score`, so the calibrated manifold does not move when a -family is added. The last three came from a mentor's margin comments on a real -Letter; their human baseline fractions per bucket are in the structure baseline +family is added. The last three came from a mentor's margin comments on a +manuscript under review; their human baseline fractions per bucket are in the structure baseline (`paper_agent_frac`, `wh_cleft_frac`, `modifier_stack_frac`). These are advisories. A detector match identifies a construction to inspect; it does diff --git a/docs/architecture/DISPOSITIONS.md b/docs/architecture/DISPOSITIONS.md index 25c3e16..31cce4f 100644 --- a/docs/architecture/DISPOSITIONS.md +++ b/docs/architecture/DISPOSITIONS.md @@ -56,7 +56,7 @@ audit and keeping the suite and validator green, and updates this table and | `L0.register` operating point | **Closed by refuting its premise** | Swept 5 → 50 uses against 203 held-out refereed papers and 173 machine documents: rank AUC is below 0.5 at **every** setting, so the axis fires more on refereed prose than on machine prose everywhere on the curve and tightening silences the machine side faster. No setting makes it a detector; the knob buys advisory volume. Cut at 15, the first point where a referee-grade paper is not flagged more often than not: 0.0858 per 1,000 words on 44.83% of held-out documents, AUC 0.2856 (EVALUATION §18.4). | | `L0.register` threshold as the vocabulary instrument | **Complemented by the zero-hit audit (v0.36.0); the 15-use rule stays as the selective instrument** | The 15-use floor kept the axis from flagging refereed prose more often than not, at the cost of missing every word a draft uses once. The owner's rule sits beside the knob: every body word with corpus df 0 is listed, and the only exemptions are mechanical (a formation of an attested stem, a term defined here at first use, a proper name capitalised at every use; a cited method's own name is an author disposition). Measured on 203 held-out refereed papers against 173 machine documents the audit fires on every refereed paper (3.37 words per 1,000 body prose words under v0.36.2's denominator; 2.21 per raw-source word under v0.36.1's float projection, 2.66 before it) and has rank AUC 0.174 (0.246 before the denominator fix) — it is not a detector and is not shipped as one; it is exhaustive advice whose dispositions are the author's (EVALUATION §23.1). | | Asymmetric projection, third instance: headings | **Fixed — one regex owner** | Section headings sat in the manuscript's body projection but never in a corpus passage, so `Validation` fused with the first sentence under it and read as a zero-hit word; the same fusion put heading words into collocation pairs. `extract_sections.RE_HEADING_COMMAND` is now the single owner and `deai_reference.units`/`sections`, `deai_register.body_only` and `length_gate` all consume it (EVALUATION §23.1). | -| Collocation as an axis (`L2.collocation`) | **Shipped (v0.36.0), `measured`; glossary reading v0.38.0** | Per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (v0.36.3 rebuild; 41,710 before), leave-one-out reference per bucket at sentence unit; distinct pairs, breaks at punctuation, placeholders, dashes, slashes, periods and digits, 11,282 admitted partner words (11,286 before). Flags five of the six mentor-marked phrases present in the private Letter and separates machine documents from held-out refereed papers on the document fraction (EVALUATION §23.2). | +| Collocation as an axis (`L2.collocation`) | **Shipped (v0.36.0), `measured`; glossary reading v0.38.0** | Per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (v0.36.3 rebuild; 41,710 before), leave-one-out reference per bucket at sentence unit; distinct pairs, breaks at punctuation, placeholders, dashes, slashes, periods and digits, 11,282 admitted partner words (11,286 before). Flags five of the six mentor-marked phrases present in a manuscript under review and separates machine documents from held-out refereed papers on the document fraction (EVALUATION §23.2). | | Residue axis (`L4.residue`) with a diff gate | **Shipped (v0.36.0), strengths set on refereed prose; absence rule added v0.37.0** | Four static rules (self-history, edit-meta, negative-label, and `residue-absence`, a body sentence defining the paper's own object by what it never does, §23.4a) and one diff rule. Strengths were measured, not assumed: on 203 held-out papers the first families put a strong finding in 154 documents; the body projection (preamble macros and "Planck Collaboration XXX" are not prose), dropping `used to` (174 instrumental hits) and demoting `initially`/`originally`/`at first` to ordinary bring that to 34 (16.7%), and v0.36.2 — marks read across line breaks, `we have added` a mark only with a document object (fourteen procedural hits otherwise) — to 27 (13.3%), 8 of them `\textcolor{red}` in tables. The static negative-label rule is ordinary (26% of refereed papers); the diff rule gates. The skill mirrors the family lists and the validator proves the mirror. Exit 1 on a strong finding is the plugin's third narrow exit contract (EVALUATION §23.4). | | Structure families from the mentor's comments | **Shipped as auxiliary (v0.36.0)** | Paper-as-agent, wh-cleft and modifier-stack join the auxiliary class: named on the sentence, excluded from `template_score`, baseline fractions recorded per bucket (EVALUATION §23.3). | | Condensation as a read-and-trim pass | **Replaced by the removal map (v0.36.0)** | The skill's own sweep removed a few percent by hand; `condense_map.py` lists every entry with the words it frees and `length_gate.py --require-shrink` turns the map's target into an exit code. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%; 3.1% before v0.36.2 counted each unit once), so a pass that cuts less than that has not found what the map lists (EVALUATION §23.5). | diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index 1a7b205..b78da54 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -114,10 +114,10 @@ result. | L0 lexical/punctuation | measured | Deterministic Tier A, em-dash, and Tier B cap implementation with CLI regression tests. | Continue regression coverage when policy changes. | | L1 distribution | degraded (refuted) | §16: measured on 500 human papers against 173 `docval` AI documents, one observation per document. Burstiness reverses sign — adversarial prose is *more* bursty than the human median (1.036 vs 0.775, AUC 0.181) and long-form sits inside the human band (AUC 0.441) — while flagging 7.2% of humans. Signposting has an AUC of 0.247, below chance, and flags 0 of 173 AI documents at the shipped default. | None. `deai_policy.json` is withdrawn as a roadmap item: the statistics do not support an operating point, so the axis stays advisory by measurement rather than by absence. | | L1 UID | degraded | `style-profile/wgl/uid_baseline.json` records paragraph-level GPT-2-large summaries. | A documented operating point and human false-flag behavior; audit sensitivity to mathematics and jargon. | -| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2705 per passage on 203 held-out ApJ/ApJL/A&A papers (§17.5's headline; 0.2776 when re-run in the v0.36.2 process) against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nfields}{63}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | +| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2705 per passage on 203 held-out ApJ/ApJL/A&A papers (§17.5's headline; 0.2776 when re-run in the v0.36.2 process) against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nsamples}{12}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | | L0 register | measured | §14: document frequency over 41,644 corpus passages, 53,367 terms (v0.36.3 profile rebuild; 41,710 and 53,414 before it, and the 57 held-out findings below are the same 57 under both); compound-by-rarest-part and macro-subscript handling; native-term controls pass. Since v0.32.0 a `-` profile whose own bank cannot resolve the 1e-4 gate judges against `` and names the borrowed bank in every finding: on 36 letter-format documents the 706-passage `wgl-letter` bank produced 262 findings the field bank did not (`sne`, `bao`, `pantheon`, and `letter` itself) against 2 the other way (§18.5). | Recall below the 15-use floor, and a **measured** false-positive rate: §18.4 put it at 0.0858 findings per 1,000 words on 203 held-out refereed papers (44.83% of documents), rank AUC 0.2856 against machine text — it still fires more on human prose than on AI drafts — and §23.1 re-measures the same papers at 0.0351 (30.05%, AUC 0.352) after the heading projection fix and at 0.0247 (22.2%, AUC 0.392) after v0.36.1's float projection fix, same direction; v0.36.2 counts per body prose word instead of per raw-source word, and the same 57 findings read 0.0371 (AUC 0.391). §17.3, re-measured: 94.44% of the 198 remaining flags would be suppressed by the paper's own bank membership (98.2% of the 57 under v0.36.1). The operating point is no longer an open item: swept 5→50 uses, rank AUC stays below 0.5 everywhere, so no setting makes this a detector and the knob buys advisory volume only. It was cut at 15, the first point where a referee-grade paper is not flagged more often than not (§18.4). §21 replicates that refutation on a second population — 22 papers by one author, 1996–2015, sampled by author rather than by journal and spanning a different two decades — where rank AUC is **0.328**, again below 0.5 (0.412 on the v0.36.3 profile rebuild, 5 findings on the 22 papers). | | L2 sentence structure | measured for deterministic matches; degraded for strength | `style-profile/wgl/structure_baseline.json` provides section-level reference fractions over 27,841 paragraphs — method 9,478; results 3,978; data 3,894; intro 3,812; discussion 3,635; conclusion 2,611; abstract 433. §23.3 adds the per-bucket human fractions of the three auxiliary families taken from a mentor's comments (paper-as-agent 0.13–1.00%, wh-cleft 0.00–0.23%, modifier stacks 2.5–15.9%). | Author-labelled difficult cases. A calibrated strong-advisory threshold is no longer expected from `deai_policy.json` (§16). | -| L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (530,504 pairs over 11,282 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,391; method 27,207; intro 14,321; results 12,509; data 12,048; discussion 12,183; conclusion 8,588 after the v0.36.3 profile rebuild; 41,710 passages and 541,309 → 530,677 pairs under v0.36.2); p90 gates 0.44–0.59, p95 0.50–0.67. On the first measured draft of the private Letter it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at AUC 0.704 (7,237 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same Letter: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; a second generation process (§20's lesson). | +| L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (530,504 pairs over 11,282 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,391; method 27,207; intro 14,321; results 12,509; data 12,048; discussion 12,183; conclusion 8,588 after the v0.36.3 profile rebuild; 41,710 passages and 541,309 → 530,677 pairs under v0.36.2); p90 gates 0.44–0.59, p95 0.50–0.67. On the first measured draft of a manuscript under review it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at AUC 0.704 (7,237 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same manuscript: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; a second generation process (§20's lesson). | | L4 residue | measured | §23.4: four static rules (self-history, edit-meta, negative-label, and since v0.37.0 the absence rule, §23.4a) and one diff rule, deterministic, strengths set on 203 held-out refereed papers (70,848 body prose sentences): strong self-history fires in 20 papers, strong edit-meta in 8 (68 of 69 are `\textcolor{red}` in tables), any strong in 27 (13.3%); the static negative-label rule is ordinary at 26% of papers. Four passes to get there, the first a projection asymmetry and the last the procedural sense of `we have added`. | Recall against a labelled edit history; only the author's own drafts can supply it. | | L0 register zero-hit audit | measured, not a detector | §23.1: every body word with corpus df 0, on both sides of one projection (headings now excluded on both). All 203 held-out refereed papers carry zero-hit words, 3.37 per 1,000 body prose words (2.21 per raw-source word under v0.36.1, 2.66 before its float fix), against 1.05 for the 173 machine documents — rank AUC 0.174, the audit fires more on human prose — so it ships as exhaustive advice with author dispositions, never as a gate. The two projection fixes moved the thresholded rule from 0.0858 to 0.0247 findings per 1,000 raw-source words on the same papers (0.0371 per body prose word). | None as a detector. Whether the mechanical exemptions (attested stem, defined term, proper name) are the right three. | | L2 document structure | measured | §9: cross-paragraph dispersion calibrated one-observation-per-paper over 493 complete human `wgl` papers — the 2026-08-17 scoring; the v0.36.3 rebuild (2026-09-05) re-fit the artifact on 507 documents and the operating-point figures were not re-scored on it — human false-flag at the shipped conformal operating point 0.0325 (manifold) and 0.0426 (role) against nominal α = 0.05. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. | Re-score §9 through `eval_docscale.py` on the 507-document artifact; continue recalibration when the corpus changes. diff --git a/docs/architecture/evaluation/narrative-salience-register.md b/docs/architecture/evaluation/narrative-salience-register.md index 5b91962..cfb061e 100644 --- a/docs/architecture/evaluation/narrative-salience-register.md +++ b/docs/architecture/evaluation/narrative-salience-register.md @@ -171,7 +171,7 @@ discipline's vocabulary. All measurements below use the `wgl` field profile. ### 14.1 The reduction defect that preceded both The first prototype measured `recital_frac = 0.0` on an abstract carrying -`$500$`, `$14{,}850{,}000$`, `$62\%$`, `$0.98\%$`, `$34.6\%$`, `$\AUC=0.817$` +`$200$`, `$1{,}250{,}000$`, `$57\%$`, `$0.884\%$`, `$23.9\%$`, `$\mathrm{AUC}=0.927$` and five more quantities. `extract_style.latex_to_plain` — the shared front end for every axis — replaces each math span with the token `[math]`, so **every numeral in a LaTeX manuscript is destroyed before any detector sees it**. That @@ -184,7 +184,7 @@ set and differs in one decision: numerals inside *inline* math survive. Displayed equations are dropped by both, because their digits are the constants of a definition — counting the 3 in a volume formula as a reported quantity made every derivation paragraph read as a recital of measurements. The LaTeX -thousands form `14{,}850{,}000` collapses to one numeral rather than three. +thousands form `1{,}250{,}000` collapses to one numeral rather than three. `latex_to_plain` is untouched, so no existing calibration asset moved then. One did later, and in the other direction: the bank stored only the `latex_to_plain` text, so the salience reference was calibrated on paragraphs @@ -281,15 +281,15 @@ test. ### 14.3 Salience on the case document -`Letter/main.tex` (804 lines, read 2026-08-16) against the wgl reference: +A manuscript under review (804 lines, read 2026-08-16) against the wgl reference: | Passage | `max_recital_run` | `run_frac` (pct) | `recital_frac` (pct) | `num/sent` (pct) | |---|---|---|---|---| | abstract | 4 of 8 | 0.50 (p91) | 0.50 (p78) | 1.38 (p79) | -| grid definition (L137) | 2 of 5 | 0.40 (p90) | 0.60 (p91) | 8.20 (p100) | -| source model (L179) | 4 of 4 | 1.00 (p100) | 1.00 (p100) | 4.00 (p99) | -| detector block (L210) | 3 of 3 | 1.00 (p100) | 1.00 (p100) | 3.33 (p99) | -| twin fit (L541) | 5 of 6 | 0.83 (p98) | 0.83 (p97) | 1.67 (p95) | +| body paragraph (L137) | 2 of 5 | 0.40 (p90) | 0.60 (p91) | 8.20 (p100) | +| body paragraph (L179) | 4 of 4 | 1.00 (p100) | 1.00 (p100) | 4.00 (p99) | +| body paragraph (L210) | 3 of 3 | 1.00 (p100) | 1.00 (p100) | 3.33 (p99) | +| body paragraph (L541) | 5 of 6 | 0.83 (p98) | 0.83 (p97) | 1.67 (p95) | Run length is the discriminating feature for the abstract (p91) while density is unremarkable (p78/p79). That is the intended behaviour: a quantitative abstract @@ -331,7 +331,6 @@ Controls that must not fire: |---|---:|---|---:|---| | `accuracy` | 774 | itself | 5.0e-2 | not flagged | | `epoch` | 402 | itself | 2.6e-2 | not flagged | -| `same-plane` | 536 | `plane` | 3.4e-2 | not flagged | | `aperture-mass` | 313 | `aperture` | 2.0e-2 | not flagged | | `training` | 155 | itself | 9.9e-3 | not flagged | | `benchmark` | 81 | itself | 5.2e-3 | not flagged | @@ -371,11 +370,11 @@ threshold: corpus-rare: `aperture-mass`, this field's core observable, appears in 8 passages as a string. Judging a compound by its rarest part fixes it while keeping `cross-validation` foreign via `validation`. -- *Possessives.* `sub-halo's` and `campaign's` fold onto their bare terms. +- *Possessives.* `point-source's` and `lens's` fold onto their bare terms. After the fix: 3 findings, all substantive. -**Recall cost, stated.** `probit` (df 0) and `pooled` (df 0) are used 3 and 4 +**Recall cost, stated.** Two statistics terms at df 0 are used 3 and 4 times, below the manuscript-use floor, and are not reported. The floor buys precision at the price of terms used a handful of times; below it the corpus's own sampling gaps dominate the comparison. @@ -589,8 +588,8 @@ A 254-document corpus cannot express a non-zero rate below 1/254 = 3.94e-3, **39.4× coarser than the threshold**; a single occurrence lands at the threshold only at about 10,000 documents. -The consequence is not hypothetical. Under the subfield reference `saddle` — -the central concept of the case document — flips from native (16/13,642) to +The consequence is not hypothetical. Under the subfield reference the central +concept of the case document flips from native (16/13,642) to foreign (0/254), as do `classifier`, `recall`, and `ablation`, all on zero counts rather than on any property of the field's vocabulary. There are not 10,000 weak-lensing ApJ/ApJL/A&A papers in the window, so the register diff --git a/docs/architecture/evaluation/projection-and-operating-point.md b/docs/architecture/evaluation/projection-and-operating-point.md index 80d1093..117ed22 100644 --- a/docs/architecture/evaluation/projection-and-operating-point.md +++ b/docs/architecture/evaluation/projection-and-operating-point.md @@ -307,11 +307,11 @@ dated (§18.7). §18.2 found the salience axis reading digits that were not quantities. This is the same class in the other direction: quantities the axis could not read. -A manuscript that writes `\newcommand{\Nfields}{63}` in its preamble and -`\Nfields{}` in its results has put a measured number where neither named +A manuscript that writes `\newcommand{\Nsamples}{12}` in its preamble and +`\Nsamples{}` in its results has put a measured number where neither named projection can see it. `RE_TEX_SIMPLE_CMD` reduces a command to its argument, so the *use* contributes nothing at all, while the *definition* contributes -`63` once — in the preamble, attributed to no reported section. Two errors in +`12` once — in the preamble, attributed to no reported section. Two errors in opposite directions, which is why the net was small enough to go unnoticed. Found by real-machine review of a manuscript under review that uses the habit diff --git a/docs/architecture/evaluation/vocabulary-and-residue.md b/docs/architecture/evaluation/vocabulary-and-residue.md index 69804ea..e15ed6f 100644 --- a/docs/architecture/evaluation/vocabulary-and-residue.md +++ b/docs/architecture/evaluation/vocabulary-and-residue.md @@ -8,8 +8,8 @@ raw source; none in any calibration bank), the 544 in-sample refereed papers, and the 173 `docval` machine documents of §16. Every per-1,000-word rate is per body prose word since v0.36.2 — the register projection with placeholders dropped — where it was per raw-source word, bibliography and preamble included. -The private Letter that motivated the work is measured too; nothing from it is -quoted beyond single words and the two-word phrases its mentor marked. +A manuscript under review that motivated the work is measured too; nothing +from it is quoted, and only the counts the tools report on it are given. ## 23.1 The zero-hit audit: every word the corpus never wrote @@ -21,7 +21,7 @@ reason — the paper defines it, or it names a method the field has not used before. `deai_register` now carries that audit as `register-zero:` beside the thresholded rule. The mechanical exemptions, reported ordinary, are a formation of an attested stem (`clears` from `clear`), a term the paper defines -at first use, and a proper name (`benjamini--hochberg`); every other zero-hit +at first use, and a proper name (`kruskal--wallis`); every other zero-hit word is strong and takes one of the §5.2 step-1b dispositions. **It is exhaustive, and it is not a detector.** Rates per 1,000 body prose @@ -45,14 +45,13 @@ bank membership buys — a paper whose own vocabulary is in the denominator loses two thirds of its zero-hit words — and is not a leakage estimate, for the era reason §17.3 gives. -**What it finds on the Letter** (re-measured on the current draft, 2026-09-04). -Thirteen zero-hit words, ten strong. Three are statistics vocabulary the -field's corpus does not use (`probit`, `deciles`, `exceedance`), three are this -plugin's own software vocabulary that had leaked into an astronomy manuscript -(`unmeasured`, `rescored`, `outranks`), one is a coinage the paper needs to -define (`resolvability`), and the ordinary hits are stem formations (`clears`, -`earns`) and a name (`benjamini--hochberg`). Every one of those is a decision -the author can make in a second, which is the point of listing them all. +**What it finds on the manuscript under review** (re-measured on the current +draft, 2026-09-04). Thirteen zero-hit words, ten strong. Three are statistics +vocabulary the field's corpus does not use, three are this plugin's own +software vocabulary that had leaked into an astronomy manuscript, one is a +coinage the paper needs to define, and the ordinary hits are two stem +formations and a name. Every one of those is a decision the author can make +in a second, which is the point of listing them all. **A third projection asymmetry.** Building the audit exposed the third instance of the class recorded in §17.4 and §18.1: section headings sat in the @@ -80,13 +79,13 @@ table above is the re-taken one. ## 23.2 `L2.collocation`: sentences that join words the field never joins -**What the mentor marked.** Of 25 margin comments on the Letter, eight were of -one kind — "I don't know what this means", "this is jargon" — and each sat on a -two-word phrase (`physical cells`, `controlled grid`, …) whose words are -ordinary field vocabulary and whose *pair* no passage of the corpus has -written. The register axis cannot see this: both words have healthy document +**What the mentor marked.** Of 25 margin comments on the manuscript under +review, eight were of one kind — "I don't know what this means", "this is +jargon" — and each sat on a two-word phrase whose words are ordinary field +vocabulary and whose *pair* no passage of the corpus has written (a pair of +the kind, invented here: `binned kernel`). The register axis cannot see this: both words have healthy document frequency. A prototype on the first 1,500 words of each document, ranking -documents by their novel-pair fraction, reached AUC 0.855 and put the Letter +documents by their novel-pair fraction, reached AUC 0.855 and put that manuscript at 0.557 against a human p90 of 0.300; that prototype is superseded by the shipped axis below, which is judged per sentence and reports its document fraction as evidence only. @@ -144,12 +143,12 @@ written (the §17.3 argument). And the in-sample row is the bank-membership effect at its largest — a paper's own pairs are attested by itself — so it is not a leakage estimate either. -**The Letter** (re-measured on the current draft, 2026-09-04). 29 sentence +**The manuscript under review** (re-measured on the current draft, 2026-09-04). 29 sentence findings, 22 strong, document novel-pair fraction 0.603 over 481 judged pairs. On the draft first measured, of the eight phrases the mentor marked two had already been rewritten out and **five** of the six present were in flagged -sentences (`physical cells` and `controlled grid` among them). The current -draft carries neither phrase, so the mentor-phrase check is closed on the +sentences, the two phrases the comments quoted among them. The current +draft carries neither of those two, so the mentor-phrase check is closed on the draft it was made on rather than repeatable: the phrases it would test are gone, which is the outcome the check exists to produce. On the shipped examples the axis rises from 2 findings to 3 across the revision, because the synthetic paper repeats its own coined parameter and the @@ -169,9 +168,9 @@ recalibrated and the rate re-measured. ## 23.3 Three structure families from the mentor's margin The remaining comments that named a *sentence shape* rather than a phrase fell -into three kinds: a paper-as-agent subject with a mental verb ("This Letter -asks whether …" — "is it an AI prompt?"), a wh-cleft opener ("What it can -conclude is limited by …" — "feels AI-ish"), and a modifier stack (three or +into three kinds: a paper-as-agent subject with a mental verb (of the form +"This study asks whether …" — "is it an AI prompt?"), a wh-cleft opener (of +the form "What matters here is …" — "feels AI-ish"), and a modifier stack (three or more tokens before the head noun with at least two hyphenated compounds). `deai_structure` names them under `structure-auxiliary`, outside `template_score`, and `--calibrate` records their human fraction per bucket: @@ -191,7 +190,7 @@ The `n` column is the v0.32.0 bank's; the v0.36.3 `structure_baseline.json` hold A first cut of the stack rule counted any run of non-function tokens and put 15% of human method paragraphs in it; cutting the run at the head noun and requiring two hyphenated compounds brought the fraction to what the table -shows. On the Letter the families name one paper-agent sentence, one wh-cleft +shows. On the manuscript under review the families name one paper-agent sentence, one wh-cleft and nine stacks, and the two stacks the mentor marked are among the nine. ## 23.4 `L4.residue`: the trace an edit leaves @@ -228,15 +227,15 @@ have added a paragraph`). What remains strong fires in 13% of refereed papers; 68 of the 69 edit-meta hits are `\textcolor{red}` inside tables, an author's emphasis that the disposition `kept` answers in a word. The static negative-label rule fires on 26% of refereed papers and is ordinary; only the -diff rule gates. The Letter carries 0 residue findings. +diff rule gates. The manuscript under review carries 0 residue findings. ### 23.4a `residue-absence`: the prose form of the negative label The author's name for the defect is a menu line reading "tomato and egg (no braised pork)": a sentence that tells the reader what the paper's own object -never does or has, where the thing itself should stand (`the head never -participates in the decision`, `the reference carries no quoted number`, `no -support threshold is applied, because …`). The negative-label rule catches it +never does or has, where the thing itself should stand (`the photometric +catalog never enters the shape measurement`, `the comparison sample carries no +redshift estimate`, `no colour cut is applied, because …`). The negative-label rule catches it in headings and captions; this rule reads body sentences, and a citation in the sentence exempts it as a baseline contrast. The families were tiered on the held-out full texts (442 files, 1,899,092 prose words), counting sentences @@ -262,21 +261,21 @@ case`, `not a surprise`), the qualifier class §6 protects. On the author's pipeline paper before its sweep the rule found 15 strong and 17 ordinary sentences in 11,661 prose words, `never` alone at 0.94 per 1,000 words, 118 times the refereed rate; after the sweep, 0 strong and 2 ordinary, both -physics kept under a recorded disposition. The Letter went from 2 strong to -0 strong and 1 ordinary. +physics kept under a recorded disposition. The manuscript under review went +from 2 strong to 0 strong and 1 ordinary. **Delete first (v0.37.1).** The first sweep rewrote every class-A sentence positively. Read in context, a share of those rewrites carried nothing the -neighbouring sentences did not: `downstream of detection` had reached five -occurrences in the pipeline paper, an appendix restated the method section's -`catalog coordinates`, a one-line topic sentence repeated the sentence -after it. The author's ruling, now the rule's action text, is +neighbouring sentences did not: one qualifying phrase had reached five +occurrences in the pipeline paper, an appendix restated a convention the +method section had already given, a one-line topic sentence repeated the +sentence after it. The author's ruling, now the rule's action text, is to delete the clause first and rewrite only what the page lacks; the second -sweep cut 17 such clauses or sentences across the two manuscripts (Letter -body 2235 → 2221 words). The §6 fidelity gate then reports what the ruling +sweep cut 17 such clauses or sentences across the two manuscripts (the +shorter one's body 2235 → 2221 words). The §6 fidelity gate then reports what the ruling implies: with the absence clause gone, its `not` / `no` / `never` is a missing negation invariant. On the final text `rewrite_reward` flagged -4 of 7 changed Letter paragraphs and 38 of 54 changed +4 of 7 changed paragraphs of the shorter manuscript and 38 of 54 changed pipeline-paper paragraphs, every flag the dropped absence marker or a token inside the deleted clause (`\ref`, `because`, a math symbol), plus one tokenizer artifact (`CDM-consistent` unpacked to `CDM`, read as a dropped diff --git a/docs/audits/codex-review-2026-09-04.md b/docs/audits/codex-review-2026-09-04.md index fb88264..3405873 100644 --- a/docs/audits/codex-review-2026-09-04.md +++ b/docs/audits/codex-review-2026-09-04.md @@ -93,7 +93,7 @@ Read-only review of `git diff 9f0b920..edd480c`; session `01a06ebe-b4c5-7802-bfe 触发：`\section{Results TODO}\nClean prose.`，或 `\begin{figure}\n\caption{TODO}\n\end{figure}` 放在 Results 下，均没有 residue finding；正文独立 `TODO` 则 strong。建议：residue 的 edit-meta 扫描保留可见 headings、captions、table text；不要复用排除这些内容的词汇投影。另将跨行 `in the revised\nversion` 按规范化空白扫描，当前逐行 regex 漏报。`tools/deai_register.py:344`、`tools/deai_residue.py:204` 8. **major · [tools/deai_residue.py:288](../../tools/deai_residue.py) · diff rule 检查的是任一消失的词干，不是完整对象。** - 触发：before 为 `M + "The correction is applied."`；after 为 `M + "The measurement is applied.\n\\caption{Without the saddle correction}"`。得到 strong，虽然 `saddle correction` 从未在 before 出现。另把旧 caption 的 `Blue points` 改成 `Red points`，也会将原有 negation 当成新增。建议：比较 canonical negated object；确认完整对象 before 存在、after 缺失；对象新旧不能由整段 caption 字符串决定。`tools/deai_residue.py:280`、`tools/deai_residue.py:285` + 触发：before 为 `M + "The correction is applied."`；after 为 `M + "The measurement is applied.\n\\caption{Without the dust correction}"`。得到 strong，虽然 `dust correction` 从未在 before 出现。另把旧 caption 的 `Blue points` 改成 `Red points`，也会将原有 negation 当成新增。建议：比较 canonical negated object；确认完整对象 before 存在、after 缺失；对象新旧不能由整段 caption 字符串决定。`tools/deai_residue.py:280`、`tools/deai_residue.py:285` 9. **major · [tools/deai_reference.py:162](../../tools/deai_reference.py) · 无 section 的文档静默漏检。** 触发：纯文本 `We no longer use this method.`，或前加 `\section {Methods}` / `\section[Short]{Methods}`，self-history 均无 finding；condense 的句子扫描同样为空。原因是 section regex 不接受这些合法形式，随后 `(document)` 被丢弃。建议：共享 section parser；无 heading 时保留 document unit，不能套用 bucket baseline 的轴明确标为 unavailable。`tools/deai_metrics.py:25`、`tools/deai_metrics.py:120`、`tools/condense_map.py:119` @@ -198,7 +198,7 @@ Read-only review of `git diff 9f0b920..edd480c`; session `01a06ebe-b4c5-7802-bfe | `test_projection_parity_matrix` | 对 §3 每个构造比较实际 corpus pipeline 与 manuscript tokens；覆盖 multi-line cite/ref、unit macro、外置 tablenote、非星号 deluxetable、minipage、深层 braces。现有 float test 主要覆盖 star 环境和 manuscript 一侧。`tests/test_deai_register.py:316` | | `test_float_with_blank_lines_stays_excluded` | figure/table 内空行和 comment-only 行不能让 caption/cells 进入 collocation units。`tools/deai_reference.py:210` | | `test_edit_meta_in_visible_labels` | heading、caption、table cell 中的 `TODO` 和跨行 revision phrase 必须被正确扫描。当前 literal 测试没有这些上下文。`tests/test_deai_residue.py:72` | -| `test_diff_matches_objects_not_shared_stems` | `correction → without saddle correction` 不得声称删除了 saddle object；旧 negation 仅改 caption 颜色不算新增 negation。`tests/test_deai_residue.py:128` | +| `test_diff_matches_objects_not_shared_stems` | `correction → without dust correction` 不得声称删除了 dust object；旧 negation 仅改 caption 颜色不算新增 negation。`tests/test_deai_residue.py:128` | | `test_no_sections_and_empty_measurement` | sectionless history 能扫描；没有 eligible sentences 的 calibrated axis 不得返回无说明的 measured；LF/CRLF 结果一致。`tools/deai_reference.py:162`、`tools/deai_collocation.py:192` | | `test_offsets_survive_multiline_markup` | §2.12 分别断言 L5、L4；abstract 正文和 acronym 返回真实行；include finding 返回 child path/line。现有 line test 未覆盖这些场景。`tests/test_deai_register.py:271` | | `test_definition_only_exempts_defined_term` | `We define flux using quuxification` 不能给后者 defined-here；Unicode 名称不切成伪词。`tests/test_deai_register.py:185`、`tools/deai_register.py:76` | @@ -215,7 +215,7 @@ Read-only review of `git diff 9f0b920..edd480c`; session `01a06ebe-b4c5-7802-bfe |---|---| | “The only mechanical exemption … formation of an attested stem”。`docs/architecture/evaluation/vocabulary-and-residue.md:19`；standard 同样只列 stem。`docs/SCIPAPER_STANDARD.md:160` | 还自动返回 `"defined-here"`、`"name"`。这符合本次用户给出的意图，应该修文档，同时收紧定义对象识别。`tools/deai_register.py:447` | | “This paper presents” 属于 paper-as-agent。`CHANGELOG.md:85`、`docs/SCIPAPER_STANDARD.md:181` | regex 没有 `presents`；测试明确名为 `test_paper_that_merely_presents_is_not_an_agent`。`tools/deai_structure.py:59`、`tests/test_deai_structure.py:79` | -| “three-plus tokens **before a head noun**”。`docs/SCIPAPER_STANDARD.md:183` | `MODIFIER_STACK_RUN=3` 检查的是含 head 的 phrase 长度；`non-compensated 500-configuration subfamily` 仅两个 pre-modifiers 已足够。`tools/deai_structure.py:80`、`tools/deai_structure.py:113` | +| “three-plus tokens **before a head noun**”。`docs/SCIPAPER_STANDARD.md:183` | `MODIFIER_STACK_RUN=3` 检查的是含 head 的 phrase 长度；`non-weighted 200-image stack` 仅两个 pre-modifiers 已足够。`tools/deai_structure.py:80`、`tools/deai_structure.py:113` | | restatement：“≥80% … another section (**or** ≥60% … one sentence)”。`skills/condense/SKILL.md:84` | 实际为 `union >= .80 and best >= .60`，且没有要求另一 section。`tools/condense_map.py:137`、`tools/condense_map.py:144` | | “outside the abstract/conclusion carve-out”。`docs/SCIPAPER_STANDARD.md:575` | zero-gain finding 不带 `genre_carve_out`，budget 将缺失值视为未豁免。`tools/condense_map.py:183`、`tools/condense_map.py:342` | | “Comments and mathematics do not count … rendered prose”。`docs/SCIPAPER_STANDARD.md:584` | `latex_to_plain(...).split()` 将 `[math]`、`[MATH]`、`[FIGURE-OR-TABLE]` 计为词。`tools/length_gate.py:45`、`tools/extract_sections.py:195` | diff --git a/docs/audits/full-review-2026-09-27.md b/docs/audits/full-review-2026-09-27.md index ea72872..a945733 100644 --- a/docs/audits/full-review-2026-09-27.md +++ b/docs/audits/full-review-2026-09-27.md @@ -56,7 +56,7 @@ | E7 | `parse_args(["--field","wgl","--corpus-root","/tmp/x","sample"])`（label_findings 的构造方式） | `field=None`，`corpus_root` 为默认值 | 子命令默认值覆盖根解析结果 | | C12 | `ref.quantiles(list(range(100)))["0.9"]`；n=30 | 90（P=0.91）；`q[0.1]`=3（P=0.133）、`q[0.9]`=27（P=0.933） | 网格偏一位 | | C3 | `\section{Appendix}\nTODO fix this later.` 经 `body_lines` 后 `edit_meta_findings` | `[]`；同文本放在 `\section{Data availability}` 下 → `['residue-edit-meta']` | `skip` 段落里的编辑标记不可见 | -| C4 | `res.RE_LABEL.findall("\section {No saddle}")` | `[]` | 上次审计 A9 未落到 residue | +| C4 | `res.RE_LABEL.findall("\section {No dust}")` | `[]` | 上次审计 A9 未落到 residue | | C13 | `ref.units("无标题散文")`、`ref.units("\section{Weak lensing}…")` | bucket 均为 `'unknown'` | 逐 bucket 轴无此 bucket，整篇跳过 | | D1 | 读 `fetch_arxiv_abstracts._candidate_ids` 本地捷径条件 | `if jsonl.exists() and not exclude and not args.author:`，不看 `journals` | `--journals` 静默失效 | | D2 | 读 `:399`/`:403` | `range(args.start_at, …)` 配 `min(args.page, args.per_query - start)` | `--start-at 2000` 请求 `max_results=-1600` | diff --git a/skills/de-ai/SKILL.md b/skills/de-ai/SKILL.md index 7d4c875..66834c5 100644 --- a/skills/de-ai/SKILL.md +++ b/skills/de-ai/SKILL.md @@ -146,7 +146,7 @@ Axes and tools: exemption. A strong hit is answered by one of the §4.2 dispositions, never by silence. - **L2 collocation** (`deai_collocation`) — sentences that join common words - the field never joins (`physical cells`, `controlled grid`): the fraction of a + the field never joins (`binned kernel`, `smoothed boundary`): the fraction of a sentence's adjacent content-word pairs that no corpus passage attests, against a per-section leave-one-out reference. A pair the field writes by chance would have been seen; a pair it never wrote is a coinage or a figure of speech, and diff --git a/skills/paper-review/SKILL.md b/skills/paper-review/SKILL.md index eb636c7..c33536d 100644 --- a/skills/paper-review/SKILL.md +++ b/skills/paper-review/SKILL.md @@ -144,7 +144,7 @@ uncertainty、多重比较、prior）与**声明-证据纪律**（动词强度 阐释式冒号（linter `ing-tail` / `colon-elaboration`）是 advisory。 - 词汇层两条证据来自语料而非词表：`register-zero:`（全文逐词零命中审计， strong 除非是已证实词干的机械派生）与 `collocation-novel:
`（一句中相邻 - 常用词对在语料里从未同现的比例，如 `physical cells`）。零命中 strong 必须走 + 常用词对在语料里从未同现的比例，如 `binned kernel`）。零命中 strong 必须走 de-ai §4.2 的三种 disposition 之一（本文定义 / 首次引入并引用 / 换成领域词）。 - `deai_structure` 的 auxiliary 家族（paper-as-agent 主语、wh-cleft 开头、三词以上 且含两个连字符复合的修饰堆）只命名句子，不进 template score；命中是 advisory。 From 6aaf9f3de063fb5b1d5f80b8d26856e8caf46bf5 Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 12:27:17 -0400 Subject: [PATCH 23/27] audit remainder in code: the document scale cuts prose, one owner per fact, the L0 mirror checked Audit 2026-09-27, the items the first tier and d42ef14 left open: - B22: deai_docshape.shape_units blanks headings and floats before the paragraph cut (a heading fused into its section's first block; a float holding a blank line split its paragraph in two). deai_partition keeps what the blanking took out as fixed, unscored blocks. - B24: the in-sample percentile fallback for a manifold without split-conformal calibration is gone from deai_docshape, deai_docstructure, deai_partition and eval_docscale. The axis reports degraded with a rebuild reason (deai_docstructure.uncalibrated_manifold_reason), and deai_partition prints that reason with status unmeasured instead of "not measurable". - Hedging runs on `intro` and `method` (deai_discourse AXES), as the 2026-09-27 re-measurement of separation and transfer supports. - H8: ai_ism_lint.validator_check holds the paper skill's Tier A table, opener row and Tier B sentence to the four L0 patterns in both directions; validate_plugin's skills check runs it (still 11 checks), and test discovery runs once per validator process (F29). - E23: extract_sections.bundle_documents is the one bundle loader both evaluators read. E24: register findings once per document, math density once per audit, one cross_validate pass, and the linter measures the document shape and the anchoring once per run. E25: voice_audit uses eval_docscale.rank_auc. E26: voice_audit says what it fits. E29: train_voice_model re-exports no private helper. - A22-A25: aliases that only forwarded are removed, condense_map and deai_residue take cli_common.report_options, and the tests share one LaTeX bundle writer (tests/_texbundle.py). - D17/D18: the dossier's no-sections hint joins the `unknown` table, and its header names the source files the run skipped. D21/D23/D25: unused constants and imports go, VALID_SECTIONS is CLASSIFIED_BUCKETS, and extract_md_negatives drops a local path and states that it overwrites. - The projection's placeholder vocabulary has one owner (PLAIN_INLINE, PLACEHOLDER_SHAPE); the arXiv API is queried over HTTPS with one user agent. - Tests: B30 (the conformal path end to end), D27 (a PDF through a stand-in pymupdf), D28/D29, F30/F31/F37/F38, and one for each behaviour above. 801 tests pass; the validator's 11 checks pass once the recorded counts follow in the documentation commit. Co-Authored-By: Claude Opus 5.5 --- tests/_texbundle.py | 26 +++++ tests/test_ai_ism_lint_cli.py | 142 +++++++++++++++++++++++++--- tests/test_deai_anchoring.py | 14 +++ tests/test_deai_discourse.py | 31 +++--- tests/test_deai_docstructure.py | 113 ++++++++++++++++++++++ tests/test_deai_partition.py | 44 ++++++++- tests/test_deai_structure.py | 33 +++++++ tests/test_eval_docscale.py | 23 +---- tests/test_extract_style.py | 70 +++++++------- tests/test_extractor_statistics.py | 93 ++++++++++++++++++ tests/test_fetch_arxiv_abstracts.py | 19 ++-- tests/test_published_figures.py | 32 ++++--- tests/test_tex_assembly.py | 33 ++----- tests/test_train_voice_model.py | 29 +++--- tools/ai_ism_lint.py | 108 ++++++++++++++++++++- tools/build_profile.py | 13 +-- tools/condense_map.py | 3 +- tools/deai_anchoring.py | 17 ++-- tools/deai_discourse.py | 46 +++++---- tools/deai_docshape.py | 36 +++---- tools/deai_docstructure.py | 98 ++++++++++--------- tools/deai_partition.py | 125 ++++++++++++------------ tools/deai_reference.py | 2 +- tools/deai_residue.py | 7 +- tools/deai_salience.py | 4 +- tools/eval_docscale.py | 27 ++---- tools/eval_findings.py | 47 ++++----- tools/extract_md_negatives.py | 12 +-- tools/extract_sections.py | 30 ++++-- tools/extract_style.py | 44 ++++----- tools/fetch_arxiv_abstracts.py | 8 +- tools/label_findings.py | 13 +-- tools/length_gate.py | 7 +- tools/retrieve_exemplars.py | 10 +- tools/train_ai_ism_classifier.py | 31 +++--- tools/train_voice_model.py | 15 +-- tools/validate_plugin.py | 8 +- tools/voice_audit.py | 57 +++++------ 38 files changed, 1008 insertions(+), 462 deletions(-) create mode 100644 tests/_texbundle.py diff --git a/tests/_texbundle.py b/tests/_texbundle.py new file mode 100644 index 0000000..2605cff --- /dev/null +++ b/tests/_texbundle.py @@ -0,0 +1,26 @@ +"""A throwaway LaTeX source bundle: named `.tex` bodies written to one directory. + +The root-selection tests (`test_extract_style`) and the include-assembly tests +(`test_tex_assembly`) carried four copies of this writer between them by +2026-09-27. The name starts with an underscore so `unittest discover` never +collects it. +""" + +from __future__ import annotations + +from pathlib import Path + + +def write_bundle(tmp: str, files: dict[str, str]) -> Path: + """Write each `name: body` under `/bundle/` and return that directory. + + A name may carry a subdirectory (`sections/intro.tex`), as an unflattened + arXiv source does; its parent directories are created. + """ + directory = Path(tmp) / "bundle" + directory.mkdir(parents=True, exist_ok=True) + for name, body in files.items(): + path = directory / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body, encoding="utf-8") + return directory diff --git a/tests/test_ai_ism_lint_cli.py b/tests/test_ai_ism_lint_cli.py index ba6df9d..b2b2369 100644 --- a/tests/test_ai_ism_lint_cli.py +++ b/tests/test_ai_ism_lint_cli.py @@ -1,11 +1,17 @@ from __future__ import annotations import json +import re import subprocess import sys import tempfile import unittest from pathlib import Path +from unittest import mock + +from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path + +import ai_ism_lint as lint ROOT = Path(__file__).resolve().parents[1] LINTER = ROOT / "tools" / "ai_ism_lint.py" @@ -355,16 +361,7 @@ def test_markdown_front_matter_and_rules_are_not_em_dashes(self): class DocumentAssemblyTests(unittest.TestCase): - """The lint path must read the same document the corpus path reads. - - `extract_sections.latex_to_plain` promises that its projections "can never - drift apart in how they treat comments". The L0 lexical scan is a third - view -- it must see `---` as authored, so it can use neither projection -- - and it sat outside that promise, reading raw file text. On a real - manuscript that made 3 of 3 em-dash targets false, every one a - `% --- lane A: ...` comment rule, and reading only the root file measured - 678 of 23,505 words while four axes still reported `measured`. - """ + """The lint path measures `ai_ism_lint.document_source`; its docstring says why.""" def lint(self, files: "dict[str, str]"): with tempfile.TemporaryDirectory() as temporary: @@ -381,7 +378,7 @@ def lint(self, files: "dict[str, str]"): def test_an_em_dash_inside_a_latex_comment_is_not_a_target(self): result = self.lint({"main.tex": - "% --- lane A: all from notes/v19.json, run 2026-07-09 ---\n" + "% --- calibration block: values from run 3 ---\n" "\\section{Methods}\nThe shear catalog is measured.\n"}) self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertIn("L0=0", result.stdout) @@ -401,5 +398,128 @@ def test_an_included_file_is_measured_not_skipped(self): self.assertNotIn("L0=0", result.stdout) +class LegacyClassifierTests(unittest.TestCase): + """`--ai-classifier` (the L3 legacy axis), with a stub in place of the joblib.""" + + TEXT = ("\\section{Methods}\nWe fit the model to the binned counts.\n\n" + "The residuals stay within the quoted errors.\n") + + class Stub: + """A fitted classifier that gives every paragraph one probability.""" + + classes_ = [0, 1] + + def __init__(self, probability: float): + self.probability = probability + + def predict_proba(self, paragraphs): + return [[1.0 - self.probability, self.probability] for _ in paragraphs] + + def lexical(self, loaded: dict, **options): + """(L3 findings, L3 axis status or None, the patched loader).""" + with mock.patch.object(lint, "load_ai_classifier", **loaded) as loader: + findings, axes = lint.lexical_findings(self.TEXT, Path("d.tex"), None, **options) + status = {axis["axis"]: axis for axis in axes}.get("L3.legacy_classifier") + return [f for f in findings if f["layer"] == "L3"], status, loader + + def test_a_paragraph_at_or_above_the_threshold_is_a_degraded_advisory(self): + legacy, status, _ = self.lexical({"return_value": self.Stub(0.9)}, ai_classifier=True) + self.assertEqual(status["status"], "degraded") + self.assertEqual([(f["location"]["start_line"], f["location"]["end_line"]) + for f in legacy], [(1, 2), (4, 4)]) + for finding in legacy: + self.assertEqual((finding["kind"], finding["rule"], finding["measurement_status"]), + ("advisory", "legacy-classifier-high", "degraded")) + self.assertEqual(finding["observed"], {"legacy_similarity_score": 0.9}) + self.assertEqual(finding["reference"]["user_threshold"], 0.7) + self.assertAlmostEqual(finding["normalized_distance"], 0.2) + # The test is `score < threshold`, so a score at the threshold is flagged. + at_threshold, _, _ = self.lexical({"return_value": self.Stub(0.5)}, + ai_classifier=True, ai_threshold=0.5) + self.assertEqual(len(at_threshold), 2) + + def test_a_score_below_the_threshold_leaves_the_axis_degraded_and_silent(self): + legacy, status, _ = self.lexical({"return_value": self.Stub(0.5)}, ai_classifier=True) + self.assertEqual((legacy, status["status"]), ([], "degraded")) + + def test_no_classifier_is_unmeasured(self): + for loaded, reason in ( + ({"return_value": None}, "ai_ism_classifier.joblib is unavailable"), + ({"side_effect": ImportError("No module named 'joblib'")}, "joblib")): + legacy, status, _ = self.lexical(loaded, ai_classifier=True) + self.assertEqual((legacy, status["status"]), ([], "unmeasured")) + self.assertIn(reason, status["reason"]) + + def test_with_the_option_off_no_legacy_axis_is_reported_or_loaded(self): + legacy, status, loader = self.lexical({"return_value": self.Stub(0.9)}) + self.assertEqual((legacy, status), ([], None)) + loader.assert_not_called() + + +class SkillMirrorTests(unittest.TestCase): + """skills/paper/SKILL.md mirrors the four L0 patterns and the validator + holds it to them both ways (audit H8): `paved` and `showcased` once sat in + the skill's table while the linter passed them.""" + + SKILL = (ROOT / "skills" / "paper" / "SKILL.md").read_text(encoding="utf-8") + + def failures(self, skill_text: str | None = None) -> list[str]: + """`validator_check`'s failure messages on a repository holding `skill_text`.""" + messages: list[str] = [] + + def require(condition, message): + if not condition: + messages.append(message) + + with tempfile.TemporaryDirectory() as temporary: + skill = Path(temporary) / "skills" / "paper" / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text(self.SKILL if skill_text is None else skill_text, + encoding="utf-8") + lint.validator_check(Path(temporary), require) + return messages + + def test_the_repository_passes_its_own_check(self): + self.assertEqual(self.failures(), []) + + def test_a_form_the_skill_drops_is_reported(self): + for name, listed, shorter, form in ( + ("TIER_A_PATTERN", "`pave / paves / paved / paving`", + "`pave / paves / paving`", "paved"), + ("TIER_A_OPENER_PATTERN", "`It is worth noting`, ", "", "it is worth noting"), + ("TIER_A_PARAGRAPH_CONNECTOR_PATTERN", ", `Notably,`", "", "Notably,"), + ("TIER_B_PATTERN", "`intricate`, ", "", "intricate")): + with self.subTest(name): + self.assertEqual(self.SKILL.count(listed), 1) + failures = self.failures(self.SKILL.replace(listed, shorter)) + self.assertEqual(len(failures), 1, failures) + self.assertIn(f"mirror of {name} ", failures[0]) + self.assertIn(f"matched but not listed=[{form!r}]", failures[0]) + + def test_a_form_the_linter_drops_is_reported(self): + for name, alternative, form in ( + ("TIER_A_PATTERN", "paved|", "paved"), + ("TIER_A_OPENER_PATTERN", "|it is worth noting", "It is worth noting"), + ("TIER_A_PARAGRAPH_CONNECTOR_PATTERN", "|Notably", "Notably,"), + ("TIER_B_PATTERN", "intricate|", "intricate")): + with self.subTest(name): + pattern = lint.MIRRORED[name] + self.assertEqual(pattern.pattern.count(alternative), 1) + narrower = re.compile(pattern.pattern.replace(alternative, ""), pattern.flags) + with mock.patch.dict(lint.MIRRORED, {name: narrower}): + failures = self.failures() + self.assertEqual(len(failures), 1, failures) + self.assertIn(f"mirror of {name} ", failures[0]) + self.assertIn(f"listed but not matched=[{form!r}]", failures[0]) + + def test_a_pattern_language_is_its_forms_without_their_context(self): + self.assertEqual(lint.pattern_language(re.compile(r"(?i)\bfoster(?:s|ing|ed)?\b")), + {"foster", "fosters", "fostering", "fostered"}) + self.assertEqual(lint.pattern_language(lint.TIER_A_PARAGRAPH_CONNECTOR_PATTERN), + {"Importantly,", "Interestingly,", "Notably,", "Crucially,"}) + with self.assertRaises(ValueError): # an unbounded pattern has no finite mirror + lint.pattern_language(re.compile(r"\bdelv\w+")) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_deai_anchoring.py b/tests/test_deai_anchoring.py index 82f0bf6..ad8149f 100644 --- a/tests/test_deai_anchoring.py +++ b/tests/test_deai_anchoring.py @@ -6,6 +6,7 @@ import tempfile import unittest from pathlib import Path +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -236,6 +237,19 @@ def test_read_path_without_a_field_is_unmeasured_not_a_traceback(self): "--profile-root", str(empty)]) self.assertEqual((code, "file not found" in err), (2, True)) + def test_the_cli_measures_the_document_once(self): + # The findings and the axis status each ran the sweep (audit B22). + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text(document({"Results": UNANCHORED}), encoding="utf-8") + (root / "profiles").mkdir() + with mock.patch.object(anchoring, "document_anchoring", + wraps=anchoring.document_anchoring) as measure: + code, _out, _err = self.run_main([str(draft), "--profile-root", + str(root / "profiles")]) + self.assertEqual((code, measure.call_count), (0, 1)) + def test_calibrate_needs_an_existing_field(self): with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) diff --git a/tests/test_deai_discourse.py b/tests/test_deai_discourse.py index e1d1369..bd2fd1f 100644 --- a/tests/test_deai_discourse.py +++ b/tests/test_deai_discourse.py @@ -296,34 +296,43 @@ def test_no_bank_at_all_means_nothing_calibrated(self): class TestBucketRestriction(unittest.TestCase): - """Hedging is calibrated for introductions only, and has to say so.""" + """Hedging is calibrated for introductions and methods only, and has to say so.""" def test_hedging_declares_its_restriction_and_cohesion_declares_none(self): - self.assertEqual(discourse.AXES["hedging"]["buckets"], ("intro",)) + self.assertEqual(discourse.AXES["hedging"]["buckets"], ("intro", "method")) self.assertIsNone(discourse.AXES["cohesion"]["buckets"]) def test_a_resolving_bucket_outside_the_restriction_stays_silent(self): - with fixture.temp_profile(graded_bank(section="method"), + with fixture.temp_profile(graded_bank(section="results"), prefix="discourse-") as profile: discourse.calibrate(profile) # The reference resolves; the restriction is what silences it. baseline = discourse.LOADERS["hedging"](profile) self.assertTrue( - discourse.resolves_below_gate(baseline["method"], "hedging")) + discourse.resolves_below_gate(baseline["results"], "hedging")) self.assertEqual(discourse.live_buckets("hedging", profile), []) - self.assertEqual(discourse.live_buckets("cohesion", profile), ["method"]) + self.assertEqual(discourse.live_buckets("cohesion", profile), ["results"]) - def test_status_and_detection_agree_on_which_buckets_are_live(self): + def test_a_method_reference_speaks_for_hedging(self): with fixture.temp_profile(graded_bank(section="method"), prefix="discourse-") as profile: discourse.calibrate(profile) + self.assertEqual(discourse.live_buckets("hedging", profile), ["method"]) + rules = {f["rule"] for f in discourse.discourse_findings( + document(POOR, "Methods"), profile)} + self.assertIn("discourse-hedging:method", rules) + + def test_status_and_detection_agree_on_which_buckets_are_live(self): + with fixture.temp_profile(graded_bank(section="results"), + prefix="discourse-") as profile: + discourse.calibrate(profile) statuses = {s["axis"]: s for s in discourse.discourse_axis_status(profile)} self.assertEqual(statuses["L2.hedging"]["status"], "degraded") rules = {f["rule"] for f in discourse.discourse_findings( - document(POOR, "Methods"), profile)} - self.assertNotIn("discourse-hedging:method", rules) - self.assertIn("discourse-cohesion:method", rules) + document(POOR, "Results"), profile)} + self.assertNotIn("discourse-hedging:results", rules) + self.assertIn("discourse-cohesion:results", rules) class TestLocalReference(unittest.TestCase): @@ -348,8 +357,8 @@ def test_both_axes_are_measured(self): self.assertEqual(statuses["L2.cohesion"], "measured") self.assertEqual(statuses["L2.hedging"], "measured") - def test_hedging_is_live_for_introductions_only(self): - self.assertEqual(discourse.live_buckets("hedging", self.PROFILE), ["intro"]) + def test_hedging_is_live_for_introductions_and_methods_only(self): + self.assertEqual(discourse.live_buckets("hedging", self.PROFILE), ["intro", "method"]) def test_an_unlinked_flat_introduction_is_flagged_on_both_axes(self): rules = {f["rule"] for f in discourse.discourse_findings( diff --git a/tests/test_deai_docstructure.py b/tests/test_deai_docstructure.py index 7b5a638..538e509 100644 --- a/tests/test_deai_docstructure.py +++ b/tests/test_deai_docstructure.py @@ -2,6 +2,7 @@ import contextlib import io +import itertools import json import pathlib import sys @@ -473,6 +474,118 @@ def test_a_tail_the_reference_cannot_resolve_is_not_strong(self): return_value=self._shape(0.35)): self.assertIn("needs 20", docstructure.docstructure_axis_status("x", root)["reason"]) + def test_a_manifold_without_a_conformal_block_is_degraded_not_scored(self): + # The in-sample fallback that scored such a baseline went with audit + # B24, since no calibrate writes one. Hand-built, it reads degraded with + # a rebuild instruction: no crash, and no finding passed off as scored. + names = docstructure.DISPERSION_FEATURE_NAMES + rows = [{name: 1.0 + 0.1 * ((index * (k + 2)) % 7) for k, name in enumerate(names)} + for index in range(docstructure.MIN_MANIFOLD_DOCUMENTS)] + baseline = {"schema": "sci-paper.docstructure-baseline.v2", "n_documents": 40, + "strong_percentile": 0.95, "metrics": {}, "dispersion": {}, + "dispersion_manifold": docstructure.fit_dispersion_manifold(rows)} + self.assertIsNone(docstructure.manifold_operating_point(baseline, rows[0], 6)) + text = document([REPEATED_PARAGRAPH]) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / docstructure.BASELINE_NAME).write_text(json.dumps(baseline), + encoding="utf-8") + self.assertEqual(docstructure.document_findings(text, root), []) + status = docstructure.docstructure_axis_status(text, root) + self.assertEqual(status["status"], "degraded") + self.assertIn("deai_docstructure --calibrate", status["reason"]) + + +class BlankedSweepTests(unittest.TestCase): + """Headings and floats are blanked before the paragraph split, as the + anchoring sweep blanks them, and the CLI measures once (audit B22).""" + + VARIED = [REPEATED_PARAGRAPH, SHORT_PARAGRAPH, LONG_PARAGRAPH, + SHORT_PARAGRAPH, LONG_PARAGRAPH, REPEATED_PARAGRAPH] + + def assert_same_shape(self, first: str, second: str) -> None: + one, two = docstructure.document_shape(first), docstructure.document_shape(second) + for key in ("n_paragraphs", "metrics", "dispersion"): + self.assertEqual(one[key], two[key], key) + + def test_a_heading_does_not_fuse_into_the_first_block(self): + spaced = document(self.VARIED) + fused = spaced.replace("}\n\n", "}\n") # no blank line under any heading + start, _end, block = docstructure.shape_units(fused)[0]["blocks"][0] + self.assertEqual(start, 2) + self.assertNotIn("Introduction", block) + self.assert_same_shape(fused, spaced) + + def test_a_float_does_not_split_a_paragraph(self): + head, tail = LONG_PARAGRAPH.split("; ", 1) + figure = ("\\begin{figure}\n\\includegraphics{map}\n\n" + "\\caption{The smoothed map behind every count in this section.}\n" + "\\end{figure}\n") + plain = document([REPEATED_PARAGRAPH, head + ";\n" + tail] + self.VARIED[2:]) + with_float = plain.replace(head + ";\n", head + ";\n" + figure) + blocks = docstructure.shape_units(with_float)[0]["blocks"] + self.assertEqual(len(blocks), 2) + start, end, block = blocks[1] + self.assertEqual(end - start, 1 + figure.count("\n")) + self.assertNotIn("caption", block) + self.assert_same_shape(with_float, plain) + + def test_the_cli_measures_the_shape_once(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + draft = root / "draft.tex" + draft.write_text(document(self.VARIED), encoding="utf-8") + (root / "profiles").mkdir() + with mock.patch.object(docstructure, "document_shape", + wraps=docstructure.document_shape) as measure, \ + contextlib.redirect_stdout(io.StringIO()): + code = docstructure.main([str(draft), "--profile-root", str(root / "profiles")]) + self.assertEqual((code, measure.call_count), (0, 1)) + + +class ConformalEndToEndTests(unittest.TestCase): + """Calibrate -> findings through the split-conformal and Mondrian paths + on generated documents (audit B30). The manifold is fit on int(0.6 n) + documents and needs MIN_MANIFOLD_DOCUMENTS (33) of them, so 55 is the + smallest corpus that writes a `conformal` block; a length stratum gets a + manifold of its own only with 33 training and 30 calibration documents in + it, which 75 documents of one length give.""" + + def test_calibrate_then_score_on_the_stratum_and_the_pooled_manifold(self): + fraction = docstructure.CONFORMAL_TRAIN_FRACTION + self.assertEqual([int(n * fraction) >= docstructure.MIN_MANIFOLD_DOCUMENTS + for n in (54, 55)], [False, True]) + pairs = itertools.permutations((REPEATED_PARAGRAPH, SHORT_PARAGRAPH, LONG_PARAGRAPH), 2) + combos = itertools.islice(itertools.product(list(pairs), repeat=3), 75) + corpus = [(f"generated-{index}", document([p for pair in combo for p in pair])) + for index, combo in enumerate(combos)] + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + conformal = docstructure.calibrate(corpus, root)["conformal"] + manifold = conformal["manifold"] + # six paragraphs each: one length, so every document is stratum 0 + self.assertEqual(conformal["strata_edges"], [6.0, 6.0]) + self.assertEqual((manifold["n_train"], len(manifold["calibration"])), (45, 30)) + self.assertEqual(sorted(manifold["stratum"]), ["0"]) + # a uniform document of the calibrated length is scored on the + # stratum's own manifold; a longer one falls to the pooled one + for per_section, operating_point, basis in ( + (2, "split-conformal, stratum manifold", "stratum 0 manifold"), + (3, "split-conformal", "pooled")): + text = "\n\n".join( + f"\\section{{{name}}}\n\n" + "\n\n".join([REPEATED_PARAGRAPH] * per_section) + for name in ("Introduction", "Methods", "Results")) + "\n" + found = [f for f in docstructure.document_findings(text, root) + if f["rule"] == "document-dispersion-manifold"] + self.assertEqual(len(found), 1, per_section) + reference = found[0]["reference"] + self.assertEqual((reference["operating_point"], reference["calibration_basis"], + reference["n_calibration"]), (operating_point, basis, 30)) + # beyond all 30 calibration papers: the smallest p they allow + self.assertAlmostEqual(found[0]["observed"]["p_value"], 1 / 31) + self.assertEqual(docstructure.docstructure_axis_status(text, root)["status"], + "measured") + class FeatureRuntimeTests(unittest.TestCase): """`deai_features` without its optional dependencies: unmeasured markers diff --git a/tests/test_deai_partition.py b/tests/test_deai_partition.py index 06a9e7a..4efe108 100644 --- a/tests/test_deai_partition.py +++ b/tests/test_deai_partition.py @@ -2,6 +2,7 @@ import contextlib import io +import json import tempfile import unittest from pathlib import Path @@ -155,6 +156,23 @@ def test_any_heading_command_fixes_its_block(self): self.assertFalse(section["blocks"][op["block"]]["fixed"]) self.assertFalse(section["blocks"][op["block"] + 1]["fixed"]) + def test_what_the_blanking_removes_stays_a_fixed_unscored_landmark(self): + # The measured blocks no longer carry headings or floats (audit B22). + # A heading sharing its line with a \label keeps its block fixed, and a + # float between two paragraphs is a landmark: no merge across it, and + # the scored blocks are exactly `shape_units`' own. + text = ("\\section{Methods}\\label{sec:m}\n" + LENSING_A + "\n\n" + "\\begin{figure}\n\\caption{A map.}\n\n\\end{figure}\n\n" + + LENSING_B + "\n\n" + LENSING_B + "\n") + section, = partition._parse(text) + self.assertEqual([(b["lines"], b["fixed"], b.get("landmark", False)) + for b in section["blocks"]], + [((1, 2), True, False), ((4, 7), True, True), + ((9, 9), False, False), ((11, 11), False, False)]) + self.assertEqual([op["block"] for op in partition._merge_candidates([section], 0.0)], [2]) + self.assertEqual([b["lines"] for b in section["blocks"] if not b.get("landmark")], + [(start, end) for start, end, _ in ds.shape_units(text)[0]["blocks"]]) + def fake_operating_point(p_by_n: dict, distance_by_n: dict): """A stand-in for `manifold_operating_point` keyed by paragraph count, @@ -183,9 +201,6 @@ def test_improves_orders_by_p_then_by_distance_within_one_basis(self): "calibration_basis": "stratum 1 manifold"} self.assertTrue(partition._improves(same_basis_closer, worse_p)) self.assertFalse(partition._improves(other_basis_closer, worse_p)) - # a legacy baseline (no conformal block) compares pooled distances - self.assertTrue(partition._improves({"distance": 1.0, "calibration_basis": "in-sample percentile"}, - {"distance": 2.0, "calibration_basis": "in-sample percentile"})) def test_plan_prefers_the_higher_p_and_records_the_basis(self): text = uniform_document(FOUR_SENTENCES) @@ -249,6 +264,29 @@ def test_without_a_profile_the_tool_exits_2_with_a_message(self): "--profile-root", str(empty)]) self.assertEqual((code, "file not found" in err), (2, True)) + def test_a_manifold_without_conformal_calibration_names_the_rebuild(self): + # It printed "document or manifold not measurable" and exited 0; the + # document axis already said why, in words this tool now shares. + names = ds.DISPERSION_FEATURE_NAMES + rows = [{name: 1.0 + 0.1 * ((index * (k + 2)) % 7) for k, name in enumerate(names)} + for index in range(ds.MIN_MANIFOLD_DOCUMENTS)] + baseline = {"n_documents": 40, "dispersion_manifold": ds.fit_dispersion_manifold(rows)} + text = uniform_document(FOUR_SENTENCES) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / "f").mkdir() + (root / "f" / ds.BASELINE_NAME).write_text(json.dumps(baseline), encoding="utf-8") + draft = root / "draft.tex" + draft.write_text(text, encoding="utf-8") + code, out, _err = self.run_main([str(draft), "--profile-root", str(root), + "--field", "f", "--json", str(root / "plan.json")]) + plan = json.loads((root / "plan.json").read_text(encoding="utf-8")) + reason = ds.docstructure_axis_status(text, root / "f")["reason"] + self.assertEqual(code, 0) + self.assertIn("deai_docstructure --calibrate", reason) + self.assertEqual(out, f"[deai_partition] {reason}\n") + self.assertEqual((plan["status"], plan["reason"]), ("unmeasured", reason)) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_deai_structure.py b/tests/test_deai_structure.py index 3d4add5..b2fb44f 100644 --- a/tests/test_deai_structure.py +++ b/tests/test_deai_structure.py @@ -270,5 +270,38 @@ def test_no_reference_quotes_no_fraction(self): self.assertNotIn("fraction", message) +class CalibrateTests(unittest.TestCase): + """`calibrate` turns the exemplar bank into per-bucket fractions (audit B30).""" + + def test_a_tiny_bank_yields_per_bucket_fractions(self): + rows = [{"section": "method", "text": TEMPLATED}, + {"section": "method", "text": PLAIN_PROSE}, + {"section": "results", "text": ANTITHESIS_HEAVY}, + {"section": "results", "text": "Too short to be a reference paragraph."}] + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + (root / "exemplar_paragraphs.jsonl").write_text( + "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8") + baseline = structure.calibrate(root) + written = json.loads((root / "structure_baseline.json").read_text(encoding="utf-8")) + # the detector reads the written reference back for its bucket + finding, = [f for f in structure.structure_findings( + "\\section{Methods}\n\n" + TEMPLATED, root) + if f["rule"] == "structure-template:method"] + self.assertEqual(written, baseline) + # the paragraph under MIN_WORDS is not a reference observation + self.assertEqual({bucket: entry["n"] for bucket, entry in baseline.items()}, + {"method": 2, "results": 1}) + method, results = baseline["method"], baseline["results"] + self.assertEqual((method["templated_frac"], method["tricolon_frac"], + method["modal_frac"], method["auxiliary_frac"]), (0.5, 0.5, 0.5, 0.0)) + self.assertEqual((results["templated_frac"], results["auxiliary_frac"], + results["antithesis_cluster_frac"]), (0.0, 1.0, 1.0)) + self.assertEqual((finding["reference"]["templated_fraction"], finding["reference"]["n"], + finding["measurement_status"]), (0.5, 2, "degraded")) + with tempfile.TemporaryDirectory() as temporary, self.assertRaises(SystemExit): + structure.calibrate(Path(temporary)) # no bank: a stated failure + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_eval_docscale.py b/tests/test_eval_docscale.py index 2635360..231c7df 100644 --- a/tests/test_eval_docscale.py +++ b/tests/test_eval_docscale.py @@ -11,11 +11,10 @@ def _points(n: int, *, distance: float = 1.0, p_value: float = 0.5, - alpha: float | None = 0.05, - operating_point: str = "split-conformal", **extra) -> list[dict]: + alpha: float = 0.05, operating_point: str = "split-conformal") -> list[dict]: """`n` scored documents as `manifold_operating_point` returns them.""" return [{"distance": distance, "p_value": p_value, "alpha": alpha, - "operating_point": operating_point, "n_paragraphs": 30, **extra} + "operating_point": operating_point, "n_paragraphs": 30} for _ in range(n)] @@ -57,22 +56,13 @@ def _report(rows: dict, alpha: float | None = 0.05) -> dict: class OperatingPointTests(unittest.TestCase): - """A flag is decided by the operating point that scored the document. - - Reading a legacy in-sample percentile against the baseline's conformal - alpha compared a within-sample percentile with a level it was never - calibrated to. - """ + """A flag is decided by the operating point that scored the document.""" def test_a_conformal_point_is_flagged_at_its_own_alpha(self) -> None: self.assertTrue(ed.flagged({"p_value": 0.04, "alpha": 0.05})) self.assertFalse(ed.flagged({"p_value": 0.06, "alpha": 0.05})) self.assertTrue(ed.flagged({"p_value": 0.10, "alpha": 0.10})) - def test_a_legacy_percentile_point_keeps_its_own_decision(self) -> None: - self.assertFalse(ed.flagged({"p_value": 0.03, "alpha": None, "flagged": False})) - self.assertTrue(ed.flagged({"p_value": 0.50, "alpha": None, "flagged": True})) - def test_each_row_names_its_operating_point(self) -> None: rows = ed.summarize({ "human": _points(ed.MIN_DOCUMENTS), @@ -86,13 +76,6 @@ def test_each_row_names_its_operating_point(self) -> None: self.assertIn("split-conformal, stratum manifold", rendered) self.assertIn("alpha=0.05", rendered) - def test_a_legacy_baseline_reports_no_alpha(self) -> None: - rows = ed.summarize({"human": _points( - ed.MIN_DOCUMENTS, alpha=None, operating_point="in-sample percentile", - flagged=False)}) - self.assertEqual(rows["human"]["operating_point"], "in-sample percentile") - self.assertIn("alpha=-", ed.render(RowFloorTests._report(rows, alpha=None))) - class ResolutionTests(unittest.TestCase): def test_no_human_document_is_a_named_exit_not_a_statistics_error(self): diff --git a/tests/test_extract_style.py b/tests/test_extract_style.py index ffcfd91..4ca86da 100644 --- a/tests/test_extract_style.py +++ b/tests/test_extract_style.py @@ -20,6 +20,7 @@ from pathlib import Path from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path +from _texbundle import write_bundle import extract_style as es @@ -286,18 +287,36 @@ class ReExportContractTests(unittest.TestCase): def test_every_public_name_is_re_exported(self): # An imported module is not a name this module defines, and naming each # one to exclude it (`re`, `defaultdict`, `Path`) put the test one - # `import` behind its own subject: adding `tex_macros` failed it. + # `import` behind its own subject: adding `tex_macros` failed it. A name + # borrowed from `tex_assembly` is part of the surface too; excluded by + # its `__module__`, it let two dead private aliases sit here unseen. import types import extract_sections as sections expected = {n for n in vars(sections) if not n.startswith("__") and not isinstance(vars(sections)[n], types.ModuleType) and getattr(vars(sections)[n], "__module__", "extract_sections") - in ("extract_sections", "re", None) - and n != "annotations"} + in ("extract_sections", "tex_assembly", "re", None)} + self.assertIn("read_tex_document", expected) missing = sorted(n for n in expected if not hasattr(es, n)) self.assertEqual(missing, [], f"extract_style does not re-export: {missing}") + def test_a_tex_assembly_name_is_its_own_object_under_its_own_name(self): + # `tex_assembly` owns the include assembler, so what `extract_style` + # re-exports from it is that module's object under that module's name: + # `_include_targets` and `_resolve_include` were aliases of + # `include_targets` and `resolve_include` that no caller used. + import types + import tex_assembly + owned = {id(value) for name, value in vars(tex_assembly).items() + if not name.startswith("__") and not isinstance(value, types.ModuleType)} + borrowed = sorted(name for name, value in vars(es).items() + if not name.startswith("__") and id(value) in owned) + self.assertIn("read_tex_document", borrowed) + aliased = [name for name in borrowed + if getattr(tex_assembly, name, None) is not getattr(es, name)] + self.assertEqual(aliased, [], f"re-exported under another name: {aliased}") + class NumberedHeadingTests(unittest.TestCase): """Journals set section titles in title case behind a number. @@ -337,16 +356,9 @@ class DocumentRootTests(unittest.TestCase): same pseudoreplication the project refuses elsewhere. """ - def _bundle(self, tmp: str, files: dict[str, str]) -> list[Path]: - d = pathlib.Path(tmp) / "bundle" - d.mkdir(parents=True, exist_ok=True) - for name, body in files.items(): - (d / name).write_text(body, encoding="utf-8") - return sorted(d.glob("*.tex")) - def test_included_fragments_are_dropped(self): with tempfile.TemporaryDirectory() as tmp: - tex = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\documentclass{article}" "\n" r"\begin{document}" "\n" r"\include{chap_1}" "\n" @@ -355,41 +367,41 @@ def test_included_fragments_are_dropped(self): "chap_1.tex": r"\section{Results}" "\nfirst\n", "chap_2.tex": r"\section{Discussion}" "\nsecond\n", }) - roots = es.select_document_roots(tex) + roots = es.select_document_roots(sorted(d.glob("*.tex"))) self.assertEqual([p.name for p in roots], ["main.tex"]) def test_unmarked_sibling_of_a_marked_root_is_dropped(self): # bib.tex is neither \input nor marked, but it is not a paper either. with tempfile.TemporaryDirectory() as tmp: - tex = self._bundle(tmp, { + d = write_bundle(tmp, { "ms.tex": r"\documentclass{aastex}" "\n" r"\begin{document}" "\nbody\n" r"\end{document}" "\n", "bib.tex": r"\bibitem{a} A. Author, 2001" "\n", }) - roots = es.select_document_roots(tex) + roots = es.select_document_roots(sorted(d.glob("*.tex"))) self.assertEqual([p.name for p in roots], ["ms.tex"]) def test_a_lone_unmarked_file_is_still_the_paper(self): # Plain-TeX papers predating LaTeX2e carry no document marker at all; # Schneider (1996) in the wgl corpus is one, and it IS the paper. with tempfile.TemporaryDirectory() as tmp: - tex = self._bundle(tmp, { + d = write_bundle(tmp, { "aperture.tex": r"\def\ave#1{\langle #1\rangle}" "\n" r"\section{Introduction}" "\nbody\n", }) - roots = es.select_document_roots(tex) + roots = es.select_document_roots(sorted(d.glob("*.tex"))) self.assertEqual([p.name for p in roots], ["aperture.tex"]) def test_documentstyle_counts_as_a_document_marker(self): # LaTeX 2.09; still used by pre-1995 arXiv sources such as # Kaiser, Squires & Broadhurst (1995) and Refregier (2003). with tempfile.TemporaryDirectory() as tmp: - tex = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\documentstyle[aaspp]{article}" "\nbody\n", "macros.tex": r"\def\x{1}" "\n", }) - roots = es.select_document_roots(tex) + roots = es.select_document_roots(sorted(d.glob("*.tex"))) self.assertEqual([p.name for p in roots], ["main.tex"]) def test_separate_bundles_do_not_shadow_each_other(self): @@ -404,22 +416,16 @@ def test_separate_bundles_do_not_shadow_each_other(self): self.assertEqual(len(roots), 2) -class TexDocumentAssemblyTests(unittest.TestCase): - r"""Dropping a fragment must not drop the prose it holds. +class BundleRootSelectionTests(unittest.TestCase): + """A bundle subdirectory is one paper, and that paper is read once. - `WeakLens.tex` is 72 words of \include calls; the ~40,000-word review it - names lives in the eleven chapter files. Selecting the root and reading - it with `read_text` is as wrong as counting each chapter as its own - paper -- it replaces a twelvefold overcount with a total loss. + Given the source directory as `bundle_root`, each subdirectory under it is + one arXiv submission and keeps only the root with the most classified + prose, because bundles ship the journal's worked example beside the + manuscript; a file lying directly in the source directory stays a paper + of its own. The root kept is read with its includes spliced in once. """ - def _bundle(self, tmp: str, files: dict[str, str]) -> pathlib.Path: - d = pathlib.Path(tmp) / "bundle" - d.mkdir(parents=True, exist_ok=True) - for name, body in files.items(): - (d / name).write_text(body, encoding="utf-8") - return d - def test_a_bundle_yields_one_paper_not_the_journal_template(self): # 27 of the 500 wgl arXiv bundles ship the journal's class # documentation next to the manuscript; mnras_template.tex is a @@ -452,7 +458,7 @@ def test_loose_files_in_the_source_root_stay_independent(self): def test_the_whole_bundle_is_read_exactly_once(self): with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\documentclass{article}" "\n" r"\include{ch}" "\n", "ch.tex": r"\section{Results}" "\nUNIQUE MARKER\n", diff --git a/tests/test_extractor_statistics.py b/tests/test_extractor_statistics.py index b7c69df..820cb57 100644 --- a/tests/test_extractor_statistics.py +++ b/tests/test_extractor_statistics.py @@ -5,10 +5,15 @@ from __future__ import annotations +import io import json import pathlib +import sys import tempfile +import types import unittest +from contextlib import redirect_stderr, redirect_stdout +from unittest import mock from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path @@ -63,6 +68,16 @@ def test_placeholders_are_not_words_and_do_not_drop_a_paragraph(self): text = "[CITE] showed the effect.\n\n[MATH]\n\n[FIGURE-OR-TABLE] and more." self.assertEqual(es.paragraph_initial_words(text), ["showed", "and"]) + def test_every_placeholder_the_projection_writes_is_dropped_by_its_readers(self): + # One vocabulary (`PLAIN_INLINE`, `PLAIN_PLACEHOLDERS`); the three + # patterns that read it were typed out by hand until audit D23. + import deai_reference + for token in (es.PLAIN_INLINE, *es.PLAIN_PLACEHOLDERS.values()): + with self.subTest(token=token): + self.assertRegex(token.strip(), es.RE_PLACEHOLDER) + self.assertEqual(es.without_placeholders(token).strip(), "") + self.assertEqual(deai_reference.RE_PLACEHOLDER_TOKEN.sub("", token).strip(), "") + def test_words_are_unicode_letters(self): self.assertEqual(es.words("naïve Poincaré"), ["naïve", "Poincaré"]) @@ -157,6 +172,84 @@ def write(texts): self.assertEqual(embeddings[:, 0].tolist(), [4.0, 2.0]) +class DossierTests(unittest.TestCase): + """What the dossier says about the corpus behind it (audit D17, D18).""" + + PROSE = ("The sample is small. The method is simple and the result is stable. " + "We repeat the measurement twice.\n") + + def _dossier(self, files: dict[str, str]) -> str: + """Run the extractor over one curated tier holding `files`; return the dossier.""" + with tempfile.TemporaryDirectory() as tmp: + corpus = pathlib.Path(tmp) / "corpus" + tier = corpus / "g" / "tier-1-top" + tier.mkdir(parents=True) + for name, body in files.items(): + (tier / name).write_text(body, encoding="utf-8") + profile = pathlib.Path(tmp) / "profile" + with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): + code = es.main(["--field", "g", "--corpus-root", str(corpus), + "--profile-root", str(profile)]) + self.assertEqual(code, 0) + return (profile / "g" / "style_dossier.md").read_text(encoding="utf-8") + + def test_the_no_sections_hint_fires_when_only_unknown_was_measured(self): + # `.txt` and sectionless `.tex` fill `unknown`, which reaches the + # statistics, so a hint keyed on an empty dict could never fire. + unsectioned = self._dossier({"notes.txt": self.PROSE}) + self.assertIn("No sections detected", unsectioned) + self.assertIn("| unknown |", unsectioned) # the hint adds; the row stays + sectioned = self._dossier({"paper.tex": "\\section{Results}\n" + self.PROSE}) + self.assertNotIn("No sections detected", sectioned) + + def test_the_header_counts_the_files_the_run_skipped(self): + # Without pymupdf a standalone PDF is skipped: the run's summary said + # so, and the dossier's "Built from N corpus papers" did not. + missing = ImportError("pymupdf not installed") + with mock.patch.object(es, "extract_pdf_text", side_effect=missing): + dossier = self._dossier({"notes.txt": self.PROSE, "scan.pdf": "%PDF-1.4\n"}) + header = next(line for line in dossier.splitlines() if line.startswith("Built from")) + self.assertIn("Built from 1 corpus papers", header) + self.assertIn("1 source file(s) SKIPPED (pymupdf unavailable)", header) + + +class PdfPathTests(unittest.TestCase): + """The PDF path end to end on a stand-in `pymupdf` (audit D27): only its + helpers had tests, never `extract_pdf_text` reading a text layer.""" + + def test_ligatures_expand_and_line_fragments_rejoin_into_paragraphs(self): + blocks = [(0, 0, 1, 1, "1. Introduction", 0, 0), + (0, 0, 1, 1, "The first effect is significant, and the", 1, 0), + (0, 0, 1, 1, "inflated estimate follows from it.", 2, 0), + (0, 0, 1, 1, "an image block the text layer must drop", 3, 1), + (0, 0, 1, 1, "2. Results", 4, 0), + (0, 0, 1, 1, "The final figure is stable.", 5, 0)] + + class Page: + def get_text(self, mode): + return blocks if mode == "blocks" else "" + + class Document: + closed = False + + def __iter__(self): + return iter([Page()]) + + def close(self): + Document.closed = True + + fake = types.SimpleNamespace(open=lambda path: Document()) + with mock.patch.dict(sys.modules, {"pymupdf": fake}): + analysis = es.analyse_paper(pathlib.Path("paper.pdf")) + sections = analysis["by_section"] + self.assertEqual(sorted(sections), ["intro", "results"]) + self.assertEqual(sections["intro"]["plain_text"], + "The first effect is significant, and the inflated " + "estimate follows from it.") + self.assertEqual(sections["results"]["plain_text"], "The final figure is stable.") + self.assertIn("significant", sections["intro"]["word_counter"]) + self.assertTrue(Document.closed) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_fetch_arxiv_abstracts.py b/tests/test_fetch_arxiv_abstracts.py index 0ae597e..5391cad 100644 --- a/tests/test_fetch_arxiv_abstracts.py +++ b/tests/test_fetch_arxiv_abstracts.py @@ -13,7 +13,8 @@ from _toolpath import TOOLS # noqa: E402 -- because importing it is what puts tools/ on sys.path import fetch_arxiv_abstracts as fetch # noqa: E402 -from _fetch_harness import ARXIV_NS, ATOM_NS, _bank, _record, _sweep, _write_bank # noqa: E402 +from _fetch_harness import ( # noqa: E402 -- because the harness imports the tool _toolpath exposes + ARXIV_NS, ATOM_NS, _bank, _offline, _record, _sweep, _write_bank) class ClassifyJournalTest(unittest.TestCase): @@ -139,10 +140,6 @@ def test_published_dates_v1_but_updated_dates_the_returned_text(self) -> None: class QuerySetTest(unittest.TestCase): - def test_broad_set_is_the_pre_existing_query_list(self) -> None: - self.assertEqual(fetch.QUERY_SETS["broad"], - fetch.QUERIES + fetch.AUTHOR_QUERIES) - def test_wl_set_is_weak_lensing_only(self) -> None: self.assertTrue(fetch.QUERY_SETS["wl"]) for query in fetch.QUERY_SETS["wl"]: @@ -236,7 +233,6 @@ def test_non_429_http_error_propagates_and_is_not_retried(self) -> None: def test_throttled_is_not_a_generic_exception_subclass_catch(self) -> None: # The abstract sweep distinguishes the two: a hiccup is skipped, a # Throttled stops the sweep. That only works if it is its own type. - self.assertTrue(issubclass(fetch.Throttled, Exception)) self.assertFalse(issubclass(fetch.Throttled, urllib.error.HTTPError)) @@ -395,6 +391,17 @@ def test_writing_a_heldout_set_into_the_calibration_dir_is_refused(self) -> None "--fulltext-dir", fetch.REFERENCE_DIR]) self.assertIn("held-out", captured.getvalue()) + def test_an_author_pull_into_the_calibration_dir_is_refused(self) -> None: + # An author's papers are a population to MEASURE; under the default + # --fulltext-dir the next profile rebuild would calibrate on them. + # Offline, so a missing guard fails here instead of downloading. + with tempfile.TemporaryDirectory() as name, _offline(Path(name)) as (_, err): + with self.assertRaises(SystemExit) as caught: + fetch.main(["--field", "wgl", "--profile-root", name, + "--fulltext", "--author", "Someone"]) + self.assertEqual(caught.exception.code, 2) + self.assertIn("--author writes a population", err.getvalue()) + def test_a_non_calibration_fulltext_dir_is_allowed(self) -> None: # The interlock must key on the calibration directory specifically, not # refuse --exclude-known outright. diff --git a/tests/test_published_figures.py b/tests/test_published_figures.py index 0f7aee8..5554a7b 100644 --- a/tests/test_published_figures.py +++ b/tests/test_published_figures.py @@ -53,11 +53,12 @@ AXIS_TABLE_ORDER = ("abstract", "method", "data", "intro", "discussion", "results", "conclusion") # The per-rule table in examples/README.md, in its order: one rule named with -# its variant, one with its layer, six with neither. Which rules the +# its variant, one with its layer, seven with neither. Which rules the # walkthrough tabulates is the document's choice; every count in it is the # linter's. EXAMPLE_RULE_ROWS = ( ("`discourse-cohesion`", "discourse-cohesion"), + ("`discourse-hedging`", "discourse-hedging"), ("`em-dash` (L0)", "em-dash"), ("`ing-tail:highlighting`", "ing-tail:highlighting"), ("`document-uniformity`", "document-uniformity"), @@ -95,12 +96,6 @@ def bucket_n(name: str) -> dict[str, int]: return {key: value["n"] for key, value in artifact(name).items()} -def by_dot(counts: dict[str, int]) -> str: - """The `name n · name n` ordering §19.1 uses: largest bucket first.""" - return " · ".join(f"{key} {counts[key]:,}" - for key in sorted(counts, key=lambda key: -counts[key])) - - def gate(name: str, feature: str, bucket: str, places: int) -> str: """The feature value at the advisory gate, as §19 prints it.""" reference = artifact(name)[bucket]["percentiles"][feature] @@ -109,7 +104,7 @@ def gate(name: str, feature: str, bucket: str, places: int) -> str: def by_n(counts: dict[str, int]) -> str: - """The `name n · name n` ordering both READMEs use: largest bucket first.""" + """The `name n · name n` ordering both READMEs and §19.1 use: largest first.""" return " · ".join(f"{key} {counts[key]:,}" for key in sorted(counts, key=lambda key: -counts[key])) @@ -245,8 +240,8 @@ def expectations(document: str) -> list[tuple[str, str]]: # error the axis exists to prevent -- so both are pinned to their own # artifact, including the p10 gate each bucket abstains or fires at. return [ - ("cohesion buckets", f"| paragraph | {by_dot(bucket_n('cohesion_baseline.json'))} |"), - ("hedging buckets", f"| section | {by_dot(bucket_n('hedging_baseline.json'))} |"), + ("cohesion buckets", f"| paragraph | {by_n(bucket_n('cohesion_baseline.json'))} |"), + ("hedging buckets", f"| section | {by_n(bucket_n('hedging_baseline.json'))} |"), ("bank size", f"the {bank_size():,}-paragraph `{FIELD}` bank"), ] + [(f"hedging gate: {bucket}", f"| {bucket} | {gate('hedging_baseline.json', 'hedging', bucket, 3)} |") @@ -255,7 +250,6 @@ def expectations(document: str) -> list[tuple[str, str]]: salience = bucket_n("salience_baseline.json") register = artifact("register_lexicon.json") return [ - ("bucket-history total", f"| **{sum(structure.values()):,}** |"), ("axis table: register", f"{register['n_passages']:,} corpus passages, " f"{len(register['document_frequency']):,} terms"), @@ -281,6 +275,9 @@ def expectations(document: str) -> list[tuple[str, str]]: "mean surprisal {} ± {}.".format(*pooled_uid("mean_surprisal"))), ("structure total", f"contains {sum(structure.values()):,} paragraph observations"), + # §5's bank-history table ends on the current build; its total row is + # the artifact's, the earlier columns are history. + ("bucket-history total", f"| **{sum(structure.values()):,}** |"), ] for bucket in BUCKETS: # Section 5's `name n` list wraps across source lines, so the whole @@ -324,7 +321,7 @@ def test_the_uid_and_structure_reference_sections(self) -> None: self.assert_document_carries_its_figures(UID_DOC) @needs_profile - def test_the_evaluation_hub_bucket_table(self) -> None: + def test_the_evaluation_hub_axis_table(self) -> None: self.assert_document_carries_its_figures(EVALUATION_DOC) @needs_profile @@ -350,4 +347,13 @@ def test_an_absent_profile_skips_rather_than_passes(self) -> None: # than no check: it reports agreement it never established. self.assertEqual(HAVE_PROFILE, (PROFILE / "uid_baseline.json").is_file()) - self.assertIs(needs_profile, needs_profile) + + class Probe(unittest.TestCase): + @needs_profile + def test_figure(self) -> None: + """Stands in for any figure case above.""" + + result = unittest.TestResult() + Probe("test_figure").run(result) + self.assertTrue(result.wasSuccessful()) + self.assertEqual(len(result.skipped), 0 if HAVE_PROFILE else 1) diff --git a/tests/test_tex_assembly.py b/tests/test_tex_assembly.py index c0e6d37..e362e9c 100644 --- a/tests/test_tex_assembly.py +++ b/tests/test_tex_assembly.py @@ -14,6 +14,7 @@ import unittest from _toolpath import TOOLS # noqa: F401,E402 -- because importing it is what puts tools/ on sys.path +from _texbundle import write_bundle import extract_style as es import tex_assembly as assembly @@ -28,16 +29,9 @@ class TexDocumentAssemblyTests(unittest.TestCase): paper -- it replaces a twelvefold overcount with a total loss. """ - def _bundle(self, tmp: str, files: dict[str, str]) -> pathlib.Path: - d = pathlib.Path(tmp) / "bundle" - d.mkdir(parents=True, exist_ok=True) - for name, body in files.items(): - (d / name).write_text(body, encoding="utf-8") - return d - def test_included_chapters_are_spliced_in(self): with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\documentclass{article}" "\n" r"\begin{document}" "\n" r"\include{chap_1}" "\n" @@ -54,7 +48,7 @@ def test_commented_out_includes_are_not_spliced(self): # arXiv sources routinely park an alternative build in comments # (`% \includeonly{WeakLens_7}`); resolving those duplicates a chapter. with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\documentclass{article}" "\n" r"% \input{alt}" "\n" "kept\n", "alt.tex": "ALTERNATE\n", @@ -70,7 +64,7 @@ def test_flattened_subdirectory_targets_resolve_to_siblings(self): # dropped as a fragment and never spliced back -- 2101.09097v3 kept # 2 of its 9,743 words that way. with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "article.tex": r"\documentclass{article}" "\n" r"\input{sections/introduction}" "\n" r"\input{sections/forecast.tex}" "\n", @@ -85,7 +79,7 @@ def test_the_selector_and_the_reader_agree_on_targets(self): # The bug was an asymmetry, not a missing feature: whatever # select_document_roots treats as included must be spliceable. with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "article.tex": r"\documentclass{article}" "\n" r"\input{sections/introduction}" "\n", "introduction.tex": "intro body\n", @@ -96,7 +90,7 @@ def test_the_selector_and_the_reader_agree_on_targets(self): def test_missing_include_target_leaves_the_call_in_place(self): with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\documentclass{article}" "\n" r"\input{aa.cls}" "\n" "body\n", }) @@ -105,7 +99,7 @@ def test_missing_include_target_leaves_the_call_in_place(self): def test_include_cycles_terminate(self): with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "a.tex": r"\documentclass{article}" "\n" r"\input{b}" "\n", "b.tex": r"\input{a}" "\n" "cycle body\n", }) @@ -117,7 +111,7 @@ def test_an_include_is_spliced_in_place_and_may_repeat(self): # whole line was replaced by the child, and a second `\input{body}` # returned nothing because "seen anywhere" was mistaken for a cycle. with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": "Before \\input{body} After\n\\input{body} % \\input{alt}\n", "body.tex": "CHILD", "alt.tex": "ALT", @@ -127,16 +121,9 @@ def test_an_include_is_spliced_in_place_and_may_repeat(self): class IncludeResolutionTests(unittest.TestCase): - def _bundle(self, tmp: str, files: dict[str, str]) -> pathlib.Path: - d = pathlib.Path(tmp) - for name, body in files.items(): - (d / name).parent.mkdir(parents=True, exist_ok=True) - (d / name).write_text(body, encoding="utf-8") - return d - def test_a_target_with_a_suffix_is_read_as_written(self): with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, {"main.tex": r"\input{fig.tikz}" "\n", + d = write_bundle(tmp, {"main.tex": r"\input{fig.tikz}" "\n", "fig.tikz": "TIKZ\n", "fig.tex": "WRONG FILE\n"}) self.assertEqual(assembly.read_tex_document(d / "main.tex"), "TIKZ\n\n") @@ -145,7 +132,7 @@ def test_a_nested_include_resolves_against_the_document_root(self): # in; resolved against the child's directory, `sections/table.tex` # shadowed the root-level file the author meant. with tempfile.TemporaryDirectory() as tmp: - d = self._bundle(tmp, { + d = write_bundle(tmp, { "main.tex": r"\input{sections/intro}" "\n", "sections/intro.tex": r"INTRO \input{table}" "\n", "table.tex": "ROOT-LEVEL TABLE", diff --git a/tests/test_train_voice_model.py b/tests/test_train_voice_model.py index b053580..144022f 100644 --- a/tests/test_train_voice_model.py +++ b/tests/test_train_voice_model.py @@ -16,6 +16,8 @@ import deai_features as features import deai_voice as voice import train_voice_model as training +import voice_audit +import voice_dataset def _unbound_names(source: str, filename: str = "") -> list[str]: @@ -67,13 +69,15 @@ def tables(table): class ModuleSplitContractTests(unittest.TestCase): - """`train_voice_model` must re-export everything the split modules define. + """`train_voice_model` must re-export the split modules' public entry points. The dataset and audit layers were split out on 2026-08-26 (1,174 lines against a 750-line budget). The re-export list is hand-written, so it can silently fall behind the modules it mirrors -- the drift the equivalent `extract_style` and `deai_docstructure` tests have caught four times - between them. Every test above reaches these names as `training.`. + between them. The tests below reach the public names as `training.` + and a private helper through its own module, so the list carries no + private name (audit 2026-09-27, E29: it once re-exported twelve). """ def _public_names(self, module, module_name): @@ -84,27 +88,30 @@ def _public_names(self, module, module_name): return { name for name, value in vars(module).items() - if not name.startswith("__") + if not name.startswith("_") and not isinstance(value, types.ModuleType) and getattr(value, "__module__", module_name) == module_name } def test_dataset_names_are_re_exported(self): - import voice_dataset - missing = sorted(n for n in self._public_names(voice_dataset, "voice_dataset") if not hasattr(training, n)) self.assertEqual(missing, [], f"train_voice_model does not re-export: {missing}") def test_audit_names_are_re_exported(self): - import voice_audit - missing = sorted(n for n in self._public_names(voice_audit, "voice_audit") if not hasattr(training, n)) self.assertEqual(missing, [], f"train_voice_model does not re-export: {missing}") + def test_no_private_helper_is_re_exported(self): + leaked = sorted( + name for name, value in vars(training).items() + if name.startswith("_") and not name.startswith("__") + and getattr(value, "__module__", None) in ("voice_dataset", "voice_audit")) + self.assertEqual(leaked, [], f"train_voice_model re-exports private {leaked}") + def test_the_split_modules_stay_within_the_line_budget(self): # The split exists to get under the budget; a test keeps it there. for name in ("train_voice_model.py", "voice_dataset.py", "voice_audit.py"): @@ -321,13 +328,13 @@ def test_bootstrap_auc_ci_reproducible_and_brackets_point(self): self.skipTest("numpy is unavailable") y = [0, 0, 0, 1, 1, 1] scores = [0.1, 0.2, 0.3, 0.7, 0.8, 0.9] - first = training._bootstrap_auc_ci(y, scores, n_boot=500, seed=7) - second = training._bootstrap_auc_ci(y, scores, n_boot=500, seed=7) + first = voice_audit._bootstrap_auc_ci(y, scores, n_boot=500, seed=7) + second = voice_audit._bootstrap_auc_ci(y, scores, n_boot=500, seed=7) self.assertEqual(first, second) # seeded => reproducible self.assertEqual(first["auc"], 1.0) # perfect separation self.assertLessEqual(first["ci95_low"], first["auc"]) self.assertLessEqual(first["auc"], first["ci95_high"] + 1e-9) - self.assertIsNone(training._bootstrap_auc_ci([1, 1], [0.2, 0.9])) # single class + self.assertIsNone(voice_audit._bootstrap_auc_ci([1, 1], [0.2, 0.9])) # single class def test_undefined_f1_is_excluded_from_aggregation(self): report_with_positives = {"overall": training.binary_metrics( @@ -425,7 +432,7 @@ def test_a_blank_line_in_a_bank_is_skipped_not_parsed(self): bank = Path(raw) / "exemplar_paragraphs.jsonl" record = json.dumps({"text": "word " * 30, "source": "s"}) bank.write_text(record + "\n\n" + record + "\n \n", encoding="utf-8") - loaded = training._load_jsonl(bank, 1, "corpus") + loaded = voice_dataset._load_jsonl(bank, 1, "corpus") self.assertEqual(len(loaded), 2) self.assertEqual({record["label"] for record in loaded}, {1}) diff --git a/tools/ai_ism_lint.py b/tools/ai_ism_lint.py index 4f00819..c02945e 100644 --- a/tools/ai_ism_lint.py +++ b/tools/ai_ism_lint.py @@ -32,7 +32,8 @@ EM_DASH_PATTERN = re.compile(r"—|---|\\textemdash") # A line that is nothing but dashes is a Markdown rule or front-matter fence. RULE_LINE_PATTERN = re.compile(r"\s*-{3,}\s*") -# The canonical Tier A list (skills/paper/SKILL.md). `paved` and `showcased` +# The canonical Tier A list; skills/paper/SKILL.md mirrors it, and the +# validator holds the mirror to it (`validator_check`). `paved` and `showcased` # were in the skill's table and grep but not here, so "paved the way" -- the # commonest form of the idiom -- passed the linter and failed the review grep. TIER_A_PATTERN = re.compile( @@ -352,7 +353,7 @@ def lexical_findings(text: str, path: Path, observed={"word": word, "occurrence_in_section": index + 1, "section_count": len(occurrences)}, reference={"cap_per_section_per_word": TIER_B_CAP, - "provenance": "skills/paper/SKILL.md"}, + "provenance": "docs/SCIPAPER_STANDARD.md"}, normalized_distance=float(index + 1 - TIER_B_CAP), message=(f"Tier B word {word!r} exceeds the per-section cap " f"of {TIER_B_CAP}; this is occurrence {index + 1}."), @@ -428,7 +429,7 @@ def document_source(path: Path) -> str: they treat comments". The L0 lexical scan is a third view -- it must see `---` and `\\textemdash` as authored, so it cannot use either projection -- and it was never brought into that guarantee. It read raw file text, and on - one manuscript it reported three em-dash targets that were all `% --- lane A:` + one manuscript it reported three em-dash targets that were all `% --- ... ---` comment rules: 3 of 3 false, with no non-comment line in the document carrying an em-dash at all. @@ -494,8 +495,11 @@ def collect_feedback(path: Path, field_profile_dir: Path | None, *, findings.extend(deai_structure.structure_findings(text, field_profile_dir, path)) axes.append(deai_structure.structure_axis_status(field_profile_dir)) if document_structure: - findings.extend(deai_docstructure.document_findings(text, field_profile_dir, path)) - axes.append(deai_docstructure.docstructure_axis_status(text, field_profile_dir)) + shape = deai_docstructure.document_shape(text) # once: findings and status read one measurement + findings.extend(deai_docstructure.document_findings( + text, field_profile_dir, path, shape=shape)) + axes.append(deai_docstructure.docstructure_axis_status( + text, field_profile_dir, shape=shape)) if oracle: try: import deai_oracle @@ -558,6 +562,100 @@ def lint(path: Path, field_profile_dir: Path | None, summary: bool = False, return 1 if report["summary"]["by_kind"]["l0_target"] else 0 +# --- validator hook ---------------------------------------------------------- + +# skills/paper/SKILL.md mirrors these four patterns for human reading: the Tier +# A table's word rows, its opener row (the phrases, and the one-word paragraph +# connectors with their comma), and the Tier B sentence after the pattern's +# name. The mirror drifted once (`paved`, `showcased`), so the validator reads +# it in both directions (audit H8). +MIRRORED = { + "TIER_A_PATTERN": TIER_A_PATTERN, + "TIER_A_OPENER_PATTERN": TIER_A_OPENER_PATTERN, + "TIER_A_PARAGRAPH_CONNECTOR_PATTERN": TIER_A_PARAGRAPH_CONNECTOR_PATTERN, + "TIER_B_PATTERN": TIER_B_PATTERN, +} +MIRROR_ROW_PATTERN = re.compile(r"^\| (动词类|形容词/副词|名词类|段首套话) \| (.+) \|$", + re.MULTILINE) +MIRROR_TIER_B_PATTERN = re.compile(r"`TIER_B_PATTERN`([^。]*)") +BACKTICKED_PATTERN = re.compile(r"`([^`]+)`") + + +def pattern_language(pattern: re.Pattern[str]) -> set[str]: + """Every surface form `pattern` accepts: the L0 patterns are finite. + + They are built from literals, alternation, groups and `?`. What frames a + form -- `\\b`, `^`, the opener's lookbehind and the whitespace before it -- + is context and adds nothing to it. Any other construct raises, so a pattern + that stops being enumerable fails the check instead of passing it. + """ + from re import _parser # private module, so imported only where it is walked + + whitespace_run = [(_parser.IN, [(_parser.CATEGORY, _parser.CATEGORY_SPACE)])] + + def language(items) -> set[str]: + forms = {""} + for op, av in items: + if op is _parser.LITERAL: + options = {chr(av)} + elif op is _parser.SUBPATTERN: + options = language(av[-1]) + elif op is _parser.BRANCH: + options = set().union(*map(language, av[1])) + elif op is _parser.MAX_REPEAT and av[:2] == (0, 1): + options = {""} | language(av[2]) + elif op in (_parser.AT, _parser.ASSERT) or ( + op is _parser.MAX_REPEAT and list(av[2]) == whitespace_run): + options = {""} + else: + raise ValueError(f"{pattern.pattern!r}: cannot enumerate {op} {av!r}") + forms = {head + tail for head in forms for tail in options} + return forms + + return language(_parser.parse(pattern.pattern)) + + +def skill_mirror(skill_text: str) -> dict[str, set[str]]: + """The forms skills/paper/SKILL.md lists, keyed by the pattern they mirror.""" + listed: dict[str, set[str]] = {name: set() for name in MIRRORED} + cells = MIRROR_ROW_PATTERN.findall(skill_text) + [ + ("Tier B", cell) for cell in MIRROR_TIER_B_PATTERN.findall(skill_text)] + for row, cell in cells: + for span in BACKTICKED_PATTERN.findall(cell): + for form in (part.strip() for part in span.split("/")): + if row == "Tier B": + listed["TIER_B_PATTERN"].add(form) + elif row != "段首套话": + listed["TIER_A_PATTERN"].add(form) + elif re.fullmatch(r"\w+,", form): # a connector: one word and its comma + listed["TIER_A_PARAGRAPH_CONNECTOR_PATTERN"].add(form) + else: # `In recent years,...` names the phrase the opener pattern holds + listed["TIER_A_OPENER_PATTERN"].add(form.rstrip(".,")) + return listed + + +def validator_check(repo: Path, require) -> str: + """The `validate_plugin` mirror check, run from its skills check. + + Every form the skill lists must be fully matched by its pattern, and every + form the pattern accepts must be listed -- compared case-folded where the + pattern ignores case. + """ + listed = skill_mirror((repo / "skills" / "paper" / "SKILL.md").read_text("utf-8")) + for name, pattern in MIRRORED.items(): + fold = str.lower if pattern.flags & re.IGNORECASE else str + unmatched = sorted(form for form in listed[name] if not pattern.fullmatch(form)) + unlisted = sorted({fold(form) for form in pattern_language(pattern)} + - {fold(form) for form in listed[name]}) + require(not unmatched and not unlisted, + f"skills/paper/SKILL.md mirror of {name} differs from ai_ism_lint: " + f"listed but not matched={unmatched}, matched but not listed={unlisted}") + return (f"skills/paper/SKILL.md mirrors the {len(MIRRORED)} L0 patterns both ways " + f"({sum(len(forms) for forms in listed.values())} forms)") + + +# --- CLI --------------------------------------------------------------------- + def main(argv: list[str] | None = None) -> int: cli_common.utf8_stdout() parser = cli_common.base_parser(__doc__, parents=[cli_common.field_options()]) diff --git a/tools/build_profile.py b/tools/build_profile.py index 6525168..9aea1b0 100644 --- a/tools/build_profile.py +++ b/tools/build_profile.py @@ -20,7 +20,6 @@ from __future__ import annotations -import argparse import subprocess import sys import time @@ -30,16 +29,6 @@ import cli_common # noqa: E402 -- because the sys.path insert above must run first REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" - - -def list_fields(profile_root: Path) -> list[str]: - if not profile_root.exists(): - return [] - return sorted( - p.name for p in profile_root.iterdir() - if p.is_dir() and not p.name.startswith(".") - ) def run_step(label: str, cmd: list[str]) -> tuple[bool, float]: @@ -124,7 +113,7 @@ def main(argv: list[str] | None = None) -> int: print(f" total: {total:.1f} s") print() - fields = list_fields(args.profile_root) + fields = cli_common.list_fields(args.profile_root) if fields: print(f"Basic profile(s) present: {fields}") print("This command did not create calibrated operating points.") diff --git a/tools/condense_map.py b/tools/condense_map.py index b0d8e81..c53179a 100644 --- a/tools/condense_map.py +++ b/tools/condense_map.py @@ -483,8 +483,7 @@ def main(argv: list[str] | None = None) -> int: cli_common.utf8_stdout() parser = cli_common.base_parser(__doc__) parser.add_argument("file", type=Path) - parser.add_argument("--format", choices=("text", "json"), default="text") - parser.add_argument("--output", type=Path) + cli_common.report_options(parser) args = parser.parse_args(argv) if not args.file.exists(): print(f"[condense_map] file not found: {args.file}", file=sys.stderr) diff --git a/tools/deai_anchoring.py b/tools/deai_anchoring.py index a150c45..659d854 100644 --- a/tools/deai_anchoring.py +++ b/tools/deai_anchoring.py @@ -235,9 +235,10 @@ def class_tests(baseline: dict[str, Any], result: dict[str, Any] "testable": [cls for cls in measured if cls not in underpowered]} -def anchoring_axis_status(text: str, field_profile_dir: Path | None - ) -> dict[str, Any]: - result = document_anchoring(text) +def anchoring_axis_status(text: str, field_profile_dir: Path | None, *, + result: dict[str, Any] | None = None) -> dict[str, Any]: + """The axis status; `result` is `document_anchoring(text)` when the caller has it.""" + result = document_anchoring(text) if result is None else result if result["status"] != "measured": return feedback.axis_status("L2.claim_anchoring", "unmeasured", reason="no section long enough to measure", @@ -267,9 +268,10 @@ def anchoring_axis_status(text: str, field_profile_dir: Path | None def anchoring_findings(text: str, field_profile_dir: Path | None, - path: str | Path | None = None) -> list[dict[str, Any]]: + path: str | Path | None = None, *, + result: dict[str, Any] | None = None) -> list[dict[str, Any]]: baseline = load_baseline(field_profile_dir) - result = document_anchoring(text) + result = document_anchoring(text) if result is None else result if baseline is None or result["status"] != "measured": return [] # Bonferroni share: a document measures up to k classes; testing each at @@ -381,10 +383,11 @@ def main(argv: list[str] | None = None) -> int: return 2 field_dir = cli_common.optional_field_dir(args, tool="deai_anchoring") text = args.file.read_text(encoding="utf-8", errors="replace") + result = document_anchoring(text) # once: the findings and the status read one measurement report = feedback.build_report( path=args.file, - findings=anchoring_findings(text, field_dir, args.file), - axes=[anchoring_axis_status(text, field_dir)], + findings=anchoring_findings(text, field_dir, args.file, result=result), + axes=[anchoring_axis_status(text, field_dir, result=result)], ) print(feedback.render_text(report)) return 0 diff --git a/tools/deai_discourse.py b/tools/deai_discourse.py index a1249f7..fe6ae75 100644 --- a/tools/deai_discourse.py +++ b/tools/deai_discourse.py @@ -18,7 +18,7 @@ hedges nowhere is entirely ordinary. Calibrating there would have produced a gate no passage can fall below, and the axis would have reported a confident zero findings forever. Regrouped so one section is one unit, six of the seven -buckets separate (p10 from 1.05 in `data` to 3.35 in `discussion`); `abstract` +buckets separate (p10 from 0.82 in `data` to 3.38 in `discussion`); `abstract` stays flat at 0.00 and abstains, since an abstract IS one passage. Each artifact records its own `unit` so the two can never be read against each other. @@ -29,13 +29,13 @@ Neither is an authorship claim, and the separation evidence is narrower than it first looks. Against 203 held-out refereed papers and six independent machine -generation regimes, exactly one bucket separates for every regime: `intro`, at -rank AUC 0.676-0.830 for cohesion (human/human null 0.515) and 0.613-0.816 for -hedging (null 0.460). Everywhere else at least one regime collapses -- -cohesion 0.461 in `discussion`, hedging 0.473 in `results` and 0.376 in -`conclusion`, all at or below chance -- and `method`'s apparent hedging -separation (0.603-0.796) sits against a 0.574 human/human null, so most of it is -not separation at all. EVALUATION section 19 carries the full table. +generation regimes, one bucket separates on both features for every regime: +`intro`, at rank AUC 0.677-0.832 for cohesion (human/human null 0.489) and +0.605-0.811 for hedging (null 0.487); hedging separates in `method` as well, +0.731-0.898 against a 0.442 null. Elsewhere at least one regime lands at or +near chance -- cohesion 0.508 in `data`, hedging 0.510 in `results`, 0.421 in +`discussion` and 0.358 in `conclusion`. EVALUATION section 19 carries the full +table. That is why these are advisories against the field's own distribution and not a detector. The reference says what this field's prose does; a passage below its @@ -140,21 +140,25 @@ def hedging_features(block: str) -> dict[str, Any] | None: # `buckets` is the second: None means every bucket whose reference resolves, and # a tuple restricts the axis to the section genres where its operating point was # shown to hold. Hedging is restricted, and two independent measurements over -# 203 held-out refereed papers put the restriction in the same place: +# 203 held-out refereed papers put the restriction in the same place (the +# 2026-09-27 profile, a manuscript section being the paper's whole bucket as on +# the reference side): # # held-out rate at a 10% gate worst-of-six regime AUC (null) -# intro 7.89% 0.613 (0.460) -# conclusion 15.48% 0.376 (0.469) -# discussion 16.34% 0.526 (0.475) -# results 24.17% 0.473 (0.520) -# data 22.98% 0.459 (0.508) -# method 26.77% 0.603 (0.574) +# intro 9.09% 0.605 (0.487) +# method 9.62% 0.731 (0.442) +# discussion 8.23% 0.421 (0.472) +# data 12.23% 0.464 (0.489) +# conclusion 13.64% 0.358 (0.520) +# results 14.17% 0.510 (0.440) # -# Outside `intro` the gate fires at two to three times its nominal rate on prose -# a referee accepted, and at least one generation regime lands at or below -# chance. `method` looks respectable until its human-vs-human null is read: 0.574 -# of that 0.603 is not separation. Cohesion needs no such restriction -- it runs -# 6.58%-14.63% across all seven buckets against the same 10% gate. +# Only `intro` and `method` pass both: the gate fires near its nominal rate on +# prose a referee accepted, and every regime sits clear of the human-vs-human +# null. `discussion` transfers but a regime lands below chance; the other three +# over-fire. The v0.33.0 measurement, when a manuscript section was one heading +# span, read `method` at 26.77% against a 0.574 null and restricted the axis to +# `intro` alone. Cohesion needs no restriction -- it runs 6.76%-14.88% across +# all seven buckets against the same 10% gate. # # This was measured on one field. A new field inherits the restriction, and # widening it means re-running that measurement, not editing this tuple. @@ -179,7 +183,7 @@ def hedging_features(block: str) -> dict[str, Any] | None: "unit": "section", "spans": reference.sections, "extract": hedging_features, - "buckets": ("intro",), + "buckets": ("intro", "method"), "message": ("{bucket} section states everything flatly: {value:.2f} " "epistemic markers per 1,000 words (p{pct}), against an " "n={n} human {bucket} reference."), diff --git a/tools/deai_docshape.py b/tools/deai_docshape.py index c7d0aa5..677f6f7 100644 --- a/tools/deai_docshape.py +++ b/tools/deai_docshape.py @@ -33,6 +33,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import deai_features as features # noqa: E402 resolves after the path insert import deai_metrics as metrics # noqa: E402 canonical section/paragraph ranges +import deai_reference as reference # noqa: E402 because it resolves only after the path insert import deai_structure as structure # noqa: E402 sentence template features import extract_style as es # noqa: E402 canonical LaTeX cleanup/tokenizer @@ -172,7 +173,7 @@ def _pairwise_similarity(vectors: list[list[float]]) -> float | None: def shape_units(text: str) -> list[dict[str, Any]]: - """The prose sections a document is measured on, with their raw blocks. + """The prose sections a document is measured on, with their prose blocks. One dict per unit of `deai_metrics.section_units` whose bucket is prose: `{"label", "start_line", "end_line", "blocks": [(start, end, block), ...]}`. @@ -184,6 +185,13 @@ def shape_units(text: str) -> list[dict[str, Any]]: measurements as body prose; a docstructure baseline built before the change must be rebuilt, since its every value carried them. `deai_partition` cuts its candidate blocks from this same list. + + Headings and floats are blanked first (`deai_reference.without_headings`, + as the anchoring sweep does), line count preserved. A heading alone on its + line ends a paragraph and a float does not, so a float's lines are cut as + lines of the paragraph around it. Until 2026-09-27 the raw text was cut: a + heading fused into its section's first block and a float holding a blank + line split its paragraph in two, so such a baseline must be rebuilt too. """ lines = text.splitlines() units: list[dict[str, Any]] = [] @@ -191,8 +199,14 @@ def shape_units(text: str) -> list[dict[str, Any]]: if bucket in (metrics.PREAMBLE_BUCKET, "skip"): continue segment = "\n".join(lines[start - 1:end]) + blanked = reference.without_headings(segment).split("\n") + cut = es.RE_TEX_ENV_FIGURE_TABLE.sub( # every float line is prose to the cut + lambda m: "\n".join(["float"] * (m.group(0).count("\n") + 1)), + es.blank_preserving(segment, es.RE_HEADING_COMMAND)) + blocks = [(p_start, p_end, "\n".join(blanked[p_start - start:p_end - start + 1])) + for p_start, p_end, _ in metrics.paragraph_line_ranges(cut, start)] units.append({"label": label, "start_line": start, "end_line": end, - "blocks": metrics.paragraph_line_ranges(segment, start)}) + "blocks": [block for block in blocks if block[2].strip()]}) return units @@ -536,8 +550,10 @@ def manifold_operating_point(baseline: dict, row: dict[str, float], baseline carries one (length-aware metric); otherwise scores against the pooled manifold with stratum-then-pooled calibration. Distances from different manifolds are never mixed in one calibration comparison. - Returns None when the axis is unmeasurable (no manifold or missing - features). + Returns None when the axis is unmeasurable: no manifold, missing + features, or no split-conformal calibration. The in-sample percentile + fallback for a baseline without a `conformal` block went on 2026-09-27; + no calibrate writes one, and `deai_docstructure` reports it degraded. """ pooled = baseline.get("dispersion_manifold") conformal = baseline.get("conformal") @@ -569,18 +585,6 @@ def manifold_operating_point(baseline: dict, row: dict[str, float], "calibration_basis": basis, "n_calibration": len(cal), "n_train": axis.get("n_train")} - if pooled: - distance = manifold_distance(pooled, row) - if distance is None: - return None - null = pooled["null_distances"] - return {"distance": distance, - "p_value": 1.0 - (sum(d <= distance for d in null) - / (len(null) + 1)), - "alpha": None, - "operating_point": "in-sample percentile", - "threshold": pooled["threshold"], - "flagged": distance > float(pooled["threshold"])} return None diff --git a/tools/deai_docstructure.py b/tools/deai_docstructure.py index 71d24b5..7f6128c 100644 --- a/tools/deai_docstructure.py +++ b/tools/deai_docstructure.py @@ -66,9 +66,27 @@ def baseline_tail_floor(baseline: dict[str, Any]) -> int: (baseline.get("role_coupling") or {}).get("low_percentile", ROLE_LOW_PERCENTILE))) -def docstructure_axis_status(text: str, field_profile_dir: Path | None - ) -> dict[str, Any]: - shape = document_shape(text) +def uncalibrated_manifold_reason(baseline: dict[str, Any]) -> str | None: + """Why the baseline's manifold rule goes unscored, or None when it can be. + + A manifold with no split-conformal calibration predates it. The in-sample + percentile fallback that scored one went with audit B24 (no calibrate + writes such a baseline), so its rule is left unscored: this axis reports + it degraded, and `deai_partition`, which ranks states by that rule, gives + the same reason for having no plan. + """ + conformal_manifold = (baseline.get("conformal") or {}).get("manifold") or {} + if baseline.get("dispersion_manifold") and not conformal_manifold.get("calibration"): + return ("the baseline's dispersion manifold carries no split-conformal " + "calibration, so the manifold rule was not scored; rebuild the " + "baseline with deai_docstructure --calibrate") + return None + + +def docstructure_axis_status(text: str, field_profile_dir: Path | None, *, + shape: dict[str, Any] | None = None) -> dict[str, Any]: + """The axis status; `shape` is `document_shape(text)` when the caller has it.""" + shape = document_shape(text) if shape is None else shape if shape["status"] != "measured": return feedback.axis_status("L2.document_structure", "unmeasured", reason=shape["reason"], @@ -88,13 +106,18 @@ def docstructure_axis_status(text: str, field_profile_dir: Path | None f"tail percentile needs {floor}; its in-sample findings are " "ordinary context"), detector="deai_docstructure") + uncalibrated = uncalibrated_manifold_reason(baseline) + if uncalibrated: + return feedback.axis_status("L2.document_structure", "degraded", + reason=uncalibrated, detector="deai_docstructure") return feedback.axis_status("L2.document_structure", "measured", detector="deai_docstructure") def document_findings(text: str, field_profile_dir: Path | None, - path: str | Path | None = None) -> list[dict[str, Any]]: - shape = document_shape(text) + path: str | Path | None = None, *, + shape: dict[str, Any] | None = None) -> list[dict[str, Any]]: + shape = document_shape(text) if shape is None else shape baseline = load_baseline(field_profile_dir) if shape["status"] != "measured" or baseline is None: return [] @@ -177,49 +200,33 @@ def document_findings(text: str, field_profile_dir: Path | None, else: per_feature_status = "degraded" else: + # Split-conformal only: `manifold_operating_point` leaves a manifold + # with no conformal block unscored (None), and the axis status says so. operating = manifold_operating_point(baseline, flat_row, shape["n_paragraphs"]) if operating is not None: distance = operating["distance"] p_value = operating["p_value"] - if operating["alpha"] is not None: - alpha = operating["alpha"] - cal_basis = operating["calibration_basis"] - n_cal = operating["n_calibration"] - flagged = p_value <= alpha - op_reference = {"operating_point": operating["operating_point"], - "alpha": alpha, - "n_calibration": n_cal, - "calibration_basis": cal_basis, - "n_train": operating.get("n_train"), - "provenance": BASELINE_NAME} - op_margin = alpha - p_value - op_confidence = { - "value": min(1.0, n_cal / 100.0), - "basis": (f"split-conformal p against {n_cal} held-out " - f"human papers ({cal_basis}); P(false flag) <= " - f"{alpha:g} finite-sample for exchangeable " - "human documents")} - op_clause = (f"conformal p = {p_value:.4f} <= alpha {alpha:g} " - f"against {n_cal} held-out human papers " - f"({cal_basis})") - else: # legacy baseline without a conformal block - flagged = operating["flagged"] - op_reference = {"operating_point": "in-sample percentile", - "n_documents": manifold["n_documents"], - "threshold": manifold["threshold"], - "percentile": manifold["percentile"], - "leave_one_document_out_flag_rate": manifold[ - "leave_one_document_out_flag_rate"], - "provenance": BASELINE_NAME} - op_margin = distance - float(manifold["threshold"]) - op_confidence = { - "value": min(1.0, manifold["n_documents"] / 100.0), - "basis": (f"{manifold['n_documents']} complete reference " - "documents; joint band distance in log " - "dispersion-ratio space")} - op_clause = (f"reference {manifold['percentile']:.0%} " - f"threshold {float(manifold['threshold']):.2f}") + alpha = operating["alpha"] + cal_basis = operating["calibration_basis"] + n_cal = operating["n_calibration"] + flagged = p_value <= alpha + op_reference = {"operating_point": operating["operating_point"], + "alpha": alpha, + "n_calibration": n_cal, + "calibration_basis": cal_basis, + "n_train": operating.get("n_train"), + "provenance": BASELINE_NAME} + op_margin = alpha - p_value + op_confidence = { + "value": min(1.0, n_cal / 100.0), + "basis": (f"split-conformal p against {n_cal} held-out " + f"human papers ({cal_basis}); P(false flag) <= " + f"{alpha:g} finite-sample for exchangeable " + "human documents")} + op_clause = (f"conformal p = {p_value:.4f} <= alpha {alpha:g} " + f"against {n_cal} held-out human papers " + f"({cal_basis})") if flagged: findings.append(feedback.make_finding( kind="advisory", layer="L2", @@ -479,10 +486,11 @@ def main(argv: list[str] | None = None) -> int: return 2 field_dir = cli_common.optional_field_dir(args, tool="deai_docstructure") text = args.file.read_text(encoding="utf-8", errors="replace") + shape = document_shape(text) # once: the findings and the status read one measurement report = feedback.build_report( path=args.file, - findings=document_findings(text, field_dir, args.file), - axes=[docstructure_axis_status(text, field_dir)], + findings=document_findings(text, field_dir, args.file, shape=shape), + axes=[docstructure_axis_status(text, field_dir, shape=shape)], ) print(feedback.render_text(report)) return 0 diff --git a/tools/deai_partition.py b/tools/deai_partition.py index 76b51e7..ea398a1 100644 --- a/tools/deai_partition.py +++ b/tools/deai_partition.py @@ -58,39 +58,62 @@ def _overlap(a: str, b: str) -> float: def _parse(text: str) -> list[dict]: - """Sections as ordered raw paragraph blocks with original line spans. - - The blocks are cut from `deai_docshape.shape_units`, so the preamble and - every `skip` unit are absent here exactly as they are absent from the - measurement a plan is scored on. A block that starts with a heading - command is a fixed landmark: merging or splitting into it would be a - nonsense suggestion. Any heading command counts (`\\subsection`, - `\\chapter`, ...); until 2026-09-27 only `\\section` did, and a subsection - heading was offered as a merge partner. + """Sections as ordered paragraph blocks with original line spans. + + The prose blocks are `deai_docshape.shape_units`' own, headings and floats + blanked, so the preamble and every `skip` unit are absent here exactly as + they are absent from the measurement a plan is scored on. What the + blanking took out between them -- a heading on its own line, a float + standing between paragraphs -- stays as a `landmark` block of raw text: + fixed and never scored, so no merge is offered across it ("delete the + blank line" would not describe one). A prose block whose raw text still + starts with a heading command (a heading sharing its line with a `\\label`) + is fixed too: merging or splitting into a heading would be a nonsense + suggestion. Any heading command counts (`\\subsection`, `\\chapter`, ...); + until 2026-09-27 only `\\section` did, and a subsection heading was + offered as a merge partner. """ + lines = text.splitlines() sections = [] for unit in ds.shape_units(text): - blocks = [{"lines": (p_start, p_end), "text": block, - "fixed": bool(extract_sections.RE_HEADING_COMMAND.match( - block.lstrip()))} - for p_start, p_end, block in unit["blocks"]] + blocks, cursor = [], unit["start_line"] + for p_start, p_end, block in unit["blocks"]: + blocks += _landmark(lines, cursor, p_start - 1) + raw = "\n".join(lines[p_start - 1:p_end]) + blocks.append({"lines": (p_start, p_end), "text": block, + "fixed": bool(extract_sections.RE_HEADING_COMMAND.match( + raw.lstrip()))}) + cursor = p_end + 1 + blocks += _landmark(lines, cursor, unit["end_line"]) if blocks: sections.append({"label": unit["label"], "blocks": blocks}) return sections +def _landmark(lines: list[str], first: int, last: int) -> list[dict]: + """The raw text between lines `first` and `last` that no prose block holds, + as one fixed, unscored block (none when those lines are blank).""" + filled = [number for number in range(first, last + 1) if lines[number - 1].strip()] + if not filled: + return [] + return [{"lines": (filled[0], filled[-1]), "fixed": True, "landmark": True, + "text": "\n".join(lines[filled[0] - 1:filled[-1]])}] + + def _state_statistics(sections: list[dict], baseline: dict) -> dict | None: - """Manifold distance (+ conformal p when available) of a partition state. - - The blocks scored are exactly the ones `document_shape` scores, through - the shared `measurable_sections` filter (word floor, paragraphs per - section, sections per document), so the starting distance printed here - is the one `deai_docstructure` reports for the same file. `calibration_basis` - names the manifold and calibration set the numbers came from; a state - below the document-shape floor, or one the axis cannot score, is None. + """Manifold distance and split-conformal p of a partition state. + + The blocks scored -- every block but the landmarks -- are exactly the ones + `document_shape` scores, through the shared `measurable_sections` filter + (word floor, paragraphs per section, sections per document), so the + starting distance printed here is the one `deai_docstructure` reports for + the same file. `calibration_basis` names the manifold and calibration set + the numbers came from; a state below the document-shape floor, or one the + axis cannot score, is None. """ units = [{"blocks": [(b["lines"][0], b["lines"][1], b["text"]) - for b in section["blocks"]]} for section in sections] + for b in section["blocks"] if not b.get("landmark")]} + for section in sections] kept = ds.measurable_sections(units) if len(kept) < ds.MIN_SECTIONS: return None @@ -104,13 +127,9 @@ def _state_statistics(sections: list[dict], baseline: dict) -> dict | None: operating = ds.manifold_operating_point(baseline, row, len(vectors)) if operating is None: return None - out = {"distance": operating["distance"], "n_paragraphs": len(vectors), - "calibration_basis": (operating.get("calibration_basis") - or operating["operating_point"])} - if operating["alpha"] is not None: - out["conformal_p"] = operating["p_value"] - out["alpha"] = operating["alpha"] - return out + return {"distance": operating["distance"], "n_paragraphs": len(vectors), + "calibration_basis": operating["calibration_basis"], + "conformal_p": operating["p_value"], "alpha": operating["alpha"]} def _split_at(block_text: str, cut: int) -> tuple[str, str]: @@ -217,21 +236,19 @@ def _improves(trial: dict, over: dict) -> bool: from the stratum-2 manifold are different quantities (`deai_docshape.manifold_operating_point`). Until 2026-09-27 the greedy step compared raw distances across that switch, and stopped on p while - advancing on distance. A legacy baseline without a conformal block has - one pooled manifold, so there its distances are the comparison. + advancing on distance. """ - if "conformal_p" in trial and "conformal_p" in over: - if trial["conformal_p"] != over["conformal_p"]: - return trial["conformal_p"] > over["conformal_p"] - return (trial["calibration_basis"] == over["calibration_basis"] - and trial["distance"] < over["distance"]) - if "conformal_p" in trial or "conformal_p" in over: - return False # one scored conformally, one not: no common scale - return trial["distance"] < over["distance"] + if trial["conformal_p"] != over["conformal_p"]: + return trial["conformal_p"] > over["conformal_p"] + return (trial["calibration_basis"] == over["calibration_basis"] + and trial["distance"] < over["distance"]) def suggest(text: str, baseline: dict, max_ops: int) -> dict: """Greedy band-seeking plan of admissible merge/split operations.""" + uncalibrated = ds.uncalibrated_manifold_reason(baseline) + if uncalibrated: # no state is scored against such a baseline, whatever the document + return {"status": "unmeasured", "reason": uncalibrated, "plan": []} sections = _parse(text) start = _state_statistics(sections, baseline) if start is None: @@ -247,8 +264,7 @@ def suggest(text: str, baseline: dict, max_ops: int) -> dict: state, current = sections, start plan = [] for _ in range(max_ops): - alpha = current.get("alpha") - if alpha is not None and current.get("conformal_p", 0.0) > alpha: + if current["conformal_p"] > current["alpha"]: break # inside the human band: nothing left to fix candidates = (_merge_candidates(state, floor) + _split_candidates(state, floor)) @@ -305,14 +321,10 @@ def main(argv: list[str] | None = None) -> int: return 0 start, end = result["start"], result["end"] print(f"[deai_partition] {args.file}") - line = (f" manifold distance {start['distance']:.3f} " - f"({start['calibration_basis']})") - if "conformal_p" in start: - line += f", conformal p {start['conformal_p']:.4f} (alpha {start['alpha']:g})" - print(line) + print(f" manifold distance {start['distance']:.3f} ({start['calibration_basis']}), " + f"conformal p {start['conformal_p']:.4f} (alpha {start['alpha']:g})") if not result["plan"]: - if ("conformal_p" in start and start.get("alpha") is not None - and start["conformal_p"] > start["alpha"]): + if start["conformal_p"] > start["alpha"]: print(" already inside the human band; nothing to fix") else: print(" no admissible partition operation moves the document " @@ -321,24 +333,17 @@ def main(argv: list[str] | None = None) -> int: for index, step in enumerate(result["plan"], 1): print(f" {index}. {step['description']}") extra = (f", p {step['conformal_p_before']:.4f} -> " - f"{step['conformal_p_after']:.4f}" - if step.get("conformal_p_after") is not None else "") + f"{step['conformal_p_after']:.4f}") if step["calibration_basis_before"] != step["calibration_basis_after"]: extra += (f" [calibration basis {step['calibration_basis_before']} -> " f"{step['calibration_basis_after']}: the two distances " "are not on one scale]") print(f" distance {step['distance_before']:.3f} -> " f"{step['distance_after']:.3f}{extra}") - verdict = "" - if "conformal_p" in end and end.get("alpha") is not None: - verdict = (" (inside the human band)" - if end["conformal_p"] > end["alpha"] - else " (still outside the band; apply and re-measure)") - print(f" projected: conformal p {start['conformal_p']:.4f} -> " - f"{end['conformal_p']:.4f}{verdict}") - else: - print(f" projected: distance {start['distance']:.3f} -> " - f"{end['distance']:.3f}") + verdict = (" (inside the human band)" if end["conformal_p"] > end["alpha"] + else " (still outside the band; apply and re-measure)") + print(f" projected: conformal p {start['conformal_p']:.4f} -> " + f"{end['conformal_p']:.4f}{verdict}") print(" suggestions only - apply by hand where the argument allows; " "merge/split change zero tokens, so fidelity is preserved by " "construction.") diff --git a/tools/deai_reference.py b/tools/deai_reference.py index b03dbf3..415e0c6 100644 --- a/tools/deai_reference.py +++ b/tools/deai_reference.py @@ -356,7 +356,7 @@ def units(text: str) -> list[tuple[int, int, str, str]]: for start, end, _label, bucket, block in paragraphs(text)] -RE_PLACEHOLDER_TOKEN = re.compile(r"\[[A-Za-z\-]+\]") +RE_PLACEHOLDER_TOKEN = re.compile(es.PLACEHOLDER_SHAPE) # the projection's own token shape # What may sit between two consecutive words of one projected sentence in the # raw block: anything but a letter (space, line break, punctuation, digit), a # command with its bracketed and braced arguments (a citation, a reference), diff --git a/tools/deai_residue.py b/tools/deai_residue.py index b1a2a28..a2826b8 100644 --- a/tools/deai_residue.py +++ b/tools/deai_residue.py @@ -65,7 +65,7 @@ import deai_register as register # noqa: E402 because the tools directory joins sys.path just above import extract_sections as sections # noqa: E402 because the tools directory joins sys.path just above import extract_style as es # noqa: E402 because the tools directory joins sys.path just above -import length_gate # noqa: E402 because the tools directory joins sys.path just above +import tex_assembly # noqa: E402 because the tools directory joins sys.path just above # Rule 1. Single source of the word families (see the module docstring). # Strength was set on 203 held-out refereed papers, not by intuition. `used to` @@ -544,8 +544,7 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument("after", type=Path, help="the document to check") parser.add_argument("--before", type=Path, help="pre-edit snapshot (enables the diff rule)") parser.add_argument("--git-ref", default=None, help="read the pre-edit version from git") - parser.add_argument("--format", choices=("text", "json"), default="text") - parser.add_argument("--output", type=Path) + cli_common.report_options(parser) args = parser.parse_args(argv) if not args.after.exists(): print(f"[deai_residue] file not found: {args.after}", file=sys.stderr) @@ -560,7 +559,7 @@ def main(argv: list[str] | None = None) -> int: if args.before is not None: before = ai_ism_lint.document_source(args.before) elif args.git_ref is not None: - before = _blank_comments(length_gate.read_git_version(args.after, args.git_ref)) + before = _blank_comments(tex_assembly.read_git_document(args.after, args.git_ref)) else: before = None if before is not None: diff --git a/tools/deai_salience.py b/tools/deai_salience.py index 47ef890..71eb77f 100644 --- a/tools/deai_salience.py +++ b/tools/deai_salience.py @@ -13,7 +13,9 @@ at the median; a draft whose run covers half its sentences is in the top decile of recital. -This detector is the only consumer of :func:`extract_style.latex_to_numeral_text`. +This is the only detector that reads :func:`extract_style.latex_to_numeral_text`; +`rewrite_reward` also calls it, and the exemplar bank stores each paragraph +under the same projection (`extract_style.paired_paragraphs`). Every other axis reads ``latex_to_plain``, which replaces each math span with ``[math]`` and so measures zero numerals on any real `.tex` file. That is the right reduction for lexical and sentence-shape statistics and the wrong one diff --git a/tools/eval_docscale.py b/tools/eval_docscale.py index 6e40399..aca3890 100644 --- a/tools/eval_docscale.py +++ b/tools/eval_docscale.py @@ -14,10 +14,9 @@ A row under `MIN_DOCUMENTS` scored documents reports `unmeasured`, never a rate: one document once printed a flag rate and an AUC of its own. Each row -names the operating point its flags were decided by, because a baseline can -carry either a split-conformal calibration (flag: p <= its own alpha) or the -older in-sample percentile threshold (flag: distance above it, alpha none), -and the two are not read against the same level. +names the operating point its flags were decided by (a stratum or the pooled +manifold); every point is split-conformal and flags at p <= its own alpha, +because a baseline without that calibration scores no document (audit B24). Run: python tools/eval_docscale.py --field wgl [--format json] """ @@ -112,17 +111,11 @@ def collect(baseline: dict, field: str, profile_root: Path, def flagged(point: dict) -> bool: """Whether the operating point that scored a document flags it. - A split-conformal point carries its own `alpha`, and the flag is - `p_value <= alpha`. A legacy in-sample percentile point carries - `alpha=None`, a `p_value` that is a within-sample percentile, and the - `flagged` decision its threshold already made; reading that percentile - against the baseline's conformal alpha compared it with a level it was - never calibrated to. + Every point is split-conformal and carries its own `alpha`; the flag is + `p_value <= alpha`. The branch for a legacy in-sample percentile point + (no alpha, its threshold's own decision) went with that point in B24. """ - alpha = point.get("alpha") - if alpha is not None and point.get("p_value") is not None: - return point["p_value"] <= alpha - return bool(point.get("flagged", False)) + return point["p_value"] <= point["alpha"] def operating_point(points: list[dict]) -> str: @@ -206,10 +199,8 @@ def build_report(field: str, profile_root: Path, corpus_root: Path) -> dict: f"the baseline scores none of them: rebuild it with `python " f"tools/deai_docstructure.py --calibrate --field {field} " f"--corpus-dir style-corpus/{field}`.") - # The level every flag in this run was decided at: the points carry it, - # and a legacy percentile baseline carries none. - alphas = {point.get("alpha") - for points in scored.values() for point in points} - {None} + # The level every flag in this run was decided at: the points carry it. + alphas = {point["alpha"] for points in scored.values() for point in points} return { "schema": "sci-paper.docscale-eval.v1", "field": field, diff --git a/tools/eval_findings.py b/tools/eval_findings.py index 3e5bf61..6e35a86 100644 --- a/tools/eval_findings.py +++ b/tools/eval_findings.py @@ -89,40 +89,26 @@ DOCUMENT_SCORES = {"L2.collocation": "collocation_novel_fraction"} -def _bundle_documents(root: Path) -> list[tuple[str, str]]: - """(name, text) per paper under a directory of arXiv source bundles. - - One bundle is one paper even when it ships a dozen `.tex` fragments, so the - roots are selected and their includes spliced back in, rather than each file - counting as its own document. - """ - out: list[tuple[str, str]] = [] - for bundle in sorted(p for p in root.iterdir() if p.is_dir()): - tex = sorted(bundle.rglob("*.tex")) - if not tex: - continue - for chosen in es.select_document_roots(tex, bundle): - text = es.read_tex_document(chosen) - if text.strip(): - out.append((f"{bundle.name}/{chosen.name}", text)) - return out - - def _flat_documents(root: Path) -> list[tuple[str, str]]: - """(name, text) for a directory of single-file documents.""" + """(name, text) for a directory of single-file documents; a directory of + source bundles is read by `extract_sections.bundle_documents` instead.""" return [(p.name, p.read_text(encoding="utf-8", errors="replace")) for p in sorted(root.glob("*.tex"))] -def score_document(text: str, field_dir: Path, path: str) -> dict[str, float]: +def score_document(text: str, field_dir: Path, path: str, + register: list[dict] | None = None) -> dict[str, float]: """Per-axis finding counts, length, and the salience denominator. `n_salience_units` is the count of passages the salience gate could have fired on. Without it the axis can only be reported per 1,000 words, and its gate is defined per passage — so there would be no way to check the measured - rate against the rate the percentile is supposed to produce. + rate against the rate the percentile is supposed to produce. `register` is + the document's `deai_register.register_findings` when the caller holds it + (`collect` runs that pass once and hands it to `leakage_paired` as well). """ - register = deai_register.register_findings(text, field_dir, path) + if register is None: + register = deai_register.register_findings(text, field_dir, path) zero = [f for f in register if f["rule"].startswith("register-zero:")] salience = deai_salience.salience_findings(text, field_dir, path) strong = sum(1 for f in salience @@ -190,7 +176,8 @@ def _density(rows: list[dict[str, float]], axis: str) -> list[float]: return [1000.0 * r[axis] / r["n_words"] for r in rows if r["n_words"]] -def leakage_paired(text: str, field_dir: Path, path: str) -> tuple[int, int]: +def leakage_paired(text: str, field_dir: Path, path: str, + register: list[dict] | None = None) -> tuple[int, int]: """(flagged, still flagged had this paper been in the bank) for one document. Comparing the held-out population against the in-sample one estimates @@ -228,7 +215,9 @@ def leakage_paired(text: str, field_dir: Path, path: str) -> tuple[int, int]: with_own = ChainMap({term: int(table.get(term, 0)) + count for term, count in own_df.items()}, table) flagged = survives = 0 - for finding in deai_register.register_findings(text, field_dir, path): + if register is None: + register = deai_register.register_findings(text, field_dir, path) + for finding in register: # The zero-hit audit is suppressed by own membership by construction # (one occurrence anywhere clears "absent"), so pairing it measures # nothing; the rarity rule is the one whose leakage is a question. @@ -266,14 +255,16 @@ def collect(field: str, corpus_root: Path, profile_root: Path, heldout_dir: str for label, root, is_bundle in sources: if not root.is_dir(): continue - documents = _bundle_documents(root) if is_bundle else _flat_documents(root) + documents = es.bundle_documents(root) if is_bundle else _flat_documents(root) rows = [] for name, text in documents: if len(text.split()) < deai_salience.MIN_WORDS: continue - rows.append(score_document(text, field_dir, name)) + # one register pass per document: the row and the pairing read it + register = deai_register.register_findings(text, field_dir, name) + rows.append(score_document(text, field_dir, name, register)) if label == "published-heldout": - one, two = leakage_paired(text, field_dir, name) + one, two = leakage_paired(text, field_dir, name, register) flagged += one survives += two if rows: diff --git a/tools/extract_md_negatives.py b/tools/extract_md_negatives.py index 7e721d1..e3a4c69 100644 --- a/tools/extract_md_negatives.py +++ b/tools/extract_md_negatives.py @@ -2,7 +2,7 @@ markdown. Strictly paper-voice prose, NOT documentation. Source-quality matters MORE than quantity. Empirically (2026-04-28 test -on D:/Projects/weak-gravitational-lensing/instructions/), feeding the +on a set of the author's project documentation), feeding the classifier compact technical-documentation prose drops CV F1 from 0.88 to 0.71 because doc-voice and paper-voice are different distributions — the model has to span both as "AI-ish" and gets confused. @@ -24,9 +24,9 @@ keeps paragraphs that look like prose (30–400 words, ≥ 60% alphabetic, < 15% digits to filter out tables-as-prose). -Output is appended to `style-profile//ai_ism_negatives_extracted.txt` -(gitignored — content comes from your private project files and shouldn't -ship with the plugin). +Output goes to `style-profile//ai_ism_negatives_extracted.txt`, +overwritten unless `--append` is given (gitignored — content comes from your +private project files and shouldn't ship with the plugin). Run with: python tools/extract_md_negatives.py --source-dir @@ -40,7 +40,6 @@ from __future__ import annotations -import argparse import re import sys from pathlib import Path @@ -48,9 +47,6 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" - # Strip ordering matters: code fences first (they may contain # / | / -), # then headers / blockquotes / tables / lists, then inline syntax. RE_FENCED_CODE = re.compile(r"```.*?```", re.DOTALL) diff --git a/tools/extract_sections.py b/tools/extract_sections.py index d93e1cf..e60ea30 100644 --- a/tools/extract_sections.py +++ b/tools/extract_sections.py @@ -228,17 +228,23 @@ def _project(text: str, *, inline, display: str, figure: str, cite: str) -> str: return RE_TEX_TILDE.sub(" ", text) +# What the plain projection writes for an inline math span, a display, a float +# and a citation. The one owner of that vocabulary: the placeholder patterns in +# `extract_style` and `deai_reference` are built from these names, not retyped. +PLAIN_INLINE = " [math] " PLAIN_PLACEHOLDERS = {"display": " [MATH] ", "figure": " [FIGURE-OR-TABLE] ", "cite": " [CITE] "} def latex_to_plain(text: str) -> str: - return _project(text, inline=" [math] ", **PLAIN_PLACEHOLDERS) + return _project(text, inline=PLAIN_INLINE, **PLAIN_PLACEHOLDERS) # A projection placeholder (`[math]`, `[MATH]`, `[FIGURE-OR-TABLE]`, `[CITE]`) # is not a word: counted, two deleted equations read as two words of prose cut. -RE_PLACEHOLDER = re.compile(r"^\[[A-Za-z\-]+\]$") +# Matched by the shape every token above has; `RE_PLACEHOLDER` holds a whole token. +PLACEHOLDER_SHAPE = r"\[[A-Za-z\-]+\]" +RE_PLACEHOLDER = re.compile(rf"^{PLACEHOLDER_SHAPE}$") def prose_words(text: str) -> list[str]: @@ -288,9 +294,6 @@ def latex_to_numeral_text(text: str) -> str: RE_TEX_DOC_MARKER = tex_assembly.RE_TEX_DOC_MARKER -RE_TEX_INCLUDE = tex_assembly.RE_TEX_INCLUDE -_include_targets = tex_assembly.include_targets -_resolve_include = tex_assembly.resolve_include CLASSIFIED_BUCKETS = frozenset( @@ -355,7 +358,7 @@ def select_document_roots(tex_files: list[Path], stems = {p.stem: p for p in group} included: set[Path] = set() for p, t in texts.items(): - for name in _include_targets(t): + for name in tex_assembly.include_targets(t): target = stems.get(Path(name).stem) if target is not None and target != p: included.add(target) @@ -426,6 +429,21 @@ def named(parent: Path, first: Path) -> str: return documents +def bundle_documents(root: Path) -> list[tuple[str, str]]: + """`(name, text)` per paper in a directory of per-paper bundles (a `--fulltext` + pull); the one loader both evaluators read. Unlike `corpus_documents`, each + bundle is its own `bundle_root`, so every root directly in it counts: the 200 + `wgl` held-out bundles yield 203 documents (measured 2026-09-27).""" + out: list[tuple[str, str]] = [] + for bundle in sorted(p for p in root.iterdir() if p.is_dir()): + tex = sorted(bundle.rglob("*.tex")) + for chosen in select_document_roots(tex, bundle) if tex else []: + text = read_tex_document(chosen) + if text.strip(): + out.append((f"{bundle.name}/{chosen.name}", text)) + return out + + def split_into_sections(raw_tex: str) -> dict[str, str]: """Best-effort split by \\section / \\subsection / \\chapter headers. diff --git a/tools/extract_style.py b/tools/extract_style.py index eb9d5b9..2a37b57 100644 --- a/tools/extract_style.py +++ b/tools/extract_style.py @@ -37,20 +37,16 @@ RE_HEADING_MATH, RE_HEADING_TEXORPDF, RE_PDF_LINE_HEADER, RE_PLACEHOLDER, RE_SECTION, RE_TEX_BEGIN_END, RE_TEX_BRACES, RE_TEX_CITE, RE_TEX_CITE_SILENT, RE_TEX_CITE_TEXT, RE_TEX_COMMENT, RE_TEX_DISPLAY_MATH, RE_TEX_ENV_FIGURE_TABLE, RE_TEX_INCLUDEGRAPHICS, - RE_TEX_INLINE_MATH, RE_TEX_DOC_MARKER, RE_TEX_INCLUDE, RE_TEX_LABEL_REF, RE_TEX_MATH_CMD, + RE_TEX_INLINE_MATH, RE_TEX_DOC_MARKER, RE_TEX_LABEL_REF, RE_TEX_MATH_CMD, RE_TEX_SIMPLE_CMD, RE_TEX_THIN_COMMA, RE_TEX_TILDE, RE_SENTENCE_TERMINAL, SECTION_PATTERNS, PDF_HEADING_MIN_LETTER_FRAC, PDF_HEADING_MIN_LETTERS, PDF_HEADING_MIN_WORDS, - _classify_pdf_heading, _include_targets, _math_numerals, _resolve_include, - _rejoin_pdf_paragraphs, blank_preserving, classify_section, clean_heading, corpus_documents, - extract_pdf_text, latex_to_numeral_text, latex_to_plain, PLAIN_PLACEHOLDERS, _project, - prose_words, read_tex_document, select_document_roots, split_into_sections, + _classify_pdf_heading, _math_numerals, PLACEHOLDER_SHAPE, PLAIN_INLINE, + _rejoin_pdf_paragraphs, blank_preserving, bundle_documents, classify_section, clean_heading, + corpus_documents, extract_pdf_text, latex_to_numeral_text, latex_to_plain, PLAIN_PLACEHOLDERS, + _project, prose_words, read_tex_document, select_document_roots, split_into_sections, split_pdf_into_sections, ) -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_CORPUS_ROOT = REPO_ROOT / "style-corpus" -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" - # The curated tiers. The weights are RECORDED metadata (they travel with each # observation) and are NOT applied: every aggregate pools the tiers equally, # as `aggregate_sentence_stats` says. The dict also names which directories @@ -84,7 +80,9 @@ def resolve_field(arg_field: str | None, corpus_root: Path) -> str: # the opener statistics, so `intro` lost its citation-led paragraphs. Nor are # the tokens words: counted, they inflated sentence lengths, the em-dash # denominator and the lexicon, and let a 25-word paragraph into the bank. -_RE_PLAIN_PLACEHOLDER_TOKEN = re.compile(r"\[(?:MATH|math|FIGURE-OR-TABLE|CITE)\]") +# Built from the owner's vocabulary, so a placeholder added there is dropped here. +_RE_PLAIN_PLACEHOLDER_TOKEN = re.compile("|".join( + re.escape(token.strip()) for token in (PLAIN_INLINE, *PLAIN_PLACEHOLDERS.values()))) def without_placeholders(text: str) -> str: # every placeholder becomes a space @@ -393,7 +391,7 @@ def _math_numerals_slotted(match: "re.Match[str]") -> str: `$\\vec{\\nabla}$` alone on a line is a `[math]` paragraph in the plain view and would be a swallowed blank line in the numeral one.""" reduced = _math_numerals(match) - return reduced if reduced.strip() else " [math] " + return reduced if reduced.strip() else PLAIN_INLINE def paired_paragraphs(text: str) -> tuple[str, list[str] | None]: @@ -507,11 +505,16 @@ def write_dossier( lexicon: dict, n_papers: int, field: str, + skipped: int = 0, ) -> None: lines = [] lines.append(f"# Style Dossier — field: `{field}` (auto-generated)\n") + # The run's final summary names the files it could not read; so does the + # header of the dossier built without them. + skipped_note = (f" {skipped} source file(s) SKIPPED (pymupdf unavailable): this " + "dossier does not describe them." if skipped else "") lines.append(f"Built from {n_papers} corpus papers under " - f"`style-corpus/{field}/`. Re-run " + f"`style-corpus/{field}/`.{skipped_note} Re-run " f"`python tools/extract_style.py --field {field}` after " "corpus changes.\n") lines.append("> Descriptive evidence only. Normative policy lives in " @@ -520,10 +523,13 @@ def write_dossier( "and regenerate.\n") lines.append("\n## 1. Sentence length per section\n") - if not sentence_stats: + # `unknown` is not a section: sectionless `.tex`, `.txt` and heading-less + # PDF text all land there, so "no sections" never meant an empty dict. The + # hint joins the table rather than replacing it: `unknown` was measured. + if not set(sentence_stats) - {DEFAULT_SECTION_BUCKET}: lines.append("_No sections detected. Are you using `.tex` source with " "`\\section{}` markers?_\n") - else: + if sentence_stats: lines.append("| Section | n | mean | median | stdev | p25 | p75 | p95 |") lines.append("|---|---|---|---|---|---|---|---|") for sec, st in sorted(sentence_stats.items()): @@ -618,14 +624,7 @@ def main(argv: list[str] | None = None) -> int: # build_profile.py does when it captures this tool's output -- the default # encoder raises UnicodeEncodeError and the run dies after the work is done. cli_common.utf8_stdout() - p = cli_common.base_parser(__doc__) - p.add_argument("--field", default=None, - help="Field name (subdir under style-corpus/). " - "Auto-detected when only one field exists.") - p.add_argument("--corpus-root", type=Path, default=DEFAULT_CORPUS_ROOT, - help="Root corpus dir (default: style-corpus/).") - p.add_argument("--profile-root", type=Path, default=DEFAULT_PROFILE_ROOT, - help="Root profile dir (default: style-profile/).") + p = cli_common.field_parser(__doc__, corpus=True) args = p.parse_args(argv) field = resolve_field(args.field, args.corpus_root) @@ -724,6 +723,7 @@ def main(argv: list[str] | None = None) -> int: lexicon, n_papers=len(per_paper), field=field, + skipped=skipped, ) n_exemplars = write_exemplar_bank(per_paper + reference_papers, field_profile) diff --git a/tools/fetch_arxiv_abstracts.py b/tools/fetch_arxiv_abstracts.py index 0267402..945645f 100644 --- a/tools/fetch_arxiv_abstracts.py +++ b/tools/fetch_arxiv_abstracts.py @@ -44,8 +44,8 @@ from tex_assembly import RE_TEX_DOC_MARKER # noqa: E402 -- same sys.path reason REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" -API = "http://export.arxiv.org/api/query" +API = "https://export.arxiv.org/api/query" +USER_AGENT = "sci-paper-voice/0.14 (corpus builder)" # one identity for both endpoints ATOM = "{http://www.w3.org/2005/Atom}" ARXIV = "{http://arxiv.org/schemas/atom}" BANK = "human_abstracts_extra.jsonl" # the abstract bank every downstream reader opens @@ -182,7 +182,7 @@ def fetch_page(query: str, start: int, n: int, date_lo: str, date_hi: str) -> li "search_query": q, "start": start, "max_results": n, "sortBy": "submittedDate", "sortOrder": "descending"}) url = f"{API}?{params}" - req = urllib.request.Request(url, headers={"User-Agent": "sci-paper-voice/0.13"}) + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) xml = urlopen_backoff(req, timeout=60).decode("utf-8", "replace") root = ET.fromstring(xml) out = [] @@ -238,7 +238,7 @@ def fetch_page(query: str, start: int, n: int, date_lo: str, date_hi: str) -> li def _eprint_bytes(arxiv_id: str) -> bytes: req = urllib.request.Request( EPRINT + urllib.parse.quote(arxiv_id), - headers={"User-Agent": "sci-paper-voice/0.14 (corpus builder)"}) + headers={"User-Agent": USER_AGENT}) return urlopen_backoff(req, timeout=120) diff --git a/tools/label_findings.py b/tools/label_findings.py index 0b8eb60..243bc02 100644 --- a/tools/label_findings.py +++ b/tools/label_findings.py @@ -144,20 +144,13 @@ def _documents(root: Path, *, bundles: bool) -> list[tuple[str, str]]: """(name, text) per document, one paper per entry. An arXiv source bundle ships a dozen `.tex` fragments and is still one - paper, so its roots are selected and their includes spliced back in rather - than each fragment counting as a document of its own. + paper, so a directory of bundles goes through the loader `eval_findings` + reads the same population with (`extract_sections.bundle_documents`). """ if not bundles: return [(p.name, p.read_text(encoding="utf-8", errors="replace")) for p in sorted(root.rglob("*.tex"))] - out: list[tuple[str, str]] = [] - for bundle in sorted(p for p in root.iterdir() if p.is_dir()): - tex = sorted(bundle.rglob("*.tex")) - for chosen in es.select_document_roots(tex, bundle) if tex else []: - text = es.read_tex_document(chosen) - if text.strip(): - out.append((f"{bundle.name}/{chosen.name}", text)) - return out + return es.bundle_documents(root) def _load_population(root: Path) -> list[tuple[str, str]]: diff --git a/tools/length_gate.py b/tools/length_gate.py index 7baf90a..4a6bbb2 100644 --- a/tools/length_gate.py +++ b/tools/length_gate.py @@ -60,11 +60,6 @@ def section_word_counts(text: str) -> dict[str, int]: return dict(counts) -def read_git_version(after: Path, ref: str) -> str: - r"""The assembled document at `ref`: root and every \input child (`tex_assembly`).""" - return tex_assembly.read_git_document(after, ref) - - def parse_allowances(entries: list[str]) -> dict[str, str]: allowances: dict[str, str] = {} for entry in entries: @@ -290,7 +285,7 @@ def main(argv: list[str] | None = None) -> int: # the root alone never saw a child shrink, while the removal map it # closes was built on the whole paper. if args.git_ref is not None: - before_text = read_git_version(args.after, args.git_ref) + before_text = tex_assembly.read_git_document(args.after, args.git_ref) else: before_text = tex_assembly.read_tex_document(args.before) after_text = tex_assembly.read_tex_document(args.after) diff --git a/tools/retrieve_exemplars.py b/tools/retrieve_exemplars.py index 5132365..d7044da 100644 --- a/tools/retrieve_exemplars.py +++ b/tools/retrieve_exemplars.py @@ -20,21 +20,19 @@ from __future__ import annotations -import argparse import json import sys from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) import cli_common # noqa: E402 -- because the sys.path insert above must run first -from extract_style import REFERENCE_DIR, TIER_WEIGHTS +from extract_style import CLASSIFIED_BUCKETS, REFERENCE_DIR, TIER_WEIGHTS -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_PROFILE_ROOT = REPO_ROOT / "style-profile" DEFAULT_MODEL = "all-MiniLM-L6-v2" -VALID_SECTIONS = {"abstract", "intro", "method", "data", "results", - "discussion", "conclusion"} +# The named section buckets (neither `skip` nor `unknown`), derived rather than +# retyped: the hand-typed set learned about `data` only after 112 bank rows had it. +VALID_SECTIONS = CLASSIFIED_BUCKETS # The bank serves two roles. Its curated rows are the imitation target; its # `REFERENCE_DIR` rows are the breadth corpus that makes the per-section diff --git a/tools/train_ai_ism_classifier.py b/tools/train_ai_ism_classifier.py index 3d928e8..6bc1eba 100644 --- a/tools/train_ai_ism_classifier.py +++ b/tools/train_ai_ism_classifier.py @@ -44,8 +44,8 @@ def load_positives(jsonl_path: Path, The source travels with the text because paragraphs from one paper are not independent: an ungrouped split puts siblings in train and test at once and - reports an optimistic score. 28 source papers back the 1957 `wgl` - paragraphs, so grouping is available and is used. + reports an optimistic score. The bank records each paragraph's source + paper, so grouping is available and is used. """ pos = [] with jsonl_path.open(encoding="utf-8") as f: @@ -99,7 +99,7 @@ def train_and_save( from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import Pipeline - from sklearn.model_selection import cross_val_score + from sklearn.model_selection import cross_validate import joblib X = positives + negatives @@ -128,9 +128,10 @@ def train_and_save( # and GROUPED by source paper when groups are supplied: paragraphs from one # paper are not independent observations, and an ungrouped split lets # siblings sit in train and test simultaneously. Measured on the `wgl` - # corpus the difference is F1 0.876 ungrouped vs 0.823 grouped, so the - # ungrouped number was optimistic by ~0.05. `cv_grouped` travels with the - # metrics so a consumer can never read one for the other. + # corpus at v0.27.0 (2026-08-16, CHANGELOG-ARCHIVE) the difference was F1 + # 0.876 ungrouped vs 0.823 grouped, so the ungrouped number was optimistic + # by ~0.05. `cv_grouped` travels with the metrics so a consumer can never + # read one for the other. from sklearn.model_selection import StratifiedGroupKFold, StratifiedKFold n_splits = 5 cv_grouped = bool(groups) and len(set(groups)) >= n_splits @@ -142,10 +143,11 @@ def train_and_save( splitter = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=seed) split_kwargs = {} - acc_scores = cross_val_score(pipeline, X, y, cv=splitter, - scoring="accuracy", **split_kwargs) - f1_scores = cross_val_score(pipeline, X, y, cv=splitter, scoring="f1", - **split_kwargs) + # One pass: each fold is fitted once and scored on both metrics. Two + # cross_val_score passes fitted the same folds twice for the same numbers. + scores = cross_validate(pipeline, X, y, cv=splitter, + scoring=("accuracy", "f1"), **split_kwargs) + acc_scores, f1_scores = scores["test_accuracy"], scores["test_f1"] pipeline.fit(X, y) output_path.parent.mkdir(parents=True, exist_ok=True) @@ -219,10 +221,11 @@ def main(argv: list[str] | None = None) -> int: print(f"\n[train_ai_ism_classifier] OK.") print(f" n_pos = {metrics['n_pos']}, n_neg = {metrics['n_neg']}") - # Accuracy alone is unreadable at this class ratio: the corpus bank grew to - # 25k paragraphs against ~20 negatives, so always-predict-corpus already - # scores ~0.999. Print that baseline next to it so the headline number - # cannot be mistaken for skill; F1 on the minority class is the real signal. + # Accuracy alone is unreadable at this class ratio: corpus paragraphs far + # outnumber the negatives, so always-predict-corpus already scores close to + # 1. Print that baseline, from this run's own counts, next to it so the + # headline number cannot be mistaken for skill; F1 on the minority class is + # the real signal. majority = metrics["n_pos"] / max(1, metrics["n_pos"] + metrics["n_neg"]) print(f" class ratio: {metrics['n_pos'] / max(1, metrics['n_neg']):.0f}:1 " f"(always-predict-corpus accuracy = {majority:.3f})") diff --git a/tools/train_voice_model.py b/tools/train_voice_model.py index 0e831a0..86aa981 100644 --- a/tools/train_voice_model.py +++ b/tools/train_voice_model.py @@ -30,20 +30,18 @@ # Re-exported so `train_voice_model.` keeps resolving for existing # callers and tests after the 2026-08-26 split. A contract test asserts this -# list still covers every public name in both modules. +# list still covers every public name in both modules and carries no private +# helper: a test that needs one imports it from its own module. from voice_dataset import ( # noqa: E402,F401 -- re-export, unused here by design CHECKPOINT_EVERY, build_features, build_field_lexicon, embeddings_failed_at_run_time, feature_cache_fingerprint, load_records, source_family, - _atomic_savez, _load_jsonl, _record_embeddings, _tokens, ) from voice_audit import ( # noqa: E402,F401 -- re-export, unused here by design HARDSET_AI_CATEGORIES, HARDSET_HUMAN_CATEGORIES, aggregate_audits, audit_split_seed, binary_metrics, confound_audit, first_valid_group_split, hardset_evaluation, repeated_group_audit, section_normalize_uid, split_corpus_cos, - _auc, _bootstrap_auc_ci, _breakdown, _math_bin, _metric_maps, _quantile, - _series_summary, _three_way_bin, ) @@ -168,8 +166,12 @@ def _make(name): f"{results['hgb']['auc']:.3f} but weaker OOD — see comment)") print(f"[audit] primary confound strata for {best}", file=sys.stderr) + # Once for both primary audits, as `repeated_group_audit` does for its + # own: a record's math density depends on neither the split nor the scores. + math_density = [df.math_marker_density(record["text"]) for record in recs] primary_raw_audit = confound_audit( - recs, X_primary, tr_idx, va_idx, yva, validation_scores[best]) + recs, X_primary, tr_idx, va_idx, yva, validation_scores[best], + math_density=math_density) normalized_X, normalization_meta = section_normalize_uid( X_primary, recs, tr_idx) normalized_scaler = StandardScaler().fit(normalized_X[tr_idx]) @@ -179,7 +181,8 @@ def _make(name): normalized_scores = normalized_model.predict_proba( normalized_scaler.transform(normalized_X[va_idx]))[:, normalized_column] primary_normalized_audit = confound_audit( - recs, X_primary, tr_idx, va_idx, yva, normalized_scores) + recs, X_primary, tr_idx, va_idx, yva, normalized_scores, + math_density=math_density) print(f"[audit] repeated source-grouped comparison: " f"{args.audit_splits} splits", file=sys.stderr) diff --git a/tools/validate_plugin.py b/tools/validate_plugin.py index ebe7147..41fd7b5 100644 --- a/tools/validate_plugin.py +++ b/tools/validate_plugin.py @@ -10,6 +10,7 @@ import ast import contextlib +import functools import importlib import io import json @@ -213,7 +214,9 @@ def check_skills() -> str: for token in (SCHEMA, "integrity_blocker", "l0_target", "advisory", "measured", "degraded", "unmeasured", "not_applicable", "rejected_as_false_positive"): require(token in standard, f"SCIPAPER_STANDARD.md missing contract token {token!r}") - return f"skills and normative standard agree ({len(documents)} skills)" + sys.path.insert(0, str(TOOLS)) # not popped: importing ai_ism_lint inserts tools/ itself + mirror = importlib.import_module("ai_ism_lint").validator_check(REPO, require) + return f"skills and normative standard agree ({len(documents)} skills); {mirror}" RE_HEADING = re.compile(r"^#{1,6}\s+(.*?)\s*$", re.MULTILINE) @@ -421,8 +424,9 @@ def check_documentation_boundaries() -> str: f"{len(skills)} skills named only where shipped; heading order clean)") +@functools.lru_cache(maxsize=None) def _discovered_test_count() -> tuple[int, int]: - """(test count, test-file count) from real unittest discovery.""" + """(test count, test-file count) from real unittest discovery, run once per process.""" import unittest suite = unittest.defaultTestLoader.discover(str(TESTS), top_level_dir=str(TESTS)) diff --git a/tools/voice_audit.py b/tools/voice_audit.py index 03892f9..d3fae57 100644 --- a/tools/voice_audit.py +++ b/tools/voice_audit.py @@ -8,8 +8,11 @@ Split out of `train_voice_model.py` on 2026-08-26, which had reached 1,174 lines against the repository's 750-line budget and could no longer be edited. -Nothing here fits a model; a bundle stays degraded until an operating point and -a confound audit are recorded, and these functions produce that evidence. +Nothing here fits a model that ships: `repeated_group_audit` fits two +audit-only logistic regressions per split (raw and section-normalized UID) and +keeps only their held-out scores. A bundle stays degraded until an operating +point and a confound audit are recorded, and these functions produce that +evidence. """ from __future__ import annotations @@ -23,6 +26,8 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import deai_features as df # noqa: E402 resolves only after the sys.path insert import voice_dataset as vd # noqa: E402 resolves only after the sys.path insert +# The suite's one midrank AUC, because a second copy here could drift from it. +from eval_docscale import rank_auc # noqa: E402 because of the sys.path insert # Hard-set provenance categories. They live here rather than in the CLI because # `hardset_evaluation` below is their only consumer; `train_voice_model` @@ -60,24 +65,11 @@ def _math_bin(value: float, nonzero_median: float) -> str: return "present-high" -def _auc(y_values: list[int], scores: list[float]) -> float | None: - n_pos = sum(y_values) - n_neg = len(y_values) - n_pos - if not n_pos or not n_neg: - return None - ordered = sorted(zip(scores, y_values), key=lambda pair: pair[0]) - rank_sum = 0.0 - position = 1 - cursor = 0 - while cursor < len(ordered): - end = cursor + 1 - while end < len(ordered) and ordered[end][0] == ordered[cursor][0]: - end += 1 - average_rank = (position + position + (end - cursor) - 1) / 2.0 - rank_sum += average_rank * sum(label for _, label in ordered[cursor:end]) - position += end - cursor - cursor = end - return (rank_sum - n_pos * (n_pos + 1) / 2.0) / (n_pos * n_neg) +def _by_class(y_values, scores) -> tuple[list[float], list[float]]: + """(label-0 scores, label-1 scores): `rank_auc`'s argument order, which + gives P(a label-1 score outranks a label-0 one), ties at midrank.""" + pairs = list(zip(y_values, scores)) + return [s for y, s in pairs if not y], [s for y, s in pairs if y] def binary_metrics(y_values, scores, threshold: float = 0.5) -> dict: @@ -113,7 +105,7 @@ def binary_metrics(y_values, scores, threshold: float = 0.5) -> dict: "n_negative": len(labels) - sum(labels), "score_mean": statistics.mean(probabilities) if probabilities else None, "score_median": statistics.median(probabilities) if probabilities else None, - "auc": _auc(labels, probabilities), + "auc": rank_auc(*_by_class(labels, probabilities)), "positive_recall": tpr, "negative_false_positive_rate": fpr, "f1_positive": f1, @@ -210,7 +202,7 @@ def _bootstrap_auc_ci(y_values: list[int], scores: list[float], *, Small author-labelled strata have wide sampling error; a point AUC without an interval invites the exact over-reading this hard set previously caused. """ - point = _auc(y_values, scores) + point = rank_auc(*_by_class(y_values, scores)) if point is None: return None import numpy as np @@ -219,7 +211,7 @@ def _bootstrap_auc_ci(y_values: list[int], scores: list[float], *, draws: list[float] = [] for _ in range(n_boot): idx = rng.integers(0, n, size=n) - value = _auc([y_values[i] for i in idx], [scores[i] for i in idx]) + value = rank_auc(*_by_class([y_values[i] for i in idx], [scores[i] for i in idx])) if value is not None: draws.append(value) draws.sort() @@ -362,14 +354,20 @@ def hardset_evaluation(field_dir: Path, bundle: dict, model_name: str) -> dict: def confound_audit(recs: list[dict], X, train_indices, validation_indices, - y_validation, scores, threshold: float = 0.5) -> dict: - """Build one split's confound report without creating an operating point.""" + y_validation, scores, threshold: float = 0.5, *, + math_density: list[float] | None = None) -> dict: + """Build one split's confound report without creating an operating point. + + `math_density` is every record's `df.math_marker_density`, which no split + changes; `repeated_group_audit` measures it once and passes it to each call. + """ train_indices = [int(i) for i in train_indices] validation_indices = [int(i) for i in validation_indices] lexicon, lexicon_meta = vd.build_field_lexicon(recs, train_indices) word_index = df.FEATURE_NAMES.index("word_count") word_counts = [float(X[i, word_index]) for i in range(len(recs))] - math_density = [df.math_marker_density(record["text"]) for record in recs] + if math_density is None: + math_density = [df.math_marker_density(record["text"]) for record in recs] jargon_density = [df.lexicon_density(record["text"], lexicon) for record in recs] train_lengths = [word_counts[i] for i in train_indices] @@ -559,6 +557,9 @@ def repeated_group_audit(recs: list[dict], X, y, groups, *, n_splits: int, normalized_reports: list[dict] = [] paired_deltas: dict[str, list[float]] = defaultdict(list) split_records: list[dict] = [] + # Once per audit: a record's math density does not depend on the split, and + # each split's two confound_audit calls read the same list. + math_density = [df.math_marker_density(record["text"]) for record in recs] attempts = 0 maximum_attempts = max(20, n_splits * 20) while len(raw_reports) < n_splits and attempts < maximum_attempts: @@ -586,7 +587,7 @@ def repeated_group_audit(recs: list[dict], X, y, groups, *, n_splits: int, raw_scaler.transform(X_split[validation_indices]))[:, positive_column] raw_report = confound_audit( recs, X_split, train_indices, validation_indices, y_validation, - raw_scores) + raw_scores, math_density=math_density) normalized_X, normalization = section_normalize_uid( X_split, recs, train_indices) @@ -600,7 +601,7 @@ def repeated_group_audit(recs: list[dict], X, y, groups, *, n_splits: int, normalized_scaler.transform(normalized_X[validation_indices]))[:, positive_column] normalized_report = confound_audit( recs, X_split, train_indices, validation_indices, - y_validation, normalized_scores) + y_validation, normalized_scores, math_density=math_density) raw_reports.append(raw_report) normalized_reports.append(normalized_report) From 2c70054d0fdd21e9381a0f803d201c4d870378b0 Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 12:27:54 -0400 Subject: [PATCH 24/27] skills: strong advisories only where the standard allows them, and the audit's open skill items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H14 (the author's decision: the skills follow the standard): standard §4 makes an advisory strong only when a measured effect exceeds a calibrated operating point. De-AI's Pass-2 density hits, paper-review's structural hits, physics' naming and unit checks and mainline's narrative findings have no such point; each is now an ordinary advisory ranked first. The standard is unchanged. The skill items of audit 2026-09-27 that the verification after the first tier found still open: - H15: an unlabelled schematic formula is a ranked-first advisory only when its dimensions close; otherwise P1 makes it an integrity blocker. - H16: proposal-polish proposes a firmer central hypothesis to the author and never applies it silently (§6 keeps stance and modality). - H17: mainline asks an eighth question, the one thesis line every contribution branch serves (standard §5.4). - H18/H22: under --orchestrated, paper-review also skips the dimension-D de-ai audit and the dimension-G figure-review call, which the parent runs; the partition suggestions come from de-ai Pass 3, not from --audit-only. - H37: paper-review's M.1 runs physics P1-P6 for its first and third passes instead of restating them, and dimension K says how it runs alone and under --orchestrated. - H39/H40: argument hints list only defined flags; brainstorm drops the no-op --min-frameworks, and names cc-enforcer, FACTS.md and cc-tree as optional. - H42: the residue gate joins the stopping semantics. - H32/H33: narration of removed flags and dated source notes leave the skills; the attribution and adoption boundaries live in ACKNOWLEDGMENTS.md. - F15: the Tier B cap is per heading unit, as the linter counts it. - H45-H51: repetition inside a skill is cut to one statement and a pointer. Repetition across skills stays (the author's decision), because each skill is loaded alone and does not follow pointers. - H21 needs no change: brainstorm's NEEDS-MORE-INFO is a provisional verdict that convergence forbids, which is what paper-review requires. Also: the paper skill's review grep finds every form the linter does (`delving`; `Moreover` mid-sentence), and calibrate's document-structure row says what an uncalibrated manifold does since B24. Co-Authored-By: Claude Opus 5.5 --- ACKNOWLEDGMENTS.md | 24 +++++++++--- skills/brainstorm/SKILL.md | 36 +++++++----------- skills/calibrate/SKILL.md | 12 +++--- skills/de-ai/SKILL.md | 14 ++++--- skills/final-review/SKILL.md | 33 +++++++---------- skills/mainline/SKILL.md | 27 ++++++++------ skills/paper-review/SKILL.md | 61 +++++++++++++++--------------- skills/paper/SKILL.md | 66 ++++++++++++--------------------- skills/physics/SKILL.md | 5 ++- skills/proposal-polish/SKILL.md | 16 +++++--- 10 files changed, 143 insertions(+), 151 deletions(-) diff --git a/ACKNOWLEDGMENTS.md b/ACKNOWLEDGMENTS.md index 2ee6bba..469186b 100644 --- a/ACKNOWLEDGMENTS.md +++ b/ACKNOWLEDGMENTS.md @@ -8,20 +8,26 @@ implicit. ## AIScientists-Dev/academic-humanizer - (MIT) + (MIT License, +Copyright (c) 2026 AIScientists-Dev) **Adopted.** The 2026-07-16 lexicon extensions (`underscore*`, `pivotal`, `tapestry`, `testament`, `realm*`, `intricate`, `foster*`); the `serves as`, `ing-tail`, and `colon-elaboration` linter rules; the Claim–Evidence Discipline -and Preserve List sections of [`skills/paper/SKILL.md`](skills/paper/SKILL.md); +(its Layer 4, recalibrated on the astronomy corpus) and Preserve List (its +Layer 3) sections of [`skills/paper/SKILL.md`](skills/paper/SKILL.md); the [`proposal-polish`](skills/proposal-polish/SKILL.md) skill; and the [`de-ai`](skills/de-ai/SKILL.md) skill's Layer 1–5 audit catalog adapt its -material. +material. The first five words entered Tier A only after both curated field +corpora showed zero occurrences of them; `intricate` and `foster*` occur once +each there, so they entered Tier B. **Declined.** Venue-specific rules that conflict with astronomy usage — -`landscape`, and the blanket `demonstrate` / `significantly` bans. Every lexical -adoption was re-verified against the curated field corpora before tier -assignment. +`landscape`, a legitimate field term (`detection landscape`) frequent in the +corpus, and the blanket `demonstrate` / `significantly` bans: in the combined +curated astronomy corpus `demonstrate*` runs at 0.147/1k and `significantly` at +0.274/1k (measured 2026-07-16). Every lexical adoption was re-verified against +the curated field corpora before tier assignment. academic-humanizer itself builds on blader/humanizer (MIT). @@ -38,6 +44,12 @@ self-interrogation step, and its false-positive guards. chatbot artifacts, curly-quote flags) and its `landscape`-flagging word list. Only the academically relevant structural tells were absorbed. +## Project-owner rules + +Two rules in [`skills/paper/SKILL.md`](skills/paper/SKILL.md) were set by the +project owner on 2026-07-16: condense, do not accumulate (改写、删减、精简，而不是堆叠; +now standard §5.3), and the `colon-elaboration` rule. + --- Licensing: [MIT](LICENSE) covers code, skills, documentation, and tooling diff --git a/skills/brainstorm/SKILL.md b/skills/brainstorm/SKILL.md index 559ec93..d30f40b 100644 --- a/skills/brainstorm/SKILL.md +++ b/skills/brainstorm/SKILL.md @@ -1,6 +1,6 @@ --- name: brainstorm -description: Fully automatic radial exploration engine for divergent research ideation, or for exhaustively finding ways to solve one concrete hard problem. The data model is a phylogenetic tree: root = the starting point (topic / problem / current research state), node = one idea, depth = number of divergence layers, width = number of final results. Every node is brainstormed through twelve framing passes (first-principles, inversion, cross-disciplinary transport, adversarial, constraint relaxation, scale extrapolation, office-hours, contrarian, failure-driven, high-risk, meta), every branch gets a complete rigorous derivation (mathematics / physics / logic / literature check / feasibility / falsifiability), and divergence recurses until every deepest leaf is fully advanced. A mandatory glossary grill locks the root node's terminology to the project FACTS.md before exploration starts. width / depth / rounds default to unlimited and stop on convergence criteria. Deferred or incomplete leaves ('defer', 'cost limits', 'future work', 'TODO') are hard-banned. Use when: "brainstorm" / "explore options" / "find research directions" / "how do I solve this problem" / 发散思考 / 找研究方向 / 怎么解决这个难题 / 穷尽某个研究问题的解法 / 论文 motivation 阶段需要 radial 探索. +description: Fully automatic radial exploration engine for divergent research ideation, or for exhaustively finding ways to solve one concrete hard problem. The data model is a phylogenetic tree: root = the starting point (topic / problem / current research state), node = one idea, depth = number of divergence layers, width = number of final results. Every node is brainstormed through twelve framing passes (first-principles, inversion, cross-disciplinary transport, adversarial, constraint relaxation, scale extrapolation, office-hours, contrarian, failure-driven, high-risk, meta), every branch gets a complete rigorous derivation (mathematics / physics / logic / literature check / feasibility / falsifiability), and divergence recurses until every deepest leaf is fully advanced. A mandatory glossary grill locks the root node's terminology to the project glossary (FACTS.md if available) before exploration starts. width / depth / rounds default to unlimited and stop on convergence criteria. Deferred or incomplete leaves ('defer', 'cost limits', 'future work', 'TODO') are hard-banned. Use when: "brainstorm" / "explore options" / "find research directions" / "how do I solve this problem" / 发散思考 / 找研究方向 / 怎么解决这个难题 / 穷尽某个研究问题的解法 / 论文 motivation 阶段需要 radial 探索. disable-model-invocation: false argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max-branches N|∞] [--field ] [--out ] [--seed ] [--no-online] — 不传 topic 则自动从当前项目状态推断" --- @@ -22,7 +22,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max - **node（节点）** = 每一个想法本身。无论位于哪一层、哪一支，**任何一个发散/推导/结果点都是 node**。 - **width（最外弧 / "宽度"）** = 最终交付的结果数，即树达到收敛/终止时**叶节点（不再扩展的终态节点）**的总数。**宽度无上限**（除非用户用 `--width N` 显式封顶）。 - **生长准则**：一个 node 若能再开出"不同尝试 / 不同方向"，则**必须**伸出新子节点继续向外（depth + 1）；只有当一个 node 已经被 §4 的 12 字段全部填满、§5 评分定论、且§3 的 12 个 framing pass 在该节点上跑过仍**无新颖增益**时，它才允许作为最终叶（计入 width）。 -- **完整推进准则（硬性，§0.8 强制）**：进入 width 计数的每一片最终叶节点都**必须完整推进**——derivation 走通、predictions 给数、falsifiability 给判据、novelty_vs_literature 给真实文献对比。任何含 "defer / 时间不够 / 算力不够 / future work / 留作 TODO" 字样的节点**不算最终叶**，必须继续推进或被显式标记为 `INCOMPLETE_FORBIDDEN` 并触发再循环。 +- **完整推进准则（硬性，§0.8 强制）**：进入 width 计数的每一片最终叶节点都**必须完整推进**——derivation 走通、predictions 给数、falsifiability 给判据、novelty_vs_literature 给真实文献对比；半成品的判定与处理见 §0.8。 --- @@ -46,11 +46,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max 6. **禁止用户中断决策**——本 skill 是全自动的。 遇到歧义优先选**信息量最大**的分支继续；只有当 (a) 触及不可逆操作、(b) 触及 §0.7 资源安全阀、(c) 用户原始 topic 完全无法解析、(d) §2.0 glossary grill 判定 MISSING / AMBIGUOUS（一次性提问）或 CONFLICT（硬停，等用户裁决）时才停下。术语歧义是 root 节点的缺陷，绕过它会让整棵树建立在错误定义上。 -7. **资源参数（默认全部不限；caps 仅在用户显式提供数值时生效）**： - - `--width N` 默认 ∞ — 最终叶节点总数上限 - - `--depth N` 默认 ∞ — 树深度上限 - - `--rounds N|conv` 默认 `conv` — 发散轮次，由§6 收敛判据终止 - - `--max-branches N` 默认 ∞ — 每节点单轮新增分支上限（注意：§3 强制 12 条 framing pass 各产 ≥1 分支，所以下限实际是 12；该 flag 仅可放大） +7. **资源参数（默认全部不限；caps 仅在用户显式提供数值时生效；flag 与默认值见 §1 flag 表）**： - **caps 触顶的语义**：当用户**显式**设置了 `--width N` / `--depth N` / `--rounds N` 并触顶时，**已展开的节点必须先全部完整推进到§4 12 字段填满、§5 verdict 定论之后**才允许停止；不允许"刚到上限立即停留下半成品"。报告以 `WIDTH_CAP_REACHED` / `DEPTH_CAP_REACHED` / `ROUNDS_EXHAUSTED` 标记，**但所有可见叶节点必须完整**。 - **不允许 skill 内部自行扩大 cap**；也不允许内部自行缩小默认 ∞ 为某个有限值。 - "探索成本太大"不构成停止理由；这是本 skill 与普通 brainstorm 工具的关键区别。 @@ -61,7 +57,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max - "因成本限制" / "因算力限制" / "因时间限制" / "时间不够" / "算力不够" - "future work" / "留作 future work" / "TODO" / "FIXME" - "暂不展开" / "略" / "details omitted" / "省略" / "暂略" - - "应该" / "大概" / "我相信" / "通常" / "可能"（→ 触发 cc-enforcer rule 01） + - "应该" / "大概" / "我相信" / "通常" / "可能"（→ 若装有 cc-enforcer，触发其 rule 01） - 若推导**真的**需要外部资源（特定数据集、特定计算、特定文献全文），必须当轮通过 WebFetch / WebSearch / Bash / Read 获取；获取失败 → 改用§3.X / §3.E（约束变换）派生**替代方案分支**并完整推导。**禁止**留半成品节点声称"算最终叶"。 - 这条禁令在**每一个节点**处都强制执行，不分主分支次分支、不分高分低分。 @@ -89,12 +85,9 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max | `--out ` | `brainstorm-out/__/` | 树输出目录 | | `--seed ` | 无 | 额外种子提示，作为 root 节点的 hint | | `--no-online` | 关 | 关闭 WebSearch / WebFetch；只用本地 + 已读引用 | -| `--min-frameworks N` | 12 | 每节点至少跑过的 framing pass 数（§3）；下限即 §3.A–§3.L 全 12 条 | | `--min-novelty-ratio R` | 0.15 | 收敛要求的"近 2 轮新颖比"下限（详§6） | | `--no-grill` | 关 | 跳过 §2.0 glossary grill 预热（弱收敛模式；root 节点术语标 `unverified`） | -> **关于"无上限"**：本 skill 的设计哲学是用§6 收敛判据（substantive convergence）而不是用资源 cap（resource exhaustion）来终止。当用户既不传 `--width / --depth / --rounds` 也不触§6 收敛 → 它会一直跑直到收敛，这是预期行为而非 bug。 - **§1.1 field 选择**：与 `/sci-paper:de-ai` 的 field 解析一致 — 解析 `style-profile/` 下子目录：1 个 → 自动选；多个 → 要求 `--field`；0 个 → 跳过文献先验加权（不阻塞，仅警告）。 @@ -113,7 +106,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max **操作**（一次一问、每问给推荐答；用户传 `--no-grill` 才跳过）： 1. **定位 glossary 来源**（按优先级 Read 第一个存在的）： - - `wgl-suite/FACTS.md` + `wgl-suite/KEY_NUMBERS.md`（WGL 项目 single source of truth） + - `wgl-suite/FACTS.md` + `wgl-suite/KEY_NUMBERS.md`（可选，若存在；WGL 项目 single source of truth） - 项目根的 `CLAUDE.md` - `style-profile//style_dossier.md` 中含 glossary 表的段落 - 兜底：从最近 git log + .tex / .md draft 抽常见 noun-phrase 自建临时 glossary @@ -127,7 +120,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max - **EXACT MATCH** → 默认用 glossary 定义，无须打扰用户，记入 `/glossary-anchors.md` - **ALIAS** → 直接用 canonical 名替换，在 `glossary-anchors.md` 注 `aka: "<别名>"`，无须打扰用户 - **MISSING**（glossary 没收）→ 一次性问用户："你说的 X 是不是指 glossary 里的 Y？还是 Z？还是一个 glossary 没收的新概念？" → 给 ≤3 个选项 + 推荐答（按 grill-me 的"一次一问"惯例） - - **AMBIGUOUS**（glossary 有 ≥2 条相似条目 A vs B）→ 一次性问 "X 是 A（FACTS.md §3.A）还是 B（FACTS.md §3.B）？" + 推荐答 + - **AMBIGUOUS**（glossary 有 ≥2 条相似条目 A vs B）→ 一次性问 "X 是 A（glossary §3.A）还是 B（glossary §3.B）？" + 推荐答 - **CONFLICT**（用户 topic 的用法跟 glossary 已收定义矛盾）→ **立刻停 brainstorm**，等用户裁决：改 brainstorm 用法 vs 改 glossary 收新定义。**不允许**绕过冲突静默继续。 4. **产物**：写 `/glossary-anchors.md`：root 节点用的术语 + 对应 glossary 定义 + file:line 引用。后续每个 framing pass / 每个分支生成时用本文件做术语锁——若新分支引入新 noun，要追加到本文件并 grill 一次。 @@ -177,7 +170,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max > **核心创新点**：每个节点走完 §3.A–§3.L 全部 12 条 framing pass，每条至少产出 1 个分支； > 之后由§4 完整推导每个分支，§5 评估并决定是否进一步展开。 -> "全部"是硬性要求 —— `--min-frameworks` 的下限即全 12 条（§1 flag 表、§3 完成判据、§9 反模式同此）。它只能放大所需的 framing 数，不能低于 12。 +> "全部"是硬性要求。 ### §3.A — First-principles / Constructor-theoretic - 把当前节点的所有"约定俗成"假设列出来；逐条问"如果这条不成立呢？" @@ -242,7 +235,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max 7. 现在树里的"最 weird" 分支真的足够 weird 吗？如果不够，强制再生成一个。 - 输出至少 1 个分支，必须是元层自审中暴露出的盲区 -> **§3 完成判据**：上述 12 条全部跑完 + `--min-frameworks` 下限满足 + 每条至少有 1 条分支带完整推导。 +> **§3 完成判据**：上述 12 条全部跑完 + 每条至少有 1 条分支带完整推导。 ### §3.X — 在线工具/插件检索（每节点至少 1 次，除非 `--no-online`） @@ -279,7 +272,7 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max - `derivation` 中含数值时**当轮**用 Bash + python（sympy / numpy）跑一次自检脚本；输出贴入字段；无法跑则在字段最后写 `[unverified — needs symbolic check]` - `novelty_vs_literature` 中每篇引用必须 WebFetch arXiv abs / DOI 页确认作者+年份+标题至少匹配；不可仅凭 WebSearch 摘要下结论 - 任何字段写出 "应该 / 大概 / 我相信 / 通常 / 应当" → 该字段无效，必须重写 -- **§0.8 完整推进禁令在此强制生效**：任何字段含 "defer / 留后 / 待定 / 因成本限制 / 因时间限制 / 因算力限制 / future work / TODO / FIXME / 略 / details omitted / 暂略 / 暂不展开" 之一 → 该字段无效，节点状态强制改为 `INCOMPLETE_FORBIDDEN`，必须继续推进至该字段完整。**不允许把半成品节点提交进 verdict**。 +- **§0.8 完整推进禁令在此强制生效**：任一字段含 §0.8 所列字样 → 该字段无效，节点状态强制改为 `INCOMPLETE_FORBIDDEN`，必须继续推进至该字段完整。**不允许把半成品节点提交进 verdict**。 --- @@ -336,7 +329,6 @@ argument-hint: "[topic] [--width N|∞] [--depth N|∞] [--rounds N|conv] [--max | Topic 完全无法解析 | `EARLY_STOP=topic_unparseable` | 必须在§2 baseline 阶段；进入§3 之后此项不再适用 | CONVERGED / WIDTH_CAP_REACHED / DEPTH_CAP_REACHED / ROUNDS_EXHAUSTED 时输出**最终报告**（§7）。 -**"资源不够"不构成停止理由**——本 skill 不接受 "exploration cost too high" 之类的隐含 early-stop（与 §0.7 / §0.8 一致）。 --- @@ -415,7 +407,7 @@ CONVERGED / WIDTH_CAP_REACHED / DEPTH_CAP_REACHED / ROUNDS_EXHAUSTED 时输出** --- -## 8. 工具使用规范（cc-enforcer 投影） +## 8. 工具使用规范（取自 cc-enforcer 规则；不依赖该插件） | 任务 | 必用工具 | 禁止 | |---|---|---| @@ -429,7 +421,7 @@ CONVERGED / WIDTH_CAP_REACHED / DEPTH_CAP_REACHED / ROUNDS_EXHAUSTED 时输出** **子代理使用建议**： - 当树宽度 ≥ 5 时，把每个 framing pass 派给一个 Explore subagent 并行；汇总后由主 agent 做§5 评估 - 子代理 prompt 必须自包含（背景 + 当前节点 + 该 pass 的硬要求） -- 子代理返回的引用主 agent 必须再 verify（cc-enforcer rule 04） +- 子代理返回的引用主 agent 必须再 verify（若装有 cc-enforcer，即其 rule 04） --- @@ -444,9 +436,9 @@ CONVERGED / WIDTH_CAP_REACHED / DEPTH_CAP_REACHED / ROUNDS_EXHAUSTED 时输出** - ❌ "WebSearch 找了，没找到，就是新的" — 至少要换 3 种关键词组合 + 检查相邻 field。 - ❌ "用户没说要并行，我就串行做" — `--rounds conv` 模式下并行是性能必需。 - ❌ "树太大用户看不动，我手动剪一下" — 用户要的是穷尽，不是好看。 -- ❌ "我跑完了 framing A 到 D 觉得够了" — §3.A–§3.L 全部必跑（`--min-frameworks` 下限即 12）。 -- ❌ **"这条分支留作 future work"** / **"因时间限制暂不展开"** / **"因算力限制 defer 到将来"** / **"细节暂略"** / **"TODO: 进一步推导"** — 全部违反§0.8 完整推进禁令。节点必须完整推进或显式标 `INCOMPLETE_FORBIDDEN` 触发再循环。 -- ❌ "探索成本太高，提前停止" — §0.7 已移除时间 / 节点数硬 cap；只有§6 收敛或用户**显式** width/depth/rounds cap 触顶才能停，且停时所有叶必须完整。 +- ❌ "我跑完了 framing A 到 D 觉得够了" — §3.A–§3.L 全部必跑。 +- ❌ **"这条分支留作 future work"** / **"因时间限制暂不展开"** / **"因算力限制 defer 到将来"** / **"细节暂略"** / **"TODO: 进一步推导"** — 全部违反§0.8 完整推进禁令。 +- ❌ "探索成本太高，提前停止" — 只有§6 收敛或用户**显式** width/depth/rounds cap 触顶才能停（§0.7）。 - ❌ "用户说 problem-solving，我就跳过§3 的某些 framing" — 不允许。研究 mode 与问题求解 mode 都跑全部 12 条 framing；问题求解 mode 只是 baseline 不同。 --- diff --git a/skills/calibrate/SKILL.md b/skills/calibrate/SKILL.md index fb2c32f..c48a163 100644 --- a/skills/calibrate/SKILL.md +++ b/skills/calibrate/SKILL.md @@ -123,9 +123,9 @@ python tools/deai_discourse.py --calibrate --field | `L2.sentence_structure` | per-section reference fractions | `measured` for deterministic matches, `degraded` for strength — there is no calibrated strong-advisory operating point | | `L0.register` | ≥ 500 corpus passages, and enough of them to resolve a 1e-4 document-frequency rate | under 500 the axis is silent; unable to resolve the rate it is `degraded`, or borrows ``'s bank and says so | | `L2.claim_anchoring` | ≥ 30 documents per section class | classes below the floor are omitted from the band, honestly, rather than estimated | -| `L2.document_structure` | ≥ 3 complete documents, each ≥ 3 sections with ≥ 2 substantial paragraphs | `unmeasured`; legacy baselines without a conformal block fall back to percentile thresholds | +| `L2.document_structure` | ≥ 3 complete documents, each ≥ 3 sections with ≥ 2 substantial paragraphs | `unmeasured`; under the 20 documents a 5% tail needs, `degraded`; a manifold without a conformal block is left unscored and the axis `degraded` until `--calibrate` is rerun | | `L2.cohesion` | ≥ 30 reference paragraphs per bucket, each ≥ 3 sentences and ≥ 40 words | that bucket abstains; a reference with no spread *below* the p10 gate reports nothing rather than everything | -| `L2.hedging` | ≥ 30 reference **sections** per bucket, each ≥ 150 words — and the bucket must be one the axis is calibrated for | `degraded`. It ships restricted to `intro` because that is the only bucket where its gate was shown to transfer (EVALUATION §19.4); a thin profile is `degraded` there too | +| `L2.hedging` | ≥ 30 reference **sections** per bucket, each ≥ 150 words — and the bucket must be one the axis is calibrated for | `degraded`. It ships restricted to `intro` and `method`, the buckets where its gate was shown both to transfer and to separate (EVALUATION §19.3–19.4); a thin profile is `degraded` there too | If you point `--corpus-dir` anywhere, point it at the field root or at `fulltext-arxiv/`. Aimed at the field root it collects the calibration corpus @@ -231,10 +231,10 @@ and a surname is not a person. **Sample size.** Any cell under 20 labels reports `unmeasured`. The sampler prints, before you start, every cell its populations cannot fill — and some cannot be filled by any sheet size, only by more papers. Measured on the `wgl` -corpus: fifteen held-out single-author-group papers yield 8 register findings -and 2 hedging findings, and the entire 203-paper held-out set yields 15 hedging -findings, because hedging speaks only about introductions and fires below a -tenth percentile. Salience and cohesion fill from a handful of papers. +corpus: the entire 203-paper held-out set yields 33 hedging findings, because +hedging speaks only about introductions and methods and fires below a tenth +percentile, so a hedging cell needs most of that set. Salience and cohesion +fill from a handful of papers. ## 8. Verify, then record what stayed unmeasured diff --git a/skills/de-ai/SKILL.md b/skills/de-ai/SKILL.md index 66834c5..6a054cd 100644 --- a/skills/de-ai/SKILL.md +++ b/skills/de-ai/SKILL.md @@ -76,7 +76,7 @@ condense's ranked sweep. `degraded` or `unmeasured`, never a clean result. No universal style verdict exists. 6. **L0 is the only axis rewritten to zero.** Tier A, em-dash, and Tier B - above the section cap are `l0_target`. L1-L4 findings are ranked + above its heading-unit cap are `l0_target`. L1-L4 findings are ranked advisories with dispositions, never forced to zero at the cost of clarity. 7. **Condense, do not accumulate (§5.3).** The default direction of every rewrite is shorter; the explanatory patch — answering a finding by @@ -202,8 +202,10 @@ Axes and tools: ## 3. Pass 2 — humanizer structural-tell audit (vendored) Audit against the catalog below, then merge every hit into the Pass-1 ranked -list (Layer 1-2 structural hits are style-class advisories, density excess -upgrades to strong; Layer-4 claim-evidence failures are integrity-class). +list (Layer 1-2 structural hits are ordinary style-class advisories: a +catalog count has no calibrated operating point, so density excess puts a hit +first among the ordinary advisories but never makes it strong (standard §4); +Layer-4 claim-evidence failures are integrity-class). Skip LaTeX comments and scaffolding macros. This pass never edits. ### Layer 1 — general AI tells @@ -315,8 +317,8 @@ is the measured voice baseline. Funding proposals route to ## 4. Pass 3 — claim-first rewrite Select work in ranked order: L0 targets in the requested span first, then -strong L1/L2/L3 advisories with concrete actions, then document-shape -findings. For each selected paragraph: +strong L1/L2/L3 advisories with concrete actions, then Pass-2 density hits, +then document-shape findings. For each selected paragraph: 1. **Protect.** Write the protected claim record (`/claim.txt`): every claim and its evidence relation, all numbers/units, citations, named @@ -360,7 +362,7 @@ findings. For each selected paragraph: and negation/causal/comparison markers as bags, so a marker moved between clauses passes it; its `count mismatch` line flags only lost repeats. 4. **Re-measure.** Re-run the Pass-1 linter on the candidate in enough - section context for section caps. A good rewrite introduces no new + heading-unit context for the Tier B cap. A good rewrite introduces no new integrity blocker or L0 target, stays eligible, acts on the selected advisory, worsens no higher-priority finding, does not grow the passage without a stated reason (report the length delta), and reports residual diff --git a/skills/final-review/SKILL.md b/skills/final-review/SKILL.md index 0c89405..d89937e 100644 --- a/skills/final-review/SKILL.md +++ b/skills/final-review/SKILL.md @@ -23,7 +23,7 @@ The required review components are: 6. `/sci-paper:mainline` for the narrative-spine cold read; 7. `/sci-paper:logic` for reasoning, statistics and claim-evidence discipline. -Components 5-7 are measurement primitives that paper-review also composes. The parent +Components 3-7 are measurement primitives that paper-review also composes. The parent launches them at its own level so their cold read is independent of paper-review's context rather than nested inside it. @@ -57,9 +57,7 @@ condense during §3.7. Keeping detection in one lane avoids duplicate review sur `unmeasured`/`degraded`, not clean. 10. **Minimum effective fixes only.** Every edit maps to a finding ID; no opportunistic rewriting or unrelated refactor. -11. **Stable rounds verify state, not zero suggestions.** `--require-consecutive` rounds must - reproduce a disposition-complete state with no new blockers/L0 and no unexplained change - in strong advisories. +11. **Stable rounds verify state, not zero suggestions.** The criterion is §4. 12. **Do not expand budget silently.** `--max-rounds` exhaustion returns `BREAK_WITH_USER_DECISION` with the exact unresolved state. @@ -74,7 +72,7 @@ Defaults: - `--max-rounds 10` - `--require-consecutive 2` - no skipped reviewers -- output under `final-review-out/__/` +- output under `final-review-out/__/`; `--out ` sets another root (the `` of §2) Valid skips: `paper-review`, `figure-review`, `de-ai`, `physics`, `mainline`, `logic`. A skip must be user-explicit, remains visible as `unmeasured`, and cannot be described as @@ -130,7 +128,8 @@ Launch a worktree agent with a self-contained prompt: - return the complete typed report, including A–R coverage (dimension E narrative-spine answers and dimension M record included), measurement states, blockers, L0 targets, strong/ordinary advisories, dispositions and build evidence; -- set physics, mainline and logic to `SKIPPED_FOR_ORCHESTRATOR`. +- set physics, mainline, logic, the dimension-D de-ai audit and the dimension-G figure-review + call to `SKIPPED_FOR_ORCHESTRATOR`; the parent runs all five itself (§3.3-§3.5). If it attempts nesting, return `NESTED_AGENT_REJECTED`; the round becomes `PROMPT_VIOLATION` and must be reissued with the corrected prompt. @@ -156,8 +155,8 @@ Launch a separate worktree agent to invoke `/sci-paper:de-ai --audit-on - the full `sci-paper.feedback.v1` measurement report with every axis state; - L0 targets (Tier A / em-dash / Tier B excess) as `l0_target` findings; - structural-tell and distribution findings as ranked advisories; -- document-shape findings with their fidelity-free partition suggestions, - clearly marked apply-by-hand. +- document-shape findings as advisories; their fidelity-free partition suggestions come from + de-ai Pass 3 (`deai_partition.py`), which the parent runs in §3.7, not from this audit. The parent applies L0 fixes and selected strong-advisory rewrites itself in §3.7 (or runs de-ai Pass 3 in the parent context); the auditor never edits. @@ -230,12 +229,11 @@ For each action: Close the round with the §5.3 loop-close gates against the round's baseline (§2 step 7): `python tools/length_gate.py --before ` and `python tools/deai_residue.py --before ` (or `--git-ref`). A strong -`length-growth`, `length-shrink-short` or residue finding needs a disposition before the round -can be disposition-complete; record `--allow "
="` justifications in the -round record. +`length-growth`, `length-shrink-short` or residue finding needs a disposition (§4); record +`--allow "
="` justifications in the round record. -Subjective strong advisories that require author preference may remain pending with a precise -question. Do not erase them merely to make counts zero. +A strong advisory whose fix turns on the author's preference may remain pending with a precise +question. Do not erase it merely to make counts zero. ## 4. Stable-round criterion @@ -323,13 +321,8 @@ unmeasured axes. ## 7. Completion meaning -`DISPOSITION_COMPLETE` means the mandated review workflow reached a stable state in which: - -- scientific blockers were resolved or disproved; -- L0 targets are zero; -- strong advisories have explicit author/process dispositions; -- ordinary residuals and limitations are visible; -- independent reviewers reproduced that state for the required consecutive rounds. +`DISPOSITION_COMPLETE` means the mandated review workflow reached the §4 disposition-complete +state and independent reviewers reproduced it for `--require-consecutive` rounds. It does not mean the paper is guaranteed correct, accepted, human-authored, aesthetically unique or free of every possible reviewer objection. diff --git a/skills/mainline/SKILL.md b/skills/mainline/SKILL.md index edc1d65..2549bea 100644 --- a/skills/mainline/SKILL.md +++ b/skills/mainline/SKILL.md @@ -1,6 +1,6 @@ --- name: mainline -description: 叙事主线核查（测量原件，只产 finding，不改稿）：建 paper-level purpose record 与 contribution graph，再按 cold reader 回答七问，定位读者需要回溯、补隐藏上下文或在竞争解读间抉择的地方。findings 走 sci-paper.feedback.v1；不假设三幕模板，多贡献论文可有多个明确相关的分支。可单独运行，也是 /sci-paper:paper-review 维度 E 的唯一来源。Use for: 叙事主线, 主线检查, contribution graph, 冷读七问, narrative spine, 文章结构是否讲得通, 读者会不会卡住。 +description: 叙事主线核查（测量原件，只产 finding，不改稿）：建 paper-level purpose record 与 contribution graph，再按 cold reader 回答八问，定位读者需要回溯、补隐藏上下文或在竞争解读间抉择的地方。findings 走 sci-paper.feedback.v1；不假设三幕模板，多贡献论文可有多个明确相关的分支，但都从属于唯一的 thesis line。可单独运行，也是 /sci-paper:paper-review 维度 E 的唯一来源。Use for: 叙事主线, 主线检查, contribution graph, 冷读八问, narrative spine, 文章结构是否讲得通, 读者会不会卡住。 disable-model-invocation: false argument-hint: " [--field ]" --- @@ -45,11 +45,12 @@ argument-hint: " [--field ]" - **不预设三幕。** motivation → method → validation 是常见默认，不是所有论文 必须套用的模板。 - **多贡献合法。** 多贡献论文可以有多个明确相关的分支；问题是**关系有没有讲 - 清楚**，而不是图是否恰好只有一个连通分量。 + 清楚**，而不是图是否恰好只有一个连通分量。但每个分支都必须从属于唯一的 thesis + line（标准 §5.4：全文只有一个中心结果，可用一句话说出）。 - **disconnected node 是待查证据，不是自动 fatal。** - **禁手**：用连接词桥接逻辑断链（拿 Furthermore / Moreover 顶替缺失的前提）。 -## 3. 第三步：cold reader 七问 +## 3. 第三步：cold reader 八问 按第一次读的读者依次回答： @@ -59,9 +60,10 @@ argument-hint: " [--field ]" 4. 各自的方法 / 论证是什么？ 5. 各自的关键证据是什么？ 6. take-home 与 scope 是什么？ -7. 读者在哪里需要**回溯**、**补隐藏上下文**，或在**竞争解读**之间抉择？ +7. thesis line 是什么？能否用一句话说出唯一的中心结果，且每个贡献分支都从属于它？ +8. 读者在哪里需要**回溯**、**补隐藏上下文**，或在**竞争解读**之间抉择？ -前六问答不上来，就是主线本身的缺陷，而不是读者的问题。第七问的每个落点都是 +前七问答不上来，就是主线本身的缺陷，而不是读者的问题。第八问的每个落点都是 一条候选 finding。 ## 4. 后果分类 @@ -69,13 +71,14 @@ argument-hint: " [--field ]" | 情形 | 类别 | |---|---| | 矛盾，或关键 claim 缺必要支撑论证 | `integrity_blocker` | -| 高曝光、可复现的困惑 + 能给出具体修复 | strong `advisory` | +| 高曝光、可复现的困惑 + 能给出具体修复 | `advisory`，排在最前 | | 局部措辞、可读性偏好 | `advisory` | - 缺失支撑关键 claim 的必要论证可升为 `integrity_blocker`。 -- 其余 narrative finding 按证据强度分 strong 或 ordinary advisory。 -- **不得要求所有结构 advisory 归零。** strong advisory 要 disposition， - 普通 residual 报告即可。 +- 其余 narrative finding 都是普通 advisory，按证据强度和读者曝光排序（标准 §4）。 + 不升 strong：strong 要求效应超出校准过的操作点，冷读没有操作点。 +- **不得要求所有结构 advisory 归零。** 普通 residual 报告即可；作者可以逐条处置， + 但这不是收尾条件。 ## 5. 边界 —— 本 skill 不做什么 @@ -101,18 +104,18 @@ argument-hint: " [--field ]" ## Contribution graph | node | type | file:line | edges (type → target) | -## Cold-read seven questions +## Cold-read eight questions | # | question | answer or gap | file:line | ## Findings | id | kind | location | reader confusion | proposed fix | disposition | ``` -不打印 PASS/FAIL 行。终止态是七问都有明确答案或明确记录的缺口。 +不打印 PASS/FAIL 行。终止态是八问都有明确答案或明确记录的缺口。 ## 7. 反模式 -- ❌ "整体看结构还行" —— 没有 purpose record 和七问逐条答案的总评无效。 +- ❌ "整体看结构还行" —— 没有 purpose record 和八问逐条答案的总评无效。 - ❌ 拿摘要或 intro 概括代替全文冷读。 - ❌ 因为图不是单连通就判 fatal。 - ❌ 要求每篇论文都是 motivation → method → validation。 diff --git a/skills/paper-review/SKILL.md b/skills/paper-review/SKILL.md index c33536d..641eb00 100644 --- a/skills/paper-review/SKILL.md +++ b/skills/paper-review/SKILL.md @@ -1,6 +1,6 @@ --- name: paper-review -description: Strict, source-traced scientific paper review. Runs all A-R dimensions under the SCIPAPER_STANDARD feedback contract: mathematics, physics, logic and statistics, language and de-AI, document structure and narrative spine (contribution graph plus a seven-question cold read), citation existence and relevance, data and figures, interfaces, redundancy, reproducibility, modern-physics checks, cross-section consistency, adversarial derivation verification (three passes plus twelve-framing escalation to CONFIRMED / REFUTED / MARGINAL), staleness and drift, process artifacts, internal draft language, citation precision, and glossary alignment. Scientific-integrity blockers must be resolved, L0 targets must reach zero, and every strong advisory needs an explicit disposition; ordinary editorial advisories are never disguised as a universal paper PASS/FAIL. Use for: pre-submission audit, full manuscript review, source tracing, mathematics and physics re-derivation, 投稿前 audit, 完整论文审查, 严格迭代修订. +description: Strict, source-traced scientific paper review. Runs all A-R dimensions under the SCIPAPER_STANDARD feedback contract: mathematics, physics, logic and statistics, language and de-AI, document structure and narrative spine (contribution graph plus an eight-question cold read), citation existence and relevance, data and figures, interfaces, redundancy, reproducibility, modern-physics checks, cross-section consistency, adversarial derivation verification (three passes plus twelve-framing escalation to CONFIRMED / REFUTED / MARGINAL), staleness and drift, process artifacts, internal draft language, citation precision, and glossary alignment. Scientific-integrity blockers must be resolved, L0 targets must reach zero, and every strong advisory needs an explicit disposition; ordinary editorial advisories are never disguised as a universal paper PASS/FAIL. Use for: pre-submission audit, full manuscript review, source tracing, mathematics and physics re-derivation, 投稿前 audit, 完整论文审查, 严格迭代修订. disable-model-invocation: false argument-hint: " [--max-iter N] [--no-fix] [--skip-final-physics] [--orchestrated] [--field ]" --- @@ -39,7 +39,7 @@ measure → type → rank → edit → re-measure → disposition M.pass-1/2/3；任一命中后修复并重跑全部三 pass。 8. **后果类别不能混用。** - `integrity_blocker`: 科学、来源、引用、构建或必需制品错误；不可由风格偏好豁免。 - - `l0_target`: Tier A、em-dash、同 section 同 Tier B 词的第 2 次及以后。 + - `l0_target`: Tier A、em-dash、同一 heading 单元内同一 Tier B 词的第 2 次及以后。 - `advisory`: editorial、narrative、结构、corpus distance、learned field-similarity。 9. **缺失测量不等于 0。** 所有 axes 显式标 `measured`、`degraded`、 `unmeasured` 或 `not_applicable`。 @@ -143,30 +143,32 @@ uncertainty、多重比较、prior）与**声明-证据纪律**（动词强度 几何、burstiness、UID、document shape、learned field-similarity、-ing 分词尾巴与 阐释式冒号（linter `ing-tail` / `colon-elaboration`）是 advisory。 - 词汇层两条证据来自语料而非词表：`register-zero:`（全文逐词零命中审计， - strong 除非是已证实词干的机械派生）与 `collocation-novel:
`（一句中相邻 - 常用词对在语料里从未同现的比例，如 `binned kernel`）。零命中 strong 必须走 + strong 除非是已证实词干的机械派生、本文首次使用处定义的术语或专有名词）与 + `collocation-novel:
`（一句中相邻常用词对在语料里从未同现的比例，如 + `binned kernel`）。零命中 strong 必须走 de-ai §4.2 的三种 disposition 之一（本文定义 / 首次引入并引用 / 换成领域词）。 - `deai_structure` 的 auxiliary 家族（paper-as-agent 主语、wh-cleft 开头、三词以上 且含两个连字符复合的修饰堆）只命名句子，不进 template score；命中是 advisory。 - learned signals 只表示 field-similarity/triage，不证明作者身份；没有 calibrated operating point 时必须 `degraded`。 -- **结构 tell 审计（每轮必跑；`/sci-paper:de-ai` `--audit-only` 模式，其 Pass 2）**： +- **结构 tell 审计（`/sci-paper:de-ai` `--audit-only` 模式，其 Pass 2；单独运行时每轮必跑，`--orchestrated` 时标 `SKIPPED_FOR_ORCHESTRATOR`、由父级独立启动）**： 对全文执行 humanizer Layers 1--2 结构清单——2.11 长句堆叠（约 30 词以上、 3 层以上从句链或双层括号嵌套）、negative parallelism（"X, not Y" 模板密度）、 elegant variation（同一对象换名）、rule-of-three padding、公式化开场白、 - connective 连排。结构命中为 advisory（同类超密度升 strong advisory）；该 skill + connective 连排。结构命中为普通 advisory（同类超密度只把它排在普通 advisory + 最前，不升 strong：人工清单没有校准过的操作点，标准 §4）；该 skill 的 Layer-4 claim--evidence 命中并入上方 §C 纪律（`integrity_blocker`）。其 Corpus overrides 优先于上游词表：landscape / demonstrate / significantly 不做词表级标记，只走证据条件。 - 修复 de-AI finding 时遵守 `/sci-paper:paper` 的 Preserve List：不得把证据绑定的 hedging 改强，不得为避词改动数字、引用或 stance——制造 over-claim 的"修复" 本身构成 claim-evidence defect。 -- 不能要求所有结构 advisory 归零。strong advisory 要 disposition，普通 residual 要报告。 +- 结构 advisory 不要求归零，处置按 §0 第 10 条。 ### E. Document structure and narrative spine **叙事主线协议委派 `/sci-paper:mainline`**——purpose record、contribution graph、 -cold reader 七问与它们的后果分级都由该 skill 拥有，本维度不复述。 +cold reader 八问与它们的后果分级都由该 skill 拥有，本维度不复述。 本维度自己拥有的是**章节层面的编排**：依赖顺序、section function、abstract coverage、intro promise、results/discussion/conclusion closure、图表首次引用位置。 @@ -190,7 +192,8 @@ coverage、intro promise、results/discussion/conclusion closure、图表首次 - abstract、正文、table、caption、figure、conclusion 同名数字与单位一致。 - table 逐 cell 与上游数据对位。 - **figure 制品本身的检查调用 `/sci-paper:figure-review`**（150 DPI 编译页上的曲线、 - 点、误差棒、轴、单位、legend、caption 与画布平衡），不在此复述。本维度只负责 + 点、误差棒、轴、单位、legend、caption 与画布平衡），不在此复述；`--orchestrated` + 时该调用标 `SKIPPED_FOR_ORCHESTRATOR`，由父级独立启动 figure-review。本维度只负责 数字与 caption 声明的**溯源与一致性**。 - baseline 比较使用相同 protocol；uncertainty 的定义与样本量明确。 - mismatch、stale result、missing required artifact 或不可复现 claim 为 @@ -227,7 +230,9 @@ coverage、intro promise、results/discussion/conclusion closure、图表首次 - de-AI review 不得覆盖物理判断。 - 不适用项由 physics 标 `not_applicable`，不能虚构检查结果。 -- 隔离冷读模式见 §5。 + +单独运行 paper-review 时在本维度内联执行 physics protocol，收尾另跑 §5 隔离冷读； +`--orchestrated` 时标 `SKIPPED_FOR_ORCHESTRATOR`，由父级独立启动 physics。 ### L. Systemic consistency @@ -239,11 +244,12 @@ coverage、intro promise、results/discussion/conclusion closure、图表首次 ### M. Adversarial verification -**M.1 — Three-pass derivation verification.** 对每个关键推导块和数值结论独立执行： +**M.1 — Three-pass derivation verification.** 对每个关键推导块和数值结论独立执行。 +pass-1 与 pass-3 的检查项归 `/sci-paper:physics`，本维度不复述，只拥有三 pass 的纪律与 pass-2： -- **pass-1:** 量纲、代数、近似、边界、数值复算。 +- **pass-1:** 按 physics P1、P5、P6 从源重做（`--orchestrated` 时由父级的 physics 运行承担）。 - **pass-2:** 以对手 reviewer 视角提出最强的具体反驳，并用当前证据回答。 -- **pass-3:** 守恒、对称、渐近、概率与统计前提。 +- **pass-3:** 按 physics P2、P3、P4 从源重做（同上）。 规则： @@ -269,7 +275,7 @@ coverage、intro promise、results/discussion/conclusion closure、图表首次 CONFIRMED 自动降 MARGINAL。verdict 只表证据状态；consequence class 按 §C claim-evidence 影响单独赋予，CONFIRMED 的纯 editorial critique 仍是 advisory。 - M.2 在进程内执行、不 spawn 子代理；需要大宽度并行探索时移交用户级 - `cc-tree:attack`（同一方法学的通用引擎），结论以 finding 形式回流本审查。 + `cc-tree:attack`（若已安装；同一方法学的通用引擎），结论以 finding 形式回流本审查。 ### N. Staleness and drift @@ -342,20 +348,19 @@ stale scientific content、冲突副本或 required artifact drift 是 `integrit 2. 对每个 blocker 修根因；对每个 L0 做最小有效修改。修复遵守标准 §5.3 长度纪律：默认删减或精简，被修段落净增长必须有科学必要性理由， 报告给出每段字数差。解释性补丁（对被标记文本追加说明而非重写）是缺陷。 -3. 对 strong advisory：行动、接受、验证为 false positive，或带原因 pending。 -4. ordinary advisory 不要求消失；保留并报告即可。 -5. 每个 edit 后重新 Read 相关上下文。 -6. 改公式/数字/物理链后重跑 M 三 pass；改 figure/table 后重新查看制品与来源； +3. strong 与 ordinary advisory 按 §0 第 10 条处置。 +4. 每个 edit 后重新 Read 相关上下文。 +5. 改公式/数字/物理链后重跑 M 三 pass；改 figure/table 后重新查看制品与来源； 改 citation 后重新读原文；改 prose 后重新跑 shared linter。 -7. 重跑权威 build 和所有受影响测试/脚本。 -8. **收尾前跑 length gate**（标准 §5.3 机械执行）： +6. 重跑权威 build 和所有受影响测试/脚本。 +7. **收尾前跑 length gate**（标准 §5.3 机械执行）： `python tools/length_gate.py --before /length-baseline.tex` （或 `--git-ref `）。exit 1 = 存在无理由净增长——精简回预算内， 或用 `--allow "
=<理由>"` 记录作者批准的理由后重跑至 exit 0。 `length-growth` finding 未 disposition 时循环不得收尾。本轮若执行了 condense，加 `--require-shrink `： 删减未达目标产生 strong `length-shrink-short` 且 exit 1。 -9. **收尾前跑 residue diff**：`python tools/deai_residue.py --before +8. **收尾前跑 residue diff**：`python tools/deai_residue.py --before /length-baseline.tex`；exit 1（strong residue）时循环不得收尾。 伪代码： @@ -363,10 +368,7 @@ stale scientific content、冲突副本或 required artifact drift 是 `integrit ```text for iter in 1..max_iter: report = full_A_to_R_review_from_current_sources() - if no integrity_blocker \ - and no l0_target \ - and no UNDER_SCRUTINY derivation \ - and every strong advisory has a disposition: + if stopping_semantics_hold(report): # 条件见 §7 stop with DISPOSITION_COMPLETE if no_fix: stop with REVIEW_ONLY @@ -389,10 +391,9 @@ isolated worktree agent，重新读取 `/sci-paper:physics` skill 并 cold-read - `--skip-final-physics`: 仅用户显式选择时跳过，报告 `unmeasured`/`SKIPPED_BY_USER`。 - `--orchestrated`: 仅供 final-review 等父 orchestrator 避免 nested agent。physics、 - mainline、logic 三个测量原件改由父级在同级独立启动；本 skill 对应维度报告 - `SKIPPED_FOR_ORCHESTRATOR`。 -- isolated review 产生的 critique 先判断 evidentiary verdict，再单独赋 consequence class。 - CONFIRMED editorial critique 不自动成为 blocker。 + mainline、logic、figure-review 与 de-ai `--audit-only` 五个测量原件改由父级在同级 + 独立启动；本 skill 对应维度（D、G 为维度内的该调用）报告 `SKIPPED_FOR_ORCHESTRATOR`。 +- isolated review 的 critique 按 §2 开头与 M.2 的规则赋 consequence class。 - 新 blocker/L0/strong advisory 回注主循环；agent failure 必须显式报告，不能伪装通过。 ## 6. Report contract @@ -448,6 +449,8 @@ Review may stop as disposition-complete only when: - required build and artifacts are valid; - the length gate (standard §5.3) exits 0 against the loop's pre-edit baseline, every recorded growth justification included in the report; +- the residue diff (`deai_residue.py --before`, standard §5.3) exits 0 against + the same baseline, so no strong residue finding remains; - every strong advisory has an explicit disposition or a stated pending reason; - ordinary advisories and unavailable axes are reported. diff --git a/skills/paper/SKILL.md b/skills/paper/SKILL.md index 1bb8493..ddc48cf 100644 --- a/skills/paper/SKILL.md +++ b/skills/paper/SKILL.md @@ -105,14 +105,10 @@ deleting its hedges has been damaged. - ✅ "Equation (3) gives Y, which we then evaluate at $z = 0.5$." - ✅ "We adopt $H_0 = 67.4\,\mathrm{km\,s^{-1}\,Mpc^{-1}}$." -读者**绝对不该**看到（无论初稿还是修订稿）： +读者**绝对不该**看到（无论初稿还是修订稿；修订场景的更多例句见下方 Patch/Structural 对照表）： - ❌ "我们一开始尝试了 A，发现 A 不对，所以改用 B" -- ❌ "We did A, found it gave incorrect results, and therefore switched to B" - ❌ "Method A would have suggested X, but in fact ..."（除非这是本文有意做的 baseline 对照，见下"唯一例外"） -- ❌ "An earlier version of Eq. (5) had a factor-of-2 error; the corrected form is ..." -- ❌ "Note: the previous draft used $H_0 = 70$; the present analysis uses $H_0 = 67.4$." -- ❌ "After several iterations we converged on the following architecture." **Patch（禁）vs Structural（要）—— 修订时的硬区分** @@ -161,8 +157,7 @@ deleting its hedges has been damaged. ### 改写不堆叠 / Condense, Do Not Accumulate -> 规范条文：`docs/SCIPAPER_STANDARD.md` §5.3。用户规则 2026-07-16： -> "改写、删减、精简，而不是堆叠！" +> 规范条文：`docs/SCIPAPER_STANDARD.md` §5.3（改写、删减、精简，而不是堆叠）。 - 每次修改的默认方向是**更短**。优先级：删除 > 原位精简 > 等长改写 > 增长。 - 增长只有两种合法理由：用户要求的新内容，或来源可验证的科学必需 @@ -171,7 +166,7 @@ deleting its hedges has been damaged. 而不是重写句子本身。上节 forward narrative 禁止堆叠**状态**；本节禁止 堆叠**字数**。 - 每处修改报告字数差；靠加字消除 detector 信号是缺陷，不是修复。 -- **机械执行**（标准 §5.3 v3.3）：改写候选经 `rewrite_reward.py --original` +- **机械执行**（标准 §5.3）：改写候选经 `rewrite_reward.py --original` 硬门（超长即 `-inf`）；编辑循环收尾经 `length_gate.py` delta 门 （无理由净增长 = exit 1，循环不得收尾）。增长的唯一合法路径是 `--allow`/`--allow-growth` 记录的作者批准理由。 @@ -249,7 +244,7 @@ python tools/ai_ism_lint.py --field \ --format json --output /writing-feedback.json ``` -- Tier A、em-dash 和超过每节每词 cap 的 Tier B 是 `l0_target`。 +- Tier A、em-dash 和超过每 heading 单元每词 cap 的 Tier B 是 `l0_target`。 - 句式模板、burstiness、UID、document shape 与 learned field-similarity 是 advisory；必须保留 `measured` / `degraded` / `unmeasured` / `not_applicable` 区别，不能把缺失测量当作零命中。 @@ -268,7 +263,7 @@ python tools/ai_ism_lint.py --field \ **Tier A — L0 target** -正文命中必须重写。canonical set： +正文命中必须重写。规范词表是 `tools/ai_ism_lint.py` 的 `TIER_A_PATTERN`（段首套话对应 `TIER_A_OPENER_PATTERN` 与 `TIER_A_PARAGRAPH_CONNECTOR_PATTERN`），下表是它的人读镜像： | 类别 | 词 | |---|---| @@ -277,27 +272,22 @@ python tools/ai_ism_lint.py --field \ | 名词类 | `tapestry`, `testament`, `realm / realms` | | 段首套话 | `Recent advances in...`, `Despite significant progress...`, `With the advent of...`, `In recent years,...`, `It is worth noting`, 段首 `Crucially,`, `Importantly,`, `Notably,`, `Interestingly,` | -2026-07-16 扩充（`underscore*`, `pivotal`, `tapestry`, `testament`, `realm*`） -采自 academic-humanizer 词表（MIT，见文末 Provenance），并经两域 curated -corpus 复核为 0 出现后才入 Tier A；`landscape` 虽在其词表中，但它是本领域 -正当术语（detection landscape 等，corpus 高频），**不入表**。 +`landscape` 是本领域正当术语（detection landscape 等，corpus 高频），**不入表**。 替换原则：使用直接、具体、可核验的动词或范围，不做机械同义词交换。例如 `leverages X to Y` → `uses X to Y`，`pave the way for` → `enable`， `comprehensively` → 明确列出覆盖范围。 -**Tier B — per-section/per-word cap** +**Tier B — per-heading-unit/per-word cap** Tier B 可以使用，但同一个 Tier B 词在同一 heading 单元（`\section` 与 `\subsection` 各算一个单元，linter 按此计）最多出现 1 次。第 2 次及 以后是 `l0_target`；cap 内的出现不是 finding。规范词表是 -`tools/ai_ism_lint.py` 的 `TIER_A_PATTERN` / `TIER_B_PATTERN`，本表是它的 -人读镜像，常见项包括 `Furthermore`, `Moreover`, `Additionally`, -`robust/robustly`, `comprehensive`, `utilize/utilized`, `leverage`, -`Importantly`, `Interestingly`, `Notably`, `intricate`, -`foster/fosters/fostering/fostered`（后两组 2026-07-16 加入；curated corpus -各有 1 次出现，非零故不入 Tier A）。经验频率只从当前 profile 读取，不在 -本文件复制。优先用直接陈述或可验证数字，但不要为了避词而损害准确性。 +`tools/ai_ism_lint.py` 的 `TIER_B_PATTERN`，本表是它的人读镜像：`Furthermore`, +`Moreover`, `Additionally`, `robust/robustly`, `comprehensive`, +`utilize/utilized`, `leverage`, `Importantly`, `Interestingly`, `Notably`, +`intricate`, `foster/fosters/fostering/fostered`。经验频率只从当前 profile +读取，不在本文件复制。优先用直接陈述或可验证数字，但不要为了避词而损害准确性。 `robust` 紧接 `estimator(s)` / `estimation` / `statistic(s)` / `regression` 时是 robust statistics 的方法名，不计入 Tier B（跨行也算）；`robust estimate` 仍计入，因为它同样常指"可靠的估计"。 @@ -329,7 +319,7 @@ Tier B 可以使用，但同一个 Tier B 词在同一 heading 单元（`\sectio - **阐释式冒号（colon-appositive）**：`X: the rule that ...` 这类 "名词: 展开" 结构是 `X — that is, Y` 的冒号变体。改写为限定从句、 两个句子或 ", so ..."；caption 标签（`Left: ...`）与真正的列表规格 - 说明可保留（linter `colon-elaboration` advisory；用户规则 2026-07-16）。 + 说明可保留（linter `colon-elaboration` advisory）。 结构模式必须结合 section、样本量、calibration 和科学功能判断。技术列表若编码真实 分类，不应为了制造参差而破坏可读性。 @@ -346,13 +336,12 @@ Tier B 可以使用，但同一个 Tier B 词在同一 heading 单元（`\sectio grep -n -E '—|---|\\textemdash' main.tex # Tier A（必删；正文中不允许出现，包括变体） -grep -n -E -i '(delve|leveraged|leverages|leveraging|paved?|paves|paving|shed[s]?|shedding|showcase[sd]?|showcasing|seamless(ly)?|holistic(ally)?|comprehensively|crucially|utilizes|utilizing|underscor(e|es|ed|ing)|tapestry|testament|pivotal|realms?|recent advances|despite significant|with the advent|in recent years|it is worth)' main.tex +grep -n -E -i '(delv(e|es|ed|ing)|leveraged|leverages|leveraging|paved?|paves|paving|shed[s]?|shedding|showcase[sd]?|showcasing|seamless(ly)?|holistic(ally)?|comprehensively|crucially|utilizes|utilizing|underscor(e|es|ed|ing)|tapestry|testament|pivotal|realms?|recent advances|despite significant|with the advent|in recent years|it is worth)' main.tex # 段首套话（Tier A）：只有位于段落第一行时才算段首；linter 按段落判定，不按物理行 grep -n -E '^\s*(Importantly|Interestingly|Notably|Crucially),' main.tex # Tier B（定位；是否超过每 heading 单元每词 cap 由 linter 计算） -grep -n -E -i '^\s*(Furthermore|Moreover|Additionally),' main.tex -grep -n -E -i '\b(robust|robustly|comprehensive|utilize|utilized|leverage|importantly|interestingly|notably|intricate|foster(s|ing|ed)?)\b' main.tex +grep -n -E -i '\b(furthermore|moreover|additionally|robust|robustly|comprehensive|utilize|utilized|leverage|importantly|interestingly|notably|intricate|foster(s|ing|ed)?)\b' main.tex # 顽固替换组（不分级） grep -n -E -i '\b(in order to|aim to|facilitate|serves as)\b' main.tex @@ -363,7 +352,7 @@ grep -n -E '([A-Za-z0-9]|\}): [a-z$\\]' main.tex ``` **Tier A / em-dash 残留** = `l0_target`。 -**Tier B 超频** = 同词在同 section 的第 2 次及以后为 `l0_target`。 +**Tier B 超频** = 同词在同一 heading 单元内的第 2 次及以后为 `l0_target`。 > **Companion evidence from `/sci-paper:de-ai` calibration:** corpus assets supply > descriptive frequencies and calibration. Re-run `python tools/extract_style.py` @@ -373,8 +362,7 @@ grep -n -E '([A-Za-z0-9]|\}): [a-z$\\]' main.tex ### Claim–Evidence Discipline / 声明-证据纪律 > QD 类规则（claim-evidence defects 在 SCIPAPER_STANDARD §2 QD 下是 -> `integrity_blocker`）。本节给出操作化检查；条目改编自 academic-humanizer -> Layer 4（MIT，见文末 Provenance），并按天体物理语料重校准。 +> `integrity_blocker`）。本节给出操作化检查。 对每个经验性声明检查两件事：(a) 它是否有正文内的数字、图、表或引用支撑； (b) 动词强度是否不超过证据强度。 @@ -393,15 +381,14 @@ grep -n -E '([A-Za-z0-9]|\}): [a-z$\\]' main.tex 区间优于单一均值（除非均值方法已声明）；每个数字注明方法、指标、基线。 做比较时先打最强对手，不打平凡基线。 - **`significantly` 必须有伴随检验或数字**；孤立的 "significantly better" - 是声明缺陷。注意这是**证据条件规则，不是词法禁令**：astro curated corpus - 中 `demonstrate*`（0.147/1k）与 `significantly`（0.274/1k，合并语料实测 - 2026-07-16）都是正常用词，禁词式移植（ML 会议口味）会误伤本领域写作—— + 是声明缺陷。注意这是**证据条件规则，不是词法禁令**：`demonstrate*` 与 + `significantly` 在本领域 curated corpus 中都是正常用词，禁词会误伤本领域写作； 只有"动词/副词超出证据"才构成 finding。 ### 防过度纠正 / Preserve List（rewrite 护栏） -> 改编自 academic-humanizer Layer 3（MIT）。De-AI 重写循环的反向风险是 -> "把校准的 hedging 改强"——这会**制造** over-claiming，比留下 AI 词更糟。 +> De-AI 重写循环的反向风险是"把校准的 hedging 改强"——这会**制造** +> over-claiming，比留下 AI 词更糟。 > 与 SCIPAPER_STANDARD §6 rewrite eligibility（stance/modality/qualifier > 不可变）同源；此处是写作端明细。 @@ -436,12 +423,5 @@ grep -n -E '([A-Za-z0-9]|\}): [a-z$\\]' main.tex ### Provenance / 借鉴出处 -The 2026-07-16 additions (Tier A/B word extensions, `serves as`, the --ing-tail and colon-elaboration structure tells, the Claim–Evidence -Discipline section, and the Preserve List) adapt material from -[academic-humanizer](https://github.com/AIScientists-Dev/academic-humanizer) -(MIT License, Copyright (c) 2026 AIScientists-Dev, itself building on -blader/humanizer, MIT). Every lexical adoption was re-verified against the -curated field corpora before tier assignment; venue-specific rules that -conflict with astro usage (`landscape`, blanket `demonstrate`/ -`significantly` bans) were deliberately not adopted. +部分条目改编自 MIT 许可的外部项目；出处、版权声明、日期与采纳边界见仓库根目录 +`ACKNOWLEDGMENTS.md`。 diff --git a/skills/physics/SKILL.md b/skills/physics/SKILL.md index 9958d77..3b85462 100644 --- a/skills/physics/SKILL.md +++ b/skills/physics/SKILL.md @@ -116,9 +116,12 @@ argument-hint: " [--field ]" | 类别 | 本 skill 的来源 | |---|---| | `integrity_blocker` | 量纲错、物理错、推导错、数值不一致、编译失败、未声明的必要假设、伪造引用 | -| strong `advisory` | 命名/单位混用、精度不一、bracketing 超出物理域、schematic 公式无标注 | +| `advisory`，排在最前 | 命名/单位混用、精度不一、bracketing 超出物理域、schematic 公式无标注（仅限量纲闭合时；不闭合者按 P1 为 `integrity_blocker`） | | `advisory` | 纯表述偏好 | +两行都是普通 advisory：strong 要求效应超出校准过的操作点（标准 §4），人工核查没有 +操作点。第一行是有出处可查的清晰度缺陷，按标准 §4 第 3 条排在第二行之前。 + 每条 finding 带 measurement state。不适用的检查标 `not_applicable` 并说明为什么 不适用；拿不到证据标 `unmeasured` 并说明缺什么。 diff --git a/skills/proposal-polish/SKILL.md b/skills/proposal-polish/SKILL.md index d4efb5f..9f3b31d 100644 --- a/skills/proposal-polish/SKILL.md +++ b/skills/proposal-polish/SKILL.md @@ -97,7 +97,9 @@ most editing effort there. to the research: named programs, named courses or tools, measurable outreach. - **Hedged central hypothesis.** The Aims-page hypothesis is a falsifiable commitment, not "we will explore whether possibly ...". Calibrated hedging - belongs in the Approach's interpretation, not in the central claim. + belongs in the Approach's interpretation, not in the central claim. Firming + a hedged hypothesis changes stance and modality (§6), so propose the committed + wording to the author and apply it only with their approval, never silently. ## Preserve and deploy (funded-proposal craft) @@ -135,23 +137,25 @@ These read as strength; keep or add them rather than editing them out. `python tools/ai_ism_lint.py --field ` (skip the field flag when no profile applies; structural axes calibrated on journal corpora are `degraded` for proposals and advisory-only). -3. **Rewrite.** Same aims, same evidence, same citations. Fix weak moves, +3. **Rewrite.** Same aims, same evidence, same citations. Fix weak moves + (a firmer central hypothesis stays a proposal until the author approves it), attach feasibility footings beside ambitious claims, keep the vision, clear L0 targets, and edit the score-forming pages hardest. Rewrite-eligibility invariants (SCIPAPER_STANDARD §6) hold: no number, citation, stance, or qualifier changes without the author's source. 4. **Report.** Cleaned text plus a change log: weak moves fixed by type, feasibility gaps flagged for the author (with what evidence would close - them), L0 before/after counts, and any structural expectations the draft - still misses. Confirm no number, result, or citation was altered and no - support was invented. + them), any central-hypothesis wording proposed for approval, L0 before/after + counts, and any structural expectations the draft still misses. Confirm no + number, result, or citation was altered and no support was invented. ## Anti-patterns - Flattening the long-term vision because paper mode would trim it. - "Improving" a proposal by inventing preliminary results, partners, letters, or funding history: that is fabrication, not editing. -- Leaving a hedged central hypothesis because hedging is legitimate in papers. +- Leaving a hedged central hypothesis unflagged because hedging is legitimate in + papers, or firming it without the author's approval. - Spending effort on later sections while the Aims/Summary page is weak. - Treating funder page limits, formatting, and deadline rules from memory as current: consult the funder's live policy documents (NSF PAPPG, NIH From 4dfa9a41ba5f4db96907882abc92cd1310c77ed7 Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 12:32:39 -0400 Subject: [PATCH 25/27] docs: every figure re-taken on the 2026-09-27 rebuild, and the audit's document items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `wgl` profile was rebuilt on the corpus-side fixes of this release, and each figure below was re-taken through the shipped path rather than carried forward: - both READMEs' asset rows: the exemplar bank (28,444 paragraphs from 18 of the 19 curated and 498 of the 500 reference papers), register passages and terms, pooled UID, the structure and salience bucket counts, the document-structure baseline (504 documents, strata edges 49 and 78) and the collocation bank (535,517 pairs); - the document-scale table and a new §9 notice. The 2026-08-25 notice, whose tables repeated §9.1-§9.8, gives way to it (R6); four changes between the two measurements are named and not separated; - the L3 retrain (45,153 records; §7.0 retrain equivalence), the held-out register row under the E17 estimator, hedging's §19 re-measurement, and the worked example in examples/README.md. Audit 2026-09-27 document items: G4, G6, G12, G28, G29, G30, G31, G34, H34, H35, E27, F15, I26-I29, I37, M3/I35 (the 2026-09-04 review's per-item dispositions), B24 and B25 where the docs described them, the L0 mirror in tools/README and the standard (H8), and R1-R10: each measurement keeps one home and the other places point to it. The suite size reads 801 tests in 32 files. CHANGELOG gains the audit remainder and the rebuild; v0.35.0-v0.35.1 move to CHANGELOG-ARCHIVE-v0.35.md because the live file had reached 806 lines. latency.json, which nothing read, is removed (M4/G19). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG-ARCHIVE-EARLY.md | 1 + CHANGELOG-ARCHIVE-RECENT.md | 1 + CHANGELOG-ARCHIVE-v0.33-v0.34.md | 1 + CHANGELOG-ARCHIVE-v0.35.md | 148 +++++++++++ CHANGELOG-ARCHIVE.md | 1 + CHANGELOG.md | 236 +++++++----------- README.md | 148 +++++------ README.zh-CN.md | 139 +++++------ docs/README.md | 13 +- docs/SCIPAPER_STANDARD.md | 24 +- docs/architecture/DEAI_SUBSYSTEM.md | 82 +++--- docs/architecture/DISPOSITIONS.md | 29 +-- docs/architecture/EVALUATION.md | 130 +++------- docs/architecture/RESPONSIBILITIES.md | 16 +- .../evaluation/discourse-and-citation.md | 146 ++++++----- .../architecture/evaluation/document-scale.md | 201 +++++---------- .../evaluation/held-out-labels.md | 71 +++--- docs/architecture/evaluation/learned-model.md | 172 ++++++------- .../evaluation/lexical-structure-uid.md | 101 +++++--- .../evaluation/narrative-salience-register.md | 126 ++++------ .../evaluation/vocabulary-and-residue.md | 134 +++++----- docs/audits/codex-review-2026-09-04.md | 28 +++ examples/README.md | 37 ++- latency.json | 14 -- style-corpus/wgl/README.md | 16 +- style-profile/README.md | 34 +-- tools/README.md | 49 ++-- 27 files changed, 1028 insertions(+), 1070 deletions(-) create mode 100644 CHANGELOG-ARCHIVE-v0.35.md delete mode 100644 latency.json diff --git a/CHANGELOG-ARCHIVE-EARLY.md b/CHANGELOG-ARCHIVE-EARLY.md index 9fbc2e7..5a82641 100644 --- a/CHANGELOG-ARCHIVE-EARLY.md +++ b/CHANGELOG-ARCHIVE-EARLY.md @@ -6,6 +6,7 @@ The earliest history of the plugin, from the first release through v0.21.0 is edited; the history is verbatim. - Current releases: [CHANGELOG.md](CHANGELOG.md) +- **v0.35.0 through v0.35.1**: [CHANGELOG-ARCHIVE-v0.35.md](CHANGELOG-ARCHIVE-v0.35.md) - **v0.33.0 through v0.34.0**: [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md) - **v0.27.1 through v0.32.0**: [CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md) - **v0.22.0 through v0.27.0**: [CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md) diff --git a/CHANGELOG-ARCHIVE-RECENT.md b/CHANGELOG-ARCHIVE-RECENT.md index c195ae8..0a8732c 100644 --- a/CHANGELOG-ARCHIVE-RECENT.md +++ b/CHANGELOG-ARCHIVE-RECENT.md @@ -6,6 +6,7 @@ live changelog passed the repository's 750-line budget. Nothing here is edited; history is verbatim. - Current releases: [CHANGELOG.md](CHANGELOG.md) +- **v0.35.0 through v0.35.1**: [CHANGELOG-ARCHIVE-v0.35.md](CHANGELOG-ARCHIVE-v0.35.md) - **v0.33.0 through v0.34.0**: [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md) - **v0.22.0 through v0.27.0**: [CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md) - **v0.21.0 and earlier**: [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md) diff --git a/CHANGELOG-ARCHIVE-v0.33-v0.34.md b/CHANGELOG-ARCHIVE-v0.33-v0.34.md index d71d126..75935d4 100644 --- a/CHANGELOG-ARCHIVE-v0.33-v0.34.md +++ b/CHANGELOG-ARCHIVE-v0.33-v0.34.md @@ -6,6 +6,7 @@ archive was within 65 lines of it. Nothing here is edited; the history is verbatim. - Current releases: [CHANGELOG.md](CHANGELOG.md) +- **v0.35.0 through v0.35.1**: [CHANGELOG-ARCHIVE-v0.35.md](CHANGELOG-ARCHIVE-v0.35.md) - **v0.27.1 through v0.32.0**: [CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md) - **v0.22.0 through v0.27.0**: [CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md) - **v0.21.0 and earlier**: [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md) diff --git a/CHANGELOG-ARCHIVE-v0.35.md b/CHANGELOG-ARCHIVE-v0.35.md new file mode 100644 index 0000000..f57ae50 --- /dev/null +++ b/CHANGELOG-ARCHIVE-v0.35.md @@ -0,0 +1,148 @@ +# Changelog archive — v0.35.0 through v0.35.1 + +Entries moved out of [CHANGELOG.md](CHANGELOG.md) on 2026-09-27 (v0.39.0) when +the live changelog passed the repository's 750-line budget. Nothing here is +edited; the history is verbatim. + +- Current releases: [CHANGELOG.md](CHANGELOG.md) +- **v0.33.0 through v0.34.0**: [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md) +- **v0.27.1 through v0.32.0**: [CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md) +- **v0.22.0 through v0.27.0**: [CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md) +- **v0.21.0 and earlier**: [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md) + +--- + +## v0.35.1 — 2026-09-03 + +A post-release audit of the documentation, not of the code. An escape the +writer lost had truncated the tools table in both READMEs, the one published +figure with no artifact behind it was wrong, and several figures and dates in +the record had drifted from what they describe. + +### A lost backslash ended the tools table four rows early + +The `tools/tex_macros.py` row wrote `\newcommand` without its backslash, and +the consumed newline split the row, so the table stopped rendering there — +`retrieve_exemplars`, `fetch_arxiv_abstracts`, `train_ai_ism_classifier` and +`extract_md_negatives` fell out of it, in both READMEs. The same damage sat in +the v0.32.0 changelog entry at `\nocite`. + +### The worked example published a count the linter does not report + +`examples/README.md` said 19 total advisories where `ai_ism_lint` reports 18. +It was the only published-figure document with no artifact behind it, so +nothing read it. `tests/test_published_figures.py` now renders both of its +tables — the before/after summary and the per-rule counts — by running the +linter on the two shipped manuscripts and looking for the result, the same way +every other pinned figure is rendered from the artifact it was read from. A +cell the linter does not produce fails the case, so a document that agrees with +a stale run and a document nobody updated fail identically. + +### Figures and dates the record had drifted from + +- **203**, not 200, held-out refereed papers, in the two `DISPOSITIONS.md` rows + that still carried the count from before §18 re-measured it. +- The v0.30.0–v0.33.0 changelog headings were dated 2026-08-27; their tagger + dates are **2026-08-26**. +- The release-gate label in `EVALUATION.md` §12 stamped a version on a suite + size measured after it. It now reads "as of 2026-09-03; last tagged release + v0.35.1", so the version names the tag and the date names the measurement. +- Both README latency tables paired 394 tests with 81.4 s, a wall time taken on + the 393-test suite. The suite row is re-taken: **73.0 s**, median of 3, the + three runs spanning 70.0–86.9 s. Every other row still stands from the + 2026-08-27 take and the preamble now says which rows carry which date. +- `docs/README.md` called the evidence record five files where it is eight, + gave three of them section maps missing the sections added since, counted + four kinds of document where six live there, and described the validator's + version and suite-size checks narrower than they are. +- The unit-pattern anchor in both READMEs pointed at `rewrite_reward.py:41`; + the pattern is at `:56`. +- Both READMEs said CI runs on every push. `ci.yml` filters `branches: [main]`, + so it runs on every push to `main` and every pull request. + +### Also + +- `.gitignore` covers `models/hf-cache/`, `.ce/` and `.ccm/` — relocated model + cache and machine-local tool state, never repository content. +- Suite: 394 tests across 20 files; `validate_plugin.py` 9/9. + +## v0.35.0 — 2026-08-27 + +The axis that counts numbers could not see numbers written as macros, a public +repository was carrying an unpublished manuscript's method summary, and the +worked example that demonstrates any of this now runs on a synthetic paper +instead of a real one. + +### Numbers held in macros were invisible, in both directions + +A manuscript that writes `\newcommand{\Nsamples}{12}` in its preamble and +`\Nsamples{}` in its results put a measured quantity where neither named text +projection could read it. `RE_TEX_SIMPLE_CMD` reduces a command to its +argument, so the use site contributed nothing while the definition site +contributed the digits once, in the preamble, attributed to no reported +section. Two errors running in opposite directions, which is why the net stayed +small enough to go unnoticed: on the manuscript that surfaced it, expanding the +uses adds 650 digits and dropping the definitions removes 493. + +Found by running the tools on a real manuscript, not by a test. The salience +axis had been reporting recital percentiles for that paper computed on 91% of +its digits, and correcting it moved that paper from 16 recital findings to 26. + +`tools/tex_macros.py` expands numeric-literal macros once, on the assembled +document root, because that is the only scope holding both a preamble +definition and a body use — the same reason `read_tex_document` folds +`\include` in the first place. Only a bare numeric literal expands, so +`\newcommand{\Msun}{M_\odot}` and every macro taking an argument are untouched. + +Both salience baselines were rebuilt rather than left to score expanded +manuscripts against an unexpanded reference. They move by **zero to four +decimals** at the p90 and p95 gates, across all three features and all seven +buckets, in both `wgl` and `wgl-letter`: 88.2% of 390 corpus documents never +use the construction. That is what separates a correction from a rescaling, and +it means no published salience figure in `EVALUATION.md` changes (§22). + +### An unpublished manuscript's method summary was in a public repository + +This repository is public. Content from the author's unpublished manuscripts +had accumulated in the evaluation record since 2026-07-12, in increments that +each looked locally harmless — a codename in a test fixture, a macro and its +value in a docstring, one fidelity-preserving rewrite quoted in full. + +The aggregate was a complete method summary: enough of the method for a competitor to reconstruct the contribution. Public for 46 days. + +The quoted rewrite was the largest single exposure, and it was labelled a +proposal rather than a quotation, which is exactly why it survived review: the +rewrite gate protects numbers, citations, macros, entities and claim/evidence +relations, so a fidelity-preserving rewrite discloses what the original +discloses. Being a rewrite is not a defence. + +Removed: the quoted block, the method-component list, manuscript codenames +throughout, one real measured value, and manuscript-derived test fixtures, +which now use invented macros. Kept: every measurement. Finding counts, +percentiles, rule names, before/after tables and the panel results describe how +the tools behave, not what the paper found, and they are why the record exists. + +Git history was rewritten to remove the same content from every earlier commit. +That does not undo publication — the content may persist in existing clones, +forks and upstream caches. + +### A worked example that is safe to ship + +`examples/` now carries a synthetic manuscript and the same paper after acting +on the findings. The topic is a textbook one and every value is invented, so +the demonstration no longer depends on anyone's unpublished work. + +It is also a more honest demonstration than a clean sweep would be. L0 targets +go to zero and `discourse-cohesion` from 3 findings to 1, while +`salience-recital` **rises** from 4 to 6 — because carrying a noun forward to +link two sentences pulls the subject into sentences that also carry a numeral. +In a number-dense passage cohesion and recital want opposite things and no +rewrite satisfies both. Both findings are true, and which to act on is the +author's judgement, which is why neither is a blocker and there is no score. + +### Also + +- The re-export contract test named its excluded module imports one at a time + (`re`, `defaultdict`, `Path`), so it failed the moment a new import appeared. + It now excludes module objects by type. +- Suite: 393 tests across 20 files; `validate_plugin.py` 9/9. diff --git a/CHANGELOG-ARCHIVE.md b/CHANGELOG-ARCHIVE.md index a6154e7..99e2a7a 100644 --- a/CHANGELOG-ARCHIVE.md +++ b/CHANGELOG-ARCHIVE.md @@ -6,6 +6,7 @@ the repository's 750-line budget. Nothing here is edited; the history is verbatim. - Current releases: [CHANGELOG.md](CHANGELOG.md) +- **v0.35.0 through v0.35.1**: [CHANGELOG-ARCHIVE-v0.35.md](CHANGELOG-ARCHIVE-v0.35.md) - **v0.33.0 through v0.34.0**: [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md) - **v0.27.1 through v0.32.0**: [CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md) - **v0.21.0 and earlier**: [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35901fe..20fee30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,9 @@ All notable changes to the `sci-paper` plugin. Versions follow the `docs/audits/full-review-2026-09-27.md` recorded a full-repository review of v0.38.0: nine parallel line-by-line reads of the tools, skills and documents, -every high and medium finding reproduced first-party. This release acts on its -first tier ("应修") and the documentation corrections that go with it. No +every high and medium finding reproduced first-party. This release settles +every item: its first tier ("应修") and the documentation corrections that go +with it first, then the rest and a profile rebuilt on them. No consequence class changed; the standard is v3.9 (a fourth narrow exit contract, `verify_references.py`, and the residues and readings v0.37–v0.38 had added without a version stamp). @@ -95,9 +96,10 @@ sweep drops the preamble and every `skip` unit on both sides. The partition tool compares candidate states by conformal p instead of distances from different manifolds, simulates a split on the original text (the joined text let a trailing `%` comment swallow half a paragraph), and treats every heading -command as fixed. The tricolon wrap-up needs a wrap-up. `structure_baseline`, -`docstructure_baseline` and `anchoring_baseline` must be rebuilt for these to -take effect; the EVALUATION figures tied to them predate the rebuild and say so. +command as fixed. The tricolon wrap-up needs a wrap-up. A field's +`structure_baseline`, `docstructure_baseline` and `anchoring_baseline` must be +rebuilt for these to take effect; the shipped `wgl` ones were, and their +figures re-taken (below). **The rewrite gate runs on a fresh clone.** A missing profile, a missing or unusable `voice_model.joblib`, or a missing sentence-transformers was exit 2, @@ -143,8 +145,8 @@ marked `[WGL]` and `--query` serves any other field. skipped.** `quantiles` stored the ⌊qn⌋+1-th smallest value, one order statistic high: p90 of 100 values was the 91st, and at the 30-unit floor the tails read at P = 0.133 and 0.933. It stores the ⌈qn⌉-th now; an artifact -calibrated before this keeps the old grid until it is recalibrated, and the -EVALUATION rates measured on the old grid predate it. An artifact recording +calibrated before this keeps the old grid until it is recalibrated, as the +shipped `wgl` profile was (below). An artifact recording another unit than the axis measures is refused and named. A document whose units fall in the `unknown` bucket, a bucket the baseline lacks, and a salience reference no bucket resolves above are `degraded` with counts, where @@ -164,8 +166,86 @@ The voice axis is `unmeasured` without the surprisal runtime instead of a RuntimeError, `--scores` lists paragraphs by source line, and every one of these tools resolves `--field` through the shared resolver. -The suite is 764 tests in 32 files (v0.38.0: 529 in 27); the validator runs -11 checks and passes. +### The rest of the audit, and the profile rebuilt on it + +**Every item is settled.** Past the first tier, each finding of the audit is +fixed or closed with its reason, and §10 of the audit record names the commit +that carries it. + +**The document scale reads prose, at calibrated operating points only.** +Headings and floats are blanked before paragraphs are cut: a heading had fused +into its section's first block, and a float holding a blank line split its +paragraph in two (audit B22). The length-stratum edges are fitted on the +training papers alone, as exchangeability requires (B17), and a feature whose +reference cannot resolve its own tail is `degraded` and ordinary (B18). The +in-sample percentile fallback for a manifold without split-conformal +calibration is gone (B24): no calibration writes such a baseline, so the +manifold rule goes unscored and the axis `degraded` with a rebuild instruction, +and `deai_partition` gives that reason instead of "not measurable". + +**Hedging speaks for methods.** Re-measured per section on the rebuilt +profile, hedging separates machine text in `method` at 0.731–0.898 against a +0.442 human/human null and fires on 9.62% of held-out refereed method sections +at its 10% gate, so the axis runs on `intro` and `method` (EVALUATION §19.4). +The v0.33.0 measurement, taken when a manuscript section was one heading span, +had read `method` at 26.77%. + +**Skills defer to the standard on strong advisories.** Standard §4 makes an +advisory strong only when a measured effect exceeds a calibrated operating +point. De-AI's structural density hits, paper-review's structural hits, +physics' naming and unit checks and mainline's narrative findings are +judgement calls with no such point; each is now an ordinary advisory ranked +first, and the standard is unchanged. A firmer central hypothesis in +proposal-polish is proposed to the author, never applied silently. Under +`--orchestrated` the parent runs the de-AI audit and figure-review itself, as it +already ran physics, mainline and logic. Mainline's cold read asks an eighth +question, the one thesis line every contribution branch serves (standard +§5.4). Brainstorm drops `--min-frameworks`, whose floor was already all twelve +framings. The paper skill's review grep finds every form the linter does +(`delving`, and `Moreover` mid-sentence). + +**One copy of each fact.** The L0 word lists are the linter's patterns, and the +validator now holds the paper skill's table to them in both directions (audit +H8). The loader both evaluators read, the midrank AUC, the placeholder +vocabulary and the named section buckets each have one owner; register +findings, math density and cross-validation folds are computed once where they +were computed twice. The evidence documents keep each measurement in one place +and point to it, the validator's checks and the optional dependencies are +listed once (`tools/README.md`, `requirements.txt`), and the orphaned +`latency.json` is gone. + +**Smaller corrections.** `robust` before estimator, estimation, statistic or +regression is a method name and no longer counts toward the Tier B cap (F26). +The paired leakage estimate counts a paper the way the bank holds it (E17). +`corpus_cos` without the embedder or the corpus centroid scores unmeasured +instead of 0.0 (E20). CrossRef requests carry a polite-pool address from +`SCI_PAPER_MAILTO` and retry once on 429 and 5xx (F22). The exemplar embedding +cache is rebuilt when its fingerprint disagrees with the bank (D20). The arXiv +API is queried over HTTPS, and the style dossier names the source files it +skipped. Three regressions the pull request itself introduced are fixed: a +hyphenated compound is one word again, a unit reason names only buckets the +document reaches, and a commented-out `\newcommand` no longer beats the live +one. Examples in code, tests and documents that had been taken from a +manuscript under review are replaced by invented ones; no measurement changes. + +**The profile is rebuilt and the figures re-taken.** The `wgl` profile was +rebuilt on 2026-09-27 on every corpus-side fix above, and each evaluation figure +that reads an artifact was re-taken through the shipped path; a figure that was +not names the build it was taken on. The exemplar bank holds 28,444 paragraphs +from 18 of the 19 curated and 498 of the 500 reference papers. The +document-structure baseline is 504 papers, and its table moved (natural-AI +length-fair manifold AUC 0.933 → 0.893, long-form 0.740 → 0.840; human +false-flag 0.020 manifold, 0.042 role) under four changes the record does not +separate (EVALUATION §9). The learned model, retrained on 45,153 records, keeps +its grouped-split AUC at 0.9487 and the ranking its degraded mode consumes +(median Spearman ρ 1.000; 68 of 69 documents surface the same three +paragraphs). Held-out register reads 56 findings, 0.0364 per 1,000 body words, +87.5% of which would vanish had the paper been in its own bank. The worked +example reads 22 advisories before the revision and 17 after. + +The suite is 801 tests in 32 files (v0.38.0: 529 in 27); the validator runs +11 checks and passes. v0.35.0–v0.35.1 moved to `CHANGELOG-ARCHIVE-v0.35.md`; this +file had reached 806 lines. ## v0.38.0 — 2026-09-16 @@ -583,144 +663,10 @@ from 458 ms to 1.07 s, most of it the 541,309-pair bank load, which `--no-collocation` drops. v0.30.0–v0.31.0 moved to `CHANGELOG-ARCHIVE-RECENT.md`; this file had reached 790 lines. -## v0.35.1 — 2026-09-03 - -A post-release audit of the documentation, not of the code. An escape the -writer lost had truncated the tools table in both READMEs, the one published -figure with no artifact behind it was wrong, and several figures and dates in -the record had drifted from what they describe. - -### A lost backslash ended the tools table four rows early - -The `tools/tex_macros.py` row wrote `\newcommand` without its backslash, and -the consumed newline split the row, so the table stopped rendering there — -`retrieve_exemplars`, `fetch_arxiv_abstracts`, `train_ai_ism_classifier` and -`extract_md_negatives` fell out of it, in both READMEs. The same damage sat in -the v0.32.0 changelog entry at `\nocite`. - -### The worked example published a count the linter does not report - -`examples/README.md` said 19 total advisories where `ai_ism_lint` reports 18. -It was the only published-figure document with no artifact behind it, so -nothing read it. `tests/test_published_figures.py` now renders both of its -tables — the before/after summary and the per-rule counts — by running the -linter on the two shipped manuscripts and looking for the result, the same way -every other pinned figure is rendered from the artifact it was read from. A -cell the linter does not produce fails the case, so a document that agrees with -a stale run and a document nobody updated fail identically. - -### Figures and dates the record had drifted from - -- **203**, not 200, held-out refereed papers, in the two `DISPOSITIONS.md` rows - that still carried the count from before §18 re-measured it. -- The v0.30.0–v0.33.0 changelog headings were dated 2026-08-27; their tagger - dates are **2026-08-26**. -- The release-gate label in `EVALUATION.md` §12 stamped a version on a suite - size measured after it. It now reads "as of 2026-09-03; last tagged release - v0.35.1", so the version names the tag and the date names the measurement. -- Both README latency tables paired 394 tests with 81.4 s, a wall time taken on - the 393-test suite. The suite row is re-taken: **73.0 s**, median of 3, the - three runs spanning 70.0–86.9 s. Every other row still stands from the - 2026-08-27 take and the preamble now says which rows carry which date. -- `docs/README.md` called the evidence record five files where it is eight, - gave three of them section maps missing the sections added since, counted - four kinds of document where six live there, and described the validator's - version and suite-size checks narrower than they are. -- The unit-pattern anchor in both READMEs pointed at `rewrite_reward.py:41`; - the pattern is at `:56`. -- Both READMEs said CI runs on every push. `ci.yml` filters `branches: [main]`, - so it runs on every push to `main` and every pull request. - -### Also - -- `.gitignore` covers `models/hf-cache/`, `.ce/` and `.ccm/` — relocated model - cache and machine-local tool state, never repository content. -- Suite: 394 tests across 20 files; `validate_plugin.py` 9/9. - -## v0.35.0 — 2026-08-27 - -The axis that counts numbers could not see numbers written as macros, a public -repository was carrying an unpublished manuscript's method summary, and the -worked example that demonstrates any of this now runs on a synthetic paper -instead of a real one. - -### Numbers held in macros were invisible, in both directions - -A manuscript that writes `\newcommand{\Nsamples}{12}` in its preamble and -`\Nsamples{}` in its results put a measured quantity where neither named text -projection could read it. `RE_TEX_SIMPLE_CMD` reduces a command to its -argument, so the use site contributed nothing while the definition site -contributed the digits once, in the preamble, attributed to no reported -section. Two errors running in opposite directions, which is why the net stayed -small enough to go unnoticed: on the manuscript that surfaced it, expanding the -uses adds 650 digits and dropping the definitions removes 493. - -Found by running the tools on a real manuscript, not by a test. The salience -axis had been reporting recital percentiles for that paper computed on 91% of -its digits, and correcting it moved that paper from 16 recital findings to 26. - -`tools/tex_macros.py` expands numeric-literal macros once, on the assembled -document root, because that is the only scope holding both a preamble -definition and a body use — the same reason `read_tex_document` folds -`\include` in the first place. Only a bare numeric literal expands, so -`\newcommand{\Msun}{M_\odot}` and every macro taking an argument are untouched. - -Both salience baselines were rebuilt rather than left to score expanded -manuscripts against an unexpanded reference. They move by **zero to four -decimals** at the p90 and p95 gates, across all three features and all seven -buckets, in both `wgl` and `wgl-letter`: 88.2% of 390 corpus documents never -use the construction. That is what separates a correction from a rescaling, and -it means no published salience figure in `EVALUATION.md` changes (§22). - -### An unpublished manuscript's method summary was in a public repository - -This repository is public. Content from the author's unpublished manuscripts -had accumulated in the evaluation record since 2026-07-12, in increments that -each looked locally harmless — a codename in a test fixture, a macro and its -value in a docstring, one fidelity-preserving rewrite quoted in full. - -The aggregate was a complete method summary: enough of the method for a competitor to reconstruct the contribution. Public for 46 days. - -The quoted rewrite was the largest single exposure, and it was labelled a -proposal rather than a quotation, which is exactly why it survived review: the -rewrite gate protects numbers, citations, macros, entities and claim/evidence -relations, so a fidelity-preserving rewrite discloses what the original -discloses. Being a rewrite is not a defence. - -Removed: the quoted block, the method-component list, manuscript codenames -throughout, one real measured value, and manuscript-derived test fixtures, -which now use invented macros. Kept: every measurement. Finding counts, -percentiles, rule names, before/after tables and the panel results describe how -the tools behave, not what the paper found, and they are why the record exists. - -Git history was rewritten to remove the same content from every earlier commit. -That does not undo publication — the content may persist in existing clones, -forks and upstream caches. - -### A worked example that is safe to ship - -`examples/` now carries a synthetic manuscript and the same paper after acting -on the findings. The topic is a textbook one and every value is invented, so -the demonstration no longer depends on anyone's unpublished work. - -It is also a more honest demonstration than a clean sweep would be. L0 targets -go to zero and `discourse-cohesion` from 3 findings to 1, while -`salience-recital` **rises** from 4 to 6 — because carrying a noun forward to -link two sentences pulls the subject into sentences that also carry a numeral. -In a number-dense passage cohesion and recital want opposite things and no -rewrite satisfies both. Both findings are true, and which to act on is the -author's judgement, which is why neither is a blocker and there is no score. - -### Also - -- The re-export contract test named its excluded module imports one at a time - (`re`, `defaultdict`, `Path`), so it failed the moment a new import appeared. - It now excludes module objects by type. -- Suite: 393 tests across 20 files; `validate_plugin.py` 9/9. - --- -Older entries: [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md) +Older entries: [CHANGELOG-ARCHIVE-v0.35.md](CHANGELOG-ARCHIVE-v0.35.md) +(v0.35.0-v0.35.1), [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md) (v0.33.0-v0.34.0), [CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md) (v0.27.1-v0.32.0), [CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md) (v0.22.0-v0.27.0) and diff --git a/README.md b/README.md index 30ed9ea..acafe1c 100644 --- a/README.md +++ b/README.md @@ -5,14 +5,14 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-764%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-801%20passing-success.svg)](tests/) **A Claude Code plugin that writes, reviews, de-AIs, and condenses scientific manuscripts for top-tier journals — under one typed standard, with every claim traced to a source and every unavailable measurement labelled as unavailable.** Built for ApJ / MNRAS / PRD / JCAP-class papers and NSF / NIH proposals. -**12 skills · 40 tools · 764 tests · one normative contract · zero authorship verdicts.** +**12 skills · 40 tools · 801 tests · one normative contract · zero authorship verdicts.** [中文文档](README.zh-CN.md) — [What it does](#what-it-does) · [How it works](#how-it-works) · [See it work](#see-it-work) · [Benchmarks](#benchmark-dashboard) · [Install](#install) · @@ -65,7 +65,7 @@ problem: | **standard** | [`paper`](skills/paper/SKILL.md) | The writing framework in context: accuracy rules, formula and citation conventions, forward narrative, the L0 lexical policy with canonical examples, positive-voice guidance, measurement states, stopping semantics. | | **measure** | [`physics`](skills/physics/SKILL.md) | Measurement primitive, findings only. First principles P1-P8: dimensional consistency, asymptotic limits, symmetry and parity, conservation, the preconditions of the information-theoretic bounds, algebraic re-derivation, numerical provenance, foundational citations, build integrity. The single source for `paper-review` dimension K; register and L0 delegate to the shared axes, not a private word list. | | **measure** | [`logic`](skills/logic/SKILL.md) | Measurement primitive, findings only. Claim graph (circular arguments, broken chains, switched conditions, undeclared assumptions), empirical statistics (splits, leakage, multiple comparisons, priors), and the review side of claim–evidence discipline: verb strength may not exceed evidence strength. The single source for `paper-review` dimension C. | -| **measure** | [`mainline`](skills/mainline/SKILL.md) | Measurement primitive, findings only. Builds a paper-level purpose record and contribution graph, then answers a cold reader's seven questions — where must a reader backtrack, supply hidden context, or choose between competing readings? No three-act template assumed. The single source for `paper-review` dimension E. | +| **measure** | [`mainline`](skills/mainline/SKILL.md) | Measurement primitive, findings only. Builds a paper-level purpose record and contribution graph, then answers a cold reader's eight questions — where must a reader backtrack, supply hidden context, or choose between competing readings? No three-act template assumed. The single source for `paper-review` dimension E. | | **measure** | [`figure-review`](skills/figure-review/SKILL.md) | Reviews **compiled pages at 150 DPI**, not source. Traces figure/caption/data provenance, measures canvas balance at the pixel level, separates scientific and build contradictions from readability advisories. | | **act** | [`de-ai`](skills/de-ai/SKILL.md) | Three chained passes — subsystem measurement (L0–L4), a structural-tell audit, then **claim-first rewriting** that rebuilds prose from the protected claim graph instead of polishing in place. `--audit-only` stops after the structural-tell audit (Pass 2). | | **act** | [`condense`](skills/condense/SKILL.md) | Whole-document redundancy elimination under one-canonical-home-per-fact, loop-until-dry convergence, and a **mechanical length gate** as the closing proof that the document actually shrank. | @@ -81,7 +81,7 @@ object with the same ID. A composite calls primitives; it never restates their c | It will not | Because | |---|---| -| Output an authorship verdict or an "% AI" score | The learned axis is **field-similarity triage**, capped at 0.5 confidence at paragraph scale. Its false-positive rate on field-topic AI prose is 30–42% ([why](#why-there-is-no-single-score-the-l3-confound)). | +| Output an authorship verdict or an "% AI" score | The learned axis is **field-similarity triage**, capped at 0.5 confidence at paragraph scale. Its false-positive rate on field-topic AI prose is 29–42% ([why](#why-there-is-no-single-score-the-l3-confound)). | | Emit a universal paper-level PASS/FAIL | Terminal state is *disposition-complete*, not "zero advisories". | | Convert a missing baseline into zero findings | Unavailable axes report `unmeasured` / `degraded` with the reason. | | Optimise prose to evade detectors | Rewrite ranking optimises faithful scientific prose. Fidelity is a hard gate, not a weight. | @@ -135,8 +135,8 @@ still leaves them uniform *relative to each other*. So the primary detector is a the per-document vector of log dispersion ratios scored by Mahalanobis distance against the human centre and covariance — pure-stdlib, 11-D, ridge-stabilised. The joint geometry catches what independent marginals cannot: a shape adversary can land plausible per-feature spreads with the wrong covariance. An orthogonal axis measures -**role coupling** — humans vary paragraph shape *where the argument demands it*. On the 493-paper human corpus -the two axes' 5% tails are **exactly disjoint** (0 flagged by both; independence predicts ~1.3). +**role coupling** — humans vary paragraph shape *where the argument demands it*. On the 504-paper human corpus +the two axes' 5% tails are **exactly disjoint** (0 of 26 flagged by both; independence predicts 1.34). **3 · Operating points are split-conformal and length-stratified.** Percentile thresholds fit on the papers that set them are in-sample. The shipped operating @@ -367,7 +367,7 @@ isolated worktrees and merges typed findings from six independent passes: | `paper-review` | 18 dimensions **A–R**: mathematics, physics, logic and statistics, language and de-AI, structure and narrative spine, citation existence and relevance, data/results/figures, interfaces, redundancy, reproducibility, modern-physics checks, systemic consistency, adversarial verification, staleness, process artifacts, draft language, reference precision, glossary alignment | | `figure-review` | every figure re-rendered from the compiled PDF at 150 DPI — caption/data consistency, units, readability at print size, colour accessibility, float placement, cross-figure coherence | | `de-ai --audit-only` | the L0–L4 measurement stack plus the vendored humanizer structural-tell catalogue | -| `physics` · `mainline` · `logic` | three measurement primitives, one sibling agent each: first principles P1–P8; the cold reader's seven questions against the contribution graph; claim graph, empirical statistics and claim–evidence discipline | +| `physics` · `mainline` · `logic` | three measurement primitives, one sibling agent each: first principles P1–P8; the cold reader's eight questions against the contribution graph; claim graph, empirical statistics and claim–evidence discipline | The merge rule is the point: **integrity blockers must be resolved, L0 targets must reach zero, every strong advisory needs a recorded disposition, and ordinary @@ -395,34 +395,35 @@ The last row moved with the reference: the 2026-08-27 profile read the revision Two kinds of number, not interchangeable: **discrimination and calibration**, read from the evaluation record; and **latency**, measured first-party for this -README. All discrimination figures below were **re-scored on 2026-08-25** against -the rebuilt profile, through the shipped path — not carried forward. +README. The document-scale table and the L3 figures below were **re-taken on 2026-09-27** +against the rebuilt profile, through the shipped path — not carried forward. ### Document-scale discrimination and its false-positive control AUC is length-fair: each document is compared only against human papers in its -own length stratum (170 in the short stratum). Flag rates are split-conformal, +own length stratum (197 in the short stratum). Flag rates are split-conformal, Mondrian-stratified by length tercile, giving **finite-sample, distribution-free `P(false flag) ≤ α`** for exchangeable human papers. Source: [§9.5](docs/architecture/evaluation/document-scale.md). | Contrast | Manifold AUC | Role AUC | Manifold tail power | Role | union | |---|---:|---:|---:|---:|---:| -| human vs **natural** AI | **0.933** | 0.690 | 0.250 | 0.036 | 0.286 | -| human vs **de-AI'd** AI (paragraph de-AI as an attack) | **0.943** | 0.742 | 0.400 | 0.268 | **0.600** | -| human vs **shape-adversarial** AI (deliberate raggedness) | 0.927 | **0.810** | 0.184 | 0.316 | 0.447 | -| human vs **skeleton-matched** structural clone | 0.914 | 0.507 | 0.292 | 0.042 | 0.333 | -| human vs **long-form** AI *(standing falsification target)* | 0.740 | 0.634 | **0.000** | 0.069 | 0.069 | -| **human false-flag**, scored over 492 papers, α = 0.05 | — | — | **0.033** | **0.043** | 0.075 | - -The axes are complementary by construction: the manifold carries discrimination -against every tier including structure clones, while the role axis concentrates -on the tier that narrows the manifold's margin. Every published AUC reproduced -within **0.012** across the 42× rebuild — all eight, not a matching total. +| human vs **natural** AI | 0.893 | 0.687 | 0.071 | 0.036 | 0.107 | +| human vs **de-AI'd** AI (15 documents, under the 20-document floor) | unmeasured | unmeasured | unmeasured | unmeasured | unmeasured | +| human vs **shape-adversarial** AI (deliberate raggedness) | **0.953** | **0.803** | 0.184 | 0.316 | **0.447** | +| human vs **skeleton-matched** structural clone | 0.866 | 0.526 | 0.250 | 0.083 | 0.292 | +| human vs **long-form** AI *(standing falsification target)* | 0.840 | 0.610 | **0.000** | 0.069 | 0.069 | +| **human false-flag**, scored over 504 papers, α = 0.05 | — | — | **0.020** | **0.042** | 0.062 | + +The axes are complementary by construction: the manifold carries discrimination against every tier including +structure clones, while the role axis concentrates on the tier that narrows the manifold's margin. Since the +2026-08-25 table the length-fair AUCs moved −0.05 to +0.10 (natural 0.933 → 0.893, long-form 0.740 → 0.840) under +four unseparated changes: the v0.36.3 corpus, dropping appendices and references, blanking headings and floats, and +fitting the length strata on the training split alone. Tail power is reported plainly, including where it is weak and where it is zero, and **the tail-power column -above is a single seed draw**: the per-seed spread on those tiers is 0.04–0.18, so over 12 seeds natural reads -0.170 ± 0.110. Long-form is not caught at the strict operating point at all, in any configuration tried. The +above is a single seed draw**: the per-seed spread on those tiers is 0.04–0.18, so over 12 seeds (2026-08-26) natural +read 0.170 ± 0.110. Long-form is not caught at the strict operating point at all, in any configuration tried. The estimator-noise model that was supposed to fix short documents has now been built and refuted — it moves the human rate not at all ([§9.4c](docs/architecture/evaluation/document-scale.md)). Re-run the whole table with `python tools/eval_docscale.py --field wgl`. @@ -430,31 +431,31 @@ human rate not at all ([§9.4c](docs/architecture/evaluation/document-scale.md)) ### Why there is no single score: the L3 confound The learned paragraph-scale model is good — and ships `degraded` **anyway**, for -a measured reason. Retrained 2026-09-05 on 44,636 records. Source: +a measured reason. Retrained 2026-09-27 on 45,153 records. Source: [§7](docs/architecture/evaluation/learned-model.md). | Metric | Value | 95% split range | 17,299-record bank | |---|---:|---|---:| -| Grouped-split AUC (20 splits, complete papers held out) | **0.9487** | 0.9409 – 0.9583 | 0.9320 | -| Matched-stratum AUC (section × length × math × field-term) | **0.9262** | 0.9063 – 0.9494 | 0.9236 | -| Balanced accuracy | **0.8704** | 0.8588 – 0.8881 | 0.8445 | -| Author hard-set, **true provenance** | 0.938 | 0.846 – 0.998 | 0.937 | -| False-positive rate — public-generic AI text | 0.055 | 0.026 – 0.076 | 0.086 | -| False-positive rate — **field-topic AI text** | **0.295** | 0.230 – 0.356 | 0.318 | -| False-positive rate — **field-jargon-dense AI text** | **0.421** | 0.236 – 0.543 | 0.417 | - -A 0.95 AUC headline and a 30–42% false-positive rate on field-topic AI prose are the same model: the learned +| Grouped-split AUC (20 splits, complete papers held out) | **0.9487** | 0.9392 – 0.9587 | 0.9320 | +| Matched-stratum AUC (section × length × math × field-term) | **0.9250** | 0.9058 – 0.9495 | 0.9236 | +| Balanced accuracy | **0.8705** | 0.8602 – 0.8883 | 0.8445 | +| Author hard-set, **true provenance** | 0.937 | 0.842 – 0.999 | 0.937 | +| False-positive rate — public-generic AI text | 0.057 | 0.026 – 0.076 | 0.086 | +| False-positive rate — **field-topic AI text** | **0.292** | 0.230 – 0.349 | 0.318 | +| False-positive rate — **field-jargon-dense AI text** | **0.418** | 0.212 – 0.528 | 0.417 | + +A 0.95 AUC headline and a 29–42% false-positive rate on field-topic AI prose are the same model: the learned score partly measures *field register*, so it is unreliable on the exact distribution a de-AI pass must catch. -**Four retrains across a 2.6× bank range and a profile rebuild agree** — field-topic went 0.318 → 0.285 → 0.280 -→ 0.295, every step inside one retrain's own split range, while headline AUC rose and eased (0.9320 → 0.9518 → -0.9487). More data buys the easy contrast and nothing on the one that matters, so the roadmap item closes by +**Five retrains across a 2.6× bank range and two profile rebuilds agree** — field-topic went 0.318 → 0.285 → +0.280 → 0.295 → 0.292, every step inside one retrain's own split range, while headline AUC rose, eased and held +(0.9320 → 0.9518 → 0.9487 → 0.9487). More data buys the easy contrast and nothing on the one that matters, so the roadmap item closes by decision: **not obtainable from this feature set**; reopening needs different features, not a bigger bank. The document-level surprisal path measured 0.757, weaker than the model-free manifold (0.881), and adds nothing to it. -Retraining is not behaviour-preserving and is not claimed to be: on 1,808 -paragraphs scored by the last two bundles the degraded mode's ranking holds at -Spearman **ρ = 0.991** and the three surfaced paragraphs overlap **0.889** (0.846 -and 0.654 for the 2.6× bank growth before it). Same schema, features and posture. +Retraining is not behaviour-preserving and is not claimed to be: on 2,080 +paragraphs scored by the last two bundles the degraded mode's ranking holds at a median +Spearman **ρ = 1.000** and 68 of 69 documents surface the same three paragraphs (0.846 +and 0.654 for the 2.6× bank growth two retrains back). Same schema, features and posture. ### Latency and repository health @@ -474,11 +475,11 @@ at 100%, and a number taken under that describes the contention, not the axis. V | `+ --oracle` (GPT-2-large token surprisal) | 33.8 s (2026-08-27) | `transformers` + `torch` | | `+ --voice` (learned L3 triage) | 37.2 s (2026-08-27) | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` — **11/11 checks pass** | 2.24 s | stdlib | -| Full test suite — **764 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | +| Full test suite — **801 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | The headline: **a complete model-free pass over a 5,084-word manuscript costs ~1.0 s above the interpreter floor**, with no optional dependency installed — and 0.6 s of that is loading the collocation bank (530,677 -pairs in the 2026-09-04 bank the timing was taken on; 530,504 after the v0.36.3 rebuild), which `--no-collocation` drops to ~0.4 s. The two model-backed axes cost 30×–35× more than the full +pairs in the 2026-09-04 bank the timing was taken on; 535,517 after the v0.39.0 rebuild), which `--no-collocation` drops to ~0.4 s. The two model-backed axes cost 30×–35× more than the full model-free pass and are opt-in flags — you should not need a GPU to lint a paper. CI runs validator and suite on every push to main and on every PR, Python 3.11, Ubuntu. --- @@ -494,20 +495,13 @@ Skills are then namespaced `/sci-paper:`. **Python ≥ 3.11.** The shared schema, the deterministic L0 linter, the model-free L1/L2 axes, the document-structure analysis, and the validator are **standard -library only**. Optional capabilities add dependencies: +library only**. Optional capabilities add dependencies, each listed with what it +enables in [`requirements.txt`](requirements.txt): ```bash pip install -r requirements.txt # all optional extras ``` -| Package | Enables | -|---|---| -| `pymupdf` | PDF corpus extraction, compiled-page inspection | -| `sentence-transformers` | semantic exemplar retrieval, embedding features | -| `scikit-learn` + `joblib` | legacy and learned field-similarity models | -| `transformers` + `torch` | token-surprisal / UID measurement | -| `numpy` | learned feature, cache, and rewrite-score utilities | - > Never install an optional dependency merely to turn an unavailable axis into a > nominal score. A missing package keeps its axis `unmeasured`, by design. @@ -551,12 +545,11 @@ gates, `build` corpus and profile construction, `eval` evidence. Detail: [tools/ | `tools/rewrite_reward.py` | core | Ranks rewrite candidates **after** hard scientific-fidelity eligibility. Dropping *or inventing* a protected invariant scores `-inf`. | | `tools/deai_register.py` | L0 | Domain register: terms the manuscript leans on that the field's own corpus does not carry, judged by corpus document frequency rather than a curated cross-discipline list. Compounds are judged by their rarest part. The zero-hit audit lists every body word with df 0. Advisories only. | | `tools/deai_collocation.py` | L2 | Sentences that join common words the field never joins: the fraction of adjacent content-word pairs unattested in the corpus, against a leave-one-out per-section reference; each pair carries its chance-absence probability. Advisories only; a defined term keeps its pair. `--glossary` reads the same bank document-wide and lists the unattested pairs used twice or more, the manuscript's own terms, with the line of first use and whether that sentence defines them. | -| `tools/ai_ism_negatives_handcrafted.txt` | L0 | Seed negative examples for the legacy classifier (data asset). | | `tools/deai_metrics.py` | L1 | Model-free information-distribution findings — sentence-length variation, connective openers — with explicit calibration state. | | `tools/deai_oracle.py` | L1 | Optional token-surprisal and Uniform Information Density evidence. Unavailable assets and compatibility thresholds stay explicit. | | `tools/deai_structure.py` | L2 | Sentence and paragraph construction: enumeration, repeated frames, parallel runs, symmetry, and related template families; auxiliary families (antithesis, reversal beats, paper-as-agent, wh-cleft, modifier stacks) name the sentence without entering the score. | | `tools/deai_salience.py` | L2 | Salience hierarchy: how far a passage's measured quantities run without an interpreting sentence between them, against a per-section human reference. Sole consumer of the numeral-preserving LaTeX projection. | -| `tools/deai_discourse.py` | L2 | Discourse texture on the LOW tail: given/new linkage per paragraph, and epistemic-marker rate per **section** — hedging has no paragraph-scale lower tail, so the two axes calibrate at different units and each artifact records its own. Hedging speaks only for introductions, where its operating point was shown to transfer. | +| `tools/deai_discourse.py` | L2 | Discourse texture on the LOW tail: given/new linkage per paragraph, and epistemic-marker rate per **section** — hedging has no paragraph-scale lower tail, so the two axes calibrate at different units and each artifact records its own. Hedging speaks only for introductions and methods, where its operating point was shown to transfer. | | `tools/deai_reference.py` | L2 | The one `(feature, unit)` percentile reference every per-bucket axis shares: quantile grid, plateau-top percentile reader, sample floor, paragraph and section sweeps, calibration loop. Holds no policy; its invariant is that calibration and detection never drift apart on what a unit is. | | `tools/deai_docshape.py` | L2 | Document-shape measurement and complete-document calibration: the per-paragraph feature vector, cross-paragraph dispersion, the joint Mahalanobis manifold, role coupling, split-conformal operating points, and the baseline builder. Split from `deai_docstructure.py` on 2026-08-25; that module re-exports every public name here. | | `tools/deai_docstructure.py` | L2 | Whole-document rhetorical shape and complete-document calibration: dispersion band, per-length-stratum joint manifold, role coupling, split-conformal operating points. | @@ -582,6 +575,7 @@ gates, `build` corpus and profile construction, `eval` evidence. Detail: [tools/ | `tools/fetch_arxiv_abstracts.py` | build | Fetches dated abstract corpora for controlled evaluation and training, optionally restricted to a subfield query set and named refereed journals, or complete LaTeX sources for one author (`--author` + `--author-is` + `--max-authors`). Rate limiting **stops the sweep and exits 2** rather than writing a truncated corpus as if it were complete. | | `tools/train_ai_ism_classifier.py` | legacy | Trains the legacy word-ngram classifier, used only as degraded advisory evidence. | | `tools/extract_md_negatives.py` | legacy | Harvests candidate generated paragraphs for controlled evaluation and training. | +| `tools/ai_ism_negatives_handcrafted.txt` | legacy | Seed negative examples for the legacy classifier (data asset). | > `tools/validate_plugin.py` is a release tool, not a product tool, and is excluded above. @@ -594,7 +588,7 @@ Every finding carries exactly one **consequence class**: | Class | Meaning | Required consequence | |---|---|---| | `integrity_blocker` | The scientific record may be wrong, unsupported, inconsistent, unreproducible, or unusable | **Must** be resolved from sources. Cannot be waived as a style preference. | -| `l0_target` | A Tier A word, an em-dash, or a Tier B word beyond one occurrence per section | Rewrite to zero. Not a claim that the paper is scientifically invalid. | +| `l0_target` | A Tier A word, an em-dash, or a Tier B word beyond one occurrence per heading unit | Rewrite to zero. Not a claim that the paper is scientifically invalid. | | `advisory` | Structural, distributional, learned, rhetorical, clarity, or aesthetic evidence | Rank, act on the strongest, then record a disposition for the rest. | Every axis reports one **measurement state** — `measured`, `degraded`, @@ -625,17 +619,16 @@ For scale, the reference profile behind every measured number here carries: | Asset | Scale | |---|---| -| `exemplar_paragraphs.jsonl` | **27,851** section-typed paragraphs from 19 curated + 500 reference papers | -| `register_lexicon.json` | 41,644 passages · 53,367 terms | -| `uid_baseline.json` | 27,851 paragraphs under GPT-2-large · pooled global UID 3.303 ± 0.420 | -| `structure_baseline.json` | method 9,478 · results 3,978 · data 3,894 · intro 3,812 · discussion 3,635 · conclusion 2,611 · abstract 433 | -| `salience_baseline.json` | abstract 13,971 · method 6,967 · results 3,240 · intro 3,239 · data 3,020 · discussion 2,954 · conclusion 1,999 | -| `docstructure_baseline.json` | 507 complete documents · conformal α 0.05 · length strata [46, 75] | +| `exemplar_paragraphs.jsonl` | **28,444** section-typed paragraphs from 18 of the 19 curated and 498 of the 500 reference papers | +| `register_lexicon.json` | 42,237 passages · 53,668 terms | +| `uid_baseline.json` | 28,444 paragraphs under GPT-2-large · pooled global UID 3.301 ± 0.413 | +| `structure_baseline.json` | method 9,654 · results 4,065 · data 3,953 · intro 3,848 · discussion 3,739 · conclusion 2,747 · abstract 433 | +| `salience_baseline.json` | abstract 13,970 · method 7,209 · results 3,337 · intro 3,280 · data 3,098 · discussion 3,051 · conclusion 2,018 | +| `docstructure_baseline.json` | 504 complete documents · conformal α 0.05 · length strata [49, 78] | | `anchoring_baseline.json` | 517 documents · all six section classes above the 30-document retention minimum and the 119-document floor of the six-class Bonferroni share | -| `voice_model.joblib` | 44,636 records · 14 features · **no operating point**, `degraded` | +| `voice_model.joblib` | 45,153 records · 14 features · **no operating point**, `degraded` | -Every bucket clears the 30-passage floor — untrue before 2026-08-25, when -`results` held 26 and that was misread as a corpus-size limit. +Every bucket clears the 30-passage floor. Corpus contents are **read-only, copyright-sensitive inputs**, never committed. A dossier is evidence, not a standard and not proof of authorship. @@ -663,7 +656,7 @@ dossier is evidence, not a standard and not proof of authorship. | Corpus reference | **User-supplied, tiered, gitignored** | Style is field-relative. A generic prior is the thing being replaced. | | Optional models | `transformers`+`torch`, `scikit-learn`, `sentence-transformers` | Strictly opt-in flags. Absence degrades an axis, never the run. | | Distribution | **Claude Code plugin** (`.claude-plugin/plugin.json`) | Skills at `skills//SKILL.md`, namespaced `/sci-paper:`. | -| Contract enforcement | `tools/validate_plugin.py` + GitHub Actions | 11 checks over manifests, registries, doc authority and links, recorded counts, imports, CLI entry points, exit semantics, tests, CI wiring, the residue contract, and the 750-line budget. Drift fails CI instead of accumulating. | +| Contract enforcement | `tools/validate_plugin.py` + GitHub Actions | 11 checks, listed once in [tools/README.md](tools/README.md). Drift fails CI instead of accumulating. | --- @@ -677,18 +670,15 @@ sci-paper/ │ ├── SCIPAPER_STANDARD.md the single normative contract (v3.8) │ ├── architecture/ DEAI_SUBSYSTEM.md · EVALUATION.md (hub) + evaluation/ │ └── design-notes/ frozen, dated reasoning records (not status) -├── skills//SKILL.md 12 skills ├── tests/ 32 files, 764 tests +├── skills//SKILL.md 12 skills ├── tests/ 32 files, 801 tests ├── tools/ 40 product tools ├── CHANGELOG.md · ACKNOWLEDGMENTS.md ├── style-corpus// user-supplied read-only corpus (gitignored) └── style-profile// generated and calibrated evidence (gitignored) ``` -`python tools/validate_plugin.py` runs 11 contract checks and `python -m unittest discover -s tests -v` runs the -764-test suite; both must pass before a release. The validator covers release metadata, skill frontmatter, -standard references, documentation boundaries and index completeness, in-page anchors, recorded suite sizes -against real discovery, product registries, syntax, runtime imports, CLI entry points, schema fields, and linter -exit semantics — `tools/validate_plugin.py` itself is the authoritative list. A release also requires -independent review, clean-checkout verification, and green hosted CI. +`python tools/validate_plugin.py` runs 11 contract checks (listed in [tools/README.md](tools/README.md)) and +`python -m unittest discover -s tests -v` runs the 801-test suite; both must pass before a release. A release +also requires independent review, clean-checkout verification, and green hosted CI. --- @@ -708,16 +698,16 @@ de-AI standard. | Limitation | Current state | |---|---| | **No learned-model operating point** | L3 ships `degraded`. The document-level surprisal path was *measured* not to provide one (0.757 vs the model-free manifold's 0.881). | -| **Field-topic false positives** | 30–42% on field-topic and jargon-dense AI prose. **Closed by decision:** four retrains across a 2.6× bank range agree the confound is in the feature set, so no field-topic-robust operating point is obtainable from it. | -| **Short-document tail power, and it is a seed draw** | Manifold 5%-tail power on short natural-AI documents averages **0.170 ± 0.110** over 12 seeds against a 0.933 length-fair ranking; per-tier spread reaches 0.18, wider than several differences the record once read as improvements. Three fixes were built and all three fail: distance normalisation (rejected), finer stratification, and an explicit estimator-noise covariance. `tools/eval_docscale.py` re-runs the table instead of quoting it. | -| **Long-form generation is not caught** | At α = 0.05, manifold tail power on long-form AI is **0.000** — stable across 2 metrics × 4 calibration splits × 12 seeds. Rank AUC is 0.729, so the signal exists and the operating point cannot reach it. | +| **Field-topic false positives** | 29–42% on field-topic and jargon-dense AI prose. **Closed by decision:** five retrains across a 2.6× bank range agree the confound is in the feature set, so no field-topic-robust operating point is obtainable from it. | +| **Short-document tail power, and it is a seed draw** | Manifold 5%-tail power on short natural-AI documents averaged **0.170 ± 0.110** over 12 seeds (2026-08-26) against a 0.933 length-fair ranking; the 2026-09-27 draw reads 0.071 against 0.893; per-tier spread reaches 0.18, wider than several differences the record once read as improvements. Three fixes were built and all three fail: distance normalisation (rejected), finer stratification, and an explicit estimator-noise covariance. `tools/eval_docscale.py` re-runs the table instead of quoting it. | +| **Long-form generation is not caught** | At α = 0.05, manifold tail power on long-form AI is **0.000** — stable across 2 metrics × 4 calibration splits × 12 seeds. Its length-fair rank AUC is 0.840 (2026-09-27), so the signal exists and the operating point cannot reach it. | | **Cooperative-layer tools** | `deai_provenance` and `deai_personal` are honestly `unmeasured` until the author supplies their own draft history or ≥ 3 prior papers. | | **`L1.distribution` / `L2.sentence_structure`** | `degraded` — and now for a *measured* reason. Burstiness reverses sign on adversarial prose (AUC 0.181) and signposting runs below chance (0.247), so no operating point is available to write. | -| **Retrains are not behaviour-preserving** | Rebuilding the profile refits L3. Ranking holds at ρ 0.991 and triage overlap 0.889 for the 2026-09-05 retrain (0.846 / 0.654 across the 2.6× bank growth before it), but an old triage list will not reproduce exactly. | +| **Retrains are not behaviour-preserving** | Rebuilding the profile refits L3. Ranking holds at ρ 1.000 and triage overlap 0.995 for the 2026-09-27 retrain (0.846 / 0.654 across the 2.6× bank growth), but an old triage list will not reproduce on every document ([§7.0](docs/architecture/evaluation/learned-model.md)). | | **A quarter of the corpus is never used** | Headings matching no section bucket are dropped rather than guessed: **2,334 of 9,178 (25.4%)** in `wgl`, 42 of 148 in `wgl-letter`. The remainder is mostly topic headings ("Matter power spectrum"); "Measurements" and "Background" were refused as genuinely ambiguous. | -| **Register fires on accepted prose, and the vocabulary audits are advice** | Measured on 203 held-out refereed ApJ/ApJL/A&A papers it never saw: **0.0371 findings per 1,000 body words** (57 findings; the same 57 read 0.0247 against a raw-source denominator until v0.36.2, and 0.0858 before the v0.36.0 heading and v0.36.1 float fixes), 22.2% of documents, rank AUC **0.391** against machine text — it still fires *more* on human papers than on AI drafts, and 98.2% of the 57 remaining flags would vanish if the paper sat in its own bank. Sweeping the use floor 5 → 50 keeps AUC below 0.5 **everywhere**, so no setting makes this a detector (replicated by author rather than journal, 22 papers, AUC **0.328**, [§21](docs/architecture/evaluation/held-out-labels.md)). The exhaustive zero-hit audit is the same fact at full strength: every refereed paper carries words the corpus never wrote (3.37 per 1,000 body words, AUC 0.174), so it and the collocation axis ship as advice with author dispositions, and the collocation bank costs 0.6 s per run that `--no-collocation` drops ([§23](docs/architecture/evaluation/vocabulary-and-residue.md)). | -| **Advice quality is still unlabelled** | Provenance answers "does it fire on accepted prose", not "is this advisory right". Salience's p90 gate fires at **0.203** per passage on 203 held-out refereed papers, under its 0.271 three-gate union bound ([§17.5](docs/architecture/evaluation/held-out-labels.md)); it read 0.454 under v0.36.2 because the bank stored only the `[math]` projection, so the reference held no numeral inside math while every manuscript did — closed in v0.36.3 by storing each paragraph's numeral projection beside it. 7.00% of the digits it read on LaTeX were citation years until v0.32.0; precision and recall for the advice itself need `tools/label_findings.py`, and running it is the author's act. | -| **Hedging only speaks about introductions** | The epistemic-marker axis ships restricted to `intro`, where its p10 gate fires at 7.89% on 203 held-out refereed papers. Elsewhere it fires at 15–27% on prose a referee accepted, and at least one generation regime lands below chance. Cohesion needs no such restriction (6.6–14.6% across all seven buckets). | +| **Register fires on accepted prose, and the vocabulary audits are advice** | Measured on 203 held-out refereed ApJ/ApJL/A&A papers it never saw: **0.0364 findings per 1,000 body words** (56 findings), 22.2% of documents, rank AUC **0.391** against machine text — it fires *more* on human papers than on AI drafts, and 87.5% of the 56 flags would vanish if the paper sat in its own bank ([§17.4](docs/architecture/evaluation/held-out-labels.md)). Sweeping the use floor 5 → 50 keeps AUC below 0.5 **everywhere**, so no setting makes this a detector (replicated by author rather than journal, 22 papers, AUC **0.411**, [§21](docs/architecture/evaluation/held-out-labels.md)). The exhaustive zero-hit audit is the same fact at full strength: every refereed paper carries words the corpus never wrote (3.34 per 1,000 body words, AUC 0.177), so it and the collocation axis ship as advice with author dispositions, and the collocation bank costs 0.6 s per run that `--no-collocation` drops ([§23](docs/architecture/evaluation/vocabulary-and-residue.md)). | +| **Advice quality is still unlabelled** | Provenance answers "does it fire on accepted prose", not "is this advisory right". Salience's p90 gate fires at **0.201** per passage on 203 held-out refereed papers, under its 0.271 three-gate union bound ([§17.5](docs/architecture/evaluation/held-out-labels.md)); precision and recall for the advice itself need `tools/label_findings.py`, and running it is the author's act. | +| **Hedging speaks only about introductions and methods** | The epistemic-marker axis ships restricted to `intro` and `method`, where its p10 gate fires at 9.09% and 9.62% on 203 held-out refereed papers and every generation regime separates. In `data`, `conclusion` and `results` it fires at 12–14% on prose a referee accepted, and in `discussion` a generation regime lands below chance ([§19](docs/architecture/evaluation/discourse-and-citation.md)). Cohesion needs no such restriction (6.8–14.9% across all seven buckets). | | **Two axes can want opposite things** | Cohesion asks a sentence to reuse the previous sentence's nouns; recital counts sentences bearing numerals. In a number-dense passage the noun worth carrying forward is the one the numbers are about, so satisfying one axis costs the other and no rewrite satisfies both ([`examples/`](examples/README.md): the revision carries `bias` into numeral-bearing sentences and restates its headline pair, and its two strong recital findings are led by numeral density at p96 and p99 rather than by a run). Both findings are true; the contract is advisory precisely because the trade-off is the author's to make. | | **A fresh clone measures nothing** | All profile assets are gitignored. Until you build a profile from your own papers, every corpus-referenced axis is `unmeasured`. | @@ -727,7 +717,7 @@ Empty. `label_findings.py` ships; running it is the author's business, not the r this project measures against is provenance — `eval_findings.py`, on labels that already exist. **Closed by refutation, not by shipping.** The length-aware manifold, a larger conformal calibration -set, long-form tail power, four L3 retrains, the leakage contrast, the register operating point, and +set, long-form tail power, five L3 retrains, the leakage contrast, the register operating point, and now citation placement were built or tested and refuted. The last was the strongest model-free discriminator in the record (AUC 0.866) and did not survive its own pre-registered test: a second bank from a different model scores **0.053** on the same statistic with no citation instruction and **0.734** diff --git a/README.zh-CN.md b/README.zh-CN.md index 8ab8399..9098383 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -5,13 +5,13 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-764%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-801%20passing-success.svg)](tests/) **一个 Claude Code 插件：在同一套 typed 标准下完成科研论文的写作、审查、去 AI 化与精简。 每条结论都可溯源，每个测不出来的轴都如实标为测不出来。** 面向 ApJ / MNRAS / PRD / JCAP 级别的论文，以及 NSF / NIH 基金申请书。 -**12 个 skill · 40 个工具 · 764 个测试 · 一份规范 · 零作者身份判决。** +**12 个 skill · 40 个工具 · 801 个测试 · 一份规范 · 零作者身份判决。** [English](README.md) — [它做什么](#它做什么) · [怎么做到的](#怎么做到的) · [实际效果](#实际效果) · [Benchmark 面板](#benchmark-面板) · [安装](#安装) · @@ -59,11 +59,11 @@ sci-paper 把一个 Claude Code 会话变成一张论文工作台，由唯一一 | **规范** | [`paper`](skills/paper/SKILL.md) | 把写作框架载入上下文：准确性规则、公式与引用规范、正向叙述、带 canonical 例子的 L0 词汇政策、正向 voice 指引、measurement state、停止语义。 | | **测量** | [`physics`](skills/physics/SKILL.md) | 测量原件，只产 finding。第一原则 P1–P8：量纲一致性、渐近极限、对称性与宇称、守恒律、信息论与统计不等式的前提、代数再推导、数值溯源、基础引用、编译完整性。是 `paper-review` 维度 K 的唯一来源；语域与 L0 委派给共享轴，不自带词表。 | | **测量** | [`logic`](skills/logic/SKILL.md) | 测量原件，只产 finding。claim graph（循环论证、断链、偷换条件、未声明假设）、经验统计方法学（split、泄漏、多重比较、prior），以及声明-证据纪律的审查端：动词强度不得超过证据强度。是 `paper-review` 维度 C 的唯一来源。 | -| **测量** | [`mainline`](skills/mainline/SKILL.md) | 测量原件，只产 finding。建 paper-level purpose record 与 contribution graph，再按冷读者回答七问——读者在哪里需要回溯、补隐藏上下文，或在竞争解读间抉择？不预设三幕模板。是 `paper-review` 维度 E 的唯一来源。 | +| **测量** | [`mainline`](skills/mainline/SKILL.md) | 测量原件，只产 finding。建 paper-level purpose record 与 contribution graph，再按冷读者回答八问——读者在哪里需要回溯、补隐藏上下文，或在竞争解读间抉择？不预设三幕模板。是 `paper-review` 维度 E 的唯一来源。 | | **测量** | [`figure-review`](skills/figure-review/SKILL.md) | 审的是 **150 DPI 的编译页**，不是源码。追溯 figure/caption/数据的 provenance，在像素级测量画布平衡，并把科学与构建矛盾同可读性建议分开。 | -| **动作** | [`de-ai`](skills/de-ai/SKILL.md) | 三过串联 —— 子系统测量（L0–L4）、结构 tell 审计，然后 **claim-first 改写**：从受保护的 claim graph 重建文字，而不是原地润色。`--audit-only` 只跑到测量为止。 | +| **动作** | [`de-ai`](skills/de-ai/SKILL.md) | 三过串联 —— 子系统测量（L0–L4）、结构 tell 审计，然后 **claim-first 改写**：从受保护的 claim graph 重建文字，而不是原地润色。`--audit-only` 跑完结构 tell 审计（Pass 2）就停。 | | **动作** | [`condense`](skills/condense/SKILL.md) | 全文去冗余，遵循"每个事实只有一个 canonical 位置"、loop-until-dry 收敛，并以**机械长度门**作为"确实变短了"的收尾证明。 | -| **组合** | [`paper-review`](skills/paper-review/SKILL.md) | 溯源式 **A–R 审查**：数学、物理、逻辑与统计、语言、结构与叙事主线、引用、数据与图表、接口、冗余、可复现性、现代物理核查、跨章节一致性、对抗式验证（三 pass + 12-framing 升级）、staleness、过程残影、引用精确度、术语对齐。 | +| **组合** | [`paper-review`](skills/paper-review/SKILL.md) | 溯源式 **A–R 审查**：数学、物理、逻辑与统计、语言、结构与叙事主线、引用、数据与图表、接口、冗余、可复现性、现代物理核查、跨章节一致性、对抗式验证（三 pass + 12-framing 升级）、staleness、过程残影、内部草稿语言、引用精确度、术语对齐。 | | **组合** | [`final-review`](skills/final-review/SKILL.md) | 父级编排器：在**独立 worktree agent** 里跑 paper-review、figure-review、de-ai `--audit-only` 与 physics / mainline / logic 三个原件，合并 typed finding，并验证 disposition-complete 状态在连续多轮里稳定。 | | **体裁** | [`proposal-polish`](skills/proposal-polish/SKILL.md) | NSF Project Summary/Description、NIH Specific Aims、fellowship。保留论文会删掉的"愿景+可行性"语域，强制 claim 与可行性匹配，最狠地打磨决定评分的前几页。 | | **探索** | [`brainstorm`](skills/brainstorm/SKILL.md) | 辐射状研究方向探索器：每节点 12 条 framing pass、术语锚定到 glossary、每分支完整推导、递归发散直到收敛。严禁 defer / future-work / 半成品叶节点。 | @@ -77,7 +77,7 @@ sci-paper 把一个 Claude Code 会话变成一张论文工作台，由唯一一 | 它不会 | 因为 | |---|---| -| 给出作者身份判决或 "AI 百分比" | 学习型轴只是**领域相似度分诊**，在段落尺度封顶 0.5 置信度。它在领域主题 AI 文本上的假阳率是 30–42%（[为什么](#为什么没有单一分数l3-的混淆)）。 | +| 给出作者身份判决或 "AI 百分比" | 学习型轴只是**领域相似度分诊**，在段落尺度封顶 0.5 置信度。它在领域主题 AI 文本上的假阳率是 29–42%（[为什么](#为什么没有单一分数l3-的混淆)）。 | | 给出论文级 PASS/FAIL | 终止态是 *disposition-complete*，不是"零条 advisory"。 | | 把缺失的基线当成零个 finding | 测不了的轴报 `unmeasured` / `degraded`，并写明原因。 | | 为了绕过检测器而优化文字 | 改写排序优化的是忠实的科学文字。保真是硬门，不是权重。 | @@ -129,8 +129,8 @@ em-dash**，而全文级 dispersion 几乎没动 —— 0.47 → 0.49，人类 向量，对人类中心与协方差做 Mahalanobis 距离 —— 纯标准库的 11 维实现，带 ridge 稳定化。 联合几何能抓住独立边缘分布抓不住的东西：形状对抗者可以凑出合理的逐特征离散度， 却凑不出正确的协方差。另一条正交轴测 **role coupling** —— 人类是在**论证需要的地方** -改变段落形状。在 507 篇的人类语料上，两条轴的 5% 尾部**完全不相交** -（0 篇被两条同时标记；若独立应约 1.3 篇）。 +改变段落形状。在 504 篇的人类语料上，两条轴的 5% 尾部**完全不相交** +（各 26 篇里 0 篇被两条同时标记；若独立应为 1.34 篇）。 **三 · 操作点是 split-conformal 且按长度分层的。** 用同一批论文既拟合又定阈值， 那是 in-sample。发布的操作点在留出的人类论文上标定 nonconformity 分数， @@ -343,7 +343,7 @@ linter 只是其中一路输入。`/sci-paper:final-review` 会在隔离 worktre | `paper-review` | **A–R** 共 18 个维度：数学、物理、逻辑与统计、语言与去 AI、文档结构与叙事主线、引用存在性与相关性、数据/结果/图表、接口、冗余、可复现性、现代物理检查、系统一致性、对抗式验证、陈旧与漂移、过程残留、内部草稿语言、参考文献精度、术语表对齐 | | `figure-review` | 从编译后的 PDF 以 150 DPI 重渲每张图 —— 图/caption/数据一致性、单位、印刷尺寸下的可读性、色彩可及性、浮动体位置、跨图一致性 | | `de-ai --audit-only` | L0–L4 测量栈，加上引入的 humanizer 结构 tell 目录 | -| `physics` · `mainline` · `logic` | 三个测量原件，各起一个兄弟 agent：第一性原理 P1–P8；对着贡献图回答冷读者的七个问题；claim 图、经验统计与 claim–evidence 纪律 | +| `physics` · `mainline` · `logic` | 三个测量原件，各起一个兄弟 agent：第一性原理 P1–P8；对着贡献图回答冷读者的八个问题；claim 图、经验统计与 claim–evidence 纪律 | 合并规则才是重点：**integrity blocker 必须解决、L0 target 必须清零、每条 strong advisory 必须有明确 disposition，而普通 advisory 与不可用的轴保持可见**。轮次会一直 @@ -374,26 +374,26 @@ advisory 必须有明确 disposition，而普通 advisory 与不可用的轴保 ### 全文尺度的判别力与假阳控制 -AUC 是长度公平的：每份文档只跟**自己那个长度层**的人类论文比（短层 170 篇）。 +AUC 是长度公平的：每份文档只跟**自己那个长度层**的人类论文比（短层 197 篇）。 标记率走 split-conformal，按长度三分位做 Mondrian 分层，对可交换的人类论文给出 **有限样本、分布无关的 `P(误标) ≤ α`**。 来源：[§9.5](docs/architecture/evaluation/document-scale.md)。 | 对比 | 流形 AUC | role AUC | 流形尾功效 | role | 并集 | |---|---:|---:|---:|---:|---:| -| 人类 vs **自然** AI | **0.933** | 0.690 | 0.250 | 0.036 | 0.286 | -| 人类 vs **去 AI 化后**的 AI（段落级去 AI 当攻击） | **0.943** | 0.742 | 0.400 | 0.268 | **0.600** | -| 人类 vs **形状对抗**的 AI（刻意制造参差） | 0.927 | **0.810** | 0.184 | 0.316 | 0.447 | -| 人类 vs **骨架匹配**的结构克隆 | 0.914 | 0.507 | 0.292 | 0.042 | 0.333 | -| 人类 vs **长文**生成 *（长期证伪靶）* | 0.740 | 0.634 | **0.000** | 0.069 | 0.069 | -| **人类误标率**，492 篇直接打分，α = 0.05 | — | — | **0.033** | **0.043** | 0.075 | +| 人类 vs **自然** AI | 0.893 | 0.687 | 0.071 | 0.036 | 0.107 | +| 人类 vs **去 AI 化后**的 AI（15 篇，低于 20 篇下限） | 未测 | 未测 | 未测 | 未测 | 未测 | +| 人类 vs **形状对抗**的 AI（刻意制造参差） | **0.953** | **0.803** | 0.184 | 0.316 | **0.447** | +| 人类 vs **骨架匹配**的结构克隆 | 0.866 | 0.526 | 0.250 | 0.083 | 0.292 | +| 人类 vs **长文**生成 *（长期证伪靶）* | 0.840 | 0.610 | **0.000** | 0.069 | 0.069 | +| **人类误标率**，504 篇直接打分，α = 0.05 | — | — | **0.020** | **0.042** | 0.062 | 两条轴天生互补：流形对包括结构克隆在内的每一档都保持稳健判别，而 role 轴的功效 -恰好集中在把流形逼到最窄的那一档上。跨越 42 倍的语料重建，八个已发表 AUC -**全部复现在 0.012 以内** —— 这是逐值核对，不是总数对上就算数。 +恰好集中在把流形逼到最窄的那一档上。与 2026-08-25 那张表相比，长度公平 AUC 变动 −0.05 到 +0.10 +（自然 0.933 → 0.893，长文 0.740 → 0.840），其间四处改动未拆开：v0.36.3 语料、剔除附录与参考文献、切段前挖掉标题与浮动体、长度分层边界只用训练集拟合。 尾部功效照实报，包括弱的地方和为零的地方，而且**上表的尾部功效一列是单次 seed -抽样**：这些档上逐 seed 的离散度是 0.04–0.18，12 个 seed 下 natural 读作 +抽样**：这些档上逐 seed 的离散度是 0.04–0.18，12 个 seed 下（2026-08-26）natural 读作 0.170 ± 0.110。长文生成在严格操作点上任何配置都**完全抓不到**。本该修好短文档的 估计量噪声模型已经建出来并被证伪 —— 它对人类误报率毫无影响 （[§9.4c](docs/architecture/evaluation/document-scale.md)）。整张表可用 @@ -402,30 +402,30 @@ AUC 是长度公平的：每份文档只跟**自己那个长度层**的人类论 ### 为什么没有单一分数：L3 的混淆 学习型段落尺度模型效果不错 —— 而它**依然**以 `degraded` 发布，理由是测出来的。 -2026-09-05 用 44,636 条记录重训。 +2026-09-27 用 45,153 条记录重训。 来源：[§7](docs/architecture/evaluation/learned-model.md)。 | 指标 | 值 | 95% 切分区间 | 上一版语料 | |---|---:|---|---:| -| 分组切分 AUC（20 次切分，整篇论文留出） | **0.9487** | 0.9409 – 0.9583 | 0.9320 | -| 匹配层 AUC（section × 长度 × 数学 × 领域词） | **0.9262** | 0.9063 – 0.9494 | 0.9236 | -| 平衡准确率 | **0.8704** | 0.8588 – 0.8881 | 0.8445 | -| 作者 hard-set、**真实 provenance** | 0.938 | 0.846 – 0.998 | 0.937 | -| 假阳率 —— 通用公开 AI 文本 | 0.055 | 0.026 – 0.076 | 0.086 | -| 假阳率 —— **领域主题 AI 文本** | **0.295** | 0.230 – 0.356 | 0.318 | -| 假阳率 —— **领域术语密集 AI 文本** | **0.421** | 0.236 – 0.543 | 0.417 | - -0.95 的 AUC 头条数字，和 30–42% 的领域主题假阳率，是同一个模型。学习型分数有一 +| 分组切分 AUC（20 次切分，整篇论文留出） | **0.9487** | 0.9392 – 0.9587 | 0.9320 | +| 匹配层 AUC（section × 长度 × 数学 × 领域词） | **0.9250** | 0.9058 – 0.9495 | 0.9236 | +| 平衡准确率 | **0.8705** | 0.8602 – 0.8883 | 0.8445 | +| 作者 hard-set、**真实 provenance** | 0.937 | 0.842 – 0.999 | 0.937 | +| 假阳率 —— 通用公开 AI 文本 | 0.057 | 0.026 – 0.076 | 0.086 | +| 假阳率 —— **领域主题 AI 文本** | **0.292** | 0.230 – 0.349 | 0.318 | +| 假阳率 —— **领域术语密集 AI 文本** | **0.418** | 0.212 – 0.528 | 0.417 | + +0.95 的 AUC 头条数字，和 29–42% 的领域主题假阳率，是同一个模型。学习型分数有一 部分测的是*领域 register*，所以它恰恰在"去 AI 必须抓住的那个分布"上不可靠。 -**把语料放大 2.6 倍、再重建 profile 都没有修好它** —— 领域主题档 0.318 → 0.285 → 0.280 → 0.295， -每一步都落在单次重训自身的切分区间内 —— 四次重训现在一致指向：这个混淆属于特征集，不属于某一份 +**把语料放大 2.6 倍、再两次重建 profile 都没有修好它** —— 领域主题档 0.318 → 0.285 → 0.280 → 0.295 → 0.292， +每一步都落在单次重训自身的切分区间内 —— 五次重训现在一致指向：这个混淆属于特征集，不属于某一份 训练语料。这就是 L3 只做分诊、在段落尺度封顶 0.5 置信度、永远不是作者身份判决的 原因。文档级 surprisal 路径也测了（0.757），比 model-free 流形（0.881）更弱， 且对它毫无增益。 -重训**不保证行为等价，也不宣称等价**：同一批 1,808 个段落经最近两个 bundle 打分， -degraded 模式真正消费的那个**排序**保持在 Spearman **ρ = 0.991**，被推到人前复核的 -三个段落平均重合 **0.889**（此前语料放大 2.6 倍那次是 0.846 与 0.654）。schema、 +重训**不保证行为等价，也不宣称等价**：同一批 2,080 个段落经最近两个 bundle 打分， +degraded 模式真正消费的那个**排序**中位数保持在 Spearman **ρ = 1.000**，69 篇里有 68 篇 +被推到人前复核的三个段落完全相同（两次重训之前语料放大 2.6 倍那次是 0.846 与 0.654）。schema、 特征、`degraded` 姿态都没变，也没有凭空引入阈值。 ### 延迟与仓库健康度 @@ -448,10 +448,10 @@ LaTeX include 组装而成。标准库的每一行都重测了：解释器地板 | `+ --oracle`（GPT-2-large token surprisal） | 33.8 s（2026-08-27） | `transformers` + `torch` | | `+ --voice`（学习型 L3 分诊） | 37.2 s（2026-08-27） | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` —— **11/11 通过** | 2.24 s | 标准库 | -| 完整测试套件 —— **764 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | +| 完整测试套件 —— **801 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | 一句话：**一份 5,084 词的稿子跑完全部 model-free 通道，在解释器地板之上约花 1.0 s**， -且不需要任何可选依赖——其中 0.6 s 是加载 collocation 词对库（计时所用的 2026-09-04 库有 530,677 对，v0.36.3 重建后为 530,504 对）， +且不需要任何可选依赖——其中 0.6 s 是加载 collocation 词对库（计时所用的 2026-09-04 库有 530,677 对，v0.39.0 重建后为 535,517 对）， `--no-collocation` 能把它降回约 0.4 s。两条模型驱动的轴比完整的 model-free 通道贵 30–35 倍，并且是显式 opt-in 的 flag —— lint 一篇论文不该需要一块 GPU。CI 每次推到 main 的 push 与每个 PR 都跑 validator + 套件，Python 3.11，Ubuntu。 @@ -468,20 +468,13 @@ claude --plugin-dir /path/to/sci-paper # 开发模式 之后 skill 的命名空间是 `/sci-paper:`。 **Python ≥ 3.11。** 共享 schema、确定性 L0 linter、model-free 的 L1/L2 轴、 -全文结构分析和 validator **只用标准库**。可选能力才需要额外依赖： +全文结构分析和 validator **只用标准库**。可选能力才需要额外依赖，每个包启用什么 +列在 [`requirements.txt`](requirements.txt) 里： ```bash pip install -r requirements.txt # 全部可选依赖 ``` -| 包 | 启用什么 | -|---|---| -| `pymupdf` | PDF 语料抽取、编译页检查 | -| `sentence-transformers` | 语义范例检索、embedding 特征 | -| `scikit-learn` + `joblib` | 旧版与学习型领域相似度模型 | -| `transformers` + `torch` | token surprisal / UID 测量 | -| `numpy` | 学习型特征、缓存、改写打分工具 | - > 不要为了把一个"测不出来的轴"变成一个名义分数而去装可选依赖。 > 缺包就让那个轴保持 `unmeasured`，这是设计。 @@ -538,12 +531,11 @@ profile 构建，`eval` 可复现的证据。逐工具细节见 [tools/README.md | `tools/rewrite_reward.py` | core | **先过**科学保真硬门再对改写候选排序。丢掉*或凭空加上*受保护不变量 → `-inf`。 | | `tools/deai_register.py` | L0 | 领域 register：稿子反复依赖、但本领域语料里不存在的术语。判据是语料的 document frequency，而不是一张手工整理的"外来词表"。复合词按其最罕见的部分判定。零命中审计列出正文里每一个 df 为 0 的词。只产 advisory。 | | `tools/deai_collocation.py` | L2 | 把本领域从不并置的常用词并置起来的句子：相邻实词对在语料里零共现的比例，对照按 section 分桶的留一法参照；每个词对带自己的"按机会缺席"概率。只产 advisory；本文定义的术语保留它的词对。 `--glossary` 用同一个词对库按全文读：列出用到两次及以上的未匹配词对（本文自己的术语），带首次出现的行号和那句是否已经给出定义。 | -| `tools/ai_ism_negatives_handcrafted.txt` | L0 | 旧版分类器的种子负样本（数据资产）。 | | `tools/deai_metrics.py` | L1 | model-free 的信息分布 finding —— 句长变化、连接词开头 —— 带显式标定状态。 | | `tools/deai_oracle.py` | L1 | 可选的 token surprisal 与 UID 证据。资产不可用与兼容性阈值保持显式。 | | `tools/deai_structure.py` | L2 | 句子与段落构造：announced enumeration、重复框架、并列串、对称结构等模板家族；辅助家族（对偶、短反转句、论文当施事者、wh-cleft、修饰语堆叠）只命名句子，不进分数。 | | `tools/deai_salience.py` | L2 | Salience hierarchy：一段文字里的数值能连续跑多远而中间没有一句解释性句子，对照按 section 分桶的人类参照。唯一消费"保留数字"那条 LaTeX 投影的工具。 | -| `tools/deai_discourse.py` | L2 | 语篇质地，两条都打**低尾**：`cohesion` 逐段的 given/new 衔接（一句话的实词有多大比例在上一句已经出现过），`hedging` 逐 **section** 的认知情态标记密度（每千词）。这里的缺陷是「缺席」而不是「过量」，所以打低尾。两条轴**单位不同、各自的产物各写各的 `unit`**：hedging 在段落尺度根本没有低尾（`wgl` 七个桶的 p10 全是 0.000），只有按 section 重组才分得开。hedging 只对 `intro` 说话 —— 那是它的操作点被验证能迁移的唯一桶（EVALUATION §19）。 | +| `tools/deai_discourse.py` | L2 | 语篇质地，两条都打**低尾**：`cohesion` 逐段的 given/new 衔接（一句话的实词有多大比例在上一句已经出现过），`hedging` 逐 **section** 的认知情态标记密度（每千词）。这里的缺陷是「缺席」而不是「过量」，所以打低尾。两条轴**单位不同、各自的产物各写各的 `unit`**：hedging 在段落尺度根本没有低尾（`wgl` 七个桶的 p10 全是 0.000），只有按 section 重组才分得开。hedging 只对 `intro` 和 `method` 说话 —— 那是它的操作点被验证既能迁移、又分得开机器稿的两个桶（EVALUATION §19）。 | | `tools/deai_reference.py` | L2 | 所有按桶参照的轴共用的那一份 `(feature, unit)` 百分位参照：0.01 分位网格、并列平台顶端的百分位读法、30 单位样本下限、段落与 section 两条扫描、产物读取与标定循环。不持有任何策略；它唯一的不变量是「标定与检测共用同一个单位、同一张网格」—— 正是这条检查抓出了 hedging 标定在一个没有低尾的单位上。 | | `tools/deai_docshape.py` | L2 | 全文形状测量与完整文档标定：逐段特征向量、跨段 dispersion、联合 Mahalanobis 流形、role coupling、split-conformal 操作点，以及 baseline 构建器。2026-08-25 从 `deai_docstructure.py` 拆出，后者 re-export 这里每一个公开名字。 | | `tools/deai_docstructure.py` | L2 | 全文修辞形状与完整文档标定：dispersion band、按长度分层的联合流形、role coupling、split-conformal 操作点。 | @@ -557,7 +549,7 @@ profile 构建，`eval` 可复现的证据。逐工具细节见 [tools/README.md | `tools/deai_provenance.py` | L4 | 基于作者**自己**草稿历史的编辑 provenance 账本；按 token 编辑比把每段标为 AI-untouched → author-original。不是检测器；没有 AI 草稿祖先时为 `unmeasured`。 | | `tools/deai_personal.py` | L4 | 个人 dispersion 基线，对照作者自己以前的论文 —— 一个无混淆的同作者参照。少于三篇时为 `unmeasured`。 | | `tools/label_findings.py` | eval | 把五条会产出 finding 的轴（register、salience、cohesion、hedging、collocation）抽成人工标注表，再盲发一个子集算 intra-rater 一致性，按 `--population NAME=DIR` 命名的总体分层，逐轴报 precision、**合并**报 recall。任何不足 20 条标注的分层一律报 `unmeasured`，不给数字。 | -| `tools/eval_findings.py` | eval | 用**出处**当标签来测 register、salience、cohesion、hedging 与 collocation，而不是靠人工标注：在**留出**的 ApJ/ApJL/A&A 已发表论文、同体裁但参与过标定的论文（泄漏对照）、以及 `docval` 机器 tier 上各自的命中率，外加机器对留出人类的秩 AUC。register 那行是误报率；salience 那行不是——它的门是百分位，非零命中率是设计值，测的是标定迁移。 | +| `tools/eval_findings.py` | eval | 用**出处**当标签来测 register（阈值与零命中）、salience 与 collocation，而不是靠人工标注：在**留出**的 ApJ/ApJL/A&A 已发表论文、同体裁但参与过标定的论文（泄漏对照）、以及 `docval` 机器 tier 上各自的命中率，外加机器对留出人类的秩 AUC。register 那行是误报率；salience 那行不是——它的门是百分位，非零命中率是设计值，测的是标定迁移。 | | `tools/eval_docscale.py` | eval | 复现 §9 的全文尺度表 —— 人类误标率与逐 tier 尾部功效 —— 把语料与每个 `docval` tier 都送进 finding 用的同一个操作点。 | | `tools/build_profile.py` | build | 构建基础 field profile：抽取、可选的旧版分类器、范例缓存预热。 | | `tools/cli_common.py` | build | 共享的命令行前置：UTF-8 stdout，以及每个 field 感知工具都要的 `--field` / `--profile-root` 选项。不持有任何策略。 | @@ -569,6 +561,7 @@ profile 构建，`eval` 可复现的证据。逐工具细节见 [tools/README.md | `tools/fetch_arxiv_abstracts.py` | build | 抓取带日期的摘要语料用于受控评估与训练，可限定子领域 query set 与指定的 refereed 期刊；也可抓单个作者的完整 LaTeX 源（`--author` + `--author-is` + `--max-authors`）。触发限流时**停止抓取并 exit 2**，而不是把被截断的语料当作完整的写下去。 | | `tools/train_ai_ism_classifier.py` | legacy | 训练旧版 word-ngram 分类器，仅作为 degraded 的 advisory 证据使用。 | | `tools/extract_md_negatives.py` | legacy | 为受控评估与训练收集候选生成段落。 | +| `tools/ai_ism_negatives_handcrafted.txt` | legacy | 旧版分类器的种子负样本（数据资产）。 | > `tools/validate_plugin.py` 是开发/发布工具，不是产品工具，不计入上面的数量。 @@ -581,7 +574,7 @@ profile 构建，`eval` 可复现的证据。逐工具细节见 [tools/README.md | 类别 | 含义 | 强制后果 | |---|---|---| | `integrity_blocker` | 科学记录可能是错的、无支撑的、自相矛盾的、不可复现的或不可用的 | **必须**从源头解决。不能以"风格偏好"豁免。 | -| `l0_target` | 一个 Tier A 词、一个 em-dash，或同一 section 内第二次及以后出现的 Tier B 词 | 改写到零。这不等于说论文在科学上无效。 | +| `l0_target` | 一个 Tier A 词、一个 em-dash，或同一 heading 单元内第二次及以后出现的 Tier B 词 | 改写到零。这不等于说论文在科学上无效。 | | `advisory` | 结构、分布、学习型、修辞、清晰度或审美证据 | 排序，处理最强的几条，其余记录 disposition。 | 每个轴报告一个 **measurement state** —— `measured`、`degraded`、`unmeasured` @@ -610,14 +603,14 @@ style-profile// 生成的证据（gitignore） | 制品 | 规模 | |---|---| -| `exemplar_paragraphs.jsonl` | **27,851** 个按 section 分类的段落，来自 19 篇精选 + 500 篇参照论文 | -| `register_lexicon.json` | 41,644 个 passage · 53,367 个词条 | -| `uid_baseline.json` | 27,851 段（GPT-2-large）· 合并 global UID 3.303 ± 0.420 | -| `structure_baseline.json` | method 9,478 · results 3,978 · data 3,894 · intro 3,812 · discussion 3,635 · conclusion 2,611 · abstract 433 | -| `salience_baseline.json` | abstract 13,971 · method 6,967 · results 3,240 · intro 3,239 · data 3,020 · discussion 2,954 · conclusion 1,999 | -| `docstructure_baseline.json` | 507 篇完整文档 · conformal α 0.05 · 长度分层 [46, 75] | +| `exemplar_paragraphs.jsonl` | **28,444** 个按 section 分类的段落，来自 19 篇精选中的 18 篇与 500 篇参照论文中的 498 篇 | +| `register_lexicon.json` | 42,237 个 passage · 53,668 个词条 | +| `uid_baseline.json` | 28,444 段（GPT-2-large）· 合并 global UID 3.301 ± 0.413 | +| `structure_baseline.json` | method 9,654 · results 4,065 · data 3,953 · intro 3,848 · discussion 3,739 · conclusion 2,747 · abstract 433 | +| `salience_baseline.json` | abstract 13,970 · method 7,209 · results 3,337 · intro 3,280 · data 3,098 · discussion 3,051 · conclusion 2,018 | +| `docstructure_baseline.json` | 504 篇完整文档 · conformal α 0.05 · 长度分层 [49, 78] | | `anchoring_baseline.json` | 517 篇文档 · 六个 section 类全部高于 30 篇下限 | -| `voice_model.joblib` | 44,636 条记录 · 14 个特征 · **无操作点**，`degraded` | +| `voice_model.joblib` | 45,153 条记录 · 14 个特征 · **无操作点**，`degraded` | 每个桶都过了 30 passage 下限，没有一个是"仅排序" —— 这在 2026-08-25 之前并不成立： 当时 `results` 只有 26，而那被误诊为语料量不足。语料内容是**只读、受版权保护的输入**， @@ -646,7 +639,7 @@ style-profile// 生成的证据（gitignore） | 语料参照 | **用户提供、分 tier、gitignore** | 风格是相对领域而言的。通用先验正是被替换掉的那个东西。 | | 可选模型 | `transformers`+`torch`、`scikit-learn`、`sentence-transformers` | 严格 opt-in flag。缺失只降级一条轴，从不让整次运行失败。 | | 分发 | **Claude Code 插件**（`.claude-plugin/plugin.json`） | skill 放在 `skills//SKILL.md`，命名空间 `/sci-paper:`。 | -| 契约执行 | `tools/validate_plugin.py` + GitHub Actions | 11 项检查覆盖 manifest、注册表、文档权威、记录数字、import、CLI 入口、退出语义、测试、CI 接线。漂移让 CI 挂掉，而不是慢慢堆积。 | +| 契约执行 | `tools/validate_plugin.py` + GitHub Actions | 11 项检查，清单只列在 [tools/README.md](tools/README.md) 一处。漂移让 CI 挂掉，而不是慢慢堆积。 | --- @@ -662,7 +655,7 @@ sci-paper/ │ └── design-notes/ 冻结的、带日期的设计记录（不是现状文档） ├── skills//SKILL.md 12 个 skill ├── tools/ 40 个产品工具 + 仓库 validator -├── tests/ 32 个测试文件、764 个测试 +├── tests/ 32 个测试文件、801 个测试 ├── style-corpus// 用户提供的只读语料（gitignore） ├── style-profile// 生成与标定的证据（gitignore） ├── ACKNOWLEDGMENTS.md 改编来源的致谢与采纳边界 @@ -672,19 +665,15 @@ sci-paper/ ## 开发与发布 -`python tools/validate_plugin.py` 跑 11 项契约检查， -`python -m unittest discover -s tests -v` 跑 764 个测试；发布前两者都必须通过。 -Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权威边界与索引完整性、 -记录的测试规模与真实发现的一致性、过期契约标记、产品注册表、Python 语法、 -运行时 import、CLI 入口、schema 字段、linter 退出语义、Tier B 行为、测试与 CI 接线 —— -权威清单以 `tools/validate_plugin.py` 本身为准。发布还额外要求独立代码审查、 -干净 checkout 验证，以及发布 commit 上的绿色 CI。 +`python tools/validate_plugin.py` 跑 11 项契约检查（清单见 [tools/README.md](tools/README.md)）， +`python -m unittest discover -s tests -v` 跑 801 个测试；发布前两者都必须通过。 +发布还额外要求独立代码审查、干净 checkout 验证，以及发布 commit 上的绿色 CI。 --- ## 现状、已知限制与路线图 -当前版本：**v0.39.0**。完整逐版本历史见 [CHANGELOG.md](CHANGELOG.md)，更早的条目见 [CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md)（v0.33.0–v0.34.0）、[CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md)（v0.27.1–v0.32.0）、[CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md)（v0.22.0–v0.27.0）与 [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md)（v0.1.0–v0.21.0）。 +当前版本：**v0.39.0**。完整逐版本历史见 [CHANGELOG.md](CHANGELOG.md)，更早的条目见 [CHANGELOG-ARCHIVE-v0.35.md](CHANGELOG-ARCHIVE-v0.35.md)（v0.35.0–v0.35.1）、[CHANGELOG-ARCHIVE-v0.33-v0.34.md](CHANGELOG-ARCHIVE-v0.33-v0.34.md)（v0.33.0–v0.34.0）、[CHANGELOG-ARCHIVE-RECENT.md](CHANGELOG-ARCHIVE-RECENT.md)（v0.27.1–v0.32.0）、[CHANGELOG-ARCHIVE.md](CHANGELOG-ARCHIVE.md)（v0.22.0–v0.27.0）与 [CHANGELOG-ARCHIVE-EARLY.md](CHANGELOG-ARCHIVE-EARLY.md)（v0.1.0–v0.21.0）。 **规范核心：** `docs/SCIPAPER_STANDARD.md` v3.8 —— 完整的去 AI 标准全在这一个文件里 （分层模型、全文尺度检测核心、协作层与 residue 轴、`calibration_unit` 置信度封顶、§5.2 去 AI 化 @@ -697,17 +686,17 @@ Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权 |---|---| | **没有学习型模型的操作点** | L3 以 `degraded` 发布。文档级 surprisal 路径已被*测量证明*给不出操作点（0.757 vs model-free 流形的 0.881）。 | | **尾部功效数字是单次抽样** | 流形各 tier 的逐 seed 标准差是 0.04–0.18，比记录里几处曾被读作"提升"的差还大。`tools/eval_docscale.py` 用来重跑，而不是引用。 | -| **领域主题假阳** | 在领域主题与术语密集 AI 文字上 30–42%。**已以决定收口**：跨 2.6 倍语料区间的四次重训一致表明混淆在特征集里，因此这套特征拿不到对领域主题稳健的操作点。 | -| **短文档尾部功效** | 流形对短的自然 AI 文档，12 个 seed 平均 **0.170 ± 0.110**，而长度公平排序是 0.933。三条修法都做了、都失败：距离归一化（已否决）、更细分层、显式估计噪声协方差。 | -| **长文生成抓不到** | α = 0.05 下流形对长文 AI 的尾部功效是 **0.000** —— 在 2 种度量 × 4 种标定切分 × 12 个 seed 下都稳定。排序 AUC 是 0.729，说明信号在，操作点够不着。 | +| **领域主题假阳** | 在领域主题与术语密集 AI 文字上 29–42%。**已以决定收口**：跨 2.6 倍语料区间的五次重训一致表明混淆在特征集里，因此这套特征拿不到对领域主题稳健的操作点。 | +| **短文档尾部功效** | 流形对短的自然 AI 文档，12 个 seed 平均 **0.170 ± 0.110**（2026-08-26），当时长度公平排序是 0.933；2026-09-27 的单次抽样是 0.071，排序 0.893。三条修法都做了、都失败：距离归一化（已否决）、更细分层、显式估计噪声协方差。 | +| **长文生成抓不到** | α = 0.05 下流形对长文 AI 的尾部功效是 **0.000** —— 在 2 种度量 × 4 种标定切分 × 12 个 seed 下都稳定。长度公平排序 AUC 是 0.840（2026-09-27），说明信号在，操作点够不着。 | | **协作层工具** | `deai_provenance` 与 `deai_personal` 在作者提供自己的草稿历史或 ≥ 3 篇既往论文之前，诚实地保持 `unmeasured`。 | | **`L1.distribution` / `L2.sentence_structure`** | `degraded` —— 现在是有*测量依据*的。burstiness 在对抗文本上符号反转（AUC 0.181），signposting 低于随机（0.247），根本写不出一个操作点。 | -| **重训不保证行为等价** | 重建 profile 会重拟 L3。2026-09-05 那次重训排序保持 ρ 0.991、分诊重合 0.889（此前跨 2.6 倍语料增长的一步是 0.846 / 0.654），但旧的分诊清单不会一字不差复现。 | +| **重训不保证行为等价** | 重建 profile 会重拟 L3。2026-09-27 那次重训排序保持 ρ 1.000、分诊重合 0.995（跨 2.6 倍语料增长那一步是 0.846 / 0.654），但旧的分诊清单不会在每篇上都一字不差复现（[§7.0](docs/architecture/evaluation/learned-model.md)）。 | | **语料有四分之一从未被用上** | 匹配不到任何 section 桶的标题是被丢弃，而不是被猜进某个桶：`wgl` 里 9,178 个标题中有 **2,334 个（25.4%）**，`wgl-letter` 里 148 个中有 42 个。剩下的多是主题标题（"Matter power spectrum"）；"Measurements" 与 "Background" 因真歧义被拒绝加入。 | -| **register 在已发表文字上照样开火** | 在它从没见过的 203 篇留出 ApJ/ApJL/A&A 已发表论文上实测：v0.36.0 修掉标题投影、v0.36.1 修掉浮动体投影后**每千正文词 0.0371 条**（同样这 57 条在 v0.36.2 之前按原始源码分母读作 0.0247，两次修复之前是 0.0858），22.2% 的文档命中，对机器文本的秩 AUC **0.391** —— 它在人类论文上仍比在 AI 草稿上更爱开火。剩下 57 条 flag 里，98.2% 只要论文自己在库里就会消失。把用词门槛从 5 扫到 50，AUC **处处低于 0.5**：没有任何设置能把它变成检测器，它就是一条建议，切在「一篇够格送审的论文不会过半被点名」的第一个点上。这个结论已在第二个群体上复现 —— 按作者而非期刊取样的 22 篇导师论文（1996–2015），AUC **0.328**（[§21](docs/architecture/evaluation/held-out-labels.md)）。 | -| **建议质量仍未被标注** | 出处只能回答「它是否在已发表文字上开火」，回答不了「这条建议对不对」。salience 的 p90 门在 203 篇留出已审稿论文上逐 passage 开火 **0.203**，低于三门并集上界 0.271（[§17.5](docs/architecture/evaluation/held-out-labels.md)）；v0.36.2 时读作 0.454，原因是语料库只存了 `[math]` 投影，参照里数学内一个数字都没有，而每篇手稿都有 —— v0.36.3 给每个段落并排存下数字投影后关闭。v0.32.0 之前它在 LaTeX 上读到的数字有 7.00% 是引用年份；建议本身的精确率与召回率仍需 `tools/label_findings.py`，跑不跑是作者自己的事。 | -| **hedging 只对引言说话** | 认知情态轴发布时被收窄到只管 `intro` —— 在 203 篇留出的已审稿论文上，它的 p10 门在 `intro` 上开火率是 7.89%，而其他桶是 15–27% —— 那是审稿人已经接受的文字，且至少有一套生成流程落到随机以下。cohesion 不需要这条限制（七个桶 6.6–14.6%）。 | -| **两条词汇审计是建议，且要加载词对库** | 零命中审计列出语料从未写过的每一个词，而已发表论文里这种词反而比机器草稿多（每千正文词 3.37 个，203 篇留出论文篇篇都有，秩 AUC 0.174）；collocation 轴每次运行都要加载 530,504 个词对的库，model-free 那一行延迟上升的 0.6 s 就是它。两者都不是检测器；`--no-collocation` 可以跳过词对库（[§23](docs/architecture/evaluation/vocabulary-and-residue.md)）。 | +| **register 在已发表文字上照样开火** | 在它从没见过的 203 篇留出 ApJ/ApJL/A&A 已发表论文上实测：**每千正文词 0.0364 条**（56 条），22.2% 的文档命中，对机器文本的秩 AUC **0.391** —— 它在人类论文上比在 AI 草稿上更爱开火。这 56 条 flag 里，87.5% 只要论文自己在库里就会消失（[§17.4](docs/architecture/evaluation/held-out-labels.md)）。把用词门槛从 5 扫到 50，AUC **处处低于 0.5**：没有任何设置能把它变成检测器，它就是一条建议，切在「一篇够格送审的论文不会过半被点名」的第一个点上。这个结论已在第二个群体上复现 —— 按作者而非期刊取样的 22 篇导师论文（1996–2015），AUC **0.411**（[§21](docs/architecture/evaluation/held-out-labels.md)）。 | +| **建议质量仍未被标注** | 出处只能回答「它是否在已发表文字上开火」，回答不了「这条建议对不对」。salience 的 p90 门在 203 篇留出已审稿论文上逐 passage 开火 **0.201**，低于三门并集上界 0.271（[§17.5](docs/architecture/evaluation/held-out-labels.md)）；建议本身的精确率与召回率仍需 `tools/label_findings.py`，跑不跑是作者自己的事。 | +| **hedging 只对引言和方法说话** | 认知情态轴只管 `intro` 和 `method` —— 在 203 篇留出的已审稿论文上，它的 p10 门在这两个桶的开火率是 9.09% 和 9.62%，六套生成流程都分得开；在 `data`、`conclusion`、`results` 上是 12–14%（那是审稿人已经接受的文字），`discussion` 上有一套生成流程落到随机以下（[§19](docs/architecture/evaluation/discourse-and-citation.md)）。cohesion 不需要这条限制（七个桶 6.8–14.9%）。 | +| **两条词汇审计是建议，且要加载词对库** | 零命中审计列出语料从未写过的每一个词，而已发表论文里这种词反而比机器草稿多（每千正文词 3.34 个，203 篇留出论文篇篇都有，秩 AUC 0.177）；collocation 轴每次运行都要加载 535,517 个词对的库，model-free 那一行延迟上升的 0.6 s 就是它。两者都不是检测器；`--no-collocation` 可以跳过词对库（[§23](docs/architecture/evaluation/vocabulary-and-residue.md)）。 | | **全新 clone 什么都测不出来** | 全部 profile 制品都 gitignore。在你用自己的论文建出 profile 之前，每条语料参照的轴都是 `unmeasured`。 | ### 路线图 @@ -716,7 +705,7 @@ Validator 覆盖发布元数据、skill frontmatter、规范引用、文档权 自己的参照基准是 provenance：`eval_findings.py` 拿**已经存在的标注**给各轴打分。 **以证伪收口，不是以发布收口。** 长度感知流形、扩大 conformal 标定集、长文尾部功效、 -四次 L3 重训、留出对样本内的泄漏估计、register 操作点，以及 citation placement —— +五次 L3 重训、留出对样本内的泄漏估计、register 操作点，以及 citation placement —— 全部做出来、测过、然后被否决。最后一项曾是记录里最强的 model-free 判别量（AUC 0.866）， 却没通过自己预先登记的检验：换一个模型生成的第二个库，不提引用时同一统计量得 **0.053**， 提了就得 **0.734**，所以信号是提示词（[§20](docs/architecture/evaluation/discourse-and-citation.md)）。 diff --git a/docs/README.md b/docs/README.md index edb4568..1762631 100644 --- a/docs/README.md +++ b/docs/README.md @@ -60,7 +60,7 @@ universal paper PASS/FAIL. | Per-tool registry, calibration, and failure behavior | [../tools/README.md](../tools/README.md) | | Supplying a field corpus | [../style-corpus/README.md](../style-corpus/README.md) | | Generated profile assets and build boundaries | [../style-profile/README.md](../style-profile/README.md) | -| Per-version history | [../CHANGELOG.md](../CHANGELOG.md) · [../CHANGELOG-ARCHIVE-v0.33-v0.34.md](../CHANGELOG-ARCHIVE-v0.33-v0.34.md) (v0.33.0-v0.34.0) · [../CHANGELOG-ARCHIVE-RECENT.md](../CHANGELOG-ARCHIVE-RECENT.md) (v0.27.1-v0.32.0) · [../CHANGELOG-ARCHIVE.md](../CHANGELOG-ARCHIVE.md) (v0.22.0-v0.27.0) · [../CHANGELOG-ARCHIVE-EARLY.md](../CHANGELOG-ARCHIVE-EARLY.md) (v0.1.0-v0.21.0) | +| Per-version history | [../CHANGELOG.md](../CHANGELOG.md) · [../CHANGELOG-ARCHIVE-v0.35.md](../CHANGELOG-ARCHIVE-v0.35.md) (v0.35.0-v0.35.1) · [../CHANGELOG-ARCHIVE-v0.33-v0.34.md](../CHANGELOG-ARCHIVE-v0.33-v0.34.md) (v0.33.0-v0.34.0) · [../CHANGELOG-ARCHIVE-RECENT.md](../CHANGELOG-ARCHIVE-RECENT.md) (v0.27.1-v0.32.0) · [../CHANGELOG-ARCHIVE.md](../CHANGELOG-ARCHIVE.md) (v0.22.0-v0.27.0) · [../CHANGELOG-ARCHIVE-EARLY.md](../CHANGELOG-ARCHIVE-EARLY.md) (v0.1.0-v0.21.0) | | Worked example: a synthetic manuscript before and after acting on the findings | [../examples/README.md](../examples/README.md) | | Working rules for this repository | [../CLAUDE.md](../CLAUDE.md) | | Adapted-material attribution and adoption boundaries | [../ACKNOWLEDGMENTS.md](../ACKNOWLEDGMENTS.md) | @@ -79,13 +79,10 @@ accumulating: - no tracked source or document exceeds 750 lines, the same cap the author's local editing hook enforces; - every relative link, in-page anchor and numbered cross-reference in the READMEs and `docs/` resolves. -One convention it cannot enforce lives in the suite instead. The figures these -documents quote come from `style-profile//`, which is gitignored, so no -CI check can open the source of a published number. -`tests/test_published_figures.py` therefore renders each figure *from* the -artifact and looks for it in the document, and skips — rather than passes — when -a clean clone has no profile to read. The drift it exists to stop is recorded in -[§18.8](architecture/evaluation/projection-and-operating-point.md). +One convention it cannot enforce lives in the suite instead: the figures these +documents quote come from the gitignored `style-profile//`, so +`tests/test_published_figures.py` checks each against its artifact +([§18.8](architecture/evaluation/projection-and-operating-point.md)). Historical `CHANGELOG.md` entries quote the documentation paths that were current at that release; documents moved into `architecture/` and `design-notes/` in diff --git a/docs/SCIPAPER_STANDARD.md b/docs/SCIPAPER_STANDARD.md index 1bb360a..cfc4334 100644 --- a/docs/SCIPAPER_STANDARD.md +++ b/docs/SCIPAPER_STANDARD.md @@ -131,10 +131,11 @@ quality dimensions. ### L0: preserved lexical de-AI standard -The canonical Tier A and Tier B lists, em-dash rule, grep patterns, and writing -examples remain in [`../skills/paper/SKILL.md`](../skills/paper/SKILL.md). The -field dossier and generated lexicon are calibration data, not a competing -policy authority. +The canonical Tier A and Tier B lists are the linter's own patterns in +`tools/ai_ism_lint.py`; [`../skills/paper/SKILL.md`](../skills/paper/SKILL.md) +mirrors them, with the em-dash rule, grep patterns, and writing examples, and the +validator checks that mirror in both directions. The +field dossier and generated lexicon are calibration data, not a competing policy authority. L0 contains exactly: @@ -201,8 +202,7 @@ Advisory; a defined term keeps its pair, and no claim is changed to dissolve one A **blind perceptual panel** — independent cold-read judges who score AI-feel and must name concrete tells with quotes across document versions — is a recognized L2 validation instrument, read by *tell inventory turnover* rather -than mean score: judges saturate on the most visible family, so removing it -exposes the next stratum at a similar score (protocol in `EVALUATION.md`). +than mean score (protocol and evidence: EVALUATION §13). Document-level evidence concerns shape rather than repeated subject matter, and is the de-AI center of gravity: field register shifts the *level* of @@ -270,12 +270,9 @@ text must not say that a low score proves machine authorship. A universal The per-paragraph learned classifier (`deai_voice`, axis `L3.voice`) is an **offline audit instrument, not a product detector one calibration away from an -operating point**: a decided status, fixed by the three measured facts recorded -in EVALUATION §10, §7 and §9.8 (the paragraph unit is near-unjudgeable, the -field-topic level confound cannot be escaped at that unit, and document-level -surprisal adds nothing to the model-free manifold). L3 therefore stays `degraded` -with no operating point, emits rank-only triage, and its per-paragraph findings -are confidence-capped (§3, `calibration_unit`). +operating point**, a decided status (evidence: EVALUATION §7, §9.8 and §10). L3 +therefore stays `degraded` with no operating point, emits rank-only triage, and +its per-paragraph findings are confidence-capped (§3, `calibration_unit`). ### L4: positive scientific voice @@ -742,8 +739,7 @@ engineering item has a decided disposition, so the standard rests on no undecided obstacle. The register itself — every item, its disposition, and the measurement behind it -— is [`architecture/DISPOSITIONS.md`](architecture/DISPOSITIONS.md), moved there -on 2026-08-25 when this file passed the repository's line budget. It is a record +— is [`architecture/DISPOSITIONS.md`](architecture/DISPOSITIONS.md). It is a record of decisions, not independent policy: this document stays the single normative contract. Adoption of any item requires passing the §9 confound audit and keeping the suite and validator green, and updates that register and diff --git a/docs/architecture/DEAI_SUBSYSTEM.md b/docs/architecture/DEAI_SUBSYSTEM.md index e6f263c..f42ba6c 100644 --- a/docs/architecture/DEAI_SUBSYSTEM.md +++ b/docs/architecture/DEAI_SUBSYSTEM.md @@ -45,7 +45,7 @@ A narrow deterministic rewrite target: - Tier A lexical occurrence; - em-dash occurrence; -- Tier B occurrence above the cap of one occurrence per section and word. +- Tier B occurrence above the cap of one occurrence per heading unit and word. The target count is reduced to zero during editing, but an L0 target does not imply that the manuscript is scientifically invalid. @@ -110,8 +110,9 @@ axis at this layer that never joins the to-zero set. It asks whether the draft speaks its own field's vocabulary, by comparing terms the manuscript leans on (≥ 15 uses) against document frequency in the field's own corpus. The evidence is frequency, never a curated list of a neighbouring discipline's words, because a -list cannot separate `AUC` (1 corpus passage) from `epoch` (402) and `accuracy` -(774). Three constructions are handled rather than thresholded: hyphenated +list cannot separate `AUC`, which the field's corpus almost never writes, from +`epoch` and `accuracy`, which it writes routinely (EVALUATION §14.4). Three +constructions are handled rather than thresholded: hyphenated compounds are judged by their rarest part, `\mathrm{}` preceded by `_` or `^` in a macro body is a subscript decoration rather than a term, and possessives fold onto the bare term. Macro bodies are read because the shared reduction erases @@ -191,8 +192,10 @@ to find. Where a reference has no spread above the advisory gate the feature abstains rather than reporting an ordinary passage as the 100th percentile. This detector is the only one that reads -[`extract_style.latex_to_numeral_text`](../../tools/extract_style.py), the second -LaTeX projection (the bank writer runs it too, to store `numeral_text`). `latex_to_plain` replaces every math span with `[math]`, which +[`extract_sections.latex_to_numeral_text`](../../tools/extract_sections.py), the second +LaTeX projection (`rewrite_reward` reads numbers through it too, and the bank +stores each paragraph's `numeral_text` under the same projection through +`extract_style.paired_paragraphs`). `latex_to_plain` replaces every math span with `[math]`, which is right for lexical and shape statistics and zeroes every numeral signal on `.tex` input; the numeral-preserving projection shares the same pattern set and differs only in what happens inside an inline math span. Displayed equations are @@ -200,15 +203,14 @@ dropped by both. The reference is built from the same projection: every bank row carries `numeral_text` beside `text`, paired paragraph by paragraph by `extract_style.paired_paragraphs`, and the calibration reads that field. Until v0.36.3 it read `text`, so the reference held no numeral inside math while the -manuscript did, and the p90 gate fired at 0.45 per held-out passage against a -0.27 bound (EVALUATION §17.5). +manuscript did (EVALUATION §17.5). ### L2: collocation [`../tools/deai_collocation.py`](../../tools/deai_collocation.py) asks a question the register axis cannot: whether a sentence joins words the field never joins. -Each word may be ordinary — `physical` and `cells` both are — and the pair still -one no passage of 41,644 has written. The unit is the sentence, the feature the +Each word may be ordinary — `binned` and `kernel` both are — and the pair still +one no passage of the field's banks has written. The unit is the sentence, the feature the fraction of its distinct adjacent content-word pairs that the bank does not attest, and the reference is leave-one-out per bucket: at calibration a pair the bank saw in exactly one passage is treated as that passage's own, so the @@ -219,8 +221,8 @@ breaks at punctuation, a `[math]`/`[CITE]` placeholder, a dash, a slash, a period or a digit, because `yields, separate` is two clauses. Each flagged pair carries its expected passage co-presence λ = df(a)·df(b)/N and e^−λ (`expected_copresent_passages`, `p_copresence_absent`; a ranking aid, never a -filter), so a reader can see that `measurements projected` (λ = 183) is a -stronger absence than `sub-halo abundances` (λ = 0.3). +filter), so a reader can see that a pair of two frequent words is a stronger +absence than a pair of two rare ones. A document-level novel-pair fraction is reported as evidence and used by `eval_findings`; it is not a percentile, because the reference is per sentence. @@ -241,11 +243,10 @@ appeared in the sentence before it, and `L2.hedging`, epistemic markers per 1,000 words. They are the first axes in the subsystem to measure at **different units**, and -that is forced rather than chosen. Hedging has no paragraph-scale lower tail at -all: on the `wgl` bank its tenth percentile is exactly 0.000 in every one of the -seven section buckets, because a 40-word paragraph that hedges nowhere is -entirely ordinary. Calibrating there produces a gate no passage can fall below. -Regrouped so one section is one unit, six of seven buckets separate. So cohesion +that is forced rather than chosen. Hedging has no paragraph-scale lower tail +(EVALUATION §19.1), because a 40-word paragraph that hedges nowhere is entirely +ordinary, and calibrating there produces a gate no passage can fall below. +Regrouped so one section is one unit, the buckets separate. So cohesion calibrates and detects per paragraph, hedging per section, each artifact records its own `unit`, and the two are never read against each other. @@ -260,7 +261,7 @@ drift apart on what "the reference cannot resolve this" means. That guard is wha surfaced the hedging unit problem instead of shipping it. Hedging additionally speaks only for buckets whose operating point was shown to -transfer — `intro` on `wgl`. Two independent measurements put the boundary in the +transfer — `intro` and `method` on `wgl`. Two independent measurements put the boundary in the same place, and both are recorded in EVALUATION §19. Neither axis is an authorship claim: a passage below the field's tenth percentile is unusual for the field, which is worth telling an author, and is not evidence about who wrote it. @@ -284,7 +285,7 @@ The sweep drops the preamble and every `skip` unit (acknowledgements, appendices references) on both sides, and a document-shape metric is strong when it exceeds the quoted quantile threshold. If the corpus does not contain enough complete and measurable papers, the axis is `unmeasured`; the `wgl` figures in EVALUATION §9 -predate the rebuild these changes require. The implementation must not synthesize a +were re-measured on the rebuilt reference on 2026-09-27. The implementation must not synthesize a document baseline from paragraph exemplars. ### L3: learned field similarity @@ -298,13 +299,12 @@ field-similarity triage. A bundle without a documented calibrated operating point is degraded. Evaluation must separate source-paper groups and audit mathematical-placeholder density, jargon density, section type, and paragraph length. The mathematical-density -confound was audited (EVALUATION §7.1: matched-stratum AUC within 0.02 of -overall); the field-topic confound is decided not resolvable from this feature -set (§7.0a). +confound was audited (EVALUATION §7.1); the field-topic confound is decided +not resolvable from this feature set (§7.0a). The per-paragraph learned classifier is an **offline audit instrument, not a product detector one calibration away from an operating point**: the paragraph -unit is near-unjudgeable for AI-ness (perceptual AUC 0.444, EVALUATION.md §7), and the document-level +unit is near-unjudgeable for AI-ness (EVALUATION §7, §13), and the document-level surprisal path is now measured (EVALUATION.md §9.8) to be weaker than the model-free manifold and to add nothing to it. Accordingly, `make_finding` carries a `calibration_unit` (paragraph|section|document) that structurally caps @@ -323,22 +323,24 @@ sentence defining the paper's own object by what it never does or has, the prose form of the negative label, deleted outright where the neighbouring sentences already carry the positive statement and rewritten only where they do not: `never` and the `nothing is` / `none sees` / -`no … is applied` forms are strong, at 0.008 per 1,000 words in refereed prose; -`carries no`, `is not applied`, `does not participate` are ordinary, at -0.02–0.05 per 1,000 and mostly procedure; a citation in the sentence makes it a +`no … is applied` forms are strong, being rare in refereed prose; `carries no`, +`is not applied`, `does not participate` are ordinary, being common there and +mostly procedure (EVALUATION §23.4a); a citation in the sentence makes it a baseline contrast and exempts it); `residue-edit-meta` (`TODO`, `see previous version`, case-sensitive for the upper-case markers; `we have added` only with a document object, since in refereed prose it is a procedure); `residue-negative-label` (a heading or caption whose object head stems never occur in the body, on documents of 400 words or more — ordinary, -because it names 26% of refereed papers); and, given `--before` or `--git-ref`, +because refereed papers carry it routinely, EVALUATION §23.4); and, given +`--before` or `--git-ref`, `residue-negative-label-added` (a label the edit introduced and the body does not earn — strong). The literal and label rules read `deai_register.body_only`, because a `\newcommand{\TODO}` in a preamble and a bibliography title are not prose an edit left. The history and absence families are defined once in the tool and mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` calls the tool's own `validator_check`, which proves the mirror -and scans the shipped documentation for the edit-meta literals. A strong finding +and scans both READMEs, the standard and every skill for the edit-meta literals. +A strong finding exits 1, one of the four narrow exit contracts the standard's §0.1 registers (with `length_gate`, `rewrite_reward` and `verify_references`). @@ -440,26 +442,10 @@ standard. ## 10. Validation and release boundary -[`../tools/validate_plugin.py`](../../tools/validate_plugin.py) checks: - -- manifest, README, and CHANGELOG version agreement; -- skill frontmatter and standard references; -- stale review-contract markers; -- README and manifest skill/tool counts; -- exact README product-tool registry; -- Python syntax and core imports; -- core command-line entry points; -- shared schema fields and allowed enums; -- linter exit and Tier B cap semantics; -- normative/evaluation document authority boundaries; -- required tests and CI wiring; -- the residue contract: the history families the paper skill quotes match the - tool's, and no shipped document carries an edit-meta literal; -- every tracked file within the 750-line budget; -- relative links, in-page anchors and numbered cross-references resolve. - -The authoritative check list is `validate_plugin.py` itself (`tools/README.md` -mirrors it); this summary is descriptive. +[`../tools/validate_plugin.py`](../../tools/validate_plugin.py) runs the +repository's release checks. What each one covers is listed once, in +[`tools/README.md`](../../tools/README.md), which mirrors the validator's own +`CHECKS` tuple. CI also runs the unit and CLI test suite. A release additionally requires an independent code review, a clean-checkout verification, release metadata updates, @@ -476,7 +462,7 @@ open: invariant verification — EVALUATION §11 and §13; 2. learned-model audits for mathematics, jargon, section, length, and source-paper confounds — §7; -3. complete-document calibration — §9, measured over 507 complete papers; +3. complete-document calibration — §9, measured over the field's complete papers; 4. UID and learned-model operating points — both `degraded` by measurement, §6 and §7, with the refutations that keep them so in [`DISPOSITIONS.md`](DISPOSITIONS.md); diff --git a/docs/architecture/DISPOSITIONS.md b/docs/architecture/DISPOSITIONS.md index 31cce4f..6d266a7 100644 --- a/docs/architecture/DISPOSITIONS.md +++ b/docs/architecture/DISPOSITIONS.md @@ -1,8 +1,7 @@ # De-AI subsystem status and open-item dispositions -The disposition register for `sci-paper`. Split out of -[`SCIPAPER_STANDARD.md`](../SCIPAPER_STANDARD.md) §11 on 2026-08-25, when that -file passed the repository's 750-line budget and could no longer be edited. +The disposition register for `sci-paper`, split out of +[`SCIPAPER_STANDARD.md`](../SCIPAPER_STANDARD.md) §11 on 2026-08-25. This is a **record of decisions**, not independent policy. The standard remains the single normative contract and nothing here can redefine a consequence class, @@ -15,9 +14,7 @@ this register and `EVALUATION.md` together. The ranked de-AI frontier is complete ([`DEAI_FRONTIER.md`](../design-notes/DEAI_FRONTIER.md)). Every remaining engineering item has a decided disposition, so the standard rests -on no undecided obstacle. Adoption of any item requires passing the §9 confound -audit and keeping the suite and validator green, and updates this table and -`EVALUATION.md` together. +on no undecided obstacle. | Item | Disposition | Reason | |---|---|---| @@ -32,7 +29,7 @@ audit and keeping the suite and validator green, and updates this table and | Subfield reference for the register axis | **Rejected as harmful** | 254 documents cannot express a rate below 1/254, 39.4× coarser than the 1e-4 threshold, so core field terms flip to foreign on zero counts (EVALUATION §15.5). | | Cooperative layer (`deai_partition`, `deai_anchoring`, `deai_provenance`, `deai_personal`) | **Shipped** | Partition/anchoring `measured`; provenance/personal `unmeasured` by design until the author supplies own history/papers. | | `L3.voice` operating point | **Decided degraded** | Offline audit instrument; per-paragraph unit near-unjudgeable and document-level surprisal refuted (§2 L3, EVALUATION §7, §9.8). | -| Field-topic-robust L3 operating point | **Not obtainable from this feature set** | Four retrains across a 2.6× bank range and a profile rebuild (17,299 / 41,641 / 44,576 / 44,636 records), each with its own 20-split grouped audit: field-topic false positives 0.318 → 0.285 → 0.280 → 0.295, every step inside one retrain's own split range, while headline AUC rose and eased 0.9320 → 0.9518 → 0.9487. The confound is in the features, not the sample. Reopening requires a different feature set, not a larger bank (EVALUATION §7.0a). | +| Field-topic-robust L3 operating point | **Not obtainable from this feature set** | Five retrains across a 2.6× bank range and two profile rebuilds (17,299 / 41,641 / 44,576 / 44,636 / 45,153 records), each with its own 20-split grouped audit: field-topic false positives 0.318 → 0.285 → 0.280 → 0.295 → 0.292, every step inside one retrain's own split range, while headline AUC rose, eased and held: 0.9320 → 0.9518 → 0.9487 → 0.9487. The confound is in the features, not the sample. Reopening requires a different feature set, not a larger bank (EVALUATION §7.0a). | | Human-labelled validation for salience and register | **Closed on the repository side; the labelling itself is an author act** | The provenance half is measured (EVALUATION §17) on 203 held-out refereed papers. What only a labeller can say is whether an individual advisory is good *advice*, and recall — neither is obtainable from provenance, and neither is a repository item: provenance labels are the chosen ground truth (2026-08), and whether to hand-label is the author's decision, taken outside this record. The harness stands (`tools/label_findings.py`): finding-level labels, one labeller plus a blind re-label subset, named populations stratified, `unmeasured` below 20 per stratum. Extended 2026-08-26 from two axes to all four that emit findings (register, salience, cohesion, hedging) and on 2026-09-04 to collocation; recall is pooled over them, because a control row records that a passage was missed but not by which axis. | | `L1.distribution` operating point | **Decided degraded — refuted, not merely absent** | `deai_policy.json` is withdrawn as an adoption candidate. Measured on 500 human papers against 173 AI documents, one observation per document: burstiness reverses sign on adversarial prose (AUC 0.181) and long-form sits inside the human band (0.441) while flagging 7.2% of humans; signposting runs below chance at AUC 0.247 and flags 0 of 173 AI documents. Same criterion that rejected the inference-connective rate. No threshold may be built on either statistic (EVALUATION §16). | | `L1.uid` operating point | **Decided degraded** | No field-policy-calibrated compatibility operating point; the surprisal path is measured not to add document-level power. | @@ -43,22 +40,22 @@ audit and keeping the suite and validator green, and updates this table and | Baseline unification into one `(feature, unit)` object (rank 2) | **Done (v0.33.0)** | It stopped being elegance debt when a second per-bucket axis arrived: `deai_salience` and `deai_discourse` would otherwise have carried two copies of the quantile grid, the plateau-top percentile reader, the passage sweep and the calibration loop, and a percentile would have meant two different things depending on which tool computed it. `tools/deai_reference.py` now owns all of it, plus the `(feature, unit)` invariant itself — which immediately earned its keep: it is what caught the hedging axis calibrating at a unit that has no lower tail (EVALUATION §19.1). The artifact reader had five copies before this module existed. | | Jargon-conditional per-paragraph operating point (rank 3) | **Won't pursue as scoped** | The jargon confound is handled at document scale by per-stratum + conformal; a per-paragraph operating point is inconsistent with the L3-degraded decision. | | `corpus_cos` ablation (rank 4) | **Won't pursue — cannot change a shipped decision** | `confound_audit` bins on record metadata the feature cache does not carry, so an ablation runnable from the cache computes a different statistic from the four recorded retrains (§7.0a) and cannot be compared with them. Its only consumer is a `degraded`, audit-only classifier with no shipped operating point, whose status rests on independently refuted grounds, so no result could change shipped behaviour (EVALUATION §18.7). | -| Citation placement as an axis | **Refuted (v0.33.0) — on its own pre-registered condition** | v0.32.0 measured it as the strongest model-free discriminator in the record (section-matched AUC 0.866 in `method`, surviving the section, length and human-vs-human controls) and declined to ship it for one stated reason: all 173 machine documents came from one generation process, so one bank could not separate "AI cites more" from "these prompts made it cite more". Two new 20-document banks from a **different model** (Codex `gpt-5.6-terra`), differing in one prompt line, answer it: with no citation instruction the same statistic scores **0.053** — separation of nearly equal strength in the opposite direction — and with one it scores **0.734**. Density swings 12.5× (1.00 to 12.55 per 1,000 words) around a human median of 6.20, so the machine extremes bracket the human distribution instead of sitting on one side of it. Same failure mode as §15.1, §16 and §18.4: a statistic whose sign depends on which machine population you sampled is not a discriminator, whatever its AUC on the sample you have (EVALUATION §20). | -| Topic-orthogonal axes: cohesion, hedging (rank 6) | **Shipped (v0.33.0), one of them restricted** | `tools/deai_discourse.py`, both flagging the LOW tail at p10 — the defect here is absence, not excess. Cohesion ships at paragraph unit across all seven buckets: held-out transfer 10.78% on the v0.36.3 profile (10.87% at v0.33.0) against a 10% design point, worst-of-six-regimes AUC 0.676 in `intro` against a 0.515 null. Hedging has **no paragraph-scale lower tail** (p10 is 0.000 in all seven buckets, because a 40-word paragraph that hedges nowhere is ordinary), so it calibrates and detects at SECTION unit, and ships restricted to `intro` — two independent measurements agree on that boundary: held-out transfer 7.80% there on the v0.36.3 profile (7.89% at v0.33.0) against 15.48–26.77% elsewhere, and worst-of-six AUC 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376 (EVALUATION §19). | +| Citation placement as an axis | **Refuted (v0.33.0) — on its own pre-registered condition** | v0.32.0 declined to ship its strongest model-free discriminator for one stated reason: all 173 machine documents came from one generation process, so one bank could not separate "AI cites more" from "these prompts made it cite more". Two banks from a different model, differing in one prompt line, put the statistic on opposite sides of the human distribution. A statistic whose sign depends on which machine population was sampled is not a discriminator — the failure mode of §15.1, §16 and §18.4 (EVALUATION §20). | +| Topic-orthogonal axes: cohesion, hedging (rank 6) | **Shipped (v0.33.0), one of them restricted** | `tools/deai_discourse.py`, both flagging the LOW tail at p10 — the defect here is absence, not excess. Cohesion ships at paragraph unit across all seven buckets; hedging has no paragraph-scale lower tail, so it calibrates and detects at SECTION unit and ships restricted to `intro` and `method`, where a transfer test and a separation test agree (widened from `intro` alone on 2026-09-27, by re-running both). Figures: EVALUATION §19.3–19.5. | | Long-form generation (`ai_long`) | **Standing falsification target — robustly measured** | A recorded measured limit (EVALUATION §9.7). No longer a single-configuration result: tail power holds at 0.000 across two metrics, four calibration splits and 12 seeds, while rank AUC is 0.729 — the signal is present and the operating point cannot reach it (EVALUATION §9.4c). | | Single-seed tail-power figures as estimates | **Rejected as a reporting practice** | Per-seed spread on the manifold tiers is 0.04–0.18, wider than several differences the record previously read as improvements. §9 figures now carry the spread, and `tools/eval_docscale.py` reproduces the table rather than it being quoted (EVALUATION §9.4c). | | Hand labels as the only route to register/salience precision | **Superseded — half of it never needed a labeller** | A refereed ApJ/ApJL/A&A paper's provenance is already a label for "does this axis fire on accepted prose". 203 held-out papers, verified disjoint from all three calibration banks, now measure both axes (EVALUATION §17). `label_findings.py` keeps the remaining half: whether an individual advisory is good advice. | -| Comparing held-out against in-sample papers to estimate leakage | **Rejected — confounded** | The two populations are era-disjoint (2020–2021 vs 2012–2018), so the contrast charges six years of vocabulary drift to calibration leakage. Replaced by a paired test on one population where bank membership is the only thing that differs: 86.3% of 887 held-out register flags would be suppressed by the paper's own membership (EVALUATION §17.3; re-measured at 94.44% of 198 in §18.4 after the citation fix and the new use floor). | -| Asymmetric projection between calibration and detection | **Fixed — it was 58.7% of the measured false positives** | The corpus df is built from body prose (`extract_style` drops the preamble and every `skip` bucket) while `manuscript_terms` read the whole raw file, so an affiliation line or a `\bibitem` surname had df 0 on one side and count ≥ 5 on the other. Measured on the 203 held-out papers: 27.5% preamble, 26.3% bibliography, 4.1% TeX control words, 0.9% `skip` sections. `deai_register.body_only` restores the symmetry, taking the held-out rate 0.991 → 0.384 per 1,000 words with no threshold changed. `deai_salience` was checked and is clean — 0 of its 1,077 findings on the same papers fall in a bibliography (EVALUATION §17.4). **Superseded figures:** a second instance of the same class (citation commands) and the new use floor took the rate to 0.0858 in §18. | +| Comparing held-out against in-sample papers to estimate leakage | **Rejected — confounded** | The two populations are era-disjoint (2020–2021 vs 2012–2018), so the contrast charges six years of vocabulary drift to calibration leakage. Replaced by a paired test on one population where bank membership is the only thing that differs (EVALUATION §17.3); its reading on the current build is the last row of §17.4's table. | +| Asymmetric projection between calibration and detection | **Fixed — it was 58.7% of the measured false positives** | The corpus df is built from body prose (`extract_style` drops the preamble and every `skip` bucket) while `manuscript_terms` read the whole raw file, so an affiliation line or a `\bibitem` surname had df 0 on one side and count ≥ 5 on the other. `deai_register.body_only` restores the symmetry with no threshold changed; `deai_salience` was checked and is clean. The breakdown, and every later move of the rate, is §17.4's table. | | Count floor as the guard for a rate gate | **Rejected — replaced by a resolution guard** | `MIN_CORPUS_PASSAGES` (a count) cannot protect `RARE_DF_RATE` (a rate): a bank of n passages expresses no non-zero rate below 1/n, so under 10,000 passages the rule silently becomes `df == 0`. §15.5 derived this in v0.26.1 and used it only to reject one bank; the shipped `wgl-letter` profile (706 passages, 14.2x coarser) then reported `measured` with no reason. The axis now reports `degraded` and its findings carry `measurement_status: degraded`, without being silenced (EVALUATION §17.4). | | Asymmetric projection, second instance: citation commands | **Fixed — 46 command names, 8,835 leaking uses** | `RE_TEX_CITE` listed four names and required the brace to follow the name directly, so `\citep[e.g.][]{Smith2020}` and six other natbib forms fell through to `RE_TEX_SIMPLE_CMD`, which substitutes a command's argument as text: the bibliography key became a word. 64 of 887 held-out register findings (7.2%) were leaked surnames, and 7.00% of every digit the salience axis read on LaTeX was a citation year. Matched by shape rather than by an allowlist, because the tail is per-paper local macros (EVALUATION §18.1–18.2). | | Held-out bundles collectable as calibration input | **Fixed — a structural guard, not a caller convention** | `corpus_documents` walks with `rglob` and filtered nothing, so the documented `--calibrate --corpus-dir style-corpus/wgl` collected 717 documents where the shipped baseline had 517 — the extra 200 being the whole evaluation set, absorbed silently. The existing held-out test pinned `extract_style`'s source tuple, which this path never consults. The guard now lives in the collector so every caller inherits it (EVALUATION §18.3). | -| `L0.register` operating point | **Closed by refuting its premise** | Swept 5 → 50 uses against 203 held-out refereed papers and 173 machine documents: rank AUC is below 0.5 at **every** setting, so the axis fires more on refereed prose than on machine prose everywhere on the curve and tightening silences the machine side faster. No setting makes it a detector; the knob buys advisory volume. Cut at 15, the first point where a referee-grade paper is not flagged more often than not: 0.0858 per 1,000 words on 44.83% of held-out documents, AUC 0.2856 (EVALUATION §18.4). | -| `L0.register` threshold as the vocabulary instrument | **Complemented by the zero-hit audit (v0.36.0); the 15-use rule stays as the selective instrument** | The 15-use floor kept the axis from flagging refereed prose more often than not, at the cost of missing every word a draft uses once. The owner's rule sits beside the knob: every body word with corpus df 0 is listed, and the only exemptions are mechanical (a formation of an attested stem, a term defined here at first use, a proper name capitalised at every use; a cited method's own name is an author disposition). Measured on 203 held-out refereed papers against 173 machine documents the audit fires on every refereed paper (3.37 words per 1,000 body prose words under v0.36.2's denominator; 2.21 per raw-source word under v0.36.1's float projection, 2.66 before it) and has rank AUC 0.174 (0.246 before the denominator fix) — it is not a detector and is not shipped as one; it is exhaustive advice whose dispositions are the author's (EVALUATION §23.1). | +| `L0.register` operating point | **Closed by refuting its premise** | Swept 5 → 50 uses against 203 held-out refereed papers and 173 machine documents: rank AUC is below 0.5 at **every** setting, so the axis fires more on refereed prose than on machine prose everywhere on the curve and tightening silences the machine side faster. No setting makes it a detector; the knob buys advisory volume. Cut at 15, the first point where a referee-grade paper is not flagged more often than not (EVALUATION §18.4; the current rate is the last row of §17.4's table). | +| `L0.register` threshold as the vocabulary instrument | **Complemented by the zero-hit audit (v0.36.0); the 15-use rule stays as the selective instrument** | The 15-use floor kept the axis from flagging refereed prose more often than not, at the cost of missing every word a draft uses once. The owner's rule sits beside the knob: every body word with corpus df 0 is listed, and the only exemptions are mechanical (a formation of an attested stem, a term defined here at first use, a proper name capitalised at every use; a cited method's own name is an author disposition). Measured on 203 held-out refereed papers against 173 machine documents, the audit fires on every refereed paper and more on them than on machine text (rank AUC 0.177) — it is not a detector and is not shipped as one; it is exhaustive advice whose dispositions are the author's (EVALUATION §23.1). | | Asymmetric projection, third instance: headings | **Fixed — one regex owner** | Section headings sat in the manuscript's body projection but never in a corpus passage, so `Validation` fused with the first sentence under it and read as a zero-hit word; the same fusion put heading words into collocation pairs. `extract_sections.RE_HEADING_COMMAND` is now the single owner and `deai_reference.units`/`sections`, `deai_register.body_only` and `length_gate` all consume it (EVALUATION §23.1). | -| Collocation as an axis (`L2.collocation`) | **Shipped (v0.36.0), `measured`; glossary reading v0.38.0** | Per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (v0.36.3 rebuild; 41,710 before), leave-one-out reference per bucket at sentence unit; distinct pairs, breaks at punctuation, placeholders, dashes, slashes, periods and digits, 11,282 admitted partner words (11,286 before). Flags five of the six mentor-marked phrases present in a manuscript under review and separates machine documents from held-out refereed papers on the document fraction (EVALUATION §23.2). | -| Residue axis (`L4.residue`) with a diff gate | **Shipped (v0.36.0), strengths set on refereed prose; absence rule added v0.37.0** | Four static rules (self-history, edit-meta, negative-label, and `residue-absence`, a body sentence defining the paper's own object by what it never does, §23.4a) and one diff rule. Strengths were measured, not assumed: on 203 held-out papers the first families put a strong finding in 154 documents; the body projection (preamble macros and "Planck Collaboration XXX" are not prose), dropping `used to` (174 instrumental hits) and demoting `initially`/`originally`/`at first` to ordinary bring that to 34 (16.7%), and v0.36.2 — marks read across line breaks, `we have added` a mark only with a document object (fourteen procedural hits otherwise) — to 27 (13.3%), 8 of them `\textcolor{red}` in tables. The static negative-label rule is ordinary (26% of refereed papers); the diff rule gates. The skill mirrors the family lists and the validator proves the mirror. Exit 1 on a strong finding is the plugin's third narrow exit contract (EVALUATION §23.4). | +| Collocation as an axis (`L2.collocation`) | **Shipped (v0.36.0), `measured`; glossary reading v0.38.0** | Per-sentence fraction of adjacent common-word pairs unattested in the field's passage banks, leave-one-out reference per bucket at sentence unit; distinct pairs, breaks at punctuation, placeholders, dashes, slashes, periods and digits. Flags five of the six mentor-marked phrases present in a manuscript under review and separates machine documents from held-out refereed papers on the document fraction (EVALUATION §23.2). | +| Residue axis (`L4.residue`) with a diff gate | **Shipped (v0.36.0), strengths set on refereed prose; absence rule added v0.37.0** | Four static rules (self-history, edit-meta, negative-label, and `residue-absence`, a body sentence defining the paper's own object by what it never does, §23.4a) and one diff rule. Strengths were measured, not assumed: four passes on 203 held-out papers took the documents with a strong finding from 154 to 27 — the body projection, dropping the instrumental `used to`, demoting `initially`/`originally`/`at first` to ordinary, and `we have added` a mark only with a document object — and read with comments blanked, as both CLIs read a manuscript, the three original static rules put one in 24 of them on 2026-09-27 (EVALUATION §23.4). The static negative-label rule is ordinary (26% of refereed papers); the diff rule gates. The skill mirrors the family lists and the validator proves the mirror. Exit 1 on a strong finding is the plugin's third narrow exit contract (EVALUATION §23.4). | | Structure families from the mentor's comments | **Shipped as auxiliary (v0.36.0)** | Paper-as-agent, wh-cleft and modifier-stack join the auxiliary class: named on the sentence, excluded from `template_score`, baseline fractions recorded per bucket (EVALUATION §23.3). | -| Condensation as a read-and-trim pass | **Replaced by the removal map (v0.36.0)** | The skill's own sweep removed a few percent by hand; `condense_map.py` lists every entry with the words it frees and `length_gate.py --require-shrink` turns the map's target into an exit code. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%; 3.1% before v0.36.2 counted each unit once), so a pass that cuts less than that has not found what the map lists (EVALUATION §23.5). | +| Condensation as a read-and-trim pass | **Replaced by the removal map (v0.36.0)** | The skill's own sweep removed a few percent by hand; `condense_map.py` lists every entry with the words it frees and `length_gate.py --require-shrink` turns the map's target into an exit code. Held-out refereed papers carry a median default target of 1.22% of prose (p90 3.39%), so a pass that cuts less than that has not found what the map lists (EVALUATION §23.5). | | Spaced units outside the unit vocabulary (rewrite fidelity, §8) | **Accepted limit** | A unit written with an ASCII space and absent from the vocabulary (`5 furlongs`) loses unit-level protection; its number stays a protected invariant, and `rewrite_reward` gates eligibility on it without emitting a finding. Recorded here so EVALUATION §8 no longer carries it as an open item. | | Enlarging `wgl-letter` to resolve the gate | **Rejected — format is not a domain** | Impractical (ApJL is 24 papers in 5,364 arXiv records, so 10,000 passages needs ~90,000 fetched) and wrong: the register axis measures domain vocabulary and a letter is a format. On 36 letter-format documents the 706-passage letter bank produced 262 findings the field bank did not — `sne`, `bao`, `pantheon`, `posteriors`, and `letter` itself — against 2 the other way. A `-` profile now judges against `` and names the borrowed bank in every finding (EVALUATION §18.5). | diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index b78da54..3549990 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -1,7 +1,8 @@ # EVALUATION: de-AI subsystem for `sci-paper` v0.39.0 -First recorded 2026-07-12; every section-keyed measurement re-derived against the -rebuilt `wgl` profile on 2026-08-17. +First recorded 2026-07-12. The profile-dependent figures in §2 were re-taken on the +`wgl` profile rebuilt on 2026-09-27; a row that was not re-taken names the build it +was measured on. ## 0. Section map @@ -30,7 +31,7 @@ is cited from. | **14** | 14. Salience hierarchy and domain register (v0.26.0) | [`narrative-salience-register.md`](evaluation/narrative-salience-register.md) | | **15** | 15. Narrative salience: two more refuted features, and two reference nulls (v0.26.1) | [`narrative-salience-register.md`](evaluation/narrative-salience-register.md) | | **16** | 16. `L1.distribution`: the operating point is refuted, not merely absent (v0.28.0) | [`lexical-structure-uid.md`](evaluation/lexical-structure-uid.md) | -| **17** | 17. Held-out refereed papers as labels: register and salience measured (v0.30.0, corrected v0.31.0, superseded by 18) | [`held-out-labels.md`](evaluation/held-out-labels.md) | +| **17** | 17. Held-out refereed papers as labels: register and salience measured (v0.30.0; every re-measurement since is tabled in §17.4) | [`held-out-labels.md`](evaluation/held-out-labels.md) | | **18** | 18. Projection symmetry, the register operating point, and citation placement (v0.32.0) | [`projection-and-operating-point.md`](evaluation/projection-and-operating-point.md) | | **19** | 19. Discourse texture: cohesion and hedging (v0.33.0) | [`discourse-and-citation.md`](evaluation/discourse-and-citation.md) | | **20** | 20. Citation placement refuted by the second bank (v0.33.0) | [`discourse-and-citation.md`](evaluation/discourse-and-citation.md) | @@ -52,78 +53,31 @@ result. ## 2. Current axis status -> ### ⚠️ Every section-keyed figure below is post-2026-08-25. Rebuild before reading them. -> -> Two rounds of corpus-layer defects were fixed on 2026-08-25 and the profile -> was rebuilt against both. **A machine still holding an older profile must -> treat every section-keyed axis below as `degraded`, whatever this table -> says**, until it reruns the rebuild. The repository ships no baseline (all -> are gitignored), so a fresh clone is `unmeasured` and unaffected. -> -> **Round 1 — section labels.** `classify_section` matched titles in the -> singular only, so `Results`/`Conclusions`/`Systematics` fell to `method`; -> `method` was itself the default bucket and absorbed every unnamed heading; -> PDF table cells were accepted as headings and PDF "paragraphs" were line -> fragments. An unrecognised heading is now `unknown` and is dropped rather -> than guessed. -> -> **Round 2 — what counts as a paper.** The corpus layer treated a *file* as a -> paper and could only see the three curated tiers. Four consequences, each -> measured: `\include` fragments counted as separate papers (one review entered -> every distribution twelve times); selecting the root instead lost the body it -> includes (72 words in place of 64,657); the root-selector and the reader -> resolved `\input` targets differently, costing four arXiv bundles most of -> their prose; and a `\subsection` did not inherit its `\section`, sending -> 54.8% of all section words to `unknown`. Separately, the 500-paper -> `fulltext-arxiv/` breadth corpus — already on disk for §9 — was invisible to -> every paragraph-level baseline. -> -> | bucket | v0.27.1 (31 files) | v0.28.0 | v0.29.0 (heading coverage) | v0.32.0 (citation fix) | v0.36.3 (profile rebuild) | -> |---|---:|---:|---:|---:|---:| -> | abstract | 15 | 433 | 433 | 433 | 433 | -> | intro | 109 | 3,753 | 3,844 | 3,840 | 3,812 | -> | data | 112 | 3,929 | 3,915 | 3,908 | 3,894 | -> | method | 163 | 8,144 | 9,522 | 9,512 | 9,478 | -> | discussion | 118 | 3,088 | 3,653 | 3,647 | 3,635 | -> | conclusion | 48 | 2,533 | 2,610 | 2,609 | 2,611 | -> | results | 26 | **3,118** | **3,964** | **3,958** | **3,978** | -> | **total** | **593** | **25,005** | **27,951** | **27,907** | **27,841** | -> -> Consequences, stated rather than smoothed over: -> -> - **Every bucket now clears the 30-passage floor**, `results` by 104×. No -> bucket is rank-only, and the `results` limitation recorded since v0.27.0 is -> cleared — the binding constraint was never corpus availability but a -> corpus layer that could not see it. -> - The curated tiers and the breadth corpus are **distinct roles**. The tiers -> carry every weighted aggregate and the dossier; the breadth corpus is -> unweighted and feeds the reference distributions only, so it cannot restyle -> the imitation target. `retrieve_exemplars` reads the curated tiers by -> default. -> - Register composition changes materially and for the better: abstracts fall -> from 96% of the reference to 35%. -> - One arXiv bundle in 500 still loses 35% of its prose — it ships chapter -> files with no root that assembles them. -> -> Rebuild: `python tools/build_profile.py --field ` then the -> `--calibrate` commands listed in `style-profile/README.md`, including a full -> `train_voice_model.py` retrain (§7). +> **Every section-keyed figure below reads the profile rebuilt on 2026-09-27.** A +> machine holding an older profile must treat every section-keyed axis as +> `degraded`, whatever this table says, until it reruns the rebuild: +> `python tools/build_profile.py --field `, then the `--calibrate` +> commands in `style-profile/README.md`, including a full `train_voice_model.py` +> retrain (§7). The repository ships no baseline (all are gitignored), so a +> fresh clone is `unmeasured` and unaffected. How the paragraph bank reached its +> present shape, and why no count before v0.28.0 compares with one after, is +> recorded once, in [§5](evaluation/lexical-structure-uid.md). | Axis | Status | Current evidence | Required next evidence | |---|---|---|---| | L0 lexical/punctuation | measured | Deterministic Tier A, em-dash, and Tier B cap implementation with CLI regression tests. | Continue regression coverage when policy changes. | -| L1 distribution | degraded (refuted) | §16: measured on 500 human papers against 173 `docval` AI documents, one observation per document. Burstiness reverses sign — adversarial prose is *more* bursty than the human median (1.036 vs 0.775, AUC 0.181) and long-form sits inside the human band (AUC 0.441) — while flagging 7.2% of humans. Signposting has an AUC of 0.247, below chance, and flags 0 of 173 AI documents at the shipped default. | None. `deai_policy.json` is withdrawn as a roadmap item: the statistics do not support an operating point, so the axis stays advisory by measurement rather than by absence. | +| L1 distribution | degraded (refuted) | §16 (v0.28.0; not re-taken on later builds): measured on 500 human papers against 173 `docval` AI documents, one observation per document. Burstiness reverses sign — adversarial prose is *more* bursty than the human median (1.036 vs 0.775, AUC 0.181) and long-form sits inside the human band (AUC 0.441) — while flagging 7.2% of humans. Signposting has an AUC of 0.247, below chance, and flags 0 of 173 AI documents at the shipped default. | None. `deai_policy.json` is withdrawn as a roadmap item: the statistics do not support an operating point, so the axis stays advisory by measurement rather than by absence. | | L1 UID | degraded | `style-profile/wgl/uid_baseline.json` records paragraph-level GPT-2-large summaries. | A documented operating point and human false-flag behavior; audit sensitivity to mathematics and jargon. | -| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,971; method 6,967; data 3,020; intro 3,239; discussion 2,954; results 3,240; conclusion 1,999 after the v0.36.3 profile rebuild); P(X ≤ x) on a 0.01 quantile grid; abstains where the reference cannot resolve above the gate. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; recall. §17.5: the p90 gate was measured to transfer to unseen refereed papers essentially exactly — 0.2705 per passage on 203 held-out ApJ/ApJL/A&A papers (§17.5's headline; 0.2776 when re-run in the v0.36.2 process) against the 1 − 0.9³ = 0.2710 three-feature union bound — **while the manuscript side bucketed every subsection whose title matched no bucket as `unknown` and never measured it** (344 of one paper's 540 paragraphs). With subsections inheriting their parent bucket (v0.36.2) the same papers fired at **0.4542** per passage over 9,849 scorable paragraphs. That excess was a projection seam after all, on the reference side: the bank stored only the `[math]` projection, so the reference counted no numeral inside math while the manuscript side counted every one. With each bank row carrying its numeral projection and the reference calibrated on it (v0.36.3) the same papers fire at **0.2034** per passage (2,003 findings), under the union bound as three correlated gates must be, and machine text separates at AUC **0.663** on body-word density (0.572 before). §18.2: 7.00% of the digits this axis read on LaTeX were citation years, now removed; 0 of its 2,759 findings on those papers start on a bibliography line. §22: the same class in the other direction — a quantity written as `\newcommand{\Nsamples}{12}` was unreadable at its use site and mis-attributed to the preamble at its definition site; expansion now runs once on the assembled root. 88.2% of 390 corpus documents never use the construction, and both rebuilt baselines move by **zero to four decimals** at the p90 and p95 gates in all seven buckets, so no figure here changes. Every bucket clears the 30-passage floor by two orders of magnitude. | -| L0 register | measured | §14: document frequency over 41,644 corpus passages, 53,367 terms (v0.36.3 profile rebuild; 41,710 and 53,414 before it, and the 57 held-out findings below are the same 57 under both); compound-by-rarest-part and macro-subscript handling; native-term controls pass. Since v0.32.0 a `-` profile whose own bank cannot resolve the 1e-4 gate judges against `` and names the borrowed bank in every finding: on 36 letter-format documents the 706-passage `wgl-letter` bank produced 262 findings the field bank did not (`sne`, `bao`, `pantheon`, and `letter` itself) against 2 the other way (§18.5). | Recall below the 15-use floor, and a **measured** false-positive rate: §18.4 put it at 0.0858 findings per 1,000 words on 203 held-out refereed papers (44.83% of documents), rank AUC 0.2856 against machine text — it still fires more on human prose than on AI drafts — and §23.1 re-measures the same papers at 0.0351 (30.05%, AUC 0.352) after the heading projection fix and at 0.0247 (22.2%, AUC 0.392) after v0.36.1's float projection fix, same direction; v0.36.2 counts per body prose word instead of per raw-source word, and the same 57 findings read 0.0371 (AUC 0.391). §17.3, re-measured: 94.44% of the 198 remaining flags would be suppressed by the paper's own bank membership (98.2% of the 57 under v0.36.1). The operating point is no longer an open item: swept 5→50 uses, rank AUC stays below 0.5 everywhere, so no setting makes this a detector and the knob buys advisory volume only. It was cut at 15, the first point where a referee-grade paper is not flagged more often than not (§18.4). §21 replicates that refutation on a second population — 22 papers by one author, 1996–2015, sampled by author rather than by journal and spanning a different two decades — where rank AUC is **0.328**, again below 0.5 (0.412 on the v0.36.3 profile rebuild, 5 findings on the 22 papers). | -| L2 sentence structure | measured for deterministic matches; degraded for strength | `style-profile/wgl/structure_baseline.json` provides section-level reference fractions over 27,841 paragraphs — method 9,478; results 3,978; data 3,894; intro 3,812; discussion 3,635; conclusion 2,611; abstract 433. §23.3 adds the per-bucket human fractions of the three auxiliary families taken from a mentor's comments (paper-as-agent 0.13–1.00%, wh-cleft 0.00–0.23%, modifier stacks 2.5–15.9%). | Author-labelled difficult cases. A calibrated strong-advisory threshold is no longer expected from `deai_policy.json` (§16). | -| L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 41,644 passages (530,504 pairs over 11,282 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,391; method 27,207; intro 14,321; results 12,509; data 12,048; discussion 12,183; conclusion 8,588 after the v0.36.3 profile rebuild; 41,710 passages and 541,309 → 530,677 pairs under v0.36.2); p90 gates 0.44–0.59, p95 0.50–0.67. On the first measured draft of a manuscript under review it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at AUC 0.704 (7,237 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same manuscript: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; a second generation process (§20's lesson). | -| L4 residue | measured | §23.4: four static rules (self-history, edit-meta, negative-label, and since v0.37.0 the absence rule, §23.4a) and one diff rule, deterministic, strengths set on 203 held-out refereed papers (70,848 body prose sentences): strong self-history fires in 20 papers, strong edit-meta in 8 (68 of 69 are `\textcolor{red}` in tables), any strong in 27 (13.3%); the static negative-label rule is ordinary at 26% of papers. Four passes to get there, the first a projection asymmetry and the last the procedural sense of `we have added`. | Recall against a labelled edit history; only the author's own drafts can supply it. | -| L0 register zero-hit audit | measured, not a detector | §23.1: every body word with corpus df 0, on both sides of one projection (headings now excluded on both). All 203 held-out refereed papers carry zero-hit words, 3.37 per 1,000 body prose words (2.21 per raw-source word under v0.36.1, 2.66 before its float fix), against 1.05 for the 173 machine documents — rank AUC 0.174, the audit fires more on human prose — so it ships as exhaustive advice with author dispositions, never as a gate. The two projection fixes moved the thresholded rule from 0.0858 to 0.0247 findings per 1,000 raw-source words on the same papers (0.0371 per body prose word). | None as a detector. Whether the mechanical exemptions (attested stem, defined term, proper name) are the right three. | -| L2 document structure | measured | §9: cross-paragraph dispersion calibrated one-observation-per-paper over 493 complete human `wgl` papers — the 2026-08-17 scoring; the v0.36.3 rebuild (2026-09-05) re-fit the artifact on 507 documents and the operating-point figures were not re-scored on it — human false-flag at the shipped conformal operating point 0.0325 (manifold) and 0.0426 (role) against nominal α = 0.05. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. | Re-score §9 through `eval_docscale.py` on the 507-document artifact; continue recalibration when the corpus changes. -| L3 learned field similarity | degraded (confound-audited) | Confound-aware audit complete (§7): repeated grouped-split AUC 0.9487, matched-stratum AUC 0.9262, hard-set true-provenance AUC 0.938 (2026-09-05 retrain), but a 30–42% false-positive rate on field-topic AI text. Document-level now measured (§9.8): surprisal dispersion (0.757) is weaker than the model-free manifold (0.881) and adds nothing to it, so L3 stays degraded for a measured reason. | None from this feature set: four retrains agree the confound is in the features (§7.0a), so reopening needs a different feature set; the surprisal path is measured not to provide one. | -| L2 cohesion | measured | §19: given/new linkage per paragraph against the field's own bank (abstract 13,967; method 6,886; intro 3,228; results 3,211; data 2,989; discussion 2,926; conclusion 1,980 after the v0.36.3 profile rebuild), flagged on the LOW tail at p10. The gate transfers to 203 held-out refereed papers at 10.78% (10.87% at v0.33.0) against a 10% design point, in every bucket (6.58%–14.61%), and separates all six machine regimes in `intro` at worst-of-six 0.676 against a 0.515 human-vs-human null. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether an individual advisory is good advice; recall. Whether the `intro` separation holds for a second generation process — §20 is what happens when that question is not asked. | -| L2 hedging | measured, restricted to `intro` | §19: epistemic markers per 1,000 words per SECTION (abstract 10,404; intro 502; method 438; conclusion 383; discussion 327; results 317; data 299 after the v0.36.3 profile rebuild). It has no paragraph-scale lower tail — p10 is 0.000 in all seven buckets — so calibration and detection both run at section unit, and each artifact records its own. Restricted to `intro` by two independent measurements that agree: held-out transfer is 7.80% there (7.89% at v0.33.0) against 15.48–26.77% elsewhere, and worst-of-six AUC is 0.613 (null 0.460) there while `conclusion` runs below chance at 0.376. Measured on the quantile grid the axes used until 2026-09-27, which sat one order statistic high; the artifact must be recalibrated and the rate re-measured before this figure moves. | Whether the restriction generalizes beyond `wgl`. On `wgl-letter` the axis is `degraded`: no bucket clears the 30-unit floor after the restriction. | +| L2 salience hierarchy | measured | §14: per-bucket passage reference from the field's own banks, calibrated on the numeral projection of each passage (abstract 13,970; method 7,209; data 3,098; intro 3,280; discussion 3,051; results 3,337; conclusion 2,018 on the 2026-09-27 rebuild); P(X ≤ x) on a 0.01 quantile grid at nearest rank; abstains where the reference cannot resolve above the gate. The smallest bucket clears the 30-passage floor 67-fold. The p90 gate transfers to 203 held-out refereed papers at 0.2007 per passage (1,975 findings over 9,840 scorable paragraphs) against the 1 − 0.9³ = 0.2710 three-feature union bound, under it as three correlated gates must be, and machine text separates at rank AUC 0.666 on body-word density. §17.5 is the record of the two projection seams that moved this rate; §18.2 of the citation years removed from its numerals; §22 of the numbers held in macros. | Whether an individual advisory is good advice; recall. | +| L0 register | measured | §14: document frequency over 42,237 corpus passages, 53,668 terms (2026-09-27 rebuild); compound-by-rarest-part and macro-subscript handling; native-term controls pass. A `-` profile whose own bank cannot resolve the 1e-4 gate judges against `` and names the borrowed bank in every finding (§18.5). The false-positive rate is measured: on 203 held-out refereed papers the rule fires 0.0364 times per 1,000 body prose words (56 findings, 22.17% of documents) at rank AUC 0.391 against machine text — more on human prose than on AI drafts — and 87.5% of those flags would be suppressed by the paper's own bank membership. The table closing §17.4 is the one account of how the rate reached this value. Swept 5 → 50 uses, rank AUC stays below 0.5 at every setting (§18.4), and a second population, 22 papers by one author, agrees at 0.411 (§21). | Recall below the 15-use floor. There is no operating point to find: no setting makes this a detector, so the floor buys advisory volume only, and it is cut at 15, the first point where a referee-grade paper is not flagged more often than not. | +| L2 sentence structure | measured for deterministic matches; degraded for strength | `style-profile/wgl/structure_baseline.json` provides section-level reference fractions over 28,439 paragraphs — method 9,654; results 4,065; data 3,953; intro 3,848; discussion 3,739; conclusion 2,747; abstract 433. §23.3 adds the per-bucket human fractions of the three auxiliary families taken from a mentor's comments (paper-as-agent 0.11–0.95%, wh-cleft 0.00–0.22%, modifier stacks 2.6–15.9%). | Author-labelled difficult cases. A calibrated strong-advisory threshold is no longer expected from `deai_policy.json` (§16). | +| L2 collocation | measured | §23.2: per-sentence fraction of adjacent common-word pairs unattested in 42,237 passages (535,517 pairs over 11,368 partner words), leave-one-out reference per bucket at sentence unit (abstract 89,377; method 28,114; intro 14,529; results 12,911; data 12,346; discussion 12,584; conclusion 8,741 on the 2026-09-27 rebuild); p90 gates 0.44–0.58, p95 0.50–0.67. On the first measured draft of a manuscript under review it flagged five of the six mentor-marked phrases present; the document novel-pair fraction separates machine documents from held-out refereed papers at rank AUC 0.703 (7,172 findings on the 203 papers). The document-scope glossary reading (v0.38.0) on the same manuscript: 252 distinct unattested pairs, 42 used twice or more, 16 with a definition cue at first use; the default run's finding set is unchanged (86 before and after, set difference 0). | Whether an individual advisory is good advice; a second generation process (§20's lesson). | +| L4 residue | measured | §23.4: four static rules (self-history, edit-meta, negative-label, and since v0.37.0 the absence rule, §23.4a) and one diff rule, deterministic, strengths set on 203 held-out refereed papers and re-measured on 2026-09-27 on the source as both CLIs read it, comments blanked: strong self-history fires in 20 papers, strong edit-meta in 5 (53 of its 54 hits are `\textcolor{red}`), strong absence in 16; any strong in 38 (18.7%), 24 (11.8%) without the absence rule. The static negative-label rule is ordinary at 26% of papers. | Recall against a labelled edit history; only the author's own drafts can supply it. | +| L0 register zero-hit audit | measured, not a detector | §23.1: every body word with corpus df 0, on both sides of one projection (headings excluded on both). All 203 held-out refereed papers carry zero-hit words, 3.34 per 1,000 body prose words, against 1.05 for the 173 machine documents — rank AUC 0.177, the audit fires more on human prose — so it ships as exhaustive advice with author dispositions, never as a gate. | None as a detector. Whether the mechanical exemptions (attested stem, defined term, proper name) are the right three. | +| L2 document structure | measured | §9: cross-paragraph dispersion calibrated one-observation-per-paper over 504 complete human `wgl` papers and re-scored through the shipped path on 2026-09-27 — human false-flag at the shipped conformal operating point 0.020 (manifold) and 0.042 (role) against nominal α = 0.05, in-sample; length-fair manifold AUC 0.840–0.953 across the natural, adversarial, skeleton and long-form tiers, long-form tail power 0.000. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. | Continue recalibration when the corpus changes. | +| L3 learned field similarity | degraded (confound-audited) | Confound-aware audit complete (§7): repeated grouped-split AUC 0.9487, matched-stratum AUC 0.9250, hard-set true-provenance AUC 0.937 (2026-09-27 retrain), but a 29–42% false-positive rate on field-topic AI text. Document-level now measured (§9.8): surprisal dispersion (0.757) is weaker than the model-free manifold (0.881) and adds nothing to it, so L3 stays degraded for a measured reason. | None from this feature set: five retrains agree the confound is in the features (§7.0a), so reopening needs a different feature set; the surprisal path is measured not to provide one. | +| L2 cohesion | measured | §19: given/new linkage per paragraph against the field's own bank (abstract 13,966; method 7,138; intro 3,269; results 3,309; data 3,069; discussion 3,022; conclusion 1,988 on the 2026-09-27 rebuild), flagged on the LOW tail at p10. The gate transfers to 203 held-out refereed papers at 11.15% against a 10% design point, in every bucket (6.76%–14.88%), and separates all six machine regimes in `intro` at worst-of-six 0.677 against a 0.489 human-vs-human null. | Whether an individual advisory is good advice; recall. Whether the `intro` separation holds for a second generation process — §20 is what happens when that question is not asked. | +| L2 hedging | measured, restricted to `intro` and `method` | §19: epistemic markers per 1,000 words per SECTION (abstract 10,400; intro 503; method 438; conclusion 384; discussion 329; results 317; data 303 on the 2026-09-27 rebuild). It has no paragraph-scale lower tail — p10 is 0.000 in all seven buckets — so calibration and detection both run at section unit, and each artifact records its own. Restricted to `intro` and `method` by two independent measurements that agree: held-out transfer is 9.09% and 9.62% there against 12.23–14.17% in `data`, `conclusion` and `results`, and worst-of-six AUC is 0.605 (null 0.487) and 0.731 (null 0.442) there, while `discussion`, which transfers at 8.23%, has a regime below chance at 0.421. | Whether the restriction generalizes beyond `wgl`. On `wgl-letter` it is inherited, and the `method` reference sits exactly at the 30-unit floor. | | Citation placement | **refuted, not shipped** | §20: the v0.32.0 candidate (section-matched AUC 0.866 in `method`) does not hold its sign across generation processes. A second bank from a different model scores 0.053 with no citation instruction and 0.734 with one — one prompt line, a 12.5× density swing, and the two machine extremes bracket the human distribution rather than sitting on one side of it. | None. Reopening requires a statistic that holds its sign across independently produced banks. | | Rewrite scientific fidelity | measured for protected invariants | Unit tests cover preserved invariants, dropped number, dropped citation, reversed comparison, display-math values and exponents (v0.27.0), and the punctuation/adjacent-word tokenizer boundaries (§8). | Real manuscript before/after demonstration, including scope and stance review. The plain-ASCII-space unit boundary in §8 is an accepted limit ([DISPOSITIONS](DISPOSITIONS.md)): the number stays protected, the unit does not. | @@ -131,48 +85,34 @@ A missing baseline is not interpreted as zero findings. ## 3. Repository verification -The repository validator checks manifest/version agreement, skill contract references, -product registries, Python syntax, runtime imports, CLI entry points, feedback schema, -linter exit semantics, Tier B cap behavior, tests, and CI wiring: +The repository validator runs the release checks listed once, in +[`tools/README.md`](../../tools/README.md); the unit and CLI suite covers the rest: ```bash python tools/validate_plugin.py python -m unittest discover -s tests -v ``` -The working tree passes the validator and all 764 unit/CLI tests (32 test files, collected +The working tree passes the validator and all 801 unit/CLI tests (32 test files, collected 2026-09-27). These commands must be rerun after every subsequent code or release-metadata change; the release record must quote the fresh output rather than a past result. -Figures quoted from a generated profile are pinned the same way, by -`tests/test_published_figures.py`: each one is rendered *from* the artifact it -came from and then looked for, so a document that quotes a number the artifact -no longer holds fails. Those artifacts are gitignored and no CI runner can see -them, so on a clean clone the cases **skip** rather than pass — absence is -reported as absence, which is the same rule this record applies to every axis. -The three drift events it exists to stop are recorded in §18.8. +Figures quoted from a generated profile are pinned by +`tests/test_published_figures.py`, which renders each one from its artifact and +skips, rather than passes, on a clean clone that has none; §18.8 records how it +works and the drift it exists to stop. ## 12. Release evidence boundary Current release gates (as of 2026-09-27; last tagged release v0.38.0, this release v0.39.0): `validate_plugin.py` all 11 checks pass and the full unit/CLI suite -(764 tests, 32 files) passes on a clean tree; both are rerun before every tag, +(801 tests, 32 files) passes on a clean tree; both are rerun before every tag, and as of v0.25.1 the hosted CI run on the release commit must also be green (first green runs: 31133202443 push, 31133215203 manual dispatch). -Historical record — v0.14.0 release gates and their status on 2026-07-12: - -- validator and 36 unit/CLI tests after final edits — met (rerun before tag); -- independent multi-agent review and fixes — met: an adversarially verified Opus review - (4 dimensions × 2 verifiers per finding) confirmed 16 findings, all fixed this cycle; -- real introduction rewrite evidence — met (§11, proposal-only, author disposition - pending); -- confound-aware learned-model status with an explicit degraded result — met (§7): the - audit and author hard set keep L3 degraded with no operating point; -- documentation and release metadata — updated for this release; -- clean-checkout verification — performed before tag; -- commit, tag, push, and GitHub release — performed after the gates above are re-run - green. +Earlier releases' gates are recorded with each release in the CHANGELOG and its +archives (v0.14.0's, the first set, in +[`CHANGELOG-ARCHIVE-EARLY.md`](../../CHANGELOG-ARCHIVE-EARLY.md)). Author decisions outside this record, which do not block a release: accepting or rejecting the §11 rewrite proposal ([DISPOSITIONS](DISPOSITIONS.md)). The L3 diff --git a/docs/architecture/RESPONSIBILITIES.md b/docs/architecture/RESPONSIBILITIES.md index 43f23b2..719493a 100644 --- a/docs/architecture/RESPONSIBILITIES.md +++ b/docs/architecture/RESPONSIBILITIES.md @@ -2,8 +2,8 @@ **Incorporated by reference into [`SCIPAPER_STANDARD.md`](../SCIPAPER_STANDARD.md) §§7-8.** These tables bind every component and carry the same force as the rest -of the standard; they were moved here on 2026-08-25 when that file passed the -repository's 750-line budget. The annex has **no independent authority**: it +of the standard, from which they moved here on 2026-08-25. The annex has **no +independent authority**: it cannot create a consequence class, a measurement state, or a stopping rule, and where it and the standard could be read to differ, the standard wins. @@ -27,7 +27,7 @@ overrides the standard. | `final-review` | Preserve independent isolated reviews (paper-review, figure-review, de-ai audit, physics, mainline, logic); merge typed findings; resolve blockers and L0 targets; record advisory dispositions and unmeasured axes. | | `calibrate` | Walk one field from an empty corpus to calibrated axes and, where a labeller is available, to measured advice quality; report every axis's floor and leave a stratum that cannot support a rate `unmeasured`; never lower a floor, never admit a held-out document into calibration, and never turn a calibrated distribution into a detector or a verdict. | | `physics` | Measurement primitive. First-principles verification (P1-P8): dimensions, asymptotics, symmetry/parity/conservation, the preconditions of information-theoretic bounds, algebraic re-derivation, numerical provenance, foundational citations, build integrity. Emit findings only; never carry a private word list or a zero-advisory gate. | -| `mainline` | Measurement primitive. Build the purpose record and contribution graph, then answer the cold reader's seven questions; classify each confusion by consequence. Assume no three-act template and no single-component graph. Emit findings only. | +| `mainline` | Measurement primitive. Build the purpose record and contribution graph, then answer the cold reader's eight questions; classify each confusion by consequence. Assume no three-act template and no single-component graph. Emit findings only. | | `logic` | Measurement primitive. Claim graph, empirical statistical methodology, and the review side of claim-evidence discipline: verb strength may not exceed evidence strength. Emit findings only; never mark register by word list. | | `proposal-polish` | Funding-proposal register (vision plus feasibility): keep backed ambition, enforce claim-feasibility matching, apply the L0 policy and §6 rewrite invariants unchanged; never fabricate support. | @@ -49,9 +49,9 @@ scope, not a standard-reference obligation. | `deai_metrics.py` | L1 | Emit calibrated distribution findings and explicit missing-baseline status. | | `deai_oracle.py` | L1 | Emit surprisal/UID findings with observed and reference values; advisory-success exit 0. | | `deai_structure.py` | L2 sentence | Emit template evidence with calibration metadata; advisory-success exit 0. The auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) name the sentence under `structure-auxiliary` and never enter `template_score`. | -| `deai_salience.py` | L2 salience | Measure recital structure against a per-section human passage reference at one shared unit; one finding per passage; report `measured` only where a bucket built at that unit clears the floor and resolves above the gate, and count the document's units left in `unknown`. Sole consumer of `extract_style.latex_to_numeral_text`, the numeral-preserving projection (`latex_to_plain` replaces math with `[math]`, which zeroes every numeral signal on `.tex` input). | -| `deai_register.py` | L0 register | Compare manuscript vocabulary against field-corpus document frequency; judge compounds by their rarest part; read macro bodies but not subscript decorations; emit advisories only, never `l0_target`s. The zero-hit audit (`register-zero:`) is exhaustive over body words on both sides of the same projection (headings, comments, preamble and bibliography excluded on both), strong unless the manuscript supplies a reason the tool can read: a mechanical formation of an attested stem, a defining sentence or acronym expansion at first use, or a proper name capitalised at every use (a cited method's own name is an author disposition, not an exemption). | -| `deai_collocation.py` | L2 collocation | Judge, per sentence, the fraction of adjacent common-word pairs (unigram df at or above `COMMON_RATE`) that no corpus passage attests, against a leave-one-out per-section reference at sentence unit; a pair breaks at punctuation, a placeholder or a dash; carry each pair's chance-absence probability so a reader can weigh it; advisories only, and the action names the relation, never the claim. Read document-wide (`--glossary`), list every unattested pair used twice or more with its first-use line and whether a definition cue follows it there; every occurrence counts as a use. Report `degraded` with counts where units fall in the `unknown` bucket or a bucket's artifact records another unit. | +| `deai_salience.py` | L2 salience | Measure recital structure against a per-section human passage reference at one shared unit; one finding per passage; report `measured` only where a bucket built at that unit clears the floor and resolves above the gate, and count the document's units left in `unknown`. The only detector that reads `extract_sections.latex_to_numeral_text`, the numeral-preserving projection (`latex_to_plain` replaces math with `[math]`, which zeroes every numeral signal on `.tex` input); `rewrite_reward` also calls it, and the exemplar bank stores each paragraph under the same projection through `extract_style.paired_paragraphs`. | +| `deai_register.py` | L0 register | Compare manuscript vocabulary against field-corpus document frequency; judge compounds by their rarest part; read macro bodies but not subscript decorations; emit advisories only, never `l0_target`s. The zero-hit audit (`register-zero:`) is exhaustive over body words on both sides of the same projection (headings, floats, comments, preamble and bibliography excluded on both), strong unless the manuscript supplies a reason the tool can read: a mechanical formation of an attested stem, a defining sentence or acronym expansion at first use, or a proper name capitalised at every use (a cited method's own name is an author disposition, not an exemption). | +| `deai_collocation.py` | L2 collocation | Judge, per sentence, the fraction of adjacent common-word pairs (unigram df at or above `COMMON_RATE`) that no corpus passage attests, against a leave-one-out per-section reference at sentence unit; a pair breaks at punctuation, a placeholder, a dash, a slash, a period or a digit; carry each pair's chance-absence probability so a reader can weigh it; advisories only, and the action names the relation, never the claim. Read document-wide (`--glossary`), list every unattested pair used twice or more with its first-use line and whether a definition cue follows it there; every occurrence counts as a use. Report `degraded` with counts where units fall in the `unknown` bucket or a bucket's artifact records another unit. | | `deai_residue.py` | L4 residue | Report self-referential drafting history (first-person sentence, no citation), a body sentence that defines the paper's own object by what it never does or has (`residue-absence`; `never` strong, a citation in the sentence exempts), edit-meta literals, and a heading or caption whose object the body never names; with `--before`/`--git-ref`, the added-label diff rule; exit 1 on a strong finding (§0.1); keep the family lists in one place and let the validator prove the skills quote them. Scan for editing marks in every visible line, the `skip` sections included. | | `condense_map.py` | QD (§5.3) | Enumerate every removable entry in six scans, each with `removable_words` and, for a restatement, its canonical home; total a default target that excludes the abstract/conclusion carve-out; never delete anything, never emit a blocker, exit 0/2. | | `deai_discourse.py` | L2 cohesion + L2 hedging | Measure given/new linkage and epistemic-marker rate against the field's own distribution, each at its declared unit, and flag the LOW tail. Report one status per feature, never a joint one: they calibrate from the same corpus at different granularities, so a field can support one and not the other. Speak only for buckets whose operating point was shown to transfer. Refuse a bucket whose artifact records another unit, and say so; a section is one unit per bucket; report `degraded` with counts where units fall in `unknown`. | @@ -66,8 +66,8 @@ scope, not a standard-reference obligation. | `deai_partition.py` | L4 | Suggest fidelity-free merge/split operations toward the human dispersion band; zero-token, suggest-only. | | `deai_provenance.py` | L4 | Label author edit depth vs a designated AI-draft ancestor from the author's own history; `unmeasured` without one. | | `deai_personal.py` | L4 | Compare a draft to the author's own prior papers (confound-free reference); `unmeasured` below three papers. | -| `eval_findings.py` | evidence | Score the register, salience, cohesion, hedging and collocation axes (the last on its document novel-pair fraction, because its reference is per sentence) on a **held-out** refereed corpus that fed no calibration bank, and report the same-genre in-sample population beside it so the leakage gap stays visible. Never present the salience rate as a false-positive rate — its gate is a percentile, so a non-zero rate is its design point — and never present either rate as precision for the advice itself. Added 2026-08-27: the axes' precision had stood at `unmeasured` pending hand labels, while a refereed paper's provenance was already a label for half of the question. | -| `eval_docscale.py` | evidence | Reproduce the EVALUATION §9 table through the same `manifold_operating_point` that findings use — never a private scoring path. Label the human rate **in-sample** wherever it includes the manifold's own train and calibration documents, and never present a single-seed tail-power figure as an estimate. Added 2026-08-26: nothing bound the evidence-producing path before, and §9 accumulated headline figures produced by a 95th-percentile cut rather than by the shipped conformal rule. Report `unmeasured` for any row under the 20-document floor it shares with `eval_findings`, decide each flag by the operating point's own alpha (never a legacy percentile against the conformal alpha), and name that operating point per row. | +| `eval_findings.py` | evidence | Score the register axis (thresholded and zero-hit), salience and collocation (the last on its document novel-pair fraction, because its reference is per sentence) on a **held-out** refereed corpus that fed no calibration bank, and report the same-genre in-sample population beside it so the leakage gap stays visible. Never present the salience rate as a false-positive rate — its gate is a percentile, so a non-zero rate is its design point — and never present either rate as precision for the advice itself. Added 2026-08-27: the axes' precision had stood at `unmeasured` pending hand labels, while a refereed paper's provenance was already a label for half of the question. | +| `eval_docscale.py` | evidence | Reproduce the EVALUATION §9 table through the same `manifold_operating_point` that findings use — never a private scoring path. Label the human rate **in-sample** wherever it includes the manifold's own train and calibration documents, and never present a single-seed tail-power figure as an estimate. Added 2026-08-26: nothing bound the evidence-producing path before, and §9 accumulated headline figures produced by a 95th-percentile cut rather than by the shipped conformal rule. Report `unmeasured` for any row under the 20-document floor it shares with `eval_findings`, decide each flag by the operating point's own alpha, and name that operating point per row. | | `verify_references.py` | QD (dimension F) | Verify every bibliography entry against the registries (CrossRef, DataCite for the prefixes CrossRef does not register, arXiv) and, with `--tex`, the assembled document's `\cite` keys against the entries; an unresolvable identifier or a cited key with no entry is an `integrity_blocker` and exit 1 (§0.1); a registry outage marks the entry `unmeasured` and the axis `degraded`, never clean; relevance is the reviewer's, not measured here. | | `label_findings.py` | evidence | Sample findings from every finding-emitting axis into a labelling sheet, re-serve a blind subset, and score precision, pooled recall and intra-rater agreement; any stratum under 20 labels is `unmeasured`; nothing here enters calibration or creates an operating point. | diff --git a/docs/architecture/evaluation/discourse-and-citation.md b/docs/architecture/evaluation/discourse-and-citation.md index fdcf9b8..d393293 100644 --- a/docs/architecture/evaluation/discourse-and-citation.md +++ b/docs/architecture/evaluation/discourse-and-citation.md @@ -29,7 +29,7 @@ boilerplate. ### 19.1 The two axes measure at different units, because one of them has to Hedging has **no paragraph-scale lower tail at all**. Calibrated per paragraph -over the 27,851-paragraph `wgl` bank, the tenth percentile is exactly 0.000 +over the 28,444-paragraph `wgl` bank, the tenth percentile is exactly 0.000 markers per 1,000 words in every one of the seven section buckets: more than a tenth of real human paragraphs contain no hedge, because a 40-word paragraph that hedges nowhere is entirely ordinary. A gate there is one no passage can @@ -38,16 +38,18 @@ fall below, and the axis would have reported a confident zero findings forever. the defect surfaced rather than shipping. Regrouped so that one section is one unit — every paragraph sharing a source -document and a bucket joined back together — six of the seven buckets separate: +document and a bucket joined back together, on the reference side and, since +v0.39.0, on the manuscript side as well, however many headings split the +bucket there — six of the seven buckets separate: | bucket | hedging p10, section unit (markers / 1,000 words) | |---|---:| -| discussion | 3.350 | -| results | 3.008 | -| method | 2.172 | -| intro | 1.986 | -| conclusion | 1.647 | -| data | 1.034 | +| discussion | 3.378 | +| results | 3.128 | +| method | 2.273 | +| intro | 2.066 | +| conclusion | 1.600 | +| data | 0.824 | | abstract | **0.000** — abstains | `abstract` stays flat because an abstract *is* one passage; regrouping cannot @@ -60,16 +62,18 @@ granularities are both valid and are not comparable: | artifact | unit | bucket sizes | |---|---|---| -| `cohesion_baseline.json` | paragraph | abstract 13,967 · method 6,886 · intro 3,228 · results 3,211 · data 2,989 · discussion 2,926 · conclusion 1,980 | -| `hedging_baseline.json` | section | abstract 10,404 · intro 502 · method 438 · conclusion 383 · discussion 327 · results 317 · data 299 | +| `cohesion_baseline.json` | paragraph | abstract 13,966 · method 7,138 · results 3,309 · intro 3,269 · data 3,069 · discussion 3,022 · conclusion 1,988 | +| `hedging_baseline.json` | section | abstract 10,400 · intro 503 · method 438 · conclusion 384 · discussion 329 · results 317 · data 303 | -Sizes and gates are the v0.36.3 profile rebuild (2026-09-04); the 2026-08-27 -build read method 6,903 · intro 3,252 · results 3,183 · data 2,992 · -discussion 2,932 · conclusion 1,975 at paragraph unit, and the `results` and -`data` hedging gates were 2.853 and 1.055. +Sizes and gates are the 2026-09-27 profile rebuild, read on the nearest-rank +quantile grid (audit C12); the CHANGELOG records the earlier builds. ### 19.2 Both floors were measured, not chosen +Both sweeps below were run on the v0.33.0 bank, when the floors were set, and +not again since; on the rebuilt bank every non-abstract bucket still resolves +at the 150-word floor (§19.1). + **Hedging, section word floor.** A rate per 1,000 words computed over too few words turns on the presence of one or two of them. Sweeping the floor against the bank: @@ -94,95 +98,87 @@ unchanged (0.676 at three sentences against 0.674 at four). ### 19.3 What separates, and what does not 203 held-out refereed papers (`fulltext-heldout`, disjoint from all calibration -banks) against the six `docval` generation regimes. Rank AUC, human over -machine; 0.5 is no separation. The **null** row is the same held-out set split -in half and scored against itself — the only number that says what a given AUC -is worth. +banks) against the six `docval` generation regimes, on the 2026-09-27 profile. +Rank AUC, human over machine; 0.5 is no separation. The **null** row is the +same held-out set split in half and scored against itself — the only number +that says what a given AUC is worth. A regime with fewer than five units in a +bucket is `—`, and the worst is taken over the rest. **Cohesion (paragraph unit)** | regime | intro | method | results | discussion | conclusion | data | |---|---:|---:|---:|---:|---:|---:| -| ai | 0.744 | 0.648 | 0.658 | — | — | — | -| ai_adversarial | 0.830 | 0.693 | 0.767 | 0.601 | — | 0.502 | -| ai_deai | 0.738 | 0.648 | 0.683 | 0.547 | 0.751 | 0.649 | -| ai_long | 0.676 | 0.545 | 0.614 | 0.623 | 0.631 | 0.516 | -| ai_natural | 0.684 | 0.624 | 0.679 | 0.559 | 0.580 | 0.533 | -| ai_skeleton | 0.711 | 0.562 | 0.573 | 0.667 | 0.707 | 0.538 | -| **worst of six** | **0.676** | 0.545 | 0.573 | 0.547 | 0.580 | 0.502 | -| null (human/human) | 0.515 | 0.487 | 0.505 | 0.496 | 0.510 | 0.505 | -| n human units | 960 | 1,128 | 593 | 869 | 631 | 868 | +| ai | 0.746 | 0.645 | 0.668 | — | — | — | +| ai_adversarial | 0.832 | 0.698 | 0.772 | — | — | 0.508 | +| ai_deai | 0.739 | 0.624 | 0.678 | 0.558 | — | 0.650 | +| ai_long | 0.677 | 0.552 | 0.546 | 0.609 | 0.627 | 0.519 | +| ai_natural | 0.686 | 0.599 | 0.671 | 0.573 | — | 0.538 | +| ai_skeleton | 0.713 | 0.569 | 0.585 | 0.744 | 0.716 | 0.544 | +| **worst of six** | **0.677** | 0.552 | 0.546 | 0.558 | 0.627 | 0.508 | +| null (human/human) | 0.489 | 0.520 | 0.485 | 0.465 | 0.443 | 0.531 | +| n human units | 1,040 | 2,038 | 1,326 | 1,412 | 631 | 1,391 | **Hedging (section unit)** | regime | intro | method | results | discussion | conclusion | data | |---|---:|---:|---:|---:|---:|---:| -| ai | 0.776 | 0.796 | 0.683 | — | — | — | -| ai_adversarial | 0.613 | 0.692 | 0.473 | 0.842 | 0.948 | 0.585 | -| ai_deai | 0.811 | 0.747 | 0.613 | 0.795 | 0.948 | 0.589 | -| ai_long | 0.682 | 0.603 | 0.607 | 0.568 | 0.591 | 0.459 | -| ai_natural | 0.816 | 0.761 | 0.622 | 0.850 | 0.948 | 0.544 | -| ai_skeleton | 0.750 | 0.613 | 0.490 | 0.526 | 0.376 | 0.609 | -| **worst of six** | **0.613** | 0.603 | 0.473 | 0.526 | **0.376** | 0.459 | -| null (human/human) | 0.460 | 0.574 | 0.520 | 0.475 | 0.469 | 0.508 | -| n human units | 190 | 310 | 120 | 202 | 155 | 248 | +| ai | 0.770 | 0.898 | 0.769 | — | — | — | +| ai_adversarial | 0.605 | 0.832 | 0.510 | — | — | 0.649 | +| ai_deai | 0.806 | 0.858 | 0.670 | — | — | — | +| ai_long | 0.671 | 0.731 | 0.551 | 0.593 | 0.601 | 0.464 | +| ai_natural | 0.811 | 0.866 | 0.675 | — | — | 0.609 | +| ai_skeleton | 0.744 | 0.788 | 0.578 | 0.421 | 0.358 | 0.641 | +| **worst of six** | **0.605** | **0.731** | 0.510 | 0.421 | 0.358 | 0.464 | +| null (human/human) | 0.487 | 0.442 | 0.440 | 0.472 | 0.520 | 0.489 | +| n human units | 198 | 156 | 120 | 158 | 154 | 139 | Read the worst-of-six row against the null row, not the best cell. Hedging in -`conclusion` looks impressive at 0.948 for three regimes and is **0.376** — below -chance, pointing the wrong way — for `ai_skeleton`. Hedging in `method` looks -respectable at 0.603–0.796 until its human-vs-human null is read: at 0.574, most -of that is not separation at all. Only `intro` clears the null for every regime -on both features. +`conclusion` reads 0.601 for `ai_long` and **0.358** — below chance, pointing +the wrong way — for `ai_skeleton`, the only two regimes that reach it. Hedging +clears its null for every regime in `intro` and in `method` (0.731–0.898 +against 0.442); cohesion does so in `intro`, and in `data` its worst regime sits +below its null. The v0.33.0 table, taken when a manuscript section was one +heading span, read hedging in `method` at 0.603–0.796 against a 0.574 null. ### 19.4 The transfer test agrees with the separation test, independently Held-out flag rate at the p10 gate: the design point is 10% by construction, so this measures whether the reference *transfers* to unseen refereed papers, not -whether those papers are defective. 203 papers: +whether those papers are defective. 203 papers, 2026-09-27 profile: | bucket | cohesion (paragraph) | hedging (section) | |---|---:|---:| -| intro | 8.33% | **7.89%** | -| method | 9.93% | 26.77% | -| results | 10.62% | 24.17% | -| discussion | 12.20% | 16.34% | -| conclusion | 10.14% | 15.48% | -| data | 14.63% | 22.98% | -| abstract | 6.58% | (abstains) | -| **all** | **10.87%** | 19.67% | - -Re-taken under v0.36.2, with subsections inheriting their parent bucket on the -manuscript side (§17.5): cohesion **10.81%** over 7,865 paragraphs (intro 7.91, -method 10.11, results 9.95, discussion 11.60, conclusion 10.03, data 14.61, -abstract 6.58) and hedging in `intro` 7.80% over 205 sections — the transfer -holds on the paragraphs the old bucketing dropped, where salience's did not -until its reference was calibrated on the numeral projection (§17.5). On the -v0.36.3 profile rebuild the same sweep reads cohesion **10.78%** (848 of -7,865; `data` 14.46, every other bucket unchanged to two decimals) and hedging -in `intro` 7.80% again. - -Cohesion transfers across every bucket — 10.87% against a 10% design point, a -quality of transfer `deai_salience` showed only on the paragraphs its old -bucketing measured (§17.5). Hedging transfers **only in `intro`**. Everywhere else it -fires at two to three times its nominal rate on prose a referee accepted, which -means the reference does not describe the held-out population there. +| intro | 8.17% | **9.09%** | +| method | 10.50% | **9.62%** | +| results | 10.33% | 14.17% | +| discussion | 11.97% | 8.23% | +| conclusion | 10.30% | 13.64% | +| data | 14.88% | 12.23% | +| abstract | 6.76% | (abstains) | +| **all** | **11.15%** (882 of 7,912) | 10.92% (101 of 925) | + +Cohesion transfers across every bucket, `data` highest. Hedging transfers in +`intro`, `method` and `discussion` and over-fires in the other three. Two independent measurements — one against machine text, one against unseen -human text — put the restriction in the same place. So hedging ships restricted -to `intro`, and `deai_discourse.AXES["hedging"]["buckets"]` carries the -restriction with this table beside it. A new field inherits it; widening it means -re-running this measurement, not editing the tuple. +human text — agree on `intro` and `method`; `discussion` transfers, but a +regime lands below chance there (§19.3). So hedging ships restricted to `intro` +and `method`, and `deai_discourse.AXES["hedging"]["buckets"]` carries the +restriction with this table beside it. A new field inherits it; widening it +means re-running this measurement, not editing the tuple — which is how +`method` joined on 2026-09-27, after the v0.33.0 measurement had put it at +26.77% and restricted the axis to `intro` alone. ### 19.5 What ships | axis | unit | live buckets | held-out rate at a 10% gate | worst-of-six AUC (null) | |---|---|---|---:|---| -| `L2.cohesion` | paragraph | all seven | 10.87% (557 of 5,125 units; 163 of 203 documents) | 0.676 in `intro` (0.515) | -| `L2.hedging` | section | `intro` only | 7.89% (15 of 190 units; 15 of 203 documents) | 0.613 in `intro` (0.460) | +| `L2.cohesion` | paragraph | all seven | 11.15% (882 of 7,912 units; 170 of 203 documents) | 0.677 in `intro` (0.489) | +| `L2.hedging` | section | `intro`, `method` | 9.32% (33 of 354 units; 31 of 203 documents) | 0.605 in `intro` (0.487); 0.731 in `method` (0.442) | -On `wgl-letter`, hedging reports `degraded`: no bucket clears the 30-unit floor -after the restriction. That is the correct answer for a 36-document profile and -is reported rather than papered over. +On `wgl-letter` both hedging buckets reach the 30-unit floor on the rebuilt +profile (`intro` 37 sections, `method` exactly 30), so the axis reports +`measured` there under the restriction measured on `wgl`. **What this evidence does not license.** It says the reference distributions are real and section-bound. It does not say a low value means a machine wrote the diff --git a/docs/architecture/evaluation/document-scale.md b/docs/architecture/evaluation/document-scale.md index ed1d489..a0c91f6 100644 --- a/docs/architecture/evaluation/document-scale.md +++ b/docs/architecture/evaluation/document-scale.md @@ -14,141 +14,73 @@ nothing here can redefine it. All machine-readable findings use the ## 9. Whole-document cross-paragraph dispersion (the keystone axis) -> ### ⚠️ Re-measured twice — 2026-08-25 +> ### Re-measured — 2026-09-27 (v0.39.0) > -> **Second pass (v0.28.0).** The corpus-layer rebuild changed section splitting, -> and `document_shape` is built on sections, so §9 could not carry its figures -> forward on assertion. Every tier was re-scored through the shipped path -> against the rebuilt baseline, with the **length-fair protocol restored** -> (each document compared only against human papers in its own length -> stratum). The result is a per-value reproduction, not a matching total: +> The v0.39.0 profile rebuild recalibrated the baseline over **504** complete +> papers — length-stratum edges 49 and 78 paragraphs, the manifold trained on 302 +> and calibrated on 202, the role axis calibrated on all 504 — and every tier was +> re-scored through the shipped path. Length-fair AUC compares each AI document +> only with the human papers in its own length stratum (197 in the short one); the +> flag columns are the conformal operating point at α = 0.05: > -> | length-fair AUC | recorded | re-measured | Δ | -> |---|---:|---:|---:| -> | manifold, natural | 0.928 | 0.933 | +0.005 | -> | manifold, de-AI'd | 0.939 | 0.943 | +0.004 | -> | manifold, adversarial | 0.919 | 0.927 | +0.008 | -> | manifold, skeleton | 0.916 | 0.914 | -0.002 | -> | role, natural | 0.702 | 0.690 | −0.012 | -> | role, de-AI'd | 0.751 | 0.742 | −0.009 | -> | role, adversarial | 0.818 | 0.810 | −0.008 | -> | role, skeleton | 0.507 | 0.507 | 0.000 | +> | tier (documents) | manifold AUC | role AUC | manifold flag | role flag | union | +> |---|---:|---:|---:|---:|---:| +> | natural (28) | 0.893 | 0.687 | 0.071 | 0.036 | 0.107 | +> | de-AI'd (15) | unmeasured | unmeasured | unmeasured | unmeasured | unmeasured | +> | adversarial (38) | 0.953 | 0.803 | 0.184 | 0.316 | 0.447 | +> | skeleton (24) | 0.866 | 0.526 | 0.250 | 0.083 | 0.292 | +> | long-form (29) | 0.840 | 0.610 | 0.000 | 0.069 | 0.069 | +> | human, all 504 (in-sample) | — | — | 0.020 | 0.042 | 0.062 | > -> All eight within 0.012. Human false-flag at the shipped operating point, -> scored directly over 492 papers: **manifold 0.0325, role 0.0427**, union -> 0.0752, against nominal α = 0.05. +> The de-AI'd tier holds 15 documents, under the 20-document floor of +> `tools/eval_docscale.py`, so it has no rate. The human row is 10, 21 and 31 of +> 504 papers. No paper is flagged by both axes (independence predicts 0.42), and +> the pre-conformal 5% tails stay disjoint (the two 26-paper tails share none; +> independence predicts 1.34). +> `python tools/eval_docscale.py --field wgl` reproduces the manifold flag column +> and gives the pooled-AUC view: natural 0.867, adversarial 0.929, skeleton +> 0.855, long-form 0.832. > -> **A second correction landed in the same release, and it moved the axis.** -> `_paper_documents` assembled each bundle by sorted FILENAME, so -> `Conclusion.tex` preceded `Introduction.tex` — and this axis measures section -> arc. 122 of the 500 bundles hold more than one `.tex`, 12 provably out of -> order, and all 122 also folded appendices, acknowledgements and the journal's -> own class documentation in as body prose. Rebuilding in include order raises -> manifold tail power on natural AI **0.214 → 0.250** and on adversarial -> **0.158 → 0.184** while *lowering* the human false-flag rate 0.0346 → 0.0325. -> The role axis trades the other way (natural 0.107 → 0.036, de-AI'd 0.333 → -> 0.267), so the union moves 0.321 → 0.286 on natural and 0.421 → 0.447 on -> adversarial. Current tail powers: manifold 0.250 / 0.400 / 0.184 / 0.292; -> role 0.036 / 0.267 / 0.316 / 0.042; union 0.286 / 0.600 / 0.447 / 0.333. +> **What moved since the 2026-08-25 table, and why it is not attributed.** The +> manifold's length-fair AUCs moved by −0.05 to +0.10 (natural 0.933 → 0.893, +> adversarial 0.927 → 0.953, skeleton 0.914 → 0.866, long-form 0.740 → 0.840). +> Four changes landed between the two measurements and were not run one at a +> time: the v0.36.3 corpus rebuild; dropping appendices, reference lists and the +> other `skip` sections from every measured document; blanking headings and +> floats before paragraphs are cut (audit B22/B23); and fitting the length-stratum +> edges on the training split alone (audit B17). The role axis needs no fit: +> its flag counts reproduce on natural, adversarial and long-form, skeleton moves +> from 1 to 2 of 24, and its AUCs move by at most 0.024 — the paragraph cut, the +> strata and the human reference changed, not a fit. The manifold flag rates are single seed +> draws whose per-seed spread is 0.04–0.18 (§9.4c), so none of their changes is +> read as a result. > -> **Read every manifold tail-power figure in this paragraph as a single seed -> draw (§9.4c, 2026-08-26).** The per-seed spread on those tiers is 0.04–0.18, -> so `0.214 → 0.250` and `0.158 → 0.184` are both well inside it. The -> *human-side* improvement and the role axis's exact reproduction are the -> durable parts of this correction; the manifold deltas are not resolvable -> against seed noise, and the ordering fix should be justified on the grounds -> that reading a paper in its own order is correct, not on those two deltas. -> -> The 42× *paragraph-bank* growth did not move document-scale discrimination — -> the expected result, since §9 always calibrated against `fulltext-arxiv`. -> What moved it was reading those same papers in the right order. A fifth tier -> is added below: `ai_long` scores manifold AUC 0.740 with **0.000** tail -> power, the standing falsification target. -> -> **First pass.** The 2026-08-17 rebuild regenerated `docstructure_baseline.json`, -> but §9 was **not** on the v0.27.1 re-measurement list (that release re-measured -> §5, §6, §7 and §14, and checked §15). Its figures therefore still described the -> **507**-paper reference while the shipped artifact carried **493**. The notice -> below records the re-measurement that closed that gap. -> -> **The control that makes the rest credible.** The role-coupling axis needs no -> fit, so a refit cannot move it — and it reproduces the pre-rebuild record -> exactly: conformal flag rates 0.107 / 0.333 / 0.316 / 0.042 across the four -> tiers, identical to the published table, and length-fair AUC within 0.001 of -> every published value. Same scoring code, same tier documents, same strata. -> Whatever *did* move is the refit, not the pipeline. -> -> **What moved: the manifold improved, and not by trading away false-flag -> control.** Re-measured on the 493-paper baseline against the same AI tiers: -> -> | quantity | 507-paper record | 493-paper rebuild | -> |---|---:|---:| -> | conformal flag rate, natural | 0.071 | **0.214** | -> | conformal flag rate, de-AI'd | 0.133 | **0.400** | -> | conformal flag rate, adversarial | 0.026 | **0.158** | -> | conformal flag rate, skeleton | 0.125 | **0.292** | -> | length-fair AUC, natural | 0.873 | **0.928** | -> | length-fair AUC, de-AI'd | 0.901 | **0.939** | -> | length-fair AUC, adversarial | 0.836 | **0.919** | -> | length-fair AUC, skeleton | 0.822 | **0.916** | -> -> Human false-flag control held. Scoring all 493 reference papers through the -> shipped operating point directly: **manifold 16/493 = 0.0325**, **role -> 21/493 = 0.0426**, both under the nominal α = 0.05 (union 37/493 = 0.0751). -> Leave-one-out over the conformal calibration set alone agrees — manifold -> 0.0404, role 0.0487. The extra tail power is not borrowed from the human side. -> -> **The rest of §9, re-measured against the same corpus.** The human corpus is -> user-supplied and gitignored — CI and a fresh clone cannot see it — but it is -> present on any machine that holds the profile, at -> `style-corpus//fulltext-arxiv/`, keyed by the same arXiv IDs the -> baseline's `documents` list carries. Every remaining human-side figure was -> therefore re-measured rather than left at its 507-paper vintage: -> -> | quantity | 507-paper record | 493-paper rebuild | -> |---|---:|---:| -> | §9.1 disjointness, pre-conformal 5% tails | 0 of 507 (independence ~1.3) | **0 of 493** (independence ~1.27) | -> | §9.1 disjointness, at the conformal operating point | — | **0** (independence ~0.68) | -> | §9.3 paired skeleton-vs-its-own-twin AUC | 0.934 (twins held out of the fit) | **0.958** (shipped scorer, twins in reference — *not* the same protocol) | -> | §9.4 r(manifold distance, paragraph count) | 0.353 | **−0.080** | -> -> Two of those deserve a sentence. The **disjointness result survives the -> rebuild**: no human paper is flagged by both axes, on either the pre-conformal -> tails or the shipped operating point, so the axes remain complementary rather -> than redundant. And the **length confound has essentially gone** on the -> shipped path — 0.353 was measured against the *pooled* manifold, whereas the -> shipped scorer now sends each document to its own length-stratum manifold, and -> against that the correlation is −0.080 with median distances of 2.481 / 2.099 / -> 2.186 across the three strata. The Mondrian stratification of §9.5 is doing -> the job it was introduced to do. -> -> **Still not re-measured:** the document-level surprisal sweep of §9.8, which -> needs GPT-2 over all 493 papers (hours of compute, and its conclusion — the -> surprisal path is weaker than the model-free manifold and adds nothing to it — -> is only reinforced by a rebuild that raised the model-free numbers and left -> the surprisal path untouched). Bootstrap CIs were not recomputed for any -> post-rebuild point estimate; the pre-rebuild intervals belong to the -> pre-rebuild points and are not transferred. -> -> Reproduce: score `style-profile//docval/ai_*` and the corpus papers -> through `manifold_operating_point` and `document_role_coupling`, using the -> baseline's own per-stratum calibration as the human reference. +> **Everything below this notice is the 2026-08-25/26 record** on the 493-paper +> baseline (edges 46 and 76, manifold split 295 / 198), kept for its method and +> reasoning; where a figure below differs from the table above, the table is +> current. That record also carried a bundle-assembly correction: +> `_paper_documents` had joined each bundle's files in sorted filename order, so +> `Conclusion.tex` could precede `Introduction.tex` — 122 of the 500 bundles hold +> more than one `.tex`, 12 provably out of order, and all 122 folded appendices, +> acknowledgements and the journal's class documentation in as body prose. +> Reading a paper in include order is correct on its face; the tail-power changes +> it produced are inside the seed spread and are not the evidence for it. The architecture reflection ([`DEAI_ARCHITECTURE_ROADMAP.md`](../../design-notes/DEAI_ARCHITECTURE_ROADMAP.md)) identified the document scale as the confound-orthogonal signal: field register shifts the *level* of per-paragraph features, while AI-uniformity compresses their *spread* across a -document, which no per-paragraph score can see (the 32–41% field-topic FPR of §7.2). +document, which no per-paragraph score can see (the 29–42% field-topic false-positive rate of §7.2). [`tools/deai_docstructure.py`](../../../tools/deai_docstructure.py) measures, per model-free per-paragraph feature, the cross-paragraph dispersion of a complete document (`deai_features.cross_paragraph_dispersion`), and flags a document that varies a feature below the human low tail. The axis was first calibrated one-observation-per-paper over 14 complete human `wgl` papers (multi-file papers concatenated to avoid pseudoreplication); -the current reference is **493** complete papers, and `L2.document_structure` is -`measured`. Per-stratum leave-one-paper-out false-flag rates on the percentile-threshold -path are 0.091 / 0.077 / 0.087, above the nominal 0.05 — which is why the shipped -operating points are the split-conformal ones of §9.5, whose leave-one-out human rate is -0.040. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. +the current reference is **504** complete papers, and `L2.document_structure` is +`measured`. On the 493-paper reference the in-sample percentile thresholds flagged +0.091 / 0.077 / 0.087 of human papers per stratum under leave-one-paper-out, above the +nominal 0.05 — which is why the manifold scores only at the split-conformal operating +points of §9.5, whose leave-one-out human rate was 0.040. The `docstructure_baseline.json` artifact is gitignored and rebuilt per field. ### 9.1 Validation against AI documents, including an adversarial de-AI tier @@ -191,10 +123,10 @@ two stages: adversarial contrast to AUC 0.801 (CI 0.733–0.867)** while keeping natural at 0.894 and de-AI'd at 0.932 — on the same held-out protocol. -The shipped detector implements the band view at three levels, calibrated over **493 +The shipped detector implements the band view at three levels, calibrated over **504 complete human papers** (the bulk full-text fetch grew the reference 14 → 507 within two -days; the 2026-08-17 rebuild re-ingested it at 493 — see the notice at the top of §9, -which also carries the post-rebuild operating characteristics): +days; rebuilds re-ingested it at 493 on 2026-08-17 and at 504 on 2026-09-27 — the notice +at the top of §9 carries the current operating characteristics): 1. **Joint manifold statistic (primary).** The per-document vector of log dispersion ratios is scored by Mahalanobis distance against the human center and covariance @@ -307,8 +239,8 @@ enforced only at the section/paragraph level. > median distances 2.481 / 2.099 / 2.186 across the three strata. In other words the > confound this section identifies is real, and the Mondrian stratification introduced in > §9.5 in response to it has largely removed it from the shipped path. The role axis's -> operating characteristics were re-measured and are unchanged to the digit (see the §9 -> notice), so the mechanism described here stands. +> operating characteristics were re-measured and are unchanged to the digit (§9.5), so +> the mechanism described here stands. The adversarial arc in 9.1 proved dispersion *magnitude* is gameable: forced variety lands inside (or past) the human band. What forced variety cannot do without actually @@ -390,8 +322,10 @@ Conformal p = (1 + #{calibration ≥ score}) / (n_cal + 1), flag at p ≤ α = 0 giving P(false flag) ≤ α finite-sample and distribution-free for exchangeable human papers. Calibration is Mondrian-stratified by document-length terciles (edges 46 and 76 paragraphs) — length is the measured confound (9.4); a thin stratum falls back to -the pooled calibration set, keeping marginal validity. Legacy baselines without the -conformal block fall back to the percentile thresholds. +the pooled calibration set, keeping marginal validity. A baseline whose manifold has no +conformal block leaves the manifold rule unscored and the axis `degraded` with a rebuild +reason (audit B24); without one, the role axis falls back to its in-sample percentile +threshold. Stratification exposed a second, larger instance of the same confound: **short human papers score systematically higher manifold distances** (stratum-0 95th percentile @@ -408,7 +342,8 @@ operating point (α = 0.05) the measured rates are: | role-decoupling | 0.107 | 0.333 | 0.316 | 0.042 | | union | **0.321** | **0.667** | **0.421** | **0.333** | -Post-rebuild (2026-08-25, n = 28 / 15 / 38 / 24 measurable tier documents). The +The first 2026-08-25 pass, before the bundle-assembly correction (n = 28 / 15 / 38 / 24 +measurable tier documents). The pre-rebuild record on the 507-paper reference read manifold 0.071 / 0.133 / 0.026 / 0.125 and union 0.179 / 0.467 / 0.342 / 0.167; the role row is unchanged to the digit, which is the control — the role z needs no fit, so a refit cannot move it, and its exact @@ -443,7 +378,8 @@ conformal calibration set only, which *is* out-of-fit, gives **manifold 0.0404 p manifold's higher tail power is not paid for on the human side. The discrimination signal itself survives length matching. Length-fair AUC against the -**169** same-stratum (short) humans, post-rebuild: +**169** same-stratum (short) humans, post-rebuild (the 2026-09-27 figures are in the +notice at the top of §9): | length-fair AUC | natural | de-AI'd | adversarial | skeleton | long | |---|---|---|---|---|---| @@ -572,8 +508,9 @@ rather than quoted. long-form tail power is 0.003 ± 0.010 at worst and 0.000 in most configurations. That is no longer a single-configuration result: it is stable across every variation tried, so the axis does not detect long-form generation at α = 0.05 and no tuning of the -calibration or the covariance changes that. The tier's rank AUC is 0.729, so the signal -is present and the operating point cannot reach it. +calibration or the covariance changes that. The tier's mixed-length rank AUC was 0.729 on +the 507-paper reference (§9.7), and its length-fair AUC reads 0.840 on 2026-09-27, so the +signal is present and the operating point cannot reach it. ### 9.6 Claim anchoring: hypothesis refuted for strong-model generations @@ -707,6 +644,6 @@ a guess. dispersion reference: for each shape feature it places the draft's within-document dispersion in the distribution of the author's own papers. Because that reference is same-author, same-field, same-jargon, it sidesteps the field-topic confound -behind the classifier's 32–41% false-positive rate entirely — the comparison is +behind the classifier's 29–42% false-positive rate entirely — the comparison is you-versus-you. It flags a draft that varies paragraph shape far less than the author usually does, and is `unmeasured` below three prior papers. diff --git a/docs/architecture/evaluation/held-out-labels.md b/docs/architecture/evaluation/held-out-labels.md index 53f8550..f658d30 100644 --- a/docs/architecture/evaluation/held-out-labels.md +++ b/docs/architecture/evaluation/held-out-labels.md @@ -18,14 +18,14 @@ thing: which papers are held out, and what it costs when one quietly is not. --- -## 17. Held-out refereed papers as labels: register and salience measured (v0.30.0, corrected v0.31.0, superseded by §18) +## 17. Held-out refereed papers as labels: register and salience measured (v0.30.0; every re-measurement since is tabled in §17.4) -> **Read §18 for the current register numbers.** Every register figure below was -> measured at `MIN_MANUSCRIPT_USES = 5`; v0.32.0 raised it to 15, which cut the -> held-out findings from 887 to 198 and the rate from 0.3842 to 0.0858 per 1,000 -> words. §18's table carries the pairs. The section is kept because the reasoning -> — why provenance is a label, why held-out had to be enforced, what the paired -> leakage measurement controls for — is unchanged by the threshold. +> **The current register numbers are the last row of the table closing §17.4**, +> which carries every re-measurement of this population. The figures above that +> table were measured at `MIN_MANUSCRIPT_USES = 5`, the setting v0.31.0 shipped. +> The section is kept because the reasoning — why provenance is a label, why +> held-out had to be enforced, what the paired leakage measurement controls for +> — is unchanged by the threshold. Both axes had stood at "precision `unmeasured`, pending hand labels" since v0.26.0. Half of that gap never needed a labeller. A refereed ApJ/ApJL/A&A @@ -128,27 +128,27 @@ rate. That open item was closed in [§18.4](projection-and-operating-point.md) by refuting its premise: rank AUC stays below 0.5 at every setting, so no operating point makes this a detector. -> **⚠️ Superseded (v0.32.0).** Every register figure in §17.2–17.4 was -> re-measured in [§18.4](projection-and-operating-point.md): **0.0858** per -> 1,000 words on **44.83%** of documents, AUC **0.2856**, leakage **94.44% of -> 198**. Two changes, both in that release — the citation projection stopped -> leaking bibliography keys into prose (§18.1), and `MIN_MANUSCRIPT_USES` moved -> 5 → 15 once the sweep showed rank AUC below 0.5 at every setting. The salience -> figures below also moved slightly, because removing 7.00% of the digits the -> numeral projection carried (§18.2) shifted both the reference and the measured -> side. Read §18 for current values; this section is kept for its method and its -> history. -> -> **⚠️ Correction (2026-08-27, v0.31.0).** The figures above replace those -> v0.30.0 published — **0.991** per 1,000 words on **93.6%** of documents, AUC -> 0.080, leakage 72.7% of 2,287. Those were measured with a projection defect: -> `manuscript_terms` read the whole raw file while the corpus df was built from -> body prose only, so front matter and bibliographies were compared against a -> reference that had excluded them. **58.7%** of the findings came from outside -> body prose (27.5% preamble, 26.3% bibliography, 4.1% TeX control words, 0.9% -> `skip` sections). No threshold changed; the projection did. Salience was -> unaffected and reproduces byte-for-byte — 0 of its 1,077 findings on these -> papers fall in a bibliography. +**How the held-out rate moved.** This table is the one account of the chain; +every other place that quotes the rate gives its last row and points here. One +population throughout, the 203 papers of §17.1, and one statistic. Every step is +a projection fix or the use floor, never a change of population. + +| release | what changed | findings | per 1,000 words | documents flagged | rank AUC vs machine | own-membership suppression | measured in | +|---|---|---:|---:|---:|---:|---:|---| +| v0.30.0 | first measurement: the manuscript side read the whole raw file, the corpus side body prose only | — | 0.991 | 93.6% | 0.080 | 72.7% of 2,287 | — | +| v0.31.0 | one projection on both sides: 58.7% of the v0.30.0 findings came from outside body prose (27.5% preamble, 26.3% bibliography, 4.1% TeX control words, 0.9% `skip` sections) | 887 | 0.384 | 87.2% | 0.148 | 86.3% of 887 | §17.2–17.4 | +| v0.32.0 | citation keys no longer leak into prose (§18.1); use floor 5 → 15 (§18.4) | 198 | 0.0858 | 44.83% | 0.2856 | 94.44% of 198 | §18.4 | +| v0.36.0 | headings out of the manuscript's body projection | 81 (196 before, same process) | 0.0351 | 30.0% | 0.352 | 95.1% of 81 | §23.1 | +| v0.36.1 | floats blanked on the manuscript side as on the corpus side | 57 | 0.0247 | 22.2% | 0.392 | 98.2% of 57 | §23.1 | +| v0.36.2 | per body prose word instead of per raw-source word | 57 | 0.0371 | 22.2% | 0.391 | — | §23.1 | +| **v0.39.0** (2026-09-27) | profile rebuilt on the audit's corpus-side fixes; own membership counted as the bank holds a paper (audit E17) | **56** | **0.0364** | **22.17%** | **0.391** | **87.5% of 56** | this table | + +No threshold changed at v0.31.0; the projection did. Salience was unaffected and +reproduced byte-for-byte — 0 of its 1,077 findings on these papers fell in a +bibliography. The own-membership column before v0.39.0 matched terms by +substring over blank-line blocks of the raw source, preamble and bibliography +included, which overstated suppression; from v0.39.0 the paper's own passages +are the ones the bank would admit, indexed as the bank indexes them. **Guarded since v0.30.1.** §15.5 derived that a bank of n passages cannot express a rate below 1/n, so under 10,000 passages this gate collapses to "df == 0". That @@ -217,6 +217,12 @@ papers whose rows the reference contains — with per-bucket rates method 0.233, abstract 0.342 (26 of 76). Rank AUC against machine text on body-word density rises to **0.663**. Nothing about the salience gate is open. +**Re-taken on the 2026-09-27 rebuild**, on the nearest-rank quantile grid +(audit C12): **1,975 findings over 9,840 scorable paragraphs = 0.2007** per +passage, and rank AUC **0.666** on body-word density. This paragraph and the +one above are the record of the seam; everything else that quotes the rate +gives this value and points here. + ### 17.6 Limits The register figure is an upper bound on advice-quality false positives, not a @@ -227,7 +233,9 @@ one journal family, 2012–2018, and the fetch stopped at the `--max-papers 200` cap rather than at exhaustion, so 203 is not a power calculation. `machine:ai` tiers are short (median 1,266–1,788 words) against 9,793, so every machine-vs-human AUC inherits the length asymmetry of §9. The paired leakage -test adds only the paper's own passages, so 86.3% is a lower bound. +test adds only the paper's own passages, so it bounds leakage from below; +before v0.39.0 it also matched terms by substring over raw-source blocks, which +pushed the other way (audit E17). --- @@ -333,7 +341,10 @@ same way §17.5 does once subsections inherit their parent bucket (v0.36.2): and back the same way once the reference is calibrated on the numeral projection (v0.36.3): **0.1943**, 197 findings over the same 1,014, with salience's rank AUC 0.567 and register's **0.412** (5 findings) on the -rebuilt profile, register still below 0.5. +rebuilt profile, register still below 0.5. On the 2026-09-27 rebuild the same +22 papers read salience **0.1933** (196 findings over the same 1,014) at rank +AUC 0.569, and register **0.411** (5 findings, all 5 suppressed by the paper's +own membership under the corrected estimator), still below 0.5. **What this does not establish.** One field, one advisor, 22 papers. The population is small enough that one paper is 4.5 percentage points of the diff --git a/docs/architecture/evaluation/learned-model.md b/docs/architecture/evaluation/learned-model.md index 93dc515..985eb15 100644 --- a/docs/architecture/evaluation/learned-model.md +++ b/docs/architecture/evaluation/learned-model.md @@ -16,44 +16,47 @@ nothing here can redefine it. All machine-readable findings use the The current `style-profile/wgl/voice_model.joblib` bundle -was **retrained locally on 2026-09-05** (RTX 4060 Ti) after the v0.36.3 profile -rebuild took the curated-field bank to 42,371 positive records, and re-evaluated with -the same confound-aware audit. It supersedes the 2026-08-26 retrain, which superseded -2026-08-25, 2026-08-17 and the 2026-07-12 cloud run. The full machine-readable audit is -`style-profile/wgl/voice_model_evaluation.json` -(schema `sci-paper.voice-model-evaluation.v1`, `generated_utc` `2026-09-05T06:41:31Z`). - -| Metadata | 2026-09-05 | 2026-08-26 | 2026-08-25 | 2026-08-17 | -|---|---:|---:|---:|---:| -| classifier | logistic regression | logistic regression | logistic regression | logistic regression | -| positive-class records | 42,371 | 42,311 | 39,376 | 15,034 | -| negative-class records | 2,265 | 2,265 | 2,265 | 2,265 | -| total records | **44,636** | 44,576 | 41,641 | 17,299 | -| grouped-split AUC (20 splits) | **0.9487** | 0.9518 | 0.9502 | 0.9320 | -| grouped-split F1 (positive class) | 0.9324 | 0.9345 | 0.9394 | 0.9172 | -| grouped-split balanced accuracy | 0.8704 | 0.8761 | 0.8724 | 0.8445 | -| feature count | 14 | 14 | 14 | 14 | -| operating point in bundle | absent | absent | absent | absent | -| `measurement_status` | degraded | degraded | degraded | degraded | +was **retrained locally on 2026-09-27** (RTX 4060 Ti) after the v0.39.0 profile +rebuild took the curated-field bank to 42,888 positive records, and re-evaluated with +the same confound-aware audit. It supersedes the 2026-09-05 retrain, which superseded +2026-08-26, 2026-08-25, 2026-08-17 and the 2026-07-12 cloud run. The full +machine-readable audit is `style-profile/wgl/voice_model_evaluation.json` +(schema `sci-paper.voice-model-evaluation.v1`, `generated_utc` `2026-09-27T09:55:51Z`). + +| Metadata | 2026-09-27 | 2026-09-05 | 2026-08-26 | 2026-08-25 | 2026-08-17 | +|---|---:|---:|---:|---:|---:| +| classifier | logistic regression | logistic regression | logistic regression | logistic regression | logistic regression | +| positive-class records | 42,888 | 42,371 | 42,311 | 39,376 | 15,034 | +| negative-class records | 2,265 | 2,265 | 2,265 | 2,265 | 2,265 | +| total records | **45,153** | 44,636 | 44,576 | 41,641 | 17,299 | +| grouped-split AUC (20 splits) | **0.9487** | 0.9487 | 0.9518 | 0.9502 | 0.9320 | +| grouped-split F1 (positive class) | 0.9323 | 0.9324 | 0.9345 | 0.9394 | 0.9172 | +| grouped-split balanced accuracy | 0.8705 | 0.8704 | 0.8761 | 0.8724 | 0.8445 | +| feature count | 14 | 14 | 14 | 14 | 14 | +| operating point in bundle | absent | absent | absent | absent | absent | +| `measurement_status` | degraded | degraded | degraded | degraded | degraded | ### 7.0a The field-topic confound is a property of the feature set, decided The roadmap carried "a field-topic-robust L3 operating point, **or a recorded decision -that one is not obtainable from this feature set**". Four retrains on banks differing -by 2.6×, the last on the rebuilt v0.36.3 profile, answer it, each with its own 20-split -grouped audit: - -| negative control | 2026-09-05 (44,636) | 2026-08-26 (44,576) | 2026-08-25 (41,641) | 2026-08-17 (17,299) | -|---|---:|---:|---:|---:| -| public-generic generated | **0.055** (0.026–0.076) | 0.052 | 0.053 | 0.086 | -| field-topic generated | **0.295** (0.230–0.356) | 0.280 | 0.285 | 0.318 | -| field-jargon-dense generated | **0.421** (0.236–0.543) | 0.393 | 0.410 | 0.417 | +that one is not obtainable from this feature set**". Five retrains on banks differing +by 2.6×, the last two on the rebuilt v0.36.3 and v0.39.0 profiles, answer it, each with +its own 20-split grouped audit. This is the one table of the negative-control +false-positive rates: the fraction of generated negatives the model scores as +curated-field-like, mean over the 20 splits. + +| negative control | 2026-09-27 (45,153) | 2026-09-05 (44,636) | 2026-08-26 (44,576) | 2026-08-25 (41,641) | 2026-08-17 (17,299) | +|---|---:|---:|---:|---:|---:| +| public-generic generated | **0.057** (0.026–0.076) | 0.055 | 0.052 | 0.053 | 0.086 | +| field-topic generated | **0.292** (0.230–0.349) | 0.295 | 0.280 | 0.285 | 0.318 | +| field-jargon-dense generated | **0.418** (0.212–0.528) | 0.421 | 0.393 | 0.410 | 0.417 | Ranges are 2.5–97.5 percentiles over the 20 splits. **The decision is recorded: not -obtainable from this feature set.** Every movement across a 2.6× bank increase and a -profile rebuild lies inside a single retrain's own split range — field-topic 0.318 → -0.285 → 0.280 → 0.295 against a range of 0.230–0.356 — while the *headline* AUC first -rose and then eased, 0.9320 → 0.9518 → 0.9487, itself inside 0.9409–0.9583. More data +obtainable from this feature set.** Every movement across a 2.6× bank increase and two +profile rebuilds lies inside a single retrain's own split range — field-topic 0.318 → +0.285 → 0.280 → 0.295 → 0.292 against a range of 0.230–0.349 — while the *headline* +AUC rose, eased and held, 0.9320 → 0.9518 → 0.9487 → 0.9487, itself inside +0.9392–0.9587. More data buys separation on the easy contrast and buys nothing on the one that matters, which is what a feature-set confound looks like rather than a sampling limit. The model partly measures field register, and field-topic AI prose is precisely the distribution on which field register is uninformative. @@ -67,29 +70,30 @@ verdict. Reopening this requires a *different feature set*, not a larger bank. The retrain is checked at the unit the bundle is actually used on. `voice_findings` scores every paragraph of ≥ 30 words and, because the bundle is degraded, surfaces the three lowest-ranked ones; feeding it a whole document is out of distribution and says -nothing about product behaviour. The 2026-08-26 and 2026-09-05 bundles therefore -scored the **same 1,808 paragraphs** from 63 documents (every third of the `docval` -tiers and the curated tiers), against the same feature pipeline, so only the -classifier differs. The 2026-08-26 column is the same check between that bundle and -its 2026-08-25 predecessor (1,845 paragraphs, 54 documents). - -| Quantity | 2026-09-05 | 2026-08-26 | -|---|---|---| -| feature schema, feature names, classifier, `measurement_status` | unchanged | unchanged | -| operating point | absent in both — no threshold was introduced | absent | -| per-paragraph score change | median 0.007, p90 0.044, max 0.168 | median 0.034, p90 0.268, max 0.776 | -| within-document rank correlation (Spearman ρ) | median **0.991**, p10 0.972 | median 0.846, p10 0.698 | -| triage paragraphs unchanged, mean overlap of the 3 surfaced | **0.889** | 0.654 | -| documents whose 3 surfaced paragraphs are identical | 42/63 = 0.667 | 11/54 = 0.204 | +nothing about product behaviour. Each column compares a bundle with its predecessor +on the same paragraphs (every third document of the `docval` tiers and the curated +tiers), featurized once through the same pipeline, so only the classifier differs: +2,080 paragraphs from 69 documents for 2026-09-27, 1,808 from 63 for 2026-09-05, and +1,845 from 54 for 2026-08-26. + +| Quantity | 2026-09-27 | 2026-09-05 | 2026-08-26 | +|---|---|---|---| +| feature schema, feature names, classifier, `measurement_status` | unchanged | unchanged | unchanged | +| operating point | absent in both — no threshold was introduced | absent | absent | +| per-paragraph score change | median 0.0007, p90 0.004, max 0.022 | median 0.007, p90 0.044, max 0.168 | median 0.034, p90 0.268, max 0.776 | +| within-document rank correlation (Spearman ρ) | median **1.000**, p10 0.999 | median 0.991, p10 0.972 | median 0.846, p10 0.698 | +| triage paragraphs unchanged, mean overlap of the 3 surfaced | **0.995** | 0.889 | 0.654 | +| documents whose 3 surfaced paragraphs are identical | 68/69 = 0.986 | 42/63 = 0.667 | 11/54 = 0.204 | **Exact behavioural equivalence does not hold, and it was not available to hold**: a refitted logistic regression is a different function, and the 2026-08-26 retrain, on a -bank grown 2.6×, moved the marginal third of every triage list. The 2026-09-05 retrain -on the rebuilt profile (60 more positive rows) is close to equivalent at the unit that -matters: the ranking the degraded mode consumes is preserved at ρ = 0.991, two thirds -of the documents surface the identical three paragraphs, and the per-paragraph score -moves by 0.007 at the median. That is reported rather than assumed: a reviewer who -reran an old triage list will get the same list back on most documents, not all. +bank grown 2.6×, moved the marginal third of every triage list. The two retrains on +rebuilt profiles are close to equivalent at the unit that matters. With 60 more +positive rows, 2026-09-05 kept the ranking the degraded mode consumes at ρ = 0.991 and +two thirds of the documents surfaced the identical three paragraphs; with 517 more, +2026-09-27 keeps it at ρ = 1.000 and 68 of 69 documents surface the identical three. +That is reported rather than assumed: a reviewer who reran an old triage list will get +the same list back on most documents, not all. The labels represent curated field prose versus generated negative examples. The resulting probability is exposed as `field_similarity`, not a probability that a human @@ -102,45 +106,35 @@ recomputes `corpus_cos` against a training-only curated centroid so held-out pap cannot inflate their own similarity feature. Intervals summarize split-to-split variation; they are not independent-sample confidence intervals. -| Metric | mean | 2.5% | 97.5% | 2026-08-26 mean | +| Metric | mean | 2.5% | 97.5% | 2026-09-05 mean | |---|---:|---:|---:|---:| -| overall AUC (raw UID) | **0.9487** | 0.9409 | 0.9583 | 0.9518 | -| overall balanced accuracy | **0.8704** | 0.8588 | 0.8881 | 0.8761 | -| matched-stratum AUC (section × length × math × field-term) | **0.9262** | 0.9063 | 0.9494 | 0.9306 | +| overall AUC (raw UID) | **0.9487** | 0.9392 | 0.9587 | 0.9487 | +| overall balanced accuracy | **0.8705** | 0.8602 | 0.8883 | 0.8704 | +| matched-stratum AUC (section × length × math × field-term) | **0.9250** | 0.9058 | 0.9495 | 0.9262 | -The matched-stratum AUC stays within ~0.02 of the overall AUC, so the separation is not +The matched-stratum AUC stays within 0.024 of the overall AUC, so the separation is not merely a topic, length, or mathematical-density artifact. -**What the v0.36.3 profile rebuild moved.** Every headline figure eased by less than -its own interval: AUC −0.003, balanced accuracy −0.006, matched-stratum AUC −0.004, -each inside the 2.5–97.5% range of the audit before it. The `joint_matched_support` -cell holds 1,358 records across 6 qualifying strata in the primary split (1,283 on -average over the 20 audit splits). The posture is unchanged: no operating point, -`degraded`, for the reason §7.2 gives. +**What the v0.39.0 profile rebuild moved.** Nothing beyond its own interval: AUC ++0.00001, balanced accuracy +0.0002, matched-stratum AUC −0.0012, each inside the +2.5–97.5% range of the audit before it. The `joint_matched_support` cell holds 1,365 +records across 6 qualifying strata in the primary split (1,277 on average over the 20 +audit splits). The posture is unchanged: no operating point, `degraded`, for the +reason §7.2 gives. ### 7.2 Negative controls — the confound the audit exposes -The false-positive rate is the fraction of generated negatives the model wrongly scores -as curated-field-like (mean across 20 splits): - -| Generated-negative control | false-positive rate | 95% split range | 2026-08-26 | -|---|---:|---:|---:| -| public-generic AI text | **0.055** | 0.026–0.076 | 0.052 | -| field-topic AI text | **0.295** | 0.230–0.356 | 0.280 | -| field-jargon-dense AI text | **0.421** | 0.236–0.543 | 0.393 | - -Generic public AI prose is easy (5.5% FPR), but AI text written in the field's topic and -jargon fools the model 30–42% of the time. The learned score partly measures field -register, so it is unreliable on the exact distribution — field-topic AI prose — that a -manuscript de-AI pass must catch. +The negative-control false-positive rates, per retrain and with their split ranges, +are the §7.0a table. Generic public AI prose is easy (5.7% FPR), but AI text written in +the field's topic and jargon fools the model 29–42% of the time. The learned score +partly measures field register, so it is unreliable on the exact distribution — +field-topic AI prose — that a manuscript de-AI pass must catch. -**Neither the corpus growth nor the profile rebuild fixed this.** Field-jargon-dense -prose has read 0.417 → 0.410 → 0.393 → 0.421 across four retrains and field-topic 0.318 -→ 0.285 → 0.280 → 0.295, every step inside the split-to-split range; public-generic sits -at 5%, the easy case. Four retrains on banks of very different sizes and two profile -generations now agree that the confound is a property of the feature set, not of any -one training bank — which is the measured reason L3 ships `degraded` and will keep -doing so until a feature set separates field register from provenance. +**Neither the corpus growth nor the two profile rebuilds fixed this**: across five +retrains every step of every control lies inside the split-to-split range (§7.0a). The +confound is a property of the feature set, not of any one training bank — which is the +measured reason L3 ships `degraded` and will keep doing so until a feature set +separates field register from provenance. ### 7.3 Author hard set — true provenance is the yardstick, perception is not @@ -152,11 +146,11 @@ the provenance is the meaningful one. Every AUC below is reported with a seeded | Question | AUC | 95% CI | |---|---:|---:| -| Does the **model** score separate true generated from human? (low compatibility = generated) | 0.938 | 0.846–0.998 | +| Does the **model** score separate true generated from human? (low compatibility = generated) | 0.937 | 0.842–0.999 | | Can the **author's `ai_feel`** separate the same true provenance? | 0.444 | 0.304–0.582 | -Mean compatibility by true source on the retrained bundle: human papers 0.916 and the -author's own drafts 0.887, against RAID generations 0.189 and Claude generations 0.373. +Mean compatibility by true source on the retrained bundle: human papers 0.917 and the +author's own drafts 0.887, against RAID generations 0.185 and Claude generations 0.372. The author's drafts score with the published papers, not with the generations. The model separates true AI-vs-human prose well (0.94). The author's single-paragraph @@ -177,8 +171,8 @@ secondary line in `voice_model_evaluation.json`, not as evidence about the model L3 stays `degraded` with **no operating point** — but for the well-powered reasons, not the hard-set perception metric: -1. the field-topic and field-jargon-dense negative controls (§7.2, n=170/48 in the - primary split) show a 31–42% false-positive rate on exactly the AI prose a manuscript +1. the field-topic and field-jargon-dense negative controls (§7.2, n=170/47 in the + primary split) show a 31–43% false-positive rate on exactly the AI prose a manuscript pass must catch; 2. AI-ness in scientific writing is substantially a document- and cross-paragraph property, and no document-level calibration set exists yet (§9). @@ -192,7 +186,7 @@ never a universal cutoff. - Grouping by source paper reduces same-paper leakage; the matched-stratum result adds section/length/math/jargon control, but observational separation is not causal proof. -- The `results` stratum holds 687 records in the primary split and no generated +- The `results` stratum holds 706 records in the primary split and no generated negative, so its per-section figures are reported and not interpreted. - Held-out classification performance alone is insufficient for rewrite ranking outside the training distribution; §8 gates ranking on measured calibration. diff --git a/docs/architecture/evaluation/lexical-structure-uid.md b/docs/architecture/evaluation/lexical-structure-uid.md index f272f5b..65155c7 100644 --- a/docs/architecture/evaluation/lexical-structure-uid.md +++ b/docs/architecture/evaluation/lexical-structure-uid.md @@ -37,28 +37,66 @@ These tests are in ## 5. Sentence-structure reference evidence `style-profile/wgl/structure_baseline.json` -contains 27,841 paragraph observations across seven section buckets — `method` 9,478, -`results` 3,978, `data` 3,894, `intro` 3,812, `discussion` 3,635, `conclusion` 2,611, +contains 28,439 paragraph observations across seven section buckets — `method` 9,654, +`results` 4,065, `data` 3,953, `intro` 3,848, `discussion` 3,739, `conclusion` 2,747, `abstract` 433. The file records reference fractions for announced enumeration, ordinal runs, tricolon-like setup/list patterns, anaphora, balanced closers, and aggregate templating. -Counts are post-2026-09-04 (v0.36.3) and reflect five rounds of corpus-layer fixes -plus the 500-paper breadth corpus (EVALUATION §2). The third round is the -heading-coverage work in limit 5 below, which added 2,946 paragraphs (25,005 -> 27,951); -the fourth is v0.32.0's citation projection fix, which removed 44 (27,951 -> 27,907) -because a paragraph made only of leaked bibliography keys is no longer a paragraph of -prose; the fifth is the v0.36.3 profile rebuild on the v0.36.0–v0.36.2 corpus-side -fixes (heading whitespace and `\texorpdfstring`, floats and citations blanked across -lines, one assembly reader), which re-cut 66 bank rows (27,907 -> 27,841): 207 rows -gone, 131 added and 478 re-paragraphed, across 48 of the 516 papers with bank -rows. -The v0.27.1 file read 593 observations with -`results` at 26, under its 30-passage floor; the v0.27.0 file read 1,942 with `method` -at 1,671, but `method` was then `DEFAULT_SECTION_BUCKET` and absorbed every unnamed -heading, and paragraphs were split from PDF line fragments rather than reconstructed. -None of the three is comparable to the others as a count of anything. **Every bucket -now clears the floor**, so no bucket is rank-only. +**How the bank reached this shape.** This is the record's one account of the +corpus-layer history; every section-keyed count elsewhere is read against it. + +| bucket | v0.27.1 (31 files) | v0.28.0 | v0.29.0 (heading coverage) | v0.32.0 (citation fix) | v0.36.3 (profile rebuild) | v0.39.0 (profile rebuild) | +|---|---:|---:|---:|---:|---:|---:| +| abstract | 15 | 433 | 433 | 433 | 433 | 433 | +| intro | 109 | 3,753 | 3,844 | 3,840 | 3,812 | 3,848 | +| data | 112 | 3,929 | 3,915 | 3,908 | 3,894 | 3,953 | +| method | 163 | 8,144 | 9,522 | 9,512 | 9,478 | 9,654 | +| discussion | 118 | 3,088 | 3,653 | 3,647 | 3,635 | 3,739 | +| conclusion | 48 | 2,533 | 2,610 | 2,609 | 2,611 | 2,747 | +| results | 26 | **3,118** | **3,964** | **3,958** | **3,978** | **4,065** | +| **total** | **593** | **25,005** | **27,951** | **27,907** | **27,841** | **28,439** | + +- **v0.28.0 (2026-08-25), two rounds of corpus-layer defects.** Section labels: + `classify_section` matched titles in the singular only, so + `Results`/`Conclusions`/`Systematics` fell to `method`; `method` was the default + bucket and absorbed every unnamed heading; PDF table cells were accepted as + headings and PDF "paragraphs" were line fragments. An unrecognised heading is now + `unknown` and is dropped rather than guessed. What counts as a paper: `\include` + fragments counted as separate papers (one review entered every distribution + twelve times); selecting the root instead lost the body it includes (72 words in + place of 64,657); the root selector and the reader resolved `\input` targets + differently; a `\subsection` did not inherit its `\section`, sending 54.8% of all + section words to `unknown`; and the 500-paper `fulltext-arxiv/` breadth corpus was + invisible to every paragraph-level baseline. Register composition moved with it: + abstracts fell from 96% of the reference to 35%. +- **v0.29.0**, the heading-coverage work in limit 5 below: 2,946 paragraphs added. +- **v0.32.0**, the citation projection fix: 44 removed, because a paragraph made + only of leaked bibliography keys is no longer a paragraph of prose. +- **v0.36.3**, the rebuild on the v0.36.0–v0.36.2 corpus-side fixes (heading + whitespace and `\texorpdfstring`, floats and citations blanked across lines, one + assembly reader): 66 fewer — 207 rows gone, 131 added and 478 re-paragraphed, + across 48 of the 516 papers with bank rows. +- **v0.39.0 (2026-09-27)**, the rebuild on the corpus-side fixes of the 2026-09-27 + audit: a paragraph that opens on a citation or an equation enters the bank (D16), + placeholders are no longer words (D7), non-ASCII letters are letters (D10), and + `\[…\]` and `$$…$$` are mathematics on the corpus side as on the manuscript side + (A5, A6). 598 more on the same 516 papers; how the 598 divide between those fixes + was not measured. + +The v0.27.1 file read 593 observations with `results` at 26, under its 30-passage +floor; the v0.27.0 file read 1,942 with `method` at 1,671, when `method` absorbed +every unnamed heading and paragraphs were split from PDF line fragments. None of the +early counts is comparable to the others as a count of anything. **Every bucket now +clears the floor** — `results`, under it in v0.27.1, holds 4,065 — so no bucket is +rank-only. + +The curated tiers and the breadth corpus are distinct roles. The tiers carry every +weighted aggregate and the dossier; the breadth corpus is unweighted and feeds the +reference distributions only, so it cannot restyle the imitation target. +`retrieve_exemplars` reads the curated tiers by default. At v0.28.0 one arXiv +bundle in 500 lost 35% of its prose, because it ships chapter files with no root +that assembles them. Interpretation limits: @@ -98,30 +136,31 @@ Interpretation limits: ## 6. UID reference evidence -`style-profile/wgl/uid_baseline.json` records **27,851** paragraphs that met its +`style-profile/wgl/uid_baseline.json` records **28,444** paragraphs that met its 25-token requirement. It stores pooled and section-level means, standard deviations, and counts for global UID, local UID, and mean surprisal under GPT-2-large. Pooled -global UID is **3.303 ± 0.420**; local UID 3.417 ± 0.445; mean surprisal 3.563 ± 0.524. +global UID is **3.301 ± 0.413**; local UID 3.412 ± 0.439; mean surprisal 3.558 ± 0.519. | bucket | n | global UID | |---|---:|---| -| method | 9,485 | 3.31 ± 0.45 | -| results | 3,979 | 3.28 ± 0.26 | -| data | 3,895 | 3.36 ± 0.51 | -| intro | 3,812 | 3.26 ± 0.41 | -| discussion | 3,636 | 3.29 ± 0.37 | -| conclusion | 2,611 | 3.33 ± 0.43 | +| method | 9,659 | 3.30 ± 0.44 | +| results | 4,065 | 3.28 ± 0.26 | +| data | 3,953 | 3.35 ± 0.50 | +| intro | 3,848 | 3.26 ± 0.39 | +| discussion | 3,739 | 3.29 ± 0.37 | +| conclusion | 2,747 | 3.35 ± 0.43 | | abstract | 433 | 3.23 ± 0.49 | -Counts are post-2026-09-05 (fifth rebuild, on the 27,851-row bank). Neither the 593-paragraph / 3.383 ± 0.680 +Counts are from the v0.39.0 rebuild (2026-09-27), on the 28,444-row bank. Neither the 593-paragraph / 3.383 ± 0.680 nor the 1,942-paragraph / 3.329 ± 0.391 predecessor is comparable: the first was blind to the breadth corpus, and the second labelled most of its paragraphs `method` by default and split them from PDF line fragments, so both its size and its section keys -described something other than what they claimed (§14.2, EVALUATION §2). +described something other than what they claimed (§5). -The section means are strikingly tight — 3.23 to 3.36 across seven buckets, a spread of -0.13 against within-bucket standard deviations of 0.26–0.51. It was 0.15 before the -fourth rebuild and 0.13 after, so removing the leaked keys did not disturb the null. **Section identity barely +The section means are strikingly tight — 3.23 to 3.35 across seven buckets, a spread of +0.12 against within-bucket standard deviations of 0.26–0.50. It was 0.15 before the +v0.32.0 citation fix, 0.13 after it and after v0.36.3, and is 0.12 now, so neither removing +the leaked keys nor the later rebuilds disturbed the null. **Section identity barely moves paragraph-level UID in this corpus.** That is a null worth recording rather than a defect: it means a per-section UID operating point would be calibrating on a difference smaller than its own noise, which is consistent with §9.8's finding that the diff --git a/docs/architecture/evaluation/narrative-salience-register.md b/docs/architecture/evaluation/narrative-salience-register.md index cfb061e..1fa62d9 100644 --- a/docs/architecture/evaluation/narrative-salience-register.md +++ b/docs/architecture/evaluation/narrative-salience-register.md @@ -186,13 +186,11 @@ of a definition — counting the 3 in a volume formula as a reported quantity ma every derivation paragraph read as a recital of measurements. The LaTeX thousands form `1{,}250{,}000` collapses to one numeral rather than three. `latex_to_plain` is untouched, so no existing calibration asset moved then. -One did later, and in the other direction: the bank stored only the -`latex_to_plain` text, so the salience reference was calibrated on paragraphs -with no numeral inside math while the manuscript side counted them, and the -p90 gate fired at 0.45 per held-out passage against the 0.27 union bound. Since -v0.36.3 every bank row carries `numeral_text`, the same paragraph under this -projection, and the calibration reads it; the record of the asymmetry and its -closure is §17.5 in [held-out-labels.md](held-out-labels.md). +One did later: until v0.36.3 the bank stored only the `latex_to_plain` text, +so the reference counted no numeral inside math while the manuscript side +counted every one. Every bank row now carries `numeral_text` and the +calibration reads it; §17.5 in [held-out-labels.md](held-out-labels.md) is the +record. ### 14.2 Salience reference and operating points @@ -200,58 +198,32 @@ Calibrated per section bucket at one shared unit (a passage) on the reference and measurement sides, from the field's own banks, on the numeral projection of each passage since v0.36.3: -| Bucket | n (v0.36.3) | 2026-08-25 | v0.27.1 | 2026-08-17 | Sources | -|---|---:|---:|---:|---:|---| -| abstract | **13,971** | 13,971 | 13,438 | 13,438 | `human_abstracts_extra.jsonl`, `exemplar_paragraphs.jsonl` | -| method | **6,967** | 6,959 | 130 | 1,303 | `exemplar_paragraphs.jsonl` | -| data | **3,020** | 3,016 | 102 | — | `exemplar_paragraphs.jsonl` — bucket added 2026-08-25 | -| intro | **3,239** | 3,264 | 94 | 88 | `exemplar_paragraphs.jsonl` | -| discussion | **2,954** | 2,958 | 93 | 78 | `exemplar_paragraphs.jsonl` | -| results | **3,240** | 3,206 | 26 | 10 | `exemplar_paragraphs.jsonl` — clears the floor by 108× | -| conclusion | **1,999** | 1,994 | 39 | 41 | `exemplar_paragraphs.jsonl` | - -The 2026-08-25 column is the rebuild that made `method` a reference, and `method` is the headline: **it was never a -1,303-passage reference.** `method` had no pattern of its own — it was -`DEFAULT_SECTION_BUCKET`, so it absorbed every heading the classifier could not name, -and 92% of it was residue. Three sources fed it, all measured: `.tex` corpus files -carrying no `\section` markup at all (one 100-page review, split into chunks, -contributed 101 and 88 paragraphs at 100% "method"); PDF table cells accepted as -headings by the ALL-CAPS heuristic (305 of 325 headings detected across the 90 corpus -PDFs were cells such as "S", "RA", "NFW", "A85", each one switching the bucket that -following prose was filed under); and data/observation sections with no bucket of their -own. Giving `method` explicit vocabulary, giving `data` its own bucket, defaulting to -`unknown`, and requiring an ALL-CAPS heading to carry ≥ 2 words, ≥ 4 letters and ≥ 75% -letters produces the left-hand column. - -A second fix landed in the same rebuild and pulls the other way. `extract_pdf_text` -took pymupdf's `get_text("blocks")` to mean paragraphs; on real journal PDFs blocks -run to a median of 5–16 words and only 21–23% end a sentence, so roughly four in five -were mid-paragraph line fragments. Rejoining them produces real paragraphs — a corpus -PDF goes from 438 fragments (median 5 words) to 105 paragraphs (median 84), and -`document_shape` on it moves from `insufficient_evidence` to `measured` — but real -paragraphs are fewer than fragments, so every bucket loses some count. - -**`results` therefore does not clear the floor.** It moved 10 → 31 on the classifier -fix alone and settled at **26** once paragraphs and headings were reconstructed -correctly. Intermediate values of 31 and 24 were recorded here while the PDF fixes were -landing; 26 is the number that survived all of them. - -That 26 was then read as a corpus-size limit — `gather_corpus_files` ingested 31 -documents, so growing `results` was said to need more papers. **That diagnosis was -wrong**, and the second round of 2026-08-25 fixes says why: the corpus layer treated a -*file* as a paper, and could only see the three curated tiers. The 500-paper -`fulltext-arxiv/` corpus that §9 has calibrated against since v0.26 was on the same -disk, gitignored, and invisible to every paragraph-level baseline. Reading it — with -`\include` fragments folded back into their documents rather than counted separately, -and `\subsection`s inheriting their `\section` — takes `results` to **2,541** and the -bank to 25,005. EVALUATION §2 records all four defects and their measured effect. - -The bank therefore reads 1,942 → 593 → 25,005 across three rebuilds, and none of the -three counts the same thing: 1,942 was inflated by mislabelling and fragment-splitting, -593 was honest but blind to 94% of the available corpus, and 25,005 is the count of -correctly-paragraphed passages whose section is identifiable across everything on disk. -The `abstract` bucket moves only 13,438 → 13,823, because abstracts come from their own -bank and only the exemplar-side contribution grew. +| Bucket | n (v0.39.0) | v0.36.3 | 2026-08-25 | v0.27.1 | 2026-08-17 | Sources | +|---|---:|---:|---:|---:|---:|---| +| abstract | **13,970** | 13,971 | 13,971 | 13,438 | 13,438 | `human_abstracts_extra.jsonl`, `exemplar_paragraphs.jsonl` | +| method | **7,209** | 6,967 | 6,959 | 130 | 1,303 | `exemplar_paragraphs.jsonl` | +| data | **3,098** | 3,020 | 3,016 | 102 | — | `exemplar_paragraphs.jsonl` — bucket added 2026-08-25 | +| intro | **3,280** | 3,239 | 3,264 | 94 | 88 | `exemplar_paragraphs.jsonl` | +| discussion | **3,051** | 2,954 | 2,958 | 93 | 78 | `exemplar_paragraphs.jsonl` | +| results | **3,337** | 3,240 | 3,206 | 26 | 10 | `exemplar_paragraphs.jsonl` — clears the floor by 111× | +| conclusion | **2,018** | 1,999 | 1,994 | 39 | 41 | `exemplar_paragraphs.jsonl` | + +The earlier columns are history, not comparable counts; §5 in +[lexical-structure-uid.md](lexical-structure-uid.md) is the one account of the +corpus-layer fixes between them. The measurements taken on this reference: the +2026-08-17 `method` was never a 1,303-passage reference. As the default bucket +it absorbed every heading the classifier could not name, and 92% of it was +residue: `.tex` files with no `\section` markup (one 100-page review, split +into chunks, contributed 101 and 88 paragraphs at 100% `method`), PDF table +cells accepted as headings by the ALL-CAPS heuristic (305 of the 325 headings +detected across the 90 corpus PDFs were cells such as "S", "RA", "NFW", "A85"), +and data sections with no bucket of their own. An ALL-CAPS heading now needs +≥ 2 words, ≥ 4 letters and ≥ 75% letters. PDF blocks ran to a median of 5–16 +words and only 21–23% ended a sentence; rejoined, one corpus PDF goes from 438 +fragments (median 5 words) to 105 paragraphs (median 84), which is why every +bucket lost count in that rebuild and `results` settled at 26 before the +breadth corpus took it past the floor. `abstract` moves least, because +abstracts come from their own bank. Human abstract percentiles: @@ -259,7 +231,7 @@ Human abstract percentiles: |---|---:|---:|---:|---:| | `max_recital_run_frac` | 0.20 | 0.33 | 0.50 | 0.67 | | `recital_frac` | 0.29 | 0.50 | 0.67 | 0.78 | -| `numerals_per_sentence` | 0.57 | 1.25 | 2.00 | 2.70 | +| `numerals_per_sentence` | 0.57 | 1.25 | 2.09 | 2.71 | The gate is P(X ≤ x) > 0.90 for an advisory and > 0.95 for a strong one. Two implementation facts are load-bearing: @@ -309,39 +281,39 @@ as observed evidence, gives 19. ### 14.4 Register reference and precision -Document frequency over **41,644** passages (27,851 exemplar paragraphs + 13,793 -abstracts), **53,367** terms, after the v0.36.3 profile rebuild (41,710 and -53,414 before it). Firing rule: ≥ 15 manuscript uses **and** corpus df +Document frequency over **42,237** passages (28,444 exemplar paragraphs + 13,793 +abstracts), **53,668** terms, on the 2026-09-27 rebuild. Firing rule: ≥ 15 +manuscript uses **and** corpus df rate < 1e-4. (The use count was 5 until v0.32.0 raised it to 15, and this paragraph still said 5 until the 2026-08-26 sweep caught it. §17 records the held-out rates at the old setting; §18 records them at the new one.) The composition bias recorded in §14.6 is largely resolved. Reading the 500-paper breadth corpus took the body contribution from 593 to 25,005 passages and it now -stands at 27,851, so abstracts fell from **96% of the reference to 33%** (13,793 -of 41,644). The axis was biased toward flagging body-section vocabulary precisely +stands at 28,444, so abstracts fell from **96% of the reference to 33%** (13,793 +of 42,237). The axis was biased toward flagging body-section vocabulary precisely because body text was almost absent from its reference; that reference is now -two-thirds body prose. The threshold's resolution also improves by 2.9×: 41,644 +two-thirds body prose. The threshold's resolution also improves by 3.0×: 42,237 passages can express a df rate of 2.4e-5, comfortably below the 1e-4 firing rule, where 14,235 could express 7.0e-5. -Controls that must not fire: +Controls that must not fire (re-read on the 2026-09-27 bank of 42,237 passages): | Term | df | Judged on | Rate | Result | |---|---:|---|---:|---| -| `accuracy` | 774 | itself | 5.0e-2 | not flagged | -| `epoch` | 402 | itself | 2.6e-2 | not flagged | -| `aperture-mass` | 313 | `aperture` | 2.0e-2 | not flagged | -| `training` | 155 | itself | 9.9e-3 | not flagged | -| `benchmark` | 81 | itself | 5.2e-3 | not flagged | -| `classifier` | 29 | itself | 1.9e-3 | not flagged | -| `held-out` | 24 | `held` | 1.5e-3 | not flagged | +| `accuracy` | 1,612 | itself | 3.8e-2 | not flagged | +| `epoch` | 615 | itself | 1.5e-2 | not flagged | +| `aperture-mass` | 627 | `aperture` | 1.5e-2 | not flagged | +| `training` | 658 | itself | 1.6e-2 | not flagged | +| `benchmark` | 132 | itself | 3.1e-3 | not flagged | +| `classifier` | 47 | itself | 1.1e-3 | not flagged | +| `held-out` | 75 | `held` | 1.8e-3 | not flagged | `epoch` and `accuracy` are why the rule cannot be a curated list: both are ordinary astronomy vocabulary (an observation time; plain English), and both appear on any hand-written "machine-learning words" list. -Positives on the case document: +Positives on the case document, measured on the 15,599-passage bank of v0.26.0: | Term | Manuscript uses | Corpus df | Rate | |---|---:|---:|---:| @@ -353,7 +325,9 @@ The third term is the concept the paper introduces, so its correct disposition is the third the action offers (confirm the first occurrence carries a definition) rather than replacement. The axis cannot distinguish an introduced concept from a borrowed one and does not try to, which is why it emits -advisories only. +advisories only. On the 2026-09-27 bank `AUC` has df 8 and `logit` df 9, both +above the gate, so neither would be flagged now: the breadth corpus brought both +terms into the field's own writing. **Precision history.** The unguarded first implementation produced 48 findings on this document, of which roughly six were real. Three construction classes diff --git a/docs/architecture/evaluation/vocabulary-and-residue.md b/docs/architecture/evaluation/vocabulary-and-residue.md index e15ed6f..d287704 100644 --- a/docs/architecture/evaluation/vocabulary-and-residue.md +++ b/docs/architecture/evaluation/vocabulary-and-residue.md @@ -1,15 +1,17 @@ # EVALUATION §23 — vocabulary the field never wrote, sentence families from a mentor's comments, the removal map, and the residue an edit leaves Part of the evidence record whose hub is [`../EVALUATION.md`](../EVALUATION.md); -section numbers are global. Everything here was measured on 2026-09-04 against -the rebuilt `wgl` profile (41,710 passages), the 203 held-out refereed +section numbers are global. Everything here was first measured on 2026-09-04; +the tables were re-taken on 2026-09-27 against the profile rebuilt that day +(42,237 passages), the 203 held-out refereed ApJ/ApJL/A&A papers of §17 (median 6,944 body prose words, 9,793 counting the raw source; none in any calibration bank), the 544 in-sample refereed papers, and the 173 `docval` machine documents of §16. Every per-1,000-word rate is per body prose word since v0.36.2 — the register projection with placeholders dropped — where it was per raw-source word, bibliography and preamble included. A manuscript under review that motivated the work is measured too; nothing -from it is quoted, and only the counts the tools report on it are given. +from it is quoted, only the counts the tools report on it are given, and those +paragraphs keep their 2026-09-04 date. ## 23.1 The zero-hit audit: every word the corpus never wrote @@ -26,23 +28,23 @@ word is strong and takes one of the §5.2 step-1b dispositions. **It is exhaustive, and it is not a detector.** Rates per 1,000 body prose words, and the fraction of documents with at least one finding -(`eval_findings.py --field wgl`, 2026-09-04, re-taken under v0.36.2): +(`eval_findings.py --field wgl`, re-taken on the 2026-09-27 rebuild): | population | n | `register-zero` /1k words | documents flagged | `register` (15-use rule) /1k words | |---|---:|---:|---:|---:| -| held-out refereed | 203 | **3.374** | 100.0% | 0.037 | -| in-sample refereed (bank members) | 544 | 0.812 | 75.0% | 0.007 | -| machine, all tiers | 173 | 1.054 | 79.8% | 0.002 | +| held-out refereed | 203 | **3.337** | 100.0% | 0.036 | +| in-sample refereed (bank members) | 544 | 0.790 | 73.7% | 0.006 | +| machine, all tiers | 173 | 1.049 | 78.6% | 0.002 | | machine `ai_adversarial` | 40 | 2.132 | 92.5% | 0.000 | -| machine `ai_long` | 29 | 1.043 | 100.0% | 0.003 | +| machine `ai_long` | 29 | 1.040 | 100.0% | 0.003 | -Rank AUC of machine over held-out on the per-1k rate is **0.174**: a refereed +Rank AUC of machine over held-out on the per-1k rate is **0.177**: a refereed paper carries more words the corpus never wrote than a machine draft does, which is the same direction §18.4 found for the thresholded rule at every floor from 5 to 50. The audit therefore ships as advice with author dispositions and never as a gate, and the standard says so (§2 L0). The in-sample row is what bank membership buys — a paper whose own vocabulary is in the denominator -loses two thirds of its zero-hit words — and is not a leakage estimate, for the +loses three quarters of its zero-hit words — and is not a leakage estimate, for the era reason §17.3 gives. **What it finds on the manuscript under review** (re-measured on the current @@ -74,8 +76,8 @@ documents, AUC 0.392, own-membership 98.2% of 57. v0.36.2 moved the denominator, not the findings: the same 57 read **0.0371** per 1,000 body prose words (AUC 0.391) and the audit 3.374 (AUC 0.174), because a refereed paper's raw source carries a bibliography and a preamble that a single-file -machine draft does not, so the old rate was diluted on one side only; the -table above is the re-taken one. +machine draft does not, so the old rate was diluted on one side only. The +table closing §17.4 carries this chain to the current build. ## 23.2 `L2.collocation`: sentences that join words the field never joins @@ -91,11 +93,10 @@ shipped axis below, which is judged per sentence and reports its document fraction as evidence only. **Design.** `deai_collocation.py --calibrate` builds a bank from the two human -passage banks: unigram document frequency for the 11,286 words at or above -`COMMON_RATE` (2 × 10⁻⁴), and passage frequency for 530,677 adjacent pairs of -them (the v0.36.2 bank this section was measured on; the v0.36.3 rebuild holds -11,282 words and 530,504 pairs, and the per-bucket sizes below are the v0.36.2 -ones). A pair is two words the writer put side by side, so it breaks at +passage banks: unigram document frequency for the 11,368 words at or above +`COMMON_RATE` (2 × 10⁻⁴), and passage frequency for 535,517 adjacent pairs of +them (the 2026-09-27 rebuild, which the tables below were re-taken on). A pair +is two words the writer put side by side, so it breaks at punctuation, at a `[math]`/`[CITE]` placeholder, at a dash, at a slash and at a digit (`yields, separate` is two clauses; `50/50` and `2 sigma` are not word pairs), and a sentence's pairs are judged distinct. The unit is the sentence; @@ -113,29 +114,29 @@ stronger absence than a pair of two rare ones. | bucket | n sentences | p50 | p90 (advisory) | p95 (strong) | |---|---:|---:|---:|---:| -| abstract | 89,392 | 0.250 | 0.588 | 0.667 | -| method | 27,293 | 0.200 | 0.500 | 0.600 | -| intro | 14,382 | 0.200 | 0.444 | 0.500 | -| results | 12,419 | 0.200 | 0.500 | 0.571 | -| data | 12,054 | 0.200 | 0.500 | 0.600 | -| discussion | 12,205 | 0.200 | 0.500 | 0.571 | -| conclusion | 8,571 | 0.200 | 0.500 | 0.556 | +| abstract | 89,377 | 0.250 | 0.583 | 0.667 | +| method | 28,114 | 0.200 | 0.500 | 0.600 | +| intro | 14,529 | 0.200 | 0.444 | 0.500 | +| results | 12,911 | 0.200 | 0.500 | 0.571 | +| data | 12,346 | 0.200 | 0.500 | 0.600 | +| discussion | 12,584 | 0.200 | 0.500 | 0.571 | +| conclusion | 8,741 | 0.200 | 0.467 | 0.545 | **Transfer and separation** (`eval_findings.py`, findings per 1,000 words and documents with at least one): | population | n | findings /1k words | documents flagged | |---|---:|---:|---:| -| held-out refereed | 203 | 4.700 | 99.0% | -| in-sample refereed (bank members) | 544 | 0.203 | 43.4% | -| machine, all tiers | 173 | 4.811 | 100.0% | -| machine `ai_long` | 29 | 4.534 | 100.0% | -| machine `ai_adversarial` | 40 | 6.128 | 100.0% | +| held-out refereed | 203 | 4.663 | 99.0% | +| in-sample refereed (bank members) | 544 | 0.128 | 32.4% | +| machine, all tiers | 173 | 4.791 | 100.0% | +| machine `ai_long` | 29 | 4.519 | 100.0% | +| machine `ai_adversarial` | 40 | 6.146 | 100.0% | On the document novel-pair fraction the rank AUC of machine over held-out is -**0.704** — the axis separates, unlike the two vocabulary rules, and more than -salience now does (0.572 on body-word density; §17.5 records why that figure -moved). Two things the table must not be read as. The held-out row +**0.703** — the axis separates, unlike the two vocabulary rules, and more than +salience does (0.666 on body-word density, §17.5). Two things the table must +not be read as. The held-out row is an upper bound on the design rate, not the rate itself: the reference is leave-one-out over bank passages of one era, and a 2020–2021 paper's pairs drift from a 2012–2018 bank for reasons that have nothing to do with how it is @@ -157,12 +158,14 @@ exception at work, recorded in [`examples/README.md`](../../../examples/README.m **Two limits recorded on 2026-09-27.** The bank keeps rows of 30–400 words and the manuscript side has no such band, so a manuscript paragraph outside it -is measured against a reference that holds none of its length (audit C17, open). -Each artifact bucket now records `text_key` (the projection its rows were read -on) and `text_fallback_rows` (how many fell back to `text`), and the quantile -grid is stored at nearest rank; the gates quoted above were measured on the -earlier grid, one order statistic high, and move only when the artifact is -recalibrated and the rate re-measured. +is measured against a reference that holds none of its length. Recording the +asymmetry here is the fix the audit asked for (C17): the band is the bank's +admission rule, and applying it to a manuscript would skip exactly the +paragraphs that fall outside the field's range. The references built through +`deai_reference.calibrate` (salience, cohesion, hedging) record per bucket +`text_key` (the projection its rows were read on) and `text_fallback_rows` (how +many fell back to `text`); every quantile grid, this one included, is stored at +nearest rank (audit C12), and the gates above are read on it. ## 23.3 Three structure families from the mentor's margin @@ -178,14 +181,14 @@ more tokens before the head noun with at least two hyphenated compounds). | bucket | n paragraphs | paper-agent | wh-cleft | modifier stack | |---|---:|---:|---:|---:| | abstract | 433 | 0.92% | 0.00% | 15.94% | -| intro | 3,840 | 0.29% | 0.13% | 5.83% | -| method | 9,512 | 0.13% | 0.05% | 3.44% | -| data | 3,908 | 0.23% | 0.15% | 4.32% | -| results | 3,958 | 0.28% | 0.03% | 2.53% | -| discussion | 3,647 | 0.41% | 0.16% | 3.32% | -| conclusion | 2,609 | 1.00% | 0.23% | 5.83% | +| intro | 3,848 | 0.29% | 0.13% | 5.98% | +| method | 9,654 | 0.11% | 0.06% | 3.56% | +| data | 3,953 | 0.23% | 0.15% | 4.40% | +| results | 4,065 | 0.27% | 0.02% | 2.58% | +| discussion | 3,739 | 0.40% | 0.16% | 3.34% | +| conclusion | 2,747 | 0.95% | 0.22% | 5.82% | -The `n` column is the v0.32.0 bank's; the v0.36.3 `structure_baseline.json` holds intro 3,812, method 9,478, data 3,894, results 3,978, discussion 3,635, conclusion 2,611, and the fractions were not re-taken on it. +Counts and fractions are the 2026-09-27 rebuild's `structure_baseline.json`. A first cut of the stack rule counted any run of non-function tokens and put 15% of human method paragraphs in it; cutting the run at the head noun and @@ -202,14 +205,15 @@ edit-meta literals, a heading or caption whose object the body never names, and — given `--before` or `--git-ref` — a label the edit added and does not earn. A strong finding exits 1. The strengths were set on the 203 held-out papers (70,848 body prose sentences; 85,324 under the raw-source projection the first three -passes counted), in four passes: +passes counted), in four passes, and re-measured on 2026-09-27: | pass | self-history strong | edit-meta strong | negative-label | documents with a strong finding | |---|---:|---:|---:|---:| | raw document, first families | 243 | 184 | 78 (strong) | 154 / 203 | | body projection, `initially`/`originally` ordinary | 203 | 67 | 74 (ordinary) | 106 / 203 | | `used to` dropped, `at first` ordinary | 24 in 20 papers | 67 in 15 papers | 74 in 58 papers | 34 / 203 (16.7%) | -| v0.36.2: marks read across lines, `we have added` needs a document object | **24 in 20 papers** | **69 in 8 papers** | 67 in 53 papers | **27 / 203 (13.3%)** | +| v0.36.2: marks read across lines, `we have added` needs a document object | 24 in 20 papers | 69 in 8 papers | 67 in 53 papers | 27 / 203 (13.3%) | +| 2026-09-27: the source as both CLIs read it, comments blanked; the label rule's `-ate` family and appendix prose | **24 in 20 papers** | **54 in 5 papers** | 64 in 52 papers | **24 / 203 (11.8%)** | The first pass was the projection asymmetry again: `\newcommand{\TODO}` in a preamble and "Planck Collaboration XXX" in a bibliography are not prose an @@ -223,9 +227,15 @@ wrapped at a line break had been invisible, and once seen it fired fourteen times, every one a procedure (`we have added uniform Gaussian noise`) or another paper's history (`in the revised version of \citet{…}`) — the `used to` lesson again — so the phrase is a mark only with a document object (`we -have added a paragraph`). What remains strong fires in 13% of refereed -papers; 68 of the 69 edit-meta hits are `\textcolor{red}` inside tables, an -author's emphasis that the disposition `kept` answers in a word. The static +have added a paragraph`). The last row corrects the measurement, not the +rules: `ai_ism_lint` and `deai_residue` both read a manuscript through +`ai_ism_lint.document_source`, which blanks `%` comments, while the v0.36.2 +sweep read the raw source, so 15 edit-meta hits in 3 papers and 2 negative +labels sat in comments no user is shown; the label rule's fixes removed one +more. What remains strong fires in 11.8% of refereed papers (18.7% with the +absence rule of §23.4a, strong in 16 papers); 53 of the 54 edit-meta hits are +`\textcolor{red}`, at least 48 of them in table rows, an author's emphasis that +the disposition `kept` answers in a word. The static negative-label rule fires on 26% of refereed papers and is ordinary; only the diff rule gates. The manuscript under review carries 0 residue findings. @@ -286,25 +296,29 @@ clause, so the human disposition decides, and the record names which. ## 23.5 The removal map: what a refereed paper has to remove -`condense_map.py` on the 203 held-out papers (median 7,174 prose words) and the +`condense_map.py` on the 203 held-out papers (median 6,622 prose words) and the 173 machine documents, removable words per 1,000 prose words — candidate mass -per scan, overlaps included: +per scan, overlaps included, p90 at nearest rank (re-taken 2026-09-27): | scan | held-out median | held-out p90 | |---|---:|---:| -| `condense-restatement` | 10.01 | 35.04 | -| `condense-zero-gain` | 8.37 | 23.29 | -| `condense-dead:*` | 0.49 | 8.21 | -| `condense-verbose` | 2.65 | 5.78 | -| `condense-regloss` | 0.00 | 0.50 | +| `condense-restatement` | 5.55 | 28.53 | +| `condense-zero-gain` | 8.62 | 26.53 | +| `condense-dead:*` | 0.40 | 8.00 | +| `condense-verbose` | 2.73 | 7.32 | +| `condense-regloss` | 0.00 | 0.00 | | `condense-duplicate` | 0.00 | 0.00 | The default target (restatement plus zero-gain outside the abstract/conclusion carve-out, each unit counted once since v0.36.2 — a sentence that was both a restatement and zero-gain, or a sentence inside a removed paragraph, had been -counted twice) is a median **1.59%** of prose on refereed papers (p90 4.11%), -total removable 2.48%; on machine documents 0.20% and 0.29%. The map is not a -detector either (AUC of machine over human on the default target 0.097 — short +counted twice) is a median **1.22%** of prose on refereed papers (p90 3.39%), +total removable 2.06%; on machine documents 0.19% and 0.26%. It was 1.59% (p90 +4.11%) before the map and gate fixes of the 2026-09-27 audit (the carve-out on +either side of a restatement pair, negation- and number-aware restatement, one +prose count for map and gate); how the difference divides between them was not +measured. The map is not a +detector either (AUC of machine over human on the default target 0.098 — short machine documents restate less) and is not meant as one: it exists because the `condense` skill's own sweep removed 1.5% of the shipped sample manuscript by reading, and a pass that cuts less than a refereed paper's own median has not diff --git a/docs/audits/codex-review-2026-09-04.md b/docs/audits/codex-review-2026-09-04.md index 3405873..c68f8d9 100644 --- a/docs/audits/codex-review-2026-09-04.md +++ b/docs/audits/codex-review-2026-09-04.md @@ -53,6 +53,34 @@ Read-only review of `git diff 9f0b920..edd480c`; session `01a06ebe-b4c5-7802-bfe 2. **应改文档/元数据**：`/1k` 分母口径；collocation scope=sentence；`presents`；定义豁免、名字豁免写进 standard；p_absent 降为 co-presence heuristic 或改名；tie policy。 3. **可延后**：A10 相邻 null；A15/A16 评估器与标注器；A14 pair 边界；统一 TeX 分解层（大工程）。 +## 逐条处理（2026-09-27 补记） + +v0.36.2 的发布说明按根因而非按编号写（`CHANGELOG.md` v0.36.2，提交 `3ea7699`）；本表把它逐条对回编号。三条在 v0.36.2 只落地了一侧，另一侧由 [2026-09-27 全库审计](full-review-2026-09-27.md) 发现并在 v0.39.0 补上。 + +| # | 处理 | 版本 | +|---|---|---| +| A1 | 已修：restatement 与 zero-gain 取并集，被删段落内的句子不再重计 | v0.36.2 | +| A2 | 已修：v0.36.2 挡住 canonical home 丢掉副本所带否定、比较、数字这一方向；反方向在 v0.39.0 补上（2026-09-27 审计 A12） | v0.36.2、v0.39.0 | +| A3 | 已修：开头套语只在它自己至多三个内容词会被删时才算整句零增益 | v0.36.2 | +| A4 | 已修：`--require-shrink` 读作百分比、分数或词数，拒绝 `100%`、`0%`、`inf`、`30%%`，分数向上取整到一个词 | v0.36.2 | +| A5 | 已修：gate 一侧不计占位符；map 一侧在 v0.39.0 补上（2026-09-27 审计 A13） | v0.36.2、v0.39.0 | +| A6 | 已修：统一段落扫描 `deai_reference.paragraphs`，float 在两侧都整块置空 | v0.36.2 | +| A7 | 已修：edit-meta 在可见正文（含 heading、caption）上整体扫描 | v0.36.2 | +| A8 | 已修：diff 规则比较完整的被否定对象，旧版本已有的否定不算新增 | v0.36.2 | +| A9 | 已修：`section_units` 改由 `extract_sections.RE_SECTION` 派生并继承，无标题文档为一个单元；`deai_residue.RE_LABEL` 一侧在 v0.39.0 补上（2026-09-27 审计 C4） | v0.36.2、v0.39.0 | +| A10 | 以改名收口：权重改名为 `expected_copresent_passages` / `p_copresence_absent`，只用于排序与展示，不进 gate | v0.36.2 | +| A11 | 已收窄：只有定义短语的宾语获得 `defined-here`，不再豁免整句 | v0.36.2 | +| A12 | 已修：宏定义按行首匹配，不再吞掉前面的空行 | v0.36.2 | +| A13 | 已修：`tex_assembly.py` 保留同一行 `\input` 前后的文字，重复 include 再次展开，只有当前 include 栈内的子文件算环 | v0.36.2 | +| A14 | 已修：词对在 `/`、`.` 和数字处断开，bank 重建 | v0.36.2 | +| A15 | 已修：可打分文档少于 20 篇时 AUC 为 `unmeasured` | v0.36.2 | +| A16 | 已修：对照样本按行跨度排除任何被 finding 触及的段落 | v0.36.2 | +| 结构 | 已修：structure 读统一段落扫描，heading 先置空 | v0.36.2 | +| 口径 | 已修：每千词分母是正文 prose 词（`prose_words(body_only(text))`） | v0.36.2 | +| 漂移（scope） | 已修：collocation 的 scope 与 calibration unit 都是 `sentence` | v0.36.2 | +| 漂移（`presents`） | 以改文档收口：标准与 skill 在 v0.36.2 改；DEAI_SUBSYSTEM 的例句在 v0.39.0 改（2026-09-27 审计 I25） | v0.36.2、v0.39.0 | +| tie policy | 已记录：plateau 顶端读法写入 EVALUATION §23.2；分位网格在 v0.39.0 改为最近秩（2026-09-27 审计 C12） | v0.36.2、v0.39.0 | + --- # Part 2 —— Codex 报告原文 diff --git a/examples/README.md b/examples/README.md index 9726662..c900f0b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -21,7 +21,7 @@ gitignored — so on a fresh clone both files report `unmeasured` for the corpus-referenced axes instead of the numbers below. Build one first with `python tools/build_profile.py --field ` and the `--calibrate` commands in `style-profile/README.md`. The figures here were produced against the `wgl` -profile on 2026-09-05 (v0.36.3) and will move with the corpus. +profile on 2026-09-27 (v0.39.0) and will move with the corpus. ## What it reports @@ -29,22 +29,23 @@ profile on 2026-09-05 (v0.36.3) and will move with the corpus. |---|---:|---:| | L0 targets | 1 | **0** | | integrity blockers | 0 | 0 | -| total advisories | 20 | **15** | -| strong advisories | 7 | **5** | -| document-scale findings | 8 | **6** | +| total advisories | 22 | **17** | +| strong advisories | 8 | **6** | +| document-scale findings | 9 | **7** | Per rule: | rule | before | after | | |---|---:|---:|---| | `discourse-cohesion` | 3 | **1** | intro passage cleared from p1 | +| `discourse-hedging` | 0 | 1 | the revised method states no hedge — see below | | `em-dash` (L0) | 1 | **0** | | | `ing-tail:highlighting` | 1 | **0** | | | `document-uniformity` | 6 | **5** | | | `document-role-decoupling` | 1 | **0** | | | `structure-template` | 1 | 2 | announced enumeration gone, two announced counts remain | | `salience-recital` | 4 | **3** | fell, and the two strong ones changed kind — see below | -| `collocation-novel` | 2 | **3** | rose for the same reason, and for one of its own — see below | +| `collocation-novel` | 3 | 3 | the same count, not the same pairs — see below | ## Nothing was deleted to make a finding go away @@ -63,11 +64,13 @@ that fake a link. ## A coined term keeps its pair `collocation-novel` counts, per sentence, the adjacent common-word pairs that -no passage of the field's corpus has ever written. It went from 2 findings to 3 -because the revision says `blending fraction` and `blending term` — this -synthetic paper's own parameter, a pair the real literature does not use — in -more sentences, and because carrying `bias` forward for cohesion puts it next -to words it had not stood beside before (`bias runs`). That is the axis's stated exception, not a defect to fix: +no passage of the field's corpus has ever written. It reads 3 findings before +and 3 after, and they are not the same three. Both versions are flagged for +this synthetic paper's own parameter — `blending term`, `blending fraction`, a +pair the real literature does not use — and the revision adds one of its own +making: carrying `bias` forward for cohesion puts it next to words it had not +stood beside before (`bias runs`, the one strong collocation finding left). +The coined pair is the axis's stated exception, not a defect to fix: a term this paper defines keeps its pair and gets its definition at first use, and the disposition is recorded as *kept*, never answered by dissolving the pair or changing the claim. @@ -104,13 +107,23 @@ paragraph that writes its numbers inside math looked numeral-free, so any density looked like recital. v0.36.3 calibrates on the numeral projection of each passage (EVALUATION §17.5), and the same revision reads 3. +## A finding the revision earned + +`discourse-hedging` speaks for methods as well as introductions since v0.39.0 +([§19](../docs/architecture/evaluation/discourse-and-citation.md)). The revised +method section states its procedure in 160 words without one epistemic marker, +the 2nd percentile of human method sections, so it is flagged; the original's +is not. A method section certain of what it did may be right to say so: the +finding is an advisory, and whether to act on it is the author's call. + ## The part that barely moved, and why -Document-scale findings go from 8 to 6, and five of the six that remain are +Document-scale findings go from 9 to 7, and five of the seven that remain are `document-uniformity`. Cross-paragraph dispersion is measured over the paragraphs a document has, and -this one has eleven. An eleven-paragraph file cannot demonstrate that its paragraph +the measured parts of these files hold six (original) and ten (revision). A file +that short cannot demonstrate that its paragraph shape varies with rhetorical role, because there is little variation to measure and no room to show a pattern is not chance. The conformal p-values behind these axes are calibrated against complete papers of ordinary length, so a diff --git a/latency.json b/latency.json deleted file mode 100644 index d6ec86d..0000000 --- a/latency.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "validate_plugin": { - "median": 2.2390009999999165, - "min": 2.2119057999998404, - "max": 2.5338272000008146, - "n": 7 - }, - "suite": { - "median": 51.01080809999985, - "min": 50.331643200000144, - "max": 51.9070141000002, - "n": 3 - } -} \ No newline at end of file diff --git a/style-corpus/wgl/README.md b/style-corpus/wgl/README.md index 0c3d90b..394233c 100644 --- a/style-corpus/wgl/README.md +++ b/style-corpus/wgl/README.md @@ -1,8 +1,7 @@ # style-corpus/wgl/ -Weak-gravitational-lensing (WGL) field corpus. This is the only populated -field in v0.1; additional fields will live alongside it as -`style-corpus//` subdirectories (e.g. `cosmology/`, `ml-methods/`). +Weak-gravitational-lensing (WGL) field corpus. Other fields live alongside it +as `style-corpus//` subdirectories (e.g. `cosmology/`, `ml-methods/`). ## Tiers @@ -13,15 +12,16 @@ field in v0.1; additional fields will live alongside it as weak-lensing methodology, cluster lensing, NFW fits, aperture-mass formalism, etc.). Default weight 0.2. -See `../README.md` (one level up) for general corpus rules. +See `../README.md` (one level up) for general corpus rules and the +`fulltext-*` pulls. ## Building this field's profile ```bash python tools/extract_style.py --field wgl -# (or just `python tools/extract_style.py` while wgl is the only field — -# the tool auto-detects single-field corpora.) +# (`--field` may be omitted when one field is present — the tool auto-detects +# a single-field corpus; with several fields it is required.) ``` -Outputs to `style-profile/wgl/{lexicon.json, sentence_stats.json, -transition_inventory.json, style_dossier.md}`. +Its outputs, and every `--calibrate` step after it, are listed in +[`style-profile/README.md`](../../style-profile/README.md). diff --git a/style-profile/README.md b/style-profile/README.md index d8a1657..36d4afb 100644 --- a/style-profile/README.md +++ b/style-profile/README.md @@ -101,28 +101,12 @@ python tools/deai_anchoring.py --field --calibrate python tools/train_voice_model.py --field ``` -> **Recalibrate after every corpus-side fix the CHANGELOG records** (the last -> full rebuild shipped with v0.36.3 on 2026-09-05: 27,851 bank rows, `results` -> 3,240). The v0.28.0 notice below is kept as the record of why. -> -> **Recalibrate after v0.28.0.** Two rounds of corpus-layer defects were fixed -> on 2026-08-25. Section labelling: `classify_section` matched titles in the -> singular only, `method` was itself the default bucket and absorbed every -> unnamed heading, and PDF "paragraphs" were line fragments. Then what counts as -> a paper: `\include` fragments were counted as separate papers, selecting the -> root instead lost the body it includes, the root-selector and the reader -> resolved `\input` targets differently, and a `\subsection` did not inherit -> its `\section` — which sent 54.8% of all section words to `unknown`. The -> 500-paper `fulltext-arxiv/` breadth corpus was also invisible to every -> paragraph-level baseline. -> -> Rebuilding against all of it takes the exemplar bank from 593 to **25,005** -> paragraphs and `results` from 26 to **3,118**, so every bucket clears the -> 30-passage floor and none is rank-only. -> -> **Until you rebuild, treat every section-keyed axis as `degraded`** whatever -> its recorded status says: rerun `build_profile.py` and then each `--calibrate` -> above. `docs/architecture/EVALUATION.md` §2 carries the same notice. +> **Recalibrate after every corpus-side fix the CHANGELOG records, and until +> you do, treat every section-keyed axis as `degraded`** whatever its recorded +> status says: rerun `build_profile.py` and then each `--calibrate` above. The +> last full rebuild shipped with v0.39.0 on 2026-09-27. +> `docs/architecture/EVALUATION.md` §2 carries the same notice, and its §5 +> records how the bank reached its present shape. Snapshot before any rebuild — a recalibration overwrites artifacts that can take hours of GPU time to regenerate: @@ -138,7 +122,11 @@ rules exclude, so it must not drift outside the repository's ignore rules. `deai_policy.json` is not synthesized by the basic builder. Add it only when the sample unit, corpus selection, uncertainty, applicability, operating point, and -validation behavior are documented in the asset and `EVALUATION.md`. For the +validation behavior are documented in the asset and `EVALUATION.md`. It holds +one block per axis: `distribution` (`burstiness_ratio` and `signpost_fraction`, +which replace the compatibility defaults 0.60 and 0.20) and `structure` +(`rare_template_fraction`, at or below which a template counts as rare); a file +written without blocks is read as the block itself. For the `wgl` field this asset is **not** obtainable: EVALUATION §16 measures both statistics it would threshold and finds burstiness reverses sign on adversarial prose while signposting runs below chance, so `L1.distribution` and diff --git a/tools/README.md b/tools/README.md index b308df6..c5108fa 100644 --- a/tools/README.md +++ b/tools/README.md @@ -9,30 +9,30 @@ serialize evidence but do not define an independent paper verdict. | File | Purpose | Calibration / failure behavior | |---|---|---| | `build_profile.py` | Builds the basic descriptive profile, optionally trains the legacy classifier, and warms the exemplar cache. | Aborts on extraction failure; reports optional-stage failures and continues. It does not create calibrated L1/L2/L3 policy. | -| `extract_style.py` | Builds descriptive lexicon, sentence statistics, transition inventory, dossier, and exemplar bank from `.tex`, `.pdf`, or text corpora (`.txt` feeds the statistics under `unknown`, never the bank). Re-exports every public name from `extract_sections.py`. | PDF extraction is unavailable without pymupdf (skipped files are counted in the summary); ligatures are expanded on the PDF path. Section titles match singular and plural; a `.pdf` is accepted only at source-directory depth 1. Statistics count prose without projection placeholders, words are Unicode letters, and a citation-led paragraph stays in the bank. The curated tiers carry every aggregate and the dossier (their 0.5/0.3/0.2 weights are recorded metadata; the aggregates pool the tiers unweighted); `REFERENCE_DIR` (`fulltext-arxiv/`) is gathered **unweighted** and feeds the exemplar bank only, so breadth cannot restyle the imitation target. Corpus observations do not redefine policy. | +| `extract_style.py` | Builds descriptive lexicon, sentence statistics, transition inventory, dossier, and exemplar bank from `.tex`, `.pdf`, or text corpora (`.txt` feeds the statistics under `unknown`, never the bank). Re-exports every public name from `extract_sections.py`. | PDF extraction is unavailable without pymupdf (skipped files are counted in the run summary and on the dossier's "Built from" line); ligatures are expanded on the PDF path. Section titles match singular and plural; a `.pdf` is accepted only at source-directory depth 1. Statistics count prose without projection placeholders, words are Unicode letters, and a citation-led paragraph stays in the bank. The curated tiers carry every aggregate and the dossier (their 0.5/0.3/0.2 weights are recorded metadata; the aggregates pool the tiers unweighted); `REFERENCE_DIR` (`fulltext-arxiv/`) is gathered **unweighted** and feeds the exemplar bank only, so breadth cannot restyle the imitation target. Corpus observations do not redefine policy. | | `extract_sections.py` | Source-text projection, section splitting, and corpus document assembly: the section vocabulary and `classify_section`, both named LaTeX projections, the PDF heading heuristic, and `select_document_roots` / `read_tex_document` / `corpus_documents`. Split out of `extract_style.py` on 2026-08-25. | Section buckets key every per-section reference distribution, so a change here requires a profile rebuild. `method` has explicit rules and the default is `unknown`: before 2026-08-25 `method` WAS the default and absorbed every unmatched heading (1,671 "method" passages against 10 "results"). The ALL-CAPS PDF branch requires >= 2 words, >= 4 letters and >= 75% letters, rejecting the table cells ("S", "RA", "NFW", "S/N") that were 305 of 325 detected headings. **A paper is a document, not a file**: `\include`/`\input` fragments are folded back into their root (one review shipped as 12 files), a `\subsection` inherits its `\section` (54.8% of section words previously fell to `unknown`), and a bundle contributes one paper. | | `tex_assembly.py` | Document assembly shared by every reader: `\input`/`\include` children spliced in place (mid-line calls keep their neighbours, repeats splice again, cycles stop), from disk or from a git ref, a target read as written when it has a suffix and resolved against the document root's directory first. | An unreadable ROOT raises (no reader may take it for an empty document); an unresolvable or unreadable child is left in place as the literal call; a git ref without the root raises `ValueError`, and the git reader resolves children at the ref too. | | `tex_macros.py` | Expands numeric-literal `\newcommand` / `\def` macros once on the assembled document, so a quantity written as a macro reaches the projections that count numerals; owns the LaTeX comment pattern. | Conservative: a symbolic body, an argument-taking macro and a definition inside a comment are left exactly as written; the live definition is dropped so its digits are not counted in the preamble. | | `retrieve_exemplars.py` | Retrieves section- and topic-matched exemplar paragraphs. | Uses sentence-transformer cosine when installed; keyword fallback requires `--allow-fallback`. Reads the **curated tiers only** by default (`--include-reference` widens to the breadth corpus): the bank serves both imitation and reference-distribution roles, and only the first is what retrieval wants. Tier filtering happens after the embedding lookup, because the `.npy` cache is positional against the full bank. | -| `train_ai_ism_classifier.py` | Trains the legacy word-ngram paragraph classifier. | Output is degraded L3 advisory evidence, not authorship evidence or an L0 gate. Cross-validation is grouped by source paper (`StratifiedGroupKFold`) because paragraphs from one paper are not independent; `cv_grouped` travels with the metrics (ungrouped F1 0.876 vs grouped 0.823 on `wgl`). | +| `train_ai_ism_classifier.py` | Trains the legacy word-ngram paragraph classifier. | Output is degraded L3 advisory evidence, not authorship evidence or an L0 gate. Cross-validation is grouped by source paper (`StratifiedGroupKFold`) because paragraphs from one paper are not independent; `cv_grouped` travels with the metrics (on `wgl` grouped F1 read 0.823 against 0.876 ungrouped when the grouping was introduced, and 0.822 ± 0.116 on the 2026-09-27 rebuild). | | `extract_md_negatives.py` | Harvests candidate negative paragraphs from a document tree. | Generated data are privacy-sensitive and remain profile-local/gitignored. | | `ai_ism_negatives_handcrafted.txt` | Seed negative examples for the legacy classifier. | Data asset, not normative prose policy. | | `deai_feedback.py` | Implements `sci-paper.feedback.v1`: stable IDs, consequence classes, measurement states, dispositions, ranking, summaries, text/JSON rendering, and tuple compatibility. | Stdlib-only shared contract. | | `ai_ism_lint.py` | Unified L0 and advisory CLI. Emits ranked text or JSON from the same findings; every L0–L2 axis is on by default (`--no-` disables one), `--oracle`, `--voice` and `--glossary` are opt-in. | Exit 0 = no L0 target; 1 = L0 target present; 2 = invalid input/configuration/execution (an unreadable root, an unparseable lexicon). Advisories never cause exit 1. A paragraph-initial connector is judged on the paragraph's first prose line and an opener where a sentence starts, never per physical line; command arguments (`\label`, `\ref`, `\cite`, `\url`) are blanked before the lexical scan; the Tier B cap is per heading unit and per word. | | `length_gate.py` | Per-section rendered-prose length-budget delta gate between two document versions (standard §5.3). Growing sections emit strong advisories; `--allow` records justifications; `--require-shrink ` demands a net cut of at least that much (0 = no cut required, so a map with nothing to cut still closes). | Exit 0 = net unjustified growth within tolerance and any shrink target met; 1 = growth beyond it or a cut short of the target (strong `length-shrink-short`); 2 = invalid input/execution, including a document or baseline that cannot be read (a mistyped `--before` once read as a zero-word baseline and every section as growth). Both versions are assembled documents, the `--git-ref` baseline with its `\input` children resolved at the ref. Comments (blanked before segmenting, so a commented-out heading opens no section) and mathematics in every display form are excluded; a pure section rename nets to zero. | | `verify_references.py` | Bibliography verification against CrossRef (DataCite fallback for the DOI prefixes CrossRef does not register) and the arXiv API: DOI resolvability (from the `doi` field or a doi.org `url`) and first-author / year / title / journal / volume / first-page agreement per entry, `@string` names expanded, plus `--tex` cross-checking of the assembled document's citation keys (LaTeX, natbib and biblatex commands including `\cites{a}{b}`; `\nocite{*}` cites every entry; a citation inside a `%` comment is text) against the entries. `--cache` keeps fetched records between rounds; a registry miss is never cached, and a finding built from a cached record says so (`observed.cached`). | Exit 0 = no blocker; 1 = unresolvable identifier or cited key without an entry; 2 = invalid input or execution failure (malformed BibTeX names the entry and line). Title match tolerates the CrossRef title/subtitle split; a one-year print/online seam is ordinary, a larger gap strong; a first author agrees when the family names are equal or share a part of four or more letters. A registry outage or an answer of unexpected shape marks the entry unmeasured and the axis `degraded`, never clean; an entry with no identifier is unmeasured. Relevance (whether a citation supports its sentence) is not measured here; that half of dimension F stays with the reviewer. | -| `condense_map.py` | The removal map behind `/sci-paper:condense`: six scans — restatement (with the canonical home), zero-gain sentences, dead figure/table/label/macro/acronym, verbose constructions, repeated symbol glosses, duplicated paragraphs — each entry with `removable_words`, totalled into `condense_budget` with a default target that excludes the abstract/conclusion carve-out. | Deletes nothing and emits no blocker; exit 0 on a measured map, 2 on invalid input. Every word count is the gate's rendered-prose count (placeholders and heading words excluded), and the abstract/conclusion carve-out applies to either side of a restatement pair. Held-out refereed papers carry a median default target of 1.59% of prose (p90 4.11%, EVALUATION §23.5), so a pass that cuts less than that has not found what the map lists. | -| `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan shipped documentation for edit-meta literals. The literal rule reads the visible lines of the whole paper minus preamble and bibliography (headings, captions and appendices included; an appendix `TODO` was invisible after v0.36.2 and the gate exited 0; a `\newcommand{\TODO}` in a preamble is not prose) and the label rule looks for the object in that same span, blanked to prose; strengths were set on held-out refereed papers: any strong finding in 13.3% of them, the static negative-label rule ordinary, the strong absence forms at 0.008 per 1,000 words (EVALUATION §23.4). `\section {X}` is a label; `-ation/-ated/-ating` forms share one root; an unwritable `--output` is exit 2 rather than the gate's exit 1 (2026-09-27). | +| `condense_map.py` | The removal map behind `/sci-paper:condense`: six scans — restatement (with the canonical home), zero-gain sentences, dead figure/table/label/macro/acronym, verbose constructions, repeated symbol glosses, duplicated paragraphs — each entry with `removable_words`, totalled into `condense_budget` with a default target that excludes the abstract/conclusion carve-out. | Deletes nothing and emits no blocker; exit 0 on a measured map, 2 on invalid input. Every word count is the gate's rendered-prose count (placeholders and heading words excluded), and the abstract/conclusion carve-out applies to either side of a restatement pair. Held-out refereed papers carry a median default target of 1.22% of prose (p90 3.39%, EVALUATION §23.5), so a pass that cuts less than that has not found what the map lists. | +| `deai_residue.py` | L4 residue: self-referential drafting history (`residue-self-history:`, first-person sentence without a citation), a sentence defining the paper's own object by what it never does or has (`residue-absence`; `never` strong, `carries no` ordinary, a citation in the sentence exempts it), edit-meta literals (`residue-edit-meta`), a heading or caption whose object the body never names (`residue-negative-label`), and with `--before`/`--git-ref` the added-label diff rule. | Deterministic, no profile needed. Exit 0 = no strong finding, 1 = a strong finding (the third narrow exit contract), 2 = invalid input. The history and absence families live once in the tool and are mirrored between markers in `skills/paper/SKILL.md`; `validate_plugin` runs the tool's `validator_check` to prove the mirror and to scan both READMEs, the standard and every skill for edit-meta literals. The literal rule reads the visible lines of the whole paper minus preamble and bibliography (headings, captions and appendices included; an appendix `TODO` was invisible after v0.36.2 and the gate exited 0; a `\newcommand{\TODO}` in a preamble is not prose) and the label rule looks for the object in that same span, blanked to prose; strengths were set on held-out refereed papers: the static negative-label rule ordinary, the absence forms strong only where refereed prose rarely writes them (EVALUATION §23.4). `\section {X}` is a label; `-ation/-ated/-ating` forms share one root; an unwritable `--output` is exit 2 rather than the gate's exit 1 (2026-09-27). | | `deai_metrics.py` | L1 information-distribution analysis: sentence-length variation and connective-openers. | Strong status requires applicable policy calibration; compatibility heuristics are degraded. The reference rate is computed over `CONNECTIVE_OPENERS` from the profile's `paragraph_initial_counts`, i.e. the same set the draft is measured against; a profile predating that key omits the reference clause rather than quoting an incomparable one. A `sentence_stats.json` with no classified bucket (only `unknown`) leaves the axis `degraded`: its pooled CV is 0 and the burstiness rule cannot fire. | | `deai_structure.py` | L2 sentence-template analysis: enumeration, ordinal/modal/parallel runs, setup-list-wrap-up patterns, and balanced closers; plus the auxiliary families (antithesis, reversal beat, paper-as-agent, wh-cleft, modifier stack) reported under `structure-auxiliary`. | Strong status requires calibrated policy and sufficient reference sample. Auxiliary families never enter `template_score`; their per-bucket human fractions are written by `--calibrate` (`paper_agent_frac`, `wh_cleft_frac`, `modifier_stack_frac`). A template finding is `measured` only against a reference for its own bucket plus a policy, `degraded` when the baseline lacks the bucket, `unmeasured` without one; the auxiliary message quotes the reference fraction and calls a figure rare only at or below the policy gate. The wrap-up beat requires a sentence opening `These N ` closed by a summing verb (define/form/constitute/together/…). | -| `deai_salience.py` | L2 salience hierarchy: how far a passage's numerals run without an interpreting sentence between them, plus density and per-sentence numeral count, against a per-section human passage reference. | Sole consumer of `extract_style.latex_to_numeral_text` (the numeral-preserving projection; `latex_to_plain` zeroes every numeral on `.tex` input), and it calibrates on the bank's `numeral_text` field, the same projection of each paragraph, so the reference holds the numerals the manuscript side counts. Until v0.36.3 it calibrated on `text`, which holds none inside math, and the p90 gate fired at 0.45 per held-out passage against a 0.27 bound; it now fires at 0.20 (EVALUATION §17.5). Percentiles are read as P(X <= x) on a 0.01 grid at the top of a tie plateau. Buckets under 30 reference passages are degraded; the axis is `measured` only where a bucket both clears that floor and has spread above the gate (a flat reference read `measured` with zero findings until 2026-09-27), and the status counts the document's `unknown` units. As of the v0.36.3 profile rebuild every bucket clears that floor (smallest: `conclusion` 1,999). One finding per passage, led by its most extreme feature. `--calibrate` records `text_fallback_rows` per bucket and warns on stderr when a bank row lacked `numeral_text`, instead of calibrating silently on the `[math]` projection (2026-09-27). | -| `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- `AUC` (df 1) must separate from `epoch` (402) and `accuracy` (774). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the manuscript supplies a reason the tool can read (a formation of an attested stem, a defining sentence or acronym expansion at first use, a proper name capitalised at every use); it is exhaustive advice, not a detector (rank AUC 0.174, EVALUATION §23.1). `\newcommand*` definitions are read; calibrating on no passage writes no lexicon and exits 2 (2026-09-27). | -| `deai_collocation.py` | L2 collocation: per sentence, the fraction of distinct adjacent common-word pairs that no corpus passage attests, against a leave-one-out per-bucket reference at sentence unit (p90 advisory, p95 strong). Each flagged pair carries λ = df(a)·df(b)/N and e^−λ so the reader can weigh it. `--glossary` reads the same bank document-wide: every unattested pair used at least twice, with its first-use line and whether that sentence defines it, as the list of the manuscript's own terms to define at first use or replace (2026-09-16). | `--calibrate` writes `collocation_bank.json` (unigram and pair document frequency; 11,282 partner words and 530,504 pairs over 41,644 `wgl` passages) and `collocation_baseline.json`. Unmeasured without the bank; degraded where no bucket clears the 30-sentence floor. Pairs break at punctuation, `[math]`/`[CITE]` and dashes; a sentence needs four judged pairs to carry a fraction. The action names the relation a pair compresses and never changes a claim; a defined term keeps its pair. Since 2026-09-27 `--glossary` is an argparse option (refused beside `--calibrate`, exit 2); a definition cue counts only in the eight words after the pair or a parenthesis opening right after it; `uses` counts every occurrence; the glossary status depends on the bank alone. Sentence findings carry the sentence's own lines. Calibrating a field with no passage writes nothing and exits 2, and the axis status is degraded, with counts, when units fall in `unknown` or a bucket's artifact records another unit. | -| `deai_discourse.py` | L2 discourse texture, two axes on the LOW tail (p10 advisory, p05 strong): `cohesion`, the mean fraction of a sentence's content words that already appeared in the sentence before it, and `hedging`, epistemic markers per 1,000 words. | The defect here is absence, not excess, so both flag below the gate. The two run at **different units and each artifact records its own**, and the reader refuses a bucket recorded at the other unit: cohesion per paragraph, hedging per SECTION, because hedging has no paragraph-scale lower tail at all (p10 is 0.000 in all seven `wgl` buckets). Hedging ships restricted to `intro`, where held-out transfer is 7.89% against 15.48-26.77% elsewhere and every generation regime separates (EVALUATION section 19). Advisories against the field's distribution, never a provenance claim. A manuscript section is one unit per bucket, its subsections pooled, as the reference pools a paper's paragraphs. The status takes the text: a document whose units sit in `unknown` is degraded with the count, never `measured` with nothing read (2026-09-27). | +| `deai_salience.py` | L2 salience hierarchy: how far a passage's numerals run without an interpreting sentence between them, plus density and per-sentence numeral count, against a per-section human passage reference. | The only detector that reads `extract_sections.latex_to_numeral_text` (the numeral-preserving projection; `latex_to_plain` zeroes every numeral on `.tex` input; `rewrite_reward` also calls it, and the exemplar bank stores each paragraph under the same projection through `extract_style.paired_paragraphs`), and it calibrates on the bank's `numeral_text` field, the same projection of each paragraph, so the reference holds the numerals the manuscript side counts. Until v0.36.3 it calibrated on `text`, which holds none inside math (EVALUATION §17.5). Percentiles are read as P(X <= x) on a 0.01 grid at the top of a tie plateau. Buckets under 30 reference passages are degraded; the axis is `measured` only where a bucket both clears that floor and has spread above the gate (a flat reference read `measured` with zero findings until 2026-09-27), and the status counts the document's `unknown` units. Every `wgl` bucket clears that floor (EVALUATION §14.2). One finding per passage, led by its most extreme feature. `--calibrate` records `text_fallback_rows` per bucket and warns on stderr when a bank row lacked `numeral_text`, instead of calibrating silently on the `[math]` projection (2026-09-27). | +| `deai_register.py` | L0 domain register: terms the manuscript leans on (>= 15 uses) whose document frequency in the field's own corpus is below 1e-4. | Corpus frequency, never a curated cross-discipline list -- a list cannot tell a term the field rarely writes (`AUC`) from field words that also sit on any machine-learning list (`epoch`, `accuracy`; EVALUATION §14.4). Compounds are judged by their rarest part; `\mathrm{}` after `_`/`^` is a subscript, not a term; possessives fold. Advisories only, never `l0_target`s. The floor was 5 until v0.32.0, when a sweep against 203 held-out refereed papers put it at 15 (EVALUATION section 18.4). Corpora under 500 passages are silent; one that cannot resolve the 1e-4 rate is degraded, and a `-` profile borrows ``'s bank and names it. The zero-hit audit (`register-zero:`) lists every body word with df 0 — headings, floats, comments, preamble and bibliography excluded on both sides of the comparison — strong unless the manuscript supplies a reason the tool can read (a formation of an attested stem, a defining sentence or acronym expansion at first use, a proper name capitalised at every use); it is exhaustive advice, not a detector (EVALUATION §23.1). `\newcommand*` definitions are read; calibrating on no passage writes no lexicon and exits 2 (2026-09-27). | +| `deai_collocation.py` | L2 collocation: per sentence, the fraction of distinct adjacent common-word pairs that no corpus passage attests, against a leave-one-out per-bucket reference at sentence unit (p90 advisory, p95 strong). Each flagged pair carries λ = df(a)·df(b)/N and e^−λ so the reader can weigh it. `--glossary` reads the same bank document-wide: every unattested pair used at least twice, with its first-use line and whether that sentence defines it, as the list of the manuscript's own terms to define at first use or replace (2026-09-16). | `--calibrate` writes `collocation_bank.json` (unigram and pair document frequency over the field's passage banks) and `collocation_baseline.json`. Unmeasured without the bank; degraded where no bucket clears the 30-sentence floor. Pairs break at punctuation, `[math]`/`[CITE]`, dashes, slashes, periods and digits; a sentence needs four judged pairs to carry a fraction. The action names the relation a pair compresses and never changes a claim; a defined term keeps its pair. Since 2026-09-27 `--glossary` is an argparse option (refused beside `--calibrate`, exit 2); a definition cue counts only in the eight words after the pair or a parenthesis opening right after it; `uses` counts every occurrence; the glossary status depends on the bank alone. Sentence findings carry the sentence's own lines. Calibrating a field with no passage writes nothing and exits 2, and the axis status is degraded, with counts, when units fall in `unknown` or a bucket's artifact records another unit. | +| `deai_discourse.py` | L2 discourse texture, two axes on the LOW tail (p10 advisory, p05 strong): `cohesion`, the mean fraction of a sentence's content words that already appeared in the sentence before it, and `hedging`, epistemic markers per 1,000 words. | The defect here is absence, not excess, so both flag below the gate. The two run at **different units and each artifact records its own**, and the reader refuses a bucket recorded at the other unit: cohesion per paragraph, hedging per SECTION, because hedging has no paragraph-scale lower tail at all (EVALUATION section 19.1). Hedging ships restricted to `intro` and `method`, the buckets where its gate transfers to held-out papers and every generation regime separates (EVALUATION section 19). Advisories against the field's distribution, never a provenance claim. A manuscript section is one unit per bucket, its subsections pooled, as the reference pools a paper's paragraphs. The status takes the text: a document whose units sit in `unknown` is degraded with the count, never `measured` with nothing read (2026-09-27). | | `deai_reference.py` | Library, no CLI. One `(feature, unit)` percentile reference shared by every per-bucket axis: the 0.01 quantile grid at nearest rank (the stored p_q is the ceil(q·n)-th smallest value; until 2026-09-27 it was the floor(q·n)-th plus one, one order statistic high), the plateau-top percentile reader, the 30-unit sample floor, the paragraph and section sweeps, the artifact loader, and the calibration loop. | Holds no policy -- gate quantiles, feature names and what a finding means belong to the axis. Its one invariant is that calibration and detection share one unit and one grid; that check is what caught the hedging axis calibrating where no lower tail exists. The artifact loader had five copies across the suite before this module (EVALUATION section 19.1). The invariant is checked, not only declared: `usable_buckets` refuses a bucket whose artifact records another `unit` and the status names it; `sections()` pools a bucket's headings into one unit as the reference side pools a paper; `unbucketed_reason` counts the units a document leaves in `unknown`; `sentence_lines` anchors a sentence-scope finding on its own lines; each bucket records `text_key` and `text_fallback_rows`; `calibrate` returns None (exit 2 in `axis_main`) when no record contributed. Artifacts calibrated before 2026-09-27 keep the old grid (the 30-unit floor read its tails at P = 0.133 and 0.933) until `--calibrate` is rerun. | -| `deai_docshape.py` | Document-shape measurement and complete-document calibration: the per-paragraph model-free feature vector, cross-paragraph dispersion, the joint Mahalanobis manifold (pooled and per-length-stratum), role coupling, the split-conformal operating points, and the baseline builder. | Split out of `deai_docstructure.py` on 2026-08-25 at 1,092 lines against a 750-line budget; the seam is measurement versus feedback. `deai_docstructure._paper_documents` assembles a bundle through `es.corpus_documents` in `\include` order — sorted-filename concatenation put `Conclusion.tex` before `Introduction.tex` in 12 of the 500 `wgl` bundles, on the one axis that measures section arc. `document_shape` sweeps `deai_metrics.section_units` and drops the preamble and every `skip` unit (acknowledgements, appendices, references) on both sides; `shape_units`/`measurable_sections` are the one section/paragraph filter `deai_partition` scores with. | -| `deai_docstructure.py` | L2 whole-document rhetorical-shape findings and the CLI: turns the measurements above into `sci-paper.feedback.v1` findings. Re-exports every public name from `deai_docshape.py`. | Requires at least three measurable complete documents and sufficient sections/paragraphs; otherwise unmeasured. Legacy baselines without a conformal block fall back to percentile thresholds; factor-drifted role references disable that axis. A document-shape metric is strong when it exceeds the interpolated quantile quoted as `strong_threshold` (the rule the leave-one-document-out rate uses). `--strong-percentile` must lie strictly in (0, 1) (exit 2); `--field` auto-resolves on the read path and must name an existing profile for `--calibrate`. | -| `deai_partition.py` | Fidelity-free partition suggestions (merge/split of paragraph blocks) that move a document toward the calibrated human dispersion band. Suggest-only; the author applies changes by hand. | Zero-token operations, so rewrite fidelity holds by construction; cohesion is self-normalized to the document's median adjacent-pair overlap; heading-command blocks (`\section`, `\subsection`, `\chapter`, …) are never candidates; reordering is deliberately not offered. Without a calibrated manifold the tool exits with an explicit message. States are ranked by conformal p (higher = deeper inside the band), distance only as a tie-break within one calibration basis; every plan step records `calibration_basis_before/after`. A split is simulated on the original text at the sentence boundary, line breaks kept; scored blocks pass the same filter as `document_shape`. | +| `deai_docshape.py` | Document-shape measurement and complete-document calibration: the per-paragraph model-free feature vector, cross-paragraph dispersion, the joint Mahalanobis manifold (pooled and per-length-stratum), role coupling, the split-conformal operating points, and the baseline builder. | Split out of `deai_docstructure.py` on 2026-08-25 at 1,092 lines against a 750-line budget; the seam is measurement versus feedback. `deai_docstructure._paper_documents` assembles a bundle through `es.corpus_documents` in `\include` order — sorted-filename concatenation put `Conclusion.tex` before `Introduction.tex` in 12 of the 500 `wgl` bundles, on the one axis that measures section arc. `document_shape` sweeps `deai_metrics.section_units` and drops the preamble and every `skip` unit (acknowledgements, appendices, references) on both sides; `shape_units` blanks headings and floats before it cuts paragraphs, and it and `measurable_sections` are the one section/paragraph filter `deai_partition` scores with; the partition keeps what the blanking took out as fixed, unscored blocks. | +| `deai_docstructure.py` | L2 whole-document rhetorical-shape findings and the CLI: turns the measurements above into `sci-paper.feedback.v1` findings. Re-exports every public name from `deai_docshape.py`. | Requires at least three measurable complete documents and sufficient sections/paragraphs; otherwise unmeasured. A baseline whose manifold carries no split-conformal calibration leaves the manifold rule unscored and reports the axis `degraded` with a rebuild reason; role coupling without a conformal block falls back to its in-sample percentile threshold, and a factor-drifted role reference disables that rule. A document-shape metric is strong when it exceeds the interpolated quantile quoted as `strong_threshold` (the rule the leave-one-document-out rate uses). `--strong-percentile` must lie strictly in (0, 1) (exit 2); `--field` auto-resolves on the read path and must name an existing profile for `--calibrate`. | +| `deai_partition.py` | Fidelity-free partition suggestions (merge/split of paragraph blocks) that move a document toward the calibrated human dispersion band. Suggest-only; the author applies changes by hand. | Zero-token operations, so rewrite fidelity holds by construction; cohesion is self-normalized to the document's median adjacent-pair overlap; heading-command blocks (`\section`, `\subsection`, `\chapter`, …) are never candidates; reordering is deliberately not offered. Without a calibrated manifold the tool exits with an explicit message; a manifold with no split-conformal calibration gets the reason `deai_docstructure` gives (rebuild with `--calibrate`), status `unmeasured`, exit 0. States are ranked by conformal p (higher = deeper inside the band), distance only as a tie-break within one calibration basis; every plan step records `calibration_basis_before/after`. A split is simulated on the original text at the sentence boundary, line breaks kept; scored blocks pass the same filter as `document_shape`. | | `deai_anchoring.py` | L2 claim-anchoring band: section-class conditional anchored-sentence rates against the human corpus, low-tail conformal with a Bonferroni share per class. | A writing-quality axis, not an AI-discrimination axis (EVALUATION.md 9.6 records the refuted tell). Classes below the 30-document minimum are honestly omitted; unmeasurable documents degrade to unmeasured. A class whose calibration n cannot reach its Bonferroni share (1/(n+1) > alpha/k) is skipped and the axis reported `degraded`; `--calibrate` records `underpowered_classes` and warns. The scan runs on section units with headings, floats and the bibliography blanked and `skip` units dropped. | | `deai_provenance.py` | Editing-provenance ledger (frontier idea 4): matches each current paragraph to a designated AI-draft ancestor (file or git ref from the author's own history) and labels the span AI-untouched / lightly-edited / rewritten / author-original by a deterministic difflib token edit ratio. | Not an AI detector; reads only the author's own history. `unmeasured` without an ancestor or when the file has no git history for the given ref (never a guess). Paragraphs come from the shared section sweep, so a title block is not a ledger paragraph (2026-09-27). | | `deai_personal.py` | Personal dispersion baseline (frontier idea 6): places a draft's per-feature within-document dispersion in the distribution of the author's own prior papers, a confound-free same-author reference. | Model-free; reuses `document_shape`. `unmeasured` below three measured prior papers. Flags a draft that varies paragraph shape far less than the author's own papers on at least 40% of features. | @@ -41,14 +41,14 @@ serialize evidence but do not define an independent paper verdict. | `deai_voice.py` | Optional learned field-similarity triage. | Refuses bundles with drifted feature names/schema and degrades on corrupt bundles. A measured operating point gates threshold findings; an uncalibrated bundle yields rank-based triage of the lowest-scoring paragraphs, never a universal 0.5 cutoff. Without the optional transformers/torch runtime the axis is `unmeasured` with that reason (the feature vector needs token surprisal); `--scores` prints the status line and exits 0, and lists paragraphs by source line otherwise; `--field` resolves through the shared resolver (2026-09-27). | | `train_voice_model.py` | Trains the optional learned field-similarity/rewrite-ranking model and writes `voice_model_evaluation.json`. | Cache reuse requires a content/model/centroid fingerprint; featurization checkpoints every 500 rows for preemptible runs. Grouped splits recompute `corpus_cos` from training-only centroids; repeated source-group splits audit source, section, length, mathematical-marker, field-term, matched-stratum, and UID-normalization sensitivity, plus the author-labelled hard-set stratum, without creating an operating point. Requires `--field` to resolve (auto-detected when one profile exists); a missing numpy/scikit-learn/joblib is one line and exit 2. Repeated audit splits start one seed past the primary split. | | `voice_dataset.py` | Record loading, source-family grouping, the training-only field lexicon, and the fingerprinted feature-matrix cache for the voice model. Split out of `train_voice_model.py` on 2026-08-26 (1,174 lines against a 750-line budget). | Depends on nothing in `voice_audit`; the split is one-way and a test asserts it. The split's hand-written import block left `time` and `CHECKPOINT_EVERY` unbound and the suite stayed green, because no test reached `build_features` — a scope-aware symbol-table check for unbound names now runs in `test_train_voice_model.py`. A run whose encoder fails while sentence-transformers imports never becomes the cache: the language-model rows are kept as a finished checkpoint and the embedder is retried next run. | -| `voice_audit.py` | Held-out metrics, bootstrap AUC intervals, the section-normalized UID alternative, the author hard set, and the repeated grouped confound audits. Owns `HARDSET_AI_CATEGORIES` / `HARDSET_HUMAN_CATEGORIES`. | Fits no model and creates no operating point: it produces the evidence a bundle needs before it can leave `degraded`. `train_voice_model` re-exports every public name, guarded by a contract test. | +| `voice_audit.py` | Held-out metrics, bootstrap AUC intervals, the section-normalized UID alternative, the author hard set, and the repeated grouped confound audits. Owns `HARDSET_AI_CATEGORIES` / `HARDSET_HUMAN_CATEGORIES`. | Fits nothing that ships (`repeated_group_audit` fits two audit-only regressions per split and keeps their held-out scores) and creates no operating point: it produces the evidence a bundle needs before it can leave `degraded`. `train_voice_model` re-exports every public name, and a contract test forbids it re-exporting a private helper. | | `label_findings.py` | Samples findings from all five finding-emitting axes — register, salience, cohesion, hedging, collocation — into a human-labelling sheet (`sample`), re-serves a blind subset with the prior answers stripped (`relabel`), and reports precision, recall and intra-rater agreement (`score`). | The one roadmap item computation cannot close: these axes are corpus-referenced, so nothing here says whether a human agrees a given advisory is right. Built to the scheme chosen 2026-08-26 — finding-level labels, one labeller plus a blind re-label subset, and populations named with `--population NAME=DIR` so each is sampled and reported separately. Recall is **pooled over the axes**: a control row asks whether a passage should have been flagged, and a labeller who says yes has not said by which axis, so a per-axis recall would charge every axis with every other axis's misses. Calibrates nothing; any stratum under 20 labels reports `unmeasured` rather than a rate, and the intra-rater kappa is printed as the ceiling no axis can be held above. `--n` is the flagged-row quota per (population, axis) cell (default 20, the floor); every finding is collected first and each cell drawn at random with a per-document cap, the summary printing how many documents each cell came from; emitter failures are printed per axis rather than read as an exhausted corpus; pooled recall counts a passage several axes flagged once, by `(source, start_line, end_line)`. | | `eval_findings.py` | Scores the register (thresholded and zero-hit), salience and collocation axes against provenance labels instead of hand labels (collocation on its document novel-pair fraction, since its reference is per sentence): firing rate on **held-out** refereed ApJ/ApJL/A&A papers, on the same-genre in-sample papers (leakage control), and on the `docval` machine tiers, plus the machine-over-held-out rank AUC. | Read the two rows differently. Register is an absolute rarity test, so its held-out rate **is** a false-positive rate. Salience is a percentile gate (`ADVISORY_PERCENTILE` 0.90), so a non-zero rate is the design point and the number tests calibration transfer, not defectiveness. Neither is precision for the *advice*: publication does not mean every sentence is beyond improvement, so the register figure is an upper bound. Recall stays `unmeasured` — that is `label_findings.py`. A population with no body words reports `unmeasured` rather than a rate; the 20-document floor is `eval_docscale.MIN_DOCUMENTS`. | | `eval_docscale.py` | Reproduces the EVALUATION §9 document-scale table — human false-flag rate, per-tier tail power, and rank AUC — by scoring the corpus and every tier found under `docval/` through `manifold_operating_point`; each row names the operating point its flags were decided by and reports `unmeasured` under 20 scored documents (the floor shared with `eval_findings`). | The human row is **in-sample** (it includes the manifold's own train and calibration documents) and is labelled as such; the distribution-free guarantee is the split-conformal alpha, not this rate. Point estimates carry substantial seed variance (§9.4c), so a single run is not a stable figure — compare configurations, not runs. | | `cli_common.py` | Shared command-line preamble and field resolution: UTF-8 stdout, the `--field` / `--profile-root` / `--corpus-root` options, `resolve_field` for a tool that needs a field, `optional_field_dir` for one that can run without (one profile auto-resolves, several print a note and run without one), and `axis_main` for the calibrated axes (`--calibrate` needs an existing field and refuses to write an empty artifact; an axis flag goes through `extra_arguments`). | Holds no policy — decides no default beyond the two roots, reads no profile, emits no findings. **21 of 30 tools were retrofitted on 2026-08-26**; the sweep is recorded with its allowlist and refusals. One divergence is kept deliberately: `ai_ism_lint` returns `None` instead of raising (warning on an explicit field that does not exist), because its L0 pass must still run with no profile present; `extract_style` resolves against the corpus root where `tier-*` is not a field (expressed as `exclude_prefixes`, not a second implementation). | | `rewrite_reward.py` | Evaluates rewrite candidates after hard scientific-fidelity eligibility. | Bidirectional: dropping OR inventing protected numbers, units, citations, math, acronyms, semantic LaTeX macros, comparison direction, negation, or causal direction makes a candidate ineligible (`-inf`). Numbers and the three marker categories are compared as bags (a dropped second `not` is a loss; the report names count mismatches); a marker moved between clauses is the stated residual limit and belongs to the by-hand check. Ranking is led by L0 advisory reduction (signed, fidelity-floored) and semantic fidelity; the learned score is tie-break weight unless its bundle is measured. Its numeral tokenizer has been corrected six times, five of them for one root cause — a neighbour absorbed into the token, or the token started inside a neighbour: the Oxford comma, the spaced unit, (2026-08-25) a hyphen between numerals, which made `0.5-1.2` yield `-1.2`, and (2026-09-27) the LaTeX en-dash `0.5--1.2`, the Unicode minus and exponent sign, and identifiers (`M200` yielded `00`). Each is pinned by a test. `--field` is optional: one field auto-detects, none or several run without a profile. A missing profile, a missing or unusable `voice_model.joblib`, or a missing sentence-transformers leaves that term unmeasured at weight 0 with one stderr line, never exit 2. Exit 0 = at least one eligible candidate; 1 = every candidate ineligible (a measured outcome: preserve the original and regenerate tighter); 2 = invalid input or a crash. | | `fetch_arxiv_abstracts.py` | Fetches dated arXiv corpora, abstracts or complete LaTeX sources, for controlled evaluation and calibration. The built-in query sets, keyword filter and journal table are `[WGL]` (`--query-set wl` narrows to the weak-lensing subfield; `--journals apj,apjl,aa` keeps only those refereed venues); any other field passes its own `--query` terms, which replace them. `--fulltext --author "Surname" --author-is --max-authors N` builds a per-author full-text population instead. | Network failures are explicit; fetched text is evidence/training data, not policy. The author path keeps the search term and the identity test separate, because `au:` matches a **surname** and a surname is not a person — `au:"Dell'Antonio"` returns an astronomer and a mathematical physicist. The abstract sweep's `AUTHOR_QUERIES` have no such filter, so each of the ten was checked once against the names it really returns: nine resolve to one person, and `au:"Schneider, P"` returned Donald P. Schneider (SDSS quasars) on 58 papers against the intended Peter Schneider's 33, so that entry now spells the given name out. That check needs the network and cannot be a test; adding an eleventh query means repeating it by hand. `--author` refuses to write into the calibration-breadth directory: a paper in a calibration bank has its own vocabulary in the register denominator, so the axis would go quiet on exactly the prose it was fetched to judge (1 finding across three in-calibration papers against 8 across fifteen held out). Team size is read from the API's author list, never from counting `\author` in the LaTeX, which picks up template placeholder lines. Journal selection is client-side against observed `journal_ref` strings because the API's `jr:` prefix is a loose token match that returns "Research in Astronomy and Astrophysics" for an A&A query. HTTP 429 escalates through a backoff schedule and then stops the sweep with exit 2 and a TRUNCATED report; `--resume` extends a cut-short corpus instead of truncating it, and a journal/subfield run must pass `--out-name` so the broad corpus cannot be clobbered. `--date-hi` bounds the v1 submission, not the returned abstract, which is always the latest version's; `--updated-before` is the only control that dates the text, and a record with no `updated` is dropped rather than assumed clean. Since 2026-09-27 the flags hold in both modes: `--journals` applies on the local full-text shortcut, `--start-at` sizes its pages from where the band began, a throttled full-text run is TRUNCATED and exit 2, `--updated-before` applies under `--fulltext` while `--query-set`, `--out-name` and `--resume` are refused there rather than ignored, candidates and output directories key on the bare arXiv id so a version bump is not a second copy, a `\documentstyle` or comment-led file is a LaTeX source and colliding tarball basenames are both kept, and omitting `--field` is a message and exit 2. | -| `validate_plugin.py` | Validates manifests, version/count agreement (including the versioned doc headers of `docs/architecture/DEAI_SUBSYSTEM.md` and `docs/architecture/EVALUATION.md`), skill frontmatter and standard references, documentation authority boundaries and index completeness, recorded suite sizes against real test discovery, stale review markers, Python syntax, runtime imports, CLI entry points, feedback schema, linter exits, required tests, CI wiring, and the residue contract (`deai_residue.validator_check`: the history families the paper skill quotes match the tool's, and no shipped document carries an edit-meta literal). | Stdlib-only, 11 checks; rejects a document reappearing at a location it was moved away from, an unindexed document under `docs/`, a `design-notes/` file that does not declare itself one, a relative link or numbered cross-reference whose target does not exist, any recorded test count that disagrees with discovery, and any tracked file over the 750-line budget. Exits nonzero on contract drift. | +| `validate_plugin.py` | Validates manifests, version/count agreement (including the versioned doc headers of `docs/architecture/DEAI_SUBSYSTEM.md` and `docs/architecture/EVALUATION.md`), skill frontmatter and standard references, documentation authority boundaries and index completeness, recorded suite sizes against real test discovery, stale review markers, Python syntax, runtime imports, CLI entry points, feedback schema, linter exits, required tests, CI wiring, and the residue contract (`deai_residue.validator_check`: the history families the paper skill quotes match the tool's, and neither the two READMEs, the standard nor any skill carries an edit-meta literal), and the L0 mirror (`ai_ism_lint.validator_check`, run by the skills check: the paper skill's Tier A table, opener row and Tier B sentence list exactly the forms the linter's four L0 patterns accept, in both directions). | Stdlib-only, 11 checks; rejects a document reappearing at a location it was moved away from, an unindexed document under `docs/`, a `design-notes/` file that does not declare itself one, a relative link or numbered cross-reference whose target does not exist, any recorded test count that disagrees with discovery, and any tracked file over the 750-line budget. Exits nonzero on contract drift. | The top-level manifest counts 40 product tools: the 39 shipped Python tools plus the `ai_ism_negatives_handcrafted.txt` data asset. `validate_plugin.py` is a @@ -57,15 +57,8 @@ development/release tool and is excluded from that count. ## Dependencies The shared schema, regex linter, model-free L1/L2 analysis, document structure, -and validator use the standard library. Optional capabilities add: - -```text -pymupdf PDF corpus extraction and compiled-page rendering -sentence-transformers semantic exemplar retrieval and embedding features -scikit-learn + joblib legacy and learned field-similarity models -transformers + torch token-surprisal / UID measurement -numpy learned feature/model utilities -``` +and validator use the standard library. Each optional capability and the package +it needs is listed once, in [`requirements.txt`](../requirements.txt). Do not install optional dependencies merely to turn an unavailable axis into a nominal score. Missing assets remain `unmeasured`; compatibility thresholds remain @@ -88,16 +81,8 @@ python tools/ai_ism_lint.py draft.tex --field wgl \ --structure --distribution --document-structure --oracle --voice \ --format json --output feedback.json -# Calibrate whole-document shape from a directory of verified complete -# documents (independent .tex/.md papers, not paragraphs of one paper) -python tools/deai_docstructure.py --field --calibrate \ - --corpus-dir - -# Per-section salience reference and corpus document frequency for register -python tools/deai_salience.py --field --calibrate -python tools/deai_register.py --field --calibrate -python tools/deai_discourse.py --field --calibrate -python tools/deai_collocation.py --field --calibrate # pair bank + leave-one-out reference +# Calibrate every reference: the ordered list of --calibrate commands is kept +# once, in style-profile/README.md ("Build boundaries") # The removal map, the shrink target it sets, and the residue diff gate python tools/condense_map.py draft.tex --format json --output condense-map.json From ed97c35630194843c320b63868125c483099460b Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 12:38:59 -0400 Subject: [PATCH 26/27] audit record: every item mapped to its commit or to a closing reason Section 10 of docs/audits/full-review-2026-09-27.md now covers all 318 items. The PR's own rows stay, with E32 in the rewrite_reward row and the two "this section's commit" cells naming the PR commits they meant. New rows list the items the PR fixed without naming them (each re-checked before merge), d42ef14's follow-ups including C20, the invented examples of bbd355a, and the code, skill and document commits of this merge. G26, M6, H21 and H23-H30 are closed with their reasons. The "artifacts to rebuild" paragraph becomes the record of the 2026-09-27 rebuild and re-take, and two measured properties of the held-out set are noted: its 200 bundles read as 203 documents, and 15 of them have an odd number of unescaped `$` once comments are removed. Co-Authored-By: Claude Opus 5.5 --- docs/audits/full-review-2026-09-27.md | 33 +++++++++++++++++++-------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/audits/full-review-2026-09-27.md b/docs/audits/full-review-2026-09-27.md index a945733..cc02c1f 100644 --- a/docs/audits/full-review-2026-09-27.md +++ b/docs/audits/full-review-2026-09-27.md @@ -378,7 +378,7 @@ ## 10. 落地记录（v0.39.0，2026-09-27） -§8 第一梯队与其耦合的第二梯队已在 v0.39.0 落地，提交按范围列出；每个提交正文写明它覆盖的条目号。 +§1–§7 的每个条目都已在 v0.39.0 落地或附理由关闭，提交按范围列出；每个提交正文写明它覆盖的条目号。前十行属 PR #1，其后是合并前的收尾提交。 | 范围 | 条目 | 提交 | |---|---|---| @@ -386,14 +386,27 @@ | 装配、投影与宏；length_gate 与 condense_map；ai_ism_lint 与 validate_plugin；extract_style 与 retrieve_exemplars | A1–A20、A28，F3/F4/F7/F9/F12/F14/F16–F20/F25/F27/F28/F33，D6–D11/D14/D16/D19，M2/M3 | `5ce670c` | | verify_references | F1/F2/F5/F6/F8/F10/F11/F13/F21/F23/F24/F32 | `46cf4ce` | | 文档级轴：docshape、docstructure、metrics、structure、features、partition、anchoring | B1–B15、B19–B21、B23、B28，E20 的 features 侧 | `1c9736d`、`f211ece`、`2d2685d` | -| rewrite_reward | E1–E4、E12、E30，H1 | `116f074` | +| rewrite_reward | E1–E4、E12、E30、E32，H1 | `116f074` | | eval_findings、eval_docscale、label_findings、voice_dataset、voice_audit、train_voice_model、train_ai_ism_classifier | E5–E8、E10/E11、E13–E16、E18/E19、E21/E22、E28、E31、E33，M5 | `9b30c47`、`e47d250` | | fetch_arxiv_abstracts | D1–D5、D12/D13、D24、D26 | `41a29a8` | -| 逐 bucket 轴：reference、collocation、discourse、salience、register、residue、oracle、provenance、personal、voice | C1–C19、C21、C22（C20 属 deai_structure，未动），E20 的 voice 侧；ai_ism_lint 把带文本的状态接入统一报告 | `95c988f`、`3e13594`、`7daf3e9` 及本节所在提交 | -| 文档与元数据：§6.2 全部数字（I1–I20、G14–G25）、H3–H6/H12/H13/H36 的 skill 补丁、I21/I25/I36、I22/I31–I34、I38、G11、G27、D15（改为"记录权重，未应用"）、H31（标准 v3.9）及各部分的 registry 行、发布说明与记录的测试数 | | `1ad6ed7`、`de925a9`、`470290d`、`b25f4cd` 及本节所在提交 | - -未落地（§8.3 及本轮范围之外）： - -- §5 的去重（R1–R10、H23–H30/H45–H51）；待核实项 B16/B17/B18/E17/E32/F22/F26/D20；G19（`latency.json`）；M3/I35 的 disposition 列。A31/D26/E33/B28 的 `__main__` 位置已随各自文件修复。 -- 需重建的制品：`structure_baseline` 与 `docstructure_baseline`（B12/B15）、`anchoring_baseline`（B13）；逐 bucket 轴的制品需重新校准才采用最近秩分位网格（C12）。重建前，EVALUATION 中据旧制品测得的比率沿用旧值，各处已注明测得日期。 -- 抓取器的内置查询集、关键词过滤与期刊表标为 `[WGL]`，其他领域改用 `--query`（D24）；`TIER_WEIGHTS` 仍只记录不应用（D15 按第二梯队的"改措辞"落地）。 +| 逐 bucket 轴：reference、collocation、discourse、salience、register、residue、oracle、provenance、personal、voice | C1–C19、C21、C22（C20 见 `d42ef14` 行），E20 的 voice 侧；ai_ism_lint 把带文本的状态接入统一报告 | `95c988f`、`3e13594`、`7daf3e9`、`04ae5b0` | +| 文档与元数据：§6.2 全部数字（I1–I20、G14–G25）、H3–H6/H12/H13/H36 的 skill 补丁、I21/I25/I36、I22/I31–I34、I38、G11、G27、D15（改为"记录权重，未应用"）、H31（标准 v3.9）及各部分的 registry 行、发布说明与记录的测试数 | | `1ad6ed7`、`de925a9`、`470290d`、`b25f4cd`、`e5f69c4`、`04ae5b0` | +| 上面各行未列号、PR #1 已修的条目（收尾前逐条复核为已修） | A21、A26、A27、A29、A30、A32，B26，E34，F34，G1–G3、G5、G7、G8、G10、G13、G33、G35、G36，H7、H9–H11、H19、H20、H38、H41、H43、H44，I23、I24、I30、I39 | PR #1（`6107991..04ae5b0`） | +| §8.3 的待核实项与 PR 自身的回归 | B16（关闭，理由写在 `DEFAULT_BURSTINESS_RATIO` 旁）、B17、B18、B23 余项、C20（注释例句换成检测器命中的一类）、D20、E17、E20 余项、F22、F26；复合词分词、`unit_reason` 的范围、注释里的 `\newcommand` 三处回归；代码侧取自在审手稿的例子换成虚构例子 | `d42ef14` | +| 文档与记录里取自在审手稿的例子 | 换成同类虚构例子或工具报出的计数，不改任何测量 | `bbd355a` | +| 代码余项 | A22–A25，B22、B24、B30，D17、D18、D21、D23、D25 的代码侧、D27–D29，E23–E26、E29，F29–F31、F37、F38，H8 的 validator 镜像检查；hedging 按 2026-09-27 的重测扩到 `method` | `6aaf9f3` | +| skill 余项 | H14（作者裁定 skill 服从标准 §4：四处人工判断的 strong 改为排在最前的普通 advisory），H15–H18、H22、H32、H33、H37、H39、H40、H42，F15 的 skill 侧，H45–H51（skill 内去重） | `2c70054` | +| 文档余项与重建后的重测 | G4、G6、G12、G28–G31、G34，H34、H35，E27，F15 与 F35 的文档侧，I26–I29、I37，M3/I35，B24/B25 的文档侧，R1–R10，M4/G19（删除 `latency.json`）；重建后 EVALUATION、两份 README 与 examples 的数字全部重取 | `4dfa9a4` | +| 本节 | 落地记录的收尾 | 本节所在提交 | + +附理由关闭、不改的条目： + +- G26：`plugin.json` 与 `marketplace.json` 的描述由不同的消费方读取（插件加载器与 marketplace 列表），validator 的 manifest 检查要求两者一致，逐字相同是它守护的状态，不是漂移。 +- M6：标准 §0.1 对 advisory 工具只要求配置失败时非零退出；这些工具没有 finding 退出码，exit 1 不会与 finding 语义混淆。 +- H21：brainstorm 的 NEEDS-MORE-INFO 是临时 verdict，收敛判据禁止它留到终态（§6 第 1 条，终报的节点分布里为 0），与 paper-review 的要求一致。 +- H23–H30：跨 skill 的重复保留（作者裁定）：每个 skill 单独加载，不跟随指针。 +- A31/D26/E33/B28 的 `__main__` 位置已随各自文件修复；D24 的内置查询集、关键词过滤与期刊表标为 `[WGL]`，其他领域用 `--query`；`TIER_WEIGHTS` 只记录不应用（D15 按第二梯队的"改措辞"落地）。 + +重建与重取：2026-09-27 按本版全部语料侧修复重建 `wgl` profile（exemplar bank、lexicon、register、collocation、cohesion、hedging、salience、structure、anchoring、UID 与 docstructure），并重训 L3；B22 落地后 docstructure 再校准一次。逐 bucket 轴的制品随重建采用最近秩分位网格（C12）。EVALUATION、两份 README 与 examples 的数字随之重取，没重取的行写明取数所在的构建。 + +已知性质（记录备查，不是缺陷，2026-09-27 测）：留出集 200 个 bundle 读出 203 篇文档，因为三个 bundle 各有第二个根文件被选为独立文档（`AuthorList_P13_CIB_Lensing_cpp.tex`、`Planck_bib.tex`、`aa.tex`）；203 篇里有 15 篇去掉注释后未转义的 `$` 个数为奇数，简单配对下至少有一个 `$` 落单。 From 37432af8c805e4118b49eb1ef7047b3c8dfb8b6a Mon Sep 17 00:00:00 2001 From: skymanbp Date: Sun, 27 Sep 2026 13:13:13 -0400 Subject: [PATCH 27/27] deai_partition: the cohesion floor reads adjacent paragraphs only (pre-merge review of 6aaf9f3) Since B22 kept headings and floats as landmark blocks, suggest() took the median overlap over every adjacent block pair, landmarks included. Each section heading and each float between paragraphs added near-zero pairs, so the floor fell: 0.0 on the new test's three-section document, against 0.078 over its paragraph pairs, which admits every merge and only splits whose halves share no content word. The floor now reads consecutive prose blocks per section, the blocks _state_statistics scores. Test: the floor equals the median over the document's paragraph pairs, with and without a float between two of them; it failed on the previous code. Suite 802; the recorded counts and the audit record's section 10 follow. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- README.md | 10 +++++----- README.zh-CN.md | 10 +++++----- docs/architecture/EVALUATION.md | 4 ++-- docs/audits/full-review-2026-09-27.md | 1 + tests/test_deai_partition.py | 23 +++++++++++++++++++++++ tools/deai_partition.py | 10 +++++++--- 7 files changed, 44 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 20fee30..ff08c56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -243,7 +243,7 @@ paragraphs). Held-out register reads 56 findings, 0.0364 per 1,000 body words, 87.5% of which would vanish had the paper been in its own bank. The worked example reads 22 advisories before the revision and 17 after. -The suite is 801 tests in 32 files (v0.38.0: 529 in 27); the validator runs +The suite is 802 tests in 32 files (v0.38.0: 529 in 27); the validator runs 11 checks and passes. v0.35.0–v0.35.1 moved to `CHANGELOG-ARCHIVE-v0.35.md`; this file had reached 806 lines. diff --git a/README.md b/README.md index acafe1c..a493cbc 100644 --- a/README.md +++ b/README.md @@ -5,14 +5,14 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-801%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-802%20passing-success.svg)](tests/) **A Claude Code plugin that writes, reviews, de-AIs, and condenses scientific manuscripts for top-tier journals — under one typed standard, with every claim traced to a source and every unavailable measurement labelled as unavailable.** Built for ApJ / MNRAS / PRD / JCAP-class papers and NSF / NIH proposals. -**12 skills · 40 tools · 801 tests · one normative contract · zero authorship verdicts.** +**12 skills · 40 tools · 802 tests · one normative contract · zero authorship verdicts.** [中文文档](README.zh-CN.md) — [What it does](#what-it-does) · [How it works](#how-it-works) · [See it work](#see-it-work) · [Benchmarks](#benchmark-dashboard) · [Install](#install) · @@ -475,7 +475,7 @@ at 100%, and a number taken under that describes the contention, not the axis. V | `+ --oracle` (GPT-2-large token surprisal) | 33.8 s (2026-08-27) | `transformers` + `torch` | | `+ --voice` (learned L3 triage) | 37.2 s (2026-08-27) | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` — **11/11 checks pass** | 2.24 s | stdlib | -| Full test suite — **801 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | +| Full test suite — **802 passing**, 32 files (timing from the 2026-09-05 run of the suite as it then stood, median of 3 spanning 49.8 – 51.7 s; not re-taken since) | 51.5 s | stdlib | The headline: **a complete model-free pass over a 5,084-word manuscript costs ~1.0 s above the interpreter floor**, with no optional dependency installed — and 0.6 s of that is loading the collocation bank (530,677 @@ -670,14 +670,14 @@ sci-paper/ │ ├── SCIPAPER_STANDARD.md the single normative contract (v3.8) │ ├── architecture/ DEAI_SUBSYSTEM.md · EVALUATION.md (hub) + evaluation/ │ └── design-notes/ frozen, dated reasoning records (not status) -├── skills//SKILL.md 12 skills ├── tests/ 32 files, 801 tests +├── skills//SKILL.md 12 skills ├── tests/ 32 files, 802 tests ├── tools/ 40 product tools ├── CHANGELOG.md · ACKNOWLEDGMENTS.md ├── style-corpus// user-supplied read-only corpus (gitignored) └── style-profile// generated and calibrated evidence (gitignored) ``` `python tools/validate_plugin.py` runs 11 contract checks (listed in [tools/README.md](tools/README.md)) and -`python -m unittest discover -s tests -v` runs the 801-test suite; both must pass before a release. A release +`python -m unittest discover -s tests -v` runs the 802-test suite; both must pass before a release. A release also requires independent review, clean-checkout verification, and green hosted CI. --- diff --git a/README.zh-CN.md b/README.zh-CN.md index 9098383..30e9856 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -5,13 +5,13 @@ [![Version](https://img.shields.io/badge/version-0.39.0-informational.svg)](CHANGELOG.md) [![Claude Code plugin](https://img.shields.io/badge/Claude%20Code-plugin-8A5CF6.svg)](https://docs.claude.com/en/docs/claude-code/plugins) [![Python](https://img.shields.io/badge/python-%E2%89%A5%203.11-3776AB.svg)](requirements.txt) -[![Tests](https://img.shields.io/badge/tests-801%20passing-success.svg)](tests/) +[![Tests](https://img.shields.io/badge/tests-802%20passing-success.svg)](tests/) **一个 Claude Code 插件：在同一套 typed 标准下完成科研论文的写作、审查、去 AI 化与精简。 每条结论都可溯源，每个测不出来的轴都如实标为测不出来。** 面向 ApJ / MNRAS / PRD / JCAP 级别的论文，以及 NSF / NIH 基金申请书。 -**12 个 skill · 40 个工具 · 801 个测试 · 一份规范 · 零作者身份判决。** +**12 个 skill · 40 个工具 · 802 个测试 · 一份规范 · 零作者身份判决。** [English](README.md) — [它做什么](#它做什么) · [怎么做到的](#怎么做到的) · [实际效果](#实际效果) · [Benchmark 面板](#benchmark-面板) · [安装](#安装) · @@ -448,7 +448,7 @@ LaTeX include 组装而成。标准库的每一行都重测了：解释器地板 | `+ --oracle`（GPT-2-large token surprisal） | 33.8 s（2026-08-27） | `transformers` + `torch` | | `+ --voice`（学习型 L3 分诊） | 37.2 s（2026-08-27） | `scikit-learn` + `sentence-transformers` | | `validate_plugin.py` —— **11/11 通过** | 2.24 s | 标准库 | -| 完整测试套件 —— **801 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | +| 完整测试套件 —— **802 通过**，32 个文件（计时取自 2026-09-05 对当时套件的一次运行，3 次取中位数，落在 49.8 – 51.7 s；此后未重测） | 51.5 s | 标准库 | 一句话：**一份 5,084 词的稿子跑完全部 model-free 通道，在解释器地板之上约花 1.0 s**， 且不需要任何可选依赖——其中 0.6 s 是加载 collocation 词对库（计时所用的 2026-09-04 库有 530,677 对，v0.39.0 重建后为 535,517 对）， @@ -655,7 +655,7 @@ sci-paper/ │ └── design-notes/ 冻结的、带日期的设计记录（不是现状文档） ├── skills//SKILL.md 12 个 skill ├── tools/ 40 个产品工具 + 仓库 validator -├── tests/ 32 个测试文件、801 个测试 +├── tests/ 32 个测试文件、802 个测试 ├── style-corpus// 用户提供的只读语料（gitignore） ├── style-profile// 生成与标定的证据（gitignore） ├── ACKNOWLEDGMENTS.md 改编来源的致谢与采纳边界 @@ -666,7 +666,7 @@ sci-paper/ ## 开发与发布 `python tools/validate_plugin.py` 跑 11 项契约检查（清单见 [tools/README.md](tools/README.md)）， -`python -m unittest discover -s tests -v` 跑 801 个测试；发布前两者都必须通过。 +`python -m unittest discover -s tests -v` 跑 802 个测试；发布前两者都必须通过。 发布还额外要求独立代码审查、干净 checkout 验证，以及发布 commit 上的绿色 CI。 --- diff --git a/docs/architecture/EVALUATION.md b/docs/architecture/EVALUATION.md index 3549990..81bffae 100644 --- a/docs/architecture/EVALUATION.md +++ b/docs/architecture/EVALUATION.md @@ -93,7 +93,7 @@ python tools/validate_plugin.py python -m unittest discover -s tests -v ``` -The working tree passes the validator and all 801 unit/CLI tests (32 test files, collected +The working tree passes the validator and all 802 unit/CLI tests (32 test files, collected 2026-09-27). These commands must be rerun after every subsequent code or release-metadata change; the release record must quote the fresh output rather than a past result. @@ -106,7 +106,7 @@ works and the drift it exists to stop. Current release gates (as of 2026-09-27; last tagged release v0.38.0, this release v0.39.0): `validate_plugin.py` all 11 checks pass and the full unit/CLI suite -(801 tests, 32 files) passes on a clean tree; both are rerun before every tag, +(802 tests, 32 files) passes on a clean tree; both are rerun before every tag, and as of v0.25.1 the hosted CI run on the release commit must also be green (first green runs: 31133202443 push, 31133215203 manual dispatch). diff --git a/docs/audits/full-review-2026-09-27.md b/docs/audits/full-review-2026-09-27.md index cc02c1f..6e1beb5 100644 --- a/docs/audits/full-review-2026-09-27.md +++ b/docs/audits/full-review-2026-09-27.md @@ -398,6 +398,7 @@ | skill 余项 | H14（作者裁定 skill 服从标准 §4：四处人工判断的 strong 改为排在最前的普通 advisory），H15–H18、H22、H32、H33、H37、H39、H40、H42，F15 的 skill 侧，H45–H51（skill 内去重） | `2c70054` | | 文档余项与重建后的重测 | G4、G6、G12、G28–G31、G34，H34、H35，E27，F15 与 F35 的文档侧，I26–I29、I37，M3/I35，B24/B25 的文档侧，R1–R10，M4/G19（删除 `latency.json`）；重建后 EVALUATION、两份 README 与 examples 的数字全部重取 | `4dfa9a4` | | 本节 | 落地记录的收尾 | 本节所在提交 | +| 合并前复审 | B22 在 `deai_partition` 的余项：内聚下限只取相邻段落对，标题与浮动体不再把它拉低 | 本行所在提交 | 附理由关闭、不改的条目： diff --git a/tests/test_deai_partition.py b/tests/test_deai_partition.py index 4efe108..9826bdf 100644 --- a/tests/test_deai_partition.py +++ b/tests/test_deai_partition.py @@ -3,6 +3,7 @@ import contextlib import io import json +import statistics import tempfile import unittest from pathlib import Path @@ -173,6 +174,28 @@ def test_what_the_blanking_removes_stays_a_fixed_unscored_landmark(self): self.assertEqual([b["lines"] for b in section["blocks"] if not b.get("landmark")], [(start, end) for start, end, _ in ds.shape_units(text)[0]["blocks"]]) + def test_landmarks_do_not_move_the_cohesion_floor(self): + # The floor is the median overlap of adjacent paragraphs. Counted as + # blocks, every heading and float added near-zero pairs that pulled it + # down (the pre-merge review of audit B22). + figure = "\\begin{figure}\n\\caption{A map of the field.}\n\\end{figure}\n\n" + + def document(between: str) -> str: + return ("\\section{Introduction}\n\n" + LENSING_A + "\n\n" + LENSING_B + "\n\n" + "\\section{Methods}\n\n" + LENSING_A + "\n\n" + between + UNRELATED + + "\n\n" + LENSING_B + "\n\n" + "\\section{Results}\n\n" + LENSING_B + "\n\n" + LENSING_A + "\n") + pairs = [(LENSING_A, LENSING_B), (LENSING_A, UNRELATED), + (UNRELATED, LENSING_B), (LENSING_B, LENSING_A)] + expected = statistics.median(partition._overlap(a, b) for a, b in pairs) + # six paragraphs reach the manifold (UNRELATED is under MIN_WORDS); + # p inside the band, so the plan stops before its first step + operating = fake_operating_point(p_by_n={6: 0.5}, distance_by_n={6: 1.0}) + with mock.patch.object(ds, "manifold_operating_point", operating): + floors = [partition.suggest(document(between), {"dispersion_manifold": {}}, + 1)["cohesion_floor"] for between in ("", figure)] + self.assertEqual(floors, [expected, expected]) + def fake_operating_point(p_by_n: dict, distance_by_n: dict): """A stand-in for `manifold_operating_point` keyed by paragraph count, diff --git a/tools/deai_partition.py b/tools/deai_partition.py index ea398a1..f9a2627 100644 --- a/tools/deai_partition.py +++ b/tools/deai_partition.py @@ -254,9 +254,13 @@ def suggest(text: str, baseline: dict, max_ops: int) -> dict: if start is None: return {"status": "unmeasured", "reason": "document or manifold not measurable", "plan": []} - # self-normalized cohesion reference: median adjacent-pair overlap - adjacent = [_overlap(s["blocks"][i]["text"], s["blocks"][i + 1]["text"]) - for s in sections for i in range(len(s["blocks"]) - 1)] + # self-normalized cohesion reference: the median overlap of adjacent + # paragraphs, over the prose blocks alone. A landmark is no paragraph; + # counted, every heading and float added near-zero pairs that pulled the + # floor down, admitting more merges and fewer splits. + prose = [[b["text"] for b in s["blocks"] if not b.get("landmark")] for s in sections] + adjacent = [_overlap(first, second) + for texts in prose for first, second in zip(texts, texts[1:])] if not adjacent: return {"status": "unmeasured", "reason": "no adjacent paragraph pairs", "plan": []}