Skip to content

test: expand paired dataset differential acceptance and fix zero-rank completeness - #258

Merged
codeasier merged 6 commits into
mainfrom
test/251-random-differential
Oct 11, 2026
Merged

codeasier merged 6 commits into
mainfrom
test/251-random-differential

Conversation

@codeasier

@codeasier codeasier commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Depends on #255; its two infrastructure commits are included in this branch so CI selection works before merging. Review/merge #255 first.

Summary

Refs #251. Built on the CI tier infrastructure from #248.

  • Preserve the original eight regression seeds and twenty query shapes; expand to 20 deterministic seeds in regular PR tests and 200 in a required scheduled/manual step, in both compatibility-v1 and native-v2 layouts.
  • Compare all eleven dataset-supported templates with layout-matched standalone SQL over the same independently generated history. Cover filters, sorting, pagination, exact totals, static inclusion, rank caps, valid event/member ranges, and 5–40 shards. Native inputs pass actual manifest/hash/reference validation. Unsupported frame templates are tested separately.
  • Check critical allocation/free-request/free-completion/reuse boundaries against an independent active-set/counter model. Assert actual scratch execution, unchanged input hashes and reproducible seed/format/parameter evidence. One-sided and two-sided unexpected query errors always fail.
  • Fix the zero-sized inner-rank completeness mismatch found by the expanded suite: top_n=0 remains conservatively incomplete unless exact_total resolves it. Nonzero rank behavior is unchanged. Preserve test: clarify lifecycle contracts across all dataset query paths #254's standalone/dataset leak-scope, unproved-identity and observation-state contracts.
  • Add a required fail-closed nightly runner that verifies all 400 executed test identities, matching per-seed evidence and zero unexpected errors. Always upload compact log/XML/summary/timeline/coverage/failure evidence; exclude reproducible SQLite files from retained artifacts.

Validation

Completed locally:

  • Default profile: 72 tests passed; 40 seed-format cases, 2,896 paired comparisons, 50.73 seconds.
  • Direct 200-seed profile: 400 seed-format cases passed; 28,864 comparisons and 2,128 actual scratch builds, 498.91 seconds, zero unexpected errors.
  • Regular non-slow suite: 2,671 passed, 14 skipped, 13 deselected. Skips comprise twelve optional browser cases and two intentionally unsupported leak-slice combinations.
  • Focused query/completeness/test: clarify lifecycle contracts across all dataset query paths #254/CLI/API/JSON checks passed; independent review additionally passed a 3,600-case zero/nonzero-rank matrix.
  • Nightly runner/wiring/governance tests: 93 passed. Real runner smoke executed both formats while removing injected seed and pytest-selection overrides. The evidence validator accepted the complete direct 200-seed run.
  • Ruff, Black and diff checks passed. Type checking: zero errors; existing informational warnings remain.

Pending at draft creation:

  • The new required runner itself is now running end-to-end with --seed-count 200 against frozen local commit 85250731fc7c6621d0bb6040e09691be7edcbc9f. Its final result is not yet claimed as passed.
  • Remote CI and scheduled/manual workflow execution remain unverified.

Reproduce the required runner from the checked-out revision:

python .github/scripts/run_differential_acceptance.py \
  --seed-count 200 --output-dir .tmp/differential-acceptance

Evidence boundaries

These generated fixtures validate reader/query semantics. They do not claim native exporter, live PyTorch/NPU collector, paid-model or real-user-session acceptance. Installed-wheel and producer gates remain separate. No changes are made under src/pt_snap_cli/snapshot/.

Copy link
Copy Markdown
Owner Author

The previously pending required 200-seed runner has now completed end-to-end on the exact reviewed local tree 2a22646a6d7c3b4be96cf168dec1d68558290666 (identical to published head 8b28ae53257bfa5c3b9d7a9cb6e5e8806b0e1f7c).

400 passed in 577.19s: every expected seed × v1/v2 identity was verified, 28,864 strict comparisons, zero unexpected errors, and 2,128 actual scratch builds. The fail-closed runner validated matching per-case artifacts and wrote status=passed with pytest_exit_code=0. No edits/builds occurred during the run.

This is completed local required-runner evidence. Published-head PR CI is still running; remote scheduled/manual execution remains unverified and is not claimed as passing.

Copy link
Copy Markdown
Owner Author

Latest-head CI verification (2026-10-11): commit 8b28ae53257bfa5c3b9d7a9cb6e5e8806b0e1f7c, Test run 38146231472 completed successfully. All seven applicable jobs passed (Python 3.10–3.13, lint/format/type checks, provenance, browser regression). Python 3.13: 2745 passed, 14 skipped, 1 deselected; coverage 93.18%. Wheel audit and clean-wheel Agent acceptance passed. The separate installed-wheel dataset/nightly job was intentionally skipped on the PR event; this is not remote nightly verification. Local scale evidence in the description/earlier comments remains separate. Draft status and issue status are unchanged.

Copy link
Copy Markdown
Owner Author

检视结论:approve with nits

对照 #251:20/200 种子差分 + v1/v2 与单库对照。产品修复 top_n=0 → rank_full=True(dataset_active_sql / attribution / query_service)正确。zero_rank 与定向测试通过。

nit:match="unexpected query errors.*seed" 遇多行 AssertionError 可能偏脆,建议加 (?s) 或收窄匹配。

建议在 #255 之后、#261 之前合入。本评论不 approve/merge。

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with nits,已转为正式 PR。GitHub 不允许作者账号提交 APPROVE。合入前建议顺手修 match 正则((?s));可在 #255 之后、#261 之前合。

Copy link
Copy Markdown
Owner Author

已处理检视 nit(https://github.com/codeasier/pt-snap-cli/pull/258#issuecomment-6110740929),追加提交 32bcd9e93e621961b9bad4342ae6fb32de89ade5。

tests/core/test_dataset_differential_cases.py 的匹配改为 (?s)unexpected query errors.*seed,仍同时要求 query-error 与 seed,不改变生产逻辑或错误判定。通过实际 assertion-rewritten helper 的三种错误组合复现:旧正则无法匹配多行诊断,新正则可以,双方结果及 seed/format 证据仍完整。

本地 19 项定向/样本溯源 + 60 项真实随机差分测试通过;另有 6 项正则正反例探针通过,Ruff/Black/空白检查通过。远端树与本地审阅树一致;最新提交 CI 正在运行,尚未称为通过。

Retain the reviewed independent 200-seed step and evidence upload. No differential runtime, test or runner bytes changed.

Copy link
Copy Markdown
Owner Author

已与当前 main d2b5399e60143a312df089bf04718b94b301b740 集成,分支更新为 0f0089a23494749b863fd7a23e5b7b352c61d43e,树 198c8032989b0aadec1b5a5ab20a2d9db557fb4e。

主分支要求线性历史,#255–#257 已采用 squash。此次仅解决共享 CI 基础带来的上下文冲突:保留 differential runner 索引、同一 nightly job 内独立的 200-seed step 及其 evidence upload。上述 workflow/runner、差分生产逻辑及测试字节与原审阅 head 完全一致;继承 main 的平台/fixture 变更。

当前集成树本地验证:100 项 runner/CI/provenance/governance 检查 + 88 项实际差分/zero-rank 测试通过,git diff --check 通过。新 head 的托管 Test/Platform CI 待验证;旧 head 的通过记录不作为新 head CI 证据。PR 正文只将自动关闭关键字改为 Refs,议题状态保留。

Keep the full non-slow union, twenty-minute job cap and max-parallel two. Update fail-closed partition guards and bilingual reproduction docs; production behavior and differential test cases are unchanged.

Copy link
Copy Markdown
Owner Author

托管平台集成暴露 Windows 时间预算问题,现追加 CI-only 修复 6a7da43da80379407a8a8768a0b34dcb1d5f9978(独立审阅树 de1de59ea492d74ed08a9e83ace2f8cf5e374b82)。

旧 head 的 Test/Linux 全绿;Platform run 38157189623 的 Windows shard 0 用时 17m43s 成功,shard 1 达到原有 20 分钟上限后取消,日志为 985 passed、8 capability skips、71% 完成,无 assertion failure。旧失败日志保留,不计为通过。

改为四个确定性 Windows 分片,保持 max-parallel=2、每任务 20 分钟、完整 non-slow 集合和警告门禁;macOS 仍完整单任务。不减少种子,不添加 skip,不放宽测试。同步 fail-closed 分片回归及英文/中文说明,生产代码和差分用例字节不变。

本地 52 项 wiring/governance/provenance 测试通过;独立复核 20 项通过,Ruff/Black/空白检查通过。四次真实 collect-only 的相同完整集合共 2,785 项,分片 [697, 696, 696, 696],互斥并集精确相等;40 个默认 seed/format 用例保留,每片 10 项。旧日志重分配的已观测工作约 484/512/512/637 秒,其中旧超时片有未测尾部,不能当作新片完整耗时。新的托管 Test/Platform CI 才验证真实结果;当前待运行。

@codeasier
codeasier merged commit 25ae3ad into main Oct 11, 2026
13 checks passed

Copy link
Copy Markdown
Owner Author

最新 head 6a7da43da80379407a8a8768a0b34dcb1d5f9978 的 Test 38159018984 和 Platform 38159019075 已全部通过。

四个 Windows 任务实际耗时 9m06s / 9m26s / 8m27s / 10m02s,均在原有 20 分钟内;日志对应 691/690/689/688 passed 和 6/6/7/8 capability skips。实际执行身份共 2,785 项、零重复,与独立 non-slow 收集精确相等,40 个 default seed/format 用例均执行。macOS 完整任务成功。

已按授权顺序 squash 合入,main 25ae3ad48788433678e310093e5acb5e711abd27 的树与该审阅/验证树完全一致。正在等待新的 main-push CI;后续 PR 尚未合入。nightly/manual 仍非本次 PR 触发执行,不能计作远端 nightly 验收。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant