Skip to content

Add handwritten TPC-DS benchmark and sampled-world validation - #39

Draft
ila wants to merge 1 commit into
upgrade/duckdb-2.0from
bench/tpcds-handwritten-validation
Draft

ila wants to merge 1 commit into
upgrade/duckdb-2.0from
bench/tpcds-handwritten-validation

Conversation

@ila

@ila ila commented Nov 11, 2026

Copy link
Copy Markdown
Member

Add handwritten stochastic TPC-DS queries and independent sampled-world references, with a runner for timing and checking the expanded workload. This separates benchmark development from the DuckDB 2.0 upgrade in #35.

Changes

  • Preserve 66 handwritten AS queries and 66 per-world SQL references, the SF30 configuration, and the benchmark runner/validation additions.
  • Document the distinction between compiled release behavior and per-world aggregate references.
  • Keep the SUM accuracy unit correction directly in its SQL query.

This draft targets upgrade/duckdb-2.0 for a benchmark-only diff. It is intended to merge after #35 and be retargeted to main once that upgrade lands.

Validation and remaining work

The C++ runner builds against the cleaned upgrade; Python and JSON parse checks pass. A nonempty Q52 smoke succeeds in all four timing modes with five measured runs, and compiler/handwritten outputs match exactly.

Per-world validation remains WIP: setup unconditionally defines a macro using as_stddev_samp, which has no registered implementation. This prevents the AS per-world comparison even for queries that do not use standard deviation; Q17 also requires an actual STDDEV rewrite. The conditional-aggregate Q52 oracle matches. Resolve these paths before treating the per-world validation as complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant