Pre-submission checks
Is your feature request related to a problem?
I'd like to raise a concern regarding the data generation strategy used for the Join Order Benchmark (JOB) workload in BenchBox.
Background & Motivation
The original JOB paper ("How Good Are Query Optimizers, Really?", Leis et al., PVLDB 2015) explicitly chose the IMDB dataset precisely because it is real-world data. The paper demonstrates that real-world datasets exhibit strong correlations, skew, and non-uniform value distributions that systematically mislead traditional cardinality estimators (e.g., independence assumptions, histogram-based methods). These estimation errors then cascade into catastrophic join ordering decisions.
This is the entire point of JOB as a benchmark — it exposes optimizer weaknesses that only manifest on data with realistic, complex statistical properties (e.g., certain directors consistently casting the same actors, popularity skew across decades, genre co-occurrence patterns).
The Problem
From my understanding, BenchBox's current data generation for the JOB benchmark produces data with uniformly random distributions. If this is the case, the generated data fundamentally lacks the correlated, skewed characteristics that make JOB challenging and meaningful.
With uniformly distributed data:
Cardinality estimates based on independence assumptions become accidentally accurate.
Join order selection becomes trivial because the optimizer's simple statistical model happens to match the data.
The benchmark no longer differentiates between a naive optimizer and a sophisticated one.
In other words, uniform random generation removes the very property that JOB was designed to test.
References
- Leis, V., Gubichev, A., Mirber, A., Boncz, P., Kemper, A., & Neumann, T. (2015). How Good Are Query Optimizers, Really? PVLDB, 9(3), 204–215.
- Leis, V., Radke, B., Gubichev, A., Mirber, A., Boncz, P., Kemper, A., & Neumann, T. (2018). Query Optimization Through the Looking Glass, and What We Found Running the Join Order Benchmark. VLDB Journal, 27(5), 643–668.
Thank you for building BenchBox — it's a great tool. I hope this feedback helps make the JOB workload even more representative and useful for the community.
Describe the solution you'd like
I completely understand that programmatically generating synthetic data that preserves the complex statistical distributions and cross-table correlations of the original real-world dataset is incredibly difficult.
Because of this difficulty, I believe a more reasonable solution is to remove/disable the scaling capability for the JOB workload. Instead, BenchBox could simply reference and ingest the original frozen IMDB dataset.
Allowing synthetic scaling with uniform distributions risks severely misleading researchers and engineers who use BenchBox to evaluate their optimizers' performance on JOB. By enforcing the use of the original dataset, BenchBox ensures the benchmark remains faithful to its original, rigorous intent.
Describe alternatives you've considered
No response
Feature category
Data generation
Priority
High - Essential for current project
Use case details
Testing JOB with current BenchBox will lead to a wrong result.
Additional context
No response
Pre-submission checks
Is your feature request related to a problem?
I'd like to raise a concern regarding the data generation strategy used for the Join Order Benchmark (JOB) workload in BenchBox.
Background & Motivation
The original JOB paper ("How Good Are Query Optimizers, Really?", Leis et al., PVLDB 2015) explicitly chose the IMDB dataset precisely because it is real-world data. The paper demonstrates that real-world datasets exhibit strong correlations, skew, and non-uniform value distributions that systematically mislead traditional cardinality estimators (e.g., independence assumptions, histogram-based methods). These estimation errors then cascade into catastrophic join ordering decisions.
This is the entire point of JOB as a benchmark — it exposes optimizer weaknesses that only manifest on data with realistic, complex statistical properties (e.g., certain directors consistently casting the same actors, popularity skew across decades, genre co-occurrence patterns).
The Problem
From my understanding, BenchBox's current data generation for the JOB benchmark produces data with uniformly random distributions. If this is the case, the generated data fundamentally lacks the correlated, skewed characteristics that make JOB challenging and meaningful.
With uniformly distributed data:
Cardinality estimates based on independence assumptions become accidentally accurate.
Join order selection becomes trivial because the optimizer's simple statistical model happens to match the data.
The benchmark no longer differentiates between a naive optimizer and a sophisticated one.
In other words, uniform random generation removes the very property that JOB was designed to test.
References
Thank you for building BenchBox — it's a great tool. I hope this feedback helps make the JOB workload even more representative and useful for the community.
Describe the solution you'd like
I completely understand that programmatically generating synthetic data that preserves the complex statistical distributions and cross-table correlations of the original real-world dataset is incredibly difficult.
Because of this difficulty, I believe a more reasonable solution is to remove/disable the scaling capability for the JOB workload. Instead, BenchBox could simply reference and ingest the original frozen IMDB dataset.
Allowing synthetic scaling with uniform distributions risks severely misleading researchers and engineers who use BenchBox to evaluate their optimizers' performance on JOB. By enforcing the use of the original dataset, BenchBox ensures the benchmark remains faithful to its original, rigorous intent.
Describe alternatives you've considered
No response
Feature category
Data generation
Priority
High - Essential for current project
Use case details
Testing JOB with current BenchBox will lead to a wrong result.
Additional context
No response