From d1e165d220c2ad5c199dbdac9f794148d8bf296d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 16 Jul 2026 03:21:30 +0000 Subject: [PATCH 1/3] docs(tuning): fix env contract, rewrite stale README, document precedence; honest auto-mode message Re-verified the 2026-07-12 tuning review findings (R5) against HEAD (post-#1172 resolver hints/keyword-first fixes) before implementing: - w1: BENCHBOX_TUNING_ENABLED set config key tuning.enabled, which nothing at runtime read except a unit test. Removed both env-var mapping mechanisms (module-level env_mappings dict and the ConfigManager default's environment_overrides entry) from benchbox/cli/config.py, the dead "enabled" default key, and the false "activates tuned runs in CI" doc claims in docs/usage/configuration.md and docs/reference/cli/configuration.md. BENCHBOX_TUNING_CONFIG and BENCHBOX_TUNING_PATH behavior is unchanged. Added a CHANGELOG entry. - w2: rewrote examples/tunings/README.md - the real flag is --tuning (not --tuning-config), the named example scripts don't exist (pointed at the real examples/unified_runner.py and examples/features/tuning_comparison.py instead), file layout matches examples/tunings//_tuned.yaml, auto-discovery via --tuning tuned is the primary UX, and dropped the unqualified "2-10x" performance claim. - w3: documented the actual --tuning resolution order in docs/reference/cli/tuning.md (default notuning -> keyword check before path check -> tuned's default_config_file/BENCHBOX_TUNING_CONFIG beats discovery -> BENCHBOX_TUNING_PATH -> cwd-relative examples/ -> cwd -> fallback with warning), the cwd-relative caveat for installed-package users, the tuning list/show/platforms subcommands, and the correct --output default (_tuning.yaml / DataFrame's __tuning.yaml, not tuning_config.yaml). - w4: --tuning auto now warns explicitly on SQL platforms that smart defaults are DataFrame-only today and the run proceeds with a basic config, instead of silently building an untuned config while claiming smart defaults were applied. Message change only, no behavior change. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QTnjb2i44JUL71kmu9sk1t --- CHANGELOG.md | 12 ++ ...ning-docs-env-contract-fixes-20260712.yaml | 145 +++++++++++++ ...ning-docs-env-contract-fixes-20260712.yaml | 85 -------- benchbox/cli/commands/run.py | 11 + benchbox/cli/config.py | 4 - benchbox/cli/tuning_resolver.py | 16 +- docs/reference/cli/configuration.md | 7 +- docs/reference/cli/tuning.md | 112 +++++++++- docs/usage/configuration.md | 3 +- examples/tunings/README.md | 201 +++++++++--------- 10 files changed, 398 insertions(+), 198 deletions(-) create mode 100644 _project/DONE/main/active/tuning-docs-env-contract-fixes-20260712.yaml delete mode 100644 _project/TODO/main/planning/tuning-docs-env-contract-fixes-20260712.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index ccd762ee3..d8f3daf97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Removed dead `BENCHBOX_TUNING_ENABLED` env var** - This variable set the + `tuning.enabled` config key, which nothing at runtime ever read (only a unit + test did); docs incorrectly claimed it "activates tuned runs in CI". Use + `--tuning tuned` / `--tuning auto` on `benchbox run` to actually enable + tuning; `BENCHBOX_TUNING_CONFIG` still works to point at a default tuning + file. If you were setting `BENCHBOX_TUNING_ENABLED`, it had no effect and can + simply be removed. +- **Honest `--tuning auto` messaging on SQL platforms** - Smart defaults for + `--tuning auto` are implemented for DataFrame platforms only today; SQL + platform runs now get an explicit warning that they're proceeding with a + basic (untuned) configuration instead of a message implying real smart + defaults were applied. - **`-df` platform names select DataFrame mode** - `benchbox run --platform datafusion-df` (and `lakesail-df`) now runs the DataFrame adapter without needing `--mode dataframe`; alias normalization previously dropped the diff --git a/_project/DONE/main/active/tuning-docs-env-contract-fixes-20260712.yaml b/_project/DONE/main/active/tuning-docs-env-contract-fixes-20260712.yaml new file mode 100644 index 000000000..5420ee453 --- /dev/null +++ b/_project/DONE/main/active/tuning-docs-env-contract-fixes-20260712.yaml @@ -0,0 +1,145 @@ +id: tuning-docs-env-contract-fixes-20260712 +title: "Fix broken tuning docs/env contract: dead env var, stale README, undocumented precedence, dishonest auto mode" +worktree: main +priority: Medium-High +status: Completed +completed_date: "2026-07-16" +category: Documentation + +description: | + From the 2026-07-12 tuning review, finding R5 (evidence pinned to commit acfb8992). + + - BENCHBOX_TUNING_ENABLED sets config key tuning.enabled which NOTHING at + runtime reads (only reader is a test); yet docs/usage/configuration.md:91-94 + and docs/reference/cli/configuration.md:93 claim it activates tuned runs in CI. + - examples/tunings/README.md is wholesale stale: documents a --tuning-config + flag that does not exist, example scripts that do not exist, stale flat + filenames, an env-var recipe that cannot work, and an unqualified "2-10x" + performance promise. + - The actual precedence order (flag default notuning -> keyword/path -> yaml + default_config_file [BENCHBOX_TUNING_CONFIG] -> BENCHBOX_TUNING_PATH -> + cwd-relative examples/ -> cwd -> fallback) is documented nowhere. + - --tuning auto on SQL platforms prints "using smart defaults based on system + profile" but delivers a default-constructed config (run.py:1021-1024); + real smart defaults exist only for DataFrame mode (tuning_runtime.py:49-92). + - docs/reference/cli/tuning.md: wrong --output default, three subcommands + (list/show/platforms) undocumented. + +work: + - id: w1 + summary: "Decide-in-PR the env var: remove the doc claims AND either delete BENCHBOX_TUNING_ENABLED handling or emit a deprecation warning when set (recommend delete + release note)" + status: done + needs: [] + notes: > + Re-verified at HEAD (706a64c3): tuning.enabled is set by two mechanisms + (module-level _apply_environment_overrides env_mappings dict, and the + ConfigManager instance's generic apply_environment_overrides driven by + the default config's tuning.environment_overrides dict) but consumed + nowhere at runtime except tests/unit/cli/test_config_coverage.py. + Deleted both mappings, the docstring bullet, and the dead + "enabled": False default key from benchbox/cli/config.py. Fixed doc + claims in docs/usage/configuration.md and + docs/reference/cli/configuration.md (env-var table, config-section + table, and example config block). Added a CHANGELOG.md [Unreleased] -> + Fixed entry. BENCHBOX_TUNING_CONFIG (tuning.default_config_file, read by + tuning_resolver.py:_resolve_tuned) is untouched and still documented. + No test asserted the removed default value, so no test changes were + needed; test_config_coverage.py's + test_instance_apply_environment_overrides_updates_tuning sets its own + synthetic environment_overrides mapping and still passes unchanged. + - id: w2 + summary: "Rewrite examples/tunings/README.md against reality: --tuning flag, real template paths, auto-discovery as the primary UX, drop unfalsifiable performance claims" + status: done + needs: [] + notes: > + Rewrote examples/tunings/README.md. Verified against HEAD: the real + flag is --tuning (no --tuning-config anywhere in the codebase); the + named example scripts (examples/duckdb_tpch.py etc.) do not exist, but + examples/unified_runner.py and examples/features/tuning_comparison.py + are real and do use --tuning correctly (noted unified_runner.py + defaults to "tuned", unlike the main CLI's "notuning"); file layout + confirmed via `find examples/tunings` (duckdb/, databricks/, and a flat + dataframe/_.yaml layout that is NOT auto-discovered); + dropped the unqualified "2-10x faster" claim in favor of "run both and + compare"; added BENCHBOX_TUNING_PATH and `benchbox tuning list/show` + usage. + - id: w3 + summary: "Write the precedence table into docs/reference/cli/tuning.md; document tuning list/show/platforms; fix --output default" + notes: > + Re-checked the review's verified rules against HEAD before writing, per + the note. Confirmed in benchbox/cli/tuning_resolver.py:resolve_tuning + that keyword checks (notuning/auto/tuned) run before the path-existence + check (post-#1172), that --tuning defaults to notuning + (cli/commands/run.py option default), that _resolve_tuned checks + tuning.default_config_file (BENCHBOX_TUNING_CONFIG override) before + template discovery, and that get_tuning_template_paths searches + BENCHBOX_TUNING_PATH -> cwd-relative examples/tunings/... -> cwd + /... in that order, falling back to a basic config with a + warning (+ wizard offer when interactive, per + run.py:_load_unified_tuning_config). Documented all of this plus the + cwd-relative caveat (review finding N8). Fixed the --output default + claim: SQL mode is `_tuning.yaml`, DataFrame mode is + `__tuning.yaml` (cli/tuning_group.py:78-82, + 109-115,137,169-171 - not tuning_config.yaml as previously documented). + Documented the list/show/platforms subcommands, including that `tuning + list` (unlike `--tuning tuned`) does NOT honor BENCHBOX_TUNING_PATH. + status: done + needs: [] + - id: w4 + summary: "Make --tuning auto honest on SQL platforms: warn that smart defaults are DataFrame-only and the run proceeds with basic config (message change only; behavior change gated on ADR-2)" + status: done + needs: [] + notes: > + Re-verified: resolve_dataframe_tuning_config (tuning_runtime.py:49) + returns real smart defaults only when mode_name == "dataframe"; SQL + platforms fall through to run.py:_load_unified_tuning_config's final + else-branch (still at lines ~1021-1024 at HEAD), which was previously + silently building a bare UnifiedTuningConfiguration() while the info + message claimed "using smart defaults based on system profile". Message + change only, no behavior change: reworded _resolve_auto's info message + in tuning_resolver.py to say "DataFrame platforms only today", and added + an explicit yellow console warning in run.py's else-branch, gated on + `tuning_resolution.mode == TuningMode.AUTO and s.resolved_mode != + "dataframe"` (s.resolved_mode is set earlier by _resolve_platform_mode, + confirmed via call order in the run command). + +deferred: + - summary: "Ship discoverable templates with the installed package (package resources) so --tuning tuned works outside a repo checkout" + reason: "Behavior/packaging change beyond a docs fix; needs its own sizing after the precedence doc makes the limitation visible (review finding N8)" + +deps: + needs: [] + +must_preserve: + - "BENCHBOX_TUNING_CONFIG and BENCHBOX_TUNING_PATH behavior exactly as implemented" + +anti_patterns: + - "Do not document aspirational behavior; every documented rule must have a file:line anchor at time of writing" + +scope_limit: + only_modify: + - "docs/**" + - "examples/tunings/README.md" + - "benchbox/cli/config.py" + - "benchbox/cli/commands/run.py" + - "benchbox/cli/tuning_resolver.py" + - "tests/unit/**" + +verification: + - description: "No doc references to the dead env var remain (or the var actually works)" + command: "grep -rn 'BENCHBOX_TUNING_ENABLED' docs/ benchbox/ | grep -v test" + expected_output: "Either zero doc hits + removed handling, or a functioning documented code path" + - description: "Docs build clean" + command: "make lint" + expected_output: "Passing" + +metadata: + owners: + - "benchbox-team" + tags: + - tuning + - documentation + - cli + estimated_effort: Medium + created_date: "2026-07-12" + source_review: "tuning-system deep review 2026-07-12, finding R5" diff --git a/_project/TODO/main/planning/tuning-docs-env-contract-fixes-20260712.yaml b/_project/TODO/main/planning/tuning-docs-env-contract-fixes-20260712.yaml deleted file mode 100644 index b76aa2b98..000000000 --- a/_project/TODO/main/planning/tuning-docs-env-contract-fixes-20260712.yaml +++ /dev/null @@ -1,85 +0,0 @@ -id: tuning-docs-env-contract-fixes-20260712 -title: "Fix broken tuning docs/env contract: dead env var, stale README, undocumented precedence, dishonest auto mode" -worktree: main -priority: Medium-High -status: Not Started -category: Documentation - -description: | - From the 2026-07-12 tuning review, finding R5 (evidence pinned to commit acfb8992). - - - BENCHBOX_TUNING_ENABLED sets config key tuning.enabled which NOTHING at - runtime reads (only reader is a test); yet docs/usage/configuration.md:91-94 - and docs/reference/cli/configuration.md:93 claim it activates tuned runs in CI. - - examples/tunings/README.md is wholesale stale: documents a --tuning-config - flag that does not exist, example scripts that do not exist, stale flat - filenames, an env-var recipe that cannot work, and an unqualified "2-10x" - performance promise. - - The actual precedence order (flag default notuning -> keyword/path -> yaml - default_config_file [BENCHBOX_TUNING_CONFIG] -> BENCHBOX_TUNING_PATH -> - cwd-relative examples/ -> cwd -> fallback) is documented nowhere. - - --tuning auto on SQL platforms prints "using smart defaults based on system - profile" but delivers a default-constructed config (run.py:1021-1024); - real smart defaults exist only for DataFrame mode (tuning_runtime.py:49-92). - - docs/reference/cli/tuning.md: wrong --output default, three subcommands - (list/show/platforms) undocumented. - -work: - - id: w1 - summary: "Decide-in-PR the env var: remove the doc claims AND either delete BENCHBOX_TUNING_ENABLED handling or emit a deprecation warning when set (recommend delete + release note)" - status: pending - needs: [] - - id: w2 - summary: "Rewrite examples/tunings/README.md against reality: --tuning flag, real template paths, auto-discovery as the primary UX, drop unfalsifiable performance claims" - status: pending - needs: [] - - id: w3 - summary: "Write the precedence table into docs/reference/cli/tuning.md; document tuning list/show/platforms; fix --output default" - notes: "Re-check the review's verified rules against HEAD before writing. Explicitly document that examples/ discovery is cwd-relative, so installed-package users hit the fallback unless BENCHBOX_TUNING_PATH is set (review finding N8)." - status: pending - needs: [] - - id: w4 - summary: "Make --tuning auto honest on SQL platforms: warn that smart defaults are DataFrame-only and the run proceeds with basic config (message change only; behavior change gated on ADR-2)" - status: pending - needs: [] - -deferred: - - summary: "Ship discoverable templates with the installed package (package resources) so --tuning tuned works outside a repo checkout" - reason: "Behavior/packaging change beyond a docs fix; needs its own sizing after the precedence doc makes the limitation visible (review finding N8)" - -deps: - needs: [] - -must_preserve: - - "BENCHBOX_TUNING_CONFIG and BENCHBOX_TUNING_PATH behavior exactly as implemented" - -anti_patterns: - - "Do not document aspirational behavior; every documented rule must have a file:line anchor at time of writing" - -scope_limit: - only_modify: - - "docs/**" - - "examples/tunings/README.md" - - "benchbox/cli/config.py" - - "benchbox/cli/commands/run.py" - - "benchbox/cli/tuning_resolver.py" - - "tests/unit/**" - -verification: - - description: "No doc references to the dead env var remain (or the var actually works)" - command: "grep -rn 'BENCHBOX_TUNING_ENABLED' docs/ benchbox/ | grep -v test" - expected_output: "Either zero doc hits + removed handling, or a functioning documented code path" - - description: "Docs build clean" - command: "make lint" - expected_output: "Passing" - -metadata: - owners: - - "benchbox-team" - tags: - - tuning - - documentation - - cli - estimated_effort: Medium - created_date: "2026-07-12" - source_review: "tuning-system deep review 2026-07-12, finding R5" diff --git a/benchbox/cli/commands/run.py b/benchbox/cli/commands/run.py index 352b93586..b43850e37 100644 --- a/benchbox/cli/commands/run.py +++ b/benchbox/cli/commands/run.py @@ -1020,6 +1020,17 @@ def _load_unified_tuning_config(s: types.SimpleNamespace) -> None: s.logger.debug("Using basic constraints-only configuration (fallback)") else: s.loaded_unified_config = UnifiedTuningConfiguration() + if tuning_resolution.mode == TuningMode.AUTO and s.resolved_mode != "dataframe": + # Real smart defaults (resolve_dataframe_tuning_config in tuning_runtime.py) + # only exist for DataFrame platforms today; SQL platforms land here with a + # freshly-constructed, untuned UnifiedTuningConfiguration. + if not s.quiet: + console.print( + "[yellow]Warning: --tuning auto smart defaults are DataFrame-only today; " + "this SQL run proceeds with a basic (untuned) configuration.[/yellow]" + ) + if s.logger: + s.logger.debug("Tuning mode auto on a non-DataFrame platform: using basic unified config") if s.logger: s.logger.debug(f"Using basic unified config for mode: {tuning_resolution.mode.value}") diff --git a/benchbox/cli/config.py b/benchbox/cli/config.py index 65a0ab064..bb5267503 100644 --- a/benchbox/cli/config.py +++ b/benchbox/cli/config.py @@ -128,7 +128,6 @@ def _apply_environment_overrides(config_dict: dict[str, Any]) -> dict[str, Any]: - BENCHBOX_SCALE_FACTOR: Override benchmarks.default_scale - BENCHBOX_VERBOSE: Override execution.verbose - BENCHBOX_MAX_WORKERS: Override execution.max_workers - - BENCHBOX_TUNING_ENABLED: Override tuning.enabled - BENCHBOX_TUNING_CONFIG: Override tuning.default_config_file Args: @@ -143,7 +142,6 @@ def _apply_environment_overrides(config_dict: dict[str, Any]) -> dict[str, Any]: "BENCHBOX_SCALE_FACTOR": ("benchmarks", "default_scale", float), "BENCHBOX_VERBOSE": ("execution", "verbose", lambda v: v.lower() in ["true", "1", "yes", "on"]), "BENCHBOX_MAX_WORKERS": ("execution", "max_workers", int), - "BENCHBOX_TUNING_ENABLED": ("tuning", "enabled", lambda v: v.lower() in ["true", "1", "yes", "on"]), "BENCHBOX_TUNING_CONFIG": ("tuning", "default_config_file", str), "BENCHBOX_OUTPUT_DIR": ("output", "directory", str), "BENCHBOX_MEMORY_LIMIT_GB": ("execution", "memory_limit_gb", int), @@ -285,12 +283,10 @@ def _get_default_config(self) -> BenchBoxConfig: }, }, tuning={ - "enabled": False, "default_config_file": None, "validate_on_load": True, "allow_platform_incompatible": False, "environment_overrides": { - "BENCHBOX_TUNING_ENABLED": "enabled", "BENCHBOX_TUNING_CONFIG": "default_config_file", }, }, diff --git a/benchbox/cli/tuning_resolver.py b/benchbox/cli/tuning_resolver.py index 3eed9299b..cb2b88c68 100644 --- a/benchbox/cli/tuning_resolver.py +++ b/benchbox/cli/tuning_resolver.py @@ -228,15 +228,25 @@ def _resolve_notuning(logger: Logger | None) -> TuningResolution: def _resolve_auto(logger: Logger | None) -> TuningResolution: - """Resolve auto (smart defaults) mode.""" + """Resolve auto (smart defaults) mode. + + Real system-profile smart defaults are only implemented for DataFrame + platforms today (see ``resolve_dataframe_tuning_config`` in + ``tuning_runtime.py``); SQL platforms fall back to a basic, untuned + configuration. This function does not know the resolved platform/mode, so + the concrete "SQL gets a basic config" warning is emitted downstream once + that is known (see ``run.py::_load_unified_tuning_config``). + """ resolution = TuningResolution( mode=TuningMode.AUTO, source=TuningSource.SMART_DEFAULTS, enabled=True, ) - resolution.info_messages.append("Tuning mode: auto (using smart defaults based on system profile)") + resolution.info_messages.append( + "Tuning mode: auto (smart defaults based on system profile - DataFrame platforms only today)" + ) if logger: - logger.debug("Tuning mode: auto (smart defaults)") + logger.debug("Tuning mode: auto (smart defaults; DataFrame platforms only)") return resolution diff --git a/docs/reference/cli/configuration.md b/docs/reference/cli/configuration.md index 21ca707d6..c84011a36 100644 --- a/docs/reference/cli/configuration.md +++ b/docs/reference/cli/configuration.md @@ -90,8 +90,7 @@ These variables override the corresponding settings in the configuration file (e | `BENCHBOX_SCALE_FACTOR` | `benchmarks.default_scale` | float | `0.01` | Default scale factor | | `BENCHBOX_VERBOSE` | `execution.verbose` | boolean | `true` | Enable verbose output (`true`/`1`/`yes`/`on`) | | `BENCHBOX_MAX_WORKERS` | `execution.max_workers` | integer | `4` | Maximum parallel worker threads | -| `BENCHBOX_TUNING_ENABLED` | `tuning.enabled` | boolean | `false` | Enable table tuning by default | -| `BENCHBOX_TUNING_CONFIG` | `tuning.default_config_file` | string | - | Path to tuning configuration file | +| `BENCHBOX_TUNING_CONFIG` | `tuning.default_config_file` | string | - | Path to tuning configuration file. Used when `--tuning tuned` resolves with no explicit file (see [Tuning Commands](tuning.md)) | | `BENCHBOX_OUTPUT_DIR` | `output.directory` | string | `./benchmark_runs/results` | Output directory for result files | | `BENCHBOX_MEMORY_LIMIT_GB` | `execution.memory_limit_gb` | integer | `0` (auto) | Memory limit in GB; `0` means no limit | @@ -306,8 +305,7 @@ Table tuning and optimization settings. | Setting | Type | Default | Description | |---------|------|---------|-------------| -| `enabled` | boolean | `false` | Apply tuning configurations by default | -| `default_config_file` | string | `null` | Path to a default tuning YAML file | +| `default_config_file` | string | `null` | Default tuning YAML file used when `--tuning tuned` finds no explicit file; see [Tuning Commands](tuning.md) for the full resolution order | | `validate_on_load` | boolean | `true` | Validate tuning configurations when loaded | | `allow_platform_incompatible` | boolean | `false` | Allow tuning configs that contain platform-incompatible directives | @@ -341,7 +339,6 @@ execution: warm_up_iterations: 2 tuning: - enabled: true default_config_file: ./tuning/my_tuning.yaml ``` diff --git a/docs/reference/cli/tuning.md b/docs/reference/cli/tuning.md index 8b44b9c89..30aaacc3e 100644 --- a/docs/reference/cli/tuning.md +++ b/docs/reference/cli/tuning.md @@ -5,6 +5,62 @@ Commands for creating and managing tuning configurations for SQL and DataFrame platforms. +(cli-tuning-resolution)= +## How `--tuning` is resolved + +`benchbox run --tuning VALUE` accepts the keywords `notuning`, `tuned`, `auto`, +or an explicit path to a YAML file. If `--tuning` is omitted it **defaults to +`notuning`** (baseline, all optimizations disabled). + +Resolution happens in this order (see `benchbox/cli/tuning_resolver.py:resolve_tuning`): + +0. **Default**: no `--tuning` given -> `notuning`. +1. **Keyword check, before any file lookup**: if the value is exactly + `notuning`, `auto`, or `tuned` (case-insensitive), it is resolved as that + keyword immediately - a local file or directory that happens to be named + `tuned` or `auto` can never shadow the keyword's meaning. + - `notuning` -> baseline: all constraints disabled. + - `auto` -> smart-defaults mode (see the DataFrame-only caveat below). +2. **`tuned` keyword resolution** (`_resolve_tuned`): + 1. If `benchbox.yaml`'s `tuning.default_config_file` is set (overridable + with the `BENCHBOX_TUNING_CONFIG` environment variable) and that file + exists, it is used - this beats template auto-discovery. + 2. Otherwise, auto-discovery searches, in order: + 1. `$BENCHBOX_TUNING_PATH//_tuned.yaml`, if + `BENCHBOX_TUNING_PATH` is set. + 2. `examples/tunings//_tuned.yaml`, resolved + relative to the **current working directory**. + 3. `/_tuned.yaml`, also cwd-relative. + 3. If none of those exist, tuning falls back to a basic, untuned + configuration and prints a warning; in an interactive terminal the user + is also offered the tuning wizard. +3. **Explicit file path**: if the value is not one of the three keywords, it + is checked as a path; if it exists, that file is loaded directly. +4. **Invalid value**: if it's not a keyword and the path doesn't exist, + `benchbox run` raises an error (with a hint to run `benchbox tuning list`). + +```{note} +Step 2.2 is **cwd-relative**. `--tuning tuned` only auto-discovers the +templates bundled in `examples/tunings/` when `benchbox` is run from a +checkout of this repository (or another directory with its own +`examples/tunings/`). If BenchBox is installed as a package and run from +elsewhere, auto-discovery falls through to the basic-config fallback unless +`BENCHBOX_TUNING_PATH` is set to a directory with the same +`/_tuned.yaml` layout. +``` + +```{warning} +`--tuning auto` prints a "smart defaults" message, but real system-profile +smart defaults (`benchbox/cli/tuning_runtime.py:resolve_dataframe_tuning_config`) +are implemented for **DataFrame platforms only** today. On SQL platforms, +`--tuning auto` proceeds with a basic, untuned configuration and +`benchbox run` prints an explicit warning saying so. +``` + +`BENCHBOX_TUNING_CONFIG` and `BENCHBOX_TUNING_PATH` are the only tuning-related +environment variables that affect resolution; see +[Configuration - Environment Variables](configuration.md#cli-env-vars). + (cli-tuning-init)= ## `tuning init` - Create Tuning Configuration @@ -13,12 +69,14 @@ Generate sample unified tuning configurations for specific platforms. ### Options - `--platform TEXT`: Target platform (required) - duckdb, databricks, snowflake, etc. -- `--output TEXT`: Output file path (default: tuning_config.yaml) +- `--output TEXT`: Output file path. Default depends on mode: + - SQL mode: `_tuning.yaml` (e.g. `duckdb_tuning.yaml`) + - DataFrame mode: `__tuning.yaml` (e.g. `polars_optimized_tuning.yaml`) ### Usage Examples ```bash -# Create sample tuning for Databricks +# Create sample tuning for Databricks (writes databricks_tuning.yaml) benchbox tuning init --platform databricks # Create with custom output path @@ -84,6 +142,56 @@ DataFrame tuning supports these configuration sections: For complete configuration reference, see [DataFrame Platforms - Tuning](../../platforms/dataframe.md#dataframe-tuning). +(cli-tuning-list)= +## `tuning list` - List Available Templates + +List the tuning templates found under `examples/tunings/` (cwd-relative, +same as `--tuning tuned` auto-discovery step 2.2 above), optionally filtered +by platform and/or benchmark. Unlike `--tuning tuned`, `tuning list` does +**not** consult `BENCHBOX_TUNING_PATH`. + +```bash +benchbox tuning list +benchbox tuning list --platform duckdb +benchbox tuning list --platform duckdb --benchmark tpch +benchbox tuning list --benchmark tpcds +``` + +### Options + +- `--platform TEXT`: Filter to a specific platform (e.g. `duckdb`, `snowflake`) +- `--benchmark TEXT`: Filter to a specific benchmark (e.g. `tpch`, `tpcds`) + +(cli-tuning-show)= +## `tuning show` - Show Resolved Tuning Configuration + +Show what a given `--tuning` value would actually resolve to for a +platform/benchmark - useful for checking which file (if any) `--tuning tuned` +or `--tuning auto` would pick up before running a full benchmark. + +```bash +benchbox tuning show tuned --platform duckdb --benchmark tpch +benchbox tuning show ./my-tuning.yaml +benchbox tuning show auto --platform polars +benchbox tuning show notuning +``` + +### Arguments and Options + +- `TUNING_ARG` (positional, default: `tuned`): `tuned`, `notuning`, `auto`, or a file path +- `--platform TEXT`: Target platform, used for template discovery +- `--benchmark TEXT`: Target benchmark, used for template discovery + +(cli-tuning-platforms)= +## `tuning platforms` - List Platform Tuning Capabilities + +Show which SQL and DataFrame platforms support tuning and a summary of their +key tuning options; does not take any options. + +```bash +benchbox tuning platforms +``` + ## Related - [Run Command](run.md) - Using `--tuning` option during benchmark execution diff --git a/docs/usage/configuration.md b/docs/usage/configuration.md index 1fd318056..6a2d53b8c 100644 --- a/docs/usage/configuration.md +++ b/docs/usage/configuration.md @@ -88,10 +88,9 @@ Some tuning settings can be toggled at deploy time without editing files. The de | Environment variable | Maps to | Typical values | | --- | --- | --- | -| `BENCHBOX_TUNING_ENABLED` | `tuning.enabled` | `true`, `false` | | `BENCHBOX_TUNING_CONFIG` | `tuning.default_config_file` | Path to a YAML file | -Set `BENCHBOX_TUNING_ENABLED=true` to activate tuned runs in CI, or point `BENCHBOX_TUNING_CONFIG` at a configuration checked into your repo. +Point `BENCHBOX_TUNING_CONFIG` at a configuration checked into your repo to make it the default whenever `--tuning tuned` is used (see [Tuning Commands](../reference/cli/tuning.md) for the full `--tuning` resolution order). To actually enable tuned runs in CI, pass `--tuning tuned` (or `--tuning auto`) on the `benchbox run` command line; there is no environment variable that turns tuning on by itself. ## Validating Configuration diff --git a/examples/tunings/README.md b/examples/tunings/README.md index b0ca11c3a..8adb5da92 100644 --- a/examples/tunings/README.md +++ b/examples/tunings/README.md @@ -1,117 +1,132 @@ -# Tuning Configuration Guide +# Tuning Configuration Examples -This directory contains pre-built tuning configuration files for all benchmark examples in BenchBox. These configurations demonstrate performance comparison between optimized and baseline configurations. +This directory contains example tuning configuration YAML files consumed by +`benchbox run --tuning`. They demonstrate a "tuned" configuration (constraints, +partitioning/sorting, platform-specific optimizations) against a baseline +"notuning" configuration for the same platform/benchmark pair. -## Available Configurations +## File layout -### DuckDB Configurations +SQL platforms follow `examples/tunings//_tuned.yaml` (and +a matching `_notuning.yaml`): -#### TPC-H Benchmark -- `duckdb_tpch_tuned.yaml` - Maximum performance with all DuckDB-supported optimizations -- `duckdb_tpch_notuning.yaml` - Baseline performance with all optimizations disabled +- `duckdb/` - `tpch`, `tpcds`, `clickbench`, `ssb`, `amplab`, `h2odb`, + `read_primitives`, `joinorder`, `tpchavoc` +- `databricks/` - `tpch`, `tpcds`, `ssb`, `read_primitives`, `tpchavoc`, plus + `tpch_liquid_tuned.yaml` / `tpcds_liquid_tuned.yaml` (Liquid Clustering AUTO + variants of the same logical profile, alongside the legacy Z-ORDER + `tpch_tuned.yaml` / `tpcds_tuned.yaml`) -#### TPC-DS Benchmark -- `duckdb_tpcds_tuned.yaml` - Maximum performance with partitioning on date columns and sorting -- `duckdb_tpcds_notuning.yaml` - Baseline performance for comparison +DataFrame platforms live under `dataframe/` with a flat +`_.yaml` naming (e.g. `polars_optimized.yaml`, +`dask_memory_constrained.yaml`, `cudf_default.yaml`). These are **not** +auto-discovered (see below) - pass the path to `--tuning` explicitly. -#### ClickBench Benchmark -- `duckdb_clickbench_tuned.yaml` - Optimized for web analytics workloads with date partitioning -- `duckdb_clickbench_notuning.yaml` - Unoptimized baseline +## Using with the CLI -#### Star Schema Benchmark (SSB) -- `duckdb_ssb_tuned.yaml` - Star schema optimizations with fact table partitioning -- `duckdb_ssb_notuning.yaml` - Baseline configuration +The `benchbox run` flag is `--tuning` (there is no `--tuning-config` flag). +It accepts one of the keywords `tuned`, `notuning`, `auto`, or an explicit +path to a YAML file; it defaults to `notuning` when omitted. -#### AMPLab Big Data Benchmark -- `duckdb_amplab_tuned.yaml` - Optimized for big data queries with sorting on key columns -- `duckdb_amplab_notuning.yaml` - Baseline configuration - -#### H2O.ai Database Benchmark -- `duckdb_h2odb_tuned.yaml` - Optimized for data science workloads -- `duckdb_h2odb_notuning.yaml` - Baseline configuration - -#### Read Primitives Benchmark -- `duckdb_primitives_tuned.yaml` - Basic OLAP optimizations using TPC-H schema -- `duckdb_primitives_notuning.yaml` - Baseline configuration +```bash +# Auto-discover the platform/benchmark tuned template (the primary UX) +benchbox run --platform duckdb --benchmark tpch --tuning tuned -#### Write Primitives Benchmark -- `duckdb_primitives_tuned.yaml` - Basic OLAP optimizations using TPC-H schema -- `duckdb_primitives_notuning.yaml` - Baseline configuration +# Explicit baseline (no tuning) +benchbox run --platform duckdb --benchmark tpch --tuning notuning -#### Join Order Benchmark -- `duckdb_joinorder_tuned.yaml` - Optimized for complex join queries -- `duckdb_joinorder_notuning.yaml` - Baseline configuration +# Point directly at a file +benchbox run --platform duckdb --benchmark tpch \ + --tuning examples/tunings/duckdb/tpch_tuned.yaml -#### TPC-Havoc Benchmark -- `duckdb_tpchavoc_tuned.yaml` - Optimized for query optimizer stress testing -- `duckdb_tpchavoc_notuning.yaml` - Baseline configuration +# DataFrame platform - these files must be referenced explicitly +benchbox run --platform polars --benchmark tpch --mode dataframe \ + --tuning examples/tunings/dataframe/polars_optimized.yaml +``` -### Databricks Configurations +`examples/unified_runner.py` is a lighter-weight alternative to the `benchbox` +CLI for scripting/automation; it accepts the same `--tuning` values but +**defaults to `tuned`** (the main CLI defaults to `notuning`): -#### TPC-H Benchmark -- `databricks/tpch_tuned.yaml` - Legacy Delta Lake Z-ORDER rendering, auto-optimize, and bloom filters -- `databricks/tpch_liquid_tuned.yaml` - Liquid Clustering AUTO rendering of the same logical TPC profile -- `databricks/tpch_notuning.yaml` - Baseline performance +```bash +python examples/unified_runner.py --platform duckdb --benchmark tpch --scale 0.1 --tuning notuning +``` -#### TPC-DS Benchmark -- `databricks/tpcds_tuned.yaml` - Legacy Delta Lake Z-ORDER rendering for large-scale analytics -- `databricks/tpcds_liquid_tuned.yaml` - Liquid Clustering AUTO rendering of the same logical TPC profile -- `databricks/tpcds_notuning.yaml` - Baseline configuration +See `examples/features/tuning_comparison.py` for a runnable walkthrough of +comparing baseline vs. tuned performance. -#### TPC-Havoc Benchmark -- `databricks_tpchavoc_tuned.yaml` - Delta Lake optimizations for optimizer stress testing -- `databricks_tpchavoc_notuning.yaml` - Baseline configuration +## Auto-discovery (`--tuning tuned`) -#### Star Schema Benchmark (SSB) -- `databricks_ssb_tuned.yaml` - Delta Lake star schema optimizations -- `databricks_ssb_notuning.yaml` - Baseline configuration +`--tuning tuned` is the primary day-to-day UX: you don't reference a path, +BenchBox finds the matching template for you. The full resolution order +(covering every `--tuning` value, not just `tuned`) is documented in +[docs/reference/cli/tuning.md](../../docs/reference/cli/tuning.md); the part +relevant to these templates is: -#### Primitives Benchmark -- `databricks_primitives_tuned.yaml` - Delta Lake basic OLAP optimizations -- `databricks_primitives_notuning.yaml` - Baseline configuration +1. `benchbox.yaml`'s `tuning.default_config_file` (overridable with the + `BENCHBOX_TUNING_CONFIG` environment variable), if set and the file exists. +2. `$BENCHBOX_TUNING_PATH//_tuned.yaml`, if + `BENCHBOX_TUNING_PATH` is set. +3. `examples/tunings//_tuned.yaml`, resolved relative to + the current working directory. +4. `/_tuned.yaml`, also resolved relative to cwd. +5. If none of the above exist, the run falls back to a basic, untuned + configuration and prints a warning (an interactive terminal is also + offered the tuning wizard). -## Usage +Step 3 is **cwd-relative**, so `--tuning tuned` only auto-discovers the +templates in this directory when `benchbox` is run from a checkout of this +repository (or another directory that contains its own `examples/tunings/`). +If BenchBox is installed as a package and run elsewhere, set +`BENCHBOX_TUNING_PATH` to a directory with the same +`/_tuned.yaml` layout - see the next section. -### Using with Example Scripts +## Custom tuning directory - `BENCHBOX_TUNING_PATH` -All example scripts support loading tuning configurations via the `--tuning-config` parameter: +Point `--tuning tuned` at a different template collection without relying on +the working directory: ```bash -# Run with tuned configuration -python examples/duckdb_tpch.py --scale 0.01 --tuning-config examples/duckdb_tpch_tuned.yaml +# Directory must use the same /_tuned.yaml layout +export BENCHBOX_TUNING_PATH=/path/to/my-tunings +benchbox run --platform duckdb --benchmark tpch --tuning tuned +``` -# Run with no-tuning baseline -python examples/duckdb_tpch.py --scale 0.01 --tuning-config examples/duckdb_tpch_notuning.yaml +## Default file via `benchbox.yaml` / `BENCHBOX_TUNING_CONFIG` -# Compare performance between configurations -python examples/duckdb_tpch.py --scale 0.01 --tuning-config examples/duckdb_tpch_tuned.yaml --save-results tuned_results.json -python examples/duckdb_tpch.py --scale 0.01 --tuning-config examples/duckdb_tpch_notuning.yaml --save-results baseline_results.json -``` +Set a default file that `--tuning tuned` uses before falling back to +auto-discovery: -### Using with CLI +```yaml +# benchbox.yaml +tuning: + default_config_file: ./tuning/my_tuning.yaml +``` -The BenchBox CLI also supports tuning configurations: +or override it at runtime without editing the file: ```bash -# Run benchmark with tuning configuration -benchbox run --platform duckdb --benchmark tpch --scale 0.01 --tuning-config examples/duckdb_tpch_tuned.yaml - -# Compare different configurations -benchbox run --platform duckdb --benchmark tpch --scale 0.01 --tuning-config examples/duckdb_tpch_notuning.yaml +export BENCHBOX_TUNING_CONFIG=./tuning/my_tuning.yaml +benchbox run --platform duckdb --benchmark tpch --tuning tuned ``` -### Environment Variable - -You can also set a default tuning configuration using an environment variable: +## Inspecting templates ```bash -export BENCHBOX_TUNING_CONFIG=examples/duckdb_tpch_tuned.yaml -python examples/duckdb_tpch.py --scale 0.01 +# List everything available under examples/tunings/ +benchbox tuning list + +# Filter by platform and/or benchmark +benchbox tuning list --platform duckdb --benchmark tpch + +# Show what --tuning would actually resolve to (including which file, if any) +benchbox tuning show tuned --platform duckdb --benchmark tpch ``` -## Configuration Structure +## Configuration structure -All tuning configurations follow the unified tuning format with these sections: +All tuning configurations follow the unified tuning format with these +sections: ### Constraint Configuration - `primary_keys` - Primary key constraint settings @@ -164,8 +179,7 @@ basic-constraints fallback as equivalent tuned runs. ### Tuned Configurations - Enable all appropriate constraints (primary keys, foreign keys, unique constraints, check constraints) - Include table-level optimizations (partitioning on date columns, sorting on key columns) -- Enable platform-specific features (Databricks: Z-ordering, auto-optimize, bloom filters) -- Optimized for maximum query performance +- Enable platform-specific features (Databricks: Z-ordering/Liquid Clustering, auto-optimize, bloom filters) ### No-Tuning Configurations - Disable all constraints for fastest data loading @@ -173,24 +187,15 @@ basic-constraints fallback as equivalent tuned runs. - No platform-specific features enabled - Provide baseline performance for comparison -## Performance Impact - -Tuned configurations typically provide: -- **Query Performance**: 2-10x faster query execution for analytical workloads -- **Data Loading**: Slower initial loading due to constraint validation and optimization -- **Storage**: More efficient storage with partitioning and compression -- **Maintenance**: Better long-term performance with auto-optimization features - -No-tuning configurations provide: -- **Data Loading**: Fastest possible data loading with no overhead -- **Query Performance**: Baseline performance without optimizations -- **Consistency**: Predictable performance for benchmarking baseline -- **Simplicity**: No complex optimization dependencies +Actual performance impact varies by platform, benchmark, and data volume; run +both configurations yourself with `benchbox run --tuning tuned` / +`--tuning notuning` and compare results rather than relying on a fixed +multiplier. ## Best Practices -1. **Development and Testing**: Use no-tuning configurations for fast iteration -2. **Performance Evaluation**: Use tuned configurations for realistic production performance +1. **Development and Testing**: Use `--tuning notuning` for fast iteration +2. **Performance Evaluation**: Use `--tuning tuned` for realistic production performance 3. **Benchmarking**: Compare both configurations to understand optimization impact 4. **Production**: Adapt tuned configurations to your specific workload requirements @@ -223,4 +228,6 @@ table_tunings: order: 2 ``` -For more information about the unified tuning system, see the main BenchBox documentation. +For the full `--tuning` precedence order and the `tuning` command group +(`init`, `validate`, `defaults`, `list`, `show`, `platforms`), see +[docs/reference/cli/tuning.md](../../docs/reference/cli/tuning.md). From 9a1cbc05a0b171b953f9a153be4bb7ccded2ca00 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 16 Jul 2026 12:52:34 +0000 Subject: [PATCH 2/3] fix(cli): relocate SQL auto-mode warning to resolver; merge develop Merged origin/develop (PR #1172's module-size allowlist bump to 2946, plus #1174/#1175/#1170/#1177 tuning soundness/explorer work) - merge was conflict-free. The auto-mode warning added in the previous commit pushed benchbox/cli/commands/run.py to 2957 lines, 11 over the just-bumped 2946 limit in tests/system/test_module_size_thresholds.py. Relocated the warning logic to benchbox/cli/tuning_resolver.py (which already owns tuning messaging via display_tuning_resolution) as a new warn_sql_auto_mode() function, and replaced the ~13-line inline block in run.py with a single call. Folded the branch's existing generic "using basic unified config" debug log into the same function (it only ever fires alongside the auto-mode case), which brought run.py to exactly 2946 lines without bumping the allowlist. Added unit coverage for warn_sql_auto_mode in test_tuning_resolution.py: warns on SQL platforms, silent on DataFrame platforms and non-auto modes, respects quiet=True, and always logs the debug line. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01QTnjb2i44JUL71kmu9sk1t --- benchbox/cli/commands/run.py | 15 +------ benchbox/cli/tuning_resolver.py | 30 ++++++++++++++ tests/unit/cli/test_tuning_resolution.py | 53 ++++++++++++++++++++++++ 3 files changed, 85 insertions(+), 13 deletions(-) diff --git a/benchbox/cli/commands/run.py b/benchbox/cli/commands/run.py index b43850e37..2fe86a835 100644 --- a/benchbox/cli/commands/run.py +++ b/benchbox/cli/commands/run.py @@ -55,6 +55,7 @@ TuningSource, display_tuning_resolution, resolve_tuning, + warn_sql_auto_mode, ) from benchbox.cli.tuning_runtime import ( build_baseline_unified_config, @@ -1020,19 +1021,7 @@ def _load_unified_tuning_config(s: types.SimpleNamespace) -> None: s.logger.debug("Using basic constraints-only configuration (fallback)") else: s.loaded_unified_config = UnifiedTuningConfiguration() - if tuning_resolution.mode == TuningMode.AUTO and s.resolved_mode != "dataframe": - # Real smart defaults (resolve_dataframe_tuning_config in tuning_runtime.py) - # only exist for DataFrame platforms today; SQL platforms land here with a - # freshly-constructed, untuned UnifiedTuningConfiguration. - if not s.quiet: - console.print( - "[yellow]Warning: --tuning auto smart defaults are DataFrame-only today; " - "this SQL run proceeds with a basic (untuned) configuration.[/yellow]" - ) - if s.logger: - s.logger.debug("Tuning mode auto on a non-DataFrame platform: using basic unified config") - if s.logger: - s.logger.debug(f"Using basic unified config for mode: {tuning_resolution.mode.value}") + warn_sql_auto_mode(tuning_resolution, s.resolved_mode, console, s.logger, quiet=bool(s.quiet)) def _resolve_data_organization(s: types.SimpleNamespace) -> None: diff --git a/benchbox/cli/tuning_resolver.py b/benchbox/cli/tuning_resolver.py index cb2b88c68..c229ab696 100644 --- a/benchbox/cli/tuning_resolver.py +++ b/benchbox/cli/tuning_resolver.py @@ -374,6 +374,36 @@ def display_tuning_resolution( console.print(f" [dim]{path}[/dim] ({exists_marker})") +def warn_sql_auto_mode( + resolution: TuningResolution, + resolved_mode: str | None, + console: Console, + logger: Logger | None = None, + quiet: bool = False, +) -> None: + """Log and, on SQL platforms, warn about a basic-config `--tuning auto` resolution. + + Callers reach here whenever ``_load_unified_tuning_config`` falls through + to a basic, freshly-constructed ``UnifiedTuningConfiguration`` (the only + resolution source that does so is ``TuningMode.AUTO``). Real + system-profile smart defaults (``resolve_dataframe_tuning_config`` in + ``tuning_runtime.py``) only exist for DataFrame platforms today, so SQL + platforms get an explicit warning instead of silently proceeding while the + `auto` info message implies real smart defaults were applied. + """ + if logger: + logger.debug(f"Using basic unified config for mode: {resolution.mode.value}") + if resolution.mode != TuningMode.AUTO or resolved_mode == "dataframe": + return + if not quiet: + console.print( + "[yellow]Warning: --tuning auto smart defaults are DataFrame-only today; " + "this SQL run proceeds with a basic (untuned) configuration.[/yellow]" + ) + if logger: + logger.debug("Tuning mode auto on a non-DataFrame platform: using basic unified config") + + def display_tuning_list( console: Console, platform: str | None = None, diff --git a/tests/unit/cli/test_tuning_resolution.py b/tests/unit/cli/test_tuning_resolution.py index 28c3d11d1..278728b91 100644 --- a/tests/unit/cli/test_tuning_resolution.py +++ b/tests/unit/cli/test_tuning_resolution.py @@ -25,6 +25,7 @@ get_tuning_template_paths, list_available_tuning_templates, resolve_tuning, + warn_sql_auto_mode, ) pytestmark = [ @@ -438,6 +439,58 @@ def test_display_tuning_list_filtered(self, mock_console): assert mock_console.print.called + +@pytest.mark.unit +class TestWarnSqlAutoMode: + """Test warn_sql_auto_mode: the honest --tuning auto messaging on SQL platforms.""" + + def _auto_resolution(self) -> TuningResolution: + return TuningResolution( + mode=TuningMode.AUTO, + source=TuningSource.SMART_DEFAULTS, + enabled=True, + ) + + def test_warns_on_sql_platform(self, mock_console): + """SQL platforms resolving --tuning auto should get an explicit warning.""" + warn_sql_auto_mode(self._auto_resolution(), resolved_mode="sql", console=mock_console) + + assert mock_console.print.called + printed = " ".join(str(call) for call in mock_console.print.call_args_list) + assert "DataFrame-only" in printed + + def test_no_warning_on_dataframe_platform(self, mock_console): + """DataFrame platforms have real smart defaults elsewhere; no warning needed.""" + warn_sql_auto_mode(self._auto_resolution(), resolved_mode="dataframe", console=mock_console) + + assert not mock_console.print.called + + def test_no_warning_for_non_auto_mode(self, mock_console): + """Only TuningMode.AUTO triggers the warning, regardless of resolved_mode.""" + resolution = TuningResolution( + mode=TuningMode.NOTUNING, + source=TuningSource.BASELINE, + enabled=False, + ) + + warn_sql_auto_mode(resolution, resolved_mode="sql", console=mock_console) + + assert not mock_console.print.called + + def test_quiet_suppresses_console_output(self, mock_console): + """quiet=True must not print, even on a SQL platform.""" + warn_sql_auto_mode(self._auto_resolution(), resolved_mode="sql", console=mock_console, quiet=True) + + assert not mock_console.print.called + + def test_logs_debug_regardless_of_platform(self): + """The generic 'using basic unified config' debug log fires for both modes.""" + logger = MagicMock() + + warn_sql_auto_mode(self._auto_resolution(), resolved_mode="dataframe", console=MagicMock(), logger=logger) + + assert logger.debug.called + def test_display_tuning_list_no_results(self, mock_console): """Test display_tuning_list with no matching templates.""" display_tuning_list(mock_console, platform="nonexistent") From 2bece9caffe6aec171c8ec489c0f170fded17489 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 16 Jul 2026 12:58:26 +0000 Subject: [PATCH 3/3] fix(cli): don't call the SQL --tuning auto fallback config "untuned" warn_sql_auto_mode's warning called the fallback UnifiedTuningConfiguration() "basic (untuned)", but ConstraintConfiguration.enabled defaults to True, so that config has primary/foreign/unique/check constraints all enabled - the same "constraints-only" state the FALLBACK resolution path already labels correctly elsewhere in this module. A user comparing against --tuning notuning (all constraints disabled) could read "untuned" and assume parity with that baseline when it isn't one. Message-only fix; the constraints-enabled behavior itself is unchanged. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01SKw2UhgHuzYPo3y5MfarD6 --- benchbox/cli/tuning_resolver.py | 4 +++- tests/unit/cli/test_tuning_resolution.py | 13 +++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/benchbox/cli/tuning_resolver.py b/benchbox/cli/tuning_resolver.py index c229ab696..e0f7eff65 100644 --- a/benchbox/cli/tuning_resolver.py +++ b/benchbox/cli/tuning_resolver.py @@ -398,7 +398,9 @@ def warn_sql_auto_mode( if not quiet: console.print( "[yellow]Warning: --tuning auto smart defaults are DataFrame-only today; " - "this SQL run proceeds with a basic (untuned) configuration.[/yellow]" + "this SQL run proceeds with a basic constraints-only configuration " + "(primary/foreign/unique/check constraints enabled; no other tunings applied), " + "not an untuned baseline.[/yellow]" ) if logger: logger.debug("Tuning mode auto on a non-DataFrame platform: using basic unified config") diff --git a/tests/unit/cli/test_tuning_resolution.py b/tests/unit/cli/test_tuning_resolution.py index 278728b91..4ef14e5f6 100644 --- a/tests/unit/cli/test_tuning_resolution.py +++ b/tests/unit/cli/test_tuning_resolution.py @@ -459,6 +459,19 @@ def test_warns_on_sql_platform(self, mock_console): printed = " ".join(str(call) for call in mock_console.print.call_args_list) assert "DataFrame-only" in printed + def test_warning_does_not_call_the_config_untuned(self, mock_console): + """The fallback UnifiedTuningConfiguration() this warning describes has + primary/foreign/unique/check constraints enabled by default + (ConstraintConfiguration.enabled defaults to True) -- calling it + "untuned" misleads a user comparing against --tuning notuning, whose + baseline has every constraint disabled. + """ + warn_sql_auto_mode(self._auto_resolution(), resolved_mode="sql", console=mock_console) + + printed = " ".join(str(call) for call in mock_console.print.call_args_list) + assert "basic (untuned) configuration" not in printed + assert "constraints" in printed.lower() + def test_no_warning_on_dataframe_platform(self, mock_console): """DataFrame platforms have real smart defaults elsewhere; no warning needed.""" warn_sql_auto_mode(self._auto_resolution(), resolved_mode="dataframe", console=mock_console)