docs(tuning): fix env contract, rewrite stale README, document precedence; honest auto-mode message

CHANGELOG.md

@@ -46,6 +46,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **Removed dead `BENCHBOX_TUNING_ENABLED` env var** - This variable set the
+  `tuning.enabled` config key, which nothing at runtime ever read (only a unit
+  test did); docs incorrectly claimed it "activates tuned runs in CI". Use
+  `--tuning tuned` / `--tuning auto` on `benchbox run` to actually enable
+  tuning; `BENCHBOX_TUNING_CONFIG` still works to point at a default tuning
+  file. If you were setting `BENCHBOX_TUNING_ENABLED`, it had no effect and can
+  simply be removed.
+- **Honest `--tuning auto` messaging on SQL platforms** - Smart defaults for
+  `--tuning auto` are implemented for DataFrame platforms only today; SQL
+  platform runs now get an explicit warning that they're proceeding with a
+  basic (untuned) configuration instead of a message implying real smart
+  defaults were applied.
 - **`-df` platform names select DataFrame mode** - `benchbox run --platform
   datafusion-df` (and `lakesail-df`) now runs the DataFrame adapter without
   needing `--mode dataframe`; alias normalization previously dropped the

_project/DONE/main/active/tuning-docs-env-contract-fixes-20260712.yaml

@@ -0,0 +1,145 @@
+id: tuning-docs-env-contract-fixes-20260712
+title: "Fix broken tuning docs/env contract: dead env var, stale README, undocumented precedence, dishonest auto mode"
+worktree: main
+priority: Medium-High
+status: Completed
+completed_date: "2026-07-16"
+category: Documentation
+
+description: |
+  From the 2026-07-12 tuning review, finding R5 (evidence pinned to commit acfb8992).
+
+  - BENCHBOX_TUNING_ENABLED sets config key tuning.enabled which NOTHING at
+    runtime reads (only reader is a test); yet docs/usage/configuration.md:91-94
+    and docs/reference/cli/configuration.md:93 claim it activates tuned runs in CI.
+  - examples/tunings/README.md is wholesale stale: documents a --tuning-config
+    flag that does not exist, example scripts that do not exist, stale flat
+    filenames, an env-var recipe that cannot work, and an unqualified "2-10x"
+    performance promise.
+  - The actual precedence order (flag default notuning -> keyword/path -> yaml
+    default_config_file [BENCHBOX_TUNING_CONFIG] -> BENCHBOX_TUNING_PATH ->
+    cwd-relative examples/ -> cwd -> fallback) is documented nowhere.
+  - --tuning auto on SQL platforms prints "using smart defaults based on system
+    profile" but delivers a default-constructed config (run.py:1021-1024);
+    real smart defaults exist only for DataFrame mode (tuning_runtime.py:49-92).
+  - docs/reference/cli/tuning.md: wrong --output default, three subcommands
+    (list/show/platforms) undocumented.
+
+work:
+  - id: w1
+    summary: "Decide-in-PR the env var: remove the doc claims AND either delete BENCHBOX_TUNING_ENABLED handling or emit a deprecation warning when set (recommend delete + release note)"
+    status: done
+    needs: []
+    notes: >
+      Re-verified at HEAD (706a64c3): tuning.enabled is set by two mechanisms
+      (module-level _apply_environment_overrides env_mappings dict, and the
+      ConfigManager instance's generic apply_environment_overrides driven by
+      the default config's tuning.environment_overrides dict) but consumed
+      nowhere at runtime except tests/unit/cli/test_config_coverage.py.
+      Deleted both mappings, the docstring bullet, and the dead
+      "enabled": False default key from benchbox/cli/config.py. Fixed doc
+      claims in docs/usage/configuration.md and
+      docs/reference/cli/configuration.md (env-var table, config-section
+      table, and example config block). Added a CHANGELOG.md [Unreleased] ->
+      Fixed entry. BENCHBOX_TUNING_CONFIG (tuning.default_config_file, read by
+      tuning_resolver.py:_resolve_tuned) is untouched and still documented.
+      No test asserted the removed default value, so no test changes were
+      needed; test_config_coverage.py's
+      test_instance_apply_environment_overrides_updates_tuning sets its own
+      synthetic environment_overrides mapping and still passes unchanged.
+  - id: w2
+    summary: "Rewrite examples/tunings/README.md against reality: --tuning flag, real template paths, auto-discovery as the primary UX, drop unfalsifiable performance claims"
+    status: done
+    needs: []
+    notes: >
+      Rewrote examples/tunings/README.md. Verified against HEAD: the real
+      flag is --tuning (no --tuning-config anywhere in the codebase); the
+      named example scripts (examples/duckdb_tpch.py etc.) do not exist, but
+      examples/unified_runner.py and examples/features/tuning_comparison.py
+      are real and do use --tuning correctly (noted unified_runner.py
+      defaults to "tuned", unlike the main CLI's "notuning"); file layout
+      confirmed via `find examples/tunings` (duckdb/, databricks/, and a flat
+      dataframe/<platform>_<profile>.yaml layout that is NOT auto-discovered);
+      dropped the unqualified "2-10x faster" claim in favor of "run both and
+      compare"; added BENCHBOX_TUNING_PATH and `benchbox tuning list/show`
+      usage.
+  - id: w3
+    summary: "Write the precedence table into docs/reference/cli/tuning.md; document tuning list/show/platforms; fix --output default"
+    notes: >
+      Re-checked the review's verified rules against HEAD before writing, per
+      the note. Confirmed in benchbox/cli/tuning_resolver.py:resolve_tuning
+      that keyword checks (notuning/auto/tuned) run before the path-existence
+      check (post-#1172), that --tuning defaults to notuning
+      (cli/commands/run.py option default), that _resolve_tuned checks
+      tuning.default_config_file (BENCHBOX_TUNING_CONFIG override) before
+      template discovery, and that get_tuning_template_paths searches
+      BENCHBOX_TUNING_PATH -> cwd-relative examples/tunings/... -> cwd
+      <platform>/... in that order, falling back to a basic config with a
+      warning (+ wizard offer when interactive, per
+      run.py:_load_unified_tuning_config). Documented all of this plus the
+      cwd-relative caveat (review finding N8). Fixed the --output default
+      claim: SQL mode is `<platform>_tuning.yaml`, DataFrame mode is
+      `<platform>_<profile>_tuning.yaml` (cli/tuning_group.py:78-82,
+      109-115,137,169-171 - not tuning_config.yaml as previously documented).
+      Documented the list/show/platforms subcommands, including that `tuning
+      list` (unlike `--tuning tuned`) does NOT honor BENCHBOX_TUNING_PATH.
+    status: done
+    needs: []
+  - id: w4
+    summary: "Make --tuning auto honest on SQL platforms: warn that smart defaults are DataFrame-only and the run proceeds with basic config (message change only; behavior change gated on ADR-2)"
+    status: done
+    needs: []
+    notes: >
+      Re-verified: resolve_dataframe_tuning_config (tuning_runtime.py:49)
+      returns real smart defaults only when mode_name == "dataframe"; SQL
+      platforms fall through to run.py:_load_unified_tuning_config's final
+      else-branch (still at lines ~1021-1024 at HEAD), which was previously
+      silently building a bare UnifiedTuningConfiguration() while the info
+      message claimed "using smart defaults based on system profile". Message
+      change only, no behavior change: reworded _resolve_auto's info message
+      in tuning_resolver.py to say "DataFrame platforms only today", and added
+      an explicit yellow console warning in run.py's else-branch, gated on
+      `tuning_resolution.mode == TuningMode.AUTO and s.resolved_mode !=
+      "dataframe"` (s.resolved_mode is set earlier by _resolve_platform_mode,
+      confirmed via call order in the run command).
+
+deferred:
+  - summary: "Ship discoverable templates with the installed package (package resources) so --tuning tuned works outside a repo checkout"
+    reason: "Behavior/packaging change beyond a docs fix; needs its own sizing after the precedence doc makes the limitation visible (review finding N8)"
+
+deps:
+  needs: []
+
+must_preserve:
+  - "BENCHBOX_TUNING_CONFIG and BENCHBOX_TUNING_PATH behavior exactly as implemented"
+
+anti_patterns:
+  - "Do not document aspirational behavior; every documented rule must have a file:line anchor at time of writing"
+
+scope_limit:
+  only_modify:
+    - "docs/**"
+    - "examples/tunings/README.md"
+    - "benchbox/cli/config.py"
+    - "benchbox/cli/commands/run.py"
+    - "benchbox/cli/tuning_resolver.py"
+    - "tests/unit/**"
+
+verification:
+  - description: "No doc references to the dead env var remain (or the var actually works)"
+    command: "grep -rn 'BENCHBOX_TUNING_ENABLED' docs/ benchbox/ | grep -v test"
+    expected_output: "Either zero doc hits + removed handling, or a functioning documented code path"
+  - description: "Docs build clean"
+    command: "make lint"
+    expected_output: "Passing"
+
+metadata:
+  owners:
+    - "benchbox-team"
+  tags:
+    - tuning
+    - documentation
+    - cli
+  estimated_effort: Medium
+  created_date: "2026-07-12"
+  source_review: "tuning-system deep review 2026-07-12, finding R5"

benchbox/cli/commands/run.py

@@ -55,6 +55,7 @@
     TuningSource,
     display_tuning_resolution,
     resolve_tuning,
+    warn_sql_auto_mode,
 )
 from benchbox.cli.tuning_runtime import (
     build_baseline_unified_config,
@@ -1020,8 +1021,7 @@ def _load_unified_tuning_config(s: types.SimpleNamespace) -> None:
             s.logger.debug("Using basic constraints-only configuration (fallback)")
     else:
         s.loaded_unified_config = UnifiedTuningConfiguration()
-        if s.logger:
-            s.logger.debug(f"Using basic unified config for mode: {tuning_resolution.mode.value}")
+        warn_sql_auto_mode(tuning_resolution, s.resolved_mode, console, s.logger, quiet=bool(s.quiet))
 
 
 def _resolve_data_organization(s: types.SimpleNamespace) -> None:

benchbox/cli/config.py

@@ -128,7 +128,6 @@ def _apply_environment_overrides(config_dict: dict[str, Any]) -> dict[str, Any]:
     - BENCHBOX_SCALE_FACTOR: Override benchmarks.default_scale
     - BENCHBOX_VERBOSE: Override execution.verbose
     - BENCHBOX_MAX_WORKERS: Override execution.max_workers
-    - BENCHBOX_TUNING_ENABLED: Override tuning.enabled
     - BENCHBOX_TUNING_CONFIG: Override tuning.default_config_file
 
     Args:
@@ -143,7 +142,6 @@ def _apply_environment_overrides(config_dict: dict[str, Any]) -> dict[str, Any]:
         "BENCHBOX_SCALE_FACTOR": ("benchmarks", "default_scale", float),
         "BENCHBOX_VERBOSE": ("execution", "verbose", lambda v: v.lower() in ["true", "1", "yes", "on"]),
         "BENCHBOX_MAX_WORKERS": ("execution", "max_workers", int),
-        "BENCHBOX_TUNING_ENABLED": ("tuning", "enabled", lambda v: v.lower() in ["true", "1", "yes", "on"]),
         "BENCHBOX_TUNING_CONFIG": ("tuning", "default_config_file", str),
         "BENCHBOX_OUTPUT_DIR": ("output", "directory", str),
         "BENCHBOX_MEMORY_LIMIT_GB": ("execution", "memory_limit_gb", int),
@@ -285,12 +283,10 @@ def _get_default_config(self) -> BenchBoxConfig:
                 },
             },
             tuning={
-                "enabled": False,
                 "default_config_file": None,
                 "validate_on_load": True,
                 "allow_platform_incompatible": False,
                 "environment_overrides": {
-                    "BENCHBOX_TUNING_ENABLED": "enabled",
                     "BENCHBOX_TUNING_CONFIG": "default_config_file",
                 },
             },

benchbox/cli/tuning_resolver.py

@@ -228,15 +228,25 @@ def _resolve_notuning(logger: Logger | None) -> TuningResolution:
 
 
 def _resolve_auto(logger: Logger | None) -> TuningResolution:
-    """Resolve auto (smart defaults) mode."""
+    """Resolve auto (smart defaults) mode.
+
+    Real system-profile smart defaults are only implemented for DataFrame
+    platforms today (see ``resolve_dataframe_tuning_config`` in
+    ``tuning_runtime.py``); SQL platforms fall back to a basic, untuned
+    configuration. This function does not know the resolved platform/mode, so
+    the concrete "SQL gets a basic config" warning is emitted downstream once
+    that is known (see ``run.py::_load_unified_tuning_config``).
+    """
     resolution = TuningResolution(
         mode=TuningMode.AUTO,
         source=TuningSource.SMART_DEFAULTS,
         enabled=True,
     )
-    resolution.info_messages.append("Tuning mode: auto (using smart defaults based on system profile)")
+    resolution.info_messages.append(
+        "Tuning mode: auto (smart defaults based on system profile - DataFrame platforms only today)"
+    )
     if logger:
-        logger.debug("Tuning mode: auto (smart defaults)")
+        logger.debug("Tuning mode: auto (smart defaults; DataFrame platforms only)")
     return resolution
 
 
@@ -364,6 +374,38 @@ def display_tuning_resolution(
             console.print(f"  [dim]{path}[/dim] ({exists_marker})")
 
 
+def warn_sql_auto_mode(
+    resolution: TuningResolution,
+    resolved_mode: str | None,
+    console: Console,
+    logger: Logger | None = None,
+    quiet: bool = False,
+) -> None:
+    """Log and, on SQL platforms, warn about a basic-config `--tuning auto` resolution.
+
+    Callers reach here whenever ``_load_unified_tuning_config`` falls through
+    to a basic, freshly-constructed ``UnifiedTuningConfiguration`` (the only
+    resolution source that does so is ``TuningMode.AUTO``). Real
+    system-profile smart defaults (``resolve_dataframe_tuning_config`` in
+    ``tuning_runtime.py``) only exist for DataFrame platforms today, so SQL
+    platforms get an explicit warning instead of silently proceeding while the
+    `auto` info message implies real smart defaults were applied.
+    """
+    if logger:
+        logger.debug(f"Using basic unified config for mode: {resolution.mode.value}")
+    if resolution.mode != TuningMode.AUTO or resolved_mode == "dataframe":
+        return
+    if not quiet:
+        console.print(
+            "[yellow]Warning: --tuning auto smart defaults are DataFrame-only today; "
+            "this SQL run proceeds with a basic constraints-only configuration "
+            "(primary/foreign/unique/check constraints enabled; no other tunings applied), "
+            "not an untuned baseline.[/yellow]"
+        )
+    if logger:
+        logger.debug("Tuning mode auto on a non-DataFrame platform: using basic unified config")
+
+
 def display_tuning_list(
     console: Console,
     platform: str | None = None,

docs/reference/cli/configuration.md

@@ -90,8 +90,7 @@ These variables override the corresponding settings in the configuration file (e
 | `BENCHBOX_SCALE_FACTOR` | `benchmarks.default_scale` | float | `0.01` | Default scale factor |
 | `BENCHBOX_VERBOSE` | `execution.verbose` | boolean | `true` | Enable verbose output (`true`/`1`/`yes`/`on`) |
 | `BENCHBOX_MAX_WORKERS` | `execution.max_workers` | integer | `4` | Maximum parallel worker threads |
-| `BENCHBOX_TUNING_ENABLED` | `tuning.enabled` | boolean | `false` | Enable table tuning by default |
-| `BENCHBOX_TUNING_CONFIG` | `tuning.default_config_file` | string | - | Path to tuning configuration file |
+| `BENCHBOX_TUNING_CONFIG` | `tuning.default_config_file` | string | - | Path to tuning configuration file. Used when `--tuning tuned` resolves with no explicit file (see [Tuning Commands](tuning.md)) |
 | `BENCHBOX_OUTPUT_DIR` | `output.directory` | string | `./benchmark_runs/results` | Output directory for result files |
 | `BENCHBOX_MEMORY_LIMIT_GB` | `execution.memory_limit_gb` | integer | `0` (auto) | Memory limit in GB; `0` means no limit |
 
@@ -306,8 +305,7 @@ Table tuning and optimization settings.
 
 | Setting | Type | Default | Description |
 |---------|------|---------|-------------|
-| `enabled` | boolean | `false` | Apply tuning configurations by default |
-| `default_config_file` | string | `null` | Path to a default tuning YAML file |
+| `default_config_file` | string | `null` | Default tuning YAML file used when `--tuning tuned` finds no explicit file; see [Tuning Commands](tuning.md) for the full resolution order |
 | `validate_on_load` | boolean | `true` | Validate tuning configurations when loaded |
 | `allow_platform_incompatible` | boolean | `false` | Allow tuning configs that contain platform-incompatible directives |
 
@@ -341,7 +339,6 @@ execution:
     warm_up_iterations: 2
 
 tuning:
-  enabled: true
   default_config_file: ./tuning/my_tuning.yaml
 ```