Skip to content

cloud-sorted-ingestion.md

Latest commit

 

History

History
90 lines (64 loc) · 2.57 KB

cloud-sorted-ingestion.md

File metadata and controls

90 lines (64 loc) · 2.57 KB
Raw

Cloud Sorted Ingestion

BenchBox supports an opt-in sorted-ingestion strategy for cloud data warehouse runs so you can compare:

  • Deep sorted ingestion
  • Platform-native tuning (for example clustering or Z-ORDER)
  • Baseline runs with no sorted-ingestion rewrite

Strategy Controls

Use unified tuning platform_optimizations:

platform_optimizations:
  sorted_ingestion_mode: force   # off | auto | force
  sorted_ingestion_method: ctas  # auto | ctas | z_order | liquid_clustering | vacuum_sort

sorted_ingestion_mode:

  • off: disable sorted ingestion
  • auto: enable only when the platform capability matrix supports a method
  • force: require sorted ingestion, fail if unsupported

sorted_ingestion_method:

  • auto: choose platform default
  • ctas: use CTAS rewrite flow where available
  • z_order: Databricks Z-ORDER path
  • liquid_clustering: Databricks liquid clustering path
  • vacuum_sort: Redshift vacuum sort path

CLI Overrides

You can override strategy at runtime:

benchbox run \
  --platform snowflake \
  --benchmark tpch \
  --scale 1 \
  --tuning ./unified_tuning.yaml \
  --sorted-ingestion-mode force \
  --sorted-ingestion-method ctas

Comparison Workflow

  1. Run baseline:
benchbox run --platform snowflake --benchmark tpch --scale 1 --tuning notuning
  1. Run platform-native tuning:
benchbox run --platform snowflake --benchmark tpch --scale 1 --tuning ./snowflake-native.yaml
  1. Run deep-sorted ingestion:
benchbox run --platform snowflake --benchmark tpch --scale 1 --tuning ./snowflake-sorted.yaml
  1. Compare result files:
benchbox compare <baseline.json> <native.json> <sorted.json>

Platform Notes

  • Snowflake: supports ctas sorted-ingestion path when mode is enabled.
  • Databricks: supports ctas, z_order, and liquid_clustering strategy methods. TPC templates now distinguish legacy databricks_z_order from databricks_liquid_auto; do not combine Liquid Clustering with ZORDER or per-table partitioning fields.
  • Redshift: supports ctas and vacuum_sort.
  • Amazon Athena and Azure Synapse Analytics: support ctas method.
  • BigQuery: forcing sorted ingestion currently returns an actionable error for CTAS-hook execution path.
  • ClickHouse Cloud: post-load sorted ingestion is unsupported; use table ORDER BY at create time.

Result Metadata

BenchBox writes sorted-ingestion metadata into execution_metadata.sorted_ingestion:

  • Configured mode/method
  • Resolved mode/method
  • Platform capability details
  • Applied table list/count
  • Total sorted-ingestion apply time