Ch. 2: Background — Notes

[jathavaan]

Purpose

Working notes for Chapter 2: Background of the master thesis
"Benchmarking Cloud-Native and Traditional Geospatial Technologies."

Sections covered

• 2.1 Cloud-native data architecture (decoupled storage/compute, object storage, distributed vs. non-distributed, cloud cost model)
• 2.2 Geometry and indexing (geometry model/encodings, spatial indexing — R-tree, GiST, SP-GiST, BRIN)
• 2.3 Spatial data formats and storage models (Shapefile, GeoJSON, GML, GeoParquet, MVT, PMTiles, SpatiaLite, GeoPackage, FGDB, PostGIS, DuckDB + spatial)
• 2.4 Distributed spatial processing (Spark lifecycle, spatial partitioning, distributed operations, Apache Sedona, Databricks as runtime)

Key topics to document

• Decoupled storage and compute as architectural foundation for cloud-native formats
• Format comparison table (Table 2.3.3): indexing, access pattern, cloud-native support
• PostGIS vs. DuckDB architectural contrast (server vs. in-process)
• Sedona layered architecture on Spark: SpatialRDD, ST functions, partitioners, per-partition indexes
• Databricks runtime properties: cluster startup, autoscaling, per-second billing, storage separation
• Broadcast vs. partitioned join execution strategies

How to use this thread

Post notes, draft paragraphs, open questions, and revision requests as comments.

[jathavaan]

From PR #328

§2.4.4 Apache Sedona (p31–32)

The section describes Sedona's layered architecture (Figure 2.4.4) with two user-facing APIs (SQL/DataFrame and SRDD) sitting above the Catalyst optimizer with Sedona's extensions registered. Add a note that Databricks Photon replaces the classic Spark execution engine with a C++ vectorized pipeline that bypasses spark.experimental.extraStrategies. When Photon is active, Sedona's JoinQueryDetector strategy — which detects ST_Intersects in join predicates and plans RangeJoin or BroadcastIndexJoin — is silently ignored, and joins fall back to Spark's BroadcastNestedLoopJoin (brute-force cross product). This is relevant because Figure 2.4.4 implies that the Catalyst optimizer layer uniformly mediates both APIs, but on Databricks with Photon this is not the case for physical planning.

§2.4.5 Databricks as a Sedona run-time (p33)

The section describes four platform properties (cluster startup, autoscaling, separation of compute from storage, per-second billing) but does not mention Photon. Add a paragraph noting that Databricks clusters run the Photon engine by default, a proprietary C++ vectorized execution engine that replaces Spark's classic JVM-based execution for supported operations. Photon accelerates common SQL workloads but does not honour spark.experimental.extraStrategies, the extension point through which Sedona registers its spatial join planner (JoinQueryDetector). When Photon is active, spatial joins degrade to BroadcastNestedLoopJoin regardless of Sedona configuration. The benchmark clusters therefore disable Photon (spark.databricks.photon.enabled=false) to ensure Sedona's optimizer extensions take effect.