Ch. 5: System Architecture and Implementation — Notes

[jathavaan]

Purpose

Working notes for Chapter 5: System Architecture and Implementation of the master thesis
"Benchmarking Cloud-Native and Traditional Geospatial Technologies."

Use this thread to draft, discuss, and refine content for the following sections:

Sections covered

• 5.1 Architecture overview
• 5.2 Infrastructure (compute platforms, storage/identity, determinism and reproducibility)
• 5.3 Data layer (GeoParquet on ADLS Gen2, PostGIS on Azure PostgreSQL, Shapefile bundle)
• 5.4 Compute configuration (DuckDB, PostGIS, Apache Sedona on Databricks)
• 5.5 Orchestration and pairing
• 5.6 Measurement instrumentation and cost model (host-level instrumentation, Spark phase metrics, cost model implementation, raw-sample persistence, what is not measured)

Key implementation topics to document

• Clean Architecture layout: domain → application → infrastructure → presentation
• Dependency injection via dependency_injector and container wiring
• ACI-based orchestration (main.py → benchmark_runner.py → entrypoints)
• Batch pairing constraints (same query type, same dataset size, single PostGIS, Databricks vCPU ≤ 80)
• @monitor decorator and the two stopping-rule paths
• Cost model: how Azure pricing feeds into per-query cost estimates
• Reproducibility: .env forwarding, deterministic seeding, benchmarks.yml as experiment manifest

How to use this thread

Post notes, draft paragraphs, open questions, and revision requests as comments.

[jathavaan]

From issue #292 — Sequential stopping rule (Ch.5 sections)

§5.6.1 Host-level instrumentation & §5.6.3 Cost model implementation

• The cost-analytics window (start_time to end_time) spans the full timed window including any soft-ceiling extension past the BenchmarkIteration value. This is wider than the window in which actual timed iterations occurred if the soft ceiling triggered.

[jathavaan]

From issue #312 — Transient failure tolerance (Ch.5 sections)

§5.6.3 Cost model implementation

• Cost analytics still span the full timed window including failed iterations' wall-clock contribution. A failed iteration that ran for 3 minutes before crashing is billed by Azure just like a successful one. The cost model is therefore conservative (overestimates per-successful-iteration cost when failures occur).

[jathavaan]

From issue #309

Section 5.5 — Orchestration and pairing

The default strategy matrix has been pruned from 11 cells to 3 (large tier only: 2 / 8 / 16 nodes). If the thesis describes the full default matrix or counts 37 RQ2 experiments, update to 29 experiments and note that default is retained at large tier only as a within-Sedona baseline.

Section 5.4.3 — Apache Sedona on Databricks

If this section describes the three strategy variants and their node counts, update the default row to reflect large-tier-only coverage at 2 / 8 / 16 nodes.

[jathavaan]

From issue/PR #296

§5.5 — Orchestration and pairing

• The regional Standard_D4s_v3 vCPU quota has been raised from 80 to 200. The batch constraint (d) now reads "Databricks cluster vCPU sum ≤ 200" instead of ≤ 80.
• When a batch member fails at the orchestrator level, a warning is now logged identifying the incomplete batch and the failed member(s), noting that fair-comparison assumptions may not hold.

§5.4.3 — Apache Sedona on Databricks

• Cluster provisioning now raises a typed QuotaExhaustedError (instead of generic RuntimeError) when the Databricks API or cluster state message indicates quota exhaustion. This surfaces in two places: the clusters/create API response and the cluster polling loop (_wait_for_cluster_running). The error is defined in src/domain/exceptions/.

[jathavaan]

From PR #326

§5.4.3 Apache Sedona on Databricks (stub)

When this section is written, it should document the regional vCPU quota constraint. The quota was increased from 80 to 200 vCPUs (PR #326), computed as (workers + 1) × 4 per Sedona member on Standard_D4s_v3. The system now raises a typed QuotaExhaustedError (in src/domain/exceptions/) instead of a generic RuntimeError when Databricks returns QuotaExceeded during cluster creation or polling.

§5.5 Orchestration and pairing (stub)

When this section is written, it should cover batch-failure detection. The orchestrator now:

1. Tracks which experiment IDs failed within each batch.
2. Distinguishes total batch failure (all members crashed — no usable data) from partial failure (some peers survived without matched counterparts — fair-comparison assumptions weakened).
3. Logs a warning identifying the failed member IDs and the failure mode.

[jathavaan]

From PR #328

§5.4.3 Apache Sedona on Databricks (p57, currently empty)

When writing this section, document the following cluster configuration choices and why each is necessary:

1. Photon disabled (spark.databricks.photon.enabled=false). Photon's C++ execution pipeline does not honour spark.experimental.extraStrategies, where SedonaContext.create(spark) registers JoinQueryDetector. Without this, all spatial joins degrade to BroadcastNestedLoopJoin.

2. KryoSerializer (spark.serializer=org.apache.spark.serializer.KryoSerializer). Recommended by Sedona for efficient geometry serialization between executors.

3. Partitioned variant: AQE and auto-broadcast disabled for the timed section only (spark.sql.adaptive.enabled=false, spark.sql.autoBroadcastJoinThreshold=-1). AQE can rewrite Sedona's RangeJoin plan, and auto-broadcast causes Spark to broadcast the 357-row municipalities side, bypassing Sedona's KDB-tree spatial partitioner. Both are restored after the join.

4. Sedona libraries installed via Libraries API (not init scripts). This means spark.sql.extensions cannot be set in spark_conf (the JAR is not on the classpath at cluster startup). Instead, SedonaContext.create(spark) handles Catalyst rule registration at notebook runtime.

5. DataFrame API used for all three notebook variants (broadcast, partitioned, default) instead of sedona.sql(). Structural consistency across variants; both paths go through the same Catalyst optimizer.

Also mention the per-iteration hard timeout (3600 s) added to the benchmark monitor as a safety net against runaway Databricks iterations.

[jathavaan]

From PR #338

Section 5.4.3 — Apache Sedona on Databricks

When this section is written, the Spark cluster configuration table (or equivalent listing) should include spark.driver.maxResultSize = 8g alongside the existing spark.driver.memory = 9g and spark.driver.memoryOverhead = 512m.

Why it matters: the broadcast join strategy wraps the 357-polygon municipalities table in broadcast(), which forces Spark to serialize the intermediate join output (~100M rows before the groupBy aggregation) through the driver. At the large dataset tier (~100M buildings), the serialized task results total ~4.0 GiB, which hits the Spark default cap of 4 GiB and crashes the run. Raising the cap to 8 GiB gives 2× headroom over the observed failure point. The setting is applied to all cluster variants (broadcast, partitioned, default) as a defensive measure; only the broadcast variant hits the limit because the other strategies avoid routing large intermediate results through the driver.

This is also relevant context for explaining why the broadcast strategy's driver collection phase is inherently heavier than the partitioned strategy's — the intermediate result set must pass through the driver before the groupBy reduces it to 357 rows.

[jathavaan]

From issue #335 / PR #336

Section 5.5 — Orchestration and pairing

When this section is written, it should mention that the orchestrator retries transient Azure CLI failures in _run_cmd with up to 3 attempts and linear backoff (10s/20s/30s). Transient errors are detected by matching stderr markers (Connection aborted, ConnectionError, BadStatusLine, status code 0). Non-transient failures still fail immediately.

Why it matters: after long benchmark runs (~3h), the Azure IMDS token endpoint can return malformed responses (HTTP/1.1 000 status code 0), causing az container create to fail with ConnectionError. Without retries, this kills an entire batch even though the failure is transient. This was observed in batch 18/18 of a 3.1h run — both experiments in the batch failed at orchestrator level before containers were created.

Section 5.6.5 — What is not measured

Retry delays (up to 30s per attempt) occur at the orchestration level during container creation, not during benchmark timing. The @monitor timing window opens only after the container is running, so retries do not affect measured elapsed time or cost.

[jathavaan]

From issue #254

Section 5.2.1 — Compute platforms

When writing this section, note that all ACI containers now run at 4 vCPU / 16 GB RAM (previously 3 vCPU / 8 GB). This matches the Databricks worker class (Standard_D4s_v3: 4 vCPU, 16 GB), removing container spec as a confounding variable between single-node and distributed benchmarks.

Section 5.4 — Compute configuration

• 5.4.1 DuckDB / 5.4.2 PostGIS: Client containers provision 4 vCPU and 16 GB. The bump from 8 GB was required after DuckDB large-tier OOM (exit code 137) during national-scale spatial join warmup.
• 5.4.3 Apache Sedona on Databricks: spark.driver.maxResultSize is now 16g (was 8g). The broadcast join on the large dataset produced serialized results exceeding 8 GiB, triggering Spark errors. The 16 GB cap provides ~2x headroom while preserving fail-fast behavior over unlimited (0).

Section 5.6.3 — Cost model implementation

ACI per-second pricing constants remain the same (AZURE_ACI_VCPU_PRICE_PER_SECOND, AZURE_ACI_MEMORY_GB_PRICE_PER_SECOND), but the actual compute cost per container-second increases because each container now allocates more vCPU and memory. Ensure cost figures in this section use the 4/16 spec.

Section 5.5 — Orchestration and pairing

Data release version bumped from 2026-05-16.1 to 2026-05-23.1 — results collected on the old 3/8 spec are discarded.

[jathavaan]

From issue #254 — Hard timeout increased to 450 minutes

§5.6.1 Host-level instrumentation

The stopping-rule hard timeout has been raised from 5400 s (90 min) to 27 000 s (450 min). This value (BENCHMARK_MAX_TIMED_WINDOW_SECONDS in src/config.py) gates two checks: (1) any single iteration exceeding 450 min triggers an immediate TIMEOUT stop, and (2) cumulative timed-iteration wall-clock exceeding 450 min triggers the same.

When documenting the concrete stopping-rule parameters (min iterations, CI target, timed-window floor, hard timeout), use the updated value.

[jathavaan]

From PR #343

Chapter 5 sections are currently stubs. The following notes describe content that should be written based on the cost model corrections in PR #343.

§5.2.1 Compute platforms

Mention that Databricks is deployed in Sweden Central because Norway East does not offer Databricks workspaces. This cross-region topology is the reason Databricks blob reads incur egress charges that other configurations avoid.

§5.4.3 Apache Sedona on Databricks

Driver memory configuration: spark.driver.memory is set to 9 GB (with 1 GB overhead). The Standard_D4s_v3 node type caps custom driver memory at ~10,069 MB. An earlier configuration of 14 GB was rejected by the Databricks API. spark.driver.maxResultSize is set to 4 GB.

§5.6.3 Cost model implementation

When writing this section, describe the following implementation details that realize Equation 4.1:

1. Driver node inclusion: Databricks cost uses num_workers + 1 nodes (driver is the same Standard_D4s_v3 VM type). Both DBU and VM costs are multiplied by the total node count, not just the worker count.
2. Cross-region blob egress: When is_cross_region_blob=True (Databricks only), blob storage egress uses the intra-Europe cross-region rate ($0.02/GB) instead of the intra-region rate ($0.00). The rate is sourced from AzurePricingService.
3. Dataset-size-aware blob counting: The blob operation count (used for c_ops) is derived from the actual dataset tier (small/medium/large) rather than always counting blobs in the small-tier path. The dataset size is resolved from the DI container at cost-analytics time.
4. Blob storage cost for Databricks: Sedona executors read the same GeoParquet files from ADLS Gen2 as DuckDB. The Databricks CostConfiguration now includes include_blob_storage=True.

§5.6.5 What is not measured

Document the following exclusions:

• Blob writes for monitoring data (parquet samples, cost analytics) are omitted for all configurations — shared overhead that cancels in pairwise comparisons.
• ACI cost represents idle-wait time for PostGIS/Databricks containers vs. actual query compute for DuckDB/Shapefile.
• Cluster provisioning/teardown time is excluded from the Databricks cost window (only the timed-iteration window is priced).
• Blob operation count is approximate: one read operation per blob is assumed, but DuckDB may skip files via bbox pushdown or issue multiple range requests per file.

[jathavaan]

From issue #341

Section 5.4.3 — Apache Sedona on Databricks

• If the section enumerates cluster sizes (worker counts), remove the 12-node configuration. Valid levels are now {2, 4, 8, 16}.

Section 5.5 — Orchestration and pairing

• The large-tier Sedona batch structure has changed. Previously:

  • broadcast-4-large ↔ broadcast-12-large
  • partitioned-4-large ↔ partitioned-12-large
  • broadcast-2-large ↔ broadcast-16-large
  • partitioned-2-large ↔ partitioned-16-large

  Now (12-node removed, 4-node folded into the 2+16 batch):

  • broadcast-{2, 4, 16}-large (3-way batch, vCPU sum = 100)
  • partitioned-{2, 4, 16}-large (3-way batch, vCPU sum = 100)

  Update any batch listing or pairing table accordingly.

Section 5.6 — Measurement instrumentation and cost model

• A new cumulative wall-clock ceiling has been added for fixed-iteration (non-sequential-stopping) benchmarks: BENCHMARK_MAX_FIXED_WINDOW_SECONDS = 4500 (75 minutes). This caps worst-case cost on the Databricks national-scale-spatial-join branch, which previously had no wall-clock ceiling when use_sequential_stopping=False. The per-iteration timeout (BENCHMARK_MAX_ITERATION_SECONDS = 4500) remains unchanged and fires independently.

[jathavaan]

From PR #347

Section 5.4.3 — Apache Sedona on Databricks

When writing this section, document the two Spark configuration overrides that prevent Sedona's Catalyst strategies (JoinQueryDetector) from being bypassed:

1. Photon disabled (spark.databricks.photon.enabled = false): Photon bypasses Sedona's custom Catalyst extraStrategies, causing spatial joins to fall back to Spark's native BroadcastNestedLoopJoin instead of Sedona's BroadcastIndexJoin. Already set at cluster creation in databricks_service.py.

2. AQE disabled (spark.sql.adaptive.enabled = false): Adaptive Query Execution can rewrite Sedona's physical plan after the Catalyst optimizer has selected BroadcastIndexJoin, substituting Spark's native BroadcastNestedLoopJoin. Both the broadcast and partitioned notebooks now disable AQE before the timed join section and restore the original value afterward. PR #346 Disable AQE in broadcast notebook #347 added this guard to the broadcast notebook (the partitioned notebook already had it).

These are independent failure modes — either one alone causes the same fallback to a brute-force cross product — so both must be disabled simultaneously.

Section 5.2.3 — Determinism and reproducibility

The AQE disable is also relevant here: without it, the physical join strategy selected by Spark is non-deterministic across runs (AQE decisions depend on runtime statistics), which would make the benchmark measure different algorithms on different iterations. Pinning AQE off ensures every iteration executes the same physical plan.

[jathavaan]

From issue #348

§5.5 — Orchestration and pairing (p. 59)

The orchestrator now supports per-experiment replication limits. Each experiment in benchmarks.yml declares a runs field (e.g., runs: 30 for RQ1, runs: 3 for RQ2). During the benchmark loop, main.py checks whether benchmark_run > experiment["runs"] before launching a batch. When exceeded, the experiment and all its related_script_ids are marked as completed and skipped. This enables differentiated replication without changing the execution order or pairing protocol.

If §5.5 describes the orchestration loop, it should note that the loop runs BENCHMARK_RUNS (30) times total, but individual experiments may exit the loop earlier based on their runs field. The runs field defaults to BENCHMARK_RUNS if omitted, preserving backward compatibility.

[jathavaan]

From issue #357

When writing §5.4.3 (Apache Sedona on Databricks) and §5.5 (Orchestration and pairing), note that the worker-count levels are now 2, 4, 8, 12, 16 (five levels, not four). The 12-node cluster configuration and its batch pairing constraints will need to be documented alongside the existing sizes.

The batch constraint on Databricks vCPU sum ≤ 200 may affect how 12-node experiments are paired — each 12-node cluster uses (12 + 1) × 4 = 52 vCPUs.