Skip to content

Repository navigation

Increase resources for ACI containers and PostgreSQL #254

Closed
#339
Closed
Increase resources for ACI containers and PostgreSQL#254
#339
Assignees
jathavaan
Labels
enhancementNew feature or request

Description

@jathavaan
jathavaan
opened on May 15, 2026

Task description

Task description

Align the ACI benchmark container spec with the Databricks worker class and the PostgreSQL Flex Server class so the benchmark comparison reflects engine performance, not host-spec differences. The current ACI spec (3 vCPU, 8 GB) lets DuckDB and Shapefile run at a lower compute class than PostGIS (server at 4 vCPU, 16 GB per the pricing service config) and Sedona workers (4 vCPU, 16 GB on Standard_D4s_v3), which biases CPU-bound spatial predicates (PIP, KNN, intersection-heavy joins) against the ACI-hosted engines.

Target compute class for all single-node hosts: 4 vCPU, 16 GB.

Primary change: bump every benchmark container in benchmarks.yml from cpu: 3, memory_gb: 8 to cpu: 4, memory_gb: 16.

Re-run requirement: all previously collected benchmark results must be discarded and re-run on the new spec. Mixing results across spec classes is worse than picking one and being consistent.

Documentation drift to fix: the README.md PostgreSQL setup section still references Standard_B2ms (2 vCPU, 8 GB), while src/infra/infrastructure/services/azure_pricing_service.py already prices Standard_D4ads_v5 (4 vCPU, 16 GB) as of March 2026. Confirm the actual SKU of doppa-db in the Azure Portal and bring the README into line with reality.

Developer tasks

  • Verify Azure student subscription quota for the new ACI core count in Norway East
  • Update cpu from 3 → 4 and memory_gb from 8 → 16 across every entry in benchmarks.yml
  • Confirm doppa-db Flex Server is actually running at Standard_D4ads_v5; if not, resize it
  • Update README.md PostgreSQL setup section — replace Standard_B2ms and "2 vCores, 8 GiB RAM" references with the actual SKU
  • Re-verify PostgreSQL server parameters (shared_buffers, effective_cache_size, work_mem) match the new memory budget
  • Search the repo for any remaining references to B2ms, 3 vCPU, or 8 GB benchmark-host specs and update them
  • Update the pricing constants in azure_pricing_service.py if the verified SKU differs from Standard_D4ads_v5
  • Discard previously collected benchmark results in the dev-benchmarks and benchmarks blob containers
  • Run a single full benchmark sweep at the new spec to confirm the setup before locking it in
  • Record the spec-class alignment decision in the thesis methodology chapter

Developer tasks

No response

Activity

  1. jathavaan
    self-assigned this
    on May 15, 2026
  2. jathavaan
    added
    enhancementNew feature or request
    on May 15, 2026

  3. jathavaan commented on May 24, 2026

    @jathavaan
    jathavaan
    on May 24, 2026
    CollaboratorAuthor
    Image This is during the most intensive run

  4. jathavaan commented on May 24, 2026

    @jathavaan
    jathavaan
    on May 24, 2026
    CollaboratorAuthor

    Update: OOM on RQ2 DuckDB large + resource alignment

    OOM on national-scale-spatial-join-duckdb-large

    Container terminated with exit code 137 (OOMKilled). Warmup alone took ~62 min, then the timed loop started but the container was killed before completing iteration 1. DuckDB has no internal memory limit set — it consumed all 8 GB and the OS killed it. This confirms the need to bump to 16 GB.

    All RQ2 ACI containers → 4 vCPU / 16 GB

    Every RQ2 container (DuckDB, PostGIS client, and Databricks clients) should run at the same spec: cpu: 4, memory_gb: 16. This matches the Databricks worker class (Standard_D4s_v3: 4 vCPU, 16 GB) and removes container spec as a confounding variable.

    Shared PostGIS server affects both RQ1 and RQ2

    RQ1 and RQ2 use the same doppa-db PostgreSQL Flex Server. Any server resize (SKU, shared_buffers, work_mem) applies to both research questions. The server must be verified/aligned once — not separately per RQ.

    RQ1 ACI containers → also 4 vCPU / 16 GB

    Since the goal is comparability and the PostGIS server is shared, RQ1 ACI containers (DuckDB, PostGIS client, Shapefile) should also be bumped to the same 4/16 spec. This ensures neither RQ has a compute-class asymmetry.

    Spark driver.maxResultSize must increase

    Multiple Sedona large-tier benchmarks failed with:

    Total size of serialized results of 53 tasks (8.2 GiB) is bigger than spark.driver.maxResultSize 8.0 GiB.

    Current config: DATABRICKS_DRIVER_MAX_RESULT_SIZE = "8g" in src/config.py. The broadcast join on the large dataset produces a result set exceeding 8 GiB that needs to be collected on the driver. Bump to 16g (or 0 for unlimited). This is orthogonal to the ACI/server memory alignment — it's a Spark-internal config on the Databricks cluster.

    Additionally, the partitioned-2-nodes-large and default-2/8-nodes-large experiments failed with executor-level OutOfMemoryError: Java heap space and heartbeat timeouts. These are worker OOMs on Standard_D4s_v3 (16 GB per worker) — indicating that 2-node and some 8-node clusters don't have enough aggregate memory for the large-tier spatial join with the default/partitioned strategy. This is an expected finding (the point of the scaling curve) and should be recorded as stop_reason="failed" rather than treated as infrastructure misconfiguration.

    PostgreSQL: keep general-purpose, do not switch to memory-optimized

    PostGIS large ran 2.5h+ without OOM — it's slow but not memory-starved. Switching to memory-optimized (Standard_E4ads_v5: 4 vCPU, 32 GB) would double PostGIS RAM vs other engines and break the uniform compute-class comparison. The slowness is the benchmark result.

    Bump data release version

    Update BENCHMARK_DOPPA_DATA_RELEASE in src/config.py from "2026-05-16.1" to "2026-05-23.1" — previous results were collected on the old 3-CPU/8-GB spec and must be discarded.

  5. jathavaan
    added a commit that references this issue on May 24, 2026

    #254 Increase ACI resources, reduce RQ2 iterations, drop default stra…

    f82e814
  6. jathavaan
    linked a pull request that will close this issue#254 Increase ACI resources, tune RQ2, drop default strategy #339on May 24, 2026
  7. jathavaan
    added 3 commits that reference this issue on May 24, 2026

    #254 Remove default-strategy containers from CI, docker-compose, and …

    4e74184

    #254 Handle Stopped/Terminated ACI states in container polling

    685a5b5

    #254 Remove leftover default-strategy notebook and fix driver memory …

    235f1ac
  8. jathavaan
    closed this as completedin #339on May 24, 2026
  9. jathavaan
    added 2 commits that reference this issue on May 24, 2026

    Merge pull request #339 from kartAI/feature/254-increase-resources-fo…

    301673e

    Merge pull request #340 from kartAI/feature/254-increase-resources-fo…

    5d14fa8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions