[DataLoader] Simplify integration tests

[robreeves]

Summary

The data loader integration tests were getting brittle and complicated. They wrote data locally with PyIceberg and then manually manipulated table metadata to add the data and different snapshots. As the use cases grow (branches, ORC support) this is not sustainable.

This PR refactors the data loader integration tests to use the oh-hadoop-spark docker recipe. Then the integration tests can use Spark to do writes and the data loader to do reads. Using oh-hadoop-spark is a more expensive setup so I moved it to its own workflow to only run when data loader changes are made.

Details

Integration tests run inside a Docker container on the same network as the oh-hadoop-spark Docker Compose services. The test container needs Python dependencies with Linux x86_64 native extensions (pyarrow, datafusion, etc.) plus JRE 8 and Hadoop 2.8 client jars for HDFS access via PyArrow's libhdfs JNI bridge.

Cross-platform problem
Dependencies like pyarrow and datafusion include platform-specific compiled extensions (.so files). On CI (Linux x86_64), these match the Docker container's platform natively. On a macOS ARM dev machine, a normal pip install produces macOS ARM binaries that can't run inside the Linux container.

How it's handled
uv pip install --target supports a --python-platform flag that downloads pre-built wheels for a different platform. The Makefile uses this to install all dependencies with Linux x86_64 native extensions regardless of the host OS. The Dockerfile then copies the pre-built site-packages directory into the image and sets PYTHONPATH.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

New file: integrations/python/dataloader/tests/Dockerfile — Test runner container with
Python 3.12, JDK, Hadoop 2.8.5 client (for PyArrow's libhdfs JNI bridge), and uv.

integrations/python/dataloader/tests/integration_tests.py — Removed _LocalCommitCatalog,
_append_data, _copy_metadata_from_container, _get_metadata_path, _create_table,
_delete_table, _cleanup_table and their imports. Added LivySession class that creates a
Livy SQL session and executes Spark SQL statements. All test_* functions and assertions are
unchanged.

integrations/python/dataloader/Makefile — integration-tests target now builds a Docker
image and runs it on the oh-hadoop-spark_default network. Token is passed via OH_TOKEN env var.

.github/workflows/build-run-tests.yml — Removed data loader steps.

integrations/python/dataloader/CLAUDE.md — Updated to reflect oh-hadoop-spark and
containerized test execution.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

All test functions and assertions are unchanged — only the test data setup mechanism changed
(Spark SQL via Livy instead of PyIceberg metadata manipulation). make verify passes (lint,
format, typecheck, unit tests).

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

[cbb330]

This PR switches CI from oh-only to oh-hadoop-spark, which adds HDFS, Spark, and Livy containers + shadowJar to the build. That's a significant increase in startup time and build cost that will slow down every PR, including Java-only changes that don't touch the dataloader at all.

Split the workflow into two independent jobs:

1. build-and-run-tests — stays exactly as it is on main today (gradlew build, oh-only, scripts/python/integration_test.py). No new dependencies.
2. dataloader-tests (new) — runs in parallel, owns the oh-hadoop-spark recipe, shadowJar, Livy wait, make verify, and make integration-tests.

No needs: between them, so Java CI is never blocked on the heavier Python/Spark setup. The existing build-tag-publish.yml gate (needs: build-and-run-tests) stays on the Java job only.

Example workflow sketch

jobs:
  # Unchanged from main — Java build + lightweight API integration test
  build-and-run-tests:
    name: Build and Run Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-java@v5
        with:
          distribution: 'microsoft'
          java-version: '17'
      - uses: gradle/actions/setup-gradle@v5
      - run: ./gradlew clean build

      - run: docker compose -f infra/recipes/docker-compose/oh-only/docker-compose.yml up -d --build
      - run: sleep 30

      - uses: actions/setup-python@v6
        with:
          python-version: '3.12'
      - run: pip install -r scripts/python/requirements.txt
      - run: python scripts/python/integration_test.py ./tables-test-fixtures/tables-test-fixtures-iceberg-1.2/src/main/resources/dummy.token

      - if: always()
        run: docker compose -f infra/recipes/docker-compose/oh-only/docker-compose.yml down

  # New parallel job — dataloader lint + unit + integration
  dataloader-tests:
    name: Dataloader Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-java@v5
        with:
          distribution: 'microsoft'
          java-version: '17'
      - uses: gradle/actions/setup-gradle@v5
      - run: ./gradlew shadowJar

      - run: docker compose -f infra/recipes/docker-compose/oh-hadoop-spark/docker-compose.yml up -d --build
      - name: Wait for services
        run: |
          for i in $(seq 1 30); do
            if curl -sf http://localhost:8000/v1/databases > /dev/null 2>&1 && \
               curl -sf http://localhost:9003/sessions > /dev/null 2>&1; then
              echo "Services ready after $((i * 5))s"
              exit 0
            fi
            sleep 5
          done
          echo "Timed out after 150s"
          exit 1

      - uses: actions/setup-python@v6
        with:
          python-version: '3.12'
      - uses: astral-sh/setup-uv@v7
        with:
          enable-cache: true

      - working-directory: integrations/python/dataloader
        run: make sync verify
      - working-directory: integrations/python/dataloader
        run: make integration-tests TOKEN_FILE=../../../tables-test-fixtures/tables-test-fixtures-iceberg-1.2/src/main/resources/dummy.token

      - if: always()
        run: docker compose -f infra/recipes/docker-compose/oh-hadoop-spark/docker-compose.yml down

[robreeves]

This PR switches CI from oh-only to oh-hadoop-spark, which adds HDFS, Spark, and Livy containers + shadowJar to the build. That's a significant increase in startup time and build cost that will slow down every PR, including Java-only changes that don't touch the dataloader at all.

Split the workflow into two independent jobs:

1. build-and-run-tests — stays exactly as it is on main today (gradlew build, oh-only, scripts/python/integration_test.py). No new dependencies.
2. dataloader-tests (new) — runs in parallel, owns the oh-hadoop-spark recipe, shadowJar, Livy wait, make verify, and make integration-tests.

No needs: between them, so Java CI is never blocked on the heavier Python/Spark setup. The existing build-tag-publish.yml gate (needs: build-and-run-tests) stays on the Java job only.

Example workflow sketch

I like that idea. I feel a lot better about not adding the cost for every PR. I was also not satisfied with the change yet, but wanted to see how it ran in the PR check (still in draft state).

[robreeves]

@cbb330 this is ready for review

[Copilot]

Pull request overview

Refactors the Python DataLoader integration tests to provision Iceberg tables via Spark SQL through Livy (using the oh-hadoop-spark docker recipe), and moves these heavier integration tests into a dedicated GitHub Actions workflow that runs only when the dataloader subtree changes.

Changes:

• Reworked integration_tests.py to use a LivySession helper for Spark SQL table setup/inserts instead of local PyIceberg metadata manipulation.
• Added a dedicated Docker-based integration test runner image + Makefile targets to build platform-correct dependencies and run tests on the Compose network.
• Introduced a new CI workflow for dataloader-only unit + integration tests; removed dataloader steps from the shared build workflow.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
integrations/python/dataloader/tests/integration_tests.py	Uses Livy/Spark SQL to create/populate tables in HDFS and validates DataLoader reads (filters, projections, snapshots).
integrations/python/dataloader/tests/Dockerfile	Adds a minimal test runner image with Java 8 + Hadoop 2.8 client and prebuilt Python deps for HDFS access.
integrations/python/dataloader/Makefile	Builds platform-targeted site-packages for the test image and runs integration tests in Docker on the Compose network.
integrations/python/dataloader/CLAUDE.md	Updates local validation instructions for oh-hadoop-spark and containerized integration tests.
.github/workflows/dataloader-tests.yml	New workflow to run dataloader unit + integration tests when integrations/python/dataloader/** changes.
.github/workflows/build-run-tests.yml	Removes dataloader test steps from the general Gradle build/test workflow.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.