[XGBoost] Gamma testing by Jyothirmaikottu · Pull Request #6039 · aws/deep-learning-containers

Jyothirmaikottu · 2026-05-05T21:02:34Z

Purpose

Test Plan

Test Result

Toggle if you are merging into master Branch

By default, docker image builds and tests are disabled. Two ways to run builds and tests:

Using dlc_developer_config.toml
Using this PR description (currently only supported for PyTorch, TensorFlow, vllm, and base images)

How to use the helper utility for updating dlc_developer_config.toml

Assuming your remote is called origin (you can find out more with git remote -v)...

Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp origin

Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -t sanity_tests -cp origin

Restore TOML file when ready to merge

python src/prepare_dlc_dev_environment.py -rcp origin

NOTE: If you are creating a PR for a new framework version, please ensure success of the local, standard, rc, and efa sagemaker tests by updating the dlc_developer_config.toml file:

sagemaker_remote_tests = true
sagemaker_efa_tests = true
sagemaker_rc_tests = true
sagemaker_local_tests = true

How to use PR description

Use the code block below to uncomment commands and run the PR CodeBuild jobs. There are two commands available:

# /buildspec <buildspec_path>
- e.g.: # /buildspec pytorch/training/buildspec.yml
- If this line is commented out, dlc_developer_config.toml will be used.
# /tests <test_list>
- e.g.: # /tests sanity security ec2
- If this line is commented out, it will run the default set of tests (same as the defaults in dlc_developer_config.toml): sanity, security, ec2, ecs, eks, sagemaker, sagemaker-local.

# /buildspec <buildspec_path>
# /tests <test_list>

Toggle if you are merging into main Branch

PR Checklist

[] I ran pre-commit run --all-files locally before creating this PR. (Read DEVELOPMENT.md for details).

Jyothirmaikottu · 2026-05-08T19:04:05Z


    def test_gpu_single_instance(self, image_uri, role):
-        hp = {**BASE_HP, "tree_method": "gpu_hist"}
+        hp = {**BASE_HP, "tree_method": "hist"}


XGBoost 3.2.0 removed gpu_hist as a tree method entirely. The tree method and device selection are now decoupled. hist is the algorithm, device picks where it runs.

Jyothirmaikottu · 2026-05-11T22:01:44Z

 def gpu_trained_model(image_uri, role):
    """Train a GPU model once for GPU e2e tests."""
-    hp = {**E2E_HP, "tree_method": "gpu_hist"}
+    hp = {**E2E_HP, "tree_method": "hist"}


XGBoost 3.2.0 removed gpu_hist as a tree method entirely. The tree method and device selection are now decoupled. hist is the algorithm, device picks where it runs.

Same Flask conflict fix as PR workflow — sagemaker-containers pins flask==1.1.1 but we need Flask==3.1.3.

The _wait_healthy() method only caught ConnectionError, so a ReadTimeout on the first /ping poll escaped the retry loop and failed the test immediately instead of retrying for 120s.

XGBoost 3.2.0 removed the 'gpu_hist' tree method. GPU training now uses 'hist' with 'device': 'cuda'. Valid tree methods are: {'approx', 'auto', 'exact', 'hist'}.

sagemaker_containers runs 'pip install .' without --no-build-isolation, so pip tries to fetch setuptools from PyPI which fails under network isolation. This is a container-level issue, not a test bug.

nvidia/cuda:12.9.1-base only includes driver stubs. XGBoost GPU needs libcudart.so from the runtime image.

Bumps cache-bust to pick up sagemaker-xgboost-container fix that adds 'device' to the algorithm_toolkit hyperparameter whitelist. Without this, GPU training jobs fail with 'Extraneous hyperparameter found: device'.

- Pipe mode intentionally unsupported (MLIO removed, SageMaker deprecated it) - Sparse protobuf fails with scipy 1.15 vstack on zero-feature records

…distributed - Remove 'device': 'cuda' from all e2e tests — algorithm mode rejects unknown HPs; container auto-detects GPU via SM_NUM_GPUS - Mark pipe mode tests as xfail (MLIO removed, pipe mode unsupported) - Mark container distributed tests as xfail (Rabit protocol changed) - Remove csv-pipe from benchmark parametrize - Fix generate_models workflow to use xgboost==3.2.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ge changes)

- Remove 'device: cuda' from all algorithm-mode GPU e2e tests (container rejects it as extraneous HP; GPU auto-detected via SM_NUM_GPUS) - Remove csv-pipe from benchmark parametrize (pipe mode removed) - Dockerfile: use prebuilt wheel from CI artifact instead of cloning repo every build. Fallback to clone from XGBOOST_CONTAINER_BRANCH for local builds. - PR/release workflows: add build-wheel job that clones the container repo, builds the wheel, and passes it to Docker build via GitHub Actions artifacts. - Add XGBOOST_CONTAINER_BRANCH env for branch testing.

…, CVE-2026-34282)

…n AL2023 repo yet)

… than MLIO C++)

aws-deep-learning-containers-ci Bot added the authorized label May 5, 2026

Jyothirmaikottu force-pushed the xgboost-promote-preprod-to-gamma branch 3 times, most recently from 9085206 to 151218a Compare May 8, 2026 17:24

Jyothirmaikottu commented May 8, 2026

View reviewed changes

Jyothirmaikottu force-pushed the xgboost-promote-preprod-to-gamma branch 2 times, most recently from c5ebbe1 to 848ed49 Compare May 11, 2026 20:42

Jyothirmaikottu commented May 11, 2026

View reviewed changes

Jyothirmaikottu and others added 22 commits May 11, 2026 22:02

[XGBoost] Gamma testing

a9c623d

fix: use --no-deps in release workflow unit test Dockerfile

adf4e82

Same Flask conflict fix as PR workflow — sagemaker-containers pins flask==1.1.1 but we need Flask==3.1.3.

fix: catch ReadTimeout in health check retry loop

0a718c2

The _wait_healthy() method only caught ConnectionError, so a ReadTimeout on the first /ping poll escaped the retry loop and failed the test immediately instead of retrying for 120s.

fix: replace gpu_hist with hist+device=cuda for XGBoost 3.2.0

2c18373

XGBoost 3.2.0 removed the 'gpu_hist' tree method. GPU training now uses 'hist' with 'device': 'cuda'. Valid tree methods are: {'approx', 'auto', 'exact', 'hist'}.

test: xfail network isolation script mode test

0b15c34

sagemaker_containers runs 'pip install .' without --no-build-isolation, so pip tries to fetch setuptools from PyPI which fails under network isolation. This is a container-level issue, not a test bug.

style: fix pre-commit formatting (ruff)

c8ee9cc

fix: use cuda runtime image for GPU support

9533f6f

nvidia/cuda:12.9.1-base only includes driver stubs. XGBoost GPU needs libcudart.so from the runtime image.

fix: register 'device' hyperparameter for XGBoost 3.2.0 GPU support

735b18d

Bumps cache-bust to pick up sagemaker-xgboost-container fix that adds 'device' to the algorithm_toolkit hyperparameter whitelist. Without this, GPU training jobs fail with 'Extraneous hyperparameter found: device'.

test: xfail pipe mode and sparse protobuf tests

85f0944

- Pipe mode intentionally unsupported (MLIO removed, SageMaker deprecated it) - Sparse protobuf fails with scipy 1.15 vstack on zero-feature records

test: xfail GPU endpoint deploy test (MMS startup timeout on g4dn)

a43807d

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

revert: restore Dockerfile to main (remove cache-bust and runtime ima…

f88c705

…ge changes)

fix: clone sagemaker-xgboost-container from master (branch merged)

73fce75

ci: retrigger PR workflow after container fix merge

a56f6b3

ci: bump cache-bust to rebuild with dmlc_timeout fix

f74de30

fix: pin java-11-amazon-corretto-headless to 11.0.31+11 (CVE-2026-22016…

478b02c

…, CVE-2026-34282)

fix: revert java corretto pin (dnf update pulls latest automatically)

05db5cb

fix: allowlist CVE-2026-22016, CVE-2026-34282 (corretto 11.0.31 not i…

6b91740

…n AL2023 repo yet)

test: branch testing with fix-dask-gpu-complete

fba9951

fix: bump urllib3 to 2.7.0 (GHSA-qccp-gfcp-xxvc)

d93cac7

chore: switch XGBOOST_CONTAINER_BRANCH back to master

84c7172

fix: simplify Dockerfile wheel-builder to just use prebuilt wheel

6b601f3

Jyothirmaikottu force-pushed the xgboost-promote-preprod-to-gamma branch from bcdc8f5 to 6b601f3 Compare May 11, 2026 22:02

Jyothirmaikottu and others added 4 commits May 11, 2026 22:03

chore: restore agent-fix.py from main

029d9d5

fix: increase benchmark timeout to 2400s (pure Python RecordIO slower…

e634763

… than MLIO C++)

fix: increase multi-softmax-15class benchmark timeout to 2700s

4bf78f8

make gamma release true

243f2a5

Jyothirmaikottu temporarily deployed to gamma May 12, 2026 19:20 — with GitHub Actions Inactive

Jyothirmaikottu temporarily deployed to gamma May 12, 2026 19:23 — with GitHub Actions Inactive

Jyothirmaikottu deployed to gamma May 12, 2026 19:24 — with GitHub Actions Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[XGBoost] Gamma testing#6039

[XGBoost] Gamma testing#6039
Jyothirmaikottu wants to merge 27 commits into
mainfrom
xgboost-promote-preprod-to-gamma

Jyothirmaikottu commented May 5, 2026

Uh oh!

Jyothirmaikottu May 8, 2026

Uh oh!

Jyothirmaikottu May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Jyothirmaikottu commented May 5, 2026

Purpose

Test Plan

Test Result

PR Checklist

Uh oh!

Jyothirmaikottu May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Jyothirmaikottu May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant