[cuda] enforce max_depth on CUDA tree learner

[maxwbuckley]

Summary

Two related bugs caused the CUDA tree learner to silently ignore the max_depth parameter:

1. CUDABestSplitFinder::FindBestSplitsForLeaf had no max_depth check. CPU's SerialTreeLearner::BeforeFindBestSplit (src/treelearner/serial_tree_learner.cpp:347-355) invalidates a leaf's gain when its depth has reached config_->max_depth, but the CUDA path never did the equivalent.

2. CUDATree::Split / SplitCategorical updated the GPU-side cuda_leaf_depth_ (via the split kernel) but never updated the host-side leaf_depth_ vector. So tree->leaf_depth(idx) always returned 0 on CUDA. Without fixing (2), even adding the check at (1) would have done nothing.

Symptom

With max_depth=2, varying num_leaves:

num_leaves	CPU (correct)	CUDA before fix
4	depth=2, leaves=4	depth=2, leaves=4
7	depth=2, leaves=4	depth=3, leaves=7
15	depth=2, leaves=4	depth=5, leaves=15
31	depth=2, leaves=4	depth=7, leaves=31

After the fix, CUDA caps at the requested depth (2) for every num_leaves.

Why this is a real bug, not FP-precision drift

This is a missing feature on CUDA, not the kind of "numerical precision differences caused by different degrees of parallelism" that issue lightgbm-org#6055 discusses. CUDA is silently ignoring a documented regularization parameter — any user who set max_depth on CUDA was getting models 2-7x deeper than requested, with substantially weaker regularization.

Fix

• src/io/cuda/cuda_tree.cpp — mirror the host-side leaf_depth_ update in CUDATree::Split and CUDATree::SplitCategorical (matching CPU Tree::Split in include/LightGBM/tree.h:578-580).
• src/treelearner/cuda/cuda_best_split_finder.{hpp,cpp} — plumb two new flags smaller_leaf_below_max_depth and larger_leaf_below_max_depth into FindBestSplitsForLeaf and AND them into the is_*_leaf_valid checks.
• src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp — compute the flags at the call sites: config_->max_depth <= 0 || tree->leaf_depth(idx) < config_->max_depth.

Test plan

• Verified locally on RTX 5090 / CUDA 13.2: CUDA now caps trees at the requested max_depth for num_leaves from 4 to 31.
• CPU/CUDA parity sweep — reg_max_depth case (which used max_depth=3 with num_leaves=7) now matches CPU at FP epsilon, down from max|Δ|=0.25 raw_score.
• Existing tests on CUDA (not yet run by author).

🤖 Generated with Claude Code

[BelixRogner]

Thanks Max, and Claude Code. The symptom table (max_depth=2 producing depth-7 trees) is the kind of regression that should never have shipped — good catch. Fix is well-scoped: mirror CPU's Tree::Split for host-side leaf_depth_, plumb two flags, AND them into the existing validity check. Eight parametrized depth-cap tests plus an end-to-end raw-score equality check is solid.

Two small things before merge:

1. Env var convention — same as on PRs [cuda] fix unweighted percentile formula (L1 & quantile leaf renewal) #6, [cuda] fix illegal-memory-access crash in weighted L1 / quantile training #8, [cuda] fix unweighted percentile formula in PercentileGlobalKernel (init scores) #9, [cuda] make BitonicArgSort_1024 / _2048 stable on tied values #10: align LIGHTGBM_TEST_CUDA to the existing getenv("TASK", "") != "cuda" pattern.
2. Existing CUDA test suite — your test plan checkbox for "Existing tests on CUDA" is unchecked. Since max_depth enforcement touches every FindBestSplitsForLeaf call, can you run the suite under TASK=cuda and confirm green before merge? Environmental skips like graphviz / the device_type=cpu-baked-in test you flagged on PR [cuda] honor min_data_per_group in categorical split kernels #4 are fine to enumerate and ignore.

[maxwbuckley]

Both addressed.

1. Env var aligned to TASK=cuda — pushed as eaa55986.

2. Full Python test suite on CUDA — built the PR branch on RTX 5090 + CUDA 13.2 (sm_120) and ran:

TASK=cuda pytest tests/python_package_test/ --ignore=tests/python_package_test/test_dask.py
5 failed, 1111 passed, 21 skipped, 8 xfailed, 4 xpassed in 66.25s

The 5 failures are all environmental, none caused by this PR — same set as on #4:

• test_engine.py::test_all_expected_params_are_written_out_to_model_text — asserts the trained model text contains [device_type: cuda] when TASK=cuda, but the test trains with no device_type param and a CUDA build still defaults to device_type=cpu. Test body is byte-identical to upstream master, so this fails the same way against origin/master — unrelated to this PR.
• test_plotting.py × 4 — graphviz.backend.execute.ExecutableNotFound: failed to execute PosixPath('dot'). The dot binary isn't installed on this machine; apt install graphviz away.

The new max_depth tests all pass:

tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[1-2] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[1-7] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[2-4] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[2-7] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[2-31] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[3-7] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[3-31] PASSED
tests/python_package_test/test_dual.py::test_cuda_respects_max_depth[5-31] PASSED
tests/python_package_test/test_dual.py::test_cuda_max_depth_matches_cpu_predictions PASSED
9 passed, 1 skipped in 0.55s

[BelixRogner]

Quick nudge — the chronic Python - latest versions failure that was hitting every open PR was fixed in #15 (merged earlier today) and the propagated update-branch on this PR confirms CI is otherwise green. Only thing keeping this one from merging now is the ruff/cpplint issue on test_dual.py (and on cuda_rank_objective.cpp for #14):

• PR [cuda] enforce max_depth on CUDA tree learner #7: PT006 — parametrize("max_depth,num_leaves", …) wants a tuple.
• PR [cuda] fix illegal-memory-access crash in weighted L1 / quantile training #8: I001 — imports unsorted.
• PR [cuda] make BitonicArgSort_1024 / _2048 stable on tied values #10: I001 + ruff format reformat.

Single pre-commit run --all-files && git commit --amend --no-edit && git push --force-with-lease should clear all three (and #14 if you do those together).

Let me know when pushed and I'll merge.