Skip to content

Fixed an error after backporting Upstream PR180310 to release/2.12 br…#3376

Open
glen-amd wants to merge 1 commit into
release/2.12from
fix__to__release_2.12__after__backporting__PR180310
Open

Fixed an error after backporting Upstream PR180310 to release/2.12 br…#3376
glen-amd wants to merge 1 commit into
release/2.12from
fix__to__release_2.12__after__backporting__PR180310

Conversation

@glen-amd

Copy link
Copy Markdown

Error after backporting the Upstream PR180310

Traceback (most recent call last):
  File "/workspace/pytorch/test/glen_test.py", line 14, in <module>
    out = F.interpolate(x, scale_factor=2, mode='nearest')
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/functional.py", line 4827, in interpolate
    return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: invalid configuration argument

Summary

  • Root cause: F.interpolate(x, scale_factor=2, mode='nearest') on a default contiguous tensor takes PATH 2 (NCHW) in upsample_nearest2d_out_cuda_template. That path computed: int grid_z = std::min<int>(maxGridSize[2], ceil_div(nc, (int64_t) block_z * 4));.
  • The change in aten/src/ATen/native/cuda/UpSampleNearest2d.cu:351 correctly targets the path used by a normal contiguous F.interpolate(x, scale_factor=2, mode='nearest'): the NCHW 3D-grid forward launch. It avoids the bad std::min<int>(maxGridSize[2], requested_grid_z) behavior on ROCm, computes grid_z in int64_t, casts to unsigned only at the end, and preserves correctness because the kernel already grid-strides over the NC/Z dimension.
  • On ROCm/HIP, hipDeviceProp_t::maxGridSize[2] is reported through the signed int field as -1 (uint32 max wrapped). So std::min<int>(-1, requested) -> -1 -> stored into dim3 as 0xFFFFFFFF. HIP/HSA stores each dimension's global work size (gridDim*blockDim) in a uint32_t AQL field, so grid.z = 4294967295 overflows it -> hipErrorInvalidConfiguration -> the exact CUDA error: invalid configuration argument.
  • This was a latent bug inherited verbatim from upstream PR [ROCm] Workaround for UpSamplingNearest2D Fwd due to HIP UINT32_MAX limit pytorch/pytorch#180310 (its NCHW comment wrongly says "No HIP-specific clamp needed"). The backport surfaced it by removing @skipCUDAIfRocm from test_upsampling_64bit_indexing_channels_last, whose contiguous reference branch hits NCHW with large nc.
  • The fix (aten/src/ATen/native/cuda/UpSampleNearest2d.cu, PATH 2) on ROCm only: stop reading maxGridSize[2]; compute grid_z in int64, clamp to [1, UINT32_MAX/block_z], and build dim3 from unsigned values. Safe because the NCHW kernel already grid-strides over the NC dimension. grid_x/grid_y are left exact (the kernel doesn't stride over X/Y; they're bounded by INT_MAX/block_dim). The CUDA path is unchanged (kept under #else, including its TORCH_CHECK and maxGridSize usage).

Notes

  • The corresponding backward continuous path is still explicitly left as a TODO.

@glen-amd glen-amd requested a review from jeffdaily June 25, 2026 22:36
@rocm-repo-management-api

rocm-repo-management-api Bot commented Jun 25, 2026

Copy link
Copy Markdown

Jenkins build for b8fc04681280710a1737186e87d398d9e5f6b0ad commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant