Offer precompiled binaries for newer versions of NVSHMEM

[byhemechi]

Current builds target cuda 13.0 libraries which are not included in more "bleeding-edge" distros

[polvalente]

As a quick local fix you can change these to whatever versions you have:

xla/lib/xla.ex

Lines 345 to 349 in 90e1eed

	~s/--repo_env=HERMETIC_CUDA_VERSION="13.0.0"/,
	~s/--repo_env=HERMETIC_CUDNN_VERSION="9.12.0"/,
	~s/--repo_env=HERMETIC_NVSHMEM_VERSION="3.3.20"/,
	~s/--repo_env=HERMETIC_NCCL_VERSION="2.27.7"/,
	~s/--repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES="sm_75,sm_80,sm_90,sm_100,compute_120"/

However, what's the error you get when trying to run on cuda 13.1?

[jonatanklosko]

The builds should be forward compatible. Jax also builds against CUDA 13 at the moment. If there are specific errors let us know, perhaps it's a build problem.

[byhemechi]

Apologies for the delay. I got around the issue by downloading an older version of cuda/nvshmem and putting them first in my ldpath, but it would be good to have a precompiled release that links against a newer version

$ ldd libxla_extension.so | grep "not found"
        nvshmem_transport_ibrc.so.3 => not found
        libnvrtc-builtins.so.13.0 => not found

$ tar tf libnvshmem-linux-x86_64-3.5.21_cuda13-archive.tar.xz | grep nvshmem_transport_ibrc
lib/nvshmem_transport_ibrc.so.4.0.0
lib/nvshmem_transport_ibrc.so
lib/nvshmem_transport_ibrc.so.4

$ tar tf cuda-13.1.1-1-x86_64.pkg.tar.zst | grep libnvrtc-builtins.so
opt/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so
opt/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so.13.1
opt/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so.13.1.115

19:17:35.593 [warning] The on_load function for module Elixir.EXLA.NIF returned:
{%RuntimeError{
   message: "Failed to load NIF library.\nFollow the steps in the :exla README Troubleshooting section for more information.\n\n:load_failed\nFailed to load NIF library /home/zsty/things/exla_dep_error/_build/dev/lib/exla/priv/libexla: 'nvshmem_transport_ibrc.so.3: cannot open shared object file: No such file or directory'\n"
 },
 [{EXLA.NIF, :__on_load__, 0, [file: ~c"lib/exla/nif.ex", line: 13, ...]}, ...]}

[byhemechi]

Bumping this because I'm still getting the error on every environment i've tried, libexla.so still links against a version of nvshmem_transport_ibrc.so the system doesn't have, no matter what i set HERMETIC_*_VERSION to

[polvalente]

@byhemechi you might have to check if your system has it in the LD_LIBRARY_PATH. It seems that some CUDA installers don't properly set up links

[byhemechi]

Yeah that was the first thing I checked. I have a nvshmem_transport_ibrc.so.5 in my ldpath, but not nvshmem_transport_ibrc.so.3. So far I have got things working by downloading an old nvshmem tarball but it's awkward to set up

[polvalente]

How about we change the target here to add more documentation around this instead of supporting newer versions in precompiled binaries?