[fma-rc]

On a Linux PC with an NVIDIA GPU and NVIDIA's VA-API driver installed, opening NVENC and VA-API at the same moment in one process never returns. That happens in cargo test: the two GPU round trips run in parallel, deadlock, and every other share worker test in the run hangs behind them (seen on an RTX 5080, driver 610.57).

The stacks show a lock-order inversion between CUDA and the loader. The NVENC thread is in cuInit, which holds CUDA's lock and loads a library, waiting for the loader's lock. The VA-API thread is in vaInitialize -> dlopen(nvidia_drv_video.so), holding the loader's lock, and that driver's constructor calls cuInit, waiting for CUDA's.

GPU drivers now start one at a time in a process: NvEncoder::open and VaEncoder::open hold one lock while they run. Encoding is not affected.

nvenc_and_vaapi_open_at_the_same_time opens both at once in a child process, so a deadlock fails the test after 30 s and does not hang the run. Without the change it never ended in 5 of 5 runs; with it 5 of 5 pass, and the share worker's unit tests finish in 3 s with the round trips included. With ANYPLAY_REQUIRE_HW=1 the NVENC round trip passes on the RTX 5080; the VA-API one reports that NVIDIA's VA-API driver has no H.264 encoder, as expected there.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[AlxcBTW]

Thank you for finding this, the diagnosis is excellent: the stacks (CUDA's lock against the loader's lock, through the VA-API driver's constructor calling cuInit) make the cause plain, and one process-wide lock around the two open calls is the smallest correct fix. Recovering from a poisoned lock is right, and the lock covers only the open, not encoding. Running the race in a child process with a 30 s deadline, so a regression fails instead of hanging the whole run, is a good test design.

The failing macOS check is real_window::the_worker_saves_a_clip_of_a_real_window ("no_window"), unrelated to this change; it was fixed on main in #127 and #131 after this branch was tested. I will close and reopen the PR so CI runs again against current main, and merge when all checks are green.

Approved.