Skip to content

Fix single-GPU refine CUDA IPC failures - #134

Open
jdickerson95 wants to merge 1 commit into
development_v1.3from
jd_optimize_patch
Open

Fix single-GPU refine CUDA IPC failures#134
jdickerson95 wants to merge 1 commit into
development_v1.3from
jd_optimize_patch

Conversation

@jdickerson95

Copy link
Copy Markdown
Contributor

Summary

  • Run single-GPU core_refine_template / core_inspect_template in-process instead of spawning a child process after CUDA init
  • Avoids pidfd_getfd: Operation not permitted / empty torch.cat failures seen during optimize_template on one GPU
  • Multi-GPU paths still use run_multiprocess_jobs
  • Add unit tests covering single- vs multi-device dispatch

Test plan

  • pytest tests/backend/test_core_refine_template.py
  • Re-run single-GPU optimize_template on the failing machine/config
  • Spot-check multi-GPU refine still dispatches via multiprocessing

Made with Cursor

Avoid spawning a child process after CUDA init, which can fail with
pidfd_getfd errors and empty worker results during optimize/refine.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant