Skip to content

RR runtime: give the pipeline compile thread a 64 MB stack; log failed pipelines - #196

Merged
phantoslayer21 merged 1 commit into
phantoslayer21:mainfrom
j0nsh1n:fix/rr-compile-thread-stack
Nov 14, 2026
Merged

phantoslayer21 merged 1 commit into
phantoslayer21:mainfrom
j0nsh1n:fix/rr-compile-thread-stack

Conversation

@j0nsh1n

@j0nsh1n j0nsh1n commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

What this changes and why

Fixes #149.

Under Proton, the Ray Reconstruction runtime's background thread that compiles the kernels' pipelines dies of a stack overflow on the opt-in verification kernels (WG_RR_HEAD=exact/strict/debug, WG_RR_BODY=X/Z, WG_RR_PATHS=verified). vkd3d-proton translates each kernel on the thread that calls CreateComputePipelineState, and the structurizer in current dxil-spirv recurses about once per branch (reported as HansKristian-Work/dxil-spirv#366). A std::thread gets the game's default stack: 1 MB in rr_test_host (stage.IN0 overflows), 2 MB in Cyberpunk 2077 (stage.D4-F overflows). Nothing was logged, and compile_failed was never set, so the opt-in variants stayed "still being prepared". The menu modes (Exact, Accurate, Fast) are compiled in earlier passes and are not affected. Windows with AMD's driver is not affected either.

One file, clean/rr_f/runtime/src/rr_runtime.cpp:

  • A CompileThread class replaces the std::thread. It uses CreateThread with a 64 MB stack reservation (STACK_SIZE_PARAM_IS_A_RESERVATION: address space only, pages are committed as they are used). It has joinable() and join(), so the call sites stay as they are.
  • make_pso logs the kernel name, its size and the HRESULT when CreateComputePipelineState fails, so a failed pipeline shows in wronggpu-rr.log.
  • set-device fails cleanly (logs, drops the device, returns WG_E_D3D_API) if the thread cannot start, as its other failures do.

This changes how the compile thread starts for every player, on Windows too. The maintainer said an MSVC build and a Windows run will be done before merge.

What each new or substantially changed file is based on (public document, test result or own design)

rr_runtime.cpp only. Own design, using Microsoft's documented Win32 CreateThread and STACK_SIZE_PARAM_IS_A_RESERVATION. The stack sizes come from my own measurements of the translation on Linux (stage.D4-F needs 2 MB, stage.IN0 1 MB), with synthetic data.

How it was tested (CI runs the CPU tests; say what else you ran: GPU or game tests, with synthetic data)

Built with llvm-mingw only (the Linux route of clean/linux/README.md). No MSVC build and no Windows run here.

Test: rr_test_host under GE-Proton 11-11, vkd3d-proton 5b1089c8 + dxil-spirv #362, RX 9070 XT (RADV), the perf job at 1706x960 -> 2560x1440 with the verification head (exact), WG_RR_COMPILE_WAIT_ALL=1, a new prefix and Mesa cache for each run, on current main (6622148) with and without this change.

Before (main) After (this PR)
Log stops at pso 0 stage.IN0: start (808528 bytes) pso 0 stage.IN0: 1047 ms, then opt-in variant kernels prepared after 113311 ms
Wine err:virtual:virtual_setup_exception stack overflow no stack overflow
Frames 30 frames, 0 ok (30 failed) 30 frames, 30 ok (0 failed)

Checklist

  • I make the clean-room statement of
    CONTRIBUTING.md
    for this and all my contributions (required once, in my first pull request).
  • Every commit has a Signed-off-by: line (git commit -s; Developer Certificate of Origin 1.1).
  • New source files carry the SPDX licence line for MIT, or are covered in REUSE.toml. (No new files.)
  • No files of the original upscaler (see CONTRIBUTING.md), no model data, game files, captures, screenshots or
    binaries; no paths, user or machine names of my PC.
  • Any new third-party component is under a permitted licence and listed in THIRD-PARTY-NOTICES and REUSE.toml. (None added.)
  • Numbers in documents come with their measurement record. (No document changes.)
  • Kernel or runtime changes: the runtime changes, the kernels do not. Only the compile thread's stack and a log line change; the output is unchanged.

🤖 Generated with Claude Code

…d pipelines

Under Proton, vkd3d-proton translates each kernel on the thread that calls
CreateComputePipelineState. The structurizer in current dxil-spirv recurses
about once per branch (HansKristian-Work/dxil-spirv#366), so the opt-in
kernels stage.IN0 and stage.D4-F overflowed the std::thread's default stack
(1 MB in rr_test_host, 2 MB in Cyberpunk 2077). The thread died without
setting compile_failed and the opt-in variants never became ready.

- CompileThread (CreateThread, 64 MB with STACK_SIZE_PARAM_IS_A_RESERVATION:
  address space only) replaces std::thread; it keeps joinable() and join(),
  so the call sites do not change.
- make_pso logs the kernel name, its size and the HRESULT when
  CreateComputePipelineState fails.
- set-device logs and fails (WG_E_D3D_API) if the thread cannot start.

Verified with a llvm-mingw build only (no MSVC build, no Windows run):
rr_test_host under GE-Proton 11-11, vkd3d-proton 5b1089c8 + dxil-spirv #362,
RX 9070 XT, verification head (perf ... exact). Before: the log stops at
"pso 0 stage.IN0: start", Wine reports a stack overflow, 0 of 30 frames ok.
After: "opt-in variant kernels prepared", no overflow, 30 of 30 frames ok.

Fixes phantoslayer21#149

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
@phantoslayer21
phantoslayer21 merged commit a784207 into phantoslayer21:main Nov 14, 2026
9 checks passed
@j0nsh1n
j0nsh1n deleted the fix/rr-compile-thread-stack branch November 14, 2026 23:13
j0nsh1n added a commit to j0nsh1n/wronggpu that referenced this pull request Nov 15, 2026
Brings the launcher branch up to upstream main (217 commits: phantoslayer21#47, phantoslayer21#105,
phantoslayer21#196, phantoslayer21#197, phantoslayer21#209, Model K/L, frame generation, ...). Changes beyond
resolving the three conflicts (CMakeLists.txt project(), util.h's includes,
the restorer test):

- POSIX layer: the Win32 calls main's handle-based restore uses
  (SetFileInformationByHandle with FileBasicInfo, FileDispositionInfo and
  FileDispositionInfoEx's POSIX delete, GetFileInformationByHandleEx
  FileBasicInfo, SetEndOfFile, ExitProcess, GetProcessHeap/HeapFree, CDECL,
  ERROR_TOO_MANY_LINKS/FILE_CORRUPT/READ_FAULT), with posix-layer tests.
- run_launcher_tests.py stays below the 256 KB source limit: the edition
  section moves unchanged to launcher_edition_cases.py and the Linux run's
  plumbing to native_linux.NativeT.
- launcher-native runs only in a -DWRONGGPU_PRESET_K=OFF build (like CI's
  launcher build: the suite expects no seeded wronggpu-asi.ini); its two
  checks that fail on main's own launcher too (phantoslayer21#219) are skipped while
  they fail (native_linux.KNOWN_ON_MAIN).

Verified: fresh build, ctest 22/22 (launcher-native 365 passed, 0 failed,
6 skipped: 4 Windows-only sections, the 2 phantoslayer21#219 checks); reuse lint passes.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RR runtime: compile thread dies of a stack overflow under Proton with current vkd3d-proton (fix included)

2 participants