Skip to content

RR runtime: keep an instance's heaps until its last frame completes; no wait at destroy - #82

Merged
phantoslayer21 merged 9 commits into
mainfrom
fix/63-rr-heaps-retire
Nov 14, 2026
Merged

phantoslayer21 merged 9 commits into
mainfrom
fix/63-rr-heaps-retire

Conversation

@phantoslayer21

@phantoslayer21 phantoslayer21 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Fixes #63

The RR runtime released an instance's shader-visible descriptor heap and timestamp query heap at destroy after a
bounded wait, whether or not the game's command list that uses them had finished, and the destroy could stall the game
thread for up to 2 s + 2 s while holding the runtime-wide lock. This PR applies the lifetime rule shared by #63, #58
and #62, with completion proven per command list. The first version of this PR used a single runtime-wide marker,
and the batch review blocked it; that is fixed here (see "Batch review findings").

Batch review findings, one by one

1. Blocker: a later serial does not prove that an earlier recorded list ran. Fixed.

  • Every command list the runtime records into ends with a write of its serial into a marker slot of its own
    (rr_retire.h ListMarks). This covers a frame (evaluate), a pass-through and a transient copy (retain).
    A list counts as complete only when its own slot holds its serial. A later list completing proves nothing about it.
  • A marker slot is handed to a new list only after the list that held it has proved complete. So a slot's value always
    belongs to its current holder: a stale value from the previous holder never proves the new one.
  • An object is released only when every list that may use it has proved complete:
    • an instance's buffers, descriptor heap and query heap wait for all of the instance's lists that have not proved
      complete (Instance::lists);
    • an old output texture or an old arena (resize / re-layout) waits for the instance's lists, the current frame's
      included;
    • a transient copy waits for its copy list and for every frame or pass-through that read it.
  • The existing serial-based retirement shared the flaw, and this PR fixes it too:
    • defer_release of buffers (old arena, old output texture, transient copies) now takes the lists that use the
      object, not a serial;
    • slot reuse (the per-instance upload / descriptor / timestamp slots and the pass-through descriptor slots) used
      serial % 8 and waited until the shared marker passed the slot's old serial. A slot is now taken only when its own
      last list has proved complete. The wait is still at most 5 s, and the evaluate fails as before if no slot frees up.
    • The timing harvest reads a slot only when that slot's list has proved complete.
  • If a list is never submitted, its objects and its marker slot are kept, however many later lists complete. They are
    released when the device is removed. At most 1024 lists can be pending per device; past that, nothing more is recorded
    until one completes, and one log line says so.

2. Major: a device switch released the old device's pending objects. Fixed.

  • drop_device (set-device with another device, the end of prepare, test shutdown) no longer releases anything for
    which a list of that device has not proved complete. Instead it parks the old device's state (ParkedSet) until
    each of its lists proves complete, or GetDeviceRemovedReason reports the old device removed. The parked state is:
    • its ListMarks and deferred objects;
    • the objects every such list uses: pipelines, root signature, model and AGS buffers, pass-through heap and buffer,
      and its live transient copies;
    • the marker buffer the lists write;
    • the device reference, which is released last.
  • Bound: at most 4 old devices are tracked. Beyond that, the oldest one's objects are left allocated without
    tracking (reclaimed at process exit) and are never released without proof.
  • Log: one line when a device is parked (lists pending, objects kept), one when it is released (every list
    complete, or device removed), and one if the bound applies.
  • A device with no pending lists, or a removed one, is released at once, as before.

Final review findings, one by one

1. Major: a stale marker could falsely complete a new list after the 32-bit value wraps. Fixed.

  • The GPU writes only the low 32 bits of a serial, and a free marker slot keeps its last completed list's value.
  • ListMarks now records the value each slot holds: 0 when fresh, or the low 32 bits of its last proven holder.
  • begin skips any serial whose low 32 bits equal the chosen slot's current value, so the value a new list waits for
    always differs from what the slot already holds, however many serials have gone by. A skipped serial is never handed
    out. This replaces the old "never low 32 bits = 0" rule, which covered fresh slots only.
  • CPU test (wraparound_stale_marker): a slot is left free holding x while serials jump past x + 2^32 - 1 (test
    hook skip_to), and the next list is assigned to that slot. Its serial differs from x in the low 32 bits, and the
    stale value does not prove it. The test also checks that a fresh slot never gets a serial ending in 32 zero bits.

2. Major: the cap of 4 tracked old devices forgot pending generations, leaking them permanently. Fixed.

  • An old device now leaves the parked set only when it is released: every list proved complete, or its device is
    removed. RetireList::abandon is removed.
  • On a 5th park, the oldest is released at once if its marker slots show every list complete, or if
    GetDeviceRemovedReason reports its device removed.
  • Otherwise its marker slots are polled for at most 2 s (1 ms sleeps; under the runtime lock, only at this device
    switch), and it is released as soon as they complete.
  • If it is still pending after the wait, it stays tracked: the cap is soft. One log line names it and the number of
    old devices now tracked. Later pumps still release it when its lists complete or its device is removed.
  • CPU test (device_switch_soft_cap), 6 switches with work pending:
    • nothing completes: all 6 stay tracked and nothing is released; the 5th and 6th switches each wait at most
      2000 ms; the soft cap is logged. When the lists complete later, every object of all 6 is released exactly once;
    • the oldest's list completes during the 5th switch's wait: it is released once, and the wait ends then;
    • the oldest's device is removed: it is released at the 5th switch with no wait.

Other changes

  • Per-step progress in the RR e2e (test tools only). rr_test_host writes stdout and stderr unbuffered. It prints
    one step N <command> at T s line per job command and one line per 100 perf frames, so a run stopped by a time cap
    keeps its output and names the step it was in. e2e.py gets --gpu-timeout (default 900 s); on a failure or
    timeout the error names the last step. It also prints a line when the GPU run starts and when the CPU reference
    starts.
  • destroy still waits for nothing (rule point 3 of RR releases its descriptor heap and query heap at destroy without waiting for the GPU #63). The 2 s timing-harvest wait stays removed: frames still in
    flight at destroy are left out of the last timing window.

Test

python -m unittest -v clean.rr_f.runtime.tests.test_rr_retire builds rr_retire_test.cpp, which uses fake objects
and a fake marker buffer; "the GPU runs list s" is the write of s into s's slot. Passes, 40 of 40 checks, MSVC /W4
with no warnings. Besides the wraparound and soft-cap tests above, the checks cover:

  • out-of-order submission: A is recorded first and B later; B is submitted and completes first. A's buffer,
    descriptor heap and query heap are not released. They are released once A completes. The same holds for one
    instance with frames 1 and 2 in flight when frame 2 completes first;
  • a list that is never submitted: 100 later lists complete. None of them gets its marker slot, and its objects are
    still kept. They are released once each at device removal;
  • marker-slot reuse: with the pool full nothing more is recorded; a slot goes only to the list that follows a
    completed holder; a stale slot value does not prove the new holder; a cancelled (unrecorded) list gives its slot back;
  • device switch with work pending: the old device's objects, marker buffer and device are all kept while its list
    is pending. Each is released once when the list completes on the old device, or when the old device is removed,
    with one log line.

For comparison, the same out-of-order scenario against the first version's rr_retire.h (head 94e02b0, shared
marker) releases A's heap while A has not run. This was a scratch program, not committed.

GPU run

Built from this branch: the runtime (rr_runtime.cpp and the rest of src/) and rr_test_host. The embedded kernel
library comes from a build of main 234515a (kernels.json sha256 879012b1…, the same manifest as a second,
independent build of main). This is valid for three reasons:

  • no RR kernel source changed between 234515a and this branch's base;
  • build_dx12.py changed only in its kernel-set selection (Model L, frame generation), which invalidates the cache key
    but not the RR compiles;
  • the wg_platform.hlsli addition sits behind #if WG_REV_ORDER, which no RR kernel defines.

I also recompiled 16 RR kernels from this tree, and all 16 are byte-identical to that build's.

Full e2e: python -m clean.rr_f.runtime.tests.e2e --perf 0 with WG_RR_COMPILE_WAIT_ALL=1, tagloop included, the
GPU part in a GPU time slot (cap 170 s). PASS.

check frames passed
copies, exposure, head, input constants, input stage, network body, output stage, PH2 pool, resolve 74 / 74 each
lifecycle (par / fc / reset / rebuild / failed call) 75 / 75
failed call leaves state unchanged 1 / 1
aliased run = unaliased run (outputs, state) 75 / 75
inputs released with no instance: video memory growth (limit 256 MiB) 13.5 MiB over 200 frames

Host checks in the same run: retag (a copy released before its list ran is kept until the list runs, then freed) ok
twice; arrayin ok; border ok twice; leak 6 cycles with live resources 3 → 3, instances 0, deferred 0, video memory
unchanged; end with live 3, instances 0, deferred 0.

After the final-review fixes (marker wraparound, soft park cap), the full e2e was run once more on this branch
(--perf 0, the GPU part in a GPU time slot: 7.0 s, rc 0). PASS, with the same results as the table above:
every stage 74 / 74, lifecycle 75 / 75, aliased = unaliased 75 / 75, video memory growth 13.5 MiB over 200 frames.
The host checks (retag twice, arrayin, border twice, leak 6 cycles live 3 -> 3, end with deferred 0) were all ok.
Since that run, main was merged once more; the merge touched no RR or D3D12 kernel/build file.

10 repeat runs of the same GPU job, each in its own GPU time slot, each written to its own folder. Every dumped
frame file (outputs, alpha outputs, and the arena, which holds every state image), 372 files per run, was compared
byte for byte with the run verified above, together with the host's own checks. 10 of 10 passed; the longest took
8.6 s.

run 1 2 3 4 5 6 7 8 9 10
wall (s) 8.6 7.7 7.6 7.6 8.2 7.1 8.1 7.6 7.6 7.6
result pass pass pass pass pass pass pass pass pass pass

The one run that hit the time cap. The first attempt used e2e's default --perf 600. That job appends timing-only
perf lines after the checks: 600 exact + 600 exact-ext + 1200 fast frames at 2560x1440, then 60 more. The checks
themselves had all finished within about 8 s, every frame with rc 0. The exact head costs about 0.65 s of GPU time
per frame at that size, so the perf lines alone need more than 13 minutes, and the 170 s cap stopped the run. It was
not a hang. A perf-only A/B run of the same job (30 exact + 60 fast frames, same host, one slot each) gives the same
speed on main 234515a and on this branch: 649.1 vs 649.0 ms of GPU time per exact frame, 19.48 s wall each. The
host's output of that first attempt was lost to stdout buffering when the cap stopped it; the per-step progress lines
above fix that. The perf timings are not part of the correctness checks.

Tested

  • test_rr_retire: 40 of 40 checks (MSVC 19.51, Windows); python -m unittest discover -s clean/rr_f/tests -t .:
    241 tests OK (7 skipped); test_rr_retire + test_model_pack + test_model_bounds: 8 tests OK.
  • rr_runtime.cpp and rr_test_host.cpp build with the runtime build's compiler flags (/W3, no new warnings).
  • GPU e2e (above): PASS, plus 10 of 10 repeat runs byte-identical.
  • Perf A/B against main on the exact and fast heads (above): same GPU time per frame.

Not tested

  • A real game that destroys an RR instance between recording and submitting a list, a list a game never submits, a
    real device switch with work pending, and a real device removal. The CPU test covers the logic; the e2e covers the
    normal create / evaluate / destroy paths and the transient-copy paths on hardware.
  • An evaluate that throws after its list was started: its serial stays pending, so the instance's objects are kept
    (never released early). The runtime already disables RR for the session after such an exception.
  • The e2e with the default --perf 600 inside a GPU slot: it cannot fit in 170 s on main or on this branch (above).

…roy unchanged)

The deferred-release list of the RR runtime moves into a header-only RetireList (any
object with Release, a counted flag for the live-resource total) so that a CPU test can
drive it with a fake marker value and fake objects that count Release. The instance
destroy goes through retire_instance with its current behaviour: the buffers wait for the
frame marker, the descriptor and query heaps are released after the bounded wait.

test_rr_retire builds tests/rr_retire_test.cpp and fails here: a destroy with the marker
behind the instance's last frame releases both heaps (issue #63). The next commit fixes it.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…no wait at destroy

Fixes the premature release of issue #63:
- the shader-visible descriptor heap and the timestamp query heap go to the retire list
  with the instance's last serial, as its buffers do (not counted in live resources);
  they are released when the frame marker reaches that serial, at device drop, or once
  the device is removed (pump releases everything kept then); never on a timeout;
- a destroy while that frame is still in flight logs one line and keeps the objects;
- the 2 s wait before the heap release is gone, and so is the 2 s wait for the timing
  harvest in destroy_locked: destroy harvests what has completed and waits for nothing
  while it holds the runtime-wide lock.

test_rr_retire now passes.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…lete

Review of #82: a later serial does not prove that an earlier list ran
(recording order is not submission order), and a device switch released
the old device's in-flight objects without proof.

- Every list the runtime records into (frame, pass-through, transient
  copy) writes its serial into a marker slot of its own (rr_retire.h
  ListMarks); a slot is reused only after its holder proved complete.
- An object is released only when every list that may use it proved
  complete: instance objects wait for all the instance's lists, an old
  arena or output texture for the instance's lists, a transient copy for
  its copy list and every frame / pass-through that read it.
- Frame and pass-through slots are taken only when their last list
  proved complete (no "serial % 8" reuse on a later serial).
- Device switch: the old device's pending lists, deferred objects,
  shared pipelines / buffers, marker buffer and device reference are
  parked until each list proved complete or the device is removed; at
  most 4 old devices are tracked, beyond that the oldest is left
  allocated (never released early); both logged.
- CPU test: out-of-order submission, a never-submitted list, slot reuse,
  device switch with work pending, removal, the bound (27 checks).

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
A host run stopped by a time cap lost its buffered output, so a long
step (the exact-head perf frames at 2560x1440 take most of a second each)
could not be told from a hang.

- rr_test_host: stdout / stderr unbuffered; one "step N <command> at T s"
  line per job command and one per 100 perf frames on stderr.
- e2e.py: --gpu-timeout (default 900 s) for the host run; on a failure or
  timeout host.log keeps what the host printed and the error names the
  last step; phase lines for the GPU run and the CPU reference; --perf
  help says to use 0 inside a short GPU time slot.

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 marked this pull request as draft November 14, 2026 18:41
Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 marked this pull request as ready for review November 14, 2026 18:51
Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…orgotten

Final review of #82:

- Marker wraparound: the GPU writes only the low 32 bits of a serial and a
  free slot keeps its last completed value. ListMarks now remembers each
  slot's value and skips a serial whose low 32 bits equal it (0 for a
  fresh slot), so a stale value can never prove a new list after the
  serials wrap. CPU test forces the wrap (skip_to test hook).
- Device-park cap: an old device is never forgotten without being
  released. Beyond 4 tracked, the oldest is released at once if its lists
  completed or its device is removed (GetDeviceRemovedReason); otherwise
  its marker slots are polled for at most 2 s and it is released when
  they complete; if still pending it stays tracked (soft cap, logged).
  RetireList::abandon is removed. CPU test: 6 switches with work pending
  (none completes: all kept, then released once each when they do; the
  oldest completes during the wait; the oldest's device removed).
- The fake GPU of the CPU test maps serials to slots in a std::map (the
  wraparound test uses serials above 2^32).

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 merged commit 6d2b34b into main Nov 14, 2026
9 checks passed
@phantoslayer21
phantoslayer21 deleted the fix/63-rr-heaps-retire branch November 14, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RR releases its descriptor heap and query heap at destroy without waiting for the GPU

1 participant