Repository navigation
CUDA: prefetch the fixed pages the host writes before a bang (experiment) - #2014
nicolas-abril wants to merge 3 commits into
Conversation
An experiment for #1992, not for merge: comp.ts is 192 ttok over its cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
To remove before any merge. A line at exit gives the medians over the second half of the bangs, the corpus address and the region offsets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@nuxyel this is the experiment for #1992. Could you follow the steps under "How to measure" and post the results here? Step 3 decides whether prefetching helps, and it doesn't need |
|
Thanks for putting this together. I ran it with the program from #1992 (SmolLM2 135M, 8 tokens, plus 360M, 6 tokens). Setup: RTX 4050 Laptop 6 GB, driver 615.78.08, Core Ultra 7 155H, Linux. PR head 693eb30 and its base 017f83f as a control. Build: it builds and runs as written against both CUDA 13.4 and 12.9 (the binaries link Step 3, your loop (
Every run printed the same tokens as above,
The Step 4, page lists, 135M, CUDA 13.4 (two runs each way, second half of the run, faults a bang):
The ring and header faults are gone, as intended; one ring page (offset 142737408) alone took 156-161 faults a bang before. But the heap's GPU faults about double, and a third to a half of them land on one page, offset 1139347456, which wasn't among the busiest GPU pages without the prefetch (it faulted about 0.5 a bang on the CPU side). That page sits 4 KB above 1139343360, one of the heap pages that fault either way. CPU faults go up by 6-9 a bang. For what it's worth, on #1854 I measured prefetching only the header page before the launches at +56 µs a bang over the build without the entry check, so the header page alone doesn't pay for itself; the gain here comes with the ring's pages, whose faults are gone in step 4. |
The experiment for #1992 again. BEND_NEAR now picks the prefetches by letter: h the header, r ring row 0's get and put, s the ring slot, t the task's tail block, and w widens each one to its aligned 64 KB, clamped to the corpus. Unset or 1 still means all four at 4 KB, and 0 none. The exit line also reports the last tail block's offset and how often its page moved, so the fault lists can tell which heap page is the tail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@nuxyel thank you, this is very clear. 17-21% with identical outputs settles that prefetching helps. Before turning this into a PR to merge, I'd like to know which of the prefetches pay for themselves. Your #1854 number suggests the header one may not, and a 4 KB prefetch seems to make the heap page next to it (1139347456) much worse. Can you run one more round? I pushed e33c525 to this branch.
All on 135M, CUDA 13.4: 1. Get the new commit and build git fetch origin pull/2014/head:pr-2014-b
git checkout pr-2014-b
bun bend2/main.ts /path/to/your_program.bend -o near_test2. Time each setting, 5 rounds rm -f near.txt
for i in 1 2 3 4 5; do
for m in 0 hrst h r s t rst hrs hrstw; do
BEND_NEAR=$m ./near_test +RTS --gpu on > /dev/null 2>> near.txt
done
done
grep -h 'bend: near' near.txt | sort -s -k2,23. Page faults for each setting This uses the same for m in 0 hrst h r s t rst hrs hrstw; do
BEND_NEAR=$m nsys profile --trace=cuda --cuda-um-gpu-page-faults=true --cuda-um-cpu-page-faults=true --force-overwrite=true -o near_$m ./near_test +RTS --gpu on > /dev/null 2> near_$m.log
nsys export --type=sqlite --force-overwrite=true --output=near_$m.sqlite near_$m.nsys-rep
echo "=== BEND_NEAR=$m"
grep 'bend: near' near_$m.log
python3 pages.py near_$m.sqlite near_$m.log
done4. Post back the output of step 2's What I checked here: both sides of |
Refs #1992. An experiment for someone with a CUDA GPU to measure, not for merge. It has never run on a real GPU.
Before each bang, this prefetches the pages whose addresses the runtime already knows: the header, ring row 0's get, put and slot words, and the task's tail block. That's 5 of the 11 or so pages in #1992. It doesn't cover the reused heap blocks or the input vector, so this tests whether prefetching beats faulting per page, not the full possible saving.
How to measure
You need your program from #1992, the CUDA toolkit,
bun, andnsysfor step 4. Add your program's own arguments before+RTSin steps 3 and 4.1. Get this branch
git clone https://github.com/bendlang/bend.git bend-near cd bend-near git fetch origin pull/2014/head:pr-2014 git checkout pr-20142. Build your program
3. Compare prefetch off and on, 5 runs each
The binary prefetches by default, and
BEND_NEAR=0turns it off, so both use the same build. Each run prints onebend: near=...line to stderr when it exits.The times are medians over the second half of the bangs, in µs:
cycle_usbang_us + host_us. This is the number that decides itbang_ushost_usprefetches_per_bangprefetch_uscuMemPrefetchAsynccall (a mean over the whole run)failedThe
near=offlines should be close to the 1,614 µs #1992 reports for 135M. Ifnear=onisn't lower thannear=offby more than the spread between runs, prefetching doesn't help and this PR can be closed.4. See which pages still fault
Save the script:
Then profile both ways and print the page lists:
Look for how many heap pages are left, and whether a page's faults per bang stay near the same value. A busy heap page means the same block keeps being reused, which a log in the allocator could catch. The ring slot page moves on every bang, so spread-out ring pages are expected with prefetch off.
5. Post back
The output of step 3's
grep, the output of step 4, and your GPU, driver and CUDA version.What was checked
CUDA_VERSION >= 13000. Each bang made 5 prefetches, all page-aligned, inside the allocation and before the launch.BEND_NEAR=0made none, a failing call was reported, and stdout was the same either way.pages.pyran on a hand-made database with the columns in NVIDIA's export docs. A newernsysmay name things differently, and if a table is missing the script says so.nsyscommands, and the realcuda.h. The stub headers were written from docs, so CUDA 13 could still need a fix. The timing fields read near 0 against the fake, so only real runs can show those values are sensible.017f83fd.comp.tsis 66,986 ttok, 986 over its 66k cap, so the CI check will be red. It was exactly at the cap before this. The second commit is only the measuring code and goes away before any merge, but the first (about 192 ttok) would still need to find room.🤖 Generated with Claude Code