[Roadmap Feedback] Allow GPU-side descriptor creation for descriptor heaps

[tareksander]

Problem statement:

Descriptor heaps increase flexibility and allow GPU management of descriptors, but only in one way: Copying existing descriptors in the heap, or on and off the heap. BDA decreases the need for dynamically creating buffer descriptors, however e.g. filling a region of a storage buffer and letting other shaders access is as a uniform buffer could have improvements depending on how uniform buffers are implemented. The bigger problem is images though: You need an image descriptor to access the sampling hardware on the GPU. As it stands you'd need a CPU roundtrip if you need the image size to be dependent on GPU memory contents, reading the size from the GPU, creating the image and then continuing to dispatch shaders. Additionally this could be useful for a future extension that makes input attachments dynamic, which is mentioned in the descriptor heap feature proposal document. Together with BDA this would allow full GPU-side memory management (except creating new VkDeviceMemory allocations). It could be considered for optional features of the final KHR extension to create different descriptor types GPU-side.

[Tobski]

Hi @tareksander thanks for the feedback. You've described roughly what you'd like to see, but can you elaborate on what you would want to do with it at a higher level? It would help us both prioritise this and figure out what to actually design/what is possible.

Currently there's a lot of open questions here - for instance it's not obvious how you would do something like create an image descriptor without at least some sort of base image descriptor.

If you can provide more information on how it would be used and why, it would really help.

[tareksander]

For the use case: Imagine something like a work graph model where you just push input data into the GPU and ideally it should complete the whole work graph without CPU intervention. If the size of an intermediate or output image is determined by the input data, you'd have to either:

• Not use an actual image (I tried looking into how cuda does this, I think it just uses buffers with a good layout for shader access instead of actual image descriptors)
• Have a CPU roundtrip in the middle, create the image with the requested size, put it on the heap and continue the computations afterwards.

As a simple example, doing some calculations on a point cloud and outputting as an image. The dimensions of the image depends on the coordinates of the points (and scale maybe). This is more a scientific workload, but as we've seen with tensor cores being used in frame generation, a variety of things can also have applications in real-time graphics.

As for how to create the descriptor: Ideally an image descriptor is just the data (or only parts of it) that is passed to create the descriptor on the CPU. In that case, you could just have a SPIR-V instruction that takes the appropriate arguments and is lowered to the correct descriptor construction code by the driver. I don't know if hardware actually works like that, it could also work more like the border color registration case with more internal implementation work in side buffers, in which case it may not be that easy.

Since descriptor heaps seem to use the same data needed for ImageViews and there is no "non-view image" type in spirv, that'd probably be more like the ImageCreateInfo struct. Along with an instruction to query the memory requirements of such an image you'd be able to allocate a big chunk of memory via a physical pointer (e.g. a bump allocator), pass that as the "base pointer" for the image, and together with the other information (format, layers, mip levels, etc.) create a descriptor for the new image. Image alignment would need to be guaranteed for the base pointer of course, and maybe other things like memory heap of the pointer, which would need to be queried as properties and relayed to the shader beforehand.

Indirect dispatches could then iterate over the whole image, you could even sample it in a draw call after that (maybe for dynamic texture generation with LOD).

In any case, it seems like a reasonable extension to descriptor heaps, to be able to have "full" descriptor management on the GPU. Even if it is neither necessary for current rendering techniques nor implementable by some vendors, it should be a thing to keep in mind for the future.

[Tobski]

Thanks for the detail! That does help with the understanding.

I think the problem is that creating an image, with a BDA pointer or similar available to allocate from, is probably not trivial. If we took a subset of image creation and did it that way, something like that might be doable, but it's a lot of work and the performance would likely be ...not great. We'd basically be reimplementing a chunk of the driver in shader code.

And that all said, in the use cases you've defined, why not just allocate an image with the max possible size at the start of rendering, and suballocate from that manually? This would work today, and requires no additional API/shader features.

[tareksander]

While that would work, that's falling back to the "absolutely worst-case memory allocation" problem shaders have, and which shader forward progress guarantees should hopefully at least solve for BDA memory (since you'd just be able to block workgroups on memory availability, which means you don't need to e.g. reserve a 10GB buffer (a bit exaggerated) for all the invocations or split the dispatch up into multiple ones, with memory+execution barriers in between, that could hurt occupancy more than the driver deciding to somehow try to advance all workgroups into a memory-heavy section for some reason (and needing to block a large chunk of them for a time)). Since I read that the WG is working on defining what kind of forward progress guarantees can be given over multiple vendors, for that one a solution is on the horizon. It'd be nice to also have a solution for images on the horizon, even if it's 5 year later than that. I'm not advocating for introducing that feature for the KHR version of descriptor heaps if it's not possible to implement in a good way, I'd just like it to be somewhere on the list for future consideration.

How about other descriptor types? E.g. IIRC Nvidia still has a fast path for uniform buffers, so allocating a chunk of BDA memory in a compute prepass and then creating UBO descriptors for rendering access is another use case. And for that only an address should really be needed.

[Tobski]

I suspect for this we'd be more likely to be talking about dynamic allocations in some capacity, rather than pure descriptor creation, but that's fine, this is the sort of detail I was hoping to get at, so thanks for entertaining my questions.

We can certainly keep this on the radar, but it will definitely take a while.

How about other descriptor types? E.g. IIRC Nvidia still has a fast path for uniform buffers, so allocating a chunk of BDA memory in a compute prepass and then creating UBO descriptors for rendering access is another use case. And for that only an address should really be needed.

This type of thing you should just use device addresses directly. I don't want to speak for NV here, but my expectation is that creating descriptors would be more trouble than it's worth.

[oddhack]

@Tobski @r-potter I'm taking this off the maintenance agenda - it seems like something to put on the feature request list for the development call.