diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 104595c..f8b7b3f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -347,17 +347,16 @@ Without subgroup operations the prefix sum costs two barriers and about 20 workg ### Inverse transform -A workgroup of 256 invocations produces a 32x32 tile of one level's output: +A workgroup of 256 invocations produces a 32x32 tile of one level's output from the tile's 40x40 neighbourhood in the interleaved image. Each coordinate of the neighbourhood is mirrored about the level's true size, using whole-sample symmetry, which preserves the parity of the coordinate; the parity picks the band (LL, HL, LH or HH) and the mirrored coordinate halved picks the sample. Mirroring keeps the parity up to the level's width less two past an edge, and the stored samples read no further than four, so the horizontal pass can take a row's even samples from one band and its odd ones from another while a level is at least six samples wide. Every level is at least eight, by the minimum image size of 128. Four samples of extension on each side are what four lifting steps need. -1. It loads a 40x40 window of the interleaved image into workgroup memory (6400 bytes). Each window coordinate is mirrored about the level's true size, using whole-sample symmetry, which preserves the parity of the coordinate; the parity picks the band (LL, HL, LH or HH) and the mirrored coordinate halved picks the sample. Four samples of extension on each side are what four lifting steps need. -2. Horizontal pass: 160 invocations each take 16 samples of a window row, apply the gain and the four lifting steps in registers and keep the 8 valid results (workgroup memory, 5120 bytes). The apron of a segment is recomputed rather than shared. -3. Vertical pass: 128 invocations do the same down a column and store the 8 results into the LL band of the next level. At the last level of a component they are 8-bit samples instead, packed as described next. +1. Horizontal pass: 160 invocations each read 16 samples of a row of the neighbourhood straight from the bands, apply the gain and the four lifting steps in registers and keep the 8 valid results (workgroup memory, 5120 bytes). A row's even samples come from one band and its odd ones from another. The apron of a segment is read and recomputed rather than shared. +2. Vertical pass: 128 invocations do the same down a column of those results and store the 8 results into the LL band of the next level. At the last level of a component they are 8-bit samples instead, collected in workgroup memory (4096 bytes) and packed as described next. -The lifting, and the loads and stores around it, are written out with constant indices, which keeps the samples in registers (D24). Total workgroup memory is 11520 bytes, and it is not zeroed first: the load and the two passes write every element before it is read (D25). Tiles that stick out of the image (the coarsest levels can be 8x8) are computed in full and stored with bounds checks, and mirroring always uses the true level size, never the picture size. The component is the z index of the dispatch; at level 0 a 4:2:0 frame has luma only. +The lifting, and the loads and stores around it, are written out with constant indices, which keeps the samples in registers (D24). Total workgroup memory is 9216 bytes, and it is not zeroed first: the two passes write every element before it is read (D25). Tiles that stick out of the image (the coarsest levels can be 8x8) are computed in full and stored with bounds checks, and mirroring always uses the true level size, never the picture size. The component is the z index of the dispatch; at level 0 a 4:2:0 frame has luma only. Until D68 a first phase loaded the neighbourhood into workgroup memory, and the horizontal pass read it from there. ### 8-bit output -The last level of each component writes the decoded plane as 8-bit samples, which is what upstream's UNORM storage images hold. A sample is `round(clamp(v + 0.5, 0, 1) * 255)` (`to_unorm8` in `idwt.wgsl`); the float picture never goes to memory. The vertical pass leaves the 8-bit levels of its 32x32 tile in the window array, which is free by then, and after one barrier each of the 256 invocations writes one word: four samples of a tile row. Every level waits at that barrier, though only the last needs it. Inside `if (is_final)` the barrier would be legal, since `is_final` is the same for the whole workgroup, but there it crashed WARP, Direct3D's software rasteriser, in CI; a bisection of the commits pinned the crash on it. +The last level of each component writes the decoded plane as 8-bit samples, which is what upstream's UNORM storage images hold. A sample is `round(clamp(v + 0.5, 0, 1) * 255)` (`to_unorm8` in `idwt.wgsl`); the float picture never goes to memory. The vertical pass leaves the 8-bit levels of its 32x32 tile in the levels array, and after one barrier each of the 256 invocations writes one word: four samples of a tile row. Every level waits at that barrier, though only the last needs it. Inside `if (is_final)` the barrier would be legal, since `is_final` is the same for the whole workgroup, but there it crashed WARP, Direct3D's software rasteriser, in CI; a bisection of the commits pinned the crash on it. WGSL's `round` rounds ties to even on every backend (naga emits `rint`, `round` and `RoundEven` for Metal, HLSL and SPIR-V), which is what a UNORM storage write does. Four pixels share a word, first pixel in the low byte, and rows are padded to whole words with zero samples. A decoder's rows are then padded with words it never writes to a multiple of 256 bytes, the rows a copy into a texture takes, and its planes start at multiples of 512 bytes (D41). The CPU side cuts the padding off. @@ -598,7 +597,7 @@ PSNR is not what either encoder minimises: both minimise the squared error of th |---|---|---|---| | Input from textures | `gather.wgsl` | 1, in the transform's pass: a workgroup per 64x32 samples of a plane, the component in z | Each sample of the R8 or Rg8 textures, its byte back from the unorm float, packed four to a word with rows padded to whole words: the words the upload writes for the same picture | | Input in RGB | `gather_rgb.wgsl` | 1, in the transform's pass: a workgroup per 128x32 pixels for 4:2:0, 64x16 for 4:4:4 | Each pixel's R', G' and B' bytes, converted by `ForwardMatrix` in integers, a 4:2:0 chroma sample from the sums of the pixels it covers: the words the upload writes for the planes `rgb_to_ycbcr8` makes (D65) | -| Forward transform | `fdwt.wgsl` | 5: one per level, finest first; a workgroup per 32x32 tile of the level's image and component | The inverse transform's shape and parameters (`TransformLevel`), the other way round: a 40x40 window of the image, rows and then columns lifted in registers, and the four bands written. At a component's finest level the image is the 8-bit plane, its last column and row repeated up to the aligned size | +| Forward transform | `fdwt.wgsl` | 5: one per level, finest first; a workgroup per 32x32 tile of the level's image and component | The inverse transform's shape and parameters (`TransformLevel`), the other way round: a 40x40 window of the image, its rows read straight from the image and lifted, then its columns, in registers, and the four bands written. At a component's finest level the image is the 8-bit plane, its last column and row repeated up to the aligned size | | Quantisation | `quantise.wgsl`, `equant.wgsl` | 1: a workgroup of 128 invocations per block | Scales, magnitudes and bit-planes of `quant.rs`, then the block's size and distortion for 0 to 15 dropped planes, its cost table. Each invocation adds up the squared errors of its own 4x2 subblock in registers, and the subblocks are added up per 8x8 block, in the grouping the CPU adds in. Each 8x8 block is measured only up to the drop that leaves nothing of it; beyond, its error is the same sum of the same terms | | Rate control | `rate.wgsl`, `scan.wgsl` | 4, and two prefix sums of 3 | Each block's steps, and their saved words added into a histogram of the keys; the key at which the frame fits; what each block's steps of that key save, summed in block order; each block's drop and size, and their sums in block order, which are the blocks' places in the payload | | Packing | `quantise.wgsl`, `pack.wgsl` | 1: a workgroup per block | The block quantised again, the chosen planes dropped, its bytes assembled in workgroup memory and written at its place; a block of the coarsest level with nothing left, below `pristine_blocks` in block index order, as an empty 8x8 block (D58) | @@ -700,7 +699,7 @@ Short-form ADRs, newest last. Each entry is dated. **Context.** Upstream loads the inverse transform's input through `textureGather` and a mirrored sampler, and stores the wavelet pyramid as FP16 or FP32 storage images. `r16float` storage images are not in the portable set, and the sampler's mirroring is not the whole-sample symmetric extension the format needs without per-coordinate fix-ups. -**Decision.** Explicit mirroring of the interleaved coordinates and plain `f32` storage buffers. A 32x32 output tile per workgroup with a 40x40 window in workgroup memory, as described above. FP16 packing of the pyramid is an optimisation for later, to be weighed against the accuracy it costs (the original's default FP16 storage is why it sits about 20 dB below its FP32 mode). +**Decision.** Explicit mirroring of the interleaved coordinates and plain `f32` storage buffers. A 32x32 output tile per workgroup with a 40x40 window in workgroup memory, as described above (D68 later read the window's rows straight from the bands). FP16 packing of the pyramid is an optimisation for later, to be weighed against the accuracy it costs (the original's default FP16 storage is why it sits about 20 dB below its FP32 mode). ### 2026-10-23 — D14: Output as packed 8-bit buffers @@ -1718,3 +1717,33 @@ The last frame's detail, always, gains everything on a still picture and loses u - #136 is closed. Within its objective the encoder has under 1 % left, and measured by a metric of perceived quality, the objective itself has a few percent of bytes left. - If game footage of more than one frame, such as #82's, shows the chroma tilt's gain, the tilt goes into `gen_encoder_weights.py` as chroma's own sensitivity, for both encoders, and D57's measure keeps upstream's weights. - SSIMULACRA2 stays out of the repository. In `emberwave-cli quality` it would add a dependency, and its BSD-2-Clause notice to every release, for a study that ran once. + +### 2026-10-31 — D68: The transforms read their rows straight from the bands + +**Context.** On an Apple M3 Max the inverse transform is 70 % of a 4K decode's GPU time frame after frame, 0.38 of 0.54 ms. A workgroup loaded its tile's 40x40 neighbourhood into workgroup memory first, then lifted the neighbourhood's rows from there, then the tile's columns from the rows' results: three phases with a barrier after each, the first all loads. The GPU encoder's forward transform has the same shape. + +**Measured** on the M3 Max, frame after frame, by the GPU's timestamps of a transform's five dispatches (`emberwave-cli bench` and `bench-encode` on the generated scene streams), the median of 400 frames for the decoder and of 300 for the encoder, three rounds in turn with the build before. The Mac was busy, at a load of 4 to 6, and the rounds it disturbed are left out. + +| Inverse transform | 1080p | 4K | Output | +|---|---|---|---| +| Before: the neighbourhood loaded first, 11.25 KiB of workgroup memory | 0.104 to 0.107 ms | 0.378 to 0.381 ms | | +| The rows read straight from the bands, 9 KiB | 0.067 to 0.070 ms | 0.300 to 0.302 ms | The same, byte for byte | +| The same, the last level's samples kept in the rows' array, 5 KiB | 0.070 to 0.071 ms | 0.302 to 0.306 ms | The same | +| The same, rows inside the level read without mirroring | 0.068 ms | 0.301 to 0.303 ms | 3360 of 746 million samples one level off | +| Tiles of 32x48 or 32x64, so that more invocations lift columns | 0.074 to 0.079 ms | 0.303 to 0.310 ms | The same | + +| Forward transform, from planes in CPU memory | 1080p | 4K | Output | +|---|---|---|---| +| Before | 0.107 to 0.109 ms | 0.378 to 0.381 ms | | +| The rows read straight from the image | 0.087 ms | 0.317 to 0.320 ms | The same frames, byte for byte | + +**Decision.** +- Both transforms lift the rows of the neighbourhood as they read them, straight from the bands or the image. Mirroring keeps a coordinate's parity, so in the inverse transform a row's even samples come from one band and its odd ones from another. Far enough past an edge, where the second reflection is clamped, it does not, but no sample that is stored reads that far while a level is at least six samples wide, and the minimum image size of 128 makes every level at least eight. A test holds the levels to that. The review compared the old and new shaders' whole outputs at 5760 sizes from 8 to 140 and found them equal. The load phase, its 6400 bytes of workgroup memory and a barrier go. +- The rest stays as it was: the last level's samples in an array of their own, every coordinate mirrored, and tiles of 32x32. No variant beat them. Reading without mirroring also changed the output, probably where the compiler fused multiplies and adds differently. +- The output is the same to the byte: the decodes of the generated streams and of the test vectors, and the encoder's frames of the generated scenes and of the game frame of D49. D9's comparisons stand as they were. + +**Consequence.** +- The decoder's GPU time for a 4K frame falls from about 0.54 to 0.47 ms frame after frame, and for a 1080p frame from 0.16 to 0.12 ms. The encoder's falls by about 0.06 ms at 4K. +- Less workgroup memory alone, 5 KiB instead of 9, made the inverse transform no faster: the load phase held it back, not the memory a workgroup takes. Halving workgroup memory, which D48 counted among the sources of upstream's FP16 gain, would not help emberwave's kernel by itself. +- `BENCHMARKS.md` keeps its tables until the next full run on a quiet machine. +- The test of the workgroup variables checked for writes before reads (D25) lists the new ones. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e335fd..8870e9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ All notable changes to emberwave. The format follows [Keep a Changelog](https:// - Frames drawn in part conceal the blocks of detail they lack: where the coarsest level around such a block changed by little since the last decode, the picture is taken to be still and the block keeps the last frame's coefficients, so a still picture keeps its detail through lost packets, and a moving one decodes as before (decision D66, [#129](https://github.com/with-developer/emberwave/issues/129)). Off by default: `gpu::Decoder::set_concealment`, the C API's `emberwave_decoder_set_concealment` and the browser player's `set_concealment` turn it on, and `cpu::Concealer` is the CPU's. Measured with the CPU's: frames drawn from 40 to 60 % of their blocks of a still game frame came out as the whole frame, 3 to 4 dB of luma PSNR above the zeros of before, those of a mostly still scene 12 to 16 dB above them, and those of a picture moving by one to eight pixels a frame within 0.03 dB of them. A 4K frame takes 0.06 ms more on an Apple M3 Max at most, where its blocks are zeroed. +### Changed + +- The wavelet transforms are faster on the GPU, with the same output to the byte (decision D68): a workgroup lifts the rows of its tile's neighbourhood straight from the bands or the image instead of loading the neighbourhood into workgroup memory first. On an Apple M3 Max, frame after frame, the decoder's inverse transform takes 0.30 ms instead of 0.38 ms at 4K and 0.067 ms instead of 0.105 ms at 1080p, and the encoder's forward transform 0.32 ms instead of 0.38 ms at 4K. + ## [0.15.0] - 2026-10-31 Pictures in RGB encoded as hosts' captures hold them. The encoder converts BGRA or RGBA to Y'CbCr as it reads them, from textures of Metal, Vulkan and Direct3D or from CPU memory, in integers that make the same bytes on every GPU and on the CPU. Also a frame assembler that no stream of packets can slow down, and late packets ignored as the docs say. diff --git a/crates/emberwave-core/src/gpu/layout.rs b/crates/emberwave-core/src/gpu/layout.rs index b3c2600..b2c2f7e 100644 --- a/crates/emberwave-core/src/gpu/layout.rs +++ b/crates/emberwave-core/src/gpu/layout.rs @@ -760,6 +760,29 @@ mod tests { assert_eq!(MAX_BLOCK_WORDS, 622); } + #[test] + fn every_transform_level_is_at_least_eight_samples_wide() { + // idwt.wgsl takes a row's even samples from one band and its odd ones + // from the other, as mirroring keeps a coordinate's parity. Past the + // level's width less two it does not, and the samples that are stored + // read up to four past an edge, so levels have to be at least six + // samples wide (decision D68). The minimum image size makes the + // smallest frames' coarsest levels eight. + for chroma in [ChromaSubsampling::Yuv420, ChromaSubsampling::Yuv444] { + let layout = FrameLayout::new(2, 2, chroma).unwrap(); + let buffers = BufferLayout::new(&layout).unwrap(); + for level in 0..DECOMPOSITION_LEVELS { + let params = TransformLevel::new(&layout, &buffers, level).params; + assert!( + params.image_width >= 8 && params.image_height >= 8, + "{chroma:?} level {level}: {}x{}", + params.image_width, + params.image_height + ); + } + } + } + #[test] fn the_largest_frame_fits_the_scans_and_32_bit_word_counts() { // 16384x16384 at 4:4:4: three components of 262144 blocks. diff --git a/crates/emberwave-core/src/gpu/shaders/fdwt.wgsl b/crates/emberwave-core/src/gpu/shaders/fdwt.wgsl index 4056c4c..9179b8f 100644 --- a/crates/emberwave-core/src/gpu/shaders/fdwt.wgsl +++ b/crates/emberwave-core/src/gpu/shaders/fdwt.wgsl @@ -5,8 +5,9 @@ // ("Wavelet transform") as emberwave's CPU encoder does it // (src/encode/fdwt.rs), in the shape of idwt.wgsl: a workgroup takes a 32x32 // tile of the level's image with 4 samples of whole-sample symmetric extension -// on each side (a 40x40 window), lifts its rows and then its columns in -// registers, and writes the 16x16 corner of each band the tile gives. At a +// on each side (a 40x40 window), lifts the window's rows, read straight from +// the image, into workgroup memory, then the tile's columns from there, and +// writes the 16x16 corner of each band the tile gives. At a // component's finest level the image is the 8-bit plane, extended to the // aligned size by repeating its last column and row, at the other levels the // LL band the level before wrote. Workgroup z selects the colour component. @@ -61,15 +62,12 @@ const DELTA: f32 = 0.443506852043971; const K: f32 = 1.230174104914001; const INV_K: f32 = 0.8128930661159609; -// Image tile and the window of it, with its extension, that feeds a workgroup. +// Image tile, and the extension of its window on each side. const TILE: u32 = 32u; -const WINDOW: u32 = 40u; const APRON: i32 = 4; -// Workgroup memory is not zeroed for this pipeline (decision D25): the load -// fills all of `window` and the horizontal pass all of `horizontal` before -// either is read. -var window: array; +// Workgroup memory is not zeroed for this pipeline (decision D25): the +// horizontal pass fills all of `horizontal` before it is read. var horizontal: array; // Whole-sample symmetric mirroring: -1 -> 1, n -> n - 2. @@ -151,38 +149,32 @@ fn main(@builtin(local_invocation_index) tid: u32, @builtin(workgroup_id) wg: ve let tile_x = wg.x * TILE; let tile_y = wg.y * TILE; - for (var i = tid; i < WINDOW * WINDOW; i = i + 256u) { - let row = i32(i / WINDOW); - let col = i32(i % WINDOW); - let x = mirror(i32(tile_x) + col - APRON, image_width); - let y = mirror(i32(tile_y) + row - APRON, image_height); - window[i] = image_at(comp, u32(x), u32(y)); - } - workgroupBarrier(); - - // Horizontal pass: one invocation per segment of a window row. Window - // columns 8s..8s+16 give the tile columns 8s..8s+8. + // Horizontal pass, straight from the image: one invocation per segment of + // a window row. Window columns 8s..8s+16 give the tile columns 8s..8s+8. + // Loading the window into workgroup memory first made the transform take + // a fifth longer at 4K (decision D68). if (tid < 160u) { let row = tid / 4u; let segment = tid % 4u; + let y = u32(mirror(i32(tile_y + row) - APRON, image_height)); + let x0 = i32(tile_x + 8u * segment) - APRON; var v: array; - let base = row * WINDOW + 8u * segment; - v[0] = window[base + 0u]; - v[1] = window[base + 1u]; - v[2] = window[base + 2u]; - v[3] = window[base + 3u]; - v[4] = window[base + 4u]; - v[5] = window[base + 5u]; - v[6] = window[base + 6u]; - v[7] = window[base + 7u]; - v[8] = window[base + 8u]; - v[9] = window[base + 9u]; - v[10] = window[base + 10u]; - v[11] = window[base + 11u]; - v[12] = window[base + 12u]; - v[13] = window[base + 13u]; - v[14] = window[base + 14u]; - v[15] = window[base + 15u]; + v[0] = image_at(comp, u32(mirror(x0 + 0, image_width)), y); + v[1] = image_at(comp, u32(mirror(x0 + 1, image_width)), y); + v[2] = image_at(comp, u32(mirror(x0 + 2, image_width)), y); + v[3] = image_at(comp, u32(mirror(x0 + 3, image_width)), y); + v[4] = image_at(comp, u32(mirror(x0 + 4, image_width)), y); + v[5] = image_at(comp, u32(mirror(x0 + 5, image_width)), y); + v[6] = image_at(comp, u32(mirror(x0 + 6, image_width)), y); + v[7] = image_at(comp, u32(mirror(x0 + 7, image_width)), y); + v[8] = image_at(comp, u32(mirror(x0 + 8, image_width)), y); + v[9] = image_at(comp, u32(mirror(x0 + 9, image_width)), y); + v[10] = image_at(comp, u32(mirror(x0 + 10, image_width)), y); + v[11] = image_at(comp, u32(mirror(x0 + 11, image_width)), y); + v[12] = image_at(comp, u32(mirror(x0 + 12, image_width)), y); + v[13] = image_at(comp, u32(mirror(x0 + 13, image_width)), y); + v[14] = image_at(comp, u32(mirror(x0 + 14, image_width)), y); + v[15] = image_at(comp, u32(mirror(x0 + 15, image_width)), y); lift(&v); let out = row * TILE + 8u * segment; horizontal[out + 0u] = v[4]; diff --git a/crates/emberwave-core/src/gpu/shaders/idwt.wgsl b/crates/emberwave-core/src/gpu/shaders/idwt.wgsl index 42d049d..edde024 100644 --- a/crates/emberwave-core/src/gpu/shaders/idwt.wgsl +++ b/crates/emberwave-core/src/gpu/shaders/idwt.wgsl @@ -9,9 +9,10 @@ // // One dispatch combines the LL, HL, LH and HH bands of a level into the LL band // of the next finer level (or into the final plane). A workgroup produces a -// 32x32 tile of the output: it loads the 40x40 interleaved neighbourhood -// (4 samples of extension on each side, one per lifting step), filters -// horizontally and then vertically in workgroup memory, and stores the tile. +// 32x32 tile of the output from its 40x40 interleaved neighbourhood (4 samples +// of extension on each side, one per lifting step): it lifts the rows of the +// neighbourhood, read straight from the bands, into workgroup memory, then the +// columns of the tile from there, and stores the tile. // At the last level of a component the tile becomes 8-bit samples, which is // what upstream's UNORM storage images hold; the float picture never goes to // memory. Workgroup z selects the colour component. @@ -62,20 +63,20 @@ const DELTA: f32 = 0.443506852043971; const K: f32 = 1.230174104914001; const INV_K: f32 = 0.8128930661159609; -// Output tile and the interleaved window that feeds it. +// Output tile, and the extension of its neighbourhood on each side. const TILE: u32 = 32u; -const WINDOW: u32 = 40u; const APRON: i32 = 4; // Shifts the DC level back (bitstream.md, "Inverse DC shift"). const DC_SHIFT: f32 = 0.5; // Workgroup memory is not zeroed for this pipeline (decision D25): every -// element is written before it is read. The load fills all of `window`, the -// horizontal pass all of `horizontal`, and at the last level the vertical -// pass rewrites the 32x32 corner of `window` that the packing reads. -var window: array; +// element is written before it is read. The horizontal pass fills all of +// `horizontal`, and at the last level the vertical pass all of `levels`, +// which the store reads. var horizontal: array; +// The last level's tile, as 8-bit levels. +var levels: array; // Whole-sample symmetric mirroring: -1 -> 1, n -> n - 2. fn mirror(index: i32, n: i32) -> i32 { @@ -148,7 +149,7 @@ fn store_ll(comp: CompParams, x: i32, y: i32, value: f32) { // A final sample as an 8-bit level: the DC shift is added by the caller, the // result clamped to [0, 1] and rounded like a UNORM store. WGSL's round() // rounds ties to even on every backend. Kept as a float so that it can wait in -// the window array; it is a whole number from 0 to 255. +// the levels array; it is a whole number from 0 to 255. fn to_unorm8(v: f32) -> f32 { return round(clamp(v, 0.0, 1.0) * 255.0); } @@ -161,42 +162,43 @@ fn main(@builtin(local_invocation_index) tid: u32, @builtin(workgroup_id) wg: ve let tile_x = i32(wg.x * TILE); let tile_y = i32(wg.y * TILE); - // Load the interleaved window. The low bit of a mirrored coordinate says - // which band the sample comes from: LL, HL (odd column), LH (odd row), HH. - for (var i = tid; i < WINDOW * WINDOW; i = i + 256u) { - let row = i32(i / WINDOW); - let col = i32(i % WINDOW); - let x = mirror(tile_x + col - APRON, out_width); - let y = mirror(tile_y + row - APRON, out_height); - let band = u32(x & 1) | (u32(y & 1) << 1u); - let index = comp.band_offset[band] + u32(y >> 1u) * params.band_width + u32(x >> 1u); - window[i] = pyramid[index]; - } - workgroupBarrier(); - - // Horizontal pass: one invocation per segment of a window row. Window - // columns 8s..8s+16 give the filtered columns 8s+4..8s+12. + // Horizontal pass, straight from the bands: one invocation per segment of + // a row of the neighbourhood, whose columns 8s..8s+16 give the filtered + // columns 8s+4..8s+12. Each coordinate is mirrored about the level's size; + // the low bit of a mirrored coordinate says which band the sample comes + // from, LL, HL (odd column), LH (odd row) or HH, and mirroring keeps that + // bit, so a row's even columns come from one band and its odd ones from + // another. It keeps it up to the level's width less two past an edge, and + // the samples that are stored read no further than four: the levels must + // be at least six wide, and the minimum image size makes them eight + // (gpu/layout.rs tests it). Loading the neighbourhood into workgroup memory + // first made the transform take a quarter longer at 4K (decision D68). if (tid < 160u) { let row = tid / 4u; let segment = tid % 4u; + let y = mirror(tile_y + i32(row) - APRON, out_height); + let row_bands = u32(y & 1) << 1u; + let band_row = u32(y >> 1u) * params.band_width; + let even = comp.band_offset[row_bands] + band_row; + let odd = comp.band_offset[row_bands | 1u] + band_row; + let x0 = tile_x + i32(8u * segment) - APRON; var v: array; - let base = row * WINDOW + 8u * segment; - v[0] = window[base + 0u]; - v[1] = window[base + 1u]; - v[2] = window[base + 2u]; - v[3] = window[base + 3u]; - v[4] = window[base + 4u]; - v[5] = window[base + 5u]; - v[6] = window[base + 6u]; - v[7] = window[base + 7u]; - v[8] = window[base + 8u]; - v[9] = window[base + 9u]; - v[10] = window[base + 10u]; - v[11] = window[base + 11u]; - v[12] = window[base + 12u]; - v[13] = window[base + 13u]; - v[14] = window[base + 14u]; - v[15] = window[base + 15u]; + v[0] = pyramid[even + u32(mirror(x0 + 0, out_width) >> 1u)]; + v[1] = pyramid[odd + u32(mirror(x0 + 1, out_width) >> 1u)]; + v[2] = pyramid[even + u32(mirror(x0 + 2, out_width) >> 1u)]; + v[3] = pyramid[odd + u32(mirror(x0 + 3, out_width) >> 1u)]; + v[4] = pyramid[even + u32(mirror(x0 + 4, out_width) >> 1u)]; + v[5] = pyramid[odd + u32(mirror(x0 + 5, out_width) >> 1u)]; + v[6] = pyramid[even + u32(mirror(x0 + 6, out_width) >> 1u)]; + v[7] = pyramid[odd + u32(mirror(x0 + 7, out_width) >> 1u)]; + v[8] = pyramid[even + u32(mirror(x0 + 8, out_width) >> 1u)]; + v[9] = pyramid[odd + u32(mirror(x0 + 9, out_width) >> 1u)]; + v[10] = pyramid[even + u32(mirror(x0 + 10, out_width) >> 1u)]; + v[11] = pyramid[odd + u32(mirror(x0 + 11, out_width) >> 1u)]; + v[12] = pyramid[even + u32(mirror(x0 + 12, out_width) >> 1u)]; + v[13] = pyramid[odd + u32(mirror(x0 + 13, out_width) >> 1u)]; + v[14] = pyramid[even + u32(mirror(x0 + 14, out_width) >> 1u)]; + v[15] = pyramid[odd + u32(mirror(x0 + 15, out_width) >> 1u)]; lift(&v); let out = row * TILE + 8u * segment; horizontal[out + 0u] = v[4]; @@ -239,16 +241,15 @@ fn main(@builtin(local_invocation_index) tid: u32, @builtin(workgroup_id) wg: ve lift(&v); if (is_final) { - // The window array is free now: the horizontal pass is done. let at = 8u * segment * TILE + column; - window[at + 0u * TILE] = to_unorm8(v[4] + DC_SHIFT); - window[at + 1u * TILE] = to_unorm8(v[5] + DC_SHIFT); - window[at + 2u * TILE] = to_unorm8(v[6] + DC_SHIFT); - window[at + 3u * TILE] = to_unorm8(v[7] + DC_SHIFT); - window[at + 4u * TILE] = to_unorm8(v[8] + DC_SHIFT); - window[at + 5u * TILE] = to_unorm8(v[9] + DC_SHIFT); - window[at + 6u * TILE] = to_unorm8(v[10] + DC_SHIFT); - window[at + 7u * TILE] = to_unorm8(v[11] + DC_SHIFT); + levels[at + 0u * TILE] = to_unorm8(v[4] + DC_SHIFT); + levels[at + 1u * TILE] = to_unorm8(v[5] + DC_SHIFT); + levels[at + 2u * TILE] = to_unorm8(v[6] + DC_SHIFT); + levels[at + 3u * TILE] = to_unorm8(v[7] + DC_SHIFT); + levels[at + 4u * TILE] = to_unorm8(v[8] + DC_SHIFT); + levels[at + 5u * TILE] = to_unorm8(v[9] + DC_SHIFT); + levels[at + 6u * TILE] = to_unorm8(v[10] + DC_SHIFT); + levels[at + 7u * TILE] = to_unorm8(v[11] + DC_SHIFT); } else { let x = tile_x + i32(column); let y = tile_y + i32(8u * segment); @@ -264,7 +265,7 @@ fn main(@builtin(local_invocation_index) tid: u32, @builtin(workgroup_id) wg: ve } // At the last level every invocation stores four samples of a row of the - // tile, from the window array, in the file behind this one. + // tile, from the levels array, in the file behind this one. // // The barrier is outside the branch on purpose: every level waits, though // only the last needs it. Inside `if (is_final)`, uniform as that is, it diff --git a/crates/emberwave-core/src/gpu/shaders/idwt_packed.wgsl b/crates/emberwave-core/src/gpu/shaders/idwt_packed.wgsl index f4d6f81..503d3e6 100644 --- a/crates/emberwave-core/src/gpu/shaders/idwt_packed.wgsl +++ b/crates/emberwave-core/src/gpu/shaders/idwt_packed.wgsl @@ -9,12 +9,12 @@ @group(0) @binding(1) var packed: array; -// The 8-bit level of tile sample `at` collected in the window array, or 0 for +// The 8-bit level of tile sample `at` collected in the levels array, or 0 for // a padding sample past the right edge of the plane. `select` evaluates both -// arms, so the window is read for padding samples too: the vertical pass has +// arms, so the array is read for padding samples too: the vertical pass has // to write the whole tile, not only the part inside the plane. fn level_at(at: u32, x: u32, width: u32) -> u32 { - return select(0u, u32(window[at]), x < width); + return select(0u, u32(levels[at]), x < width); } // Every invocation writes one word of the tile: four samples of a row. Rows diff --git a/crates/emberwave-core/src/gpu/shaders/idwt_planes.wgsl b/crates/emberwave-core/src/gpu/shaders/idwt_planes.wgsl index 4ac7ace..cc6a73f 100644 --- a/crates/emberwave-core/src/gpu/shaders/idwt_planes.wgsl +++ b/crates/emberwave-core/src/gpu/shaders/idwt_planes.wgsl @@ -40,16 +40,16 @@ fn store_plane(comp: CompParams, component: u32, tile_x: u32, tile_y: u32, tid: let at = row * TILE + column; if (y < comp.plane_height) { if (x < comp.plane_width) { - store_level(component, vec2(x, y), window[at]); + store_level(component, vec2(x, y), levels[at]); } if (x + 1u < comp.plane_width) { - store_level(component, vec2(x + 1u, y), window[at + 1u]); + store_level(component, vec2(x + 1u, y), levels[at + 1u]); } if (x + 2u < comp.plane_width) { - store_level(component, vec2(x + 2u, y), window[at + 2u]); + store_level(component, vec2(x + 2u, y), levels[at + 2u]); } if (x + 3u < comp.plane_width) { - store_level(component, vec2(x + 3u, y), window[at + 3u]); + store_level(component, vec2(x + 3u, y), levels[at + 3u]); } } } diff --git a/crates/emberwave-core/tests/shaders.rs b/crates/emberwave-core/tests/shaders.rs index 30d6bd6..930c1d5 100644 --- a/crates/emberwave-core/tests/shaders.rs +++ b/crates/emberwave-core/tests/shaders.rs @@ -86,7 +86,8 @@ fn the_workgroup_variables_are_the_ones_checked_for_writes_before_reads() { ("sign_counts", 512), ("block_sign_totals", 64), ]; - let idwt: &[Variable] = &[("window", 6400), ("horizontal", 5120)]; + let idwt: &[Variable] = &[("horizontal", 5120), ("levels", 4096)]; + let fdwt: &[Variable] = &[("horizontal", 5120)]; // quantise.wgsl's, in front of equant.wgsl's and pack.wgsl's. let quantise: &[Variable] = &[ ("subblock_largest", 512), @@ -120,7 +121,7 @@ fn the_workgroup_variables_are_the_ones_checked_for_writes_before_reads() { ("yuv2rgb", Vec::new()), ("gather", Vec::new()), ("gather_rgb", Vec::new()), - ("fdwt", idwt.to_vec()), + ("fdwt", fdwt.to_vec()), ("equant", [quantise, equant].concat()), ("rate", Vec::new()), ("scan", vec![("sums", 1024)]), @@ -203,7 +204,7 @@ fn the_inverse_transform_that_stores_planes_differs_only_in_its_store() { .collect(); assert_eq!( workgroup, - [("window".to_owned(), 6400), ("horizontal".to_owned(), 5120)] + [("horizontal".to_owned(), 5120), ("levels".to_owned(), 4096)] ); let store = naga::StorageAccess::STORE; assert_eq!(