Skip to content

Frame generation: CPU reference of the networks' building blocks - #153

Merged
phantoslayer21 merged 2 commits into
mainfrom
fg/cpu-reference
Nov 14, 2026
Merged

phantoslayer21 merged 2 commits into
mainfrom
fg/cpu-reference

Conversation

@phantoslayer21

Copy link
Copy Markdown
Owner

A NumPy CPU reference for the building blocks of frame generation's two convolution networks (lane h). It follows #148, which added the FG weight layout and converter preset.

clean/fg/reference/ops.py (float64 arithmetic with only the float16 roundings, like the other oracles here)

  • Float16 convolution (3×3 and 1×1, stride 1, same size), with bias, one float16 rounding per output, then optional ReLU.
  • 2× pooling over 2×2 blocks, a bilinear 2× upscale in the align-corners form, element-wise add and channel concatenation.
  • net1's per-pixel input features: luma with fixed RGB weights, the warp error (mean absolute RGB difference, clamped to [0, 1]) and a mask with its complement. Also the 2×2 fold of ten features into one cell of 40 values padded to 48, and its inverse.
  • weight_shape(layer) gives the canonical weight shape [out, k, k, in] for every layer of clean/fg/layout.py.

Behaviour the spec side has not delivered is never guessed:

  • These are required arguments with no default: the convolution border rule (pad: zero, replicate or mirror), ReLU after a net2 layer (relu_after), the pooling kind (max or avg), net1's feature channel order, and the fold order inside a cell.
  • These raise OpenItem: float16 accumulation in the matrix unit's order (acc="f16"), net1 blob 2 (1×1 or 2×2), and both networks end to end (net1_forward, net2_forward).
  • clean/fg/FORMATS.md lists every OPEN item.

Refs #96

Tested

  • python -m unittest clean.fg.tests.test_fg_reference clean.fg.tests.test_fg_layout: 18 tests pass locally.
    • The convolution matches a separately written plain-loop version bit for bit, for both kernel sizes and all three border rules, with and without ReLU.
    • Pooling (max and avg) and the upscale are checked against hand formulas: corners copied exactly, a linear ramp kept linear, a single-row input.
    • Features and fold: round trips for three fold orders, and one cell value checked by position.
    • Every OPEN choice is checked to have no default or to raise.
    • Every layer of both layout tables runs as one convolution.
  • Line-ending check passes. Public gates pass, with no hits in clean/fg.

Not tested

  • No real weights and no comparison with the original's output: the axis order inside each blob, the network wiring and the map file are not delivered yet.
  • Float16 accumulation, which is OPEN, so the reference uses float64 sums.

clean/fg/reference/ops.py (NumPy, float64 with the float16 roundings):
float16 convolution (3x3, 1x1) with bias and ReLU, 2x pooling, bilinear 2x
upscale in the align-corners form, element-wise combines, net1's input
features (luma, warp error, mask pair) and the 2x2 fold into 48-value cells.
Behaviour not yet delivered is a required argument (border rule, ReLU per
layer, pooling kind, channel and fold order) or raises OpenItem (float16
accumulation order, both networks' wiring). FORMATS.md lists them.

Refs #96

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 marked this pull request as ready for review November 14, 2026 07:59
Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21
phantoslayer21 merged commit 5607e8f into main Nov 14, 2026
9 checks passed
@phantoslayer21
phantoslayer21 deleted the fg/cpu-reference branch November 14, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant