Skip to content

Add extra signal support to 2DGS rasterization - #999

Open
yunusstalha wants to merge 1 commit into
nerfstudio-project:mainfrom
yunusstalha:feat/2dgs-extra-signals
Open

Add extra signal support to 2DGS rasterization #999
yunusstalha wants to merge 1 commit into
nerfstudio-project:mainfrom
yunusstalha:feat/2dgs-extra-signals

Conversation

@yunusstalha

@yunusstalha yunusstalha commented Jun 29, 2026

Copy link
Copy Markdown

#998

This PR adds auxiliary per-Gaussian signal rendering support to rasterization_2dgs(), mirroring the existing extra_signals API in the 3DGS rasterizer.

The motivation is to allow 2DGS users to render learned features, semantic logits, supervision targets, or other auxiliary Gaussian-attached signals through the same visibility ordering and alpha compositing path as the normal 2DGS render, while keeping those auxiliary outputs separate from RGB/depth outputs.

Previously, users could only approximate this by packing extra channels into colors, which conflates appearance with auxiliary outputs and makes it harder to preserve a clean render contract. This PR adds a dedicated path instead.

Main Changes

  • Adds extra_signals and extra_signals_sh_degree arguments to rasterization_2dgs().

  • Supports post-activation auxiliary signal layouts:

    • [..., N, E]
    • [..., C, N, E]
  • Supports SH-evaluated auxiliary signals with layout:

    • [N, K, E]
  • Renders auxiliary signals through the same 2DGS alpha compositing path as colors.

  • Stores the auxiliary render separately in: 'meta["render_extra_signals"]

  • Keeps RGB/depth outputs separate from auxiliary outputs, including for RGB+D, D, and ED render modes.

  • Adds explicit metadata describing the auxiliary signal receipt/round-trip contract:

meta["extra_signal_layout"]
meta["extra_signal_channels"]
meta["extra_signals_sh_degree"]
meta["extra_signal_source"]
meta["extra_signal_compositing"]
  • Adds channel chunking support for large combined color/depth/signal feature tensors.

Testing
This PR adds focused 2DGS tests covering:

  • Numeric compatibility when extra_signals=None.
  • Metadata and shape checks for rendered auxiliary signals.
  • Per-Gaussian and per-camera signal layouts.
  • Packed rasterization mode.
  • RGB, RGB+D, D, and ED render modes.
  • SH-based auxiliary signals.
  • Channel chunking.
  • Gradient flow through auxiliary signal values and Gaussian parameters.
  • A tiny deterministic stats fixture to help detect future changes to compositing or visibility semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant