Skip to content

feat: export sky and calibrated source-trajectory renders - #11

Open
heiwang1997 wants to merge 2 commits into
NVIDIA:mainfrom
heiwang1997:feat/render-sky
Open

feat: export sky and calibrated source-trajectory renders#11
heiwang1997 wants to merge 2 commits into
NVIDIA:mainfrom
heiwang1997:feat/render-sky

Conversation

@heiwang1997

@heiwang1997 heiwang1997 commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

This addresses the sky-rendering request in issue #5's follow-up comment and makes the reference renderer honor the source camera calibration and trajectory.

  • derive a renderable cubemap from the released decoder's canonical RGB and SKY semantics (the public checkpoints omit the learned sky.* head)
  • retain the observation mask, infill unseen directions from pruned ROAD Gaussians, and merge chunk cubemaps with mask-aware weights
  • export a versioned <stem>.sky.npz sidecar plus <stem>.sky.png atlas next to every PLY, including affine-to-sensor metadata
  • render <stem>.render.png with the original NCore F-theta calibration, exact per-pixel world rays, exposure START/END poses, and rolling shutter
  • add --merge --render-video to stream every original source exposure to an H.264 <stem>.render.mp4 at the source cadence without materializing the full ray sequence
  • pin the public gsplat F-theta convergence fix and explicit 3DGUT world-ray path at a337c3dc
  • fail explicitly for unsupported camera/distortion models, truncated --max-chunks, multi-sequence video input, missing libx264, or encoder failures

Standard 3DGS PLY has no environment-map or per-camera ISP fields, so existing PLY viewers continue to show the Gaussian foreground only. The sidecar preserves that non-PLY state for a sidecar-aware renderer, while the PNG/MP4 outputs provide immediately testable source-view renders.

Validation

  • uv sync --frozen --extra render
  • .venv/bin/pytest -q542 passed
  • .venv/bin/ruff check --no-cache .
  • uv lock --check
  • git diff --check
  • live CUDA exact-ray F-theta/3DGUT smoke with the pinned gsplat commit
  • full public example clip, two chunks plus merge and all 599 source frames:
INSTANT_NUREC_RUN_ID=calibrated-sky-final .venv/bin/python run_inference.py \
  --model pa-front \
  --ncore-path demo_clip/clips/000da9de-0ee5-465a-9a2d-e7e91d3016bb/pai_000da9de-0ee5-465a-9a2d-e7e91d3016bb.json \
  --output-dir /tmp/instant-nurec-calibrated-final \
  --max-chunks 2 --merge --render-preview --render-video

The example completed successfully with 1,918,461 merged Gaussians. The MP4 is H.264 High/yuv420p, 784x448, exactly 599 frames, 29.9994 fps, and 19.967 s; a full decode found no corrupt, blank, or duplicate frames. Exact source/render comparisons at frames 0/299/598 measured static-feature displacement medians of 0.782 / 0.943 / 0.995 px (p90 1.630 / 2.043 / 2.487 px), with stable horizon and edge-to-edge geometry.

Renderer scope

The optional outputs are calibrated source-trajectory checks, not arbitrary novel-view renders. This public path currently supports NCore F-theta cameras without external windshield distortion and rejects other models rather than silently approximating them. Dynamic people/vehicles in the released public inference path remain subject to the model's current empty dynamic-layer output; that affects scene content, not camera calibration.

@heiwang1997 heiwang1997 changed the title feat: export and render observation-derived sky feat: export sky and calibrated source-trajectory renders Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant