Skip to content

Add experimental Steam Frame port and eye-tracked streaming - #27

Closed
coal-rock wants to merge 1 commit into
JMS1717:mainfrom
coal-rock:feat/steam-frame-eye-tracked-streaming
Closed

coal-rock wants to merge 1 commit into
JMS1717:mainfrom
coal-rock:feat/steam-frame-eye-tracked-streaming

Conversation

@coal-rock

Copy link
Copy Markdown

Adds an experimental Steam Frame Android/OpenXR client in Lepton and gaze-driven PyroWave streaming from the existing Windows gaming PC server. Fresh installs use Frame defaults: runtime-recommended stream size, 3240×3240 PC render, prefer 72 Hz, 500 Mbps, CDF 5/3 compute, and no Quest clock/colour overrides. Existing sessions retain their values; Frame profiles come first in the dashboard and headset menu.

The opt-in eye-tracked profile negotiates an actual OpenXR gaze source, projects it into both eye frusta, publishes both centers coherently, and reconstructs each selected frame with its own inverse layout and filter regions. Invalid or stale gaze recentres. Optional clock conversion is requested only when advertised; the fallback clock is explicitly estimated and does not prove latency. The protocol version changes to 20.13.0-frame.pyro.1.

Validation:

  • Android ARM64 release APK and native decoder libraries built locally with the pinned Rust/NDK/loader; development signing.
  • Windows GNU target Rust checks pass for server core and client.
  • 165 Rust core/graphics/settings/packet/device tests, 12 dashboard tests, and one standalone clock test pass; one existing visual preview is ignored.
  • Python: 189 passed, 107 subtests. C++ concurrent gaze publication test passes.
  • Software GLES: 490 fixed and 7,840 moving-layout cases pass; worst moving UV error 1.11e-7.
  • All 686 reconstructed recipe files match the build source tree byte for byte.

At 2176×2176 expanded per eye, the gaze profile encodes 1632×1632, 43.75% fewer pixels. This is geometry arithmetic, not measured Frame FPS or quality. Eye-tracked mode remains opt-in.

Still unverified: native Windows C++/D3D11 server package, Steam Frame launch and Vulkan/AHardwareBuffer import, live gaze/controller/audio behavior, fresh FPS, worn quality, sustained play, thermals and optical latency. The local full-driver test stopped at the separate Linux FFmpeg development bundle. Frame hardware is available later, so this PR stays draft and no release is published. docs/STEAM-FRAME.md contains setup, timing limitations and the first device comparison; HANDOFF and PLAN are current.

@coal-rock coal-rock closed this Oct 17, 2026
@coal-rock
coal-rock deleted the feat/steam-frame-eye-tracked-streaming branch October 17, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants