perf(video): pipeline hardware encode/decode and harden display recovery - #192
Merged
mrjeeves merged 86 commits intoJul 19, 2026
Merged
Conversation
…ing, refinement, CPU cuts The two field failures this buries: - GPU/CPU load froze (and could abort) the stream: the MF async pump parked the capture thread up to 1 s per frame and, worse, discarded drained access units and NeedInput events under backlog — starving the encoder for good. - Window switches smeared and un-blurred in slow motion: dropped reference units + a static-skip gate that froze the picture at its burst-crushed quality with nothing ever re-encoding it. What changed: - Media Foundation pump rewritten lossless: EncodeOutcome seam returns every drained unit in order + a consumed flag; NeedInput events bank as input credits; worst-case per-call stall drops 1000 ms -> 50 ms grace. VideoToolbox gets the same drain shape (its pending-unit path skipped encoding the input frame entirely); FFmpeg hwenc drains per-unit instead of concatenating. - HealingEncoder: encoder errors rebuild through the ladder (bounded: 3 per 120 s, 2 s spacing) instead of zombifying the route into a GDI screenshot loop against a dead encoder; permanent failure ends the route with an in-band status. - Post-quiesce refinement: after the convergence IDR, two spaced non-IDR re-encodes sharpen the retained frame once motion stops. - win_capture CPU pass: persistent cursor-free desktop buffer (deletes a full-frame clone + zeroed alloc per damage frame), ReleaseFrame before Map, BGRA->RGBA swizzle moved to the opt-level-3 pixels crate. - NV12 fed directly to the MFT (scale_rgba_to_nv12; encoder-side interleave deleted) + a contiguous no-scale fast path for both 4:2:0 layouts. - os_perf: 1 ms timer-resolution guard per stream; media threads get ABOVE_NORMAL + EcoQoS opt-out + P-core CPU-set preference on hybrid CPUs. - Adapter pin: ALLMYSTUFF_VIDEO_ENCODE_ADAPTER=intel|nvidia|amd|<index> via MFTEnum2 + MFT_ENUM_ADAPTER_LUID (encode on the idle iGPU while the dGPU is loaded); MF_MT_DEFAULT_STRIDE declared for Intel-MFT correctness. - Remote control: ScreenMap re-queries on a 2 s TTL (mouse no longer lands wrong after a resolution/DPI change); allocation-free fleet-membership probe on the per-event gate. - Hardening: short capture buffers are stream errors, not panics (release aborts on panic); duplicate Tunes no longer restart the capture. - Ignored-by-default bench suite (pipeline decomposition, MF latency histogram + units conservation, micro-benches, sleep-granularity probe). Measured on the dev box (1440p, NVENC, release): capture-thread busy 21.2 -> 17.7 ms/frame, live-screen arrival 34.6 -> 40.8 fps, MF call max 12.3 -> 9.3 ms, units conservation lossless by construction + tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…riorities The consolidated writeup of the 2026-07 encoder pass for review: symptom -> root-cause -> fix map, per-stage profiling before/after, syntheses of the QuickSync / AV1-HEVC / multi-encoder / stream-upscaling / Apple-platform research, the Game Mode design, and the prioritized plan (thread split -> zero-copy -> transport -> codec). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…T low-latency WAN gaming is becoming first-class; this is the preset half that ships on today's pipeline: - ALLMYSTUFF_GAME_MODE=1: off-LAN automatic fps floor rises 30 -> 60 (the deliberate open-loop trade until pacer/BWE land — documented as such), and the encoder burst posture tightens from 2x-peak/1s-VBV to 1.5x/0.5s so a scene change queues for half the time; the post-quiesce refinement passes repair what the tighter budget costs a transition. Shared burst_bounds feeds both the MF (Windows) and FFmpeg vendor encoders. - h264_qsv gains the low-latency screen posture: scenario=displayremoting, async_depth=1 (the default queues four frames), b_strategy=0. - macOS: media threads now set QOS_CLASS_USER_INTERACTIVE — the only P/E-core steering that exists on Apple silicon; an untagged thread can sit on E-cores while P-cores idle. - VideoToolbox: request the low-latency rate controller (macOS 11.3+) in the encoder spec — keyed by name, with a create-retry without it so older systems keep working. Strict one-in-one-out + faster rate adaptation. macOS/hwenc arms are compile-checked only on Windows here — needs one Mac/Linux build pass to verify (flagged in docs/ENCODER-PASS-2026-07.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…te threads Roadmap #1. The pump is now a two-stage pipeline: the capture side drains the backend, orients, and converts to the encode side's advertised 4:2:0 layout while a scoped encode thread runs the codec — conversion of frame N overlaps the encode of frame N-1. The stage channel is shallow and drained to freshest, so latency never accumulates; the encode side owns the quiet-spell logic (quiesce IDR + refinement) and publishes its input needs (layout + fit edge) through atomics, so a healing rebuild that changes either costs at most one dropped, layout-tagged frame — never wrong-order chroma. H264Stream::encode splits into convert + encode_prepared; the single-threaded paths (oneshot fallback, tests) keep the fused entry. Honest numbers: stage costs are unchanged (convert 8.7 ms / encode 10.1 ms at 1440p, within desktop noise of the pre-split run) and semantics are covered by the existing pump-behavior tests (153/153). End-to-end synthetic throughput is ~58 packets/s — still the serial rate, not the pipelined ~99/s ceiling: with the stages concurrent, each frame's ~35 MB of fresh large allocations across three threads (source copy, convert output, MF input sample) serializes on the allocator/page-zeroing. The split is the structure; buffer reuse is the unlock and lands next (into-variant converters + recycled staging buffers + a reused MF input buffer). A new bench_pump_pipelined_throughput measures exactly this number. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…by interleaved A/B The pipelined pump now recycles its multi-megabyte buffers instead of allocating fresh per frame: the pixels converters gain _into variants writing into reused buffers, spent convert outputs ride a return lane from the encode side back to the capture side (including frames displaced by freshest-wins draining, which now also count into the dropped stat so the dial-in line distinguishes "encode behind" from downstream loss), and win_capture grows a reclaim lane so the per-frame copy-out lands in warm pages. Measured honestly: an interleaved A/B (4 runs per arm, alternating with rebuilds, on the live dev box) reads 44.5 pkt/s (split only) vs 45.4 (reuse) at synthetic 1440p — throughput-neutral under ambient load; earlier apparent deltas were between-session desktop drift. Kept anyway on the stability argument: the lanes cost ~a try_send per frame and pin the stream's working set to a stable, already-touched page set instead of racing the OS allocator, which matters exactly when a game is eating memory next to the stream. An MF input-buffer ring was prototyped and DROPPED: it rests on an unproven MFT buffer-lifetime assumption and bought nothing measurable — unproven assumptions for free are the wrong trade. The real conclusion the numbers force: allocation churn was not the limiter — the copies themselves are. The GPU zero-copy path (roadmap mrjeeves#10) is promoted from "4K enabler" to the primary-path throughput unlock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ion, p95 tails Three reliability/smoothness levers for the general-use path: - Media-pipe writes are bounded (5 s): a daemon that stops READING its socket (wedged, not dead) never errors the write — it just never completes, silently stalling every audio and video send behind the shared writer mutex forever. The timeout converts that into the same drop-and-reconnect a write error already gets, on both the JSON media pipe and the binary track pipe (handshake included). This was the last known silent-freeze vector in the node. - DXGI re-promotion: a route that fell back to per-frame GDI screenshots (fullscreen-exclusive app, UAC desktop, driver reset) used to stay degraded for its whole life. The fallback now runs in 30 s spells and re-attempts a real duplication session between them, with a fresh monitor handle each time (mode changes move things around). - The per-stream stats line grows p95 columns for scale and encode: smoothness lives in the tail, not the average — a clean 8 ms mean with a 40 ms p95 IS the stutter the viewer feels, and the dial-in line now shows it directly. (Drain-displaced frames were already folded into `dropped` last commit, so "encode side behind" is visible too.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fed MFT, proven end-to-end New gpu_pipeline module: a GpuConvert stage that owns a video-capable D3D11 device (multithread-protected — MF shares it across its workers), a D3D11 VideoProcessor doing BGRA→NV12 colour conversion (and scaling) entirely on the GPU, a 4-deep NV12 output-texture ring, and the IMFDXGIDeviceManager the encoder MFT opens against. Media Foundation side: open_with_manager() passes that manager to the MFT via MFT_MESSAGE_SET_D3D_MANAGER, and encode_texture() feeds an NV12 *texture* straight in through MFCreateDXGISurfaceBuffer — the encoder reads it in place. CPU pixel touches per frame on this lane: zero. The interleaved A/B on the pipelined pump showed the copies themselves are the CPU lane's limiter (staging Map + swizzle ~15 MB, copy-out, RGBA→NV12 ~5.5 MB, MFT ingest), and this deletes all of them. Ordering find, caught by the test: an async MFT rejects MFT_MESSAGE_SET_D3D_MANAGER with MF_E_TRANSFORM_ASYNC_LOCKED until MF_TRANSFORM_ASYNC_UNLOCK has been set, so the manager message goes after the unlock block, before stream negotiation. Colour: full-range RGB in, BT.709 limited out (the HD convention; the CPU lane's BT.601 stays as-is — lanes never mix within one stream). Lands proven but unwired: gpu_lane_end_to_end drives synthetic BGRA textures through convert → MFT (device-manager path) → Annex-B → openh264 decode and asserts dimensions + per-region luma, so the novel COM plumbing is validated on real hardware before the integration slice (duplication on the shared device, GPU cursor composite, Staged::Gpu lane, ladder wiring with CPU fallback) switches the live path over. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…places the CPU pipeline where it can Windows H.264 screen routes now try a fully-GPU path before building any CPU encoder: duplication, cursor composite, and BGRA→NV12 colour conversion all happen on one D3D11 device (win_capture::start_gpu), and the encoder MFT — opened against the lane's device manager, enumerated on the lane's adapter LUID — reads the NV12 ring textures in place (video::run_gpu_lane). Per frame the CPU touches at most a cursor-sized save-under patch; the deleted work is the staging Map + full-frame swizzle, the copy-out, the RGBA→NV12 convert, and the MFT's system-memory ingest. Measured (bench_gpu_lane_cycle, 1440p, release, 240 frames, 240 units — lossless): convert 0.036 ms/frame of blt-queue cost vs the CPU lane's 5–8 ms scale column; encode_texture 6.25 ms feed+drain. Stability first — the lane is pure addition and fails soft everywhere: - Any setup failure (rotated output, no VideoProcessor, no MFT on the display adapter, manager refused) logs its reason and the untouched CPU pipeline takes over; the CPU encoder isn't even built until then, so a GPU-lane route holds ONE hardware encode session, not two. - Live errors reopen the encoder under the shared RebuildPolicy bounds; persistent failure falls back mid-route. A watchdog catches the silent-brick MFT (consumes frames, never emits) since the ladder's synthetic send test can't run on a texture lane. - Mode changes and auto-adapt edge moves restart the lane with fresh geometry (restart-budgeted); the operator's adapter pin disables the lane entirely — pinning means cross-adapter encode, which is the CPU lane's job. Kill-switch: ALLMYSTUFF_GPU_LANE=0. The lane mirrors the CPU pump's cadence machinery: freshest-wins drains with slot reclaim, fps pacing, adaptive IDR + viewer refresh, quiesce IDR + refinement passes re-encoding the retained last texture (its ring slot stays checked out so the capture side can't blt over it), and first-frame wake pressure. Damage-driven duplication replaces the static-skip byte compare outright. Cursor: Desktop Duplication never draws the pointer, so its rect — and only its rect — round-trips through a small staging texture: pristine save-under out, CPU blend (the existing three-shape compositor, parameterized over BGRA order), patched rect back, blt, restore. The clean desktop texture stays cursor-free for pointer-only re-emits, which re-blt without recapture (15 ms rate limit, as before). NV12 ring slots are now explicitly checked out and released (consumer → capture return lane) instead of blind rotation; release ordering keeps the async MFT ≥3 slot-reuses behind any texture it could still read. Shared plumbing extracted, no behaviour change: duplicate_output, fetch_cursor_shape, packetize_units, effective_h264_edge, fit_within_even (pub(crate)); StreamStats grows set_mode so the stats line tracks an MJPEG-floor fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bps, 150 frames, full-frame shift) so A/B/C encode columns compare like-for-like
MediaFoundationH264::set_bitrate re-aims the rate controller mid-stream via ICodecAPI (mean, plus peak/VBV riding along best-effort): no reopen, no forced IDR. The knob receiver-driven adaptation needs — today a bitrate move costs a codec rebuild whose first unit is an IDR, a burst on the wire at the exact moment the link is struggling. Pilot findings, pinned by hardware_bitrate_reconfigures_in_place (720p60 on the NVIDIA MFT): - Downshift 12 -> 2 Mbps: accepted instantly, the next one-second window lands on the new target within 8%, and the stream decodes cleanly across the change — no internal reset, reference chain intact. This is the direction congestion control needs NOW-semantics for. - Upshift is only partially honored: the mean stores verbatim (reads back exactly), but the driver self-clamps peak/VBV to an internal ceiling and tracked ~5.7 Mbps of a 2 -> 12 ask. Operational rule set by the pilot: in-place for downshifts and modest upshifts; large upshifts keep the rebuild path, whose IDR a healthy link absorbs. (The direct-NVENC rung's reconfigure moves all three for real.) Consumer lands with BWE; the test carries the proof until then. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tive The game-mode encoder rung: NVIDIA's Video Codec SDK driven straight, fed the same NV12 ring textures the GPU lane's VideoProcessor produces. The MFT wraps this hardware but withholds the WAN-gaming levers; this rung has them: - Intra-refresh/GDR: infinite GOP, intra data riding refresh waves (~2 s period, ~15% duty) instead of IDR walls — the encoder-side answer to the transport's unpaced bursts: there is no keyframe wall left to spray. Proven by nvenc_intra_refresh_replaces_idr_walls: 90 frames, ZERO IDRs after the entry frame, decodes cleanly throughout. Forced IDRs (viewer joins, quiesce rescue) still cut through and restart the wave. - Guaranteed in-place reconfigure: mean/peak/VBV all move via NvEncReconfigureEncoder, no reset, no IDR (the MF rung only honors the mean — see the pilot commit). - Reference-picture invalidation is a per-frame call away (nvEncInvalidateRefFrames is in the loaded table) once per-frame loss feedback exists on the transport. No build-time dependency: the C function table loads from nvEncodeAPI64.dll at runtime, so every failure — no NVIDIA driver, old driver (< API 12.0), no free session — is soft and the lane keeps the MF rung. The FFI subset is hand-transcribed from NVIDIA's MIT-licensed nvEncodeAPI.h (ffnvcodec n12.0.16.0); every exact struct is pinned by a compile-time size assert, and the two codec unions are deliberately oversized + zeroed (pre-union offsets exact, driver writes land inside our larger allocation, both sides read zeros past what we set). Wiring: opt-in via ALLMYSTUFF_NVENC=1 until soaked. When on, the GPU lane opens the SDK session first (game mode => GDR shape, and the periodic-IDR cadence stands down — refresh waves ARE the convergence mechanism; stable arc keeps the familiar IDR cadence); any session failure or mid-stream error steps down to the MF rung in-lane. Sync mode (one frame in flight) — the lane's encode thread exists to wait on exactly this; async events can come later if a measurement asks. nvenc_sdk_end_to_end proves the full chain on hardware: VideoProcessor NV12 -> registered texture -> encode -> Annex-B -> openh264 decode with luma asserts, plus a mid-stream set_bitrate accepted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…om this Windows dev loop; clippy them on every push (soft while they stabilize, Windows required)
… changes Per Chris's constraint: the mesh stays a dumb fast pipe; the stream owner shapes the stream. When ALLMYSTUFF_PACED_SLICES=1 (opt-in until soaked): - Encoders emit byte-capped slices (~24 KB): NVENC sliceMode=1, the MF codec API's SliceControl (best-effort, vendor-dependent), openh264 max_slice_len. A keyframe leaves the encoder as several independently-decodable slices instead of one wall. - The mesh forwarder — already the route's single ordered video sender — splits each access unit at slice-NAL boundaries (split_annexb_paced: cuts only before slice NALs, SPS/PPS/SEI glued to the slice that follows, ranges partition the unit byte-exactly) and writes each chunk as its own track send with ~1.5 ms gaps, 10 ms total cap. On the wire, one 200-packet keyframe train becomes a handful of ~20-packet bursts a shallow bottleneck queue absorbs — libwebrtc-style pacing, implemented entirely in AllMyStuff. - Non-final chunks carry duration_us = 0, so every chunk shares one RTP timestamp. Verified against the daemon's own H264AuAssembler: it emits per marker and its contiguity anchor spans consecutive writes, so each chunk arrives as its own sample with no daemon involvement. The sleeps also release the media pipe's writer between chunks, letting audio frames interleave instead of queueing behind a keyframe. A lost packet now stalls only its slice behind NACK, not the whole frame. The receive side needs no coalescer, and the tests pin why: paced_slice_chunks_decode_independently drives a real slice-capped bitstream chunk-by-chunk through openh264 and asserts every picture surfaces on ITS OWN frame's final chunk — the decoder completes pictures by macroblock accounting, not next-AU detection, so chunked arrival adds zero latency. (The test also documents a trap found on the way: pure noise makes openh264's rate control skip alternate frames outright — an encoder artifact, not a chunking one.) Wire-format note, disclosed by design: the RTP marker bit now means end-of-chunk rather than strictly end-of-AU on paced streams. The only consumers are the daemon's assembler (per-marker by construction) and our viewer (per-unit decode); a third-party RTP consumer pointed at these tracks would need to know. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nc rung's true encode-to-bits latency against the MF rung's pipelined call time
…, per-route game mode, verbose field log - GPU-lane MF rung: depth-2 texture retirement (ring 6) — the async MFT can still be reading frame N-1 when N is fed; releasing N-1's slot at N's consume let the capture blt tear it mid-read (the field's window-open tearing). A slot now returns only two consumed frames later. - GDR heals in ~0.5s (was 2s) + recovery-point SEI — in-game loss artifacts bounded to a blink. - Per-route game mode: Tune.game rides the session protocol (serde-default, old peers unaffected) viewer->host; the GUI's pill row leads with Mode - Balanced/Game and Res/FPS/Rate stay as expert overrides. ALLMYSTUFF_GAME_MODE=1 remains the node-wide override. - serve auto-writes ./allmystuff-serve.log (debug, module targets on, ALLMYSTUFF_CWD_LOG=0 to disable) and the GPU lane logs which physical adapter each monitor's lane landed on; NVENC logs the driver API rev. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… H.264 -> MJPEG morph A healing rebuild went through make_encoder, whose MJPEG floor could silently swap a struggling H.264 route onto chunked MJPEG mid-stream: a quality lurch the viewer never chose, and (Chris's field read) the chunk-loss artifacts present as tearing. Heals now rebuild the SAME negotiated mode or fail the route, which then restarts and renegotiates honestly. The MJPEG floor still applies at route start, where the viewer sees what it's getting from frame one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eeps MF) + viewer decode-thread media boost
… strings and comments back to clean ×·—≈ glyphs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ingle-frame VBV, pacer default-on, slice-mode fix Per-route game posture now reaches the pacer (VideoBridge::route_game): game runs 1ms gaps under a 6ms cap vs balanced 1.5/10 — the frame tail lands ~4ms sooner. The GPU lane's quiet path wakes at 50ms in game mode (quiesce IDR + refinement land the moment motion stops; balanced keeps 250ms). Game NVENC runs a single-frame VBV: no frame queues behind an oversized predecessor — posture A/B (interleaved, 1440p/30Mbps): avg/p95 unchanged ~8ms, worst-case tail 23.4->17.9ms r1, 22.9->21.3 r2, 150/150 units all runs. Slice pacing is default-on (=0 pins single-write) — and the bench caught a field-breaker before it shipped: real drivers reject byte-based slice mode outright, which would have killed the whole SDK rung at open. NVENC now uses universal count-based slicing (8 slices >=1080p, 4 below) and retries single-slice if a driver rejects even that: pacing config can cost cut points, never the rung. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… game control New InputAction::MouseMoveRel carries the viewer's Pointer Lock deltas raw (no normalization, no screen clamp — FPS camera control wants deltas, not positions); the host injects them relative. Old receivers ignore the variant via the existing catch-all, so nothing breaks cross-version. GUI: in fullscreen control a click captures the mouse, Esc (the platform gesture) or leaving fullscreen/control releases it; absolute-move throttling doesn't apply to the aim path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-state (v3 arc) The LAN fidelity posture: mode rides the tune chain by name (serde default, old peers untouched; the legacy game bool still steers hosts that predate it). Studio is LAN-gated host-side — asked off-LAN it degrades to Balanced with a log — and spends the pipe when it has one: 150 Mbps auto budget (250 Mbps ceiling, vs the 80 Mbps stability ceiling elsewhere), NVENC P5 + high-quality tuning, 1 s VBV, near-mean peak. tuned_bitrate() centralizes the posture-aware budget across all encoder rungs. GUI Mode pill cycles the three postures. 4:4:4 chroma and the lossless preset slot in behind the hardware-decode epic (mrjeeves#24) — openh264 cannot decode High 4:4:4, so fidelity's ceiling raises with the viewer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stream codec picker described transports by bare name (H.264 / H.264 native / MJPEG) — reworded to what the viewer actually chooses between now that the host picks its own best encoder (NVENC/MF/software, all H.264): 'H.264 - hardware decode', 'H.264 - software decode', 'MJPEG - compatibility', each with a hover hint. Studio mode gets a one-time confirmation on the transition in (150-250 Mbps, meant for LAN) with Cancel / Don't ask again / Use Studio, the ack persisted in localStorage so it never nags twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… user's Studio is no longer LAN-gated: the warning dialog is the guardrail, then the user owns the wire. It runs on any link when chosen, floors to 150 Mbps auto, and its Rate ceiling opens to 500 Mbps. Game's ceiling opens to 200 Mbps too — its single-frame VBV converts extra bits into constant-per-frame motion quality (crisp fast pans) at zero latency cost, since every frame stays one interval's size regardless of budget. Balanced keeps the 80 Mbps stability ceiling. tuned_bitrate() is now a clean posture match. Rate pill gains 60/100/200 Mbps options; the Studio dialog copy reflects that it always applies now (lower the Rate if the link stutters). Off-LAN studio logs the honored ask rather than downgrading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dio 150M interleaved
…F_SOAK_ONLY, 60fps-paced, 15s windows + percentile ladder + slowest-frames log First soak findings recorded in the report artifact: GPU clock state moves encode ~2.5x (8-10ms boosted vs ~23-26ms settled at idle duty cycle — the tight-loop benches only ever saw boost); all four postures soak-stable for 6 min with zero unit loss and flat post-settle windows; game 200M costs +2.3ms avg over game 30M cold for 6.7x the bits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he popout stage When the decoded stream is smaller than the display box, the plain path was the browser's bilinear stretch — soft edges, smeared text. A WebGL2 port of AMD's MIT-licensed FidelityFX-FSR pipeline (EASU edge-adaptive upsampling, then RCAS sharpening) now runs on an overlay canvas, engaged automatically at >1.1x upscale, killable from the new Scale pill (Auto/FSR/Off, persisted). RCAS is text-adaptive: high local-contrast neighborhoods (glyph strokes) sharpen toward full strength while smooth gradients ease off — crisper text without fried video. Pointer events never touch the overlay (the normalizer and pointer lock stay on the base canvas); WebGL trouble falls back to the plain path with one console warning. One GPU-GPU canvas upload per frame; the decode pipeline is untouched. This is also Reach mode's presentation layer: a deliberately low-res stream over a starved link gets rebuilt at the viewer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Studio's aspirational tier, made measurable: open_lossless_on_device opens a constQP-0 session with transform bypass (Hi444PP profile, P5 under the dedicated lossless tuning) — mathematically lossless relative to the lane's NV12. No rate control exists in this mode, so set_bitrate honestly reports false and bandwidth is whatever content entropy demands. Which is why one number would mislead: the new bench_nvenc_lossless_content brackets it with four deterministic content classes (static UI, scrolling text, panning video + occluder, noise) against lossy studio-150M as the reference column, and the soak and posture-bench matrices gain a studio-lossless row. First real-driver numbers at 1440p60: ~3.5 Mbps static (nearly all of it the two IDRs), 65-76 Mbps scrolling text / panning video, ~4.9 Gbps on noise (the ceiling — CABAC expands what it cannot predict) at 46 ms a frame. Realistic-content encode latency ~10.5-12.7 ms vs studio's 8-10. Measurement-only: nothing in the viewer decodes Hi444PP yet (that is the hardware-decode epic's gate). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lossless soak measured 7-9 Mbps where the comment predicted a stress ceiling: value(j, i+1) = value(j+7, i), so the per-byte rolling gradient is a smooth ~1.75 px/frame pan that quarter-pel motion search tracks almost perfectly. Rewrite the comment to what the data proved — that row is the sustained-motion floor and the latency probe; the content-classed bench owns the bandwidth brackets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
open_lossless_hevc_on_device: the same constQP-0 contract as the H.264 rung, but HEVC carries transquant bypass as core syntax — the driver autoselects Main profile (the smoke test verifies VPS/SPS/PPS + IDR NAL structure and signaled profile_idc 1), which is exactly what hardware decoders open. Exact FFI from the n12.0.16.0 header: NV_ENC_CONFIG_HEVC (1560 bytes, size-asserted; chromaFormatIDC and pixelBitDepthMinus8 are multi-bit fields inside the flags word, unlike H.264), HEVC codec GUID, autoselect profile GUID; the codec-config init and slice-retry paths branch per codec. Transcribing against the real header also caught the H.264 HIGH_444 GUID Data4 tail mis-remembered in the first pass — the driver had been silently autoselecting past it; now exact. Measured at 1440p (content bench, soak row added): HEVC lossless runs FASTER than H.264 lossless — 7.3-8.6 ms avg vs 10.5-12.2, level with lossy studio-150M — at ~20% more bits on desktop content (95 Mbps worst realistic vs 76) and half the pathological ceiling (noise 2.8 Gbps @ 25 ms vs 4.9 Gbps @ 45 ms). Also adds a D3D11 decoder-profile diagnostic probe (gpu_pipeline): this box exposes HEVC VLD Main and Main10, so the decode gate is the webview layer, not silicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The receive twin of the encoder: nvcuvid + nvcuda loaded at runtime from the driver (no build dep), FFI hand-transcribed from the MIT dynlink headers (n12.0.16.0) with size-asserted layouts; CUVIDPICPARAMS stays opaque parser-to-decoder. One owner thread, callbacks synchronous inside cuvidParseVideoData, ulMaxDisplayDelay=0 so IPP streams surface every picture in its own call. Output is display-cropped NV12 copied tight; nv12_to_rgba (BT.709 limited) readies it for the native lane. The round-trip test is the load-bearing proof: paint -> GPU convert -> NVENC HEVC-lossless -> Annex-B -> NVDEC, with the encoder input read back via staging and compared byte-exact — 60/60 frames identical, which also hardware-verifies the encoder rung claim of mathematical losslessness. One real bug found on the way, kept as a comment: a coded-sized map target makes NVDEC scale the display rect up to the CTB-padded height (720->768) — a subtle vertical stretch that produced deterministic ink<->page swaps growing with y. Display-sized target = 1:1 crop, exactness restored. The failure diagnostics (plane split, shift search, ref->dec value pairs, BMP dumps behind ALLMYSTUFF_DIAG_DIR) stay in the test for the next such hunt. Measured at 1440p (bench_nvdec_hevc_1440p): decode+copy 3.7 ms avg / p95 4.9; scalar NV12->RGBA 11.5 ms avg — the lane's next optimization target, still inside the 60 fps budget. Test painters moved to nvenc::tests_support so both sides feed identical pixels. The full suite minus the audio test passes 154/0; that audio test crashes in isolation on this box (environmental, filed separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ort console is one shared open Two boundary guarantees the user asked for. 1) Pipeline tuning is now backend-only. The pure-pipeline feedback signals (est_kbps, delay_trend_us_per_s) no longer live as typed fields in Chris's protocol crate — RouteControl::VideoFeedback and Tune each carry a single opaque `ext: serde_json::Value` that the protocol and session crates relay VERBATIM and never inspect. The node video backend owns its shape via `video::PipelineFeedback` (to_ext/from_ext at the mesh edge): a new receiver-side signal or viewer-requested encoder knob is a field on THAT struct — no protocol, session, mobile-core, or GUI change ever again. The session crate's round-trip test now asserts the ext is relayed opaquely. Wire-compat: `ext` is serde-default + skip-if- null, so a 0.2.46 peer that sends none reads identically, both directions. This is the seam that lets Chris develop the protocol/ session/GUI freely while all encoder/decoder/pacing work stays in the node video modules. 2) The support (CEC) console is a literal clone of the main console. Both openConsole and openCecConsoleDirect now funnel through ONE `openConsoleResolved` — same ConsoleHost → Console.svelte, same video wiring, provably no drift. The only difference between them is authorization (a CEC customer arrives after their approval instead of a fleet relationship), which is upstream of the shared open, not inside the console. Gate: protocol/session/mobile-core 140+/0 (ext-relay assertion added), node 169/0, stub build compiles, clippy baseline, svelte-check 0/0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HANDOFF and the integration report now document the guarantee that encoder/decoder/pacing tuning stays in the node video modules — the opaque `ext` bag on VideoFeedback/Tune and labs.rs are the seams, so Chris's protocol/session/GUI never need touching for pipeline work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…caffolding) Both field boxes now support AV1 (5070 Blackwell: encode+decode; 9060 XT RDNA4: encode via AMF + decode), so the arc is unblocked. This lands the seams so implementing AV1 is filling rung bodies, not hunting branch points — all in node/src video modules, zero wire/GUI/protocol touch (the codec is carried key-to-key by the sniff, per the pipeline boundary). The structural gotcha handled: AV1 has no Annex-B start codes (it's OBUs). `sniff_codec` now falls through to a new `sniff_av1_obu` when no start code is found — detects a leading sequence-header OBU (the AV1 analog of repeated SPS/PPS), leb128-aware, conservative, and provably collision-free with H.264/HEVC (unit-tested: sniff_routes_h264_hevc_ and_av1_obu). `split_annexb_paced` documents where the OBU-aware split goes; today AV1 returns whole-AU (safe unpaced fallback). Stubs (compile, fail soft, dormant until an encoder emits AV1): - decode: AuCodec::Av1 + Av1Rung dispatch (NVDEC→D3D11VA) in video_decode.rs; NvdecAv1 (nvdec.rs, CUDA_VIDEO_CODEC_AV1=11 named); D3d11vaAv1 (d3d11va.rs, AV1_VLD_PROFILE0 GUID named). - encode: NV_ENC_CODEC_AV1_GUID (nvenc.rs); AMF_VIDEO_ENCODER_AV1 (amf.rs). probe_nvenc_av1_lossless already exists — run on the 5070. - Active::Av1 decode arm shares HEVC's NV12→RGBA paint path (same NvFrame seam), so the rung body is all that differs. docs/AV1-SEAMS.md maps every stub to what fills it, with the implement-in-this-order plan (probe → decode → encode → pacer → gate). Gate: node 170/0, clippy 38 (zero new; below baseline), stub build compiles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the wire-boundary defect Audit of docs/EXPERIMENTAL-ARC-PLAN-2026-07.md against the current tree (it was written at b992776, before labs.rs, the opaque ext seam, the Posture enum, AMF, and the AV1 stubs). The one real defect: §1.4 told a future implementer to add typed `caps: u32` / `last_good_ts_us` / `labs: u32` fields to Chris's protocol crate for the experimental handshake — which now violates the backend-only boundary (8e1308c). Corrected: those signals ride the opaque `ext` bag (video::PipelineFeedback + a PipelineHints), backend- only, zero protocol/session/GUI churn per feature. labs.rs now carries EVERY dial the plan names as a real Feature variant (Damage, PaintPace, SubFrame, QpMap, Grain, WaveStretch, Present, Ltr, GapNack, Rescue, EncAsync) with its ALLMYSTUFF_X_* env and labs_set name — so implementing a feature is filling its `if labs::on(...)` branch, never adding a gate. Added a reconciliation preamble: the gate + Experimental toggle are BUILT (toggle lives in ModePill, not a separate LabsPill; only the per-feature counter sheet remains), §1.4 is corrected, T2.7 wave-length already shipped, and the seam renames since writing (wave_flags is AtomicU32, GpuCodec gained Amf, route_rates exists, Tune.mode is a Posture enum). Verified the per-feature seams the plan depends on still exist (enqueue_decoded, parseVideoPacket, the lane-budget trio, LockBitstream slice_offsets); the damage-rect fetch is correctly marked to-add, not assumed. Gate: node 170/0, clippy 39 (baseline), stub build compiles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…f the upstream PR
The fork's own pipeline docs are now grouped under docs/fork/ and kept
off the upstream PR, so the PR stays code + review-dossier only and
Chris's docs tree is untouched.
Moved to docs/fork/ (fork-internal, excluded from the PR):
AV1-SEAMS, EXPERIMENTAL-ARC-PLAN, SMOOTHNESS-IDEAS, ENCODER-PASS.
New:
docs/fork/PIPELINE.md — the consolidated "how to interact with the
encoders/decoders/video pipeline" reference: the encode/decode
ladders, postures, the codec sniff, the pacer + closed loop, every
operator dial, the three backend-only seams you extend (ext bag,
labs gate, codec sniff), the log lines, and the boundary.
docs/fork/README.md — what's fork-only vs PR-facing, and the
PR-exclusion mechanism.
.gitattributes — docs/fork/ + HANDOFF.md marked export-ignore.
Stays PR-facing at docs/ root: INTEGRATION-REPORT + TESTER-KIT (the
review dossier written for Chris). HANDOFF.md stays at repo root as the
next-agent entry point but is fork-internal and excluded from the PR.
The mechanism (documented in TESTER-KIT §5 + docs/fork/README): a GitHub
PR is a whole-branch diff, so the PR opens from a curated `for-upstream`
branch that `git rm -r docs/fork && git rm HANDOFF.md` first — those
paths stay on fork main and never travel upstream.
Updated every cross-reference to the moved docs: 6 source-comment paths
(the AV1 stub + labs pointers), HANDOFF's review-set block, the
INTEGRATION-REPORT and TESTER-KIT links, and the two memory files.
Source changes are comment/string-literal only — fmt clean, compiles.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rdening Prep for the demonstration prototype — three workstreams, backend-only (no crates/ or gui/ touched; the ext/labs seams hold). Cross-platform builds (the fork had broken all three node-check legs): - videotoolbox: add the missing EncodeOutcome.input_ts field (macOS) - os_perf: declare pthread_set_qos_class_self_np locally (libc omits it) - node-check.yml: install libwayland-dev (Linux wayland-sys build script) - clippy 1.97: transmute annotations (nvdec), doc-list indent (os_perf), items-after-test-module (telemetry) - verified ARM/RISC-V run soft: scalar SIMD fallback, Windows-gated GPU rungs, soft-fail decode, 32-bit-safe leb128 Telemetry prod-stamp — new `field-telemetry` default feature. Dev and PROTOTYPE builds keep the telemetry line + verbose cwd debug log; a stamped prod build (--no-default-features --features host) ships silent. The ALLMYSTUFF_TELEMETRY / ALLMYSTUFF_CWD_LOG env dials still override either way. Red team of every encoder/decoder/pipeline backend (panic=abort makes any panic on peer input a remote DoS): - HIGH d3d11va: SPS conformance-window crop components are u32 *2 and the guard only checked their sum, which wraps in release — a crafted SPS passed the check while a single component became a ~2GB absolute byte offset in read_surface (OOB read → abort / adjacent-memory disclosure). Now checked_mul/checked_add, rejecting any overflow. - HIGH video: AIMD `next.clamp(RATE_FLOOR, ceiling)` aborts when a route ceiling is configured below the 8 Mbps floor (ALLMYSTUFF_VIDEO_BITRATE bypasses the floor) and two peer congestion reports arrive — clamp is now self-consistent (RATE_FLOOR.min(ceiling)). - MED d3d11va: a never-closing picture accumulated slice NALs unbounded (heap exhaustion) — per-picture slice cap. - MED video: peer recv_fps multiply overflow — saturating_mul. - defense-in-depth: nvdec nv12_to_rgba length guard; nvenc fps divisor guard + PresetConfig layout assert; win_capture Instant-underflow checked_sub; os_perf CPU-set record bound; gpu_pipeline null-texture guard; d3d11va SPS field range checks; leb128 u64 accumulation; est_bps u64 clamp before truncation. AMF reviewed by hand: FFI signatures, refcounts, and untrusted-input guards all sound; the vtable pad-run offsets remain hardware-gated (verify against the AMF 1.4.35 header on the 9060 XT — not editable blind). node 177/0, clippy -D warnings clean, fmt clean. Version stays 0.2.46 (Chris's release; prototype artifacts are distinguished by commit hash). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keeps a local AI-tool config dir (Codex) out of the tree, matching how .idea/.vscode are handled. Not project source; must not travel to the upstream PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Preserve AMF output under input backpressure, recover queue drops only through an accepted keyframe, clear stale GDR/lossless fallback state, and cap pacing to the real frame interval. Add a default-off Labs route-isolated sender prototype without changing media requests, signaling, or wire formats.
mrjeeves
marked this pull request as ready for review
July 19, 2026 17:40
Owner
|
Some failing integration tests still. |
Contributor
Author
|
Status update for the macOS report and integration-test follow-up:
The fork-local PR remains a draft CI trigger and should not be merged. |
Contributor
Author
|
Follow-up: the maintainer approval gate has cleared. Upstream CI run 815 and node-check run 6 both succeeded on the generated merge commit against current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the serialized display capture/convert/encode path with a pipelined video backend and adds native Windows hardware rungs:
Media stays on the existing authenticated ICE data plane. This PR adds no signaling channel, daemon request, or media path.
Defaults and boundaries
ALLMYSTUFF_CWD_LOG=1.--features field-telemetryorALLMYSTUFF_TELEMETRY=1..myownmesh-rev, release versioning, signaling, RTP packetization, FEC, and retransmission are unchanged.PR hygiene
The upstream-facing branch is
codex/video-pipeline-upstream-pr. Its final diff contains production code, automated regression tests, CI, and two maintainer documents.The cleanup at
c23d2c5and4402f9bremoved:docs/fork/**GitHub currently reports 46 changed files, +20,322 / -1,137. No prototype installer, log, dump, local tool configuration,
HANDOFF.md,docs/fork/**, ornode/examples/**path is in the proposed diff.Validation
Exact head:
4402f9b39697b1b85f86d56d0f56977d3767c261.Fork-local Actions are green on that exact tree:
hwenc, no-default-features, GUI check/build, GUI backends, and Android mobileThe two macOS errors reported from a development checkout were already fixed in
5fc7347: the Darwin QoS binding is declared locally and VideoToolbox initializesEncodeOutcome.input_ts. The exact final head passes macOS strict Clippy, node tests, root tests, and GUI-backend tests.The upstream merge-target workflows are also green on the generated merge commit against current
main: CI run 815 passed all 11 jobs, and node-check run 6 passed strict Clippy on macOS, Windows, and Ubuntu.Remaining limits