camera: lossless raw thermal viewing and playback - #1760
Merged
Merged
Conversation
Decode FFV1 frames with their capture-time metadata for temperature readout, native sample saves and paused pixel projection. Default to greyscale with optional palettes and show temperatures in a readable panel below the image.
thermal_to_video.py writes 16-bit FFV1/Matroska thermal recordings whose per-frame apcg.telemetry.v1 snapshot lives in Matroska block additions, not H.264/H.265 SEI. Teach the player to handle them. VideoIndex detects an FFV1 stream and indexes it with PyAV, reading each frame's block-additional apcg.thermal.v1 metadata and exposing the nested telemetry, with the thermal lens HFOV taken from the outer metadata when the snapshot lacks one. FrameReader gains a PyAV thermal decoder: it seeks by presentation timestamp (every frame is independently coded) and auto-ranges the 16-bit radiometric image to 8-bit grey with 1st/99th percentile scaling. OpenCV alone flattens the 16-bit frames to near-black. Frames captured before the flight log started keep their reconstructed clock with a null pose and simply show without an overlay. Adds parser tests for the block-additional metadata.
The camera omits Matroska DefaultDuration because the stream rate can change live, so FFmpeg probed the frame rate by reading frames until the 64 KB probesize was spent. Highly compressible frames (SITL test pattern, or a uniform scene at a low rate) made that take longer than the 3 s open timeout; the interrupted probe left the demuxer's I/O context in error while the open still succeeded, and the reader failed as soon as the probe-buffered packets were consumed. Codec parameters come from CodecPrivate and the first decoded packet, so opening now completes on the first frame.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add viewing and playback of the lossless 16-bit raw thermal stream that AP_CameraGimbal MT11 firmware advertises as stream 3 (ArduPilot/AP_CameraGimbal#31).
camera view rawthermal(orcamera view 3) opens the FFV1/Matroska HTTP stream with PyAV and shows it with white-hot greyscale by default, optional Inferno and Turbo palettes, per-frame minimum and maximum temperatures, and hover temperature readout. The popup menu can pause and save the native 640x512 little-endian samples with theirapcg.thermal.v1metadata. While paused, a left click projects the pixel to terrain using that frame's embedded position and attitudes.mavvidplayplays the.raw.mkvrecordings written by the camera and bythermal_to_video.py, indexing the per-frame telemetry from Matroska block additions and auto-ranging the 16-bit radiometric frames for display.MAVLink stream type 200 and the APCG block-additional mapping are private firmware/MAVProxy values, not upstream allocations, and are documented as experimental in the module README. Requires PyAV 18.1 or later with packet side-data support.
Tested with the AP_CameraGimbal SITL raw thermal stream and SupportProxy relay tests, and with
tests/test_raw_thermal.py,tests/test_video_telemetry.pyandtests/test_mavproxy_camera.py.