Add lossless raw thermal streaming, recording and SupportProxy transport - #31
Conversation
Map vehicle boot clocks into local time and match vehicle yaw to gimbal feedback so transport jitter does not mix samples of different ages. Derive stationary ROI line-of-sight rates directly from position and velocity, avoiding quantisation from finite differences of rounded coordinates, and stop prediction when telemetry is stale.
Preserve yaw history when corrected sample times coincide, allow ATTITUDE fallback at prediction expiry, and retain vertical-pitch metadata without using an undefined yaw rate for control. Carry measured gimbal yaw rate into video metadata so prediction avoids differentiating quantised angles, with regression tests and documentation for both changes.
Preserve native 16-bit thermal samples and capture-time telemetry in FFV1/Matroska on MT11 and SITL. Expose independent live streaming and recording rates through MAVLink and web parameters, with raw recording following the existing video recording policy. Add legacy-frame conversion and extraction tools, interoperability tests, and hardware validation results.
Old raw thermal .bin captures carry only a timestamp, so their per-frame apcg.telemetry.v1 snapshot was null. Add --bin FLIGHT.bin to fill it from an ArduPilot dataflash log: GPS week/ms give the log absolute UTC, and each frame's absolute filename (or mtime) time is matched against it. Position, NED velocity, vehicle attitude with yaw rate and gimbal attitude are interpolated between the bracketing samples, yaw along the shortest arc, with a per-field age. Frames outside the log's coverage keep a reconstructed clock and a null pose. Gimbal backends fill MNT differently, so per axis the reported angle is preferred, then the demanded angle, and vehicle-relative yaw is preferred over earth-referenced yaw (converted with the vehicle yaw); the chosen source is recorded in each frame's gimbal_pose_source. Adds sampler and gimbal-selection unit tests.
Expose the thermal proxy port and stream name through camera configuration and advertise the proxy Matroska URL over MAVLink. Publish native FFV1 frames and capture metadata through an independent worker that retains only the newest queued frame and tolerates slow uplinks without blocking local recording. Add configuration, queue and end-to-end SITL coverage, including proxy restart and operation with the local thermal HTTP listener disabled.
Previous review (2026-09-21)Automated review note — AI-generated (Claude), validated against the live diff. Please sanity-check before acting. Full report: https://uav.tridgell.net/DevCallReviews/2026_09_22_AIReview/devcall_pr_reviews.html#prAP_CameraGimbal-31 Reviewed at head REQUEST CHANGES. The FFV1/Matroska pipeline is genuinely lossless and is the strongest part of this change — I verified that end to end. What blocks it is more prosaic: four CI jobs are red and three are caused directly by this PR, each traced to its actual failure line in the job log. There's also one wire-protocol bug that two independent passes found separately. Given this repo merged #24 thirty-eight minutes before its review landed, please hold this one until at least findings 1-4 are addressed. It's open and unmerged as I write. 1. BUG — This is reachable, and the decisive line is if (stream == 0U) {
ca_thermal_stream_enable(enabled);
for (unsigned i = 0; i < APCAM_NUM_STREAMS; i++) server->stream_enabled[i] = enabled;
}
else if (stream == CA_RAW_THERMAL_STREAM_ID && ca_thermal_stream_available())
ca_thermal_stream_enable(enabled);
else if (stream <= APCAM_NUM_STREAMS) server->stream_enabled[stream - 1U] = enabled;
else return MAV_RESULT_DENIED;2. BUG — 3. BUG — 4. BUG — 5. BUG — 6. ISSUE — 7. ISSUE — 8. ISSUE — Notes (detail in the report): 640×512 is hardcoded in five files while The fourth red job, What I checked and cleared: the lossless claim holds end to end — FFV1 level 3, |
…s accepted VIDEO_START/STOP_STREAMING for stream 3 toggled the encoder before the command was checked, then returned MAV_RESULT_DENIED when the local raw HTTP listener was disabled. With SupportProxy as the only transport the stream changed state while the GCS was told the command failed. Gate on the same availability stream_count() already uses, so proxy-only publishing accepts the command. The SupportProxy SITL raw thermal case now runs without the local listener and checks stream 3 start/stop. Load the checkout's ThermalReader by path there, since the shared test helpers have already imported the installed MAVProxy package.
The camera-app-host build uses -Werror=unused-parameter, which failed the non-SITL arm of configure_raw_thermal().
SupportProxy settings are deliberately kept out of the camera definition, matching the other PROXY_ keys; the setting remains available through the INI file, environment and MAVLink parameters.
The link rules hardcoded -lpthread -lm, which misses libavutil's platform dependencies; on Cygwin random_seed.o needs BCrypt and the SITL link failed. Take the extra libraries from the pkg-config files ffmpeg's configure installed so new dependencies follow automatically.
ArduPilot logs RATE.Y with unit 'k' (deg/s) but it was written unchanged into the rad/s telemetry field. Split the series builder out of the log loader so the unit conversions can be tested without a dataflash file.
The time-aligned yaw history rejected any query more than 250 ms past its newest sample, and the vehicle position expired after 250 ms, where the previous code held both for 1 s and 1.5 s with prediction capped at 250 ms. A short telemetry gap flipped GIMBAL_DEVICE_ATTITUDE_STATUS between earth and vehicle frames and stopped ROI tracking; the MT11 integration and manual-control tests, which send vehicle state once, timed out in CI. Cap extrapolation at 250 ms on either side of the history and hold the newest sample for 1 s, restore the 1.5 s position hold, and fall back to the current held yaw when gimbal feedback cannot be aligned with the history.
|
Automated review note — AI-generated (Claude), cross-checked by a second independent Claude pass against the live diff. Please sanity-check before acting. Full report: https://uav.tridgell.net/DevCallReviews/followups/2026_09_22_2315/devcall_pr_reviews.html#prAP_CameraGimbal-31 Re-reviewed at head All five findings from last round are genuinely fixed, and three of the four red CI jobs are now green. Verdict stays REQUEST CHANGES for one reason: Previous round
A finding that was queued and then dropped, because the author is right. I was going to report that the new 🔴 BLOCKER —
|
| job | now | last round |
|---|---|---|
build (Cygwin) |
pass | fail → fixed |
exposure |
pass | fail → fixed |
release |
pass | fail → fixed |
legacy-python (3.10.11 / 3.10.13 / 3.12) |
pass | pass |
mt11-sitl |
fail | fail → partly fixed |
mt11-sitl per step, comparing job 106304530396 (old head) with 106674548488 (new): "Test ArduPilot SIYI drivers against MT11 SITL" passed both rounds; "Test ArduPilot MAVLink drivers over NET against MT11 SITL" no longer dies at test_mavlink_integration.py:203 — that step now prints PASS native MAVLink camera/gimbal capabilities over TCP, UDP and UART with position targeting enabled — and instead fails further on, at the stream assertion above.
What was not checked
The mt11-sitl job end to end (it needs an ArduCopter SITL build plus the pinned camera dependencies; both job logs were read instead, and the attribution above is from the job's own camera.log plus the source path, not a local run). sitl/test_support_proxy.py --raw-thermal was reviewed statically but not executed — it needs a SupportProxy checkout with Matroska relay support. camera_app/tests/test_telemetry_time was built and run and passes; 7 of 11 tests/test_thermal_to_video.py tests pass locally, the other 4 erroring on a missing local av/numpy (an environment gap, and they are green in release). The parts cleared last round — FFV1/Matroska losslessness, wire-input bounds, the SupportProxy wire contract — were not re-audited, since thermal_stream.cpp is byte-identical.
This run's usual second reviewer (Codex) was unavailable — hard-blocked on a usage limit until 2026-09-27 — so the cross-check was a second independent Claude context rather than a different vendor. Both passes reached the same blocker and the same frame error independently.
…test The MT11 SITL advertises stream 3 whenever its raw listener is open, so the routed VIDEO_STREAM_INFORMATION check now expects three streams and verifies the raw stream's type, URI, flags and geometry.
RATE logs body gyro rates, while yaw_rate_rad_s is the earth-frame Euler yaw rate the camera receives from AUTOPILOT_STATE_FOR_GIMBAL_DEVICE. Apply the same conversion the camera uses for ATTITUDE, with roll and pitch interpolated from ATT onto the RATE timestamps, and drop samples near vertical pitch or outside the attitude coverage.
…AVProxy The --mavproxy default was a personal checkout path; use ../MAVProxy like the SupportProxy default and let MAVPROXY_REPO override it. Add the --raw-thermal --reconnect case to sitl-supportproxy-test so the proxy-only stream 3 start/stop regression is exercised by the target.
It applies to any query older than the whole history, not only to feedback captured just after a reset.
2 participants
Add lossless raw thermal streaming and recording on MT11 and SITL-MT11, preserving the sensor's complete 640×512 16-bit radiometric output together with capture-time telemetry. This makes temperature measurements and approximate map projection possible without extracting data from a colourised display stream.
The first two commits align camera telemetry timestamps with the vehicle clock and derive ROI line-of-sight rates from position and velocity, so the per-frame telemetry carried by the raw stream refers to capture time. They were previously proposed separately as #30, which is superseded by this PR.
RAW_STREAM_FPSandRAW_RECORD_FPSparameters through MAVLink and the web UI. Recording follows existing manual, automatic and while-armed video policies; both rates default to 5 fps and recording rate 0 disables raw recording.PROXY_VID3_PORTand a configurable stream name for publishing through SupportProxy. Proxy requests receive/v3.mkv; a separate publisher keeps only the latest queued frame and tolerates slow uplinks without waiting in the capture/recording path. Proxy streaming also works with the local thermal HTTP listener disabled.tools/thermal_to_video.pyfor lossless conversion of legacy captures and extraction back to individual images. Optional--binreconstructs per-frame telemetry from an ArduPilot DataFlash log.Review fixes (one commit per subsystem):
mavlink_server:VIDEO_START/STOP_STREAMINGfor stream 3 toggled the encoder before denying the command when only SupportProxy carried the stream. It is now gated on the same availabilitystream_count()uses. The SupportProxy SITL raw-thermal case runs without the local listener and checks stream 3 start/stop.mavlink_server: the time-aligned yaw history dropped vehicle telemetry 250 ms after the last sample, and position after 250 ms, where the previous code held them for 1 s and 1.5 s with prediction capped at 250 ms. This flipped attitude feedback between earth and vehicle frames on short gaps and failed the MT11 integration and manual-control tests. Extrapolation is capped at 250 ms on either side of the history, the holds are restored, and status falls back to the current held yaw when feedback cannot be aligned.media: unused parameter in the host stub under-Werror.camera_definition:PROXY_VID3_PORTis no longer advertised in the camera definition, matching the otherPROXY_settings; it remains an INI, environment and MAVLink parameter.Makefile: the static libav archives' platform libraries are taken from the pkg-config files ffmpeg installs, which fixes the Cygwin SITL link against BCrypt.thermal_to_video:RATErates are logged in deg/s and are body gyro rates, while the schema'syaw_rate_rad_sis the earth-frame Euler yaw rate the camera receives live. The converter now applies the camera's own conversion,(q sin(roll) + r cos(roll)) / cos(pitch), with roll interpolated along the shortest arc fromATT; near vertical pitch the rate is undefined and the snapshot omits it rather than bridging the gap. The series builder is a separate function with unit tests for units, bank, roll wrap and the vertical gap.tests: the routed ArduPilot MAVLink test expected two video streams; the MT11 SITL advertises the raw stream too, so it now checks stream 3's type, URI, flags and geometry.sitl: the SupportProxy test's--mavproxydefault is repo-relative (../MAVProxy, orMAVPROXY_REPOthrough make), andmake sitl-supportproxy-testruns the--raw-thermal --reconnectcase that covers proxy-only stream 3 control.Validation completed:
The MAVProxy raw thermal reader needs ArduPilot/MAVProxy#1760, which disables FFmpeg frame-rate probing: with no DefaultDuration in the live stream, probing read up to 64 KB of frames inside the 3 s open timeout, and highly compressible frames (SITL test pattern, uniform scenes at low rates) made an interrupted probe poison the demuxer. With that change
sitl/test_raw_thermal_stream.pypasses end to end.The proxy path requires the companion SupportProxy Matroska changes. Temperature-aware viewing requires the MAVProxy raw thermal viewer or the companion desktop viewer; the proxy's existing MPEG-TS browser player cannot decode this stream.
Interoperability remains experimental: MAVLink stream type 200 and the APCG BlockAdditional mapping are private values, not upstream allocations. Timestamps have microsecond representation but describe USB reception rather than calibrated sensor exposure. Sustained 25 fps hardware operation and surveyed projection accuracy have not been validated. Truncation tolerance does not guarantee SD/filesystem persistence after power loss; the camera currently forces a flush when recording stops.
Outstanding review follow-ups: raw frame ingest runs at sensor rate even with no consumer, an encoder failure leaves stream 3 advertised, a flush error when stopping raw recording rolls back the rate while the file is already closed, plus isolating raw startup/storage failures from normal video and unrelated configuration changes, SD-write stalls, raw-file rollover/failure reporting, encoder recovery, and converter output on filesystems without hard links.