Repository navigation
fix(patchcore): stamp Insight video frames with the source frame identity - #678
Yukkta-Seelam wants to merge 3 commits into
Conversation
…tity The annotated frame was pushed to Insight as a bare tensor, so the video got its own timestamp while the verdict metadata carried the source frame's. Insight pairs the two by timestamp, so verdicts could land on the wrong frame when inference dropped or delayed frames. Push each annotated frame as a sample stamped with the source pts and frame ID, and send the verdict with the same values. Video-file mode stamps both with the same computed pts. Same change in C++ and Python. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The video sender's input left appsrc do-timestamp on, so it restamped each pushed frame with its push time and dropped the source timestamp the previous commit set. On a DevKit only 2 of 600 RTSP verdicts landed on a video frame. With do_timestamp off, all 600 RTSP and 450 video-file verdicts match their frame within 1 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Develop (#698) now pushes the NV12 source frame in RTSP mode and BGR in video-file mode. The frame identity fix is re-applied on that code: each pushed frame carries the source pts_ns and frame_id, and the verdict uses the same values.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd5546e400
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| input_options.fps_n = output_fps | ||
| input_options.fps_d = 1 | ||
| # Keep each pushed frame's source timestamp; Insight pairs it with the verdict by timestamp. | ||
| input_options.do_timestamp = False |
There was a problem hiding this comment.
Assert video/metadata timestamp pairing end to end
The Python and C++ live-source E2E tests verify verdict metadata, frame IDs, and snapshots, but neither captures the video RTP timestamp and compares it with the metadata timestamp. Consequently, removing this setting or dropping sample.pts_ns would recreate the original Insight pairing failure while every automated test still passes. Add an end-to-end pairing assertion for both implementations, using the same RTP-versus-metadata timestamp rule described in the validation.
AGENTS.md reference: AGENTS.md:L100-L102
Useful? React with 👍 / 👎.
Refs #649 (review comment #649 (comment)).
Summary
PatchCore sends each annotated frame to Insight through a separate push graph, and the verdict through
MetadataSender. The frame was pushed as a bare tensor, so the video got its own timestamp, while the verdict carried the source frame'spts_ns(RTSP) or a computed time (video file). Insight pairs metadata with video by timestamp (timestamp_ms × 90against the frame's RTP timestamp), so a verdict could be drawn on the wrong frame, or on none.pts_nsandframe_id. The verdict uses the samepts_nsandframe_id. The app's frame counter is used only when the source frame has no ID.(n - 1) / fps.do_timestampis off, so the input keeps the pushed timestamp instead of restamping each frame with its push time.multi-stream-blazepose3dbuilds its pushed image samples.Skill updates
None. Pushing an image sample with a copied source identity is already the pattern in
multi-stream-blazepose3d.Validation
Office DevKit, Core
0.6.1+develop.f70653154707, built from this branch (cd5546e4) with CMake on the board. A local GStreamer RTSP server served the PatchCore test clip as H.264. A UDP listener on the app's video and metadata ports applied Insight's pairing rule: a verdict counts as matched when some video frame's RTP timestamp is within 90 ticks (1 ms) oftimestamp_ms × 90.cd5546e4, including the live video-file and RTSP end-to-end runs in both languages.