Skip to content

fix(patchcore): stamp Insight video frames with the source frame identity - #678

Open
Yukkta-Seelam wants to merge 3 commits into
developfrom
fix-patchcore-frame-identity
Open

Yukkta-Seelam wants to merge 3 commits into
developfrom
fix-patchcore-frame-identity

Conversation

@Yukkta-Seelam

@Yukkta-Seelam Yukkta-Seelam commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Refs #649 (review comment #649 (comment)).

Summary

PatchCore sends each annotated frame to Insight through a separate push graph, and the verdict through MetadataSender. The frame was pushed as a bare tensor, so the video got its own timestamp, while the verdict carried the source frame's pts_ns (RTSP) or a computed time (video file). Insight pairs metadata with video by timestamp (timestamp_ms × 90 against the frame's RTP timestamp), so a verdict could be drawn on the wrong frame, or on none.

  • RTSP mode: each annotated NV12 frame is pushed as a sample stamped with the source frame's pts_ns and frame_id. The verdict uses the same pts_ns and frame_id. The app's frame counter is used only when the source frame has no ID.
  • Video-file mode: the BGR frame and the verdict are stamped with the same computed timestamp, (n - 1) / fps.
  • Video sender input: do_timestamp is off, so the input keeps the pushed timestamp instead of restamping each frame with its push time.
  • C++ and Python make the same change. The sample is built the way multi-stream-blazepose3d builds its pushed image samples.
  • Develop is merged in. The fix now sits on refactor(examples): simplify input memory and streamline live demos #698's NV12/BGR push path.

Skill updates

None. Pushing an image sample with a copied source identity is already the pattern in multi-stream-blazepose3d.

Validation

Office DevKit, Core 0.6.1+develop.f70653154707, built from this branch (cd5546e4) with CMake on the board. A local GStreamer RTSP server served the PatchCore test clip as H.264. A UDP listener on the app's video and metadata ports applied Insight's pairing rule: a verdict counts as matched when some video frame's RTP timestamp is within 90 ticks (1 ms) of timestamp_ms × 90.

Run Frames Verdicts matched to a frame Nearest frame, median / max (ticks)
develop, Python, RTSP 600 41 (6.8%) 264 / 127,962
this branch, Python, RTSP 600 600 (100%) 56 / 85
this branch, Python, video file 450 450 (100%) 29 / 60
this branch, C++, RTSP 600 600 (100%) 33 / 62
this branch, C++, video file 450 450 (100%) 29 / 60
  • Python unit tests: 36 passed. C++ unit test: passed.
  • Vulcan CI run 43547716766 passed at cd5546e4, including the live video-file and RTSP end-to-end runs in both languages.
  • Not covered by an automated test: the live end-to-end runs check each verdict's metadata against the printed verdict, but none pairs a verdict's timestamp with a video frame. The table above is the evidence for that.
  • Not verified: the view in the Insight UI itself.

…tity

The annotated frame was pushed to Insight as a bare tensor, so the video
got its own timestamp while the verdict metadata carried the source
frame's. Insight pairs the two by timestamp, so verdicts could land on
the wrong frame when inference dropped or delayed frames.

Push each annotated frame as a sample stamped with the source pts and
frame ID, and send the verdict with the same values. Video-file mode
stamps both with the same computed pts. Same change in C++ and Python.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The video sender's input left appsrc do-timestamp on, so it restamped
each pushed frame with its push time and dropped the source timestamp
the previous commit set. On a DevKit only 2 of 600 RTSP verdicts landed
on a video frame. With do_timestamp off, all 600 RTSP and 450
video-file verdicts match their frame within 1 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Develop (#698) now pushes the NV12 source frame in RTSP mode and BGR in
video-file mode. The frame identity fix is re-applied on that code: each
pushed frame carries the source pts_ns and frame_id, and the verdict uses
the same values.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T03:38:31.474467Z cd5546e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd5546e400

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

input_options.fps_n = output_fps
input_options.fps_d = 1
# Keep each pushed frame's source timestamp; Insight pairs it with the verdict by timestamp.
input_options.do_timestamp = False

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert video/metadata timestamp pairing end to end

The Python and C++ live-source E2E tests verify verdict metadata, frame IDs, and snapshots, but neither captures the video RTP timestamp and compares it with the metadata timestamp. Consequently, removing this setting or dropping sample.pts_ns would recreate the original Insight pairing failure while every automated test still passes. Add an end-to-end pairing assertion for both implementations, using the same RTP-versus-metadata timestamp rule described in the validation.

AGENTS.md reference: AGENTS.md:L100-L102

Useful? React with 👍 / 👎.

This branch was successfully deployed

4 active deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant