Skip to content

Add histogram and optical flow comparison methods to video_describer's extract_significant_frames #10

@chigwell

Description

@chigwell

User Story
As a video analysis user,
I want multiple frame comparison methods in extract_significant_frames
so that semantic changes between visually similar frames are captured more effectively.

Background
The current grayscale pixel difference method in VideoDescriber.extract_significant_frames (lines 33-49 of video_describer.py) only detects absolute pixel value changes. This fails to capture meaningful content changes when frames have similar brightness distributions but different color patterns or object movements (e.g., a red object turning blue while maintaining luminance). The existing threshold-based approach also lacks context-aware comparison capabilities.

Acceptance Criteria

  • Modify frame_story/video_describer.py to implement:
    • Histogram comparison using OpenCV's cv2.compareHist with correlation method
    • Dense optical flow analysis via cv2.calcOpticalFlowFarneback
  • Add method parameter to VideoDescriber.__init__ with options:
    ["grayscale" (default), "histogram", "optical_flow"]
  • Update frame significance logic to:
    • For histogram mode: Compare HSV histograms using 16-bin quantization
    • For optical flow: Calculate magnitude thresholding on flow vectors
  • Validate through unit tests in tests/test_video_describer.py:
    1. Ensure all methods detect frame changes in test videos with:
      • Same composition but different color schemes (must fail with grayscale)
      • Static background with moving foreground objects
    2. Maintain ≤5% false positive rate compared to current implementation
  • Keep backward compatibility by preserving default grayscale behavior
  • Add configuration parameters for histogram/optical flow thresholds in VideoDescriber initialization

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions