Reimplement video utils

[SkalskiP]

Description

The functionalities currently available in supervision.utils.video should be reimplemented and consolidated within a new Video class. Importantly, all features supported by the old video API must remain available in the new implementation.

• get video info (works for files, RTSP, webcams)

  import supervision as sv
   
  # static video
  sv.Video("source.mp4").info
  
  # video stream
  sv.Video("rtsp://...").info
  
  # webcam
  sv.Video(0).info

• simple frame iteration (object is iterable)

  import supervision as sv
  
  video = sv.Video("source.mp4")
  for frame in video:
      ...

• advanced frame iteration (stride, sub-clip, on-the-fly resize)

  import supervision as sv
  
  for frame in sv.Video("source.mp4").frames(stride=5, start=100, end=500, resolution_wh=(1280, 720)):
      ...

• process the video

  import cv2
  import supervision as sv
  
  def blur(frame, i):
      return cv2.GaussianBlur(frame, (11, 11), 0)
  
  sv.Video("source.mp4").save(
      "blurred.mp4",
      callback=blur,
      show_progress=True
  )

• overwrite target video parameters

  import supervision as sv
  
  sv.Video("source.mp4").save(
      "timelapse.mp4",
      fps=60,
      callback=lambda f, i: f,
      show_progress=True
  )

• complete manual control with explicit VideoInfo

  from supervision import Video, VideoInfo
  
  source = Video("source.mp4")
  target_info = VideoInfo(width=800, height=800, fps=24)
  
  with src.sink("square.mp4", info=target_info) as sink:
      for f in src.frames():
          f = cv2.resize(f, target_info.resolution_wh)
          sink.write(f)

• multi-backend support decode/encode; implement PyAV and OpenCV

  import supervision as sv
  
  video = sv.Video("source.mkv", backend="pyav")
  
  video = sv.Video("source.mkv", backend="opencv")

  suggested minimal protocol

  class Backend(Protocol):
      def open(self, path: str) -> Any: ...
      def info(self, handle: Any) -> VideoInfo: ...
  
      def read(self, handle: Any) -> tuple[bool, np.ndarray]: ...
      def grab(self, handle: Any) -> bool: ...
      def seek(self, handle: Any, frame_idx: int) -> None: ...
  
      def writer(self, path: str, info: VideoInfo, codec: str) -> Writer: ...
  
  class Writer(Protocol):
      def write(self, frame: np.ndarray) -> None: ...
      def close(self) -> None: ...

Additional

• Please share a Google Colab with minimal code to test the new feature. We know it's additional work, but it will speed up the review process. The reviewer must test each change. Setting up a local environment to do this is time-consuming. Please ensure that Google Colab can be accessed without any issues (make it public). Thank you! 🙏🏻
• Mark all methods from the old video API as deprecated. Find examples of already deprecated methods or classes in the current codebase and use the same approach to mark the old video API methods. A deprecation period of at least 5 releases is required.
• Reimplement the internals of the old video API using the new video API.
• Provide full unit-test coverage matching existing test style.
• Update the library docs (docstrings, mkdocs) to reflect the new API.
• Take into account the comments in Issue with sv.VideoInfo FPS Handling for Precise Video Metadata Retrieval #1687.

[Ashp116]

Hey @SkalskiP,

Could you elaborate on this feature, "complete manual control with explicit VideoInfo"? I was trying to implement this, but was confused about how the new Video class is used after instantiation?

from supervision import Video, VideoInfo

source = Video("source.mp4")
target_info = VideoInfo(width=800, height=800, fps=24)

with src.sink("square.mp4", info=target_info) as sink:
    for f in src.frames():
        f = cv2.resize(f, target_info.resolution_wh)
        sink.write(f)

[timmermansjoy]

@Ashp116 You maybe want to work together on this? Otherwise i'm also just gonna give it a shot.

To your question. you have your video class source = Video("source.mp4") and you will later use it by adding the VideoInfo to it like source.sink("square.mp4", info=target_info). You maybe are confused because its source and src. which should be same in the example. This is a great idea and allows us to integrate different backends and they all just have to implement this VideoInfo class features to work

[SkalskiP]

Hi @timmermansjoy and @Ashp116! I highly encourage both of you to work on it (together or separately). This task seems easy at first glance but requires a lot of prototyping and design considerations.

@Ashp116 as for your question @timmermansjoy is correct. The idea is that you can overwrite VideoInfo of the source video to change fps or resolution all at once.

[Ashp116]

Hi @SkalskiP and @timmermansjoy,

@timmermansjoy Thank you for the clarification. I initially predicted to do what you mentioned, but due to the code error, I was confused about how that feature should be implemented. As for working on this together, I don't mind working with you. But I pretty much have everything done over here. Right now I am just adding the "Additions". Currently, finishing up a fix for #1687 and writing unit tests.

@SkalskiP Thank you for your response. I do want to discuss with you on #1687. I was benchmarking and testing out myself to try to resolve this issue. Out of curiosity, are we going to depreciate FPSMonitor? Also, I think the resolution for the issue here is to just round the float you get from OpenCV and then convert to an int; it produces much more accurate results.

Results from heatmap_and_track with input as people-walking-1

Float FPS: 7.95795
Int FPS: 8.084745762711865
Int FPS (Rounded): 7.95

Thoughts on rounding and then converting to int?

[timmermansjoy]

@Ashp116 you just added stuff to the existing utils/video file? I Believe @SkalskiP is asking to create a new dedicated video class in a new directory with a structure like so:

supervision/
├── __init__.py
├── video/
│   ├── __init__.py
│   ├── core.py          # Contains the Video class
│   └── utils.py
│   └── backends
│      └── opencv.py 
│      └── pyAV.py 
└── utils/
    └── video.py         # Old video utilities (to be deprecated)

where its easy to add different backends in the future

you also install imageio-ffmpeg but then just use subprocess instead? Also i would highly reccomend to use pyAV instead, this is kinda the defacto library for working with ffmpeg in a pythonic way and even the authors of imageio-ffmpeg say so in their own readme:

This library was created before PyAV was a thing. But now they have binary wheels for many platforms. You should probably use PyAV instead; it is faster and offers more features.

One last thing, what exactly do you mean with rounding and converting to int? all of your numbers that you give as example are decimals?

@SkalskiP If you could confirm that the above structure is kinda what you are looking for I'll give it a try.

[SkalskiP]

Hi @Ashp116 and @timmermansjoy 👋🏻

To be honest, I did not specify the expected file structure in the task description; I wanted to leave that decision to whoever implements the task. The layout proposed by @timmermansjoy seems very sensible, though.

Regarding imageio-ffmpeg, OpenCV, and pyAV, I would like audio handling to depend on the chosen backend. Since OpenCV cannot process audio, the Video class should ignore audio when it uses the OpenCV backend. In contrast, pyAV does support audio, so the Video class should enable it with the pyAV backend.

Please add all new dependencies to pyproject.toml.

[Ashp116]

Hey @timmermansjoy and @SkalskiP,

Yes, @timmermansjoy, I have currently added everything to one file, which was one of my concerns. I wanted @SkalskiP's thoughts here. Using imageio-ffmpeg instead of pyAV was because OpenCV doesn't have audio loading capabilities. Initially, imageio-ffmpeg was used to have process_video pack the audio stream into the outfile. The file structure you proposed is valid. Would you like to append those changes to the PR or do you want me to apply those changes?

One last thing, what exactly do you mean with rounding and converting to int? all of your numbers that you give as example are decimals?

I am proposing a solution to fix #1687 where the FPS is rounded down, which generates a lost buffer for ByteTrack: track_seconds * FPS.

After @SkalskiP's response, I will have audio processing to be dependent on the type of backend.

[SkalskiP]

@Ashp116 Let me know when you'll be ready for code review. 🔥🔥🔥

[Ashp116]

@SkalskiP could you code review right now? I wasn't able to send a code review request since the previous review request was not completed.

[SkalskiP]

@Ashp116 I’ve left only high-level comments for now.

@Ashp116 and @timmermansjoy this week we experimented with automated code generation. One of the resulting PRs — #1941 — is an attempt to solve this task. It’s not perfect, but I think it’s close enough to be worth studying and drawing inspiration from.