Frame Processor is a high-performance, multi-threaded C++ application designed for robust video ingestion, processing, and distribution. It ingests multiple RTSP streams, performs real-time frame transformations (cropping, scaling), and efficiently batches and publishes the results to Redis Streams for downstream analytics.
Designed for reliability and scalability, it employs a supervisor process pattern to ensure high availability and uses a modern C++20 codebase with efficient memory management.
- Multi-Stream Ingestion: Concurrent handling of multiple RTSP video streams.
- Horizontal Scalability: Run multiple instances safely distributed via consistent hashing & subset filtering.
- Testing & Debugging: Includes pipeline 'dry-run' toggles for network testing and raw/processed frame disk-saving tools for model verification.
- Dynamic Reconfiguration: Add, remove, or modify streams and processing rules at runtime without service interruption.
- High-Performance Processing:
- Efficient thread pooling for parallel frame processing.
- FFmpeg-based filter graphs for high-quality cropping and scaling.
- Zero-copy optimizations where possible.
- Robust Architecture:
- Supervisor Pattern: A parent process monitors the application and automatically restarts it in case of failures.
- Graceful Shutdown: Ensures all resources are released and queues flushed upon termination.
- Efficient Output:
- Batches processed frames to reduce network overhead.
- Serializes data into a compact, padding-free binary format (YUV420P).
- Publishes to Redis Streams for decoupled consumption.
- Observability: Structured logging (via
fmtlog) and detailed error reporting.
The application follows a pipelined producer-consumer architecture:
- Stream I/O Manager: Connects to RTSP sources, demuxes, and decodes video frames. Pushes raw frames to the Decode Queue.
- Worker Pool: Consumes frames from the Decode Queue. It applies configured transformations (Crop/Scale) using FFmpeg filter graphs and pushes the results to the Processed Queue.
- Publisher: Consumes from the Processed Queue. It batches frames based on stream ID and event type, serializes them, and publishes the batches to Redis.
- Operating System: Linux (Ubuntu 24.04 recommended)
- Compiler: C++20 compatible compiler (Clang 18+ recommended)
- Build System: CMake 3.16+
- Dependencies:
- FFmpeg 8.0 (libavcodec, libavformat, libavutil, libswscale, libavfilter)
- fmt (11.0)
- simdjson (3.11)
- hiredis
- libcurl
- jemalloc (optional, recommended for performance)
The project includes a multi-stage Dockerfile that builds all dependencies from source to ensure version compatibility and optimization.
-
Build the image:
docker build -t frame-processor . -
Run the container:
docker run -d \ --net=host \ -v $(pwd)/config:/etc/fp/configs \ -e FP_REDIS_HOST=localhost \ frame-processor
-
Install system dependencies:
sudo apt-get update && sudo apt-get install -y \ build-essential cmake git pkg-config \ libcurl4-openssl-dev libhiredis-devNote: You may need to build FFmpeg 8.0, fmt 11, and simdjson from source if your distribution's repositories are outdated.
-
Build the project:
mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=Release .. make -j$(nproc)
The application is configured via a JSON file and environment variables.
| Variable | Description | Default |
|---|---|---|
FP_REDIS_HOST |
Required. Hostname or IP of the Redis server. | - |
FP_REDIS_PORT |
Port of the Redis server. | 6379 |
FP_REDIS_QUEUE_BASE_NAME |
Base name for Redis Stream keys. | fp_batches |
FP_NUM_WORKER_THREADS |
Number of processing threads. | CPU Cores - 2 |
FP_PUBLISHER_THREADS |
Number of concurrent Redis publisher threads. | 4 |
FP_INSTANCE_SLOT |
Instance slot for horizontal scaling (e.g. 0). |
0 |
FP_TOTAL_INSTANCES |
Total configured instances for horizontal scaling. | 1 |
FP_STREAM_IDS |
Comma-separated allowlist of stream IDs to process exclusively. | "" |
FP_PUBLISHER_DRY_RUN |
Skips Redis I/O but simulates latency for testing backpressure. | false |
FP_SAVE_DECODED_FRAMES |
Saves raw decoded frames to disk prior to processing. | false |
FP_SAVE_PROCESSED_FRAMES |
Saves finalized processed frames to disk for inspection. | false |
FP_SAVED_FRAMES_OUTPUT_DIR |
Base output directory for saved frames. | /tmp/output |
The configuration file defines the streams and their processing rules. It is typically located at /etc/fp/configs/config.json.
{
"global_settings": {
"log_level": "INFO"
},
"streams": [
{
"id": 1,
"url": "rtsp://user:pass@ip:port/stream",
"enabled": true,
"processing": {
"crop": { "x": 0, "y": 0, "width": 1920, "height": 1080 },
"scale": { "width": 640, "height": 360 }
},
"events": [
{
"name": "analytics_event_A",
"target_fps": 5.0,
"batch_size": 10
}
]
}
]
}- processing: Defines the crop region and target scale resolution.
- events: Defines logical events associated with the stream.
batch_size: Number of frames to accumulate before publishing to Redis.target_fps: (Logic handled by StreamIO) Target frame rate for this event.
Data is published to Redis Streams using the key pattern: <FP_REDIS_QUEUE_BASE_NAME>:<stream_id>.
Each entry contains:
- event: The event name (e.g., "analytics_event_A").
- data: A binary blob containing the batched frames.
The data blob consists of concatenated frames. Each frame in the batch has the following layout:
- Header (16 bytes):
struct FrameHeader { int64_t pts; // Presentation Timestamp uint32_t width; // Frame width uint32_t height; // Frame height };
- Y Plane:
width * heightbytes. - U Plane:
(width/2) * (height/2)bytes. - V Plane:
(width/2) * (height/2)bytes.
Note: The image data is tightly packed (no stride/padding) in YUV420P format.
This project is licensed under the MIT License - see the LICENSE file for details.