FormulIA1 is a modular Computer Vision pipeline designed to extract physical trajectories and structural insights from Formula 1 broadcast video feeds. It leverages instance segmentation, restricted ego-motion homography, and optional monocular depth estimation to perform highly stable, centroid-based tracking and motion trail visualization on fast-paced racing footage.
Note
Work In Progress (v0.01): This project is in its very early stages. While the core tracking architecture is there, the inference results are far from perfect, and the team classifier is currently even worse.
I am actively working on building better datasets, manually labeling more edge cases, and refining the overall logic.
See the Roadmap & TODO section below for upcoming features.
- Broadcast-Style HUD: Replaces standard bounding boxes with sleek, dynamic telemetry-style overlays showing team names and tracker IDs.
- Instance Segmentation Tracking: Utilizes YOLOv8-seg for pixel-accurate vehicle and track masking, eliminating bounding box jitter.
- Restricted Ego-Motion Stabilization: Computes robust visual odometry by isolating optical flow calculations exclusively to the semantic road surface.
- Monocular Depth-SLAM: Integrates Depth Anything V2 for experimental 3D camera space unprojection and rigid-body transformation tracking.
- Smart Re-Identification (ReID): Custom anti-occlusion merging logic that validates geographical proximity and team livery to reconnect broken trajectories.
- Cascade Classification Architecture: Isolates tracking and team identification into a two-stage process for optimized inference.
- Automated Data Engineering: Includes utilities integrating SegFormer and Segment Anything Model (SAM) for automated road and vehicle dataset labeling.
Requires Python 3.10+ and a CUDA-capable GPU for real-time inference.
git clone https://github.com/Moloshow/formulia1.git
cd formulia1
# Install dependencies
pip install -r requirements.txtRather than training a single YOLO model to detect 10 different F1 teams simultaneously, FormulIA1 decouples the problem into two models:
- Model A (Segmentation/Detection): Trained solely on a single class (
f1_car). It learns the structural shape of a Formula 1 car and is highly robust to scale, blur, and lighting. - Model B (Classification): A lightweight classifier that analyzes tight crops of the detected cars to identify the specific team (e.g., Ferrari, Mercedes).
- The "Why": Scalability and maintainability. When a new F1 season starts with new car liveries, only the lightweight classifier (Model B) needs fine-tuning on a small set of images. You completely avoid the massive cost of re-annotating tens of thousands of complex bounding boxes or polygons for the detection model.
Manual pixel annotation for semantic segmentation is extremely time-consuming. To achieve pixel-perfect tracking and ego-motion masking without manual labor, this pipeline relies on Data Engineering via Foundation Models:
- NVIDIA SegFormer is used zero-shot to extract the drivable surface (the track) mask, allowing optical flow to ignore TV HUDs and grandstands.
- Meta SAM (Segment Anything Model) is used to automatically upgrade coarse bounding boxes into precise F1 car polygons.
This project relies on a cascading data engineering pipeline to transform a basic object detection dataset into complex instance segmentation and classification datasets.
The Source Dataset (f1_generic)
- Origin: F1 Car Recognition 2 by Yoav Fogel.
- License: CC BY 4.0
- Structure: Standard YOLO Object Detection format (Bounding Boxes).
- Classes: 10 classes corresponding to the F1 teams (Ferrari, Mercedes, Red-Bull, etc.).
Generated Datasets (Automated via tools/)
Running the data engineering scripts transforms f1_generic into three distinct, specialized datasets:
f1_detection: A pure bounding-box dataset where all 10 team classes are collapsed into a single0: f1_carclass. Used as the spatial prompt for SAM.f1_classification: An image classification dataset (ImageNet folder structure) containing only tight crops of the cars, categorized by team. Used to train the Team Classifier model.f1_segmentation: The final dataset. Upgrades thef1_detectionboxes into pixel-perfect polygons using SAM, and adds a1: roadsemantic mask using SegFormer. Used to train the main YOLOv8-seg model.
The pipeline relies on custom-trained YOLO models. You must first generate the datasets and train these models using the provided scripts (see Usage section), which will output the weights to the following default directories. You can also override these paths using CLI arguments if you store your weights elsewhere.
- Segmentation Model:
runs/segment/f1_road_segmentation/weights/best.pt - Classification Model:
runs/classify/f1_team_classifier/weights/best.pt - Depth Model: Automatically fetched from Hugging Face (
depth-anything/Depth-Anything-V2-Small-hf)
Run the main orchestrator on a video source.
Standard tracking using semantic segmentation:
python run_pipeline.py --source videos/sample_f1.mp4 --output outputs/result.mp4Enable the cascade Team Classifier to identify specific F1 teams:
python run_pipeline.py --source videos/sample_f1.mp4 --output outputs/result.mp4 --classifyEnable experimental 3D Depth-SLAM tracking:
python run_pipeline.py --source videos/sample_f1.mp4 --depth-slamDebug mode (visualize semantic road mask and freeze edge tracking):
python run_pipeline.py --source videos/sample_f1.mp4 --show-mask --freeze-edgesUse classic YOLO bounding boxes instead of the broadcast-style HUD:
python run_pipeline.py --source videos/sample_f1.mp4 --classic-bboxTo avoid manual pixel annotation, this project relies on an automated dataset generation pipeline.
Auto-label segmentation masks from raw bounding box datasets: Context: Fuses SegFormer (to extract the racing track plane) and SAM (to extract perfect vehicle silhouettes from coarse bounding boxes) into YOLO segmentation labels.
python tools/auto_label_road.py --input-dir datasets/f1_detection --output-dir datasets/f1_segmentationVerify and render generated semantic masks:
Context: Debugging utility that reads YOLO segmentation .txt labels and overlays them as colored polygons on the original images to ensure the automated labeling process succeeded.
python tools/visualize_segmentation.py --dataset-dir datasets/f1_segmentationPrepare Two-Stage Cascade datasets: Context: To train the Team Classifier, we must extract tightly cropped images of the F1 cars. This script splits a generic detection dataset into two parts: a pure detection dataset and a folder structure of cropped cars ready for image classification training.
python tools/prepare_datasets.py --base-dir datasets/f1_generic --det-dir datasets/f1_detection --cls-dir datasets/f1_classificationTrain the YOLOv8 instance segmentation model:
python training/train_segmentation.py --model yolov8m-seg.pt --data datasets/f1_segmentation/data.yaml --epochs 30 --batch 8Train the YOLOv8 image classification model (Team Identifier):
python training/train_classifier.py --model yolov8n-cls.pt --data-dir datasets/f1_classification --epochs 20 --batch 16Run the automated test suite:
pytestformulia1/
├── run_pipeline.py
├── README.md
├── tests/
│ └── test_run_pipeline.py
├── tools/
│ ├── auto_label_road.py
│ ├── prepare_datasets.py
│ └── visualize_segmentation.py
└── training/
├── evaluate_model.py
├── train_classifier.py
└── train_segmentation.py
Transparency on technical debt and current system boundaries:
- Optical Flow Instability: Fast whip-pans with heavy motion blur can cause the homography matrix to degrade if too few trackable corners are found on the road surface.
- Auto-Labeling Artifacts: The SAM auto-labeling pipeline struggles when F1 cars are heavily occluded by fences or other cars, sometimes merging two cars into a single polygon.
- Track Surface Segmentation: The current zero-shot road extraction using Cityscapes-trained SegFormer is not perfectly adapted to F1 circuits (struggles with kerbs and runoff areas). Future improvements require manual labeling or sourcing a dedicated GP circuit dataset.
- Dataset Temporal Bias: The current
f1_genericdataset is based on the 2024 F1 season. Because team liveries and sponsors change annually, the Team Classifier will suffer severe accuracy drops if run on broadcast videos from previous or future seasons. Generalization may also drop in severe weather conditions (e.g., heavy rain spray) or low-light night races.
- Phase 1: Stable Segmentation Tracking: Mask-centroid tracking and restricted ego-motion to eliminate bounding-box jitter.
- Phase 2: Dataset & Labeling Refinement: Expand the dataset to include multiple F1 seasons (to resolve temporal bias) and manually label complex track surfaces (kerbs/run-offs) to fix zero-shot segmentation artifacts.
- Phase 3: Edge-Case Tracking Logic: Implemented Smart ReID fallback mechanisms for tracking occlusions. Still need to handle fast whip-pan optical flow failures and refine mask-merging artifacts to improve overall tracker stability.
- Phase 4: Radar Mini-Map: Projection of
$(cx, cy)$ pixel coordinates onto a 2D satellite orthophoto using multi-point homography registration. - Phase 5: Telemetry Sync: Integration with the
FastF1API via Optical Character Recognition (OCR) syncing to overlay real-time throttle/brake telemetry on the tracked vehicles.
