A Unified Deep Learning Framework for Image Quality Assessment (IQA) and Video Quality Assessment (VQA).
This framework provides an end-to-end solution for training, evaluating, and deploying quality assessment models. It features a unified architecture that seamlessly handles both image and video inputs, multi-dataset support, cross-validation pipelines, and production-ready inference APIs.
Note
This framework is primarily tested on AutoDL cloud GPU instances. You can run the command below to open proxy on AutoDL cloud instances:
source /etc/network_turbo- Architecture & Design Decisions
- Model Architecture
- Training Pipeline
- Evaluation & Metrics
- Deployment & Inference API
- Project Main Structure
- System Overview
- Configuration Guide
- Troubleshooting
- Dependency Security
- License
- Contributors
- Acknowledgements
The framework implements a dimension-aware routing system that automatically switches between image (4D tensors) and video (5D tensors) processing modes.
Key Design Decisions:
| Decision | Implementation | Rationale |
|---|---|---|
| Unified Model | A single IQAVQANet handling both 4D and 5D inputs |
Eliminates code duplication; ensures consistency in quality metrics |
| Flexible Backbone | Swin-T / ResNet50 with automatic feature adaptation | Balances accuracy against GPU memory consumption |
| Temporal Fusion | Transformer encoder for video frame aggregation | Captures long-range dependencies between frames |
| Task-Aware Loss | MSE + Rank + PLCC, reweighted based on task_type |
Simultaneously optimizes absolute prediction and relative ranking |
| Multi-dataset Support | YAML-based configuration and Factory pattern | Allows easy addition of new datasets without code modification |
| Config & Path Management | Pydantic-based Config.paths and typed methods |
Single source of truth; eliminates hard-coding; enables dataset-isolated storage |
| File Index Resolution | CaseInsensitiveAssetResolver pre-builds a case-insensitive index |
Reduces O(n) directory traversal to O(1) memory lookup; resolves parsing failures caused by filename case inconsistencies |
| Model Serving | FastAPI + secure loading via weights_only=True |
Decouples training from inference; provides standardized HTTP interfaces; ensures secure model loading |
# Architecture overview
Input (4D: [B,3,H,W] or 5D: [B,F,3,H,W])
↓
Backbone (Swin-T / ResNet50)
↓
Spatial Pooling (AdaptiveAvgPool2d)
↓
[Temporal Fusion] ← TransformerEncoder (only for video)
↓
Quality Head (3-layer MLP + Sigmoid)
↓
Output: Quality Score (0-1 range)| Backbone | Parameters | IQA | VQA | Memory (per sample) |
|---|---|---|---|---|
| ResNet50 | 25M | ✅ | ✅ | ~2GB (8 frames) |
| Swin-T | 28M | ✅ | ✅ | ~4GB (8 frames) |
IQAVQALoss combines three components, weighted differently depending on task_type (iqa vs vqa):
Total Loss = w_mse × MSE + w_rank × RankLoss + w_plcc × (1 − PLCC)
| Task | MSE | Rank | PLCC |
|---|---|---|---|
IQA (resnet_iqa) |
0.7 | 0.3 | 0.0 |
VQA (timeswin_vqa) |
0.4 | 0.3 | 0.3 |
- MSE Loss: Absolute prediction accuracy
- Rank Loss: Pairwise ranking consistency (sampled, capped at
max_pairspairs) - PLCC Loss:
1 − Pearson correlation, weighted in for VQA to directly optimize linear alignment with human MOS
# Initialize environment and install dependencies
make setup
# Check environment status
make info
# Download datasets, unrar and make symbol links
make dataRun training directly with uv:
# TID2013 (Image Quality Assessment)
uv run python -m src.main --model resnet_iqa --dataset tid2013
# KoNViD-1k (Video Quality Assessment)
uv run python -m src.main --model timeswin_vqa --dataset konvid-1k
# T2VQA-DB (Text-to-Video Quality Assessment)
uv run python -m src.main --model timeswin_vqa --dataset t2vqa-dbNote: By default, DEBUG=0 is applied in make commands. You can override it by appending DEBUG=1 if needed.
Note
Only two model configs ship today: resnet_iqa (image/ResNet50) and timeswin_vqa (video/Swin-T). Model configs are auto-discovered from config/models/*.yaml — drop a new YAML there (e.g. resnet_vqa.yaml) to register another combination before referencing it in commands.
You can extend the framework capabilities using the following training and debugging modes:
| Mode | Use Case | uv / Shell Command |
|---|---|---|
| Smoke Test | Quick functionality check | uv run python -m src.main --smoke_test |
| Debug Mode | Enable breakpoints & verbose logs | LOG_LEVEL=DEBUG uv run python -m src.main |
| Background | Run on remote server persistently | nohup uv run python -m src.main > results/scripts_logs/train.log 2>&1 & |
Tip
Monitor real-time training progress with:
tail -f results/scripts_logs/train.log| Metric | Full Name | Interpretation |
|---|---|---|
| PLCC | Pearson Linear Correlation Coefficient | Linear relationship (accuracy) |
| SROCC | Spearman Rank Order Correlation Coefficient | Monotonic relationship (ranking) |
| KROCC | Kendall Rank Correlation Coefficient | Ordinal agreement |
| RMSE | Root Mean Square Error | Prediction error magnitude |
| R² | Coefficient of Determination | Variance explained |
The framework automatically generates:
-
Training History: Loss curves, PLCC/SROCC progression
-
Residual Analysis: Scatter plots, error distribution
-
Cross-Model Comparison: Bar charts for multiple models
Output location: results/{dataset}/plots/
A standalone FastAPI service (deploy/api.py) exposes trained checkpoints for inference, decoupled from the training stack. It loads one IQA model and one VQA model at startup and serves a unified 0-5 MOS scale regardless of which model answers the request.
deploy/
├── api.py # FastAPI service (this file)
├── infer.py # Preprocessing + checkpoint loading + prediction helpers
├── iqa-models/
│ └── tid2013_best.pt # Default IQA checkpoint (resnet_iqa, trained on TID2013)
└── vqa-models/
└── konvid_best.pt # Default VQA checkpoint (timeswin_vqa, trained on KoNViD-1k)
Warning
Checkpoint paths are resolved relative to api.py's own location (Path(__file__).resolve().parent / "iqa-models" / ...), so .pt files must sit in deploy/iqa-models/ and deploy/vqa-models/ — not in results/model_outputs/.
cd deploy
uv run python -m deploy.apiThe service listens on 0.0.0.0:8000 and loads both checkpoints eagerly on startup; if neither model file is found, startup fails with RuntimeError: 没有成功加载任何模型，服务启动失败.
| Endpoint | Method | Purpose |
|---|---|---|
/health |
GET | Returns loaded model IDs and inference device |
/evaluate |
POST | Runs inference on an uploaded image/video and returns a unified MOS score |
/evaluate accepts multipart/form-data with:
| Field | Type | Notes |
|---|---|---|
file |
file | Image or video to score |
media_type |
string | "image" or "video" |
task_type |
string | Echoed back in the response, not used for routing |
model |
string | "resnet_iqa" or "timeswin_vqa" — selects which cached model handles the request |
Cross-media inference is supported in both directions: resnet_iqa averages predictions across sampled frames when given a video, and timeswin_vqa expands a single image into a pseudo-video (image_to_video_tensor) when given an image.
The response includes both mos (unified 0–5 scale, raw_score × 5) and dataset_mos (denormalized back to the source dataset's original MOS range, for debugging).
Note
MOS range (mos_min/mos_max) is stored in the checkpoint's config and restored on load for checkpoints saved with v0.5.0+. Denormalization is automatic.
Tip
The dataset_mos field returns scores in the original dataset's MOS scale (e.g., TID2013: 0-9, KoNViD-1k: 1-5), while mos is always normalized to 0-5 for cross-dataset comparison.
deep-vqa-framework/
├── Makefile # Automation & workflow commands
├── README.md # Project overview
├── DISCLAIMER.md # Legal liability & resource usage policy
├── pyproject.toml # Dependency & environment management (uv)
│
├── config/ # YAML configuration files (user-editable)
│ ├── basic.yaml # System & training global defaults
│ ├── dataset_config.yaml # Dataset-specific metadata
│ └── models/ # Model architecture parameters
│
├── datasets/ # Data storage & symlink routing
│ ├── KoNViD-1k/ # Video quality dataset
│ ├── T2VQA-DB/ # Text-to-Video QA dataset
│ └── TID2013/ # Image quality dataset
│
├── docs/ # Interactive architecture & manuals
│ ├── pipeline.html # System execution & module flow
│ └── Cloud_Platform_Rental_Guide.md
|
├── reports/ # Security reports from pip-audit, SBOM, and safety
|
├── results/
| ├── {dataset}/
│ │ ├── train_logs/ # Training history, CSV logs
│ │ ├── plots/ # Loss curves, residual plots
│ │ ├── model_outputs/ # Checkpoints (.pt files)
│ │ └── corrupted/ # Corrupted files from integrity check
│ └── scripts_logs/ # Shell script logs (setup, data, etc.)
|
├── scripts/ # Infrastructure automation
│ ├── manage_data.sh # Download & data preparation
│ ├── setup_env.sh # Environment & system initialization
│ ├── archive_results.sh # Package results
│ └── *.sh # Auxiliary maintenance & cleanup scripts
│
├── deploy/ # Standalone inference service (decoupled from training)
│ ├── api.py # FastAPI service — see note below on module naming
│ ├── infer.py # Preprocessing + checkpoint loading + prediction
│ ├── iqa-models/ # IQA .pt checkpoints served by api.py
│ └── vqa-models/ # VQA .pt checkpoints served by api.py
│
└── src/ # Core framework logic
├── main.py # Global execution entry point
├── core/ # Training engine & evaluation pipeline
├── data/ # Data loaders, EDA & integrity analysis
├── models/ # Architecture definitions (IQAVQA-Net)
├── utils/ # Configuration, logging & path management
└── config/ # Pydantic config system (code)
To help you quickly grasp the system architecture and execution flow, we provide an interactive pipeline visualization.
→ Open Interactive Architecture Map
This map illustrates:
- How data flows through each stage of the system
- Key components and their interactions
- Execution order of the entire workflow
Charts are rendered using Mermaid and support interactive viewing in the browser.
Configuration is assembled by load_config() which returns a Pydantic Config object.
All settings are validated at load time with type checking.
Paths are resolved via cfg.paths.xxx_dir(dataset_name) methods.
| Stage | File | How it's merged |
|---|---|---|
| 1 (Base) | basic.yaml |
Loaded first as the starting config |
| 2 (Model) | models/{model}.yaml |
Deep-merged on top of stage 1 (matching keys override) |
| 3 (Dataset) | dataset_config.yaml |
Not merged into top-level keys — the matched dataset entry is attached wholesale as config["dataset_info"] |
The merged result is validated against config_loader.validate_config_schema()'s required-field list before training starts. Path resolution is handled by Pydantic PathsConfig with typed methods.
When OOM occurs, adjust the following parameters in your model's YAML config (e.g., config/models/timeswin_vqa.yaml):
Reduce memory consumption:
- Lower
preprocessing.batch_size— reduces per-batch memory - Reduce
model.num_frames— fewer video frames to process - Switch to a lighter
model.backbone(e.g.,resnet50instead ofswin_t) - Lower
model.transformer_layers— shallower temporal fusion
Compensate for smaller batch size:
- Increase
train.gradient_accumulation_steps— maintains effective batch size =batch_size × gradient_accumulation_steps - Ensure
amp: true— mixed precision significantly reduces memory
Note
Gradient checkpointing is not currently implemented in IQAVQANet — don't set gradient_checkpointing: true in configs yet; it has no effect.
Warning
On AutoDL or similar cloud GPU instances, OpenCV's VideoCapture may fail due to missing system dependencies.
Solution: Use Decord
uv add decordDecord is pre-configured as the default backend. If Decord is not available, the framework automatically falls back to OpenCV, but on AutoDL this fallback may fail. Always use Decord for video training on AutoDL.
| Issue | Optimization |
|---|---|
| Data loading bottleneck | Increase num_workers: 8 |
| Small batch size | Use gradient_accumulation_steps |
| Video decoding slow | Ensure Decord is installed |
results/{dataset}/corrupted/ stores files moved aside by DataEDA.check_integrity().
- Clear manually if no longer needed
- Or add a cleanup step to
scripts/cache_clean.sh
Tip
These files are corrupted media files isolated during integrity checks — safe to delete if you don't need them for debugging.
If a nohup/background training run appears frozen with no new log lines, check for pdb.set_trace() breakpoints in main.py or engine.py. These are triggered on exceptions and will wait for stdin input, blocking non-interactive processes.
Solutions:
- Run training interactively (without
nohup) when debugging - Check
results/{dataset}/train_logs/*.logfor error details - Kill the process and fix the underlying issue before retrying
The framework includes security tools to audit dependencies:
| Command | Purpose |
|---|---|
make vuln-audit |
Scan dependencies for known vulnerabilities |
make sbom |
Generate Software Bill of Materials (CycloneDX) |
make safety |
Check dependencies with Safety (legacy, requires login) |
make security-all |
Run all security checks |
Note
pip-audit is the primary vulnerability scanner. safety requires registration or login.
- Framework: MIT
- Author: @autentisitet
- Version: 0.5.1-beta (pre-release)
| Name | Role | Contributions |
|---|---|---|
| @autentisitet | Project Lead / Core Developer | Framework architecture, training pipeline, inference engine, deployment API |
| @yss0120 | Frontend Developer | Interactive UI/UX (index.html), subjective blind rating system, quality passport visualization |
| @Zed-23 | DevOps & Quality Assurance | CI/CD pipeline, automated & smoke testing, Shell script fixes, CUDA OOM debugging |
| @bazhina-5566 | Backend API Developer | FastAPI service (deploy/api.py), model checkpoint loading, inference API design |
Note
We welcome contributions! Please see CONTRIBUTING.md for guidelines.
- PyTorch team for deep learning framework
- Decord developers for efficient video loading
- FastAPI for the production-ready API framework
- TID2013, KoNViD-1k, T2VQA-DB dataset providers
For details regarding third-party tool usage, dataset compliance, and resource usage, please refer to the DISCLAIMER.md file.
For detailed contribution guidelines and issue reporting, please check the .github folder.
Built with ❤️ for the research community