diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index 828a873..b9bad11 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -28,6 +28,14 @@ on: required: false default: "12" type: string + platform: + description: "GitHub-hosted runner label for the bench job" + required: false + default: "ubuntu-latest" + type: choice + options: + - ubuntu-latest + - ubuntu-24.04-arm # Bench is on-demand: each workflow_dispatch run is independent and # parallel-safe (different `runs` get different run IDs and thus @@ -41,7 +49,7 @@ permissions: jobs: bench: - runs-on: ubuntu-latest + runs-on: ${{ inputs.platform }} timeout-minutes: 60 steps: - uses: actions/checkout@v4 @@ -126,6 +134,7 @@ jobs: env: BENCH_MODELS: ${{ steps.variants.outputs.BENCH_MODELS }} BENCH_VARIANT_INPUT: ${{ inputs.variant }} + BENCH_PLATFORM: ${{ inputs.platform }} run: | set -euo pipefail provenance_path=benchmark_provenance.txt @@ -138,7 +147,7 @@ jobs: printf 'models=%s\n' "$BENCH_MODELS" printf 'image_path=assets/tench.jpg\n' printf 'build_configuration=Release\n' - printf 'platform=ubuntu-latest\n' + printf 'platform=%s\n' "$BENCH_PLATFORM" printf 'quantization=f16 only\n' } > "$provenance_path" variant_list="${BENCH_MODELS//,/ }" @@ -164,17 +173,20 @@ jobs: BENCH_MODELS: ${{ steps.variants.outputs.BENCH_MODELS }} BENCH_REPEATS: ${{ inputs.repeats }} BENCH_THREADS: ${{ inputs.threads }} + BENCH_PLATFORM: ${{ inputs.platform }} BENCH_BUILD_CONFIG: Release run: | set -euo pipefail echo "models selected: $BENCH_MODELS" echo "repeats: $BENCH_REPEATS" echo "threads: $BENCH_THREADS" + echo "platform: $BENCH_PLATFORM" chmod +x scripts/bench.sh bash scripts/bench.sh \ --models "$BENCH_MODELS" \ --repeats "$BENCH_REPEATS" \ --threads "$BENCH_THREADS" \ + --platform "$BENCH_PLATFORM" \ --image assets/tench.jpg \ --bin build/bin/dinov2-cli \ --out benchmark_results.txt diff --git a/README.md b/README.md index b8de8b1..d56ca1a 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,7 @@ each input gets one JSON line on stdout, identical to running it alone ./bin/dinov2-cli -m models/model.gguf -i a.jpg -i b.jpg --batch 2 --print-embeddings ``` -That's it. Up to **3x faster than PyTorch on CPU** with up to **4x less memory** (see [docs/benchmarks.md](docs/benchmarks.md)). +That's it. Up to **3x faster than PyTorch on CPU** with up to **4x less memory** (i9-14900HX, see [benchmarks methodology](docs/benchmarks.md)). ## C API @@ -138,6 +138,11 @@ Opt-in surfaces, off by default and not covered by the stability contract | **PyTorch-parity outputs** | CLS + patch embeddings as JSON, matching the reference implementation. | | **Cross-platform prebuilts** | macOS arm64, Linux x64/arm64, Windows x64. | +**Limitations:** image decode is vendored stb_image only (JPEG, PNG, BMP, +TGA; no HEIC, WebP, RAW, or AVIF, convert those first with `sips`, +`heif-convert`, or `ffmpeg`). Embeddings are image-to-image: this is not +CLIP and there are no text queries. Classification is ImageNet-1k only. + ## Pre-converted GGUF weights Ready-to-download f16 GGUF weights, published by CI to the [`dinov2-cpp-core`](https://huggingface.co/dinov2-cpp-core) Hugging Face profile: diff --git a/benchmark_results.txt b/benchmark_results.txt index cfd8a04..043dbf4 100644 --- a/benchmark_results.txt +++ b/benchmark_results.txt @@ -1,3 +1,4 @@ +# ===== platform: ubuntu-latest ===== # Generated by scripts/bench.sh on 2026-09-21T17:14:58Z. Actual Ubuntu workflow evidence. # Platform: ubuntu-latest (GitHub-hosted runner), reported runner: linux-x86_64 # Git SHA: 9d0a6265ae53088e423e4c2417ec8589de181e99 @@ -14,3 +15,21 @@ # Measurement: forward-pass wall time from dinov2-cli; image load, preprocessing, classification, and PCA are excluded. # Unmeasured: register-token variants, base/large/giant sizes, non-Ubuntu platforms, GPU backends, and quantization comparisons. + +# ===== platform: ubuntu-24.04-arm ===== +# Generated by scripts/bench.sh on 2026-09-22T22:59:52Z. Actual Ubuntu ARM64 workflow evidence. +# Platform: ubuntu-24.04-arm (GitHub-hosted runner), reported runner: linux-aarch64 +# Git SHA: 1f5c16148f75fc3b75fe5ff964943d406f3bfd10 +# Build configuration: Release +# Precision: f16 GGUF; backend: CPU; batch: 1 +# Image path: assets/tench.jpg +# Methodology: 5 timed runs after 1 warmup; threads: 4. +# Command: scripts/bench.sh --models small --repeats 5 --threads 4 --platform ubuntu-24.04-arm --image assets/tench.jpg --bin build/bin/dinov2-cli --out benchmark_results.txt +# Workflow run: https://github.com/espetro/dinov2.cpp/actions/runs/35794974726 + +| Model | GGUF path | GGUF SHA256 | mean (ms) | stddev (ms) | min (ms) | max (ms) | peak RSS (MB) | +|:------|:----------|:------------|----------:|------------:|---------:|---------:|---------------:| +| small | models/dinov2-vit-small-patch14/model.gguf | b7ca009aa416f6be85ea95363f4ef13d3c139999f96ca3fd47226849afb712e6 | 133.8 | 8.2 | 128 | 148 | 96 | + +# Measurement: forward-pass wall time from dinov2-cli; image load, preprocessing, classification, and PCA are excluded. +# Unmeasured: register-token variants, base/large/giant sizes, non-Ubuntu platforms, GPU backends, and quantization comparisons. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 8152ce1..7043e4d 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -1,20 +1,28 @@ # Benchmarks -The current committed benchmark evidence is Ubuntu-only, CPU-only, and generated by -`.github/workflows/bench.yml` on a GitHub-hosted `ubuntu-latest` runner. It is -not a multi-platform performance claim. No quantization comparison is included. +The current committed benchmark evidence is Ubuntu-only (x64 and arm64), +CPU-only, and generated by `.github/workflows/bench.yml` on GitHub-hosted +`ubuntu-latest` and `ubuntu-24.04-arm` runners. It is not a multi-platform +performance claim. No quantization comparison is included. ## Current evidence -The following row is an actual measurement from workflow run -[35630688827](https://github.com/espetro/dinov2.cpp/actions/runs/35630688827), -not a seed or estimate. +The following rows are actual measurements from workflow runs +[35630688827](https://github.com/espetro/dinov2.cpp/actions/runs/35630688827) +and +[35794974726](https://github.com/espetro/dinov2.cpp/actions/runs/35794974726), +not seeds or estimates. -| Model | GGUF | Precision | Backend | Mean (ms) | Stddev (ms) | Min (ms) | Max (ms) | Peak RSS (MB) | -|:------|:-----|:----------|:--------|----------:|------------:|---------:|---------:|---------------:| -| small | `dinov2-cpp-core/dinov2-small-gguf` | f16 | CPU | 255.0 | 9.1 | 249 | 271 | 104 | +| Model | Platform | GGUF | Precision | Backend | Mean (ms) | Stddev (ms) | Min (ms) | Max (ms) | Peak RSS (MB) | +|:------|:---------|:-----|:----------|:--------|----------:|------------:|---------:|---------:|---------------:| +| small | `ubuntu-latest` (x86_64) | `dinov2-cpp-core/dinov2-small-gguf` | f16 | CPU | 255.0 | 9.1 | 249 | 271 | 104 | +| small | `ubuntu-24.04-arm` (aarch64) | `dinov2-cpp-core/dinov2-small-gguf` | f16 | CPU | 133.8 | 8.2 | 128 | 148 | 96 | -### Provenance +The two rows use different thread counts (12 on x64, 4 on arm64, matching the +runner CPU counts) and different microarchitectures; they are not directly +comparable as an x64-versus-arm64 claim. + +### Provenance: `ubuntu-latest` row - Date: 2026-09-21T17:14:58Z - Platform: `ubuntu-latest`, reported by the runner as `linux-x86_64` @@ -30,6 +38,25 @@ not a seed or estimate. - Timed runs: 5 - Measurement: forward-pass wall time from `dinov2-cli`; image loading, preprocessing, classification, and PCA are excluded - Command: `scripts/bench.sh --models small --repeats 5 --threads 12 --image assets/tench.jpg --bin build/bin/dinov2-cli --out benchmark_results.txt` +- Workflow run: [35630688827](https://github.com/espetro/dinov2.cpp/actions/runs/35630688827) + +### Provenance: `ubuntu-24.04-arm` row + +- Date: 2026-09-22T22:59:52Z +- Platform: `ubuntu-24.04-arm`, reported by the runner as `linux-aarch64` +- Repository SHA: `1f5c16148f75fc3b75fe5ff964943d406f3bfd10` +- GGUF path: `models/dinov2-vit-small-patch14/model.gguf` +- GGUF SHA-256: `b7ca009aa416f6be85ea95363f4ef13d3c139999f96ca3fd47226849afb712e6` +- Image: `assets/tench.jpg` +- Precision: f16 GGUF +- Backend: CPU +- Threads: 4 +- Batch: 1 +- Warmup: 1 +- Timed runs: 5 +- Measurement: forward-pass wall time from `dinov2-cli`; image loading, preprocessing, classification, and PCA are excluded +- Command: `scripts/bench.sh --models small --repeats 5 --threads 4 --platform ubuntu-24.04-arm --image assets/tench.jpg --bin build/bin/dinov2-cli --out benchmark_results.txt` +- Workflow run: [35794974726](https://github.com/espetro/dinov2.cpp/actions/runs/35794974726) The runner's exact result and checksum provenance are retained in the workflow artifact alongside the committed result. Runner scheduling and CPU frequency @@ -47,6 +74,9 @@ gh workflow run bench.yml --ref feat/batched-inference \ -f variant=small -f repeats=5 -f threads=12 ``` +Add `-f platform=ubuntu-24.04-arm` to run the same job on the ARM64 runner +(the committed arm64 row above used `-f threads=4` to match its 4 CPUs). + The workflow supports the regular `small`, `base`, `large`, and `giant` models and their explicitly named `small-registers`, `base-registers`, `large-registers`, and `giant-registers` variants. `all` currently means the diff --git a/examples/dedup/README.md b/examples/dedup/README.md index eab6b58..523e46f 100644 --- a/examples/dedup/README.md +++ b/examples/dedup/README.md @@ -28,6 +28,15 @@ open dupes.html grouping merely similar images. Tune on a small folder first. - `-o/--out`: where the HTML review page goes (default `dedup-report.html`). +## Supported formats + +The CLI decodes images with vendored stb_image, so only `.jpg`, `.jpeg`, +`.png`, `.bmp`, and `.tga` are scanned. WebP, HEIC, RAW, and AVIF files are +skipped by `dedup.py` and would fail decode in `dinov2-cli` anyway; convert +them first. On macOS, `sips -s format jpeg in.heic --out out.jpg` covers +HEIC/WebP/RAW; elsewhere `heif-convert in.heic out.jpg` (libheif) or +`ffmpeg -i in.webp out.jpg` do the same. + The script uses `pooled` (`[cls || mean(patches)]`) when present and falls back to `cls`. For large folders the pairwise pass is O(n^2) in Python; a few thousand images is fine, hundreds of thousands will want a real index diff --git a/examples/dedup/dedup.py b/examples/dedup/dedup.py index 0455fa0..4b4c3df 100644 --- a/examples/dedup/dedup.py +++ b/examples/dedup/dedup.py @@ -20,7 +20,9 @@ import sys from pathlib import Path -EXTS = {".jpg", ".jpeg", ".png", ".bmp", ".tga", ".webp"} +# Formats decodable by vendored stb_image (v2.30). WebP, HEIC, RAW, and AVIF +# are not supported: convert them first (sips, heif-convert, ffmpeg). +EXTS = {".jpg", ".jpeg", ".png", ".bmp", ".tga"} def embed(cli: str, model: str, images: list[Path]) -> list[dict]: