Skip to content

benchmarks.md

Latest commit

 

History

History
132 lines (108 loc) · 6.09 KB

benchmarks.md

File metadata and controls

132 lines (108 loc) · 6.09 KB
Raw

Benchmarks

The current committed benchmark evidence is Ubuntu-only (x64 and arm64), CPU-only, and generated by .github/workflows/bench.yml on GitHub-hosted ubuntu-latest and ubuntu-24.04-arm runners. It is not a multi-platform performance claim. No quantization comparison is included.

Current evidence

The following rows are actual measurements from workflow runs 35630688827 and 35794974726, not seeds or estimates.

Model Platform GGUF Precision Backend Mean (ms) Stddev (ms) Min (ms) Max (ms) Peak RSS (MB)
small ubuntu-latest (x86_64) dinov2-cpp-core/dinov2-small-gguf f16 CPU 255.0 9.1 249 271 104
small ubuntu-24.04-arm (aarch64) dinov2-cpp-core/dinov2-small-gguf f16 CPU 133.8 8.2 128 148 96

The two rows use different thread counts (12 on x64, 4 on arm64, matching the runner CPU counts) and different microarchitectures; they are not directly comparable as an x64-versus-arm64 claim.

Provenance: ubuntu-latest row

  • Date: 2026-09-21T17:14:58Z
  • Platform: ubuntu-latest, reported by the runner as linux-x86_64
  • Repository SHA: 9d0a6265ae53088e423e4c2417ec8589de181e99
  • GGUF path: models/dinov2-vit-small-patch14/model.gguf
  • GGUF SHA-256: b7ca009aa416f6be85ea95363f4ef13d3c139999f96ca3fd47226849afb712e6
  • Image: assets/tench.jpg
  • Precision: f16 GGUF
  • Backend: CPU
  • Threads: 12
  • Batch: 1
  • Warmup: 1
  • Timed runs: 5
  • Measurement: forward-pass wall time from dinov2-cli; image loading, preprocessing, classification, and PCA are excluded
  • Command: scripts/bench.sh --models small --repeats 5 --threads 12 --image assets/tench.jpg --bin build/bin/dinov2-cli --out benchmark_results.txt
  • Workflow run: 35630688827

Provenance: ubuntu-24.04-arm row

  • Date: 2026-09-22T22:59:52Z
  • Platform: ubuntu-24.04-arm, reported by the runner as linux-aarch64
  • Repository SHA: 1f5c16148f75fc3b75fe5ff964943d406f3bfd10
  • GGUF path: models/dinov2-vit-small-patch14/model.gguf
  • GGUF SHA-256: b7ca009aa416f6be85ea95363f4ef13d3c139999f96ca3fd47226849afb712e6
  • Image: assets/tench.jpg
  • Precision: f16 GGUF
  • Backend: CPU
  • Threads: 4
  • Batch: 1
  • Warmup: 1
  • Timed runs: 5
  • Measurement: forward-pass wall time from dinov2-cli; image loading, preprocessing, classification, and PCA are excluded
  • Command: scripts/bench.sh --models small --repeats 5 --threads 4 --platform ubuntu-24.04-arm --image assets/tench.jpg --bin build/bin/dinov2-cli --out benchmark_results.txt
  • Workflow run: 35794974726

The runner's exact result and checksum provenance are retained in the workflow artifact alongside the committed result. Runner scheduling and CPU frequency can vary, so this result is a reproducibility point rather than a universal hardware claim.

Reproduce on Ubuntu

The workflow is the canonical reproducible path. It downloads the published f16 GGUF, builds the Release CLI, runs the script, and uploads a small, diffable result plus provenance file. Trigger it with the GitHub CLI:

gh workflow run bench.yml --ref feat/batched-inference \
  -f variant=small -f repeats=5 -f threads=12

Add -f platform=ubuntu-24.04-arm to run the same job on the ARM64 runner (the committed arm64 row above used -f threads=4 to match its 4 CPUs).

The workflow supports the regular small, base, large, and giant models and their explicitly named small-registers, base-registers, large-registers, and giant-registers variants. all currently means the four regular sizes. Register-token variants, base/large/giant sizes, and any cross-platform or GPU measurements remain unmeasured in the committed result. They must be run and recorded separately before being described as evidence.

For an already staged f16 GGUF on Ubuntu:

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DDINOV2_BUILD_TESTS=OFF
cmake --build build --target dinov2-cli -j"$(nproc)"
scripts/bench.sh --models small --repeats 5 --threads 12 \
  --warmup 1 --image assets/tench.jpg --bin build/bin/dinov2-cli \
  --out benchmark_results.txt

scripts/bench.sh records the UTC timestamp, runner information, repository SHA, exact model path and SHA-256, precision, backend, batch, image, thread count, warmup and timed-run counts, and command inputs. It reports mean, standard deviation, minimum, maximum, and peak RSS for each model. The timed block is the model forward pass. A standard deviation above roughly five percent of the mean is a reason to repeat the run on a less-loaded runner.

Model variants

Regular and register-token checkpoints are distinct published models and must remain distinct result rows. Register tokens are useful for dense feature and local feature-map workflows, while regular checkpoints remain useful for exact baseline comparisons. This document makes no quality or speed claim for an unmeasured variant.

The benchmark scope is deliberately limited to f16 CPU inference. The repository may load other GGUF types, but loading them is not benchmark evidence and no quantization-specific benchmark mode or table is provided here.

Historical comparison

The following comparison is retained as historical context only. It was measured on an Intel Core i9-14900HX with 24 threads and is not reproducible from this Ubuntu workflow. It is not part of the current-result section and does not establish a current cross-platform claim.

Model PyTorch speed dinov2.cpp speed
small with registers 297 ms 64 ms
base with registers 436 ms 200 ms
large with registers 1331 ms 597 ms
giant with registers 4472 ms 1995 ms
small without registers 181 ms 62 ms
base without registers 462 ms 197 ms
large without registers 1288 ms 600 ms
giant without registers 4384 ms 1969 ms