Skip to content

Add YOLOv9-T and RF-DETR-S benchmarks on DGX Spark - #14

Merged
EHxuban11 merged 2 commits into
LibreYOLO:mainfrom
DavidDiazMerino:benchmark/dgx-spark-libreyolo-v1.5.0
Sep 16, 2026
Merged

EHxuban11 merged 2 commits into
LibreYOLO:mainfrom
DavidDiazMerino:benchmark/dgx-spark-libreyolo-v1.5.0

Conversation

@DavidDiazMerino

@DavidDiazMerino DavidDiazMerino commented Sep 13, 2026

Copy link
Copy Markdown
Contributor
  • Add two unmodified harness runs for YOLOv9-T and RF-DETR-S on DGX Spark (GB10), PyTorch CUDA FP32.
  • Canonical mini500 latency runs, pointed at the HF dataset directly, without --limit. Harness defaults kept: batch 1, conf 0.001, IoU 0.6, max_det 300, native input sizes 640 / 512.
  • Pin LibreYOLO v1.5.0 (c25f6dffb521ea60bc0f63ae3dffb168a7edc466, the tagged release commit) and clean harness f55a528cdd714259405f08b22c44d32be4fc7ab7; the PR proposes adding that release commit to the support matrix.
  • GPU occupancy was checked before and during each run; no other compute job was detected.
  • mini500 mean latency: YOLOv9-T 10.016 ms, RF-DETR-S 16.690 ms.
  • Full-val (5000-image) reference runs were also produced locally (YOLOv9-T 38.09, RF-DETR-S 52.93) and are deliberately not part of this batch. See the review note below.
  • GB10 metadata limitation: the harness emits CPU Unknown and gpu_memory_gb: 0.0 for unified memory. These fields are unchanged and need maintainer review.
  • validate_submission.py and build_verified_results.py passed. Exactly two new records; no existing record was dropped.
  • First batch for maintainer review. No private football data or predictions are included.

Drop the two full-val (5000-image) reference runs from this first batch. They
are valid, but a full-val record for the same model/hardware/runtime/precision
makes the site prefer that record for the model's leaderboard row, which would
move yolov9t and rfdetr-s onto the Spark full-val accuracy. Keep the canonical
mini500 runs that the protocol asks for latency.

The full-val references are kept locally and available on request:
yolov9t 38.09, rfdetr-s 52.93 (they reproduce the project's Port Fidelity
values within 0.1).

Rebuilt generated/verified-results.v1.json.
@DavidDiazMerino

Copy link
Copy Markdown
Contributor Author

Review notes on this first batch, written before you dig in. Everything below comes from reading the branch, re-running the repo validators locally and cross-checking the existing corpus. I have not re-measured on the Spark.

What matches the protocol

  • The LibreYOLO pin is a release pin: c25f6dff... is exactly the v1.5.0 tag commit, so the support-matrix addition is not a stray dev commit.
  • The harness commit recorded in both JSONs (f55a528c) is the current head of vision-analysis-benchmark main, with harness_dirty: false.
  • The mini500 dataset id and revision (6c5d8d10...) match the HF repo, the runs use the dataset directly with no --limit, and the harness defaults are untouched (batch 1, conf 0.001, IoU 0.6, max_det 300, native 640 / 512).
  • validate_submission.py reports all submissions valid and build_verified_results.py rebuilds the committed generated/verified-results.v1.json byte for byte. 710 records, the two new ones added, none dropped.
  • Sanity check against the corpus: the Spark mini500 accuracy sits inside the range of the existing mini500 rows for both models (yolov9t 41.77 vs 41.77-41.82 elsewhere, rfdetr-s 55.15 vs 55.11-55.28).

Two things I deliberately held back, and would like your call on

  1. Full-val reference runs. I also ran full COCO val2017 (5000 images) on the Spark for both models: YOLOv9-T 38.09 and RF-DETR-S 52.93. Those reproduce the Port Fidelity page numbers (38.2 and 53.0) within 0.1, so the ARM / unified-memory path measures the same as the 5070 Ti. I left them out of this batch because of a site behaviour rather than a measurement problem: a full-val record for the same model / hardware / runtime / precision shows up as a second row with an identical Hardware and Runtime label (the model page table has no dataset-variant column), and the leaderboard candidate score prefers full-val batch 1, so yolov9t and rfdetr-s would switch to the full-val accuracy together with the Spark timing (41.8 -> 38.1 and 55.1 -> 52.9). Adding a device should not move other models' leaderboard rows, so I would rather you decide: submit them as a follow-up when the site can label the variant, or bring them in now.

  2. Cross-hardware latency comparisons. These rows are LibreYOLO 1.5.0 / harness 2.1.0 / torch 2.14.0+cu130, while the existing 5070 Ti rows are LibreYOLO 1.2.0.dev0 / harness 2.0.0 / torch 2.11.0+cu128. For a 4 GFLOPs model like yolov9t both are dominated by host and framework overhead, so an apparent "GB10 faster than a 5070 Ti" in the tables would be a version effect, not a hardware result. Flagging it so the numbers are not read that way.

Hardware metadata

cpu: "Unknown" and gpu_memory_gb: 0.0 are the harness reading unified memory on GB10 (cpu_cores: 20 and ram_gb: 121 are detected fine). I kept the fields as emitted instead of hand-editing the JSON. Happy to help with a small harness change if you want the Spark represented properly.

CI

Both workflows on this branch sit at action_required (0s), so they need your approval before they run. Local validation passes.

Not touching further models until you confirm the format and the hardware-metadata question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants