Skip to content

bench: ask the community row's plan probes about the cards the run will use, not GPU 0 - #474

Open
erkrali wants to merge 1 commit into
jesdga95:mainfrom
erkrali:bench-community-probe-gpu
Open

erkrali wants to merge 1 commit into
jesdga95:mainfrom
erkrali:bench-community-probe-gpu

Conversation

@erkrali

@erkrali erkrali commented Nov 19, 2026

Copy link
Copy Markdown

Closes #463.

--community refuses --gpu, because a row has to be comparable, and then chooses the cards the run
uses exactly as the server does, from CUDA_VISIBLE_DEVICES. The plan probes — the model's name and
architecture for the row, which --kv the model takes, its fixtures, the quality gate's window —
still asked about --gpu 0. On a box whose first visible card is not the main card's architecture,
which is precisely what -DPEER_ADA=ON puts there, that probe plans an engine that could never
start: gpu_plan_json emits arch and model only when the plan did not error, so the name comes
back empty and the run is refused with a message pointing at the pack's metadata. The boxes the
feature exists for cannot file the rows that would validate it.

What changed

  • bench_main.cpp: the card selection moves above the probes and the five "--gpu", "0" probes take
    the value the run is given (probe_gpu, which is gpu_value, which is what select_gpus picked).
    Nothing about a row's measurement moves: the probes are engine_plan calls, not runs.
  • The refused run now says what the plan said. plan_complaint() reads the probe's own log — the
    answer is empty in exactly this case, so the log is the only thing naming the cause — and the error
    gains a second line, the engine's plan said: <line>.
  • An engine that fails to start leaves the temp directory its own message named. Cleanup was
    deleting it, log and all, on the way out.
  • docs/community.md: one sentence for a box that enumerates the older card first —
    CUDA_VISIBLE_DEVICES=1 for a single-card row, 1,0 with the peer — since the harness samples
    power and clocks from the first visible card while it measures.

Parity

Where card 0 is the main card's architecture, select_gpus resolves --gpu 0 to 0 and the probes
are handed the string they were handed before. For a community run the probe's plan goes from one
card to the set the run uses; the fields taken from it are model, arch and which --kv flag the
model accepts, none of which the card set changes. The --quality gate reads its window from its
probe, and there seeing the cards it will actually run on is the correction rather than a change. The
one visible reordering: a card-selection error now surfaces before the --community flag refusal.

Verification

g++ -std=c++20 -fsyntax-only over src/program/bench_main.cpp at ad274aa and with these changes:
clean, no new diagnostics. No new test — the thing under test is a subprocess call whose answer comes
from the engine's plan, so a test would need a GPU and a pack. On hardware, the failure this fixes is
reproducible in one line and the fix is the two CUDA_VISIBLE_DEVICES spellings the docs now give:

basalt: gpu auto: GPU 0 (NVIDIA GeForce RTX 4060 Ti) is not sm_120, using it as an expert-tier peer only
basalt-bench: error: --community: the engine reports no community model name for <pack> (architecture 'qwen4exp'); no row was written

We have that pair (5060 Ti + 4060 Ti, CUDA 13) and can post both rows — single-card and Ada-peer — as
soon as a build with this in it is running here, if a 5060 Ti + 4060 Ti row would be worth having.

…ll use

--community rightly refuses --gpu, since a row has to be comparable, and then picks the cards the
run uses the way the server does, from CUDA_VISIBLE_DEVICES.  The plan probes that read the row's
model name, its cache format and its fixtures still asked about "--gpu 0".  On a box whose first
visible card is not the main card's architecture - a Blackwell card with an Ada card built in by
-DPEER_ADA=ON, where enumeration may name the older one first - that probe plans an engine that
could never start: the answer has no plan in it, the model name comes back empty, and the run is
refused with a line pointing at the pack's metadata.  The boxes -DPEER_ADA=ON exists for are then
the ones that cannot file the rows that would validate it.

Move the card selection above the probes and hand the probes the value the run is given.  A probe
is not a measurement, so no figure in a row changes.  Two small things ride along, both about
reaching the real cause: a refused community run now quotes the plan log's own error line, and an
engine that fails to start leaves behind the temp directory its message names instead of deleting
it.  docs/community.md says what such a box should pass in CUDA_VISIBLE_DEVICES, which is the
question this reached as.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant