Repository navigation
Conversation
…ll use --community rightly refuses --gpu, since a row has to be comparable, and then picks the cards the run uses the way the server does, from CUDA_VISIBLE_DEVICES. The plan probes that read the row's model name, its cache format and its fixtures still asked about "--gpu 0". On a box whose first visible card is not the main card's architecture - a Blackwell card with an Ada card built in by -DPEER_ADA=ON, where enumeration may name the older one first - that probe plans an engine that could never start: the answer has no plan in it, the model name comes back empty, and the run is refused with a line pointing at the pack's metadata. The boxes -DPEER_ADA=ON exists for are then the ones that cannot file the rows that would validate it. Move the card selection above the probes and hand the probes the value the run is given. A probe is not a measurement, so no figure in a row changes. Two small things ride along, both about reaching the real cause: a refused community run now quotes the plan log's own error line, and an engine that fails to start leaves behind the temp directory its message names instead of deleting it. docs/community.md says what such a box should pass in CUDA_VISIBLE_DEVICES, which is the question this reached as.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #463.
--communityrefuses--gpu, because a row has to be comparable, and then chooses the cards the runuses exactly as the server does, from
CUDA_VISIBLE_DEVICES. The plan probes — the model's name andarchitecture for the row, which
--kvthe model takes, its fixtures, the quality gate's window —still asked about
--gpu 0. On a box whose first visible card is not the main card's architecture,which is precisely what
-DPEER_ADA=ONputs there, that probe plans an engine that could neverstart:
gpu_plan_jsonemitsarchandmodelonly when the plan did not error, so the name comesback empty and the run is refused with a message pointing at the pack's metadata. The boxes the
feature exists for cannot file the rows that would validate it.
What changed
bench_main.cpp: the card selection moves above the probes and the five"--gpu", "0"probes takethe value the run is given (
probe_gpu, which isgpu_value, which is whatselect_gpuspicked).Nothing about a row's measurement moves: the probes are
engine_plancalls, not runs.plan_complaint()reads the probe's own log — theanswer is empty in exactly this case, so the log is the only thing naming the cause — and the error
gains a second line,
the engine's plan said: <line>.Cleanupwasdeleting it, log and all, on the way out.
docs/community.md: one sentence for a box that enumerates the older card first —CUDA_VISIBLE_DEVICES=1for a single-card row,1,0with the peer — since the harness samplespower and clocks from the first visible card while it measures.
Parity
Where card 0 is the main card's architecture,
select_gpusresolves--gpu 0to0and the probesare handed the string they were handed before. For a community run the probe's plan goes from one
card to the set the run uses; the fields taken from it are
model,archand which--kvflag themodel accepts, none of which the card set changes. The
--qualitygate reads its window from itsprobe, and there seeing the cards it will actually run on is the correction rather than a change. The
one visible reordering: a card-selection error now surfaces before the
--communityflag refusal.Verification
g++ -std=c++20 -fsyntax-onlyoversrc/program/bench_main.cppatad274aaand with these changes:clean, no new diagnostics. No new test — the thing under test is a subprocess call whose answer comes
from the engine's plan, so a test would need a GPU and a pack. On hardware, the failure this fixes is
reproducible in one line and the fix is the two
CUDA_VISIBLE_DEVICESspellings the docs now give:We have that pair (5060 Ti + 4060 Ti, CUDA 13) and can post both rows — single-card and Ada-peer — as
soon as a build with this in it is running here, if a 5060 Ti + 4060 Ti row would be worth having.