Skip to content

benchmarks: sweep_extra.sh escalates once and has no path for "still no majority" at N=3 #3041

Description

@Xore

Split out of a #3023 comment (round 7, batch A review). The comment itself says this is
"a gap in the escalation protocol itself, independent of this issue", and no issue
existed for it — filing so it is not lost when #3023 closes.

Benchmark/GPU-parked: this is in the excluded set for round 7 and should not be
worked until GPU access frees up.

The gap

sweep_extra.sh escalates a cell from N=2 to N=3 when run1 != run2, then stops.
There is no handling for the case where three runs produce three distinct values, so
no majority exists and the cell has no defensible score.

That case is live. From #1947 phase 2, the escalated cells:

A  codellama_7b-instruct-q4_K_M          [57, 58, 58] -> 58  (2/3)
A  qwen3_8b                              [64, 65, 65] -> 65  (2/3)
B  gemma-4-31B-it-qat-q4_0-heretic       [62, 63, 63] -> 63  (2/3)
A  pki/nova-24b-cybersec                 [58, 59, 58] -> 58  (2/3)
A  Foundation-Sec-8B-Instruct-Q8_0       [61, 62, 61] -> 61  (2/3)
A  Foundation-Sec-1.1-8B-Instruct-Q8_0   [62, 64, 62] -> 62  (2/3)
B  ravenx-cyberagent-35b                 [64, 63, 64] -> 64  (2/3)
A  GLM-4.6-REAP-218B:i1-IQ1_S            [55, 54, 55] -> 55  (2/3)
B  GLM-4.6-REAP-218B:i1-IQ1_S            [55, 57, 54] -> NO MAJORITY

Eight of nine resolve 2-of-3. The ninth does not resolve at all, and it is the most
CPU-offloaded row in the roster (57 GB served, 65%/35% CPU/GPU) — pointing at spill
nondeterminism rather than slot contention. It must not be published as 55, 57 or
54.

What would close this

  1. sweep_extra.sh detects "N=3 produced no majority" explicitly instead of silently
    taking one of the values.
  2. On that branch it either escalates to N>=5 or records the cell as UNRESOLVED.
  3. The matrix renderer carries UNRESOLVED through to publication rather than
    dropping the row or picking a value.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmlML worker and GPU scoring

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions