Skip to content

test(e2e): fix four failing Nebius cases (87, 99, 104, 107) - #758

Merged
thxCode merged 1 commit into
mainfrom
fix/e2e-case-87-zone-guard
Oct 10, 2026
Merged

thxCode merged 1 commit into
mainfrom
fix/e2e-case-87-zone-guard

Conversation

@thxCode

@thxCode thxCode commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

/kind failing-test
/area testing

What this PR does / why we need it:

Four e2e cases failed on the Nebius run of role mutability for reasons that sit in the test scripts, not in the product. Each root cause below was checked against the run logs or the code.

  • Case 87 aborted with jq: Cannot iterate over null on a cluster whose Nodes carry no zone label. It now prints one SKIP row that names the measured shape, then exits 0.
  • Case 99 wrote the Hugging Face token into its Secret through a here-string, which appends a newline. The gated download then sent a malformed Authorization header. It now pipes the token with printf '%s', as case 101 already does.
  • Cases 101 and 104 read plugin metrics through the API server pod proxy (:32444/proxy/metrics). The model-manager NetworkPolicy admits only the plugins, the worker and the configured scrapers on that port, so the proxy timed out. A new plugin_metrics helper reads them from inside the plugin container over the loopback, which the policy never filters.
  • Case 107 ran on a pool of one AMD and four Intel Nodes, which Kueue splits across two ResourceFlavors. Kueue picks the flavor first and TAS ranks Nodes only inside it, so a preferred term that names a Node of the other flavor cannot win. All 20 trials landed on the same Node. The case now keeps the largest same-model group and exits 2 when no group has two Nodes. The TASRespectNodeAffinityPreferred gate was loaded (loaded=true), so it is not the cause.

Which issue(s) this PR links to:

NONE

Special notes for your reviewer:

  • Test scripts and the e2e skill table only. No product code and no chart change. The NetworkPolicy is correct and stays as it is.
  • Not run on a cluster. Checked so far: make lint agents-shell, make lint docs, git diff --check, and a local simulation of the case 107 grouping with four pool shapes. The simulation caught one bug in my first version (the CPU id had no trailing newline, so sort -u merged all Nodes); it is fixed.
  • Two assumptions are still open until the next cluster run: the main container has curl (the image installs it, and case 111 already relies on it), and Kueue gives the same-flavor pool the 1/K baseline that case 107 tests against.
  • Case 25 (MIG) did not finish on that run because the platform took the Node down after MIG was enabled. It is out of scope here.
  • Case 99 stopped at its gated-deployment step on the last run, so its cache-path assertions have not been exercised yet.

Does this PR introduce a user-facing change?

NONE

Case 87 aborted with no row on a cluster whose Nodes carry no zone
label. It now prints one SKIP row naming the measured shape and exits 0.

Case 99 wrote the Hugging Face token into its Secret through a
here-string, which appends a newline, so the gated download sent a
malformed Authorization header. It now pipes the token with printf '%s'.

Cases 101 and 104 read plugin metrics through the API server pod proxy.
The model-manager NetworkPolicy admits only the plugins, the worker and
the configured scrapers on the secure port, so the proxy timed out on a
CNI that enforces it. A new plugin_metrics helper reads them from inside
the plugin container over the loopback, which the policy never filters.

Case 107 mixed Nodes of two CPU models, so Kueue split them across two
ResourceFlavors. It picks a flavor first and TAS ranks Nodes only inside
it, so a preferred term naming a Node of the other flavor could not win.
The case now keeps the largest same-model group and exits 2 when none
has two Nodes.

Test scripts and the e2e skill table only. No product code or chart
change. Not yet run on a cluster.

Signed-off-by: thxCode <thxcode0824@gmail.com>
@gpustack-code-review

Copy link
Copy Markdown

✅ OpenCodeReview: Review skipped: no items were selected.

@thxCode
thxCode merged commit 1e4ff35 into main Oct 10, 2026
7 checks passed
@thxCode
thxCode deleted the fix/e2e-case-87-zone-guard branch October 10, 2026 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant