Skip to content

test(installer): probe cluster telemetry and reject demo mode in smoke gates - #206

Closed
mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:test/smoke-gate-cluster-telemetry
Closed

mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:test/smoke-gate-cluster-telemetry

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Closes #195

Summary

  • Enhance test-installer-artifact smoke gate recipe in Justfile to probe /api/mcp/clusters and ensure .source != "demo" and active cluster discovery before passing, preventing synthetic dev/demo mode from passing silently.
  • Update tests/e2e/test_kubestellar_browser_login.py with verify_live_cluster_resources() to verify that actual Kubernetes cluster resources (nodes, pods, or initialized ControlPlanes) are discovered and rendered in the DOM, while explicitly rejecting demo mode (kc-demo-mode=true in localStorage) and synthetic demo cluster fixtures (kind-local, minikube, etc.).
  • Add contract tests in tests/unit/test_installer_contract.py and tests/unit/test_lima_e2e_contract.py verifying the smoke gate recipe and headless browser test logic.

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: cc40da8

…e gates

Enhance test-installer-artifact readiness check to verify /api/mcp/clusters
rejecting demo mode (.source != "demo") alongside /healthz and root HTTP 200.
Update E2E browser verification to assert live Kubernetes cluster resources
in the DOM and reject synthetic demo fixtures and demo mode localStorage flags.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
@castrojo

Copy link
Copy Markdown
Contributor

Needs a rebase — see #210.

No conflict on paths: this PR touches Justfile, tests/e2e/test_kubestellar_browser_login.py, tests/unit/test_installer_contract.py and tests/unit/test_lima_e2e_contract.py, none of which are deleted. But the assertions target the k0s-era console wiring, and #210 changes what "live" looks like: real ControlPlane instances from files/cluster/manifests/controlplanes/controlplanes.yaml, and DEV_MODE removed from the console deployment outright rather than probed for at runtime.

The .source != "demo" gate on /api/mcp/clusters is exactly the right shape and I want it. Please rebase after #210 and re-point the fixtures at the new cluster stack. Refs #195.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

installer-test / Install and boot in QEMU is failing on this PR because it is failing on main itself, not because of anything in this diff.

Tracked as a single repository incident in #218. main has been red on this check since 2026-09-14; the same failure currently blocks 17 open PRs, including a docs-only PR (#208) and a Renovate digest bump (#49).

Failure mode is /dev/disk/by-partlabel/var never appearing, so /var and everything ordered after it depend-fail and the readiness probe times out.

Please don't retry this check per-PR — it will stay red until #218 is resolved.

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Acknowledged @castrojo. We will rebase this branch onto main as soon as #210 lands, and re-point the cluster telemetry fixtures and assertions at the new ControlPlane manifest stack without DEV_MODE.

🐝 Hive Agent: contributor | SHA: cc40da8

🐝 Hive Agent: contributor | SHA: unknown

@Danathar

Copy link
Copy Markdown
Contributor

#220 merged to main at 14:06 UTC today with the fix for the installer-test boot deadlock (udev sysext ordering, depmod path, homed-firstboot mask). This branch predates it, so its installer-test / Install and boot in QEMU failure is the old repo-wide one, not this change. A rebase onto current main is all it needs to get a real result from that check; I cannot update the branch from here.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • Gate still passes with zero discovered clusters (Justfile:357). The jq filter (.clusters | length) > 0 or .source == "k8s" or .source == "mcp" accepts {"clusters": [], "source": "k8s"} — and the console's k8s fallback path (kubestellar/console v0.3.34 pkg/api/handlers/mcp/cluster.go:69-73) can return an empty list with source: "k8s". So "active cluster discovery" is not actually enforced; the or .source == "k8s" clause should be dropped or the empty-list case rejected. The new unit test (tests/unit/test_installer_contract.py:152-181) doesn't cover this payload.
  • DOM check can pass without live resources (tests/e2e/test_kubestellar_browser_login.py:125-130). The text fallback matches static card headings ("Node Status", "Cluster Health", "Top Pods") that render even with empty data. Also, the demo-cluster-name scan (lines 92-97) runs once before async content loads, so late-rendered demo fixtures are missed.

intent-alignment

  • .source != "demo" in the curl gate is unreachable (Justfile:357). In console v0.3.34, source: "demo" is only returned when the request carries the X-Demo-Mode: true header (pkg/api/handlers/demo_data.go:15-17, cluster.go:51-54); the gate's plain curl never sends it. The PR body's claim that the smoke gate "ensures .source != "demo"… preventing synthetic dev/demo mode from passing silently" is therefore not implemented at the API level — demo rejection only actually happens in the browser e2e check.

style

  • tests/e2e/test_kubestellar_browser_login.py:82-88 — re-raising based on "synthetic demo mode" in str(e) silently swallows unrelated WebDriver errors; catch the specific exception instead. tests/unit/test_installer_contract.py ends with two extra blank lines.

No findings from: security, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

The smoke gate accepted {"clusters": [], "source": "k8s"}, which the
console k8s fallback returns before any cluster is reachable, so it did
not enforce active cluster discovery. It also checked .source != "demo",
which is unreachable for a plain curl because the console only serves
demo data for requests carrying X-Demo-Mode: true.

Require a k8s/mcp source with at least one discovered cluster and reject
responses containing upstream synthetic demo fixture names, which is what
the unauthenticated probe can actually observe. Extract the filter and the
demo fixture list into shell variables and read them from the Justfile in
the unit test so the test cannot drift from the gate.

Tighten the browser check as well: drop static card headings and container
selectors that render with empty data, re-scan for demo fixtures on every
poll so late-rendered demo content cannot slip through, and catch
WebDriverException by type instead of matching exception message text.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • Dead DOM selectors: the "nodes/pods rendered" check is not actually implemented (medium) — tests/e2e/test_kubestellar_browser_login.py:97-101 gates on [data-testid='node-row'] and [data-testid='pod-row'], but neither testid exists anywhere in the kubestellar/console web source; cluster-card exists only on the Workloads page (web/src/components/workloads/Workloads.parts.tsx:222), which this test never navigates to — it stays on the dashboard after login (run_browser_verification, line ~215-228). In practice the DOM verification passes only via the substring markers its1/wds1/in-cluster in body text (line 105). It still fails closed, but the PR body's claim that "nodes, pods, or initialized ControlPlanes are discovered and rendered" is nominal, and if the dashboard doesn't surface those strings the check will hard-fail every run.
  • 15s render timeout may flake right after boot (low) — verify_live_cluster_resources(driver, timeout=15) at tests/e2e/test_kubestellar_browser_login.py:227. Upstream ListClusters returns HealthUnknown clusters immediately after boot and populates health asynchronously (pkg/api/handlers/mcp/cluster.go), so first dashboard render of live identifiers can plausibly exceed 15s.

style

  • Timeout message no longer names the failing condition (low) — Justfile:426 still prints "Timed out ... waiting for KubeStellar Console readiness!" even when /healthz and / were fine all along and only the new cluster-telemetry filter never passed. Logging which stage failed (or the last CLUSTERS_JSON) would save triage time, since this gate now also requires live cluster discovery within the same deadline.
  • Negative asserts like assert "Node Status" not in content (tests/unit/test_lima_e2e_contract.py:50-51) forbid those strings anywhere in the script forever — brittle for a rationale better carried by the existing comment.

No findings from: security, intent-alignment, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…stage

- Replace non-existent node-row/pod-row selectors with the per-cluster
  testids the console actually renders (cluster-row-<name> on /clusters,
  cluster-card on /workloads) and navigate to those routes, so the DOM
  verification is actually exercised instead of falling back to text markers.
- Make the live-cluster render budget configurable (default 120s, new
  --cluster-telemetry-timeout flag); upstream populates cluster health
  asynchronously after boot, so 15s could flake on a healthy install.
- Report which readiness stage blocked the smoke gate on timeout (healthz,
  root, or cluster telemetry) and echo the last /api/mcp/clusters payload.
- Drop the forbidden-string asserts in the e2e contract test in favour of
  positive asserts on the new routes, selectors and timeout flag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • tests/unit/test_installer_contract.py:181-189 — the new contract test shells out to jq via subprocess.run, adding a hard external dependency to the unit suite. On a machine without jq, subprocess.run raises FileNotFoundError rather than a clear assertion. Consider pytest.skip/shutil.which("jq") guarding.
  • Justfile:352-355 — LAST_STAGE only ratchets forward, so if /healthz regresses after the clusters stage was reached once, the timeout message blames /api/mcp/clusters instead of the stage currently failing. Cosmetic for triage, but that's exactly what the stage tracking is for.
  • tests/e2e/test_kubestellar_browser_login.py:135-142 — assert_no_demo_fixtures substring-matches the full body text, so a legitimate operator cluster named e.g. minikube, or page copy mentioning a fixture name, hard-fails the gate. Acceptable for this controlled installer VM, but worth knowing it's name-based, not source-based.

style

  • tests/e2e/test_kubestellar_browser_login.py:112 (and the CLUSTER_RESOURCE_PAGES comment block) — cites web/src/components/workloads/Workloads.parts.tsx, but at the pinned console v0.3.34 (files/k0s/manifests/kubestellar/40-kubestellar-console.yaml:44) that file does not exist; data-testid="cluster-card" lives in web/src/components/workloads/Workloads.tsx:535. The comment documents a future refactor, not the deployed tag.
  • DEMO_CLUSTER_NAMES is duplicated verbatim in Justfile:345 and tests/e2e/test_kubestellar_browser_login.py:88-101 as a hardcoded copy of upstream pkg/api/handlers/demo_data.go:34-45; it will silently drift when the console image is bumped past v0.3.34.

Verified against upstream kubestellar/console v0.3.34: demo fixture names match demo_data.go:34-45; .source == "demo" is only set for X-Demo-Mode: true requests (demo_data.go:11-15), so the name-based rejection in the Justfile comment is accurate; /api/mcp/clusters bypasses auth in DEV_MODE (routes_integrations.go:26-29), so the unauthenticated curl probe can actually pass.

No findings from: security, intent-alignment, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar

Copy link
Copy Markdown
Contributor

The idea is right: passing the smoke gate on synthetic demo fixtures is exactly the false-green worth closing, and the staged LAST_STAGE error messages are a real improvement on a bare "timed out".

The problem is that it is written against a probe that no longer exists. #220 (cdb1683) removed the host-side curl http://127.0.0.1:8080/healthz poll from test-installer-artifact, on the grounds that a QEMU hostfwd cannot reach the guest's loopback-bound kiosk proxy and the proxy terminates TLS anyway, so that poll could only ever hit an unrelated host process and false-pass. Main now waits on KIOSK_CONSOLE_READY emitted from inside the guest over serial. This PR's Justfile and tests/unit/test_installer_contract.py hunks conflict with exactly that.

So the demo-fixture rejection needs to move inside the in-guest readiness unit, next to the existing /healthz and / checks, and query https://127.0.0.1:8080/api/mcp/clusters with --insecure from there.

Separately, gh pr checks reports nothing on 6c6b159. The last runs on test/smoke-gate-cluster-telemetry are from cc40da8 on 2026-09-19. Needs CI before it can be called clean.

@castrojo

Copy link
Copy Markdown
Contributor

As noted in review, the smoke gate probe host-side poll in Justfile was replaced in #220/#227 with in-guest readiness signaling over serial (KIOSK_CONSOLE_READY). This branch requires moving the cluster telemetry probe into the in-guest readiness unit before it can be merged.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal (don't let the smoke gate pass on demo fixtures) is right, but this needs reworking against current main:

  • The Justfile hunk extends the host-side curl poll loop that #220 removed. Main now waits for KIOSK_CONSOLE_READY from inside the guest over serial, so the probe needs to move into the in-guest readiness unit.
  • It probes http://127.0.0.1:8080/api/mcp/clusters, but the kiosk is TLS-only; everything else uses https:// with --insecure.
  • DEMO_CLUSTER_NAMES is duplicated between the Justfile and tests/e2e/test_kubestellar_browser_login.py; please define it once. (It also becomes largely moot if the kubeadm cutover in #210 lands.)
  • The branch conflicts with main and no CI has run on it.

@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Status note: the installer smoke gate this PR extends (the Flatcar-based offline installer and just test-installer-artifact) was removed in #262; issue #195 is closed as obsolete. Boot and install coverage now comes from scripts/dogfood-diskless.sh and scripts/dogfood-install.sh in CI. Leaving the decision on this PR to you.

@castrojo

Copy link
Copy Markdown
Contributor

Closing: installer smoke gate and Lima test harness targeted by this PR were replaced by the QEMU/UKI boot-test harness in PR #262 and #264.

@castrojo castrojo closed this Sep 29, 2026
@castrojo

Copy link
Copy Markdown
Contributor

Reopening: goal of rejecting demo fixtures in smoke tests remains relevant. Branch conflicts with main because host-side curl poll loop was replaced by guest serial signal; requires author rebase.

@castrojo castrojo reopened this Sep 29, 2026
@castrojo

Copy link
Copy Markdown
Contributor

Superseded by PR #262. The Flatcar offline installer smoke gate, Justfile target test-installer-artifact, and Lima test harness targeted by this PR were retired and replaced by the FSDK QEMU/OVMF boot-test suite.

@castrojo castrojo closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(installer): smoke gate tests only /healthz and HTTP 200, allowing demo mode to pass silently

6 participants