Workload Plane isolation detection and remediation

[TeddyAndrieux]

Component: salt, containers, build, tests

Context:

MetalK8s already detects Workload Plane (WP) isolation: node-problem-detector
(shipped in 133.0.12) exposes the WorkloadPlaneNetworkUnavailable node
condition when a node can no longer reach the majority of its peers over the
Workload Plane. This PR adds the remediation that acts on that condition, the
alerts that surface it, and an end-to-end test.

Summary:

• Deploy node-warden-operator v1.0.0 as a kustomize addon in the
  metalk8s-monitoring namespace, ordered after Prometheus (it ships a
  ServiceMonitor, labelled so our Prometheus scrapes it) and cert-manager (its
  webhook certificate). Added to the deployment sanity checks.
• Default remediation policy (workload-plane-isolation, a cluster-scoped
  NodeRemediationPolicy): when a WP node reports
  WorkloadPlaneNetworkUnavailable, apply a reversible
  node.scality.io/workload-plane-unreachable:NoExecute taint to drain it and
  drop it from Service endpoints, then remove the taint once WP connectivity
  recovers. A guard suppresses remediation when the majority of nodes are
  affected (cluster-wide outage). Infra components tolerate the taint.
• Prometheus alerts, split by the component that produces the signal:
  NodeWorkloadPlaneUnavailable (node-problem-detector) for the condition, and
  NodeWorkloadPlaneRemediated + WorkloadPlaneOutage (node-warden-operator)
  for the taint / suppressed-guard cases; also registered in the lib-alert-tree.
• End-to-end BDD test on a segregated-network cluster: an isolated node is
  drained and recovers (link-down and blocked modes), while a Calico rolling
  restart, a short blip, and a majority outage do not trigger remediation.

Acceptance criteria:

• CI is green, including the new workload_plane_isolation.feature on a
  multi-node, segregated-network cluster.
• A WP node that loses connectivity gets the taint, its workloads are evicted,
  and NodeWorkloadPlaneUnavailable / NodeWorkloadPlaneRemediated fire; all
  revert once connectivity is restored.
• A cluster-wide WP outage raises WorkloadPlaneOutage and does not taint
  any node.

---

Fixes: MK8S-320, MK8S-321, MK8S-323

[bert-e]

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

[claude]

LGTM

Review by Claude Code

[TeddyAndrieux]

/create_integration_branches

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

Peer approvals must include at least 1 approval from the following list:

• @g-carre

• @m4nch0t

• @anthony-treuillier-scality

• @eg-ayoub

• @thomasdanan

• @ChengYanJin

• @JBWatenbergScality

• @TeddyAndrieux

• @ZiyangLinScality

• @MonPote

• @damiengillesscality

• @ezekiel-alexrod

• @JeanMarcMilletScality

• @hervedombya

The following options are set: create_integration_branches