Skip to content

[BUG]: deleting a runtime-required NodeWright leaves its taint on nodes where it never completed #714

Open

Description

@rice-riley

Summary

Deleting a runtimeRequired: true NodeWright does not remove the runtime-required taint from nodes where that NodeWright never completed. Nothing else removes it afterwards, so the node stays NoSchedule with no NodeWright left to finish the work.

Why

Phase 3 of the finalizer calls only node.Uncordon() and node.CleanupSCRMetadata() (operator/internal/controller/skyhook_controller.go, "Phase 3" block); neither touches spec.taints. The only removal path is HandleRuntimeRequired, which builds its candidate list from groupSkyhooksByNode(clusterState) and untaints a node once every runtime-required NodeWright targeting it is complete there (getRuntimeRequiredTaintCompleteNodes). A deleted NodeWright is no longer in the cluster state, so a node it left incomplete drops out of that map and is never considered.

Reproduce

  1. One runtimeRequired: true NodeWright targets a node that carries the runtime-required taint (applied by provisioning or by autoTaintNewNodes). A package on it is erroring on that node.
  2. Delete the NodeWright with uninstall.enabled: false on its packages, so Phase 3 runs immediately.
  3. The finalizer is released. The node still has nodewright.nvidia.com=runtime-required:NoSchedule.

Expected

Either the finalizer removes the recognised runtime-required taints from the NodeWright's nodes when no other runtime-required NodeWright still targets them, or docs/user-guide/runtime-required.md ("When is the runtime-required taint removed from a node") and docs/architecture/lifecycle.md (Deletion) state that deletion leaves the taint and that removing it is the operator's user's job. Today the lifecycle page claims deletion removes it, which #713 corrects; this issue is the behaviour decision behind that wording.

Found by @ayuskauskas reviewing #713.

Metadata

Metadata

Assignees

No one assigned

    Labels

    component/operatorSkyhook operator (controller-manager)

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions