You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG]: deleting a runtime-required NodeWright leaves its taint on nodes where it never completed #714
Deleting a runtimeRequired: true NodeWright does not remove the runtime-required taint from nodes where that NodeWright never completed. Nothing else removes it afterwards, so the node stays NoSchedule with no NodeWright left to finish the work.
Why
Phase 3 of the finalizer calls only node.Uncordon() and node.CleanupSCRMetadata() (operator/internal/controller/skyhook_controller.go, "Phase 3" block); neither touches spec.taints. The only removal path is HandleRuntimeRequired, which builds its candidate list from groupSkyhooksByNode(clusterState) and untaints a node once every runtime-required NodeWright targeting it is complete there (getRuntimeRequiredTaintCompleteNodes). A deleted NodeWright is no longer in the cluster state, so a node it left incomplete drops out of that map and is never considered.
Reproduce
One runtimeRequired: true NodeWright targets a node that carries the runtime-required taint (applied by provisioning or by autoTaintNewNodes). A package on it is erroring on that node.
Delete the NodeWright with uninstall.enabled: false on its packages, so Phase 3 runs immediately.
The finalizer is released. The node still has nodewright.nvidia.com=runtime-required:NoSchedule.
Expected
Either the finalizer removes the recognised runtime-required taints from the NodeWright's nodes when no other runtime-required NodeWright still targets them, or docs/user-guide/runtime-required.md ("When is the runtime-required taint removed from a node") and docs/architecture/lifecycle.md (Deletion) state that deletion leaves the taint and that removing it is the operator's user's job. Today the lifecycle page claims deletion removes it, which #713 corrects; this issue is the behaviour decision behind that wording.
Activity
test(agent): assert what gracefulShutdown guarantees in sigterm_grace
test(agent): assert what gracefulShutdown guarantees in sigterm_grace
test(agent): assert what gracefulShutdown guarantees in sigterm_grace (…
chore(release): backport #708, #713 and #720 to release/agent/v7.0.x (#…