Skip to content

NodeProfile controller can replay a consumed plan after a crash #255

Description

@spyroot

Summary

consumedPlanHash is only recorded in the CR status AFTER the reconcile handler returns: build_status returns the dict carrying status["consumedPlanHash"] (k8s/controller/redfish_node_profile_controller.py:255) and kopf patches the object post-return. The BMC write (the approved one-shot apply, e.g. BootOneShot per the sample) executes inside the handler before that return.

Consequence

A pod failure after the BMC write but before the Kubernetes status patch leaves the plan looking unconsumed while approvedPlanHash still matches — the next reconcile can REPLAY the mutating action. This is an inference from the control flow, not a reproduced failure, but the window is structural.

Fix shape

Record intent before acting: patch consumedPlanHash (or an explicit in-progress/intent marker) BEFORE issuing the BMC write, or make the apply verify-before-apply idempotent so a replay is a no-op. Regression test: simulate handler death between apply and status persistence (fake-kopf harness in tests/test_k8s_controller_concurrency.py is reusable) and assert the second reconcile does not re-issue the write.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions