feat: Sanitize host BMC during tenant release

[spydaNVIDIA]

Is this a new feature, an enhancement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

We have seen that host BMCs between various SKUs (GB200, Grace-Grace, various SMC machines) have their BMCs reporting an invalid boot order during tenant deletion. In the latest case, the BMC was wedged in such a way that it was both reporting an incorrect boot order and the UEFI would not accept boot order changes from the BMC.

Feature Description

During the tenant deletion flow, sanitize the host BMC to avoid such issues. When a tenant releases an instance we do various cleanup procedures such as power cycling the machine, cleaning up the NVMe drives, ensuring the bios is configured appropriately. Lets include sanitizing the host BMC as part of that so we can proactively avoid these BMC issues.

Describe your ideal solution

1. Factory Reset BMC
2. Wait for BMC to come back up
3. Change password from factory default to site-wide password
4. Reboot host to reload the boot options into the BMC (at this point, I have observed that the boot options come up as empty in the BMC).
5. Check and set the boot order on the host if it is not configured to boot off the DPU

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[zanezhuNVIDIA]

before the BMC reset to default, should we save the necessary settings during ingestion (like KCS, host interface, lockdown mode, security, etc) and then re-apply those settings back to the BMC after reset-default?

[spydaNVIDIA]

@zanezhuNVIDIA we actually do this today during tenant deletion already: we make sure that the BMC and BIOS of the host server are configured appropriately after deletion.

[AlexGridnev]

We're blocked on validating this until instance creation blockers are fixed, cc @relliott40 to take this when these bugs are fixed