[BMC] Wait for Redfish readiness after BMC firmware update

[william8545]

Description

After a BMC firmware update the BMC is restarted to activate the new image. Ping/L3 connectivity returns noticeably earlier than the BMC's Redfish/HTTPS management plane, so the post-reset wait now gates completion on Redfish being usable rather than on ping.

The readiness logic is encapsulated so callers don't reach into RedfishClient directly:

• RedfishClient.wait_until_redfish_ready(timeout, interval) — polls a quiet login() until the Redfish service accepts it or the timeout elapses, returning a RedfishClient return code (ERR_CODE_OK when ready). It drops any cached session first via invalidate_session() (a local-only call that cannot emit an error log), so a cached token can't report a false "ready"; it sleeps min(interval, remaining) so it never overshoots the deadline; and it fails fast only on a local, clearly permanent error.
• BMCBase.wait_until_redfish_ready(...) — a thin delegator to self.rf_client, deliberately not decorated with @with_session_management (that decorator would try to log in while the BMC is still rebooting and defeat the wait).
• bmc_fw_update.py — after the reset, waits a short settle interval (so a not-yet-rebooted BMC isn't mistaken for ready) and then calls bmc.wait_until_redfish_ready(). The previous ping-based wait loop is removed: a successful Redfish login implies L3 is back, so ping is redundant.
• RedfishClient.login() gains a keyword-only log_errors flag (default True). When False, login failures are logged at notice instead of error, so the readiness poll's expected failures during the reboot window don't add error-level entries to the log. The default keeps behaviour unchanged for all existing callers.

Motivation and Context

When a BMC firmware update triggers a BMC restart, L3/ping recovers before the BMC's Redfish management plane. The previous post-reset wait only checked ping, so the update could report completion while the Redfish version-read API was still unavailable. Follow-on tooling that reads the BMC firmware version (for example fwutil show status) would then race the restart and log transient Redfish login errors during that window. Gating completion on a successful quiet Redfish login closes that window for callers serialized after the update, and the quiet-login flag keeps the readiness probe itself from adding error-level entries.

AUTH_FAILURE (HTTP 401 on the login POST) is treated as transient, since during BMC boot the web server can come up before the auth backend; the poll only fails fast on a local, unrecoverable error. The readiness wait is bounded by a timeout; if Redfish does not become ready within the bound, the update path logs an error and exits non-zero instead of hanging.

How Has This Been Tested?

Unit tests in tests/redfish_client_test.py, tests/bmc_base_test.py, and tests/bmc_fw_update_test.py cover wait_until_redfish_ready (first-try ready; ready after N retries; timeout returns the last non-OK code; fail-fast on a local permanent error; a transient auth failure keeps polling; sleep capped to the remaining budget; and a cached session not short-circuiting the probe), the BMCBase delegation, the quiet-vs-error login() logging, and the bmc_fw_update flow (settle + readiness call, success, and failure exit).

Additional Information (Optional)

• The default log_errors=True preserves existing logging behaviour, so current RedfishClient.login() callers are unaffected.
• This change targets the readiness window for callers serialized after the firmware update; hardening readers that race the restart on a separate command channel is out of scope.

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yxieca]

Reviewed via interactive multi-model review — the design is sound and I'd be comfortable with this going in; leaving a couple of non-blocking nits rather than a formal state.

Nice change: replacing the fixed sleep(20) + 5×get_status() ping loop with a Redfish-login readiness poll is a genuine improvement — a successful login proves the web server, auth backend, and L3 are all back, which is a stronger "ready" signal than a ping. The correctness details hold up well: the local-only invalidate_session() before the loop (so a cached token can't short-circuit login() into a false "ready", and no spurious ERROR-level DELETE), the 20s settle before polling, the min(interval, remaining) deadline cap, fail-fast only on the local-permanent PASSWORD_UNAVAILABLE while keeping AUTH_FAILURE/network codes transient, and the login(*, log_errors=False) quiet-poll flag that leaves existing callers untouched. Test coverage of the edge cases (first-try, retries, timeout, fail-fast, transient 401, sleep cap, cached-session drop, delegation/default-args) is excellent.

Non-blocking nits:

1. The two readiness docstrings for wait_until_redfish_ready are written as bare triple-quoted strings placed above the def (module-level string statements), so they aren't attached as the function's __doc__. This matches the existing style in this module, so it's consistent — just noting that if the intent was a real docstring, it would need to move inside the def.
2. On timeout, wait_until_redfish_ready returns the last observed code (e.g. -10 SERVER_UNREACHABLE) rather than normalizing to ERR_CODE_TIMEOUT (-6). The caller only checks != 0 and logs the code, so this is harmless — and returning the last real cause is arguably more useful for debugging than a generic timeout. Purely a readability call.

Neither is required for merge.

AI-assisted review, reviewed by Ying Xie (@yxieca).

[judyjoseph]

LGTM @william8545 can you confirm that these changes are applicable if we use the redfish from switch-host --> BMC

[mssonicbld]

Cherry-pick PR to msft-202608: Azure/sonic-platform-common.msft#118