Skip to content

[action] [PR:707] [BMC] Wait for Redfish readiness after BMC firmware update#118

Merged
yxieca merged 1 commit into
Azure:202608from
mssonicbld:cherry/msft-202608/707
Jul 13, 2026
Merged

[action] [PR:707] [BMC] Wait for Redfish readiness after BMC firmware update#118
yxieca merged 1 commit into
Azure:202608from
mssonicbld:cherry/msft-202608/707

Conversation

@mssonicbld

Copy link
Copy Markdown
Collaborator

Description

After a BMC firmware update the BMC is restarted to activate the new image. Ping/L3 connectivity returns noticeably earlier than the BMC's Redfish/HTTPS management plane, so the post-reset wait now gates completion on Redfish being usable rather than on ping.

The readiness logic is encapsulated so callers don't reach into RedfishClient directly:

  • RedfishClient.wait_until_redfish_ready(timeout, interval) — polls a quiet login() until the Redfish service accepts it or the timeout elapses, returning a RedfishClient return code (ERR_CODE_OK when ready). It drops any cached session first via invalidate_session() (a local-only call that cannot emit an error log), so a cached token can't report a false "ready"; it sleeps min(interval, remaining) so it never overshoots the deadline; and it fails fast only on a local, clearly permanent error.
  • BMCBase.wait_until_redfish_ready(...) — a thin delegator to self.rf_client, deliberately not decorated with @with_session_management (that decorator would try to log in while the BMC is still rebooting and defeat the wait).
  • bmc_fw_update.py — after the reset, waits a short settle interval (so a not-yet-rebooted BMC isn't mistaken for ready) and then calls bmc.wait_until_redfish_ready(). The previous ping-based wait loop is removed: a successful Redfish login implies L3 is back, so ping is redundant.
  • RedfishClient.login() gains a keyword-only log_errors flag (default True). When False, login failures are logged at notice instead of error, so the readiness poll's expected failures during the reboot window don't add error-level entries to the log. The default keeps behaviour unchanged for all existing callers.

Motivation and Context

When a BMC firmware update triggers a BMC restart, L3/ping recovers before the BMC's Redfish management plane. The previous post-reset wait only checked ping, so the update could report completion while the Redfish version-read API was still unavailable. Follow-on tooling that reads the BMC firmware version (for example fwutil show status) would then race the restart and log transient Redfish login errors during that window. Gating completion on a successful quiet Redfish login closes that window for callers serialized after the update, and the quiet-login flag keeps the readiness probe itself from adding error-level entries.

AUTH_FAILURE (HTTP 401 on the login POST) is treated as transient, since during BMC boot the web server can come up before the auth backend; the poll only fails fast on a local, unrecoverable error. The readiness wait is bounded by a timeout; if Redfish does not become ready within the bound, the update path logs an error and exits non-zero instead of hanging.

How Has This Been Tested?

Unit tests in tests/redfish_client_test.py, tests/bmc_base_test.py, and tests/bmc_fw_update_test.py cover wait_until_redfish_ready (first-try ready; ready after N retries; timeout returns the last non-OK code; fail-fast on a local permanent error; a transient auth failure keeps polling; sleep capped to the remaining budget; and a cached session not short-circuiting the probe), the BMCBase delegation, the quiet-vs-error login() logging, and the bmc_fw_update flow (settle + readiness call, success, and failure exit).

Additional Information (Optional)

  • The default log_errors=True preserves existing logging behaviour, so current RedfishClient.login() callers are unaffected.
  • This change targets the readiness window for callers serialized after the firmware update; hardening readers that race the restart on a separate command channel is out of scope.

Signed-off-by: Sonic Build Admin sonicbld@microsoft.com

#### Description

After a BMC firmware update the BMC is restarted to activate the new image. Ping/L3 connectivity returns noticeably earlier than the BMC's Redfish/HTTPS management plane, so the post-reset wait now gates completion on Redfish being usable rather than on ping.

The readiness logic is encapsulated so callers don't reach into `RedfishClient` directly:

- **`RedfishClient.wait_until_redfish_ready(timeout, interval)`** — polls a quiet `login()` until the Redfish service accepts it or the timeout elapses, returning a `RedfishClient` return code (`ERR_CODE_OK` when ready). It drops any cached session first via `invalidate_session()` (a local-only call that cannot emit an error log), so a cached token can't report a false "ready"; it sleeps `min(interval, remaining)` so it never overshoots the deadline; and it fails fast only on a local, clearly permanent error.
- **`BMCBase.wait_until_redfish_ready(...)`** — a thin delegator to `self.rf_client`, deliberately **not** decorated with `@with_session_management` (that decorator would try to log in while the BMC is still rebooting and defeat the wait).
- **`bmc_fw_update.py`** — after the reset, waits a short settle interval (so a not-yet-rebooted BMC isn't mistaken for ready) and then calls `bmc.wait_until_redfish_ready()`. The previous ping-based wait loop is removed: a successful Redfish login implies L3 is back, so ping is redundant.
- **`RedfishClient.login()`** gains a keyword-only `log_errors` flag (default `True`). When `False`, login failures are logged at notice instead of error, so the readiness poll's expected failures during the reboot window don't add error-level entries to the log. The default keeps behaviour unchanged for all existing callers.

#### Motivation and Context

When a BMC firmware update triggers a BMC restart, L3/ping recovers before the BMC's Redfish management plane. The previous post-reset wait only checked ping, so the update could report completion while the Redfish version-read API was still unavailable. Follow-on tooling that reads the BMC firmware version (for example `fwutil show status`) would then race the restart and log transient Redfish login errors during that window. Gating completion on a successful quiet Redfish login closes that window for callers serialized after the update, and the quiet-login flag keeps the readiness probe itself from adding error-level entries.

`AUTH_FAILURE` (HTTP 401 on the login POST) is treated as transient, since during BMC boot the web server can come up before the auth backend; the poll only fails fast on a local, unrecoverable error. The readiness wait is bounded by a timeout; if Redfish does not become ready within the bound, the update path logs an error and exits non-zero instead of hanging.

#### How Has This Been Tested?

Unit tests in `tests/redfish_client_test.py`, `tests/bmc_base_test.py`, and `tests/bmc_fw_update_test.py` cover `wait_until_redfish_ready` (first-try ready; ready after N retries; timeout returns the last non-OK code; fail-fast on a local permanent error; a transient auth failure keeps polling; sleep capped to the remaining budget; and a cached session not short-circuiting the probe), the `BMCBase` delegation, the quiet-vs-error `login()` logging, and the `bmc_fw_update` flow (settle + readiness call, success, and failure exit).

#### Additional Information (Optional)

- The default `log_errors=True` preserves existing logging behaviour, so current `RedfishClient.login()` callers are unaffected.
- This change targets the readiness window for callers serialized after the firmware update; hardening readers that race the restart on a separate command channel is out of scope.

Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Original PR: sonic-net/sonic-platform-common#707

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run

@yxieca
yxieca merged commit b3ee41d into Azure:202608 Jul 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants