OCPBUGS-77924: cleanup orphaned boot entries in IPC

[danmanor]

Summary

• Add removeOrphanedBootEntries() to the IPC idle handler cleanup flow, which removes /boot/ostree/ directories that don't match any stateroot listed in
  rpm-ostree deployments
• This prevents storage exhaustion on /boot caused by stale boot entries (e.g. from the seed image or IBI) that have no corresponding deployed stateroot
• Fix pre-existing test failures in ipc_idle_handlers_test.go by overriding the osReadDir package variable for tests that exercise
  CleanupUnbootedStateroots

Details

On RHCOS formatted disks there is only room for 2 boot entries at the same time. We observed cases where an old boot entry from the seed image (dated 2022)
persisted in /boot/ostree/ with no matching stateroot in ostree admin status. This left no space for the IP Configuration flow to create its target
stateroot, causing failures in the pre-pivot phase.

The existing removeBootDirsByStaterootPrefixes() only cleans up boot entries for stateroots that appear as unbooted deployments in rpm-ostree. It does
not handle entries whose stateroot was already fully removed from all deployments (i.e. truly orphaned entries). The new removeOrphanedBootEntries()
function complements it by querying all deployed stateroots and removing any /boot/ostree/ directory that doesn't match any of them.

Test plan

• New unit tests for removeOrphanedBootEntries: basic removal, missing /boot/ostree/ dir, multiple deployments
• Updated mock expectations in existing integration tests for additional QueryStatus/ReadDir calls
• All TestIPCIdleStageHandler_Handle subtests pass
• Linter passes (make golangci-lint)

[openshift-ci-robot]

@danmanor: This pull request references Jira Issue OCPBUGS-77924, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references Jira Issue OCPBUGS-77924, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Add removeOrphanedBootEntries() to the IPC idle handler cleanup flow, which removes /boot/ostree/ directories that don't match any stateroot listed in
  rpm-ostree deployments
• This prevents storage exhaustion on /boot caused by stale boot entries (e.g. from the seed image or IBI) that have no corresponding deployed stateroot
• Fix pre-existing test failures in ipc_idle_handlers_test.go by overriding the osReadDir package variable for tests that exercise
  CleanupUnbootedStateroots

Details

On RHCOS formatted disks there is only room for 2 boot entries at the same time. We observed cases where an old boot entry from the seed image (dated 2022)
persisted in /boot/ostree/ with no matching stateroot in ostree admin status. This left no space for the IP Configuration flow to create its target
stateroot, causing failures in the pre-pivot phase.

The existing removeBootDirsByStaterootPrefixes() only cleans up boot entries for stateroots that appear as unbooted deployments in rpm-ostree. It does
not handle entries whose stateroot was already fully removed from all deployments (i.e. truly orphaned entries). The new removeOrphanedBootEntries()
function complements it by querying all deployed stateroots and removing any /boot/ostree/ directory that doesn't match any of them.

Test plan

• New unit tests for removeOrphanedBootEntries: basic removal, missing /boot/ostree/ dir, multiple deployments
• Updated mock expectations in existing integration tests for additional QueryStatus/ReadDir calls
• All TestIPCIdleStageHandler_Handle subtests pass
• Linter passes (make golangci-lint)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danmanor]

/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jc-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jc-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment