Skip to content

[release-4.19] Avoid using lb-ext.kubeconfig for seedgen cleanup#6009

Closed
openshift-cherrypick-robot wants to merge 2 commits into
openshift-kni:release-4.19from
openshift-cherrypick-robot:cherry-pick-6008-to-release-4.19
Closed

[release-4.19] Avoid using lb-ext.kubeconfig for seedgen cleanup#6009
openshift-cherrypick-robot wants to merge 2 commits into
openshift-kni:release-4.19from
openshift-cherrypick-robot:cherry-pick-6008-to-release-4.19

Conversation

@openshift-cherrypick-robot
Copy link
Copy Markdown

This is an automated cherry-pick of #6008

/assign jc-rh

/cherrypick release-4.18 release-4.17 release-4.16

omertuc and others added 2 commits April 19, 2026 14:10
# Background

During cleanup before seed generation, we want to delete all pods in the
Succeeded or Failed phase.

# Issue

The current implementation uses `oc delete pod --all-namespaces
--field-selector=status.phase==...` with the node `lb-ext.kubeconfig` as
a kubeconfig. This is in contrast to the rest of the controller, which
uses the in-cluster config via a client loaded with
ctrl.GetConfigOrDie().

We've had users report TLS errors when the controller tries to use the
lb-ext.kubeconfig, which seems expected since that kubeconfig is meant
for external use and may not have the correct CA or credentials for
in-cluster API access.

Before they hit that error, the client has already been used, so it
proves that the in-cluster config works, and this is specifically an
issue with using the lb-ext.kubeconfig for this cleanup step.

# Solution

The solution is to avoid using the lb-ext.kubeconfig for this cleanup
step, and instead use the same in-cluster client that the rest of the
controller uses.

# Caveat

As the existing TODO states:

```go
// TODO: Can this be done cleanly via client? The client.DeleteAllOf
seems to require a specified namespace, so maybe loop over the
namespaces
```

The Kubernetes API does not support DeleteCollection without a namespace
for namespaced resources, so we cannot simply use DeleteAllOf with no
namespace. Instead, we copy the behavior of kubectl when using `delete
--all-namespaces`, which is to list the resources with the field
selector, and then delete them one by one. This is less efficient than a
single DeleteCollection call, but it's already what kubectl was doing
when we were calling it, so we know it works and is acceptable for this
use case.

# Implementation Details

We cannot use `client.MatchingFields` because we do not have an index
set up and it seems overkill to do so.

# Agent Interaction

Agent assisted. The conversation went something like this:

- Where does r.Client get its kubeconfig? → ctrl.GetConfigOrDie() → in-cluster SA
- Where does the oc call get its kubeconfig? → lb-ext.kubeconfig on the host via chroot executor
- Can we generate a kubeconfig from rest.Config? → no built-in utility, hand-rolled versions kept being incomplete
- Just use DeleteAllOf with no namespace? → tried it, in a toy project using `sigs.k8s.io/controller-runtime/pkg/envtest`, API rejects it (no cluster-scoped DeleteCollection for namespaced resources)
- Inspect kubectl source - what does it do when delete --all-namespaces? → list + delete one by one, sequentially
- Did that, modified toy project to prove it works, updated the controller, added comment explaining the reasoning
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 19, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign donpenney for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented Apr 20, 2026

/test ibi-e2e-flow ibu-e2e-flow

6 similar comments
@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented Apr 20, 2026

/test ibi-e2e-flow ibu-e2e-flow

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented Apr 21, 2026

/test ibi-e2e-flow ibu-e2e-flow

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented Apr 21, 2026

/test ibi-e2e-flow ibu-e2e-flow

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented May 5, 2026

/test ibi-e2e-flow ibu-e2e-flow

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented May 13, 2026

/test ibi-e2e-flow ibu-e2e-flow

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented May 15, 2026

/test ibi-e2e-flow ibu-e2e-flow

@jc-rh
Copy link
Copy Markdown
Member

jc-rh commented May 15, 2026

/test ibi-e2e-flow

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 15, 2026

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/ibu-e2e-flow 77b7108 link false /test ibu-e2e-flow
ci/prow/ibi-e2e-flow 77b7108 link false /test ibi-e2e-flow

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@omertuc
Copy link
Copy Markdown
Collaborator

omertuc commented May 26, 2026

Obsoleted by #6881

@omertuc omertuc closed this May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants