Orphan managed resources when deleting GitOpsClaim

[irizzant]

Hi @vfarcic ,
how do you manage Crossplane managed resources deletion dependencies?

In my setup I have a cluster created with vcluster and I install a XR called GitOps which does exactly the same as your GitOpsClaim (install ArgoCD, install some projects, some applications...)

As far as I can see when you delete the GitOps XR Crossplane leaves orphaned object resources for ArgoCD Application objects like this.

Is there a way to get rid of them?

[vfarcic]

If I remember correctly, when I created that composition Argo CD did not yet have the option to change the tracking method. You should be able to fix it by adding application.resourceTrackingMethod: annotation.

Here's an example of Argo CD Helm values file I used recently: https://github.com/vfarcic/idp-demo/blob/main/argocd/helm-values.yaml#L6

[irizzant]

Hi @vfarcic
Thanks for the update, actually I am using that tracking annotation too.

Here you can see my crossplane XR https://github.com/irizzante/port-demo/blob/455fe2472ebdf288f1f09ec74ff624d0662bedb8/crossplane/working/gitops.yaml

And here the Argo configuration I use https://github.com/irizzante/port-demo/blob/455fe2472ebdf288f1f09ec74ff624d0662bedb8/argocd/values/argocd.yaml#L52

When I delete the GitOps claim I see this orphan object left
https://github.com/irizzante/port-demo/blob/455fe2472ebdf288f1f09ec74ff624d0662bedb8/crossplane/working/gitops.yaml#L227

And the status says that it cannot be deleted because the corresponding AppProject https://github.com/irizzante/port-demo/blob/455fe2472ebdf288f1f09ec74ff624d0662bedb8/crossplane/working/gitops.yaml#L197 doesn't exist anymore.

It looks like crossplane first deleted the AppProject leaving the Application orphaned.

[vfarcic]

I misunderstood your question. Sorry for that...

There is an issue in Crossplane with orphaned resources. Since there is no order (yet), when a resource where another resource should be running is deleted, Crossplane fails (rightfully) to delete the other. That should be solved soon in one of the upcoming releases. Until then, you might have to delete metadata.finalizers[]."finalizer.managedresource.crossplane.io". With that gone, Crossplane will delete the resource right away without waiting for the API on the other end to delete the destination resource.

[irizzant]

Hi @vfarcic
Thanks I think ordered resources deletion native in crossplane would be much better!

Removing the finalizers automatically is possible but it's tricky.
For example if I use a job to do it whenever users of my Port Idp trigger a delete action they have to wait until the job starts to see the resources are actually removed.

In one of "you choose" videos (which I love by the way) you suggested ArgoCD can also be set in the management cluster to deploy ArgoCD in other clusters created by crossplane instead of having one independent ArgoCD instance in each cluster.
This way GitOps setup in managed clusters would be managed by ArgoCD instead of crossplane.

Any other suggestions?
Thank you in advance

[vfarcic]

Argo CD can be set up centrally. However, there is a small issue with that. When you create a Kubernetes cluster from the management cluster, a secret is generate (in the management cluster). The issue is that the secret contains kubeconfig and Argo CD expects a different format (unlike Flux which does understand kubeconfig). So, some type of a transformation is needed from that secret to the one Argo CD understands. One option could be Kyverno which has Mutating Admission Controllers that can do just that.

[irizzant]

So, some type of a transformation is needed from that secret to the one Argo CD understands.

I found a workaround for this using ESO (external secret), see here https://github.com/irizzante/port-demo/blob/dd6ed3778d5494df272b11791489cf7de76f66fb/eso/kubernetes-provider.yaml

This provider configuration allows you to replicate a secret in the same cluster ESO is running using templates

[vfarcic]

I'm not sure that's it. The secret needs to be transformed from having kubeconfig to whatever Argo CD expects to be the format.

[irizzant]

I didn't report all the needed resources, sorry for that.

You can find the ESO SecretStore configuration here

The actual ExternalSecret object is here.
This ExternalSecret ensures that ArgoCD can see and manage the created clusters.

I believe this is a better approach than installing and managing multiple ArgoCD instances in crossplane created clusters, at least when it comes to clusters leaving in the same region or vclusters

[vfarcic]

Yeah. That's really good.