Update the Operator to work with an existing instance of LiteLLM already running

[adamdolman]

Problem

Support attaching a LiteLLMInstance to an existing, externally-managed deployment

Ask

An opt-out that lets a LiteLLMInstance describe a LiteLLM deployment the operator did not create, so the entity CRDs can be used against an existing installation without handing the operator the workload.

We have this working today by blocking workload reconciliation with RBAC. It works well enough to be worth supporting properly — and badly enough to be adopted as a proper feature!

Use case

A LiteLLM proxy already exists and is managed by something else — a Helm chart, a GitOps pipeline, an internal platform, whatever. What's wanted from the operator is the entity CRDs: LiteLLMTeam, LiteLLMVirtualKey, LiteLLMBudget, LiteLLMModel. Declarative teams and keys reconciled from Kubernetes beats scripting the admin API.

What isn't wanted is for the operator to own the Deployment, as it already exists.

Today, the entity CRDs are unreachable, even though nothing about them depends on the operator owning a Deployment.

Observation

The entity controllers' entire dependency on the instance is resolveInstance (internal/controller/common.go), which needs three things:

• status.ready
• status.endpoint
• the master key from spec.masterKey.secretRef

Both status fields are derived from the instance name, not from anything the operator created:

// updateInstanceStatus
var dep appsv1.Deployment
if err := r.Get(ctx, types.NamespacedName{Name: instance.Name, Namespace: instance.Namespace}, &dep); err == nil {
    instance.Status.Ready = dep.Status.ReadyReplicas > 0
}
instance.Status.Endpoint = fmt.Sprintf("%s://%s.%s.svc:%d", scheme, instance.Name, instance.Namespace, port)

The Get carries no ownership check and the endpoint is a formula rather than a lookup. So a LiteLLMInstance named after an existing Deployment and Service already resolves correctly.

The only thing in the way is that reconcileResources runs first and adopts those objects by name — reconcileDeployment and reconcileService both Get-then-overwrite rather than declining to touch a resource they didn't create.

What we validated

We have got this working with an existing deployment, by using RBAC to prevent the operator from being able to deploy the instance. We specified the CR with the existing deployment's name, port, key etc, and the workload reconciliation was blocked by RBAC, but other than logging the failure proceeded to then work fine:

• status.ready: true and status.endpoint resolved against the existing deployment
• the existing Deployment, Service and ConfigMap were byte-identical before and after — same containers, same generation, no new ReplicaSet, no pod restart
• a LiteLLMTeam reached Synced with a real team ID from the admin API
• deleting that CR propagated through to LiteLLM — recreating it produced a new ID
• health probing still worked, since it only needs the endpoint

So the operator is entirely capable of this. It just has no way to be told to do it.
It also reports failure permanently: reconcileResources fails on every loop, so the instance sits Degraded and logs a Forbidden per resource forever. status.ready is true while the Ready condition is False with a ReconcileError. This doesn't stop it working, it just doesn't look nice, especially to someone unfamiliar with this workaround.

Proposed Solution

Proposal

apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: litellm
spec:
  workload:
    managed: false        # default true — operator provisions the workload as today
  masterKey:
    secretRef: {name: litellm-master-key, key: LITELLM_MASTER_KEY}
  service:
    port: 80

When managed: false, skip workload reconciliation and keep everything else:

var reconcileErr error
if instance.Spec.Workload == nil || instance.Spec.Workload.Managed {
    reconcileErr = r.reconcileResources(ctx, &instance, labels, licenseSecretName, guardrails)
    r.reconcileAutoRollback(ctx, &instance)
}
r.updateInstanceStatus(ctx, &instance, reconcileErr)

updateInstanceStatus still populates ready and endpoint, health probing still runs, and config sync is a separate controller making API calls, so none of it is affected. Status would describe the instance honestly instead of reporting Degraded about resources it doesn't manage.

Worth considering alongside

Resolving the endpoint from the instance name works, but it makes correctness depend on a naming coincidence — the CR has to be named after a Deployment and Service it doesn't own. Extend to specify the endpoint fully:

spec:
  workload:
    managed: false
    endpoint: http://my-litellm.my-namespace.svc:80   # optional; defaults to today's formula
    readinessRef:                                      # optional; defaults to metadata.name
      kind: Deployment
      name: my-litellm

That would also allow attaching to a proxy that isn't a Deployment — a StatefulSet, something outside the cluster, or a Service fronting either.

Alternatives Considered

No response

Additional Context

No response

[schneidermr]

Thanks for the unusually thorough write-up — I've checked the analysis against the code and it holds up line for line:

• updateInstanceStatus gets the Deployment by CR name with no ownership check (litellminstance_controller.go:1087-1091)
• the endpoint is a formula rather than a lookup (:1109)
• reconcileDeployment Get-then-overwrites rather than declining (:577-580); createOrUpdate (:1468) is blunter still and replaces the whole object
• resolveInstance really does depend on nothing but status.ready, status.endpoint and the master key (common.go:145-180)

Deletion is already safe, for what it's worth: handleDeletion only drops the finalizer and relies on ownerReferences, so with nothing owned there is nothing to clean up.

So this is accepted in principle — the operator can clearly do it, and spec.workload.managed is the right shape. Before building it I'd like to settle a few things that the proposal doesn't cover.

The migration Job needs guarding too. reconcileMigrationStatus is called at :114, before reconcileResources at :117, and the proposed guard doesn't cover it. With spec.database.migration.enabled: true the operator would create a Job running prisma migrate deploy against the database of a proxy someone else owns. Your RBAC workaround was blocking that; the feature as proposed would hand it back. This is the one part I'd call dangerous rather than untidy.

How much of the CRD goes inert. With no ConfigMap and no Deployment, everything that renders into proxy_server_config.yaml or Deployment env stops applying: sso, scim, jwtAuth, oauth2Auth, rbac, security, logging, adminUI, caching, routerSettings, callbacks, passThroughEndpoints, secretManager, license injection — and the entire LiteLLMGuardrail CRD, which is config-level and only materialises through the instance ConfigMap. That is a coherent scope (LiteLLMInstance becomes a connection descriptor plus health probe), but it needs to be explicit, and probably surfaced as a status condition rather than only documented. Does that match what you want from it, or were you expecting any of the config-level settings to keep working?

Pod health would report a comforting lie. inspectPodHealth (:961-966) selects on labelsForInstance(), which includes managed-by: litellm-operator. An externally-managed proxy's pods don't carry it, so PodsHealthy reads clean and unhealthyPods stays empty however badly the pods are doing. Better to skip it explicitly than report healthy.

masterKey.autoGenerate becomes a footgun. Combined with managed: false the operator would mint a key the running proxy has never seen, and every entity controller would get a 401 with no obvious cause. managed: false should require masterKey.secretRef.

I'd want endpoint and readinessRef in the first version, not as a follow-up. resolveInstance hard-fails on !status.Ready, and status.Ready comes from a Deployment Get by CR name — so the flag-only version works right until someone's proxy is a StatefulSet or simply named differently, and then the failure surfaces as instance "x" is not ready with nothing pointing at the naming coincidence. Your "worth considering alongside" section is what stops that being a trap.

No timeline on this yet — it needs the design settled first, and it is a bigger change than it looks from the diff. Two questions that would help:

1. In your setup, is the existing proxy a Deployment named the same as the CR, or were you working around the naming constraint?
2. Are you using any of the config-level features listed above, or is it purely the entity CRDs (LiteLLMTeam, LiteLLMVirtualKey, LiteLLMBudget, LiteLLMModel)?

[adamdolman]

Hi @schneidermr - the existing deployment was named as the CR as a coincidence!

Our use case was the entity CRDs to create Teams, Keys, Budgets and define models, to avoid calling the LiteLLM API itself to do the same. Day 2 jobs basically. The guardrail CRD is something that might be used in the future, I hadn't looked in to that yet.
We aren't using the other config features you mention here, as that is all set up in the deployment of the proxy as day 1 via a seed script. I didn't look in to how those could be made to work, of course there's value in them - but clearly a much more complicated job!

For a bit more info, we are using Crossplane to build XRDs spanning multiple providers and compositions that include deployment of new Teams/Keys on to LiteLLM. This today is concentrated on those 4 key ones.