Add developer guide for CDC architecture and lifecycle management

docs/developer-guide/01-architecture.md

@@ -0,0 +1,49 @@
+# Architecture
+
+How the CDC is organized, how it boots, and — the question everyone asks first — **how a CDC instance knows which composition it is responsible for**.
+
+## The CDC supplies logic; the runtime owns the loop
+
+The CDC does not implement an informer, a work queue, or a reconcile loop. Those live in **unstructured-runtime**. The CDC implements one thing — the four reconcile operations (`Observe`, `Create`, `Update`, `Delete`) — registers them with the runtime, and lets the framework drive them. Everything operates on dynamic, untyped objects; the CDC knows nothing about the chart's types at compile time.
+
+## The main parts
+
+- **The handler** — implements the four operations. This is the entirety of the CDC's business logic.
+- **The chart resolver** — works out the chart's location, version, and credentials for a given resource: either a fixed chart (when one is configured) or, more usually, by reading it from the resource's `CompositionDefinition`.
+- **The chart-inspector client** — asks chart-inspector which resources the chart touches.
+- **The RBAC generator and installer** — turn that list into roles and bindings, and apply them.
+- **The release processor** — decodes the rendered release and computes a digest used to detect drift.
+
+## Component view
+
+```mermaid
+flowchart TB
+    UR[unstructured-runtime: informer + queue + workers] -->|drives| H[handler: Observe/Create/Update/Delete]
+    H -->|resolve chart| CD[CompositionDefinition]
+    H -->|ask which resources the chart touches| CI[chart-inspector]
+    H -->|generate + apply RBAC| K8s[(Kubernetes API)]
+    H -->|install / upgrade / uninstall| HELM[Helm engine]
+    HELM --> K8s
+```
+
+The authoritative reconcile flowchart is [`../../_diagrams/cdc-flow.puml`](../../_diagrams/cdc-flow.puml) (at the repository root).
+
+## How it boots
+
+Startup is short: read configuration, build a structured JSON logger, connect to the cluster, pick the chart resolver (fixed or dynamic), set up telemetry, build the label selector that scopes this controller to its composition version, construct the handler, and hand everything to the runtime to run.
+
+One important choice happens here: the CDC tells the runtime to treat an **update** to a watched resource as an **observe** event. So even a spec change first runs `Observe` — which reconciles Helm and decides whether anything is actually out of date — rather than jumping straight to an update path. See [`02-reconcile-lifecycle.md`](./02-reconcile-lifecycle.md).
+
+## How a CDC is launched, and which composition it manages
+
+A CDC instance is pinned to one composition type by **three things together**:
+
+1. **The watched resource type** — the group, version, and resource it is told to watch at launch. This is the informer's target.
+2. **A composition-version label selector** — so the controller only sees instances of *its* version. This is what lets multiple CDC versions coexist for the same kind.
+3. **The namespace** — all namespaces, or a specific one.
+
+core-provider sets all of these when it deploys the CDC, along with the chart-inspector URL and the identity the controller should use for the RBAC it generates.
+
+### Finding the CompositionDefinition
+
+Unless a fixed chart is configured, the chart's coordinates come from the `CompositionDefinition`, resolved at reconcile time. The resolver first tries labels on the resource that point directly at its definition; failing that, it lists the definitions and matches on the kind and version they report. With several candidate definitions and no disambiguating labels, it stops with an error rather than guessing. Once it has the definition, it reads the chart reference (URL, version, repository, credentials) from it.

docs/developer-guide/02-reconcile-lifecycle.md

@@ -0,0 +1,53 @@
+# Reconcile lifecycle
+
+What happens each time a `Composition` is reconciled — the Helm flow, drift detection — and the one behavior you must understand before changing anything: **`Observe` mutates the cluster**.
+
+The runtime calls the four operations; what `Observe` reports decides whether a create or update follows. Because the CDC treats an update as an observe (see [`01`](./01-architecture.md)), almost every event lands in `Observe`.
+
+## Observe — the center of gravity
+
+In order, `Observe`:
+
+1. **Computes and persists the release name** (sticky once set; by default it gets a unique suffix).
+2. **Honors graceful pause** — if the resource is paused, it returns immediately as "exists and up to date".
+3. **Resolves the chart** and records the `CompositionDefinition` labels on the resource.
+4. **Looks at the existing Helm release.** If there is none, it reports "doesn't exist" (which triggers a create). If the release is stuck in a pending state, it rolls back to the previous revision before continuing.
+5. **Scopes RBAC** — asks chart-inspector which resources the chart touches and applies the corresponding roles (see [`03-rbac-generation.md`](./03-rbac-generation.md)).
+6. **Builds the values** from the resource's spec, injects the global values, and prepares a post-renderer that stamps composition-ownership labels onto every rendered object.
+7. **Runs a Helm upgrade.** This actually changes the cluster on every reconcile — see the box below.
+8. **Detects drift** by computing a digest of the rendered release and comparing it to what was recorded. If it changed (or the chart version changed), it reports "out of date" (which triggers an update); otherwise it marks the composition available.
+
+> ### ⚠️ `Observe` mutates the cluster
+> The Helm upgrade in step 7 runs **unconditionally, on every reconcile, before** the drift comparison. The practical effect is roughly one Helm revision per reconcile, plus a steady stream of "up-to-date" logs. It is intentional today — Helm's three-way merge quietly corrects manual drift on the live objects as a side effect — but it is the single biggest footgun in this codebase. The release-history cap limits how many revisions are *kept*, not the churn. Read `docs/observe-reconciliation-loop-184.md` before touching `Observe`.
+
+## Create, Update, Delete
+
+- **Create** installs the chart (or upgrades an existing release), records the resulting objects in the status, and marks the composition available.
+- **Update** re-reads the release, recomputes the digest and the recorded objects, and updates status.
+- **Delete** uninstalls the release, confirms it is gone, then removes the generated RBAC. Finalizer handling is the runtime's job, not the handler's.
+
+## The Helm flow
+
+All Helm operations (get, install, upgrade, uninstall, rollback) go through the shared Helm library; the actual rendering and apply happen inside it. The CDC contributes two things: the **values** (the resource's spec plus a `global` block carrying the composition's identity and the Krateo namespace) and a **post-renderer** that labels every rendered object with composition ownership. Changing the labels or values applied to rendered objects is done at that post-renderer.
+
+## Drift detection
+
+Drift is detected by hashing the **rendered** manifest and comparing it to the recorded digest. This is rendered-vs-rendered, not rendered-vs-live: it does not directly notice edits to live objects — only the side-effecting upgrade (see the box) corrects those.
+
+## Adoption of existing resources
+
+The observable behavior is covered user-side in [Reconciliation & Lifecycle](https://docs.krateo.io). Two mechanism notes:
+
+- **Release by name.** `Observe`/`Create` look the release up by its computed name and `helm upgrade` it if present, so a re-triggered create is idempotent rather than a "release already exists" failure.
+- **Arbitrary live objects are not adopted.** The post-renderer only stamps composition-ownership labels; the CDC does **not** enable Helm's take-ownership / force / replace, so an object that already exists outside the release trips Helm's ownership check. The take-ownership option exists in the shared Helm library but is not wired into the CDC today.
+
+## Disabling specific operations (management & deletion policies)
+
+The CDC runs on unstructured-runtime, which honors the `krateo.io/management-policy` and `krateo.io/deletion-policy` annotations on the **`Composition`**. Mechanically these gate the create/update/delete events the reconcile loop would otherwise enqueue (`meta.ShouldCreate` / `ShouldUpdate` / `ShouldDelete` in unstructured-runtime). The values, their effects, and examples are user-facing — see [Lifecycle Policies](https://docs.krateo.io) and [Reconciliation & Lifecycle](https://docs.krateo.io) on docs.krateo.io.
+
+> ### ⚠️ `observe` / `observe-create-update` do not fully freeze an existing release
+> These policies suppress the **enqueued** create/update actions — but remember that **`Observe` itself runs a Helm upgrade** on every reconcile when a release already exists (see the box near the top of this page). The management policy does *not* gate that side-effecting upgrade, so a composition whose release already exists is **not** frozen by `observe`; the upgrade inside `Observe` still runs. (Create is genuinely prevented, because `Observe` reports "doesn't exist" and returns before upgrading when there is no release.) **To truly freeze a composition, use graceful pause**, which returns early from all four operations.
+
+## Graceful pause
+
+Graceful pause is a CDC-specific signal, distinct from the runtime's standard pause. When set, all four operations return early without touching the release, and the paused duration is tracked. It's how a composition can be frozen without being deleted.

docs/developer-guide/03-rbac-generation.md

@@ -0,0 +1,31 @@
+# RBAC generation
+
+How the CDC grants *itself* exactly the permissions a chart needs — by asking chart-inspector which resources the chart touches, turning that into roles, and applying them under its own identity. This runs on every reconcile, before the Helm upgrade.
+
+## The pipeline
+
+```mermaid
+sequenceDiagram
+    participant H as handler
+    participant RG as RBAC generator
+    participant CI as chart-inspector
+    participant INST as RBAC installer
+    participant K as Kubernetes API
+
+    H->>RG: generate the RBAC this chart needs
+    RG->>CI: which resources does this chart touch?
+    CI-->>RG: list of resources
+    RG->>RG: split cluster-scoped vs namespaced, build roles + bindings
+    RG-->>H: the roles to apply
+    H->>INST: apply them
+    INST->>K: create or merge cluster-scoped and per-namespace roles
+```
+
+1. **Ask chart-inspector** which resources the chart touches (see the **chart-inspector** developer guide; remember its result reflects what was *touched* and may contain duplicates).
+2. **Build roles** — a cluster-scoped role and binding for cluster-scoped resources, and a per-namespace role and binding for namespaced ones, all bound to the CDC's own identity. The grants are **broad** (all verbs on the reported resources) — this is the obvious place to tighten policy if you want least-verb rather than least-resource.
+3. **Apply them.** Application is **additive**: it only adds rules that aren't already present.
+
+## Two consequences worth flagging
+
+- **Permissions only grow during a composition's life.** Because application is additive, if a chart's resource set *shrinks*, the now-unnecessary rules are not removed — cleanup happens only when the composition is deleted.
+- **Identity coupling is load-bearing.** The identity configured for the bindings must be the same identity the CDC pod actually runs as. If they differ, the bindings grant permissions to a different identity than the pod uses, so the pod still can't manage the chart's resources even though the roles were "applied successfully." When debugging "forbidden" errors despite RBAC being present, check this first.

docs/developer-guide/04-extending.md

@@ -0,0 +1,18 @@
+# Extending the CDC
+
+Where to make the common changes. The CDC's logic is concentrated in a few places.
+
+| You want to… | Change |
+| --- | --- |
+| Change reconcile behavior | The four operations (`Observe` / `Create` / `Update` / `Delete`) — the single business-logic seam the runtime calls. |
+| Resolve chart coordinates differently | The chart resolver — provide a new source of a chart's location/version/credentials. A fixed-chart option already swaps in an alternative resolver. |
+| Tighten the generated RBAC | The RBAC generator — it grants all verbs today; narrow it here. |
+| Change labels or values on rendered objects | The post-renderer and value injection (in the shared Helm library). |
+| Change which event a watched change triggers | The controller's launch options (for example, the update-becomes-observe remap). |
+| Add a configuration knob | A new flag/env at startup. |
+
+Keep the four operations **idempotent and non-blocking** — the runtime can re-enqueue and retry them, so a create must tolerate an already-existing release and a delete a missing one.
+
+## Worked example: a new chart-coordinate source
+
+Suppose chart coordinates should come from a ConfigMap rather than the `CompositionDefinition`. The shape of the change is: implement a new chart resolver that reads the ConfigMap referenced by the resource and produces the same chart-coordinate result the rest of the pipeline expects, then select it at startup (the same way the fixed-chart option is selected today). Nothing downstream changes, because the reconcile flow only depends on the resolved coordinates, not on where they came from.

docs/developer-guide/README.md

@@ -0,0 +1,36 @@
+# composition-dynamic-controller (CDC) — Developer Guide
+
+A contributor-facing guide to **Krateo's execution engine**: the controller that turns each `Composition` instance into a reconciled Helm release.
+
+> Audience: engineers who **contribute to, extend, or debug the CDC** — not end users. This guide explains *ideas and flows*, not line-by-line code. For product concepts, see [docs.krateo.io](https://docs.krateo.io/key-concepts/kco/cdc/overview).
+
+## Role in KCO
+
+KCO turns Helm charts into Kubernetes-native APIs. **core-provider** generates a CRD from a chart's values schema and deploys, **per `CompositionDefinition`, one CDC** — this repo — telling it which resource to watch. The CDC (built on **unstructured-runtime**) watches that single resource; for each `Composition` instance it resolves the chart, renders it using the instance's spec as Helm values, and applies it as a Helm release, then keeps it reconciled. To scope its own **least-privilege RBAC**, the CDC asks **chart-inspector** which resources the chart touches and generates roles for exactly those.
+
+```mermaid
+flowchart LR
+    K8s[(Kubernetes API)] -->|watch one resource type| CDC[composition-dynamic-controller]
+    CDC -->|asks which resources the chart touches| CI[chart-inspector]
+    CDC -->|apply RBAC| K8s
+    CDC -->|render chart with the spec, install/upgrade| Release[(Helm release)]
+```
+
+The CDC is **not composition-specific at compile time** — a single binary watches whatever resource it is told to. Everything composition-specific comes from the chart and from how the controller is launched. The reconcile loop itself lives in **unstructured-runtime**; the CDC supplies only the business logic. See [`01-architecture.md`](./01-architecture.md).
+
+## Documents in this folder
+
+| Document | What it covers |
+| --- | --- |
+| [`01-architecture.md`](./01-architecture.md) | The main parts, how the controller boots, and **how a CDC is launched / which composition it manages**. |
+| [`02-reconcile-lifecycle.md`](./02-reconcile-lifecycle.md) | What happens on each reconcile, the Helm flow, drift detection, graceful pause, and the **Observe-mutation** behavior. |
+| [`03-rbac-generation.md`](./03-rbac-generation.md) | How the CDC scopes its own permissions per chart, and the identity coupling that makes it work. |
+| [`04-extending.md`](./04-extending.md) | Where to make common changes. |
+
+## See also
+
+- **Ecosystem overview (canonical)** — the whole KCO pipeline lives in the **core-provider** repo: `core-provider/docs/developer-guide/00-ecosystem-overview.md`.
+- **The framework** — the CDC is built on **unstructured-runtime**; its developer guide documents the reconcile loop and the contract the CDC plugs into.
+- **Logging** — `docs/logs-ingester-compatibility.md`. **Telemetry / metrics** — `telemetry/`.
+- **Design note** — `docs/observe-reconciliation-loop-184.md` analyses the Observe-mutation behavior discussed in [`02`](./02-reconcile-lifecycle.md).
+- **Reconcile flow diagram (source)** — [`../../_diagrams/cdc-flow.puml`](../../_diagrams/cdc-flow.puml) (at the repository root).