diff --git a/architecture/adr/001-why-dcm-exists.md b/architecture/adr/001-why-dcm-exists.md new file mode 100644 index 0000000..7c1bf56 --- /dev/null +++ b/architecture/adr/001-why-dcm-exists.md @@ -0,0 +1,31 @@ +# ADR-001: Why DCM Exists + +**Status:** Accepted +**Date:** March 2026 + +## Context + +Enterprise data centers run hundreds of thousands of resources — VMs, containers, network segments, storage volumes — across multiple infrastructure platforms. Today, each platform has its own provisioning workflow, API, data format, and lifecycle model. The result: + +- **No unified view of what's deployed.** Intended state, deployed state, and actual state diverge silently. Nobody can answer "what's running, who owns it, and does it match what was approved?" +- **No consistent governance.** Policy enforcement is tribal knowledge. Security reviews are manual gates. Compliance is verified after the fact rather than enforced at request time. +- **No common abstraction.** A team requesting a VM goes through one process; requesting a database goes through another; requesting a three-tier application requires manually coordinating both plus networking. + +Public cloud solves this with unified control planes (AWS CloudFormation, Azure Resource Manager, GCP Deployment Manager). On-premises infrastructure has no equivalent. + +## Decision + +Build DCM — a management plane for enterprise data center infrastructure that provides: +- A unified data model and API across all infrastructure platforms +- Policy-as-code enforcement on every request before provisioning +- Full lifecycle management from request through decommission with tamper-evident audit +- A provider abstraction that makes any infrastructure platform consumable through the same interface + +DCM is **not** a provisioning tool. It is the governance and orchestration layer that sits above provisioning tools (Ansible, Terraform, operators) and governs what gets requested, approved, built, owned, and decommissioned. + +## Consequences + +- DCM must be infrastructure-agnostic — it cannot favor any single platform +- The data model must be extensible to any resource type without code changes +- Policy evaluation must be mandatory, not optional — governance is the value proposition +- Audit must be tamper-evident to satisfy regulated environments (the primary adopters) diff --git a/architecture/adr/002-three-abstractions.md b/architecture/adr/002-three-abstractions.md new file mode 100644 index 0000000..fc07867 --- /dev/null +++ b/architecture/adr/002-three-abstractions.md @@ -0,0 +1,27 @@ +# ADR-002: Three Foundational Abstractions — Data, Provider, Policy + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Foundations (UDLM) + +## Context + +A management plane for infrastructure must handle many concerns: data storage, external integrations, governance rules, audit trails, placement decisions, dependency resolution, lifecycle events, and more. Without a unifying model, the architecture becomes a collection of ad-hoc services with unclear boundaries. + +## Decision + +Every component of DCM maps to exactly one of three foundational abstractions: + +**DATA** — Everything stored and versioned. The unified data model, entity lifecycle states, field-level provenance, data layers, and audit records. Data flows through a deterministic pipeline: Intent → Requested → Realized → Discovered. + +**PROVIDER** — Everything external. Any system DCM interacts with through a defined contract. Providers receive data from DCM, act on it, and return data to DCM. Providers are distinguished by declared CAPABILITY, not a fixed type enum (**superseded on this point by ADR-005**, which abolished rigid provider types; the UDLM provider-contract kinds are service / information / process / peer_dcm, with auth expressed as a capability). The earlier 'six provider types' framing (service, information, meta, auth, peer_dcm, process) is retained here only as history. + +**POLICY** — Everything that decides. Rules that fire when data matches conditions and produce typed outputs: allow/deny, validation, field mutations, recovery actions, orchestration directives. Policies govern every transition and transformation in DCM. + +The interaction model: Data changes trigger Policy evaluation. Policy decisions may mutate Data or select Providers. Providers produce new Data. The cycle repeats. + +## Consequences + +- Any new capability must map to one of these three abstractions — if it doesn't fit, the abstraction model needs revision, not a fourth pillar +- Documentation, APIs, and code are organized around these three concepts +- Team members only need deep knowledge of 1-2 abstractions for their area of work diff --git a/architecture/adr/003-four-lifecycle-states.md b/architecture/adr/003-four-lifecycle-states.md new file mode 100644 index 0000000..53b7ca4 --- /dev/null +++ b/architecture/adr/003-four-lifecycle-states.md @@ -0,0 +1,29 @@ +# ADR-003: Four Lifecycle States + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Four States (UDLM) + +## Context + +A resource entity goes through multiple stages: the consumer declares intent, the system processes and approves, the provider provisions, and discovery observes what actually exists. If we track this as a single mutable record, we lose the ability to answer: "What did they ask for? What did we approve? What got built? What exists now?" + +These four questions are the foundation of governance, audit, compliance, and drift detection. + +## Decision + +Every resource entity flows through four immutable states: + +1. **Intent** — What the consumer asked for (raw declaration, no processing) +2. **Requested** — What was approved after layer assembly and policy evaluation (write-once) +3. **Realized** — What the provider actually created (snapshot from provider callback) +4. **Discovered** — What exists right now (independent observation via polling) + +The `entity_uuid` links all four states for the same resource. States are immutable — updates create new records. Drift is the delta between Realized and Discovered. Compliance is provable because Requested State records the policy-approved payload. + +## Consequences + +- Every resource has exactly 4 records linked by entity_uuid +- Drift detection is a comparison: Realized ≠ Discovered +- Rehydration (disaster recovery) re-enters at Intent with current policies +- Audit can trace any resource from consumer's original ask through to what's running diff --git a/architecture/adr/004-service-catalog-consumer-experience.md b/architecture/adr/004-service-catalog-consumer-experience.md new file mode 100644 index 0000000..3027ce7 --- /dev/null +++ b/architecture/adr/004-service-catalog-consumer-experience.md @@ -0,0 +1,38 @@ +# ADR-004: Service Catalog and Consumer Experience + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Resource Type Hierarchy (UDLM), Resource/Service Entities (UDLM) + +## Context + +Consumers need a way to discover what services are available and request them. The service catalog must abstract away infrastructure complexity — a developer requesting a VM should not need to know which hypervisor, which datacenter, or which network configuration is required. + +## Decision + +A four-level hierarchy separates what consumers see from what providers implement: + +1. **Resource Type Category** — Broad groupings (Compute, Network, Storage, Database) +2. **Resource Type** — Specific resource kinds (Compute.VirtualMachine, Network.VLAN) +3. **Resource Type Specification** — Vendor-neutral field schemas, constraints, lifecycle rules +4. **Provider Catalog Item** — A specific provider's offering (pricing, SLAs, availability) + +Consumers browse the catalog, select a catalog item, and submit a request with only the fields they care about (e.g., CPU count, memory, OS). DCM handles everything else: layer assembly, policy evaluation, provider selection, dependency resolution. + +**Consumer request surface** is a JSON payload via the Consumer API: + +```json +POST /api/v1/requests +{ "catalog_item_uuid": "...", "fields": { "cpu_count": 4, "memory_gb": 8, "os_family": "rhel" } } +``` + +## Open Question — Application Definition Language + +The current consumer interface is an API call with a JSON payload. This works for single resources. For multi-resource applications (three-tier web app, data pipeline, ML training environment), the consumer needs a way to define the application as a whole. This is an open design question — see [ADR-016: Application Definition Language](016-application-definition-language.md). + +## Consequences + +- Resource types are vendor-neutral; provider catalog items are provider-specific +- Multiple providers can offer catalog items for the same resource type +- Consumers never choose a provider directly — placement does that +- The catalog is queryable via API; RHDH provides the frontend diff --git a/architecture/adr/005-provider-abstraction.md b/architecture/adr/005-provider-abstraction.md new file mode 100644 index 0000000..984a4c4 --- /dev/null +++ b/architecture/adr/005-provider-abstraction.md @@ -0,0 +1,51 @@ +# ADR-005: Why Providers Exist and What They Do + +**Status:** Accepted +**Date:** April 2026 +**Docs:** Doc A (Provider Contract), Capability Discovery (UDLM) + +## Context + +DCM must interact with many external systems: hypervisors, container platforms, network controllers, IPAM systems, identity services, other DCM instances, ITSM tools, FinOps platforms, and more. Each has its own API, data format, and operational model. Without a common abstraction, DCM becomes tightly coupled to specific infrastructure platforms. + +## Decision + +A **Provider** is any external system DCM interacts with through a defined contract. All providers share the same base contract: registration, health check, sovereignty declaration, accreditation, zero trust authentication, and provenance emission. + +What varies is the **capabilities** the provider declares. Capabilities define what the provider can do — not a rigid type assignment, but a profile of operations: + +| Capability | What it means | Example | +|-----------|--------------|---------| +| `realize_resources` | Provisions, updates, and decommissions infrastructure resources | OpenStack Nova, KubeVirt, ACM | +| `serve_data` | Responds to queries with authoritative external data | CMDB, DNS, IPAM (InfoBlox) | +| `authenticate` | Authenticates identities and returns tokens/roles/groups | Keycloak, LDAP, FreeIPA | +| `federate` | Another DCM instance — mTLS mandatory, dual audit | Cross-region DCM | +| `execute_workflows` | Runs ephemeral workflows without producing persistent resources | Approval chains, ITSM, runbooks | + +**A provider can declare multiple capabilities.** An IPAM system that both serves IP availability data AND allocates IP addresses registers once with `capabilities: [serve_data, realize_resources]` — not twice as two separate providers. + +The key mechanism is **Naturalization/Denaturalization**: DCM sends a unified payload to the provider. The provider translates (naturalizes) it into its native API format, acts on it, then translates (denaturalizes) the result back into DCM's unified format. + +## Capability Discovery + +DCM and providers discover each other's capabilities bidirectionally: + +- **DCM advertises** its capabilities via `GET /api/v1/capabilities` — external systems query what DCM offers (cost data, audit trail, entity lifecycle events, placement decisions) and subscribe to data streams automatically +- **Providers declare** what they offer to DCM (capabilities) AND what they need from DCM (data streams, events) at registration time. DCM matches needs to available capabilities and offers subscription endpoints. + +This replaces the old one-directional model where providers register with DCM but DCM doesn't advertise anything back. + +## Alternatives Considered + +1. **12 provider types** (original design) — rejected because credential, notification, message bus, registry, storage, meta, policy, and ITSM providers were implementation details or data concepts, not architectural abstractions +2. **5 rigid types** (interim design) — rejected because it still forced providers into exactly one type, preventing multi-capability providers and providing no discovery mechanism +3. **Unified model with capability declarations** (current) — one provider type with capability profiles, bidirectional discovery, and automatic pipeline establishment + +## Consequences + +- Adding a new infrastructure platform means writing one provider — not changing DCM core +- Consumers don't know or care which provider fulfills their request +- Provider selection is policy-driven (placement), not consumer-chosen +- All provider interactions are audited and sovereignty-checked +- Multi-capability providers register once, not once per capability +- External systems discover DCM's data streams without reading docs diff --git a/architecture/adr/006-policy-engine.md b/architecture/adr/006-policy-engine.md new file mode 100644 index 0000000..c777d5f --- /dev/null +++ b/architecture/adr/006-policy-engine.md @@ -0,0 +1,36 @@ +# ADR-006: Why Policy-as-Code and What It Governs + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Doc B (Policy Contract) + +## Context + +Enterprise infrastructure requires governance: sizing limits, security constraints, compliance rules, sovereignty requirements, cost controls, naming conventions. Today this governance is tribal knowledge enforced by manual review gates. Manual gates are slow, inconsistent, and unauditable. + +## Decision + +Every request is policy-evaluated before provisioning. Policies are code artifacts (Rego), not configuration. They fire automatically when data matches conditions and produce typed outputs. + +**What policies govern:** +- **Who can request what** (Gating Policy: allow/deny based on role, tenant, resource type) +- **Whether the request is valid** (Validation: field constraints, range checks, format) +- **How the request is enriched** (Transformation: inject monitoring agents, set backup policies, apply naming conventions) +- **What happens when things fail** (Recovery: retry, requeue, compensate) +- **How pipeline stages are ordered** (Orchestration Flow: dependency sequencing) +- **What crosses boundaries** (Governance Matrix: sovereignty, data classification) +- **Lifecycle-stage behavior** (Lifecycle Policy: per-operation rules over the entity lifecycle) + +That is the **7 base typed policies**; ADR-019 adds **Placement Policy** as the 8th. The README's "8 policy types" = these 7 + Placement. + +**Key design choices:** +- Multi-pass evaluation with convergence — transformation policies can inject fields that other policies depend on +- Lifecycle-scoped — a CPU-sizing policy fires on provisioning and scaling, not on hostname changes +- Override model with 5 mechanisms — governance is not rigid; legitimate exceptions are handled through audited overrides + +## Consequences + +- No request bypasses policy evaluation — this is mandatory, not opt-in +- Policies are versioned, have lifecycle (developing → active → retired), and support shadow mode for safe testing +- Every policy evaluation produces an audit record regardless of outcome +- Policy complexity is managed through templates (Gatekeeper ConstraintTemplate pattern) and a Constraint Type Registry diff --git a/architecture/adr/007-placement-engine.md b/architecture/adr/007-placement-engine.md new file mode 100644 index 0000000..344391d --- /dev/null +++ b/architecture/adr/007-placement-engine.md @@ -0,0 +1,32 @@ +# ADR-007: How DCM Decides Where Things Run + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Placement (UDLM), Profiles (UDLM) + +## Context + +When a consumer requests a VM, they don't specify which provider or datacenter. Multiple providers may be capable of fulfilling the request. DCM must select the best provider based on sovereignty requirements, capacity, compliance, cost, and organizational policy. + +## Decision + +The Placement Engine selects providers through a multi-stage scoring process: + +1. **Sovereignty pre-filter** — Eliminate providers that don't satisfy data residency requirements (e.g., EU-WEST resources can only go to EU-WEST providers). This is a hard gate, not a score. + +2. **Capability filter** — Eliminate providers that don't support the requested resource type or lack required capabilities. + +3. **Reserve query** — Query remaining providers for capacity availability and get confidence scores. + +4. **Policy-driven scoring** — Apply placement policies that score providers on criteria like cost, performance tier, organizational preference, and existing affinity (e.g., co-locate with related resources). + +5. **Selection** — Highest-scoring provider wins. Ties broken by configurable rules. + +For composite services (composite resource type specifications), placement runs per-constituent — the database may land on a different provider than the app server, each scored independently but subject to the same sovereignty constraints. + +## Consequences + +- Consumers never choose providers — placement is always policy-driven +- Adding new providers to a zone automatically makes them candidates for placement +- Placement decisions are audited with full scoring rationale +- Provider health affects placement — unhealthy providers are excluded diff --git a/architecture/adr/008-dependency-resolution.md b/architecture/adr/008-dependency-resolution.md new file mode 100644 index 0000000..3a3500b --- /dev/null +++ b/architecture/adr/008-dependency-resolution.md @@ -0,0 +1,32 @@ +# ADR-008: How Resources Know What They Need + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Service Dependencies (UDLM), Composite Service Composition Model (UDLM) + +## Context + +Infrastructure resources have dependencies. A VM needs an IP address. A database needs a network port. A three-tier application needs all of its components provisioned in the right order with runtime values (IP addresses, connection strings) flowing from one resource to the next. + +## Decision + +Dependencies are declared at two levels: + +**Type-level** (in the Resource Type Specification): "Every VM requires exactly one IP address." These are portable, provider-agnostic, and apply to all implementations of the resource type. DCM automatically creates sub-requests for type-level dependencies. + +**Binding fields** (in composite service definitions): "The backend's db_host field gets its value from the database's realized ip_address." These connect resources via runtime values — the output of one resource becomes the input of another. + +**How it works:** +1. Request Processor reads the resource type spec and identifies dependencies +2. Dependencies without parents are dispatched first (topological sort) +3. When a dependency is realized, its output values are injected into dependent resources via dependency payload passing (with full provenance tracking) +4. Dependent resources are dispatched after their dependencies are satisfied + +For composite services, the composite resource type spec declares the full dependency graph with binding fields. + +## Consequences + +- Consumers don't manage dependencies — they request a catalog item and DCM resolves the graph +- Each dependency is a first-class DCM entity with its own audit trail and lifecycle +- Decommission reverses the dependency order — dependents are torn down before their dependencies +- Circular dependencies are detected at resource type registration time, not at request time diff --git a/architecture/adr/009-api-gateway-control-plane.md b/architecture/adr/009-api-gateway-control-plane.md new file mode 100644 index 0000000..ad25310 --- /dev/null +++ b/architecture/adr/009-api-gateway-control-plane.md @@ -0,0 +1,39 @@ +# ADR-009: Why an API Gateway and What the Control Plane Services Do + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Control Plane Services (UDLM), OpenAPI Specs + +## Context + +DCM has multiple consumers (developers, platform engineers, admins, providers, external systems) that interact via different APIs with different authorization scopes. Internally, DCM has multiple services that process requests through a pipeline. These services need a single entry point that handles authentication, routing, rate limiting, and API versioning. + +## Decision + +The **API Gateway** is the single entry point for all external traffic. It handles: +- Authentication (JWT validation, API key verification) +- Route multiplexing (consumer API, admin API, provider callback API) +- Rate limiting and throttling per tenant +- TLS termination +- API versioning (v1, v1alpha1) + +Behind the gateway, **9 control plane services** process requests through the pipeline: + +| Service | What it does | +|---------|-------------| +| API Gateway | Routes external traffic to internal services | +| Catalog Manager | Serves the service catalog and resource type registry | +| Request Processor | Assembles layers, resolves dependencies, builds requested state | +| Policy Engine | Evaluates all matching policies against the request payload | +| Placement Engine | Scores and selects providers for fulfillment | +| Request Orchestrator | Dispatches to providers, manages async callbacks, handles retries | +| Audit Service | Records tamper-evident audit trail with Merkle tree | +| Discovery Service | Polls providers for current state, detects drift | +| Provider Manager | Manages provider registration, health monitoring, sovereignty declarations | + +## Consequences + +- All external traffic goes through one endpoint — simplifies network policy and TLS +- Services communicate internally via direct calls or PostgreSQL LISTEN/NOTIFY +- Each service has its own health endpoint and can be scaled independently +- The pipeline is deterministic: assembly → policy → placement → dispatch → callback diff --git a/architecture/adr/010-audit-tamper-evidence.md b/architecture/adr/010-audit-tamper-evidence.md new file mode 100644 index 0000000..7a3b7ab --- /dev/null +++ b/architecture/adr/010-audit-tamper-evidence.md @@ -0,0 +1,31 @@ +# ADR-010: Why Tamper-Evident Audit and How It Works + +**Status:** Accepted +**Date:** April 2026 +**Docs:** Universal Audit (UDLM) + +## Context + +Regulated industries (financial services, government, healthcare) require provable audit trails. "We logged it" is insufficient — auditors need mathematical proof that records haven't been modified or deleted after the fact. This is a hard requirement for sovereign cloud deployments. + +## Decision + +DCM uses a **Merkle tree** audit model (RFC 9162 — the same pattern used in Certificate Transparency): + +- Every pipeline stage produces a signed audit record (Ed25519 signature) +- Records are leaves in a Merkle tree — a binary hash tree where modifying any leaf changes the root hash +- **Inclusion proofs** prove a specific record exists in the tree +- **Consistency proofs** prove the tree has only grown (no deletions) +- **Signed tree heads** provide non-repudiation by the DCM instance + +**Configurable granularity** because not every deployment needs the same detail: +- **Stage** (~6 leaves/request): one leaf per pipeline stage — sufficient for dev/homelab +- **Mutation** (~15-30 leaves/request): one leaf per field change — standard for production +- **Field** (mutation + per-field hashes): required for FedRAMP/sovereign deployments + +## Consequences + +- Any modification to audit records is mathematically detectable +- Auditors can independently verify the audit trail without trusting DCM +- Granularity is profile-governed — organizations choose their audit depth +- Three SQL tables support the model: audit_records, signed_tree_heads, merkle_tree_nodes diff --git a/architecture/adr/011-sovereignty-data-residency.md b/architecture/adr/011-sovereignty-data-residency.md new file mode 100644 index 0000000..74b8b93 --- /dev/null +++ b/architecture/adr/011-sovereignty-data-residency.md @@ -0,0 +1,28 @@ +# ADR-011: Why Sovereignty Is a First-Class Concept + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Profiles (UDLM), Policy Contract §18 Overrides (UDLM), Governance Matrix (UDLM) + +## Context + +Organizations operating in regulated industries or across jurisdictions face data residency requirements: EU data must stay in EU, classified data must stay on approved infrastructure, healthcare data must meet HIPAA locality requirements. Public clouds handle this with regions. On-premises infrastructure has no equivalent enforcement mechanism. + +## Decision + +Sovereignty is enforced at three levels: + +1. **Provider declaration** — Every provider declares its sovereignty zones and data residency scope at registration. This is not self-reported trust — it's validated against the accreditation model. + +2. **Policy enforcement** — Sovereignty spans THREE policy homes after ADR-019/020 (no single home): **Gating** (hard allow/deny on placement zone, this section), **Governance-Matrix** (cross-boundary + migration permission, ADR-020), and **Placement Policy** (residency as a placement constraint, ADR-019). Sovereignty policies are Gating policies with hard enforcement. They fire on every lifecycle operation (not just initial provisioning). A resource in EU-WEST stays in EU-WEST for its entire lifecycle, including updates, scaling, and rehydration. + +3. **Placement pre-filter** — The placement engine eliminates non-compliant providers before scoring begins. Sovereignty is a hard gate, not a soft preference. + +**Override governance:** Sovereignty policies can be overridden, but only through dual-approval (two approvers from different roles). Every override is audited at field granularity. + +## Consequences + +- Sovereignty violations are caught at request time, not after deployment +- Cross-zone data movement is impossible without explicit, audited override +- Rehydration (disaster recovery) respects current sovereignty policies — rebuilding in a non-compliant zone is blocked +- Profiles (minimal, standard, fsi, sovereign) set sovereignty enforcement minimums diff --git a/architecture/adr/012-data-assembly-layering.md b/architecture/adr/012-data-assembly-layering.md new file mode 100644 index 0000000..fc74103 --- /dev/null +++ b/architecture/adr/012-data-assembly-layering.md @@ -0,0 +1,29 @@ +# ADR-012: How Organizational Data Merges with Consumer Requests + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Layering and Versioning (UDLM) + +## Context + +When a consumer requests a VM with 4 CPUs, the provisioning system needs much more information: which datacenter, which network, what monitoring agent, what backup policy, what compliance requirements apply. This organizational data shouldn't be the consumer's responsibility — they just want a VM. + +## Decision + +**Data Layers** carry organizational context that gets merged into every request: + +- **System layers** — Datacenter configurations, environment defaults, compliance requirements +- **Tenant layers** — Organization-specific overrides (monitoring agents, naming conventions) +- **Provider layers** — Provider-specific defaults (image mappings, flavor resolution) +- **Consumer intent** — What the consumer actually asked for + +Layers merge in precedence order (system → tenant → provider → consumer). Consumer values override layer defaults. Every field in the merged payload carries **provenance** — where the value came from and what modified it. + +**Layers are Data, not Logic.** Layers provide values. Policies provide decisions. A layer says "the datacenter is EU-WEST-DC1." A policy says "EU-WEST resources must use the EU-WEST monitoring endpoint." This separation means layers can be managed by infrastructure teams while policies are managed by security/governance teams. + +## Consequences + +- Consumers declare only what they need — organizational data is injected automatically +- Adding a new datacenter or changing a monitoring agent is a layer change, not a code change +- Provenance on every field answers "why does this VM have this backup policy?" +- Layer conflicts are resolved deterministically by precedence order diff --git a/architecture/adr/013-override-exception-governance.md b/architecture/adr/013-override-exception-governance.md new file mode 100644 index 0000000..9922d43 --- /dev/null +++ b/architecture/adr/013-override-exception-governance.md @@ -0,0 +1,28 @@ +# ADR-013: How to Handle Legitimate Exceptions Without Undermining Governance + +**Status:** Accepted +**Date:** April 2026 +**Docs:** Doc B §18 (Override Model) + +## Context + +Policies will block legitimate requests. A data residency policy may block a valid exception for a disaster recovery scenario. A sizing policy may block a temporary capacity burst for a product launch. If the only options are "change the policy" or "work around the system," governance degrades. + +## Decision + +Five override mechanisms, layered from least to most disruptive: + +1. **Override Policy** — A planned exception registered in advance (e.g., "DR events may use US-EAST zone") +2. **Exception Grant** — A pre-authorized waiver with compensating controls and expiry +3. **Manual Override** — Immediate single-request authorization with written justification +4. **Compensating Control** — Replace a blocked requirement with an equivalent risk-reduction measure +5. **Dual-Approval** — Required modifier for hard-enforcement policies (two approvers, different roles) + +**The consumer experience:** When a policy blocks a request, the consumer sees the blocking reason, compliant value suggestions, and four options: modify the request, request an override, cancel, or escalate. Override is one path among four — not the default. + +## Consequences + +- Every override is audited with full Merkle tree leaf +- Frequently-overridden policies are surfaced in metrics for policy review +- Block timeout auto-cancels requests where the consumer takes no action +- The governance model is flexible without being permissive diff --git a/architecture/adr/014-multi-tenancy-isolation.md b/architecture/adr/014-multi-tenancy-isolation.md new file mode 100644 index 0000000..a57642f --- /dev/null +++ b/architecture/adr/014-multi-tenancy-isolation.md @@ -0,0 +1,27 @@ +# ADR-014: How Tenants Are Separated + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Data Store Contracts (UDLM), Universal Groups (UDLM) + +## Context + +DCM serves multiple teams (tenants) within an organization. Each tenant's data, resources, policies, and audit trails must be isolated. A developer on Team A must not see Team B's resources, and Team A's policies must not affect Team B's requests (unless they're system-level policies that apply to everyone). + +## Decision + +**Row-Level Security (RLS)** in PostgreSQL enforces tenant isolation at the database layer. Every query is automatically scoped to the actor's tenant — application code cannot accidentally leak cross-tenant data. + +Tenants are **DCMGroups** with type `tenant_boundary`. Groups can be nested (organization → department → team) and support the universal group model for flexible organizational mapping. + +**Policy domain precedence** respects tenancy: system > platform > tenant > resource_type > entity. A system-level sizing policy applies to all tenants. A tenant-level naming convention applies only to that tenant. + +**Per-tenant key binding + crypto-shredding.** A Tenant's data-at-rest key material is bound to it by reference — `dcm-group.key_bindings` → a `Credential.EncryptionKey` (envelope DEK/KEK) issued by a credential-capability provider. That makes the key **addressable**, so the realization can keep it in-boundary (sovereign profile), rotate it, and **crypto-shred** it on decommission: destroying the tenant's KEK renders its data-at-rest unrecoverable (GDPR Art. 17 erasure) while the immutable audit ledger survives (UDLM GRP-013). UDLM carries the binding (data); the realization owns the key lifecycle and the KMS/HSM integration. This is the architecture primitive, not the implementation. + +## Consequences + +- Tenant isolation is enforced by the database, not application logic — defense in depth +- Cross-tenant operations (ownership transfer, shared resources) require explicit policy authorization +- RLS adds a small query overhead (~2-5%) — acceptable for the security guarantee +- 18 SQL tables all include tenant_uuid columns with RLS policies +- Tenant offboarding erases data-at-rest by **crypto-shredding the per-tenant key** (`key_bindings`), not by mutating the audit ledger — right-to-erasure without breaking tamper-evidence diff --git a/architecture/adr/015-minimal-infrastructure.md b/architecture/adr/015-minimal-infrastructure.md new file mode 100644 index 0000000..02d01f5 --- /dev/null +++ b/architecture/adr/015-minimal-infrastructure.md @@ -0,0 +1,30 @@ +# ADR-015: Why PostgreSQL Is the Only Required Dependency + +**Status:** Accepted +**Date:** March 2026 +**Docs:** Data Store Contracts (UDLM), Deployment (UDLM) + +## Context + +Infrastructure management platforms often require heavy middleware stacks: message brokers, secret managers, identity providers, search engines. This creates a bootstrap problem — you need significant infrastructure just to manage infrastructure. It also blocks adoption in resource-constrained environments (homelab, edge, evaluation). + +## Decision + +PostgreSQL is the only required dependency. DCM implements internal equivalents for every capability that optional services provide: + +| Capability | Internal (default) | External (optional) | +|-----------|-------------------|-------------------| +| Events | PostgreSQL LISTEN/NOTIFY | Kafka | +| Secrets | Envelope-encrypted table | Vault | +| Auth | Built-in bcrypt + JWT | Keycloak/OIDC | +| Search | PostgreSQL full-text + GIN | OpenSearch | +| Notifications | PostgreSQL LISTEN/NOTIFY + webhooks | External notification service | + +Every optional dependency follows the same pattern: internal by default, externally delegable by configuration. The same API surface is exposed regardless of which implementation is active. + +## Consequences + +- Bootstrap is `docker-compose up` with one PostgreSQL container +- Production deployments can delegate to Kafka, Vault, Keycloak when scale or policy requires it +- Internal implementations have performance ceilings (LISTEN/NOTIFY: ~1K events/sec vs Kafka: millions) +- Every new cross-cutting service must implement the internal path first diff --git a/architecture/adr/016-application-definition-language.md b/architecture/adr/016-application-definition-language.md new file mode 100644 index 0000000..61eb7de --- /dev/null +++ b/architecture/adr/016-application-definition-language.md @@ -0,0 +1,114 @@ +# ADR-016: Application Definition Language + +**Status:** OPEN — Design Decision Required +**Date:** April 2026 +**Raised by:** Ondra (machacekondra), repeatedly + +## Context + +DCM currently has two consumer interfaces: + +1. **Single resource** — A JSON payload to the Consumer API: `POST /api/v1/requests { "catalog_item_uuid": "...", "fields": {...} }` +2. **Compound service** — A composite resource type spec that defines constituent resources, dependencies, and binding fields in YAML + +The single-resource API works well for atomic resources. The composite service definition (composite service model) works for platform engineers who define reusable application templates. But there is a gap: + +**How does a consumer define a custom application?** Not a pre-defined catalog item, but an ad-hoc composition: "I need a database, two app servers, and a load balancer, and here's how they connect." Today, this requires a platform engineer to create a composite resource type spec first. + +Comparable projects have made different choices: +- **Radius** uses Bicep (a DSL) for application definitions, with Recipes (Terraform/Bicep templates) for infrastructure implementation +- **KRO** uses ResourceGraphDefinitions with CEL expressions, generating CRDs from the definition +- **Crossplane** uses Compositions with embedded resource templates and patch sets + +## The Question + +What is DCM's application definition language? Options to evaluate: + +### Option A: API-Only (Current State) +Consumers submit JSON payloads. Compound services require pre-defined composite resource type specs. Platform engineers author specs; consumers consume them. + +**Pros:** Simple, API-first, no custom language to learn +**Cons:** No self-service composition. Every new application pattern requires a platform engineer. + +### Option B: YAML Application Manifests +A YAML document defining resources, dependencies, and binding fields — similar to the composite resource type spec but authored by consumers, not platform engineers. + +```yaml +apiVersion: dcm.io/v1 +kind: Application +metadata: + name: pet-clinic +spec: + resources: + - name: database + type: Database.PostgreSQL + fields: { engine: postgresql, storage_gb: 50 } + - name: backend + type: Compute.VirtualMachine + depends_on: [database] + bindings: + - from: database.ip_address + to: config.db_host + fields: { cpu_count: 4, memory_gb: 8 } + - name: frontend + type: Compute.VirtualMachine + depends_on: [backend] + bindings: + - from: backend.ip_address + to: config.api_host + fields: { cpu_count: 2, memory_gb: 4, replicas: 2 } +``` + +**Pros:** Declarative, GitOps-friendly, reviewable, versionable +**Cons:** New format to learn. Validation complexity. How does this interact with the service catalog? + +### Option C: Reference Existing DSL (Bicep, CEL, HCL) +Adopt an existing language like Radius does with Bicep or KRO does with CEL. Leverage existing tooling and developer familiarity. + +**Pros:** Existing tooling, IDE support, community +**Cons:** Tight coupling to an external project. Bicep is Azure-originated. CEL is K8s-specific. HCL is HashiCorp-specific. + +### Option D: Catalog Composition via API +Consumers compose applications by linking multiple catalog requests through the API, declaring dependencies between them. No new language — just structured API calls. + +```json +POST /api/v1/applications +{ + "name": "pet-clinic", + "components": [ + { "name": "database", "catalog_item_uuid": "pg-standard", "fields": {...} }, + { "name": "backend", "catalog_item_uuid": "vm-standard", "fields": {...}, + "depends_on": ["database"], + "bindings": [{ "from": "database.ip_address", "to": "config.db_host" }] } + ] +} +``` + +**Pros:** API-first, no DSL, consistent with existing patterns +**Cons:** JSON is verbose for complex compositions. Not as readable/reviewable as YAML. No GitOps-friendly file format. + +## Evaluation Criteria + +1. **Consumer UX** — How easy is it for a developer to define a three-tier app? +2. **Platform engineer UX** — How easy is it to create reusable templates? +3. **GitOps compatibility** — Can definitions be stored in Git and applied via PR? +4. **Validation** — Can DCM validate the definition before execution? +5. **Existing tooling** — Does it work with existing editors, linters, CI pipelines? +6. **Consistency with DCM patterns** — Does it align with the API-first, JSON, snake_case conventions? + +## Recommendation + +This decision needs team input. The author's preliminary assessment: + +**Option B (YAML manifests) or Option D (API composition) are most aligned** with DCM's existing patterns. Option B is better for GitOps. Option D is better for API-first consistency. They could coexist — the YAML manifest could be a file format that the API endpoint accepts. + +**Option C (external DSL) is least aligned** — it introduces a dependency on an external project's language and tooling, which conflicts with DCM's technology-agnostic principle. + +**Regardless of choice, the composite resource type spec remains the implementation mechanism.** The application definition language is a consumer-facing UX that ultimately produces a composite resource type spec (or equivalent) for execution. + +## Actions Required + +- [ ] Team discussion to evaluate options +- [ ] Prototype consumer UX for three-tier app with top 2 options +- [ ] Evaluate interaction with RHDH (Backstage) scaffolding templates +- [ ] Decision by [date TBD] diff --git a/architecture/adr/017-brownfield-greening-discovered-ingestion.md b/architecture/adr/017-brownfield-greening-discovered-ingestion.md new file mode 100644 index 0000000..2606f3e --- /dev/null +++ b/architecture/adr/017-brownfield-greening-discovered-ingestion.md @@ -0,0 +1,90 @@ +# ADR-017: Brownfield Greening — Ingesting Existing Resources via the Discovered Store + +**Status:** Accepted +**Date:** June 2026 +**Docs:** ADR-003 (Four Lifecycle States), ADR-005 (Provider Abstraction), ADR-007 (Placement Engine); UDLM `foundations/four-states.md`, `foundations/ownership-sharing-allocation.md` +**Tracking:** #221 (this design) → #222–#227 (build steps) + +## Context + +DCM's normal flow runs **forward**: Intent → Requested → Realized → Discovered (ADR-003). But real estates are **brownfield** — resources already exist before DCM ever models them (the homelab is the motivating case). DCM needs a defined process to bring existing resources **into** the managed model, working **backward** from observation. We call this **greening the brownfield**. + +The anchoring distinction is *who controls the record*: + +- **Discovered** = observed to exist, **no provider attached** → **unclaimed**. +- **Realized** = a provider has asserted "I control this" → **claimed**, attributed, managed. + +Ingestion is the process of moving a resource across that line. This is not a novel invention — it is the well-established brownfield **import/adoption** pattern (Terraform `import`, Crossplane observe/import, Cluster API / ACK / Config-Connector "acquisition") mapped onto UDLM's four states. + +## Decision + +Define a **discovered-resource ingestion pipeline**: + +1. **Discovery** populates the **Discovered store** by one of two avenues: + - **Provider-generated** (ADR-005 Level-2 discovery): a registered provider enumerates the resources it controls and emits Discovered records **with provider attribution** — effectively pre-claimed. + - **Third-party-generated**: a non-provider observer (probes, scanners, CMDB, import tools — e.g. the homelab's `virsh`/`oc`/`ceph`/ansible probes) emits Discovered records with **no provider attribution** — **unclaimed**. +2. The **Discovered store** holds both, durably and queryably (see Decision A). +3. **Reverse placement** (the inverse of the ADR-007 placement engine): given an unclaimed discovered resource, identify the provider(s) that own or can claim it — by `resourceType` + attributes + provider capability / adopted-standard matrices. +4. **Provider claim / adoption**: the identified provider asserts control → the record moves **Discovered → Realized**, preserving the entity UUID (UDLM SPEC-DESIGN-REQUIREMENTS §28 (raw/unallocated lifecycle entry) adopt-then-append). +5. **Intent backport / synthesis** (optional): derive Intent from the Realized/Discovered state so the resource becomes rebuildable (see Decision B). + +``` +existing resource + ├─ provider-generated discovery ─┐ (attributed) + └─ third-party discovery ────────┤ (unclaimed) + ▼ + DISCOVERED store + │ (unclaimed only) + reverse placement → owning provider + │ + provider claim/adoption + ▼ + REALIZED ──(optional backport)──► INTENT +``` + +### Decision A — the Discovered store has a dual role (clarified, not a new store) + +UDLM today frames Discovered as *ephemeral* (per-cycle snapshots for drift detection). We **clarify** it as having two roles: (1) the ephemeral snapshot stream (drift), **and** (2) a **durable, per-UUID entity inventory** that is the **source of truth for what exists — including discovered-but-unclaimed resources**. We do **not** introduce a separate "inventory" store. This is consistent with UDLM §28 raw/unallocated resources (which already live durably in Discovered with `lifecycle_state: available`). *Landed 2026-07: `foundations/four-states.md` §2.4 now carries the Discovered dual-role clarification (udlm greening PR); #222 closed. `correlation_ids` is a real field on `realized-entity.schema.json`.* + +### Decision B — Intent may be generated from Discovered, independent of Realized + +Backporting Intent is **permitted and valued**: from Realized (the normal case) **and** directly from Discovered to produce a **provider-agnostic, rehydratable desired state** for DR/portability (rebuild the estate onto different hardware/providers). This is a legitimate state — Intent-declared-but-not-yet-built already exists in the forward flow. Such inferred Intent is marked **`provenance: discovered-derived`** (not human-declared) so consumers trust it appropriately. + +### Decision C — discovered records carry correlation identifiers; ingestion resolves to one entity + +The two avenues can observe the **same** real-world resource (a 3rd-party probe sees it, and later a provider enumerates it). To avoid double-counting, every discovered record carries **correlation identifiers** — stable natural keys independent of the observer: + +- hosts: SMBIOS/system UUID, chassis/system serial, BMC id; +- VMs: hypervisor domain/instance UUID; +- NICs: MAC; disks: WWN / serial; storage clusters: cluster FSID; +- and, when provider-generated, the **provider's own resource ID**. + +Ingestion runs **entity resolution**: a new observation is matched against existing entities by these keys (strongest/globally-unique keys first); a match **merges into the existing entity UUID** (the UDLM universal linking key, four-states §3) instead of minting a new one. So when a provider later enumerates a resource a 3rd-party probe already discovered, it **correlates to the same entity and claims it** (Discovered-unclaimed → Realized) rather than creating a duplicate. This is what minimizes 3rd-party-discovered items that are subsequently also injected by a DCM provider. + +This builds on the the component `Identity` block (UDLM common-elements §27 identity) (serial/wwn/mac/location) but applies at the **top-level resource**, so the discovered/realized record carries a dedicated `correlation_ids` field — *shipped on `realized-entity.schema.json` (udlm greening PR).* + +## Options considered + +- **In-between store for "described but unclaimed".** (a) Reuse **Discovered** ✅ — it is already defined as observed ground truth that may contain unprovisioned/brownfield resources. (b) A new dedicated inventory store — rejected: duplicates Discovered's purpose and the four-state model. +- **Discovered durability.** (a) Clarify dual-role ✅. (b) Keep ephemeral-only and force unclaimed resources into Realized — rejected: Realized requires provider attribution, which unclaimed resources lack. +- **Intent from Discovered.** (a) Allow, provider-agnostic, flagged inferred ✅ — unlocks DR/portability. (b) Forbid; require Realized first — rejected: blocks rehydration of brownfield that no provider has claimed yet. + +## Consequences + +- The **homelab is the first application** and is entirely **Avenue 2** (no providers yet): its estate store (#219) is a Discovered store of **unclaimed** resources. +- As providers arrive (Kea #208, Ceph #218, a libvirt provider), they **reverse-place and claim** those resources into Realized; Intent backport then makes the estate rebuildable for DR — DCM-at-home dogfooding DCM. +- **Reverse placement** is a new engine alongside the forward placement engine (ADR-007). +- The UDLM `four-states.md` clarification for Decision A (#222) has LANDED. +- The dependency graph stays **emergent** from per-resource Discovered records (#219), never a hand-authored file. + +## Best practices + +These govern every discovered-ingestion flow: + +1. **Minimize unclaimed resources — they are an anti-pattern.** A resource stuck in `Discovered`/unclaimed has **no owning provider**, so it gets no lifecycle management, no authoritative drift reconciliation, and cannot be rehydrated for DR. Lingering unclaimed resources are a tracked **gap**: surface the unclaimed count, alert on it, and drive it toward zero — every discovered resource should converge to **claimed** (a provider owns it) or be explicitly **retired/excluded**. Recorded as a UDLM **Antipattern** (`entities/knowledge-family.md` §4.4): *"long-lived unclaimed discovered resource → claim it (reverse-place + adopt) or retire it."* +2. **Prefer provider-generated discovery (Avenue 1) over third-party (Avenue 2).** Provider discovery arrives already attributed (effectively claimed), skipping the unclaimed limbo and most correlation work. Third-party is the bootstrap/interim — migrate sources onto real providers as they come online (how the homelab graduates from probes to the Kea/Ceph/libvirt providers). +3. **Every discovery source MUST emit correlation identifiers** (Decision C). No identifiers → no reliable entity resolution → duplicates. +4. **One real resource, one entity UUID.** On a correlation match, merge into the existing entity; never mint a second. +5. **Reconcile, don't overwrite.** A discovered value that contradicts Realized is **drift** — surfaced (OBS-001), never silently merged. +6. **Backport Intent for anything that must survive DR.** Claimed-but-no-Intent can't be rebuilt; synthesize provider-agnostic Intent (`provenance: discovered-derived`). +7. **Unclaimed = inventoried, not managed.** Unclaimed resources are queryable for inventory but excluded from lifecycle operations until claimed — they describe reality, they don't yet control it. diff --git a/architecture/adr/018-wire-serialization-event-conventions.md b/architecture/adr/018-wire-serialization-event-conventions.md new file mode 100644 index 0000000..42a153b --- /dev/null +++ b/architecture/adr/018-wire-serialization-event-conventions.md @@ -0,0 +1,44 @@ +# ADR-018: Wire Serialization & Event Conventions + +**Status:** Accepted +**Date:** June 2026 +**Docs:** ADR-003 (Four Lifecycle States), ADR-009 (API Gateway & Control Plane); UDLM `registry/naming-conventions.md` §4; AEP (`aep.dev`); CNCF CloudEvents +**Tracking:** companion to the UDLM data-model casing decision (naming-conventions §4). Supersedes the camelCase first draft of this ADR. + +## Context + +UDLM defines the **data model** and fixes its on-the-wire casing — **`snake_case` keys** (UDLM `naming-conventions.md` §4). DCM is the **runtime** that serializes and transports that model — over REST/gRPC at the API gateway (ADR-009) and over the event bus — between services written in **Go and Python**. This ADR covers the DCM-side conventions so the snake_case contract flows end to end without per-hop key-translation layers. (The casing of the data model itself is UDLM's call; this ADR is how the runtime honors it.) + +**Why snake_case (the reversal):** UDLM is a **canonical data model consumed natively** — the model *is* the wire form, so there is no separate "API casing" to translate to. DCM's API is the consumer with a hard external constraint: it conforms to **AEP** (`aep.dev`, the dcm-project engineering team's adopted API-design standard, enforced by `aep-dev/aep-openapi-linter`), whose prescribed fields are snake_case (`page_size`, `*_time`). Native-universal consumption **+** AEP-bound API jointly force one casing — snake_case. The first draft of this ADR chose camelCase on research that had not accounted for AEP; this revision reverses it. (Empirically: the AEP linter reports zero casing findings against the existing snake_case OpenAPI specs.) + +## Decision + +1. **Wire payloads are `snake_case`** — request/response bodies and event payloads match the UDLM data model and AEP. No snake_case↔camelCase translation layer between the API, the event bus, and services. + +2. **Go services:** PascalCase exported struct fields with explicit tags — `json:"snake_case" yaml:"snake_case"`. Go keeps its idiom; the wire stays snake_case. + ```go + type ResourceDefinition struct { + ResourceID string `json:"resource_id" yaml:"resource_id"` + MemoryLimit string `json:"memory_limit" yaml:"memory_limit"` + } + ``` + +3. **Python services:** native `snake_case` attributes via **Pydantic** — attribute names map **directly** to wire keys, so no alias generator is needed (the camelCase draft required `alias_generator=to_camel`; that is now removed). Python keeps PEP 8 *and* the wire matches it. + +4. **Events use the CloudEvents envelope** (CNCF). Event **payload** property keys are `snake_case`. Event **type identifiers / topics** are lowercase **dot notation** — `resource.discovered`, `entity.realized`, `resource.definition.created` — so brokers (Kafka/NATS/etc.) can **wildcard-route** (`resource.definition.*`). Topics are an event-naming concern, orthogonal to payload casing. (CloudEvents *context attributes* are themselves flatcase per the CloudEvents spec — neither snake nor camel — and are unaffected.) + +5. **No ad-hoc translation.** Frontends, third-party webhooks, and microservices consume the same snake_case shape; serialization config is centralized (Go tags), not reinvented per service. The one place casing changes is an **export adapter** at a foreign-domain boundary (e.g. projecting a resource into a Kubernetes CRD, which is camelCase by convention) — never inside the DCM/UDLM core. + +## Options considered + +- **camelCase on the wire** — rejected: conflicts with AEP (the adopted, lint-enforced API standard) and with native-universal UDLM consumption; would re-introduce the translation layer native consumption exists to remove. (This was the first draft; reversed.) +- **PascalCase keys** — rejected: legacy CloudFormation idiom only. +- **camelCase topics** — moot: event names use dot-notation regardless of payload casing, for broker wildcard routing. + +## Consequences + +- API specs are AEP-conformant on casing; the `aep-openapi-linter` casing checks pass. +- Python services need **no** Pydantic alias generator — attribute = wire key. Go adds snake_case struct tags (mechanical, centralized). +- Frontends and third-party webhook consumers ingest payloads directly — no key-translation layer; one convention from API request → event bus → service. +- Events are CloudEvents-compliant; dot-notation topics enable wildcard subscriptions. +- This ADR is the DCM realization of UDLM `naming-conventions.md` §4 — the data-model casing is owned by UDLM; the transport/serialization conventions are owned here. Both are now snake_case, so the contract is identity end to end. diff --git a/architecture/adr/019-placement-policy.md b/architecture/adr/019-placement-policy.md new file mode 100644 index 0000000..acf4495 --- /dev/null +++ b/architecture/adr/019-placement-policy.md @@ -0,0 +1,36 @@ +# ADR-019: Placement Policy + +**Status:** Proposed +**Date:** June 2026 +**Docs:** ADR-002 (Three Abstractions), ADR-007 (Placement Engine), ADR-011 (Sovereignty); UDLM **ADR-001 (`Topology`)**, **ADR-002 (Capacity/Utilization)**, **ADR-004 (Provider capability declaration)** +**Tracking:** raised by dcm-project/dcm #64 (pkliczewski: "what about placement policy?") + +## Context + +The Placement Engine (ADR-007) tie-breaks on `affinity`/`cost`/`load`, but nothing **authors** placement constraints — there is no policy that lets a consumer/org express "spread across failure domains," "co-locate app+db," "keep in EU," or "prefer provider X." The data to resolve against now exists in UDLM (`Topology`, capacity/utilization, provider capability); this ADR adds the **policy** that consumes it. + +## Decision + +Introduce **Placement Policy** — the **8th typed Policy** (alongside Gating Policy, Validation, Transformation, Recovery, Orchestration-Flow, Governance-Matrix, Lifecycle). + +- **Authors declarative placement constraints**: affinity / anti-affinity / co-locate / spread / prefer / avoid / pin, plus weights — keyed on abstract `Topology` **`kind`s** (`zone`/`host`/`power-domain`/…), never provider-native ids. +- **Declarative only** (no embedded expressions); the **Placement Engine evaluates** it. Output feeds the engine's scoring/tie-break stage. +- **Enforces the portability discipline** (UDLM ADR-001 §17): rejects intent that names provider-native topology ids — those belong to realized state. +- **Consumes**: UDLM `Topology` (the map), capacity/utilization (the load/headroom), and provider `topology_capability` (ADR-004) to filter/score candidates. + +## Data · Policy · Provider (required lens — see ADR README) + +- **Data (UDLM):** `Topology` (kinds + concrete domains), capacity/utilization overlay, each resource's locality reference — the facts placement reads. *(UDLM ADR-001/002.)* +- **Policy (DCM, this ADR):** the Placement Policy constraint grammar + the engine evaluation/scoring + portability enforcement. *(The decision/compute.)* +- **Provider:** declares `topology_capability` + `mobility` (ADR-004) and naturalizes abstract kinds to its native topology; populates the concrete `Topology`. *(What's possible + execution.)* + +## Options considered +- **Provider-native placement constraints** — rejected: destroys portability. +- **Affinity via relationships only** — rejected: doesn't carry weights/spread/prefer or org-authored intent. +- **Placement Policy (8th typed policy) over abstract `Topology`** — **chosen**; fits the typed-policy model and the Data⇄Policy boundary. + +## Consequences +- New typed policy; the taxonomy's typed-Policy set grows from 7 → 8. +- Cross-domain: governs placement of **any** resource type (the `Topology`/capability data it reads is cross-cutting). +- Sovereignty/residency becomes a placement constraint over jurisdiction-labeled domains (unifies with ADR-011). +- Companion: ADR-020 (migration & operational gating). diff --git a/architecture/adr/020-migration-and-operational-gating.md b/architecture/adr/020-migration-and-operational-gating.md new file mode 100644 index 0000000..ed553a9 --- /dev/null +++ b/architecture/adr/020-migration-and-operational-gating.md @@ -0,0 +1,38 @@ +# ADR-020: Migration & Operational Gating + +**Status:** Proposed +**Date:** June 2026 +**Docs:** ADR-002 (Three Abstractions), ADR-006 (Policy Engine), ADR-011 (Sovereignty), ADR-013 (Override & Exception Governance), ADR-019 (Placement Policy); UDLM **ADR-003 (Data mobility + process validation, T6)**, **ADR-004 (Provider capability declaration)** +**Tracking:** placement-data family — the control-plane side of UDLM data mobility + process validation + +## Context + +UDLM ADR-003 makes data mobility declarable (`data_mobility` requirements, `process_validation` lifecycle) and ADR-004 makes provider mobility/operational capability declarable. The control plane must **govern** when/whether a migration may run, **gate** on validation freshness, and **schedule** rehearsals — without inventing new policy machinery. + +## Decision + +Migration and operational gating **reuse the existing typed policies** (no new types beyond ADR-019); they are configurations of them: + +- **Migration *permission*** → **Governance-Matrix** policy: every cross-boundary move is evaluated (sovereignty/jurisdiction — ADR-011); cross-jurisdiction migrations `DENY` / `ALLOW_WITH_CONDITIONS` / dual-approval (ADR-013). +- **Migration *sequence*** → **Orchestration-Flow** policy: the ordered cutover steps (provision target → replicate → verify → switch → drain) — provider executes each. +- **Capability match** → **Placement Policy** (ADR-019) extends to require a provider whose `mobility` (ADR-004) satisfies the resource's `data_mobility` (RTO/RPO/online). +- **Process-validation gating (T6)** → **Gating Policy** policy: `gate_on_stale` ⇒ deny placing/depending-on a critical workload whose `mobility_validation` is `stale`/`failing`. Compliance-class (fail-safe). +- **Rehearsal scheduling** → an operational policy fires `simulated`/`rehearsal` runs on the `process_validation.cadence`; the provider runs them (`operational_capability.rehearsal_support`), evidence is recorded, freshness refreshed. + +**A real incident executes the same validated path (T6)** — it validates the outcome; the gates above just ensure the path was proven first. + +## Data · Policy · Provider (required lens) + +- **Data (UDLM):** `data_mobility`, `process_validation`, `mobility_validation` evidence (ADR-003) — requirements + observed proof. +- **Policy (DCM, this ADR):** permission (Governance-Matrix), sequence (Orchestration-Flow), gating (Gating Policy), scheduling (operational) — when/whether/how-gated. +- **Provider:** declares `mobility` + `operational_capability` (ADR-004); **executes** the migration mechanism and the rehearsals (naturalization) — unmodeled "how." + +## Options considered +- **A bespoke "migration policy" type** — rejected: migration permission/sequence/gating are already expressible as Governance-Matrix / Orchestration-Flow / Gating Policy configurations; minimal-core. +- **Reuse existing typed policies + Placement Policy capability match** — **chosen.** + +## Consequences +- No new policy *type* (beyond ADR-019); migration/operational gating are policy *configurations*. +- Cross-domain: applies to any resource carrying `data_mobility` (any stateful domain). +- Makes ADR-019's "change topology → rebuild" governed + proven: the rebuild is permission-checked, freshness-gated, and sequence-orchestrated. +- DAV connection: the `process_validation` findings/evidence are the same validation-backed records an assessment realization surfaces — one evidence model. diff --git a/architecture/adr/021-adopting-external-standards.md b/architecture/adr/021-adopting-external-standards.md new file mode 100644 index 0000000..b077d0b --- /dev/null +++ b/architecture/adr/021-adopting-external-standards.md @@ -0,0 +1,70 @@ +# ADR-021: Adopting External Standards by Reference + +**Status:** Accepted +**Date:** June 2026 +**Docs:** `architecture/adopted-standards-dcm.md` (ADS-001…010), `architecture/data-policy-boundary.md`; UDLM `design-principles/core-tenets.md` (T5), `design-principles/adopted-standards.md` + +## Context + +DCM and UDLM repeatedly meet domains that a mature, vendor-neutral external standard **already models**: +cost/usage (**FOCUS** — FinOps Open Cost & Usage Specification), Kubernetes cost allocation +(**OpenCost**), compliance (**OSCAL**), identity (**SCIM**). The question recurs: do we model that data +*inside* DCM/UDLM, or reference the external standard? + +Cost forced the decision. A first attempt modeled a cost type with a rate schema and outputs inside the +data substrate; it was retired. Modeling adjacent-domain data inside DCM/UDLM (a) duplicates an external +system's record and its lifecycle, (b) couples the substrate to one vendor's vocabulary — a cost type +expressed in Koku's metric names and `Infrastructure/Supplementary` taxonomy is serviceable by **one** +provider — and (c) means re-expressing (badly) a standard that already exists and that providers already +emit. That breaks the core requirement that contracts be **vendor-agnostic**. + +## Decision + +**Adopt external standards by reference; do not absorb them.** Decided by a test: + +| Disposition | Meaning | When | +|---|---|---| +| **Absorb** | define the schema inside DCM/UDLM | only when **no** credible external standard exists **and** the data's lifecycle is genuinely the substrate's to custody | +| **Embed** | bake the fields into every entity | never | +| **Adopt** | reference the standard by conformance + binding | whenever a credible external standard already models the data | + +Under Adopt, the **data** (UDLM) carries only: resource **identity** (the join key), a **version-pinned +conformance reference**, and the **binding**; never the standard's schema. **DCM** owns the verbs: +providers declare an `adopted_standard_support` matrix (which standard versions they emit/consume); DCM +**negotiates**, **translates**, and **enforces** versions, and records the **effective version** as +provenance (`adopted-standards-dcm.md`, ADS-001…010). Implementor-bounded parameters (e.g. cost rate +ranges, budgets) are enforced via policy-as-code, not hard-coded. + +**Cost is the first case:** the cost data conforms to **FOCUS** (+ **OpenCost** for k8s allocation), +served via the Cost SP's Information-Provider query API — a `serve_data` + `realize_resources` provider +(ADR-005). The Koku backing is an *implementation detail of one provider*, never the contract. + +## Alternatives Considered + +1. **Absorb — define a cost type / schema in the registry** — rejected: duplicates FOCUS, breaks T1 + (the data model is a custodian, not the owner of every domain's data), and vendor-locks the contract + to Koku's vocabulary. +2. **Embed cost fields on every entity** — rejected: a stronger T1 violation; cost is not intrinsic to + every resource's lifecycle. +3. **Adopt FOCUS/OpenCost by reference, negotiate versions in DCM** — chosen: vendor-agnostic by + construction, no duplication, tracks the standard's evolution without schema churn. + +## Consequences + +- Cost — and every future adopted domain — is **vendor-agnostic by construction**: any FOCUS-emitting + provider satisfies the same binding; the backing implementation (Koku, Kubecost, a cloud export) is + swappable. +- UDLM stays **thin** — identity, lifecycle, relationships, bindings — not a model of every adjacent + domain. The architectural form of "don't reinvent the wheel." +- DCM gains a **version-negotiation/translation** responsibility (ADS-003/004) and a provider + **support-matrix** at registration (ADS-001) — so providers detail which standard versions they + support and implementors get the version they need. +- The external standard can **evolve without DCM schema changes** (FOCUS 1.x → next): DCM bumps a + pointer, it does not migrate a copied schema. +- The litmus going forward: **if a new external-standard version would force a change to a DCM/UDLM + schema, we absorbed it by mistake — it should have been adopted.** +- Adoption has **two tiers**, and the integration machinery is routed by kind: **value/codelist** + standards (ISO 4217, ISO 8601, RFC 4122) are adopted as a referenced field constraint — *no* support + matrix or version negotiation; **record/schema** standards (FOCUS, OpenCost, OSCAL, SCIM) get the full + ADS-001…010 apparatus. The full routing table is in UDLM `design-principles/adopted-standards.md` §1a. + This keeps ADS from being mis-applied to a codelist (e.g. Koku's ISO 4217 change vs its FOCUS export). diff --git a/architecture/adr/022-trust-model.md b/architecture/adr/022-trust-model.md new file mode 100644 index 0000000..1fe4d41 --- /dev/null +++ b/architecture/adr/022-trust-model.md @@ -0,0 +1,120 @@ +# ADR-022: DCM Trust Model (incl. Credential API Selection) + +**Status:** Proposed +**Date:** 2026-06-28 +**Type:** Architecture Decision Record (a `DecisionRecord` with architecture scope) +**Related:** ADR-005 (Provider Abstraction), ADR-007 (Placement Engine), ADR-010 (Audit & Tamper Evidence), ADR-011 (Sovereignty), ADR-021 (Adopting External Standards by Reference); UDLM ADR-004 (Provider capability declaration); CPX-001…012; AAL (NIST 800-63B); NIST SP 800-207 (Zero Trust). **Flows:** [`architecture/trust-flows.md`](../trust-flows.md). **Profiles:** [`architecture/trust-profiles.md`](../trust-profiles.md). **Attestation of this model:** [`architecture/trust-attestation.md`](../trust-attestation.md). +**Note:** consolidates the credential-API-selection and trust-model drafts into one ADR (minimize surface / avoid drift). + +## Context + +Trust in DCM cannot be self-declared — not for the platform and not for a provider. DCM spans organizational and sovereign boundaries, so it must be a **first-class participant in the standard trust fabric** (PKI/TLS, OAuth2/OIDC, attestation), not a bespoke trust island. And **a declared capability is only a *claim*** — "I support FIPS L3" is not trustworthy because the provider says so; trust must be backed by **external attestation** of a strength the target market demands. + +## DCM's role: trust **broker** (introducer/matchmaker), not credential authority + +DCM/UDLM **broker** trust; they do **not** custody, pass, or negotiate brokered credentials. + +- **DCM does** — discover + **match** producer↔consumer; **gate** on attestation; **broker the bootstrap** (endpoints + trust anchors + a short-lived, scoped **Introduction Grant**) so the parties stand up a **direct, mutually-authenticated** channel; **audit**. +- **DCM does NOT** — hold/pass credential **values** (CPX-001) or sit in the credential **negotiation** path. The consumer speaks the *selected* standard API (ACME/EST/OAuth/KMIP) **directly** to the producer. +- **"Trusted" = introduction integrity, not secret custody** — a minimal surface: a compromised broker can't leak secrets (it never holds them); at worst it mis-introduces, which mutual-auth + audit catch. + +**DCM's second hat — participant for its OWN needs.** DCM also *consumes* credentials (component mTLS, user auth, internal trust) and *produces* its own component identity (Internal CA), through the **same** declare→select→attest machinery (no bypass). Two custody categories: + +| Category | Owner | Custody | +|---|---|---| +| **Managed** (brokered consumer↔producer) | the parties | **CPX-001 — value never in DCM** | +| **DCM-operational** (own TLS/JWKS/component keys) | DCM | DCM **must** hold these — protected per profile (sw → HSM), attested, rotated | + +## Decision — one trust model across five planes (adopt-by-reference, ADR-021) + +Each plane is **upheld** (enforced on every interaction), **participated-in** (DCM is a member of the standard system), and **exposed** (DCM publishes its own verifiable posture). DCM's posture across all five is *broker*. + +| Plane | Standards (referenced) | Uphold / Participate / Expose | +|---|---|---| +| **Identity & transport** | X.509/PKIX (RFC 5280), mTLS, Trust Anchors, PCA, SPIFFE (opt) | validate chain→anchor + revocation / present own cert + accept external CAs / publish anchors + certs | +| **Authorization** | OAuth2 (RFC 6749), OIDC, introspection (RFC 7662), JWKS, revocation | verify every token / integrate any OIDC IdP / own introspection+JWKS | +| **Credential issuance** | ACME/EST/SCEP/CMP, OAuth, KMIP — **selected** (below) | gate on capability+attestation / naturalize any compliant backend / publish the capability+attestation matrix | +| **Attestation** | CMVP(FIPS 140-3), CC, FedRAMP, eIDAS, PCI-DSS, SOC 2, SecNumCloud, C5, IRAP; RATS (RFC 9334) | gate on tier+framework / consume recognized authorities / publish own + providers' attestations | +| **Federation** | peer posture exchange; CONFORMANCE verification | verify peer before federating / exchange anchors / expose posture | + +Invariants: **CPX-001**; **everything declared data** (anchors, identities, attestations) → queryable + governable; **market-appropriate strength** set by `profile × region`; every trust decision **audited** (ADR-010). + +## Credential API Selection (the credential-issuance plane in detail) + +Credentials are brokered like **placement** (ADR-007): providers **declare** a granular credential capability matrix (per type × operation: API specs, algorithms, `key_usage`, AAL/FIPS/HSM, lifetime, rotation, revocation, retrieval transport+auth, residency, **attestation**); requests state **requirements**; DCM **selects** — `sovereignty pre-filter → accreditation (attestation) filter → capability filter → score → select → negotiate` — then mints the Introduction Grant and steps out. A provider exposing only a native API **fails the filter** for any standardized requirement. + +- **Attestation ladder** (gate on the attestation, not the claim): `self_asserted → vendor_attested → independently_verified → accredited → hardware_attested`; the profile sets the minimum tier + accepted frameworks for the market. Carried on the existing **Accreditation** artifact. +- **Priority — security/trust/fit-for-purpose first, portability retained but subordinate.** A *deliberate inversion* of the general UDLM stance (where portability is near-sacrosanct): for credentials, security+trust+fit are the **hard gate**; portability is a **scoring tiebreak** (prefer standardized > `vendor-native`), never traded below the security floor. `vendor-native` is opt-in (`portability: provider-specific`), selected only when it's the fit-for-purpose secure choice the market needs *and* the request permits. +- **Queryable + governable:** the matrix is registry data ("who issues x509/ACME at FIPS L3 with EU custody?"), and `credential_profile`/Governance-Matrix gate on the *declared* fields before selection. +- **Per-type adopt-by-reference:** x509→ACME/EST/SCEP/CMP; token→OAuth2+RFC7662; key→KMIP/PKCS#11 — the standard *is* the API; the thin DCM envelope is only the selection/negotiation wrapper. + +## Anchoring & the internal TLS root + +**Can all trust anchor to one internal TLS certificate root? For the operational majority, yes — and at homelab, for *everything*. For attestation, deliberately no.** + +- **Identity / transport / issuance-brokering** (mTLS between components, PCA, the consumer↔producer channel, and the Introduction Grant signed by DCM and validated via DCM's JWKS) **all chain to a single internal TLS root.** This is the largest trust surface and the **default + bootstrap anchor** (ICOM-009 — components accept only registered anchors). At **homelab/minimal**, the internal root anchors *everything*, because the only attestation tier there is `self_asserted` (self-rooted by definition). +- **Attestation deliberately extends to *external* roots as assurance rises.** You cannot self-certify FIPS L3 / eIDAS / FedRAMP — those must be signed by a **recognized external authority** (CMVP roots, accreditation bodies, TEE/HSM manufacturer attestation roots for RATS). Anchoring attestation to your own internal root *is* `self_asserted` — meaningless for regulated markets. So `accredited` / `hardware_attested` (fsi/sovereign) add external anchors; they don't replace the internal root, they sit alongside it for the assurance claims. **This is not a limitation — it is the entire reason the attestation plane is separate from identity.** +- The internal root's **own** trust scales by profile: a software root (homelab) → an **HSM-protected / externally-attested** root (sovereign), with standard PKI hygiene (offline root, intermediates, rotation). A single root is also a single point of compromise — so higher profiles protect/attest the root itself. + +- **Disconnected / air-gapped sovereign:** where no external authority is reachable, external attestations are **imported** — their signed evidence pre-validated out-of-band and cached as **Accreditation** artifacts re-anchored into the local trust domain — rather than fetched live. The internal root can then anchor *everything operationally* in a self-contained domain, while the assurance still **derives from the original external authority's signature** (imported), not from self-assertion. This is how a sovereign disconnected cloud keeps high-assurance trust without live connectivity. + +**Summary:** internal TLS root = the universal anchor for *identity/transport/brokering* (and the homelab-everything anchor); external roots are *added* for *attestation* as the market bar rises. + +### Anchor methods (`anchor_type` — pluggable, extensible) + +The root of trust is **not** fixed to PKI. The trust anchor is declared data (ICOM-009), so **`anchor_type` is a pluggable dimension**: support a small core now, extend by configuration (adopt-by-reference, profile-gated) without redesign. Each profile lists *suggested* methods + a *settable minimum* (see `trust-profiles.md`). + +| Family | `anchor_type` | Anchor = trust in… | Tier | +|---|---|---|---| +| Hierarchical PKI | `internal-ca`, `public-acme` (LE/ISRG), `private-acme` (step-ca), `enterprise-pki` | a root cert chained to | **v1 core** | +| Federated identity | `spiffe-bundle`, OIDC/JWKS | node/platform attestation + bundle | core (OIDC) / deferred (SPIFFE) | +| Out-of-band | `tofu` (pin/PSK) | first-seen key / manual install | **v1 core** (bootstrap) | +| Authority / trust-list | `authority-list` (eIDAS Trusted Lists, CMVP, root stores) | a curated, signed authority list | deferred (= attestation plane) | +| Hardware / silicon | `hardware-rats` (TPM/HSM/TEE: SGX/SEV-SNP/TDX/CCA) | manufacturer root, appraised via RATS (RFC 9334) | deferred (fsi/sovereign) | +| Transparency-backed | `transparency-log` (Certificate Transparency, **Sigstore** Fulcio/Rekor) | append-only public logs + auditability | deferred (recommended standard+) | +| Decentralized | `did` / `ledger` (W3C DIDs+VCs), web-of-trust | distributed proofs / peer endorsement | declare-extensible only (not built) | +| Quorum | `threshold` (m-of-n / MPC / key ceremony) | N parties must agree | **root-protection** (sovereign root key), not a general anchor | + +**Core (v1):** `internal-ca` / `public-acme` / `private-acme` + OIDC + `tofu` bootstrap. **Deferred (declared, built when a market needs it):** `authority-list`, `hardware-rats`, `transparency-log`, `spiffe-bundle`. **Declare-extensible only:** `did`/`ledger`, web-of-trust. **`threshold`** is the answer to "a single root is a single point of compromise" — used to protect the sovereign root key (cf. DNSSEC root KSK ceremony), not as a day-to-day anchor. + +Caveat (homelab): a **`public-acme` (Let's Encrypt) leaf can root the public TLS edge but cannot issue mesh/client certs** (`CA:FALSE`) — pair it with `internal-ca` or `private-acme` (step-ca) for mTLS/workload identity. + +## v1 mandated core vs declared-but-deferred (minimization) + +The model is comprehensive; the **mandated v1 implementation is small**, and unbuilt mechanisms are *declared but deferred* — gated by the fail-safe rule: **a profile may require a framework only if the mechanism to verify it exists.** + +| v1 mandated DCM core (implement now) | Declared, deferred (build when a market needs it) | +|---|---| +| X.509/mTLS identity + Trust Anchors (internal root) | accredited-authority registry + accredited-tier verification | +| OIDC + JWKS + introspection (RFC 7662) | hardware attestation (RATS RFC 9334), TEE/HSM | +| **Introduction Grant** (P1) | HSM/PKCS#11/KMIP key custody | +| minimal attestation (`self_asserted`/`vendor_attested`) + revocation | external CA protocols beyond the default (EST/SCEP/CMP) — *producer-side anyway* | + +**The broker boundary is the great minimizer:** because DCM only brokers, it does **not** implement the credential-type protocols (ACME/EST/SCEP/CMP/KMIP/PKCS#11) — those are **producer-side**. DCM core ≈ four things: mTLS, OIDC, Introduction Grant, attestation-verify+revocation. + +## Proposed new primitives (the only net-new — see trust-flows.md) +- **P1 — Introduction Grant** *(required v1)*: short-lived, audience-scoped token for direct consumer↔producer issuance (OAuth2 Token Exchange RFC 8693; validated vs DCM JWKS). **Introduction-only** — the producer still independently authenticates the consumer per the selected spec; it must never become a bearer-secret. +- **P2 — Attestation Verifier + Accreditation-Authority registry** *(thin v1: self/vendor; accredited+RATS deferred)*: extends the Trust-Anchor model + RATS. +- **P3 — Bootstrap anchor/token** *(small)*: kubeadm/SPIRE-style one-time enrolment; *is* the internal root install. +- **P4 — Credential scoring profile** *(config, not new engine)*: portability as tiebreak. + +## Trust attestation (we attest to ourselves) +DCM/UDLM earn trust by **self-application** — running this model on themselves and publishing a verifiable **Trust Posture Statement** (a *projection* of existing records: DecisionRecord + Accreditation + CONFORMANCE + Audit; **zero new primitives**) at `/.well-known/udlm/trust-posture`. Same bar, same tooling we require of producers. The render/assess role is fulfilled by **any conformant, independent assessor** — *non-normative*; nothing here names or depends on a specific tool. (A separate assessment/testbed consumer may exercise it.) Full model: `trust-attestation.md`. + +## Options considered +- **Implicit/internal trust (assert our own trustworthiness)** — rejected: unverifiable across boundaries. +- **Invent a DCM trust/credential framework** — rejected (ADR-021): adopt PKI/OAuth/OIDC/CMVP/CC/eIDAS/RATS. +- **Route credential values through DCM to normalize** — rejected (CPX-001). +- **Single fixed credential envelope** — rejected: re-specs solved standards, loses per-type fidelity. +- **Capability-declared / requirement-selected, value direct, broker-mediated, attestation-gated, market-graded** — **chosen.** + +## Data · Policy · Provider +- **Data (UDLM):** `credential_capability` + `attestation[]` + `trust_posture` on the provider declaration; `credential_requirements` on the request; the Trust Posture projection. +- **Policy (DCM):** selection (filter/gate/score/select/negotiate) + the five-plane enforcement + profile gating, all audited; broker — never in the value/negotiation path. +- **Provider:** declares the matrix, furnishes market-required attestation, implements the selected spec, issues **direct** to the consumer under the Introduction Grant. + +## Consequences +- One coherent model over mostly-existing primitives (Placement, Policy Engine, Governance Matrix, Accreditation, revocation, PCA/Trust-Anchors, OIDC/introspection, Audit). Net-new ≈ P1 + thin P2. +- New UDLM data: provider capability declaration gains `credential_capability` + `trust_posture`; request gains `credential_requirements`. +- New conformance: a realization **exposes** its trust posture and **upholds** validation on every plane; a Credential Provider implements each spec it declares + furnishes market-required attestation. +- Trust becomes **provably** market-appropriate (gated on attested, validatable data) and **self-attested** (we hold ourselves to the same bar). diff --git a/architecture/adr/023-provider-naturalization-boundary.md b/architecture/adr/023-provider-naturalization-boundary.md new file mode 100644 index 0000000..8fc15dc --- /dev/null +++ b/architecture/adr/023-provider-naturalization-boundary.md @@ -0,0 +1,64 @@ +# ADR-023: Provider Naturalization Boundary (naturalize / denaturalize) + +**Status:** Proposed +**Date:** 2026-07-11 +**Type:** Architecture Decision Record (a `DecisionRecord` with architecture scope) +**Related:** ADR-002 (Data·Policy·Provider triad), ADR-005 (Provider Abstraction), ADR-019 (Placement & Policy), ADR-021 (Adopting External Standards by Reference); UDLM ADR-004 (Provider capability declaration), UDLM ADR-008 (the UDLM/DCM boundary). **Prior art:** Open Service Broker API; DDD *anti-corruption layer*; Crossplane Claim→Composition. + +## Context + +UDLM is a **generic** substrate: typed records, four states, intent + data. A provider executes a **specific** mechanism — an Ansible playbook, libvirt domain XML, a Kubernetes manifest, a KMIP call. Something has to translate between the two, in **both** directions, and the whole system's integrity depends on **where that translation lives**. + +If the mechanism leaks into the substrate — e.g. a `Automation.Playbook` resource type, or a "terraform module" field — the UDLM/DCM boundary (UDLM ADR-008) breaks: a peer that realizes the same intent with a different tool can no longer read the data. So the translation must be named, owned by the provider, and kept out of the substrate. This ADR names that boundary. + +## Decision + +**1. Naturalization / denaturalization — the provider's anti-corruption boundary.** Every provider translates at its edge: +- **Naturalize** (inbound) — take the generic UDLM request + data and render it into the provider's **native** form (playbook vars, a domain definition, a manifest). Generic → provider-native. +- **Denaturalize** (outbound) — take the provider's native results and render them back into **generic UDLM** (realized/discovered state, outputs). Provider-native → generic. + +The native form **never enters the substrate**. UDLM holds only generic intent, data, and denaturalized results; the mechanism is the provider's business. + +**2. Two provider modes — do not conflate them.** A "mode" here is a **capability grouping, not a provider type** (ADR-PROV-002): *resource* mode = declaring `realize_resources` (typed, four-state, lifecycle contract); *process* mode = declaring `execute_workflows` (catalog, ephemeral). Both are capability verbs in the one vocabulary, and a single provider MAY declare both — "mode" names which capability it is exercising, never a mutually-exclusive kind. +- **Resource-mode providers** provision **and manage resources** — they publish **defined resource types** and honor **lifecycle-management contracts** (the four states: intent → requested → realized → discovered; converge; decommission — `contracts/provider-contract.md`). This is **day-0/day-1**: bring a thing into being and keep it converged. *Example: OCP/podman realizing `Compute.Container`; the libvirt provider realizing `Compute.VirtualMachine`.* +- **Process-mode providers** run **processes / automation only** — they publish a **catalog of processes** and the data each needs, execute, and report; they own **no resource** and manage no lifecycle. This is **day-2**: operate, remediate, scan, report. *Example: "run a compliance scan," "rotate now."* + +Both modes naturalize/denaturalize. The difference is only the **outcome shape**: a resource provider persists a typed, four-state resource; a process provider reports an ephemeral result. **A process provider MUST NOT provision or manage resources** — anything that owns a thing is a resource provider with a typed contract. + +**3. The realize loop (shared spine).** catalog → DCM **presents** → consumer **picks + supplies data** → **intent** → **policy** (ADR-002/019, re-entrant) → **request** → **placement** (UDLM ADR-004 capability match) → **naturalize** → **execute** → **denaturalize** → **report → DCM** (realized/discovered) → **report → user**. + +**4. Consequence for the type system.** No provider-mechanism types in UDLM. Capabilities are **declared** (UDLM ADR-004) and offered as `CatalogItem`s; mechanisms are provider-internal. `Automation.Job` carries a chosen `CatalogItem` reference + generic supplied data only — never a playbook, module, or manifest. + +**5. Ownership split — three tiers of "configuration," not two.** The line is *not* "config in the substrate vs not." It is: +- **(i) The resource's *typed, declared* config — UDLM owns it, as intent.** The resource-type schema: a VM's `vcpu`/`memory`/`disks`, a Container's `image`/`ports`, a file-share's `shares`/`workgroup`. This *is* configuration and DCM's interface manages it. A VM is **not** special — model any resource's semantic config as typed properties and it lives here. (The original error was excluding Samba's shares from this tier by leaving Samba unmodeled.) +- **(ii) The provider's *serialization / mechanism* + execution — provider owns it, naturalized (§1), never stored.** Rendering tier (i) into native form — libvirt domain XML, the `smb.conf` *file*, the IPA `[global]` block — and applying it. `smb.conf`-the-file lives here, not with the typed config. +- **(iii) Config *below/outside* the resource's typed schema — handled at the integration scale it warrants (§6).** Files inside a VM's guest OS, an app's own config in a container, opaque knobs not lifted into the type. This tier is met at whatever **scale** the provider supports: **basic text passthrough** (DCM edits it, UDLM stores it as an opaque blob, provider applies) or a **reference/redirect** to the provider's own interface — always via the resource's **managing-provider** link. +- **DCM/UDLM own the lifecycle across all three** — the record, intended state, four-state transitions, ordering, policy (the *what state, when/why*); the provider **executes** the transitions (the *how*). So a VM and Samba are consistent: typed config → UDLM (`vcpu` ≙ `shares`), serialization → provider-naturalized (XML ≙ smb.conf), deep/opaque → referenced. + +**6. Scales of integration — one interface, depth set by the provider (the OpenShift route).** DCM is *always* the configuration interface (single pane of glass); how deep it goes is a **spectrum**, chosen per provider by how much interface detail the provider exposes: +- **Basic (text passthrough).** For records DCM has no native typed interface for, DCM offers a **basic text-level config edit** and **passes it to the provider**; UDLM stores it as **opaque config-as-intent** (a blob), the provider applies it. Like `oc edit` on a raw resource, or a ConfigMap holding a file. Shallow, but still one interface. +- **Full (native typed).** For providers that expose enough interface detail (rich capability declaration, ADR-004), **DCM is the full config interface and UDLM stores the *typed* config** (tier (i)), then passes it to the provider to naturalize + apply. A VM's `vcpu`, a file-share's `shares`. +- **In between / reference.** Partial typing, or a pure **reference/redirect** to the provider's own interface (embedded delegation) when neither fits. + +The provider **always owns execution + the serialization** (naturalization, §1). DCM storing config here is **config-as-intent for a provider to realize**, at the fidelity the integration supports — **not** authoritative config management for its own sake, and **not** a universal store: depth is opt-in per provider, and a resource with no integration is just a record + reference. The deferred first-class-config capability (Consequences) is simply the deep end of this same scale. **Prior art:** Kubernetes' spectrum — deeply-typed native resources ↔ opaque ConfigMaps/Secrets ↔ CRDs+operators, all under one `oc`/`kubectl`; API **aggregation**; LDAP/DNS **referrals**; the Open Service Broker delegation model. + +## Data · Policy · Provider (required lens) + +- **Data** — UDLM carries generic intent + data + denaturalized results only; naturalized/native forms are never Data. `CatalogItem` (offerings + required-data schema) and `Automation.Job` (the generic request) are the Data touchpoints. +- **Policy** — policy evaluates the generic intent before naturalization and gates the request (tenancy, sovereignty, approval, re-entry on drift/denial). Placement policy matches request → capable provider. +- **Provider** — the provider **owns** naturalize/denaturalize and the mechanism. Resource providers additionally honor the lifecycle contract; process providers honor the catalog/execute/report contract. + +## Options considered + +- **(A) Mechanism types in the substrate** (a `Playbook`/`Module` type). Rejected: breaks UDLM ADR-008 — a peer with a different tool can't read the data; couples the substrate to one provider's tooling. +- **(B) Fully opaque provider (no declared capabilities).** Rejected: with nothing declared there is no catalog to present and nothing for placement to match; DCM can't broker. +- **(C) [chosen] Declared capabilities (ADR-004) + a provider-owned naturalize/denaturalize boundary.** Generic in, mechanism hidden, generic out. + +## Consequences + +- **+** The boundary is explicit and enforceable — reviewers can reject any substrate field that is a mechanism. +- **+** Peers interoperate at the generic layer; providers stay swappable (Ansible today, AAP/Terraform tomorrow) with no substrate change. +- **+** Clean home for day-0/1 (resource providers) vs day-2 (process providers) as both arrive. +- **−** Every provider must implement two translations; a thin provider still pays the naturalize/denaturalize cost. +- **Open** — a shared *denaturalization conformance* (how faithfully results must round-trip to generic state) may warrant its own follow-up. +- **Deferred — first-class config ownership.** DCM/UDLM deliberately do **not** absorb application configuration today: §5–6 keep the provider owning config + execution while DCM references and can redirect to it. A **future version may take config ownership as a first-class capability if demand warrants** — customers wanting DCM to be their config source of truth. The embedded/redirect interface (§6) is the on-ramp, so adopting it later is **additive, not a redesign**. Revisit when enough consumers ask for it (same "defer, add-when-demanded" discipline as the `ownership_model` deferral). diff --git a/architecture/adr/024-reference-resolution-and-change-impact.md b/architecture/adr/024-reference-resolution-and-change-impact.md new file mode 100644 index 0000000..00bc6ca --- /dev/null +++ b/architecture/adr/024-reference-resolution-and-change-impact.md @@ -0,0 +1,44 @@ +# ADR-024: Reference Resolution & Change-Impact Cascade + +**Status:** Proposed +**Date:** July 2026 +**Docs:** UDLM ADR-012 (Data References — the data side); [ADR-012](012-data-assembly-layering.md) (Data Assembly & Layering); [ADR-006](006-policy-engine.md) (Policy Engine); [ADR-011](011-sovereignty-data-residency.md) + +## Context + +UDLM lets a field reference shared, governed data by object reference — `{ref_uuid, ref_name, ref_version, reference_data_type}` pointing at an immutable Reference Data Layer version — instead of inlining a copy (UDLM ADR-012). Referenced entities are immutable: a change mints a new version (new uuid), and lineage is a single explicit `supersedes` DAG. That is the **Data**. Two things are left for DCM (**Policy**): *when* references are resolved into a request payload, and *what to do* when referenced data a live payload was built against is later superseded — the change-impact question ("a library embedded in a container image is bumped; what is affected?"). + +The trap is auto-cascade: quietly re-minting dependents when referenced data changes. That rewrites authored intent (someone pinned a version deliberately), assumes forward-compatibility that may not hold, and fans a single base change into an unreviewable blast radius. So impact and action must be separated. + +## Decision + +**1. DCM resolves references at assembly time.** During payload assembly (ADR-012), the assembly engine dereferences each `ref_uuid` to its immutable Reference Data Layer version and injects that version's structured data into the payload, recording the reference (uuid + version) as the field's provenance. Because the target is immutable, resolution is deterministic and reproducible. A **retired** target is refused at resolution; a merely superseded one resolves normally (the pin is honored — immutability). + +**2. Change-impact is consumed, not computed.** UDLM derives the impact map from the `supersedes` DAG + the reverse reference graph, cascading transitively (deployment → image → library). DCM **consumes** that map; it does not maintain a parallel lineage. Impact is advisory data — surfacing it changes nothing. + +**3. Acting on impact is a governed cascade policy — never automatic.** Whether and how to move a dependent off a superseded version is an 8th-family Policy decision (ADR-006), profile-governed along a spectrum: +- **notify** — surface the impact map to the referrer's owner (default; the only floor for sovereign/fsi). +- **propose** — open a new-version PR/change for the owner to review and approve (GitOps). +- **auto-adopt** — mint the dependent's new version automatically. Permitted **only** under an explicitly permissive profile (e.g. dev), **never** sovereign/fsi, and never across a major-version bump of the referenced data. + +The referrer's **owner** decides; the referenced data's change never rewrites a dependent's intent by itself. Cascade runs as an ordinary policy evaluation (auditable per ADR-010), and any adopted change is a normal new immutable version — not an in-place edit. + +## Data · Policy · Provider (required lens) + +- **Data (UDLM)** — the reference shape, the immutable Reference Data Layer versions, the `supersedes` lineage DAG, and the derived transitive impact map (UDLM ADR-012). DCM adds no lineage data of its own. +- **Policy (DCM)** — resolves references into the assembled payload with provenance; consumes the impact map; runs the profile-governed cascade policy (notify → propose → auto-adopt); refuses retired targets and gates deprecated ones per profile. +- **Provider** — n/a for lineage: a provider receives the **naturalized payload with references already resolved and injected** (ADR-023), never raw references, and never resolves or cascades. Its declared capability set is unaffected. + +## Options considered + +- **(A) Auto-cascade in the substrate** — UDLM auto-bumps dependents when referenced data changes. Rejected: rewrites authored intent, crosses the UDLM/DCM boundary (data mutating on a policy trigger), and creates unbounded blast radius. +- **(B) No cascade support — manual only** — ignore the impact map. Rejected: drift becomes invisible; the map is cheap, and "who is pinned to a superseded base?" is exactly the security/compliance question operators must answer. +- **(C) [chosen] DCM resolves references + consumes UDLM's transitive impact map + a profile-governed cascade policy (notify → propose → auto-adopt)** — impact is advisory data, action is a governed decision. + +## Consequences + +- **+** Impact ("who is behind on a superseded base") is answerable across the whole graph, transitively, without mutating anything — the input to security/compliance action. +- **+** Intent is never silently rewritten: moving a dependent forward is always an owner-approved (or profile-permitted) new immutable version, auditable like any other. +- **+** The floor is safe — sovereign/fsi only ever *notify*; auto-adoption is opt-in and profile-scoped. +- **−** Owners must act on notifications; the platform will surface pinned-to-superseded references but not fix them for regulated tenants. That is the intended trade (no silent auto-upgrade of governed intent). +- Pairs with ADR-012 (assembly/provenance) and ADR-023 (references resolved before the provider boundary). diff --git a/architecture/adr/README.md b/architecture/adr/README.md new file mode 100644 index 0000000..ec7da34 --- /dev/null +++ b/architecture/adr/README.md @@ -0,0 +1,46 @@ +# Architecture Decision Records + +Short, reviewable summaries of the major architectural decisions in DCM. Each ADR answers **"Why does this exist and what does it do?"** — not implementation details. + +**Required lens (every ADR / DecisionRecord).** Each decision MUST state its **Data · Policy · Provider** aspects — the three foundational abstractions (ADR-002). *Data* = what's modeled/held (UDLM); *Policy* = what's decided/computed/governed (DCM); *Provider* = what's declared as possible and what executes the mechanism. A decision that can't name all three (or explicitly say "n/a, because…") isn't fully scoped. This is foundational across UDLM and DCM (and any consumer). + +**Reading order:** ADRs 001-003 establish the foundations. Read those first, then jump to whichever ADRs are relevant to your area. + +> **ADRs and the UDLM `DecisionRecord`.** These ADRs are the human-authored, prose form of "why the architecture +> is the way it is." UDLM defines that same WHY as a first-class, machine-trackable record type — the +> **`DecisionRecord`** in the Knowledge entity-type family (`udlm/entities/knowledge-family.md` §4.5) — anchored to +> the capability/decision it justifies, paired with [Audit & Tamper Evidence](010-audit-tamper-evidence.md) and +> field-level provenance ([ADR-012](012-data-assembly-layering.md)). A `DecisionRecord` is the substrate-level, +> **validation-backed** counterpart of an ADR (it reaches `CANONICAL` only with passing use-case validation); the +> authoring/validation loop is realized by a conformant assessment realization (non-normative; nothing here depends on a specific tool). Adopt-by-reference per +> [ADR-021](021-adopting-external-standards.md): DCM records its decisions *as* UDLM DecisionRecords rather than a +> parallel form. + +**DecisionRecord scope & validation.** A `DecisionRecord` is **scoped** (Data·Policy·Provider): **architecture-scoped** records are these ADRs; **policy-scoped** and **provider-scoped** records capture the *why* of a policy or a provider/capability adoption. A record reaches `CANONICAL` via **scope-appropriate validation** — *architecture:* use-case / conformance validation; *policy:* Policy-Engine validation + **Shadow Mode** (a `proposed` policy evaluated against real traffic, never applied); *provider:* attestation verification + conformance. This is **distinct from runtime decisions** (a policy firing, a provider selection), which are captured as **Audit + provenance** ([ADR-010](010-audit-tamper-evidence.md)), **not** DecisionRecords — a DecisionRecord is the deliberate *why* (authoring time, any scope); audit is *what happened* (runtime). The validation runner is therefore not one component but the scope's existing mechanism (conformance / Shadow Mode / attestation verification) — DCM-owned, no external dependency. + +| ADR | Decision | One-Line Summary | +|-----|----------|-----------------| +| [001](001-why-dcm-exists.md) | Why DCM Exists | Unified management plane for on-prem infrastructure — the governance layer above provisioning tools | +| [002](002-three-abstractions.md) | Three Foundational Abstractions | Everything in DCM is Data, Provider, or Policy — no exceptions | +| [003](003-four-lifecycle-states.md) | Four Lifecycle States | Intent → Requested → Realized → Discovered — immutable states linked by entity_uuid | +| [004](004-service-catalog-consumer-experience.md) | Service Catalog & Consumer UX | Four-level hierarchy from resource types to catalog items; consumers declare what, not how | +| [005](005-provider-abstraction.md) | Provider Abstraction | Unified provider model with capability declarations; bidirectional discovery; any platform, same interface | +| [006](006-policy-engine.md) | Policy Engine | Policy-as-code on every request; 8 policy types from gating to orchestration flow | +| [007](007-placement-engine.md) | Placement Engine | Multi-stage scoring: sovereignty pre-filter → capability → capacity → policy scoring | +| [008](008-dependency-resolution.md) | Dependency Resolution | Type-level dependencies trigger automatic sub-requests; binding fields inject runtime values | +| [009](009-api-gateway-control-plane.md) | API Gateway & Control Plane | Single entry point routing to 9 internal services; deterministic pipeline | +| [010](010-audit-tamper-evidence.md) | Audit & Tamper Evidence | Merkle tree (RFC 9162) with configurable granularity; mathematically provable integrity | +| [011](011-sovereignty-data-residency.md) | Sovereignty & Data Residency | First-class enforcement on every lifecycle operation; dual-approval for overrides | +| [012](012-data-assembly-layering.md) | Data Assembly & Layering | Organizational data merges with consumer requests; field-level provenance on everything | +| [013](013-override-exception-governance.md) | Override & Exception Governance | 5 mechanisms from planned exceptions to dual-approval; governance with flexibility | +| [014](014-multi-tenancy-isolation.md) | Multi-Tenancy & Isolation | PostgreSQL RLS enforces tenant isolation at the database layer | +| [015](015-minimal-infrastructure.md) | Minimal Infrastructure | PostgreSQL is the only required dependency; everything else is optional | +| [016](016-application-definition-language.md) | Application Definition Language | **OPEN** — How should consumers define multi-resource applications? Options under evaluation | +| [017](017-brownfield-greening-discovered-ingestion.md) | Brownfield Greening / Discovered Ingestion | Bring existing resources into the four states — two discovery avenues (provider / 3rd-party), Discovered store holds unclaimed, reverse placement → claim → Realized, optional Intent backport; correlation IDs dedup the same resource across sources | +| [018](018-wire-serialization-event-conventions.md) | Wire Serialization & Event Conventions | snake_case payloads end-to-end (AEP-conformant; Go json tags, Python Pydantic native — no alias generator); CloudEvents envelope; event topics use lowercase dot-notation for broker wildcard routing — the runtime side of UDLM's data-model casing | +| [019](019-placement-policy.md) | Placement Policy | The 8th typed policy — declarative affinity/anti-affinity/spread/co-locate/pin over abstract `Topology` kinds; engine evaluates; enforces portability. Consumes UDLM ADR-001/002/004 | +| [020](020-migration-and-operational-gating.md) | Migration & Operational Gating | Migration permission (Governance-Matrix) + sequence (Orchestration-Flow) + freshness gating (compliance Validation) + rehearsal scheduling — reuses existing policy types; control-plane side of UDLM ADR-003/004 | +| [021](021-adopting-external-standards.md) | Adopting External Standards | Adopt (reference) standards like FOCUS/OpenCost/OSCAL/SCIM — don't absorb; providers declare version support, DCM negotiates | +| [022](022-trust-model.md) | DCM Trust Model (incl. Credential API Selection) | Uphold·participate·expose a full trust model across 5 planes (PKI/mTLS, OAuth/OIDC, credential issuance, attestation, federation); DCM is a trust **broker**, not a credential authority; credentials brokered like placement (declare→select→attest, value direct CPX-001); security/trust/fit > portability; market-graded by profile; self-attested | +| [023](023-provider-naturalization-boundary.md) | Provider Naturalization Boundary | Generic intent + data in, mechanism hidden, generic denaturalized results out; naturalized/native forms are never Data; the provider owns naturalize/denaturalize (UDLM ADR-008) | +| [024](024-reference-resolution-and-change-impact.md) | Reference Resolution & Change-Impact Cascade | DCM resolves data references into the payload at assembly; consumes UDLM's transitive impact map; the cascade action (bump dependents) is a profile-governed policy (notify → propose → auto-adopt), never automatic — the policy counterpart to UDLM ADR-012 |