diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..aaf1ae2 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,29 @@ +# Contributing to DCM + +DCM — the Data Center Management control plane — is open-source under Apache License 2.0. Contributions +to the control plane, the provider ecosystem, and the architecture docs are welcome. The architecture is +captured in `architecture/`; the major decisions are recorded as ADRs in `architecture/adr/`. + +## Subject-scoped pull requests (default) + +The default unit of contribution is **one subject per PR** — a single, complete logical change, titled +by its subject (e.g. "Enable cost provider", "Adopt FOCUS 1.4 for cost", "Add EgressFirewall to the +namespace"). Keep PRs to roughly ≤2–3k lines; if a subject is larger, split it along logical boundaries +into a sequence of independently reviewable, subject-scoped PRs rather than forcing one oversized change. +Prefer logical boundaries over size-driven cuts, and never bundle unrelated subjects. Lead every PR +description with a short **Why** (the rationale), linking the ADR or requirement when one exists. + +## Document the why + +Every non-trivial change records its rationale, not just its diff: +- **Architectural decisions** get an ADR in `architecture/adr/` (next available number; follow the + existing shape — Context, Decision, Alternatives Considered, Consequences). One decision per ADR; + don't bundle. +- **Requirement changes** update the relevant requirement set (`dcm-platform-requirements.md` and the + ID series — `ADS-`, `AUD-`, `RDG-`, …). +- A reviewer should be able to reconstruct *why* a change exists from the repo, not just *what* changed. + +## Licensing + +By contributing to DCM you agree your contributions are licensed under Apache License 2.0, matching the +project license. diff --git a/architecture/00-split-manifest.md b/architecture/00-split-manifest.md new file mode 100644 index 0000000..6d434ce --- /dev/null +++ b/architecture/00-split-manifest.md @@ -0,0 +1,867 @@ +--- +status: ✅ Decisions locked — ready for execution +created: 2026-05-26 +purpose: Plan and record the split of architecture/data-model/ into two repos (udlm + dcm) +permanence: Kept in dcm as a permanent contextual artifact (helps future contributors understand the boundary) +--- + +# UDLM / DCM Split Manifest + +This document is the **plan of record** for splitting the existing +`architecture/data-model/` directory into two independent repos: + +- **`udlm`** — Universal Data Lifecycle Model (substrate) — `github.com/croadfeldt/udlm` +- **`dcm`** — Data Center Management (operational platform built on udlm) — this repo + +This document is kept in dcm as a **permanent contextual artifact**. It captures +the rationale and boundary work that drove the split; future contributors can +read it to understand why files live where they do and which decisions are +load-bearing. + +Companion doc: [`00-layering-data-model-vs-dcm.md`](00-layering-data-model-vs-dcm.md) +— captures the conceptual layering that justifies this split. (Held uncommitted +until the split lands so its repo references can be made concrete.) + +--- + +## The boundary rule (recap) + +For each file or section, the test is: + +> *"Could a peer of DCM, built independently, choose to do this differently and +> still be a valid realization of the same data?"* + +- **Yes →** belongs in **dcm** (it's an implementation choice) +- **No, it would break interop or invalidate the data →** belongs in **udlm** + (it's a substrate invariant) + +**udlm owns** entity types, four states + transitions + invariants, contracts +(provider, policy, event payloads, data store), provenance, identity, reference +taxonomies. The state vocabulary lives here because peers must share it to +interoperate. + +**dcm owns** the convergence engine, runtime/orchestration, deployment topology, +monitors, integrations with specific external systems, ease-of-use packaging, +implementation specifics. + +### Compatibility model (LOCKED) + +**udlm enforces wire-level compatibility at the data/event/contract boundary; +it does not enforce implementation portability.** + +Concretely: +- Any system conformant to udlm version X produces data that any other system + conformant to the same major version of udlm can read, interpret, and + exchange — **versioning applicability rules withstanding**. +- Federation between peers is **literal interop**, not "architecturally similar + systems requiring adapters." +- A peer realization's storage, internal APIs, control-plane components, and + runtime mechanics are NOT constrained by udlm — those are dcm-layer choices. + +**Implications for udlm spec authoring:** +1. Wire formats are **normative** (identifier strings, timestamps, event payloads, error envelopes). +2. Error/code/state vocabularies that cross interop boundaries are **closed**. +3. udlm **must** define a schema-sharing mechanism so peers can exchange schemas + for their custom types and resolve each other's data with context. +4. Versioning is a first-class concern — every wire contract carries a version + and a compatibility window. + +This position is the K8s precedent: K8s API + CRDs are wire-compatible across +distributions; controllers are not portable. We are in the same shape. + +### Sidebar — the validating analogy + +A useful test (user-provided, captured here for future contributors): + +| Analogy | Layer | +|---|---| +| **Directions** — where you can go, what destinations exist | udlm | +| **Goals** for the rules of the road (safety, predictability, interop) | udlm | +| **Rules-of-the-road requirements** (what cars + drivers must satisfy) | udlm | +| **Driver requirements** (license classes, competencies) | udlm | +| **Published rules-of-the-road manual** (RFCs, NIST cited as substrate) | udlm | +| **The road itself** (control-plane components, persistence) | dcm | +| **Turn signals** (the physical lights and signaling infrastructure) | dcm | +| **Cars actually driving** (convergence engine, the intent→realized loop) | dcm | +| **Actual rules of the road** (specific enforcement, matrix evaluator) | dcm | +| **DMV licensing process** (profile thresholds, approval enforcement, GitOps PR) | dcm | + +The analogy stress-tests classifications. When unsure: "is this a *direction +or requirement* (udlm) or *infrastructure or enforcement* (dcm)?" + +--- + +## File-level classification (61 files total) + +Counts: **udlm 22 / dcm 18 / both 21** = 61. (Plus 7 net-new udlm contract docs +authored during the split — see "Newly identified udlm contracts" below.) + +### Pure udlm (22 files) — move as-is + +``` +00-context-and-purpose.md 12-audit-provenance-observability.md +00-foundations.md 15-universal-groups.md +01-entity-types.md 16-universal-audit.md +02-four-states.md 30-composite-service-model.md +03-layering-and-versioning.md 33-event-catalog.md +04b-ownership-sharing-allocation.md 50-subscription-lifecycle.md +05-resource-type-hierarchy.md 52-test-framework-specification.md +07-service-dependencies.md A-provider-contract.md +08-resource-grouping.md B-policy-contract.md +09-entity-relationships.md +10-information-providers.md +11-data-store-contracts.md +11-storage-providers.md +``` + +### Pure dcm (18 files) — move as-is + +``` +14-policy-profiles.md 39-dcm-self-health.md +17-deployment-redundancy.md 41-operational-reference.md +18-webhooks-messaging.md 42-itsm-integration.md +22-dcm-federation.md 44-kessel-integration-evaluation.md +23-notification-model.md 45-consistency-review.md +25-control-plane-components.md 46-workload-analysis.md +29-scoring-model.md 47-accreditation-monitor.md +34-api-versioning-strategy.md 49-implementation-specifications.md +35-session-revocation.md +36-internal-component-auth.md +``` + +### Needs per-section split (21 files) — see detail below + +``` +00-design-priorities.md 31-credential-management.md +04-examples.md 31-credential-provider-model.md +06-resource-service-entities.md 32-authority-tier-model.md +13-ingestion-model.md 37-scheduled-requests.md +19-auth-providers.md 38-request-dependency-graph.md +20-registry-governance.md 40-standards-catalog.md +21-information-providers-advanced.md 43-provider-callback-auth.md +24-operational-models.md 48-location-topology-layers.md +26-accreditation-and-authorization-matrix.md 51-infrastructure-optimization.md +27-governance-matrix.md 53-capability-discovery.md +28-federated-contribution-model.md +``` + +--- + +## Proposed `udlm` repo layout (LOCKED — numeric prefixes dropped) + +``` +udlm/ +├── README.md # what udlm is, who consumes it, how to extend +├── CONFORMANCE.md # NEW — what a conformant realization must provide (wire contract surface) +├── foundations/ +│ ├── context-and-purpose.md +│ ├── foundations.md +│ ├── entity-types.md +│ ├── four-states.md +│ ├── layering-and-versioning.md +│ ├── examples.md # ALL examples (kept clean — see resolved #1) +│ └── ownership-sharing-allocation.md +├── entities/ +│ ├── resource-type-hierarchy.md +│ ├── resource-service-entities.md # udlm portion +│ ├── service-dependencies.md +│ ├── resource-grouping.md +│ ├── entity-relationships.md +│ └── composite-service-model.md +├── contracts/ +│ ├── provider-contract.md # was A- +│ ├── policy-contract.md # was B- +│ ├── information-providers.md +│ ├── data-store-contracts.md +│ ├── storage-providers.md +│ ├── information-providers-advanced.md # udlm portion +│ ├── event-catalog.md +│ ├── provider-callback-auth.md # udlm portion (mechanism-neutral two-layer abstract — see resolved #3) +│ ├── capability-discovery.md # udlm portion +│ ├── identifier-scheme.md # NEW +│ ├── time-and-clock.md # NEW +│ ├── error-model.md # NEW +│ ├── retry-semantics.md # NEW (extract from 24/25/recovery profiles) +│ ├── rate-limit-and-backpressure.md # NEW (extract from 49) +│ └── schema-sharing.md # NEW (peer schema exchange — required by wire-compat) +├── lifecycle/ +│ ├── ingestion-model.md # udlm portion +│ ├── operational-models.md # udlm portion (timeouts, cancellation, orphan contracts) +│ ├── scheduled-requests.md # udlm portion +│ ├── request-dependency-graph.md # udlm portion +│ └── subscription-lifecycle.md +├── governance/ +│ ├── auth-providers.md # udlm portion (auth mode taxonomy) +│ ├── registry-governance.md # udlm portion +│ ├── accreditation-and-authorization-matrix.md # udlm portion +│ ├── governance-matrix.md # udlm portion +│ ├── federated-contribution-model.md # udlm portion +│ ├── credentials.md # NEW — merged 31a + 31b (see resolved #6) +│ └── authority-tier-model.md # udlm portion +├── observability/ +│ ├── audit-provenance-observability.md +│ ├── universal-groups.md +│ └── universal-audit.md +├── topology/ +│ └── location-topology-layers.md # udlm portion: layered-topology contract + assembly rules + lifecycle ONLY (specific 9-layer hierarchy moved to dcm — see resolved #4) +├── design-principles/ +│ ├── design-priorities.md # udlm portion (the four principles as contracts) +│ └── infrastructure-optimization.md # udlm portion (data-contract principle + four domains; PostgreSQL mandate moved to dcm — see resolved #5) +├── reference/ +│ └── standards-catalog.md # udlm portion (the normative external standards list) +├── docs/ +│ └── consumer-perspective.md # NEW — the "driver's handbook" (narrative perspective for consumers) +└── tests/ + └── test-framework-specification.md +``` + +**Notes on the layout:** +- Numeric prefixes dropped. Reading order is conveyed by README + section names. +- `CONFORMANCE.md` at top-level: defines what any peer realization must provide + to be wire-compatible. This is the conformance surface DAV will validate + against. +- `contracts/` is the wire-compatibility surface — every doc here is normative. +- `docs/` is the single narrative directory (no separate `guides/`). +- udlm has no `deployment/`, no `mcp-servers/`, no API surface — it's pure + specification. + +--- + +## Proposed `dcm` repo layout + +dcm gets a layout shaped around its concerns, not mirroring udlm: + +``` +dcm/ +├── README.md # what dcm is, links to udlm +├── architecture/ +│ ├── overview.md # links to udlm as substrate +│ ├── layering.md # adapted from 00-layering-data-model-vs-dcm.md +│ ├── control-plane/ +│ │ ├── components.md # ← 25-control-plane-components +│ │ ├── self-health.md # ← 39-dcm-self-health +│ │ ├── internal-component-auth.md # ← 36 +│ │ ├── session-revocation.md # ← 35 +│ │ └── api-versioning.md # ← 34 +│ ├── convergence-engine/ +│ │ ├── overview.md # new — the intent → realized loop +│ │ ├── policy-evaluation.md # ← 14, dcm parts of 27 +│ │ ├── scoring.md # ← 29 +│ │ ├── recovery-and-retry.md # ← dcm parts of 24 +│ │ └── dependency-orchestration.md # ← dcm parts of 38 +│ ├── ingestion/ +│ │ ├── engine.md # ← dcm parts of 13 +│ │ └── workload-analysis.md # ← 46 +│ ├── credentials-and-auth/ +│ │ ├── auth-implementation.md # ← dcm parts of 19 +│ │ ├── credentials.md # ← dcm parts of 31a + 31b (CONSOLIDATED) +│ │ ├── provider-callback.md # ← dcm parts of 43 (mTLS + interaction credential mechanism) +│ │ └── authority-enforcement.md # ← dcm parts of 32 +│ ├── governance-enforcement/ +│ │ │ # NOTE: matrix-evaluator content merged into convergence-engine/policy-evaluation.md (← dcm parts of 27) +│ │ ├── accreditation-monitor.md # ← 47, dcm parts of 26 +│ │ ├── registry-enforcement.md # ← dcm parts of 20 +│ │ ├── contribution-pipeline.md # ← dcm parts of 28 +│ │ └── policy-profiles.md # ← 14 +│ ├── runtime-features/ +│ │ ├── scheduling.md # ← dcm parts of 37 +│ │ ├── notifications.md # ← 23 +│ │ ├── webhooks-messaging.md # ← 18 +│ │ └── federation-runtime.md # ← 22 +│ ├── topology/ +│ │ ├── canonical-9-layer-hierarchy.md # ← MOVED from udlm 48 (Country → ... → Unit as DCM's canonical default) +│ │ └── placement-and-priority-bands.md # ← dcm parts of 48 +│ ├── persistence/ +│ │ ├── postgres-mandate.md # ← MOVED from udlm 51 (single-required-infrastructure decision) +│ │ └── postgres-implementation.md # ← dcm parts of 51 +│ ├── integrations/ +│ │ ├── itsm.md # ← 42 +│ │ └── kessel-evaluation.md # ← 44 +│ ├── design-principles.md # ← dcm parts of 00-design-priorities +│ ├── operator-perspective.md # NEW — how to operationalize udlm (companion to udlm's consumer-perspective) +│ └── consistency-review.md # ← 45 (meta doc, lives here) +├── deployment/ # (existing) +├── requirements/ +│ └── dcm-platform-requirements.md +├── examples/ +│ └── three-tier-application.md # ← dcm parts of 04-examples (orchestration scenarios) +└── reference/ + └── implementation-standards.md # ← dcm parts of 40-standards-catalog +``` + +**Notes on the dcm layout:** +- Organized by **architectural concern**, not by file number. Numbers were + scaffolding; the split is the moment to drop them. +- Every dcm spec opens with: `> Implements contracts defined in udlm: [link]` +- The `convergence-engine/` group is new — it's the heart of dcm and currently + scattered across 24, 27, 38, 14. Worth consolidating. +- `runtime-features/` is the catch-all for "things dcm does that aren't core + convergence": scheduling, notifications, federation, webhooks. + +--- + +## Per-section split for the 21 "both" files + +For each file: which sections go to udlm, which to dcm, and how the dcm doc +references the udlm doc. + +> **Note on paths**: the path references in the per-section blocks below use +> the **legacy numeric layout** (e.g., `udlm/40-governance/27-...md`). The final +> layout drops numeric prefixes per the LOCKED udlm layout above. Mapping is +> mechanical: drop the numeric directory prefix, drop the numeric file prefix. +> Example: `udlm/40-governance/27-governance-matrix.md` → +> `udlm/governance/governance-matrix.md`. The execution phase will apply this +> rename uniformly. + +### 1. `00-design-priorities.md` + +**udlm sections** → `udlm/70-design-principles/00-design-priorities.md` +- *Design Principles as Interoperability Substrate* — Four invariant principles (consumer sovereignty, zero trust, federation, policy as code) form the contract foundation any realization must honor. +- *Authority Tiers (model definition)* — Ordered decision authority vocabulary (auto, reviewed, verified, authorized) plus custom extensions. +- *Profile Scaling Model (definition)* — Named profiles (minimal, dev, standard, prod, fsi, sovereign) with constraint matrices. + +**dcm sections** → `dcm/architecture/design-principles.md` +- *Design Priorities: Implementation Choices* — Specific trade-offs (latency vs governance rigor, velocity vs stability). +- *Approval Tier Model (runtime enforcement)* — Tier-to-capability mappings and enforcement gates. +- *Profile-Governed System Constraints* — Per-profile enforcement of limits and validation modes. +- *Policy as Code Requirement* — Integration with external policy engines (OPA, etc.) and audit logging. +- *Documentation Discipline Requirements* — Internal governance for document lifecycle (status badges, related-doc links). + +**Cross-ref:** dcm doc opens with: *"Implements the design principles defined in [udlm/70-design-principles/00-design-priorities.md]."* + +--- + +### 2. `04-examples.md` — **resolved: all examples → udlm, separate dcm doc** + +**udlm sections** → `udlm/00-foundations/04-examples.md` (kept clean — all examples) +- *VM Provisioning Example (basic intent-to-realized lifecycle)* +- *IP Allocation Example (allocation ownership)* +- *VLAN Attachment Example (cross-entity coordination)* +- *Brownfield Ingestion Example* +- *Drift Detection Example* + + Each example is rewritten to stay contract-level — describe what happens at + each state without orchestration mechanics. Any paragraphs that wandered into + "DCM does X" get dropped (those scenarios re-surface in the dcm examples doc). + +**dcm sections** → `dcm/examples/orchestration-scenarios.md` (new — composed scenarios) +- *Three-Tier Application Example (full dependency group orchestration)* +- *VM Provisioning with timeout/cancellation propagation* +- *IP Allocation with provider's internal lifecycle reconciliation* +- New scenarios specific to dcm features (retry, scoring-driven placement, etc.) + +**Cross-ref:** dcm doc opens with: *"Builds on the canonical examples in [udlm/00-foundations/04-examples.md] to illustrate dcm-specific orchestration features."* + +**Resolution:** Originally proposed per-paragraph splitting; analogy made the +cleaner answer obvious — keep all udlm examples as clean contract illustrations, +let dcm write its own orchestration-scenarios doc fresh. + +--- + +### 3. `06-resource-service-entities.md` + +**udlm sections** → `udlm/10-entities/06-resource-service-entities.md` +- *Resource/Service Request vs Entity (fundamental distinction)* +- *Ownership Models (allocation, whole_allocation, full_transfer, hybrid_transfer)* +- *Entity Lifecycle (provider-side: requested, creating, created, deleting, deleted)* +- *Provider Internal Lifecycle and Notification Model* — what events providers emit and what fields they may update. + +**dcm sections** → distributed across `dcm/architecture/convergence-engine/` and `dcm/architecture/control-plane/` +- *Request/Entity Relationship Management* — operational tracking → `convergence-engine/` +- *Ownership Model Enforcement at Dispatch* — dispatcher logic → `convergence-engine/` +- *Provider Notification Consumption* — receipt, validation, reconciliation → `runtime-features/` or `convergence-engine/` +- *Entity Lifecycle Monitoring* — polling/webhook detection → `runtime-features/` + +**Cross-ref:** dcm convergence engine doc cites this udlm doc as the authoritative entity-lifecycle contract. + +--- + +### 4. `13-ingestion-model.md` + +**udlm sections** → `udlm/30-lifecycle/13-ingestion-model.md` +- *Brownfield Ingestion Problem Statement and Flow* +- *Enrichment Stages (discovery, enrichment, readiness)* +- *Transitional Tenant Mechanism* +- *Auto-Assignment Signals* — the contract; rules of which signals exist. +- *Ingestion Lifecycle (states: discovered, enriching, ready, ingested)* + +**dcm sections** → `dcm/architecture/ingestion/engine.md` +- *Ingestion Engine Implementation* +- *Information Provider Integration (polling/webhook orchestration)* +- *Enrichment Policy Enforcement (specific rules, profile-driven)* +- *Transitional Tenant and Auto-Assignment Execution* +- *Ingestion Scheduling* + +**Cross-ref:** dcm engine doc opens: *"Realizes the ingestion contract defined in [udlm/30-lifecycle/13-ingestion-model.md]."* + +--- + +### 5. `19-auth-providers.md` + +**udlm sections** → `udlm/40-governance/19-auth-providers.md` +- *Authentication Modes (built-in, GitHub/GitLab OAuth, LDAP, AD, OIDC, mTLS)* — taxonomy of supported mechanisms. +- *Multiple Provider Authentication* — multi-provider routing as a contract. +- *Credential Types and Issuance* — data model only. + +**dcm sections** → `dcm/architecture/credentials-and-auth/auth-implementation.md` +- *Authentication Implementation within DCM* — library choices, integration mechanics. +- *Credential Management Service Integration* +- *Provider Authentication Routing Logic* +- *Session Management and Token Lifecycle* + +**Cross-ref:** dcm doc cites udlm/40-governance/19-auth-providers.md as the mode-taxonomy contract. + +--- + +### 6. `20-registry-governance.md` + +**udlm sections** → `udlm/40-governance/20-registry-governance.md` +- *Three-Tier Registry Model (submission, review, publication)* +- *Proposal/Review/Publication Workflow* — the artifact lifecycle contract. +- *Versioning and Deprecation Lifecycle* +- *Resource Type Registry (standard type definitions + extension contract)* + +**dcm sections** → `dcm/architecture/governance-enforcement/registry-enforcement.md` +- *Registry Governance Enforcement* — operational enforcement of the three-tier workflow. +- *Provider Selection Tie-Breaking* — selection algorithm when multiple providers match. +- *Artifact Lifecycle Management* — storage, versioning, deprecation warnings. +- *Review Queue and Approval Workflow* — review-tier mechanics. + +**Cross-ref:** dcm doc: *"Enforces the registry contract in [udlm/40-governance/20-registry-governance.md]."* + +--- + +### 7. `21-information-providers-advanced.md` + +**udlm sections** → `udlm/20-contracts/21-information-providers-advanced.md` +- *Confidence Scoring and Hybrid Descriptor Model* +- *Authority and Priority Declarations* +- *Schema Versioning for Providers* +- *Well-Known Provider Registry (the contract for canonical providers)* + +**dcm sections** → fold into `dcm/architecture/ingestion/engine.md` +- *Ingestion-Time Conflict Detection and Resolution* +- *Write-Back Capability Implementation* +- *Air-Gapped Verification Model* +- *Provider Priority and Fallback Logic* + +**Cross-ref:** dcm ingestion engine doc cites this udlm doc as the trust/authority contract. + +--- + +### 8. `24-operational-models.md` + +**udlm sections** → `udlm/30-lifecycle/24-operational-models.md` +- *Timeout Model and State Machine* — deadline contract. +- *Cancellation Request and Propagation Model* — cancel state contract. +- *Orphan Detection and Prevention* — accountability contract. +- *Discovery Scheduling and Continuous Reconciliation* — reality-vs-intent contract. +- *Recovery Policy Model* — failure semantics contracts. +- *Compensation* — rollback contract. + +**dcm sections** → `dcm/architecture/convergence-engine/recovery-and-retry.md` +- *Timeout Enforcement Mechanisms* +- *Cancellation Execution and Cleanup* +- *Orphan Detection Implementation* +- *Discovery Job Scheduling and Execution* +- *Recovery Policy Evaluation* +- *Compensation Execution* + +**Cross-ref:** dcm doc: *"Implements the operational contracts in [udlm/30-lifecycle/24-operational-models.md]."* + +--- + +### 9. `26-accreditation-and-authorization-matrix.md` + +**udlm sections** → `udlm/40-governance/26-accreditation-and-authorization-matrix.md` +- *Data Classification Levels (restricted, sensitive, internal, public)* +- *Accreditation Model and Lifecycle* +- *Accreditation Gap Handling (policy options)* +- *Authorization Matrix (data/capability × subject/context)* +- *Zero Trust Interaction Model (five-check boundary)* +- *Federation Tunnel Model (the contract for secure inter-DCM channels)* + +**dcm sections** → split between `dcm/architecture/governance-enforcement/accreditation-monitor.md` and `dcm/architecture/runtime-features/federation-runtime.md` +- *Accreditation Governance Enforcement* → accreditation-monitor +- *Authorization Evaluation at Runtime* → governance-enforcement +- *Zero Trust Boundary Implementation* → spans multiple dcm areas +- *Federation Tunnel Establishment and Maintenance* → federation-runtime +- *Profile-Governed Accreditation Constraints* → governance-enforcement + +**Cross-ref:** all dcm fragments cite this udlm doc as the security-substrate contract. + +--- + +### 10. `27-governance-matrix.md` + +**udlm sections** → `udlm/40-governance/27-governance-matrix.md` +- *Unified Governance Matrix as Single Enforcement Point* (the architectural invariant) +- *Matrix Four Axes (Subject/Data/Target/Context)* +- *Rule Structure and Decision Vocabulary (ALLOW, DENY, STRIP_FIELD, REDACT, AUDIT_ONLY)* +- *Soft vs Hard Enforcement* (the distinction as a contract) +- *Field-Level Controls* (the granular policy contract) + +**dcm sections** → `dcm/architecture/convergence-engine/policy-evaluation.md` (matrix-evaluator content merged here; no separate matrix-evaluator.md) +- *Evaluation Algorithm* +- *Hard Enforcement Mechanics* +- *Soft Enforcement Execution* +- *Sovereignty Zone Management* +- *Profile-Governed Policy Configurations* +- *Policy Caching and Invalidation* + +**Cross-ref:** dcm doc: *"Implements the governance matrix contract in [udlm/40-governance/27-governance-matrix.md]."* + +--- + +### 11. `28-federated-contribution-model.md` — **resolved by wire-compat decision** + +**udlm sections** → `udlm/40-governance/28-federated-contribution-model.md` +- *Four Contributor Types (Platform Admin, Consumer/Tenant, Service Provider, Peer DCM)* — federated actor taxonomy. +- *Contribution Artifact Types (resource types, policies, profiles, accreditations, locations, credentials, provider definitions)* +- *Universal Contribution Pipeline (submission → review → publication)* — governance invariant. +- *Consumer/Provider/Federation Contribution Models* — what each contributor type may contribute (this is contract: who may write what). +- *Artifact Lifecycle and Versioning* + +**dcm sections** → `dcm/architecture/governance-enforcement/contribution-pipeline.md` +- *Contribution Store Structure* +- *Review Queue and Approval Workflow* (GitOps PR mechanics live here) +- *Contribution Pipeline Orchestration* +- *Consumer Contribution Enforcement* +- *Provider Contribution Integration* +- *Federation Contribution Synchronization* + +**Cross-ref:** dcm doc: *"Operationalizes the contribution contract in [udlm/40-governance/28-federated-contribution-model.md] via a GitOps-style PR workflow."* + +**Note:** The GitOps-specific bits (PR review, branch protection rules) are a dcm choice. A peer of dcm could use a different review channel. + +--- + +### 12 + 13. `31-credential-management.md` + `31-credential-provider-model.md` — **resolved: merged on both sides** + +**udlm sections** → `udlm/governance/credentials.md` (one consolidated doc) +- *Credential Scope (DCM-internal vs consumer-facing)* +- *Credential Types (api_key, JWT, mTLS cert, SSH key, secret, signing key, HSM-backed, dcm_interaction)* — full taxonomy +- *Credential Lifecycle (issuance, active, rotation, revocation, expired)* +- *Rotation Protocol (parallel validity windows)* — contract +- *Revocation Model and Propagation* — contract +- *Consumer Credential Delivery* — the contract, not the mechanism +- *Provider API Contract for Credentials* — how providers accept credentials +- *Cryptographic Requirements* (defers to standards catalog) +- *Registration and Profile-Governed Configuration* — the constraint vocabulary + +**dcm sections** → `dcm/architecture/credentials-and-auth/credentials.md` (one consolidated doc) +- *Credential Storage and Access Control* +- *Credential Generation Implementation* +- *Issuance Flow Orchestration* +- *Rotation Job Scheduling and Execution* +- *Revocation Enforcement Across Providers* +- *Consumer Delivery Mechanics* +- *Provider Authentication Validation* +- *Profile-Governed Constraints (enforcement)* +- *Integration with External Services* + +**Cross-ref:** dcm doc opens: *"Implements the credential contracts in [udlm/governance/credentials.md]."* + +**Resolution:** The historical 31a/31b split (management vs provider-model) was confusing — heavy overlap, unclear boundary. Wire-compat decision forced consolidation: peers must agree on a single credential model for interop. + +--- + +### 14. `32-authority-tier-model.md` + +**udlm sections** → `udlm/40-governance/32-authority-tier-model.md` +- *Core Authority Tier Model (auto, reviewed, verified, authorized)* +- *Decision Gravity Vocabulary* — what decisions require what tiers. +- *Custom Tier Definition and Contribution* — extension contract. +- *Tier Registry Change Impact Detection with Degradation Review Gate* — approval continuity contract. + +**dcm sections** → `dcm/architecture/credentials-and-auth/authority-enforcement.md` +- *Tier Evaluation Algorithm* +- *Approval Authority Mapping* +- *Profile Threshold Configuration* +- *DCMGroup Assignment* +- *Tier Enforcement at Decision Points* +- *Degradation Review Orchestration* + +**Cross-ref:** dcm doc: *"Enforces the authority tier contract in [udlm/40-governance/32-authority-tier-model.md]."* + +--- + +### 15. `37-scheduled-requests.md` + +**udlm sections** → `udlm/30-lifecycle/37-scheduled-requests.md` +- *Scheduling Model (immediate, at, window, recurring)* — deferral contract. +- *Request State During Deferral (SCHEDULED state, paused at ACKNOWLEDGED)* +- *Maintenance Windows* — coordination contract. +- *Deadline Enforcement* — scheduling deadline contract. + +**dcm sections** → `dcm/architecture/runtime-features/scheduling.md` +- *Request Scheduler Component* +- *Deferred Request Lifecycle Management* +- *Maintenance Window Scheduling Logic* +- *Deadline Evaluation and Timeout Enforcement* +- *Consumer API Additions* (endpoints) +- *New Events* (request.scheduled, request.activation_pending, etc.) +- *Profile-Governed Scheduling Constraints* + +**Cross-ref:** dcm doc: *"Implements the scheduling contract in [udlm/30-lifecycle/37-scheduled-requests.md]."* + +--- + +### 16. `38-request-dependency-graph.md` + +**udlm sections** → `udlm/30-lifecycle/38-request-dependency-graph.md` +- *Request Dependency Group Structure* (group_uuid, group_handle, member set) +- *wait_for Values (acknowledged, approved, dispatched, realized)* — activation contract. +- *Field Injection Mechanism* — propagation contract. +- *PENDING_DEPENDENCY Status* — blocked-state contract. +- *Failure Handling (on_failure: cancel_remaining | continue)* — propagation policy contract. +- *Group Timeout* — group-level deadline contract. +- *Relationship to composite service definitions* — scoping guidance. + +**dcm sections** → `dcm/architecture/convergence-engine/dependency-orchestration.md` +- *Request Dependency Graph Submission and Parsing* +- *Dependency Resolution and Dispatch Orchestration* +- *PENDING_DEPENDENCY State Lifecycle* +- *Failure Handling Execution* +- *Group Timeout Enforcement* +- *Consumer API Endpoints* +- *New Events* +- *Profile-Governed Constraints* + +**Cross-ref:** dcm doc cites this udlm doc as the multi-request coordination contract. + +**Note:** Agent's output referenced `30-meta-provider-model.md` which doesn't exist — actual file is `30-composite-service-model.md`. Reference is to **composite service model**, which is pre-defined ordering vs ad-hoc dependency groups. + +--- + +### 17. `40-standards-catalog.md` + +**udlm sections** → `udlm/90-reference/40-standards-catalog.md` +- All six normative standards groups (identity/access, auth protocols, crypto, data model/serialization, operational, compliance) — these are external standards we cite as substrate requirements. + +**dcm sections** → `dcm/reference/implementation-standards.md` +- *Cryptographic Implementation Details* (which algorithms chosen) +- *Certificate and Key Management Procedures* +- *Authentication Protocol Integration* (which OAuth/OIDC/LDAP impl) +- *OpenAPI Implementation* (endpoint design choices) +- *Observability Implementation* (Prometheus/OTel choices) +- *Kubernetes Integration* +- *Compliance Configuration* (which standards enforced per profile) + +**Cross-ref:** dcm doc: *"Selects implementations of the standards listed in [udlm/90-reference/40-standards-catalog.md]."* + +**Note:** udlm portion is reference-only (citations, not new substrate). Still belongs in udlm because peers consuming udlm need to know what standards apply. + +--- + +### 18. `43-provider-callback-auth.md` — **resolved: udlm contract is mechanism-neutral** + +**udlm sections** → `udlm/contracts/provider-callback-auth.md` (mechanism-neutral two-layer contract) +- *Two-Layer Authentication Contract* — abstract: any callback MUST be validated via two independent identity factors. Specific mechanisms (mTLS, JWT, signed assertions, etc.) are realization choices declared via schema-sharing. +- *Provider Identity Attestation Contract* — peers MUST attest provider identity at registration via a verifiable mechanism (the verification approach is realization-declared). +- *Callback Credential Lifecycle* — issuance, active, rotation, revocation states (technology-neutral). +- *Authentication-at-Callback-Time Contract* — every callback MUST present both factors; the receiving peer MUST validate both before accepting. +- *Entity-Level Authorization Contract* — credentials are scoped; peer MUST verify the provider's authorization to update the target entity. +- *Bootstrap Contract* — initial registration requires an authenticated single-use token; mechanism is realization-declared. +- *Credential Revocation Contract* — revocation is immediate; peers MUST recognize and reject revoked credentials. + +**dcm sections** → `dcm/architecture/credentials-and-auth/provider-callback.md` (DCM's specific mechanism: mTLS + interaction credential) +- *mTLS as Layer 1 — DCM's identity-attestation mechanism* +- *Interaction Credential as Layer 2 — DCM's credential mechanism* +- *Provider Certificate Storage and Validation* +- *Interaction Credential Issuance and Management* +- *mTLS Enforcement at Callback Endpoint* +- *Credential Validation Logic at Callback Time* +- *Entity Authorization Checks* +- *Registration Token Generation and Validation* +- *Revocation Enforcement* +- *Emergency Revocation Response* + +**Cross-ref:** dcm doc opens: *"Realizes the two-layer auth contract in [udlm/contracts/provider-callback-auth.md] using mTLS + interaction credential. The specific mechanism is declared in DCM's schema bundle per [udlm/contracts/schema-sharing.md]."* + +**Resolution:** Per user direction — udlm defines abstract contract; dcm picks specific mechanism. Peer realizations declare their chosen mechanism via the schema-sharing protocol so federation peers can interoperate. + +--- + +### 19. `48-location-topology-layers.md` — **resolved: tighten udlm to contract-only; hierarchy moves to dcm** + +**udlm sections** → `udlm/60-topology/48-location-topology-layers.md` (contract only) +- *Layered-Topology Contract* — "topology consists of layers; layers have parent/child relationships; layers carry typed fields" (abstract, not the specific 9 layers) +- *Location Layer Instance Format* — the data structure for representing a layer instance +- *Hierarchy Assembly Rules* — what makes a parent/child relationship valid (contract, not the specific tree) +- *Location Layer Lifecycle (active, deprecated, decommissioned)* +- *Custom/Extension Mechanism* — how new layer types are added + +**dcm sections** → split between: +1. `dcm/architecture/topology/canonical-9-layer-hierarchy.md` — the specific Country → Region → Zone → Site → Data Center → Hall → Cage → Rack → Unit hierarchy (DCM's canonical default; a peer realization could pick differently) +2. `dcm/architecture/topology/placement-and-priority-bands.md` + - *Location Topology Database and Query Interface* + - *Priority Band Allocation* (premium/standard/budget) + - *Consumer Selection Model* (preference matching) + - *Authority and Ownership Model* + - *Relationship to Placement Engine* + - *Location Layer Lifecycle Management* (operational draining, re-placement) + - *Profile-Governed Topology Constraints* + +**Cross-ref:** dcm hierarchy doc opens: *"Realizes the layered-topology contract in [udlm/60-topology/48-location-topology-layers.md] with DCM's canonical 9-layer scheme."* + +**Resolution:** Analogy made the call clear — "addresses are layered" is rule of the road; "the layers are Country, Region, Zone..." is the specific addressing scheme this jurisdiction picked. + +--- + +### 20. `51-infrastructure-optimization.md` — **resolved: PostgreSQL mandate moves to dcm** + +**udlm sections** → `udlm/70-design-principles/51-infrastructure-optimization.md` +- *Data Contracts vs Abstraction Layers* (the principle — no abstraction-hiding allowed) +- *Four Data Domains (Intent, Requested, Realized, Discovered)* — restatement as foundational domains. +- *Mandatory Persistence Requirement* — the contract that all four domains must be persistently queryable. **Note: persistence is required; the technology is not specified here.** + +**dcm sections** → split between: +1. `dcm/architecture/persistence/postgres-mandate.md` — the decision that this dcm realization mandates PostgreSQL (a dcm-level architectural choice; a peer realization could pick differently while honoring the udlm persistence contract) +2. `dcm/architecture/persistence/postgres-implementation.md` + - *Enforcement Mechanisms for Required Infrastructure* + - *Data Domain Implementation Details* (table structures, schema) + - *Query Optimization and Indexing* + - *Data Retention and Archival Policies* + +**Cross-ref:** dcm doc: *"Realizes the persistence contract in [udlm/70-design-principles/51-infrastructure-optimization.md] by mandating PostgreSQL for DCM."* + +**Resolution:** Per analogy — "must have a road" is rule of the road; "the road is paved asphalt with painted lines" is a jurisdictional infrastructure choice. + +--- + +### 21. `53-capability-discovery.md` + +**udlm sections** → `udlm/20-contracts/53-capability-discovery.md` +- *Problem Statement (types vs capabilities)* — modeling contract. +- *Unified Provider Model* — provider registration contract. +- *Capability Declaration Format and Semantics* — provider description contract. + +**dcm sections** → fold into `dcm/architecture/convergence-engine/overview.md` (or new file) +- *Provider Registry Implementation with Capabilities* +- *Capability Matching for Dispatch Decisions* +- *Backward Compatibility with Type-Based Model* +- *Capability Validation and Conflict Resolution* + +**Cross-ref:** dcm doc cites this udlm doc as the capability declaration contract. + +--- + +## Hardest calls — all resolved by wire-compatibility decision + +1. ~~**`04-examples.md`**~~ — **RESOLVED**: all examples kept clean in udlm; dcm gets its own `orchestration-scenarios.md`. No per-paragraph splitting. + +2. ~~**`28-federated-contribution-model.md`**~~ — **RESOLVED**: split as proposed. Wire-compatibility means contributor types + artifact formats ARE wire-level concerns peers must agree on; GitOps PR is dcm's transport for contributions, not the contract. + +3. ~~**`43-provider-callback-auth.md`**~~ — **RESOLVED**: udlm defines the **two-layer auth contract abstractly** (any peer must validate provider identity via two independent factors). dcm specifies **mTLS + interaction credential** as its specific mechanism. Peer realizations could pick different layers and still conform — provided they declare their auth mechanism via the schema-sharing protocol. + +4. ~~**`48-location-topology-layers.md`**~~ — **RESOLVED**: layered-topology contract + assembly rules + lifecycle → udlm. Specific 9-layer hierarchy → dcm canonical default. + +5. ~~**`51-infrastructure-optimization.md`**~~ — **RESOLVED**: PostgreSQL mandate moves to dcm. udlm keeps data-contract principle + four-domains contract + persistence-required contract (technology-neutral). + +6. ~~**`31-credential-management.md` + `31-credential-provider-model.md`**~~ — **RESOLVED**: merge to `udlm/governance/credentials.md` (udlm side) and `dcm/architecture/credentials-and-auth/credentials.md` (dcm side). + +--- + +## Newly identified udlm contracts (from "rules of the road" sweep) + +User clarification (a/b/c) and a follow-up sweep of existing docs surfaced +**7 net-new substrate documents** that were missing, partially specified, or +scattered. **All 7 have been drafted** in `architecture/data-model/` and will +migrate to udlm during the split. + +### Created (drafts in dcm/architecture/data-model/, target paths in udlm shown) + +| Doc | Target path | Draft location | Status | +|---|---|---|---| +| **Identifier scheme contract** | `udlm/contracts/identifier-scheme.md` | `architecture/data-model/identifier-scheme.md` | ✅ Drafted | +| **Time and clock model** | `udlm/contracts/time-and-clock.md` | `architecture/data-model/time-and-clock.md` | ✅ Drafted | +| **Error model contract** | `udlm/contracts/error-model.md` | `architecture/data-model/error-model.md` | ✅ Drafted (incl. `conformance.*` namespace) | +| **Retry semantics contract** | `udlm/contracts/retry-semantics.md` | `architecture/data-model/retry-semantics.md` | ✅ Drafted | +| **Rate limit + backpressure contract** | `udlm/contracts/rate-limit-and-backpressure.md` | `architecture/data-model/rate-limit-and-backpressure.md` | ✅ Drafted | +| **Schema sharing protocol** | `udlm/contracts/schema-sharing.md` | `architecture/data-model/schema-sharing.md` | ✅ Drafted | +| **Conformance specification** | `udlm/CONFORMANCE.md` | `architecture/data-model/CONFORMANCE.md` | ✅ Drafted | + +### Sufficient as-is (no new doc needed) + +- **Idempotency contract** — `33-event-catalog.md` already covers it at substrate quality (event_uuid as idempotency key, at-least-once semantics, consumer-supplied Idempotency-Key). Just needs to be cross-referenced from new related docs. + +### Perspective docs (the "handbooks") — authored + +Two complementary perspective docs, one per layer. **Authored** during the +split execution phase: + +| Doc | Target path | Purpose | Status | +|---|---|---|---| +| **Consumer perspective (driver's handbook)** | `udlm/docs/consumer-perspective.md` | How a consumer sees the system: onboarding, mental models, request lifecycle, common patterns, troubleshooting — written from the user's POV against the substrate | ✅ Done | +| **Operator perspective (DMV operator's manual)** | `dcm/architecture/operator-perspective.md` | How an implementer/operator sees the system: how DCM operationalizes udlm, where the realization choices live, deployment perspective, ops playbook entry point | ✅ Done | + +### Sweep findings — what was checked and verdict + +| Concept | Found | Verdict | +|---|---|---| +| Identifier scheme | Scattered across 06, 02, 33, A, 45 | Gap — needs new substrate doc | +| Idempotency | Thorough in 33, 18, 06, 23, 25, 31b, A | Substrate quality — keep | +| Backpressure / rate limit | Strong in 49, 18, 37, 53 | Mixed — extract substrate portion | +| Time / clock model | Scattered in 40, 33, 16, 12, 37, 52 | Gap — needs new substrate doc (UTC contract, ms precision, skew tolerance, total ordering) | +| Error model | Minimal in consumer-api-spec, 31b, B | Gap — needs new substrate doc with closed error vocabulary | +| Retry semantics | Operational in 24, 25, 7, 30, 37, 23, 50 | Partial — extract + new substrate framing | +| Consumer perspective | Minimal (04-examples is closest) | Gap — needs net-new driver's handbook | + +--- + +## Execution sequence (Phase 2-3) + +Mechanical work, in order: + +1. **Create empty `udlm` git repo** at `github.com/croadfeldt/udlm`. +2. **Use `git filter-repo`** to extract `architecture/data-model/` history into the new repo, preserving commits. Reorganize into the LOCKED udlm layout in a single restructure commit (drop all numeric prefixes; move files into their target directories). +3. **For each "both" file**: split into udlm + dcm fragments per the per-section blocks above. Apply the rename mapping at the same time. Single commit per file ("split N-foo.md: udlm/dcm portions"). +4. ~~Author the 6 new udlm contract docs + CONFORMANCE.md~~ ✅ **DONE** — drafted in `architecture/data-model/`. Execution moves them to their target paths in udlm during step 2. +5. ~~Author the two perspective docs~~ ✅ **DONE** — authored at `udlm/docs/consumer-perspective.md` and `dcm/architecture/operator-perspective.md`. +6. **Update cross-references**: add the `> Implements...` header to every dcm doc that has a udlm counterpart. Cross-link the new substrate docs from anywhere they're referenced. Update intra-doc references in the 7 new contract docs to drop their `(N-...)` legacy path hints. +7. ~~Add `conformance.version_deprecated` federation event~~ ✅ **DONE** — wired into `udlm/contracts/event-catalog.md` (introduced by `CONFORMANCE.md` §9.2). +8. **Delete migrated files from dcm** in one cleanup commit. +9. **Update dcm `README.md` and `project-overview`** to reference udlm as the substrate spec. +10. **Commit `00-layering-data-model-vs-dcm.md`** with concrete repo links now resolvable. + +## After split (Phase 4-5) — DAV plumbing + +- DAV MCP doc-fetcher gets two source repos (`udlm` + `dcm`). +- UC YAML `spec_refs` use namespaced paths: `udlm/governance/governance-matrix.md` and `dcm/architecture/convergence-engine/policy-evaluation.md`. +- Source ConfigMap split: separate mountpaths. +- Run one sample UC eval to validate cross-repo resolution. + +--- + +## Decision log + +All blocking decisions are settled. This section is the durable record of +what was decided and why. + +### Locked decisions + +- ✅ **Compatibility model**: wire-compatible at data/event/contract boundary (versioning rules apply). udlm is K8s-shaped: API + CRD wire-compatible across distributions; controllers not portable. +- ✅ **Repo location**: `github.com/croadfeldt/udlm` +- ✅ **udlm numbering**: dropped (directory structure carries ordering) +- ✅ **dcm numbering**: dropped (flat, organized by concern) +- ✅ **Split manifest**: kept as permanent contextual doc in dcm (helps future contributors understand the boundary) +- ✅ **Examples**: all-udlm + new dcm `orchestration-scenarios.md` +- ✅ **Location topology**: contract → udlm, specific 9-layer hierarchy → dcm canonical default +- ✅ **PostgreSQL mandate**: → dcm (substrate requires persistence; technology choice is dcm-level) +- ✅ **Credentials (31a + 31b)**: merged on both sides (`udlm/governance/credentials.md`, `dcm/architecture/credentials-and-auth/credentials.md`) +- ✅ **Federated contribution**: split (wire-compat makes contributor types + artifact format a peer contract; GitOps PR is dcm's transport) +- ✅ **Provider callback auth**: udlm = abstract two-layer auth contract; dcm = mTLS + interaction credential mechanism; peers declare their chosen mechanism via schema-sharing +- ✅ **7 new udlm substrate docs drafted**: identifier-scheme, time-and-clock, error-model (with `conformance.*` namespace), retry-semantics, rate-limit-and-backpressure, schema-sharing, CONFORMANCE +- ✅ **Consumer perspective + operator perspective**: paired narrative docs authored at `udlm/docs/consumer-perspective.md` and `dcm/architecture/operator-perspective.md` +- ✅ **Single narrative directory**: `docs/` (no separate `guides/`) + +### Open items (cosmetic / non-blocking) + +- [ ] Final read-through could flag any reclassifications on the pure-udlm and pure-dcm file lists. Default: trust the agent's classification + the analogy validation. + +### Items deferred to execution phase + +- Path rename: per-section split blocks (1-21 above) reference legacy numeric paths. Execution will apply the mapping uniformly (drop numeric prefixes from directories and filenames). +- ✅ `conformance.version_deprecated` federation event wired into `udlm/contracts/event-catalog.md` (introduced by CONFORMANCE.md §9.2). +- ✅ Two perspective docs authored (consumer-perspective.md, operator-perspective.md). diff --git a/architecture/adopted-standards-dcm.md b/architecture/adopted-standards-dcm.md new file mode 100644 index 0000000..7e0f1b4 --- /dev/null +++ b/architecture/adopted-standards-dcm.md @@ -0,0 +1,94 @@ +# DCM — Adopted External Standards (the DCM-domain requirements) + +UDLM defines the **Adopt** disposition: when a credible external standard already models a domain's data +(FOCUS for cost/usage, OpenCost for k8s allocation, OSCAL for compliance, SCIM for identity), the data +substrate carries only *identity*, a *version-pinned conformance reference*, and the *binding* — never +the standard's schema. See UDLM `design-principles/core-tenets.md` **T5** and +`design-principles/adopted-standards.md`. + +> **License verdicts & the full adoption ledger live in UDLM** — `registry/standards-adoption-register.md` (what/why/where/when/who + license-compatibility verdict per standard, CI-enforced by ADOPT-001). This document does not restate verdicts; it states the DCM RUNTIME requirements. + +That is the **Data** half. This document states the **DCM** half: the requirements DCM must implement +and enable so adopted standards actually work at runtime. It is an application of the +**Data ⇄ Policy boundary** (`data-policy-boundary.md`): **the data declares which standard versions are +in play; DCM (Policy/Provider runtime) negotiates, enforces, and translates between them.** + +DCM **MUST** adhere to T5: it does not absorb an external standard's schema into its own persistence, +and does not become the system of record for adopted data — that data is referenced via an Information +Provider, lookup-only (`contracts/information-providers.md`). + +> **Scope — Tier 2 only.** These requirements apply to **record/schema** standards (FOCUS, OpenCost, +> OSCAL, SCIM), which version in ways that change their shape. **Value/codelist** standards (ISO 4217, +> ISO 8601, RFC 4122) are adopted as a plain referenced field constraint — *no* support matrix, *no* +> version negotiation, *no* ADS requirements. Route by kind first (UDLM `adopted-standards.md` §1a). + +## Requirements (ADS — Adopted Standards) + +### Registration & discovery +- **ADS-001 — Provider support matrix.** DCM **MUST** accept and validate a provider's + `adopted_standard_support[]` declaration at registration: for each adopted standard, the supported + version range, a `preferred` version, and `direction` (`emit` | `consume` | `both`). It is validated + and trust-stamped like any other provider capability. +- **ADS-002 — Compatibility discovery.** DCM **MUST** expose, for discovery, which providers can serve + which standard versions, so an implementor can select a compatible provider before binding. Silent + incompatibility is not permitted. + +### Negotiation, translation, enforcement (Policy) +- **ADS-003 — Version negotiation.** At binding time DCM **MUST** resolve the intersection of the + consumer/type **required** version range and the provider's **supported** range, selecting the + effective version (highest common, or `preferred` when inside the overlap). +- **ADS-004 — Translation as Policy.** When required and supported do not directly overlap, DCM **MAY** + translate between versions via a **registered, deterministic** mapping — preferably the standard's own + published migration. Translation is a **Policy act** (transformation is Policy, UDLM T2), evaluated and + **audited**; it is never an evaluator embedded in the portable data. +- **ADS-005 — Enforcement / reject.** If there is no compatible version **and** no registered + translation path, DCM **MUST** reject the binding as non-conformant and **surface** it — never + silently drop or downgrade. +- **ADS-006 — Implementor-bounded parameters.** Parameters defined by the standard but constrained by + the implementor (e.g. cost rate ranges, markup minimums, budget ceilings) **MUST** be enforced via + **policy-as-code** (OPA/Rego), not hard-coded — consistent with DCM's policy-governance model. + +### Identity, provenance, audit (recording the decision) +- **ADS-007 — Identity join, no ownership.** DCM **MUST** resolve an adopted-standard binding using the + UDLM identity ↔ standard-column join (e.g. resource `uuid` ↔ FOCUS `ResourceId`) and **MUST NOT** + cache or persist the external records as a system of record — the data is served by the Information + Provider with the freshness/authority the IP contract provides + (`contracts/information-providers-advanced.md`). +- **ADS-008 — Effective-version provenance.** DCM **MUST** record the negotiated **effective version** + (and, when it translated, the source version + translation reference) as **provenance** on the + realized entity, and **MUST** lower confidence/authority for translated/derived values. +- **ADS-009 — Auditability.** The negotiation outcome (accept / translate / reject) and any translation + **MUST** be written to the tamper-evident audit log (`AUD-001/002`) as a decision — reproducible from + the immutable record. + +### Lifecycle +- **ADS-010 — Standard version lifecycle.** DCM **MUST** track adopted-standard version deprecation and + allow a configured **minimum** and **preferred** version per environment, so an operator can require, + e.g., "FOCUS ≥ 1.3" platform-wide and let negotiation/translation satisfy older providers. + +## How this maps to the boundary + +| Step | Data (UDLM) — the noun | DCM (Policy/Provider) — the verb | +|---|---|---| +| Provider declares support | `adopted_standard_support[]` record | validate + register (ADS-001) | +| Consumer/type needs a version | `adopts[].version` range | — | +| Pick the version | — | **negotiate** required ∩ supported (ADS-003) | +| Versions don't match | registered migration reference | **translate** (ADS-004) or **reject** (ADS-005) | +| Bind to the resource | identity ↔ standard column | **resolve** join, IP lookup (ADS-007) | +| Record what happened | effective-version provenance slot | **write** provenance + audit (ADS-008/009) | +| Constrain parameters | the declared parameter values | **enforce** via Rego (ADS-006) | + +> Test (same as the boundary doc): a **noun** (a support record, a version pin, a join key, a provenance +> slot) is UDLM's; a **verb** (negotiate, translate, enforce, reject, resolve, record) is DCM's. + +## Worked example — cost (FOCUS + OpenCost) + +The Cost Management Service Provider (`dcm-project/enhancements#57`) is the reference case: it declares +`adopted_standard_support` for FOCUS (`≥1.2 <2.0`, preferred 1.4, emit) and OpenCost (`1.x`); DCM +negotiates against a chargeback view that requires FOCUS ≥ 1.3 (allocation columns); rate ranges and +budgets are enforced by Rego (ADS-006); the effective version is recorded as provenance (ADS-008). The +cost data itself is **never** modeled in DCM/UDLM — it conforms to FOCUS and is served via the SP's +Information-Provider query API. This is what "adopt, don't absorb" looks like end-to-end. + +See also: `data-policy-boundary.md`, UDLM `design-principles/adopted-standards.md` (the Data-side +contract), `dcm-platform-requirements.md` (the broader requirement set). diff --git a/architecture/data-policy-boundary.md b/architecture/data-policy-boundary.md new file mode 100644 index 0000000..93a00e2 --- /dev/null +++ b/architecture/data-policy-boundary.md @@ -0,0 +1,58 @@ +# DCM ↔ UDLM — the Data / Policy responsibility boundary + +DCM and UDLM are two domains separated by one **responsibility boundary** — a service / contract seam. +Getting it right is what keeps the data model portable, auditable, and sovereign, and keeps DCM's logic +where it belongs. This is the DCM-side statement of the boundary defined in UDLM's +`design-principles/core-tenets.md`. + +| Domain | Owner | Responsibility | +|---|---|---| +| **Data** | UDLM | Custody of data through its lifecycle: identity, the four states, versioning, relationships, provenance, audit records, sovereignty fields. *Hold, move, reference, version, audit.* | +| **Policy** | **DCM** | **Application of policy** — transformation, enrichment, derivation, decision, governance. *Compute, derive, evaluate, decide, enforce.* | + +**UDLM defines the contracts (Data, Provider, and Policy); DCM is where Policy is *applied*.** UDLM +carries the data policy acts on and records the decisions policy makes; it never executes logic. DCM +never becomes the system of record for lifecycle state; it applies logic over UDLM data. + +## What DCM owns (the verbs) +- **Assembly** — Intent → Requested: merge Layers (data) under Policy (logic), recording per-field + provenance back into the UDLM record. +- **Policy evaluation** — Validation Policy, Transformation; the **Governance Matrix** and + **sovereignty/accreditation/trust** decisions; placement. +- **Dependency-graph application** — validate the DAG (`RDG-001`), order forward execution, run + compensation in reverse, schedule rehydration in dependency order. +- **Realization** — dispatch to Providers, collect Realized state, run Discovery → drift, resolve + conflicts (field ownership / server-side apply). +- **Audit production** — write the synchronous, append-only, Merkle-chained log (`AUD-001/002`). +- **Adopted-standard runtime** — for externally-adopted standards (FOCUS, OpenCost, OSCAL, SCIM), the + data carries identity + version pins; **DCM negotiates, translates, and enforces standard versions** + and records the effective version as provenance. Full requirements: `adopted-standards-dcm.md` + (`ADS-001…010`); the Data-side contract is UDLM `design-principles/adopted-standards.md` (tenet T5). +- **Enforcement** — reject changes to `immutable`/createOnly fields; reject sovereignty-boundary + violations; reject non-conformant data at the seam. + +## What DCM must NOT do (boundary violations) +- Become the durable system of record for lifecycle state — that is UDLM data. +- **Absorb an adopted standard's schema** into DCM persistence, or become the system of record for + adopted data (cost, compliance, identity). That data conforms to its standard and is referenced via an + Information Provider, lookup-only (`adopted-standards-dcm.md`, `ADS-007`). +- Push executable logic *into* the portable data model. UDLM carries **no embedded expression + language**; all transformation/enrichment is DCM policy. This is what makes the contract layer + **deterministic and reproducible** (the precondition for tamper-evident audit and sovereignty): + determinism is structural in the data because the evaluator lives only in DCM. +- Let a runtime decision that legitimately depends on live state (e.g. placement by current capacity) + silently alter the contract. Such decisions are **recorded as decisions in the audit log**, never + written back as if they were the reproducible definition. + +## Why the boundary holds the line on the four pillars +- **Audit:** data is immutable + version-pinned (`$id`); DCM produces the proofs. Reproducible forever. +- **Observability:** UDLM declares relationships + typed outputs; DCM reconciles observed vs declared. +- **Dependency graph:** UDLM carries typed edges; DCM constructs/validates/orders the DAG. +- **Sovereignty:** UDLM marks sovereignty fields `immutable` and bundles offline closures; DCM enforces + the Governance Matrix. Because no expression rides in the data, nothing can route around the boundary. + +> Test: a **noun** (record, contract, edge, marker, pin) is UDLM's. A **verb** (assemble, evaluate, +> decide, enforce, transform, resolve) is DCM's. + +See UDLM `design-principles/core-tenets.md` (T1–T4) and `cross-cutting-requirements.md` for the +substrate side; the Resource Type Registry (`registry/`) is the concrete Data-domain contract DCM applies. diff --git a/architecture/dcm-documentation-review-strategy.md b/architecture/dcm-documentation-review-strategy.md new file mode 100644 index 0000000..d774315 --- /dev/null +++ b/architecture/dcm-documentation-review-strategy.md @@ -0,0 +1,193 @@ +# DCM Architecture Documentation — Review Strategy + +**Author:** Chris, Principal Architect +**Date:** April 2026 +**Audience:** DCM Engineering Team + +--- + +## The Problem + +The DCM architecture now comprises 58 data model documents, 15 specifications, 4 OpenAPI schemas, and a capabilities matrix with 331 entries — roughly 30,000 lines of technical documentation. This body of work defines the complete control plane, data model, provider contract, policy engine, audit system, and governance model. + +This documentation needs engineering review, but the experience with PRs #7 and #8 demonstrated the challenge: even GitHub's Sourcery bot hit its 20,000-line review limit. Nobody is going to read 30,000 lines linearly, and asking the team to do so would be neither productive nor respectful of their time. + +This document proposes a strategy to make the architecture documentation consumable, reviewable, and useful for the engineering team. + +--- + +## Principles + +1. **Nobody needs to read everything.** Each team member needs the 3-5 documents relevant to their area, not all 58. +2. **Decisions are reviewable. Reference material is not.** The team should review *what we decided and why* — not the full specification text that implements those decisions. +3. **Concrete examples beat abstract specifications.** A worked example tracing one request through the full pipeline teaches more than 10 documents of structural definitions. +4. **The documentation should be queryable, not just readable.** The AI prompt (5,768 lines, 125 sections) enables conversational exploration of any architectural topic. + +--- + +## Proposal 1: Role-Based Reading Guide + +**Effort:** Low (1 page) +**Impact:** Immediate + +Create a single-page reading guide that tells each person where to start based on what they work on. Example: + +| If you work on... | Start with these documents | Then reference... | +|---|---|---| +| **Provider development** (VM, ACM, network) | Doc A (Provider Contract), Doc 06 (Resource Entities), Provider Dev Guide | Docs 10, 22, 30 (provider types) | +| **Policy and placement** | Doc B (Policy Contract §1-7), Doc 14 (Profiles) | Doc B §8-9 (Constraint Registry, Templates), §18 (Overrides) | +| **Request pipeline** (orchestrator, processor) | Doc 02 (Four States), Doc 03 (Layering), Doc 04 (Examples) | Doc 25 (Control Plane), Consumer API Spec | +| **Catalog and API gateway** | Doc 06 (Resource Entities), Consumer API Spec, Admin API Spec | Doc 36 (API Versioning), AEP conventions | +| **Audit, security, compliance** | Doc 16 (Universal Audit §1-8), Doc 31 (Credentials), Doc 43 (Callback Auth) | Doc 26 (Governance Matrix), Doc 14 §8-10 | +| **Database and storage** | Doc 11 (Data Store Contracts), Doc 02 §4 (Data Domains), SQL Schema | Doc 17 (Deployment) | + +This cuts the perceived scope by 80% for any individual reviewer. Each path is 3-5 documents, not 58. + +--- + +## Proposal 2: Architecture Decision Records (ADRs) + +**Effort:** Medium (1 page per major decision) +**Impact:** High — this is what the team actually needs to review + +The team does not need to review the full 1,344-line Policy Contract to understand the override model. They need a 1-page summary that says: + +> **Decision:** 5 override mechanisms layered by severity. +> **Context:** Requests can be blocked by hard enforcement policies. The team needs a way to handle legitimate exceptions without undermining policy governance. +> **Options considered:** Override policies only, manual override only, exception grants, compensating controls. +> **Decision:** All five, layered: Override Policy (planned) → Exception Grant (pre-authorized) → Manual Override (immediate) → Compensating Control (structural) → Dual-Approval (modifier for hard policies). +> **Consequences:** `override_requests` SQL table added. 5 new events. Consumer sees POLICY_BLOCKED with resolution options (modify, override, cancel, escalate). Admin API gets 4 new endpoints. + +The major decisions that need ADRs (roughly 15): + +1. One required infrastructure (PostgreSQL only) +2. Internal auth/secrets/events with optional external delegation +3. 5 provider types (why these 5, what was consolidated) +4. 2 policy evaluation modes (Internal/External) +5. Evaluation Context with multi-pass convergence +6. Constraint Type Registry and Policy Templates +7. Lifecycle-scoped policy evaluation (10 operation types) +8. Policy Override Model (5 mechanisms) +9. Policy Block Resolution (consumer options, not auto-override) +10. Merkle tree audit with configurable granularity +11. Data-driven policy matching (4 sources) +12. Stage signing and payload chain of custody +13. AEP API conventions adoption +14. Traefik over KrakenD (API Gateway decision) +15. PostgreSQL consolidation (stores model) + +ADRs are individually reviewable. The full specification documents become reference material that people consult when implementing — not review material that blocks progress. + +--- + +## Proposal 3: Domain-Split PRs + +**Effort:** Medium (splitting existing content into themed PRs) +**Impact:** High — makes the merge process manageable + +Instead of one PR with the entire architecture, submit 6-8 PRs by domain. Each is reviewable in one sitting and can be assigned to the most relevant reviewer: + +| PR | Content | Primary Reviewer | Size | +|---|---|---|---| +| Core Data Model | Docs 00-04, 11, SQL schema | machacekondra | ~3,000 lines | +| Provider Contract | Doc A, docs 06, 10, 22, 30, provider callback | pkliczewski, ygalblum | ~3,500 lines | +| Policy Contract | Doc B, doc 14 | gabriel-farache | ~4,000 lines | +| Audit and Security | Docs 16, 31, 43, 26, 27 | jenniferubah | ~2,500 lines | +| Request Pipeline and Lifecycle | Docs 25, 50, 36, consumer/admin API specs | machacekondra | ~3,000 lines | +| Capabilities and Examples | Capabilities Matrix, doc 04, doc 52, taxonomy | All (light review) | ~2,500 lines | +| Infrastructure and Deployment | Doc 51, doc 17, doc 41, OpenAPI schemas | ygalblum | ~2,000 lines | +| Federation and Advanced | Docs 20, 44, 48, DISCUSSION-TOPICS | pkliczewski | ~2,000 lines | + +Each reviewer focuses on their area of expertise. Cross-cutting concerns (like "does the policy model work with the provider contract?") are handled in a final integration review after the domain PRs merge. + +--- + +## Proposal 4: End-to-End Walkthrough Document + +**Effort:** Medium (1 document, ~500 lines) +**Impact:** Very High — the single most useful onboarding artifact + +The team responded well to concrete examples. machacekondra specifically requested the three-tier app example. A dedicated walkthrough document that traces one request through the *entire* pipeline — with actual YAML payloads at each stage — would be worth more than 10 specification documents. + +The walkthrough would cover: + +1. Consumer submits a VM request (show the API call and intent payload) +2. Layer assembly (show which layers merge and the assembled payload) +3. Policy evaluation (show which policies fire, what constraints emit, how conflicts resolve) +4. A policy blocks the request (show the POLICY_BLOCKED response with resolution guidance) +5. Consumer modifies the request (show the modified payload) +6. Placement (show candidate scoring and selection) +7. Dispatch to provider (show the naturalized payload) +8. Provider callback (show the realized state) +9. Audit trail (show the Merkle tree leaves at mutation granularity) + +Every stage shows real data structures, not abstract descriptions. The walkthrough references the specification documents for detail but stands alone as a readable narrative. + +--- + +## Proposal 5: Interactive Architecture Map on the Website + +**Effort:** Higher (requires frontend work) +**Impact:** High for onboarding and navigation + +The Hugo website could host a visual diagram (Mermaid or D3) showing the 9 control plane services, the pipeline flow, and provider interactions. Each node links to the relevant documentation. People orient visually before drilling into text. + +This is lower priority than proposals 1-4 but would be a strong differentiator for the project's public presence and for onboarding new contributors. + +--- + +## Proposal 6: AI Prompt as Team Onboarding Tool + +**Effort:** Zero (already built) +**Impact:** Immediate for anyone willing to use it + +> **Note (2026-07-11): the AI prompt has since been RETIRED** — it duplicated the specs and drifted stale (it taught the pre-ADR-PROV-002 typed-provider model). Orientation now routes through `AGENTS.md` → the authoritative spec docs. The paragraph below is retained for historical context only. + +The AI prompt (DCM-AI-PROMPT.md) is the most comprehensive single document in the project: 5,768 lines covering every architectural decision, every capability, every data structure, and every cross-reference. When loaded into Claude (or any capable LLM), it enables conversational exploration: + +- "How does the override model work?" → 2-paragraph answer with doc references +- "What happens when a sovereignty policy blocks a request?" → full flow with YAML examples +- "What's the difference between Internal and External policy evaluation?" → comparison table + +This is faster than reading documentation for exploratory understanding. The team should be encouraged to use it as a first stop before reading specification documents. + +--- + +## Proposal 7: Session Changelogs + +**Effort:** Low (produced as part of each work session) +**Impact:** Keeps the team current without re-reading everything + +After each significant architecture session, produce a short changelog: + +> **Session: April 4-7, 2026** +> +> **Added:** Policy Override Model (doc B §18) — 5 mechanisms layered by severity. Policy Block Resolution (doc B §18.8) — consumer gets compliant value suggestions, not just a deny. Merkle tree audit (doc 16 §8) — configurable granularity (stage/mutation/field). Lifecycle-scoped policy evaluation (doc B §2.2-2.3) — 10 operation types, changed_field_filter. Test Framework Specification (doc 52) — 60 invariants, machine-readable YAML summary. OpenStack Nova example provider. +> +> **Changed:** SQL schema now 18 tables (+override_requests, signed_tree_heads, merkle_tree_nodes). Capabilities matrix 309→331. Admin API 57→61 paths. Events 101→109. +> +> **Why it matters for you:** If you're working on the policy engine, read doc B §18 (override model) and §2.2 (lifecycle scope). If you're working on audit, read doc 16 §8 (Merkle tree). If you're working on providers, the OpenStack Nova example in dcm-examples shows the complete naturalization/denaturalization pattern. + +The team reads the changelog (~1 page). They drill into specific documents only when something affects their work. + +--- + +## Recommended Implementation Order + +| Priority | Proposal | Effort | Timeline | +|---|---|---|---| +| **1** | Reading Guide | 1 day | This week | +| **2** | Session Changelogs | Ongoing | Start immediately | +| **3** | Domain-Split PRs | 2-3 days | Before next review cycle | +| **4** | ADRs for top 15 decisions | 3-5 days | Next two weeks | +| **5** | End-to-End Walkthrough | 2-3 days | Before summit demo prep | +| **6** | AI Prompt onboarding | 0 days | Announce to team | +| **7** | Interactive Architecture Map | 1-2 weeks | Future | + +--- + +## Summary + +The architecture is comprehensive and consistent. The problem is not the documentation quality — it's the volume. The team needs navigational aids, decision summaries, and concrete examples to make 30,000 lines of specification consumable. The proposals above provide a layered approach: immediate wins (reading guide, changelogs, AI prompt), near-term improvements (domain-split PRs, ADRs), and longer-term investments (walkthrough, interactive map). + +The goal is that no team member ever needs to read more than 3,000 lines to understand their area, and that every architectural decision is reviewable in a 1-page ADR — not buried in a 1,344-line specification document. diff --git a/architecture/integrations/automation-outcome-providers.md b/architecture/integrations/automation-outcome-providers.md new file mode 100644 index 0000000..334af5b --- /dev/null +++ b/architecture/integrations/automation-outcome-providers.md @@ -0,0 +1,126 @@ +# Automation as DCM services — build for the outcome, not the method + +> **North star — an intent-based service model, where the outcome *is* the intent.** The goal is to get +> organizations to focus on **outcomes, not methods**: declaring the desired **outcome** *is* the +> expression of intent — there is no separate intent artifact to interpret. The catalog/system deploys +> that outcome; *how* it's achieved is the platform's concern. This maps directly onto UDLM's four +> states, which open with **Intent**: the **Intent state is the declared outcome**, and realization is +> the journey Intent → Requested → Realized → Discovered. DCM is the system that deploys outcomes. This +> document applies the north star to automation — the principle generalizes to every service in the +> catalog. + +**Why:** the consumer's ask is *"I need this thing on this target"* — `Observability.LogShipper` on +`host-Z`. That is an **outcome**. The automation that makes it true (an Ansible role, an AAP job +template, a script, a container) is a *method*. DCM providers must be built for the **outcome**, and the +method must be an **encapsulated, swappable internal mechanism** — never a thing the consumer, the +Resource Type, or DCM placement sees. This note settles "do we need a generic automation spec?" — **no** +— and how the homelab's ansible roles become DCM-consumable services. + +## The rule + +> **Provider = outcome. Method = hidden.** + +- **Consumer contract:** Resource Type + target. `LogShipper` on `host-Z`. Nothing about *how*. +- **Provider:** identified by the **outcome capability** it offers — it `realize_resources` of type + `Observability.LogShipper` (and other outcome types). It is **not** "the Ansible provider" or "the AAP + provider." Its name is the outcome family, not the engine. +- **Method:** inside the provider. Today: `ansible-runner` invoking the the estate's private Ansible repo `alloy` role. + Tomorrow: an AAP job template, or a container. **Swapping the method is an internal provider change + with zero impact** on the type, the consumer, or DCM. That swap-invisibility *is* the proof the + boundary is correct. + +## Outcome-derived services — declare the goal, derive the work + +The leaf outcome (`LogShipper` on a host) is the floor. The **goal** is to drive **outcome-derived +services**: a consumer declares a *higher-order outcome* — "host-Z is **observable**", "host-Z is +**production-baseline**" — and the concrete services it needs are **derived** from it, not hand-picked. + +``` +Outcome (goal) "host-Z is observable" + | derive +Derived services Observability.LogShipper + Observability.MetricsExporter ( + … ) + | realize (provider; method hidden) +Realized on host-Z +``` + +This is the **Composite Service** model — a composite outcome whose **constituents are the derived +services** (the existing depends-on DAG) — and it is the answer to the open **Application Definition +Language** question (`adr/016-application-definition-language.md`): an *outcome is the application*, and +it derives its constituents. + +**Where the derivation lives splits on the Data ⇄ Policy line:** +- **Fixed** outcome (observable *always* = this service set) → a **declarative composite** (data). +- **Target-conditional** outcome (a Pi derives X, a server derives Y, by host attributes) → **Policy**. + +No new machinery — the same boundary, the same four-state lifecycle, now on the composite outcome (its +Discovered state aggregates its constituents' health). The single data-driven provider still realizes +each leaf; the derivation sits above it. + +## Why this beats method-providers + +- **Four-state lifecycle works on the outcome.** Intent (want shipping) → Requested (assembled with + sink + labels) → Realized (shipper running) → **Discovered** (is it healthy / still shipping? → drift). + You can reconcile *"is the LogShipper healthy,"* which you **cannot** do with *"did the playbook run."* + A method-provider (`Process.AnsiblePlaybook`) is fire-and-forget; an outcome-provider is reconcilable. +- **Audit/provenance** attach to a durable resource, not a one-shot job. +- **Engine independence by construction:** `ansible-runner` → AAP is a backend change, not a re-model. + +## No generic automation spec — the genericity lives in the provider + +There is **no automation in the data model**. The only "generic" part is an *implementation* detail +inside the outcome provider: a **data-driven Type → method-binding table**. + +``` +Observability.LogShipper -> ansible role 'alloy' (var-map: spec.sink.url -> loki_url, …) +Observability.MetricsExporter -> ansible role 'node_exporter' +… -> … +``` + +Adding a new outcome is **"define a Resource Type + add a mapping row"** — *not* writing a new provider. +The method binding (which role / which AAP template realizes which type, and the spec→vars mapping) is +the provider's **private catalog config** — the vendor-specific layer. It lives in the provider, **never +in the universal type**, exactly as Koku's native metric names live in its catalog item and not in the +FOCUS type (`koku-focus-adoption.md`). + +## raw ansible-runner vs AAP + +Not a consumer choice and not two Resource Types — both are **execution backends of the same outcome +provider**: +- **ansible-runner** — lightweight, homelab-grade. The starting backend. +- **AAP** — enterprise execution: RBAC, credential vault, job history, and **surveys (≈ UDLM E1 + constraint profiles)**. Slots in later as the provider's backend with **zero** type/consumer change. + +If that swap is truly zero-change, the abstraction held. + +## Running an automation is *also* an outcome — the automation as a service + +Even "just run automation X" fits the model — there is no second class. The outcome is **a service that +runs X**, not a fire-and-forget job. Its realized form is a **registered, invokable automation service**: +an **AAP job template** *is* exactly "a service to run an automation," as is a DCM-registered job or a +CronJob. That service is itself persistent and reconcilable — *does it exist? can it run? is its +definition current?* are all drift-checkable — and **each invocation is an audited run**. + +So every consumer ask is an outcome → a service. The only thing that differs is **what the service +provides**: + +| The service provides… | Example | Realized form | +|---|---|---| +| a running / configured **resource** | `Observability.LogShipper` | an agent running on the host | +| an **invokable automation** | a backup service, a cert-rotation service | a registered job (AAP template / CronJob) you can run | + +The generic **`Process.Automation`** type models the second row — *the automation-runner service*, a peer +outcome, not an exception. (Executor-neutral: the specific playbook / AAP template it wraps is provider +catalog config, never in the type.) The four-state lifecycle applies to the **service**; running it is an +audited event against that service. + +## Worked example (homelab) + +`Observability.LogShipper` (`udlm/registry/resource-types/observability.log-shipper.json`) — spec: +`{ target.host, sink.url, source, labels }` — realized by an outcome provider that runs the +the estate's private Ansible repo **`alloy`** role (journald → Loki). The consumer asks for a LogShipper on a host; the +provider naturalizes the spec into role vars, runs it, and reports `status` / `last_shipped_at` for drift. +The homelab's roles (`alloy`, `node_exporter`, `smartctl`, `fan_control`, …) become the first real DCM +**outcome** catalog — the live reference proving the model on actual automation. + +See also: `data-policy-boundary.md`, `adr/021-adopting-external-standards.md`, and UDLM +`design-principles/adopted-standards.md`. diff --git a/architecture/integrations/koku-focus-adoption.md b/architecture/integrations/koku-focus-adoption.md new file mode 100644 index 0000000..e105545 --- /dev/null +++ b/architecture/integrations/koku-focus-adoption.md @@ -0,0 +1,70 @@ +# Koku — upstream requirements for FOCUS adoption (cost provider) + +The DCM Cost provider (`dcm-project/enhancements` #57/#60) must serve **vendor-neutral** cost data so any +cost backend is swappable behind the same `cost` service type. Per ADR-021 (adopt external standards by +reference) and `adopted-standards-dcm.md` (ADS-001…010), the cost data conforms to **FOCUS** (Tier-2 +record standard) + **OpenCost** for Kubernetes allocation — not a bespoke, Koku-shaped vocabulary. + +This doc records the **upstream changes Koku** (`project-koku/koku`) needs so the cost provider can emit +FOCUS-conformant data. GitHub issues are disabled on that repo (it tracks via the **COST** Jira); these +are written as ready-to-file tickets — one subject per ticket, each with a *Why* — for COST Jira or a +koku GitHub Discussion. Current state (June 2026): no FOCUS export exists upstream; ISO 4217 currency is +already in flight (koku PR #6097). + +> **Tiering (why the work is uneven):** FOCUS/OpenCost are **Tier-2** record/schema standards → they need +> a real export + version negotiation + identity join. ISO 4217 is a **Tier-1** codelist → a referenced +> field constraint, already underway. Don't flatten them; see `adopted-standards.md` §1a. + +--- + +## A. FOCUS data export (serializer) + +**Why:** [FOCUS](https://focus.finops.org/) (FinOps Foundation, v1.4) is the vendor-neutral cost/usage +standard AWS/Azure/GCP already emit. Koku already normalizes multi-cloud + OpenShift cost into a unified +model; exposing it **as FOCUS** lets any FOCUS-aware consumer (FinOps tooling, the DCM cost provider) read +Koku cost without a Koku-specific integration. + +**What:** A FOCUS-conformant export/serializer projecting Koku's unified data into FOCUS columns — +`BilledCost`/`EffectiveCost`/`ListCost`/`ContractedCost`, `BillingCurrency`, `ChargeCategory`/`ChargeClass`, +`ConsumedQuantity`/`ConsumedUnit`, `PricingQuantity`/`PricingUnit`, `ChargePeriodStart/End`, +`ServiceCategory`/`ServiceName`, `ResourceId`/`ResourceType`, and the 1.3+ allocation columns. A projection +over existing data, not new metering. + +**Scope:** the export itself; version selection (B), `ResourceId` join (C), and OpenCost alignment (D) are +separate subjects. + +## B. FOCUS export — version selection + advertise supported versions + +**Why:** consumers need a specific FOCUS `major.minor` (allocation columns require ≥1.3), and a negotiating +platform must know which versions Koku can emit (the cost provider's `adopted_standard_support` matrix). + +**What:** accept a requested FOCUS version on the export (e.g. `?focus_version=1.4`), emit that version's +shape, and advertise the supported set (e.g. 1.2–1.4) via the API/capabilities. Depends on **A**. + +## C. FOCUS export — stable `ResourceId` for the identity join + +**Why:** FOCUS rows must carry a stable `ResourceId` so an external system can join cost back to the +resource it manages (e.g. a cluster/VM identity). Without a stable key, cost can't be attributed to a +managed resource. + +**What:** emit a stable, documented `ResourceId` in the FOCUS output (derived from existing tags/labels or +an accepted external correlation id). Depends on **A**. + +## D. Align OpenShift cost allocation with OpenCost + +**Why:** [OpenCost](https://opencost.io/) (CNCF) is the vendor-neutral standard for Kubernetes cost +allocation (workload/idle split, `max(request,usage)` over CPU/memory/GPU/PV/network). Aligning Koku's +OpenShift allocation to OpenCost — or documenting the precise mapping — makes Koku's container cost +portable and comparable with the ecosystem. + +**What:** align (or document the mapping of) Koku's OpenShift allocation to the OpenCost spec, and expose +it in the FOCUS export's allocation columns. Related to **A**. + +--- + +## E. (Not Koku) cost-dcm-provider — the DCM-side seam + +`pgarciaq/cost-dcm-provider` (the cost SP adapter) declares `adopted_standard_support` (FOCUS/OpenCost +versions), serves FOCUS via a `serve_data` capability, and binds cost to the target by identity +(`uuid` ↔ FOCUS `ResourceId`). It can perform interim FOCUS translation if A lands slowly — but the durable, +reusable home for the FOCUS projection is **Koku itself** (A), so every Koku consumer benefits, not just DCM. diff --git a/docs/engineering/service-taxonomy-reconciliation.md b/docs/engineering/service-taxonomy-reconciliation.md new file mode 100644 index 0000000..c577383 --- /dev/null +++ b/docs/engineering/service-taxonomy-reconciliation.md @@ -0,0 +1,162 @@ +# Service Taxonomy Reconciliation (DCM/UDLM ↔ Engineering ↔ OSAC) + +**Status:** Proposal — basis for the alignment conversation with **dcm-project engineering** (OSAC convergence is context that motivates one of the proposals, not a party whose sign-off is required). Not yet adopted. +**Date:** 2026-06-27 +**Scope:** The *"Service"* family of terms — Service, Service Provider, Atomic/Composite Service, Realize/Realized, Infrastructure Platform, Region/Zone. +**Related:** `taxonomy/DCM-Taxonomy.md` (Part 2 Anti-Vocabulary), `docs/engineering/ENGINEERING-ALIGNMENT.md` (implementation gap map), `architecture/adr/002-three-abstractions.md`, `architecture/adr/003-four-lifecycle-states.md`, `architecture/adr/005-provider-abstraction.md`; dcm-project/dcm `taxonomy/` (engineering vocabulary); osac-project (`fulfillment-service`, `bare-metal-fulfillment-operator`). + +> This is a **terminology** reconciliation. It is distinct from `ENGINEERING-ALIGNMENT.md`, which maps *implementations* against the architecture. Here we reconcile *words and concepts* across three taxonomies that are converging. + +--- + +## 1. The core insight: a Service is an *act*; a Resource is a *thing* + +The whole knot unties on one distinction: + +- A **Resource** is a *thing* (a noun) — a VM, a cluster, a database, an IP pool. +- A **Service** is the *act of doing* (an offering, a rendering) — "X **as a service**" is the act of providing and managing X, not the X itself. + +From which the load-bearing sentence follows: + +> **A Resource Provider provides a resource via its service.** + +The provider *renders a service* (the act of provisioning + managing); that service *yields a resource*. "Service" therefore sits at **both ends of the same contract** — the consumer *requests* a service (the offering they experience), the provider *renders* a service (the act it performs) — and the **resource flows between them**. Same concept, two viewpoints. + +This is consistent with the engineering taxonomy's own wording, which already calls a Service "the **capability** supported by a catalog item" — a *capability* is an ability to *do*, not a thing. We are sharpening an instinct the team already had, not overruling it. + +### Consequence: name providers by what their service *yields* + +Every provider renders a service, so "Service" does not distinguish one provider from another — it is common to all of them. The discriminator is **what the service yields**: + +| Provider | Its service yields | Example | +|---|---|---| +| **Resource Provider** | a **resource** (thing) | kubevirt → a VM; acm-cluster → a cluster | +| **Process Provider** | an **act / outcome** (the service *is* the deliverable; no persistent resource) | run an automation, execute a remediation | +| **Information Provider** | **data** (served, not owned) | FOCUS cost data, inventory | +| **Auth Provider** | **credentials / identity acts** | issue/rotate/revoke a credential | +| **Peer DCM** | **federated capability** | another DCM contributing registry/policy | + +This is why **"Service Provider" is the wrong name** — it describes *all* of them. The fix in §2 follows directly. + +--- + +## 2. Resource Provider (renames `service_provider`) — and the OSAC collision + +**Proposal: rename the provider type `service_provider` → `resource_provider`.** + +Two independent reasons: + +1. **It tells the truth (§1).** The provider's declared capability is to *provide resources*; naming it for the generic act ("service") it shares with every other provider hides what it actually does. "Resource Provider" names the yield. + +2. **OSAC collision.** As DCM and OSAC converge, "Service Provider" becomes genuinely overloaded — OSAC uses it for the **Cloud Service Provider / operator persona** (the org *running* the sovereign cloud), not a software adapter: + - osac-project/enhancement-proposals: *"As a **Cloud Service Provider admin**, I want to install OSAC…"* + - *"PublicIPPools are defined by the **service provider**; tenants manage…"* + + That is a *persona/organization*. Ours is a *pluggable component that provisions resources* (`kubevirt-service-provider`, `acm-cluster-service-provider`). Renaming ours to **Resource Provider** frees "Service Provider" for OSAC's operator sense and removes the contradiction we have been carrying ("a Service Provider does not provide services — it provisions resources"). + +The five provider types restated under the named-by-yield rule: + +`resource_provider` (was `service_provider`) · `process_provider` · `information_provider` · `auth_provider` · `peer_dcm` + +**Blast radius (why this is a proposal, not an edit):** every `*-service-provider` repo, `service-provider-manager`, the provider-contract, and the `service_provider` enum value across schemas/docs. This is a rename of *our* provider type, so it needs **dcm-project engineering** consensus. **OSAC consensus is not required** — OSAC keeps "service provider" for its operator persona; we are renaming *ours*, and doing so simply leaves their term uncontested. OSAC is the *motivation* (collision avoidance), not an approver. + +### Data · Policy · Provider lens +- **Data (UDLM):** the provider declaration carries `provider_type: resource_provider`; `supported_resource_types` is the yield it advertises. +- **Policy (DCM):** placement/capability matching is unchanged — it already matches on *what a provider yields*, which this rename makes explicit. +- **Provider:** renders a service (the act) that yields resources; the rename is purely nominal at the contract level (no behavioral change). + +--- + +## 3. Realize is the act; Realized is the state + +One root word does both jobs, and that is the point — no second concept is needed: + +| Form | Part of speech | Meaning | +|---|---|---| +| **realize** | verb (**the act**) | make intent *hold true in reality* | +| **Realized** | state noun (**the result**) | the state produced by that act — `Intent → Requested → Realized → Discovered` (ADR-003), linked by `entity_uuid` | + +**Why *realize* is the right act verb** — it is the most **general** option, and that generality is load-bearing: + +- **provision** — resource-only. You do not "provision" a process or an automation outcome. +- **fulfill** — request-only. It names servicing the *ask*, not the making-real of the *thing*. +- **realize** — making *intent* hold true in reality, true for a **resource** *and* an **act/outcome** alike. As DCM grows `process_provider` / automation-outcome providers, only *realize* stays correct. + +**Resolution:** *realize* is the act; *Realized* is the state. Zero blast radius on ADR-003 + the `entity_uuid` chain — and zero new vocabulary. **We do not adopt "fulfillment" as the act.** + +**On engineering's anti-`realize` entry:** the objection targets the *casual English* verb ("ah, now I realize…"), which is fair for prose. But as a defined term-of-art — the act that produces the Realized state — *realize* is exactly right and more general than the alternatives. **Proposal: narrow the anti-vocab entry to the prose sense only; keep `realize`/`Realized` as the canonical act/state pair.** ("Fulfillment" remains fine where engineering/OSAC use it as a label for the request/order *flow* — e.g. OSAC's `fulfillment-service` — but that is a flow name, not our term for the act of making intent real.) + +--- + +## 4. Composite **Resource**, not Compound Service — and Atomic **Resource** + +Two reconciliations here, one consequence of the other. + +**(a) Composite, not Compound.** A *composite* is assembled from parts that **retain their identity**; a *compound* (chemistry) is elements bonded into a new substance where the parts **lose** identity. Our Composition Visibility model (`transparent`/`selective` expose constituents as their own addressable DCM entities) proves the constituents keep their identity — so **composite is the technically correct word**. + +**(b) Resource, not Service — naming the *thing*, consistent with §1–2.** Under *Service = act, Resource = thing*, you **compose things, not acts**: a 3-tier app is web + app + db — a composition of **resources**, not of acts. So the composed noun is **Composite Resource**, and the indivisible leaf is **Atomic Resource**. This falls straight out of the `service_provider → Resource Provider` (name-by-yield) rule — the catalog names the *resource* yielded; the *act* of offering/rendering it is still a Service. + +- **Atomic Resource** — the smallest complete, actionable, indivisible yielded thing; one catalog item tightly coupled to a single resource. *(renames engineering's "Atomic Service" — which already wanted a nickname — and resolves the nickname by naming the thing, not the act.)* +- **Composite Resource** — composed of multiple constituents (Atomic Resources / resource types) that remain individually addressable; one catalog item, one composite entity, one `entity_uuid`. *(renames "Composite Service".)* +- A **Service** (the act) yields an Atomic or Composite **Resource**; the offering/catalog item is the authored definition of that resource, consumed as a service. (The offering's own lifecycle — author/version/approve/manage — is the composite-resource management capability; see the composite-offering enhancement.) + +This supersedes "Composite Service" in `composite-service-model.md` / doc 30 / ADRs — those rename Service→Resource on the *yielded-thing* noun (the act-side wording stays "service"). + +--- + +## 5. Adopted from engineering as-is (clean gap-fills, no conflict) + +- **Infrastructure Platform** — the native substrate a Resource Provider wraps (bare metal, KubeVirt, a Kubernetes distribution, a storage array). UDLM/DCM left this implicit ("provider-native"); engineering names it well. **Adopt it.** It slots cleanly: `Service` (requested) → `Resource Provider` (renders the act) → **`Infrastructure Platform`** (the substrate that backs it). +- **Region / Zone** — engineering defines these as geo + availability-zone topology. They are **named kinds within UDLM `Topology`** (ADR-001: abstract `kind` ∈ {region, zone, rack, host, …}). No conflict: Region/Zone are two concrete topology kinds; UDLM generalizes the dimension so placement/sovereignty/fault-domain gating all resolve against one model. Reference engineering's Region/Zone definitions as the canonical prose for those two kinds. +- **Six-domain "layer cake"** (Value / Application / Control Plane / Resource / Data Center / Governance & FinOps) — this is an **orthogonal** framing to Data · Policy · Provider (ADR-002), not a competitor. The layer cake is a *responsibility/deployment* view (who experiences which layer); Data · Policy · Provider is an *ontological* view (what kind of thing each artifact is). Both stand; we map our components into the layer cake rather than replacing it. + +--- + +## 6. Net: who absorbs what (a genuine two-way reconciliation) + +| Decision | Absorbed by | Cost | +|---|---|---| +| **Service** accepted as the Application-domain act/offering term; anti-vocab narrowed to ban only *unqualified* "Service" | **us (croadfeldt)** | small — edit Anti-Vocabulary entry | +| **Infrastructure Platform** adopted | **us (croadfeldt)** | small — add term | +| **Service Provider → Resource Provider** (named-by-yield; OSAC collision is the motivation, not an approver) | **engineering** | large — repos, contract, enum | +| **Compound → Composite** | **engineering** | small — one vocab entry | +| **`realize` = act, `Realized` = state** kept as the canonical pair; anti-`realize` entry narrowed to the prose sense | **engineering** | small — scope anti-vocab to prose | +| **Region/Zone** kept as canonical kinds within `Topology` | **shared** | none — compose | + +It is not us dictating to engineering: two of the substantive moves are ours to absorb, two are theirs, the rest compose. + +--- + +## 7. The reconciled spine (one picture) + +``` + requests + Consumer ───────────────────────► a SERVICE (the act/offering: "X as a service") + │ + │ the DCM pipeline REALIZES the intent (realize = the act) + ▼ + ┌──────────────────────────────────────────────┐ + │ a PROVIDER renders a service that yields … │ + ├───────────────┬──────────────┬───────────────┤ + │ Resource Prov.│ Process Prov.│ Info/Auth/Peer │ + │ → a RESOURCE │ → an ACT/ │ → data/creds/ │ + │ (a thing) │ OUTCOME │ federation │ + └───────┬───────┴──────────────┴───────────────┘ + │ wraps + ▼ + an INFRASTRUCTURE PLATFORM (the native substrate) + + …which REALIZES the intent → the REALIZED state (Intent → Requested → Realized → Discovered) + Region / Zone = named kinds within Topology, resolved during placement. +``` + +--- + +## 8. Open items for the alignment conversation + +1. **Resource Provider rename** — agree in principle on the named-by-yield rule + OSAC rationale, then sequence the rename (proposal: alias first, flip the canonical name second, deprecate `service_provider` last). +2. **Service at both ends** — confirm "Service = the act" is acceptable as the shared definition (consumer offering ⇄ provider rendering), with **Resource** as the yielded thing. +3. **Realize / Realized** — confirm *realize* = the act, *Realized* = the state (one canonical pair, no "fulfillment" as the act); engineering narrows its anti-`realize` entry to the casual-prose sense. +4. **Composite Resource** (not Compound Service, not "Composite Service") — confirm composing **things** → the noun is Resource; the act stays Service. +5. **Atomic Resource** — confirm; this *also resolves* engineering's "Atomic Service (need a nickname)" by naming the thing, not the act. (renames "Atomic Service".) +6. Whether these land as **UDLM DecisionRecords / DCM ADRs** once agreed (they are architecturally significant enough to warrant the WHY-record). diff --git a/docs/specifications/cncf-strategy.md b/docs/specifications/cncf-strategy.md new file mode 100644 index 0000000..d296b2c --- /dev/null +++ b/docs/specifications/cncf-strategy.md @@ -0,0 +1,256 @@ +# DCM — CNCF Strategy and Community Engagement Plan + +**Document Status:** 📋 Draft — Ready for Implementation Feedback +**Document Type:** Strategy Document + + + + +**Version:** 0.1.0-draft +**Status:** Draft +**Document Type:** Strategic Planning +**Maintainers:** Red Hat FlightPath Team +**Last Updated:** 2026-03 + +--- + +## 1. Strategic Intent + +DCM's goal is to become the community standard for enterprise data center and private cloud management — a neutral, open standard that the industry adopts the way it adopted CSI, CNI, and CRI. This requires DCM to exist in a community-trusted home, not as a vendor product. + +The CNCF (Cloud Native Computing Foundation) is the appropriate home. It provides the neutral governance model, the community infrastructure, and the ecosystem relationships needed to drive broad adoption. The FSI consortium already engaging with DCM (leading FSI consortium members and others) provides the multi-organization sponsorship and real production use case evidence needed for a credible CNCF proposal. + +--- + +## 2. CNCF Landscape and Positioning + +### 2.1 Where DCM Fits + +The CNCF landscape has strong coverage of Kubernetes runtime concerns — container runtimes, networking, storage, service mesh, observability. It has weaker coverage of the management plane — the layer above Kubernetes that governs what gets provisioned, owned, and decommissioned across multiple clusters and infrastructure types. + +DCM fills this gap. It is not competing with existing CNCF projects — it extends and governs them. + +**Related CNCF projects and how DCM relates:** + +| CNCF Project | Relationship to DCM | +|-------------|---------------------| +| **Kubernetes** | DCM is a superset — extends Kubernetes upward to the management plane | +| **Crossplane** | Complementary — Crossplane provisions cloud resources via Kubernetes CRDs; DCM governs what Crossplane provisions and adds the management plane | +| **Cluster API (CAPI)** | DCM can manage Kubernetes clusters via CAPI as a Service Provider | +| **Argo CD / Flux** | Complementary — DCM governs provisioning requests; GitOps manages deployment | +| **OpenCost** | DCM's cost analysis is a superset — OpenCost data can feed DCM cost attribution | +| **Kessel** | Shares inventory and relationship goals — potential collaboration or alignment | +| **OPA/Gatekeeper** | DCM's Policy Engine uses OPA internally; Gatekeeper is the cluster-level enforcement | + +### 2.2 The Gap DCM Fills + +No current CNCF project addresses all of: +- Multi-cluster, multi-infrastructure lifecycle management from a single control plane +- First-class multi-tenancy with Tenant ownership model +- Policy governance with field-level override control across the full request lifecycle +- Data sovereignty and compliance evidence for regulated industries +- Service catalog with self-service consumer experience +- Cost attribution across heterogeneous infrastructure + +This is the gap DCM fills. The positioning is not "another Kubernetes tool" — it is "the management plane that governs your entire data center, of which Kubernetes is one component." + +--- + +## 3. CNCF Submission Path + +### 3.1 CNCF Maturity Levels + +CNCF accepts projects at three maturity levels: + +| Level | Requirements | DCM Target Timeline | +|-------|-------------|---------------------| +| **Sandbox** | Alignment with CNCF mission, basic governance, active development | Target for initial submission | +| **Incubating** | Production users, healthy contributor base, defined governance, security audit | 12-18 months post-Sandbox | +| **Graduated** | Broad adoption, stable API, long-term maintainer commitment | 24-36 months post-Sandbox | + +### 3.2 Sandbox Submission Requirements + +For CNCF Sandbox acceptance, DCM needs: + +**Technical requirements:** +- Clear alignment with CNCF's cloud native mission +- Open source license (Apache 2.0 — already in place) +- Publicly accessible source code (GitHub — already in place) +- Documented roadmap +- Basic security practices (vulnerability disclosure process, etc.) + +**Governance requirements:** +- Defined governance model (maintainers, decision process) +- Code of conduct +- Multi-organization contributor base (this is the key requirement — Red Hat alone is insufficient) + +**Community requirements:** +- Evidence of community interest beyond the founding organization +- At least one non-founding organization actively contributing + +**DCM's strong position:** +The FSI consortium provides exactly the multi-organization evidence CNCF requires. Having leading FSI consortium members as active contributors or committed users is an unusually strong foundation for a Sandbox proposal. Most projects submit to Sandbox without any production users — DCM can submit with evidence of production interest from systemically important financial institutions. + +### 3.3 Recommended Submission Path + +**Step 1 — CNCF TAG (Technical Advisory Group) engagement** +Before formal submission, engage with CNCF TAG App Delivery and TAG Runtime. These groups review cloud native tooling proposals and can provide informal feedback before the formal Due Diligence process. Presenting DCM at a TAG meeting builds awareness and surfaces concerns early. + +**Step 2 — Prepare the Due Diligence document** +The CNCF Due Diligence document is a detailed technical and governance questionnaire. Key sections: project description, statement on alignment with CNCF mission, comparison to similar projects, security practices, roadmap, adopters. The FSI consortium adopters section will be a significant differentiator. + +**Step 3 — TOC sponsor identification** +CNCF Technical Oversight Committee (TOC) members sponsor project proposals. Red Hat's relationships in the Kubernetes community make identifying a TOC sponsor feasible. Target TOC members with expertise in multi-cluster management or enterprise Kubernetes. + +**Step 4 — Sandbox vote** +TOC votes on Sandbox acceptance. With a strong Due Diligence document, FSI adopter evidence, and a TOC sponsor, acceptance probability is high. + +--- + +## 4. Community Engagement Strategy + +### 4.1 Operator Ecosystem — The Primary Leverage Point + +The DCM Operator Interface Specification is the primary community artifact for driving ecosystem adoption. The strategy is to make conformance attractive enough that operator maintainers want to implement it. + +**Priority operator communities for engagement:** + +| Operator | Community | Why Priority | Engagement Approach | +|----------|-----------|-------------|---------------------| +| **KubeVirt** | Red Hat/Community | Active DCM development already | Direct contribution — DCM team contributes Level 2 support | +| **CloudNativePG** | CNPG Community | High FSI adoption — databases in regulated environments | Present DCM at CNPG community calls, contribute SDK example | +| **Strimzi (Kafka)** | Red Hat/Community | Messaging infrastructure — DCM Message Bus use case | Direct contribution via Red Hat maintainership | +| **Cert-Manager** | Jetstack/Venafi | Security resources — every DCM deployment needs certificates | SDK contribution, present at KubeCon | +| **ACM** | Red Hat | Cluster management — natural DCM complement | Direct — internal Red Hat alignment | +| **Rook (Ceph)** | CNCF | Storage operator — core DCM service provider use case | CNCF relationship — present at SIG Storage | + +### 4.2 KubeCon Strategy + +KubeCon is the primary conference for Kubernetes ecosystem influence. DCM needs a presence at KubeCon North America and Europe: + +**KubeCon NA (target — next edition):** +- Submit a talk: "DCM — Managing the Management Plane: Kubernetes as a Component of Enterprise Infrastructure" +- Submit a contribfest session: hands-on DCM Operator SDK implementation workshop +- Engage Kubernetes SIG Cluster Lifecycle about CAPI integration + +**KubeCon EU (following year):** +- Present CNCF Sandbox submission (if accepted by then) +- Case study talk with FSI consortium member (FSI consortium members presenting their DCM deployment) +- Operator Interface Specification BOF (Birds of a Feather) session + +### 4.3 The Developer Value Proposition — What We Need to Communicate + +The community message must be concrete and compelling, not abstract. Avoid "unified management plane" as the opener — lead with what operators get: + +**For operator developers:** +> "Add DCM support to your operator and your users get self-service catalog, multi-tenancy, cost attribution, and cross-cluster management — for free. It takes one day using our SDK." + +**For platform engineering teams:** +> "Manage your entire data center from one control plane. VMs, databases, Kubernetes clusters, networking — all with the same declarative model, the same policy engine, and the same audit trail." + +**For FSI/regulated industry teams:** +> "Every provisioning request produces a complete audit chain — who asked for what, what policies applied, what was approved, what was built. Sovereignty constraints enforced at the management plane, not bolted on afterward." + +### 4.4 Contributor Onboarding + +A project cannot become a standard without contributors beyond the founding organization. The contributor onboarding strategy: + +**Good first issues:** +Maintain a curated list of well-scoped, well-documented issues labeled `good-first-issue`. These should be achievable in a few hours without deep DCM knowledge — documentation improvements, test coverage, example implementations, SDK feature additions. + +**Operator SDK examples:** +Each operator SDK example is a potential contributor touchpoint. An operator maintainer who wants to add DCM support to their operator is a natural contributor. The example for their specific CRD framework (kubebuilder, operator-sdk, raw controller-runtime) lowers the barrier. + +**RFC process:** +Establish a lightweight RFC (Request for Comments) process for significant changes to the DCM Operator Interface Specification. This gives external contributors a formal path to influence the specification direction — which is essential for community trust. + +**Monthly community calls:** +Regular community calls (video, recorded, published) signal active project health and give contributors a forum to discuss ideas. Target: bi-weekly during active development, monthly once stable. + +--- + +## 5. Standards Positioning + +Beyond CNCF, DCM should engage with relevant standards bodies where appropriate: + +### 5.1 DMTF (Distributed Management Task Force) +DMTF maintains the TOSCA (Topology and Orchestration Specification for Cloud Applications) and other cloud management standards. DCM's data model has some conceptual overlap with TOSCA. Rather than competing, DCM should position as a Kubernetes-native, GitOps-native evolution of the same problem TOSCA addressed — bringing the conversation into the cloud native era. + +### 5.2 FinOS Foundation +FinOS is the open source community for financial services. The FSI consortium involvement makes FinOS a natural secondary community for DCM. Presenting DCM at FinOS events reaches exactly the regulated industry audience that benefits most from DCM's sovereignty and compliance capabilities. + +### 5.3 OpenInfra Foundation +The OpenInfra Foundation hosts OpenStack, Kata Containers, and StarlingX — all relevant to DCM's target environments (private cloud, edge, regulated infrastructure). DCM should present at OpenInfra Summit to the platform engineering teams who manage these environments. + +--- + +## 6. What Needs to Exist Before CNCF Submission + +The following artifacts must be ready before a CNCF Sandbox submission is credible: + +| Artifact | Status | Owner | Target | +|----------|--------|-------|--------| +| DCM Operator Interface Specification v1.0 | 🔄 Draft | DCM Project | Ready | +| DCM Operator SDK v0.1.0 (Level 1 + Level 2) | 📋 Not started | DCM Project | 3-6 months | +| KubeVirt reference implementation (Level 2) | 🔄 In progress | DCM/KubeVirt teams | 3-6 months | +| Conformance test suite (Level 1 + Level 2) | 📋 Not started | DCM Project | 3-6 months | +| CNCF Due Diligence document | 📋 Not started | DCM Project | 6 months | +| Governance model document | 📋 Not started | Red Hat/Consortium | 3 months | +| Security vulnerability disclosure process | 📋 Not started | Red Hat Security | 1 month | +| FSI consortium adopter statements | 📋 Not started | Consortium members | 3 months | +| Second non-Red Hat maintainer | 📋 Not started | Community | 6 months | + +--- + +## 7. Risk Considerations + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| CNCF TAG sees overlap with Crossplane | Medium | Medium | Prepare clear differentiation — DCM governs, Crossplane provisions; they are complementary | +| Operator communities resist specification adoption | Medium | High | Lead with SDK ease, reference implementations, concrete value; don't mandate, make it attractive | +| Red Hat perceived as controlling the standard | Medium | High | Establish CNCF governance early, actively recruit non-Red Hat maintainers, FSI consortium co-ownership | +| Specification fragmentation — forks or competing standards | Low | High | CNCF neutral governance prevents this; be the first mover in this space | +| Key contributor departure | Low | Medium | CNCF governance ensures project continuity beyond any single contributor | + +--- + +## 8. Open Questions + +| # | Question | Impact | Status | +|---|----------|--------|--------| +| 1 | Should the CNCF submission be for DCM as a whole or for the DCM Operator Interface Specification as a standalone standard? | Scope of submission | ✅ Resolved | +| 2 | Which FSI consortium members are willing to be named as public adopters in the CNCF submission? | Submission strength | ✅ Resolved | +| 3 | Is there a TOC member with relevant expertise who could sponsor the DCM proposal? | Submission path | ✅ Resolved | +| 4 | Should DCM engage with the Kubernetes SIG structure before or after CNCF Sandbox submission? | Community positioning | ✅ Resolved | +| 5 | What is the timeline for the KubeVirt reference implementation reaching Level 2 conformance? | Readiness milestone | ✅ Resolved | + +--- + + + +## Resolution Notes + +**Q1:** Submit the DCM Operator Interface Specification as a CNCF specification project first. CNCF Sandbox project submission for DCM as a whole follows once a reference implementation reaches Level 2 conformance. Submitting the specification standard separately lowers the implementation bar for initial acceptance and establishes the interface contract independently of any single implementation. + +**Q2:** Identify a minimum of two named production evaluators and one FSI design partner before submission. At least one named organization should be willing to go on record. This is a project team action item — the architecture does not determine who those organizations are. + +**Q3:** Target the App Delivery TAG and Runtime TAG for initial sponsor identification. Engage SIG App Delivery and SIG Cluster Lifecycle before submission — SIG members frequently become TOC sponsors. Project team action item. + +**Q4:** SIG engagement comes before Sandbox submission. SIG App Delivery and SIG Cluster Lifecycle are the primary targets. The Cluster API overlap specifically must be addressed with SIG Cluster Lifecycle before submission. Pre-submission SIG engagement surfaces conflicts, identifies sponsors, and positions DCM as a collaborative project rather than a competing one. + +**Q5:** Level 2 conformance requires: full dispatch/cancel/discover cycle, full realized state reporting, governance matrix enforcement at the provider boundary, and health check compliance. These requirements are now formally defined in the Operator Interface Specification. The project team estimates timeline based on available engineering resources against this defined scope. + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* + +## Red Hat Developer Hub / Backstage Integration + +DCM implements a Backstage plugin suite (`@dcm/backstage-plugin-*`) for deployment as RHDH Dynamic Plugins. This is the primary consumer-facing deployment model. See [RHDH Integration Specification](dcm-rhdh-integration-spec.md) for the complete architecture. + +**CNCF alignment:** Backstage is a CNCF incubating project. DCM's RHDH integration follows CNCF best practices for developer portals and internal developer platforms (IDPs). + +| Component | CNCF Status | DCM Use | +|-----------|------------|---------| +| Backstage | Incubating | Primary consumer GUI platform | +| Backstage Software Templates | Backstage feature | Auto-generated from DCM catalog items | +| Backstage Catalog | Backstage feature | DCMService and DCMResource entity kinds | +| Backstage Permission Framework | Backstage feature | DCM role → Backstage permission bridge | diff --git a/schemas/openapi/AEP-CONFORMANCE.md b/schemas/openapi/AEP-CONFORMANCE.md new file mode 100644 index 0000000..8a0430a --- /dev/null +++ b/schemas/openapi/AEP-CONFORMANCE.md @@ -0,0 +1,48 @@ +# DCM OpenAPI — AEP conformance + +DCM's public APIs adopt the **[API Enhancement Proposals](https://aep.dev/)** (AEP) — resource-oriented +design, the standard methods (Get/List/Create/Update/Delete), and the RFC 9457 error model — per +**ADR-AEP-001** (croadfeldt/udlm). Conformance is checked by the AEP **Spectral** OpenAPI ruleset. + +## Run the linter locally + +``` +npm install @stoplight/spectral-cli @aep_dev/aep-openapi-linter +npx spectral lint "schemas/openapi/*.yaml" --ruleset .spectral.yaml +``` + +CI runs the same lint on every PR that touches `schemas/openapi/**` (`.github/workflows/lint-openapi.yml`). +It is **advisory** today (`continue-on-error: true`) while the baseline below is burned down; flip it to +blocking once the error count reaches zero. + +## Baseline (2026-07-08, first run) + +| Spec | Errors | Warnings | Notes | +|------|--------|----------|-------| +| `dcm-consumer-api.yaml` | 144 | 236 | | +| `dcm-operator-api.yaml` | 32 | 24 | | +| `dcm-provider-callback-api.yaml` | 22 | 25 | | +| `dcm-admin-api.yaml` | — | — | **P0: invalid YAML — does not parse / lint (see below)** | + +### Top rule categories (across the three parseable specs) + +| Count | Rule(s) | Category | +|------:|---------|----------| +| ~110 | `aep-158-*` (next-page-token, max-page-size, page-token) | **Pagination** — List methods need `page_size`/`page_token` params + `next_page_token` in the response | +| ~90 | `aep-131/132/133/135-*` (operation-id, request-body, response-body) | **Standard methods** — Get/List/Create/Update/Delete operationId + body conventions | +| ~62 | `aep-142-time-field-*` | **Time fields** — timestamp fields must be named `*_time` | +| 77 | `operation-description` | Every operation needs a description | +| 35 | `oas3-schema` | Structural OpenAPI validity (non-AEP) | +| ~7 | `aep-193-error-response-schema` | **Errors** — responses should be RFC 9457 Problem Details | +| misc | `aep-140-uri-property-naming`, `aep-136-operation-id` (LRO) | naming / long-running operations | + +## Remediation plan (ratchet the baseline down) + +1. **P0 — fix `dcm-admin-api.yaml` structural corruption.** It has **two `components:` blocks** (~line 1535 and ~line 1838) and duplicated/misplaced `/api/v1/admin/overrides…` path blocks nested under `components:`; the YAML is invalid (fails to parse at ~line 1846). Repair: consolidate to one `components:`, move the misplaced path blocks under `paths:`, de-duplicate. Then it can lint. +2. **RFC 9457 errors (`aep-193`).** Replace the bespoke `Error` / `OperatorError` / `ProviderError` schemas with a shared `ProblemDetails` schema (`type/status/title/detail/instance` + extension members), `application/problem+json` — consistent with the UDLM error model (croadfeldt/udlm `contracts/error-model.md`, ADR-AEP-001). +3. **Pagination (`aep-158`).** Give every List method `page_size` + `page_token` parameters and `next_page_token` in the response. +4. **Standard methods (`aep-131/132/133/135`).** Normalize operationIds and request/response bodies to the AEP method shapes. +5. **Time fields (`aep-142`).** Rename timestamp fields to the `*_time` suffix. +6. **Descriptions.** Add operation descriptions. + +Flip CI to blocking after step 4; steps 5–6 are polish.