From 317e848a0a87b735eac37d4150db9673184c5c51 Mon Sep 17 00:00:00 2001 From: Chris Roadfeldt Date: Tue, 16 Jun 2026 16:43:23 -0500 Subject: [PATCH 1/2] architecture: convergence engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The convergence engine — the intent to realized loop (load-bearing). Signed-off-by: Chris Roadfeldt Signed-off-by: croadfeldt --- .../dependency-orchestration.md | 348 ++++++++++++ architecture/convergence-engine/overview.md | 214 +++++++ .../convergence-engine/policy-evaluation.md | 262 +++++++++ .../convergence-engine/recovery-and-retry.md | 453 +++++++++++++++ architecture/convergence-engine/scoring.md | 535 ++++++++++++++++++ 5 files changed, 1812 insertions(+) create mode 100644 architecture/convergence-engine/dependency-orchestration.md create mode 100644 architecture/convergence-engine/overview.md create mode 100644 architecture/convergence-engine/policy-evaluation.md create mode 100644 architecture/convergence-engine/recovery-and-retry.md create mode 100644 architecture/convergence-engine/scoring.md diff --git a/architecture/convergence-engine/dependency-orchestration.md b/architecture/convergence-engine/dependency-orchestration.md new file mode 100644 index 0000000..41f0f91 --- /dev/null +++ b/architecture/convergence-engine/dependency-orchestration.md @@ -0,0 +1,348 @@ +--- +Document Status: ✅ Stable — DCM implementation +Document Type: Architecture Reference — Dependency Orchestration +Established: 2026-05-26 +Maps to: udlm/lifecycle/request-dependency-graph.md +--- + +# Convergence Engine — Dependency Orchestration + +> **Implements contracts defined in UDLM**: +> [udlm/lifecycle/request-dependency-graph.md](https://github.com/croadfeldt/udlm/blob/main/lifecycle/request-dependency-graph.md). +> UDLM defines the Request Dependency Group structure, `wait_for` values, +> field injection mechanism, `PENDING_DEPENDENCY` state contract, failure +> handling propagation policy, group timeout contract, and the relationship +> to composite service definitions. DCM operationalizes the submission, +> parsing, dispatch orchestration, state lifecycle, failure execution, and +> consumer API. + +--- + +## 1. Request dependency graph submission and parsing + +Consumers submit ad-hoc cross-request ordering via the dependency group +endpoint: + +``` +POST /api/v1/request-groups + +{ + "group_handle": "three-tier-app-deploy", + "on_failure": "cancel_remaining", + "timeout": "PT2H", + "requests": [ + { "ref": "db", + "catalog_item_uuid": "", + "fields": { ... } }, + { "ref": "app", + "catalog_item_uuid": "", + "fields": { ... }, + "depends_on": [ + { "ref": "db", + "wait_for": "realized", + "inject_fields": [ + { "from_field": "realized_fields.primary_ip", + "to_field": "fields.db_host" } + ] + } + ] + }, + { "ref": "lb", + "catalog_item_uuid": "", + "fields": { ... }, + "depends_on": [ + { "ref": "app", "wait_for": "realized", + "inject_fields": [ ... ] } + ] + } + ] +} +``` + +### 1.1 Parsing and validation + +The Request Orchestrator parses the submission and: + +1. **Validates DAG-ness** — runs a topological sort; rejects with 422 if a + cycle is detected (`RDG-001`) +2. **Validates group size** — rejects with 422 if request count exceeds the + profile's max_group_size +3. **Validates nesting depth** — rejects with 422 if dependency chain depth + exceeds the profile's max_nesting_depth +4. **Validates field injection paths** — checks `from_field` paths against + the dependency's resource type spec; per profile, validation may be + advisory (warn), enforced (reject), or policy-gated (also pass through + GateKeeper) +5. **Allocates entity UUIDs** for each request (so the group response can + return entity_uuid immediately) +6. **Resolves local refs** to actual entity UUIDs in the dependency graph + +On valid submission, DCM writes the group record and returns 202 with the +full request list in `PENDING_DEPENDENCY` status. + +### 1.2 Quota enforcement at group submission + +`PENDING_DEPENDENCY` requests count against the consumer's quota immediately +(`RDG-004`). Resources are reserved at group submission, not at dispatch +time. This prevents a consumer from submitting a 50-request group and then +finding only 30 fit in their quota when they start dispatching. + +--- + +## 2. PENDING_DEPENDENCY state mechanics + +`PENDING_DEPENDENCY` is a new status DCM adds to the Intent State lifecycle: + +``` +ACKNOWLEDGED → PENDING_DEPENDENCY → [dependency met] → LAYERS_ASSEMBLED → ... → REALIZED +``` + +PENDING_DEPENDENCY requests: + +- Are visible via `GET /api/v1/requests?status=PENDING_DEPENDENCY` +- Are cancellable via `DELETE /api/v1/requests/{uuid}` +- Receive a `request.pending_dependency` event (urgency: info) +- Do NOT have independent timeouts — the group-level `timeout` governs + +The Request Orchestrator maintains an in-memory dependency graph per group +plus a persistent state in `request_dependency_groups` and +`request_dependencies` tables. On every dependency state change, the +orchestrator re-evaluates dependents: when a dependency reaches its +`wait_for` state, blocked dependents transition out of `PENDING_DEPENDENCY` +and enter the standard assembly pipeline. + +### 2.1 Field injection execution + +When a dependency reaches its `wait_for` state and dependents have +`inject_fields`: + +``` +Dependency realized → Realized State written + │ + ▼ Request Orchestrator looks up inject_fields declarations for dependents + │ For each injection: + │ Extract from_field path from Realized State + │ If extraction fails: per profile, warn (advisory) or fail dispatch (enforced) + │ Inject value at to_field path in dependent's fields + │ + ▼ Dependent request transitions out of PENDING_DEPENDENCY + │ Enters standard nine-step assembly with injected fields + │ + ▼ Injected values pass through Transformation policies normally (RDG-003) +``` + +Injected values are not exempt from policy evaluation. A Transformation +policy that modifies `db_host` will modify the injected value just as it +would a consumer-declared one. + +--- + +## 3. wait_for state evaluation + +DCM tracks four `wait_for` states: + +| Value | Triggered when | +|---|---| +| `acknowledged` | Dependency has entity_uuid (post-Request Orchestrator acknowledgment) | +| `approved` | Dependency has passed all approvals (post-approval tier evaluation) | +| `dispatched` | Dependency has been sent to its provider | +| `realized` | Dependency is fully realized (Realized State written, entity.realized event emitted) | + +The Request Orchestrator subscribes to the events that mark each +transition; when any dependency state matches a dependent's `wait_for`, +the orchestrator unblocks the dependent. + +`realized` is the most common and default. + +--- + +## 4. Failure handling execution + +UDLM defines two propagation policies: `cancel_remaining` and `continue`. +DCM executes: + +### 4.1 cancel_remaining + +``` +Request fails (provider error, recovery policy DISCARD_NO_REQUEUE, etc.) + │ + ▼ Orchestrator inspects the group's on_failure policy + │ on_failure: cancel_remaining + │ + ▼ For all requests in the group with status in + │ {PENDING_DEPENDENCY, ACKNOWLEDGED}: + │ Transition to CANCELLED + │ failure_reason: dependency_failed + │ Emit request.cancelled event + │ + ▼ For already-dispatched requests: follow standard cancellation model + │ (Section 3 in recovery-and-retry.md) + │ + ▼ Emit request.failed for the original failure + │ Emit request.group_failed for the group + │ + ▼ Group status → failed +``` + +### 4.2 continue + +The failed request is marked FAILED. Dependents that depended on it are also +marked FAILED with `failure_reason: dependency_failed`. Independent requests +in the group continue unaffected. The group transitions to `failed` once all +requests reach a terminal state. + +--- + +## 5. Group timeout enforcement + +The Request Orchestrator maintains a per-group timer. When `timeout` elapses +without all requests reaching a terminal state: + +``` +Group timeout reached + │ + ▼ For all non-terminal requests in the group: + │ Transition to FAILED + │ failure_reason: group_timeout + │ Emit request.failed event + │ + ▼ Group status → failed + │ Emit request.group_failed event +``` + +Group timeout is measured from group submission (not from first dispatch). +Individual requests do not have independent timeouts while in +PENDING_DEPENDENCY status — only the group timeout governs that window +(`RDG-005`). + +--- + +## 6. Consumer API endpoints + +### 6.1 Submit dependency group + +``` +POST /api/v1/request-groups +``` + +See Section 1 above for the request body schema. + +Response 202: +```json +{ + "group_uuid": "", + "group_handle": "three-tier-app-deploy", + "requests": [ + { "ref": "db", "request_uuid": "", "entity_uuid": "", "status": "ACKNOWLEDGED" }, + { "ref": "app", "request_uuid": "", "entity_uuid": "", "status": "PENDING_DEPENDENCY" }, + { "ref": "lb", "request_uuid": "", "entity_uuid": "", "status": "PENDING_DEPENDENCY" } + ], + "estimated_completion": "" +} +``` + +### 6.2 Add a request to an existing group + +``` +POST /api/v1/request-groups/{group_uuid}/members +{ + "request_uuid": "", + "depends_on": [ ... ] +} +``` + +A request may belong to at most one group (`RDG-006`); adding to a second +group returns 409 Conflict. + +### 6.3 Query group status + +``` +GET /api/v1/request-groups/{group_uuid} + +→ { + "group_uuid": "", + "group_handle": "three-tier-app-deploy", + "status": "in_progress | completed | failed | cancelled", + "requests": [ + { "request_uuid": "", "ref": "db", "status": "REALIZED" }, + { "request_uuid": "", "ref": "app", "status": "DISPATCHED" }, + { "request_uuid": "", "ref": "lb", "status": "PENDING_DEPENDENCY" } + ], + "created_at": "", + "timeout_at": "" + } +``` + +### 6.4 Cancel a group + +``` +DELETE /api/v1/request-groups/{group_uuid} + +# Cancels all PENDING_DEPENDENCY and ACKNOWLEDGED requests in the group. +# Already-dispatched requests follow the standard cancellation model. +Response 204 +``` + +--- + +## 7. New events + +| Event | Urgency | Trigger | +|---|---|---| +| `request.pending_dependency` | info | Request entered PENDING_DEPENDENCY | +| `request.dependency_met` | info | Dependency reached wait_for state; request proceeding | +| `request.group_completed` | medium | All requests in group reached terminal state | +| `request.group_failed` | high | Group failed or timed out | + +These events are added to the UDLM event catalog ( +[udlm/contracts/event-catalog.md](https://github.com/croadfeldt/udlm/blob/main/contracts/event-catalog.md)) +under the `request.*` domain. + +--- + +## 8. Profile-governed constraints + +| Profile | Max group size | Max timeout | Field injection validation | Max nesting depth | +|---|---|---|---|---| +| minimal | 100 | P30D | advisory | 3 | +| dev | 100 | P7D | advisory | 3 | +| standard | 50 | P3D | enforced | 3 | +| prod | 25 | P1D | enforced + audited | 3 | +| fsi | 10 | PT8H | enforced + audited + policy gated | 2 | +| sovereign | 5 | PT4H | enforced + audited + policy gated | 2 | + +`RDG-002` sets an absolute upper bound of 100 — profiles may set lower limits +but no profile may set higher. + +--- + +## 9. Relationship to composite service definitions + +Request dependency groups and composite service definitions solve overlapping +but distinct problems: + +| | Request Dependency Group | Composite Service Definition | +|---|---|---| +| Who declares | Consumer at request time | Platform team at catalog time | +| Reusable | No — ad hoc | Yes — catalog item | +| Type constraints | None — any resources | Defined by composite spec | +| Policy governance | Standard consumer request policies | Composite Service policies (CMP-*) | +| Field injection | Consumer-declared inject_fields | Composite handles internally | +| Use case | Ad-hoc deployment ordering | Standard composite service | + +When a standard composite service exists as a composite service definition, +consumers should use it. Request dependency groups are for deployments that +don't fit a predefined composite pattern. + +--- + +## 10. Policy IDs (DCM realization) + +| Policy | Rule | +|---|---| +| `RDG-001-DCM` | DCM rejects circular dependency graphs at submission time (422) via topological sort validation | +| `RDG-002-DCM` | DCM enforces a profile-governed maximum group size with absolute upper bound of 100 | +| `RDG-003-DCM` | DCM applies all active Transformation policies to injected field values; injection does not bypass policy | +| `RDG-004-DCM` | DCM counts PENDING_DEPENDENCY requests against consumer quota at group submission time | +| `RDG-005-DCM` | DCM enforces group-level timeout from group submission; individual requests have no independent timeout while PENDING_DEPENDENCY | +| `RDG-006-DCM` | DCM enforces single-group membership; attempts to add a request to a second group return 409 Conflict | diff --git a/architecture/convergence-engine/overview.md b/architecture/convergence-engine/overview.md new file mode 100644 index 0000000..4aa67ab --- /dev/null +++ b/architecture/convergence-engine/overview.md @@ -0,0 +1,214 @@ +--- +Document Status: 📋 Draft — Initial Specification +Document Type: Architecture Reference — Convergence Engine +Established: 2026-05-26 +Maps to: UDLM four-states contract, capability discovery contract +--- + +# Convergence Engine — Overview + +> **Implements contracts defined in UDLM**: +> [udlm/foundations/four-states.md](https://github.com/croadfeldt/udlm/blob/main/foundations/four-states.md), +> [udlm/contracts/provider-contract.md](https://github.com/croadfeldt/udlm/blob/main/contracts/provider-contract.md), +> [udlm/contracts/event-catalog.md](https://github.com/croadfeldt/udlm/blob/main/contracts/event-catalog.md), +> [udlm/contracts/capability-discovery.md](https://github.com/croadfeldt/udlm/blob/main/contracts/capability-discovery.md). + +The convergence engine is the heart of DCM. It walks data through the four +UDLM states — **intent → requested → realized → discovered** — and +continuously reconciles realized state against intent. Everything else in +DCM exists to support this loop. + +--- + +## 1. What the engine does + +UDLM defines the four states and the allowed transitions between them. UDLM +does **not** prescribe how a realization drives those transitions. The DCM +convergence engine is one specific answer: + +1. **Accepts intent** (via the API gateway from any ingress: API, GitOps, + CLI, message bus, scheduled trigger). Writes an Intent State record. +2. **Assembles a Requested State** by running the nine-step assembly: + layer resolution → layer merge → policy evaluation → placement → score → + approval routing → Requested State persistence → dispatch preparation → + audit emission. +3. **Dispatches** to the selected provider with a scoped, short-lived + `dcm_interaction` credential. Waits for realization within configured + timeout. +4. **Persists Realized State** on provider callback. Updates the entity + lifecycle state, fires `entity.realized` events. +5. **Reconciles continuously** via the Discovery Service. Polls providers + on schedule, writes Discovered State, compares to Realized State, fires + drift events. The Policy Engine evaluates each drift through Recovery + Policies. + +The loop runs forever — entities continuously move toward their declared +intent until decommissioned. + +--- + +## 2. The engine's responsibilities + +| Responsibility | How DCM fulfills it | +|---|---| +| Walk an entity from intent to realized | Request Orchestrator drives the nine-step pipeline; Request Processor performs assembly | +| Evaluate policy at every transition | Policy Manager evaluates GateKeeper / Validation / Transformation / Recovery / Orchestration Flow / Governance Matrix policies via OPA | +| Select a provider for placement | Placement Manager runs the six-step placement algorithm: sovereignty pre-filter → eligibility filter → capability filter → reserve query → scoring → tie-break | +| Dispatch with scoped credentials | API Gateway requests a `dcm_interaction` credential from the Credential Provider, scoped to the specific provider + entity + operation, valid for PT15M–PT1H per profile | +| React to provider events | Provider callbacks land at the Provider Callback API; the Request Orchestrator routes to Realized State persistence and event emission | +| Detect drift | Discovery Service polls per Resource Type Spec's `discovery_schedule`; Drift Detection compares Discovered to Realized field-by-field | +| Evaluate recovery on failure | Recovery Policies fire on declared triggers (timeout, cancellation failure, partial realization, compensation failure); evaluate via the same Policy Manager | +| Audit everything | Audit Service appends a record on every state transition, policy decision, credential issuance, and provider call; SHA-256 hash chain provides tamper evidence | + +--- + +## 3. Pipeline routing — how events flow + +DCM uses PostgreSQL's `LISTEN/NOTIFY` for pipeline routing in standard +deployments (Kafka added as an enhancement for high-throughput deployments). +Every state transition writes a row to `pipeline_events`; a trigger fires +`pg_notify`; subscribed services consume. + +``` +Consumer submits intent + │ POST /api/v1/requests + ▼ +API Gateway — authenticates, injects X-DCM-Tenant + │ + ▼ +Request Orchestrator — writes intent_records row, emits intent.acknowledged + │ LISTEN/NOTIFY + ▼ +Request Processor — assembles, writes requested_records row, emits requested.assembled + │ LISTEN/NOTIFY + ▼ +Policy Manager — evaluates policies, computes score, emits policy.evaluated + score.computed + │ LISTEN/NOTIFY + ▼ +Placement Manager — selects provider, emits placement.decided + │ LISTEN/NOTIFY + ▼ +Request Orchestrator — issues interaction credential, dispatches to provider + │ HTTP POST to provider's dispatch endpoint + ▼ +Provider — realizes the resource, calls back to /api/v1/instances/{id}/status + │ + ▼ +Request Orchestrator — writes Realized State, emits entity.realized + │ LISTEN/NOTIFY + ▼ +Audit Service — appends audit record with hash chain + │ + ▼ +Discovery Service — runs scheduled discovery for the resource type + │ + ▼ +Drift Detection — compares Discovered to Realized, emits drift events if differ + │ LISTEN/NOTIFY + ▼ +Policy Manager — evaluates Recovery Policies, fires configured action +``` + +Every step also writes provenance, so the full chain is reconstructable from +the audit trail. + +--- + +## 4. Capability discovery and provider matching + +UDLM's +[capability-discovery.md](https://github.com/croadfeldt/udlm/blob/main/contracts/capability-discovery.md) +defines the unified provider model: a provider is an external system that +declares **capabilities** (`realize_resources`, `serve_data`, `authenticate`, +`federate`, `execute_workflows`), not a fixed type. + +DCM's provider registry implements this: + +- Each provider registration includes a `capabilities` block plus declared + `supported_resource_types`. +- The Placement Manager matches a request's resource_type and constraints + against providers whose declared `realize_resources` capability includes + that type. +- A provider declaring multiple capabilities (e.g., InfoBlox declaring both + `serve_data` for IP availability queries and `realize_resources` for + Network.IPAddress allocation) is matched separately for each capability. + +**DCM exposes `GET /api/v1/capabilities`** — the machine-readable advertisement +of what this DCM instance can do (lifecycle management, policy evaluation, +cost analysis, audit trail, placement decisions, drift detection, entity +lifecycle events, subscribe endpoints). External systems (FinOps tools, audit +tools, DAV, federation peers) query this endpoint to discover DCM's +capabilities before integrating. + +**Backward compatibility:** the legacy typed-provider names +(`service_provider`, `information_provider`, etc.) are retained as resolved +labels derived from declared capabilities. Existing registrations continue +to work. + +System policies: `DISC-001` through `DISC-005` (in +[udlm/contracts/capability-discovery.md](https://github.com/croadfeldt/udlm/blob/main/contracts/capability-discovery.md)) +govern capability advertisement authentication, tenant scoping, rate limiting, +and the advisory nature of needs_from_dcm matching. + +--- + +## 5. State semantics — what each state means inside DCM + +| State | What's in it | Where it lives in DCM | Who writes it | +|---|---|---|---| +| **Intent** | Consumer's raw declaration before any processing | `intent_records` table | Request Orchestrator on ingress | +| **Requested** | Assembled, policy-evaluated, placed payload | `requested_records` table | Request Processor after nine-step assembly | +| **Realized** | What the provider built, with provider-side fields | `realized_entities` table (versioned, `is_current` flag) | Request Orchestrator on provider callback | +| **Discovered** | What the provider currently reports | `discovered_records` table (ephemeral snapshots) | Discovery Service on scheduled poll | + +The **Realized State only changes via an authorized request that produces a +corresponding Requested State record** (UDLM invariant RSE-010). Drift detection, +discovery cycles, and lifecycle events do not write to the Realized Store — +they write to other domains and trigger policy evaluation that may produce a +new Requested State. + +--- + +## 6. Reading order for engine internals + +If you're operationalizing or extending the convergence engine, read in this +order: + +1. This overview. +2. [`policy-evaluation.md`](policy-evaluation.md) — how DCM evaluates the unified + governance matrix, hard/soft enforcement, caching, sovereignty zones. +3. [`scoring.md`](scoring.md) — the hybrid scoring model, signals, approval routing. +4. [`recovery-and-retry.md`](recovery-and-retry.md) — timeouts, cancellation, + orphan detection, recovery policy execution, compensation. +5. [`dependency-orchestration.md`](dependency-orchestration.md) — consumer + request dependency graphs, PENDING_DEPENDENCY lifecycle, field injection, + failure handling. + +For governance enforcement specifically: +- [`../governance-enforcement/accreditation-monitor.md`](../governance-enforcement/accreditation-monitor.md) +- [`../governance-enforcement/registry-enforcement.md`](../governance-enforcement/registry-enforcement.md) +- [`../governance-enforcement/contribution-pipeline.md`](../governance-enforcement/contribution-pipeline.md) + +For provider interaction: +- [`../credentials-and-auth/provider-callback.md`](../credentials-and-auth/provider-callback.md) +- [`../credentials-and-auth/credentials.md`](../credentials-and-auth/credentials.md) + +--- + +## 7. Design invariants (DCM-level) + +These hold across all profiles and deployments: + +- The nine-step assembly is the only path from Intent to Requested +- The convergence loop is the only path from Requested to Realized +- The Discovery Service is the only writer of Discovered State +- Every state transition emits an event to `pipeline_events` +- Every state transition produces an audit record +- Every provider call carries a scoped, short-lived interaction credential +- Recovery Policies fire on every closed-vocabulary trigger; the action is + evaluated through the same Policy Manager as any other policy + +These are realization invariants for **DCM specifically**. A peer realization +might choose a different routing mechanism (e.g., Kafka instead of +`LISTEN/NOTIFY`), a different assembly algorithm, or a different audit +storage — and remain UDLM-conformant as long as the wire contracts are honored. diff --git a/architecture/convergence-engine/policy-evaluation.md b/architecture/convergence-engine/policy-evaluation.md new file mode 100644 index 0000000..94259de --- /dev/null +++ b/architecture/convergence-engine/policy-evaluation.md @@ -0,0 +1,262 @@ +--- +Document Status: ✅ Stable — DCM implementation +Document Type: Architecture Reference — Governance Matrix Evaluator +Established: 2026-05-26 +Maps to: udlm/governance/governance-matrix.md +--- + +# Convergence Engine — Policy Evaluation (Governance Matrix Evaluator) + +> **Implements contracts defined in UDLM**: +> [udlm/governance/governance-matrix.md](https://github.com/croadfeldt/udlm/blob/main/governance/governance-matrix.md). +> UDLM defines the unified governance matrix as the single enforcement point +> for cross-boundary decisions, the four matrix axes (subject / data / target / +> context), the decision vocabulary (ALLOW / DENY / ALLOW_WITH_CONDITIONS / +> STRIP_FIELD / REDACT / AUDIT_ONLY), the hard-vs-soft enforcement +> distinction, and field-level controls. DCM operationalizes the evaluation +> algorithm, the caching, the sovereignty zone management, and the integration +> with the convergence pipeline. + +> **Manifest note**: this file is THE matrix evaluator. The DCM split +> manifest had a duplication bug that listed file 14-policy-profiles under +> both `convergence-engine/policy-evaluation.md` and +> `governance-enforcement/policy-profiles.md`. Resolution: file 14 lives at +> `governance-enforcement/policy-profiles.md` (it's pure-dcm and already +> moved). This file holds only the DCM matrix evaluator content from #10 +> (27-governance-matrix). The redundant `governance-enforcement/matrix-evaluator.md` +> was not created — see report. + +--- + +## 1. Evaluation algorithm + +The Policy Manager evaluates the governance matrix at every interaction +boundary: provider dispatch, federation tunnel data transmission, +notification delivery, registration acceptance, and any cross-boundary +capability invocation. The evaluation runs the same algorithm against the +same rule set on every interaction. No parallel enforcement paths exist. + +``` +Interaction attempt arrives at boundary: + subject = { type, identity, accreditation_level, tenant } + data = { classification, resource_type, field_paths, capability } + target = { type, identity, sovereignty_zone, accreditation_held, trust_posture } + context = { profile, zero_trust_posture, federated, cross_jurisdiction, ... } + +Step 1: Collect matching rules + Load all active governance matrix rules across all tiers (system, platform, + tenant, resource_type, entity) + For each rule: evaluate the four axes against the interaction + Result: set of matching rules with decisions and enforcement levels + +Step 2: Evaluate hard constraints first + For each hard DENY rule that matches → DENY immediately; record rule_uuid; + no further evaluation + For each hard ALLOW rule that matches → record as hard allow candidate + If hard DENY exists → terminal decision = DENY + +Step 3: Evaluate soft constraints by domain precedence + Sort matching soft rules: entity > resource_type > tenant > platform > system + At each precedence level, most restrictive wins: + DENY > STRIP_FIELD > REDACT > ALLOW_WITH_CONDITIONS > AUDIT_ONLY > ALLOW + If DENY at any level → terminal decision = DENY + +Step 4: Evaluate conditions for ALLOW_WITH_CONDITIONS + For each ALLOW_WITH_CONDITIONS rule that survived Steps 2-3: + Evaluate all declared conditions + If any condition fails → downgrade to DENY + If all conditions pass → decision remains ALLOW_WITH_CONDITIONS + +Step 5: Apply field permissions + If terminal decision is ALLOW or ALLOW_WITH_CONDITIONS: + Apply field_permissions per governing rule: + allowlist mode: strip all fields not in allowed list + blocklist mode: strip all fields in blocked list + passthrough mode: all fields pass + For each stripped field: + If field is required → escalate to DENY_REQUEST + If field is optional → STRIP_FIELD (proceed without it) + +Step 6: Produce audit record + Record: interaction_uuid, all matching rules, terminal decision, + fields stripped or redacted, governing rule_uuid + Notification: if terminal decision is in rule's notification_on list + +Step 7: Enforce decision + ALLOW / ALLOW_WITH_CONDITIONS → interaction proceeds with permitted fields + DENY → interaction blocked; 403 response with governance_matrix_rule_uuid + STRIP_FIELD → interaction proceeds with stripped payload + REDACT → interaction proceeds with redacted field values + AUDIT_ONLY → interaction proceeds; flagged audit record written +``` + +--- + +## 2. Hard enforcement mechanics + +Hard rules (UDLM contract: `enforcement: hard`) cannot be relaxed by any +downstream rule at any domain level. DCM enforces this by: + +- At evaluation time, **hard DENY short-circuits** the entire rule set — + Steps 3-5 are skipped +- **No tenant-level, entity-level, or operator override** can permit an + interaction blocked by a hard DENY; the override is rejected at policy + contribution time by the Governance Matrix evaluator itself (the meta-rule + that consumer/tenant policies cannot modify system-domain hard rules) +- **Hard ALLOW** is rare and explicitly tracked; auditors can query for all + hard ALLOW rules to confirm none have been added inadvertently + +DCM ships pre-configured hard DENY rules for: + +- `sovereign` and `classified` data classifications crossing any boundary in + any profile (GMX-004) +- PHI without HIPAA BAA at federation boundaries (HIPAA compliance domain) +- Cross-jurisdiction transfer of restricted data in `fsi` profile + +These hard rules are activated automatically by the active profile and +compliance domain. The set is documented in the profile activation manifest. + +--- + +## 3. Soft enforcement execution + +Soft rules establish defaults that downstream (more specific) rules can +tighten — but never relax. DCM enforces this by: + +- Sorting matching rules by domain precedence: entity > resource_type > + tenant > platform > system +- At each level, computing the most restrictive decision across all matching + rules at that level (`DENY > STRIP_FIELD > REDACT > ALLOW_WITH_CONDITIONS + > AUDIT_ONLY > ALLOW`) +- Walking precedence levels from most-specific to least-specific; **a more + restrictive decision at any level wins**; a less restrictive decision at a + more specific level is rejected at policy contribution time (the rule never + becomes active) + +A soft DENY at the system level cannot be relaxed to ALLOW by a +tenant-level rule. The Governance Matrix evaluator detects the attempted +relaxation and rejects the policy contribution at submission time with +`GMX_SOFT_DENY_CANNOT_BE_RELAXED`. + +--- + +## 4. Caching and invalidation + +Policy evaluation is on the hot path for every interaction. DCM caches: + +- **Compiled rule sets** — the active matrix rule set is compiled to an + in-memory evaluation tree per Policy Manager instance; recompiled on rule + change events (`policy.activated`, `policy.deactivated`, + `policy.deprecated`) +- **Per-actor permission cache** — for a session, the actor's role/tenant + scope and matching subject-axis rules are cached for the session TTL + (PT15M–PT8H per profile) +- **Per-provider accreditation cache** — accreditation state for target + evaluations cached for `accreditation_cache_ttl` per profile (PT5M + standard, PT1M fsi/sovereign) + +**Invalidation triggers:** + +- Policy activation/deactivation event → recompile rule set; emit + `policy.cache_invalidated` to all Policy Manager instances +- Actor session revocation (via SES-001 model) → drop actor's permission cache +- Accreditation status change (from Accreditation Monitor) → drop provider's + accreditation cache entry +- Credential revocation event → permission cache entries referencing the + credential are dropped + +Cache invalidation propagates via PostgreSQL `LISTEN/NOTIFY` and completes +within PT5S in standard deployments. In `fsi`/`sovereign` profiles, +cache TTL is shortened to PT1M (or per-call evaluation in `sovereign`) to +limit stale-cache risk. + +--- + +## 5. Sovereignty zone management + +Sovereignty zones are first-class UDLM artifacts ( +[udlm/governance/governance-matrix.md](https://github.com/croadfeldt/udlm/blob/main/governance/governance-matrix.md)). +DCM operationalizes them via: + +- **Zone registry** — sovereignty zones stored as DCM artifacts with handle, + jurisdictions, regulatory frameworks, data residency guarantee, inter-zone + agreements +- **Resolution layer** — at policy evaluation, target sovereignty zone is + resolved from the target's registration (provider sovereignty declaration, + peer DCM zone, etc.) +- **Hard rule for sovereign data** — sovereign-classified data carries a + hard DENY for all federation and external-provider interactions in all + profiles including minimal; the rule is shipped pre-activated and cannot + be modified by tenant or platform policies + +### 5.1 Zone evaluation in placement + +When the Placement Manager runs Step 1 (sovereignty pre-filter), it queries +the sovereignty zone of every eligible provider and eliminates any provider +whose zone is not in the request's permitted zones list. This is a +pre-filter, not a tie-breaker — sovereignty failures eliminate providers +from the placement loop entirely. + +### 5.2 Inter-zone agreements + +A sovereignty zone may declare `inter_zone_agreements` — explicit data +transfer agreements with other zones (e.g., EU adequacy decision for +transfers within EU member states). The matrix evaluator consults +`inter_zone_agreements` when evaluating cross-zone interactions; transfers +permitted by an agreement are evaluated at the agreement's permitted +classification cap. + +--- + +## 6. Profile-governed policy configurations + +DCM ships pre-configured rule sets per profile. The rule set is activated +automatically when the profile is set: + +| Profile | Rule set characteristics | +|---|---| +| `minimal` | Hard DENY only for sovereign/classified; soft ALLOW for public/internal; permissive | +| `dev` | Inherits minimal; adds soft ALLOW_WITH_CONDITIONS for confidential to verified targets | +| `standard` | Inherits minimal; adds soft ALLOW_WITH_CONDITIONS for restricted (third_party accreditation required); soft DENY for PHI without HIPAA | +| `prod` | Inherits standard; tightens federation to verified peers only for confidential+; STRIP_FIELD for restricted in notifications | +| `fsi` | Inherits prod; hard DENY for cross-jurisdiction with regulated data; hard ALLOW_WITH_CONDITIONS for PHI requiring HIPAA BAA + verified + ZT full | +| `sovereign` | Inherits fsi; hard DENY for any sensitive data crossing DCM federation; hardware attestation required for any federation | + +When a compliance domain is active (HIPAA, GDPR, PCI-DSS, FedRAMP), its +matrix rules are automatically added to the active rule set. They compose +with profile rules — they do not replace them. + +See [`../governance-enforcement/policy-profiles.md`](../governance-enforcement/policy-profiles.md) +for the complete profile definitions. + +--- + +## 7. Where the evaluator runs + +The Policy Manager service hosts the matrix evaluator. It is invoked from: + +| Call site | When | +|---|---| +| Request Processor (during assembly) | Step 5 of nine-step assembly — evaluates all GateKeeper + Validation + Transformation + Governance Matrix rules against the assembled payload | +| Provider Dispatcher | Before provider dispatch — evaluates outbound governance matrix against the dispatch payload + target provider | +| Federation Tunnel | Before every cross-DCM message — evaluates outbound governance matrix against the message + remote DCM peer | +| Notification Router | Before every notification delivery — evaluates outbound governance matrix against the notification payload + destination | +| Webhook Subscription Resolver | When a webhook subscription is established — evaluates the matrix on the subscriber's authority for the subscribed event domain | +| Contribution Submission | When any contributor submits an artifact — evaluates contributor-permission matrix rules (see [`../governance-enforcement/contribution-pipeline.md`](../governance-enforcement/contribution-pipeline.md)) | + +Every invocation produces an audit record (GMX-005), regardless of outcome. + +--- + +## 8. Realization-specific notes + +- **OPA as the evaluation engine.** DCM uses OPA (Open Policy Agent) to + evaluate matrix rules; rules are translated to Rego at compile time. A peer + DCM realization could use a different engine while remaining UDLM-conformant. +- **PostgreSQL as the rule store.** Active rules live in the `policies` + table with status `active` and tier metadata. A peer could use a different + store. +- **`LISTEN/NOTIFY` for invalidation propagation.** A peer could use Kafka + or any other pub/sub. + +These are DCM implementation choices, not UDLM contracts. diff --git a/architecture/convergence-engine/recovery-and-retry.md b/architecture/convergence-engine/recovery-and-retry.md new file mode 100644 index 0000000..ab8f20a --- /dev/null +++ b/architecture/convergence-engine/recovery-and-retry.md @@ -0,0 +1,453 @@ +--- +Document Status: ✅ Stable — DCM implementation +Document Type: Architecture Reference — Recovery and Retry +Established: 2026-05-26 +Maps to: udlm/lifecycle/operational-models.md +--- + +# Convergence Engine — Recovery and Retry + +> **Implements contracts defined in UDLM**: +> [udlm/lifecycle/operational-models.md](https://github.com/croadfeldt/udlm/blob/main/lifecycle/operational-models.md). +> UDLM defines the timeout model and state machine, the cancellation +> propagation contract, the orphan-detection contract, the discovery +> scheduling contract, the recovery policy model, and the compensation +> contract. DCM operationalizes the timeout enforcement, cancellation +> execution, orphan detection, discovery job scheduling, recovery policy +> evaluation, and compensation execution. + +--- + +## 1. Timeout enforcement mechanisms + +UDLM defines three independent timeout scopes (assembly, dispatch, +reserve_query). DCM enforces each with per-step deadlines and dedicated +recovery triggers. + +### 1.1 Assembly timeout enforcement + +The Request Processor runs the nine-step assembly inside a budget governed +by `assembly_timeout` (profile-governed; see table below). The total budget +is allocated across steps as proportional fractions: + +| Step | Fraction of assembly_timeout | +|------|----------------------------| +| Layer Resolution | 20% | +| Layer Merge | 10% | +| Policy Evaluation (each) | 15% total; 5% per Mode 1/2; 30s per Internal policy; PT2M per External evaluator | +| Placement Engine Loop | 40% | +| Requested State Persistence | 10% | + +DCM enforces sub-step deadlines via `context.WithDeadline` (Go) or equivalent +in the Request Processor. A step exceeding its sub-deadline immediately fires +`ASSEMBLY_TIMEOUT` recovery trigger regardless of overall budget remaining. + +A external policy evaluation that exceeds PT2M per query causes +`ASSEMBLY_TIMEOUT` even if the overall assembly budget would tolerate it — +this prevents a single slow External Policy Evaluator from consuming the +entire assembly budget. + +| Profile | assembly_timeout default | +|---|---| +| minimal | PT5M | +| dev | PT5M | +| standard | PT3M | +| prod | PT2M | +| fsi | PT2M | +| sovereign | PT2M | + +### 1.2 Dispatch timeout enforcement + +The Request Orchestrator starts a dispatch deadline timer when it sends the +dispatch payload to the provider. If the provider has not posted a final +Realized State callback by the deadline, `DISPATCH_TIMEOUT` fires. + +| Profile | dispatch_timeout default | +|---|---| +| minimal | PT2H | +| dev | PT1H | +| standard | PT1H | +| prod | PT30M | +| fsi | PT30M | +| sovereign | PT30M | + +Resource-type overrides extend the timeout for legitimately long-running +provisioning (Compute.BareMetalServer: PT4H, Storage.LargeVolume: PT2H). +Overrides are declared in the Resource Type Specification. + +### 1.3 Reserve query timeout enforcement + +The Placement Manager queries each eligible provider's reserve endpoint in +parallel during the placement loop, with a short per-call deadline: + +| Profile | reserve_query_timeout default | +|---|---| +| minimal | PT30S | +| dev | PT30S | +| standard | PT10S | +| prod | PT5S | +| fsi | PT5S | +| sovereign | PT10S | + +A reserve query timeout does NOT immediately fire a recovery trigger — the +Placement Manager skips the timed-out provider and continues the placement +loop with remaining candidates. Only when ALL candidates have timed out or +been rejected does `RESERVE_QUERY_ALL_EXHAUSTED` fire. + +### 1.4 Timeout audit records + +Every timeout writes an audit record: + +```yaml +audit_record: + action: ASSEMBLY_TIMEOUT | DISPATCH_TIMEOUT | RESERVE_QUERY_TIMEOUT | + RESERVE_QUERY_ALL_EXHAUSTED + actor: + type: system + system_actor: + component: request_processor | request_orchestrator | placement_manager + trigger: timeout + entity_uuid: + details: + timeout_duration: + actual_elapsed: + step_at_timeout: + recovery_policy_triggered: +``` + +--- + +## 2. Cancellation execution and cleanup + +UDLM defines the three cancellation scenarios (pre-dispatch, post-dispatch +not-yet-started, mid-execution) and the propagation model. DCM enforces: + +### 2.1 Pre-dispatch cancellation (Scenario 1) + +Consumer submits `DELETE /api/v1/requests/{uuid}` while entity is in +pre-DISPATCHED state. DCM: + +1. Marks the Intent State record CANCELLED +2. Halts assembly immediately (cancels the in-flight Request Processor context) +3. Transitions entity to CANCELLED (terminal) +4. Writes `REQUEST_CANCELLED` audit record +5. No recovery policy triggered (clean cancel) +6. Returns `200 OK` with `{ "status": "CANCELLED" }` + +### 2.2 Post-dispatch, provider not yet started (Scenario 2) + +DCM sends cancellation payload to the provider's declared `cancellation_endpoint` +(if provider declares `supports_cancellation: true`). The provider acknowledges; +DCM transitions entity to CANCELLED. Response is `202 Accepted` while DCM +awaits provider confirmation; consumer polls status for the final CANCELLED. + +### 2.3 Mid-execution cancellation (Scenario 3) + +DCM consults `provider.supports_cancellation` and `provider.cancellation_supported_during`: + +- If provider supports cancellation during PROVISIONING: DCM sends the cancellation + payload; provider attempts rollback. Outcomes: + - Rollback clean → entity → CANCELLED (terminal) + - Rollback partial → fires `CANCELLATION_FAILED` recovery trigger + - No response → fires `CANCELLATION_FAILED` recovery trigger +- If provider does NOT support cancellation: entity enters `CANCEL_PENDING`; + DCM waits for provider to complete normally; on completion, fires + `LATE_RESPONSE_RECEIVED` (action typically `DISCARD_AND_REQUEUE` in + cancellation context) + +### 2.4 Cancellation payload + +```json +{ + "cancellation_uuid": "", + "entity_uuid": "", + "requested_state_uuid": "", + "reason": "consumer_requested | timeout | policy_triggered", + "requested_at": "", + "best_effort": true +} +``` + +`best_effort: true` is always set — DCM never guarantees cancellation success. + +--- + +## 3. Orphan detection implementation + +When cleanup cannot be guaranteed, DCM runs an orphan detection pass to find +provider resources with no corresponding DCM Realized State record. + +### 3.1 Triggers + +DCM fires orphan detection on: + +- Dispatch timeout with cancellation sent +- Cancellation failed +- Compensation failed +- `DISCARD_NO_REQUEUE` action taken +- Manual platform admin trigger + +### 3.2 Query implementation + +The Orphan Detection Service queries the provider's discovery endpoint with +narrow criteria: + +```yaml +orphan_detection_query: + provider_uuid: + time_window: + from: + to: + match_criteria: + resource_type: + characteristics: + name_pattern: + size_class: + tags: + exclude: + known_realized_state_uuids: [] +``` + +DCM compares discovery results against known Realized State entities; +unmatched provider-side entities are flagged as orphan candidates. + +### 3.3 Orphan candidate lifecycle + +Orphan candidates enter the platform admin review queue: + +```yaml +orphan_candidate: + orphan_candidate_uuid: + suspected_request_uuid: + provider_entity_id: + provider_uuid: + discovered_at: + characteristics: { ... } + status: under_review | confirmed_orphan | adopted | false_positive + resolution: + action: manual_decommission | adopt_into_dcm | mark_false_positive + resolved_by: + resolved_at: +``` + +Orphan candidates surface in the Platform Admin dashboard and generate a +notification with `urgency: high`. + +--- + +## 4. Discovery job scheduling and execution + +The Discovery Scheduler maintains a priority queue and dispatches discovery +jobs to provider discovery endpoints. UDLM defines the three trigger types +(scheduled / event / on-demand); DCM implements the queue and dispatcher. + +### 4.1 Priority queue + +``` +Priority order: + 1. Critical — COMPENSATION_FAILED orphan detection, sovereignty violation + 2. High — on-demand from platform admin, event-triggered (provider.degraded) + 3. Standard — event-triggered (entity.realized, drift.resolved) + 4. Background — scheduled discovery passes +``` + +Queue depth is bounded per profile. When the queue is full, new Background +items are dropped (with a log entry). Standard and above are never dropped — +they wait. + +### 4.2 Scheduled discovery + +Resource Type Specifications declare `discovery_schedule.default_interval`, +overridable per profile. The Discovery Scheduler runs each schedule via cron +(LISTEN/NOTIFY-based timer + work-stealing across Discovery Service replicas +for HA). + +### 4.3 Event-triggered discovery + +Specific DCM events automatically enqueue an out-of-cycle discovery: + +| Event | Delay | Scope | Reason | +|---|---|---|---| +| `entity.realized` | PT30S | this_entity | Confirm realization matches Requested State | +| `drift.resolved` | PT60S | this_entity | Confirm remediation took effect | +| `provider_update.approved` | PT30S | this_entity | Confirm provider update reflected | +| `provider.degraded` | PT0S | all_entities_on_provider | Assess impact | +| `TIMEOUT_PENDING` | PT5M | this_entity | Orphan detection after timeout | +| `COMPENSATION_FAILED` | PT0S | this_entity_and_dependents | Find orphans | + +### 4.4 On-demand discovery API + +``` +POST /api/v1/admin/discovery:trigger +{ + "scope": "entity | resource_type | provider | tenant", + "entity_uuid": "", + "resource_type": "", + "provider_uuid": "", + "tenant_uuid": "", + "reason": "incident investigation", + "priority": "high" +} +``` + +### 4.5 Discovery audit + +Every discovery cycle writes an audit record: + +```yaml +audit_record: + action: DISCOVERY_CYCLE_COMPLETED | DISCOVERY_CYCLE_FAILED + actor: + type: system + system_actor: + component: discovery_scheduler + trigger: scheduled | event_triggered | on_demand + trigger_event_uuid: + entity_uuid: + details: + entities_discovered: 47 + new_entities_found: 2 + duration: PT8S +``` + +--- + +## 5. Recovery policy evaluation + +UDLM defines Recovery Policies as a formal policy type alongside GateKeeper, +Validation, and Transformation. DCM evaluates Recovery Policies via the same +Policy Manager. The evaluation precedence is the same as all other policies: + +``` +1. Resource-type-level override (most specific) +2. Tenant-level override +3. Active profile's recovery posture group +4. System default (recovery-automated-reconciliation) + +First matching policy for the trigger condition wins. +Multiple recovery policies for the same trigger at the same domain level +→ policy conflict; CONFLICT_ERROR at ingestion; platform admin notified. +``` + +### 5.1 Built-in recovery profile groups + +DCM ships four built-in recovery profile groups (declared as Policy Groups +with `concern_type: recovery_posture`): + +| Group | Posture | +|---|---| +| `recovery-automated-reconciliation` | Let drift detection converge; default for minimal/dev/standard | +| `recovery-discard-and-requeue` | On ambiguity, clean up and start fresh — prioritize consistency | +| `recovery-notify-and-wait` | Never act automatically — always notify and wait for human; default for prod/fsi/sovereign | +| `recovery-aggressive-retry` | Retry everything before giving up | + +Profile bindings: + +```yaml +profile_recovery_defaults: + minimal: recovery-automated-reconciliation + dev: recovery-automated-reconciliation + standard: recovery-automated-reconciliation + prod: recovery-notify-and-wait + fsi: recovery-notify-and-wait + sovereign: recovery-notify-and-wait +``` + +Tenant and resource-type overrides are permitted; resource-type overrides +the most specific. + +### 5.2 NOTIFY_AND_WAIT consumer interface + +When a recovery policy fires `NOTIFY_AND_WAIT`, DCM sends a notification to +the entity owner with a time-bounded decision interface: + +``` +GET /api/v1/resources/{entity_uuid}/recovery-decisions +→ { + "recovery_decision_uuid": "", + "trigger": "DISPATCH_TIMEOUT", + "entity_uuid": "", + "deadline": "", + "available_actions": [ + { "action": "DRIFT_RECONCILE", "description": "..." }, + { "action": "DISCARD_AND_REQUEUE", "description": "..." }, + { "action": "DISCARD_NO_REQUEUE", "description": "..." } + ] + } + +POST /api/v1/resources/{entity_uuid}/recovery-decisions/{recovery_decision_uuid} +{ + "action": "DISCARD_AND_REQUEUE", + "reason": "Provider was known to be degraded at time of timeout" +} +``` + +If the deadline passes without resolution, the configured +`on_deadline_exceeded` action fires automatically. + +--- + +## 6. Compensation execution + +UDLM defines the composite service compensation contract (reverse-dependency +ordering, declarative per-component compensation behavior). DCM executes: + +### 6.1 Reverse-order execution + +When a composite service partially fails: + +``` +Successful so far: vm ✓, ip ✓ +Failed: dns ✗ (atomic) + +Compensation triggered: + Step 1: decommission vm (compensation_order: 3 → runs first in reverse) + Step 2: release ip allocation (compensation_order: 1 → runs second in reverse) + Compound entity → FAILED (terminal for this request cycle) +``` + +The Composite Service Orchestrator maintains the dependency graph from the +composite service spec and walks it in reverse for compensation. + +### 6.2 Compensation failure + +If a compensation step itself fails: + +1. Entity enters `COMPENSATION_FAILED` state +2. `COMPENSATION_FAILED` recovery trigger fires (default action: `ESCALATE`) +3. Orphan detection triggered immediately, scoped to provider + entity + characteristics +4. ORPHAN_CANDIDATE record created +5. Platform admin notified + +--- + +## 7. New lifecycle states (DCM-internal) + +DCM adds five lifecycle states to operationalize the UDLM recovery contracts: + +| State | Meaning | Recovery trigger | +|---|---|---| +| `TIMEOUT_PENDING` | Dispatch timeout fired; cancellation sent; awaiting outcome | `DISPATCH_TIMEOUT` | +| `LATE_REALIZATION_PENDING` | Provider responded after timeout; NOTIFY_AND_WAIT active | `LATE_RESPONSE_RECEIVED` | +| `INDETERMINATE_REALIZATION` | State ambiguous; drift detection resolving | — | +| `COMPENSATION_IN_PROGRESS` | Composite service rollback underway | — | +| `COMPENSATION_FAILED` | Rollback itself failed; orphaned resources possible | `COMPENSATION_FAILED` | + +These states are DCM-internal — they describe the recovery pipeline mechanics +and are mapped to UDLM lifecycle states (typically as sub-states of +PROVISIONING or FAILED) for external interop. + +--- + +## 8. Policy IDs + +DCM-side policy IDs governing recovery and retry execution: + +| Policy | Rule | +|---|---| +| `OPS-010-DCM` | DCM enforces assembly, dispatch, and reserve-query timeouts independently with profile-governed defaults and resource-type overrides | +| `OPS-011-DCM` | DCM cancellation is always best-effort; outcomes flow through Recovery Policy evaluation | +| `OPS-014-DCM` | DCM evaluates Recovery Policies via the same Policy Manager as all other policy types; same shadow mode, same audit | +| `OPS-017-DCM` | DCM runs composite compensation in reverse dependency order; compensation failure fires immediate orphan detection | +| `OPS-019-DCM` | DCM's NOTIFY_AND_WAIT actions carry a deadline; if exceeded, the configured `on_deadline_exceeded` action fires automatically | diff --git a/architecture/convergence-engine/scoring.md b/architecture/convergence-engine/scoring.md new file mode 100644 index 0000000..0c62d70 --- /dev/null +++ b/architecture/convergence-engine/scoring.md @@ -0,0 +1,535 @@ +--- +Document Status: ✅ Complete +Document Type: Architecture Reference — Scoring Model Specification +--- + +# DCM Data Model — Hybrid Scoring Model + +> **DCM-native scoring engine; no single UDLM contract counterpart.** +> The hybrid scoring model is a realization-layer extension of the Policy +> abstraction — UDLM defines no scoring contract. A peer DCM realization could +> use a different signal-weighting scheme and still satisfy every UDLM Policy +> and Governance Matrix contract. The Governance Matrix remains a pure boolean +> gate; scoring never applies to cross-boundary data decisions. + + +**Document Status:** ✅ Complete +**Document Type:** Architecture Reference — Scoring Model Specification +**Related Documents:** [Foundational Abstractions](https://github.com/croadfeldt/udlm/blob/main/foundations/foundations.md) | [Provider Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/provider-contract.md) | [Policy Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/policy-contract.md) | [Policy Profiles](../governance-enforcement/policy-profiles.md) | [Control Plane Components](../control-plane/components.md) | [Governance Matrix](https://github.com/croadfeldt/udlm/blob/main/governance/governance-matrix.md) | [Federated Contribution Model](https://github.com/croadfeldt/udlm/blob/main/governance/federated-contribution-model.md) + +> **This document maps to: DATA + POLICY** +> +> The Scoring Model is an extension of the Policy abstraction. Scored signals are Data artifacts with lifecycle and provenance. Profile thresholds are Policy-governed configuration. The Governance Matrix remains a pure boolean gate — scoring never applies to cross-boundary data decisions. +> See also: [Provider Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/provider-contract.md) | [Policy Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/policy-contract.md) +> > **See also:** [Authority Tier Model](https://github.com/croadfeldt/udlm/blob/main/governance/authority-tier-model.md) — the ordered authority tier list, custom tier definition, dynamic threshold format, and ATM system policies. + +> **Design Priority:** The Scoring Model is the primary mechanism for Priority 2 (ease of use) in service of Priority 1 (security). The auto-approval threshold (SMX-008: ≤ 50) and compliance-class GateKeepers are non-negotiable security properties. Profile thresholds and signal weights are the ease-of-use scaling mechanism. See [Design Priorities](https://github.com/croadfeldt/udlm/blob/main/design-principles/design-priorities.md). + +--- + +## 1. Purpose and Governing Principle + +DCM uses a **hybrid scoring model**: some decisions are boolean gates (facts), others are scored signals (degrees). The governing principle is explicit: + +> **Questions of fact use boolean gates. Questions of degree use scoring.** + +A secondary test for any ambiguous decision: +> **Can a regulator accept "the score was below threshold" as a complete explanation? If not, the decision must be boolean.** + +This document specifies the scoring half of the hybrid. For boolean decisions, see [Governance Matrix](https://github.com/croadfeldt/udlm/blob/main/governance/governance-matrix.md) and the compliance enforcement model in [Policy Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/policy-contract.md). + +### 1.1 What This Model Does + +The scoring model adds three capabilities to the existing architecture: + +1. **Operational GateKeeper policies** contribute a weighted risk score instead of producing a binary deny. The aggregate score drives approval routing. +2. **Advisory Validation policies** produce a completeness score and warning list without blocking the request. +3. **Five scoring signals** aggregate into a request risk score that determines approval routing tier — replacing the current per-policy approval flag with a continuous, profile-governed threshold system. + +### 1.2 What This Model Does Not Do + +The scoring model does **not**: +- Apply to Governance Matrix decisions — these remain boolean always +- Apply to compliance-class GateKeeper policies — PHI→BAA, sovereign data→sovereign provider remain hard gates +- Apply to authentication, authorization, or five-check boundary enforcement +- Apply to lifecycle state transitions +- Replace the Policy Engine — it is a function within it + +--- + +## 2. GateKeeper Enforcement Classes + +Every GateKeeper policy declares an `enforcement_class`. This is a required field in the Policy base contract (added in this document). + +```yaml +enforcement_class: compliance | operational +``` + +### 2.1 Compliance Class + +Behavior: **boolean gate**. A compliance-class GateKeeper that fires produces a `deny` decision. The request is halted immediately. No score is produced. + +**Use for:** +- Data classification boundary rules (PHI requires BAA accreditation) +- Sovereignty violations (classified data leaving declared zone) +- Security hard requirements (unencrypted data, expired certificates) +- Regulatory mandates with no legitimate override path +- Any rule where "score-around" creates legal or compliance liability + +```yaml +# Example compliance-class GateKeeper +policy_type: gatekeeper +enforcement_class: compliance +handle: "system/compliance/phi-baa-required" +match: + payload_type: request.layers_assembled + conditions: + - field: payload.data_classification + operator: contains + value: phi + - field: payload.provider.accreditations + operator: not_contains + value: baa_active +output: + decision: deny + reason: "PHI data requires provider with active BAA. Provider has no active BAA." + audit_required: true + notify_on: [DENY] +``` + +### 2.2 Operational Class + +Behavior: **risk score contribution**. An operational-class GateKeeper that fires contributes a weighted score to the request risk score. The request is not immediately halted. Instead the aggregate score determines routing. + +**Use for:** +- Cost ceiling policies (request cost exceeds Tenant recommendation) +- Resource sizing policies (CPU/memory above recommended maximums) +- Unusual timing or context (off-hours request, unusual field combinations) +- Quota pressure (Tenant approaching quota limit) +- Business rule preferences that should escalate review, not block + +```yaml +# Example operational-class GateKeeper +policy_type: gatekeeper +enforcement_class: operational +handle: "tenant/payments/gatekeeper/cost-ceiling" +scoring_weight: 35 # contribution to request risk score when fired +match: + payload_type: request.layers_assembled + conditions: + - field: payload.cost_estimate.per_month + operator: gt + value: 500 +output: + risk_score_contribution: 35 + reason: "Estimated monthly cost ${{payload.cost_estimate.per_month}} exceeds Tenant ceiling $500" + label: "cost_ceiling_exceeded" + audit_required: true +``` + +### 2.3 Profile-Level Enforcement Class Override + +Profiles can override the enforcement class of individual policies. This is the mechanism for making the scoring system tunable without touching individual policies. + +```yaml +# In a profile definition: +policy_enforcement_overrides: + - policy_handle: "tenant/payments/gatekeeper/cost-ceiling" + override_enforcement_class: compliance # escalate to hard gate in this profile + rationale: "FSI profile: all cost violations are hard gates" + + - policy_handle: "system/security/off-hours-request" + override_enforcement_class: operational # demote to soft score in dev profile + rationale: "Dev profile: off-hours requests are expected; score but don't block" +``` + +**Hard constraint:** A profile can **only** override `operational → compliance` or `compliance → operational` for explicitly non-regulatory policies. Policies with `regulatory_mandate: true` in their metadata cannot be demoted to operational by any profile. + +--- + +## 3. Validation Output Classes + +Every Validation policy declares an `output_class`. This is a required field. + +```yaml +output_class: structural | advisory +``` + +### 3.1 Structural Class + +Behavior: **boolean pass/fail**. A structural Validation that fails halts the request. No score is produced. + +**Use for:** +- Required field presence (missing required fields) +- Type correctness (wrong field type) +- Referential integrity (UUID references that don't resolve) +- Format validation (malformed handle, invalid semver) +- Schema conformance + +### 3.2 Advisory Class + +Behavior: **completeness score contribution + warning list**. An advisory Validation that fires contributes to the completeness score and adds a warning to the advisory_warnings list. The request is not halted. + +**Use for:** +- Recommended fields absent (cost_center not provided) +- Unusual values (memory_gb at 1 for a database VM — unusual but not invalid) +- Low-confidence field values (field sourced from a provider with confidence < 0.5) +- Naming convention violations (non-compliant resource name — advisory only) + +```yaml +policy_type: validation +output_class: advisory +handle: "platform/advisory/cost-center-recommended" +scoring_weight: 10 +match: + payload_type: request.layers_assembled +output: + completeness_contribution: 10 + warning_code: "recommended_field_absent" + warning_message: "cost_center not provided — cost attribution will use Tenant default" + field: "fields.cost_center" +``` + +--- + +## 4. The Five Scoring Signals + +The request risk score is assembled from five independent signals. Each signal is normalized to 0–100. The aggregate is a weighted sum, also normalized to 0–100. + +### 4.1 Signal 1 — Operational GateKeeper Score + +**Source:** All operational-class GateKeeper policies that fired during policy evaluation. +**Composition:** Sum of `risk_score_contribution` values from all fired operational GateKeepers. +**Normalization:** Capped at 100 before weighting. Multiple GateKeepers can fire; their contributions accumulate. +**Default weight in aggregate:** 0.45 + +```yaml +operational_gatekeeper_score: + fired_policies: + - handle: "tenant/payments/gatekeeper/cost-ceiling" + contribution: 35 + reason: "Cost $620/month exceeds ceiling $500" + - handle: "platform/gatekeeper/off-hours" + contribution: 15 + reason: "Request submitted outside business hours" + raw_score: 50 # sum of contributions + normalized: 50 # already within 0-100 +``` + +### 4.2 Signal 2 — Policy Completeness Score + +**Source:** All advisory-class Validation policies that fired. +**Composition:** Sum of `completeness_contribution` values from all fired advisory Validations. +**Normalization:** Capped at 100. Score represents "how incomplete is this request" — higher = more warnings. +**Default weight in aggregate:** 0.15 + +### 4.3 Signal 3 — Actor Risk History Score + +**Source:** Decay-weighted history of the actor's previous request outcomes. +**Composition:** Each historical event has a base score contribution and a time-decay multiplier. +**Decay model:** `contribution × e^(-λt)` where `t` is days since event, `λ` = 0.1 (half-life ≈ 7 days). +**Normalization:** 0–100. A clean history = 0. Recent consecutive failures approach 100. +**Default weight in aggregate:** 0.20 + +```yaml +# Events that contribute to actor risk history score +actor_risk_events: + - event: validation_failure # base_contribution: 5 + - event: gatekeeper_deny # base_contribution: 10 + - event: compliance_deny # base_contribution: 20 + - event: policy_override_requested # base_contribution: 8 + - event: drift_caused # base_contribution: 15 + - event: decommission_forced # base_contribution: 12 + - event: request_abandoned # base_contribution: 3 +``` + +**Privacy constraint:** Actor risk history scores are never exposed in consumer-facing API responses beyond the actor's own history. They are available in the Admin API for platform admins and in the audit trail. + +### 4.4 Signal 4 — Tenant Quota Pressure Score + +**Source:** Current quota utilization for the resource type being requested. +**Composition:** `max(0, (utilization_pct - free_threshold) / (1 - free_threshold)) × 100` +**Free threshold:** 0.75 (quota pressure score = 0 below 75% utilization). +**At 100% utilization:** quota pressure = 100, but the hard quota gate also fires (blocking the request regardless of score). +**Default weight in aggregate:** 0.10 + +```yaml +quota_pressure_score: + resource_type: "Compute.VirtualMachine" + current_usage: 87 + limit: 100 + utilization_pct: 0.87 + free_threshold: 0.75 + score: 48 # (0.87 - 0.75) / (1 - 0.75) × 100 = 48 +``` + +### 4.5 Signal 5 — Provider Accreditation Richness Score + +**Source:** Accreditation portfolio of the selected/candidate provider. +**Composition:** Weighted sum of accreditation types held, normalized against the maximum possible portfolio. +**Usage:** Used in placement tie-breaking (supplements existing tie-breaking algorithm). Also contributes inversely to request risk score — a richly accredited provider reduces risk. +**Default weight in aggregate:** 0.10 (inverse — higher richness = lower risk contribution) + +```yaml +accreditation_weights: + self_declared: 5 + third_party_audit: 15 + iso_27001: 20 + soc2_type2: 20 + fedramp_moderate: 30 + fedramp_high: 40 + hipaa_baa: 25 + pci_dss: 25 + sovereign_authorization: 50 + +# richness_score = sum(weights for held accreditations) / max_possible × 1 + +# Verification currency multipliers (applied per accreditation, see doc 47) +# Multiplier reduces an accreditation's weight contribution based on how recently +# it was externally verified by the Accreditation Monitor +verification_multipliers: + external_registry_verified_within_P1D: 1.0 # full weight — verified today + external_registry_verified_within_P7D: 0.9 + document_verified_within_P30D: 0.85 + contract_webhook_active: 0.9 + expiry_only_no_external_check: 0.7 # never been externally verified + verification_stale: 0.4 # check overdue + verification_failed_threshold_reached: 0.1 # Monitor cannot reach registry00 +# risk_contribution = (1 - richness_score/100) × 10 [lower richness = higher risk] +``` + +### 4.6 Aggregate Request Risk Score + +``` +request_risk_score = + (operational_gatekeeper_score × 0.45) + + (completeness_score × 0.15) + + (actor_risk_history_score × 0.20) + + (quota_pressure_score × 0.10) + + (provider_risk_contribution × 0.10) + +# Normalized: 0–100 +# 0 = clean request, no concerns +# 100 = maximum risk signal across all dimensions +``` + +Signal weights are profile-governed and can be adjusted per deployment. The weights above are the `standard` profile defaults. + +--- + +## 5. Profile-Governed Thresholds + +Every profile declares scoring thresholds that map the continuous risk score to a discrete approval routing decision. + +```yaml +# Approval routing uses named tier thresholds — see Authority Tier Model (doc 32) +# Tier names are resolved from the ordered authority tier list; numeric weights are derived. +scoring_thresholds: + approval_routing: + - tier: auto + max_score: 24 # score 0–24: auto-approve (SMX-008: never exceed 50) + - tier: reviewed + max_score: 59 # score 25–59: reviewed tier required + - tier: verified + max_score: 79 # score 60–79: verified tier required + - tier: authorized + max_score: 100 # score 80–100: authorized tier required + # Custom tiers (if defined) are inserted into this list; existing names unchanged. + # "authorized" means DCM holds the pipeline and notifies the declared DCMGroup; + # the review process and deliberation are the organization's responsibility. + # DCM records votes via Admin API; external systems (ServiceNow, Jira, Slack) + # may call the API on behalf of authorized group members. See [Design Priorities](https://github.com/croadfeldt/udlm/blob/main/design-principles/design-priorities.md). + # Note: compliance-class GateKeeper deny always halts regardless of score +``` + +### 5.1 Per-Profile Threshold Defaults + +| Profile | auto_approve | reviewed | verified | authorized | signal_weights | +|---------|-------------|-------------|--------------|-----------|----------------| +| `minimal` | < 45 | 45–74 | 75–100 | — | default | +| `dev` | < 40 | 40–69 | 70–100 | — | default | +| `standard` | < 25 | 25–59 | 60–79 | 80–100 | default | +| `prod` | < 15 | 15–49 | 50–74 | 75–100 | gatekeeper_weight: 0.50 | +| `fsi` | < 10 | 10–39 | 40–69 | 70–100 | gatekeeper_weight: 0.55, actor_weight: 0.25 | +| `sovereign` | < 5 | 5–29 | 30–59 | 60–100 | gatekeeper_weight: 0.60 | + +### 5.2 Resource-Type Threshold Overrides + +Profiles can declare tighter thresholds for specific resource types: + +```yaml +resource_type_threshold_overrides: + - resource_type: "Compute.VirtualMachine" + # tier: auto, max_score: 20 # use named-tier threshold format # tighter than profile default + - resource_type: "Network.VLAN" + # tier: auto, max_score: 10 # use named-tier threshold format # VLANs require more scrutiny + - resource_type: "Storage.Volume" + # tier: verified, max_score: 40 # use named-tier threshold format # storage changes escalate earlier +``` + +### 5.3 Tenant Threshold Overrides + +Platform admins can declare Tenant-level scoring threshold adjustments: + +```yaml +tenant_scoring_config: + tenant_uuid: + threshold_overrides: + # tier: auto, max_score: 15 # use named-tier threshold format # more conservative for this Tenant + signal_weight_overrides: + actor_risk_history_weight: 0.30 # higher actor scrutiny for this Tenant + trusted_actors: + - actor_uuid: + actor_risk_history_score_override: 0 # zero out risk history for trusted automation +``` + +### 5.4 Switching Between Scoring and Boolean Per Policy + +A profile can declare that a specific operational-class policy should behave as boolean (compliance-class) in that profile's context: + +```yaml +# In profile definition: +policy_enforcement_overrides: + - policy_handle: "platform/gatekeeper/cpu-size-limit" + override_enforcement_class: compliance + rationale: "Prod profile: CPU limit is a hard constraint, not a risk signal" + applies_to_resource_types: ["Compute.VirtualMachine"] +``` + +And conversely, a compliance-class policy that is **not** a regulatory mandate can be demoted to operational in lower-trust profiles: + +```yaml + - policy_handle: "platform/gatekeeper/naming-convention" + override_enforcement_class: operational + scoring_weight_override: 20 + rationale: "Dev profile: naming violations are warnings, not blocks" + requires_regulatory_mandate_false: true # safety check +``` + +--- + +## 6. Score Lifecycle and Audit Trail + +### 6.1 Score Record Structure + +Every scored evaluation produces a Score Record stored in the Audit Store alongside the standard audit record: + +```yaml +score_record: + score_record_uuid: + request_uuid: + entity_uuid: + evaluated_at: + + request_risk_score: 47 + routing_decision: reviewed + routing_threshold_applied: 25 # the threshold that triggered this tier + profile_uuid: + + signal_breakdown: + operational_gatekeeper: + score: 50 + weight: 0.45 + weighted_contribution: 22.5 + fired_policies: + - handle: "tenant/payments/gatekeeper/cost-ceiling" + contribution: 35 + - handle: "platform/gatekeeper/off-hours" + contribution: 15 + completeness: + score: 20 + weight: 0.15 + weighted_contribution: 3.0 + advisory_warnings: 2 + actor_risk_history: + score: 30 + weight: 0.20 + weighted_contribution: 6.0 + recent_events: 2 + quota_pressure: + score: 48 + weight: 0.10 + weighted_contribution: 4.8 + provider_risk: + score: 15 + weight: 0.10 + weighted_contribution: 1.5 + + compliance_gates_evaluated: 3 + compliance_gates_fired: 0 # if any > 0: request halted regardless of risk score +``` + +### 6.2 Score Immutability + +Score Records are immutable once written. Threshold changes do not retroactively alter historical Score Records. If thresholds change, requests evaluated before the change retain their original routing decisions in the audit trail. + +### 6.3 Human Override of Score-Based Routing + +A platform admin or reviewer can override a score-based routing decision with a recorded justification. The override is audited, but the Score Record is never modified — instead an Override Record is written referencing the original Score Record. + +--- + +## 7. Score Exposure in APIs + +### 7.1 Consumer-Facing Score Exposure + +Consumers receive a simplified score view: +- `risk_score` on request status (integer 0–100) +- `routing_decision` (auto_approved | pending_review | pending_verified | pending_authorized) +- `advisory_warnings` list from advisory Validation +- `score_drivers` — human-readable list of the top 3 contributing factors (no raw weights) + +Consumers **do not** receive: +- Actor risk history score breakdown (privacy) +- Signal weights +- Provider accreditation richness detail + +### 7.2 Platform Admin Score Exposure + +Platform admins receive full Score Record detail via the Admin API including all signal breakdowns, weights, and actor risk history detail. + +--- + +## 8. Relationship to Existing Decision Model + +The scoring model slots into the existing pipeline without replacing any component: + +``` +Policy Engine evaluation run: + 1. Evaluate all matching policies (existing behavior) + 2. Compliance-class GateKeeper fires → HALT (existing deny behavior) + 3. Structural Validation fails → HALT (existing fail behavior) + 4. Governance Matrix DENY fires → HALT (existing behavior, unchanged) + 5. NEW: Collect operational GateKeeper contributions → Signal 1 + 6. NEW: Collect advisory Validation contributions → Signal 2 + 7. NEW: Fetch actor risk history score → Signal 3 + 8. NEW: Calculate quota pressure score → Signal 4 + 9. NEW: Calculate provider accreditation richness → Signal 5 + 10. NEW: Aggregate → request_risk_score + 11. NEW: Apply profile thresholds → routing_decision + 12. NEW: Write Score Record to Audit Store + 13. Route request: auto_approve | queue_for_review | queue_dual | queue_authorized +``` + +Steps 2–4 handle standard policy evaluation. Steps 5–13 extend the model with scoring and approval routing. + +--- + +## 9. System Policies + +| Policy | Rule | +|--------|------| +| `SMX-001` | Every GateKeeper policy must declare `enforcement_class: compliance` or `enforcement_class: operational`. Policies without a declared enforcement_class are treated as compliance-class. | +| `SMX-002` | Every Validation policy must declare `output_class: structural` or `output_class: advisory`. Policies without a declared output_class are treated as structural. | +| `SMX-003` | Compliance-class GateKeeper policies with `regulatory_mandate: true` cannot be overridden to operational by any profile. This flag is set by platform admins and is audited. | +| `SMX-004` | The Governance Matrix is always boolean. No Governance Matrix Rule may declare a scoring weight or enforcement_class. | +| `SMX-005` | Signal weights in a profile must sum to 1.00. Profiles with invalid weight sums fail validation at activation time. | +| `SMX-006` | Score Records are immutable. Threshold changes do not retroactively alter historical Score Records. | +| `SMX-007` | Actor risk history scores are not exposed to consumers beyond the actor's own history. Platform admins have full access. | +| `SMX-008` | A profile's `auto_approve_below` threshold may not exceed 50. Auto-approving requests with risk scores above 50 is prohibited in all profiles. | +| `SMX-009` | Operational-class GateKeeper `scoring_weight` values must be declared between 1 and 100. Weights above 100 are validation errors. The aggregate of all fired policies is capped at 100 before weighting. | +| `SMX-010` | Score breakdown must be included in the audit trail for every request that receives a routing decision. A request with no Score Record is an audit integrity violation. | + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* From 9061f7988509523dcd2031f0cf0f7b2d1b8d8a88 Mon Sep 17 00:00:00 2001 From: Chris Roadfeldt Date: Sun, 28 Jun 2026 18:23:50 -0500 Subject: [PATCH 2/2] rename: GateKeeper policy -> Gating Policy (avoid OPA Gatekeeper collision) Sync from upstream croadfeldt/dcm. GateKeeper policy type -> Gating Policy (action gatekeep -> gate, op gatekeeping -> gating, enum gatekeeper -> gating). OPA Gatekeeper / Gatekeeper ConstraintTemplate references preserved. Files: architecture/convergence-engine/dependency-orchestration.md architecture/convergence-engine/overview.md architecture/convergence-engine/policy-evaluation.md architecture/convergence-engine/recovery-and-retry.md architecture/convergence-engine/scoring.md Co-Authored-By: Claude Opus 4.8 Signed-off-by: croadfeldt --- .../dependency-orchestration.md | 2 +- architecture/convergence-engine/overview.md | 2 +- .../convergence-engine/policy-evaluation.md | 2 +- .../convergence-engine/recovery-and-retry.md | 2 +- architecture/convergence-engine/scoring.md | 72 +++++++++---------- 5 files changed, 40 insertions(+), 40 deletions(-) diff --git a/architecture/convergence-engine/dependency-orchestration.md b/architecture/convergence-engine/dependency-orchestration.md index 41f0f91..b96c6dc 100644 --- a/architecture/convergence-engine/dependency-orchestration.md +++ b/architecture/convergence-engine/dependency-orchestration.md @@ -72,7 +72,7 @@ The Request Orchestrator parses the submission and: 4. **Validates field injection paths** — checks `from_field` paths against the dependency's resource type spec; per profile, validation may be advisory (warn), enforced (reject), or policy-gated (also pass through - GateKeeper) + Gating Policy) 5. **Allocates entity UUIDs** for each request (so the group response can return entity_uuid immediately) 6. **Resolves local refs** to actual entity UUIDs in the dependency graph diff --git a/architecture/convergence-engine/overview.md b/architecture/convergence-engine/overview.md index 4aa67ab..b45fc8b 100644 --- a/architecture/convergence-engine/overview.md +++ b/architecture/convergence-engine/overview.md @@ -52,7 +52,7 @@ intent until decommissioned. | Responsibility | How DCM fulfills it | |---|---| | Walk an entity from intent to realized | Request Orchestrator drives the nine-step pipeline; Request Processor performs assembly | -| Evaluate policy at every transition | Policy Manager evaluates GateKeeper / Validation / Transformation / Recovery / Orchestration Flow / Governance Matrix policies via OPA | +| Evaluate policy at every transition | Policy Manager evaluates Gating Policy / Validation / Transformation / Recovery / Orchestration Flow / Governance Matrix policies via OPA | | Select a provider for placement | Placement Manager runs the six-step placement algorithm: sovereignty pre-filter → eligibility filter → capability filter → reserve query → scoring → tie-break | | Dispatch with scoped credentials | API Gateway requests a `dcm_interaction` credential from the Credential Provider, scoped to the specific provider + entity + operation, valid for PT15M–PT1H per profile | | React to provider events | Provider callbacks land at the Provider Callback API; the Request Orchestrator routes to Realized State persistence and event emission | diff --git a/architecture/convergence-engine/policy-evaluation.md b/architecture/convergence-engine/policy-evaluation.md index 94259de..d7d7e25 100644 --- a/architecture/convergence-engine/policy-evaluation.md +++ b/architecture/convergence-engine/policy-evaluation.md @@ -237,7 +237,7 @@ The Policy Manager service hosts the matrix evaluator. It is invoked from: | Call site | When | |---|---| -| Request Processor (during assembly) | Step 5 of nine-step assembly — evaluates all GateKeeper + Validation + Transformation + Governance Matrix rules against the assembled payload | +| Request Processor (during assembly) | Step 5 of nine-step assembly — evaluates all Gating Policy + Validation + Transformation + Governance Matrix rules against the assembled payload | | Provider Dispatcher | Before provider dispatch — evaluates outbound governance matrix against the dispatch payload + target provider | | Federation Tunnel | Before every cross-DCM message — evaluates outbound governance matrix against the message + remote DCM peer | | Notification Router | Before every notification delivery — evaluates outbound governance matrix against the notification payload + destination | diff --git a/architecture/convergence-engine/recovery-and-retry.md b/architecture/convergence-engine/recovery-and-retry.md index ab8f20a..1939cd0 100644 --- a/architecture/convergence-engine/recovery-and-retry.md +++ b/architecture/convergence-engine/recovery-and-retry.md @@ -314,7 +314,7 @@ audit_record: ## 5. Recovery policy evaluation -UDLM defines Recovery Policies as a formal policy type alongside GateKeeper, +UDLM defines Recovery Policies as a formal policy type alongside Gating Policy, Validation, and Transformation. DCM evaluates Recovery Policies via the same Policy Manager. The evaluation precedence is the same as all other policies: diff --git a/architecture/convergence-engine/scoring.md b/architecture/convergence-engine/scoring.md index 0c62d70..ecfde1b 100644 --- a/architecture/convergence-engine/scoring.md +++ b/architecture/convergence-engine/scoring.md @@ -23,7 +23,7 @@ Document Type: Architecture Reference — Scoring Model Specification > See also: [Provider Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/provider-contract.md) | [Policy Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/policy-contract.md) > > **See also:** [Authority Tier Model](https://github.com/croadfeldt/udlm/blob/main/governance/authority-tier-model.md) — the ordered authority tier list, custom tier definition, dynamic threshold format, and ATM system policies. -> **Design Priority:** The Scoring Model is the primary mechanism for Priority 2 (ease of use) in service of Priority 1 (security). The auto-approval threshold (SMX-008: ≤ 50) and compliance-class GateKeepers are non-negotiable security properties. Profile thresholds and signal weights are the ease-of-use scaling mechanism. See [Design Priorities](https://github.com/croadfeldt/udlm/blob/main/design-principles/design-priorities.md). +> **Design Priority:** The Scoring Model is the primary mechanism for Priority 2 (ease of use) in service of Priority 1 (security). The auto-approval threshold (SMX-008: ≤ 50) and compliance-class Gating Policies are non-negotiable security properties. Profile thresholds and signal weights are the ease-of-use scaling mechanism. See [Design Priorities](https://github.com/croadfeldt/udlm/blob/main/design-principles/design-priorities.md). --- @@ -42,7 +42,7 @@ This document specifies the scoring half of the hybrid. For boolean decisions, s The scoring model adds three capabilities to the existing architecture: -1. **Operational GateKeeper policies** contribute a weighted risk score instead of producing a binary deny. The aggregate score drives approval routing. +1. **Operational Gating policies** contribute a weighted risk score instead of producing a binary deny. The aggregate score drives approval routing. 2. **Advisory Validation policies** produce a completeness score and warning list without blocking the request. 3. **Five scoring signals** aggregate into a request risk score that determines approval routing tier — replacing the current per-policy approval flag with a continuous, profile-governed threshold system. @@ -50,16 +50,16 @@ The scoring model adds three capabilities to the existing architecture: The scoring model does **not**: - Apply to Governance Matrix decisions — these remain boolean always -- Apply to compliance-class GateKeeper policies — PHI→BAA, sovereign data→sovereign provider remain hard gates +- Apply to compliance-class Gating policies — PHI→BAA, sovereign data→sovereign provider remain hard gates - Apply to authentication, authorization, or five-check boundary enforcement - Apply to lifecycle state transitions - Replace the Policy Engine — it is a function within it --- -## 2. GateKeeper Enforcement Classes +## 2. Gating Policy Enforcement Classes -Every GateKeeper policy declares an `enforcement_class`. This is a required field in the Policy base contract (added in this document). +Every Gating policy declares an `enforcement_class`. This is a required field in the Policy base contract (added in this document). ```yaml enforcement_class: compliance | operational @@ -67,7 +67,7 @@ enforcement_class: compliance | operational ### 2.1 Compliance Class -Behavior: **boolean gate**. A compliance-class GateKeeper that fires produces a `deny` decision. The request is halted immediately. No score is produced. +Behavior: **boolean gate**. A compliance-class Gating Policy that fires produces a `deny` decision. The request is halted immediately. No score is produced. **Use for:** - Data classification boundary rules (PHI requires BAA accreditation) @@ -77,8 +77,8 @@ Behavior: **boolean gate**. A compliance-class GateKeeper that fires produces a - Any rule where "score-around" creates legal or compliance liability ```yaml -# Example compliance-class GateKeeper -policy_type: gatekeeper +# Example compliance-class Gating Policy +policy_type: gating enforcement_class: compliance handle: "system/compliance/phi-baa-required" match: @@ -99,7 +99,7 @@ output: ### 2.2 Operational Class -Behavior: **risk score contribution**. An operational-class GateKeeper that fires contributes a weighted score to the request risk score. The request is not immediately halted. Instead the aggregate score determines routing. +Behavior: **risk score contribution**. An operational-class Gating Policy that fires contributes a weighted score to the request risk score. The request is not immediately halted. Instead the aggregate score determines routing. **Use for:** - Cost ceiling policies (request cost exceeds Tenant recommendation) @@ -109,10 +109,10 @@ Behavior: **risk score contribution**. An operational-class GateKeeper that fire - Business rule preferences that should escalate review, not block ```yaml -# Example operational-class GateKeeper -policy_type: gatekeeper +# Example operational-class Gating Policy +policy_type: gating enforcement_class: operational -handle: "tenant/payments/gatekeeper/cost-ceiling" +handle: "tenant/payments/gating/cost-ceiling" scoring_weight: 35 # contribution to request risk score when fired match: payload_type: request.layers_assembled @@ -134,7 +134,7 @@ Profiles can override the enforcement class of individual policies. This is the ```yaml # In a profile definition: policy_enforcement_overrides: - - policy_handle: "tenant/payments/gatekeeper/cost-ceiling" + - policy_handle: "tenant/payments/gating/cost-ceiling" override_enforcement_class: compliance # escalate to hard gate in this profile rationale: "FSI profile: all cost violations are hard gates" @@ -196,20 +196,20 @@ output: The request risk score is assembled from five independent signals. Each signal is normalized to 0–100. The aggregate is a weighted sum, also normalized to 0–100. -### 4.1 Signal 1 — Operational GateKeeper Score +### 4.1 Signal 1 — Operational Gating Policy Score -**Source:** All operational-class GateKeeper policies that fired during policy evaluation. -**Composition:** Sum of `risk_score_contribution` values from all fired operational GateKeepers. -**Normalization:** Capped at 100 before weighting. Multiple GateKeepers can fire; their contributions accumulate. +**Source:** All operational-class Gating policies that fired during policy evaluation. +**Composition:** Sum of `risk_score_contribution` values from all fired operational Gating Policies. +**Normalization:** Capped at 100 before weighting. Multiple Gating Policies can fire; their contributions accumulate. **Default weight in aggregate:** 0.45 ```yaml -operational_gatekeeper_score: +operational_gating_score: fired_policies: - - handle: "tenant/payments/gatekeeper/cost-ceiling" + - handle: "tenant/payments/gating/cost-ceiling" contribution: 35 reason: "Cost $620/month exceeds ceiling $500" - - handle: "platform/gatekeeper/off-hours" + - handle: "platform/gating/off-hours" contribution: 15 reason: "Request submitted outside business hours" raw_score: 50 # sum of contributions @@ -235,7 +235,7 @@ operational_gatekeeper_score: # Events that contribute to actor risk history score actor_risk_events: - event: validation_failure # base_contribution: 5 - - event: gatekeeper_deny # base_contribution: 10 + - event: gating_deny # base_contribution: 10 - event: compliance_deny # base_contribution: 20 - event: policy_override_requested # base_contribution: 8 - event: drift_caused # base_contribution: 15 @@ -302,7 +302,7 @@ verification_multipliers: ``` request_risk_score = - (operational_gatekeeper_score × 0.45) + + (operational_gating_score × 0.45) + (completeness_score × 0.15) + (actor_risk_history_score × 0.20) + (quota_pressure_score × 0.10) + @@ -339,7 +339,7 @@ scoring_thresholds: # the review process and deliberation are the organization's responsibility. # DCM records votes via Admin API; external systems (ServiceNow, Jira, Slack) # may call the API on behalf of authorized group members. See [Design Priorities](https://github.com/croadfeldt/udlm/blob/main/design-principles/design-priorities.md). - # Note: compliance-class GateKeeper deny always halts regardless of score + # Note: compliance-class Gating Policy deny always halts regardless of score ``` ### 5.1 Per-Profile Threshold Defaults @@ -349,9 +349,9 @@ scoring_thresholds: | `minimal` | < 45 | 45–74 | 75–100 | — | default | | `dev` | < 40 | 40–69 | 70–100 | — | default | | `standard` | < 25 | 25–59 | 60–79 | 80–100 | default | -| `prod` | < 15 | 15–49 | 50–74 | 75–100 | gatekeeper_weight: 0.50 | -| `fsi` | < 10 | 10–39 | 40–69 | 70–100 | gatekeeper_weight: 0.55, actor_weight: 0.25 | -| `sovereign` | < 5 | 5–29 | 30–59 | 60–100 | gatekeeper_weight: 0.60 | +| `prod` | < 15 | 15–49 | 50–74 | 75–100 | gating_weight: 0.50 | +| `fsi` | < 10 | 10–39 | 40–69 | 70–100 | gating_weight: 0.55, actor_weight: 0.25 | +| `sovereign` | < 5 | 5–29 | 30–59 | 60–100 | gating_weight: 0.60 | ### 5.2 Resource-Type Threshold Overrides @@ -390,7 +390,7 @@ A profile can declare that a specific operational-class policy should behave as ```yaml # In profile definition: policy_enforcement_overrides: - - policy_handle: "platform/gatekeeper/cpu-size-limit" + - policy_handle: "platform/gating/cpu-size-limit" override_enforcement_class: compliance rationale: "Prod profile: CPU limit is a hard constraint, not a risk signal" applies_to_resource_types: ["Compute.VirtualMachine"] @@ -399,7 +399,7 @@ policy_enforcement_overrides: And conversely, a compliance-class policy that is **not** a regulatory mandate can be demoted to operational in lower-trust profiles: ```yaml - - policy_handle: "platform/gatekeeper/naming-convention" + - policy_handle: "platform/gating/naming-convention" override_enforcement_class: operational scoring_weight_override: 20 rationale: "Dev profile: naming violations are warnings, not blocks" @@ -427,14 +427,14 @@ score_record: profile_uuid: signal_breakdown: - operational_gatekeeper: + operational_gating: score: 50 weight: 0.45 weighted_contribution: 22.5 fired_policies: - - handle: "tenant/payments/gatekeeper/cost-ceiling" + - handle: "tenant/payments/gating/cost-ceiling" contribution: 35 - - handle: "platform/gatekeeper/off-hours" + - handle: "platform/gating/off-hours" contribution: 15 completeness: score: 20 @@ -497,10 +497,10 @@ The scoring model slots into the existing pipeline without replacing any compone ``` Policy Engine evaluation run: 1. Evaluate all matching policies (existing behavior) - 2. Compliance-class GateKeeper fires → HALT (existing deny behavior) + 2. Compliance-class Gating Policy fires → HALT (existing deny behavior) 3. Structural Validation fails → HALT (existing fail behavior) 4. Governance Matrix DENY fires → HALT (existing behavior, unchanged) - 5. NEW: Collect operational GateKeeper contributions → Signal 1 + 5. NEW: Collect operational Gating Policy contributions → Signal 1 6. NEW: Collect advisory Validation contributions → Signal 2 7. NEW: Fetch actor risk history score → Signal 3 8. NEW: Calculate quota pressure score → Signal 4 @@ -519,15 +519,15 @@ Steps 2–4 handle standard policy evaluation. Steps 5–13 extend the model wit | Policy | Rule | |--------|------| -| `SMX-001` | Every GateKeeper policy must declare `enforcement_class: compliance` or `enforcement_class: operational`. Policies without a declared enforcement_class are treated as compliance-class. | +| `SMX-001` | Every Gating policy must declare `enforcement_class: compliance` or `enforcement_class: operational`. Policies without a declared enforcement_class are treated as compliance-class. | | `SMX-002` | Every Validation policy must declare `output_class: structural` or `output_class: advisory`. Policies without a declared output_class are treated as structural. | -| `SMX-003` | Compliance-class GateKeeper policies with `regulatory_mandate: true` cannot be overridden to operational by any profile. This flag is set by platform admins and is audited. | +| `SMX-003` | Compliance-class Gating policies with `regulatory_mandate: true` cannot be overridden to operational by any profile. This flag is set by platform admins and is audited. | | `SMX-004` | The Governance Matrix is always boolean. No Governance Matrix Rule may declare a scoring weight or enforcement_class. | | `SMX-005` | Signal weights in a profile must sum to 1.00. Profiles with invalid weight sums fail validation at activation time. | | `SMX-006` | Score Records are immutable. Threshold changes do not retroactively alter historical Score Records. | | `SMX-007` | Actor risk history scores are not exposed to consumers beyond the actor's own history. Platform admins have full access. | | `SMX-008` | A profile's `auto_approve_below` threshold may not exceed 50. Auto-approving requests with risk scores above 50 is prohibited in all profiles. | -| `SMX-009` | Operational-class GateKeeper `scoring_weight` values must be declared between 1 and 100. Weights above 100 are validation errors. The aggregate of all fired policies is capped at 100 before weighting. | +| `SMX-009` | Operational-class Gating Policy `scoring_weight` values must be declared between 1 and 100. Weights above 100 are validation errors. The aggregate of all fired policies is capped at 100 before weighting. | | `SMX-010` | Score breakdown must be included in the audit trail for every request that receives a routing decision. A request with no Score Record is an audit integrity violation. | ---