From b07855e1fdbcea0289b919454c43a130fba53c87 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 11:35:36 -0500 Subject: [PATCH 01/25] docs: propose internal DNS component architecture --- docs/architecture/README.md | 3 + docs/architecture/internal-dns/README.md | 246 +++++++++++++++++++++++ 2 files changed, 249 insertions(+) create mode 100644 docs/architecture/internal-dns/README.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 48eea0aa..b3a2d6b7 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -161,6 +161,9 @@ See the [API Reference](./api-reference.md) for complete field documentation. ## Learn More +- [Internal DNS proposal](./internal-dns/README.md) — Proposed private DNS + control plane, publication contracts, and shared serving architecture + - [Record Ownership](./record-ownership.md) — Who owns a name in a zone, and when that changes - [Conditions and Reasons](./conditions.md) — What each status reason means, and diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md new file mode 100644 index 00000000..29957e2a --- /dev/null +++ b/docs/architecture/internal-dns/README.md @@ -0,0 +1,246 @@ +# Internal DNS component design + +Status: Proposed. These contracts describe the local prototype and require API +and serving qualification before release. + +The [product enhancement](https://github.com/datum-cloud/enhancements/pull/922) +defines the consumer experience. The [cross-service overview](https://github.com/datum-cloud/enhancements/blob/docs/internal-dns-product-proposal/architecture/deliver/dns/internal-dns/README.md) +defines integration boundaries. This document owns the DNS control plane, +publication contracts, tenant isolation, and shared serving configuration. + +## Control planes + +The DNS service runs in its own VPC and serves many consumer networks through a +shared fleet. The following diagram shows API boundaries and update paths. +Controllers can run in a management cluster while writing to these APIs. + +```mermaid +flowchart TB + P[Consumer project API] + D[DNS service project API] + K[Karmada API] + E[Edge API + DNS fleet] + T[DNS publication transport] + P -->|DNS reconciliation| D + P -->|Network and workload intent| K + K -->|Placed desired state| E + D -->|Committed updates| T + T -->|DNS records and configuration| E +``` + +DNS acknowledgments return to the service project. Network and workload +observations return through Karmada. Owning controllers project status into the +consumer project. + +| Control plane | Resources and ownership | +| --- | --- | +| Consumer project | Private zones, records, associations, managed naming, registrations, grants, and contributions. Trusted networking integration manages `DNSResolverContext` and `DNSResolverAccessBinding`; DNS writes their status. Product services publish records. | +| DNS service project | DNS-owned publication ownership, manifests, chunks, outboxes, internal `DNSResolverBinding` assignments, and allocation claims. Also holds the service's own network intent. | +| Karmada | Network and workload projections, location placement, propagation policies, and selected edge observations. Networking and product controllers own these projections. | +| Edge | Local network contexts, interfaces, VPCs, attachments, `ServiceEndpoint` and `ServiceRoutePolicy` resources, shared serving workloads, and checkpoints. Edge controllers and serving agents own local state. | + +DNS coordination retains each consumer's trusted project and source identities. +Controller replicas share authoritative storage for each publication and +serving-plan ownership domain. Product publishers cannot write that storage or +grant resolver access. + +Networking `NetworkContext` describes a network at a location; +`DNSResolverContext` defines its DNS scope. Resolver settings needed at the edge +must travel as desired state in the networking projection. The existing +propagation contract does not carry source status. DNS publications follow their +own delivery path. + +## Contexts and regional access + +One logical network has one DNS context across its locations. That context +selects its managed namespace and associated private zones. Separate contexts +can use overlapping names and addresses. Zone sharing requires explicit +associations. + +```mermaid +flowchart TB + C["DNSResolverContext
Managed namespace + private zones"] + C --> AC["DNSResolverAccessBinding
Central"] + C --> AE["DNSResolverAccessBinding
East"] + subgraph Central[Central region] + WC[Workloads] --> GC[Galactic private endpoint] + GC --> FC[Shared DNS fleet] + end + subgraph East[East region] + WE[Workloads] --> GE[Galactic private endpoint] + GE --> FE[Shared DNS fleet] + end + AC -.->|Authorizes context access| FC + AE -.->|Authorizes context access| FE +``` + +Access bindings live in the consumer project API. Each has its own serving +target, destination, renewal, deadline, and readiness. Expiring East access +leaves Central access and the context intact. Network or context recreation +receives a new lifetime identity; stale updates cannot restore old access. + +The current `region` field identifies the serving target because the project +API can hold access for several regions. Trusted networking integration supplies +it. A future location reference could determine placement instead. Regional +access does not change zone contents or choose nearby application endpoints. +Region-to-cell mapping, resolver failover, and discovery locality remain open. + +DNS treats the context's consumer identity as opaque. Networking owns its mapping +to logical networks and regional VPCs. DNS does not discover VPCs or attachments. +Adding a VPC adds data and authorization to the shared fleet. + +## Context and access contracts + +Trusted networking integration writes context and access specs in the consumer +project API. DNS writes status and issues writer epochs. Product publishers +cannot create contexts or renew access. DNS does not read VPC or attachment APIs. + +The following proposed manifests show the integration contract. Names, UIDs, +addresses, and deadlines are illustrative; the resources are not in the released +API bundle. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverContext +metadata: + name: application-network-dns + namespace: project-a +spec: + # Immutable logical consumer lifetime; opaque to DNS. + consumerID: "22222222-2222-4222-8222-222222222222" + managedNamespace: + enabled: true +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverAccessBinding +metadata: + name: application-network-dns-central + namespace: project-a +spec: + contextRef: + name: application-network-dns + # API-assigned context UID prevents access surviving context recreation. + uid: "33333333-3333-4333-8333-333333333333" + # Serving target; not a new zone scope or discovery-locality policy. + region: us-central-1 + queryIdentity: + type: DestinationAddress + # Destination received by DNS after Galactic frontend translation. + value: "fd70:100::10" + port: 53 + transports: [UDP, TCP] + authorization: + # DNS-issued epoch and increasing writer sequence fence stale renewals. + writerEpoch: 3 + sequence: 27 + validUntil: "2026-10-08T12:05:00Z" +``` + +`consumerID` identifies the logical consumer independently of provider VPC UIDs. +The context UID fences DNS state. `queryIdentity` binds one authorized serving +destination to that context. The integration renews authorization before +`validUntil`; DNS serving agents enforce expiration locally. The managed suffix +and DNS-issued epoch are exposed through context status. + +`DNSZoneAssociation` and `DNSNamingPolicy` use UID-pinned `resolverContextRef` +references. `DNSResolverBinding` is a service-owned fleet plan. The prototype +retains legacy `vpcUID` wire fields for context UIDs; this compatibility does not +permit DNS to discover networking resources. + +## Distributed publication + +Product publishers reserve names with `DNSRegistration`, receive scoped +`DNSContributionGrant` authorization, and publish typed records with +`DNSRecordContribution`. Contributions report eligibility, writer epoch, +sequence, and validity deadline. Private static records use `DNSRecordSet`. + +```mermaid +flowchart TB + P[Project DNS APIs] --> C[Validate and compile] + C --> S[DNS service project: ownership and committed publications] + S --> O[Durable outbox exporter] + O --> N[NATS JetStream] + N --> A[Regional authoritative materializers] + N --> R[Resolver configuration agents] + A --> V[Verified revision acknowledgments] + R --> V + V --> S + S --> T[Project status projection] +``` + +API storage holds durable ownership, manifests, chunks, outboxes, and serving +assignments. Conditional writes and writer epochs fence concurrent controllers. +Exporters retry committed updates; regional agents reject stale generations and +preserve expiration deadlines across replay and restart. The design does not +require PostgreSQL. NATS transports DNS intent; it is not queried to answer DNS. + +Authoritative materializers apply accepted publications to private PowerDNS +replicas. Resolver agents apply destination mappings, views, and shared-pool +membership. Acknowledgments report verified applied revisions. A broker +acknowledgment alone does not establish query readiness. + +## Shared serving and isolation + +The serving candidate uses dnsdist, shared BIND resolver processes with isolated +views, and PowerDNS Authoritative for private zone variants. Node and regional +tiers use shared processes; adding a context adds configuration and data. +Resolver choice and capacity remain subject to qualification. + +```mermaid +flowchart LR + G[Authorized Galactic destination] --> D[Shared node dnsdist] + D --> B[Shared node BIND: isolated context views] + B --> R[Shared regional dnsdist] + R --> C[Shared regional BIND: isolated context views] + C -->|Private zones| A[Shared PowerDNS Authoritative: zone variants] + C -->|Public recursion| P[Public authoritative DNS] +``` + +The destination selects the context before any cache lookup. dnsdist forwards +trusted original-destination metadata to private resolver listeners; each BIND +view forwards through its assigned regional destination. Service-owned source +markers select authoritative variants. Consumer DNS metadata and source address +ranges do not select views. Backend listeners accept only authorized service +traffic. Unknown mappings fail closed; answers and caches remain isolated. + +A shared-pool replica becomes eligible only after it has every assigned context +at the required configuration revision. Failover stays within the same context +and serving assignment. The [Private Service Connect design](https://github.com/datum-cloud/galactic/blob/docs/private-service-connect-architecture/docs/enhancements/networking/private-service-connect/README.md) owns +attachment authorization, destination translation, and the return path. DNS +owns listener configuration and context selection. + +## Publication and service discovery + +Product services such as Compute and Connect publish addresses and endpoint +eligibility through their project DNS APIs. DNS validates ownership, compiles +records, and distributes them to serving locations. Zone owners manage custom +records. Record publication and workload resolver setup are separate paths. + +Service discovery follows this lifecycle: + +```mermaid +stateDiagram-v2 + [*] --> Pending + Pending --> Serving: Fresh eligible publication applied + Serving --> Withdrawn: Unhealthy, unavailable, or expired + Withdrawn --> Serving: Fresh eligible publication applied + Pending --> Deleted: Resource deleted + Serving --> Deleted: Resource deleted + Withdrawn --> Deleted: Resource deleted + Deleted --> [*] +``` + +Instance identity names follow instance and address lifecycle; application +health does not automatically remove them. Services with stable addresses manage +backend health themselves. Connector exports publish selected reachable +services. Custom records do not receive automatic health checks. + +Serving uses applied state without consulting a product control plane per query. +Lifetime and ordering checks prevent delayed updates from restoring deleted +records or replacing newer state. During an outage, records and access remain +usable only until their validity deadlines. Discovery names with no eligible +endpoints return no endpoint addresses. Unsafe private resolution fails. + +Cached answers can persist until their DNS time to live (TTL) expires. +Publication, withdrawal, freshness, and access targets remain release decisions. +Product status distinguishes record readiness from resolver access readiness. From b3892cc9ccb56c485fde512bc35056202a5110b3 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 15:28:42 -0500 Subject: [PATCH 02/25] docs: document internal DNS architecture and API contracts --- docs/architecture/README.md | 5 +- docs/architecture/internal-dns/README.md | 407 ++++++++--------- docs/architecture/internal-dns/api-design.md | 449 +++++++++++++++++++ 3 files changed, 648 insertions(+), 213 deletions(-) create mode 100644 docs/architecture/internal-dns/api-design.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index b3a2d6b7..73ce0347 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -161,9 +161,10 @@ See the [API Reference](./api-reference.md) for complete field documentation. ## Learn More -- [Internal DNS proposal](./internal-dns/README.md) — Proposed private DNS +- [Internal DNS architecture](./internal-dns/README.md) — Proposed private DNS control plane, publication contracts, and shared serving architecture - +- [Internal DNS API design](./internal-dns/api-design.md) — Annotated YAML for + contexts, regional access, private zones, and product publication - [Record Ownership](./record-ownership.md) — Who owns a name in a zone, and when that changes - [Conditions and Reasons](./conditions.md) — What each status reason means, and diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 29957e2a..f9456fe1 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -1,246 +1,231 @@ -# Internal DNS component design +# Internal DNS architecture -Status: Proposed. These contracts describe the local prototype and require API -and serving qualification before release. +Status: Proposed. The API contracts and serving components require qualification +before release. See [API design](api-design.md) for annotated YAML examples. + +Internal DNS lets resources resolve private names within a VPC. One DNS context +selects the names visible to that network, including an automatically managed +namespace and optional custom zones. Many contexts share the same serving fleet; +adding a VPC does not create a resolver deployment. The [product enhancement](https://github.com/datum-cloud/enhancements/pull/922) -defines the consumer experience. The [cross-service overview](https://github.com/datum-cloud/enhancements/blob/docs/internal-dns-product-proposal/architecture/deliver/dns/internal-dns/README.md) -defines integration boundaries. This document owns the DNS control plane, -publication contracts, tenant isolation, and shared serving configuration. +defines the consumer experience. The +[Galactic design](https://github.com/datum-cloud/galactic/blob/main/docs/enhancements/networking/private-service-connect/README.md) +defines private connectivity and network authorization. This document defines +the DNS control plane, publication contracts, and query serving architecture. -## Control planes +## Control-plane boundaries -The DNS service runs in its own VPC and serves many consumer networks through a -shared fleet. The following diagram shows API boundaries and update paths. -Controllers can run in a management cluster while writing to these APIs. +The DNS service runs in its own VPC. Galactic exposes it through a private +endpoint in each consumer VPC. The diagram shows which API owns each resource; +it does not prescribe where controller processes run. ```mermaid flowchart TB - P[Consumer project API] - D[DNS service project API] - K[Karmada API] - E[Edge API + DNS fleet] - T[DNS publication transport] + Products[Product services] -->|Records and eligibility| P + Integration[Trusted VPC integration] -->|DNS access and network intent| P + P["Consumer project API
DNS resources and network intent"] + D["DNS service project API
Publication state, fleet plans, and service network intent"] + K["Karmada API
Placement and propagation"] + E["Edge API and shared fleet
Local networking, DNS agents, and serving state"] P -->|DNS reconciliation| D P -->|Network and workload intent| K + D -->|Service network and workload intent| K K -->|Placed desired state| E - D -->|Committed updates| T - T -->|DNS records and configuration| E + D -->|Committed DNS updates through regional NATS| E + E -.->|Serving acknowledgments| D + D -.->|DNS status| P ``` -DNS acknowledgments return to the service project. Network and workload -observations return through Karmada. Owning controllers project status into the -consumer project. - -| Control plane | Resources and ownership | -| --- | --- | -| Consumer project | Private zones, records, associations, managed naming, registrations, grants, and contributions. Trusted networking integration manages `DNSResolverContext` and `DNSResolverAccessBinding`; DNS writes their status. Product services publish records. | -| DNS service project | DNS-owned publication ownership, manifests, chunks, outboxes, internal `DNSResolverBinding` assignments, and allocation claims. Also holds the service's own network intent. | -| Karmada | Network and workload projections, location placement, propagation policies, and selected edge observations. Networking and product controllers own these projections. | -| Edge | Local network contexts, interfaces, VPCs, attachments, `ServiceEndpoint` and `ServiceRoutePolicy` resources, shared serving workloads, and checkpoints. Edge controllers and serving agents own local state. | - -DNS coordination retains each consumer's trusted project and source identities. -Controller replicas share authoritative storage for each publication and -serving-plan ownership domain. Product publishers cannot write that storage or -grant resolver access. - -Networking `NetworkContext` describes a network at a location; -`DNSResolverContext` defines its DNS scope. Resolver settings needed at the edge -must travel as desired state in the networking projection. The existing -propagation contract does not carry source status. DNS publications follow their -own delivery path. +- **Consumer project:** Owns private zones, records, associations, naming policies, + registrations, grants, and contributions. Trusted VPC integration writes + context and access specifications; DNS writes their status. Product services, + such as Compute and Connect, publish their records and eligibility. +- **DNS service project:** Owns durable publication state, writer leases, + allocation claims, regional serving plans, and the service's network intent. +- **Karmada:** Places network and workload intent at the edge. It is not the DNS + publication store. Guest resolver settings must travel as desired state in the + networking projection; source status is not a substitute for that contract. +- **Edge:** Owns local network contexts, interfaces, private endpoints, route + policies, shared DNS workloads, and serving checkpoints. + +DNS controllers use project discovery and authenticated project clients. They +do not watch VPCs, network interfaces, or product resources. VPC integration +translates networking authorization into DNS access. Product publishers translate +resource state into DNS records. These integrations keep networking and product +lifecycles outside the DNS controller. ## Contexts and regional access -One logical network has one DNS context across its locations. That context -selects its managed namespace and associated private zones. Separate contexts -can use overlapping names and addresses. Zone sharing requires explicit -associations. +A logical network has one DNS context across its locations. The context selects +its managed namespace and associated zones. Separate contexts can use identical +zone names and overlapping IP addresses. Sharing a zone requires an explicit +association. ```mermaid -flowchart TB - C["DNSResolverContext
Managed namespace + private zones"] - C --> AC["DNSResolverAccessBinding
Central"] - C --> AE["DNSResolverAccessBinding
East"] - subgraph Central[Central region] - WC[Workloads] --> GC[Galactic private endpoint] - GC --> FC[Shared DNS fleet] - end - subgraph East[East region] - WE[Workloads] --> GE[Galactic private endpoint] - GE --> FE[Shared DNS fleet] - end - AC -.->|Authorizes context access| FC - AE -.->|Authorizes context access| FE -``` - -Access bindings live in the consumer project API. Each has its own serving -target, destination, renewal, deadline, and readiness. Expiring East access -leaves Central access and the context intact. Network or context recreation -receives a new lifetime identity; stale updates cannot restore old access. - -The current `region` field identifies the serving target because the project -API can hold access for several regions. Trusted networking integration supplies -it. A future location reference could determine placement instead. Regional -access does not change zone contents or choose nearby application endpoints. -Region-to-cell mapping, resolver failover, and discovery locality remain open. - -DNS treats the context's consumer identity as opaque. Networking owns its mapping -to logical networks and regional VPCs. DNS does not discover VPCs or attachments. -Adding a VPC adds data and authorization to the shared fleet. - -## Context and access contracts - -Trusted networking integration writes context and access specs in the consumer -project API. DNS writes status and issues writer epochs. Product publishers -cannot create contexts or renew access. DNS does not read VPC or attachment APIs. - -The following proposed manifests show the integration contract. Names, UIDs, -addresses, and deadlines are illustrative; the resources are not in the released -API bundle. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSResolverContext -metadata: - name: application-network-dns - namespace: project-a -spec: - # Immutable logical consumer lifetime; opaque to DNS. - consumerID: "22222222-2222-4222-8222-222222222222" - managedNamespace: - enabled: true ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSResolverAccessBinding -metadata: - name: application-network-dns-central - namespace: project-a -spec: - contextRef: - name: application-network-dns - # API-assigned context UID prevents access surviving context recreation. - uid: "33333333-3333-4333-8333-333333333333" - # Serving target; not a new zone scope or discovery-locality policy. - region: us-central-1 - queryIdentity: - type: DestinationAddress - # Destination received by DNS after Galactic frontend translation. - value: "fd70:100::10" - port: 53 - transports: [UDP, TCP] - authorization: - # DNS-issued epoch and increasing writer sequence fence stale renewals. - writerEpoch: 3 - sequence: 27 - validUntil: "2026-10-08T12:05:00Z" +flowchart LR + C[DNSResolverContext] --> AC[Central access binding] + C --> AE[East access binding] + AC --> FC[Central shared fleet] + AE --> FE[East shared fleet] ``` -`consumerID` identifies the logical consumer independently of provider VPC UIDs. -The context UID fences DNS state. `queryIdentity` binds one authorized serving -destination to that context. The integration renews authorization before -`validUntil`; DNS serving agents enforce expiration locally. The managed suffix -and DNS-issued epoch are exposed through context status. +Both access bindings live in the consumer project's control plane. `region` +selects a serving target; it does not select the API where the object lives. +Each binding has its own UID, trusted destination, authorization deadline, and +readiness. East access can expire while Central access remains available. -`DNSZoneAssociation` and `DNSNamingPolicy` use UID-pinned `resolverContextRef` -references. `DNSResolverBinding` is a service-owned fleet plan. The prototype -retains legacy `vpcUID` wire fields for context UIDs; this compatibility does not -permit DNS to discover networking resources. +Publication reaches regions asynchronously. A region reports readiness after +its required serving members apply the current state. A failed region must not +block updates to healthy regions. Regional access does not, by itself, promise +that service discovery answers contain only endpoints from that region. -## Distributed publication +## Query path and network identity -Product publishers reserve names with `DNSRegistration`, receive scoped -`DNSContributionGrant` authorization, and publish typed records with -`DNSRecordContribution`. Contributions report eligibility, writer epoch, -sequence, and validity deadline. Private static records use `DNSRecordSet`. +Both VPCs can expose the same DNS address and contain the same client IP. Galactic +authorizes the private path and translates its destination into a distinct +service-side identity. DNS maps that identity to a context before consulting a +cache or selecting private zones. ```mermaid flowchart TB - P[Project DNS APIs] --> C[Validate and compile] - C --> S[DNS service project: ownership and committed publications] - S --> O[Durable outbox exporter] - O --> N[NATS JetStream] - N --> A[Regional authoritative materializers] - N --> R[Resolver configuration agents] - A --> V[Verified revision acknowledgments] - R --> V - V --> S - S --> T[Project status projection] -``` - -API storage holds durable ownership, manifests, chunks, outboxes, and serving -assignments. Conditional writes and writer epochs fence concurrent controllers. -Exporters retry committed updates; regional agents reject stale generations and -preserve expiration deadlines across replay and restart. The design does not -require PostgreSQL. NATS transports DNS intent; it is not queried to answer DNS. - -Authoritative materializers apply accepted publications to private PowerDNS -replicas. Resolver agents apply destination mappings, views, and shared-pool -membership. Acknowledgments report verified applied revisions. A broker -acknowledgment alone does not establish query readiness. - -## Shared serving and isolation - -The serving candidate uses dnsdist, shared BIND resolver processes with isolated -views, and PowerDNS Authoritative for private zone variants. Node and regional -tiers use shared processes; adding a context adds configuration and data. -Resolver choice and capacity remain subject to qualification. - -```mermaid -flowchart LR - G[Authorized Galactic destination] --> D[Shared node dnsdist] - D --> B[Shared node BIND: isolated context views] - B --> R[Shared regional dnsdist] - R --> C[Shared regional BIND: isolated context views] - C -->|Private zones| A[Shared PowerDNS Authoritative: zone variants] - C -->|Public recursion| P[Public authoritative DNS] + subgraph A[VPC A] + ClientA[Client 10.0.0.5] --> FrontA[DNS endpoint fd53::53] + end + subgraph B[VPC B] + ClientB[Client 10.0.0.5] --> FrontB[DNS endpoint fd53::53] + end + FrontA --> GrantA[Galactic authorized path A] + FrontB --> GrantB[Galactic authorized path B] + GrantA --> DestA[Service destination fd70:100::10] + GrantB --> DestB[Service destination fd70:100::11] + subgraph Shared[Shared DNS service fleet] + DestA --> Dist[dnsdist: trusted destination to context] + DestB --> Dist + Dist --> ViewA[Context A resolver view and cache] + Dist --> ViewB[Context B resolver view and cache] + ViewA -->|Context A source marker| Auth[Private authoritative fleet] + ViewB -->|Context B source marker| Auth + Auth --> ZoneA[Context A: api.prod.internal = 10.20.0.10] + Auth --> ZoneB[Context B: api.prod.internal = 10.20.0.20] + end ``` -The destination selects the context before any cache lookup. dnsdist forwards -trusted original-destination metadata to private resolver listeners; each BIND -view forwards through its assigned regional destination. Service-owned source -markers select authoritative variants. Consumer DNS metadata and source address -ranges do not select views. Backend listeners accept only authorized service -traffic. Unknown mappings fail closed; answers and caches remain isolated. - -A shared-pool replica becomes eligible only after it has every assigned context -at the required configuration revision. Failover stays within the same context -and serving assignment. The [Private Service Connect design](https://github.com/datum-cloud/galactic/blob/docs/private-service-connect-architecture/docs/enhancements/networking/private-service-connect/README.md) owns -attachment authorization, destination translation, and the return path. DNS -owns listener configuration and context selection. +Client source addresses, ECS, and client-supplied PROXY headers are not tenant +authority. Galactic checks the source VPC lifetime and route authorization before +translating the destination. A consumer must not be able to select another +context by addressing its service-side destination. Galactic handles the return +path to the consumer's DNS endpoint. + +The serving candidate uses shared node and regional tiers: + +1. **dnsdist** selects the context from the authorized destination. Its private + packet cache is disabled. +2. **BIND resolver views** isolate positive and negative caches. Node views + forward to context-specific regional destinations; regional views resolve + private zones and provide recursive resolution for other names. +3. **PowerDNS Authoritative** serves private zone variants selected by + service-owned source markers from the regional resolvers. + +[dnsdist PROXYv2](https://www.dnsdist.org/advanced/passing-source-address.html) +preserves the destination when forwarding to a shared resolver listener. +[BIND's PROXY access controls](https://bind9.readthedocs.io/en/v9.20.2/reference.html#namedconf-statement-allow-proxy) +restrict that listener to approved service peers and listener addresses. Each +regional resolver replica uses a distinct marker for each context. A shared pool member +is eligible only when all its assigned contexts and authorizations are current. + +[PowerDNS views](https://doc.powerdns.com/authoritative/views.html) are experimental +and require the LMDB backend and enabled zone cache. Unmatched clients bypass +view selection, and ordinary variantless zones are implicitly visible to views. +Therefore, private zones must have no variantless copy, and private authoritative +listeners must admit only approved resolver markers. + +The original proposal called for PowerDNS Recursor at both resolver tiers. The +prototype has not qualified Recursor's cache isolation for this contract. BIND +views are the current candidate; PowerDNS Authoritative is a different component. +Replacing BIND requires positive and negative cache isolation tests against the +selected Recursor version. Adding a context changes shared configuration and +zone data, not the number of resolver processes. ## Publication and service discovery -Product services such as Compute and Connect publish addresses and endpoint -eligibility through their project DNS APIs. DNS validates ownership, compiles -records, and distributes them to serving locations. Zone owners manage custom -records. Record publication and workload resolver setup are separate paths. - -Service discovery follows this lifecycle: +A registration reserves a name and record types. A grant authorizes a product +publisher. A contribution supplies records and a time-limited eligibility +observation. DNS compiles accepted contributions into a complete zone snapshot. ```mermaid -stateDiagram-v2 - [*] --> Pending - Pending --> Serving: Fresh eligible publication applied - Serving --> Withdrawn: Unhealthy, unavailable, or expired - Withdrawn --> Serving: Fresh eligible publication applied - Pending --> Deleted: Resource deleted - Serving --> Deleted: Resource deleted - Withdrawn --> Deleted: Resource deleted - Deleted --> [*] +sequenceDiagram + participant Product as Product publisher + participant Project as Consumer project API + participant DNS as DNS compiler + participant Store as DNS service project API + participant Transport as Regional NATS + participant Agent as Serving agent + Product->>Project: Records and fresh eligibility observation + DNS->>Project: Validate identities, grants, and deadlines + DNS->>Store: Stage immutable chunks and manifest + DNS->>Store: Commit active snapshot under writer lease + Store->>Transport: Export committed update through durable outbox + Transport->>Agent: Deliver update, including original deadlines + Agent->>Agent: Verify, apply, and enforce local expiry + Agent->>Store: Acknowledge applied snapshot + DNS->>Project: Report serving status + Product->>Project: Higher-sequence ineligible observation + DNS->>Store: Commit snapshot that withdraws the endpoint ``` -Instance identity names follow instance and address lifecycle; application -health does not automatically remove them. Services with stable addresses manage -backend health themselves. Connector exports publish selected reachable -services. Custom records do not receive automatic health checks. - -Serving uses applied state without consulting a product control plane per query. -Lifetime and ordering checks prevent delayed updates from restoring deleted -records or replacing newer state. During an outage, records and access remain -usable only until their validity deadlines. Discovery names with no eligible -endpoints return no endpoint addresses. Unsafe private resolution fails. - -Cached answers can persist until their DNS time to live (TTL) expires. -Publication, withdrawal, freshness, and access targets remain release decisions. -Product status distinguishes record readiness from resolver access readiness. +Each zone has one logical compiler owner. Compare-and-swap updates to durable +ownership fence that writer. Takeover advances its epoch; revisions increase +within an epoch. Serving agents reject older epochs and revisions. Controller +leader election alone does not fence a writer after a partition. + +Kubernetes API storage holds ownership, snapshots, and outboxes; this design +does not require Postgres. Regional exporters use +[NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) for at-least-once +delivery. Every serving replica receives its complete assigned stream. The +exporter records an outbox before sending and reconciles committed snapshots +that lack an outbox after a crash. A broker acknowledgment confirms delivery to +the broker, not application by a DNS serving member. Queries use local state and +do not contact the broker or project APIs. + +Products define eligibility for each record's purpose. An instance identity name +can follow interface readiness; service discovery can follow application health. +A stable service address can remain published while its backend membership +changes. DNS does not infer these policies from product APIs. User-authored +static records have no automatic endpoint health withdrawal. + +Contribution freshness and resolver access have separate deadlines. Replay, +restart, or recompilation cannot extend either original deadline. Serving agents +withdraw expired contributions locally; a watchdog removes a member that cannot +enforce expiry. Deletions retain tombstones and writer fences for the supported +replay window. + +A reserved name with no eligible addresses returns NODATA. An authorized context +whose required private state is unavailable returns SERVFAIL; an unknown or +expired access identity returns REFUSED. Missing private state must never fall +back to public resolution. Dynamic TTLs are bounded, and stale private answers +are disabled. Withdrawal budgets include publication delay, local expiry, and +client cache TTL: removing a served record cannot erase an answer already cached +by a client. + +## Integration and qualification + +Reuse project discovery, authenticated project clients, record types, condition +patterns, certificates, release tooling, and PowerDNS/LMDB operating experience. +The [infra configuration](https://github.com/datum-cloud/infra/tree/main/apps/dns-operator) +provides these integration points; its public serving and replication paths do +not establish private tenant isolation. + +New components include private API admission, context allocation, publication +ownership and compilation, durable export, shared fleet configuration, and +serving acknowledgments. Private snapshots need their own activation contract; +public zone replication through LightningStream does not replace it. + +Before release, validate overlapping zones and addresses, positive and negative +cache isolation, forged identities, UDP and TCP, access revocation, endpoint +withdrawal, writer failover, replay, restart, project API and broker outages, and partial regional +failure. Include Galactic's private path and guest resolver configuration in the +end-to-end environment. Measure fleet capacity and configuration reload behavior +at the expected context count and query rate. Resolve regional failover policy, +broker placement, withdrawal budgets, and tombstone retention before promising +service guarantees. diff --git a/docs/architecture/internal-dns/api-design.md b/docs/architecture/internal-dns/api-design.md new file mode 100644 index 00000000..1e6ec220 --- /dev/null +++ b/docs/architecture/internal-dns/api-design.md @@ -0,0 +1,449 @@ +# Internal DNS API design + +Status: Proposed. These examples describe the internal DNS contract, not an +installation recipe or a released API. See [the architecture](README.md) for +control-plane ownership and query serving. + +All examples use `dns.networking.miloapis.com/v1alpha1`. The project API assigns +object UIDs and generations. Reference UIDs, timestamps, addresses, and allocated +suffixes below are illustrative. `status` examples show controller output and +are not fields consumers submit when creating resources. + +## Ownership and validation + +- Trusted VPC integration creates resolver contexts and regional access bindings. + DNS allocates managed namespaces and reports serving status. +- Project users manage custom private zones, associations, static records, and + naming policy. Platform policy controls namespace reservations and grants. +- Product publishers write contribution records and eligibility. They cannot + authorize resolver access, grant themselves publication rights, or write DNS + service project state. +- DNS owns grant epochs, publication revisions, serving plans, and acknowledgments. + Admission must enforce field ownership, including fields sharing a status + subresource; RBAC on that subresource alone is insufficient. + +New references pin API-assigned UIDs. References that authorize publication also +pin the policy generation. The authenticated project and source-cluster identity +scope each reference; resource names and self-reported cluster IDs are not +credentials. Cross-project references are outside this proposal. + +## Context and regional access + +VPC integration creates an opaque DNS scope. DNS does not dereference the +`consumerID` or watch a networking resource. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverContext +metadata: + name: application +spec: + # Immutable lifetime identity supplied by trusted VPC integration. + consumerID: "11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222" + # Allocate a default private namespace without asking users to choose a zone. + managedNamespace: + enabled: true +status: + # DNS-owned output; publishers use this allocated zone and suffix. + managedNamespace: + dnsZoneRef: + name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + suffix: vpc-a7c9.project-p4e2.internal + # DNS-owned fencing epoch for trusted access authorization. + accessWriterEpoch: 3 + # Regional assignments do not move this object out of the project API. + servingTargets: + - region: central + shard: shared-0 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverAccessBinding +metadata: + name: application-central +spec: + # Pin the context lifetime so recreation cannot inherit old access. + contextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 + # Select the serving region, not a separate project control plane. + region: central + queryIdentity: + # Galactic supplies an authorized service-side destination before DNS lookup. + type: DestinationAddress + # This is distinct from the well-known address exposed inside the VPC. + value: fd70:100::10 + # Permit DNS over both transports at the authorized destination. + port: 53 + transports: [UDP, TCP] + authorization: + # Match the DNS-issued context epoch; stale issuers cannot restore access. + writerEpoch: 3 + # Renewals advance monotonically within this access-binding lifetime. + sequence: 27 + # Preserve this deadline through transport, replay, and local installation. + validUntil: "2026-10-08T20:05:00Z" +``` + +The context reports `Ready` when its namespace and assignment are available. +An access binding reports `Accepted` after authorization validation and `Ready` +after the required serving members apply current access and configuration. +`status.observedSequence` identifies the authorization renewal DNS has observed; +`status.bindingRef` identifies the DNS-owned serving plan. + +Context, region, query identity, port, transports, and authorization epoch are +immutable for an access-binding lifetime. Renewals advance the sequence with a +bounded deadline. Destinations must be unique within their routing scope. A +second region gets another binding against the same context. Galactic owns the +consumer-side address, private route, and network lease separately. + +## Private zones and associations + +Users can add a custom zone to the context and create static records in it. +Private zones do not require public domain verification or public delegation. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSZone +metadata: + name: production +spec: + # Visibility is immutable. The existing default remains Public. + visibility: Private + # Different contexts can associate separate zones with this same apex. + domainName: prod.internal + # Select private backend policy; users do not choose fleet instances. + dnsZoneClassName: datum-internal-dns +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSZoneAssociation +metadata: + name: production-application +spec: + # Both objects belong to this project. Pin each object's lifetime. + dnsZoneRef: + name: production + uid: 88888888-8888-4888-8888-888888888888 + # DNS consumes a context reference, not a VPC reference. + resolverContextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSRecordSet +metadata: + name: database +spec: + # Existing record API: a same-project name reference to its zone. + dnsZoneRef: + name: production + recordType: A + records: + # Static records have no endpoint eligibility lease. + - name: database + ttl: 30 + a: + content: 10.20.0.50 +``` + +A context can associate several zones, including its managed zone. Associating +two different zones with the same apex to one context is rejected. Parent and +child zones require deterministic longest-suffix selection. Two contexts can +share one zone through explicit associations, or associate separate zones with +the same apex for isolated answers. Admission must also reject conflicts +between static records and reserved registration names. + +## Automatic naming and additional names + +DNS owns the lifecycle of the managed namespace and its association. Publishers +read its zone reference from context status. A DNS-owned `DNSManagedNamespace` +can track allocation and cleanup; consumers should not need to create it. + +A naming policy adds names in custom zones. It supplements the managed name and +does not require a zone choice on each Compute or Connect resource. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSNamingPolicy +metadata: + name: application-production-names +spec: + # Apply naming rules to this DNS scope. + resolverContextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 + additionalNames: + # Add instance identity names, such as web-01.instances.prod.internal. + - registrationClass: InstanceIdentity + dnsZoneRef: + name: production + uid: 88888888-8888-4888-8888-888888888888 + namePrefix: instances + # Add discovery names, such as api.services.prod.internal. + - registrationClass: ServiceDiscovery + dnsZoneRef: + name: production + uid: 88888888-8888-4888-8888-888888888888 + namePrefix: services +``` + +Every target zone must be associated with the context. The proposed classes are +`InstanceIdentity`, `ServiceDiscovery`, `ServiceVIP`, and `ServiceExport`. +Product integrations apply the appropriate rule when reserving names. DNS does +not inspect the underlying product resource to determine its class. + +## Product publication + +Compute can reserve a name in the managed namespace, obtain a scoped publisher +grant, and publish eligible interface addresses. Connect uses the same contract +with its own publisher principal and export eligibility policy. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSRegistration +metadata: + name: instance-web-01 +spec: + # Select the managed zone allocated for the resource's DNS context. + dnsZoneRef: + name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + # Reserve this relative owner name, rather than one particular IP address. + name: web-01.instances + # Constrain which record types contributions can populate. + recordTypes: [A, AAAA] + # Publish only contributions with a current eligible observation. + publicationPolicy: EligibleContributions + # Bound client caching; this does not replace the observation deadline. + ttlSeconds: 30 + # Optional descendant reservations must not overlap other owners. + reservedDescendants: [] +status: + # DNS-owned output for product status and consumer display. + canonicalFQDN: web-01.instances.vpc-a7c9.project-p4e2.internal +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSContributionGrant +metadata: + name: web-01-compute +spec: + # Pin both the registration lifetime and the policy being authorized. + registrationRef: + name: instance-web-01 + uid: 55555555-5555-4555-8555-555555555555 + generation: 1 + # Audit label; the authenticated principal below authorizes the writer. + producerID: compute + principal: + # Trusted source-cluster identity distinguishes distributed publishers. + clusterUID: 99999999-9999-4999-8999-999999999999 + subject: system:serviceaccount:compute-system:dns-publisher + # Scope the writer to these types and owner names. + recordTypes: [A, AAAA] + nameScopes: [web-01.instances] +status: + # DNS allocates the epoch; the producer cannot choose its own fencing token. + activeWriterEpoch: 3 + # Identify the grant and registration policy versions DNS accepted. + observedGrantGeneration: 1 + observedRegistrationGeneration: 1 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSRecordContribution +metadata: + name: web-01-endpoint +spec: + # Bind the contribution to the reserved name and its current policy. + registrationRef: + name: instance-web-01 + uid: 55555555-5555-4555-8555-555555555555 + generation: 1 + # Bind the producer to its authorized grant lifetime and policy. + grantRef: + name: web-01-compute + uid: 66666666-6666-4666-8666-666666666666 + generation: 1 + # Reuse the existing typed record representation. + recordSets: + - recordType: A + records: + - name: web-01.instances + ttl: 30 + a: + content: 10.20.0.10 + - recordType: AAAA + records: + - name: web-01.instances + ttl: 30 + aaaa: + content: fd20::10 +status: + # Producer-owned: this observation covers the current contribution spec. + observedGeneration: 1 + # Producer-owned: echo the DNS-issued grant epoch. + writerEpoch: 3 + # Producer-owned: order observations within that epoch. + sequence: 18 + # Producer-owned: product policy determines readiness for this name's purpose. + eligible: true + reason: NetworkInterfaceProgrammed + # Producer-owned: DNS must preserve this original freshness deadline. + validUntil: "2026-10-08T20:01:00Z" + # DNS-owned: the publication revision that included this contribution. + publishedRevision: 12 +``` + +Status updates use the current resource version and preserve fields owned by +other controllers. Changing contribution records requires an observation for +the new generation. Admission verifies the authenticated principal, pinned +references, accepted grant epoch, increasing sequence, permitted records, and +bounded freshness interval. + +When an endpoint becomes ineligible, its publisher submits a higher-sequence +observation with `eligible: false`. DNS withdraws its addresses through the same +publication path. If the publisher disappears, the original deadline still +expires locally. Recovery requires a fresh eligible observation; replaying the +old one does not restore an endpoint. The product defines whether interface +readiness, application health, or another condition determines eligibility. + +The prototype also defines `Persistent`, but its distinct lifetime semantics +need API review. Use `DNSRecordSet` for static records in this proposal. + +## DNS service project contracts + +These resources are internal to the DNS service project. Consumers and product +publishers cannot write them. Examples show selected fields that explain +coordination; they are not complete generated serving plans or transport payloads. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverBinding +metadata: + name: application-central + namespace: dns-platform +spec: + # Preserve the authenticated source project identity in shared storage. + projectUID: 11111111-1111-4111-8111-111111111111 + # Prototype wire name: this carries a DNS context UID, not a VPC lookup key. + vpcUID: 33333333-3333-4333-8333-333333333333 + # Identify the shared regional serving assignment. + region: central + shard: shared-0 + # Fence plan replacement separately from configuration changes. + bindingGeneration: 1 + configurationRevision: 7 + # Trusted node-tier and regional-tier destinations for this context. + consumerAddress: fd70:100::10 + clusterAddress: fd70:200::10 + port: 53 + transports: [UDP, TCP] + # Use isolated caches in shared processes, with no private dnsdist packet cache. + resolverEngine: BIND9 + resolverIsolation: View + deploymentModel: SharedShard + dnsdistPacketCache: Disabled + # Only explicitly associated private zones enter the context's plan. + zoneUIDs: + - 44444444-4444-4444-8444-444444444444 + - 88888888-8888-4888-8888-888888888888 + # Preserve the accepted access authorization through every serving tier. + authorizationIssuerEpoch: 3 + authorizationRevision: 27 + authorizationValidUntil: "2026-10-08T20:05:00Z" +``` + +The publication resources divide coordination from immutable data: + +- `DNSPublicationOwnership` holds the zone's compiler lease, writer epoch, + revision allocator, and active manifest pointer. Updates use compare-and-swap. +- `DNSPublicationChunk` holds a bounded part of a snapshot with its index, epoch, + revision, hash, and payload. Chunks are immutable. +- `DNSPublicationManifest` identifies the complete snapshot, its chunks, + serving targets, and original contribution deadlines. +- `DNSTransportOutbox` records a committed update's subject, identity, hash, + payload, and dependencies before sending. Ambiguous retries reuse the same + event identity. Transport acknowledgment is separate from serving acknowledgment. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSPublicationOwnership +metadata: + name: managed-zone-owner + namespace: dns-platform +spec: + # One logical compiler owner per zone lifetime. + zoneUID: 44444444-4444-4444-8444-444444444444 + # Identify the current controller lease holder. + holderIdentity: compiler-a + # Takeover increases the epoch; revisions increase within it. + writerEpoch: 2 + nextRevision: 13 + # A writer must hold a current lease to commit the active pointer. + leaseUntil: "2026-10-08T20:01:00Z" + # The committed snapshot is the activation authority. + activeManifestName: managed-zone-e2-r12 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSPublicationManifest +metadata: + name: managed-zone-e2-r12 + namespace: dns-platform +spec: + # Retain the source zone lifetime and its private apex. + zoneRef: + name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + zoneApex: vpc-a7c9.project-p4e2.internal + # Select this zone's private authoritative variant. + variant: zone-44444444 + # Prototype wire name: values are DNS context UIDs, not networking references. + vpcUIDs: + - 33333333-3333-4333-8333-333333333333 + # Agents reject snapshots behind their accepted writer fence. + writerEpoch: 2 + revision: 12 + previousRevision: 11 + # Deletion uses a fenced tombstone rather than silently dropping history. + tombstone: false + # Verify all chunks before activation; these hashes are illustrative. + chunks: + - name: managed-zone-e2-r12-c0 + sha256: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + size: 512 + contentHash: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb + generatedAt: "2026-10-08T20:00:00Z" + contributionFences: + # Original observation deadlines survive recompilation and replay. + - uid: 77777777-7777-4777-8777-777777777777 + grantUID: 66666666-6666-4666-8666-666666666666 + epoch: 3 + sequence: 18 + validUntil: "2026-10-08T20:01:00Z" + # Each required replica receives the complete regional assignment. + servingTargets: + - region: central + shard: shared-0 +``` + +The compiler stages immutable chunks and a manifest, then commits the active +pointer under its current ownership lease. Exporters reconcile that pointer and +repair a missing outbox after a crash. Agents verify identities, hashes, writer +fences, and deadlines before applying a complete snapshot. Acknowledgments must +identify the exact binding or zone lifetime, epoch, and revision applied. +Serving agents retain fences for expired, withdrawn, and deleted state so an +older replay cannot restore it. + +## Compatibility and API review + +The target integration uses `resolverContextRef` and never requires DNS to read +networking resources. The prototype retains legacy `vpcRef` inputs and internal +`vpcUID`/`vpcUIDs` wire names. Context-based execution treats the latter as opaque +DNS context identities. Rename these internal fields and define migration rules +before stabilizing the API; do not make the old VPC coupling a supported contract. + +API review must settle namespace reservation policy, naming conflicts, grant +issuance and revocation, field-level status ownership, deadline bounds, regional +readiness aggregation, snapshot size limits, and the complete internal serving +plan schema. Private visibility and exclusion from public controllers form a +separate implementation boundary in +[DNS operator #230](https://github.com/datum-cloud/dns-operator/pull/230). From e97d8699b450590766c9f0447a411bfdc75b757f Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 15:34:55 -0500 Subject: [PATCH 03/25] docs: follow enhancement template for internal DNS design --- docs/architecture/internal-dns/README.md | 229 ++++++++++++++++++++--- 1 file changed, 207 insertions(+), 22 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index f9456fe1..5a338221 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -1,20 +1,109 @@ -# Internal DNS architecture +--- +status: provisional +stage: alpha +latest-milestone: "TBD" +--- -Status: Proposed. The API contracts and serving components require qualification -before release. See [API design](api-design.md) for annotated YAML examples. +# Internal DNS architecture -Internal DNS lets resources resolve private names within a VPC. One DNS context -selects the names visible to that network, including an automatically managed -namespace and optional custom zones. Many contexts share the same serving fleet; -adding a VPC does not create a resolver deployment. +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User stories](#user-stories) + - [Notes, constraints, and caveats](#notes-constraints-and-caveats) + - [Risks and mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Control-plane boundaries](#control-plane-boundaries) + - [Contexts and regional access](#contexts-and-regional-access) + - [Query path and network identity](#query-path-and-network-identity) + - [Publication and service discovery](#publication-and-service-discovery) + - [API design](#api-design) + - [Integration boundaries](#integration-boundaries) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed](#infrastructure-needed) + +## Summary + +Internal DNS lets resources resolve private names within a VPC. Each network has +a managed namespace and can use additional private zones. Product services +publish names automatically, and shared regional fleets serve many VPCs with +isolated DNS contexts. The [product enhancement](https://github.com/datum-cloud/enhancements/pull/922) defines the consumer experience. The [Galactic design](https://github.com/datum-cloud/galactic/blob/main/docs/enhancements/networking/private-service-connect/README.md) -defines private connectivity and network authorization. This document defines -the DNS control plane, publication contracts, and query serving architecture. +defines private connectivity. This enhancement defines the DNS architecture and +[API design](api-design.md). + +## Motivation + +Compute instances, Connect services, and other resources need names that resolve +within their network. Consumers should receive useful default names without +choosing a zone for every resource. Shared infrastructure must isolate networks +that use identical names and addresses, and stop returning endpoints that are no +longer eligible. + +### Goals + +- Serve private zones through the VPC's inherited resolver configuration. +- Allocate default names and support additional custom zones. +- Isolate overlapping names, addresses, and resolver caches across DNS contexts. +- Accept publications from distributed product control planes and withdraw + expired or ineligible endpoints. +- Scale shared serving capacity across regions without deployments per VPC. + +### Non-Goals + +- Designing Galactic's private connectivity APIs or packet handling. +- Having DNS controllers inspect networking or product resources. +- Defining application health policies for product services. +- Changing public DNS behavior or defining cross-project zone sharing. + +## Proposal + +Trusted VPC integration provisions a DNS context and regional resolver access. +DNS allocates a managed namespace, accepts explicit custom zone associations, +and compiles product publications into private serving state. Consumers use a +stable resolver address in their VPC while the platform selects shared capacity. -## Control-plane boundaries +### User stories + +- A Compute user resolves an instance's default private hostname without + selecting a zone or resolver deployment. +- A project adds `prod.internal` and uses names such as `api.prod.internal`. +- Two VPCs use the same zone name and resolver address but receive isolated + answers. Removing an eligible endpoint withdraws its discovery address. + +### Notes, constraints, and caveats + +These contracts are proposed; API and serving qualification remain required. +Regional publication is asynchronous. Resolver access and endpoint eligibility +have separate deadlines. Client caches can retain an answer until its TTL +expires. Example addresses, identities, and allocated suffixes are illustrative. + +### Risks and mitigations + +- Tenant leakage: authorize network identity before cache lookup, isolate + resolver views, and test overlapping zones and forged identities. +- Stale endpoints or access: preserve original deadlines, fence writers and + replay, and enforce expiry locally at serving nodes. +- Shared fleet failure: qualify configuration activation, readiness, capacity, + and regional failure handling before release. + +## Design Details + +### Control-plane boundaries The DNS service runs in its own VPC. Galactic exposes it through a private endpoint in each consumer VPC. The diagram shows which API owns each resource; @@ -55,7 +144,7 @@ translates networking authorization into DNS access. Product publishers translat resource state into DNS records. These integrations keep networking and product lifecycles outside the DNS controller. -## Contexts and regional access +### Contexts and regional access A logical network has one DNS context across its locations. The context selects its managed namespace and associated zones. Separate contexts can use identical @@ -80,7 +169,7 @@ its required serving members apply the current state. A failed region must not block updates to healthy regions. Regional access does not, by itself, promise that service discovery answers contain only endpoints from that region. -## Query path and network identity +### Query path and network identity Both VPCs can expose the same DNS address and contain the same client IP. Galactic authorizes the private path and translates its destination into a distinct @@ -147,7 +236,7 @@ Replacing BIND requires positive and negative cache isolation tests against the selected Recursor version. Adding a context changes shared configuration and zone data, not the number of resolver processes. -## Publication and service discovery +### Publication and service discovery A registration reserves a name and record types. A grant authorizes a product publisher. A contribution supplies records and a time-limited eligibility @@ -208,7 +297,14 @@ are disabled. Withdrawal budgets include publication delay, local expiry, and client cache TTL: removing a served record cannot erase an answer already cached by a client. -## Integration and qualification +### API design + +The [API design](api-design.md) provides YAML examples with field comments for +contexts, regional access, private zones, associations, naming policies, +registrations, grants, contributions, and DNS-owned publication state. It also +defines writer ownership, validation, and prototype compatibility boundaries. + +### Integration boundaries Reuse project discovery, authenticated project clients, record types, condition patterns, certificates, release tooling, and PowerDNS/LMDB operating experience. @@ -221,11 +317,100 @@ ownership and compilation, durable export, shared fleet configuration, and serving acknowledgments. Private snapshots need their own activation contract; public zone replication through LightningStream does not replace it. -Before release, validate overlapping zones and addresses, positive and negative -cache isolation, forged identities, UDP and TCP, access revocation, endpoint -withdrawal, writer failover, replay, restart, project API and broker outages, and partial regional -failure. Include Galactic's private path and guest resolver configuration in the -end-to-end environment. Measure fleet capacity and configuration reload behavior -at the expected context count and query rate. Resolve regional failover policy, -broker placement, withdrawal budgets, and tombstone retention before promising -service guarantees. +## Production Readiness Review Questionnaire + +Production readiness review remains open. Resolve the following requirements +before release. + +### Feature Enablement and Rollback + +Alpha needs separate activation controls for product publication and private +serving. Flag names and deployment wiring require implementation review. Public +visibility remains the default. Rollback must revoke consumer access before +draining private serving capacity; workloads that depend on internal DNS lose +resolution. Reenabling requires fresh authorizations and observations, with +retained writer fences preventing stale replay. + +### Rollout, Upgrade and Rollback Planning + +Deploy shared capacity before granting access, then qualify a limited set of +contexts and expand by region. Validate overlapping zones and addresses, +positive and negative cache isolation, forged identities, UDP and TCP, access +revocation, endpoint withdrawal, writer failover, replay, restart, API and broker +outages, and partial regional failure. Include Galactic's private path and guest +resolver configuration. Upgrade, rollback, and reenabling tests remain release +requirements. Resolve withdrawal budgets and tombstone retention before rollout. + +### Monitoring Requirements + +Measure query latency and failures, publication lag, outbox backlog, expiry +withdrawal delay, failed activations, and watchdog removal of serving members. +Keep metric labels bounded; use resource status for context-level diagnosis. +Consumers inspect access readiness and published names, then query from their +VPC. Query availability, latency, and withdrawal SLOs require measured targets +before release. + +### Dependencies + +- Project APIs and discovery support intent, renewal, and durable coordination. + An outage stops updates; installed state remains subject to its deadlines. +- Galactic and Karmada provide authorized private paths and placed network and + workload intent. Network authorization expires independently of DNS state. +- NATS JetStream carries committed updates. Outages delay publication; queries + continue from local state while original freshness deadlines still apply. +- dnsdist, BIND, and PowerDNS Authoritative provide query serving. Versions and + configuration must pass the tenant isolation qualification described above. + +### Scalability + +API traffic includes project watches, eligibility and access renewals, compiler +leases, snapshot writes, export progress, and serving acknowledgments. Bound +snapshot chunks and retained history. Contexts add views, addresses, and zone +data to shared processes. Measure API write rates, cache memory, configuration +reload time, transport fanout, and query throughput at the expected context +count. Supported limits and regional failover capacity remain open decisions. + +### Troubleshooting + +Trace project, context, and access UIDs through the accepted authorization, +committed manifest epoch and revision, and serving acknowledgments. For REFUSED, +inspect access expiry and the authorized network destination. For SERVFAIL, +inspect missing private state and serving readiness. For NODATA, inspect product +eligibility and contribution deadlines. During API or broker outages, inspect +outbox lag and local expiry; replay must not renew stale state. + +## Implementation History + +- 2026-10-08: [DNS operator #229](https://github.com/datum-cloud/dns-operator/pull/229) + proposes the DNS architecture and annotated API contracts. +- 2026-10-08: [DNS operator #230](https://github.com/datum-cloud/dns-operator/pull/230) + proposes the private-zone visibility boundary. The complete feature remains + under development. + +## Drawbacks + +Shared view configuration and experimental authoritative views add operational +complexity. Deadline enforcement, distributed writer fencing, and durable +publication introduce coordination state and ongoing API traffic. A fleet +failure can affect many contexts, so capacity and activation need qualification. + +## Alternatives + +- **PowerDNS Recursor:** The original resolver choice. Replacing BIND remains an + option after qualifying positive and negative cache isolation for the selected + version. +- **CoreDNS:** An alternative serving component that needs a tenant-aware query + and cache design, plus throughput and operational qualification. +- **Resolvers per VPC:** Provide separate processes but conflict with the shared + fleet requirement and add workload overhead for every network. +- **DNS reads product and networking APIs:** Avoids publisher integrations but + couples DNS to their lifecycles and health policies. Explicit publications and + opaque contexts keep those responsibilities with their owners. + +## Infrastructure Needed + +Provide a DNS service VPC, Galactic private connectivity, shared node and regional +serving capacity, protected destination and source-marker allocation, durable +project API storage, and secured NATS JetStream capacity. Reuse platform identity, +certificates, and deployment tooling. Regional broker placement, capacity, and +failover policy need review before production deployment. From 0fe93422106e571f3a261b65065bca6f38b565dc Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 15:37:31 -0500 Subject: [PATCH 04/25] docs: inline internal DNS API design in enhancement --- docs/architecture/README.md | 2 +- docs/architecture/internal-dns/README.md | 456 ++++++++++++++++++- docs/architecture/internal-dns/api-design.md | 449 ------------------ 3 files changed, 452 insertions(+), 455 deletions(-) delete mode 100644 docs/architecture/internal-dns/api-design.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 73ce0347..8f45a21e 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -163,7 +163,7 @@ See the [API Reference](./api-reference.md) for complete field documentation. - [Internal DNS architecture](./internal-dns/README.md) — Proposed private DNS control plane, publication contracts, and shared serving architecture -- [Internal DNS API design](./internal-dns/api-design.md) — Annotated YAML for +- [Internal DNS API design](./internal-dns/README.md#api-design) — Annotated YAML for contexts, regional access, private zones, and product publication - [Record Ownership](./record-ownership.md) — Who owns a name in a zone, and when that changes diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 5a338221..33114ad9 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -20,6 +20,13 @@ latest-milestone: "TBD" - [Query path and network identity](#query-path-and-network-identity) - [Publication and service discovery](#publication-and-service-discovery) - [API design](#api-design) + - [Ownership and validation](#ownership-and-validation) + - [Context and regional access](#context-and-regional-access) + - [Private zones and associations](#private-zones-and-associations) + - [Automatic naming and additional names](#automatic-naming-and-additional-names) + - [Product publication](#product-publication) + - [DNS service project contracts](#dns-service-project-contracts) + - [Compatibility and API review](#compatibility-and-api-review) - [Integration boundaries](#integration-boundaries) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - [Feature Enablement and Rollback](#feature-enablement-and-rollback) @@ -44,7 +51,7 @@ The [product enhancement](https://github.com/datum-cloud/enhancements/pull/922) defines the consumer experience. The [Galactic design](https://github.com/datum-cloud/galactic/blob/main/docs/enhancements/networking/private-service-connect/README.md) defines private connectivity. This enhancement defines the DNS architecture and -[API design](api-design.md). +[API design](#api-design). ## Motivation @@ -299,10 +306,449 @@ by a client. ### API design -The [API design](api-design.md) provides YAML examples with field comments for -contexts, regional access, private zones, associations, naming policies, -registrations, grants, contributions, and DNS-owned publication state. It also -defines writer ownership, validation, and prototype compatibility boundaries. +All examples use `dns.networking.miloapis.com/v1alpha1`. The project API assigns +object UIDs and generations. Reference UIDs, timestamps, addresses, and allocated +suffixes below are illustrative. `status` examples show controller output and +are not fields consumers submit when creating resources. + +#### Ownership and validation + +- Trusted VPC integration creates resolver contexts and regional access bindings. + DNS allocates managed namespaces and reports serving status. +- Project users manage custom private zones, associations, static records, and + naming policy. Platform policy controls namespace reservations and grants. +- Product publishers write contribution records and eligibility. They cannot + authorize resolver access, grant themselves publication rights, or write DNS + service project state. +- DNS owns grant epochs, publication revisions, serving plans, and acknowledgments. + Admission must enforce field ownership, including fields sharing a status + subresource; RBAC on that subresource alone is insufficient. + +New references pin API-assigned UIDs. References that authorize publication also +pin the policy generation. The authenticated project and source-cluster identity +scope each reference; resource names and self-reported cluster IDs are not +credentials. Cross-project references are outside this proposal. + +#### Context and regional access + +VPC integration creates an opaque DNS scope. DNS does not dereference the +`consumerID` or watch a networking resource. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverContext +metadata: + name: application +spec: + # Immutable lifetime identity supplied by trusted VPC integration. + consumerID: "11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222" + # Allocate a default private namespace without asking users to choose a zone. + managedNamespace: + enabled: true +status: + # DNS-owned output; publishers use this allocated zone and suffix. + managedNamespace: + dnsZoneRef: + name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + suffix: vpc-a7c9.project-p4e2.internal + # DNS-owned fencing epoch for trusted access authorization. + accessWriterEpoch: 3 + # Regional assignments do not move this object out of the project API. + servingTargets: + - region: central + shard: shared-0 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverAccessBinding +metadata: + name: application-central +spec: + # Pin the context lifetime so recreation cannot inherit old access. + contextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 + # Select the serving region, not a separate project control plane. + region: central + queryIdentity: + # Galactic supplies an authorized service-side destination before DNS lookup. + type: DestinationAddress + # This is distinct from the well-known address exposed inside the VPC. + value: fd70:100::10 + # Permit DNS over both transports at the authorized destination. + port: 53 + transports: [UDP, TCP] + authorization: + # Match the DNS-issued context epoch; stale issuers cannot restore access. + writerEpoch: 3 + # Renewals advance monotonically within this access-binding lifetime. + sequence: 27 + # Preserve this deadline through transport, replay, and local installation. + validUntil: "2026-10-08T20:05:00Z" +``` + +The context reports `Ready` when its namespace and assignment are available. +An access binding reports `Accepted` after authorization validation and `Ready` +after the required serving members apply current access and configuration. +`status.observedSequence` identifies the authorization renewal DNS has observed; +`status.bindingRef` identifies the DNS-owned serving plan. + +Context, region, query identity, port, transports, and authorization epoch are +immutable for an access-binding lifetime. Renewals advance the sequence with a +bounded deadline. Destinations must be unique within their routing scope. A +second region gets another binding against the same context. Galactic owns the +consumer-side address, private route, and network lease separately. + +#### Private zones and associations + +Users can add a custom zone to the context and create static records in it. +Private zones do not require public domain verification or public delegation. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSZone +metadata: + name: production +spec: + # Visibility is immutable. The existing default remains Public. + visibility: Private + # Different contexts can associate separate zones with this same apex. + domainName: prod.internal + # Select private backend policy; users do not choose fleet instances. + dnsZoneClassName: datum-internal-dns +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSZoneAssociation +metadata: + name: production-application +spec: + # Both objects belong to this project. Pin each object's lifetime. + dnsZoneRef: + name: production + uid: 88888888-8888-4888-8888-888888888888 + # DNS consumes a context reference, not a VPC reference. + resolverContextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSRecordSet +metadata: + name: database +spec: + # Existing record API: a same-project name reference to its zone. + dnsZoneRef: + name: production + recordType: A + records: + # Static records have no endpoint eligibility lease. + - name: database + ttl: 30 + a: + content: 10.20.0.50 +``` + +A context can associate several zones, including its managed zone. Associating +two different zones with the same apex to one context is rejected. Parent and +child zones require deterministic longest-suffix selection. Two contexts can +share one zone through explicit associations, or associate separate zones with +the same apex for isolated answers. Admission must also reject conflicts +between static records and reserved registration names. + +#### Automatic naming and additional names + +DNS owns the lifecycle of the managed namespace and its association. Publishers +read its zone reference from context status. A DNS-owned `DNSManagedNamespace` +can track allocation and cleanup; consumers should not need to create it. + +A naming policy adds names in custom zones. It supplements the managed name and +does not require a zone choice on each Compute or Connect resource. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSNamingPolicy +metadata: + name: application-production-names +spec: + # Apply naming rules to this DNS scope. + resolverContextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 + additionalNames: + # Add instance identity names, such as web-01.instances.prod.internal. + - registrationClass: InstanceIdentity + dnsZoneRef: + name: production + uid: 88888888-8888-4888-8888-888888888888 + namePrefix: instances + # Add discovery names, such as api.services.prod.internal. + - registrationClass: ServiceDiscovery + dnsZoneRef: + name: production + uid: 88888888-8888-4888-8888-888888888888 + namePrefix: services +``` + +Every target zone must be associated with the context. The proposed classes are +`InstanceIdentity`, `ServiceDiscovery`, `ServiceVIP`, and `ServiceExport`. +Product integrations apply the appropriate rule when reserving names. DNS does +not inspect the underlying product resource to determine its class. + +#### Product publication + +Compute can reserve a name in the managed namespace, obtain a scoped publisher +grant, and publish eligible interface addresses. Connect uses the same contract +with its own publisher principal and export eligibility policy. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSRegistration +metadata: + name: instance-web-01 +spec: + # Select the managed zone allocated for the resource's DNS context. + dnsZoneRef: + name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + # Reserve this relative owner name, rather than one particular IP address. + name: web-01.instances + # Constrain which record types contributions can populate. + recordTypes: [A, AAAA] + # Publish only contributions with a current eligible observation. + publicationPolicy: EligibleContributions + # Bound client caching; this does not replace the observation deadline. + ttlSeconds: 30 + # Optional descendant reservations must not overlap other owners. + reservedDescendants: [] +status: + # DNS-owned output for product status and consumer display. + canonicalFQDN: web-01.instances.vpc-a7c9.project-p4e2.internal +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSContributionGrant +metadata: + name: web-01-compute +spec: + # Pin both the registration lifetime and the policy being authorized. + registrationRef: + name: instance-web-01 + uid: 55555555-5555-4555-8555-555555555555 + generation: 1 + # Audit label; the authenticated principal below authorizes the writer. + producerID: compute + principal: + # Trusted source-cluster identity distinguishes distributed publishers. + clusterUID: 99999999-9999-4999-8999-999999999999 + subject: system:serviceaccount:compute-system:dns-publisher + # Scope the writer to these types and owner names. + recordTypes: [A, AAAA] + nameScopes: [web-01.instances] +status: + # DNS allocates the epoch; the producer cannot choose its own fencing token. + activeWriterEpoch: 3 + # Identify the grant and registration policy versions DNS accepted. + observedGrantGeneration: 1 + observedRegistrationGeneration: 1 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSRecordContribution +metadata: + name: web-01-endpoint +spec: + # Bind the contribution to the reserved name and its current policy. + registrationRef: + name: instance-web-01 + uid: 55555555-5555-4555-8555-555555555555 + generation: 1 + # Bind the producer to its authorized grant lifetime and policy. + grantRef: + name: web-01-compute + uid: 66666666-6666-4666-8666-666666666666 + generation: 1 + # Reuse the existing typed record representation. + recordSets: + - recordType: A + records: + - name: web-01.instances + ttl: 30 + a: + content: 10.20.0.10 + - recordType: AAAA + records: + - name: web-01.instances + ttl: 30 + aaaa: + content: fd20::10 +status: + # Producer-owned: this observation covers the current contribution spec. + observedGeneration: 1 + # Producer-owned: echo the DNS-issued grant epoch. + writerEpoch: 3 + # Producer-owned: order observations within that epoch. + sequence: 18 + # Producer-owned: product policy determines readiness for this name's purpose. + eligible: true + reason: NetworkInterfaceProgrammed + # Producer-owned: DNS must preserve this original freshness deadline. + validUntil: "2026-10-08T20:01:00Z" + # DNS-owned: the publication revision that included this contribution. + publishedRevision: 12 +``` + +Status updates use the current resource version and preserve fields owned by +other controllers. Changing contribution records requires an observation for +the new generation. Admission verifies the authenticated principal, pinned +references, accepted grant epoch, increasing sequence, permitted records, and +bounded freshness interval. + +When an endpoint becomes ineligible, its publisher submits a higher-sequence +observation with `eligible: false`. DNS withdraws its addresses through the same +publication path. If the publisher disappears, the original deadline still +expires locally. Recovery requires a fresh eligible observation; replaying the +old one does not restore an endpoint. The product defines whether interface +readiness, application health, or another condition determines eligibility. + +The prototype also defines `Persistent`, but its distinct lifetime semantics +need API review. Use `DNSRecordSet` for static records in this proposal. + +#### DNS service project contracts + +These resources are internal to the DNS service project. Consumers and product +publishers cannot write them. Examples show selected fields that explain +coordination; they are not complete generated serving plans or transport payloads. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSResolverBinding +metadata: + name: application-central + namespace: dns-platform +spec: + # Preserve the authenticated source project identity in shared storage. + projectUID: 11111111-1111-4111-8111-111111111111 + # Prototype wire name: this carries a DNS context UID, not a VPC lookup key. + vpcUID: 33333333-3333-4333-8333-333333333333 + # Identify the shared regional serving assignment. + region: central + shard: shared-0 + # Fence plan replacement separately from configuration changes. + bindingGeneration: 1 + configurationRevision: 7 + # Trusted node-tier and regional-tier destinations for this context. + consumerAddress: fd70:100::10 + clusterAddress: fd70:200::10 + port: 53 + transports: [UDP, TCP] + # Use isolated caches in shared processes, with no private dnsdist packet cache. + resolverEngine: BIND9 + resolverIsolation: View + deploymentModel: SharedShard + dnsdistPacketCache: Disabled + # Only explicitly associated private zones enter the context's plan. + zoneUIDs: + - 44444444-4444-4444-8444-444444444444 + - 88888888-8888-4888-8888-888888888888 + # Preserve the accepted access authorization through every serving tier. + authorizationIssuerEpoch: 3 + authorizationRevision: 27 + authorizationValidUntil: "2026-10-08T20:05:00Z" +``` + +The publication resources divide coordination from immutable data: + +- `DNSPublicationOwnership` holds the zone's compiler lease, writer epoch, + revision allocator, and active manifest pointer. Updates use compare-and-swap. +- `DNSPublicationChunk` holds a bounded part of a snapshot with its index, epoch, + revision, hash, and payload. Chunks are immutable. +- `DNSPublicationManifest` identifies the complete snapshot, its chunks, + serving targets, and original contribution deadlines. +- `DNSTransportOutbox` records a committed update's subject, identity, hash, + payload, and dependencies before sending. Ambiguous retries reuse the same + event identity. Transport acknowledgment is separate from serving acknowledgment. + +```yaml +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSPublicationOwnership +metadata: + name: managed-zone-owner + namespace: dns-platform +spec: + # One logical compiler owner per zone lifetime. + zoneUID: 44444444-4444-4444-8444-444444444444 + # Identify the current controller lease holder. + holderIdentity: compiler-a + # Takeover increases the epoch; revisions increase within it. + writerEpoch: 2 + nextRevision: 13 + # A writer must hold a current lease to commit the active pointer. + leaseUntil: "2026-10-08T20:01:00Z" + # The committed snapshot is the activation authority. + activeManifestName: managed-zone-e2-r12 +--- +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSPublicationManifest +metadata: + name: managed-zone-e2-r12 + namespace: dns-platform +spec: + # Retain the source zone lifetime and its private apex. + zoneRef: + name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + zoneApex: vpc-a7c9.project-p4e2.internal + # Select this zone's private authoritative variant. + variant: zone-44444444 + # Prototype wire name: values are DNS context UIDs, not networking references. + vpcUIDs: + - 33333333-3333-4333-8333-333333333333 + # Agents reject snapshots behind their accepted writer fence. + writerEpoch: 2 + revision: 12 + previousRevision: 11 + # Deletion uses a fenced tombstone rather than silently dropping history. + tombstone: false + # Verify all chunks before activation; these hashes are illustrative. + chunks: + - name: managed-zone-e2-r12-c0 + sha256: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + size: 512 + contentHash: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb + generatedAt: "2026-10-08T20:00:00Z" + contributionFences: + # Original observation deadlines survive recompilation and replay. + - uid: 77777777-7777-4777-8777-777777777777 + grantUID: 66666666-6666-4666-8666-666666666666 + epoch: 3 + sequence: 18 + validUntil: "2026-10-08T20:01:00Z" + # Each required replica receives the complete regional assignment. + servingTargets: + - region: central + shard: shared-0 +``` + +The compiler stages immutable chunks and a manifest, then commits the active +pointer under its current ownership lease. Exporters reconcile that pointer and +repair a missing outbox after a crash. Agents verify identities, hashes, writer +fences, and deadlines before applying a complete snapshot. Acknowledgments must +identify the exact binding or zone lifetime, epoch, and revision applied. +Serving agents retain fences for expired, withdrawn, and deleted state so an +older replay cannot restore it. + +#### Compatibility and API review + +The target integration uses `resolverContextRef` and never requires DNS to read +networking resources. The prototype retains legacy `vpcRef` inputs and internal +`vpcUID`/`vpcUIDs` wire names. Context-based execution treats the latter as opaque +DNS context identities. Rename these internal fields and define migration rules +before stabilizing the API; do not make the old VPC coupling a supported contract. + +API review must settle namespace reservation policy, naming conflicts, grant +issuance and revocation, field-level status ownership, deadline bounds, regional +readiness aggregation, snapshot size limits, and the complete internal serving +plan schema. Private visibility and exclusion from public controllers form a +separate implementation boundary in +[DNS operator #230](https://github.com/datum-cloud/dns-operator/pull/230). ### Integration boundaries diff --git a/docs/architecture/internal-dns/api-design.md b/docs/architecture/internal-dns/api-design.md deleted file mode 100644 index 1e6ec220..00000000 --- a/docs/architecture/internal-dns/api-design.md +++ /dev/null @@ -1,449 +0,0 @@ -# Internal DNS API design - -Status: Proposed. These examples describe the internal DNS contract, not an -installation recipe or a released API. See [the architecture](README.md) for -control-plane ownership and query serving. - -All examples use `dns.networking.miloapis.com/v1alpha1`. The project API assigns -object UIDs and generations. Reference UIDs, timestamps, addresses, and allocated -suffixes below are illustrative. `status` examples show controller output and -are not fields consumers submit when creating resources. - -## Ownership and validation - -- Trusted VPC integration creates resolver contexts and regional access bindings. - DNS allocates managed namespaces and reports serving status. -- Project users manage custom private zones, associations, static records, and - naming policy. Platform policy controls namespace reservations and grants. -- Product publishers write contribution records and eligibility. They cannot - authorize resolver access, grant themselves publication rights, or write DNS - service project state. -- DNS owns grant epochs, publication revisions, serving plans, and acknowledgments. - Admission must enforce field ownership, including fields sharing a status - subresource; RBAC on that subresource alone is insufficient. - -New references pin API-assigned UIDs. References that authorize publication also -pin the policy generation. The authenticated project and source-cluster identity -scope each reference; resource names and self-reported cluster IDs are not -credentials. Cross-project references are outside this proposal. - -## Context and regional access - -VPC integration creates an opaque DNS scope. DNS does not dereference the -`consumerID` or watch a networking resource. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSResolverContext -metadata: - name: application -spec: - # Immutable lifetime identity supplied by trusted VPC integration. - consumerID: "11111111-1111-4111-8111-111111111111/22222222-2222-4222-8222-222222222222" - # Allocate a default private namespace without asking users to choose a zone. - managedNamespace: - enabled: true -status: - # DNS-owned output; publishers use this allocated zone and suffix. - managedNamespace: - dnsZoneRef: - name: managed-application - uid: 44444444-4444-4444-8444-444444444444 - suffix: vpc-a7c9.project-p4e2.internal - # DNS-owned fencing epoch for trusted access authorization. - accessWriterEpoch: 3 - # Regional assignments do not move this object out of the project API. - servingTargets: - - region: central - shard: shared-0 ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSResolverAccessBinding -metadata: - name: application-central -spec: - # Pin the context lifetime so recreation cannot inherit old access. - contextRef: - name: application - uid: 33333333-3333-4333-8333-333333333333 - # Select the serving region, not a separate project control plane. - region: central - queryIdentity: - # Galactic supplies an authorized service-side destination before DNS lookup. - type: DestinationAddress - # This is distinct from the well-known address exposed inside the VPC. - value: fd70:100::10 - # Permit DNS over both transports at the authorized destination. - port: 53 - transports: [UDP, TCP] - authorization: - # Match the DNS-issued context epoch; stale issuers cannot restore access. - writerEpoch: 3 - # Renewals advance monotonically within this access-binding lifetime. - sequence: 27 - # Preserve this deadline through transport, replay, and local installation. - validUntil: "2026-10-08T20:05:00Z" -``` - -The context reports `Ready` when its namespace and assignment are available. -An access binding reports `Accepted` after authorization validation and `Ready` -after the required serving members apply current access and configuration. -`status.observedSequence` identifies the authorization renewal DNS has observed; -`status.bindingRef` identifies the DNS-owned serving plan. - -Context, region, query identity, port, transports, and authorization epoch are -immutable for an access-binding lifetime. Renewals advance the sequence with a -bounded deadline. Destinations must be unique within their routing scope. A -second region gets another binding against the same context. Galactic owns the -consumer-side address, private route, and network lease separately. - -## Private zones and associations - -Users can add a custom zone to the context and create static records in it. -Private zones do not require public domain verification or public delegation. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSZone -metadata: - name: production -spec: - # Visibility is immutable. The existing default remains Public. - visibility: Private - # Different contexts can associate separate zones with this same apex. - domainName: prod.internal - # Select private backend policy; users do not choose fleet instances. - dnsZoneClassName: datum-internal-dns ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSZoneAssociation -metadata: - name: production-application -spec: - # Both objects belong to this project. Pin each object's lifetime. - dnsZoneRef: - name: production - uid: 88888888-8888-4888-8888-888888888888 - # DNS consumes a context reference, not a VPC reference. - resolverContextRef: - name: application - uid: 33333333-3333-4333-8333-333333333333 ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSRecordSet -metadata: - name: database -spec: - # Existing record API: a same-project name reference to its zone. - dnsZoneRef: - name: production - recordType: A - records: - # Static records have no endpoint eligibility lease. - - name: database - ttl: 30 - a: - content: 10.20.0.50 -``` - -A context can associate several zones, including its managed zone. Associating -two different zones with the same apex to one context is rejected. Parent and -child zones require deterministic longest-suffix selection. Two contexts can -share one zone through explicit associations, or associate separate zones with -the same apex for isolated answers. Admission must also reject conflicts -between static records and reserved registration names. - -## Automatic naming and additional names - -DNS owns the lifecycle of the managed namespace and its association. Publishers -read its zone reference from context status. A DNS-owned `DNSManagedNamespace` -can track allocation and cleanup; consumers should not need to create it. - -A naming policy adds names in custom zones. It supplements the managed name and -does not require a zone choice on each Compute or Connect resource. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSNamingPolicy -metadata: - name: application-production-names -spec: - # Apply naming rules to this DNS scope. - resolverContextRef: - name: application - uid: 33333333-3333-4333-8333-333333333333 - additionalNames: - # Add instance identity names, such as web-01.instances.prod.internal. - - registrationClass: InstanceIdentity - dnsZoneRef: - name: production - uid: 88888888-8888-4888-8888-888888888888 - namePrefix: instances - # Add discovery names, such as api.services.prod.internal. - - registrationClass: ServiceDiscovery - dnsZoneRef: - name: production - uid: 88888888-8888-4888-8888-888888888888 - namePrefix: services -``` - -Every target zone must be associated with the context. The proposed classes are -`InstanceIdentity`, `ServiceDiscovery`, `ServiceVIP`, and `ServiceExport`. -Product integrations apply the appropriate rule when reserving names. DNS does -not inspect the underlying product resource to determine its class. - -## Product publication - -Compute can reserve a name in the managed namespace, obtain a scoped publisher -grant, and publish eligible interface addresses. Connect uses the same contract -with its own publisher principal and export eligibility policy. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSRegistration -metadata: - name: instance-web-01 -spec: - # Select the managed zone allocated for the resource's DNS context. - dnsZoneRef: - name: managed-application - uid: 44444444-4444-4444-8444-444444444444 - # Reserve this relative owner name, rather than one particular IP address. - name: web-01.instances - # Constrain which record types contributions can populate. - recordTypes: [A, AAAA] - # Publish only contributions with a current eligible observation. - publicationPolicy: EligibleContributions - # Bound client caching; this does not replace the observation deadline. - ttlSeconds: 30 - # Optional descendant reservations must not overlap other owners. - reservedDescendants: [] -status: - # DNS-owned output for product status and consumer display. - canonicalFQDN: web-01.instances.vpc-a7c9.project-p4e2.internal ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSContributionGrant -metadata: - name: web-01-compute -spec: - # Pin both the registration lifetime and the policy being authorized. - registrationRef: - name: instance-web-01 - uid: 55555555-5555-4555-8555-555555555555 - generation: 1 - # Audit label; the authenticated principal below authorizes the writer. - producerID: compute - principal: - # Trusted source-cluster identity distinguishes distributed publishers. - clusterUID: 99999999-9999-4999-8999-999999999999 - subject: system:serviceaccount:compute-system:dns-publisher - # Scope the writer to these types and owner names. - recordTypes: [A, AAAA] - nameScopes: [web-01.instances] -status: - # DNS allocates the epoch; the producer cannot choose its own fencing token. - activeWriterEpoch: 3 - # Identify the grant and registration policy versions DNS accepted. - observedGrantGeneration: 1 - observedRegistrationGeneration: 1 ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSRecordContribution -metadata: - name: web-01-endpoint -spec: - # Bind the contribution to the reserved name and its current policy. - registrationRef: - name: instance-web-01 - uid: 55555555-5555-4555-8555-555555555555 - generation: 1 - # Bind the producer to its authorized grant lifetime and policy. - grantRef: - name: web-01-compute - uid: 66666666-6666-4666-8666-666666666666 - generation: 1 - # Reuse the existing typed record representation. - recordSets: - - recordType: A - records: - - name: web-01.instances - ttl: 30 - a: - content: 10.20.0.10 - - recordType: AAAA - records: - - name: web-01.instances - ttl: 30 - aaaa: - content: fd20::10 -status: - # Producer-owned: this observation covers the current contribution spec. - observedGeneration: 1 - # Producer-owned: echo the DNS-issued grant epoch. - writerEpoch: 3 - # Producer-owned: order observations within that epoch. - sequence: 18 - # Producer-owned: product policy determines readiness for this name's purpose. - eligible: true - reason: NetworkInterfaceProgrammed - # Producer-owned: DNS must preserve this original freshness deadline. - validUntil: "2026-10-08T20:01:00Z" - # DNS-owned: the publication revision that included this contribution. - publishedRevision: 12 -``` - -Status updates use the current resource version and preserve fields owned by -other controllers. Changing contribution records requires an observation for -the new generation. Admission verifies the authenticated principal, pinned -references, accepted grant epoch, increasing sequence, permitted records, and -bounded freshness interval. - -When an endpoint becomes ineligible, its publisher submits a higher-sequence -observation with `eligible: false`. DNS withdraws its addresses through the same -publication path. If the publisher disappears, the original deadline still -expires locally. Recovery requires a fresh eligible observation; replaying the -old one does not restore an endpoint. The product defines whether interface -readiness, application health, or another condition determines eligibility. - -The prototype also defines `Persistent`, but its distinct lifetime semantics -need API review. Use `DNSRecordSet` for static records in this proposal. - -## DNS service project contracts - -These resources are internal to the DNS service project. Consumers and product -publishers cannot write them. Examples show selected fields that explain -coordination; they are not complete generated serving plans or transport payloads. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSResolverBinding -metadata: - name: application-central - namespace: dns-platform -spec: - # Preserve the authenticated source project identity in shared storage. - projectUID: 11111111-1111-4111-8111-111111111111 - # Prototype wire name: this carries a DNS context UID, not a VPC lookup key. - vpcUID: 33333333-3333-4333-8333-333333333333 - # Identify the shared regional serving assignment. - region: central - shard: shared-0 - # Fence plan replacement separately from configuration changes. - bindingGeneration: 1 - configurationRevision: 7 - # Trusted node-tier and regional-tier destinations for this context. - consumerAddress: fd70:100::10 - clusterAddress: fd70:200::10 - port: 53 - transports: [UDP, TCP] - # Use isolated caches in shared processes, with no private dnsdist packet cache. - resolverEngine: BIND9 - resolverIsolation: View - deploymentModel: SharedShard - dnsdistPacketCache: Disabled - # Only explicitly associated private zones enter the context's plan. - zoneUIDs: - - 44444444-4444-4444-8444-444444444444 - - 88888888-8888-4888-8888-888888888888 - # Preserve the accepted access authorization through every serving tier. - authorizationIssuerEpoch: 3 - authorizationRevision: 27 - authorizationValidUntil: "2026-10-08T20:05:00Z" -``` - -The publication resources divide coordination from immutable data: - -- `DNSPublicationOwnership` holds the zone's compiler lease, writer epoch, - revision allocator, and active manifest pointer. Updates use compare-and-swap. -- `DNSPublicationChunk` holds a bounded part of a snapshot with its index, epoch, - revision, hash, and payload. Chunks are immutable. -- `DNSPublicationManifest` identifies the complete snapshot, its chunks, - serving targets, and original contribution deadlines. -- `DNSTransportOutbox` records a committed update's subject, identity, hash, - payload, and dependencies before sending. Ambiguous retries reuse the same - event identity. Transport acknowledgment is separate from serving acknowledgment. - -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSPublicationOwnership -metadata: - name: managed-zone-owner - namespace: dns-platform -spec: - # One logical compiler owner per zone lifetime. - zoneUID: 44444444-4444-4444-8444-444444444444 - # Identify the current controller lease holder. - holderIdentity: compiler-a - # Takeover increases the epoch; revisions increase within it. - writerEpoch: 2 - nextRevision: 13 - # A writer must hold a current lease to commit the active pointer. - leaseUntil: "2026-10-08T20:01:00Z" - # The committed snapshot is the activation authority. - activeManifestName: managed-zone-e2-r12 ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSPublicationManifest -metadata: - name: managed-zone-e2-r12 - namespace: dns-platform -spec: - # Retain the source zone lifetime and its private apex. - zoneRef: - name: managed-application - uid: 44444444-4444-4444-8444-444444444444 - zoneApex: vpc-a7c9.project-p4e2.internal - # Select this zone's private authoritative variant. - variant: zone-44444444 - # Prototype wire name: values are DNS context UIDs, not networking references. - vpcUIDs: - - 33333333-3333-4333-8333-333333333333 - # Agents reject snapshots behind their accepted writer fence. - writerEpoch: 2 - revision: 12 - previousRevision: 11 - # Deletion uses a fenced tombstone rather than silently dropping history. - tombstone: false - # Verify all chunks before activation; these hashes are illustrative. - chunks: - - name: managed-zone-e2-r12-c0 - sha256: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - size: 512 - contentHash: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb - generatedAt: "2026-10-08T20:00:00Z" - contributionFences: - # Original observation deadlines survive recompilation and replay. - - uid: 77777777-7777-4777-8777-777777777777 - grantUID: 66666666-6666-4666-8666-666666666666 - epoch: 3 - sequence: 18 - validUntil: "2026-10-08T20:01:00Z" - # Each required replica receives the complete regional assignment. - servingTargets: - - region: central - shard: shared-0 -``` - -The compiler stages immutable chunks and a manifest, then commits the active -pointer under its current ownership lease. Exporters reconcile that pointer and -repair a missing outbox after a crash. Agents verify identities, hashes, writer -fences, and deadlines before applying a complete snapshot. Acknowledgments must -identify the exact binding or zone lifetime, epoch, and revision applied. -Serving agents retain fences for expired, withdrawn, and deleted state so an -older replay cannot restore it. - -## Compatibility and API review - -The target integration uses `resolverContextRef` and never requires DNS to read -networking resources. The prototype retains legacy `vpcRef` inputs and internal -`vpcUID`/`vpcUIDs` wire names. Context-based execution treats the latter as opaque -DNS context identities. Rename these internal fields and define migration rules -before stabilizing the API; do not make the old VPC coupling a supported contract. - -API review must settle namespace reservation policy, naming conflicts, grant -issuance and revocation, field-level status ownership, deadline bounds, regional -readiness aggregation, snapshot size limits, and the complete internal serving -plan schema. Private visibility and exclusion from public controllers form a -separate implementation boundary in -[DNS operator #230](https://github.com/datum-cloud/dns-operator/pull/230). From 53576a8f4e82ad00d0131f70b1b975b6171c9eb2 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 15:42:53 -0500 Subject: [PATCH 05/25] docs: organize internal DNS resolver binding contract --- docs/architecture/internal-dns/README.md | 117 ++++++++++++++++------- 1 file changed, 85 insertions(+), 32 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 33114ad9..a070a32d 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -617,6 +617,11 @@ These resources are internal to the DNS service project. Consumers and product publishers cannot write them. Examples show selected fields that explain coordination; they are not complete generated serving plans or transport payloads. +The proposed `DNSResolverBinding` groups four concerns: source identity, +regional placement, generated serving configuration, and access authorization. +It binds a consumer DNS context to shared capacity; it does not describe a VPC +or create a resolver deployment. + ```yaml apiVersion: dns.networking.miloapis.com/v1alpha1 kind: DNSResolverBinding @@ -624,36 +629,80 @@ metadata: name: application-central namespace: dns-platform spec: - # Preserve the authenticated source project identity in shared storage. - projectUID: 11111111-1111-4111-8111-111111111111 - # Prototype wire name: this carries a DNS context UID, not a VPC lookup key. - vpcUID: 33333333-3333-4333-8333-333333333333 - # Identify the shared regional serving assignment. - region: central - shard: shared-0 - # Fence plan replacement separately from configuration changes. - bindingGeneration: 1 - configurationRevision: 7 - # Trusted node-tier and regional-tier destinations for this context. - consumerAddress: fd70:100::10 - clusterAddress: fd70:200::10 - port: 53 - transports: [UDP, TCP] - # Use isolated caches in shared processes, with no private dnsdist packet cache. - resolverEngine: BIND9 - resolverIsolation: View - deploymentModel: SharedShard - dnsdistPacketCache: Disabled - # Only explicitly associated private zones enter the context's plan. - zoneUIDs: - - 44444444-4444-4444-8444-444444444444 - - 88888888-8888-4888-8888-888888888888 - # Preserve the accepted access authorization through every serving tier. - authorizationIssuerEpoch: 3 - authorizationRevision: 27 - authorizationValidUntil: "2026-10-08T20:05:00Z" + source: + # Scope source references to the authenticated consumer project. + projectUID: 11111111-1111-4111-8111-111111111111 + # Pin the DNS context lifetime; DNS does not dereference a VPC. + resolverContextRef: + name: application + uid: 33333333-3333-4333-8333-333333333333 + # Retain the exact regional access authorization that produced this plan. + accessBindingRef: + name: application-central + uid: aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa + placement: + # Identify shared capacity; these fields do not select the source API. + region: central + shard: shared-0 + configuration: + # Fence plan replacement. This is distinct from metadata.generation. + generation: 1 + # Order configuration changes within this plan generation. + revision: 7 + listeners: + # Node-tier service destination, reached through the private network path. + node: + address: fd70:100::10 + port: 53 + transports: [UDP, TCP] + # Regional-tier destination used by the context's node resolver view. + regional: + address: fd70:200::10 + port: 53 + transports: [UDP, TCP] + zoneRefs: + # Include only private zones explicitly associated with the source context. + - name: managed-application + uid: 44444444-4444-4444-8444-444444444444 + - name: production + uid: 88888888-8888-4888-8888-888888888888 + authorization: + # Copy the accepted access epoch, sequence, and deadline without renewal. + writerEpoch: 3 + sequence: 27 + validUntil: "2026-10-08T20:05:00Z" ``` +`source` references and `configuration.zoneRefs` identify objects in the consumer +project selected by `source.projectUID`. They are provenance references, not +lookups in the DNS service project's namespace. Admission pins their UIDs and +restricts writes to DNS controllers. Source identity is immutable for a binding +lifetime. + +`placement` selects shared regional capacity. `configuration.listeners.node` +is the service-side destination, not the well-known address inside the consumer +VPC. Each listener has its own address, port, and transports so the two serving +tiers do not share ambiguous endpoint fields. The region and node listener must +match the source access binding; DNS allocates the regional listener. + +`configuration.generation` replaces the prototype's `bindingGeneration`; +`configuration.revision` replaces `configurationRevision`. Access renewals +advance `authorization.sequence`. Configuration and authorization counters fence +different state; agents check both. A configuration change cannot extend the +access deadline. Serving acknowledgments +must identify the binding UID, configuration generation and revision, and the +exact authorization epoch, sequence, and deadline applied. Readiness requires +current acknowledgments from all required members. + +Resolver engine, cache isolation, deployment model, and dnsdist cache policy +belong to DNS-owned fleet configuration. Bindings cannot override those shared +policies. The initial fleet uses BIND views, shared processes, and a disabled +private dnsdist packet cache as described in the query path. + +This example proposes a reorganized contract. The local prototype still uses +a flat specification; schema, controller, transport, and serving-agent changes +must preserve its existing identity, version, and expiry fences. + The publication resources divide coordination from immutable data: - `DNSPublicationOwnership` holds the zone's compiler lease, writer epoch, @@ -738,10 +787,14 @@ older replay cannot restore it. #### Compatibility and API review The target integration uses `resolverContextRef` and never requires DNS to read -networking resources. The prototype retains legacy `vpcRef` inputs and internal -`vpcUID`/`vpcUIDs` wire names. Context-based execution treats the latter as opaque -DNS context identities. Rename these internal fields and define migration rules -before stabilizing the API; do not make the old VPC coupling a supported contract. +networking resources. The prototype retains legacy `vpcRef` inputs and flat +`DNSResolverBinding` fields. The binding example above proposes nested groups, +UID-pinned DNS source references, explicit node and regional listeners, and fleet +policy outside individual bindings. It requires coordinated schema, controller, +transport, and agent migration. The manifest prototype still uses `vpcUIDs` for +opaque DNS context identities; rename that field before stabilizing the API. +Migration must retain accepted UIDs, configuration versions, authorization +fences, and original deadlines. API review must settle namespace reservation policy, naming conflicts, grant issuance and revocation, field-level status ownership, deadline bounds, regional From 2658d737d8c19c6d3bf52391e86389f4f625607a Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 15:49:40 -0500 Subject: [PATCH 06/25] docs: tighten internal DNS enhancement proposal --- docs/architecture/internal-dns/README.md | 503 ++++++----------------- 1 file changed, 130 insertions(+), 373 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index a070a32d..2efe6b25 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -11,8 +11,6 @@ latest-milestone: "TBD" - [Goals](#goals) - [Non-Goals](#non-goals) - [Proposal](#proposal) - - [User stories](#user-stories) - - [Notes, constraints, and caveats](#notes-constraints-and-caveats) - [Risks and mitigations](#risks-and-mitigations) - [Design Details](#design-details) - [Control-plane boundaries](#control-plane-boundaries) @@ -20,21 +18,8 @@ latest-milestone: "TBD" - [Query path and network identity](#query-path-and-network-identity) - [Publication and service discovery](#publication-and-service-discovery) - [API design](#api-design) - - [Ownership and validation](#ownership-and-validation) - - [Context and regional access](#context-and-regional-access) - - [Private zones and associations](#private-zones-and-associations) - - [Automatic naming and additional names](#automatic-naming-and-additional-names) - - [Product publication](#product-publication) - - [DNS service project contracts](#dns-service-project-contracts) - - [Compatibility and API review](#compatibility-and-api-review) - [Integration boundaries](#integration-boundaries) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - - [Feature Enablement and Rollback](#feature-enablement-and-rollback) - - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) - - [Monitoring Requirements](#monitoring-requirements) - - [Dependencies](#dependencies) - - [Scalability](#scalability) - - [Troubleshooting](#troubleshooting) - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) @@ -55,11 +40,9 @@ defines private connectivity. This enhancement defines the DNS architecture and ## Motivation -Compute instances, Connect services, and other resources need names that resolve -within their network. Consumers should receive useful default names without -choosing a zone for every resource. Shared infrastructure must isolate networks -that use identical names and addresses, and stop returning endpoints that are no -longer eligible. +Resources need private names without a zone choice on every resource. The +platform must isolate overlapping names and addresses across networks and +withdraw endpoints that are no longer eligible. ### Goals @@ -84,21 +67,6 @@ DNS allocates a managed namespace, accepts explicit custom zone associations, and compiles product publications into private serving state. Consumers use a stable resolver address in their VPC while the platform selects shared capacity. -### User stories - -- A Compute user resolves an instance's default private hostname without - selecting a zone or resolver deployment. -- A project adds `prod.internal` and uses names such as `api.prod.internal`. -- Two VPCs use the same zone name and resolver address but receive isolated - answers. Removing an eligible endpoint withdraws its discovery address. - -### Notes, constraints, and caveats - -These contracts are proposed; API and serving qualification remain required. -Regional publication is asynchronous. Resolver access and endpoint eligibility -have separate deadlines. Client caches can retain an answer until its TTL -expires. Example addresses, identities, and allocated suffixes are illustrative. - ### Risks and mitigations - Tenant leakage: authorize network identity before cache lookup, isolate @@ -133,23 +101,17 @@ flowchart TB D -.->|DNS status| P ``` -- **Consumer project:** Owns private zones, records, associations, naming policies, - registrations, grants, and contributions. Trusted VPC integration writes - context and access specifications; DNS writes their status. Product services, - such as Compute and Connect, publish their records and eligibility. -- **DNS service project:** Owns durable publication state, writer leases, - allocation claims, regional serving plans, and the service's network intent. -- **Karmada:** Places network and workload intent at the edge. It is not the DNS - publication store. Guest resolver settings must travel as desired state in the - networking projection; source status is not a substitute for that contract. -- **Edge:** Owns local network contexts, interfaces, private endpoints, route - policies, shared DNS workloads, and serving checkpoints. - -DNS controllers use project discovery and authenticated project clients. They -do not watch VPCs, network interfaces, or product resources. VPC integration -translates networking authorization into DNS access. Product publishers translate -resource state into DNS records. These integrations keep networking and product -lifecycles outside the DNS controller. +- **Consumer project:** Owns contexts, access bindings, zones, naming policy, and + publications. VPC integration writes access intent; product services publish + records and eligibility; DNS reports serving status. +- **DNS service project:** Owns writer leases, snapshots, outboxes, fleet plans, + and the service's network intent. +- **Karmada:** Places network and workload intent. Guest resolver settings travel + as desired state; DNS publications use their own delivery path. +- **Edge:** Owns local networking, shared serving workloads, and checkpoints. + +DNS uses authenticated project clients. VPC integration and product publishers +translate their resources into DNS intent; DNS does not watch their APIs. ### Contexts and regional access @@ -166,15 +128,10 @@ flowchart LR AE --> FE[East shared fleet] ``` -Both access bindings live in the consumer project's control plane. `region` -selects a serving target; it does not select the API where the object lives. -Each binding has its own UID, trusted destination, authorization deadline, and -readiness. East access can expire while Central access remains available. - -Publication reaches regions asynchronously. A region reports readiness after -its required serving members apply the current state. A failed region must not -block updates to healthy regions. Regional access does not, by itself, promise -that service discovery answers contain only endpoints from that region. +Bindings live in the consumer project API; `region` selects serving capacity. +Each has independent identity, authorization, and readiness. Publication is +asynchronous, and a failed region must not block healthy regions. Regional access +does not imply that discovery returns only region-local endpoints. ### Query path and network identity @@ -207,41 +164,30 @@ flowchart TB end ``` -Client source addresses, ECS, and client-supplied PROXY headers are not tenant -authority. Galactic checks the source VPC lifetime and route authorization before -translating the destination. A consumer must not be able to select another -context by addressing its service-side destination. Galactic handles the return -path to the consumer's DNS endpoint. +Galactic verifies the source VPC lifetime and route authorization before +translating the destination. DNS trusts that path, not client addresses, ECS, or +client-supplied PROXY headers. Consumers cannot select another context by +addressing its service-side destination. -The serving candidate uses shared node and regional tiers: +Shared node and regional tiers use: -1. **dnsdist** selects the context from the authorized destination. Its private - packet cache is disabled. -2. **BIND resolver views** isolate positive and negative caches. Node views - forward to context-specific regional destinations; regional views resolve - private zones and provide recursive resolution for other names. -3. **PowerDNS Authoritative** serves private zone variants selected by - service-owned source markers from the regional resolvers. +1. **dnsdist** to select the context before lookup, with private packet caching + disabled. +2. **BIND views** to isolate positive and negative caches. Node views forward to + context-specific regional destinations; regional views resolve private zones + and recurse for other names. +3. **PowerDNS Authoritative** to select private zone variants using service-owned + source markers, unique to each regional resolver replica and context. [dnsdist PROXYv2](https://www.dnsdist.org/advanced/passing-source-address.html) -preserves the destination when forwarding to a shared resolver listener. -[BIND's PROXY access controls](https://bind9.readthedocs.io/en/v9.20.2/reference.html#namedconf-statement-allow-proxy) -restrict that listener to approved service peers and listener addresses. Each -regional resolver replica uses a distinct marker for each context. A shared pool member -is eligible only when all its assigned contexts and authorizations are current. +preserves the destination. [BIND PROXY access controls](https://bind9.readthedocs.io/en/v9.20.2/reference.html#namedconf-statement-allow-proxy) +restrict trusted peers and listeners. A shared pool member must have current +configuration and authorization for every assigned context. [PowerDNS views](https://doc.powerdns.com/authoritative/views.html) are experimental -and require the LMDB backend and enabled zone cache. Unmatched clients bypass -view selection, and ordinary variantless zones are implicitly visible to views. -Therefore, private zones must have no variantless copy, and private authoritative -listeners must admit only approved resolver markers. - -The original proposal called for PowerDNS Recursor at both resolver tiers. The -prototype has not qualified Recursor's cache isolation for this contract. BIND -views are the current candidate; PowerDNS Authoritative is a different component. -Replacing BIND requires positive and negative cache isolation tests against the -selected Recursor version. Adding a context changes shared configuration and -zone data, not the number of resolver processes. +and require LMDB with zone caching. Private zones must have no variantless copy, +and authoritative listeners must allow only approved resolver markers: unmatched +clients bypass views, and ordinary zones are implicitly visible to views. ### Publication and service discovery @@ -270,39 +216,27 @@ sequenceDiagram DNS->>Store: Commit snapshot that withdraws the endpoint ``` -Each zone has one logical compiler owner. Compare-and-swap updates to durable -ownership fence that writer. Takeover advances its epoch; revisions increase -within an epoch. Serving agents reject older epochs and revisions. Controller -leader election alone does not fence a writer after a partition. - -Kubernetes API storage holds ownership, snapshots, and outboxes; this design -does not require Postgres. Regional exporters use -[NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) for at-least-once -delivery. Every serving replica receives its complete assigned stream. The -exporter records an outbox before sending and reconciles committed snapshots -that lack an outbox after a crash. A broker acknowledgment confirms delivery to -the broker, not application by a DNS serving member. Queries use local state and -do not contact the broker or project APIs. - -Products define eligibility for each record's purpose. An instance identity name -can follow interface readiness; service discovery can follow application health. -A stable service address can remain published while its backend membership -changes. DNS does not infer these policies from product APIs. User-authored -static records have no automatic endpoint health withdrawal. - -Contribution freshness and resolver access have separate deadlines. Replay, -restart, or recompilation cannot extend either original deadline. Serving agents -withdraw expired contributions locally; a watchdog removes a member that cannot -enforce expiry. Deletions retain tombstones and writer fences for the supported -replay window. - -A reserved name with no eligible addresses returns NODATA. An authorized context -whose required private state is unavailable returns SERVFAIL; an unknown or -expired access identity returns REFUSED. Missing private state must never fall -back to public resolution. Dynamic TTLs are bounded, and stale private answers -are disabled. Withdrawal budgets include publication delay, local expiry, and -client cache TTL: removing a served record cannot erase an answer already cached -by a client. +Each zone has one compiler owner, fenced by a compare-and-swap lease. Takeover +advances the writer epoch; revisions increase within it. Agents reject older +versions. The compiler stages immutable chunks and a manifest, then commits its +active pointer. Kubernetes API storage retains this state and a durable outbox; +exporters repair missing outboxes after crashes. + +Regional [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) delivery is +at least once. Every serving replica receives its complete assignment and +acknowledges applied state separately from broker receipt. Queries use local +state without contacting project APIs or the broker. + +Products define eligibility for instance identity, application discovery, or +stable service addresses. Serving agents enforce contribution freshness and +access deadlines independently. Replay and restart cannot extend them; expired +endpoints are withdrawn locally. Retained tombstones and writer fences prevent +restoration, and watchdogs remove members that cannot enforce expiry. + +A reserved name without eligible addresses returns NODATA. Missing required +private state returns SERVFAIL; unknown or expired access returns REFUSED. +Private names never fall back to public resolution. Disable stale private +answers and bound dynamic TTLs; withdrawal budgets include client caching. ### API design @@ -313,21 +247,10 @@ are not fields consumers submit when creating resources. #### Ownership and validation -- Trusted VPC integration creates resolver contexts and regional access bindings. - DNS allocates managed namespaces and reports serving status. -- Project users manage custom private zones, associations, static records, and - naming policy. Platform policy controls namespace reservations and grants. -- Product publishers write contribution records and eligibility. They cannot - authorize resolver access, grant themselves publication rights, or write DNS - service project state. -- DNS owns grant epochs, publication revisions, serving plans, and acknowledgments. - Admission must enforce field ownership, including fields sharing a status - subresource; RBAC on that subresource alone is insufficient. - -New references pin API-assigned UIDs. References that authorize publication also -pin the policy generation. The authenticated project and source-cluster identity -scope each reference; resource names and self-reported cluster IDs are not -credentials. Cross-project references are outside this proposal. +Admission pins reference UIDs and publication policy generations, authenticates +project and source-cluster identity, and enforces field ownership, including +shared status fields. Publishers cannot grant themselves rights or authorize +resolver access. Consumer references stay within a project. #### Context and regional access @@ -387,17 +310,12 @@ spec: validUntil: "2026-10-08T20:05:00Z" ``` -The context reports `Ready` when its namespace and assignment are available. -An access binding reports `Accepted` after authorization validation and `Ready` -after the required serving members apply current access and configuration. -`status.observedSequence` identifies the authorization renewal DNS has observed; -`status.bindingRef` identifies the DNS-owned serving plan. - -Context, region, query identity, port, transports, and authorization epoch are -immutable for an access-binding lifetime. Renewals advance the sequence with a -bounded deadline. Destinations must be unique within their routing scope. A -second region gets another binding against the same context. Galactic owns the -consumer-side address, private route, and network lease separately. +Access reports `Accepted` after validation and `Ready` after required members +apply current authorization and configuration. Context, region, query identity, +port, transports, and authorization epoch are immutable per access lifetime. +Renewals advance the sequence with a bounded deadline; destinations are unique +within their routing scope. Galactic owns the consumer frontend and network +lease separately. #### Private zones and associations @@ -448,21 +366,15 @@ spec: content: 10.20.0.50 ``` -A context can associate several zones, including its managed zone. Associating -two different zones with the same apex to one context is rejected. Parent and -child zones require deterministic longest-suffix selection. Two contexts can -share one zone through explicit associations, or associate separate zones with -the same apex for isolated answers. Admission must also reject conflicts -between static records and reserved registration names. +A context can use several zones. Reject duplicate apexes within a context and +conflicts between static records and reserved names. Select parent and child +zones by longest suffix. Separate contexts can explicitly share a zone or use +independent zones with the same apex. Static records have no health lease. #### Automatic naming and additional names -DNS owns the lifecycle of the managed namespace and its association. Publishers -read its zone reference from context status. A DNS-owned `DNSManagedNamespace` -can track allocation and cleanup; consumers should not need to create it. - -A naming policy adds names in custom zones. It supplements the managed name and -does not require a zone choice on each Compute or Connect resource. +DNS allocates the managed zone and exposes it in context status. A naming policy +adds custom names without a zone choice on each Compute or Connect resource. ```yaml apiVersion: dns.networking.miloapis.com/v1alpha1 @@ -595,21 +507,11 @@ status: publishedRevision: 12 ``` -Status updates use the current resource version and preserve fields owned by -other controllers. Changing contribution records requires an observation for -the new generation. Admission verifies the authenticated principal, pinned -references, accepted grant epoch, increasing sequence, permitted records, and -bounded freshness interval. - -When an endpoint becomes ineligible, its publisher submits a higher-sequence -observation with `eligible: false`. DNS withdraws its addresses through the same -publication path. If the publisher disappears, the original deadline still -expires locally. Recovery requires a fresh eligible observation; replaying the -old one does not restore an endpoint. The product defines whether interface -readiness, application health, or another condition determines eligibility. - -The prototype also defines `Persistent`, but its distinct lifetime semantics -need API review. Use `DNSRecordSet` for static records in this proposal. +Publishers preserve DNS-owned status fields and observe the current contribution +generation. An ineligible endpoint gets a higher-sequence observation with +`eligible: false`; recovery requires a fresh eligible observation. Publisher +loss does not renew the original deadline. Admission checks the principal, +grant epoch, sequence, record scope, and freshness bound. #### DNS service project contracts @@ -673,148 +575,37 @@ spec: validUntil: "2026-10-08T20:05:00Z" ``` -`source` references and `configuration.zoneRefs` identify objects in the consumer -project selected by `source.projectUID`. They are provenance references, not -lookups in the DNS service project's namespace. Admission pins their UIDs and -restricts writes to DNS controllers. Source identity is immutable for a binding -lifetime. - -`placement` selects shared regional capacity. `configuration.listeners.node` -is the service-side destination, not the well-known address inside the consumer -VPC. Each listener has its own address, port, and transports so the two serving -tiers do not share ambiguous endpoint fields. The region and node listener must -match the source access binding; DNS allocates the regional listener. - -`configuration.generation` replaces the prototype's `bindingGeneration`; -`configuration.revision` replaces `configurationRevision`. Access renewals -advance `authorization.sequence`. Configuration and authorization counters fence -different state; agents check both. A configuration change cannot extend the -access deadline. Serving acknowledgments -must identify the binding UID, configuration generation and revision, and the -exact authorization epoch, sequence, and deadline applied. Readiness requires -current acknowledgments from all required members. - -Resolver engine, cache isolation, deployment model, and dnsdist cache policy -belong to DNS-owned fleet configuration. Bindings cannot override those shared -policies. The initial fleet uses BIND views, shared processes, and a disabled -private dnsdist packet cache as described in the query path. - -This example proposes a reorganized contract. The local prototype still uses -a flat specification; schema, controller, transport, and serving-agent changes -must preserve its existing identity, version, and expiry fences. - -The publication resources divide coordination from immutable data: - -- `DNSPublicationOwnership` holds the zone's compiler lease, writer epoch, - revision allocator, and active manifest pointer. Updates use compare-and-swap. -- `DNSPublicationChunk` holds a bounded part of a snapshot with its index, epoch, - revision, hash, and payload. Chunks are immutable. -- `DNSPublicationManifest` identifies the complete snapshot, its chunks, - serving targets, and original contribution deadlines. -- `DNSTransportOutbox` records a committed update's subject, identity, hash, - payload, and dependencies before sending. Ambiguous retries reuse the same - event identity. Transport acknowledgment is separate from serving acknowledgment. +Source and zone references resolve in `source.projectUID`, not the DNS service +namespace. Source identity is immutable. The region and node listener match the +access binding; DNS allocates the regional listener. -```yaml -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSPublicationOwnership -metadata: - name: managed-zone-owner - namespace: dns-platform -spec: - # One logical compiler owner per zone lifetime. - zoneUID: 44444444-4444-4444-8444-444444444444 - # Identify the current controller lease holder. - holderIdentity: compiler-a - # Takeover increases the epoch; revisions increase within it. - writerEpoch: 2 - nextRevision: 13 - # A writer must hold a current lease to commit the active pointer. - leaseUntil: "2026-10-08T20:01:00Z" - # The committed snapshot is the activation authority. - activeManifestName: managed-zone-e2-r12 ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSPublicationManifest -metadata: - name: managed-zone-e2-r12 - namespace: dns-platform -spec: - # Retain the source zone lifetime and its private apex. - zoneRef: - name: managed-application - uid: 44444444-4444-4444-8444-444444444444 - zoneApex: vpc-a7c9.project-p4e2.internal - # Select this zone's private authoritative variant. - variant: zone-44444444 - # Prototype wire name: values are DNS context UIDs, not networking references. - vpcUIDs: - - 33333333-3333-4333-8333-333333333333 - # Agents reject snapshots behind their accepted writer fence. - writerEpoch: 2 - revision: 12 - previousRevision: 11 - # Deletion uses a fenced tombstone rather than silently dropping history. - tombstone: false - # Verify all chunks before activation; these hashes are illustrative. - chunks: - - name: managed-zone-e2-r12-c0 - sha256: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa - size: 512 - contentHash: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb - generatedAt: "2026-10-08T20:00:00Z" - contributionFences: - # Original observation deadlines survive recompilation and replay. - - uid: 77777777-7777-4777-8777-777777777777 - grantUID: 66666666-6666-4666-8666-666666666666 - epoch: 3 - sequence: 18 - validUntil: "2026-10-08T20:01:00Z" - # Each required replica receives the complete regional assignment. - servingTargets: - - region: central - shard: shared-0 -``` +Configuration generation fences plan replacement; revision orders its updates. +Authorization copies the accepted access epoch, sequence, and deadline. Serving +acknowledgments identify the binding UID and both version sets; readiness +requires current acknowledgments from all required members. + +Resolver engine, isolation, deployment model, and dnsdist cache policy belong to +shared fleet configuration. The nested binding is proposed; the prototype is flat. -The compiler stages immutable chunks and a manifest, then commits the active -pointer under its current ownership lease. Exporters reconcile that pointer and -repair a missing outbox after a crash. Agents verify identities, hashes, writer -fences, and deadlines before applying a complete snapshot. Acknowledgments must -identify the exact binding or zone lifetime, epoch, and revision applied. -Serving agents retain fences for expired, withdrawn, and deleted state so an -older replay cannot restore it. +DNS-owned publication resources retain the compiler lease and active pointer +(`DNSPublicationOwnership`), immutable snapshot data (`DNSPublicationManifest` +and `DNSPublicationChunk`), and durable delivery intent (`DNSTransportOutbox`). +Their storage schemas can be reviewed with the implementation. #### Compatibility and API review -The target integration uses `resolverContextRef` and never requires DNS to read -networking resources. The prototype retains legacy `vpcRef` inputs and flat -`DNSResolverBinding` fields. The binding example above proposes nested groups, -UID-pinned DNS source references, explicit node and regional listeners, and fleet -policy outside individual bindings. It requires coordinated schema, controller, -transport, and agent migration. The manifest prototype still uses `vpcUIDs` for -opaque DNS context identities; rename that field before stabilizing the API. -Migration must retain accepted UIDs, configuration versions, authorization -fences, and original deadlines. - -API review must settle namespace reservation policy, naming conflicts, grant -issuance and revocation, field-level status ownership, deadline bounds, regional -readiness aggregation, snapshot size limits, and the complete internal serving -plan schema. Private visibility and exclusion from public controllers form a -separate implementation boundary in -[DNS operator #230](https://github.com/datum-cloud/dns-operator/pull/230). +Prototype schema, controller, transport, and agent migration must preserve UID, +configuration, and authorization fences and original deadlines. Review naming +conflicts, grant revocation, field ownership, freshness bounds, regional +readiness, and snapshot limits before stabilizing the APIs. ### Integration boundaries -Reuse project discovery, authenticated project clients, record types, condition -patterns, certificates, release tooling, and PowerDNS/LMDB operating experience. -The [infra configuration](https://github.com/datum-cloud/infra/tree/main/apps/dns-operator) -provides these integration points; its public serving and replication paths do -not establish private tenant isolation. - -New components include private API admission, context allocation, publication -ownership and compilation, durable export, shared fleet configuration, and -serving acknowledgments. Private snapshots need their own activation contract; -public zone replication through LightningStream does not replace it. +Reuse project discovery, authenticated clients, record types, certificates, +release tooling, and PowerDNS/LMDB experience from +[infra](https://github.com/datum-cloud/infra/tree/main/apps/dns-operator). +Add private admission, compilation, export, and serving agents. Public +LightningStream replication does not provide private snapshot activation. ## Production Readiness Review Questionnaire @@ -823,93 +614,59 @@ before release. ### Feature Enablement and Rollback -Alpha needs separate activation controls for product publication and private -serving. Flag names and deployment wiring require implementation review. Public -visibility remains the default. Rollback must revoke consumer access before -draining private serving capacity; workloads that depend on internal DNS lose -resolution. Reenabling requires fresh authorizations and observations, with -retained writer fences preventing stale replay. +Define separate alpha controls for publication and serving. Public visibility +remains the default. Revoke access before draining private capacity; dependent +workloads lose resolution. Reenabling requires fresh authorization and observations. ### Rollout, Upgrade and Rollback Planning -Deploy shared capacity before granting access, then qualify a limited set of -contexts and expand by region. Validate overlapping zones and addresses, -positive and negative cache isolation, forged identities, UDP and TCP, access -revocation, endpoint withdrawal, writer failover, replay, restart, API and broker -outages, and partial regional failure. Include Galactic's private path and guest -resolver configuration. Upgrade, rollback, and reenabling tests remain release -requirements. Resolve withdrawal budgets and tombstone retention before rollout. +Qualify tenant isolation, expiry, failover, replay, and upgrade/rollback over +Galactic UDP and TCP paths, including guest resolver configuration. Expand by +context and region after setting withdrawal budgets and tombstone retention. ### Monitoring Requirements -Measure query latency and failures, publication lag, outbox backlog, expiry -withdrawal delay, failed activations, and watchdog removal of serving members. -Keep metric labels bounded; use resource status for context-level diagnosis. -Consumers inspect access readiness and published names, then query from their -VPC. Query availability, latency, and withdrawal SLOs require measured targets -before release. +Measure query latency/errors, publication lag, outbox backlog, expiry delay, and +activation failures. Use bounded metric labels and per-context API status. +Availability, latency, and withdrawal SLOs remain to be defined. ### Dependencies -- Project APIs and discovery support intent, renewal, and durable coordination. - An outage stops updates; installed state remains subject to its deadlines. -- Galactic and Karmada provide authorized private paths and placed network and - workload intent. Network authorization expires independently of DNS state. -- NATS JetStream carries committed updates. Outages delay publication; queries - continue from local state while original freshness deadlines still apply. -- dnsdist, BIND, and PowerDNS Authoritative provide query serving. Versions and - configuration must pass the tenant isolation qualification described above. +Project APIs, Galactic/Karmada, NATS, and the DNS fleet support updates and +connectivity. API or broker outages delay updates while installed state remains +subject to its original deadlines; network authorization expires independently. ### Scalability -API traffic includes project watches, eligibility and access renewals, compiler -leases, snapshot writes, export progress, and serving acknowledgments. Bound -snapshot chunks and retained history. Contexts add views, addresses, and zone -data to shared processes. Measure API write rates, cache memory, configuration -reload time, transport fanout, and query throughput at the expected context -count. Supported limits and regional failover capacity remain open decisions. +Measure context capacity, renewal/write rates, snapshot size, transport fanout, +cache memory, reload time, and query throughput. Bound chunks and retained +history; establish supported limits and regional failover capacity before release. ### Troubleshooting -Trace project, context, and access UIDs through the accepted authorization, -committed manifest epoch and revision, and serving acknowledgments. For REFUSED, -inspect access expiry and the authorized network destination. For SERVFAIL, -inspect missing private state and serving readiness. For NODATA, inspect product -eligibility and contribution deadlines. During API or broker outages, inspect -outbox lag and local expiry; replay must not renew stale state. +Trace project/context/access UIDs through authorization, committed snapshots, +and serving acknowledgments. Inspect expiry for REFUSED, private state for +SERVFAIL, and product eligibility for NODATA. ## Implementation History -- 2026-10-08: [DNS operator #229](https://github.com/datum-cloud/dns-operator/pull/229) - proposes the DNS architecture and annotated API contracts. -- 2026-10-08: [DNS operator #230](https://github.com/datum-cloud/dns-operator/pull/230) - proposes the private-zone visibility boundary. The complete feature remains - under development. +2026-10-08: Architecture proposal +[#229](https://github.com/datum-cloud/dns-operator/pull/229) and initial private-zone +boundary [#230](https://github.com/datum-cloud/dns-operator/pull/230). ## Drawbacks -Shared view configuration and experimental authoritative views add operational -complexity. Deadline enforcement, distributed writer fencing, and durable -publication introduce coordination state and ongoing API traffic. A fleet -failure can affect many contexts, so capacity and activation need qualification. +Shared views and experimental authoritative views increase fleet complexity and +failure scope. Leases, snapshots, and freshness renewals add coordination state +and API traffic. ## Alternatives -- **PowerDNS Recursor:** The original resolver choice. Replacing BIND remains an - option after qualifying positive and negative cache isolation for the selected - version. -- **CoreDNS:** An alternative serving component that needs a tenant-aware query - and cache design, plus throughput and operational qualification. -- **Resolvers per VPC:** Provide separate processes but conflict with the shared - fleet requirement and add workload overhead for every network. -- **DNS reads product and networking APIs:** Avoids publisher integrations but - couples DNS to their lifecycles and health policies. Explicit publications and - opaque contexts keep those responsibilities with their owners. +PowerDNS Recursor and CoreDNS remain options if they meet the same tenant +isolation and capacity requirements. ## Infrastructure Needed -Provide a DNS service VPC, Galactic private connectivity, shared node and regional -serving capacity, protected destination and source-marker allocation, durable -project API storage, and secured NATS JetStream capacity. Reuse platform identity, -certificates, and deployment tooling. Regional broker placement, capacity, and -failover policy need review before production deployment. +A DNS service VPC, Galactic private connectivity, shared serving capacity, +protected destination/source-marker allocation, durable project storage, and +secured NATS capacity. Review regional broker placement and failover policy. From 9e2699fe42dc7be3a85406f376003013e27f08bf Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Thu, 8 Oct 2026 16:23:26 -0500 Subject: [PATCH 07/25] docs: use BIND views for private DNS authority --- docs/architecture/internal-dns/README.md | 46 ++++++++++++------------ 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 2efe6b25..9c92045b 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -155,12 +155,10 @@ flowchart TB subgraph Shared[Shared DNS service fleet] DestA --> Dist[dnsdist: trusted destination to context] DestB --> Dist - Dist --> ViewA[Context A resolver view and cache] - Dist --> ViewB[Context B resolver view and cache] - ViewA -->|Context A source marker| Auth[Private authoritative fleet] - ViewB -->|Context B source marker| Auth - Auth --> ZoneA[Context A: api.prod.internal = 10.20.0.10] - Auth --> ZoneB[Context B: api.prod.internal = 10.20.0.20] + Dist --> ViewA[Context A BIND view and cache] + Dist --> ViewB[Context B BIND view and cache] + ViewA --> ZoneA[Private zone in view A: api.prod.internal = 10.20.0.10] + ViewB --> ZoneB[Private zone in view B: api.prod.internal = 10.20.0.20] end ``` @@ -174,20 +172,18 @@ Shared node and regional tiers use: 1. **dnsdist** to select the context before lookup, with private packet caching disabled. 2. **BIND views** to isolate positive and negative caches. Node views forward to - context-specific regional destinations; regional views resolve private zones - and recurse for other names. -3. **PowerDNS Authoritative** to select private zone variants using service-owned - source markers, unique to each regional resolver replica and context. + context-specific regional destinations. Regional views host the context's + private authoritative zones and recurse for other names. Views share resolver + processes; each context has its own zone selection and cache. [dnsdist PROXYv2](https://www.dnsdist.org/advanced/passing-source-address.html) preserves the destination. [BIND PROXY access controls](https://bind9.readthedocs.io/en/v9.20.2/reference.html#namedconf-statement-allow-proxy) restrict trusted peers and listeners. A shared pool member must have current configuration and authorization for every assigned context. -[PowerDNS views](https://doc.powerdns.com/authoritative/views.html) are experimental -and require LMDB with zone caching. Private zones must have no variantless copy, -and authoritative listeners must allow only approved resolver markers: unmatched -clients bypass views, and ordinary zones are implicitly visible to views. +[BIND views](https://bind9.readthedocs.io/en/v9.20.2/reference.html#view-block-grammar) +select the private zones visible to each context. Keep caches separate and reject +unmatched destinations before lookup. ### Publication and service discovery @@ -222,6 +218,9 @@ versions. The compiler stages immutable chunks and a manifest, then commits its active pointer. Kubernetes API storage retains this state and a durable outbox; exporters repair missing outboxes after crashes. +Regional agents render committed snapshots as zone files, validate and activate +them in the associated BIND views, then acknowledge serving. + Regional [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) delivery is at least once. Every serving replica receives its complete assignment and acknowledges applied state separately from broker receipt. Queries use local @@ -595,14 +594,16 @@ Their storage schemas can be reviewed with the implementation. #### Compatibility and API review Prototype schema, controller, transport, and agent migration must preserve UID, -configuration, and authorization fences and original deadlines. Review naming -conflicts, grant revocation, field ownership, freshness bounds, regional -readiness, and snapshot limits before stabilizing the APIs. +configuration, and authorization fences and original deadlines. The prototype's +authoritative backend must be replaced with BIND zone materialization and +revalidated for tenant isolation and expiry. Review naming conflicts, grant +revocation, field ownership, freshness bounds, regional readiness, and snapshot +limits before stabilizing the APIs. ### Integration boundaries -Reuse project discovery, authenticated clients, record types, certificates, -release tooling, and PowerDNS/LMDB experience from +Reuse project discovery, authenticated clients, record types, certificates, and +release tooling from [infra](https://github.com/datum-cloud/infra/tree/main/apps/dns-operator). Add private admission, compilation, export, and serving agents. Public LightningStream replication does not provide private snapshot activation. @@ -656,9 +657,8 @@ boundary [#230](https://github.com/datum-cloud/dns-operator/pull/230). ## Drawbacks -Shared views and experimental authoritative views increase fleet complexity and -failure scope. Leases, snapshots, and freshness renewals add coordination state -and API traffic. +Shared BIND views increase fleet complexity and failure scope. Leases, snapshots, +and freshness renewals add coordination state and API traffic. ## Alternatives @@ -668,5 +668,5 @@ isolation and capacity requirements. ## Infrastructure Needed A DNS service VPC, Galactic private connectivity, shared serving capacity, -protected destination/source-marker allocation, durable project storage, and +protected listener-address allocation, durable project storage, and secured NATS capacity. Review regional broker placement and failover policy. From f8ff47e9e0d042a9f3f53e4a5bf017ce70d2165d Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 14:08:03 -0500 Subject: [PATCH 08/25] Define consumer ownership of managed DNS domain policy --- docs/architecture/internal-dns/README.md | 27 ++++++++++++++---------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 9c92045b..009ae0db 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -62,8 +62,9 @@ withdraw endpoints that are no longer eligible. ## Proposal -Trusted VPC integration provisions a DNS context and regional resolver access. -DNS allocates a managed namespace, accepts explicit custom zone associations, +Network services selects the default VPC domain, `datum.internal`, and provisions +a DNS context and regional resolver access. DNS allocates the requested managed +namespace, accepts explicit custom zone associations, and compiles product publications into private serving state. Consumers use a stable resolver address in their VPC while the platform selects shared capacity. @@ -253,8 +254,9 @@ resolver access. Consumer references stay within a project. #### Context and regional access -VPC integration creates an opaque DNS scope. DNS does not dereference the -`consumerID` or watch a networking resource. +Network services creates an opaque DNS scope and requests its managed domain. +DNS does not dereference the `consumerID` or watch a networking resource. Other +trusted integrations can request a different domain through the same contract. ```yaml apiVersion: dns.networking.miloapis.com/v1alpha1 @@ -267,13 +269,15 @@ spec: # Allocate a default private namespace without asking users to choose a zone. managedNamespace: enabled: true + # Consumer integration policy selects the apex; it is immutable in this context. + domainSuffix: datum.internal status: # DNS-owned output; publishers use this allocated zone and suffix. managedNamespace: dnsZoneRef: name: managed-application uid: 44444444-4444-4444-8444-444444444444 - suffix: vpc-a7c9.project-p4e2.internal + suffix: datum.internal # DNS-owned fencing epoch for trusted access authorization. accessWriterEpoch: 3 # Regional assignments do not move this object out of the project API. @@ -372,7 +376,8 @@ independent zones with the same apex. Static records have no health lease. #### Automatic naming and additional names -DNS allocates the managed zone and exposes it in context status. A naming policy +DNS allocates the requested managed zone and exposes its zone reference and +suffix in context status. Product publishers use these outputs. A naming policy adds custom names without a zone choice on each Compute or Connect resource. ```yaml @@ -422,7 +427,7 @@ spec: name: managed-application uid: 44444444-4444-4444-8444-444444444444 # Reserve this relative owner name, rather than one particular IP address. - name: web-01.instances + name: web-01 # Constrain which record types contributions can populate. recordTypes: [A, AAAA] # Publish only contributions with a current eligible observation. @@ -433,7 +438,7 @@ spec: reservedDescendants: [] status: # DNS-owned output for product status and consumer display. - canonicalFQDN: web-01.instances.vpc-a7c9.project-p4e2.internal + canonicalFQDN: web-01.datum.internal --- apiVersion: dns.networking.miloapis.com/v1alpha1 kind: DNSContributionGrant @@ -453,7 +458,7 @@ spec: subject: system:serviceaccount:compute-system:dns-publisher # Scope the writer to these types and owner names. recordTypes: [A, AAAA] - nameScopes: [web-01.instances] + nameScopes: [web-01] status: # DNS allocates the epoch; the producer cannot choose its own fencing token. activeWriterEpoch: 3 @@ -480,13 +485,13 @@ spec: recordSets: - recordType: A records: - - name: web-01.instances + - name: web-01 ttl: 30 a: content: 10.20.0.10 - recordType: AAAA records: - - name: web-01.instances + - name: web-01 ttl: 30 aaaa: content: fd20::10 From 854647cb266b27fa938468ade068d5adbc96877b Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 17:31:04 -0500 Subject: [PATCH 09/25] Use name-only references for customer DNS configuration --- docs/architecture/internal-dns/README.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 009ae0db..d86b5db4 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -247,10 +247,11 @@ are not fields consumers submit when creating resources. #### Ownership and validation -Admission pins reference UIDs and publication policy generations, authenticates -project and source-cluster identity, and enforces field ownership, including -shared status fields. Publishers cannot grant themselves rights or authorize -resolver access. Consumer references stay within a project. +Customers select zones and contexts by name within their project namespace. DNS +resolves these references and retains their UIDs in controller-owned status. +Internal publication and access contracts pin UIDs and policy generations. +Admission authenticates writers and enforces field ownership. Publishers cannot +grant themselves rights or authorize resolver access. #### Context and regional access @@ -343,14 +344,12 @@ kind: DNSZoneAssociation metadata: name: production-application spec: - # Both objects belong to this project. Pin each object's lifetime. + # Select the zone by name within this project namespace. dnsZoneRef: name: production - uid: 88888888-8888-4888-8888-888888888888 # DNS consumes a context reference, not a VPC reference. resolverContextRef: name: application - uid: 33333333-3333-4333-8333-333333333333 --- apiVersion: dns.networking.miloapis.com/v1alpha1 kind: DNSRecordSet @@ -369,6 +368,10 @@ spec: content: 10.20.0.50 ``` +DNS pins each association's resolved zone and context lifetimes. A replacement +with the same name does not inherit the association; recreate the association +to attach it. Customers cannot change the resolved identities in status. + A context can use several zones. Reject duplicate apexes within a context and conflicts between static records and reserved names. Select parent and child zones by longest suffix. Separate contexts can explicitly share a zone or use @@ -389,23 +392,22 @@ spec: # Apply naming rules to this DNS scope. resolverContextRef: name: application - uid: 33333333-3333-4333-8333-333333333333 additionalNames: # Add instance identity names, such as web-01.instances.prod.internal. - registrationClass: InstanceIdentity dnsZoneRef: name: production - uid: 88888888-8888-4888-8888-888888888888 namePrefix: instances # Add discovery names, such as api.services.prod.internal. - registrationClass: ServiceDiscovery dnsZoneRef: name: production - uid: 88888888-8888-4888-8888-888888888888 namePrefix: services ``` -Every target zone must be associated with the context. The proposed classes are +Every target zone must be associated with the context. DNS retains the policy's +resolved lifetimes; recreate the policy to use a replacement zone or context +with the same name. The proposed classes are `InstanceIdentity`, `ServiceDiscovery`, `ServiceVIP`, and `ServiceExport`. Product integrations apply the appropriate rule when reserving names. DNS does not inspect the underlying product resource to determine its class. From 950c8506d5442d8e34df8dfb1e8eb3cd62198ef8 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sun, 11 Oct 2026 19:06:46 -0500 Subject: [PATCH 10/25] Describe product-neutral DNS naming selectors --- docs/architecture/internal-dns/README.md | 26 +++++++++++++++--------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index d86b5db4..121fc771 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -393,24 +393,27 @@ spec: resolverContextRef: name: application additionalNames: - # Add instance identity names, such as web-01.instances.prod.internal. - - registrationClass: InstanceIdentity + # Match DNS registration labels from any authorized publisher. + - selector: + matchLabels: + app: storefront dnsZoneRef: name: production - namePrefix: instances - # Add discovery names, such as api.services.prod.internal. - - registrationClass: ServiceDiscovery - dnsZoneRef: + namePrefix: storefront + # An omitted selector adds this name for every registration in the context. + - dnsZoneRef: name: production namePrefix: services ``` Every target zone must be associated with the context. DNS retains the policy's resolved lifetimes; recreate the policy to use a replacement zone or context -with the same name. The proposed classes are -`InstanceIdentity`, `ServiceDiscovery`, `ServiceVIP`, and `ServiceExport`. -Product integrations apply the appropriate rule when reserving names. DNS does -not inspect the underlying product resource to determine its class. +with the same name. Selectors match `DNSRegistration.metadata.labels` using +`matchLabels` or `matchExpressions`. An omitted or empty selector matches every +registration in the context. Matching rules add +`..`; overlapping rules produce one +reservation per zone and name. Product integrations apply the accepted rules +when reserving additional names. Scoped grants authorize publication. #### Product publication @@ -423,6 +426,9 @@ apiVersion: dns.networking.miloapis.com/v1alpha1 kind: DNSRegistration metadata: name: instance-web-01 + # Naming policies match these generic registration labels. + labels: + app: storefront spec: # Select the managed zone allocated for the resource's DNS context. dnsZoneRef: From daf9ae6d223f594af84eaee3afbee1106e0f9b6b Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 16:40:43 -0500 Subject: [PATCH 11/25] Document RBAC boundaries for private DNS resource access --- docs/architecture/internal-dns/README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 121fc771..78a1b7d7 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -250,8 +250,11 @@ are not fields consumers submit when creating resources. Customers select zones and contexts by name within their project namespace. DNS resolves these references and retains their UIDs in controller-owned status. Internal publication and access contracts pin UIDs and policy generations. -Admission authenticates writers and enforces field ownership. Publishers cannot -grant themselves rights or authorize resolver access. +The project API server authenticates callers and uses RBAC to authorize resource +and status operations. Separate roles govern product publication, grant issuance, +network integration, and DNS controllers. Publishers cannot issue grants or +modify resolver authority. Admission validates live DNS references, lease limits, +and the relationship between an authenticated publisher and its current grant. #### Context and regional access From 6d3517c03b6d038bda8810e8dfbb426997060508 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 18:00:50 -0500 Subject: [PATCH 12/25] Document persistent internal DNS serving during control plane outages --- docs/architecture/internal-dns/README.md | 94 ++++++++++++------------ 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 78a1b7d7..81da55a9 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -50,7 +50,8 @@ withdraw endpoints that are no longer eligible. - Allocate default names and support additional custom zones. - Isolate overlapping names, addresses, and resolver caches across DNS contexts. - Accept publications from distributed product control planes and withdraw - expired or ineligible endpoints. + endpoints after an explicit health update or deletion. +- Keep serving applied DNS configuration during control plane outages. - Scale shared serving capacity across regions without deployments per VPC. ### Non-Goals @@ -72,8 +73,10 @@ stable resolver address in their VPC while the platform selects shared capacity. - Tenant leakage: authorize network identity before cache lookup, isolate resolver views, and test overlapping zones and forged identities. -- Stale endpoints or access: preserve original deadlines, fence writers and - replay, and enforce expiry locally at serving nodes. +- Delayed health updates: retain the last reported endpoint state during outages + and reconcile changes when connectivity returns. +- Tenant identity reuse: retain destination allocations until withdrawal is + confirmed across the serving fleet. - Shared fleet failure: qualify configuration activation, readiness, capacity, and regional failure handling before release. @@ -189,7 +192,7 @@ unmatched destinations before lookup. ### Publication and service discovery A registration reserves a name and record types. A grant authorizes a product -publisher. A contribution supplies records and a time-limited eligibility +publisher. A contribution supplies records and an explicit eligibility observation. DNS compiles accepted contributions into a complete zone snapshot. ```mermaid @@ -200,13 +203,13 @@ sequenceDiagram participant Store as DNS service project API participant Transport as Regional NATS participant Agent as Serving agent - Product->>Project: Records and fresh eligibility observation - DNS->>Project: Validate identities, grants, and deadlines + Product->>Project: Records and eligibility observation + DNS->>Project: Validate identities, grants, and versions DNS->>Store: Stage immutable chunks and manifest DNS->>Store: Commit active snapshot under writer lease Store->>Transport: Export committed update through durable outbox - Transport->>Agent: Deliver update, including original deadlines - Agent->>Agent: Verify, apply, and enforce local expiry + Transport->>Agent: Deliver committed update + Agent->>Agent: Verify and apply local configuration Agent->>Store: Acknowledge applied snapshot DNS->>Project: Report serving status Product->>Project: Higher-sequence ineligible observation @@ -228,13 +231,18 @@ acknowledges applied state separately from broker receipt. Queries use local state without contacting project APIs or the broker. Products define eligibility for instance identity, application discovery, or -stable service addresses. Serving agents enforce contribution freshness and -access deadlines independently. Replay and restart cannot extend them; expired -endpoints are withdrawn locally. Retained tombstones and writer fences prevent -restoration, and watchdogs remove members that cannot enforce expiry. +stable service addresses. Records and resolver bindings persist until an explicit +update or withdrawal. Control plane and messaging outages delay changes but do +not expire applied DNS configuration. Agents can restore verified configuration +from local checkpoints without contacting the control plane. + +Local DNS probes and backend health checks detect serving failures independently +of update delivery. Retained tombstones and writer fences prevent replay from +restoring withdrawn records or access. A new serving node requires a complete +configuration before accepting queries. A reserved name without eligible addresses returns NODATA. Missing required -private state returns SERVFAIL; unknown or expired access returns REFUSED. +private state returns SERVFAIL; unknown or withdrawn access returns REFUSED. Private names never fall back to public resolution. Disable stale private answers and bound dynamic TTLs; withdrawal budgets include client caching. @@ -253,7 +261,7 @@ Internal publication and access contracts pin UIDs and policy generations. The project API server authenticates callers and uses RBAC to authorize resource and status operations. Separate roles govern product publication, grant issuance, network integration, and DNS controllers. Publishers cannot issue grants or -modify resolver authority. Admission validates live DNS references, lease limits, +modify resolver authority. Admission validates live DNS references and the relationship between an authenticated publisher and its current grant. #### Context and regional access @@ -311,18 +319,15 @@ spec: authorization: # Match the DNS-issued context epoch; stale issuers cannot restore access. writerEpoch: 3 - # Renewals advance monotonically within this access-binding lifetime. + # Order updates monotonically within this access-binding lifetime. sequence: 27 - # Preserve this deadline through transport, replay, and local installation. - validUntil: "2026-10-08T20:05:00Z" ``` Access reports `Accepted` after validation and `Ready` after required members apply current authorization and configuration. Context, region, query identity, port, transports, and authorization epoch are immutable per access lifetime. -Renewals advance the sequence with a bounded deadline; destinations are unique -within their routing scope. Galactic owns the consumer frontend and network -lease separately. +Updates advance the sequence; destinations are unique within their routing +scope. Galactic owns the consumer frontend and network authorization. #### Private zones and associations @@ -436,14 +441,13 @@ spec: # Select the managed zone allocated for the resource's DNS context. dnsZoneRef: name: managed-application - uid: 44444444-4444-4444-8444-444444444444 # Reserve this relative owner name, rather than one particular IP address. name: web-01 # Constrain which record types contributions can populate. recordTypes: [A, AAAA] # Publish only contributions with a current eligible observation. publicationPolicy: EligibleContributions - # Bound client caching; this does not replace the observation deadline. + # Bound client caching after a record update or withdrawal. ttlSeconds: 30 # Optional descendant reservations must not overlap other owners. reservedDescendants: [] @@ -464,8 +468,7 @@ spec: # Audit label; the authenticated principal below authorizes the writer. producerID: compute principal: - # Trusted source-cluster identity distinguishes distributed publishers. - clusterUID: 99999999-9999-4999-8999-999999999999 + # Match the authenticated publisher within this project's API context. subject: system:serviceaccount:compute-system:dns-publisher # Scope the writer to these types and owner names. recordTypes: [A, AAAA] @@ -516,17 +519,15 @@ status: # Producer-owned: product policy determines readiness for this name's purpose. eligible: true reason: NetworkInterfaceProgrammed - # Producer-owned: DNS must preserve this original freshness deadline. - validUntil: "2026-10-08T20:01:00Z" # DNS-owned: the publication revision that included this contribution. publishedRevision: 12 ``` Publishers preserve DNS-owned status fields and observe the current contribution generation. An ineligible endpoint gets a higher-sequence observation with -`eligible: false`; recovery requires a fresh eligible observation. Publisher -loss does not renew the original deadline. Admission checks the principal, -grant epoch, sequence, record scope, and freshness bound. +`eligible: false`; recovery requires a newer eligible observation. Publisher +unavailability leaves the last reported state in place. Admission checks the +principal, grant epoch, sequence, and record scope. #### DNS service project contracts @@ -584,10 +585,9 @@ spec: - name: production uid: 88888888-8888-4888-8888-888888888888 authorization: - # Copy the accepted access epoch, sequence, and deadline without renewal. + # Copy the accepted access epoch and sequence. writerEpoch: 3 sequence: 27 - validUntil: "2026-10-08T20:05:00Z" ``` Source and zone references resolve in `source.projectUID`, not the DNS service @@ -595,12 +595,12 @@ namespace. Source identity is immutable. The region and node listener match the access binding; DNS allocates the regional listener. Configuration generation fences plan replacement; revision orders its updates. -Authorization copies the accepted access epoch, sequence, and deadline. Serving +Authorization copies the accepted access epoch and sequence. Serving acknowledgments identify the binding UID and both version sets; readiness requires current acknowledgments from all required members. Resolver engine, isolation, deployment model, and dnsdist cache policy belong to -shared fleet configuration. The nested binding is proposed; the prototype is flat. +shared fleet configuration. DNS-owned publication resources retain the compiler lease and active pointer (`DNSPublicationOwnership`), immutable snapshot data (`DNSPublicationManifest` @@ -609,12 +609,10 @@ Their storage schemas can be reviewed with the implementation. #### Compatibility and API review -Prototype schema, controller, transport, and agent migration must preserve UID, -configuration, and authorization fences and original deadlines. The prototype's -authoritative backend must be replaced with BIND zone materialization and -revalidated for tenant isolation and expiry. Review naming conflicts, grant -revocation, field ownership, freshness bounds, regional readiness, and snapshot -limits before stabilizing the APIs. +Schemas, transport, and checkpoints must preserve object lifetimes and ordered +configuration versions. Review naming conflicts, grant revocation, field +ownership, outage recovery, regional readiness, and snapshot limits before +stabilizing the APIs. ### Integration boundaries @@ -637,32 +635,33 @@ workloads lose resolution. Reenabling requires fresh authorization and observati ### Rollout, Upgrade and Rollback Planning -Qualify tenant isolation, expiry, failover, replay, and upgrade/rollback over +Qualify tenant isolation, outage continuity, failover, replay, and upgrade/rollback over Galactic UDP and TCP paths, including guest resolver configuration. Expand by context and region after setting withdrawal budgets and tombstone retention. ### Monitoring Requirements -Measure query latency/errors, publication lag, outbox backlog, expiry delay, and +Measure query latency/errors, publication lag, outbox backlog, withdrawal delay, and activation failures. Use bounded metric labels and per-context API status. Availability, latency, and withdrawal SLOs remain to be defined. ### Dependencies Project APIs, Galactic/Karmada, NATS, and the DNS fleet support updates and -connectivity. API or broker outages delay updates while installed state remains -subject to its original deadlines; network authorization expires independently. +connectivity. API or broker outages delay updates while the fleet continues +serving its applied configuration. Health changes and deletions take effect +after delivery resumes. ### Scalability -Measure context capacity, renewal/write rates, snapshot size, transport fanout, +Measure context capacity, update rates, snapshot size, transport fanout, cache memory, reload time, and query throughput. Bound chunks and retained history; establish supported limits and regional failover capacity before release. ### Troubleshooting Trace project/context/access UIDs through authorization, committed snapshots, -and serving acknowledgments. Inspect expiry for REFUSED, private state for +and serving acknowledgments. Inspect network identity for REFUSED, private state for SERVFAIL, and product eligibility for NODATA. ## Implementation History @@ -673,8 +672,9 @@ boundary [#230](https://github.com/datum-cloud/dns-operator/pull/230). ## Drawbacks -Shared BIND views increase fleet complexity and failure scope. Leases, snapshots, -and freshness renewals add coordination state and API traffic. +Shared BIND views increase fleet complexity and failure scope. Snapshots and +writer coordination add state and API traffic. During control plane outages, +DNS retains endpoints whose health may have changed. ## Alternatives From 66edd3cdd09c8667ebf711dec1461d461e7888c4 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 18:03:24 -0500 Subject: [PATCH 13/25] Define outage continuity for private network connectivity --- docs/architecture/internal-dns/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 81da55a9..31c8ca5c 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -648,7 +648,8 @@ Availability, latency, and withdrawal SLOs remain to be defined. ### Dependencies Project APIs, Galactic/Karmada, NATS, and the DNS fleet support updates and -connectivity. API or broker outages delay updates while the fleet continues +connectivity. Galactic must retain applied private connectivity during control +plane outages. API or broker outages delay updates while the fleet continues serving its applied configuration. Health changes and deletions take effect after delivery resumes. From 34e60c914bd20753d9953fde72404a9fd76c4037 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 18:52:46 -0500 Subject: [PATCH 14/25] docs: Define independent DNS availability and update delivery --- docs/architecture/internal-dns/README.md | 29 ++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 31c8ca5c..f830524f 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -222,6 +222,12 @@ versions. The compiler stages immutable chunks and a manifest, then commits its active pointer. Kubernetes API storage retains this state and a durable outbox; exporters repair missing outboxes after crashes. +Contexts, access bindings, and publications share one stable shard assignment +per region. Capacity limits admit established contexts before new allocations. +An allocation that exceeds capacity reports a condition without blocking other +contexts or explicit withdrawals. Destination ownership follows the context +lifetime, so recreating its access binding can retain the same destination. + Regional agents render committed snapshots as zone files, validate and activate them in the associated BIND views, then acknowledge serving. @@ -236,12 +242,16 @@ update or withdrawal. Control plane and messaging outages delay changes but do not expire applied DNS configuration. Agents can restore verified configuration from local checkpoints without contacting the control plane. -Local DNS probes and backend health checks detect serving failures independently -of update delivery. Retained tombstones and writer fences prevent replay from +Agents retain desired version fences and an activated configuration separately. +A rejected expansion retains that configuration; a rejected withdrawal gates +serving. An independent watchdog probes local daemons while updates are staged +and validated. DNS query probes and backend health checks isolate serving +failures from update delivery. Retained tombstones and writer fences prevent replay from restoring withdrawn records or access. A new serving node requires a complete configuration before accepting queries. -A reserved name without eligible addresses returns NODATA. Missing required +A reserved name without eligible addresses returns NODATA. A pending private zone returns SERVFAIL only beneath its own apex; other zones +and public recursion continue. Missing required private state returns SERVFAIL; unknown or withdrawn access returns REFUSED. Private names never fall back to public resolution. Disable stale private answers and bound dynamic TTLs; withdrawal budgets include client caching. @@ -578,6 +588,10 @@ spec: address: fd70:200::10 port: 53 transports: [UDP, TCP] + # Reserve private apices even while their publication is pending. + zoneApices: + 44444444-4444-4444-8444-444444444444: datum.internal + 88888888-8888-4888-8888-888888888888: production.internal zoneRefs: # Include only private zones explicitly associated with the source context. - name: managed-application @@ -597,7 +611,10 @@ access binding; DNS allocates the regional listener. Configuration generation fences plan replacement; revision orders its updates. Authorization copies the accepted access epoch and sequence. Serving acknowledgments identify the binding UID and both version sets; readiness -requires current acknowledgments from all required members. +requires current proof from available capacity in each serving tier. +`FullyReplicated` reports whether every assigned member has applied the plan; +`ObservationHealthy` reports current serving observations. Stale observations +make serving status unknown without withdrawing applied access. Resolver engine, isolation, deployment model, and dnsdist cache policy belong to shared fleet configuration. @@ -638,6 +655,10 @@ workloads lose resolution. Reenabling requires fresh authorization and observati Qualify tenant isolation, outage continuity, failover, replay, and upgrade/rollback over Galactic UDP and TCP paths, including guest resolver configuration. Expand by context and region after setting withdrawal budgets and tombstone retention. +Compatible upgrades retain transport state and checkpoints. Roll out serving +capabilities before enabling new controller output. Unsupported formats preserve +state and reject activation; conversion must retain authority and withdrawal +fences. ### Monitoring Requirements From f99a38a542500c98dd9ff3fc1be29e38523141bb Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Mon, 12 Oct 2026 19:11:14 -0500 Subject: [PATCH 15/25] docs: Separate resolver access and zone publication readiness --- docs/architecture/internal-dns/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index f830524f..46ebb843 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -334,7 +334,8 @@ spec: ``` Access reports `Accepted` after validation and `Ready` after required members -apply current authorization and configuration. Context, region, query identity, +apply current authorization and configuration. Zone publication readiness is +reported separately; a pending zone does not block resolver access. Context, region, query identity, port, transports, and authorization epoch are immutable per access lifetime. Updates advance the sequence; destinations are unique within their routing scope. Galactic owns the consumer frontend and network authorization. From 4234b81dff72bcf5546eb413b182a74b20d2e31d Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 08:30:08 -0500 Subject: [PATCH 16/25] docs: Keep reference identities in DNS state --- docs/architecture/internal-dns/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 46ebb843..88f46ffc 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -108,7 +108,7 @@ flowchart TB - **Consumer project:** Owns contexts, access bindings, zones, naming policy, and publications. VPC integration writes access intent; product services publish records and eligibility; DNS reports serving status. -- **DNS service project:** Owns writer leases, snapshots, outboxes, fleet plans, +- **DNS service project:** Owns internal reference records, writer leases, snapshots, outboxes, fleet plans, and the service's network intent. - **Karmada:** Places network and workload intent. Guest resolver settings travel as desired state; DNS publications use their own delivery path. @@ -266,7 +266,8 @@ are not fields consumers submit when creating resources. #### Ownership and validation Customers select zones and contexts by name within their project namespace. DNS -resolves these references and retains their UIDs in controller-owned status. +resolves these references and stores ownership identities in internal reference +records in the DNS service project. Project status exposes names and conditions. Internal publication and access contracts pin UIDs and policy generations. The project API server authenticates callers and uses RBAC to authorize resource and status operations. Separate roles govern product publication, grant issuance, @@ -298,7 +299,6 @@ status: managedNamespace: dnsZoneRef: name: managed-application - uid: 44444444-4444-4444-8444-444444444444 suffix: datum.internal # DNS-owned fencing epoch for trusted access authorization. accessWriterEpoch: 3 From 9728572b37d37450e90f759416b6cf866d18c418 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 09:51:03 -0500 Subject: [PATCH 17/25] docs: Clarify DNS integration boundaries --- docs/architecture/internal-dns/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 88f46ffc..35a7fd01 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -69,6 +69,11 @@ namespace, accepts explicit custom zone associations, and compiles product publications into private serving state. Consumers use a stable resolver address in their VPC while the platform selects shared capacity. +DNS APIs reference resolver contexts and zones. Network services owns the mapping +from a network to its DNS context and the consumer-facing resolver address. DNS +reserves service-side listener addresses and accepts registrations from any +authorized publisher; naming rules match registration labels. + ### Risks and mitigations - Tenant leakage: authorize network identity before cache lookup, isolate From 6e91a878834d47a64969ffa3d6fafc5786b48a96 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 12:31:14 -0500 Subject: [PATCH 18/25] docs: Simplify internal DNS integration --- docs/architecture/internal-dns/README.md | 115 ++++++++--------------- 1 file changed, 39 insertions(+), 76 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 35a7fd01..18e767fe 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -138,7 +138,7 @@ flowchart LR ``` Bindings live in the consumer project API; `region` selects serving capacity. -Each has independent identity, authorization, and readiness. Publication is +Each has independent identity, access mapping, and readiness. Publication is asynchronous, and a failed region must not block healthy regions. Regional access does not imply that discovery returns only region-local endpoints. @@ -187,8 +187,8 @@ Shared node and regional tiers use: [dnsdist PROXYv2](https://www.dnsdist.org/advanced/passing-source-address.html) preserves the destination. [BIND PROXY access controls](https://bind9.readthedocs.io/en/v9.20.2/reference.html#namedconf-statement-allow-proxy) -restrict trusted peers and listeners. A shared pool member must have current -configuration and authorization for every assigned context. +restrict trusted peers and listeners. A shared pool member must have the current +configuration and access version for every assigned context. [BIND views](https://bind9.readthedocs.io/en/v9.20.2/reference.html#view-block-grammar) select the private zones visible to each context. Keep caches separate and reject @@ -196,9 +196,9 @@ unmatched destinations before lookup. ### Publication and service discovery -A registration reserves a name and record types. A grant authorizes a product -publisher. A contribution supplies records and an explicit eligibility -observation. DNS compiles accepted contributions into a complete zone snapshot. +A registration reserves a name and record types. An RBAC-authorized product +service publishes records and explicit endpoint eligibility through a contribution. +DNS validates ownership and observation versions, then compiles a zone snapshot. ```mermaid sequenceDiagram @@ -209,7 +209,7 @@ sequenceDiagram participant Transport as Regional NATS participant Agent as Serving agent Product->>Project: Records and eligibility observation - DNS->>Project: Validate identities, grants, and versions + DNS->>Project: Validate references and observation versions DNS->>Store: Stage immutable chunks and manifest DNS->>Store: Commit active snapshot under writer lease Store->>Transport: Export committed update through durable outbox @@ -275,10 +275,9 @@ resolves these references and stores ownership identities in internal reference records in the DNS service project. Project status exposes names and conditions. Internal publication and access contracts pin UIDs and policy generations. The project API server authenticates callers and uses RBAC to authorize resource -and status operations. Separate roles govern product publication, grant issuance, -network integration, and DNS controllers. Publishers cannot issue grants or -modify resolver authority. Admission validates live DNS references -and the relationship between an authenticated publisher and its current grant. +and status operations. Separate roles govern product publication, network integration, and DNS +controllers. Product services are trusted within their project permissions. +Admission validates live references; DNS checks name conflicts and record scope. #### Context and regional access @@ -305,8 +304,8 @@ status: dnsZoneRef: name: managed-application suffix: datum.internal - # DNS-owned fencing epoch for trusted access authorization. - accessWriterEpoch: 3 + # DNS-owned epoch for ordering access mapping updates. + accessEpoch: 3 # Regional assignments do not move this object out of the project API. servingTargets: - region: central @@ -331,17 +330,17 @@ spec: # Permit DNS over both transports at the authorized destination. port: 53 transports: [UDP, TCP] - authorization: - # Match the DNS-issued context epoch; stale issuers cannot restore access. - writerEpoch: 3 + version: + # Match the context epoch to reject stale mapping updates. + epoch: 3 # Order updates monotonically within this access-binding lifetime. sequence: 27 ``` Access reports `Accepted` after validation and `Ready` after required members -apply current authorization and configuration. Zone publication readiness is +apply the current access mapping and configuration. Zone publication readiness is reported separately; a pending zone does not block resolver access. Context, region, query identity, -port, transports, and authorization epoch are immutable per access lifetime. +port, transports, and access epoch are immutable per access lifetime. Updates advance the sequence; destinations are unique within their routing scope. Galactic owns the consumer frontend and network authorization. @@ -437,13 +436,13 @@ with the same name. Selectors match `DNSRegistration.metadata.labels` using registration in the context. Matching rules add `..`; overlapping rules produce one reservation per zone and name. Product integrations apply the accepted rules -when reserving additional names. Scoped grants authorize publication. +when reserving additional names. Project RBAC authorizes publication. #### Product publication -Compute can reserve a name in the managed namespace, obtain a scoped publisher -grant, and publish eligible interface addresses. Connect uses the same contract -with its own publisher principal and export eligibility policy. +Compute reserves names in the managed namespace and publishes eligible interface +addresses. Connect uses the same contribution API with its export eligibility +policy. Each service authenticates to the project API with scoped RBAC access. ```yaml apiVersion: dns.networking.miloapis.com/v1alpha1 @@ -472,31 +471,6 @@ status: canonicalFQDN: web-01.datum.internal --- apiVersion: dns.networking.miloapis.com/v1alpha1 -kind: DNSContributionGrant -metadata: - name: web-01-compute -spec: - # Pin both the registration lifetime and the policy being authorized. - registrationRef: - name: instance-web-01 - uid: 55555555-5555-4555-8555-555555555555 - generation: 1 - # Audit label; the authenticated principal below authorizes the writer. - producerID: compute - principal: - # Match the authenticated publisher within this project's API context. - subject: system:serviceaccount:compute-system:dns-publisher - # Scope the writer to these types and owner names. - recordTypes: [A, AAAA] - nameScopes: [web-01] -status: - # DNS allocates the epoch; the producer cannot choose its own fencing token. - activeWriterEpoch: 3 - # Identify the grant and registration policy versions DNS accepted. - observedGrantGeneration: 1 - observedRegistrationGeneration: 1 ---- -apiVersion: dns.networking.miloapis.com/v1alpha1 kind: DNSRecordContribution metadata: name: web-01-endpoint @@ -506,11 +480,6 @@ spec: name: instance-web-01 uid: 55555555-5555-4555-8555-555555555555 generation: 1 - # Bind the producer to its authorized grant lifetime and policy. - grantRef: - name: web-01-compute - uid: 66666666-6666-4666-8666-666666666666 - generation: 1 # Reuse the existing typed record representation. recordSets: - recordType: A @@ -528,9 +497,7 @@ spec: status: # Producer-owned: this observation covers the current contribution spec. observedGeneration: 1 - # Producer-owned: echo the DNS-issued grant epoch. - writerEpoch: 3 - # Producer-owned: order observations within that epoch. + # Producer-owned: increase for each eligibility observation. sequence: 18 # Producer-owned: product policy determines readiness for this name's purpose. eligible: true @@ -542,8 +509,8 @@ status: Publishers preserve DNS-owned status fields and observe the current contribution generation. An ineligible endpoint gets a higher-sequence observation with `eligible: false`; recovery requires a newer eligible observation. Publisher -unavailability leaves the last reported state in place. Admission checks the -principal, grant epoch, sequence, and record scope. +unavailability leaves the last reported state in place. The compiler rejects stale +generations, sequence rollback, and records outside the registration. #### DNS service project contracts @@ -552,7 +519,7 @@ publishers cannot write them. Examples show selected fields that explain coordination; they are not complete generated serving plans or transport payloads. The proposed `DNSResolverBinding` groups four concerns: source identity, -regional placement, generated serving configuration, and access authorization. +regional placement, generated serving configuration, and access versions. It binds a consumer DNS context to shared capacity; it does not describe a VPC or create a resolver deployment. @@ -570,7 +537,7 @@ spec: resolverContextRef: name: application uid: 33333333-3333-4333-8333-333333333333 - # Retain the exact regional access authorization that produced this plan. + # Identify the regional access mapping that produced this plan. accessBindingRef: name: application-central uid: aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa @@ -583,17 +550,13 @@ spec: generation: 1 # Order configuration changes within this plan generation. revision: 7 - listeners: - # Node-tier service destination, reached through the private network path. - node: - address: fd70:100::10 - port: 53 - transports: [UDP, TCP] - # Regional-tier destination used by the context's node resolver view. - regional: - address: fd70:200::10 - port: 53 - transports: [UDP, TCP] + query: + # Galactic maps the consumer resolver address to this context destination. + nodeAddress: fd70:100::10 + # Node resolvers forward to this regional context destination. + regionalAddress: fd70:200::10 + # Both destinations use DNS port 53 and the same transports. + transports: [UDP, TCP] # Reserve private apices even while their publication is pending. zoneApices: 44444444-4444-4444-8444-444444444444: datum.internal @@ -604,9 +567,9 @@ spec: uid: 44444444-4444-4444-8444-444444444444 - name: production uid: 88888888-8888-4888-8888-888888888888 - authorization: - # Copy the accepted access epoch and sequence. - writerEpoch: 3 + version: + # Copy the current access mapping version. + epoch: 3 sequence: 27 ``` @@ -633,7 +596,7 @@ Their storage schemas can be reviewed with the implementation. #### Compatibility and API review Schemas, transport, and checkpoints must preserve object lifetimes and ordered -configuration versions. Review naming conflicts, grant revocation, field +configuration versions. Review naming conflicts, record withdrawal, field ownership, outage recovery, regional readiness, and snapshot limits before stabilizing the APIs. @@ -654,7 +617,7 @@ before release. Define separate alpha controls for publication and serving. Public visibility remains the default. Revoke access before draining private capacity; dependent -workloads lose resolution. Reenabling requires fresh authorization and observations. +workloads lose resolution. Reenabling requires a newer access version and current observations. ### Rollout, Upgrade and Rollback Planning @@ -688,7 +651,7 @@ history; establish supported limits and regional failover capacity before releas ### Troubleshooting -Trace project/context/access UIDs through authorization, committed snapshots, +Trace project/context/access UIDs through access mappings, committed snapshots, and serving acknowledgments. Inspect network identity for REFUSED, private state for SERVFAIL, and product eligibility for NODATA. From c2748bede60983cf669b7d3815abaa7e4b450afd Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 12:56:26 -0500 Subject: [PATCH 19/25] docs: Clarify query destination ownership --- docs/architecture/internal-dns/README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 18e767fe..ec3b07ab 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -157,10 +157,10 @@ flowchart TB subgraph B[VPC B] ClientB[Client 10.0.0.5] --> FrontB[DNS endpoint fd53::53] end - FrontA --> GrantA[Galactic authorized path A] - FrontB --> GrantB[Galactic authorized path B] - GrantA --> DestA[Service destination fd70:100::10] - GrantB --> DestB[Service destination fd70:100::11] + FrontA --> PathA[Galactic private path A] + FrontB --> PathB[Galactic private path B] + PathA --> DestA[Service destination fd70:100::10] + PathB --> DestB[Service destination fd70:100::11] subgraph Shared[Shared DNS service fleet] DestA --> Dist[dnsdist: trusted destination to context] DestB --> Dist @@ -323,11 +323,11 @@ spec: # Select the serving region, not a separate project control plane. region: central queryIdentity: - # Galactic supplies an authorized service-side destination before DNS lookup. + # Galactic maps verified VPC traffic to this service-side destination. type: DestinationAddress # This is distinct from the well-known address exposed inside the VPC. value: fd70:100::10 - # Permit DNS over both transports at the authorized destination. + # Serve DNS at this destination over both transports. port: 53 transports: [UDP, TCP] version: @@ -342,7 +342,8 @@ apply the current access mapping and configuration. Zone publication readiness i reported separately; a pending zone does not block resolver access. Context, region, query identity, port, transports, and access epoch are immutable per access lifetime. Updates advance the sequence; destinations are unique within their routing -scope. Galactic owns the consumer frontend and network authorization. +scope. Galactic enforces network access to each destination. Project RBAC controls +who can manage the DNS mappings. #### Private zones and associations From ba4ff9fcbf9576a4df50ceb96f64bd74e060a0b1 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 13:31:10 -0500 Subject: [PATCH 20/25] docs: Simplify private DNS integration Describe atomic product publication, fixed query protocols, and DNS-owned access ordering. Resolver contexts directly own their managed zones and associations while preserving tenant isolation and outage continuity. --- docs/architecture/internal-dns/README.md | 66 ++++++++++-------------- 1 file changed, 26 insertions(+), 40 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index ec3b07ab..402c83f1 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -198,7 +198,7 @@ unmatched destinations before lookup. A registration reserves a name and record types. An RBAC-authorized product service publishes records and explicit endpoint eligibility through a contribution. -DNS validates ownership and observation versions, then compiles a zone snapshot. +DNS validates ownership and resource generations, then compiles a zone snapshot. ```mermaid sequenceDiagram @@ -208,8 +208,8 @@ sequenceDiagram participant Store as DNS service project API participant Transport as Regional NATS participant Agent as Serving agent - Product->>Project: Records and eligibility observation - DNS->>Project: Validate references and observation versions + Product->>Project: Atomically update records and eligibility + DNS->>Project: Validate references and resource generation DNS->>Store: Stage immutable chunks and manifest DNS->>Store: Commit active snapshot under writer lease Store->>Transport: Export committed update through durable outbox @@ -217,7 +217,7 @@ sequenceDiagram Agent->>Agent: Verify and apply local configuration Agent->>Store: Acknowledge applied snapshot DNS->>Project: Report serving status - Product->>Project: Higher-sequence ineligible observation + Product->>Project: Update contribution with eligible=false DNS->>Store: Commit snapshot that withdraws the endpoint ``` @@ -304,8 +304,6 @@ status: dnsZoneRef: name: managed-application suffix: datum.internal - # DNS-owned epoch for ordering access mapping updates. - accessEpoch: 3 # Regional assignments do not move this object out of the project API. servingTargets: - region: central @@ -322,26 +320,17 @@ spec: uid: 33333333-3333-4333-8333-333333333333 # Select the serving region, not a separate project control plane. region: central - queryIdentity: - # Galactic maps verified VPC traffic to this service-side destination. - type: DestinationAddress - # This is distinct from the well-known address exposed inside the VPC. - value: fd70:100::10 - # Serve DNS at this destination over both transports. - port: 53 - transports: [UDP, TCP] - version: - # Match the context epoch to reject stale mapping updates. - epoch: 3 - # Order updates monotonically within this access-binding lifetime. - sequence: 27 + # Galactic maps verified VPC traffic to this service-side destination. + # This differs from the well-known address exposed inside the VPC. + destinationAddress: fd70:100::10 + ``` Access reports `Accepted` after validation and `Ready` after required members apply the current access mapping and configuration. Zone publication readiness is -reported separately; a pending zone does not block resolver access. Context, region, query identity, -port, transports, and access epoch are immutable per access lifetime. -Updates advance the sequence; destinations are unique within their routing +reported separately; a pending zone does not block resolver access. Context, region, +and destination are immutable per access lifetime. DNS serves UDP and TCP on port 53 +and generates internal mapping versions. Destinations are unique within their routing scope. Galactic enforces network access to each destination. Project RBAC controls who can manage the DNS mappings. @@ -403,8 +392,8 @@ independent zones with the same apex. Static records have no health lease. #### Automatic naming and additional names -DNS allocates the requested managed zone and exposes its zone reference and -suffix in context status. Product publishers use these outputs. A naming policy +The context directly owns its managed zone and association. DNS allocates the +requested zone and exposes its reference and suffix in context status. Product publishers use these outputs. A naming policy adds custom names without a zone choice on each Compute or Connect resource. ```yaml @@ -461,8 +450,6 @@ spec: name: web-01 # Constrain which record types contributions can populate. recordTypes: [A, AAAA] - # Publish only contributions with a current eligible observation. - publicationPolicy: EligibleContributions # Bound client caching after a record update or withdrawal. ttlSeconds: 30 # Optional descendant reservations must not overlap other owners. @@ -481,6 +468,10 @@ spec: name: instance-web-01 uid: 55555555-5555-4555-8555-555555555555 generation: 1 + # Publish eligibility and addresses together in one desired-state update. + eligible: true + # Product policy determines eligibility for this name's purpose. + reason: NetworkInterfaceProgrammed # Reuse the existing typed record representation. recordSets: - recordType: A @@ -496,22 +487,18 @@ spec: aaaa: content: fd20::10 status: - # Producer-owned: this observation covers the current contribution spec. + # DNS-owned: the desired generation processed by DNS. observedGeneration: 1 - # Producer-owned: increase for each eligibility observation. - sequence: 18 - # Producer-owned: product policy determines readiness for this name's purpose. - eligible: true - reason: NetworkInterfaceProgrammed # DNS-owned: the publication revision that included this contribution. publishedRevision: 12 ``` -Publishers preserve DNS-owned status fields and observe the current contribution -generation. An ineligible endpoint gets a higher-sequence observation with -`eligible: false`; recovery requires a newer eligible observation. Publisher -unavailability leaves the last reported state in place. The compiler rejects stale -generations, sequence rollback, and records outside the registration. +Publishers update records and eligibility atomically with optimistic concurrency. +An explicit update with `eligible: false` withdraws the endpoint; recovery sets it +to true. Kubernetes assigns a new generation to each desired-state change. DNS +retains UID and generation fences across controller takeover and replay. Status +belongs to DNS. Publisher unavailability leaves the last desired state in place. +The compiler rejects stale generations and records outside the registration. #### DNS service project contracts @@ -557,7 +544,6 @@ spec: # Node resolvers forward to this regional context destination. regionalAddress: fd70:200::10 # Both destinations use DNS port 53 and the same transports. - transports: [UDP, TCP] # Reserve private apices even while their publication is pending. zoneApices: 44444444-4444-4444-8444-444444444444: datum.internal @@ -579,7 +565,7 @@ namespace. Source identity is immutable. The region and node listener match the access binding; DNS allocates the regional listener. Configuration generation fences plan replacement; revision orders its updates. -Authorization copies the accepted access epoch and sequence. Serving +DNS generates private access versions and advances them for withdrawal and recovery. Serving acknowledgments identify the binding UID and both version sets; readiness requires current proof from available capacity in each serving tier. `FullyReplicated` reports whether every assigned member has applied the plan; @@ -618,7 +604,7 @@ before release. Define separate alpha controls for publication and serving. Public visibility remains the default. Revoke access before draining private capacity; dependent -workloads lose resolution. Reenabling requires a newer access version and current observations. +workloads lose resolution. Reenabling requires current access intent and verified serving state. ### Rollout, Upgrade and Rollback Planning From 2fba135c31141b150f1817232913e18fa60db216 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 13:31:57 -0500 Subject: [PATCH 21/25] docs: Clarify DNS-owned mapping versions Keep consumer access examples focused on destination intent and describe mapping versions only in DNS service contracts. --- docs/architecture/internal-dns/README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 402c83f1..e5b763c3 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -323,7 +323,6 @@ spec: # Galactic maps verified VPC traffic to this service-side destination. # This differs from the well-known address exposed inside the VPC. destinationAddress: fd70:100::10 - ``` Access reports `Accepted` after validation and `Ready` after required members @@ -543,7 +542,6 @@ spec: nodeAddress: fd70:100::10 # Node resolvers forward to this regional context destination. regionalAddress: fd70:200::10 - # Both destinations use DNS port 53 and the same transports. # Reserve private apices even while their publication is pending. zoneApices: 44444444-4444-4444-8444-444444444444: datum.internal @@ -555,7 +553,7 @@ spec: - name: production uid: 88888888-8888-4888-8888-888888888888 version: - # Copy the current access mapping version. + # DNS generates ordering fences for this mapping's lifetime. epoch: 3 sequence: 27 ``` From 6e57cf5b60a19045d3eda1a31d15690611527711 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 15:58:50 -0500 Subject: [PATCH 22/25] Document automatic project discovery for internal DNS --- docs/architecture/internal-dns/README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index ec3b07ab..27417dbd 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -119,8 +119,11 @@ flowchart TB as desired state; DNS publications use their own delivery path. - **Edge:** Owns local networking, shared serving workloads, and checkpoints. -DNS uses authenticated project clients. VPC integration and product publishers -translate their resources into DNS intent; DNS does not watch their APIs. +DNS discovers ready Milo projects and watches every namespace through +authenticated project clients. Discovery outages retain existing sources and +applied serving state. Admission uses the authenticated project parent identity +for live reference validation. VPC integration and product publishers translate +their resources into DNS intent; DNS does not watch their APIs. ### Contexts and regional access From 56559edb1eb30bc82be761131fde52889592fd71 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 16:41:36 -0500 Subject: [PATCH 23/25] docs: Explain DNS activation continuity --- docs/architecture/internal-dns/README.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index bee750ca..52932181 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -252,11 +252,14 @@ from local checkpoints without contacting the control plane. Agents retain desired version fences and an activated configuration separately. A rejected expansion retains that configuration; a rejected withdrawal gates -serving. An independent watchdog probes local daemons while updates are staged -and validated. DNS query probes and backend health checks isolate serving -failures from update delivery. Retained tombstones and writer fences prevent replay from -restoring withdrawn records or access. A new serving node requires a complete -configuration before accepting queries. +serving. Healthy DNS continues through normal agent stops and restarts. +Readiness verifies durable activation state, the active checkpoint, and local +DNS daemons over UDP and TCP without contacting the broker or project APIs. +Incomplete or uncertain activation keeps serving gated until recovery verifies +the saved configuration. DNS query probes and backend health checks isolate +serving failures from update delivery. Retained tombstones and writer fences +prevent replay from restoring withdrawn records or access. A new serving node +requires a complete configuration before accepting queries. A reserved name without eligible addresses returns NODATA. A pending private zone returns SERVFAIL only beneath its own apex; other zones and public recursion continue. Missing required From 76e681d0ea23df54fdbc8dbc63f25d98628edc59 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Tue, 13 Oct 2026 17:00:35 -0500 Subject: [PATCH 24/25] docs: Explain serving activation ownership --- docs/architecture/internal-dns/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 52932181..1de4f079 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -255,8 +255,9 @@ A rejected expansion retains that configuration; a rejected withdrawal gates serving. Healthy DNS continues through normal agent stops and restarts. Readiness verifies durable activation state, the active checkpoint, and local DNS daemons over UDP and TCP without contacting the broker or project APIs. -Incomplete or uncertain activation keeps serving gated until recovery verifies -the saved configuration. DNS query probes and backend health checks isolate +The serving supervisor retains approved configuration independently of the +agent. A transaction lock owns candidate verification; incomplete or orphaned +activation stays gated until recovery verifies the saved configuration. DNS query probes and backend health checks isolate serving failures from update delivery. Retained tombstones and writer fences prevent replay from restoring withdrawn records or access. A new serving node requires a complete configuration before accepting queries. From 76bd5dc9c13afec397727832acafc2aadde9c9e5 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Wed, 14 Oct 2026 17:18:15 -0500 Subject: [PATCH 25/25] docs: describe the CoreDNS context architecture --- docs/architecture/internal-dns/README.md | 44 +++++++++++++----------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/architecture/internal-dns/README.md b/docs/architecture/internal-dns/README.md index 1de4f079..696878b0 100644 --- a/docs/architecture/internal-dns/README.md +++ b/docs/architecture/internal-dns/README.md @@ -77,7 +77,7 @@ authorized publisher; naming rules match registration labels. ### Risks and mitigations - Tenant leakage: authorize network identity before cache lookup, isolate - resolver views, and test overlapping zones and forged identities. + DNS contexts, and test overlapping zones and forged identities. - Delayed health updates: retain the last reported endpoint state during outages and reconcile changes when connectivity returns. - Tenant identity reuse: retain destination allocations until withdrawal is @@ -167,10 +167,10 @@ flowchart TB subgraph Shared[Shared DNS service fleet] DestA --> Dist[dnsdist: trusted destination to context] DestB --> Dist - Dist --> ViewA[Context A BIND view and cache] - Dist --> ViewB[Context B BIND view and cache] - ViewA --> ZoneA[Private zone in view A: api.prod.internal = 10.20.0.10] - ViewB --> ZoneB[Private zone in view B: api.prod.internal = 10.20.0.20] + Dist --> ViewA[CoreDNS context A] + Dist --> ViewB[CoreDNS context B] + ViewA --> ZoneA[Private zone in context A: api.prod.internal = 10.20.0.10] + ViewB --> ZoneB[Private zone in context B: api.prod.internal = 10.20.0.20] end ``` @@ -183,19 +183,23 @@ Shared node and regional tiers use: 1. **dnsdist** to select the context before lookup, with private packet caching disabled. -2. **BIND views** to isolate positive and negative caches. Node views forward to - context-specific regional destinations. Regional views host the context's - private authoritative zones and recurse for other names. Views share resolver - processes; each context has its own zone selection and cache. +2. **CoreDNS with a context plugin** to select private zones from the trusted + destination. Node members forward to the context's regional destination; + regional members answer private queries from accepted publication snapshots. +3. **PowerDNS Recursor** to resolve public names. CoreDNS keeps public caches + separate by context and bypasses caching for private names. [dnsdist PROXYv2](https://www.dnsdist.org/advanced/passing-source-address.html) -preserves the destination. [BIND PROXY access controls](https://bind9.readthedocs.io/en/v9.20.2/reference.html#namedconf-statement-allow-proxy) -restrict trusted peers and listeners. A shared pool member must have the current -configuration and access version for every assigned context. +preserves the destination. The CoreDNS transport adapter accepts PROXYv2 only +from configured service peers and preserves the destination over UDP and TCP. +Unknown contexts are rejected before record or cache lookup. -[BIND views](https://bind9.readthedocs.io/en/v9.20.2/reference.html#view-block-grammar) -select the private zones visible to each context. Keep caches separate and reject -unmatched destinations before lookup. +The publication agent validates and persists a complete snapshot. The context +plugin builds a replacement lookup table, then atomically activates it without +restarting DNS listeners. Invalid updates retain the accepted table. Persistent +snapshots allow serving to recover without an available control plane or broker. +A shared member must prove the current configuration and publication before it +receives traffic for that context. ### Publication and service discovery @@ -237,7 +241,7 @@ contexts or explicit withdrawals. Destination ownership follows the context lifetime, so recreating its access binding can retain the same destination. Regional agents render committed snapshots as zone files, validate and activate -them in the associated BIND views, then acknowledge serving. +them in the associated CoreDNS contexts, then acknowledge serving. Regional [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream) delivery is at least once. Every serving replica receives its complete assignment and @@ -638,7 +642,7 @@ after delivery resumes. ### Scalability Measure context capacity, update rates, snapshot size, transport fanout, -cache memory, reload time, and query throughput. Bound chunks and retained +cache memory, snapshot activation time, and query throughput. Bound chunks and retained history; establish supported limits and regional failover capacity before release. ### Troubleshooting @@ -655,14 +659,14 @@ boundary [#230](https://github.com/datum-cloud/dns-operator/pull/230). ## Drawbacks -Shared BIND views increase fleet complexity and failure scope. Snapshots and +The context plugin adds code that needs isolation, capacity, and protocol testing. Snapshots and writer coordination add state and API traffic. During control plane outages, DNS retains endpoints whose health may have changed. ## Alternatives -PowerDNS Recursor and CoreDNS remain options if they meet the same tenant -isolation and capacity requirements. +BIND views provide another shared serving option. The selected engine must meet +the same tenant isolation and capacity requirements. ## Infrastructure Needed