[scotwells]

Summary

Keep validated private DNS available during configuration agent restarts and control plane outages.

CoreDNS restores accepted snapshots from persistent storage and activates record changes without restarting DNS listeners. Failed restart installations retry activation. The fleet verifies installed publications before reporting readiness. Rejected additions preserve accepted records; if an update removes records or resolver access and cannot be applied, the member stops serving the affected configuration.

Test plan

• Focused restart tests, including a regression that fails without the retry fix
• CI: API and controller tests
• CI: Lint and generated manifests
• CI: Container packaging
• CI: Repository end-to-end tests

Stacked on #236. Architecture: #229.

Related: datum-cloud/enhancements#921

[ecv]

Approving. Every failure path here fails closed for isolation. Follow-ups are in #261, and the first two are worth doing before staging relies on restarts: restoreActive does not mark activation needed when ApplyForce fails, and probeLocalDaemons accepts REFUSED and SERVFAIL, so a gated node can report Ready. Every restart test sets UnsafeAllowNoDNSVerify, so none of them runs the probe. The body says invalid updates retain accepted state, but a rejected record removal or version bump drops all serving. Say which is intended.

The merge-base changed after approval.

[ecv]

Re-approving at 0f08c61. A failed restore now marks activation needed and retries, and fail-closed writes its marker, so a restart no longer reports Ready over a gated node. The probe still accepts REFUSED and SERVFAIL, and a failing restore retries at 1 Hz with no backoff. Both are in #261.

[ecv]

Re-approving. Only the parent merge changed this PR. Its own commits and diff are unchanged apart from line offsets.

[kevwilliams]

Hold: a withdrawal whose render fails keeps serving the withdrawn data.

• Agent.reconcile returns on a renderPublications error, and on the store.Save after Active = nil, before any gate. The old files and daemons keep answering for a revoked binding or deleted record, and every tick retries the same error. Still true at the top of the stack (#244).
• Fix: call failClosed whenever !canRetainActive() and reconcile fails for any reason, not only when Apply fails.
• Test: inject a render error on a withdrawal and assert serving stops.

[scotwells]

@kevwilliams Fixed in 3f8fa1a. Agent.reconcile now checks canRetainActive() before it starts and calls failClosed on any reconcile error when the approved configuration cannot be retained, including a renderPublications error and the checkpoint save after Active = nil. A shutdown is excluded because restoreActive gates again on restart. TestWithdrawalRenderFailureStopsServing injects a render error on a binding withdrawal and asserts that serving is gated, and that the same error without a withdrawal leaves serving alone. The fix is merged up through #243, and into #244 locally.

This push also carries 50d1585, which fixes the Chainsaw E2E hold on #239 and #243: the local readiness probe queried CoreDNS over UDP, but CoreDNS listens only on TCP.

[ecv]

Re-approving at 50d1585. Reconcile now fails closed on any error when the desired state withdraws something the approved configuration serves, so a revoked binding stops answering even if rendering fails, and a test covers both the withdrawal and the no-withdrawal case. The readiness probe checks CoreDNS over TCP only, matching #236's TCP-only listener. probeLocalDaemons still accepts any DNS response from dnsdist, which #261 tracks.

The merge-base changed after approval.

[ecv]

Re-approving. The head is unchanged; only the base moved after the parent merged.