Skip to content

fix: Report DNS records that wait on their zone as pending - #657

Merged
vvoytovych merged 1 commit into
mainfrom
fix/537-dns-pending-aggregate
Oct 18, 2026
Merged

vvoytovych merged 1 commit into
mainfrom
fix/537-dns-pending-aggregate

Conversation

@vvoytovych

@vvoytovych vvoytovych commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

An HTTPProxy reported every hostname programmed while one of its DNS records was refused (#537). #650 fixed that, and #655 reverted it: a new hostname raised a false "Failed to program DNS" while its DNS zone settled. This change carries the condition again, and the Gateway now reports that wait as pending.

Before After
A hostname whose DNSZone or Domain is still being set up False, PartialFailure False, Pending
A hostname that waits on the customer: a registrar, a zone held elsewhere, a Domain to verify False, PartialFailure False, PartialFailure
A refused or conflicting DNS record False, PartialFailure False, PartialFailure
One hostname pending, another failed or waiting on the customer False, PartialFailure False, PartialFailure
A Gateway with no hostname left for Datum-managed DNS Keeps the condition it wrote before No condition
DNSRecordsProgrammed on the HTTPProxy Removed (#656) Carried from the Gateway, as in #650

Notes

  • Each hostname's own reason says who must act. Datum: DNSZoneNotReady while its zone settles, Pending while Datum checks its Domain or reads its nameservers. The customer: DomainNotVerified or DNSAuthorityMissing.
  • The status stays False. Gateway API lists Pending among the reasons for a False Programmed, beside AddressNotAssigned for an address not yet allocated.
  • A failed read of a Domain's DNS zones is returned and retried. It never reads as a wait on the customer.
  • The line fix: Drop a stale DNSRecordsProgrammed from the HTTPProxy #656 added to remove the condition goes.
  • fix: Report DNSRecordsProgrammed on the HTTPProxy as its Gateway does #650 also dropped a Pending once no DNS record was left, for andrewb3000's review. That drop goes too. Its cause was the Gateway keeping an old condition, which the Gateway now removes.
  • dnsRecordFailed is unchanged. Like its sibling certificateFailed, it warns when the stored condition is absent or holds another reason.

Blast radius

reads the condition where effect
the dnsRecordFailed activity event alb_activity_events.go Fires again for a failure. A pending hostname does not fire it
the reason catalog internal/agent/catalog.go Explains Pending, as in #650, so a pending HTTPProxy does not read as broken
cloud-portal waitForHttpProxyReady rejects False unless the reason is Pending; it has no callers It accepts a pending hostname's False/Pending
the Gateway's and the HTTPProxy's status Milo One status write for each object whose condition changes
the per-hostname DNSRecordProgrammed The Gateway computes it, with the reasons above, in memory and stores none. The HTTPProxy builds its own from DNS record sets No change. cloud-portal's getDnsRecordProgrammedIssue and infra's hostname-dns-reap read the HTTPProxy's

On 2026-10-15, 15 production Gateways kept a False from an earlier state with no hostname left for Datum-managed DNS; they lose it. 8 reported a DNS failure that this change keeps, so each HTTPProxy over one of them raises dnsRecordFailed once on upgrade.

Test plan

  • TestGatewayDNSWhileZoneAndDomainSettle replays the order of infra's hostname-dns-reap test. TestGatewayDNSRecordsProgrammedByClass covers each class of hostname, its own reason, and the mixes. TestGatewayWithoutDatumDNSHostnameCarriesNoDNSRecordsProgrammed covers the Gateway with no hostname left. TestGatewayDNSReturnsAFailedZoneRead covers the failed read. Each fails on main; the customer and failure rows pass there.
  • envtest TestHTTPProxyCarriesAPendingDNSRecordsConditionWithoutARecord: the HTTPProxy carries the Gateway's False/Pending while no hostname has a DNS record. TestDNSRecordFailedWarnsAsCertificateFailedDoes pins the warning rule: a pending aggregate does not warn. fix: Report DNSRecordsProgrammed on the HTTPProxy as its Gateway does #650's tests pass.
  • go test ./internal/controller/ ./internal/util/dns/ ./internal/agent/ ./api/..., on main at ff07521. Eighteen planted defects, each caught. Among them: a wait read as a failure, or the reverse. A refused or held zone read as settling. A hostname with the wrong reason. Pending hiding a failure. A failed read taken for a Domain without a zone. A pending aggregate that warns.
  • Staging, after the merge: the integration run's new HTTPProxy reads False/Pending while its zone settles. It raises no DNSRecordFailed.

Related to #537

@vvoytovych
vvoytovych requested a review from a team as a code owner October 15, 2026 20:41
@vvoytovych
vvoytovych marked this pull request as draft October 15, 2026 20:42
@vvoytovych
vvoytovych marked this pull request as ready for review October 16, 2026 08:40
@vvoytovych
vvoytovych marked this pull request as draft October 16, 2026 08:46
@vvoytovych
vvoytovych force-pushed the fix/537-dns-pending-aggregate branch from cd720df to e355c4f Compare October 16, 2026 11:44
@vvoytovych
vvoytovych marked this pull request as ready for review October 18, 2026 15:02
@vvoytovych
vvoytovych merged commit 957e22d into main Oct 18, 2026
17 checks passed
@vvoytovych
vvoytovych deleted the fix/537-dns-pending-aggregate branch October 18, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants