Repository navigation
fix: Keep external-dns running through large DNS change batches - #10
Merged
Merged
Conversation
A POST /records that ran past the webhook server's 30s write timeout had its connection closed with no response while the handler kept applying changes. ExternalDNS saw a transport EOF, which it treats as fatal, and exited. Production crash-loops on any batch of roughly 40 or more changes, since each change costs 2-3s against the Datum DNS API and was applied one at a time. Key changes: - Bound each apply by --apply-timeout (default 5m) and derive the server write timeout from it with 30s of headroom, so the handler always answers before the server gives up on the connection - When the deadline is reached, stop starting new changes and return a 503 that says how many changes were attempted and which flag raises the limit; ExternalDNS treats a 5xx as a soft error and retries on its next sync instead of exiting - Apply changes within each phase in parallel, bounded by --apply-concurrency (default 4) - Run a delete before the creates when the same record set is recreated in the batch; other deletes still run last, so replacements are made before old records are removed - Recover a panic in a single change so it fails that change instead of the process Tests drive the real ExternalDNS v0.15 webhook client against the server, so the soft-error contract is checked against the code that decides whether ExternalDNS exits.
2 tasks
ecv
self-requested a review
September 26, 2026 16:33
ecv
approved these changes
Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
External-dns crash-loops in production whenever it sends a large batch of DNS changes, because the provider closes the connection without answering once a batch runs past 30 seconds.
External-dns reads a dropped connection as fatal and exits, even though the provider keeps applying the changes in the background.
Large batches now either finish, applied several changes at a time, or stop at a configurable limit and answer with an error that names the limit. External-dns treats that error as retryable and picks up the remaining changes on its next sync instead of restarting.
The limit defaults to five minutes and four parallel changes, and both are adjustable at startup.
Test plan
Fixes #4