Wait 120s after cluster is ACTIVE before connecting (awscli-cp-with-pcp) - #599
Open
ronaldngounou wants to merge 1 commit into
Open
Wait 120s after cluster is ACTIVE before connecting (awscli-cp-with-pcp)#599ronaldngounou wants to merge 1 commit into
ronaldngounou wants to merge 1 commit into
Conversation
…-pcp The loadtest can fail during connect with a DNS 'no such host' error for the cluster API server endpoint, because the endpoint instances take time to come up and the Route53 record has a ~60s TTL. Wait after the cluster is ACTIVE (create succeeded) before kubectl connects.
ronaldngounou
force-pushed
the
add-cp-settle-sleep-pcp
branch
from
August 11, 2026 20:00
9573beb to
8a265f1
Compare
ronaldngounou
changed the title
| # become resolvable/discoverable before connecting with kubectl: the endpoint | ||
| # instances take time to come up and the Route53 record has a ~60s TTL, so | ||
| # connecting immediately can fail with "no such host". | ||
| sleep 120 |
Contributor
There was a problem hiding this comment.
have you checked why wait won't internally wait until endpoint reachable ?
Contributor
Author
There was a problem hiding this comment.
aws eks wait cluster-active won't wait until the endpoint is reachable because the waiter model will only evaluate whether the cluster is active, failed, or deleting. It doesn't wait until the endpoint is reachable.
"ClusterActive": {
"delay": 30,
"operation": "DescribeCluster",
"maxAttempts": 40,
"acceptors": [
{ "state": "failure", "matcher": "path", "argument":
"cluster.status", "expected": "DELETING" },
{ "state": "failure", "matcher": "path", "argument":
"cluster.status", "expected": "FAILED" },
{ "state": "success", "matcher": "path", "argument":
"cluster.status", "expected": "ACTIVE" }
]
}
Contributor
There was a problem hiding this comment.
please ensure the check is - loop until it becomes active and break and/or 120sec if that's the SLO for r53 recs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add
sleep 120afteraws eks wait cluster-activein theawscli-eks-cluster-create-with-pcp-stacktask, so subsequent steps don't connect to the cluster API server endpoint immediately after creation.Why
Loadtests intermittently failed during connect with:
This is a transient issue right after creation: the API server endpoint instances take time to come up and be discoverable, and the Route53 record has a ~60s TTL. The fix is to wait until the cluster create is signaled successful (
wait cluster-active) and then give the endpoint time to become resolvable before kubectl connects.