docs: manual rolling upgrade runbook (clone workerpool approach) - #1392
Merged
Anna Pendleton (annapendleton) merged 2 commits intoSep 3, 2026
Merged
Conversation
Haven Xia (HavenXia)
force-pushed
the
runbook
branch
from
September 2, 2026 15:33
0c5471a to
0bca460
Compare
Haven Xia (HavenXia)
changed the title
Haven Xia (HavenXia)
force-pushed
the
runbook
branch
from
September 2, 2026 18:34
92e979b to
eae8e5a
Compare
Anna Pendleton (annapendleton)
previously approved these changes
Sep 2, 2026
Haven Xia (HavenXia)
force-pushed
the
runbook
branch
2 times, most recently
from
September 2, 2026 23:51
315db1a to
f398e50
Compare
| node at a time, so no actor loses state and at most one node's worth | ||
| of capacity is out of service while the rest of the fleet keeps | ||
| serving. It needs `kubectl`, `kubectl ate`, `go run ./cmd/ate-setup`, | ||
| `jq`, `grpcurl`, and on GKE `gcloud`. All of its state lives in |
Collaborator
There was a problem hiding this comment.
"GKE gcloud." - should this be more agnostic? does this only work on GKE or does it work for all substrate components?
Haven Xia (HavenXia)
force-pushed
the
runbook
branch
from
September 3, 2026 00:06
f398e50 to
60feb49
Compare
| | name | what it is | how to get it | | ||
| |---|---|---| | ||
| | `$CLUSTER`, `$ZONE` | the GKE cluster and its location | `gcloud container clusters list` | | ||
| | `$NODEPOOL` | the GKE node pool | `gcloud container node-pools list --cluster $CLUSTER --zone $ZONE` | |
Collaborator
There was a problem hiding this comment.
these gcloud commands - possibly should be kind commands (eg. assume kind is our base provider in substrate repo)
Anna Pendleton (annapendleton)
approved these changes
Sep 3, 2026
Anna Pendleton (annapendleton)
left a comment
Anna Pendleton (annapendleton)
left a comment
Collaborator
There was a problem hiding this comment.
not worth blocking review on - but we likely need to revisit some of the wording here so it's more cloud generic
Anna Pendleton (annapendleton)
merged commit Sep 3, 2026
2cd4943
into
agent-substrate:main
9 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runbook for moving a running substrate to a new build without losing actor state. An user runs the roll by hand with
kubectl,kubectl ate,ate-setup,jq, andgrpcurl, and every piece of upgrade state lives in cluster objects, so the roll can stop and resume at any point.The order follows the upgrade design: CRDs, then
ate-controller, then the dataplane node by node, thenate-api-serverandatenet.The dataplane moves by version label. The new atelet DaemonSet sits next to the old one, each serving WorkerPool is cloned with the new worker image and the new version pin, and each node is drained (workers marked
DRAININGthrough theDrainWorkerRPC), emptied by suspending its actors, relabeled, and cleared of old worker pods. The old objects stay untouched until a separate retire step, so rollback is one label flip per node.Tested end to end locally on a GKE cluster: install at one build, run counter actors, upgrade to a second build following the document, and confirm the actors resume on the new pool with their state intact. Rollback and retire were exercised the same way.
Fix #1272, fix #1273.