Repository navigation
feat: Add drain to EgressShard - #32
Merged
Merged
Conversation
Compute nodes are moving from sending each tenant to one configured egress shard to spreading tenants across every shard in the cluster. Taking a shard out of that rotation without breaking the sessions it holds needs a signal the compute nodes can read from the shard itself. Add spec.drain. A draining shard gets no share of new tenant addresses, but compute nodes keep sending it the flows already pinned to it until they go idle. The shard itself changes nothing: it keeps translating, advertising its SID and receiving replies, so it is safe to remove once its session count reaches zero. The field is mutable, unlike the shard's identity, and shows as a DRAIN print column. Compute nodes still running in ordered mode, where each tenant goes to the first configured shard that resolves, ignore it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scotwells
approved these changes
Oct 10, 2026
This was referenced Oct 10, 2026
privateip
added a commit
to datum-cloud/galactic
that referenced
this pull request
Oct 12, 2026
…mode
galactic-cni run now owns how tenant VRFs reach the egress shards. GALACTIC_CNI_EGRESS_MODE=ordered, the default, keeps today's behaviour: each VRF uses the first reachable shard in GALACTIC_CNI_EGRESS_SHARD_SIDS. GALACTIC_CNI_EGRESS_MODE=hashed makes every eligible EgressShard in the cluster a member of the shard groups usid_egress picks from, so each shard carries about 1/N of the metro's egress and losing one costs only its sessions.
In hashed mode the installer watches EgressShards. A shard is a candidate once its status carries a shardSID with Programmed and Ready true; it is draining while spec.drain is set, and a terminating shard is left out. It joins a translation class's group only if its status can translate that class: an IPv6 address for NAT66; an IPv4 address and the matching nat64Prefix, or translatesWellKnownPrefix for 64:ff9b::/96, for each NAT64 prefix. More than three NAT64 prefixes in hashed mode stops the container at startup. A candidate takes tenants only while its SID resolves to an uplink neighbor, so a dead node's stale status does not matter. A netlink route and neighbor watch rebuilds the groups as soon as a route moves, and the 30s sweep stays as a backstop. Until the watch has listed the cluster once, the groups keep their last members.
Going back to ordered mode publishes every enabled group as CLOSING, moves every route back to the static list, and disables the groups only after a sweep that read and wrote every route without error and found none still naming a group, other namespaces' routes included. A failed sweep leaves the groups forwarding and is retried.
GALACTIC_CNI_EGRESS_HASH picks source (default, one public address per tenant address) or flow hashing, GALACTIC_CNI_EGRESS_PIN_IDLE sets how long a tenant address stays pinned to its shard after its last packet (default 2h4m, the shard's established-TCP timeout; 0 turns pins off), and GALACTIC_CNI_EGRESS_POOL_MIN_ACTIVE sets the floor the pool alert compares against. An invalid value stops the container at startup.
New metrics: galactic_cni_egress_shard_{packets,bytes}_total{shard_sid}, galactic_cni_egress_shard_selections_total{result}, galactic_cni_egress_pool_members{class,state} (with an ineligible state), galactic_cni_egress_pool_min_active, galactic_cni_egress_pool_enabled{class}, and the control loop's own state: galactic_cni_egress_mode, galactic_cni_egress_shard_watch_synced, galactic_cni_egress_group_routes{class}, galactic_cni_egress_sweep_errors_total{stage} and the last successful sweep's timestamp. A node told to run hashed mode that never published a group says so instead of exporting nothing.
galactic-cni's ClusterRole gains get, list and watch on egressshards. go.datum.net/network moves to the commit that adds EgressShard spec.drain (datum-cloud/network#32); it moves to the release once that is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Galactic compute nodes are gaining a mode that spreads tenants across every egress shard in a cluster instead of sending each tenant to one configured shard. Removing a shard from that rotation without breaking the sessions it holds needs a signal the compute nodes can read from the shard itself.
This adds
spec.draintoEgressShard. A draining shard gets no share of new tenant addresses, but compute nodes keep sending it the flows already pinned to it until they go idle. The shard itself behaves the same: it keeps translating, advertising its SID and receiving replies. The field is mutable and shows as aDRAINprint column.Compute nodes in ordered mode, the current default, ignore it.
Compatibility
Additive and optional. Shards stored before this change read as not draining. Galactic's lab and staging channel install CRDs from main, and nothing reads the field until the galactic change that uses it merges.
Galactic's active/active egress branch currently pins this branch's pseudo-version; after this merges it needs a release to re-pin to.
Test plan
task generateleaves no diffdrainround-trips and is omitted when false)spec.drain🤖 Generated with Claude Code