Skip to content

feat: Serve instance shell sessions from compute cells - #390

Merged
scotwells merged 20 commits into
mainfrom
feat/instance-console-session-agent
Sep 30, 2026
Merged

feat: Serve instance shell sessions from compute cells#390
scotwells merged 20 commits into
mainfrom
feat/instance-console-session-agent

Conversation

@scotwells

@scotwells scotwells commented Sep 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Each compute cell can now serve shell sessions into its instances. Shell traffic reaches the cell only through an encrypted Datum Connect tunnel, and the agent stops every process a session started when the session ends, whatever ends it. A session is claimed only after the cell has checked the instance, its container, a shell and the requested command, so a request that cannot work fails at once with a clear reason. This first build serves general-purpose (Kata) instances.

Stacked on #387. Design: #384. The tunnel endpoint depends on datum-cloud/app#192, which is not released yet, so the endpoint image tag is a placeholder until it is.

What's included

  • Shell agent, a new binary in the existing compute image. It shares the session API and the connection protocol with the manager, so it ships in the same image under the same tag rather than in a new one.
  • Claiming with optimistic concurrency. Each instance gets at most three sessions, and each agent holds at most 200 open ones.
  • Connection protocol v1, using the shared protocol package: signed requests, the 403/404/409/410/503 refusals, the connection recorded before any byte is relayed, then exactly one closing status and a normal close.
  • Every ending stops every process the session started, background jobs included: normal exit, disconnect, expiry, revocation, crash takeover and maintenance drain. The agent stops the whole session the command leads, not just its process group, and keeps retrying from the orphan sweep until nothing is left. A process that calls setsid itself leaves the session and is not stopped.
  • Disconnect detection: the agent pings the client every 10 seconds and ends a session as Disconnected once the client has sent nothing, not even a pong, for 20 seconds, instead of waiting for the tunnel to idle out.
  • Endpoint identity keys, created by the agent and rotated every 30 days, one agent at a time so the cell keeps taking sessions.
  • Cell packaging as an opt-in component: agents and endpoints paired by ordinal, least-privilege access, an endpoint with no cluster credential, and portable plus Cilium network policies.

Security notes

  • No hub access. The agent holds only its cell's credentials. It claims and ends sessions on the copy in its cell, and Karmada carries that status back to the hub.
  • Exec limited to tenant pods. An admission policy shipped with the component lets the agent exec only into pods in tenant (ns-) namespaces, on top of the agent's own checks.
  • No egress out of the cell. The Cilium policy lets the agent reach only the cell apiserver and DNS.
  • Session existence is observable. Before the upgrade, an unknown session gets 404 and a known one with a bad signature gets 403, so a caller can learn that a session UID exists. Session UIDs are random and unguessable, so this is accepted.

Cleanup confirmation

The agent holds the cell copy with its finalizer from the claim until the session's processes are confirmed stopped.

It records a session's end at once, so the client learns of it, but sets endedAt only when nothing the session started is left running. If a stop fails it keeps retrying, and sets endedAt once a retry succeeds. The control plane takes endedAt as confirmed cleanup, because Karmada deletes the cell copy without waiting for it and nothing can report back after that.

Where the contract did not fit

  • A frame over 16 MiB ends the session as Invalid.
  • A cell that does not run the session's instance leaves the session alone. Sessions no cell claims are ended by the control plane as Unavailable, and the agent never claims a session that has ended or been revoked.
  • A client message split across several WebSocket frames is joined before it reaches the apiserver, which reads only the first frame of a fragmented message.

Test plan

  • Build, lint and tests pass and both components render. Unit tests cover claim races (including losing to a revoke and a lagging cache retrying the agent's own claim), the exact three-session cap, liveness takeover, deadlines, revoke and deletion, drain, the orphan sweep, failed stops that leave the end unconfirmed until a retry succeeds, signature and refusal codes, the closing status for every ending, ping/pong disconnect detection, fragmented client messages, bounded exec calls, and the kill script against a fake /proc
  • End to end on a local kind cluster running Kata, before status moved to the cell copy: a relay-only tunnel through the app#192 endpoint image, exit codes from the guest, revocation, disconnect, expiry, not connected, no shell, command not found, the three-session cap, drain notice, crash recovery, orphan sweep, and wrong-key and second-connection refusals
  • On a local Karmada v1.18.3 hub with pull-mode cells and no hub credential anywhere, including a Kata cell: claims reached the hub in well under a second, revoke, disconnect and expiry each stopped a background job under busybox sh and a nohup ... & disown job under bash before the end was confirmed, and another cell's refusals and forged claims were ignored
  • Staging, with Cilium enforcing the policies

Related to datum-cloud/enhancements#900

@scotwells
scotwells force-pushed the feat/instance-console-session-agent branch 2 times, most recently from a3eea28 to ed43bad Compare September 30, 2026 02:24
@scotwells
scotwells marked this pull request as ready for review September 30, 2026 02:37
@scotwells
scotwells requested a review from a team as a code owner September 30, 2026 02:37
ecv
ecv previously approved these changes Sep 30, 2026
View reviewed changes
@scotwells
scotwells marked this pull request as draft September 30, 2026 11:10
@scotwells
scotwells force-pushed the feat/instance-console-session-agent branch from ed43bad to 711b200 Compare September 30, 2026 11:47
@scotwells
feat: Serve shell sessions from a cell agent
15d994d 
Each compute cell needs a process that turns a delivered
InstanceConsoleSession into a running command in the instance and
guarantees the command stops when the session ends, whatever ends it.

Key changes:
- Claim session copies with optimistic concurrency after checking the
  cell Instance, its pod, the container, a shell and the executable;
  publish the paired endpoint's ID, relays and target on the hub copy
- Cap sessions per instance with three atomic Lease slots; a slot also
  records where the command runs and is held until its processes are
  confirmed stopped
- Renew a liveness Lease; survivors end a lost agent's sessions with
  AgentLost and stop their processes
- Serve connection protocol v1 through the shared consolesession
  package: signed requests, refusal codes, Connected recorded before any
  byte is relayed, and one closing status then close code 1000
- Wrap every command so its process group is stopped on exit,
  disconnect, expiry, revocation, takeover and shutdown
- Drain on SIGTERM, sweep orphaned processes every minute, cap each
  agent at 200 open sessions, and create and rotate the endpoint key
- Hold a finalizer on the hub and cell copies until processes stop, so
  a hub copy that disappears confirms the cell cleaned up
@scotwells
feat: Package the shell agent for compute cells
f0041be 
A cell opts into shell sessions by deploying the agent beside Datum
Connect tunnel endpoints, so the agent is reachable only through its
paired endpoint and the endpoint holds no cluster credential.

Key changes:
- Two two-replica StatefulSets in compute-shell-system, paired by
  ordinal through a headless Service
- Least-privilege RBAC for the agent; the endpoint mounts no service
  account token and reads its key read-only
- Portable NetworkPolicies, and Cilium policies in their own component
  that also police traffic to the node's apiserver
- An opt-in overlay that keeps the namespace intact, since the cell
  overlay renames every namespace it renders

The datum-connect image tag is a placeholder until datum-cloud/app#192
is released.
@scotwells
chore: Ship the shell agent in the compute image
e7864d5 
The agent shares the session API and the connection protocol with the
manager, so building it into the same image and stamping the same tag
into its component keeps the two from skewing, and reuses the existing
release plumbing instead of adding an image.
@scotwells
fix: Retry lost claim races without an error
b4fc139 
Two agents claiming at once is expected: the loser's finalizer or status
write conflicts. Treat that as a retry rather than a reconcile error, and
avoid the deprecated immediate requeue.
@scotwells
fix: Leave unclaimed sessions to the control plane
4232c11 
The control plane now ends sessions no cell claims within 30 seconds as
Unavailable, deciding on the hub copy with optimistic concurrency. The
agent's own two-minute InstanceNotFound fallback would race it, so cells
without the session's instance now simply leave the session alone.

A client that vanishes behind the tunnel is only noticed when the QUIC
connection idles out, 25 to 30 seconds later, and it ended as Completed.
The agent now pings clients and ends silent or departed clients as
Disconnected, stopping the command's process group.

Key changes:
- Use the API's session labels and finalizers instead of local copies
- Treat Unavailable and Disconnected as terminal, so an ended hub copy is
  never claimed; claims still lose to any concurrent write on the hub copy
- Ping the client every 10s; a client silent for 20s is disconnected
  (--ping-interval, --pong-timeout); pongs to the agent's pings are not
  relayed to the apiserver
- Test that gorilla/websocket and client-go's WebSocket executor answer
  the pings and read TTY and non-TTY output through the agent
@scotwells
test: Keep clients alive on unsolicited pongs
1cc3491 
A client whose output is backed up may send unsolicited empty pongs
instead of answering each ping. The agent already counts any inbound
frame as a sign of life; pin that down so a later change cannot start
matching pongs to pings and drop busy clients.
@scotwells
fix: Give commands without stdin an empty input
9c33e83 
Kata leaves an exec's standard input open when the request asks for
none, so a one-off command that reads it, such as
`datumctl compute exec <instance> -- cat`, waited for the whole session
instead of ending. Runc gives such a command /dev/null.

The wrapper now redirects standard input from /dev/null when the session
asks for neither standard input nor a terminal, so the command sees an
empty input on every runtime.
@scotwells
fix: Claim sessions only while the endpoint is ready
4b4b0de 
An agent whose paired tunnel endpoint was down, for example while its
image could not be pulled, still claimed sessions and published the
endpoint's ID. With two agents per cell, half of all sessions then
failed at the client's dial with an unreachable-endpoint timeout and
ended as NotConnected a minute later.

The agent now leaves sessions to its peer, or to the control plane's
Unavailable ending, until its endpoint's pod is ready.
@scotwells
fix: Read hub sessions by name instead of watching
cc98709 
Each cell's hub identity may now only get and patch sessions and write
their status; it cannot list or watch them, so a leaked cell credential
cannot enumerate other locations' sessions. The agent's hub informer
would fail to start under that grant.

The agent now watches only the session copies delivered to its cell and
fetches each one's hub copy by the same namespace and name. Hub-side
changes, such as a deletion or the control plane ending the session, are
picked up by rereading the hub copy every 10 seconds while the session is
open or still holds the agent's finalizer.

Key changes:
- Drop the hub cache and its watch from the controller and the binary
- Poll the hub copy of open and finalizer-held sessions (HubPollInterval)
- Run every test against a hub client that refuses list, watch, create,
  update and delete, as the hub now does
@scotwells
fix: Bound each exec the agent runs in an instance
1c28ea2 
A pod exec that never answers, such as one into a wedged guest, held its
caller forever. Stopping a session's processes runs from reconcile
workers and from the cell-wide orphan sweep, so one hung instance could
tie up workers and stall the sweep for every other session.

Each probe, marker listing and process stop now gives up after the kill
grace plus 30 seconds. A stop that times out counts as not confirmed, so
the slot and marker are kept for the sweep to retry.
@scotwells
fix: Name liveness Leases by the full endpoint hash
0aa7417 
An agent's liveness Lease was named by the first 16 hex digits of its
endpoint ID. Two endpoint keys sharing that prefix would share a Lease,
so one agent's renewals could keep a crashed peer looking alive and delay
the takeover of its sessions.

The Lease is now named by the full SHA-256 of the endpoint ID. Lease
names are DNS subdomains, which fit all 64 hex digits, and hashing keeps
the name valid whatever the hub copy's status claims the ID is.
@scotwells
fix: Reassemble fragmented client messages
64910e9 
The agent relayed a client's frames to the apiserver as they arrived.
The apiserver reads only the first frame of a fragmented message and
drops the rest, so input a client sent in several frames, such as a
large paste, silently lost everything after the first frame.

The agent now joins a fragmented text or binary message and sends it to
the apiserver as one masked frame. Control frames in the middle of a
message are still relayed at once. A continuation with no message to
continue, or a new message inside an unfinished one, ends the session as
Invalid, and a reassembled message over 16 MiB ends it as a single
oversized frame does.
@scotwells
fix: Stop background jobs when a session ends
0bedcd9 
Background jobs outlived the sessions that started them. The agent
stopped a session by signalling the process group of the command it
started, but an interactive shell's job control moves each background
job to a process group of its own. On Kata, `sleep 60 &` and
`nohup sleep 60 &` in busybox sh, and `nohup ... & disown` in bash, kept
running after a disconnect or a revocation, and the orphan sweep could
not find them because the kill had already removed their marker.

Job control never moves a job out of its session, and every exec leads a
session of its own under Kata and runc. The wrapper now records the
session it leads, starting one with setsid where it does not, and the
kill script signals every process in that session.

Key changes:
- Record the wrapper's PID, start time and whether it leads a session or
  only a process group
- Signal each live member with SIGHUP and SIGCONT, then SIGKILL after the
  grace period, and skip zombies and processes older than the wrapper
- Skip the kill when the recorded PID now belongs to a younger process,
  which means the session is long gone
- Keep the marker, and report failure, until no member is left, so the
  sweep retries stragglers
- Test the kill script against a fake /proc, and the wrapper against the
  real one on Linux

A process that calls setsid itself leaves the session and is not
stopped. Verified on Kata with busybox sh and bash, and on runc.
@scotwells
feat: Limit the shell agent's exec to tenant pods
b9926a1 
The agent's ClusterRole grants pod exec in every namespace, because
tenant namespaces are created per project and cannot be listed in RBAC.
A compromised agent could therefore exec into any pod in the cell,
including system workloads and the agent's own namespace.

The shell-agent component now ships a ValidatingAdmissionPolicy on
CONNECT that lets the agent's service account exec only into pods in
tenant namespaces (the `ns-` prefix), and refuses it attach and port
forwarding. The agent's own checks before a claim are unchanged.

Verified on a Kubernetes 1.34 kind cluster: the agent's identity can
exec into a pod in a tenant namespace and is refused in `default` and in
`compute-shell-system`; other users are unaffected.
@scotwells
fix: Allow the shell agent out only to its hub
0b3c8dd 
The base Cilium policy let the agent reach any address on the internet
over HTTPS and left narrowing it to each deployment. A deployment that
forgot the patch shipped an agent holding cell and hub credentials with
open egress.

The base policy now allows the agent out of the cell only to one hub
name, through Cilium's DNS proxy. The name is a placeholder that
resolves nowhere, so an unset deployment reaches no hub instead of the
whole internet, and deployments replace it with their hub's name.
@scotwells
fix: Keep the shell agent off the Karmada hub
c2b3577 
No cell component may write to the Karmada hub any longer. The agent now
claims, connects and ends sessions only on the cell copy, and Karmada's
status reflection carries that status back to the hub copy.

Karmada deletes a cell copy without waiting for it, so a deleted hub copy
cannot report the session's end. The control plane therefore revokes an
open session with an annotation first, and the agent records Revoked only
after it has stopped the session's processes, which is what the control
plane takes as confirmed cleanup.

Key changes:
- drop the hub client, its kubeconfig flags and the hub poll interval
- claim with optimistic concurrency on the cell copy, and keep the slot of
  a claim that already stands when a lagging cache retries it
- hold only the cell copy with the shell-agent finalizer
- end sessions on the revoke annotation, and refuse connections to them
- read the cell copy for the orphan sweep and for draining
@scotwells
fix: Drop the shell agent's hub credential and egress
6313c76 
The agent no longer talks to the Karmada hub, so the cell packaging needs
neither the hub kubeconfig and client certificate nor a way out of the
cell. The agent may now reach only the cell apiserver and DNS.

Key changes:
- remove the federation kubeconfig flag and its projected volume
- let the agent write session status in the cell
- remove the hub FQDN egress rule and its DNS proxy from the Cilium policy
@scotwells
test: Return after nil-check fatals in build tests
81681de 
Lint on this branch fails with staticcheck SA5011 on three nil checks in
the build analysis tests, a file this branch never touches. It is a
golangci-lint cache bug, not a code defect (golangci/golangci-lint#5979,
same root cause as #3228): with a warm analysis cache, staticcheck loses
the fact that t.Fatal never returns, so every nil check followed by a
dereference reads as a possible nil pointer.

Key evidence:
- A clean golangci-lint v2.12.2 run of this head reports 0 issues, on
  macOS and in a Linux container.
- The go.mod change gives this branch its own lint cache key. The cold
  runs that built that cache passed in 130-165s; every warm run since
  has failed in 14-22s with the same six findings.

An explicit return after each fatal makes the nil path terminate
without relying on that fact, so lint passes whatever state the cache
is in.
@scotwells
fix: Confirm a session's end only once its processes stop
8cd6012 
The agent recorded a session's end with endedAt even when it could not stop
the session's processes, and the control plane took that end as confirmed
cleanup while the processes might still run.

The agent still records the end at once, so the client and the project
learn of it, but now sets endedAt only when nothing the session started is
left running. When the stop fails it keeps the slot, retries from the
reconcile and the orphan sweep, and sets endedAt once a retry succeeds. The
control plane waits for endedAt, and records CleanupUnconfirmed if it does
not arrive within five minutes. No API field changes: endedAt already means
when the session ended, and a session whose processes still run has not.

Key changes:
- record endedAt with the end only when the stop succeeded
- set endedAt from the settle, cleanup and sweep paths once a later stop
  succeeds, only for the session that owns the copy
- test failed stops on takeover and on a dropped client, and the sweep
@scotwells
scotwells force-pushed the feat/instance-console-session-agent branch from 1fb6983 to 8cd6012 Compare September 30, 2026 12:23
@scotwells
scotwells changed the base branch from feat/instance-console-session-api to feat/instance-console-session-controllers September 30, 2026 12:24
@scotwells
test: Read the prompt before ending a platform-ended session
34f3026 
TestPlatformEndings ended the session as soon as it reported Connected,
which could race the relay of the command's first output, so the client
sometimes read no prompt and the test failed on CI. The client now reads
the prompt before the session is ended.
@scotwells
scotwells marked this pull request as ready for review September 30, 2026 13:21
Base automatically changed from feat/instance-console-session-controllers to main September 30, 2026 14:58
@scotwells
scotwells dismissed ecv’s stale review September 30, 2026 14:58

The base branch was changed.

@scotwells
scotwells requested a review from ecv September 30, 2026 14:59
@scotwells
scotwells merged commit ba4c87d into main Sep 30, 2026
11 checks passed
@scotwells
scotwells deleted the feat/instance-console-session-agent branch September 30, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ecv ecv ecv approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@scotwells @ecv