openssh multiplexing Connect() hangs until ControlPersist expires (or SIGINT)

[kke]

openssh multiplexing Connect() hangs until ControlPersist expires (or SIGINT)

Environment

• rig v2.1.0 (same code present since v2.0.0)
• protocol/openssh with multiplexing enabled (the default; DisableMultiplexing == false)
• Surfaced via k0sctl v0.32.1: k0sctl v0.32.1 connection hangs k0sctl#1112

Summary

When connecting with the openssh protocol and multiplexing enabled, Connection.Connect() can block for up to ControlPersist (600s ≈ 10 minutes) even though the SSH control master starts successfully and authentication succeeds. Pressing Ctrl-C (which SIGINTs the whole process group) unblocks it immediately and the connection then reports as established.

In k0sctl this manifests as the "Connect to hosts" phase appearing to hang; only after the user interrupts do all hosts print "connected", after which the run aborts on the now-cancelled context.

Root cause

Connection.Connect() starts the control master like this (protocol/openssh/connection.go):

args = append(args, "-N", "-f")          // -f: daemonize (fork to background) after auth
args = append(args, opts.ToArgs()...)    // ControlMaster=true, ControlPersist=600, ...
args = append(args, c.args()...)

cmd := exec.CommandContext(ctx, "ssh", args...)
errBuf := bytes.NewBuffer(nil)
cmd.Stderr = errBuf                       // <-- a *bytes.Buffer, NOT an *os.File
...
if err := cmd.Run(); err != nil {         // Run = Start + Wait

Because cmd.Stderr is not an *os.File, os/exec connects the child's stderr through an os.Pipe and starts a copier goroutine. cmd.Wait() (inside Run) does not return until that goroutine sees EOF — i.e. until every process holding the pipe's write end has closed it.

ssh -N -f daemonizes: after authenticating, it forks a background control-master process (kept alive ControlPersist=600s) that inherits the stderr pipe write end. So the foreground ssh exits 0 and the master is up, but the inherited pipe stays open, and cmd.Run() blocks in Wait() until:

• the backgrounded master exits (idle ControlPersist = ~10 min), or
• the process group is signalled (Ctrl-C), which kills the master, closes the pipe, and lets Wait() return nil.

This exactly matches the k0sctl report: a ~10-minute hang, and "connected" printed only after Ctrl-C.

Minimal reproduction of the os/exec mechanism (no SSH host needed)

A foreground process that forks a backgrounded child inheriting a non-*os.File stderr makes Run() block until the child exits:

func TestPipeLeak(t *testing.T) {
    cmd := exec.Command("sh", "-c", "(sleep 20 &) ; echo hi 1>&2 ; exit 0")
    cmd.Stderr = &bytes.Buffer{} // non-*os.File → os/exec uses a pipe + copier goroutine
    start := time.Now()
    _ = cmd.Run()
    t.Logf("Run() returned after %v", time.Since(start))
}

Run() returned after 20.025046692s

The foreground sh exits immediately, but Run() waits the full 20s for the backgrounded sleep to release the inherited stderr pipe. ssh -N -f behaves the same way, with ControlPersist (600s) as the effective ceiling.

Suggested fix

Don't let os/exec create a pipe for the control master's stderr, since the pipe write end is inherited by the long-lived daemon. Options, best first:

1. Redirect the master's stdout+stderr to a real *os.File (e.g. a temp file), then read it back for host-key detection. When the writer is an *os.File, os/exec dups the fd directly and starts no copier goroutine, so Wait() returns as soon as the foreground ssh exits regardless of what the daemon inherits:

   f, err := os.CreateTemp("", "rig-ssh-master-*.stderr")
   if err != nil { /* ... */ }
   defer os.Remove(f.Name())
   cmd.Stdout = f
   cmd.Stderr = f
   if err := cmd.Run(); err != nil {
       _ = f.Sync()
       b, _ := os.ReadFile(f.Name())
       errOut := string(b)
       if isHostKeyError(errOut) { /* ErrNonRetryable ... */ }
       /* ... */
   }
   _ = f.Close()

2. Use os.Pipe() explicitly and close rig's own write end right after cmd.Start(), draining the read end in a bounded goroutine. Then the copier is under rig's control and EOF no longer depends on the daemon. (More code than option 1 for the same effect.)

3. Set cmd.WaitDelay (Go 1.20+) as a mitigation so Wait() doesn't block forever on the leaked fd. This bounds the hang but reports an error and forcibly severs I/O, so it's a fallback rather than a real fix.

Option 1 is the smallest correct change and also removes the incidental capture of the daemon's stdout.

Notes / open questions

• The DisableMultiplexing code path is unaffected: it runs ssh ... -- "exit 0" with no -f, so nothing daemonizes and the pipe closes when ssh exits. Workaround for users: set disableMultiplexing: true on the openssh connection (or use the pure-Go ssh protocol, which spawns no subprocess).
• The v1 (v0.21.x) OpenSSH.Connect() used a very similar pattern (-N -f with cmd.Stderr = io.MultiWriter(os.Stderr, &errBuf), also non-*os.File), so this hazard may be long-standing rather than a clean v2 regression; whether it actually fires appears to depend on the local ssh client's fd handling after -f. A bisect / confirmation of when it started biting in practice would help, but the fix above stands regardless.