Skip to content

fix(scaffold): servers a cli app starts never outlive it; io.pilot.redis 8.6.3 - #115

Draft
TeoSlayer wants to merge 2 commits into
fix/otto-lifecyclefrom
fix/redis-lifecycle
Draft

TeoSlayer wants to merge 2 commits into
fix/otto-lifecyclefrom
fix/redis-lifecycle

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #114 (fix/otto-lifecycle), which is stacked on #111. This PR adds one commit. It depends on #114's child guard (childguard.go), childProcAttr/lockForChild and shutdown drain, and on #111's watchParent.

Problem

The harness sweep ran the published io.pilot.redis 8.6.2 bundles on darwin/arm64, darwin/amd64 (Rosetta), linux/arm64 and linux/amd64 (docker --init). It found these leaks on every platform:

Published 8.6.2 (measured)
Daemonized server redis.start runs redis-server --daemonize yes. The server setsid()s out of the adapter's process group and is reparented to pid 1. It survives the adapter's SIGTERM (LEFT pid=93603 ppid=1 pgid=93603 redis-server, darwin-arm64), the supervisor's stop, which SIGKILLs the adapter's pid, and supervisor death. On Linux it survives even when Pdeathsig killed the adapter (PASS: app died 3ms after parent SIGKILL … LEFT pid=1567 ppid=1 redis-server). Pdeathsig, a group kill, reapStale and #114's guard cannot reach it.
Passthrough daemon redis.exec ["redis-server",…,"--daemonize","yes"] leaks the same way.
Passthrough escape args[0] goes to the redis dispatcher, which only checks [ -x "$dir/$tool" ]. ["../"×40+"bin/sh","-c","echo ESCAPED uid=$(id -u)"] prints ESCAPED uid=501 (macOS) and uid=1000 (Linux).
Binds all interfaces CONFIG GET bind returns * -::* and lsof shows TCP *:46391 (LISTEN), although the docs say 127.0.0.1.
False success A second redis.start on a busy port returns exit 0, because the server forks before it binds.

Fix

Template:

  • cli.service (service.go.tmpl, svcproc_{linux,other}.go.tmpl) marks a method that starts a long-running server the adapter owns.
    • The server runs as the adapter's foreground child in its process group.
    • force_args are appended last, so --daemonize no wins over the caller and any config file.
    • The call returns once the server is ready: ready_tcp accepts, or, on a passthrough route, the server has stayed up for ready_after.
    • A ready_tcp address that is already accepting fails the call before anything starts.
    • A server that is not ready within ready_timeout is stopped and the call fails.
    • A process that exits before it is ready (--version, a config error) returns its output like a command.
  • How a service is stopped:
  • cli.tools is an allowlist for a passthrough route's args[0] (bare names only).
  • Validation:
    • A passthrough service needs both cli.tools and service.tools.
    • ready_tcp must be host:port.
    • ready_timeout must be below the method timeout.
  • Documented in docs/CLI-ADAPTER.md.

submissions/io.pilot.redis → 8.6.3:

  • redis.start becomes a service: --bind 127.0.0.1, ready_tcp 127.0.0.1:${port}, force_args --daemonize no, and its log tail is returned on failure.
  • redis.exec allows only the six bundled tools, and runs redis-server and redis-sentinel as services.
  • The docs text no longer says "daemonized".
  • The bundled Redis is still 8.6.2, with the native assets and shas unchanged. See the version note below.

Tests

TestCLIServiceLifetime builds a real adapter. It covers:

Every stop must leave the adapter's process group empty, and a clean stop must SIGTERM the server rather than leave it to the guard's SIGKILL. TestCLIServiceConfig covers the validation rules.

Verified on real redis bundles built from this commit

I built all four platforms through publish.BuildBundle from submissions/io.pilot.redis/submission.json at 8.6.3, signed with a TEST-ONLY key; nothing was published. pilot-app verify passes on each tarball, and verify-submission passes. Compared with the published 8.6.2 bundle, the file layout, grants, exposes and install.json assets are identical; only the version and the signing key differ.

The harness ran 7 passes per platform: help×200, exec×200, server (start then ping×200), setget×200, ptdaemon (passthrough --daemonize yes), inflight (a running redis-cli SUBSCRIBE), and probes. Each pass has lifecycle, supervisor-stop (SIGKILL of the pid) and orphan phases:

  • darwin/arm64 and darwin/amd64 (Rosetta):
    • Every phase ends with pgroup_left=0 descendants_left=0 run_dir_procs_left=0.
    • SIGTERM: exit 0 within 1–92 ms.
    • Orphan without Pdeathsig: SELF-EXIT after 103–308 ms.
    • fd is flat (10→10, 11→11, 12→12, 15→15).
    • On 8.6.2 the server survived SIGTERM, SIGKILL and orphaning, and on darwin the orphaned adapter kept serving.
  • Probes:
    • A second redis.start fails with 127.0.0.1:<port> is already accepting connections.
    • The path escape is refused with args[0] must be one of redis-server, redis-cli, ….
    • redis-server --version over redis.exec returns its output with exit 0.
    • CONFIG GET bind returns 127.0.0.1.
  • linux/arm64 and linux/amd64 (docker --init, uid 1000, golang:1.25 image; amd64 emulated), run at 6400bfc:
    • Go: TestCLIServiceLifetime, TestCLIServiceConfig, TestCLIInflightServerE2E and TestGeneratedAdapterLifecycleE2E pass on both (ok … 21.9s arm64, ok … 35.0s amd64). They also pass on darwin/arm64.
    • Bundles: a second checker drove the same workload on all 4 platforms: help×200 and ping×200 (200/200), redis.start, a passthrough redis-server --daemonize yes, and an in-flight redis-cli SUBSCRIBE. It ran three stop phases: SIGTERM, a pid-only SIGKILL, and supervisor SIGKILL. On Linux the supervisor-kill phase ran with and without Pdeathsig.
    • 8.6.3: 14/14 runs PASS. No process under the run dir remains, and neither server port accepts connections afterwards. SIGTERM exits with code 0 in 28–165 ms. With Pdeathsig the app dies 12–13 ms after the supervisor is SIGKILLed; without it the app exits on its own after 306–309 ms.
    • Published 8.6.2, same checker: 14/14 FAIL. On Linux, after the app has died from Pdeathsig, the process list still shows 42 ppid=1 pgid=42 …/redis-server and both ports stay open. Without Pdeathsig the app survives: ORPHAN: app alive 6s after supervisor SIGKILL, still serving=true.
    • Probes on Linux: the path escape is refused, CONFIG GET bind returns 127.0.0.1 (8.6.2 returns * -::* and ESCAPED uid=1000), and a second start on the same port fails.
    • The adapter binaries are reproducible: rebuilt with a different signing key, all 4 bin/redis-app sha256s are byte-identical. For the real-key rebuild, only the manifest signature and tarball sha change.

Before merging

  • Upload first. This is a RICH submission, so on merge publish-rich-from-r2.sh builds the catalogue entry from bundles/io.pilot.redis/8.6.3/ on the prod R2 bucket. Rebuild with the redis publisher key (ed25519:QXvscnyXJ5L/Bmy7oCo5dPC1HrEOGa7WFWBfH5ktejM=) and upload all four tarballs before merging (docs/UPDATING-BUNDLES.md).
  • Version number. 8.6.3 is the adapter version; the server is still Redis 8.6.2. A bump is required: the registry is write-once, and the supervisor only restarts an app when app_version changes. Upstream Redis already has 8.6.3–8.6.7, so the alternative is to stage Redis 8.6.7 native assets and ship 8.6.7.
  • Native assets. They are still on the R2 dev bucket (pub-2328865f…), and so is the manifest's net.dial grant, so a first spawn without network fails. This PR does not change that.

🤖 Generated with Claude Code

… uses them

io.pilot.redis 8.6.2 leaked on darwin/arm64, darwin/amd64, linux/arm64 and
linux/amd64 (harness runs on the published bundles):

- redis.start ran `redis-server --daemonize yes`. The server setsid()s out of
  the adapter's process group and is reparented to pid 1, so it survived the
  adapter's SIGTERM ("LEFT pid=93603 ppid=1 pgid=93603 redis-server"), the
  supervisor's stop (SIGKILL of the adapter's pid) and the supervisor's own
  death, even on Linux where Pdeathsig killed the adapter. Pdeathsig, the
  supervisor's group stop, reapStale and the child guard cannot reach it.
- redis.exec ["redis-server",...,"--daemonize","yes"] leaked the same way, and
  a relative args[0] ran any host binary through the `redis` dispatcher:
  ["../"x40+"bin/sh","-c","id -u"] printed "ESCAPED uid=501" (macOS) and
  "uid=1000" (Linux).
- redis.start bound *:port (`CONFIG GET bind` = "* -::*") although the docs
  say 127.0.0.1, and a second start on a busy port returned exit 0 because
  the daemonized server forks before it binds.

Template:
- cli.service: a method that starts a long-running server the adapter owns.
  The server runs as the adapter's foreground child in its process group, and
  force_args are appended last, so `--daemonize no` wins over the caller and
  any config file. The call returns once the server is ready: ready_tcp
  accepts, or, on a passthrough route, it has stayed up for ready_after. A
  ready_tcp address that already accepts fails the call before anything
  starts. A server that is not ready within ready_timeout is stopped, and a
  process that exits first (--version, a config error) returns its output
  like a command. How the server is stopped:
  - on a clean stop, StopServices sends SIGTERM, then SIGKILL after 8s;
  - while a server runs it holds the child guard (childguard.go), so a
    SIGKILLed adapter takes it down on every OS;
  - on Linux the server also gets SIGTERM from the kernel (Pdeathsig) when
    the adapter dies.
- cli.tools: an allowlist for a passthrough route's args[0] (bare names only).
- Validation: a passthrough service requires cli.tools and service.tools, and
  ready_timeout must be below the method timeout.
- Documented in docs/CLI-ADAPTER.md.

redis submission (8.6.2 -> 8.6.3; the bundled Redis stays 8.6.2):
- redis.start is a service: ready_tcp 127.0.0.1:${port}, --bind 127.0.0.1 and
  force_args --daemonize no.
- redis.exec allows only the six bundled tools, and runs redis-server and
  redis-sentinel as services.

Tests:
- TestCLIServiceLifetime covers SIGTERM and SIGKILL for enumerated and
  passthrough services, supervisor death through the mini supervisor, three
  services stopped by one SIGTERM, one-shot replies, allowlist refusals, a
  busy port and a server that never gets ready. Each stop must leave an empty
  process group, and a clean stop must SIGTERM the server.
- Mutation checks on darwin/arm64:
  - With the service not holding the guard, both SIGKILL cases fail ("timed
    out after 10s waiting for server to die after adapter sigkill").
  - Without StopServices, the three SIGTERM cases fail ("server was not
    stopped with SIGTERM").
- TestCLIServiceConfig covers the validation rules.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

A/B report — io.pilot.redis (vanilla vs Pilot)

Equivalent commands run two ways: the vanilla CLI binary vs the Pilot adapter (socket mode). CI runs non-VM commands only (GitHub runners have no KVM). Full HTML report (commands, outputs, timings, adapter-generated help): download from the run artifacts.

Command Vanilla ms Pilot ms Δ Exit match
redis-cli --version 4 7 +3 ✓
redis-cli --help 4 6 +2 ✓
redis-server --help 6 9 +3 ✓
redis-cli --version (exec) 3 6 +3 ✓

…, not after a fixed wait

A passthrough service counted as ready once it had stayed up for ready_after
(1s for redis.exec). The harness showed that a freshly staged binary's first
launch takes longer than that to listen. `redis.start`, which probes its
ready_tcp, took this long to become ready in the lifecycle phase:

- darwin/arm64: 1.5-1.9s (a relaunch of the same files: 53ms)
- darwin/amd64 under Rosetta: 3.2-4.5s (relaunch: 112ms)

So `redis.exec ["redis-server","--port",P,...]` reported "started" and the
next ping got "Could not connect to Redis at 127.0.0.1:P: Connection refused"
(darwin/arm64 and darwin/amd64, 2 of 2 runs).

cli.service.ready_port_flag (passthrough only) names the argv flag that
carries the server's port. When a call passes it (`--port N` or `--port=N`,
the last one wins), the server is ready once 127.0.0.1:N accepts, and the call
fails fast if something already accepts there. Without the flag, readiness is
still ready_after. redis.exec sets ready_port_flag to "--port".

Test: TestCLIServiceLifetime/passthrough-ready-port uses a server that
listens only after 1.5s, in both forms, and checks that a busy port is
refused. With the port probe disabled it fails ("returned after 309ms,
before the server listened"; "reported started but 127.0.0.1:53188
refuses").

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants