Skip to content

fix(scaffold): VMs a smol call starts never outlive the app; io.pilot.smol 1.2.1 - #121

Draft
TeoSlayer wants to merge 2 commits into
fix/redis-lifecyclefrom
fix/smol-lifecycle
Draft

TeoSlayer wants to merge 2 commits into
fix/redis-lifecyclefrom
fix/smol-lifecycle

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Stacked on #115 (fix/redis-lifecycle), which is stacked on #114 and #111. This PR uses their child guard (childguard.go), shutdown drain, watchParent and cancel-with-SIGTERM, and adds two commits. Review those PRs first. Once they merge, retarget this PR to main.

Problem

These leaks were measured on the published io.pilot.smol 1.2.0 darwin/arm64 bundle, running real smolvm 1.2.0 VMs. The supervisor's spawn flags were used, with an isolated HOME and app dir.

Published 1.2.0 (measured)
Ephemeral VM orphaned smol.exec ["machine","run",…] was stopped mid-run by an app SIGTERM. Its VM process was left running: LEFT 61336 ppid=1 pgid=61336 smolvm-bin _boot-vm …, still alive 20 s later. Upstream machine run tears its VM down only on SIGINT (SigintGuard), and SIGTERM or SIGKILL of the CLI orphans it. The earlier sweep saw the same after a process-group SIGTERM (2/2) and after the 60 s call deadline (1/1).
Persistent VMs outlive the app machine start --name t1 and machine run -d --name t2 were followed by an app SIGTERM. Both VMs were LEFT … ppid=1, and machine ls still showed them running 30 s later. The earlier sweep found the same after the supervisor's SIGKILL. Nothing stops them on uninstall.
Junk darwin/amd64 bundle BuildBundle builds all 4 DefaultPlatforms, although install.json has no smolvm for darwin/amd64 (upstream ships none). The unlisted io.pilot.smol-1.2.0-darwin-amd64.tar.gz on R2 exits 1 on every spawn: no asset for darwin/amd64.

Fix

cli.env_rules (passthrough only) adds environment to the child of a matching call. smol sets SMOLVM_BOOT_BINARY=${command_dir}/smolvm-bin for a machine run without -d/--detach. That arms smolvm's own parent-death watchdog in the VM process (src/agent/manager.rs, watch_parent), so the VM dies with its CLI whatever signal the CLI gets. A detached run does not get the variable: its VM has to outlive the CLI.

cli.teardown (passthrough only) handles a call that leaves something running outside the process group. When a call matches, a record is opened: "run <base> <run argv> with this env", with ${name} taken from name_flags or default_name. The record is opened before the CLI runs, so a start that is cut short is still covered.

  • Clean stop (SIGTERM, or watchParent seeing the daemon gone): main runs every open record (30 s each, concurrently) before it releases the guard.
  • Adapter SIGKILLed: the child guard stays up while any record is open. It gets each record over its pipe, and on the adapter's death it runs the records before it kills the group.
  • Closing a record: a successful call under clear_prefixes for the same name closes it, so nothing is stopped twice. A start that exits non-zero closes the record it opened.
  • Flags are matched only before --, so a guest command's own flags don't count.

smol.exec declares two teardown rules:

  • machine start: stopped with machine stop --name ${name} (default name default).
  • machine run -d|--detach: stopped the same way.

The smol.exec description now says that VMs stop when the app stops and that their disks persist. This is a behaviour change: product needs to sign off on it. The version is bumped to 1.2.1.

BuildPlatforms: an app with native assets is built only for the platforms it has an asset for. smol now builds darwin/arm64, linux/amd64 and linux/arm64.

Verification

  • GOWORK=off go test ./... -count=1: all packages ok. That includes the new tests:
    • TestCLITeardownE2E, which uses a fake smolvm that models the VM's own process group, the watchdog and the SIGINT-only teardown.
    • TestValidateCLILifecycle
    • TestBuildPlatforms_AssetsLimitTargets
  • TestCLITeardownE2E passed 3 times out of 3 (-count=3).
  • Negative control: with the rules disabled (LifecycleGo returning ""), 7 of the 8 subtests fail. The 8th only checks that nothing is stopped needlessly.

Real smolvm 1.2.0 on darwin/arm64, using bundles built from this branch with a test-only publisher key:

Scenario 1.2.0 (published) This branch
machine run in flight, app SIGTERM VM LEFT (ppid 1) after 20 s 0 left, gone after 540 ms
… app SIGKILL (the supervisor's stop) orphaned in earlier sweep (4/5) 0 left, 645 ms
… process-group SIGTERM orphaned in earlier sweep (2/2) 0 left, 151 ms
machine start + run -d, app SIGTERM 2 VMs LEFT, machine ls = running 0 left, 154 ms; machine ls = stopped
… app SIGKILL 2 LEFT in earlier sweep 0 left, 148 ms (the guard ran machine stop)
… process-group SIGTERM – 0 left, 105 ms
… supervisor SIGKILLed (orphan) app kept serving app exits in 337 ms; 0 VMs left
machine run … echo hi-from-vm – exit 0, stdout hi-from-vm; 0 VMs after

Other checks:

  • smol.version ×200:
    • darwin/arm64: 200/200, fd 10→11 (the guard pipe), RSS 17→21 MB.
    • linux/arm64 in Docker: 200/200, fd 8→11.
    • linux/amd64 in Docker: 200/200, fd 20→21.
    • SIGTERM exits in 1–5 ms with code 0; the socket is removed and the process group is empty.
  • Not covered:
    • Linux VM paths were not run (Docker Desktop has no /dev/kvm). Linux relies on the same watchdog and teardown code, which the fake-smolvm e2e covers.
    • machine fork, pack run and the hidden machine network-test (which auto-starts a VM) have no teardown rule.
    • VM state still lives in the user's real $HOME (~/Library/Caches/smolvm), so disks persist after uninstall.
    • A guard running machine stop from $APP while uninstall deletes $APP is untested.

Release (needs maintainers)

  1. Merge fix(scaffold): generated adapters exit with their daemon and drop abandoned calls #111 → fix(scaffold): a cli adapter's in-flight children never outlive it (io.pilot.otto) #114 → fix(scaffold): servers a cli app starts never outlive it; io.pilot.redis 8.6.3 #115 → this PR.
  2. Build io.pilot.smol 1.2.1 with the smol publisher key (ed25519:54rnZ8…). That produces 3 platforms.
  3. Upload the bundles to R2 under bundles/io.pilot.smol/1.2.1/. Never overwrite 1.2.0.
  4. Update the web4 catalogue entry: the bundles map, plus top-level bundle_url/bundle_sha256/bundle_size from linux/amd64. Then re-sign catalogue.json.
  5. Delete the unlisted 1.2.0 darwin-amd64 bundle.
  6. Keep darwin/amd64 unlisted: pilotctl ≥ v1.12 refuses cleanly with "has no bundle for this platform".

🤖 Generated with Claude Code

teovl and others added 2 commits September 24, 2026 14:28
…as an asset for

io.pilot.smol 1.2.0 published a darwin/amd64 adapter bundle although
install.json has no smolvm for darwin/amd64 (upstream ships none). The
bundle was left unlisted, but it exits 1 on every spawn: "install assets:
stage: no asset for darwin/amd64". BuildBundle now builds DefaultPlatforms
filtered to the asset platforms (BuildPlatforms); apps without assets still
build for all four.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
….smol 1.2.1

Measured on io.pilot.smol 1.2.0 (darwin/arm64, real smolvm 1.2.0 VMs):
- an ephemeral `machine run` VM is orphaned (ppid 1, own process group)
  when its CLI gets SIGTERM or SIGKILL: app stop, group stop, call deadline.
  smolvm tears it down only on SIGINT.
- VMs from `machine start` and `machine run -d` keep running after the app's
  SIGTERM, the supervisor's SIGKILL and the supervisor's death.

New passthrough route options:
- cli.env_rules: environment for the child of a matching call. smol sets
  SMOLVM_BOOT_BINARY=${command_dir}/smolvm-bin for a foreground run, which
  arms smolvm's own parent-death watchdog in the VM (manager.rs).
- cli.teardown: a matching call opens a record "run <stop argv>"; the
  adapter runs open records on a clean stop, and the child guard (which stays
  up while records are open) runs them if the adapter is killed. An explicit
  stop/delete closes the record; a failed start closes what it opened.

smol.exec declares both; the description says VMs stop with the app.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

A/B report — io.pilot.smol (vanilla vs Pilot)

Equivalent commands run two ways: the vanilla CLI binary vs the Pilot adapter (socket mode). CI runs non-VM commands only (GitHub runners have no KVM). Full HTML report (commands, outputs, timings, adapter-generated help): download from the run artifacts.

Command Vanilla ms Pilot ms Δ Exit match
Version 5 6 +1 ✓
Top-level help 4 6 +2 ✓
machine --help 4 6 +2 ✓
pack --help 4 6 +2 ✓
config --help 4 6 +2 ✓

@TeoSlayer
TeoSlayer marked this pull request as draft October 1, 2026 14:31
@TeoSlayer

Copy link
Copy Markdown
Contributor Author

Converted to draft rather than merged. It is based on draft #115 and carries its commits, the description says product still has to sign off on VMs stopping with the app, and the smol 1.2.1 bundles are not in R2 yet (publish-on-merge would fail). It also needs rework after #116 landed today: r.base[0] is no longer the staged binary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants