fix(machine): wait for the distro's own init before reporting a Machine ready (CORE-66)

[AprilNEA]

Closes CORE-66.

A client operation issued right after MachineService.Start returned could fail with ENETUNREACH. eth0 is configured twice during boot — once by our machine-init, then again from scratch by the distro's own init — and the machine was advertised as ready in between. The second configuration begins by flushing the interface, which is the outage.

Root cause

Tight-loop sampler on /proc/net/route inside the Machine, dumping ps at every change (LINES counts wc -l, so the header makes LINES=1 an empty table):

=== T=0.46 LINES=3          healthy: default via 10.0.2.1 + subnet
   97     1 /arcbox/bin/arcbox-agent serve      <- already serving RPC
  376   355 {networking} /sbin/openrc-run /etc/init.d/networking start
  397    97 /bin/sh -c wget ... http://10.0.2.1:49285/blob    <- succeeds

=== T=0.50 LINES=1          EMPTY
  461   377 ifup -i /etc/network/interfaces eth0
  507   461 {dhcp} /bin/sh /usr/libexec/ifupdown-ng/dhcp
  518   507 /sbin/udhcpc -b -R -p /var/run/udhcpc.eth0.pid -i eth0

=== T=0.51 LINES=3          restored

The flush belongs to the distro's own networking service, not ours. The boot shim's ordering is correct and was never the problem: machine-init runs synchronously, then the agent is backgrounded, then exec chroot . /sbin/init. The gap is one step later — the agent is answering RPCs, which is what readiness gated on, while the distro init that runs after it has not settled. Host-side: agent ready at guest uptime ≈0.5 s, failure at 0.52 s. The window is ~20 ms.

Fix

SystemInfo.distro_init_pending gates wait_for_machine_ready, reusing its existing backoff and 60 s timeout. The guest derives it from a sentinel: machine_init installs a hook ordered at the end of the distro's own boot — a systemd unit After=multi-user.target, or an openrc service that depend()s after * — which writes the boot id to /run/arcbox-boot-done. Pending = the hook exists and the sentinel does not name the current boot.

Not init internals, deliberately. The first implementation polled /run/openrc/rc.starting and systemctl is-system-running, and was completely inert — it never fired once. rc.starting is absent both before openrc runs and after it finishes, so "has not started yet" read as "already done", which is exactly the window being closed. Unit ordering is a public contract; a runtime directory's layout is not. Unit tests pin that ambiguity.

This is the shape comparable runtimes use — none inspect init internals from outside:

runtime	signal
Lima	/run/lima-boot-done holds the instance id, written by injected boot scripts (pkg/hostagent/requirements.go)
Multipass	[ -e /var/lib/cloud/instance/boot-finished ] (wait_for_cloud_init)
Incus	guest declares itself: PATCH /1.0 {"state":"Ready"} on devIncus → volatile.last_state.ready

cloud-init would be the standard vehicle, but the mirrored images are the linuxcontainers default variant, which does not ship it — verified by booting ubuntu-noble and finding no cloud-init at all. The hook is installed directly instead, which the writable overlay already allows.

Safe proto default. The field is distro_init_pending, not _complete, so an agent predating it decodes false and the host proceeds exactly as before rather than waiting out the readiness timeout on a signal that agent never sends.

Validation

Matched conditions, both arms prebuilt with SKIP_BUILD=1:

	machine_network e2e
baseline (master binaries)	4/8 failed
this branch	0/8 failed

Direct evidence the gate engages, rather than only that the flake stopped: alpine/openrc 4 gate hits, ready 0.50 s → 2.16 s; ubuntu/noble systemd 26 gate hits, ready → 13.8 s, no timeout. The hits prove both halves — the gate can only block if the hook was written into the distro, and can only release if the sentinel matched the current boot id.

Also: cargo test -p arcbox-core machine:: (35 pass, incl. 4 new gate tests), agent unit tests for the sentinel and the ambiguity, buf breaking clean, clippy/fmt clean.

Cost and residual risk, stated plainly

• Start now costs what the distro's boot costs: ~2.2 s on alpine, ~14 s on ubuntu/systemd, versus ~0.5 s before. That is the honest price — the machine was not usable during that time; readiness merely claimed it was. Lima and multipass block the same way for the same reason. If that is too long, the alternative is to wait for a network target rather than full boot completion, which trades contract clarity for latency — worth deciding explicitly rather than by accident.
• A hook that installs but never runs would hang readiness to the 60 s timeout and fail Start. Install is best-effort and rolls back on partial failure, so a failed install degrades to the old no-wait behaviour; what cannot be ruled out is a distro whose init ignores the hook. Verified on alpine 3.24 and ubuntu noble; debian and the other streams in mirror.toml are unverified.
• Adjacent and still open (not this PR): SystemInfo.ip_addresses comes from hostname -I falling back to hostname -i, and busybox's -i resolves the guest's hostname through DNS rather than enumerating interfaces — which is why alpine reports a 198.18.11.51 fake-IP while ubuntu (coreutils -I) correctly reports 10.0.2.2.

[linear-code]

CORE-66

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Greptile Summary

The PR prevents MachineService.Start from reporting a machine ready before the guest distro’s own init has settled.

• Adds systemd and OpenRC boot-completion hooks that write a boot-ID sentinel.
• Exposes the pending state through SystemInfo and incorporates it into both readiness probe paths.
• Preserves compatibility with older agents through the proto3 false default.
• Adds unit coverage for readiness gating and sentinel matching.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code failure established for the supported and validated guest configurations.

The readiness gate is applied consistently across both probe transports, retains compatibility with older agents, and the installed hooks use a boot-specific sentinel to release the gate only after distro initialization.

Important Files Changed

Filename	Overview
guest/arcbox-agent/src/boot_done.rs	Adds best-effort systemd/OpenRC hook installation and boot-ID sentinel validation; no actionable defect was established for supported images.
guest/arcbox-agent/src/agent/linux/system_info.rs	Reports distro initialization as pending only when a hook exists and the current boot’s sentinel is absent.
guest/arcbox-agent/src/init.rs	Installs the boot-completion hook after guest network initialization and before handoff to distro init.
app/arcbox-core/src/machine.rs	Gates both blocking and asynchronous readiness probes on distro initialization completion before selecting an IP.
rpc/arcbox-protocol/proto/agent.proto	Adds a backward-compatible pending-state field whose default preserves behavior with older agents.
app/arcbox-core/src/machine/tests.rs	Covers pending, settled, legacy-agent default, and missing-routable-address readiness states.

Sequence Diagram

sequenceDiagram
    participant Host as Host MachineManager
    participant Agent as Guest Agent
    participant Init as Distro Init
    participant Hook as Boot-done Hook

    Agent->>Agent: machine-init installs hook
    Agent-->>Host: Agent begins answering RPCs
    Agent->>Init: Boot shim execs /sbin/init
    loop Readiness polling
        Host->>Agent: GetSystemInfo
        Agent-->>Host: "distro_init_pending=true"
    end
    Init->>Hook: Run after boot target/runlevel
    Hook->>Hook: Write current boot ID sentinel
    Host->>Agent: GetSystemInfo
    Agent-->>Host: "pending=false and routable IP"
    Host-->>Host: Report Machine ready

Loading

_{Reviews (1): Last reviewed commit: "chore: follow the Connect migration for ..." | Re-trigger Greptile}

[pullfrog]

Important

The handshake itself is sound, but Start now fail-closes on the distro's whole boot inside the same 60 s budget that used to cover only agent-up — and the flake this closes has no regression guard in the tree.

Reviewed changes

• guest/arcbox-agent/src/boot_done.rs (new): installs a guest-side boot-completion hook — a systemd unit After=/WantedBy=multi-user.target or an openrc service with depend() { after * } — that writes the current kernel boot_id into /run/arcbox-boot-done. Enabled by hand-created symlink because neither init is running at machine-init time. Best-effort with rollback on partial failure.
• guest/arcbox-agent/src/init.rs: calls boot_done::install() at the end of machine_init.
• guest/arcbox-agent/src/agent/linux/system_info.rs: reports distro_init_pending = hook_installed() && !boot_complete().
• rpc/arcbox-protocol/proto/agent.proto + regenerated prost: additive field 12 bool distro_init_pending, phrased pending so proto3's false default preserves pre-CORE-66 behaviour for older agents.
• app/arcbox-core/src/machine.rs: new readiness_ip() wraps select_routable_ip and returns None while the guest reports the init pending; both the blocking and async probes route through it, reusing the existing 50→500 ms backoff.
• app/arcbox-core/src/machine/tests.rs: 4 tests over readiness_ip, including the old-agent default and the settled-but-no-address case.
• app/AGENTS.md: records the readiness contract and the explicit warning against re-deriving it from init runtime state.

⚠️ A hook that installs but never fires turns a working Start into a hard 60-second failure, and the error blames the agent

The gate is one-directional: once hook_installed() is true, the machine can never be ready until the sentinel appears. There is no path back. So any image where the hook lands but the sentinel is never written — a stripped systemd image whose default.target doesn't pull in multi-user.target, or an openrc image whose active runlevel isn't default — goes from "starts in ~0.5 s" to "fails after 60 s". That is a regression in reachability, not just latency, and the failure text misattributes it.

# Fail-closed readiness on unverified distro streams

## Affected sites

- `app/arcbox-core/src/machine.rs:649` — `PROBE_TIMEOUT` is 60 s, and it is now *shared* between "agent answers" and "distro finishes booting". Previously the whole budget covered the first alone.
- `app/arcbox-core/src/machine.rs:679` — on exhaustion the error reads `Machine '{name}' agent did not report a routable IP within timeout`. When the real cause is a hook that will never fire, this points the operator at the agent, which is up and answering correctly.
- `app/arcbox-core/src/machine_image.rs:24,343` — the selectable stream set is fetched at runtime from `https://image.arcboxcdn.com/linux` `index.json`. It can grow without any change in this repo, so "we verified the streams we ship" is not a closed set. The PR description verifies alpine 3.24 and ubuntu noble only.
- `guest/arcbox-agent/src/boot_done.rs:123-133` — `install()` probes for the systemd *binary* and, if present, installs the systemd hook. A binary-present-but-not-PID-1 image installs a hook nothing will ever run.

## Required outcome

`Start` should not become less likely to succeed than it was before this PR on a stream the project offers for selection. Either the wait degrades to a warning after its own sub-budget, or the timeout error distinguishes "agent never answered" from "agent answered, distro init never completed" so the failure is diagnosable from the message alone.

## Suggested approach

Give the distro-init wait its own deadline nested inside `PROBE_TIMEOUT`, and on expiry log a warn naming the sentinel and the detected init, then fall through to `select_routable_ip`. That keeps the fix's benefit for the images it was measured on (the gate cleared in 2.2 s on alpine and 13.8 s on ubuntu, both far inside any sub-budget) while capping the downside on an unverified one.

## Open questions for the human

The PR description asks for an explicit decision on the ~14 s ubuntu latency. The latency itself seems defensible — a Machine that isn't network-usable isn't started. Two things about it are not settled by that answer:

1. During those 14 s the client gets no progress signal at all. `abctl machine start` issues `.start()` with no deadline, so it simply sits there; there is nothing to distinguish a slow systemd boot from a wedged one.
2. Fail-closed vs. degrade-with-warning is the sharper half of the same decision, and the description doesn't address it.

⚠️ The flake this PR closes has no regression guard in the tree

The validation table cites a machine_network e2e going from 4/8 baseline failures to 0/8. That target does not exist anywhere in the repository — tests/e2e/tests/ contains machine.rs and machine_pty.rs, and neither asserts that the network is usable after Start returns. The 4 new unit tests cover readiness_ip's branching (and they do genuinely fail if the gate is removed — I checked), but nothing exercises the guest hook or the post-Start window that CORE-66 is about. Remove readiness_ip's early return and the unit tests catch it; regress the hook installation, the sentinel path, or the ordering and nothing does.

# No end-to-end coverage for the window CORE-66 closes

## Affected sites

- `tests/e2e/tests/` — only `machine.rs` and `machine_pty.rs` exist. `machine_network` is referenced in the PR description but is not in the repo.
- `tests/e2e/tests/machine.rs` — `START_BUDGET` is 120 s and `READY_TIMEOUT` 180 s, so the ~14 s systemd wait fits comfortably; the budgets are not the obstacle to adding this.

## Required outcome

Either the cited test lands with this change, or the description drops the claim so the numbers aren't read as in-tree coverage. A test that issues a network operation immediately after `Start` returns and asserts it does not hit `ENETUNREACH` is the thing that keeps this fixed.

ℹ️ Nitpicks

• boot_done.rs:42 — SENTINEL is pub but has no consumer outside the module.
• boot_done.rs:123 — install() returns bool, and its sole call site (init.rs:504) discards it. Either the caller should act on it or the signature can be (); the guest-side reader keys off hook_installed(), not this value.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

SENTINEL on line 42 is authoritative for the reader (boot_complete()), but both hook bodies hardcode /run/arcbox-boot-done as a literal — here and again on line 76. Editing SENTINEL alone leaves the hooks writing the old path, and nothing in the build or the tests notices.

The symptom of that divergence is precisely the 60-second readiness timeout this module's own docs warn about: the hook installs, the hook runs, the sentinel is written, and the reader looks somewhere else forever.

# Sentinel path is duplicated across three sites

## Affected sites

- `guest/arcbox-agent/src/boot_done.rs:42` — `SENTINEL`, read by `boot_complete()`.
- `guest/arcbox-agent/src/boot_done.rs:60` — `SYSTEMD_UNIT_BODY` `ExecStart`, literal path.
- `guest/arcbox-agent/src/boot_done.rs:76` — `OPENRC_SERVICE_BODY` `start()`, literal path.

## Required outcome

Changing the sentinel path in one place should either update every writer or fail loudly. A silent divergence must not be possible.

## Suggested approach

The hook bodies are `&'static str`, so they can't interpolate a const directly — but a unit test alongside the existing three closes the gap for free:

```rust
#[test]
fn both_hook_bodies_write_the_sentinel_the_reader_watches() {
    assert!(SYSTEMD_UNIT_BODY.contains(SENTINEL));
    assert!(OPENRC_SERVICE_BODY.contains(SENTINEL));
}
```

Same for `BOOT_ID`, which is duplicated the same way.

[pullfrog]

This overstates what systemd guarantees. After=multi-user.target orders the unit against the target unit's own activation, not against the completion of every service pulled into that target. A service reached via a .wants symlink gets no implicit Before= on the target it is wanted by, so ordering between it and the target's other members is unconstrained — the hook can in principle run concurrently with services still coming up.

The measured ubuntu behaviour (26 gate hits, ready at 13.8 s) says the ordering is strong enough on the images tested, and I'm not claiming a bug. But this module's entire premise is that the init's ordering contract is the dependable part — the alternative (polling init runtime state) was tried and rejected for exactly that reason — so the comment shouldn't claim more than the contract gives.

# Comment claims a stronger systemd ordering guarantee than `After=` provides

## Affected sites

- `guest/arcbox-agent/src/boot_done.rs:50-51` — "which is the documented way to run something once the boot transaction has settled."

## Required outcome

The comment should describe what the unit actually orders against, so a future reader debugging an early-firing sentinel isn't misled into trusting a guarantee that was never there.

## Open questions for the human

Lima anchors its equivalent on `network-online.target` (`Wants=` + `After=`) rather than `multi-user.target`, which is a closer match to what the readiness gate actually cares about — the interface being configured. It carries its own trade-off: on an image with no `wait-online` provider the target may never activate, which is the failure mode you're already exposed to. Worth a note either way; the current choice may well be the right one for the mirrored streams.