Skip to content

feat(setup): install and start Pilot behind an HTTPS egress proxy (Meta Muse) - #23

Merged
TeoSlayer merged 8 commits into
mainfrom
feat/proxy-aware-install
Sep 24, 2026
Merged

TeoSlayer merged 8 commits into
mainfrom
feat/proxy-aware-install

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

npx pilotprotocol-mcp setup does not work in proxy-only agent sandboxes such as Meta Muse. In Muse:

  • the agent runs as root, with no systemd;
  • outbound traffic is allowed only through an authenticating HTTPS_PROXY (CONNECT host:443 only);
  • UDP is blocked;
  • DNS for the Pilot hostnames is poisoned;
  • traffic that bypasses the proxy is killed;
  • the proxy credentials in HTTPS_PROXY rotate every few minutes. A process keeps the credentials it started with, so once they rotate its new CONNECTs fail with 407 ("node online, all apps broken").

Three problems follow:

  • Node's built-in fetch ignores the proxy environment, so the runtime download fails.
  • The UDP probe sent a payload the beacon never answers, so every network was reported as UDP-blocked.
  • Setup had no way to start the daemon through a proxy.
  • Neither the downloads nor the daemon picked up rotated credentials.

What changed

Proxy-aware downloads.

  • The manifest and runtime archive now tunnel through the proxy with CONNECT by hostname. TLS stays end to end, and the SHA-256 check is unchanged.
  • src/netproxy.js ports proxy selection from common/netproxy v0.5.14, which pilot-daemon -proxy also uses:
    • order: HTTPS_PROXY > https_proxy > ALL_PROXY > all_proxy;
    • an unusable ALL_PROXY is skipped, while an unusable HTTPS_PROXY means no proxy;
    • NO_PROXY uses Go's semantics;
    • loopback is never proxied;
    • userinfo runs to the last @.
  • common's resolver test vectors run here as a parity table.

Daemon start behind a proxy. planDaemonStart resolves the proxy the way the daemon does. What setup does depends on the runtime:

Runtime Setup behaviour
Has -transport=auto Passes no --transport and records nothing, so the daemon picks.
Has -proxy but not auto Starts with pilotctl daemon start --transport compat and records "transport":"compat" plus "transport_set_by":"pilot-mcp". A later run removes that record when UDP works or no proxy is set.
Older (v1.13.10 and earlier have no -proxy) Keeps the runtime and prints the pilot-sandbox pointer. In a proxy-only sandbox (Linux without systemd, credentials in HTTPS_PROXY) it does not start that daemon at all (it would dial around the proxy) and skips its own UDP probe; the summary says the node cannot reach the Pilot network and gives the way forward; exit 1. Elsewhere it starts it: if it registers but fails the trust check (UDP blocked, TCP allowed), the summary prints the compat-mode switch that needs no proxy (exit 0); if it never comes up, the summary says the node cannot reach the network, a daemon setup's own start created is stopped, and setup exits 1.

On every runtime:

  • Setup never passes --proxy, so a config.json proxy or $PILOT_PROXY still wins.
  • A transport the user set is never overwritten.

Rotating proxy credentials. This follows the proxy-command convention of common v0.5.15 and pilot-daemon -proxy-cmd: the command runs with sh -c for at most 10s, its output must be one http(s):// URL, and the output is never logged. The command comes from:

  1. PILOT_PROXY_CMD;
  2. otherwise, config.json "proxy_cmd";
  3. otherwise, in a Linux container or VM without systemd whose HTTPS_PROXY/https_proxy carries credentials, the sandbox default bash -c 'case $https_proxy in *@*) printf %s "$https_proxy";; *) printf %s "${HTTPS_PROXY:-$https_proxy}";; esac', byte-identical to pilotctl's sandboxProxyCmd and release#49's SANDBOX_PROXY_CMD (a saved proxy_cmd with this or the earlier pre-release command is recognised as the sandbox default).
  • Downloads run the command before every request and every redirect hop. On a 407, or a CONNECT reply that does not parse, they run it once more and retry once. If a run fails, the last good URL stays in use (at first, the environment's proxy).
  • Daemon start depends on the runtime:
Daemon Setup behaviour
Has -proxy-cmd Gets the sandbox default as PILOT_PROXY_CMD for this start only. It is not saved: the pilotctl that ships with -proxy-cmd derives it on every start, and only while the proxy comes from HTTPS_PROXY. A configured command is left for the daemon to read.
Has -proxy but not -proxy-cmd HTTPS_PROXY is pointed at the pilot-sandbox egress_relay.py on 127.0.0.1:3128. Setup looks for it in ~/workspace/skills and the other skill folders, or at PILOT_EGRESS_RELAY. It is started with python3 unless an instance is already running. A port held by another program is never used.
Neither works Setup warns that the credentials will go stale and points at the relay.
No -proxy (v1.13.10 and earlier) Unchanged: the pilot-sandbox pointer.

Setup never makes its own proxy command replace an explicit PILOT_PROXY or config.json "proxy" URL. A command the user configured does replace it, because pilot-daemon -proxy-cmd supplies the URL -proxy would use. Setup and doctor report that case as a replacement instead of naming the explicit proxy. A config.json "proxy_cmd" that equals the sandbox default (install.sh saves it) is a special case: setup's downloads use it only with the proxy environment, and setup and doctor warn when pilot-daemon would use it over an explicit proxy, naming pilotctl config --set proxy_cmd=.

The setup summary adds proxy_refresh (proxy-cmd, relay or stale) and proxy_replaced_by. doctor --json adds network.proxy_cmd {source, daemon_support, replaces?}. The command itself is never printed.

"transport": "auto". Re-verified at this head against a daemon built from pilotprotocol 1dd9e8e7. auto in config.json or PILOT_TRANSPORT is accepted in any case and kept. It is never called invalid and never overwritten with compat, and setup passes no --transport, so the daemon picks. The reviewer's repro-auto.mjs cases A through E and the doctor case are all clean.

Settings validation.

  • PILOT_PROXY accepts auto, off and an http(s):// URL. off has the aliases none, no, false and direct, and every alias reaches the daemon as off.
  • An unusable PILOT_PROXY is ignored with a warning and not passed on.
  • PILOT_TRANSPORT is lower-cased, and it is withheld from runtimes that would refuse it.

Safe runtime upgrades.

  • An unmanaged per-user runtime is upgraded only to a strictly newer stable release.
  • The staged daemon must list -proxy before it is swapped in.
  • Managed nodes keep their pinned runtime. A node counts as managed if it has a managed-runtime-* tag, enterprise-control.json, enterprise_control in config.json, or PILOT_ENTERPRISE_CONTROL.
  • A runtime of unknown version is left alone.

Probe and reporting.

  • The UDP probe now sends the real discover message, 3 times over 2s.
  • doctor --json adds:
    • network.proxy (redacted);
    • network.mode / network.setting;
    • network.warnings;
    • network.transport {value, set_by};
    • network.proxy_cmd {source, daemon_support};
    • daemon_proxy_support.
  • The README and CHANGELOG describe the proxy and sandbox path. Root without systemd is supported.

Compatibility

  • No proxy in the environment: behavior is unchanged apart from the corrected UDP probe.
  • Hosts that set HTTPS_PROXY but relied on direct downloads can set PILOT_PROXY=off.
  • Runtime tiers:
  • Publishing: npm publishing is tag-triggered, so merging does not publish.

Security

  • Proxy credentials are redacted (http://***@host:port) in every setup, doctor and warning message.
  • Credentials never reach pilotctl or daemon argv. They travel only through the environment, and only when the daemon needs them.
  • Tests assert that no credential appears in any warning.
  • E2E below: 0 credentials in output or daemon argv.

Tests

  • npm run lint: clean.
  • npm test (Node v25.2.1): 152/152 pass at head a45d981. The earlier rounds added 15; review round 4 adds 4 and rewrites 1. They cover:
    • a proxy whose credentials rotate after every CONNECT, across requests and redirect hops, with a stale-credentials control;
    • 407 and malformed-reply refresh with exactly one retry;
    • fallback to the last good URL when the command fails;
    • sh -c semantics: the timeout kills the process group, output is capped at 64 KiB, and no output appears in errors;
    • proxyCommandFor precedence and the sandbox-default conditions;
    • the real runtime install through the TLS mirror with rotating credentials;
    • each daemon tier (-proxy-cmd, relay, stale, legacy);
    • relay discovery and start/reuse/port-conflict;
    • doctor output.
  • A mutation check confirms the new tests catch these regressions: dropping the env override, dropping the per-request refresh, and dropping the retry each fail the tests.
  • test/netproxy.test.js has 11 common v0.5.14 parity tests.
  • test/proxy-daemon-start.test.js covers:
    • all three runtime tiers;
    • managed-node guards, which make zero network requests;
    • the no-downgrade rule;
    • a real CONNECT proxy with a TLS mirror.
  • A matrix against the real daemons (a3ed59f4, proxy-without-auto, v1.13.9) passed all 12 cases.
  • Earlier, in the Muse sim as root with no systemd (auth proxy, poisoned DNS, NET_RAW dropped), node cli.js setup downloaded through the proxy, kept v1.13.9, printed the pilot-sandbox pointer, exited 0 and printed no credentials.

End-to-end: scenario 7, npm path through the proxy (passed)

This ran on the Mac with a local authenticating CONNECT proxy. The Muse sim was down because Docker hung.

  • Install: npm pack of e849003, then npm install through HTTPS_PROXY=http://muse:***@127.0.0.1:3129 with a clean cache. It took 6.8s and made 15 ALLOW registry.npmjs.org:443 requests through the proxy.
  • 7a, v1.13.9 runtime: ran with env -i, HOME=<tmp> and PILOT_TRANSPORT=compat.
    • pilot-mcp setup --all finished with rc=0 in 14.2s.
    • The runtime download went through the proxy (pilotprotocol.network, github.com, release-assets, raw.githubusercontent.com) and v1.13.9 was verified.
    • Setup printed "pilot-daemon predates egress-proxy support … Keeping the installed runtime" and the pilot-sandbox pointer.
  • 7b, branch daemon binaries in ~/.pilot/bin:
    • setup printed Egress proxy http://***@127.0.0.1:3129 (HTTPS_PROXY): starting pilot-daemon with -transport=compat through it;
    • the daemon argv had --transport compat and no credentials;
    • the daemon registered, and every daemon socket went to the proxy (checked with lsof);
    • doctor --json returned ok with the proxy redacted;
    • the list-agents query succeeded in 3.2s.

End-to-end: rotating credentials (phase 3, on the Mac; Docker is still hung)

These runs used a local Muse-like proxy (CONNECT and :443 only, Basic auth) whose credentials rotate and whose old credentials are refused at once. BASH_ENV exports the current URL, so a fresh bash -c sees rotated credentials the way it does in Muse. Each run used a throwaway HOME and socket, never ~/.pilot or /tmp/pilot.sock.

  • A, downloads: credentials rotate after every CONNECT, with HTTPS_PROXY stale from the start. ensurePilotRuntime downloaded and verified the real v1.13.9 from pilotprotocol.network → github.com → release-assets, one fresh generation per CONNECT (gen 0, 1, 2). The control run without the command failed with refused CONNECT github.com:443: HTTP 407.
  • B, -proxy-cmd daemon (1dd9e8e7): credentials rotate every 45s. installDaemon with PILOT_TRANSPORT=compat passed PILOT_PROXY_CMD and saved "proxy_cmd". The daemon logged proxy=... (refreshed from the proxy command) and registered (node 251972), and trust was verified.
    • Over 8 rotations, the daemon handled 5 proxy 407s with proxy rejected the credentials; retrying with refreshed ones, each followed by an ALLOW with the new generation. No 407 went unresolved.
    • info and a list-agents query succeeded at t+55, 109, 157, 206 and 253s.
    • The query at t+330s failed with connection_failed ... 0:0000.0002.BBE4. This is the same peer-level error phase 2 also saw on released v1.13.9 in UDP mode. The proxy log shows no rejected CONNECT at that point.
  • C, relay tier (a3ed59f4 daemon: -proxy, no -proxy-cmd): setup started the pilot-sandbox egress_relay.py (a copy moved to port 3139 so it would not disturb other sessions) and pointed HTTPS_PROXY at it. The relay log shows 407 followed by 200 Connection Established after rotations. Results over 8 rotations (gen 9 to 16):
    • the relay turned 6 upstream 407s into a successful retry and made 33 200 CONNECTs;
    • the daemon registered (node 251973);
    • list-agents succeeded at t+130, 181 and 311s;
    • the first trust check and the t+262s query failed with connection_failed / no reply ... within 30s from 0:0000.0002.BBE4, and the daemon log shows rekey desync with a peer. The next query, after further rotations, succeeded, so the credential path held.
  • The intermittent list-agents failures in B and C are the same peer-level flake phase 2 recorded on released v1.13.9 over UDP. In neither run did the proxy show a rejected CONNECT that stayed rejected.

Not covered, and cosmetic:

  • The run was not inside the Linux Muse sim, because Docker is down. Round 4 exercised the "UDP blocked, the v1.13.9 daemon cannot get out" path on the Mac instead (see below). Setup now reports the node as unreachable, stops the daemon it started, and exits 1.
  • The summary shows Address: (fetching…), or (none: not on the network) when unreachable. It shows Hostname: null when PILOT_HOSTNAME is unset.
  • 7a: when the daemon lacks -proxy but reached the network directly, the summary now says Proxy: … (not used: pilot-daemon has no -proxy and reached the network directly).
  • In auto mode the daemon's transport check can take up to about 5s. The 10s trust wait can then report "TRUST NOT YET VERIFIED" on a slow proxy; doctor clears it.
  • The relay tier applies only to unreleased builds that have -proxy but not -proxy-cmd. The first release with -proxy also has -proxy-cmd. The relay listens on the fixed port 3128. It is not persisted: after a VM restart it has to be started before the daemon, and setup says so.

Review round 4 (phase-3 findings)

mcp-p3-released-runtime-onboarding-says-resolves (medium), fixed.

  • installDaemon now handles the proxy-unsupported plan when the trust check fails:
    • it returns network_unreachable with a hint pointing at pilot-sandbox or a runtime with -proxy;
    • it stops a daemon that setup itself started (daemon status --check ran before the start) and that never came up (pilotctl daemon start failed);
    • if the stop fails, it says so rather than claiming success;
    • it leaves a pre-existing daemon alone;
    • it leaves a daemon alone that reached the network directly.
  • runSetup then prints THIS NODE CANNOT REACH THE PILOT NETWORK with the reason and the way forward, lists pilot-sandbox/doctor as next steps instead of tour/peers, and returns { ok: false }; cli.js exits 1.
  • README and CHANGELOG now make Muse support conditional on a runtime with -proxy.
  • End to end on the Mac with the reviewer's harness (sandbox-exec proxy-only profile, authenticating CONNECT proxy, throwaway HOME and socket, real v1.13.9 downloaded through the proxy):
    • The headline, the pilot-sandbox pointer and exit 1 all appear, with no "resolves within 60s".
    • Under sandbox-exec, v1.13.9's daemon stop cannot run the setuid ps to identify the process. It reports not running, and setup prints Could not stop the pilot-daemon setup started (...). This is an artifact of the Mac harness: on Linux, pilotctl reads /proc/<pid>/cmdline. The v1.13.9 daemon also gives up by itself after 10 registry attempts, about 50s.
    • The same run outside sandbox-exec, with registry and beacon pointed at dead local ports, printed Stopped the pilot-daemon setup started and left no daemon running.

mcp-p3-saved-sandbox-proxy-cmd-overrides-explicit-proxy (low), fixed.

  • The sandbox default is no longer saved as config.json "proxy_cmd".
  • A configured command that replaces an explicit proxy is now reported as such, and a saved sandbox default gets a warning (see above).
  • End to end with the current pilotprotocol#470 head (d80ac75e, built with GOWORK=off: real pilotctl and pilot-daemon), proxy A (credentialed, the sandbox's HTTPS_PROXY) and proxy B (explicit). All three daemons registered (node 251984).
    • Stage 1, sandbox with HTTPS_PROXY=A: PILOT_PROXY_CMD was handed over and config.json has no proxy_cmd. The daemon logged proxy="auto: http://***@…:41293 (credentials refreshed by command)", and every CONNECT went to A.
    • Stage 2, PILOT_PROXY=B added (A still in the environment): the daemon logged proxy=http://127.0.0.1:41294. Every daemon CONNECT (registry ×4, beacon, raw.githubusercontent) and setup's manifest download went to B, and A got none. Setup and doctor both said http://127.0.0.1:41294 (PILOT_PROXY).
    • Stage 3, proxy_cmd set to the sandbox default as install.sh saves it, PILOT_PROXY=B:
      • setup's step-4 line was Egress proxy from the proxy command in config.json proxy_cmd (in place of http://127.0.0.1:41294 from PILOT_PROXY), followed by the warning and the removal command;
      • the result had proxy_replaced_by: "config.json proxy_cmd";
      • the daemon went through A, as reported;
      • the manifest download went through B;
      • doctor printed the replacement and the warning;
      • trust was verified.
    • No proxy password appeared in any output, config.json or daemon log. All daemons and proxies were stopped afterwards.
  • Remaining work outside this repo: install.sh in installer: transport auto/compat, HTTPS-proxy sandboxes, root without systemd release#49 still saves the sandbox default as "proxy_cmd", so a later explicit proxy on such a host is still replaced in pilot-daemon. pilot-mcp now warns about it. The root fix belongs in #49 (stop saving it, since pilotctl derives it) or in #470 (treat a saved default like the derived one).
  • A mutation check confirms the new tests catch each of these regressions: persisting the default again, dropping the unreachable flag, never stopping the daemon, stopping a daemon that was already running, using the saved default over an explicit PILOT_PROXY for downloads, dropping the replacement report, and cli.js ignoring the result.

Review round 5 (this push: e430e4a, 3d1c760)

  • mcp-p3r2-unreachable-verdict-when-daemon-registered (medium), fixed. When setup's own start of a daemon without -proxy succeeds (pilotctl blocks until registration) but the trust check fails, the host let it out directly and UDP is what is blocked. Setup now returns compat_hint and prints "the daemon registered, but it cannot reach peers yet" with the switch that works there without the proxy (pilotctl config --set transport=compat, plus registry=registry.pilotprotocol.network:443 for runtimes whose pilotctl hands config.json's registry over, then restart). No "CANNOT REACH", no pilot-sandbox pointer, exit 0.
  • mcp-p3r2-stops-daemon-it-did-not-start (low), fixed. Presence before the start also counts a live pid in ~/.pilot/pilot.pid (a daemon still registering answers no IPC yet) and install.sh's launchd agent (stopping it would unload the agent for good); setup stops only when its own start created the pid file. This also fixes the start-result check: an uncaptured execPilotctl resolves the bare exit code, so started.code had always been undefined.
  • R49-A3 (release#49 review), fixed. SANDBOX_PROXY_CMD is the credential-preferring command pilotctl and release#49 use and save, so isSavedSandboxDefault recognises install.sh's proxy_cmd again (the earlier pre-release command too); with an explicit PILOT_PROXY, setup's downloads use it, not the saved sandbox default.
  • E2E scenario 7 bypass traffic, fixed. In a proxy-only sandbox a daemon without -proxy is not started (it sent 10 SYNs to 34.71.57.205:9000 and 4 UDP datagrams around the proxy before setup stopped it), and setup's own 3-datagram UDP probe is skipped there. A node already up (the pilot-sandbox recipe's) is used as it is.
  • Docs: "v1.13.10 and earlier lack -proxy" (it said "v1.13.9 and v1.13.10-rc.1"); summary prints Hostname: (not set) and Transport: (daemon not started) where they apply.

Tests: npm test 155/155 on macOS (Node 25), npm run lint clean; in a node:22 Linux container without systemd 146 pass / 7 skipped (openssl, python3, darwin-only) / 0 fail. New: the registered-directly compat hint, an alive-but-unanswering outside daemon (never stopped, no "setup started" claim), the proxy-only sandbox (no start, no stop, daemon_not_started; a running node used as is; a proxy without credentials is not taken as proxy-only), the launchd agent never stopped (macOS), proxyOnlySandbox, the sandbox command byte-for-byte plus the legacy one. The fake pilotctl now keeps a pid file like the real one.

E2E, round 3 Muse sim (rotating-credential CONNECT-443-only proxy, password rotates every 60s; root, no systemd, poisoned DNS; packet sink logging every direct dial). Node v22 and npm install -g of this branch's npm pack through the proxy; pilot-mcp setup started with docker exec and node directly (stale HTTPS_PROXY):

  • Released v1.13.10 runtime: Proxy-only sandbox … UDP probe skipped, Keeping the installed runtime: the latest stable release v1.13.10 is not newer, Not starting it: this host … goes out only through the proxy; summary "THIS NODE CANNOT REACH THE PILOT NETWORK … setup did not start it", pilot-sandbox pointer, rc=1. The sink logged 0 packets from the VM (phase 3: 10 SYNs + 4 UDP + 3 probe datagrams). Every download CONNECT used fresh credentials.
  • Next-release runtime (pilotprotocol#470 30c034bb binaries in ~/.pilot/bin; re-run after Docker Desktop recovered from a host disk-full that interrupted the first attempt): setup's UDP probe skipped; -transport=auto picked compat through the proxy; PILOT_PROXY_CMD in the daemon's environment (credentials refreshed by command); registered in 8s (node 252148, deregistered after); trust with list-agents verified, pilot-mcp summary "installed and running", rc=0; pilot-mom query ok. The sink logged exactly 2 packets from the VM: the daemon's own -transport=auto UDP probe to 34.71.57.205:9001 (setup's 3-datagram probe is gone).

Release

  • How publishing works: npm publishing runs only from .github/workflows/publish.yml, on a pushed v* tag or a manual dispatch for an existing tag. The workflow first checks that the tag equals v + package.json version, then runs lint and tests. It then publishes to npm with OIDC (no token, and it is a no-op if that version is already published), the Official MCP Registry (server.json) and ghcr.io/pilot-protocol/pilot-mcp. Merging this PR publishes nothing.
  • Next version: 0.3.0. The last published version is 0.2.13. This is a behaviour change: downloads honour HTTPS_PROXY, setup may write "transport" (never "proxy_cmd"), and setup exits 1 when the node cannot reach the Pilot network.
  • Timing against the runtime release: the README and CHANGELOG now make Muse support conditional on a runtime whose pilot-daemon has -proxy, so tagging before pilotprotocol#470 ships does not publish a false claim. Until #470 is latest_stable, though, a Muse onboarding through pilot-mcp ends with the "cannot reach the Pilot network" summary and the pilot-sandbox pointer, not an online node. For a clean Muse onboarding, release #470 as latest_stable first, then tag v0.3.0. Bump the version in package.json and package-lock.json, server.json (both fields), .well-known/mcp/server-card.json, src/version.js, src/openclaw-plugin/openclaw.plugin.json and src/openclaw-plugin/evaluate.js (test/release-contract.test.js keeps them locked). Also update the pinned pilotprotocol-mcp@0.2.13 strings in test/{hermes-setup,openclaw-plugin,native-harness-setup,harness-config-contracts}.test.js. Then move [Unreleased] to [0.3.0] and tag v0.3.0.

Merge order and dependencies

Deploy order (all six PRs)

  1. installer: transport auto/compat, HTTPS-proxy sandboxes, root without systemd release#49 merge → notify-install-sh-sync dispatches to pilot-protocol/website → sync-install-sh.yml puts install.sh into R2 pilot-release-assets/install.sh (the pilot-release Worker reads R2 per request; no Worker deploy). Check: curl -fsSL https://pilotprotocol.network/install.sh | shasum -a 256 → 4384f0cb2890177cb87a2e2bc49e1b814d2850434d24c54d1730c9955fbbb671 (up to 5 min cache).
  2. install.sh: sync the Pages fallback with pilot-protocol/release#49 website#263 merge (Pages fallback public/install.sh, same bytes) → the sync workflow's drift step goes green.
  3. feat: native HTTPS-proxy support and -transport=auto (Meta Muse / proxy-only sandboxes) pilotprotocol#470: re-run install.sh matches pilot-protocol/release, merge.
  4. Tag pilotprotocol v1.13.11 on main (release workflow: binaries, checksums.txt, signed latest.json). v1.13.10 shipped without #470, so until this tag every install gets a daemon without -proxy. Check pilot-daemon -h lists -proxy and -proxy-cmd.
  5. feat(muse): one-shot Pilot-in-Muse installer and pilot-up.sh TeoSlayer/pilot-skills#34 merge → CI regenerates skills.json/setups.json and skills/pilotctl/SKILL.md, dispatches the website deploy; the Muse one-shot (raw.githubusercontent.com/TeoSlayer/pilot-skills/main/muse/install.sh) is live at merge.
  6. feat(setup): install and start Pilot behind an HTTPS egress proxy (Meta Muse) #23 merge → release 0.3.0 (behaviour change; last published 0.2.13): bump package.json, package-lock.json, server.json (both fields), .well-known/mcp/server-card.json, src/version.js, src/openclaw-plugin/openclaw.plugin.json, src/openclaw-plugin/evaluate.js and the pinned pilotprotocol-mcp@0.2.13 strings in test/{hermes-setup,openclaw-plugin,native-harness-setup,harness-config-contracts}.test.js (test/release-contract.test.js locks them), move [Unreleased] to [0.3.0], push tag v0.3.0 → publish.yml (tag must equal v+version; lint + tests; npm via OIDC, MCP Registry server.json, ghcr.io/pilot-protocol/pilot-mcp).
  7. Add marker-gated skill targets (gatedTools) and the Meta Muse format skillinject#42 merge → tag v0.2.5 → pilotprotocol follow-up PR go get github.com/pilot-protocol/skillinject@v0.2.5 && go mod tidy (GOWORK=off) → next daemon release (v1.13.12; or fold the bump in before tagging v1.13.11).
  8. inject-manifest: gate Meta Muse under a new gatedTools key TeoSlayer/pilot-skills#35 merge (manifest gatedTools muse row). Released daemons drop the key (CI released-skillinject v0.2.2–v0.2.4). No manifest signing step: DefaultManifestPublicKeyHex is empty; only hosts that installed ~/.pilot/skillinject.pub verify inject-manifest.json.sig.

🤖 Generated with Claude Code

teovl and others added 4 commits September 24, 2026 00:09
Proxy-only agent sandboxes (Meta Muse) block UDP, poison DNS for the Pilot
hostnames and only allow CONNECT :443 through an authenticating
HTTPS_PROXY. Node's built-in fetch ignores HTTPS_PROXY, so `setup` could
not even download the runtime there.

- src/netproxy.js: zero-dependency proxy-aware fetch. Resolves
  HTTPS_PROXY/https_proxy (HTTP_PROXY for http:), then ALL_PROXY, honours
  NO_PROXY with Go httpproxy rules, and PILOT_PROXY=auto|off|URL. Tunnels
  with CONNECT by hostname (no local DNS), keeps TLS end-to-end with
  hostname verification, follows redirects per hop, redacts credentials.
- runtime.js: manifest and archive downloads use it; the SHA-256 gate is
  unchanged. requireProxy upgrades a ~/.pilot/bin runtime whose daemon
  lacks -proxy to the latest release (once; never brew/PATH/managed).
- transport.js: the UDP probe sends the beacon's real discover (node 0,
  as pilot-daemon does) and needs a 0x02 reply; the old "PILOT_PROBE"
  payload was never answered, so every network looked UDP-blocked.
  PILOT_TRANSPORT=udp|compat skips the probe.
- daemon.js: with a proxy and UDP blocked, a daemon that lists -proxy in
  `pilot-daemon -h` starts via `pilotctl daemon start --transport compat
  --proxy auto` with the proxy env forwarded, and "transport": "compat"
  is recorded in ~/.pilot/config.json (never the proxy URL). Older
  daemons keep today's start plus a pointer to the pilot-sandbox skill.
- doctor reports the egress proxy (redacted) and daemon -proxy support.
- Tests: Muse-like authenticating CONNECT-only :443 proxy routing by
  hostname, TLS end-to-end, full runtime download/upgrade through the
  proxy with checksum rejection, daemon start plans, UDP probe.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tion with common/netproxy

Proxy-driven runtime upgrade (review findings mcp-managed-runtime-swapped,
mcp-proxy-upgrade-downgrades):
- upgradeRuntimeForProxy replaces ~/.pilot/bin only when the node is not
  managed (managed-runtime-* tag, ~/.pilot/managed/enterprise-control.json,
  config.json enterprise_control, or PILOT_ENTERPRISE_CONTROL), the installed
  tag is a known release, latest_stable is strictly newer (semver), and the
  staged pilot-daemon lists -proxy before anything is swapped. Otherwise the
  runtime is kept, the reason is logged, and setup points at pilot-sandbox.
- installDaemon gates the upgrade on the same managed detection, not only on
  --managed-url for this run.

Transport persistence (mcp-compat-pinned-forever):
- "transport": "compat" is written only for a compat-through-proxy start,
  marked "transport_set_by": "pilot-mcp", and removed by a later setup that
  finds UDP working or no proxy. A transport the user or install.sh set is
  never changed. The UDP probe retries 3 times before concluding "blocked".
- doctor reports the recorded transport and who set it.

Proxy flag (mcp-proxy-flag-overrides-config):
- setup no longer passes --proxy auto (the daemon default); planning reads
  config.json "proxy" first, then PILOT_PROXY, then the environment, as the
  daemon does.

Resolution (mcp-js-go-resolution-divergence):
- First non-empty of HTTPS_PROXY/https_proxy/ALL_PROXY/all_proxy decides;
  an unusable value means no proxy plus a warning. HTTP_PROXY/http_proxy is
  best-effort for http:// only (ignored under REQUEST_METHOD). First non-empty
  of NO_PROXY/no_proxy, comma or space separated. Loopback is never proxied,
  even with an explicit proxy. PILOT_PROXY off|none|false|direct = off; a
  scheme-less PILOT_PROXY is http://. Resolution never throws: an invalid
  PILOT_PROXY is ignored with a warning, and planDaemonStart returns before
  reading proxy settings on UDP hosts. Userinfo with unescaped / ? # is
  rejected instead of becoming the proxy host.

Root/no-systemd: setup has no uid or service-manager check; verified as
root (HOME=/root, no systemd) in a Muse-like container behind an
authenticating CONNECT-only proxy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…; port common v0.5.14 resolution

Re-review findings on feat/proxy-aware-install:

- mcp-pilot-proxy-off-synonyms-break-daemon: PILOT_PROXY off aliases
  (off, none, no, false, direct; any case, as pilot-daemon's proxyconf
  accepts) now reach the daemon as "off", which every daemon build takes.
  Before, "none" was forwarded verbatim and the 3a89f62d daemon dialed
  everything through http://none. A config.json "proxy" pilotctl would
  refuse (an alias, a case variant, no scheme, another scheme) is reported
  on every transport with the `pilotctl config --set proxy=...` fix.
- mcp-invalid-pilot-proxy-kills-daemon: a PILOT_PROXY setup ignores (another
  scheme, a host:port without a scheme, a bare word) is handed to the daemon
  as '' (its auto default) instead of verbatim, so pilot-daemon neither exits
  with "-proxy: ..." nor takes a typo as the proxy host, on udp or compat.
  PILOT_TRANSPORT is passed lower-cased, or as '' when it is not udp/compat
  (pilotctl refuses anything else). The warnings say what setup does.
- mcp-netproxy-diverges-from-common-v0.5.14: src/netproxy.js ports common
  v0.5.14 parseProxyURL (optional scheme, userinfo up to the last '@',
  PathUnescape byte-exact, withheld errors) and fromEnv's per-variable
  semantics (unusable ALL_PROXY/all_proxy/HTTP_PROXY/http_proxy skipped,
  unusable HTTPS_PROXY/https_proxy means no proxy). NO_PROXY splits on
  exactly Go's separators. common's zz_resolver_test vectors run as a parity
  table in test/netproxy.test.js. Proxy settings (PILOT_PROXY, config.json)
  require an http(s):// scheme, as pilot-daemon and pilotctl do.
- mcp-pilot-transport-advice-noop-on-v1139: PILOT_TRANSPORT counts only for
  a runtime whose pilot-daemon lists -proxy (its pilotctl applies it);
  v1.13.9 ignores it, so setup warns and the summary reports the transport
  the daemon really runs. Compat without a proxy now prints commands that
  work on the installed runtime: `pilotctl config --set transport=compat`,
  plus `registry=registry.pilotprotocol.network:443` for pre--proxy
  runtimes whose pilotctl passes config.json's raw-TCP registry explicitly.

doctor reports network.mode and network.setting ("Egress proxy: off
(PILOT_PROXY)") and the refused settings. The test pilotctl shim now refuses
what pilot-daemon/pilotctl refuse, so a forwarded bad value fails the smoke
test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…overwrite a user transport

The unified web4 runtime (feat/native-https-proxy a3ed59f4) adds
-transport=auto: install.sh saves "transport": "auto" on fresh installs,
pilotctl accepts udp|compat|auto in any case and asks for auto when no
transport is set, and the daemon picks udp or compat (through the proxy)
on every start.

- Runtime features come from one `pilot-daemon -h` read (daemonFeatures):
  -proxy, and 'auto' named in the -transport usage (the check install.sh
  and pilotctl make).
- config.json "transport": "auto" and PILOT_TRANSPORT=auto (any case) are
  valid for such a runtime: no warning in setup or doctor, and
  PILOT_TRANSPORT=auto is passed on unchanged. Setup and doctor warn only
  when the installed daemon predates auto (and hold PILOT_TRANSPORT=auto
  back from a -proxy runtime whose pilotctl would refuse it).
- On an auto runtime setup passes no --transport, records nothing, and
  removes a "compat" an earlier setup recorded, so the node is not pinned
  to relay-only compat on later networks.
- syncRecordedTransport never overwrites a config.json "transport" without
  setup's marker, whatever its value ("auto", or one setup warns about).
- For an auto runtime PILOT_PROXY beats config.json "proxy", matching its
  pilotctl and daemon precedence (plan, summary and doctor).

Fixes re-review findings mcp-auto-transport-rejected and
mcp-overwrites-install-auto-with-compat.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
teovl and others added 4 commits September 24, 2026 05:00
… daemon

Meta Muse rotates the credentials in HTTPS_PROXY every few minutes; a
process keeps the ones it started with, and its new CONNECTs then fail
with 407 ("node online, all apps broken").

- Proxy command (the common v0.5.15 / pilot-daemon -proxy-cmd convention:
  sh -c, 10s, output never logged, http(s):// URL required):
  PILOT_PROXY_CMD, then config.json "proxy_cmd", then, in a Linux
  container or VM without systemd whose HTTPS_PROXY/https_proxy carries
  credentials, the sandbox default
  bash -c 'printf %s "${https_proxy:-$HTTPS_PROXY}"' that pilotctl and
  install.sh use.
- Downloads: the proxy command runs before every request and redirect
  hop, and once more on a 407 (or an unparseable CONNECT reply) with a
  single retry; a failed run keeps the last good URL (at first the
  environment's proxy). No background refresher: setup is short-lived.
- Daemon start: daemonFeatures reads -proxy-cmd. A daemon with it gets
  the sandbox default as PILOT_PROXY_CMD, also saved as "proxy_cmd" in
  config.json (never over an existing one), or reads the configured one.
  A daemon with -proxy but no -proxy-cmd has HTTPS_PROXY pointed at the
  pilot-sandbox egress_relay.py on 127.0.0.1:3128 (found in the skill
  folders or PILOT_EGRESS_RELAY, started with python3 unless an instance
  runs; a port held by another program is never used). Otherwise setup
  warns that the credentials will go stale and points at the relay.
  Explicit PILOT_PROXY / config.json "proxy" URLs are never replaced.
- Summary proxy_refresh (proxy-cmd | relay | stale); doctor
  network.proxy_cmd {source, daemon_support} and a warning when the
  daemon ignores a configured command. PILOT_PROXY_CMD joins the
  forwarded daemon environment.

"transport": "auto" handling from e849003 re-verified against a daemon
built from pilotprotocol 1dd9e8e7: kept, never called invalid, never
overwritten with compat, PILOT_TRANSPORT=auto passed on.

Tests: 148/148 (15 new), eslint clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e sandbox proxy_cmd

On a proxy-only host (UDP blocked, egress only through HTTPS_PROXY) every
released runtime lacks -proxy, so installDaemon ends in 'proxy-unsupported'
and the daemon can never reach the registry. Setup still printed "TRUST NOT
YET VERIFIED ... usually resolves on its own within 60s" and exited 0, so an
agent waited and re-ran doctor instead of switching to pilot-sandbox.

- installDaemon: in 'proxy-unsupported', when the trust check fails the
  result carries network_unreachable and a hint naming pilot-sandbox or a
  runtime with -proxy. A daemon setup itself started (daemon status --check
  before the start) that never came up (pilotctl start failed) is stopped
  again; a failed stop is reported, never claimed as done. A daemon that
  reached the network directly (trust verified) is left alone.
- runSetup: a "THIS NODE CANNOT REACH THE PILOT NETWORK" headline with the
  way forward, pilot-sandbox next steps instead of tour/peers, and
  { ok: false }, which cli.js turns into exit 1. When the daemon lacks -proxy
  but got out directly, the summary says the proxy is not used.
- README/CHANGELOG: Muse support is stated as conditional on a runtime whose
  pilot-daemon has -proxy (pilotprotocol#470; v1.13.9 and v1.13.10-rc.1 lack
  it), with what setup does until then.

The sandbox default proxy command is no longer saved as config.json
"proxy_cmd". Once saved it counted as configured everywhere: pilot-daemon's
-proxy-cmd supplied its URL in place of a PILOT_PROXY or config.json "proxy"
set later, pilotctl skipped its own guarded default, and setup's downloads
used it too, while setup and doctor still reported the explicit proxy.
Setup now hands it over as PILOT_PROXY_CMD for its own start only; the
pilotctl that comes with -proxy-cmd derives it on every start, and only
while the proxy comes from the environment.

- A configured command that replaces an explicit proxy is reported as such:
  the step-4 line, proxy_replaced_by in the result and summary, and
  network.proxy_cmd.replaces plus the Egress proxy line in doctor.
- A config.json "proxy_cmd" equal to the sandbox default (install.sh saves
  it) is treated as that default for setup's downloads (used only with the
  proxy environment), and setup and doctor warn when pilot-daemon would use
  it over an explicit proxy, naming `pilotctl config --set proxy_cmd=`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…verdict when it registered directly; stop only what setup started; sandbox proxy_cmd synced

Review round 2 (mcp-p3r2-*) and release#49 R49-A3, plus E2E scenario 7:

- A registered daemon is not "unreachable": when setup's own start of a
  daemon without -proxy succeeds (pilotctl blocks until registration) but the
  trust check fails, the host let it out directly and UDP is what is
  blocked. Setup prints the compat-mode switch that needs no proxy there
  (`transport=compat`, plus the TLS registry for runtimes whose pilotctl
  hands config.json's registry over) instead of THIS NODE CANNOT REACH THE
  PILOT NETWORK and the Linux-root-only pilot-sandbox skill, and exits 0.
- Setup stops only a daemon its own start created: presence before the start
  now also counts a live pid in ~/.pilot/pilot.pid (a daemon still
  registering answers no IPC yet) and install.sh's launchd agent (stopping
  it would unload the agent for good); the stop needs the pid file to have
  appeared. It used to stop an outside daemon whose start it had been
  refused and then say it stopped the one it started. Also fixes the start
  result check: an uncaptured execPilotctl resolves the bare exit code, so
  `started.code` was always undefined.
- Proxy-only sandbox (Linux without systemd, credentials in
  HTTPS_PROXY/https_proxy): a daemon without -proxy is not started at all
  (E2E s7: it sent 10 SYNs to 34.71.57.205:9000 and 4 UDP datagrams around
  the proxy before setup stopped it), and setup's own UDP probe is skipped.
  A node already up there (the pilot-sandbox recipe's) is used as it is.
- SANDBOX_PROXY_CMD is now byte-identical to pilotctl's sandboxProxyCmd and
  release#49's SANDBOX_PROXY_CMD (the credential-preferring case command);
  isSavedSandboxDefault also recognizes the earlier pre-release command.
- Docs: released runtimes up to and including v1.13.10 lack -proxy (not
  "v1.13.9 and v1.13.10-rc.1"); summary prints "Hostname: (not set)".

npm test 155/155 on macOS (Node 25); in a node:22 Linux container without
systemd 146 pass, 7 skipped (openssl/python3 absent, darwin-only), 0 fail.
npm run lint clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…port it never ran

E2E (Muse sim, released v1.13.10 runtime): with no daemon started, the
summary still printed 'Transport: udp'.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants