Skip to content

linux arm64/capsules - #338

Merged
ccozianu merged 20 commits into
ccozianu:mainfrom
powerofsoul:ws-linux-arm64/capsules
Nov 4, 2026
Merged

ccozianu merged 20 commits into
ccozianu:mainfrom
powerofsoul:ws-linux-arm64/capsules

Conversation

@powerofsoul

Copy link
Copy Markdown
Contributor

DevCapsule now runs on an Apple Silicon Mac. The launcher runs natively against Docker Desktop, and a project's capsules run either natively as linux-arm64 or as linux-amd64 under Docker Desktop's emulation. On my M-series Mac, VSCodium, VS Code, Eclipse, PyCharm, Rider and IntelliJ IDEA all start natively as ARM.

It is experimental and built from source. Nothing is published by this PR: no macOS executable, no ARM executable, no arm64 base image. Those are decisions for you; see the end.

It comes from two workstreams, the second built on the first and carrying it merged:

Trying it

The macOS page has the whole recipe: build three executables from one commit, build the arm64 base once, and then for each project:

devcapsule project init --ide codium --authorize base-image default
devcapsule project config authorize base-image devcapsule-base-local:arm64
devcapsule project run

On a Mac, init now asks which platform a new project is for. Enter takes linux-arm64, the native one. --platform answers without the question. A Linux host runs only one platform, so it is never asked, and nothing changes for Linux users.

What a Mac needed

Docker Desktop is not a Linux host, and the launcher assumed one. The differences are now properties of HostOs in platforms.py. Feature code asks "can I bind a Unix socket here?" instead of "am I on a Mac?" (D-0006 keeps that knowledge in one module):

  • The Docker socket shows up as root:root inside every container, so the capsule user joins group 0.
  • Host networking belongs to Docker Desktop's virtual machine, not to the Mac. A project authorized for network host launches on bridge networking, with a warning, and services on the Mac are reachable as host.docker.internal.
  • Unix sockets can't be bind-mounted, so the host-browser bridge is unavailable, with a warning. The contained desktop still opens in your browser.
  • Bind mounts refuse chown, even from root. So root writes the development-sudo policy into a small Docker volume. The session removes the volume when it ends, and a later launch sweeps up any that a killed launch left behind (only unused ones older than a day, so it never removes a concurrent launch's volume).
  • The capsule can't run a macOS executable, so the launcher can't deliver itself into the capsule as it does on Linux (D-0009). You build the Linux runtime once per capsule platform and point DEVCAPSULE_RUNTIME_PEX at them. Each launch checks that the runtime is the launcher's own build, the same version and, where both record one, the same revision (R-RUNTIME-001).

What a second platform needed

Platform.LINUX_ARM64 joins linux-amd64. A Mac runs both, native first. A project runs as the first platform it has a lock for, so a project's locks are the only thing that decides it. There is no setting to forget and no flag that changes the answer between two commands.

  • Every component is pinned for arm64 at the same versions as amd64, with each archive's sha256 computed from its downloaded bytes and checked against the vendor's checksums and Ubuntu's signed indexes (components/linux_arm64_pins.py). Eclipse's native packages come from ports.ubuntu.com, plus libunwind8, which arm64's GStreamer needs and amd64's doesn't. Playwright's Chromium is Chrome for Testing's ARM build.
  • The arm64 base is the existing recipe 11 built natively on the Mac, pinned at the digest a push of that exact image keeps. Until it is published, init says so, and a launch without a locally built base stops and gives the command to build one.
  • Six verified edges, one per IDE, from native smokes. The .NET SDK and Playwright went along for the ride but weren't exercised, so they still need --unverified.
  • Owner edits keep every lock in step. config require, offer, drop and recommend regenerate every lock the project carries, each from its own platform's resolution matrix, in one interrupted-edit journal. An owner on a Mac can't leave the committed linux-amd64 lock behind the manifest. Recovery still reads the journal an older launcher wrote.
  • Below the configuration, the platform is the lock's. It is a required argument of the pull, the build, the runtime choice and the launch. Docker only hears --platform when its daemon's architecture differs from the capsule's.

What I need from you

  1. Yes or no on macOS and ARM Linux as experimental platforms built from source, as described here.
  2. D-0006 says current() and parse() are the only ways to get a Platform. This adds runnable(), the platforms a host runs, and of_elf_machine(), the platform an executable is for. The decision should say so, or tell me to do it differently.
  3. D-0009: on a Mac the launcher can't deliver its own bytes, so DEVCAPSULE_RUNTIME_PEX is an explicit override, checked by version. Is that acceptable, or does D-0009 need reopening?
  4. Publishing the arm64 base to docker.io/mycodespaceai/devcapsule-base. Pushing the image I built keeps the pinned digest, and then one line in the matrix makes init and run work on a Mac without building anything locally. Until then, an arm64 lock should stay out of shared repositories.

What is not done

  • A checkout keeps one base-image answer. Deleting the arm64 lock to go back to emulation means answering the base question again. The macOS page says how.
  • The JetBrains smokes saw the splash window, not the first-use screen.
  • No release work: release-pex.yml still builds only the Linux x86-64 executable.
  • Found along the way, for maintenance: Ubuntu removed libsoup 3.4.4-5ubuntu0.8 on 2026-10-08, so the amd64 Eclipse closure's two libsoup URLs now return 404 and Eclipse can't be acquired from scratch on amd64.

How I tested it

  • Unit suite on the Mac: 2,764 passed, 25 skipped. mypy --platform linux reports no issues in 142 files.
  • Unit suite in a linux/amd64 container, the CI's platform: 2,779 passed, 5 failed at 9d285a5. The same five recursive tests fail on main in that container, because it has no host X11 directory, runs as root and has no Docker group.
  • IDE smokes, natively as linux-arm64, one at a time, each run's images removed afterwards:
    • VSCodium 20261104T120158Z-e8f78f
    • VS Code 20261104T120217Z-ee18ce
    • Eclipse 20261104T120256Z-aa127a
    • PyCharm 20261104T120351Z-38cdd0
    • Rider 20261104T120546Z-c9c1b3
    • IntelliJ IDEA 20261104T121007Z-289109
  • After the last launcher changes: VSCodium again, natively from a lock init --platform linux-arm64 wrote, without --unverified (20261104T130201Z-392364), and under emulation (20261104T130042Z-36f56a).
  • My own ~/hello-devcapsule, switched from emulated to native, runs aarch64 with an ARM VSCodium.
  • Reviews: two reviews of the whole diff, plus one against the codebase's own conventions. Section 5 of the linux-arm64 design note lists what they found and how each finding was fixed.

Ofc, major kudos to Opus 5.5 for this work.

@powerofsoul powerofsoul changed the title Ws linux arm64/capsules linux arm64/capsules Nov 4, 2026
powerofsoul and others added 20 commits November 4, 2026 15:46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the launcher natively on Apple Silicon Macs against Docker Desktop,
capsules as linux-amd64 under emulation, at the human's direction of
2026-11-03. The design note records the host assumptions found and the
slice plan.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Platform.current() now names the platform this host runs capsules as,
linux-amd64 on a Mac, and HostOs names the host for host mechanisms.
Docker runs, builds and pulls carry --platform when the host emulates
the capsule platform; native Linux commands are unchanged. A launcher
whose own binary cannot run in the capsule delivers the runtime that
DEVCAPSULE_RUNTIME_PEX names, or refuses with that remedy.

On macOS the host-browser bridge is unavailable with a warning, since
Docker Desktop refuses a host Unix socket as a bind source; its peer
check reads LOCAL_PEERCRED and two macOS socket races are fixed. The
capsule user joins group 0 for Docker Desktop's socket. The test suite
gets a short temporary root on macOS and skips container-side and
X11-dependent tests there.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Docker Desktop's host network is its VM's: under --network host the
contained desktop and console listen where the Mac cannot reach them
(observed on 29.3.1), while a bridge publish to 127.0.0.1 answers. An
authorized network host therefore launches on bridge with a warning.

The unit suite pins the host OS to Linux by default, as it already pins
the host-not-container default; macOS tests patch it, and the
real_host_os marker opts a test out.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The IDE smoke matched only the ready line a launcher inside a capsule
prints, which names the URL. A launcher outside a container opens the
URL instead and prints the ready line without it, so the smoke waited
out its timeout on every host but a capsule. It now takes either line,
with the URL from the launcher's earlier announcement. Its runtime
digest check compares with the runtime the launcher delivers, which on
macOS is the Linux one DEVCAPSULE_RUNTIME_PEX names.

With both, nox -s ide-smoke -- --surface pycharm passes on an Apple
Silicon Mac against Docker Desktop. The design note records the launch
and two probes beside it: the desktop in a Mac browser, identity and
ownership, and Docker access through group 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The macOS platform page said DevCapsule does not run on macOS. The
source tree's launcher now runs on Apple Silicon against Docker
Desktop, so the page says what the release still lacks, how the Mac
differs from Linux, and how to build both executables. Its commands
were run on an Apple Silicon Mac. The install page sends Mac readers
there.

The workstream's design note records that a supported macOS release
reopens D-0009: the launcher cannot deliver its own bytes there, and
today's path is the explicit override the decision admits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sudo ignores a policy file root does not own. On Linux the launcher
chowns its generated policy to root, through a constrained helper
container when the user cannot. Docker Desktop shares Mac files with
its virtual machine as the Mac user, so inside any container a
bind-mounted file stays the user's and not even root can chown it: a
macOS launch with development sudo stopped with "Permission denied".

On macOS the policy is now written by root into a Docker volume and
mounted from there, read-only, at the same path. The volume is named
after the container and removed when the session ends. The Linux
helper also gains --platform when the host emulates the capsule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Platform gains LINUX_ARM64. A host now runs a list of capsule
platforms, preferred first: a Linux host its own, Apple Silicon
linux-arm64 natively and linux-amd64 under emulation, an Intel Mac
linux-amd64. Every project command selects, for its process, the first
of those the project has a lock for, else the preferred one, so a Mac
runs a project locked only for linux-amd64 as before and a new one
natively.

DEVCAPSULE_RUNTIME_PEX may name a directory holding one runtime per
capsule platform as devcapsule-<platform>. Eclipse accepts arm64
native packages, Codex maps its arm64 npm package, and the JetBrains
runtime check names the machine's skiko library. The linux-arm64
resolution matrix shares amd64's vocabulary and recipes and has, for
now, no bases or pins of its own.

The unit suite runs capsules as linux-amd64 by default, as it runs
the Linux host by default; real_host_os tests keep the real host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The linux-arm64 matrix gains its recipe-11 base: the amd64 recipe built
natively on an Apple Silicon Mac by a launcher at public main b3f10b7.
It is not published. Its reference is the digest of the image index in
the builder's containerd store, which a push of that exact image keeps;
until the owner pushes it, a checkout selects the local image as a
local base.

`project --platform PLATFORM` names the capsule platform for one
command, refused when the host cannot run it. `project --platform
linux-arm64 init` is how a project locked for linux-amd64 gains an
arm64 lock, since init completes what a project lacks; on Apple Silicon
`--platform linux-amd64` runs a project under emulation on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The linux-arm64 matrix now pins the same component versions as
linux-amd64 with their arm64 artifacts: PyCharm, IntelliJ IDEA and
Rider, the .NET SDK, Eclipse and its WebKitGTK package closure,
VSCodium, VS Code, the Codex, Claude Code and Antigravity CLIs, and
Playwright with Chromium. Each sha256 was computed from the downloaded
bytes and matched every vendor checksum, npm integrity value and signed
Ubuntu index available; components/linux_arm64_pins.py records where
each came from. An amd64 version added without an arm64 pin fails at
import.

Ubuntu removed libsoup 3.4.4-5ubuntu0.8 from every noble pocket on
2026-10-08, so the arm64 Eclipse closure pins 3.4.4-5ubuntu0.10. The
amd64 closure's libsoup URLs return 404 as well; that is maintenance's.

The matrix has no verified edges yet: a validation on amd64 establishes
nothing on arm64, so resolution needs --unverified until arm64 smokes
record their evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A checkout keeps one set of personal pins, and on Apple Silicon a
project may run as linux-arm64 or linux-amd64. Adding an arm64 lock to
a project whose checkout had chosen VSCodium under linux-amd64 kept the
x86-64 VSCodium in the checkout's arm64 pins, because the retained
choice's table was reused as it stood; the capsule then failed at
start with "rosetta error: failed to open elf at
/lib64/ld-linux-x86-64.so.2".

Personal pins now take the platform's own embedded table for every
component version that platform pins, both when a selection is pinned
and when pins are laid over a lock, which also heals pins already
written wrongly. A version the platform does not pin, such as a channel
upgrade, is kept on its own platform and otherwise asks for a new
choice.

The macOS test session also removes the short temporary root it
creates; each run had left one in /tmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
arm64's libgstreamer1.0-0 depends on libunwind8, which the amd64
closure does not carry; the Eclipse smoke's offline dpkg install
stopped at it. Pinned from noble-updates, the index checked against
the signed InRelease and the package bytes against the index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The human's /review of every change since main rejected the branch with
seven high, fourteen medium and three low findings. Each is fixed:

- A Mac creates linux-amd64 projects again, as the human's ruling says
  (arm64 when the project is locked for it). Platform.current() is host
  detection: the platform a host creates projects for; runnable() lists
  what it runs. Which platform a project runs as is the configuration's
  question, storage.project_platform, read-only and answered from the
  project's locks; project --platform and images build --platform name
  one. The process-wide ContextVar selection is gone, so images build
  follows the project's locks too.
- Below the configuration the lock's platform is passed down: the base
  pull, the buildx build, the delivered runtime and the docker run
  arguments. Emulation is judged against the Docker daemon's
  architecture, asked once per process. Known-good version sets are the
  lock platform's only.
- A delivered runtime that is not the launcher's own bytes must be its
  build: the same version, and the same revision where both record one
  (D-0009, R-RUNTIME-001, whose record is updated). A plain PEX on a
  non-Linux host is not delivered.
- The arm64 base pin is marked unpublished: init says so, and a launch
  without a local base stops with the remedy instead of a failed pull.
- Personal pins are remapped only when written for another platform.
- The macOS sudo-policy volume is labelled, and each launch prunes
  unused ones, so an interrupted launch no longer leaks it.
- Docker Desktop's conventions are HostOs properties: the socket group,
  an unreachable host network, unbindable Unix sockets, bind mounts that
  refuse chown. The host-open code that only tests on a Mac reached is
  removed and those tests are Linux-only.
- The successor plan compares --platform in either spelling.
- Tests: the arm64 hash test walks every artifact; the suite's default
  host is amd64 Linux down to the daemon; real_host_os is renamed
  own_host_patches; the recursive orchestrator skips off Linux only; a
  failed macOS session keeps its temporary root.
- The macOS page describes linux-amd64 by default and linux-arm64 per
  project with a local base; the design note records the review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pins_for_platform now judges each component table by what it is, not
by the pins' label: a table that is exactly another platform's embedded
pin for its version takes this platform's; one that matches no
embedded table is a record kept as it stands on its labelled platform.
This keeps the review's rule, never re-deriving same-platform records,
and heals the pins an earlier build wrote with an amd64 VSCodium under
a linux-arm64 label, which re-choosing the IDE could not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Linux runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second review of all changes since main, and a review against the
codebase's own conventions, found that one checkout could switch
platform per command and that owner edits on a Mac left the committed
linux-amd64 lock behind its manifest. The design note's sections 3 and
5 record the outcome:

- project --platform and the module-global selection behind it are
  gone. A project runs as the first platform this host runs that it has
  a lock for; project init --platform adds that platform's lock, and
  images build --platform names a base build's platform. One argparse
  type parses both and refuses a platform the host cannot run.
- An owner's edit of the shared contract regenerates every lock the
  project carries under one journal. Recovery still reads the
  single-lock journal an older launcher wrote.
- Below the configuration the lock's platform is required: the base
  pull, the build, the run options and the recursive successor take it.
  The Docker daemon's architecture is asked once per launch, not cached
  for the process; Platform.emulated_docker_platform is the one
  emulation decision.
- The macOS sudo-policy sweep removes only unused policy volumes older
  than a day, so a concurrent launch's fresh volume survives.
- The retained successor plan carries its planned --platform, so the
  independent inspection checks it too.
- linux-arm64 records native smokes of VSCodium, VS Code, Eclipse,
  PyCharm, Rider and IntelliJ IDEA as verified edges; the .NET SDK and
  Playwright stay unverified.
- The arm64 pin module holds plain literals under unambiguous names,
  libunwind8 in its sorted place.
- Conventions: Warning: prefixes, owner-free comments, upper-case
  placeholders, inline Linux-only skips, one host fixture and helper in
  the tests, matrix_for and lock_platform for a lock's platform, and the
  macOS-only drain in host_open removed.
- Docs: the CLI reference and the macOS page describe init --platform;
  the unit and end-to-end test pages list the fixtures, the marker and
  the smoke's variables; index.md lists the linux-arm64 status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On an Apple Silicon Mac, init chose linux-amd64 for a new project
without a question. Init now asks before the identity questions when
the host runs more than one capsule platform, with the native one,
Platform.current(), as the default and the answer of a run without a
terminal. --platform answers it; a project with a lock the host runs
is not asked, and a Linux host never is. The macOS page, the CLI
reference and the linux-arm64 records follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@powerofsoul
powerofsoul force-pushed the ws-linux-arm64/capsules branch from a345c15 to dcc2a7d Compare November 4, 2026 13:49
@ccozianu
ccozianu merged commit bf7e5d6 into ccozianu:main Nov 4, 2026
1 check passed
ccozianu pushed a commit that referenced this pull request Nov 4, 2026
#338 merged this work before its move into the unit mirror, so main's
legacy test_host_open, test_recursive_orchestrator and test_supervisor
carry the Linux-only skips that the move took out against the older
main. Taking them out now would change legacy tests, so they stay. The
macOS host-browser test leaves test_host_open.py, since the mirror
holds the same test.

test_supervisor.py assigned pytestmark twice, and the second assignment
dropped the legacy marker; the two marks are now one list.

test conventions: 108 legacy files holding 1461 legacy tests; 22 files
follow the convention; 0 breaches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kivdG1oNN7W4zZ12jHFEG
ccozianu pushed a commit that referenced this pull request Nov 4, 2026
This Mac's arm64 recipe-11 base was pushed as
mycodespaceai/devcapsule-base:linux-arm64-recipe-11-20261104, and the
registry returns the pinned index digest, sha256:7d32268065b6..., for
linux/arm64, so the pin and its six verified edges stand unchanged.

The pin's published flag, init's note and report line about an
unpublished base, and the launch's refusal to pull one served only
this base until its push, so they are removed: an arm64 checkout pulls
its base as an amd64 one does. The macOS page no longer builds or
selects a local base, and says how a checkout that selected one
returns to the project's base.

The records note #338: linux-arm64 and macos-launcher are on main, and
this branch turns main's CI green again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kivdG1oNN7W4zZ12jHFEG
ccozianu added a commit that referenced this pull request Nov 4, 2026
#338 went in from `dcc2a7db`, three commits short of the branch's tip,
and `main` has been red since: the test-conventions gate counts 115
breaches, the tests that work added or changed inside legacy files. This
brings in the rest, pins the arm64 base as published now that it is on
Docker Hub, and lets a pull request from a fork pass CI.

## What's here

- **The three commits that missed the merge**, unchanged: the fit to
`main`'s non-Docker marker set and its e2e builder calls, the move of
every test the branch added or changed into `tests/unit` with a contract
docstring, and the record of both.
- **`main`'s Linux-only skips stay.** `main` now carries the skips that
the move had taken out of `test_host_open.py`,
`test_recursive_orchestrator.py` and `test_supervisor.py`, and taking
them out again would change legacy tests. The macOS host-browser test
leaves `test_host_open.py`, since the mirror holds the same test.
`test_supervisor.py` assigned `pytestmark` twice and so lost its
`legacy` marker; the two marks are one list now.
- **The arm64 base is published.** It went up as
`mycodespaceai/devcapsule-base:linux-arm64-recipe-11-20261104`, the
exact image the six native IDE smokes ran on, and the registry returns
the pinned digest `sha256:7d32268065b6…` for `linux/arm64`. The pin and
its verified edges don't move. With no unpublished base left, the pin's
`published` flag, init's note about it and the launch's refusal to pull
it are gone, and the macOS page no longer asks you to build a base and
select it per checkout.
- **Coverage numbers from a fork.** A pull request from a fork runs with
a read-only `GITHUB_TOKEN`, whatever the job asks for, so the step that
writes the coverage numbers into the description got HTTP 403 and failed
this PR after the conventions check and every test had passed. That step
now runs only when the PR's branch is in this repository; for a fork,
the job summary and the uploaded reports still carry the figures.

## Validation

- Test conventions against today's `main`: 108 legacy files holding
1,461 tests, 22 conforming files, 0 breaches.
- mypy: no issues in 302 files.
- Unit suite in a `linux/amd64` container: 2,820 passed, 5 failed, the
five recursive-dogfood tests that fail on `main` in the same container,
which has no host X11 directory, runs as root and has no Docker group.
They pass on CI's runners.
- Unit suite on an Apple Silicon Mac: 2,800 passed, 1 failed,
`test_broker_rejects_a_different_peer_uid`, a Linux-only legacy test
that fails on `main` on a Mac as well.
- `docker buildx imagetools inspect` on the pinned digest shows an OCI
index with `linux/arm64` and its attestation, and `docker pull
--platform linux/arm64` by that digest resolves to the recipe-11 arm64
image.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_013kivdG1oNN7W4zZ12jHFEG
ccozianu added a commit that referenced this pull request Nov 15, 2026
The owner's decision of 2026-11-07 recorded as the scope and cut decision;
the release overview with the headline, what main carried at the cut, the
agent freshness review read today and the preparation checklist; the bug
table with the Eclipse amd64 libsoup record filed from PR #338's note and
confirmed against the archive; the changelog since 0.3.0; the index, the
registry row and the maintenance status at the cut.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants