RL post-training with your Apple-silicon Macs as the MLX rollout fleet.

stoa disaggregates RL post-training into two halves. Disposable Apple-silicon Macs, anywhere from one to thousands, generate rollouts in MLX; one or more CUDA GPUs learn through Megatron / slime. The two sides meet only at a Cloudflare R2 bucket, where workers stream rollouts up and the trainer streams weights back. A Mac can join, leave, or crash mid-run and the trainer keeps learning from whatever landed.

Status: proof of concept. This project is just a proof of concept at this stage, and it is not production-hardened. A production-grade version is in development and will be released.

How it works

rollouts/ holds immutable per-worker records. The weight plane is a current.json version pointer plus anchors/ checkpoints and per-version versions/ objects, with PULSE sparse-lossless deltas between anchors; a worker polls the pointer and pulls only when the version advances. The trainer filters rollouts by staleness and reuse, then applies dual-clip PPO plus the DPPO gate on each worker's recorded logprobs before one Megatron step. GRPO stays the RL algorithm. The full design is in docs/architecture.md.

Before you start

You need:

• uv on each Mac. It installs Python 3.12 and the worker dependencies.
• A Cloudflare R2 bucket and S3 token. See docs/r2-setup.md.
• A CUDA GPU with NVIDIA drivers, model and checkpoint storage, and SSH. The 1.5B quickstart trains on an A100 or H100; the 8B reference run used one B200.
• Apple-silicon Macs. Run ops/detect_mac_tier.sh for each Mac's memory tier and decode batch G.

Optional: a Hugging Face token for gated models, Weights & Biases for training and eval curves, and a separate GPU for held-out eval.

Quickstart

1. Test rollouts on your Mac. This runs Qwen2.5-1.5B in MLX, scores GSM8K rollouts, and reads the records back through a temporary R2 prefix. No trainer yet.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra worker --extra data
cp .r2env.example ~/.r2env && chmod 600 ~/.r2env && $EDITOR ~/.r2env
uv run python worker/run_local_e2e.py 2 4 128

The arguments are 2 groups, 4 samples, 128 tokens. Omit them for the default 3 groups, 4 samples, 256 tokens. Set MODEL to use another model: MODEL=Qwen/Qwen2.5-3B-Instruct uv run python worker/run_local_e2e.py.

2. Run the trainer and worker. Bring up a slime/Megatron container and convert the base model as in envs/gsm8k/README.md. The trainer repo must live at /root/dRL (the scripts hardcode .r2env, patches/, and PYTHONPATH there); the detailed step 2 sets that up. Then:

# on the trainer
DRL_RUN=my-gsm8k DRL_PULSE_PLANE=bf16 bash envs/gsm8k/run_qwen25_gsm8k_decoupled.sh

# on your Mac
uv run python trainer/prep_gsm8k.py ~/dRL_data
DRL_RUN=my-gsm8k DRL_PULSE_PLANE=bf16 bash worker/run_dRL_worker_gsm8k.sh

Use the same DRL_RUN, DRL_PULSE_PLANE, and R2 bucket on both. GSM8K reward climbs over a few tens of steps. envs/gsm8k/README.md covers GPU setup, SSH, model conversion, the curve, and teardown; runs/psqa-decoupled/REPRODUCE.md is the same shape at 8B.

Run a fleet

This is the PaperSearchQA reference fleet: ops/join_and_run.sh brings up the PSQA retriever and worker. GSM8K has no retriever, so for a GSM8K fleet run worker/run_dRL_worker_gsm8k.sh on each Mac (per the quickstart) instead of join_and_run.sh.

1. Start the trainer in the slime/Megatron container with --rollout-num-gpus 0 --rollout-function-path r2_rollout.generate_rollout. It publishes the initial anchor, then consumes rollouts and publishes PULSE weights.
2. Join each Mac with bash ops/join_and_run.sh --name "<label>". It starts the task retriever if needed, syncs weights from R2, and streams rollouts. No registration, and you can add or drop boxes any time.
3. Watch the trainer's W&B streams under train/*, rollout/*, and worker/*, plus a held-out eval on a separate GPU.
4. Stop the trainer with ray stop --force. Stop a Mac with pkill -f dRL_producer.py or uninstall its supervisor. The trainer keeps whatever rollouts already landed.

Renting Macs is optional. Use Macs you own, or rent by the hour from Scaleway, Flow Swiss, or AWS EC2 Mac (24-hour minimum per dedicated host).

Our reference recipe (the fleet launcher and preflight, per-tier run.env, and the trainer setup we used) is in runs/psqa-decoupled/REPRODUCE.md.

Reference run

Our reference run trains LFM2.5-8B-A1B with GRPO on PaperSearchQA across a mixed Apple-silicon fleet plus one B200. On the full 5,000-question held-out test set, cover-EM pass@1 climbs from about 0.29 to 0.63 over tens of versions (pass@8 to about 0.83), and search rate rises from about 0.22 to 0.85. runs/psqa-decoupled/REPRODUCE.md is the full runnable recipe: the LFM2 slime plugin ships in-repo and the trainer runs on the public slime base, so you can rerun it end to end on your own trainer GPU plus Macs. You retrain from the public base model, so you reproduce the curve, not our exact checkpoint (that one stays private, and off-policy RL over a live fleet is stochastic). The GSM8K quickstart above is the faster smoke test.

License

stoa is MIT licensed. A few vendored or adapted files keep their upstream Apache-2.0 terms, recorded in NOTICE and THIRD_PARTY_NOTICES.md, with full texts in third_party_licenses/. The reference run's model and datasets carry their own licenses, noted in runs/psqa-decoupled/REPRODUCE.md.

Citation

If you use stoa in your research, please cite it:

@software{stoa2026,
  author  = {Miahi, Erfan},
  title   = {{stoa: RL post-training with Apple-silicon Macs as the rollout fleet}},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/PluralisResearch/stoa}
}

Companion write-up: Erfan Miahi, "RL Post-Training on Macs", Pluralis Research Blog, July 2026.

References

• GRPO, Shao et al., DeepSeekMath: https://arxiv.org/abs/2402.03300
• DPPO (the off-policy gate, trainer/dppo_gate.py), Qi et al.: https://arxiv.org/abs/2602.04879
• slime: https://github.com/THUDM/slime · MLX: https://github.com/ml-explore/mlx