diff --git a/source/evaluation.md b/source/evaluation.md index eb7a286..d543591 100644 --- a/source/evaluation.md +++ b/source/evaluation.md @@ -23,10 +23,22 @@ The `robot_result` entry stores the robot's outcome, timing, and configuration. Planner failures and environment errors are distinct from valid task failures. Keep them visible when reporting evaluation results. +## Start a single episode + +`/embodied-run` starts one episode of an experiment created by `/embodied-setup`. Run it from the directory that holds the experiment directories. It reads each experiment's recorded setting, asks for the task flags, a GPU, the budget, and the panels, then runs the episode in the terminal, in the background through the agent, or prints the command. A request selects the robot or overrides the setting, for example `/embodied-run metaworld privileged code`. + ## Inspect a session Add `--dashboard=true` to an interactive run and open the URL printed at startup. The dashboard shows the session's observations, tool calls, and robot state. It follows pi's session history, including resumed or forked sessions. +`--dashboard-port 8765` fixes the port; `/embodied-run` uses 8765, or the next free port when it is taken. The dashboard listens on `127.0.0.1`, so forward the port to view it from another computer: + +```bash +ssh -N -L 8765:127.0.0.1:8765 +``` + +Some integrations also serve their own panel. CaP-X's Viser 3D view (`--viser --viser-port 8080`) shows point clouds, camera frustums, the end effector, and grasp candidates for LIBERO and Franka. `/embodied-run` offers a reproduced work's own panel first. + Use `/export` in pi to export the session. With this fork's export hook, the dashboard also offers JSONL and HTML downloads. ## Parallel runs diff --git a/source/index.md b/source/index.md index cf81cf0..f560092 100644 --- a/source/index.md +++ b/source/index.md @@ -9,13 +9,13 @@ RoboAgentLab connects a coding agent to robot environments, policies, and tools. :link: quickstart :link-type: doc -Install the extension and use `/embodied-setup` to configure LIBERO. +Install the extension, configure LIBERO with `/embodied-setup`, and start it with `/embodied-run`. ::: :::{grid-item-card} Choose an agent interface :link: modes :link-type: doc -Use action units, native tools, or generated Python programs. +Pick an observation × execution cell, a reproduced setting, and a planner. ::: :::{grid-item-card} Evaluate and inspect :link: evaluation diff --git a/source/installation.md b/source/installation.md index 1f93776..d6139c6 100644 --- a/source/installation.md +++ b/source/installation.md @@ -8,15 +8,14 @@ RoboAgentLab contains the full pi fork, its robotics extension, and Python servi git clone https://github.com/RoboAgentLab/RoboAgentLab ~/RoboAgentLab cd ~/RoboAgentLab npm ci --ignore-scripts -node scripts/hydrate-ci-models.mjs -npm run build:offline +npm run build npm link --ignore-scripts --workspace @earendil-works/pi-coding-agent pi install ./packages/embodied ``` -The package initially loads only onboarding. `/embodied-setup` creates an experiment configured with one robot. +The package initially loads only onboarding: `/embodied-setup` creates an experiment configured with one robot, and `/embodied-run` starts its episodes. -The catalog command downloads and verifies the pinned model catalog. `build:offline` then builds the repository's packages and CLI from local sources. The first installation and catalog download require a network connection. +`npm run build` regenerates the model catalog, then builds the repository's packages and the CLI, so it needs a network connection. `npm run build:offline` skips the catalog step and uses the model data already in the checkout. Build from the checkout's real path rather than through a symbolic link to it: TypeScript otherwise resolves the workspace packages twice. ## Check your local CLI @@ -32,7 +31,7 @@ The linked executable resolves to this checkout's `packages/coding-agent/dist/bu node ~/RoboAgentLab/packages/coding-agent/dist/bundle/cli.js ``` -After pulling source updates, run `npm ci --ignore-scripts`, hydrate the pinned catalog, and rebuild. The global link continues to use the rebuilt CLI. +After pulling source updates, run `npm ci --ignore-scripts` and `npm run build`. The global link continues to use the rebuilt CLI. Do not run `pi update` in this setup: it would replace the linked source build with a published release. Robot extensions load from the TypeScript sources, so changes under `packages/embodied` take effect at the next pi start without a rebuild. ## Install a backend manually diff --git a/source/modes.md b/source/modes.md index f240f3f..4752f41 100644 --- a/source/modes.md +++ b/source/modes.md @@ -1,14 +1,52 @@ # Agent interfaces -RoboAgentLab separates the model from the action interface. Keep the environment and task fixed while changing what the agent can call. +RoboAgentLab separates the model from what it observes and how it acts. Keep the environment and task fixed while changing one of three choices: the **observation** the agent is given, the **execution interface** it acts through, and the **planner** that decides. -| Interface | Select | Agent output | +## Observation × execution + +`/embodied-setup` presents the survey's Agent-as-Policy taxonomy as a grid. Each level is a flag; O1 and A2 are a robot's defaults. + +| Level | Definition | In RoboAgentLab | Flags | +|---|---|---|---| +| O1 Sensory Inputs | Decisions start from sensory inputs; agent-built perception remains O1. | Camera frames only | none | +| O2 Perception Tools | Decisions use supplied semantic or geometric perception. | SAM3 masks with ids: `detect`, `select_detection`, `reject_detection` | `--detections` (Franka, dual Franka and UR5e: `--segment`), with a SAM3 service | +| O3 Privileged State | Decisions use oracle state unavailable at deployment. | The simulator's ground-truth poses; the result is marked privileged | `--privileged` (simulators only) | +| A1 Code | The agent specifies behaviors; a fixed programmed executor realizes them. | Show-Harness action units run by the robot's motion code | `--units=true` | +| A1 Model | The agent specifies behaviors; a fixed learned executor realizes them. | The robot's VLA skill realizes each request | `--require-skills pi0` (LIBERO), `rldx` (RoboCasa), `lingbot` (RoboTwin), `--vla` (Franka, dual Franka) | +| A2 Tool Orchestration | The agent selects execution APIs and coordinates them. | The robot's own tools: perception, motion, grasping, policies | none | +| A3 Code Synthesis | The agent writes executable logic with conditions, loops, and computation. | `run_code`: Python over the robot's declared primitives | `--code=true` | + +A cell combines one observation level with one execution level, for example O2 × A3 is `--code=true --detections`. Not every robot has every cell: RoboCasa and RoboTwin have no A1 Code because their tools are a VLA, HumanCLAW acts only through its motion model (A1 Model), and physical robots have no O3. `--units` and `--code` are alternative interfaces; do not enable both. + +## Reproduce a published setting + +After you choose a cell, the wizard lists the published settings that RoboAgentLab reproduces in it. Type part of the short name to filter the list. A setting runs as a preset or tier that expands to that work's flags. + +| Short name | Work | Cell | Flags | Robots | +|---|---|---|---|---| +| `showharness` | Show-Harness | O1 × A1 Code | `--preset showharness` | any robot with action units | +| `motormind` | MotorMind | O2 × A1 Code | `--preset motormind` | LIBERO | +| `humanclaw` | HumanCLAW | O1 × A1 Model | `--preset humanclaw` | HumanCLAW | +| `rpent` | RPent | O1 × A2 | `--preset rpent` | LIBERO, RoboCasa, Franka, dual Franka | +| `openeta` | OpenETA | O2 × A2 | `--preset openeta` | any | +| `capx-S1` | CaP-X S1 | O3 × A3 | `--tier S1` | simulators with code | +| `capx-S2` … `capx-S4` | CaP-X S2–S4 | O2 × A3 | `--tier S2`, `S3`, `S4` | any robot with code | +| `capx-M1` … `capx-M4` | CaP-X M1–M4 | O2 × A3 | `--tier M1` … `M4` | any robot with code | + +The other surveyed works are counted under each cell and recognized by name in a request such as `/embodied-setup reproduce on `. A work that is not reproduced yet gets a notice with its cells, and the experiment starts from the cell's flags. A name the wizard does not recognize is reported as possibly unsupported. + +## Planners + +The planner decides each step. Every planner drives every execution interface through the same tool calls, so sessions, results, and evaluation scripts are identical whichever planner ran. + +| Planner | Select | Notes | |---|---|---| -| Native tools | default | Calls to perception, motion, grasping, or policy tools | -| Action units | `--units=true` | Discrete movement, rotation, grasp, and release units | -| Generated Python | `--code=true` | Programs calling the robot's declared primitives | -| Fine-tuned policy | a `finetuned/` model with units | Action units from a served small model or adapter | -| Flash replay | `--model flash/replay` | A recorded plan grounded in current observations | +| Frontier API model | `--model provider/model` | Configure with `/login` or `models.json`. | +| Self-hosted model | `--model selfhost/` | An OpenAI-compatible server you run; see [self-hosted planners](services.md#self-hosted-planners). | +| Distilled student | `--model finetuned/` | A small VLM with a LoRA adapter served by vLLM; released adapters cover action units. | +| Replay | `--model flash/replay` | A recorded plan replayed with live grounding, without a model. | + +The wizard's planner list is searchable: it shows your configured models, the self-hosted planners, and, for A1 Code, the robot's fine-tuned adapter. ## Action units @@ -32,7 +70,7 @@ Start the robot without `--units=true`. The selected robot registers its availab `--code=true` exposes `run_code`, which executes Python programs through the environment service. The primitive manifest defines the available operations and parameters. `--code=both` retains native tools as well. -`--code-api=high`, `low`, `low-noexamples`, or `raw` selects an API tier where supported. The available tier depends on the integration. Units and code are alternative interfaces; do not enable both. +`--code-api=high`, `low`, `low-noexamples`, or `raw` selects an API tier where supported. The available tier depends on the integration. `--privileged` explicitly enables additional simulator state for configurations that support it. Keep that flag recorded with results. @@ -40,4 +78,4 @@ Start the robot without `--units=true`. The selected robot registers its availab The setup wizard can configure supported fine-tuned adapters served through vLLM. Choose the adapter, prompt convention, and robot together; these are part of the experiment configuration. -Flash uses stored plans with live grounding in supported integrations, including LIBERO, RoboCasa, RoboTwin, ManiSkill, RoboLab, and RoboDojo. Consult the [complete reference](reference.md) for per-robot flags, prompts, and replay requirements. +Flash replays stored plans with live grounding in supported integrations, including LIBERO, RoboCasa, RoboTwin, ManiSkill, RoboLab, and RoboDojo. It is started by hand rather than from the wizard. Consult the [complete reference](reference.md) for per-robot flags, prompts, and replay requirements. diff --git a/source/quickstart.md b/source/quickstart.md index 9c6cc94..f3ded15 100644 --- a/source/quickstart.md +++ b/source/quickstart.md @@ -12,8 +12,7 @@ Build pi from the RoboAgentLab checkout and link that local CLI: git clone https://github.com/RoboAgentLab/RoboAgentLab ~/RoboAgentLab cd ~/RoboAgentLab npm ci --ignore-scripts -node scripts/hydrate-ci-models.mjs -npm run build:offline +npm run build npm link --ignore-scripts --workspace @earendil-works/pi-coding-agent pi install ./packages/embodied mkdir -p ~/robot-runs @@ -21,9 +20,9 @@ cd ~/robot-runs pi ``` -`npm link` makes `pi` run the CLI built inside this checkout. Rebuild with `npm run build:offline` after changing pi's source. See [installation](installation.md#check-your-local-cli) if another pi installation is already on your `PATH`. +`npm link` makes `pi` run the CLI built inside this checkout. See [installation](installation.md#check-your-local-cli) if another pi installation is already on your `PATH`, and for updating the checkout. -Use a planner that accepts images. Inside pi, run `/login` to configure a supported provider. For a custom endpoint, configure `~/.pi/agent/models.json` using [pi's model configuration guide](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/models.md). +Use a planner that accepts images. Inside pi, run `/login` to configure a supported provider, or add a [self-hosted model](services.md#self-hosted-planners). ## 2. Run the setup wizard @@ -33,47 +32,74 @@ Inside the interactive pi session, enter: /embodied-setup ``` -Choose the following for a first run: +The wizard asks one question at a time. Choose the following for a first run: -| Setting | Choose | +| Question | Choose | |---|---| -| Robot or benchmark | LIBERO (sim) | -| Mode | units | -| Services | here (this machine) | -| Planner | your configured image-capable model | -| Experiment directory | `~/robot-runs/embodied-libero` | +| Simulation or real robot? | sim | +| Robot or benchmark | LIBERO | +| Observation × Execution | **O1 × A1 Code**: camera frames only, action units | +| Work to reproduce | `showharness` (Show-Harness's setting), or none for the plain cell | +| Where do the services run? | here (this machine) | +| Planner model | your image-capable model; type to filter the list | +| Experiment directory | the default, `~/robot-runs/embodied-libero` | -Review the setup plan and confirm it. The agent follows the bundled `embodied-quickstart` skill to install the Python environment and scene assets, run the preflight checks, and print the launch command. It also writes the experiment's `.pi/settings.json`, which loads the robot and dashboard. +The observation × execution grid is the survey's Agent-as-Policy taxonomy. Use the arrow keys to move between cells; cells a robot does not support are shown as `–`. Each cell shows its definition, the flags it adds, and how many surveyed works use it. See [agent interfaces](modes.md) for every level. + +You can also describe the experiment after the command, and the wizard asks only for what you did not name: + +```text +/embodied-setup reproduce Show-Harness on LIBERO here +``` + +The confirmation lists the robot, the setting and its flags, what the install needs, and the launch command. After you confirm, the agent follows the bundled `embodied-quickstart` skill: it installs the Python environment and scene assets, runs the preflight checks, and ends with the launch command. It does not start an episode. The wizard writes the experiment's `.pi/settings.json`, which loads the robot and dashboard, and records the chosen setting in `.pi/embodied-setup.json`. If you select a remote GPU box, pi, the checkout, and the experiment run on that box over SSH. See [installation](installation.md) for a manual setup. ## 3. Start the episode -Exit the setup session and run the printed command. With the paths above: +Exit the setup session, start pi in the same directory where you ran the wizard, and enter: + +```text +/embodied-run +``` + +It finds the experiment directories there and reads their recorded setting. Then it asks for the task flags, a GPU (the one with the most free memory is listed first), the episode budget, and how to run: + +| Choice | What happens | +|---|---| +| here | pi takes this terminal and starts the episode immediately | +| headless | the agent runs the episode in the background and reports the result | +| print | the command is shown for you to run yourself | + +The first start in an experiment directory asks whether to trust the project; accept it so that the robot extension loads. The command `/embodied-run` builds for this setup is equivalent to: ```bash cd ~/robot-runs/embodied-libero source ~/RoboAgentLab/services/.venv-libero/pi-embodied.env pi --libero-type standard --suite libero_10 --task 0 --seed 0 \ - --units=true --max-turns 40 --time-limit 900 --dashboard=true -``` - -Accept pi's project trust prompt so that the experiment's robot extension loads. When the scene is ready, type: - -```text -Solve the task. + --preset showharness --max-turns 40 --time-limit 900 \ + --dashboard=true --dashboard-port 8765 "Solve the task." ``` The agent receives camera observations and chooses movement, grasp, and release units. The run is bounded to 40 turns or 900 seconds. For a model that supports configurable reasoning, you can add `--thinking low`. ## 4. Inspect what happened -Open the dashboard URL printed in the terminal to follow observations and tool calls. At the end, inspect the `robot_result` entry: `terminated` is the environment's success signal, while `claimed` is the agent's reported outcome. +Open the dashboard at `http://127.0.0.1:8765/` to follow observations and tool calls. The dashboard listens only on the machine that runs pi. From your own computer, forward the port first; `/embodied-run` prints this command when pi runs over SSH: + +```bash +ssh -N -L 8765:127.0.0.1:8765 +``` + +A `LocalForward 8765 127.0.0.1:8765` line in that host's SSH configuration makes every SSH session carry it. + +At the end, inspect the `robot_result` entry: `terminated` is the environment's success signal, while `claimed` is the agent's reported outcome. `planner_budget_exhausted` shows when the turn or time budget ended the episode. Use `/export` for a shareable session view. Continue to [evaluation and replay](evaluation.md) to run several tasks or [data collection](data.md) to export trajectories. :::{tip} -For a non-interactive episode, use the same flags with `pi -a -p` and append `"Solve the task."`. The `-a` option approves loading this experiment's extensions for that invocation. +For a non-interactive episode, use the same flags with `pi -a -p` and close standard input: `pi -a -p "Solve the task." < /dev/null`. Without the redirect, `pi -p` waits for standard input to end, which never happens over SSH or from an agent's shell. The `-a` option approves loading this experiment's extensions for that invocation. ::: ## Common setup issues @@ -85,4 +111,6 @@ For a non-interactive episode, use the same flags with `pi -a -p` and append `"S | Python or environment import fails | Source the `pi-embodied.env` file for the selected backend. | | Missing LIBERO assets | Run `services/setup.sh libero`; the default installation downloads the scene assets. | | Duplicate tools or flags | Load one robot per process. Run evaluation scripts from the checkout, outside a configured experiment directory. | +| The dashboard does not open from your computer | Forward its port with `ssh -L` as shown above; it listens on `127.0.0.1` only. | +| A headless run never starts | Add `< /dev/null` to the `pi -p` command. | | GPU renderer fails | Check the NVIDIA driver and EGL installation on the machine running the services. | diff --git a/source/reference.md b/source/reference.md index f6be040..5ed4582 100644 --- a/source/reference.md +++ b/source/reference.md @@ -34,22 +34,25 @@ Prerequisites: git clone https://github.com/RoboAgentLab/RoboAgentLab ~/RoboAgentLab cd ~/RoboAgentLab npm ci --ignore-scripts -node scripts/hydrate-ci-models.mjs -npm run build:offline +npm run build npm link --ignore-scripts --workspace @earendil-works/pi-coding-agent pi install ./packages/embodied mkdir -p ~/robot-runs && cd ~/robot-runs && pi # a notice points to /embodied-setup ``` -`/embodied-setup` asks for the robot or benchmark (and what it needs: GPU, Isaac Sim, real hardware -and an operator), the mode (tools, units, fine-tuned, flash), where the services run (here, or a -remote GPU box over ssh), the planner model (tools and units modes) and the experiment directory -(`Experiment directory (empty: /embodied-)`). Without a model it says how to add one +`/embodied-setup` asks whether the target is a simulator or a real robot, then the robot or +benchmark, its cell in the observation × execution grid ([agent interfaces](modes.md)), optionally a +reproduced work's setting in that cell, where the services run (here, or a remote GPU box over ssh), +the planner model (a searchable list that includes the [self-hosted planners](services.md#self-hosted-planners)) +and the experiment directory (`Experiment directory (empty: /embodied-)`). A request +after the command, such as `/embodied-setup reproduce RPent on LIBERO-PRO`, settles what it names. +What a robot needs (GPU, Isaac Sim, real hardware and an operator) appears in the confirmation. Without a model it says how to add one (`/login`, or `~/.pi/agent/models.json`) and exits without writing anything. After you confirm, it writes the experiment directory's `.pi/settings.json` and asks the agent to follow the `embodied-quickstart` skill: run `services/setup.sh ` (venv, pyproject extra, assets, `--weights` for checkpoints), then the preflight (`node src/infra/check.ts `; `/robot-check` -inside a robot session), and report the launch command. The agent runs each step with its bash +inside a robot session), and report the launch command; it does not start an episode, which +`/embodied-run` does. The agent runs each step with its bash tool; pi does not ask per step, so your confirmation covers the install, and the agent asks before sudo, real hardware or a download the plan does not list. @@ -66,8 +69,9 @@ declares, so they cannot add a robot): } ``` -Settings cannot carry extension flags, so the task, the mode and the budget stay on the command -line. The launch command `/embodied-setup` prints for LIBERO in units mode (with absolute paths): +Settings cannot carry extension flags, so the task, the setting and the budget stay on the command +line; the wizard records the setting in `.pi/embodied-setup.json` for `/embodied-run`. The launch +command `/embodied-setup` prints for LIBERO in O1 × A1 Code (with absolute paths): ```bash cd ~/robot-runs/embodied-libero && source ~/RoboAgentLab/services/.venv-libero/pi-embodied.env && pi --libero-type standard --suite libero_10 --task 0 --seed 0 --units=true --max-turns 40 --time-limit 900 --dashboard=true @@ -78,7 +82,8 @@ levels). The robot resets the scene and waits for a prompt: type `Solve the task.` to start the episode. In print mode the prompt goes on the command line: -`pi -a -p "Solve the task."`. Every turn sends camera images to the planner, and +`pi -a -p "Solve the task." < /dev/null` (close stdin: `pi -p` reads a piped stdin to its +end first, and over ssh that pipe never closes). Every turn sends camera images to the planner, and without `--max-turns` / `--time-limit` (default 0, no limit) an episode runs until the model calls `finish`, so give every run against a paid API a budget. The first pi there asks to trust the project; `pi -p` cannot ask, so it runs without the robot unless diff --git a/source/services.md b/source/services.md index fc3974d..73dd059 100644 --- a/source/services.md +++ b/source/services.md @@ -30,6 +30,73 @@ Example profile — replace the paths and endpoints for your machine: Select it with `--deployment gpu-box` and inspect it with `/embodied-config`. +## Self-hosted planners + +A planner can be a model you serve yourself through an OpenAI-compatible API. The setup wizard always offers three self-hosted Qwen3.8 planners: + +| Planner | Engine | Weights | Hardware | +|---|---|---|---| +| `selfhost/qwen3.8-27b-bf16` | [vLLM](https://docs.vllm.ai/) | Qwen3.8-27B BF16, unquantized (55.6 GB) | a GPU with 80 GB or more | +| `selfhost/qwen3.8-27b` | [NInfer](https://github.com/Neroued/ninfer) | Qwen3.8-27B NVFP4 | RTX 5090 or DGX Spark | +| `selfhost/qwen3.8-flash-next-unsloth-q8_0` | [Strata](https://github.com/Niko1221/Strata) | Qwen3.8-Flash-Next Q8_0 | an NVIDIA GPU and about 140 GB of RAM for the experts | + +Describe where each one runs in the deployment's `selfhost` entry, keyed by model id: + +```json +{ + "deployments": { + "default": { + "selfhost": { + "qwen3.8-27b-bf16": { + "base_url": "http://127.0.0.1:18180/v1", + "start": "selfhost-planner vllm start", + "stop": "selfhost-planner vllm stop", + "run_on": "the workstation that reaches the GPU host" + } + } + } + } +} +``` + +`base_url` is the address as the robot machine reaches the server. Requests carry the episode's camera images, often more than a megabyte each, so route the tunnel through a host with a fast link to the GPU host rather than through a slow uplink. `start` and `stop` are shell commands, and `run_on` names the machine where they must run when it is not the robot machine. When you pick a self-hosted planner, the setup agent starts the server if `base_url` does not answer (with `run_on`, it gives you the command to run there), registers the model under the `selfhost` provider in `~/.pi/agent/models.json`, and checks one image request and one tool call. Without a deployment entry, it sets the engine up on a machine you name. `/embodied-run` checks that the server answers before an episode and prints its start command when it does not. + +A registered self-hosted model looks like this in `models.json`: + +```json +{ + "providers": { + "selfhost": { + "api": "openai-completions", + "apiKey": "none", + "compat": { "supportsDeveloperRole": false, "maxTokensField": "max_tokens", "thinkingFormat": "qwen-chat-template" }, + "models": [ + { "id": "qwen3.8-27b-bf16", "baseUrl": "http://127.0.0.1:18180/v1", "input": ["text", "image"], + "reasoning": true, "contextWindow": 65536, "maxTokens": 8192 } + ] + } + } +} +``` + +vLLM serves any Hugging Face checkpoint in BF16. Start it with the Qwen reasoning and tool-call parsers so that the planner's thinking and tool calls are parsed: + +```bash +vllm serve --served-model-name qwen3.8-27b-bf16 --dtype bfloat16 \ + --max-model-len 65536 --reasoning-parser qwen3 \ + --enable-auto-tool-choice --tool-call-parser qwen3_coder \ + --enable-prefix-caching --max-num-seqs 8 --limit-mm-per-prompt '{"image": 8}' +``` + +NInfer builds only for RTX 5090 and DGX Spark, and its artifacts are quantized. Strata keeps the Flash-Next experts in system RAM; under a smaller memory limit it can read them from disk with `--set mmap-experts=on`, at roughly one token per second, which is too slow for episodes. + +When the GPU host and the robot machine cannot reach each other, forward the port through a machine that reaches both: + +```bash +ssh -f -N -L 18180:127.0.0.1:8000 +ssh -f -N -R 18180:127.0.0.1:18180 +``` + ## Start or attach Robot extensions can start their environment server from the configured services checkout. `--env-url` attaches to an existing server. Model services have separate endpoints.