Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions source/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,22 @@ The `robot_result` entry stores the robot's outcome, timing, and configuration.

Planner failures and environment errors are distinct from valid task failures. Keep them visible when reporting evaluation results.

## Start a single episode

`/embodied-run` starts one episode of an experiment created by `/embodied-setup`. Run it from the directory that holds the experiment directories. It reads each experiment's recorded setting, asks for the task flags, a GPU, the budget, and the panels, then runs the episode in the terminal, in the background through the agent, or prints the command. A request selects the robot or overrides the setting, for example `/embodied-run metaworld privileged code`.

## Inspect a session

Add `--dashboard=true` to an interactive run and open the URL printed at startup. The dashboard shows the session's observations, tool calls, and robot state. It follows pi's session history, including resumed or forked sessions.

`--dashboard-port 8765` fixes the port; `/embodied-run` uses 8765, or the next free port when it is taken. The dashboard listens on `127.0.0.1`, so forward the port to view it from another computer:

```bash
ssh -N -L 8765:127.0.0.1:8765 <gpu-host>
```

Some integrations also serve their own panel. CaP-X's Viser 3D view (`--viser --viser-port 8080`) shows point clouds, camera frustums, the end effector, and grasp candidates for LIBERO and Franka. `/embodied-run` offers a reproduced work's own panel first.

Use `/export` in pi to export the session. With this fork's export hook, the dashboard also offers JSONL and HTML downloads.

## Parallel runs
Expand Down
4 changes: 2 additions & 2 deletions source/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ RoboAgentLab connects a coding agent to robot environments, policies, and tools.
:link: quickstart
:link-type: doc

Install the extension and use `/embodied-setup` to configure LIBERO.
Install the extension, configure LIBERO with `/embodied-setup`, and start it with `/embodied-run`.
:::
:::{grid-item-card} Choose an agent interface
:link: modes
:link-type: doc

Use action units, native tools, or generated Python programs.
Pick an observation × execution cell, a reproduced setting, and a planner.
:::
:::{grid-item-card} Evaluate and inspect
:link: evaluation
Expand Down
9 changes: 4 additions & 5 deletions source/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,14 @@ RoboAgentLab contains the full pi fork, its robotics extension, and Python servi
git clone https://github.com/RoboAgentLab/RoboAgentLab ~/RoboAgentLab
cd ~/RoboAgentLab
npm ci --ignore-scripts
node scripts/hydrate-ci-models.mjs
npm run build:offline
npm run build
npm link --ignore-scripts --workspace @earendil-works/pi-coding-agent
pi install ./packages/embodied
```

The package initially loads only onboarding. `/embodied-setup` creates an experiment configured with one robot.
The package initially loads only onboarding: `/embodied-setup` creates an experiment configured with one robot, and `/embodied-run` starts its episodes.

The catalog command downloads and verifies the pinned model catalog. `build:offline` then builds the repository's packages and CLI from local sources. The first installation and catalog download require a network connection.
`npm run build` regenerates the model catalog, then builds the repository's packages and the CLI, so it needs a network connection. `npm run build:offline` skips the catalog step and uses the model data already in the checkout. Build from the checkout's real path rather than through a symbolic link to it: TypeScript otherwise resolves the workspace packages twice.

## Check your local CLI

Expand All @@ -32,7 +31,7 @@ The linked executable resolves to this checkout's `packages/coding-agent/dist/bu
node ~/RoboAgentLab/packages/coding-agent/dist/bundle/cli.js
```

After pulling source updates, run `npm ci --ignore-scripts`, hydrate the pinned catalog, and rebuild. The global link continues to use the rebuilt CLI.
After pulling source updates, run `npm ci --ignore-scripts` and `npm run build`. The global link continues to use the rebuilt CLI. Do not run `pi update` in this setup: it would replace the linked source build with a published release. Robot extensions load from the TypeScript sources, so changes under `packages/embodied` take effect at the next pi start without a rebuild.

## Install a backend manually

Expand Down
56 changes: 47 additions & 9 deletions source/modes.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,52 @@
# Agent interfaces

RoboAgentLab separates the model from the action interface. Keep the environment and task fixed while changing what the agent can call.
RoboAgentLab separates the model from what it observes and how it acts. Keep the environment and task fixed while changing one of three choices: the **observation** the agent is given, the **execution interface** it acts through, and the **planner** that decides.

| Interface | Select | Agent output |
## Observation × execution

`/embodied-setup` presents the survey's Agent-as-Policy taxonomy as a grid. Each level is a flag; O1 and A2 are a robot's defaults.

| Level | Definition | In RoboAgentLab | Flags |
|---|---|---|---|
| O1 Sensory Inputs | Decisions start from sensory inputs; agent-built perception remains O1. | Camera frames only | none |
| O2 Perception Tools | Decisions use supplied semantic or geometric perception. | SAM3 masks with ids: `detect`, `select_detection`, `reject_detection` | `--detections` (Franka, dual Franka and UR5e: `--segment`), with a SAM3 service |
| O3 Privileged State | Decisions use oracle state unavailable at deployment. | The simulator's ground-truth poses; the result is marked privileged | `--privileged` (simulators only) |
| A1 Code | The agent specifies behaviors; a fixed programmed executor realizes them. | Show-Harness action units run by the robot's motion code | `--units=true` |
| A1 Model | The agent specifies behaviors; a fixed learned executor realizes them. | The robot's VLA skill realizes each request | `--require-skills pi0` (LIBERO), `rldx` (RoboCasa), `lingbot` (RoboTwin), `--vla` (Franka, dual Franka) |
| A2 Tool Orchestration | The agent selects execution APIs and coordinates them. | The robot's own tools: perception, motion, grasping, policies | none |
| A3 Code Synthesis | The agent writes executable logic with conditions, loops, and computation. | `run_code`: Python over the robot's declared primitives | `--code=true` |

A cell combines one observation level with one execution level, for example O2 × A3 is `--code=true --detections`. Not every robot has every cell: RoboCasa and RoboTwin have no A1 Code because their tools are a VLA, HumanCLAW acts only through its motion model (A1 Model), and physical robots have no O3. `--units` and `--code` are alternative interfaces; do not enable both.

## Reproduce a published setting

After you choose a cell, the wizard lists the published settings that RoboAgentLab reproduces in it. Type part of the short name to filter the list. A setting runs as a preset or tier that expands to that work's flags.

| Short name | Work | Cell | Flags | Robots |
|---|---|---|---|---|
| `showharness` | Show-Harness | O1 × A1 Code | `--preset showharness` | any robot with action units |
| `motormind` | MotorMind | O2 × A1 Code | `--preset motormind` | LIBERO |
| `humanclaw` | HumanCLAW | O1 × A1 Model | `--preset humanclaw` | HumanCLAW |
| `rpent` | RPent | O1 × A2 | `--preset rpent` | LIBERO, RoboCasa, Franka, dual Franka |
| `openeta` | OpenETA | O2 × A2 | `--preset openeta` | any |
| `capx-S1` | CaP-X S1 | O3 × A3 | `--tier S1` | simulators with code |
| `capx-S2` … `capx-S4` | CaP-X S2–S4 | O2 × A3 | `--tier S2`, `S3`, `S4` | any robot with code |
| `capx-M1` … `capx-M4` | CaP-X M1–M4 | O2 × A3 | `--tier M1` … `M4` | any robot with code |

The other surveyed works are counted under each cell and recognized by name in a request such as `/embodied-setup reproduce <work> on <robot>`. A work that is not reproduced yet gets a notice with its cells, and the experiment starts from the cell's flags. A name the wizard does not recognize is reported as possibly unsupported.

## Planners

The planner decides each step. Every planner drives every execution interface through the same tool calls, so sessions, results, and evaluation scripts are identical whichever planner ran.

| Planner | Select | Notes |
|---|---|---|
| Native tools | default | Calls to perception, motion, grasping, or policy tools |
| Action units | `--units=true` | Discrete movement, rotation, grasp, and release units |
| Generated Python | `--code=true` | Programs calling the robot's declared primitives |
| Fine-tuned policy | a `finetuned/<adapter>` model with units | Action units from a served small model or adapter |
| Flash replay | `--model flash/replay` | A recorded plan grounded in current observations |
| Frontier API model | `--model provider/model` | Configure with `/login` or `models.json`. |
| Self-hosted model | `--model selfhost/<id>` | An OpenAI-compatible server you run; see [self-hosted planners](services.md#self-hosted-planners). |
| Distilled student | `--model finetuned/<adapter>` | A small VLM with a LoRA adapter served by vLLM; released adapters cover action units. |
| Replay | `--model flash/replay` | A recorded plan replayed with live grounding, without a model. |

The wizard's planner list is searchable: it shows your configured models, the self-hosted planners, and, for A1 Code, the robot's fine-tuned adapter.

## Action units

Expand All @@ -32,12 +70,12 @@ Start the robot without `--units=true`. The selected robot registers its availab

`--code=true` exposes `run_code`, which executes Python programs through the environment service. The primitive manifest defines the available operations and parameters. `--code=both` retains native tools as well.

`--code-api=high`, `low`, `low-noexamples`, or `raw` selects an API tier where supported. The available tier depends on the integration. Units and code are alternative interfaces; do not enable both.
`--code-api=high`, `low`, `low-noexamples`, or `raw` selects an API tier where supported. The available tier depends on the integration.

`--privileged` explicitly enables additional simulator state for configurations that support it. Keep that flag recorded with results.

## Fine-tuned models and replay

The setup wizard can configure supported fine-tuned adapters served through vLLM. Choose the adapter, prompt convention, and robot together; these are part of the experiment configuration.

Flash uses stored plans with live grounding in supported integrations, including LIBERO, RoboCasa, RoboTwin, ManiSkill, RoboLab, and RoboDojo. Consult the [complete reference](reference.md) for per-robot flags, prompts, and replay requirements.
Flash replays stored plans with live grounding in supported integrations, including LIBERO, RoboCasa, RoboTwin, ManiSkill, RoboLab, and RoboDojo. It is started by hand rather than from the wizard. Consult the [complete reference](reference.md) for per-robot flags, prompts, and replay requirements.
72 changes: 50 additions & 22 deletions source/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,17 @@ Build pi from the RoboAgentLab checkout and link that local CLI:
git clone https://github.com/RoboAgentLab/RoboAgentLab ~/RoboAgentLab
cd ~/RoboAgentLab
npm ci --ignore-scripts
node scripts/hydrate-ci-models.mjs
npm run build:offline
npm run build
npm link --ignore-scripts --workspace @earendil-works/pi-coding-agent
pi install ./packages/embodied
mkdir -p ~/robot-runs
cd ~/robot-runs
pi
```

`npm link` makes `pi` run the CLI built inside this checkout. Rebuild with `npm run build:offline` after changing pi's source. See [installation](installation.md#check-your-local-cli) if another pi installation is already on your `PATH`.
`npm link` makes `pi` run the CLI built inside this checkout. See [installation](installation.md#check-your-local-cli) if another pi installation is already on your `PATH`, and for updating the checkout.

Use a planner that accepts images. Inside pi, run `/login` to configure a supported provider. For a custom endpoint, configure `~/.pi/agent/models.json` using [pi's model configuration guide](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/models.md).
Use a planner that accepts images. Inside pi, run `/login` to configure a supported provider, or add a [self-hosted model](services.md#self-hosted-planners).

## 2. Run the setup wizard

Expand All @@ -33,47 +32,74 @@ Inside the interactive pi session, enter:
/embodied-setup
```

Choose the following for a first run:
The wizard asks one question at a time. Choose the following for a first run:

| Setting | Choose |
| Question | Choose |
|---|---|
| Robot or benchmark | LIBERO (sim) |
| Mode | units |
| Services | here (this machine) |
| Planner | your configured image-capable model |
| Experiment directory | `~/robot-runs/embodied-libero` |
| Simulation or real robot? | sim |
| Robot or benchmark | LIBERO |
| Observation × Execution | **O1 × A1 Code**: camera frames only, action units |
| Work to reproduce | `showharness` (Show-Harness's setting), or none for the plain cell |
| Where do the services run? | here (this machine) |
| Planner model | your image-capable model; type to filter the list |
| Experiment directory | the default, `~/robot-runs/embodied-libero` |

Review the setup plan and confirm it. The agent follows the bundled `embodied-quickstart` skill to install the Python environment and scene assets, run the preflight checks, and print the launch command. It also writes the experiment's `.pi/settings.json`, which loads the robot and dashboard.
The observation × execution grid is the survey's Agent-as-Policy taxonomy. Use the arrow keys to move between cells; cells a robot does not support are shown as `–`. Each cell shows its definition, the flags it adds, and how many surveyed works use it. See [agent interfaces](modes.md) for every level.

You can also describe the experiment after the command, and the wizard asks only for what you did not name:

```text
/embodied-setup reproduce Show-Harness on LIBERO here
```

The confirmation lists the robot, the setting and its flags, what the install needs, and the launch command. After you confirm, the agent follows the bundled `embodied-quickstart` skill: it installs the Python environment and scene assets, runs the preflight checks, and ends with the launch command. It does not start an episode. The wizard writes the experiment's `.pi/settings.json`, which loads the robot and dashboard, and records the chosen setting in `.pi/embodied-setup.json`.

If you select a remote GPU box, pi, the checkout, and the experiment run on that box over SSH. See [installation](installation.md) for a manual setup.

## 3. Start the episode

Exit the setup session and run the printed command. With the paths above:
Exit the setup session, start pi in the same directory where you ran the wizard, and enter:

```text
/embodied-run
```

It finds the experiment directories there and reads their recorded setting. Then it asks for the task flags, a GPU (the one with the most free memory is listed first), the episode budget, and how to run:

| Choice | What happens |
|---|---|
| here | pi takes this terminal and starts the episode immediately |
| headless | the agent runs the episode in the background and reports the result |
| print | the command is shown for you to run yourself |

The first start in an experiment directory asks whether to trust the project; accept it so that the robot extension loads. The command `/embodied-run` builds for this setup is equivalent to:

```bash
cd ~/robot-runs/embodied-libero
source ~/RoboAgentLab/services/.venv-libero/pi-embodied.env
pi --libero-type standard --suite libero_10 --task 0 --seed 0 \
--units=true --max-turns 40 --time-limit 900 --dashboard=true
```

Accept pi's project trust prompt so that the experiment's robot extension loads. When the scene is ready, type:

```text
Solve the task.
--preset showharness --max-turns 40 --time-limit 900 \
--dashboard=true --dashboard-port 8765 "Solve the task."
```

The agent receives camera observations and chooses movement, grasp, and release units. The run is bounded to 40 turns or 900 seconds. For a model that supports configurable reasoning, you can add `--thinking low`.

## 4. Inspect what happened

Open the dashboard URL printed in the terminal to follow observations and tool calls. At the end, inspect the `robot_result` entry: `terminated` is the environment's success signal, while `claimed` is the agent's reported outcome.
Open the dashboard at `http://127.0.0.1:8765/` to follow observations and tool calls. The dashboard listens only on the machine that runs pi. From your own computer, forward the port first; `/embodied-run` prints this command when pi runs over SSH:

```bash
ssh -N -L 8765:127.0.0.1:8765 <gpu-host>
```

A `LocalForward 8765 127.0.0.1:8765` line in that host's SSH configuration makes every SSH session carry it.

At the end, inspect the `robot_result` entry: `terminated` is the environment's success signal, while `claimed` is the agent's reported outcome. `planner_budget_exhausted` shows when the turn or time budget ended the episode.

Use `/export` for a shareable session view. Continue to [evaluation and replay](evaluation.md) to run several tasks or [data collection](data.md) to export trajectories.

:::{tip}
For a non-interactive episode, use the same flags with `pi -a -p` and append `"Solve the task."`. The `-a` option approves loading this experiment's extensions for that invocation.
For a non-interactive episode, use the same flags with `pi -a -p` and close standard input: `pi -a -p <flags> "Solve the task." < /dev/null`. Without the redirect, `pi -p` waits for standard input to end, which never happens over SSH or from an agent's shell. The `-a` option approves loading this experiment's extensions for that invocation.
:::

## Common setup issues
Expand All @@ -85,4 +111,6 @@ For a non-interactive episode, use the same flags with `pi -a -p` and append `"S
| Python or environment import fails | Source the `pi-embodied.env` file for the selected backend. |
| Missing LIBERO assets | Run `services/setup.sh libero`; the default installation downloads the scene assets. |
| Duplicate tools or flags | Load one robot per process. Run evaluation scripts from the checkout, outside a configured experiment directory. |
| The dashboard does not open from your computer | Forward its port with `ssh -L` as shown above; it listens on `127.0.0.1` only. |
| A headless run never starts | Add `< /dev/null` to the `pi -p` command. |
| GPU renderer fails | Check the NVIDIA driver and EGL installation on the machine running the services. |
Loading
Loading