Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/development/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ Repository-wide contributor guidance.

- [Coding Standards](coding-standards.md) — coding, writing, and testing standards for all contributors and agents.
- [Security Rules](security.md) — security requirements for `src/physicalai/`.
- [Component config](component-config.md) — canonical accepted design note for captured component configuration.
1,057 changes: 1,057 additions & 0 deletions docs/development/component-config.md

Large diffs are not rendered by default.

17 changes: 14 additions & 3 deletions docs/development/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,20 @@ These rules apply when writing, editing, or reviewing code under `src/physicalai
raise ValueError(f"Path escapes base directory: {user_path!r}")
```

4. No arbitrary `class_path` import from untrusted manifests or YAML. `instantiate_component` in `inference/component_factory.py` resolves `class_path` via `ComponentRegistry` and `importlib`. Only register trusted short names; treat manifest `class_path` values as untrusted unless the export directory is trusted. Prefer registered `type` names for built-ins.

5. Enforce component nesting limits. `_MAX_COMPONENT_DEPTH` in `component_factory.py` caps recursive manifest/YAML instantiation — do not raise or bypass without a security review.
4. No arbitrary `class_path` import from untrusted manifests, YAML, or peer
payloads. `instantiate_component` in `inference/component_factory.py`
resolves `class_path` via `ComponentRegistry` and `importlib`. Only register
trusted short names; treat manifest `class_path` values as untrusted unless
the export directory is trusted. Prefer registered `type` names for
built-ins. `physicalai.config.instantiate` is a separate trusted-local /
parent→child-only construction boundary: never pass robot/camera network
metadata, Zenoh payloads, or other untrusted peer data into it.

5. Enforce component nesting limits. `_MAX_COMPONENT_DEPTH` in
`component_factory.py` caps recursive manifest/YAML instantiation;
`_MAX_CONFIG_DEPTH` in `physicalai.config` caps recursive
`to_config` / `instantiate` trees — do not raise or bypass either without a
security review.

6. Never use `pickle`, `eval()`, `exec()`, `joblib`, `dill`, or `cloudpickle` on untrusted data. Prefer `json` for structured metadata, `safetensors` for weights, and `numpy.load(..., allow_pickle=False)` for arrays.

Expand Down
16 changes: 16 additions & 0 deletions docs/development/shared-construction-wire-decision.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Shared construction wire decision

**Status:** Accepted (recorded in the component-config design)

Private robot-owner and camera-publisher startup stdin is a same-package
ephemeral `Popen` handshake, not a persisted peer protocol. Change the
construction payload to `robot: ComponentConfig` / `camera: ComponentConfig`
with a **hard cutover**: rewrite writers, readers, and fixtures in the same
PR as the Shared\* spawn path. Do **not** add `config_format`, dual-read, or
shape-detection fallback.

Do not bump `ROBOT_TRANSPORT_PROTOCOL_VERSION` or camera frame
`PROTOCOL_VERSION` for this change; those version network/frame payloads, not
startup envelopes.

Canonical detail: [component-config.md — Private startup envelopes](component-config.md#private-startup-envelopes-hard-cutover).
27 changes: 24 additions & 3 deletions docs/explanation/cameras.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,37 @@ Camera instances are not thread-safe. Use one thread per camera instance or add

## SharedCamera

For multi-process or multi-thread access, use `SharedCamera`. It wraps any camera and handles IPC transport automatically.
`SharedCamera` lets one publisher subprocess own a camera's exclusive hardware
connection while any number of subscribers read frames over iceoryx2. It
satisfies the same `Camera` protocol as a direct driver.

Construction is ComponentConfig-only (`from_config` / `from_camera`), matching
`SharedRobot`. Prefer `from_config` (or YAML) when sharing. `from_camera` is
export-only sugar after disconnect — it does not hand off an open device into
the child; the publisher always opens fresh. Never keep a direct camera
connected to the same device while sharing: another connected holder causes
open failure.

```python
from physicalai.capture import SharedCamera, UVCCamera
from physicalai.capture import SharedCamera
from physicalai.config import to_config

shared = SharedCamera(UVCCamera(device="/dev/video0"))
# Prefer config-only / disconnected export — no live direct camera held open
shared = SharedCamera.from_config(
{
"class_path": "physicalai.capture.UVCCamera",
"init_args": {"device": "/dev/video0", "backend": "v4l2"},
},
)
# Equivalent after to_config(disconnected_driver):
# shared = SharedCamera.from_config(to_config(driver))
shared.connect()

# Safe to read from multiple threads/processes
frame = shared.read_latest()
```

`create_camera(..., shared=True)` remains a convenience for shareable built-ins
(`uvc`, `realsense`, `basler`) that packs a type into `SharedCamera(camera=...)`.

`SharedCamera` is the recommended approach for production deployments where multiple consumers need camera frames. It avoids the need for manual synchronization and handles frame distribution efficiently.
44 changes: 24 additions & 20 deletions docs/how-to/runtime/share-a-robot.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
while any number of other processes read its state and send actions over
[Zenoh](https://zenoh.io/). It satisfies the same `Robot` protocol as a
direct driver, so it is a drop-in replacement anywhere a robot is expected
(including `PolicyRuntime`).
(including `RobotRuntime`).

## Install

Expand All @@ -17,21 +17,24 @@ pip install "physicalai[transport]"
Every `SharedRobot` has a required, caller-chosen logical `name` — it keys
the Zenoh topics directly. The first `SharedRobot` constructed for a given
`name` that finds no existing owner spawns one (in a detached subprocess);
later instances (same or different process, same `name`) attach to it:
later instances (same or different process, same `name`) attach to it.

Construction is ComponentConfig-only (`robot=` / `from_config` / `from_robot`).
Prefer `from_config` when you already have a recipe; `from_robot` is sugar
that exports a disconnected `@export_config` driver:

```python
import numpy as np
from physicalai.robot import SharedRobot
from physicalai.robot.so101 import SO101

robot = SharedRobot(
"left-arm",
robot_class=SO101,
robot_kwargs={
"port": "/dev/ttyUSB0",
"calibration": "~/.cache/calibration/so101.json", # a path — kwargs must be serializable
},
from physicalai.config import to_config
from physicalai.robot import SO101, SharedRobot

driver = SO101(
port="/dev/ttyUSB0",
calibration="~/.cache/calibration/so101.json", # path stays relative/as given
)
robot = SharedRobot.from_config(to_config(driver), name="left-arm")
# or: SharedRobot.from_robot(driver, name="left-arm")
# or: SharedRobot("left-arm", robot={"class_path": "physicalai.robot.SO101", "init_args": {...}})
robot.connect()

obs = robot.get_observation() # pull latest state, non-blocking
Expand All @@ -40,10 +43,9 @@ robot.send_action(np.asarray(obs.joint_positions), goal_time=0.1)
robot.disconnect() # detaches; the owner keeps running
```

`robot_class` can be the class object (normalized to its dotted import path)
or the path itself, e.g. `robot_class="physicalai.robot.so101.SO101"` — any
importable class works, including third-party plugin robots, with no
registry to update.
Any importable `@export_config` robot class works (including third-party
plugins) — pass its public `class_path` + `init_args`; there is no flat
`robot_class` / `robot_kwargs` API.

## Serve a robot in the foreground

Expand Down Expand Up @@ -145,10 +147,12 @@ loopback port from `name`.
Opt into cross-host reachability explicitly when you need it:

```python
robot = SharedRobot(
"left-arm",
robot_class=SO101,
robot_kwargs={"port": "/dev/ttyUSB0", "calibration": "calibration.json"},
robot = SharedRobot.from_config(
{
"class_path": "physicalai.robot.SO101",
"init_args": {"port": "/dev/ttyUSB0", "calibration": "calibration.json"},
},
name="left-arm",
allow_remote=True,
)
```
Expand Down
12 changes: 7 additions & 5 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,13 @@ with runtime:
physicalai robot serve --config examples/so101/serve.yaml
```

The flat YAML fields are `name`, `robot_class`, optional `robot_kwargs`, optional
`allow_remote` (default `false`), and optional positive `rate_hz` (default 100 Hz).
Direct CLI arguments use the same names, including nested constructor values such as
`--robot_kwargs.port /dev/ttyACM0`. Serving stays in the foreground until SIGINT or
SIGTERM and returns nonzero for startup, loop, repeated-read, or disconnect failures.
The YAML fields are `name`, required `robot` (a ComponentConfig with
`class_path` + `init_args`), optional `allow_remote` (default `false`), and
optional positive `rate_hz` (default 100 Hz). Direct CLI arguments use the same
names, including nested robot constructor values such as
`--robot.init_args.port /dev/ttyACM0`. Flat `robot_class` / `robot_kwargs` are
unsupported. Serving stays in the foreground until SIGINT or SIGTERM and returns
nonzero for startup, loop, repeated-read, or disconnect failures.

Use `--verbose` to include driver construction, lock acquisition, initial observation,
endpoint declaration, and cleanup details.
Expand Down
8 changes: 4 additions & 4 deletions examples/capture/shared_camera.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"the camera, buffer management, or copy overhead.\n",
"\n",
"Because the transport layer is [iceoryx2](https://github.com/eclipse-iceoryx/iceoryx2),\n",
"any language with iceoryx2 bindings (Rust, C, C++, Python, ) can subscribe to\n",
"any language with iceoryx2 bindings (Rust, C, C++, Python, \u2026) can subscribe to\n",
"the same camera stream.\n",
"\n",
"**Why use SharedCamera:**\n",
Expand Down Expand Up @@ -81,16 +81,16 @@
"metadata": {},
"outputs": [],
"source": [
"from physicalai.capture import SharedCamera\n",
"from physicalai.capture import SharedCamera, create_camera\n",
"\n",
"# --- Edit these to match your setup ---\n",
"CAMERA_TYPE = CameraType.UVC\n",
"DEVICE_KWARGS = {\"device\": 0} # or {\"serial_number\": \"123456\"} for realsense/basler\n",
"# ---------------------------------------\n",
"\n",
"cam = SharedCamera(CAMERA_TYPE, zero_copy=True, **DEVICE_KWARGS)\n",
"cam: SharedCamera = create_camera(CAMERA_TYPE, shared=True, zero_copy=True, **DEVICE_KWARGS) # type: ignore[assignment]\n",
"cam.connect()\n",
"print(f\"Connected to {cam.device_id}\")"
"print(f\"Connected to {cam.device_id}\")\n"
]
},
{
Expand Down
52 changes: 16 additions & 36 deletions examples/runtime/runtime.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,21 +47,23 @@ runtime:
overhead: # scene/workspace view
class_path: physicalai.capture.SharedCamera
init_args:
camera_type: uvc
camera_kwargs:
device: /dev/video0 # CHANGE_ME: use /dev/v4l/by-id/... for stable identity
width: 640
height: 480
fps: 30
camera:
class_path: physicalai.capture.UVCCamera
init_args:
device: /dev/video0 # CHANGE_ME: use /dev/v4l/by-id/... for stable identity
width: 640
height: 480
fps: 30
gripper: # wrist-mounted view (rename to ``arm`` if the model expects that key)
class_path: physicalai.capture.SharedCamera
init_args:
camera_type: uvc
camera_kwargs:
device: /dev/video2 # CHANGE_ME
width: 640
height: 480
fps: 30
camera:
class_path: physicalai.capture.UVCCamera
init_args:
device: /dev/video2 # CHANGE_ME
width: 640
height: 480
fps: 30

fps: 30.0

Expand All @@ -71,32 +73,10 @@ runtime:
# mode: connect
# connect_addr: "127.0.0.1:9876"
# and start ``rerun`` separately.
#
# Cameras are declared again here so the callback can subscribe to the
# same shared-memory streams as the runtime. SharedCamera matches
# publishers by ``camera_type`` + ``camera_kwargs``, so keep these blocks
# in sync with the runtime ``cameras:`` section above.
# Camera frames come from each tick's ``TickEvent.camera_frames`` (same
# values the action source saw); no separate camera wiring on this callback.
- class_path: physicalai.runtime.RerunCallback
init_args:
cameras:
overhead:
class_path: physicalai.capture.SharedCamera
init_args:
camera_type: uvc
camera_kwargs:
device: /dev/video0 # CHANGE_ME: same as overhead camera above
width: 640
height: 480
fps: 30
gripper:
class_path: physicalai.capture.SharedCamera
init_args:
camera_type: uvc
camera_kwargs:
device: /dev/video2 # CHANGE_ME: same as gripper camera above
width: 640
height: 480
fps: 30
mode: spawn
log_images: true
image_decimation: 1 # log every Nth frame; raise to reduce viewer load
Expand Down
11 changes: 6 additions & 5 deletions examples/so101/serve.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@
# host B walkthrough (serve on the host with the physical robot attached,
# attach from anywhere else with a `SharedRobot` runtime config).
name: follower-arm
robot_class: physicalai.robot.SO101
robot_kwargs:
port: /dev/ttyACM0
calibration: /path/to/so101-calibration.json
role: follower
robot:
class_path: physicalai.robot.SO101
init_args:
port: /dev/ttyACM0
calibration: /path/to/so101-calibration.json
role: follower
allow_remote: false
rate_hz: 100.0
Loading
Loading