diff --git a/README.md b/README.md index 76853304..e1b3e6ca 100644 --- a/README.md +++ b/README.md @@ -10,26 +10,41 @@

- Robot Control for Strands Agents + Robot Control & Simulation for Strands Agents

PyPI Version GitHub stars License + MuJoCo GR00T LeRobot

Strands Docs + β—† MuJoCo β—† NVIDIA GR00T β—† LeRobot β—† Jetson Containers

-Control robots with natural language through [Strands Agents](https://github.com/strands-agents/sdk-python). Integrates [NVIDIA Isaac GR00T](https://github.com/NVIDIA/Isaac-GR00T) for vision-language-action policies and [LeRobot](https://github.com/huggingface/lerobot) for universal robot support. +Control and simulate robots with natural language through [Strands Agents](https://github.com/strands-agents/sdk-python). Simulate 60+ robots in MuJoCo, run policies, record LeRobot datasets, and deploy to real hardware β€” all from the same API. + +## The 5-Line Promise + +```python +from strands_robots import Robot +from strands import Agent + +robot = Robot("so100") # MuJoCo sim, auto-downloads assets +agent = Agent(tools=[robot]) # 64 simulation actions as AgentTool +agent("Pick up the red cube") # Agent orchestrates sim via natural language +``` + +That's it. `Robot("so100")` auto-detects simulation mode, downloads the MJCF model from [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie), builds a physics scene with ground plane and lighting, and exposes **64 actions** (step, render, run_policy, record, randomize, ...) as a Strands AgentTool. ## How It Works @@ -37,11 +52,14 @@ Control robots with natural language through [Strands Agents](https://github.com graph LR A[Natural Language
'Pick up the red block'] --> B[Strands Agent] B --> C[Robot Tool] - C --> D[Policy Provider
GR00T/Mock] - C --> E[LeRobot
Hardware Abstraction] - D --> F[Action Chunk
16 timesteps] - F --> E - E --> G[Robot Hardware
SO-101/GR-1/G1] + C --> D{Mode?} + D -->|Simulation| E[MuJoCo Backend
64 actions] + D -->|Hardware| F[LeRobot
Real Robot] + E --> G[Policy Provider
Mock / GR00T / LeRobot] + F --> G + G --> H[Action Chunks
Joint positions] + H --> E + H --> F classDef input fill:#2ea44f,stroke:#1b7735,color:#fff classDef agent fill:#0969da,stroke:#044289,color:#fff @@ -49,443 +67,481 @@ graph LR classDef hardware fill:#bf8700,stroke:#875e00,color:#fff class A input - class B,C agent - class D,F policy - class E,G hardware + class B,C,D agent + class E,F hardware + class G,H policy ``` -## Architecture +## Installation -```mermaid -flowchart TB - subgraph Agent["πŸ€– Strands Agent"] - NL[Natural Language Input] - Tools[Tool Registry] - end - - subgraph RobotTool["🦾 Robot Tool"] - direction TB - RT[Robot Class] - TM[Task Manager] - AS[Async Executor] - end - - subgraph Policy["🧠 Policy Layer"] - direction TB - PA[Policy Abstraction] - GP[GR00T Policy] - MP[Mock Policy] - CP[Custom Policy] - end - - subgraph Inference["⚑ Inference Service"] - direction TB - DC[Docker Container] - ZMQ[ZMQ Server :5555] - TRT[TensorRT Engine] - end - - subgraph Hardware["πŸ”§ Hardware Layer"] - direction TB - LR[LeRobot] - CAM[Cameras] - SERVO[Feetech Servos] - end - - NL --> Tools - Tools --> RT - RT --> TM - TM --> AS - AS --> PA - PA --> GP - PA --> MP - PA --> CP - GP --> ZMQ - ZMQ --> TRT - TRT --> DC - AS --> LR - LR --> CAM - LR --> SERVO - - classDef agentStyle fill:#0969da,stroke:#044289,color:#fff - classDef robotStyle fill:#2ea44f,stroke:#1b7735,color:#fff - classDef policyStyle fill:#8250df,stroke:#5a32a3,color:#fff - classDef infraStyle fill:#bf8700,stroke:#875e00,color:#fff - classDef hwStyle fill:#d73a49,stroke:#a72b3a,color:#fff - - class NL,Tools agentStyle - class RT,TM,AS robotStyle - class PA,GP,MP,CP policyStyle - class DC,ZMQ,TRT infraStyle - class LR,CAM,SERVO hwStyle +```bash +pip install strands-robots +``` + +### With simulation (MuJoCo) + +```bash +pip install "strands-robots[sim-mujoco]" +``` + +### With everything + +```bash +pip install "strands-robots[all]" ``` +| Extra | What it adds | When you need it | +|-------|-------------|------------------| +| `sim` | `robot_descriptions` | Robot model descriptors (URDF/meshes) | +| `sim-mujoco` | `mujoco`, `imageio`, `imageio-ffmpeg` (includes `sim`) | MuJoCo simulation runtime | +| `lerobot` | `lerobot>=0.5` | LeRobot policy inference + dataset recording | +| `groot-service` | `pyzmq`, `msgpack` | NVIDIA GR00T inference | +| `mesh` | `eclipse-zenoh` | Peer-to-peer mesh networking | +| `all` | All of the above | Full development | + ## Quick Start +### Simulation (no hardware needed) + +```python +from strands_robots import Robot + +# Create simulation β€” auto-downloads robot model +sim = Robot("unitree_g1") + +# Step physics +sim.step(n_steps=100) + +# Render a frame +frame = sim.render(width=640, height=480) # returns dict with PNG bytes + +# Run a policy +sim.run_policy( + robot_name="unitree_g1", + policy_provider="mock", + instruction="walk forward", + duration=5.0, + video={"path": "/tmp/g1_walk.mp4", "fps": 30}, +) + +sim.destroy() +``` + +### Agent-Driven Simulation + +```python +from strands_robots import Robot +from strands import Agent + +robot = Robot("so100") +agent = Agent(tools=[robot]) + +# The agent figures out the tool calls +agent(""" +1. Add a red box at [0.3, 0, 0.05] +2. Run mock policy for 3 seconds to pick it up +3. Record video to /tmp/demo.mp4 +4. Show me the final state +""") +``` + +### Dataset Recording (LeRobot v3 format) + +```python +from strands_robots import Robot + +sim = Robot("so100") + +# Start recording to LeRobot dataset +sim.start_recording( + repo_id="my-org/so100-pick-cube", + task="pick up the red cube", + fps=30, + root="/tmp/my_dataset", +) + +# Run policy β€” frames auto-captured +sim.run_policy( + robot_name="so100", + policy_provider="mock", + instruction="pick up the red cube", + duration=5.0, + fast_mode=True, +) + +# Save episode +sim.stop_recording() +sim.destroy() + +# Output: /tmp/my_dataset/ +# meta/info.json β€” LeRobot v3 metadata +# meta/tasks.parquet β€” task descriptions +# data/chunk-000/ β€” observation.state + action parquet +``` + +### Real Hardware + ```python from strands import Agent from strands_robots import Robot, gr00t_inference -# Create robot with cameras +# Create robot with cameras (new-style factory API) robot = Robot( - tool_name="my_arm", - robot="so101_follower", + "so101", + mode="real", cameras={ "front": {"type": "opencv", "index_or_path": "/dev/video0", "fps": 30}, - "wrist": {"type": "opencv", "index_or_path": "/dev/video2", "fps": 30} + "wrist": {"type": "opencv", "index_or_path": "/dev/video2", "fps": 30}, }, - port="/dev/ttyACM0", - data_config="so100_dualcam" + data_config="so100_dualcam", ) -# Create agent with robot tool agent = Agent(tools=[robot, gr00t_inference]) -# Start GR00T inference service +# Start GR00T inference agent.tool.gr00t_inference( action="start", checkpoint_path="/data/checkpoints/model", port=8000, - data_config="so100_dualcam" + data_config="so100_dualcam", ) -# Control robot with natural language +# Natural language control agent("Use my_arm to pick up the red block using GR00T policy on port 8000") ``` -## Installation +## Architecture -```bash -pip install strands-robots ``` - -From source: - -```bash -git clone https://github.com/strands-labs/robots -cd robots -pip install -e . + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Strands Agent β”‚ + β”‚ (natural language in) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Robot Factory β”‚ + β”‚ Robot("so100") dispatchesβ”‚ + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β” β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Simulation β”‚ β”‚ HardwareRobot β”‚ + β”‚ (MuJoCo) β”‚ β”‚ (LeRobot) β”‚ + β”‚ 64 actions β”‚ β”‚ real servos β”‚ + β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Policy Layer β”‚ + β”‚ mock β”‚ groot β”‚ lerobot_local β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Dataset Recorder β”‚ + β”‚ LeRobot v3 parquet + video β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` -
-🐳 Jetson Container Setup (Required for GR00T Inference) +## Mesh Networking -GR00T inference requires the Isaac-GR00T Docker container on Jetson platforms: +Every `Robot()` and `Simulation()` is automatically a peer on a peer-to-peer +[Zenoh](https://zenoh.io) mesh. Robots, simulations, and agents on the same +LAN discover each other with zero configuration; cross-network discovery +works through any Zenoh router or Cloudflare/Tailscale tunnel. -```bash -# Clone jetson-containers -git clone https://github.com/dusty-nv/jetson-containers -cd jetson-containers +```python +from strands_robots import Robot -# Run Isaac GR00T container (background) -jetson-containers run $(autotag isaac-gr00t) & +# Each Robot() auto-joins the mesh β€” no extra setup +sim_a = Robot("so100") # process A +sim_b = Robot("so100") # process B (same machine or remote) -# Container exposes inference service on port 5555 (ZMQ) or 8000 (HTTP) +print(sim_a.mesh.peers) # discovers sim_b within ~1s +sim_a.mesh.tell(sim_b.mesh.peer_id, "pick up the cube") +sim_a.mesh.emergency_stop() # broadcast E-STOP, audited to ~/.strands_robots/mesh_audit.jsonl ``` -**Tested Hardware:** -- NVIDIA Thor Dev Kit (Jetpack 7.0) -- NVIDIA Jetson AGX Orin (Jetpack 6.x) +### What every peer publishes -See [Jetson Deployment Guide](https://github.com/NVIDIA/Isaac-GR00T/blob/main/deployment_scripts/README.md) for TensorRT optimization. +``` +strands/{peer_id}/presence β€” 2 Hz heartbeat (peer discovery) +strands/{peer_id}/state β€” 10 Hz joints / sim time / task status +strands/{peer_id}/cmd β€” incoming RPC commands +strands/{peer_id}/response/{turn}β€” RPC replies (turn_id correlated) +strands/{peer_id}/stream β€” VLA execution steps (publish_step) +strands/{peer_id}/pose β€” SE(3) pose from SLAM/odometry/VIO +strands/{peer_id}/imu β€” Roll/pitch/yaw, gyro, accel +strands/{peer_id}/odom β€” Dead-reckoning odometry +strands/{peer_id}/health β€” Battery, CPU, memory, temps +strands/{peer_id}/lidar/summary β€” Point cloud stats +strands/{peer_id}/hand/{name}/state β€” End-effector joints / force +strands/{peer_id}/map/info β€” Map metadata +strands/{peer_id}/input/{device} β€” Teleoperator action stream +strands/broadcast β€” Fan-out RPC to every peer +``` -
+Sensor topics auto-publish only when the host robot exposes the relevant +attribute (e.g. `robot._imu`, `robot._lidar_summary`) β€” zero cost when unused. -## Robot Control Flow +### Mesh-aware agent tool -```mermaid -sequenceDiagram - participant User - participant Agent as Strands Agent - participant Robot as Robot Tool - participant Policy as GR00T Policy - participant HW as Hardware - - User->>Agent: "Pick up the red block" - Agent->>Robot: execute(instruction, policy_port) - - loop Control Loop @ 50Hz - Robot->>HW: get_observation() - HW-->>Robot: {cameras, joint_states} - Robot->>Policy: get_actions(obs, instruction) - Policy-->>Robot: action_chunk[16] - - loop Action Horizon - Robot->>HW: send_action(action) - Note over Robot,HW: 20ms sleep (50Hz) - end - end - - Robot-->>Agent: Task completed - Agent-->>User: "βœ… Picked up red block" -``` +The `robot_mesh` tool exposes the mesh to a Strands agent through 10 actions: -## Tools Reference +| Action | What it does | +|--------|--------------| +| `peers` | List local + remote peers | +| `status` | One-line mesh summary | +| `tell` | Natural-language instruction to a specific peer | +| `send` | Raw JSON RPC command to a peer | +| `broadcast` | Fan-out RPC to every peer | +| `stop` | Stop a single peer's running task | +| `emergency_stop` | Broadcast E-STOP (audited to disk) | +| `subscribe` | Subscribe to any Zenoh topic (wildcards supported) | +| `watch` | Watch another peer's VLA execution stream | +| `inbox` | Read buffered messages from a subscription | -### Robot Tool +```python +from strands import Agent +from strands_robots import Robot +from strands_robots.tools import robot_mesh -The `Robot` class is a Strands AgentTool that provides async robot control with real-time status reporting. +sim = Robot("so100") +agent = Agent(tools=[sim, robot_mesh]) +agent("Find every robot on the mesh and ask each one to report its status") +``` -| Action | Parameters | Description | Example | -|--------|------------|-------------|---------| -| `execute` | `instruction`, `policy_port`, `duration` | Blocking execution until complete | `"Pick up the cube"` | -| `start` | `instruction`, `policy_port`, `duration` | Non-blocking async start | `"Wave your arm"` | -| `status` | - | Get current task status | Check progress | -| `stop` | - | Interrupt running task | Emergency stop | +### Teleoperation over the mesh -**Natural Language Examples:** +Stream a leader arm's joint positions to a follower on another machine via +`InputPublisher` / `InputReceiver`: ```python -# Blocking execution (waits for completion) -agent("Use my_arm to pick up the red block using GR00T policy on port 8000") +from strands_robots.mesh import InputPublisher, InputReceiver -# Async execution (returns immediately) -agent("Start my_arm waving using GR00T on port 8000, then check status") +# Machine A β€” leader arm publishes at 50 Hz +leader = Robot("so100", mode="real") +pub = InputPublisher(leader.mesh, leader_teleoperator, device_name="leader") +pub.start() -# Stop running task -agent("Stop my_arm immediately") +# Machine B β€” follower receives + applies actions +follower = Robot("so100", mode="real") +rec = InputReceiver(follower.mesh, follower.robot, source_peer_id=leader.mesh.peer_id) +rec.start() ``` -
-Robot Constructor Parameters - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `tool_name` | `str` | required | Name for this robot tool | -| `robot` | `str\|RobotConfig` | required | Robot type or config | -| `cameras` | `Dict` | `None` | Camera configuration | -| `port` | `str` | `None` | Serial port for robot | -| `data_config` | `str` | `None` | GR00T data config name | -| `control_frequency` | `float` | `50.0` | Control loop Hz | -| `action_horizon` | `int` | `8` | Actions per inference | +Topic schema for `strands/{peer_id}/input/{device}`: + +```json +{ + "peer_id": "leader-a1b2c3d4", + "device": "leader", + "method": "arm", + "t": 1736975234.123, + "seq": 42, + "action": {"shoulder.pos": 1.23, "elbow.pos": -0.5, "gripper.pos": 0.0}, + "events": {"terminate_episode": false} +} +``` -
+### Safety: emergency stop with audit log ---- +`mesh.emergency_stop()` broadcasts `{"action": "stop"}` to every peer and +appends a tamper-evident record to `~/.strands_robots/mesh_audit.jsonl` +(file mode `0o600`, parent dir `0o700`). Override the location with +`STRANDS_MESH_AUDIT_DIR`. -### GR00T Inference Tool +### Disable + +| How | Scope | +|-----|-------| +| `STRANDS_MESH=false` | Process-wide kill switch | +| `Robot("so100", mesh=False)` | Per-robot opt-out | + +Mesh networking requires the `[mesh]` extra (or `[all]`): + +```bash +pip install "strands-robots[mesh]" +``` + +The base install (`pip install strands-robots`) does **not** include Zenoh. + +## Simulation Features -Manages GR00T policy inference services running in Docker containers. +The MuJoCo simulation backend exposes **64 actions** as a Strands AgentTool: -| Action | Parameters | Description | Example | -|--------|------------|-------------|---------| -| `start` | `checkpoint_path`, `port`, `data_config` | Start inference service | `"Start GR00T on port 8000"` | -| `stop` | `port` | Stop service on port | `"Stop GR00T on port 8000"` | -| `status` | `port` | Check service status | `"Is GR00T running?"` | -| `list` | - | List all running services | `"List inference services"` | -| `find_containers` | - | Find GR00T containers | `"Find available containers"` | +| Category | Actions | +|----------|---------| +| **World** | `create_world`, `load_scene`, `reset`, `get_state`, `destroy` | +| **Robots** | `add_robot`, `remove_robot`, `list_robots`, `get_robot_state` | +| **Objects** | `add_object`, `remove_object`, `move_object`, `list_objects` | +| **Cameras** | `add_camera`, `remove_camera` | +| **Policies** | `run_policy`, `start_policy`, `stop_policy`, `eval_policy`, `replay_episode` | +| **Rendering** | `render`, `render_depth`, `open_viewer`, `close_viewer` | +| **Physics** | `step`, `set_gravity`, `set_timestep`, `get_contacts` | +| **Recording** | `start_recording`, `stop_recording`, `get_recording_status` | +| **Randomization** | `randomize` (colors, physics, lighting, cameras) | +| **Assets** | `list_urdfs`, `register_urdf`, `get_features` | -**TensorRT Acceleration:** +### Supported Robots (60+ robots, 120+ aliases) + +Any robot in the registry works in simulation. Assets auto-download from MuJoCo Menagerie on first use. ```python -agent.tool.gr00t_inference( - action="start", - checkpoint_path="/data/checkpoints/model", - port=8000, - use_tensorrt=True, - trt_engine_path="gr00t_engine", - vit_dtype="fp8", # ViT: fp16 or fp8 - llm_dtype="nvfp4", # LLM: fp16, nvfp4, or fp8 - dit_dtype="fp8" # DiT: fp16 or fp8 -) -``` +from strands_robots import list_robots ---- +# List all simulation-capable robots +for r in list_robots(): + print(f"{r['name']}: {r['description']}") +``` -### Camera Tool +**Key robots tested**: `so100` (6-DOF arm), `unitree_g1` (30 joints), `panda` (Franka), `unitree_h1` (humanoid), `aloha` (bimanual). -LeRobot-based camera management with OpenCV and RealSense support. +### Domain Randomization -| Action | Parameters | Description | Example | -|--------|------------|-------------|---------| -| `discover` | - | Find all cameras | `"Discover cameras"` | -| `capture` | `camera_id`, `save_path` | Single image capture | `"Capture from /dev/video0"` | -| `capture_batch` | `camera_ids`, `async_mode` | Multi-camera capture | `"Capture from all cameras"` | -| `record` | `camera_id`, `capture_duration` | Record video | `"Record 10s video"` | -| `preview` | `camera_id`, `preview_duration` | Live preview | `"Preview camera 0"` | -| `test` | `camera_id` | Performance test | `"Test camera speed"` | +```python +sim.randomize( + target="colors", # or "physics", "lighting", "camera", "all" + robot_name="so100", +) +``` ---- +### Policy Evaluation -### Serial Tool +```python +result = sim.eval_policy( + robot_name="so100", + policy_provider="mock", + instruction="pick up the cube", + num_episodes=10, + max_steps_per_episode=200, +) +# Returns success rate, mean reward, per-episode stats +``` -Low-level serial communication for Feetech servos and custom protocols. +## Policy Providers -| Action | Parameters | Description | Example | -|--------|------------|-------------|---------| -| `list_ports` | - | Discover serial ports | `"List serial ports"` | -| `feetech_position` | `port`, `motor_id`, `position` | Move servo | `"Move motor 1 to center"` | -| `feetech_ping` | `port`, `motor_id` | Ping servo | `"Ping motor 1"` | -| `send` | `port`, `data/hex_data` | Send raw data | `"Send FF FF to robot"` | -| `monitor` | `port` | Monitor serial data | `"Monitor /dev/ttyACM0"` | +| Provider | Description | Requirements | +|----------|-------------|-------------| +| `mock` | Sinusoidal test actions | None | +| `groot` | NVIDIA GR00T N1.5/N1.6 | `[groot-service]` + inference container | +| `lerobot_local` | HuggingFace LeRobot direct inference | `[lerobot]` + model weights | ---- +```python +from strands_robots.policies.factory import create_policy -### Teleoperation Tool +# Mock (for testing β€” no deps) +policy = create_policy(provider="mock") -Record demonstrations for imitation learning with LeRobot. +# GR00T (requires inference server) +policy = create_policy(provider="groot", host="localhost", port=8000, data_config="so100_dualcam") -| Action | Parameters | Description | Example | -|--------|------------|-------------|---------| -| `start` | `robot_type`, `teleop_type` | Start teleoperation | `"Start teleoperation"` | -| `stop` | `session_name` | Stop session | `"Stop recording"` | -| `list` | - | List active sessions | `"List teleop sessions"` | -| `replay` | `dataset_repo_id`, `replay_episode` | Replay episode | `"Replay episode 5"` | +# LeRobot local (direct inference) +policy = create_policy(provider="lerobot_local", policy_path="lerobot/act_so100_pick") +``` ---- +## Tools Reference -### Pose Tool +### Robot Tool (Simulation Mode) -Store, retrieve, and execute named robot poses. +When `Robot("name")` detects simulation mode, it creates a MuJoCo `Simulation` with 64 actions accessible via natural language or direct calls. -| Action | Parameters | Description | Example | -|--------|------------|-------------|---------| -| `store_pose` | `pose_name` | Save current position | `"Save as 'home'"` | -| `load_pose` | `pose_name` | Move to saved pose | `"Go to home pose"` | -| `list_poses` | - | List all poses | `"List saved poses"` | -| `move_motor` | `motor_name`, `position` | Move single motor | `"Move gripper to 50%"` | -| `incremental_move` | `motor_name`, `delta` | Small movement | `"Move elbow +5Β°"` | -| `reset_to_home` | - | Safe home position | `"Reset to home"` | +### Robot Tool (Hardware Mode) ---- +| Action | Description | +|--------|-------------| +| `execute` | Blocking policy execution until complete | +| `start` | Non-blocking async start | +| `status` | Get current task status | +| `stop` | Emergency stop | -## Supported Robots +### GR00T Inference Tool -| Robot | Config | Cameras | Description | -|-------|--------|---------|-------------| -| SO-100/SO-101 | `so100`, `so100_dualcam`, `so100_4cam` | 1-4 | Single arm desktop robot | -| Fourier GR-1 | `fourier_gr1_arms_only` | 1 | Bimanual humanoid arms | -| Bimanual Panda | `bimanual_panda_gripper` | 3 | Dual Franka Emika arms | -| Unitree G1 | `unitree_g1` | 1 | Humanoid robot platform | +| Action | Description | +|--------|-------------| +| `start` | Start GR00T inference service (Docker) | +| `stop` | Stop inference service | +| `status` | Check service health | +| `list` | List running services |
-GR00T Data Configurations +TensorRT Acceleration -| Config | Video Keys | State Keys | Description | -|--------|------------|------------|-------------| -| `so100` | `video.webcam` | `state.single_arm`, `state.gripper` | Single camera | -| `so100_dualcam` | `video.front`, `video.wrist` | `state.single_arm`, `state.gripper` | Front + wrist | -| `so100_4cam` | `video.front`, `video.wrist`, `video.top`, `video.side` | `state.single_arm`, `state.gripper` | Quad camera | -| `fourier_gr1_arms_only` | `video.ego_view` | `state.left_arm`, `state.right_arm`, `state.left_hand`, `state.right_hand` | Humanoid arms | -| `bimanual_panda_gripper` | `video.right_wrist_view`, `video.left_wrist_view`, `video.front_view` | EEF pos/quat + gripper | Dual arm EEF | -| `unitree_g1` | `video.rs_view` | `state.left_arm`, `state.right_arm`, `state.left_hand`, `state.right_hand` | G1 humanoid | +```python +agent.tool.gr00t_inference( + action="start", + checkpoint_path="/data/checkpoints/model", + port=8000, + use_tensorrt=True, + trt_engine_path="gr00t_engine", + vit_dtype="fp8", + llm_dtype="nvfp4", + dit_dtype="fp8", +) +```
-## Policy Providers +### Additional Tools -```mermaid -classDiagram - class Policy { - <> - +get_actions(observation, instruction) - +set_robot_state_keys(keys) - +provider_name - } - - class Gr00tPolicy { - +data_config - +policy_client: ZMQ - +get_actions() - } - - class MockPolicy { - +get_actions() - Returns random actions - } - - class CustomPolicy { - +get_actions() - Your implementation - } - - Policy <|-- Gr00tPolicy - Policy <|-- MockPolicy - Policy <|-- CustomPolicy -``` +| Tool | Description | +|------|-------------| +| `lerobot_camera` | Camera discovery, capture, recording (OpenCV + RealSense) | +| `lerobot_calibrate` | Motor calibration management | +| `lerobot_teleoperate` | Record demonstrations for imitation learning | +| `pose_tool` | Store, retrieve, execute named robot poses | +| `serial_tool` | Low-level Feetech servo communication | -```python -from strands_robots import create_policy +
+🐳 Jetson Container Setup (for GR00T Inference) -# GR00T policy (requires inference server) -policy = create_policy( - provider="groot", - data_config="so100_dualcam", - host="localhost", - port=8000 -) +GR00T inference requires the Isaac-GR00T Docker container on Jetson platforms: -# Mock policy (for testing) -policy = create_policy(provider="mock") +```bash +git clone https://github.com/dusty-nv/jetson-containers +cd jetson-containers +jetson-containers run $(autotag isaac-gr00t) & ``` -## Project Structure +**Tested Hardware:** +- NVIDIA Thor Dev Kit (Jetpack 7.0) +- NVIDIA Jetson AGX Orin (Jetpack 6.x) -``` -strands-robots/ -β”œβ”€β”€ strands_robots/ -β”‚ β”œβ”€β”€ __init__.py # Package exports -β”‚ β”œβ”€β”€ robot.py # Universal Robot class (AgentTool) -β”‚ β”œβ”€β”€ policies/ -β”‚ β”‚ β”œβ”€β”€ __init__.py # Policy ABC + factory -β”‚ β”‚ └── groot/ -β”‚ β”‚ β”œβ”€β”€ __init__.py # Gr00tPolicy implementation -β”‚ β”‚ β”œβ”€β”€ client.py # ZMQ inference client -β”‚ β”‚ └── data_config.py # Robot embodiment configurations -β”‚ └── tools/ -β”‚ β”œβ”€β”€ gr00t_inference.py # Docker service manager -β”‚ β”œβ”€β”€ lerobot_camera.py # Camera operations -β”‚ β”œβ”€β”€ lerobot_calibrate.py # Calibration management -β”‚ β”œβ”€β”€ lerobot_teleoperate.py # Recording/replay -β”‚ β”œβ”€β”€ pose_tool.py # Pose management -β”‚ └── serial_tool.py # Serial communication -β”œβ”€β”€ test.py # Integration example -└── pyproject.toml # Package configuration -``` +See [Jetson Deployment Guide](https://github.com/NVIDIA/Isaac-GR00T/blob/main/deployment_scripts/README.md) for TensorRT optimization. -## Example: Complete Workflow +
-```python -#!/usr/bin/env python3 -from strands import Agent -from strands_robots import Robot, gr00t_inference, lerobot_camera, pose_tool +## GR00T Data Configurations -# 1. Create robot with dual cameras -robot = Robot( - tool_name="orange_arm", - robot="so101_follower", - cameras={ - "wrist": {"type": "opencv", "index_or_path": "/dev/video0", "fps": 15}, - "front": {"type": "opencv", "index_or_path": "/dev/video2", "fps": 15}, - }, - port="/dev/ttyACM0", - data_config="so100_dualcam", -) +| Config | Video Keys | Description | +|--------|------------|-------------| +| `so100` | `video.webcam` | Single camera setup | +| `so100_dualcam` | `video.front`, `video.wrist` | Front + wrist cameras | +| `so100_4cam` | `video.front`, `video.wrist`, `video.top`, `video.side` | Quad camera | +| `fourier_gr1_arms_only` | `video.ego_view` | Humanoid bimanual arms | +| `bimanual_panda_gripper` | 3 camera views | Dual Franka Emika arms | +| `unitree_g1` | `video.rs_view` | G1 humanoid platform | -# 2. Create agent with all robot tools -agent = Agent( - tools=[robot, gr00t_inference, lerobot_camera, pose_tool] -) +## Development -# 3. Start inference service -agent.tool.gr00t_inference( - action="start", - checkpoint_path="/data/checkpoints/gr00t-wave/checkpoint-300000", - port=8000, - data_config="so100_dualcam", -) +```bash +git clone https://github.com/strands-labs/robots +cd robots + +# Create environment +uv venv --python 3.12 .venv +source .venv/bin/activate -# 4. Interactive control loop -while True: - user_input = input("\nπŸ€– > ") - if user_input.lower() in ["exit", "quit"]: - break - agent(user_input) +# Install with simulation + dev tools +uv pip install -e ".[sim,dev]" -# 5. Cleanup -agent.tool.gr00t_inference(action="stop", port=8000) +# Run tests (34 tests, ~1s) +uv run pytest tests/ -v + +# Lint +uv run ruff check . +uv run ruff format --check . ``` +See [AGENTS.md](AGENTS.md) for detailed testing guide, manual E2E validation scripts, and contribution workflow. + ## Configuration ### Environment Variables @@ -496,15 +552,21 @@ agent.tool.gr00t_inference(action="stop", port=8000) | `STRANDS_ROBOT_MODE` | Default mode for `Robot()` factory: `sim` / `real` / `auto` | `sim` | | `STRANDS_TRUST_REMOTE_CODE` | Allow downloading + executing model code | `false` | | `MUJOCO_GL` | GL backend for the MuJoCo renderer | auto | -| `STRANDS_MESH` | Set to `false` to disable Zenoh mesh networking globally | `true` | -| `STRANDS_MESH_PORT` | TCP port for the local Zenoh router | `7447` | -| `ZENOH_CONNECT` | Comma-separated list of remote Zenoh endpoints to connect to | - | -| `ZENOH_LISTEN` | Comma-separated list of endpoints for the local Zenoh listener | - | -| `STRANDS_MESH_AUDIT_DIR` | Directory for the safety audit log (`mesh_audit.jsonl`) | `~/.strands_robots/` | | `GROOT_API_TOKEN` | API token for GR00T inference service | - | -| `STRANDS_ROBOT_MODE` | Override `Robot()` factory mode detection (`sim`, `real`, `auto`) | `auto` | -| `STRANDS_TRUST_REMOTE_CODE` | Set to `1` to opt into HuggingFace `trust_remote_code` for `lerobot_local` policies | unset | -| `MUJOCO_GL` | OpenGL backend for MuJoCo (`egl`, `osmesa`, `glfw`) | auto-detected | +| `STRANDS_MESH` | Set to `false` to disable Zenoh mesh networking globally | `true` | +| `STRANDS_MESH_PORT` | TCP port for the local Zenoh router (validated, falls back to default on bad input) | `7447` | +| `ZENOH_CONNECT` | Comma-separated remote Zenoh endpoints to connect to | - | +| `ZENOH_LISTEN` | Comma-separated endpoints for the local listener | - | +| `STRANDS_MESH_AUDIT_DIR` | Directory for the safety audit log `mesh_audit.jsonl` | `~/.strands_robots/` | +| `STRANDS_MESH_POSE_HZ` | Pose-loop frequency (0 disables) | `10.0` | +| `STRANDS_MESH_IMU_HZ` | IMU-loop frequency (0 disables) | `10.0` | +| `STRANDS_MESH_ODOM_HZ` | Odometry-loop frequency (0 disables) | `10.0` | +| `STRANDS_MESH_HEALTH_HZ` | Health-loop frequency (0 disables) | `0.5` | +| `STRANDS_MESH_LIDAR_SUMMARY_HZ` | LiDAR summary frequency | `5.0` | +| `STRANDS_MESH_LIDAR_STATE_HZ` | LiDAR state frequency | `1.0` | +| `STRANDS_MESH_HAND_HZ` | End-effector state frequency | `50.0` | +| `STRANDS_MESH_MAP_INFO_HZ` | Map metadata frequency | `0.2` | +| `STRANDS_MESH_CAMERA_HZ` | Camera-frame publish rate (0 disables; opt-in) | `0` | | `STRANDS_LIBERO_ACTION_LOG` | Set to `1` to emit per-step diagnostic logs from the LIBERO OSC controller (action keys, delta scale, EEF tracking, gripper polarity, qpos/ctrl deltas). Logs the first N steps per episode. | unset | | `STRANDS_LIBERO_ACTION_LOG_MAX` | Max number of `apply()` calls to log per episode when `STRANDS_LIBERO_ACTION_LOG=1`. | `50` | | `STRANDS_LIBERO_STATE_LOG` | Set to `1` to emit per-step diagnostic logs of the state values (`state.x/y/z/roll/pitch/yaw/gripper`) the LIBERO adapter feeds to the GR00T policy. Pairs with `STRANDS_LIBERO_ACTION_LOG` for end-to-end interface bisection. | unset | @@ -551,7 +613,7 @@ To change the cache location: `export STRANDS_ASSETS_DIR=/path/to/custom/dir` ## Simulation (MuJoCo) -`strands-robots` ships a MuJoCo-backed simulation AgentTool - 58 actions +`strands-robots` ships a MuJoCo-backed simulation AgentTool β€” 64 actions exposed to any Strands agent for world composition, physics, policy execution, and video/dataset recording. @@ -580,7 +642,7 @@ sim.run_policy(robot_name="arm", policy_provider="mock", n_steps=200, frame = sim.render(camera_name="topdown") # returns {status, content:[text, image]} ``` -### 58 actions grouped +### 64 actions grouped - **World & objects**: `create_world`, `load_scene`, `add_robot`, `add_object`, `move_object`, `list_objects`, `list_robots`, @@ -654,18 +716,19 @@ For the full action contract and test coverage see ## Contributing We welcome contributions! Please see: +- [AGENTS.md](AGENTS.md) for development guidelines - [GitHub Issues](https://github.com/strands-labs/robots/issues) for bug reports - [Pull Requests](https://github.com/strands-labs/robots/pulls) for contributions +- [Project Board](https://github.com/orgs/strands-labs/projects/2) for planned work ## License -Apache-2.0 - see [LICENSE](LICENSE) file. - -## Links +Apache-2.0 β€” see [LICENSE](LICENSE).
GitHub β—† PyPI + β—† MuJoCo β—† NVIDIA GR00T β—† LeRobot β—† Strands Docs diff --git a/examples/01_sim_quickstart.py b/examples/01_sim_quickstart.py new file mode 100644 index 00000000..850c1af6 --- /dev/null +++ b/examples/01_sim_quickstart.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 +"""Quickstart: 5 lines from import to simulation. + +The Robot() factory auto-detects mode (defaults to sim when no hardware +is connected) and returns a ready-to-use MuJoCo simulation. + +Requirements: + pip install strands-robots[sim-mujoco] + +Usage: + python examples/01_sim_quickstart.py +""" + +from strands_robots import Robot + +# Create a simulated SO-100 arm β€” assets auto-download from MuJoCo Menagerie +sim = Robot("so100") + +# Inspect the world +state = sim.get_state() +print(state["content"][0]["text"]) + +# Step physics and render a frame +sim.step(n_steps=100) +frame = sim.render(width=640, height=480) +print(f"Rendered frame: {frame['content'][0]['text']}") + +sim.destroy() +print("Done β€” simulation complete") diff --git a/examples/02_sim_agent.py b/examples/02_sim_agent.py new file mode 100644 index 00000000..7135fc1c --- /dev/null +++ b/examples/02_sim_agent.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python3 +"""The 5-Line Promise: natural language robot control. + +Robot() returns an AgentTool with 64 simulation actions. Hand it to a +Strands Agent and control the robot through conversation. + +Requirements: + pip install strands-agents strands-robots[sim-mujoco] + +Usage: + python examples/02_sim_agent.py +""" + +from strands import Agent + +from strands_robots import Robot + +# Robot("so100") auto-detects mode="sim", picks the "mujoco" backend, +# constructs a Simulation instance, calls add_robot() to load the SO-100 +# model (auto-downloading URDF/meshes on first run), and returns that +# Simulation as an AgentTool. You get full access to all Simulation +# actions β€” step(), render(), run_policy(), get_observation(), etc. +robot = Robot("so100") + +# The sim IS the tool β€” pass it directly to Agent +agent = Agent(tools=[robot]) + +# Natural language β†’ simulation actions +result = agent("Get the simulation state, then run a mock policy for 1 second in fast mode") +print(result) + +robot.destroy() diff --git a/examples/03_sim_recording.py b/examples/03_sim_recording.py new file mode 100644 index 00000000..0733751b --- /dev/null +++ b/examples/03_sim_recording.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""Record simulation rollouts as LeRobot v3 datasets. + +Runs a mock policy in MuJoCo, captures joint states + video, and saves +everything as a LeRobot-compatible dataset (parquet + AV1 video). + +Note: + ``start_recording`` produces a LeRobotDataset (parquet + per-camera + MP4), so you need the ``lerobot`` extra in addition to ``sim-mujoco``. + For plain MP4 only (no dataset schema), use ``start_cameras_recording``. + +Requirements: + pip install "strands-robots[sim-mujoco,lerobot]" + +Usage: + python examples/03_sim_recording.py +""" + +from strands_robots import Robot + +sim = Robot("so100") + +try: + # Start recording - creates LeRobot v3 dataset structure + sim.start_recording( + repo_id="local/so100_demo", + task="reach target", + fps=30, + root="/tmp/so100_dataset", + ) + + # Run a mock policy (random actions) for 2 seconds. + # Video kwargs go inside the ``video`` dict, NOT as top-level args. + result = sim.run_policy( + robot_name="so100", + policy_provider="mock", + instruction="reach target", + duration=2.0, + fast_mode=True, + video={"path": "/tmp/so100_rollout.mp4", "fps": 30}, + ) + print(result["content"][0]["text"]) + + # Finalize the episode + stop = sim.stop_recording() + print(stop["content"][0]["text"]) +finally: + sim.destroy() + +print("Dataset saved to /tmp/so100_dataset/") diff --git a/examples/04_real_hardware.py b/examples/04_real_hardware.py new file mode 100644 index 00000000..a17a38a0 --- /dev/null +++ b/examples/04_real_hardware.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +"""Control real robot hardware with the same factory API. + +Robot(mode="real") returns a HardwareRobot backed by LeRobot. The same +Agent workflow works β€” just swap the mode. + +Requirements: + pip install strands-agents strands-robots[lerobot] + # Hardware: SO-100/SO-101 arm connected via USB (Feetech servos) + +Usage: + # Auto-detect (switches to real if USB servo controller found) + STRANDS_ROBOT_MODE=real python examples/04_real_hardware.py + + # Or set mode explicitly in code (see below) +""" + +from strands import Agent + +from strands_robots import Robot + +# Explicit real mode with camera config +robot = Robot( + "so100", + mode="real", + cameras={ + "wrist": { + "type": "opencv", + "index_or_path": "/dev/video0", + "fps": 15, + "fourcc": "MJPG", + }, + }, +) + +# Same Agent interface as simulation +agent = Agent(tools=[robot]) +agent("Connect to the robot, read the current joint positions, and report status") diff --git a/examples/05_real_groot_policy.py b/examples/05_real_groot_policy.py new file mode 100644 index 00000000..0b67b0e4 --- /dev/null +++ b/examples/05_real_groot_policy.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Run NVIDIA GR00T policy on real hardware. + +Starts a GR00T inference server, connects to a real SO-101 arm with +dual cameras, and runs the policy through an Agent. + +Requirements: + pip install strands-agents strands-robots[all] + # Hardware: SO-101 arm + 2 USB cameras + # Model: Download from HuggingFace (e.g., cagataydev/gr00t-wave) + +Usage: + python examples/05_real_groot_policy.py +""" + +from strands import Agent + +from strands_robots import Robot, gr00t_inference, lerobot_camera, pose_tool + +# Initialize robot to None so finally block is safe if construction fails +robot = None +groot_started = False + +try: + # Real robot with dual cameras + robot = Robot( + "so101", + mode="real", + cameras={ + "wrist": { + "type": "opencv", + "index_or_path": "/dev/video0", + "fps": 15, + "fourcc": "MJPG", + }, + "front": { + "type": "opencv", + "index_or_path": "/dev/video2", + "fps": 15, + "fourcc": "MJPG", + }, + }, + ) + + # Build agent with robot + inference tools + agent = Agent( + tools=[robot, gr00t_inference, lerobot_camera, pose_tool], + ) + + # Start GR00T inference server + agent.tool.gr00t_inference( + action="start", + checkpoint_path="/data/checkpoints/gr00t-wave/checkpoint-300000", + port=5555, + data_config="so100_dualcam", + embodiment_tag="new_embodiment", + ) + groot_started = True + + # Interactive control loop + print("GR00T policy running. Type instructions or 'quit' to exit.") + while True: + query = input("\n# ") + if query.lower() in ("quit", "exit", "q"): + break + agent(query) + +finally: + # Always stop the inference server and release hardware, even on error. + if groot_started: + stop_result = gr00t_inference(action="stop", port=5555) + print(f"GR00T stop: {stop_result}") + if robot: + robot.cleanup() diff --git a/examples/06_list_robots.py b/examples/06_list_robots.py new file mode 100644 index 00000000..137648d3 --- /dev/null +++ b/examples/06_list_robots.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +"""List all supported robots and their capabilities. + +The registry contains 60+ robots with simulation assets and/or hardware +support. Use list_robots() to discover what's available. + +Requirements: + pip install strands-robots + +Usage: + python examples/06_list_robots.py +""" + +from strands_robots import list_robots + + +def _flag(present: bool, label: str) -> str: + return f"[{label}]" if present else f"[{' ' * len(label)}]" + + +print("=== All Robots ===") +for r in list_robots(mode="all"): + sim_flag = _flag(r.get("has_sim", False), "sim") + real_flag = _flag(r.get("has_real", False), "real") + print(f" {sim_flag} {real_flag} {r['name']:25s} {r.get('description', '')}") + +# Sim-capable: all robots that have simulation assets (includes those with +# hardware support too). For truly sim-only robots, filter out has_real. +sim_capable = list_robots(mode="sim") +sim_only = [r for r in sim_capable if not r.get("has_real", False)] +print(f"\n=== Sim-capable ({len(sim_capable)} robots, {len(sim_only)} sim-only) ===") +for r in sim_capable[:5]: + print(f" {r['name']}") + +print(f"\n=== Real hardware ({len(list_robots(mode='real'))} robots) ===") +for r in list_robots(mode="real"): + print(f" {r['name']}") diff --git a/examples/act_policy_simulation.py b/examples/act_policy_simulation.py new file mode 100755 index 00000000..77e3d5f1 --- /dev/null +++ b/examples/act_policy_simulation.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Run a HuggingFace ACT policy in MuJoCo and export a LeRobot dataset. + +Downloads a pretrained ACT policy, runs it in simulation, records multi-camera +video + joint data as a LeRobot v3 dataset. + +Requirements: + pip install "strands-robots[sim-mujoco,lerobot]" torch + +Usage: + python examples/act_policy_simulation.py +""" + +from strands_robots import Robot + +# 1. Create simulated Aloha bimanual robot (14 actuators, 6 cameras) +sim = Robot("aloha") + +# 2. Start recording a LeRobot dataset (parquet + AV1 video) +sim.start_recording( + repo_id="local/act_aloha_sim_demo", + task="transfer cube", + fps=50, + root="/tmp/act_aloha_dataset", +) + +# 3. Run a pretrained ACT policy from HuggingFace (~51M params). +# NOTE: This downloads model weights (~200MB) on first run. +# For a lightweight test, swap policy_provider="mock" and drop policy_config. +# Provider-specific kwargs (pretrained_name_or_path, device, ...) go inside +# ``policy_config``. Video output goes inside the ``video`` dict. +result = sim.run_policy( + robot_name="aloha", + policy_provider="lerobot_local", + policy_config={ + "pretrained_name_or_path": "lerobot/act_aloha_sim_transfer_cube_human", + }, + instruction="transfer cube", + duration=2.0, # seconds of sim time + fast_mode=True, # no wall-clock sleep between steps + video={"path": "/tmp/act_aloha_rollout.mp4", "fps": 30}, +) +print(result["content"][0]["text"]) + +# 4. Save the episode to disk +stop = sim.stop_recording() +print(stop["content"][0]["text"]) + +sim.destroy() diff --git a/examples/physics_agent.py b/examples/physics_agent.py new file mode 100644 index 00000000..e00eea96 --- /dev/null +++ b/examples/physics_agent.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Strands Agent with full MuJoCo physics introspection. + +Demonstrates the new Physics API β€” direct Python access to MuJoCo C functions: + mj_ray, mj_jacBody, mj_applyFT, mj_fullM, mj_inverse, mj_contactForce, + mj_getState/mj_setState, mj_energyPos/mj_energyVel, and more. + +An agent that can reason about physics: cast rays, compute Jacobians, +apply forces, checkpoint/restore state, read sensors, and analyze contacts +β€” all through natural language. + +Requirements: + pip install strands-agents strands-robots[sim-mujoco] + +Usage: + python examples/physics_agent.py +""" + +from strands import Agent + +from strands_robots import Robot + +# Create a simulated SO-100 robot arm +sim = Robot("so100") + +# Give the agent the simulation tool β€” all 64 actions available via NL +agent = Agent( + tools=[sim], + system_prompt=( + "You are a robotics physicist. You have a simulated SO-100 robot arm " + "in MuJoCo. Use the simulation tool to explore its physics. " + "Be concise and use real numbers from the simulation." + ), +) + +# --- Example 1: Full physics analysis in natural language -------------------- +print("=" * 70) +print("Example 1: Agent-driven physics analysis") +print("=" * 70) + +result = agent( + "Analyze the robot's physics: " + "1) Get the total mass breakdown, " + "2) Compute the mass matrix and tell me its condition number, " + "3) Read all sensor values, " + "4) Get the system energy. " + "Summarize the physical properties." +) +print(result) + +# --- Example 2: Raycasting for obstacle detection --------------------------- +print("\n" + "=" * 70) +print("Example 2: Agent uses raycasting for spatial reasoning") +print("=" * 70) + +result = agent( + "Cast rays downward from 5 points above the robot (height=1m) at " + "x=-0.2, -0.1, 0, 0.1, 0.2 (y=0) to map what's below. " + "Use multi_raycast for efficiency. Report the distance map." +) +print(result) + +# --- Example 3: State checkpointing + force experiments --------------------- +print("\n" + "=" * 70) +print("Example 3: Save state, experiment, then restore") +print("=" * 70) + +result = agent( + "I want to experiment with forces without breaking the sim: " + "1) Save the current state as 'pristine', " + "2) Step 200 times to let things settle, " + "3) Get the energy, " + "4) Apply a 50N upward force to the robot's end-effector body, " + "5) Step 100 more times, " + "6) Get the energy again and compare, " + "7) Restore the 'pristine' state, " + "8) Verify we're back by checking energy matches the original." +) +print(result) + +# --- Example 4: Jacobian + inverse dynamics --------------------------------- +print("\n" + "=" * 70) +print("Example 4: Dynamics analysis") +print("=" * 70) + +result = agent( + "Compute the Jacobian for the end-effector and run inverse dynamics. " + "What forces are needed at each joint to hold the current pose?" +) +print(result) + +# --- Example 5: Contact analysis -------------------------------------------- +print("\n" + "=" * 70) +print("Example 5: Contact force analysis") +print("=" * 70) + +result = agent( + "Step the simulation 500 times to let everything settle, " + "then get detailed contact forces. " + "Which bodies are in contact and what are the normal forces?" +) +print(result) + +# Clean up +sim.destroy() +print("\nDone. All physics examples complete.") diff --git a/tests/mocks/torch_mock.py b/tests/mocks/torch_mock.py index c40b59b3..6266e45a 100644 --- a/tests/mocks/torch_mock.py +++ b/tests/mocks/torch_mock.py @@ -322,6 +322,7 @@ def install_torch_mock(): torch_mock.no_grad = _NoGrad torch_mock.inference_mode = _NoGrad + # Seeding (used by policy_runner._set_eval_seed) torch_mock.manual_seed = lambda seed: None # torch.nn