VLA Lab

This is a repository purely for educational porpuse made during the last semester of my AI Bachalor. Visual Language Action model is any input data turn into action tokens that will become action in a real/simulated robot. In specific this repository works with code generation through LLM, where the action tokens are a sequency of actions (through APIs) in code format. This is heavly inspired by the work of RoboCodeX

Features

• ROS2 Foxy
• API control for the panda arm using the repository Panda ROS2 Gazebo
• Code Generation Through LLMs
  • OpenAI application (Cloud)
  • Ollama application (Local)

Table of Contents

• Prerequisites
• Installation
• Configuration
• How to Run
• Architecture overview

Prerequisites

• Python 3.10+
• OpenAI API Key
• GPU (Cuda 11+)
• Docker Engine 24.0+

Installation

Docker development

git submodule update --init --recursive
docker build -f docker/Dockerfile -t panda_ros2_gazebo:v0.1
# If you want with the LLM part
docker compose -f docker/docker-compose.yaml --env-file docker/.env --project-name vla up -d
# If you want only the ros2 container
chmod +x ./run.sh
./run.sh

Setup Dependency Packages (inside docker)

cd /root/workspace/src
vcs import < panda_ros2_gazebo/workspace.repos

Colcon Build (inside docker)

source /opt/ros/${ROS2_DISTRO}/setup.bash
export COLCON_PYTHON_EXECUTABLE=/usr/bin/python3.8
colcon build --merge-install --symlink-install --cmake-args -DIDYNTREE_USES_PYTHON=True -DPYTHON_EXECUTABLE=/usr/bin/python3.8 -DIDYNTREE_USES_IPOPT:BOOL=ON -DCMAKE_BUILD_TYPE=Release

Configuration

Create a .env file in the root directory with the following structure:

OPENAI_API_KEY=<your_key>

How to run

First terminal run the gazebo launch and the task library node:

ros2 launch panda_ros2_gazebo gazebo.launch.py & ros2 launch panda_ros2_gazebo bringup.launch.py mode:=tasklibrary & wait

On a second terminal, run the NL Command Bridge that will allow to send requests to a LLM

ros2 run nl_command_bridge node

Finnaly, you can send a command publishing in the topic

ros2 topic pub /nl_command std_msgs/String "data: 'Create a stack of 3 boxes'"

Some other options can be:

ros2 topic pub /manip_task/cmd std_msgs/String "data: 'HOME'"
ros2 topic pub /manip_task/cmd std_msgs/String "data: 'GO 0.55 0.10 0.30 0 90 0'"
ros2 topic pub /manip_task/cmd std_msgs/String "data: 'PICK 0.55 0.00 0.00 0.05'"
ros2 topic pub /manip_task/cmd std_msgs/String "data: 'PLACE 0.30 0.50 0.00 0.05'"

Architecture overview