diff --git a/README.md b/README.md index 1f042d7..2bdf6d7 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,67 @@

# What is Heracles? -This repository provides a library for encoding a [Hydra](https://github.com/MIT-SPARK/Hydra) 3D scene graph into a Neo4j database. It expects a scene graph to initially be loaded from file via [spark\_dsg](https://github.com/MIT-SPARK/Spark-DSG) and provides utilities for translating back and forth between spark\_dsg and the Neo4j database. -## Setup +This repository provides a library for encoding a +[Hydra](https://github.com/MIT-SPARK/Hydra) 3D scene graph into a Neo4j +database. It expects a scene graph to initially be loaded from file via +[spark\_dsg](https://github.com/MIT-SPARK/Spark-DSG) and provides utilities for +translating back and forth between spark\_dsg and the Neo4j database. + +We provide three examples that should be pretty easy to run. They require that +Docker is installed, but they should have no other dependencies. + +## Minimal Example + +To run the first example, run +```bash +./heracles_demo.sh +``` + +This script will load an example scene graph into the graph database and put +you in a python terminal where you can run example queries. For example, try +running the following query to print the location of all trashcans: +```python +db.query("MATCH (n: Object {class: 'trash'}) RETURN n.center as center""") +``` + +## Interactive ROS Example + +The second example will render the current state of the scene graph in the +database in Rviz. You do _not_ need ROS installed to run this demo. However, +you will need to specify an OpenAI api key `HERACLES_OPENAI_API_KEY` if you +want to use the LLM interaction functionality. + +```bash +export HERACLES_OPENAI_API_KEY= +xhost +local:docker && ./chatdsg_ros_demo.sh +``` + +Try asking the LLM how many objects are in the scene graph (press `ctrl-b` to +send your message to the LLM). Then try telling it to update some of the +objects (e.g., turn `sign`s into `light`s). + +## Interactive Viser Example + +Finally, we provide an example of visualizing the scene graph using Viser, so +that no ROS is necessary. Again, you will need an OpenAI API key if you want to +use the LLM functionality. Make sure to go to http://127.0.0.1:8081 in your +browser after running the script! + +```bash +export HERACLES_OPENAI_API_KEY= +./chatdsg_viser_demo.sh # Then navigate to http://127.0.0.1:8081 in your browser +``` + +You may have to pan the camera around if the scene graph is rendered out of the +initial camera frame. + + +# Development Setup + +If you want functionality beyond these basic demos, you will want to install +`heracles` locally. This requires 1) spinning up a database, and 2) installing +`heracles. Pull the Neo4j database image: ```bash @@ -35,15 +93,13 @@ for an example, as that successfully installs and runs the library for CI. If you don't have a scene graph to test with, you can use a large example scene graph [here](https://drive.google.com/file/d/1aktyS792PUrj2ACRu1DoxMGse55GWloB/view?usp=drive_link). This scene graph has 2D places and objects, and 3D places (that don't make much -sense), but no regions or rooms. - -The script `examples/dsg_test.py` is a small example of loading an existing -scene graph file into a graph database and running some simple queries. You can -run it in interaction mode (`ipython3 -i heracles/examples/dsg_test.py`) and -then try executing some of the other example queries below. +sense), but no regions or rooms. This repo contains an example scene graph as well. ## Demo - Loading a Scene Graph -We provide a demo program that loads a scene graph from file, encodes it into a database, and performs a simple query for validation. The demo program is implemented with interactive mode in mind to make it easy to try other kinds of queries. +We provide a demo program that loads a scene graph from file, encodes it into a +database, and performs a simple query for validation. The demo program is +implemented with interactive mode in mind to make it easy to try other kinds of +queries. ### Environment Variables We expect certain environment variables to be set, listed below. `HERACLES\_NEO4J\_URI` can alternatively be provided via the command line. An example of `HERACLES\_NEO4J\_URI` is `neo4j://127.0.0.1:7688`. diff --git a/chatdsg_ros_demo.sh b/chatdsg_ros_demo.sh new file mode 100755 index 0000000..69626c4 --- /dev/null +++ b/chatdsg_ros_demo.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -e + +docker compose -f docker/docker-compose.yaml up -d neo4j hydra_visualization +docker compose -f docker/docker-compose.yaml run --rm chatdsg # Blocks until terminal exits +docker compose -f docker/docker-compose.yaml down hydra_visualization diff --git a/chatdsg_viser_demo.sh b/chatdsg_viser_demo.sh new file mode 100755 index 0000000..0c192a5 --- /dev/null +++ b/chatdsg_viser_demo.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -e + +docker compose -f docker/docker-compose.yaml up -d neo4j viser_visualization +docker compose -f docker/docker-compose.yaml run --rm chatdsg +docker compose -f docker/docker-compose.yaml down viser_visualization diff --git a/docker/chatdsg_demo.Dockerfile b/docker/chatdsg_demo.Dockerfile new file mode 100644 index 0000000..d969133 --- /dev/null +++ b/docker/chatdsg_demo.Dockerfile @@ -0,0 +1,8 @@ +FROM python:3.12-slim + +RUN apt-get update +RUN apt-get install -y git +COPY heracles/examples/scene_graphs/example_dsg.json example_dsg.json +RUN git clone https://github.com/GoldenZephyr/heracles_agents.git +RUN pip install ./heracles_agents[all] +WORKDIR /heracles_agents/examples/chatdsg diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml new file mode 100644 index 0000000..dddfd9f --- /dev/null +++ b/docker/docker-compose.yaml @@ -0,0 +1,87 @@ +--- +services: + + # Graph database storing scene graph + neo4j: + image: neo4j:5.25.1 + environment: + NEO4J_server_bolt_listen__address: :7683 + NEO4J_server_http_advertised__address: :7469 + NEO4J_server_http_listen__address: :7469 + NEO4J_AUTH: neo4j/neo4j_pw + network_mode: host + healthcheck: + test: [CMD-SHELL, wget --no-verbose --tries=1 --spider http://localhost:7469 || exit 1] + interval: 1s + timeout: 10s + retries: 10 + start_period: 2s + + # Standalone python interpreter example for running queries against database + cli: + build: + context: .. + dockerfile: docker/minimal_db_cli.Dockerfile + depends_on: + neo4j: + condition: service_healthy + environment: + HERACLES_NEO4J_USERNAME: neo4j + HERACLES_NEO4J_PASSWORD: neo4j_pw + HERACLES_NEO4J_URI: neo4j://127.0.0.1:7683 + network_mode: host + stdin_open: true + tty: true + entrypoint: [ipython, -i, /heracles/examples/load_scene_graph.py, --, --scene_graph, /heracles/examples/scene_graphs/example_dsg.json] + + # Interactive LLM-based agent interaction with database + chatdsg: + build: + context: .. + dockerfile: docker/chatdsg_demo.Dockerfile + depends_on: + neo4j: + condition: service_healthy + environment: + HERACLES_NEO4J_USERNAME: neo4j + HERACLES_NEO4J_PASSWORD: neo4j_pw + HERACLES_NEO4J_URI: neo4j://127.0.0.1:7683 + HERACLES_OPENAI_API_KEY: $HERACLES_OPENAI_API_KEY + HERACLES_AGENTS_PATH: /heracles_agents + ADT4_HERACLES_IP: 127.0.0.1 + ADT4_HERACLES_PORT: 7683 + network_mode: host + stdin_open: true + tty: true + entrypoint: [python3, chatdsg.py, --scene-graph, /example_dsg.json] + + # Visualize scene graph in RVIZ using Hydra ROS visualizer + hydra_visualization: + build: + dockerfile: hydra.Dockerfile + depends_on: + neo4j: + condition: service_healthy + environment: + HERACLES_NEO4J_USERNAME: neo4j + HERACLES_NEO4J_PASSWORD: neo4j_pw + HERACLES_IP: 127.0.0.1 + HERACLES_PORT: 7683 + HERACLES_VENV: '' + DISPLAY: $DISPLAY + QT_X11_NO_MITSHM: 1 + network_mode: host + volumes: [/tmp/.X11-unix:/tmp/.X11-unix:rw] + entrypoint: [/bin/bash, /run_hydra_visualization.sh] + + # Visualize scene graph in Viser using spark_dsg Viser visualization + viser_visualization: + build: + context: .. + dockerfile: docker/spark_dsg_viser.Dockerfile + environment: + HERACLES_NEO4J_USERNAME: neo4j + HERACLES_NEO4J_PASSWORD: neo4j_pw + HERACLES_NEO4J_URI: neo4j://127.0.0.1:7683 + network_mode: host + entrypoint: [python, /heracles/src/heracles/heracles_viser_publisher.py, --ip, 127.0.0.1, --port, '8081'] diff --git a/docker/hydra.Dockerfile b/docker/hydra.Dockerfile new file mode 100644 index 0000000..1cbae96 --- /dev/null +++ b/docker/hydra.Dockerfile @@ -0,0 +1,26 @@ +FROM ros:jazzy-ros-core + +RUN apt-get update +RUN apt-get install -y git ros-dev-tools python3.12-venv python3-pip libgoogle-glog-dev +RUN rosdep init && rosdep update + +RUN mkdir -p /hydra_ws/src +WORKDIR /hydra_ws +RUN echo "build: {cmake-args: [-DCMAKE_BUILD_TYPE=Release]}" > colcon_defaults.yaml + +WORKDIR /hydra_ws/src +RUN git clone https://github.com/MIT-SPARK/Hydra-ROS.git hydra_ros +RUN git clone https://github.com/GoldenZephyr/heracles_agents.git +RUN git clone https://github.com/GoldenZephyr/heracles.git + +RUN python3 -m venv /venv --system-site-packages +RUN git clone https://github.com/MIT-SPARK/config_utilities.git +RUN git clone --branch ros2 https://github.com/MIT-SPARK/Kimera-PGMO.git +RUN git clone https://github.com/MIT-SPARK/Spark-DSG.git +RUN git clone https://github.com/MIT-SPARK/ianvs.git + +RUN rosdep install --from-paths . --ignore-src -r -y +WORKDIR /hydra_ws +RUN . /opt/ros/jazzy/setup.sh && colcon build --packages-up-to hydra_visualizer && colcon build --packages-up-to heracles heracles_ros heracles_agents +RUN . /venv/bin/activate && pip install ./src/heracles_agents[all] +COPY run_hydra_visualization.sh /run_hydra_visualization.sh diff --git a/docker/minimal_db_cli.Dockerfile b/docker/minimal_db_cli.Dockerfile new file mode 100644 index 0000000..1f2102e --- /dev/null +++ b/docker/minimal_db_cli.Dockerfile @@ -0,0 +1,5 @@ +FROM python:3.12-slim + +RUN apt update +COPY heracles heracles +RUN pip install ./heracles ipython diff --git a/docker/run_hydra_visualization.sh b/docker/run_hydra_visualization.sh new file mode 100755 index 0000000..9ba40dc --- /dev/null +++ b/docker/run_hydra_visualization.sh @@ -0,0 +1,5 @@ +#!/bin/sh +. /opt/ros/jazzy/setup.sh +. /hydra_ws/install/setup.sh +. /venv/bin/activate +ros2 launch heracles_ros demo.launch.yaml launch_heracles_publisher:=true launch_hydra_visualizer:=true launch_rviz:=true diff --git a/docker/spark_dsg_viser.Dockerfile b/docker/spark_dsg_viser.Dockerfile new file mode 100644 index 0000000..4216e63 --- /dev/null +++ b/docker/spark_dsg_viser.Dockerfile @@ -0,0 +1,7 @@ +FROM python:3.12-slim + +RUN apt update +RUN apt install -y git libzmq3-dev nlohmann-json3-dev build-essential +COPY heracles heracles +RUN git clone -b feature/viser_aaron --single-branch https://github.com/MIT-SPARK/Spark-DSG.git spark_dsg +RUN pip install ./heracles ipython ./spark_dsg diff --git a/heracles_demo.sh b/heracles_demo.sh new file mode 100755 index 0000000..b366ff4 --- /dev/null +++ b/heracles_demo.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash +set -e +docker compose -f docker/docker-compose.yaml up -d neo4j +docker compose -f docker/docker-compose.yaml run --rm cli