doomworld

A tiny codebase for building world models for Doom.

First environment is VizDoom take_cover

collect rollouts -> learn V -> learn M -> train C in dreams -> compare dream vs real -> evaluate in Doom

• V: visual model that compresses Doom frames into latents
• M: memory/world model that predicts future latents
• C: controller trained against the learned world

Why world models?

World models have a little bit of everything:

0. They seem fun
1. Pixels need compression
2. Actions need dynamics
3. Rewards and done states are easy to exploit
4. Planning exposes model errors quickly
5. Bad predictions are visible as videos

Inspired by World Models

Setup

uv sync

Dashboard & API Server

1. Start the API Server (Backend)

Run the following command from the root of the repo to start the API:

uv run python -m api.app --runs-root runs/demo --port 8787

2. Start the Dashboard

Navigate to the web directory and start the server:

cd web
npm run dev

The dashboard will be available at http://localhost:5173.

Modal

modal run infra/modal_app.py::run_pipeline_remote --spec-name baseline_debug
modal run --detach infra/modal_app.py::run_pipeline_remote --spec-name proper_world_v1
modal deploy infra/modal_app.py