METALBOX

Docker Containerization of METAL

Graph dataset

The .tmg graph files under metal/graphdata and metal/grapharchives are not tracked in git (or Git LFS). They're large, they change independently of the application code, and keeping them out of the repo is what keeps clones fast and the repo small. Instead they're packaged as .tar.zst archives and attached to GitHub Releases, and Docker Compose fetches whichever version you need automatically.

There are two datasets, handled the same way but independently versioned:

• metal/graphdata — the current, in-use graph set. Required; the app reads it directly. Version: GRAPH_DATA_VERSION. Fetched by the graphs-init Compose service (scripts/get-graphs.sh under the hood).
• metal/grapharchives — the historical/dated graph sets shown in the app's "archived graph set" dropdown. Optional and much larger. Version: GRAPH_ARCHIVES_VERSION. Fetched by the graph-archives-init Compose service (scripts/get-graph-archives.sh), only when you ask for it.

Quick start on a fresh clone

git clone https://github.com/HL5195/METALBOX-docker.git
cd METALBOX-docker

docker compose up --build

That's it — a graphs-init service in compose.yaml downloads and extracts metal/graphdata before the app container starts, so there's no separate download step to remember. The app is then available at http://localhost:9000.

Want the historical graph-archive sets too? Add the archives profile:

docker compose --profile archives up --build

or, equivalently, using the convenience wrapper:

./start.sh              # same as docker compose up --build
./start.sh --archives   # same as docker compose --profile archives up --build

If you'd rather fetch the data without starting Docker at all (e.g. to pre-warm it, or on a machine without Docker), the underlying scripts still work standalone: ./scripts/get-graphs.sh and ./scripts/get-graph-archives.sh.

How versioning works

All configuration lives in one file, .env, at the repo root (Docker Compose loads it automatically, and the scripts source it too):

GITHUB_REPOSITORY=HL5195/METALBOX-docker
GRAPH_DATA_VERSION=2026.08
GRAPH_ARCHIVES_VERSION=2026.08
GRAPH_COMPRESSION_LEVEL=9

To move to a newer graph dataset: bump GRAPH_DATA_VERSION (and/or GRAPH_ARCHIVES_VERSION) in .env, then re-run docker compose up --build (or ./scripts/get-graphs.sh directly). Every run compares the requested version against the .graph-version marker file written last time, and only re-downloads if they differ — safe and fast to run every time, including every docker compose up.

The resulting GitHub Release asset URL follows the standard https://github.com/<GITHUB_REPOSITORY>/releases/download/v<VERSION>/<asset> structure, e.g. https://github.com/HL5195/METALBOX-docker/releases/download/v2026.08/graphs-v2026.08.tar.zst.

Packaging a new graph release

From a machine with the new metal/graphdata (and/or metal/grapharchives) contents in place:

# packages metal/graphdata -> ./graphs-v<GRAPH_DATA_VERSION>.tar.zst
./scripts/package-graphs.sh

# packages metal/grapharchives -> ./graph-archives-v<GRAPH_ARCHIVES_VERSION>.tar.zst
./scripts/package-graph-archives.sh

# or both, and publish via `gh release` in one step (falls back to
# printing manual instructions if `gh` isn't installed):
./scripts/release-graphs.sh --archives

Version comes from .env by default, or an explicit argument (./scripts/package-graphs.sh 2026.09). Compression level comes from GRAPH_COMPRESSION_LEVEL in .env (1-22, higher = smaller/slower — 19+ can take a very long time on multi-GB datasets with limited CPU, so the default is a more practical 9).

To publish manually instead of release-graphs.sh:

1. Bump the version in .env (GRAPH_DATA_VERSION and/or GRAPH_ARCHIVES_VERSION) and commit that change.
2. Create a GitHub Release tagged v<VERSION> (e.g. v2026.09) — via the GitHub UI, or gh release create v2026.09.
3. Upload the .tar.zst file(s) produced above as release assets, keeping their generated filenames (graphs-v2026.09.tar.zst, graph-archives-v2026.09.tar.zst).
4. Anyone who runs docker compose up --build (or ./scripts/get-graphs.sh) after pulling the version bump picks up the new dataset automatically.

How Docker accesses the graph files

The graph directories are not copied into the application image — the Dockerfile/.dockerignore already exclude them from the build context, so docker build never sees them and the image stays small. Fetching happens in two small one-shot Compose services (graphs-init and, only under the archives profile, graph-archives-init), built from scripts/graphs-init.Dockerfile — a tiny Alpine image with bash, curl, tar, and zstd. Each mounts scripts/ and .env read-only plus its target directory, runs the matching get-graphs*.sh script unmodified, and exits; server is configured to wait for graphs-init to finish (depends_on: ... condition: service_completed_successfully) before it starts.

Once populated, the same host directories are bind-mounted read-only into server, the one container that actually serves them:

volumes:
  - ./metal/graphdata:/var/www/html/metal/graphdata:ro
  - ./metal/grapharchives:/var/www/html/metal/grapharchives:ro

There's a single local copy on disk either way, nothing is duplicated per container. If you add another container that needs the graph files, mount the same ./metal/graphdata (or ./metal/grapharchives) path into it rather than copying the data again.

Local layout

metal/
  graphdata/            # current dataset (graphs-init / scripts/get-graphs.sh)
    .graph-version       # installed version marker, e.g. "2026.08"
    usa.tmg
    ny.tmg
    ...
  grapharchives/        # optional historical sets (graph-archives-init / scripts/get-graph-archives.sh)
    .graph-version
    aug2025/
      ...
    jan2026/
      ...