One fast shared storage pool for large AI files: models, caches, git checkouts and Pinokio folders. You do not need to copy these files to each computer.
This repository is the storage layer. It does not contain a chat system. Use ai-gateway for the language-model part.
v0.2.4 · site · changelog · Unraid NFS runbook · Grafana sample
| Layer | What we use | Upstream |
|---|---|---|
| Drives | Two NVMe devices (prefer /dev/disk/by-id/…) |
— |
| Filesystem | Btrfs RAID0 (fast, no parity — one dead drive loses the pool) | Btrfs |
| Space saving | bees shares identical chunks so near-duplicate weights do not eat the disk twice | Zygo/bees |
| Sharing | Host NFS so Mac and Linux clients can mount the pool | kernel NFS |
| Scope | Storage only | ai-gateway for chat/agents |
| Measure | Value |
|---|---|
| Pool | 3.7 TB device; 2.60 TiB data used of 2.68 TiB allocated; 1.1 TB free (72% used) |
| bees | Hash table 6 GiB, 75% full. The crawl is current with the file system. |
Space that only each folder uses (btrfs filesystem du -s, "exclusive") |
models 1,759 GB · pinokio 250 GB · work 196 GB · github 56 GB |
Lesson: a duplicate-file audit overstates the space that you can get back, because bees already shares
those blocks. Measure "exclusive" space with btrfs filesystem du -s on the server before you delete.
On this pool, a 65 GB copy of a model folder had 0 GB of exclusive space.
The persist script on the flash drive cannot be executed. Unraid /boot
is FAT32. It has no Unix execute bits. If /boot/config/go calls the script
as a bare path, the console prints Permission denied, host nfsd never
starts, and every client mount fails. The container can look perfectly healthy
while that happens.
# correct — in /boot/config/go, after emhttp
( sleep 45; bash /boot/config/plugins/fast-models/host-nfs-export.sh ) &Wrong: rc.nfsd start alone (empty /etc/exports). Wrong: Settings → NFS
(array shares stay SMB). Details:
docs/UNRAID_NFS_PERSIST.md ·
what broke on 2026-08-15.
This Unraid host-nfsd layout is the stopgap until the pool lives on its own non-Unraid host.
The container is privileged and sees the raw NVMe devices through the host kernel. That is convenient. It is not the same as giving a VM exclusive PCIe ownership of the drives.
Do not also put those NVMes in the Unraid array, a cache pool, or Unassigned Devices auto-mount while this stack is using them.
- BIOS: PCIe bifurcation for the dual-M.2 adapter slot (
x4x4/ vendor equivalent). - Unraid plugins (recommended): Unassigned Devices for inventory only — do not auto-mount these two. Dynamix SSD TRIM optional.
- Host free: leave both NVMe unassigned.
- Docker Compose: Compose Manager plugin, Portainer, or CLI
docker compose.
lsblk -o NAME,SIZE,TYPE,MODEL,SERIAL
ls -l /dev/disk/by-id/nvme-*-
Copy this folder to the array, e.g.
/mnt/user/appdata/fast-models/stack/. -
cp .env.example .envand setNVME0/NVME1to by-id paths. -
Create appdata dirs:
mkdir -p /mnt/user/appdata/fast-models/{bees,config} -
Compose Manager → add project → this
docker-compose.yml→ Compose Up.
cd /mnt/user/appdata/fast-models/stack
cp .env.example .env
# edit .env
mkdir -p /mnt/user/appdata/fast-models/{bees,config}
docker compose --env-file .env up -d --buildEmpty drives only:
# in .env
ALLOW_FORMAT=1
FORCE_FORMAT=0docker compose --env-file .env up -d --build
docker logs -f fast-modelsWhen healthy, set ALLOW_FORMAT=0 and recreate. FORCE_FORMAT=1 (with
ALLOW_FORMAT=1) is a destructive wipe. Never leave it set.
Inside the container at /ai-data (NFS root with fsid=0):
/ai-data/
models/
uv-cache/
hf-cache/
github/
pinokio/
stability-matrix/
work/
docker-registry/
.fast-models-info
| Service | Role | Port 2049 |
|---|---|---|
| Unraid array NFS | Off (shareNFSEnabled=no) — array paths are SMB-only |
— |
| Host NFS export | /mnt/ai-data = same dual-NVMe Btrfs as the container |
Owns 2049 (v3+v4, rpcbind-registered) |
| fast-models container | Btrfs RAID0 + bees; ENABLE_NFS=0 |
Does not run nfsd |
Alpine-in-Docker nfsd listens on 2049 and works from Linux, but often fails
to register with host rpcbind, so macOS mount_nfs returns RPC prog. not
avail. Host nfsd + host mount of the same Btrfs is the path that works for
both clients.
fsid=0 → mount path is / (not /mnt/ai-data):
sudo apt-get install -y nfs-common # once
sudo mkdir -p /mnt/ai-data
sudo mount -t nfs -o vers=4.2,rsize=1048576,wsize=1048576,hard,noatime,nconnect=4 \
<nas-host-ip>:/ /mnt/ai-data
test -f /mnt/ai-data/.fast-models-info && cat /mnt/ai-data/.fast-models-infofstab:
<nas-host-ip>:/ /mnt/ai-data nfs vers=4.2,rsize=1048576,wsize=1048576,hard,noatime,nconnect=4,_netdev,x-systemd.automount 0 0
Fallback: vers=4.1, or vers=3 with <nas-host-ip>:/mnt/ai-data.
Lesson from this lab (August–September 2026): NFS mounts on macOS were not reliable. NFS v3, v4 and v4.1 mounts hung many times, also after a macOS update. A hung mount stops every program that reads the share. We moved the Mac clients to SMB3. Linux clients stay on NFSv4.2, which has worked without these hangs. Do not expect one NFS setup to serve Linux (v4.2) and macOS (v3) equally well.
Recommended (Finder → Go → Connect to Server, or the command line):
mkdir -p ~/ai-data
mount_smbfs //<user>@<nas-host-ip>/<share> ~/ai-dataUse the share for reading and for normal copies. Run large file walks, hashes and bulk moves on the server itself, not over the network share.
Fallback only (NFS v3; the v4 root often fails with RPC prog. not avail):
sudo mkdir -p /Volumes/ai-data
sudo mount_nfs -o resvport,vers=3,tcp,rw,hard,intr \
<nas-host-ip>:/mnt/ai-data /Volumes/ai-data
test -f /Volumes/ai-data/.fast-models-info && echo OKmacOS defaults to uid 501, most Linux first users are 1000. After a bulk copy, keep shared dirs world-writable (LAN-only) or both uids can write:
ssh root@<nas-host-ip> 'chmod -R a+rwX /mnt/ai-data/{models,pinokio,uv-cache,work}'Host-split uv caches (uv-cache/<host>) if more than one machine writes
the cache. A shared uv cache grows host-specific symlinks and then breaks.
bees walks the Btrfs pool and shares identical chunks. It is always on
when ENABLE_BEES=1 (not a nightly cron).
| Hash table | When |
|---|---|
| 1G | Small pools / first experiments |
| 2G | Default for multi-TiB pools |
| 4G | Only if 2G stays nearly full after a full re-crawl; ~4 GiB sticky RAM, no swap |
| 6G | This pool today (2.6 TiB, mostly model files): 75% full with evictions; ~6 GiB sticky RAM |
| 8G | Planned next step as the pool grows past ~3 TiB or evictions stay high; ~8 GiB sticky RAM |
Reality check (this pool, 2.6 TiB of mostly model files): the 1G table was 99% full by July 2026. We increased it to 4G, then to 6G on 2026-09-27. At 6G it is 75% full and still evicts old fingerprints (about 1.4 million evictions), so some duplicates are missed. For a 2–3 TiB pool of model files, plan for 4–6G. The table size is also the RAM that bees keeps in use.
A table at 100% full is not a full disk and is not caused by Unraid parity. bees still runs; it just forgets old fingerprints. Sizing notes: docs/BEES_HASH_SIZING.md.
Image pins bees v0.11 (small musl/gettid fix for Alpine). Default: 1
thread, scan-mode 4, nice / ionice. The hash file lives on the array
under appdata, not on the NVMe pool.
docker exec fast-models pgrep -a bees
ls -lh /mnt/user/appdata/fast-models/bees/beeshash.dat
docker exec fast-models sh -c 'grep -E "cells occupied|Uptime" /var/lib/bees/beesstats.txt'Optional deeper pass (IO-heavy; bees remains the everyday tool):
docker exec fast-models duperemove -r -d -h /ai-data/models
docker exec fast-models compsize /ai-data/models 2>/dev/null || trueKeep BEES_THREADS=1 on ~32 GiB hosts unless you clearly have spare capacity.
scripts/bees-health.sh and scripts/duperemove-weekly.sh are for Unraid
User Scripts or cron.
Today the pool is a privileged container on Unraid plus a host nfsd hook.
That works. It is also how a FAT32 go line can take the share down after
every reboot.
A dedicated host (bare metal, or a small VM with the NVMe card passed
through) would own Btrfs + bees + nfsd with a real Unix /boot. Unraid
becomes just another NFS client. Until that move, follow the persist
runbook and do not turn the container into the NFS server.
| Rule | Why |
|---|---|
| RAID0 | One drive failure loses the entire pool |
| Models re-downloadable | Prefer HF / Forgejo rehydrate over treating the pool as sole backup |
| No array membership | Prevents Unraid and the container fighting for the same devices |
ALLOW_FORMAT=0 steady-state |
Avoids accidental mkfs on restart |
Tighten NFS_CLIENTS |
Default 192.168.0.0/16 is LAN-wide |
If Unraid NFS settings also export shares on 2049, pick one owner of that port. This stack wants the host to own it.
If you later prefer Unraid to own the Btrfs cache pool
(Main → Add Pool → RAID0) and only run NFS export in Docker, bind
/mnt/fast-models as a volume and drop devices: / format logic. That is
the “host pool” topology. This compose is the device-bound path.
Bottleneck that we measured: the server has one 1 GbE network port. Linux NFS clients and Mac SMB clients share it, so copies stop at about 110 MB/s. Hashing is not the limit: one modern CPU core hashes SHA-256 at 0.4–3 GB/s.
Plan: move storage to a dedicated host. Alpine Linux in Docker on a dual-Xeon server, with a 10 Gb (or faster) SFP+ back-end link between the storage host and the compute hosts. The 1 GbE network then serves only normal client traffic. Until then, the NFS and SMB rules above apply.
VFIO passthrough of the dual-NVMe card + ZFS bclone + NFS inside a minimal VM is the dedicated-host story. Not implemented in this compose.
docker compose ps—fast-modelshealthy.docker exec fast-models btrfs fi show— both NVMe devices in one FS.docker exec fast-models ls /ai-data— layout dirs present.- Client: NFS 4.2 mount + multi-GiB read/write (
test -f …/.fast-models-info). - After reboot: host
/mnt/ai-datais a mount,showmountlists it, console did not print Permission denied on the persist script. - (Later) two near-identical GGUFs under
models/→ bees/compsize shows savings.
Shared models and tool trees got too big to copy onto every machine (desk Mac, GPU host, NAS). After the caregiving pivot (13 April 2026) and the race toward the ACL Phase 1 deadline (31 July 2026), the home lab needed one honest pool: fast NVMe, Btrfs, bees for near-duplicates, NFS for clients.
This repo is that storage plane, spun out of the M.A.N.A.G.E.R. monorepo so the gateway and coding agents have somewhere real to put weights. Public release followed the ACL submission week as part of opening the stack piece by piece.
Timeline anchors: vibecoding from 22 March 2026; caregiving mission from 13 April 2026; LLC 20 April 2026; public modular repos late July 2026.
| File | Role |
|---|---|
docker-compose.yml |
Unraid import target |
Dockerfile |
Alpine + btrfs + bees + nfs-utils |
entrypoint.sh |
format/mount → layout → bees |
scripts/host-nfs-export.sh |
Host mount + nfsd persist (invoke with bash) |
config/exports.template |
NFS export (container path; unused when ENABLE_NFS=0) |
config/bees.conf.template |
bees notes / UUID stamp |
.env.example |
device paths + safety flags |
docs/ |
Runbook, incident, Grafana sample, GitHub Pages |
docs/observability/ |
Importable Grafana dashboard + Prometheus scrape example |
deploy/observability/ |
node_exporter textfile script for bees + btrfs |
© 2026 M.A.N.A.G.E.R. LLC — prepare for the care when we cannot be there