FoG Detection

FoG Detection is a local workstation app for markerless 3D gait and freezing-of-gait (FoG) analysis from ordinary video. It uses Meta's SAM 3D Body to recover a 3D body mesh per frame, derives lower-body kinematics, and computes a kinematics-only FoG probability signal that you can review, annotate, and benchmark — all in a browser-based 3D viewer that runs entirely on your machine.

What you can do

• Upload a video in the viewer.
• Detect & pick the subject — a streaming, text-prompted detector scans the whole video and previews every detected person with a box; select the one to reconstruct, or merge several track fragments of the same person into one subject before reconstructing.
• Reconstruct with SAM 3D Body to get a per-frame 3D mesh and joint kinematics, streamed live into the 3D viewer.
• Compute and tune a kinematics-only FoG probability signal (auto-computed when you open a run).
• Annotate freezing episodes and compare predictions against ground truth with event/frame F1 metrics.
• Export one folder containing the full per-joint kinematics, the FoG probability, your annotation track, any ground truth, and a "tracking box" MP4 — each channel in the format you choose (CSV / JSON / ELAN).

How it works

FoG Detection is a single Next.js app: one process serves both the browser UI and the backend API. When you process a video, the app spawns the Python pipeline (uv run sam3d …) and reads/writes the repository's input/ and output/ folders directly. The model weights, the Python pipeline, and the heavy compute all live wherever this app runs — your laptop for local use, or a GPU server for remote use (see Run on a server).

There is no separate backend process to start: launching the app from the repository root with uv, ffmpeg, and the model weights available is all it takes. ./dev.sh wires it together for you.

Subject detection across platforms. The fast in-viewer subject-detection preview uses an MLX build of SAM 3 that is Apple-Silicon–only. On Linux/Windows the reconstruction pipeline still detects and tracks the subject itself (PyTorch SAM 3, CUDA/CPU) — you process a video the same way, the in-viewer detect-preview step is simply skipped. The heavy 3D reconstruction (SAM 3D Body) is PyTorch and runs on every platform.

Requirements

Tool	Notes
Python >=3.12,<3.13	provisioned automatically by uv
uv	Python environment + runner
Node.js 18+ and npm	the web viewer
ffmpeg + ffprobe	must be on your PATH (video I/O)
Hugging Face account	the model weights are gated (see step 2)
~6 GB disk	for the model weights

Supported machines

FoG Detection runs on any machine — a GPU is optional. The pipeline automatically picks the best available compute device (NVIDIA CUDA → Apple Silicon MPS → CPU); pass --force-cpu to override.

Platform	Acceleration	Notes
Apple Silicon (M-series Mac)	MPS (partial) + CPU	./dev.sh enables the per-op CPU fallback the SAM 3 detector needs on MPS.
Intel Mac	CPU only	Works out of the box; slower than a GPU.
Linux + NVIDIA GPU	CUDA	Fastest. Install a CUDA-capable driver; PyTorch CUDA wheels come with uv sync.
Linux / any CPU-only	CPU	Works; expect longer processing times.
Windows	CUDA or CPU	See the Windows note below.

Install ffmpeg for your OS:

# macOS (Homebrew)
brew install ffmpeg
# Debian / Ubuntu
sudo apt-get install -y ffmpeg
# Windows (PowerShell, winget) — or use choco / scoop, or ffmpeg.org + add to PATH
winget install --id Gyan.FFmpeg

1. Install

From the repository root:

uv sync                                  # creates .venv and installs the backend
cd web-viewer && npm install && cd ..    # installs the web viewer

uv sync registers the sam3d command used throughout this guide.

2. Download the models

FoG Detection needs two gated Hugging Face models. First request access (one click, usually instant) on each model page, then log in with a token:

• SAM 3D Body — https://huggingface.co/facebook/sam-3d-body-dinov3
• SAM 3 — https://huggingface.co/facebook/sam3

uv run hf auth login    # paste a token from https://huggingface.co/settings/tokens

Download the weights (~6 GB total):

# SAM 3D Body — 3D mesh recovery (model.ckpt + mhr_model.pt + config), ~2.7 GB
uv run hf download facebook/sam-3d-body-dinov3 --local-dir models/sam-3d-body-dinov3

# SAM 3 — open-vocabulary subject detector, ~3.2 GB (downloaded into the HF cache)
uv run hf download facebook/sam3 sam3.pt config.json

The SAM 3D Body command lands the files at the exact paths the pipeline expects:

models/sam-3d-body-dinov3/model.ckpt
models/sam-3d-body-dinov3/model_config.yaml
models/sam-3d-body-dinov3/assets/mhr_model.pt

SAM 3 is loaded from the Hugging Face cache (~/.cache/huggingface), so the hf download above is all that's needed. After this one-time online download the app runs fully offline (it sets HF_HUB_OFFLINE=1).

Notes

• Your first processing run also fetches the DINOv3 backbone code via torch.hub (cached under ~/.cache/torch/hub), so keep internet on for the first run.
• sam3d doctor also accepts a copy of SAM 3 at models/sam3/sam3.pt, but the running app loads it from the cache and does not require that copy.
• Apple Silicon only: the first time you use the in-viewer subject-detection preview, it downloads the MLX build of SAM 3 (mlx-community/sam3-image, ~3.4 GB, ungated) into the HF cache automatically. No extra step is needed on Linux/Windows, where that preview is skipped.

Models, sources & licenses

All weights are downloaded from their original publishers — none are redistributed in this repository. Review and accept each model's license on its page before use.

Model	Used for	Source	License
SAM 3D Body (sam-3d-body-dinov3, incl. the MHR head)	per-frame 3D body mesh	Meta — huggingface.co/facebook/sam-3d-body-dinov3	gated; see the model card
SAM 3	open-vocabulary subject detection (PyTorch, all platforms)	Meta — huggingface.co/facebook/sam3	gated; see the model card
SAM 3 (MLX)	fast in-viewer detect preview (Apple Silicon)	mlx-community — huggingface.co/mlx-community/sam3-image	inherits SAM 3's terms
DINOv3 backbone	image features inside SAM 3D Body	Meta — github.com/facebookresearch/dinov3	see the repo license

Upstream code is vendored under vendor/ so the pipeline runs straight after clone — Meta's SAM 3 and SAM 3D Body, and the MLX SAM 3 port — each under its own upstream license (see the headers/LICENSE files in those folders). Only the heavy sample media and demo notebooks are excluded.

Verify everything is in place:

uv run sam3d doctor --json

doctor reports Python/torch versions, CUDA/MPS availability, ffmpeg/ffprobe presence, and whether each model file is found.

3. Run the app

From the repository root:

./dev.sh

Then open http://127.0.0.1:4001/.

./dev.sh installs the web dependencies if needed, sets the backend environment (including the Apple-Silicon MPS fallback), and starts the app on port 4001. Run it from the repository root — the app locates input/, output/, and the Python environment relative to that directory.

Manual equivalent (any OS):

cd web-viewer
npm run dev -- --hostname 127.0.0.1 --port 4001

For a production build, run npm run build then npm run start -- --hostname 127.0.0.1 --port 4001 from web-viewer/.

Windows

./dev.sh is a bash script: run it from Git Bash or WSL2, or use the manual npm commands above from PowerShell (the Apple-Silicon fallback variable it sets is harmless/ignored on Windows). Install ffmpeg (see above) and make sure ffmpeg/ffprobe are on PATH (ffmpeg -version should work in a fresh shell). CUDA acceleration works if you have an NVIDIA GPU and driver; otherwise the pipeline runs on CPU. Cancelling a running job terminates the whole process tree via taskkill (POSIX uses process groups), so no Python child is left orphaned. The fast in-viewer detect preview is Apple-Silicon-only and is skipped on Windows (the pipeline detects the subject itself — see How it works).

Run on a server (access from another machine)

The compute (Python pipeline + model weights) runs wherever the app runs, so the simplest way to "put the backend on a server" is to run FoG Detection on that server and open its UI from another machine's browser. This is useful when the server has a GPU and your laptop does not.

On the server (after Install + Download models), bind to all interfaces and allow your own URL as a browser origin:

cd web-viewer
npm run build
FOGDET_ALLOWED_ORIGINS="http://SERVER_HOST:4001" \
  npm run start -- --hostname 0.0.0.0 --port 4001

Then from any machine on the network, open http://SERVER_HOST:4001/. The UI and the API share the same origin, so processing jobs run on the server's GPU/CPU and stream results back to your browser. Put it behind a reverse proxy (nginx, Caddy, Tailscale, …) for TLS and authentication if it is exposed beyond a trusted network.

Advanced: a separate frontend host. You can also deploy the frontend on one host and point it at a backend on another. Set NEXT_PUBLIC_FOGDET_BACKEND_URL on the frontend to the backend's URL, and FOGDET_ALLOWED_ORIGINS on the backend to the frontend's origin (exact origins or https://*.example.com wildcards are supported). See Configuration and web-viewer/.env.example.

Using the app

The viewer walks you through four steps. (Screenshots live in docs/screenshots/ — add your own with the filenames below to illustrate your build.)

1. Upload a video.

   

2. Detect & pick the subject. Enter a text prompt (default person) and the detector scans the whole video, progressively drawing a coloured box around every person it finds. Each distinct identity becomes a chip. If one person is split into several chips (e.g. after they leave and re-enter, or a hard cut), select those chips and Merge them into one subject. Then pick the subject to reconstruct. (On Linux/Windows this preview is skipped — see How it works.)

   

3. Reconstruct. The 3D viewer streams the reconstructed mesh and kinematics as the job runs (one job at a time). The left pane toggles between the original Video and the Tracking box overlay.

   

4. Analyze, annotate & export. Review the FoG probability strip, tune the threshold, and label freezing episodes. Export writes one .zip that unpacks to a ‹video›_‹timestamp›/ folder with a separate file per channel — full per-joint kinematics, FoG probability, your annotation track, any ground truth, and the tracking-box MP4 — each in the format you pick (CSV / JSON / ELAN .eaf).

   

Command-line usage (optional)

The viewer drives these for you, but you can also run the pipeline directly.

Process one video (automatic patient detection):

uv run sam3d run \
  --video-input input/example.mp4 \
  --run-id example_processed \
  --inference-target body \
  --precision float32 \
  --auto-init-mode sam3 \
  --auto-select-strategy patient \
  --sam3-text-prompts "the patient" \
  --no-preview --output-codec h264

Process with a manual patient box (x1,y1,x2,y2):

uv run sam3d run \
  --video-input input/example.mp4 \
  --run-id example_manual \
  --auto-init-mode off \
  --prompt-bbox "210,0,830,720" --prompt-bbox-frame 0 \
  --inference-target body --precision float32 \
  --no-preview --output-codec h264

--run-id is optional; it defaults to <video_name>_processed. On a machine with no GPU, add --force-cpu.

Analyze a run (FoG probability + kinematics):

uv run sam3d analyze --run-id example_processed --preset clinical_fog_v1

Artifacts are written under output/<run_id>/analysis/<analysis_id>/. The analysis exports fog.probability (kinematics-only FoG signal), fog.threshold (default binary threshold, 0.37), and a guarded XY root trajectory. See docs/fog_score.md for the signal definition.

Configuration

Most setups need nothing here. Copy web-viewer/.env.example to web-viewer/.env.local to override defaults.

Variable	Default	Purpose
FOGDET_RUNS_ROOT	output	where processed runs are stored
FOGDET_UPLOADS_ROOT	input	where uploaded videos are stored
NEXT_PUBLIC_FOGDET_BASIC_UI	0	1 = minimal single-video UI
NEXT_PUBLIC_FOGDET_BACKEND_URL	empty	only set to split the frontend onto another host
FOGDET_ALLOWED_ORIGINS	127.0.0.1:4001	browser origins allowed to call the API (set on the backend for remote/split-host setups)

Validate

uv run sam3d doctor --json                       # environment + model files
uv run python -m unittest discover -s tests      # backend tests
cd web-viewer && npx tsc --noEmit && npm run build

Layout

FoG-Detection/
  input/         source videos (selected from the UI or CLI)
  output/        processed runs, analyses, logs
  models/        local model weights (gitignored)
  src/           Python backend: inference, streaming detection + analysis (sam3d)
  tests/         backend tests
  web-viewer/    Next.js 3D viewer UI + API
  vendor/        SAM 3D Body, SAM 3, and MLX SAM 3 upstream code (required to run)
  docs/          FoG signal reference + screenshots