[marcvergees]

Closes #748

---

Overview

This PR implements the backend half of the local-first desktop production deployment for FireForm.
It makes the FastAPI application fully self-contained: a single PyInstaller binary starts the
server, initialises and migrates a SQLite database, and runs all background work in-process —
no Docker, no PostgreSQL, no Redis, no Celery worker required on the user's machine.

Server and Docker deployments are completely unaffected. Desktop mode is opt-in via a single
environment variable (FIREFORM_DESKTOP=true). Every decision below was made with this hard
constraint in mind.

---

Decision Log

1. CI Release Assets (Approach A) over shipping a pre-built binary in the repo

Decision: PyInstaller binaries are built in CI on every v* tag push and attached to the
GitHub Release. The fireform-frontend release workflow downloads the matching platform binary
before running electron-builder.

Rationale:

• Binaries are platform-specific and change frequently; storing them in git would bloat the repo
  history permanently.
• Approach A (CI assets) decouples backend and frontend release cadences: the frontend can consume
  any tagged backend binary without a full monorepo sync.
• GitHub Releases are already the distribution channel for Electron installers; attaching backend
  binaries to the same release keeps everything in one place.
• The alternative (embedding the binary in the frontend repo) would require the frontend to own
  Python build tooling, which is a worse separation of concerns.

Where: .github/workflows/build-desktop-binaries.yml

---

2. SQLite over PostgreSQL for desktop; PostgreSQL remains for server

Decision: When FIREFORM_DESKTOP=true (or DATABASE_URL is not set), the backend defaults
to sqlite:///{DATA_DIR}/fireform.db. PostgreSQL is still used for all Docker/server deployments.

Rationale:

• PostgreSQL as an embedded binary requires shipping and managing a ~40 MB server process, a
  pg_ctl lifecycle, WAL housekeeping, and an OS-level service on every target platform. This
  is the problem we are explicitly trying to avoid.
• SQLite is a file, not a service. It lives at a known path, starts instantly, needs no init,
  and is effectively impossible to corrupt during a clean shutdown.
• All existing migrations are pure SQLAlchemy/SQLModel with no PostgreSQL-specific types or
  SQL. Switching the connection string is the only change required at the ORM layer.
• Tests already use SQLite in-memory (conftest.py). The existing postgres CI service was dead
  weight — it is removed in this PR.
• The only SQLite incompatibility in Alembic is ALTER TABLE … ALTER COLUMN / DROP COLUMN,
  which SQLite does not support natively. This is solved by setting render_as_batch=True in
  alembic/env.py, which rewrites those statements as table-copy-and-swap.

What we did NOT do: migrate the Docker stack to SQLite. It runs PostgreSQL as before.

Where: app/core/config.py (URL defaulting), app/db/database.py (WAL + foreign keys
pragmas), alembic/env.py (batch mode).

---

3. Remove Redis; replace Celery worker with in-process eager mode

Decision: In desktop mode there is no Redis broker and no Celery worker process. Celery is
configured with task_always_eager=True and a memory:// broker so tasks run synchronously
in the same process that enqueues them.

Rationale:

• Redis is a network service. Embedding it would require the same binary-management story we
  rejected for PostgreSQL.
• The tasks FireForm runs in desktop mode (PDF generation, LLM inference) are either fast
  enough to block on or are already async (the LLM streaming route does not use Celery at all).
• Celery's eager mode is an officially supported configuration, not a hack. It is equivalent to
  calling the task function directly while keeping all the application code unchanged.
• Celery Beat (the periodic task scheduler) is also disabled in desktop mode: the daily
  submission purge is not critical for single-user local use. A future iteration can use
  APScheduler or a threading.Timer if retention becomes important.
• Server deployments continue to use Redis and Celery workers exactly as before.

Where: app/core/celery.py, app/core/config.py (CELERY_BROKER_URL default).

---

4. PyInstaller as the packaging tool

Decision: The FastAPI application is compiled into a single executable using PyInstaller
with a hand-written spec file (fireform-backend.spec).

Rationale:

• PyInstaller produces a single directory or a single file that contains the Python runtime,
  all dependencies, and the application code. No Python installation required on the user's
  machine.
• Alternatives considered:
  • Nuitka: faster runtime, but compilation is significantly slower and the toolchain is
    less stable for FastAPI + SQLAlchemy.
  • Shiv / zipapp: still require a Python interpreter on the target machine.
  • Docker: explicitly ruled out for desktop mode (the whole point is zero infrastructure).
• The spec file excludes torch, torchvision, ultralytics, cv2, and scipy — these are
  present in .venv (pulled in by other optional services) but are never imported by the FastAPI
  app itself. Excluding them reduces the binary from ~600 MB to ~68 MB.
• Non-Python assets (alembic/, alembic.ini, data/inputs/file_template_manual.pdf,
  app/services/form_filler/prompt.txt) are bundled via the datas list in the spec.
• sys._MEIPASS detection in config.py ensures BASE_DIR resolves to the bundle root at
  runtime, so all relative-path lookups work identically inside and outside the binary.

Where: fireform-backend.spec, app/core/config.py (frozen detection), app/desktop.py
(entry point).

---

5. FIREFORM_DESKTOP as the single feature flag

Decision: A single boolean env var (FIREFORM_DESKTOP) gates all desktop-specific behaviour.
It defaults to false; app/desktop.py sets it to true before importing the app.

Rationale:

• One flag is easier to reason about than a combination of DATABASE_URL, CELERY_BROKER_URL,
  WHISPER_ENABLED, etc. The derived defaults flow from that one decision point.
• Keeps the Docker .env.example clean — desktop users don't set this; server operators never
  need to know it exists.
• Testable: any test can set FIREFORM_DESKTOP=true to simulate desktop mode.
• Decisions driven by FIREFORM_DESKTOP:
  • DATABASE_URL default → SQLite
  • CELERY_BROKER_URL default → empty string → in-process eager mode
  • WHISPER_ENABLED default → false
  • DB_ECHO default → false (quieter logs in a GUI context)

Where: app/core/config.py.

---

6. OS data paths via FIREFORM_DATA_DIR (passed from Electron)

Decision: The backend does not resolve OS-specific paths itself. Electron calls
app.getPath('userData') and passes the result as FIREFORM_DATA_DIR. The backend stores all
mutable data under that directory.

Rationale:

• app.getPath('userData') is the cross-platform, OS-sanctioned location for per-user
  application data. It maps to:
  • macOS: ~/Library/Application Support/FireForm
  • Windows: %LOCALAPPDATA%\FireForm
  • Linux: ~/.config/FireForm
• Keeping the path resolution in Electron (which owns the app identity and userData root) is
  cleaner than duplicating platform detection in Python.
• The backend only needs to know the directory; it does not need to care which OS it is on.
• DATA_DIR is created by config.py on import (mkdir(parents=True, exist_ok=True)), so
  the backend is resilient to first-run scenarios where the directory does not yet exist.
• paths.py and purge.py were updated to treat DATA_DIR as a trusted root alongside
  PROJECT_ROOT, so path-safety checks do not block operations on user data files.

Where: app/core/config.py (DATA_DIR, DATA_DIR.mkdir), app/core/paths.py,
app/tasks/purge.py, Electron src/main/index.ts.

---

7. Whisper disabled in desktop mode

Decision: Voice transcription is unavailable in the desktop build. The health check returns
healthy (not degraded) when WHISPER_ENABLED=false.

Rationale:

• The Whisper service runs in a separate Docker container. Embedding it in the PyInstaller
  binary would add ~1 GB of model weights and GPU/CPU inference libraries to the bundle.
• Voice transcription is not in scope for the initial desktop release (issue [FEAT]: Desktop Production Deployment #748 acceptance
  criteria do not mention it).
• Disabling it cleanly via a flag is better than leaving the health check in a permanently
  degraded state, which would confuse users and monitoring.
• When voice is needed in a future release it can be re-enabled by running the whisper container
  alongside the binary and setting WHISPER_ENABLED=true WHISPER_HOST=http://....

Where: app/core/config.py (WHISPER_ENABLED), app/api/routes/system.py (health check).

---

8. Alembic migrations run at startup; render_as_batch=True for SQLite

Decision: app/db/init_db.py runs alembic upgrade head every time the backend starts.
Alembic is configured with render_as_batch=True for SQLite connections.

Rationale:

• Running migrations at startup is the simplest way to keep the schema in sync across app
  updates without requiring the user to run a CLI command.
• It is idempotent — if the schema is already at head, Alembic does nothing.
• render_as_batch=True is Alembic's documented solution for SQLite's lack of native
  ALTER TABLE … ALTER COLUMN. It rewrites those operations as:
  1. Create a new table with the target schema.
  2. Copy all rows.
  3. Drop the old table.
  4. Rename the new table.
     This is fully transparent to the application.
• ALEMBIC_INI is resolved from BASE_DIR (not __file__) so it works both from the repo
  and from inside the PyInstaller bundle.

Where: alembic/env.py, app/db/init_db.py.

---

9. CI: drop the postgres service from tests.yml

Decision: The services: postgres: block is removed from .github/workflows/tests.yml.
The alembic migration check now targets a SQLite database.

Rationale:

• The test suite already used SQLite in-memory via conftest.py. The postgres service was
  never used by any test — it was a resource that spun up and was immediately ignored.
• Removing it cuts ~30 seconds off every CI run (postgres container startup + health check).
• The alembic migration check (alembic upgrade head && alembic check) is now run against a
  fresh SQLite file, which is what the desktop binary will actually use.

Where: .github/workflows/tests.yml.

---

Files Changed

File	Type	Purpose
app/desktop.py	➕ New	PyInstaller entry point; sets FIREFORM_DESKTOP=true, starts uvicorn on 127.0.0.1
fireform-backend.spec	➕ New	PyInstaller spec; bundles assets, excludes heavy ML libs
.github/workflows/build-desktop-binaries.yml	➕ New	Matrix CI: macOS + Ubuntu + Windows binary builds on v* tags
app/core/config.py	✏️ Modified	FIREFORM_DESKTOP flag; SQLite default URL; DATA_DIR auto-create; WHISPER_ENABLED; DB_ECHO; Celery broker defaults
app/db/database.py	✏️ Modified	SQLite WAL mode + PRAGMA foreign_keys=ON via SQLAlchemy event
app/core/celery.py	✏️ Modified	In-process eager mode (memory:// broker, task_always_eager=True) when FIREFORM_DESKTOP
alembic/env.py	✏️ Modified	render_as_batch=is_sqlite in both offline and online migration paths
app/core/paths.py	✏️ Modified	DATA_DIR added to allowed path roots for file-safety checks
app/tasks/purge.py	✏️ Modified	_safe_delete_file allows files under DATA_DIR
app/api/routes/system.py	✏️ Modified	Health check returns healthy immediately when WHISPER_ENABLED=false
app/db/init_db.py	✏️ Modified	ALEMBIC_INI resolved from BASE_DIR (PyInstaller-safe)
.github/workflows/tests.yml	✏️ Modified	Removed postgres service; alembic check uses SQLite
docs/DESKTOP.md	✏️ Modified	Full deployment guide (arch, data paths, CI pipeline, dev setup, troubleshooting)
.gitignore	✏️ Modified	Added build/, dist/, .pyinstaller_cache/, *.spec.bak

---

Verification

PYTHONPATH=. .venv/bin/pytest tests/ → 436 passed, 2 warnings ✅
DATABASE_URL="sqlite:///test.db" alembic upgrade head → 7 migrations applied cleanly ✅
PyInstaller build → dist/api-backend (68 MB, macOS arm64) ✅
PORT=8005 FIREFORM_DATA_DIR=/tmp/ff GET /api/v1/health → 200 healthy ✅
PORT=8005 FIREFORM_DATA_DIR=/tmp/ff GET /api/v1/templates → seeded template returned ✅

---

What This Does NOT Change

• Docker Compose development workflow (make fireform, make up) — untouched.
• PostgreSQL schema or migrations — SQLite uses the same migration files.
• Any existing API contract — same FastAPI routes, same response shapes.
• Production server deployment — PostgreSQL + Redis + Celery worker unchanged.

[vharkins1]

Hey @marcvergees, really nice work on this. I built both PRs into an actual FireForm.app to try it, and a few things keep it from running yet:

• backend exits on launch since alembic's script_location resolves from the cwd (setting it to BASE_DIR / "alembic" fixed it for me)
• uploads and filled PDFs go to the PyInstaller temp dir, so they're gone after a restart
• AI Schema fails with No module named 'ultralytics', and /forms/jobs with No module named 'celery.loaders.app'
• packaged window is blank on fe fix: resolve off-by-one whitespace stripping bug in handle_plural_values #65, loadFile needs ../renderer/index.html , so it didn't boot when I ran it as an app.

The fix I found for it was changing the packaged window which is blank on fe #65, loadFile needed ../renderer/index.html instead of ../../

Also needs a quick rebase on development (small conflict in celery.py). Happy to help on any of it, excited to get this shipped!

[chetanr25]

Hi @marcvergees, Over all PR description looks good to me, I have few questions:

Configuration in production

Here I can see that in desktop mode SQLite is used by default when DATABASE_URL is missing. As developers we can change any configuration through .env.dev, but how will users update the configuration once this goes to production?

We won't be shipping a .env inside the PyInstaller package, and the secrets should come from the host machine anyway. Right now everything comes from env variables that Electron passes when it starts the binary, and the host is fixed to 127.0.0.1 in desktop.py, so there is no way for a user to change anything after install.

I think we need two things here:

• An option to add the configuration during setup, like the database URL, the LLM provider and its API key.
• An option to change these configurations later from the app, with a warning, so users can actually update them when something changes.

---

FIREFORM_DESKTOP flag

I came across the FIREFORM_DESKTOP variable. Does this mean that when it is true, both the server and the client app are on the same machine, and when it is false, it is a server only setup? Is my assumption correct?

Rest of the changes

All the other changes look promising, but I need some time to go through the whole implementation properly. I should be able to finish it before Saturday, 14th November. But if you feel we should move forward quickly, then we can go ahead and merge this PR.