Workflow Engine

Durable workflow automation on Django + PostgreSQL. Users define workflows as trigger → condition → action graphs; a Postgres-backed job queue executes them reliably, with retries, backoff, dead-lettering, and idempotency. The frontend is server-rendered with HTMX and Alpine.

Run locally

Requires Python 3.11+ and a running PostgreSQL.

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# Generate a Fernet key and paste it into .env as FERNET_KEY:
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# Create the database (matches DATABASE_URL in .env):
createdb workflow_engine

python manage.py migrate
python manage.py seed

# Terminal 1 — web UI at http://127.0.0.1:8000
python manage.py runserver
# Terminal 2 — the worker loop
python manage.py run_worker

Open the seeded workflow, trigger it with {"amount": 250} (true branch: delay → HTTP) or {"amount": 50} (false branch: log), and watch the run timeline update live. The timeline's Manual tick button advances the worker on demand without waiting for the loop; Retry run re-queues a failed run.

Run two workers at once (python manage.py run_worker in two terminals) to see parallel-safe claiming — no job is ever processed twice.

Architecture decisions & trade-offs

Queue on Postgres, not a broker. Jobs are StepExecution rows. Workers claim them inside a transaction with SELECT ... FOR UPDATE SKIP LOCKED, so N workers never grab the same row. Trade-off: no push delivery and polling latency, versus zero extra infrastructure, transactional enqueue-with-state, and one datastore to operate. At this scale that is the right call; a broker (Redis/RabbitMQ/Celery) adds moving parts we do not need.

skip_locked for concurrency. The claim query locks only the rows it selects and skips rows locked by other workers, giving lock-free horizontal scaling without advisory locks or a leader.

Templates + HTMX vs. SPA. The engine is the product; the UI is thin. HTMX gives live timelines (polling), the four-state save form, and inline error recovery with no build step and no client state to keep in sync. A SPA would add a toolchain and a second source of truth for little benefit here.

Own expression interpreter vs. a library / eval. Conditions and {{...}} interpolations run through a small tokenizer + recursive-descent parser supporting > < >= <= == != and or () over templated values. No eval/exec — untrusted workflow definitions can never execute arbitrary code. Trade-off: a deliberately minimal grammar instead of a full expression language.

Definition cache. Active workflow graphs are cached (django.core.cache) keyed by workflow_id + updated_at, so any edit/activate/archive (which bumps updated_at) misses naturally; the engine still falls back to the DB, so the cache is safe across worker processes.

Assumptions (unspecified in the brief)

• Editing is draft-only. A workflow is mutable while draft; activation is one-way to active, and active/archived graphs are immutable. This prevents deleting nodes that historical runs' steps reference. Iterate on a draft, then activate; to change an active workflow, create a new one.
• dedup_key = run-<run>:node-<node>, stable across retries. Enqueue uses get_or_create, so a node reachable by multiple paths is queued once (fan-in has no join/barrier semantics — a node runs when first enqueued).
• delay stores its wake time in output and reschedules next_retry_at without consuming an attempt; http_request treats status ≥ 400 as a retryable failure and sends Idempotency-Key: <dedup_key>.
• max_attempts is read from each node's config (default 3). Backoff is exponential (base 2s, cap 300s) with jitter.
• Auth is out of scope; the API is open. Secret values are encrypted at rest with Fernet and never returned by the API.

Index justification

• StepExecution(status, next_retry_at) — the worker's hot path filters status='pending' AND next_retry_at <= now; this composite index makes claiming ready jobs an index range scan instead of a table scan.
• StepExecution.dedup_key unique — enforces per-step idempotency at the DB and backs the get_or_create enqueue lookup.
• Run(workflow, status) — dashboard filters and status aggregations per workflow.
• StepExecution(run) — loading a run's timeline fetches all its steps by run.

Deploy (Render)

render.yaml provisions a PostgreSQL database, a web service (gunicorn config.wsgi, static files via WhiteNoise), and a worker service (python manage.py run_worker). Build installs requirements and runs collectstatic; migrate runs as the pre-deploy step. Set FERNET_KEY in the workflow-engine-secrets group; SECRET_KEY is generated and DATABASE_URL is wired from the database automatically.

API

POST /api/workflows (create/update draft, rejects cycles) · POST /api/workflows/{id}/activate · POST /api/workflows/{id}/trigger · GET /api/runs/{id} (run + step timeline) · POST /api/runs/{id}/retry · GET|POST /api/secrets, GET|DELETE /api/secrets/{key}. Errors are typed: { "error": { "code", "message", "details" } }.