Workflow Engine

A full-stack, visual workflow automation engine built with Node.js, React Flow, PostgreSQL, and Redis. Design pipelines on a drag-and-drop canvas, then execute them through a distributed Bull queue with sandboxed JavaScript and Python runtimes.

---

Features

• Visual editor — React Flow canvas with custom nodes, drag-and-drop, and live config panels.
• Distributed execution — Express API enqueues jobs to Redis (Bull); a separate worker process consumes and runs them.
• Sandboxed runtimes — JavaScript transformer nodes run inside Node's native vm module; Python nodes execute in an isolated container.
• Persistent state — Workflows, executions, users, and audit logs are stored in PostgreSQL via Knex migrations and repositories.
• Real-time updates — WebSocket channel streams execution status and logs to the UI.
• Auth & hardening — JWT auth, bcrypt password hashing, Helmet, CORS, and express-rate-limit.
• Node types — HTTP request, PostgreSQL query (parameterized), JavaScript transformer, Python script, and conditional branching.

---

Architecture

graph LR
   UI[React + Vite SPA] -->|REST + WS| API[Express API]
   API --> DB[(PostgreSQL)]
   API --> Q[(Redis / Bull)]
   Q --> W[Bull Worker]
   W --> JS[JS vm Sandbox]
   W --> PY[Python Executor]
   W --> DB

Loading

• Frontend (frontend/) — Vite + React 18 + React Flow + Tailwind, talking to the backend via Axios and WebSockets.
• Backend API (backend/src/) — Express app exposing REST routes, WebSocket gateway, and queueing jobs to Bull.
• Worker (backend/src/queue/worker.js) — Out-of-process Bull consumer that runs node implementations.
• Python executor (python-executor/) — Container hosting the Python runtime used by Python nodes.

---

Project structure

.
├── backend/
│   ├── src/
│   │   ├── api/              # routes, controllers, websocket
│   │   ├── core/             # engine, executor, scheduler
│   │   ├── db/               # knex migrations + repositories
│   │   ├── middleware/       # security, errorHandler
│   │   ├── nodes/            # http, database, python, transformer, conditional
│   │   ├── queue/worker.js   # Bull worker entry point
│   │   ├── services/         # authService
│   │   └── index.js          # API entry point
│   ├── knexfile.js
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── components/       # CustomNode, ConfigModal, LoginModal
│   │   ├── services/api.js   # Axios client
│   │   └── App.jsx
│   └── Dockerfile
├── python-executor/
│   ├── Dockerfile
│   └── requirements.txt
├── database/init.sql
└── docker-compose.yml

---

Quick start (Docker)

docker-compose up -d --build

Services and ports:

Service	URL / Port
Frontend (Vite)	http://localhost:3001
Backend API	http://localhost:3002
PostgreSQL	localhost:5432 (workflow_user / workflow_pass, db workflow_engine)
Redis	localhost:6379

Default login (seeded via database/init.sql / backend/reset_admin.js):

• Username: admin
• Password: admin123

Change JWT_SECRET, DB credentials, and the admin password before deploying anywhere non-local.

---

Local development (without Docker)

You'll need Node.js ≥ 18, PostgreSQL, and Redis running locally.

# 1. Backend API
cd backend
npm install
npm run migrate
npm run dev          # starts API on PORT (default 3002)

# 2. Bull worker (separate terminal)
cd backend
npm run worker

# 3. Frontend
cd frontend
npm install
npm run dev          # Vite dev server on 3001

Environment variables used by the backend (see docker-compose.yml for defaults):

• PORT — API port (default 3002)
• DATABASE_URL — Postgres connection string
• REDIS_HOST, REDIS_PORT
• JWT_SECRET
• FRONTEND_URL — used for CORS

---

Useful scripts

Backend (backend/package.json):

• npm run dev — start API with nodemon
• npm run worker — start the Bull worker
• npm run migrate / migrate:rollback — Knex migrations
• npm test — Jest tests
• npm run lint / format

Frontend (frontend/package.json):

• npm run dev — Vite dev server
• npm run build — production build
• npm run preview — preview built bundle

---

Node types

Node	What it does
HTTP	Issues an HTTP request via Axios and returns the response.
Database	Runs a parameterized SQL query against PostgreSQL.
Transformer	Executes user JS inside a sandboxed vm context.
Python	Runs a script in the Python executor container.
Conditional	Branches based on a predicate evaluated against node input.

Implementations live in backend/src/nodes.

---

License

MIT — see LICENSE.