Open Refinery

A factory with the lights on.

---

open-refinery is a self-hosted control plane for AI-driven software work. Teams define the pipeline their work moves through — the stage graph a run follows from a ticket to a pull request — connect their repositories, and ship work through it. Every step is owned, authorized, recorded, and queryable. It runs "dark" (lights-out automation) but stays "open": nothing happens without an attributable, auditable trail.

Before you adopt it, read LIMITATIONS.md. It is short, and it is the half of the picture most READMEs leave out: SQLite and one process, a schema frozen at 1.0, a pre-1.0 dependency carrying a pillar, and exactly which rungs of the ladder this product does not carry. Every item there is a deliberate boundary or a known gap, not a surprise waiting in production.

Try it in two minutes

No accounts, no API key, no network. The local forge writes a pull request as a file and the offline stub stands in for the model, so the whole governed loop runs on a throwaway repository seed makes for you:

make setup          # .env + a seeded database + a demo git repository
make dev &          # then sign in at http://localhost:8000

Sign in as owner@example.com / owner — it holds every permission, so every screen is reachable. Name a ticket under Work, ship it, and watch the run hold at the plan gate.

What it is

The platform layer between your harnesses (agents, scripts, CI — the in-process, app-owned side) and your targets (repos, models, tools). It governs how work reaches those targets: identity and roles, authorization, provenance, an append-only audit trail, oversight gates, and metrics. Install it on any VPS with pip install and one command; manage everything from the web dashboard.

What it does

• Governance policy layer — org-wide (effect, role, action, resource) rules with deny-overrides, plus strict rules a lower layer can't override. Policies are authored and governed: a change to governance walks a per-layer approval workflow (accept / deny / feedback). Packs seed starter rules, skills, commands, and standards (TDD, ATDD, spec-driven, UI verification, tech-debt, infrastructure, org policy…).
• Configurable oversight strategy — a per-repository human-oversight dial (L0 manual → L4 fully dark) deciding how much of a run waits for a person. ask never degrades to allow: unattended, it refuses instead.
• Human approval gates — a gated stage holds the run and waits. Clearing it needs approve:code, and nobody clears their own run. Changes to governance go through a separate per-layer approval workflow with an ordered chain and a distinct signer per slot.
• Proactive enforcement — an org-wide mode of audit (default-allow, opt-in deny) or strict (whitelist / default-deny). A pre-action authorize seam lets a harness verify identity + intent against policy before it runs a tool / command / host-egress action; per-namespace whitelists scope rules; every refused attempt is audited.
• Teams, concurrency caps & spend ceilings — group users into teams; a team's live in-flight cap bounds how many runs it can have going at once, and budgets cap what they may spend: max_run_units on a repository bounds one run, and an org / team / repo budget over a rolling window bounds the burn. GET /usage reports what runs actually cost.
• Connects your code hosts and issue trackers — GitHub, GitLab (code hosts); GitHub Issues, Jira, Linear (issue trackers), connected by token or OAuth, credentials encrypted at rest. Trackers expose workflow discovery — the tool's own columns/statuses.
• Ships work through customizable pipelines — a stage graph with gates and feedback loops, versioned, and pinned per run.
• Records a complete, attributed audit trail — who did what, to which work item, with what inputs — fans it out to webhooks, and derives metrics plus debt-audit health scores (factory / harness / charter) with insights.
• Runs live — background job runner, scheduled ingest, and a WebSocket live channel (job status, new audit events, per-run live logs) feeding a visibility-first dashboard (an overview that surfaces what needs attention, a work board, and a right-hand detail/action drawer).
• Enforces ownership with admin-configurable roles: developers see their own work; platform sets org-wide policy; admins audit everything.

What it doesn't do

• It is not a CI runner, a build system, or an agent framework — it governs work, it doesn't execute your builds or own your prompts.
• It is not multi-tenant SaaS. One install serves one organization, self-hosted and single-tenant by design.
• It doesn't hide anything: no opaque automation, no un-owned actions.

Philosophy & goals

Automation you can't audit isn't trustworthy. open-refinery's goal is to make lights-out software work safe to trust by making it legible — every production authorized, owned, provenanced, and logged, with human oversight configurable to each team's philosophy. Minimal to run (one process, SQLite, env-light), everything managed through the UI, and completely open source.

The orchestrator is a queue, not an agent. Work advances through the stage graph via deterministic code (workers over a durable store), not an LLM deciding what happens next. That determinism is the point: it's cheap (no model call to move a step), reproducible, and auditable — the agent's judgment is confined to the work inside a step, while sequencing stays plain software. It also keeps work items partitionable: independent items advance through the queue without intercommunication, which — per Brooks in The Mythical Man-Month — is the condition under which adding effort actually adds throughput, whereas work requiring communication incurs overhead that a central agent bottleneck would impose.

Status: 2.0.0 — feature-complete platform (schema frozen at 1.0; every release since is additive, backward-compatible — no breaking change at 2.0). The full governed loop plus the platform around it is in place: admin- configurable roles; policies as authored harness artifacts (rule / skill / command / agent) with a strict override lock and layered precedence; proactive enforcement (audit / strict modes, a pre-action /authorize gate, per-namespace whitelists); packs — a curated marketplace of starter standards; per-layer approval workflows that govern changes to governance itself; teams + concurrency caps; governance landscape + analysis; repo coverage & debt-audit health with GitHub ingest (on a schedule); evals & experiments; webhooks; background jobs and a WebSocket live channel with per-run live logs; oversight, the async metrics, and a full audit trail — behind a visibility-first dashboard. Config lives in the database, not the env — encrypted, UI-managed, so only SECRET_KEY is required in the environment. Self-hosted API docs with live "Try it out" at /api-docs. See CHANGELOG.md.

Quickstart

Requires Python 3.11+. SQLite ships with Python — there is no separate database to install.

pip install open-refinery        # or: uv pip install open-refinery
open-refinery serve              # server + dashboard on port 8000

Open http://your-host:8000 — on a fresh instance the dashboard walks you through creating the first admin (no CLI needed), then signs you in. From there, manage repos, pipelines, work, oversight, and the audit trail across nine screens. The UI (React + shadcn/ui, light/dark/auto themes) is bundled in the package — no Node to run.

Prefer the CLI to seed the admin? open-refinery create-admin --email you@x.dev still works.

Background it on a VPS however you like — &, nohup, screen, tmux, or your process manager:

open-refinery serve --port 9000 &                # or: PORT=9000 open-refinery serve
curl localhost:9000/health                       # {"status": "ok"}

Upgrading

Schema migrations run automatically on startup (serve). To apply them explicitly after upgrading the package — before starting the server — run:

pip install -U open-refinery
open-refinery migrate                            # applies pending migrations
open-refinery migrate --to 9 --yes               # pin an older schema (down; destructive)

The schema is frozen at 1.0 — upgrades are additive only, so existing data is preserved. Downgrading to a pinned version drops the newer columns (and their data), so it requires --yes.

Using the API

Authenticate every request with the admin token from create-admin:

TOKEN=<paste the token>
H="Authorization: Bearer $TOKEN"

# register a repository (a repo = a project, whatever the code architecture)
curl -s -H "$H" localhost:9000/repositories \
  -d '{"name":"my-app","git_url":"git@github.com:me/my-app.git"}'

# how closely runs in this repo are watched (manual → dark)
curl -s -X PUT -H "$H" localhost:9000/repositories/<repo> \
  -d '{"oversight":"supervised"}'

# name a ticket, then put it through the factory
curl -s -H "$H" localhost:9000/work-items \
  -d '{"repo_id":"<repo>","title":"CVE-1234"}'
curl -s -H "$H" localhost:9000/runs -d '{"work_item_id":"<item>"}'

# what is waiting on a person, and clearing it
curl -s -H "$H" localhost:9000/approvals
curl -s -H "$H" localhost:9000/runs/<run>/approve -d '{}'

# what it cost, and a ceiling so it cannot cost that again
curl -s -H "$H" localhost:9000/usage
curl -s -H "$H" localhost:9000/budgets \
  -d '{"scope":"org","limit":5000000,"window_seconds":86400}'

# read the audit trail — every move, owned and attributed
curl -s -H "$H" "localhost:9000/events?subject=<item>"

Config is env-only, all optional: PORT (or --port), DATABASE_URL (sqlite:///open-refinery.db default), LOG_LEVEL.

Sign in with GitHub (optional)

Set these and the dashboard shows a "Sign in with GitHub" button:

export GITHUB_CLIENT_ID=...       # from your GitHub OAuth App
export GITHUB_CLIENT_SECRET=...
export APP_BASE_URL=https://or.example.com   # optional; used to build the callback

Register the OAuth App's callback as <APP_BASE_URL>/auth/github/callback. A GitHub login is accepted only when its verified primary email matches an existing user — provision accounts first (an admin creates them in the UI); unknown emails are denied. (Roadmap: OAuth client id/secret move to DB settings, editable in the UI by platform/admin — so only SECRET_KEY is required in the environment; everything else lives in the database.) API/CI accounts keep using tokens.

Library

open-refinery is also an embeddable core — the same governed-production loop without the server:

from open_refinery import Factory

factory = Factory()

@factory.recipe("upper")
def upper(text: str) -> str:
    return text.upper()

artifact, record = factory.produce("upper", actor="ian", text="hello")

open-refinery demo prints one such record.

Pillars

Pillar	Where it lives
Authorization	Authorizer (AllowAll, AllowList) — checked before produce
Provenance	Record — recipe, actor, timestamp, input/output digests
Ownership	owner on every record (defaults to the actor)
Auditability	AuditSink (MemorySink, JsonlSink) — append-only trail
Logging	stdlib logging, logger name open_refinery
Oversight	Per-repository autonomy levels L0–L4; a gated stage holds the run for a person
Observability	GET /metrics — WIP, event counts, per-actor activity, lead times; per-run live logs
Governance	Policy layer (audit / strict enforcement, layered strict overrides, per-namespace whitelists) + pre-action /authorize gate
Cost & limits	Per-run and org/team/repo spend ceilings in model units, metered off the provider's own usage; live per-team concurrency caps

Durable audit trail

from open_refinery import Factory, JsonlSink

factory = Factory(audit=JsonlSink("audit.jsonl"))

Each production appends one JSON line — a replayable record of who produced what, from which inputs, and when.

Development

make install            # backend: uv sync --extra dev
make test               # uv run pytest
make serve              # run the server locally
make help               # list all tasks

open-refinery seed      # optional: load EXAMPLE data + login tokens (never auto-run)

Frontend (dashboard) lives in frontend/ — React + TypeScript + Vite + Tailwind

• shadcn/ui, built with bun:

make ui-dev             # Vite dev server (proxies API to :8000)
make ui                 # build the SPA into the package
make dist               # build UI + wheel (the wheel bundles the SPA)

The build step is release-time only; end users never need bun/node.

For a full fresh-VPS walkthrough — install, background the server, create the admin, ports, and HTTPS/TLS on a public host — see USER_GUIDE.md.

See also PLAN.md, CONTRIBUTING.md, and docs/ARCHITECTURE.md.

Credits

The harness-vs-platform framing that shapes open-refinery's design — the platform as the out-of-process governance layer that harnesses call through — draws on Traefik Labs' mental model: Harness engineering vs platform engineering.

The deterministic-queue orchestrator — the orchestrator is the queue (plain code), not an agent, for cost, determinism, and parallelism — is inspired by Mike Piccolo's Loop Engineering.

The stance that decomposition belongs in deterministic orchestration rather than agent-spawned sub-agents (sub-agent delegation is a harness concern, not the platform's) draws on Anton Vinogradov's "You don't need sub-agents".

License

MIT © Ian Johnson