dxforge is the orchestration suite within the Divergex ecosystem, designed to serve service meshes, secure communication, and containerized infrastructure management for high-performance quantitative research and trading systems.
It is basically just a multi-tenant scheduling serverless platform, but self-hosted on your own infrastructure.
The main purpose of dxforge is to use it to orchestrate and schedule processes. Suppose you have a small prop trading setup, for a retailer or family member, or even a small desk at a trading firm.
We believe that everyone should own their own trading setup, and dxforge is the tool to help you start with that. At some point, you will want a more robust syste, to organize your strategies (or strategy! dxforge is aimed to both the small and large investors).
Moved to dxcore.
- Different users have different needs, and dxforge aims to allows you to run your services either in isolated containers or directly on your host machine. This should ensure that dependencies and configurations between strategies do not conflict depending on your needs.
- Works with Docker and (in future also Kubernetes and Podman) to deploy, manage, and scale services.
These are our main objectives/requiisites for all future features
- Scalability: Easily scale services up and down based or number of strategies.
- Fault Tolerance: Incorporate resiliency features like retries, fallbacks, and circuit breakers.
- Interoperability: Full integration with (but still decoupled to) other components of the divergex ecosystem, including
cadlaganddxlib.
- Python 3.12+
- Docker with the
composeplugin
The compose file, the container bootstrap scripts, and the container image reference ship inside the package:
pip install dxforge
forge init ~/<your-stack>k # renders docker-compose.yml, bootstrap/, .env.example, secrets/
forge up --dir ~/dxforge-stack # starts Postgres, MinIO, OpenBao and the registry
forge bootstrap --dir ~/dxforge-stack # bootstraps credentials, service accounts and migrations
forge run api --dir ~/dxforge-stackforge init writes the published image tag matching the installed dxforge version into docker-compose.yml; re-run it with --force after
upgrading. Commands resolve the stack directory from --dir, then $FORGE_STACK_DIR, then the working directory.
Root secrets are generated once into <stack>/secrets and never rewritten. forge down --volumes --dir <stack> erases all stack data and
credentials. The images above the dxforge one (Postgres, MinIO, OpenBao, registry) are pinned by digest; re-resolve a pin with
docker buildx imagetools inspect <image>:<tag> --format '{{.Manifest.Digest}}'git clone https://github.com/divergex/dxforge.git
cd dxforge
python -m venv .venv && source .venv/bin/activate
pip install -e .
make setup # forge init .stack --force && forge bootstrap --dir .stackmake reset tears the stack down and deletes its volumes and generated secrets. The compose file is generated, so forge init .stack --force
re-renders it after template changes.
Once set up, you can configure your services to interact via dxforge for service discovery, mesh registration, and secure communication. To enable these features:
- Service Registration: Register services with the built-in discovery mechanism.
- Inter-Service Communication: Utilize the provided protocols and channels for secure messaging.
- Scaling: Leverage Kubernetes or Docker Swarm to scale services automatically based on load.
Start the api and access it at http://localhost:8000 with
forge run api
Code runs as a host process (executor_backend: "host", the default). Upload a zip, build a version,
create a function, then submit a schedule (POST /api/v1/schedules); a worker (forge worker start) starts it.
- Schedules:
trigger_atis set to the submit (or update) time, so the schedule fires as soon as the worker sees it; the worker is woken immediately through PostgresNOTIFY, no waiting for the poll interval.rule: nullmeans run once and never again; with arule({"every": ...}/{"dates": ...}) it repeats after that first run. To run again,PUTthe schedule: any update re-arms it. - Timeouts (per function):
hard_timeout_secondskills the process after that long regardless (null= never; default 300).health_url(loopback only) is polled every few seconds; if it does not answer 2xx foralive_timeout_seconds(counted from process start) the process is killed with statusunhealthy. - Decoupled execution: each execution gets its own detached supervisor process that owns the program,
its logs, heartbeat and timeouts. Stopping the API or the worker does not stop running programs; a restarted
worker adopts them, and closes ones whose supervisor died as
lost. - Observe / stop:
GET /executions/{id},GET /executions/{id}/logs(live while running),POST /executions/{id}/stop. Statuses:running,success,failed,timeout,unhealthy,stopped,lost.
Host processes run on the dxforge host with its network. Supervisors read their state from
FORGE_EXECUTION_DIR (default ./var/executions), which API and workers must share.
If you'd like to contribute to the development of dxforge, see the CONTRIBUTING guidelines for details on how to get started.
This project is licensed under the MIT License - see the LICENSE file for details.
