Triad — AI-Driven Market Microstructure Trading System

Triad is a research-grade, event-driven algorithmic trading engine written in Rust and Python. It ingests live Level-2 order book data from Binance, computes real-time microstructure signals, and uses a combination of a neural network (ONNX) and episodic memory (HNSW k-NN) to make sub-millisecond paper trading decisions.

---

Table of Contents

1. Architecture Overview
2. System Components
3. Data Flow
4. Signal Glossary
5. Database Schema
6. Prerequisites
7. Infrastructure Setup
8. Model Generation
9. Running the System
10. Project Structure
11. Configuration Reference
12. Testing
13. Roadmap

---

Architecture Overview

Triad is structured as a three-layer pipeline:

┌────────────────────────────────────────────────────────────────────┐
│                        FAST PATH (Rust)                            │
│                                                                    │
│  Binance WebSocket                                                 │
│      │ (trade + depth@100ms)                                       │
│      ▼                                                             │
│  ┌────────────────┐    NATS publish    ┌──────────────────────┐   │
│  │ Witness Node   │ ─────────────────► │ Execution Engine     │   │
│  │ (monitoring/)  │  ObservationEvent  │ (fast-path/)         │   │
│  │                │                   │                        │   │
│  │ • LocalOrderBook│                  │ • ModelLoader (hot)   │   │
│  │ • VpinTracker  │                   │ • MemoryBank (HNSW)   │   │
│  │ • Commitment   │                   │ • OnnxBrain           │   │
│  │   Hash Chain   │                   │ • PaperPortfolio      │   │
│  └────────────────┘                   └──────────┬───────────┘   │
│                                                   │ ClosedEpisode  │
│                                                   │ NATS publish   │
└───────────────────────────────────────────────────┼───────────────┘
                                                    │
┌───────────────────────────────────────────────────▼───────────────┐
│                    SLOW PATH (Python)                              │
│                                                                    │
│  ┌─────────────────────┐      ┌──────────────────────────────┐   │
│  │  episode_store.py   │ ───► │  Postgres (pgvector)         │   │
│  │  (experiencing/)    │      │  • episodes table (128-dim   │   │
│  │                     │      │    context vectors)           │   │
│  │  • JetStream sub    │      │  • IVFFlat ANN index         │   │
│  │  • pgvector insert  │      └──────────────────────────────┘   │
│  └─────────────────────┘                                          │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘
                 ▲ HNSW index rebuilt every 60s from Postgres
                 └── MemoryBank background task (in fast-path)

---

System Components

1. monitoring/ — Witness Node (Rust)

The Witness Node is the data ingestion layer. It connects to the Binance Combined WebSocket Stream and maintains a live, incrementally-updated Level-2 order book.

Key files:

File	Responsibility
src/main.rs	WebSocket connection, reconnection loop, NATS publisher
src/orderbook.rs	[LocalOrderBook]: diff-based order book with spoofing detection; [VpinTracker]: rolling VPIN metric
src/events.rs	Canonical event types (ObservationEvent, ClosedEpisode) and SHA-256 commitment hash chain
src/lib.rs	Re-exports events and orderbook for use by fast-path

Produces: ObservationEvent messages on NATS subject observations.microstructure.BTC-USD

---

2. fast-path/ — Execution Engine (Rust)

The Execution Engine subscribes to the observation stream and makes real-time paper trading decisions using a 3-stage decision pipeline.

Key files:

File	Responsibility
src/main.rs	Main event loop: hot-reload check → exit logic → entry logic
src/model_loader.rs	Watches a binary model file for changes and hot-reloads weights without restart
src/classifier.rs	Numerically-stable softmax classifier for regime detection
src/memory.rs	HNSW-based episodic memory bank with background Postgres refresh
src/onnx_engine.rs	ONNX Runtime wrapper for the neural net decision model

Consumes: ObservationEvent from NATS
Produces: ClosedEpisode on NATS subject episodes.closed.BTC-USD

Decision Pipeline (per tick, ~10 ms)

ObservationEvent
    │
    ├─ [1] Hot-reload check (every 5s) ──► ModelLoader
    │
    ├─ [2] Active trade? ──► YES: track PnL, check Take Profit / Stop Loss
    │                         NO:  ──► [3]
    │
    └─ [3] Entry evaluation
               ├─ KNN win-rate lookup  ──► MemoryBank (HNSW, <1ms)
               └─ ONNX inference       ──► OnnxBrain (RandomForest, <1ms)
                       └─ direction ∈ {long, short, flat}
                               └─ open position if not flat

Risk Rules (Paper Trading)

Rule	Threshold
Take Profit	+$20 unrealized PnL
Stop Loss	-$10 unrealized PnL
Position size	1.0 BTC fixed
Starting capital	$100,000 USD

---

3. experiencing/ — Slow Path Episode Archiver (Python)

The Episode Archiver (episode_store.py) consumes closed trade episodes from NATS JetStream and persists them to Postgres with a 128-dimensional pgvector context embedding.

This store is the long-term memory of the system — future episodes will be retrieved from it via the MemoryBank's HNSW index for KNN win-rate estimation.

---

4. learning/ — Model Training & Export (Python)

Script	Purpose
export_fast_brain.py	Exports the 116-byte regime classifier binary (regime_fast_v1.bin)
export_onnx_brain.py	Trains a Random Forest and exports it as decision_brain.onnx
nats_consumer.py	Debug utility: prints a one-liner summary every 100 observation events

---

5. infra/ — Infrastructure

Resource	Description
docker-compose.yml	NATS 2.10 (JetStream), Postgres 16 + pgvector, TimescaleDB
postgres/init.sql	Database schema: episodes, chain_heads, signal_scores, model_metadata, sandbox schema
models/	Model binaries: regime_fast_v1.bin, onnx/decision_brain.onnx

---

Data Flow

Observation Event (NATS: observations.microstructure.BTC-USD)

Every order-book depth update emits one ObservationEvent JSON message:

{
  "id": "uuid-v4",
  "timestamp_ns": 1710000000000000000,
  "asset_id": "BTC-USD",
  "signal_class": "microstructure",
  "regime_hint": "unknown",
  "confidence": 0.5,
  "signal_vector": [0.45, 0.72, 0.0, 0.31, 0.0, ...],
  "payload": { "best_bid": 68000.0, "best_ask": 68001.0 },
  "system_health": { "latency_p99_ms": 1.0, "slippage_bps": 0.0, "api_status": "ok", "queue_consumer_lag": 0 },
  "feature_version": "v1.0",
  "prev_hash": [0, 0, ...],
  "commitment_hash": [34, 17, ...]
}

Active signal_vector indices:

Index	Signal	Description
0	spread	Absolute bid-ask spread in USD
1	imbalance	Top-5 level volume imbalance [0, 1]
2	spoofing	Volume cancelled at top-of-book this tick
3	VPIN	Rolling Volume-Synchronized PIN estimate [0, 1]

Closed Episode (NATS: episodes.closed.BTC-USD)

When a trade is closed, a ClosedEpisode is published and consumed by episode_store.py for Postgres archival.

---

Signal Glossary

Signal	Range	High value means
Spread	0 → ∞ (USD)	Low liquidity, wider market
Imbalance	0 → 1	Near 1.0: buy pressure; near 0.0: sell pressure
Spoofing Score	0 → ∞	Large orders placed and rapidly cancelled near top-of-book
VPIN	0 → 1	High informed order flow; elevated adverse selection risk
KNN Win-rate	0 → 1	Historical win fraction in similar market contexts

---

Database Schema

Managed by infra/postgres/init.sql, initialized automatically by Docker on first start.

Key Tables

Table	Purpose
episodes	One row per closed trade. Stores entry/exit prices, PnL, MAE, regime, and the 128-dim pgvector context embedding.
chain_heads	Records periodic snapshots of the SHA-256 hash chain head for auditability.
signal_scores	Rolling accuracy metrics per signal class × feature × regime (signal marketplace).
model_metadata	Tracks model lifecycle: SHADOW → LIVE → RETIRED with accuracy and drawdown stats.
sandbox.episodes	Physically isolated copy of episodes for black-swan stress testing (all rows enforced synthetic = TRUE).

ANN Index

The context_vector column uses an IVFFlat (Inverted File + Flat) index from pgvector for approximate nearest-neighbour retrieval on the 128-dimensional embeddings:

CREATE INDEX ON episodes
    USING ivfflat (context_vector vector_l2_ops)
    WITH (lists = 100)
    WHERE synthetic = FALSE;

---

Prerequisites

System

• OS: Ubuntu 20.04+ (or WSL2 on Windows)
• Rust: 1.75+ (rustup toolchain install stable)
• Python: 3.10+
• Docker & Docker Compose: for NATS and Postgres

Python dependencies

pip install asyncpg nats-py numpy scikit-learn skl2onnx

ONNX Runtime

The ort crate downloads ONNX Runtime automatically via Cargo. No manual installation is needed.

---

Infrastructure Setup

Start NATS (with JetStream) and Postgres (with pgvector) using Docker Compose:

cd infra/
docker compose up -d

Verify services are healthy:

docker compose ps

Service	Port	Purpose
nats	4222 (client), 8222 (monitoring)	Message bus with JetStream persistence
postgres	5435	Episode storage with pgvector ANN
timescaledb	5433	Time-series signal storage (for future use)

---

Model Generation

Both model files must exist before the fast-path engine can start.

1. Regime Classifier (116-byte binary)

cd learning/
python export_fast_brain.py
# → infra/models/regime_fast_v1.bin

2. ONNX Decision Model

cd learning/
python export_onnx_brain.py
# → infra/models/onnx/decision_brain.onnx

---

Running the System

Start all processes in separate terminals in this order:

Terminal 1 — Infrastructure

cd infra/
docker compose up

Terminal 2 — Witness Node (market data ingestion)

cd /path/to/triad/
cargo run -p monitoring

Terminal 3 — Fast-Path Execution Engine

cargo run -p fast-path

Terminal 4 — Slow-Path Episode Archiver

python experiencing/episode_store.py

Terminal 5 — (Optional) Observation Stream Inspector

python learning/nats_consumer.py

---

Project Structure

triad/
├── Cargo.toml                    # Workspace manifest (fast-path + monitoring crates)
├── Cargo.lock
│
├── fast-path/                    # Rust: Real-time execution engine
│   ├── Cargo.toml
│   └── src/
│       ├── main.rs               # Entry point + main event loop
│       ├── classifier.rs         # Softmax regime classifier
│       ├── memory.rs             # HNSW episodic memory bank
│       ├── model_loader.rs       # Hot-reloadable binary model loader
│       └── onnx_engine.rs        # ONNX Runtime wrapper
│
├── monitoring/                   # Rust: Witness Node (data ingestion)
│   ├── Cargo.toml
│   └── src/
│       ├── main.rs               # WebSocket loop + NATS publisher
│       ├── orderbook.rs          # LocalOrderBook + VpinTracker
│       ├── events.rs             # Shared event types + SHA-256 hash chain
│       └── lib.rs                # Re-exports for fast-path dependency
│
├── experiencing/                 # Python: Slow-path data pipeline
│   └── episode_store.py          # NATS JetStream → Postgres archiver
│
├── learning/                     # Python: Model training & debugging
│   ├── export_fast_brain.py      # Export 116-byte regime classifier binary
│   ├── export_onnx_brain.py      # Train + export ONNX decision model
│   └── nats_consumer.py          # Debug: inspect live observation stream
│
├── infra/                        # Infrastructure configuration
│   ├── docker-compose.yml        # NATS + Postgres + TimescaleDB
│   ├── models/
│   │   ├── regime_fast_v1.bin    # Regime classifier weights (116 bytes)
│   │   └── onnx/
│   │       └── decision_brain.onnx
│   └── postgres/
│       └── init.sql              # DB schema (auto-applied on first docker compose up)
│
└── shared/                       # (Reserved) Future shared Rust crate

---

Configuration Reference

All configuration is currently via hardcoded constants or environment assumptions. Key values:

Setting	Location	Default
NATS URL	monitoring/src/main.rs, fast-path/src/main.rs	nats://localhost:4222
Postgres DSN	fast-path/src/memory.rs, experiencing/episode_store.py	postgres://triad:triad@localhost:5435/triad
Starting capital	fast-path/src/main.rs	$100,000 USD
Take-profit threshold	fast-path/src/main.rs	$20.00
Stop-loss threshold	fast-path/src/main.rs	-$10.00
Model hot-reload interval	fast-path/src/main.rs	5 seconds
MemoryBank refresh interval	fast-path/src/memory.rs	60 seconds
KNN neighbours	fast-path/src/main.rs	k = 5
KNN distance gate	fast-path/src/memory.rs	avg_dist > 2.0 → reject
VPIN bucket size	monitoring/src/main.rs	1.0 BTC
VPIN window	monitoring/src/main.rs	50 buckets
Confidence threshold	infra/models/regime_fast_v1.bin	0.90
Signal vector dimensions	monitoring/src/events.rs	32 (4 active)

---

Testing

Rust Unit Tests

# Run all unit tests across the workspace
cargo test

# Run tests for a specific crate
cargo test -p fast-path
cargo test -p monitoring

Covered by tests:

• model_loader: valid blob parsing, size/magic rejection, mtime-based reload logic
• classifier: correct softmax output, Unknown threshold gating, insufficient-features guard

Integration Testing (Manual)

1. Start infrastructure and all processes.
2. Watch fast-path logs for TRADE OPENED and TRADE CLOSED events.
3. Verify episode_store.py logs Archived Episode ... for each closed trade.
4. Connect to Postgres and inspect:

   SELECT id, direction, outcome_pnl, max_adverse_excursion FROM episodes ORDER BY created_at DESC LIMIT 10;

---

Roadmap

• Live regime labeling — slow-path updates regime_authoritative in Postgres after each episode
• Proper ML training pipeline — replace synthetic weights with Postgres-trained LogisticRegression
• Signal marketplace — populate signal_scores table with rolling per-feature accuracy
• Model promotion gates — automate SHADOW → LIVE based on model_metadata thresholds
• TimescaleDB integration — write raw signal time-series for backtesting
• Blockchain anchoring — periodically commit chain_heads.head_hash to a public chain
• Multi-asset support — parameterize asset symbol and add per-asset NATS subjects
• Configuration file — move hardcoded thresholds to a config.toml
• Live exchange integration — replace PaperPortfolio with a real exchange OMS adapter

---

License

This project is for research and educational purposes. All trading decisions are simulated (paper trading only). Do not use in production with real funds without appropriate risk controls.