From 44fdd27a68e46772a53a5b302fe0cc11487fab3a Mon Sep 17 00:00:00 2001 From: Chizy Date: Wed, 19 Aug 2026 23:14:49 -0400 Subject: [PATCH 1/2] docs: add contributing guide, code of conduct, and community health files --- .github/CODEOWNERS | 3 + .github/SUPPORT.md | 26 ++ CODE_OF_CONDUCT.md | 83 +++++ CONTRIBUTING.md | 162 ++++++++++ README.md | 638 ++++++--------------------------------- docs/JSON_DATA_EXPORT.md | 61 ++++ docs/SCRIPTS.md | 98 ++++++ 7 files changed, 524 insertions(+), 547 deletions(-) create mode 100644 .github/CODEOWNERS create mode 100644 .github/SUPPORT.md create mode 100644 CODE_OF_CONDUCT.md create mode 100644 CONTRIBUTING.md create mode 100644 docs/JSON_DATA_EXPORT.md create mode 100644 docs/SCRIPTS.md diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..8628b39 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,3 @@ +# Default owner for everything in the repo. +# Later matches take precedence over earlier ones. +* @chizy7 diff --git a/.github/SUPPORT.md b/.github/SUPPORT.md new file mode 100644 index 0000000..caf2523 --- /dev/null +++ b/.github/SUPPORT.md @@ -0,0 +1,26 @@ +# Getting Help with PinnacleMM + +Thanks for using PinnacleMM! Here's how to get help: + +## Documentation + +Start with the docs — most questions are covered there: + +- [Getting Started Guide](../docs/user_guide/getting_started.md) +- [System Architecture](../docs/architecture/system_overview.md) +- [API Reference](../docs/api/reference.md) +- [Full documentation index](../README.md#documentation) + +## Questions and Bug Reports + +- **Bugs**: open an issue using the [bug report template](https://github.com/chizy7/PinnacleMM/issues/new?template=bug_report.md) +- **Feature ideas**: use the [feature request template](https://github.com/chizy7/PinnacleMM/issues/new?template=feature_request.md) +- **General questions**: open an issue, or email [chizy@chizyhub.com](mailto:chizy@chizyhub.com) + +## Security Issues + +**Do not open a public issue.** Please follow the [security policy](SECURITY.md) to report vulnerabilities privately. + +## Contributing + +Want to fix it yourself? See the [Contributing Guide](../CONTRIBUTING.md). diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..429e82f --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,83 @@ +# Contributor Covenant 3.0 Code of Conduct + +## Our Pledge + +We pledge to make our community welcoming, safe, and equitable for all. + +We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant. + +## Encouraged Behaviors + +While acknowledging differences in social norms, we all strive to meet our community's expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language. + +With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including: + +1. Respecting the **purpose of our community**, our activities, and our ways of gathering. +2. Engaging **kindly and honestly** with others. +3. Respecting **different viewpoints** and experiences. +4. **Taking responsibility** for our actions and contributions. +5. Gracefully giving and accepting **constructive feedback**. +6. Committing to **repairing harm** when it occurs. +7. Behaving in other ways that promote and sustain the **well-being of our community**. + +## Restricted Behaviors + +We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct. + +1. **Harassment.** Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop. +2. **Character attacks.** Making insulting, demeaning, or pejorative comments directed at a community member or group of people. +3. **Stereotyping or discrimination.** Characterizing anyone's personality or behavior on the basis of immutable identities or traits. +4. **Sexualization.** Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community. +5. **Violating confidentiality**. Sharing or acting on someone's personal or private information without their permission. +6. **Endangerment.** Causing, encouraging, or threatening violence or other harm toward any person or group. +7. Behaving in other ways that **threaten the well-being** of our community. + +### Other Restrictions + +1. **Misleading identity.** Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions. +2. **Failing to credit sources.** Not properly crediting the sources of content you contribute. +3. **Promotional materials**. Sharing marketing or other commercial content in a way that is outside the norms of the community. +4. **Irresponsible communication.** Failing to responsibly present content which includes, links or describes any other restricted behaviors. + +## Reporting an Issue + +Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm. + +When an incident does occur, it is important to report it promptly. To report a possible violation, email the project maintainer at **[chizy@chizyhub.com](mailto:chizy@chizyhub.com)**. All reports will be handled with discretion and confidentiality. + +Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution. + +## Addressing and Repairing Harm + +If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped. + +1. Warning + 1. Event: A violation involving a single incident or series of incidents. + 2. Consequence: A private, written warning from the Community Moderators. + 3. Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations. +2. Temporarily Limited Activities + 1. Event: A repeated incidence of a violation that previously resulted in a warning, or the first incidence of a more serious violation. + 2. Consequence: A private, written warning with a time-limited cooldown period designed to underscore the seriousness of the situation and give the community members involved time to process the incident. The cooldown period may be limited to particular communication channels or interactions with particular community members. + 3. Repair: Examples of repair may include making an apology, using the cooldown period to reflect on actions and impact, and being thoughtful about re-entering community spaces after the period is over. +3. Temporary Suspension + 1. Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation. + 2. Consequence: A private written warning with conditions for return from suspension. In general, temporary suspensions give the person being suspended time to reflect upon their behavior and possible corrective actions. + 3. Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted. +4. Permanent Ban + 1. Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member. + 2. Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior. + 3. Repair: There is no possible repair in cases of this severity. + +This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. + +## Attribution + +This Code of Conduct is adapted from the Contributor Covenant, version 3.0, permanently available at [https://www.contributor-covenant.org/version/3/0/](https://www.contributor-covenant.org/version/3/0/). + +Contributor Covenant is stewarded by the Organization for Ethical Source and licensed under CC BY-SA 4.0. To view a copy of this license, visit [https://creativecommons.org/licenses/by-sa/4.0/](https://creativecommons.org/licenses/by-sa/4.0/) + +For answers to common questions about Contributor Covenant, see the FAQ at [https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq). Translations are provided at [https://www.contributor-covenant.org/translations](https://www.contributor-covenant.org/translations). Additional enforcement and community guideline resources can be found at [https://www.contributor-covenant.org/resources](https://www.contributor-covenant.org/resources). The enforcement ladder was inspired by the work of [Mozilla's code of conduct team](https://github.com/mozilla/inclusion). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..aac1c08 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,162 @@ +# Contributing to PinnacleMM + +Thank you for your interest in contributing to PinnacleMM! This document explains how to set up your development environment, the conventions the project enforces in CI, and how to get your changes merged. + +By participating in this project, you agree to abide by our [Code of Conduct](CODE_OF_CONDUCT.md). + +## Ways to Contribute + +- **Report bugs** using the [bug report template](https://github.com/chizy7/PinnacleMM/issues/new?template=bug_report.md) +- **Request features** using the [feature request template](https://github.com/chizy7/PinnacleMM/issues/new?template=feature_request.md) +- **Report security vulnerabilities** privately — see [SECURITY.md](.github/SECURITY.md). **Never open a public issue for a security vulnerability.** +- **Improve documentation** — guides live in [`docs/`](docs/) +- **Submit code** — bug fixes, performance improvements, new strategies, exchange connectors, and tests are all welcome. Check the [Roadmap](docs/ROADMAP.md) for planned work and good areas to help. + +For large changes, please open an issue first to discuss the approach before investing significant time. + +## Development Setup + +### Prerequisites + +- C++20 compatible compiler (GCC 10+, Clang 10+, or MSVC 2019+) +- CMake 3.14+ +- Boost libraries 1.72+ +- OpenSSL +- nlohmann_json +- Google Test and Google Benchmark (for tests and benchmarks) +- Python 3 with `pre-commit` (for the git hooks — see below) + +spdlog and fmt are auto-downloaded at pinned versions if not installed locally. + +On macOS: + +```bash +brew install cmake boost openssl nlohmann-json googletest google-benchmark pre-commit +``` + +On Ubuntu/Debian: + +```bash +sudo apt-get install -y cmake build-essential libboost-all-dev libssl-dev \ + nlohmann-json3-dev libgtest-dev libbenchmark-dev +pip install pre-commit +``` + +### Build and Test + +```bash +# Fork the repo on GitHub, then: +git clone https://github.com//PinnacleMM.git +cd PinnacleMM + +# Build (auto-checks dependencies) +scripts/run-native.sh build + +# Run the full test suite +scripts/run-native.sh test + +# Run performance benchmarks +scripts/run-native.sh benchmark +``` + +Or manually: + +```bash +mkdir build && cd build +cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON -DBUILD_BENCHMARKS=ON .. +make -j$(nproc || sysctl -n hw.ncpu) +``` + +For memory-safety validation during development, build with sanitizers: + +```bash +cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_SANITIZERS=ON .. && make -j8 +``` + +Note that CI builds with AddressSanitizer enabled, so memory errors will fail your PR even if they don't crash locally. + +## Pre-commit Hooks (Required) + +The project uses [pre-commit](https://pre-commit.com/) to enforce formatting and linting. Install the hooks once after cloning: + +```bash +pre-commit install +``` + +After this, every `git commit` automatically runs: + +- **clang-format** (v17, using the repo's `.clang-format`) on C/C++ files +- **cmake-format** on CMake files +- **hadolint** on Dockerfiles +- **shellcheck** on shell scripts +- General checks: trailing whitespace, end-of-file newlines, YAML validity, merge-conflict markers, large files, line endings + +If a hook fails, the commit is aborted. Formatting hooks fix the files in place — just `git add` the fixed files and commit again. To run all hooks against the whole repo manually: + +```bash +pre-commit run --all-files +``` + +CI runs the same hooks on every PR (`.github/workflows/pre-commit.yml`), so skipping local installation just means you find out later. + +## Coding Guidelines + +- **C++20**, matching the style of the surrounding code +- Formatting is defined by `.clang-format` — don't hand-format; let the hook do it +- This is an ultra-low latency system: on hot paths, avoid allocations, locks, and exceptions; prefer lock-free structures and `std::atomic` consistent with the existing core engine +- Add unit tests (Google Test) for new functionality; add benchmarks (Google Benchmark) for performance-sensitive changes +- Update the relevant docs in `docs/` when behavior or interfaces change + +## Commit Messages and PR Titles + +PR titles **must** follow [Conventional Commits](https://www.conventionalcommits.org/) — this is enforced by CI (`pr-title-lint.yml`) and PRs with non-conforming titles cannot merge: + +``` +(): +``` + +- Allowed types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, `revert` +- The subject must start with a **lowercase** letter +- The full header must be at most 100 characters + +Examples: + +``` +feat(strategy): add inventory-skew quoting to basic market maker +fix(exchange): handle coinbase websocket reconnect race +perf(core): reduce order book update latency with arena allocator +``` + +Using the same convention for individual commit messages is encouraged (PRs are typically squash-merged using the PR title). + +## Pull Request Process + +1. Create a topic branch from `main` in your fork +2. Make your changes, with tests +3. Run locally before pushing: + - `pre-commit run --all-files` + - `scripts/run-native.sh test` +4. Open a PR against `main` and fill out the PR template completely +5. Make sure CI passes. On every PR, CI runs: + - **Build and Test** on Ubuntu and macOS, Debug and Release, with AddressSanitizer + - **Performance benchmarks** (Release builds) + - **Pre-commit hooks** (formatting/linting) + - **PR title lint** (conventional commit format) + - **Docker build test** +6. Address review feedback; a maintainer will merge once approved + +## Performance Expectations + +PinnacleMM targets microsecond-level latency. If your change touches the core engine, order book, or execution paths: + +- Run the relevant benchmarks before and after (`scripts/run-native.sh benchmark`) +- Include the numbers in your PR description +- Regressions on hot paths need a strong justification + +## Questions? + +- Check the [documentation](docs/) and [Getting Started Guide](docs/user_guide/getting_started.md) first +- Open a [discussion or issue](https://github.com/chizy7/PinnacleMM/issues) +- Contact the maintainer: [chizy@chizyhub.com](mailto:chizy@chizyhub.com) + +Thank you for contributing! diff --git a/README.md b/README.md index c1314cb..344d528 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ - PinnacleMM Logo + PinnacleMM Logo

Ultra-Low Latency Market Making System

@@ -30,35 +30,18 @@ PinnacleMM is a high-performance, production-grade market making system designed ## Key Features -- **Ultra-Low Latency Core**: Optimized C++ engine with lock-free data structures -- **Nanosecond Precision**: High-resolution timing for accurate execution -- **Crash Recovery**: Memory-mapped persistence system for reliable operation -- **Dynamic Market Making**: Adaptive bid-ask spread based on market conditions -- **Position Management**: Intelligent inventory management with customizable risk parameters -- **Exchange Simulation**: Realistic market simulation for strategy development and testing -- **Live Exchange Connectivity**: Real-time connection to Coinbase Pro WebSocket feeds -- **FIX Protocol Support**: Professional-grade FIX connectivity for institutional exchanges -- **Advanced Order Routing**: Smart order routing with 4 algorithms (BEST_PRICE, TWAP, VWAP, MARKET_IMPACT) -- **Multi-Venue Execution**: Intelligent order distribution across multiple exchanges -- **ML-Enhanced Trading**: Neural network-based spread optimization with sub-microsecond predictions -- **Market Regime Detection**: Real-time detection of 8 market regimes using Hidden Markov Models -- **Predictive Analytics**: Market impact prediction and reinforcement learning parameter adaptation +- **Ultra-Low Latency Core**: C++20 engine with lock-free data structures and nanosecond-precision timing +- **Dynamic Market Making**: Adaptive bid-ask spreads with intelligent inventory and position management +- **Live Exchange Connectivity**: Coinbase Pro WebSocket feeds and institutional FIX protocol support (Interactive Brokers, Coinbase, Kraken, Binance) +- **Smart Order Routing**: Multi-venue execution with BEST_PRICE, TWAP, VWAP, and MARKET_IMPACT algorithms +- **ML-Enhanced Trading**: Neural network spread optimization, Hidden Markov Model regime detection, market impact prediction, and reinforcement learning parameter adaptation +- **Risk Management**: Lock-free pre-trade checks, position/exposure limits, circuit breaker, real-time VaR, and a 16-type alerting system +- **Multi-Instrument Trading**: Simultaneous multi-symbol trading with cross-exchange arbitrage detection and cross-market correlation analysis - **Advanced Backtesting**: Historical data replay with Monte Carlo analysis and A/B testing -- **Real-Time Visualization**: Professional web dashboard with live performance monitoring (access at `visualization/static/index.html` when running with `--enable-visualization`) -- **Structured Data Export**: JSON Lines (JSONL) logging for market data, strategy metrics, and trading events with `--json-log` flag -- **Risk Management**: Pre-trade risk checks (~750ns), position/exposure limits, drawdown tracking, daily loss limits, and auto-hedging -- **Circuit Breaker**: Automatic market halt on rapid price moves, spread widening, volume spikes, latency degradation, or crisis regime detection -- **Real-Time VaR**: Value at Risk using historical, parametric, and Monte Carlo (10K simulations) methods with lock-free double-buffered reads -- **Alerting System**: 16 alert types with throttling, severity levels, and WebSocket delivery to the dashboard -- **Disaster Recovery**: Atomic risk state persistence, position reconciliation, and labeled backup management -- **Kubernetes Deployment**: Production-ready StatefulSet with health probes, PVC, network policies, and pod disruption budget -- **Enterprise Security**: AES-256-CBC encryption with unique salts, 100,000 PBKDF2 iterations, secure password input, comprehensive input validation, audit logging, rate limiting, and certificate pinning -- **Multi-Instrument Trading**: Simultaneous trading across multiple symbols with `InstrumentManager` orchestration -- **Cross-Exchange Arbitrage**: Venue price discrepancy detection with fee-adjusted opportunity scanning and dry-run execution -- **Cross-Market Correlation**: Pearson/rolling correlation, lead-lag analysis, Engle-Granger cointegration, and signal-based spread adjustment -- **Per-Symbol Risk Tracking**: Atomic per-symbol position, PnL, and volume tracking with configurable per-symbol limits -- **Dynamic Resource Allocation**: CPU core distribution and thread pinning for multi-instrument deployments -- **Comprehensive Testing**: Extensive test suite ensuring reliability and performance +- **Observability**: Real-time web dashboard (`--enable-visualization`) and structured JSONL data export (`--json-log`) +- **Crash Recovery**: Memory-mapped persistence with disaster recovery and backup tooling +- **Enterprise Security**: AES-256 encrypted credentials, input validation, audit logging, rate limiting, and certificate pinning +- **Production Deployment**: Docker images and a production-ready Kubernetes StatefulSet ## System Architecture @@ -79,588 +62,149 @@ Read more about the [system architecture](docs/architecture/system_overview.md). - C++20 compatible compiler (GCC 10+, Clang 10+, or MSVC 2019+) - CMake 3.14+ - Boost libraries 1.72+ -- spdlog library (for structured logging; auto-downloaded at v1.15.3 if not installed) -- fmt library (for formatting, spdlog dependency; auto-downloaded at 11.2.0 if not installed) -- OpenSSL library (for secure credential handling) -- nlohmann_json library (for configuration handling) +- OpenSSL (secure credential handling) +- nlohmann_json (configuration handling) +- spdlog and fmt (auto-downloaded at pinned versions if not installed) -### Quick Start with Scripts +### Quick Start -PinnacleMM includes convenient bash scripts for easy execution: - -#### **Native Execution** (Recommended for Development) ```bash # Clone and setup git clone https://github.com/chizy7/PinnacleMM.git cd PinnacleMM -# One-command setup and run -scripts/run-native.sh # Simulation mode (auto-builds if needed) -scripts/run-native.sh -m live -v # Live trading with verbose logs -scripts/run-native.sh --enable-ml # ML-enhanced simulation mode -scripts/run-native.sh --enable-visualization # With real-time dashboard -scripts/run-native.sh --setup-credentials # Configure API keys -``` +# Native execution (recommended for development) — auto-builds if needed +scripts/run-native.sh # Simulation mode +scripts/run-native.sh test # Run test suite +scripts/run-native.sh benchmark # Run performance benchmarks -#### **Docker Execution** (Recommended for Production) -```bash -# Build and run in one command -scripts/run-docker.sh # Simulation mode -scripts/run-docker.sh -m live -v # Live trading mode -scripts/run-docker.sh build # Build Docker image -scripts/run-docker.sh logs # View container logs +# Docker execution (recommended for production) +scripts/run-docker.sh # Simulation mode ``` -### Manual Building from Source +Pre-built Docker images are available at `ghcr.io/chizy7/pinnaclemm`. See the [Scripts Reference](docs/SCRIPTS.md) for all script commands, options, and container usage. -```bash -# Clone the repository -git clone https://github.com/chizy7/PinnacleMM.git -cd PinnacleMM +### Manual Build -# Build with native script (recommended) -scripts/run-native.sh build - -# Or build manually +```bash mkdir build && cd build -cmake .. -make -j$(sysctl -n hw.ncpu) # macOS -# make -j$(nproc) # Linux +cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON -DBUILD_BENCHMARKS=ON .. +make -j$(nproc || sysctl -n hw.ncpu) ``` -### Script Features Comparison +## Running PinnacleMM -| Feature | Native Script (`scripts/run-native.sh`) | Docker Script (`scripts/run-docker.sh`) | -|---------|-----------------------------------|-----------------------------------| -| **Simulation Mode** | Perfect | Perfect | -| **Live Trading** | Real WebSocket data | WebSocket config issue | -| **Auto-Build** | Builds if needed | Auto Docker build | -| **Test Runner** | `scripts/run-native.sh test` | Not included | -| **Benchmarks** | `scripts/run-native.sh benchmark` | Not included | -| **Credential Setup** | Interactive setup | Volume mounting | -| **Dependency Check** | cmake, make, g++ | Docker only | -| **Best For** | Development & Live Trading | Production & Simulation | +### Simulation Mode -### Running PinnacleMM - -#### Simulation Mode ```bash -# Using scripts (recommended) -scripts/run-native.sh # Native execution -scripts/run-docker.sh # Docker execution - -# Manual execution cd build && ./pinnaclemm --mode simulation --symbol BTC-USD -# ML-enhanced simulation with visualization -cd build && ./pinnaclemm --mode simulation --enable-ml --enable-visualization - -# Custom ports for visualization (useful for running multiple instances) -cd build && ./pinnaclemm --mode simulation --enable-ml --enable-visualization --viz-ws-port 8089 --viz-api-port 8090 - -# Debug mode with enhanced WebSocket logging -cd build && SPDLOG_LEVEL=debug ./pinnaclemm --mode simulation --enable-ml --enable-visualization --verbose - -# Enable JSON data export (logs market data and strategy metrics to JSONL file) -cd build && ./pinnaclemm --mode simulation --enable-ml --json-log --json-log-file simulation_data.jsonl - -# Combined: ML + visualization + JSON logging -cd build && ./pinnaclemm --mode simulation --enable-ml --enable-visualization --json-log --json-log-file sim_ml_data.jsonl - -# Multi-instrument simulation -cd build && ./pinnaclemm --mode simulation --symbols BTC-USD,ETH-USD - -# Arbitrage detection (dry-run) -cd build && ./pinnaclemm --mode simulation --symbol BTC-USD --enable-arbitrage --arb-dry-run - -# The visualization dashboard will be available at: -# - WebSocket: ws://localhost:8080 (or custom port with --viz-ws-port) -# - REST API: http://localhost:8081 (or custom port with --viz-api-port) -# - Dashboard: Open visualization/static/index.html in your browser -# - Test Dashboard: Open build/test_dashboard.html for WebSocket connection testing -``` - -#### Live Exchange Mode -```bash -# Setup credentials first -scripts/run-native.sh --setup-credentials - -# Live trading with scripts -scripts/run-native.sh -m live -v # Native (recommended for live) -scripts/run-docker.sh -m live -v # Docker - -# Manual execution -cd build && ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --verbose - -# Live trading with ML and visualization (custom ports) -cd build && ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --enable-ml --enable-visualization --viz-ws-port 8085 --viz-api-port 8086 --verbose - -# Debug mode for live trading (with enhanced logging) -cd build && SPDLOG_LEVEL=debug ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --enable-ml --enable-visualization --viz-ws-port 8085 --viz-api-port 8086 --verbose - -# Live trading with JSON data export (captures real market data) -cd build && ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --enable-ml --json-log --json-log-file live_btc_data.jsonl - -# Full-featured live trading: ML + visualization + JSON logging -cd build && ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --enable-ml --enable-visualization --viz-ws-port 8085 --viz-api-port 8086 --json-log --json-log-file live_trading_session.jsonl --verbose - -# Access the live dashboard: -# - Main Dashboard: visualization/static/live_dashboard.html -# - Test Dashboard: build/test_dashboard.html (for connection testing) -# - WebSocket: ws://localhost:8085 (or your custom port) -# - REST API: http://localhost:8086 (or your custom port) +# With ML strategy and real-time dashboard +./pinnaclemm --mode simulation --enable-ml --enable-visualization ``` -When running in live mode, you'll be prompted for your master password to decrypt API credentials. +### Live Trading Mode -#### Backtest Mode -```bash -# Run backtest with synthetic data (no API keys needed, auto-generates data if no CSV found) -cd build && ./pinnaclemm --mode backtest --symbol BTC-USD - -# Backtest with ML-enhanced strategy -cd build && ./pinnaclemm --mode backtest --symbol BTC-USD --enable-ml - -# Custom parameters -cd build && ./pinnaclemm --mode backtest --symbol BTC-USD \ - --initial-balance 50000 --trading-fee 0.002 --slippage-bps 5.0 - -# Custom output directory -cd build && ./pinnaclemm --mode backtest --symbol BTC-USD --backtest-output my_results - -# Using your own historical data: place CSV at /data/.csv -# e.g. backtest_results/data/BTC-USD.csv with format: -# timestamp,symbol,price,bid,ask,volume -# 1640995200000000000,BTC-USD,47892.50,47890.00,47895.00,1250.75 -``` - -Backtest mode runs the strategy against historical (or synthetic) data, prints a detailed performance report (Sharpe ratio, drawdown, win rate, etc.), saves JSON results to the output directory, and exits cleanly. - -## JSON Data Export - -PinnacleMM provides comprehensive structured data export capabilities through JSON Lines (JSONL) format logging. This feature enables detailed analysis, backtesting, monitoring, and debugging of trading strategies and market data. - -### Features - -- **Market Data Logging**: Real-time price, volume, bid/ask data with timestamps -- **Strategy Metrics**: Position, P&L, quote counts, and performance statistics -- **Order Book Updates**: Complete order book state with bid and ask arrays -- **Connection Events**: WebSocket connections, disconnections, and errors -- **Trading Events**: Order placements, fills, cancellations, and status updates -- **Thread-Safe**: Concurrent logging without performance impact -- **JSONL Format**: One JSON object per line for easy parsing and streaming - -### Usage - -Enable JSON logging with the `--json-log` flag and optionally specify a custom file path: - -```bash -# Basic JSON logging (default file: pinnaclemm_data.jsonl) -./pinnaclemm --mode simulation --symbol BTC-USD --json-log - -# Custom file path -./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --json-log --json-log-file my_trading_data.jsonl - -# Combined with other features -./pinnaclemm --mode live --enable-ml --enable-visualization --json-log --json-log-file full_session.jsonl -``` - -### Sample Output - -```json -{"format":"jsonl","timestamp":"2025-09-29T16:27:06.770Z","type":"session_start","version":"1.0.0"} -{"metrics":{"ask_price":67252.80,"bid_price":67248.20,"market_price":67250.45,"pnl":0.0,"position":0.0,"quote_updates":1,"strategy_name":"BasicMarketMaker","volume":1234.56},"strategy_name":"BasicMarketMaker","symbol":"BTC-USD","timestamp":"2025-09-29T16:27:12.011Z","type":"strategy_metrics"} -{"ask_price":67253.00,"bid_price":67249.50,"event_timestamp":1695736032610,"is_buy":true,"price":67250.75,"symbol":"BTC-USD","timestamp":"2025-09-29T16:27:12.610Z","type":"market_update","volume":0.5} -``` - -### Data Types - -- **`session_start`**: Session initialization marker with format and version -- **`strategy_metrics`**: Trading strategy performance and position data -- **`market_update`**: Real-time market data from exchange feeds -- **`order_book_update`**: Complete order book snapshots with bid/ask arrays -- **`trading_event`**: Order lifecycle events and trading actions -- **`connection_event`**: Exchange connectivity status and errors - -### File Management - -JSON log files are created in the current working directory by default. For production use, consider: - -```bash -# Save to logs directory (create and add to .gitignore) -mkdir -p logs -./pinnaclemm --json-log --json-log-file logs/trading_$(date +%Y%m%d_%H%M%S).jsonl - -# Save to data directory (existing, likely gitignored) -./pinnaclemm --json-log --json-log-file data/market_data_$(date +%Y%m%d).jsonl - -# Save outside project directory -./pinnaclemm --json-log --json-log-file ~/trading_logs/pinnaclemm_session.jsonl -``` - -## API Credential Management - -PinnacleMM securely stores and manages exchange API credentials: - -- **Encryption**: AES-256-CBC encryption with PBKDF2 key derivation -- **Master Password**: Single password to unlock all exchange credentials -- **Secure Storage**: Credentials encrypted in `config/secure_config.json` -- **Interactive Setup**: User-friendly credential configuration interface - -### Setting Up API Credentials - -1. **Run credential setup**: ```bash +# One-time credential setup (AES-256 encrypted, master password protected) scripts/run-native.sh --setup-credentials -# or manually: -./pinnaclemm --setup-credentials -``` - -2. **Enter master password** (secure input with hidden characters - this encrypts all API keys with AES-256-CBC + unique salt + 100,000 PBKDF2 iterations) - -3. **Configure exchange credentials**: - - **Coinbase Pro**: API Key + API Secret + Passphrase - - **Other exchanges**: API Key + API Secret (+ optional passphrase) - - All inputs are validated and sanitized before encryption -4. **Verify setup**: -```bash -scripts/run-native.sh -m live -v -# or manually: +# Start live trading ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --verbose ``` -### Supported Exchanges - -#### WebSocket Connectivity -- **Coinbase Pro**: Live market data via WebSocket - -#### FIX Protocol Connectivity -- **Interactive Brokers**: FIX 4.2 support (requires IB FIX API agreement) -- **Coinbase Pro**: FIX 4.4 institutional connectivity (framework ready) -- **Kraken**: FIX 4.4 institutional connectivity (framework ready) -- **Binance**: FIX 4.4 institutional connectivity (framework ready) - -#### In Development -- **Kraken, Gemini, Binance, Bitstamp**: WebSocket connectors in development - -For more detailed instructions, see the [Getting Started Guide](docs/user_guide/getting_started.md). - -## Script Documentation - -### Native Script (`scripts/run-native.sh`) - -**Available Commands:** -```bash -# Execution modes -scripts/run-native.sh # Simulation mode (default) -scripts/run-native.sh -m live -v # Live mode with verbose logging -scripts/run-native.sh -s ETH-USD # Custom trading symbol -scripts/run-native.sh -e coinbase # Specify exchange - -# Build commands -scripts/run-native.sh build # Build project -scripts/run-native.sh clean # Clean build directory -scripts/run-native.sh rebuild # Clean and rebuild - -# Testing and benchmarks -scripts/run-native.sh test # Run all tests -scripts/run-native.sh benchmark # Run performance benchmarks - -# Setup -scripts/run-native.sh --setup-credentials # Configure API credentials (secure input) -scripts/run-native.sh --help # Show help - -# Cleanup -scripts/cleanup.sh # Interactive cleanup utility -``` - -**Features:** -- **Auto-build**: Builds project if executable not found -- **Dependency checking**: Validates cmake, make, g++/clang++ -- **Cross-platform**: Works on macOS and Linux -- **Test runner**: Comprehensive test suite execution -- **Live trading**: Real WebSocket connections to exchanges +See [Security & API Key Management](docs/security/credentials.md) for credential details. -### Docker Script (`scripts/run-docker.sh`) +### Backtest Mode -**Available Commands:** ```bash -# Execution modes -scripts/run-docker.sh # Simulation mode (detached) -scripts/run-docker.sh -m live -v # Live mode (interactive) -scripts/run-docker.sh -s ETH-USD # Custom trading symbol - -# Container management -scripts/run-docker.sh build # Build Docker image -scripts/run-docker.sh logs # View container logs -scripts/run-docker.sh stop # Stop and remove containers -scripts/run-docker.sh clean # Remove containers and image -scripts/run-docker.sh --help # Show help +# Runs against historical data (auto-generates synthetic data if no CSV found) +./pinnaclemm --mode backtest --symbol BTC-USD ``` -**Features:** -- **Containerized**: Isolated execution environment -- **Auto-build**: Builds Docker image if not found -- **Container lifecycle**: Complete start/stop/clean management -- **Volume mounting**: Credential persistence for live mode -- **Production ready**: Optimized for deployment - -## Docker Deployment - -### Using Docker Script (Recommended) -```bash -# Quick start -scripts/run-docker.sh # Simulation mode -scripts/run-docker.sh -m live -v # Live trading - -# Container management -scripts/run-docker.sh logs # Monitor logs -scripts/run-docker.sh stop # Stop trading -``` - -### Using Pre-built Images (GitHub Container Registry) -```bash -# Pull the latest image -docker pull ghcr.io/chizy7/pinnaclemm:latest +Backtest mode prints a detailed performance report (Sharpe ratio, drawdown, win rate) and saves JSON results. See the [Advanced Backtesting Guide](docs/ADVANCED_BACKTESTING.md) for custom data and parameters. -# Run simulation mode -docker run --rm ghcr.io/chizy7/pinnaclemm:latest +For all command-line options, run modes, multi-instrument setups, and the visualization dashboard, see the [Getting Started Guide](docs/user_guide/getting_started.md). -# Setup credentials for live trading -docker run -it --rm -v $(pwd)/config:/app/config \ - ghcr.io/chizy7/pinnaclemm:latest --setup-credentials - -# Run live mode with credentials -docker run -it --rm -v $(pwd)/config:/app/config \ - ghcr.io/chizy7/pinnaclemm:latest --mode live --exchange coinbase --symbol BTC-USD --verbose -``` - -### Manual Docker Commands -```bash -# Build the Docker image locally -docker build -t pinnaclemm . - -# Run simulation mode -docker run -d --name pinnaclemm pinnaclemm +## Performance -# Run live mode with credentials -docker run -it --name pinnaclemm-live \ - -v $(pwd)/config:/app/config \ - pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --verbose -``` +- **Order Book Update Latency**: <1 μs +- **Order Execution Latency**: <50 μs end-to-end +- **Pre-Trade Risk Check**: ~750 ns (lock-free) +- **Circuit Breaker Check**: ~5 ns (single atomic load) +- **ML Prediction Latency**: 1–3 μs (neural network inference) +- **Throughput**: 100,000+ messages per second +- **Recovery Time**: <5 seconds for full system recovery +- **Memory Footprint**: <100 MB for core engine -## Key Components - -- **Order Book Engine**: Ultra-fast matching engine with lock-free operations -- **Market Making Strategy**: Adaptive pricing based on market conditions -- **Risk Manager**: Lock-free pre-trade risk checks with position, exposure, and loss limits -- **Circuit Breaker**: Market circuit breaker with 8 triggers and automatic recovery -- **VaR Engine**: Real-time Value at Risk with Monte Carlo simulations on a background thread -- **Alert Manager**: Alerting system with throttling and real-time WebSocket delivery -- **Disaster Recovery**: Atomic state persistence, position reconciliation, and backup management -- **ML Spread Optimization**: Neural network-based spread prediction with ~1-2μs latency -- **Order Book Flow Analyzer**: Real-time analysis of order flow patterns and market microstructure -- **Market Impact Predictor**: Advanced models for predicting price impact of trades -- **Market Regime Detector**: Hidden Markov Model-based detection of 8 market regimes -- **RL Parameter Adapter**: Reinforcement learning for dynamic strategy parameter optimization -- **Advanced Backtesting Engine**: Historical replay with Monte Carlo analysis and statistical testing -- **Instrument Manager**: Multi-instrument orchestration with per-symbol order books, strategies, and simulators -- **Arbitrage Detector**: Cross-exchange price discrepancy detection with fee-adjusted scanning -- **Cross-Market Correlation**: Statistical lead-lag analysis and cointegration testing for signal generation -- **Real-Time Visualization**: Web-based dashboard with Chart.js and D3.js visualization -- **FIX Protocol Engine**: Professional-grade FIX connectivity for institutional trading -- **Persistence System**: Crash recovery with memory-mapped files -- **Exchange Simulator**: Realistic market simulation for testing +See [Performance Benchmarks](docs/PERFORMANCE_BENCHMARKS.md) for methodology and full results. ## Documentation ### Core System -- [System Architecture](docs/architecture/system_overview.md) - [Getting Started Guide](docs/user_guide/getting_started.md) +- [System Architecture](docs/architecture/system_overview.md) - [API Reference](docs/api/reference.md) +- [Scripts Reference](docs/SCRIPTS.md) - [Project Roadmap](docs/ROADMAP.md) -### Risk Management & Production -- [Risk Management](docs/RISK_MANAGEMENT.md) - **Pre-trade checks, VaR, circuit breaker, alerting** -- [Disaster Recovery](docs/DISASTER_RECOVERY.md) - **Operational runbook for crash recovery and backups** -- [Kubernetes Deployment](docs/KUBERNETES_DEPLOYMENT.md) - **Production K8s deployment guide** - -### Advanced Features (ML) -- [ML Spread Optimization](docs/ML_SPREAD_OPTIMIZATION.md) - **Neural network-based spread prediction** -- [Order Book Flow Analysis](docs/ORDER_BOOK_FLOW_ANALYSIS.md) - **Real-time market microstructure analysis** -- [Market Impact Prediction](docs/MARKET_IMPACT_PREDICTION.md) - **Advanced trade impact modeling** -- [RL Parameter Adaptation](docs/RL_PARAMETER_ADAPTATION.md) - **Reinforcement learning optimization** -- [Market Regime Detection](docs/MARKET_REGIME_DETECTION.md) - **Hidden Markov Model regime detection** -- [Advanced Backtesting](docs/ADVANCED_BACKTESTING.md) - **Historical data replay and Monte Carlo analysis** -- [Strategy Performance Visualization](docs/STRATEGY_PERFORMANCE_VISUALIZATION.md) - **Real-time web dashboard** -- [WebSocket Testing Guide](docs/WEBSOCKET_TESTING.md) - **WebSocket connection testing and troubleshooting** - -## Real-Time Visualization Dashboard - -The PinnacleMM system includes a professional web-based dashboard for real-time performance monitoring and analysis: - -### Features -- **Live Performance Metrics**: Real-time P&L, position, Sharpe ratio, and win rate tracking -- **Interactive Charts**: Time-series visualization with Chart.js for performance trends -- **Market Data Visualization**: Live order book, spread analysis, and trade flow -- **ML Model Metrics**: Neural network accuracy, prediction times, and regime detection -- **WebSocket Real-Time Updates**: Sub-100ms latency for live data streaming -- **Multiple Dashboard Types**: Main dashboard, live trading dashboard, and test dashboard -- **Fixed WebSocket Issues**: Resolved segmentation faults and connection stability issues - -### Access Dashboard - -#### Simulation Mode Dashboard -1. **Start the system with visualization enabled:** - ```bash - cd build && ./pinnaclemm --mode simulation --enable-ml --enable-visualization --viz-ws-port 8089 --viz-api-port 8090 - ``` - -2. **Open the dashboard:** - - **Main Dashboard**: `visualization/static/index.html` (full-featured dashboard) - - **Test Dashboard**: `build/test_dashboard.html` (WebSocket connection testing) - - **WebSocket**: `ws://localhost:8089` (or your custom port) - - **REST API**: `http://localhost:8090` (or your custom port) - -#### Live Trading Dashboard -1. **Start live trading with visualization:** - ```bash - cd build && ./pinnaclemm --mode live --exchange coinbase --enable-ml --enable-visualization --viz-ws-port 8085 --viz-api-port 8086 --verbose - ``` - -2. **Access live dashboard:** - - **Live Dashboard**: `visualization/static/live_dashboard.html` (optimized for live trading) - - **Test Dashboard**: `build/test_dashboard.html` (connection diagnostics) - - **WebSocket**: `ws://localhost:8085` - - **REST API**: `http://localhost:8086` - -### Dashboard Components -- **Performance Cards**: Key metrics at-a-glance with real-time updates -- **Interactive Charts**: P&L, positions, spreads, ML accuracy with Chart.js -- **Real-Time Status**: Connection status indicators and live data feed monitoring -- **Strategy Controls**: Multiple strategy monitoring and comparison -- **Market Regime Visualization**: Real-time regime detection with confidence indicators -- **ML Metrics Panel**: Model accuracy, prediction latency, and retrain statistics -- **Risk Analysis**: Real-time VaR (historical, parametric, Monte Carlo), circuit breaker status, position/exposure limits, drawdown tracking, and alerting - -### Technical Details -- **Frontend**: HTML5/CSS3/JavaScript with Chart.js and D3.js -- **Backend**: Boost.Beast WebSocket and HTTP servers (migrated from websocketpp for compatibility) -- **Data Format**: JSON with efficient real-time streaming -- **Update Frequency**: 1-second intervals with configurable rates -- **Connection Stability**: Fixed shared_ptr lifecycle management for stable WebSocket connections -- **Debug Support**: Enhanced logging with `SPDLOG_LEVEL=debug` for troubleshooting - -### Testing WebSocket Connections -If you encounter connection issues, use the test dashboard: -```bash -# Start with debug logging -SPDLOG_LEVEL=debug ./pinnaclemm --mode simulation --enable-visualization --viz-ws-port 8089 --viz-api-port 8090 --verbose - -# Open test dashboard -open build/test_dashboard.html -# or manually: file:///path/to/PinnacleMM/build/test_dashboard.html -``` +### Risk Management & Production Operations +- [Risk Management](docs/RISK_MANAGEMENT.md) — pre-trade checks, VaR, circuit breaker, alerting +- [Disaster Recovery](docs/DISASTER_RECOVERY.md) — operational runbook for crash recovery and backups +- [Kubernetes Deployment](docs/KUBERNETES_DEPLOYMENT.md) — production K8s deployment guide +- [Persistence System](docs/architecture/persistence.md) +- [Security & API Key Management](docs/security/credentials.md) +- [Certificate Pinning](docs/security/CERTIFICATE_PINNING.md) + +### Machine Learning & Analytics +- [ML Spread Optimization](docs/ML_SPREAD_OPTIMIZATION.md) +- [Market Regime Detection](docs/MARKET_REGIME_DETECTION.md) +- [Market Impact Prediction](docs/MARKET_IMPACT_PREDICTION.md) +- [RL Parameter Adaptation](docs/RL_PARAMETER_ADAPTATION.md) +- [Order Book Flow Analysis](docs/ORDER_BOOK_FLOW_ANALYSIS.md) +- [Advanced Backtesting](docs/ADVANCED_BACKTESTING.md) +- [Strategy Performance Visualization](docs/STRATEGY_PERFORMANCE_VISUALIZATION.md) +- [JSON Data Export](docs/JSON_DATA_EXPORT.md) ### Multi-Instrument & Optimization -- [Multi-Instrument Guide](docs/MULTI_INSTRUMENT_GUIDE.md) - **Multi-symbol trading with InstrumentManager** -- [Cross-Exchange Arbitrage](docs/CROSS_EXCHANGE_ARBITRAGE.md) - **Venue spread detection and execution** -- [Cross-Market Correlation](docs/CROSS_MARKET_CORRELATION.md) - **Statistical correlation and signal generation** -- [Performance Optimization Guide](docs/PERFORMANCE_OPTIMIZATION_GUIDE.md) - **LTO, CPU affinity, object pooling, lock-free fixes** - -### Exchange Integration -- [FIX Protocol Integration Guide](docs/FIX_PROTOCOL_INTEGRATION.md) -- [FIX Testing Guide](docs/TESTING_GUIDE.md) -- [Interactive Brokers Setup](docs/IB_TESTING_GUIDE.md) +- [Multi-Instrument Guide](docs/MULTI_INSTRUMENT_GUIDE.md) +- [Cross-Exchange Arbitrage](docs/CROSS_EXCHANGE_ARBITRAGE.md) +- [Cross-Market Correlation](docs/CROSS_MARKET_CORRELATION.md) +- [Performance Optimization Guide](docs/PERFORMANCE_OPTIMIZATION_GUIDE.md) -### System Administration -- [Kubernetes Deployment](docs/KUBERNETES_DEPLOYMENT.md) -- [Disaster Recovery Runbook](docs/DISASTER_RECOVERY.md) -- [Persistence System](docs/architecture/persistence.md) -- [Recovery Guide](docs/user_guide/recovery.md) -- [Security & API Key Management](docs/security/credentials.md) -- [Certificate Pinning Guide](docs/security/CERTIFICATE_PINNING.md) +### Exchange Integration & Testing +- [FIX Protocol Integration](docs/FIX_PROTOCOL_INTEGRATION.md) +- [Interactive Brokers Setup](docs/IB_TESTING_GUIDE.md) +- [Testing Guide](docs/TESTING_GUIDE.md) +- [WebSocket Testing](docs/WEBSOCKET_TESTING.md) ## Technology Stack -- **Core Engine**: C++20 +- **Core Engine**: C++20, lock-free algorithms, `std::atomic` - **Build System**: CMake -- **Testing**: Google Test -- **Performance Benchmarking**: Google Benchmark -- **Concurrency**: Lock-free algorithms, std::atomic -- **Networking**: Boost.Beast WebSocket, hffix FIX protocol +- **Testing**: Google Test, Google Benchmark +- **Networking**: Boost.Beast (WebSocket), hffix (FIX protocol) - **Machine Learning**: Custom neural networks, Hidden Markov Models, reinforcement learning -- **Visualization**: HTML5/CSS3/JavaScript frontend, Chart.js, D3.js, WebSocket real-time updates -- **Security**: OpenSSL for encryption +- **Visualization**: HTML5/JavaScript frontend with Chart.js and D3.js +- **Security**: OpenSSL, AES-256-CBC encryption, PBKDF2 key derivation - **Configuration**: nlohmann/json -- **Containerization**: Docker -- **Security**: AES-256-CBC encryption, PBKDF2 key derivation, input validation, audit logging, rate limiting - -## Performance +- **Deployment**: Docker, Kubernetes -PinnacleMM achieves exceptional performance metrics: +## Contributing -- **Order Book Update Latency**: <1 μs (microsecond) -- **Order Execution Latency**: <50 μs (end-to-end) -- **Pre-Trade Risk Check**: ~750ns (lock-free, 8 sequential checks) -- **Circuit Breaker Check**: ~5ns (single atomic load) -- **ML Prediction Latency**: 1-3 μs (neural network inference) -- **Throughput**: 100,000+ messages per second -- **Recovery Time**: <5 seconds for full system recovery -- **Memory Footprint**: <100 MB for core engine -- **Dashboard Update Latency**: <100ms (real-time visualization) -- **Regime Detection**: Real-time with 85%+ confidence accuracy +Contributions are welcome! Please read the [Contributing Guide](CONTRIBUTING.md) for development setup, coding standards, pre-commit hooks, and the pull request process. All participants are expected to follow our [Code of Conduct](CODE_OF_CONDUCT.md). -### Testing Integration - -```bash -# Test FIX protocol components -cd build -./fix_basic_test - -# Expected output: -# ✓ Factory instance created -# ✓ Interactive Brokers FIX support: Yes -# ✓ Configuration system working -# ✓ Order creation working - -# Test advanced order routing system -./routing_test - -# Expected output: -# All OrderRouter tests passed successfully! -# ✓ BestPriceStrategy, TWAP, VWAP, MarketImpact all working -# ✓ Multi-venue execution with 1ms latency -# ✓ 8 completed fills across multiple strategies - -# Test risk management components (Phase 4) -./risk_manager_tests # 11 tests - pre-trade checks, position limits -./circuit_breaker_tests # 10 tests - state machine, triggers -./var_engine_tests # 8 tests - VaR calculations -./alert_manager_tests # 8 tests - alerting, throttling -./disaster_recovery_tests # 8 tests - state persistence, backups -./risk_check_benchmark # Risk check latency benchmarks - -# Test multi-instrument and optimization components (Phase 5) -./instrument_manager_tests # 9 tests - lifecycle management -./arbitrage_detector_tests # 8 tests - opportunity detection, fees -./cross_market_correlation_tests # 7 tests - correlation, lead-lag -./multi_instrument_benchmark # Startup scaling benchmarks - -# Memory safety validation with Address Sanitizer (development builds) -cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_SANITIZERS=ON .. && make -j8 -./pinnaclemm --mode simulation --symbol BTC-USD --verbose -# ASan will detect memory leaks, buffer overflows, and use-after-free errors -``` +- **Bugs & feature requests**: use the [issue templates](https://github.com/chizy7/PinnacleMM/issues/new/choose) +- **Security vulnerabilities**: report privately via the [Security Policy](.github/SECURITY.md) — never in a public issue +- **Looking for ideas?** Check the [Roadmap](docs/ROADMAP.md) ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. -## Contact Me +## Contact For questions, feedback, or collaboration opportunities: - **Email**: [chizy@chizyhub.com](mailto:chizy@chizyhub.com) -- **X(Twitter)**: [![Twitter Follow](https://img.shields.io/twitter/follow/chizyization?style=social)](https://x.com/Chizyization) +- **X (Twitter)**: [![Twitter Follow](https://img.shields.io/twitter/follow/chizyization?style=social)](https://x.com/Chizyization) diff --git a/docs/JSON_DATA_EXPORT.md b/docs/JSON_DATA_EXPORT.md new file mode 100644 index 0000000..f5841b1 --- /dev/null +++ b/docs/JSON_DATA_EXPORT.md @@ -0,0 +1,61 @@ +# JSON Data Export + +PinnacleMM provides comprehensive structured data export capabilities through JSON Lines (JSONL) format logging. This feature enables detailed analysis, backtesting, monitoring, and debugging of trading strategies and market data. + +## Features + +- **Market Data Logging**: Real-time price, volume, bid/ask data with timestamps +- **Strategy Metrics**: Position, P&L, quote counts, and performance statistics +- **Order Book Updates**: Complete order book state with bid and ask arrays +- **Connection Events**: WebSocket connections, disconnections, and errors +- **Trading Events**: Order placements, fills, cancellations, and status updates +- **Thread-Safe**: Concurrent logging without performance impact +- **JSONL Format**: One JSON object per line for easy parsing and streaming + +## Usage + +Enable JSON logging with the `--json-log` flag and optionally specify a custom file path: + +```bash +# Basic JSON logging (default file: pinnaclemm_data.jsonl) +./pinnaclemm --mode simulation --symbol BTC-USD --json-log + +# Custom file path +./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --json-log --json-log-file my_trading_data.jsonl + +# Combined with other features +./pinnaclemm --mode live --enable-ml --enable-visualization --json-log --json-log-file full_session.jsonl +``` + +## Sample Output + +```json +{"format":"jsonl","timestamp":"2025-09-29T16:27:06.770Z","type":"session_start","version":"1.0.0"} +{"metrics":{"ask_price":67252.80,"bid_price":67248.20,"market_price":67250.45,"pnl":0.0,"position":0.0,"quote_updates":1,"strategy_name":"BasicMarketMaker","volume":1234.56},"strategy_name":"BasicMarketMaker","symbol":"BTC-USD","timestamp":"2025-09-29T16:27:12.011Z","type":"strategy_metrics"} +{"ask_price":67253.00,"bid_price":67249.50,"event_timestamp":1695736032610,"is_buy":true,"price":67250.75,"symbol":"BTC-USD","timestamp":"2025-09-29T16:27:12.610Z","type":"market_update","volume":0.5} +``` + +## Data Types + +- **`session_start`**: Session initialization marker with format and version +- **`strategy_metrics`**: Trading strategy performance and position data +- **`market_update`**: Real-time market data from exchange feeds +- **`order_book_update`**: Complete order book snapshots with bid/ask arrays +- **`trading_event`**: Order lifecycle events and trading actions +- **`connection_event`**: Exchange connectivity status and errors + +## File Management + +JSON log files are created in the current working directory by default. For production use, consider: + +```bash +# Save to logs directory (create and add to .gitignore) +mkdir -p logs +./pinnaclemm --json-log --json-log-file logs/trading_$(date +%Y%m%d_%H%M%S).jsonl + +# Save to data directory (existing, likely gitignored) +./pinnaclemm --json-log --json-log-file data/market_data_$(date +%Y%m%d).jsonl + +# Save outside project directory +./pinnaclemm --json-log --json-log-file ~/trading_logs/pinnaclemm_session.jsonl +``` diff --git a/docs/SCRIPTS.md b/docs/SCRIPTS.md new file mode 100644 index 0000000..af0282c --- /dev/null +++ b/docs/SCRIPTS.md @@ -0,0 +1,98 @@ +# Helper Scripts Reference + +PinnacleMM ships with bash scripts that handle building, running, testing, and container management. Both scripts print full usage with `--help`. + +## Native Script (`scripts/run-native.sh`) + +Recommended for development and live trading. + +```bash +# Execution modes +scripts/run-native.sh # Simulation mode (default) +scripts/run-native.sh -m live -v # Live mode with verbose logging +scripts/run-native.sh -s ETH-USD # Custom trading symbol +scripts/run-native.sh -e coinbase # Specify exchange + +# Build commands +scripts/run-native.sh build # Build project +scripts/run-native.sh clean # Clean build directory +scripts/run-native.sh rebuild # Clean and rebuild + +# Testing and benchmarks +scripts/run-native.sh test # Run all tests +scripts/run-native.sh benchmark # Run performance benchmarks + +# Setup +scripts/run-native.sh --setup-credentials # Configure API credentials (secure input) +scripts/run-native.sh --help # Show help + +# Cleanup +scripts/cleanup.sh # Interactive cleanup utility +``` + +**Features:** + +- **Auto-build**: Builds project if executable not found +- **Dependency checking**: Validates cmake, make, g++/clang++ +- **Cross-platform**: Works on macOS and Linux +- **Test runner**: Comprehensive test suite execution +- **Live trading**: Real WebSocket connections to exchanges + +## Docker Script (`scripts/run-docker.sh`) + +Recommended for production and simulation deployments. + +```bash +# Execution modes +scripts/run-docker.sh # Simulation mode (detached) +scripts/run-docker.sh -m live -v # Live mode (interactive) +scripts/run-docker.sh -s ETH-USD # Custom trading symbol + +# Container management +scripts/run-docker.sh build # Build Docker image +scripts/run-docker.sh logs # View container logs +scripts/run-docker.sh stop # Stop and remove containers +scripts/run-docker.sh clean # Remove containers and image +scripts/run-docker.sh --help # Show help +``` + +**Features:** + +- **Containerized**: Isolated execution environment +- **Auto-build**: Builds Docker image if not found +- **Container lifecycle**: Complete start/stop/clean management +- **Volume mounting**: Credential persistence for live mode +- **Production ready**: Optimized for deployment + +## Pre-built Docker Images + +Images are published to GitHub Container Registry: + +```bash +# Pull the latest image +docker pull ghcr.io/chizy7/pinnaclemm:latest + +# Run simulation mode +docker run --rm ghcr.io/chizy7/pinnaclemm:latest + +# Setup credentials for live trading +docker run -it --rm -v $(pwd)/config:/app/config \ + ghcr.io/chizy7/pinnaclemm:latest --setup-credentials + +# Run live mode with credentials +docker run -it --rm -v $(pwd)/config:/app/config \ + ghcr.io/chizy7/pinnaclemm:latest --mode live --exchange coinbase --symbol BTC-USD --verbose +``` + +## Choosing Between Them + +| Feature | Native (`run-native.sh`) | Docker (`run-docker.sh`) | +|---------|--------------------------|--------------------------| +| Simulation mode | Yes | Yes | +| Live trading | Real WebSocket data | Known WebSocket config limitation | +| Auto-build | Builds if needed | Auto Docker build | +| Test runner | `test` command | Not included | +| Benchmarks | `benchmark` command | Not included | +| Credential setup | Interactive setup | Volume mounting | +| Dependencies | cmake, make, g++/clang++ | Docker only | +| Best for | Development and live trading | Production and simulation | From c62d9f7dca7f7a136207461534cc2fad648bb6a7 Mon Sep 17 00:00:00 2001 From: Chizy Date: Wed, 19 Aug 2026 23:51:39 -0400 Subject: [PATCH 2/2] docs: address coderabbit review feedback --- CODE_OF_CONDUCT.md | 2 +- CONTRIBUTING.md | 4 ++-- README.md | 4 ++-- docs/JSON_DATA_EXPORT.md | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 429e82f..b3948b3 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -68,7 +68,7 @@ If an investigation by the Community Moderators finds that this Code of Conduct 2. Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior. 3. Repair: There is no possible repair in cases of this severity. -This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community. +This enforcement ladder is intended as a guideline. It does not limit the ability of Community Moderators to use their discretion and judgment, in keeping with the best interests of our community. ## Scope diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index aac1c08..fd78b7c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,7 @@ CI runs the same hooks on every PR (`.github/workflows/pre-commit.yml`), so skip PR titles **must** follow [Conventional Commits](https://www.conventionalcommits.org/) — this is enforced by CI (`pr-title-lint.yml`) and PRs with non-conforming titles cannot merge: -``` +```text (): ``` @@ -121,7 +121,7 @@ PR titles **must** follow [Conventional Commits](https://www.conventionalcommits Examples: -``` +```text feat(strategy): add inventory-skew quoting to basic market maker fix(exchange): handle coinbase websocket reconnect race perf(core): reduce order book update latency with arena allocator diff --git a/README.md b/README.md index 344d528..d22e551 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ cd build && ./pinnaclemm --mode simulation --symbol BTC-USD scripts/run-native.sh --setup-credentials # Start live trading -./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --verbose +cd build && ./pinnaclemm --mode live --exchange coinbase --symbol BTC-USD --verbose ``` See [Security & API Key Management](docs/security/credentials.md) for credential details. @@ -119,7 +119,7 @@ See [Security & API Key Management](docs/security/credentials.md) for credential ```bash # Runs against historical data (auto-generates synthetic data if no CSV found) -./pinnaclemm --mode backtest --symbol BTC-USD +cd build && ./pinnaclemm --mode backtest --symbol BTC-USD ``` Backtest mode prints a detailed performance report (Sharpe ratio, drawdown, win rate) and saves JSON results. See the [Advanced Backtesting Guide](docs/ADVANCED_BACKTESTING.md) for custom data and parameters. diff --git a/docs/JSON_DATA_EXPORT.md b/docs/JSON_DATA_EXPORT.md index f5841b1..ba5053b 100644 --- a/docs/JSON_DATA_EXPORT.md +++ b/docs/JSON_DATA_EXPORT.md @@ -9,7 +9,7 @@ PinnacleMM provides comprehensive structured data export capabilities through JS - **Order Book Updates**: Complete order book state with bid and ask arrays - **Connection Events**: WebSocket connections, disconnections, and errors - **Trading Events**: Order placements, fills, cancellations, and status updates -- **Thread-Safe**: Concurrent logging without performance impact +- **Thread-Safe**: Concurrent writes are serialized; logging adds mutex and file-I/O overhead - **JSONL Format**: One JSON object per line for easy parsing and streaming ## Usage