trade-statistics — Trade History Analytics & Review Automation Pipeline

Purpose

A pipeline that takes discretionary-trading history CSVs and statistically visualizes which conditions lose money and when decision quality degrades, automating the human review loop.

The design philosophy is not to automate trading decisions, but to produce early-warning material that protects human decision making. False alarms (flagging harmless trades) and misses (overlooking dangerous decision patterns) are treated as distinct costs — the same evaluation design used in equipment diagnostics and anomaly detection.

Features

• 9-step analysis pipeline: performance report → feature enrichment (100+ indicators) → avoid-condition mining → feature importance → currency strength → deep-dive diagnostics → HTML report → Pine parameter optimization → Pine code generation
• Multi-method feature importance: combines Gini importance, Random Forest, permutation importance, SHAP, and statistical tests into a composite rank to avoid single-method bias
• Leakage-aware validation: Pine parameter thresholds are validated with a chronological train/test split; in-sample performance alone never selects a threshold
• Human-discretion diagnostics (human_discretion_diagnostics.py): extracts contexts where judgment quality degrades — re-entries right after a loss (revenge-trade candidates), additional trades after exceeding a daily stop line, losing positions held too long
• Monte Carlo simulation: estimates risk-of-ruin and drawdown distributions via resampling
• Multi-broker support: normalizes CSV formats from different brokers into a unified round-trip format (merge_brokers.py)

Quick Start (reproducible with synthetic data)

No real data, broker account, or MT5 required — three commands from synthetic data to diagnostics.

pip install -r requirements.txt

# 1. Generate synthetic trade history (fixed seed, reproducible)
python scripts/generate_synthetic_sample.py --out data/sample_trades.csv

# 2. Performance profile (loss clusters, time buckets, daily-stop candidates)
python scripts/sample_data_profile.py data/sample_trades.csv --out-dir output

# 3. Discretion-degradation diagnostics
python scripts/human_discretion_diagnostics.py data/sample_trades.csv --out-dir output

Outputs are written to output/ as CSV / JSON / Markdown review files.

For real broker CSVs (supported: OANDA MT5, GMO Click Securities, Matsui Securities), see docs/OPERATIONS.md (Japanese).

Input Format

Normalized round-trip CSV: entry_datetime, exit_datetime, symbol, side_buy (1=long / 0=short), qty, entry_price, exit_price, pnl.

Architecture

broker CSV (OANDA / GMO / Matsui)
        │  merge_brokers.py (format normalization)
        ▼
normalized round-trip CSV
        │
        ├── sample_data_profile.py           performance segmentation
        ├── human_discretion_diagnostics.py  decision-degradation events
        ├── trade_feature_analysis.py        OHLCV feature enrichment (MT5 required)
        │       ├── avoid_conditions.py      avoid-condition mining
        │       ├── feature_importance_analysis.py  SHAP/RF/permutation composite
        │       └── generate_html_report.py  12-tab Plotly dashboard
        └── monte_carlo_sim.py               risk simulation

Details: docs/ARCHITECTURE.md, docs/OPERATIONS.md, docs/PINE_INTEGRATION.md (Japanese).

Evaluation Design

• Avoid-conditions and thresholds are validated with chronological train/test splits (random splits would leak future information)
• Feature importance uses a composite of six methods for robustness
• Permutation tests check whether performance differences are distinguishable from chance
• Warning events are evaluated by their impact on subsequent PnL, not by raw counts

Limitations & Untested Areas

• The full feature-enriched pipeline (Step 2 onward) requires MT5 on Windows; synthetic data reproduces the performance profile and discretion diagnostics only
• Synthetic data mimics statistical properties (negative edge, fat tails) but not real market microstructure
• Avoid-conditions can overfit on small segments; out-of-sample re-validation is assumed before operational use
• Discretion-event definitions (e.g., revenge trades) are heuristics, not direct measurements of psychological state

Background

Built to solve a personal pain point in FX/CFD trading — "verbalizing losing patterns depends on subjective memory" — by learning the required statistics and ML techniques and implementing the review loop end to end. An example of solving a field problem by acquiring and applying the necessary technology.

License

MIT