Tools for detecting when a backtest is fooling you.
Trying many strategy configurations and reporting the best one's Sharpe is the most common way to produce a spurious "edge" — the winner is often just the luckiest of many coin-flips. This library quantifies how much of an apparent edge is an artifact of selection and multiple testing, using the standard methods from the Bailey / López de Prado literature.
import numpy as np
from overfit import analyze, format_report
returns = np.random.normal(0, 0.01, size=(1000, 200)) # 200 strategies × 1000 days, no real edge
print(format_report(analyze(returns)))
# VERDICT: OVERFIT: not significant after deflation AND high overfitting probability| Function | Question it answers |
|---|---|
probabilistic_sharpe_ratio |
Is a Sharpe significantly above a benchmark, given the returns' skew & kurtosis and the sample length? |
expected_max_sharpe |
What Sharpe would I expect from the best of N random trials with no true edge? |
deflated_sharpe_ratio (DSR) |
Is the Sharpe significant after deflating for how many strategies I tried? |
min_track_record_length |
How long must a track record be before I can trust a Sharpe? |
min_backtest_length |
How many years of backtest do N trials require before a given Sharpe means anything? |
pbo_cscv |
Probability of Backtest Overfitting via combinatorially symmetric cross-validation. |
analyze / format_report |
One-call workup on a (T × N) matrix of strategy returns. |
monte_carlo_drawdown |
If this edge is real, how bad could the ride actually get? Shuffles the same trades into thousands of orderings and reports the max-drawdown distribution — see case study #4. |
python demo.py runs two scenarios and the min-backtest-length table:
- Scenario A — data mining. 200 pure-noise strategies; the best shows an annualized Sharpe of 1.34 (looks publishable). The tools expose it: Deflated Sharpe 0.36, PBO 0.50, negative IS→OOS degradation → OVERFIT.
- Scenario B — a strong real edge (Sharpe ~3) hidden among 199 noise strategies. It's selected and survives: DSR 0.998, PBO 0.04. (A modest real edge of ~1.3 gets drowned by 200 noise trials — which is itself the honest lesson.)
- Min backtest length: try 1000 configurations and an annualized Sharpe of 1.0 needs ~10 years of data before it means anything.
A tool you can't trust is worse than none, so the test suite asserts both directions: it must flag cherry-picked noise and let a genuine edge through.
case_studies/zero-dte-audit.md — I ran the toolkit against
a 6-config strategy sweep from my own 0DTE research. With no code inspection, "pick the best
in-sample Sharpe" selected exactly the config containing a lookahead bug, and the tools
convicted it (DSR 0.34, PBO 0.60, IS→OOS slope −1.44 → OVERFIT) — reproducing in seconds
a verdict that originally took a manual bug-hunt.
case_studies/daily-meanrev.md — the mirror image: SPY
daily mean reversion is a real anomaly (18 trials, PBO 0.002, every long variant
profitable after costs) that still fails as a strategy (DSR 0.59; walk-forward Sharpe 0.26
vs buy-and-hold 0.53). Real-but-weak edges produce exactly this PBO≈0 / DSR<0.95 signature —
the two metrics measure different things, and you need both. Fully reproducible from public
data: python case_studies/daily_meanrev_repro.py.
case_studies/insider-gap.md — the third failure mode:
an execution assumption. SEC Form 4 insider purchases show t≈3 abnormal returns and a
net Sharpe of +3.1 entering at the filing-day close — but filings mostly arrive after hours,
and at the honest entry (next day's open) the edge vanishes entirely (all |t|<2, every
variant net-negative → OVERFIT). The whole effect lives in the untradeable overnight gap.
case_studies/turn-of-month-drawdown.md — the
case where every gate passes: turn-of-month t5 scores DSR 1.0, PBO 0.01,
walk-forward 5/5 — the cleanest statistical profile in my knowledge base. DSR and PBO
answer "is this edge real, and not an artifact of trying many configs?" They say nothing
about what living through it would feel like. Reshuffling the same 3,750 daily returns
5,000 times puts the typical max drawdown at 26.0% and the unlucky-but-ordinary p95 at
37.6% — the single historical path you observed is one draw from a much wider range.
Surviving the significance tests is not the same as being survivable.
Code bug (study #1), weak edge (study #2), false execution assumption (study #3), and a real edge you may not be able to sit through (study #4) — four different ways a good-looking backtest misleads you.
pip install -r requirements.txt # numpy, scipy
python demo.py # see it in action
pytest -q # 9 property/behaviour tests
Use it on your own research by passing a (T observations × N strategies) matrix of period
returns to analyze(...).
- Bailey & López de Prado (2014), The Deflated Sharpe Ratio, Journal of Portfolio Management.
- Bailey, Borwein, López de Prado & Zhu (2015), The Probability of Backtest Overfitting, Journal of Computational Finance.
- Bailey, Borwein, López de Prado & Zhu (2014), Pseudo-Mathematics and Financial Charlatanism, Notices of the AMS.
MIT licensed.