Regime-Shift Market Detection using Hidden Markov Models

Overview

This project implements a quantitative trading framework using Hidden Markov Models (HMMs) to detect hidden market regimes and dynamically allocate portfolio exposure based on changing market conditions.

The system uses macro and asset-based indicators such as SPY returns, volatility, VIX changes, Gold (GLD), and Treasury Bonds (TLT) to identify latent market states and evaluate strategy performance against benchmark portfolios.

The notebook includes:

• Hidden Markov Model regime detection
• Walk-forward validation
• Dynamic allocation strategy
• Benchmark backtesting
• Equity curve visualization
• Performance tear sheet analytics

---

Project Architecture

Architecture Decisions

1. Hidden Markov Model (HMM)

A Gaussian Hidden Markov Model was selected because financial markets exhibit hidden latent states that cannot be directly observed.

HMMs are well-suited for:

• Regime detection
• Sequential time-series modeling
• Probabilistic state transitions
• Financial market state inference

The model identifies hidden regimes such as:

• Bull Market
• Bear Market
• High Volatility / Crisis Regimes

The transition probability matrix (model.transmat_) is analyzed to understand:

• Regime persistence
• Transition likelihoods
• Market state dynamics

---

2. Feature Engineering

The following features were selected to capture market structure and macro behavior:

Feature	Purpose
SPY Returns	Market direction
Rolling Volatility	Risk regime detection
VIX Change	Fear and uncertainty indicator
GLD Returns	Flight-to-safety behavior
TLT Returns	Bond market reaction

Rolling volatility uses a 20-day rolling window.

---

3. Robust Scaling

RobustScaler was used instead of StandardScaler because financial data contains:

• Outliers
• Fat tails
• Extreme volatility spikes

Robust scaling improves model stability during crisis periods.

---

4. Dynamic Allocation Strategy

Portfolio exposure changes depending on detected regimes.

Regime	Allocation Logic
Bull	Higher SPY exposure
Bear	Defensive positioning
Crisis / Neutral	Increased bond and gold allocation

This improves:

• Risk-adjusted returns
• Drawdown protection
• Portfolio stability

---

5. Walk-Forward Validation

To reduce overfitting and improve robustness, the project implements walk-forward validation using rolling training and testing windows.

Workflow:

• Train HMM on rolling historical windows
• Predict unseen future regimes
• Refit periodically
• Aggregate out-of-sample predictions

This improves:

• Generalization
• Robustness
• Realistic evaluation

---

6. Benchmark Comparison

The strategy is evaluated against:

• Buy & Hold SPY
• Traditional 60/40 Portfolio

Performance metrics include:

• CAGR
• Volatility
• Sharpe Ratio
• Sortino Ratio
• Maximum Drawdown
• Calmar Ratio
• Win Rate
• Profit Factor
• Alpha
• Beta
• Information Ratio
• Portfolio Turnover

---

Repository Structure

regime-shift/
│
├── notebook/
│   └── regime-strategy.ipynb
│
├── data/
├── README.md
└── requirements.txt

---

Installation

Clone the Repository

git clone https://github.com/your-username/regime-shift.git
cd regime-shift

Install Dependencies

pip install -r requirements.txt

---

Running the Project

Launch Jupyter Notebook

jupyter notebook

Open Notebook

notebook/regime-strategy.ipynb

Run all notebook cells sequentially.

---

Testing Instructions

1. Data Loading Test

Verify financial data downloads correctly:

data.head()

Expected Output

• SPY
• GLD
• TLT
• ^VIX columns

---

2. Feature Engineering Test

Check feature creation:

features.head()

Expected Features

• SPY_returns
• SPY_volatility
• VIX_change
• GLD_returns
• TLT_returns

---

3. HMM Training Test

Run model training:

model.fit(X_scaled)

Expected Output

• No convergence errors
• Hidden states generated successfully

---

4. Regime Detection Test

Check regime assignments:

regime_data[["Regime"]].head()

Expected Output

• Integer regime labels
• Multiple regime states detected

---

5. Walk-Forward Validation Test

Verify rolling out-of-sample regime predictions:

walk_forward_df.head()

Expected Output

• Predicted regime labels
• Rolling out-of-sample predictions
• Multiple validation windows

---

6. Backtesting Test

Verify strategy outputs:

portfolio_returns.head()

Expected Output

• Non-empty return series
• Dynamic allocation changes

---

7. Performance Metrics Test

Run evaluation metrics:

comparison

Expected Metrics

• CAGR
• Volatility
• Sharpe Ratio
• Sortino Ratio
• Max Drawdown
• Calmar Ratio
• Turnover
• Alpha/Beta
• Information Ratio

---

Reproducibility Guide

Environment

Recommended:

• Python 3.10+
• Jupyter Notebook

---

Random Seed

The HMM model uses:

random_state = 42

to ensure reproducibility.

---

Data Source

Market data is downloaded using:

yfinance

Assets

• SPY
• GLD
• TLT
• ^VIX

Time Period

• 2015 – Present

---

Deterministic Results

To reproduce results:

1. Install exact dependencies
2. Use same notebook execution order
3. Keep identical random seed
4. Use same date range
5. Execute all cells sequentially

---

Results

The strategy achieved:

• Higher Sharpe Ratio than Buy & Hold SPY
• Lower volatility
• Lower maximum drawdown
• Better downside protection
• Improved risk-adjusted returns

---

Example Comparison

Metric	Regime Strategy	SPY
Sharpe Ratio	Higher	Lower
Volatility	Lower	Higher
Max Drawdown	Lower	Higher

---

Future Improvements

Potential upgrades:

• Regime probability weighting
• Transaction cost modeling
• Multi-asset optimization
• Deep learning regime models
• Macro-economic indicators
• Reinforcement learning allocation systems

---

Technologies Used

• Python
• pandas
• numpy
• matplotlib
• scikit-learn
• hmmlearn
• yfinance

---

Disclaimer

This project is for educational and research purposes only and should not be considered financial advice.