This is a Reinforcement Learning project I put together to explore market microstructure, specifically the dynamics of the Glosten-Milgrom (1985) model.
Instead of just deriving the math, I wanted to see if an RL agent (using PPO) could naturally learn to protect itself against toxic order flow just by interacting with a simulated market and losing virtual money.
If you're a market maker, you make money by quoting a spread to random "noise" traders. But you face one massive risk: Informed Traders. If someone secretly knows the true price of the asset is about to drop, they will sell everything to you at your Bid.
In this custom Gymnasium environment:
- The Agent is the market maker. It's blind to the true price. It only sees its current inventory and the direction of the last trade.
- Noise Traders buy/sell randomly.
- Informed Traders only trade when they can exploit the agent's quotes.
The agent's job is to choose its spread width at each tick to maximize PnL while managing its inventory risk (using a quadratic penalty).
I kept the stack standard and lightweight: Gymnasium for the environment and Stable-Baselines3 for the RL implementation.
pip install gymnasium stable-baselines3 numpy matplotlib