AI-Driven Snake Game using Deep Q-Learning

This project implements a classic Snake game where an AI agent learns to play autonomously using Deep Q-Learning (DQN). The game is built with pygame, featuring a modern dark-neon UI, and the learning agent uses a small feed-forward neural network implemented in PyTorch.

The agent learns by interacting with the environment:

• Observes the current game state (11 handcrafted features)
• Chooses an action (go straight, turn right, or turn left)
• Receives a reward (positive for eating food, negative for dying)
• Updates its policy over many games to survive longer and collect more food

---

✨ Features & UI/UX Upgrades

The project has been fully visually upgraded from a basic grid to a premium modern aesthetic:

• Dark Neon Theme: Deep navy background with glowing neon accents.
• Live Stats Sidebar: Real-time Heads-Up Display showing Game #, Score, Record, Snake Length, Epsilon rate, and FPS.
• Snake Design: Smooth gradient body segments with a glowing neon head and directional eyes.
• Visual Effects: Pulsing food with glow rings, and dynamic particle burst effects upon eating.
• Dark-Themed Plotting: Training progress auto-generates beautiful dark-themed charts.

---

11-Dimensional State Representation

Instead of feeding the full game grid into the network, we use a compact 11-feature state vector that captures what the agent really needs to know at each step.

The state is:

state = [
    # Danger (3)
    danger_straight,   # 0 or 1
    danger_right,      # 0 or 1
    danger_left,       # 0 or 1

    # Direction (4)
    dir_left,          # 0 or 1
    dir_right,         # 0 or 1
    dir_up,            # 0 or 1
    dir_down,          # 0 or 1

    # Food (4)
    food_left,         # 0 or 1
    food_right,        # 0 or 1
    food_up,           # 0 or 1
    food_down          # 0 or 1
]

Group 1 — Danger Detection (3 features)

These indicate whether moving in a given relative direction will cause an immediate collision in the next move:

• danger_straight
• danger_right
• danger_left

They are defined relative to the current direction of travel (e.g. if moving up, “straight” means up, “right” means right, etc.).

Group 2 — Current Direction (4 features)

One-hot encoding of the snake’s current movement direction:

• dir_left
• dir_right
• dir_up
• dir_down

Exactly one of these is 1 at any time.

Group 3 — Food Location (4 features)

Binary indicators of where the food is relative to the snake’s head:

• food_left
• food_right
• food_up
• food_down

Multiple of these can be 1 at once (for example, when the food is diagonally positioned).

---

Reward Structure

The agent is trained with a simple reward scheme:

Event	Reward
Eat food	+10
Game continues	0
Collision	-10

---

Tech Stack

• Python for all logic
• Pygame for the modern Snake game environment and rendering
• PyTorch for the Deep Q-Network
• NumPy for numerical operations
• Matplotlib for plotting scores and training progress

---

Algorithms and AI Techniques

This project demonstrates several core reinforcement learning and deep learning ideas:

• Deep Q-Learning (DQN)

  • Approximates the Q-function (Q(s,a)) with a neural network instead of a table.
  • Network architecture: 11 → 256 → 256 → 3 fully connected layers, outputting Q-values for relative actions [straight, right, left].

• Handcrafted State Representation (Feature Engineering)

  • Compact 11-dimensional binary state capturing: immediate danger directions, current movement direction, and food location relative to the head.
  • Greatly reduces input size compared to raw pixels, making training faster and more stable.

• Epsilon-Greedy Exploration

  • Balances exploration (random actions) and exploitation (greedy actions from the DQN).
  • The exploration rate epsilon decays with the number of games so the agent explores more at the beginning and becomes greedier as it learns.

• Experience Replay (Replay Buffer)

  • Stores past transitions ((state, action, reward, next_state, done)) in a fixed-size deque.
  • Samples random mini-batches during training to break correlation between consecutive frames and improve data efficiency.

• Online Training with Discounted Rewards

  • Uses a discount factor (\gamma) to weigh future rewards when computing target Q-values.
  • Trains both after each step (short memory) and after each episode over a sampled batch (long memory).

Together, these components allow the agent to learn a robust policy for playing Snake from raw interaction with the environment.

---

Example Training Progress

Below is an example of how the training curve typically looks after the agent has played a number of games. The blue curve is the raw score for each game, and the orange curve is the running mean score. (This plot auto-generates silently in the background during training).

You should expect a phase of mostly low scores at the beginning, followed by a clear upward trend in both maximum and mean scores as the agent learns better navigation strategies.

---

Running the Project

1. Install dependencies:

pip install -r requirements.txt

2. Train the agent:

python train.py

This will:

• Open a Pygame window with the Snake game.
• Let the agent play continuously, improving over time.
• Generate a Training_Progress.png plot in the folder that updates silently after each game.

3. Watch the AI play more clearly (slower, more visual):

In train.py, change the last line:

if __name__ == "__main__":
    # Set watch_mode=True to slow down and better visualize gameplay
    train(watch_mode=True)

This uses a lower frame rate so you can visually inspect the learned behavior and enjoy the neon particle UI.

---

Project Structure

• ai_snake_env/snake_game.py – Modern Pygame-based Snake environment, reward logic, and 11-feature state.
• model.py – Deep Q-Network (11 → 256 → 256 → 3) and training helper.
• agent.py – RL agent with replay memory, epsilon-greedy policy, and training methods.
• helper.py – Matplotlib plotting of scores and mean scores (runs in background).
• train.py – Main training loop and watch mode entrypoint.
• requirements.txt – Python dependencies.

---

Learning Loop Summary

On each step:

1. The environment exposes the 11-feature state.
2. The agent chooses an action (straight/right/left) using an epsilon-greedy policy over DQN Q-values.
3. The game advances one step, returning reward, next_state, done, score.
4. The agent stores the transition in a replay buffer and updates the DQN.
5. At the end of each game, the agent performs additional training from a mini-batch of past experiences and saves the model when a new record score is reached.