🧠 Mini GPT — A 10.8M Parameter Transformer Language Model

A from-scratch implementation of a GPT-style Transformer language model built entirely with PyTorch. This project follows the decoder-only Transformer architecture (the same architecture behind ChatGPT, Claude, and Gemini) to learn and generate text character by character.

---

📊 Model Specifications

Property	Value
Total Parameters	10.8 Million
Architecture	Decoder-Only Transformer (GPT-style)
Embedding Dimension	384
Attention Heads	6
Transformer Blocks (Layers)	6
Context Window (Block Size)	256 characters
Dropout	0.2
Vocabulary Size	~87 (unique characters in the training text)
Tokenization	Character-Level
Optimizer	AdamW
Learning Rate	3e-4

---

🏗️ Architecture Overview

The model is built from the following components, stacked together to form a complete Transformer:

Input Text
    │
    ▼
┌─────────────────────────┐
│  Token Embedding        │  Each character → 384-dim vector
│  + Position Embedding   │  Each position → 384-dim vector
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│   Transformer Block x6  │  ◄── This block is repeated 6 times
│  ┌───────────────────┐  │
│  │ Layer Norm        │  │
│  │ Multi-Head Attn   │  │  6 heads, each looking at 64 dimensions
│  │ + Residual Conn   │  │
│  ├───────────────────┤  │
│  │ Layer Norm        │  │
│  │ FeedForward (MLP) │  │  384 → 1536 → 384 (4x expansion)
│  │ + Residual Conn   │  │
│  └───────────────────┘  │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│  Final Layer Norm       │
│  Linear Head → Logits   │  384 → vocab_size (predicts next char)
└─────────────────────────┘

What each component does (in plain English):

• Token Embedding: Converts each character (like 'h', 'e', 'l') into a list of 384 numbers that the neural network can understand.
• Position Embedding: Tells the model where each character is in the sentence. Without this, the model wouldn't know if 'h' is the 1st character or the 100th.
• Multi-Head Self-Attention: This is the brain of the Transformer. Each "head" looks at the sentence from a different angle, figuring out which characters are related to each other. For example, in the word "the", the 'e' learns to pay attention to 't' and 'h' before it.
• FeedForward Network: After the attention heads gather information, this layer lets each character "think" about what it learned. It expands the 384 numbers to 1536 (to give it more room to think), and then compresses it back to 384.
• Residual Connections (x + ...): These are shortcuts that let information flow directly through the network without getting lost. Without them, deep networks (6 layers) would forget what they learned in the early layers.
• Layer Normalization: Keeps the numbers inside the network from becoming too big or too small, which helps the model train smoothly.
• Dropout (0.2): Randomly turns off 20% of the neurons during training. This forces the model to not rely on any single neuron and makes it generalize better (prevents overfitting).

---

🔤 Why Character-Level Tokenization Instead of BPE?

This is one of the most important design decisions in this project. Here's why character-level was chosen:

What's the difference?

Feature	Character-Level (This Project)	BPE (e.g., tiktoken / cl100k_base)
Vocabulary Size	~87 tokens	~100,277 tokens
What is a "token"?	A single character ('a', 'b', ' ')	A word or subword ('hello', ' the', 'ing')
Embedding Table Size	87 × 384 = 33,408 parameters	100,277 × 384 = 38,506,368 parameters
Memory for Embeddings	~130 KB	~147 MB

The Memory Problem with BPE

When we first tried to use BPE tokenization (using tiktoken's cl100k_base encoder with 100,277 tokens), the very first layer of the model tried to create a matrix of size 100,277 × 100,277. Here's the math:

100,277 × 100,277 = 10,055,476,729 numbers
10,055,476,729 × 4 bytes (Float32) = ~40 GB of RAM ❌

Caution

40 GB just for one layer! Most laptops have 8-16 GB of total RAM. The computer immediately crashes with: RuntimeError: DefaultCPUAllocator: not enough memory: you tried to allocate 40221906916 bytes.

Why Character-Level Works for Learning

Advantage	Explanation
Tiny vocabulary	Only ~87 unique characters → all layers stay small and fast
No external dependencies	No need for tiktoken or sentencepiece libraries
Easy to understand	You can literally see what each "token" is — it's just a letter!
Trains on any hardware	Runs on a laptop CPU in minutes, or a basic GPU in seconds
Great for learning	Focuses your attention on the Transformer architecture, not tokenization complexity

When to use BPE (for future reference)

BPE becomes essential when you scale up to production-level models. At that point, you would use an embedding_dim much smaller than vocab_size (e.g., nn.Embedding(100277, 384) instead of nn.Embedding(100277, 100277)), which brings memory down to a manageable ~147 MB. Models like GPT-2, GPT-3, and ChatGPT all use BPE.

---

🚀 How to Run

Prerequisites

pip install torch

Important

For GPU acceleration, install the CUDA-enabled version of PyTorch instead:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

You can verify GPU access with:

import torch
print(torch.cuda.is_available())      # Should print: True
print(torch.cuda.get_device_name(0))  # Should print your GPU name

Running the Model

python bigram.py

This will:

1. Load and tokenize book.txt (character-level)
2. Split the data into 90% training / 10% validation
3. Train the Transformer for 5,000 iterations
4. Print training and validation loss every 500 steps
5. Generate 500 characters of new text at the end

Expected Output

10.805847 M parameters
step 0 : train loss 4.5360, val loss 4.5367
step 500 : train loss 2.4375, val loss 2.4548
step 1000 : train loss 1.4382, val loss 1.4550
...
step 4500 : train loss 0.4839, val loss 0.4998

[Generated text will appear here]

---

☁️ Recommended: Run on Google Colab

Tip

This model trains much faster on a GPU. If you don't have an NVIDIA GPU, or if your laptop is running out of memory, we strongly recommend running this on Google Colab, which gives you a free GPU!

Steps to run on Colab:

1. Go to Google Colab
2. Click "New Notebook"
3. Go to Runtime → Change runtime type → Select "T4 GPU"
4. Upload book.txt to Colab (drag and drop into the file panel on the left)
5. Copy-paste the entire bigram.py code into a cell and run it

Why Colab is better for this project:

Feature	Your Laptop (MX450)	Google Colab (T4 GPU)
GPU Memory (VRAM)	2 GB	15 GB
GPU Speed	~3.5 TFLOPS	~8.1 TFLOPS
Cost	Free (your electricity)	Free (Google's electricity)
Training Time (5000 iters)	~5-10 minutes	~1-2 minutes
Can scale to bigger models?	Limited by 2 GB VRAM	Can handle 100M+ parameter models

With Colab's T4 GPU, you can experiment with larger models by increasing n_embd, n_head, n_layer, and block_size without worrying about running out of memory.

---

📁 Project Structure

LLM/
├── bigram.py      # The main Transformer model (this is the complete, runnable script)
├── book.txt       # Training data (~700K characters of text)
├── llm.ipynb      # Jupyter Notebook with step-by-step exploration and experiments
├── imp.txt        # Personal notes and reference snippets
└── README.md      # You are here!

---

📚 Key Concepts for Beginners

What is a Language Model?

A language model is an AI that predicts the next word (or character) in a sequence. If you give it "The cat sat on the", it tries to predict "mat". By doing this millions of times during training, it learns the patterns and structure of language.

What is a Transformer?

A Transformer is a type of neural network architecture introduced in the 2017 paper "Attention Is All You Need". Before Transformers, AI models processed text one word at a time (sequentially). Transformers process all words at once (in parallel) using a mechanism called Self-Attention, which makes them dramatically faster and more powerful.

What is Self-Attention?

Imagine you're reading the sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to? The animal or the street? You instantly know it's the animal. Self-Attention is the mechanism that lets the AI figure this out too — by letting every word "look at" every other word and decide which ones are relevant.

What are Parameters?

Parameters are the numbers inside the neural network that get adjusted during training. Think of them as the "brain cells" of the AI. More parameters = more capacity to learn complex patterns. For comparison:

• This model: 10.8 Million parameters
• GPT-2 (Small): 124 Million parameters
• GPT-3: 175 Billion parameters
• GPT-4: Estimated 1.7+ Trillion parameters

---

🔮 Future Roadmap & Production Engineering

Taking a language model from a prototype to a production-grade AI system involves several key engineering milestones:

┌────────────────────────┐      ┌────────────────────────┐      ┌────────────────────────┐
│  1. Evaluation Metrics │ ───► │     2. Fine-Tuning     │ ───► │ 3. Production Hosting  │
│   (Track Performance)  │      │  (Customize Behavior)  │      │  (Deploy as a Web App) │
└────────────────────────┘      └────────────────────────┘      └────────────────────────┘

1. 📊 Evaluation Metrics (Measuring Model Quality)

As a coder, how do you know if your model is actually getting smarter during training?

• Train vs. Validation Loss: The primary health check. If your training loss drops but validation loss starts rising, your model is overfitting (memorizing text instead of learning language rules).
• Perplexity ($PPL = e^{\text{Loss}}$): The standard benchmark in NLP. It measures how "surprised" the model is when reading new text. A lower score means better, more confident predictions.
• Sample Prompt Probing: Test the model at regular checkpoints with fixed prompts (e.g., "Once upon a time", "The king said") and log them (via TensorBoard or Weights & Biases) to visually evaluate improvements in grammar, coherence, and repetition.

2. 🎯 Fine-Tuning & Alignment (SFT & LoRA)

• Supervised Fine-Tuning (SFT / Instruction Tuning): Train the base language model on structured prompt-response pairs ({"instruction": "...", "response": "..."}) to turn a text-completion model into an interactive conversational assistant.
• LoRA (Low-Rank Adaptation) & QLoRA: Fine-tune only a fraction (<1%) of the parameters by injecting low-rank decomposition matrices, enabling efficient domain adaptation on consumer GPUs.
• Preference Alignment (DPO / RLHF): Use Direct Preference Optimization (DPO) to teach the model to prefer helpful, safe, and coherent responses over low-quality completions.

3. 🌐 Production Deployment & Hosting

• High-Performance Inference: Serve the model using specialized engines like vLLM, TGI (Text Generation Inference), or Ollama with continuous batching and PagedAttention.
• FastAPI Backend: Build lightweight REST/WebSocket APIs with real-time token streaming via Server-Sent Events (SSE).
• Interactive Web Demo: Create user-facing chat UIs with Gradio or Streamlit for quick demos and user testing.
• Containerization & Cloud Hosting: Package the application with Docker and deploy to platforms like Hugging Face Spaces, RunPod, or AWS/GCP.