TinyGPT — A From-Scratch Transformer model implementation

This repository contains a minimal, fully self-contained GPT-style language model implemented from scratch in PyTorch. It includes a custom Byte Pair Encoding (BPE) tokenizer, a full Transformer architecture, μ-Parametrized training, and a complete text-generation pipeline.

The goal of this project is to provide a clear, understandable reference implementation of the core components behind modern transformer-based language models without relying on external frameworks such as HuggingFace.

---

Features

Custom BPE Tokenizer

• Completely implemented from scratch
• Pair statistics computed directly on GPU tensors
• Deterministic merging
• Fully reversible decode function
• Supports large merge counts efficiently

Transformer Architecture

• Multi-head self-attention
• Causal masking
• RMSNorm for stable training
• Feed-forward MLP blocks
• Residual connections with proper scaling
• Positional embeddings

Training System

• μ-Parametrized per-tensor learning rate scaling
• Warmup schedule
• AdamW optimizer
• Gradient clipping
• Batched training over context windows

Text Generation

• Temperature scaling
• Top-p nucleus sampling
• Full inspection of logits and probability distributions
• Stable fallback behavior if probability mass collapses

---

Getting Started

1. Prepare the Dataset

Place a text file (such as shakes.txt) in the repository directory. The included example uses Shakespeare, but any dataset may be used.

2. Train the Model

Run the training script:

python main.py

This performs BPE construction, initializes model weights, and begins training on the dataset.

3. Generate Text

In Python:

seed_2 = torch.tensor([[1]], dtype=torch.long, device="cuda")
generated_2 = generate(seed_2, max_new_tokens=50, temperature = .6)
tokens_1, text_1 = decode(generated_2[0].tolist(), merged_dict, itos)
print(text_1)

This will append new tokens to the seed sequence and return the full generated output.

---

Repository Structure

/TinyGPT
│
├── main.py        # Full implementation: BPE, model, training, generation
├── shakes.txt     # Example dataset used for training
└── README.md      # Project documentation

• a shorter README
• a more academic/technical version
• an “explainer” section walking through each module
• architecture diagrams Just let me know.