minigpt — a language model from scratch

One file. ~300 readable lines. Zero borrowed weights. A decoder-only Transformer — the same architecture family as GPT, Llama, and Claude — built from raw PyTorch so every moving part is visible and hackable.

text  →  tokens  →  embeddings  →  N transformer blocks  →  next-token probabilities

Train it by making those predictions match the real next token. That's the whole trick; everything else is engineering.

Architecture

flowchart LR
    A[Input text] --> B[Tokenizer]
    B --> C[Token + Positional<br/>embeddings]
    C --> D[Transformer block × N]
    D --> E[LayerNorm]
    E --> F[Linear → vocab logits]
    F --> G[Softmax → next token]
    subgraph Block
      H[Masked multi-head<br/>self-attention] --> I[Residual + LayerNorm]
      I --> J[Feed-forward MLP] --> K[Residual + LayerNorm]
    end

Loading

Each block is the classic pre-norm Transformer: causal (masked) self-attention so a token can only see what came before it, then a position-wise MLP, each wrapped in a residual connection. Stack n_layer of them and you have the engine behind every modern LLM.

Run it

pip install torch
python minigpt.py            # trains on data.txt, then samples
python minigpt.py --sample   # load checkpoint and just generate

Swap data.txt for any plain-text file — a corpus of your own writing, the King James Bible, Tennessee statutes — and it learns that style.

The four numbers that are the model

Everything you've read about "bigger models" is turning these knobs in Config:

Knob	What it controls
n_layer	how many Transformer blocks stack — depth
n_head	attention heads per block — how many relationships it tracks at once
n_embd	embedding width — how much each token "knows"
block_size	context length — how far back it can see

Scale them up and you walk the same road from a toy model to a frontier one. Nothing else about the architecture changes.

Read it like a guided tour

The single file is organized top-to-bottom as a tour, each section commented for what it does and why it's there:

1. Config — the knobs above
2. Tokenizer — text ↔ integers
3. Attention — the masked self-attention head
4. Block — attention + MLP + residuals
5. Model — the embedding → blocks → logits stack
6. Training loop — batching, loss, optimizer step
7. Sampling — autoregressive generation with temperature

Why it exists

Reading a real implementation beats reading ten explainers. Built as a study companion for the cumberland local-LLM project, where the same ideas scale up into a real tokenizer, pretraining run, and agent harness.

Authorship

Designed and directed by William Brewer (Nashville, TN). Built in collaboration with AI (Anthropic's Claude). MIT licensed.