Deep Learning Notes

A personal collection of Jupyter notebooks working through deep learning from first principles: Python/NumPy basics, perceptrons, linear and logistic regression, multilayer networks, CNN architectures (GoogLeNet, ResNet, DenseNet), RNNs, transformers (GPT-2 and the original encoder–decoder), and classical SVM/kernel methods for contrast.

Most notebooks are built around PyTorch and follow the same five-part skeleton — prepare dataset → design model → construct loss and optimizer → training cycle → evaluate — so the same shape recurs as the models get more complex. Two notebooks (ch2, ch3) follow the Dive into Deep Learning book.

The two transformer notebooks break that pattern deliberately: they are implementation-detail notes rather than theory, focused on tensor shapes and PyTorch API pitfalls, and each one is structured as TODO exercise cells followed by reference-answer cells. Run them straight through and the answer cells overwrite the TODOs, so they work end to end either way. Both also carry their own correctness proof — see below.

Suggested reading order

#	Notebook	What it covers
1	Basic_python_numpy_matplotlib.ipynb	Python essentials (dicts, functions, classes), NumPy arrays and broadcasting, Matplotlib basics
2	Perceptron.ipynb	The perceptron as a threshold unit; AND / NAND / OR built by hand, and why XOR needs more than one layer
3	Neuro_network.ipynb	Activation functions from scratch in NumPy: step, sigmoid, ReLU
4	linear regression.ipynb	requires_grad and autograd by hand, then the same model rebuilt with nn.Module, MSELoss and optim.SGD
5	logistic regression.ipynb	Binary classification, BCELoss vs BCEWithLogitsLoss, and BCE recomputed by hand to check it
6	multi_layer.ipynb	Stacked Linear layers on the diabetes dataset; Sigmoid vs ReLU as the hidden activation
7	Dataloader_and_Dataset.ipynb	Writing a custom Dataset (__getitem__ / __len__), batching with DataLoader, train/test split
8	batch&stochastic&mini_batch.ipynb	Batch vs stochastic vs mini-batch gradient descent: convergence speed, noise signature, and the path taken across the loss surface
9	softmax_classifier.ipynb	MNIST with a 5-layer fully connected net; why the last layer has no activation, and how CrossEntropyLoss relates to LogSoftmax + NLLLoss
10	CNN.ipynb	Conv2d, padding, stride, pooling — then a full MNIST CNN, plus GoogLeNet (Inception), ResNet (residual blocks) and DenseNet (dense concatenation + transition layers)
11	RNN.ipynb	RNNCell with a manual loop, nn.RNN over a whole sequence, and Embedding + stacked nn.RNN + Linear, with the three compared side by side
12	transformer_from_scratch.ipynb	GPT-2 small rebuilt from scratch and verified against HuggingFace's real weights element by element: embeddings, LayerNorm, causal self-attention, MLP, pre-LN blocks, temperature sampling, plus a KV-cache appendix
13	transformer_original_paper.ipynb	The encoder–decoder Transformer of Attention Is All You Need: sinusoidal positional encoding, one multi-head attention reused three ways, padding + causal masks, post-LN, cross-attention, and a toy sequence-reversal task trained end to end
14	SVM_and_Kernel_Methods.ipynb	Maximum-margin hyperplanes, soft margins and C, the kernel trick geometrically, and the effect of gamma in the RBF kernel (scikit-learn)

Reference notebooks, read alongside the book rather than in sequence:

Notebook	What it covers
ch2.ipynb	D2L ch. 2 — tensor manipulation and memory, pandas preprocessing, linear algebra (reductions, dot/matrix products, norms), automatic differentiation
ch3.ipynb	D2L ch. 3 — the object-oriented training API built from scratch: HyperParameters, ProgressBoard, Module, DataModule, Trainer, then linear regression end to end on synthetic data

Extra topics worth noting

A few things covered inside the notebooks that are easy to miss from the titles:

• Saving the best model (CNN.ipynb) — tracking the lowest loss seen so far and calling torch.save(model.state_dict(), ...) whenever a new minimum appears, since the final epoch is rarely the best one. The resulting best_model.pth is checked in.
• Dummy-input shape inference (CNN.ipynb) — feeding an all-zero tensor through just the convolutional stack to find the flattened size for nn.Linear, instead of computing it by hand.
• ResNet vs DenseNet (CNN.ipynb) — a side-by-side table: addition vs concatenation, constant vs growing channel counts, and why DenseNet needs transition layers.
• Conv1D vs nn.Linear (transformer_from_scratch.ipynb) — HF GPT-2 stores those weights as [in, out] while nn.Linear wants [out, in], so loading them requires a transpose. The single most common reason a from-scratch GPT-2 fails to reproduce the official logits.
• KV-cache (transformer_from_scratch.ipynb, appendix A) — caching past K/V to take generation from O(N²) to O(N), including how to slice the causal mask once a cache exists, a token-for-token equivalence check against naive generation, and a measured speedup.
• pre-LN vs post-LN (transformer_original_paper.ipynb) — GPT-2's x + sublayer(norm(x)) against the paper's norm(x + sublayer(x)), and why the latter is more sensitive to the learning rate and leans on warmup.
• Two ways to prove an implementation is right — the GPT-2 notebook loads real HuggingFace weights and asserts the logits match element-wise; the paper notebook has no such reference weights, so it instead cross-checks its multi-head attention against nn.MultiheadAttention and trains a toy sequence-reversal task to 100% accuracy.

Setup

pip install torch torchvision numpy pandas matplotlib scikit-learn transformers einops jupyter

Verified with Python 3.13, PyTorch 2.11 (CUDA 12.8), NumPy 1.26, Matplotlib 3.10, scikit-learn 1.7, pandas 2.3, transformers 5.13 and einops 0.8. The transformer notebooks are the only ones needing transformers (for the GPT-2 tokenizer and reference weights) and einops (used in one aside comparing it to view + transpose). The notebooks that use a GPU select it with torch.device("cuda" if torch.cuda.is_available() else "cpu"), so CPU-only machines work too — just more slowly.

jupyter lab

Data

• data/house_tiny.csv — the tiny table used for the pandas preprocessing section of ch2.ipynb.

• data/diabetes.csv.gz — the Pima diabetes dataset (8 features, binary label) used by multi_layer.ipynb and Dataloader_and_Dataset.ipynb. Those notebooks read the uncompressed data/diabetes.csv, so decompress it first:

  gzip -dk data/diabetes.csv.gz

• MNIST is downloaded automatically by torchvision on first run, into ../dataset/mnist/ — i.e. a sibling of this repository, not inside it.

• GPT-2 small (~124M parameters) is fetched by transformers on first run of transformer_from_scratch.ipynb and cached in the usual HuggingFace cache (~/.cache/huggingface), not in this repository. transformer_original_paper.ipynb needs no download — it trains its toy task from scratch in a few hundred steps.

Note that the diabetes paths are written Windows-style ('data\diabetes.csv') and one cell in multi_layer.ipynb reads a bare 'diabetes.csv'; adjust the path if you run these on Linux or macOS.