A minimal, educational neural network and energy-based model library implemented from scratch in pure NumPy.
tinny was created to demystify deep learning internals: no opaque autograd magic, no heavy framework bloat — just explicit mathematical operations, vectorized tensor calculus, and property-tested analytical gradient backpropagation. See the notebooks/ directory for experiment implementations (including MLPs, CNNs, Autoencoders, GANs, and DBNs) along with solutions and commentary for course assignments.
- Pure NumPy Core: Transparent matrix implementations of forward and backward passes with zero black-box dependencies.
-
Energy-Based Models (EBMs): Complete implementations of Restricted Boltzmann Machines (RBMs) and Deep Belief Networks (DBNs), trained via Contrastive Divergence (
$CD_k$ ) and Persistent Contrastive Divergence (PCD). -
Extensible Tensor Layer API: Layers are defined as parametrized tensor functions
$y = f(x; \theta)$ supporting sequential composition and residual skip connections (Residual). -
Comprehensive Layer Suite: Dense (Linear), 2D Convolution (
Conv2Dwith vectorized im2col),MaxPool2D,Flatten,Dropout, andBatchNorm. - Rigorously Tested Gradients: Analytical gradients are validated against numerical finite differences via property-based testing with Hypothesis and pytest.
- Accompanying Theory Notes: Mathematical derivations and conceptual notes available at barhanc.github.io/notes.
Ensure you have uv installed.
git clone https://github.com/barhanc/tinny.git
cd tinny
# Synchronize the environment and install dependencies in seconds
uv syncfrom tinny import nn
from tinny.utils import softmax, chunks
# Define a Multi-Layer Perceptron
model = nn.Sequential(
nn.Linear(vsize=784, hsize=128),
nn.ReLU(),
nn.Dropout(p=0.2),
nn.Linear(vsize=128, hsize=10),
)
optimizer = nn.Adam(model.parameters(), lr=1e-3)
# Training step across mini-batches
for X_batch, y_batch in zip(chunks(X_train, 128), chunks(y_train, 128)):
# Forward pass
logits = model.forward(X_batch, training=True)
# Compute cross-entropy loss gradient
grad_y = (softmax(logits) - y_batch) / len(X_batch)
# Backward pass & parameter update
model.backward(grad_y)
optimizer.apply(model.gradients())