micrograd-lab

A small learning lab for scalar automatic differentiation, neural-network training, and an offline personal texting-style utility. Maintained by Ramanjeet Singh.

This is an extended fork of Andrej Karpathy's micrograd. The original engine, neural-network foundation, notebooks, and illustrations are credited to their upstream authors. The original MIT license and Git history are retained.

What this fork adds

• Differentiable tanh, sigmoid, exp, log, and numerically stable softplus.
• Configurable hidden activations and input-dimension checks for small neural networks.
• SGD with momentum, Adam, optional L2 weight decay, and mean MSE / binary cross-entropy losses.
• A reproducible XOR training example that runs without third-party runtime dependencies.
• A personal texting-style rewriter with editable JSON profiles and style inference from sample messages.
• A local WhatsApp export-to-training-data builder, with synthetic examples to try the workflow.
• Iterative backpropagation for deep graphs, regression tests, and GitHub Actions checks.

The library still works on individual scalar values. It is intended for learning and small experiments, not large-model training. The texting utility is a separate, rule-based tool that adjusts surface style. Its rules are deliberately limited; review the result for grammar and meaning. It is not a language model trained with this engine.

Install your copy

Use Python 3.10 or newer:

git clone https://github.com/raman2221/micrograd-lab.git
cd micrograd-lab
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

On Windows, activate with .venv\Scripts\activate instead. The distribution is named micrograd-lab; imports remain micrograd for compatibility with the original examples. Use a dedicated environment rather than installing both distributions together.

Train a tiny neural network

python -m examples.train_xor
python -m examples.train_xor --optimizer sgd --lr 0.3 --epochs 1500

The default uses a fixed seed, eight tanh hidden units, Adam, and binary cross entropy. It prints training loss and predictions for the four XOR inputs. Experiment with --activation, --lr, --epochs, or --seed.

The training API is deliberately small:

from micrograd.nn import MLP
from micrograd.optim import Adam
from micrograd.losses import binary_cross_entropy_with_logits

model = MLP(2, [8, 1], activation="tanh")
optimizer = Adam(model.parameters(), lr=0.03)
inputs = [(0.0, 0.0), (0.0, 1.0), (1.0, 0.0), (1.0, 1.0)]
targets = [0.0, 1.0, 1.0, 0.0]

for _ in range(500):
    loss = binary_cross_entropy_with_logits([model(x) for x in inputs], targets)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Pass raw scores to binary cross entropy; use model(x).sigmoid().data to obtain a probability afterward. Both loss functions average over nonempty, equally sized iterables. mse_loss is available for regression. Adam uses classic coupled L2 weight decay; it does not implement AdamW.

Rewrite a message in your texting style

python -m micrograd.texting "I am on my way. Are you coming?"

The starter style uses lowercase text and natural contractions. It does not need an API key or send messages anywhere. HTTP(S)/www URLs, email addresses, and code spans are protected from rewriting. Review the returned text before using it.

To learn your preferences, put several representative messages into private/messages.txt, one per line, then run:

mkdir -p private
# Create private/messages.txt with your sample messages first.
python -m micrograd.texting --learn-from private/messages.txt --save-profile private/style.json
python -m micrograd.texting "Are you bringing your laptop?" --profile private/style.json

The learner looks for casing, ending punctuation, contractions, and supported shortcuts such as u and ur. It saves style settings rather than copying your messages into the profile. The private/ directory is ignored by Git. An editable starter profile is in examples/texting_profile.json.

You can also call it from Python:

from micrograd.texting import learn_style, rewrite_text

style = learn_style(["are u coming?", "bring ur laptop", "i'm already here"])
print(rewrite_text("I am bringing your laptop.", style))

This is a lightweight formatter, so it cannot infer your full personality, invent new phrasing, or reproduce a personal style accurately from a single short sample. Add more examples or edit the profile to tune it.

Prepare training data from WhatsApp

Try the pipeline with entirely invented sample messages:

python -m micrograd.chat_data examples/synthetic_chat.txt --self-name "Taylor Example" --output-dir private/dataset

For your own data, export a direct chat without media, save it under private/, and pass your exact sender name with --self-name. The tool pairs incoming messages with your replies, removes duplicate examples, redacts common contact details, and writes chronological training and validation JSONL files. Review both files before training: simple redaction is not complete anonymization. Only the synthetic sample is included in this repository.

See the dataset guide for supported formats and limitations. This feature prepares data; it does not train a conversational model, connect to WhatsApp, or send replies. A practical next step is to fine-tune an existing language model on reviewed examples and evaluate drafts against held-out conversations. Synthetic examples test the pipeline but do not teach your personal voice.

Explore automatic differentiation

from micrograd.engine import Value

x = Value(0.5)
y = (x.tanh() + x.sigmoid()).log()
y.backward()
print(y.data, x.grad)

Calling backward() again accumulates gradients on leaf values and recomputes intermediate gradients. Call model.zero_grad() or optimizer.zero_grad() before each optimizer update. Rebuild the forward graph after changing parameter values.

The original demo.ipynb and trace_graph.ipynb remain available for the upstream classification and Graphviz walkthroughs; their optional notebook dependencies are separate from the dependency-free core.

Development

python -m pip install -e '.[dev]'
python -m pytest -q

Development dependencies include PyTorch to check the original autograd examples. Additional tests cover finite-difference gradients, deep graphs, optimizer updates, XOR learning, profile inference, and the texting CLI. CI runs on Python 3.10 and 3.12.

Places to extend the project next:

• Add checkpoint saving for model parameters and optimizer state.
• Add small datasets and compare optimizer learning curves.
• Extend the texting profile with explicit, opt-in phrase preferences.

Attribution and license

Original project: karpathy/micrograd, copyright Andrej Karpathy. Fork baseline: 7bc720e951fe422b8f8814aa5aa1b64121d26b4c. The features listed above are additions in this fork. See LICENSE for the retained MIT license and copyright notice.