Build and train a mini Kimi K3 from scratch — a pure-PyTorch playground from a custom 88M model to 2T-scale LLM profiles.

Production-honest small language model training factory: data import, pretraining, SFT, eval gates, contamination checks, and GPU runbooks.

Annotated study fork of LLMs-from-scratch — GPT-2 architecture in PyTorch, tokenisation, pre-training loop, instruction fine-tuning, RLHF pipeline notes, and extended implementation commentary.

Parameter-efficient small LLM training experiments achieve <16MB/10 min training using recurrent/shared-block transformers, unigram/bigram logit calibration, and INT8-compressed language model architectures with MLX and PyTorch.

Examples showing multiple options how to train a model with Python