Tiny Coder

A GPT-style Python code language model built from scratch in PyTorch, covering tokenization, dataset preparation, transformer implementation, training, checkpointing, and autoregressive generation.

The project has progressed from small local experiments to a 36.4M-parameter Colab training run. It is an experimental language-modeling project focused on understanding the full training pipeline and improving its efficiency.

What is currently implemented

• A decoder-only transformer with causal multi-head attention, learned positional embeddings, pre-layer normalization, residual connections, and GPT-style initialization.
• Byte-level BPE tokenization using Hugging Face tokenizers, plus a separate from-scratch BPE implementation for studying the algorithm.
• Fixed-length next-token prediction windows, cached token streams, training/validation loops, and checkpoint serialization.
• A GPU training variant with BF16 mixed precision, optimized scaled dot-product attention, compiled execution, AdamW, and a cosine learning-rate schedule.
• Autoregressive generation with temperature and top-k sampling.

Stack: Python, PyTorch, Hugging Face Tokenizers/Datasets/Hub, NumPy, Matplotlib, Google Colab.

Model and data

Training uses the Python subset of CodeSearchNet. Documents are separated with <|endoftext|> and converted into shifted input/target windows.

Setting	Colab configuration
Transformer blocks	8
Attention heads	8
Token embedding dimension	512
Attention projection dimension	1,024
Feed-forward hidden dimension	2,048
Context length	1,536 tokens
Tokenizer vocabulary	2,000 BPE tokens + 1 end-of-text token
Trainable parameters, recorded output	36,407,296
Training tokens per pass, recorded output	Approximately 163.7M
Optimizer	AdamW
Initial learning rate	0.001

The reusable implementation in model.py uses manually implemented attention and ReLU feed-forward layers. The Colab notebook defines an accelerated variant with optimized attention and GELU. These are distinct experimental versions rather than interchangeable checkpoint-loading targets.

Tokenizer vocabulary size was selected through a compression sweep using held-out samples. Because the original sweep uses the dataset's test split, those samples should not be described as an untouched final evaluation set.

Recorded training results

The saved output in 05-colab-training.ipynb contains the following next-token cross-entropy losses:

Recorded epoch	Training loss	Validation loss
1	4.4292	3.7744
2	2.8864	2.0792
3	1.7565	1.7003
4	1.5816	1.6495

These are recorded notebook outputs. The current configuration cell specifies three epochs while the saved output contains four; a clean rerun is needed to reconcile the configuration and run history.

The loss reduction demonstrates learning on the code corpus, but it does not establish functional code-generation quality. The saved generation example still shows repetition and invalid indentation; executable-code benchmarks have not yet been reported.

Training efficiency experiments

06-training-accel.ipynb explores training overhead with a smaller 5.35M-parameter model. Its timing notes record:

Variant described in the notebook notes	Recorded time for 10 epochs
FP32, manual attention	6.52 minutes
BF16 and optimized attention	2.51 minutes
BF16, optimized attention, and compiled execution	1.92 minutes

The notes identify an A100 High-RAM Colab environment. They are exploratory timings rather than a controlled benchmark suite: historical variants are not preserved as separate runnable configurations, and compilation/warmup costs are not separately reported. This speed experiment trains on the validation token stream and uses the test stream for its evaluation loop, so its losses are not a clean held-out language-model result.

Repository guide

File	Purpose
01-tokenizer.ipynb	Train and compare tokenizers
02-dataset.ipynb	Encode and cache the dataset splits
03-model.ipynb	Explore the architecture and training budget
04-train.ipynb	Smaller local training experiments
05-colab-training.ipynb	Larger GPU run, generation, and checkpointing
06-training-accel.ipynb	Training efficiency experiments
model.py, train.py, dataset.py	Reusable model, training, and dataset components
hf_bpe_tokenizer.py	Hugging Face BPE training helper
bpe_tokenizer.py	From-scratch reference BPE implementation
tokenizer.json	Saved tokenizer

Running the project

Clone the repository and install the declared environment with uv:

git clone https://github.com/kuzeyaldemir0/tiny-coder.git
cd tiny-coder
uv sync
uv run --with jupyter jupyter lab

Start with the numbered notebooks in order. Use 04-train.ipynb for smaller local experiments; 05-colab-training.ipynb contains CUDA-specific BF16 training and Google Drive checkpoint paths that need an appropriate Colab GPU runtime.

Data access: the training notebooks download tokenizer.json, train_indices.pt, val_indices.pt, and test_indices.pt from the private Hugging Face dataset repository kuzey-aldemir/tiny-coder. They require authorized access, or you must regenerate the token caches through the preparation notebooks and update the loading cells to use your local files. The public repository includes the tokenizer but not the token caches or trained model weights.

For generation, use the predict() function defined in the Colab notebook after training or restoring a compatible model. It returns token IDs, which are decoded with the tokenizer. Checkpoint loading must match the architecture variant and account for compiled-model state-dictionary keys.

Next steps

• Reconcile saved outputs with explicit, versioned experiment configurations and consolidate the notebook/model variants.
• Add a reproducible small public-data run and publish compatible model weights.
• Evaluate code generation with syntax checks and executable tests, alongside validation loss.
• Run controlled architecture experiments with rotary positional embeddings, grouped-query attention, and weight tying.
• Preserve an evaluation split that is excluded from tokenizer selection and subsequent experimentation.