Hands-on exercises for learning the core ideas of model optimization for edge deployment.
| # | Exercise | One line | Core to focus on |
|---|---|---|---|
| 1 | Quantization | Use fewer bits to represent numbers | quantization scheme & formula, compression ratio |
| 2 | Knowledge Distillation | A big model teaches a small one | distillation loss formula, teacher-student architecture |
| 3 | Pruning | Remove unused signals | granularity, ratio, criteria |
| 4 | Neural Architecture Search | Search for an optimal architecture | search space & strategy |
| 5 | Kernel Optimization | Make the same math run faster on the GPU | tiling, grid layout, memory access |
Kernel optimization is more hardware-constrained and usually speeds up the computation without changing the math. Meanwhile, the remaining ideas are more abstract and usually change what the model computes. They are orthogonal in concept and may be coupled in realization. For example, a quantized matmul kernel is entirely possible.
- Open the notebook for an exercise (links above).
- Click "Open in Colab" (badge at the top of each notebook), or upload the
.ipynbto Colab manually. - Set the runtime to GPU (
Runtime -> Change runtime type -> GPU). Runtime -> Run all.
To run locally instead:
pip install -r requirements.txt
jupyter labThe core concept is highlighted in every notebook. Everything else - data loading, training loops, plotting - can be skimmed. Every exercise records model size, latency and accuracy before vs. after the optimization so the trade-off can be considered.
Deploying optimized models on a laptop:
- apps/tinyml_chat/ - a from-scratch LLM chat app, following TinyChatEngine and the tinychat-tutorial.
- apps/vla.cpp/ - run SmolVLA with
vla.cpp, a C++ VLA inference runtime built onllama.cpp.
