Skip to content

About

Own the full LLM lifecycle on an Apple M1 (8GB), no rented GPU: train a transformer from scratch, MLX-QLoRA + hand-rolled DPO on a <=1.5B model, quantize, serve behind a from-scratch micro-batcher, govern with eval + safety. Capstone 2/3.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

On-Device Model Lifecycle (train → align → serve → govern on an M1)

Own the entire modern LLM lifecycle on the hardware you have. Two tracks: (T1) train a small (10–50M-param) transformer from scratch, and (T2) MLX-QLoRA fine-tune + DPO-align a ≤1.5B open model — then quantize, serve with a from-scratch micro-batcher, and govern with an eval + safety layer. One reproducible pipeline on an Apple M1 (8 GB). No rented GPU required.

Status: 🔨 M0–M3 of 8 done (2026-07-27). Domain committed (clinical/medical Q&A), held-out benchmark frozen and stratified (300 MedMCQA items, sha 41716514e227…), toolchain verified on the M1 (make check 4/4), and a clean, decontaminated 145,838-row corpus whose cleanliness is shown by a positive control, not asserted: a leakage probe scores 99.3% on a deliberately contaminated corpus and 0% on the shipped one. M2 trains a 10.79M-param char-level transformer from scratch in MLX — held-out perplexity 16.9 → 4.24, coherent medical MCQ text from random init, and a headline finding that overturns the M1 prediction: the over-parameterized model did not memorize (gap −0.07, 0.96 novelty) because a sub-epoch budget plus dropout regularizes it. M3 QLoRA-SFTs a Qwen2.5-0.5B-4bit base and reports an honest negative: SFT does not beat the base (letter-only 31.0% vs base 31.3%; the base already sits at the 31.7% majority floor via a degenerate always-A policy that SFT fixes without adding accuracy), and reasoning-completion SFT regresses to 7.0% by teaching the small model verbose, non-committal generation (72% never emit an answer). The 0.5B is capacity-bound; beating it needs a 1.5B+ base, which 8 GB can't train. The SFT pipeline (letter-logprob + CoT-generation evaluator, both completion styles, fuse) is complete and correct. 82 tests pass. Build from docs/02-build-plan.md. Next: M4 (hand-rolled DPO).

Capstone #2 of 3. Siblings: self-improving-agent-platform, realtime-decision-intelligence. Effort: 6–12 months solo.


Architecture

architecture

Interactive/exportable version: docs/assets/architecture.html.

The one-sentence pitch

Most engineers only ever call models. This project proves you understand them from the inside — data, training dynamics, alignment, quantization, inference optimization, and safety — by carrying models through the full lifecycle on a laptop, without pretending to run infrastructure you don't have.

Why two tracks (and why on-device)

An 8 GB M1 physically cannot do billion-scale continued-pretraining or multi-GPU FSDP. So instead of faking that on cloud rentals, we own two lifecycles completely on hardware we have:

  • T1 — from-scratch tiny LM (10–50M params). The purest "I built the model, not just the app" artifact. Trains on the M1 GPU in hours. The mid-2026 fine-tuning caveat (~80% of jobs don't ask for it) does not apply — this is understanding, not a hire-checkbox.
  • T2 — adapt an open model (≤1.5B). The practical skill: MLX-QLoRA SFT + DPO alignment at a size the M1 actually runs. Proves you can bend a real model to a domain.

Extends the existing llm-quantization-bench (already proven on this exact M1) into the full stack above it.

Headline deliverables (what to demo)

  1. A from-scratch tiny LM that goes from noise → coherent domain text (loss curve + generation samples).
  2. A QLoRA-adapted 1.5B model that beats its base on a domain benchmark.
  3. A full quantization Pareto chart (size × speed × accuracy) for both.
  4. A served endpoint hitting stated p50/p99 latency SLOs via a from-scratch micro-batcher.
  5. An auto-generated model card + safety report (red-team before/after).
  6. One command replays raw-data → governed endpoint on the M1.

What runs where (Apple M1, 8 GB)

Task Tool Notes
T1 from-scratch training nanoGPT-in-MLX 10–50M params, fits in memory, trains in hours
T2 SFT + DPO mlx-lm (QLoRA, 4-bit) ≤1.5B base, --batch-size 1
Quantize llama.cpp (GGUF Q4/Q8, Metal) reuse quantization-bench harness
Serve from-scratch asyncio micro-batcher + FastAPI vLLM won't fit — hand-roll it
Optional large run (7B QLoRA / distributed) rented A100 (Lambda/RunPod/Colab) off critical path, documented only

PyTorch / DeepSpeed / vLLM / FSDP appear only in the optional cloud-burst script, never on the laptop.

Architecture (one glance)

 raw domain corpus
      │ dedup · quality-filter · decontaminate · synth-gen
      ▼
 ┌─────────────┐        ┌──────────────────────────┐
 │ T1: from    │        │ T2: adapt ≤1.5B open base │
 │ scratch     │        │  QLoRA SFT → DPO align    │
 │ 10–50M MLX  │        │  (measure alignment tax)  │
 └──────┬──────┘        └────────────┬─────────────┘
        └──────────┬─────────────────┘
                   ▼
         quantize (GGUF Q4/Q8)  → Pareto: size × speed × accuracy
                   ▼
         serve (from-scratch micro-batcher + KV-cache, p50/p99 SLO)
                   ▼
         govern: eval @ every stage · model card · red-team · drift

Detail: docs/01-architecture.md.

Repository layout (target)

ondevice-model-lifecycle/
├── README.md · .gitignore · requirements.txt
├── docs/  00-what-it-is · 01-architecture · 02-build-plan · 03-setup
├── data_engine/     # collect, dedup, filter, decontaminate, synth
├── t1_scratch/      # nanoGPT-in-MLX tiny LM
├── t2_adapt/        # QLoRA SFT + DPO
├── quantize/        # GGUF conversion + Pareto bench
├── serve/           # micro-batcher + FastAPI endpoint
├── govern/          # eval suite, model card gen, red-team
└── models/          # gitignored artifacts

Quickstart

source ~/miniconda3/etc/profile.d/conda.sh && conda activate personal
make install     # pinned deps: mlx, mlx-lm, llama-cpp-python (Metal), ...
make check       # verify MLX/Metal, mlx-lm, llama.cpp and the frozen benchmark
make test        # fast offline suite

Milestone targets (m1-data, m2-t1, m3-sft, …) land as each milestone does; see the stub list at the bottom of the Makefile. make help lists what currently works.

Tech stack

Python 3.12 · Apple MLX / mlx-lm · nanoGPT-in-MLX · HF datasets · llama.cpp (GGUF, Metal) · from-scratch asyncio micro-batcher · FastAPI streaming · MLflow/W&B · DVC · matplotlib. Cloud (optional burst only): PyTorch · DeepSpeed/FSDP · vLLM · AWQ/GPTQ.

License

Private. All rights reserved.

About

Own the full LLM lifecycle on an Apple M1 (8GB), no rented GPU: train a transformer from scratch, MLX-QLoRA + hand-rolled DPO on a <=1.5B model, quantize, serve behind a from-scratch micro-batcher, govern with eval + safety. Capstone 2/3.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages