Skip to content

quickstart.md

Latest commit

 

History

History
122 lines (86 loc) · 4.55 KB

quickstart.md

File metadata and controls

122 lines (86 loc) · 4.55 KB
Raw

Quickstart — zero to a running compressed model

You will install BigSmall, compress a model, let the autopilot decide how to run it, and run it. No prior knowledge assumed. Every command below was executed on a real machine and the output shown is what it printed (an 8-core Windows workstation; your sizes and times will differ, the shape of the output will not).

1. Install

pip install bigsmall

Check it works:

bigsmall --version
bigsmall 4.0.0

2. Profile your hardware (once, ~10 seconds)

bigsmall profile
profile: C:\Users\you\.bigsmall\profile.json
  cpu: 8 logical cores | ram: 30.04 GB total, 21.32 GB free at probe
  disk: 0.657 GB/s sequential (FILE_FLAG_NO_BUFFERING)
  gpu: NVIDIA RTX A4500 | 19.5 GB usable [UNVERIFIED (GPU busy)]
  kernels: bf16 decode 484.4 MB/s, int4 decode 853.7 MB/s (CPU, measured)

This measures your RAM, disk read speed, and how fast your CPU decodes BigSmall's formats. It is saved and never asked again (--force re-probes). Two things to know:

  • The GPU is read through NVML only — BigSmall never allocates GPU memory or interrupts whatever your GPU is doing. If the GPU is busy at probe time (like above), the GPU values are static estimates and are marked unverified.
  • Every later decision (plan, run) is computed from this file. The numbers in it are measurements, not guesses.

3. Compress your first model

Grab a small HuggingFace model:

python -c "from huggingface_hub import snapshot_download; snapshot_download('gpt2', local_dir='./gpt2_src')"

Compress it:

bigsmall compress ./gpt2_src/model.safetensors -o ./gpt2.bs
compressed gpt2_src\model.safetensors -> gpt2.bs
  source:     548,105,171 bytes
  compressed: 413,973,591 bytes (75.53%)
  saved:      134,131,580 bytes
  elapsed:    19.4s

GPT-2 is stored in F32, so it saves ~24%; BF16 models (most modern LLMs) save ~34%. Verify it:

bigsmall verify gpt2.bs --fast
OK

--fast checks the container structure in seconds. Plain bigsmall verify gpt2.bs decodes everything and checks every tensor's md5 against the original — it also prints OK, and it is the reason you can trust the file: if a single bit were different, it would fail.

4. Ask the autopilot what it would do

bigsmall plan gpt2.bs
Running perfect mode (bit-exact, receipt verified) in CPU RAM at full CPU speed while the GPU is busy.

That one sentence is the whole interface. How to read it:

  • "perfect mode (bit-exact, receipt verified)" — the fidelity it picked. perfect means the original weights, with the verification receipt found in the file. Files can also offer fast (lossy INT4) — and if the planner ever picks anything below the file's best fidelity, it must say so in a separate announcement line first. That rule is enforced by the test suite.
  • "in CPU RAM" — the placement it picked (your GPU, your RAM, or streaming from disk), computed from your profile and the file's actual sizes.
  • "at full CPU speed" — the honest speed statement. Where the answer is a measured estimate it shows a number ("~0.8 tok/s"); resident placements run at whatever your hardware runs at.
  • "while the GPU is busy" — it noticed the GPU was in use and routed around it. It will not touch a busy GPU.

--json dumps the full decision (every placement considered, fit math, alternatives) if you want to see the reasoning.

5. Run it

bigsmall run gpt2.bs
Running perfect mode (bit-exact, receipt verified) in CPU RAM at full CPU speed while the GPU is busy.
loaded 160 tensors into host RAM in 7.1s [mode=perfect] (bit-exact receipt honoured)

Same sentence, then the load. The model is now in host RAM, bit-identical to the original, ready for whatever loads state dicts.

6. Get the original bytes back (any time)

bigsmall decompress gpt2.bs -o gpt2_back.safetensors
decompressed gpt2.bs -> gpt2_back.safetensors  (13.8s)

Where to go next

  • How it helps — find your situation (gaming GPU, checkpoint hoard, shipping models, fp8 lab) and the one command for it.
  • The Ferrell Duo — one .bsd file, two models: the fast one (INT4) and the real one (the bit-exact original).
  • Streaming — run models bigger than your RAM, with a measured receipt.
  • Delta compression — store fine-tunes as patches (the biggest storage win).
  • CLI reference — every command, every flag, real output.