Fine-tunes LLaMA-2-13B with QLoRA to generate coherent, structured board game rulebooks (objectives, setup, turn structure, scoring, and win conditions) from a corpus of real rulebooks scraped, OCR'd, and normalized to Markdown.
Designing a board game means writing a rule set that is simultaneously complete (no undefined edge cases), consistent (mechanics don't contradict each other), and playable. It is slow, and it is the bottleneck for most first-time designers.
Generic LLMs produce rulebook-shaped text that falls apart under scrutiny. Win conditions can't be reached and resource loops never terminate. The hypothesis: domain fine-tuning on real rulebooks teaches the structural grammar of a rule set in a way prompting alone does not.
Rulebook PDFs (BoardGameGeek corpus)
│
├─► Text extraction (PyMuPDF) with OCR fallback (Tesseract via pdf2image)
│
├─► Markdown normalization ──► structured sections
│ (objective · components · setup · gameplay · scoring · winning)
│
├─► BGG XML API enrichment ──► game description as conditioning input
│ (throttle-aware client: backoff on HTTP 202/429, URL-encoded queries)
│
└─► Instruction-formatted JSONL, resumable with per-record fsync
│
┌─────────▼──────────┐
│ LLaMA-2-13B │ 4-bit NF4 QLoRA · double quantization
│ + LoRA adapters │ r=32 · α=64 · all 7 projection modules
│ (A100-40GB) │ ~125M trainable params (≈0.95% of 13B)
└─────────┬──────────┘
│
Structured Markdown generation with header-aware early stopping
│
W&B experiment tracking
- Corpus construction. board game rulebooks passed through a text-extraction → OCR-fallback → Markdown pipeline (
PyMuPDF,pdf2image,pytesseract), with headings promoted to a consistent H2 section schema. Ingestion is resumable: each record is flushed and fsynced as it is written, so an interrupted Colab session picks up where it left off. - Throttle-aware BGG client. the BoardGameGeek XML API signals rate limiting via HTTP 429 and queued responses via HTTP 202. The ingestion client retries with exponential backoff, waits politely between calls, and URL-encodes game names (
Dungeons & Dragonswould otherwise truncate the query at the&). - Parameter-efficient fine-tuning. LoRA adapters (
peft) on all seven Llama-2 projection modules, with 4-bit NF4 double-quantized weights (bitsandbytes) and a paged 8-bit AdamW optimizer, so a 13B model trains on a single A100-40GB. - Loss masking that matches the task. a custom collator prepends BOS and appends EOS as token IDs, masks the entire prompt with
-100, and builds the attention mask from true sequence lengths, so loss is computed only on the rulebook the model must generate. - Structured generation. A few-shot exemplar primes the Markdown structure. A stopping criterion halts generation when a second top-level header appears (a new game starting), and a bad-words filter suppresses boilerplate sections.
- Leakage-safe evaluation harness. A seeded 90/10 split written to disk at training time. The evaluation cell refuses to score anything except the held-out file, decodes greedily for determinism, and reports coverage-aware metrics (below).
651 instruction-formatted examples, each pairing a game's BGG description (input) with its normalized Markdown rulebook (output). Truncation: 512 source tokens, 2,048 target tokens.
An engineering note worth knowing: the raw corpus was roughly 1,000 rulebooks. The gap to 651 traced to silent API throttling: BGG's queued/rate-limited responses were coming back empty and those games were dropped as "no description." The current throttle-aware client exists because of that diagnosis, and re-running ingestion is expected to recover most of the difference.
Rulebooks are copyrighted works. The corpus was assembled for academic research and is not redistributed in this repository.
| Base model | meta-llama/Llama-2-13b-hf (gated; requires HF access) |
| Quantization | 4-bit NF4, double quantization, bfloat16 compute |
| Adapters | LoRA r=32, α=64, dropout 0.05, on q/k/v/o/gate/up/down projections |
| Trainable params | ~125M (≈0.95% of 13B) |
| Optimizer | paged 8-bit AdamW, lr 1e-4, cosine schedule, 5% warmup |
| Batch | 1 physical × 4 gradient accumulation |
| Epochs | 4, with validation loss every epoch |
| Model selection | best epoch by validation loss (not the last epoch) |
| Hardware | single NVIDIA A100-SXM4-40GB, gradient checkpointing enabled |
Scoring runs only on held-out games the model never saw, generated greedily for determinism. Because rulebooks are sectioned documents, a single overlap score can mislead, a model that writes two of eight sections well would look strong. Four numbers are therefore reported together:
| Metric | What it answers |
|---|---|
rougeL_common |
How good are the sections the model did write? (flattering alone) |
coverage |
What fraction of the reference's sections did it write at all? |
rougeL_adjusted |
rougeL_common × coverage, quality that pays for missing sections |
rougeL_fulldoc |
Whole-document ROUGE-L, immune to sectioning entirely |
BLEU and BERTScore are logged per section alongside. Held-out numbers from the current training run will be added here, together with a sample generation; qualitatively, generated rule sets produce coherent objectives, setup instructions, scoring systems, and reachable win conditions.
PyTorch · Hugging Face Transformers · PEFT / LoRA · bitsandbytes (4-bit NF4) · Accelerate · Tesseract OCR · PyMuPDF · pdf2image · Weights & Biases · CUDA
git clone https://github.com/Drew-Patrick/gameboard-genesis.git
cd gameboard-genesis
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtLLaMA-2 is gated on Hugging Face. Request access, then:
export HF_TOKEN=... # your Hugging Face token
export WANDB_API_KEY=... # optional, for tracking
jupyter notebook notebooks/gameboard_genesis_lora.ipynbThe notebook was developed in Colab and expects a corpus directory of rulebook PDFs, so point RULEBOOK_DIR at the corpus location. Run order matters: the training cell writes the held-out split that the evaluation cell requires, and evaluation reuses the model loaded by the inference cell. No credentials are hardcoded. Tokens are read from the environment, and .gitignore blocks token files.
├── notebooks/
│ └── gameboard_genesis_lora.ipynb # ingestion → dataset → QLoRA fine-tune → generation → eval
├── docs/
│ └── gb_genesis_report.pdf # full write-up
└── requirements.txt
- ROUGE is a weak proxy for rulebook quality. It rewards n-gram overlap, not whether a game is playable. A rule set can score well and still contain an unreachable win condition. Next: a playability rubric, human or LLM judged checks for reachable win conditions, terminating resource loops, and non-contradictory rules, reported alongside overlap metrics.
- No prompted baseline yet. Comparing against the un-tuned base model with the same few-shot prompt would isolate how much the LoRA adaptation actually contributes.
- Corpus recovery. Re-running ingestion with the throttle-aware client should recover most of the games lost to API rate limiting, growing the training set.
Drew Patrick, M.S. Artificial Intelligence, Kennesaw State University.
MIT (code only), see LICENSE. Training data is not included or licensed here.