Fine-tuning Qwen2.5-Coder-3B with LoRA to generate SVG markup from natural-language prompts. Built for the NYU DL Spring 2026 Kaggle competition.
- Base model:
unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit(4-bit quantized, ~2 GB VRAM) - Fine-tuning: QLoRA (r=32, α=32) via
trl'sSFTTraineron 30k competition training samples - Prompt format: ChatML with a system prompt that enforces valid, well-formed SVG output
- Post-processing: SVG extraction, canvas normalization (256×256), tag sanitization, fallback generation for failures
- Inference: Batch generation with temperature=0.3, top_p=0.95, repetition_penalty=1.1
The trained LoRA adapter is publicly available on HuggingFace. The easiest way to run inference is directly on Kaggle using the provided notebook.
- Open
notebooks/kaggle-inference.ipynbon Kaggle (enable GPU T4 x2) - Add the following datasets to the notebook:
- Competition dataset:
dl-spring-2026-svg-generation-from-text-prompts-extended-deadline - Trained adapter: khamidovsarvar/qwen-svg-lora-v3
- Competition dataset:
- The notebook will:
- Install dependencies
- Load the base model from HuggingFace (
unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit) - Load the adapter from
/kaggle/input/qwen-svg-lora-v3 - Read
test.csvfrom the competition dataset - Generate SVGs and save
submission.csv
No local setup needed.
Open notebooks/colab-training.ipynb in Google Colab (A100 or L4 GPU recommended).
The notebook covers:
- Mounting Google Drive
- Downloading the competition dataset via Kaggle API
- Fine-tuning with QLoRA
- Saving the adapter to Drive or pushing to HuggingFace Hub
For clusters with Singularity and SLURM.
1. Configure your environment
Copy the example config and fill in your cluster-specific paths:
cp src/cluster.env.example src/cluster.env
# then edit src/cluster.env2. Download base model and data
# Requires ~/.huggingface/token and ~/.kaggle/kaggle.json
bash src/download_assets.sh3. Submit training job
sbatch src/run_training.sbatch4. Submit inference job
sbatch src/run_inference.sbatch # batch_size=8
sbatch src/run_inference_fast.sbatch # batch_size=16, fasterLogs go to logs/, submission CSVs go to outputs/.
Edit the #SBATCH directives in the .sbatch files to match your cluster's account name, partition, and GPU constraints.
| Variable | Description | Example |
|---|---|---|
SINGULARITY_OVERLAY |
Path to your Singularity overlay image | /scratch/you/pytorch.ext3:ro |
SINGULARITY_SIF |
Path to the CUDA/cuDNN SIF image | /share/apps/images/cuda12.6.3-...sif |
SINGULARITY_PYTHON |
Python binary inside the overlay | /ext3/miniforge3/bin/python |
SINGULARITY_ENV |
Env activation script inside overlay | /ext3/env.sh |
.
├── notebooks/
│ ├── kaggle-inference.ipynb # Kaggle inference (primary submission path)
│ ├── colab-training.ipynb # Google Colab fine-tuning
│ ├── midterm-kaggle.ipynb # End-to-end experiment notebook
│ ├── SVG_and_LoRA_Tutorial.ipynb # Background: SVG basics + LoRA intro
│ └── starter-notebook.ipynb # Competition starter code
├── src/
│ ├── train_hpc.py # HPC training script (CLI)
│ ├── inference_hpc.py # HPC inference script (CLI)
│ ├── download_assets.sh # Downloads model + data (HPC)
│ ├── cluster.env.example # Cluster config template
│ ├── run_training.sbatch # SLURM training job
│ ├── run_inference.sbatch # SLURM inference job
│ └── run_inference_fast.sbatch # SLURM inference job (larger batch)
├── data/ # gitignored — Kaggle CSVs go here
├── models/ # gitignored — model weights go here
├── logs/ # gitignored — SLURM job logs
├── outputs/ # gitignored — generated submissions
└── requirements.txt
- Python 3.10+
- CUDA-capable GPU (≥16 GB VRAM recommended for training; 8 GB sufficient for inference with 4-bit quant)
- Linux (required for
bitsandbytes)
pip install -r requirements.txtInstall PyTorch with CUDA support separately: https://pytorch.org/get-started/locally/
| Component | Value |
|---|---|
| Random seed | 42 (set in all scripts) |
| Base model | unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit |
| LoRA rank / alpha | 32 / 32 |
| Training samples | 30,000 (filtered from train.csv) |
| Epochs | 1 |
| Learning rate | 2e-4 (cosine schedule, 100 warmup steps) |
| Batch size | 2 × 8 grad accum = effective 16 |
| Inference temperature | 0.3 |
| SVG canvas | 256 × 256 |