Skip to content

About

QLoRA fine-tuning of Qwen2.5-Coder-3B to generate SVG markup from natural-language prompts

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

7 Commits

Folders and files

Repository files navigation

Text-to-SVG Generation

Fine-tuning Qwen2.5-Coder-3B with LoRA to generate SVG markup from natural-language prompts. Built for the NYU DL Spring 2026 Kaggle competition.

Approach

  • Base model: unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit (4-bit quantized, ~2 GB VRAM)
  • Fine-tuning: QLoRA (r=32, α=32) via trl's SFTTrainer on 30k competition training samples
  • Prompt format: ChatML with a system prompt that enforces valid, well-formed SVG output
  • Post-processing: SVG extraction, canvas normalization (256×256), tag sanitization, fallback generation for failures
  • Inference: Batch generation with temperature=0.3, top_p=0.95, repetition_penalty=1.1

Quick Start — Kaggle Inference (Recommended)

The trained LoRA adapter is publicly available on HuggingFace. The easiest way to run inference is directly on Kaggle using the provided notebook.

  1. Open notebooks/kaggle-inference.ipynb on Kaggle (enable GPU T4 x2)
  2. Add the following datasets to the notebook:
  3. The notebook will:
    • Install dependencies
    • Load the base model from HuggingFace (unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit)
    • Load the adapter from /kaggle/input/qwen-svg-lora-v3
    • Read test.csv from the competition dataset
    • Generate SVGs and save submission.csv

No local setup needed.

Training

Option A — Google Colab

Open notebooks/colab-training.ipynb in Google Colab (A100 or L4 GPU recommended).

The notebook covers:

  • Mounting Google Drive
  • Downloading the competition dataset via Kaggle API
  • Fine-tuning with QLoRA
  • Saving the adapter to Drive or pushing to HuggingFace Hub

Option B — HPC / SLURM

For clusters with Singularity and SLURM.

1. Configure your environment

Copy the example config and fill in your cluster-specific paths:

cp src/cluster.env.example src/cluster.env
# then edit src/cluster.env

2. Download base model and data

# Requires ~/.huggingface/token and ~/.kaggle/kaggle.json
bash src/download_assets.sh

3. Submit training job

sbatch src/run_training.sbatch

4. Submit inference job

sbatch src/run_inference.sbatch        # batch_size=8
sbatch src/run_inference_fast.sbatch   # batch_size=16, faster

Logs go to logs/, submission CSVs go to outputs/.

SLURM configuration

Edit the #SBATCH directives in the .sbatch files to match your cluster's account name, partition, and GPU constraints.

Environment variables (cluster.env)

Variable Description Example
SINGULARITY_OVERLAY Path to your Singularity overlay image /scratch/you/pytorch.ext3:ro
SINGULARITY_SIF Path to the CUDA/cuDNN SIF image /share/apps/images/cuda12.6.3-...sif
SINGULARITY_PYTHON Python binary inside the overlay /ext3/miniforge3/bin/python
SINGULARITY_ENV Env activation script inside overlay /ext3/env.sh

Repository Structure

.
├── notebooks/
│   ├── kaggle-inference.ipynb      # Kaggle inference (primary submission path)
│   ├── colab-training.ipynb        # Google Colab fine-tuning
│   ├── midterm-kaggle.ipynb        # End-to-end experiment notebook
│   ├── SVG_and_LoRA_Tutorial.ipynb # Background: SVG basics + LoRA intro
│   └── starter-notebook.ipynb      # Competition starter code
├── src/
│   ├── train_hpc.py                # HPC training script (CLI)
│   ├── inference_hpc.py            # HPC inference script (CLI)
│   ├── download_assets.sh          # Downloads model + data (HPC)
│   ├── cluster.env.example         # Cluster config template
│   ├── run_training.sbatch         # SLURM training job
│   ├── run_inference.sbatch        # SLURM inference job
│   └── run_inference_fast.sbatch   # SLURM inference job (larger batch)
├── data/                           # gitignored — Kaggle CSVs go here
├── models/                         # gitignored — model weights go here
├── logs/                           # gitignored — SLURM job logs
├── outputs/                        # gitignored — generated submissions
└── requirements.txt

Requirements

  • Python 3.10+
  • CUDA-capable GPU (≥16 GB VRAM recommended for training; 8 GB sufficient for inference with 4-bit quant)
  • Linux (required for bitsandbytes)
pip install -r requirements.txt

Install PyTorch with CUDA support separately: https://pytorch.org/get-started/locally/

Reproducibility

Component Value
Random seed 42 (set in all scripts)
Base model unsloth/Qwen2.5-Coder-3B-Instruct-bnb-4bit
LoRA rank / alpha 32 / 32
Training samples 30,000 (filtered from train.csv)
Epochs 1
Learning rate 2e-4 (cosine schedule, 100 warmup steps)
Batch size 2 × 8 grad accum = effective 16
Inference temperature 0.3
SVG canvas 256 × 256

About

QLoRA fine-tuning of Qwen2.5-Coder-3B to generate SVG markup from natural-language prompts

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages