GLiNER2 fine-tuning: Zig-native LoRA training at result parity with upstream

[timkaye11]

Summary

Adds end-to-end GLiNER2 LoRA fine-tuning to Antfly’s Zig inference stack, including full-task training, native and Metal execution, checkpoint/resume, evaluation, adapter materialization, and deterministic parity validation against fastino-ai/GLiNER2.

The implementation supports entities, classifications, JSON structures, relations, and count supervision through the gliner2-total-loss objective.

What’s included

Training and data pipeline

• DeBERTa-v3 forward and backward graphs with disentangled attention and relative-position handling.
• Full GLiNER2 objective: classification, structure, and count losses.
• Count-conditioned, per-instance span scoring for repeated structured fields.
• PEFT-compatible LoRA targeting, alpha / rank scaling, inverted dropout, and deterministic initialization.
• AdamW with decoupled weight decay and PyTorch-compatible per-parameter step semantics.
• Upstream-compatible schema conditioning using [DESCRIPTION], [EXAMPLE], and [OUTPUT].
• Pinned tokenizer, normalizer, Unicode, punctuation-aware word splitting, and span-grid behavior.
• Gradient accumulation, clipping, linear/cosine/constant schedules, evaluation, early stopping, and best-checkpoint selection.

Runtime and artifacts

• Native CPU correctness path and compiled resident-Metal training.
• Fail-closed Metal requirements for production-length attention and compiled execution.
• Exact checkpoint/resume, including optimizer moments, step counts, scheduler state, and run fingerprints.
• Adapter inspection, validation, held-out evaluation, and transactional merged-model materialization.
• Unified recipe and CLI integration under antfly inference finetune.
• Release manifests binding the model, tokenizer, training data, evaluation data, configuration, and generated artifacts.

See GLINER2.md for the operator and release contract.

Fastino parity

The correctness oracle is a pinned upstream Fastino GLiNER2 checkout running under the locked Python 3.12 environment.

Comparison	Result
Input IDs, attention masks, word/span indices, schema positions, and task metadata	Exact
Classification, structure, count, and total losses	Pass
Recorded deterministic loss deltas	approximately 2e-7
Loss gate	1e-4 absolute + 5e-6 relative
Full-task parity matrix	8/8 fixtures
Same-artifact adapter round-trip	Exact
Three-step AdamW per-parameter step counts	0 mismatches
Recorded independently trained adapter max tensor delta	2.97e-3 (<= 6e-3)
Native/Metal per-parameter gradient parity	3/3 objectives
Worst native/Metal relative gradient error	approximately 4e-6 (<= 1e-3)

The eight-fixture matrix covers entity extraction, classification, JSON structures, relations, mixed full-task records, repeated instances, all-negative classification, descriptions, and few-shot examples.

Metal parity additionally requires real GPU dispatches, the Metal optimizer path, zero trainable device-transfer fallbacks, finite losses, and no graph-executor fallback reasons.

Local real-model validation

Validated locally against the complete fastino/gliner2-base-v1 model bundle:

Gate	Result
Checkpoint tensors loaded	255/255
Real tokenizer/data tests	43/43 passed
Real-model training tests	2/2 passed
Two-step LoRA training loss	1.799863 -> 1.559474
Loss reduction	13.4%
Saved-adapter reload score	Exact at 0.249336
Full-task nonzero training smoke	Loss 20.804317, grad norm 8.8604
CI-equivalent Zig 0.16 x86_64 Linux build	Pass

The full-task smoke deliberately introduces one misclassified binary target because the pretrained model solves every supervised cell in the tiny fixture with margins above 20, causing the unmodified f32 BCE and gradient to saturate to zero. This keeps the test focused on proving a real backward/update path.

Performance

Diagnostic three-step configuration: batch 2, sequence length 64, LoRA rank 4.

Runtime	Approximate time per step
Fastino / PyTorch CPU	0.20 s
Zig Metal	0.55 s
Zig native	15 s

The native backend is the correctness reference. Performance qualification targets compiled Metal at production shapes—batch 32, sequence length 128—with five repeated runs, a warm-step median no slower than the Python baseline, no individual run above 1.10x, and a 3 GiB device-owned peak-live ceiling.

Scope

Parity covers the deterministic training core: preprocessing, losses, gradients, optimizer behavior, parameter updates, and artifact interoperability. Exact reproduction of upstream PyTorch RNG traces—shuffle order, dropout masks, stochastic negative masking, and augmentation sampling—is intentionally outside the contract.

The local model run proves that the complete checkpoint can train, update, save, and reload through the Zig stack. It is not a held-out quality benchmark. Final release qualification remains artifact-driven and requires representative disjoint train/eval data, all nine held-out quality floors, five paired independent Fastino/Zig training seeds, and production-shape Metal performance evidence.