A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM
-
Updated
Sep 10, 2026 - Python
A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM
MAPLE (EMNLP 2026): metadata-conditioned LLM pretraining and LocalNewsQA for locale-aware question answering.
Repository containing paper's code, results, and LaTeX source.
GPT-style LM pretrained from scratch on FineWeb-Edu under 100M params: RoPE, RMSNorm, QK-norm, SwiGLU, Muon
Double Trouble (EMNLP 2026): controlled bilingual pretraining and language-conditioned representations.
Project page: Double Trouble (EMNLP 2026 Outstanding Paper), bilingual pretraining and language-conditioned representations.
Implementation of ELECTRA pretraining architecture on a logic-focused corpus. Achieved 96.42% discriminator accuracy with 80% reduction in discriminator loss. Includes generator-discriminator training pipeline, domain adaptive pretraining, and comparison with RoBERTa baseline
Unofficial reproduction of Nemotron-CLIMB (CLustering-based Iterative Data Mixture Bootstrapping, NeurIPS 2025, arXiv:2504.13161) on Ascend NPUs — 5 rounds, ≈3,900 NPU-hours; learned mixtures beat every fixed-ratio baseline (+0.014–0.043 STEM); the selection mechanism is the weak link — fixed and validated live.
High-performance MinHash and Locality-Sensitive Hashing (LSH) deduplication engine for pretraining corpora, SFT datasets, and benchmark de-contamination.
Research-style write-ups: LLM pretraining under fixed compute, agents with a human in charge, Kaggle.
Architecture-agnostic data profiling via field physics — improves LLM pretraining by 1.0–1.9% on Transformers
Benchmark decontamination for LLM pretraining data: fast, auditable n-gram detection and removal of evaluation-benchmark text in training corpora (Rust).
Repository containing project page for "The Compute Floor of Data Mixing: When Loss-Based Reweighters Start to Drain Easy Synthetic Text" paper
LLM pretraining data mixture research insights
To associate your repository with the llm-pretraining topic, visit your repo's landing page and select "manage topics."