A Qwen3-style causal language model implemented in PyTorch, with FineWeb streaming and Megatron-style tensor parallelism.
Use Python 3.13 or later and uv. The project requires PyTorch 2.10 or later; multi-GPU training requires CUDA.
git clone https://github.com/stochi0/yuntun.git
cd yuntun
uv sync --lockedimport torch
from yuntun import Qwen3Config, Qwen3ForCausalLM
config = Qwen3Config(
vocab_size=256,
hidden_size=64,
intermediate_size=128,
num_hidden_layers=2,
num_attention_heads=4,
num_key_value_heads=2,
)
model = Qwen3ForCausalLM(config)This constructs a small random model without downloading pretrained weights. See the model implementation for its forward API.
The training entrypoint reads configs/train.json. It uses
FineWeb's sample-10BT configuration and a Qwen tokenizer from Hugging Face,
so a training run can download data and model assets.
uv run python scripts/train.py --config configs/train.json --max-steps 100For two-way tensor parallelism, set parallelism.tp_size to 2 in the config
and launch two processes:
uv run python -m torch.distributed.run --nproc_per_node=2 \
scripts/train.py --config configs/train.jsonThe process count must match the tensor-parallel configuration. The config also contains model dimensions, sequence length, batch size, learning rate, and output directory. Start with a model and batch size that fit the available device.
uv run python scripts/verify_architecture.pyThis downloads Qwen/Qwen3-0.6B and compares parameters, forward outputs, and
generation behavior against the Hugging Face implementation. These compare the architecture, not training quality or throughput.
The tensor-parallel smoke check covers the distributed path.
Reference implementation and pretrained weights: Qwen3-0.6B.
@software{bodadeprojectyuntun,
author = {Bodade, Ayush},
title = {{Yuntun}},
url = {https://github.com/stochi0/yuntun},
note = {Software implementation; cite the original research separately. Specify the commit or version used.}
}