Skip to content
stochi0Public

About

Qwen3-style language modeling with FineWeb streaming and tensor parallelism.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

38 Commits

Folders and files

Repository files navigation

Yuntun

A Qwen3-style causal language model implemented in PyTorch, with FineWeb streaming and Megatron-style tensor parallelism.

Installation

Use Python 3.13 or later and uv. The project requires PyTorch 2.10 or later; multi-GPU training requires CUDA.

git clone https://github.com/stochi0/yuntun.git
cd yuntun
uv sync --locked

Build a model

import torch
from yuntun import Qwen3Config, Qwen3ForCausalLM

config = Qwen3Config(
    vocab_size=256,
    hidden_size=64,
    intermediate_size=128,
    num_hidden_layers=2,
    num_attention_heads=4,
    num_key_value_heads=2,
)
model = Qwen3ForCausalLM(config)

This constructs a small random model without downloading pretrained weights. See the model implementation for its forward API.

Training

The training entrypoint reads configs/train.json. It uses FineWeb's sample-10BT configuration and a Qwen tokenizer from Hugging Face, so a training run can download data and model assets.

uv run python scripts/train.py --config configs/train.json --max-steps 100

For two-way tensor parallelism, set parallelism.tp_size to 2 in the config and launch two processes:

uv run python -m torch.distributed.run --nproc_per_node=2 \
  scripts/train.py --config configs/train.json

The process count must match the tensor-parallel configuration. The config also contains model dimensions, sequence length, batch size, learning rate, and output directory. Start with a model and batch size that fit the available device.

Architecture checks

uv run python scripts/verify_architecture.py

This downloads Qwen/Qwen3-0.6B and compares parameters, forward outputs, and generation behavior against the Hugging Face implementation. These compare the architecture, not training quality or throughput.

The tensor-parallel smoke check covers the distributed path.

Reference implementation and pretrained weights: Qwen3-0.6B.

Citation

@software{bodadeprojectyuntun,
  author = {Bodade, Ayush},
  title = {{Yuntun}},
  url = {https://github.com/stochi0/yuntun},
  note = {Software implementation; cite the original research separately. Specify the commit or version used.}
}

About

Qwen3-style language modeling with FineWeb streaming and tensor parallelism.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages