Repository navigation

tinymoe

Public

About

💥 Tiny 9M MoEs with Dynamic Routing

13 stars

1 watching

Report repository

Name		Name	Last commit message	Last commit date
data		data
docs		docs
experiments		experiments
src		src
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
README.md		README.md
requirements.txt		requirements.txt

Repository files navigation

TinyMoE: 9M Mixture of Experts Models

Heavily inspired by Karpathy's Makemore as well as AviSoori's MakeMoe.

✍️ Experiment Logbook: https://www.notion.so/dailyink/Dynamic-Routing-in-Tiny-9M-MoE-Models-4205e3283ef3479e8fcfdbc5514f0bdd

About

💥 Tiny 9M MoEs with Dynamic Routing

13 stars

1 watching

Report repository

Releases

Packages

Used by

Contributors

Languages