Skip to content

About

26M parameter language model with Mixture of Experts architecture, Multi-Latent Attention and efficient expert routing. Only activates ~13M parameters per token.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

About

26M parameter language model with Mixture of Experts architecture, Multi-Latent Attention and efficient expert routing. Only activates ~13M parameters per token.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages