A 50M parameter large language model made in Pytorch! Can be locally trained, inferenced and tinkered with, and comes with a fun demo (more to come) that generates wikipedia pages. Modelfiles can be found on demo pages as they are too large for Github.
Tinyapper's flagship conversational model (the other model isn't conversational)! Try it out at https://tinyapper.craisin.tech!
Tinyapper's pretraining generator completes a heading, generating a fake Wikipedia article. Try it out at https://yapperpedia.craisin.tech!
Developed with Python 3.13, use older versions at your own risk:
git clone https://github.com/craisined/tinyapper.git
pip install -r requirements.txt # use a virtual environment if desiredThis hosts a server on port 8000:
cd demos/tinyapper
curl -o static/tinyapper.pt https://tinyapper.craisin.tech/static/tinyapper.pt
python3 app.pyThis hosts a demo server on port 8000:
cd demos/yapperpedia
curl -o static/yapperpedia.pt https://yapperpedia.craisin.tech/static/yapperpedia.pt
python3 app.pyContainers of the demos is published at ghcr.io/craisined/tinyapper and ghcr.io/craisined/yapperpedia.
curl -o tinyapper.pt https://yapperpedia.craisin.tech/static/tinyapper.pt
docker run -p 8000:8000 -v ./tinyapper.pt:/app/demos/tinyapper/static/tinyapper.pt ghcr.io/craisined/tinyapper:latestcurl -o yapperpedia.pt https://yapperpedia.craisin.tech/static/yapperpedia.pt
docker run -p 8000:8000 -v ./yapperpedia.pt:/app/demos/yapperpedia/static/yapperpedia.pt ghcr.io/craisined/yapperpedia:latestFor local training / inference, start with local.ipynb. If more flexibility is desired, take a gander through infer.py and train.py. Finally, all model architecture stuff is in model.py. Enjoy!
Tinyapper uses a transformer based architecture with 12 transformer layers. Each layer is comprised of an 8 headed multi attention layer along with a feed forward network. GeLU is used throughout as an activation. GPT-2s tokenizer was used for tokenization, and AdamW was used for optimization.
- Wikitext-103-raw-v1 for pretraining and yapperpedia
- Ultrachat-200k for conversation tuning
- AI usage was kept to research, debugging, and boilerplate code, as I am trying to work on my Actual Intellegence. An exception was made for RegEx parsing and web streaming because RegEx stinks
- README note: The header emojis got removed bc a Stardance shipwright thought they were AI (╥﹏╥). Oh well D:
- Pull requests are appreciated if you ever improve on this!
- The demo is currently hosted on a CPU only machine D: If I undergo enough boredom/support, future iterations might be on the cloud :O
- This project was made for Hackclub's Stardance! Feel free to leave a good rating to fund my cloud credit :D
- Thank you for all the support Stardance! Seeing the support on Slack and the reviews truly made my day. Also 17.57x multiplier for my first ship... y'all really funded for my cloud credit and more 🥹