PlayJev: A Multimodal JEV-Like Model for Small Games

🎮  Show the live demo  🎮

PlayJev is Qwen3.5-0.8B-Base fine-tuned to play ten classic browser games from raw pixels. One frame goes in, one forward pass runs, one move comes out, 43 ms on an H200. Every picture above is the trained model playing, each a frame from a recorded held-out episode with the score it had reached by then. The weights are on Hugging Face, with every record we trained on beside them in data/.

🎮 The Ten Games

Plain HTML5/JS with one hook each (window.pj: start(seed), step(action), frame(), score(), done(), actions), and the same page is both training environment and demo tile. Every tile below opens that game on the demo, with the model playing it.

Tetris 5 moves	Snake 4 moves	Pacman 4 moves	Javascript Racer 6 moves	Space Invaders 3 moves
Sokoban 4 moves	Infinite Mario 7 moves	Floppy Bird 2 moves	Breakout 3 moves	2048 4 moves

🔍 Why Pixels

Every Jev-shaped game demo before this one feeds the model text. TypeSafe's Mario parses emulator RAM into JSON and says the model gets no screenshots, the open-jev Doom demo writes one line of text per frame, and VideoGameBench and lmgame-Bench run frontier models with reasoning loops at seconds per move. PlayJev plays ten games from the frame alone, in real time, and returns a probability the game loop can act on.

🧠 How a Decision Is Made

The model never sees a game's name. It sees the current frame and the list of moves, and it answers with one of them.

The demo's single game view is the whole model in one picture: the frame on the left is the only input, the bars are what the forward pass returns, the line below them is how sure it was at every step so far. Moves are shuffled in every training sample, so position carries no information. One frame per decision: these weights see a single still image. The vision tower's temporal patch of 2 can carry a second frame at no extra token cost, and this release leaves that off. The prompt is built in playjev/model.py, and the demo prints the exact one for every game.

▶️ Run It

The model plays a game, one command:

python -m playjev.play snake --policy local --ckpt OmniJev/PlayJev-0.8B --episodes 1

It pulls the weights from Hugging Face, opens Snake in headless Chromium and plays it. scripts/reproduce.sh builds the whole model from nothing, from the teachers through three DAgger rounds to the closed-loop score, and the records we trained on can be drawn back into frames (Training Data below). Setup once (Python 3.12, a CUDA GPU, about 3 GB for inference and 17 GB for training at batch 64):

git clone https://github.com/OmniJev/PlayJev && cd PlayJev
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt && playwright install chromium

Command	What it does
playjev.play <game> --policy teacher (or random)	the two reference rows; --delay 1 decides one step late
playjev.collect <game> --steps 4000 --shard s0	(frame, teacher target) pairs under data/<game>/s0
playjev.rebuild data	the frames of the released records, each checked against the MD5 of the one we trained on
playjev.train_sft --games <game> --model Qwen/Qwen3.5-0.8B-Base --out ckpt/x	fine-tune, one epoch
playjev.serve --ckpt <ckpt> --port 18732	the checkpoint at /v1/systemone in the OpenJev request shape; the demo switches every tile to it with ?server=http://127.0.0.1:18732
bash scripts/reproduce.sh	the whole model from nothing: the teachers collect, the base model clones them, three DAgger rounds with the replay mix, the closed loop
python scripts/build_demo.py	rebuild demo/ from the games and the recorded runs, then serve it and open index.html

📊 Results

Ten games, one model, 16 held-out episodes per game, argmax move. Random play and the teacher run the same seeds through the same harness; vs teacher is (model - random) / (teacher - random). Each version is one more DAgger round: the model plays, the teachers label what it visited. Bold is the released model.

Game	Random	Cloning	Version 1	Version 2	Version 3	Teacher	vs teacher
Space Invaders	215	400	400	400	400	400	1.00
Racer	238	6211	6704	6707	6709	6712	1.00
Sokoban	6.6	57.9	102.3	102.1	102.1	102.2	1.00
Snake	1.0	77.8	107.9	89.5	102.9	114	0.90
Pacman	113	1036	3209	3702	3965	7026	0.56
Tetris	162	1034	1561	4718	5792	15288	0.37
Infinite Mario	613	1156	1170	1764	1757	4229	0.32
2048	1021	3174	2170	3386	4868	19593	0.21
Floppy Bird	0.0	8.9	9.3	13.8	15.5	84.0	0.18
Breakout	496	611	1552	2712	2785	16547	0.14
mean vs teacher		0.37	0.49	0.53	0.57

🧭 General Ability

Two held-out sets the training never saw, 200 questions each, asked under the game contract (picture or passage as the state, question as the instruction, answers as the options). Version 3 spends a fifth of every epoch on general image questions and a fifth on text; it beats the base model on MMBench and matches it on MMLU.

How much replay it takes: four runs of 1500 steps from the base model, same budget, a growing share of the batches drawn from general image and text questions. Game agreement is validation agreement with the teachers.

Replay share	Game agreement	MMBench dev	MMLU test
base model, no training		0.66	0.33
games only	0.430	0.48	0.29
10 percent	0.431	0.65	0.42
20 percent (the release mix)	0.586	0.78	0.44
30 percent	0.547	0.83	0.47
chance		0.40	0.25

🏋️ Training Recipe

Stage	Setting
Teachers	One search program per game on the internal state: BFS (Snake), expectimax (2048), Dellacherie (Tetris), A* (Sokoban), exact physics (Floppy Bird), ghost occupancy (Pacman), ball flight (Breakout), dodge-and-aim DP (Invaders), lookahead steering (Racer), physics rollouts (Mario). Soft target 0.9 / 0.1 / 0.
Collection	100k frames per game, 448 px JPEGs, 2 to 30 percent random moves.
Cloning	One epoch, batch 64, lr 2e-5, full fine-tuning, bf16 autocast on fp32 master weights.
DAgger round	The model plays 40k frames per game, the teachers label them, one epoch at lr 1e-5.
Replay (version 3)	60 percent game batches, 20 percent general image questions (A-OKVQA, ScienceQA), 20 percent text (MMLU auxiliary train, SciQ, ARC); 30 percent of game samples get an option rewrite. Restarts from the base model on all 1.8M frames (12 h on one H200), then one more round (7 h).
Closed loop	16 held-out episodes per game, the same seeds as random play and the teacher.

Random play and every teacher on the same seeds: docs/BASELINES.md.

🗂️ Training Data

Every record we trained on sits beside the weights on Hugging Face, in data/: 2,166,984 decisions from 11,416 episodes, 853 MB of JSON lines with the teacher's target, the move taken and the MD5 of each frame. The frames themselves are drawn again here, since an episode is fixed by its seed and its moves:

hf download OmniJev/PlayJev-0.8B --include "data/*" --local-dir .
python -m playjev.rebuild data --pages 16

The rebuild replays every episode in headless Chromium (about two hours at 16 pages), checks the score and the clock at each step and counts the frames that match the MD5. On x86-64 Linux all of them do (checked on two clusters). On ARM the states are the same and five games match byte for byte, the other five differ by a few grey levels where sprites are scaled. bash scripts/reproduce.sh then keeps every complete shard and trains on it.

Round	Played by	Records
clone	the teachers, 2 to 30 percent random moves	966,744
dagger1	the cloning model, 5 percent random moves	400,080
dagger2	version 1	400,080
dagger3	the replay model version 3 continues from	400,080

📈 More Charts

Hand the least confident steps to a System Two (playjev.play --handover) and the score climbs to the teacher's; the same number of steps picked at random does not.

The probability means something in every game: how often the move matched the teacher against how sure the model was.

One step of latency takes the reflex games apart and leaves the slow ones alone (cloning model, both bars).

📁 Repository

games/<id>/        vendored game, pj.json manifest, pj_hook.js, NOTES.md, TEACHER.md
games/_shared/     pj_shim.js: virtual clock, seeded Math.random, synthetic keys, frame grab; fonts/: the only fonts the browser sees
playjev/           env.py (Playwright driver), collect.py, rebuild.py, teachers/, model.py, train_sft.py, play.py, serve.py
demo/              the GitHub Pages site; scripts/build_demo.py assembles it from games/ and runs/replays/
docs/              HARNESS.md (the hook contract), BASELINES.md (the reference scores), DEMO.md

🔗 Related

OpenJev, the text-state System One server this model plugs into, and Awesome-JEV, the reading list behind System One models.

⚖️ Licence and Credits

Code and trained weights: Apache-2.0.

The ten games are other people's work, vendored under games/<id>/ with the author's own licence file and a vendor.patch of every line we changed.

• Tetris, github.com/jakesgordon/javascript-tetris, MIT
• Snake, github.com/patorjk/JavaScript-Snake, MIT
• Pacman, github.com/daleharvey/pacman, WTFPL
• Javascript Racer, github.com/jakesgordon/javascript-racer, MIT
• Space Invaders, github.com/StrykerKKD/SpaceInvaders, MIT
• Sokoban, github.com/taniarascia/sokoban, MIT
• Infinite Mario, github.com/robertkleffner/mariohtml5, Unlicense
• Floppy Bird, github.com/nebez/floppybird, Apache-2.0
• Breakout, github.com/jakesgordon/javascript-breakout, MIT
• 2048, github.com/gabrielecirulli/2048, MIT

Three of those READMEs say the art is not the author's to license: Mario's sprites are Nintendo's, Floppy Bird's come from the original Android game and belong to Dong Nguyen and .GEARS, the Racer's are placeholder art from the Mega Drive OutRun. The licence above covers the code each author wrote. Sokoban's Microban levels are by David Skinner. The DejaVu Sans fonts in games/_shared/fonts/ keep their own licence, in LICENSE beside them.

The roster ships silent. Every sound and music file was deleted, which costs nothing: the driver already aborted every audio request (playjev/env.py), the shim forces media elements muted, and Chromium runs with --mute-audio. Every frame in this repository, training or demo, was produced in silence.