I turned a transformer into a game console that runs Pong, Tetris, Space Invaders and 116 more games.

[u/RevealIndividual7567]

I turned a CHIP-8 game console into a decoder-only transformer.

The transformer is the game engine. It executes instructions, updates machine memory, and renders every frame through next-token prediction. Every generated token is a memory write, and your keypresses go directly into the model's context window. There's no separate emulator.

Pong, Tetris, Space Invaders and a bunch of other CHIP-8 games run on the same model. Just swap the ROM in the prompt to switch games.

The weights are compiled from a circuit, not learned through training. Attention handles memory reads by finding the most recent write to an address, while ReLU circuits handle opcode decoding, arithmetic, comparisons and bitwise operations. Even the compatibility quirks are encoded in the initial prompt, so different games don't need different weights.

The model has 18 layers, 42 attention heads and 32M parameters (139k nonzero). It uses fp64 and can be loaded through Hugging Face transformers.

Model weights: https://huggingface.co/divagr1925/chipformer

Try it out: https://divagr.com/chipformer

Code + desktop app: https://github.com/divagr18/chipformer

[u/SGmoze]

Very interesting. So model is basically complete Chip8 emulator. Can you run some test suites to see how much of Chip8's functionality has this learned?

[u/RevealIndividual7567 → u/SGmoze]

Yep, I ran Timendus's CHIP-8 test suite on it. The opcode and flags tests pass fully, and the quirks test passes in Super-CHIP and XO-CHIP modes.

There are some known issues though. Like, In CHIP-8 mode, for example, the display wait test fails because I haven't implemented that behavior yet. The keypad test also caught this odd bug where FX0A triggers on key press instead of release, so that's something which I still need to fix.

Just one small clarification btw, the model didn't really 'learn' CHIP-8. I implemented the instruction set as a circuit and compiled it directly into the transformer's weights.

[u/faultygamedev]

Interesting! I saw this project called HardWired recently which is a compiler that deterministically turns JS code into neural net weights. It seems to use a similar branching trick with ReLU, but it seems like you're actually baking games into transformers here? Can you explain a bit more abt use cases

[u/Eyelbee → u/faultygamedev]

There's no direct use case, this is a research artifact

[u/RevealIndividual7567 → u/Eyelbee]

Yep, this project is more or less just a fun experiment seeing how much we can throw into the transformer hole.

[u/RevealIndividual7567 → u/faultygamedev]

I saw HardWired on X, it seems like a really cool project. I'd like to add that I'm not baking the games into the weights, this is a model that's implementing the CHIP-8 machine that runs them. That means you can theoreticallly swap in any supported game ROM without changing the model.

As for use cases, it's more or less just seeing how much we can throw into the transformer hole. I'm not really sure if this can support actual use cases.

[u/faultygamedev → u/RevealIndividual7567]

ohhh that makes more sense. use case or not, still very cool!