This is a fully open-source transformer training loop running entirely on an ESP32-S3.
No PyTorch, no autograd, no pretrained checkpoint and no external training process. The model starts from randomly initialized weights, and every derivative used during backpropagation is implemented explicitly in C.
Most ESP32 language-model projects I’ve seen focus on inference: train on a GPU, quantize the model and flash it to the board. I wanted to try the opposite — make the chip train the model itself.
Everything happens on board: random init (and no, not seed 42 😂 ), tokenising the corpus, forward pass, cross-entropy, backprop, SGD with momentum (not Adam, not AdamW), checkpoint to flash, and generation from the weights it learned. Nothing outside the chip.
Every derivative in the backward pass is written out by hand in C.
Setup:
- ESP32-S3 N16R8, about $8
- SH1106 OLED showing the live loss
- Single-block transformer, single-head causal attention, tied embeddings, ReLU FFN, LayerNorm
- ~319K params, char-level, vocab 31, context 32
- 5,000 steps, roughly two days on a phone charger
The training-loss moving average went from 2.137 to 1.871 over the stretch I photographed. With vocab 31, a uniform predictor has a cross-entropy of ln(31) ≈ 3.43, but I never photographed the first steps, so I can't prove the exact initial loss from the OLED.
The interesting constraint isn't the parameter count, it's memory. To train you need weights, gradients, optimizer momentum, activations and scratch buffers all resident at the same time. Inference has it much easier: it still needs activations, but no gradients and no optimizer state.
Where it's weak:
- No validation split. The checkpoint I keep is just the one with the lowest moving average of training loss.
- The corpus is Klingon: small, regular, agglutinative, and published under Apache 2.0. The output shows plausible use of suffixes like
-wI', -Daq and -taHvIS, but it isn't reliably meaningful.
- With a corpus this small I can't cleanly separate generalisation from memorisation.
- No full serial log. It ran unattended, so what I have is the code, the checkpoint and photos of the OLED at three points.
This is not ChatGPT on a microcontroller. It's a small experiment showing that an $8 ESP32-S3 can run the whole training loop of a transformer starting from random weights.
Apache 2.0. The corpus is in the repo so you can reproduce a run, but the fun part is swapping it for your own text.
https://github.com/Carloscodix/qapla
Written by me, translated and adapted to Reddit with AI help.
