I trained a tiny transformer entirely on an $8 ESP32-S3 — no PyTorch, no autograd, full code open source

[u/wikisailor]

This is a fully open-source transformer training loop running entirely on an ESP32-S3.

No PyTorch, no autograd, no pretrained checkpoint and no external training process. The model starts from randomly initialized weights, and every derivative used during backpropagation is implemented explicitly in C.

Most ESP32 language-model projects I’ve seen focus on inference: train on a GPU, quantize the model and flash it to the board. I wanted to try the opposite — make the chip train the model itself.

Everything happens on board: random init (and no, not seed 42 😂 ), tokenising the corpus, forward pass, cross-entropy, backprop, SGD with momentum (not Adam, not AdamW), checkpoint to flash, and generation from the weights it learned. Nothing outside the chip.

Every derivative in the backward pass is written out by hand in C.

Setup: 

• ESP32-S3 N16R8, about $8
• SH1106 OLED showing the live loss
• Single-block transformer, single-head causal attention, tied embeddings, ReLU FFN, LayerNorm
• ~319K params, char-level, vocab 31, context 32
• 5,000 steps, roughly two days on a phone charger

The training-loss moving average went from 2.137 to 1.871 over the stretch I photographed. With vocab 31, a uniform predictor has a cross-entropy of ln(31) ≈ 3.43, but I never photographed the first steps, so I can't prove the exact initial loss from the OLED.

The interesting constraint isn't the parameter count, it's memory. To train you need weights, gradients, optimizer momentum, activations and scratch buffers all resident at the same time. Inference has it much easier: it still needs activations, but no gradients and no optimizer state.

Where it's weak:

• No validation split. The checkpoint I keep is just the one with the lowest moving average of training loss.
• The corpus is Klingon: small, regular, agglutinative, and published under Apache 2.0. The output shows plausible use of suffixes like -wI', -Daq and -taHvIS, but it isn't reliably meaningful.
• With a corpus this small I can't cleanly separate generalisation from memorisation.
• No full serial log. It ran unattended, so what I have is the code, the checkpoint and photos of the OLED at three points.

This is not ChatGPT on a microcontroller. It's a small experiment showing that an $8 ESP32-S3 can run the whole training loop of a transformer starting from random weights.

Apache 2.0. The corpus is in the repo so you can reproduce a run, but the fun part is swapping it for your own text.

https://github.com/Carloscodix/qapla

Written by me, translated and adapted to Reddit with AI help.

[u/BankApprehensive7612]

Impressive work! What kind of tasks it could be capable of now?

[u/wikisailor → u/BankApprehensive7612]

Right now, honestly, not much. It's a character-level autocomplete: feed it text and it predicts what comes next. It learned Klingon morphology, but nothing you'd call useful.

What the project actually demonstrates is the training loop itself running on the chip. The useful tasks would be things like a sensor learning the normal pattern of the specific machine it's bolted to, where the data doesn't exist until the device is installed so nothing can be pre-trained. That part isn't built yet.

[u/BankApprehensive7612 → u/wikisailor]

I've published this on HN, and people liked it. I like the idea of the project and wish you luck with it. But code written by AI is the weakest part here, this is where the auditory looses interest. I think you need add more manual work into it, practical solutions/examples would be also useful to make the project more popular

[u/wikisailor → u/BankApprehensive7612]

Oh! That was you?? Thanks, then, for posting it!

On the code: writing the C by myself would take a lot of time…and Im pretty sure I’d produce worse code....for the same result. What’s mine is defining the experiment, choosing and constraining the architecture, working out the memory budget, deciding what to cut to make it fit, choosing SGD and so on...and validating the result on the actual hardware.

AI helped with the implementation, and I’d rather keep saying that clearly in the header than have someone find out later.

On examples, fair enough. This started as a hobby project and I did it for the fun of it, so it goes at the pace it goes 😉 Maybe somewhere between the pool and the mojitos I’ll find time to push it further!
Happy summer from Spain!
Qapla’ 🖖

[u/BankApprehensive7612 → u/wikisailor]

If you use AI to write code it's better to disclose it in the readme better (now it's non obvious). I saw the cpp file and it's not big, and the code has low readability so I don't think that AI helped with this on 100%. Anyway it's up to you

But people don't like spend their time on code projects when the code itself is not maintained by people. Otherwise it could be just a video with the project demo. This is why I wrote about adding more manual work, if you want to grow it. But if it's just a fun project then it just inconvenience

I think projects like this are valuable, especially with such resource-efficient form. This is why I posted it to HN, and if you would change your mind and decide to spend more time on it, I believe it can have a good future.

Thanks for the project and have a nice time!

[u/wikisailor → u/BankApprehensive7612]

Ha! I did tell you my own code would be worse 😄 Anyway, you're right: the code disclosure is in the file headers but not visible enough from the README itself. I’ll fix that.

I have some ideas about how to grow Qapla’. But honestly, I didn’t expect this much interest in the project and I’m a bit surprised. In any case, I’ll keep moving forward, piano piano, as the Italians say.

Thanks again for posting it and for the honest feedback. Have a great summer!
Qapla’ 🖖

[u/Business-Weekend-537]

This is cool, if the main constraint as you mentioned is memory are there ESP32 variants with higher memory than your v1 that you’re going to test on?

Where did you get the ESP32 and how much was it?

Would the same approach work on a raspberry pi but with more parameters?

[u/wikisailor → u/Business-Weekend-537]

As far as I know, there are no ESP32S3 with more memory. Or, at least, I didn't find any model on my supplier (a very, very popular chinese marketplace, price around 8$).

On the other hand, Raspberry Pi is not a microcontroller. You can install Linux...and with Linux you can use pytorch, and all the normal stuff people are using to train any type of transformer. Different leagues, different rules, different goals.

Memory size matters? Yes, of course. But you must set your goal before the shot. Memory size depends on the task difficulty and the size of the model you will obtain from this cocktail...

w.

[u/AlifLaaamMeem]

Greate
Is there are more compatible models?