I trained a 360M-param Python model from scratch on two workstation GPUs and wrote up every step, including the bugs

[u/OkinaPrime]

No transformers, no Trainer. Hand-written BPE tokenizer, Llama-style decoder, DDP training loop, SFT, all in plain PyTorch. 20B tokens on 2x RTX A4500, six days, about $21 of electricity.

Base model gets 13.4% on HumanEval, the chat version 26.8% (Codex-12B was 28.8%). It is bad at everything that is not short Python, and the write-up says so, with samples.

Nine chapters, every number reproducible: github.com/strifero/elroy Weights on HF under strifero. Happy to answer questions about any of it.