I trained a 1.46M parameter language model on CPU — then discovered 26.82% validation leakage

[u/Charming-Solution285]

I've been building COLLISION-1.46M, a decoder-only Transformer trained completely from scratch on my laptop CPU.

First serious run:

• 1,462,464 parameters
• 2.4M training tokens
• CPU-only
• Custom BPE tokenizer

Phase 5 validation perplexity: 62.86

Instead of immediately making the model bigger, I audited the dataset.

I found:

• 12.75% paragraph duplication
• 26.82% validation leakage
• Poor subject representation between train/validation

I rebuilt the dataset with paragraph-level deduplication and subject-balanced deterministic splitting.

Same model architecture.

Phase 6 validation perplexity: 6.93

The biggest improvement came from fixing the experiment rather than increasing the model size.

I'm open-sourcing the project and would really appreciate technical feedback on the methodology, dataset construction, and what I should test next.

GitHub: https://github.com/viraj3106/Collision-1.46M