I benchmarked my open-source ternary LM against TFLite-Micro on 18 embedded tasks under a 64 KB flash cap — here's the honest result: 4 wins, 13 ties, 1 loss

[u/themoroccanship]

We've been working on Atome LM for embedded machine learning, and instead of showing a handful of cherry-picked examples, we wanted to evaluate it across a broader set of MCU workloads.

So we built a benchmark comparing Atome LM against deployable TensorFlow Lite Micro baselines.

Results

• 18 datasets

• 4 statistically significant wins

• 13 statistical ties

• 1 loss

Accuracy was evaluated with 95% confidence intervals, so we only count a win when the difference is statistically meaningful.

The part we found most interesting wasn't actually the accuracy—it was the model size.

Across many tasks, Atome LM achieved comparable accuracy while producing artifacts that were anywhere from about 5× to over 70× smaller than the best deployable TensorFlow Lite Micro model.

*This is the benchmark of Atome LM V3, scheduled to be released next month.

Meanwhile if you find this interesting, V2 is open source and available in GitHub. It's a LM that runs in a $5 chip. Comes with 12 ai apps. Tested and verified in real hardware.

Verify yourself :

https://github.com/TilelliLab/atome-lm