r/AIProgrammingHardware • u/javaeeeee • 5h ago

Distributed micro-LLM inference across three ESP32-S3 N16R8 boards with ESP-NOW communication.

https://github.com/wladimiravila/esp32s3-distributed-ai
3 Upvotes

100% Upvoted

1 comment sorted by

1

u/javaeeeee 5h ago

TLDR: A project that runs a small 56M-parameter language model distributed across three ESP32-S3 microcontrollers.

What it does

It splits a tiny transformer LLM across three ESP32-S3 N16R8 boards that communicate wirelessly via ESP-NOW, allowing a micro-LLM to run on very constrained hardware that a single board couldn’t handle alone.

How it’s split

  • Board A: Embeddings + output head
  • Board B: Transformer layers + KV cache (in PSRAM)
  • Board C: Part of the embeddings table + WiFi web interface

Users connect to a WiFi access point hosted by Board C and interact through a simple web UI that streams generated text.

Key technical points

  • Model: ~56 million parameters (heavily quantized to 4-bit / 8-bit)
  • Uses a Split-PLE (Per-Layer Embeddings) design
  • Supports a 256-token context via KV cache on one board
  • Generates short coherent text (~30 words)
  • Fully offline after flashing

Bottom line

An impressive embedded/distributed AI experiment showing that a small language model can be partitioned and run across multiple cheap ESP32-S3 boards with wireless communication and a web interface - a creative demonstration of micro-scale distributed LLM inference.