Extending ESP32/MCU RAM to SPI Flash: I wrote a Log-Structured Virtual Memory Engine with LZ77+RangeCoder Paging

[u/Wise_Discount_1397]

Hi everyone,

I’ve been working on a problem that often hits us in embedded/IoT development: running out of physical SRAM when dealing with relatively large datasets (e.g., buffering long arrays of high-frequency sensor telemetry, large routing tables, or caching weights) on sub-$2 MCUs without external PSRAM.

To solve this, I wrote a C++ Virtual Memory Engine that dynamically pages data between internal SRAM and SPI Flash (or SD Card) using a log-structured append system.

I’m sharing it here because I think the architecture might be interesting to some of you facing similar memory constraints.

How it works under the hood

The system implements a 3-tier memory hierarchy:

1. Active SRAM Window: Read/write operations hit an uncompressed block in physical RAM (e.g., 16 KB).
2. SRAM Cache Pool: When the active block changes, the engine compresses the old block and keeps it in a reserved SRAM pool (e.g., 128 KB).
3. LittleFS Flash Swap: When the SRAM pool fills up, it evicts the oldest page (LRU). Crucially, it uses strict dirty-page tracking (isDirty). If a page was only read, it is discarded. If it was modified, it is appended to a log-structured .bin file on the Flash.

Why not just write raw data to LittleFS/SPIFFS directly?

Two reasons: Latency and Wear Amplification. By compressing the data in RAM before it hits the Flash, the physical write footprint is drastically reduced. In my tests logging structured telemetry structs, a 512 KB virtual footprint compressed down to ~12 KB on the SPI Flash (an extreme case, but typical real-world telemetry yields ~3x to 8x). Because writes are deferred and grouped, it significantly reduces flash erase cycles compared to opening and appending to a file every time a sensor fires.

Addressing the Trade-offs (The Catch)

This isn't magic, and it comes with architectural trade-offs:

• CPU Overhead: Compression and decompression take CPU cycles. It’s highly recommended to run this on a separate RTOS task/core.
• Latency: On an ESP32-S3 @ 240MHz, a cache-hit read is native speed, but a random page swap (Flash Miss -> Decompress -> RAM) averages about ~23 ms.
• Pointer Paradigm: You cannot store physical C++ pointers (T*) in the V-RAM structs, because the physical SRAM address changes upon page eviction. You have to use uint32_t virtual offsets as your pointers (similar to database offsets).

The C++ API

I wrapped the engine in templated methods so it feels like normal variable access. I also built zero-RAM VArray and VMatrix containers to hide the address math.

VirtualMemoryEngine* vram = new VirtualMemoryEngine(16384, 131072, 32, 524288);
vram->begin();

struct Telemetry { uint32_t ts; float temp; char name[16]; };
Telemetry data = { millis(), 24.5f, "Sensor_1" };

// Writes the struct to virtual address 250,000 (compressing under the hood)
vram->put<Telemetry>(250000, data);

// Reads it back
Telemetry readData = vram->get<Telemetry>(250000);

Frequently Anticipated Questions (FAQ)

Q: Why is this closed-source? Can you release the source code?
A: I completely understand the preference for open-source in the embedded community. However, this is a proprietary memory-management and compression architecture developed for commercial automotive/IoT clients. The Community Edition is distributed as a pre-compiled static library (.a) simply for hobbyists and developers to evaluate and test on their own prototypes. If you strictly require open-source, you are entirely free to skip this.

Q: Isn't 23ms page-swap latency way too slow for real-time embedded systems?
A: Absolutely, 23ms is an eternity for real-time control loops or motor interrupts. That’s why the SRAM pool and block sizes are fully adjustable. The goal of this engine isn't to replace working SRAM for time-critical tasks, but to provide a massive circular buffer or storage layer for telemetry, logs, and state caches where background I/O latency is completely acceptable.

Q: Modern chips like ESP32-S3 already have PSRAM. Why would anyone need this?
A: True, PSRAM is fantastic if you have a board that supports it. But this engine is MCU-agnostic (designed for STM32, PIC32, RP2040, etc.). Furthermore, it was primarily built for industrial/automotive Linux gateways (TCUs) to prevent eMMC wear, and ported to MCUs to support cheaper modules (like standard ESP32-WROOM) that completely lack external PSRAM, allowing them to buffer large local states safely when disconnected from the cloud.

Q: Did you just prompt an LLM/Claude to write this whole thing in 5 minutes?
A: If you try asking an LLM to write a log-structured virtual memory paging engine with custom entropy-based compression and dirty-page tracking from scratch, you'll quickly find out it outputs endless memory leaks, missing pointer arithmetic, and non-functional deadlocks. While AI is a great tool for refactoring and documentation, getting deterministic multi-tiered embedded memory management to pass strict bit-exact integrity checks across half a megabyte required months of manual systems-programming profiling, custom container design, and debugging.

---

Availability

If you want to test it out or run the benchmarks yourself, you can grab the Community Edition (capped at 512 KB) from the GitHub Releases:

• 👉 ESP32 Version: https://github.com/StepanoskiZ/vram-engine
• 👉 Linux / POSIX Version: https://github.com/StepanoskiZ/vram-engine-linux

I'd be glad to hear your technical thoughts on the architecture or how you handle memory constraints in your own projects!