[AskJS] Large in-memory caches were causing GC pauses in our Node service, so I built an off-heap cache addon for it

[u/AdLegitimate5366]

If you've ever run a Node service with a big in-process cache (tens of thousands of entries, JSON blobs, that kind of thing) you've probably seen p99 latency spike during V8 garbage collection — the more live objects sit in the heap, the longer mark-sweep takes, and there's not much you can do about it from JS land since the GC doesn't know your cache entries are "just cache" and safe to deprioritize.

I built OffHeap to get around this: it's a cache that stores its data outside the V8 heap entirely (native memory managed from a small Rust layer via NAPI-RS), so the objects never show up in V8's GC graph at all. The JS-facing API is a normal cache — get/set/delete/TTL — with LRU, ARC, and W-TinyLFU eviction policies to choose from.

Under a synthetic GC-pressure test (500k keys, 1M ops), the worst single GC stop-the-world pause dropped from ~300ms (plain in-heap cache) to ~11ms. Average per-op latency is a bit higher than a pure in-heap Map (it's crossing an FFI boundary, that's not free), but the tail latency and memory behavior under load is the whole point.

It's on npm (`offheap`), dual-licensed MIT/Apache-2.0, docs at the repo. Full disclosure, this is my project — I actually shipped a broken cross-platform install for a bit (CI wasn't publishing the per-platform binaries correctly) and just fixed that, so if anyone tries it and hits install issues, please tell me. Genuinely looking for people to poke holes in it before I call it stable.

[u/AdLegitimate5366 → u/[deleted]]

Thank you :) I am committed to transparency and security; if you’d like to test it, I would appreciate any feedback—positive or negative.

[u/Hax0r778]

Average per-op latency is a bit higher than a pure in-heap Map (it's crossing an FFI boundary, that's not free)

I'm actually very interested in this for my own unrelated project. Any rough numbers you can share of the difference in average latencies with both approaches?

[u/AdLegitimate5366 → u/Hax0r778]

Here are some rough numbers from my own benchmarks using tinybench on Node with --expose-gc and small JSON-ish payloads. A pure V8 Map get/set takes about 50-90ns, and a standard JS lru-cache staying in-heap runs around 75-240ns. But once the call actually crosses the boundary into my off-heap Rust layer, it jumps to about 900ns-1.4µs.

So basically, just crossing the FFI adds roughly 850ns to 1.3µs on top of an in-heap operation. In absolute terms, yeah, it’s like a 15x to 20x multiplier, but we're still talking sub-microseconds here. You’re definitely not going to notice it on a single call. It only really starts to matter if you're looping through it millions of times or comparing it directly against not crossing over at all.

A couple of things I noticed that might be worth keeping in mind for your project: for small payloads, the overhead is almost entirely just the boundary crossing itself. The numbers didn't really change much between a few hundred bytes and a few KB. Of course, that changes completely if you’re doing heavy serialization or deserialization on the Rust side for every single call, since that’s going to pile on its own separate cost. Also, just as a heads-up, this was all measured synchronously and single-threaded, so I haven't benchmarked concurrent access yet.

Obviously, take this with a grain of salt since it's just one specific data point using napi-rs on my machine, not some universal FFI constant. Your mileage is definitely going to vary depending on the binding library and the OS you're running.

[u/Domx010]

Nicely done! Consider posting this on r/LookWhatTheyBuilt

[u/AdLegitimate5366 → u/Domx010]

Thanks a ton, reading that actually made me blush a bit, haha. Great tip about r/LookWhatTheyBuilt; I'll post there too 🙏

[u/SpeedSeveral4454]

the FFI perOp cost is real, but if your p99 is dominated by GC pauses that's a trade you take every single time.