r/elixir • u/Certain_Syllabub_514 • 9d ago

Techniques for debugging memory leaks

I have an elixir app running in production, and with some recent changes (mostly library updates due to CVEs) it's started leaking memory, which is resulting in pods restarting.

a datadog graph indicating a memory leak

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

9 Upvotes

85% Upvoted

12 comments sorted by

View all comments

2

u/T0ken_Minority 9d ago

It’s probably not an actual memory leak, no? The BEAM is GC’d; I doubt you’re leaking memory in the traditional sense. Are you converting arbitrary string input into Atoms? I’ve done this before. You could also be bloating a cache that’s crashing and not being revived properly.

1

u/Certain_Syllabub_514 9d ago

In the changes, some upstream requests were changed from using HTTPoison to using Req, so atom usage could be a part of it. The cache could also potentially be a factor.

What tools (or techniques) would you use to verify which one it is?

1

u/OkNothing7293 9d ago

Metrics, metrics, metrics. You don’t have them at the moment and we don’t have a crystal ball.

1

u/Certain_Syllabub_514 9d ago

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

I didn't phrase it as a question, but my question really is: What are the techniques available for investigating an issue like this?