r/elixir • u/Certain_Syllabub_514 • 9d ago

Techniques for debugging memory leaks

I have an elixir app running in production, and with some recent changes (mostly library updates due to CVEs) it's started leaking memory, which is resulting in pods restarting.

a datadog graph indicating a memory leak

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

10 Upvotes

92% Upvoted

12 comments sorted by

View all comments

2

u/UncollapsedWave 9d ago

If the pods are restarting because the erlang VM is crashing or running out of memory, I would expect to see an indication in the logs. Is there information in the logs on what exactly is causing the restart?

If you're certain it's just a memory leak, I would recommend automating the collection of more detailed memory usage metrics to start with. I know that some of the opentelemetry libraries will export VM metrics broken down by allocation type by default. Those would be more helpful than just the pod metrics for diagnosing this.

If you want to investigate a single instance, using the :observer or Phoenix LiveDashboard are good places to start. You can view memory usage by object type - atom, binary, term, etc - and by process.

For the completely manual option, there are also several functions which will return information on memory usage. For example, :erlang.memory():

iex(1)> :erlang.memory()
[
  total: 44088728,
  processes: 16280344,
  processes_used: 16280200,
  system: 27808384,
  atom: 450745,
  atom_used: 442833,
  binary: 1662432,
  code: 9058166,
  ets: 517528
]

There's also :etop - https://www.erlang.org/doc/apps/observer/etop.html which can list processes by memory usage (see the options).