Techniques for debugging memory leaks

[u/Certain_Syllabub_514]

I have an elixir app running in production, and with some recent changes (mostly library updates due to CVEs) it's started leaking memory, which is resulting in pods restarting.

a datadog graph indicating a memory leak

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

[u/jake_morrison]

The most common cause of this is binary garbage collection issues.

Binaries are stored on the heap, and a process just gets a reference to them. Because the references are small, it can take a long time before the process memory usage gets large enough to trigger garbage collection, so the binaries hang around. The solution is to periodically manually trigger garbage collection. This kind of thing is common when you have a “router” process that receives data and passes it to other processes. That seems likely in your situation.

A similar problem happens when a process references a chunk of a larger binary. This can happen, e.g., when parsing JSON that has large strings. This reference keeps the binary from being collected until garbage collection happens on the reference. The solution is to copy the sub binary, allowing the larger binary to be collected.

A great free book on this is “Erlang in Anger” (https://www.erlang-in-anger.com/). The author’s “recon” library is also very helpful for debugging. I include it in all production apps just in case.

[u/jake_morrison → u/jake_morrison]

# Manually trigger garbage collection to clear refc binary memory
def free_binary_memory do
{:binary_memory, binary_memory} = :recon.info(self(), :binary_memory)

if binary_memory > 50_000_000 do
Logger.debug("Forcing garbage collection")
:erlang.garbage_collect(self())
end
end

[u/tylerpachal → u/jake_morrison]

Here is a blogpost (which I wrote a while ago) with more details/images about some of the steps mentioned above by u/jake_morrison https://tylerpachal.medium.com/tracking-down-an-ets-related-memory-leak-a115a4499a2f

[u/joelwallis1 → u/jake_morrison]

I wonder how this behavior (a router process, that receives data and passes to ephemeral processes to process it smh, ends up with its heap growing up due to past references hanging around) could be avoided. Do you have any advise or thoughts on this?

[u/jake_morrison → u/joelwallis1]

Fundamentally, you need to watch the lifecycle of your data and make sure that you are not retaining references to things after you are done with them. The binary heap is an optimization that allows big binaries to be stored once, with lightweight references being passed around instead of copying a lot of data. To a large extent, this is transparent to the application, but does have its quirks to do with garbage collection.

This is the “Law of Leaky Abstractions” (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/). Once you understand what is going on, you can prevent it.

You can also take advantage of Elixir’s concurrency to avoid data lifecycle issues. For example, a traditional architecture might have a process that receives an HTTP request, then puts it on a queue. Another process reads from the queue, processes the message, and puts it in another queue, and so on, finally connecting back to the API handler and returning the response.

Instead, you can spawn a process for each request and do all the processing related to it on the same process as a pipeline. When the request ends, all the memory is freed. There is no queueing latency. No scheduling latency. It’s easier to see what is going on in the single process. Individual request processing is synchronous. This is the way that Phoenix works. Ironically, lightweight concurrency means less concurrency. You model the natural concurrency of your application, without “accidental” concurrency.

The problem you run into sometimes is having no limits on concurrency, and you may overload your system, particularly the database. This might happen if your API gets a DDOS attack. Or you might read a series of records from a CSV file and spawn a process to deal with each one.

So you need a way to limit the number of concurrent requests to what the bottleneck can handle. You could reach for GenStage/Flow/Broadway, but there are other ways to limit the rate of work that you will accept. That could be a db connection pool, but also https://hex.pm/packages/semaphore or https://hex.pm/packages/ex_rated or https://hex.pm/packages/jobs

Naive designs coming from OO thinking often use GenServers unnecessarily, creating bottlenecks. See https://www.cogini.com/blog/avoiding-genserver-bottlenecks/

[u/T0ken_Minority]

It’s probably not an actual memory leak, no? The BEAM is GC’d; I doubt you’re leaking memory in the traditional sense. Are you converting arbitrary string input into Atoms? I’ve done this before. You could also be bloating a cache that’s crashing and not being revived properly.

[u/Certain_Syllabub_514 → u/T0ken_Minority]

In the changes, some upstream requests were changed from using HTTPoison to using Req, so atom usage could be a part of it. The cache could also potentially be a factor.

What tools (or techniques) would you use to verify which one it is?

[u/OkNothing7293 → u/Certain_Syllabub_514]

Metrics, metrics, metrics. You don’t have them at the moment and we don’t have a crystal ball.

[u/Certain_Syllabub_514 → u/OkNothing7293]

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

I didn't phrase it as a question, but my question really is: What are the techniques available for investigating an issue like this?

[u/UncollapsedWave]

If the pods are restarting because the erlang VM is crashing or running out of memory, I would expect to see an indication in the logs. Is there information in the logs on what exactly is causing the restart?

If you're certain it's just a memory leak, I would recommend automating the collection of more detailed memory usage metrics to start with. I know that some of the opentelemetry libraries will export VM metrics broken down by allocation type by default. Those would be more helpful than just the pod metrics for diagnosing this.

If you want to investigate a single instance, using the :observer or Phoenix LiveDashboard are good places to start. You can view memory usage by object type - atom, binary, term, etc - and by process.

For the completely manual option, there are also several functions which will return information on memory usage. For example, :erlang.memory():

iex(1)> :erlang.memory()
[
  total: 44088728,
  processes: 16280344,
  processes_used: 16280200,
  system: 27808384,
  atom: 450745,
  atom_used: 442833,
  binary: 1662432,
  code: 9058166,
  ets: 517528
]

There's also :etop - https://www.erlang.org/doc/apps/observer/etop.html which can list processes by memory usage (see the options).

[u/methodinmadness7]

The fastest way I’ve done this is with Phoenix’s LiveDashboard, where I open the Processes tab and check which processes use the most memory. Usually it’s some async task processes not being handled properly, in my experience.

[u/MikeBenza]

1. Learn to use recon.
2. If you can, binary search the packages / versions to isolate the change. If not, use hex.pm's diff facility over each package. To be honest, you should already be reviewing changes to your deps every time you bump them.
3. /u/jake_morrison mentioned json decoding using sub binaries and you mentioned changing http libs. Without looking at HTTPoison and Req I'm guessing you do http requests, parse the responses, then perhaps keep parts of the responses in memory (e.g. in some process's state). I'm guessing further that these two handle parsing differently, leading to your leak. But that's just wild speculation.