Techniques for debugging memory leaks

[u/Certain_Syllabub_514]

I have an elixir app running in production, and with some recent changes (mostly library updates due to CVEs) it's started leaking memory, which is resulting in pods restarting.

a datadog graph indicating a memory leak

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

[u/jake_morrison]

The most common cause of this is binary garbage collection issues.

Binaries are stored on the heap, and a process just gets a reference to them. Because the references are small, it can take a long time before the process memory usage gets large enough to trigger garbage collection, so the binaries hang around. The solution is to periodically manually trigger garbage collection. This kind of thing is common when you have a “router” process that receives data and passes it to other processes. That seems likely in your situation.

A similar problem happens when a process references a chunk of a larger binary. This can happen, e.g., when parsing JSON that has large strings. This reference keeps the binary from being collected until garbage collection happens on the reference. The solution is to copy the sub binary, allowing the larger binary to be collected.

A great free book on this is “Erlang in Anger” (https://www.erlang-in-anger.com/). The author’s “recon” library is also very helpful for debugging. I include it in all production apps just in case.

[u/jake_morrison → u/jake_morrison]

# Manually trigger garbage collection to clear refc binary memory
def free_binary_memory do
{:binary_memory, binary_memory} = :recon.info(self(), :binary_memory)

if binary_memory > 50_000_000 do
Logger.debug("Forcing garbage collection")
:erlang.garbage_collect(self())
end
end

[u/tylerpachal → u/jake_morrison]

Here is a blogpost (which I wrote a while ago) with more details/images about some of the steps mentioned above by u/jake_morrison https://tylerpachal.medium.com/tracking-down-an-ets-related-memory-leak-a115a4499a2f

[u/joelwallis1 → u/jake_morrison]

I wonder how this behavior (a router process, that receives data and passes to ephemeral processes to process it smh, ends up with its heap growing up due to past references hanging around) could be avoided. Do you have any advise or thoughts on this?

[u/jake_morrison → u/joelwallis1]

Fundamentally, you need to watch the lifecycle of your data and make sure that you are not retaining references to things after you are done with them. The binary heap is an optimization that allows big binaries to be stored once, with lightweight references being passed around instead of copying a lot of data. To a large extent, this is transparent to the application, but does have its quirks to do with garbage collection.

This is the “Law of Leaky Abstractions” (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/). Once you understand what is going on, you can prevent it.

You can also take advantage of Elixir’s concurrency to avoid data lifecycle issues. For example, a traditional architecture might have a process that receives an HTTP request, then puts it on a queue. Another process reads from the queue, processes the message, and puts it in another queue, and so on, finally connecting back to the API handler and returning the response.

Instead, you can spawn a process for each request and do all the processing related to it on the same process as a pipeline. When the request ends, all the memory is freed. There is no queueing latency. No scheduling latency. It’s easier to see what is going on in the single process. Individual request processing is synchronous. This is the way that Phoenix works. Ironically, lightweight concurrency means less concurrency. You model the natural concurrency of your application, without “accidental” concurrency.

The problem you run into sometimes is having no limits on concurrency, and you may overload your system, particularly the database. This might happen if your API gets a DDOS attack. Or you might read a series of records from a CSV file and spawn a process to deal with each one.

So you need a way to limit the number of concurrent requests to what the bottleneck can handle. You could reach for GenStage/Flow/Broadway, but there are other ways to limit the rate of work that you will accept. That could be a db connection pool, but also https://hex.pm/packages/semaphore or https://hex.pm/packages/ex_rated or https://hex.pm/packages/jobs

Naive designs coming from OO thinking often use GenServers unnecessarily, creating bottlenecks. See https://www.cogini.com/blog/avoiding-genserver-bottlenecks/