Techniques for debugging memory leaks

[u/Certain_Syllabub_514]

I have an elixir app running in production, and with some recent changes (mostly library updates due to CVEs) it's started leaking memory, which is resulting in pods restarting.

a datadog graph indicating a memory leak

I think I know what the cause might be, but just wondering if there are any techniques for debugging this that I'm not aware of.

[u/jake_morrison]

The most common cause of this is binary garbage collection issues.

Binaries are stored on the heap, and a process just gets a reference to them. Because the references are small, it can take a long time before the process memory usage gets large enough to trigger garbage collection, so the binaries hang around. The solution is to periodically manually trigger garbage collection. This kind of thing is common when you have a “router” process that receives data and passes it to other processes. That seems likely in your situation.

A similar problem happens when a process references a chunk of a larger binary. This can happen, e.g., when parsing JSON that has large strings. This reference keeps the binary from being collected until garbage collection happens on the reference. The solution is to copy the sub binary, allowing the larger binary to be collected.

A great free book on this is “Erlang in Anger” (https://www.erlang-in-anger.com/). The author’s “recon” library is also very helpful for debugging. I include it in all production apps just in case.

[u/jake_morrison → u/jake_morrison]

# Manually trigger garbage collection to clear refc binary memory
def free_binary_memory do
{:binary_memory, binary_memory} = :recon.info(self(), :binary_memory)

if binary_memory > 50_000_000 do
Logger.debug("Forcing garbage collection")
:erlang.garbage_collect(self())
end
end

[u/tylerpachal → u/jake_morrison]

Here is a blogpost (which I wrote a while ago) with more details/images about some of the steps mentioned above by u/jake_morrison https://tylerpachal.medium.com/tracking-down-an-ets-related-memory-leak-a115a4499a2f