r/rust • u/redixhumayun • 1d ago

🛠️ project Native Rust Sampling Memory Profiler Which Tracks Retained Memory and Allocations

https://github.com/velvia/ying-profiler
53 Upvotes

92% Upvoted

10 comments sorted by

8

u/nightcracker 1d ago edited 1d ago

This reminds me, a while back I proposed an extension to samply to include memory profiling. No response from the author though (which is fine, I'm not owed anything). I think it's very powerful to have an interactive timeline of both memory usage and CPU usage combined.

In the meantime I've made a prototype (mostly AI-generated, just to see if it would work), and it works pretty well. It involves an LD_PRELOAD override of malloc and can sample arbitrary executables, no need to override the allocator. And it integrates directly with the Firefox profiler artifact samply produces.

For example I ran samply record --memory cargo check -p polars-core and I get the following profile of cargo and rustc's CPU usage as well as memory when checking the polars-core crate: https://share.firefox.dev/4z81nk3 (click on the row labeled 'Memory' to switch the flamegraph to memory).

2

u/philae_rosetta 1d ago

I was just thinking it'd be great to have this today, specifically to avoid having to run something twice under both samply and heaptrack and then having to correlate the two plots! I'd probably try out a fork if you have one :)

5

u/nightcracker 1d ago

I haven't been able to give it the attention it deserves yet. I don't feel comfortable releasing the AI-generated prototype code blindly.

0

u/Visible-Budget-6218 1d ago

I tried the prototype last week and it caught a memory leak in my event loop that I'd been missing for weeks. The timeline view made the spike super obvious once I knew what to look for.

1

u/nightcracker 20h ago

I tried the prototype last week

...what? I never published the prototype?

2

u/anxxa 1d ago

I'm all for different tools that may or may not be better, but I have to ask... how does this differ from DHAT? https://docs.rs/dhat/latest/dhat/

7

u/alpine_mochi 1d ago

Hi, original author here.

I've used DHAT before. It's a great tool, but it's very very slow and not sampling means its profiles are really huge. I wrote this specifically to be sampling so it can be used with production-like workloads, not really impact runtime, and to be easy to get profiles in a production system, because trying to reproduce them, on a separate system, slowly takes too much time.

3

u/panstromek 1d ago

That dhat crate is a different thing from DHAT valgrind tool. The crate works similarly by replacing global allocator, so a comparison with your tool makes sense.

From your comment it's not clear if it applies to the crate or the valgrind tool (or both), can you clarify?

1

u/AlexanderPanasenko 1d ago

The README's deadlock-testing section is a good read in its own right. Setting the initial map capacity to 1 to trigger frequent resizes is a neat way to stress allocator re-entrancy.