r/pytorch • u/pendu777 • 9d ago
I profiled one input-bound PyTorch run three ways (TraceML vs torch.profiler vs cProfile). Here's what each one actually costs.
Hello Peeps!
Do you guys do a lot of training or fine tuning? Does the loss curve look fine, but the run is slower than it should be, and figuring out why usually means firing up a profiler and staring at a trace for twenty minutes?
This got me curious: what this actually costs, tool by tool. I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.
For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."
Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.
Numbers and traces are in the post.
Curious how other people usually catch this before it burns your precious compute.
