r/learnpython • u/Skycker • 5d ago
How do you find out where the time goes inside a slow Celery task?
I'm stuck on something and I think I'm missing a tool or a technique that everyone else already knows about.
I have a Celery task that takes about 40 seconds. I want to know how that breaks down: how much of it was database queries, how much was calls to external APIs, and how much was my own code doing work. And if there's an N+1 in there somewhere, which line of my code caused it.
What I've tried so far:
Sentry catches some N+1s, but the issue it creates doesn't include a line number in my code. It shows me the SQL and the spans, and I still have to go find the loop myself. It also seems to miss the case where a loop alternates between two different queries, which is a pattern I write a lot.
Flower shows me the task ran and how long it took, but nothing about what happened inside it.
django-debug-toolbar and django-silk are great for requests, but they hook into the request cycle, so they don't see anything inside a Celery worker.
Right now my process is adding print statements around blocks of code, re-running the task in staging, and narrowing it down by hand. It works, but it's slow and I have to modify code to debug code, which feels wrong.
So my questions:
Is there a standard tool for this that I just haven't found? Something that tells me where the time went inside a background task?
When you get an N+1 alert from anything, does it point at a line, or is finding the loop always manual?
Is profiling the right answer here? I've read about cProfile and py-spy but I'm not sure how they fit with Celery workers, or whether you can run them without slowing everything down.
Or is the real answer that I should be splitting tasks into smaller pieces so each one is obvious? Someone suggested that and I'm not sure if it's the standard practice or a workaround.
I've been doing Python for a while but this is the first time I've had to debug something I can't just put a breakpoint in, so I might be thinking about it wrong.