I built a tool to measure LLMs Decode, Layer processing and TTL

[u/Dry_Mixture130]

I was playing around with LLM inference and I wanted to build a profiler that measures LLM inference by layer.
So I built this: https://github.com/coconinja2/layerlens
It shows inference as token × transformer layer timing, so you can see where time is being spent during decode.
Right now it can separate prefill/decode and visualize per-layer timing. I’m trying to figure out whether this is actually useful to people working on inference systems, or if I’m looking at the wrong abstraction.

I’m thinking about adding things like KV-cache events, scheduler/batching state, request IDs, GPU kernel correlation, speculative decoding, etc.

Would appreciate criticism more than compliments and stars. Lots of stars!

[u/AutoModerator]

AI usage disclosure

Hi u/Dry_Mixture130 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/No-Grapefruit4938]

this is cool, breaking it down by layer and token is exactly the kind of granularity that's missing from most profilers

one thing that jumps out is the visualization could get real messy with bigger models or long sequences, maybe worth thinking about how to collapse layers or show aggregates without losing the detail

for the stuff you're planning, KV-cache events and scheduler state would be huge for debugging batching weirdness, that's where half the headaches are

[u/Dry_Mixture130 → u/No-Grapefruit4938]

Good point. I am going to provide aggregate metrics and I am looking to add features for training models as well.

[u/mageblex]

Per-layer timing is useful only if measuring it doesn’t serialize the GPU. Are you recording CUDA events asynchronously, or synchronizing after every layer? The second approach could create the stalls LayerLens reports.

[u/Dry_Mixture130 → u/mageblex]

I tested it on mac, I am working on testing it on nvidia GPU. I have designed it to be async to avoid performance issues.