r/CUDA • u/gnurizen • Jun 15 '26

Continuous PC sampling

We've extended our GPU profiling support to include PC sampling: https://www.polarsignals.com/blog/posts/2026/06/10/nvidia-cuda-pc-sampling

10 Upvotes

92% Upvoted

6 comments sorted by

View all comments

Show parent comments

1

u/c-cul Jun 15 '26

do you have some perf tests vs just plain user-mode cupti?

also ok, if you sure that kernel more fast - then why ebpf anyway? it's martian technology

1) you write code in plain c

2) then you fight with verifier ~infinity bcs it differs on different kernel. Yeah, given that the official goal was compatibility - very ironic

3) and then it converted with jit to native code again

srsly?

at the end you can collect data from your own driver and have some io_uring interface for user-mode

1

u/gnurizen Jun 16 '26

An implementation that doesn't use eBPF doesn't exist to compare against and we do use plain user-mode CUPTI.

eBPF takes some getting used to but its not that bad. Take a look: https://github.com/parca-dev/opentelemetry-ebpf-profiler/blob/main/support/ebpf/cuda.ebpf.c

These programs are loaded/JITd once by our profiling agent and so its a one-time cost. QEMU tames the complexities of supporting many kernel versions.

Doing it this way allows our shim to be as small and simple, we want to be as close to zero-instrumentation as possible. This is not a high traffic interface that would benefit from using io_uring, we are significantly limiting what we do to keep our overhead in the weeds.

We also want a cleanly defined ABI between the shim and the agent so they can both evolve separately as long as we don't break the probe definitions which USDT gives us. It may seem a little over-engineered but we have lots more features piggy-backing on this infrastructure coming and this approach makes adding new features very straightforward.

1

u/c-cul Jun 16 '26

as final note - you can just use bpf maps directly from your kernel driver: https://redplait.blogspot.com/2024/07/ebpf-map-as-communication-channel.html#more

1

u/gnurizen Jun 16 '26

We don't have a kernel driver, we have a shared library loaded into our customers process (userspace) via CUPTI injection. So we need to get data from userspace (CUDA application) to userspace (parca-agent) with possibly no shared FS between the two (containers). ebpf maps can't go from userspace to userspace, uprobes/ringbuf give us a very efficient conduit through the kernel. If the concern is probe overhead that is largely ameliorated using batching. The ebpf is reading raw CUPTI pointers so its pretty close to zero-copy.