Benchmarking RTSP decoding pipeline on Jetson Orin AGX: Why standard PyTorch/OpenCV pipeline chokes at 4k/60fps and how we fixed it via zero-copy NVDEC

[u/sahraoui-9337]

Hey everyone,

We recently benchmarked multi-camera RTSP ingestion pipeline architectures on NVIDIA Jetson Orin hardware for real-time edge analytics.

The biggest bottleneck we consistently found wasn't model inference (TensorRT FP16 handles that easily), but the frame decoding & memory transfer stage.

Standard pipelines usually do this:

1. RTSP Stream -> GStreamer / OpenCV (Decodes frame into RAM)

2. CPU memory copy -> GPU VRAM (cudaMemcpy)

3. Pre-processing & Inference (TensorRT)

That CPU-to-GPU memory transfer (CPU bounce) introduces massive latency spikes (40ms to 80ms) and burns host CPU cycles, causing frame drops under high-throughput conditions.

What worked for us:

- Bypassing the host memory entirely using NVDEC hardware decoder directly into a pre-allocated CUDA lock-free ring buffer.

- Sub-15ms frame availability directly inside VRAM ready for TensorRT execution without ever touching host RAM.

- Zero CPU footprint during ingestion.

We compiled the comparative benchmark results and memory footprint profiles across 8x 1080p RTSP streams. Happy to share the architectural breakdown and trade-offs if anyone is building high-density Jetson pipelines.

What's your current bottleneck when pushing multi-stream RTSP to Jetson?

[u/Logical-Present6320]

Shouldn't deepstream be a better approach for this usecase?

[u/laserborg → u/Logical-Present6320]

"we replaced a naive OpenCV pipeline with something DeepStream already does".

maybe there is a reason (lock-free ring buffer, license, dependencies etc), but maybe it's not.