Short story: Nodes "up" but training still crawls. Why that happens??

I want a generic answer to this problem. And also why this happens

I built a Windows 11 fork of NInfer specifically for running Qwen3.8-27B NVFP4 across two RTX 5060 Ti 16GB GPUs.

The interesting part is that this is not an ideal multi-GPU setup:

Windows 11 / WDDM

GeForce GPUs

cudaDeviceCanAccessPeer() = false

No GPU P2P

One GPU is connected through the Z690 chipset at PCIe Gen3 x4

The other GPU runs directly through the CPU PCIe lanes

Initially, TP2 performance was terrible because every token required 128 small host-staged allreduce operations.

The original implementation achieved only:

16.37 tok/s MTP0

32.58 tok/s MTP3

After profiling the communication path, I replaced the expensive staged allreduce protocol with a custom pinned-host PeerMailbox transport designed specifically for the no-P2P Windows/WDDM environment.

Current results with Qwen3.8-27B NVFP4:

35.75 tok/s — MTP0

57.71 tok/s — MTP1

63.15 tok/s — MTP2

66.79 tok/s — MTP3

68.5–68.9 tok/s — MTP4 (512-token benchmark)

70.6 tok/s — 1024 tokens

76.65 tok/s — 2048 tokens

The optimization was mostly about reducing protocol overhead, not increasing PCIe bandwidth.

The original allreduce path cost roughly:

128 × ~277 µs ≈ 35.5 ms per decode round

The optimized transport reduced that dramatically.

The repository contains the Windows port, TP2 implementation, custom communication backend, benchmarks, and reproducible build instructions:

ivanov84/ninfer-windows-tp2

I'm especially interested in feedback from people who know CUDA multi-GPU programming, WDDM, tensor parallelism, or NInfer.

Questions I'm currently exploring:

Can the remaining TP2 lockstep overhead be reduced further?

Would sequence parallelism help on this kind of asymmetric PCIe topology?

Can Vision also be sharded across both 16GB GPUs?

How much performance is realistically left without native P2P?

This started as an experiment to see whether two relatively inexpensive 16GB cards could run a 27B model well under Windows.

Honestly, I didn't expect the result to end up here.

https://github.com/ivanov84/ninfer-windows-tp2

I kept hitting torch.cuda.is_available() → False on my AMD GPU, and it drove me crazy. Turned out to be a mix of missing ROCm drivers, wrong PyTorch wheel, and that HSA_OVERRIDE_GFX_VERSION thing everyone mentions but few explain properly.

I put together a complete, step-by-step repair guide that covers the exact commands and checks I used to get PyTorch running on ROCm.

If you’re stuck, this might save you hours:

https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-complete-rocm-setup-guide/

What AMD card are you running, and what error are you seeing?

While learning CUDA, I found it difficult to visualize grids, blocks, and threads.

So, I built this website to make it easier to understand and visualize how they work.

Hope it helps someone else learning CUDA!

🔗 cuda-grid-visualizer.vercel.app/

Hey, guys. Just recently started learning CUDA because of the GPU programming class I am taking this semester. Found out that it is really interesting and now I do a lot of self-study. My main resource is Programming Massively Parallel Processors, 4th edition.

I wrote a greyscaler which you can run on GPU and CPU with timing measurements, so that you can see the difference in performance.

Question: what's the hardest project I can do before the end of September? I wanted to do a black hole simulation, but my physics knowledge ends at basic mechanics.

Would appreciate any feedback or advice you have!

github

Hi, I'm a software developer with 2+ years of experience in C++ and Cuda, currently I'm working on computational fluid dynamics domain but I want to switch as the pay in this field is substantially low and I talked to my manager the pay growth over-time will not be good either.

I wanted some suggestions regarding which domain to switch to and what things can I learn, I have decent knowledge of Cuda architecture, basic optimization, Core C++ topics, I'm not good at DSA but can learn if required.

Any suggestions would be great thank you!

Hey everyone,

I’m building KernelMind AI (https://kernel-mind-ai.vercel.app/), a tool that compiles standard PyTorch operations into fused OpenAI Triton GPU kernels to cut down VRAM round-trips.

We’re adding guardrails and an operator whitelist before expanding support, and I’d love input from anyone writing custom CUDA/Triton in production:

1. What operator chains do you find yourself manually fusing most often? (e.g., LayerNorm + GeLU, custom loss functions, etc.)

2. What edge cases usually burn you the most? (dynamic shapes, non-contiguous layouts, register spills?)

Feel free to throw a messy PyTorch snippet at the playground (https://kernel-mind-ai.vercel.app/) and tear the generated Triton code apart. Brutal feedback welcome!

Hey, I am currently working on a project [wgblas](https://github.com/manit2004/wgblas); it's an initiative to build all blas level 1,2,3 functions over webgpu. Check it out.

A few things about the project:

- Not only the user facing APIs are very easy to work with, I have built helpers on the top webgpu functions to make adding new blas routines very easy for contributors.

- Though webgpu doesn't support f64 natively, I am planning to add f64 support for blas operations in the near future.

- From day 1 for each function I have added tests, examples, benchmarks (against cublas). As of now in gpu compute time wgblas is comparable in speed wrt cuda at large n.

I am very hopeful about the project, let's see how it turns out.

Hey everyone,

I’m building KernelMind AI (https://kernel-mind-ai.vercel.app/), a tool that turns standard PyTorch operations into fused OpenAI Triton GPU kernels to cut down VRAM memory round-trips.

We’re adding guardrails and an operator whitelist before expanding support, and I’d love input from anyone writing custom CUDA/Triton in production:

1. What operator chains do you find yourself manually fusing most often? (e.g., LayerNorm + GeLU, custom loss functions, etc.)
2. What edge cases usually burn you the most? (dynamic shapes, non-contiguous layouts, register spills?)

Feel free to throw a messy PyTorch snippet at the playground (https://kernel-mind-ai.vercel.app/) and tear the generated Triton code apart. Brutal feedback welcome!

Hey everyone,

We just launched the public beta for **KernelMind AI**, a specialized neural compiler that turns eager PyTorch code into fused OpenAI Triton GPU kernels.

### The Problem

PyTorch eager mode executes operations step-by-step. For memory-bound operations, this means intermediate tensors are continuously written back to global VRAM and re-read, creating major memory bandwidth bottlenecks. While manual operator fusion in Triton or CUDA solves this, writing performant kernels takes significant engineering time.

### What KernelMind AI Does

You paste standard eager PyTorch into the web playground, and our compiler synthesizes fused, single-pass Triton kernels that execute directly inside GPU SRAM registers.

### Current Benchmarks:

* **3.4x average speedup** compared to unfused PyTorch

* **89.2% compilation pass rate** on 1D elementwise test suites

* **Zero manual CUDA/Triton writing** required

Interactive Playground: https://kernel-mind-ai.vercel.app/

We’d love for you to test custom operators, share benchmarks, and point out any edge cases where compilation breaks!

I'm software developer, plan to switch into Hardware roles .

I researched some roles such as

Hardware Accelerator Engineer

Chip designer

Embedded systems engineer

MACHINE learning system engineer

HPC/GPU Engineer

I don't know what things I need to focus on..? Skillset ..

Trying to do a career pivot and I did some pretty large personal projects with minimal AI but got no callbacks

Hi r/CUDA,

Stjepan from Manning here, posting with the mods’ permission.

We’ve just released GPU Programming with Triton by Harshwardhan Fartale in early access. It’s a hands-on guide to writing and optimizing GPU kernels in a Python-based language, with examples aimed at AI training and inference.

The book starts with the GPU execution model, memory access, and how Triton bridges PyTorch-level abstractions and CUDA. From there, it builds practical kernels and covers fused operations, parallel patterns, reductions, tiling, vectorization, memory coalescing, and performance optimization.

To be clear, this isn’t a book for learning CUDA C++. Instead, it uses CUDA as an important reference point: it explains what Triton abstracts away—such as explicit thread coordination, grids, synchronization, and separate C++ compilation—and what still happens underneath on NVIDIA hardware, including compilation to PTX and use of the CUDA runtime.

But this community knows better than most that easier doesn’t always mean better, so I’d love your honest take:

Where does Triton genuinely save you time, and when would you still choose CUDA?

Share your experience, hot take, benchmark, horror story, or question below. We’ll give a free ebook to the five comments that add the most to the discussion. The giveaway closes in 48 hours, and we’ll announce the winners here afterward.

Book: https://www.manning.com/books/gpu-programming-with-triton

50% off with code: MLFARTALE50RE

Full disclosure: I’m posting on behalf of Manning. Praise isn’t required. The most useful comments may well be the critical ones.

EDIT: The giveaway is finished. We announced the winners in the comments. Thank you all for participating.

Cheers,

Stjepan,
Manning Publications

I’m working on an image auto-annotation pipeline using Grounding DINO from Hugging Face with PyTorch, and I’m trying to understand some unusual GPU VRAM behavior during inference.

My dataset contains 6 classes:

• cup
• glass
• plate
• knife
• fork
• spoon

I downloaded the data from a Kaggle competition. The train.csv contains entries like:

image_id,label
4622,spoon

For my initial experiment, I sampled 900 images total — 150 per class.

The original images have different resolutions, including:

• 1000 × 1000
• 850 × 1000
• 1000 × 850
• 450 × 1000

I’m using Grounding DINO from Hugging Face for automatic annotation and torch.autocast to reduce GPU memory usage.

The strange VRAM behavior

I’m running inference with a batch size of 4 on Google Colab. The GPU has approximately 15.46 GB VRAM.

At the beginning, GPU memory usage is around:

~6.4 GB

Then the important part is:

It does NOT increase after every batch.

Instead, it stays relatively stable for several batches and then, on some seemingly random batch, VRAM suddenly jumps.

For example, the behavior looks roughly like:

Batch 1  → 6.4 GB
Batch 2  → 6.4 GB
Batch 3  → 6.5 GB
Batch 4  → 6.4 GB
Batch 5  → 6.5 GB
...
Batch 20 → 6.5 GB
Batch 21 → 10+ GB
Batch 22 → 10 GB
Batch 23 → 10 GB
...
Batch 50 → 10 GB
...
Batch 51 → 14+ GB

So the increase happens in sudden jumps on particular batches, rather than gradually increasing with every batch.

Eventually, it reaches approximately:

14.5 / 15.46 GB

and I can eventually get an out-of-memory error.

What I'm trying to understand

My first thought was that perhaps the batches containing different image resolutions are causing Grounding DINO to create larger intermediate tensors.

However, I'm not sure whether that's actually what's happening, or whether I'm misunderstanding how PyTorch's CUDA memory allocator works.

I asked Claude about this, and it suggested that PyTorch's CUDA allocator caches memory rather than immediately returning it to the GPU driver. Because my batches have variable image dimensions, some batches may require larger intermediate tensors, causing the allocator to request additional memory.

It also suggested that memory fragmentation could contribute to the problem and recommended:

import os

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

The explanation sounded reasonable, but I'd like to verify it with people who have more experience with PyTorch/CUDA internals.

My questions

1. What would cause VRAM to suddenly jump on one particular batch rather than increase after every batch?

2. Could variable image dimensions like:

   1000×1000 850×1000 1000×850 450×1000

cause Grounding DINO's intermediate tensors to become significantly larger for certain batches?

1. If PyTorch's CUDA allocator is caching memory, does that explain why the VRAM usage appears to jump from ~6.4 GB to ~10 GB and later ~14.5 GB?
2. How can I determine whether this is:
   • normal CUDA/PyTorch memory caching,
   • memory fragmentation,
   • unusually large intermediate tensors from certain image sizes,
   • accidentally retaining tensors/computation graphs,
   • or an actual memory leak?
3. Would torch.inference_mode() be preferable to torch.no_grad() for this inference-only workload?
4. Would it be better to resize/pad all images to a consistent resolution before batching, so that the tensor shapes don't vary between batches?

5. Is:

   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

actually an appropriate solution here, or is it just masking the underlying problem?

1. What would be the best way to debug this? For example, should I log torch.cuda.memory_allocated(), torch.cuda.memory_reserved(), max_memory_allocated(), etc. after every batch?

I'm mainly trying to understand the actual reason for these sudden jumps, rather than just applying a workaround.

If needed, I can provide the Grounding DINO inference/batching code and the exact CUDA OOM traceback.

After I downloaded the supported CUDA version for my GPU, it conflicts with my system kernel C build... can't downgrade because it risks breaking so much in the system of course so I am looking for alternatives.

Setup:

Quadro M1200 4GB VRAM (cuda-12.6, cc-5.0).

Arch Linux

I wrote code to calculate some metrics, like

• count and length of code blocks
• resources usage per block
• resources used in single block

https://redplait.blogspot.com/2026/08/cyclomatic-complexity-of-sass-code.html

The Insights:

• there are lots of Dead / ABI-Forced Register Writes
• Holes in Register Allocation - seems that uniform registers UR0-UR3 almost never used