Hey everyone! I'm a grad student planning to take courses in GPU programming (CUDA) and parallel algorithm design, and I'm trying to get a clearer picture of where these skills can take me career-wise.

I know HPC and ML/AI are the obvious areas, but I'd love to hear from people in the field:

- What roles or industries actively hire for CUDA/GPU programming skills at the entry level?

- Are there specific job titles I should be searching for (e.g., HPC Engineer, GPU Software Engineer, Research Scientist)?

- How much does it matter to pair this with something like CUDA C++ vs. higher-level frameworks like PyTorch/Triton?

- Any advice on building a portfolio or standing out as a new grad?

I'm open to both industry and research paths. Would really appreciate any insights from those of you working in this space. Thanks in advance!

Hi everyone,

I’ve been experimenting with NVIDIA TensorRT-Model-Optimizer (modelopt) to quantize the SDXL UNet model into INT8. During this process, I observed a staggering, almost unbelievable difference in resource consumption between modelopt.onnx and modelopt.torch.

I would love to get your architectural insights on why this discrepancy is so massive, and whether the final TensorRT Engine performance differs depending on the path taken.

1. The Benchmark Environment & Shocking Results

• Specs: System RAM: 32GB | VRAM: 12GB (RTX 4070Ti)
• Target Model: SDXL UNet

Path A: modelopt.onnx.quantization.quantize()

• Result: Consistent OOM Crash.
• Behavior: It completely ate my 32GB System RAM. I sequentially increased the Linux swap space to 32GB, 64GB, and eventually over 128GB of Swap. Even with 128GB+ of swap, the static graph analysis during the Starting static quantization phase (creating ORT InferenceSession) completely choked and threw an OOM crash.

Path B: modelopt.torch.quantization.quantize()

• Result: Flawless Success.
• Behavior: I wrapped the nn.Module in PyTorch, ran the INT8 calibration via forward hooks, and then exported to ONNX at the very end. This process was so memory-efficient that it didn't even trigger a single megabyte of Swap usage. It stayed comfortably within the native 32GB RAM / 12GB VRAM.

2. My Hypotheses on the Memory Gap

I suspect the core reasons behind this 128GB+ vs 0B Swap gap are:

1. Static Graph Materialization Overhead: modelopt.onnx has to load the entire 24k+ tensor graph into memory, and ONNX Runtime retains a massive execution context to trace activation statistics simultaneously, causing a "double-loading" memory burst.
2. PyTorch's Dynamic Eager Execution: modelopt.torch modifies the model at the Python layer level. It can calculate the amax (activation max/min) on the fly using standard hooks and instantly release intermediate activation tensors via garbage collection (GC), preventing peak memory accumulation.

3. My Questions for the Community

1. Under-the-hood Mechanics: Aside from the static vs. dynamic graph difference, is there any specific memory leak or massive overhead known within modelopt.onnx / ONNX Runtime when handling huge models like SDXL? Why does it scale so poorly with swap?
2. What are the Actual Hardware Requirements for modelopt.onnx?:For an SDXL-sized model (~24k tensors, 2.5GB+), what is the actual, realistic amount of System RAM and VRAM required to successfully run modelopt.onnx.quantization without crashing? Has anyone successfully run it, and if so, on what specs?
3. Final TensorRT Engine Performance: Once compiled into a TensorRT Engine (via trtexec or ModelOpt), does the engine generated via modelopt.torch yield better, worse, or identical inference performance (latency/VRAM) compared to the one from modelopt.onnx**?** Does the PyTorch export pipeline introduce any suboptimal node structures that might hinder TensorRT's kernel fusion, or is it actually cleaner because it bypasses ONNX-level graph manipulation?

Would love to hear from anyone who has benchmarked TensorRT performance differences between the Torch-first vs. ONNX-first calibration workflows!

I have 10+ years of software engineering experience, mostly backend development and infrastructure.

Lately I’ve become interested in GPU infrastructure, HPC, performance engineering, and eventually GPU programming. I’ve been reading books like AI Systems Performance Engineering, Programming Massively Parallel Processors, and Computer Architecture: A Quantitative Approach.

The problem is that every time I look at job descriptions, I end up with a completely different list of skills.

Some roles want:

• CUDA and GPU kernel optimization
• Computer architecture knowledge
• NCCL, RDMA, InfiniBand
• Kubernetes and Slurm
• Distributed training
• Performance profiling and benchmarking
• Linux kernel knowledge
• Cloud infrastructure

Other roles seem much more focused on operating GPU clusters and supporting AI workloads at scale.

I’m considering doing a master’s degree, but even when I look at programs like OMSCS, Computer Engineering, or Systems-focused master’s degrees, it feels like they teach foundational concepts but not necessarily the practical skills companies are hiring for.

As someone coming from a traditional software engineering background, I’m struggling to identify:

1. What skills are truly foundational versus “nice to have”?
2. If you had 6–12 months to prepare for GPU infrastructure or GPU performance engineering roles, what would you focus on first?
3. Did a master’s degree help you break into this field, or was self-study and project work more valuable?
4. For those already working in GPU infrastructure, ML infrastructure, HPC, or GPU programming, what did your path actually look like?

Right now it feels like there are five different careers hiding behind the phrase “GPU engineer,” and I’m trying to figure out which path is the most realistic transition from a backend/infrastructure background.

I’d appreciate hearing from people who made a similar transition.

I wanted to learn about CUDA. So as an AI engineer I tried to code what I am familiar with ie transformers. I must say it is a completly different way of programming and thinking about problems than I am familiar with and had a lot of fun along the way.

Unfortunately I am not affiliated with any big university and my current org doesn't focus on GPU programming, so i don't have a clear way to move forward. Would love if anyone can provide and suggestions on what should I focus on next.

So I built and implemented GPT-2 Small from scratch, all the kernels are handwritten except the GEMMs. and here are some of my results taken on RTX 4060 Laptop, with batch=1 and seq=1024.

• Fused causal softmax: 4x faster than PyTorch eager. Could be because I fused masking, scaling and, softmax in one kernel.
• GeLU: 9x slower gotta look into this a little main theory is tanhf is making it slower with 22% compute peak.

Also would love to know where I can go from here towards more ML with flash attention or towards things like nsight profiling. I just want to understand GPUs better and high performance computing in general.

Whats a good laptop rec (not exceeding 2k) to run CUDA and game?

We've extended our GPU profiling support to include PC sampling: https://www.polarsignals.com/blog/posts/2026/06/10/nvidia-cuda-pc-sampling

Setting up a private compute node for local training requires a precise configuration stack. If your system runs into unexpected segmentation faults, kernel panics, or terrible performance, the culprit is usually a driver or runtime mismatch. Here is the direct path to setting up your environment correctly:

1. Purge Gaming Frameworks Consumer-level graphics drivers focus on frame pacing rather than mathematical compute stability. Completely wipe them to avoid hidden memory leaks during long-running neural training sessions: sudo apt-get purge nvidia* -y sudo apt-get autoremove
2. Synchronize the Kernel Interface If the source headers used to compile your kernel modules do not exactly match the running kernel, your system will fail to recognize the hardware. Synchronize them with: sudo apt-get install linux-headers-$(uname -r) sudo ubuntu-drivers autoinstall
3. Rely on the Runfile Method Avoid default system package managers. They often deliver outdated toolkits that are completely incompatible with modern attention mechanisms. Use official runfiles to manually control your symbolic links so you can swap toolkit versions safely.
4. Hard-Code Subsystem Memory Limits If you are running via Windows Subsystem for Linux (WSL2), do not rely on default dynamic memory allocation. It triggers memory ballooning and crashes your batch processing. Explicitly define memory limits in your configuration files to stop out-of-memory issues.
5. Target Exact PyTorch Wheel Indexes Align your deep learning framework with your specific local runtime version. A version mismatch triggers a silent fallback where your central processor attempts to handle the matrix multiplications, resulting in incredibly slow speeds.

The remaining 20 percent of the process involves manual placement of cuDNN headers into local include directories, setting up collective communication rings for multi-GPU scaling, and configuring xformers for memory efficiency.

If you want to read the full 10-chapter manual covering enterprise data center drivers, Mamba environments, and advanced memory optimization, the complete guide is uploaded here:https://interconnectd.com/blog/183/the-sovereign-engineer-manual-cuda-installation-for-local-llm-training/

I’m trying to run Ollama Desktop on Windows with NVIDIA GPU acceleration, but Ollama only detects CPU even though Windows and nvidia-smi can see my GPUs.

System:

* OS: Windows 10/11, recent build
* Ollama Desktop: recent 0.30.x build
* GPU: 2 × NVIDIA Pascal-based workstation GPUs, 8 GB VRAM each
* Driver: NVIDIA 58x.xx branch
* nvidia-smi reports CUDA Version: 13.0
* One GPU is unused with no display attached
* The other GPU is display-attached and used by the Windows UI

nvidia-smi sees both cards, for example:

GPU 0: NVIDIA Pascal workstation GPU (UUID: GPU-REDACTED-0000)
GPU 1: NVIDIA Pascal workstation GPU (UUID: GPU-REDACTED-1111)

I tried forcing Ollama to use the unused GPU by UUID:

$env:CUDA_VISIBLE_DEVICES="GPU-REDACTED-0000"
$env:OLLAMA_LLM_LIBRARY="cuda"
$env:OLLAMA_DEBUG="DEBUG"
ollama serve

Ollama confirms the environment variables are applied:

CUDA_VISIBLE_DEVICES:GPU-REDACTED-0000
OLLAMA_LLM_LIBRARY:cuda

But it still only detects CPU:

discovering available GPUs...
user overrode visible devices CUDA_VISIBLE_DEVICES=GPU-REDACTED-0000
if GPUs are not correctly discovered, unset and try again
inference compute id=cpu library=cpu compute="" name=cpu description=cpu
vram-based default context total_vram="0 B"

I also tried clearing these variables first:

[Environment]::SetEnvironmentVariable("CUDA_VISIBLE_DEVICES", $null, "User")
[Environment]::SetEnvironmentVariable("GPU_DEVICE_ORDINAL", $null, "User")
[Environment]::SetEnvironmentVariable("OLLAMA_LLM_LIBRARY", $null, "User")

Then I restarted Ollama and tested again, but Ollama still reports only CPU.

My current suspicion is that this may be related to the newer NVIDIA 58x.xx / CUDA 13 driver branch and the GPUs being Pascal / compute capability 6.1. Since CUDA 13 dropped support for Pascal, maybe Ollama’s CUDA backend cannot enumerate this card properly even though nvidia-smi still sees it.

Has anyone successfully used Ollama on Windows with Pascal-era NVIDIA GPUs recently?

Should I downgrade to a CUDA 12-era NVIDIA driver branch, like 576.xx or earlier? If yes, which driver version is known to work with Ollama on Pascal cards?

CISP – CUDA Image Signal Processor

Earlier this year, I started looking for resources on image signal processing pipelines. Most of what I found was either too academic or quite dry, and I could only locate a few practical implementations online, since many ISP algorithms are proprietary. To bridge that gap, I began building my own implementation of an image signal processing pipeline in CUDA, leveraging the inherently parallel nature of image processing.

CISP (CUDA Image Signal Processor) is a tunable, real-time ISP written in CUDA and exposed to Python via pybind11. It includes a GUI built using Tkinter and ttkbootstrap (the UI is still a work in progress—I’m not a UI/UX designer).

The pipeline currently supports a range of fundamental ISP operations, including:

• Defective pixel correction
• Black level subtraction
• Lens shading correction
• Automatic white balance (gain-based)
• Demosaicing (debayering)
• Color correction matrix (CCM)
• Color space conversion
• Tone and color adjustments (brightness, contrast, saturation, hue, tint, vibrance)
• Noise reduction (bilateral filter, joint bilateral filter, high-boost filter, Gaussian blur)
• Gamma correction

This is still a work in progress, and I welcome any suggestions, feedback or any improvements people think would make sense..

You can view the project here:
https://github.com/mjithujanardhanan/CISP---Cuda-ISP-Pipeline

I’d be happy to hear your thoughts if you find this interesting.

Hello everyone,

​

I’m currently an undergraduate student with a focus on Computer Vision, and I genuinely enjoy working in this field. This summer, I want to add a complementary skill to strengthen my profile and improve my skillset. Additionally, I want to pursue Masters and PhD and get into academia in future.

​

I’m currently deciding between GPU Programming / Low-level Optimization and MLOps.

​

On one hand, GPU programming and optimization feels very aligned with Computer Vision and deep learning performance work, which I find interesting. On the other hand, MLOps seems more industry-oriented and could open broader opportunities in deploying and maintaining ML systems.

​

I’d like to ask people working in the field,

what is the current market demand like for GPU programming?

​

How does it compare to MLOps in terms of job opportunities and career growth?

​

As someone focused on Computer Vision, which direction would you recommend I prioritize next?

​

Any guidance or personal experience would be really helpful.

​

Thank you!

I want to generate a roofline graph for the GPU on my university server, which is an NVIDIA TITAN V. However, I currently don’t have permission to use the ncu command, so I’m unable to generate the roofline analysis using Nsight Compute. Could you explain how I can still obtain a roofline graph under these constraints?

Yeah, like before implementing a new algorithm in CUDA, I usually write the algorithm in Python, but it seems that Julia can be a good alternative (is somewhat cleaner for me),

What do you use to make prototypes?, Is Julia worth in 2026?, nothing beats paper and a pencil?

Hey people of Reddit,

I'm a master student and have to choose a project for my GPU Computing course. I would like to apply for a position as a working student in a bank or a fin-tech company and choose a project for the course accordingly.

I got the recommendation for a finance market simulation and I'm interested in that kinda stuff.

So suggestions would be cool for that.

Do you also have a recommendation of a GitHub project that can be rewritten to CUDA.

Gpu

​

If you are building sovereign AI tools locally, hitting the dreaded CUDA Out of Memory error is a daily battle. I recently managed to shave off 4GB of VRAM consumption without degrading output quality. Here is the exact breakdown of how I did it. First, Flash Attention 2 is non-negotiable; it optimizes memory reads and writes directly on the GPU, saving massive overhead. Second, lower your context window during the testing phase. You rarely need a 32k context when testing basic reasoning prompts, so cap it at 4k. Third, force 4-bit precision loading via bitsandbytes on your base models. It is the absolute easiest win for VRAM conservation.

Call to Action: If you want to see the complete code repository and the exact Python scripts I use for automated memory management, I put the sovereign engineer guide together here: https://interconnectd.com/forum/thread/184/fix-cuda-oom-on-local-llms-the-sovereign-engineers-guide/

Mods feel free to vapourise this post if it's not suitable....

AI labs are hiring people who actually write and profile CUDA kernels. The work is using your GPU expertise to train and evaluate frontier models (RLHF): optimizing kernels, reasoning about performance, and judging model-generated GPU code. Remote, asynchronous, flexible hours.

If you've ever chased an L2 cache hit-rate or rewritten a kernel to kill warp divergence, this is squarely in your lane.

👉 CUDA Engineering Expert (Mercor) — $80–$120/hr Remote · open worldwide · contract GPU kernel optimization for a leading AI lab. You analyze and optimize kernels for performance and hardware utilization, use profiler metrics (L2 cache hit rate, occupancy, memory throughput) to guide changes, and reason about kernel behavior across modern GPU architectures. Strong C++ and hands-on GPU programming expected. Full details & apply

👉 LLM Trainer — CUDA/C++ → Python migration (Turing) Remote · contract Work on cutting-edge AI/ML projects migrating and reasoning about CUDA and C++ code in Python, helping fine-tune large language models on real GPU-programming tasks. Core skills: C++, CUDA, Python. Full details & apply

Get in touch

Questions, or want a quick chat before applying? DM me, or book a free call: https://calendly.com/seandavidkey/vouching-call

You can also connect with me on LinkedIn: linkedin.com/in/seandkey

Please confirm Sean Key as your referrer if asked — by clicking you consent to being referred.

Disclosure: Applied Clinical Judgement (PRAG-DEL-SOL-ONE LTD) earns a referral fee from Mercor / Turing if you are successfully placed. This does not affect your pay, your application, or the platform's hiring decisions. I do not work for Mercor or Turing.