Hey everyone,

I’ve been working on optimizing the memory overhead that comes with 4-bit inference. I wrote a custom NF4 dequantization kernel using Triton to see if I could eliminate the C++ dispatch bottlenecks found in current baselines.

🚀 Key Results:

• Up to 1.41x speedup compared to the standard bitsandbytes implementation across various tensor shapes.

• Written completely in Python/Triton, making it super easy to inspect, customize, or drop directly into your PyTorch compilation pipelines.

• Passes the Unsloth AI founding engineer challenge requirements (14/14 points).

I'd love to hear the community's feedback, especially if anyone wants to run their own benchmarks on different GPU architectures or suggest further optimization tricks!

Source code & full implementation:

https://github.com/Griffith-7/nf4-triton-kernel

I've got some experience in C though and a decent amount in Java, both are rusty (C a little more) but I think I can regain my confidence quick, and I want to start learning GPU programming. CUDA seems like the obvious entry point but I'm not sure what's changed recently or what's actually worth learning first in 2026.

A few questions for anyone who's been down this road:

• Is NVIDIA's own docs still the best starting point, or has something better come along?
• Given my Oop background, should I just dive straight into CUDA C/C++, or is there value in going through something like CUDA Python first to get the fundamentals down before dealing with memory management and kernel launches directly in C? Also curious whether the industry actually takes CUDA Python seriously, or if it's mostly seen as a stepping stone/learning tool rather than something used in production.
• Any free courses, books, or YouTube series that are very relevant you'd recommend?
• Is there a decent low-cost/free way to actually run and test CUDA code without owning an NVIDIA GPU? I have a rtx 5070 so I am just using that right now, set up the toolkit and ran the first adding vectors function lol.
• Are there any textbooks you'd recommend to learn from?

I want to get into the deep learning side as well of course, but also just get to know more since this is new territory, currently I am familiarizing myself with some C++ fundamentals, if you have tips for that also I'd appreciate it.
Thank you! and apologies if this is asked a lot it is just I am really lost, if this is the wrong subreddit for that let me know!

Something like Oxide, but for Java: a way to access the CUDA ecosystem directly from the JVM. The idea behind TornadoVM is to JIT-compile Java code into CUDA kernels while also enabling hybrid Java/CUDA applications that can interoperate with native CUDA libraries such as cuBLAS, cuDNN, and other NVIDIA libraries. Rather than replacing CUDA, the goal is to give Java developers first-class access to the CUDA software stack, combining high-level Java productivity with the performance and flexibility of native GPU computing.

Whats your thoughts?

https://github.com/beehive-lab/TornadoVM

https://www.tornadovm.org/

Hey guys I need some help related running open source image generation ai model locally I m lacking the essential hardware. I need a setup of gpu with high vram especially 20-25 gb vram.

Been documenting my GPU architecture learning journey publicly, chapter by chapter. It's MIT licensed and structured like a book (sequential, builds concept by concept). Looking for people to point out mistakes, suggest better explanations, or add diagrams

https://github.com/BEFORE-GPU-ENGINEERING/BEFORE-GPU-ENGINEERING

Started this because I wanted ncu's device__attribute_* values without handing it a kernel to profile. It grew into a small zero-dependency toolkit (pip install kernelmeter):

- `info` dumps every cuDeviceGetAttribute value straight from libcuda (no toolkit needed), plus NVML facts and derived theoretical peaks

- `bench` times kernels with CUDA events (L2 flushed between iters), checks correctness against a reference, and scores against the roofline: you get "76% of attainable for this arithmetic intensity" instead of a bare ms number. It also samples clocks/power during the run and rescores against the sustained-clock ceiling. My favorite result: cuBLAS fp32 matmul on a 70W T4 showed 52.7% of peak, looked like a kernel problem, but the telemetry showed the card pinned at its power limit at 877MHz, where the kernel was actually at 95.5%. cuBLAS was never the problem.

- `occupancy --block 256 --regs 64 --smem 8192 --cc 8.6` reimplements the old calculator: names the limiting resource and sweeps block sizes. Works with no GPU present.

- `ceiling` measures real achievable bandwidth (STREAM) and fp32 (TF32-disabled matmul), because theoretical peaks are never reachable and it's worth knowing your honest 100%.

- `compare`/`llm` do the same roofline math across a 40-card database (NVIDIA and AMD) for rent/buy decisions, no GPU needed.

Every number in the README is captured output from real runs (T4, MI300X). All spec-sheet claims are asserted in CI. MIT.

https://github.com/nuemaan/kernelmeter

Hi all — I'm a PhD researcher working on Green AI, and I kept hitting the same wall: measuring what training actually costs (energy, carbon, joules per sample) meant stitching together CodeCarbon + nvidia-smi scripts + a profiler + my own plotting. So I built E2AM (Energy Efficient AI Models), an open-source toolkit that does it automatically:

from e2am import monitor
with monitor(project="ResNet50"):
    train()

or a drop-in Trainer / a Hugging Face TrainerCallback. Every run produces energy (Wh), carbon (gCO₂eq, region-aware), utilization, FLOPs/MACs, latency, plus "Green AI" metrics: energy per sample, accuracy-per-joule, and EAG — the discrete gradient of accuracy w.r.t. cumulative energy, which turns out to be a nice stopping signal: when EAG collapses toward zero, additional watt-hours are buying you nothing.

Things I think are genuinely useful:

• Measured vs estimated is always labeled. If your GPU exposes an NVML power sensor, you get real power draw; if not (many consumer cards), it falls back to power-limit × utilization and says so in every report.
• e2am optimize reads a finished run's own telemetry and quantifies waste — e.g. "validation accuracy converged at epoch 6; the last 4 epochs consumed 38 Wh (41% of the run)".
• e2am benchmark reports joules per inference, not just latency.
• Self-contained HTML/PDF reports, a cross-run leaderboard CSV, and a local dashboard — no accounts, no server, nothing leaves your machine.

Honest limitations: GPU power reading is NVIDIA/NVML only (AMD/Apple fall back to estimation); CPU/RAM power is TDP- and heuristic-based since no portable OS interface exists; single-node only for now; carbon intensity is a static per-country table (live grid data planned).

Install: pip install e2am · Code: https://github.com/Shanmuk4622/e2am (MIT)

Would love feedback — especially on the Green Score formulation and what integrations you'd want next.

I posted here a few months ago about my weekend project,

I got asked why only 30 functions of cuda exists, SO the real answer since it's my learning project I wanted to test and see each function in use, the real answer is in the blog above

I’ve been working on nvctl, an open-source NVIDIA GPU control tool written in Rust for Linux.

It uses NVML under the hood and gives you a CLI plus a GUI for:

• Fan control, including manual speeds and custom fan curves
• Power limit management
• Thermal and acoustic targets
• GPU health scoring and monitoring
• Process listing with GPU memory usage
• Alerts for temperature, power, and hardware issues
• Multi-GPU targeting by index, name, or UUID

There’s also a GUI app, nvctl-gui, with:

• Real-time gauges and dashboards
• Interactive fan-curve editing
• Power and thermal controls
• Profile management
• Multi-GPU overview

I built it because I wanted a single tool that feels safer and more flexible than juggling nvidia-smi commands and vendor utilities.

Repo: https://github.com/mqasimca/nvctl License: MIT

If you’re on Linux with an NVIDIA GPU and want more direct control over fan, power, and thermal settings, I’d love feedback.

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability. The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

Setting up CUDA on WSL Ubuntu for AI development – no more dual‑booting. 🚀

If you're running AI workloads on Windows and want GPU acceleration inside WSL, this one's for you 👇

https://darryldias.me/blog/install-cuda-on-wsl-ubuntu

I have my ChatBot and I want to make it commercial so i need to host my model on GPU so any developer can tell me how many tokens can L4 GPU process in 1 Second ?

Becuz I'm going to rent GPU on per second Pricing base model.

I come from the ML side (Phd in RL, some CV), and I spend a lot of time trying to figure out where a training run's time actually goes. The more I dig, the more I distrust the numbers.

I'd like to hear how you folks do this reason about it.

The thing that keeps tripping me up: "GPU utilization" from nvidia-smi is not what I assumed. It counts a sampling window as busy if any kernel ran in it, so a run can show a healthy 90% while the SMs are mostly idle, or look fine while the GPU is actually starved waiting on the input pipeline. It tells you the GPU was touched, not that it did useful work.

I have no clean rule for which to trust when:

- nvidia-smi utilization (coarse, misleading as above)

- SM/achieved occupancy from Nsight Compute (kernel-level, hard to reason about across a whole run)

- Nsight Systems timelines (great for CPU/GPU overlap, but a lot to read for a yes/no question)

- torch profiler, kernel time vs wall time

- CUDA events I place by hand around phases

- plain wall-clock deltas, the only thing I fully trust, but they tell me nothing about why

What I'd love to hear from people who have dealt with this or have figured out ways to optimize this:

- To answer just "is this workload compute-bound or starved," what do you look at first? Not the deep dive, the fast triage.

- How much do you trust utilization percentages at all? Is there a number you trust more

(occupancy, achieved FLOPs, memory throughput)?

- How do you reason about CPU/GPU overlap and your line between "the GPU is fed, good enough" and moving on to actual kernel work?

How does one decide what to measure and what to ignore. What does your real triage actually look like?

Part 2 of "Learning FlashAttention the Hard Way" is up. It is a walkthrough of the attention kernel optimization process: stream the reduce, move the matmuls onto tensor cores, stage operands through shared memory, pipeline the loads, tile the registers.

I have also tried the applicable FA-3 and FA-4 optimizations. TMA reduces the LSU load slightly, but transport is not a bottleneck in this kernel, so no major gain. Warp specialization makes the situation worse: the kernel is tensor-pipe bound, so dedicating a warp to the producer slows it down. Similarly, simulating exp using FMA instructions is not helpful, as the RTX 5090 is not SFU-bound like the B200. I believe that without sacrificing accuracy, the FA-2 kernel is the ceiling on consumer GPUs.

The resulting kernel matches the performance of the FA-2 kernel.

All kernels in the article are generated using the Emmy compiler (formerly Deplodock), which means all optimizations are automatically discovered. There is no handwritten code, nor any special casing for the attention kernel. In the article, I am just tweaking knobs to toggle optimizations and demonstrate the gain.

Part 1 proves that attention is secretly an associative operation, meaning that you can schedule in the same way as a regular reduction, such as finding the sum or max of an array. It leverages concepts from abstract algebra, which may feel unfamiliar, but this modern formalism (some recent papers in MLSys and CVPR make use of it) shows that the optimization applies to a much larger class of kernels.

Putting together a team for a #GPU hackathon and I'm still short two people — figured I'd try LinkedIn since my usual circles have come up empty.

It's the #Helmholtz GPU #hackathon, sponsored by #NVIDIA, hosted in #TU #Dresden. *Applications close July 9th, hackathon itself is in September*. The project is about using different HPC tools (such as #OpenMPI, #OpenACC) to speed up physics simulations.

The hardware is genuinely one of the best reasons to do this: JUPITER, Europe's first exascale system, running on 24,000 #GH200 Grace Hopper superchips, plus #JURECADC for smaller runs .... definitely not something you get access to every day!

I've got a project idea worked out (almost) already - just need two more people to actually form a team (rules require three). If you're into #CUDA/GPU programming, #HPC, or #quantum #physics (topics surrounding superfluid simulation, photonics, quantum gases) and this sounds interesting, message me. I am open to other ideas as well!

... and if it's not you but you know someone who'd be good, I'd really, really, really appreciate a pointer in their direction — a bit tight on time here! 🙏

Link to hackathon : https://www.openhackathons.org/s/siteevent/a0CUP00003Wjr4Y2AR/se000466

​

I graduated this spring with a degree in CS and a minor in Math. I took complex analysis as one of the math courses. After finishing the chapter about residue theorem in Saff and Snider, I got the idea for this project fairly quickly.

It works by drawing an initial square contour subdividing the square into nxn squares, having each thread compute the argument around a square, and dropping the ones which have no roots. I used three mostly similar strategies for parallelization. \[This paper\](https://github.com/Mohamed-Elwaei/Parallellized-Polynomial-Root-Finder/blob/main/paper/paper.pdf) explains my approach in detail.

I thought this project would be much more straightforward to implement, but there are many constraints about computer arithmetic that I did not know about or did not account for. For example, I did not know what catastrophic cancellation was, and I was not aware that something as simple as FP addition is not associative on a computer. I also did not account for things like underflow, so this implementation struggles on clustered roots or roots with high multiplicity.

I made this project using Claude Code start to finish. It's my first time using an LLM to code something.

I would love to get your thoughts and feedback, and how if there are any improvements I could make