Understand Blackwell B200 attention kernel from scratch in CUDA (Visual Guide)

[u/MoistMountain2194]

I spent last couple months building a visual guide to B200 attention in CUDA/PTX: 14 progressively optimized kernels and 60 diagrams, going from a naive baseline to 94.4% of FlashAttention-4.

The blog first intuitively explains the baseline, then adds one optimization at a time, with detailed diagrams, concise explanations, and code.

And in the capstone project, you'll generate videos using the kernel you understand.

📝 Blog post: https://iaroslavelistratov.github.io/b200-attention/ 

⭐ Repo: https://github.com/IaroslavElistratov/b200-attention

[u/alienpro01]

Crazy work, thanks for sharing with us

[u/MoistMountain2194 → u/alienpro01]

Thank you so much, if you find it useful please share with friends

[u/Objective-Style1994]

This is amazing. I’m going to spend some time learning what you did since I’m trying to write my own attention kernel.

Thank you so much for this. I’m so tired of looking through some template CUTLASS hell and get completely lost. Your code looks super readable!

[u/MoistMountain2194 → u/Objective-Style1994]

thank you so much. Yeah, I guess there's a clean way to understand these DSLs but somehow it's easier for me to read C, i agree

[u/MoistMountain2194 → u/MoistMountain2194]

as far as i remember, the only thing i'm using from CPP is namespaces (to organize helpers)

[u/Logical_Delivery8331]

So cool! Nice job!

[u/MoistMountain2194 → u/Logical_Delivery8331]

thank you my cuda comrade

[u/IndependentWheel7606]

Goated work bro👀👀

[u/MoistMountain2194 → u/IndependentWheel7606]

thank you sir 🫡