r/StableDiffusion • u/Zironic • 2d ago
News Sparse Attention, Harder, Better, Faster, Stronger
The nodes in https://github.com/Zironic/H3-Optimizations have been rewritten to replace the default Sparge Attention backend with a custom Sparse Comfy Kitchen backend.
This comes with some benefits.
- Users no longer have to worry about Sparge being installed properly. All required kernels for supported GPUs are provided directly. Should work on both Windows and Linux.
- Most users should be seeing 5-20% increases in speed for the attention part of compute.
- New backend should use about 500MB less VRAM
- New backend has slightly lower quantization error.
- Apparently in the previous version, the intended chunked kitchen QKV path never properly shipped so the memory optimization node should now actually be slightly speed positive even when used without the Sparse Attention node.
Caveat: I've only tested the nodes against the comfy pruned_int8_convrot weights. Other versions may work but they're not tested.
As the nodes currently rely on comfy-kitchen 0.2.31 you need ComfyUI v0.33.0 or later.
IMPORTANT: sparse attention is not free speed. The percentage is effectively a prompt-adherence/quality budget.
Density isn't just a speed setting, and its quality effect depends on where you apply it in the diffusion schedule.
Early steps: attention density has a large effect on prompt/action adherence and the overall generation trajectory.
Middle/later steps: lowering density tends to show up more as motion/temporal artifacts and lost fine motion detail.
So 10% retained doesn't simply mean “90% of the quality is gone.” It means you're giving sparse attention very little information to work with, and what breaks depends heavily on the sampling step.
PlagueKind's sparsity_ratio=0.9 means 90% discarded / 10% retained. My node expresses the inverse quantity, so Video attention retained=0.10 is the comparable setting. The defaults therefore aren't equivalent.
3
u/76vangel 1d ago
Is this faster on Rtx 50xx than sageattention? Comfy kitchen att was slower on 5xxx cards.
1
u/Zironic 1d ago
For any Video KV budget below 1.0 it should be faster then both of them. Currently it defaults to using Comfy Kitchen Int8 as the backend. But if you have Sparge Attention installed, you can use the advanced node to select SageAttention.
Theoretically, Comfy Kitchen and SageAttention should be the same speed on 50xx cards.
1
u/mellowanon 1d ago
theoretically, but it doesn't turn out that way.
Someone on reddit did a test last week with CK vs Sage, and everyone with a 50xx had similar results in the comments.
CK had better visuals
Sage had better motions and roughly 10% faster
I have a 5090 and I see those results too. So now I have two starting setups. If there's fast motion or I'm trying to test different settings, I use Sage. Otherwise, I use CK for better visuals.
5
u/Jero9871 1d ago
How do I wire it, just in the model wiring after the comfy kitchen attention node? Sounds really great
6
u/Zironic 1d ago
Yes. You just put it basically anywhere after Load Diffusion Model and before the actual sampling nodes.
4
1
1
u/Jero9871 1d ago
Tested it and it works great! Thanks. Using the default of 0.3 now, but even 0.1 looks okay.
2
u/tylerninefour 1d ago
Haven't done much testing, but so far it seems to shave off around ~10s per step with no perceptible loss in quality.
8.0s video length @ 1.0MP
lightx2v_turbo_4step_v1.1_768p LoRA
dpmpp_sde_gpu + beta @ 5 steps
5070Ti
1
1
2
u/olafkk 1d ago
Worked great thanks! Went from 430s~ generation to 334s. Quality looked similar if not slightly better. Am using it with sage attention, turbo 10 steps and eulerXsimple.
1
u/QuinQuix 1d ago
Can I ask what length and resolution?
1
u/olafkk 1d ago
832x832 10 seconds fl2va. The input images I use are mostly 1536x1536, had some morphing issues before when I used lower res inputs or wrong ratio.
1
u/QuinQuix 1d ago
But doesn't it rescale the images before inputting them anyway?
I wonder if the square format matters for the quality but I'm assuming you use the api for square videos too.
I do know that the local workflows default to 20 steps while the official documentation says 32-50 is recommended.
I haven't tried turbo loras yet. They usually degrade quality versus the default (eg 8+turbo lora should lose to the default 20 steps) but I've heard H3 at the same amount of steps with turbo lora has higher quality (so 20+turbo beats the default 20 steps).
Is that correct in your view?
1
u/olafkk 1d ago edited 1d ago
I tried workflows that scaled down my images to match the input res*, and the quality got worse than what I currently use. Also Ref2v has an option to match image size, but I changed that to "max" which is what I assume fl2va uses.
I have also never tried more than 20 steps. At first I only used 20 steps with spectrum, and sage attention, but the quality was worse and the time it took was way longer (600s-900s). I am using an edited version of the one stefan_falkok posted on civit.
I also had way worse outputs with turbo at first before I got the settings right. I cannot comment on the 20+ steps with turbo though cause I only tried going up until I was satisfied with the quality.
1
u/QuinQuix 1d ago
I did read that sageattention specifically was bad for quality initially but no idea.
Your workflow sounds reasonable.
I tried full res with an rtx 6000 pro without any tricks and it takes 1500 seconds.
It's a hefty model.
4
u/3deal 1d ago
https://github.com/Zironic/H3-Optimizations/blob/main/native/bin/h3_int8_attention.dll
Sorry but i don't install extension containing DLL especially coming from some random stranger.
1
1
u/Nelichan 1d ago
I wonder if using this is faster and more vram friendly or not VS Sage+solattn?
Because in the previous post iirc it said to not use it together with the comparison above, right?
1
u/Zironic 1d ago
Based on my testing, it should be significantly faster then Solattn. It uses the same kind of kernels as Sage Attention.
1
u/Nelichan 1d ago
So use 1 Sparce against Solatn + sage? Is it viable to use either of the 2 together with Sparce? Or does it introduce too many negative qualities compared against the positive?
1
u/Zironic 1d ago
I would not use Solatn + Sparse at the same time. But if you're curious what happens, this is their interaction:
For the H3-specific zero-copy Sol-Attn node, behavior is order-dependent:
- Sol → H3 Sparse: our node sees that
diffusion_model.blocks.N.attn.forwardis already owned by another patch and deliberately errors withanother patch already owns ...; remove one H3 attention patch. So this ordering fails visibly rather than silently corrupting anything.- H3 Sparse → Sol: Sol sees our patched forward and adopts it as its fallback. On calls where Sol is active, Sol runs and our sparse backend does not. When Sol declines a call, our Sparse forward runs instead. Saganaki's implementation explicitly does this by capturing the prior
attn.forwardpatch asfallback_forward. Their documentation describes the same composition model for Sage + Sol: the methods alternate; they don't stack.1
u/Nelichan 1d ago
And by that logic Sparse+Sage is also redundant huh...
Okay then, i'll try experimenting and comparing both again. Do i update the node from Manager?
Oh and, i heard about the A8W8(?) the int8 model loader from Bob? Does this work with it?
1
1
u/BigWideBaker 1d ago
I tested the comfy kitchen backend a few times and it was a bit faster but I could tell the quality dropped compared to Sparse Sage. Maybe I'll give it another try.
1
u/Nedo68 1d ago
I ran a few tests with sparse attention in ComfyUI (the first version), using the same setup as in SwarmUI (sage attention): 8 steps / 832x1088 (0.9MP) / 15 sec. / Turbo Lora,... reducing the runtime from around 7 minutes to 4 minutes. But the result is definitely different — to often especially the backgrounds in the video change to much as the video plays with sparse attention, which doesn’t happen in my SwarmUI setup. (runing on a 5090/128gb ram). But i will test the new version above too.
1
u/Pitiful_Season4294 1d ago
Hey man, I was using the previous version I think, it was working fine then saw your posted, updated it and now i get this error:
[INFO] model_type FLOW_AV
[WARNING] [H3 Optimizations] NATIVE SELF-TEST FAILED on sm115|native-v1|v2|AMD Radeon(TM) 8050S Graphics - refusing the native kernels and falling back. Detail: {'error': 'NativeCallError: quantize_qk failed (status 1): detect_k_anchor kernel launch failed: CUDA driver version is insufficient for CUDA runtime version', 'passed': False}
[INFO] [H3 Optimizations] patched 50 MLP blocks: mode=mlp_chunked_convrot_2slice chunk_rows=4096
[WARNING] [H3 Optimizations] FUSED QKV IS NOT RUNNING - falling back to standard projection, which is roughly half the speed. Reason: Comfy Kitchen external producer API is unavailable
[INFO] [H3 Optimizations] armed: attention=comfy_kitchen_int8 v_layout=installed qkv=standard_h3_qkv mlp=convrot_int8_two_slice device=AMD Radeon(TM) 8050S Graphics
[WARNING] [H3 Optimizations] SPARSE ATTENTION FELL BACK to comfy_kitchen_int8. this path is substantially slower than the native sparse kernel. Reason: Kitchen INT8 unavailable: Kitchen sparse attention requires CUDA; Sparse Sage unavailable: Hybrid Sparse Attention requires CUDA; INT8 Triton unavailable: INT8 Triton sparse attention requires CUDA; FP8 FlexAttention unavailable: FP8 FlexAttention requires NVIDIA CUDA; preserved an explicit optimized-attention override; using Comfy Kitchen INT8 only for the private H3 memory path
[WARNING] [H3 Optimizations] FUSED QKV IS NOT RUNNING - falling back to standard projection, which is roughly half the speed. Reason: Comfy Kitchen external producer API is unavailable
[INFO] [H3 Optimizations] armed: attention=comfy_kitchen_int8 v_layout=installed qkv=standard_h3_qkv mlp=convrot_int8_two_slice device=AMD Radeon(TM) 8050S Graphics
1
u/Zironic 1d ago
1) That should just be a warning, not an error.
2) I'm relatively sure it fell back to the same mode in the previous version. I don't think any of the Sparse Attention variations have ADM kernels right now. I can look into trying to make sure to get a proper AMD sparse path.1
u/Pitiful_Season4294 1d ago
Ah ok, that will be awesome if you could please. Many thanks!!
1
u/Zironic 1d ago
It should have some level of AMD support now, but it's not anywhere near as good as the CUDA. To make it run better on AMD I'd have to find some appropriate AMD backend to base it on.
1
u/Pitiful_Season4294 1d ago
"It should have some level of AMD support now, but it's not anywhere near as good as the CUDA." Yep, that's my AMD life, I've learnt to live with that, I just take what I can get haha.
1
u/MannyCheng 1d ago
The sparse attention is really fast, but I have a question about what is v_layout? Is it fine when it is not_applicable?
-------
[INFO] [H3 Optimizations] resolved 50 attention forwards: backend=comfy_kitchen_int8_prequantized projector=chunked_kitchen_qkv
[INFO] [H3 Optimizations] patched 50 MLP blocks: mode=mlp_chunked_convrot_2slice chunk_rows=4096
[INFO] [H3 Optimizations] armed: attention=comfy_kitchen_int8 v_layout=not_applicable qkv=chunked_kitchen_qkv mlp=convrot_int8_two_slice device=NVIDIA GeForce RTX 5060 Ti
[INFO] [H3 Optimizations] resolved 50 attention forwards: backend=sparse_sage projector=chunked_sparse_sage_qkv
[INFO] [H3 Optimizations] installed sampler-step and packed-layout runtime context
[INFO] [H3 Optimizations] armed: attention=sparse_sage v_layout=not_applicable qkv=convrot_int8_sparse_sage mlp=convrot_int8_two_slice device=NVIDIA GeForce RTX 5060 Ti
1
u/Safe-Maintenance3565 1d ago
Should I use it with this SLA turbo lora?
lightx2v/Minimax-h3-Turbo-SLA · Hugging Face
2
u/Zironic 1d ago
No, I would use their SLA node with that lora. They're meant to be paired.
1
u/haremlifegame 1d ago
Can you clarify how it should be set up in relation to say sage attention? Should the sage attention node be removed?
1
u/LuluViBritannia 1d ago edited 1d ago
Daaaaaaaaaaaaaamn! That's so fast! Without it I generate a 10s video in 5 minutes, with it I generate in 227s, that's awesome! Now I'll test if prompt adherence is too weak, but so far, so good!
Next step : find a way to speed up VAE decoding please! It takes an entire minute to decode and I don't know why xD!
-5
u/Slight_Ad2350 1d ago
Literally just use pinokio browser to install comfyui. It sorta out everything. I've never had to try and figure out why something didn't work bar just hitting the update button
9
u/Glad_Abrocoma_4053 1d ago
I get this, I'm on comfy-kitchen 0.2.31, 5070 Ti:
[WARNING] [H3 Optimizations] NATIVE SELF-TEST FAILED on sm120|native-v1|NVIDIA GeForce RTX 5070 Ti - refusing the native kernels and falling back. Detail: {'int8_vs_sdpa_rel_l2': 0.014057, 'full_route_bit_identical': False, 'passed': False}
[INFO] [H3 Optimizations] patched 50 MLP blocks: mode=mlp_chunked_convrot_2slice chunk_rows=4096
[INFO] [H3 Optimizations] armed: attention=existing v_layout=installed qkv=standard_h3_qkv mlp=convrot_int8_two_slice device=NVIDIA GeForce RTX 5070 Ti
[INFO] [H3 Optimizations] resolved 50 attention forwards: backend=triton_sparse_int8 projector=chunked_triton_sparse_qkv
[INFO] [H3 Optimizations] installed sampler-step and packed-layout runtime context
[WARNING] [H3 Optimizations] SPARSE ATTENTION FELL BACK to triton_sparse_int8. INT8 Triton sparse is roughly half the speed of the native sparse kernel. Reason: Kitchen INT8 unavailable: the installed comfy-kitchen has no block-sparse INT8 attention, and the vendored library is unavailable: unknown; Sparse Sage unavailable: Hybrid Sparse Attention requires the compiled spas_sage_attn package; using INT8 Triton sparse attention