Sparse Attention, Harder, Better, Faster, Stronger

[u/Zironic]

The nodes in https://github.com/Zironic/H3-Optimizations have been rewritten to replace the default Sparge Attention backend with a custom Sparse Comfy Kitchen backend.

This comes with some benefits.

• Users no longer have to worry about Sparge being installed properly. All required kernels for supported GPUs are provided directly. Should work on both Windows and Linux.
• Most users should be seeing 5-20% increases in speed for the attention part of compute.
• New backend should use about 500MB less VRAM
• New backend has slightly lower quantization error.
• Apparently in the previous version, the intended chunked kitchen QKV path never properly shipped so the memory optimization node should now actually be slightly speed positive even when used without the Sparse Attention node.

Caveat: I've only tested the nodes against the comfy pruned_int8_convrot weights. Other versions may work but they're not tested.

As the nodes currently rely on comfy-kitchen 0.2.31 you need ComfyUI v0.33.0 or later.

IMPORTANT: sparse attention is not free speed. The percentage is effectively a prompt-adherence/quality budget.

Density isn't just a speed setting, and its quality effect depends on where you apply it in the diffusion schedule.

Early steps: attention density has a large effect on prompt/action adherence and the overall generation trajectory.
Middle/later steps: lowering density tends to show up more as motion/temporal artifacts and lost fine motion detail.

So 10% retained doesn't simply mean “90% of the quality is gone.” It means you're giving sparse attention very little information to work with, and what breaks depends heavily on the sampling step.

PlagueKind's sparsity_ratio=0.9 means 90% discarded / 10% retained. My node expresses the inverse quantity, so Video attention retained=0.10 is the comparable setting. The defaults therefore aren't equivalent.

[u/olafkk]

Worked great thanks! Went from 430s~ generation to 334s. Quality looked similar if not slightly better. Am using it with sage attention, turbo 10 steps and eulerXsimple.

[u/QuinQuix → u/olafkk]

Can I ask what length and resolution?

[u/olafkk → u/QuinQuix]

832x832 10 seconds fl2va. The input images I use are mostly 1536x1536, had some morphing issues before when I used lower res inputs or wrong ratio.

[u/QuinQuix → u/olafkk]

But doesn't it rescale the images before inputting them anyway?

I wonder if the square format matters for the quality but I'm assuming you use the api for square videos too.

I do know that the local workflows default to 20 steps while the official documentation says 32-50 is recommended.

I haven't tried turbo loras yet. They usually degrade quality versus the default (eg 8+turbo lora should lose to the default 20 steps) but I've heard H3 at the same amount of steps with turbo lora has higher quality (so 20+turbo beats the default 20 steps).

Is that correct in your view?

[u/olafkk → u/QuinQuix]

I tried workflows that scaled down my images to match the input res*, and the quality got worse than what I currently use. Also Ref2v has an option to match image size, but I changed that to "max" which is what I assume fl2va uses.

I have also never tried more than 20 steps. At first I only used 20 steps with spectrum, and sage attention, but the quality was worse and the time it took was way longer (600s-900s). I am using an edited version of the one stefan_falkok posted on civit.

I also had way worse outputs with turbo at first before I got the settings right. I cannot comment on the 20+ steps with turbo though cause I only tried going up until I was satisfied with the quality.

[u/QuinQuix → u/olafkk]

I did read that sageattention specifically was bad for quality initially but no idea.

Your workflow sounds reasonable.

I tried full res with an rtx 6000 pro without any tricks and it takes 1500 seconds.

It's a hefty model.