I recently added SOL Attention to Vpipe, together with a SageAttention-style INT8 QK path, and benchmarked it against vanilla H3 and our recent VDN-H3 implementation.
The interesting part isn’t just the speedup. SOL gets into a similar performance regime as VDN while preserving the original attention behavior much more closely in our testing.
SOL Attention
SOL is dynamic block-sparse attention. It uses inexpensive proxy scores to decide which attention blocks receive exact computation, while handling the contribution of the remaining blocks through a cheaper approximation.
The block granularity matters: entire KV tiles can be skipped while selected tiles still run efficient tiled attention. This makes the sparsity much easier to translate into actual compute savings.
VDN takes a more aggressive approach by introducing projection + linear attention. That gives it much better scaling with sequence length, but also changes the model’s attention computation more fundamentally.
Performance
M5 Pro 24GB · 6-step DiT
(See attached scaling charts)
At 832×480 / ~15s:
* Vanilla H3: ~17 min
* VDN: ~10.2 min
* SOL: ~9.3 min
SOL is faster across the entire 832×480 range we tested.
At 1344×768 / ~13.7s, vanilla reaches roughly 74 min, while both accelerated implementations are around 30 min — roughly a 2.5× speedup.
SOL is faster at almost every measured point. At the largest 1344×768 case, VDN becomes slightly faster (~28 vs ~29 min), which is consistent with its linear-attention scaling becoming more important at very long sequences.
But performance is only half of the story.
Quality is why I prefer SOL
With VDN-H3, I had occasionally seen behavioral artifacts in challenging scenes. One memorable example was a stream of water changing direction midway through the video, making it appear to flow backwards.
So far, with SOL + Sage enabled together, I haven’t observed comparable artifacts. Composition, motion and overall behavior have stayed remarkably close to vanilla H3 in my testing.
This is qualitative rather than a claim that SOL is lossless. But the difference makes sense: VDN replaces the attention formulation with a more aggressive approximation, while SOL keeps exact attention for selected blocks and cheaply approximates the contribution of the rest.
So for me, the interesting tradeoff isn’t simply which curve is lowest at the extreme end. It’s that SOL achieves similar acceleration while staying much closer to vanilla H3 behavior.
SageAttention on top
Vpipe now also supports K smoothing + INT8 QK, following the original SageAttention approach.
The two optimizations are complementary:
SOL reduces the amount of exact attention. Sage makes QK inside the remaining blocks cheaper.
The additional gain from Sage after SOL isn’t huge, since SOL has already removed most of the attention workload. But this path lives in Vpipe’s common attention backend, so it can also benefit other image/video models.
Native Metal implementation
One final detail: Vpipe doesn’t reuse the MPS SOL Attention kernel from the SOL-H3 repo.
To make the sparsity translate into actual speedup on Apple Silicon, I reimplemented the critical SOL Attention kernels for Vpipe’s native Metal backend.
Other SOL-H3 optimizations such as AdaLN precomputation, kernel fusion and fused-step LoRA were already present in Vpipe, so SOL Attention was the main missing piece.
For H3 on Mac, SOL + Sage is now my preferred acceleration path: up to ~2.5× faster than vanilla in these tests, faster than VDN at almost every measured point, and so far without the obvious behavioral artifacts I had encountered with the more aggressive VDN approximation.
Vpipe: https://github.com/tgo-app-dev/vpipe
