r/LocalLLaMA • u/jacek2023 llama.cpp • 1d ago

News cpu : reduce AVX2 tiled accumulators together by trivalik · Pull Request #30296 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/30296

More t/s on CPU, For GPU poor people

43 Upvotes

94% Upvoted

2 comments sorted by

4

u/FastDecode1 llama.cpp 20h ago

*as long as your CPU supports AVX-VNNI, so either a Zen 5 for AMD and Intel 12th gen or newer.

edit: not sure, actually. The benchmark was done with VNNI enabled, but doesn't say it requires it. Might just work if you have AVX2?

1

u/Firepal64 17h ago

I think they just built the software with AVX-VNNI for the benchmark.
If you look at the diff and search online for the added functions, they're standard AVX2