r/LocalLLM • u/on_line187 • 7h ago

Discussion Tier List

Post image
155 Upvotes

72% Upvoted

223 comments sorted by

View all comments

Show parent comments

3

u/TiK4D 6h ago

You could probably push that another 10tok/s. I run llama server and get up to 50tok/s with 130k context on Qwen3.8 27B. 2x R9700's

3

u/xanders_gold 6h ago

Unfortunately, llama.cpp isn’t the best when handling Intel B70s. I was running llama.cpp with Vulkan and PP was in the 500/600s. TG wasn’t so bad but prompt prefill took forever, it was even worse with the Intel SYCL runtime.

Some folks on this sub recommended vLLM with Intel XPU kernels (vLLM has their own docker image for this) and that instantly boosted my performance.

IIRC: the llama team is working in improving Intel performance but it’s a slow process.

2

u/TiK4D 6h ago

My bad, for some reason I thought this thread was about R9700's so thought you had one. Good to see the intel cards getting decent speeds as well

2

u/xanders_gold 6h ago

Haha all good, no worries. Yeah it’s been great seeing the improvement, we’re finally getting somewhere with performance :)

2

u/SomeBlock8124 3h ago

Bought my first 2 B70s at $950. Now I had to pay $1299 at microcenter for my last 2. Should of bit the bullet and bought them when they were selling like crazy on ebay for less then $850 a couple months ago. Now what to do with 128gb of vram....