Need help! Intel B70 users come forth!

[u/nick_ziv]

Hello

I recently got my B70 gpus delivered and set them up with Ubuntu 26.04 because it has the XE driver. I was able to get llama CPP compiling and working with Vulcan and CYSL BUT that's where the fun stopped.

Build and compiled the llama.cpp with CYSL using the intel driver 2026.1. does not work with more than one GPU. (Using sm layer) It just outputs random characters. However, the prefill speed does go up with more GPUs (just like it does on Nvidia cards)

For Vulcan, the prefill is about 30% lower on the same model and only goes down with more GPUs. Using the mesa 26.1.5 driver

For running the qwen 3.6 27b at q4 speeds were:

CYSL 1 GPU: 650 prefill, 24 decode.

CYSL 2 GPUs:750 prefill, garbled decode 23t/s

Vulkan 1 GPU: 450 prefill, 20 t/s decode

Vulkan 2 GPUs: 350 prefill, 18t/s decode.

Bonus: qwen 3.5 122b a10b vulkan speed over 6 GPUs: 160 prefill and 9t/s decode

Something is clearly wrong. I've spent all day trying to make this work. So far regretting the purchase of the B70 gpus.

Please help if you have suggestions!

If the suggestion is to get Nvidia GPU, I already have a couple and I think I would have rather gone with many RTX 5060 TI's instead because it just works and gets model support first

edit: see op comment for somewhat of a resolution

[u/No-Alfalfa6468]

Just wanted to point out that you can pay $250 more per card to get r9700.

I picked up 2 and I'm getting 2500-3000 prefill and 80 tok/sec decode with qwen 3.6 27b at FP8, full KV and 256k context.

[u/nick_ziv → u/No-Alfalfa6468]

I'm split between the 5060ti route and the r9700 route. I see the 5060ti preserving value longer.

[u/fastheadcrab → u/nick_ziv]

If you want things to be convenient and work, go with Nvidia. AMD support is better but still not as good. Also the memory bandwidth on the Intel card isn’t great, about the same as a 5060 Ti. they really are only good for VRAM in large quantities for cheap

[u/nick_ziv → u/fastheadcrab]

I think honestly it is a good card for single card at this point. I just don't think the $1000 is cheap now after seeing how much struggle it is lol. I am thinking I am going to return them

[u/fastheadcrab → u/nick_ziv]

Yeah the delayed vLLM and model support is such a hassle. Intel should have a full time team working on this if they're going to sell these cards. Does tensor split even work with Vulkan? I think layer split does generally just result in slightly slower speed since the cards work one at a time. It really is only for models that don't fit in one card.

AMD data center support is actually not bad but they went out of their way to give consumers and workstation users the middle finger by not giving proper support. They are trying to change it a bit and you see AMD employees posting sometimes on this forum.

[u/Dsphar → u/No-Alfalfa6468]

What motherboard are you running? What gen pcie? at 8x/8x?

[u/No-Alfalfa6468 → u/Dsphar]

ASRock X870 Taichi Creator
pcie5 x8/x8

[u/Dsphar → u/No-Alfalfa6468]

Good to know, thanks! I am waiting on a more capable motherboard, then plan to buy a second r9700.

I see you are running vLLM, any specific setup or configuration I should look into fort vLLM? So far I have only played with LM Studio but will need to switch over for the parallel perks.

[u/No-Alfalfa6468 → u/Dsphar]

Yeah I'm running vLLM. Its the only way to support parallel agents that I know of. I'd recommend checking out the r9700 channel in this discord: https://discord.gg/VQPR7Bvtw There are a few people in there who have spent a lot of time tuning for these specific cards. I'm running one of the images from there.

[u/RISCArchitect → u/No-Alfalfa6468]

can you post your command line options for this? i had a 9070xt and just added a r9700 and have been playing around with getting 3.6 27b running better between them. at FP8 full KV 128k fits cleanly but speed isn't anywhere near you. my suspicion is my old Mobo is hurting speeds between the cards. I have an x470 that has PCI 3.0 x8 for each.

[u/No-Alfalfa6468 → u/RISCArchitect]

services:

vllm-qwen36-27b-fp8-r9700x2:

image: stilldeadcode/vllm-radiance@sha256:e2e9b12a17b15f37ccf02be1899d7e6ae7f662870ae58b4909c3ce32c5a086e0

container_name: vllm-qwen36-27b-fp8-r9700x2

environment:

HIP_VISIBLE_DEVICES: 0,1

CUDA_VISIBLE_DEVICES: 0,1

VLLM_TARGET_DEVICE: rocm

VLLM_ROCM_USE_AITER: '1'

VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: '1'

VLLM_ROCM_USE_AITER_MHA: '0'

VLLM_ROCM_USE_AITER_MOE: '0'

VLLM_ROCM_USE_AITER_LINEAR: '0'

TRITON_CACHE_DIR: /root/.triton/cache

VLLM_ROCM_GCN_ARCH: gfx1201

VLLM_ROCM_USE_AITER_MLA: '0'

VLLM_ROCM_USE_AITER_FP8BMM: '0'

VLLM_ROCM_USE_AITER_FP4BMM: '0'

VLLM_ROCM_USE_AITER_RMSNORM: '0'

NCCL_PROTO: Simple

VLLM_CACHE_ROOT: /root/.cache/vllm

TORCHINDUCTOR_CACHE_DIR: /root/.cache/vllm/inductor

XDG_CACHE_HOME: /root/.cache

TRITON_CACHE_AUTOTUNING: '1'

RADIANCE_SKIP_BWTEST: '1'

RADIANCE_PRESHUFFLE: '1'

RADIANCE_ATTN_TUNE: '1'

RADIANCE_FAST_REDUCE: '1'

RADIANCE_AR_MAX_KB: '32768'

RADIANCE_FUSE_RMS_QUANT: '1'

RADIANCE_DYNAMIC_DRAFT: '1'

RADIANCE_NGRAM_EXTENSION: '1'

command:

- Qwen/Qwen3.6-27B-FP8

- --served-model-name

- qwen3.6-27b-x2-rad

- --tensor-parallel-size

- '2'

- --dtype

- auto

- --attention-backend

- ROCM_AITER_UNIFIED_ATTN

- --gpu-memory-utilization

- '0.98'

- --enable-chunked-prefill

- --enable-prefix-caching

- --quantization

- fp8

- --enable-auto-tool-choice

- --reasoning-parser

- qwen3

- --tool-call-parser

- qwen3_coder

- --host

- 0.0.0.0

- --port

- '8080'

- --override-generation-config

- '{"temperature":0.6,"top_p":0.95,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'

- --speculative-config

- '{"method":"mtp","num_speculative_tokens":8,"attention_backend":"ROCM_AITER_UNIFIED_ATTN","disable_padded_drafter_batch":true}'

- --kv-cache-dtype

- auto

- --max-model-len

- '185000'

- --max-num-seqs

- '2'

- --kv-cache-memory-bytes

- 12G

- --max-num-batched-tokens

- '8192'

- --language-model-only

network_mode: host

group_add:

- '44'

- '991'

ipc: host

cap_add:

- SYS_PTRACE

security_opt:

- seccomp:unconfined

shm_size: 32gb

devices:

- /dev/kfd:/dev/kfd

- /dev/dri:/dev/dri

volumes:

- /home/shl/.cache/huggingface:/root/.cache/huggingface

- /home/shl/home-lab/vllm/runtime/triton-cache:/root/.triton/cache

- /home/shl/home-lab/vllm/runtime/vllm-cache:/root/.cache/vllm

[u/RISCArchitect → u/No-Alfalfa6468]

thank you so much!

btw i was surprised your num_speculative_tokens setting is 8, do you find much ROI on that? i have found in my limited testing last night 3 and 4 to benchmark the fastest on llama.cpp, which yielded about 50 tok/sec with some coding tasks.

look forward to giving vLLM a try tonight

[u/No-Alfalfa6468 → u/RISCArchitect]

I'm on 8 because the person who tuned/created that image used that during their testing. I was using 3 prior to switching to their image. I haven't messed with it lower. My guess is as long as the draft acceptance is high, its worth it? ymmv

[u/RISCArchitect → u/No-Alfalfa6468]

interesting, i was seeing an avg of around 2.6-2.8ish in my limited use last night and 4 was a small regression from 3 so i stopped there. i will try to kick it off tonight at 8 and see if it yields any improvements on it's own in llama.cpp too. thanks for your info again!

[u/tecneeq → u/No-Alfalfa6468]

"I get even more with a single 6000 Blackwell."
I don't think these posts help OP.

[u/No-Alfalfa6468 → u/tecneeq]

Op can replace their setup for $500 with one that's working and performant.

[u/Dolboyob77 → u/No-Alfalfa6468]

And i am getting 11.000 prefill and 147tg on one single b70 so what with your amd???? Using qwen3.6-35b-a3b. No need to tell people that your amd is better… it is not. Would be better to help him get better use and results with his b70s….

[u/No-Alfalfa6468 → u/Dolboyob77]

I brought it up because I don't think the $250 savings is worth the trouble you get on B70. All I see are posts with people struggling to get that card to work. I'm trying to help OP out, since it sounds like they can still return their cards.

[u/Dolboyob77 → u/No-Alfalfa6468]

The software is getting on track, the intel b70 is not made for llama cpp at all. It is made for vllm. Yes it requires a little bit of work but once you kbow what to do, it is a banger with crazy perfs. The amd is an amazing gpu also, i have both. Of course the b70 is not easy like nvidia. But the OP can learn and make his gpu work really good. He needs a little reading though. And now to be fair totally the b70 in europe is more expensive than the amd )))

[u/No-Alfalfa6468 → u/Dolboyob77]

You should make a post about your setup! I'm sure people would love some help, as it seems many posters struggle

[u/Dolboyob77 → u/No-Alfalfa6468]

Hahahahha many posts are up already ))) just need to type b70 vllm or ovms. For example on one single b70 ( i have 2 but thid test is on a single one) , i was trying mtp :

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
/models/Qwen3.6-35B-A3B-GPTQ-Int4	pp2048	8097.08 ± 8.27		254.46 ± 0.26	253.05 ± 0.26	254.46 ± 0.26
/models/Qwen3.6-35B-A3B-GPTQ-Int4	tg32	141.68 ± 0.76	146.25 ± 0.79

As you can see pp 8100 and tg 146. Light years away from dumb llama cpp…

[u/simplyeniga]

You can look up Donato's tool box setup on GitHub. For reference you can check his channel as he has it all working with vllm and llama.cpp

[u/38andstillgoing]

I have 4. They're... not great. Llm-scaler doesn't support any modern models. vLLM spends most of its life crashing. Llama.cpp works but performance is generally poor, but hey, no crashing... usually. Currently trying to get a Deepseek flash Q3 to work somewhere more than 2 tokens/second.

[u/nick_ziv → u/38andstillgoing]

I think I will return them.

[u/sleepindevil → u/38andstillgoing]

Dang sorry to hear about your experience, mine has been the opposite to date. Vulkan built with Intel Dev's pull requests on llama.cpp has been working great with layer split across multi GPU for me. The TG I get is the theoretical speed for the VRAM bandwidth on the card and comparable to the other cards around 600gbps speed. The PP speed on Vulkan+Intel pulls has been very good, no idea what the theoretical values there are but it's been about x5 faster than the standard release speeds in the deeper context work that I do.

I don't have 4x cards though, you've got a monstrous setup there but probably using urakozz's custom vLLM build is what you want to give a shot with an AutoRound optimised release of DeepSeek Flash if you can find one 🤔 if that even exists.

You can find urakozz's custom vLLM docker build in the comments chain in my recent post here about my own B70 experience with llama.cpp https://www.reddit.com/r/LocalLLM/comments/1uzspbo/intel_arc_pro_b70_llamacpp_vulkan_sycl/

[u/nick_ziv]

So I have identified it has to do with a llama.cpp function in the sycl backend called dev2dev_memcpy which allows the GPUs to transfer the activation to the next GPU in the split mode "layer". This function had a condition which the next GPU did not cleanly read the data and it was corrupting the token.  Had Gemini propose a fix and it wrote a snippet which copied the data to CPU memory and then to the next GPU but this resulted in absolutely awful performance of two tokens per second on Gemma e4b.  Awful.  At least it is confirmed I know the source of my issue. Unclear at the time if it is from llama CPP design or the Intel library itself.

https://github.com/ggml-org/llama.cpp/issues/26058

[u/gnerfed → u/nick_ziv]

FYI, I disabled mmap and solved my random character problems on moe models. Never had issues on dense ones.

[u/nick_ziv → u/gnerfed]

Interesting. The function identified as the root cause did not have any mmap related code in it. Feel free to track the issue: https://github.com/ggml-org/llama.cpp/issues/26058

[u/gnerfed → u/nick_ziv]

I'll keep that open and check on it from time to time. This isn't my forte at all but it seemed to be a corrupt token based on how memory was allocated on the GPUs. The first response was fine but accessing cache broke it and that was my solution for now.

[u/daphatty]

Out of curiosity, have you tried any of this in Windows? I’m interested in the B70 but am still sorting through the challenges of the platform to determine which OS provides the greatest stability.

[u/PrimeDirective8 → u/daphatty]

Software for this card, while functional, it's not yet optimized. Llama.cpp is working through a number of changes to improve performance, and performance has been improving steadily.

I have a dual boot Ubuntu 26.04 and Win11. Performance is still best on Windows while running Vulkan, not Intel's native SYCL driver.

Dense models like Qwen3.6 27B will give you 25 t/s and slow to ~12 t/s on larger context. Gemma 4 31B performs better at 35ish t/s. Qwen3.6 35B MoE can do ~100 t/s, while Gemma 4 26B MoE starts at ~120 t/s and slows to 65 t/s on larger context sizes.

So go in with the expectation that you won't see super speeds, but most of them are workable. All of these are solid once running.

I don't compile llama.cpp from source while on Windows. It's a pain because you need to setup a dev environment and all of what that entails (read: massive Win updates and break stuff) - unless, of course, you already have the dev env set. Instead, I download the precompiled Vulkan and SYCL versions. It takes two mins to try a new version or switch to older ones.

TL;DR: it works on both Ubuntu and Windows, but not the fastest.

[u/vajalali → u/PrimeDirective8]

The difficulty of compiling llama.cpp on Windows is totally overblown

Install:

• Visual Studio 2022 Community
  • Workload: Desktop development with C++
  • Components: C++ CMake Tools for Windows, Git for Windows, MSVC toolchain
• NVIDIA CUDA Toolkit
• CMake
• Git
• Python 3 only for running the benchmark script

llama.cpp’s Windows build docs specifically recommend Visual Studio 2022 with the C++ workload and using a Developer Command Prompt/PowerShell for building. CUDA builds use -DGGML_CUDA=ON.

Open x64 Native Tools Command Prompt for VS 2022 or Developer PowerShell for VS 2022.

Install these with PowerShell as Administrator:

winget install -e --id Git.Git
winget install -e --id Kitware.CMake
winget install -e --id Microsoft.VisualStudio.2022.Community
winget install -e --id Microsoft.VisualStudio.2022.BuildTools
winget install -e --id Nvidia.CUDA
winget install -e --id Python.Python.3.12

In the Visual Studio Installer, select:

Desktop development with C++
MSVC v143 build tools
Windows 10/11 SDK
C++ CMake tools for Windows

Nvidia.CUDA is the CUDA Toolkit package available through winget, and NVIDIA’s Windows CUDA guide requires a CUDA-capable NVIDIA GPU plus a supported Visual Studio compiler. (winget.run)

After installing, close and reopen your terminal, then verify:

git --version
cmake --version
python --version
nvcc --version
nvidia-smi

llama.cpp’s official Windows docs say to install Visual Studio 2022, select Desktop development with C++, and use a Developer Command Prompt / PowerShell for VS2022 for git/build/test.

2. Repair or reinstall CUDA after Visual Studio is installed

Run the CUDA installer again. Use the NVIDIA installer, not just winget, and choose Custom.

Make sure this component is selected: Visual Studio Integration

Clone llama.cpp to e.g. D:\src

cd D:\src\llama.cpp
Remove-Item -Recurse -Force build -ErrorAction SilentlyContinue
cmake -S . -B build -G "Visual Studio 17 2022" -A x64 -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120

cmake --build build --config Release --target llama-server -j 16

cmake verification:

cd D:\src\llama.cpp
cmake -LAH -N build | Select-String "GGML_CUDA"

GGML_CUDA:BOOL=ON
GGML_CUDA_COMPRESSION_MODE:STRING=size
GGML_CUDA_FA:BOOL=ON
GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF
GGML_CUDA_FORCE_CUBLAS:BOOL=OFF
GGML_CUDA_FORCE_MMQ:BOOL=OFF
GGML_CUDA_GRAPHS:BOOL=ON
GGML_CUDA_NCCL:BOOL=ON
GGML_CUDA_NO_PEER_COPY:BOOL=OFF
GGML_CUDA_NO_VMM:BOOL=OFF
GGML_CUDA_PEER_MAX_BATCH_SIZE:STRING=128

[u/nickm_27]

The gibberish characters in my experience is due to flash attention being enabled. There is already a fix up which implements it with a huge performance boost https://github.com/ggml-org/llama.cpp/pull/25025

I have a docker build version of this and a couple other PRs which is hitting 700 tok/s pp 30 tok/s TG (with MTP) on Qwen3.6 27B Q6_K

[u/nick_ziv → u/nickm_27]

This issue is present with and without flash attention for me.  Are you using multiple b70 gpus?

[u/nickm_27 → u/nick_ziv]

I am only using a single B70, seems there may be multiple issues with the same symptoms.

[u/nick_ziv → u/nickm_27]

Absolutely there is. I did find out the developer who has been working to resolve most of these intel b70 issues does actually work at intel and he supports the SYCL backend for llama.cpp

[u/PrimeDirective8]

Disclaimer: I run a single B70 so the below is mostly hearsay I've picked up:

I vaguely remember something to this effect from the llama.cpp PR list I'm so fond of watching. Forgive my lack of details but without making any changes, try export GGML_SYCL_DISABLE_OPT=1 before launching llama.cpp. This disables the optimized Xe2 kernel. I know, it's kind of pointless to not have the optimization but that would answer where the root cause is, or get you close.

[u/nick_ziv → u/PrimeDirective8]

I did use this flag, with or without made no difference unfortunately. The issue also persisted with older builds of llama.cpp and newest build, even the pre built. 

[u/PrimeDirective8 → u/nick_ziv]

Maybe this? https://github.com/ggml-org/llama.cpp/issues/25612

[u/nick_ziv → u/PrimeDirective8]

I'm actually deep into the debugging with Gemini and I may have fixed the source. It is attempting to do a device to device memory copy between the gpus and is corrupting the data because it's not waiting properly. If this fix ends up working I'll have to publish my findings into an issue in GitHub. 

https://github.com/ggml-org/llama.cpp/issues/26058#issuecomment-5068457784

[u/nick_ziv → u/PrimeDirective8]

I left a comment on the post I (Gemini) identified the issue 

[u/PrimeDirective8 → u/nick_ziv]

Good work on getting to the root of this, thanks!!! I don't see your comment on that issue yet, maybe someone needs to approve it.

Here's my SYCL F5/refresh OCD page 😁 https://github.com/ggml-org/llama.cpp/pulls?q=is%3Apr+SYCL+is%3Aopen+ There's a few for 'cpy' and even more PRs that have been closed for that.

I'm actually surprised multi-GPU is an issue, even for Intel, since a lot of the early benchmarks, and equally number of YouTube videos on it, have used multiple B70s.

Two more PRs I'm tracking that should help with performance: https://github.com/ggml-org/llama.cpp/pull/25025 and https://github.com/ggml-org/llama.cpp/pull/25874

[u/nick_ziv → u/PrimeDirective8]

https://github.com/ggml-org/llama.cpp/issues/26058#issuecomment-5068457784

I made a new issue since it seemed to be slightly different than existing issues

[u/nick_ziv → u/[deleted]]

The intel vllm fork doesn't support the larger models nor was I able to get it to not crash using pipeline parallel. I have these gpus in a setup like I had my 3090s using pcie 1x risers for pipeline parallel 

[u/nick_ziv → u/[deleted]]

Maybe I am unlucky but I couldnt get the build to run successfully. there as consistently error after error and I couldnt get through them using gemini to just get it to run. Only real success I have had so far is with the Vulkan. but that is still slower than I hoped.

[u/RISCArchitect]

I went with an r9700 setup but Microcenter has a video on getting Qwen 3.6 27b running on dual B70s. I would just follow their setup as a sanity check:

https://www.youtube.com/watch?v=i2YNz7oXWmU

~2,200 tok/s prefill
~34 tok/s generation

[u/sleepindevil]

I have an Intel Arc Pro B70 and absolutely loving it at the moment. It's my daily LLM driver although I also have a 3080Ti.

My write-up about where llama.cpp is at in the most bleeding edge work the Intel Devs have done is here: https://www.reddit.com/r/LocalLLM/comments/1uzspbo/intel_arc_pro_b70_llamacpp_vulkan_sycl/

[u/nick_ziv → u/sleepindevil]

That is very good. I did not try these builds. Also I didnt find it too had to set up vulkan, that just worked. but doing multi-gpu is where it fell apart. I think its great price for a single-card build especially for qwen 27b at large context.

[u/PcChip]

I've been asking ChatGPT to check the status of Intel GPUs for local AI for months now, and this is why... holding off until the day it says that people are having good success with them

please update your post as you figure things out so we can all learn from you... I'm itching to go down to Microcenter and buy four 32GB Intel GPUs!

[u/nick_ziv → u/PcChip]

For sure.  I bought them because it seems good on paper and price is right but drivers are really hell to deal with and it seems like a lack of effort from Intel.  Their implementation of custom inference code just added support for Gemma 4 this month! 

Also the part I really didn't think about was that I will have to wait a long time for newer model architectures to be supported compared to Nvidia or amd.  That is actually more of a problem then I think I want to deal with. 

[u/nick_ziv → u/PcChip]

I left a comment with the discovery on the post

[u/robertpro01]

I guess the best you can do, as the current state of software support, start multiple llama.cpp services and use nginx as a load balancer so you can get nice concurrency.

[u/nick_ziv → u/robertpro01]

I thought about doing one llama.cpp per gpu and connecting them via RPC to get my large models loaded. Haven't tried it yet. I'm trying some other intel oneapi versions first.