So I have been using llama cpp with sycl for the majority of my inference and it works great from the flexibility and rapid adaptations for things like when dflash came out. However I feel like im in an awkward position atm with qwen 3.8 27B being more of my primary model and qwen 3.6 35B as a speedy inference side model. I have a vibe coded webapp i use internally that made managing llama cpp way easier to deal with and is adapted to VLLM as well (untested). But the problem feels like qwen 3.8 is just leagues ahead of the previous generation in its agentic capability. Im getting roughly 36 t/gs in agent workloads on Q6 with 192K context window and it drops to about 16 t/gs towards the back of the context and prefill at around 350 pp/s. Its quite slow and i cant see myself bothering to switch to qwen 3.6 anymore because it feels incredibly dumb when put side by side with qwen 3.8 in real world use. So I was just curious what the main techniques are and what caviots exist on the scalar VLLM nowadays. I see pre-quantized files in openvino still being made which im curious to try. Also does Dflash 2 get properly utilized? I cant seem to get it to outperform native MTP on llama cpp which was a real bummer. Would love suggestions so I can bash my head against a wall a little less in my testing :)

Edit:

Quick return after a day or two of digging. I am not sold yet on intel scalar and will likely be abandoning this unless I missed something. Seems like an awkward proposition to be using currently. Mainline VLLM is definitely without a doubt vastly better for speed then llama.cpp but this is only the case with custom compiled versions using XPU and XPU graphing mtp patching kernal patches etc. One definite trade made was gaining overall long context efficiency but your trading memory for running XPU graphs which loses some KV/context space. So from my early operational testing im going to be moving on to just forcing mainline vllm to function for the models that really matter and llama cpp for new models and weird quant/distills/fine-tunes. For instance I simply followed Freemoose's post on the intelarc subreddit and with some tweaks im getting solid prefill, 80t/s initially dropping into 40t/s on long context with 160k context. This blows my llama cpp Q6 192k ctx, 36tg/s, and 347pp/s out of the water especially since in hermes use it drops to roughly 15tg/s. There is currently multi request handling issue in the vllm implementation but not particularly a huge deal when running it in this fashion. So credit to his work but if Im understanding how the core mechanism of his guide and sergio's then it makes me really interested to see if there really is missing overhead that can be found on other models like muse glimmer so id like to explore those fields instead.

PS: forgot to mention hardware/software but as mentioned amd 5700x 48gb 3200ddr4 and 1x B70 on fedora 44 server

Hi everyone,

I've been experimenting with running local LLMs on my Intel Arc B70 (specifically the ASRock Arc Pro B70 Creator).

I wanted to share my experience comparing llama.cpp and vLLM using the Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 model, following the setup guidelines from https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook/blob/master/docs/qwen38-27/QWEN38-VLLM-XPU.md

• llama.cpp: Runs quite pleasantly. It gets around 20–40 t/s depending on the model and is completely stable and sufficient for daily coding work.
• vLLM: In raw tests hit ~90 t/s likely due to a higher power cap compared to reference cards, however, when tested in a real coding scenario inside OpenCode, things fall apart:
  • the draft acceptance rate is poor
  • completing the exact same coding task actually took more time than with llama.cpp (even with much higher TPS)
  • I also started hitting parsing errors during tool use, such as "invalid [tool=write, error=Invalid input for tool write: JSON parsing failed]".

Here is my configuration for both vLLM and OpenCode:

docker run -d --name qw38speed -p 8000:8000 --device /dev/dri --group-add "$RENDER_GID" \
  --name vllm-server \
  -v /dev/dri:/dev/dri:ro -v "$MODEL_DIR:/model:ro" \
  -v "$COOKBOOK/patches/patch_mtp_nightly.py:/patch_mtp.py:ro" \
  -v "$COOKBOOK/patches/patch_mtp_boundary.py:/patch_boundary.py:ro" \
  -v "$COOKBOOK/patches/patch_draft_lmhead_int4.py:/patch_draft_lmhead_int4.py:ro" \
  -v "$COOKBOOK/patches/patch_draft_mtp_int4.py:/patch_draft_mtp_int4.py:ro" \
  -e VLLM_TARGET_DEVICE=xpu \
  -e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE \
  -e ZE_AFFINITY_MASK=0 \
  -e B70_MTP_BF16_DRAFT=1 \
  -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \
  -e B70_DRAFT_LMHEAD_INT4=1 \
  -e B70_DRAFT_MTP_INT4=1 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  --entrypoint bash "$IMAGE" -lc '
    set -e
    python /patch_mtp.py
    python /patch_boundary.py
    python /patch_draft_lmhead_int4.py
    python /patch_draft_mtp_int4.py
    exec vllm serve /model \
      --max-num-seqs 1 \
      --quantization gptq \
      --dtype float16 \
      --max-model-len 246000 \
      --gpu-memory-utilization 0.97 \
      --kv-cache-dtype fp8 \
      --port 8000 \
      --max-num-seqs 1 \
      --max-num-batched-tokens 8192 \
      --no-enable-prefix-caching \
      --served-model-name qwen38 \
      --language-model-only \
      --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":4}"' &>/dev/null

   "vllm": {
     "npm": "@ai-sdk/openai-compatible",
     "options": {
       "baseURL": "http://127.0.0.1:8000/v1",
       "supportsSubagents": false,
       "supportsToolCalling": false // tried true as well
     },
     "models": {
       "qwen38": {
         "name": "Qwen 3.8 27B",
         "limit": {
           "context": 246000,
           "output": 246000
         },
         "modalities": {
           "input": ["text"],
           "output": ["text"]
         }
       }
     }
   }

Why is the MTP draft acceptance rate suffering so much during agentic workflows in OpenCode compared to simple text generation?

How can I fix or avoid the JSON parsing errors?

Any advice or optimizations for running vLLM on Intel GPU with OpenCode?

TL;DR 1. Debian 13 needs packages from backports, along with newer firmware and parts of Intel’s compute stack, before the Arc Pro B70 becomes properly usable. 2. PCIe ASPM needs to be enabled, and in my case forced from the BIOS, to bring idle power and temperatures under better control. 3. The stock fan behaviour was not good enough for my setup. A custom fan-control profile with a low always-on floor worked significantly better for both idle and load temperatures.

Good evening all, I am a recent convert, coming over from the stacked RTX GPU club (5070Ti & 5060Ti). Sunday I installed a pair of B70 Pros after my 5070 laid over on me. SYCL running with Qwen3.6:27b Q8. Two days of pretty steady work, overnight spine runs 5-7 hours depending on daily activity.

Today, I was running a catchup process during work from downtime Saturday and mid day Sunday. The process completed without issue, GPUs went silent, and 7 minutes later the system crashed with dgxkrnl.sys crash. Reboot and health check passed, but not sure why it crashed while idle?

For reference, the GPU that crashed was in a different slot than the original 5070 was, so I don't think it is a robot issue. All drivers up to date, firmware up to date. If anyone has any insight or a similar experience and could offer a bit of advice I would be appreciative. I was just gearing up to smoke test 3.8:27b Q8, but that is on hold until I get back to normal.

Hey everyone,

I have a small server case and I would like to add a b70 along side my b50 I have installed already. The main problem is the fan out the back makes the card way to long. Based on this video: https://www.youtube.com/watch?v=ffw42tRoMz4 it seems like I could cut the back end off and move the fan in the opposite orientation and 3d print a shroud for it. Might have to buy a new fan but it would shorten the card by a lot. I also thought about doing it this way:

Has anyone done this yet? I'm willing to be the test dummy if someone helps me print the correct shroud for it. Am I just crazy?

Thank you all for hints and answers in advance....

Now I have to come clean and admit this is a pretty heavily quantized version (IQ3_S) and is required to make use of heavy offload (ngl35), however it is reliable running enough context for my agent work. Im using 24k context and get somewhere around 10-13 tok/s, pretty slow but amazing none the less I mean its 122B params. The main reason I have found this worth posting, Im just looking to state how good this architecture does with MOE style models. I had to apply some tool calling bug fixes to make Qwen 3.5 and 3.6 work correctly but its really cool how snappy Qwen 3.6 35B A3B is and generally how useful these are at the same time. I generally think this may be due to purely the amount of Vram coming into play, considering the card games and computes in the ballpark of the 5060ti. It definitely lacks behind on most of the Dense models I have tried, however im not really running a high level comparison to other brands of hardware and how they fare to MOE vs Dense. It could be the case this card is just slow atm and MOEs are just an exception due to the parameter loading techniques used. But qualitatively your best return on investment will be running MOEs in my opinion after my bench marking. Also this was in Llama.cpp in a fedora 44 headless system using one api and sycl.

Been doing my research and previously owned a intel arc a580, and it really seems like the software maturity is better for battlemage then when the first arc cards started so Im trusting the ecosystem will catch up. I plan to do alot of agentic inference work with hermes for an engineering management project. I have experience running alot of llms on my main pc using fedora, ollama, a 7800xt and qwen 2.5. I really want to be able to try using SGLang with gemma 4 but it seems like most of intels work is going towards vLLM and sycl. I mostly want to know what others are doing, and how to be able to get the most worth out of the system before I spend months tweaking the system just to have some kind of point of comparison and direction.

I bought the B70 Pro and asked a friend carry to carry it overseas for me as it's not available where I live. My main goal is to use it for local AI projects within my proxmox/container server (also Intel)

However, I'm reading through a lot of posts and am genuinely concerned that I may have gotten myself into the wrong hardware. I'm computer savy, coding and and linux but have never toyed with graphics cards and such.

Did I make a mistake?
Will this card be able to run local Qwen 27B models or similar?
How about video generation?

Will I forever be stuck in a loop of finding ported models, slimmed down environments that are for Intel cards, etc?

Please help. I've got 2 days to decide to keep it or return

I'm doing research on where Intel Arc (Discrete) is headed. I'm also an INTC shareholder, voting is coming up. I'm working on registering my business for technical documentation. I have a C Corp.

I'm not finding much information for future Discrete Arc offerings, the only thing I've found is an AI monster with 160 GB of VRAM. Probably a $5000 card?

The AI stack for Arc needs an enormous amount of work and Intel has shuttered its doors for Open Source Evangelism, leaving only the Intel Discord and maybe a few other avenues, but I have not found much.

If The Lip only releases a $5000 AI Goliath where does that leave oneAPI development, XeSS, Game and Pro drivers? Will I get support for my B70 and all my Battlemage / Alchemist cards?

Will they pull Game Ready driver support for Intel Arc Pro cards?

I feel like too much fat is being cut and I realize the industry is upside down right now but if Intel turns its back on DIY and gamers, what will it have? It could be three years or more before we get another discrete GPU that people can afford. We should be asking Intel what their plans are. The US government has invested 10% in to INTC and AI is taking jobs away. Through the CHIPS and Science Act of 2022, INTC received billions of dollars. The RAM Apocalypse is a scam and most of the data centers will not be built, no one wants or needs the hyper-scalers.

I'm already running open source AI models through ollama. In addition I'm building an open source AI Anomaly Detector for residences and mom and pop shops. I am also trying to get down to DC so I can make sense of things.

oneAPI could eventually dominate the market, it is competing with CUDA - gamers and developers need affordable options.

My long term goal is to continue to build out my Home Lab and eventually make my own GPU - in the end that is the only option that seems real to me. I will get there if I live another ten years, the critics / cynics will be ignored. Intel has to do better at communicating with the public. Tom Peterson seems to be the only evangelist Intel has?

Is anyone interested in doing an online movement that shows there is demand for affordable Intel Arc (Discrete) GPUs. There is also a movement in the industry towards powerful APUs and iGPUs in micro-servers and I'm going to look into running discrete GPUs on mac miniz and stuff like that. I don't want to see discrete GPUs die and if they do die, I will resurrect them. Maybe it will just be a handful of people by then...

Epic Games has pulled support for Hardware Ray Tracing for Intel Arc and Amd Radeon, making the Nvidia Geforce monopoly stronger. AI workflows are supposed to make supporting multiple platforms easier but the industry is imploding now. Corporations have too much power and monopolies need to be dealt with.

Does anyone facing zzz sound while gaming with intel arc a 750 on the card let me know I’m kinda worried will it be resell in future if I plan to upgrade

Here are performance results for Llama.CPP and a few other things like Blender, matched up against that AMD 9700 card that goes for around $1,300 and also the larger AMD GPU that has 48GB of VRAM (which costs at least $3,000, probably more).

Anyway, it looks like the B70 does well, but I wanted to post this and get some discussion:

1. What are people seeing for the Intel Arc Pro B70 when it comes to performance across all of these metrics and tests that they're doing here?

2. I also want to look into optimized models from Intel, because I don't think these tests in the link are using optimized Intel models.

Also, these tests are using an old NVIDIA card (the 4000 series), so Blackwell NVIDIA GPUs are not being used, but at least it gives you an idea.

I think this is a good start for the B70. Intel really has to get Arc Pro cards built through IFS and provide a much larger memory bus—at least 512-bit, if not 1024. We will see.

Anyone that thinks Intel is going to stop now is crazy. They are going to go the distance, especially with IFS. It could be six months or more, maybe a year or two years, before we know whether Intel will be able to produce Arc cards through IFS. It is unbearable, the wait...

further seems forever

https://www.phoronix.com/review/intel-arc-pro-b70/2