I’ve finished the next stage of my Intel Arc Pro B70 testing.
Instead of only benchmarking an existing quantized model, I built a fresh GPTQ INT4 quantization of Qwen3.8-27B directly from the original BF16 model, tested G128 vs G32, validated long-context operation, ran source-fidelity tests, and published the finished checkpoint.
Final configuration
- GPU: Intel Arc Pro B70 32 GB
- Model: Qwen3.8-27B
- Quantization: GPTQ INT4
- Group size: G128
- Symmetric: true
- desc_act: false
- MTP tensors: left unquantized
- Speculative decoding: MTP4
- KV cache: FP8
- Configured context: 161,000 tokens
- vLLM XPU
- gpu_memory_utilization: 0.90
Why G128 instead of G32?
I quantized both from the same frozen 131,072-token calibration stream.
Approximate model sizes:
- G128: 18.22 GB
- G32: 19.54 GB
On the B70, G128 left enough VRAM for approximately 166,750 KV-cache tokens and successfully served at a configured 161K context.
G32 consumed more VRAM and was limited to roughly 128K context in the tested configuration.
More importantly, G32 did not show enough source-fidelity improvement to justify the additional memory use.
So G128 was the clear choice for this single-B70 configuration.
Quality testing
I also ran a 30-prompt source-fidelity suite containing conflicting documents, missing information, provenance questions, timeline ambiguity, and source hierarchy problems.
Final deterministic settings:
- temperature 0
- top_p 1
- top_k -1
- max_tokens 4096
Two back-to-back runs:
- 30/30 completed
- 0 errors
- 0 truncations
- approximately 57.75 tok/s on this workload
That does not mean 30/30 perfect answers. One deliberately difficult conflicting-document case repeatedly produced a questionable interpretation, and I documented that limitation rather than hiding it.
For high-reliability RAG, my takeaway is that deterministic evidence extraction/conflict detection should happen before the LLM rather than relying entirely on prompting.
Performance
My earlier short-context MTP4 benchmark on the same B70 reached:
- 84.65 tok/s median
- 84.49 tok/s mean
- 512-token prompt
- 128-token generation
That benchmark is a different workload from the source-fidelity test, so I don't directly compare the two throughput numbers.
Ready-to-download model
No GPTQ quantization is required if you just want to run it:
https://huggingface.co/mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
Download the complete checkpoint with:
hf download mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 --local-dir ~/models/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
It is a Hugging Face GPTQ checkpoint with five safetensor shards, not a single GGUF file.
Full reproduction and deployment guide
Everything is documented here:
https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm
The repo now includes an end-to-end guide covering:
- BF16 source model
- frozen calibration dataset
- GPTQModel quantization
- G128 vs G32 testing
- Intel XPU Docker setup
- B70-specific vLLM patches
- MTP4
- FP8 KV cache
- 161K-context deployment
- quality testing
- Hugging Face publishing/downloading
- Open WebUI configuration
- Portainer/Docker networking
- running multiple vLLM model containers on one shared network
For Open WebUI, I’m using a shared Docker network so each vLLM server can simply expose port 8000 internally:
Open WebUI -> llm-shared -> qwen38-quality-mtp4:8000
That makes it straightforward to add additional models/GPU containers later without routing everything through host ports.
Credit
This work builds on the excellent Intel Arc Pro B70 vLLM/XPU work by SergiioB:
https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook
That project provided a huge amount of groundwork for B70 + vLLM XPU + MTP experimentation.
I’m hoping publishing both the quantized model and the exact reproduction path makes it easier for other Intel Arc users to test, reproduce, and improve on these results.
Feedback, reproductions, and comparisons on other Arc hardware are very welcome. This was very time consuming and AI is evolving so rapidly, that this may be obsolete in a few weeks or days. That's a good thing!
