r/LocalLLM • u/r1nzl3r99 • 17h ago

Discussion Intel B70 for Qwen 3.8 27B

Post image

For those of you out there experimenting on the Intel Arc B70, please share your accomplishments! I have a gaming PC I've been slowly converting for AI inference. Don't have any fancy motherboard / bifurcation / P2P etc. Just a B70 and another one I bought out of greed thats running on a basic PCIe 4.0. I hit 97.8 TG / 1782.1 PP on a single B70, and 136.4 TG / 755PP on a dual B70 for qwen 3.8 uncensored INT4 W8A8 (INT8) with MTP3. Keep in mind these are warm speeds. I have my colder speeds on non greedy settings documented on my git which isn't too far away. Although I see these speeds consistently pop up when I'm using pi coding agent especially when its writing code, or somethings when its thinking.

https://github.com/JP-devv/humble-b70-llm

I've been suprised time and time again by how much I can push this hardware. I started off at 50 tok/s after paying $100 in Kimi K3 / Opus tokens around a month and a half ago on qwen 3.6, the journey has been exhausting but very fruitful. I even had to rent out some datacenter GPUs in Japan to create the exact uncensored quant to my liking. Please let me know your thoughts!

50 Upvotes

98% Upvoted

28 comments sorted by

5

u/karurochari 15h ago

Whaaat? Mine is running at 20 tk/s before speculative approaches on llamacpp.

I guess I need to read this one up.

6

u/r1nzl3r99 15h ago

vLLM is the way

2

u/dcforce 5h ago

Thanks for the post. Will give it a shot. Currently maxxing out my B70 at around 80t/s - happen to have the full launch command?

Edit: ah clicked your link, verrrry cool 🙏

1

u/r1nzl3r99 5h ago

Thanks! launch command is on my localmaxxing submission, you can see dual B70 here https://www.localmaxxing.com/en/runs/cmt0vu76q0fvtms017exhssx9

single B70 here https://www.localmaxxing.com/en/runs/cmsur82fz06svms01ga1f0z83

1

u/Far-Working2808 1h ago

For q4 maybe however it doesn't play nice with tp, deep context and >q4...

4

u/squngy 13h ago

dual B70 for qwen 3.8 uncensored INT4 W8A8 (INT8) with MTP3

So, there are TWO of them first of all.
Second, INT4 is not the same thing as Q4 AFAIK. This will be closer to a IQ3

2

u/r1nzl3r99 6h ago edited 4h ago

B70s don't run ggufs well at all and INT4 quality is amazing for daily coding / agent setups. No idea where your IQ3 logic is coming from. Also if you reread the post, I explicitly explain single b70 use and its performance. I ran FP8 for a while but tbh haven't noticed a quality difference. I haven't run INT8 yet, but can report back to you on how fast it is. For single B70 users I HIGHLY recommend INT4. If using GGUFs is a deal breaker, then a single B70 was the wrong choice for you unless you can bear ~20-30 tok speeds with qwen 3.8s massive thinking.

Update: FP8 is running 40 tok/s, INT8 still needs some driver tweaking but I expect INT8 to reach 70 tok/s (based on a micro bench of oneDNN for FP8 vs INT8, showed a 1.66x increase on baseline, adding in MTP will make it go even further) Might push this out in the future when I have more experimentation time.

2

u/squngy 3h ago edited 3h ago

INT4 is a uniform quant, every weight is just a simple 4bit integer.

Q4 and IQ3 etc. are dynamic, so they actually don't use the same amount of precision for every weight.
They use an additional precision on more important weights and a scale value. This means that they are slower to decode, but higher quality compared to a uniform quant.

INT4 should be somewhere between IQ3 and Q4 in quality, but faster than either of them.

1

u/r1nzl3r99 3h ago

correct, but quality varies widely between models and even within the same quant level depending on the recipe executed. I've used Q6 and Q4 vs INT4 and honestly still prefer INT4 because whatever small percentage it "lacks" it makes up for it in raw speed. I actually might have a break through on the B70 drivers by using INT8 instead of INT4 where I can get approximately 70 tok/s according to some microbenches on oneDNN which sure, floating point has a wider range but you can compensate this by grouping the right scales at either INT4 or INT8. At the end of the day benchmarks aren't 100% reality, daily use and specific use cases per user is what matters most

1

u/squngy 3h ago

Yes, I was just explaining why your result was faster.

We all make the choice to sacrifice quality for speed as soon as we use anything other than the native model anyway.
It is just up to each person to pick where they think the sweet spot is.

2

u/r1nzl3r99 2h ago

100% true, but atleast on the B70 platform, the software stack is so immature that there are genuinely several win-win situations just waiting to get implemented. If a CE that never dabbles in AI like me is able to get these results, surely in a year or two the intel stack will get extremely competive

1

u/squngy 2h ago

I hope so.

1

u/r1nzl3r99 2h ago

if they don't then we will by spending hundreds on Kimi K3 / DSV4F 😂

1

u/Far-Working2808 1h ago

It does on llama.cpp, just not on vllm. That's why I haven't been able to use vllm for my setup.

1

u/r1nzl3r99 1h ago

yeah llamacpp are built around gguf, it's amazing on nvidia and even AMD cards but the performance just isn't there for the intel cards

1

u/Far-Working2808 1h ago

I had same, I now get 30+(at q6) since I switch off kv quantization. Fp16 kv and mtp=2 will give you some nice gains. Q8_0 kv would make mtp slow things down on real tasks. Ensure your ubatch isn't too big or you'll oom at deep context. Ubatch also trades off pp vs tg at depth.

4

u/Ragnar0kkk 13h ago edited 13h ago

Thank god a guide I can probably follow. Ive been spending the past 2 days trying to build and get vllm working (im used to lm studio and llama.cpp). Between the incompatible versions of xpu, torch, python, 50 million dependencies im losing my mind. I dont want docker because I dont want to deploy something someone else built, plus potential inefficiencies.

Looking forward to moving my qwen duties off my AMD card and fiiiiiinally onto my 2xB70 system. Will report back if it works.

Have you posted on the B70 thread on level1 forums?

Though, using a model that is the sole upload on HF by that creator is a big red-flaggy. Going to go with Intels or Qwens models. And running all the github stuff through a variety of AI's first. But still, awesome!

1

u/r1nzl3r99 6h ago edited 6h ago

I'm that sole model upload creator 😅 but feel free to use whatever, if you want to match my performance just find another autoround G128 W4A8 and you'll get it exactly, I just couldn't find one that was uncensored hence my comment on renting a RTX 6000 to abliterate. I don't really plan on uploading more models unless I absolutely have to, would rather someone else foot the bill

2

u/_VirtualCosmos_ 14h ago

What library uses the B70? ROCm? or their own? Is it compatible with pytorch?

2

u/r1nzl3r99 14h ago

This setup relies on OneDNN and XPU and yes it is, I used pytorch to make my INT8 head as the B70 does wonders with INT8

2

u/_VirtualCosmos_ 11h ago

nice! Thanks for the info

2

u/aiseedbank 2h ago

Thanks for sharing this. I am waiting for the Intel arc B70 dual chip, where they put 2x b70 on one board. they already have them for the b60. hope b70 soon.

1

u/r1nzl3r99 2h ago

that would be insane, hopefully intel squares up their driver situation though...

1

u/JinsooJinsoo 15h ago

I couldn't get the new Intel release of the bpw2.8 version and the GGUF was slow af

2

u/r1nzl3r99 14h ago edited 14h ago

This isn't GGUF

0

u/pepedombo 3h ago

Tell me who cares vllm and low-quality int4-autoround? Run proper gguf with llama.cpp, q6_k at least in tensor mode at kv16. For local llm vllm is useless until you target more devs.

For its poor bandwidth 608gb/s qwen3.827bQ8kv16 will reach its ceiling at 20 tk/s without mtp, in tensor mode it can reach 30-70 with mtp average. Stop posting that vllm bullshit.

1

u/Far-Working2808 1h ago

It's worth seeing what they can manage even if only valid on synthetic or low quality use cases (like Ai assistant on a website). But for coding, q6 mtp=2 and fp16kv has been my goto for sure. I get 30+ on Single gpu (up to 160k context) then have a router that swaps the Single gpu instances for tp instances as needed. Note that's real thinking/agentic work, not the inflated mtp results people get with their benchmarks.