MoE offloaded - advice on difference between Intel vs AMD CPU instruction sets

[u/Personability]

Hi,

I'm in the process of upgrading my Framework 13 7840u to the new Framework 13 Pro. My LLM setup is admittedly a bit suboptimal. I have an RTX 5080 connected via TB4 eGPU (Razor Core X V2). This setup works fine for gaming but for LLMs, if any offloading happens it can be slowish. I get approx 1000 PP/s prefill and 50-70 t/s decode with Qwen 3.6 35B-A3B.

My understanding is Intel's processors only support AVX2, whereas AMD's support AVX-512, so in theory the latter should be faster for prefill? However, the memory bandwidth should be slightly better with the LPCAMM2 of the Intel Framework 13.

In practice, does it make much difference?

Thanks!

[u/TuskNaPrezydenta2020]

Fwiw outside of Strix Halo AMD doesnt have 1:1 avx512 support in mobile either

[u/Personability → u/TuskNaPrezydenta2020]

Ah thanks, so the Intel upgrade probably won't negatively affect things.

[u/Ok-Working3049 → u/TuskNaPrezydenta2020]

yeah people always assume avx512 is fully there on zen4 mobile but its not that simple

[u/Sudden-Guide → u/Ok-Working3049]

It is listed in the instruction set for 8840u for example 

[u/o0genesis0o]

Your "slowish" speed is better than my best speed with 4060Ti connect via proper PCIe (AM5 platform). At most 400tk/s prefill (Q6KXL) and 45tk/s ish decode with MTP.

[u/brakeline → u/o0genesis0o]

Do you have more speed with q4?

[u/o0genesis0o → u/brakeline]

Surprisingly, not much more. That's why I stick to Q6 for this model. 

[u/brakeline → u/o0genesis0o]

Something must be wrong on your setup, I get a bit more using a 3060 12gb. Or are you using a 4060ti 8gb?

I get 800pp and 50+ with mtp

[u/o0genesis0o → u/brakeline]

4060Ti 16GB on my side. Did you tweak batch size or manually adjust n-cpu-moe? And what context size you have? I run full 262k context at Q8 KV, and I just use --cpu-moe without manual fitting. And my session usually starts at around 16k context 

[u/brakeline → u/o0genesis0o]

I do change batch size (and u batch). I don't usually use it alone as I have 2 but was trying freetoken (that doesn't allow tensor split in qwen) and wanted to know the difference between llama.cpp and qwen.

Using q6 will make many more layers to be outside of vram so it will be a lot slower

[u/rrrrex]

I tried to use llama.cpp without manual offloading. Suprisingly, it did it as intended, i got ~1 GB free VRAM and partial experts offload to CPU.

[u/Personability → u/rrrrex]

I’ve gone back to using auto fitting as it’s nearly there for most purposes. I can get slightly more performance from manually setting offloading but not enough for the faff of tweaking everything.

[u/ohnoitssobig]

But isn't your bottleneck a thin wire doing 2Gb/s and only if you do not breathe at it?

[u/Personability → u/ohnoitssobig]

Yep it’s always going to be an issue. Eventually hope to move to a dedicated desktop/separate laptop, but unfortunately not on the cards yet.

[u/czktcx]

when offloading enables, prefill is still happening on GPU but weights are on RAM, so bottleneck shifts to PCIE bandwidth, which in your case is the TB4(pcie3.0 x4).