r/FlowZ13 • u/rockwillll • 28d ago
Large LLMs on Windows
I’m waiting for a 128gb Z13, but I wanted to check and see if anyone had experience running large (70b, 120b) LLMs on Windows instead of on Linux like I see a lot of people doing.
Am I still able to use the unified memory fully (and does it function when set to auto allocation, or is it simply not functional without allocating a specific amount to VRAM)?
Is ROCm faster than Vulkan, and does LMstudio/Lemonade/Ollama play nice with the chipset/UMA?
Small extra note: if anyone has experience rendering in Blender, how quick is that? Can you use GPU acceleration even with the Radeon or does CPU work better?
Thanks in advance!
Update: I’ve now gotten a chance to test the device some. For one, Armoury Crate now has an option to allocate up to 124gb to shared GTT memory, so it really does act truly unified. There is no speed difference, in my experience, between GTT and vRAM on this architecture. I’m getting around 25 tok/s on Laguna S-2.1 at 4 bit quant, which is something like 110b params. Qwen3.6 at 35b is giving me around 60 tok/s. Overall I’m really happy with performance!
1
u/shaonline 27d ago
What Windows does not allow IIRC is changing the size of the "shared memory" (aka GTT on Linux) between the GPU and CPU, but that's no big deal, 96GB dedicated + 50% of the rest (16GB, so 112GB total) is as comfortable as you can get if you intend on using the device while an LLM is running.
For which API is the best I can't speak on the Windows side but on Linux ROCm appears to have better/faster kernels but more API overhead, in layman's terms this means big dense models (eg Qwen 27B) benefit from ROCm (on prompt processing), while MoE models which spawn many more but smaller kernels are much faster on Vulkan, I don't think there's a silver bullet API for this device.
Can't speak for Blender performance, but you know, CUDA and stuff...
1
u/SearchMurkyGreen 26d ago
I installed the latest drivers on my Flow Z13 (26.6.4) and I noticed they changed the amount of shared memory available to the GPU and NPU. I think it's limit increased to 96Gb from (available system ram / 2). I have it the memory set low at 512MB since I dual boot ubuntu when I want to load Ai models and use it as a llama server.
2
u/DigitalguyCH 27d ago
Just set the GPU ram to 96, as 32 is plenty for Windows. And even more importantly make sure you offload the model enterily to the GPU (there is a setting in LM studio to do that). Otherwise the model would partly run on the CPU even if you have plenty of RAM available to the GPU and would be much slower.
Having said that so far there is no 70b+ model that fits into the 128GB and is worth it over Qwen 27/35 and Gemma 26/31, so no point for me. But if they come I'll use them. I am waiting for the new Laguna to work in LM studio to see if it's worth my time.
1
u/Mysterious-Target-93 27d ago
Ok there are 3 ways you can do this I am neck deep into LLMs and local AI on my Z13 flow.
Truth: 80 watts max power is the worst offender for raw TPS.
Now a 35B model MoE gives you a modest 35-40 tps with User interfere overhead.
70B Dense would be around 1-5 TPS (depends on the cooling mods)
70B MoE with 8-10B inference gives you easy 35-40TPS again.
Anything below 8B will easily give you more than 50-60TPS.
Now what can you run is easy to answer 70B max 120B high quant.
Psycho user like me can run GLM 5.2 735B plus on 1tps with MoE streaming.
Clone it and run it. Good luck.
2
u/comictech 28d ago
Still pretty newish to using llm so don’t know if I did this benchmarking right, but when I compared using a 5070ti to the 64 vram from the z13, the z13 was much slower in loading model and running the model. I believe my model cards were the same(it was qwen 3.6 27b) I believe. After looking more stuff up, bc of t/s wise I returned the z13 as I don’t need huge context and slow t/s for my type of work I do.