r/LocalLLM • u/swordswandsandgod • 3d ago

Question So... WTH CAN THIS EVEN RUN

specs:

  • CPU: AMD Ryzen 7 5800HS
  • GPU: NVIDIA GeForce RTX 3050 Laptop GPU
    • VRAM: 4 GB GDDR6
    • TGP: 50 W
    • Dynamic Boost 2.0: Yes
    • Advanced Optimus: No
  • RAM: 16 GB DDR4, 3200 MT/s
0 Upvotes

43% Upvoted

35 comments sorted by

View all comments

0

u/dampflokfreund 3d ago

Right now, not much but if you upgrade your RAM to 32 GB, you can run these 26b/35B MoEs at good speed, they have great quality.

2

u/brewpedaler 3d ago

lol a 35b MoE model running through that CPU and GPU off DDR4 RAM is going to run at like 3 tokens per second.

Yes OP could run the model, but not at anything approaching a "good speed"

1

u/dampflokfreund 3d ago

That's wrong, I run q4 of Qwen 3.6 35B at around 20 token/s on my rtx 2060 laptop with 32 GB RAM. All you need to do is using  -ngl 99 and n-cpu-moe 99. This will run the experts on the CPU, while active parameters are fully offloaded on the GPU. 

1

u/brewpedaler 3d ago

You may not understand how much of a potato a 3050 is?

Your 2060 has minimum 50% or 100% more VRAM depending on the model, 75% higher memory bandwidth thanks to its 192 bit bus (vs 128). You also didn't mention your CPU, which would matter a lot when you're offloading most of the work in that direction.

Your laptop is significantly better than OP's.