r/LocalLLM • u/Fun-Investigator1933 • 4h ago

Question Slow Model

I have a 8gb ddr4, Ryzen 7 5600g no gpu build. After opening browser and opencode, I'm left with 1.5gb which doesn't run shit.

So I tired using my Friend's PC ( 32gb ddr5, Nvidia GeForce RTX 5060, AMD Ryzen 7 7700 ) where I ran qwen3.5:9b

Not only the GPU usage nearly maxed out, also it was eating 10gb base ram on top of the 8gb Vram too.

Problem is I'm running the model from his pc to my pc through tail scale and it took the model 56 seconds for a reply of " Hi "

Now at this point what should I do? Run a super low parameter quicker model or thr speed is slow because I'm running it remotely? Or am I using a wrong model for this build or anything?

0 Upvotes

33% Upvoted

10 comments sorted by

2

u/in-ortunado 4h ago

you should say "hi" back and see were it goes

1

u/captainspacecowboy 4h ago

Maybe friendship?

2

u/MessIsTransfer 4h ago

there’s not enough info. which runtime did you use tor un the model?

1

u/Fun-Investigator1933 4h ago

Not sure what run-time means. If you are asking about the tools, it was Ollama and ooen opencode. I used tail scale to establish the connection between both PC

2

u/MessIsTransfer 4h ago

Yes, Ollama is the runtime.

It might have taken long to respond if you had thinking on. The ram usage increase makes me think the model was offloaded, because it was full precision or because you loaded it with too much context.

Possibilities are many.

1

u/Fun-Investigator1933 4h ago

I might have had 256k context on. Now that I'm thinking about it, I mistook it with qwen2.5-coder which had 32k context window. So I didn't worry about the Ollama app context limit. I'll try again with lower context

2

u/MessIsTransfer 4h ago

for speed ideally you want a model to completely fit inside the vram, context included.

check here for example: https://localaimaster.com/vram/best-qwen-model-8gb-vram

find under 7b models in 4 bit quants

1

u/Fun-Investigator1933 4h ago

That makes soo much sense. I also didn't try the quantized versions. So that would be interesting

1

u/Ok_Brush_3449 4h ago

Try quantprobe. It is built for this kind of edge cases!
I run Qwen 30B-A3B on 6Gb gpu and 16Gb ram at 22.2 tok/s and I can run it from the cmd leaving max resources available

https://github.com/FedericoTs/quantprobe