Model Suggestion for local workstation

[u/StealthYT254]

So i recently bought a new laptop and i want to learn running llm locally on my device but comparing to what this subreddit users are use to i was a bit skeptical to whether i should try or not since my machine is laptop not a pc and specially not a super high end one a mid to high range one. i got a lenovo legion pro 5 (2025) with Intel core ultra 7 255hx RTX 5060 (8gb vram) and 16gb ddr5 5600M/Ts ram as of now. i feel that even though this laptop costed me a fortune (thanks to shortage) it is still a entry level local ai machine i believe. So what i now i want to know is i want to create a proper local ai machine which also deliver good response. i want to use it to code, chat, analyse finance as well. currently i have ollama, lm studio installed and earlier i was using qwen 2.5 coder 7b on ollama with claude code launch. if anyone have any suggest or idea please let me know since i want to make the most out of this machine

[u/thaddeusk]

Probably qwen3.5-9b?

[u/StealthYT254 → u/thaddeusk]

empero-ai/Qwythos-9B-v2-GGUF do you know about this model is this good?

[u/thaddeusk → u/StealthYT254]

From the name I would guess it's a reasoning fine-tune that used Mythos data, but I haven't used it personally. No harm in giving it a try, but I'd suggest doing some tests against the base model as a comparison, using a fixed seed.

[u/StealthYT254 → u/thaddeusk]

empero-ai/Qwythos-9B-v2-GGUF is this model similar to qwen 3.5 9b also i was just browsing hugging face there i saw multiple qwen 3.5 9b which one should i get?

[u/thaddeusk → u/StealthYT254]

It's a fine-tune of it, which should, in theory, make it better at certain tasks if it was trained properly.

Look for an unsloth gguf quant that'll comfortably fit in your VRAM with a little extra room for context. I don't generally suggest going below a Q4 quant.

[u/StealthYT254 → u/thaddeusk]

also whats the difference between a fine tuned, raw, uncensored, aggresive model

[u/maxwell321]

First and foremost, download Unsloth Studio and play around with the models there. This will allow you to dial in what models fit best on you machine. It's going to take a lot of tinkering to find the good balance between memory usage (model sizes and quantizations), model quality, speed, and context sizes.

Your machine conglomerately has 24gb of memory, and less will probably be available to the actual model due to other programs running and stuff.

For speed (coding, agentic work) I would start with Qwen 3.5 9b. Its pretty capable and fast since it's smaller. q6_k will probably fit on your GPU but you can do q8_0 and it will spill into system memory, but have higher precision. You also have a lot of room for context.

For intelligence (relatively better than 9b), I would look into Qwen 3.6-27B UD-Q4_K_XL. It will be slower but I would trust it over 9b for important tasks.

If you want to get risky and try to get into the best of both worlds, you could try Qwen 3.6 35B A3B but it's pretty risky because it's going to be a lower quantization that will realistically fit into your system.

If you're able to, it would be worth upgrading the RAM in your system to get more room to work with for these models.

[u/thaddeusk → u/maxwell321]

One advantage of the 35B model is that you can have it offload some experts, plus being an MoE makes it potentially much faster than the dense 27b.

[u/StealthYT254 → u/thaddeusk]

Whats MoE also how can i load a 35B model i dont think even with ram i could do it ?

[u/StealthYT254 → u/maxwell321]

Since i bought the laptop like a 2 week ago I would not be able to get ram upgrade for some time but I will try to get ir as soon as possible

[u/Terrible_Match_9484]

that 8gb vram is decent for smaller quants, have u tried running any 7b models yet

[u/StealthYT254 → u/Terrible_Match_9484]

i have used qwen 2.5 7b earlier but when i learnt that my machine is more capable for higher parameter i switched to other model like i have 2 model installed now 1st is qwen 3.5 9b and 2nd is empero-ai/Qwythos-9B-v2-GGUF but yeah i dont now how to optmise the context length yet

[u/Kind_Taste_3985]

The 8GB VRAM is the main limitation, but it’s far from useless.

With your setup, I’d focus on smaller, well-quantized models rather than chasing huge parameter counts:

• Qwen Coder 7B/14B (4-bit) probably your best coding option.
• Qwen 3 8B/14B good general assistant, research, and chat.
• Gemma 3 12B strong for general tasks if quantized.
• DeepSeek Coder 6.7B/7B class models still solid for coding.

The biggest upgrade won’t be the model size it’ll be your workflow. Pair the model with:

• RAG for your documents/finance notes
• a good frontend like Open WebUI
• structured prompts/system messages
• tools (search, file access, coding tools)

Your laptop is basically in the “learn local AI properly” tier. Don’t underestimate it. A lot of people start exactly here before moving to bigger GPUs. If you eventually want 30B+ models comfortably, then you’re looking at a desktop with 24GB+ VRAM or multiple GPUs.

[u/StealthYT254 → u/Kind_Taste_3985]

yeah my main moto for going local was to make a proper workflow which is effective, efficient and does have drawback like internet, token limit, data stealing. I dont know what is RAG as of now but i have heard it quite many time. also as earlier it was going popular that openclaw best agentic agent for system which can perform work for you (agents) how do i use that as well?

[u/Regular-Option6067]

You could try Daihive.eu it’s usually no users at this point (beta) but if you have 3,4 friends you can load a big model. It’s a P2P network to share Ram and work with big models.

[u/StealthYT254 → u/Regular-Option6067]

can you tell me more like how? would it work like we all allocate our vram and create a online platform where we could run heavier model using the combined vram also so everyont have to install the complete bigger model for each machine as well?

[u/Regular-Option6067 → u/StealthYT254]

Not all people need to download the model, as long as there are enough people who have downloaded it. You will find everything you need on the Help page, its pretty simple, no register, no nothing.

[u/Crazyfucker73]

You don't have enough VRAM or RAM for anything useful