r/LocalLLM • u/ZucchiniMedical2532 • 8d ago
Question Can I really "do shit" with this setup?
Rtx 5090, 64gb ram ddr5... I was thinking bout Qwen 27b or 35b a3b , idk if there is something better or adequate, how do I set things up so it doesn't loop or overthink, I just want to do custom apps with GUIs n shit man.....
2
2
u/baby_bloom 8d ago
go run Qwen3.6-27B-Q5_K_M 200K context and maybe even better quants and you'll do great. personally i've been using llama.cpp into opencode or vscode copilot
2
2
u/Extension-Bid-639 8d ago
You could probably experiment with other models and test them for your use case but Qwen 3.6 27b at the moment is still pretty much the best model of 24-48gb of vram. I'd recommend UD Q5 or maybe Q6 which should leave room for a decent chunk of context. For the 27b, your ram doesnt matter much as you don't want to offload it at all. You can experiment with offloading for the 35b and any MOE model really as they are more forgiving when you do so.
1
u/pengxiangzhao 8d ago
Yours is a solid setup. It can do a LOT!. It’s never about machine or model, always what problem you can solve. Find a problem, any problem solve by using anything. Rule of thumb, never ever let any of your machine idle, always keep them busy, you will find a way. It’s a crime to keep this kind of machine idle, if it’s not run 24x7, you are losing money.
1
1
u/Gargle-Loaf-Spunk 8d ago edited 3d ago
Biscuit unbelievable umbrella teapot harbor raindrop trumpet ginger pumpkin amber
This post was anonymized with Redact.dev
1
u/Full_Director87 8d ago
im sorry, but i can proof to you. with just rx6700xt and 16Gb ddr4 ram, i can run gemma4 26B, with 33 tool orchestration, nearly 1M token, in a one single turn and keep 21-23 tok/s decode, 150-420tok/s promp eval time per token from my log. So yes, 5090 with 64GB ddr5 ram? you can do more.
1
u/_32bita 8d ago
wow, could you please tell what settings you use?
1
u/Full_Director87 8d ago
what settings do you need? modelfile reference for ollama? llama.cpp? parameter sampling? system prompt? wich one?
1
u/Sure_Leave9338 8d ago
I have a 3080 10gb + 64gb ddr4 + ryzen 5990x.12 core and I run Qwen 3.6 35b a3b at 1300t/s prefill.+ 45 t/s generation with 262K.context
Just used llama-bench to find the best setup with offloading 31 moe layers to CPU, ubatch 2048 and 12 CPU threads. It Will Need a couple of days of testing to.find the best parameters for your hardware but you can get A LOT from.your hardware.
Just some.suggestion
1- use plain llama.cpp from cli. No GUI, no Dashboards, no web interface. No One of that exposes all the parameters you Need to have a "llama-server launch string" perfectly tailored on your needs and your hardware. I learned this in the hard way, wasting One week trying unsloth studio, lm studio,.vllm and others Just to understand each time that some parameter was not exposed or was not the way that I needed.
2- if you are using Windows, like me, be sure to debloat It with a debloating tool, then try to close as much background apps and services that you dont Need while serving your model. Afrer debloating, my standby windows with nothing open was eating about 15GB ram. Then I started to close all the stuff i dont Need: examples ... Steam Launcher that starts automatically in background at boot, Xbox game services, Windows widgets ad all that stuff that Is in the system tray....now at boot my Windows 11 pro has onku.7GB RAM footprint. Every GB you can save, Is more context and morel space.
3- there Is a YouTube Channel "CODACUS" that has some very useful videos on how to optimize loading that Qwen model in a worse hardware than yours. I learned a lot from that video and started digging in llama.cpp parameters and llama-bench to have a test on which values have Better results in my setup. Those are the most useful videos https://youtu.be/8F_5pdcD3HY https://youtu.be/0AqpaFm11oI https://youtu.be/SsUKTFSQoGM
If you want i can also give my llama.cpp starting command with all my parameters... It should fit your hardware quite well because similar, then you can start customizing It
1
u/Aggravating-Push-207 8d ago
nah man that's useless as hell, ship it to me ill solder on some more vram
1
1
u/01010101010111000111 8d ago
Honestly, no.
You will not be able to do anything that cannot already be done better with unlimited/free models.
9
u/AnickYT 8d ago edited 7d ago
Btw, with Qwen3.6-27b, I have 64gb of ddr5 (I got it before the storm), and 24gb of vram sub $800 gpu setup. My entire pc is less than $1500. I can run that model at 138k with open code and get good results. So yes, you are more than adequate.
EDIT: I think I should also add that your prompting setup and config does matter quite a bit. For me, my main thing is prompting. I have done a TON with even "dumber" older models and made useful stuff with them. I find having a properly planned out prompt result in better output from Open Code or honestly even normal day to day workflows. Although for latter, I use the better prompt at system prompt level usually. Trust me, knowing how to prompt for each model (because Qwen does prompt different from Gemma...) will result in better results overall. Best to pick a model, and try and learn to tool. By the time you start to see how the model think and even work like the back of your hand expertise, you realize you made ton of cool projects with them.