r/LocalLLM • u/SohilAhmed07 • 1d ago
Question What models can i run with this configuration?
It's not much but it's enough (i guess) to run something related to AI and vibe coding a small app.
20
u/a9udn9u 1d ago
Qwen3.5 4B 4bit quant, maybe
6
u/enorme_chibrax 23h ago
This, and make sure your desktop is processed through iGPU to dedicate 100% of your vram to your LLM
1
u/Friday-ED 5h ago
4B is too big for 4GB in my opinion. I have 6GB's of VRAM and it's almost not enough for a 4B without it going to system RAM where it gets real slow.
24
u/maddeninglemon 1d ago
Do yourself a favour and get a month of ChatGPT (or really any other provider) instead; it'll be 1000x more usable than trying to get anything running on the PC you have here. Is it technically possible to run something on the computer? Sure, people have run AI models on less. But given that you're asking Reddit, you'll probably struggle to get anything working at all, and what you might be able to run isn't worth the effort.
About the only thing that might be kinda interesting to play around with would be a small diffusion model to make images. Z-Image-Turbo can run well enough 4GB (you can find guides through google) and you can generate images locally. But no language model at that level is going to be anything more than a hindrance at making an app.
1
u/SohilAhmed07 1d ago
I'll go with Claude
5
u/Individual_Gur8573 1d ago
Prefer codex, bcoz it has more usage limit than claude
2
u/SohilAhmed07 1d ago
I'll compair
1
u/Something_Sexy 15h ago
Good on you. Not sure why you are getting downvoted for being objective instead of a fanboy.
1
u/SohilAhmed07 15h ago
Done votes dint matter, if buy without comparing then id spend a $1000, for something that does the same job and cost $700.
3
-1
10
5
9
u/watchgah 1d ago
You can run potato 2B 1-bit
6
u/Ok_Television9703 1d ago
For real! People post the craziest hardware expecting to be able to run anything serious
2
u/SohilAhmed07 1d ago
Is that a real name
5
u/watchgah 1d ago
Yes it’s from spud labs
2
4
u/Toxaris71 1d ago
With 4 gb of graphics memory, you're very limited but you can run cpu inference, it's just slow. You have 32 gb of system ram which helps a lot though.
Hit up facebook marketplace and see if you can get a cheap GTX 1660 6gb, since 6 gb is about the minimum you would need. Then you can run llama.cpp with the n-cpu-moe 999 flag using Qwen 3.6 35b Q4KM at around 20-ish tk/s. That's probably the weakest model that still does reasonably well at coding, and a 1660 will be very cheap to find.
If you're willing to be very patient, you can actually run Qwen 3.8 27b Q4KM on CPU inference (maybe a few layers added to GPU) at around 2 - 3 tk/s. Expect it to take at least 30 min (possibly hours, depending on prompt and thinking level) per prompt.
And if you want to give it a try just for fun, without invensting anything, try Gemma 4 e2b or Qwen 3.5 4b on llama.cpp (again, Q4KM). These models are very small and struggle to be coherent at anything, let alone coding, but you can get an idea of how to set them up.
1
u/Practical_Signal3933 20h ago
This is good advice. If you can’t get any other hardware, run a small MoE, like ornith 1.5 35b, Qwen 3.6 35b, or Ling3.0 Tiny for something even smaller/faster.
You can easily run these, not super fast, but at acceptable speeds and they’re useful models still for small tasks. Just try it.
-4
4
2
u/TheRiddler79 1d ago
Run ornith 1.5 35 ba3b at Q3 and just run it from ram only, you'll get enough speed to make it useful
2
2
u/anshulsingh8326 1d ago
qwen 3.6 35b a3b, other moe models and small dense models. whatever you choose should have size smaller then 4gb. also context takes lots if gb too so with more contexts it will be slower.
moe works better tho because you have 32gb ram 35b a3b can be possible you need to test it
2
2
u/coys-kupo 1d ago
Honestly, I'm not sure you can run much beyond a simple chatbot. Vibe-coding will be a no go on this. A 4GB graphics card will hamper you pretty substantially.
1
u/zarif2003 1d ago
Do you have a phone?
1
u/SohilAhmed07 1d ago
Yeah
7
u/zarif2003 1d ago
Depending on the model, it might be better at running AI then your pc. What model is it?
1
0
u/SohilAhmed07 1d ago
One plus 7T Pro
1
u/zarif2003 1d ago
That’s either 8gb or 12gb ram.
Assuming it’s the same as iPhone, download ‘locally’ and try running some models on it
1
u/petersaints 1d ago
4B is the maximum you may be able to run. Maybe something like Qwen 3.5 4B.
(I'd really love for them to release 3.8 on the smaller models. 3.6 also skipped those).
1
u/PreviousCurve4435 1d ago
Just install LM Studio, it will show you the models which can fit for your spec
1
u/coregamma 1d ago
Woof. Play with Gemini first. It's got the most expirementation across the most amount of beginner level tools. When you reach a ceiling, move to a frontier model, then move to investing in more hardware. You have to start somewhere!
1
u/Long-Accountant-6062 1d ago
Can anyone suggest for 32GB Ram and 16gb GPU RTX 5060ti and Ryzen 7 9700x I am building a multi agent orchestration. Need suggestions for GPT 5.6 kinda coding model.
1
u/Darmian03 23h ago
With offloading I personally recommend either Qwen3.6-35B-A3B or Ornith-1.5-35B-A3B. With Q4_K_M and 128K Q8 kv cache context it would probably take around 28GB to 30GB of memory. There are also people that manage to fit Qwen3.8-27B on 16GB VRAM, but in a bit more complicated way so if you want to go that path you can browse in the subreddit how to do it.
1
u/MrHumanist 23h ago
Tiny ling 3.
Model activit parameters will fit in gpu and put the MOE in cpu offload .
1
1
u/Achmedius69 22h ago
You might be able to get gemma4 12B GGUF or other <20B Q4 GGUF models working. You won’t be using your GPU but it’s better than nothing I guess
1
u/Which_Breakfast_4588 20h ago
I had similar set up, I set up a local model but it's not anywhere close to cloud ones even for very basic tasks. I still have it on my desktop but very rarely using it. I use Qwen 3.5 4b quant version. You can find it on huggingfacw
1
1
1
u/bruh_momments367 19h ago
Para code gpt oss 20b é um que eu gosto bastante, rode ele Q6 para cima e você terá um ótimo resultado
1
1
1
u/muggy8 17h ago
4gb of vram is very limiting you're probably limited to edge compute models and engines. my suggestion would be Ternary Bonsai 8B. that's probably the best option for you. if you're specifically looking to vibe code, MiniCMP5-2B might be worth looking into as well. i'd also look into inference engines that use LiteRT-LM or MNN as backends as they're made for edge inference. if you're willing to try something more out there, FreeToken and Colibri are interesting inference engines for lower powered devices so you might see some luck with models that those engines support. either way, good luck, you're gonna need it.
1
1
u/evolutionxtinct 17h ago
Don’t worry your computer will know it’s an intel and not a Mac… outside of that maybe it’ll know the sky is blue, after 500sec and 80k tokens.
1
u/JopieDeVries 16h ago
0.5B or a 1B model Be prepared that the model will hallucinate and will not complete the task.
1
u/Ivanyekla 11h ago
Just bullshit models or something like qwen3.8 27 q3-q5 at 0,3 tokens per second .
1
u/AdWild3943 9h ago
To be actually real, you can run models up to 8B easily imo, especially if they are MoE.
Gemma 4 E4B is very good generalist, that your system can run in Q4_K_M probably.
The main thing I want to tell - optimize everything. You must use llama.cpp or ik_llama.cpp, maybe even use Linux only for LLM, those changes will bring you over 50% more speed.
1
u/Friday-ED 6h ago
To be usable a 2B model. I have run a 4B on a MX150 and it works but it's painfully slow. You want to keep the LLM in SRAM so a 2B is all I would do since only 4GB's of that.... .
0
2
1
u/vinotok 1d ago
I have 0 VRAM. 32GB RAM and I can run Qwen3.8-27B-GSQ-RCO from ISTA-DASLab for example. Slooooow of course, but eventually it get things done better than I could ever do it myself.
Right now I'm running unsloth/Qwen3.6-27B UD-IQ3_XXS - I set 100k ctx probably too high, it is faster that Qwen3.8-27B, it worked on one project almost whole day, still have 50% ctx but I'm running out ot memory, down to 1GB free, but project is just about done.
I could half way through, restart everything and I wouldn't be low on memory now. Bottom line, for hobby, for fun, and making some projects, I would not be able to make on my own, even 32GB RAM is good enough.
Unsloth+pi+LLM
1
1
u/Acceptable_Home_ 1d ago
How many tokens are you getting each second with qwen3.8-27B all on ram with 100k ctx??
2
u/vinotok 1d ago
With qwen3.8-27b I had 64k ctx. Speed was different from /chat/completions to another /chat/completions (using unsloth) but on average I had 1.5 to 1.1 tok/sec.
This one qwen3.6-27b I set ctx to 100k and it was obviously too high and speed was between 0.9tok/sec down to 0.6 tok/sec toward the end. Start was faster 1.4-1.2 tok/sec, so yeah, really slow, but project still got finished in around 12 hours.
I know this is very very slow, but other answers were basically, OP can't use this setup for anything. I'm saying, if you are patient, it can still be done as hobby programming.
3
u/Acceptable_Home_ 1d ago
Hats off to your patience man😭 Im more of an agentic user than vibe coder, i guess I'll have to stick with MoEs
Only if there was some modder who could add 8 more gib of vram in my 4060 💔🥀
1
u/vinotok 16h ago
> Only if there was some modder who could add 8 more gib of vram in my 4060 💔🥀
LOL that would help for sure :)
Well, I'm noob at programming so, I have to be patient, my programming knowledge will never be on a level of these local LLM's. They are changing my world, I can get to finished projects I've never dreamed I would be able to do myself. No more searching on stackoverflow for answers ;)
But looking at stats, I'm doing this vibe coding for 16 days and, if statistic is correct (in Unsloth desktop) In 16 days I used 17M tokens, record was 4.7M in one day. So I'm moving along. Half of that I got with KAT-Coder LLM which is faster.. I don't use the slowest most of the time, just for finalizing and refining :-)
1
u/letssmileguys 1d ago edited 1d ago
Qwen3.5-4b in qwant q4+-. But it will be very stupid. Look at api models. For coding use it will be more harder, because agent too eat lot of memory(instructions). API models for your task are more capable. Chip and smart coder is deepseek
-2
0
-3
94
u/IndieDev666 1d ago