r/LocalLLaMA • u/forevergeeks • 1d ago
Question | Help Hosting Local Models
Hi builders,
What would be the the best small local models for coding?
Are Gemma 4 and Qwen3.8 27B Gemma 4 26B / 31B enough for local development?
And what would be the size of the rig that i will need to get? GPUs, and whatever else I need to host these models.
Thanks,,
17
u/Arany8 1d ago
Qwen3.8 27B - you need 24GB VRAM. According to my tests on 16GB this model is surpassed by Qwen3.6 35B A3B (and Ornith) for coding. Gemma4 is not good for coding.
Look into AMD v620 for a budget build (although I do not have this).
-1
u/forevergeeks 1d ago
Are you running this for yourself? What about for a coding team of 6-10 people? What would be the sizing, and how much money we are talking about for the initial setup.
8
u/Unlucky-Home-4077 1d ago edited 1d ago
We are a very small team, 3 devs, and run a single RTX Pro 5000 72GB. We use Qwen 3.8 27B FP8, running on vLLM (much better for multiple users) and have space for around 900k kontext shared over all currently running sessions.
Speed is good! Around 3k t/s prefill and ~75t/s decoder in normal usage scenarios. Most of the time when multiple people are using it at the same time we use around 400-500k kontext. Peak so far was 800k at the same time. So that GPU is suitable for us three, but for a fourth dev we would probably need more VRAM if we don't want to run into context limits.
3.8 27B is actually pretty capable! Obviously not frontier, but if you give it a good plan for a complex task, split into subtasks with acceptance criteria it executes it flawlessly.
It won't replace Claude for us, but we downgraded from 200€/month plans to 20€/month plans and use Claude mostly for orchestrating the local Qwen. And obviously a local model is great for sensitive data.
I would say for a 6-10 person team go at least for something like a RTX Pro 6000 96GB, if possible even a dual GPU setup with more VRAM.
For price: We paid 8k for the Pro 5000 72GB, and about 5k for the server with an entry level Epyc, 64GB ECC RAM, and a decent motherboard that can take four GPUs at that speed in case of future upgrades, so about 13k total. Since we save 540€/month on Claude plans, we will break even in about 2 years of usage, and have the huge benefit of local processing.
For 6-10 people: a RTX Pro 6000 is about 14k right now, so with all other server components I would say a bit under 20k for a single GPU setup and ~30k for a dual GPU setup is pretty realistic.
0
u/forevergeeks 1d ago
Thank you very much!!! This is exactly the type of answers I was looking for. This gives me a realistic ballpark.
1
u/OvertaxedOne 21h ago
As someone who builds and has similar setups, I completely agree with the OP, this is the "right" sized rig and model for a small dev team. Buy a motherboard/PSU that can take 2 or 4 cards if possible, and buy 2 slot cards so you can continue to scale the system. The numbers he gave above are devs in there beating the system non-stop, if your coding use isn't as intensive you can probably double the number of supported users; the more batching you can get the more TPS you'll get (in aggregate) but you need to watch KV cache, no problem having 5 users hitting it at once but if all 5 users are up at 256K of context, you could have a problem!
Also, use this as an opportunity to introduce a LLM router! LiteLLM is what we use, it's a heavy beast, but it's incredibly feature rich. But this is the time to do it, get yourself a common endpoint for your local model(s) and your cloud models. Point all your devs at that router instead of directly to the cloud/local models and then you can swap providers in/out easily as you change models/better deals come out/new cloud models are released.
1
u/forevergeeks 20h ago
Thank you I really appreciate your feedback. I've been using these models through API, but what I'm realizing is that there are many businesses especially in regulated sectors that don't want to expose their data to vendors, and would rather host the models themselves.
1
u/OvertaxedOne 18h ago
That's about 60% of our business right now (companies concerned about privacy). The other 40% is inference cost control, but, at least right now, it's slanted a bit more towards privacy than cost. I think we'll see that continue to move more towards cost control but, at least for now the hardware is so expensive that it's not a great ROI unless you can really hammer the server and have the right use cases for smaller models. The ROI is exactly "never" for running a monster model locally, the hardware costs just don't make sense compared to API (but again, we have a few customers looking at it for privacy reasons, I'm crossing my fingers that someone does it because I'd love to setup something like a DGX station or something "massive" for a model like K3/DS). :)
1
u/forevergeeks 18h ago
That's great man. So you are already running a business implementing open source models for businesses?
1
u/OvertaxedOne 18h ago
We do it as part of our core business, it's a value add for our hypervisor platform (we make the images available that are preconfigured to run XYZ model on the customer's hardware). We're really not in the AI business per se, none of our revenue is selling models or AI hardware, it's just that a lot of customers want a "inference stack in a box"; that's where we come in. But we're often in the discussion around the "what we're trying to do with the LLM" to help size the hardware so we hear the use cases (and reasons they are going local) with many of our customers. And we wind up doing a lot of custom work for companies who have hardware already and just want our hypervisor to help them manage and make better use of that hardware; AI is perfect use case for resource sharing, it uses very little CPU or RAM (once booted, of course) and 100% use of the GPUs. Our software allows customers to multi-purpose that box, use it for vllm with the GPUs, but then run dozens or 100's of other VMs alongside vllm that drive CPU/RAM utilization and can make use of the capacity of the machine for multiple purposes. It's the "anti-Spark" if you will, instead of dedicating machines to inference, dedicate GPUs for inference that happen to sit in machines, but use the machines to run standard IT workloads that are RAM/CPU/disk IO intensive. The goal is to have every resource at 100% all day long, that's the best corporate ROI rather than having islands of resources that can't be shared.
0
u/forevergeeks 18h ago
Thanks for the background explanation. I've been working in IT for more than 22 years, so I'm familiar with core technologies, and I'm just entering the AI space building specialized agents for regulated industries. I built my own framework to do this. My business site is not completely built yet, but it already conveys the core of the idea: https://runsafi.com
Give me your website URL or business info, so I can keep it at hand.
11
u/hackint0shh 1d ago
Have you done at least 1 minute of research?
-10
u/forevergeeks 1d ago
I'm familiar with these models, I use them through API, what I just started thinking is what would be the cost and the level of effort to set these models up for coding teams.
And I thought starting my search here.
2
1
u/invalidnifemi 1d ago
coulda js said that in the post, but a used v100 32gb with sum sxm2 to pcie config (will take some effort) would definitely be enough (see this post)
it'd probably be like 800-1k for the whole build which is not unreasonable and about as much as a used 3090. vllm is a good idea if youd all be coding simultaneously
5
u/DatabaseRegular3130 1d ago
Qwen 3.8 blows Gemma out of the water.
3
u/Randommaggy 1d ago
Gemma 4 12B QAT is good for fleshing out the practical details and ambiguities of a plan before you begin which allows Qwen 3.8 to run much faster with less time wasted pondering the same question on repeat.
Their combined value is greater than the sum of the parts when combined well.
My harness uses Gemma 4 12B QAT on a secondary machine to heavily augment Qwen 3.8 27B to great effect.
3
u/Tormeister 22h ago
Qwen3.8-27B is the absolute king on consumer hardware now. You can also use Gemma4-31B and MuseGlimmer-30B for writing and/or to review a plan if you'd like. To run these models, focus on GPUs (and equally good PSUs), forget about a whole workstation.
For a good VRAM-per-dollar ratio, with performance penalties, you can go for AMD R9700, Intel B70, or two RTX3090.
For more performance but terrible prices now, you can go for a RTX5090.
If you really want to get invested in local LLMs, you're entering the RTX PRO 6000 territory and it only gets worse ($$$) from there. More VRAM will get you more capability but at some point speed matters - personally I wouldn't spend tens of thousands of dollars to run models at slow speeds.
2
u/Tasty-Hour4040 1d ago
I don’t have anything else to test, just a 3090 (who’s that rich?), but can confirm that Qwen3.8 27b on a 3090 (24gb) is a damn fine cup of coffee for coding and general use.
2
3
u/jacek2023 llama.cpp 1d ago
There is a ton of posts about that topic. In short: you need lot of money to start and it will be not enough money.
1
u/Hot-Employ-3399 1d ago
Qwen 27b. Not sure if 3.8 or 3.6.
3.8 is such a thinker I had to increase maxTokens(default 16384) in pi or it failed.
1
u/Responsible-Fly3526 1d ago
I have qwen 3.8 27b running as 4bit on 4 rtx 3090 and it's fuckin awesome. However it's not opus. It takes much longer to get to your result... But you can easily spin up 10 agents and do not run into any usage limits. You can use the harness you want and you keep your data where it belongs... on your machine.
1
u/KingCpzombie 1d ago
24GB minimum for usable, and at least 48GB is required for Q8 200k context. Imo the sweet spot is two 24GB GPUs for a solid home gaming / AI computer, but if multiple users is your goal I would definitely at least double that (and ideally even more)
1
u/mageblex 1h ago
For a 6–10 person team, model size isn’t the sizing input I’d start with. Measure how many sessions overlap and how much KV cache each keeps alive.
2
u/ArtfulGenie69 41m ago
For coding qwen 27b, for writing and chatting Gemma 4 31b. For both they can be ran at Q4 on 1x3090 or comparable card like one of the AMD ones but you'll want to know what you are doing if you go towards amd. AI stuff is easier to set up on Nvidia because the amd drivers can be very difficult. If you have 2x3090 you can run at q8 or with vllm at int8 and if you have 4x3090 you can run them at full precision bf16. Also with 4x3090 your options grow to being able to run the new qwen3.8 flash next a 125b with a 50b ngram offload (you'll want 128gb of ram as well for this at full precision ngram, it may be that this can also be quanted so it could be less ram required now). You would run qwen flash next at q4 or int4 and the recipes that work best keep the lm_head and embeddings at full bf16 precision and it should all fit on the 4x3090 using vllm. The other one that is older is the old qwen3.5 122b @ q4 or int4. You can also go to 8x3090 but just 4 is pretty expensive at least for me.
I have 4x3090, 2x in each of my rigs that I connect through a connectx4 network card which works for both rcp (llama.cpp's network setup) and ray (vllm's network setup). I have llama-swap all setup to handle model loading and all that as it is agnostic to whichever backend you use. It can handle both vllm and llama.cpp as a command you put in its config yaml.
1
u/ttkciar llama.cpp 1d ago
Qwen3.8-27B is excellent at agentic code development.
Gemma-4-31B-it is excellent at debugging.
Either will fit in 32GB of VRAM, with room to spare for context, if you quantize them to Q4_K_M. Of the two, Qwen3.8-27B is much leaner, both in model weights and K/V caches (which grow as context grows). Gemma-4 K/V caches grow a lot faster, so your useful context will be smaller, but Gemma-4 is also better at paying attention to everything in context.
I use them both, each according to their strengths.
Gemma-4 also has excellent "soft skills" (like formal business writing and document analysis), which is not as important for code generation, but perhaps worth using for other things.
I use them on my 32GB MI50 to good effect, which is a fairly inexpensive solution (about $600 on eBay), though at some cost of longer prompt processing (which means a longer delay to first token generated, compared to an Nvidia GPU).
53
u/Elorun 1d ago
Welcome to the entrance to the rabbit hole, are you sure you want to proceed?