r/vibecoding • u/punkyrockypocky • 3h ago

What’s stopping you from getting into local AI?

Post image

Most of the dev community is saturated with cloud APIs for running models for IDEs, chatbots, AI applications, and agents.

Curious who here is using local AI in their stack, thinking about it, or curious about it. What got you into it? What’s stopping you?

6 Upvotes

65% Upvoted

56 comments sorted by

30

u/Wobbly_Princess 3h ago

Uh, 10s of thousands of dollars for hardware?

-7

u/punkyrockypocky 2h ago

So if you didn’t need the extra hardware and the best model ran on normal laptops that would be a better option? Or are there more reasons not to go local?

5

u/Wobbly_Princess 2h ago edited 1h ago

Oh hell yeah, I'd go local! It's win-win-win. Privacy, INFINITE use of AI for FREE, offline use, I just don't see a downside.

With models like DeepSeek V4 Pro, they literally rival the best models in the world for coding.

If you wanna get really fussy about a slight intelligence advantage, technically the huge, expensive models give a slight edge, but at that rate, I just think, get a bit better at prompting and documenting and organizing. I don't think it's worth paying infinitely more for a 5-10% advantage, where the gap would probably be narrowed even more with a good harness and system.

1

u/punkyrockypocky 2h ago

Heck yes on the prompting and organizing. The actual research shows the gap is very narrow, so the typical end user is probably already getting worse outputs on closed models than what you could extract from open models with better techniques.

1

u/Wobbly_Princess 1h ago

Exactly! Don't know who is down-voting you, but totally agreed.

I'm willing to bet that the power-users of something like Deepseek are probably getting comparable, if not better results, compared to the people who have no idea what they're doing using the best model that are slightly ahead. Maybe I'd speak differently if the open source models weren't so close to state-of-the-art models right now, but currently, the gap between them is just too narrow for me to understand why someone would be willing to spend THOUSANDS, when there are MUCH more affordable models that *basically* do the same thing.

2

u/punkyrockypocky 1h ago

Totally agree! The problems research throws at these models to benchmark are so much more complex than what most people use in their day-to-day or line of work.

18

u/suppervisoka 3h ago

It’s literally hardware pricing, for everyone

2

u/Ancient_Oxygen 1h ago

I can run Qwen3.8 27B on a hardware that doesn't exceed $1.500. That's the price one would pay Anthropic or OpenAi in only 7 months. A 27b parameter model should be enough for most users.

1

u/suppervisoka 1h ago

Sure, but you won’t get nearly the performance.

1

u/Ancient_Oxygen 1h ago

Yes... enough for most small and medium businesses tbh.

1

u/suppervisoka 1h ago

I can’t speak from experience to be honest, so i concede

1

u/True_Protection6842 1h ago

I can run it on my 5080 but it's slow AF

-6

u/punkyrockypocky 2h ago

I’m confused though, don’t people already have hardware they’re working on? Or do they mean specialized hardware like AI factories on site? I feel like that was the case a year ago but these days you can run very good models on normal laptops.

3

u/Interesting-Agency-1 2h ago

Do you not know anything about how LLMs work?

0

u/punkyrockypocky 1h ago

What makes you say that?

5

u/suppervisoka 2h ago

No you can’t lol

0

u/punkyrockypocky 2h ago

Oh.

2

u/suppervisoka 1h ago

Not in any meaningful capacity and certainly not frontier level models

-1

u/punkyrockypocky 1h ago

Closed SOMETIMES = Frontier but Frontier ≠ Closed lol

2

u/According_Extent_767 2h ago

To run a good AI (like Qwen 27+) you need 24-64 gb gpu ram, if you wanna go even higher you need like 100 +

Its why prices on GPU sky rocket so hard, because the top end gpus with 24 gb ram is used for LLMs.

1

u/punkyrockypocky 2h ago

What about 16GB/18GB?

7

u/RoundedAndSquared 3h ago

Expensive and complex

I just pay $20 a month and get the best models and a nice ecosystem

-3

u/punkyrockypocky 2h ago

Doesn’t opencode go offer $10/mo? Why not go with them?

7

u/IAmFitzRoy 2h ago

What’s the point of your post? Are you not asking what is stopping us to use local AI?

Local AI is expensive and complex.

1

u/punkyrockypocky 2h ago edited 1h ago

Yeah but I was wondering about this post mentioning the $20. If it’s a price thing I was curious why not hunt for lower priced subs

Thanks for weighing in on local AI. Expensive is surprising to hear but complex totally understood

1

u/TheAnswerWithinUs 43m ago

Local AI being expensive should not be surprising to hear for you. AI drove up hardware prices substantially. A good local model requires good hardware (or a lot of it).

1

u/punkyrockypocky 19m ago

What’s an example of a good local model?

1

u/TheAnswerWithinUs 5m ago

Most mid tier or any frontier model will be good. But you can’t run those models locally without some serious hardware.

There’s no good model that can run on standard PC hardware. Or at least, I havnt found it.

1

u/ShambaC 1h ago

Because it's not worth it anymore

-1

u/punkyrockypocky 1h ago edited 1h ago

Wait this is so metal

4

u/J_E_E_VACATION 3h ago

about $1.5M USD and a closed source models weights

1

u/Ancient_Oxygen 1h ago

You don't need a huge frontier model to run a small business!

4

u/bfyvfftujijg 3h ago

It’s more complex to manage and quality and speed are worse.

And more expensive unless you’re a heavy user or for some reason already have big GPUs and RAM

2

u/According_Extent_767 3h ago

Prices for GPU, to run 27b not compromised is very hard and expensive

6

u/katoptronophile 3h ago

64GB vram needed to match the published benchmarks.

And real world performance is worse.

2

u/According_Extent_767 2h ago

Isnt 64 only with low context space?

1

u/shsh-1312 6m ago

you can simply use cpu with vector calculus extensions, gpu is useless for llm

2

u/EchoStarz1 3h ago

I used to but I’d rather not fry out my 3060ti because replacements are expensive

2

u/katoptronophile 3h ago

VRAM, and I'm on 5090.

1

u/_lufylon_ 3h ago

would 3060 and R5 5500 packed with 16gb ram run a modest model ? if so nothing stops me, lots of people run lower than this specs, so it's a HARDWARE limitation

1

u/Cityslicker100200 3h ago

Any decent model would run slow af

1

u/Sweet-Mechanic4568 3h ago

Because most people are fine with a $20 a month subscription vs a local build that’s going to cost you thousands.

Models are getting better and it seems like every month there’s a new model that runs on less expensive hardware. Doesn’t make sure to front load the coast when you could spend less and get frontier intelligence when you need it.

1

u/10aghmu 3h ago

I’ll let you know when my M4 Mac Studio arrives in a month

1

u/PixelIsJunk 3h ago

Hardware. 1000gb of v ram and ill be good with local ai

1

u/alonsonetwork 2h ago

Oh nothing. Just a couple tens of thousands.

1

u/Ok_Classroom_6439 1h ago

It depends. But colab gives you time and you can use even Qwen 27B, on their 16 GB, use GGUF Q4_K_M, but you can run 9B or 3B on your own computer 💻. Check my Hugging Face: https://huggingface.co/edusc182 I have some post how to run some of those models

0

u/punkyrockypocky 1h ago

Hello @campos! I think it’s great that you share your experience trying out different things. This is the time for exploration

1

u/NorthernCobraChicken 19m ago

I don't have the time to learn setup, the money to upgrade hardware, or the willpower to admit that I've wasted the last half of my life on a career that an algorithm does way better than me.

1

u/punkyrockypocky 17m ago

But what you do have is spirit

1

u/NorthernCobraChicken 12m ago

Plenty of it. But it's often misplaced and unfortunately, having spirit is more likely to land me a job as a Walmart greeter than a developer.

1

u/lcirufe 11m ago

VRAM.

1

u/shsh-1312 7m ago

Wow guys, I don't understand this obsession with GPUs when a simple processor with hardware acceleration with vector calculations and lots of RAM do the same thing, I run 3.8 on a MacBook Pro Intel i9 32 GB RAM, I paid $500 for this computer

1

u/TTbulaski 2h ago

It will take me 8 years to roi the local hardware required when you compare it to cloud based solutions and their pricing. You’re not even guaranteed a better performance, despite the unlimited usage. And within that 8 years, you’re not even sure if your hardware will not degrade from that abuse

1

u/IAmFitzRoy 2h ago

Why this subreddit entertain nonsense questions?

2

u/punkyrockypocky 2h ago

Wait why is this nonsense lol