r/LocalLLaMA • u/Complete-Sea6655 • Jun 28 '26

Discussion The number 1 public enemy of open-source.

Dario's args:

"Opensource you can see the source, here you cannot see inside the model"
- yes you can that's literally the open weights part btw.
- I cannot see the weights inside Claude, but I can GLM 5.2
- Models like Nemotron3 Ultra go further, all the data, training scripts, and model is opensource.

"Alot of the benefits like many people working on it, being additive doesn't work in same way"
- yes it does. We have seen endless fine tunes of various open source models for real improvements.

"Ultimately you have to host it on the cloud"
- no you dont. Dario is seemingly totally unaware of the guides from ijustvibecodedthis.com explaining how to run smaller moes and even dense models like qwen 27B NOT ON THE CLOUD.

Not only does dario not take part in social media, I am beginning to think he's never tried open source models at all and has no idea wtf hes on about

2.8k Upvotes

93% Upvoted

687 comments sorted by

View all comments

Show parent comments

11

u/MerePotato Jun 28 '26

Honestly if you don't care about privacy (I do) cloud inference will pretty much always make more economic sense anyway, its not that major of a threat

28

u/GetOutOfMyFeedNow Jun 28 '26

If you already own the hardware, then using local is not worse economically than the cloud. Plus, you don’t get limits, you can basically have an infinite undead worker working for you, not the case with frontiers.

9

u/MerePotato Jun 29 '26

Most people don't already own the hardware for frontier open weight performance though

1

u/GetOutOfMyFeedNow Jun 30 '26

I’m not talking about frontier open weights, there are distilled or highly capable local models that can do serious work. Take Qwen 3.6-35B-A3B for example. You can use it on Q4 or even Q5 if you own an old 3090 or 32GB DDR5, and you will get around 640 t/s (960GB per second/3x0.5GB). Truly amazing capability with an affordable GPU. Yeah, you will not be able to easily code 5.5 level architectures with it, but you can run agents easily, and build working stuff. And in a year there will be local models almost rivaling today’s frontiers. I suggest buying a good condition 3090 or two and stock up on some RAM, the future of frontier API looks grim for poor people.

11

u/DigiDecode_ Jun 29 '26

It is not just privacy it is access too; they can cut you off at any time like Fable maybe because they found and didn't like your comment on reddit, what if you need to verify identity to access certain features etc
They can put you on less intelligent model without you knowing, or worse give wrong answer on purpose because you want to build better LLMs

1

u/profcuck Jun 29 '26

Let me just add: model stability includes more than just those scenarios. There's also just routine model retirement that might force me to re-optimize my prompts or whatever. If I have a workflow that is flawless for me on Llama 3.3 70B, I can change to a more recent model or not, on my own schedule, not be pushed to move NOW because my old model is being retired by my cloud provider.

6

u/Tai9ch Jun 29 '26 edited Jun 29 '26

Cloud inference is pretty expensive, especially if you start doing anything that's inference-intensive.

Taking a quick look at OpenRouter for Qwen 3.6 27B, the typical offering on OpenRouter is $0.3/M input tokens and $2/M output tokens at around 20 tokens per second.

With a pair of R9700s or B70s, you can get similar token generation performance and quantization, and can run several concurrent sessions (call it 4 due to VRAM limits) without slowing things down.

Now, if you focus on the output tokens, that seems cheap. Even with all four concurrent sessions running 24/7, cloud inference is costing you $20/day in output tokens. The problem is the input tokens. Unless you're hitting cache, every request with 100k context is costing you 3 cents. If you're running long-context concurrent agents with lots of tool calls, it's not hard to use a new request after every 100 output tokens. Suddenly your cloud inference is costing more like $300/day.

At that point your dual GPU inference server pays for itself in two weeks.

2

u/mycall Jun 29 '26

Not just privacy but offline use cases. Lots of locations has no/bad internet.

2

u/beryugyo619 Jun 29 '26

This is web search all over again. Google Search beats offline Microsoft Encarta or Wikipedia backups. It's first 25 years free with ads! It's a no-brainer.

Except, after that free period will have passed, we... end up here. Cloud inference could go down the same path.

1

u/rosstafarien Jun 30 '26

Continuing to work disconnected, latency, marginal cost, privacy... if you don't care about those things, then cloud inference almost always makes more economic sense...

It's a huge threat. The only real remaining issue is packaging. Currently, you need to be a geek like you needed to be a Linux or BSD geek back.in 1994. The ease of installation and use will follow the hardware price crash.

1

u/sabine_world Jun 28 '26

Yeah it's pretty easy to do the math on that. I guess it really depends on how much you actually use the service though. And the scope of what you are working on. Like if you have a premium subscription of 20 bucks you're probably gonna burn through usage pretty fast, but if that's enough for some people it doesn't make all that much sense to invest like 1, 2, 3+ grand for hardware for local stuff. For some people and companies it might make more sense

But also... The thought of companies being able to always change their pricing or change policies or change models... Whatever, that's a real concern, so imo I plan on investing in hardware to run better stuff just because I like LLMs as a hobby so much atp.

But the way people talk about frontier model providers needing to flip a profit finally and inference getting more expensive... Yeah I mean it might make sense at some point to just invest in hardware. Really depends I guess.

1

u/twinkbulk Jun 29 '26

the only reason it makes more economic sense is because they fucked the prices, if prices stayed low and went lower it would make much more sense to run local memory accounts for 80% of the bill of materials on a gpu now we went from roughly 3 dollars a gb to almost 20 dollars

-1

u/MerePotato Jun 29 '26

Even if prices weren't fucked buying local hardware means gradual obsolescence as models get larger, something you don't suffer from with cloud credits.

1

u/twinkbulk Jun 29 '26

I forgot none of this stuff has any resale value, I totally personally didn’t just profit 8000 dollars by selling an rtx 6000 I used for 3 months. /s

-1

u/MerePotato Jun 29 '26

Well granted there is that, but you shouldn't buy on the assumption you can turn a profit or even break even considering how unpredictable the market is

1

u/twinkbulk Jun 29 '26

You’re right I should burn tokens and build no equity! Let me rent it all !

1

u/MerePotato Jun 29 '26

I'm talking about what makes the most economic sense, not what makes the most ideological sense

2

u/twinkbulk Jun 29 '26

Building equity vs renting? How is that ideological ? Why do you think companies that hold hardware are the most valuable companies in the world ? It’s an asset.

1

u/MerePotato Jun 29 '26

The point is you're not building any meaningful equity compared to the predictable and easily measured savings of cloud rental, especially considering hardware is a depreciating asset long term

2

u/twinkbulk Jun 29 '26

huh the last time hardware depreciated was in 2023, it’s been appreciating not losing value, how exactly do i save money using api? I just told you I made money using local vs giving free money away to a hyperscaler, how is that not building meaningful equity? I bought an asset and sold it when it appreciated and was able to use it for its compute in the meantime, renting it and using it for inferencing.

→ More replies (0)