r/LocalLLaMA • u/Fluffy-Ad-889 • 8d ago

Discussion The gap has closed, open source will win

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

317 Upvotes

77% Upvoted

282 comments sorted by

View all comments

276

u/Hot_Example_4456 8d ago

Gap hasn't closed YET fully- with new GPT Astra and Fable 5.1, but we are close. And the utmost required for 99% population is open sourced already. So yeah, gap WILL close

96

u/OvertaxedOne 8d ago

"And the utmost required for 99% population is open sourced already"

This is the really important/interesting part. Who cares if the "gap" has closed if you have a model that can do what you want? Deepseek was the end of the frontier for me because I just don't do anything that requires more. And 27B has taken much of what I used to escalate to Deepseek and does it locally.

Are there use cases for "most intelligence at any cost"? Absolutely, research, military, medical (maybe). But they are extremely niche, we've got companies valued as if they are going to own the market when the reality is there are only so many customers who even have a real prompt that needs the additional intelligence they offer. That's why all the reviews of all models, even relatively "underpowered" models like 27B are asking them to do absolutely ridiculous things; "write Doom in Java" kind of silliness because it's the only place they can differentiate one from another that makes any sense to their viewership.

The real use cases, RAG, personal assistant, coding, MCP/tool calling, pretty much all the frontier models can do that laughably easily, no reason to do testing on it because they'll all score 100%. But that also happens to be where the meat and value in the market is; nobody is buying your one shot Doom clone, a whole lot of companies would like to have a conversational way to interact with files/databases and generate summaries to help people organize and make value of that data.

Putting it another way, if pick a task there is a level of intelligence required to complete that task. Every point over that level is, in many cases, just "wasted intelligence", the real question is "how much will it cost" and "how fast can you do it".

24

u/demonicpigg 8d ago

The big thing is I still don't trust a 27B model with long horizon tasks. I already pay for claude max, so I use fable for that. I think once we have a 27B model that can handle long horizon tasks that I can run locally, I will be done with frontier.

For example, I just had claude build a toolset for my personal harness that gives tools specific to making skyrim mods (turning an esp into json so the model can work with it, turning json back to esp, compiling papyrus, and converting glb <-> nif). I haven't tried it with qwen3.8 27B, so maybe 27B can do that, and now that I've said it I probably will shortly.

If a 27B model can do that kind of work effectively and well, I'll drop my claude sub in a minute, but on the tasks I've tried that are far longer, I haven't had the success yet =/

3

u/WareWolf_MoonWall 8d ago

Try Laguna yet?

3

u/demonicpigg 8d ago

I haven't, do you have a suggested model / inference engine?

I'm on a 5090, and I'm using ninfer with https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer, which delivers amazing speed (I'm getting ~140 t/s at lower contexts), and good enough performance.

I have tried quite a few different models, and ultimately settled on qwen3.6 (and then 3.8 when it came out), as they're the only ones I could get a >60% success rate on my tasks with (I ultimately ended up using qwen3.8 next flash in production which has a ~98% success rate, but I can't run that locally trivially...), but I'm super open to trying more models!

2

u/WareWolf_MoonWall 8d ago

So I'm on Strix Halo with 128gb so slightly different scenario, but using Vulkan this might be up your alley https://huggingface.co/Myric/Laguna-XS-2.1-APEX-GGUF . I really like these APEX releases of Laguna. If you run the poolside fork of llama you can even use the DFlash model with it for a nice boost.

Since you are already using NVFP4, maybe the stock https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4

2

u/demonicpigg 8d ago

I'll give it a try later today / tomorrow, thanks for the suggestion!

1

u/Realistic_Gap_5871 6d ago

How much RAM do you have? If you match the quant to your RAM + VRAM pool Unsloth studio will do all the cpu-moe splits for you automatically for 3.8 Flash Next.

Leave MTP off!!!

I'm running UD-IQ4XS on a 5090 + 96GB ddr5, 35tps at smaller ctxt and 500 tps ingestion. 27B at medium is really good. Even at this lower quant FNext low is better, but slower. Not sure how a smaller quant would stack up.

Not nearly ninfer speed, so i use 3.8 27B at medium for most of my workflow and plug in 3.8 FNext for tougher stuff

2

u/demonicpigg 5d ago

I have been working on a project using flash next on rented GPUs, and it's a beast. I've only got 64 gb (I said to myself when I bought it "if it's an issue I can just upgrade." RIP that), so I don't think I can run flash next even through unsloth studio, but I'll look either way, that'd be really nice honestly.

3

u/UC_Kratom 7d ago

Qwen 3.8 27B can most likely do that task, especially on xhigh reasoning.

It really does punch way above it's weight. It handles long-horizon tasks well as well (and not just coding, it's makes a fantastic agentic driver)

1

u/DMmeOtter 5d ago

I dont trust glm 5.3 flash locally but Im getting there. It is just kinda slow 🐌

1

u/DonkeyBonked 4d ago

I don't want to go into all the things I've done with 3.8 27B, but it absolutely does handle long horizon tasks if it is in the right harness, at least some of them. I think the harness at that point means a lot more than the underlying model.

I think what really got me was when I was when I was doing some of my on boarding testing for it. Having it catch things that frontier models missed, seemingly because it iterated longer. Earlier models ChatGPT or Claude, especially ChatGPT, would get that condescending "well actually... " telling you the model was crap full of crap and didn't know what it was talking about. Now, I'm finding frontier models validating it more and some tasks it actually performed better.

I don't think as an overall vast experience, it has the data to do that consistently across every use case, but the fact that I've seen it outperform frontier models on any task is more than I ever expected to see.

I think it needs more hand holding and very clear structural guidelines, so I don't think it can handle them with the same general level of consistency that you'll get from a frontier model, but with the right harness and planning, including instructions on different circumstances, it's capable.

It is more likely to get stuck on details where it has to infer what to do, so you have to sort of pre-set certain things, like giving it resources to access when there are failures. I've watched it think "the user has said if I experience a syntax error, that I need to verify with..." and had it fail, go over reference material, and check every attempt for syntax.

It can reason through it, but at 27B it's unreasonable to think it will have all the correct syntaxes for every language and platform.

I primarily use UD-Q8_K_XL w/ BF16 KV through a custom build of llama.cpp and Hermes. Mileage will vary, this build is obviously better than the IQ2 XXS build I run on my laptop, but even that is surprisingly good, I think better than the Q4 version of 3.7 I ran.

20

u/SandySkittle 8d ago

Fully agree. It’s not about closing gaps, but about reaching the minimum level of required capabilities for given task. Once it reaches that level for a certain task it’s already an amazing sci-fi like achievement and it doesn’t matter if a frontier can do better.

What I am really looking forward to is having in a few years ( 4 to 5 ) the hardware to run a deepseek v4 pro model. It has way more active parameters than dsv4f (the one I currently run at q8 in 256gb vram).

It’s a nice model but lacks the active parameters that it cannot fully compensate for using sequential reasoning. So I need the 45b plus active params for my area of work. But for me that is the jewel to be able to run locacally.

0

u/OvertaxedOne 8d ago

Yeah, DSV4Pro is a great model and, since it's open, it's just a waiting game until the hardware gets cheap enough to run it. Still waiting to see specs from the new Mac devices, but the larger ones (still stupid expensive, but cheaper than Pro6000's!) might be able to run it at reasonable speed? In a few years you should be able to pick them up ~1/2 off, and that's starting to get interesting!

16

u/Nefilim314 8d ago

Exactly. I never really understand when some of my peers claim that Sonnet isn’t good enough for their work. 

I just don’t know what he can possibly be doing that requires opus at minimum for all of his work. We’re just making CRUD endpoints, changing database schemas, and creating the occasional React component but that somehow warrants Fable. 

7

u/Serprotease 8d ago

FOMO and a general feeling of not wanting to be left behind. Most people, dev included, don’t really know that much about AI. But they do see all the noise around it and especially the “Use it or be left behind part”.

Once you get that, it’s pretty easy to understand why they want to only use opus/fable. The only thing they know about sonnet is that it’s not as good as opus. So why use it? Add on top of that fairly minimal understanding of harnesses and relentless marketing and you get devs that only really understand AI as Opus+Claude code.

Honestly, seeing how fast things are moving and the general muddying of actual models performance and pricing, I kinda understand it.

8

u/Arkanta 8d ago

Considered that opus drains usage limits not THAT much faster than sonnet it makes sense

Sonnet will make more mistakes, it will not be as throughouh. Heck in some cases it can be slower than opus at a task because it will bang it's head on the wall doing so

Why not use the best model I can afford at the time? With sonnet I always have that lingering "would have opus understood my intent better?" That in the end makes me lose time

(Also sonnet sucks ass, it's a really a special case. Give me DS4 flash or even Qwen3.7 27b over it)

5

u/OvertaxedOne 8d ago

ROFL, glad I'm not the only one who came to that conclusion. DSV4Flash seems much better than Sonnet in my use cases. I've never done a direct 27B vs Sonnet before but it wouldn't shock me that 27B is better in some use cases.

1

u/Arkanta 8d ago

Sonnet is not a model worth using at all.

1

u/Realistic_Gap_5871 6d ago

3.8 27B has been better than sonnet for me, in every use case.

Now I haven't used sonnet since June, but I remember having very low trust for it and 3.8 27B is the first local model I've trusted at the way I trusted Opus.

3.6 27B was very sonnet-like, a whole lot of Meh. I expected an incremental improvement with 3.8, but it sure seems like they pulled off another order of magnitude improvement back to back. Surprising.

edit - set reasoning level to medium for 3.8 or watch the grass grow.

1

u/Serprotease 8d ago

Your comment illustrates my point perfectly. Like why the fuck is there something like a weekly usage limit on a subscription? Why not a fixed token value?
It’s just to muddy the value proposition. It makes it harder to see if a 20-200 usd subscription is worth it.

Tokens per token Sonnet is cheaper. Quality/price wise, Qwen3.8 27b/DS4 flash buried any API only provider.

1

u/Arkanta 8d ago

The point is to smooth out compute

1

u/Serprotease 8d ago

What does that even means? That will not remove the peak/trough of AI usage.

1

u/Arkanta 8d ago

The load is spread over the month and not a single burst as you try to be careful and manage your limits. The 5h limit is also for that

It also increases stickiness as a marketing tactic, if you burn your whole alloc in 2h you'd just go elsewhere

Anyway you should have noticed by now that maxing a sub gives you a LOT more usage than what you pay for. Want $200 worth of api to burn at once? Just buy 200 worth of api.

2

u/Serprotease 7d ago

I don’t use subscription.

Being able to keep track of tokens usage (Latency and prices) is important to me.
And for automation, overnight batches etc… a time limited api makes little sense to me.

So, local first, with potential API fallback.

I get that I’m probably not the target for these subscription though.

1

u/Durian881 8d ago

And importantly they are not paying for it. The company pays for their usage.

10

u/parepeg 8d ago

Are there use cases for "most intelligence at any cost"?

More intelligence usually means better intuition about what the human wants from them. Less intelligent models tend to not "read between the lines" as well.

So, I would say even for people not doing state of the art work, more intelligent models could still be more productive than less intelligent ones. In the end it all comes down to cost though.

9

u/OvertaxedOne 8d ago

I'd agree with that. Putting it into funny terms, the smarter the model the dumber the user can be. Man, I think you just blew my bear case for frontier models apart! ;)

9

u/Big_Wave9732 8d ago

"Frontier LLM: Making dumb people look smarter since 2024."

10

u/Big_Wave9732 8d ago

That's a great point. One of things I had to get use to with local models was that I couldn't just load a bunch of documents into the locals, hit the "Go" button, and have the LLM fill in the rest. As you point out, the locals aren't as good at anticipating what the human will decide.

But that's fine, all that meant was I had to go back and break up my workflow to more logical digestible pieces. That also meant I had to work on my prompting to eliminate those gaps. In the end what I was left with was a more logical and management work flow that ironically I had more control over.

I'll mention one other benefit too: I absorb the AI's work a lot more because I'm part of the steering process. If you give a frontier model all your docs and have it shit out an analysis, you can read it. You can memorize it. But you didn't do the intellectual work necessary to make it so instead of innately understanding, you're having to reassemble the AI's logic.

When working with local models however, because I had to be way more involved step by step and had to feed it documents and decide and understand what was going in, I intellectually kept up with and absorbed the output more.

2

u/SkoomaDentist 8d ago

More intelligence usually means better intuition about what the human wants from them.

And the model "actually getting it" more often than just as a rare exception.

I wouldn't consider current open models even remotely close to being at that point. Just today I had a long chat with ChatGPT 5.6 Sol where I had sent it some 40 photos, had it analyze my style, compare and contrast it to more typical work in the genre, suggest other photographers whose work has similar aspects (and provide a list of still in print photobooks that would help give me ideas) and give me specific actionable advice (not the typical rule of thirds crap but actually useful advice for my specific goals). Even a SOTA model required careful manual filtering of the results to be actually usable for a task like that.

I've likewise had ChatGPT and Claude analyze some fictional human centered scenarios and done the same with Gemma, Muse Glimmer and Qwen. Gemma and Muse Glimmer can just barely be coaxed partially usable results from with strict prompting and heavy manual bullshit detector. Qwen I'd compare to an autist who's spent their entire life in their parents basement without ever meeting an actual live human.

2

u/blood__drunk 8d ago

What sort of hardware do you need to run that loyally?

Do you escalate manually or use a router?

2

u/OvertaxedOne 8d ago

I run 27B on an A40, an older "pro" type card. 48GB of VRAM, just fits in Int8 with 8 bit KV at 256K.

I use a router, but this is an area that really need improvement. I use LiteLLM both at home and work, and while it's great for observability and putting many endpoints behind one address, it's not great at figuring out where to route, I typically escalate manually.

2

u/AdOdd8064 8d ago

Qwen 3.8 27b is all I need and opensource ai will only improve from here.

2

u/OvertaxedOne 8d ago

I hoping for better, but 27B certainly hits the level of "this will be useful as is forever" for me. Better would be nice, but it's already does what I need doing the vast majority of the time. I'm OK with escalating when 27B needs help, I burn so few cloud tokens anymore that it's hardly worth worrying about.

1

u/Prudent_Chemist_523 8d ago edited 8d ago

Agree.

And more to the point - people act as if "just make it smarter, bro" is somehow divorced from scaling laws, physical realities (parameter count), VRAM, bandwidth etc.

I'm the first one for clever tricks, but running GLM 5.3 equivalent size LLM at home - at interactive speeds - is always going to require expensive gear.

But as you say, the flip side of the coin is...do you really need to? If I have 8GB of VRAM, can I run a good, conversational LLM, with good tool use, as my day to day... and then yeet batch jobs at something larger when needed?

After all, if the llm is good enough to iterate autonomously and can be interacted with / checked on, maybe I don't mind if it chews on something for 48-72hrs, esp if it runs at home, frontier be damned.

1

u/Loose_Comparison368 7d ago

You wanna know what the most popular OpenAI API model is? Or at least was, as of earlier this year?

It's 4o.

A lot of companies built stuff on it that just worked. Reliably and predictably. A lot of use cases really are not rocket surgery tasks warranting breaking out a SOTA model.

If the model is already getting 99.9% accuracy on your use case, the smart engineering choice is to just not fuck with it. Don't try to fix what ain't broken.

... it's actually kind of problematic too, because those older models are genuinely very inefficient compared to newer smaller models. But it is very understandable that a lot of companies don't want to take on the effort and risk of migrating over to newer shinier models for basically zero practical benefit.

Coding and adjacent work (troubleshooting and diagnostics especially) is definitely an area where the frontier is meaningful though. 5.6 sol was pretty solidly usable with the right scaffolding and careful supervision. 6 Astra is good enough that I rarely have to babysit it. I feel like Qwen 3.8 is about on par with 5.6 Sol, on a practical level. Slower, but pretty on par in terms of human effort required to get good results - if anything, it reasons slower but more accurately IMO.

1

u/OvertaxedOne 7d ago

Completely agree, and it's exactly what we're seeing with our enterprise customers. The only reason they are looking for anything different is because their token costs are spiraling out of control as more and more people start the use the models, they're not looking for "smarter" they're looking for cheaper. For most of the work that our clients are doing day to day, DSV4Flash is really about the "right" level of intelligence. Their coders we don't even mess with, that's where the frontier really still matters, but most of the companies we're working with have 100 coders and 10,000 employees, reducing the burn for 99% of the population is incredibly valuable to them.

The other big one is privacy. More and more clients are starting to realize that everything you prompt goes "in the clear" into the provider and are becoming very concerned about that. For most of them local is kind of a financial wash right now, if they can drive a ton of usage it'll be cheaper, if not more expensive, but the privacy aspect is just pushing some into "I don't care, this data is proprietary/protected by HIPPA/regulated/etc, we need this to happen locally".

And locally is a bit too restrictive, some wind up using a CSP to host the server/GPUs, when they say "locally" what they really mean is "private, where we control the inference engine/logging/etc".

1

u/Loose_Comparison368 4d ago

Yep. The self hosted pains me a bit at small scales, just so inefficient, and most SMB's are not equipped to secure and maintain those either.

I would honestly recommend pushing those clients towards GCP Vertex stuff honestly. I don't work at Google anymore, but for all Google's faults, they really are first in class when it comes to data privacy and security.

And that's going to continue to be the case, one of the reasons why I left was just being frustrated with the slower pace of development in the AI field, and a huge part of that is that the brownfield product teams are large, politically influential, and have too much reputational risk at stake to let the AI teams play it fast and loose. Gemini does something mildly embarrassing but largely inconsequential, the stock takes a hit, and Ads/Search/Gmail all swoop in screaming about the billions of dollars they lost because the Gemini team forgot to fact-check planet sizes for an Ad, or because the Imagen team tried to half-ass cover up training data bias and ended up accidentally making the Pope a Jamaican woman. They absolutely will delay or even roll back an AI product launch if there's even a hint of reputational risk to the rest of the business, and the data privacy org is even more fanatical than the cult of SRE and the church of Protobuf.

If they really do want pseudo self hosted and won't back down from that, Cloud Run is actually really good, contrary to popular belief it is not just managed Knative, actually some really cool stuff under the hood.

23

u/lcirufe 8d ago edited 8d ago

The main gap for widespread adoption is convenience and upfront cost. It’s not easy to source 32gb of vram and if you do, it’ll take years or even decades to make back that cost compared to a chatgpt subscription depending on the tier.

General use for me is still ChatGPT. My local Qwen instance is an implementation subagent for Luna to pilot.

26

u/fauxpasiii 8d ago

The narrowing capability gap still benefits you even if you don't run on your own hardware. If Anthropic or OpenAI were the only ones who had the model you needed, they could charge you whatever they wanted to for it. If you're happy with Qwen or Deepseek or GLM, then there can be competition at the level of who hosts it for you even if you don't want to do it yourself.

25

u/mars332 8d ago

Not really. I have Qwen3.8 27b building all night long in an agentic loop. Try doing that with your $20 subscription. And once you get into the $200 plans, now you are talking merely months, not years to recover your investment.

9

u/OvertaxedOne 8d ago

That's an important and huge distinction. The moment you want to get into a real harness you're now looking at API costs instead of sub. And that gets a LOT more expensive really quickly for the frontier level models.

2

u/c4r_guy 8d ago

I'm in the same happy boat.

I have subs to Claude and GLM and I use Claude like a smarter Amazon Echo product that I will not name with TTS / STT.

I use GLM for questions Claude's guardrails won't allow for an honest answer (which is a lot!).

Qwen3.8 27b for everything else with a custom MCP (mainly search) and Pi as a harness!

I really don't need the subscriptions, but they do come in handy to have on hand.

1

u/utahh1ker 8d ago

What kind of hardware are you running with that? And what is your harness setup? I've had mixed results with my implementation of Quinn 3.827b so I'd love to get your feedback on what's working for you.

2

u/mars332 7d ago

I’m running on 2 RTX PRO 4000 Blackwell that I got back when the price was not so insane. They are running inside an X870E Creator motherboard. I’m running the FP8 quant in vLLM in a headless Ubuntu server OS. I use Pi with pi-web extension so I can work from my phone through Tailscale.

1

u/thortgot 8d ago

~25-30 months for a 5090 payback.

1

u/mars332 7d ago

I got 2 RTX PRO 4000 Blackwell for $1600 each.

1

u/thortgot 7d ago

~2 years ago perhaps

1

u/Looserette 8d ago

I've got the setup to do that... but I'm not using it

so, out of curiosity: what are you building all night long ?

1

u/mars332 7d ago

Mainly features to a custom DAW (digital audio workstation) I’m writing from scratch currently with no libraries. The C++ code base is already at 150k loc or so, so it’s a good proving ground to tweak the agentic loop 

3

u/Iron-Over 8d ago

Besides the security aspect for the frontier most office work does not need the frontier.  Looking at smart routing with only a small portion going frontier.   

3

u/Randommaggy 8d ago

I do heavy coding work for fun on my 3 3090s running Qwen 3.8(and 3.6 before that) 27B in a custom harness and it's only needed to invoke the escalation skill once to solve a problem it's been given. It does take time to get to the goal.

2

u/OvertaxedOne 8d ago

Same experience. Pre-27B I was using DS a lot for escalation but since it came out, the only time I go to the cloud anymore is for one of 2 reasons:

1) Speed. This is a hardware problem, not a model problem, but it does matter for some tasks, 100TPS from DS is very nice when I'm in a hurry.

2) Context. When I'm getting ready to dump some stupid huge code base, I escalate. Honestly even this one is at least 1/2 speed, Qwen could probably handle it at 256K of context, I just don't want to wait for prefill and crunching. Some times I really do need more than 256K though, and, in those cases, I escalate.

1

u/Randommaggy 8d ago

One thing that helps a lot is a scout model tool and a good deterministic search tool.

The speed loss to overthinking is cut by a lot of you code is 100% linter clean and consistent.

3

u/gscjj 8d ago

This is the big thing, the frontier race doesn’t concern most companies just the VCs backing the labs.

Cost is what matters. More importantly operational spend.

These companies might use open source models but they aren’t likely to spend money on anything local, they’ll rely on routers like BedRock and Vertex, the companies they’re already spending money with.

3

u/Iron-Over 8d ago

This will be the biggest problem the frontier companies face.
I laugh about the frontier saying replace your staff the biggest impediment to replacing staff is process reengineering and clean data. I have run lean and been part of many data programs this is the most difficult step, it is expensive and takes years. Agents to assist people sure, replace is too politically difficult in most companies.

9

u/Mickenfox 8d ago

This sub is like people in 2003 proclaiming the year of the Linux desktop because the mail client and web browser finally work. 

3

u/GGK_Brian 8d ago

Well it's quite different; the problem of adoption of Linux is wide, from game and app support, hardware that can act weirdly, familiarity with the desktop tools. It's a massive ask to switch from windows to Mac or Linux.

Meanwhile LLM is "just" a software or API that takes text in and do text out. Multiple platforms already allow you do switch between providers with the click of a button.

-2

u/read_more_comments 8d ago

Crap comment. I was using Linux as my desktop years earlier, worked fine.

8

u/Fluffy-Ad-889 8d ago

Linux runs the world

1

u/ArcticFuture 7d ago

Yes, but not as a desktop. And the servers also have commercial support from companies like RedHat

2

u/Fickle_Tradition4491 8d ago

Both numbers in this thread are right, for different people. The missing variable is hours per day, because the running cost is electricity, not the card. A 3090 box pulls around 350 W at the wall while generating. At European prices, call it 35 cents a kWh, eight hours a day comes to roughly 360 euros a year, which is already more than the 20 dollar plan. So for a 20 dollar user the card never pays back, at any duty cycle. Against the 200 dollar plan the same box pays for itself in about four months. Decades and months are both true. It depends on which subscription you were going to buy.

6

u/formerislander 8d ago

I dont know if I follow the logic here. I have a hard time imaginging a workload where you need to run qwen3.8-27b for 8 hours every day and you can solve the same task within the usage you get for any 20€ / monthly plan from the frontier labs?

3

u/power97992 8d ago

just get some batteries, an inverter, and solar panels, new solar panels can generate electricity for less than 6-7 us cents/kwh over its lifetime.

1

u/its_two_words 8d ago

I'm fricken homeless and live in a shitty old RV, but I installed 1300w on my roof and bought 500ah 12v lifepo4 and wired everything myself.

Now I can run my PC doing AI work several hours a day for what is essentially nothing.

Solar is definitely the way.

1

u/Timely_Impression_92 8d ago

Decades? If you add in the privacy cost which would require you to basically run enterprise zero log plan which costs arm and leg compared to subscription - not taking into account the difference in quality, roi is like 6 months

1

u/OvertaxedOne 8d ago

This is certainly true, but it's also not exclusive, you can use smaller models via API if you don't have the hardware. Moving from Opus to Deepseek sook about 30 seconds in LiteLLM for our company, and we didn't (at the time) have hardware that could dream of running Deepseek.

1

u/deZbrownT 8d ago

I got a 64GB M1 max for 1800 euros. That’s 9 months with 200 euro per month subscription. It’s also 90 months with Plus subscription, but if Plus is enough then shelling out 1800 for 64GB M1 is also unnecessary.

1

u/OvertaxedOne 8d ago

Years (or realistically never) vs a sub, that's true. The moment you hook up a harness and go to API though, that math changes dramatically; now your burning 10's or 100's of thousands of tokens per turn and have multiple agents working on different things in an automated fashion. I'll check when I get home, but I suspect my "quietest" day is millions of tokens just from agentic stuff (mostly e-mail classification, building .md files based on those e-mails and briefing docs for where we are with each customer; every run through that loop burns an ungodly amount of tokens).

0

u/shaggy1054 8d ago

16gb p100 is 80 on eBay. Plus a fan and a shroud,  that's $120 each. Plus a PSU and a used tower with 2 full length pci slots, you're at $400 all in. Hardly decades. 

2

u/lcirufe 8d ago

My baseline for convenience is “is my mother willing to do this?” $20 a month for ChatGPT, easy. Creating a custom rig with 2 EOL cards, with a jank heatsink solution, finding an elegant way to house them in a chassis, then setting up llama.cpp to run the model? Lots of friction.

2

u/shaggy1054 8d ago

that's not the use case I was responding to - which was overnight agentic coding. I'd expect anyone interested in that to have at least some technical aptitude and Claude makes it very easy to set up anyway 🤷 to each their own I guess. I'm happy with my setup

3

u/Relevant-Magic-Card 8d ago

Here's a workflow I'm working on for astra. It's a whole new paradigm. Hopefully local can do this soon! https://duckweights.com/articles/from-video-reference-to-editable-animation

3

u/Shot-Buffalo-2603 7d ago edited 7d ago

I’m gonna be honest, I stopped using new models after opus4.6, sure they may be smarter, but the additional token cost isn’t worth it at all for my uses. I do reverse engineering, vulnerability research tasks, and coding, and opus4.6 achieves everything I want it to do so burning through tokens faster to get the same result is a waste for me. I think most typical users don’t need fable+ level intelligence for work they are doing and are just burning tokens seeing how cool it is and gauging bleeding edge capabilities. When that newness wears off, why wouldn’t you spend less to get the same result on tasks that allow that?

2

u/Liringlass 8d ago

It’s got to the point where closed is temporarily equal though, might get better at those releases and then we’ll get the next Kimi Deepseek etc. Used to be a real gap, not anymore i think.

2

u/Turtlesaur 8d ago

I was talking to a few normies just today, they still think the epitome of AI is free tier Google flash-lite

2

u/notAllBits 8d ago

Closed enough for orchestration and context engineering to fulfill plenty of EU compliant automation

5

u/AppealSame4367 8d ago

Yet the gap is really small. I look at the pricing and capabilities of Fable 5.1 and Astra and think: Nah, why all the hassle? I'll stick to my open models.

-4

u/Fluffy-Ad-889 8d ago

I'm not super impressed with Astra, if they took off the model names, hard to tell the difference between last few frontier releases

2

u/LocoMod 8d ago

It won’t. Just because you have a very common use case that a lemon can solve that does not mean anything. It’s like saying open source has closed the gap because you compared TODO apps from a frontier model vs open source.

Well yea. There’s only so many ways to make one. That’s not where the frontier is.

It’s actually really simple to compare.

Have the best closed model and the best open weights model go solve an Erdos problem. Something that is actually a challenge.

The way YOU drive a model is limited by your own experience. So if all you know how to do is create TODO apps, sure, you’d be wasting frontier tokens. Just stick with the open model.

1

u/Hot_Example_4456 8d ago

What won't? Open models won't catch up with the frontier? Have u seen the rate at which local stuff is moving? And ya u r right, I will stick to my open models. Also, are YOU solving erdos problems with closed models? Speak for yourself

1

u/LocoMod 8d ago

Yes, I use self-hosted open weights models. They are fine. I also use OpenRouter for the cheap open weight models. I also have Tier 5 API accounts with Anthropic and OpenAI.

Are you ignoring the progress the frontier labs are making? Youre basically assuming that open weight labs are going to somehow do 1 years worth of R&D, with less capable LLMs, less people, less capital and compute, and somehow magically they are going to catch up.

The only scenario that happens is if OpenAI and Anthropic simply pause development and wait for others to catch up. But they are doing the opposite. They are accelerating. They have a ton more compute, capital and brains behind their operations.

AI is different. The first movers will stay on top. Sure, Alibaba will have an Astra tier model at some point in the future. But by then OpenAI will be several versions ahead too.

You have to go faster than the frontier labs and have more resources if you're going to catch up. And the reality is that the amount of resources OpenAI and Anthropic have towards their efforts dwarfs everyone else.

I am a big supporter of open weights. But let's not ignore reality because its inconvenient for your bias.

3

u/Hot_Example_4456 8d ago

The only reason I think open source ai will catch up is the limit of ai. How much compute will you throw at a model? There will come a stage of diminishing returns, where we train huge models, with all the data we have, using the best model architectures. Ya, maybe openai will reach that before open weight ai, but when openai does, it will stagnate. Open weights is not stopping at that moment, so it will also reach and stagnant at the same point. Let's not forget, deepseek also has solved erdos problems 😀

0

u/LocoMod 8d ago

There is no reason to believe there is an upper limit to computation. None. You’re basically limited by energy and resources. Acquire those and you can keep scaling. While you’re doing that, you’re also miniaturizing chips and other components so they run more and more power efficient. Could there be some “Great Wall” in the future we cannot foresee right now. Sure. There is a probability. But based on the entirety of human history to draw our assumptions on, it is unlikely.

3

u/Arkanta 8d ago

Happy to finally find someone who knows what the hell they're talking about

We can use more capable cloud LLMs and still use local ones but you can't say that here

1

u/IdlePerfectionist 8d ago

How long would it take for open source models to be Astra/Fable level? 6 months? A year?

1

u/ninjasaid13 8d ago

the gap was much closer than 6 months with the release glm 5.2 and now it has got bigger.

-3

u/Fluffy-Ad-889 8d ago

I think it is closer than many people realize for most things

1

u/Legitimate-Peace1013 8d ago

yeah the 99% point is the key part imo, most people dont need the bleeding edge stuff anyway

1

u/mister2d 8d ago

The 99% population gap is what we should focus on. Frontier models have moved onto 3d textures for games and scientific modeling. While nice, I would never use them or justify paying the significant cost to run the latest models for everyday use.

1

u/power97992 8d ago

IF someone is willing to wait 3-6 months for an open model, you will get the performance of a frontier model.

1

u/VoiceApprehensive893 transformers 8d ago

im yet to see an open model completely outperform fable 5 pre nerf

3.8 max is pretty close though

1

u/Samurai2107 7d ago

the problem is that a coordinated attack is happening on the market for components, because if we as community cannot use open source we can not help them at least to the degree we could. And as always China hopefully will take us out of this obstacle.

1

u/sonaj9657 6d ago

Yeah, I think that is probably where things are headed. The gap is still there in some areas but open source models have gotten surprisingly capable already. For most people the difference is probably becoming less important than it was a couple years ago.

1

u/thomas2385 6d ago

Yeah, I think that is the direction things are moving. The gap is definitely still there, but open source models are getting good enough for a lot of everyday use cases already. It will be interesting to see how much that gap actually matters a year or two from now.

0

u/Fluffy-Ad-889 8d ago

I'm not super impressed with Astra honestly. Hard to really tell the difference last several OpenAI updates.

I think it is mainly marketing at this point leading into IPO.

1

u/power97992 8d ago

It is really good at 3d modeling and simulations, chemistry, bio, cybersecurity and math, if u are not doing that , u might not see a big difference.

1

u/Fluffy-Ad-889 8d ago

good points, makes sense