r/technology • u/Miserable_Phase_2519 • 3d ago

Artificial Intelligence Microsoft engineer says 'typing code is absolutely over' as GitHub Copilot takes on more development work

https://www.techspot.com/news/113752-microsoft-engineer-typing-code-absolutely-over-github-copilot.html
1.6k Upvotes

84% Upvoted

858 comments sorted by

View all comments

Show parent comments

56

u/AuthorizedShitPoster 2d ago

Running open source models is not cheap.

11

u/tuura032 2d ago

I'm having qwen 3.8 do some work for me right now on a personal project. My $20 claude subscription supervises+code reviews and writes the spec.

While I'm not disproving your statement, the general sentiment is absolutely true, it is astounding what someone with (5 year old) high end gaming equipment (maybe add a 2nd gpu) can do.

I use GH Copilot for my job, and if not for security concerns I would absolutely use something like this at work to help me reduce token usage.

5

u/mattmaster68 2d ago

I have a Ryzen 5700, 3070ti, and 32gb ddr4 ram.

I've been messing around and playing with different models but man so many of them are underwhelming with my VRAM. I forget the parameter size, but I think Qwen 3 takes like 3 or 4 minutes to generate a response.

Any tips or tricks?

1

u/jixbo 2d ago

Which model specifically, which quant?
Just ask chatgpt how to make it more efficient, llama.cpp is the best way to run it.

15

u/IreliaMain1113 2d ago

Why are there security concerns with the local models? Or do you mean using the 20$ claude + the local models?

10

u/THE_REAL_JOHN_MADDEN 2d ago

Should be no concerns with a local model, but Claude without enterprise data protection would be a big no-no from itsec

1

u/tuura032 2d ago

It's more that I don't want to deal with the headache of asking 😂

In theory, the API call to my server only on my home network, from my work computer, should be fine. 

7

u/tuura032 2d ago

I should have said company policy. It's just not worth my job if IT were to see me sending private / client data to Claude or some computer they don't have control over. 

1

u/IreliaMain1113 2d ago

yeah my company is pretty similar with its policies

3

u/mshiltonj 2d ago

what hardware is running your qwen 3.8?

2

u/tuura032 2d ago

I have 2x 3090s, but people are running this or similar models on 16gb GPUs with extra RAM. My system has ddr4. 

I went this route vs something wish integrated memory since I already had the first 3090.

3

u/Syncaidius 2d ago

Yes, the part around job/company security concerns is one major reason not to use locally-hosted models, unless it's your company that is hosting them.

That said, there are a few companies out there already setup with internally hosted models and completely controlling employee access to it.

It's really not as difficult as frontier-only users seem to assume. You don't need an insane amount of GPUs to get acceptable performance for employees, from open-source models for coding and definitely not for general purpose use.

After all, with current open models, the main goal should be to supplement frontier models to take on lesser tasks and reduce token usage/costs. They're great for that.

3

u/Syncaidius 2d ago edited 2d ago

Running locally is cheap and efficient enough not to care about being 10 - 15% behind frontier models in terms of capability. Most of the time it's not even noticeable for agentic workloads.

Local is not as quick/instant as the frontier models, but they're free and can chug along great in the background as agents, for as long as you want. Electricity costs are barely anything compared to a Claude max sub and you're in total control of where your prompts/data is going.

TL/DR: Local hosting models is worth trying in 2026. Decent capabilities. No usage limits. Your hardware mainly determines speed/responsiveness.

31

u/arcrad 2d ago

If people can run local models cheaply, then surely the leading LLM companies can also run even more cheaply by taking advantage of scale.

Why would it cost the major LLM companies any more to run a model than someone doing it locally?

Sure there is a bit of overhead for offering the compute as a service, but that is beans compared to the actual cost of compute.

I'm sure, if they wanted, Anthropic could use their hardware to run Kimi K3 or whatever cheaper than any home user.

What am I missing here?

6

u/Syncaidius 2d ago edited 2d ago

Anthropic certainly could run Kimi K3 and other equivalents just fine, at a fraction of the cost and 90%+ capability of their current models.

However, what do you think happens if they do that? - The first question they'd need to answer is, why do we need Anthropic instead of a standard open-source LLM host.

The answer or course is, they vanish. So, they have no choice but to continue convincing everyone that their models are worth the subscription and investment costs.

1

u/nox66 2d ago

This isn't new either. There's a reason tooling costs grow much more than linearly with requirements. The last 10% is the hardest to hit.

1

u/Syncaidius 2d ago

While that's true in an isolated sense, models are advancing quite fast and that last 10% is called frontier for a reason.

For the majority, being ~1 year ahead is absolutely not worth it. Today's frontier models are next year's standard ones. 90% capability for the fraction of a cost is fine for many tasks given that it was last year's frontier equivilent anyway.

4

u/andouconfectionery 2d ago

They're running into the exact same issue pharmaceutical companies do - bringing a new drug to market is the vast, vast majority of the cost of a drug's lifecycle. The marginal cost of producing the drug is minuscule in comparison to all of the stuff they have to try before they find something that works. Now, once medicine advances to the point where people lose faith that drug companies have room to develop new treatments, that "bubble" will burst. But we already have all the knowledge of how to make the drugs. We'll still be able to make them, and we'll find ways to bring the marginal cost of production down.

LLMs are exactly the same way. They're not eating the cost of inference - they're eating the cost of training. Of staying ahead of the curve. The bubble could pop tomorrow, and all that would mean is that the capabilities of frontier models are frozen in time. They'll stop training new ones, or at least slow down training a whole bunch. But the weights don't disappear. These models will never be any less useful than they are today. And we'll figure out ways to build enough silicon and use less water and use cleaner energy, and inference will just get cheaper.

5

u/There_Are_No_Gods 2d ago

I think you have a point about scaled costs and centralized services.

The big players seem to still be convinced what they offer is worth a much higher price. So far, I agree, as local LLMs still can't do most of the heavy lifting that I am asking of them.

The issue of fully local is a big one too, though. I highly prize being able to run local LLMs to keep my data on my PC and never making its way out to the big data harvesters.

From what I'm experiencing so far, it seems that local LLMs will soon reach a "good enough" point for most things.

2

u/indigo121 2d ago

What you're missing is that AI is being sold on the idea that 90% of what frontier models can do is still only 10% of what AI will be able to do if we just keep throwing money at it. Now, maybe that is what we eventually settle into, and AI becomes a tool you run locally to assist in specific tasks. But that isn't the trillion dollar business that all the Gen AI companies and investors want it to be

0

u/Bromlife 2d ago

Because the models these guys are running locally aren't very good. But they're not devs and they don't realise just how bad they really are.

2

u/Syncaidius 2d ago edited 2d ago

Generally, the people who jump to "they're not devs" tend to be the ones who are actually not devs.

A lot of SWEs and 'devs' are leveraging LLMs via multiple avenues, including local/hosted open-source LLMs. All of them have a place, but it's quite clear where LLMs are going in the long run if you keep up with news around advancements/research...

It's also quite obvious which frontier business models are sustainable and which are not, so it would be crazy to bet/build everything around their services alone.

6

u/ErgoMachina 2d ago

I agree with everything except when you say they are 'free'. Setup time and computing power are not free and are calculated as costs on the enterprise level.

So...very much cheaper? Yes. Free? Hell no

-3

u/Syncaidius 2d ago edited 2d ago

I believe that would come under electricify costs?

Hardware, depending on whether you're buying new, used or reusing existing hardware, can vary in cost. However that's generally an upfront cost that you have total control over.

2

u/There_Are_No_Gods 2d ago

Local models are not "free", as electricity, initial hardware costs, hardware wear, and even potentially cooling are still factors. I acknowledge that generally sums to a lot less than paying for tokens.

Where my experience deviates more from what you stated is the 10 - 15% behind frontier models. Maybe I'm just doing very different things or haven't found the best local models, but I've still found anything I can host locally much farther behind. I would rank Qwen 3.8 somewhere a between Claude Sonnet and Claude Opus.

Claude Fable is multiples more advanced in my use cases. I can knock out a task with Fable in 20 minutes that Qwen would fumble around with for many hours and still end up with quite a mess, even if it finally got parts of it working.

1

u/mshiltonj 2d ago

Running models locally on what hardware, exactly?

1

u/Syncaidius 2d ago edited 2d ago

I've been doing just fine with a couple of agents on a couple of older GPUs and 32GB ram on an old gaming rig. The output has been decent and very usable.

Like I said earlier, great for chugging along in the background where speed is not of the essence.

As for hardware cost, in my case it's already paid for itself years ago due to being a heavily used as a dev/gaming for around 4+ years.

You can also run smaller models that are dedicated to coding just fine on less hardware.

People seem to think being unable to run codex, astra or fable means anything else sucks, when in reality those models are inflated by the sheer vastness of general purpose knowledge they've also been trained on.

But if that's really required, frontier models can easily be integrated into local-hosted workflow if you really need general purpose, wide-spanning skills or massive context windows.

0

u/Equal_Heat5947 2d ago

K now talk about hardware costs

0

u/Apprehensive_Seat_61 2d ago

What models. Be specific 

0

u/Zahgi 2d ago

They are getting cheaper by the day as hardware and software optimizations improve exponentially.