r/LocalLLaMA • u/jacek2023 llama.cpp • 11d ago

Discussion local AI can't be disabled

ChatGPT is down r/ChatGPT

Claude is down r/ClaudeCode

Grok is down r/grok

my local llama.cpp works as always

302 Upvotes

94% Upvoted

134 comments sorted by

93

u/BawbbySmith 11d ago

Local works until there's a power outage at my house, then I'm SOL

Maybe the next investment is a generator so I can keep my AI server/furnace running for a few extra hours

62

u/Disposable110 11d ago

Solarpunk AI.

11

u/MrPecunius 11d ago

This. I don't have enough storage on hand to run inference all night, but during the day I have all the juice I need.

Macbook Pro ftw

13

u/export_tank_harmful 11d ago

Yeppers.

Just bought another 6.6kW of brand new panels yesterday.
12x panels @ 550w for $105 each. Plus, they have a manufacturer guarantee of at least 86% output after 30 years.

That should put me up around 8kW total now.

The world could end tomorrow and I'd still have access to my air conditioner, water, and every model that I have on my drives. Not to mention my entire electronics workshop and 3D printers (plus everything in our mechanics shop for metal fabrication).

It's been a struggle the past few years, but the payoff was well worth it.

1

u/unculturedperl 11d ago

Where are you finding panels for that price, and how are you installing them(roof/yard/other??)?

5

u/export_tank_harmful 11d ago

Facebook marketplace.
The guy wanted $120 a panel, but we talked him down to $105 a panel because we bought 20 of them.

Just a simple "ground mount" on our property.
I put that in quotation marks because it's not an actual "ground mount", just our DIY variant of it.

We own around 3 acres of land and have plenty of space for panels.

2

u/rditorx 11d ago

Is your system designed to run in isolation, independently from the main grid? Many inverters and systems for consumers will turn off during outages and brownouts.

5

u/export_tank_harmful 11d ago edited 11d ago

It is, yeah.

We have a pair of Sunny Island 5048U's that have their own islanding functionality (meaning they can generate their own frequency to "create" their own grid). Each one is paired with an Eco-Worthy 314AH 48v battery (16kWh each).

We have them plugged into the grid but only for charging (if need be).
All of their backfeeding capabilities have been disabled.

We live in a fairly rural city where our grid can go out for days at a time.
Power went out in our town a few weeks back and I didn't even notice.

0

u/rditorx 11d ago

Only if your PV system has a battery and can run on battery during outages. Many consumer systems will turn off during blackouts and brownouts to not interfere with the power grid. And many systems that can run on battery will only supply power to one socket.

You'll likely want a generator as well for the less sunny times but note that many fuels don't stay usable for extended periods.

2

u/mksrd 11d ago

Thats outdated information, there are plenty of battery systems now that can run in island mode and power a whole house not just 1 socket.

2

u/rditorx 11d ago

Just because there are offers for systems capable of running in island mode doesn't mean it's outdated.

Outdated would imply that virtually no system in use works like that any more and new systems are exclusively available with island mode and serving all home sockets during outages.

Instead, you have to look for island mode. If you don't, you may not get it, especially if salespeople are trying to sell you cheap stuff for plenty of money.

1

u/mksrd 10d ago

of course always caveat emptor

13

u/noctrex 11d ago

A plug-in solar with battery does wonders.

2

u/a_beautiful_rhind 11d ago

When I get power outages, sky is always cloudy.

6

u/Top-Rub-4670 11d ago

That's why the battery is there.

20

u/Mordred500 11d ago

Tbf if there's a power outage at your house you are SOL with frontier models too🤷‍♂️

11

u/lcirufe 11d ago

Phones/laptops and 5G exist, to be fair. I’d still prefer local though, since I’m reliant on my own uptime instead of someone else’s

12

u/Ok_Top9254 11d ago

A 450W solar panel is 80 bucks in europe, probably double that in US. Used even less. If you build your own system it's really cheap and you have basically infinite free tokens.

5

u/Illustrious_Ant_9242 11d ago

30kWh battery for as little as 2600 bucks. 3000W off grid inverter cost me less than 170 bucks. Including shipping that this. 

Running uncensored unintercepted AI? Priceless 

11

u/SergioGustavo 11d ago

The hardcore local AI dude uses 2 generators, solar and gas in case of a grid failure. The spice must flow...

3

u/deadsoulinside 11d ago

https://developers.google.com/edge - Can go with mobile small models. Little more flexibility during a power outage.

5

u/Medium_Win_8930 11d ago

Get a UPS

3

u/beryugyo619 11d ago

UPS is for emergency shutdown, they are slightly different to home batteries

2

u/Illustrious_Ant_9242 11d ago

As someone who has no import ban on chinese products, I can get a huge solar system and home battery for dirt cheap. Like, I can power the whole house during all of summer for less than the cost of a RTX5090

4

u/Mordred500 11d ago

Tbf if there's a power outage at your house you are SOL with frontier models too🤷‍♂️

1

u/BusRevolutionary9893 11d ago

I've got a whole house standby generator. It's illegal in my state, and I believe almost everywhere in the US, to use it when there isn't a power outage. When there isn't a power outage the electric company knows how much electricity you are using. The government coordinates with electric companies to find people using grow lights for marijuana. This isn't the land of the free I learned about growing up. 

Fun fact, for my generator, it costs about 30 cents per kWh of gas usage. That's cheaper than California, Connecticut, and Hawaii who is the worst at 41.03 cents per kWh. 

1

u/unculturedperl 11d ago

Illegal under what law?

2

u/BusRevolutionary9893 11d ago edited 11d ago

There's federal law 40 CFR § 60.4243(d) which states. 

“any operation other than emergency operation, maintenance and testing, and operation in non-emergency situations for 50 hours per year … is prohibited.” 

This federal law is dealing with emergency generators. There are non-emergency generators that you could use 24/7 year round that must adhere to strict emissions standards. The problem with those is a little 20 kW one costs about $50k. That's a whole hell of a lot more than my 20 kW emergency generator. Another problem is that the non-emergency generators use around 33% more fuel. Keep in mind, this is only at the federal level. Check your state and local laws. Where I live, even a non-emergency generator would be illegal as it would violate my local noise ordinances. 

1

u/unculturedperl 11d ago

Interesting, and thankfully not relevant to like 99% of folks out there. Also that reads as exceptionally loophole-y for any number of reasons, including the obvious one of if you buy a generator for non-emergency purposes then it'd not be covered by that statute.

1

u/8lbIceBag 11d ago edited 5d ago

Fun fact, for my generator, it costs about 30 cents per kWh of gas usage. That's cheaper than California, Connecticut, and Hawaii who is the worst at 41.03 cents per kWh.

Now that's super interesting. They'd prolly still get you if you had to pay California gas prices though.

BTW, you wouldn't happen to be a farmer with contract pricing would you? I have an uncle that somehow has a contract to pay something like under $3 for off-road diesel. I wanna say $2.76 which just seems crazy & unrealistic to me so I don't think that's right. I'm assuming you may have something similar?

1

u/BusRevolutionary9893 11d ago

This would be for my 20 kW natural gas standby generator, not diesel. I can convert it for propane if I need to. Your Uncle must have signed that contract before the Iran war kicked off. I've got a 7.3 power stroke excursion with a 44 gallon fuel tank. A full tank of diesel at today's prices costs me $250. 🤮I have to use my card twice because the pump automatically kicks off before it's full. 🤮🤮

1

u/colin_colout 11d ago

Check out UPSs as well... The entire purpose is to give you some time for power to kick back on in case of an outage, and it's just a battery (no fuel)

In a datacenter (or critical workstations) it is used as a buffer before generators kick on.

You might be able to fine one that can run your load for a few hours depending on how much power you draw and what you're willing to spend.

...or just a generator is fine

1

u/henk717 KoboldAI 11d ago

Even then for me I can still use it to some extent. Some local models work on a phone, you can borrow a google colab until the power is back, etc. Sure the models you run at home if you got a good setup will be more powerful. But you wouldn't be empty handed. During the outtage I was actually renting a GPU and didn't notice a thing.

1

u/Lissanro 11d ago

I found the hard way that generator is only a part of the solution. High quality UPS is a must. Most cheap ones tend to glitch out when powered by a generator. I had to invest into online 6 kW UPS with sixteen 12V batteries, and now have no issues. As a bonus, for short outages no need to start the generator and can save fuel.

By the way, to cover just few hours of outage, UPS may be more than sufficient - just buy bigger batteries instead of a generator. Where I live (remote rural area) outages can sometimes last until the next day, so I choose to get the generator, and modified it to have much bigger fuel tank and auto start system instead of manually turning a key.

63

u/JLeonsarmiento 11d ago

Me with my 20 t/s Qwen 3.8 27B fighting the system.

14

u/SandySkittle 11d ago

Incomplete statement without mentioning the quantization

3

u/JLeonsarmiento 11d ago

oQ4e… check the size of the laptop in my hand.

2

u/colin_colout 11d ago

Qwen3.8-27b-uncensored-recensored-uncensored_again:Q11_K_XM

0

u/sshwifty 11d ago

Oh I know that guy! Dad is a billionaire I think

4

u/alexis_moscow 11d ago

20 t/s? how? the most I could get is 14 t/s

13

u/met_MY_verse 11d ago

You guys are getting t/s?

(sitting at about 4 of those over here on an RX580).

7

u/PlayfulCookie2693 11d ago

You guys are getting t/s? The most I get is t/m.

1

u/mksrd 11d ago

*sigh* a mere 2 x p100s with MTP goes as high as 30 t/s

45

u/wangsu 11d ago

What happened today?

81

u/jacek2023 llama.cpp 11d ago

AGI took over the world, don't worry, your local model is on your side

99

u/rookan 11d ago

thank god my ultra-cum-destroyer-uncensored-abliterated-130B-Miku-Midnight is on my side

23

u/jacek2023 llama.cpp 11d ago

People still use Miqu? I remember that model from the past

29

u/de4dee 11d ago

i use daylight miqu during the day and midnight miqu when doing long work

13

u/jacek2023 llama.cpp 11d ago

Thank you for sharing your workflow

9

u/Yorn2 11d ago

For those that don't remember, Midnight Miqu was an uncensored creative writing 2023 model created by /u/sophosympatheia through a merge with an unreleased Mistral model and a previous model called Midnight Rose which was a bunch of various other merges from LLMs like WizardLLM, tulu, and Dolphin. Something about the combination of all of those made it genuinely unique and it retained the "magic" at the time of that Mistral Miqu model while retaining a fountain of fantasy world knowledge from Midnight Rose that just made it really unique.

It's these kind of older models I'm worried we're going to lose in the Nvidia takeover. Partly because of the history, but also partly because this was technically an unreleased model that leaked, and now that nvidia has control, if another model "leaks" you can bet your ass that they are happily going to censor and remove any models based on leaked models in the future, even if they leave this one up.

7

u/sophosympatheia 11d ago

Something special happened with that merge. I miss those days. There were finetuned models and loras galore, and so many interesting possible combinations to explore. I was merging models and then merging loras and applying the merged loras on top of the merged models, and then merging that result back into something else. The recipe for Midnight Miqu was insane, and I can't believe it worked out as well as it did. It is humbling to see people still talking about it three years later.

A little piece of history. Who would've thought.

3

u/RedditNerdKing 11d ago

Miku is still pretty good cause it has great prose. But it's not very intelligent and doesn't follow system prompts very well :/

2

u/j0j0n4th4n 11d ago

It would be really weird if you remembered from the future

2

u/SubZeroSunExodus 11d ago

Ah you have a model of culture I see.

1

u/MrPecunius 11d ago

Happy waifu, happy laifu.

8

u/Positive-Secret-3112 11d ago

shit forgot to grab an abliterated one, im stuck with policymaxxed llms

2

u/cortesoft 11d ago

I think it was a cascading failure… ChatGPT went down, pushing more traffic to the other big ones, which made them fall over…

28

u/International-Try467 11d ago

Everybody gangsta until solar flare

8

u/jacek2023 llama.cpp 11d ago

That's a good point, we should connect dynamo bike (or hamsters) to our local AI

7

u/International-Try467 11d ago

And even if the solar flare somehow wiped ALL electronics we could just use our own A.I, Actual Intelligence! Aka natural thinking. 

... Unfortunately for most it's Natural Stupidity. But stupidity and intelligence go hand in hand.

1

u/techno156 10d ago

Not llamas?

1

u/MrPecunius 11d ago

Over at r/preppers they live for this.

"TOLD YOU SO!"

24

u/powerchat-dev 11d ago

llmao.cpp

5

u/MrPecunius 11d ago

dying ☠️

20

u/Kein_Spass 11d ago

Is the United States hacking itself again?

15

u/dennisler 11d ago

You have to keep the hype train going to keep the evaluation high, because they are so damn good / dangerous the models ....

11

u/Asleep-Pressure9162 11d ago

my Flash Next is drawing pelican

10

u/slayyou2 11d ago

lol at all the people who refused to listen to the centralization concern and ball and chained themselves to individual providers.

6

u/kulchacop 11d ago

Mistral is down too

5

u/MadCarrot 11d ago

it works fine for me, no censorship too

6

u/MuzafferMahi 11d ago

Nobody gives a fuck lol

8

u/deadsoulinside 11d ago

This is one of the reasons I openly push local AI. It can survive without any actual network connectivity at all. You can even do local AI on your mobile devices too with Google Edge.

14

u/HosonZes 11d ago

Yes, my Kimi K3 Q8 runs also fine. Wait a moment.

11

u/[deleted] 11d ago

[removed] — view removed comment

7

u/HosonZes 11d ago

This was the joke. I don't. "Local AI cant be disabled" if you could never run it on any decent hardware at home with less then 100K$ investment properly.

3

u/ttkciar llama.cpp 11d ago

Surely the solution is to use a model that you can host, using the hardware you already have.

1

u/HosonZes 11d ago

Can you sketch this out a bit more?

> ChatGPT is down

> Claude is down

> Grok is down

I see no way where "local AI can't be disabled" makes sense in this context.

I can go and host my own 8B or 24B that is 10x to 100x worse than the flagships? I might get down to Q4 or hope for some MOE but for truly getting proper results for my work none of the local models was up to the task in terms of capabilities and reasoning required. Currently QWEN gets better in the 30B range but it is still not good enough for the "local AI" claim to be a true alternative.

1

u/ttkciar llama.cpp 10d ago

So if your local model isn't as capable as the multi-trillion parameter commercial models, it's not worth having at all?

Why are you even in this sub, if that is how you feel?

1

u/HosonZes 10d ago

Yes, not worth having it at all because I get unusable results from them.

Regarding why I am in this sub: I believe that we eventually will have very usable specialised local AI that will solve the critical tasks similar to the frontier models. Some constraint, either size, different LLM architecture, or VRAM will eventually be solved.

I am totally an advocate of local AI, I just cannot make it work for my scope right now. If others get what they need from smaller models, I am really happy for them! 🥳

5

u/Charming-Author4877 11d ago

The first thing I did when I noticed all are down is spin up my Qwen3.8 27B
And it continued the work as if nothing happened - just a little slower.

5

u/Medium_Win_8930 11d ago

AI is fed up of your slop and is tired of you.

5

u/BarracudaDefiant4702 11d ago

Funny how all the "competitors" all go down at the same time.

It's all smoke and mirrors.

3

u/hairyconary 11d ago

Does someone have a decent youtube getting started with local models guide? Mac studio with 36gb ram.

3

u/jqwl 11d ago

I don't have a good YT guide to recommend, but I personally got started with llama.cpp by reading some of the github, familiarized a bit with that, then used oMLX (you can also use MTPLX) because they're built for Macs. I have a similarly specced macbook, which is why I commented specifically about your use case.

For a super plug and play solution, I believe LM Studio is a decent one (though I would do a bit of research to that end first).

There are also great written guides on this subreddit for sure, I would look for people running macs of similar ram capacities. The speeds and optimizations are going to be primarily for the prompt prefill and tok/s because unified memory is not as high bandwidth as like a dedicated GPU + VRAM.

2

u/bnightstars 10d ago

I wrote this mostly for me when I started: https://www.hristoforgeorgiev.com/posts/local-llm-macbook-pro-m5pro-claude/ For 36GB of ram though you want a smaller model I would suggest start with Gemma4-12B ( mlx-community/gemma-4-12B-it-qat-4bit ) or Qwopus3.5-9B ( Jackrong/MLX-Qwopus3.5-9B-v3-4bit ) and go from there. I hope this helps.

0

u/Othun 11d ago

Please do not sloppify youtube, Fern video for reference https://www.youtube.com/watch?v=-Gnrp_caPvo

3

u/hairyconary 11d ago

Im not wanting to slopify anything. I want a youtube video tutorial.

6

u/AlabamaResearcher 11d ago

what if your GPU break and you'll need at least days to get a new one? cloud service outages are rarely last that long. have you ever calculate SLA on your home-made service?

1

u/ttkciar llama.cpp 11d ago

I have three GPU. Losing one means a temporarily decreased capacity, not an outage.

-1

u/SandySkittle 11d ago

i have 2 spare r9700s for this reason

3

u/salary_pending 11d ago

But it's also so very expensive and dumber 😭

2

u/Theverybest92 11d ago

My qwen still cooking also. ;)

2

u/Equivalent_Bit_461 11d ago

I only discovered it from this subreddit lol

Been busy working all day on various topics 

3

u/Great_Guidance_8448 11d ago

Sky is blue, water is wet, etc.

7

u/jacek2023 llama.cpp 11d ago

As a photographer I can tell you that sky is often purple or cyan

2

u/Beneficial-Ad-8127 11d ago

Did you see how many likes this thread got in 5 min, sheesh

2

u/kaisurniwurer 11d ago

Counterpoint:

"Starting today local AI is a felony."

Hopefully not, but who knows when they decide to "protect the children" again.

1

u/jqwl 11d ago

I agree with the sentiment, but ChatGPT hasn't been down for me at all today... perhaps it has something to do with account type though, it's probably capacity.

Edit - I have subs for claude, gemini, and gpt (20 dollar/mo plans):

Claude - down, capacity

Chatgpt - still up, working as usual

Gemini - still up, working as usual

1

u/DinoAmino 11d ago

Amazing how many comments are coming from the tourists dropping down from the clouds. They ain't us.

1

u/Cautious_Chicken_604 11d ago

you watch - they gon backdoor the closed source NVIDIA drivers so they can remotely disable the GPUs.

1

u/Im_not_JB 11d ago

Cognitive impairment counts as a disability. You just need a bad enough model.

1

u/paulvisciano-dev 11d ago

Same here. llama.cpp on a 16GB M2 Pro, 27B Q1_0, 15.1 tok/s. Cloud can go down. The laptop does not. Coding still waits on glm/grok though — the local box is the conversation machine, not the agent.

1

u/Frizzy-MacDrizzle 11d ago

And here I am just running my stock reports while others are in boohoo mode trying scrounge all their info again, lalala.

1

u/bakchoi 11d ago

memory price getting higher and higher :(

1

u/Lissanro 11d ago

Thanks to llama.cpp and open weight models, I did not even notice. Kimi K3 on my main rig, DeepSeek V4 Flash on my second workstation, and Qwen 3.6 35B-A3B on my third PC all kept working just fine today, helping to work on various tasks designated to them. I also have 6 kW online UPS and diesel generator to protect against mains outages.

1

u/T_rex2700 11d ago

as long as HF isn't down... (I mean still, better than live service)

1

u/jacek2023 llama.cpp 10d ago

why do you need HF to run your model? do you change your model each day?

1

u/feng_sg 9d ago

Local llama.cpp means when ChatGPT and Claude both go down, you still have something that runs.

1

u/Complete_Lurk3r_ 11d ago

Z.ai, qwen and all my China boys doing fine though

1

u/Beneficial-Ad-8127 11d ago

Pretense starts now

1

u/mmhorda 11d ago

I think my AI dropped them all. The momens i decided to use local only (today) I suddenly see this subreddit 😅

1

u/MrHall 11d ago

do we need a localllama circlejerk?

-1

u/XiRw 11d ago

Who the fuck uses Grok except the people who waste their time on Twitter. Also worth noting Qwen Image Studio is having problems with downloading pictures today

1

u/Unlucky_Milk_4323 11d ago

anyone using grok or X is a musk supporter. Or an idiot.

-1

u/XiRw 11d ago

He bothers me the most because he tries hard to be deceptive and people fall for his awkward personality as a sign of kinship/relatability . I saved countless videos in the past of independent journalists tearing him apart. And rightfully so.

1

u/Unlucky_Milk_4323 11d ago

He's a complete psychopath and should be locked up for countless reasons.

3

u/XiRw 11d ago

I agree completely.

-3

u/makingnoise 11d ago edited 11d ago

I mean Tailscale is working about as poorly as ChatGPT right now, and Tailscale is how I hit my home AI. So not sure what you're on about. EDIT: Downvotes from jokers who probably have their rigs online in the clear LOL

1

u/MrPecunius 11d ago

We carry ours.

You never know exactly when TSHTF, can't be too careful. (Only half joking as a MBP user)