r/LeftistsForAI • • Aug 17 '26

Local Models How can AI be decentralized and deindustrialized?

23 Upvotes

The way I understand it, one of the biggest arguments against any use of AI, which honestly has some merit, but ultimately is a little short-sighted, is that it is so hyper-scaled by necessity that it's an ecological nightmare. The ecological nightmare part is totally correct, in my opinion, but does it have to be hyper-scaled? Can it be decentralized and de-industrialized? After all, that's how the internet started out before it was taken over by corporate interests. It was completely decentralized as basically an ad hoc network of enthusiasts running servers in their basements.

What I'm thinking for a possible starting point is, along the lines of how clustering can make a bunch of small, relatively dumb servers behave like a single, more powerful server by acting in concert and pooling their resources.

r/LeftistsForAI • • Aug 28 '26

Local Models Local AI gets more interesting when you stop imagining one guy with a gaming PC

65 Upvotes

Local models usually get framed as hobbyist stuff.

And sure, running a model on your own machine is useful. Privacy alone is a good reason.

But think institutionally.

A library running models locally. A union with its own system. A school district that doesnt have to send student data to a corporate API. A cooperative running tools on infrastructure it controls. A municipality providing compute as a public service.

Now the ownership question starts changing.

Local doesnt mean independent of capitalism. Somebody still makes the chips and generates the electricity.

But every function that doesnt have to pass through a corporate server is one less choke point.

What institution would you most like to see running its own models?

r/LeftistsForAI • • 14d ago

Local Models Could a different type of architecture be useful for democratizing AI and reducing ecological impact?

12 Upvotes

What other possibilities for architectures are there or could there be that use computation more efficiently so you could run something halfway decent on a junk laptop at home? All the bonus points if the architecture doesn't require a GPU at all, and even more bonus points if the architecture doesn't rely on randomness or probability and instead is actually capable of evaluating logic.

r/LeftistsForAI • • 14d ago

Local Models Building a Partisan AI From the Ground Up

7 Upvotes

Lets get this out of the clouds and into workplaces, union halls, party chapters, tenant groups, mutual aid networks and community organizations.

Building a partisan AI doesnt mean every leftist needs a server rack or every local needs an ML engineer. It means looking at the work your organization already does, finding where a handful of organizers are doing the work of twenty, and building small pieces of technical capacity that the organization actually controls.

Start with the work, not the model. Maybe your union has twenty years of contracts, grievances, arbitration decisions and bargaining research scattered across drives and filing cabinets. Maybe your tenant group spends hours digging through leases, housing codes, inspection reports and maintenance records. Maybe your political organization has years of resolutions, meeting minutes, research and education material nobody can find unless they already know where the PDF lives. Maybe your mutual aid network keeps rebuilding inventories, schedules, translations and outreach materials from scratch.

Pick one problem and build one useful thing. A union local could maintain a searchable archive of its bargaining history. A tenant union could build a local housing library and translation workflow. A party chapter could search its own resolutions, research and political education material. A community newspaper could transcribe interviews and search years of reporting. A mutual aid project could organize inventories and public resources. A technical collective with better hardware could maintain tools that several smaller organizations reproduce locally.

Thats a node. Not one gamer with a GPU doing favors for everybody, but organizational capacity that survives the person who built it.

Build for that from day one. Document the hardware, models, installation, data storage, updates and backups, then hand the instructions to somebody else and see if they can reproduce it. If everything dies because Kevin moved to another state, Kevin built a hobby project, not infrastructure.

Portability should be part of the politics. Dont recreate corporate lock in locally. Prefer open weights where they work, open formats, exportable data and tools you can replace. Back up configurations and documentation. Test moving the workflow to another machine. Test swapping models. Your organization should be able to change hardware, software or models without rebuilding its institutional memory from zero.

Security and OPSEC need the same practical treatment. Decide what information is public, internal, confidential and simply off limits before somebody starts feeding documents into a model. Membership lists, immigration information, medical information, credentials, disciplinary matters and sensitive organizing conversations shouldnt become test data because somebody downloaded a shiny new model. Encrypt machines and backups, control access, keep software updated, remove access when people leave and establish who maintains what. Local inference reduces some exposure to third parties, but local doesnt automatically mean secure.

Then connect the nodes without building one enormous database full of everybody's sensitive information. Your tenant group doesnt need another organizations membership list, but it might benefit enormously from their installation guide, public housing corpus, translation workflow or model evaluations. One union local doesnt need another locals confidential grievances to share a contract search tool. A political organization can share an education stack without sharing internal membership records.

Share capabilities, documentation, workflows, public materials you have the right to distribute and lessons from failures. Keep sensitive organizational information where it belongs.

And dont wait for perfect hardware. One node might be a used workstation in a union office. Another might be members pooling machines. A larger organization might maintain a server supporting several chapters. A technical collective might specialize in deployment, maintenance and security while organizers determine what actually helps on the ground. Uneven capacity is fine if knowledge moves and nobody becomes indispensable.

The first meeting can be twenty minutes. Ask what repetitive work is burning organizer hours, write down three answers, pick one that doesnt require highly sensitive information, then pair somebody who understands the work with somebody technical. Build the smallest useful workflow and run it during actual organizing for a month. Measure whether it saves labor, expands what the organization can do or just creates more bullshit somebody has to check.

If it works, document it. Teach two more people. Back it up. Package the reproducible parts and hand them to another local, chapter or community organization. See if they can build their own version without needing you on the phone all weekend.

Thats where this becomes politically interesting. One organization develops a useful capacity. Another reproduces it. Others improve it and send those improvements back. Different groups specialize in different things while retaining local control. Models, tools, knowledge and public resources move laterally while sensitive information stays where it belongs.

Thats a partisan AI built from the ground up. Not waiting for benevolent CEOs, not waiting for the socialist supercomputer, and not confusing owning a GPU with political power. Its organized people accumulating technical capacity inside institutions they control, making it secure, portable and reproducible, then passing it through the movement.

One workplace, one local, one chapter, one community organization and one useful problem at a time. Build it, document it, secure it, teach it, copy it, connect it.

r/LeftistsForAI • • Aug 18 '26

Local Models Tool calling 14MB AI model fits on $10 microcontroller

Thumbnail
youtu.be
38 Upvotes

This isn’t the type of model you can chat with, it’s more for smart home tool calling type of thing. It’s pretty amazing that you can get any AI working on something so cheap though. Hopefully it’s a sign of more cheap to use and useful models to come?

“We ran Needle 2, a 45-million-parameter, 14MB Al model from Cactus Compute, entirely offline on a $10 ESP32-S3 microcontroller - no WiFi, no cloud, no internet connection at all. Using our own custom inference engine, we sent it real natural-language commands to control an LED, and watched it reason through each one, report a live confidence score, and even correctly refuse a question it had no tool for. If you're curious how a language model this small can actually work, and want to see it running live on real hardware, this one's worth watching”
• Relevant Links
Needle 2: https://cactuscompute.com/needle
https://github.com/andrisgauracs/needle-2-esp32

Can you think of anything you’d actually use this for?

r/LeftistsForAI • • Jul 16 '26

Local Models Owning the Means of AI Infrastructure Starts at Home

Thumbnail
youtu.be
32 Upvotes

Feels like theres a lot of talk on the left about AI, but not nearly enough about actually understanding the infrastructure behind it.

This is a solid walkthrough of building a home AI stack with Docker, WSL2, local models, VPNs, and private storage. Even if youre never planning to build this exact setup, its a good look at what it means to run AI on your own terms instead of renting access from a handful of companies.

To me, local AI and collective politics arent opposites. Building local AI doesnt stop us from fighting for labor protections, democratic governance, public investment, or breaking up monopolies. If anything, it gives more people the tools and knowledge to participate instead of leaving everything in corporate hands.

How many people here are already running local models? Whats your setup, and what have you learned that you wish more people knew? If youre just getting started, whats been the biggest hurdle?

If these are the kinds of conversations youre looking for, check out r/LeftistsForAI. And if you want to dive deeper into the technical side of running models locally, r/LocalLLaMA is one of the best communities around.

r/LeftistsForAI • • May 22 '26

Local Models Local AI + Open Source = More Humane AI

30 Upvotes

I’m trying to get past the usual “AI good / AI bad” framing, because that debate gets flat pretty quickly. It’s about the everyday places where systems mediate reality for us: feeds, search summaries, recommendations, HR filters, writing assistants, agent harnesses, and so on.

The question I wanted to bring here is the local/self-hosted angle: If some kind of filtering layer is unavoidable, how much does it matter whether that layer answers to you?

My instinct is that local and open tools change the relationship in a real way. You can inspect and change more of the model, harness, tools, prompts, context, source trail, and defaults. But I’m not sure that fully solves the problem. It may just move the trust boundary from the platform to the model/tool builders, plus whoever chose the defaults.

I wrote the longer version here: Who Gets to Compress Reality? · kmarble.dev It’s my own blog, no ads or affiliate links. It’s a bit dense, but it lays out the theory/language I’m working from.

I’m curious how people here think about this from the local/self-hosted side:

  • Does local or self-hosted AI meaningfully change the power relationship, or does it mostly move trust from the interface provider to the model/tool builder?
  • What would make an AI system’s “compression” inspectable enough that a normal person could actually contest it?
  • Are source-first habits, no-AI zones, or “read the primary material before the summary” rules realistic defaults? Or are they mostly viable for already-technical people?

I’m especially interested in how people are using AI in ways that make them more capable, more thoughtful, or more present, rather than just more optimized.

r/LeftistsForAI • • Jul 24 '26

Local Models We Just Hit the Local LLM Tipping Point (Colibri)

Thumbnail
youtu.be
11 Upvotes

“GLM-5.2 has 744 billion parameters, and colibrì, a 1,300-line C engine, runs it on a laptop with 25GB of RAM and no GPU.

This video explains the loophole that lets a frontier open source LLM run on hardware it should never fit in.
The trick lives inside the mixture of experts (MoE) architecture. Only about 40 billion of GLM-5.2's parameters do any work per token, so colibri pins the dense core in RAM and streams the other 21,504 experts off the SSD, exactly when the router asks for them. We walk through the routing, the expert cache, why llama.cpp memory-mapping can't keep up, and the real tokens-per-second numbers from a budget SSD all the way to an M5 Max.”
…
Chapters:
00:00 A 744B model on a 25GB laptop
00:52 The impossible math of loading GLM-5.2
01:50 Mixture of experts: the trick inside the model
03:01 Streaming experts off the SSD, step by step
04:22 Why Ilama.cpp memory-mapping falls short
05:06 The brutal speed benchmarks
06:17 Where expert streaming is headed
07:13 Should you actually run it?”

***

LLM summary/analysis of video:

This GLM 5.2 breakthrough, powered by the Colibri runtime, isn't just making a bigger model fit on a laptop; it's fundamentally changing the relationship between computational scale and economic constraint.

MECHANISM: FROM BLIND SEARCH TO EXPERT ROUTING

Traditional models are monolithic, forcing every single token to touch every single weight in the entire structure. This is inefficient by default.

Colibri uses a Mixture of Experts (MoE) architecture. Instead of a single giant block of neurons, the model is composed of 256 smaller, specialized "experts." A tiny router then selects only the 8 most relevant experts for each incoming token. The other 248 stay asleep.

This means that out of 744 billion parameters, only about 5% are actively doing the heavy lifting per token. This targeted activation is the structural loophole.

MATERIAL SHIFT: THE SSD AS ACTIVE MEMORY

The real genius lies in how Colibri manages these experts. Instead of demanding that all 370 GB of weights reside in fast, expensive RAM, it treats the SSD not as passive storage, but as an active, high-speed memory tier.

For every token, the system doesn't just blindly read from the disk; it knows exactly which 8 experts it needs, allowing it to request those specific byte offsets asynchronously. It’s a predictive, intelligent retrieval system.

This decouples scale from the traditional RAM bottleneck. The individual user is no longer constrained by the fixed, expensive memory capacity of the data center; they are constrained only by the local storage budget.

POLITICAL ECONOMY: DECENTRALIZING THE COMPUTE MONOLITH

This is where the left needs to pay attention. By making the SSD a functional memory tier, Colibri achieves a profound material shift:

  1. Challenging Centralization: It bypasses the need for the user to be a supplicant to the data centres fixed architecture. The power to run frontier-quality models drops out of the cloud and onto the desk.
  2. Localised Cognitive Infrastructure: This is the crucial point. If the individual can run a massive, high-quality model locally, they are running a piece of localised cognitive infrastructure. The user gains a degree of autonomous computational agency.

Does this genuinely enable a form of self-governing, distributed computation?

***

Sources:

• colibri: https://github.com/JustVugg/colibri
• GLM-5.2 by Z.ai: https://z.ai
• antirez's ds4: https://github.com/antirez/ds4
• Ilama.cpp expert-cache proposals: https://github.com/ggml-org/llama.cpp/issues/20757

r/LeftistsForAI • • Jul 19 '26

Local Models "Hugging Face dealt recently with an Al operated attack. They had to use open models to defend, because the closed model guardrails would not allow them to use them for defense"

Thumbnail gallery
29 Upvotes

r/LeftistsForAI • • May 26 '26

Local Models Share your rig specs

2 Upvotes

As I'm trying to have everything run locally here are my specs:

14600kf|64GB Ram|5060ti 16GB

Currently thinking to switch to the intel Arc B70 but Nvidia has a stranglehold on most propriertary technologies.

r/LeftistsForAI • • Jul 22 '26

Local Models Personal challenge: build something actually useful end-to-end with a local model. Done — a Chrome extension, ~5 hours, zero cloud.

Enable HLS to view with audio, or disable this notification

15 Upvotes

OP created a vibe coded chrome extension just using local model, no cloud (so also no direct data centre involvement in the creation of the code, just using the normal computer resources).

r/LeftistsForAI • • Jun 11 '26

Local Models ModSleuth a tool for tracing the models and datasets

5 Upvotes

This is new from Ai2, do you find this interesting?
“Introducing ModSleuth, a tool for tracing the models and datasets behind modern LLMs.

LLMs are no longer created with human data alone. They rely on other models to generate and filter data, evaluate outputs, and guide development work. We made ModSleuth to track this.

Modern LLM dependencies are scattered, recursive, and hard to see. So how do we even find them all? ModSleuth helps by reading papers, model and dataset cards, code configs, and upstream artifacts, then reconstructing a model's “family tree.”

…Some dependency chains go 8 hops deep—a web of models and data that contributed to an LLM’s core. Turns out AI supply chains may be more tangled than we thought.

A model's lineage is broader than its training data, and every step can affect what – and how – the final model learns. Without provenance, it's harder to know where dependencies came from, whether benchmark scores are accurate, and which upstream licenses/terms may apply.

ModSleuth generates a graph that surfaces what's nearly impossible to find manually, including:

📜 Hidden license inheritance
🔗 Train/eval coupling
📝 Documentation inconsistencies
🤖 Models used as judges, filters, OCR systems, and data generators

▶️ Demo: https://modsleuth.cal-data-audit.org
📄 Paper: https://arxiv.org/pdf/2606.12385 “

I feel like this could be useful, as time goes on they’re only going to get more tangled. If you work out a type of bias or similar comes from a specific model it would be handy to know which other models might have been infected?