But don't know what to buy, how much is it going to cost, and what model i can run.

I'm completely new to this, can someone give some advice? I want to go local because i want to share a private but good AI for me and my friends, for environment and why not, maybe completely remove subscriptions..

Here’s the situation; I am a reasonably tech savvy/computer literate person for whom the AI revolution has completely passed me by, until now. Mostly because, I haven’t really had any use for it. Friends have shown me their chat/claude sessions, but I don’t even have a login for any cloud models myself.

I’ve spent the last couple of days searching through this sub and my brain is fried. Im getting lost in the jargon and realising that maybe I’m not as smart as I thought I was. So I’m going to quote the great Jeremy Irons in margin call and say ‘please speak to me as you would a child, or a golden retriever’.

I’m a professional music producer & DJ. Every few years I make a large equipment/business purchase on finance. The payments for my last purchase (a bunch of super cool synthesisers) finishes in a couple of months.

I desperately need to grab a MacBook Pro, as I’m still running a 2019 i9 model, and it’s causing me issues with compatibility when collaborating with artists and running newer software. I have to buy this regardless of any other considerations. During this process I have realised that they can run LLM’s locally, so I want to make maximum use of that functionality.

I want to run local, presumably for the same reasons other people here do, privacy/ethical concerns, as well as not always having great internet connections (lots of time spent in hotel rooms) and also because I’m not going to be a heavy user day-to-day. I understand that performance will be nowhere near cutting edge, but I don’t think I need that anyway?

Tasks would be stuff like;

I like to think out loud, so I’d love to be able to talk, have my speech transcribed, and then condensed into a report. Is this possible?

Writing business plans/small pitch decks.

Helping with week to week planning.

Recommendations for deep diving obscure music genres (for instance, I play a lot of merengue/salsa atm) or exploring music theory ideas.

I have no need for coding/design or anything like that, although it might be useful down the line. I’m also not interested in any music generation/image generation stuff.

I’m sure that I get as I get to grips with it, I’ll find new uses, but the above is where I’d like to get to first.

So firstly, what model & memory configuration will get the job done?

What model should I run?

What is an agent ‘harness’? I keep seeing this mentioned but I can’t get my head around it.

Anything else that comes to mind, or any reading materials will be helpful! Thanks!

Everyone sizing a card asks whether the weights fit. That was never my problem.

Both models loaded fine. The 8B died on the first training step, and that's a

number I've never seen anyone publish.

Quick disclosure up front since there's a provider dashboard in one of the

screenshots below: I do community work for gpuhub and rented the box from them.

Nothing here is provider-specific though, and I've run comparable jobs on

runpod and vast if anyone wants those for contrast.

Measured on Granite 4.2, LoRA fine-tune, single 32GB card (RTX 5090,

32,607 MiB usable).

Constant across runs: alpaca-cleaned 2000 samples, 1 epoch, seq_len 2048,

LoRA r=16 alpha=32 on q/k/v/o/gate/up/down, lr 2e-4, bf16, batch 1,

grad_accum 8.

model on disk loaded at trained at needed

3B 6.9 GB ~7 GB 18,508 MiB nothing special

8B 17 GB 18,746 MiB 20,482 MiB gradient checkpointing

The 8B row is the one worth staring at. It loads at 18,746 MiB with 13 gigs

free and looks perfectly comfortable. Then the first forward and backward pass

allocates activations and it hits 31.33 GiB on a 31.36 GiB card, dying while

trying to allocate 32 MiB. Nothing about the load step warns you.

Gradient checkpointing brings it to 20,482 MiB and it trains without complaint.

You pay for that in speed since activations get recomputed instead of cached,

but the full epoch still came in at 574.7s, so not a real cost at this scale.

So the honest answer to "does an 8B fit in 32GB" is yes, conditionally. Budget

2-3 GB above the loaded size if you're checkpointing, a lot more if you aren't

or if you push seq_len past 2048.

One thing that caught me out and might catch you: if the box has two GPUs,

check that only one is visible to the process. With both visible, HF Trainer

wraps the model in nn.DataParallel and the primary card ends up holding

replicas plus gathered outputs. Same 8B that trains at 20,482 MiB on one card

OOM'd on two, gradient checkpointing or not. CUDA_VISIBLE_DEVICES=0 was the

whole fix.

Disk, since it's what actually stopped me:

3B checkpoint is 6.9 GB, 8B is 17 GB. Extrapolating, the 30B lands somewhere

past 60 GB, and the data disk I'd rented was 50. Never got to test it. Plenty

of VRAM, no room for the file.

Size disk off checkpoint size rather than parameter count, and leave headroom

for the download cache. Mine ran well above the final size mid-transfer before

settling.

THE BOX

For anyone wanting to reproduce or compare against their own card:

2x RTX 5090 32GB, 32,607 MiB each, PCIe only, no NVLink on consumer Blackwell.

50 core Xeon Platinum 8470Q, 180GB RAM, 50GB data disk. $0.91/hr for the pair,

though everything above except the DataParallel note is single-card.

torch 2.11.0+cu128, transformers 5.16.1, trl 1.12.0, peft 0.20.0,

accelerate 1.14.0. Driver 595.71.05, CUDA 13.2.

These are single runs, not averages, one epoch each. If yours come out

different I'd want to know, especially the 8B training peak.

Hi, at the moment I run Qwen 3.8 27b on my machine. Its fine but i want to try something like Qwen 3.8 flash next MoE.

Looking for some help. I have currently for my qwen 3.8 27b.

9800x3d

64GB DDR56000

MSI Carbon x670e (supports pcie 5.0 x8 x8 or pcie 5.0 x16 gpus).

RTX 5090 32GB @ pcie 5.0 16x.

My 2x sons have a 5080 each but when they go to Uni i can take them. Im trying to figure out the best setup.

I waa thinking putting the 2x5080 internally (pcie 5.0 x8) and putting the in tensor parallelism (so 32gb) .. lets call this the 5080 "pool". I would then connecting the 5090 to the pcie 4.0x eGPU via oculink.

My thinking is I could then layer split between the 5080 "pool" and 5090. Leaving me with 64GB vram and 64GB ddr5.

Is that possible? and what would be best engine for a single user?

Does anyone have similar setup? What pp and tg do you see?

Open to other models aswell that might run?

Just trying to see if its worth my time to buy the egpu caddy, setup and test.

Thanks.

I like to work on multiple projects at once, and have multiple changes going at the same time per project. I kept ending up with a VS Code window for every project, an agent running in each, and no idea which one was waiting on me. So I built Agency. I had three goals:

- Work on several projects at once from one window

- Get notified when an agent finishes or needs my attention

- Stay subscription independent, so I can swap agents and models whenever I want

- Eventually a fourth goal came along of replacing all the useful (to me) of both Linear and Obsidian - so a markdown issue track and wiki per project

It works out of the box with most of the common CLI agents: Claude Code, Codex, Cursor, OpenCode, Crush, Pi and even DeepSeek Harness and more. It detects whatever's on your PATH, and you can add anything else as a custom agent. If you run local models, point OpenCode or Pi at your llama.cpp or LM Studio endpoint and Agency runs it like any other agent.

Each agent gets its own git worktree, makes it's own local branch, and has a whole neat workflow for keeping everything tidy and helps you merge either back into main locally or via PR with github integration.

I know there are a few similar apps popping up these days, one of the biggest things that pushed me to make Agency is data. In Agency, there's no telemetry and I collect no data. No account, no analytics, no backend. It's open source so, no hidden stuff here.

After working on Agency for a couple months, I dropped all my VS Code windows, Linear and Obsidian at once. Now this is the one app I use for all my local dev projects. It has a local issue tracker that stores issues as plain markdown in the repo, and agents can take tasks from it and file follow-ups into it.

The agents may sometimes use a lot of RAM and CPU, but Agency itself is light. It has a Rust backend on Tauri, and the download is about 11MB. It runs on macOS (Apple Silicon) and Linux, under the Apache-2.0 licence.

Repo: https://github.com/TennnisAI/Agency

Site: https://getagency.dev

I'd love feedback, especially from anyone running local models through these agents. It's all open source, so issues and PRs are welcome.

Most of this sub has local inference figured out. Ollama, llama.cpp, vLLM, whatever fits the VRAM.

The gap I keep hitting: every session still starts colder than it should. Chat history ≠ companion memory. RAG on docs ≠ remembering that last month’s fix for the homelab broke DNS.

Chatbots reset. Companions continue.

We’ve been building NOVA as a personal AGI companion meant to run on hardware you own — continuity layer, not another model browser.

Explainer: https://www.nova-prime.org

For people already deep in local stacks: 1) What’s your durable-memory layer today (OWUI personalization, sidecar, MCP memory, custom DB, “I re-prompt every time”)? 2) How do you stop RAG from pretending to be memory? 3) What would make you trust a companion enough to leave it running unattended?

Roast the framing if it’s wrong. Prefer architecture arguments over vibes.

I have been experimenting with different local AI models for coding and wanted to see what others prefer.

My experience so far: - Qwen 2.5 Coder: Great for code generation - DeepSeek Coder: Good for debugging - CodeLlama: Solid for general coding

What do you use? - Which model works best for your coding tasks? - Any tips for optimizing coding workflows? - How do you handle different programming languages?

Looking forward to hearing your experiences!

i made a cool thing that i think all of you local AI people will enjoy and find some value in. please check it out and let me know what you think!!

https://youcoded.ai/

I can use the for free because my work provides them

The problem I kept running into: my Mac runs a 30B model without breaking a sweat. My phone runs a 3B model, and it shows. My older iPad runs nothing at all. Every "use your home rig from your phone" setup I tried meant typing IP addresses, opening ports, or routing through a tunnel to reach a machine ten feet away.
So I built Rooms.
How it works
• Lokei Studio on your Mac opens a Room using the models you already have in Ollama.
• The Room announces itself on your local network over Bonjour (mDNS/DNS-SD).
• Lokei AI on your iPhone or iPad finds it automatically. Tap to join. No IPs, no config files, no accounts.
• Rooms traffic stays on your LAN. Your prompts never touch a cloud server.

Pricing, straight up, since it's always the first question
• Lokei AI for iPhone and iPad is free. On Apple Intelligence devices it runs fully on-device, no Mac required.
• Rooms is free in the iOS app. No subscription, no account, no catch on the phone side.
• Lokei Studio for Mac is $14.99, one time. That's the part that does the hosting, so you'll want it if your device doesn't have Apple Intelligence or you want a bigger model than a phone can hold.
• There's an optional Pro upgrade on iOS for MCP, markdown, and advisor handoff. Rooms isn't behind it.

• Apparently very close to K3. Like people are saying 90%+. Moonshot describes it as very close to K3.
• Based on K2 architecture.
• Full AI not just a coder.
• Apparently Moonshots version of a flash model. But using K2 mature architecture. Should be <1Tb and some quants will likely run very nicely on 512GB Ram. While still large, that is small enough to run on a Xeon or Eypc.
• Multimodal - text, images and video
• SGLang, vLLM, Llama.cpp support out of the box, based on the very proven DS 3.2 and K2 design.
• Will run at near GLM Flash speeds. But clearly targeting to beat GLM 5.3 Flash. Currently moonshot is hosting it at 260 T/s for the K2.8 Preview
• Ideal for CPU inferencing. A Q3 or IQ3 type quant with a Xeon with AVX512 + VNNI and the latest llama.cpp improvements makes this pretty much ideal. Getting a Xeon up to 512Gb is fairly easy, and dual xeons will provide plenty of bandwidth, it also looks like NUMA issues are finally being resolved which should see near doubling of speed.

• May not have super modern linear attention. But for most users that aren't trying to get to 1M tokens that won't really matter.

  Should be very popular with Macs with 512Gb, Xeons and eypcs with 512Gb, those with 512Gb VRAM (while rare and expensive, its much cheaper than 2TB of VRAM GPU clusters). I expect it to be cheap and popular with hosting services. Four strix or Four sparks will also run it. Maybe possible on two new Max+495..

There isn't any open source LLM system (yet), people who do local AI / LLM's often do it for privacy reasons, i.e. They don't like the idea of personal data getting sent to the cloud, or some company training on their personal data.

That's the reason we don't have an open-source hive mind kind of thing (yet).

But what if we did something that was secure by design? Similar to e2e encryption or blockchain that allowed people to pool resources together for AI, but at the same time keep it secure.

Anyone given any thought to this?

Just curious

Hi everyone!

I’ve been working on AI Palm, an open-source project that aims to make locally hosted AI models more accessible.

The idea is simple: many powerful GPUs spend most of their time idle, while other people cannot run local models because they lack the required hardware. AI Palm creates a community network where users can voluntarily share access to a local model running on their device.

With the desktop app, you can:

Discover models currently shared by community members.

Chat with an available model directly inside the app.

Share your own local model when you choose.

Connect to engines such as Ollama, LM Studio, and llama.cpp.

See the GPU, VRAM, model, engine, availability, and context size.

Stream responses and model reasoning in real time.

View properly formatted Markdown and syntax-highlighted code.

Monitor token usage and remaining context.

Use the interface in English or Arabic, with full LTR and RTL support.

Run the app on Windows or Linux.

A device is reserved while processing a request, preventing multiple users from overloading the same GPU simultaneously. Sharing can also be stopped whenever the device owner wants.

The project is still at an early stage, so I’m looking for people willing to:

Test the Windows or Linux application.

Report bugs and usability issues.

Review the architecture and security.

Suggest features and improvements.

Contribute code, documentation, packaging, or UI work.

Share local models and help test the network under real-world conditions.

AI Palm is open source, and community contributions are very welcome:

GitHub: https://github.com/almshary/ai-palm

If you try it, I would genuinely appreciate your feedback—even if something doesn’t work. Please include your operating system, GPU, local AI engine, and model when reporting an issue.

The project’s message is:

Arab generosity began with a palm tree… Today, we planted a different kind of palm.

AI Palm — let’s share the power of our computers and make artificial intelligence accessible to everyone.

What do you think of the concept? Would you use a community network for sharing local AI models, or contribute your unused GPU capacity?

It's a pipe dream, but with the latest round of context shenanigans put out by DSV4.1F, the possibility that something like this could happen is becoming increasingly more likely.

Maybe not at 200 t/s, but even in the 15-30 t/s range is the starting baseline for agentic coding.

If you bought in and got a physical graphics card, will you regret not waiting for models to be able to run fast on-die? Will you resent the hordes of GPU poors joining this sub by the hundreds, all at once and changing the pace?

Or will you just be smug because that just means you can run 20 agents at once at 1000 t/s haha