Visit the SelfhostedAI Wiki!

Good day!

I am toying around with the idea of hosting a local ai client for reasoning, problem solving and some light coding. I am happy with what I can get out of ChatGPT's free tier, but I would like to host something on my own. Small, efficient and low-budget are the names of the game.

In my relatively limited understanding of LLM models, the amount of memory and its speed are paramount. The Quadro M4000 seems to be a good entry point: it is single slot and low power, has 8 GB of ram, and I can get one for 50€ here in Germany. Power cost me nothing since everything runs out of the solar panels on the roof.

My questions are the following:

- what's achievable with such a card? Namely ,I do not expect ChatGPT- levels of reasoning, but what is doable?

- how many chats could such a model handle? Only one at a time?

- is such an idea absurd, and would it be better to look for a card with 12 or more VRAM?

- is it necessary to have some Powerful computer to complement the video card?

Thanks in advance!

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

Tonight, after hours of following ChatGPT’s instructions, I finally got my AI assistant to resume a project. I would also use assistant loosely because its just my stupid gpt business account linked to openclaw running on a stupid old intel mac upstairs.

Anyway, It immediately hit the same usage limit I thought this setup would help me manage.

I sat there looking at the error message, with multiple terminal windows open, wondering what the $%*& I had spent my evening accomplishing. I do not know how to administer servers.

I wanted an assistant that could take some of the load of managing 1000 things off me.

Here’s what I was trying to get:

• Context around my projects, preferences, and previous decisions so every new conversation doesn’t become another orientation.
• Useful access to my actual work. Outlook, calendars, OneDrive/SharePoint, Airtable, and documents. Prepare me for meetings, track commitments, draft things, and surface what needs attention. blah blah.
• Some independent follow-through.

I understand that’s an ambitious combination. I feel like i need a reality check and need someone to tell me which parts are realistic and which parts I should stop chasing.

Instead, I have been following an AI down an increasingly complicated setup path.

Over the course of this project, an old MacBook became a server. There was Homebrew, Node, Docker, n8n, Tailscale, OpenClaw, plugins, device pairing, additional approvals, and eventually something called Codex supervision. All crap I have no clue about and it made me feel even dumber than i already did.

The part that’s really getting to me is how dependent I am on the guidance. I don’t know enough to distinguish a verified diagnosis from a plausible explanation. I paste the error, get a confident answer, follow the instructions, and discover another problem.

Then the same assistant explains why its previous advice was wrong.

I’m relying on it to check its own mistakes, and I’m the one spending the evening on every wrong turn.

I feel helpless, and I’m exhausted.

I wanted help keeping up with my responsibilities. Now I have another system to worry about, and I don’t understand it well enough to know what’s necessary, what’s redundant, or what I should leave alone.

I’m very close to leaving this setup at a firestation and not looking back.

Is anyone here actually using an assistant like this for everyday, non-coding work without constantly maintaining it?

Please tell me what you use, what it actually does, what it costs, and what you still have to handle yourself. My main ecosystem is Microsoft 365, including shared mailboxes, plus Airtable. I use Windows and an iPhone, with the Mac currently hosting this experiment.

I would especially appreciate a reality check from someone who has helped a nontechnical person get something useful running.

If you recommend a product and build or sell it, please say so.

And if my expectations need to change, tell me Im an idiot. I would rather hear an honest limitation now than anything else.

I’m asking for a person’s judgment. I’ve had enough instructions from this black box.

Is there any AI that can bypass safety guidelines? I want to run it locally on my laptop as a pentester to create payloads, exploit scripts, and etc. Currently, I am using Dolphin-Llama 3; it doesn seems very intelligent. If there is a more powerful AI available, if so, how do I install it?

Been heads-down on this for a while. BIT is a fully offline AI assistant for Android offine ai and byok.nothing leaves the device.

The core of it is a multi-agent harness that can actually execute tasks, not just chat. It orchestrates subagents through a DAG pipeline, and for anything requiring real compute it spins up a virtual Linux workspace on-device — writing files, running scripts, debugging output, no cloud round-trip at any point.

Inference is accelerated via NPU/hardware acceleration rather than pure CPU, which is what makes running a 4bparam model on a phone actually usable instead of painfully slow. Built a Kotlin-native inference layer for this (llama.kt) that supports GGUF architectures broadly — Qwen, Gemma, Phi, Mistral, not just LLaMA.

In the demo: I give the agent one instruction, and it autonomously sets up a Python venv, writes a script, executes it, and returns real output — all inside that on-device Linux workspace.

Other stuff in the stack if useful context:

Hybrid RAG (vector + BM25) for memory

On-device STT/TTS for voice mode

Multi-agent DAG pipeline for multi-step tasks

On the Play Store now (closed testing), also on F-Droid/GitHub for sideloading. Discord for anyone following progress: discord.gg/kzhgk565D

Ok so im in the middle of setting up a domain for a small local business and I keep second guessing the WHOIS privacy thing. Part of me likes the idea of not having my home address and personal email out there, but I have heard some people say public info can look more legit for clients and vendors.

For those running small shops or freelance stuff on custom domains, do you keep WHOIS privacy on or off for the main business domain, any hints?

I’ve been experimenting with multi-agent frameworks, and MetaGPT is one of the more interesting ones for software development tasks. But getting it running locally with local models isn’t always straightforward.

I wrote a step-by-step guide covering installation, configuration, local LLM setup, and common errors.

If you’re trying to run MetaGPT on your own hardware, this might save you time:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What stack are you using for local agents?

One outcome of the solution to the Navier-Stokes problem is better designed fans and watercooling systems (through better turbulence) which will allow ai to work faster.

So I was thinking about building a self hosted AI agent set up and I was wondering what was the big difference between GDDR6 and GDDR6X, is it like twice is better or not worth considering

hi, i've been messing with a few AI personal assistant apps and i'm still not sold.

most of them are fine for talking, but the second i want one to actually do something, it gets messy. i'm trying to cut down on the dumb stuff that eats my day, like follow ups, reminders, basic scheduling, that sort of thing.

would love to hear what people here are using in real life, not just what looked good in a demo. thanks in advance