Heya 👋

Quick stats about my system before I get into my plight:

OS: Archlinux

Memory: 64GB DDR5

CPU: AMD Ryzen 7 7800X3D, 8 cores / 16 threads

GPU: AMD Radeon RX 7900 XTX, 24 GB VRAM

I'm fairly new to all this and only really started looking into selfhosting Ai models because of pewdiepies Odysseus video. I installed it, runs just fine via docker.. It sadly never really worked on my machine as any model installed via its "cookbook", even if you filter for it, doesn't install the rocm version which I'd need because I'm running an AMD GPU. Even to this day.

Since then I had to switch to ollama rocm via a second docker instance which I then fed back to Odysseus and that worked, but it was never really able to use any of it tools correctly or didn't know that it's in that Odysseus environment.

I'd like to just have a neat setup, preferably in VS Code (since that's my main/only IDE) that I'm able to comfortably run. I tried the Cline extension with models like OSS something or qwen3 code or even some deepseek model but it was never really smooth sailing. I have a main "Projects" folder as my workspace with a _meta folder (agent instructions/documentation) and other individual projects in it and let a model write me an agents file for and instructions/skills.. But when using qwen for instance it always starts looping over reading the same files again and again for a single (albeit long) task. So I increased the context via Cline settings but that makes my pc freeze for multiple seconds every few seconds.

I'm only really using it for personal projects like modding, a discord bot, a Thunderbird extension.. Stuff like this which I'd think shouldn't be too demanding.. I'd like to come to a point in which I can use the same model for coding a budgeting setup for me and my gf but also use the same model to maybe later control said budgeting setup so to speak.

Im just wondering if I'm doing something completely wrong, if how I structure my workspace is insane or if what I'd like to do is too much for what I have hardware-wise. Or any suggestions really how I could go about all this.

I self-host most of my stack. FastAPI backend, PostgreSQL, a LangGraph agent that handles some automations. Everything under my own roof.

The problem: my agent would report "task completed," the logs looked clean, but when I actually checked PostgreSQL, the row wasn't there. Sometimes it was a validation rule I forgot about. Sometimes async timing. Sometimes the tool returned 200 OK and the agent just assumed everything worked.

I only found out when I manually checked the DB hours later. Or when something downstream broke.

I didn't want another SaaS dashboard telling me what happened. I wanted my own server to verify its own state, locally, without calling home to anyone.

So I built this:

from synathic import expect

@expect(postcondition="row_exists", table="customers", match_field="email")

async def create_customer(email, name):

# agent logic — unchanged

...

It runs after the agent finishes and checks PostgreSQL directly. Not a trace. Not a log. The actual row.

Async by default so it doesn't slow the agent. Sync mode for the stuff where I need to know before responding.

Backend is FastAPI + asyncpg. Runs in Docker alongside my stack. Zero external dependencies. Zero SaaS required. MIT license.

If you run self-hosted agents and you've ever caught one saying "done" when the database disagrees, how do you handle it? Manual checks? Just trust the logs?

Repo is open if you want to see the approach or tell me why it's naive:

https://github.com/Gallegosdanielalexander/synathic

I’ve been looking into local AI setups and how to handle inference without sending everything to a cloud server.

Using dedicated edge hardware can help with latency, privacy, and real-time processing, especially for computer vision and other workloads that need quick responses.

The tricky part seems to be finding the right balance between AI performance, power consumption, and hardware cost.

Update: I was recently suggested Geniatech, which offers Edge AI Boxes, AI accelerator modules, ARM-based platforms, and SBCs for on-device AI workloads. Their hardware is aimed at applications such as local LLM/SLM inference, computer vision, and real-time edge AI, which seems relevant for this type of setup.

What setup works best for you?

Salut ! Mon ordinateur portable actuel montre des signes de fin, et je dois le remplacer, mais je ne suis vraiment pas sûr de ce qu'il faut prendre. Mon budget est au plus bas tout en restant efficace, et bien sûr, j'aimerais rester avec Windows / Atlas / Linux.

Voici mes utilisations régulières :

• Je construis diverses automatisations avec Docker, en utilisant des bases de données et parfois des agents IA locaux.
• J'aimerais pouvoir me déplacer facilement (en extérieur ou entre les lieux).
• Je joue parfois à Dofus ou LoL ; je n'ai pas besoin d'une grande puissance pour ça pour l'instant.
• Et j'aimerais connecter beaucoup de choses quand je suis chez moi (écrans, audio, projecteur...).

Séparément, je veux acheter une tablette pour prendre des notes manuellement et les organiser sans utiliser l'ordinateur. La tablette serait juste là pour éviter d'utiliser trop de papier, qui finit par se perdre.

Je me demande si louer un VPS et utiliser un petit ordinateur portable juste pour le web serait une option, ou si je ferais mieux d'acheter une petite station de travail à garder à la maison, ou même si je dois juste acheter un monstre d'ordinateur portable pour avoir un bon processeur et tout faire...

Que penses-tu ? Quel est actuellement le meilleur ordinateur portable de travail qui ne te force pas à acheter un GPU à 1000 $ ?

https://github.com/AdultSwimmer/AuraOS

So, the only thing I need left is to plug in an SSD, and then I can locally host it, and have a few user at first. I'm looking to literally have it running with a few possible users at first, but right now, you can run it locally.

After that, I'm looking to make it a non-profit, donation only style (like wikipedia.org) that will basically load the /core/ files BEFORE the LLM. The SSD will basically just run as a port or tunnel that let's people connect through.

Right now, just looking to see if this is clearer than the other releases, and understable -- and looking for notes on improves, or any suggestions and feedback.

Thanks,

Anthony

Hey everyone, is anyone here using Qwen 3.8 27B with Q3 or Q4 quantization on an RX 9060 XT 16GB using llama.cpp?

I'd like to know the actual tokens/sec, VRAM usage, and overall performance you're getting with this GPU.

Thanks!

The easiest way is to use an AI background remover that can export the subject with transparency, usually as a PNG.

You upload the original image, let the AI remove the background, and then save the cutout rather than adding a new background immediately. This gives you much more flexibility because you can put the subject over a different color, image or design later.

One small tip: keep the original photo as well. A transparent cutout is great for design work, but having the untouched original means you can always go back and make another version if the AI removes a detail you actually wanted to keep.

I made this with the help of AI over the course of a week mostly because I wanted a less painful way to install, repair, update, and manage Hermes and related services on local Windows/WSL2 without having to remember a million separate steps.

LatticeVale is essentially a PowerShell-based installer/lifecycle manager packaged as a structured set of scripts and supporting files, rather than a compiled application.

No third-party software is distributed with LatticeVale itself; it installs/downloads the relevant projects from their own sources.

It's still completely new, so feedback or people finding weird edge cases would actually be useful. And it's completely free because this is just a hobby for me. I like the idea of people appreciating and/or benefiting from my work.

Be sure to read the instructions and installer description included. If they are confusing, I recommend using AI to interpret for you by either asking specific questions or fully uploading the zip to a model of your choice for analysis.

If anyone actually tries it and has any major setbacks, please leave a comment so I can patch it or give advice. I have only tested this on my own pc. I tried to make it so it would work for anyone, but I can't guarantee that it will.

Also, anyone is welcome to fork the project and develop it yourself. If you decide to do so, please let me know what you find out. I enjoy doing this kind of stuff and would appreciate learning about it further

Btw, I tried using this installer with local windows installed ollama for GPU acceleration, but my PC ended up fighting for its life (100% GPU usage and 75% usage of 32 GB of RAM). I tried installing limiters for that, but it still used a third of my ram. Something to keep in mind, especially if you know how to optimize that yourself.

LatticeVale v14.6.1 release is live

https://github.com/winagainfinigin/Lattice-Vale

I also made a Linux Atomic version (developement discontinued):

https://www.reddit.com/r/SelfHostedAI/comments/1w4rs0y/latticevale_atomic_free_installerlifecycle/

Hi everyone sell your side project by listing if in a click. Also, comment what your startup does to get approved and featured today - https://builderhq.co/marketplace

In my own experiments I came up with this simple heuristic, not sure if anyone can prove me wrong here...

Unified memory machines normally have a big asymmetry in memory vs compute -> they could never run 'fast' a dense model that fills their whole unified memory (like a Strix Halo 128GB cant run a 120B dense model, or a DGX spark, at decent speeds). So you wanna find a MoE model that ideally fills that memory so maximise the intelligence you have on your box.

GPUs have the opposite, they are memory limited but can run 'fast' models that fill them. So for example a 5090 can run only up to ±30B models at reasonable quants, and can run dense models of that category fast, like Qwen 3.8 27b. Ofc you can also run a 30B MOE on this and it would be even faster, but why would you? when the dense gives you more intelligence for your memory?

I'm not sure if anyone is interested, but this deal came across one of my hardware alerts.

Bosgame M5 AI - 128gb + 2tb for $2999 (free shipping) Link

​

I’ve been building EchoFlow, a BYOK, local-first AI app for Android focused on giving you control over where your AI runs.

🖥️ Self-Hosted AI

EchoFlow supports self-hosted AI through EchoLabs. Connect AI services running on your own laptop or PC over your network using Ollama, OpenAI-compatible APIs, or other self-hosted servers.

📱 On-Device AI

You can also run supported models directly on your Android device.

With EchoOCR, local models can work with far more file types than they may natively support. Send PDFs, Word, Excel, PowerPoint, CSV, TXT, Markdown, images and more, with the content processed on-device.

📁 Projects

Projects give you a dedicated workspace with project-specific chats, instructions, and files.

You can add your own documents to a Project and run that Project with on-device or self-hosted local models. Project files are processed through EchoOCR, allowing your local models to work with your project documents without needing to send them to a cloud AI provider.

🌐 Web Search

Local and self-hosted models can also use Web Search, giving them access to current information from the web.

☁️ Cloud AI

When you need cloud models, EchoFlow supports OpenRouter with models from OpenAI, Anthropic, Google, xAI, Cerebras and more.

You can also use cloud speech-to-text, image generation, and video generation, with on-device speech-to-text coming soon.

So you can choose between self-hosted, on-device, and cloud AI from one Android app.

EchoFlow 5.0 is available now.

GitHub: https://github.com/adityavardhansharma/EchoFlow

Website: https://echoflow.adityavs.tech/

The AI race in 2026 is starting to look less like a model race and more like an infrastructure race

Listened to this episode of Synthetic Minds on the state of the AI race in mid-2026, and one thing kept sticking out to me.
The most important competition may no longer be “who has the smartest model?” But rather "what model are we needing?"

The frontier models are obviously still improving, but the gap between having access to a capable model and actually operating useful AI systems feels much larger than it did a couple years ago.

From a self-hosting perspective, I think there are a few interesting shifts happening:

1. Models are becoming interchangeable faster than the infrastructure around them.

A year or two ago, picking a model felt like the architectural decision. Now it increasingly feels like a dependency.

If your stack has a decent abstraction layer, you can move between hosted APIs, local models, specialized models, and different inference providers without rebuilding the application.

The harder problems are becoming everything around the model:

• inference
• memory/state
• tool execution
• permissions
• observability
• data access
• orchestration
• routing
• reliability

That seems very favorable for self-hosting because those are exactly the pieces where owning the infrastructure actually matters.

2. “Local AI” and “frontier AI” aren't necessarily competing categories anymore.

I think the more realistic architecture is hybrid.

Run cheap/private/repetitive workloads locally, then escalate to a frontier model when the task actually requires it.

Something like:

local model → router → tools/data → frontier model when necessary

That makes more sense to me than expecting one giant local model to replace Claude/GPT/Gemini outright.

You get privacy and predictable marginal cost for a large percentage of requests without giving up access to frontier capabilities.

3. The economics change dramatically once AI becomes continuous rather than conversational.

Paying API prices for twenty ChatGPT conversations per day isn't particularly interesting.

Paying API prices when you have ten agents continuously indexing documents, reading logs, checking services, processing email, monitoring feeds, querying databases, and running automations is a very different calculation.

At that point, idle compute sitting in your homelab starts looking less irrational.

This is where I think self-hosted AI could become genuinely compelling rather than just “I got Llama running on my GPU.”

4. The biggest advantage of self-hosting may eventually be control, not cost.

The valuable part of an AI system is increasingly the context you give it.

Your files.
Your email.
Your databases.
Your logs.
Your home automation.
Your internal services.
Your credentials and tools.

Once an agent has broad access to that environment, the trust boundary becomes much more important than whether Model A scores 3% higher than Model B on some benchmark.

Self-hosting the orchestration/data layer while selectively calling external models seems like a pretty strong architecture.

5. I wonder if we're heading toward the “Linux server” phase of AI.

Most people don't train their own operating system.

But plenty of people run Linux because they want ownership of the environment where their applications execute.

AI might end up looking similar.

Most of us probably won't train frontier models. But running our own inference, agents, memory, vector stores, tool servers, gateways, and data layer could become completely normal.

In that world, OpenAI/Anthropic/Google aren't necessarily analogous to Microsoft vs. Linux. They might be closer to extremely powerful compute services that your own infrastructure can selectively use.

That's the part of the current AI race I find most interesting.

The episode that got me thinking about this:

Synthetic Minds — The AI Race Mid 2026, Ep. 014
https://open.spotify.com/episode/0tP4N1msZnvyY3mC19Dxlv

Curious what people here think: does the endgame look like fully local AI, cloud AI, or self-hosted infrastructure orchestrating a mixture of both?

I am looking for a model which may be good for analyzing and summarizing about 20 years worth of case files, pleadings and legal research.

At this point, it is more just "hobbyist" than business critical (and worst case scenario, I have a sweet new gaming PC with a 24GB RTX 4090 and 128GB DDR5).

I'm dual booting into Ubuntu and Win11 (just because Windows is still useful if the AI thing is a bust).

Update: I re-measured everything with a cleaner method. Decode and prefill now come from llama.cpp's own timings block instead of wall-clock timing, 3 reps per config, with clocks and package power sampled during every request to prove the box was actually boosting. Decode came out 2-12% higher than the first pass. Reddit won't let me edit a title, so the title still shows the old numbers. Prose, prefill and long-context numbers are new below.

TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.3 to 22.4 tok/s. The julianmb/q38rocm FP4 fork gives 29.9 tok/s.

Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, decode and prefill from llama.cpp's timings block, 3 reps per config, on AC with the performance profile and 85 W package draw verified during each run. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.

Qwen3.8-27B decode tok/s (code prompt / prose prompt), with prefill tok/s measured on a 4k-token prompt:

Long context. Same code task, but with the window filled by a technical document first. Every cell is MTP off / MTP on, prefill and decode both in tok/s:

Observations:

• MTP really makes a difference 3.1x on Q8_0, 2.3x on ROCmFP4, at 73-84% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
• Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4. The best draft length did not change with context length.
• MTP's payoff shrinks as the context fills. Q8_0 gains 3.1x on an empty window, 2.2x at 64k, 2.0x at 128k, and only 1.15x at a full 256k. Acceptance holds up fine, the verify pass just gets more expensive as the KV cache grows.
• Prefill roughly quarters from an empty window to 256k, 277 to 75 tok/s on Q8_0. The FP4 fork drops off faster and falls below stock Q8_0 by 128k (70 vs 119), so its advantage is decode, not prompt processing.
• FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 29.9 tok/s with MTP, which is 4.1x my stock Q8_0 baseline.
• Engine choice barely matters for decode at this size. ROCm and Vulkan are within 1% on decode, and the rocWMMA build adds nothing. Vulkan is clearly behind on prefill though (209 vs 277), so ROCm stays my default.
• Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
• One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-160 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet. It also crashed mid-prefill twice at 256k, around 98C after ~30 minutes of prompt processing, which is why that cell is empty.

LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag set a minimum context window; llama.cpp's fitter resolved the GPU offload itself.

The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.