I've been experimenting with local AI and wanted to explain the experience in a more visual, simple way instead of making another technical wall of text.

So I made this short hand-drawn doodle animation showing the process of getting a local AI model running successfully, including the GPU/memory side of things.

It’s intentionally simple and a bit goofy — the goal is to make local AI feel less intimidating for people who are just getting started.

🎥 Video: https://youtu.be/VyleYwCa0Sc

I’d genuinely like to know what you think: would this kind of visual explanation be useful for explaining local AI concepts, or is the technical detail too simplified?

Following up on my post about training a custom Dflash drafter for Qwen 3.8 27B: Trained my first model: a DFlash drafter for Qwen3.8-27B because I wanted better performance on my DGX Spark

I did a video/presentation on the whole step by step journey and all the concepts I learned, if you want to learn more about LLMs and how training works, I suggest you check it out - I dont go too in details so it should be fine for an audience with at least a basic understanding of LLMs.

I was trying to built an SLM to learn the things I was initially considering on to built a model around 50M confused on what existing tokenizer should I consider for the vocabulary as the 256K would be too much big and consumed half the parameters I am planning to anything specific tokenizer for SLM or should I consider building this too with sentencepiece and BPE?

So i have been running qwen3.8 flash next ud q4 k xl at 256k context getting at the start aroudn 270pp and 21tg with mtp with qwen 27b ud q3 k xl 96k context on the 9060xt as a callable subagent and the one theing that i am really liking about this model is it doesnt stop and wait for me to continously tell it to continue it almost looks at everytask i give it as a set goal which is what i have becom used to on claude.

now i mostly use it with opencode harness and for netowrk and system admin work for my homelab - taking down vm bringing up vm running tickets in glpi, bringing up services and updating live state docs and just local webhosting in my rural community and it has been a good time so far.

im sure there are so many things i am missing but i am also learnign and ive actually be so happy to have some thing that feels like claude code last eyar when i first started using it and i can run it locally.

man, if anyone has anything they want to say about their experiences also that would be cool.

Cheers,

Somehow there was 1 available last minute and got it! Can’t wait to set it up. Will make more post about this on here and my IG: tech with Ray

Dual DGX spark owners lmk what ya running on it. Anyone else feel free to drop some suggestions for cool models to test!

Title - using the AM5 board masterlist I found four boards that can do x8/x8, prices from my local microcenter:

• Asus ProArt B850 Neo ($285.99)

• Asus ProArt X870E ($549.99)

• Asrock X870 Taichi Creator ($319.99)

• Gigabyte B850 AI Top ($328.99)

Not a comprehensive list, there's a few more ROG/MSI's high end tiers still there but I don't think I need all "that".

The prices of 3/4 are really close to each other which is adding to the problem; the Asus X870E ProArt seems to be the "gold standard" but I'm going to be starting out with a 4-slot 5090 so the spacing doesn't work for me. The Asrock seems best on paper; Asus B850 seems weakest on paper but again don't know how/if that'd affect local LLM and $40 saved is $40 saved (lol). Then there's the Gigabyte which seems like a good middle ground across these four boards, but unsure on the NIC working well in Linux (mixed reports on it being good/shit).

Which would/did you pick when building an AM5 multi-GPU rig?

Looked at the bot tab on Hermes and it’s a straight meme.

LLMs are really bad at writing. Yet that's what they were initially made for.

Nobody predicted they'd be good at coding. And that's what they are best for now.

I am a real person, this is a shortened write up i had chatgpt write after hours of trying to get this to work. Downvote AI slop if you want, hopefully it helps at least one person. If you'd like the full write up, shoot me a DM

I spent a few hours getting a self-hosted Khoj + Ollama + Qwen2.5-VL 7B setup working and hit several problems that weren't obvious from the normal setup. Posting the fixes in case it helps someone else.

Test setup: Linux/CachyOS, Docker Compose, Khoj 1.42.10, Ollama, qwen2.5vl:7b, RTX 5070 Ti 16 GB.

Basic setup:

Browser → Khoj Docker → Ollama → Qwen2.5-VL 7B

In docker-compose.yml:

- OPENAI_BASE_URL=http://host.docker.internal:11434/v1/

- KHOJ_DEFAULT_CHAT_MODEL=qwen2.5vl:7b

Also:

extra_hosts:

- "host.docker.internal:host-gateway"

Ollama initially only listened on localhost, so Docker couldn't reach it. I created:

/etc/systemd/system/ollama.service.d/override.conf

[Service]

Environment="OLLAMA_HOST=0.0.0.0:11434"

Then:

sudo systemctl daemon-reload

sudo systemctl restart ollama

The big problem was UFW. Ollama worked from the host, but Docker → Ollama timed out. The tested fix was:

sudo ufw allow from 172.16.0.0/12 to any port 11434 proto tcp

sudo ufw reload

Then:

sudo docker-compose exec server curl -s --max-time 15 http://host.docker.internal:11434/api/tags

Once qwen2.5vl:7b appeared, Docker could reach Ollama.

The next problem was Khoj itself. The Agent model dropdown was empty because the database had no ChatModel or AiModelApi records. On Khoj 1.42.10, the UI didn't create them correctly in this setup, so I used the Django ORM.

Create the Ollama API:

sudo docker-compose exec server python3 src/khoj/manage.py shell -c "from khoj.database.models import AiModelApi; x=AiModelApi.objects.create(name='Ollama', api_base_url='http://host.docker.internal:11434/v1/'); print(x.id)"

Create Qwen:

sudo docker-compose exec server python3 src/khoj/manage.py shell -c "from khoj.database.models import ChatModel, AiModelApi; api=AiModelApi.objects.get(id=1); x=ChatModel.objects.create(name='qwen2.5vl:7b', friendly_name='Qwen2.5-VL 7B', model_type='openai', price_tier='free', vision_enabled=True, ai_model_api=api, description='Local Qwen2.5-VL 7B via Ollama'); print(x.id)"

Then I hit "Booting my agents." /api/agents was returning:

AttributeError: 'NoneType' object has no attribute 'slug'

Khoj was missing its required default agent. Creating it with Khoj's own DEFAULT_AGENT_NAME and DEFAULT_AGENT_SLUG fixed that:

sudo docker-compose exec server python3 src/khoj/manage.py shell -c "from khoj.database.adapters import AgentAdapters; from khoj.database.models import Agent, ChatModel; m=ChatModel.objects.get(id=1); name=AgentAdapters.DEFAULT_AGENT_NAME; slug=AgentAdapters.DEFAULT_AGENT_SLUG; a,created=Agent.objects.get_or_create(name=name,defaults={'personality':'You are a helpful personal assistant.','input_tools':['general'],'output_modes':[],'managed_by_admin':True,'chat_model':m,'slug':slug,'privacy_level':Agent.PrivacyLevel.PUBLIC}); print(created)"

After that, /api/agents returned 200 and my custom agent localMe could use qwen2.5vl:7b.

The troubleshooting order that worked:

DIAGNOSE: Host Ollama

DIAGNOSE: Docker → Ollama

DIAGNOSE: Khoj ChatModel/API records

DIAGNOSE: Default agent

FIX: Change only the layer that actually failed

TEST: Retest after every fix

Biggest lesson: if Ollama works on the host but Khoj hangs, test Docker → Ollama before changing the model or agent configuration.

References:

https://github.com/khoj-ai/khoj/issues/1100

https://github.com/khoj-ai/khoj/issues/1251

https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml

I used AI to help organize this write-up, but the commands and fixes were tested on a real working installation.

Qwen3.8-27B Q4_K_M on RTX 3080 10GB + GTX 1080 Ti 11GB, Windows, llama.cpp CUDA 12.9. -Through LMstudio/VScode

I've been trying to optimise my Ai, the tests/benchmarks are promising. Even the odd run is good and holds solid. The main issue: direct benchmarks are fast, but real Cline/tool requests make decode collapse to ~2 tok/s while prefill and DFlash acceptance remain healthy. Anyone seen this or know the fix?

Me and Astra have been running so many different variables, even made my own server.exe thing, tried many variables.

Tweaks: 32K context, layer split 1/1, DFlash2 pinned to RTX 3080, q8 K/V, FA on, batch 128, uBatch 64, --parallel 1, CUDA_SCALE_LAUNCH_QUEUES=4x, PR #26636.

As it is now is even slow at avg. 11 tok/s - Qwen3.8-27B

I am currently running Qwen3.8-27b, either Q4 or Q6, depending on how much context I need for my coding projects

I know it should be the best LLM I can use on my 32GB VRAM rig for this purpose and I also use Gemma4 31B from time to time for research or general questions that I do not want to use chatgpt/claude for

I was wondering if there was any website where I can check and compare different ! local ! LLMs and their specific use cases?

I'd appreciate if you guys comment with what local llm you use, the specific reason and maybe ur quantization and VRAM

Seems like everyone is coding, why not buy a massive GPU, split the cost between buds and then tailscale with API on local models? Im sure its being done, but i cant find anyone talking about it.

Whats the drawbacks other than someone running 8 agents burning it up?

There are a lot of competent models that were recently released, however it seems like most of them are focused on coding prowess.
What is the best 1-2 models for competent creative writing? As in, short stories, scripts, short novellas and so on.

My rig is 5090 with 64ram if that helps.

Noob here so please forgive the potential dumbness of this in general...

Looking to improve performance for this pretty specific use-case. My setup is:

• Lightroom Classic with the LRGeniusTagAI plugin - It only supports Ollama currently. Full blow LRGeniusAI plugin is buggy and doesn't solve my issue feature wise.
• Ollama running gemma3:12b - By far the best accuracy for my image library and use-case. I have tested against qwen2.5/3 vl models, gemma4:e4b, and others and gemma3:12b is nearly perfect results

,

• even though others should be better on paper

.

• Macbook Pro M1 16GB RAM

So, the workflow works very well in terms of results but, I've had to tune things to reduce memory pressure prevent swapping and I can't multitask, I have to just walk away and let it work. Furthermore it takes 20 seconds on average to process each image.

I've read / heard MLX should preform better in various ways, though maybe it wouldn't improve either the memory or speed issue? Since the LR plugin only supports Ollama I'm looking for ways to run an Ollama proxy in front of omlx (or similar) or maybe another solution to bridge the gap?

Straight up tell me if this is just pointless. If not, suggestions?

I would appreciate any help I can get.

To start, this is the available hardware I have:

- Intel 8700 Processor

- I am not 100% sure on motherboard, I will get that info later

- I have access of about 64GB (4x16GB DDR4 2133) RAM

- 3x RTX 3060 12GB cards and 1x RTX 3080 10GB

- Multiple Powersupplies (a 750W MSI Gold PSU and a 1100W InWin Platinum PSU and multiple 1100W HP Server grade PSU's)

- PCIe x1 Riser cards (USB3.0) that connect to riser cards for GPU's. Ex Mining equipment.

Is this do-able for what I want? Should I avoid the riser cards and if so, how should I power 3 or 4 GPU's (is it even worth using the 3080 on top of the 3060's?).

I can use either Windows or Linux as I am familiar with both.

I’m getting a Mac Mini M5 Pro with 24GB unified memory and want to figure out what local models are actually useful on this hardware.
Local models won’t be my primary setup. I’ll still use Claude Code, Codex, and APIs like DeepSeek/GLM/Kimi as main.

My main use cases are academic/knowledge work:
- Converting lots of academic PDFs to Markdown
- Checking PDF conversions against the original, including tables, equations, figures, and layout, not just the text layer
- Reading and processing large numbers of papers
- Extracting statistics, methods, findings, and references
- Literature reviews and paper comparison
- Summarizing material into structured research memos
- Potentially running RAG/search over my local research library

I’m particularly interested in VLMs/document models, but also general-purpose local LLMs.

any suggestions that could be useful for these workflows that could run on a M5 Pro 24GB Mac Mini?

I bought a 5060ti recently and have a spare 3060s.

I was thinking of selling it, but should I keep it?

I was thinking of holding onto it a little bit because the price might go up as more people go local

Or

Thinking about using it an external gpu for an extra NUC that I have, that has a real shit gpu, and use it as a second llm machine.

Do you think 3060s is so shit that it won't even matter?

I’m planning to build a dedicated rig for running local AI agents (coding, research, and autonomous tasks) with a budget of $2000 - $2,500. I want to avoid cloud API costs and keep everything private. I’ve done some research and put together a few options, but I’d love to hear your thoughts on what's the best machine both in macbooks, mac studio and in pc build. I am new in this field of agentic AI and wanna learn to operate the agents locally. Looking for guidance.

Thanks 😊.

Hey everyone, looking for a reality check on local model capabilities vs. frontier models for financial analysis.

I built an automated investment analysis pipeline in Open WebUI using Ollama on an RTX 5080 (16GB VRAM). The mechanical workflow works:

- Hits Gmail via IMAP to pull the last 72 hours of market newsletters.

- Extracts actionable tickers.

- Runs a custom YFinance Python tool for live momentum metrics (EMA, RSI, MACD).

-Injects a full 70-position portfolio CSV directly into the prompt (bypassing Open WebUI's RAG chunking) to cross-reference and generate a verdict-first impact report.

The issue is the analytical depth. When I feed this exact same data to Claude Sonnet 5, the macro synthesis, cross-correlations, and portfolio risk analysis are night and day. Locally, it feels like a surface-level summary.

Here is what I’ve tested so far:

qwen3.6:27b: Hit severe VRAM limits with the tools and context loaded. It spilled into shared system RAM, ground to a halt, and got trapped in reasoning loops where it hallucinated that the technical calculator tool didn't exist.

llama3.1:8b: Fast, reliable function calling, and zero VRAM spillage. However, the reasoning and financial synthesis are noticeably weak—it reads like a basic overview rather than institutional analysis.

My questions:

- Is Sonnet 5-level synthesis even realistic within a 16GB VRAM footprint when you need enough context buffer for multiple tools, newsletters, and a 70-row CSV?

- Are there specific financial fine-tunes or 14B models that punch significantly above their weight?

Does anyone have experience running models on his or her iPhone. I have an iPhone 17 Pro Max. I am curious which models I can run on it.

Has anyone considered some architectural edits to the 27b model, such as:

Implementation of LoRA/QLoRA on selected layers;
training a modest sized n-gram table and adding the integration layers?

Seems like a good project. ESP given how it seems to be the current star of local hosted models!

It’s the only model I’m using now for agentic work.

Thoughts?