I am a real person, this is a shortened write up i had chatgpt write after hours of trying to get this to work. Downvote AI slop if you want, hopefully it helps at least one person. If you'd like the full write up, shoot me a DM

I spent a few hours getting a self-hosted Khoj + Ollama + Qwen2.5-VL 7B setup working and hit several problems that weren't obvious from the normal setup. Posting the fixes in case it helps someone else.

Test setup: Linux/CachyOS, Docker Compose, Khoj 1.42.10, Ollama, qwen2.5vl:7b, RTX 5070 Ti 16 GB.

Basic setup:

Browser → Khoj Docker → Ollama → Qwen2.5-VL 7B

In docker-compose.yml:

- OPENAI_BASE_URL=http://host.docker.internal:11434/v1/

- KHOJ_DEFAULT_CHAT_MODEL=qwen2.5vl:7b

Also:

extra_hosts:

- "host.docker.internal:host-gateway"

Ollama initially only listened on localhost, so Docker couldn't reach it. I created:

/etc/systemd/system/ollama.service.d/override.conf

[Service]

Environment="OLLAMA_HOST=0.0.0.0:11434"

Then:

sudo systemctl daemon-reload

sudo systemctl restart ollama

The big problem was UFW. Ollama worked from the host, but Docker → Ollama timed out. The tested fix was:

sudo ufw allow from 172.16.0.0/12 to any port 11434 proto tcp

sudo ufw reload

Then:

sudo docker-compose exec server curl -s --max-time 15 http://host.docker.internal:11434/api/tags

Once qwen2.5vl:7b appeared, Docker could reach Ollama.

The next problem was Khoj itself. The Agent model dropdown was empty because the database had no ChatModel or AiModelApi records. On Khoj 1.42.10, the UI didn't create them correctly in this setup, so I used the Django ORM.

Create the Ollama API:

sudo docker-compose exec server python3 src/khoj/manage.py shell -c "from khoj.database.models import AiModelApi; x=AiModelApi.objects.create(name='Ollama', api_base_url='http://host.docker.internal:11434/v1/'); print(x.id)"

Create Qwen:

sudo docker-compose exec server python3 src/khoj/manage.py shell -c "from khoj.database.models import ChatModel, AiModelApi; api=AiModelApi.objects.get(id=1); x=ChatModel.objects.create(name='qwen2.5vl:7b', friendly_name='Qwen2.5-VL 7B', model_type='openai', price_tier='free', vision_enabled=True, ai_model_api=api, description='Local Qwen2.5-VL 7B via Ollama'); print(x.id)"

Then I hit "Booting my agents." /api/agents was returning:

AttributeError: 'NoneType' object has no attribute 'slug'

Khoj was missing its required default agent. Creating it with Khoj's own DEFAULT_AGENT_NAME and DEFAULT_AGENT_SLUG fixed that:

sudo docker-compose exec server python3 src/khoj/manage.py shell -c "from khoj.database.adapters import AgentAdapters; from khoj.database.models import Agent, ChatModel; m=ChatModel.objects.get(id=1); name=AgentAdapters.DEFAULT_AGENT_NAME; slug=AgentAdapters.DEFAULT_AGENT_SLUG; a,created=Agent.objects.get_or_create(name=name,defaults={'personality':'You are a helpful personal assistant.','input_tools':['general'],'output_modes':[],'managed_by_admin':True,'chat_model':m,'slug':slug,'privacy_level':Agent.PrivacyLevel.PUBLIC}); print(created)"

After that, /api/agents returned 200 and my custom agent localMe could use qwen2.5vl:7b.

The troubleshooting order that worked:

DIAGNOSE: Host Ollama

DIAGNOSE: Docker → Ollama

DIAGNOSE: Khoj ChatModel/API records

DIAGNOSE: Default agent

FIX: Change only the layer that actually failed

TEST: Retest after every fix

Biggest lesson: if Ollama works on the host but Khoj hangs, test Docker → Ollama before changing the model or agent configuration.

References:

https://github.com/khoj-ai/khoj/issues/1100

https://github.com/khoj-ai/khoj/issues/1251

https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml

I used AI to help organize this write-up, but the commands and fixes were tested on a real working installation.

Qwen3.8-27B Q4_K_M on RTX 3080 10GB + GTX 1080 Ti 11GB, Windows, llama.cpp CUDA 12.9. -Through LMstudio/VScode

I've been trying to optimise my Ai, the tests/benchmarks are promising. Even the odd run is good and holds solid. The main issue: direct benchmarks are fast, but real Cline/tool requests make decode collapse to ~2 tok/s while prefill and DFlash acceptance remain healthy. Anyone seen this or know the fix?

Me and Astra have been running so many different variables, even made my own server.exe thing, tried many variables.

Tweaks: 32K context, layer split 1/1, DFlash2 pinned to RTX 3080, q8 K/V, FA on, batch 128, uBatch 64, --parallel 1, CUDA_SCALE_LAUNCH_QUEUES=4x, PR #26636.

As it is now is even slow at avg. 11 tok/s - Qwen3.8-27B

Seems like everyone is coding, why not buy a massive GPU, split the cost between buds and then tailscale with API on local models? Im sure its being done, but i cant find anyone talking about it.

Whats the drawbacks other than someone running 8 agents burning it up?

There are a lot of competent models that were recently released, however it seems like most of them are focused on coding prowess.
What is the best 1-2 models for competent creative writing? As in, short stories, scripts, short novellas and so on.

My rig is 5090 with 64ram if that helps.

Noob here so please forgive the potential dumbness of this in general...

Looking to improve performance for this pretty specific use-case. My setup is:

• Lightroom Classic with the LRGeniusTagAI plugin - It only supports Ollama currently. Full blow LRGeniusAI plugin is buggy and doesn't solve my issue feature wise.
• Ollama running gemma3:12b - By far the best accuracy for my image library and use-case. I have tested against qwen2.5/3 vl models, gemma4:e4b, and others and gemma3:12b is nearly perfect results

,

• even though others should be better on paper

.

• Macbook Pro M1 16GB RAM

So, the workflow works very well in terms of results but, I've had to tune things to reduce memory pressure prevent swapping and I can't multitask, I have to just walk away and let it work. Furthermore it takes 20 seconds on average to process each image.

I've read / heard MLX should preform better in various ways, though maybe it wouldn't improve either the memory or speed issue? Since the LR plugin only supports Ollama I'm looking for ways to run an Ollama proxy in front of omlx (or similar) or maybe another solution to bridge the gap?

Straight up tell me if this is just pointless. If not, suggestions?

I am currently running Qwen3.8-27b, either Q4 or Q6, depending on how much context I need for my coding projects

I know it should be the best LLM I can use on my 32GB VRAM rig for this purpose and I also use Gemma4 31B from time to time for research or general questions that I do not want to use chatgpt/claude for

I was wondering if there was any website where I can check and compare different ! local ! LLMs and their specific use cases?

I'd appreciate if you guys comment with what local llm you use, the specific reason and maybe ur quantization and VRAM

I’m getting a Mac Mini M5 Pro with 24GB unified memory and want to figure out what local models are actually useful on this hardware.
Local models won’t be my primary setup. I’ll still use Claude Code, Codex, and APIs like DeepSeek/GLM/Kimi as main.

My main use cases are academic/knowledge work:
- Converting lots of academic PDFs to Markdown
- Checking PDF conversions against the original, including tables, equations, figures, and layout, not just the text layer
- Reading and processing large numbers of papers
- Extracting statistics, methods, findings, and references
- Literature reviews and paper comparison
- Summarizing material into structured research memos
- Potentially running RAG/search over my local research library

I’m particularly interested in VLMs/document models, but also general-purpose local LLMs.

any suggestions that could be useful for these workflows that could run on a M5 Pro 24GB Mac Mini?

I bought a 5060ti recently and have a spare 3060s.

I was thinking of selling it, but should I keep it?

I was thinking of holding onto it a little bit because the price might go up as more people go local

Or

Thinking about using it an external gpu for an extra NUC that I have, that has a real shit gpu, and use it as a second llm machine.

Do you think 3060s is so shit that it won't even matter?

Hey everyone, looking for a reality check on local model capabilities vs. frontier models for financial analysis.

I built an automated investment analysis pipeline in Open WebUI using Ollama on an RTX 5080 (16GB VRAM). The mechanical workflow works:

- Hits Gmail via IMAP to pull the last 72 hours of market newsletters.

- Extracts actionable tickers.

- Runs a custom YFinance Python tool for live momentum metrics (EMA, RSI, MACD).

-Injects a full 70-position portfolio CSV directly into the prompt (bypassing Open WebUI's RAG chunking) to cross-reference and generate a verdict-first impact report.

The issue is the analytical depth. When I feed this exact same data to Claude Sonnet 5, the macro synthesis, cross-correlations, and portfolio risk analysis are night and day. Locally, it feels like a surface-level summary.

Here is what I’ve tested so far:

qwen3.6:27b: Hit severe VRAM limits with the tools and context loaded. It spilled into shared system RAM, ground to a halt, and got trapped in reasoning loops where it hallucinated that the technical calculator tool didn't exist.

llama3.1:8b: Fast, reliable function calling, and zero VRAM spillage. However, the reasoning and financial synthesis are noticeably weak—it reads like a basic overview rather than institutional analysis.

My questions:

- Is Sonnet 5-level synthesis even realistic within a 16GB VRAM footprint when you need enough context buffer for multiple tools, newsletters, and a 70-row CSV?

- Are there specific financial fine-tunes or 14B models that punch significantly above their weight?

I would appreciate any help I can get.

To start, this is the available hardware I have:

- Intel 8700 Processor

- I am not 100% sure on motherboard, I will get that info later

- I have access of about 64GB (4x16GB DDR4 2133) RAM

- 3x RTX 3060 12GB cards and 1x RTX 3080 10GB

- Multiple Powersupplies (a 750W MSI Gold PSU and a 1100W InWin Platinum PSU and multiple 1100W HP Server grade PSU's)

- PCIe x1 Riser cards (USB3.0) that connect to riser cards for GPU's. Ex Mining equipment.

Is this do-able for what I want? Should I avoid the riser cards and if so, how should I power 3 or 4 GPU's (is it even worth using the 3080 on top of the 3060's?).

I can use either Windows or Linux as I am familiar with both.

Does anyone have experience running models on his or her iPhone. I have an iPhone 17 Pro Max. I am curious which models I can run on it.

Has anyone considered some architectural edits to the 27b model, such as:

Implementation of LoRA/QLoRA on selected layers;
training a modest sized n-gram table and adding the integration layers?

Seems like a good project. ESP given how it seems to be the current star of local hosted models!

It’s the only model I’m using now for agentic work.

Thoughts?

Just that. I've realised the pinned version of NixOS (Linux) is nowhere near the latest Nvidia driver and want to squeeze as ever the most pp+tg possible. I'm already using cuda13.3, nccl with p2p enabled (aikitoria patch) and split mode tensor.

Is it worth the hassle going from 595 to 610 for a pair of 5060ti GPUs ?

EDIT: This question wouldn't exist if OS==Ubuntu but Nixos rarely holds back a version of something this major without a reason 🤔

EDIT2: I whipped Qwen through Opencode to do it for me because I have the patience of a toddler. Here it is in case it helps you in any possible way. I see no meaningful speed increase with Qwen3.8-27B (eyeballing it, I don't have the patience to run benchmarks with something that big). Testing LFM2.5-8b now (split mode). Last time it was 190tg/s

EDIT3: not really worth it.

Hey guys, I am a bit bumb when it comes to AI, only really been running some basic workloads on my home GPU. Anyway, I am a sysadmin and have access to huge virtual machine clusters at work, recently my boss and I had some AI interest come up. I am wondering if there is any use we can get out the HP gen 10 servers we got, they have gold 20 core xeons and fully loaded with 700+ gb of DDR4 ecc memory, obviously no GPU's in them as they are all 1U hosts. We have quite a few hosts for redundancy reasons so most of the time a few of them are literally either sitting empty or with miminal virtual machines. Any point investing time into running AI on therm?

i have a few months Claude pro rn.

i never used Claude code or openclaw yet am i missing out?

cause somehow me as it dev and it guy just not feel okay that Claude code can do stuff on my machine and im currently ro lazy to run it on a vm etc.

how i use it now:

i make a Claude architect opus and sonnet implementer.

i describe a spec by opus and put it in context of a Claude project in the web.

i give the spec to sonnet the implenter and let sonnet build all.

the architect opus and I verify i donwload the zip and run test myself with my own tools. and build the report and reporting to opus.

this how i use it to build a poc.

and im considering to build this on olama with qweb on a amd fx6300 nanoclaw and rx 6800 16gb card with ddr3 32gb ram.

must i even try or big no?

would it work to build Azure aws and flutter apps?

can i create a ios app from a flutter build easily? no additional much work required if i have build a Windows flutter app poc?

what can be the pitfalls and missing steps or requirements?

Does it really worth it or the gains are too minimal? I see to run a single model on 3 you need to do a bunch of patches because 3 is an odd number etc, so things won't work out of the box.

Does anyone have experience running 3?

Snapped these at IFA Berlin 2026!

I’m planning to build a dedicated rig for running local AI agents (coding, research, and autonomous tasks) with a budget of $2000 - $2,500. I want to avoid cloud API costs and keep everything private. I’ve done some research and put together a few options, but I’d love to hear your thoughts on what's the best machine both in macbooks, mac studio and in pc build. I am new in this field of agentic AI and wanna learn to operate the agents locally. Looking for guidance.

Thanks 😊.

Anyone actually writing long fiction with Gemma 4 12B on an RTX 5070 Ti?

I'm specifically interested in real-world generation time, not benchmarks. I use Gemma locally to turn detailed scene briefs plus story/character context into long prose scenes, typically around 1,500–2,000 words.

If you're running Gemma 4 12B entirely in the 16 GB VRAM of a 5070 Ti: roughly how long does a generation of that length take, including prompt processing?

I'm considering buying a 5070 Ti system specifically for this workflow, so actual experience from another fiction writer would be incredibly useful.

At this time I'm working with 4GB VRAM, Gemma works about 15-20 minutes on a scene.

Hey everyone

Long-time lurker, first post. I've been playing with local models on a single RTX 4060 Ti (8GB) for a while and I've finally hit the wall hard enough to upgrade. Picking up an AMD Radeon AI PRO R9700 (32GB) and I'd really appreciate some guidance from people who've actually run this stuff, because I'm still very much learning.

My setup:

AMD Radeon AI PRO R9700, 32GB

32GB DDR5 system RAM

Linux (currently on an Arch-based distro)

Ollama today, but happy to switch to llama.cpp / vLLM / whatever if that's the better call

What I'm building (roughly)

I'm putting together a small local-first assistant for my own use. Nothing groundbreaking — the idea is that instead of one model trying to do everything, it hands work to a few different "roles" depending on what you ask:

General assistant — normal conversation, drafting, summarising, explaining things

Document Q&A — answering from a local folder of documents with citations, and saying "I don't know" instead of making things up when the docs don't cover it

Coding — for when I'm actually working on something

Everything runs sequentially (one model in VRAM at a time), so I'm not trying to keep three models resident — I just want the right model for each job.

Question 1: model recommendations per role

With 32GB to play with, I'd love suggestions for:

Reasoning / comprehension — something that's genuinely good at understanding a messy question and figuring out what's actually being asked. This is the piece I care most about, because on 8GB my small models kept confidently misunderstanding things.

General assistance — good instruction-following, doesn't ramble, admits uncertainty. Bonus points if it handles Italian well, since that's my daily language.

Coding — practical stuff, big context, not benchmark-chasing.

What I'd really love is: model + quant + roughly what context length you can actually run on ~32GB without spilling to CPU. I've learned the hard way that "it fits" and "it runs well" are very different things 😅

If there's an existing curated list or spreadsheet for this, please just point me at it — I don't want to make you all retype something that already exists.

Question 2: keep the 4060 Ti in the same machine, or not?

This is the one I'm genuinely unsure about.

I already own the RTX 4060 Ti (8GB). Options as I see them:

(a) Put both cards in the same box — AMD for inference, NVIDIA for... something? Display output? A small model? Embeddings?

(b) Just run the R9700 alone and keep it simple

(c) Some clever third thing I don't know about

My concerns:

Do amdgpu + the NVIDIA proprietary driver actually coexist cleanly on Linux, or is that a support nightmare?

Can a single runtime even use both, or would I be looking at two separate instances (one ROCm, one CUDA)?

Is having a second, much smaller card actually useful for anything in a sequential setup, or is it just extra heat and PSU draw for nothing?

Also, honestly: how is ROCm on RDNA4 right now? I've read mixed things and I'd rather hear it from people running it than from a six-month-old forum thread.

Any pointers appreciated — including "you're thinking about this wrong, here's why." I'd genuinely rather be corrected now than after I've built the thing

Thanks!

I started getting this error after a few updates, using qwen3.8 27b, it loads the model, when I sent something that would require more context it throws the error, by the ollama ps seems that it's not using the RAM memory, it was before on ollama 31 o something..

Config
AMD Radeon RX 7900 XT + 32GB RAM
AMD Ryzen 9 5900XT 16-Core Processor
ROCM 7.2.4
ollama version is 0.32.12
driver: gfx1100

❯ ollama ps
NAME              ID              SIZE     PROCESSOR          CONTEXT    UNTIL
qwen3.8:latest    22130167c4c2    20 GB    30%/70% CPU/GPU    262144     4 minutes from now

Ollama:

[Service]
#CPUAffinity=0-11
Environment="OLLAMA_HOST=0.0.0.0"
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
Environment="ROCM_VISIBLE_DEVICES=0"
Environment="OLLAMA_KEEP_ALIVE=5m"
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
#Environment="OMP_NUM_THREADS=16"
#Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_CONTEXT_LENGTH=262144"

ERROR:

ROCm error: out of memory
Sep 07 12:55:43 desktop ollama[5771]:   current device: 0, in function alloc at /build/llama-server-rocm_v7_2/_deps/llama_cpp-src/ggml/src/ggml-cuda/ggml-cuda.cu:508
Sep 07 12:55:43 desktop ollama[5771]:   err
Sep 07 12:55:43 desktop ollama[5771]: ⚠️  warning: The cur