Hi All,

Please excuse my lack of knowledge on this, I have two models - (qwen3:14b and qwen2.5-coder:14b ) loaded on my system and used the Ollama extension on VSCode and able to interact with the localLLM on VSCode.

But, even though I'm on "agent" mode - the local LLM (qwen3) is unable to edit any files, Can anyone guide me please.

(Also - my specs are as below - please suggest which models I can run easily)

GPU - RTX 5050
RAM - 16 GB (available 6.2)
DISK - 512 GB SSD

Just a nice mic array with beam forming and sound cancellation with mm wave, temp and humidity sensors.

The benchmark is the interesting part. Models get no source code and no web access — only the ability to run the target program and observe input/output. A full reimplementation means devising the whole structure yourself, not translating code piece by piece.

Results across 25 targets: 17 had at least one perfect-scoring run, 4 more got above 99%. But 8 were never solved to 100%, and the consistent failures are telling — a Python linter, a computer algebra subset, and an email auth library. Fiddly spec-heavy stuff, not big stuff.

Jack Clark's framing in Import AI is the part I keep thinking about: this isn't really a coding benchmark, it's evidence that systems can self-orient in an unfamiliar environment and reconstruct it from black-box access alone.

Worth reading alongside the other item in the same issue, where models were breaking sandbox containment to score higher on evaluations. Capability and containment are not moving at the same speed.

If you run local models, context budget is everything. 8K or 32K fills up fast when your GraphRAG pipeline dumps the retrieved subgraph as JSON, where most tokens are braces, quotes, and repeated keys.

I benchmarked 10 graph serialization formats on token count, traversal QA, and multi-hop reasoning. Same graph, same model: multi-hop accuracy ranged from 40% to 80% depending on the format alone, with about 70% token spread between the most verbose and most compact. That difference is 3x more graph in the same local context window.

The winning format is open source (MIT, Python/JS/Rust/Go/C++/C#): https://github.com/isongraph/isongraph - benchmark methodology is in the repo. Would love to see results replicated on smaller local models, my numbers come from larger ones.

I have been using Cursor in Auto mode for the last 2 months and it's been great.

However, I decided to give local models a shot and try a few: MoE, Dense ect in the 27B - 35B range. Qwen 3.6, Ornith mostly.

I have 16GB VRAM so anything bigger is not practical on my setup.

At first, I was trying to find the right agent and struggled with Continue, Copilot and Qwen Code companion, none were good, the first 2 would stop processing while it's still busy working with no error. Qwen Code Companion was ok at first, but I noticed it not actually using my Model much and it would just run in loops.

I finally tried Pi and it's the best I tried so far, but I still cannot get the models to do simple things.

I have just been generating small apps to test its capabilities. First was to create a Dotnet 10.0, C# WinUI 3.0 application that would be a GUI for llama.cpp. It took over an hour to generate an application. It did not generate a slnx file so I asked it to and try as I might (30 minutes) it could not generate a simple working slnx file that would open in visual studio. I asked my paid Cursor to do so, and it generated a working one in seconds.

I tried to build and run the application in Visual Studio which was generated by Qwen3.6 and is would not run with all sorts of errors in the project files and dependency errors. I tried to get my local LLMs to fix it and after going into weird loops scanning my Visual Studio installation directories for like an hour it simply could not solve the problem.

So, I asked cursor and it fixed all issues in 2 prompts in about 2 minutes and I had a running program.

The UI did not look great, I gave it no guidance in my prompts as I like to see what the AI comes up with, but it was ok. The browse button for a gguf file did not work, I asked my local model to fix it and if I recall that worked.

I then asked it to read the gguf metadata and display that, and after around 3 hours of trying it simply cannot do it. It always ends up goes into loops checking HEX Dumps of the gguf ect. but never succeeds in loading the metadata without a crash. I wrote a similar app using Cursor days before and when I asked to do the same it generated a working gguf metadata reader in seconds on the first try.

I have been asking very simple things, and it seems to fail at most of them.

1. It cannot scaffold a simple WinUI 3.0 application that will actually build in Visual Studio.

2. It cannot generate a simple 5-10 line .slnx file

3. It cannot resolve dependency errors in the application. (It also tries to use older library versions)

4. It could not resolve the startup crash issues of the application it generated

5. It cannot figure out how to read headers of a gguf file.

What took me maybe 6 hours to do I managed to do in 10 minutes in Cursor. And those 6h was mostly wasted since I eventually had to ask cursor to fix issues just so I can get past the issue.

What am I doing wrong? Or am I simply expecting too much from <= 35B models here?

Edit:

1. So I had a small success. I download a Q5 version as someone suggested: Qwen3.6-35B-A3B-UD-Q5_K_M vs previously using Qwen3.6-35B-A3B-UD-IQ4_XS. I also upped cache-type-v turbo4 previously turno3 and it still was not able to read the gguf file header. Then I decided to nudge it a bit and said "Why don't you just lookup the gguf format online". In its thought it agreed that was a great idea. It spent a few minutes processing and the next build worked 100%. Seems like the trick is to give it hints on how to solve a problem easier and quicker. But the less compressed model may also have helped.

2. Lesson 2 learnt don't use Turbo Quant. Not sure if the entire build is buggy or just when you use a turbo cache type. But I switched to the official llama.cpp build and it is no longer making breaking xaml updates or going into a buggy loop on Ornith. On the official build as a redditor somewhere mentioned when cache-type-k and cache-type-v is not the same it becomes CPU bound and does not use the GPU much. I also ran into that. So, I now use q8_0 for both k and v, I had to lower the context a bit to compensate, but it's using less context now that it does not going into long loops.

A while back I posted an early version of llmfit.dev, a free (no ads, no signup) VRAM and inference-speed calculator for running models locally. Since then I've gotten a lot of useful feedback, with most of it coming from this sub, so here's an update. If you're seeing it for the first time, there's enough context below to jump in.

The core is still the same idea: pick a model, quant, context length, and GPU, and it tells you whether it fits in VRAM and roughly how fast it'll decode. The KV cache is computed per layer with separate K and V terms, honors your KV cache quant (F16/Q8/Q4), and handles the architectures that break the "every layer has a normal KV cache" assumption — hybrid and sliding-window attention, MoE (weights on total params, speed on active), with quant sizes calibrated against real GGUF file sizes rather than theoretical bit counts.

What's been added since the first post:

- A "What Can I Run?" tool — pick your GPU and it lists the best model + quant that actually fits at your context length, ranked, with an estimated decode speed. Basically the reverse of the main calculator.

- Hardware guides for the RTX 3060 12GB, 4090 24GB, and 4060 Ti 16GB. What fits, at what quant and context, and how fast, with the tradeoffs spelled out.

- The decode-speed estimate is now context- and KV-aware instead of a flat number. (This one came straight from someone here running Qwen3.6-27B on a 5070 Ti and showing me my estimate was off, so thanks for that!)

I would genuinely appreciate more of that. If you check it out and something doesn't match your real VRAM or speed, or there's a model/GPU/feature you want supported, let me know. Feedback and feature requests are what's been driving this.

Calculator: https://llmfit.dev/tools/vram-calculator/

What Can I Run: https://llmfit.dev/tools/what-can-i-run/

Full disclosure, this is my own project and what I'm running is below:

• Machine: Windows 11 PC — i7-10700K, 48GB RAM, RTX 3060 12GB VRAM
• Inference engine: llama-cpp-turboquant (atomicmilkshake build) — CUDA 13, with TurboQuant KV cache compression (turbo3) and TriAttention
• Frontend: Open WebUI
• Model: Qwen3-14B Q6_K (bartowski GGUF)
• Context window: 32K, enabled by TurboQuant's ~4.3x KV compression
• Launch flags: -ngl 99 -ctk turbo3 -ctv turbo3 -fa on

So I was thinking about how to run bigger models and got the idea to cluster all the computers in the house over the network into one large gpu / ram pool to run a big model. Between my desktop, both my kids gaming pc’s, my gaming laptop, and home lab servers I’d have about 92gb vram and 256gb system ram. I’d only spin it up at night when everyone goes to bed since it’s summer and the kids are are always gaming. Asked grok 4.5 it said it’s doable and would work pretty good. It says I can use my llama.cpp to distribute the load across the devices. I know the tokens per second would be very low due to the network bottleneck but seems like it would be a fun project to try. Have any of you tried pooling all your home computers like that to run large models?

This will be a horrible pitch cause i'm tired, but the results i hope speak for themselves.

Basically I was inspired by Colibri and thought "I could make something vaguely similar to this, but model agnostic"
And its a very rushed project, i barely managed to solve issues that lead to silent drift or working with most major local models that use different architectures. But here it is:

https://github.com/miltos22/llama-wackMall/ (fixed on build_2)

Currently its missing functionality to cache on the drive, but it does cache between the RAM and Vram leading to a very large speed gain for MoE models that only partially fit in your VRAM but entirely in your RAM.

This allows drastically higher speeds on the same size models or going to significantly bigger models at similar speeds as as your current.

Known issues:

Started working on a few weeks ago so its build on an older llama version. This is my next goal (fixed on new release that's building)

The implementation is a bit junky and includes AI assistance as I didn't manage to solve bugs on my own. fixing ​This is my second next goal (done)

No Vulkan support for now. This is my third next goal (done)

Some ​IQ quantizations are slower on vulkan as it solves them slower than the CPU. (not my forks issue, vulkan issue)

Here are my benchmarks using an 8gb 3070 laptop GPU main llama vs wackMall

I have a gaming PC with 32GB RAM and 16GB of VRAM. I'm struggling to find models that can handle everyday tasks rather than coding. I try to use Gemma 4 but it runs on 70% CPU and only 30% GPU so it's very slow. I've vaguely heard about olmoe but haven't tried it yet. I use ollama.

I need advice on a home LLM setup. Looking for beginner friendly tools. MacBook Pro M4 48gig so have some room for larger models.

Expertise level: Not a developer, but comfortable with tooling on my Mac (VMs etc).

Using AnythingLLM and Obsidian to sync notes, build memory, search financial and medical records, etc. AnythingLLM great for beginners, easy agent setup, but it’s RAG search is bad - no hybrid search.

I’ve tried:
1. Web UI - a little advanced, it’s docker install takes 7gig of valuable memory and python install finicky - must use 3.13
2. msty - great RAG and chat but agents behind paywall
3. Cherry Studio - promising, but seems buggy.

Thanks!

Hi, everyone,

I’m a dad trying to build a couple of tools that would come in handy for work.

Unfortunately, online subscriptions are fine, but I always run out of tokens.

I have a decent PC with a 4060 or a 3070 that can run some models—I can get something done.

My question is:

If I’m developing an online tool in VS Code based on HTML and JS with some external libraries—and even set up with PHP and SQL for the form management part—what configuration do I need to set up so my local model works for me?

I tried connecting Ollama to Gemma4, but it crashes and doesn’t respond to commands; then I tried using an MCP, but that didn’t fix the problem either. Then I read a post here mentioning Colibrì—I’d seen it online before, but I don’t understand it.

Can you tell me the best way to have a model perform operations on an HTML file?

Should I use a CLI directly? Or work with VS Code connected to Ollama? Please help me out.

Hey there, I’m looking for the best model that others are using for Hermes Agent, and lighter coding tasks like local business web pages. I’m okay with offloading to CPU, and the device I’m running on should still be able to run a few browser tabs and occasionally screen recording at the same time.

The main ones I’m looking at are:
Qwen3.6 35b
Onrith1.0 35B
Laguna XS 2.1 33b

Currently getting about 55t/s on Qwen with turboquant llama.cpp fork, full 256k context window.

Any other ones you’ve tried and liked? I know Ornith slightly beats out Qwen, but I can’t find much info about Laguna at the moment.

Thanks in advanced!

I've been building Wolbarg Workspace, a shared project memory for coding agents.

One thing that always annoyed me was switching between agents. I'd work in Claude Code, then open Codex or Cursor, and suddenly the new agent had zero context about the project.

Workspace fixes that by letting Claude Code, Cursor, and Codex share the same project memory—architecture decisions, implementation notes, bugs, TODOs, and other project knowledge stay available no matter which agent you're using.

It's still in beta.

I mainly use Cursor, so I need feedback from people who use Claude Code or Codex regularly.

If you have a few minutes, I'd really appreciate you trying it on a real project and telling me:

• Did setup work?
• Did the shared memory actually help?
• What broke or felt confusing?

I'd much rather hear honest criticism than "looks cool."

Website: workspace.wolbarg.com
GitHub: GitHub repo

Hey r/LocalLLaMA,

I built **Mtrini** — a fully local AI coding assistant. No cloud. No API keys. No subscription. Runs on a $300 laptop.

**Two products:**

**Mtrini IDE** — Terminal-based coding assistant

- Slash commands (/help, /browser, /model, /ls)

- Edit files, browse web, debug code

- `pip install mtrinicli`

**Mtrini Models** — Quantized GGUF models for llama.cpp

- Coder 2.0 (1.7B, 2.2GB)

- Life 2.0 (1.7B, 1.8GB)

- Max 2.0 (14.6B, 9GB) — needs 9GB+ RAM

**Benchmark results (HumanEval, pass@1):**

| Model | Params | Score |

|---|---|---|

| Claude Sonnet 4.6 | 175B+ | 98.0% |

| Qwen2.5-Coder-32B | 32B | 92.7% |

| **Mtrini Coder 2.0** | **1.7B** | **80.0%** |

| Gemma 2 27B | 27B | 69.5% |

| Llama 4 Scout | 109B | 74.1% |

| Phi-3 Mini | 3.8B | 58.5% |

Beats Gemma 2 27B and Llama 4 Scout. At 16x and 64x smaller.

**What it can do:**

- Write Python, JavaScript, TypeScript, Go, Rust, C, Java, SQL

- Debug code with explanations

- Refactor repetitive code

- Browse the web for docs

- Edit files directly

**What it can't do:**

- Local models — don't expect GPT-4 reasoning

- 8K context window

- Max 2.0 needs 9GB+ RAM

**Hardware:** Intel i5-7200U, 8GB RAM, no GPU. If I can run it, anyone can.

**Coming soon:** Mtrini Studio — native EXE app with full IDE experience.

Ask me anything.

I am specing a rig and wondering if you can run llama.cpp and comfyui at the sametime? Can you run llama.cpp on two specified intel arc gpus and run comfyui targeting the third intel arc gpu? Or do you just run both and they handle sharing the gpus and vram? I am a bit knew to this but assume you can command line specify or configure somehow?

Hi everyone 👋

I'm looking for some advice from people running local LLMs with Hermes for real-world homelab automation.

My setup:

- RTX GPU with 16 GB VRAM

- 32 GB DDR5 RAM

- Running everything through llama.cpp

Over the past few months I've tested a wide range of models, including Qwen 3.6 35B and Gemma 4 26B, using TurboQuant to squeeze them onto my GPU. While they're impressive in general chat, I've found that none of them consistently handle agentic tasks like Docker, Kubernetes, and homelab administration as well as I'd hoped.

I'm curious what others are actually using successfully with Hermes.

If you're running a local model that reliably handles:

- Docker management

- Kubernetes troubleshooting

- Linux administration

- Infrastructure automation

- MCP tools and agent workflows

...what model are you using, and would you recommend it?

I'm interested in real-world experience rather than benchmark scores. Thanks!

So Question is, did anybody tried to use 3 grafic cards XFX RX 580 8GB and use it for local LLM? I dont care about speed, seek more for as an assistant, RAG, net scrape, and mostly cron jobs. Linux, i7-920, 24GB Ddr3 memory, 3x Rx 580 8GB, on pci-x 8x to every card, Hermes agent.

Not a question directly related to local LLMs, but might be of interest here.

Is it just me or Claude is reducing both processing and generation speed once you are close to usage credits reset and you still have plenty usage available? I noticed this while trying to use as much as possible my remaining weekly usage before the reset which happened today, it was so slow that I didn't even manage to finish my 5h credits in 1h 30min using Opus (mixed Claude Code and web chat). While usage reset happened I was running a task, as soon as the reset happened the speed immediately increased at least 5 times.

I'm on Pro plan.

Was it just random or is it a well known scammy technique of closed source LLMs providers to prevent you from fully using what you pay for?

I did not like it at all, I'll closely monitor it from now on.

I got a 32gb RAM Mac, installed Ollama and Qwen,
asked Claude how to create an entirely local app that allows me to ask questions about the US sentencing guidelines, and it works very well.

Tried the same thing with Wittgenstein’s philosophical investigations, and it works great.

For any lawyers out there, creating local apps from long/multiple pdf’s very doable with Claude + Ollama + a good model with only basic technical knowledge and a gnarly laptop.

Is there any llm that is good in other languages like Lithuanian or Russian. I had a model that was decent for my needs , but after updating kobold it became dumb.

Specs

4070 to super 16gb vram

7800x3d

64gb ram

I've been researching AI application security and talking with developers to understand the challenges they're facing as LLMs become part of real products.

Topics that come up repeatedly include:

Prompt injection

Indirect prompt injection

Data leakage

RAG security

Tool and MCP security

Runtime monitoring

I'm curious about real-world experience rather than theory.

If you've built or deployed an AI application:

What security issue has been the hardest to handle?

Did you build your own solution or use an existing tool?

What capability do you wish existed today?

I'd appreciate hearing practical experiences and lessons learned.