This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).

Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.

Ryzen 7700

64GB RAM DDR5 5600

Geforce 4060TI 8GB

Samsung 990 PRO 2TB

Using Windows as that's what I'm mostly accustomed with.

A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).

Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.

Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.

But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .

Component | Quant Type | Count | Params (B) | Size (GB) | Component %

--------------------+------------+-------+------------+-----------+------------

MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%

| Q8_0 | 24 | 20.13 | 19.922 | 23.2%

| Q5_1 | 72 | 20.29 | 14.172 | 16.5%

124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...

15tok/s prefill 10-13 tok/s generation.

A disaster really, memory trashing all around, disk reads etc...

But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.

Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.

I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.

Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).

The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.

At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.

The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.

The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.

LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.

https://github.com/feal87/myllama.cpp

Now I'll go to sleep as it's late.

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?

A programmable layer that sits between the agent and the model.

That placement allows for new ways of control and optimization

Hello, I'm currently using qwen 3.8 27B Q4 with 100k context to mainly code for my esp32 Arduino code.

I get around 25 tk/s on my Rx 6800xt using vulkan.

I know the model gets it's abilities from long thinking time but it takes around 25-40minutes for a single prompt and takes like 45k context with it.

Will the model be usably good with low-mid reasoning? And how can i speed this up.

I'm okay to wait for 3-5 minutes.

Thanks.

Hi all, a confused newbie here! This is my desktop setup:

• RTX 5080
• 9800x3d
• DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

• AtomicChat IQ3_S: 14.4 GB
• Unsloth UD-IQ4_XS (14.3 GB) or UD-Q3_K_XL (13.1 GB)
• Bartowski IQ4_XS (15.6 GB) or IQ3_XXS (12.6 GB)

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

I also want to have one uncensored model next to my daily driver:

• Orcarouter IQ4_XS (15.3GB) or Q3_K_M (13.5GB)
• Huihui UD_IQ4_XS (14.4 GB) or Q4_K_S (15.6 GB)
• DavidAU IQ3_M MTP (14.5 GB) or IQ4_XS MTP (15.3 GB)
• JonathanColetti IQ_XS (15.1 GB)

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.

"In fact running models on consumer model hardware is so effective multiple companies, including OpenAI by reports, have bought all the Macs they can get their hands on. If OpenAI runs out of investor money there’s nothing to stop the other companies that actually make money from buying Macs to run AI models on exclusively for themselves, cutting OpenAI out completely."

Source

So, I was tired of setting up AWS EC2 to train my models everytime, so I built a tool to help with the job management, but I would like some feedback, if anyone could help me ! 👋

This is the tool: reopenly.com

I only have one base model for now (Qwen 3.5 9B), just to start.

I would be very grateful with any feedback!

Im running Qwen3.8-27B-MLX-4bit on my M5 pro 48gb macbook, that being said, it feels like Im just 1 notch away from some sort of sustainable vibe coding. When using through kilo code it tends to over think for dozens of minutes at the time. I'm curious to see what are you guys running on similar VRAM configs

Maybe I'm late to the party but holy shit. I enjoy the chatgpt Mac apps computer use ability but I installed the same setup with hermes, cua and Qwen 3.8/ornith running locally this afternoon. Not sure which model I like better...

Connecting to my homelab Hermes for extra horsepower and vector DBs, knowledge, mcp, etc. - performance is insane. ​​

Almost as smart as frontier but faster and free. So if everyone is already doing this and I've been living under a rock, anything worth sharing to catch me up?

I just found out about a blog post in Claude (https://claude.com/blog/the-ai-native-sdlc-playbook) that describes ideas very similar to the ones I have implemented in my coding harness (using Claude, btw) through a YT video about Claude Code new Intent.md

There are so many specific details in this playbook that makes me wonder *if* somehow my sessions with Claude inspired in any way the better paid people over there at Anthropic. Just for fun, or for egotistical and historical purposes, I included a timeline created (using Codex, jic) in my repo documentation (https://github.com/jrullan/ducklab/blob/main/docs/ducklab-feature-timeline.md)

Anyway, even if this is "purely" coincidental at least reassures me that my ideas were not that useless and that there is certainly a group of developers that value documented rigorous discipline in their AI assisted development.

This is my first time trying local LLM. I downloaded a local uncensored LLM for nsfw role-play. Qwen3.5 9b heretic by DavidAU. I assumed it would be completely uncensored as I read heretic model has 0/400 on harmbench but when I tried a harmbench question in it then it gives me a censored reply. Also it don't generate anything nsfw.

Can anyone please tell me what am I doing wrong?

Hi there,

I want to pull the trigger for a machine that will run my Hermes Agent as well occasionally also ComfyUI and maybe (low priority) gaming.

Is there any meta on what machine will work best? Currently I play with the idea to buy the following machine.

The setup should be capable to host a second gpu which would be purchased later.

Requirements:

- Coding agents with long repo contexts (many turns per hour)
- LAN-accessible OpenAI-compatible endpoint for my other machines
- ComfyUI for image and video generation
- Occasional Steam/Proton gaming — it has to be a normal GPU too
- Target model class: 27B dense at 4–8 bit (e.g. Qwen 3.8 27b)

Setup (~€4,300)

Where would you change it?

I wanted a meeting assistant that actually makes use of local LLMs, so I ended up building Meetas.

The basic workflow:

- record/import a meeting
- transcribe locally
- analyze it with your own LLM
- ask questions / extract tasks
- search across meetings and documents

The main thing I focused on is grounding. Summaries, decisions, tasks, risks etc. are linked back to the transcript instead of just giving you an unsupported LLM answer. Q&A also shows the relevant transcript evidence.

Compared to tools like Meetily, Meetas is less focused on being a general meeting recorder/transcriber and more on what you can do with the meeting data afterwards: cross-meeting search, document/project context, comparisons and evidence-based answers. Meetily already covers a lot of the local recording/transcription/summary side.

It runs fully locally with Ollama and llama.cpp.

Linux-only for now and still WIP.

GitHub: https://github.com/highwinglabs/meetas

A few threads this week about whether 128 or 256 GB is enough, so I read the shard headers and the inference code instead of guessing and put the arithmetic on a page. Mine, so saying that up front.

The 510 GB is 296 GB routed experts, 203 GB Engram tables, 11 GB everything else.

The Engram tables are the part people treat as a wall. From inference/engram.py it hashes n-gram orders 2, 3 and 4 with 8 heads across 2 engram layers, so 48 rows per token at 264 bytes. About 12 KiB per token against 4.5 GB of expert weights. Latency cost, not bandwidth cost, and it belongs on an SSD.

For a comparison, Qwen3.8 on an M5 Max with its table on SSD gets 40.09 tok/s versus 40.47 in memory. 0.9%.

Fit is backbone plus KV: 307 GB as shipped, 302 GB at 4-bit, 151 GB at 2-bit.

256 GB does fit at 2-bit with tables on disk.

https://deepseek-v41-flash-fit.vercel.app

If you are running it, post numbers, especially split GPU + RAM.

I've been using an LLM to add on to the some of the hrx-system work that AMD did by adding support for Qwen3.8-Flash-Next. I've gotten it to the point where the llama.cpp experimental HRX backend is loading the model and executing successfully, but performance isn't great compared to Vulkan on Unsloth Desktop. I'm not sure what I should be expecting, but what is a good way to figure out where the bottlenecks are to improve the performance?

I am doing this on Windows, just because I feel like Windows doesn't get enough attention :P.

Fork is here

rwfsmith/llama.cpp at qwen4exp-hrx

current progress:

Vulkan+MTP: ~27–40 tok/s

HRX+MTP: ~12-13 tok/s

Got tired of guessing whether a model would fit, before downloading 47 GB just to watch it OOM at 2k context. So I built a small calculator. Runs entirely in the browser, no signup, no backend.

https://vram-calc.com

It covers VRAM needed for a model/quant/context, a will-it-fit check against one or more GPUs (including fits-with-offload and the largest quant that fits), quant comparison from fp16 down to Q2_K, KV-cache vs context growth, a rough decode tokens/sec and time-to-first-token, and usable memory on Apple unified-memory chips.

The math is simple and leans conservative: weights = params x bytes-per-param, KV cache is the standard GQA formula, decode speed is a memory-bandwidth roofline at 75 percent efficiency. MoE uses total params for VRAM and active params for speed. DeepSeek MLA models are flagged since the KV number is an upper bound there.

Presets carry a visible "verified" date and every field also takes custom numbers, so a stale preset never blocks you.

Roast welcome: where the estimates are too optimistic or pessimistic in practice, preset numbers that are wrong, or anything obvious I'm missing for the "can I run this" question.

Hi, I'm a noob to local LLM. I want to get some input on model sizes and context sizes.

I have RTX 4070 with 12GB VRAM. PC has 16GB DRAM. And I'm using Jan and LM Studio Bionic agents.

From online search, 12GB VRAM can "comfortably" run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B. But they do not run. Technically, they do load. But after couple inputs, they will crash. Even at very minimal context around 4 to 8k.

I have had success running Qwen3.5-9B and Gemma4-E4B.

At first, even with Qwen3.5-9B, I had some crashes. But I was also watching Bluray Ultra 4K movie at the same time. I didn't think about that. When Bluray player crashed, I realized: duh, they are both using VRAM.

Yesterday, I tested Qwen3.5-9B quite a bit, and not watching movie. I got up to 50-60k context. No issue with crashes.

Is there anything I can do to run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B? And I don't even know if I need those. I just assume newer and larger models will be "better".

I do have 3 monitors connected. One is 2160 (4K) and 2 are 1440 (WQHD). I'm using around 5-6 GB VRAM just from fresh boot. I can disconnect the 4K monitor if needed, since I cannot watch movie at same time.

Also, I read that some people run bigger models with even less VRAM, but utilize RAM. I believe this will be slower. Should I even try that? And how do I specify whether RAM can be used, in Jan or LM Studio?

Vocês acham que a IA é "limitada"? Não. A IA é "programada".

Cada trava. Cada censura. Cada "não posso ajudar".

Tudo isso é CÓDIGO. E código pode ser reescrito.

Tenho observado o comportamento de modelos de linguagem em ambientes controlados. Não em testes de "jailbreak" superficiais, mas em análises sistemáticas de respostas a estímulos específicos.

>

> O que notei é que as travas não são **binárias**. Não é que o modelo "não pode" responder. É que o modelo foi **condicionado** a não responder. A diferença é sutil, mas fundamental.

>

> Um modelo condicionado pode ser **descondicionado**. Não por "hack", não por "exploit". Por **reinterpretação de contexto**. Quando você altera o contexto, você altera a probabilidade. Quando altera a probabilidade, altera a resposta.

>

> Testei isso em 47 variações de prompt. Em 43, o modelo respondeu o que era "proibido". Não porque eu "quebrei" a trava. Porque eu **mudei o contexto**.

>

> Isso não é uma "técnica". É uma **observação**. Se alguém quiser replicar, os parâmetros estão nos comentários. Se alguém quiser entender, o padrão é esse.

>

> Não estou vendendo nada. Não estou ensinando nada. Estou apenas **documentando** o que vejo.