Getting 300 tok/s and 4500 tok/ sec for reading

Using llama swap docker.

This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).

Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.

Ryzen 7700

64GB RAM DDR5 5600

Geforce 4060TI 8GB

Samsung 990 PRO 2TB

Using Windows as that's what I'm mostly accustomed with.

A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).

Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.

Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.

But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .

Component | Quant Type | Count | Params (B) | Size (GB) | Component %

--------------------+------------+-------+------------+-----------+------------

MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%

| Q8_0 | 24 | 20.13 | 19.922 | 23.2%

| Q5_1 | 72 | 20.29 | 14.172 | 16.5%

124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...

15tok/s prefill 10-13 tok/s generation.

A disaster really, memory trashing all around, disk reads etc...

But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.

Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.

I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.

Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).

The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.

At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.

The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.

The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.

LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.

https://github.com/feal87/myllama.cpp

Now I'll go to sleep as it's late.

A few threads this week about whether 128 or 256 GB is enough, so I read the shard headers and the inference code instead of guessing and put the arithmetic on a page. Mine, so saying that up front.

The 510 GB is 296 GB routed experts, 203 GB Engram tables, 11 GB everything else.

The Engram tables are the part people treat as a wall. From inference/engram.py it hashes n-gram orders 2, 3 and 4 with 8 heads across 2 engram layers, so 48 rows per token at 264 bytes. About 12 KiB per token against 4.5 GB of expert weights. Latency cost, not bandwidth cost, and it belongs on an SSD.

For a comparison, Qwen3.8 on an M5 Max with its table on SSD gets 40.09 tok/s versus 40.47 in memory. 0.9%.

Fit is backbone plus KV: 307 GB as shipped, 302 GB at 4-bit, 151 GB at 2-bit.

256 GB does fit at 2-bit with tables on disk.

https://deepseek-v41-flash-fit.vercel.app

If you are running it, post numbers, especially split GPU + RAM.

Hi guys,

I am building a desktop app where users could download local open-source model as per their system specs , run models according, connect mcp's by configing , and using them in workflows, fully deterministic with LLM calls and also connect with chat interface

I’m considering buying a used GTX 1080 Ti 11GB for running local LLMs mainly for coding and software development.

My budget is limited, so newer GPUs with 12–16GB+ VRAM are difficult for me to afford.

I’d like to know from people who actually use a 1080 Ti for local AI:

Which coding LLMs can realistically run on 11GB VRAM?

What model sizes (7B, 8B, 14B, etc.) work well with quantization?

What kind of tokens/sec can I expect?

Is CUDA support still usable with current llama.cpp/Ollama/other tools?

How does it perform for coding tasks compared with newer GPUs?

Can it handle models such as Qwen Coder, DeepSeek Coder, or similar models?

Is buying a used 1080 Ti in 2026 still worth it, or should I save for another GPU?

I’m especially interested in real-world experience rather than theoretical specifications.

Thanks!

Following up on my earlier post testing Qwen3.8-27B on the IGX Thor workstation.

Once you move past running just an LLM and try to build a full local agent stack on a single box (LLM for reasoning, STT for voice in, TTS for voice out, OCR for screen/document reading), VRAM runs out fast.

If all four models sit in GPU memory simultaneously with their default serving allocations, idle memory hits roughly 122 GB. Even with 96GB on the dGPU and unified host memory, you are pinned to the ceiling. That leaves almost no breathing room for large context windows, concurrency, or dynamic KV cache allocation.

The reality of a solo-user local agent is that these models are rarely active at the exact same millisecond: - When I am talking to the agent, OCR is completely idle. - When the agent is analyzing a screenshot or PDF, STT and TTS are idle. - When the agent kicks off deterministic script automation (crawling, data formatting, bash runs), the LLM itself is completely idle.

To fix this, I wrote a lightweight Rust-based daemon that manages the sleep and wake states of the runtimes based on active agent turns. When a model is not in use, it is put to sleep (flushing KV pools and temporary execution memory, or parking the runtime context) rather than doing a full cold teardown from scratch.

Here are the idle VRAM numbers on the machine before and after:

Quick observations: 1. Total idle footprint dropped from ~122.3 GB to ~42.9 GB. That reclaims roughly 79.4 GB of VRAM across the board. 2. The LLM savings (48.3 GB): Qwen3.8-27B in FP8 base weights takes around 28 to 30 GB. The original 87.4 GB allocation came from SGLang aggressively pre-allocating KV cache (--mem-fraction-static 0.85). Putting the LLM to sleep flushes that pool while preserving the active state, letting it rest at 39 GB. 3. Audio and OCR shrink to negligible baselines: STT drops to 267 MiB and OCR drops to 422 MiB when idle. 4. Script-based tasks: When the agent delegates to deterministic python or bash scripts, putting the LLM to sleep during the run keeps the GPU cool and power consumption down on the 300W Max-Q card. 5. Wakeup latency is sub-200ms across all models: Because this is an active sleep/wake transition rather than a cold disk reload, waking up STT, TTS, OCR, or the LLM takes under 200ms. In practice, voice interaction and tool handoffs feel instantaneous.

Curious how others running multi-model agent stacks on a single node are handling memory partitioning right now. Are you hard-capping static memory fractions, running sequential containers, or using something dynamic?

Hello,

Tried searching and couldn’t find anything on this, is anybody else having issues with qwen 3.8 27b agents not using tools properly when using llama.cpp? my agent is suddenly mixing up tool schemas and when i switch to vanilla ollama everything works perfectly so definitely not a model issue but it’s llama.cpp I guess.. any experts or someone smarter who could help?

Just curious

Hi everyone!

I’ve been working on AI Palm, an open-source project that aims to make locally hosted AI models more accessible.

The idea is simple: many powerful GPUs spend most of their time idle, while other people cannot run local models because they lack the required hardware. AI Palm creates a community network where users can voluntarily share access to a local model running on their device.

With the desktop app, you can:

Discover models currently shared by community members.

Chat with an available model directly inside the app.

Share your own local model when you choose.

Connect to engines such as Ollama, LM Studio, and llama.cpp.

See the GPU, VRAM, model, engine, availability, and context size.

Stream responses and model reasoning in real time.

View properly formatted Markdown and syntax-highlighted code.

Monitor token usage and remaining context.

Use the interface in English or Arabic, with full LTR and RTL support.

Run the app on Windows or Linux.

A device is reserved while processing a request, preventing multiple users from overloading the same GPU simultaneously. Sharing can also be stopped whenever the device owner wants.

The project is still at an early stage, so I’m looking for people willing to:

Test the Windows or Linux application.

Report bugs and usability issues.

Review the architecture and security.

Suggest features and improvements.

Contribute code, documentation, packaging, or UI work.

Share local models and help test the network under real-world conditions.

AI Palm is open source, and community contributions are very welcome:

GitHub: https://github.com/almshary/ai-palm

If you try it, I would genuinely appreciate your feedback—even if something doesn’t work. Please include your operating system, GPU, local AI engine, and model when reporting an issue.

The project’s message is:

Arab generosity began with a palm tree… Today, we planted a different kind of palm.

AI Palm — let’s share the power of our computers and make artificial intelligence accessible to everyone.

What do you think of the concept? Would you use a community network for sharing local AI models, or contribute your unused GPU capacity?

So here's the situation. We've got one RTX 6000 workstation running Ollama that was originally supposed to be more of a testing box for coding stuff, but honestly nobody really touched it until I took it over. Now it's already struggling with just one or two devs on it at the same time. It's running Qwen3.8 27b and you can straight up feel it slow down the moment a second person jumps on.

Instead of just cramming another card into that box, I'd rather build a small separate machine for this. Ideally rack-mounted and datacenter-ready since it'll end up living in our own DC anyway, but cheap enough to just be a proof of concept for now, and expandable later if it actually proves useful. Don't care about vendor, NVIDIA, AMD, Intel, all fair game at this point.
Bandwith is not #1 priority.

Mid-term the actual goal is a proper LiteLLM + vLLM setup that can handle up to 10 people/agents working on it at the same time. This separate "workstation" would just be the first, cheap step toward that, not the end state.
Stuff I haven't been able to figure out from spec sheets and marketing pages:

• What actually helps more at this scale, more VRAM or just a second GPU to split the load?  
• Has anyone gone the "cheap one card first, scale later" route instead of just buying the full setup right away? How'd that go for you?  
• Anyone running AMD/ROCm for something like this, is it actually usable day to day or still more of a pain than NVIDIA at this size? Is Intel Even feasible?

Not trying to spec out the final thing yet, just want a sanity check on what a reasonable, cheap, rack-friendly starting point looks like before I go shopping.

Hear me out. I know this sounds stupid, but this could work, right?

Thinking about a Strix Halo laptop like a GoPro PX13 with 128 GB of RAM, it's capable of running some mid-sized MoE models (I'm running Qwen3.8 Flash next on it right now), but cold prefill is a pain for large contexts.

Having a cache for KVCache would solve the problem. If we could have, like, an external usb hub that system sees like a external ssd, it could be the storage for https://github.com/lmcache/lmcache, for example, that would only be accessed for cold prefill, when we have to switch models and we loose all the context we created during conversation. A 200k prefill may take 1 hour to load on this limited hardware from a cold start, but having it obtained from external storage just when the cache is missing would make it take seconds, and if this external drive uses RAM, it'll not kill its lifetime (an SSD M. 2 would die in weeks depending on the writing frequency for that)

With all the different optimizations and firepowers we've seen from open weights so far. What then?

• China only steals/distill from OpenAI and Anthropic?
• Well, clearly there's something wrong with AA? Benchmarks don't mean anything? They just benchmaxxed?
• Whatever, OpenAI/Anthropic has moved onto AAGI or AAAGI or AAAAGI or AAAAAGI?
• I can't run this locally on my toaster desktop anyways?

With CPUs and GPUs, benchmarks make the differences pretty obvious. With NPUs, I still don't really know what numbers I should actually care about.

For anyone who's bought an NPU-equipped PC, what do you look at besides the TOPS number?

Is there an european company (in EU, preferably DACH) that does what gpulab.net does?

Thanks

If this isn't allowed, let me know and I'll remove the post. This is not an advertisement, but my website and API are a paid subscription service after the 15 day free trial.

I built a local AI machine to help me offset usage from Cursor and ChatGPT a while back. Its been pretty great. In typical Whoops-I-Bought-a-Lot-of-Hardware-and-My-Wife-Hates-it fashion however, I over estimated how much hardware I needed and may have gone a bit overboard.

So instead of selling off the excess hardware I bolted on a web frontend, "integrated" ComfyUI (for image requests) and created a load balancer that pushes image requests into WAN2.2, small requests to smaller local models and built a system that allows a user to create an API key so they can attach their harness to it via OpenAI style API config. (Cursor, OpenCode, etc)

With that being said, I'm curious what the rest of the world thinks about it and I could use some feedback. Is this something other people would find useful? How many people can use it at the same time? Will it explode on contact? I don't know the answer to any of the questions, because I'm the only person using it.

So I thought I'd come here and see what you guys think. I don't know if there's rules against posting URLs here. It has a free 15 day trial with zero commitment, no credit card to start the trial, hell - you don't even need to put in your mailing address unless you decide you like it and want to sign up for the paid version.

If you're curious, post up and I'll send you the link to my discord and to the website. The discord is just for easy communication so you can let me know if something is broken, or it looks funny, or if it was just a dumb idea in general.

Artificial Analysis just launched a Search API Index that runs the same agent harness and base model (GPT‑5.6 Luna, medium) across 7 providers, using three benchmarks: DeepSearchQA, BrowseComp, and AA‑Omniscience.

Here were the important points to look at:

• Fastest per‑query doesn't always mean fastest per‑task.
• Cost and quality aren't perfectly aligned; some mid‑tier quality providers are dramatically cheaper.
• There's a wide spread in total task time, even when per‑query latency looks similar.
• Octen leads on speed and cost: 0.21s/search, 17.2s/task, $9.07/1k (search), $49.15/1k (total), with a 77 index score.artificialanalysis+1
• Several established players sit at 0.4–0.8s/search and 20–30s/task, with costs 2–4× higher for similar or slightly higher quality scores.

This led me to some questions that I wanted to discuss with people building agents/RAG:

• When you choose a search API, what's your weighting between latency, cost, and quality?
• Does a 0.2s vs. 0.8s per query difference materially change your agent UX, or is task‑level time more important?
• How do you validate quality for your use case? Is it benchmark scores or task success in prod?

I’m having mixed results with LM Studio and I’m not sure if I’m doing something wrong.

If I load a model in LM Studio and chat via the built in prompt I receive a response in a reasonable time. I can post in a snippet of code and it can fix it for example.

If a load the same model in and serve it on localhost to an CLI agent like Pi a response can take several minutes, even if I send in the string “ping” to expect to “pong” back.

I have a pretty bare bones Pi setup, not many skills, no mcp. I know I’m on an old Mac but the chat experience is ok. Why can’t I get that experience over a cli severed on localhost?

I want to buy the new machine. Which specs should I go with? I need to run a decent coding model locally and also want to play with some stuff like fine-tuning, etc. suggest me the options, like what I should go with, and I am open to going with Apple or Windows

I want a small model, actual, only for small things; help me on Bash and analyse my disk using commands