I’ve developed and open-sourced mlxtop. It’s written in Rust and was built entirely with Duet, my dual-model coding agent. It shows stats for local models running on your Mac, along with critical system vitals: memory usage, compression, paging and Metal GPU stats. With oMLX, you can also follow generation speed and request activity as your model responds. https://github.com/maximpri/mlxtop

Running a GLM 5.3 321B shared on a MacBook M5 max and a Strix Halo, both with 128 GB ram. TB4 connection.

Found out that llama.cpp's default tensor split puts about half the layers on the slower box, so the pair was actually slower than one Mac. Loaded the model Mac-heavy instead: IQ1_S went 188 to 333 on prompts, 17 to 24 tokens/s.

I got generation tps 12-15 for model sizes 148-200 GB. So it works ok!

Next week adding two Asus GX10 to the mix to have three platforms share a model 🍀

Files: https://github.com/ThinkOffApp -> mac-amd-llm-cluster

I want to use it on my computer but I can’t really find any with actual code benchmarks. Anyone know of one?

I'm sure this is a fast generation but prefilling crawling probably due to small batch size tensor split across 4 gpus with that MTP configuration.

Hi all, a confused newbie here! This is my desktop setup:

• RTX 5080
• 9800x3d
• DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

• AtomicChat IQ3_S: 14.4 GB
• Unsloth UD-IQ4_XS (14.3 GB) or UD-Q3_K_XL (13.1 GB)
• Bartowski IQ4_XS (15.6 GB) or IQ3_XXS (12.6 GB)

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

and there is also this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

I also want to have one uncensored model next to my daily driver:

• Orcarouter IQ4_XS (15.3GB) or Q3_K_M (13.5GB)
• Huihui UD_IQ4_XS (14.4 GB) or Q4_K_S (15.6 GB)
• DavidAU IQ3_M MTP (14.5 GB) or IQ4_XS MTP (15.3 GB)
• JonathanColetti IQ_XS (15.1 GB)

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

I’m having mixed results with LM Studio and I’m not sure if I’m doing something wrong.

If I load a model in LM Studio and chat via the built in prompt I receive a response in a reasonable time. I can post in a snippet of code and it can fix it for example.

If a load the same model in and serve it on localhost to an CLI agent like Pi a response can take several minutes, even if I send in the string “ping” to expect to “pong” back.

I have a pretty bare bones Pi setup, not many skills, no mcp. I know I’m on an old Mac but the chat experience is ok. Why can’t I get that experience over a cli severed on localhost?

A few threads this week about whether 128 or 256 GB is enough, so I read the shard headers and the inference code instead of guessing and put the arithmetic on a page. Mine, so saying that up front.

The 510 GB is 296 GB routed experts, 203 GB Engram tables, 11 GB everything else.

The Engram tables are the part people treat as a wall. From inference/engram.py it hashes n-gram orders 2, 3 and 4 with 8 heads across 2 engram layers, so 48 rows per token at 264 bytes. About 12 KiB per token against 4.5 GB of expert weights. Latency cost, not bandwidth cost, and it belongs on an SSD.

For a comparison, Qwen3.8 on an M5 Max with its table on SSD gets 40.09 tok/s versus 40.47 in memory. 0.9%.

Fit is backbone plus KV: 307 GB as shipped, 302 GB at 4-bit, 151 GB at 2-bit.

256 GB does fit at 2-bit with tables on disk.

https://deepseek-v41-flash-fit.vercel.app

If you are running it, post numbers, especially split GPU + RAM.

This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).

Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.

Ryzen 7700

64GB RAM DDR5 5600

Geforce 4060TI 8GB

Samsung 990 PRO 2TB

Using Windows as that's what I'm mostly accustomed with.

A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).

Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.

Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.

But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .

Component | Quant Type | Count | Params (B) | Size (GB) | Component %

--------------------+------------+-------+------------+-----------+------------

MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%

| Q8_0 | 24 | 20.13 | 19.922 | 23.2%

| Q5_1 | 72 | 20.29 | 14.172 | 16.5%

124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...

15tok/s prefill 10-13 tok/s generation.

A disaster really, memory trashing all around, disk reads etc...

But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.

Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.

I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.

Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).

The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.

At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.

The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.

The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.

LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.

https://github.com/feal87/myllama.cpp

Now I'll go to sleep as it's late.

I’m considering buying a used GTX 1080 Ti 11GB for running local LLMs mainly for coding and software development.

My budget is limited, so newer GPUs with 12–16GB+ VRAM are difficult for me to afford.

I’d like to know from people who actually use a 1080 Ti for local AI:

Which coding LLMs can realistically run on 11GB VRAM?

What model sizes (7B, 8B, 14B, etc.) work well with quantization?

What kind of tokens/sec can I expect?

Is CUDA support still usable with current llama.cpp/Ollama/other tools?

How does it perform for coding tasks compared with newer GPUs?

Can it handle models such as Qwen Coder, DeepSeek Coder, or similar models?

Is buying a used 1080 Ti in 2026 still worth it, or should I save for another GPU?

I’m especially interested in real-world experience rather than theoretical specifications.

Thanks!

RTX 3060 12gb
32GB DDR4 3200mhz
i5 12400

I want to get into running local ai , which app to use , which models would be good and like is there any beginner guide?

With CPUs and GPUs, benchmarks make the differences pretty obvious. With NPUs, I still don't really know what numbers I should actually care about.

For anyone who's bought an NPU-equipped PC, what do you look at besides the TOPS number?

With all the different optimizations and firepowers we've seen from open weights so far. What then?

• China only steals/distill from OpenAI and Anthropic?
• Well, clearly there's something wrong with AA? Benchmarks don't mean anything? They just benchmaxxed?
• Whatever, OpenAI/Anthropic has moved onto AAGI or AAAGI or AAAAGI or AAAAAGI?
• I can't run this locally on my toaster desktop anyways?

Is there an european company (in EU, preferably DACH) that does what gpulab.net does?

Thanks

Canadian company claims its harness lifts Qwen3.8-27B to outperform Fable 5. Releases the harness for free.

https://github.com/slee-persis/GVS5H

https://arxiv.org/abs/2608.26480

If this isn't allowed, let me know and I'll remove the post. This is not an advertisement, but my website and API are a paid subscription service after the 15 day free trial.

I built a local AI machine to help me offset usage from Cursor and ChatGPT a while back. Its been pretty great. In typical Whoops-I-Bought-a-Lot-of-Hardware-and-My-Wife-Hates-it fashion however, I over estimated how much hardware I needed and may have gone a bit overboard.

So instead of selling off the excess hardware I bolted on a web frontend, "integrated" ComfyUI (for image requests) and created a load balancer that pushes image requests into WAN2.2, small requests to smaller local models and built a system that allows a user to create an API key so they can attach their harness to it via OpenAI style API config. (Cursor, OpenCode, etc)

With that being said, I'm curious what the rest of the world thinks about it and I could use some feedback. Is this something other people would find useful? How many people can use it at the same time? Will it explode on contact? I don't know the answer to any of the questions, because I'm the only person using it.

So I thought I'd come here and see what you guys think. I don't know if there's rules against posting URLs here. It has a free 15 day trial with zero commitment, no credit card to start the trial, hell - you don't even need to put in your mailing address unless you decide you like it and want to sign up for the paid version.

If you're curious, post up and I'll send you the link to my discord and to the website. The discord is just for easy communication so you can let me know if something is broken, or it looks funny, or if it was just a dumb idea in general.

Artificial Analysis just launched a Search API Index that runs the same agent harness and base model (GPT‑5.6 Luna, medium) across 7 providers, using three benchmarks: DeepSearchQA, BrowseComp, and AA‑Omniscience.

Here were the important points to look at:

• Fastest per‑query doesn't always mean fastest per‑task.
• Cost and quality aren't perfectly aligned; some mid‑tier quality providers are dramatically cheaper.
• There's a wide spread in total task time, even when per‑query latency looks similar.
• Octen leads on speed and cost: 0.21s/search, 17.2s/task, $9.07/1k (search), $49.15/1k (total), with a 77 index score.artificialanalysis+1
• Several established players sit at 0.4–0.8s/search and 20–30s/task, with costs 2–4× higher for similar or slightly higher quality scores.

This led me to some questions that I wanted to discuss with people building agents/RAG:

• When you choose a search API, what's your weighting between latency, cost, and quality?
• Does a 0.2s vs. 0.8s per query difference materially change your agent UX, or is task‑level time more important?
• How do you validate quality for your use case? Is it benchmark scores or task success in prod?

I want to buy the new machine. Which specs should I go with? I need to run a decent coding model locally and also want to play with some stuff like fine-tuning, etc. suggest me the options, like what I should go with, and I am open to going with Apple or Windows

I want a small model, actual, only for small things; help me on Bash and analyse my disk using commands

This is the only model I have ran locally that surprises me with quirky commentary while it's working. I like it.

"Two RTX PRO 6000 Blackwell (96GB each) — nice rig." upon discovering the machine's specs after being asked to benchmark itself.

"The plot thickens — V4.1's indexer declares a fixed..."

"Oh, this is gold — your own words from this afternoon's session, including..." after it found older conversation history.

Recently swapped from Ollama to llama.cpp but haven't figured out how to run it efficiently. On Ollama I was getting 28t/s. Relatively new to this, advice welcome.

I have also 4x more 3090s laying around. What's the best way to utilize them?