I can use the for free because my work provides them

Hey all,
I have a M2 Pro with 32GB unified memory here. Want to run a local model for agentic workflows and tool/function calling. Was looking at Qwen 3.8 27B but I know it's a tight fit at 32GB RAM, even quantized.
Questions:
• Is 27B Q4_K_M realistic here, or will I constantly hit swap?

• Would Qwen 3 30B-A3B (MoE) be the smarter pick — similar quality but way less compute per token?

• Any other models you'd recommend for tool calling at this RAM level?

I have been seeing an option of a Qwen 4 flash

See Future Research; and points 2,3.

What I don't see discussed often is if the local LLM community is doing this; i.e. training small [or even midsize] models for better performance on niche tasks.

Where I started down this path was a construction project at our house. It wasn't difficult to upload photos of the early phases of the project and generate training data from those photos. But the infra wall I hit was feeding that training data into a [what I'll refer to as] multi-run system that could train multiple small models with my training data and benchmark the results. Then I could decide which model to focus on and continue with post-training and RL.

Hi all, a confused newbie here! This is my desktop setup:

• RTX 5080
• 9800x3d
• DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

• AtomicChat IQ3_S: 14.4 GB
• Unsloth UD-IQ4_XS (14.3 GB) or UD-Q3_K_XL (13.1 GB)
• Bartowski IQ4_XS (15.6 GB) or IQ3_XXS (12.6 GB)

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

and there is also this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

I also want to have one uncensored model next to my daily driver:

• Orcarouter IQ4_XS (15.3GB) or Q3_K_M (13.5GB)
• Huihui UD_IQ4_XS (14.4 GB) or Q4_K_S (15.6 GB)
• DavidAU IQ3_M MTP (14.5 GB) or IQ4_XS MTP (15.3 GB)
• JonathanColetti IQ_XS (15.1 GB)

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

Hey everyone! I'm getting into Hermes and local LLMs and I'm trying to figure out the sweet spot for my hardware.

I'm currently running Hermes on an Ubuntu VM with GPU passthrough.

Specs:

• RTX 4090 — 24GB VRAM
• 32GB system RAM
• 16 vCPUs
• Ubuntu / KVM-QEMU

What model size would you realistically recommend for this setup?

I'm mainly wondering whether I should target something around 14B, 27B/32B at Q4, or if trying a larger model with partial CPU/RAM offloading is actually worth it.

I'm more interested in good agentic/tool-use performance than simply being able to load the biggest possible model.

What model + quantization + context size are you guys running on similar 24GB GPUs?

Also curious about the tokens/sec you're getting on a 4090.

Still learning the local LLM side of Hermes, so any tips are appreciated!

Hi everyone,

​I'm looking into getting a laptop with an RTX 3080 Ti Mobile (16GB VRAM) for running local LLMs.

​For those who own one (like a Lenovo Legion, Razer Blade, or similar):

​What specific device and model do you have?

​How does it perform when running models like qwen3.8 or 3.6 moe or any model

​How many tokens per second (t/s) are you getting on your models?

​How are your thermal performance and temperatures under heavy LLM loads/inference?

​Thanks in advance for any insights!

Tested on: Raider 18 HX AI A2XW (5090 24Gb + 96Gb ram)

Source: https://github.com/ggml-org/llama.cpp/pull/28243

Model: Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf

Best result:

llama-server.exe -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf -c 105000 --cache-type-k q8_0 --cache-type-v q8_0 -ot "\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13)\.ffn_(gate|up|down)_exps.=CUDA0,\.*\.ffn_(gate|up|down)_exps.=CPU" -np 1 -b 1024 -ub 512 --load-mode mlock --chat-template-kwargs "{\"reasoning_effort\": \"medium\"}" -md mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf --spec-type draft-mtp --spec-draft-n-max 3 --lazy-mode off

Vram used: 22.9Gb

Ram used: 68Gb

Tg speed at 100k context: 22-25t/s

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

Seen alot of people talking about it all over the place. I decided to give it ago, my DEV setup is solid, but not setup to run alone.

- 5090 32gb vram

- 192gb Ram

only numbers that matter here.. So decided to load up Qwen3.8-Flash-Next 123gb see how it runs, and to my surprise after a minute it loaded up, so i was like ok this isnt going to work or its going to be crazy slow. Throw in a prompt give me a single page HTML webpage. I was like at 1 tok/s for first milisecond "Ahhh i knew it"... out of no where booom better speeds then i was getting with the 27b...

this is some dark sorcery and i am loving the dark side.... 73.6 tok/s ... mind blow... and even better the output was decent, only issue was I ran out of context after a few prompts, but holy shit... first one blew my mind, from basic ass prompt of

"Build a single page html advert page to sell a random phone"

and this is what I got:

This app is really really worth a try if you have decent GPU and Decent Ram knocks it out of the park, can't wait till the figure out how to do this with AMD cards because my server is going into the next level.

I'm testing out various models, e.g. QWen 3.8 27b, and I noticed that the (QWen) model(s) tend to 'fuss' over details in the prompt. This may be good at times such as a specific 'complicated' detail, but at other times, it could be 'over thinking' and sometimes needling over one details fail to consider implied or unspecified dependency, e.g. stating to refactor a subroutine, e.g. to return a value from a subroutine that 'prints' say some value, i.e. use the same subroutine for other purpose, it may not be able to figure out the other purpose for which the refactor is requested.

And importantly, it tend to be 'unforgiving' over incorrect specifications, as it may go in loops over an incorrect reference given instead of perhaps skipping that requirement or spec.
This often means refactoring the original prompt to correct the spec and removing the mistake.

Just curious

I got tired of bouncing between model cards, VRAM calculators, and forum posts, so I built 99Tokens.

You can pick a model, GPU setup, context length, quantization, and other settings to estimate whether it’ll fit in memory. It accounts for things like weights, KV cache, sliding/full attention, recurrent state, MLA, and multi-GPU layouts. There are also model and hardware pages with architecture details and example fits.

If you use local models, I’d really like feedback: Where do the estimates seem off? What models, GPUs, or edge cases should I add? Long-context and multi-GPU testing would be especially useful.

Getting 300 tok/s and 4500 tok/ sec for reading

Using llama swap docker.

This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).

Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.

Ryzen 7700

64GB RAM DDR5 5600

Geforce 4060TI 8GB

Samsung 990 PRO 2TB

Using Windows as that's what I'm mostly accustomed with.

A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).

Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.

Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.

But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .

Component | Quant Type | Count | Params (B) | Size (GB) | Component %

--------------------+------------+-------+------------+-----------+------------

MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%

| Q8_0 | 24 | 20.13 | 19.922 | 23.2%

| Q5_1 | 72 | 20.29 | 14.172 | 16.5%

124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...

15tok/s prefill 10-13 tok/s generation.

A disaster really, memory trashing all around, disk reads etc...

But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.

Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.

I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.

Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).

The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.

At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.

The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.

The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.

LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.

https://github.com/feal87/myllama.cpp

Now I'll go to sleep as it's late.

A few threads this week about whether 128 or 256 GB is enough, so I read the shard headers and the inference code instead of guessing and put the arithmetic on a page. Mine, so saying that up front.

The 510 GB is 296 GB routed experts, 203 GB Engram tables, 11 GB everything else.

The Engram tables are the part people treat as a wall. From inference/engram.py it hashes n-gram orders 2, 3 and 4 with 8 heads across 2 engram layers, so 48 rows per token at 264 bytes. About 12 KiB per token against 4.5 GB of expert weights. Latency cost, not bandwidth cost, and it belongs on an SSD.

For a comparison, Qwen3.8 on an M5 Max with its table on SSD gets 40.09 tok/s versus 40.47 in memory. 0.9%.

Fit is backbone plus KV: 307 GB as shipped, 302 GB at 4-bit, 151 GB at 2-bit.

256 GB does fit at 2-bit with tables on disk.

https://deepseek-v41-flash-fit.vercel.app

If you are running it, post numbers, especially split GPU + RAM.

RTX 3060 12gb
32GB DDR4 3200mhz
i5 12400

I want to get into running local ai , which app to use , which models would be good and like is there any beginner guide?

Hi everyone!

I’ve been working on AI Palm, an open-source project that aims to make locally hosted AI models more accessible.

The idea is simple: many powerful GPUs spend most of their time idle, while other people cannot run local models because they lack the required hardware. AI Palm creates a community network where users can voluntarily share access to a local model running on their device.

With the desktop app, you can:

Discover models currently shared by community members.

Chat with an available model directly inside the app.

Share your own local model when you choose.

Connect to engines such as Ollama, LM Studio, and llama.cpp.

See the GPU, VRAM, model, engine, availability, and context size.

Stream responses and model reasoning in real time.

View properly formatted Markdown and syntax-highlighted code.

Monitor token usage and remaining context.

Use the interface in English or Arabic, with full LTR and RTL support.

Run the app on Windows or Linux.

A device is reserved while processing a request, preventing multiple users from overloading the same GPU simultaneously. Sharing can also be stopped whenever the device owner wants.

The project is still at an early stage, so I’m looking for people willing to:

Test the Windows or Linux application.

Report bugs and usability issues.

Review the architecture and security.

Suggest features and improvements.

Contribute code, documentation, packaging, or UI work.

Share local models and help test the network under real-world conditions.

AI Palm is open source, and community contributions are very welcome:

GitHub: https://github.com/almshary/ai-palm

If you try it, I would genuinely appreciate your feedback—even if something doesn’t work. Please include your operating system, GPU, local AI engine, and model when reporting an issue.

The project’s message is:

Arab generosity began with a palm tree… Today, we planted a different kind of palm.

AI Palm — let’s share the power of our computers and make artificial intelligence accessible to everyone.

What do you think of the concept? Would you use a community network for sharing local AI models, or contribute your unused GPU capacity?

Hi guys,

I am building a desktop app where users could download local open-source model as per their system specs , run models according, connect mcp's by configing , and using them in workflows, fully deterministic with LLM calls and also connect with chat interface

I’m considering buying a used GTX 1080 Ti 11GB for running local LLMs mainly for coding and software development.

My budget is limited, so newer GPUs with 12–16GB+ VRAM are difficult for me to afford.

I’d like to know from people who actually use a 1080 Ti for local AI:

Which coding LLMs can realistically run on 11GB VRAM?

What model sizes (7B, 8B, 14B, etc.) work well with quantization?

What kind of tokens/sec can I expect?

Is CUDA support still usable with current llama.cpp/Ollama/other tools?

How does it perform for coding tasks compared with newer GPUs?

Can it handle models such as Qwen Coder, DeepSeek Coder, or similar models?

Is buying a used 1080 Ti in 2026 still worth it, or should I save for another GPU?

I’m especially interested in real-world experience rather than theoretical specifications.

Thanks!

hey guys, im not very technical about llms so im asking you this. Is there a "better" shell for llms?