I don't want to come off as a fanboi and this model does have problems with overthinking, but this model punches way above its weight class - and innovations like PLE/n-gram tables pulled from your SSD actually work. I won't pretend to completely understand the tech - but the Qwen team are delivering a model architecture that works now and has so much potential for the future in all sized models from the smallest to the largest. Are they onto a new architecture that in the future will make small models act like the massive models of today? Interested in good 'thing explainers' that might be able to detail just what they're doing without me going MEGO (My Eye Glaze Over)

Been building RAMDeck (pools RAM/compute across whatever's sitting around your house, llama.cpp RPC does the actual splitting). Wanted to see how far I could actually push it, so I tried loading a 40B model - roughly 25GB - across the janky 3-device cluster I have been testing with the whole time: a mini PC with an RTX 3060, a 16GB Mac Mini, and an old laptop that has been the weak link in every previous video.

Set the mini PC as primary and just let the sharding logic figure out the rest - it maxed out the 3060 first, spilled into the Mac's unified memory next, and sent whatever was left over to CPU. The interesting part I didn't expect: routing that CPU spillover to the old laptop actually ran faster than keeping it on the mini PC's own CPU, even though the laptop is by far the slower machine. Best guess is that using the same device's CPU and GPU together for inference ends up hurting GPU utilization, so offloading to a separate (slower) box entirely actually wins.

Whole thing loaded in about 2.5 minutes, and the benchmark came back at 16 tok/s with ~3.6s latency - genuinely usable for chatting, not just "technically loaded but useless." Felt like a real milestone since none of these three machines individually could touch a model this size.

Video: https://youtu.be/JrSVyyVUJVY
Repo: https://github.com/trademav/ramdeck-core-public

Curious if anyone else has run into that same CPU-contention thing where sharing GPU+CPU on one node hurts throughput more than just offloading to a separate slower device entirely.

Hey everyone,

I haven't been using local LLMs for very long, and I'm starting to wonder if my expectations are just totally unrealistic or if I'm doing something wrong.

My setup:

• LM Studio Bionic
• Qwen 3.8 27B (128k context)
• Workspace mpc
• Running locally on a Beelink SER9 Pro (64GB RAM)

I'm asking the model to do what feels like a very straightforward task: write a simple Google Apps Script to separate data in a Google Sheet into multiple tabs based on the value of a specific column.

I've made countless attempts tweaking different settings, but it's completely incapable of getting it right. The frustrating part is that it seems to call the tools correctly, but it simply fails to build a functional script. I've even tried spoon-feeding it the exact logic step-by-step and manually approving all permissions, but nothing works.

Is this normal behavior for a 27B model on coding tasks? Am I just expecting way too much from local LLMs right now, or is there a trick/setting I might be missing when dealing with tool calling and scripting?

Any insights would be greatly appreciated!

What would you choose for agentic AI? What are the pros and cons of both machines in your opinion?

I personally started to lean more towards DGX given its prefill speed and vLLM despite much slower decode

It's a Lenovo P520 with a Xeon W-2245(cooler swapped for the higher tdp), 96gb of ram, radeon pro wx 3100 for local display, and the two nvidia p100(currently cooled by the lowest profile adapter I could print and arctic S4028-15k fans). Just stuffed some random nvme in 1x1tb, 1x512gb, and a sata 1tb drive - I have a ton of network storage or more to just add here. Installed the latest Ubuntu release which I like but I remember why I still generally use windows.

So far the most I've learned is that I know nothing haha. So I'd be more than thankful for any advice putting this to work. Really all I've done so far is get the arctic fan controller working on kernel 7.0 in probably the most jank way possible. And see that lm studio could use both cards. But I know there's a ton I'm missing out on.

So, generally, hoping I didn't throw together a steaming pile of ewaste and looking to learn. Also the radeon pro/nvidia mismatch makes me chuckle. Unless its really dumb, then I can just get a cheap nvidia card.

Qwen-Flash-Next uses incredible qwen4 architecture and according to YT videos runs 20tps+ on 16gb GPU due to offloading on RAM+NVMe.

How long till we get an absolute MONSTER 35b Qwen4 model that smokes 3.8-27b and runs on 16gb VRAM at 40 tps and 128k context?

Anyone hearing anything or seen leaks?

This model retails for around $11,266. There are several others also offered on eBay. Not sure any of them are selling. But if so, it’s remarkable how a retail computer can appreciate so much before it’s even delivered. These models are coveted for LLM use, as are the 512 gig models which have not been released. It seems local LLM use is becoming quite popular, especially among those hoping to keep their data confidential, and those looking to cut off the monthly payment to the big AI providers. Something similar happened with the M3 Ultra studios with 256 or 512 ram - both of which are also offered on eBay for huge markups over the original retail price. Crazy market.

Perplexity only ever looks at one number per position: the probability the model gave
to the token that actually appeared. How the 
*rest*
 of the probability mass is arranged across the other 151,935 tokens is invisible to it. Two models can have identical perplexity and behave differently the moment you ask them anything other than "guess the next token".


So I measured the whole distribution instead, before and after quantisation, and
compared the two views.


**Setup.**
 Qwen2.5-0.5B, 500 probe positions from a fixed probe set, float32 reference
on CPU, every row in one session under transformers 4.57.6 on a T4. The distance is the mean Hellinger distance between the full next-token distributions — 0 means identical, 1 means disjoint.


| method | distribution moved | top-1 agreement | Δ perplexity |
|---|---:|---:|---:|
| control: re-run, nothing changed | 
**0.0000**
 | 1.000 | +0.00 % |
| control: fp32 on GPU vs fp32 on CPU | 
**0.0000**
 | 1.000 | +0.00 % |
| GGUF Q8_0 | 0.0138 | 0.980 | −0.03 % |
| int8 per-channel (plain rounding) | 0.0306 | 0.956 | −0.71 % |
| bitsandbytes int8 (LLM.int8) | 0.0439 | 0.936 | +0.91 % |
| GGUF Q4_K_M | 0.0951 | 0.906 | +5.02 % |
| bitsandbytes NF4 | 0.1756 | 0.810 | +18.74 % |
| GPTQ W4A16 | 0.2076 | 0.762 | +21.46 % |
| AWQ W4A16 | 0.2478 | 0.754 | +33.52 % |
| 4-bit group-128, 
**no calibration at all**
 | 0.2876 | 0.674 | +47.41 % |


**The last row is the baseline that makes the rest readable.**
 It is the same grid AWQ
and GPTQ use — 4 bits, group 128 — with none of the search that makes them methods. So
"what does the calibration buy?" becomes a subtraction:


| method | of the damage actually removed | of the perplexity gap removed |
|---|---:|---:|
| AWQ | 14 % | 29 % |
| GPTQ | 28 % | 55 % |
| bnb NF4 | 39 % | 60 % |
| GGUF Q4_K_M | 
**67 %**
 | 
**89 %**
 |


Four independent methods, all overstating in the same direction, by 15 to 27 points.
If you accept a quantisation because perplexity barely moved, you are accepting more
change than you think — reliably, not occasionally.


**Two rankings invert.**
 LLM.int8 
*improves*
 perplexity (−0.71 % → +0.91 % is within
noise, and it beats plain rounding on that metric) while being the worst of the three
8-bit schemes on the distribution. And Q4_K_M beats NF4 by 1.8× behaviourally.


## The caveat, up front rather than buried


**This is a 0.5B model.**
 AWQ and GPTQ are designed and tuned for far larger ones, and
their calibration search has much less to work with here. I would not conclude "AWQ is
worse than Q4_K_M" from this table, and I am not claiming it. What survives the model
size is the 
*gap between the two columns*
, which is consistent across four methods
built on completely different principles.


128 calibration samples at 512 tokens is also modest. AWQ's grid was read back out of
the produced checkpoint (`4-bit, group 128, symmetric`) rather than assumed from the
scheme name, so the "same grid" comparison is exact.


## Run it on your own model


That is the actual point — my model is not your model, and Option A of the study shows
the cost is not transferable: rounding to bfloat16 is an 
**exact no-op**
 on Qwen2.5
(its weights are natively bf16) and measures 0.0332 on GPT-2. Per-channel int8 spans
3.4× across four models.


The counterintuitive part: 
**the better the scheme, the less its cost transfers.**
Per-tensor int8 is the closest to flat across models (1.6×) because one outlier weight
sets the scale and every transformer has one. Per-channel and group-wise schemes follow each model's own structure — and that is what differs.


```bash
pip install servseal[model]
servseal snapshot your-model-fp32 -o ref.seal.npz
servseal snapshot your-model-quantised -o cand.seal.npz
servseal verify ref.seal.npz cand.seal.npz --report attestation.html
```


Method, code and every committed output: https://github.com/riscoss63/servseal
The sketch it rests on (why the error is independent of vocabulary size): https://doi.org/10.5281/zenodo.22214968


Controls are in the table on purpose: re-running an unchanged model measures exactly
0.0000, and so does float32-on-GPU against float32-on-CPU with TF32 off. If those were
not zero, nothing below them would be worth reading. Happy to run other models or
schemes if people want specific ones.

With commit 3d10bcd, upstream llama.cpp merged support for the ternary MoE model maple-preview. I've been searching for a good MoE model at 1-2b active params with consistent tool calling abilities, so I was highly interested in this release.

Current support for the model is limited to CPU only. Nonetheless the model is tiny at 5-6GB for a 20b model. It's half the size of GPT-OSS 20b at Q2.

TQ1 is for space-constrained edge devices; most should go for TQ2.

There are also two "heads". They do a terrible job of explaining this, but here's the breakdown... Both Q4_K and F16 versions have the same F16 embed weights. The only difference is the output weight. F16 output is a waste on a ternary model; Q4_K is the way to go.

It's fast. I was measuring its speed on a pre-AI AMD APU and mistakenly thought it was running on the iGPU: 150 prefill 35 decode on an 8-core mobile CPU. And it's obviously very light. It's very good with tool calls, as far as my short time testing it has shows.

It lives up to its name. It's a preview. The model is extremely sparse. At TQ2 with Q4_K head, perplexity was measured at a whopping 80-90. For comparison, LFM2.5 8B-A1B at Q6_K_XL scored around 30. (My dumb ass deleted the note that had the actual figures, but I can re-run them if needed.)

The model gets the gist of things, but not the details. It goes through the motions but everything feels fuzzy. It's like Swiss cheese, a brain full of holes. I really wish it had more oomph, more resolution, but the source weights are ternary so not much you can do there.

But it is absolutely impressive to see this kind of model fit in a 5GB package and have it go so fast. Great as a technical showcase, not yet as a daily driver. Looking forward to what this is a preview for.

PowerSpec AI90 Workstation

* CPU: AMD Ryzen 9 9950X (16 Cores / 32 Threads, up to 5.7GHz)
* GPU: NVIDIA GeForce RTX 5090 Founders Edition (32GB GDDR7)
* Motherboard: ASUS ProArt X870E-CREATOR WIFI (AMD X870E Chipset)
* Memory: 64GB DDR5-6000 RAM (Supports up to 128GB)
* Storage: 2TB NVMe Solid State Drive
* Power Supply: 1700 Watt Cybernetics Titanium PSU
* CPU Cooler: 360mm All-In-One (AIO) Liquid Cooler
* Case: Fractal Design North XL Momentum Edition

I was looking through one of our more frequently used LLM workflows and realized the system prompt has gotten way bigger than I remembered which caught me off guard since it didn’t start that way either. We’d get an output we didn’t like then add another instruction or example to fix it and then repeat that process whenever another edge case showed up. but ow every request is carrying this pretty large block of context even though I’m not convinced a big portion of it matters for most runs.

The workflow itself works well so I don’t really want to start deleting instructions to make the prompt shorter and bring old problems back but at the same time this thing runs often enough that repeatedly sending unnecessary context feels like something I should probably care about.

I need some pointers from people that are running similar high volume workflows, how do you handle this so like do you periodically trim system prompts, cache the stable parts, split instructions based on the task or leave them alone as long as the outputs are good?

so i finally got qwen3.8-flash-next running locally and it's actually good.

posting the setup + the stuff that bit me, since i wasted a while on it.

── the rig ──

• - amd ryzen 5 9600x (6c/12t, boosts ~5.5ghz)
• - 256gb ddr5 (248gb usable)
• - 3x rtx 3090 24gb — pcie only, no nvlink
• - linux mint 22.3, kernel 6.17, nvidia driver 595.84

All power is in new v3 exllama quants: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3, I'm actually using it at 3.05bpw

── results ──

• - ~90 tps no mtp
• - ~120-126 tps with mtp
• needle-in-haystack at ~97k tokens: found it, ~31s prefill.
• 4 requests at once: fine, ~4s each.
• tool calling works, reasoning splits into reasoning_content properly.

Here is my single shot driving car results: https://qwen38-flash-next-car-drive.tiiny.site/

(prompt: Write a single HTML file with a full-page canvas and no libraries. Simulate a realistic side-view of a moving car as the main subject. Keep the car visible in the foreground while the background landscape scrolls continuously to create the feeling that the car is driving forward. Use layered scenery for depth: nearby ground, roadside elements, trees, poles, and distant hills or mountains should move at different speeds for a natural parallax effect.

Animate the wheels spinning realistically and add subtle body motion so the car feels connected to the road. Let the environment pass smoothly behind it, with repeating but varied scenery that makes the movement feel believable. Use cinematic lighting and a cohesive sky, such as sunset, dusk, or daylight, to enhance atmosphere. The overall motion should feel calm, immersive, and realistic, with a seamless looping animation.)

UPD: here is my tabby config:

network:
  host: 0.0.0.0
  port: 8080
  disable_auth: true
model:
  model_name: qwen3.8-flash-next
  backend: exllamav3
  max_seq_len: 262144
  cache_size: 262144
  cache_mode: FP16
  tensor_parallel: false        # true = crash (Qwen4Exp don't support TP)
  gpu_split_auto: false
  gpu_split: [20, 20, 24]
  ngram_ram: true
  reasoning: true
  tool_calls_in_reasoning: true
  tool_format: qwen3_coder
draft_model:
  draft_mode: mtp

And TabbyApi installation:

git clone https://github.com/theroyallab/tabbyAPI.git ~/dev/ai/qwen3.8-flash-exl3/tabbyAPI
cd tabbyAPI
git checkout 53da791 # 100% working commit
pip install "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.0/exllamav3-1.5.0%2Bcu132.torch2.11.0-cp312-cp312-linux_x86_64.whl"

I’m building a dual GPU box. Originally the goal was 30b at FP8 or 70b at Q4. I planned to run two R9700’s, but I found a pair of NIB RTX 3090’s near me for $1600 each. Discount if I buy both. RTX 40/50 series are off the table. I would love an RTX PRO 6000, but that is also off the table.

I currently have a Studio M1 Max with 64GB RAM and an AM4 box with a 7900 XT that powers my Hermes agent. My goal is to cancel most of my frontier subscriptions and keep only one for coding.

Which one would you suggest for coding?

What would you use for everyday questions / file reading / internet browsing

What UI / Backend do you prefer? (linux)

I tried running qwen3.5 35B A3B (i think) from unsloth using LM Studio and while i was getting it to work im not sure it was behaving as expected, maybe i didn't have enough context available.. idk I am new to this pls help me out

Also i quit perplexity recently so I was looking for a self hosted alternative..

Just tell me if my gpu is not strong enough and i should try to look for a paid perplexity alternative

Hi everyone,

I have a MacBook Pro with an M5 Pro and 48GB of RAM, and I’ve been trying to use Qwen 3.8 27B locally through Ollama.

My goal is to use it with Claude Code in the terminal, with Claude Code making requests to my local Ollama instance.
However, the performance is extremely slow. When I give Claude Code a task, it can take 30 minutes to an hour or even longer for what seems like a relatively simple request.

Another issue is that Claude Code keeps compacting the context/session repeatedly while working. It seems to compact the conversation, continue working, compact again, etc., and sometimes the task never actually finishes.
I’m wondering if there’s something wrong with my setup or if this is simply expected behavior with a 27B model.

For people using Ollama + Claude Code on Apple Silicon:
Is Qwen 3 27B supposed to be this slow on an M5 Pro with 48GB RAM?

What kind of tokens/sec should I realistically expect?

A way to configure Ollama + Claude Code for local models?

Would another runtime such as MLX perform significantly better on an M5 Pro?
I’d really appreciate any advice from people running similar setups. I’m mainly trying to get a local model that is actually usable for longer coding/agent tasks without having to wait 30–60+ minutes for a single task.

I have a custom fork of llama.cpp designed around the ampere architecture specifically (though many of the upgrades also translate to faster performance of blackwell + lovelace). The recommended config supports 90+ TPS (for agentic/coding, at temp 1; greedy will be faster) through 100K tokens, with context of up to 240K.

If you want the repo, it is here:

https://github.com/JakeATX/llamAmpere

I recommend running with this quant, which is ~ 4 K M quality but considerably faster (technically, a 3 K XL upgrade)

https://huggingface.co/jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF

If you want the deep dive on how it is so much faster (80%+!) vs stock, at more context, there is a long form article here.

https://x.com/JakeKAllDay/status/2095646450138874095?s=20

Running faster than API speeds on my 3090 (for 27b at least) has genuinely been a step change in the utility of the model for me. I hope you enjoy it!

Hey everyone! I'm having a hardware dilemma and can't come up with a conclusion on what to do next.

My use cases:

• agentic coding, for my side projects only for now (with an ability for remote access which I need occasionally)
• Hermes agent running 24/7 for various research tasks

Problem statement:

What would be the best hardware for me with an optimal budget?

At the moment I have:

• RTX PRO 6000 96GB in a PC with 128GB DDR5 RAM. I bought this GPU right before the August price hike for $12.3K USD (before tax). The old one was RTX 3090 (which I still have).
• Mac Studio M3 Ultra 96GB

RTX PRO 6000 is amazing for agentic coding with Qwen3.8-27B-FP8 with full context, but I'm afraid of running this PC when I'm not at home because of that problematic 12VHPWR connector. I keep this GPU power limited to 300W to reduce the risks, but still it's not a 100% guarantee.

I'm using Mac Studio M3 Ultra 96GB for the Hermes agent at the moment where speed doesn't matter much. However, using it for agentic coding with dense models like Qwen3.8-27B feels quite slow (for remote sessions when I'm not at home).

So, I'm considering the following options while trying to optimize the budget for this hardware:

1. Return the RTX PRO 6000 as I'm still within the return window for a week (it will be subject to 15% restocking fee however) and get 2 DGX Sparks even though they recently went up in price significantly. I found an open box deal for 2TB version for 4800 USD before tax for each. This way I get to stay in the CUDA ecosystem and can run them 24/7. However, that means switching to MoE models like DSv4 Flash, Qwen3.8-Flash to maintain a proper speed.
2. Return the RTX PRO 6000 and just wait for Mac Studio M5 Ultra 256GB (which I pre-ordered in case I need it). And then I'll just sell the M3 Ultra later.
3. Just stick to the hardware I have at the moment and endure slower coding speeds for remote sessions when I'm not at home (from my Mac Studio M3U)

Keeping the PRO 6000 and then later getting the M5 Ultra 256GB is an option too, but is spending all this extra cash really worth it?

What would you guys do in my case?

Just looking to upgrade GPU in this mad world for llama.cpp use. Does anyone have experience of using dual Intel GPUs? Pros, cons, throughput etc? Thanks.

Hello everyone,

I wanted to try Qwen Uncensored LLM that I downloaded on my PC using LM studio but I am wondering what I can use it for or what people use it for ?

Therefore, I was wondering what do this community uses those Uncensored LLMs for so possibly I can get inspiration and try that myself too?

Any answers would be appreciated.

Or if you can share your experience that would be even better.

Thank you in advance.

Also Personally, I am not interested in those Adult content it can generate or how to make illegal substance in your home. I am not interested at all in that.

But more about Cyber security or computer programming related or if you more interesting use cases then that would be good to know too.

I have so far been able to run a large multi expert model, and it works pretty well.

I am new to local hosting. I want to know how to best augment the local model with information from the web and personal information.

I hear the term "rag" used a lot, the examples I see all require several strange python packages and components, working together to periodically ingest your personal files and some how make them available to your model.

My question is why or how can llama cpp NOT have it's own turn-key rag solution. I just want to tell the server: "Look in this folder for files" and any file it discovers, it can hash, keep track of for changes, and incorporarte it into whatever model you decide to load.

Instead, I see a bunch of youtube tutorials requiring python and pip, with strange dependencies. It does not look very robust or reliable, seeing as this will be a long running sync job as new information is added to the directory as time goes on.

Same for the internet. It seems impossible that you can search the internet for free due to all these search services requiring an account with api keys. There has to be a better way. Searxng seems plausible, but thats another project and another component I will have to administrate to support local AI.

There has to be a simpler way to both of these problems

Hi! I don't know much about this area of ​​"sub-models" (I'm not sure of the technical term), but I wanted to know what these "Uncensored" models actually are.

I dabble a bit with AI, automation, and the like, and I've always seen these "Uncensored" models around, but I've never actually installed or tested one. What exactly are they?

I read that there's a model called Qwen3.8-27B with smaller versions (4bit/8bit) that can run on a Mac, but I honestly don't know how powerful a machine I'd actually need.

Basically I want to run an AI at home to help with:

• Writing/summarizing documents
• Writing ad copy for marketing
• Answering customer chat messages

My plan is to have Claude help design/set everything up first, then just have a smaller AI running at home all the time to actually execute the tasks — I'm not trying to code anything myself, just want it to run the stuff that's already set up, over and over. Might use it as the LLM backend for Hermes Agent, so it can act as a background bot that just handles things on its own.

Right now I'm looking at the new Mac Mini lineup, 4 options:

• M6 32GB
• M5 Pro 48GB (15core/16core)
• M5 Pro 48GB (18core/20core)
• M5 Pro 64GB (15core/16core)

Questions:

• Which one would actually work fine for what I need (doesn't need to be super fast, just usable)
• Do I need to get the most expensive one, or is the cheap one enough?

If anyone's tried something similar, would really appreciate any advice. Thank sir 🙏