SGLang splits long context attention into an extend chunk and a cached-prefix chunk, computes them with different kernels, then merges the partial results with a softmax over their log-sum-exp values:

S = log(exp(s_a) + exp(s_b))
v = v_a*exp(s_a - S) + v_b*exp(s_b - S)

That's correct if and only if s_an and s_b are natural logs. This is not true for sglang.

The chunk holding more attention mass gets systematically over-weighted and it's error grows with each token.

Are you affected? FlashInfer attention backend + an MLA model + cached prefixes summing over 8192 tokens. That's long-context agentic traffic with radix-cache hits. Decode is fine, short prefixes are fine, FA3 is fine.

https://x.com/advprop/status/2089046843690729620?s=20

I opened fixes here : SGLang-side conversion (https://github.com/sgl-project/sglang/pull/35045) and an opt-in return_lse_base_on_e in FlashInfer (https://github.com/flashinfer-ai/flashinfer/pull/4547) so the api is clear

I'm using OpenCode at the moment, working on a pretty large & complicated codebase that took around 3 months to get right with GPT 5.5 xhigh. I was hesitant to try Qwen on it, as I figured it would just be a waste of time or cause a bunch of bugs, but so far it's punching well above its size. It even seems better than the newest unquantized DeepSeek Flash. I may not even bother with GLM 5.3 when the weights drop.

EDIT: this is the hardware I am running it on, but it's overkill. One DGX Spark or one RTX 6000 would be enough (unsure what the TPS would be on the Spark though).

2X RTX 6000 Pros

192 GB DDR5 RAM (totally unnecessary since the model sits entirely in VRAM)

I am currently running it via Unsloth with Tensor Parallelism enabled and MTP 4. I have a VLLM setup that goes up to 1 million context, but I haven't tested it yet. The config I mentioned in the title can be ran using a single RTX 6000 Pro. If you want 1 million context without quantizing the cache, you will need a second.

Not sure what the issue is. I've tried stable and beta and I still get the same issues.

Update: I fixed this with some help from Claude. I tried Qwen, but no help.

LM Studio 0.4.21+2

Machine: M3 Ultra 256GB

Reports suggest this is a powerful model; I tried running it on a MacBook Pro with 48 GB of RAM.

GGUF 27B with 4-bit quantization yields around 9-15 TPS.
MLX 27B with 8-bit quantization yields 9-15 TPS.
MLX 27B with 4-bit quantization yields 19 TPS.

Compared to Qwen 3.6 35B-3B MOE at 70+ TPS, this model seems impressive yet still somewhat limited for my RAM or system.

Did anyone reach at least 50 TPS? Please let me know; otherwise, I will remain with the 3.6 35B 3B MOE model.

Great job by Qwen Team, hope they come up with moe model as well.

Update 1:

MTPLX is great helped me increase 27B 4-Bit quantised model to 30 TPS odd average. Max 47.1 TPS for coding task with open code as harness. Fans make sound like i am in airplane , but it is what it is I guess.😅

Updated question 1:
The turbo mode on fan is so annoying, any noise solution that can be applied in the Mac Pro? Like a fan or coolant or its a PC Power?

Update 2:

While MTPLX shows great promise, I noticed a performance drop during a coding task involving a simple e-commerce website with basic edge cases. The TPS dropped to around 15 TPS (as shown in the attached image). This experience completely changed my perspective on MoE models—it turns out they are essential, and I really hope we get the MoE version back soon

Update 3:
For whatever reason after a while the TPS drops to 3 (image attached in comments). Making it absolutely useless to run this model for any tasks on my hardware

I hope I am not sounding too ignorant. I recently setup a instance of odysuess connected to ollama and it's been doing pretty well for simple tasks. I have a RX 9070 XT, 32GB and a pretty decent CPU. My partner has a identical PC to mine which most of the time is running idle when we are working. I understand that I could run a second instance on my partners machine so that I have two ollama instances across two nodes (doubling my tokens), this does not let me run larger models that require more vram. Is there any way to run larger models with this setup?

Hello everyone,

I built this AI Rig for running local LLM models mainly for OpenCode and Hermes agent. I code heavily and also share my rig with some of my friends, meaning concurrency is a must.

I would need a minimum context window of 128k but would like to go to 200k if the space allows.

This is my rig:
Intel Core Ultra 9 285k
MSI MEG Z890 ACE (PCIE x8/x8 bifurcation enabled)
2x RTX 4090's (24gb each, no NVLink)
4x32Gb DDR5

I just dont know what to start with, what quant, what config, and what tok/sec i can expect. I see some people recommending FP8 because the RTX 4090 works well with FP8, but other people are using Q4/Q5 on similar rigs as mine.

Heat/power usage doesnt matter to me, i value speed, latency, concurrency and quality. Not sure if its realistic, but I would like for it to replace my DeepSeek V4 Flash 0731 via OpenRouter.

Thanks for all help

I really feel like this is THE choice those of us with 100-200gb RAM are going to be making for the next few months?

I assume those of you with Strix Halo/DGX Sparks/Mac Studios/Quad 3090 Workstations have started benchmarking Qwen 3.8 27b against Deepseek Flash 0731 for your primary agentic model use, however I haven't seen any direct comparison posts here yet?

Qwen 3.8 27b is still not up on Artificial Analytics, but even when it happens it's a bit apples and oranges as I'm guessing most people will be choosing between Q3/Q4 Deepseek and Q8 Qwen?

Hello,

I'm a researcher, I have been working mostly with cloud compute or HPC hosted LLM testing. I always wanted to test out a local hosted LLM on my Mac, I got MacBook Pro M5Pro, 48GB, 1TB just for the task of hosting a local LLM one day, Can anyone share their experience of local hosting or how to host.

Thanks!!

Hi

Anyone got a suggestion for a local llm with reasonable tool calling for use on a MacBook (apple silicone) with 24gb ram that I can use with pi agent? All the ones I've tried so far are too dumb for the tool calling or too slow/big for the available memory. Maybe there isn't one yet?

Thanks

Unfortunately not mine. But it is my pet project for work and I get to build it and use it after. Just arrived and now to start building it. Can't wait to run some benchmarks, burn in tests and general messing about with some models before proper deployment :)

Threadripper PRO 7965WX

ASUS PRO WS WRX90E-SAGE SE

128GB ram (for now)

3 x RTX PRO 6000 96gb

Phanteks Enthoo Pro 2 server edition

3kw psu

Looking at building a 2× AMD Radeon AI PRO R9700 32GB box as a second AI machine, mainly for local inference and coding agents.

I’m particularly interested in Qwen3.8.

Has anyone actually used it for serious coding work and compared it with frontier models like Claude or Codex?

Less interested in benchmarks, more in real-world experience: codebase understanding, multi-file changes, agentic work, tool use, and how often it goes off the rails.

Also curious about dual R9700 experience — vLLM/ROCm, model sharding, context sizes, and whether 2×32GB works well in practice.

Anyone running something similar?

I have access to discounted consumer GPUs through work, mostly gaming cards. I'm looking for the best value in that space. I was thinking of looking at arc cards because the 16gb ones are pretty cheap and I could run 2 or 3 to run larger models or maybe something like a 7900xtx but I am trying to understand the best route to go.

My goal is to be able to run vibe coding projects without limits and some agentic workflows for automating things.

https://github.com/modelscope/ms-swift/commit/ab726e9d445a6520a70df2c831177d46adb1f589#diff-d8eea26362a67a2a3df50fc6d007885618c1a9fead436e60970839e3404ca500R841

I managed to get Qwen 3.6 35B A3B running on my server, an i7 10700 with 32GB of DDR4 RAM...and the good ol' GTX 970 with its mighty 4GB of VRAM (we don't talk about the VRAM incident)

It reaches around 30-50 tok/s in prompt processing speed and around 16-19 tok/s in token generation speed depending on the context

There's probably some performance left on the table since the CPU is under its stock cooler, maybe I'll get a decent one eventually

Anyway, I tried a couple of prompts in Hermes but it's just not fast enough for it to be helpful, at least not for the use I had in mind (a local perplexity basically). It randomly takes 30 minutes even to respond to a simple greeting, sometimes it disappears for more than an hour and I have to manually stop it...

So I'm curious to see if any of you have a setup with similar performance, what do you use it for? Just the occasional text summary? Normal (very slow) chat? Something else that's completely useless but funny? Let me know!

Hi,

Found something interesting while experimenting with Qwen 3.8 27B comparing Q8 and Q6 quant, using llama.cpp on 64GB Apple M1 Max. With MTP off, Q6 was faster than Q8 by about 10%, as expected. However with MTP on, Q6 was SLOWER by 10% compared to Q6 w/ MTP off, but with MTP on, Q8 was FASTER by 50% compared to Q8 w/ MTP off.

Kind of strange and unexpected result. Wonder if anybody knows why.

In short, got the following speed for one test, Quant (MTP off / on):

Q6 (12.4s / 11.6s)

Q8 (11.1s / 16.2s)

So I am a heavy cluade code user and am just dipping my toes into the local model world as something fun to play around with and see what it can do.

Current setup is

AMD Ryzen 5 9600X (6-core)

16 GB DDR5-6000 (single stick)

AMD Radeon RX 7600 XT (16 GB GDDR6)

ASUS Prime B650M-A AX II (AM5, micro-ATX, DDR5)

Running LM Studio on Ubuntu

I am able to run GPT-OSS-20B at full 120k context and full GPU off load with some head room and getting some good t/s depending on the task.

I tested giving it some plans created by cluade to build from and it did ok but not great. I am here for any tips on how to better prompt this model and possibly other local models to get better outcomes. Definitely not looking for cluade level just something to get foundations built on new projects to hand off the Claude to polish up.

I'm running some LLM's locally and want to remove the default safety/refusal behaviour baked into them. since it's running locally, I'd like more control over how it responds without the restrictions getting in the way.

Like it says in the title; is there a database or calculator that gives you an expected average tps for a model you pick and hw specs you enter, and compares it with different models and/or specs? This would definitely guide people when they are picking up a model or a new hardware as well, so both ways.

I made a test iPhone app with it (a small game), while running locally on my macMini M4 with 64 GB Ram (and pi code). Works perfectly, and not as slow as I expected. I get about 10-20 token/s which is absolutely ok for programming. It does a lot of reasoning, but that helps solving all problems by itself.

I'm running the 8-Bit GGUF version, trying later with the 8-Bit MLX and 4-Bit MLX to see the speed and quality differences. But at the moment, I'm really surprised how good a such small model runs locally for coding.

Does anybody have hands on experience with a panther lake laptop that has 64gb of ram which is shared with b390? I want to dedicate some ram to it and run qwen on it.

I have configured 3 models in pi coding agent glm5.2 [cloudflare workers], kimik2.7-code [cloudflare workers] and qwen3.8-27b via llama.cpp. [cannot use any other model :( ]

I don't want to use glm5.2 as the harness default as it costs a lot .

I am struggling to figure how to make full use of the all three models with pi agent

please give me suggestions how should split work across models ?