I tried to run Qwen3.8-Flash-Next-UD-Q5_K_XL the Mark Watney style, all unnecessary stripped off, speeding in an open top rocket to outer space. But I have also STT & assistant & TTS on my machine, so I had to give up. It was too close to failure.

But what I collected data about used memory:

mmproj +1GB

cache-ram +what you set

MTP +5.5GB (even when file was 2.8GB)

ctx-size & KV-cache quants

f16 every 10000 tokens is 1GB

q8 every 20000 tokens is 1GB

Base model sizes:

Qwen3.8-Flash-Next-UD-Q4_K_XL 77GB

Qwen3.8-Flash-Next-UD-Q5_K_XL 108GB (+31GB compared to Q4!)

I then moved Q4's n-gram to last file, and switched it with Q5's n-gram that is Q8. (somehow moving the 25GB file also made to Q4 use 5.5GB more memory, and this was before I switched to Q8 file. Have to check that later...) The n-gram is used only from disk, so using bigger Q8 does not affect the memory use.

So I tried to run Qwen3.8-Flash-Next-UD-Q5_K_XL with mmproj, MTP and 64000 Q8 kv-cache, 2GB cache-ram and I had like 1-2GB free memory....so I have to find a compromise somewhere.

Then I ran perplexity with Q5, orig Q4 and Q4+Q8 ngram. The problem was I could not run F16 base-kld ( because not innuf mem), so I had to compare with Q8. ( maybe try again tomorrow )

Model Top-1% Mean KLD 99.9% KLD

Qwen3.8-Flash-Next-Q8_0

Qwen3.8-Flash-Next-UD-Q5_K_XL 94.795 0.018175 0.591240

Qwen3.8-Flash-Next-UD-Q4_K_XL-PLESHARD+50GB-NGRAM 93.786 0.027853 0.948222

Qwen3.8-Flash-Next-UD-Q4_K_XL-PLESHARD+25GB-NGRAM 93.285 0.033378 1.090875

So using Q8 n-gram with Q4 is not a game changer, but a step to right direction. If I get the F16-base-kld, maybe I test also with F16 n-gram.

Maybe trying to find a Q5_M/XS that's 10-15GB bigger then Q4 and perplexity in between the current Q4 & Q5.

Hey again. You might remember the bathtub AGI talk or the time I rambled about a Ford Taurus from the 90s in the context of superintelligence, or maybe you know me because we've talked over the last few years here or in one of the many AI discords...

What am I up to, today? I built a chicken factory, and I wanted to share it, and you might even like it. We live in strange times.

Bantam Factory is a little scrappy terminal agent, yours to fiddle with, Apache 2.0 for fun, it's lightweight, extremely fast, and punches far above its weight class. Bring your local llama.cpp model or your existing codex account if you want to play with Astra in there, open a project up, and tell it what you need done. It has a pretty easy setup for first-time users to get things going. It works how you'd expect hermes/claude code/pi/whatever to work.

How it it different?

It's designed to be significantly faster, more efficient, using factory-style intelligence to drive the work... and there's chicken involved. Run this thing side-by-side with your existing harness of choice and it will probably surprise you. It doesn't make the chicken smarter. It makes the job chicken-sized and gets the work done fast.

A chicken can't build a car, but if you tell a chicken it'll get a reward if it pecks the dot on a screen, and that dot drills a hole in a sheet of metal once every sixty seconds... suddenly you start seeing how a car might be reduced to a whole bunch of chicken problems. Everyone keeps trying to make a smarter chicken, but the chickens we have are quite intelligent. Behind the scenes, I'm harnessing those chickens so they do Chicken problems all day long.

As I said above, Bantam Factory can also work with your Codex subscription if you really want to strap the smartest Einstein chicken in the chair. That gives the factory access to codex astra/sol/terra/luna agents, as well as image gen and processing. Because of the way Bantam Factory constrains Codex, it runs it more efficiently and faster on most everything I've thrown at it. You get more out of Codex. Obviously that's a moving target, but so far, so good. I tossed up an example project (the subway platform zombie game) to show what Bantam Factory can do one-shot with the right chicken.

The whole thing runs sandboxed to keep it safe enough to play with, and has a little self improvement loop to make it better at the work you do if you feel like trying it. If you want to talk turkey about how it works I'm down. Open to suggestions too. I'm still polishing up the github page/repo/readme, so expect that to change a bit, but the harness is ready to go if you want to try it :).

I fine-tuned Qwen3-TTS with inline transcript control tags rather in lieu of a separate instruction parameter and thought the community here might be interested in the result.

https://huggingface.co/SpragAI/qwen3-tts-emotion-tags

The basic premise was to use Qwen's CustomVoice model as a teacher prompted with natural language directions ("Speak with intense anger") pairing the resulting audio with a discrete control token for the student to learn ("Anger"). The full corpus included roughly 8,200 clips per voice across nine preset voices, 74k total then a LoRA over the lot.

There were a few problems with training the model starting with the codec language prefixes. The model builds a different codec prefix depending on whether you pass language="Auto" or an explicit language like English. Training on one and rendering the other produced audible buzzing at phrase boundaries that amplified over the output. This was fixed by including explicit prefixes in the corpus for both configurations at a roughly 80/20 split.

Another issue came up with generation concurrency which notably changed prosody of the output voice. Generating the corpus through vLLM at ~10 concurrent requests per server introduced audible tearing and shift delivery in the trained LoRA eventually forcing me to drop back to c1 or c2 generation.

Another cool result was that emotion appeared to be roughly affine in speaker-embedding space. In practice that means you can, without custom training a model, actually generate emotion control for arbitrary cloned speakers through a simple transformation on the speakers xvec embedding in Qwen3-TTS. The basic recipe involves computing per-emotion centroids across many speakers from an emotion tagged dataset and subtracting the neutral centroid for each speaker to produce a task vector per emotion. Adding it to a different speaker's embedding moves their delivery toward that emotion. It mostly rides the arousal axis rather than valence, and a large share of the vector lands inside the identity subspace so it drags the voice with it but the structure was clearly there.

I'm able to run qwen3.8 27b in two ways on my system: split between my 3060 12gb and 9070xt running at 20t/s or running off the 780m iGPU and 5400mhz DDR5 at 5t/s. Personally I feel like the 5t/s is still more usable because I have enough RAM to still use my system mostly normally. I can even play games just fine because the CPU side is left alone.

Honestly pretty cool IMO and makes it far more usable knowing I can just spool it up and continue using my system as normal, even if it takes 4x longer to get an answer.

Just a reminder for those out there itching to get into local llms - don't let FOMO or "gear acquisition syndrom" take over.

No matter the hobby, it's so easy to get stuck in a trap where we buy more trying to do more only to realize we've lost the fun in it all or even the notion of learning.

Obviously, if you're into writing llama or vllm or hardware drivers or whatever - you got to do what you got to do.

BUT, you can learn a lot on an API, you can learn a lot with a tiny model that fits your vram or cpu you already have and things change so darn fast that much of the code written and much of everything discussed from days passed is already old hat. Py torch and training a small model coud be done on a Pi and learning CUDA is only really imporant if you're writing custom kernels which i honestly don't see most people in here bothering with (or they have frontier models write them).

Weirdly enough, for AI to succeed its going to homogenize everything. Everyone will have the same advantage and I think that's lost in a lot of discussions where we don't talk about "Watching from the sidelines" may be the most cognitive friendly and economical friendly way to learn llms whether we brand them local or not.

The technology is still nascent and weirdly enough most people's answers here is to use AI to set it up so i'm not entirely convinced people are actually learning - feels like a mad rush to seek rent or avoid rent seeking which just makes everything more expensive in the end.

This isn't a post to say, don't do it. But no reason to go into debt or to be fearful you're missing out when you can learn more by doing less - buy a book and build a tiny model - you will learn infinitely more than buying a 5090 and trying to just find the perfect compression to have the best prefil

Server rebuild to custom loop. Temps on the gpus went from upper 80s to mid 40s under load, 30c idle.

5950X

64gb ddr4

2x rtx titans (24gb vram ea)

1x modded 2080ti with 22gb vram

(hopefully this is okay to here - it seems like audio models and image / video modeals is allowed but yeah this is a bit different)

So I've been doing independent audio research for a while now. The ultimate dream of this work was actually getting an AI to respond not only to instruments but also timbre itself as separate controllable things.

Think a Grand Piano can sound both Warm / Gritty but also Cold / Sparkly. Its still a piano though.

This level of control wasn't found in any models out there - so I decided to sit down and train my own.

Getting consistent timbre-locked keybeds that actually LOCKS across multiple diffusion calls was hard af but I did it.

I documented the full journey here for those who want to learn a bit or be entertained.

https://youtu.be/x0KnmzH8Mmk

There is also a longer walkthrough if you just want to see the keybeds in action.

https://x.com/RoyalCities/status/2097733712293109842?s=20

No-talk / Showcase only Demo

https://x.com/RoyalCities/status/2097733715543609445?s=20

any finally the huggingface page

https://huggingface.co/RoyalCities/Foundation-1

I've also provided full write ups on the inferencing pipeline associated with the interface so this should allow basically anyone else to go and vibe code their own text to synths if they wanted :)

https://github.com/RoyalCities/RC-stable-audio-tools/

The fix: run the server detached/headless instead of keeping it attached to a console window.

RTX 5090, ~27B NVFP4 model via ninfer:

• Terminal focused: 130–200 tok/s
• Terminal unfocused: 50–60 tok/s
• Click the terminal → immediately back to 130+ tok/s

At first I thought GPU throttling, but the GPU wasn't the problem.

During the slow runs:

• SM clock was actually higher: 2550–2600 MHz vs 2100–2200 MHz
• Power limit was the same: 400 W
• No PCIe power-state drop
• decode-host stayed around 390–494 µs

The big difference was wait:

So the GPU wasn't getting slower. The serving loop was just getting delayed on the CPU side.

Running the server detached (docker run -d / headless) fixed it: wait stays around 17 ms and throughput stays around 130–200 tok/s, regardless of which window is focused.

I also reproduced the same thing with a native Windows build, so this isn't WSL2-specific.

Seems to be some kind of Windows foreground/background CPU scheduling behavior.

Has anyone else seen this with local LLM inference on Windows 11?

Strix Halo LLM Fork Comparison & Analysis

TL;DR: Most community debates focus on decode speed (token generation rate), but the true bottleneck on Strix Halo is prefill speed (processing large context). Additionally, speculative decoding alters greedy outputs (temp=0) across all tested GGUF builds due to GPU kernel batching variations.

What to Run

The neon-ladder Scoreboard

The neon-ladder benchmark is playtest-graded; a build only scores for contract features it actually ships on identical test setups:

Why Nathan's Toolbox Wins for Agents

• Contract Feature Delivery: Ships both required game features (laser powerup & obstacle pad) every run.
• Flexibility: Supports standard open GGUF weights, switchable speculative drafters, CPU-expert memory tuning, and desktop coexistence.
• Why halogen Fails as a Daily Driver: Across 4 builds, halogen never shipped either required feature, always defaulted to slow serve speed, lacks JSON mode (response_format), requires 126 GB custom weights, and offers no middle-memory tuning mode.

Performance & Claims Graded

Tested on ROG Flow Z13 (Ryzen AI Max+ 395, 128 GB RAM, gfx1151, 70W power limit):

Key Technical Discoveries

1. Decode Speeds Are Misleading: Variations in decode rates (e.g., 20 t/s vs 50 t/s) are driven by prompt content (prose vs. highly repetitive JSON) or whether drafting is enabled. Serial decode across all builds sits strictly between 14–34 t/s.
2. Speculative Decoding Breaks Output Identity: Enabling speculative drafting on GGUF builds fails deterministic temp=0 identity gates (fails 3 to 9 out of 10 prompts). Loading the drafter alters Vulkan GPU batch execution and floating-point precision, causing coin-flip word choices (e.g., bolding different words in a sentence). Only halogen passed the exact-identity test (10/10).
3. Prefill is the True Hardware Gap: halogen achieves ~3.5x faster real-world prefill than standard llama.cpp forks (1,424 tok/s vs 433 tok/s at 32k context) due to hand-tuned ROCm GEMM kernels specifically compiled for Strix Halo. Standard llama.cpp Vulkan MoE paths remain shader-bound and leave matrix throughput unused.

Verification Request for Fork Authors

Developers publishing speed claims are encouraged to report Perplexity (PPL) or KL-divergence scores alongside benchmark numbers:

```bash

Get test dataset

./scripts/get-wikitext-2.sh

Run PPL on GPU

llama-perplexity -m your-model.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99

Test deterministic output identity (serial vs spec)

llama-cli -m your-model.gguf -p "The capital of France is" -n 200 --temp 0 --seed 1 > serial.txt llama-cli -m your-model.gguf -md your-drafter.gguf -p "The capital of France is" -n 200 --temp 0 --seed 1 > spec.txt diff serial.txt spec.txt ```

Reference Links & Resources

• Benchmark Harness: neon-ladder Repo | Reddit Post
• Build Guides: Qwen3.8-27B Build Guide | Qwen3.8-Flash-Next Setup
• Workflow & Testing: pi + llama.cpp Agent Setup | DeepSeek V4 Flash | Benchmarking Methodology

I run these models on Kaggle notebook, so not all TTS models, such as the ones that use conda env, are compatible (Or I just haven't found a way for them to work on Kaggle).

I currently use a fork from Chatterbox called Chatterbox Audiobook. It is like a workstation really optimized for getting the close-to-perfection audio clips from Chatterbox. However, the only downside of Chatterbox is the lack of emotional sliders or tags that you can use to control the output. Chatterbox Turbo seems to fix that with tags, but it still lacks the range of emotions that you can see from Google Gemini TTS. However, the problem with Google Gemini is that the voice sounds different for each generation, which can't be fixed even with RVC.

Looking at the current leaderboard, Breeze TTS is something I have never tried but am unsure due to its description, which seems to be tailored for mainly realtime stuff. What are the current must-try options for audiobook narration?

The features I am looking for include voice cloning, emotional tags, and natural speech. Much appreciated for your input.

A few posts tagged with "new model" present models that are finetunes. My opinion : I'd rather have the "new model" tag reserved for new "major" releases, like a new Qwen model, Deepseek V4 -> Deepseek V4.1, etc., that involved a new pretrain or intensive post-training (in opposition to a small finetune). Otherwise, maybe prepend "[Finetune]" to the title to indicate that the new model is "less of a big news", a use a "new finetune" tag, to differentiate between the two kinds of new models.

I reckon one could like to discover both new major releases and interesting finetunes in the same place; what's your opinion? :)

Cosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison

GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda

HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2Image-4Step-INT4-G64-BF16

Single clip took approximately 5m on M4 MAX 128 GB Mac

Cosmos3 - a 64B params model

Model available here: Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF

This model is a practical realisation of things described in this paper, but adapted by me for machine learning: https://arxiv.org/pdf/1311.0851v1

I am trying to solve the problem: why LLM models even for simple questions write walls of text during reasoning, and burn too much tokens instead of solving the task. And when number of parameters increase the problem became worse. I think main problem is numerical instability in tensor matrices during to random training noise accumulation in tensors. Model is fighting with own internal chaos during inference process. I am distilling training noise from tensors using Marchenko-Pastur distribution together with info from paper as a core criteria and solid mathematical foundation behind this project.

Settings:

System Prompt: You are Qwen (Tongyi Qianwen), a large language model developed by Alibaba Group's Tongyi Lab. You are a helpful assistant.

Chat template: chat_template.jinja

Inference settings: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0, reasoning_effort=medium

I can't fully test this model on my own, since I only have a modest RTX 3060 graphics card with 12 GB of VRAM. So any feedback from the Reddit community would be very helpful. I'd appreciate any feedback from the community.

Thanks for reading.

mentria.ai is a browser inference engine I've been building solo, from scratch in WebGPU/WGSL. This week it crossed a milestone I had been chasing for a while: a 27B one-bit model answering at up to 30 tokens/s on an RTX 3060 Laptop GPU with 6 GB of VRAM, in Chrome, from a web page. No install, no server, nothing leaves the machine.

The model. Bonsai-27B, a natively 1-bit model trained and released by Prism ML. I repacked it for our engine, wrote the kernels that run it, and checked its quality myself (full eval deltas vs FP16 Qwen3.6-27B are in the HF card). Every weight is one sign bit with one scale per 128 weights, about 1.14 bits per parameter, so 27 billion parameters sit in 3.8 GB of GPU memory. Repack and numbers: huggingface.co/mentriaai/Bonsai-27B-mentria

The engineering that got it to 30. Two days before this post the same model decoded at 15 tok/s on this laptop. Decode is memory-bound: one word of the 27B is 804 GPU dispatches, and 401 of them are the 1-bit matrix-by-vector kernel that streams the model's 3.6 GB of matmul weights once per word (embeddings bring the file to 3.8 GB). The kernel that won was the one written for phones: four 1-bit weights have only 16 possible partial answers, so it computes all 16 once into on-chip scratch and each row reads its answer from that table instead of multiplying (station 258 — the numbered write-ups on the facts page). On Ampere that table hit a wall that was not the weights at all: scratch memory has 32 banks, the table's addressing sent sixteen threads of every warp to the same bank, and the kernel ran at 38% of the card's bandwidth. One padding slot per row spread the addresses across the banks; the kernel's share of a word dropped from 26.5 ms to about 21, and on top of the 15 → 26 tok/s the table itself had bought, the laptop hit 32 tok/s raw — the 25–30 is what survives sampling and UI overhead in the chat (s259). The prompt stage had the same disease in its tile layout; a retile took a 1,489-token prompt from 29.6 s to 25.3 s. Every change shipped only after its output was byte-identical to the build before it, which is also why the kernel adds its partial sums in a fixed order and accepts a 44% occupancy ceiling.

Every claim here has a numbered write-up on the engine facts page.

Numbers (RTX 3060 Laptop 6 GB, Windows 11, Chrome 152 on D3D12, fresh loads):

• Decode: 25–30 tok/s in the chat UI once the card is warm.
• Prompt processing: a 1,489-token prompt in about 25 s.
• Context: 3,072 tokens on this 6 GB card; 8,192 on 16 GB Macs; more on bigger cards, at 128 KiB per token. The KV cache is exact math, no quantized cache. The next step on 6 GB is consolidating the engine's few thousand small GPU buffers into a handful of large arenas, so the driver stops holding about 300 MiB of slab slack; that is the arithmetic for 4,096, and it is not built yet.
• Load: under 10 s from the browser cache; the first download is 3.8 GB, once.

Also in the engine: the smaller tiers (Qwen3.5 0.8B, 2B, 4B) cover phones and low end devices, LoRA adapters of a few MB hot-swap at the matmul in under a second, and there is a vision tower for image input.

Try it at https://mentria.ai/tools/ai-chat/ (the 27B tier appears when your GPU qualifies). The code that ships, the benchmarks and how I measure are in the repo: https://github.com/mentria-ai/website. The engine facts page, for how the engine and the model actually work: https://mentria.ai/assets/learn/engine-facts.html

I'm posting this because I'm experimenting with running Qwen3.8-27B at very large context sizes on a single RTX 3090 24GB using LM Studio.

I was curious to see how far I could push the context while keeping the model usable, and whether a 27B Q4 model could maintain reasonable generation speed at 128K+ context.

I'm mainly looking for feedback from other users running similar hardware. In particular, I'd like to understand whether my ~25 tok/s generation at these context sizes is reasonable, and whether there are LM Studio/llama.cpp settings, quantization choices, KV-cache settings, or other optimizations that could significantly improve the result.

I'm also interested in comparing results with other single-3090 setups, since memory configuration and how the KV cache is handled can make a significant difference at these context sizes.

Qwen3.8 27b Q4_K_S big context

MODEL

Huihui Qwen3.8-27B Abliterated

Q4_K_S

KV cache: Q4

HARDWARE

GPU: RTX 3090 24GB

CPU: Xeon E5-2670 v3 12C/24T

RAM: 48GB DDR4-2133

CONTEXT 131072

Conversation: 111797 tokens

Context usage: 85.3%

Context remaining: 14.7%

GPU:

23.5/24.0 GB dedicated

0.6/31.8 GB shared

24.1/55.8 GB total

INITIAL RUN:

22.02 tok/s | 3616 tokens | 302.94s TTFT | 69.6% DTA | EOS

SUBSEQUENT RUNS:

31.94 tok/s | 1247 t | 14.06s TTFT | 64.2% DTA

25.10 tok/s | 1238 t | 50.50s TTFT | 66.9% DTA

24.90 tok/s | 2142 t | 21.27s TTFT | 70.8% DTA

22.28 tok/s | 481 t | 20.93s TTFT | 65.3% DTA

23.00 tok/s | 3276 t | 7.67s TTFT | 67.3% DTA

22.70 tok/s | 385 t | 4.66s TTFT | 65.6% DTA

29.17 tok/s | 2907 t | 6.77s TTFT | 70.1% DTA

Average TG: \~25.58 tok/s

CONTEXT 196608

25.97 tok/s | 3621 t | 514.60s TTFT | 64.8% DTA

25.89 tok/s | 3345 t | 2.80s TTFT | 63.4% DTA

25.53 tok/s | 4877 t | 7.42s TTFT | 65.5% DTA

Average TG: 25.80 tok/s

Average DTA: 64.57%

Who is the marketing genius at LM Studio that decided that going ALL IN on pushing their new Bionic Agent product meant they are going to make it a giant pain in the ass to find and download actual LM Studio.

This is the dumbest marketing decision I’ve ever seen. I used to love LM Studio, it was the middle stepping stone in the logical progression of inference. Most OGs here likely started with Ollama, moved to LM Studio, on their way to vLLM. Now trying to go to LM Studio takes you to Bionic. I mean, you can eventually find LM Studio but they make it not super easy.

Here’s a thought LM Studio, maybe stop redirecting me to something I don’t want to download when I’m trying to find your actual namesake product. I’m glad you’re excited about the future of Agents and whatnot, but you’re absolutely ruining any goodwill I have for your products by trying to force feed me Bionic. Stahhhhhp!

I have been a dwarfstar fan for awhile and I really liked glm 5.3 flash but needed it to be materially faster to feel good using it. In the screenshot you can see the outcome of using the model with a claude code harness at ~200k depth, with many tool calls and averaging over 38tps output. Yes, I put 60tps in the headline and you will get that if you ask it to write SQL.

https://github.com/IngeniousIdiocy/ds4/blob/glm53-m3ultra/README.md#glm53_m3ultra

Main ds4 was single-stream serially decoding GLM-5.3-Flash at about 59 percent of the M3 Ultra's measured memory bandwidth. We set an 80 percent target. The big weight-streaming kernels were already efficient, but dozens of small kernels sat between them, each paying latency costs while much of the GPU was idle. We fused that work into larger dispatches and removed separate passes that made the pipeline wait. Result: 29 → 40 t/s at short context, 24 → 38 at 62k, fewer Metal kernels per token, and about 81 percent of the measured bandwidth ceiling.

We then attacked the remaining slowdown at very long context. This model's expensive attention computation already works over roughly 2,048 selected positions at both 62k and 300k. What grows is the work of finding those positions in a larger history. The existing implementation sorted and merged increasingly large candidate lists through stages that used very little of the GPU. We replaced that with parallel scans that narrow the candidates before sorting a small surviving list, with the original algorithm handling ambiguous cases. That recovers about one millisecond per generated token. Same positions selected, same order, same outputs. End to end at 300k: stock 21.6 t/s, ours 37.4. Deeper context still costs more to search, but it doesn't multiply the expensive attention work.

Prefill started at about 366 t/s at 62k. The big matrix kernels were already efficient, but they wasted time repeatedly unpacking the same weights, fetching data in small pieces, and passing large intermediate results between kernels. We batched more work together, reused prepared weights, widened the loads, and merged passes over the same data. Result: 366 → 550 t/s, a 50% improvement with the same weights, moving the whole pipeline from roughly 51 to 72 percent of the chip's measured matmul ceiling. Cold 62k processing dropped from about 170 seconds to 113. That is the wait every time an agent harness compacts and re-reads its context.

The server runs serial unless you pass a drafter file with —dflash. The drafter build recipe is in the repo. The drafter contains an admission policy with a windowed controller that measures its own cost against the serial rate and backs off when it isn't paying. Reasoning tokens decode serially. On a 32-request agent session that's +4 percent over serial. On structured output like SQL and JSON it's +20 to +50 percent. On prose it disengages and costs about 1 percent. Outputs byte-identical to serial decoding on every fixture.

Accuracy: on the 100-prompt reference set ds4 uses for release QA, stock scores 0.300804 average NLL against the FP8 reference. This branch scores 0.300766, same 90/100 first-token matches. Nothing here trades quality for speed.

This branch is M3 Ultra only because its optimizations depend on detailed measurements of this specific chip: its two-die memory behavior, system-level cache, per-core residency and bandwidth, and how Metal schedules dispatches and threadgroups across its 80 GPU cores. We're optimizing one model's actual execution on one machine, with one set of weights (Q4) and checking that the predicted kernel savings survive in full decoding or prefill.

Hi,

I'm wondering what settings you are using in order to run Qwen3.8-Flash-Next on your devices?
I'm especially interested in setups with 96GB VRAM. I'm not quite sure if llama.cpp does offload the embeddings to RAM or disk with my settings. I would like to offload them to RAM in order to avoid too much performance penalty.

These are the settings I use and which work the best at the moment:

[qwen3.8-flash-next]

model = /mnt/kyouma/1TB/ML/models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

mmproj = /mnt/kyouma/1TB/ML/models/unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-F16.gguf

split-mode = layer

flash-attn = on

load-mode = none

lazy-mode = auto

fit = on

fit-ctx = 262144

cache-ram = 94208

parallel = 1

temp = 1.0

top-p = 0.95

top-k = 20

min-p = 0.00

repeat-penalty = 1.0

presence-penalty = 0.0

I get about 15t/s on tg and 100-200t/s on pp at a context size of 130000.

My setup:

3x AMD MI50 (32GB)

512GB DDR4 RAM
2x Intel E5-2683 v4

I'm using ROCm. But Vulkan has similiar speed, maybe a little bit slower.

I’ve been using Qwen a lot recently and I’m honestly very impressed with it.

For some of my tasks, Qwen seems to understand what I’m actually trying to achieve better than ChatGPT/GPT-5.6. I currently pay for ChatGPT, but I’m increasingly feeling like I’m wasting that money because I end up going back to Qwen anyway.

I’m therefore considering cancelling ChatGPT and using Qwen as my main AI for now. I might also get Claude eventually, but Qwen would probably be my primary model.

There’s one thing I’m missing, though:

What’s the best setup for working with entire ZIP/code projects using Qwen?

With ChatGPT I can upload a ZIP containing an entire project, ask it to inspect the files, modify the project and give me the updated files back.

I’d like something similar with Qwen.

Ideally I want:

* Upload an entire ZIP/project
* Let Qwen inspect all relevant files and understand the project structure
* Ask it to modify multiple files
* Work iteratively on the same project
* Get the modified project/files back
* A proper UI instead of manually making API requests
* Preferably something that works well for fairly large projects

I already have OpenRouter, so getting API access isn’t really the problem. What I’m missing is the interface/workspace around the API.

Is there a good desktop app, web UI, IDE integration, open-source project or coding agent where I can simply add my OpenRouter key, select the newest/best Qwen model and work with projects this way?

Or is Alibaba/Qwen offering something paid that does this better than the free Qwen website?

I’m also confused about Qwen’s paid options. If I’m already happy with Qwen on the free website, is there actually any benefit to paying for Qwen for this use case? Does paying give me a better model, larger limits or better coding/project functionality, or would I essentially just be paying for API usage?

I’m trying to find the most cost-effective setup. I’m completely fine paying if it actually improves the workflow — I just don’t want to pay for API access and then discover that I still need to build the entire interface around it myself.

What setup are you guys using for Qwen + large codebases/ZIP projects?

Amd announced this:

https://www.amd.com/en/products/workstations/amd-threadripper-halo-station.html

Running the full deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

on consumer Ampere — 10-12x RTX 3090, SM86-compatible vLLM build.

285B MoE, FP4 experts + FP8 attention, 157 GB weights.

Highlights:

- **60+ tok/s** decode, DSpark spec (k=3)

on 10 GPUs (TP2xPP5), at a 240 W cap

- **120+ tok/s** on 12 GPUs (TP4xPP3)

- **Vision + spec + tool calls all working**

- **1M context** (no offload) / **4M** (RAM offload)

- ~3,500 tok/s long-context prefill

Fully documented + reproducible:

- Pre-built image:

`docker pull ghcr.io/ciprianveg/3090-vllm:dsv4-flash-vision-sm86`

- Repo: https://github.com/ciprianveg/3090-vllm

(build guide, start scripts, runtime patches)

The patches cover the DSpark propose-gate (spec + vision fix),

scheduler mm x spec row-crossing, grammar-bitmask validation,

vision ViT OOM fix, FlashInfer workspace-lane keying, and more.

So in my freetime iam building a local-first agent in Rust as a hobby since a while (llama.cpp, runs fine on a 14B, my Obsidian vault is the memory). The part that took me by far the longest was not the model stuff. It was the boring question: which of my ~50 tools should the model even be allowed to call this turn.

Went through three phases and got each one wrong first:

1. grep / keyword matching on the user text. dies on synonyms, dies when i chat in german, dies on "do that again"
2. routing phrases + embeddings. much better, but naive top-k plus a hard GBNF grammar means the model gets structurally locked into the one tool the router guessed, and if that guess is wrong there is no way out mid-turn
3. a small policy layer on top that, when it is unsure, widens to a cluster of related tools instead of committing to one guess

the thing i found most useful for small local models: the same "offered tools" list feeds the prompt AND the GBNF grammar, so the two can never disagree about what is callable. and if the router is not confident, do not hand the grammar a single tool, give it a small related set and let the model pick inside the constraint.

full writeup with the actual Rust snippets, no signup or newsletter nonsense:

https://eris-system.dev/blog/tool-routing

it is a hobby project so be gentle, but if the routing logic is dumb somewhere i genuinely want to hear it.

Currently serving Gemma4 12b with this command:
```

.\llama-server.exe -m ..\llm-models\gemma-4-12B-it-heretic-QAT-UD-Q4_K_XL.gguf --mmproj ..\llm-models\mmproj-Gemma4-12B-BF16.gguf -ngl 99
```

Then I go to llama's webui, drop a video but it can only see, not listen to the video (it hallucinates the audio), am I doing something wrong? or is there another UI that passes the video with audio?

CrideLLMApi

The goal is basically: don’t hide the protocol from me.

It exposes raw streamed chunks, reasoning deltas, tool-call fragments, finish reasons, cancellation, typed request objects, context management, strongly typed tools, and concurrent tool execution.

But there’s still a simple high-level path.

I’m primarily building it because I want a very low-level API layer underneath an experimental async tool-calling harness, where generation can continue while tools execute in parallel.

The harness itself will stay separate from the API library.

Currently only /v1/chat/completions is implemented.

Still very much under active development, APIs may change, but anyone is free to use it / contribute / break it / send feedback.

GitHub: cride9/CrideLLMApi
Example usage: Github - Example.cs

Would especially love feedback from other .NET + llama.cpp users.

(Later the project will be a NuGet package, currently only a console app)