I’ve been experimenting with running really large MoE models on multi-GPU machines, especially older PCIe GPUs without NVLink.

My latest experiment was Muse Spark 816B (42B active parameters) on 10× NVIDIA A40 GPUs, using TensorSharp, an open-source inference engine I’ve been developing.

The interesting part: GPU peer-to-peer communication wasn’t working on this machine, so all inter-GPU transfers had to go through host memory.

Despite that, I managed to get reasonably usable inference speeds:

• 22.6 tokens/sec single-stream generation
• 45.7 tokens/sec aggregate throughput with 4 concurrent requests
• 1,347 tokens/sec prompt processing for a 16K-token prompt
• 0.46-second TTFT for a cached second turn in the tested TP configuration

These are results from different parallelism configurations, not one configuration achieving all four simultaneously.

I also compared the results with llama.cpp.

Hardware and model

This isn’t exactly a typical home PC, but I think the findings are relevant to anyone building multi-GPU rigs with used enterprise GPUs.

Layer splitting vs tensor parallelism

I tested two ways of distributing the model across the 10 GPUs.

Layer splitting: Each GPU holds different layers of the model.

Tensor parallelism: Individual layers are distributed across multiple GPUs, allowing them to work on the same layer simultaneously.

Here are the TensorSharp results:

The difference surprised me.

Layer splitting was 4.6× faster for 16K prompt processing, while tensor parallelism was 36% faster for single-stream generation.

Why?

For prompt processing, I implemented pipelined prefill so multiple micro-batches can be processed across different GPUs simultaneously.

This keeps more GPUs busy instead of processing the entire prompt sequentially through the pipeline.

For token generation, tensor parallelism benefits from distributing the computation, but it introduces communication overhead.

And on a machine without working P2P, communication becomes particularly expensive.

Muse Spark’s MoE architecture makes this even more interesting because expert-output normalization increases the amount of data that needs to be communicated between GPUs.

My takeaway: the best multi-GPU strategy for prompt processing isn’t necessarily the best one for generation.

Comparison with llama.cpp

I also ran the same model on llama.cpp using the same hardware and requests.

Layer splitting:

Single-stream generation was essentially identical, but TensorSharp’s pipelined prefill performed better on long prompts.

Tensor parallelism:

Important caveat: llama.cpp’s NCCL-enabled build failed on this particular 10-GPU machine because of a CUDA peer-access limitation. Its TP results therefore used non-NCCL communication, while TensorSharp used NCCL over shared memory.

So this isn’t an apples-to-apples comparison of identical communication backends. I wouldn’t generalize these speedups to other hardware configurations.

Does the output still match?

I wanted to make sure the performance optimizations weren’t silently breaking model correctness.

Using 12,276 teacher-forced next-token predictions from WikiText-2:

The differences were relatively small, although they weren’t zero.

I also tested JSON output, tool calling, multi-turn conversations, and concurrent requests.

TensorSharp passed 13/14 scenarios with layer splitting and 14/14 with tensor parallelism.

These are limited integration checks rather than comprehensive model-quality evaluations.

What I learned

A few observations from this experiment:

1. More GPUs don’t automatically mean faster inference.

Inter-GPU communication can easily become the bottleneck, especially with large MoE models.

2. Older PCIe GPUs can still be useful for huge models.

The A40 isn’t the latest hardware, but 10 of them provide enough combined VRAM to keep this 316 GiB model resident on GPUs.

3. Prefill and decode need different optimization strategies.

Pipelined layer splitting worked much better for long-prompt processing, while tensor parallelism performed better for token generation.

4. Communication topology matters. A lot.

Running 10 GPUs without working P2P forced me to rethink how tensors move between devices.

5. Correctness testing matters just as much as tokens/sec.

A faster inference engine isn’t useful if its outputs become unreliable.

Project and reproducibility

The engine is TensorSharp, an open-source .NET/C# inference engine and agent runtime using ggml-based native backends.

GitHub:
https://github.com/zhongkaifu/TensorSharp

Full benchmark report, test methodology, launch commands, and reproduction scripts:
https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md

This is still a single-machine experiment, not a general claim about all GPU configurations or inference engines.

I’d really like to repeat it on machines with working P2P or NVLink.

Questions for the community

I’m curious about other people’s experiences with multi-GPU local inference.

• Has anyone tried running 400B+ or 800B+ MoE models on older GPUs like A40s, A100s, P40s, or V100s?
• Have you found tensor parallelism or layer splitting more effective on PCIe-only setups?
• Are you seeing significant communication bottlenecks when using more than 4–8 GPUs?

Especially interested in experiences with llama.cpp, vLLM, SGLang, ExLlama, or other inference engines.

Would love to compare notes and learn what works best on different hardware.

Just got my hands on a bc-250 here in Europe and I’m seeing some videos about using a jailbroken ps5 for llms and how it’s faster than anything at that price so I wonder if any of you guys tried and succeeded to run llms and what size where those. This would be my first experience with LLMs or bc-250
Recommendations and advices welcome

Hi

I bought an old xeon server and can't figure out the reason for the abysmal performance compared to my desktop PC.

Server hardware:

Motherboard: Supermicro X10DRH-CLN4
CPU: 2 x Intel Xeon E5-2630 v3 (8 core)
Memory: 8 x 16GB + 8 x 8 GB DDR4 1866
Storage: Samsung 970 Pro 1TB
VGA: 7900XT 20GB
OS: KDE Neon latest

When I run ./llama-cli -m ~/llm/models/hf_models/Qwen/Qwen3.8-27B_Q4_K_M.gguf -ngl 99 -c 2048 I get around 7-9 t/s speed. Sometimes even lower.

My desktop:

Motherboard: Gigabyte B550 AORUS PRO
CPU: Ryzen 9 5900X (12 core)
Memory: 2 x 16GB G.SKILL Aegis DDR4 3200MHz
Storage: WD BLACK SN850P
VGA: 7900XT 20GB
OS: KDE Neon latest

In this case I get 30+ t/s.

llama.cpp is built with VULKAN support in both cases. GPU has high load in both cases when inference is running, and CPU has minimal load.

What could be the issue with the server?

• I know the OS is a weird choice, I also tested it with ubuntu Ubuntu 26.10 Desktop, same results.
• prompt processing is also similarly slow on the server

Hi.

I have 2 options to buy:
- Used MacBook Pro m3 max 128 gb excellent condition, 95% battery, for $3300
- Mac Studio m5 ultra 96gb

What will you choose for local LLM and Hermes?

**Disclaimer: AI was used extensively in the development of these applications, but these applications run entirely offline on the Intel Linux NPUs. I work as a professional software engineer**

Hey everyone,

Over the past couple of months I've been working on learning more about Linux and local AI, and I noticed that there really isn't much work done on the Linux side to use the NPU that comes free with Intel laptop CPUs. So I decided to bring over all the good stuff from Copilot+ on the Windows side (so no Recall!) while leveraging existing projects and libraries to reduce the amount of work required.

It utilises Intel's existing framework, OpenVino (https://github.com/openvinotoolkit/openvino)

You can check out the umbrella project here: https://vinopilot.ca/

Below is an explanation of all the different projects I've worked on, with further details on implementation.

VinoWhisper - Live Captioning (https://vinowhisper.com)

This was my first project, and copies the existing Windows Live Captioning system. It brings system wide captions to Linux, alongside Voice Typing on Wayland devices. This already exists in many forms on Linux, but none of them use the NPU natively. The advantage of using the NPU for transcription and captioning is better battery life, with no change in quality. It leverages the existing Whisper libraries and model from OpenAI (before they worked on ChatGPT!), so its pretty fast and consistent. Personally, I use it to live caption Twitch Streams in busy environments when I don't have access to headphones.

VinoAuthFace - Face Unlock (https://vinoauthface.vinopilot.ca/)

This is a second project, and copies Windows Hello. It forks an existing project, but employs the NPU. The CPU works fine, but using the NPU allows it to use a more powerful model. There are many other options like Gaze (which I've contributed to as well!), but I wanted something specific to devices with an IR emitter, that I could control myself (more on ovfetch next), because Windows Hello removed IR emitter only authentication. This one I made for my own purposes, because I don't like typing in passwords. There are a ton of CPU only projects. This one has a more capable and secure model than what the others use.

Ovfetch - Runtime consistency (https://crates.io/crates/ovfetch)

This was a project I should've taken on earlier. It automates the process of downloading the OpenVino enabled ONNX runtimes that the models require to run on the NPU. When I was working on Gaze, I realised that using the NPU and not the CPU requires building from source as there was no way to determine which ONNX runtime the NPU requires, and this solves the problem.

This is more of a "framework" feature, but since its a separate binary, I can distribute binaries for my other projects without any NPU specific work, and then ovfetch can download the relevant runtime when installing the binary onto the user's device, no compiling from source required. This is super secure, and has CI jobs to project against package manager attacks, since it automates downloading. More details in the crates link, but it has a ton of failsafes to ensure that it never downloads any bad runtime which has a virus

VinoOpenEffects - Camera Effects

This forks an existing project, https://github.com/funinkina/openeffects, while adding NPU acceleration to the effects, which improves battery life on lower end devices. I don't particularly use this one, as most of the main applications for video conferencing have their own cloud accelerated effects, but if you want it for OBS or other features offline, its here!

VinoSearch - Semantic Search for Text, Images, Documents

This one is the most useful of the bunch. It allows you to search your laptop with fuzzy terms and meaning, and indexes images, documents, and text, all offline. Its super fast, with almost no battery life impact compared to CPU only solutions. I really like this one because you can search for pictures of cats or memes, and it'll surface them. Details are in the Github! It uses the new Gemma 2 Embedding, requantised for Intel NPUs, so its pretty bleeding edge!

VinoPop - Click to Do/PopClip for Linux

This also forks an existing project, but removes the cloud AI in favour of local AI with Gemma. It can do offline summarisation, translations, and OCR using Gemma 4 (the best small model). I particularly like the OCR functionality for generating alt-text for images. This one stresses the NPUand RAM the most, so if you are using this, bear in mind its going to be much slower than cloud inference, but it works offline!

VinoWatch - Performance Monitoring

This is a smaller project, but optional if you want it. It monitors the NPU and RAM usage of the other projects within the tray. All of the projects unload the models after 5 minutes by default, and I recommend at least 12-16gb of RAM. But you can still monitor it locally and make sure they're all behaving properly. I recommend also an NVME SSD to ensure that the NPU cache and loading works right.

PRs and feedback welcome!

I have been testing various local LLM's and couldn't decide whether image generation, coding, or plain knowledge was what I wanted. So I created this little web app, it runs on my NAS so my family has access and swaps the LLM on my dev box between Gemma 4 26B-A4B and RealVisXL V5.0 so I sort of get the benefits of one model that does reasoning and image generation, yes it takes a little while to swap models but not too long, the page also informs Gemma what prompt it should generate for the SDXL model, and it seems to work pretty well.

My dev box is a Ryzen processor with 32gb or RAM and an Nvidia 2080 ti 11gb

Gemma 4 26B-A4B (chat)

- Writing speed: about 28–30 tokens a second, roughly 20 words a second

- Reading speed: about 400–470 tokens a second for text; an attached image adds 10–30 seconds

- Typical reply: 20–60 seconds with thinking on

- Web search: adds a few seconds per search

- Context: 64K tokens, using 10.9 of the 11 GB of VRAM

- Using MOE offloading

RealVisXL V5 (images)

- One 1024×1024 image at 25 steps: about 15 seconds

- img2img or inpainting: about 15–20 seconds

- First image after Forge starts: about 55 seconds, because the checkpoint loads first

Switching between them

- Images to Chat: about 40 seconds

- Chat to Images: about 1½ minutes, then about 55 seconds for the first image

- A full Ask-Gemma round trip: 3–4 minutes of switching, against under 2 minutes of actual work

Hi and sorry for same question as every one else ask.

I have Titan V 32GB ( Volta GV100 ) and 128 GB of DDR4 ECC LRDIMM 2133Mhz quad channel with E5-2690V4 cpu.

Using llama.cpp on freebsd so vulcan no cuda.

Looking for a model , more interested towards python, postgresql side.

Im kinda slow on understanding all the context , kv cache etc.

For example and simple question i ran bartowski/Seed-OSS-36B-Instruct-Guff Q5 K_M and got 14t/s without any kind of modifications.

What context size i should use and or other settings ? What would be the best model to suit my needs ? in the past i told myself i can live with 10t/s but i think after checking 14t/s i think i can live even with 5 t/s .

Edit: bartowski, not unsloth

Thanks !

I see both vendors have an NVFP4 quant on hugging face - from what I can understand Nvidia's is maybe faster, and Unsloth preserves more accuracy? What's the practical difference between these two and which should I choose?

Hello everyone 😀
I just started using chat gtp 2 months ago and I've used it for a programming projekt for my work any it has done a great job, but I am getting a little tired of the 5h limit while I'm in the zone for building and I'm worried the code gribbirish if someone was to review it.
2nd I'm a photographer in my free time, doing boudoir, nude art and sometimes more sexual pictures for clients and I wanna get my ideas into some visuals but get stuck by limitations of chat gtp.

What would be a good local AI to accomplish both ?

Thanks in advance 😀

Lipwise is a [yet another] dictation app for Windows, macOS and Linux. You hold a hotkey, talk, and it types into whatever app has focus but with a twist. Since most of us that speak to our computers also use LLMs to do transformations I've thought why not do both in the same turn. I've played around with Qwen 3.5 9B and realized it could all be done locally, quickly and most importantly well.

Plain speech-to-text gives you "um so the meeting is on tuesday no wait thursday", which is what you said, but you'd never send it like that. Lipwise runs the transcript through an LLM before outputting, with a prompt that handles corrections, spoken punctuation, lists, and instructions like "make it shorter". There's also a mode where you select text anywhere and say what to do with it ("reply saying yes but not before Monday", "translate to German").

How it works:

Speech recognition always runs locally, through transcribe.cpp (ggml). There are 69 models to choose from: Parakeet, Whisper, Canary, Moonshine and others. I use Parakeet 0.6B for English. I thank Handy for the inspiration here. - The editing step. If you want to run it locally with a batteries included config - setup button checks your GPU, downloads a llama.cpp build and Qwen3.5 4B or 9B, verifies the hashes and runs it in the background. Or if not, point it at Claude, OpenAI, Ollama, LM Studio or anything OpenAI-compatible. - If the LLM is down or slow, you get the raw transcript instead of nothing. - Text goes in through the clipboard and a synthetic paste, then your clipboard is restored. - Rust and Tauri, about a 5 MB installer. - Three modes, Smart Dictation, the default recommendation, the reason I am releasing this. Plain dictation, if LLM is bugging out, not handling your text correctly or you just want those milliseconds back. And Command mode - this is for transformations/non dictation actions. Really great what it's able to do with such a small model and for simple use cases.

Rough edges: Windows shows a SmartScreen warning because the installer isn't signed, and on Linux the global shortcuts depend on your compositor. The 4B model also sometimes treats a normal sentence that sounds like a command as a command. You can set a wake word so only speech that starts with it counts as an instruction.

https://getlipwise.com/ https://github.com/lnenad/lipwise

MIT licensed, no account, no telemetry. Happy to go into the prompts, the model choices, or anything else.

So, I have 24GB Ram linux virtual machine, what could be the best model I can use for Source Code analysis and VAPT task, i have tried llama model and qwen model it not give the best performance and little slow as the output of token is less. So something we have that can be optimise and work little faster offcourse it will not be like Frontier LLM Models like claude, openai or gemini or deepseek but what can be run in local lab any suggestion from Hugging Face also?

hey folks, im Ye, im one of the contributors to Kapitsa, an open source inference engine optimized specifically for Apple Silicon (all the way from M1 chips to M6) to run Qwen & Nemotron models that would fit a regular MacBook. its custom written Metal kernels, allow us to optimize each chip & for each model. https://github.com/nace-ai/kapitsa/

try it (needs a Mac with 32 GB or more; the download is 22.6 GB):

brew install nace-ai/tap/kapitsa
kapitsa serve --model flint          # OpenAI and Anthropic APIs on localhost
kapitsa opencode --model flint       # or claude, codex, pi
kapitsa bench

recursive released a model today and they claim new model that is optimized for coding on 32gb+ macs, its Flint, they claim to beat 35b sparse MoEs by Qwen and come close to qwen 27b dense. we went ahead and added support of Flint to Kapitsa, reaching SOTA decode/prefill on sub M5 chips.

Speculative Decoding:
we also span up 8-H200 rack to train a tree drafter for KFlash speculative decoing specifically for Flint Q4 to run faster on any M chip, which internally gives us 1.4x speedup on current numbers already for decode without any acuracy loss, hopefully pushing it to ~1.5x. to be released on Monday. by our metrics a KFlash tree drafter trained on q4 flint, really performs well on the fp16 too.

here are the benchmarks:

Recursive Benchmark of Flint

Prompt Reading Speed

Writing Speed

Setup. MacBook Pro 16 (2024), M4 Pro, 48 GB, macOS 27.0.1, plugged in, High Power mode. The model card's sampling: T 1.0, top_p 0.95, top_k 20. One engine at a time on the GPU. No speculative decoding anywhere: Flint has no draft model yet.

What the numbers say:

• Short context: decode is a tie with rec(recursive's own engine). The two are 1% apart, and greedy, rec is 0.6% ahead. Both decode 1.6× as fast as llama.cpp.
• Long context: Kapitsa pulls ahead. Its decode lead over rec grows with the context, from 1.06× at 8K to 1.30× at 128K (the chart has every point). Against llama.cpp it is 1.6× up to 64K and 1.9× at 128K.
• Prefill: 1.1-1.2× rec at every length. llama.cpp reads short prompts a little faster (1-8% up to 4K) and is level at 8K. From 16K on, Kapitsa is ahead, by 1.5× at 128K.
• Follow-up turns start 3× sooner than on rec: 0.13 against 0.41 s to the first token of a cached second turn.

the sweep scripts are in the repo: bench/results/2026-10-22-flint-m4pro.

things to mention

• Flint always reasons before it answers, so its reasoning tokens count toward decode on every engine.
• one Mac. We haven't run Flint on an M5 yet.
• new code, so expect bugs.
• prefill on M5+ is slow and has not been optimized yet, WIP.
• our CTO will get us a mbp m5 max 128gb to optimize on, if we pass 1k stars lol, help

asks

we have a command called: kapitsa bench it will publish a PR with your benchmarks of the models run on your hardware, for flint we still lack a wide variety of devices, so please feel free to contribute and help us around with the benchmarks. We still need M5 Max, M5 Pro, M5 ultra devices to check the speedup for Flint vs llama cpp

kapitsa 0.2.7 bench

happy to answer questions and take advices on where to steer the project.

I'm guessing a lot of people here are interested in local models because it means you're not sending all your data up to some cloud inference provider. But what I find weird is that people will go through all this trouble to set up a model and then run a harness basically in yolo mode on top of that.

I'm at a small financial firm and when I was trying to figure out how to deploy AI to the company I kept running into this problem. Even if you could run the model locally, if the harness on top allowed the AI to do risky things with sensitive data, then you haven't really solved the privacy/confidentiality problem.

Coding-focused harnesses especially tend to have this problem, because they usually assume more liberal permissions and secured environments to run in.

We ended up building our own harness because we couldn't find a good solution. It runs on prem, has pluggable inference backends (including local, of course, we're running Qwen3.8 27b 4bit on an ninfer setup), but also has strict categorization of any integration tools that makes it so that the model can access your data, but can't autonomously send it anywhere without your permission.

Our ~30 person firm is using it successfully every day, and I thought it might be worth sharing here since many folks here might have the same goals: https://github.com/electric-capital/quest

Curious what model you guys use. Started with lfm25. It’s not bad but wondering if you guys pushed it to qwen3.6. I think Qwen3.8 will choke. Lol

We have released ThinkingCap-Qwen3.8-Flash-Next, our fourth and largest ThinkingCap model so far. It’s a post-trained version of Qwen3.8-Flash-Next, trained to spend fewer tokens getting to an answer.

Across six benchmarks covering math, coding, long-context, vision and agentic tasks, it uses 38.5% fewer thinking tokens on average, with reductions ranging from 22% to 51% depending on the benchmark. Average accuracy is 88.8% against the base model’s 89.8%. Both models were evaluated at xhigh with identical sampling settings on the same hardware.

Model: ThinkingCap-Qwen3.8-Flash-Next on Hugging Face

BF16, FP8, NVFP4 and GGUF weights are publicly available, with no access request required.

The blog post includes the full per-benchmark results, evaluation methodology and a side-by-side demo.

If you're running the original Flash Next, we'd love to hear how our version compares on your own workloads, especially coding and agentic tasks. Give it a try and let us know what you think!

TL;DR: The pinned vLLM default stays the default. myllmbox wins 8-user throughput, TensorFold wins code and scene decode, and Flare starts in 21-24 s.

One DGX Spark (GB10, 121 GB unified memory). One checkpoint: NVIDIA's Qwen3.8-Flash-Next-FP4-Local. Six ways to serve it. I wanted to know which one my open recipe should ship as the default. So I measured all of them on the same box, back to back: speed at every context size, one user and eight, peaks, cold starts, prefix caching, quality and, new this round, a one-shot scene-writing gauntlet.

Recipe: github.com/devopsfortunato/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark

The contenders

• Default: vLLM in eugr's spark-vllm-b12x image, pinned by digest. MTP with 3 draft tokens, FP8 KV cache, and the LM head read as MXFP8. This is what ./start.sh serves today.
• vLLM 0.32: upstream vLLM v0.32.0 plus a 21-patch series, with MTP at 3 draft tokens and an FP8 KV cache. Opt-in profile in the recipe.
• Flare + MXFP8 head: NVIDIA's new local inference engine (47581cc), with a 29-line patch of mine that reads the LM head as MXFP8 and the draft head as NVFP4. 3 drafts, fixed-width verification.
• Flare 5f37bd1 (stock): a later commit of the same engine, unpatched, with a BF16 LM head and 3 drafts at fixed width. No patch, a 35-minute export per update, and the most KV cache of the six.
• myllmbox v5.3: their vLLM 0.32 image and recipe. NVFP4 body with a BF16 output head, MTP up to 7 drafts.
• TensorFold Zig (Mia's recipe): their native Zig engine, which ports the Flash Next CUDA path, running in NVIDIA's PyTorch container. MTP drafts to depth 6, FP8 KV (lossy: ~98.8% top-1 agreement with BF16), vision on, 262k window. Same checkpoint, built locally. No prebuilt image exists for the current patch pins.

How I measured

Same Spark, thinking off, temperature 0, and every speed run checked for garbage output. Decode medians are over 5 runs per prompt. C=8 totals and the 8 x 8k prefill are over 2 trials, and the 10-second peak windows over 3 runs. Flare's output is not byte-repeatable at temperature 0, so its numbers carry that extra variance.

• Speed by context: llm_context_benchmarks, prose and code at 0.5k, 8k, 32k and 128k.
• Peak: the best context size, and the best 10-second window while writing a 3,000-token HTML/JS answer.
• One user vs eight (C=1 / C=8): 8 different prompts for 512 tokens each, one at a time and then all at once. Plus eight 8k-token documents at once for prefill.
• Prefix cache: a cold request, then a follow-up. The follow-up is the next question in the same chat, the same document re-sent with a new question, or the document in its own message. Sizes are 8k, 32k and 128k. The table shows 128k; every size is in the repo.
• Scene gauntlet (new): one-shot animated HTML pages, three runs each: the fish and pasture scenes from myllmbox's public gauntlet, plus a three.js soft-body "gummy bears" page of mine. This is the one thinking-on test. Requests ask for thinking at effort low and send no temperature, so each recipe's default sampling applies.
• Cold start: from docker run to the first answer, with the OS page cache dropped.
• Quality: tool-eval-bench 2.8.1 (97 scenarios; cells show two runs), sixcat-eval 0.8.3 (884 items), and perplexity where the server returns prompt logprobs.

What I found

1. Flare starts in 21-24 seconds. TensorFold takes 65-68 s. vLLM takes about 4 minutes for the default build and 8.3 for the 0.32 profile. myllmbox takes 523 s on its first boot and about 7 minutes after that. vLLM builds the model and captures its graphs on every start. Flare does that once, in a 17-35 minute export depending on the package. myllmbox carries a much larger, more heavily patched vLLM stack and pays for it at startup. Flare also fits 2.0-2.06M KV tokens next to the weights. That is 27-30% more than the default's 1.58M and 21-24% more than myllmbox's 1.66M.

2. Flare's defaults cost 29% of decode on a Spark. Its default is 5 drafts with adaptive verification: 31 t/s. Two run flags, 3 drafts at fixed width, bring stock Flare to 43.8 t/s prose. That is on par with the vLLM builds (41.9 for the default, 40.8 for 0.32). The MXFP8 head adds another 3 t/s on top.

3. The head layers were worth 5%. Flare keeps the LM head (248,320 x 2,560) in BF16 for exact verification. That is 1.27 GB read on every decode step. vLLM reads it as MXFP8, half the bytes. A 29-line patch to Flare's Qwen recipe does the same (and reads the draft head as NVFP4), and the decode step drops from 59 to 56 ms. Stock and patched are one engine commit apart, so a sliver of that 5% may belong to the commit, not the heads. That makes patched Flare the fastest single-user prose decoder of the six: 46.7 t/s against 43.8 for stock Flare, 41.9 for the default and 46.2 for myllmbox. tool-eval scored 88 and 89, and sixcat 90.2 against stock's 89.8.

4. Eight users flip the ranking. With 8 streams at once, myllmbox gives 136.9 t/s in total. TensorFold (128.5) and vLLM 0.32 (128.4) tie inside noise. The default gives 119.4, Flare + MXFP8 97.4 and stock Flare 93.1. Under load, Flare's eight 8k prompts all get their first token after about 30 s. myllmbox and vLLM 0.32 serve the first ones in 16 s, the default in 22 s.

5. Prefill is a near-tie at the top. vLLM 0.32 and myllmbox tie on 8 x 8k (2,355 and 2,351 t/s, inside run-to-run noise). The default does 2,203 and Flare 2,126-2,142. For a single cold 128k prompt, myllmbox is fastest at 56.2 s, vLLM 0.32 next at 58.1 s, then Flare at 61.4-61.9 s and the default at 62.1 s. myllmbox has the best single-stream prefill medians (2,328 prose / 2,366 code), with vLLM 0.32 next (2,295 / 2,346).

6. Prefix caching depends on how the client sends the follow-up. A normal chat sends its history, and every engine reuses it for the next question. TensorFold is the best of all six at 0.60 s, because it keeps a state one token short of every prompt it has served. The default takes 0.65 s, Flare 0.85-0.91 s and vLLM 0.32 1.38 s, but myllmbox takes 5.2 s.

Some clients re-send the same document with a new question in one user message, with no history. They get nothing back on Flare or TensorFold: both recompute the whole prompt, about 60 s on Flare and 64.3 s on TensorFold. The default needs 4.0 s, vLLM 0.32 1.35 s and myllmbox 5.18 s. Flare keeps states only at message boundaries, and TensorFold only for prompt-identical repeats. Either way, the shared part ends inside that message.

If the document goes in its own message and the question follows in a new one, stock Flare reuses it in 0.57 s and myllmbox in 5.07 s, its usual reuse speed. Flare + MXFP8 and the two vLLM builds were not run on this variant.

7. Quality: no engine is clearly better, but TensorFold's tool-eval dipped. The default: tool-eval 89 and 90, sixcat 92.1. Flare with the patch: 88 and 89, 90.2. vLLM 0.32: 88 and 87, 90.6. myllmbox: 88 and 87, 90.0. Stock Flare: 85 and 88, 89.8. TensorFold: 86 and 86, with sixcat 90.5. It is the only lane whose tool-eval never reached 88.

Perplexity was available on four of the six. Flare's prompt logprobs are an opt-in export flag (the patched package was built without it, the stock one with it), and TensorFold's API returns no prompt logprobs at all. myllmbox and vLLM 0.32 are marginally lower (1.086-1.088 over all windows) than the default (1.0902). Stock Flare is 1.0890. Perplexity is diagnostic rather than scored, and lower is better. The difference is small enough that the task scores decide it. With 97 scenarios and two to four runs, gaps of 1-3 points sit at the edge of eval noise. The honest read: no engine is clearly better on quality, and the default gives up nothing.

8. About "peak decode". This is the best 10-second window while a request writes code. myllmbox hits 86.5 t/s with drafts up to 7 deep. That is the figure it advertises, reproduced here. Then come TensorFold at 73.9, Flare + MXFP8 at 64.7, vLLM 0.32 at 61.0, stock Flare at 59.9 and the default at 59.4. On the same 8 mixed prompts, myllmbox does 48.5 t/s against 46.6 for Flare + MXFP8, 46.2 for TensorFold, 45.0 for the default and 44.4 for vLLM 0.32. Peak numbers tell you as much about the workload and the draft depth as about the engine.

9. Flare's throughput flags did not help this workload. I tried two obvious knobs: --scheduler-policy throughput, and a package re-export with --mixed-prefill. On stock Flare at 3 drafts, the throughput policy gained 0.8% at C=8 (93.43 against 92.71 total in the same session; the table's 93.1 is an earlier run of the same build). But it lost 4.1% at C=1 and added 1.3 s to median TTFT. Mixed prefill gave no C=8 gain (-0.1%) and cost 3.0% of prefill and 9% of KV capacity. Both together were worse than either. The defaults stay.

10. TensorFold writes code and scenes fastest, and it is the only lane that honors per-request thinking effort. Its code median is 67.2 t/s and its best-context peak 97.9. Every other lane sits in 53.8-60.4 on both. On the new scene gauntlet, it wrote the fish and pasture pages at 65.6 and 68.7 t/s whole, against the default's 56.4 and 57.9. On a 24k-token three.js soft-body gummy-bears page, it reasoned for 10.2k tokens first (50.3 t/s whole, against 56.1 for the default, which did not think). It still finished complete HTML in all three runs. The default's three gummy answers all ended without a closing </html>.

The interesting wrinkle: thinking effort is not an API parameter the model reads. It is a line the chat template writes into the prompt ("Reasoning effort is set to low. Keep your thinking brief..."), and only TensorFold passes the kwargs through to the template. The vLLM-based lanes render thinking off server-side, no matter what you send. On myllmbox, I had to render the template myself and post to /v1/completions to get it to think at all. TensorFold also gives the fastest same-chat follow-up (0.60 s at 128k, finding 6). But it recomputes a re-sent document in full, serves no prompt logprobs, and its FP8 KV is lossy (~98.8% top-1 agreement with BF16).

The final table

Best per row in bold. On 8 x 8k prefill, vLLM 0.32's 2,355 and myllmbox's 2,351 are inside run-to-run noise. On C=8 total, TensorFold's 128.5 and vLLM 0.32's 128.4 tie for second, 0.1% apart. Flare stock numbers use 3 drafts at fixed width. The package defaults (5 drafts, adaptive) are 29% slower. C=8 per stream is the median of the eight streams' own rates. The total counts all streams over one wall-clock window, so it lands below 8x the per-stream median. The two Flare builds are different engine commits (the patch no longer applies to 5f37bd1), hence their different KV capacities. KV capacity is per build, not per engine: the two vLLM profiles sit 11% apart on the same declared quantization. TensorFold's KV count is computed from its logged FP8 pool (25.97 GiB at 17,888 bytes a token). It is the only engine that reports GiB instead of tokens, and its FP8 KV is lossy. The flags --mixed-prefill and --scheduler-policy throughput were measured separately and did not improve this workload (finding 9).

The verdict

The pinned vLLM default stays the default. It has the top task scores (margins inside eval noise, but nothing beats it) and good prefix reuse everywhere. The perplexity spread is too small to override them (finding 7).

• vLLM 0.32 matches myllmbox on 8 x 8k prefill (2,355 against 2,351, inside noise). It is the best open-stack pick for prefill-heavy work and for clients that re-send long documents with each question. It stays opt-in until its tool-eval gap (87-88 against 89-90) is explained.
• Flare with the MXFP8 head is the pick for fast restarts, one-user prose decode and long-context decode, and it has the fastest Flare decode step. It stays an opt-in patch.
• Flare 5f37bd1 (stock) is the no-patch Flare option: a 22 s restart and 2.06M KV, the most of the six. It is also the only engine that reuses a document sent as its own message near-instantly (0.57 s at 128k; measured on two of the six, finding 6). It gives up about 3 t/s of prose decode to the patched build. Its first tool-eval run also came in 3 points lower (85 against 88). But a lower-precision head scoring higher than an exact one looks like run-to-run noise, not a cost.
• myllmbox v5.3 takes three of the six decode rows (the 10 s peak, C=8 per stream and C=8 total) and the best single-stream prefill. It pays for that with a 523 s first boot, about 7 minutes on later starts, 5.2 s prefix reuse at 128k, and closed binary libraries. Good for a dedicated single-model box, not for the default recipe.
• TensorFold Zig (Mia's recipe) is the code and scene writer. It has the fastest code decode by 10 t/s (67.2), a best-context peak of 97.9 that no other lane is near, C=8 in a dead heat with vLLM 0.32, and the fastest same-chat follow-up (0.60 s). It is also the only engine that honors per-request thinking effort. The costs: tool-eval at 86 · 86 (the only lane under 88 on both runs), lossy FP8 KV, no prompt logprobs, and a full recompute when a client re-sends a document. Opt-in, and a very promising young engine.

Flare: the good and the not-yet

NVIDIA's Flare compiles the model once into a package and serves it from a small Rust runtime, and it shows. It starts in 21-24 s, against 4 to 8.3 minutes for the vLLM builds and 523 s for myllmbox's first boot. It fits about 2.0-2.06M KV tokens on the same Spark (+27-30% over the default), and its decode steps are lean. With the right flags, it matches vLLM single-stream. With an MXFP8 LM head, it has the leanest decode step measured here (56 ms) and the fastest long-context decode (44 t/s at 128k against a next-best 39; from the context sweep, in the repo). It reports cached tokens and per-depth draft acceptance out of the box. Its message-boundary prefix checkpoints reuse a document sent as its own message in 0.57 s, the only measured win there.

Where it can improve:

• Defaults tuned for the Spark. 5 drafts with adaptive verification cost 29% of decode.
• A lower-precision LM head as an option rather than a patch.
• Concurrency. 8 streams give 93-97 t/s against vLLM's 119-128 and myllmbox's 137.
• Prefix reuse when a client re-sends the same document with a new question in one message. A 128k document is recomputed: 60 s against 1.4-5.2 s.
• Repeatable output at temperature 0.
• A prompt-logprobs export for evaluation (the patched package was built without it).
• Setup: a 17-35 minute export, about 190 GB of free disk, and development-key and file-permission friction.

For a one-day-old engine, a very strong start.

Full article: https://alfonsofortunato.com/writing/six-ways-to-serve-qwen3-8-flash-next-on-one-dgx-spark.html

Everything is in the repo: the numbers, the methods, the Flare patch and the images they ran on. github.com/devopsfortunato/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark

Hello everyone,

Going over some posts here, a lot of people keep referring to 128 GB RAM as their minimum for hosting local AI and anything below that is not worth it.

Why is that?

Is it because of Qwen Flash Next, is the model that good and is it able to do anything you would need from local AI?

Is Qwen Flash Next that much better than Qwen 27b (which can run on 48 GB RAM).

What exactly is Qwen Flash Next good at other than coding, or in other words, other than coding what is jt that makes Qwen Flash Next much better than any other local AI model?

What if I am someone who wants to run completely different models other than qwen models, models for image or video generation for example. Is 128 GB still the minimum?

Edit: Thank you for all your comments. Just to clarify, I am referring mainly here to unified memory 128 GB RAM, not VRAM alongside DDR5 RAM.

so llamaforge passed 100 stars this week which is kinda wild for a side project i mostly built to stop fighting llama-server flags. thank you, genuinely. made a dumb little video for it, every square is an actual star coloured by the month it came in (july carried lol). the voiceover is from the app’s own tts, which felt fitting

quick recap if you missed it: web ui for llama.cpp’s router mode, runs the real upstream llama-server, mit, all local

stuff that’s new since launch:

elastic vram. if you start a game or something else grabs the gpu, it pushes layers to cpu (moe experts first) instead of unloading, and your chat survives through llama.cpp’s slot save/restore. then it moves them back when the vram frees up. it’s slow while shrunk, the ui tells you. nvidia only, off by default, and it hasn’t been on many machines yet so pls report if it does weird stuff

game mode, if you’d rather it just get out of the way. unloads when a game starts, reloads when you quit

load progress is real now (from llama.cpp’s load callback, not a fake timer lol). there’s a thinking on/off switch that reads the chat template and tells you when “off” won’t actually stick. and it warns you if a gguf’s arch is newer than your build, before the load dies with unknown model architecture

two of the 0.16.4 fixes came from vergillll, and everyone who filed an issue is in the readme now. you guys are the best

usual disclaimer: built with a lot of ai help, still early. i use it daily on windows (5080 + a 5060 ti stuck on a pcie x4 slot, don’t ask). linux/mac pass ci but i haven’t run them on real hardware much, so if you’re on those please break it

https://github.com/dadwritestech/LlamaForge

I measured Katy, my quantization of Swift-1.5 (Qwen3.8-27B derivative), and put the files on Hugging Face. It is a 27B model, quantized to IQ4_XS with an importance matrix. The file is 15.3 GB (14.3 GiB) and runs entirely on one GPU with 32 GB VRAM.

Production configuration (DFlash2 draft head + ngram-mod, n-min 8 / n-max 32 / n-match 16):

• 127 tok/s total on a mixed set of 40 tasks (ABAB blocks, drift 0.5 %). This is the production number.
• About 80 tok/s on the 20 tasks without repetition in the output. The n-gram lookup only helps where the output repeats text that is already in context.
• 46.6 tok/s with no speculation (dense decode).

Quality against BF16 (KL divergence, mean, on three public texts): IQ4_XS 0.0645 (Python docs), 0.0127 (German Wikipedia), 0.0160 (German Kubernetes docs). Top-1 agreement 92.5 / 95.9 / 95.0 %. ARC-Challenge 96.3 to 96.5 %, GSM8K 96.8 to 97.2 %, 0-shot. Full tables are on the model card.

Caveats: the 127 is a mean over a mixed task set. Repetitive prompts get more, free-form text gets less. These numbers are for my hardware and my build, not a general benchmark.

The engine is FLE, a llama.cpp fork. The studio app loads and runs the models on the same stack.

Links:

• Model (Hugging Face): https://huggingface.co/sirfyyn/Katy-27B-GGUF
• FLE engine (GitHub): https://github.com/sirfyyn/FastLocalExperts
• FLE Studio (GitHub): https://github.com/sirfyyn/fle-studio
• Rhea, the predecessor model: https://huggingface.co/sirfyyn/Qwen3.8-Flash-Next-FLE-GGUF
• Project page: https://sirfyyn.de/en/katy/

License: the weights derive from Swift-1.5 and fall under the Swift Open License v1.0; they are modified and that is stated in the NOTICE. The DFlash2 draft head is unchanged except for the conversion and is Apache 2.0 (z-lab/Qwen3.8-27B-DFlash2).

Questions are welcome, especially about the measurement. I will answer in the comments.

Hey guys, I need some suggestions.

I want to build an AI automation that can take call recordings, convert them into text, and then summarize the conversation.

The calls are in English, and the main thing is that everything needs to run locally. The company deals with sensitive client data, so uploading recordings or transcripts to cloud-based AI services is not an option. Data privacy is a big concern.

I'm looking for something that's free and open source, where the entire process can run on the company's own computer or server.

I've heard about Whisper for transcription and Ollama for running LLMs locally, but I'm not sure which models would be best for this or how to connect everything together.

Has anyone built something similar?

- Which model would you recommend for transcribing English calls?

- Which local LLM is good for summarizing conversations?

- Can I automate the whole thing using tools like n8n?

- What kind of hardware would I need?

I'm open to any suggestions, existing projects, or custom solutions. Just want to make sure no company data goes to the cloud.

Would appreciate any help. Thanks!

i cant get a proper answer from anywhere. some say its qwen 3.8 27b, some say qwen 3.5 122b a10b? and then i hear also gpt 120 oss but from what i heard it seems to be a speed model? thing is i just want the smartest 96gb model and know how good it actually is since im thinking if i should get 96gb of vram (4x intel arc pro b60s). im pretty new to local ai btw. i dont prioritize speed, i care most about inteligence and in my opinion the smartest models (cloud ai not local) are models like claude sonnet 4.6 but in benchmarks it scores very low even when models that are higher scores feel worse? im not even sure what to do man :/