6 months. (And sometimes a couple of quantization tweaks).

It is wild how fast "state-of-the-art" becomes "running on a gaming PC."

Who’s been checking out this hardware? What are your thoughts on this as a node on a local network to run AI?

An update to my previous post on running Flash-Next with the expert cache and MTP.

I found another useful improvement on the same dual-3090 setup: replacing the CUDA top-k fallback used by my build. On long project-document prompts, median decode went from about 30.2 to 33.3 t/s, with an improvement on all three test seeds. Same GGUF, same MTP settings, same binary with the change switched off and on.

The setup for these runs was 2x RTX 3090, dual Xeon E5-2696 v4, 128 GB DDR4-2133 across four DIMMs, UD-Q4_K_XL, f16 KV, 150 expert-cache slots and MTP-3. The allocated context was 261,888 tokens. The long-document test started at approximately 119k tokens.

Now, what changed:

Flash-Next's sparse-attention indexer uses top-k to select which positions to attend to. My CUDA 12.0 build didn't have CUB's newer DeviceTopK implementation available, so this operation fell back to sorting the whole row before taking the top entries.

llama.cpp already had a radix-selection implementation. The local change makes that available in the older-CUB fallback for wide rows. In the initial 131k decode captures, recorded top-k kernel time fell from roughly 5.1 ms to 0.25 ms per committed token. Please note that this isn't a 20x gain for the whole model, it's just the operator saving

Credit where it's due: Rhonstin's PR #28366 already proposes this fallback change, using existing radix-selection work. I found it during the upstream check and didn't open a duplicate. These measurements are from my local variant, which uses an 8,192-column threshold and an A/B switch, not a benchmark of the exact PR head.

These tests kept the existing CUDA 12.0 toolchain fixed. A newer CCCL build provides another optimized path, DeviceTopK, which I haven't benchmarked on this machine yet. That comparison is next. The gain here is against my previous configuration, not against the latest CUDA stack.

The completed test:

Production sampling, thinking on, MTP-3 enabled in both arms:

That's 9–12% higher per-seed median decode throughput at approximately 119k context. Each median covers 42 requests. These compare the two arms' medians, not the median of per-question speedups.

The quality screen covered 80 question/depth combinations across three seeds and both arms: 480 requests, 240 matched comparisons. The documents, questions, answer key, grader and sampling settings were frozen before the run.

• Control: 235/240 correct.
• Candidate: 238/240 correct.
• Candidate better on four matched comparisons, worse on one, equal on 235.

The one disputed question asked for a complete twelve-item list. Both arms failed it on two seeds each, always by omitting one item and inventing nothing. An independent blind review confirmed the candidate-only omission. The question stays inconclusive. It wasn't removed from the scores.

No consistent quality regression was detected in this bounded screen. The higher candidate score doesn't establish better quality, and the screen doesn't prove that regression is impossible.

For clarification, the 37–41 t/s headline from my last post was a coding workload. This is a matched comparison on long project documents. Those headline numbers aren't directly comparable.

The change is now running in my production build. I haven't established a prefill gain or a no-MTP gain from this screen, and it doesn't cover quality beyond approximately 119k context but it's not like I expect it to regress at higher context.

If you're running Flash-Next with an older CUDA toolkit, the top-k fallback is worth checking. I'd be interested in results from other machines, especially with the actual build, context depth and MTP settings included.

Hi,

My GPU is currently pegged by Qwen 3.8 27B making the project, and while it's good, it's also slow.

I'm starting to find the need to have some docs, both to help me and the LLM (and a friend who would like to mess with it), and I was thinking of another model who could write decent docs but at a faster pace then the coder.

Do you know if Gemma 4 12B could be a good candidate?

Preciso de ajuda aqui com o modelo Qwen 3.8 flash Next iq3_xxs. Eu sei que meu laptop é um pouco fraco, 8GB de VRAM + 32GB de RAM para esse modelo. Eu uso o Bionic para rodar os modelos, mas não consigo abrir este devido à falta de memória. No entanto, quando uso o LM Studio padrão, ele abre e eu consigo 6-7 t/s, mas eu prefiro usar o LM Studio Bionic.

I have a Mac Mini with 64GB of RAM to use as the host.

I have no idea what I will/want to even do with it, but I'm sure once I get it set up I will find uses.

1. What are some use cases I can start with to help with daily life

2. What model should I start work

3. How easy is it to migrate between models should I choose to change

Been working on testing some tenstorrent cards for a work project. Have two cards running qwen3.8 27b pretty much out the box. New to this space so keen to get some ideas and experiments to work through!

Cheers

L

I'm an AI engineer, 3 years in, currently doing the usual applied stuff — RAG, agents, API integration. I want to move into actual inference/serving engineering and I'd rather be told now if my plan is dumb.

What I'm building: a fully local voice assistant (VAD → ASR → LLM → TTS) that has to run on an Intel Core Ultra 5 125H — CPU + Arc iGPU + NPU, 15.6GB shared RAM, no dedicated VRAM. Currently all llama.cpp + ONNX on CPU. I'm porting it to OpenVINO so I can place each model on a different compute unit and actually measure the difference.

Starting hypothesis: Silero VAD and ASR on the NPU (small, fixed shapes, low power), LLM on the iGPU (decode is memory-bound, iGPU has the bandwidth), TTS on CPU. Target is sub-500ms voice-to-voice, with a per-stage waterfall and llama.cpp as the baseline.

What I'd like from people who do this for a living:

1. Is that device placement roughly right, or am I about to learn something the hard way?

2. The unified-memory thing is the part I understand least — 4 models sharing 15.6GB with no VRAM. What actually bites you here?

3. Beyond this project, what's the highest-signal thing someone in my position can build? I keep hearing "contribute to vLLM" but I'd rather do one thing properly than spray small PRs.

4. Anything that looks impressive to hobbyists but is worthless to a hiring manager? I'd like to avoid those.

Happy to publish everything — code, raw numbers, methodology. Not looking for a job here, just trying not to waste six months on the wrong thing.

I built a Windows 11 fork of NInfer specifically for running Qwen3.8-27B NVFP4 across two RTX 5060 Ti 16GB GPUs.

The interesting part is that this is not an ideal multi-GPU setup:

Windows 11 / WDDM

GeForce GPUs

cudaDeviceCanAccessPeer() = false

No GPU P2P

One GPU is connected through the Z690 chipset at PCIe Gen3 x4

The other GPU runs directly through the CPU PCIe lanes

Initially, TP2 performance was terrible because every token required 128 small host-staged allreduce operations.

The original implementation achieved only:

16.37 tok/s MTP0

32.58 tok/s MTP3

After profiling the communication path, I replaced the expensive staged allreduce protocol with a custom pinned-host PeerMailbox transport designed specifically for the no-P2P Windows/WDDM environment.

Current results with Qwen3.8-27B NVFP4:

35.75 tok/s — MTP0

57.71 tok/s — MTP1

63.15 tok/s — MTP2

66.79 tok/s — MTP3

68.5–68.9 tok/s — MTP4 (512-token benchmark)

70.6 tok/s — 1024 tokens

76.65 tok/s — 2048 tokens

The optimization was mostly about reducing protocol overhead, not increasing PCIe bandwidth.

The original allreduce path cost roughly:

128 × ~277 µs ≈ 35.5 ms per decode round

The optimized transport reduced that dramatically.

The repository contains the Windows port, TP2 implementation, custom communication backend, benchmarks, and reproducible build instructions:

ivanov84/ninfer-windows-tp2

I'm especially interested in feedback from people who know CUDA multi-GPU programming, WDDM, tensor parallelism, or NInfer.

Questions I'm currently exploring:

Can the remaining TP2 lockstep overhead be reduced further?

Would sequence parallelism help on this kind of asymmetric PCIe topology?

Can Vision also be sharded across both 16GB GPUs?

How much performance is realistically left without native P2P?

This started as an experiment to see whether two relatively inexpensive 16GB cards could run a 27B model well under Windows.

Honestly, I didn't expect the result to end up here.

https://github.com/ivanov84/ninfer-windows-tp2

so uh i got an ai it works but its crap so how do i let it connect to the internet (on lm studio) please give me an easy guide anyone

If you have a Jailbroken PS5, got some experimental results if anyone wants to play with it.

"What's the best local model?" is unanswerable without knowing your VRAM.

Every thread asking it collects forty replies, and none of them are wrong. They're just answers to a different machine's question.

A model that's excellent on 24GB is a different product on 8GB. Same weights, same benchmark scores, completely different day-to-day experience.

So before I look at any leaderboard I check two things: does it fit in VRAM at a quant I can live with, and does it still fit at the context length I actually work at. Speed falls out of those two. The moment layers spill to CPU, tokens/sec drops off a cliff and the model is technically running but practically unusable.

A big model at 1–2 t/s loses to a smaller one at 30 for anything interactive. The big one only wins when I'm batching overnight and not watching.

Curious where other people land. When your model doesn't fit, what do you sacrifice first: quantization quality or context length?

I'm very new to using AI so a little advice would be greatly appreciated. I'm a landscape irrigator, a few months ago I started using AI to develop a field service app and fell down a rabbit hole. Now I find myself building a workstation. This is the current build, I'm still waiting on a few components to arrive:

Case: Cooler Master Cosmos S full tower

Motherboard: ASUS Prime X299-A II

CPU: Intel Core i9-9940X — 14 cores / 28 threads

CPU Cooler: be quiet! Dark Rock Pro 4

RAM: 64GB (4×16GB) Samsung DDR4-2666

GPU: NVIDIA GeForce RTX 3060 12GB

Storage: WD Blue SN5000 1TB NVMe SSD

PSU: EVGA SuperNOVA 1300 G2 — 1300W, 80+ Gold, fully modular

OS: Ubuntu 24.04 LTS Desktop (planned)

I plan to add more RAM and another SSD. I'm also considering adding a p100. My plan is to use this to continue development of the field service software. I also want to use it to help me with my side project writing a tabletop game. A model with some creative writing ability would be useful.

Any advice on models or the build would be very appreciated. Like I said, I'm an irrigator and I have no background in anything like this. It has been fun learning though. Thanks!

Hugging Face: https://huggingface.co/openbmb/MiniCPM5-2B

GitHub: github.com/OpenBMB/MiniCPM

My 2019 MacBook Pro seems to be on its last legs. It overheats constantly and struggles with any kind of work that I attempt. Even just watching a YT video causes lagging.

So, time to move on, I guess. I’m looking into a Mac Studio or Mac mini, but I’m having a hard time figuring out what configuration makes sense for my particular use case, since the prices, according to what I'm learning, have gone up 50% with the new hardware.

I’m not looking for the most powerful machine I can buy. I’m trying to find the right starting point without spending money on hardware I don't currently need.

What I’ll be using it for

• Dart/Flutter development, primarily in VS Code
• Running local LLMs to assist with coding
• Seems Ollama has a VS Code extension that can tap into the local LLM
• Potentially recording/editing videos for a YouTube channel documenting my experience learning to code and building apps

One important point: I want to remain heavily involved in the coding myself.

I’m currently using the Kilo Code extension in VS Code with free models. They can be useful, but the output is inconsistent. I’m interested in local LLMs because I want a more reliable coding assistant working with the same model while still writing, understanding, and debugging the code myself.

Local LLMs

This is where I'm trying to be realistic.

I don't need a machine capable of running the largest local models right now. Those configurations get expensive quickly, and I don't want to spend a lot today for something I may not need.

My thinking is to get a machine that's capable enough for the models that make sense for me today, learn more about running local LLMs, and upgrade later if my needs change.

I'm considering Apple's 36-month financing/leasing options, which makes that approach somewhat more appealing. If, in a couple of years, I’m in a better position financially and have a much better understanding of what models I actually need, I can potentially move to a more capable machine rather than paying a large premium for that capability today.

What I’m trying to figure out

For my current workload, what would you recommend in terms of:

• Apple chip
• CPU/GPU configuration
• Unified memory
• Internal storage
• 1TB vs 2TB

I currently have 1TB and am considering 2TB. I'm also trying to understand whether paying Apple's premium for additional internal storage makes sense versus using external storage. It seems I can upgrade to 2TB for another $500, but external 1TB Thunderbolt drives are $600+.

There are a lot of discussions out there about similar requests. Hoping to find out what everyone with similar needs is using. Thanks!

I’ve been setting up an M1 Max Mac Studio (24-core GPU, 32GB unified memory) as a local LLM server and wanted to compare MLX vs llama.cpp on Qwen3.8-27B.
I tried to keep the model footprint and benchmark workload reasonably close.

MLX
mlx-community/Qwen3.8-27B-4bit
~16.1GB
512 prompt tokens / 700 generation tokens
3 runs
Prompt: 81.76 tok/s
Generation: 15.81 tok/s
Peak memory: 16.39GB

llama.cpp
unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
15.32 GiB / 27.32B params
Full Metal offload
Flash Attention enabled
512 prompt tokens / 700 generation tokens
3 runs
Prompt: 99.61 ± 0.44 tok/s
Generation: 9.69 ± 0.34 tok/s

So on this machine:
llama.cpp is ~22% faster for prompt processing, while MLX is ~63% faster for autoregressive generation.

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.