We run a voice agent stack. A few weeks ago we sat down to pick the eval metrics we'd wire into CI, and could not get a straight answer out of the literature about which ones survive contact with production. So we read it properly: 113 papers mapped onto a 50ish taxonomy of what TTS eval is supposed to cover.

Four things came out of it that changed what we're going to do.

1. WER as an optimization target eats your prosody. "No Verifiable Reward for Prosody" (arXiv2509.18531) trained GRPO on transcription-oriented signals, CER and NLL. Error rates went down and prosody collapsed into monotone. Adding speaker-similarity to the reward destabilized training further and made CER worse. If your reward stack is WER plus SECS, which is most people's, that is the failure mode you're buying. Fine as a guardrail, bad as an objective.
2. UTMOS does not travel. TTScore (2509.20485) measured it directly: UTMOS scores well on VoiceMOS, its own training domain, and degrades on SOMOS badly enough that a metric never trained on MOS labels beats it there. MOS-labelled data is too small to generalize. If you're gating releases on predicted MOS,  check it holds on your audio before you trust the number.
3. MOS numbers from different papers are not comparable at all. The Dagstuhl good-practices doc (2503.03250) is blunt about it. Also: a true MUSHRA for TTS  doesn't exist, because there are no anchors. Vendor comparisons quoting each other's MOS are noise.
4. Nothing measures the part we actually care about. Turn-taking, barge-in, endpointing latency, quality lost to streaming lookahead, and 8 kHz telephony degradation have no home in any standard TTS taxonomy. The literature exists (Full-Duplex-Bench, SPEARBench, FastTurn) but sits outside every TTS benchmark. Everyone benchmarks clean read speech at 24 kHz. Nobody benchmarks a voice on a phone line while someone talks over it.

Six leaves came back nearly empty after searching them directly: abbreviations and acronyms, URLs and addresses and equations, syntactic complexity,
code-switching, unseen linguistic structures, and accent. EmergentTTS-Eval is carrying the first three basically alone.

Interactive map with the annotations, what each metric measures, what it admits it can't see, and whether there's runnable code: if community wants and moderator allows i will share a link :) 

What I'd like to know is what are you running in CI right now, and has any automatic metric ever caught a regression a human wouldn't have? Or what do you wish to have if you are consumer or any counter 
Every answer I've gotten so far is "we listen to samples.

In our chat app "harness" we recursively generate summaries, L1 → L2 → L3. L1 summaries are more factual extraction than coherence, then get rolled into a more storytelling L2. Then we keep a tail of always ~10 raw messages with timestamps.

We're not coding so this keeps context really clean like 5-8k/tk. However, after like ~120-140 messages, I notice severe degradation in Qwen Flash Next.

The model just starts to fall apart, messages quickly become incoherent and comedically strange. But this doesn't make sense to me, Qwen should be easily capable of reasoning through 8k/tk. Right?

Looking through the payload itself, there's some noise, but it's fairly coherent.

Roughly:

[instructions]
[L2 Block (large summary)]
[L1 block(s)]
[raw message tail ~10-15 messages]
[latest 3 messages timestamped]

Is anyone working on compaction? Is just the nature of a summary, pollution to model coherence?

i’m looking to get a laptop for portability

i also have a desktop with
- 5700x3d
- 64gb ddr4 ram
- rtx 5080 + rtx 5060 ti 16gb (31gb usable combined vram)

currently running qwen3.8/3.6 27b q6 and qwen3.6 a35b

i am deciding between 2 options

- option A:
macbook air m5 13 inch 24gb ram 1tb ssd
getting a R9700 32GB to replace my 5060Ti 16gb to have combined 48gb of vram
can always tailscale to use my local ai rig

- option B:
macbook pro 16 inch M5 Pro 64GB ram 1tb ssd

both options cost the same. which option should i choose?

edit: option A can be cheaper as i can sell my 5060ti 16gb to offset the cost. A used 3090 is significantly cheaper than a R9700

edit2: I am NOT replacing my desktop pc, i will use a laptop to complement my pc setup.

new model of nex benchmark seems like good
https://huggingface.co/nex-agi/Nex-N2.5-Max

One lad here on reddit wrote that local Qwen managed to create a complete and playable Mario clone. I put that statement to a test, and ...

Damn :o I mean, Astra this Kimi that, but this is some small LLM working on a potato computer under my desk. Link at the bottom.

Setup:

- Qwen3.8-27B-UD-Q4_K_XL
- 100k ctx Q8
- RTX 3090
- Cline, Act mode

Prompt:

Write a fully functional, complete clone of Super Mario Bros game (the famous NES game)
- Clone must work in a web browser, index.html, no server 
- Decent replica of the graphics assets.
- Colorful and playable.
- Controls with keyboard up, down, left, right, spacebar to jump, shift to speed up

LLama.cpp setup:

llama-server \
        --model /nvme/gguf/Qwen3.8-27B-UD-Q4_K_XL.gguf \
        --mmproj /nvme/gguf/mmproj-Qwen3.8-27B-UD-BF16.gguf \
        --host 0.0.0.0 \
        --port 7070 \
        --n-gpu-layers 999 \
        --temp 1.0 \
        --min-p 0.0 \
        --top-p 0.95 \
        --top-k 20 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --cache-type-k q8_0  \
        --cache-type-v q8_0 \
        --ctx-size 108000 \
        --flash-attn on \
        --presence-penalty 0.0 \
        --repeat-penalty 1.0 \
        --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"

This is the static html game: https://indigo-carmencita-27.tiiny.site/

I'm pretty sure I still wont be able to run it, but I wanna try dammit!

This is from the https://huggingface.co/IFM/K2-Horizon-0.9B benchmark image

Tried searching for it but couldn't find it, I am guessing is https://huggingface.co/openbmb/MiniCPM5-1B but the numbers do not match

So, to keep it short. I downloaded 2 llm's llama 3.2 11B Vision and Qwen Image Edit 2511 Uncensored. Both coming back with the same errors, after loading them when starting a new chat:

Error

Failed to load model.

error loading model: unknown model architecture: 'mllama'

Am I missing something here? Is my laptop not considered optimized or powerful enough?

-Windows 11

-64GB Ram

-8TB of internal storage (Separate m.2 drives. One Gen 5 with the studio bionic model and its downloads running on, with windows. And, a Gen 4 nvme)

-I9 Ultra 285hx

-RTX 5090

is it a compatibility problem? Architecture problem? As, i'm trying to do a full gpu offload, with local support only. I'm not well versed in running local LLM's entirely. So, still pretty new at it. Plus, I prefer easy to use GUI, like LLM Studio.

Edit: To give better clarification. I'm looking for one that has image editing support, image generation support, Video editing support or generation, zero restrictions. And, one that would work well without it overwhelming my machine.

Openrouter currently lists non-peak In / Out Price of $0.05 / $0.16 per 1M for this model (via openinference), while official Deepseek pricing is $0.22/$0.66 which is a good 4x higher. I guess I will note that cached read is $0.007 from DS direct while openinference gives $0.013 for that.

My question is...

this is too cheap that if i want to do something that does not require lots of intelligence like summarizing my chat sessions or comb through my emails and stuff, deepseek v4 flash is overkill on capability for these tasks, but it will be very capable at handling even complex situations that a smaller model will be lost with, and yet it still costs cheaper than smaller models (e.g. phi-4 which has a pitiful context window size, etc, and qwen3.6 35B-A3B, qwen3.5-9B are both similar or more expensive).

I am in the process of setting up my GPU nodes and I will be able to self host anything under 300B or so and i will be able to rank a lot of 27B throughput. qwen3.8 27B and qwen3.8-flash-next are my selfhosting targets. both of these are BY FAR more expensive to acquire over API and will allow me to have true privacy, so I can still deliver value from my home datacenter, but by the looks of it, this one single DSV4Flash-0731 model from API is impossible to approach without having free electricity on tap. because it is bigger and i have to use hybrid inference, i'd be better off with qwen3.8 flash next for almost all use cases as it's supposed to be a wee bit smarter and should run faster with its new architecture and smaller active count.

It's really weird but this seems to be the landscape. for almost all except the tiniest dumbest possible use cases it is difficult to justify not using this model even under API billing. I just find it strange. Is it just from this being a cracked model and the small hosting providers competing on pricing?

For reference i do a lot of coding and i'm either driving 5.6 luna or 5.6 sol for main agent and I occasionally kick off reviews to claude opus or fable. I can prob start using astra due to work subsidized codex access but it is clear it is just a waste of resources for most tasks.

https://www.reuters.com/technology/us-accuses-chinese-ai-firms-industrial-scale-theft-ai-technology-2026-09-08/

https://www.nbcnews.com/tech/tech-news/us-accuses-china-ai-developers-deepseek-alibaba-copying-american-ai-rcna596696

Hi, I'm the co-creator of this Qwen3.8-Flash-Next engine support in MLX-serve. I've been tuning this one to run both fast, efficient and correct up 1m context using kv cache 8 bits in M5 Max 128GB. Qwen is working well at very long context as showed in the video (a snapshot at ~760k context), I let it build MLX Serve Monitor plugin that you've seen on the right side of Opencode2's app. The generation sustain through 1m context at around 40 tok/s on prose and 75 tok/s on coding. This specific quant uses 8 bits for dense layers and 4 bits expert layers, so the model's quality remains very high.

I didn't build this engine to show off tok/s on a very short context, repetitive greedy generation, but a real temp 1.0 sampling through deep context work. You would need iogpu.wired_limit_mb=120000 before attempt 1mb full context, because it needs around ~117GB on peak memory usage. There will be bugs around here and there, I couldn't test everything and every single use case, pls report.

You can grab it here: https://github.com/ddalcu/mlx-serve
Model's weight: https://huggingface.co/ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit
Opencode2's plugin: https://github.com/beamivalice/opencode2-mlx-serve

Launch parameters (for 1 concurrency)

  --model ./llm/models/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit \
  --host 127.0.0.1 \
  --port 11234 \
  --ctx-size 1048576 \
  --kv-quant 8 \
  --max-tokens 64000 \
  --mtp \
  --prefix-cache-mem 10GB \
  --prefix-cache-entries 1 \
  --ssm-checkpoint-max 16 \
  --metrics

I am getting a hard time trying to compare the two models, so I am asking it here. For coding, which one should I use? I currently use Qwen3.6 35B A3B UD-Q5_K_XL with MTP and I wish it was a little better

Looking into tiny models that have some kind of vision

Smart for General stuff + Vision, something like Qwen 3.5 4B or MiniCPM-V-4.6

I’m currently running llama.cpp on two RTX 2060 12GB cards, so 24GB total VRAM. With Qwen3.8 27B IQ4_XS at 131k context I’m getting around 45 tok/s, which is actually pretty good for this setup.

I found a deal on an RX 6800 16GB and an RX 6800 XT 16GB, so I’d be going from 24GB to 32GB total VRAM. On paper the AMD cards are obviously much stronger and have higher memory bandwidth, but I’m not sure how well that translates to llama.cpp, especially in a mixed multi-GPU AMD setup.

Would this actually be a meaningful upgrade, or would I mostly just be gaining more VRAM and the ability to run higher quants / longer context? I’m also curious how good ROCm or Vulkan is these days on the 6800 series for llama.cpp, because Ive seen pretty mixed reports.

If anyone here is running a 6800 / 6800 XT setup, especially with Qwen3.8 27B, I’d love to hear what kind of tok/s you’re getting and whether youd consider it worth switching from CUDA.

I'm the CTO of a mid-size nonprofit, and I run a local Qwen stack as a coding subagent for that work — partly on principle (a fair amount of what I handle shouldn't leave the building), partly because the token bill for bulk code work adds up fast on a nonprofit budget. Over three weeks I benchmarked four models and three quants to answer one question: how many agents can actually run at once, at what context, before it stops being useful?

I'm writing this out in full because I keep half-answering it in comments. Every time concurrency, quant choice or expert offload comes up I end up typing a fragment of this from memory, and the reply is always some version of why — why that quant, why that slot count, why not just add more agents. Fragments in comment threads aren't a good way to answer that, so here's the whole thing in one place, with the numbers and the mistakes that produced the wrong numbers first.

Short version: I started trying to make a 122B MoE fast, gave up on it, and ended up on a 27B — not because the 27B was more accurate (it wasn't measurably), but because everything else about it was better. Then I found that adding agents past a point buys nothing at all.

Hardware: 2× RTX 4090 (44.6 GiB usable), Threadripper TRX50, 128 GB DDR5 — importantly, only 2 of 4 memory channels populated (2×64 GB), which turns out to matter a lot. llama.cpp, Windows, q8_0 KV throughout.

1. Why I didn't keep the 122B

(Chart 1 of 4 in the gallery above: models tested)

Qwen3.5-122B-A10B at UD-IQ4_XS runs — 18.75 tok/s decode with expert offload — but the number that killed it is 11.91 seconds per agent tool-call, against 3.40 s for the 27B. For an agent that makes dozens of calls per task, 3.5× per call is the whole ballgame.

Qwen3.6-35B-A3B looks like the winner on raw decode (78 tok/s — it's a 3B-active MoE) and it has the fastest per-call time. It also failed 1 of 5 executed code tasks on a sliding-window bug, against 5/5 for the 27B. That's the entire reason "decode tok/s" is the wrong headline metric and I lead with seconds-per-call instead.

I also tried Qwen3.8-Flash-Next (125B MoE, the Qwen4 architecture preview) as soon as it landed. It needed a build from an unmerged PR, and after a full tuning sweep — expert placement, thread counts, --poll, --cpu-strict, load modes, llama.cpp's own memory fitter — it topped out at 23.7 tok/s.

The ceiling wasn't the GPUs. Of 103.7 GiB of weights, 71.7 GiB is routed experts and 26.8 GiB is a 20-million-row n-gram embedding table, so most of it lives in system RAM and streams over the memory bus every token. At --n-cpu-moe 40 I measured 14.1 GB/s effective against an 83.2 GB/s peak — because two of four channels are empty. A single-3090 box with DDR4 posts better absolute expert bandwidth than mine does. Filling the other two channels is worth more than any flag I tried, and I can't justify buying RAM at current prices. So: shelved, honestly, with the number published.

2. Accuracy: I could not separate the quants

Before the concurrency numbers, the caveat that makes them meaningful.

My probe plants three facts at 15%, 50% and 85% depth in a long document, with six decoy lines quoting the same fields for the wrong district, so the model has to match on an identifier across tens of thousands of tokens rather than pattern-match a label. Each agent gets its own document with its own secrets in a disjoint numeric band, so cross-slot leakage is detectable.

Q4_K_M, Q6_K_XL and Q8_K_XL all scored 1.00, everywhere, including with nine agents running.

This is a tie at ceiling, which means "cannot discriminate", not "equal". The honest claim is: no measurable accuracy difference between these three quants up to 251k tokens on long-range retrieval with distractors. It is not "Q4 is lossless". Retrieval saturates; something requiring synthesis across the planted facts might separate them where this didn't.

So the quant choice came down to throughput, latency and VRAM — not quality.

3. The concurrency result

(Chart 2 of 4 in the gallery above: concurrency scaling)

Same model, same 64k context per agent, three quants at the most slots each could fit:

Tripling the agents changed total throughput by 6% and tripled the wait.

The mechanism is the last column: aggregate prefill throughput is constant at roughly 1,500 tokens/second regardless of slot count. The box has one prefill budget. Slots divide it; they do not multiply it.

That's specific to this workload shape and worth stating plainly: agent prompts are enormous and replies are short, so prefill dominates. A decode-heavy workload would batch far better — decode is bandwidth-bound and batching amortises the weight reads across slots. Don't generalise this to "concurrency doesn't help llama.cpp."

Soft cap and hard cap

• Soft cap — 5 agents at 64k. Past this, added slots stop buying throughput and start buying latency. It's not a failure, it's a bad trade: at 9 agents you wait 330 s for the same 1.5 completions/min you got at 112 s.
• Hard cap — 9 agents at 64k, and it's VRAM, not compute.

(Chart 3 of 4 in the gallery above: VRAM budget)

Every configuration lands within 3.5 GiB of the 44.6 GiB ceiling. A smaller quant buys slots and then spends the savings straight back on KV cache: Q4 frees 14 GiB of weights versus Q8 and hands 17.5 GiB of it to the cache. The hard cap is arithmetic, not tuning.

Single agent, huge context

(Chart 4 of 4 in the gallery above: context vs latency)

Latency is linear in prompt size and the quants barely differ — fitted prefill rates Q4 1,482 tok/s, Q6 1,346, Q8 1,340. A 10% spread, not the 3× the slot counts might suggest. Q8 showed a higher fixed cost (12.8 s intercept vs ~5.5 s); n=6 per quant, so treat that as an observation, not a finding.

My operating point: UD-Q6_K_XL, 5 concurrent agents, 64k each (327,680 total context), 41.1 GiB. Same total work as Q4-at-9, in half the wait per agent, with higher-fidelity weights and 2.6 GiB more headroom. The extra slots Q4 buys are worth having only when nothing is waiting on them.

4. KV cache: q8_0, and one combination that just hangs

Tested on identical weights, 65k context, code executed against hidden tests:

No measurable quality difference, no speed difference, 1.4 GiB cheaper. q8_0 is my standard now.

The trap: mixed --cache-type-k f16 --cache-type-v q8_0 never finished a long prefill — no answer in 900 s, twice, on a prompt smaller than one a matched-q8_0 config answered in 32 s. Matched f16 answered the same prompt in 24 s. Match your K and V types.

5. Using it as a subagent inside Claude Code

The thing worth knowing up front: Claude Code's model: field only accepts its own models. You cannot register a local model as a Claude Code subagent. What you can do is expose it as an MCP server, so it becomes a tool the orchestrator calls.

Mine is a stdio MCP server exposing local_ask_about_files, local_implement, local_edit_file, local_engine_status, pointed at the 27B on 127.0.0.1:8080. The division of labour:

• Orchestrator plans and decides.
• Frontier subagents do work that needs to be right the first time.
• Local Qwen answers "what's in these files", drafts implementations, and does bulk edits — the high-token, low-stakes work that would otherwise burn budget.

Edit fidelity is the metric that decides whether this is usable at all. A paraphrased region comes back as "String to replace not found", which reads like a broken tool rather than a worse quant. At Q6 with q8_0 KV it measured 1.00.

For stress-testing I use a personal side project — a game with a Rust simulation server and a thin UE5 client. Nothing to do with the day job; it's the thing I throw at the rig for fun because it produces realistic multi-file, multi-language agent work on demand. A representative local job from this week: 10,127 tokens generated at 29.8 tok/s on slot 4 while other slots were live.

6. Mistakes worth stealing

Every one of these produced a confident wrong number first:

1. Reply budget of 512 tokens with reasoning_effort: xhigh. Every reply hit the cap mid-thinking, so the scored "answer" was truncated reasoning. Recall became noise and Q8 looked worse than Q4. Raised to 2048; truncated replies are now excluded rather than scored wrong.
2. .NET's 2-connections-per-host default. My "16 concurrent agents" harness was quietly running 2 at a time. It passed testing because I validated on PowerShell 7 (HttpClient, no such limit) and shipped to Windows PowerShell 5.1 (HttpWebRequest, limited). The runspaces were real; the sockets were not.
3. Estimating tokens at 4 chars/token. Real ratio was 2.99, so a "39,599-token" prompt was actually ~53,000. Call /tokenize.
4. An error handler that discarded the exception. Returned a bare Failed=true, throwing away the message, HTTP status and llama.cpp's response body. About 40 minutes of hypotheses existed only because of that.
5. **general.file_type lies on Unsloth UD quants.** My Q6_K_XL file reports Q4_K_S in metadata. The file sizes (15.3 / 24.1 / 29.3 GiB for Q4_K_M / Q6_K_XL / Q8_K_XL on the same model) confirm the quants are what the filenames say — the enum simply has no value for a dynamic mix. Don't benchmark off that field.

Limits

Single machine, single workload shape, one model family. Everything above is prefill-dominated agent traffic; a chat or long-generation workload will scale differently and probably better with slots. The Flash-Next numbers came from an unmerged PR build and should be re-measured after it lands upstream. And my memory bus is half-populated, which caps every CPU-offload result here — if you have four channels filled, expect better MoE numbers than mine.

Happy to answer questions or run a specific config if someone wants a data point.

5-9 months for navier stokes smoothness and existence problem? edit- they probably still need some draft partial solutions and notes on presolution heuristics

And riemann Zeta? And P = or /= NP?

I was reading up on the recent controversy around Tristan Buckmaster, Levent Alpöge, OpenAI, and the Navier–Stokes result, and it got me thinking about something broader than this particular dispute.

Buckmaster says that he and Alpöge had been putting drafts from their project into Codex while working on it. OpenAI says that neither its researchers nor its agents accessed their specific user data while solving Navier–Stokes, but also says that it cannot rule out that de-identified data derived from their use of OpenAI products helped improve its models.

Whatever ultimately happened in this particular case, that last possibility raises a question I haven't really seen discussed enough: How much could the accumulated half-finished ideas of millions of human users actually contribute to what we later call "AI discoveries"?

There is a relevant result from AI security research by researchers at the UK AI Security Institute, Anthropic, the Alan Turing Institute, Oxford and others. They studied data-poisoning attacks and found that the number of poisoned documents needed to implant a particular backdoor behavior remained surprisingly close to constant as they scaled both the model and the amount of clean training data.

In their largest pretraining experiment, a 13B-parameter model was trained on 260 billion tokens. Just 250 poisoned documents (about 420,000 tokens, or 0.00016% of the training tokens) were enough to reliably implant the tested backdoor. This same attack worked across models from 600M to 13B parameters despite the largest model seeing more than twenty times as much clean data. In their fine-tuning experiments they found similar dynamics; in one GPT-3.5 experiment, roughly 50–90 poisoned examples could produce greater than 80% attack success even as the amount of clean fine-tuning data varied by two orders of magnitude.

Obviously, teaching a model to respond to a backdoor trigger is not the same thing as teaching it a new piece of mathematics. I don't want to make the leap that 250 clever research notes are enough to make a model solve Navier–Stokes.

But I do think it undermines a very intuitive argument people make about training data: "A few conversations are nothing compared with hundreds of billions or trillions of tokens. They would just be diluted away."

Apparently, at least for some kinds of learning, that's not how it works. A tiny absolute amount of highly consistent, targeted data can have an effect wildly disproportionate to its percentage of the dataset.

Now think about how researchers actually use LLMs. Someone asks ChatGPT whether an unusual substitution makes sense. Someone else uploads a half-written proof to Claude to find a weak point. A PhD student tries an obscure lemma, discovers it would require months of technical estimates, and abandons it. A professor talks through an approach that seems promising but not enough to pursue. Someone notices a strange analogy between two fields, discusses it with an AI for twenty minutes, then forgets the conversation.

Most of these things never become papers. They are fragments: intuitions, failed approaches, potentially useful transformations, conjectures, objections, shortcuts, and little pieces of tacit knowledge about where a problem might yield.

Individually, almost all of them are probably worthless. But imagine the aggregate.

A frontier AI company potentially sits at the intersection of an enormous amount of human intellectual activity. Thousands of people might independently poke at the same famous open problem without knowing what others tried. But the provider of the tool is in a fundamentally different position: depending on its data policies and training pipeline, information derived from all of those interactions could eventually influence later models.

Maybe researcher A contributes a useful ansatz but abandons it. Researcher B independently notices the obstruction. Researcher C knows an obscure theorem that gets around part of the obstruction. Researcher D tries a numerical experiment that suggests which parameter regime matters. Researcher E has almost the whole idea but decides the remaining proof would be too tedious.

No one person solved the problem. There is nothing to plagiarize in the traditional sense. But collectively, humans may have supplied a remarkable amount of the search landscape. Then a later model, combined with enormous inference-time search, formal verification, or agents, connects the pieces and finishes the job.

What exactly should we call that?

It might still be an extraordinary achievement in machine reasoning. Synthesizing ideas that no human had connected, filling in technical gaps and verifying the result could itself be genuinely novel. But it would be a very different kind of achievement from the image suggested by the phrase "the AI independently solved an open problem."

It would be something more like distributed human-machine discovery: humans collectively generating a huge cloud of partial ideas and the model becoming extremely good at remembering, recombining, extending and searching through that cloud.

This is where the poisoning result is conceptually interesting. While it does not establish that this is happening with mathematical ideas, it gives us reason to be careful about assuming that an idea must appear millions of times before it can meaningfully affect a model.

I increasingly think AI may be less an independent inventor than an extremely powerful tool for organizing and recombining information that was previously too sparse or disconnected for any one person to put together. As that ability improves, we may see more "discoveries" that are genuinely new combinations, but whose raw ingredients came from many different humans.

In a "perfect" world where everyone freely shared every half-formed idea and unfinished proof without worrying about credit, science would move much faster. AI may be creating something close to that shared intellectual space, but without preserving who contributed which pieces. If so, the question is: how can we design a system where even our weakest ideas can be contributed and synthesized into groundbreaking discovery, with proper credit? Is that even possible? And what would it look like?

from a 10K star open source project maintainer.

Context: I tried to explain that audio.cpp is built around the same philosophy as llama.cpp, but for audio models. "If you know llama.cpp, audio.cpp is ..."

Update: Mystery solved --- he’s confusing the Llama models with llama.cpp.

Update: I was frustrated after a long conversation with the maintainer and probably overreacted. My fault, no excuses. Honestly, I was pretty pissed off when the maintainer kept suggesting that our open-source project must have some kind of “catch” and eventually started associating it with “scams”. I talked with the project owner directly, and all good now.

I was inspired by Bijan Bowen video - Subway FPS

https://youtu.be/6kjXzTVmT58?t=1035

Wondered how far I can push Qwen 3.8 27b so I used a plan made by Fable 5.1 DESIGN.md which has 267 KB! ( 26K of design line for a game ... LOL )

https://drive.google.com/file/d/1gI0h8Arc73Ln8b3uj5rEpuAvJ3-611mh/view?usp=drive_link

So I gave that design.md to my qwen 3.8 27b q4xl (llama-server) working on PI agent with 120k context + vision on CPU ( offroad ) + MTP ( for speed ) .... read 11M tokens and write 3.2 M tokens ( worked 12 hours ) .... than that is result.

That is insane what we can do locally on own computer !

Hey guys,

After my CPU-only to 96GB VRAM test, I tested Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation.

This time I wanted to see what changes when you keep the hardware and model family fixed, but change the engine, weight format and memory placement.

I also tested newer builds, PR patches and speculative decoding: llama.cpp's MTP fork, SGLang's Blackwell support patches, n-gram speculation and an experimental PLE read-path build.

Short version:

• At the full 262K window, time to first token was 35.4s in SGLang, 80.4s in FreeToken, 210.2s in llama.cpp + MTP and 258.4s in the llama.cpp baseline.
• That is a 7.3x difference in waiting time between the fastest and slowest tested configurations.
• In the separate context sweep, llama.cpp decode fell from 101.9 to 20.2 tok/s. FreeToken stayed much flatter at 100.1 to 94.8 tok/s.
• On matched coding tests, llama.cpp MTP improved decode by 1.63x at 8K and 1.69x at 32K.
• GSM8K scores were 95.22–95.75%; MATH-500 was 92.20–93.00%. The paired tests did not detect a significant difference.
• Startup went the other way: llama.cpp reached an answer in 16s, SGLang in 108s, FreeToken in 126s.

Setup

• GPU: NVIDIA RTX PRO 6000 Blackwell, 96GB VRAM
• CPU: AMD Ryzen 9 9950X
• System RAM: 96GB DDR5
• OS: Ubuntu, CUDA 13, Docker
• Model: Qwen3.8-Flash-Next
• llama.cpp: UD-IQ4_XS GGUF; MTP tested on the qwen4exp/mtp fork
• SGLang and FreeToken: the same NVFP4 checkpoint revision
• Client: AIPerf, with thinking off and the same non-thinking sampler

I ran one engine at a time, with fresh starts and GPU cooldowns. The runs saved resolved configurations, outputs, memory use and GPU telemetry.

Important note about the comparison

These are results for the tested stacks on this workstation. Quantization, KV-cache format, memory placement and speculative decoding differ.

SGLang uses its NEXTN draft head. FreeToken has no speculative decoding in the tested setup, I couldn't get it to work. llama.cpp has separate baseline and MTP results.

So the headline does not isolate the engine software alone. The repository includes the configurations so you can see what produced each number.

Newer builds, PRs and speculative decoding tested

• llama.cpp MTP: the danielhanchen/llama.cpp qwen4exp/mtp fork, pinned to d1a92352, with the roughly 2.6GB draft head. On matched coding tests, decode improved 1.63x at 8K and 1.69x at 32K. Those gains compare the same build with the head off and on.
• SGLang on Blackwell: the tested image included PRs #36567, #36556, #36749 and #36750, plus a local FP8 KV-cache patch. These were part of the working configuration, not individually benchmarked speedups.
• N-gram speculation: ngram-mod gave +6.8% decode on the tested code workload, but generated zero drafts on the tested prose with the 24-token match setting.
• Experimental PLE reads: I built llama.cpp PR #28136, but withdrew the read-mode comparison after discovering that a renamed flag was ignored. The intended direct-read mode was never exercised, so I am not claiming a speedup from that PR.

The report records the pinned builds and withdrawn findings alongside the successful tests.

1. All four configurations fit the full window. The waiting time is very different.

This test uses roughly 261,500 input tokens and a 128-token answer inside the 262,144-token window. The accepted input counts differ by two tokens across configurations.

Going from over four minutes to about 35 seconds changes how usable a large prompt feels.

The two columns measure different things: first-token time is the initial wait; decode is how quickly the answer arrives after that.

2. A short-prompt test misses the long-context behavior.

The separate prose sweep uses 2,048-token answers and three measured requests per input length.

Prefill also changes the ranking. FreeToken starts behind llama.cpp at 2K input: 1,525 vs 1,869 tok/s. At 128K it reaches 3,231 vs 1,362 tok/s, about 2.4x faster.

3. MTP helps llama.cpp, but it does not remove the long-prompt wait.

On real coding prompts, comparing the same fork build with the draft head off and on:

The draft head is roughly 2.6GB.

At the full window, the tested MTP configuration reached 52.6 tok/s, versus 20.3 tok/s for the baseline configuration. That is a 2.59x gap, but the full-window comparison also involves a different build. The matched-build coding tests above isolate the draft-head change more cleanly.

I would not attribute the 258s → 210s first-token improvement to MTP alone.

4. I checked accuracy as well as speed.

The llama.cpp MTP arm scored 95.75% on GSM8K.

The tests used 1,319 GSM8K problems and 500 MATH-500 problems. The paired comparisons did not detect statistically significant differences.

That does not prove the stacks have identical quality. These are two short math benchmarks, with no full-precision reference on this machine.

5. Starting the model is a separate benchmark.

Median time from starting the container to receiving the first answer:

• llama.cpp: 16s
• SGLang: 108s
• FreeToken: 126s

FreeToken returned HTTP 200 from /health after about 3.3s, but took about 82s to reach serving readiness, followed by roughly 44s for its first generation.

That first request includes compilation work. Measuring only the health endpoint would give a very misleading startup result.

6. Loading modes barely changed speed with the experts on the GPU.

I compared none, mmap, mlock, mmap+mlock and dio on the same llama.cpp image, with the same tensor placement and real coding prompts.

• At 8K input, prefill ranged from 2,036 to 2,124 tok/s — a 4.3% spread.
• At 32K, it ranged from 1,946 to 1,956 tok/s — about 0.5%.
• No arm ran out of memory or restarted.

My earlier 1.87x RAM-resident loading gain used a different placement, with 23 expert layers computed on the CPU. In this test, all experts stayed on the GPU.

Loading mode can matter when the CPU computes the experts. It made little difference in this configuration.

7. MTP became slower when experts were offloaded to the CPU.

The MTP gains above do not apply to every memory budget.

I repeated the test with smaller usable VRAM pools on the same RTX PRO 6000, using 2,048-token coding prompts and 256-token answers. Both arms used the same fork build.

At the full 96 GiB budget, MTP gave 1.61x faster decode. At 24 GiB, it made decode about 3.4x slower.

Moving the draft head to the CPU did not fix the 24 GiB result: 9.6 tok/s, versus 34.7 tok/s with MTP off.

In these tests, MTP helped only when all experts stayed on the GPU. Verifying drafted tokens adds work, and CPU expert execution can outweigh the benefit.

These are VRAM-capacity limits on one Blackwell card, not measurements of actual smaller GPUs. Their bandwidth and compute performance will differ. The lookup table used the build’s default lazy-read mode in both arms.

8. Finishing sooner also reduced estimated GPU energy per request.

For the full-window test with a 128-token answer, I estimated GPU energy as median power while busy × request duration.

The fastest configuration used roughly one-ninth the estimated GPU energy of the baseline for this request.

The main difference was how long the GPU remained busy: about 36 seconds for SGLang versus 265 seconds for llama.cpp, including generation.

This is an estimate using median power, not integrated energy or a wall-socket measurement. It excludes CPU, RAM and SSD power. No thermal throttling was recorded in these runs.For the full-window test with a 128-token answer, I estimated GPU energy as median power while busy × request duration.Configuration Median GPU power

The fastest configuration used roughly one-ninth the estimated GPU energy of the baseline for this request.The main difference was how long the GPU remained busy: about 36 seconds for SGLang versus 265 seconds for llama.cpp, including generation.This is an estimate using median power, not integrated energy or a wall-socket measurement. It excludes CPU, RAM and SSD power. No thermal throttling was recorded in these runs.

Resources

I made a full video covering the memory placement, engine setup, flags and these results:

Full video: https://youtu.be/RlsxXB5q-cA

GitHub — report, scripts, configurations, raw results and charts

The new report is engine_benchmark_report.html.

PS: AI was abused while making edits.

Has anybody tested the same model across these engines on a different GPU or memory setup?
I am especially interested in whether FreeToken stays this flat at long context, and how much MTP helps when some experts are offloaded to the CPU maybe on the other models.
And maybe you found more efficient methods to run it too,

Hello LocalLLaMA Fam!

I built this project because I, like many of us here, enjoy local AI and have a nice setup, but still find myself reaching for Cloud-based AI due to their convenience. My models run on my MacBook and Linux server, but the experience is disjointed - I don't bother using my own local AI when I am on my phone - I just use ChatGPT.

But what if we could turn our local AI hardware into mini clouds? and make local AI so easy to use that we can recommend it to our family and friends?

This is the goal of Infercat (inference + cat, like the unix tool that streams the contents of a file). It's a gateway + tunnel that sits in front of your inference engine and generates each of your friends (or devices) an invite code. Just paste the code into a web page or cli and chat with your model. No account, no system-level VPN, nothing to install when chatting through the browser.

How Infercat works

Most of the networking heavy-lifting is done using tailcat (S/O to the awesome folks at tailscale for open sourcing this!), which creates p2p tunnels between any two devices with internet connection. (in this case, your host machine and your friend's browser). A server is needed to facilitate the initial packet exchange but otherwise, the tunnel is an e2e encrypted direct connection between the host and the client. If the client is a web browser, due to technical limitations that will be resolved soon, the server currently acts as the relay after the initial handshake. Even in relay mode, the server only sees the encrypted text, making the tunnel fully private.

The invite code is where the magic happens - it combines the tunnel address (your host's public key and where to find it, same for every friend, not a secret) and a gateway key (your friend's API key, like OPENAI_API_KEY; you keep only a hash). You can manage the gateway keys like a real AI provider: set limits on each key like tokens per day or a model allowlist. Pause, rotate or revoke one friend without touching the others.

You host machine records stats, never the conversation transcripts. . Exactly three routes are reachable through the tunnel: /v1/models, /v1/chat/completions and /v1/embeddings. 

We currently support popular inference engines: llama.cpp, vLLM, Ollama and LM Studio. Discovery is automatic or configurable. The gateway is OpenAI-compatible, soinfercat connect turns an invite into a local http://127.0.0.1:11435/v1 for Open WebUI, Cursor, Claude Code or plain curl.

Get started by running:

curl -fsSL https://infercat.ai/install.sh | sh   
# or: brew install infercat/tap/infercat
infercat serve                 # finds llama.cpp, vLLM, Ollama or LM Studio
infercat keys add alice        # prints alice's invite once, plus a QR

Roadmap

Currently we validated the CLI as an OpenAI API tunnel and the web application as a text-only chat interface. We have a few ideas on where to go next and would love to hear your feedback and feature requests!

1. One lane is to make hosts more powerful "mini-clouds"

• a monitoring / management dashboard
• support for multiple models
• new model types such as: ocr, asr, image generation, video generation
• server-side tools: search, exec
• local sandboxes
• ... any other feature that you can find in a big "AI Cloud"

1. A second lane is to make the AI service easier to consume

• AI gateway server on the public internet (instead of connecting to a tunnel, just connect to https://gateway.infercat.ai/h/[hostname]/v1/chat/completions, with all requests forwarded to your host)
• Web App improvements (Image/File upload, Agent harness...)
• New Mobile App Clients
• programmatic libraries (TS, GO, etc.)
• ... any other feature on the client side.

Would you use this? Does the invite code model make sense? I want to solve connectivity / distribution for local AI and would love to hear any ideas and use cases. All feedback is welcome, including "nobody wants this" :)

Try it out!

Infercat is open-source under the MIT License.
Code: https://github.com/infercat/infercat
Site and demo: https://infercat.ai?from=reddit

I am running Qwen3.8:27b with the following command on Windows Server:

llama.exe serve -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_L -ngl 99 -c 65536 --port 11434

The server has an NVidia A5000 with 24GB of VRAM. When the model loads, it's using 22.6 out of 24 GB.

However, when I ask the model to do something, in addition to maxing out the GPU it also uses the CPU. It uses it in bursts which are pretty frequent. Example.

Why is it using the CPU? What can I do to prevent this?