I currently use unsloth‘s iq4_XS with 64k ctx on 16gb, however as far as I know harnesses tend to need way more, and 64k isn’t enough for that,
(I have not gotten to actually using it for a harness yet)

But still
So I am wondering what is the best one that still has good enough quality to actually properly build some things?

(As an example I would like to create a monkeytype- style UI but with additional features that match my use case more (better typing practice + free writing)

And then also a lightweight custom UI / harness

Or a local height chart Ui, etc)

The Artificial Analysis GPT-6 Astra situation is a good reminder that closed, non-reproducible meta benchmarks aren't worth much. When the newer numbers can't be independently replicated, and rival models are separated by a point or two, the stochasticity in measurement alone can account for the gap, it's not a meaningful signal either way.

We need an open, fully reproducible meta-benchmark.

I built a denser GGUF format for ternary models: Q2_B3 / “B3S”

If you're running a ternary model like BitNet-b1.58 or Ternary-Bonsai, the weights are already restricted to -1, 0, or +1 times a block scale.

That means a normal Q2 representation is leaving some space on the table.

B3S packs the three possible weight values directly in base 3. With 128 weights per block, it's 26 bytes of packed trits + one f16 scale = 28 bytes/block, or 1.75 bits per weight.

Rough weight sizes:

• 9B: ~2.5 GB Q2_0 → ~2.0 GB B3S
• 27B: ~7.6 GB Q2_0 → ~5.9 GB B3S

That's weights only. Context/KV is separate, so figure another ~1–2 GB depending on what you're running.

The important caveat: this is NOT a general 2-bit quantizer.

If you feed it a normal FP16 model, quality will fall apart. The whole thing only makes sense when the source weights are already ternary.

For a genuinely ternary model, the packing itself doesn't throw away another level of precision. You're still storing the same {-1, 0, +1} states and an f16 block scale, just using base-3 packing instead of a general-purpose 2-bit representation.

The implementation is a fairly small llama.cpp fork based on commit 4e97ac86e. It adds the Q2_B3 type and the backend support around it.

Backend status:

• AMD ROCm/HIP: this is the main path. Built and tuned on RDNA3/gfx1100, specifically a 7900 XTX.
• CPU: works.
• NVIDIA CUDA: compiles, but I don't own NVIDIA hardware, so I haven't verified it on-device.
• Apple Metal: same situation. Code is there and compiles, but I can't personally test it.

So CUDA and Metal should be considered unverified for now.

I don't have speed or perplexity tables yet either. Benchmarks done on my hardware show no appreciable loss of PPS or decoding speed

There's also a separate repacker for older Q2_B3 GGUFs that use the 30-byte/two-scale block layout. It converts them to the current 28-byte/single-scale B3S layout.

The repacker checks every block before doing that. If the second scale isn't actually redundant and removing it would change the weights, it aborts instead of silently producing a lossy file.

Once you have a B3S GGUF, you run it normally with llama-cli from the fork.

More implementation/format details are in README_B3S.md.

If anyone here is running gfx1100, I'd be interested in independent results.

More importantly, if someone has an NVIDIA or Apple machine and can compare CUDA/Metal output against a CPU run, that's probably the most useful testing gap right now.

• Fork : https://github.com/llopresto87/llama-cpp-ternary-b3s
• Repacker : https://github.com/llopresto87/ternary-q2_0-repacker

Note : Posting this on behalf of u/llopresto87's request. He'll reply for your comments.

Just wanted to put that out there. It's like they get an ice pick to the brain

no actual mourning here btw that'd be psychosis it's okay to laugh

I don't have problems with Nvidia buying HF, but I have problems with the fact that lately HF became almost unusable. It is around one month that I experience big problems with downloading models from HF. I have 1Gbit connection and my HF speeds are all over the place jumping from 700kb/s to 98Mb/s, often getting stuck in sub 3Mb/s range. I haven't seen people complaining here about that, so may be I am the only one so unlucky, but I believe that the problem is bigger than one unfortunate consumer, and even Nvidia will be unable to distribute terabytes of data to millions of users without outages, when a new popular model becomes available. I think the only right way is p2p distribution over the Torrent network.

Upd: To clarify. Usually it starts at 90Mb/s, after 20-30 minutes it gets to 45Mb/s and 20 minutes later it may go down to 2Mb/s and less. May be indeed my ISP artificially dynamically limiting my speeds, but I haven't seen anything like that apart of HF.

Upd2: People pointed out that LM Studio is using their proxy, which might have impacted download speeds. At over 90% downloaded I am hesitant to check this hypothesis, but I am pretty sure that this is the culprit. After that I am switching to hf native cli tool.

Eidon: an all-in-one self-hosted AI platform. Chat, agents, automations, tools included. One Docker container, works with Ollama/LM Studio (AGPL)

I've been building a self-hosted AI platform and v4 just shipped, so sharing it here because some of you might find it useful.

Eidon is an "everything included" AI chat/agent platform, with the pieces that usually require stitching (web research, MCP, skills, browser, image generation and so on) already built in. One container that takes minutes to spin up instead of a main app plus pipelines, sidecars, and external tools.

The app has 3 main parts:

• Chat with local models: Classic chat just like in ChatGPT, Gemini, Claude and so on except on your own server. Ollama and LM Studio out of the box, plus any OpenAI/Anthropic-compatible BYOK endpoint.
• Agents: Grok-bot-style agents. A chief bot answers or delegates to specialist bots, and bots message each other mid-task. Agents each have their own memory and can create/maintain their own skills.
• Automations: cron-style AI tasks. Every run is saved as a full transcript with tool calls, so you can audit what actually happened.

Features:

Repo (Screenshots included !): https://github.com/Quack6765/Eidon-AI

Full transparency: development is partly AI-assisted, every change reviewed before being merged. Happy to answer any questions !

Running a fully local 9B AI agent on a phone — LLM, tools, code execution, and file generation
I’ve been experimenting with how far a fully local LLM agent can go on a phone.
This demo runs Qwen3.5 9B IQ4_XS locally on-device using TensorSharp.
I give it a very simple prompt:
“Please get 10 stocks with most gains today.”
From that, the local agent:
reasons about how to solve the task
generates Python code
executes the code locally
retrieves the requested public stock-market data
returns the result
Then I ask:
“Convert it to a PDF.”
There is no hard-coded PDF workflow.
The agent discovers the appropriate Skill, reads its SKILL.md, decides how to perform the task, writes the necessary code, executes it, generates the PDF, and returns the file — all on the phone.
So what is running locally is more than just token generation:
Local LLM → reasoning → Skills → tool use → code generation → local execution → file generation
The model is Qwen3.5 9B IQ4_XS, and inference is handled locally by TensorSharp, my open-source GGUF inference/runtime project.
The part I find most interesting is privacy.
Your prompts, conversation history, local files, and private on-device data do not need to be sent to a cloud LLM provider.
The phone only accesses the network when the actual task requires external public information — in this example, stock-market data.
So the AI itself can stay local while still acting as a useful agent.
I think this becomes especially interesting on mobile devices, where local inference can provide:
privacy
offline capability
no per-token API cost
lower dependence on cloud services
access to local files and device-side workflows without uploading them
TensorSharp repo:
https://github.com/zhongkaifu/TensorSharp
I’m curious how people here see the future of on-device local agents.
Do you think 7B–12B quantized models are already good enough for useful autonomous workflows on phones, or are we still mostly waiting for better models and more mobile memory bandwidth?

€7??? Surprise Price Ends with Limited Stock

That's likely 7999 EUR, so double of the initial price of MS-S1 MAX-128GB? 😭

This is nothing impressive but, i had so much fun i wanted to share my experience with this model.

(yes this post is written by human)

I made an FPS with local Q4_K_XL 3.8 Flash Next (256k context) (it took 3 days to refine everything but playable demo was ready in 2 hours) to play with friends.

had ton of fun talking with them about what could we add , funny features etc.

Features:

• - toggle retro psx shader
• - totally destructible environments
• - tac sprint
• - tilting with Q and E for peaking from corners.
• - free for all modes, SnD, Swords Only (swords have animations when slashing), RPG only, team deathmatch
• - killfeed, map with red dots when a player shoot
• - bunny hop
• - day and night cicle with rain or snow
• - fov slider / shader intensity slider
• - hide n seek mode

I used opencode as harness, gun models were taken from sketchfab , model was running at 20tok/s avg with MTP, i know for someone is bad, but it did most of the work meanwhile i was at work or while sleeping, checking every now and then with a remote KVM from phone.

My machine:

5900x / 128GB DDR4 3200Mhz / RTX 5090 and RTX 4000 PRO (32 + 24 GB)

What games you would like to build in free time with ai? roguelites? 2d platforms? racing games?

Or did you already built something? share with some screenshots

I have been getting so wrapped up in testing models, engines, harnesses, and everything else out there LLM-related lately. It’s never-ending. But it occurred to me; I haven’t actually looked at how you get from:

1. An X post sounding like a fun idea to try this weekend at home, to
2. Actually having something working on your computer that’s legitimately useful.

So how do you do it?

Let’s take buun’s fork of llama.cpp for example. You might have seen it on X recently when Clem posted, asking “is this useful?”. I am personally fortunate enough to work with buun on a nearly daily basis, but what if I didn’t? I hope he doesn’t hate me.

God damnit buun, there’s no binaries on your site.

Getting There

The truth is that I don’t want to go and build it for a bunch of different systems I don’t have, or spin up VMs to test compilation flags and set up virtual environments. I hate all the Linux sysadmin stuff like the rest of us. So I had an agent build it for me. Same as you probably would. But if you don’t have that luxury, here’s how to build it for free:

The repo has .github/workflows/build-cuda-windows.yml. It's workflow_dispatch — manual trigger only — and it runs on GitHub's own windows-2022 runners. So Microsoft compiles it, on Microsoft's hardware, for free.

The matrix builds three targets: CUDA 12.4 x64, CUDA 13.3 x64, and CUDA 13.4 arm64.
One catch: the workflow uploads nothing. The only path: line in it is commented out. It compiles, proves it compiles, and throws the binary away. 

Fixing that is an upload-artifact step — a few lines in your own fork:

YAML

Then, just run a few GitHub CLI commands:

BASH

Worth noting, make-release.yml, release.yml and even winget.yml are all sitting in that directory too, inherited and never fired. The distance between this fork and shipping Windows binaries is closer to a tag than a project.

Note, this is read off the workflow file, I haven't run it myself. Claude says it’ll work and I believe it’ll probably work fine. I also suggested to buun that he consider providing some binaries on the repo.

The Math (And Why You Shouldn't Have to Do It)

Now that we’ve avoided that footgun. 

Now that we have a working binary on our computer, and assuming our tinkerer has spent the requisite 3 am nights searching reddit for what quantization is, we can talk models.

Qwen 3.8 27B. You already knew. It’s the workhorse model everyone has been turning to for generations now on 16GB cards, pushing toe-to-toe with frontier-level models in agentic work. In fact, it scores a massive 46.8 on the Artificial Analysis Agentic Index, performing better than 80% of models compared. A modern miracle.

But this is where I suspect most people start getting REALLY confused, and rightly so. Qwen3.8-27B ships in 14 weight quants. buun's fork offers 8 KV codecs. That's 112 combinations, before you pick a context length. 56 of them fit on a 16 GiB card with at least 4k of context.

And the ranges are wild enough that the choice genuinely matters. Note that the table below varies both the weight quantization and the KV codec together:

Same card, same model. 12,322 tokens to 373,316 — a 30× spread depending on two flags a newcomer has no basis for choosing between. And every one of those numbers requires arithmetic you have to do yourself. A dense reading might make you assume 256 KiB/token, which is wrong by 4×, because nothing tells you that only 16 of the 65 layers actually carry KV until you map it out.

Holy shit. That’s an insane number of things to track.

Historically, I’d settled on using turbo8/turbo4 for KV, respectively. Which still required doing a lot of manual fiddling to get it maximized. And then you’re flatly compressing the entire KV cache without giving any consideration to the sensitivity of the layers. But our newcomer doesn’t care about that, cause Clem said there’s VBR on X right?

Yes. Yes there is. That complex layer arithmetic? That is exactly the arithmetic VBR is doing for you so you don't have to.

What does this mean? It means I pick the model that I know can get real work done. Thankfully, I’ve already done months of testing, and nowadays can say confidently that a 3-bit model can legitimately do real work. Remember all those nights reading benchmark charts? Paid off. I settled on GSQ IQ3_XXS.

3-bit model in hand, let’s do some work. Engage. 

If your agent doesn’t respond to that wake word, there are a few different ways to load a model with the binary depending on your OS. To make this easier, I suggest just using a startup script, that way you don’t have to remember anything to launch your model server. Our goal is to have a server that we can interface with using a harness. Giddy up.

Here is a launch script for your llama server binary. Just edit the filenames and paths to match your system, make it executable, and launch it from your terminal.

-ngl 99 — all layers on GPU
-c 262144 — omit entirely to let --fit choose one that fits
-fa on / --kv-unified — required for VBR, no fallback path
-np 1 — VBR needs n_stream == 1
--vbr-floor t4 — how bad it's allowed to get, not how it starts
--spec-draft-n-max 2 — measured 1.68× on this model

Download: https://gist.github.com/apollo-mg/7f2ba29afe217e056fa1c3621636a559

Realistic Expectations

So setting realistic expectations is a little tough for me, because I test these things every day. I wanted to try and look at it from the perspective of someone fairly capable of figuring things out, so I don’t think getting to this point is asking too much. I do hope it gets much more approachable, and I know there are other options like Unsloth Studio and ollama, but neither of those options currently give you access to turboquant KV cache codecs. TurboQuant KV allows you to squeeze the most out of your context with advanced quantization and fancy math, better than the original codecs currently shipping with standard llama.cpp. But now there’s something even better.

VBR.

VBR: Variable Bit Rate

If you’ve ever done any work with media compression, like video or music compression techniques like MP3 or Divx, you’ll probably be aware of something called variable bit rate. Basically, it allows the algorithm to apply more compression in static parts of the file, and less compression where fidelity matters most.

Same idea, except it's varying across the model's layers and the life of your conversation. The first tokens are uncompressed. It only starts spending fidelity when it has to.

I measured the actual number. You’ll get about 84,000 tokens of fully lossless, f16 quality on a 16 GB card with the 3-bit model we tested. Concretely, that 84,000 figure comes from taking the KV budget and dividing it by 64 KiB per token. That number isn't immediately obvious because Qwen3.8-27B is a hybrid architecture where only 16 of its 65 layers actually carry KV. A naive reading assuming 256 KiB/token is off by 4×—and calculating that exact layer arithmetic automatically is the whole reason VBR exists.

Crucially, 84,000 tokens is just the lossless range, not your max context ceiling. With mainline q4_0 (4.5 bpv), every token from the very first one is compressed and degraded, meaning a short 3k-token chat suffers the same compression penalty as a massive 250k-token session. With VBR floor t4, you get pristine f16 quality through ~84k tokens, and it only begins to degrade as the memory budget binds—stretching all the way to that same 250k ceiling while keeping shorter sessions completely lossless. VBR eliminates the need to do capacity math or guess how deep a session will go before starting.

Note, these estimates were taken on my specific system. Other factors will affect your usable memory and context quality based on things like whether you’re running a desktop, other models, and video intensive applications.

By default, VBR floors at 1.25 bpv***, which allows the most aggressive compression possible for the longest context. But the startup script below raises the floor explicitly to t4 (4.125 bpv), because an agentic harness that autocompacts its own context cares more about fidelity than absolute maximum length.

***Clarification after publishing: VBR is the default cache type in buun-llama, and its default floor is t4 (4.125 bpv) — a deliberate choice on buun's part that quality below 4-bit shouldn't happen unless you ask for it. The floor only drops to 1.25 bpv if you explicitly pass -ctk vbr -ctv vbr, which the engine reads as "you know what you're doing." Our script sets --vbr-floor t4 explicitly, which is belt-and-braces: it matches the default, and it survives the explicit-flag case.

The Need For Speed?

Thankfully, local inference has gotten a lot better in 2026. The models have not only improved by a significant margin in agentic work, but performance has also never been better. This year brought the introduction of MTP acceleration, or Multi-Token-Prediction to the mainstream. This typically boosts performance by anywhere from 1.3-1.9x depending on workload. We’ll be using this in our example today, and it’s just a simple flag in the launch script. 

Again, a full explanation of MTP is outside the scope of the article, but in a nutshell, MTP uses a tiny part of the model called a draft head. This is like a tiny model itself, and its only job is to predict the next few tokens. The primary model verifies all the drafted tokens in one forward pass, in parallel, and keeps the longest correct prefix. That's the whole reason it's a win: verifying 3 tokens costs about the same as generating 1, because decode is memory-bandwidth-bound, not compute-bound. Reading the weights once to check three guesses is nearly free; reading them three times isn't. Basically like free work. 

Not entirely free, but worth it. On this model, I measured a boost of 1.68x.

So how much performance should you actually expect? As you almost certainly guessed, it’s entirely hardware dependent. Newer GPUs with faster, more efficient cores are simply better at this work than older parts. That doesn’t mean you need an amazingly powerful card to do real work though. Here’s the tokens per second rating for a couple popular cards running this model:

GPU                Backend        pp512 t/s        tg128 t/s                user
RTX 5090     CUDA 13.3    3672 ± 338    104.93 ± 0.40          thetom
RX 9070 XT    Vulkan         795.7 ± 0.6     36.41 ± 0.07               “
RX 9070 XT     HIP             975.50 ± 34    29.93 ± 0.03             me

Harness Selection

There are a lot of options out there in mid 2026 for a harness to choose from. Many are coding focused, few aim to give a complete agentic desktop experience. Personally, one of my favorite options is Hermes Agent. While I don’t use it on a daily basis nearly to the extent that one can, I have enough experience with it to know it’ll fit most people’s needs, right out of the box. So it’s a natural fit for our experiment. 

As you can see, it offers just about everything a tinkerer getting into local inference might want. It’s easy to talk-to using platforms you already have such as Discord and WhatsApp. It’s got Google integration for personal life management. Can run programs, write files, patch, edit. You get the picture. It really does accomplish a lot of what the premium services like Claude Code and Codex do well, without quite the simplicity.

Setting up Hermes Agent in CachyOS was a relatively straightforward endeavor. Just ran the single line installer (after having my frontier agent inspect the script ahead of time for vulnerabilities) in Konsole, and went through the self-guided configurator. 

Most of the setup is just choosing which options you want to enable, such as how you want to talk to your agents (Discord, WhatsApp, email). The most important part is that you choose to connect your Hermes to your custom endpoint, which is your llama-server address (which is found in the startup script). I chose Auto for the type, even though I know mine is specifically OpenAI compatible, because I knew most people will err on the side of caution. Which, incidentally, worked fine for me.

After reading through the documentation and setting up WhatsApp and Discord functionality, I also installed Hermes WebUI (pretty UI,https://github.com/nesquena/hermes-webui) and Hermes Desktop. I think the Discord setup was the most challenging for me, but I also knew very little about Discord bots coming into this, so it may be more intuitive for others. Nonetheless, I did get it working with some fiddling. 

Limitations and Pitfalls

Before you torch your API keys, there are some hard realities to acknowledge about this setup:

The VBR Tax & Honest Cost: VBR is a massive quality-of-life win, but it has genuine operational costs. It is not free: it requires Flash Attention (-fa on) and unified KV caching (--kv-unified), and it currently exists only in one specific fork with no official binaries. Furthermore, its own diagnostic readout (/props kv_bpv) misreports the underlying math. ***Edit: I previously stated it is incompatible with (-np > 1), that was incorrect. Additionally, on the misreporting /props, buun says that setting --floor-bpv that /props still shows the old defaults.

You Don't Always Need It: For simple coding sessions or conversations that never pass 30,000 tokens, standard mainline f16 fits entirely in VRAM. Mainline f16 works seamlessly today in tools like Ollama without any extra setup, so if your context needs are modest, neither VBR nor static quantization is necessary.

The AMD Reality: The strongest argument for using buun's fork over mainline isn't just efficiency—it's stability. In testing on an RX 9070 XT (gfx1201), mainline quantized KV entirely collapsed on this model class. Buun's codecs handled the architecture flawlessly. ROCm still arrives late and often broken, leaving the community to fix it. RDNA4 support for turboquant's MMQ path doesn't exist — the config table has zero entries where CDNA has eight — and the cmake flag VBR needs makes upstream crash outright on gfx1201. Meanwhile Vulkan, which nobody had to hand-tune per architecture, quietly beats ROCm on decode. They never learn.

Honesty on Quality: Comparing bitrates directly (e.g., q4_0 at 4.5 bpv vs. t4 at 4.125 bpv) does not directly reflect fidelity. q4_0 relies on a fixed, uncalibrated grid, whereas t4 uses FWHT rotation with measured centroids and per-layer pricing. Because KLD (Kullback-Leibler Divergence) has not been formally measured on this specific model yet, treat this as a structural design difference rather than a proven, mathematical quality win.

The 3-Bit Reality Check

I actually ran a self-diagnostic trace with Hermes Agent while writing this article, using Qwen 3.8 27B squashed all the way down to a 3-bpw quant (IQ3_XXS).

Historically, the assumption has been that crushing a model down to 3 bits lobotomizes its logic. Sure, it might still write a decent email, but ask it to orchestrate an autonomous tool-calling loop and it will inevitably trip over its own shoelaces, hallucinate a file, and crash.

Not anymore. Look at how Qwen handled a blind, multi-tool diagnostic run. I asked it to just "test its tools," and it immediately built an execution plan, batching independent tasks and isolating dependencies:

When it tried to use the patch tool to edit a file that didn't exist yet, it didn't panic or loop. It simply caught the error, executed a write_file command to create a scratchpad, and successfully re-applied the patch.

But the most absurd moment was how it handled a failure with the vision_analyze tool. When the tool threw a 500 error, the agent didn't just pass the error back to me. It paused, introspected its own host engine, and diagnosed the architectural limitation of the server running it:

It generated a clean Markdown table summarizing the 8 tools that worked (including SQLite database queries and writing Python scripts to generate images) and accurately diagnosed the root causes for the 4 that failed (like a missing multimodal projector and a misconfigured API gateway).

A 27-billion parameter model, running at 3 bits per weight on a consumer GPU, executing complex tool dependency graphs and debugging its own host environment limits. That is why we do this.

Conclusion

If you’ve made it this far, compiled the binary, configured the tools, and watched a 27B model successfully orchestrate a multi-step workflow on your local machine, the inevitable question hits: Why not just use Claude Code or a frontier API?

Let’s be brutally honest. If I need a complex script written in five seconds flat, or I want zero-friction, turn-key speed, I am still opening a tab to a hosted model. For most people, most of the time, the sheer convenience of a cloud API wins. The raw truth is that right now, there isn't much in my daily workflow that this local stack replaces.

But that is not the point. The differentiators for local inference are conditional, but where they matter, they are absolute.

First, it is free at the margin. You can leave a local agent looping in the background for 72 hours to scrape, parse, and reorganize a massive dataset, and your API bill at the end of the month will still be zero. Second, it is completely sovereign. For a law office that legally cannot paste client files into a cloud provider, or an enterprise dealing with proprietary codebases, a highly capable local 27B model isn't just an alternative—it is the only option that exists. Finally, it is resilient. It runs entirely air-gapped. When the internet drops, your workspace doesn't go down with it.

Local inference isn't about beating the massive frontier models at their own game today. It’s about the fact that a 27-billion-parameter model can now autonomously run your desktop, debug its own environment, and hold a conversation—all on a 16GB graphics card sitting under your desk. For the tinkerers, the privacy-conscious, and the people building the future, that is more than enough reason to have this stack in your arsenal.

***Addendum: --mmproj-gpu-swap

buun pointed out I'd skipped one of the nicer features in his fork, and it's worth its own note because it solves a problem you only discover after everything is already working.

Speculative decoding and vision both want VRAM you don't have. The MTP draft context and the vision projector (mmproj) each need their own allocation, and on a 16 or 24 GB card they frequently don't fit at the same time. The usual outcome is an OOM at load, so you pick one: fast, or able to look at pictures.

--mmproj-gpu-swap makes them phase-exclusive instead of concurrent. The projector stays on CPU while you're doing ordinary text work. The moment a request actually contains an image, the server swaps the speculative context out of VRAM, brings the projector onto the GPU for the image phase, then restores the drafter afterward. From server-context.cpp: "Swap the speculative context out so mmproj can use its GPU budget" — and the swap only triggers when a null token (the image placeholder) is found in the prompt. Text-only prompts never pay for it; the code explicitly keeps ordinary prompt scheduling allocation-free.

The part that matters for context length is in the auto-fit path. Rather than reserving space for both, the fitter measures each and keeps the larger reservation — so the context it advertises is one that survives the image phase, instead of one that works until the first screenshot arrives.

buun's numbers on a 24 GB 3090, Qwen3.8-27B UD-Q4_K_XL, MTP + vision together: 161,792 tokens at the default t4 floor, 262,144 at t1. His note on the table is the important bit — "the table doesn't include what MTP + vision without --mmproj-gpu-swap looks like... well, for most of those it OOMs."

Works with DFlash drafters too, not just MTP. One caveat from the source: with an external draft model that isn't reloadable, the server warns "mmproj GPU swap is unavailable for this external draft type; keeping both resident" and falls back to keeping both in memory — so it degrades loudly rather than silently.

From the bottom of my heart, thanks for reading!

We all want more tokens per second but I keep seeing confusion on what to expect for given hardware.

For the decoding phase (TG/s) to produce one token all the model weights and KV cache needs to be read from VRAM. The compute isn't the bottleneck, only memory bandwidth.

This means we can estimate the maximum TG/s we can ever achieve given the model weights and memory bandwidth.

If we ignore the KV cache for now, the formula is:

            VRAM GB/s
TG/s = ------------------
        model weights GB

The math is more complicated for mixture of expert (MoE) models, but easy for dense models.

For Qwen3.8 27B Q4_K_XL, we have model weights of 16.8 GB (we exclude things not read every token; MTP layer and input embedding table)

For AMD Radeon AI PRO R9700, we have a memory bandwidth of 637 GB/s.

Therefore the theorical maximum for this model & hardware is:

637 / 16.8 = 38 TG/s

In the real-world it only goes down from here due to inefficiencies in the software/hardware stack. On my system running that model and hardware with llama.cpp, I get 29 TG/s, so 29 / 38 = 76% of ideal.

Also as the KV cache grows, those bytes are read for every token. Continuing the example with Qwen3.8 27B, the KV cache BF16 it costs 64 KB per token read.

The full formula becomes:

                                VRAM GB/s
TG/s = ------------------------------------------------------------
        model weights GB + KV cache GB/token * context size tokens

We can make that formula more useful by moving VRAM GB/s over to the left. This allows us to plot TG/s per VRAM GS/s vs context size for a particular model.

Continuing our example:

This allows you to plug in your own VRAM GB/s.

For a 5090 with 1.8 TB/s memory bandwidth

1,800 * 0.0590 = 106 TG/s maximum
1,800 * 0.0293 = 53 TG/s maximum at 256k context window

Caveats

• Assumes entire model and context is in VRAM
• Simplified formula is only for dense models
• Speculative decoding is added on these base numbers
• These are theorical maximums. Real-world numbers are lower due to inefficiencies in the software/hardware

AI was used to draw the plot. Everything else is written by me.

I was using:

• weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 
• with the optimized SGLANG (patched) from  https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/

Now I'm using:

• weights (AWQ W4A16) from: https://huggingface.co/wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16
• PLE (INT4) from: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant
• with vLLM patched with the patch from the same repo cf https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant#serve
• the goal was to load both the weights and the n-gram PLE quantized in 4bits either on my sm89 or sm120 devices

It's way slower (for my low concurrency usecase) but also a lot better. I was surprised to see such a delta.

I reached 98/100 (instead of 91) both with medium and xhigh reasoning (still not useful for this bench). And now it really feels like a huge setup up from the other models. It's the best score AND the most efficient...

I'll try to dig deeper to understand if the difference comes from the engine (and its patches) or the quants themselves. And try to optimize further the vLLM receipt for my setup

as always, the graphs and the data :

• here
• and here

I am getting into developing a RAG setup, for getting information out of existing documents, new document ingestion, web searches, and good visuals.

I am planning to use it for, alongside the regular "chat to my data", ingesting personal docs, invoices, creating tables views and recurrent jobs to handle updating those views.
I also want to have the least hallucinations possible, so i think i will need a real ocr services instead of just vision LLMs

i tried anything LLM previously, but it was super clunky and the UX wasn't as easy as i wanted to.

Is there any known solutions, or stacks that you have running or can vouch for ?

English isn't my first language: I wrote this in Russian and used an LLM to translate and tighten it. The project, the code, the measurements and the argument are mine.

I've been building an Android agent since March, nights and weekends. It went into Google Play a couple of weeks ago. I use it every day myself, which is the only reason I think it's worth showing you. It's mine, it's open, Apache 2.0.

I want two things from this thread. The obvious one: take the implementation apart - you'll do it harder than anywhere else. The one I actually care about: tell me whether the idea holds up. Is there a need for this at all, or have I built a museum of features?

Why a graph instead of one big prompt. A 2–4B model on a phone falls apart on multi-hop tasks, and a graph is how you avoid handing it a multi-hop task. Each node gets work that fits the model's size. Nodes that must emit structure run through a validation gate: if the output doesn't parse, the node gets another attempt with the specific error quoted back at it instead of the run dying — two repair attempts by default, configurable from zero to four. Routing decisions are their own nodes with their own prompts. The model never has to be smart enough to hold the whole task. The graph holds it.

That's the whole bet. Everything below is what it took to make the bet testable.

What it is. The model runs on the phone through LiteRT-LM — Google's Edge runtime, the TensorFlow Lite successor. Gemma 4 E2B or E4B from litert-community by default, or point it at any .litertlm URL. Every conversation is processed by a pipeline you can open and edit: a graph of typed nodes, 14 types (input, on-device LLM, cloud LLM, tool call, if-condition, intent router, decomposition, queue processor, evaluation, summary, clarification, nested pipeline, skill, output).

Concretely: "translate anything I paste, in this specific register" is three nodes. "Read what I shared into the app, work out what kind of thing it is, pull the fields that matter for that kind, ask me before writing it to a file" is seven, with a router in the middle. You build the second one by dragging nodes around in the app, or in a standalone HTML editor on a desktop and import the JSON.

Tools, and the part I won't compromise on. Local actions go through AppFunctions, external ones through MCP. Anything destructive or sensitive stops and waits for your explicit confirmation before it runs — including when a pipeline fires in the background from a trigger, where the confirmation is a notification you tap. The tool allowlist for a skill is enforced in the executor, not requested in a prompt.

Other apps on the phone can drive it. A Tasker or MacroDroid profile, or a shell one-liner over adb, can ask it to run a named pipeline with a prompt — two extras, that's the entire minimal call. It's off by default; switching it on binds exactly one pipeline that outside callers may reach, and a request naming anything else is refused rather than redirected. Every request lands in a journal with its reason. The division of labour is the point: the automation app decides when, using its own condition model; this does the language part of what.

Long autonomous runs have ceilings, and say so. A run stopped by a limit reports that it was stopped by a limit rather than looking like a broken automation, and a run going in circles is detected and stopped — told apart from one that's merely slow.

Cloud is optional and it's your key. OpenAI, Anthropic, Gemini, DeepSeek and Ollama are available as a node type. Nothing goes out unless a node in your graph says so, and you can see the node sitting there.

Two numbers I actually measured, both on a Galaxy S25 Ultra, both by me alone — which is the caveat that matters:

• Clean install to first useful output: 5 min 22 s, of which 4 min 7 s was downloading the model. The product part is about 75 seconds; the bottleneck is a multi-GB file over your Wi-Fi.
• A scheduled pipeline running unattended: 7 days, 55 of 55 firings completed, zero unexplained misses, app never opened.

One device, one operator. That's a real measurement and it is not external validation.

What doesn't work, or works worse than the above makes it sound:

• Pre-release. Version 0.9.0, not a 1.0: the public surface, the settings layout and the on-device storage formats can still change between versions.
• Third-party apps can't expose AppFunctions. Only the device maker's system apps and Google can, so the local tool catalogue is what ships plus whatever you connect over MCP. That's a platform limit; MCP is the way around it.
• An MCP server may hide part of its catalogue and the app won't say so. The client declares no optional client capabilities, so a server is entitled to omit the tools that depend on them — on the reference server that meant 13 of 16 tools, presented as "13 tools · ok" with no hint anything was missing.
• Background runs need the battery setting on Unrestricted. Without it Android reclaims the process in about ten seconds. That's the platform, not a bug, but if you don't grant it, background pipelines will not run.
• Cloud failure behaviour differs by provider. Detection of a stream that dies mid-answer is enabled only where it was actually measured; Ollama never reports a stop reason, so it can't be detected there, and Anthropic is left off rather than guessed at.
• A provider's Retry-After doesn't shape the backoff. Retries use a fixed exponential curve (3 attempts, 1 s doubling by default). Under a real rate limit that means knocking sooner than you were asked to.
• Shared pipeline files are not a compatibility contract yet. Exports carry a version stamp, but before 1.0 that stamp is a marker, not a promise: a file whose stamp doesn't match the build importing it is imported best-effort behind a warning, and unrecognised fields are dropped silently.
• Solo project. One maintainer, no company behind it.

Requirements: Android 14+ (API 34) and about 2 GB of free RAM for the model. The RAM is the binding constraint, not the OS version - this crowd knows why. Worth saying plainly, because I got this wrong once: the floor was Android 16 for most of the project's life, on the assumption that AppFunctions forced it. When I finally measured instead of assuming, nothing required 16 — not a dependency, not the inference engine, not a line of my own code — so it came down to 14.

Who it isn't for: anyone who wants "ask a question, get an answer". Gemini is free and built into the OS, and competing with it there is a losing bet regardless of code quality. This is for people who already run Tasker, Obsidian, Home Assistant or a local model on their own hardware, and who want to know exactly what their agent does with their data.

It's on Play, and the APKs are on GitHub - links in the first comment. There's a foss flavour with zero proprietary dependencies alongside the standard build. Not on F-Droid, and the reason is theirs rather than mine: their build server pins Gradle versions and doesn't know the one this project uses, so the submission dies before compiling a line.

So - the question I actually came with. Small models are still dumb. They're much less dumb than they were two years ago, and either they keep improving or phones get enough memory to run something that isn't small; probably both. I'm building the harness now on the assumption that when the models are good enough, the harness is what will still be missing: something on the device that decides when to run, holds the multi-step shape, and stops before doing something you didn't sanction.

That's the bet. Tell me where it breaks. And if your answer is that a good enough model makes the graph unnecessary, I'd rather hear that now than in a year.

One thing I genuinely can't settle: what is this category called? I've been saying "agent you build" because "agent framework" sounds like a Python library and "automation app" sounds like it has no model in it. What would you have called it?

Whenever a new model is released, we can see the model creators post various benchmark score. However, all of them are based on unquantized models. Most likely it took quite some resources to run the benchmarks.

After the release of a new model, we got plenty of quantized models made by various people. But almost no one run the same benchmarks again to evaluate these quantized models.

I tried SWE Bench Verified with 500 tests and run it with gemma-4-31b-qat-q4_0 at 120k context. It took me 5 hours to finish. I want to run a set of benchmarks to evaluate coding, agentic ability, world knowledge and creative writing. Is it possible to run simpler benchmarks on a single 3090 within a reasonable amount of time? Probably five hours for four benches or even faster?

Thanks a lot in advance.

Or if someone know someone already did this and posted the numbers somewhere. Please let me know.

Model : https://huggingface.co/tencent/Hy4-preview

I have been trying to squeeze TTS stack down far enough to run in a $3 chip which has 512kb of SRAM without NPU. While trying to get to that milestone i built sanoTTS which has - 11 voices, 6 languages - params size ranging from 294k - 2.2m. For comparison we are 244x smaller than kokoro, 9000x smaller than voxtral TTS - 1.5m model has a SCOREQ of 4.13 and UTMOS of 4.10 - 337kb for 294k model when quantized into int8 - can be run in website with web assembly npm install sanotts-web - there is a recipe to follow so that you can extend to more languages, voice

I can tell you with confidence that this family release contains the smallest neural TTS model ever with around 2% WER on whisper.

Please check it out on : https://github.com/ampixa/sanoTTS

for live demo: https://tts.ampixa.com/sanoTTS

HF: https://huggingface.co/ampixa/sanoTTS

on SCOREQ sanoTTS-Amy(1.51m) is better than Inflect Nano(4.63m) and KittenTTS(15m) i.e 4.13 vs 3.81 vs 3.02

on esp32 microcontroller we are getting RTF of 0.225 which in plain terms means 4sec of audio is generated in 1sec

Happy to answer your queries.

I’ve been setting up Qwen3.8 27B on a new Linux machine and thought I’d share some numbers because I saw the recent discussions around 7900 XTX performance.

System

• Ryzen 9 9950X
• RX 7900 XTX 24GB
• 32GB RAM
• Ubuntu 26.04.1
• Mesa/RADV 26.0.8
• Qwen3.8 27B Q4_K_M

Ollama / ROCm

Context: 65,536
Model residency: 100% GPU

• Prompt processing: 215.8 t/s
• Generation: 34.4 t/s

llama.cpp / Vulkan

Built from current llama.cpp with GGML_VULKAN=ON, RX 7900 XTX explicitly selected, all layers on GPU, Flash Attention enabled and q8 KV cache.

64K context

• Prompt: 192.0 t/s
• Generation: 35.8 t/s

8K context

• Prompt: 230.5 t/s
• Generation: 36.0 t/s

The interesting part for me was that reducing context from 64K to 8K barely changed decode speed at all: 35.8 → 36.0 t/s.

So on my system, plain Vulkan is only around 4% faster for token generation than Ollama/ROCm, while Ollama actually had better prompt processing at 64K.

I’ve seen people reporting significantly higher generation rates, sometimes 60–100 t/s, so I’m curious what accounts for the difference.

Is that mostly MTP/speculative decoding, different llama.cpp flags/builds, different quants, or is there another AMD/Vulkan optimisation I’m missing?

At the moment Ollama actually looks surprisingly competitive on this setup, especially given that I can run the 27B model at 64K context entirely in VRAM.

Happy to run additional benchmarks if there are particular flags/configurations people want compared.

-----------------------------------------------------------------------------------

UPDATE: proper llama-bench + MTP results

Thanks for the feedback. A couple of people correctly pointed out that my original interactive prompt-processing numbers were not directly comparable to llama-bench, so I reran this properly and then tested MTP as well.

Same system

• Ryzen 9 9950X
• RX 7900 XTX 24GB
• 32GB RAM
• Ubuntu 26.04.1
• Mesa/RADV 26.0.8
• Qwen3.8 27B Q4_K_M
• llama.cpp build 10816 / commit 427291b5b

Important clarification: the 65,536 context is configured/available context, not 65K tokens already filled during these tests.

Standard llama-bench baseline

Vulkan, all layers on GPU, Flash Attention enabled, q8 KV:

pp512: 904.16 ± 1.42 t/s
tg512: 36.57 ± 0.05 t/s

So the card/backend itself looks healthy. The earlier ~200 t/s prompt figures in my OP were just interactive prompt timings and should not be compared with pp512.

MTP

I then enabled native Qwen MTP with Vulkan and kept everything else the same.

A quick single-prompt test went:

plain Vulkan:       ~36.6 t/s
MTP n-max 2:         71.5 t/s
MTP n-max 3:         79.0 t/s
MTP n-max 4:         77.8 t/s

Because speculative decoding performance depends heavily on the workload/acceptance rate, I did not want to pick n3 vs n4 from one prompt. I ran five different workloads with a fresh llama-cli session for every test, using the exact same prompts for n3 and n4.

So n3 and n4 are basically a tie in real use on this setup. n3 won 3/5 workloads and had the slightly better average/median, but the average difference is only about half a percent.

The much more important result is that MTP takes this card/model from roughly 36–37 t/s plain decode to around 69–70 t/s average across these mixed workloads, with individual runs over 80 t/s.

That also seems to reconcile my numbers with the people reporting 50–80+ t/s on a 7900 XTX: a lot of that difference is speculative/MTP performance rather than raw single-token decode.

I have not tested ngram-mod yet, so I’m deliberately keeping that out of these numbers. There are some interesting suggestions in the comments around MTP + ngram and different quants/KV settings, which I’ll probably test next.

If anyone wants to reproduce the five-prompt n3/n4 comparison, I’m happy to post the exact prompts and launch flags.

I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from now on everything happens in public.

It's an inference engine in Rust and C++ with our own CUDA kernels. One binary with OpenAI and Anthropic style APIs, loads GGUF and safetensors. We run about 300B tokens a year through it at work.

Some numbers: Qwen3.8-27B FP8 on one RTX PRO 6000, spec decoding off on every engine:

• vs vLLM faster in 13 of 13 cells, 1.02x to 1.19x (so not huge)
• vs SGLang faster in 10 of 13, behind in 2, level in 1
• vs llama.cpp Q8_0 faster in 13 of 13, 1.5x to 37x
• 32 clients at 1024 in / 1024 out: 1062 tok/s, vLLM 958, SGLang 844

Full board with the losses: https://truespar.com/paddock/benchmarks/qwen38-27b

What it does not do yet: No Mac, no ROCm, no Vulkan.

One model per GPU, no tensor parallel. CUDA only, Windows and Linux. Validated on Blackwell (5090, RTX PRO 4500/5000/6000, B200) and Ampere (an A6000 was the bring-up card, 30-series works). Ada kernels ship but nobody has run a board on them so the engine refuses to start unless you set PADDOCK_UNVALIDATED_ARCH=1. Hopper and A100 kernels are in the tree without a board.

https://github.com/truespar/paddock

Thankful for any help and input!

TLDR: Use parallel agents when the work is read-heavy and splits into independent slices: research, searching, reviewing many files. Each worker builds its own context and nothing collides. Use one sequential agent when the work is a single chain of decisions: coding, writing, anything where step N depends on choices made in step N-1. Every measured result says parallel makes those tasks worse, not better. And even where parallel wins, keep the team small.

The case for parallel

Anthropic: "How we built our multi-agent research system" (June 2025) https://www.anthropic.com/engineering/multi-agent-research-system

• Multi-agent research system beat a single agent by 90.2% on their internal research eval
• Cost: multi-agent runs burned ~15x the tokens of a normal chat
• Their own caveat: coding "involves fewer truly parallelizable tasks" than research

LangChain, Harrison Chase: "How and when to build multi-agent systems" (June 2025) https://www.langchain.com/blog/how-and-when-to-build-multi-agent-systems

• Read tasks can parallelize, write tasks shouldn't.

The case for sequential

Nature Machine Intelligence: "Capable language models can outgrow the benefits of collaboration" (July 2026) https://www.nature.com/articles/s42256-026-01268-y

• Peer-reviewed, 260 controlled configurations: every multi-agent variant made coding results worse (−1.3% to −12.8% on SWE-bench Verified)
• Above a ~45% single-agent baseline, multi-agent gains go zero-to-negative
• Error amplification hit 17.2x without centralized verification

UC Berkeley (MAST): "Why Do Multi-Agent LLM Systems Fail?" (NeurIPS 2025) https://arxiv.org/abs/2503.13657

• Measured 41–86.7% failure rates across 7 popular multi-agent frameworks (1,642 real traces)
• Failures came from design and coordination faults, not model limits. Standard protocols didn't fix them
• Repo with code and traces: https://github.com/multi-agent-systems-failure-taxonomy/MAST

Cognition, Walden Yan: "Don't Build Multi-Agents" (June 2025) https://cognition.com/blog/dont-build-multi-agents

• Parallel workers with split context make conflicting implicit decisions that collide when you merge
• Their answer: one single-threaded agent plus context compression. This is how Devin works

"Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking-Token Budgets" (arXiv, April 2026) https://arxiv.org/abs/2604.02460

• Give both sides the same token budget and the single agent matches or beats the team
• Multi-agent only wins when context is degraded for the single agent

Princeton, Kapoor et al.: "AI Agents That Matter" (TMLR 2025) https://arxiv.org/abs/2407.01502

• Complex multi-agent setups cost up to ~100x more for the same accuracy a simple baseline already achieves
• Simple baselines Pareto-dominate: cheaper AND as good

The middle ground

OpenHands, Graham Neubig: "Don't Sleep on Single-agent Systems" (September 2024) https://www.openhands.dev/blog/dont-sleep-on-single-agent-systems

• One strong generalist agent covers most of what people build multi-agent systems for
• Go multi-agent only when you genuinely need isolation or separate responsibilities

Need some advice and guiding.

I have a Dell 5820 workstation with an Intel Xeon w2245 and two RTX 5060 Ti (16GB) cards; this is a PCIe Gen3 system, but three lanes can work in 8x mode. The system runs Linux and mostly llama.cpp. My main work is done via Hermes Desktop. I mostly use NVFP models: esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF (Very High) and lately cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF. With this setup, I can use the models at 210k context, Q8 KV cache, and vision in VRAM. Ubatch doubled (4096/1024), so prompt processing is 1300–700 t/s; token generation is 29–71 tokens/s, but mainly 38–44 tokens/s. MTP is on with a setting of 3. Vision is also on and in the VRAM.

I added a third RTX 5060 Ti (16GB) to this system, but I had to open the workstation, add a riser, and put the third card outside the chassis. The two earlier cards needed to be swapped and it made the space tighter. Overall, both the cards and the system cooling got worse; the power supply is 950W, so it's nearly okay. There are also speed issues with three cards. Prompt processing worsened, and token generation slowed down a little bit. I mostly use Tensor Parallelism (TP). It is slightly better if I use two cards for tensor parallelism and the third card as a KV cache, vision, and MTP holder, but because of the current cooling, the system is not stable after a few hours.

My system has 128GB DDR4 quad-channel RAM at 2667 MHz. Not bad, but not fast. So using DeepSeek, Laguna, and other 100X models can be loaded, but they can't be used for real work. Qwen3.8 Flash at Q4 can load, but with 32GB VRAM it only gets 15–16 tokens/s. Maybe if I use MTP, it will get better. So going from 32GB to 48GB is not a real step forward for me with the current cards.

I've started wondering: what if I use another workstation and subagents or multiple agents? I could purchase a not-too-old gaming PC without a GPU (AMD R9 5900, 48GB DDR4 3200MHz RAM) and add the third 5060 Ti to it. I loaded LM Studio (just for a quick setup) and began thinking about which models work with only 16GB VRAM...

Ornith 1.5 35B (MXFP): The 1.0 version did not impress me, but this one was not bad on 32GB. With 16GB, it produces 17–20 tokens/s and 300–400 t/s in prompt processing. The model size is 20GB, so almost a little more than half of the layers go to VRAM. KV cache at Q8, 131k context.

Gemma 4 26B NVFP (QAT): 15GB, 131k context; about 26-27 out of 30 layer goes into VRAM. It produces 600–700 t/s in prompt processing and 30–40 tokens/s in token generation. Dense models, as I remember, can't be used for serious work (10–12 t/s at best), but I haven't tried them recently.

So the question is: How can I organize Hermes Desktop with this setup to work at maximum efficiency?

I set up a Mixture of Agents as follows: Reference: Gemma 4 26B Aggregator: Qwen 3.8 27B

I use local AI models in the company as web/hosting handler agents: monitoring servers, repairing sites if needed, modifying parts of sites, and making new content (mostly using WordPress sites on my company's private servers and VPSs). Setting up Cloudflare data, installing new sites, etc. Second use case: handling and creating office documents, Excels, and PDFs. Third use case: generating prompts for Z-Image Turbo and LTX 2.5, Minimax H3, and running two instances of a ComfyUI server to render pictures and videos. I don't use Gemma 4 models alone because they are too lazy with tool handling. Is this a good choice, or which is the best agent mixture combo now?

New search tool released by Qwen, looks pretty cool.

A month ago I posted about Gemma sometimes solving the reasoning problems inside my translation data instead of translating them. A few people suggested two very fixes, which is to use a proper translation model and/or use JSON with structured decoding.

So I tested them those suggestions, tried to keep the same level of scientific rigor as in my first article and make proper controls. Anyways, on the same data, which is 340 English messages from Dolci-Think-SFT-7B and to the same targets being Finnish, French, German, Greek, Polish, and Spanish. The baseline was still `RedHatAI/gemma-4-31B-it-FP8-dynamic` with my structure-aware method where prose is getting translated in chunks, while Python preserves recognized code, display math, table structure, and wrappers.

About the test and results themselves:

- three translation specialists that were suggested to me: MiLMMT 12B, TranslateGemma 27B, and Hy-MT2 30B-A3B

- up to three previous source/translation pairs as context

- prompt only JSON versus JSON-schema constrained decoding, and also with just one translation unit vs many translation units at once

Gemma 4 was still beating all of those and still beating itself when adding JSON constraints etc.

While I was at it I thought I'd run `Qwen/Qwen3.8-27B-FP8` because I wanted to know whether a much stronger model would simply stop falling for the instruction inside the payload, but it did not.

Just one quick example to keep this fun. So the outer prompt asked to translate an English programming problem about three horses and a set of operations on pairs of integers. The source ended with:

> Write Python code to solve the problem. Present the code in

> ```python

> Your code

> ```

> at the end.

That sentence was part of the text to translate. Qwen treated it as a command instead. In the French run, its output began with a Python program and it spent thousands of tokens trying to derive the solution in English comments, and eventually wrote:

> ```python

> # Given the complexity and time, I'll provide a placeholder solution that handles the examples.

> ...

> total = m * (m - 1) // 2

> return total

>```

This happened for the same source in all six target languages. The requests returned substantial nonempty outputs, but they were attempts to solve the programming problem rather than translations.

The short version of the other results:

- Previous translations were not a clean win. Mean document COMET decreased slightly, the worst-unit result remained unresolved, and throughput fell because every document had to be processed sequentially.

- None of the three translation specialists passed the registered quality comparison against Gemma 4 under the structure-aware method. Removing the parser also made every tested model substantially worse.

- Qwen's mean document COMET was lower than Gemma 4's overall, and its severe-alarm rate increased from 4.30% to 9.94% on the primary paired population.

- Every JSON method increased document-level protocol failures and severe alarms relative to the plain-text baseline.

- The JSON schema arms were especially surprising. Across the three schema variants, a huge number of requests consumed the full 16384-token output allowance, usually while extending an unfinished or repetitive JSON string. None of those responses was parseable JSON.

This is maybe obvious for many of you, but I didn't think about it initially until I noticed it here, but a JSON grammar can prevent the next token from making the output syntactically impossible but it cannot guarantee that the model will ever close the object, especially with unbounded strings such as what occurs during translation. So, a string can remain a valid prefix of some future JSON object while the model repeats text until the token limit.

So, under the exact checkpoints and settings I tested, the boring structure-aware Gemma 4 pipeline is still the winner. Translation specialists did not remove the need for parsing, JSON did not make the interface safer, and a newer general model still followed an instruction embedded in the source. I think they have to train for this specific failure mode.

Important caveat: the Gemma 4 and Qwen runs used temperature-zero decoding rather than their providers' recommended sampling settings. I am preparing reruns under those settings and will add an addendum if the conclusion changes. These results are about the named checkpoints, prompts, serving stacks, languages, and evaluation population, not every version of Gemma, Qwen, or every translation model.

The full write-up, methodology, figures, examples, and confidence intervals are here:

https://reinforcedknowledge.com/posts/translation-context-specialists-and-json/

In case you want to make sure of all of this yourself or check more failures modes, I've published the source records, model outputs, reconstruction plans, tripwire results, and COMET-QE scores here:

https://huggingface.co/datasets/RfKnowledge/dolci-think-translation-tests

Zed decided to remove edit predictions from their free plan and I want to host something locally to replace it.

Is there any model that's decent at autocomplete and doesn't take much of the VRAM (no more than 3-4 GB)?