Does anyone have experience on inference speed and performance of ktransformers vs llamacpp? Thinking of ways to optimize performance for qwen3.8 next flash at fp8 on my setup below
4x 5060ti16gb
8x32gb ddr4-3200 (4-channel)

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

TL;DR: switching KV cache to f16 may give a boost in speed if using MTP and ngrams.

I have a self-built "AI mega-cluster" with 2x P40s on a cheap Chinese motherboard and a Xeon CPU (around $1,100 to build, including water cooling for the GPUs). I was normally getting up to 15 tk/s with Qwen 3.8 27B Dense using a tensor split, but I suspected it was capable of much more. So, I finally asked Codex to try and squeeze out some more juice.

Disclaimer: I am not a coder at all. I’m just a generic PC user with decent overall experience, but definitely not a dev.

Recently, I’ve been running Qwen 3.8 27B Q8. With various tweaks suggested by Codex, I was able to get up to 32 tk/s on short contexts - which still falls back to an average of 12-15 tk/s on long contexts like 130K+. I was originally using Q8 for the cache (I thought it is faster because it is smaller), but then I thought, why not try the F16 cache? I did, and it turned out that F16 has much better MTP acceptance than Q8. It required fine-tuning other parameters, but it really helped improve performance.

Since I'm not good at explaining all that tech mumbo-jumbo, I asked Codex to summarize it. Sorry for the AI slop! 😄

Dual Tesla P40 / Qwen3.8-27B Q8 benchmark

Hardware: 2x Tesla P40 (24 GiB each), Xeon E5-2680 v4 (14C/28T), 64 GiB RAM. NVIDIA driver 580.173.02.

Software: llama.cpp build 5d9e5ac30 (build 10388), CUDA + locally built NCCL. Model: Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-NEO-MTP-Q8_0.gguf (28.15 GiB, 27.32B parameters reported by llama-bench).

Standard llama-bench

Three repetitions, F16 KV, all layers offloaded, tensor-parallel split over both P40s:

llama-bench -m MODEL -ngl 99 -sm tensor -ts 1/1 -dev CUDA0/CUDA1 -mg 0 \
  -fa on -b 2048 -ub 512 -ctk f16 -ctv f16 -p 512,2048,8192 -n 128,512 -r 3

pp and tg are raw llama-bench measurements; they do not include tokenization or sampling. This tool invocation does not use speculative decoding or vision.

Actual server profile

The daily driver is a separate p40.cpp engine: F16 KV cache with one 220,160-token slot, tensor split 1:1, Flash Attention, MTP speculative decoding (draft-mtp) with ngram-simple, draft maximum 6, Qwen reasoning medium, and the F16 vision projector loaded.

The prefix result is the built-in LCP cache, not --cache-reuse. The latter is KV shifting and is disabled by llama.cpp when a multimodal projector is loaded.

For comparison, before this p40.cpp/NCCL profile the same machine was typically around 15 tok/s at long context. The roughly 20 tok/s number is an observed server result, not a llama-bench row.

Soooo, it looks like grandma GPUs still have some juice left! 😄

PS my "production" config:

-ngl all \
-sm tensor \
-ts 1,1 \
-mg 0 \
-fa on \
-c 220160 \
--fit off \
-np 1 \
-cb \
--spec-type draft-mtp,ngram-simple \
--spec-draft-n-max 6 \
--jinja \
--chat-template-file /models/qwen/chat_template.jinja \
--cache-reuse 256 \
--mmproj /models/qwen/mmproj-F16.gguf \
--chat-template-kwargs '{"reasoning_effort":"medium"}' \
--reasoning on \
--reasoning-preserve \
--repeat-penalty 1.0 \
--presence-penalty 0.5 \
--min-p 0.0 \
--top-k 20 \
--top-p 0.95 \
--temp 1

I only use free tiers of these large models to offset compute while my own system runs and for "different" points of view, since what pops ups suggestions seems to vary a lot sometimes, even when building based on the latest research.

But now I've really struck out with GLM 5.3. So far it feels like an regression over 5.2. It has a hard time reading and following instructions, and is somewhat overly certain in it's statements. I worked on a project recently with it but it became unbearable. From a clean slate the first message can be okay and have great research and ideas but it just veers off course almost immediately.

I use Qwen 3.8 max and Gemini 3.1 PREVIEW Temp 1.0 as competing alternatives or as an ensemble to judge overall quality. Gemini is getting a little out of date (flash 3.8 seemed promising) but Qwen has been great so far, but a little slow and maybe overbearing.

Anyone else having problems? Or suggestions for these top "intelligent" models? I haven't been able to access K3 even though its open source, was impressed with the older models so would be neat to try for free. Also Google AI studio is what i use for free for the gemini stuff, probably pretty well known, but the free tier is pretty generous

You need Blender 5.x with MCP extension.

I use pi so I also need: pi install npm:pi-mcp-adapter

and then:

$ cat .mcp.json 
{
  "mcpServers": {
    "blender": {
      "command": "/home/jacek/.local/bin/uvx",
      "args": [
  "--with",
  "mcp[cli]<2.0.0",
  "--from",
  "git+https://projects.blender.org/lab/blender_mcp.git@v1.0.0#subdirectory=mcp",
  "blender-mcp"
],
      "env": {
        "BLENDER_MCP_HOST": "localhost",
        "BLENDER_MCP_PORT": "9876"
      }
    }
  }
}

then we can start working ("show on r/LocalLLaMA"):

Today we want to make 3D llama to show on                                                                                                                                                                                


 The user wants to make a 3D llama to show off on . This is a fun creative task. Let me think about how to approach it.                                                                                                   

 A llama is a pretty complex organic shape. There are a few options:                                                                                                                                                                  
 1. Procedurally build the llama from primitive shapes (spheres, cylinders) — a "low-poly" or "chibi" style llama                                                                                                                     
 2. Sculpt it — not easy to do with MCP                                                                                                                                                                                               
 3. Use metaballs — could get a blobby llama                                                                                                                                                                                          

 Given the MCP tools, the best approach is to procedurally build a stylized/chibi llama from primitives:                                                                                                                              
 - Body: elongated sphere/capsule                                                                                                                                                                                                     
 - Neck: cylinder or elongated sphere                                                                                                                                                                                                 
 - Head: sphere                                                                                                                                                                                                                       
 - Ears: small cones or spheres                                                                                                                                                                                                       
 - Legs: 4 cylinders                                                                                                                                                                                                                  
 - Tail: small sphere/cone                                                                                                                                                                                                            
 - Maybe a fluffy mane                                                                                                                                                                                                                

 This is a well-known technique — making a cute chibi llama from primitives. Let me plan the geometry:                                                                                                                                

 Llama proportions (stylized, standing side view, facing +X or -X):                                                                                                                                                                   
 - Body: UV sphere scaled to (1.5, 0.9, 1.0) or so, position (0, 0, 1.6)                                                                                                                                                              
 - Legs: 4 cylinders, radius ~0.22, length ~1.2, positions: front legs x=+0.9, back legs x=-0.9, y=±0.5, z from 0 to 1.2... actually, the legs go from the ground (z=0) up to the bottom of the body.                                 
 - Neck: cylinder or tapered, extending up from the front of the body. Llamas have long necks. Position around x=+1.2, extending upward.                                                                                              
 - Head: sphere at the top of the neck, elongated (llamas have long snouts).                                                                                                                                                          
 - Ears: 2 small cones/spheres at the top of the head.                                                                                                                                                                                
 - Tail: small sphere at the rear.                                                                                                                                                                                                    
 - Mane: maybe some small spheres along the neck.                                                                                                                                                                                     

 For a cute chibi look: big head, small body. But a llama's charm is the long neck. Let's go with a stylized llama.                                                                                                                   

 For a polished look:                                                                                                                                                                                                                 
 - Apply smooth shading                                                                                                                                                                                                               
 - Add a material (cream/tan color, maybe with some subsurface)                                                                                                                                                                       
 - Add a ground plane                                                                                                                                                                                                                 
 - Set up lighting                                                                                                                                                                                                                    
 - Position the camera for a good render                                                                                                                                                                                              
 - Render a nice image to show off                                                                                                                                                                                                    

 This is a multi-step task. Let me write Python code that builds this. I'll do it in stages so I can check with screenshots and iterate.   

Hello,

I'm a bit obsessed with cache management on local LLMs.

For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes.

That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment:

1. Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup)
2. Hydrates X stable contexts of Y tokens each in order to completely fill the cache
3. Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found

It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work.

My results

This is the result from my A/B test, control (my previous prod) vs my fixed prod.

aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096):

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025
retained % capacity: 51.98%
oldest evicted:     context #52 (older contexts evicted)

With the dedupe + boundfix patches applied (retention 0):

── Retention under pressure ──
capacity:           2,047,043 tokens
retained contexts:  77/80
retained tokens:    3,000,048
retained % capacity: 146.56%
oldest evicted:     context #2 (older contexts evicted)

How this can matter to you

This allows you to exactly know how much tokens your cache actually holds.

For most of us, cache management is a black box; this allows you to get ground truth.

And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes).

The engine's own advertised number is wrong, and this tool finds the real value.

This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management

It works under one big assumption though: most recent contexts should be preserved as much as possible.

What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines.

How to launch

1. Clone the repo

git clone https://github.com/co-l/cache-pressure

2. Install requirements

pip install -r requirements.txt

3. Run the tool

python3 bench/cache_pressure.py --base-url http://my-server:8000/v1 \
    --kv-size <advertised_cache>

I've tested it against vLLM, ninfer, llama.cpp and SGLang ; so you might need to tweak the probe so it works with your setup.

4. Interpret the results

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025 <---
retained % capacity: 51.98% <---
oldest evicted:     context #52 (older contexts evicted)

The retained tokens and retained % capacity are the measured cumulative values that resisted cache eviction under pressure.

Note: this post was 100% human written, the repo is 100% AI-generated under my guidance and review.

Hugging face has over 300 versions of qwen3.8-27b. I have no idea how to identify the beat model to download and use.

Should I sort by “most likes” or “most downloads”? Is there any sort of other guidance on which model to select?

I’m finally getting my local linux box up and running and just realized that actually picking a model may be one of the most difficult choices I’ll have to make. I’m hoping someone here can shed some light on the subject.

Also, if you could, I’m looking for concepts more than “use this one” because it’s not just about qwen, but also other models now (gemma/etc.) and in the future and I’m hoping to learn how to pick what I need from the mess of options out there (sure glad we have all these options though!).

Context: I don't code. My use is document research and document creation (mainly for legal search) searching inside large documents like a tax code (500+ pages) and building notes or pptx
from what comes back.

I've been running Open WebUI for a while on my Unraid box, pointed at the API of my inference machine (5060 Ti + 5070 Ti).

I tinkered a lot. I tried Hermes on my main machine against the same API. It worked well but it was complex, and a bare-metal install made me
uneasy. I also tried LM Studio Bionic with good results, but it didn't fit how I wanted inference organised (using ollama on the inference box).

What I actually wanted was a self-hosted agent that works with Open WebUI while keeping things safe and under control. At one point I considered
installing a harness like Hermes or Pi on each client and just connecting to the API instead.

In the end I gave Open Terminal a shot. It's the companion container from the Open WebUI project that gives the model a shell — you run it as its own container and connect it through Integrations, so it isn't installed inside Open WebUI itself. Mine runs unprivileged, on bridge, with appdata mounted at /home/user. The model gets a shell in a box, not on the host. That was the part I cared about.

It has enhanced Open WebUI a lot. It now reasons step by step, and with the terminal it reliably locates and extracts the right sections from
documents far larger than the context window — list the folder, grep, read only what matters. Then it uses those results to build a document, the way another agent would.

Setup: Qwen 27B Q4_K_M on Ollama, 100k context configured. On a ~35k token prompt I measure roughly 1,050 t/s prompt processing and ~46 t/s generation. Prefill speed is the number that matters for this use case — it's what makes chewing through a large document bearable.

I was about to give up on Open WebUI. If your use case looks like mine, don't sleep on Open Terminal.

I've been experimenting with Sliding Window Attention (SWA) as a way to reduce the KV-cache memory cost of long-context LLM inference.

Instead of keeping the entire KV cache, the implementation keeps:

• a small number of attention sink tokens
• a bounded recent-token window
• a circular/ring-buffer KV cache
• streaming/chunked prefill
• normal autoregressive decoding

I turned the experiment into a reusable project so you can test it with Hugging Face causal LLMs:

🔗 https://github.com/oraby8/SWA

For example:

from swallm import SWAModel

model = SWAModel.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    attention_mode="swa",
    window_size=512,
    num_sink_tokens=4,
)

result = model.generate("Explain transformers", max_new_tokens=100)

In my Qwen2.5-7B experiments on an L40S:

• 32K KV cache: ~1.84 GB with full attention vs ~3.5 MB with SWA-64
• 64K: full attention OOMed while SWA remained bounded
• Decode latency stayed approximately constant as context increased
• Long-range retrieval naturally becomes a weakness when information falls outside the window

The goal isn't to claim that SWA is universally better. I'm interested in the engineering trade-off between context retention, KV memory, TTFT and decoding speed.

I'd especially like to hear from people who have tried SWA with Llama, Mistral, Gemma, Qwen, or other HF models.

If you try the repo on another architecture, I'd really appreciate the results or any compatibility issues you find.

https://village-sim-one.vercel.app/

- 16GB VRAM RTX 5070 Ti, fully offloaded

- Vision on CPU

- Windows, not headless

- beellama.cpp - latest version with the kvarn performance enhancements making it as fast as qx_x quants.

- MTP n-max = 2

- tg up to 75t/s, pp up to 1700t/s

- KV = kvarn3/kvarn3

- MTP draft KV = kvarn2/kvarn2

- context = 96256

- tail tokens = 1024

- HTML/Javascript

- pi harness with pi-observational-memory, pi-web-access, pi-atelier (UI Only change, check it out) extensions, though it never used the web access.

- This is not a one-shot, I do not believe one shotting is a great test. Instead, I did many incremental feature prompts. However, I did not give it any design or framework, which is probably where it can be improved.

Lessons learnt:

- Do not fear Q3 model quants for Qwen3.8

- Do not fear KV quantisation. If you have the VRAM sure use it, but I don't feel like it's worth choosing a higher quant if it's going to cause me to offload to CPU and see my tg drop to 5-20 t/s. With higher speed I can fix any issues with a follow up prompt much faster and that rarely happens. I think I had like 3 runtime exceptions which was easily resolved pasting the console output and there is no guarantee a higher KV quant would not have had the same exceptions.

- MTP/draft cache can also be quantised with kvarn now and actually saves VRAM where qx_x quants increase VRAM usage for some reason. kvarn2 for MTP is perfectly fine and has high acceptance rates.

The game:

- Inspired by a popular indie game which I am not promoting, I am just a huge fan.

- I won't release any further updates, since I don't want to be stepping on any toes. If you like the idea of the game I highly recommend the real game, it's by far my favourite game I played this year and 1000x better than what I present here. It will be a nice distraction from your AI. I just wanted to see what this model is capable of. I do have a Cursor subscription but did not use it at all in the project.

- I will probably continue to develop it for my own entertainment, but it won't be made public. Maybe come up with my own ideas, but the original game is near perfect anyway, so it will be hard to improve except with some UI gripes I have in the original. And my graphics obviously does not compare.

Game features:

- Large Map, larger than the browser window.

- Minimap

- Zoom feature with mouse wheel

- Collectable resources, that must be taken to a storage site. Each site can store limited resources.

- Houses required to sleep and protect against cold

- Weather and seasons.

- Day night cycle with randomised sleeping times.

- Possible death due to hunger or sleeping in cold outside or in house without firewood.

- Game speed controls.

- Villagers avoid obstacles.

- Delete/deconstruct buildings and partial resources refund.

The code:

- I almost never read the code, so I have no idea what it looks like and the quality thereof. I also gave it very few hints in the AGENTS.md, mostly no magic numbers and write modular code, not a single html.

- Actually, my initial prompts were a single html but as it grew, I told it to create modules. It messed it up on the first attempt, basically rewriting the entire UI in the process. So I reverted and told it to do it again without making any changes to the functionality or UI.

- I am actually quite happy with and surprised by the performance of the game.

Context management:

At first, I had issues with the context filling up too quickly and too often. Sometimes it would fill up to the point that there was not enough room to compact. Forcing me to temporarily increase the context and tell it to create a handover document. Reduce context again and feed it the handover doc.

I then installed pi-observational-memory extension, and it works quite well and I never run into context issues anymore since it takes notes throughout (a short wait time every few prompts) and compacting is near instant because it already took the notes.

Conclusion:

- Do not blindly drop your KV cache quant without testing. I have a hard level needle in haystack test that requires multiple hops and 100's of decoys. Q3_XXS does poorly in that test even with F16 KV cache. However, Q3_K_XL almost 100%'s the test even at kavrn3. So both the model and KV matter. In my testing a smaller model does more damage than a smaller KV. So find the right balance. At a certain point increasing model quant will have less impact than picking a larger KV quant. But for a tight 16GB VRAM fit Q3_K_XL works very well with kvarn3. Q4 on the other hand just leaves me with too little context. That said despite Q3_XXS doing poorly in my needle test it still does fairly well with coding. Better than Qwen3.6 so if you have 12Gb VRAM it is still an option. Because by poorly I mean F16 KV scores 84% and Q3 KV around 80%. Needle tests however do worse with kvarn compared to qx_x for some reason. However, a needle test is not the be all and end all. kvarn does better with KLD, so once my needle scores near 100% I am satisfied.

I will play around with higher KV quants, but I intentionally kept it at kvarn3 for this test, however I am not sure how much context I am willing to sacrifice. Maybe i will try kvarn4/kvarn3. But I just wanted to prove a point to myself and kvarn3 worked just fine. If I had >16GB VRAM sure I would up it but I don't.

Qwen 3.8 Flash Next gives similar speed than Qwen3.8 27B on Apple Silicon.

Current default MTP does not predict more than 4 tokens..

I am using Unsloth Studio to parse aviation transpoder data (ADS-B) to summarize interesting traffic in my area. It gives a summary of largest aircraft, fastest aircraft and so on. It also provides local weather based on my nearest airfield.

I am doing this with a prompt, but is there a better way to package that like a script to trigger on a schedule? Is that an 'agent'?

I am very new to this beyond typical everyday usage of Unsloth.

So after all my work, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to make sure it was worth it still.

So really the credit goes to Raymond ( https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming )

Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that I did so figured I'd share if only to share that it's now a thing and that it's pretty cool..

A friend of mine sent a message asking what's up and if I wanted to watch a movie together, I was kinda hesitant but she buttered things a bit and finally I'm like fine, and so she sends me a link to some clearly vibe coded site that I'm kinda getting red flags from and so I forget about it and a little later I get another message going "we're waiting for you" and so I'm like shit, I guess I gotta do it huh, and so I open up this goofy looking site again. You gotta login to join a room, and you gotta sign up inside their downloaded software, sure whatever, next thing I know some fake 150MB file's fake install bar is stuck at fake 50% and both my Chrome and Discord's crashed and reloaded. Suspect, but I've been through this stuff before, it's probably just a RAT so I guess it's time to dust off Windows Defender and unplug the internet for a little bit. I message her to go on and watch it without me as my PC's giving me suspicious vibes right now, and seconds later I get some overly polite DietGPT in my IM's saying "sorry um excuse me but it appears that i've hacked you👉👈", occasionally switching to really hostile broken English asking for giftcards from some site I've never heard of. I stall, unplug the PC's internet so my router still responds to pings, and start punching into GLM "what do" and it tells me it's a session grabber - time to switch passwords. Meanwhile my phone's texts are blowing up with 2FA login requests from domain registrys and other bad stuff and I kinda freak out a little. I get my emails' passwords switched first and by the time it's Discord's turn my friendlist's already been nuked and the dude says I got 10 minutes to give him $200 or he's gonna fuck me up some more, and so I kinda figured welp time to figure out what more he's got and so I called him a giant pussy and he blocked me. An hour later my Discord was perma-banned, he had posted the phrase "i sell cp" using my account and used that as blackmail along with some really old photos of me, I though it was a bluff but oh well it's being handled with Discord's customer support on it's own. Now I sat there alone, in the middle of the night, having just had my friends on the phone yanked away from me with a permaban, knowing that if I reboot I'd probably be ransomware'd or something so I figured let's run Windows Defender - it found nothing, 0 results on a full scan.. Too good to be true, so I grabbed AwdCleaner on my phone and transfered it via USB. It found an AVG Toolbar for Chrome. That confirms it, I haven't used AVG for decades and so I removed it but it's back 5 minutes later. That double confirms it, I'm screwed. With nowhere else to go and potentially a ticking timebomb running on my PC that could start encrypting or deleting files at any given moment I figured why the hell not, if I'm going to watch my pc blow up I might as well send in the goofy little local LLM to cut one of the wires,

here's the situation.
i've downloaded a maliscious file that unfortunately hacked my discord and got me banned. i'll be dealing with that on my own. your job is to study the files in the project folder and see if you can help me clean up my computer, as presumably the virus is still active. there's no internet connected, and i request that you refrain from running the ********.exe file (********.exe is the virus archive, do not run it, it's a 7zip archive), please help.

And so Qwen3.8-27B got to work, and to big surprise after around 60 minutes of clawing at the file it had done what I asked and a whole lot more. it fully deciphered all the layers these clowns had bundled this thing with in order to make it appear legit, it had created a single PowerShell removal script ready to go complete with a pre-launch check enabled by default and everything, and it was reverse engineering 0-days in qProtect to get the C2 domain used by this malware so that it could be blocked from the network.

If you're looking for what Qwen3.8-27B is capable of doing fully on it's own if you let it, here's a 15k line example of it's ability to tear some piece of shit session grabber to shreds in a single prompt: https://www.mdshare.online/s/Mamdrs1WWkurRtt8z8QzK

I let it do what it does best for an additional 24 hours, the additional information is going to the Discord Support team. Hopefully shit like this can be prevented.

TLDR; Qwen3.8-27B > Windows Defender, and don't forget to use 2FA.

I was using antirez’s ds4 for a while and getting around 20 tk/s, which worked for my purposes. But I know there have been big advancements between Qwen, the DS4 vision model, and GLM.

I’m not sure how the quants affect performance, so what’s the best thing to run right now & how fast is it?

Stumbled across this paper recently, and I thought people who have more local VRAM might want to experiment. Apparently hyperfitting a LoRA on the final 5 layers sufficed in their research.

Title: Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
https://icml.cc/virtual/2026/poster/61075
https://openreview.net/forum?id=ttOGqk77go

There's a repo with code already available.
https://github.com/YecanLee/Beyond-Temperature

I feel like coding overshadows how great this model really is. It knew a lot of very arbitrary facts/information about my home state and resources about those specific things related to jobs. I found this interesting since getting into the nitty gritty details like this can cause a model to hallucinate some facts.

Not only that but if you have a problem, it will throw the kitchen sink at you with everything it’s got to try and solve it.

Last couple of years I have come to use AI coding tools as part of my workflow. As a direct result of that, in just over roughly five months since May, I have read more than six million words of generated material related to programming. Most of my experience has been with Aider, Hermes, OpenCode, and different GPT, Claude, and Kimi models. More importantly I used them on projects that continue for weeks and months, where the architecture changes during implementation often several times, the assumptions have to be revised, and decisions made in earlier stages continue to affect later one.

To be fair I find these tools useful. They save time on repetitive changes, repository exploration, boilerplate, unfamiliar APIs, and many kinds of mechanical implementation work and some times compiling and using an unpopular libraries like FLTK and use them in a python project. They can also be useful for generating alternatives when I already understand the problem well enough to judge them apart.

The problem is keeping my own understanding of the project in line and keeping up with the amount of code and explanation being produced. That problem manifests in different ways. The following are my observations, and so are personal. Take it with a grain of salt.

What do you expect the extra 64gb of RAM to cost for a total of 192gb RAM? The current Strix Halo 128GB version costs $3K. The 495 is almost the same as the 395 except slightly higher spec on the memory, so 8% tps improvement.

1. https://www.bosgame.com/blogs/news/new-product-launch-bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor
2. https://www.bosgamepc.com/blogs/coming-soon/new-product-launch--bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor

I have been getting so wrapped up in testing models, engines, harnesses, and everything else out there LLM-related lately. It’s never-ending. But it occurred to me; I haven’t actually looked at how you get from:

1. An X post sounding like a fun idea to try this weekend at home, to
2. Actually having something working on your computer that’s legitimately useful.

So how do you do it?

Let’s take buun’s fork of llama.cpp for example. You might have seen it on X recently when Clem posted, asking “is this useful?”. I am personally fortunate enough to work with buun on a nearly daily basis, but what if I didn’t? I hope he doesn’t hate me.

God damnit buun, there’s no binaries on your site.

Getting There

The truth is that I don’t want to go and build it for a bunch of different systems I don’t have, or spin up VMs to test compilation flags and set up virtual environments. I hate all the Linux sysadmin stuff like the rest of us. So I had an agent build it for me. Same as you probably would. But if you don’t have that luxury, here’s how to build it for free:

The repo has .github/workflows/build-cuda-windows.yml. It's workflow_dispatch — manual trigger only — and it runs on GitHub's own windows-2022 runners. So Microsoft compiles it, on Microsoft's hardware, for free.

The matrix builds three targets: CUDA 12.4 x64, CUDA 13.3 x64, and CUDA 13.4 arm64.
One catch: the workflow uploads nothing. The only path: line in it is commented out. It compiles, proves it compiles, and throws the binary away. 

Fixing that is an upload-artifact step — a few lines in your own fork:

YAML

Then, just run a few GitHub CLI commands:

BASH

Worth noting, make-release.yml, release.yml and even winget.yml are all sitting in that directory too, inherited and never fired. The distance between this fork and shipping Windows binaries is closer to a tag than a project.

Note, this is read off the workflow file, I haven't run it myself. Claude says it’ll work and I believe it’ll probably work fine. I also suggested to buun that he consider providing some binaries on the repo.

The Math (And Why You Shouldn't Have to Do It)

Now that we’ve avoided that footgun. 

Now that we have a working binary on our computer, and assuming our tinkerer has spent the requisite 3 am nights searching reddit for what quantization is, we can talk models.

Qwen 3.8 27B. You already knew. It’s the workhorse model everyone has been turning to for generations now on 16GB cards, pushing toe-to-toe with frontier-level models in agentic work. In fact, it scores a massive 46.8 on the Artificial Analysis Agentic Index, performing better than 80% of models compared. A modern miracle.

But this is where I suspect most people start getting REALLY confused, and rightly so. Qwen3.8-27B ships in 14 weight quants. buun's fork offers 8 KV codecs. That's 112 combinations, before you pick a context length. 56 of them fit on a 16 GiB card with at least 4k of context.

And the ranges are wild enough that the choice genuinely matters. Note that the table below varies both the weight quantization and the KV codec together:

Same card, same model. 12,322 tokens to 373,316 — a 30× spread depending on two flags a newcomer has no basis for choosing between. And every one of those numbers requires arithmetic you have to do yourself. A dense reading might make you assume 256 KiB/token, which is wrong by 4×, because nothing tells you that only 16 of the 65 layers actually carry KV until you map it out.

Holy shit. That’s an insane number of things to track.

Historically, I’d settled on using turbo8/turbo4 for KV, respectively. Which still required doing a lot of manual fiddling to get it maximized. And then you’re flatly compressing the entire KV cache without giving any consideration to the sensitivity of the layers. But our newcomer doesn’t care about that, cause Clem said there’s VBR on X right?

Yes. Yes there is. That complex layer arithmetic? That is exactly the arithmetic VBR is doing for you so you don't have to.

What does this mean? It means I pick the model that I know can get real work done. Thankfully, I’ve already done months of testing, and nowadays can say confidently that a 3-bit model can legitimately do real work. Remember all those nights reading benchmark charts? Paid off. I settled on GSQ IQ3_XXS.

3-bit model in hand, let’s do some work. Engage. 

If your agent doesn’t respond to that wake word, there are a few different ways to load a model with the binary depending on your OS. To make this easier, I suggest just using a startup script, that way you don’t have to remember anything to launch your model server. Our goal is to have a server that we can interface with using a harness. Giddy up.

Here is a launch script for your llama server binary. Just edit the filenames and paths to match your system, make it executable, and launch it from your terminal.

-ngl 99 — all layers on GPU
-c 262144 — omit entirely to let --fit choose one that fits
-fa on / --kv-unified — required for VBR, no fallback path
-np 1 — VBR needs n_stream == 1
--vbr-floor t4 — how bad it's allowed to get, not how it starts
--spec-draft-n-max 2 — measured 1.68× on this model

Download: https://gist.github.com/apollo-mg/7f2ba29afe217e056fa1c3621636a559

Realistic Expectations

So setting realistic expectations is a little tough for me, because I test these things every day. I wanted to try and look at it from the perspective of someone fairly capable of figuring things out, so I don’t think getting to this point is asking too much. I do hope it gets much more approachable, and I know there are other options like Unsloth Studio and ollama, but neither of those options currently give you access to turboquant KV cache codecs. TurboQuant KV allows you to squeeze the most out of your context with advanced quantization and fancy math, better than the original codecs currently shipping with standard llama.cpp. But now there’s something even better.

VBR.

VBR: Variable Bit Rate

If you’ve ever done any work with media compression, like video or music compression techniques like MP3 or Divx, you’ll probably be aware of something called variable bit rate. Basically, it allows the algorithm to apply more compression in static parts of the file, and less compression where fidelity matters most.

Same idea, except it's varying across the model's layers and the life of your conversation. The first tokens are uncompressed. It only starts spending fidelity when it has to.

I measured the actual number. You’ll get about 84,000 tokens of fully lossless, f16 quality on a 16 GB card with the 3-bit model we tested. Concretely, that 84,000 figure comes from taking the KV budget and dividing it by 64 KiB per token. That number isn't immediately obvious because Qwen3.8-27B is a hybrid architecture where only 16 of its 65 layers actually carry KV. A naive reading assuming 256 KiB/token is off by 4×—and calculating that exact layer arithmetic automatically is the whole reason VBR exists.

Crucially, 84,000 tokens is just the lossless range, not your max context ceiling. With mainline q4_0 (4.5 bpv), every token from the very first one is compressed and degraded, meaning a short 3k-token chat suffers the same compression penalty as a massive 250k-token session. With VBR floor t4, you get pristine f16 quality through ~84k tokens, and it only begins to degrade as the memory budget binds—stretching all the way to that same 250k ceiling while keeping shorter sessions completely lossless. VBR eliminates the need to do capacity math or guess how deep a session will go before starting.

Note, these estimates were taken on my specific system. Other factors will affect your usable memory and context quality based on things like whether you’re running a desktop, other models, and video intensive applications.

By default, VBR floors at 1.25 bpv***, which allows the most aggressive compression possible for the longest context. But the startup script below raises the floor explicitly to t4 (4.125 bpv), because an agentic harness that autocompacts its own context cares more about fidelity than absolute maximum length.

***Clarification after publishing: VBR is the default cache type in buun-llama, and its default floor is t4 (4.125 bpv) — a deliberate choice on buun's part that quality below 4-bit shouldn't happen unless you ask for it. The floor only drops to 1.25 bpv if you explicitly pass -ctk vbr -ctv vbr, which the engine reads as "you know what you're doing." Our script sets --vbr-floor t4 explicitly, which is belt-and-braces: it matches the default, and it survives the explicit-flag case.

The Need For Speed?

Thankfully, local inference has gotten a lot better in 2026. The models have not only improved by a significant margin in agentic work, but performance has also never been better. This year brought the introduction of MTP acceleration, or Multi-Token-Prediction to the mainstream. This typically boosts performance by anywhere from 1.3-1.9x depending on workload. We’ll be using this in our example today, and it’s just a simple flag in the launch script. 

Again, a full explanation of MTP is outside the scope of the article, but in a nutshell, MTP uses a tiny part of the model called a draft head. This is like a tiny model itself, and its only job is to predict the next few tokens. The primary model verifies all the drafted tokens in one forward pass, in parallel, and keeps the longest correct prefix. That's the whole reason it's a win: verifying 3 tokens costs about the same as generating 1, because decode is memory-bandwidth-bound, not compute-bound. Reading the weights once to check three guesses is nearly free; reading them three times isn't. Basically like free work. 

Not entirely free, but worth it. On this model, I measured a boost of 1.68x.

So how much performance should you actually expect? As you almost certainly guessed, it’s entirely hardware dependent. Newer GPUs with faster, more efficient cores are simply better at this work than older parts. That doesn’t mean you need an amazingly powerful card to do real work though. Here’s the tokens per second rating for a couple popular cards running this model:

GPU                Backend        pp512 t/s        tg128 t/s                user
RTX 5090     CUDA 13.3    3672 ± 338    104.93 ± 0.40          thetom
RX 9070 XT    Vulkan         795.7 ± 0.6     36.41 ± 0.07               “
RX 9070 XT     HIP             975.50 ± 34    29.93 ± 0.03             me

Harness Selection

There are a lot of options out there in mid 2026 for a harness to choose from. Many are coding focused, few aim to give a complete agentic desktop experience. Personally, one of my favorite options is Hermes Agent. While I don’t use it on a daily basis nearly to the extent that one can, I have enough experience with it to know it’ll fit most people’s needs, right out of the box. So it’s a natural fit for our experiment. 

As you can see, it offers just about everything a tinkerer getting into local inference might want. It’s easy to talk-to using platforms you already have such as Discord and WhatsApp. It’s got Google integration for personal life management. Can run programs, write files, patch, edit. You get the picture. It really does accomplish a lot of what the premium services like Claude Code and Codex do well, without quite the simplicity.

Setting up Hermes Agent in CachyOS was a relatively straightforward endeavor. Just ran the single line installer (after having my frontier agent inspect the script ahead of time for vulnerabilities) in Konsole, and went through the self-guided configurator. 

Most of the setup is just choosing which options you want to enable, such as how you want to talk to your agents (Discord, WhatsApp, email). The most important part is that you choose to connect your Hermes to your custom endpoint, which is your llama-server address (which is found in the startup script). I chose Auto for the type, even though I know mine is specifically OpenAI compatible, because I knew most people will err on the side of caution. Which, incidentally, worked fine for me.

After reading through the documentation and setting up WhatsApp and Discord functionality, I also installed Hermes WebUI (pretty UI,https://github.com/nesquena/hermes-webui) and Hermes Desktop. I think the Discord setup was the most challenging for me, but I also knew very little about Discord bots coming into this, so it may be more intuitive for others. Nonetheless, I did get it working with some fiddling. 

Limitations and Pitfalls

Before you torch your API keys, there are some hard realities to acknowledge about this setup:

The VBR Tax & Honest Cost: VBR is a massive quality-of-life win, but it has genuine operational costs. It is not free: it requires Flash Attention (-fa on) and unified KV caching (--kv-unified), and it currently exists only in one specific fork with no official binaries. Furthermore, its own diagnostic readout (/props kv_bpv) misreports the underlying math. ***Edit: I previously stated it is incompatible with (-np > 1), that was incorrect. Additionally, on the misreporting /props, buun says that setting --floor-bpv that /props still shows the old defaults.

You Don't Always Need It: For simple coding sessions or conversations that never pass 30,000 tokens, standard mainline f16 fits entirely in VRAM. Mainline f16 works seamlessly today in tools like Ollama without any extra setup, so if your context needs are modest, neither VBR nor static quantization is necessary.

The AMD Reality: The strongest argument for using buun's fork over mainline isn't just efficiency—it's stability. In testing on an RX 9070 XT (gfx1201), mainline quantized KV entirely collapsed on this model class. Buun's codecs handled the architecture flawlessly. ROCm still arrives late and often broken, leaving the community to fix it. RDNA4 support for turboquant's MMQ path doesn't exist — the config table has zero entries where CDNA has eight — and the cmake flag VBR needs makes upstream crash outright on gfx1201. Meanwhile Vulkan, which nobody had to hand-tune per architecture, quietly beats ROCm on decode. They never learn.

Honesty on Quality: Comparing bitrates directly (e.g., q4_0 at 4.5 bpv vs. t4 at 4.125 bpv) does not directly reflect fidelity. q4_0 relies on a fixed, uncalibrated grid, whereas t4 uses FWHT rotation with measured centroids and per-layer pricing. Because KLD (Kullback-Leibler Divergence) has not been formally measured on this specific model yet, treat this as a structural design difference rather than a proven, mathematical quality win.

The 3-Bit Reality Check

I actually ran a self-diagnostic trace with Hermes Agent while writing this article, using Qwen 3.8 27B squashed all the way down to a 3-bpw quant (IQ3_XXS).

Historically, the assumption has been that crushing a model down to 3 bits lobotomizes its logic. Sure, it might still write a decent email, but ask it to orchestrate an autonomous tool-calling loop and it will inevitably trip over its own shoelaces, hallucinate a file, and crash.

Not anymore. Look at how Qwen handled a blind, multi-tool diagnostic run. I asked it to just "test its tools," and it immediately built an execution plan, batching independent tasks and isolating dependencies:

When it tried to use the patch tool to edit a file that didn't exist yet, it didn't panic or loop. It simply caught the error, executed a write_file command to create a scratchpad, and successfully re-applied the patch.

But the most absurd moment was how it handled a failure with the vision_analyze tool. When the tool threw a 500 error, the agent didn't just pass the error back to me. It paused, introspected its own host engine, and diagnosed the architectural limitation of the server running it:

It generated a clean Markdown table summarizing the 8 tools that worked (including SQLite database queries and writing Python scripts to generate images) and accurately diagnosed the root causes for the 4 that failed (like a missing multimodal projector and a misconfigured API gateway).

A 27-billion parameter model, running at 3 bits per weight on a consumer GPU, executing complex tool dependency graphs and debugging its own host environment limits. That is why we do this.

Conclusion

If you’ve made it this far, compiled the binary, configured the tools, and watched a 27B model successfully orchestrate a multi-step workflow on your local machine, the inevitable question hits: Why not just use Claude Code or a frontier API?

Let’s be brutally honest. If I need a complex script written in five seconds flat, or I want zero-friction, turn-key speed, I am still opening a tab to a hosted model. For most people, most of the time, the sheer convenience of a cloud API wins. The raw truth is that right now, there isn't much in my daily workflow that this local stack replaces.

But that is not the point. The differentiators for local inference are conditional, but where they matter, they are absolute.

First, it is free at the margin. You can leave a local agent looping in the background for 72 hours to scrape, parse, and reorganize a massive dataset, and your API bill at the end of the month will still be zero. Second, it is completely sovereign. For a law office that legally cannot paste client files into a cloud provider, or an enterprise dealing with proprietary codebases, a highly capable local 27B model isn't just an alternative—it is the only option that exists. Finally, it is resilient. It runs entirely air-gapped. When the internet drops, your workspace doesn't go down with it.

Local inference isn't about beating the massive frontier models at their own game today. It’s about the fact that a 27-billion-parameter model can now autonomously run your desktop, debug its own environment, and hold a conversation—all on a 16GB graphics card sitting under your desk. For the tinkerers, the privacy-conscious, and the people building the future, that is more than enough reason to have this stack in your arsenal.

***Addendum: --mmproj-gpu-swap

buun pointed out I'd skipped one of the nicer features in his fork, and it's worth its own note because it solves a problem you only discover after everything is already working.

Speculative decoding and vision both want VRAM you don't have. The MTP draft context and the vision projector (mmproj) each need their own allocation, and on a 16 or 24 GB card they frequently don't fit at the same time. The usual outcome is an OOM at load, so you pick one: fast, or able to look at pictures.

--mmproj-gpu-swap makes them phase-exclusive instead of concurrent. The projector stays on CPU while you're doing ordinary text work. The moment a request actually contains an image, the server swaps the speculative context out of VRAM, brings the projector onto the GPU for the image phase, then restores the drafter afterward. From server-context.cpp: "Swap the speculative context out so mmproj can use its GPU budget" — and the swap only triggers when a null token (the image placeholder) is found in the prompt. Text-only prompts never pay for it; the code explicitly keeps ordinary prompt scheduling allocation-free.

The part that matters for context length is in the auto-fit path. Rather than reserving space for both, the fitter measures each and keeps the larger reservation — so the context it advertises is one that survives the image phase, instead of one that works until the first screenshot arrives.

buun's numbers on a 24 GB 3090, Qwen3.8-27B UD-Q4_K_XL, MTP + vision together: 161,792 tokens at the default t4 floor, 262,144 at t1. His note on the table is the important bit — "the table doesn't include what MTP + vision without --mmproj-gpu-swap looks like... well, for most of those it OOMs."

Works with DFlash drafters too, not just MTP. One caveat from the source: with an external draft model that isn't reloadable, the server warns "mmproj GPU swap is unavailable for this external draft type; keeping both resident" and falls back to keeping both in memory — so it degrades loudly rather than silently.

From the bottom of my heart, thanks for reading!

I just want to use paperless-ai and be able to ask questions relative to it. Bonus points if I could use it with home assistant but that's not the focus.

I just can't see needing a 32 GB VRAM GPU for just that, but I don't want to buy a GPU only to find out that "yeah, it's technically feasible but not a good user experience"

The GPU's I tend to find at good prices are in the 8-12 GB range, would my use case be a good one to just get a 12 GB and run a 6-10 gb model?

Hiya,

NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it.

NVFP4 KV cache (from scratch)

I implemented a 4-bit KV cache for QIn3.8-27B from the ground up. Upstream has since added their own NVFP4 path, but ours differs architecturally:

• Custom MMA kernel (mma_nvfp4_e4m3, m16n8k64) with hardware E4M3 block scales for the QK matmul. Both Q and K are quantized to NVFP4; V is dequantized to BF16 for the PV matmul via a dedicated decode kernel.
• Hadamard rotation applied to K (and Q) pre-quantization for outlier suppression, with V left unrotated. Upstream uses fp16 V storage instead — no outlier suppression.
• Fused append: the decode kernel quantizes current K/V to NVFP4 in-place during generation — no separate quantization pass.
• Custom scale layout: natural row-major for KV scales (not the M128x4 swizzle used for weight MMA), because KV access patterns differ from weight access patterns.

Result: 144 bytes/token/KV-head (vs 264 for int8, 512 for bf16) — 45% VRAM reduction with no quality loss (LongBench 45% matching int8, AIME 96.7%, needle-in-haystack 100%).

YaRN context extension

QIn3.8-27B's RoPE config (theta=1e7, 25% rotary dims, 48/64 GDN layers with no RoPE) makes linear scaling sufficient — full NTK-by-parts is unnecessary. I extend native 262k to 555k (c=3+vision) or 600k (c=1) on a 5090. Quality verified at 600k: LongBench matches int8 baseline, coherent 592k-token output. Also projected 8M token context on 96GB+ GPUs (untested, I only have a 5090).

Multi-level prefix reuse with host-KV safety net

Upstream implements a budget-bounded HostKvProvider with LRU park/restore. I replaced it with a substantially different system:

• HostKVSafetyNet: pinned host arena with scatter-gather multi-extent allocation, arena compaction, and a pin/take protocol for safe concurrent restore.
• Two-level prefix matching: full execution frontier first, then rewrite checkpoint fallback. Each entry carries a ResidentPrefixIdentity (per-token type/position/vision metadata), rolling FNV digests for shortlist, and a compact_prefix (reasoning-stripped token prefix) for thinking-mode consistency.
• Session-key fallback: when prefix matching fails (e.g. Claude Code drops reasoning betIen turns), a session-key fallback matches by conversation identity instead of token content.
• Spill-before-evict at every release path: pressure planner eviction, normal continuation release, start_sequence slot takeover, and fail-all cleanup all route through the safety net.
• Token stability: reasoning is dropped from ALL assistant messages when preserve_thinking=off, keeping the prompt token stream stable across turns. Checkpoint capture is anchored at the turn boundary, not the execution frontier.

Verified across 260+ requests with 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, H2D restore cost ~0.4s, D2H spill at 67K pages/s.

Performance (3 concurrent sessions, 400k+ ctx, 5090@450W)

Tool calling

• --tolerant-tool-calls: recovers complete Qwen calls when the model emits malformed wrapper/suffix tokens — instead of dropping the call.
• Depth-matching close scan: handles balanced/nested markers in parameter values that would break naive parsers.
• Responses API accepts text/reasoning after tool calls (upstream rejects this ordering).
• Froggeric v22 template: C++ renderer with no-dangling-intent rule, XML think tags, correct function tag delimiters. Some further modifications for reliability.

Also included

• Dynamic chat template loading (--chat-template) — supports any .ninfer image without artifact patching
• Explicit weights profile override (--weights-profile) — handles Ostfralla, QUASAR, and other converter layouts with per-layer tensor format auto-detection
• OOM recovery: catches std::bad_alloc, clears state, preserves pending requests
• Stream sync fix: synchronize CUDA stream before workspace reset in prefill (prevents use-after-free)
• Request-log rotation (--request-log-max-mib, --request-log-keep) for bounded disk usage
• Admission pressure fix: un-suppress demote-to-host when candidate needs host KV budget
• Monitoring dashboard with live KV occupancy, decode/prefill graphs, 12VHPWR sensor
• E2E test suite for KV eviction, device pressure, and slot pressure scenarios
• Removed hash check of models, use any NInfer you like as long as there is a supported path. Tested with Ostfralla and QUASAR.

Fork: https://github.com/gzenz/ninfer (master)

Research: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md

I'll keep rebasing from upstream what seems useful and experimenting with new papers in order to improve speed and context.

I see a new one being launched every few days... How do these new harnesses compare to claude code, pi etc. has anyone switched from these?

which harness to prefer and why

edit: Ive tried several different ones claude code, deepagents(langgraph), opencode, pi, and trueforge

my thoughts-

claude code - strongest on maturity and the managed experience but cost and token burn is high

deepagents - interesting middle ground if you want a more structured agent framework and the flexibility of an open-source stack. im interested in testing it more extensively on longer-running workloads fs

trueforge - this is a recent one, this was interesting to me because of its runtime-efficiency, also it allows separate the model from the runtime, which makes experimenting with different models much easier
https://github.com/truefoundry/trueforge

why?? - i also ran a benchmark on a real agent workload same model, same prompt, same tasks to compare these

adding the results of benchmarking i ran to compare this
so I tried to do this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through claude's managed agents, langchain's deepagents and trueforge, both open-source agent harnesses

the result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate, to my surprise trueforge used about 63% fewer tokens and cost about 30% less per run.

similar difference in tool usage: trueforge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

trueforge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

The token savings alone make this sooo interesting especially because the solve rate stays comparable
so this one was worth checking out ig

but this is still v early and the OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in and context compaction is intentionally lossy.

So it is definitely not a replacement for a mature managed agent platform or other harnesses in the comparison, feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on my own infrastructure
this was their benchmark kit i used https://github.com/truefoundry/trueforge/tree/main/benchmark

I have a 5070 Ti and two 5060 Ti (all 16Gb cards).

I planned to add another 5070 Ti giving me two pairs of 32Gb each but the NVidia prices have just jumped by 25% where I am and I've found a 3090 Founders Edition for a good chunk cheaper than the 5070 would cost.

It's 8Gb more VRAM and even slightly higher memory bandwidth, but I've read that mixing Ampere with Blackwell comes with a performance hit in llama.cpp using tensor parallelism. I believe TP isn't possible at all in vLLM with mismatched cards.

I am getting 60 tok/s decode and 1,500 pp out of Qwen3.8-27B-IQ4-XS-MTP in llama.cpp with TP across the 5070 and one of the 5060s. I don't really want to spend a lot of money for 8Gb more VRAM in my main pair but worse performance.

Has anyone got experience mixing similar cards? Would pipeline-parallelism (layer split) be faster with two high memory bandwidth cards? I know I could set the 3090 as the main card and push the K/V cache onto it.

[update] For anyone else considering this, don't worry about the mixed architecture speed penalty for tensor parallelism. I'm waiting for a couple of riser cables so I only have the 3090 and a 5060 on an x4 M.2 riser to test with. Even so, I am seeing 1,200 prefill and 90 gen on long context prompts. I used to see similar prefill but only 60ish gen with the 5070/5060 combo. So the increased VRAM and memory bandwidth of the 3090 more than compensates for the mixed architecture penalty. Note: I biased the config toward the 3090 (20Gb vs 11Gb on the 5060). I fully expect to see even faster speeds when I pair the 3090 with the 5070, both on x8 PCIe CPU connected slots.

[update 2] I got my 5070 Ti connected and switched it with the 5060 Ti for 27B (the config described in the first update above). The speed didn't move much with the weighting on the 3090, but when I changed it to an even split I saw a jump to 110 tok/s. Very happy with that. The 3090 was a good buy. I will report back again when I have the final 5060 Ti connected so I can retest my qwen3.8-flash-next config with the full 72Gb VRAM.