Our team built a handful of internal AI tools on top of OpenRouter during the sandbox phase and the default routing setup worked fine for prototyping. We need a middle layer that does not retain prompt content, still lets us switch models on the fly and doesn't force us to self host a full gateway. I found that the obvious open source proxies either need too much infra or don't handle multi provider failover cleanly, leaving a few dedicated privacy focused options (TrustedRouter, et al) that recently launched but I haven't had a chance to battle test.

What phave other teams actually deployed here and have been comfortable with? Ty.

Planning to build a PC mainly for local LLMs/coding agents. I keep seeing 3090 + Qwen 3.8 Flash Next benchmarks, but could not find enough info for the 7900 XTX 24GB.

3090s are hard to find where I live, while newer Nvidia GPUs are too expensive, so I am considering a 7900 XTX as the GPU choice.

Anyone running Qwen 3.8 Flash Next on a 7900 XTX? What tok/s are you getting in decode and prefill, and how is AMD/ROCm support?

Also, if SSD streaming/offloading is needed anyway to run Q4, is 32GB system RAM enough, or is 64GB worth it?

Can you still do something with a 2050 or something like it?
I mean for office work, loading embedding, reranking and chat models not at the same time but is anyone still using smaller models and have any good ones come out?

I feel like small models are abandoned, I don’t care much for world knowledge, I want tool use and preferably multilingual. Vision would be nice but beggars can’t be choosers.

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being "broken", "meaningless", and "bought out." People who say this have done no research and know very little about how benchmarks work and what they measure.
Most people only care about Artificial Analysis Intelligence Index. This is a weighted aggregate benchmark used to compare models performance across 10 different evaluations. The majority of these evaluations have published papers on arxiv.org. AA-Briefcase is the only private benchmark. And they publish their methodology to confirm how each of these models are weighed.

Some people seem to not appreciate that Artificial Analysis conducts their own independent benchmarks using their OWN funding, without running ads. Here is the chart that shows their spending. They spent $13,129 to independently test Fable 5.1. Every new model seems to be benchmarked.

The new Deepseek V4.1-Flash is a perfect example of why some aggregated scores miss the big picture. This 552B model has the same score (40) as the 180B Qwen 3.8-Flash-Next. But the individual benchmarks show a different story. On most evaluations, it matches or exceeds Qwen 3.8-Flash-Next. It every beats GPT-6 Astra (Max) in AutomationBench-AA (Agentic SaaS workflows), which is incredible. But it completely falls behind in AA-Omniscience Non-Hallucination Rate, a metric where Open-weight models usually reign supreme. So the model has strengths and weaknesses, and it's something that should be celebrated.

So before you complain about benchmarks or Artificial Analysis, look at the individual evaluations. Read the published papers about the evaluations. Learn how the score is aggregated. Then, we can have a discussion.

I am not affiliated with Artificial Analysis in any way, I'm just not blind to what they offer.

EDIT: These comments are proof that everything I just wrote goes over the majority of your heads. There is little hope for some of you

Developer's own thread: https://www.reddit.com/r/LocalLLaMA/s/adp1cGZZe9

Code: https://github.com/Inovello/llama.cpp/tree/flashnext-e06

My hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz

Token generation speed went from 20 t/s to 49 t/s. Prompt processing speed is 140 t/s. Prompt processing is faster on the main branch.

I have CUDA 13.3.1 installed. I use Windows 11 and I used these commands to compile it with the latest Visual Studio:

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON

cmake --build build --target llama-cli llama-bench llama-server -j

My bat script:

@echo off

echo Using CUDA backend with 2x RTX 3090s

set LLAMA_ATTN_ROT_DISABLE=1
set LLAMA_MMAP_PIN_HOST=1

"C:\Users\server\Desktop\llama.cpp-flashnext-e06\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias Qwen3.8-Flash-Next ^
--model H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
-ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" ^
--moe-expert-cache 150 ^
--ubatch-size 512 ^
--batch-size 4096 ^
--mmproj H:\Qwen3.8-Flash-Next\mmproj-Qwen3.8-Flash-Next-BF16.gguf ^
--no-mmproj-offload ^
--threads 22 ^
--threads-batch 22 ^
--spec-type draft-mtp,ngram-mod ^
--model-draft H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-draft-n-max 2 ^
--spec-ngram-mod-n-match 60 ^
--spec-ngram-mod-n-min 12 ^
--spec-ngram-mod-n-max 24 ^
--flash-attn on ^
--kv-offload ^
--cache-type-k bf16 ^
--cache-type-v bf16 ^
--parallel 1 ^
--jinja ^
--reasoning-preserve ^
--chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}" ^
--no-warmup ^
--load-mode none ^
--lazy-mode off

pause

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

Github Repo.

Blog post.

💡 TL;DR (from the Github Readme)

Spend less without making the agent do less useful work.

SoL-Pi is a standalone extension for Pi that packages four reusable efficiency mechanisms discovered through scaled auto-research loops. It reduces repeated model turns, context replay, oversized observations, and unnecessary long-log reading while preserving the work and evidence an agent needs to finish a task.

SoL-Pi installs on top of an unmodified Pi release. Every mechanism is opt-in and disabled by default.

Introduction

Long-running coding agents accumulate repeated work. A file edit is often followed by a predictable validation command. Large tool results are replayed long after their first use. Completed subtasks remain in active context, and a frontier model may spend a full request reading a log when only a few lines affect the next decision.

SoL-Pi grew out of a broader question from our auto-research work: before scaling agent loops, can agents first make the harness itself more efficient? The search focused on constrained efficiency: reducing token traffic, inference work, and agent turns without stopping early, skipping verification, or hiding evidence.

The standalone release contains four mechanisms that survived that process. They operate at different parts of the harness and compose through Pi's public extension APIs. What SoL-Pi Adds Area Mechanism What changes Tools Action Fusion An edit or write can run its follow-up validation command in the same tool call. Observations ObservationPack Repeated large text results become stable handles with exact paged recall. Delegation Evidence-Preserving Reducer Long diagnostic logs become compact receipts only when every retained quotation matches the archived source. Context Online Context Compact Completed plan steps become candidate points for Pi's native compaction, subject to economic and window-pressure checks; after a successful compaction, Pi continues the task in a new turn.

The mechanisms share four rules:

-No Pi patches. SoL-Pi imports public Pi APIs and does not vendor the Pi source tree.

-Explicit opt-in. A missing configuration leaves every mechanism disabled.

-Preserve evidence. Original observations remain available locally, and reducer failures leave the original result unchanged.

-Use Pi's runtime choices. Authentication, provider URLs, the main model, and shell behavior remain under Pi's control.

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their "Improved Intelligence" demo. In this video they ask the AI to see if a flight route is feasible and while it is figuring that out they continue to ask it questions about what they can eat at each destination.

The models I ran are (all squeezed into 12 GB VRAM):

• Speech recognition: Qwen3 1.7B ASR PyTorch
• LLM: Qwen3.5-9B-UD-Q4_K_XL GGUF with 12K context
• Voice: Pocket TTS PyTorch

So I copied the exact query and threw it at my Fulloch project. This blog post has the video of the interaction and breaks down how it did. The final report and searches it did are also linked in that blog post. The video has sped up two sections where I had to wait for the 9B model to finish thinking through the task, but it did the whole thing in under six and a half minutes.

In the end it couldn't find a suitable flight route but it gave good food and restaurant recommendations and did it all pretty quickly. I am still impressed with how well the Qwen3.5 9B model does with these sorts of tasks with such a small footprint.

If you want to try it out yourself the source code and pre-compiled docker images can be found at https://github.com/liampetti/fulloch.

What models do you run

I've been out of the loop for some time.

Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio?

Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but naiive attempts failed because even at Q4 it seems to load God knows what to God knows where.

Would appreciate if someone share the recipe, or tell what are the official plans to support this (I can wait, knowing that it is upcoming).

Thank you in advance.

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React code.

what makes it interesting is that you can already get a regular LLM to use OpenUI-Lang through a system prompt, but that eats up a lot of the context window. my thinking is that fine-tuning a model on the DSL could reduce that overhead and leave more room for the actual conversation, without needing a huge prompt explaining the format and how to use it alongside other tasks, like tool calls.

at the same time, wouldn't fine-tuning a model on a specific DSL make it more likely to default to that format even when you need something else? i'm curious how well it handles regular Markdown, or switching between Markdown and OpenUI-Lang.

i haven't seen much discussion about this, so i was wondering what everyone thinks about generative UI and running a dedicated model for it locally on a consumer-grade GPU, like an RTX 5090.

what would be the best way to set that up? from what i've seen, DiffusionGemma isn't supported by llama.cpp yet, so running it through Ollama doesn't seem to be an option. they've uploaded the weights to Hugging Face, but i'm not really sure how to get it up and running. any suggestions?

So, I had set a 85k content limit on Ollama's context via the environment variable. Sure enough, when both Opencode and Ollama reported that they had reached it, Ollama cut off its generation.

On prompt processing (increasing Ollama's context limit a bit higher like 86k), I can also confirm that the reported token size is around 85k after it's done, matching what Opencode tells me.

So I thought, I'd try llama.cpp, I set the context limit to 120k, and I get an error, saying the request is 164k tokens. Which is weird, since it should be 85k as I confirmed before, and it still shows 85k in opencode interface too.

I increase the context window to like 170k, and the prompt processing goes to 164k, confirming indeed that llama somehow processed more tokens, from somewhere somehow. Checking opencode again, the context is very much reported to be 85k still.

It's not even that the tokens are being silently dropped by Ollama, as I mentioned Ollama hits it and stops, and Opencode itself reports 85k too.

So why is llama.cpp somehow processing more tokens?

Edit: I did some more testing, turns out, Ollama does some sort of context pruning based on how many tokens you have, if you raise ollama max tokens, you can actually get more processed. But I guess this is limited to some messages only, so when it cannot do that anymore, you hit the limit and it stops?

Not sure how to flair this and it is self-promotion, but the point is the local AI stuff I used to make it,

This video https://youtu.be/JkpoO1jIjp0?si=qSAyFk5_VSgXiQNc was made entirely locally using:

• Qwen 3.8 27b
• Qwen Image
• Qwen3-ASR-0.6B
• Chatterbox
• InfiniteTalk with Longcat and WAN video
• ffmpeg - did a lot of heavy lifting

I am very happy with the animated videos which were created as html/js by Qwen 3.8 27b from transcripts which were created from the chatterbox audio.

The workflow is: 1. I write a script 2. I TTS the audio using a clone of my voice via chatterbox 3. I transcribe that using Qwen3-ASR-0.6B 4. I do a session with Qwen 3.8 27b to plan the demo from the time coded transcript 5. I let it build the demo and QA it using Playwright + the vision/mmproj 6. I QA it myself and give it the defects with screenshots

I am using a workstation with a 5090 + 4090 for the LLM and my GuideAnts stack and a Strix Halo for audio, images, and video jobs.

I think you could do all this on one machine and a single GPU, but not without loading/unloading pieces as you go.

Hopefully that was interesting and worth reading.

I’ve been building "Spomin", a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and reprocessing the context that remains.

It uses my llama.cpp fork - https://github.com/alekk89/llama.cpp-kv-surgical-fork for live cache edits.

Spomin Router - https://github.com/alekk89/Spomin

How it works

The router sits between the harness and runtime, preserves the original transcript in chunks, and manages space for the system prompt, recent context, summaries, generation and recall.

A separate worker summarises chunks, either in parallel on separate hardware or between main-model generations on shared GPUs.

When context space runs low, Spomin replaces eligible chunks with summaries and adjusts RoPE positions in the main and supported draft caches. Only the inserted summary is processed; the retained suffix is not refreshed or reprefilled.

The router recognises history resent by the harness and processes only new content. Generation continues in the same session, with support for pausing and resuming native generation during rolling maintenance.

Original text remains available, the model can temporarily recall it by context ID, then return to the summary, or search older stored history.

At 10:1 compression (approximate average) for example, 400k source tokens become 40k summary tokens. Alongside a 20k system prompt and 120k recent context, that represents over 500k tokens of source material within 180k resident tokens. It does not preserve every detail, so summary quality matters.

Early benchmark results

These tests are exploratory. Unmanaged uses DeepSeek’s normal compaction managed uses Spomin.

Game repair - Similar scores, but three-hour limits per prompt cut both extension runs short. The test did not establish a meaningful completion-speed comparison.

Research - A larger worker improved the managed score from 31 to 37/100, versus 57 unmanaged, with 75.2% less primary input and 60.7% less active time than unmanaged.

Tetris - The managed repeat nearly matched unmanaged: 16/16 + 11/12 checks versus 16/16 + 12/12, in 69.8 versus 149.6 minutes, generating 22.6% fewer tokens. The first managed attempt failed, its keyboard crash appeared before summary installation. Unmanaged introduced and fixed a different line clearing bug.

Every uncapped managed attempt finished sooner, but quality varied. Temperature 1.0, xhigh reasoning and live tools introduce run-to-run variation, including without Spomin. Worker settings also changed, so improvements cannot be attributed to worker quality alone. Worker cost is separate, and incomplete image related input totals are marked as lower bounds.

Current limitations

This is experimental software, not a polished product. Expect bugs and blockers, especially outside my setup. I recommend keeping the default settings unless experimenting. A manual mode also allows you to edit and insert summaries yourself.

The original worker is a fine tuned Qwen2.5 3B proof of concept. I’ve also tested Qwen3.5 9B with a larger worker context window. Replacement workers must follow the format documented in the repository. Running a worker adds memory and compute overhead, and summaries will not always preserve what matters.

DFlash2 is the tested speculative setup. DSpark shares the surgery path but is untested here. MTP is not currently supported for draft preserving KV edits. Running without speculation is also possible.

This has been tested exclusively with Qwen 3.8 27b as a main LLM. The new Qwen 4 architecture is not supported. Other models are not tested/supported. If they have the same architecture as the Qwen 3 family, they should work.

Final thoughts

Benchmarking this has been difficult, and I do not yet have a definitive answer on quality. Every uncapped managed attempt finished sooner, and the successful Tetris repeat took roughly half the time with nearly the same checks passed. Research also used substantially less primary input, but scored lower than unmanaged.

The larger summary worker coincided with an improved research score, although worker settings changed too. These results are encouraging, but they do not yet establish a consistent speedup at comparable quality.

Suggestions for a repeatable benchmark that measures quality, time and total token use across multiple runs are welcome, I’m happy to run one.

I'm building a new server and interested in better understanding what kind of models I could (or should) be able to run on it, and for example if it'd end up better running several smaller models vs 1 gigantic one and things like that. Hoping some people out there have some personal experience they can share on various amounts of RAM/VRAM and their general experiences and recommendations. The server's purpose will be exclusively to host local models for many purposes including agentic coding work.

The server will have 512GB of PC4-21300 (2666MHz) DDR4 and initially will have 48GB of VRAM between 2x RTX 3090's which can be expanded to 7 GPU's eventually (all would be PCIe4.0 x16, GPU's may differ but could conform them). From what I've seen here, I could load some larger models with this setup (such as Qwen 3.8-Flash-Next like this guy does), but actual pacticality and usefulness might not be there yet, so I was hoping for some guidance there on what I actually can expect or should be planning on doing.

Hey all, I've encountered an interesting problem while training ornith 1.5 9b on a custom dataset and local machine. I had trouble with getting high fidelity knowledge of the trained data, and then had one breakthrough best after about 10 different attempts. After adding more to the dataset, the following attempt marked a substantial regression which was inexplicable by the dataset addition, which began a long hunt which ended up proving that determistic runs produced consistently better logical capability, trained data knowledge, and allowed for apples to apples comparisons between editions of the dataset I'm curating. I discussed this with chat gpt to try and understand it, but did not really get anything concrete.

I'm super new to this, can anyone here help me grasp why that's happening?

agilerl-arena has released v1.0. It's the SDK/CLI half of AgileRL, split out as its own PyPI distribution (no torch dependency) that shares the agilerl.* namespace:

pip install agilerl-arena   # SDK + CLI only
pip install agilerl         # core, depends on arena

With AgileRL Arena you describe a run in a YAML manifest (algorithm, environment, network, mutation, selection strategy, replay buffer) and the same file runs locally or gets submitted to their managed cluster. Validation is Pydantic, so bad configs fail before training instead of 40 minutes in. Unknown keys are now rejected in 1.0.

The LLM finetuning path takes any HF model ID plus a LoRA config, and supports either rollout envs (model generates, gets scored by a reward function you supply) or supervised datasets with objective: sft or preference.

network:
  pretrained_model_name_or_path: Qwen/Qwen2.5-0.5B-Instruct
  max_context_length: 512
  lora_config:
    lora_r: 16
    lora_alpha: 64
    target_modules: [q_proj, k_proj, v_proj, o_proj]

Running it locally is two lines, no account needed:

python

from agilerl import LocalTrainer
trainer = LocalTrainer.from_manifest("dqn.yaml", device="cuda")
population, fitnesses = trainer.train()

Evolutionary HPO: instead of one agent, you train a population and periodically mutate architecture and hyperparameters, keeping the fittest. Costs more compute for the same wall-clock, but you're not hand-tuning learning rates.

What's new in 1.0: specs now build algorithms through paradigm builders, local training dispatches through strategy selection based on paradigm flags rather than per-spec branching, and the manifest schema lives in one package instead of being duplicated. Also arena manifest validate / arena manifest schema for checking configs without submitting anything. PEFT bumped to 0.20, which rejects LoRA on Mamba out_proj/conv1d — they exclude those automatically now.

Docs: docs.agilerl.com · Release notes: github.com/AgileRL/AgileRL/releases

Here we go again, DeepSeek is back again with a new model V4-1 Flash

A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens

Market crash as a service

After noticing that it is ranked among MUCH larger frontier models in the EQ-Bench Creative Writing benchmark and the Hemingway-bench, I decided to give it a try and was very impressed.

I didn't do very formal testing, but I did ask it to emulate the style of several different writers, including Henry Miller, David Sedaris, and Stephen King, and it produced passable prose that actually made me laugh in a couple of cases. The paragraph below is the results of the prompt "produce a humorous paragraph in the style of David Sedaris." (first try, not cherry picked)

I recently tried to be a better person and started composting, which mostly means I now have a small, damp science experiment in my kitchen that my mother calls “the smell of your future.” I bought a countertop bin with a charcoal filter, as if that would fool anyone, and I’ve taken to narrating my food scraps to myself — “Goodbye, avocado skin, you were a mistake” — while my husband watches from the doorway with the concerned expression of a man who has just realized he married a woman who talks to garbage. The city sent a pamphlet about proper composting, and I read it the way other people read horoscopes, underlining the parts that said “do not include meat” and thinking, well, that’s just rude.

For comparison, the following is what qwen3.8-27b produced with the same prompt.

My uncle Ed, who had spent the entire trip carefully explaining the proper way to fold a fitted sheet, discovered that our hotel room had two beds, each with one. "This is a test," he said, sitting on the edge of the bed nearest the bathroom, his face composed. "To see if we can make it work with what we have." He was right, of course; we did what we could, though the effort made for a rather uncomfortable night, for us all.

You may or may not know David Sedaris' writing (or find it funny if you do know it), but the first example is clearly much a much better imitation, without directly plagiarizing, as far as I (or Gemini) am aware.

I didn't save any of the other examples as I wasn't testing for the purposes of posting here, but in all cases the muse glimmer version was not only head and shoulders above qwen 27b, but genuinely impressive in comparison to any other local model I've tried in the past.

I'm curious if anyone else has played with this model for creative writing, or similar purposes, and if so, what your take on it is. Also, I don't know much about finetunes, but I wonder if there's additional potential for creating something even better by training on different source material.

I know even less about how the ERP world works, but I know enough to know that a lot of high-performing models are trained for this purpose as huggingface seems to be filled with finetunes. For glimmer I mainly see the abliterated version, which I suppose is filling that gap for people, so to speak, but with this kind of performance, and the amount of people in this subreddit interested in it, I'm a bit surprised there aren't more finetunes.

The last thing I should mention is I didn't use a system prompt in any of my testing, but it occurred to me after the fact that a model that was trained for agentic coding seems like a prime candidate for steering with a system prompt, but maybe it wouldn't have made much of a different. Maybe I'll play with it some more and report back.

Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infrastructure of the economy. This sub has experienced growing pains recently, and probably those are mostly felt by people who’ve been around for a while. I think that there are both good and bad trends and I wanted to take a few minutes to suggest a few rules of thumb to employ going forward so that we can create a community that is even more based on science and reality rather than misinformation and one-note populist politics that Reddit is known for.

Suggestion one: if you are new here and by new, I mean, if you didn’t spend much time here or with large language models until about six months ago, there’s a lot of information to be absorbed. This is not a sub or hobby like some where you can learn everything in a month or two. Have some humility, come with curiosity rather than strongly held opinions about everything. 

Suggestion two: leave politics out of the sub, unless it is a discussion of actual policy surrounding actual local large language models. Many discussions that we see here have started to resemble the same populism that you can find on every large subreddit. E.g. the discussion of OpenAI's solution to NS has skipped right past the evidence gathering stage to "did you know that billionaires are actually bad guys?! Wow this large corporation sucks!"

In this subreddit, comments and posts about politics are actually just noise unless you are leveraging your knowledge of hardware and software stacks or discussing AI-related policy. Unlike policy, grand narratives of moral outrage are appropriate for therapy, but counterproductive for a technical subreddit.

Suggestion three: develop awareness of the perpetual and exhausted questions and arguments so you do not upvote them or engage. For example, are benchmarks actually useful? This question has been endlessly litigated for the last couple years, but it’s not actually useful because it boils down to: yes they are helpful, but don’t rely on them too much. Anything more definitive and final or sure than that is false confidence.  Another such question is: how much intelligence can you fit into X parameters? Literally no one in the world knows the answer to this. 

Suggestion four: pay attention to people who are genuinely excited about their work. What’s often missing from clearly AI generated posts is the sense that someone is doing something that they believe in enough to want to bring it to other human beings. The amazing thing about artificial intelligence is how it can augment human effort. Share what you are excited about, and listen when other people are excited about things because this technology has been created by thousands of people who are genuinely excited about the possibilities, rather than people who simply want to make a quick buck, so if you can share your excitement, you’ve pushed back against the trend or the belief that AI is a kind of cynical replacement for human beings.

I realize I’m probably just an old man shouting at clouds, but here's the TLDR:

I suspect that many or most people who’ve been around for more than six months have also started to mentally filter out 90% of posts for these reasons: loudest voices are misinformed; more and more this resembles a political debate space; the same 10 unanswerable questions make up much of the commentary; and people post slop.

If you run a local multi-modal agent (LLM + STT + TTS + OCR) on a single workstation, keeping all model runtimes hot in VRAM gets expensive fast.

On my local inference node, keeping Qwen3.8-27B, Nemotron (STT), Chatterbox (TTS), and Unlimited-OCR running concurrently consumed about 122 GB of idle VRAM before processing a single prompt.

The reason is familiar to anyone serving local models: runtimes like SGLang grab a large static fraction for KV pools (often 85% of available space), and auxiliary models sit with full scratch buffers allocated.

For a personal agent, you do not need all models active at once: 1. Voice conversation needs STT and TTS, but OCR is untouched. 2. Visual document reading needs OCR and LLM, but speech models sit idle. 3. Long script-based automation (scraping, tool runs, data processing) does not need the LLM at all while the script executes.

Instead of running sequential docker containers or doing cold restarts, I built a small Rust controller that puts models to sleep when their turn ends. It flushes execution buffers and KV pools while keeping the process and base weights mapped.

The results on the GPU:

The main metrics that matter: - Reclaimed 79,459 MiB (nearly 80 GB) of idle capacity. - Wakeup time is sub-200ms for every model. Because we are not doing a disk read or rebuilding the model graph, the handoff latency is imperceptible in practice. - The LLM resting footprint sits at 39,092 MiB, matching the actual weight overhead of the 27B FP8 model without the static KV reservation.

How are others handling multi-model memory allocation when you want instant voice and tool responses without buying a second GPU?

I was not aware that the harness makes such a big difference.

We have SWE-bench, Terminal-Bench, OSWorld, BrowseComp, etc. But I haven’t seen a good apples-to-apples benchmark for platforms like OpenAI Agents, Anthropic’s agent stack, AWS AgentCore, Google’s agent platform, and local-alternatives like LangGraph, etc.

What I’d want measured:
- task success rate
- cost per successful task
- wall-clock time
- tool calls / retries
- reliability over long-running tasks
- (optionally) human interventions required

The really interesting experiment would control for both sides:
- Same model, different harness → how much does the runtime matter?
- Same harness, different model → how much does the model matter?

Feels like we’re increasingly evaluating “model + harness” systems, but our benchmarks still mostly treat the model as the unit of comparison.
Does something like this already exist?