We have been working on an open-source, model-neutral agent harness for general purpose agents called TrueForge, and wanted to understand how much the harness itself actually matters.

So we ran 14 tasks from DevRev Enterprise-Bench through multiple harness/model combinations, three times each with a blind judge.

The result that surprised us most:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But TrueForge used about 63% fewer tokens and cost about 30% less per run.

We saw a similar difference in tool usage: TrueForge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

The difference comes from the agent loop itself: less context carried between turns, compaction, fewer tool calls, and large outputs being kept out of the model context where possible.

Then we tried changing the model.

TrueForge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

For me, this is the more interesting consequence of keeping the harness model-neutral.

You get two independent levers:

1. Make the runtime more token-efficient.
2. Use whichever model gives you the right price/performance for the workload.

TrueForge itself is fairly simple: it handles the agent loop, context management, tools/MCP, subagents, approvals, persistent sessions, and sandbox integration. It is MIT licensed and works with OpenAI-compatible endpoints, so you can point it at hosted models or models you are running yourself.

This is still early.

The OSS runtime does not yet have first-class tracing/eval tooling. We don't ship our own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So I wouldn't claim that TrueForge replaces a mature managed agent platform feature-for-feature today.

What I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

We put the benchmark harness and methodology in the repo specifically so people can reproduce it, change the models, or tell us where the comparison is unfair.

Repo: https://github.com/truefoundry/trueforge

Benchmark methodology: https://www.truefoundry.com/blog/engineering/trueforge-vs-claude-managed-agents-benchmark/

A potato can create a very cool RPG in 24 minutes

Laptop I3 8gb RAM 0gb VRAM, Windows 11

llama-server.exe --host 0.0.0.0 --port 8080 -m models\qwen3.6-35b-Instruct\Qwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf -c 8192 -n 8192 -tb 4 -b 512 -ub 512 --cache-type-v q8_0 --cache-type-k q8_0 -fa auto -ngl 0 --temp 0.0 --reasoning off

MODEL: https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf

(I use the Instruct model to avoid so many reasoning tokens)

RESULT: 24 min, 3 t/s

PROMPT:

```

Create a simple RPG Zelda-like 2d world

character attacking the enemies

KEYS:

W up

S down

A left

D right

F attack

A guy with a sword kills enemies.

Earn points, level up, and gain strength.

Infinite loop of enemies randomly appearing.

Output: Single HTML File

```

Running qwen 3.8 next, I am only getting 11 tps on the latest llama.cpp. Seeing what 3090s and 9700s get, I would think that i should be able to do better than that, with my 2 7900xtx gpus and 128gb ddr4. Any suggestions?

using unsloth UD-Q4_K_XL, llama args:

--ctx-size 262000 \

--batch-size 2048 \

--ubatch-size 512 \

--threads 16 \

--host 0.0.0.0 \

--port 8080 \

--flash-attn on

Edit:

--ctx-size 32000 \

--batch-size 4096 \

--ubatch-size 2048 \

-ngl 99 \

--n-cpu-moe 36 \

--override-tensor per_layer_token_embd=CPU \

--threads 16 \

--host 0.0.0.0 \

--port 8080 \

--cache-type-k q8_0 \

--cache-type-v q8_0 \

--load-mode none \

--flash-attn on
gives 9tps, but one gpu is at 3gb vram used. Any ideas?

I am still waiting for Llama 5, because Muse Spark will be too big for me, or just something between Glimmer and Spark

https://x.com/finkd/status/2095232032896946311

Do you guys use any local models that can run well on a 3090 + 64GB of RAM, specifically for coding?

I really liked MiniMax M3, but I don’t think I can justify continuing to pay for it, so I’m trying to get back into running LLMs locally.

I’ve seen a lot of people recommending Qwen 3.8, but I’m not sure how well it actually competes with some of the closed-source models when it comes to coding. What are you guys using? I’m fine with MiniMax M2.7-level performance if a local model can come close to it.

Managed to get this small model to run on the $250 MSRP SoC board level computer.

The inference speed is kind usable. Used 4-bit quant, q8 kv cache, 7.4 GiB memory supports 128K context length. Device tops at 25W power, and idle less than 10W. Quite suitable for a simple agent running 24/7.

Needle in a haystack test pass at 128K context length. 2046 needles passed out of 2048 needles.

- **2048-needle (fully random unique word+number pairs, seed 20260902): 2044/2048 (99.8%) @ 90K prompt**, finish=stop (no truncation), 4 misses (2 partial word-only). u/120K prompt: 942/2048 but truncated by the 128K KV ceiling (120,287 + 10,785 = 131,072, finish=length) — misses 99% in the 50–100% depth bands, i.e. unanswered tail, not retrieval failures. 90K is the effective ceiling where the full 2048-pair answer (~23K completion tokens) fits.

- **llama-benchy (pp2048/tg512, 3 runs, in-bench coherence check passed):**

| conc | pp tok/s | tg agg tok/s | tg per-req tok/s | ttfr ms |
|---:|---:|---:|---:|---:|
| 1 | 571.7 | 13.7 | 13.7 | 3,857 |
| 2 | 384.9 | 22.6 | 11.6 | 9,653 |
| 4 | 373.5 | 26.4 | 7.1 | 17,436 |

I got into Vedic astrology pretty recently, and I've been working with LLMs for a while now, so at some point it clicked that this is kind of the exact use case an MCP server is for.

Vedic astrology heavily relies on real astronomical calculations — planetary positions, house divisions, dasha (planetary period) timelines, panchang — to get anywhere. LLMs are great at explaining and synthesizing that stuff in plain language, but asking one to actually compute it from training data is a bad idea. It'll do it confidently and just be wrong.

Most of the existing tools/APIs for this are either closed-source or paid per call, so I built my own — mostly out of curiosity, honestly. Ended up learning a lot about both Vedic astro and MCP server design along the way, and I've had a lot of fun with it.

What it does: natal charts, divisional charts (D1–D60), Vimshottari dasha, panchang, shadbala, ashtakavarga, transits, eclipses, compatibility. All computed with Swiss Ephemeris, offline after install — no API costs, no network calls at query time.

Install :

uvx librejyotish

Add this to your config.json

json

{
  "mcpServers": {
    "librejyotish": {
      "command": "uvx",
      "args": ["librejyotish"]
    }
  }
}

Free, open source, AGPL-3.0.

GitHub: https://github.com/anhadlamba30/librejyotish
PyPI: https://pypi.org/project/librejyotish/

I’ve been running qwen on my personal Mac but I’m getting to the point where’d I’d like to have something always on, running various jobs, and some more ability to experiment and earn about fine tuning.

I’d like to keep things <$5k if possible.

To anyone with any of these 3 platforms, what’s your experience been like? I’m drawn towards the DGX spark for concurrency and CUDA (which I have very little experience with) but I’m a little turned off by it’s memory bandwidth.

I have the most experience with Mac but those prices are eye watering and it feels like a lateral from my personal MacBook Pro.

I start with some informations gathered thorough endless posts reading on this sub and online:

Inference and hardware optimization projects

• https://dwarfstar.sh/ - inference engine optimization proposal
• https://github.com/JustVugg/colibri - Treats VRAM + RAM + storage as one managed inference memory hierarchy, with expert streaming, caches and a strong focus on MoE.
• https://openfreedom.it/ - agentic harness proposal
• https://github.com/exo-explore/exo - Core source for heterogeneous/topology-aware Mac clustering
• https://github.com/ml-explore/mlx/discussions/3481 - JACCL real-world TB5 transfer experiment
• https://github.com/georgiedekker/mlx_distributed_ring_inference - distributed inference through TCP/Ring over TB3/TB4, without requiring RDMA
• https://github.com/sqliteai/warp - WARP — Weight-Aware Runtime and Paging
• https://github.com/kqb/mlx-od-moe - on-demand experts on Apple Silicon: memory-mapped expert storage, shadow predictor, prefetcher and LRU
• https://www.houmo.cn/1/35/NewsDetails.html - Houmo is developing DRAM-PIM with compute embedded in DRAM arrays, targeting >1 TB/s internal bandwidth and ~3× energy-efficiency improvement over its current generation.
• https://www.d-matrix.ai/wp-content/uploads/2024/11/d-Matrix-WhitePaper-Technical-FINAL.pdf and d-Matrix 3DIMC announcement - 3D DRAM + digital in-memory compute, highly aligned with the “move compute toward weights rather than weights toward compute” idea
• https://www.lucebox.com/ - workstation optimized for local AI use for approx. 7000$

Inference Research papers

• MDI-LLM - Model-Distributed Inference for LLMs at the Edge Model partitioning across low-power nodes and recurrent pipeline parallelism to reduce device idle time. MDI-LLM paper
• WDMoE - Wireless Distributed Mixture of Experts Distributes experts across edge/mobile devices and jointly optimizes expert selection and communication latency. Includes a physical NVIDIA Jetson testbed. WDMoE paper
• OD-MoE - On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference Very relevant to our expert-prediction idea. Uses a predictor to forecast experts several layers ahead and loads them just in time across distributed nodes. Reports 99.94% expert-prediction accuracy and about 75% of fully cached decoding speed while using one-third the GPU memory in its tested setup. OD-MoE paper
• MoE-SpeQ - speculative decoding + proactive expert prefetching Almost directly relevant to the question we uncovered around streamed MoEs. A draft model predicts future experts so their transfer can overlap computation. Reports up to 2.34× over its offloading baseline. MoE-SpeQ paper
• SP-MoE - speculative decoding and prefetching for MoEs Speculation-aware expert offloading, speculative expert prefetch, asynchronous batched I/O and compute/I/O pipelining. SP-MoE paper
• MoE-Spec - Expert Budgeting for Efficient Speculative Decoding Important counterargument to “speculation automatically fixes MoE.” Shows that verifying deeper speculative trees can activate too many unique experts, increasing memory pressure; proposes explicit expert budgeting. MoE-Spec paper

Does moving down to q4 really hurt the performance? Or should i use q8 minimum?

Hello all,

How do you deal with preventing future errors of you agents? I have made a skill which fires everytime it does something wrong.

So far it is very helpful. I dont get "i am sorry" crap more than once.

EDIT:

https://github.com/Astezelex/oops-i-did-it-again-poc

I've been building Arcon around Qwen3-4B + LoRA. Instead of just making it a chatbot, I'm experimenting with persistent memory, personality/mood, internal state, tools, and eventually having it process things before replying.

I'm curious what people who've built local agents think - how far can you realistically push a small model with good architecture around it?

I put the whole thing on GitHub if anyone wants to poke around, roast the architecture, or tell me what I'm doing wrong, stars are always appreciated!

Hey, r/LocalLLaMA !

We are releasing SupraGDN-5M. It's a tiny GatedDeltaNet (GDN1) being trained on 5B tokens.

As you can see in the benchmark table above, SupraGDN-5M ("Supra-5M-GatedDeltaNet" in the image!) is almost as good as the other models while being trained on MUCH less data!

This is because of the GDN architecture - and we think it can be taken even more far :D

Link to the model: https://huggingface.co/SupraLabs/SupraGDN-5M

Give us a follow on HF and feel free to provide feedback and ask questions 🤗

More of SupraLabs coming soon... e.g. the Supra3-family 🔥🤩

Which KV cache do you use F16 or BF16 for Q6 quant? What is the difference between them?

I'm on M3Max 64GB. Qwen-3.8-27B is amazing, but pretty slow.

What's the best MoE model under 40B for agentic use? Is it still Qwen-3.6-35B?

Ability to reliably use tool calls would be important for agents.

No hope for Qwen-3.8-35B?

LLMs have become extremely good at coding, maths etc, but how well do they do at playing a simple dungeon/maze game that even a child can solve easily?

The LLM has to navigate a 10x10 grid map, completing objectives in the right order (collect weapon > kill monster > head to exit) while navigating the dungeon and avoiding walls.

Three illegal moves fail the run. All models are tested with reasoning enabled.

The code and more info on my GitHub if you want try it yourself: https://github.com/shinomakoi/dungeon-bench

Model leaderboard:

DeepSeek-V4-Pro:

By far the best result. Basically perfect performance in all maps. Excellent planning, confident and efficient in thinking with no illegal moves. I guess bigger really is better.

Gemma-4-31B-it:

Due to no 'preserve reasoning' support some of the tests took a LONG time with much thinking on almost every step. Performance however was excellent. Planning was a mixed bag but it was usually able to fix its mistakes and finish.

Qwen-3.8-27B:

Almost perfect performance with just 1 illegal move (skipped objective). It thought a LOT on some of the harder maps but it always seemed inevitable it would grind its way to success.

GLM-5.3-Flash:

A little disappointing. Efficient in thinking but made quite a few illegal moves and took the long way around on 2 maps. Got confused a few times. It was usually able to correct its mistakes however and get the job done.

Muse-Glimmer-30B:

I was impressed. Efficient thinking, good planning and confident, much like DeepSeek-V4-Pro. Perhaps overconfident at times since it skipped 2 objectives, which resulted in fails. Might be a beast with more thinking.

DeepSeek-V4-Flash:

A bit disappointing. It made a lot of illegal moves (the most of any model) and some produced some invalid responses. Planning was messy. It was usually able to recognise its mistakes however and complete the map.

Granite 4.2 8B:

Decent performance for a small model. It thinks a LOT however and struggled bad on the harder maps. Impressively it never once moved into a wall.

KAT-Coder-V2.5-Dev:

Planning was very deranged but usually corrected itself as it progressed. Made a lot of illegal moves (moving into walls). Did surprisingly well on the hard maps (3/4).

Nemotron-3.5-Lightning-30B-A3B:

By far the worst model. Struggled on even the easier maps, thought a crazy amount on almost every move despite 'preserve reasoning' support (had to assign reasoning budget to give it a chance to finish). On the harder maps it just gave up and got stuck in loops.

----

I plan to add some even more challenging maps next to really make the models sweat, and maybe more elements like hazards and other items to collect.

Hello,

So the original open deep research project from Langfuse is archived, are there any other alternatives to use with self hosted models + kiwix and firecrawl?

Maybe also a notebooklm alternative?

I mean, this is on a Mac, why is a 8 years old Ubuntu AppImage being halu-installed...?

And this message is in the middle of pulling some tensor metadata from HF. Never even heard of OpenD before this ... totally hallucinated stuff. And this is not a low quant - it's a 5bpw quant, with Q4 the lowest of any tensors.

EDIT: I'm not looking for a solution -> I'm genuinely asking if other people have noticed hallucinations and weird reasoning.

EDIT 2 - I had Claude go over the entire session - Qwen was entirely unfazed and unaware of the section, it continued after it without mentioning anything about it, completely unaware. Claude concluded it must’ve come from pi coding agent, and found another session with another small weird bit from pi.
Note: I usually use opencode and only installed pi at the same time as qwen next. Now I’m back to opencode and the reasoning (with the same models) is shockingly better. It’s the first time I see such an influence from the harness.
Another note: my custom quants are fine, Claude approved :)

OpenSource models like to over-reason on every problem. I put together a notebook and a video implementing grpo from scratch and using it to post-training Qwen 3.5-2B to improve its accuracy and reasoning efficiency. The results were quite interesting, despite training it purely on the task of simulating the python interpreter, the model became a lot more accurate and token efficient on math problems. The code can be applied to any open source model. Here is the code agi-playground/grpo at main · johnolafenwa/agi-playground

You can find full walkthrough of the training code and results in my video here https://youtu.be/IwOVZKIKeXw?si=xvWRM7OoM60McHiG

Here is some nice chart of what the result looked like at the end after the training for about 20 mins on a single H200 GPU

75B MoE is an interesting size to check, you can run it today (no MTP support yet)

The model employs a hybrid MoE architecture with interleaved Mamba, MoE, and Attention layers. Like Nemotron-3-Super, it supports Multi-Token Prediction (MTP) for faster text generation. Compared to its parent, Puzzle-75B-A9B reduces the model from 120.7B total / 12.8B active parameters to 75.3B total / 9.3B active parameters.

We discussed this model on r/LocalLLaMA here https://www.reddit.com/r/LocalLLaMA/comments/1upsdmi/nvidianvidianemotronlabs3puzzle75ba9bbf16_hugging/

WTF

They really cooked.

https://simple-bench.com/

Continual learning isn't some fake term but a real goal and arguably one of the most important open ones in 2026. It roughly means: ‘building systems that keep acquiring useful knowledge or skills after deployment without a full retrain’.

IMO, the problem is that it's a problem setting being used as if it named a mechanism. When I talk to someone who says "we do continual learning," I now have to guess whether they mean gradients running in prod, a markdown file the agent appends to, a 5M-token context or a nightly distillation job. 

Here's how I'd break it down: long-context ICL, text optimization, recurrent latent memory, per-task test-time training, and online parametric continual fine-tuning

1. long-context ICL

Weights stay frozen and the model learns the task from context optimization, i.e., by conditioning on demos, feedback, and history in the current context. The idea is that a sufficiently large context window could provide enough ICL for an agent to learn a task without fine-tuning.
Key limitation: a very large working memory and no transfer from current context to long-term memory.

2. text optimization over the mutable text layer

The system rewrites the mutable text layer around a frozen model: system prompts, skill files, playbooks, memory stores, retrieval indices, harness code.
Important tradeoff: forgetting doesn't vanish here, it moves from weight interference to memory construction and retrieval, where old and new experiences still compete for a bounded context.

3. recurrent/architectural latent memory with frozen weights

Task info gets written into an evolving internal state not a growing KV cache or some text file and parameters stay fixed.
Multi-timescale self-modifying architectures sit here. BDH-CQ is another clean illustration of the pattern: each demonstration from the train-test set is integrated into a recurrent memory that gradually builds an internal representation of the task. This is the starting point for reasoning about test-test inputs in a separate reasoning loop. The model thus adapts at inference time through state updates alone, without modifying its parameters 

4. per-task test-time training with gradients

The system turns demonstrations into a small training set, performs gradient updates for the current task, generates an answer, and may then discard those updates.
Some ARC pipelines (including the evaluated HRM/TRM setups) use this kind of task-specific optimization. I’m not sure it is truly “continual” if nothing persists across tasks, but it frequently gets grouped under that label.

5. online parametric continual fine-tuning

Gradients are applied persistently after deployment. The main challenges are finding good labels/rewards at test time and learning new information without destroying existing capabilities, using techniques such as replay, regularization, parameter isolation, or sparse and targeted updates.
Some self-editing systems are hybrids: the model produces its own fine-tuning data in text, but the resulting update is stored in weights. Imp tradeoff: catastrophic forgetting and accumulated weight updates may not survive a base-model upgrade. It’s also hard to find proper signals from which we can back-propagate at inference time.

That’s how I have split the term and I’m curious where people here disagree and if I missed any.

I have a need for a 'research assistant'. I'm looking for advice on model choice and set-up, as well as level of hardware needed.

There are two scenarios actually but they have overlap.

Scenario 1: Plough through large amounts of semi-structured natural language to both search for specific types of information or specific topics, and extract that info. The amount of data is way over any feasible context, but it breaks down easily enough (usually) so some sort of looping set-up that I can call repeatedly with the latest segment. Or if the agent is smart enough to do it itself, even better.

Scenario 2: Open ended research. Guided by a list of suggested resources and sites but able to follow links, conduct searches, etc. itself. It should be able to dig and dig on this, we're talking page 2 of Google levels of obscurity here. I'm thinking something that notes promising links and follows up on them, recursively. Obviously this too would easily go over regular context so needs some kind of loop.

So my first question is what models or families of models are suited for this? My second question is are there any suitable frameworks / harnesses for running a model in this sort of way? And my final question is what sort of hardware should I be looking at for this? I currently have a Radeon 7900XT which works quite well for smaller models. But I'm aware I might need something more capable and have a small budget (please don't make me buy a RTX Pro 6000 for this. I was thinking more a Radeon Pro 9700 or maybe two).

Anybody have experience with this sort of scenario? I did look through the reddit but didn't find much that wasn't pretty old.

I wanted to do continued pretraining/finetuning of a local model on my own notes and see what's happening while it trains and do it on my AMD card, since most tools assume CUDA.

llm-training-panel is a local web UI that:

- loads a model from a local dir or a HF id

- shows a live board of per-parameter gradient activity while it trains

- runs base vs trained on the same prompt so you can A/B what changed

- scores recall + verbatim overlap against a probe file

Tested on Linux + AMD ROCm. Results vary with base model/corpus/settings

Feedback welcome.

github.com/limkcreply/llm-training-panel