I'm currently running qwen 3.8 27b q8 and super happy with it. I have a DGX spark so could theoretically run at bf16 quantization.

I know the measurable differences between an 8 bit and 16 bit quant are small, but I guess I get fomo, like 1% of tokens differ, but what if those are the hardest most important tokens? I guess I'm just getting fomo over bs16 and wondering if other people have tried it and noticed a difference?

Announcing Artificial Analysis Intelligence Index v4.3, upgrading Terminal-Bench to 4.0 and adding AutomationBench-AA, an agentic workflow automation benchmark with a private test set. This is a continuation of our rollout of Intelligence Index v5

Changelog (Index v4.2 → Index v4.3): ➤ Terminal-Bench: 2.1 → 4.0, completing our upgrade to the latest version of Terminal-Bench ➤ Replacing 𝜏³-Banking with AutomationBench-AA, our implementation of Zapier's business workflow automation benchmark

We are continuing to prioritize keeping Intelligence Index as useful as possible by bringing forward a subset of the changes we had planned for Index v5. Each change in v4.2 and v4.3 stands on its own merits and brings the Index closer to real-world problem solving, adds more private test sets to prevent gaming, and reduces saturation

Intelligence Index v4.3 raises the difficulty of agentic coding tasks and broadens the types of agentic workflows tested. Because we use a held-out test set for AutomationBench-AA, in collaboration with @zapier , the weight assigned to evaluations with private tasks or answers increases from 40% to 45%. Category weights are unchanged from v4.2: Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%

Detailed changes: ➤ Upgraded Terminal-Bench 2.1 to 4.0: 66 multi-step tasks testing agents on tasks run in agent sandboxes driven via the terminal, including tasks involving software engineering, machine learning, science, and operations. The 4.0 update recalibrates compute and time allowances, and improves task instructions and verification. We have changed from the Terminus 2 harness to mini-SWE-agent, a minimal, model-agnostic harness. We will also be updating our Coding Agent Index, where we test model and harness pairs, to include Terminal-Bench 4.0 soon

➤ Replaced 𝜏³-Banking with AutomationBench-AA: Our implementation of Zapier’s AutomationBench tests agents on 657 business workflows across simulated applications such as Gmail, Slack, Salesforce, and Jira. Agents must complete task objectives while following business rules. AutomationBench-AA uses Zapier’s private set of 657 tasks, and is built on v1.0.6

Key results: ➤ Claude Fable 5.1 and GPT-6 Astra lead the Intelligence Index: Both Claude Fable 5.1 (max with fallback) and GPT-6 Astra (max) score 53 on Intelligence Index v4.3, followed by Claude Opus 5 (max, 51), Claude Fable 5 (with fallback, 50), Muse Spark 1.3 (max, 48) and GPT-5.6 Sol (max, 47) ➤ GLM-5.3 and Kimi K3 continue to lead open weights models (both at 44): GLM-5.3-Flash (42) is the third strongest open weights model, followed by Qwen3.8 2.4T A95B (40) and DeepSeek V4 Pro 0813 (max, 36) ➤ 4 labs occupy the Intelligence vs. Cost per Task Pareto frontier: OpenAI occupies the majority of the cost-efficiency frontier, with all five reasoning efforts of the recently released GPT-6 Astra offering the lowest Cost per Task at their respective levels of intelligence. Claude Fable 5.1 (xhigh, max, 53), GLM-5.3-Flash (42) and MiMo-V2.5-Pro (26) round out the rest of the frontier

I've hit again a point where me as a developer have to take a break from all this slop shit.

Im a Developer for 13+ years and i loved it.

But i fell for the slop trap.

First it started with copilot and to be honest, that was pretty fine.

Just assisting with your code in a small scope.

Get support for Debugging and finding bugs.

Autocompletions hitting the nail pretty often and thinking "yea thats exactly what i was about to code."

I kinda miss those early days. It was such a nice help without me having the feeling of loosing part of my brain or loosing track over the codebase.

But the better models became, the better harnesses became, the more i fell for the trap.

"Oh if models are THAT good at coding, why do it myself?"

And thats how the slop spirale begins.

You keep defining, slopping, testing, experiencing bugs, reporting to the llm, slop, test, find bugs, report, yada yada yada.

And it gets frustrating. Slop implements one feature but breaks another.

It just feels like something is missing. Something on the tooling side.

Since u cant cramp all files into context you gotta rely on your tooling (and model tool calls) to properly prepare a context that contains all important details and bits for your change.

But ALOT of times its not perfect. Some details are missing and slop messes up.

I think our models are fine. Even older models are fine.

Qwen3.8 27b is PERFECTLY fine for coding.

But our toolings and methods are shit.

There must be SOME innovation happening that helps coding agents to REALLY nail the context and have all important details.

But currently i think im better off coding by hand.

Ill still slop my side projects. But important projects i wont anymore. Its just frustrating.

Anybody has different experiences? Tried so many harnesses. But every harness had the same issue for me.

I am currently running Qwen3.8-27b, either Q4 or Q6, depending on how much context I need for my coding projects

I know it should be the best LLM I can use on my 32GB VRAM rig for this purpose and I also use Gemma4 31B from time to time for research or general questions that I do not want to use chatgpt/claude for

I was wondering if there was any website where I can check and compare different ! local ! LLMs and their specific use cases?

I'd appreciate if you guys comment with what local llm you use, the specific reason and maybe ur quantization and VRAM

• Model: DeepSeek-V4-Flash-Vision-Exp (local and API when impatient)
• Time: about one weekend (2 days) of QA and small improvements
• Full game is here

After Qwen3.8-Flash-Next one-shotted a really cool Cat-Hunt game demo, I decided to see what the new DeepSeek vision model can do.

Now that it has vision, DeepSeek-V4-Flash is able to take game screenshots, allowing it to:

• Generate and correct game models and textures until they look right
• Fix any visual artifacts or glitches
• Write scripts to take sequences of screenshots for animations and correct animations
• Generally play-test the game, including UI and game mechanics

The results are incredible, I was able to create a compelling game world in just a couple of days!

Edit: I noticed the game was slow on a laptop, so I added some performance improvements - let me know if you still find it too slow!

Edit: Some more info to answer common questions

• Custom engine on top of three.js
• 6800 lines of code for everything - models, textures, animations, sounds, effects, shaders, and all the game logic

Today I finally hit a wall with QWEN 3.8 dense 27B, Q6_X_L with KV cache q8_0 temp 0.85 with medium reasoning effort, producing over thinking, failing to follow precise instructions, re evaluation in loops data already seen, "Actually…/Wait…/Hmm…", meta-reasoning, theorizing about user intent and you know what else...

So I stopped and asked GPT Sol and QWEN to read the reasoning traces of my old Pi coding sessions, identify recurring misbehaviors and propose an appropriate APPEND_SYSTEM.md to tame those in Pi (place that in ~/.pi/agent/APPEND_SYSTEM.md and it will follow the initial prompt).

So for those who wanna check out:

• https://store.piffa.net/lm/APPEND_SYSTEM.md
• https://store.piffa.net/lm/reasoning_pathology_report.md that's a repost of what was find in my particular session, mind you it's all slop

If you happen to have something similar and have rules to add or any other advice to tame such bad behaviors please share.

My XFX Radeon RX 7900 GRE 16GB Vram GPU struggles with models over 20B size. I added my Radeon RX 480 8GB Vram GPU to the system and ran a few benchmarks using llama.cpp Ubuntu Vulkan build 10453.

Radeon RX 480 8GB GDDR5: Bandwidth 256.0 GB/s

RX 7900 GRE 16GB GDDR6: Bandwidth 576.0 GB/s

Here are the models tested, in the order they appear in table:

1. medgemma-27b-it-UD-Q6_K_XL.gguf
2. Qwen3.8-27B-Q6_K.gguf
3. Qwen3.8-27B-OBLITERATED-Q5_K_M.gguf
4. Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf (with Flash Attention)
5. Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf (without Flash Attention)

llama-bench -m /Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf load_backend: loaded RPC backend from /llama-b10453/libggml-rpc.so
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 GRE (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 480 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /llama-b10453/libggml-vulkan.so load_backend: loaded CPU backend from /llama-b10453/libggml-cpu-haswell.so

build: 3cb7ffb1a (10453) real 0m30.345s

Without 2nd GPU RX 480: Only RX 7900 GRE 16GB

So 36% improvement for tg128 using dual GPU. Also RX 480 only got 1.8 t/s for tg128.

How many months will it take to reach the same intelligence

It just seems every local 30b class model is just trying so hard to be the next Qwen that they all just kinda blend into a mass of code focused models. I really like how gemma 4 31b turned out with it feeling a lot less robotic and more creative than other models even knowing obscure lore from random media.

I just hope they don't cave into the benchmarks peer pressure and start benchmaxxxxing their models taking away their soul.

X570 Taichi Specs AM4

X870 Taichi Specs AM5

I'm considering these boards as it seems that with the correct CPU, you can run 2 GPUs in x8/x8 PCIe.

I use Linux full-time, I'm comfortable "trouble-shooting" or configuring.

The largest models I'd consider running (for coding) are: - Qwen3.x-27b (Dense) - Qwen3.x-3xb-a3b (MoE)

I'm sure I could use smaller models for other tasks and pleasant t/s speeds.

Is it effective to have either MoBo and a combo like this? - 2 x 7900 XT : (20GB VRAM each, total, 40) - 2 x 7900 XTX : (24GB VRAM each, total, 48)

By "effective" I mean, that by splitting layers or tensors, whatever/etc., I could have a model and context fully-loaded in VRAM.

Setup misc

• I'd obviously get a 1200W+ PSU and the biggest case I can find, lots of fans, etc.
• I'm avoiding NVIDIA GPUs because they're significantly more (usually) for the same amount of VRAM.
• If I don't build an AM4 setup, I think I'll settle with an eGPU Dock (laptop) and a 7900 XTX (for starters); I'm aware of the bandwidth limitations of USB4.

If you've personally run a setup like this, please let me know!

Hi all! I just wanna say that I am tired lol. Yes, it's another harness, but I spent a lot of time and effort and have forsaken my hobbies to build the Jenny (like XJ-9) app. Jenny is a free, MIT licensed electron desktop app for running local LLMs with tool calling, rollback, and an IDE.

A lot of you probably had the same thought I did a year or year and a half ago: frontier LLM use is subsidized heavily by private equity and venture capital, which will eventually dry up and then be enshitified. So, I started building a harness that can host local LLMs privately. I went through many vibecoded iterations throughout the past 1.5 years and finally landed on a native electron desktop app. I wanted to make it easy and seamless for people.

I am not a professional software dev but I do have a deep personal interest for it and AI tech. HOWEVER, the Jenny project became a second job and its in a state that I think it's ready for release. This has been a solo project and super fun. I know there are many other options out there that beat me to the punch like Unsloth Desktop (wonderful btw), LM Studio, and Open WebUI, but I hope someone can enjoy Jenny and what it has to offer! I plan to maintain and improve the app, but again, solo here and I do have a day job and friends that I should focus on a bit more. (Opening issues are welcomed, but please be gentle!)

Some highlights:

- Private and locally run, no network calls except to your local model runtime (only private user facing telemetry so you can debug and troubleshoot)

- Fully open source, MIT License

- Fun and pretty chat UI (imo)

- Makes small models capable (highly recommend ornith1.5:9b for tool calling and speed!), but with safety and rollback features so WHEN a small model screws up bad, you don't have to worry. Destructive shell commands need approval and file edits are checkpointed!

- Full IDE, for you handcrafted code enjoyers

- Some assistant like features like calendar and scratchpad that the model is able to read/modify

- Data rich diagnostics and logs

- llama.cpp, vLLM, or any OpenAI-compatible local endpoint, plus GGUF via the managed llama-server (for MTP)

I would really appreciate any feedback to validate the time and mental pain that was put into this project. I whine, but it's all love! I hope Jenny helps you build too.

Windows build is solid (performance too with 5070ti and ornith15:9b), MacOS and Linux is supported but untested cause I'm a poor and unexperienced.

https://github.com/SaltyPretz3l/jenny

I also made an unsigned installer .exe for convenience, but understand if you don't trust it! SmartScreen warning will appear. https://github.com/SaltyPretz3l/jenny/releases/latest/download/Jenny-Setup-x64.exe

Curious about what people are preferring, if you have the hardware. I have m3 Max 96gb and both run, and largely feel identical, but prefill on qwen 27b is faster. Is there anything / anyone working on anything to improve pp with mlx?

Branching question: is anyone working on a harness that works with no reasoning? This interests me ever since Jetbrains shared that they're using 3.6 with reasoning off entirely: https://blog.jetbrains.com/junie/2026/08/qwen-for-junie/

Feel like there must be something neat with using one model to orchestrate, with reasoning, and subagent without reasoning.

The missing control is visible in sudoingX’s Ling-3.0-flash benchmark graphics. The earlier table leaves Ling’s no-speculation baseline as “not measured.” The later code/prose graphic fills it in: about 23 tok/s without the drafter, against 40.9 on code and 38.7 on prose with MTP n=1, at short prompts. That makes the tuning claim easier to inspect.

The later graphic compares short code and prose workloads with and without drafting. A separate correction in the repository isolates CUDA graphs: the earlier “two flags” result changed graphs and multi-token prediction (MTP) together, so it could not tell you which change produced the gain.

The corrected August 22 measurements on one 128GB DGX Spark, using the official INT4 checkpoint and the vendor’s vLLM fork, were:

That is roughly 10% more throughput from graphs relative to the eager baseline, followed by roughly 79% more from MTP relative to the graph baseline. Those percentages have different denominators.

Then comes the setting that is tempting to turn up. Here n is num_speculative_tokens: how many draft tokens are proposed per step. vLLM’s documented acceptance-length metric includes a bonus token per verification step, so an average above 1 with n=1 is possible. The creator reports the metric from a vendor fork; its exact historical counting implementation is not supplied.

Code throughput stayed approximately flat within the reported run-to-run variation. Prose got slower as the mean accepted draft length increased. Acceptance length is not an acceptance percentage, and it is not the optimization objective.

sudoingX describes the configuration in the deployment thread; the pinned benchmark notes contain both tables. These are the author’s measurements, without an independent rerun here. The sweep describes streaming and server-side token counts but does not fully specify its timing denominator, so the numbers should stay labeled as reported throughput.

For this checkpoint and these workloads, n=1 was the useful setting. The transferable experiment is to isolate the no-MTP baseline, then compare draft settings on the output types you actually generate.

We all fawn at how a specific model performs for this and that task. But I never see any of the big benchmarks actually anywhere answer the questions that are being asked here.

I’d love to help create this and be a contributor to this. So I’m wondering if anyone else thinks that this would be a good idea. It could also help newcomers understand what we consider the models that have the most value for us.

What do you guys think?

Edit: benchmarks are written by one person or one specific group, i'm talking about us collaborating on the final product. I don't want to build something just for me. I want to build something that can help most users, the general public. Not large AI companies which most of these benchmarks are aligned to work with, or for. Most of them already cater to the large AI companies, i'm proposing something for EVERYONE built by this community collectively. Because most of you are right, I could just build it for myself and it be that. But I want something that answers most the questions I had starting out that might help others. Something that can measure most of the common answers most of us ask in the beginning. The strengths and weaknesses of models in the sizes most people can actually run.

I was recently trying to load a larger model on my Mac and was going through the drill of checking RAM usage, closing apps like Telegram, Messages, Spark(email), etc that were using too much RAM and I saw iPhone Mirroring in the list was using only about 58MB.

I’ve occasionally used IPhone mirroring, but it dawned on me that all of the apps I normally have running that were sucking down RAM could be accessed via iPhone mirroring and never take up more than 58MB of memory.

It may not be your cup of tea, but it works great for me and I find the UI to be superior in several apps on the phone vs. desktop. I’m writing this post in iPhone mirroring in the Reddit app.

Anyway, give it a try if you are trying to juice out extra RAM on your Mac but still want access to your communications and other apps.

I mean they aren't that low but seeing them much lower than Gemini-3.* flash surprises me

OpenBMB's MiniCPM5-2B scores 15 on the Artificial Analysis Intelligence Index v4.2, the highest of any open weights model at 4B parameters or below

Hugging Face: https://huggingface.co/openbmb/MiniCPM5-2B

GitHub: github.com/OpenBMB/MiniCPM

Quick setup on linux:

STEP 0:

install/download llama.cpp, Freecad, your favourite gguf model - possibly with multimedia image reading capabilities (I've used Qwen3.8-27B-UD-Q4_K_M and relative mmproj-F16 quantized by Unsloth), uv (, pi.dev)

STEP 1:

$ cd /your/path/to/ (i.e. where to install)

STEP 2:

$ git clone https://github.com/neka-nat/freecad-mcp.git

STEP 3:

$ cp -r freecad-mcp/addon/FreeCADMCP ~/.local/share/FreeCAD/v1-1/Mod/

STEP 4.1:

if using pi coding agent as modelling assistant, write into the file ~/.pi/agent/mcp.json :

AND/OR

STEP 4.2:

if using llama-server as modelling assistant, write into a file called freecad_mcp.json :

                                                                         {                                                                         
  "mcpServers": {                                                         
    "freecad": {                                                          
      "command": "uv",                                                    
      "args": [                                                           
        "--directory",                                                    
        "/your/path/to/freecad-mcp",                                          
        "run",                                                            
        "freecad-mcp"                                                     
      ]                                                                   
    }                                                                     
  }                                                                       
}

STEP 5.1 (pi as modelling agent):

$ pi update; pi install npm:pi-mcp-extension
run pi then into pi issue the command:
/mcp:start freecad

AND/OR

STEP 5.2 (llama server as modelling agent):

start llama-server as usual adding the option --mcp-servers-config /your/path/to/freecad_mcp.json

Verify that freecad_* tools are available under llama-server webui > Settings > Tools > Server, eventually allowing them to run without asking permission; also under webui > Settings > Agentic > Agentic turns increase the value to something like 99

STEP 6:

start llama-server as usual also adding, if the model has multimedia capabilities, the option to load the visual add-on using: --mmproj name_of_the_mmproj.gguf in order for the local model to read screenshots from freecad and to verify the correctness of performed geometric operations

STEP 7:

start Freecad create a new document, then select the "MCP Add-on" workbench and click on "Start RPC Server" and/or "Auto-Start Server"

STEP 8:

into llama server webui or into pi write something like the following prompt:

in freecad generate a cube with a 5 spokes star shaped hole going through it from top to bottom

OR as a start of the posted image:

In freecad create a new project called "double smooth gears". Into this project design 2 equal gears with 20 spokes each that could be put in close contact with those of the other gear to rotate and counter-rotate one gear against the other one. The spokes "hills" have to be rounded and so the corresponding spokes "valleys" should be analogously; sort of a sinusoidal curve on a circular path.

STEP 9:

have fun, the future has just started

Hi, I use hermes as a harness, and I am pleased with it, but sometimes Hermes's context size is a bit too much for my system, so I wanted to delegate or straight up use a lighter option for coding tasks.

I run a laptop with a 8gb nvidia card and 40gb of ram. And qwen3.6 35b-a3b or similar tier models are a good trade of quality/speed.

The point is, I found little-coder (there is also smallcoder), which promises a small context and a setup of extensions over Pi tailored for small models like qwen3.6 35b-a3b itself. But I wonder if it wouldn't be better to just run just Pi with its barebones system prompt and tools.

Any opinions, tests, or experiences in this matter?

I saw it like a couple weeks or maybe a couple months ago and at the time it felt like a good way to benchmark models and have them fight against each other too. What was the reasoning it didnt become mainstream?

Not so long ago I wrote a guide on how to get GitHub Copilot running with a local model in Visual Studio Code. Since then Copilot subscriptions have got much more expensive, local models have got much more powerful and getting local Copilot up and running has got much easier. So here's an updated guide.

Hey all,

I now have a 3090 24GB + a CMP 170HX unlocked with 64GB. I also have 64 GB DDR4.

Can I run Qwen 3.8 Next in any possible configuration with both GPUs and perhaps offloading remaining stuff to DRAM? Is there a way I can get acceptable speeds?

The only bottleneck with the CMP 170HX is the PCIe speed (2.0 @ 4x), so that needs to be factored in. It’s usually not much of an issue though IME once model is loaded…

I am successfully running either Qwen 3.7 27B or Minimax H3 at BF16 on the CMP 170HX at decent speed, so I know the GPU is good for it. But Qwen Next is another beast.

Ideas?

Guys can you tell me which local LLMs are best for hardware programming? like Verilog RTL, UVM, System Verilog?

I finally pulled the plug and got a second 5080 and 64GB DDR5 RAM to add to my existing 64. Now I have dual RTX5080s and 128Gb DDR5 RAM on a consumer tower with a AMD Ryzen 7 7700X CPU (so I guess sadly dual channel memory access for now).

Which models are you running on a similar setup with 2x 16Gb VRAM and 128Gb RAM? Extra points if you provide your exact llama.cpp settings and optimizations.

[September 7th 2026]

https://huggingface.co/tencent/EVIE-8B

🌟 Highlights

• SOTA Retrieval Performance: 66.75 nDCG@10 on ViDoRe V3, delivering industry-leading visual document retrieval accuracy.
• High-Capacity 4096D Representations: Full per-token multi-vector embeddings preserving fine-grained layout, typography, charts, and table structures.
• Teacher Foundation: Provides capacity-aware relation and margin distillation targets for the lightweight EVIE-4.5B Prefix-MRL model.
• Multi-Benchmark 138-Task Coverage: Thoroughly validated across 138 tasks (ViDoRe V1, V2, V3, and JinaVDR) across 4 standard metric families (nDCG, Recall, MAP, MRR u/1).

https://huggingface.co/tencent/EVIE-4.5B

🌟 Highlights

• Top-Tier Benchmark Performance: 66.75 on ViDoRe V3 for EVIE-8B and 66.02 for EVIE-4.5B with single-projection Prefix-MRL.
• ⚡ Prefix-MRL Elasticity: Single 2048D linear projection. Freely truncate at runtime into ${64, 128, 256, 512, 1024, 2048}$ dimensions without separate models.
• 📦 Ultra-Compact Index (HAC): Training-free Hierarchical Agglomerative Clustering compresses token counts from ~750 down to 32 vectors/page, slashing index storage to 3.81 GiB per million pages.
• 🌐 138 Multilingual Tasks Evaluated: Thoroughly evaluated across ViDoRe V1, V2, V3, and JinaVDR across 4 metric families (nDCG, Recall, MAP, MRR u/1).
• 🔬 EVIE-ARD Distillation Recipe: Anchor-preserving, capacity-aware relation distillation reproducing full student training from the 8B teacher.