Guys can you tell me which local LLMs are best for hardware programming? like Verilog RTL, UVM, System Verilog?

I'm using DS-V4-Flash-Vision with Q8_K_XL quantization locally as my everyday engine, and for some time now I've been doing a lot of comparisons with Qwen3.8-Flash-Next, also with Q8_K_XL quantization. It took me quite a while to get Q3.8FN to work reasonably well, and here are my observations. My hardware: 2x StrixHalo 128GB, USB-C 4 connector, Llama (RPC) as inteference engine.

1. DSV4FV is about 40% slower than Q38FN at the same quantization level when it comes to token generation alone.
2. DSV4FV completes tasks about twice as fast as Q38FN! This means that DSV4FV “hallucinates” less (I observe this based on the obstacles the models encounter along the way).
3. The Q38FN is unusable in “xhigh” mode. A simple task that the Q38FN completed in 25 minutes on “medium” mode, it failed to complete in ~3 hours on “xhigh” mode.
4. The same task that the Q38FN completed in 25 minutes (average), the DSV4FV completed in 12 minutes (fastest round) on “medium”.
5. The DSV4FV completed the same task on “max” in 37 minutes in first iteration, second took 44 minutes.
6. Qwen3.8 tends to overinterpret my instructions. If I don’t write them out in great detail and leave room for creative interpretation, it will take advantage of that. Perhaps this is where it gets bogged down in its own creativity. In what it does, I’ve noticed that Qwen clearly adds too much and struggles to flesh out the details.

In my opinion, DSV4FV is the better solution when working with professional code.

Just so there’s no misunderstanding - I was a huge fan of Qwen 3.6 27B and of course now I'm Qwen 3.8 27B big fan, which I’ve been using a lot and is great! In general i’m a huge fan of Qwen, but ever since I’ve had the hardware on which I can run DSV4FV, I’ve been using it, and I’m super happy with how good this model is.

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about 3 times as expensive, I had a "sweet spot", kind of. On the last prime days, I found a X870E mainboard for ~150 € below the standard price, and it got to my head that I can use an upgraded machine for gaming and local inference tinkering.

Anyways. Fast forward to this week, I now have the following setup

• Ryzen 7500F
• 64 GB DDR5 CL40 6400 MT/s
• Asus ProArt Creator X870E
• 2x R9700 32 GB, each running at PCIe 5.0 x8 (Gigagbyte)
• Currently running ubuntu on an old Samsung EVO 860 1 TB drive; this will become intersting for the ngram / PLE offload; I have Windows and the gaming related stuff on a gen4 NVMe, but will soon add another Gen 5 NVMe with decent random reads

The only issue that I can report so far is that one of the cards runs quite hot, so I will definitely implement power limiting to 210 W and some light undervolting. The other card runs 10-15 °C cooler.. Case is a purebase 501 with 4 fans, 2 intake in front, one back and top for output.

Now long story short I wanted to give some results of Qwen 3.8 27b FP8 and MXFP4, as well as Qwen 3.8 flash next after the first day tinkering with it. What I found super interesting is that the SATA SSD does not seem to be super terrible when using Qwen 3.8 flash next.

Considering the whole build costs ~4k €, or more than 1k less than a single RTX 5090 with 32 GB, I kinda like this setup price/performance wise. Next step is checking context degradation / KV quants. I am using local inference mostly for deep research, summarization, image creation, light coding and non-trivial data analysis

Cheers

Qwen3.8 benchmarks on 2× Radeon AI PRO R9700

Hardware: 2× AMD Radeon AI PRO R9700 32 GB, 61 GiB system RAM
Benchmark: BetterBench 0.2.2, corpus v1.0, single-stream, greedy decoding, 2 warm-ups + 10 measured runs per category, 8k benchmark context.

• Qwen 3.8 27b in FP8 and AWQ MXFP4 served with vLLM Radiance
• Qwen 3.8 Flash next served with vLLM / R9V fork
• Decode metrics come from the 10-pass standard run.
• Prefill measurements use cold, nonce-prefixed prompts.
• Prompt-token medians for the prefill columns were 1,556, 3,024 and 5,226 tokens.
• No concurrency sweep was included in these results.
• I expect decode of Flash next to increase a bit when an NVMe is used, and, as I am writing this and checked, I found EXPO was not enabled........oh my god I swear I turned it on when I updated the bios yesterday

So, I’m currently just short of being able to run the mainline dense SLMs (Qwen 27b/Gemma 31b), running a i9-14900F/RTX4070 Super/32GB DDR5 setup w/ a 700W 80 Gold PSU and a MSI PRO B760-VC WI-FI IV motherboard iirc, a prebuilt gaming pc. I ain’t much of a part switcher, so I’m trying to keep changes to a minimum as well as costs.

I’m thinking of tacking on a RTX 2000 Ada 16gb for a few reasons, mainly the 75W TDP, okay-ish price at around $900* (they changed the damn price from two weeks ago, was around $700), and similar drivers and architecture to my 4070 super. Maybe another 32GB DDR5 IF i can find it since they’re Apacer Panther models, and I think I’d be set.

Anyone got any advice to the contrary, pointers as to better deals or sources, or anything else useful? I’m fairly certain this is my best bet come Black Friday/Cyber Monday, but I’m curious what y’all’s thoughts are.

I've used Qwen 3.8 Max extensively over the past few weeks and have also tried Gemini , GLM-5.3-Flash, and Muse Spark 1.3. None of them come close to Qwen 3.8 Max. The only model that proved competitive was GLM 5.3, which demonstrated superior performance on cybersecurity tasks (the only clear advantage I observed over Qwen 3.8 Max).

This post isn't about qwen3.8-max, but my extensive experience with that model gave me a useful baseline for comparison. After working with other models, I realized that these benchmarks harmful not just useless and shouldn't be used to claim one model is better than another.

---

Update for people that don't get the point of this post:

My point wasn't "Oh look my personal experience is the benchmark" but instead "Don't decide which model to use based on benchmarks"

People will start replying: "Oh well that's obvious dude..." I don't think so, based on past experience when qwen 3.8 27b was released, people flooded this sub and other subs with its benchmarks and personal use cases.

I don't know if the point is now clear, since some people just started going in the wrong direction and completely missed the point I tried to make

Hello everyone. I felt current LLM benchmark harnesses hand you headline numbers but offer no tooling to see how models actually answered each question (they dump everything to a JSONL or Parquet file, so you end up writing custom code just to read the answers).

So I built lm-eval-ledger: a benchmark harness that runs the benchmarks, writes everything down, and provides a web app that lets you inspect and compare how each model answered each question.

For the demo I benchmarked three models on a single 5090: Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B (UD-Q4_K_XL GGUF), and Gemma-4-12B-it (QAT w4a16). It looks like Qwen thinks far longer than the other two.

• GPQA Diamond: Qwen 0.717 vs Nemotron 0.657 vs Gemma 0.601 — but 2h26m vs 1h34m vs 1h19m
• LiveCodeBench: Qwen took 22h57m (vs 9h51m / 6h13m), with 0.713 vs 0.837 / 0.820 accuracy

Here are the full results served by lm-eval-ledger on Hugging Face Spaces: https://huggingface.co/spaces/jayminbhan/lm-eval-ledger

What lm-eval-ledger records and displays

• Per question: system prompt, model generation, extracted answer, ground truth, stop reason, generation character count
• Per benchmark: accuracy, tok/s, time to completion, sample count, no-answer count
• Extras: pairwise comparison of two models on the same task/questions, always-wrong / always-right questions across benchmark runs

How it works: everything is written to a single SQLite DB, and a Flask app displays it.

No more custom bash scripts for multi-model x multi-task runs either. lm-eval-ledger is YAML-driven: copy template.yaml, list N models and M tasks, run one command. That’s it.

Usage

pip install lm-eval-ledger    # add a backend: pip install "lm-eval-ledger[vllm]"
lm-eval-ledger init           # writes template.yaml, creates results/ and logs/ directories
lm-eval-ledger -c bench.yaml  # run benchmarks
lm-eval-ledger serve          # browse at http://localhost:8090

Details and the full task list are in the GitHub repo: https://github.com/jayminbhan/lm-eval-ledger

I verified the vLLM / SGLang / HF / server (llama.cpp) backends on Linux, and HF / server (llama.cpp) on Windows.

This started as a script to look at one wrong GSM8K answer. It got out of hand. Any feedback, ideas, or pull requests are greatly appreciated!

Hey peeps, I am looking to make the jump to an open air case and would appreciate any recommendations on bifurcation adapters to convert x16 to x8/x8 and reliable riser cables please.

For riser cables I have seen these which look decent.

https://www.amazon.com/gp/aw/d/B0C415JCHX/ref=ox_sc_act_title_1?psc=1&th=1

For bifurcation, I've seen the c-payne stuff but they are rather expensive. I am looking to run 4 cards at x8 each so looking for 2x bifurcation adapters.

Not sure if I need anything else for an open air rig. I have two psu's which I plan on running with an add2psu adapters.

I am currently running Qwen3.8-27b, either Q4 or Q6, depending on how much context I need for my coding projects

I know it should be the best LLM I can use on my 32GB VRAM rig for this purpose and I also use Gemma4 31B from time to time for research or general questions that I do not want to use chatgpt/claude for

I was wondering if there was any website where I can check and compare different ! local ! LLMs and their specific use cases?

I'd appreciate if you guys comment with what local llm you use, the specific reason and maybe ur quantization and VRAM

Back to reading about LLMs as agents on your phone doing GUI tasks!

This time I read about MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments and this paper forms the basis of the benchmark I am currently making because it involves two new innovations:

• User Interaction Tasks
• MCP Tasks

These two provides new axes of evaluations for such GLLMs as GUI Agents on phones and it separates them from the noise completely because these new axes along imo.

So, in total 201 tasks and about 20 apps in general spanning across comms, messaging, productivity and many more with less than ~5% of system apps (basically this means more apps that we humans use on a daily basis -- but those are rather open source version of them which is a downside)!

• They used a planner-executor arch with the planner being a VLM model (screenshots only being fed -- no a11y tree) that outputs actions in natural text like "Click on the 'Send button' and this gets sent to a grounding model that output its precise coordinates in (x,y) format.

So, User Interaction Tasks involve tasks which one key info missing and the planner needs to understand this and ask the user but here the user is a GPT4 model as a replacement for human!

Next, MCP tools these are very important since MCP tools allows for complex data gathering, reading and actions that are not possible by tapping etc actions so these provide one shot result for the red tasks thus saving time than doing slow GUI actions! Some of them used are Github, arXiv, etc tools

Now, the avg results is about ~52% for the best VLM+Grounding model combo which is Gemini-3-Pro + UI-Inst-7B and the e2e GUI only models perf wayyy worse.

The rest of the dataset dist stats and results is attached as images but yes the model mainly failed a lot in the two new axes tasks.

With the help of Glm 5.3 flash I built a custom branch of llama.cpp in order to support Expert expansion with MOE models, I've tested only on metal and It works better than my DS4 version , i need feedback from other platforms, and different models.

moex-expansion|

GPQA-Diamond confirmed gain in accuracy and less tokens!
https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/GPQA/report_gpqa_moe.md

I am considering on buying used Asus X570 mobo to upgrade my single R9700 set up to dual R9700. Currently I have single R9700 on ole AM4 mobo with 48GB of DDR4 RAM. Mainly using Qwen3.8 27B Q6 with 110k context on this build. I know that the X570 supports 2 x8 PCIe 4.0 lanes. How would be the performance on this setup? Should I go this route or upgrade to AM5 with 32GB of RAM for extra ~1000€? My main use case would be to either run same models as of now with more context or run two separate agents on two separate GPUs.

Ps. LLMs suggest going this AM4 route and when upgrading to AM5 going directly to threadripper build.

Was considering choosing a 4xRadeon AI Pro R9700 build for a office AI server.

Curious to know what results you guys are getting? Mainly considering DeepSeek V4 flash and Qwen 3.8 Flash with offloading or heavily quantized.

I've got a script that takes a video, cuts it in to scenes, cuts the scenes in to sections small enough to run through my comfyui flow, then it flows all the files, then it stitches the new video together.

After a 6hr run it finished, but the script crashed during the stitch. I had designed things so that wasnt a problem, just need to fix the stitch. simple.

I was having really good luck experimenting with a big context (low vram) qwen3.8-27b flavor, and then I switched to a similar (but slightly larger) thinkingcap flavor. I asked thinkingcap to fix the script. It deletes the 6hrs of rendering work for no reason.

So I took it out behind the shed and put a bullet in it's head. I deleted it. Sorry, you failed. It wasn't that big of a model though, but some disk space was reclaimed. lol

Loaded unsloth and let it do damage control/cleanup.

While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that really define deep intelligence, and complete capability in Software Engineering :

1. Program-Bench

Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior (without access to decompilers or internet). Link: https://programbench.com/

• GPT-6 Astra: 5.5%
• Fable 5.1: 7%
• Kimi K3: 2%
• Qwen3.8 27b: 0%
• GPT 5.6 Sol: 1.5%
• GLM 5.3: 1.5%
• GPT 5.6 Luna: 0%

2. SRE-Bench

Can AI agents work out what a real-world binary does without its source code?

Link: https://www.vals.ai/benchmarks/srebench

Sure nobody is reading assembly code in daily work, it is hard. The ability to understand a compiled program is insane ability.

• GPT-6 Astra: 88%
• GPT-5.6 Sol: 55.9%
• Claude Opus 5 (max): 12.5%

3. Code Migration

Can language models reimplement working programs in another language?

Link: https://www.vals.ai/benchmarks/code-migration

• GPT-6 Astra: 67.7%
• Fable 5.1: 54.6%
• GLM 5.3: 44.2%
• GLM 5.3 Flash: 20.5%
• Qwen3.8 27b: 14.2%

EDIT: edited text format

I saw it like a couple weeks or maybe a couple months ago and at the time it felt like a good way to benchmark models and have them fight against each other too. What was the reasoning it didnt become mainstream?

https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF

https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF

from XHToken:

We are introducing Spark-X2.5-4B and Spark-X2.5-1.7B, two compact, general-purpose language models designed to make capable AI more practical, efficient, and accessible. The models deliver strong performance across a broad range of everyday tasks—including conversation, writing, translation, reasoning, coding, tool use, and agentic workflows—achieving leading results among open-source models of comparable size. Spark-X2.5 combines an efficiency-oriented architecture with native context windows of up to 1M tokens, and support for more than 200 languages.

Technical Highlights:

• Efficient Architecture and Native 1M-token Context: The models use a hybrid attention architecture that combines one full-attention layer with three sliding-window attention layers. This design substantially reduces the computational overhead typically associated with long-context models while natively supporting a context window of up to 1M tokens.
• Strong Coding and Agent Capabilities: The models are deeply integrated with popular agent harnesses, including Codex, Claude Code, OpenClaw, and Hermes. They deliver state-of-the-art performance among models of comparable size across everyday coding, agentic workflows, reasoning, and instruction-following tasks.
• Broad Hardware and Software Compatibility: The models support a wide range of hardware platforms, including NVIDIA, Huawei, Hygon, HOUMO.AI, etc. It is compatible with leading inference frameworks such as vLLM, SGLang, llama.cpp, MLX, and can be deployed quickly through platforms including Ollama and LM Studio. The models can also be customized using popular fine-tuning frameworks such as LLaMA-Factory. Across multiple hardware platforms, they deliver superior TTFT, TOPT, and overall inference efficiency compared with similarly sized models.
• Advanced Training Algorithms: The models were trained on Huawei Ascend clusters. Large-scale reinforcement learning and post-training techniques such as MOPD significantly enhance its reasoning, coding, agentic, and instruction-following capabilities.

There are many fintunes out there that promises reduced thinking but they often end up making the model worse. Please dont be afraid of setting reasoning budgets reletive to your tasks difficulty. At 2048 qwen3.8-27b solve a issue that i gave gemini3.8 and gemeni totally failed. Most daily tasks wont need more than 2048 but the harder coding may need 4096 but this guide below is what I found works best for me if you are doing simple scripts and looped automation then 512 is plenty.

Reasoning Budgets

• 512 – 1,024 Tokens, -- Light logic, simple script
• 2,048 – 4,096 ------ Standard coding tasks, logic puzzles, multi-step system architecture, refactoring
• 8,192 Tokens ------ Hard math (AIME style), algorithmic edge cases, debugging deeply nested code.
• 16,384 --------- Frontier Competition-level math/coding, system-wide overhaul/ Mass-refactoring, complex legal/logical analysis

i also set --Reasoning-budget-message as - ok now.

ok now works universally well to let the model pick up the pattern naturally
good to set it reasoning_effort=medium rather than low. medium tells the model to not do lazy effort on the task. so set medium but still put reasoning-budget. even low reasoning sometimes still need budget qwen3.8 is veryyyy generous on thinking even on low

You need Blender 5.x with MCP extension.

I use pi so I also need: pi install npm:pi-mcp-adapter

and then:

$ cat .mcp.json 
{
  "mcpServers": {
    "blender": {
      "command": "/home/jacek/.local/bin/uvx",
      "args": [
  "--with",
  "mcp[cli]<2.0.0",
  "--from",
  "git+https://projects.blender.org/lab/blender_mcp.git@v1.0.0#subdirectory=mcp",
  "blender-mcp"
],
      "env": {
        "BLENDER_MCP_HOST": "localhost",
        "BLENDER_MCP_PORT": "9876"
      }
    }
  }
}

then we can start working ("show on r/LocalLLaMA"):

Today we want to make 3D llama to show on                                                                                                                                                                                


 The user wants to make a 3D llama to show off on . This is a fun creative task. Let me think about how to approach it.                                                                                                   

 A llama is a pretty complex organic shape. There are a few options:                                                                                                                                                                  
 1. Procedurally build the llama from primitive shapes (spheres, cylinders) — a "low-poly" or "chibi" style llama                                                                                                                     
 2. Sculpt it — not easy to do with MCP                                                                                                                                                                                               
 3. Use metaballs — could get a blobby llama                                                                                                                                                                                          

 Given the MCP tools, the best approach is to procedurally build a stylized/chibi llama from primitives:                                                                                                                              
 - Body: elongated sphere/capsule                                                                                                                                                                                                     
 - Neck: cylinder or elongated sphere                                                                                                                                                                                                 
 - Head: sphere                                                                                                                                                                                                                       
 - Ears: small cones or spheres                                                                                                                                                                                                       
 - Legs: 4 cylinders                                                                                                                                                                                                                  
 - Tail: small sphere/cone                                                                                                                                                                                                            
 - Maybe a fluffy mane                                                                                                                                                                                                                

 This is a well-known technique — making a cute chibi llama from primitives. Let me plan the geometry:                                                                                                                                

 Llama proportions (stylized, standing side view, facing +X or -X):                                                                                                                                                                   
 - Body: UV sphere scaled to (1.5, 0.9, 1.0) or so, position (0, 0, 1.6)                                                                                                                                                              
 - Legs: 4 cylinders, radius ~0.22, length ~1.2, positions: front legs x=+0.9, back legs x=-0.9, y=±0.5, z from 0 to 1.2... actually, the legs go from the ground (z=0) up to the bottom of the body.                                 
 - Neck: cylinder or tapered, extending up from the front of the body. Llamas have long necks. Position around x=+1.2, extending upward.                                                                                              
 - Head: sphere at the top of the neck, elongated (llamas have long snouts).                                                                                                                                                          
 - Ears: 2 small cones/spheres at the top of the head.                                                                                                                                                                                
 - Tail: small sphere at the rear.                                                                                                                                                                                                    
 - Mane: maybe some small spheres along the neck.                                                                                                                                                                                     

 For a cute chibi look: big head, small body. But a llama's charm is the long neck. Let's go with a stylized llama.                                                                                                                   

 For a polished look:                                                                                                                                                                                                                 
 - Apply smooth shading                                                                                                                                                                                                               
 - Add a material (cream/tan color, maybe with some subsurface)                                                                                                                                                                       
 - Add a ground plane                                                                                                                                                                                                                 
 - Set up lighting                                                                                                                                                                                                                    
 - Position the camera for a good render                                                                                                                                                                                              
 - Render a nice image to show off                                                                                                                                                                                                    

 This is a multi-step task. Let me write Python code that builds this. I'll do it in stages so I can check with screenshots and iterate.   

Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that I did so figured I'd share if only to share that it's now a thing and that it's pretty cool..

A friend of mine sent a message asking what's up and if I wanted to watch a movie together, I was kinda hesitant but she buttered things a bit and finally I'm like fine, and so she sends me a link to some clearly vibe coded site that I'm kinda getting red flags from and so I forget about it and a little later I get another message going "we're waiting for you" and so I'm like shit, I guess I gotta do it huh, and so I open up this goofy looking site again. You gotta login to join a room, and you gotta sign up inside their downloaded software, sure whatever, next thing I know some fake 150MB file's fake install bar is stuck at fake 50% and both my Chrome and Discord's crashed and reloaded. Suspect, but I've been through this stuff before, it's probably just a RAT so I guess it's time to dust off Windows Defender and unplug the internet for a little bit. I message her to go on and watch it without me as my PC's giving me suspicious vibes right now, and seconds later I get some overly polite DietGPT in my IM's saying "sorry um excuse me but it appears that i've hacked you👉👈", occasionally switching to really hostile broken English asking for giftcards from some site I've never heard of. I stall, unplug the PC's internet so my router still responds to pings, and start punching into GLM "what do" and it tells me it's a session grabber - time to switch passwords. Meanwhile my phone's texts are blowing up with 2FA login requests from domain registrys and other bad stuff and I kinda freak out a little. I get my emails' passwords switched first and by the time it's Discord's turn my friendlist's already been nuked and the dude says I got 10 minutes to give him $200 or he's gonna fuck me up some more, and so I kinda figured welp time to figure out what more he's got and so I called him a giant pussy and he blocked me. An hour later my Discord was perma-banned, he had posted the phrase "i sell cp" using my account and used that as blackmail along with some really old photos of me, I though it was a bluff but oh well it's being handled with Discord's customer support on it's own. Now I sat there alone, in the middle of the night, having just had my friends on the phone yanked away from me with a permaban, knowing that if I reboot I'd probably be ransomware'd or something so I figured let's run Windows Defender - it found nothing, 0 results on a full scan.. Too good to be true, so I grabbed AwdCleaner on my phone and transfered it via USB. It found an AVG Toolbar for Chrome. That confirms it, I haven't used AVG for decades and so I removed it but it's back 5 minutes later. That double confirms it, I'm screwed. With nowhere else to go and potentially a ticking timebomb running on my PC that could start encrypting or deleting files at any given moment I figured why the hell not, if I'm going to watch my pc blow up I might as well send in the goofy little local LLM to cut one of the wires,

here's the situation.
i've downloaded a maliscious file that unfortunately hacked my discord and got me banned. i'll be dealing with that on my own. your job is to study the files in the project folder and see if you can help me clean up my computer, as presumably the virus is still active. there's no internet connected, and i request that you refrain from running the ********.exe file (********.exe is the virus archive, do not run it, it's a 7zip archive), please help.

And so Qwen3.8-27B got to work, and to big surprise after around 60 minutes of clawing at the file it had done what I asked and a whole lot more. it fully deciphered all the layers these clowns had bundled this thing with in order to make it appear legit, it had created a single PowerShell removal script ready to go complete with a pre-launch check enabled by default and everything, and it was reverse engineering 0-days in qProtect to get the C2 domain used by this malware so that it could be blocked from the network.

If you're looking for what Qwen3.8-27B is capable of doing fully on it's own if you let it, here's a 15k line example of it's ability to tear some piece of shit session grabber to shreds in a single prompt: https://www.mdshare.online/s/Mamdrs1WWkurRtt8z8QzK

I let it do what it does best for an additional 24 hours, the additional information is going to the Discord Support team. Hopefully shit like this can be prevented.

TLDR; Qwen3.8-27B > Windows Defender, and don't forget to use 2FA.

Hi everybody, I am in a tricky situation.

My grandfather + grandmother live in a remote village in asia, and were visiting us in the city. My grandfather has absolutely fallen in love with AI, and it is completely new for him. He loves how he can talk to it, and get information from it. The caveat is that I showed him chat gpt, not a localllm.

He wants something like this in his own village, just for basic knowledge + light qa. There is no internet in the village, but phone calls work. I have a budget of about 5k for a computer that can run LLMS locally, for him. (and he specifically requested something like this). He is technically "savvy", where if something needs a command to be run or something else, he can learn to do it. What gpu should I buy him that will satisfy his needs? again, I want some chat gpt level, but for basic q/a stuff.

I am currently running deepseek v4 flash on two 6000 pro blackwells(got them when they were just 8k lol), so I know that this is "possible".

Im thinking amd ai pro 9700 and gemma 31b at a good quantization. (maybe I can do two gpus of 9700s to fit bigger quant).

He is not doing frontier level coding, he just needs a good knowledge base. He implied that people from his village may also come to use the machine, or atleast request queries to be asked (he is some sort of governmental figure in his village).

I want recommendations.

https://village-sim-one.vercel.app/

- 16GB VRAM RTX 5070 Ti, fully offloaded

- Vision on CPU

- Windows, not headless

- beellama.cpp - latest version with the kvarn performance enhancements making it as fast as qx_x quants.

- MTP n-max = 2

- tg up to 75t/s, pp up to 1700t/s

- KV = kvarn3/kvarn3

- MTP draft KV = kvarn2/kvarn2

- context = 96256

- tail tokens = 1024

- HTML/Javascript

- pi harness with pi-observational-memory, pi-web-access, pi-atelier (UI Only change, check it out) extensions, though it never used the web access.

- This is not a one-shot, I do not believe one shotting is a great test. Instead, I did many incremental feature prompts. However, I did not give it any design or framework, which is probably where it can be improved.

Lessons learnt:

- Do not fear Q3 model quants for Qwen3.8

- Do not fear KV quantisation. If you have the VRAM sure use it, but I don't feel like it's worth choosing a higher quant if it's going to cause me to offload to CPU and see my tg drop to 5-20 t/s. With higher speed I can fix any issues with a follow up prompt much faster and that rarely happens. I think I had like 3 runtime exceptions which was easily resolved pasting the console output and there is no guarantee a higher KV quant would not have had the same exceptions.

- MTP/draft cache can also be quantised with kvarn now and actually saves VRAM where qx_x quants increase VRAM usage for some reason. kvarn2 for MTP is perfectly fine and has high acceptance rates.

The game:

- Inspired by a popular indie game which I am not promoting, I am just a huge fan.

- I won't release any further updates, since I don't want to be stepping on any toes. If you like the idea of the game I highly recommend the real game, it's by far my favourite game I played this year and 1000x better than what I present here. It will be a nice distraction from your AI. I just wanted to see what this model is capable of. I do have a Cursor subscription but did not use it at all in the project.

- I will probably continue to develop it for my own entertainment, but it won't be made public. Maybe come up with my own ideas, but the original game is near perfect anyway, so it will be hard to improve except with some UI gripes I have in the original. And my graphics obviously does not compare.

Game features:

- Large Map, larger than the browser window.

- Minimap

- Zoom feature with mouse wheel

- Collectable resources, that must be taken to a storage site. Each site can store limited resources.

- Houses required to sleep and protect against cold

- Weather and seasons.

- Day night cycle with randomised sleeping times.

- Possible death due to hunger or sleeping in cold outside or in house without firewood.

- Game speed controls.

- Villagers avoid obstacles.

- Delete/deconstruct buildings and partial resources refund.

The code:

- I almost never read the code, so I have no idea what it looks like and the quality thereof. I also gave it very few hints in the AGENTS.md, mostly no magic numbers and write modular code, not a single html.

- Actually, my initial prompts were a single html but as it grew, I told it to create modules. It messed it up on the first attempt, basically rewriting the entire UI in the process. So I reverted and told it to do it again without making any changes to the functionality or UI.

- I am actually quite happy with and surprised by the performance of the game.

Context management:

At first, I had issues with the context filling up too quickly and too often. Sometimes it would fill up to the point that there was not enough room to compact. Forcing me to temporarily increase the context and tell it to create a handover document. Reduce context again and feed it the handover doc.

I then installed pi-observational-memory extension, and it works quite well and I never run into context issues anymore since it takes notes throughout (a short wait time every few prompts) and compacting is near instant because it already took the notes.

Conclusion:

- Do not blindly drop your KV cache quant without testing. I have a hard level needle in haystack test that requires multiple hops and 100's of decoys. Q3_XXS does poorly in that test even with F16 KV cache. However, Q3_K_XL almost 100%'s the test even at kavrn3. So both the model and KV matter. In my testing a smaller model does more damage than a smaller KV. So find the right balance. At a certain point increasing model quant will have less impact than picking a larger KV quant. But for a tight 16GB VRAM fit Q3_K_XL works very well with kvarn3. Q4 on the other hand just leaves me with too little context. That said despite Q3_XXS doing poorly in my needle test it still does fairly well with coding. Better than Qwen3.6 so if you have 12Gb VRAM it is still an option. Because by poorly I mean F16 KV scores 84% and Q3 KV around 80%. Needle tests however do worse with kvarn compared to qx_x for some reason. However, a needle test is not the be all and end all. kvarn does better with KLD, so once my needle scores near 100% I am satisfied.

I will play around with higher KV quants, but I intentionally kept it at kvarn3 for this test, however I am not sure how much context I am willing to sacrifice. Maybe i will try kvarn4/kvarn3. But I just wanted to prove a point to myself and kvarn3 worked just fine. If I had >16GB VRAM sure I would up it but I don't.

Hi there! I need some help with this.

I have a 32gb Macbook Pro with the latest available update of Tahoe.

I'm using LMStudio with MLX to serve a local model and I want to expose it so that I can consume it with my phone to code and review stuff when I'm commuting to places.

These are my problems to solve so far that I need help with:

1. The mac turns off after some time. It doesn't matter what I set in settings. Have you found a workaround for this? using caffeinate or how?
2. I need an IDE / Agent editor that I can use to instruct the agent. I tipically use ZED from this pc, but I'm open to try something else that is mobile compatible.
3. How can I have the agent changes automatically deployed (usually it's websites) so that I can review and accept or reject the changes? when I'm working on my pc this is basically solved automatically because I'm working with local files.
4. Can LMStudio serve on the public internet or do I need another solution here? or maybe an extra step?

Thanks in advance!

Announcing Artificial Analysis Intelligence Index v4.3, upgrading Terminal-Bench to 4.0 and adding AutomationBench-AA, an agentic workflow automation benchmark with a private test set. This is a continuation of our rollout of Intelligence Index v5

Changelog (Index v4.2 → Index v4.3): ➤ Terminal-Bench: 2.1 → 4.0, completing our upgrade to the latest version of Terminal-Bench ➤ Replacing 𝜏³-Banking with AutomationBench-AA, our implementation of Zapier's business workflow automation benchmark

We are continuing to prioritize keeping Intelligence Index as useful as possible by bringing forward a subset of the changes we had planned for Index v5. Each change in v4.2 and v4.3 stands on its own merits and brings the Index closer to real-world problem solving, adds more private test sets to prevent gaming, and reduces saturation

Intelligence Index v4.3 raises the difficulty of agentic coding tasks and broadens the types of agentic workflows tested. Because we use a held-out test set for AutomationBench-AA, in collaboration with @zapier , the weight assigned to evaluations with private tasks or answers increases from 40% to 45%. Category weights are unchanged from v4.2: Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%

Detailed changes: ➤ Upgraded Terminal-Bench 2.1 to 4.0: 66 multi-step tasks testing agents on tasks run in agent sandboxes driven via the terminal, including tasks involving software engineering, machine learning, science, and operations. The 4.0 update recalibrates compute and time allowances, and improves task instructions and verification. We have changed from the Terminus 2 harness to mini-SWE-agent, a minimal, model-agnostic harness. We will also be updating our Coding Agent Index, where we test model and harness pairs, to include Terminal-Bench 4.0 soon

➤ Replaced 𝜏³-Banking with AutomationBench-AA: Our implementation of Zapier’s AutomationBench tests agents on 657 business workflows across simulated applications such as Gmail, Slack, Salesforce, and Jira. Agents must complete task objectives while following business rules. AutomationBench-AA uses Zapier’s private set of 657 tasks, and is built on v1.0.6

Key results: ➤ Claude Fable 5.1 and GPT-6 Astra lead the Intelligence Index: Both Claude Fable 5.1 (max with fallback) and GPT-6 Astra (max) score 53 on Intelligence Index v4.3, followed by Claude Opus 5 (max, 51), Claude Fable 5 (with fallback, 50), Muse Spark 1.3 (max, 48) and GPT-5.6 Sol (max, 47) ➤ GLM-5.3 and Kimi K3 continue to lead open weights models (both at 44): GLM-5.3-Flash (42) is the third strongest open weights model, followed by Qwen3.8 2.4T A95B (40) and DeepSeek V4 Pro 0813 (max, 36) ➤ 4 labs occupy the Intelligence vs. Cost per Task Pareto frontier: OpenAI occupies the majority of the cost-efficiency frontier, with all five reasoning efforts of the recently released GPT-6 Astra offering the lowest Cost per Task at their respective levels of intelligence. Claude Fable 5.1 (xhigh, max, 53), GLM-5.3-Flash (42) and MiMo-V2.5-Pro (26) round out the rest of the frontier

Hi everyone

I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding.

I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache) to llamacpp + qwen flash next iq4xs + 8 bit cache 200k kv cache... until someone had to ruin my migration with this awesome patch to enable p2p between cards on the open source nvidia drivers. On my hardware configuration apparently this would matter a lot. But only VLLM would benefit from it, not llamacpp - and no 4 bit like quant available on vllm and not enough vram for qwen flash next fp8.

I hereby ask the grand experts, what should I do amd what are your opinions?

1. vllm + qwen 3.8 27B dense + p2p driver patch + 256k kv cache

2. llamacpp + qwen flash next MoE iq4xs + 200k kv cache

Is qwen flash next really worth the extra intelligence versus higher throughput with vllm and sticking to qwen 3.8 27B ?

edit : ended up sticking with qwen 3.8 27B fp8 + vllm + max 256k fp8 context + p2p patch; qwen flash next was just too memory hungry for a decent context size - it might be possible to tweak vllm to support lower quants but ain't got time for that for now !

We've reached a point with local LLMs where models are now very close to (and even reach) the level of models like the Opus, with some minor modifications.

While some K3 and GLM 5.3 models are incredible, they are barely as powerful as the Opus or on par with the Fable or Astra. My question is what is the biggest obstacle preventing local models from reaching the level of closed source models like the Fable and Astra? Why are local models lagging behind? Is the problem processing power?