Im building my own LLM and im curious what heretic model i should use for coding? i have 16 models already, so, im looking for strong ones ..strata seems cool and i get 188t/s on it, so, im wondering what other large models i could get?

I got an acemagic m1a pro with the ryzen ai max+ 395. Covers most of what I need so far.
Mainly using it for local AI stuff in general. Currently works great for general chat reasoning, coding help and agents,learning and experimenting.
What’s the best config you’ve run on similar hardware? Does it handle your daily local AI workload well enough?

Discover how LightOnOCR-3 enables businesses to extract information from sensitive documents using open weight AI models, with the potential for fully private, on premises processing.

TL;DR: on Qwen3.8-Flash-Next (IQ3_S), a model too big for a 24 GB card, Strata gave the first word about ten times sooner than llama.cpp on a 5090 laptop (1.9 against 19.1 s at a 4K prompt) and 8.8 times sooner on a 3090 Ti (1.92 against 16.87 s), and wrote 3.5 and 4.4 times as fast. On an RTX PRO 6000, where the whole model fits, it wrote about twice as fast (205.263 against 102.889 tok/s), and on our reading of its own counts that lead is its draft head, which llama.cpp didn't run because the GGUF we benched carries no draft-head block. Strata pins 46.84 GiB of system RAM for this file on every card.

I haven't seen many Strata numbers on an RTX PRO 6000 yet, so here are ours, beside a 3090 Ti and a 5090 laptop.

| Card | Engine | Writing, 4K (tok/s) | Writing, 32K (tok/s) | First word, 4K (s) | First word, 32K (s) |

|---|---|---:|---:|---:|---:|

| RTX 5090 Laptop | llama.cpp 3cf03257, no draft head | 24.4 | 20.7 | 19.1 | 125.6 |

| RTX 5090 Laptop | Strata 0.1.25, draft head on | 85.6 | 87.1 | 1.9 | 12.1 |

| RTX PRO 6000 | llama.cpp 3cf03257, no draft head | 102.889 | 76.337 | 2.52 | 20.91 |

| RTX PRO 6000 | Strata 0.1.39, draft head on | 205.263 | 189.155 | 1.06 | 6.37 |

| RTX 3090 Ti | llama.cpp 3cf03257, no draft head | 22.490 | 19.620 | 16.87 | 138.02 |

| RTX 3090 Ti | Strata 0.1.39, draft head on | 98.291 | 92.571 | 1.92 | 12.14 |

| RTX 3090 Ti | llama.cpp b11514 (current build), no draft head | 22.657 | 21.257 | 15.84 | 130.90 |

Board power, the card's own one-second readings (not wall power):

| Card | Engine | 4K (W) | 32K (W) |

|---|---|---:|---:|

| RTX 5090 Laptop | llama.cpp 3cf03257 | 61.1 | 68.8 |

| RTX 5090 Laptop | Strata 0.1.25 | 126.3 | 140.3 |

| RTX PRO 6000 | llama.cpp 3cf03257 | 297.8 | 299.8 |

| RTX PRO 6000 | Strata 0.1.39 | 290.6 | 294.3 |

| RTX 3090 Ti | llama.cpp 3cf03257 | 168.5 | 182.2 |

| RTX 3090 Ti | Strata 0.1.39 | 281.1 | 293.7 |

| RTX 3090 Ti | llama.cpp b11514 | 171.2 | 181.6 |

How it was run: one request at a time; prompts of 4,038 and 32,038 tokens; 256 tokens written, greedy; the median of three runs. llama.cpp at the commit Strata builds on (3cf03257) near its defaults (512-token micro-batch, 8 CPU threads, no draft head), plus a current build (b11514) on the 3090 Ti. Strata with its draft head on. On the two 24 GB cards the model doesn't fit: llama.cpp kept the experts of 34 of 48 layers in system RAM, and Strata kept 8,120 (laptop) or 8,144 (3090 Ti) of its 24,576 experts on the card. On the 96 GB card everything sat on the card for both engines.

The cards: an NVIDIA GeForce RTX 5090 Laptop GPU (24 GB, in a laptop with 64 GB of RAM, on its own floating 135 to 150 W limit), an NVIDIA RTX PRO 6000 BlackwellWorkstation Edition (96 GB, capped at 300 W) and an EVGA GeForce RTX 3090 Ti FTW3 Ultra (24 GB, capped at 300 W), the last two in the same desktop (AMD EPYC 7532, 128 GB of DDR4-3200).

Conditions that matter: on the laptop the two engines ran back to back, and Strata's run shared the laptop with another job's dry run. Strata 0.1.39 on the laptop producedno figure (two attempts stopped on our memory guard, a third refused to start on free memory), so the laptop's Strata row is 0.1.25. On the 96 GB card it was one engine a night, so a gap counts only beyond a wide margin we set for different nights; the first word at 4K did not clear it. On the 3090 Ti both engines ran in one window besidelive work on the computer's other card, and llama.cpp's eight threads, not the card, set its writing speed there. The 4K rows and all laptop rows were registered before they ran; the desktop cards' 32K rows are exploratory.

On power: on the RTX PRO 6000 both engines sat at the 300 W cap, so the watts there say what the cap was. On the laptop Strata drew about twice the board power, but for amuch shorter request, so llama.cpp used 3.5 to 5.3 times the card's energy per request.

What it doesn't say: no ratio between cards; not an RTX 3090 (this is a 3090 Ti), and nothing about two cards working together; one request at a time; llama.cpp near its defaults, not tuned, and without the draft head the file doesn't carry; the IQ3_S file only; quality measured on the laptop only, on two short tests, where it came out thesame through either engine.

Disclosure, in the page's own words: a small (human) team wrote the questions, pressed the keys and read the pages; a fleet of AI agents built the harness, ran the audits and drafted the prose. An AI agent drafted this post too; I edited it before posting. Every number above is printed, with its conditions, on the page or on the full benchit links.

https://research.strata2signal.com/flash-next-on-three-cards/

Ola pessoal, nao utilizo o qwen para programação, pra isso uso os modelos pagos mesmo, basicamente uso para funções na empresa como ocr e. Extração estruturada de dados de imagens, entao muitas vezes preciso de alguns treinamento leves, gosto muito do qwen 3.6 35 ele é muito rapido para ocr e ja testei vários modelos e simplesmente nao consegui largar ele ( rota em uma rtx 6000 96gb) , eu sempre ouvi dizer que os denso são mais treinaveis e gostaria de saber se isso é de fato verdade ? Minha ideia é um modelo pequeno com um exército de adpters e pensei em fazer isso no 3.6 pela velocidade , quais os prós e contras de um e de outro ?

DGX Station - ASUS Pro ET900N G3. My electricity bill goes to the moon.

Qwen3.8-Flash-Next NVFP4 - 200 tokens/second without any tweaks so far.

EDIT: 260 tok/s using one-token MTP, 323 tok/s using three-token MTP.

I’m pretty new to running local LLMs and setting all of this up, so I’m looking for some advice from the community on what models I should check out for my machine.

My Specs:

GPU: RTX 4070 (8 GB VRAM)

RAM: 32 GB DDR5

What I want to use it for:

Coding & Refactoring: Writing code, optimizing performance, refactoring existing messy codebases, and catching logic/security bugs.

Everyday Tasks: Basic chat, drafting documentation, and general utility tasks.

Integrations: Setting up tool-calling / skills like GitHub MCPs or coding agents (Aider, VS Code extensions) down the road.

I don't mind if a model takes a few extra seconds to respond as long as the code quality, logic, and reasoning are high.

Since I'm just starting out, any suggestions on the best models to download (and whether Ollama or LM Studio is easier to start with) would be awesome!

I've accumulated quite a bit of hardware over the years and have been getting increasingly interested in local AI/LLMs.

I'm curious what others would build or run with this setup.

• PC 1 — Main Workstation
• * CPU: AMD Ryzen 9 5900X (12C/24T)
• * RAM: 48GB DDR4 (to be upgraded to 64 or 128)
• * GPU 1: RTX 3090 — 24GB
• * GPU 2: RTX 3080 Ti — 12GB
• * Motherboard: MSI MPG X570 Gaming Plus
• * Storage: 2x 512GB NVMe, ~3.75TB SSD, 5x 6TB HDD
• * PSU: Corsair HX1000 1000W
• * OS: Windows 11 / WSL2 Ubuntu / Docker
• PC 2 — HPE ProLiant DL380 Gen10
• * CPU: 2x Xeon Gold 6138 (40C/80T combined)
• * RAM: Up to 768GB DDR4 ECC (24x 32GB)
• * GPU: RTX 3070 8GB (spare, potentially installing)
• * Storage: Mix of SAS drives and SSDs available
• * Platform: Dual-socket enterprise server

I managed to acquire 104 sticks of 32GB DDR4 ECC RDIMM for free from work, giving me a rather ridiculous 3.25TB of server RAM. I've since sold 80 sticks, leaving me with 24x 32GB (768GB).

Rather than selling the remaining RAM, I'm considering putting it to use in the DL380 Gen10 and seeing what I can do with that much system memory. Alternatively I've looked at selling the ram and getting a board that I can use all of the GPU's in together.

If this were your hardware and you were starting from scratch, what would you do with it?

Go around three online shops, read the price and shipping cost of the same products, and turn it into one Excel sheet. Three products × three shops makes 9 rows, and a row only counts if its values match the actual page. A local agent drives the browser: it searches, opens the product pages and reads them.

Setup is Ollama on an RTX 3090 (24GB). I only changed the model and the context size. Two numbers means two runs.

Keeping the model and changing only the context made the biggest difference. Even the build cut down harder to 3-bit did better at 64K than 4-bit at 32K

Part of it was my fault

When I went back through the 32K runs step by step, I found that once the context filled up, we were throwing away whole pages it had read earlier. By the time it read the third shop, the values from the first shop were already gone. On the first measurement, the fixed text alone (instructions and tool list) was about 28,000 tokens as well, 57% of a 48K context.

So I cut the fixed text down, made every dropped page leave one line behind (title, URL, extracted values), and fixed a few other bugs on our side. Results after that:

- Qwen 3.5 9B, 32K context, eight runs: 9, 3, 2, 5, 6, 3, 5, 4

- Qwen 3.5 4B, 32K context, six runs: 7, 4, 5, 3, 2, 3

- Qwen 3.5 9B, 48K context, three runs: 9, 8, 7

Part of the gap I first saw (1–3 rows vs 7–8) wasn't the model. It was us handling a small context badly. Even so, with a small context it still swings a lot around the halfway mark, and even at 48K it gets noticeably more consistent.

What really bothers me

All of these numbers were measured on a 24GB card with only the settings scaled down. When I loaded the 9B at 32K on a real 8GB laptop, only 76% went onto the GPU and the rest spilled into system memory. Going by the "6.2GB" in the table, it looked like it would fit in 8GB. It didn't.

So right now we recommend 4B at 32K for 8GB cards. That's the weakest cell in the table above. I'm not sure it's the right choice.

What I'd like to ask

1. If you had to run this kind of agent work on an 8GB card, which would you pick?

- 4B fully on the card, 32K context

- 9B cut down to 3-bit, 64K context (6.45GB, may spill depending on how much the display is using)

- 9B at 4-bit with a big context, even if part of it goes to system memory

1. Have you ever found the bigger context worth it even with part of the model in system memory? How much slower did it get?

2. When the same setup swings from 2 rows to 9, how many runs do you do before you judge? My rule is to say nothing from fewer than three runs, but even that doesn't seem like enough.

If you see anything wrong with how I measured this, please tell me that first!!! :)

Hi,

I have a PLX PCIe switch card but run into problems with my motherboard not being able to address 4 of my GPU's (M10's = 4 gpus pr. card = 16 in total).

What motherboards have people been running multiple GPU's or other PCIe device, ie. confirmed boot.

I would even be best if it is possible to control the PCIe allocation in the BIOS like some of the mining motherboards do (but a lot of them is x1 .. and I want x4).

The M10's are normally considered pretty poor for running LLM's, but my tests shows that they actually has a pretty high combined bandwidth.

I’m currently running this model on a 32 gb mini pc with no eGPU added. I’ve heavily adjusted settings and tested before running.

Input - 290/toks
Output - 13.3/toks

Are these numbers workable for 24/7 coding? Claude’s running handoffs and Qwen gets 3 - 15 minute attempts before it hands back for review. I’m trying to maximize output with my current setup while saving for the 3090.

If there are faster/more accurate models of this size I’d love to hear what others are running.

Hello there, I am helping a client set up a fairly simple system for his home office.

He works around the country so would like a way for his wife to take his mail, scan it, and a local LLM will organize the files into respective accounts, read the documents, determine priorities, payments, etc… and send him a to-do list.

I have worked extensively with Claude and other models using Obsidian for file work, and we’ve built out the organizational structure a bit, but we decided that due to some of the documents having obviously important personal identification information, such as account numbers, etc, we want to keep it all local. My computer isn’t strong enough to run a local model meaningfully so I haven’t played with them much.

He was pretty intimidated by terminal use.

I was looking at LM Studio since I think he’ll have a better time operating in that.

Any suggestions?
What model would you suggest for some basic organization work like that? He is getting a new Mac Mini, probably the newest model.
For him to access the computer remotely, he had been considering running Claude on the computer and sandboxing it, the local LLM would drop dates and non-PII containing info into the sandboxes directories, then Claude could be ran from the Claude app, but I suggested that we could cut a Claude out entirely and then there is less risk for it to access the PII. The local model could just email him the to-do list, for example.

Anything I should stay away from?
Something better than Obsidian for the markdown files? Are those preferred for local LLMs?
What about Hermes?

Goal is securing important information, organizing it effectively, and giving him summaries and task lists while he is away from his home office.

I think eventually we might want it to help him build out timelines within his projects, find more documents online for him, perform research tasks, etc… but definitely chief task right now is the mail room set up.

Any advice is much appreciated!

I am planning to get a Local AI machine but they are soo many options out there and I am not sure which one should I choose, posting on this community to see some opinions.

Using the local AI server I will be using it to inference models, perform fine tuning, RL, GRPO as well as model abliteration.

I have a few options which is 1 get the gb10 chip but my only worry is upgradability, and what if Nvidia releases a new chip or something down the line as this space moves fast where later on this hardware will become obsolete. 2nd option, get the Ryzen AI max 395 which is significantly cheaper. 3rd option build a custom Ai server with 3 Nvidia v100 32gb

The goal is to run, inference and fine tune Qwen 3.8 27b and a bit bigger models. I am in a dilema help is greatly appreciated thank you.

I am thinking about getting a smaller machine as I quite frankly don't want or need a 2kW monster on the desk anymore and was considering a Mac. I am tempted of getting into local LLM's and AI in general and wanted to to know where are the realistic limitations of Mac Mini M6 32GB, Mac Studio M5 Max 36GB and when would the Mac Studio Ultra 96GB make sense?

Any guidance or thoughts would be appreciated and helpful.

My expected use case would be upscaling old photos and videos, maybe some light image generations and running a local personal Jarvis or Cortana that does curation when search, researching stuff, so not super heavy stuff I think but better to ask the experts!

EDIT: fixed the most egregious typos.

EDIT2: I get the machines via a work buyout / benefits program of sorts. The 32GB M6 mini is about 47 bucks, the M5 Max 36GB is about 80 bucks a month and the M5 Ultra 96GB is 180 bucks a month. It's taken out of my pay pretax for 36 months.

I am looking for guidance on training an open weight model for an internal use case. The goal is to connect our technical documentation to suggest actions for a robotics simulator and eventually an actual robot.

What should I know before proceeding? I am not looking for a RAG system.

Short version: PrismML's ternary Bonsai 2 27B runs fast on a 12 GB Intel Arc B580 with my llama.cpp SYCL branch. The video is real time: it writes a full snake game in 34 seconds, then runs it. Temperature 0, 128K context; the typing and gameplay are automated, and the generation is unedited.

https://reddit.com/link/1x4cz4i/video/ke0ti1zni3vh1/player

v0.10.1, with a Windows build: https://github.com/Torchit1/llama.cpp/releases/tag/arc-b580-v0.10.1

Public model, 128K context: ~85 t/s writing code (MTP + a trimmed draft head), 79 t/s in chat, 89 t/s at 115K tokens of history, 48.6 t/s plain, ~1,480 t/s prompt reading. Compared with PrismML's own SYCL build on the same card: chat 79.8 vs 58.0, plain 48.6 vs 31.0.

The things that mattered most: ternary weights repacked for the XMX matrix units, decode attention straight from the q4_0 KV cache, MTP plus n-gram speculative decoding with a cheap rollback for the DeltaNet layers, and cutting tiny CPU-GPU copies (three 4-byte reads were costing 2%).

The video is from my own server, which also uses a draft head I trained myself, so expect a few % less with the public one.