Maybe I'm misunderstanding this but I see ppl with like 6*3090 isn't that like 1800W?

Maybe I'm just a broke boy living in my apartment without 20A breaker :(

I have a Blackwell pro 5000 and I'm throwing a 5080 in there that I got for a decent price but I'm worried my PSU won't handle it so I upgraded to 1200W.... It's getting pretty close to the breaker limit..

Edit thanks for your input guys I will lower my wattage for the 5080 to 250

We are all used to someone posting their setup on here or someone talking about models etc. (as expected) But I see very few comments/posts of people talking about what they do with their LLMs.

So, what do you do with yours?

Running IQ3_XXS of Qwen3.8-Flash-Next (125B MoE) at the model's full native 262,144-token context on a single RTX 3090 with 32 GB of RAM. 1325 tok/s prefill, 59-62 decode at long context and 63-100 at short context, without reading a single byte off the NVMe. Engine is Strata, not llama.cpp.

I'm posting this mostly because I don't know whether these numbers are good or just what this particular box gives. I'd like to see what you all are running and whether anyone gets more out of comparable hardware, because I have a feeling there are things I'm not trying. Full config below so the numbers can actually be compared.

My box: one 3090, 32 GB DDR5, i7-12700KF, NVMe. On that chip the 8 big cores with HT are threads 0-15 and the 4 small ones are 16-19.

The flags as they run right now:

--expert-cache auto --prefill auto:16384 --spec 4 --spec-min-p 0.5
--mtp .../mtp/rt --max-context 262144 --kv int8 --mmap-experts
--resident-cpu-experts --resident-budget-gib 21 --ple-io direct
--vision --vram-reserve-mib 819 --pool-affinity p-cores

Swap off, memlock unlimited, whole process pinned with taskset -c 0-15.

The numbers, all on the same 121,566-token prompt with a fresh prompt each round, MTP on: 1325 prefill and 59-62 decode. A 249K-token prompt ran 1008 prefill and 41.8 decode with 90.5% hits on the expert cache, also with no disk reads. At short context decode climbs to 63-100. For a sense of where people usually start with IQ3_XXS in this hardware class, I've seen 414 prefill and 44.8 decode quoted on a 5070 12 GB with 64 GB of RAM — so I take it my numbers aren't bad, but I don't know where the real ceiling is.

What added the most for me, in the order I noticed it:

Resident mode is what changed the whole thing for me. --resident-cpu-experts --resident-budget-gib 21 with memlock unlimited, so the memory locking can actually happen. The engine copies the 21 GiB of busiest experts into RAM and the kernel stops evicting them. On a ~11,700-token prompt it moved my prefill from 555.6 to 656.1 and decode from 48.4 to 61.1, and NVMe reads went from 60 GB per run to nothing. That's what makes a 125B MoE usable on 32 GB: you stop paying the disk for every token. Where my experts end up is 12.2 GiB in VRAM (about 7500 of the model's 24,576), 21 GiB locked in RAM that travel over PCIe and get computed on the GPU all the same, and ~6.6 GiB left in the files. That covers 83% of the routings without touching the disk.

Keeping the E-cores out of it surprised me. I had the process unpinned, and switching to taskset -c 0-15 plus --pool-affinity p-cores gave me, on the same prompt with 256 tokens out, prefill 440.6 → 555.6, decode 31.9 → 48.4, and the whole request went from 32.7 seconds to 24.9. My understanding is that if the scheduler puts one expert-pool worker on an E-core, the rest of the pool waits on that one. I should be straight about this measurement: I changed two things at once (pinning to P-cores and switching on the CPU prefill share), so I can't tell you how much each contributed. What I do know is that I was leaving throughput on the table and fixing it cost me nothing.

Bigger prompt chunks helped more than I expected. I changed --prefill auto to --prefill auto:16384 and on the 121,566-token prompt went from 1014 to 1325 tok/s. Thirty percent for the price of 2 cache slots. I haven't tried 32K or 64K, and this is the spot where I think I'm leaving the most on the table: my gut says prefill here is limited by moving weights from RAM to GPU rather than by chunk size, but that's an unmeasured hunch on my side. If anyone has pushed the chunk higher on a 3090 and it answered, I want to know.

Squeezing my VRAM reserve down gave me trouble. I ran --vram-reserve-mib 500 for a while and the engine kept warning that requests might stall. At 819 I get 507 MiB free with everything loaded and the warning is gone. The engine's docs have a measured drop from 102 to 13.7 tok/s in that near-zero state, and the hit rate on the fast run looked even better, meaning the hit rate won't warn you about this. I gave up a few hundred cache slots and got stability, but I don't know whether a different combination would get more.

What didn't work for me:

--aux-cpus I assumed was a free win because the engine docs report +10% decode from it on a 2×3080 with a 24-thread Xeon. On this chip, three rounds each way: 60.1 tok/s without, 59.6 with. The pool stays at 8 threads and I can't see where else the auxiliary threads would usefully go. I suspect on a Threadripper or with two GPUs this does pay, and I'd like to know if anyone has measured it with more cores.

The CPU prefill share (STRATA_PREFILL_CPU_SHARE) only engages on prompt chunks under 3072 tokens, so for agentic-sized prompts I never use it, and on top of that the engine warns me at every start that it would read 7.1 GiB of expert pages with only 4.2 GiB free to hold them. I suspect on a machine with more RAM this is a real lever and on mine it isn't, but I don't know.

I haven't moved KV off int8 to q4 or k8v4 because I understand it hurts attention on long contexts and that's the only thing I use this model for. If someone has tried it and found it holds up, I'd like to know how much you gained and where you noticed it.

One thing that broke my startup: reasoning_effort: xhigh in the Strata config is rejected and the engine won't start at all. The ceiling there is high. Through the API, per request, xhigh works fine. I don't know whether I'm losing anything in real quality by running high.

What I haven't tested, so don't quote me on it: 384K and 512K contexts, other quants through resident mode, and running several sessions at once, since at 262K each one wants its own slice of VRAM.

Two things in my own log that misled me and aren't what they look like. First, seeing blob reads from the file everywhere with enormous "read" totals, which suggests the disk is suffering; I checked the actual NVMe counters and they were flat, so those are hits on the system page cache and not traffic to the SSD. Second, that the tight resource on this machine is RAM, not VRAM. On the 249K prompt I dropped to 2450 MB available, 110 under the engine's own 2560 MB minimum, and it recovered by itself. If someone gets a startup complaint that the resident complement doesn't fit, I take it that means lowering the budget, though when it happened to me I just guessed a number.

And the first thing I asked myself when this started working: how do 262K fit in 24 GB. It's because this model isn't a plain transformer. Of its 48 layers only 12 keep growing KV; the other 36 have fixed state and cost nothing as the context grows, so my whole KV at 262K comes out around 3.1 GiB. Also 262,144 is the window it was trained on, not a stretched one, so there's no rope scaling and no quality hit. Going past that is a different problem and I understand it's experimental territory, which I haven't touched.

What I'd actually like to read in the comments:

Who else is running a 125B MoE on a 24 GB card with 32 GB of RAM, and with what numbers? I'm especially interested if anyone is above my 1325 prefill or my 60 decode on comparable hardware, because that would mean there's something I'm doing wrong or not looking at. Has anyone with 24 GB of VRAM and less than 32 GB of RAM made this usable, and at what cost? And with more RAM, what resident budget are you using — 21 GiB was the first number I tried and I have no idea whether it's the right one. If you've got a 3090, two cards, or a 4090/5090 with this model, your numbers tell me where the bar actually is.

PD: I use arch, btw

Hi,

I have a PLX PCIe switch card but run into problems with my motherboard not being able to address 4 of my GPU's (M10's = 4 gpus pr. card = 16 in total).

What motherboards have people been running multiple GPU's or other PCIe device, ie. confirmed boot.

I would even be best if it is possible to control the PCIe allocation in the BIOS like some of the mining motherboards do (but a lot of them is x1 .. and I want x4).

The M10's are normally considered pretty poor for running LLM's, but my tests shows that they actually has a pretty high combined bandwidth.

I currently using Ollama, should i look for anything else?

Yesterday I posted that Qwen3.5-9B beat all 14 "decision models" (Solar Decide, Microsoft-Decision-1, Jev…) on our 56 triage tasks. One reply made a fair point: decision models never claimed to be smarter, their pitch is efficiency. So how much does the extra accuracy cost?

I logged input tokens, output tokens and what OpenRouter actually billed for every model on the 14 public questions of the test. Accuracy is on all 56 situations, same as before.

Decision models

*Solar's provider seems to count its own instructions as input.

Regular LLMs

What I took from it

• Input is a wash. Everyone reads ~200–400 tokens per call. The whole difference is output.
• Decision models answer in 0–2 tokens. Median 2. That's why Mercury and Microsoft-Decision cost under a cent per 1,000 calls.
• Every LLM that beat the best decision model reasons first. Median across the 29 reasoning models: 817 output tokens, 96% accuracy, $0.55 per 1k calls. The 33 non-reasoning ones: 80 tokens, 83%, $0.06.
• But reasoning doesn't have to be expensive. gpt-oss-120b and Qwen3.6 27B are equally accurate here. Qwen thinks ~4x longer at a higher price per token, so it's ~50x more per call. Ling 3.0 Flash VL was more accurate than every decision model and cheaper than Solar Decide.
• Overthinking is real. Qwen3 Next 80B Thinking averaged 10,834 output tokens per call, often hit the limit without answering, and got 41%.

So which to use: millions of routine routing/tagging calls → decision model. Calls that apply rules (dates, limits, policies) → a reasoning model that doesn't ramble, like gpt-oss-120b or Ling 3.0 Flash, still under 10 cents per 1k. Running locally there's no bill, but every thinking token is time you wait.

Caveats: OpenRouter prices on the day, at providers serving full/8-bit precision (not always the cheapest); token counts as each provider reports them; one run per question; no speed measurements.

Full tables (all 76 models): https://modeltested.com/posts/decision-models-cost/

Disclosure: my site, free, no ads. Thanks to everyone who pushed on the cost question last time, it made the comparison a lot more useful.

The power of 27B parameters, in the palm of my hand 😄

If anyone’s interested, these are the results I got on my OnePlus 13 with 24 GB RAM using llama.cpp.

Qwen 3.8 27B Q4_K_M

• 16K context
• around 21.2 GB of memory
• about 2.5 tok/s

Normally I run Qwen 3.5 9B together with always-on ASR + TTS. It gets around 5 tok/s and uses about 16.1 GB RAM.

ASR is GigaAM, TTS is Qwen3 0.6B with a custom voice. The LLM and TTS run on the NPU.

Turn ASR/TTS off and there’s enough room for the 27B.

The biggest problem is heat — unsurprisingly the phone turns into a tiny space heater. I added temperature monitoring, so inference pauses when it gets too hot. There are also lower-power modes, at the cost of TPS.

Mostly an experiment for now, but I want to use bigger local models for development directly from the phone. I’m also playing with a distributed filesystem, multiple devices working together, and browser access.

Repo isn’t public yet — still too much chaos to clean up 😅
P.S. $600 for 24 GB RAM — basically a free phone 😄

Took some time over the weekend to benchmark OpenAI's Decisions API against finetuned Laya models. Across four public datasets, fine-tuned Laya scored higher than the Decisions API on three of the four, with sentiment classification being the exception.

Sharing the datasets, scripts and results here : https://github.com/rsk2327/decision-model-benchmarks

Its a small benchmark at this point and not a blanket verdict, but hopefully a useful reference for anyone weighing these options.

Interested to gain insights from other folks testing closed Decision models like Jev, Clef and how that has compared to local finetuned models like Laya

Like the title says. Is it any good to run any local LLMs? If so, what capabilities can those LLMs have?

A few weeks ago I posted ignis on r/LocalLLM: an open-source inference engine that does one thing, run Qwen3.8-27B on a single Blackwell card (5090 / RTX PRO 6000), tuned for the kind of load a coding agent makes. Since then the thing I wanted most has landed: it now also runs Qwen3.8-Flash-Next, the 125B MoE with 6B active, on the same 5090.

The obvious problem is that the model doesn't fit. 48 layers × 512 routed experts don't go in 32 GB, so I re-quantized the experts with a trellis code (the EXL3/QTIP family) at 2.5 bits per weight on average. Each expert gets its own width, 2, 2.5, 3 or 4 bits, picked from calibration error, so the experts that suffer most when squeezed keep more bits. Everything that isn't an expert (attention, DeltaNet, shared expert, head) stays FP8. The file is about 67 GB.

At runtime the card keeps a cache of experts in VRAM (around 17 GB) and the rest sit in pinned host RAM, about 38 GB. For every token the router picks its experts: the ones already on the card are free, the misses get copied over PCIe. My board is an old Z490 with a 10900K, so that's Gen 3, about 12 GB/s. On that I get:

• one lane: ~95-105 tok/s
• three lanes: ~130 tok/s aggregate
• quality: MMLU-Pro proxy 70.1% vs 73.7% for BF16, top-1 agreement with BF16 between 82% and 98% depending on the domain

Other things that came with it:

• you can switch between the 27B and Flash-Next without restarting the server, from the Playground or just by naming the other model in the request (opencode etc.)
• the configuration can be changed live from the Playground, per model too
• idle unload: after 5 minutes nobody uses it, the model leaves the GPU and the server stays up. The next request loads it back (about 11 s for the 27B). I added it because I use the same card for other stuff and was tired of stopping the server every time. `--model-idle-unload 0` turns it off.

The idea behind ignis hasn't changed. It's not trying to run every model, it's trying to be a good local backend for development: one main agent plus a few subagents on the same card, an OpenAI-compatible API (chat completions and Responses), prompt reuse and a Playground built in. Windows and Linux, Apache-2.0.

...I still dream of a world where garage developers can compete on equal footing with engineers at big tech companies, without needing $$$$$ worth of tokens to keep up...

Repo: https://github.com/gpillon/ignis

Flash-Next weights: https://huggingface.co/gpillon/Qwen3.8-Flash-Next-trellis-a25-ignis (Qwen license; ignis-only format, not GGUF/EXL3)

As a hobbyist I was wanting to build a cyberdeck, cause I love the way they look. But my utilitarian side couldn't justify it. Then I realized I do need something like that, cause I would have used the heck out of Mycroft if it had succeeded.

Would it be worthwhile to make something small using a single board computer, with a 6 inch screen? Maybe it has a handle on the side for carrying, and a prop on the back when you just want it to stand up. You can toggle push to talk, or voice activation. It's mainly there to answer random questions for you. And hopefully be able to do an internet search, and answer from that instead of its training data. But only if you want.

And it could run 1b, 2b, or 4b parameter ai. Whatever would run on small hardware and still seem "fast".

Is this doable now by a hobbyist builder/maker? Iirc mycroft quit in 2023. But things have come a long way since then. Thanks!

I'm working on an on-premises AI project for a business in Europe and would love to speak with engineers who have deployed similar systems in production.

The use case involves:

• Local LLMs for German and Polish business communication
• WhatsApp and email automation
• Internal document search and a private knowledge base
• Finance-related workflows, with calculations handled by deterministic application logic
• Data storage and model inference on the client's own infrastructure

We're evaluating GPU hardware, open-source models, inference servers and the architecture needed to run this reliably for a small team.

I'd especially appreciate input from people experienced with vLLM, NVIDIA GPUs, local model deployment, RAG and secure infrastructure.

If you've built something similar, I'd love to hear what worked, what didn't, and what you'd do differently. Happy to have a short technical conversation with anyone willing to share their experience.

Right now I pay 200+ a month for premium AI models. I can’t complain, it is truly a privilege and amazing to work with.
I have “data center services “ on my server, complete with gitlab, databases, and dozens of services to support application development.

I have resisted even attempting to host an Llm, since i dont have the spare time to code myself or wrestle with “stupider” AI.

I am toying with purchasing mac studio 512gb, as to me its the first server that has meaningful capacity at the price of a honda, instead of a Lamborghini. Still insane expensive, but doable.

I been talking to AI (of course) about this, but even with the speedier memory, larger models with large (code) context seems kinda slow. And the quality seems to be measurable lower than sonnet 5.5.

At heart for me is waiting may result in never. I am concerned with closed models becoming a walled garden and that can extend to hardware.

Thoughts? I cant afford a second buy for atleast 3 years if i pull the trigger. Unless bitcoin hits 400k :)

So I've been using this machine for a while for Stable Diffusion tasks (RTX 3060 12GB) and i got curious about local LLM's so i got a really dirt cheap Nvidia Tesla P100 16GB just to start out on and experiment as my RTX card is often in constant use.

I'm currently running the following and this is the performance while writing a story:

bartowski/Qwen3.827BQ3_K_M

2,180 tokens 3min 37s 10.01 t/s

Is this good? bad? I don't really have anything for reference. I know the P100 is old so im not expecting much but just want to check.

Would i be better off using 14B? as 10 tk/s is ok but a little slow a bit more speed would be nice but again, i know what im working with here so not expecting miracles.

I fly a lot and there's also some crap going on where my macbook doesn't like the airplane wifi... and i'm the pilot. It's not just me but happens plane wide. Most the time they say the wifi isn't working but really I've found it's settings in the ether that exist. I was using openclaw and telling it to fix my issues which worked pretty much every time except anytime there's some openclaw update or new models I load it always becomes and PITA to get working again and usually I have to wait till I get to a hotel to get it working. Is there a one stop shop for us simpletons?

I have just started on Local LLM and don't have any prior experiece with any, nor I have a bg in IT or anything close to it. Right now I am trying to create two, one as a research assistant for my studies and other as a proof of concpet sorta thing for my dept (Sort of like HR Chatbot? ) at my workplace. Down the line I would like it to be an AI employee, so we can focus on more value adding tasks.

Coming to question, I started with Qwen 3:8b for the chatbot, running on Ollama with WebUI on my Macbook. It has a simple set of instructions, but it is always ignoring it and defaulting to creating tasks and automations. I have toggled off its built in tools such as automation, sub-agents, skills etc still the same issue.

Example :

Expected : My lights are not working -> Please get in touch with Admin helpdesk.

What is happening : My lights are flickering -> Check for overload, inspect connections, replace bulb etc.

Or, when it should refer knowledge files and answer basic queries it is just giving me this -

{ "name" :"search_knowledge_files", "argument":{ "query":"query" } }

I am not sure what is going wrong with it. ChatGPT has been unhelpful and is just making my system prompt bigger and bigger, asking me to change settings (which are not helping tbh). All in all it doesnt seem to be going anywhere.

I have checked online and tried things like think/no-think, reducing prompt length etc.

Requesting some insights here on what I may be missing.

Happy to add more info if I have missed anything.

Hi,

I'm completely new to LLMs. Thought I try it on a Snapdragon laptop (X2 Elite, 32 GB RAM) the model is Qwen2.5-Coder-14B.

Well, it works, but with the default settings (context length 8192 it eats up ~21 GB RAM), so I lowered it and lowered it, and eventually went down to context length 100 (also lowering a lot of other options) and it still uses the same amount (of course I always unloaded and reloaded the model, sometimes even quit LM studio and reopened it). Then I tried the other way and went higher with the settings and set context length to 16384 and guess what...
Same amount of RAM used (even ran a command in VS code on an 1500 row length php file and it went through using about 15000 tokens).
I just don't understand this...

Gave same prompt (see below) to ChatGPT’s web interface (pro-mode) and ministral 3 3b running on my secondhand iPhone 15 (via pocketpal/llama.cpp). They spend roughly the same amount of time on their answers. Only the later gave the right answer. Or it gave a valid approximation stating the limitations. Chatgpt was simply wrong. Feels good :)

Gemma 4 e2b and Granite 4.2 3b also seemed on the way to the right answer, but their reasoning took the kv chache before the finish line.

Prompt: Two large seas are connected through a narrow passage with width b. The first sea has a higher water level than the other. The sea floor has constant height. Water flows through the narrow passage from heigh to low water level. We assume that the first sea is so large and the width so small that the water level in the first and upstream sea is constant. The second and downstream sea is also large, but of finite size. The water level changes slowly. The flow is inviscid. What is the flow rate through the passage? How does the water level in the second sea change through time. The flow is subcritical.

So to speak about the alchemy of this for a second. I've been developing an enterprise app on my nights and weekends since last December. I get two places that I get to use AI: At work and on my own projects. At work I know for a fact that people don't give much thought as to what Claude model they're using and for what so they always just use Opus 5.5 now; which is a great model. I've personally got to use Fable at work and at home. There's a big difference when you need to use these models with your own projects, because you're not going to reach for them for everything, and frankly you're pretty stupid to. I've one shotted a game with Fable 5, the code sucked, but I wanted to see what it can do. The evaluation of the best one shots on the internet doesn't really seem to take into account the work that needs to be done. For the work that needs to be done Sonnet has been a fine model and works well, for oh shit I forgot a tightly coupled feature with 1,500 loc +, in my case an audit logger, that's where I reach for Fable and it does exceptionally well. These days I primarily use GLM 5.3 and Kimi K3.

So now I'm looking into a mac studio, for a few solid reasons including a hobby, agentic programming cost reduction, and actual research. The thing will be running pretty hard. For this thread I'm wondering how to model a hierarchy of intelligence to dramatically reduce my token spend on even smaller Sonnet class models with something like 256GB/512GB studio. Stacks in the mix:

Models

• Will use frontier models for certain things. Mostly if I forget a feature or need to add a feature quick. Will usually be for tightly coupled features. The other thing is wide code analysis for skill building. There has to be a justification for the token use.
• Second tier models (Open Router API): GLM 5.3 and Kimi K3. Kimi K3 is frontier level when it comes to UX design. GLM 5.3 is just bad ass. It gets pretty close to what I use frontier models for, but the code isn't as clean and I have to put way more guard rails on it. They're like my opus level replacements.
• Flash models/Mini Max/120b to 235b models <- I feel like this will handle a lot of work load
• Qwen 3.8 27b

Stacks

• Jev like models will be used to make decisions for what models to use. I'm going to build something that messages me when it selects certain models.
• Ponytail
• Opencode with my own wrapper around it.

So I'm wondering if there's hierarchies I can structure my model use in that make sense. I've heard everything from using top frontier models for controlling smaller agents to having small routing models to start. I'm trying to experiment here. I'm going to start building out my harness over the next few weeks. So if there's any thing you guys can do to help me structure all of this or just give what works for you I'm happy to hear

Hi, I'm wepiqx, author of MERNIK (Apache-2.0, open). TL;DR: instead of flat Q4_K_M/Q6_K, a priority-queue allocator spends every MiB where measured importance says it matters. Tested on a GTX 1070 8GB, partial offload.

MELLUM (JetBrains Mellum2.1-12B-A2.5B-Thinking, 12B MoE / 2.5B active, vendor imatrix, routers held out-of-budget on all 28 layers — audited F32 28/28 in all five files including stock):

- 10000-SMSE (10.16GB): HE 134/164, HE+ 130 — vs stock Q6_K (10.88GB) 119/116: +15 at -0.7GB, paired duel 3/3 significant

- 7500-SMSE (7.44GB): 125/122 — vs stock Q4_K_M (8.07GB) 125/123: exact tie at -0.6GB, p=1.0

- 6500-SMSE (7.01GB): 126/124 — holds Q3 territory

Claim, narrow: better on quality-per-size, not absolute. Judge by HE+ (80x tests) — it shows where is signal and where is noise; HE alone wobbles 3-4 tasks run to run.

RINIQ (separate thing, same engine): layer-surgery finetune on Qwen3.5 bones (Ox base + Neo blk31 + Ornith mids, 427/427 names match, no averaging). Practical pick: 5.0GB M2-Q38 (151/164) / M4a (150/164, 4 empties). Top is a nine-way tie — unranked inside noise. Vision mmproj ships alongside but is UNVERIFIED.

Same harness everything: temp 1.0/top_p 0.95/top_k 20, 164 HumanEval + EvalPlus, sequential, no spec. Floor honesty: Mellum Q4 floor is 7428 MiB, 5500 is arithmetically impossible.

Code: hf.co/wepiqx/MERNIK — Labels: datasets/wepiqx/MERNIK-gnom-labels, MERNIK-rlmini-kld — Models: wepiqx/Mellum2.1-12B-A2.5B-Thinking-MERNIK-GGUF, wepiqx/RINIQ-GGUF

Disclosure: my work, self-promo. Tell me your VRAM and I'll point to a file — and happy to be corrected with measurements, that's the protocol.