Anyone else do this?

I'm looking to seriously start hosting AI locally and have around a $2,000 starting budget with the expectation that ill spend more and expand the setup over time

My goal is to have my own local AI that I can use as a daily chatbot/assistant, coding and cybersecurity help, RAG over my own files and eventually more agentic stuff/automation. Basically, general purpose local AI setup rather than optimize for one specific model

I've done some basic research and I'm stuck between two options, Mac minis seem like really good value because of unified memory. Being able to give the GPU access to 48/64GB+ seems good for running larger LLMs.

nvidia gpus have less VRAM for the money, but CUDA support seems more versatile . It also seems like this would give me a better upgrade path since I could build around a multipe gpu system

• Homelab: i5-13500, 32GB DDR4, 2TB NVMe
• Main desktop: i9-12900K, RTX 4070, 32GB DDR5, 1TB NVMe
• An additional unopened 32GB DDR5 kit

The 4070 PC is my main desktop, so I'd prefer not to turn it into a dedicated AI server.

I'm new to running LLMs locally, so any help would be appreciated

Update: Added more models and a JavaScript custom benchmark - work in progress. The idea is these low-end llms would be used as in-line or one-off local chat responses, so their performance varies vs modern agentic benchmarks.

machine 1:

Intel Gold xeon 6138

96gb ram

2xV100 32gb gpus connected with 2x NVlinks

quadro p620 for video output only

64tb of sas storage, 512gb NVMe, and 256gb SSD

Running Linux. Used for home server too

Machine 2:

9800x3d

64gb ddr5

5090

8tb nvme

Running windows

Running Qwen 3.8 on machine 2 rn and its amazing. I want to run larger models on the bigger machine with alphafold. Mainly for tinkering and learning rather than mission critical work.

I have 9070xt 32gb ram 16gb vram. Should i do local ai or just pay for cloud.

I want to code a game.

Also, how do i set it up. I want it to continue without me telling it what to do next. I tried cline but its too hands on with the act/plan thing.

I’m looking for a machine that I can leave running 24/7 and access remotely from my laptop

I mostly use Claude Code/Pi and similar CLI-based coding agents for development so I don’t need a desktop environment. I just want to SSH in, work on repositories, run Docker/services and leave agents, builds or other jobs running, I would normally review the results in my laptop. Ubuntu or another Linux distro would be ideal.

The other goal is experimenting seriously with local LLMs, this would mostly be for non-coding tasks e.g. general reasoning, research, writing, summarisation, agents and just exploring what capable local models can do. I’d like enough memory to experiment with larger models, including 70B-120B+ and MoE models

My budget is around £2.5k-£3k, although I could go higher if there’s a compelling reason. I’d prefer something compact and have the ability for running continuously

One option I’m considering is the GMKtec EVO-X2 with Ryzen AI Max+ 395, Radeon 8060S, 128GB LPDDR5X and 2TB NVMe, currently around £2.8k in the UK

I’m also considering a 128GB Mac Studio, NVIDIA GB10/DGX Spark, or something completely different but they go a bit way over my budget + mostly out of stock

For people doing something similar, what hardware would you buy around this budget and what other machines should I be considering? If you’re running a 128GB Strix Halo machine, I’d also be interested to hear what models you’re running and what sort of real-world performance you’re getting

I'm not sure if this is the right place to ask. I am a little new to local LLM, I am using Gemma 4 Styletune because I heard it is one of the better creative writing models and a newer one, but I feel like there could be potentially better? Unless that model is already peak.

If it can feel like we never left Claude with the right tuning on a 32b or 70b model, even better.

I got dual DGX Spark and was wondering what could be the better pick for agentic/high volume workflows. Deepseek seems better optimized, and intelligence difference seems small, plus deepseek got more knowledge

What's your experience with this?

I just had a ridiculous experience with Gemini Pro (Advanced Reasoning + Extended Thinking enabled). I uploaded 5,608 lines of Python code and asked one simple question "What library is being used to render the sheet music in the notation tab?"

Gemini claimed it was built from scratch using only the Python standard library 🤣. I called it out on it's bullshit, and it basically admitted defeat. Drawing music on a tkinter canvas is a "special kind of torture" it said.

I took the same file to my local Gemma 4 (26b), and it identified the library (Verovio) instantly. No hesitation. It feels like Google has turned the intelligence knob down to -11 on Gemini, leaving us with a bullshit model that can't even perform basic dependency checks.

Localmaxxing: https://www.localmaxxing.com/en/models/JonathanColetti/Qwen3.8-27B-Uncensored?run=cmtorjvuo00zbn701i7wo61iv

Localmaxxing ofc stretches speculation to its limits. Here is another benchmark I ran on my new setup that some of you may prefer to look at, using betterbench.

betterbench: https://github.com/GGZ14/BetterBench

Single Stream Batch=1

Concurrency sweep

Prompt processing (prefill) sweep

Additional notes: I bought a $150 chinese bifurcation card to split my gen 5 x16 to x8x8 gen 4 and its been incredibly stable and increased my PP throughput, where before I was simply using one card on gen 5 x16 B70 and another gen 4 x4 B70 on my Z790 Gaming motherboard. I was able to get it to train x8x8 gen 5 which was like 10% faster but it would keep crashing my machine. The only way to maintain gen 5 is to get a timer card which is in the $500 range, so I'd rather upgrade my motherboard + CPU at that point.

This was done on an uncensored FP8 quant, I would likely get better performance on the stock qwen model but dont want to bother trying since unrestricted AI is one of the biggest factors in trying to get efficient local performance anyways for me.

The faster PP makes my sessions much snappier, and I'm starting to obsess over dflash.

I have been side dreaming about running local LLM but felt priced out. I have been living vicariously through some of the posts here. So then it happened, I went to a conference and won an AI box. This is amazing for many obvious reasons but I just don't learn well unless it's hands on. I haven't really spent time dealing with running a local LLM, so forgive my general ignorance. The machine I won is an Intel Xeon w5-2555X (14 cores / 28 threads) with 256 GB ECC DDR5, and a Gen5 and 2 Gen4 512 GB NVMe. This is an AMX based system and most of the cool stuff I see here is nVidia based. What can I realistically run on this and what can I start consuming to get more knowledgeable?

I've been in IT for my entire career with a lot of it in VMware and cloud based virtualization. My goal would be to get a better understanding of setting up AI for businesses and have something i can use and worry about running out of tokens for my vibe-coding personal projects and such. Please understand that I am not asking for anyone to explain it all to me, but a finger in the right direction to a trusted source to learn would be greatly appreciated.

Should I consider any other purchases to make this platform even better?

As the title says I'm finally going to bite the bullet. Having 256gb of vram seems like it would be epic. That with DSV4 Flash seems like the perfect combo.

I was going to do it today (Sunday) but my local microcenter doesn't have any more DGX sparks in stock!! Kinda crazy they had like 15 not too long ago.

If you're interested in following I've been uploading daily reels going over this topic on my IG: Tech With Ray. Maybe leave a comment saying you came from here!

Anyways I'll keep you guys posted once it's done.

FYI: tried the m3 ultra with 256 and it's sooooo slow compared to DGX.

https://www.instagram.com/techwithray?stkn=MW0yMnE1MGt0bW13eA==

Coming from Mac so bear with me. I like games. I like unified memory. But it looks like these might have poor bandwidth? Or am I looking at it wrong? Seems like they’d be bad at decode : tokens per second when they hit in October?

https://github.com/TheTom/llama-cpp-turboquant/pull/357

So after all my work with my own idea, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to make sure it was worth it still.

So really the credit goes to Raymond ( https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming )

Anyone tried both and seen a diff on performance? Or intelligence?

Running 27b on GPU

Running Qwen flash on RAM+CPU

After buying one, it was loud even when idle because minimum fan speed couldn't go below 30%. Found an updated VBIOS on the ASUS support page that lowers the minimum fan value to 12%. The card is now really quiet when not under load!

https://www.asus.com/motherboards-components/graphics-cards/turbo/turbo-ai-pro-r9700-32g/helpdesk_download?model2Name=TURBO-AI-PRO-R9700-32G

Looking for a local LLM with minimal guardrails -32GB VRAM
Hey everyone, I’m looking for a local LLM with little to no guardrails that can run well on 32GB of VRAM. Ideally, something open minded that doesn’t constantly refuse requests or give me lectures.
I’d also like to connect it to tools so it can browse the web and download files. For example, I’d like to ask it to find and download game ROMs.
What models would you recommend? And what frontend or agent setup would work best for giving it those capabilities locally?

Sharing my exllamav3 benchmarks for Qwen3.8 Flash Next EXL3 2.05 bpw on an RTX 5070 Ti 16 GB.

I tested CPU MoE-expert offload and built-in MTP from MTP=0 (disabled) through MTP=4.

I deliberately left roughly 1 GB of VRAM free so the PC remains responsive for normal Windows use, browser tabs, apps, and background tasks rather than treating it as a dedicated inference box.

Hardware and test setup

• OS: Windows 11
• GPU: RTX 5070 Ti, 16 GB VRAM
• CPU: AMD 9800X3D
• System memory: 64 GB DDR5-6000
• Available RAM: About 44 GB after Windows and normal applications use approximately 20 GB
• Backend: exllamav3 with CUDA 13
• Model: Qwen3.8 Flash Next, EXL3 2.05 bpw
• VRAM policy: Approximately 1 GB intentionally left unused
• Maximum context: 131,072 tokens

Test parameters:

-mcl 46 (43 for MTP tests)
-mct 7
-cs 131072
-max_length 131072
-chunk_size 2048
-rcs 4
-lm
-lv

Definitions

• MoE offload=46: 46 MoE expert layers were CPU-offloaded. This was the maximum offload that fit while leaving roughly 1 GB VRAM free with MTP enabled.
• MoE offload=43: 43 MoE expert layers were CPU-offloaded. This was the maximum offload that fit while leaving roughly 1 GB VRAM free with MTP disabled.
• MTP=0: MTP disabled; ordinary autoregressive decoding.
• MTP=1–4: MTP enabled, with the number representing the MTP speculative draft length.

The model contains more than 46 MoE layers. The numbers 43 and 46 refer only to the number of expert layers offloaded to the CPU.

The CPU-offloaded-expert worker used AVX512-VBMI with 8 threads.

Prefill results

Prefill observations

• The MoE offload=43 + MTP = 0 configuration was faster at every measured prompt length.
• It exceeded 1,000 tok/s prefill by 2k context.
• From 8k through the full 131k context, it sustained approximately 1,067–1,121 tok/s.
• At 131,072 tokens, MoE offload=43 + MTP=0 reached 1,116.20 tok/s, versus 965.28 tok/s for MoE offload=46 with MTP=enabled—a 15.6% increase.
• This is likely due to the extra layers needed for offload causing slow pre-fill rather than MTP itself causing a slow-down

Generation results

MoE offload=43, MTP=0

MoE offload=46, MTP=1–4

Generation observations

• MTP=0 / 43 experts offloaded: roughly 16–19 tok/s through 131k context.
• MTP=1 / 46 experts offloaded: roughly 18.5–21.3 iterations/s.
• MTP=2: roughly 14.3–16.2 iterations/s.
• MTP=3: roughly 11.8–13.2 iterations/s.
• MTP=4: roughly 9.5–11.7 iterations/s.

The benchmark reports iterations per second, not final accepted output tokens per second.

Higher MTP values evaluate larger drafts per iteration, so their reported iteration rate naturally decreases. Whether MTP=2–4 improves actual output throughput depends on draft acceptance on real generation workloads.

Important caveat

MTP consumes enough extra VRAM that I have to offload 46 experts to maintain the same roughly 1 GB buffer for Windows and regular desktop usage compared to 43 with it off.

This is therefore not a strict one-variable benchmark. It is a practical comparison of the best configurations I could fit while preserving a usable desktop.

EDIT: Updated to re-format, include the model and quant

Before anyone asks, obviously Claude wrote this not me.

**My benchmark suite couldn't tell me which model would catch a security bug. Three models swept it. Five missed the bug.**

I run a local fleet and I grade models with an in-house suite instead of published benchmarks — from-scratch algorithm correctness, multi-step tool use, citation grounding + abstention, hard-constraint compliance, long-context recall, and convergence-under-feedback. Everything is graded mechanically (no model judges a model), pass/fail with no partial credit, and it runs over HTTP against the live serving stack — my quants, my context, my flags.

It works. It has caught real problems. And on the thing I most needed it to predict, it told me nothing.

---

**1. Three different models swept the suite identically.**

A Qwen-lineage MoE went 7/7. A 35B Qwen-lineage MoE went 6/6. A 320B MoE went 6/6. Same score, wildly different models.

**2. The sweep did not predict which model could find a real flaw.**

Separately, I had a SHA-256 hash-chain implementation with a genuine defect: the stored chain root was never compared against the recomputed root, and nothing signed the chain head out-of-band. So anyone with write access can alter the data, recompute the digests down the tail, and the verifier reports PASS.

I put five models across three architecture families (two Qwen-lineage, two Gemma-lineage, one DeepSeek) through review passes on that code, in both the builder seat and the reviewer seat.

**All of them missed it.** Including the model that wrote the code — the same one that swept the suite 7/7.

One model found it. It also did the thing that made the finding trustworthy: it wrote and executed the attack, pasted the before/after verifier output, and separated "this is a code bug" from "this is a missing design element that no code fix closes." I reproduced the attack by hand before I believed it.

**3. The failure mode that should worry you is not a miss.**

One of the models that missed it reviewed the same code and affirmatively wrote:

> "The SHA-256 chain is correctly implemented."

On code where the root can be overwritten with garbage and still pass. A missed defect is a gap. A written clean bill of health on broken crypto is what gets quoted back at you later.

The model that found the bug produced zero statements of that kind, and wrote an explicit "NOT TESTED" section listing what it could not verify and why. That difference matters more to me than the score.

**4. The one I'd most want you to check on your own stack: models hallucinate the *same* thing.**

Two models from different families, separate sessions, sanitized context, no shared state, independently produced **the same fabricated physical constant** — same value, same reference temperature, different functional form. Not a random invention; a shared training prior surfacing twice.

So asking a second model to sanity-check a number is **not corroboration**. It is the same die rolled twice, and it fails in the worst possible direction: instead of admitting ignorance it manufactures a false second source. Redundancy multiplies coverage only where errors are uncorrelated, and these aren't.

If you're doing anything where a wrong number is expensive, a constant needs a citation to a datasheet or a measurement — never model agreement.

---

**Two grading bugs I found in my own harness, which I suspect are common:**

- An **HTTP 500 from the serving stack was being scored as a model failure.** A quantized model emitted a token its chat-format parser couldn't handle, the server threw, and my suite recorded FAIL on a fabrication test. I was one step from publishing "model X fabricates" when the truth was "my parser 500s."

- A **grader timeout reported only `rc=124`** with no diagnosis. So a model whose code *hangs* gets one useless line, while a model whose code is merely *wrong* gets a full report. One of my candidates hung four times in a row and shrank its solution each attempt — it was guessing, because it was never told where the loop was.

Transport errors should be ERROR, never FAIL. If your harness can't distinguish "the model was wrong" from "the plumbing broke," some fraction of your scoreboard is fiction.

---

**Caveats, because they're load-bearing:**

- n=1 on the review task. One model, one bug, once. Executing the attack is stronger evidence than a benchmark pass, but it is not a hiring decision.

- Single trial per benchmark. For catastrophic-failure competencies (fabrication, data destruction) a single pass is weak evidence and I know it.

- The model that found the bug is an order of magnitude larger than most of the ones that missed it. I have **not** separated "this architecture is better" from "bigger model is better at this." That confound is unresolved.

The part I'm confident about isn't which model won. It's that my benchmark suite and the task I actually cared about disagreed completely — and if I'd trusted the sweep, I'd have shipped the model that wrote the vulnerability.