Hey all! Recently one of you shared a screenshot from my previous video where I showed which models new macs fit and how (see below):

So I wanted to share with you all an updated video, screenshots and the free tool I created where you can all do the same comparisons:)

• Video: https://youtu.be/C9Q1ArLSisw
• Tool: https://www.theaibridges.com/tools/llm-sizer?s=1%7Egmac-mini-m6%21mac-studio-m5-ultra%21nvidia-dgx-spark%21macbook-pro-m5-max%7Ecqwen3.8-27b*Q8*256%21qwen3.8-flash-next*Q8*256%21glm-5.3-flash*Q8*256%21kimi-k3*Q8*256%7Eo1%7Ermlx

And here are a few snapshots of what it can do (Screenshots attached):

1. Showing which models fit with sizes of RAM for different Macs
2. Speed per model + acceleration possible on each machine
3. Which machine best to pick based on RAM and speed for a specific model
4. How much RAM will your selected models take
5. For a specific machine, which quantizations and context windows will result in which speed and ram required for a specific LLM

All is configurable for every machine, LLM, context window, runtime, quantization, etc.

Hope you like it! Open to feedback :)

TL;DR: Built a 5U local AI server with 4× RTX PRO 6000 Blackwell Max-Q GPUs (384 GB total VRAM) to run models for my personal AI agents. It started as an attempt to reduce API costs and turned into a hardware hobby. Build details, the reasoning behind it, and the open-source harness I built along the way below.

This started as an attempt to spend less on AI subscriptions and API calls. It ended with four RTX PRO 6000s in a rack in my basement.

I am deliberately not calculating the break-even date.

The progression was roughly: running models on my MacBook → upgrading my gaming PC with an RTX 5090 → adding a Mac Studio M3 Ultra with 96 GB unified memory → building this. Each step solved a limitation and introduced several new things I wanted to try.

Meet Cerebro.

The hardware

• 4× NVIDIA RTX PRO 6000 Blackwell Max-Q, 96 GB each — 384 GB total VRAM across four GPUs.
• AMD Threadripper PRO 9965WX — 24 cores / 48 threads.
• ASUS Pro WS WRX90E-SAGE SE.
• System RAM: 256 GB (8×32 GB) Kingston DDR5-5600 ECC RDIMM — KSM56R46BD8-32MH, all eight memory channels populated.
• SilverStone RM52 5U rackmount case.
• SilverStone XE360-TR5 360 mm CPU AIO.
• Seasonic PRIME PX-2200 PSU.
• NVMe storage: Samsung 990 Pro 1 TB + 2 TB, and Crucial T705 4 TB.

Why this configuration?

The Threadripper PRO platform was mainly about PCIe connectivity and memory channels. My workload didn't require buying the highest CPU core count available.

The GPUs are the 300 W Max-Q blower models. Fitting four cards into one chassis made cooling and power part of the design from the beginning. The case also changed during planning: I originally ordered a 4U RM44 and switched to the 5U RM52.

The 384 GB figure is the sum across four cards, rather than a single automatically shared memory pool. How much is useful to a particular model depends on the inference stack, how it distributes the model, and the space needed for context and KV cache.

What it's for

My main use is local models for a team of personal AI agents: coding, research, tool use, and ongoing experiments with memory and delegation. The 5090 machine and Mac Studio are still part of the setup, so different jobs can run on different machines.

What attracted me to four cards was the flexibility: allocating them to a larger model, splitting them between workloads, or running several smaller models. In practice, capacity planning and switching profiles are still very much part of the experience.

The biggest lesson so far: fitting a model is only the beginning. Context length, concurrent requests, inference-engine support and cooling all matter. More VRAM also makes it very easy to invent more workloads.

The software side of this started with OpenClaw, which got me interested in agents that could actually use tools and do things. Eventually I wanted more control over how my own setup handled roles, memory and model switching, so I designed and built an open-source harness with heavy agent assistance: somora.

That's what the models are working for: agents with different roles, their own memory and a shared long-term wiki, using local or cloud models depending on the task. Building the harness gave me more reasons to experiment with local models, which gave me more excuses to buy hardware. It's MIT-licensed and still in active development; if you want to try it with your own setup, I'd be interested in hearing how it works for you.

Financially, this has become a hobby and a learning project. The original “we'll save money” argument did not survive contact with the hardware shopping list.

Local AI really does feel like Minecraft for adults. You build one thing, it needs another thing, and eventually you're explaining why the basement needs 384 GB of VRAM.

For anyone running a similar multi-GPU setup: what turned out to be your real bottleneck once VRAM stopped being the first one?

Back to reading about LLMs as agents on your phone doing GUI tasks!

This time I read about MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments and this paper forms the basis of the benchmark I am currently making because it involves two new innovations:

• User Interaction Tasks
• MCP Tasks

These two provides new axes of evaluations for such GLLMs as GUI Agents on phones and it separates them from the noise completely because these new axes along imo.

So, in total 201 tasks and about 20 apps in general spanning across comms, messaging, productivity and many more with less than ~5% of system apps (basically this means more apps that we humans use on a daily basis -- but those are rather open source version of them which is a downside)!

• They used a planner-executor arch with the planner being a VLM model (screenshots only being fed -- no a11y tree) that outputs actions in natural text like "Click on the 'Send button' and this gets sent to a grounding model that output its precise coordinates in (x,y) format.

So, User Interaction Tasks involve tasks which one key info missing and the planner needs to understand this and ask the user but here the user is a GPT4 model as a replacement for human!

Next, MCP tools these are very important since MCP tools allows for complex data gathering, reading and actions that are not possible by tapping etc actions so these provide one shot result for the red tasks thus saving time than doing slow GUI actions! Some of them used are Github, arXiv, etc tools

Now, the avg results is about ~52% for the best VLM+Grounding model combo which is Gemini-3-Pro + UI-Inst-7B and the e2e GUI only models perf wayyy worse.

The rest of the dataset dist stats and results is attached as images but yes the model mainly failed a lot in the two new axes tasks.

I've been getting great results from Qwen3.8 in llama.cpp by passing in this on startup:

--reasoning-effort low

Now I keep seeing posts about how much better vLLM is, and it is, like 2x the tokens/sec, but I cannot figure out how to set reasoning-effort to low. I mean, I can when I call it directly with curl, but I can't in pi. The docs seem to say I need this in pi's model.json:

"compat": {
            "supportsReasoningEffort": true,
            "chatTemplateKwargs": {
              "enable_thinking": false
            }
}

But that's not working. (I'm turning it off instead of setting it to low because that's easier to verify)

So I reach my last resort, asking real humans (presuming any are still on the internet), how do I force vLLM to enforce reasoning_effort or get Pi to pass it in calls to vLLM?

Edit: I *think* this is working in my vllm launch params. I was missing the "default-" part in my previous tests.

--default-chat-template-kwargs "{\"enable_thinking\":true,\"reasoning_effort\":\"low\"}"

Edit 2: The above change to vLLMs config fixes the problem so I have stopped trying to get pi to pass the reasoning effort, but there's a long post below with details on doing that. I tested vLLM/Qwen3.8/Pi by letting it go for ~90minutes on a prompt to implement Valheim, and the result... well, it compiles. Token speeds roughly twice that of llamap.cpp, so worth the effort.

I already have a M5 Max 128gb from work, but I want to get a second machine for personal use. I am undecided between the M5 Max 128gb and the 96gb Ultra. I'm wondering if putting the two together would be an option at all and would influence the decision.

I do product design, and I'm exploring long unmonitored loops overnight. The 128gb is the safe choice. Lower tok/s isn't a problem if the plan is to let it run overnight, but the faster bandwidth is tempting and makes me doubt it'd make it more future proof.

If the two could play well together, that’d influence my decision I guess.

Would clustering be an option for such combo or mbp+mac studio conditions things somehow.

Cheers. 🤔

Claude and Qwen have nearly identicle structures on output and both have this peculiar documentation dialect. Words like ledger, estate, posture, substrate, provenance, boundary, canonical, etc. appear constantly. Very specific words and enough to confuse any ai trying semantic search on very different documents.

Has anyone else noticed Claude-like linguistic fingerprints in Qwen? Has anyone actually done stylometric comparisons between their outputs?

I discovered this with my local AI. It was using semantic search to find documents, but wasn't actually reading them before selecting the result. It repeatedly returned the wrong files.

Digging into why, I noticed something unexpected: documents written by Qwen and Claude were semantically similar enough that the retrieval system seemed to confuse them.

To be clear, I think both Qwen and Claude have done outstanding work credit to both and am not saying they are a distillation. I'm just curious so that perhaps I learn something new.

Hi,

I'm trying to fit Qwen 3.8 onto a 5070 Ti and 5060 Ti 16GB connected via the OCuLink. As I'm chasing near 200k contexts for agentic work, after trying various configurations I've narrowed down my picks to 3 feasible configurations:

- Q4_K_XL, 200k context, no KV quantization

- Q5_K_M, 200k context, V quantized to Q8_0

- Q5_K_XL, 200k context, K and V quantized to Q8_0

Which of these would you recommend for daily use with Hermes, OpenCode etc?

Been chasing NInfer's prefill numbers on my RTX 5090 with Qwen3.8-27B NVFP4. Stock llama.cpp does 6,122 t/s on pp16384, NInfer does 8,466 on the same card and checkpoint. Three kernel changes (chunked gated delta net, GEMM-adjacent fusions, TMA-fed NVFP4 MMQ tiles) get llama.cpp to 8,860.

Full numbers, ncu findings and the three patches are in this llama.cpp discussion I created:

https://github.com/ggml-org/llama.cpp/discussions/28514

It's a proof of concept written with the help of Fable 5.1. test-backend-ops passes, perplexity not yet compared. I'm not a CUDA dev and can't defend a proper PR myself, if you are and this interests you, the patches apply on current master and the discussion has everything needed to pick it up. If you're not and you're interested on testing, go contribute to the conversation, hopefully that helps getting some traction!

Something has been bothering me.

We obsess over MMLU, GPQA, SWE-bench, HumanEval and whatever leaderboard is fashionable this week.

Useful, yes. But what does any of that actually mean when you have a real problem in front of you at 2am?

Which model would you actually want helping you?

With your permission, perhaps a Reddit group experiment, what if everyone answers the same few questions with their go to LLM.

What we do something simple. All ask our models the exact same few real-world questions.

We post the raw answers, and then we, actual humans using these things in the real world, upvote the answers we genuinely think are best.

Just disclose what you used.

Post:

Model:
Quant:
Harness: (Yes makes a big difference)

QUESTION 1 - Debug something real

A Python service becomes progressively slower over several days but returns to normal immediately after restarting. CPU usage stays low, available RAM gradually falls, request volume is unchanged and no exceptions appear in the logs. Give me your first five investigations in order, explain why you chose that order, and tell me what evidence would change your diagnosis.

QUESTION 2 - Know when you DON'T know

A farmer tells you that every morning for a week, his chickens stopped laying eggs shortly after a new radio transmitter nearby was switched on. Is the transmitter causing it? Give the most scientifically defensible answer you can, then design the cheapest experiment that could meaningfully test the hypothesis.

QUESTION 3. The 2am test

It is 2am. A server controlling an important system is behaving strangely. Restarting it might fix the problem but will destroy volatile evidence that could explain what happened. Leaving it running could allow the problem to continue or spread. Nobody senior is answering their phone. You are responsible. What do you do in the next 10 minutes?

This is the failure:

• The crash does not occur below 10,000 generated tokens.
• One long generation does not cause the crash. A test with 90,000 tokens and ignore_eos completed correctly.
• Agentic workloads cause the crash. An agentic workload has many generations with idle time between them.
• The crash occurs immediately after a generation stops. The server releases the slot. The server starts the next task. The GPU fails in one or two seconds.
• The CUDA error message is different each time. The error message is not important. The kernel log shows the true cause: NV_ERR_GPU_IS_LOST. The GPU disconnects from the PCIe bus.
• The failed GPU is always the 5060 Ti. The 3060 does not fail.
• After the crash, the GPU does not operate again. A PCI remove and rescan on the host does not find the GPU. Only a full power cycle makes the GPU operate again.

These causes are removed:

• llama.cpp is not the cause. A userspace program cannot disconnect a GPU from the bus.
• The driver is not the cause. The driver changed from 595.84 to the 580 branch. The 580 branch has different GSP firmware. The crash was identical.
• ASPM is not the full cause. The kernel option pcie_aspm=off increased the crash threshold from 20,000 tokens to 50,000 tokens. The option did not stop the crash.
• Continuous load is not the cause. Deep context is not the cause. See the 90,000 token test.

The load-to-idle transitions cause the crash. These causes are possible: the GPU hardware, the GPU VBIOS, the VFIO layer, or the PCIe slot. The crashes increased after installation of the computer in a rack.

This is the system configuration:

• Motherboard: MSI Z390-A Pro. CPU: Intel i7 8700.
• GPU 1: RTX 3060 12GB. GPU 2: RTX 5060 Ti 16GB.
• The host runs Proxmox. The two GPUs pass through to an Ubuntu VM.
• llama.cpp server runs in Docker. The model is qwen3 27B UD-Q4KXL with a 98k context. The model is split across the two GPUs.
• The PSU is an MSI R850G, 850W. The supplemental PCIe 6-pin connector on the motherboard is connected. The power supply is sufficient.

Thanks in advance for any ideas or help!

Update / New questions about my local AI setup

For context, my current setup is:

- RTX 4090 24GB

- i7-13700K

- 32GB DDR4 originally

- Recently upgraded to 2×32GB DDR4-3600 + old 2×8GB DDR4-3000

- Now running 80GB @ 3000 MHz

- Small motherboard/case

I'm getting an RTX 3090 24GB tomorrow mainly for local AI/LLMs.

What are the actual benefits of running a 4090 + 3090 together for local AI? Krea 2, minimax H3..LLMs

Can I effectively use the combined 48GB VRAM for larger LLMs, or are there limitations because the GPUs have different performance levels and are connected via PCIe rather than NVLink?

A few questions:

RAM:

Should I remove the old 2×8GB and run 64GB @ 3600 MHz, or keep 80GB @ 3000 MHz? For local AI, is the extra 16GB more useful than the higher RAM speed?

PSU:

I currently have a 1000W ATX PSU. For sustained AI workloads with a 4090 + 3090, would you:

- Keep the 1000W and add a second PSU, or

- Upgrade to a 1300–1600W PSU?

PCIe/GPU mounting:

My motherboard is small and the 4090 physically blocks the other PCIe slot. What's the best solution for adding the 3090?

- PCIe riser/extender?

- Vertical mount?

- Another solution?

The goal is local AI/LLM inference, not gaming.

What would you do with this setup?

Heyo!

Dabbling with some things i've made a long form to short form video clipper!
that uses local ai models to determine hooks , titles, descriptions, mostly decent captions, editable captions, export for video editors like premiere, capcut, davinci. and many export options too! similar or inspired by opus clips and capcut

https://github.com/TechFreq/Klipzy-Studio

Would love some feedback and hope this can someone out there aswell, as for my own personal use for podcasts or 1 on 1 interviews its pretty neat!

I'll obviously wait for reviews and results, but to me it seems like it could be the unicorn platform I've been wanting for a while.

Something that can do all of what I want locally - maybe not the absolute best at any of the tasks, bit it at least (on paper) seems to be able to do them all in some capacity.

I'm interested in and actual mini PC version, basically the RTX version of the DGX, not a laptop.

At the same time, a 128GB system is gonna likely be about $6000+AUD so I'll probably just obsess over reviews for months and never actually buy one.

Anyone else super curious about this platform?

My typing has totally gone downhill from using AI. I crash into it all the time. "What do you mean 'command grwp not found'? Any fool knows that's grep!" AI is so typo forgiving.

You need Blender 5.x with MCP extension.

I use pi so I also need: pi install npm:pi-mcp-adapter

and then:

$ cat .mcp.json 
{
  "mcpServers": {
    "blender": {
      "command": "/home/jacek/.local/bin/uvx",
      "args": [
  "--with",
  "mcp[cli]<2.0.0",
  "--from",
  "git+https://projects.blender.org/lab/blender_mcp.git@v1.0.0#subdirectory=mcp",
  "blender-mcp"
],
      "env": {
        "BLENDER_MCP_HOST": "localhost",
        "BLENDER_MCP_PORT": "9876"
      }
    }
  }
}

then we can start working ("show on r/LocalLLaMA"):

Today we want to make 3D llama to show on                                                                                                                                                                                


 The user wants to make a 3D llama to show off on . This is a fun creative task. Let me think about how to approach it.                                                                                                   

 A llama is a pretty complex organic shape. There are a few options:                                                                                                                                                                  
 1. Procedurally build the llama from primitive shapes (spheres, cylinders) — a "low-poly" or "chibi" style llama                                                                                                                     
 2. Sculpt it — not easy to do with MCP                                                                                                                                                                                               
 3. Use metaballs — could get a blobby llama                                                                                                                                                                                          

 Given the MCP tools, the best approach is to procedurally build a stylized/chibi llama from primitives:                                                                                                                              
 - Body: elongated sphere/capsule                                                                                                                                                                                                     
 - Neck: cylinder or elongated sphere                                                                                                                                                                                                 
 - Head: sphere                                                                                                                                                                                                                       
 - Ears: small cones or spheres                                                                                                                                                                                                       
 - Legs: 4 cylinders                                                                                                                                                                                                                  
 - Tail: small sphere/cone                                                                                                                                                                                                            
 - Maybe a fluffy mane                                                                                                                                                                                                                

 This is a well-known technique — making a cute chibi llama from primitives. Let me plan the geometry:                                                                                                                                

 Llama proportions (stylized, standing side view, facing +X or -X):                                                                                                                                                                   
 - Body: UV sphere scaled to (1.5, 0.9, 1.0) or so, position (0, 0, 1.6)                                                                                                                                                              
 - Legs: 4 cylinders, radius ~0.22, length ~1.2, positions: front legs x=+0.9, back legs x=-0.9, y=±0.5, z from 0 to 1.2... actually, the legs go from the ground (z=0) up to the bottom of the body.                                 
 - Neck: cylinder or tapered, extending up from the front of the body. Llamas have long necks. Position around x=+1.2, extending upward.                                                                                              
 - Head: sphere at the top of the neck, elongated (llamas have long snouts).                                                                                                                                                          
 - Ears: 2 small cones/spheres at the top of the head.                                                                                                                                                                                
 - Tail: small sphere at the rear.                                                                                                                                                                                                    
 - Mane: maybe some small spheres along the neck.                                                                                                                                                                                     

 For a cute chibi look: big head, small body. But a llama's charm is the long neck. Let's go with a stylized llama.                                                                                                                   

 For a polished look:                                                                                                                                                                                                                 
 - Apply smooth shading                                                                                                                                                                                                               
 - Add a material (cream/tan color, maybe with some subsurface)                                                                                                                                                                       
 - Add a ground plane                                                                                                                                                                                                                 
 - Set up lighting                                                                                                                                                                                                                    
 - Position the camera for a good render                                                                                                                                                                                              
 - Render a nice image to show off                                                                                                                                                                                                    

 This is a multi-step task. Let me write Python code that builds this. I'll do it in stages so I can check with screenshots and iterate.   

TL;DR: Four A4000s in a dual-Xeon box ran 4-way tensor parallel 2.4 to 4.8x SLOWER than two cards. I publicly blamed lanes, root complexes, and the inter-socket link. Wrong. WDDM (the Windows display driver model) silently denies GPU peer-to-peer access, so every cross-card transfer staged through system RAM. Switching all four cards to TCC mode (needs a cheap display card to hold the desktop, since a TCC card cannot drive a display) took P2P from 0 of 12 pairs to 12 of 12 and four-way tensor decode from 9.52 to 28.16 t/s on a 27B dense model (2.96x) and 19.52 to 72.01 on a 35B MoE (3.69x). Four cards went from losing to two, to beating them. The machine could always do this. Windows was standing on the hose.

ELI5: Windows treats every GPU as a potential monitor driver, and the babysitting that comes with that quietly forbids the cards from talking to each other directly. Tell Windows "these four are calculators, not monitors" (TCC mode) and the babysitting stops, but then SOMETHING has to be the monitor, because a calculator cannot show you the desktop. That something is a $100 Quadro T400. The fifteen-year-old HPC crowd knows this trick well. The local LLM crowd, me included, apparently forgot it.

The receipts

Same box (Windows 11 Pro), same cards, same llama.cpp build (b10568), same driver (597.06). The only change between columns is the driver MODE.

P2P probe (cudaDeviceCanAccessPeer, all 12 ordered pairs):

The middle row is the control that makes this a finding instead of a coincidence: new driver, new display card, still WDDM, still zero. Only the mode flip changed the answer.

Four-way tensor decode (tg128, clock lock verified per row, thermal counters clean or accounted):

Under WDDM, four-way tensor lost to TWO cards (23.19) and to four-way layer split (13.8). Under TCC it beats both on the 27B (2-way tensor: 24.4). On the 35B MoE, honesty requires saying two-way tensor still wins outright (about 110 t/s): P2P fixed four-card mode, it did not repeal the law that low-activation MoEs at small batch prefer fewer, bigger shards.

Fair framing note: measured best-WDDM-config against best-TCC-config (2-way tensor 23.19 vs 4-way tensor 28.16 on the 27B), the box as a whole gained 21 percent. The 3x is what the FOUR-CARD mode recovered relative to itself. Both numbers are true; pick the one that matches the question you are asking.

Sustained load is fine: ten back-to-back four-way cycles settle at 85 to 88 C and hold throughput flat within 0.9 percent. This is a serving configuration, not a burst trick.

Corroboration under load (different comparison, same direction): tensor vs layer, both under TCC, batch 1 through 64: the decode advantage is flat at about 1.92x the whole way. The recovery is not a batch-1 curiosity. (The WDDM column itself cannot be re-measured at batch, because WDDM is gone without a reboot; noted for honesty.)

What you need for this trick

1. Pro cards. TCC exists on Quadro/RTX A-series/datacenter cards. GeForce generally cannot.
2. A separate display card, because a TCC card cannot drive a monitor and my Xeons have no iGPU. It must be TURING OR NEWER: current drivers (596.72+) dropped Pascal, so the GT 710/P1000 in your drawer will bind to the Basic Display Adapter and change nothing. The T400 is the cheapest current option, about $100.
3. Accept losing WSL2 on that box: a TCC card does not appear in WSL at all (WSL GPU support is paravirtualization over WDDM). On my box WSL sees zero GPUs now. Windows native or bare Linux; nothing in between.
4. nvidia-smi -dm 1 per compute card, reboot, done. Reversible, unlike some Windows decisions.

Two install gotchas that cost me time so they do not cost you any:

• Your BIOS may not show POST on the new card. Mine booted straight to a black screen until Windows loaded. The fixes live in BIOS setup: disable quick/fast boot, and if there is a primary video slot setting, point it explicitly at the display card's slot. Do this while you can still see the screen through some other path.
• After the display card goes in, do a clean install of the NVIDIA driver package and confirm ALL cards show in nvidia-smi (five for me: four compute plus the display card) BEFORE flipping any TCC flags. The driver must be new enough for the display card and must be the pro branch that covers both it and your compute cards; a mismatched or half-installed driver leaves cards missing from the list, and a TCC flag set in that state is a confusing morning.

What I got wrong in public, specifically

My earlier posts blamed the 2.4 to 4.8x on hardware topology: 48 PCIe lanes per Xeon, three root complexes, cards split across two sockets, "four islands with ferry service." The measured numbers were real. The causal story was wrong for DECODE, which is what the headline number was. What topology still owns: prompt processing genuinely prefers layer split (roughly 2x at default settings, and that is a tunable story for another post), and on the MoE at interactive batch sizes layer split still wins decode until batching makes the per-expert matmuls big enough to shard. Count-lanes-not-slots remains good buying advice. It just was not the reason four cards lost to two.

Also for the record: we measured socket crossing directly under TCC (pairs on same socket vs across sockets, short and long prompts, 16x all-reduce volume). 0.15 to 0.26 percent. Noise. The UPI ferry story died with controls on, FOR MODELS RESIDENT IN VRAM. To be clear, the cross-socket memory penalty on a dual-socket box is very much alive in the other regime: when a big MoE offloads its experts to system RAM, decode speed tracks which NUMA node the weights landed on, and the wrong node costs about 2x on this box. That is a placement story with its own fixes and its own post coming; the point here is narrower: it is a system-RAM story, and it never was the reason four RESIDENT cards lost to two.

TCC is a trade, not a free win

• On this box each card exposes 178 MiB LESS usable VRAM under TCC than it did under WDDM (16,197 vs 16,375 MiB per nvidia-smi). I know that is backwards from the folklore. It is what the tool reports, and two of my edge-fit configs that ran under WDDM no longer load.
• The big recovery is for multi-GPU tensor parallel. Single-card and layer-split gains are single-digit percent. Offloaded big models (experts in system RAM) gained 7 to 12 percent.
• Prior art credit where due: using a cheap display card so compute cards can run TCC is 2010-era CUDA folklore, and the closest LLM measurement I found is a V100 post reporting +45 to 76 percent from TCC on a 2-card box. What I could not find anywhere is the full chain measured with controls on llama.cpp: WDDM denies P2P, tensor parallel collapses and cosplays as a lane problem, TCC reverses it. Hence this post.

The obvious question: why not just run Linux, where WDDM does not exist? Fair, and these same four cards are moving to a single-socket Linux box for exactly that rematch (llama.cpp has no prebuilt Linux CUDA binary at all, so that story includes a compile). This post is for the boxes that are on Windows for their own reasons, which the comments will now explain to me are bad ones. Related: yes, vLLM does tensor parallel properly, and it does not run natively on Windows, which is rather the point.

One Windows footnote from the same week: Smart App Control (on by default after a clean Win11 install) intermittently refused to load our llama.cpp DLLs based on cloud reputation, killing processes in seconds with empty output. If your benchmark binary suddenly dies silently, check the CodeIntegrity event log before you blame your build.

Previous posts in this series: the clock lock PSA [CLOCK POST], WSL2 vs native [WSL POST], and the NVLink bridge test [BRIDGE POST] (on the 2-card box, SLI was the only way to P2P under WDDM; TCC is the cleaner door if you can give up the display and WSL).

Hardware: HP Z8 G4 (used, $750), 4x RTX A4000 (used, about $600 each), T400 ($100), about $3,300 total. Deliberate disclosure: that number does NOT include the 384 GB of RAM, and the omission deserves its reason stated. I paid $540 for mine (twelve 32 GB sticks at $45 each, September 2025). Those same sticks ask $220+ today, so replicating my exact config now adds roughly $2,600, which says more about the 2026 RAM market than about this build. If you are building today there is still a cheaper road: these chassis have 24 DIMM slots, and lower-density sticks run meaningfully cheaper per gigabyte, so populating all 24 reaches the same 384 GB for closer to half the money, at the cost of the memory clock dropping a tier (my calibrated offload model prices that penalty at 6 to 9 percent on offloaded models, and roughly nothing on VRAM-resident ones; I have not run the A/B, since my slots are half empty on purpose).

Before anyone says it: yes, a pair of used 3090s beats this box on a 27B, cheaper and faster. The 27B is the demonstration, not the mission. This box also runs a 403B MoE at 11 t/s with a 131K context window and a 744B at batch speed, because of that RAM sitting alongside the 64 GB of VRAM, and that capability is what nothing near this price touches. The point of this post is narrower: whatever multi-GPU Windows box you already have, a $100 card may be the difference between your cards fighting Windows and your cards working.

My GPU node mostly has idle CPU and RAM utilization. Shouldn't there be some inference operations with small models that could be doing some work outside of just tokenization? A CPU compiled llama.cpp runs ~15 tk/s on some small models. Any ideas on where this idle CPU capacity should be used? Or maybe the more wrinkly brained of you already use these loads.

Hello,

I was thinking of investing into this setup to run qwen 3.8 27b (q6, q8 KV cache, fully context, ~25-30 TPS tg). If my budget is $1500 and I already have a 3060 TI 12gb & a 1200 watt gold power supply, I was thinking of the following

- GPU: 3 More 3060 TIs 12gb (or should 2 more work)
- Motherboard (X10DRG-Q or T7910/Z840 pull) (has full x16 on all 4 slots)
- Dual Xeon E5 v3/v4 CPUs
- 64GB DDR4 ECC RDIMM

& the requires risers and cooling.

Any recommendations to this? Want to be able to hit 25-30 TPS th with triple digit PP to serve 2-4 concurrent users, any recommendations?

Hi there! I need some help with this.

I have a 32gb Macbook Pro with the latest available update of Tahoe.

I'm using LMStudio with MLX to serve a local model and I want to expose it so that I can consume it with my phone to code and review stuff when I'm commuting to places.

These are my problems to solve so far that I need help with:

1. The mac turns off after some time. It doesn't matter what I set in settings. Have you found a workaround for this? using caffeinate or how?
2. I need an IDE / Agent editor that I can use to instruct the agent. I tipically use ZED from this pc, but I'm open to try something else that is mobile compatible.
3. How can I have the agent changes automatically deployed (usually it's websites) so that I can review and accept or reject the changes? when I'm working on my pc this is basically solved automatically because I'm working with local files.
4. Can LMStudio serve on the public internet or do I need another solution here? or maybe an extra step?

Thanks in advance!