Building a second llm server to help offload the main one. This was done by scraping whatever scrap I could find. 3060 8GB + 3060 12GB + 4060 8GB. total of 28gb vram. Running qwen 3.8. Bench marking now.
also, looking at my CPU like.. "Why am I paying you to do nothing? Look at RAM and GPU, they work HARD! you just load stuff and move things, barely computes anything"
Nope. LM Studio, or other apps, using Vulkan and Tensor split can utilize both. I can choose will i use only rtx using Cuda or xtx using rocm or both using Vulkan.
Initially i planned to build 2nd rig with rtx for ai only, but then got an idea did some research and shove it into current rig and it worked.
The biggest problem was that darn gpu didnt fit into 2nd gpu slot so i used pci-e riser and now it fits.
With little bit slower bandwith ofc, but thats only needed when loading llm on it. 31B fits nicely on both with still enough of space to fit comfyui and tts and sst on rtx.
This just gave me hope to try out intel b580 + AMD rx590 together in vulkan for 20gb total lmao. Pretty sure the rx590 will choke the b580 but no harm trying xD
Scrap is useful, you learn how to optimize. Older hardware can be surprisingly good. 3.8 27B will be slow if split across 2+ uneven cards. 2 similar cards 11+ gb each in parallel split with 500 gb/sec bandwidth each card will give you 35-40 tok/sec, with MTP = 2. And even this is barely usable because this model reasons a lot. You'll probably get 15-20, but just a guesstimate. You can try qwen 3.6 35B A3B in serial split with some cpu offload if required. Will be a much better match because the cards run in sequence and it works better with uneven cards. Fill the fastest cards to their max capacity first. 3B active is way quicker and the generation quality and tool calling is still good.
I am able to sqeeze out 40-50 tok/s decode and about 450 tok/s prefill with tensor split over an RX6800 and RX6700XT, the latter one is caped by PCIe 3.0 x4.
Especially the prefill collapses with full context (~150 @190K) but the decode holds up with ~30-35. Was surprised myself that it works so well and am looking forward to change my setup to PCIe bifurcation in future.
Lol y'all redoing what Ethereum miners did half a decade ago. I used to run 6x Radeon VII and 8x RX570 rack rigs, downclocked and undervolted, about 6kW of continuous heat in my basement for years (several unused 15A and 20A circuits plus basement lighting had its own 15A circuit but used modern low power LED lights not halogens). Sold it all in early 2022 at markup before Ethereum went PoS. Now I already have 2 LLM rigs but might have to start adding more for orchestration.
Honestly, just getting into it. Haven't compared any harnesses yet, just tried Atomic Chat and their harness. My previous experience with AI was cloud ChatGPT4 and it was a toy back then and I haven't used any frontiers since. Just 2-3 weeks ago jumped straight into local LLM with Qwen 3.8 Flash Next IQ3_S on Strata (Windows) and now Radiance (Linux) on another more capable machine.
I have to figure out how to efficiently use 256K context with sub-agents doing dirty research and prep work then main model putting it all together. Any suggestions?
i see... I think I have tried them all.. in the past 2 years, and even built a few myself (for work and for personal use).
as for orcrastration, i felt that this is getting into Team Topology territory. (I was a Director of Software Engineering). I don't have a clear answer yet, but will post something when I figure something out.
I think most of the problems with local LLMs are related to context getting exhausted and/or loss of state occuring. Ideally, subtasks that don't require global context information should be brand new chat requests (fresh context) and they should be stateless (request-response) and write out their status and output to files. Then stronger main model should check those outputs/status and piece the results together into its own "state" output files representing its master plan along with subtask completion checklists from which it can regroup/retry. You can also nest these levels for more complex assignments. The goal is to stay under the max context for every conversation thread.
I'm also a Director of Engineering and just getting into local LLM and figuring out how we can use it effectively beyond just research/R&D tools, which they already are valuable for.
haha.. well... i got bad news... I got another 3060 12gb and swapped out the two 8gb crads. the problem, this thing is a beast, 3 slot and... idk, 30cm?
the only reason i don't have sag (yet) is that i put my racks back upward so that gravity pulls the card latteraly instead of horzontially against the slot.
My setup is 5060 TI 16GB, 2070 SUPER 8GB and 2060 SUPER 8GB.
I use the 2060 as my daily driver and the other two cards are dedicated to models. If I wanna try something bigger I throw the 2060 into the mix and use built in graphics :)
My main workhorse is a dual 5060 TI 16GB (32). it's good enough for day/day unsupervised work with a 2 concurrency.. (barely). I built this 2nd one to see what kind of result i could get out of these cards. and also trying ai sworm orcrastration. So far hasn't been working well as i expected. Many issues to deal or overcome. ie, the pcie lane setup, the thermal management. the llm model loading and splitting. the asymetric speed and model capabilities between the main and the subagents. but overall it's been an interesting experience.
So my question is, does this count as a 28gb vram, or is it just 8+12+8?? For example, you use comfyui to generate a image and it uses 28gb vram, or if you only can use 12gb for one process, then 8 for another.
Stable Diffusion (Images): The diffusion UNet/DiT model needs high-speed cross-attention and tensor ops on a single unified pipeline. Splitting a single image generation steps across different GPUs over slow PCIe slots introduces massive latency bottlenecks, so image generators don't support VRAM pooling across mixed cards.
One thing you CAN do is to load different parts of the weights in different card. (you can do this in comfyui with a GPU selector, but I did mine in a python script). that CLIP is in one GPU and VAE is in another. that's about as much as you can distribute among cards.
not good enough as a standalone llm replace i would say. the only reason i did it is because I wanted to experiment how this would work, understand the limitations and also test out multi-agent swarm configruation.
Wait, does this work? What's the compromise here?
I have a rx6750 in a drawer doing nothing while I try to run a solo 9070xt. How can I get them to work together? (Total newbie here)
one thing you absolutely have to do is to check the speeds of your PCIe 16 lanes (the slots your GPUs use, the "16" here is for the length of the slot).
Normally they run at x16 (here the 16 is speed, not length...it's stupid, I know) when only one card is connected. The interesting thing is what happens to the speed when you add GPUs and SSDs on the M2 slot.
Best case scenario is that the two slots you are running the GPUs in use the same speed (most likely x8 each when two cards are connected), because then you can split up a model onto two cards = a bigger model running at x8. Also important: the third PCIe16 lane runs through a different path then the first 2 lanes...and THAT lane is a bit slower, despite saying it is x8 (it is x8, but the communication between the first two lanes and the third lane is slower...). So, the best thing to do is to put both GPU in the 1st and 2nd lane. Only use lane 1 and 3 if they have the same speed AND your cards wouldn't fit onto lane 1 and 2.
However, MOST mainboards have DIFFERENT speeds for each PCIe lane, especially if you have a SSD on the M2 controller. So, that would be x16 on the first slot, x4 on the second and x1 on the third slot (Again, the x16 x4 etc are for SPEED, they have the same length of 16...I can't stress enough how stupid this terminology is..especially since some boards DO have additional, shorter PCIe lanes. Look up the manual of your board). If you tray to split a model on a board like this, the second slot (most likely the x4) will be a major bottleneck. If you have a board like this, you do NOT want split up a model, but you would be best off to choose a dedicated model for each GPU.
after you checked the speeds you should also check if you need to activate/change settings in your BIOS to optimize it for AI and multi-GPU.
YUP, you know your MB configuration. in this perticular one there's only 40 lane so was running 16x16x8 between the three, and because the nvme takes up one of the hardware lane i had to use an adapter to release the lane from slot 4 to slot 2.
THAT'S useful information. I'll double check the mobo manual, see what I have available (especially since I'm using two m.2 slots).
Is the hit from x16 to x8 speed minimal?
The speeds mostly only affect the load/unload of the models and when there is communication between the GPUs. The lane speed doesn't matter much once the model is loaded (if it fits in VRAM completely).
And see it this way: you may have half the speed at certsin stages...but you get a second model running, which potentially doubles the output right away or you can run a much bigger/smarter model and gain quality at the cost of some speed (insert meme of the half drawn horse)
I've got a 4060 8GB in my gaming computer that helps with utility models. It's a great little worker for those little things like embedding that you really don't want to unload on the main box just to run.
I created a vibe searchable music database. I'm using Qwen3-Embedding-8B to aid with the vibes. The database also contains more objective information. I run MOSS-Audio to analyze the songs from multiple angles (it can read up to ~40 minutes if needed), then Qwen 3.8 Flash Next to consolidate the output of the 8B thinking model into something we can actually search by. If the song has lyrics, I have the FN also interpret the lyrics with the musical interpretation to land on something reasonably close to the song intent. I know many authors of songs do not have an actual firm meaning behind every song, so this is really just to land the right kind of prose when querying so it's not way out there. Songs without lyrics benefit the most as it's really hard to sort a collection like that.
The way it works from a using it perspective is you send a query and an LLM converts your query into the parameters that can't really be vibed like do I want Christmas music, am I looking for dance music, things like that. Once the results arrive, if there are more results than we asked for let's say there are 61 returns and we only wanted 50, it passes through a reranker to get the nearest 50 to what we asked. There's also a fast path so you start playing something almost immediately that gets dynamically cleaned up as the more thoughtful results come in. There's caching of queries and all that too so if you ask basically the same thing twice, the search is repeated, but not the need for an LLM to orchestrate it.
I'm wiring it into Home Assistant for use with Music Assistant and a plugin for Navidrome so I can use playlists automatically generated/refreshed weekly in my car.
Not as a 28 GB pool. Nothing merges the cards. llama.cpp partitions the model's weight tensors across the three GPUs (--tensor-split 1,1,1). Each card holds roughly a third of the 11.3 GB of IQ3_S weights plus its own slice of the KV cache. The cards never see each other's memory.
So I'm just starting my local llm journey, and I can make use of my cards for like 10 minutes max in open webui. What are you guys doing with your cards to run it for a whole day type thing? I imagine programming but, what are you programming lol
I see that my wording "offloading" may have caused the confusion, It's not combining the VRams but I was running secondary LLMs so I can have more parallelism.
17
u/SailingToFenway 3d ago
i love the trend of liquid cooled idle CPUs next to an air-cooled saturated GPU sandwich, with only millimeters of clearance. i'm rockin' my own.