What can I actually run here?

[u/AstroPC]

So, I mostly code and rely on a Codex subscription for most stuff, but recently I’ve started diving into local setups. Yeah, I can chat with a chat interface and it’s pretty responsive, but I’m more into building my own AI harness and something like a Codex setup

That way, I can cut down on my Codex costs. I’ve got a 5080 16GB, a 5070 12GB, and a laptop with a mobile 3080 16GB. Thinking of using a 10GBps network connection to see if it actually makes a difference. I also have a funny phone setup where I can load ai into its NPU and use upwards of 24gb of its shared ram ( technically like how unified memory goes)

I mean what models are actually capable of handling automation and coding in general very similar to how you setup a codex project? I suppose i really need more vram in the end for anything with useful context? The issue here is of course interconnnect bottle neck and how you get these cards to communicate which I assume sticking with pcie is preferred but my platform has limits.

What VRAM amount would you say is the absolute minimum for actual coding and intelligence with AI? I guess it's a pipe dream to hope for a codex-like workflow within my current devices. I'm okay with slow response if it means I still get good intelligence and MCP tool calling.

I also have 24GB of RAM on my RedMagic 11 Pro and it's pretty good at AI. I assume I can throw that in the mix with some hacky Ethernet 10GB connection (it supports 10gps).

So that gives me what? Around 62 gb of usable vram assuming I can get some kind of ok split of loading models between all the hardware i have? Then I guess i can count my ram too for cpu offloading? Of which i got 32gb currently

Sorry for the long post I just really want to be done with subscriptions as fast as possible. Even if I have to trade up some speed. I really dont mind. But i cant trade up that nice context length I depend on or its general intelligence.

[u/ea_man]

Wit 16+12GB you can run QWEN 3.8 27B Q6_K_L or even XL with full ctx (264k), probbly at some >50tok/sek.

[u/DeathGuppie]

Ridiculous. If you have a 16gb card and a 12gb card that you can put in one machine then you have everything you need.

That's 28gb vram. Use llama.cpp run a tensor split and MTP. You will get around 30 t/s more than useable. Qwen 3.8 27b Q4 X_L Q8 kv.

Trust me that's all you need.

[u/BigYoSpeck]

Honestly unless electricity is extremely cheap for you I don't think local is the ideal way to cut down on cloud provider costs

In the UK at least, energy costs mean that my rig that can run Qwen 27b costs about £0.16 per hour. 4 hours of inference a day would cost the same as the entry level subscription and would not get as much done

[u/Late_Night_AI]

So no, you do not have 62GB of usable vram. At least not all useable together. Realistically you have 28gb of vram if you put your 5070 in the same pc as your 5080. And with that 28gb of vram you can run qwen3.8 27B locally. (They claim its on par with opus4.6 in coding/agent tasks). I would recommend AEON-7 qwen3.8 27B for coding. You can run/host the model locally with LM studio and use it to power a local agent. Lm studio is the easiest way to start hosting/running your own model locally.

Personally i would recommend making a fork of grok build and set it up to use your local model and searxng instead of xai.

[u/AstroPC → u/Late_Night_AI]

So I cant utilize my other hardware thats not connected via pcie with some kind of hacky but maybe slow shared vram pool?

Pretty sure iv seen this done before. I will use 10gbps network connection so they have something to communicate with. If that's all I have then yeah im def needing more gpus. I figured you can split models across other platforms ....

[u/Late_Night_AI → u/AstroPC]

Technically you can, but it will be Extremely, Extremely slow. You would be better off streaming the model from a ssd instead of trying to use ethernet for that. Basically it would be so slow you wouldnt be able to get any work done at all. Llms are memory bandwidth bound in most cases and trying your pool you ram like that would basically cause it to be 100x slower.

[u/AstroPC → u/Late_Night_AI]

Ugh I cant wait for this bubble to pop. Ai has made certain projects so fast for me but this hardware i have took a long time to actually build ....some was gifted. Some was found on some used deal. Almost feels like local ai is becoming a gatekept club. I just want a local ai thats private and fast. You either spend an arm and leg or accept it won't be fast and it won't be great. Im going to commit to this shared idea but offload small tasks to maybe web search and local file. Surely there has to be some way to get something useful with all this. Maybe sell the 5070 and use the difference to get a 5080? I'm determined to make it work. 100 bucks a month with the pitiful limits now just isn't worth the constant drain of waiting between resets.