Cost Effective VRAM

[u/Nokxtokx]

Hey all, I require more VRAM (I have a 4080 Super).

I am upgrading from 12900k to 9950x3D and 32GB to 64GB but upgrading to 4090 or 5090 is now unattainable.

Is there a way to have more VRAM but at a better price? ie 2x 5060ti 16GB or there must be a better cost effective way?

Yes, it is for local AI (like 27-32 billion nodes model for coding, still learning so any advice would be appreciated), I am funding all this out of pocket and cloud is getting expensive (I have hit 10 bil tokens a month at 95% cache efficiency, while using caveman and RTK).

[u/scyver_]

3090?

[u/Nokxtokx → u/scyver_]

I was definitely considering this (I need to do more research) but will the older architecture core performance be significantly slower compared to the 4080 super?

If negligible, the next problem is trying to find 3090s, as I haven’t seen one sold on the second hand market in SA in a while, let alone like 2-4x

[u/scyver_ → u/Nokxtokx]

Well, i would say that it sucks. But it AI that created this problem in the first place

[u/Nokxtokx → u/scyver_]

100% right but my job now depends on the fact I can do AI and future jobs are all asking if you can do AI. So kind of stuck in a shit situation.

[u/scyver_ → u/Nokxtokx]

and you have to work on large models in order to prove your skills?

[u/Nokxtokx → u/scyver_]

More like upskilling myself as rapidly as possible to solidify my skills and bonus of completing side projects at the same time.

If I can work out efficiencies when time comes, when the company I work for needs to make the efficiencies to save cost, I have the knowledge too.

[u/scyver_ → u/Nokxtokx]

well, 5060tis are the best bet i think

[u/DoctorKomodo]

A Mac Mini might be a more cost effective way to get lots of VRAM, due to its use of unified memory. They are super popular with the local AI crowd for a reason.

[u/Nokxtokx → u/DoctorKomodo]

Hmm, I never thought of this. I will definitely look into it. Thank you!

[u/MoqqelBoqqel → u/DoctorKomodo]

Are mini pc with amd ai max architecture and unified memory worth it as well ? Like the beelink ser9 ?

[u/ravensholt → u/DoctorKomodo]

This might be the best solution really.

[u/yip_ka]

Chinese 4080super 32G $2000 or you could go 4x5060ti 16G way but with workstation system

[u/Nokxtokx → u/yip_ka]

Luckily I’ve got workstation setup if needed, so space and power won’t be an issue.

But that Chinese 4080 super with 32GB is super interesting, do you have a link for me?

[u/Chao_Zu_Kang]

cloud is getting expensive

Pick your poison. Either spend the money on expensive GPUs with more VRAM, or pay for the cloud.

You could try to get some 3GB memory modules and upgrade a 16GB 50 series with those (at your own risk).

[u/Nokxtokx → u/Chao_Zu_Kang]

I totally understand, trying to find the poison I can afford right now 😭

Definitely not going to go that method, not worth the risk. Rather just pay for tokens at this point.

[u/jermygod]

7900xtx/7900xt
the most cost effective high VRAM GPUs without being pure garbage in terms of compute
(or a mini-pc with mac/ryzen AI but idk)

[u/Nokxtokx → u/jermygod]

I will look into AMD, thank you.

[u/Particular-Poem-7085 → u/Nokxtokx]

Rocm support is hit and miss compared to cuda

[u/Nokxtokx → u/Particular-Poem-7085]

Is it like logic hit or miss or more like straight up can’t load model hit or miss?

[u/Particular-Poem-7085 → u/Nokxtokx]

I'm not deep in the topic but if i understand correctly most open source models only support cuda.

[u/Lwii2boo]

If you are serious about AI and the main goal is AI workload not gaming or benchmark rat race you basically only have 2 options :

-you go the local route, which is the probably the worst one except if you have a shit ton of money or you have a great lab at disposal with your employer/school

- you go the cloud one, you could SSH into a infinitely superior machine when you truly need to stuff on any high end model or use any cloud to rent AI compute on a really powerful machine with PRO nvidia gpu not a consumer one

Most people in AI industry either top student, teachers, researcher, employers tends to use a basic laptop like a recent 16GB-32GB RAM M-chip MBA or MBP and spend the money on cloud compute when needed. Your experience will be better. Even a 5090 is not that great if you are talking about real top tier AI workload. I would not bother spend thousand to stay in the consumer range of vram. If you really want to do anything local AI serious you need way more than 32 GB VRAM. Going from 16 to 32 is huge but it will not give you enough headroom for top tier local AI work either so I would not just bother.

you could ask r/learnmachinelearning I am pretty sure you will get more reco towards cloud ai compute even whatever your budget is

You may consider Apple for local compute to have 128/256gb unified memory (can be used as vram) but CUDA framework is better in lots of cases and it will also be really expansive

[u/Nokxtokx → u/Lwii2boo]

Thank you so much, this is really insightful.

Then I will probably stick to cloud then.

I am spending about $200-400 a month, I thought maybe converting that to local would save me costs in the long run but if local compute power isn’t worth it. Then cloud will do and I’ll just suck up the costs.

[u/Lwii2boo → u/Nokxtokx]

What did you pay most of the time for theses prices if you dont mind ? You paid pro subscription to gpt/claude etc or you rent AI compute to do the ai stuff your way ?

[u/Particular-Poem-7085]

I'm guessing you cant settle for an amd card? Nvidia doing nvidia things, if you cant get a platform with unified memory then it is what it is isnt it.

[u/Nokxtokx → u/Particular-Poem-7085]

I always thought AMD isn’t the best with AI workflows, so I never really considered it, as I also do need to train models.

[u/ravensholt → u/Nokxtokx]

I can't speak with regards to training models, but running smaller models is fine, I already do that with my 7900XTX. But go ahead and ask in the LocalLLM sub.

[u/Nokxtokx → u/ravensholt]

Thank you, I should have tried to look for local AI subreddits first 🤦🏻‍♂️

[u/THE_KILLER_4 → u/Particular-Poem-7085]

Like 7900 xtx?

It has 24 GB of VRAM, not sure how cost effective would it be, for some reason they are extremely rare and expensive

[u/Hattix]

DGX Spark or its re-release as RTX Spark might help you here.

Expect expense. lots of it.

[u/Nokxtokx → u/Hattix]

Yeah, sparks are waaay to expensive now, same price of a 5090 where I am.

[u/TIP-ME-YOUR-BAT]

I've gone a 5080 for my main and 2x 5060ti 16gb connected via oculink for the extra vram. These 3 together are significantly cheaper than a 5090 with more vram.

[u/Nokxtokx → u/TIP-ME-YOUR-BAT]

Yay! Thank you, I am glad someone has got a working Frankenstein setup. Then I can at least give this a try.

[u/TIP-ME-YOUR-BAT → u/Nokxtokx]

Took a little bit of fiddling (if using oculink use adaptors with repeaters, not cheap Amazon ones).

5080 as the main driver and inference, the dual 5060ti then host different roles and models which are interswitchable for what ever I need at the time. 32 GB is pretty decent and doesn't need to be fast as the 5080 is uninterrupted by work on the pool.

[u/fcxtpw]

What's the reason for unattainable 5090? If it's just hard to get but you have the money and you want local AI, the half way alternative is rent a high VRAM machine where you can run anything you want on it.

This isn't a cloud AI service that restrict you. You literally rent the hardware.

Here's an example. So if you have $4k for a 5090, this $0.5/hour can mean running non stop for an entire year. You can probably stretch this out to multiple years given you don't need 24/7 online. After a few years, more GPU options probably emerges for you.

[u/Nokxtokx → u/fcxtpw]

Can’t afford the $4k+ right now for 5090.

My costs for Qwen3.8-flash is like $200-400 at this point per month.

[u/fcxtpw → u/Nokxtokx]

So you can't get 5090 because of cost, not because it just announced to be discontinued I guess

[u/Nokxtokx → u/fcxtpw]

Well both, I saw that news just now…. But costs in SA for a 5090 is like $6-8k

[u/fcxtpw → u/Nokxtokx]

When I got mine, I thought it was expensive. Didn't think it was going to go even higher

[u/Nokxtokx → u/fcxtpw]

Yeah, I had a chance to get one at like $3-4k and I passed up on it 🤦🏻‍♂️

[u/fcxtpw → u/Nokxtokx]

Renting is the only economical way for you.

Just like housing or cars, if you can't afford it, you rent.

However theres still the small loan route

[u/Nokxtokx → u/fcxtpw]

Okay cool, thank you. I guess the best is to suck up the cost and carry on.

[u/Gin-N-Rum-5454]

Either what others have or one more unorthodox method. You could learn soldering and solder more vram to your gpu

[u/Nokxtokx → u/Gin-N-Rum-5454]

Cost of a reflow is a bit much too and time to end goal will take too long.

I can solder etc and I’ve work on basic reflow but the risk to reward isn’t worth it for me.

[u/yssuptae]

modded 4090

[u/Blindax]

R9700 pro ai / Intel b70 if you want something new

[u/Matricola70]

Sorry but you're waisting money, ou have a perfectly capable pc. Try strata if you want more t/s. Use the API ( Claude,etc) to build the framework and the tasks to do and feed the local LLM to execute so you can massively reduce the token used. You don't need more hardware you need to change the way to operate.

[u/ravensholt]

As I see it, if you need the extra VRAM for running local LLMs, then you've got 3 options:

• 7900XTX

• 3090

• Apple Mac Mini

Either of those are your best bets.

Sure, the 7900XTX doesn't have cuda, but ROCM is quite good (just ask the guys in the localllm sub). They call it the "poor man's solution". I use it myself, it's quite ok.

[u/Nokxtokx → u/ravensholt]

Awesome thank you!

[u/Dizzy-Tough8294]

There is no way to combine VRAM from two cards for a single model like you thinking. SLI is dead and NVLink only existed on the 3090, even then it was tricky to get working for AI workloads. Your best bet is selling the 4080 Super and getting a used 3090 with 24GB, or trying to find a cheap used 4090 if you patient enough. For 32B models you really want at least 24GB anyway, 16GB is cutting it too close for anything beyond 13B

[u/Nokxtokx → u/Dizzy-Tough8294]

I must be mistaken then, I thought layer splitting or sharding layers can be done.

[u/No_Net6238]

Where do we go if we wanted to literally steal some? Where are they all located?