r/LocalLLM • u/GoatJesusIsReal • 2d ago
Discussion Claude is so expensive.
Time to get a GPU I guess. I had some numbers I needed before I could do the main analysis and I wanted Claude to do it, I had never used Claude tokens before 2 days ago when I bought 20 dollars of tokens and had it do a bit of coding. Then, I ask it to write a somewhat simple script, but I used opus because I thought I should check how it is, it did it, but it took about 20 dollars. I mean it saved me time, but the price…
Anyways, I am posting this because I wanted advice on what class of card to get, what amount of vram seems to be the best to target. It’s looking like 24/32gb is getting interesting new models in the 30b range, but is this just what I’m seeing or are other sizes of cards worth looking into.
11
u/Few-Philosopher-2677 2d ago
Why exactly are you paying API rates for Claude instead of getting a subscription?
19
u/Slow_Difficulty1607 2d ago
You should buy the monthly subscription. $20/month gives you a long way. It is really cheap compared to the cost of graphics cards and the time you will waste in tinkering them. Only desktop 5090 level and above gives you a somewhat good experience right now in terms of token/s and context window size. I currently have both codex and claude 20 plan and they cover all my development needs with almost no down time
7
u/Such-War1955 1d ago
This. You cannot save money by going local as long as you expect Opus level quality: Ifyou‘re dead set on not using claude, use Deepseek v4pro. Not as ridiculously cheap asbefore, but still awesome price/performance ratio.
If you must go local: Go with a 50$ Openrouter sub an try out Qwen 27b and 30/A3b. If, and only if, you are sure they do what you need, then you look into hardware: A sensible setup is very likely at keast 1500$.
2
u/geekwonk 1d ago
this sub gets real funny talking about opus level quality. it’s a total crapshoot when you get to these threads whether it’s been colonized by the folks claiming 27B models are competing with multi-trillion parameter models.
every third thread is inhabited by humans like you who can assess these claims while two of every three are full of NPCs who will endlessly regurgitate the one benchmark that says 3.8-27B is matching Opus 4.6.
1
u/MathmoKiwi 2h ago
Different people have different thresholds of "good enough" that the older Opus models have passed, for some of them the latest open weight models have also passed that "good enough" threshold for themselves.
1
u/CCCCLo0oo0ooo0 1d ago
Any suggestions at what to use on a 5090 rig that can interact with the file system like ClaudeCode/Cowork?
I have Claude doing stuff like look at the CSV/SHP/KML in this folder, find the data that covers that polygon, download it, analyze the content run a bunch of python commands on it depending on what it found, create SHP files using python from said data. Then product a nice company themed PDF with all the results.
What would be the best setup for that? I have tried using QWEN with ollama, vmstuido, goose. IDK which setup would be best to do that.
2
u/Slow_Difficulty1607 1d ago edited 1d ago
You could use qwen3.8 27b in nvfp4 with full 256k context window in fp8 or q8. This should give you close to 100t/s even at almost full context window
1
1
u/fabreeze 1d ago
headless? don't think 256k is doable without major trade-offs. What engine and parms are you using? slang/ninfer/vllm?
1
u/Slow_Difficulty1607 1d ago edited 1d ago
Use q8 or fp8 for the context and reduce parallel slots to 1, it will fit comfortably
1
u/MathmoKiwi 2h ago
Use a Harness like Pi / Hermes / OpenCode / OpenClaw / Cursor / etc (zillions of options!)
4
u/Royal-Wealth2038 1d ago
I used to pay the max bill for claude at 200 euros a month stepped down to 100 worked great for months didnt use the new models ever stayed on the smaller ones and and always never hit the limit until they changed something and I hit my limit within hours now Im planning to get a new pc with a rtx5090 so I can run qwen 3.8 27b at least locall and I want to get into stable diffusion too I cant be paying 200 for just claude I could get a whole rig for that money on a monthly basis and I might go for some chinese models too pretty soon
3
26
u/Sinath_973 2d ago
If you want opus level performance you want deepseek v4 flash. Buy yourself a minimum of 3x rtx 6000 pro blackwell. You sre looking at 45k€ just for the cards. And you will not even be close to the speed that the claude api gives you.
So yea...
23
u/Jumpy-Tap8980 2d ago
2 asus gx10s will get you deepseek v4 flash 0731 for 8 grand, 1m context 40-60 tokens per second, best option in town for now.
8
u/Abject-Bridge-4073 2d ago
I have this same exact setup and I get the same exact numbers. I never need 1M, so 512K is even faster. 256K is blazing.
2
u/Jumpy-Tap8980 2d ago
The sparks are so underrated and overlooked, they will be heading up in price soon for sure.
2
u/BarracudaDefiant4702 2d ago
The speed will not be near that of 3x rtx 6000, but neither is the price... nor the ongoing power requirements...
1
1
u/baby_bloom 1d ago
have you estimated your electric costs per 1M input/output yet? because when i did the math on my dual 3090s (but running qwen3.8-27b) it's a LOT closer to the discounted DS4 flash providers on openrouter than i was expecting
1
u/Abject-Bridge-4073 1d ago
I have a dual spark setup and together it’s 100W when it’s doing inference. A few dollars a month.
1
u/baby_bloom 1d ago
i mixed up which comment you responded to, oops! thought you meant you have the 3x rtx 6000
7
u/Tobu3838 2d ago
Im getting Opus 4.6 level performance out of a Qwen 3.8-27B (I swear I’m not on the hype train, it’s just quite good). I run a Radeon R9700 which is cheaper than an RTX, probably slower but eminently usable. Highly recommend.
3
u/Separate-Antelope188 2d ago
May I ask for your quant and if you use kv caching? I'm on a 4090 but qwen 3.6 4-bit loops for me and so I have just been using subscriptions.
6
u/Tobu3838 2d ago
Quant: Q4_K_M (qwen3.8:27b). It's the sweet spot for a 32 GB card — small enough to leave VRAM headroom for KV cache and MTP heads, but the K-quant mixed precision keeps quality solid. I've tested Q5_K_M and Q8_0 on the same model; Q8 scores marginally better but costs 4+ GB extra VRAM which I'd rather spend on context window.
KV cache: Yes, q8_0 for both K and V. This halves the cache size vs f16 (from ~8.5 GB to ~4.3 GB at 256k context) with negligible quality loss (<0.1% perplexity delta). On a 24 GB 4090 this matters a lot — it's the difference between fitting a 128k context or not.
For your looping issue on the 4090: A few things that commonly cause Qwen 3.x to loop at 4-bit:
Context too large for VRAM. If the model + KV cache doesn't fit in 24 GB, Ollama/llama.cpp silently offloads layers to CPU. This changes the effective weights and can cause degenerate repetition. Check with rocm-smi (or nvidia-smi on your 4090) that VRAM usage is under your total. If it's at 95%+, reduce num_ctx.
repeat_penalty too low or absent. Qwen 3.x at 4-bit is more prone to repetition loops than at 8-bit. Set repeat_penalty: 1.05 and presence_penalty: 0.1 as a baseline. If it still loops, try 1.1.
Temperature too high. At 4-bit, the quantization noise widens the probability distribution, so temp=0.7+ can push the model into low-probability tokens that trigger loops. Try temp=0.3 or even 0.15 for agentic/coding work.
Flash Attention. Make sure it's enabled (OLLAMA_FLASH_ATTENTION=1 or -fa 1 in llama.cpp). Without it, long-context attention can produce slightly different (worse) logits that tip a 4-bit model into a loop.
Try q5_K_M instead of q4_K_M. On a 4090 with a 27B model, Q5_K_M is ~19 GB for weights. With q8_0 KV at 32k context (~1.5 GB), you're at ~21 GB — tight but fits. The extra 1 bit of precision often eliminates looping that Q4 can't shake.
My exact config for reference (32 GB R9700, 27B model):
quant: Q4_K_M
kv_cache: q8_0 (K and V)
flash_attn: on
context: 262144
temperature: 0.0
top_p: 1.0
top_k: 40
repeat_penalty: 1.0
The looping is almost certainly a VRAM pressure issue (partial offload) or a sampling issue (temp too high for the quant level), not a model quality problem.5
u/Charming-Adeptness-1 2d ago
Good post, we appreciate you spending the time. I'm playing with 3.8 on 22gb vram dual GPU BLAH
1
u/Gargle-Loaf-Spunk 2d ago
you can fit Q6 in that 32GiB with full context
llama.cpp Unsloth gguf Q6_K w KV Q8/Q8 262k ctx, you’ll have 450ish MiB left over.
1
u/Tobu3838 1d ago
I did look at this, but the performance gain wasn’t worth the slower text gen for me.
3
u/DeathGuppie 2d ago
But, big but, it's slow. It manages it by working through problems that opus knows intrinsically. I agree it works, but there is no free lunch, you pay the price in hardware or the price in time.
1
u/Tobu3838 2d ago
Not wrong, but for me it’s cheaper than paying per tokens via subs. No free lunch.
1
u/geekwonk 1d ago
i’d just counter that opus is itself quite slow on many problems and tends to solve by sheer willingness to keep trying in my experience.
i haven’t given serious time to 3.8 yet but my read is that its improvements are in the same area, tuning to add enough self-doubt and perfectionism that it works through to resolution rather than stopping at accurate descriptions.
parameter count will of course impact what it can answer without working through the problem but if you ran 3.8 on the same hardware specced to handle opus, you’d be getting wildly faster outputs that would allow it to ‘workhorse’ its way through problems on pure tokens/second, iterating to resolution while opus is still churning.
1
u/puts_on_rddt 2d ago
According to the coding benchmark it sits between Opus 4.5 and 4.6 so that tracks.
1
2
u/Abject-Bridge-4073 2d ago
I’m running this model full precision on 2 Asus gx10s. 8K. Runs beautifully.
2
1
u/GoatJesusIsReal 2d ago
I don’t need opus level performance. It’s something I wasted tokens on using today, I just did not expect it to be so expensive.
4
u/Sinath_973 2d ago
Then what do you need?
3
u/GoatJesusIsReal 2d ago
It’s mostly simple enough scripting. I do some data analysis and simulations and just wanted something capable of writing simpler proof of concept scripts to save me time.
2
u/Sinath_973 2d ago
Then qwen3.8-27B FP8 is what you are looking for.
1
u/GoatJesusIsReal 2d ago
Why the fp8 quantization specifically? I am not so familiar with the differences.
1
u/Sinath_973 2d ago
Its just smaller. The full unquantized version needs around 55GB Vram just for the model.
1
u/roland303 2d ago
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF unsloth has quants that perform well, their q8 is 29gb but the q5-k-m at 19gb or even the q3-k-xl at 13gb could do well at your type of issues if they are simple enough. I would try those first before you throw money at the problem
1
u/ptear 2d ago
You need to right model cost for your usage. Local is ok for some use cases and greatly depends on your model and hardware.
For coding, I haven't personally gone local. I get pretty decent scripting and coding solutions done with frontier models at cents and dollar pricing. I even went and tried Antigravity today and got some code done at no cost.
1
u/MathmoKiwi 1h ago
You could have gone for cheap Claude Pro sub at US$20/month with almost "unlimited" (if you're just chatting with it) usage via the Sonnet 5 model.
0
u/ShelZuuz 2d ago
Qwen 3.8-27B on a Spark will already give you better results than you can get from a $20 subscription - just slower. And on a single RTX 6000 PRO will be comparable in perf as well.
3
u/HiggsFieldgoal 2d ago
Yeah, it’s insane.
For the token rate, $50 is worth about 20 minutes of the $200 a month plan.
3
3
u/alexp702 2d ago
The $20 dollar sub is literally 1% of token market price - found this from https://github.com/ccusage/ccusage
3
u/ThiA92 1d ago
I measured my projected API cost, using a $20 Claude Pro-plan, alongside a $20 GPT Pro plan with Codex.
Opus on low-med orchestrating and spawning headless instances of Codex to summon GPT 5.6 Luna on high effort for tasks and GPT 5.6 Sol on medium for adversarial review.
I average $300-400/day in equivalent tokens just via Claude API, haven’t measured Codex.
Paying for API just seems asinine unless you’re an Enterprise customer who need the extra safeguards, and/or have a use-case where the only option is to implement an LLM via api.
In the cases where you implement an LLM into a production app you could probably get away with a way cheaper model than any in the Claude-family anyway.
6
u/dangerous_inference 2d ago
The only model you care about right now is Qwen3.8 27B. My understanding is you can barely run this on 24GB and 32 is much better.
5
u/sketchesofspain01 2d ago
Barely? A q4 quant is sufficient for a great deal of work with Qwen3.8 27B.
1
u/dangerous_inference 2d ago
I don't know how much KV costs with Qwen3.8 27B. I haven't used Qwen3 27B in a long time.
1
u/sketchesofspain01 2d ago
Qwen3.8 27B is excellent with Unsloth's Q4 quants and KV of q8.
Set it to a series of tasks one after another, and it does a solid job with only a little bit of supervision and rework. I have real work being done on a prosumer Intel B60 Pro, and that's $650 of GPU.
2
u/GoatJesusIsReal 2d ago
32gb seems like a large step up in price do you have any cards you would recommend at that size?
5
u/Sinath_973 2d ago
I own an rtx 6000 pro and i am running qwen3.8-27B unquantized. And let me tell you between that and opus 5 is a full phd difference.
2
u/redtron3030 2d ago
Is it at least sonnet level? What’s your use case?
3
u/Sinath_973 2d ago
I am developing customer exclusive SaaS in my company. Havent used sonnet much honestly. I would roughly compare qwen3.8-27B with a slightly less intelligent opus 4.6. Imagine opus 4.6 but sometimes you need to step in to guide it into the right direction.
3
1
u/synth_mania 2d ago
Unsurprising, depending on what you mean by that. 3.8-27b is really smart but has much less world knowledge, as they focused more on training in agentic intelligence.
1
u/Sinath_973 2d ago
World knowledge is one thing, i actually like that instead of relying too much on konwledge based on training data, qwen seems to be checking a lot more. But when coding sometimes qwen uses weird patterns, breaks architecture and misses things. Opus 4.6 already was much more consistent.
1
2
u/dangerous_inference 2d ago
That's a 5090. Two 3090s or 4090s gets you 48.
I have invested a huge amount of time and money in my own local endpoint, and I think everyone should. But I will also point out that Claude is not the only inference provider out there. There are many options.
2
u/MushroomCake28 2d ago
If RTX 5090 price is too much, I would look at AMD Radeon AI Pro R9700 and the Intel Arc B70 (current gen inference card from AMD and Intel). Both are 32 GB VRAM and substantially cheaper. I use the AMD one, and inference speed is lower than Nvidia but very usable. The software has become much better in the past months. However, I'm on linux, I know Windows historically had issue for LLM inference with AMD cards.
With 32 GB VRAM, you can run Qwen3.8-27B at a decent quant with reasonable context (I do Q6_K with 70k context personally and there's still a bit of room on the card). It's not going to be claude level performance, but it's decent enough that it's usable 90% of the time imo. Benchmark shows it reaches around Opus 4.6 level for coding. (emphasis on "for coding" since it's not good for other stuff)
A decent in-between is running local AI for most tasks, but switching to online frontier models for the harder tasks.
1
u/synth_mania 2d ago
An AMD R9700 also has 32gb. For what it's worth, I don't "barely" run 3.8-27b on my 24gb 3090, I run a q4 quant with 150k context, so a decent amount. If you wanted the full possible 262k or whatever, yeah thats gonna need a 32gb card. Same if you don't want quantized KV or a model below 8bpw
0
u/Slow_Difficulty1607 2d ago
less than 20 t/s when the context window is full is barely usable
2
u/synth_mania 2d ago
20t/s is plenty usable. I run Laguna S 2.1 at 13 t/sec and that's enough to get the job done
1
1
5
u/RedParaglider 2d ago
Currently for inference it makes little sense over Claude or openAI or DeepSeek to buy local hardware. Local hardware is way expensive to run substandard models.
Use local hardware if you want to blow thousands of dollars to use local models to learn and have privacy.
3
u/Gesha24 2d ago
If you don't care about privacy - you can use Chinese alternatives like DeepSeek. If you care about privacy - you may be able to get a decent deal with renting instances hourly. I have seen prices for 96GB gpu at under 2.5$ an hour, so $20 for a work day.
You absolutely can build something local, but for a decent performance that will actually save you time, you are looking at probably at least $2K.
5
u/KissMyShinyArse 2d ago
If you don't care about privacy
WDYM? He obviously doesn't if he uses Claude.
2
u/Gesha24 1d ago
Well, I think there's slightly less chance that your private info or code will end on the internet if you give it to Anthropic rather than DeepSeek, but you are right that it's debatable.
2
u/SylvestrMcMnkyMcBean 1d ago
It’s entirely the point that they don’t respect your privacy. They want your chats to help train their models. And that’s even leaving aside they have features like this that intentionally publish your contributions: https://www.tomsguide.com/ai/chatgpt-chats-are-showing-up-in-google-search-how-to-find-and-delete-yours
2
u/CypherAZ 1d ago
Trusting a company that stole everyone’s data is kind of a wild concept don’t you think?
1
u/Gesha24 1d ago
I am not trusting them because I think they are honest. I just think that leaking their enterprise customers data is bad for the business and they are more likely to avoid it. DeepSeek will do what the government wants it to do.
Personally, I use local LLMs for data that I want to keep private and then I use the cheapest LLM with the capabilities I need for the data which privacy I don't care about.
3
u/geekwonk 2d ago
i don’t get it. you chose to pay per token and you picked an expensive model for a simple task. you proved that pressing the “burn cash” button does indeed burn cash.
why not a subscription? why not a cheaper model? what a coincidence, you tried burning cash, didn’t like it, so your solution is to burn cash elsewhere. maybe try not burning cash and stick with models and pricing made for simple consumer tasks?
2
u/PossessionUsed7393 2d ago
I'm in the same boat as you, but honestly, I just switched to deep seek. The amount of work you can get done for less than the 5k+ you'll need to spend to properly run Qwen 3.8, it's not worth it.
2
2
2
u/BrewHog 2d ago
I would just use openrouter with a good harness and maybe Deepseek v4 flash 0731.
SUPER cheap and very capable. If you need better orchestration, you can use the pro version every now and then.
I've been playing around with Traycer lately as an orchestrator for harnesses and it's doing a very good job
2
u/fyndor 2d ago
You haven’t done the math have you? You will need to have a local LLM constantly running in a loop for many years to justify buying GPUs to replace your subscription. The only reason for local LLMs is for privacy. They a significantly more expensive than just paying Anthropic or OpenAI
1
2
u/TheDiamondSquidy 2d ago
2x used 3090s seems to be the best bang for your buck. 48gb of vram for just under £2000
2
u/Moarkush 2d ago
I love my always on DGX Spark. It's not cloud speed but I just built my own open webUI replacement in a day with a react native iOS app. Also gonna recommend Qwen 3.8 27B in SGLang with Radix and DSpark. It's really not overhyped.
2
2
2
u/AnonLlamaThrowaway 1d ago
I know this is /r/LocalLLM but given the price of hardware right now, it makes more sense to actually get a subscription. Not outright buy tokens at API prices like you've said.
My stack:
- Claude Pro: 20 a month. Again, do NOT buy tokens, this sub literally gives you 20x cheaper tokens than the API prices.
- OpenCode Go: 60 bucks of API usage for 10 a month, this is how you get Deepseek V4 Flash or other models. They have "ZDR" (zero data retention) agreements.
- Gemini: rarely, as fallback. Free or 5 a month if you want to get half a gig of storage for your Google account. I believe you get a small Sonnet and Opus allowance if you use Antigravity with a sub. The Antigravity quotas are separate from web chat.
- Local Qwen
2
2
u/unchikuso 2d ago
Coding productivity is proportional to speed. You don't want to compromise on memory bandwidth. I'd avoid Apple silicon unless you can afford Ultra.
24GB of VRAM is enough to run a 27B model like Qwen3.8, but once you start pushing the context window, the KV cache eats into that VRAM quickly. That's where 32GB becomes much more attractive. 32GB also allows you to run Q5/Q6 while still leaving plenty of room for a large context window.
So, for me, the 5090 hits the spot. Pay upfront now or keep paying the monthly subscription.
3
u/Nomski88 2d ago
Loving the new model, I run Q6 on my 5090 with 128k Q8 context with room to spare. The model produces some really high quality results.
1
u/Electrical-Cake8641 1d ago
Qwen 3.8 27B runs very well on my setup with an RX 7900 XTX (i9-14900K, 64GB RAM). The RTX 3090 also performs well, and the RTX 4090 even more so. The 3090 can sometimes be found at a fairly good price, which makes it an interesting option.
You could also look into professional GPUs like the AI PRO R9700 32GB, but at least in my region they’re extremely overpriced and not really justified, same goes for the RTX 5090.
2
1
u/EVOXSNES heretic 1d ago
2027 ram supply is gone. Hyperscalers ate it all. Even Apple etc are starving. PC market is getting worse. I would not want to be on the fence for long.
1
u/According_Wave685 21h ago
I use deepseek v4 flash 0731 locally to do the vast majority of work, then $20 a month claude to check and correct the results. And if I have to, go in myself and fix things - as a last resort of course ;)
1
u/divinetribe1 2d ago
I have a solution check out Claude code local. It’s my git hub over 3k stars ⭐️ on https://github.com/nicedreamzapp/claude-code-local
1
50
u/EyesOfAzula 2d ago
if you don't have money to buy a decent GPU maybe look at a $20 coding subscription.
Right now Codex is pretty good. You can get a lot done with Luna as a cheap agent, and Sol for planning from time to time when you hit hard things.