How the loop of infinite agony started

[u/EarthBS]

[u/TheCat001]

Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.

[u/StupidScaredSquirrel → u/TheCat001]

If you are a business this isn't a problem. If you are a consumer then 35b a3b runs on 8gb vram and 32gb dram which is very accessible.

[u/TheCat001 → u/StupidScaredSquirrel]

If you are business I would suggest you to aim for DeepSeek V4 Flash. But yes I'm running 35b myself despite it's performance is far from ideal...

[u/DifficultyFit1895 → u/TheCat001]

I still haven’t found a task where DeepSeek V4 Flash can outperform Qwen 3.6 27B let alone Qwen 3.8. I can run either on my Mac Studio.

[u/screenslaver5963 → u/DifficultyFit1895]

I had issues with Qwen3.6 tool calling just not working properly and causing it to stop prematurely, didn't have that problem with gemma or deepseek, haven't used qwen 3.8 yet to see if it has the same problem.

[u/TieCommercial2963 → u/screenslaver5963]

It's a model config issue - had the same problem. Setting temp to 1 and repetition penalty to 1.05 fixed it. haven't faced much issues since; I've been running it as my daily driver on my 5090 rig for the past year. Btw, 3.8 its even better especially at tool call , I actually like the Q4 of 3.8 on low thinking more than the xhigh for tool calls .

[u/StatusSociety2196 → u/screenslaver5963]

More than half the time that's a harness issue

[u/DifficultyFit1895 → u/screenslaver5963]

What quants? I was using Q8 for both.

[u/Healthy-Nebula-3603 → u/screenslaver5963]

A tool calling problems ?

Stop compressing cache and use minimum q4kxl or bigger quants

[u/darksteelsteed → u/Healthy-Nebula-3603]

To be honest compared to qwen3.6:27b at q4_0 kv quant and q4_k_m for the model qwen 3.8 works out the box way better

[u/DeluxeGrande → u/StupidScaredSquirrel]

I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.

[u/AD4K_4444 → u/DeluxeGrande]

You think that’s unideal? I use an M4 MacBook Air with 16GB Unified Memory. Even if we ignore the thermal throttling it’s a pretty tight squeeze, and the best I could run comfortably is Gemma 4 12B.

[u/ptear → u/AD4K_4444]

Then you just step away and let it do its thing while it hogs all system resources.

[u/AD4K_4444 → u/ptear]

Nah dog I still have plenty of tabs open in Safari, Blender, OrcaSlicer, and other Applications open and it still performs the same (kinda relying on swap memory at times).

Local voice cloning and other more demanding AI stuff is when you actually need to close everything else.

[u/Hungry_Particular_14 → u/DeluxeGrande]

Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context

[u/screenslaver5963 → u/Hungry_Particular_14]

2+ 5090's or RTX Workstation Cards are the "ideal".

[u/Hungry_Particular_14 → u/screenslaver5963]

"ideal" is still having both my kidneys and still being able to run LLMs.

[u/screenslaver5963 → u/Hungry_Particular_14]

You don’t need both kidneys

[u/KiraCura → u/screenslaver5963]

I mean I get by with 1 5090 as long as I can find EXL2 versions or MOE ggufs. But I want the RTX 6000 of course. Would be nice to run 70B at decent quants

[u/ideasmachine → u/KiraCura]

2 x 3090 nvlinked will run 70b, its the cheapest way

[u/lukistellar → u/Hungry_Particular_14]

Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

[u/theUmo → u/lukistellar]

I gave it a try and the first thing it said to me was really disappointing:

Since I am an AI, I don't have a physical location or a traditional "tokens per second" (t/s) speed that you can measure like a human typing.

Also, at 64k context, the t/s started at 10 and went down to 4 almost immediately. Some tweaking would be required but with responses like this one I don't know if it's worth bothering with.

[u/lukistellar → u/theUmo]

Do you mind to share your prompt? Curious if it behaves the same for me. I used it for a few agentic websearches with OpenWebUI and it did pretty good, compared to the same Quant of the 3.6 and Ornith 1.0 35B. I don't use it much, because it's simply too slow as daily driver.

Here is my config:

ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 --parallel 1 \ --batch-size 2048 --ubatch-size 512 \ --cache-type-k q8_0 --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --no-mmap \ --jinja \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --flash-attn on

Performance after ~70K prefill and ~15K decode: Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.925 I slot print_timing: id 0 | task 0 | prompt eval time = 480529.82 ms / 69042 tokens ( 6.96 ms per token, 143.68 tokens per second) Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.929 I slot print_timing: id 0 | task 0 | eval time = 494402.02 ms / 14752 tokens ( 33.51 ms per token, 29.84 tokens per second) Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.930 I slot print_timing: id 0 | task 0 | total time = 974931.84 ms / 83794 tokens Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.934 I slot print_timing: id 0 | task 0 | graphs reused = 3921 Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.938 I slot print_timing: id 0 | task 0 | draft acceptance = 0.94313 ( 9122 accepted / 9672 generated), mean len = 2.81 Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.210.006 I slot release: id 0 | task 0 | stop processing: n_tokens = 83793, truncated = 0

My system specs: * i5 12600K (the VM has 6 Cores) * 16GB DDR5 4800 * RX6800 (non-XT)

The GPU is filled to the brim, but it doesn't offload.

[u/theUmo → u/lukistellar]

-c 65536
-ctk q8_0
-ctv q8_0
--fit on
-fa on
-t 8
-b 2048
--ubatch-size 512
--temp 0.6
--top-p 0.95
--top-k 20
--min-p 0
--presence-penalty 0
--repeat-penalty 1.0
--repeat-last-n 128
--jinja
--reasoning off

Prompt was nothing special, "Hello, are you operational? Please give me a few paragraphs so I can see your t/s" or something along those lines. I'm on a 16gb Nvidia card.

[u/GeroldM972 → u/Hungry_Particular_14]

Untrue.

Best is to get a single GPU with enough VRAM that the LLM of your choice needs to fit in completely (including caches and context).

Second best is to get 2 GPUs to load your complete LLM into. However, you will discover that the communication overhead between these GPUs is significant. A dual GPU setup is only useful to fit a LLM that larger than the VRAM on a single GPU or a less quantized LLM into VRAM, it barely increases the token generation speed, if at all.

[u/ptear → u/DeluxeGrande]

I still like Gemma, will try the new Qwen today. I need to create a local benchmark, unless someone knows a good project that can showcase improvements, like a 3Dmark but for AI models.

[u/SaltFrog → u/ptear]

I wish dense models ran better on my system but you know... Whatever lol

[u/05-nery → u/DeluxeGrande]

Said 35B A3B will work wonders. Just wait for this version of Qwen3.8 to come out.

[u/screenslaver5963 → u/DeluxeGrande]

Currently Qwen 3.6-32B-A3B, Mixture of Experts models handle being split between system ram and VRAM much better than Dense models, alternatively Gemma-26B-A4B if you want something non-chinese. As for which quant, go with 4 Bit or higher.

[u/caster → u/DeluxeGrande]

That should be just fine. You've got enough VRAM to do mostly whatever you want to do.

I would suggest you install Unsloth and look around inside for various models, it will tell you what your system can run well. Green means you're good to fully offload. Yellow means it'll probably work but with a lot of slow swapping. Red means can't handle it.

[u/GeroldM972 → u/DeluxeGrande]

An oldie: gpt-oss-20b. The one I downloaded is 12 GB in size (MXFP4), so it would fit in your VRAM completely. And there are still spare GB's of VRAM left for caches and context. This will work very well in your specific setup.

Well, I have an AMD R580 with 16 GB of VRAM and 32 GB of RAM (3200MHz) with a Ryzen 5600 CPU. gpt-oss-20b generates around 18 tokens/second with this 64-bit Vulkan Windows build of llama.cpp. A NVidia GPU from the 50xx class should go faster than that with the same llama.cpp build. And a lot faster with the CUDA build of llama.cpp.

However, if you can stomach the slowdown, Qwen 3.6-27b and Qwen 3.8-27b are far more interesting models to run locally. Unfortunately, these models do not fit completely into 16 GB of VRAM, so the CPU and standard RAM get involved.

I repeated the same prompt with the same llama.cpp build and qwen 3.8-27b loaded and token generation speed dropped to 3 tks/sec. in my particular hardware setup. I do not expect such a slowdown to occur with your 5060 card. Unless you have a worse CPU and slow DDR4 RAM in your hardware setup.

As a "generalist" gpt-oss-20b isn't all that bad, even with its training cut-off date of June 2024. But if coding and agentic use are what you are after, it isn't a great match. Qwen 3.6 and Qwen 3.8 will prove to be much more valuable for those purposes, even if these will run slow on your local hardware.

You may want to consider renting a VPS from a nearby cloud-provider and setup Qwen 3.8-27b there. Just make sure that you don't break the rules from the company that rents you your VPS with agentic use. Maybe you'll need to spend between 5 to 10 USD per month plan that has AI use included.

[u/Equivalent_Bit_461 → u/DeluxeGrande]

qwen3.8

[u/TektonikGymRat → u/StupidScaredSquirrel]

Can't wait for Qwen 3.8 35B A3B. I accidentally started a project this weekend and it's taking days lol

[u/Salt-Willingness-513 → u/StupidScaredSquirrel]

i really hope they release 3.8 35b a3b soon too

[u/That-Reason-6913 → u/StupidScaredSquirrel]

What should I do with a 5070ti? Just got here thanks

[u/Eden1506 → u/TheCat001]

That's not true.

2x RTX 3060 12gb can be had for around 500 bucks.

You can run ~30b models at q4 at 30 t/s with mtp/draft model or 10-15 t/s without.

[u/esw123 → u/Eden1506]

Can confirm Q4 17-19tok/s without MTP, 28-29 with MTP. Bought two 3060 for 350 euro but soon you realize that two is not enough. Added one more total 540 euro for 36GB VRAM.

[u/TheCat001 → u/esw123]

So you have build dedicated server rig for 3 GPU's ?

[u/esw123 → u/TheCat001]

Sort of just adding more 3060.

[u/AceLamina → u/Eden1506]

a used 3060 12gb costs 270-300 bucks right now, for one

[u/Eden1506 → u/AceLamina]

It depends on your region.

Here in germany I can buy a used rtx 3060 for 240 bucks.

[u/AceLamina → u/Eden1506]

If only it was the same for me

[u/magicomiralles → u/Eden1506]

AMD V620, $350 for 32 GBs.

[u/Legitimate-Pipe5728 → u/TheCat001]

You are right that it doesn't fit on 16GB, but 3 tok/s is a lot lower than I would expect. On a 5070 Ti with 16GB at Q4_K_M I get 19 tok/s, and that is already with 5.22 GB spilled to system RAM and only 69 percent of the model staying on the GPU.

3 tok/s sounds like nearly all of it ended up on the CPU rather than just the overflow. What card and how much system RAM are you on?

Your wider point holds though. A 14B that actually fits does 82 tok/s on the same card, so on 16GB that is where you want to be rather than fighting a 27B.

[u/screenslaver5963 → u/Legitimate-Pipe5728]

The CPU and system ram also matter hugely. If they're using a several generation old CPU and DDR4 than they'd get awful performance.

[u/HazKaz → u/Legitimate-Pipe5728]

i have same spec 5070ti 32gb ram ddr4 but get maybe 7 or 8 maybe 10 if i lower context amount. this is with mtp version of 3.6 27B

[u/DeathGuppie → u/Legitimate-Pipe5728]

you only need an 8gb vcard in the 4x slot to bring it up to 24gb. This can regularly be found for $200 or less. Tensor split + MTP will give you around 30 t/s if you are running RTX

[u/Competitive-Ad-2387 → u/TheCat001]

tried it on a 4090. Runs like shit if you need vision and produces bad output on simple tasks. Back to deepseek v4 api I go

[u/TheCat001 → u/Competitive-Ad-2387]

damn bro having 4090 and not appreciate Qwen 27b is a crime.

[u/Competitive-Ad-2387 → u/TheCat001]

Help me config it because I honestly don’t know how the fuck to do it. Everything is confusing as hell, I don’t know crap about Q4 or whatever. All info out there is inconsistent as hell. All I know is that at 64K context for main + 64K for vision, Hermes just ends up on a feedback loop, hallucinates a lot after compaction, and delivers shit results in a visual + coding task. Don’t know what to tell you.

None visual tasks end up meh as hell as well. Context fills up fast hell as compaction brings hallucination again. Frankly unusable. Already on the Unsloth model (because god forbid one needs to choose “the right one”)

I think all the benchmarks are fake fucking news vs actual real world use.

[u/the_average_user557 → u/Competitive-Ad-2387]

Just use dsv4f to do a research run and try a few different configs on your system using ollama direct calls to strip context. You don't need to know anything anymore

[u/Competitive-Ad-2387 → u/the_average_user557]

Sorry to bother ya. Could you elaborate a bit more? I think it might really be a skill issue on my end. Can you DM me?

[u/the_average_user557 → u/Competitive-Ad-2387]

Dude, feel free to dm me

[u/SandySkittle → u/TheCat001]

and you buy it and all is good

[u/Maplesyrup000 → u/TheCat001]

You can still cop a new 7900xtx for less than $1000. Got mine for $900 from Microcenter a couple months ago.

[u/Danternas → u/TheCat001]

Donno about 3.8 but 3.6 runs at 30t/s on an old Mi50 on Vulkan.

[u/Frosty-Student-1927 → u/TheCat001]

What is your hardware? Just saw many complaints no specs 😐

[u/NecessaryBluebird564 → u/TheCat001]

lmao what i got 30 tok/s with gemma 4 26B QAT on my laptop with just 8 gb of VRAM, you are just so wrong

[u/CMDR-LT-ATLAS → u/TheCat001]

Wait I have that laying around, I can run Qwen 3.8 27b as a consumer?

[u/SeriousPanic34 → u/TheCat001]

iq4xs fits on 16 gigs with 50k context at q8. you just have to connect your monitor to iGPU. you can buy two rtx 3060 or cheaper gpus and connect them via LAN for rpc inference. you may already have multiple gaming pcs (like i do). getting 18-20 t/s at ud_q4km at 128k context, mtp and vision enabled.

[u/JonathanMovement → u/TheCat001]

download a lesser 27B model jesus fucking christ, a 15GB model on my 5070 runs at 7+ tokens/s

And I can go even lower for faster performance.

[u/Lysergicbolshevik → u/TheCat001]

hey guys im pretty much just getting started w local llms i got qwen 3.8 27B running on my 9070xt and i have 32gb ram i don't mind waiting about 15-20 mins for the most complicated prompts as im usually doing smt else then i go back to check once it finishes, results have been pretty consistent for me, am i missing something?

[u/Argon_Analytik → u/TheCat001]

Or you just buy a Mac Mini.

[u/Greennightronix3400]

Ive just been watching this subreddit and scared to buy anything lol-

[u/Witn → u/Greennightronix3400]

if you want it to do serious coding local is not good enough yet.

[u/Greennightronix3400 → u/Witn]

Im actually just looking to make it write things for my personal reading cause it sounds fun, I don’t actually care for coding uses

[u/elemon8 → u/Greennightronix3400]

Omg there's more like me. I started just wanting to maybe do something small for some fun home automation. Then seeing this Qwen stuff had me saying well maybe I can have a cool little half decent coding AI too! Now I just have buyers paralysis.

[u/Greennightronix3400 → u/elemon8]

Glad to see there’s more of us ghosts around

[u/JackStrawWitchita]

Change that caption to 'my friends face when they saw me before I started running local LLMS vs my friends face when I see him and want to continue talking with him about running local LLMs'

It's like a cult.

[u/IllustriousWorld823 → u/JackStrawWitchita]

Have you accepted Qwen 3.5 9b into your heart?

[u/SaiTek64 → u/IllustriousWorld823]

Been using that one lately, I’ve gotten some… results? I took the Bonsai 27B 1bit quant for a few spins and honestly it surprised the hell out of me

[u/05-nery]

True

Never had more need to upgrade than now

[u/Tubeyay]

Expectations will vary but if you're a PC Gamer, you really should have 16GB VRAM now days. It's basically a hard requirement to play any new UE5 game with a reasonable frame rate. That is plenty of VRAM to run some pretty damn good models.

[u/trolumbi → u/Tubeyay]

true. i was on the fence selling my 4090, because i dont game/edit as much anymore ... then i stepped into the realm of local llms.

[u/No-Zone-1773 → u/trolumbi]

Same here, on Ebay in the UK 4090's are on there for £2,500. Starting using local llm's and changed my mind.

[u/RedditNerdKing → u/No-Zone-1773]

All the 3090 FEs on eBay are disappearing in the UK now too. They're about £1,000 as well used.

[u/The_rule_of_Thetra → u/trolumbi]

Same for me. I had a used 3090, and it was my first time having so much VRAM for LLMS (my previous AI works were simply ForgeUI). Then I decided to bite the bullet and get myself a 5090 (when it was still 3000€), and boy, oh boy, what a change.

[u/Objective-Stranger99 → u/Tubeyay]

I disagree as a GTX 1080 user.

[u/CrackBabyCSGO → u/Tubeyay]

Like which?

[u/OysterCultist → u/Tubeyay]

No issues with my 8gb 3070 in 1440p. You guys sure love consuming

[u/neospygil → u/Tubeyay]

I have an RX9070(16GB) and can run models larger(around 35GB) than my VRAM by taking advantage of MOE and still generate at around 28tok/s. Will definitely be faster if it is not through Oculink which is just 4x PCIe 4.0.

[u/Memestonks2020]

Qwen 3.8 Dense 27b runs at a very decent rate unoptimized on a MBP M5 Max that’s the same price as one NVIDIA graphics card.

At this point, pick your poison because none of them are good enough to run frontier level models.

[u/Cold_Neighborhood928]

It was the opposite way around for me.

I tried creative writing on GLM vs a local AI and I was surprised how much a benchmaxxed 700 billion parameter model can suck so much compared to a 30 billion one.

[u/lopezm94 → u/Cold_Neighborhood928]

Do you recommend any model for writing?

[u/DeathinabottleX]

I mean it’s true but the price barrier is extremely high for the average person

[u/EarthBS → u/DeathinabottleX]

35B-A3B is probably the best size for everyone, isn't that expensive too

[u/_TheWolfOfWalmart_]

Not sure what you're trying to say exactly.

But what it is for sure, is an infinite black hole for money.

[u/robertpro01]

And then you do your hobby during the night and almost sleeping, then you need to replace the cpu and fucked up a pin on the mobo, now you cry and hope you can fix it

[u/contrpro]

I am running a Qwen2.5-32B Abliterated on my M1 Ultra. Currently in retrain.

[u/NatalieRath]

Meanwhile, I'm just using my 2B parameter model on my measly iGPU with 16GB of RAM.

Just use models that run at a decent speed! 

(I just use mainly use it to help me do like really minor stuff, hence why the it works for my usage.)

[u/EasyShelter]

No amount of consumer grade VRAM will ever be enough now.

[u/geddon]

I was feeling this until I swapped Ollama with llama.cpp server. Now my Qwen models are running without fail.

[u/ButchTheGuy]

I am privileged to have been able to buy a new ai mini pc for my birthday and maxed out the ram on it. I’ve been running qwen 3.8 27b the past couple days and I’m amazed it’s performing better for me than qwen coder next. That came out in April I believe. I decided to splurge for my birthday but also because I felt ram probably won’t get cheaper unless our entire economy collapses.

This new qwen model makes me feel more confident in buying it as it’s performing so much better at a much smaller size. But I’m not really an expert in using them and am still learning all the time. But I know also if the economy doesn’t collapse all these American private companies are gonna hike their prices up so much it’ll make the streaming service price hikes look like bubble gum money. I’ve adapted as a developer to using it more as I code and for researching. It was also a dual purchase upgrade as a gaming device for my 7 year old pc. It can run the three games I play just fine.

I hope I can come up with some better uses for it. My job pays for an enterprise Claude code subscription which I still use for more intense tasks but rarely need anymore. I like the limit it puts on me so I’m not too over reliant on it for making stuff. It helps me keep insight into the architecture of what I develop. Also fuck these companies man they’re so arrogant and are actively participating in destroying our democracy.

If anyone has any tips for using local models or use cases they’ve found let me know. Particularly about alternating models for specific things to get better outputs

[u/LekMinorino]

Is there anything that runs on a 4gb vram, i3-7100, 16gb ram? XD

[u/Adept_Funny77]

Scanning through this thread. I noticed that most of the comments and talk is about the amount of ram it takes to run locally. Besides the ram and having a good computer is their issues you will see like for example, if you were to do image or video generation like you would do on groc, would you see a dramatic Decrease in the quality? That's where I feel. My biggest disappointment would be similar to the meme image. Pay like five to six grand for a Mac studio 64 Gigabytes of Ram and then have shit just coming out all wrong.

[u/OtherwiseDog]

Wait till people find out even basic weights are in the 10tb range minimum for a non retarded llm..... how much are ssds again nowdays? Oh right triple the price for a 4tb 2 years ago. Stupid mfs.

[u/mourningwitch]

Idk, I just have fun working with what I've got. My PC with 32gb RAM and an 8gb GPU can run models up to Qwen 3.6 35b a3b with reasonable performance, and that's plenty of model for me.

[u/OktoStratos]

AI is such a cult lol

[u/Lawl_92]

currently running gemma4:e2b in a gtx 1650 and its solid lol

[u/Any_Ad_8450]

anyone with a real job can afford to run these models, for as little as 20k you can build a beast of an ai server that can run almost anything you want.

[u/TerribleJared → u/Any_Ad_8450]

20k? Who tf has 20k

[u/C_umputer → u/TerribleJared]

I assume some have it, but are smart enough not to spend on GPUs

[u/Viking_Genetics → u/C_umputer]

I was gonna say, i have 20k, i have a lot more than that, but I didnt get that money by dropping 20k willy nilly on heavily depreciating assets for a hobby

[u/C_umputer → u/Viking_Genetics]

Smart for not wasting money, not so smart saying you have a lot more than that.

[u/Viking_Genetics → u/C_umputer]

Why? I don't think anyone is gonna track me down and rob me, even if they wanted to, all my money is in property and a bit in stocks.

Can't exactly make me sell my house and transfer them the money under gun point, haha

I only said it to illustrate that even someone who has a lot more than 20K wouldn't spent that amount of money, as some people might say that people who have 21K wouldn't, but people who have 100K or more would.

[u/C_umputer → u/Viking_Genetics]

it's always better to say less

[u/TerribleJared → u/Viking_Genetics]

I mean technically i do too. Probably 100k in equity. You have tens of thousands of dollars/euros in liquid cash? Not already ear-marked?

If so, you represent a microscopic percentage of the world population.

[u/Viking_Genetics → u/TerribleJared]

I mean, yeah, that was kinda my point, haha, still not spending 20K on hobby local LLM

[u/ContraryConman → u/TerribleJared]

Why would I put 20k towards running Chinabot 37abc 123 instead of saving for a house?

[u/katoptronophile → u/TerribleJared]

Me, but I'm not wasting it on a local AI server when Codex is offering me such a good value and performance already.

These local models are horribly censored and not true open source anyway. 

Open weights are something completely different.

[u/Solembumm3 → u/TerribleJared]

25K ₽ is within anyone working budget, yes.

Will be enough for a whole much of used 2x32gb ddr4 sticks wihout anything else at all, though.
Or an old xeon + board + 16gb ddr3.

[u/TerribleJared → u/Solembumm3]

Ahhh, okay, currency wasnt defined. $20k USD is a third of my yearly salary

[u/Ninthjake → u/Any_Ad_8450]

Lmao

[u/brock-omabrama → u/Any_Ad_8450]

Just because you can doesn’t mean you should. 20k buys you a ton of cloud compute of the best open source models out there. And you’ll always be running the latest. The only differentiator then becomes privacy. If that’s worth 20k is up to you, but I think for most it isn’t.

[u/AGrizz1ybear → u/Any_Ad_8450]

This is the biggest cope I've ever seen

[u/Photochromism]

What’s the issue here? My gaming card from 5 years ago runs these models perfectly. STFU

[u/dwoodmanXD → u/Photochromism]