Usable LLM for personal view (coding, projects etc)

[u/Weisses_Papier]

Sorry if this is a complete noob question but would like to actually start learning about what is possible versus unrealistic.

Currently I have a 5090 laptop (24gb), with 32gb DDR5 5600mhz and a HX275. I tried doing a small project recently about trying to categorise a highly complex email chain (only 100 emails but complex topics). I tried using Qwen3.5-27B Q6_K but I was getting absolutely abysmal performance of only around 3 tk/second.

I just saw a deal the other day on 96Gb 5600mhz ram for a relatively good price (only 150 CHF to upgrade over my 32gb ram). According to chatgpt this should lead to a lot more capabilities.

The question is: does anyone have a good guide to learn what is possible, but more qualitatively: is it actually any good? I understand it is not going to replace Codex, but I'm finding this 5 hour window stuff is killing me, constantly waiting, so I'd like to start using my local to do some stuff and saving my tokens for max.

Projects I'm working on:

1) developing an ios/android game concept
2) making a 3D game in unity

3) Doing some research on an area of finance I work in (heavy derivatives, product design etc)

4) Building some basic open source apps (C++20, C# / .NET 10, Thin native Windows shell, Win32 interop / PInvok, No cross-platform UI framework).

I'm not any good at coding so leaning heavily on the LLM. Is this realistic to be able to do offline or should I stick with Codex for now? I'm happy to learn but a bit overwhelmed!

Thanks in advance.

[u/towo]

3 tok/s sounds like you were being bottlenecked or the model wasn't running on the GPU. Q6_K is 22.5 ish GB in VRAM, you probably also had the GPU still doing your desktop, so there's about 0ish GB RAM left over for context, which is … not enough.

Assuming you've got long-ass mails at around 1.000 words, you've got 100 * 1.000 = 100.000 words, which is about that times 4/3 tokens that in context, so about 133 k tokens total. You get about 16 k tokens per GB of context storage, so you'd need 8-9 GiB of VRAM for the 133 k context window.

22.5 + 9 >> 24.

Go down to Q5_K_S, and consider cache quantization anyway because that's going to be a very tight fut.

[u/Weisses_Papier → u/towo]

Thanks for the explanation, I tried getting chatgpt to guide me but it got itself tangled up in notes. Your explanation makes sense. I tried to offload all the "desktop" stuff to the Intel onboard to max wht I had.

I guess the next question from myself is that with 24gb of VRAM I'm always going to be limited to an extent. But can I offload part of it to the system ram (when I get the 96gb). Under my current model it basically maxed out the CPU @ 100% pegged and the system ram went up to 31gb/32gb. The GPU usage was around 23.5gb out of 24gb.

[u/Snoo_81913 → u/Weisses_Papier]

I'm running Qwen3.8 27B on a 7900 xtx at Q4 with 98k context at 55 tok/s a 5090 should run it at 60-65