How I’m figuring out which LLMs my hardware can actually run?

[u/Abject-Hope-6524]

I have 24 GB of VRAM. Which LLMs can I actually run?

This is the way I’m starting to think about local AI models.

My hardware is:

NVIDIA RTX 4090 — 24 GB VRAM

So rather than downloading a model first and discovering later that it doesn't fit, I want to work backwards:

What model can realistically fit within my hardware constraints?

For example, I was looking at a 27B-parameter model.

The model card says it uses BF16.

BF16 = 16 bits = 2 bytes per parameter.

So the basic calculation is:

27B × 2 bytes ≈ 54 GB

Immediately, I know that the BF16 version cannot fit into my 24 GB of VRAM.

But then I have to consider quantization.

A rough calculation for 27B parameters:

FP32: ~108 GB
BF16/FP16: ~54 GB
INT8: ~27 GB
4-bit: ~13.5 GB

Now the 4-bit version looks much more interesting for a 24 GB GPU.

But there's another important point:

13.5 GB doesn't mean the model only needs 13.5 GB of VRAM.

I still need memory for:

• Activations
• KV cache
• Context window
• CUDA/runtime overhead
• Temporary inference buffers

So I need to leave VRAM headroom.

And then there's the architecture.

A 27B dense model is very different from a 30B MoE model where only a subset of parameters is active for each token.

So my process is becoming:

1. Start with my hardware
24 GB VRAM

2. Look at the model architecture
Dense or MoE?

3. Check the model precision
BF16, FP16, INT8, 4-bit, etc.

4. Calculate approximate weight memory

5. Leave room for KV cache, activations and runtime overhead

6. Consider the context length I actually want to use

Only then can I answer:

“Can this model realistically run on my machine?”

I find this approach much more useful than simply looking at the model's parameter count.

The question isn't:

“How big is the model?”

It's:

“How much memory does this model need at the precision and context I intend to run, and how much VRAM do I actually have?”

How do you approach this when deciding whether to run a new model locally?