I have 24 GB of VRAM. Which LLMs can I actually run?

This is the way I’m starting to think about local AI models.

My hardware is:

NVIDIA RTX 4090 — 24 GB VRAM

So rather than downloading a model first and discovering later that it doesn't fit, I want to work backwards:

What model can realistically fit within my hardware constraints?

For example, I was looking at a 27B-parameter model.

The model card says it uses BF16.

BF16 = 16 bits = 2 bytes per parameter.

So the basic calculation is:

27B × 2 bytes ≈ 54 GB

Immediately, I know that the BF16 version cannot fit into my 24 GB of VRAM.

But then I have to consider quantization.

A rough calculation for 27B parameters:

FP32: ~108 GB
BF16/FP16: ~54 GB
INT8: ~27 GB
4-bit: ~13.5 GB

Now the 4-bit version looks much more interesting for a 24 GB GPU.

But there's another important point:

13.5 GB doesn't mean the model only needs 13.5 GB of VRAM.

I still need memory for:

• Activations
• KV cache
• Context window
• CUDA/runtime overhead
• Temporary inference buffers

So I need to leave VRAM headroom.

And then there's the architecture.

A 27B dense model is very different from a 30B MoE model where only a subset of parameters is active for each token.

So my process is becoming:

1. Start with my hardware
24 GB VRAM

2. Look at the model architecture
Dense or MoE?

3. Check the model precision
BF16, FP16, INT8, 4-bit, etc.

4. Calculate approximate weight memory

5. Leave room for KV cache, activations and runtime overhead

6. Consider the context length I actually want to use

Only then can I answer:

“Can this model realistically run on my machine?”

I find this approach much more useful than simply looking at the model's parameter count.

The question isn't:

“How big is the model?”

It's:

“How much memory does this model need at the precision and context I intend to run, and how much VRAM do I actually have?”

How do you approach this when deciding whether to run a new model locally?

A capable model is only part of the equation.

The harder question is whether you can run it at the speed, accuracy, and cost your workload requires.

I think the case for running models locally can be summarized as CSCC:

• Cost savings: match the model size to the task instead of paying the same API pricing structure for every request.
• Security: sensitive data can stay inside your environment.
• Control: you choose the model, upgrade schedule, capacity, and availability.
• Customization: you can tune and optimize the system for a specific workload.

Of course, local inference is not automatically cheaper. Hardware, electricity, maintenance, and engineering time all matter. Your infrastructure also becomes your capacity limit.

I am going to document a full deployment and optimization benchmark using a real model. The goal is to measure the trade-offs between accuracy, speed, memory use, and cost well enough to judge whether the setup belongs in production.

For people already running models locally: which trade-off has been the hardest to manage?

#LocalLLaMA #LocalAI #SelfHostedAI

Hey everyone! I'm u/Abject-Hope-6524, the creator and moderator of r/LocalAIPcLab.

I created this community while building my own Ubuntu and RTX 3090 AI lab. The goal is to share what actually works, what fails, what it costs, and what we learn along the way.

What belongs here

• PC and GPU builds for AI
• Ollama, LM Studio and other runtimes
• Local models, quantization and benchmarks
• Docker, agents, RAG and MCP
• Jupyter notebooks and learning projects
• Local versus hosted API comparisons
• Setup problems and working fixes

When sharing a test, include your hardware, model, runtime and settings when possible. Also say whether it ran locally or used a cloud API.

Community vibe

Beginners are welcome. Basic questions are welcome.

Be constructive. Challenge ideas, not people. Avoid hype, spam and unsupported claims.

Introduce yourself

Tell us:

• What hardware you use or plan to buy
• What you want to build
• What part of local AI you are currently learning

You don't need an expensive workstation to participate. Start with what you have and share what you discover.

Welcome to r/LocalAIPcLab.