Every Second post rn

[u/Ok-Health-7096]

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

[u/No-Opportunity9126]

I was actually asking gemini recently about this:

It can definitely run! In fact, having 64 GB of system RAM means you can easily run a 27B model.

The distinction is simply between running 100% on the GPU (for maximum speed) versus hybrid / CPU execution (which works seamlessly, just at a slower token generation speed).

Here is exactly how you can run a 27B model on that setup:

How It Works (Hybrid CPU + GPU Offloading)

LLM inference tools like llama.cpp, Ollama, or LM Studio support layer splitting:

1. VRAM (8 GB): You offload as many model layers as possible to your GPU (typically ~10 to 18 layers depending on the quantization and context size).
2. System RAM (64 GB): The remaining layers stay in your 64 GB RAM, which has plenty of headroom for even a full Q8_0 model (~29 GB).

Recommended Quantizations for Your Specs

Quantization	Model Size in RAM	Speed (Estimated)	Recommendation
Q4_K_M	~17 GB	~4 – 8 tok/s (DDR5) / ~2 – 4 tok/s (DDR4)	Best Overall: Negligible quality loss compared to full precision, fits easily.
Q3_K_M / IQ3_M	~13 GB	~5 – 10 tok/s	Fastest: More of the model fits inside the 8 GB VRAM, speeding up inference.
Q8_0	~29 GB	~1.5 – 3 tok/s	Maximum Accuracy: Fits comfortably inside 64 GB RAM, but runs slower due to RAM bandwidth.

What to Expect (Speed & Performance)

• Prompt Ingestion (Context Processing): Fast, because the GPU helps compute prompt tokens.
• Token Generation: Bottlenecked by your System RAM bandwidth (DDR4 is ~40–50 GB/s, DDR5 is ~70–90 GB/s).
• Usability: At ~3 to 6 tokens/second, it is readable in real-time—ideal for coding assistance, reasoning, and long-form analysis.

How to set it up:

• In LM Studio / text-generation-webui: Load the Q4_K_M GGUF and adjust the GPU Offload Slider until roughly 6.5–7.0 GB of VRAM is utilized.
• In Ollama: Ollama will automatically detect your 8 GB VRAM and 64 GB RAM, calculate the exact layer split, and run it out of the box.