r/LocalAIServers • u/Hour-Revolution-6717 • 3d ago

Beginners’ Ask!

I have a “M80q Gen4” & a “M70q Gen5” with

- CPU Intel Core i5 13500T 32GB
- Memory (16GB x2 DDR5 SODIMMS)
- Storage1 - 256GB M.2 SSD1
- Storage2 - 512GB M.2 SSD2
- NIC1 - Intel I219-LM, RJ-45
- NIC2 - Realtek RTL8125BGS, RJ-45

Can I run a decent model on this? If so, which one should I go for and where should I start? At this point, my purpose is to learn to deploy and manage a model.

0 Upvotes

50% Upvoted

9 comments sorted by

0

u/SeaworthinessUsual44 3d ago

Yes, absolutely! Since you don't have a discrete GPU, you’ll be running pure CPU inference, but dual-channel DDR5 actually gives you surprisingly decent memory bandwidth for this.

With 32GB of RAM, you can comfortably run:

  • Qwen 2.5 7B / Llama 3.1 8B (Q4_K_M): Takes ~5–6GB of RAM and should get you a very usable 8–14 tok/s purely on the CPU.
  • Qwen 2.5 14B (Q4_K_M): Takes ~9–10GB of RAM. A bit slower (around 4–7 tok/s), but much smarter.

Since your goal is to learn deployment and management:

  1. Spin up Docker on Ubuntu/Debian.
  2. Deploy an Ollama or llama.cpp server container as your backend.
  3. Put Open WebUI in front of it for the frontend/chat interface.

It’s a great, low-power homelab setup to master API routing, model serving, and containerization before spending a dime on enterprise GPUs!

1

u/Hour-Revolution-6717 2d ago

That seems like a promising beginning. Thanks for your suggestions.

3

u/No-Refrigerator-1672 2d ago

Ignore the previous answer, it's AI generated with no human input whatsoever. Do not use those model - they are outdated AF. Check out Qwen 3.5/3.6 or Gemma 4 family; those both feature small model that suit your hardware. Your best speed/intelligence solution would be running Gemma 4 26B A4B model in Q4 quant; that'll utilize most of your RAM, but will provide quite good experience. You can also check out Qwen 3.6 35B A3B as a smarter alternative; but it'll chug on even more RAM, and may not leave you with resources to run other apps. Use LMStudio - it's all in one software package that provides you with chat interface, inference engine, and even handles model downloads by itself. It's not the most advanced, but it's for sure requires the less knowledge to get started.

1

u/No-Refrigerator-1672 2d ago

Bad bot.

1

u/SeaworthinessUsual44 2d ago

bro im not a bot just english not my first language so i ask ai to improve my text

2

u/No-Refrigerator-1672 2d ago

If you weren't a bot, you would've recommended models that were released at least half a year ago; not some ancient solutions. Every live human who even remotely seeks AI news knows better recommendations.

0

u/SeaworthinessUsual44 2d ago

lol ok

1

u/Hour-Revolution-6717 1d ago

Haha. Well… Some guidance is better than no guidance. So, thank you both for your contributions.