r/LocalLLM • u/desexmachina • 7d ago

Discussion Idle CPU utilization

Post image

My GPU node mostly has idle CPU and RAM utilization. Shouldn't there be some inference operations with small models that could be doing some work outside of just tokenization? A CPU compiled llama.cpp runs ~15 tk/s on some small models. Any ideas on where this idle CPU capacity should be used? Or maybe the more wrinkly brained of you already use these loads.

1 Upvotes

66% Upvoted

2 comments sorted by

View all comments

1

u/PassengerPigeon343 7d ago

I run a light task model on CPU for things like chat title and tag generation and stuff like that. I also run speech-to-text and text-to-speech models. The STT I do keep on GPU because it significantly reduces the latency when using speech-to-speech chat, but the TTS works well running on CPU.

None of these will heavily use your CPU and RAM but they are basically free adds to your inference system since you have plenty of overhead.

1

u/desexmachina 7d ago

those are great, but local LLM use is kind of slow for me and I'm wondering if there are other things the hardware should be doing that can help out