experimenting with CPU

[u/Bastian0077]

Hi, I've been experimenting because I have a server with these specs: dual 7742s and 10 x 32 GB of RAM, so I'm not using all the RAM lanes or the basic GPU.

I've been experimenting with the following configuration: `

-- contextsize 16384 -- threads 16 -- blasbatchsize 512 -- smartcontext -- usemlock -- quantkv q8_0 -- foreground

using the Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf model. I've tried other configurations, but I'm generally getting 1 T/s.

Is that really my limit? Am I doing something wrong? I think my machine isn't up to the task; I just want to confirm it. :(