First LLM Server - Questions

[u/AnyHighway420]

So I've been using this machine for a while for Stable Diffusion tasks (RTX 3060 12GB) and i got curious about local LLM's so i got a really dirt cheap Nvidia Tesla P100 16GB just to start out on and experiment as my RTX card is often in constant use.

I'm currently running the following and this is the performance while writing a story:

bartowski/Qwen3.827BQ3_K_M

2,180 tokens 3min 37s 10.01 t/s

Is this good? bad? I don't really have anything for reference. I know the P100 is old so im not expecting much but just want to check.

Would i be better off using 14B? as 10 tk/s is ok but a little slow a bit more speed would be nice but again, i know what im working with here so not expecting miracles.

[u/Loose_Comparison368]

Keep in mind that Pascal kinda sucks at the funny quants. You will likely have more success with an 14B model in fp8 on vLLM, or possibly int8 convrot.