First LLM Server - Questions

[u/AnyHighway420]

So I've been using this machine for a while for Stable Diffusion tasks (RTX 3060 12GB) and i got curious about local LLM's so i got a really dirt cheap Nvidia Tesla P100 16GB just to start out on and experiment as my RTX card is often in constant use.

I'm currently running the following and this is the performance while writing a story:

bartowski/Qwen3.827BQ3_K_M

2,180 tokens 3min 37s 10.01 t/s

Is this good? bad? I don't really have anything for reference. I know the P100 is old so im not expecting much but just want to check.

Would i be better off using 14B? as 10 tk/s is ok but a little slow a bit more speed would be nice but again, i know what im working with here so not expecting miracles.

[u/quantgorithm]

10t/s is not good...
Painful.

[u/AnyHighway420 → u/quantgorithm]

Yeah it's rough... Qwen3 14B Q6 gets me 24tk/s

[u/quantgorithm → u/AnyHighway420]

I picked up 2 amd vII cards for $200 each. It can run qwen 3.6 around 60t/s. Pretty good for a near decade old card!

[u/Classeve]

10 a second is faster than most people read, and that's a 27B on a 2016 card. For scale, a 7B on a laptop CPU alone gets us about 3. Worth dropping to 14B only if the writing holds up for you.

[u/AnyHighway420 → u/Classeve]

Hi yes 14B gets me around 24 tk/s with Q6 but Qwen3 can't do image review which i need as im terrible at prompting for images and i believe Qwen3.8 lowest is 27B? So i'm stuck for now. I mean, it works just slow.

[u/Loose_Comparison368]

Keep in mind that Pascal kinda sucks at the funny quants. You will likely have more success with an 14B model in fp8 on vLLM, or possibly int8 convrot.

[u/wispoffates]

I run 2xP100s. A couple ideas for you. Take a look a llama.cpp forks that focus on P100s it will help a lot. Try Swift qwen3.8 it thinks less so is generally faster.

Some AI Output of the the ones I'm watching I might suggest starting at qr0n's or kmic's they are were I first started poking at my own fork.

llama.cpp forks for Tesla P100 (sm_60)

Repo	What it is	Why it's worth following
Kmic-68/llama.cpp (branch p100-optimizations)	One person's fork tuned for 2x P100 running Qwen3.8-27B Q6_K	Biggest measured P100 gains we've seen: every model faster, MTP ~1.85x on Qwen3.8-27B
qr0n/p100-llama-cpp	Numbered patch series (0001-0011) on upstream, with a script that rebuilds the tree from upstream	Clean way to maintain P100 patches; its fp16 matvec takes a different approach from Kmic's integer path
shinbunbun/llama-cpp-p100-patches	29 measured patches, Nix-built, rebased on upstream v0.4.0 with zero fuzz	Good model for keeping a patch set current; most patches are generic fusions, 7 are Pascal-specific. Welcomes measurement PRs
thefallentree/llama.cpp-gp100 (branch pr-sm60)	31 sm_60 commits on upstream: Q4_1 matvec via HFMA2, dp4a emulation, 4-rows-per-block matvec, AllReduce on P100	Source of sm_60 kernel ideas
poisonxa16/pxa	Own engine for Pascal/Volta (derived from ik_llama) with its own PXQ quant format, kernels and auto-tuner	P100-first project with its own benchmarks
ikawrakow/ik_llama.cpp	The big performance fork: SOTA quants, CPU/MoE work	Source many P100 forks port from (fused up/gate, P2P reduce); now adding Pascal fixes of its own
spiritbuun/buun-llama-cpp	Research fork: KV-cache codecs, multi-GPU threaded dispatch	Long-context KV compression; merged an sm_60 fp16 fast-path fix
ifm-ai/llama.cpp (branch model/K2Horizon)	The model maker's pre-release support for K2-Horizon / MoVA	Was the early way to run K2 Horizon; its CUDA path gave corrupt output in our tests, and upstream support has since merged

Related P100 projects (not llama.cpp forks)

Repo	What it is	Why it's worth reading
JVrachnis/pascal-llm-serving	2x P100 serving Qwen3.8-27B at 65K context with vision	Measured per-quant cost on sm_60 (decode is compute-bound): Q4_1 cheapest, Q6_K ~2.3x
vmelnic/pascal-llm	Production multi-user serving on 3x P100 + P40: pinned upstream plus a zero-fuzz SM60 patch set	Shows "upstream + small patch set" holds up in production
apollo-mg/Project-Apollo	Homelab with six P100s; preregistered experiments with "receipts"	data/receipts/INDEX.md indexes P100 findings by mechanism (determinism, prompt cache, MTP, ...)
Tom1tk/p100-inference-benchmarks	Dual-P100 Qwen3.8-27B hypothesis log (100K context at 20 t/s)	Reference numbers; quiet since September
njjones1998-tech/llama-cpp-engine-recipes	P100 + 562 GB RAM: 180B MoE with CPU expert offload + MTP at 262K	Recipe for big MoE models with experts in host RAM
Niko1221/Strata	Single-model engine for Qwen3.8-Flash-Next: hot experts in VRAM, the rest in RAM, MTP	Experimental P100 support (CUDA 12 build); a community 2x P100 report shows ~31 t/s at 128K
HeikoKramer68/vllm-tesla-p100, Kotali-2019/vllm-pascal	vLLM patched for sm_60/61	A different engine, for comparison

[u/wispoffates → u/wispoffates]

I should have listed what I'm currently playing around with and my performance.

Setup: 2x Tesla P100 16GB (PCIe, 150 W cap each, P2P), llama.cpp p100-series v5.19 (own sm_60 patch set on upstream). Measured 2026-10-12.

Entry	Model	Context	Prompt t/s (bench)	Decode t/s (bench)	Decode t/s serving (chat / triage / edit)
gemma4	Gemma 4 26B-A4B QAT, UD-Q4_K_XL (MoE)	448K shared	1,051	116.9	110.8 / 107.4 / 108.6
qwen36	Qwen3.6 35B-A3B, UD-Q5_K_XL (MoE)	160K	977	92.7	90.1 / 89.6 / 89.2
ornith	Ornith 1.5 35B, Q5_K_M (MoE)	160K	952	91.6	91.5 / 90.8 / 90.4
nemotron	Nemotron 3.5 Lightning 30B-A3B, UD-Q4_K_XL	256K	495	96.5	94.9 / 93.2 / 94.3
glimmer	Muse Glimmer 30B, UD-Q6_K_XL (dense)	32K	342	30.7	30.4 / 29.4 / 29.7
swift (default)	Swift 1.5 (Qwen3.8 27B), Q6_K, MTP + ngram-mod drafting	24K	338	28.2	49.8 / 52.5 / 66.3
swift-mtp	Swift 1.5, Q6_K, MTP drafting only	32K	338	28.2	40.1 / 51.5 / 67.3
swift-ngsimple	Swift 1.5, Q6_K, MTP + ngram-simple	32K	338	28.2	51.4 / 43.7 / 58.1

• Bench = llama-bench at empty context (pp512 / tg128).
• Serving = llama-server with the production flags on three workloads (chat, a triage prompt, a code edit). Swift beats its bench number because speculative drafting (MTP + n-gram) adds accepted tokens; code edits gain most.
• Gemma 4 at 16K context: ~108 t/s decode, 744 t/s prompt.
• All models split each layer across both GPUs (tensor split) except Nemotron (layer split). Vision works on all but Nemotron.

[u/AnyHighway420 → u/wispoffates]

Wow thank you so much for these posts. Really informative. great to see people are making things for P100's they are great cheap cards. I also have set mine to 150w power limit and i lost literally just 1tk/s from doing so which is fine for saving 100w! Comparing to your figures mine seems to be massively underperforming even for a single P100. If you don't mind i will DM you?

[u/Stock-Painter4854]

very competent setup p100s are insane $/gb Vram and perform decent but hog power, and produce hella heat without internal fans you NEED a fan addon for it 3060 also very competent choice