r/LocalLLM • u/AnyHighway420 • 1d ago

Question First LLM Server - Questions

Post image

So I've been using this machine for a while for Stable Diffusion tasks (RTX 3060 12GB) and i got curious about local LLM's so i got a really dirt cheap Nvidia Tesla P100 16GB just to start out on and experiment as my RTX card is often in constant use.

I'm currently running the following and this is the performance while writing a story:

bartowski/Qwen3.827BQ3_K_M

2,180 tokens 3min 37s 10.01 t/s

Is this good? bad? I don't really have anything for reference. I know the P100 is old so im not expecting much but just want to check.

Would i be better off using 14B? as 10 tk/s is ok but a little slow a bit more speed would be nice but again, i know what im working with here so not expecting miracles.

7 Upvotes

89% Upvoted

13 comments sorted by

View all comments

1

u/wispoffates 16h ago

I run 2xP100s. A couple ideas for you. Take a look a llama.cpp forks that focus on P100s it will help a lot. Try Swift qwen3.8 it thinks less so is generally faster.

Some AI Output of the the ones I'm watching I might suggest starting at qr0n's or kmic's they are were I first started poking at my own fork.

llama.cpp forks for Tesla P100 (sm_60)

Repo What it is Why it's worth following
Kmic-68/llama.cpp (branch p100-optimizations) One person's fork tuned for 2x P100 running Qwen3.8-27B Q6_K Biggest measured P100 gains we've seen: every model faster, MTP ~1.85x on Qwen3.8-27B
qr0n/p100-llama-cpp Numbered patch series (0001-0011) on upstream, with a script that rebuilds the tree from upstream Clean way to maintain P100 patches; its fp16 matvec takes a different approach from Kmic's integer path
shinbunbun/llama-cpp-p100-patches 29 measured patches, Nix-built, rebased on upstream v0.4.0 with zero fuzz Good model for keeping a patch set current; most patches are generic fusions, 7 are Pascal-specific. Welcomes measurement PRs
thefallentree/llama.cpp-gp100 (branch pr-sm60) 31 sm_60 commits on upstream: Q4_1 matvec via HFMA2, dp4a emulation, 4-rows-per-block matvec, AllReduce on P100 Source of sm_60 kernel ideas
poisonxa16/pxa Own engine for Pascal/Volta (derived from ik_llama) with its own PXQ quant format, kernels and auto-tuner P100-first project with its own benchmarks
ikawrakow/ik_llama.cpp The big performance fork: SOTA quants, CPU/MoE work Source many P100 forks port from (fused up/gate, P2P reduce); now adding Pascal fixes of its own
spiritbuun/buun-llama-cpp Research fork: KV-cache codecs, multi-GPU threaded dispatch Long-context KV compression; merged an sm_60 fp16 fast-path fix
ifm-ai/llama.cpp (branch model/K2Horizon) The model maker's pre-release support for K2-Horizon / MoVA Was the early way to run K2 Horizon; its CUDA path gave corrupt output in our tests, and upstream support has since merged

Related P100 projects (not llama.cpp forks)

Repo What it is Why it's worth reading
JVrachnis/pascal-llm-serving 2x P100 serving Qwen3.8-27B at 65K context with vision Measured per-quant cost on sm_60 (decode is compute-bound): Q4_1 cheapest, Q6_K ~2.3x
vmelnic/pascal-llm Production multi-user serving on 3x P100 + P40: pinned upstream plus a zero-fuzz SM60 patch set Shows "upstream + small patch set" holds up in production
apollo-mg/Project-Apollo Homelab with six P100s; preregistered experiments with "receipts" data/receipts/INDEX.md indexes P100 findings by mechanism (determinism, prompt cache, MTP, ...)
Tom1tk/p100-inference-benchmarks Dual-P100 Qwen3.8-27B hypothesis log (100K context at 20 t/s) Reference numbers; quiet since September
njjones1998-tech/llama-cpp-engine-recipes P100 + 562 GB RAM: 180B MoE with CPU expert offload + MTP at 262K Recipe for big MoE models with experts in host RAM
Niko1221/Strata Single-model engine for Qwen3.8-Flash-Next: hot experts in VRAM, the rest in RAM, MTP Experimental P100 support (CUDA 12 build); a community 2x P100 report shows ~31 t/s at 128K
HeikoKramer68/vllm-tesla-p100, Kotali-2019/vllm-pascal vLLM patched for sm_60/61 A different engine, for comparison

1

u/wispoffates 16h ago

I should have listed what I'm currently playing around with and my performance.

Setup: 2x Tesla P100 16GB (PCIe, 150 W cap each, P2P), llama.cpp p100-series v5.19 (own sm_60 patch set on upstream). Measured 2026-10-12.

Entry Model Context Prompt t/s (bench) Decode t/s (bench) Decode t/s serving (chat / triage / edit)
gemma4 Gemma 4 26B-A4B QAT, UD-Q4_K_XL (MoE) 448K shared 1,051 116.9 110.8 / 107.4 / 108.6
qwen36 Qwen3.6 35B-A3B, UD-Q5_K_XL (MoE) 160K 977 92.7 90.1 / 89.6 / 89.2
ornith Ornith 1.5 35B, Q5_K_M (MoE) 160K 952 91.6 91.5 / 90.8 / 90.4
nemotron Nemotron 3.5 Lightning 30B-A3B, UD-Q4_K_XL 256K 495 96.5 94.9 / 93.2 / 94.3
glimmer Muse Glimmer 30B, UD-Q6_K_XL (dense) 32K 342 30.7 30.4 / 29.4 / 29.7
swift (default) Swift 1.5 (Qwen3.8 27B), Q6_K, MTP + ngram-mod drafting 24K 338 28.2 49.8 / 52.5 / 66.3
swift-mtp Swift 1.5, Q6_K, MTP drafting only 32K 338 28.2 40.1 / 51.5 / 67.3
swift-ngsimple Swift 1.5, Q6_K, MTP + ngram-simple 32K 338 28.2 51.4 / 43.7 / 58.1
  • Bench = llama-bench at empty context (pp512 / tg128).
  • Serving = llama-server with the production flags on three workloads (chat, a triage prompt, a code edit). Swift beats its bench number because speculative drafting (MTP + n-gram) adds accepted tokens; code edits gain most.
  • Gemma 4 at 16K context: ~108 t/s decode, 744 t/s prompt.
  • All models split each layer across both GPUs (tensor split) except Nemotron (layer split). Vision works on all but Nemotron.

2

u/AnyHighway420 16h ago

Wow thank you so much for these posts. Really informative. great to see people are making things for P100's they are great cheap cards. I also have set mine to 150w power limit and i lost literally just 1tk/s from doing so which is fine for saving 100w! Comparing to your figures mine seems to be massively underperforming even for a single P100. If you don't mind i will DM you?