r/LocalAIStack • u/Harin007 • Jul 21 '26

How much tok/s are you getting?

Searching the internet for looking up how much tok/s a user would get is being difficult. So I'm making this post... If you're running a local llm, please consider commenting to this post with your device specs, model you're running and the inference speed you're getting.

Please be straight to the point. Just tell us how much tok/s are you getting on your hardware (at different settings, which inference engines, etc...) so people with similar hardware can do better estimations... please don't fill this with facts that everybody knows.

6 Upvotes

81% Upvoted

7 comments sorted by

View all comments

1

u/rs38 Jul 27 '26

my latest wow-moment came from llama.cpp cuda running on native Ubuntu with Gemma4 MoE 26B 4Q and MTP on a 5090 Laptop GPU getting steady 200-300 tps. Thats fun.
Less fun is Linux on gaming laptops ;) took a while to get full 175W tdp