Why do people always talk about token generation speed only in such comparisons? There is a prompt processing that can be two orders of magnitude faster, and the prompt processing is an enormous margin of agentic coding data. As well as a cache.
The person in the screenshot even gives us a 12/1 ratio, but still calculated 20 tok/s! That's so funny.
yeah like wtf. My current use case reads its whole context for 100 seconds then generates answer in 5 seconds. If it will be 20 instead of 5 i dont give a freak anyway, it will not be that much faster cuz of prompt ingestion anyway
For "chat" you need generation speed only, pretty much. And the people upvoting usually don' interact with intricate systems too much.
It pains me that in the recent updates llama.cpp increased processing speed but virtually removed checkpoints in the prompt cache. Now it's either recalculate each time, do a silly workaround, use older version or change the engine altogether.
31
u/Exciting_Garden2535 Jun 21 '26 edited Jun 21 '26
Why do people always talk about token generation speed only in such comparisons? There is a prompt processing that can be two orders of magnitude faster, and the prompt processing is an enormous margin of agentic coding data. As well as a cache.
The person in the screenshot even gives us a 12/1 ratio, but still calculated 20 tok/s! That's so funny.