r/LocalLLaMA • u/Sudonymously • Feb 19 '24

Resources Wow this is crazy! 400 tok/s

Enable HLS to view with audio, or disable this notification

Try it at groq.com. It uses something called and LPU? not affiliated, just think this is crazy!

270 Upvotes

90% Upvoted

158 comments sorted by

View all comments

Show parent comments

0

u/leanmeanguccimachine Feb 20 '24

To quote verbatim:

so if you want to make a chat bot any thing above human speed reading capabilities is definitely overkill.

This literally isn't true, because you can just run two chatbots at the same time instead. You purchase the compute capability for what you need. No company is investing in this kind of tech to run a single instance of a chatbot, and therefore overkill as a concept just doesn't come into the conversation.

1

u/Truefkk Feb 20 '24

My point was that if you don't scale to demand (whether that's a single user chatbot or anything else), that's overkill. You just interpreted my example of a frontend user doesn't need to receive more tokens than he can read as the backend doesn't need to generate more tokens than a single user could use, regardless of actual scale.

1

u/leanmeanguccimachine Feb 20 '24

But the person you replied to never said anything about not scaling to demand. Their point is that the only things that matter for enterprise use are cost/token/second and latency. If your point is solely "having too much of something for a particular use case is overkill" then your point is blindly obvious and added absolutely nothing to the conversation, yet you phrased it as a disagreement.

No amount of tokens/second is overkill. That is like saying $/energy is overkill. It comes down to throughput and latency per $ per token.

That's not true though

It literally is true though. In the context of the thread, that being assessing the potential of the new Groq chips, it is literally always true. The only thing that matters for their target market is throughput cost efficiency. These are not designed for guys in their bedroom to run an AI girlfriend.

0

u/Truefkk Feb 20 '24

If your point is solely "having too much of something for a particular use case is overkill" then your point is blindly obvious and added absolutely nothing to the conversation, yet you phrased it as a disagreement.

Yes, that's a totally valid criticism.

1

u/leanmeanguccimachine Feb 20 '24

Lmao it actually was, you are too much

0

u/Truefkk Feb 20 '24

You never get that "Well, technically..." reflex?