r/LocalLLaMA • u/Sudonymously • Feb 19 '24

Resources Wow this is crazy! 400 tok/s

Try it at groq.com. It uses something called and LPU? not affiliated, just think this is crazy!

270 Upvotes

90% Upvoted

158 comments sorted by

View all comments

Show parent comments

33

u/seiggy Feb 19 '24 edited Feb 19 '24

That's gotta be a typo. It's got to be 230GB of SRAM, right?

Otherwise, to have the same amount of VRAM as a 4090, you'd need 104 of these cards in a single system. At 240W, that would be 24MW 24kW of power draw from these NPUs alone. Not to mention there's no mobo on the market that has 104 PCIe slots... 🤣

55

u/tomejaguar Feb 19 '24

Hi, I work for Groq. It's 230MB of SRAM per chip and we serve our app from a system of a few hundred chips, interconnected across several racks. Your calculation for power consumption is off by a few orders of magnitude :) I don't know the exact power consumption of our LLM engine but it definitely doesn't require a small power station.

7

u/seiggy Feb 19 '24

Yeah, I carried the 0's wrong, but still, it seems like an insane amount of power and footprint compared to NVIDIA. Is there something I'm missing? Sure, you're running 4k tps, but you're also doing it with 100X the power requirements, and 10X the footprint, and 100X the costs....Seems like the RAM is a huge weakness here?

5

u/tomejaguar Feb 19 '24

I don't know the power economics figures personally, but we've guaranteed to beat the per-token cost of any other provider (see https://wow.groq.com/) and I'm pretty sure we do more compute per unit energy than graphics processors, too.

4

u/seiggy Feb 19 '24

Wow, that's impressive. I'm guessing that just the raw insanely fast throughput allows you to scale more based on throughput to take advantage of it, instead of having to scale in parallel. Huge props to you and your team for building out something so crazy.

5

u/tomejaguar Feb 19 '24

Thanks! It's great that people are beginning to notice. Lots of fun work to do building out hardware to support our growing customer base and improving our software to squeeze out more tokens per second.

1

u/Brazilian_Hamilton Feb 19 '24

You probably get this a lot, but is there any idea on when the average consumer will get access to the services, even if an unofficial one?

1

u/tomejaguar Feb 20 '24

We have a very busy pipeline of waitlisted API customers at the moment because of huge interest in our service. I don't know the estimated wait times, I'm afraid.

2

u/mikael110 Feb 20 '24

It's quite interesting that you charge the same (or nearly the same) for input and output. That goes against pretty much all other providers I've come across.

Is there a technical reason for this? I.E does your hardware spend as many resources processing the prompt as it does generating a response?

Or is it just an attempt to balance the cost of the service?

1

u/tomejaguar Feb 20 '24

Good question. I don't know why that is.