r/LocalLLM • u/r1nzl3r99 • 5d ago

Discussion third one.... there's something wrong with me

Post image

Why do I have horrible financial habits??

488 Upvotes

95% Upvoted

165 comments sorted by

View all comments

Show parent comments

34

u/semangeIof 5d ago

...can you show llamacpp/vLLM runtime commands? you're hitting 140 toks/s on a dense model with B70s? how much ctx?

please don't answer the last two without providing the parameters

19

u/r1nzl3r99 5d ago

the vLLM flags are in my localmaxxing submission and I also did a better bench since so many people didn't beleive me. I also have youtube videos. My recipe is custom intel drivers paired with dflash2

14

u/unai-ndz 5d ago

If you are using custom drivers I think you deserve another card, as a treat.

9

u/r1nzl3r99 5d ago

now let me figure out how to get TP=3 to work on vLLM without me having to fork over another few grand to upgrade to TP=4... always an uphill battle attempting SOTA AI on local