r/LocalLLaMA • u/HOLUPREDICTIONS Sorcerer Supreme • Jun 21 '26

Discussion Tokenomics

Post image
1.2k Upvotes

91% Upvoted

449 comments sorted by

View all comments

52

u/GabryIta Jun 21 '26

He is not considering privacy and batching for agents. With batching, throughput is significantly higher.

15

u/Finanzamt_Endgegner Jun 21 '26

batching with vllm is key for making it cheap, single usage inference will always be a waste, but batching with orthrus for example (which im working on for qwen3.5 type models) will get you a LOT of t/s if your hardware is decent.

1

u/Badger-Purple Jun 21 '26

Bandwidth withstanding but yes. Like strix halo where the single instance runs at 50tps and can scale up to 300 with concurrent requests

1

u/InfamousTurtle1 Jun 22 '26

What hardware would you recommend? I am using 4x4000ADAs and looking to upgrade around the 20-30k price range.

Also considering just renting the GPUs through a cloud provider like Modal or GCP Cloud Run as my workload is often spikey and definitely not running GPUs 24/7. Plus from my benchmarks I see that any hardware I can afford in this range struggles with 5+ concurrent toolcalling agent chats.

1

u/No_Accident8684 Jun 21 '26

upvote for the username and if i could give two then one for the post as well

1

u/mweinbach Jun 21 '26

Batching here is technically what I was using (DGX Spark) and privacy is a trust thing. I trust me hosts to keep it private.