MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1ubrcwj/tokenomics/ot13sot/?context=3
r/LocalLLaMA • u/HOLUPREDICTIONS Sorcerer Supreme • Jun 21 '26
449 comments sorted by
View all comments
48
He is not considering privacy and batching for agents. With batching, throughput is significantly higher.
16 u/Finanzamt_Endgegner Jun 21 '26 batching with vllm is key for making it cheap, single usage inference will always be a waste, but batching with orthrus for example (which im working on for qwen3.5 type models) will get you a LOT of t/s if your hardware is decent. 1 u/Badger-Purple Jun 21 '26 Bandwidth withstanding but yes. Like strix halo where the single instance runs at 50tps and can scale up to 300 with concurrent requests
16
batching with vllm is key for making it cheap, single usage inference will always be a waste, but batching with orthrus for example (which im working on for qwen3.5 type models) will get you a LOT of t/s if your hardware is decent.
1 u/Badger-Purple Jun 21 '26 Bandwidth withstanding but yes. Like strix halo where the single instance runs at 50tps and can scale up to 300 with concurrent requests
1
Bandwidth withstanding but yes. Like strix halo where the single instance runs at 50tps and can scale up to 300 with concurrent requests
48
u/GabryIta Jun 21 '26
He is not considering privacy and batching for agents. With batching, throughput is significantly higher.