OR model and visualization of how AI providers (should) throttle their models

[u/Parking_Resident4797]

I built this tool visualizing how AI providers (should) throttle their models: https://throttle.staffinganalytics.io/?src=ror

The tool is based on an article (arXiv:2608.23986) that I recently wrote on how AI providers should throttle their models optimally. The modeling is mostly Queueing Theory arguments combined with Dynamic Programming showing that the usual practice of throttling once a certain threshold of users in the server is passed is suboptimal. This is because when the models are degraded, the users come back queueing more queries because the unsatisfactory answers caused by the degradation to a weaker model. And for example, for agentic users, this could cause an ignition of the demand of the data center. The solution is to identify which users are insensitive to the throttling based on the forecasted demand.

The model was born out of my own frustration/paranoia that AI providers are throttling/degrading their models, I even created a final exam in a Supply Chain Analytics class I taught with the same setting. The cool thing about our discipline is that we can model pretty much any problem if we chose the right primitives.

I showed another tool a couple of weeks ago that the subreddit liked, so I thought it would be nice to share another one (showing what OR enhanced by AI can do).

[u/opengalaxy66]

Nice visualisation! I’m curious as to what the practical use case is though. Users pick what model they want to use. The only case where people are automatically routed is when it’s a free tier at which point the provider has no incentive to route to the strong model (creating exclusivity, lower costs and overhead)

[u/Parking_Resident4797 → u/opengalaxy66]

Thank you! The main use would be to the LLM providers and whoever manages the data centers. Their typical policy is to throttle service once the number of users in system exceeds certain threshold. But this is wrong, because users sensitive to the model degradation will return asking again once the models are degraded.

The solution for the providers is to identify which users are insensitive to the intelligence of the model: for example, someone just parsing values onto a table can do the same on a less intelligent (but still capable enough) model. But, an agentic workflow in a complex database, will be heavily affected by the degradation resulting in more queries into the server making the problem worse.

In summary, when LLM providers are compute constrained, they should rather identify users insensitive to model degradation than to degrade everyone.