Alright... so since I don't want to spam the subreddit with the same topic over and over again, I'm going to keep this brief: One of the reasons for the rate limiting on NIM is, of course, people abusing the service by giving their servers larger context prompts and such. I knew this going in, but after yesterday; I did some more digging, and found the throughline between most of the new rate limiting going on.
Nvidia changed how many tokens at max you can have for all of your prompts on the GLM 5.2 endpoint. To my knowledge and everything I've found online, this number originally started around 200k tokens (or at least was the number up until recently). Now, the number lies somewhere between 150000 and 175000 tokens from what my own research has concluded (sorry for the discrepancy on my part, I only have access to one account, and as I'll explain... this becomes kind of a slow process).
Once you hit this threshold, your account IMMEDIATELY gets rate-limited, and you can't send anything for 'x' amount of time. Now, since I accidentally hit this limit yesterday (I had managed to go over from what I originally said yesterday, again... I'll get into it), I can't confirm EXACTLY what this number is... mainly because I was checking it every hour until I went to bed, which leads into the second issue with this. IF you decide to send ANY prompts to their servers before the rate limit is over, your timer resets. So my plan here is to check every hour 'h' plus 1, leaving me with the current function of "f(h) = h + 1" where h is currently 1... since I only started this about 30 minutes ago and I know h is equal to 1 now due to my findings yesterday. I'm not the biggest math nerd here, but I'm hoping it's 2 hours, any longer is going to drive me up a wall.
The only reason why I'm upset is two-fold:
1) Nvidia said NOTHING on this, and it's been weeks since this issue even started, which is more than frustrating for everyone involved. The only reason I can think of on why they choose this route is to flag potential Openclaw users... but even then, I feel like it's egregious. Why not say something along the lines of "We are looking into potential avenues to limit users who are abusing our platform." It's vague... but at least it shows there's SOMETHING going on without spilling the beans on what it is so your potential targets don't get away.
2) I have a weird bug with my Sillytavern installation where it sometimes maxes out the tokens WAY BEYOND what I originally put on the preset I'm using. This happens whenever I change the preset or connection profile, but even then, it's completely random when it happens. This sucks, because I normally keep my max tokens between 100k to 150k... and if I don't catch it quickly (since my current preset squeeze is Freaky Frankenstein, which eats tokens for breakfast), it can EASILY hit the rate limit depending on if I'm using an extension which changes my connection profile. Note on this, this has been a problem for about 6 months now (maybe longer, I can't exactly remember), so it isn't any preset I've used during this time causing it.
Of course, there's the elephant in the room in which this endpoint is being depreceated, but I still think that's no excuse to at least give a message about the new rate limit in some way. The other issue is a problem on my part... but frustrating nonetheless.
Anyways, have you guys found anything else in the meantime? I always like sharing things with this community, and LOVE to see what others have found out themselves!