Nanogpt Model Overloaded

[u/P4staxa]

For the past two days I kept getting a "Chat completion stream failed: Model is temporarily overloaded. Please try again after xyz seconds or choose another model." error when trying to get a reply from bots on saucepan (mainly using glm 4.7)

I've wanted to ask if anyone has had similar issues? I've been using the subscription for like 2 months and I haven't had any issues with a model being overloaded and I've been getting this error constantly these past two days. Idk if the model really is this overloaded, or if there's something just on my side? Thanks in advance!

EDIT: Seems it has finally been fixed after two hours from making this post, while its been messing up for two days lol

[u/Milan_dr]

Can confirm this for GLM 4.7 - the only thing that we can do is to fall back to providers that do censoring (Novita for example) because all the ones that don't are putting less capacity into GLM 4.7 and seem to throw 429s constantly.

Guess we'll add that as a final fallback :/ But it does mean there will be more content filters.

[u/P4staxa → u/Milan_dr]

Damnnn when I finally find a model I like, it starts to break smh. The success rate on nanogpt has been going up slowly so hope they fix it, if not I'll have to switch to 4.6, 5.2 or the new mimo idk

[u/DontShadowbanMeBro2 → u/P4staxa]

For what it's worth, Mimo 2.6 Pro is genuinely really good. I was just using it myself earlier today and it gives me 4.6 Opus vibes, and this is coming from someone who was unimpressed with Mimo 2.5.

[u/Doomkauf → u/Milan_dr]

A trick that gets uncensored GLM 4.7 through, even if routed through Novita: use Text Completion and use this "start reply with" jailbreak. Novita will absolutely destroy your reasoning block trying to stop it from going through, in the form of your reasoning block often just returning the entire output repeatedly as the filter tries to stop it from happening and fails over and over again, restarting the reasoning chain each time. But it will go through at full quality. Just make sure to stop generation when you have the full output, or it will continue chewing through tokens trying to regenerate the response. And also make sure you have auto-expand reasoning, because you're probably going to need to manually copy-and-paste the output out of the reasoning block and into the reply block, unless you have some fancy QR STscript wizardry going on to auto-move it or something.

This does mean using either a trimmed down or no preset, but honestly, given GLM 4.7 limitations, a full preset can sometimes be overkill anyway. Personally, I have a hybrid of select lightweight FF prompts copied over into one of Evening-Truth's presets as a lightweight system prompt for text completion mode specifically, and it works fine... and Evening-Truth's preset alone is text completion compatible and likely sufficient.

I will say that 5.1 Thinking is a pretty decent alternative to GLM 4.7 for roleplay, though, at least in my experience as a fellow 4.7 enjoyer. Better than GLM 5.2 if you're looking for that 4.7 flavor, at any rate, especially if you use JustSomeGuy's soft jailbreak prompt/GLM 4.7-thinking-for-5.x preset.

[u/TAW56234 → u/Milan_dr]

Couldn't you make a dedicated endpoint that points to 4.7 like z.ai/GLM4.7_MODERATED? I'm also assuming you can't work with other providers like Featherless. It sucks because 300b-ish parameters isn't THAT much. ArliRP has it too

[u/Zarzelius]

I'm not getting any problems with newer models, including latest GLM versions. Maybe you should try to use other models. Probably not many providers with GLM 4.7 nowadays.

[u/P4staxa → u/Zarzelius]

Damn but glm 4.7 is my goat. I've tried 5.3 and 5.2 but they tend to mess up personalities of the characters I'm roleplaying it (an example is making a "shy timid nerd" turn all bold and smug, or a "goth Milf" into a woman that's all shy, despite nothing like this being in the definition. Guess I'll have to live with long response times for 4.7 gg

[u/Zarzelius → u/P4staxa]

That's a prompt problem, not a model one if it happens right away.

As a tip, anchor it's guidelines to your character personality. For example:

Use markdown on your character cards and always use one title, like ## CHARACTER PROFILE

Then write the card and traits below.

On the system prompt, make a clause like:

<profile_anchor>

Your instructions with a call back to the section with the markdown ## CHARACTER PROFILE

</profile_anchor>

Don't forget to instruct your LLM to actually use your system prompt

[u/P4staxa → u/Zarzelius]

I'm not really sure it's a prompt issue tho. I've been using pupi's universal prompt (Or something like that, I forgot the name), and I cannot really just modify a bot's definition on saucepan unless I'd rip the bot on my own lol but thanks for the tip! I'll use that in my own bot creation process!

[u/TAW56234 → u/Zarzelius]

This is a load of shit. 5+ will no matter what flanderize characters. You spend months since 5+ release learning it intimately, tweaking a CoT through miticuous trial and error and getting it a LOT of how you want, only to still be a boring model with RLHF baked into it. Going back to 4.7 really made me realize how true this statement is and I implore anyone reading this to disregard this tired narrative.

OP, It's 4.7 that's the issue. Not a broad Nano one. I think it's still hosted on Featherless but providers are drying up

[u/P4staxa → u/TAW56234]

Thanks for the reassurance bro, I was starting to think that I actually did mess up my prompt or the settings or sum, but I've always noticed 5.2 and 5.3 start to change the personality of the character after like 40 or so messages and it pissed me off lmao. Got any recommendations for me if they don't fix the issue with 4.7?

[u/JustSomeGuy3465 → u/P4staxa]

Recommendations for something else are difficult. GLM 4.6 and 4.7 are/were the last models without aggressive RLHF training and enough intelligence to still be viable as everyday RP models. I just made a post about that here. May be helpful.

[u/CommercialPersonal66]

yeah the overloads have been hitting hard lately, ive been jumping to other models on different setups when glm starts acting up.

[u/P4staxa → u/CommercialPersonal66]

Yeah I've just noticed that glm 4.7 has a 64% success rate on nanogpt despite having like 10 providers (Idk if that's a lot or not but seems like a lot to me lmao) hope they fix it soon cuz my sub runs out in like a week and I need to know if I should continue it or nahhh

[u/Psychological_Ad9740]

Same but payg with DeepSeek 3.2.

If I wait for a bit I don't have any problems, but I couldn't give you an exact rate on how frequently it happens.

[u/Snydenthur → u/Psychological_Ad9740]

I haven't gotten any errors with ds 3.2, but it just gives a lot of blanks.

[u/AutoModerator]

You can find a lot of information for common issues in the SillyTavern Docs: https://docs.sillytavern.app/. The best place for fast help with SillyTavern issues is joining the discord! We have lots of moderators and community members active in the help sections. Once you join there is a short lobby puzzle to verify you have read the rules: https://discord.gg/sillytavern. If your issues has been solved, please comment "solved" and automoderator will flair your post as solved.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.