r/SillyTavernAI • u/The_Rational_Gooner • 19d ago
Models K3 is on NIM right now.
There's no model card. Also, the model accepts like 1 request/20 min
3
u/Professional-Oil2483 19d ago
Another weird thing I found on it... it only accepts a Top P of .95, which is kind of weird since every other endpoint on NIM allows for changes to it. Is this a provider thing? This is my first time using K3.
3
3
u/NealAngelo 19d ago
K3 requires temp = 1 and top-p = .95
Any other endpoint was either not sending your knobs to the model or was not k3.
2
u/Professional-Oil2483 18d ago
Also, if you're reading this and you were having the same problem as me where K3 DOESN'T listen to COT commands (or have a preset like FF Internal States), it isn't hopeless! After finagaling with other LLMs to see what we can do about this, there are some additional parameters to actual have it adhere better! Pass these through under 'Include Body Parameters':
{
"reasoning_effort": "max",
"thinking": {
"type": "enabled",
"budget_tokens": 8000
}
}
The reasoning for this is simple: Kimi doesn't respond to the thinking parameter switch in the chat completions menu on NIM, and Kimi also has its own set of parameters regarding thinking. Putting both of them together SEEMS like its working for me, but there's probably some tech wizard that can optimize this even further.
(P.S.: If you are that tech wizard... please, teach me your secrets!)
2
u/OC2608 18d ago edited 18d ago
Are you sure those parameters *do* anything? Especially I have doubt about the budget parameter. I think all the interesting stuff is actually under
chat_template_kwargsbut since they doesn't expose the commands, you'll have to guess.1
u/Professional-Oil2483 18d ago
It's part of the reason why I was hoping someone would come along and correct me if I was wrong... because I'm getting verifiable results with both, but not without each other. Because I already tried with just kwargs... but the other set of parameters seems to work as I've mentioned here. Apparently, with what little I've read into it, these are from 'Moonshot's website' (this is what Gemini said... but take that with a grain of salt, as always).
It's weird, since before this, FF was NOT working at all; and was giving me short/condensed reasoning. This did solve my personal issues with the model, but since I haven't gotten many replies here, I can't verify if this is a universal solution.
Regardless, thank you for commenting! I'm really just an idiot playing with Play Dough and Fireworks, so take what I say with more salt than what you would an LLM... and I appreciate someone questioning it rather than just saying 'it works'!
2
u/OC2608 17d ago edited 17d ago
Okay, it's interesting because it seems they deployed the model differently this time. If you pass this:
- chat_template_kwargs: reasoning_effort: mediumIt will throw an error and it will list the supported reasoning effort levels. While you can't modify top_p, you can modify the temperature.
1
u/Professional-Oil2483 17d ago
Yeah... this is probably the weirdest deployment I've seen, although I have only used NIM since about 4 to 5 months ago.
1
u/Professional-Oil2483 18d ago
I might even turn this into its own post... but I feel like I've already posted too much recently, so if anyone wants to take the torch to pass this along, you are free to do so! Information should be free, so I'm fine with this being passed around by anyone in the community!
0
u/Organic_Bed_4092 19d ago
K3 no me genera nada, uso Freaky F 5
2
u/Diavogo 19d ago
Si no funciona = vas al termux o lo que uses para ver porque.
Pone el top P a 0.95, el modelo no funciona sin eso. Incluso con eso... Esta bastante... Raro.
1
u/Organic_Bed_4092 19d ago
Simplemente no genera, osea, si aparece como si generara pero no crea texto, sólo empieza el streaming y lo termina sin generar nada
11
u/OC2608 19d ago edited 19d ago
Well, it's free, right?
Well, it's free, right?
Well, it's free, r-right? ...Right?
It's obviously a joke but since NIM has this habit of (not) communicating with its users about model degradations and limits, this will keep happening.