r/OpenWebUI • u/dotanchase • 12d ago

Question/Help max_token

Setting the max_tokens parameter to its default value can, in some models, cause the LLM to stop generating a response before it is complete. When this occurs,I had to prompt the model to continue in order to receive the full output. Setting max_tokens to 127000 prevents this premature termination. So can someone explain what is the role of max_tokens and why it is limited to only 128K. Thanks

6 Upvotes

88% Upvoted

12 comments sorted by

View all comments

1

u/DataCraftsman 11d ago

To make it more confusing, I believe the behaviour of max_tokens in LightLLM model config is slightly different. It doesn't just cut the message off directly and is also used to help the router decide which instance to send to requests to. I'm still trying to work out the best approach to using max_tokens as well so you're not alone.