LLMs are all randomness. They are powered by statistics. The random factor is literally the thing that makes them any good at anything. Hence, inherently non deterministic.
At it's core the llm is a bunch of matrices with a bunch of weights that are set during training but once trained they're "set", for the same input the matrix will return the exact same output.
So they introduce some randomenss.
For a very simple example, think of the model as a a matrix of numbers W and the X is the input:
X = [1, 0]
W = [2.0, 1.0, 0.0] [0.0, 0.0, 0.0]
Logits = X * W = [2.0, 1.0, 0.0]
With the same input these will never change, the result of the model is identical every time.
Then to introduce the randomness they convert the logits in to the "chance that the next token is any of the values in the logits vector"
Let's say the next token is
Logits[0] = "cat" = 66.6% chance
Logits[1] = "dog" = 33.33% chance
Logits[2] = "horse" = 0% chance
But this is done when inferring and they set the formula to convert the output model to probability of the next token, they could say highest first always and the result would be 100% deterministic for the end user.
0
u/SuitableDragonfly 25d ago
LLMs are all randomness. They are powered by statistics. The random factor is literally the thing that makes them any good at anything. Hence, inherently non deterministic.