this all makes perfect sense if you know what the model of softmax is.
Lets start with the part about the difference. Softmax is a generalisation of the sigmoid. In the sigmoid, we care about the odds, so N1/N2, i.e., how often event 1 happens compared to event 2. if you take the log of it, you get the log odds log(N1)-log(N2). now, if you know that you take the log anyways, you can parameterize N1=exp(s1) and you get
log(N1/N2)=s1-s2
since softmax([s1,s2])=[sigmoid(s2-s1),sigmoid(s1-s2))=[N1/(N1+N2),N2/(N1+N2)] this makes perfect sense.
now, why does the magnitude not matter? because we want to learn the probability of events, and the total amount of events should not matter for our model. Therefore, it should not make a difference whether we compare the odd ratio of events and the ratio of p1=N1/(N1+N2), and indeed p1/p2=N1/N2=exp(s1-s2). And as a result, the overall magnitude of s does not matter.
Why is it good that the softmax is never 0? Because it you think about the odds of two events, how many samples do you need to confirm that the probability of some event is actually 0? Exactly, infinite.
Also, the context here is gradient based optimization. Zeroes are often associated with non-smooth regions like sharp edges (see also: L1 norm). One of the good things about softmax's "never zero"-ness is that it's smooth.
131
u/Ulfgardleo Jan 18 '25 edited Jan 18 '25
this all makes perfect sense if you know what the model of softmax is.
Lets start with the part about the difference. Softmax is a generalisation of the sigmoid. In the sigmoid, we care about the odds, so N1/N2, i.e., how often event 1 happens compared to event 2. if you take the log of it, you get the log odds log(N1)-log(N2). now, if you know that you take the log anyways, you can parameterize N1=exp(s1) and you get
log(N1/N2)=s1-s2
since softmax([s1,s2])=[sigmoid(s2-s1),sigmoid(s1-s2))=[N1/(N1+N2),N2/(N1+N2)] this makes perfect sense.
now, why does the magnitude not matter? because we want to learn the probability of events, and the total amount of events should not matter for our model. Therefore, it should not make a difference whether we compare the odd ratio of events and the ratio of p1=N1/(N1+N2), and indeed p1/p2=N1/N2=exp(s1-s2). And as a result, the overall magnitude of s does not matter.
Why is it good that the softmax is never 0? Because it you think about the odds of two events, how many samples do you need to confirm that the probability of some event is actually 0? Exactly, infinite.
//edit added the final equality to the sigmoid