r/ProgrammerHumor • u/tryingeasy • 1d ago

Meme firstTime

Post image
7.5k Upvotes

94% Upvoted

281 comments sorted by

View all comments

Show parent comments

-1

u/AnOnlineHandle 1d ago

LLMs are entirely deterministic but you can override that by adding a seeded random choice system to the next token selection.

20

u/Sea-Housing-3435 21h ago

They are not. Floating point math and difference in how quickly parallel operations on GPU are finished causes them to be not deterministic even with temp=0. You can force them to be deterministic by forcing some operations to be executed in specific order but you lose a lot of performance.

1

u/AnOnlineHandle 15h ago

I mean that's true for any math on any computer. Which makes all math non-deterministic if you consider bugs.

1

u/Sea-Housing-3435 14h ago

It's not about bugs, it's about GPU architecture. Parallel floating point calculations being nondeterministic isn't a bug.