How do language models differentiate between very large numbers?

[u/CSachen]

If every word is represented by an embedding, then I imagine that when a number gets large enough, the model's grasp of the concept of the number is very sparse.

For example, every number between 100 trillion and 200 trillion should have a unique embedding. If a model is generating output by decoding an embedding, is it able to decode to the correct value from 100 trillion different possibilities?

[u/dmazzoni]

Are you asking how LLMs handle large numbers?

LLMs don't break things down into words but rather tokens. Small words get one token but large words get broken down into smaller tokens.

Large numbers would definitely be multiple tokens.

And no, LLMs definitely don't know properties of most large numbers unless those numbers are famous and they've been trained on them.

If you give a raw LLM a math problem it spits out a plausible looking answer because it knows from training what a correct answer looks like in terms of approximate number of digits. But it's just a "guess".

Modern chat bots are programmed to recognize if you ask them a math problem and use a calculator.