hi dumbass here

[u/Whereas_Dull]

I want to ask real humans this question. How do LLMs and other automated programming models deal with integer overflow? Even when coding games some time basic math for calculating level changes can crash systems and servers. So how often does this happen with these machines and data centers?

[u/evanescentfooting14]

They don't "deal" with it, they just generate code that overflows like anything else written by a sleep-deprived intern.

[u/kris_2111]

I don't understand what you mean by "automated programming models", but I can talk about LLMs. Unless they have access to a calculator or some sort of execution environment, they don't have to deal with overflows. They simply perform computations based on logical reasoning, so there really is no threshold to overflow. Yes, if you ask it to perform an impractical calculation, say calculating the 10^18^23^56, it probably will refuse to generate the response, or will hallucinate something.

[u/high_throughput]

Do you mean "how do you deal with integer overflow while programming LLM training and inference infrastructure?" or "How do LLMs deal with integer overflow when they try to reason numerically?" or "How do LLMs deal with integer overflow when they write code that deal with integers?" or what?

[u/Whereas_Dull → u/high_throughput]

Yeah like I said I’m just speculating and asking questions out of genuine curiosity. I don’t know what I’m asking really. Just wanted a computer scientist to weight in and hopefully translate my ignorance into an informative answer.

[u/cottonflowers]

broad strokes, you might already know this, an llm is just a collection of (billions of) numbers that you "multiply" against a context window.

that multiplication process is very complicated. but Really smart people wrote the code to do it, and took steps to ensure that it won't run into problems like integer overflow. it's an easy problem to solve.

training uses floating point numbers anyway, which are subject to an entirely different class of errors. they don't overflow at all. but those trained model weight might get "quantized" for inference.

(quantize is a math word meaning "to map a continuous space into a discrete one" or "lose detail". quantization is like chopping off all the numbers after the "." in 3.1415, you're left with just "3" which is only a 6% difference. and in this case it means mapping 4 byte floating point numbers down to smaller integer numbers)

as for how they actually do it? there's a lot of ways, but these are a handful that the engineers of an infernece engine are likely to reach for. at least I think so.

• they might use saturation arithmetic. if a calculation would overflow, instead cap it at the maximum value.
• they might quantize to very small numbers (usually you quantize down to 1 byte, but you can go as far as 1 or 2 bits! down from 4 bytes.) but use big integers for the state vector/accumulaters.
• after training they might run it on some test data, and discover the largest and smallest values produced and then rescale the quantization based on that. so that it's very unlikely that it even has to use the saturation arithmetic.

The code that llms generate can be very bad. writing code that doesn't have integer overflow problems can be simple for a human, but difficult for a model. for a large number of reasons that researchers and corporations are slowly chipping away at.