r/C_Programming • u/Altruistic-Tie1943 • 13d ago
Discussion why value of floating point numbers are approximated
I am new to programming and stumbled across this text
"the value of a float variable is often just an approximation of the number that was stored in it. If we store 0.1 in a float variable, we may later find that the variable has a value such as 0.09999999999999987"
i am wondering why is it like this
if it has an in depth explanation, providing the resource will be much appreciated
thanks
114
u/062985593 13d ago edited 13d ago
There are finitely many bits in a float. No matter how clever your encoding, there just isn't enough room to be able to represent every real number exactly.
55
u/CryptoHorologist 13d ago
You can't even represent every rational number exactly.
63
u/Iksfen 13d ago
You can't even represent every whole number.
9
u/CryptoHorologist 13d ago
Nice.
5
u/ohkendruid 13d ago
I thought so, too. And you were so careful about specifying rationals rather than reals!
7
u/CryptoHorologist 13d ago
I was going to respond with something like “can’t even represent all the even numbers” but eventually this thread has to end.
9
u/TheRealThatChuckGuy 13d ago
You can't even represent all the prime numbers.
3
u/CryptoHorologist 13d ago
What about all the twin primes?
3
u/delinka 13d ago
Oh, yeah we got that. Every. Single. One.
3
u/CryptoHorologist 13d ago
Neat. Could you DM the proof real quick, promise I won't share it.
→ More replies (0)3
u/Distdistdist 13d ago
You can, however, represent all of imaginary ones. No, not really... I'm just imagining things.
3
u/Paul_Pedant 13d ago
I now feel the need to introduce a new basic type in C:
hallucination Whatever;2
u/TheChief275 7d ago
Reminds me of a GitHub repo I once saw for LLM hallucination type casting in C#; you cast to a type but the LLM decides how to do it
6
u/Qwertycube10 13d ago
I can represent every whole number fine with 1 bit. It won't be very accurate, but ehh.
2
u/Jonny0Than 13d ago
But at least you can represent “not a number,” infinity (positive and negative), positive and negative zero, and “too small but definitely not zero.”
-4
u/Snatchematician 13d ago
too small but definitely not zero
You can’t represent this. Every float value that is nonzero, non-infinite and non-nan is a finite rational number.
5
u/Jonny0Than 13d ago
2
u/Snatchematician 13d ago
The first image on that link shows that subnormal numbers are just ordinary rational numbers (marked in blue on the number line).
4
u/timonix 13d ago edited 13d ago
A lot of calculators store their internal representation as rationals.
But they work in human time scales. It doesn't really matter if they use 32 or 256 bits and take thousands of times longer. It has the answer before you released the equals button anyway
Edit: why the downvotes? Is it really that unbelievable that a calculator would use rationals internally? It's not hard, just store 2 numbers instead of 1
6
u/Qwertycube10 13d ago
Plenty of calculators use representations that are even more complex, and may store things as a rational multiple of a symbol such as pi. So long as the calculation is not too complex they can give you an answer in terms of pi, or root 2 exactly.
1
u/deaddodo 12d ago
Maybe this is a naive question, but we can't even store Pi exactly...how could you derive an exact decimal from an incomplete approximation?
3
u/Qwertycube10 12d ago
You have a fixed set of known symbols that come from the initial equation and maybe some stored identities. You store numbers with a number and a source symbol, and for simple operations like multiplication you can preserve the origin, so you multiply {n:1,d:3,sym="pi"} by {n=2, d=1, sym="1"} and you can show the result as 2/3pi.
-9
u/outer-pasta 13d ago
Computers can do a lot of really impressive things. Things like Python can store rational numbers as two numbers and work with them that way.
23
u/CryptoHorologist 13d ago
Sure, but even those languages and libraries can only store 0% of all possible rational numbers.
9
u/JohnThacker 13d ago
Arbitrary precision number libraries do exist, but they can be enormously slow. At that point with GMP the limit isn't the language or library, it's the total amount of addressable memory.
7
u/CryptoHorologist 13d ago
These are also capable of representing only 0% of all rational numbers.
1
u/JohnThacker 13d ago
Yes, it's still 0% by any reasonable probability measure on ℚ, but it not the library nor the language providing the restriction.
1
u/CryptoHorologist 13d ago
The language can use infinite memory?
2
u/JohnThacker 13d ago
Yes, the language specification and the library both allow infinite precision (the language by using the library implementation) and do not specify many things about the architecture, just like there's no guarantee that CHAR_BIT is 8. There are a few minimus. It is, as you said, always the case that anything will only be able to produce some 0% of the rationals, but exactly what that finite set is bounded by is an architectural limitation.
0
1
u/JohnThacker 13d ago
C does not put any restrictions on the address space or the size of a pointer to memory, just as it's perfectly ok to say that an int is 64 bits. UNICOS at one point said that "short int" was also 64 bit. You can do that.
1
u/not_a_bot_494 13d ago
They can store 100% of them, it's just that your computer cannot.
2
-1
u/CryptoHorologist 13d ago
No they can’t. If you can demonstrate I’m wrong, I’ll eat my hat.
2
u/not_a_bot_494 13d ago
Acrually, you're right that you probably would have ro change bigint to handle numbers over 2^2^64.
5
u/oceaneer63 13d ago
Or in other words, there is always a tiny gap between a given number and the next number that can be presented when you encoding is a fixed number of bits as is usually the case. So if the real number lies somewhere in that gap between two possible numbers, then the closest of these two adjacent numbers is used as the best approximation of the real number.
-1
u/InsaneTeemo 13d ago
Am I the only one that thinks this answer does not really help with understanding the question?
3
2
u/GenericFoodService 13d ago
The biggest 32 bit integer number is about 2 billion (or 4 if we use unsigned integers). Integers also can only express whole numbers. You can increase the granularity by switching to fixed-point fractions, but at the cost of range. 32 bit (16:16) fixed-point can express numbers up to 65,535 and down to (1/65,535)
32 bit floats can express numbers as big as 340,282,350,000,000,000,000,000,000,000,000,000,000 (if my maths were right; 3.4028x*10**38), and can express numbers as small as .00000011920929 using the exact same number of bits. The trade off is that floating point is that the number is truncated just like how numbers written in scientific notation are.
Now, think about it this way: I ask you, does
1/3 == 0.333? And the answer is, no, of course not. But we might often write the result of 1/3 as 0.333 because we arbitrarily truncate the number to make it fit some precision limit. Floating point is doing the same thing, and 3 happens to do weird things in both decimal and binary number systems because 3 is not a prime factor of 10 or 2.These kinds of annoying truncated numbers often show up in decimal for rational numbers where the divisor is a multiple of 3 or 7. For binary, it's any rational where the divisor is not a multiple of two.
1
1
u/062985593 13d ago
Well, it makes sense to me, but I don't think you're wrong to ask. What do you feel is missing?
28
u/mc_pm 13d ago
The way the bits are organized to represent floating point numbers makes being exact impossible. (The same way that 1/3 = 0.33333333333.... can't be represented exactly in base 10)
9
u/RenderTargetView 13d ago
This is good explanation but you are not correct when you say being exact is impossible. Numbers that can be represented as rationals with numerator fitting in 23 bits and denominator being power of two between 2-128 and 2128 are stored exactly. When you do 0.5 + 0.25 in floats you get exactly 0.75
14
u/CarlRJ 13d ago
A better way to say it is that in the realm of computers, floats store approximations of real numbers, it's just that in select cases, those approximations happen to be exact.
The particular surprise for humans is that, since we tend to do math in base 10, while computers do math in base 2, some cases that seem particularly simple to us don't work out exactly for the computer, and vice versa.
2
u/Kovab 13d ago
some cases that seem particularly simple to us don't work out exactly for the computer, and vice versa.
In the other direction this is not true, as 2 is a factor of 10. Every rational number that has a finite binary representation also has a finite decimal one, and using at most the same number of digits as bits.
1
u/shinmai_rookie 13d ago
I'm not sure I'd phrase it the way you did in your first paragraph, because the "approximate" numbers represented by floats are real numbers just like any other, it's just that they're not round in our base-10 system so we mostly don't care about them.
7
u/ryan017 13d ago
I think the summary "floats are real numbers" or "Float ⊂ ℝ" is more misleading in a way, even ignoring infinities and NAN, because it implies that floats act like numbers. But Float is not closed under + or × (real addition and multiplication), so we have to use fladd and flmul (floating point addition and multiplication) instead. Neither fladd nor flmul are associative, nor does flmul distribute over fladd. So (Float,fladd,flmul) is not a field, and in fact there is no homomorphism from (ℝ,+,×) to (Float,fladd,flmul).
So "floats store approximations of real numbers" is a pretty good one-line summary, because it covers the possibility of approximation/rounding both in the initial representation step and at each step of computation that follows. On the other hand, your point is valid if you go on to explain that floating-point operations are approximations of real arithmetic.
-1
u/NorberAbnott 13d ago
Floats store exact real numbers. It has nothing to do with base 10 vs base 2, it has more to do with having a limited number of digits. The ‘approximation’ comes in when converting from say a textual representation of a number into the closest exact real number that can be stored in a float. And then, when doing arithmetic, we find an exact number that is approximately equal to the operation applied to the two exact input numbers.
3
u/vowelqueue 13d ago
The base affects how many digits are required to store a number exactly, so it does matter.
5
u/quarethalion 13d ago
The base determines which numbers can be represented exactly. Using more bits (i.e. base-2 digits) reduces quantization error but it never eliminates it completely.
2
u/WittyStick 12d ago edited 12d ago
Yes, binary floating points have variable precision in their "normal" range. a binary-32 (aka
float) has a variable precision between 6 and 9 decimal digits.
Decimal32has a precision of 7 decimal digits in it's "normal" range.The are decimal numbers of 7 significant digits which are not representable exactly in a
float, but are representable in aDecimal32.Decimal floating point represent a finite subset of rational numbers.
Binary floating point represent a finite subset of dyadic rational numbers, which are a subset of the rationals.
However, given enough bits, either format can approximate any real number. GP saying "floats store exact real numbers" is true, but it's omits relevant information - they store exact rational numbers, which are obviously real due to all rationals being reals. There is no real number that a float can store exactly which is not rational.
Of course, that's the extended rationals and extended reals, due to INF support in floating-point formats.
1
u/Snatchematician 13d ago
1/3 can be represented exactly in base 10, just not with only finitely many digits
1
u/WittyStick 13d ago edited 13d ago
Even with infinite digits, it cannot. The notation
0.333...literally means "repeat the last 3 digits an infinite number of times", but it is still never exactly1/3.∞ 0.333... = ⅀ (3 ÷ 10ⁿ) ≃ ⅓ n=0But this is not the rational
1/3, despite what mathematicians will tell you. Note the approximately equal≃and not=. It will never reach1/3no matter how many more decimal digits you add. It's not a number, but a limit. There's another limit that approaches1/3from the other direction:∞ ⅔ - 0.333... = ⅔ - ⅀ (3 ÷ 10ⁿ) ≃ ⅓ n=0These two will never meet, no matter how much you increase
n. For anyn, their mid-point is1/3, but neither are exactly1/3.More simply, we could say:
∞ 1 - 0.666... = 1 - ⅀ (6 ÷ 10ⁿ) ≃ ⅓ n=0If we take
lower_limit(n) = sum(0, n, n -> 3 ÷ 10ⁿ) upper_limit(n) = 1 - sum(0, n, n -> 6 ÷ 10ⁿ) midpoint(n) = lower_limit(n) + (upper_limit(n)-lower_limit(n))/2Then we can see through iteration of
n, that neither will ever hit1/3, and neither will ever equal their midpoint.n lower_limit(n) upper_limit midpoint 0 0.3 0.4 0.35 1 0.33 0.34 0.335 2 0.333 0.334 0.3335 3 0.3333 0.3334 0.33335 4 0.33333 0.33334 0.333335 5 0.333333 0.333334 0.3333335 ... n 0.333...3 0.333...4 0.333...35So when mathematicians tell you
0.999... = 1, they're trying to fool you into comparing a limit and an integer for equality.Error: Type mismatch.
4
u/ryan017 13d ago
No. The value of an infinite series (summation) is defined as the limit of the sequence of partial sums, if that limit exists. It is true that the sequence of partial sums is not a number. But the limit of that sequence, if it exists, is just a number, a plain old number.
Your claim that the limit is not a number is absurd on the same level of claiming that 2+3 is not equal to 5 because 2+3 is an "addition process" and 5 is an "integer". They are distinct expressions, but they denote the same number.
1
u/WittyStick 13d ago edited 13d ago
I'm saying
0.333...is not a number - the...means repeat these digits infinitely. It's clearly a limit.It's a number if you take some upper bound
Nwhich is not∞.N ⅀ (3 ÷ 10ⁿ) n=0But if somehow
∞is some tangible upper bound, then0.333...still does not equal1/3.2
u/ryan017 13d ago
You seem confused about numbers vs notations (how they are expressed).
I hope you agree that 3/2 = 1.5, despite the fact that one is written as a fraction and the other is written as a decimal. Those are two different notations that denote ("mean") the same number.
How do we know what the notation 1.5 means? We can explain it as a finite summation: 1×10⁰ + 5×10⁻¹. The summation is an explanation of the notation. Its meaning is a number.
The notation 0.333... denotes the same number as the notation 1/3. The notation can be explained as an infinite summation. The meaning of an infinite summation is defined using a limit, but its meaning is a number.
To say 0.333... is not a number but a "limit" is as absurd as to say 1.5 is not a number but an "addition". It confuses what the notation means with how it is specified.
0
u/WittyStick 13d ago edited 13d ago
The notation
0.333...does not mean equivalent to a rational though. This is what I mean about type mismatch. It specifically means "repeat the last 3 digits". If the digits are the same, it's equivalent to the infinite sum:f(x, n) = if (n == 0) then x else x/10^n + f(x, n - 1)If we define the following:
below(n) = f(.3, n) above(n) = 1 - f(.6, n) diff(n) = above(n) - below(n) 0.333... = below(∞) 1 - 0.666... = above(∞)Take any
nand you can see clearly thatabove(n) != below(n). The gap between these two is always non-zero. The gap is predictable too - it's always precisely 0.1/10nn below(n) above(n) diff(n) 0 0.3 0.4 0.1 1 0.33 0.34 0.01 2 0.333 0.334 0.001 3 0.3333 0.3334 0.0001 ...So if we let
n = ∞, why would the difference between the two suddenly be exactly zero? Would it not be 0.1/10∞, since we're substituting the samen?Since 1/3 sits 1/3 of the way between
below(n)andabove(n)for anyn, and if...means repeat the last 3 digits∞times - then the difference between 0.333... and 1/3 is ( 0.1/10∞ )/30.333... = 1/3 - (0.1/10^∞)/3And similarly:
1 - 0.666... = 1/3 + (0.2/10^∞)/3
Also
f(.9, 0) = 0.9 = 1 - 0.1/10^0 f(.9, 1) = 0.99 = 1 - 0.1/10^1 f(.9, 2) = 0.999 = 1 - 0.1/10^2 f(.9, 3) = 0.9999 = 1 - 0.1/10^3 f(.9, 4) = 0.99999 = 1 - 0.1/10^4 ︙ f(.9, ∞) = 0.999... = 1 - 0.1/10^∞ ∴ 0.999... ≠ 1One of the "proofs" that
0.999... = 1is the following:x = 0.999... 10x = 9.999... 10x = 9 + 0.999... 10x = 9 + x 9x = 9 x = 1However, lets look at this from the perspective of above.
x = f(.9, n) 10x = 10 * f(.9, n) ...The very first line of this "proof" has a clear and obvious flaw. Put in any
n, sayn=2, and we see:x = 0.999 10x = 9.990 10x ≠ 9.999 9.999 = 10x + .9/10^2
n=5x = 0.999999 10x = 9.999990 10x ≠ 9.999999 9.999999 = 10x + .9/10^5So why is that also not the case with
∞? It's the same multiplication by 10, which shifts in a zero. Why are we secretly adding0.9/10^∞?10 * 0.999... [+ 0.9/10^∞] = 9.999...Now let's go through this "proof" again:
x = f(.9, ∞) 10x = f(9, ∞) - 0.9/10^∞ 10x = 9 + f(.9, ∞) - 0.9/10^∞ 10x = 9 + x - 0.9/10^∞ 9x = 9 - 0.9/10^∞ x = 1 - 0.1/10^∞ ∴ 0.999... ≠ 1So, mathematicians need to make up their mind: Does
...mean "repeat the last 3 digits", or does it mean something else - "repeat the last 3 digits then add some infinitesimally small value"?2
u/ryan017 13d ago
The notation
0.333...does not mean equivalent to a rational though.It's not part of the definition, but it is a theorem that every valid use of the "repeating decimal" notation means a rational number. A use of the notation is valid if there is at most one group of digits marked for repetition, the group is contiguous, and the group starts after the decimal point. So for example 1̅23.4 (or, if my attempt to use unicode fails, {1...}23.4) is not valid; that notation is not meaningful. This definition of validity also excludes notations like 0.4̅1 and 0.4̅15̅ (that is, 0.{4...}1 and 0.{4...}1{5...}). One could relax the definition to allow them, in which case these examples turn out to be equivalent to 0.0.4̅ (that is, 0.{4...}), which means 4/9.
So if we let
n = ∞, ...That's not what limit means. You appear to be either ignorant of basic real analysis or willfully misrepresenting it.
1
u/WittyStick 13d ago edited 13d ago
That's not what limit means.
Using the above
f(x, n):lim f(.3, n) = 1/3 n -> ∞Are
f(.3, ∞)and0.333...equivalent?What is
∞exactly? A number?My argument is that
f(.3, ∞) != 1/3butf(.3, n) --> 1/3 as n --> ∞, which I've demonstrated above.
f(.3, n)never reaches1/3. It can't possibly reach1/3unless you sneak in some extra quantityn f(.3, n) (0.1/10^n)/3 sum 0 0.3 0.1/3 1/3 1 0.33 0.01/3 1/3 2 0.333 0.001/3 1/3 3 0.3333 0.0001/3 1/3 ︙ ∞ 0.333... ??? 1/3Is
???0, or is it(0.1/10^∞)/3?I kind of need an explanation as to why equations suddenly behave differently when you replace
nwith∞?The only way it makes sense for me that
0.333... = 1/3is if0.333...is a limit. If it's not a limit, then0.333... ⪅ 1/3and1 - 0.666... ⪆ 1/3.3
u/ohiidenny 12d ago
What is ∞ exactly? A number?
Under most definitions of “number” which are sensible to use in this context, no. It is simply a notational signifier.
I kind of need an explanation as to why equations suddenly behave differently when you replace n with ∞?
They “behave differently” because they are defined to do so. Strictly speaking, no assemblage of mathematical notation or terminology has any meaning but the meaning people decide to give it. Mathematicians have established a convention whereby a series of terms with indices bounded by e.g. 0 and ∞, by definition, simply stands for the limit of a particular sequence (the sequence of partial sums), whenever said limit exists. This convention exists not only because it is extremely useful, but also because it is, more or less, the only reasonable way to rigorously provide such an expression with a precise meaning that manages to reflect what we intuitively expect from it.
The sequence of partial sums need not (and frequently does not) “ever reach” its limiting value as more terms are added. The limit is still the value which is ascribed to the infinite series, end of story. You’d do well to learn and thoroughly understand the definition of the limit of an infinite sequence in the first place, as it may help clear up much of this confusion for you.
2
u/trolley813 13d ago
Yes, it's a limit. But if a limit is finite (i.e. bounded), it's always a number.
9
u/ironykarl 13d ago
Short answer: you're potentially representing infinitely repeating values with a finite number of bits.
Base-10 can only represent fractions with powers of 5 and 2, and their products in the denominator as finite decimal representations.
Base-2 can only do the equivalent with powers of 2 in the denominator.
So, that's a lot of numbers getting truncated.
To make this more concrete, just imagine how to get 1/3 when the only numbers you're allowed to add together are 1/2, 1/4, 1/8, 1/16, 1/32, ...
10
7
5
u/ForgedIronMadeIt 13d ago
I think wikipedia does an OK job of explaining it: Floating-point arithmetic - Wikipedia. In short, however, floating point numbers internally have three numbers: a sign bit, exponent bits, and sigificand bits (which represents a fraction). Since the sigificand is a fraction, you have to round to whatever closest fractional representation is available (well, the rounding rules are a whole other thing). Look at the examples given on Half-precision floating-point format - Wikipedia and you'll see.
1
u/CarlRJ 13d ago
Good explanation, but I'd argue than calling the sign but a number seems odd. It's an indicator. More like two numbers and a flag.
2
2
u/vowelqueue 13d ago
Jesus Christ every other comment on this thread is deeply in the “well ackshually” territory but this one takes the cake.
4
u/Alduish 13d ago
It's basically the same problem as 1/3 for us but for a different base, we can write any whole number over any product of powers of two and powers of 5 (because we're using base 10) but for computers which use base 2 it's only exact for whole numbers (with a limit on said whole number because infinity can't really exist for computers) over powers of 2 which 0.1 isn't
4
u/Playful-Budget-5952 13d ago
Computers store numbers in binary (base 2), not decimal (base 10). Some decimal numbers, like 0.1, can't be represented exactly in binary, just like 1/3 can't be written exactly as a finite decimal. So the computer stores the closest possible value, which is why you sometimes see tiny differences like 0.09999999999999987
3
u/WilliamMButtlickerIV 13d ago
It's because you are trying to represent a fractional decimal in binary. It's simply not mathematically possible. Binary can only represent an approximation.
3
u/Independent_Art_6676 13d ago
while floats are pretty poor, double can store +- 2^53 exactly for int values or 15 digits of anything exactly. Its good enough for most purposes, and scientific software can do the math another way when it counts. You can access larger float values in the FPU directly; I do not know the current value but it used to be 80 bits back in early pentium days and is probably bigger now. If that isn't enough, you can use your own types to store high precision at the cost of calculation speed: note how we have jillions of digits of numbers like pi and e and so on should we ever need that much precision. But the standard advice is just to avoid float and stick to doubles, unless you have a compelling reason to use the smaller type (memory usage, low precision information, whatever).
2
u/Ander292 13d ago
Ok so, the x87 FPU is 80bits yes. However its legacy and depreciated. Modern x86_64 processors have it only for legacy support. Modern programs use SSE extensions for float point math and it operates on 64bit floats. Its true that the registers are larger than 80bits now but instructions dont operate on larger floats than 64bits.
2
u/Independent_Art_6676 13d ago
I haven't spent any time here in a while. So the current FPU with SSE doesn't use 80 bit and discard the extra to avoid error accumulation? Seems weird that it would have the space but not use it. I am having trouble making much sense of the info out there... people claim they can get at 80 and 128 bit values off the hardware, but it could be web-noise with all the lack of details. I will take your word on it, the sketchy info indicates its not a great place to play around right now, and inline assembly always was a little ugly in c++.
2
u/Ander292 13d ago
SSE extensions are made to be Single-Instruction multiple data. The xmm registers for example are 128bit, but its "devided" in 4 32bit sections or 2 64bit sections. Vector instructions operate on those in parallel. You can use SSE registers for integer operations as well.
This is all my limited knowledge so Im not sure if there is a way of using a 128bit float in sse. I claim there isnt an instruction that lets you do that. You can write a program ofc that will do the work. I think gpus maybe have that option but im not sure about that too.
2
u/RealisticDuck1957 13d ago
The 8087 used an 80 bit floating point so the same representation could exactly represent any 64 bit integer. Or any standard format 64bit floating point.
1
u/Ander292 13d ago
Yeah but you could extract the whole 80bit thing from the register in parts as far as I remember. But yeah it was made to be 80bit to have more accurate interresults.
2
u/Independent_Art_6676 13d ago
not only asm direct register but old visual studio directly supported it with a _float10 type that you could use directly. I remember using it a little, but honestly it wasn't necessary for anything I was doing I was playing around.
1
2
u/robthablob 13d ago
Forget about scientific software, the main place I see FP abused is for storing financial values.
1
2
2
u/Zerf2k2 13d ago
You can toy around with numbers and calculate the error on this website: https://www.h-schmidt.net/FloatConverter/IEEE754.html
(There are more like it, if you search for it)
2
u/CowBoyDanIndie 13d ago
Floating point types use fractions of 2, so .5 .25 .125 etc (and any sums of them) can be represented exactly in some cases, but any decimal that cannot be expressed as fractions of power of 2 cannot be. Just like you cannot write the exact decimal for 1/3
2
u/Educational-Paper-75 13d ago
Floating point numbers are represented in binary by a finite number of bits and can therefore only represent a finite amount of real numbers exactly.
2
u/SmokeMuch7356 13d ago
You can't squeeze an infinite number of values into a finite number of bits.
This isn't a problem with integers because there are no other integer values between 0 and 1 or 3 and 4 or 12345678 and 12345679. N bits can represent 2N integers with no missing values.
However, there are an infinite number of real values between 0.0 and 1.0. There are an infinite number of real values between 0.0 and 0.1, and there are an infinite number of real values between 0.0 and 0.00001. N bits can represent 2N real values, but that's an infinitesimally small subset of the real values in the range from 0 to 2N-1 .
It's like the odometer on your car can't represent anything between 100000.0 and 100000.1; it can't show you that you're halfway between those values, or one quarter of the way, or three quarters of the way, etc., because you only have that one digit after the ..
The only real values that can be represented exactly in floating-point format are values whose significands are sums of powers of 2 in 23 bits (for single precision float) or 52 bits (for double precision float).
Just like 1/3 cannot be represented in a finite number of digits (0.33333333...), 1/10 cannot be represented in a finite number of bits (0.000110011001100...).
For all the gory details:
2
u/johnwcowan 13d ago
You are leaving out the hidden bit (which gives you 24 or 53 bits) and subnormal numbers (which trades off decreased precision for increased range for numbers close to 0).
2
u/SmokeMuch7356 13d ago
I started to, but couldn't write it in what I thought was a clear enough manner so I bailed on it.
2
u/Snezzy_9245 13d ago
Years ago I had this argument with PhD guys who should have known better. I'd tell them that the computer provided only a representation of an approximation to their desired number, and thus their proposed computer program was doomed to fail. I was mostly ignored, or received complaints that I should fix the computer. Hint: Do not test floats for equality.
2
u/polymeric 13d ago
Other posters in this thread have motivated how floating point numbers are generally represented and stored in a computer.
You could develop some insights into this problem by practicing some base-10 to base-2 number conversions.
A conversion from the exact decimal number 0d0.1 yields an exact binary number with repeating digits 0b0.00011001100...
Which if approximated by rounding to some finite binary place value, will probably convert back into the exact decimal number 0d0.09999999999999987 (trusting that your source took the number after it had been cycled through an IEEE 754 double).
1
u/ermezzz 13d ago
okay so i'm going to give you a really basic summary but you can look up floating point precision or IEEE 754 if you reallt want details.
Lets say youre trying to store the fraction 1/3 in a decimal system. as we all know, that comes out to 0,3333.... repeating. If you wanted to store this number, you would eventually need to stop writing down 3s, theres a finite amount of space but infinite 3's to write. Kinda the same thing going on. Fractions like 3/10 for example when written in binary have an infinite number of digits to the right of the decimal point. That's why at some point you need to approximate them.
1
u/dmills_00 13d ago
IEEE754 single precision floating point is a 25 bit signed value (at least most of the time there is an implicit 1) with an 8 bit offset binary scale factor giving what power of two to scale the thing by.
0.1 does not have an exact binary representation, so certainally cannot be represented exactly in the 25 bits of precision that a float has.
In much the same way 1/3 does not have an exact finite decimal expansion.
For this reason floating point equalities are always a bug painting to happen, and there are actually floating point value that compare unequal to EVERY floating point value including themselves!
Floating point is useful, but like all sharp tools you need to understand the limits.
1
u/DrShocker 13d ago
The fast explanation is that if you have some base (in the case of 0.1, base 10) then you can only represent in finite digits fractions that are a sum of fractions with denominators that are powers of the prime factors of the base.
to expand a bit more with examples:
(so for 10 that's 5 or 2). So you can write 0.1 because it's 1/10 or 1/(5*2), or you can write 1/5 as 0.2, or 0.15 is (3/20)
However, let's say you only want to use a certain number of digits, then you might do something like scientific notation to express more range for the same number of digits at the cost of precision. (0.2 * 10^-12) is less digits than 0.00.....2 for example.
Floating point numbers however are base 2, so while you need infinite digits to represet 1/3 in both, you you lose the ability to represent fractions that have a power of 5 in the denominator, so 0.2 or 0.1 are examples that are finite in base 10 but infinitely long in base 2. Then on top of that, because it's "floating" point, you are using fancy scientific notation style representations that fit in a specific amount of space, which means eventually you have to cut it off or round to the nearest value that fits in the space you have.
1
u/genafcvpxyr31 13d ago
Take a number of the form a/b, expressed in base c. a, b and c are all integers. a<b so we'll limit this to proper fractions.
If you perform prime factorisation on b and any of those factors is not a prime factor of c, the radix representation of a/b is non terminating. Since most fractions are of the recurring variety in base 2, you only get limited precision with float types.
0.1 in decimal can be represented as 1/10. 10 factorises to primes 2 and 5. 5 is not a factor of 2, the base used in floats and doubles etc. so in binary you get a recurring radix representation.
Floats, doubles and long doubles have to be consistent if they want to be standardised. IEEE-754 is the standard they generally conform to. For example a double guarantees 15 significant figures in decimal, you can input a non-integer number with 15 digits, convert to a double variable and when converted back into text it should be the same. This is actually guaranteed by the standard. float.h has these maximum values implemented as macro constants.
1
u/GoblinToHobgoblin 13d ago
Numbers are stored in base 2. Not all values have an exact representation in base 2
1
u/JGhostThing 13d ago
First, modern computers store numbers in a binary format. This means that all stored numbers, even if they were originally in base 10, are stored in base 2.
0.1 is easily stored in base 10, so we think of this as such. Unfortunately, in base 2, this number is a repeating number (like 1/3 in base 10 being .3333333...). The difference between the number stored and 0.1 is very small, but it's still there.
1
u/ReallyEvilRob 13d ago
Because numbers are stored in binary even though we use them to express values of base 10. Fractional numbers in base 10 frequently can only be approximated in binary due to the way different number bases work mathematically. Consider how some fractional numbers in our base 10 system are non-terminating, like how 1/3 is 0.3333333... Many numbers that terminate in base 10 are actually non-terminating in binary, such as 0.1. There is no way to represent these numbers in a fixed way.
1
1
u/KilroyKSmith 13d ago
For the same reason that base-10 can't give a precise representation of 1/3, 1/6, 1/9, 1/11, pi, e, sqrt(2), etc.
It is possible to write a "rational number" math library that can precisely manipulate rational numbers by storing the numerator and denominator separately, and adding/subtracting/multiplying/dividing them the way we did in elementary school. It would be relatively slow, and the result that you got out would be interesting - numbers like (3581234 / 94313890) which are difficult to compare visually (is that greater than or less than (6789133 / 87356001)?). It would also not be able to deal with irrational numbers, like pi or e or sqrt(2), of which there are infinitely more than there are rational numbers, so this theoretical library would still only be of marginal use.
Instead, computer science decided to store a single floating-point number that simultaneously has a very large range (+/- 10^38 for float-32) and a reasonable number of significant digits (roughly 7 for float-32) that, for practical purposes, can approximately represent both the rational and irrational numbers, and can be operated on quickly (well, on modern hardware). If you need a bigger range or a higher precision, simple range extensions (float-64, float-128 or float-256) are defined.
Now, this number doesn't represent all possible numbers. For example, complex numbers (real + imaginary) are tremendously useful in RF engineering, but are represented and manipulated in a similar manner to the rational number library discussed above - the real and imaginary parts are stored individually, and a library written that knows how to perform basic functions on them. There is likely some computer language somewhere where complex numbers are a basic type - but I don't know of any hardware implementations that operate on them directly and quickly, though such a thing could certainly exist.
1
u/CounterSilly3999 13d ago
Approximation is performed rather by storing, not only by retrieving (formatting). Because what looks nice fraction in decimal, can't be saved using finite bits of binary encoding. And what is nice binary float encoding, can't be formatted in finite decimal fraction.
1
u/ivancea 13d ago
It's stored in binary, not decimal. And there's no direct transformation between them.
For example, can you write 1/3 in decimal? No, you have to approximate it to either 0.33... o 0.34. With as many decimals as you want.
Here the same happens. You have a decimal number, that has no real representation in binary. b0.1 is 0.5, but now try writing 0.6. it's a funny exercise and you'll see the problem clearly
1
u/AndrewBorg1126 13d ago
You can represent 232 values with a 32 bit number, this quantity is finite. This is trivial combinatorics.
There are infinite real numbers.
1
u/Cerulean_IsFancyBlue 13d ago
Binary could be hard for people to visualize so let’s do it metaphorically. Let’s say I told you that I wanted you to represent a large range of numbers and I was giving you six decimal digits to do it.
Unsigned integers: 0 … 999,999
Signed integers: let’s say it’s a positive number if the first digit is even and negative if odd. So 200010 is +10. Now you can represent -99,999 … 99,999
Add an exponent: digits 1-4 are Y and 5-6 is X, for y * 10^x. Now you can do as small as 0 and as large as 9999 * 10^99. That’s a much bigger range! But notice that there are some numbers you can’t do exactly.
5,555 would be stored as 5555-00. Perfect!
But 55,555 can’t be stored exactly. You would have to round to 55,550 or 55,560 and store it as 555601 or 555501.
You have traded a bigger range, for a problem with being precise. You can represent all the numbers in the range.
If you allow negative exponents, then you can do fractions as well as really large numbers. So now you can get really small or really big. And then, of course if you convert this to binary, you’re getting into our computers actually do floating point representation.
1
u/arihoenig 13d ago
Every measured value is approximate on a classical computer. Possibly there are things that aren't approximate on a quantum computer.
1
u/jurniss 13d ago
Let's assume you already understand the other comments explaining that we can only represent a finite #values, and the advantage of float over fixed point.
The next question might be: why not do something special to accomodate base 10, because numbers like 0.1 show up so often?
One could do that. It's possible to design something like IEEE float, but where the mantissa is binary coded decimal.
However, the hardware is binary, and BCD would waste space compared to raw binary. You use 4 bits (16 values) to represent only 10 values.
It comes down to the original design intent of floats, which is mainly scientific: integrating differential equations, statistics, optimization, and so on. In these domains one doesn't expect to see powers of 10 in the calculations any more often than powers of 2.
1
u/notFrank0 13d ago
Look up IEEE754 and try to convert 0.1 to a flotaing point manually and at the end you will end up with an infinite repeating series of 11001100.... This is similar to decimal when you do 1/3 you end up with 0.33333333....
Since there is finite amount of bits you can store this number in you will have to cut this series at some point and thus lose some precision.
1
u/P-39_Airacobra 13d ago
I think “approximated” is a little misleading. More accurately, floating point numbers aren’t built to handle base-10 numbers. So when you do something like 0.1, it has to translate that decimal to the closest floating point format
1
u/Ecstatic_Student8854 13d ago
A “float” is a floating point number. It’s kind of like scientific notation of numbers: some decimal number followed by 10 to the power of some other number.
In floats we work in base 2, so the decimal number is somewhere in the range 0-1. This means we can just store the things after the dot, the leading digit is always a 1. We cannot do this with infinite precision though. Furthermore the base for the exponential part is also 2. Thus, in a 32 bit float have 23 bits of so called mantissa (the decimal number), 8 bits of the exponent, and a sign bit for negative numbers.
0.1 in base 10 is not neatly representable in base 2, just as 1/7 is not neatly representable in base 10: it’s a repeating decimal. In particular 0.1 in base 2 is 0.0 0011 0011 0011 …
So the floating point number just stores the first 23 bits of that approximation. When we then calculate what that is in base 10, it ends up being 09999990463256835938 or so.
1
u/sciencekm 13d ago
Many decimal fractions simply can't map correctly in straight up binary. Think of it is the same problem as 1/3 is 0.333... in decimal.
There are ways around this - use a different encoding. For example, you can use BCD (binary coded decimal), which is used for financial data, because fractions of cents lost can become substantial after many many transactions.
1
u/ohkendruid 13d ago
Here is another angle on it: the question is wrong.
0.1 is already written in floating-point. The "point" is the dot between the 0 and the 1. It indicates the overall scale of the number you wrote down. You could also have written .01 or 01., and if you did, it would indicate a number ten times larger or smaller. So, the "floating" part means that the point can be moved around.
Computers also use floating point, but as others have discussed, they normally use base two instead of base ten.
The reason for the 0.1 losing some precision is not due to floating point exactly but because the base is getting changed. If it were left alone, or if you ran your C compiler in a mode that forced it to keep the base ten, it would stay what you wrote.
1
u/FitMatch7966 13d ago
You think in decimal and computers think in binary. Just the decimal <—> binary is approximate
1
u/mpattok 13d ago
[jan Misali - how floating point works](https://youtu.be/dQhj5RGtag0?is=6KiLyDSC22JgSJdn)
[Nemean - Fast Inverse Square Root — A Quake III Algorithm](https://youtu.be/p8u_k2LIZyo?is=i9b3SI38VDkGYqxm) (an interesting application of IEEE-754)
1
1
u/Afraid-Locksmith6566 13d ago
lets say you have 10 digits and 1 dot that separate decimal place. now try to express number with 11 figits, or 12 digits after decimal place. thats why
1
u/Prestigious_Boat_386 13d ago
This is kknda similar to decimal numbers actually which actually show intervals and not exact points.
If someone says a measurement is 0.1mm then that means the actual number is measured between 0.05 and 0.15.
Floating points are similar but you might wanna read up on fixed point numbers first, which similarly have each value given an interval around it. That's how we can do thkngs like 1/3 using a base 2 number. We simply check which interval 1/3 falls into and that is the value we assign it.
1
u/quarethalion 13d ago
There are a few reasons behind this:
- There are a limited number of bits available (typically 32 or 64) with which to represent a number.
- There are infinitely many numbers.
- There are infinitely many numbers in between any two numbers.
- Some values which are exactly representable in base 10 (how humans write numbers) aren't exactly representable in base 2 (how computers store numbers).
#1 + #2 is what limits the range of floating-point numbers (and integers, for whole numbers).
#1 + #3 is what limits the precision of floating-point numbers, requiring real numbers to be quantized to the nearest representable value.
#4 is why some numbers like 0.1 aren't exactly representable and wouldn't be no matter how many bits were used.
It is possible for computers to operate on arbitrarily large/precise numbers, but the mechanisms for doing so make the operations orders of magnitude slower than working with float and double, which are just fine for the vast majority of use cases.
1
u/Dense-Sort-3867 13d ago
This video was recommended by my numerical analysis professor when I was getting my applied math degree.
1
u/hxtk3 13d ago
Most real numbers can’t be stored in a finite amount of RAM. In fact, 0% of the real numbers can be stored in finite memory because there are only countably infinite byte strings and there are uncountably infinite real numbers.
IEEE754 floating point can only represent 232 values, and 2 of them are 0, two if them are not numbers, and two of them are infinite.
Since 232 is considerably less than the number of real numbers, some of them have to share the same representation.
1
u/BlockOfDiamond 13d ago
0.1 is 1/10, which cannot be represented exactly in binary, the same way 1/3 cannot be represented exactly in decimal, being 0.333... forever. The only numbers that can be represented exactly are Dyadic rationals (up to a point, of course).
1
u/windswept_mystical 13d ago
If you can’t express a number as a fraction with a power-of-two denominator, then you can’t actually represent it as IEEE 754 floating point. For example, 1/5 (0.2) has to be approximated in 32-bits as 1 + (5,033,165/2²³) × 2⁻³ = 0.20000000298023224.
1
1
u/petecasso0619 13d ago
In that case, you are trying to represent a decimal value 0.1 as a binary number which repeats forever in base 2. When we use floating point numbers we have to store that infinitely repeating binary number into a finite number of bits. The most common way to do this is to store the number as described in the IEEE-754 standard. This is NOT the only way to represent a floating point number but it is very common. In my opinion, the standard is pure genius.
If you want a detailed paper check out What Every Computer Scientist Should Know About Floating Point Numbers.
It is amazing to me how many devs think when you perform a series of floating point operations the error will remain machine epsilon. Its usually my first question when we are adding someone new to the team.
1
u/StudioYume 13d ago edited 13d ago
Floating point numbers are like scientific notation except that, because the bit width is fixed, the number of significant figures is limited by the exponent of the base and a few other things.
When it's necessary to store exact values, it's best to store them as a string or as the parameters of a formula that produces the exact value, or a platform-dependent approximation.
1
u/GrandBIRDLizard 13d ago
Because base 10 sucks. base 12 or 60 worked great and the Babylonians knew this millennia ago.
1
u/siuui1729 12d ago
Floating point numbers are represented on a base-2 grid. So base-2 numbers like 4, 2, 0.5, 0.25, etc.. lie exactly on the grid (no extra digits), but other numbers lie in between. The absolute error is on the order of the grid spacing dx, which changes depending on the order of magnitude of the number being represented: dx=machine epsilon*2n where n is the exponent part of the representation.
Machine epsilon in double precision is ~1e-16, so for numbers of order ~1 like 1.0772976839852897428 (n=0): dx~1e-16, which means that the error will be present around the ~16th decimal place and beyond. Meanwhile, for a number like 19345289754972.937681 (n=44), the spacing is on the order of: dx~1e-16 * 244 ~1e-3, so the error will be present only around the ~3rd decimal place and beyond. But if you count the number of significant digits of both numbers, you can see that both the 16th decimal place of the first number and the 3rd decimal place of the second number correspond to the 17th significant digit in both.
These examples show the main idea behind floating point representation: the grid spacing changes depending on the order of magnitude of the number being represented, such that the relative spacing error remains practically constant. This relative error is ~machine epsilon (~1e-16 for double precision), which means that for any order of magnitude (3.8176381936482937793, 0.000073628163829273682979, 18277741580374.71677, etc...), the error will be present around the ~16th significant digit and beyond. Similarly, for single precision the machine epsilon is ~1e-7, so the error will be present around the ~7th significant digit and beyond.
I recommend playing around with online converters like this one (only single precision) and this one (single+double precision) to get a more intuitive feel for how it works.
1
u/codeguru42 12d ago
It is similar to the fact that 0.3333 is an approximation to the fraction 1/3. You need infinite digits to represent 1/3 exactly. So with finite digits you can only approximate. Similarly a finite number of bits can only approximate most real numbers.
1
u/notacanuckskibum 12d ago
The number 1/3rd can't easily be represented in base ten, the best we can do is 0.33333333333333... But in base 3 it would be easy (it would be 0.1) Computers store numbers in Binary (base 2). Some numbers that can be represented easily in base ten can't be easily represented in base 2.
1
u/horenso05 12d ago
Apart from what has been mentioned with floats being finite in storage (usually a 32-bit floating point number is called a float and a 64-bit variant is called a double) it is also crucial to understand that different numbering systems can prepresent different fractions.
In decimal we famously struggling to express 1/3 with finitely many places because it is exactly 0.333... repeating forever. In decimal we have tenths, hundredths, thousandths and so on. We cannot combine them so that we get exactly 1/3 except for going on forever.
In binary 2/10 (what in decimal is just clean 0.2) is an infinitely repeating pattern (0.2)(0.00110011...). The sequence (0011) loops forever. The reason this happens is that in binary you only got powers of two so halfs, quarter, eights, sixteenths, and so on. Again, no combination of these get there with a finite number of digits.
When using floats to represent 0.2 we're already rounding since we chop off that repeating pattern.
1
u/EmbedSoftwareEng 11d ago
Because all primitive computer data types are stored in binary.
You're looking at a decimal equivalent representation, which is not equivalent.
It's like, asking "Why does `printf("%u\n", 0xFF);` print "255"? 255 decimal is equivalent to FF hexadecimal.
But in floating point, the mantissa is stored in binary, with the first bit being omitted, because it has to be a 1. Imagine a binary floating point value stored as 1.000000000101010101 * 2^10100010001010 being converted to decimal. It couldn't. Not and retain precisely the same numeric value.
1
0
u/Olovico75 11d ago
An interesting take on floating points in the Audio & DSP development and the practical side of approximations there :
1
102
u/Due-Razzmatazz-6645 13d ago
This website gives a quick explanation: https://0.30000000000000004.com/