[Gambas-user] Float hidden precision?
Benoît Minisini
g4mba5 at gmail.com
Tue Nov 17 14:06:02 CET 2020
Le 17/11/2020 à 13:47, KKing a écrit :
> Is it possible for a float variable to have unwanted hidden distortion
> in a held value?
>
> I have a conundrum buried in a lot of code but essentially I have two
> monetary values which originally arrive as strings in a csv file.
>
> I then convert these to float values for other processing but by
> debugging I have come across the following issue where essentially the
> value has been distorted.
>
> I've summarised the code below, and as a standalone test it works as
> expected but as part of much larger project I get the result below.
>
> The question is how can a float value in debugger show 429761 and when
> converted to integer ends up as 429760 ?
>
Never use floating point for storing currencies. Always use integers.
Floating point is binary, and a number with a finite number of decimal
digits in base 10 often has an infinite number of decimal digits in
binary, and so you have rounding error by converting between the two bases.
--
Benoît Minisini
More information about the User
mailing list