[Gambas-user] Float hidden precision?

[Jussi Lahtinen]

Floating points are always approximations to decimals in all languages,
because of how computers work.

Dim num1 As Single = 4.2
Dim num2 As Single = 4.3

Print num2 - num1
''0.1000004

Use integers like Benoit suggested or if speed isn't critical use Strings
and do the calculations with bc.

Excellent wiki article for further information:
https://en.wikipedia.org/wiki/Floating-point_arithmetic


Jussi


On Tue, Nov 17, 2020 at 2:48 PM KKing <kicking177 at gmail.com> wrote:

> Is it possible for a float variable to have unwanted hidden distortion
> in a held value?
>
> I have a conundrum buried in a lot of code but essentially I have two
> monetary values which originally arrive as strings in a csv file.
>
> I then convert these to float values for other processing but by
> debugging I have come across the following issue where essentially the
> value has been distorted.
>
> I've summarised the code below, and as a standalone test it works as
> expected but as part of much larger project I get the result below.
>
> The question is how can a float value in debugger show 429761 and when
> converted to integer ends up as 429760 ?
>
> This occurs in a number of times with a large amount of data being
> passed. If I step through the code I can see in one instance the example
> below.
>
> So when a float is displayed in the debugger as 4297.61 is it possible
> Gambas really believes it is 4297.609 (even though the string was
> definitely "4297.61")
> and when 429761 is seen that actually it is being held has  429760.9 and
> when the CInt() is done it's truncating to 4297.60 and 429760 respectively?
>
> I had fun with this back QB4 and powerbasic days and ended up writing a
> whole load of routines to handle monetary values as integers and
> converting to and from strings by my own routines.
>
> NB I've only added the * 100 and integers once I suddenly had Gambas
> seemingly telling me that 4297.61 did not equal 4297.61
>
>
> Dim monValue1 As Float
> Dim monValue2 As Float
>
> Dim strValue1 as string
> Dim strValue2 as string
>
> strValue1 = "4297.61"
> strValue2 = "4297.61"
>
> monValue1 = CFloat(strValue1)
> monValue2 = CFloat(strValue2)
>
> Process(ByRef monValue1 , ByRef monValue2 )
>
>   - - - - -
>
> Public Sub Process(ByRef monValue1 as Float, ByRef monValue2 as Float)
>
> Dim monValue1_Adj As Float
> Dim monValue2_Adj2 As Float
>
> Dim intValue1_Adj As Integer
> Dim intValue2_Adj2 As Integer
>
> ' value seen in debugger monValue1 = 4297.61
> ' value seen in debugger monValue2 = 4297.61
>
> monValue1_Adj = monValue1 * 100         ' value seen in debugger 429761
> monValue2_Adj = monValue2 * 100         ' value seen in debugger 429761
>
> intValue1_Adj = CInt(monValue1_Adj)      ' value seen in debugger 429760
> intValue2_Adj = CInt(monValue2_Adj)      ' value seen in debugger 429761
>
> End
>
>
>
>
>
>
>
> ----[ http://gambaswiki.org/wiki/doc/netiquette ]----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gambas-basic.org/pipermail/user/attachments/20201117/ff4e3286/attachment.htm>