[Gambas-user] Float hidden precision?
KKing
kicking177 at gmail.com
Tue Nov 17 13:47:12 CET 2020
Is it possible for a float variable to have unwanted hidden distortion
in a held value?
I have a conundrum buried in a lot of code but essentially I have two
monetary values which originally arrive as strings in a csv file.
I then convert these to float values for other processing but by
debugging I have come across the following issue where essentially the
value has been distorted.
I've summarised the code below, and as a standalone test it works as
expected but as part of much larger project I get the result below.
The question is how can a float value in debugger show 429761 and when
converted to integer ends up as 429760 ?
This occurs in a number of times with a large amount of data being
passed. If I step through the code I can see in one instance the example
below.
So when a float is displayed in the debugger as 4297.61 is it possible
Gambas really believes it is 4297.609 (even though the string was
definitely "4297.61")
and when 429761 is seen that actually it is being held has 429760.9 and
when the CInt() is done it's truncating to 4297.60 and 429760 respectively?
I had fun with this back QB4 and powerbasic days and ended up writing a
whole load of routines to handle monetary values as integers and
converting to and from strings by my own routines.
NB I've only added the * 100 and integers once I suddenly had Gambas
seemingly telling me that 4297.61 did not equal 4297.61
Dim monValue1 As Float
Dim monValue2 As Float
Dim strValue1 as string
Dim strValue2 as string
strValue1 = "4297.61"
strValue2 = "4297.61"
monValue1 = CFloat(strValue1)
monValue2 = CFloat(strValue2)
Process(ByRef monValue1 , ByRef monValue2 )
- - - - -
Public Sub Process(ByRef monValue1 as Float, ByRef monValue2 as Float)
Dim monValue1_Adj As Float
Dim monValue2_Adj2 As Float
Dim intValue1_Adj As Integer
Dim intValue2_Adj2 As Integer
' value seen in debugger monValue1 = 4297.61
' value seen in debugger monValue2 = 4297.61
monValue1_Adj = monValue1 * 100 ' value seen in debugger 429761
monValue2_Adj = monValue2 * 100 ' value seen in debugger 429761
intValue1_Adj = CInt(monValue1_Adj) ' value seen in debugger 429760
intValue2_Adj = CInt(monValue2_Adj) ' value seen in debugger 429761
End
More information about the User
mailing list