[Gambas-user] Float hidden precision?

KKing kicking177 at gmail.com
Tue Nov 17 13:47:12 CET 2020


Is it possible for a float variable to have unwanted hidden distortion 
in a held value?

I have a conundrum buried in a lot of code but essentially I have two 
monetary values which originally arrive as strings in a csv file.

I then convert these to float values for other processing but by 
debugging I have come across the following issue where essentially the 
value has been distorted.

I've summarised the code below, and as a standalone test it works as 
expected but as part of much larger project I get the result below.

The question is how can a float value in debugger show 429761 and when 
converted to integer ends up as 429760 ?

This occurs in a number of times with a large amount of data being 
passed. If I step through the code I can see in one instance the example 
below.

So when a float is displayed in the debugger as 4297.61 is it possible 
Gambas really believes it is 4297.609 (even though the string was 
definitely "4297.61")
and when 429761 is seen that actually it is being held has  429760.9 and 
when the CInt() is done it's truncating to 4297.60 and 429760 respectively?

I had fun with this back QB4 and powerbasic days and ended up writing a 
whole load of routines to handle monetary values as integers and 
converting to and from strings by my own routines.

NB I've only added the * 100 and integers once I suddenly had Gambas 
seemingly telling me that 4297.61 did not equal 4297.61


Dim monValue1 As Float
Dim monValue2 As Float

Dim strValue1 as string
Dim strValue2 as string

strValue1 = "4297.61"
strValue2 = "4297.61"

monValue1 = CFloat(strValue1)
monValue2 = CFloat(strValue2)

Process(ByRef monValue1 , ByRef monValue2 )

  - - - - -

Public Sub Process(ByRef monValue1 as Float, ByRef monValue2 as Float)

Dim monValue1_Adj As Float
Dim monValue2_Adj2 As Float

Dim intValue1_Adj As Integer
Dim intValue2_Adj2 As Integer

' value seen in debugger monValue1 = 4297.61
' value seen in debugger monValue2 = 4297.61

monValue1_Adj = monValue1 * 100         ' value seen in debugger 429761
monValue2_Adj = monValue2 * 100         ' value seen in debugger 429761

intValue1_Adj = CInt(monValue1_Adj)      ' value seen in debugger 429760
intValue2_Adj = CInt(monValue2_Adj)      ' value seen in debugger 429761

End








More information about the User mailing list