[Gambas-user] Calculation Bug?
Pablo Vera
pvera at ...729...
Tue Nov 28 00:54:30 CET 2006
The problem is that your calculation is made up of Integers, so the
partial results are also integers and this causes rounding errors.
To solve it, start with a floating number, so that all the calculations
are done as a Float, like this:
fTmp1 = 1.0 * 618 * 2 * 60 * 1000000 / (618 * 2) / 360
Which results in: 166666.666666667
Saludos,
Pablo Vera
James Linden wrote:
> Gambas does not appear to know how to handle 8 digit numbers...
>
> Dim fTmp1 As Float
> fTmp1 = 60 * 1000000
> Message.Info( CStr( fTmp1 ) )
>
> -- James
>
> PS: Isn't your formula a bit redundant -- you multiply by (618*2), then
> divide by (618*2)... ?
>
> R. Stormo wrote:
>> I'm doing a small calculation for a CDI box to a 4 wheel bike. And when I
>> putting the function togheter it does not calculate the right values
>> according to the calculator.
>>
>> >From Debug.
>> Dim fTmp1 as Float
>> fTmp1 = 618 * 2 * 60 * 1000000 / (618 * 2) / 360 =2574.51449119022
>>
>>
>> The calculator gives the value
>>
>> 166666,666666667
>>
>> Spreadsheet gives
>> 166666,666666667
>>
>> So there must be something wrong in gambas when calculation is done.
>>
>> Because if I take one zero from the 1000000 it gives me a total different
>> value
>> Dim fTmp1 as Float
>> fTmp1 = 618 * 2 * 60 * 100000 / (618 * 2) / 360 =-2638.29241280115
>>
>> Should have been 16666,6666666667
>>
>>
>>
>> Regards R. Stormo
>>
>> My Gambas Community http://gambasforum.tk
>>
>>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Gambas-user mailing list
> Gambas-user at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gambas-user
>
More information about the User
mailing list