[Gambas-user] Best ways to format float values

[Tobias Boege]

On Thu, 13 Jul 2017, Gianluigi wrote:
> I would not be misunderstood.
> I had understood that Alex wanted a forced increase "Round".
> 
> To recap:
> 
>   Dim n As Float = 26.6601666666666
>   Dim b As Byte
> 
>   Print Int(n * 100) / 100              ' Normal truncate, as already
> mentioned by other
>   Print Round(n, -2)                    ' Normal round, as already
> mentioned by other
>   If Len(CStr(Frac(n))) > 4 Then
>     b = Val(Mid(CStr(Frac(n)), 5, 1))
>     If b >= 5 Then
>       Print Round(n, -2)
>     Else
>       Print Round(n + 0.01, -2)         ' Forced increase round, as
> proposed by me
>     Endif
>   Endif
> 

I would avoid arithmetic operations involving even more floats, such as you
proposed. Consider this:

  $ gbx3 -e 'Round(0.80999999950+0.1,-9)'
  0.909999999
  $ gbx3 -e 'Round(0.80999999951+0.1,-9)'
  0.91

I only changed the very last digit (of order 10^-11) from 0 to 1 which
shouldn't influence the rouding to 9 decimals at all -- but it does!

The problem here is, famously, that the decimal 0.1 has no finite binary
representation, so *storing* the value that is represented in decimal by
the string "0.1" in a binary float already gives you an unavoidable error.
Whenever you use 0.1 in your program, this error propagates. Specifically,
the error when storing 0.1 in a Single is about 1.49*10^-9, which is how
I arrived at that example above.

Of course, the same applies to the 0.01 you use above. You can think
by yourself about a similar example where float addition with 0.01 makes
the result of a later rounding unreliable.

The other method

  Floor(n*100)/100  ' round down to two decimals
  Ceil(n*100)/100   ' round up to two decimals

is reliable, since the *integer* 100 can be stored without error in a
float and IEEE754 requires the outcome of float arithmetic to be the
same as if the operation was performed exactly and then rounded to the
limited precision of the float datatype [1].

Regards,
Tobi

[1] http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html#865

-- 
"There's an old saying: Don't change anything... ever!" -- Mr. Monk