Page 1 of 1

Why are value converted to floats in a strange way?

Posted: Sunday 07 June 2020 15:26
by EddyG
It seems that value's are converted to floats in a strange way.

Code: Select all

Woonkamer Temperatuur: 21.200000762939
This value is in the database and display 21.2
Why is that?

Re: Why are value converted to floats in a strange way?

Posted: Sunday 07 June 2020 15:41
by ronaldbro
That's because of the precision of the data type. You will see the same in Excel when you show a lot of decimals.

You can use string.format to specify how many decimals you want.
Try string.format("%.0f", 21.200000762939)

Re: Why are value converted to floats in a strange way?

Posted: Sunday 07 June 2020 16:08
by waaren
EddyG wrote: Sunday 07 June 2020 15:26 It seems that value's are converted to floats in a strange way.

Code: Select all

Woonkamer Temperatuur: 21.200000762939
This value is in the database and display 21.2
Why is that?
Short explanation:
because of the way the underlying platform handles floating point numbers, and ultimately with the inaccuracy you’ll always have when writing down numbers as a string of a fixed number of digits.
Long explanation

Re: Why are value converted to floats in a strange way?  [Solved]

Posted: Sunday 07 June 2020 16:25
by EddyG
Tnx, That explains a lot. :)