Page 1 of 1
Why are value converted to floats in a strange way?
Posted: Sunday 07 June 2020 15:26
by EddyG
It seems that value's are converted to floats in a strange way.
Code: Select all
Woonkamer Temperatuur: 21.200000762939
This value is in the database and display 21.2
Why is that?
Re: Why are value converted to floats in a strange way?
Posted: Sunday 07 June 2020 15:41
by ronaldbro
That's because of the precision of the data type. You will see the same in Excel when you show a lot of decimals.
You can use string.format to specify how many decimals you want.
Try string.format("%.0f", 21.200000762939)
Re: Why are value converted to floats in a strange way?
Posted: Sunday 07 June 2020 16:08
by waaren
EddyG wrote: Sunday 07 June 2020 15:26
It seems that value's are converted to floats in a strange way.
Code: Select all
Woonkamer Temperatuur: 21.200000762939
This value is in the database and display 21.2
Why is that?
Short explanation:
because of the way the underlying platform handles floating point numbers, and ultimately with the inaccuracy you’ll always have when writing down numbers as a string of a fixed number of digits.
Long explanation
Re: Why are value converted to floats in a strange way? [Solved]
Posted: Sunday 07 June 2020 16:25
by EddyG
Tnx, That explains a lot.
