## floating number error with a simple substraction

Community Trekker

Joined:

May 25, 2016

I cant say if JMP can substract two simple numbers correctly but I tried the following numbers:

16.11-15.12

0.99 (ok, so far so good)

15.29-15.18

0.109999999999999 (not so good any more)

15.29-15.19

0.0999999999999997 (a seven?)

So where is the hook I placed wrong?

This happens with JMP 12.2.0 (64-bit) in the script editor.

2 ACCEPTED SOLUTIONS

Accepted Solutions

Staff

Joined:

Jun 23, 2011

Solution

You might want to read this resource: JMP Numerical Accuracy You can find such white papers for other popular software such as spreadsheets. It is a know issue but generally not a problem.

Learn it once, use it forever!

Staff

Joined:

Jun 23, 2011

Solution

Regarding "significant digits", you can get that by using the "Precision" format with the "Format" function. Also, the "Char" function supports two addition arguments, one for the width and one for the number of decimal places, if you just want to skip the Round operation for display.

``````Char( 15.29 - 15.19, 10, 2); // "0.10"
Format( 12.345, "Precision", 3); // "12.3"
Format( 1.2345, "Precision", 3); // "1.23"``````

Regarding the original issue, it boils down to the fact that computers store numbers in base 2 (binary) but display them in base 10, and some base 10 numerals like "0.1" have no exact representation in base 2.

9 REPLIES

Staff

Joined:

Jun 23, 2011

In mathematics, simple arithmetic would yield exact results in these differences: 15.29 - 15.18 = 0.11 and 15.29 - 15. 19 = 0.10. But the result in JMP is finite precision arithmetic in a computer. The result depends on how the real number is represented as a floating point number and how the arithmetic operators work.with this representation. It is not difficult to find cases where the result of the operator and the mathematical result do not agree.

JMP might use a standard floating point numeric software library or a dedicated hardware floating point processor for such operations.

There are some softwares that use extended or 'unlimited' precision (limited only by physical resources) but it generally doesn't change the practical results in our statistical methods, such as regression.

One solution in your case is to round the result in a formula or script:

``Round( 15.2900000 - 15.1900000, 2 )``

Then you get 0.1 for the result. Problem solved.

Learn it once, use it forever!

Community Trekker

Joined:

May 25, 2016

not quiet, to "exten" it a bit (and the reason why its bugging me) is that the amount of precision depend on user input

If he uses 15.29mm and 15.19mm he will get the correct 0.1mm with Round(x,2).

If he uses 0.01529m and 0.01519m he will get 0m and will probably stand in my door the seccand after he sees the result *g*

And I can't see if "significant digits" is supported like round or char.

Staff

Joined:

Jun 23, 2011

I understand. My example with the Round() function was only meant to illustrate one method for dealing with the limited precision of the floating point number and associated arithmetic.You can build richer solutions to handle more complex cases.

Did you use Help > Books > Scripting Guide or Help > Scripting Index to see the capabilities of the Round() or Char() functions?

Learn it once, use it forever!

Staff

Joined:

Jun 23, 2011

Solution

Regarding "significant digits", you can get that by using the "Precision" format with the "Format" function. Also, the "Char" function supports two addition arguments, one for the width and one for the number of decimal places, if you just want to skip the Round operation for display.

``````Char( 15.29 - 15.19, 10, 2); // "0.10"
Format( 12.345, "Precision", 3); // "12.3"
Format( 1.2345, "Precision", 3); // "1.23"``````

Regarding the original issue, it boils down to the fact that computers store numbers in base 2 (binary) but display them in base 10, and some base 10 numerals like "0.1" have no exact representation in base 2.

Community Trekker

Joined:

May 25, 2016

I endet up using

`format(mean(y),6)`

so in the report a very small number gets converted into 1.574e-6 and don't use up a lot of space.

but

`Format( 1.2345, "Precision", 3)`

would be the way for  "significant digits".

Staff

Joined:

Jun 23, 2011

Solution

You might want to read this resource: JMP Numerical Accuracy You can find such white papers for other popular software such as spreadsheets. It is a know issue but generally not a problem.

Learn it once, use it forever!

Joined:

Apr 3, 2013

Just to illustrate the issue is NOT a JMP issue, here is what happens in Excel.

 X Y X - Y Formula 16.11 15.12 0.99000000000000000 15.29 15.18 0.10999999999999900 15.29 15.19 0.09999999999999960

Dan Obermiller

Community Trekker

Joined:

May 25, 2016

THX for the inputs, small question, do you consider an option as "n significant digits" for show, char and round? that would help a lot.

Staff

Joined:

Jun 23, 2011

I am not sure what you are asking. If you are asking if these functions have the option to specify significant digits as well as the number of decimal places, then the answer is no. If you are asking if I would consider adding such an option, then that request is a matter for JMP Development.

Learn it once, use it forever!