Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Discussions
- :
- floating number error with a simple substraction

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 4:08 AM
(3304 views)

I cant say if JMP can substract two simple numbers correctly but I tried the following numbers:

16.11-15.12

0.99 (ok, so far so good)

15.29-15.18

0.109999999999999 (not so good any more)

15.29-15.19

0.0999999999999997 (a seven?)

So where is the hook I placed wrong?

This happens with JMP 12.2.0 (64-bit) in the script editor.

2 ACCEPTED SOLUTIONS

Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 6:55 AM
(6474 views)

Solution

You might want to read this resource: JMP Numerical Accuracy You can find such white papers for other popular software such as spreadsheets. It is a know issue but generally not a problem.

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 16, 2017 6:44 AM
(6255 views)

Solution

Regarding "significant digits", you can get that by using the "Precision" format with the "Format" function. Also, the "Char" function supports two addition arguments, one for the width and one for the number of decimal places, if you just want to skip the Round operation for display.

```
Char( 15.29 - 15.19, 10, 2); // "0.10"
Format( 12.345, "Precision", 3); // "12.3"
Format( 1.2345, "Precision", 3); // "1.23"
```

Regarding the original issue, it boils down to the fact that computers store numbers in base 2 (binary) but display them in base 10, and some base 10 numerals like "0.1" have no exact representation in base 2.

9 REPLIES

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 6:40 AM
(3282 views)

In *mathematics*, simple arithmetic would yield exact results in these differences: 15.29 - 15.18 = 0.11 and 15.29 - 15. 19 = 0.10. But the result in JMP is *finite precision arithmetic* in a computer. The result depends on how the real number is represented as a floating point number and how the arithmetic operators work.with this representation. It is not difficult to find cases where the result of the operator and the mathematical result do not agree.

JMP might use a standard floating point numeric software library or a dedicated hardware floating point processor for such operations.

There are some softwares that use extended or 'unlimited' precision (limited only by physical resources) but it generally doesn't change the practical results in our statistical methods, such as regression.

One solution in your case is to round the result in a formula or script:

`Round( 15.2900000 - 15.1900000, 2 )`

Then you get 0.1 for the result. Problem solved.

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 6:55 AM
(3271 views)

not quiet, to "exten" it a bit (and the reason why its bugging me) is that the amount of precision depend on user input

If he uses 15.29mm and 15.19mm he will get the correct 0.1mm with Round(x,2).

If he uses 0.01529m and 0.01519m he will get 0m and will probably stand in my door the seccand after he sees the result *g*

And I can't see if "significant digits" is supported like round or char.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 6:59 AM
(3267 views)

I understand. My example with the Round() function was only meant to illustrate one method for dealing with the limited precision of the floating point number and associated arithmetic.You can build richer solutions to handle more complex cases.

Did you use **Help** > **Books** > **Scripting Guide** or **Help** > **Scripting Index** to see the capabilities of the Round() or Char() functions?

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 16, 2017 6:44 AM
(6256 views)

Regarding "significant digits", you can get that by using the "Precision" format with the "Format" function. Also, the "Char" function supports two addition arguments, one for the width and one for the number of decimal places, if you just want to skip the Round operation for display.

```
Char( 15.29 - 15.19, 10, 2); // "0.10"
Format( 12.345, "Precision", 3); // "12.3"
Format( 1.2345, "Precision", 3); // "1.23"
```

Regarding the original issue, it boils down to the fact that computers store numbers in base 2 (binary) but display them in base 10, and some base 10 numerals like "0.1" have no exact representation in base 2.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 20, 2017 2:18 AM
(3087 views)

I endet up using

format(mean(y),6)

so in the report a very small number gets converted into 1.574e-6 and don't use up a lot of space.

but

Format( 1.2345, "Precision", 3)

would be the way for "significant digits".

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 6:55 AM
(6475 views)

You might want to read this resource: JMP Numerical Accuracy You can find such white papers for other popular software such as spreadsheets. It is a know issue but generally not a problem.

Learn it once, use it forever!

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 15, 2017 8:35 AM
(3236 views)

Just to illustrate the issue is NOT a JMP issue, here is what happens in Excel.

X | Y | X - Y Formula |

16.11 | 15.12 | 0.99000000000000000 |

15.29 | 15.18 | 0.10999999999999900 |

15.29 | 15.19 | 0.09999999999999960 |

Dan Obermiller

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 16, 2017 12:40 AM
(3199 views)

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Feb 16, 2017 4:31 AM
(3183 views)

I am not sure what you are asking. If you are asking if these functions have the option to specify significant digits as well as the number of decimal places, then the answer is no. If you are asking if I would consider adding such an option, then that request is a matter for JMP Development.

Learn it once, use it forever!