turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Discussions
- :
- Calculation difference between actual(MS Excel) and JMP

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Nov 27, 2017 6:54 PM
(1481 views)

Hi,

Recently I noticed when I am doing a simple calculation using formula there is a delta of 0.0xx in JMP when compare to MS Excel or a normal calculator(refer to attached graphic). Can I know whether this is expected in JMP.

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

I have attached the data table I used. Below is an image of the table

Jim

4 REPLIES

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Running with JMP 13.2.1 and Office 365 I got the identical values between the calculations in JMP and Excel.

Jim

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Hi Jim,

I guess this could be some sort of "bug" :) I use a normal calculator and the results follow the Excel after all this calculation supposedly gave out 3-decimal point but result in JMP shows up to 9-decimal point.

I guess this could be some sort of "bug" :) I use a normal calculator and the results follow the Excel after all this calculation supposedly gave out 3-decimal point but result in JMP shows up to 9-decimal point.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

I have attached the data table I used. Below is an image of the table

Jim

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

I guess this has to do with Col Info > Format. In my JMP data, the first 2 columns were set to Fixed but the result column was set to Best. I think the calculation result actually took the values(contained 9-decimal point) from the 2 columns which was set to Fixed.

That data actually came from a test result so I guess I wasn't aware that it has so many decimal points :-)

Anyway, thanks a lot Jim for your response.