turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- JMP User Community
- :
- Discussions
- :
- Discussions
- :
- Possible to increase floating point number precision?

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Jul 30, 2016 7:22 AM
(3678 views)

Hi. Is there a way to specify the floating point precision JSL uses when performing calculations? That is, the number of significant digits used. For instance, on my computer 1 minus 1e-15 equals 0.999... (15 "9"s). But if I try 1 minus 1e-16, I just get 1.

x = 1;

y = 1e-15;

z = x - y;

Show(z); //z = 0.999999999999999 (aka 15 decimal places)

y = 1e-16;

z = x - y;

Show(z); //z = 1

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Here is the complete script that I ran to give you the results I posted.

x = **1**;

y = **1e-15**;

z = x - y;

Show**(**z**)**; //z = 0.999999999999999 (aka 15 decimal places)

y = **1e-16**;

z = x - y;

Show**(**format**(**z, "Fixed Dec", **30**, **20))**;

I ran the script in 11.21.1 and got the results you reported. So the difference is the change to version 12.2.

Jim

5 REPLIES

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

What you are seeing is JMP attempting to display the data in a default friendly format. The precision behind the value does not change when specifying a format to display the value in. So if you specify to display the value of "z" with more precision it will do that.

Show(format(z, "Fixed Dec", 30, 20)); = "0.99999999999999989000";

Jim

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Jim,

I'm sorry but that does not appear to work on my system. I used the format command and the answer still came out as "1". Obviously it works on your system, but not mine. My company still has us using JMP 11.2.1, if that matters. My computer is < 1 yr old though.

Question: the documentation for Format is a little confusing.

s = Format(x, formatstring, <currencycode>, <ndecimal>, < <<Use Locale(b=1)>)

Based on that, the "20" in your command occupies the currency code field, and the "30" occupies the ndecimal field. Naturally I care about the latter. What is the purpose of the "20"?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Hi again Jim. Also realized another important question. In your post you indicated the result to the problem 1 minus 1e-16 was:

0.999999999999999**8**9000

(Note the "8" in the 16th decimal place)

Was this a typo?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

Here is the complete script that I ran to give you the results I posted.

x = **1**;

y = **1e-15**;

z = x - y;

Show**(**z**)**; //z = 0.999999999999999 (aka 15 decimal places)

y = **1e-16**;

z = x - y;

Show**(**format**(**z, "Fixed Dec", **30**, **20))**;

I ran the script in 11.21.1 and got the results you reported. So the difference is the change to version 12.2.

Jim

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Get Direct Link
- Email to a Friend
- Report Inappropriate Content

JMP uses the standard hardware floating point support. IEEE floating point - Wikipedia, the free encyclopedia

There are about 15 significant digits, as you've noticed.

JMP does not have support for longer representations. (Big Int, Big Num libraries are not in JMP at this time)

As txnelson points out, sometimes a default format for printing a number may not show all 15 or so possibly interesting digits.

Craige