topic Re: Possible to increase floating point number precision? in Discussions

Possible to increase floating point number precision?

nikles — Sat, 30 Jul 2016 14:22:12 GMT

Hi. Is there a way to specify the floating point precision JSL uses when performing calculations? That is, the number of significant digits used. For instance, on my computer 1 minus 1e-15 equals 0.999... (15 "9"s). But if I try 1 minus 1e-16, I just get 1.

x = 1;

y = 1e-15;

z = x - y;

Show(z); //z = 0.999999999999999 (aka 15 decimal places)

y = 1e-16;

z = x - y;

Show(z); //z = 1

Thanks.

Re: Possible to increase floating point number precision?

txnelson — Sat, 30 Jul 2016 15:57:42 GMT

What you are seeing is JMP attempting to display the data in a default friendly format. The precision behind the value does not change when specifying a format to display the value in. So if you specify to display the value of "z" with more precision it will do that.

Show(format(z, "Fixed Dec", 30, 20)); = "0.99999999999999989000";

Re: Possible to increase floating point number precision?

Craige_Hales — Sat, 30 Jul 2016 17:05:14 GMT

JMP uses the standard hardware floating point support. IEEE floating point - Wikipedia, the free encyclopedia

There are about 15 significant digits, as you've noticed.

JMP does not have support for longer representations. (Big Int, Big Num libraries are not in JMP at this time)

As txnelson points out, sometimes a default format for printing a number may not show all 15 or so possibly interesting digits.

Re: Possible to increase floating point number precision?

nikles — Sat, 30 Jul 2016 18:12:37 GMT

Jim,

I'm sorry but that does not appear to work on my system. I used the format command and the answer still came out as "1". Obviously it works on your system, but not mine. My company still has us using JMP 11.2.1, if that matters. My computer is < 1 yr old though.

Question: the documentation for Format is a little confusing.

s = Format(x, formatstring, <currencycode>, <ndecimal>, < <<Use Locale(b=1)>)

Based on that, the "20" in your command occupies the currency code field, and the "30" occupies the ndecimal field. Naturally I care about the latter. What is the purpose of the "20"?

Re: Possible to increase floating point number precision?

nikles — Sat, 30 Jul 2016 18:22:09 GMT

Hi again Jim. Also realized another important question. In your post you indicated the result to the problem 1 minus 1e-16 was:

0.99999999999999989000

(Note the "8" in the 16th decimal place)

Was this a typo?

Re: Possible to increase floating point number precision?

txnelson — Sat, 30 Jul 2016 20:42:30 GMT

Here is the complete script that I ran to give you the results I posted.

x = 1;

y = 1e-15;

z = x - y;

Show(z); //z = 0.999999999999999 (aka 15 decimal places)

y = 1e-16;

z = x - y;

Show(format(z, "Fixed Dec", 30, 20));

I ran the script in 11.21.1 and got the results you reported. So the difference is the change to version 12.2.