cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Sign-in to the JMP Community will be unavailable intermittently Dec. 6-7 due to a system update. Thank you for your understanding!
  • We’re retiring the File Exchange at the end of this year. The JMP Marketplace is now your destination for add-ins and extensions.
  • JMP 19 is here! Learn more about the new features.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
lehaofeng
Level V

Why are there so many decimals in the difference between matrices?

names default to here(1);
a=[5.0443 5.0565 5.0498 5.0856 5.051 4.9921 5.0346 5.0265 5.0356 4.9521 5.0361 5.0444
5.0442 5.0741 4.9872 5.0361 5.0464 5.0291 4.9869 5.0011 5.036 5.0012 5.0562 5.0456
5.0415 4.9914 5.0389 5.015 5.0722 5.0151 5.0471 5.0445,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119];
b=[5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05
5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05,
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0];
a-b

//result
[-0.00570000000000004 0.00649999999999995 -0.000199999999999534 0.0356000000000005
0.00100000000000033 -0.0579000000000001 -0.0153999999999996 -0.0234999999999994
-0.0144000000000002 -0.0979000000000001 -0.0138999999999996 -0.00559999999999938
-0.00579999999999981 0.0240999999999998 -0.0628000000000002 -0.0138999999999996
-0.0035999999999996 -0.0209000000000001 -0.0630999999999995 -0.0488999999999997
-0.0140000000000002 -0.0488 0.00619999999999976 -0.00439999999999952
-0.00849999999999973 -0.0586000000000002 -0.0110999999999999 -0.0350000000000001
0.0221999999999998 -0.0348999999999995 -0.00289999999999946 -0.00549999999999962,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119]

Why are there so many decimals in this result?thanks!

2 ACCEPTED SOLUTIONS

Accepted Solutions
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim

View solution in original post

jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

2 REPLIES 2
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim
jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

Recommended Articles