cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Instantly extract effect sizes, F-ratios, and FDR-adjusted p-values from your models with the Calculate Effects Sizes extension, available now in the JMP Marketplace!
  • New to JMP? Join us Sept. 23-24 for the Early User Edition of Discovery Summit, tailor-made for new users. Register now for free!
  • Your voice matters! Tell us how you prefer to receive JMP updates, so we can tailor our communication to your needs. Take short survey.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
lehaofeng
Level V

Why are there so many decimals in the difference between matrices?

names default to here(1);
a=[5.0443 5.0565 5.0498 5.0856 5.051 4.9921 5.0346 5.0265 5.0356 4.9521 5.0361 5.0444
5.0442 5.0741 4.9872 5.0361 5.0464 5.0291 4.9869 5.0011 5.036 5.0012 5.0562 5.0456
5.0415 4.9914 5.0389 5.015 5.0722 5.0151 5.0471 5.0445,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119];
b=[5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05
5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05,
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0];
a-b

//result
[-0.00570000000000004 0.00649999999999995 -0.000199999999999534 0.0356000000000005
0.00100000000000033 -0.0579000000000001 -0.0153999999999996 -0.0234999999999994
-0.0144000000000002 -0.0979000000000001 -0.0138999999999996 -0.00559999999999938
-0.00579999999999981 0.0240999999999998 -0.0628000000000002 -0.0138999999999996
-0.0035999999999996 -0.0209000000000001 -0.0630999999999995 -0.0488999999999997
-0.0140000000000002 -0.0488 0.00619999999999976 -0.00439999999999952
-0.00849999999999973 -0.0586000000000002 -0.0110999999999999 -0.0350000000000001
0.0221999999999998 -0.0348999999999995 -0.00289999999999946 -0.00549999999999962,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119]

Why are there so many decimals in this result?thanks!

2 ACCEPTED SOLUTIONS

Accepted Solutions
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim

View solution in original post

jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

2 REPLIES 2
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim
jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

Recommended Articles