cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • New to JMP? Join us Sept. 23-24 for the Early User Edition of Discovery Summit, tailor-made for new users. Register now for free!
  • Use World Cup data to build models, explore spatial relationships, and create informative visualizations in JMP. Register. July 17, 2 pm US Eastern Time.
  • Your voice matters! Tell us how you prefer to receive JMP updates, so we can tailor our communication to your needs. Take short survey.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
lehaofeng
Level V

Why are there so many decimals in the difference between matrices?

names default to here(1);
a=[5.0443 5.0565 5.0498 5.0856 5.051 4.9921 5.0346 5.0265 5.0356 4.9521 5.0361 5.0444
5.0442 5.0741 4.9872 5.0361 5.0464 5.0291 4.9869 5.0011 5.036 5.0012 5.0562 5.0456
5.0415 4.9914 5.0389 5.015 5.0722 5.0151 5.0471 5.0445,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119];
b=[5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05
5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05,
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0];
a-b

//result
[-0.00570000000000004 0.00649999999999995 -0.000199999999999534 0.0356000000000005
0.00100000000000033 -0.0579000000000001 -0.0153999999999996 -0.0234999999999994
-0.0144000000000002 -0.0979000000000001 -0.0138999999999996 -0.00559999999999938
-0.00579999999999981 0.0240999999999998 -0.0628000000000002 -0.0138999999999996
-0.0035999999999996 -0.0209000000000001 -0.0630999999999995 -0.0488999999999997
-0.0140000000000002 -0.0488 0.00619999999999976 -0.00439999999999952
-0.00849999999999973 -0.0586000000000002 -0.0110999999999999 -0.0350000000000001
0.0221999999999998 -0.0348999999999995 -0.00289999999999946 -0.00549999999999962,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119]

Why are there so many decimals in this result?thanks!

2 ACCEPTED SOLUTIONS

Accepted Solutions
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim

View solution in original post

jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

2 REPLIES 2
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim
jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

Recommended Articles