cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Register to attend Discovery Summit 2025 Online: Early Users Edition, Sept. 24-25.
  • New JMP features coming to desktops everywhere this September. Sign up to learn more at jmp.com/launch.
Choose Language Hide Translation Bar
lehaofeng
Level V

Why are there so many decimals in the difference between matrices?

names default to here(1);
a=[5.0443 5.0565 5.0498 5.0856 5.051 4.9921 5.0346 5.0265 5.0356 4.9521 5.0361 5.0444
5.0442 5.0741 4.9872 5.0361 5.0464 5.0291 4.9869 5.0011 5.036 5.0012 5.0562 5.0456
5.0415 4.9914 5.0389 5.015 5.0722 5.0151 5.0471 5.0445,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119];
b=[5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05
5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05,
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0];
a-b

//result
[-0.00570000000000004 0.00649999999999995 -0.000199999999999534 0.0356000000000005
0.00100000000000033 -0.0579000000000001 -0.0153999999999996 -0.0234999999999994
-0.0144000000000002 -0.0979000000000001 -0.0138999999999996 -0.00559999999999938
-0.00579999999999981 0.0240999999999998 -0.0628000000000002 -0.0138999999999996
-0.0035999999999996 -0.0209000000000001 -0.0630999999999995 -0.0488999999999997
-0.0140000000000002 -0.0488 0.00619999999999976 -0.00439999999999952
-0.00849999999999973 -0.0586000000000002 -0.0110999999999999 -0.0350000000000001
0.0221999999999998 -0.0348999999999995 -0.00289999999999946 -0.00549999999999962,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119]

Why are there so many decimals in this result?thanks!

2 ACCEPTED SOLUTIONS

Accepted Solutions
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim

View solution in original post

jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

2 REPLIES 2
txnelson
Super User

Re: Why are there so many decimals in the difference between matrices?

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation.  If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978.  Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.  

Jim
jthi
Super User

Re: Why are there so many decimals in the difference between matrices?

Recommended Articles