Why are there so many decimals in the difference between matrices？

lehaofeng — Thu, 27 Jun 2024 07:47:25 GMT

names default to here(1);
a=[5.0443 5.0565 5.0498 5.0856 5.051 4.9921 5.0346 5.0265 5.0356 4.9521 5.0361 5.0444
5.0442 5.0741 4.9872 5.0361 5.0464 5.0291 4.9869 5.0011 5.036 5.0012 5.0562 5.0456
5.0415 4.9914 5.0389 5.015 5.0722 5.0151 5.0471 5.0445,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119];
b=[5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05
5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05 5.05,
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0];
a-b

//result
[-0.00570000000000004 0.00649999999999995 -0.000199999999999534 0.0356000000000005
0.00100000000000033 -0.0579000000000001 -0.0153999999999996 -0.0234999999999994
-0.0144000000000002 -0.0979000000000001 -0.0138999999999996 -0.00559999999999938
-0.00579999999999981 0.0240999999999998 -0.0628000000000002 -0.0138999999999996
-0.0035999999999996 -0.0209000000000001 -0.0630999999999995 -0.0488999999999997
-0.0140000000000002 -0.0488 0.00619999999999976 -0.00439999999999952
-0.00849999999999973 -0.0586000000000002 -0.0110999999999999 -0.0350000000000001
0.0221999999999998 -0.0348999999999995 -0.00289999999999946 -0.00549999999999962,
0.0023 0.0007 -0.02 0.0245 0.017 0.011 0.0557 0.0618 0.028 -0.0049 0.0661 0.0243
0.0454 0.0151 -0.0022 0.0423 0.0205 0.0285 0.0279 0.021 0.0277 -0.0036 0.0125 0.0086
0.0416 -0.004 -0.0141 0.0276 0.0055 0.0286 -0.0197 0.0119]

Why are there so many decimals in this result？thanks！

Re: Why are there so many decimals in the difference between matrices？

txnelson — Thu, 27 Jun 2024 12:28:05 GMT

The short answer is that by default, JMP numeric values are stored as floating point values. Some values can not be exactly specified in floating point notation. If you take the first value in the A matrix, 5.0443, as a floating point value it is stored as: 5.04429999999999978. Therefore, what you are seeing in the result matrix are the precision required to show the most accurate values for the calculation.

Re: Why are there so many decimals in the difference between matrices？

jthi — Thu, 27 Jun 2024 19:01:41 GMT

https://0.30000000000000004.com/

topic Re: Why are there so many decimals in the difference between matrices？ in Discussions

Why are there so many decimals in the difference between matrices？

Re: Why are there so many decimals in the difference between matrices？

Re: Why are there so many decimals in the difference between matrices？