cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Check out the JMP® Marketplace featured Capability Explorer add-in
Choose Language Hide Translation Bar
shaira
Level IV

How to get rid of extra decimal points during floating point calculations?

Hi,

I am making a script which takes a matrix as input, gets the interval between the elements in the matrix, and outputs that interval. My problem is that JMP output has extra decimal places sometimes (see log output below). How do I ensure that if my input has N decimal places, output should also have N decimal places?

 

Script:

 

outmtx = [0, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6, 1.8, 2,
		  2.2, 2.4, 2.6, 2.8, 3, 3.2, 3.4, 3.6, 3.8, 4,
		  4.2, 4.4, 4.6, 4.8, 5, 5.2, 5.4, 5.6, 5.8, 6,
		  6.2, 6.4, 6.6, 6.8, 7, 7.2, 7.4, 7.6, 7.8, 8,
		  8.2, 8.4, 8.6, 8.8, 9, 9.2, 9.4, 9.6, 9.8, 10];

mtxdiff = J(Nrows(outmtx)-1,1,0);		  

For (i=1, i<=Nrows(outmtx)-1, i++,  mtxdiff[i] = outmtx[i+1]-outmtx[i]);

Show(mtxdiff);

 

Log window output:

mtxdiff = [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999, 0.2,
0.2, 0.2, 0.2, 0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999,
0.2, 0.2, 0.199999999999999, 0.200000000000001, 0.199999999999999,
0.200000000000001, 0.199999999999999, 0.199999999999999, 0.200000000000001,
0.199999999999999, 0.200000000000001, 0.199999999999999];

Thanks,

Shaira

 

1 REPLY 1

Re: How to get rid of extra decimal points during floating point calculations?

See the Round() function. In this case, Round( mtxdiff, 1 ) should do what you want.