cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Sign-in to the JMP Community will be unavailable intermittently Dec. 6-7 due to a system update. Thank you for your understanding!
  • We’re retiring the File Exchange at the end of this year. The JMP Marketplace is now your destination for add-ins and extensions.
  • JMP 19 is here! Learn more about the new features.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
shaira
Level IV

How to get rid of extra decimal points during floating point calculations?

Hi,

I am making a script which takes a matrix as input, gets the interval between the elements in the matrix, and outputs that interval. My problem is that JMP output has extra decimal places sometimes (see log output below). How do I ensure that if my input has N decimal places, output should also have N decimal places?

 

Script:

 

outmtx = [0, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6, 1.8, 2,
		  2.2, 2.4, 2.6, 2.8, 3, 3.2, 3.4, 3.6, 3.8, 4,
		  4.2, 4.4, 4.6, 4.8, 5, 5.2, 5.4, 5.6, 5.8, 6,
		  6.2, 6.4, 6.6, 6.8, 7, 7.2, 7.4, 7.6, 7.8, 8,
		  8.2, 8.4, 8.6, 8.8, 9, 9.2, 9.4, 9.6, 9.8, 10];

mtxdiff = J(Nrows(outmtx)-1,1,0);		  

For (i=1, i<=Nrows(outmtx)-1, i++,  mtxdiff[i] = outmtx[i+1]-outmtx[i]);

Show(mtxdiff);

 

Log window output:

mtxdiff = [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999, 0.2,
0.2, 0.2, 0.2, 0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999,
0.2, 0.2, 0.199999999999999, 0.200000000000001, 0.199999999999999,
0.200000000000001, 0.199999999999999, 0.199999999999999, 0.200000000000001,
0.199999999999999, 0.200000000000001, 0.199999999999999];

Thanks,

Shaira

 

1 REPLY 1

Re: How to get rid of extra decimal points during floating point calculations?

See the Round() function. In this case, Round( mtxdiff, 1 ) should do what you want.

Recommended Articles