cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Instantly extract effect sizes, F-ratios, and FDR-adjusted p-values from your models with the Calculate Effects Sizes extension, available now in the JMP Marketplace!
  • New to JMP? Join us Sept. 23-24 for the Early User Edition of Discovery Summit, tailor-made for new users. Register now for free!
  • Your voice matters! Tell us how you prefer to receive JMP updates, so we can tailor our communication to your needs. Take short survey.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
shaira
Level IV

How to get rid of extra decimal points during floating point calculations?

Hi,

I am making a script which takes a matrix as input, gets the interval between the elements in the matrix, and outputs that interval. My problem is that JMP output has extra decimal places sometimes (see log output below). How do I ensure that if my input has N decimal places, output should also have N decimal places?

 

Script:

 

outmtx = [0, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6, 1.8, 2,
		  2.2, 2.4, 2.6, 2.8, 3, 3.2, 3.4, 3.6, 3.8, 4,
		  4.2, 4.4, 4.6, 4.8, 5, 5.2, 5.4, 5.6, 5.8, 6,
		  6.2, 6.4, 6.6, 6.8, 7, 7.2, 7.4, 7.6, 7.8, 8,
		  8.2, 8.4, 8.6, 8.8, 9, 9.2, 9.4, 9.6, 9.8, 10];

mtxdiff = J(Nrows(outmtx)-1,1,0);		  

For (i=1, i<=Nrows(outmtx)-1, i++,  mtxdiff[i] = outmtx[i+1]-outmtx[i]);

Show(mtxdiff);

 

Log window output:

mtxdiff = [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999, 0.2,
0.2, 0.2, 0.2, 0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999,
0.2, 0.2, 0.199999999999999, 0.200000000000001, 0.199999999999999,
0.200000000000001, 0.199999999999999, 0.199999999999999, 0.200000000000001,
0.199999999999999, 0.200000000000001, 0.199999999999999];

Thanks,

Shaira

 

1 REPLY 1

Re: How to get rid of extra decimal points during floating point calculations?

See the Round() function. In this case, Round( mtxdiff, 1 ) should do what you want.

Recommended Articles