cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Learn how to build custom Python data connectors and further customize JMP’s Data Connector Framework with the Python Data Connector Demo, available now in the JMP Marketplace!
  • See how to use Accelerated Life Testing (ALT) to evaluate reliability. Register for June 5 webinar, 2pm US Eastern Time.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
shaira
Level IV

How to get rid of extra decimal points during floating point calculations?

Hi,

I am making a script which takes a matrix as input, gets the interval between the elements in the matrix, and outputs that interval. My problem is that JMP output has extra decimal places sometimes (see log output below). How do I ensure that if my input has N decimal places, output should also have N decimal places?

 

Script:

 

outmtx = [0, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6, 1.8, 2,
		  2.2, 2.4, 2.6, 2.8, 3, 3.2, 3.4, 3.6, 3.8, 4,
		  4.2, 4.4, 4.6, 4.8, 5, 5.2, 5.4, 5.6, 5.8, 6,
		  6.2, 6.4, 6.6, 6.8, 7, 7.2, 7.4, 7.6, 7.8, 8,
		  8.2, 8.4, 8.6, 8.8, 9, 9.2, 9.4, 9.6, 9.8, 10];

mtxdiff = J(Nrows(outmtx)-1,1,0);		  

For (i=1, i<=Nrows(outmtx)-1, i++,  mtxdiff[i] = outmtx[i+1]-outmtx[i]);

Show(mtxdiff);

 

Log window output:

mtxdiff = [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999, 0.2,
0.2, 0.2, 0.2, 0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999,
0.2, 0.2, 0.199999999999999, 0.200000000000001, 0.199999999999999,
0.200000000000001, 0.199999999999999, 0.199999999999999, 0.200000000000001,
0.199999999999999, 0.200000000000001, 0.199999999999999];

Thanks,

Shaira

 

1 REPLY 1

Re: How to get rid of extra decimal points during floating point calculations?

See the Round() function. In this case, Round( mtxdiff, 1 ) should do what you want.

Recommended Articles