cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • DownloadSemiconductor Toolkit: Tools to create wafer maps, add wafer geometry to graphics, explore die defects & compare wafers.
  • Discovery Summit 2026: Early User Edition - September 23-24.Register. It's free.
  • See how to access JMP Marketplace - and - find, create & share add-ins to extend your JMP. Watch video.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
shaira
Level IV

How to get rid of extra decimal points during floating point calculations?

Hi,

I am making a script which takes a matrix as input, gets the interval between the elements in the matrix, and outputs that interval. My problem is that JMP output has extra decimal places sometimes (see log output below). How do I ensure that if my input has N decimal places, output should also have N decimal places?

 

Script:

 

outmtx = [0, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6, 1.8, 2,
		  2.2, 2.4, 2.6, 2.8, 3, 3.2, 3.4, 3.6, 3.8, 4,
		  4.2, 4.4, 4.6, 4.8, 5, 5.2, 5.4, 5.6, 5.8, 6,
		  6.2, 6.4, 6.6, 6.8, 7, 7.2, 7.4, 7.6, 7.8, 8,
		  8.2, 8.4, 8.6, 8.8, 9, 9.2, 9.4, 9.6, 9.8, 10];

mtxdiff = J(Nrows(outmtx)-1,1,0);		  

For (i=1, i<=Nrows(outmtx)-1, i++,  mtxdiff[i] = outmtx[i+1]-outmtx[i]);

Show(mtxdiff);

 

Log window output:

mtxdiff = [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2,
0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999, 0.2,
0.2, 0.2, 0.2, 0.199999999999999, 0.2, 0.2, 0.2, 0.2, 0.199999999999999,
0.2, 0.2, 0.199999999999999, 0.200000000000001, 0.199999999999999,
0.200000000000001, 0.199999999999999, 0.199999999999999, 0.200000000000001,
0.199999999999999, 0.200000000000001, 0.199999999999999];

Thanks,

Shaira

 

1 REPLY 1

Re: How to get rid of extra decimal points during floating point calculations?

See the Round() function. In this case, Round( mtxdiff, 1 ) should do what you want.

Recommended Articles