Wrong rounding when transferring from crosstab table to new table

[#1]

Hi,

I am using JMP version 16. 

I am transferring data from a CrossTab table. I have asked it to round to 3 decimals. It then saves the data into a string, which will be used to print the results into a table. See code below.

// Get data from CrossTab table
	crossTab = reportFitmod[Outline Box( "LSMeans Differences Student's t" )][CrosstabBox( 1 )] << get as matrix;
	meandiff = Round( crossTab[1, 2] , 3 );
	lowerCL = Round( crossTab[3, 2] , 3 );
	upperCL = Round( crossTab[4, 2] , 3 );

 // Create the Difference in mean and CL string
Insert Into( dif, Char( meandiff ) || " [" || Char( lowerCL ) || " ; " || Char( upperCL ) || "]" );

I have two questions about the reporting: 

1. Why does it report e.g., 0.019 as 0.02 instead of 0.020 when it is specified to report 3 decimals? 

2. This used to work, but now it does not - it takes the value 0.0015 and rounds it to 0.001 instead of 0.002. Do you know why it does this and how to fix it? I want it to keep the normal rounding rules (4 or less rounds down and 5 or higher rounds up).

[#2]

1. I think if you wish to have those extra zeros in JMP, you have to specify the format and convert to string 

Format(1.100, "Fixed Dec", 10, 3)

2. Are you sure you are rounding 0.0015 and not something else? What is the real value in the matrix you have? Even if the platform is showing you 0.0015 the value is most likely more accurate in reality, such as 0.00149 which would then get rounded to 0.0015

[#3]

Hi Jarmo,

Thank you for the reply! 

1. I will try this, thanks.

2. It should go to the crosstab and take from the second column, row 1, row 3 and row 4 (Mean[i]-Mean[j], Lower CL Dif and Upper CL Dif). 

Please see picture below. 

The mean difference is printed with this line 

meandiff = Round( crossTab[1, 2] , 3 );

It should be rounded to three decimals and it should be rounded according to normal rounding rules. It just did not round correctly and I cannot figure out what goes wrong. 

[#4]

Did you check the values it really has? Those values you see in the JMP report are most likely already rounded values. This is done in JMP18

Names Default To Here(1);
dt = Open("$SAMPLE_DATA/Drug.jmp");
obj = Fit Model(
	Y(:y),
	Effects(:Drug, :x),
	Personality(Standard Least Squares),
	Emphasis(Effect Leverage),
	<< Run(:Drug << {LSMeans Student's t(.05, Ordered Differences Report(1))})
);

m = Report(obj)["Response y","Drug","LSMeans Differences Student's t",CrosstabBox(1)] << get as matrix;
show(m);

[#5]

I could not get it to run the following line

m = Report(obj)[ :Result , :Person , "LSMeans Differences Student's t" , CrosstabBox(1)] << get as matrix;

So instead I opened the ordered differences report. Here I can see, that the full difference is 0.0015000, hence, it is not double rounded. 

[#6]

You already had the script for your case to extract the matrix

crossTab = reportFitmod[Outline Box( "LSMeans Differences Student's t" )][CrosstabBox( 1 )] << get as matrix;

Just verify the values by printing the crossTab

[#7]

Ahh, thank you! I got the results out and can see it is because when it rounds to 2 decimals instead of 3, then it rounds differently. 

crossTab = [0 0.00149999999999987, 0 0.000298142396959379, 0 0.000835697341910363, 0 0.00216430265808938, -0.00149999999999987 0, 0.000298142396959379 0, -0.00216430265808938 0, -0.000835697341910363 0];

Thank you a lot for your help