topic Re: Script to Save Residuals of Spline Fit in Discussions

Script to Save Residuals of Spline Fit

Peng — Wed, 16 May 2018 00:35:20 GMT

Hello,

Anyone knows how to write a script to sava residuals (or predicteds) as a new colum for spline fit? It is easy if i do it manually, just need to select Save Residuals in the dropdown red triangle. I know how to write scripts to save residuals as a new column for liner fit using Fit Model. However, I often need to fit time-trend data using standardized x spline fit, and i am not sure if there's a easier way to do so. My final goal is to aquire residuals or predicteds for several varialbles in a loop, so a simple script will help a lot.

Thanks!!

Re: Script to Save Residuals of Spline Fit

txnelson — Wed, 16 May 2018 00:51:16 GMT

You can find these types of syntax in the Scripting Index

Help==>Scripting Index==>Bivariate==>Bivariate Curve

Here is the scripting index example from that entry

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),
	Fit Polynomial( 3 )
);
obj << (Curve[1] << Save Residuals);

Re: Script to Save Residuals of Spline Fit

Peng — Wed, 16 May 2018 05:14:27 GMT

Thanks Jim! It works. If I need to add "GroupBy" in Bivariate is it posible to save all the residuals? It looks like JMP only keeps the last fit line. Thanks!

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),
        GroupBy( :Team),
	Fit Spline( 0.1, Standardized )
);
obj << (Curve[1] << Save Residuals);

Re: Script to Save Residuals of Spline Fit

txnelson — Wed, 16 May 2018 09:38:06 GMT

Each separate "Fit" that is specified has to have their residuals saved separately. The substript on the "Curve[]" reference is what handles this. The example below has a GroupBy(:Sex) which will produce a Fit for Females, "F" and one for Males, "M". Thus, 2 "Save Residuals" must be issued.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        
	GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);
obj << (Curve[1] << Save Residuals);
obj << (Curve[2] << Save Residuals);

As I was testing out this code for your example. I discovered that JMP 14 does not work for the Save Residuals for a Spline Fit. I will be passing that on to JMP Support as a bug.

Re: Script to Save Residuals of Spline Fit

gzmorgan0 — Wed, 16 May 2018 21:17:08 GMT

Jim,

I noticed the same issue for JMP 14. JMP support states it is fixed in JMP 14.1 (August?).

Here is a small modification of Jim's script. It creates a single column of residuals. It uses the new (in JMP 13) syntax for multiple subsections of a data table. This is getting the residual columns by row and finding the max (actually it gets the only non-empty value).

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);

//Find how many in a group. We know it is 2 for Big Class
summarize(_by = By(:sex));

//Find out how many columns before adding residuals
ncpre = ncol(dt);
_nby  = nitems(_by);

for(i=1, i<=_nby, i++,
  obj << (Curve[i] << Save Residuals);
  wait(0);
);

idx = ncpre+1::(ncpre+_nby);

//create a single column of residuals
resids = dt << New Column("Weight v Height Spline Residuals",
	Numeric, Continuous,	
   << Set Each Value( max(dt[row(),idx]) )
);

Re: Script to Save Residuals of Spline Fit

Peng — Thu, 17 May 2018 05:03:14 GMT

Thank you all for the solutions!! The syntax for multiple subsections of a data table does not work in JMP12. I read another discussion about returning max value of mutiple rows, a similar method was also appllied for JMP13. Is there a better way to acquire max values across mutiple rows in JMP12? Writing a formula to compare row values for unknown number of rows doesn't seem to be efficient. Thanks!

Re: Script to Save Residuals of Spline Fit

gzmorgan0 — Thu, 17 May 2018 05:48:17 GMT

I really like the new JMP 13 table subset reference. However, there still is an easy method to get the values with message Get as Matrix. The key syntax is dt << get as matrix( list | index ). There is no rowwise max function, but there is VMax function which is a columnwise max. I tested this in JMP 12.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);

//Find how many in a group. We know it is 2 for Big Class
summarize(_by = By(:sex));

//Find out how many columns before adding residuals
ncpre = ncol(dt);
_nby  = nitems(_by);

for(i=1, i<=_nby, i++,
  obj << (Curve[i] << Save Residuals);
  wait(0);
);

idx = ncpre+1::(ncpre+_nby);
cmat = Transpose(VMax(Transpose(dt << get as matrix(idx)))); //matrix _nby x nrow(dt)
/*the previous line of code is performing these tasks
  - get the columns of residuals into a matrix of size nrow(dt) x _nby
  - create the transpose: coloumns are rows and rows are columns, size _nby x nrow(dt)
  - Vmax is a vertical maximum, it finds the max of each column (previously rows)
    returns a 1 x nrow(dt) vector, a row vector
  - transpose again so it is a column vector
*/

//create a single column of residuals
resids = dt << New Column("Weight v Height Spline Residuals",
	Numeric, Continuous, Values( cmat )
);

Re: Script to Save Residuals of Spline Fit

Peng — Fri, 18 May 2018 23:14:11 GMT

Cool, this is awesome!!

Just one more question:

When I do obj << (Curve[i] << Save Residuals) and group by (:sex), JMP calculated residuals for "M" and "F" and saved in different columns with some empty cells. That's why "transpose" is needed to get all residuals in one column. However, if I do <<save Predicteds and group by (:sex), it actually calculates Predicteds for all Heights for both "M" and "F". I guess by default JMP does that even if I do it manually in the drop. Is there a way to claculate Predicteds for individual sex only?

Thanks

Re: Script to Save Residuals of Spline Fit

gzmorgan0 — Sat, 19 May 2018 01:40:58 GMT

Peng,

Yes, this seems inconsistent, however, I am guessing this is done on purpose to compare the predicteds (for example, the males are a hold-out set for the predicteds based upon females and vice versa.

To create a single column of Predicteds based upon the Group By model create a new column with a formula or set each value expression

If( :sex=="M", column for the male predicteds, :sex=="F", column for the male predicteds);

If you want a column that looks like the residual columns, then create a columnm for males and use an expression like

If( :sex=="M", column for the male predicteds, empty() );

Then do the same for females.

Re: Script to Save Residuals of Spline Fit

deveshchugh — Wed, 21 Oct 2020 21:13:46 GMT

Hi Jim

Any resolution/fix to so that we can save residual in v14?

thanks for the help