BookmarkSubscribeSubscribe to RSS Feed

Script to Save Residuals of Spline Fit

Highlighted
Peng

New Contributor

Joined:

May 15, 2018

Hello,

 

Anyone knows how to write a script to sava residuals (or predicteds) as a new colum for spline fit? It is easy if i do it manually, just need to select Save Residuals in the dropdown red triangle. I know how to write scripts to save residuals as a new column for liner fit using Fit Model. However, I often need to fit time-trend data using standardized x spline fit, and i am not sure if there's a easier way to do so. My final goal is to aquire residuals or predicteds for several varialbles in a loop, so a simple script will help a lot. 

 

Thanks!!

Spline fit.png

1 ACCEPTED SOLUTION

Accepted Solutions
txnelson

Super User

Joined:

Jun 22, 2012

Solution

Each separate "Fit" that is specified has to have their residuals saved separately.  The substript on the "Curve[]" reference is what handles this.  The example below has a GroupBy(:Sex) which will produce a Fit for Females, "F" and one for Males, "M".  Thus, 2 "Save Residuals" must be issued.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        
	GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);
obj << (Curve[1] << Save Residuals);
obj << (Curve[2] << Save Residuals);

As I was testing out this code for your example.  I discovered that JMP 14 does not work for the Save Residuals for a Spline Fit.  I will be passing that on to JMP Support as a bug.

Jim
8 REPLIES
txnelson

Super User

Joined:

Jun 22, 2012

You can find these types of syntax in the Scripting Index

     Help==>Scripting Index==>Bivariate==>Bivariate Curve

Here is the scripting index example from that entry

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),
	Fit Polynomial( 3 )
);
obj << (Curve[1] << Save Residuals);
Jim
Peng

New Contributor

Joined:

May 15, 2018

Thanks Jim! It works. If I need to add "GroupBy" in Bivariate is it posible to save all the residuals? It looks like JMP only keeps the last fit line. Thanks!

 

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),
GroupBy( :Team), Fit Spline( 0.1, Standardized ) ); obj << (Curve[1] << Save Residuals);

  

txnelson

Super User

Joined:

Jun 22, 2012

Solution

Each separate "Fit" that is specified has to have their residuals saved separately.  The substript on the "Curve[]" reference is what handles this.  The example below has a GroupBy(:Sex) which will produce a Fit for Females, "F" and one for Males, "M".  Thus, 2 "Save Residuals" must be issued.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        
	GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);
obj << (Curve[1] << Save Residuals);
obj << (Curve[2] << Save Residuals);

As I was testing out this code for your example.  I discovered that JMP 14 does not work for the Save Residuals for a Spline Fit.  I will be passing that on to JMP Support as a bug.

Jim
gzmorgan0

Community Trekker

Joined:

Jul 25, 2016

Jim,

 

I noticed the same issue for JMP 14.  JMP support states it is fixed in JMP 14.1 (August?).

 

Here is a small modification of Jim's script. It creates a single column of residuals. It uses the new (in JMP 13) syntax for multiple subsections of a data table. This is getting the residual columns by row and finding the max (actually it gets the only non-empty value).

 

 

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);

//Find how many in a group. We know it is 2 for Big Class
summarize(_by = By(:sex));

//Find out how many columns before adding residuals
ncpre = ncol(dt);
_nby  = nitems(_by);

for(i=1, i<=_nby, i++,
  obj << (Curve[i] << Save Residuals);
  wait(0);
);

idx = ncpre+1::(ncpre+_nby);

//create a single column of residuals
resids = dt << New Column("Weight v Height Spline Residuals",
	Numeric, Continuous,	
   << Set Each Value( max(dt[row(),idx]) )
);

 

 

Peng

New Contributor

Joined:

May 15, 2018

Thank you all for the solutions!! The syntax for multiple subsections of a data table does not work in JMP12. I read another discussion about returning max value of mutiple rows, a similar method was also appllied for JMP13. Is there a better way to acquire max values across mutiple rows in JMP12? Writing a formula to compare row values for unknown number of rows doesn't seem to be efficient. Thanks!

gzmorgan0

Community Trekker

Joined:

Jul 25, 2016

I really like the new JMP 13 table subset reference. However, there still is an easy method to get the values with message Get as Matrix. The key syntax is dt << get as matrix( list | index ).  There is no rowwise max function, but there is VMax function which is a columnwise max.  I tested this in JMP 12.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);

//Find how many in a group. We know it is 2 for Big Class
summarize(_by = By(:sex));

//Find out how many columns before adding residuals
ncpre = ncol(dt);
_nby  = nitems(_by);

for(i=1, i<=_nby, i++,
  obj << (Curve[i] << Save Residuals);
  wait(0);
);

idx = ncpre+1::(ncpre+_nby);
cmat = Transpose(VMax(Transpose(dt << get as matrix(idx)))); //matrix _nby x nrow(dt)
/*the previous line of code is performing these tasks
  - get the columns of residuals into a matrix of size nrow(dt) x _nby
  - create the transpose: coloumns are rows and rows are columns, size _nby x nrow(dt)
  - Vmax is a vertical maximum, it finds the max of each column (previously rows)
    returns a 1 x nrow(dt) vector, a row vector
  - transpose again so it is a column vector
*/

//create a single column of residuals
resids = dt << New Column("Weight v Height Spline Residuals",
	Numeric, Continuous, Values( cmat )
);

  

Peng

New Contributor

Joined:

May 15, 2018

Cool, this is awesome!! 

 

Just one more question: 

 

When I do  obj << (Curve[i] << Save Residualsand group by (:sex), JMP calculated residuals for "M" and "F" and saved in different columns with some empty cells. That's why "transpose" is needed to get all residuals in one column. However, if I do <<save Predicteds and group by (:sex), it actually calculates Predicteds for all Heights for both "M" and "F". I guess by default JMP does that even if I do it manually in the drop. Is there a way to claculate Predicteds for individual sex only? 

 

Thanks

gzmorgan0

Community Trekker

Joined:

Jul 25, 2016

Peng,

 

Yes, this seems inconsistent, however, I am guessing this is done on purpose to compare the predicteds (for example, the males are a hold-out set for the predicteds based upon females and vice versa.

 

To create a single column of Predicteds based upon the Group By  model create a new column with a formula or set each value expression

If( :sex=="M", column for the male predicteds, :sex=="F", column for the male predicteds);

 

If you want a column that looks like the residual columns, then create a columnm for males and use an expression like

If( :sex=="M", column for the male predicteds, empty() );

 

Then do the same for females.