World Statistics Day was yesterday, but we’re celebrating all week long! This celebration means acknowledging the impact statistics has on our world. Who is your favorite statistician? Share with us who they are and why they top your favorites list.
Choose Language Hide Translation Bar
Highlighted
Peng
Level I

Script to Save Residuals of Spline Fit

Hello,

 

Anyone knows how to write a script to sava residuals (or predicteds) as a new colum for spline fit? It is easy if i do it manually, just need to select Save Residuals in the dropdown red triangle. I know how to write scripts to save residuals as a new column for liner fit using Fit Model. However, I often need to fit time-trend data using standardized x spline fit, and i am not sure if there's a easier way to do so. My final goal is to aquire residuals or predicteds for several varialbles in a loop, so a simple script will help a lot. 

 

Thanks!!

Spline fit.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
txnelson
Super User

Re: Script to Save Residuals of Spline Fit

Each separate "Fit" that is specified has to have their residuals saved separately.  The substript on the "Curve[]" reference is what handles this.  The example below has a GroupBy(:Sex) which will produce a Fit for Females, "F" and one for Males, "M".  Thus, 2 "Save Residuals" must be issued.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        
	GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);
obj << (Curve[1] << Save Residuals);
obj << (Curve[2] << Save Residuals);

As I was testing out this code for your example.  I discovered that JMP 14 does not work for the Save Residuals for a Spline Fit.  I will be passing that on to JMP Support as a bug.

Jim

View solution in original post

8 REPLIES 8
Highlighted
txnelson
Super User

Re: Script to Save Residuals of Spline Fit

You can find these types of syntax in the Scripting Index

     Help==>Scripting Index==>Bivariate==>Bivariate Curve

Here is the scripting index example from that entry

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),
	Fit Polynomial( 3 )
);
obj << (Curve[1] << Save Residuals);
Jim
Highlighted
Peng
Level I

Re: Script to Save Residuals of Spline Fit

Thanks Jim! It works. If I need to add "GroupBy" in Bivariate is it posible to save all the residuals? It looks like JMP only keeps the last fit line. Thanks!

 

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),
GroupBy( :Team), Fit Spline( 0.1, Standardized ) ); obj << (Curve[1] << Save Residuals);

  

Highlighted
txnelson
Super User

Re: Script to Save Residuals of Spline Fit

Each separate "Fit" that is specified has to have their residuals saved separately.  The substript on the "Curve[]" reference is what handles this.  The example below has a GroupBy(:Sex) which will produce a Fit for Females, "F" and one for Males, "M".  Thus, 2 "Save Residuals" must be issued.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        
	GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);
obj << (Curve[1] << Save Residuals);
obj << (Curve[2] << Save Residuals);

As I was testing out this code for your example.  I discovered that JMP 14 does not work for the Save Residuals for a Spline Fit.  I will be passing that on to JMP Support as a bug.

Jim

View solution in original post

Highlighted
gzmorgan0
Super User

Re: Script to Save Residuals of Spline Fit

Jim,

 

I noticed the same issue for JMP 14.  JMP support states it is fixed in JMP 14.1 (August?).

 

Here is a small modification of Jim's script. It creates a single column of residuals. It uses the new (in JMP 13) syntax for multiple subsections of a data table. This is getting the residual columns by row and finding the max (actually it gets the only non-empty value).

 

 

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);

//Find how many in a group. We know it is 2 for Big Class
summarize(_by = By(:sex));

//Find out how many columns before adding residuals
ncpre = ncol(dt);
_nby  = nitems(_by);

for(i=1, i<=_nby, i++,
  obj << (Curve[i] << Save Residuals);
  wait(0);
);

idx = ncpre+1::(ncpre+_nby);

//create a single column of residuals
resids = dt << New Column("Weight v Height Spline Residuals",
	Numeric, Continuous,	
   << Set Each Value( max(dt[row(),idx]) )
);

 

 

Highlighted
Peng
Level I

Re: Script to Save Residuals of Spline Fit

Thank you all for the solutions!! The syntax for multiple subsections of a data table does not work in JMP12. I read another discussion about returning max value of mutiple rows, a similar method was also appllied for JMP13. Is there a better way to acquire max values across mutiple rows in JMP12? Writing a formula to compare row values for unknown number of rows doesn't seem to be efficient. Thanks!

Highlighted
gzmorgan0
Super User

Re: Script to Save Residuals of Spline Fit

I really like the new JMP 13 table subset reference. However, there still is an easy method to get the values with message Get as Matrix. The key syntax is dt << get as matrix( list | index ).  There is no rowwise max function, but there is VMax function which is a columnwise max.  I tested this in JMP 12.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Big Class.jmp" );

obj = Bivariate(
	Y( :Weight ),
	X( :Height ),        GroupBy( :sex),
	Fit Spline( 0.1, Standardized )
);

//Find how many in a group. We know it is 2 for Big Class
summarize(_by = By(:sex));

//Find out how many columns before adding residuals
ncpre = ncol(dt);
_nby  = nitems(_by);

for(i=1, i<=_nby, i++,
  obj << (Curve[i] << Save Residuals);
  wait(0);
);

idx = ncpre+1::(ncpre+_nby);
cmat = Transpose(VMax(Transpose(dt << get as matrix(idx)))); //matrix _nby x nrow(dt)
/*the previous line of code is performing these tasks
  - get the columns of residuals into a matrix of size nrow(dt) x _nby
  - create the transpose: coloumns are rows and rows are columns, size _nby x nrow(dt)
  - Vmax is a vertical maximum, it finds the max of each column (previously rows)
    returns a 1 x nrow(dt) vector, a row vector
  - transpose again so it is a column vector
*/

//create a single column of residuals
resids = dt << New Column("Weight v Height Spline Residuals",
	Numeric, Continuous, Values( cmat )
);

  

Highlighted
Peng
Level I

Re: Script to Save Residuals of Spline Fit

Cool, this is awesome!! 

 

Just one more question: 

 

When I do  obj << (Curve[i] << Save Residualsand group by (:sex), JMP calculated residuals for "M" and "F" and saved in different columns with some empty cells. That's why "transpose" is needed to get all residuals in one column. However, if I do <<save Predicteds and group by (:sex), it actually calculates Predicteds for all Heights for both "M" and "F". I guess by default JMP does that even if I do it manually in the drop. Is there a way to claculate Predicteds for individual sex only? 

 

Thanks

Highlighted
gzmorgan0
Super User

Re: Script to Save Residuals of Spline Fit

Peng,

 

Yes, this seems inconsistent, however, I am guessing this is done on purpose to compare the predicteds (for example, the males are a hold-out set for the predicteds based upon females and vice versa.

 

To create a single column of Predicteds based upon the Group By  model create a new column with a formula or set each value expression

If( :sex=="M", column for the male predicteds, :sex=="F", column for the male predicteds);

 

If you want a column that looks like the residual columns, then create a columnm for males and use an expression like

If( :sex=="M", column for the male predicteds, empty() );

 

Then do the same for females.

 

 

  

Article Labels

    There are no labels assigned to this post.