topic Re: Calculating linear regression slope in formula in Discussions

Calculating linear regression slope in formula

bio_grad — Mon, 27 Feb 2017 18:52:16 GMT

Hello,

I was wondering if there was a way to calculate the linear regression slope of column data within a data table formula.

A simple example of the table is below. The data is ordered and I would like to calculate the slope of a fit line of volume over time using the current row's data and previous points to perform the regression.

The idea is that I would like to see the how the calculated slope changes with increasing the number of data points. From doing this manually, the slope values end up approaching a given value and stay within +/-5% after 20 or so points.

Instance	Time (s)	Volume (mL)	Slope (mL/s)
1	0	0	n/a
2	5	0.5	m1
3	10	0.8	m2
...	...	...	...
N	T	V	mN-1

If that is possible, my next challenge would be to set this up so that I can calculate the slope formula so that it uses N data points on either side of the current row for a symmetrical moving calculation instead.

Thank you in advance.

Re: Calculating linear regression slope in formula

txnelson — Mon, 27 Feb 2017 20:15:36 GMT

Here is a script that gives you the output you want, it just does it a different way. It uses the Fit Model Platform and loops through it several time, increasing the N.

Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA\semiconductor capability.jmp" );

// Create an output table to hold the results
dtOut = New Table( "summary",
	New Column( "Instance" ),
	New Column( "N" ),
	New Column( "RSquare" ),
	New Column( "RSM" ),
	New Column( "Intercept" ),
	New Column( "b1" ),
	New Column( "b2" )
);

Loop Counter = 0;

// Loop across the different sizes
For( i = 5, i <= 111, i = i + 5,
	Loop Counter++;
	
	// Create a random sample data table from the original data table
	dt << New Column( "random", formula( Random Uniform() ) );
	dt << sort( by( dt:random ), , Order( Ascending ), replace table( 1 ) );
	dt << delete columns( "Random" );
	dt << select where( Row() <= i );
	dtSample = dt << subset( invisible, selected rows( 1 ), selected columns( 0 ) );
	
	// Run the regression
	fm = dtSample << Fit Model(
		Y( :NPN1 ),
		Effects( :PNP1, :PNP2 ),
		Personality( "Standard Least Squares" ),
		Emphasis( "Minimal Report" ),
		Run(
			:NPN1 << {Summary of Fit( 1 ), Analysis of Variance( 1 ), Parameter Estimates( 1 ),
			Lack of Fit( 0 ), Plot Actual by Predicted( 0 ), Plot Regression( 0 ),
			Plot Residual by Predicted( 0 ), Plot Effect Leverage( 0 )}
		)
	);
	
	// Capture the output data
	dtOut << Add Rows( 1 );
	dtOut:Instance[N Rows( dtOut )] = Loop Counter;
	dtOut:N[N Rows( dtOut )] = i;
	dtOut:RSquare[N Rows( dtOut )] = (Report( fm )["Summary of Fit"][1][2] << get)[1];
	dtOut:RSM[N Rows( dtOut )] = (Report( fm )["Summary of Fit"][1][2] << get)[3];
	dtOut:Intercept[N Rows( dtOut )] = (Report( fm )["Parameter Estimates"][1][3] << get)[1];
	dtOut:b1[N Rows( dtOut )] = (Report( fm )["Parameter Estimates"][1][3] << get)[2];
	dtOut:b2[N Rows( dtOut )] = (Report( fm )["Parameter Estimates"][1][3] << get)[3];
	
	// Clean up after our selfs
	fm << close window;
	Close( dtSample, nosave );
	
);

Re: Calculating linear regression slope in formula

jerry_cooper — Fri, 02 Aug 2019 15:06:32 GMT

Try plugging this formula into a new formula column (this will calculate the slope up to the current row):

(Row() * Summation( i = 1, Row(), :Time[i] * :Volume[i] )
-Summation( i = 1, Row(), :Time[i] ) * Summation( i = 1, Row(), :Volume[i] )) / (
Row() * Summation( i = 1, Row(), :Time[i] ^ 2 )
-Summation( i = 1, Row(), :Time[i] ) ^ 2)

To do the second part of your question would only require changing the index on the summation, i.e. from i=Row()-N to Row()+N.

Hope this helps.

-Jerry

Re: Calculating linear regression slope in formula

Mark_Bailey — Mon, 27 Feb 2017 21:00:29 GMT

You could also use a column formula. I opened Big Class.jmp from the Sample Data folder. (Select Help > Sample Data Library > Big Class.jmp > Open) I crated a new numeric column called moving slope. This example computes the moving slope with a span of 3 for Y = weight and X = height with the following column formula:

If( Row() > 1,
	x = J( Row(), 1, 1 ) || :height[Index( 1, Row() )]`;
	y = :weight[Index( 1, Row() )]`;
	(Inv( x` * x ) * x` * y)[2];
,
	Empty()
)

Then plot moving slope versus row in Graph Builder.

Re: Calculating linear regression slope in formula

bio_grad — Tue, 28 Feb 2017 05:17:49 GMT

This is exactly what I was looking for. I had no idea these operations were available!

Thank you!

Re: Calculating linear regression slope in formula

Steve_Kim — Thu, 27 Jul 2017 04:52:16 GMT

wow~~ Awesome!!!!!!

Thank you so much!!!!! : )

Re: Calculating linear regression slope in formula

Frank_Bowler — Tue, 23 Jan 2018 16:07:25 GMT

I am trying to apply your formula to calculate the moving slope with a window of 8, by which I mean (for Row() > 7), I want to calculate the slope of the linear regression for the first set of x and y data points [1 to 8], and then the next set of 8 data points incrementing by 1 [2 to 9], and so on. I am trying to calculate the maximum slope to get the initial rate of an enzymatic reaction.

Please can you tell me how you would modify your script to achieve this? I have tested your formula and I obtain the slope for all data points from Row 1 to Row n, rather than a moving window of defined size.

Thank you in advance

Re: Calculating linear regression slope in formula

Mark_Bailey — Tue, 23 Jan 2018 16:27:41 GMT

This modified formula produces a moving slope of span 8:

If( Row() > 7,
	x = J( 8, 1, 1 ) || :height[Index( Row() - 7, Row() )]`;
	y = :weight[Index( Row() - 7, Row() )]`;
	(Inv( x` * x ) * x` * y)[2];
)

Re: Calculating linear regression slope in formula

Frank_Bowler — Tue, 23 Jan 2018 16:43:23 GMT

Excellent, that works, thank you

Re: Calculating linear regression slope in formula

altug_bayram — Mon, 10 Sep 2018 18:36:46 GMT

Hi,

Is there any plan to put slope and intercept into formula builder of a future JMP version w/ a more straight forward direct function ?

thx.

Re: Calculating linear regression slope in formula

txnelson — Mon, 10 Sep 2018 21:25:40 GMT

Take a look at the Linear Regression() function, which could be used in a formula. It might be something you can use.

Re: Calculating linear regression slope in formula

rdzdec — Fri, 01 Feb 2019 19:16:02 GMT

Hi,

You mention a "span of 3", if we want to modify it to be moving slope of "N" samples, which value will need to change?

Thanks!

Re: Calculating linear regression slope in formula

TRR21 — Fri, 02 Aug 2019 06:55:23 GMT

Dear @txnelson, can you show how to tweak the above code to run fit models on rows selections based on each 'wafer' id/value, instead of the random sampling? Thanks!

Re: Calculating linear regression slope in formula

mksaad — Tue, 31 Mar 2020 21:47:14 GMT

@Mark_Bailey Hi mark, I am new to JMP and am trying to do something similar to bio_grad where I perform linear regression on 3 points, then 4 points, then so on in a column in my data table. I am trying to understand what your answer is actually doing and had a couple of questions. What does || mean? also what is J?

Re: Calculating linear regression slope in formula

txnelson — Tue, 31 Mar 2020 21:58:06 GMT

J() is a function that creates a matrix of identical values. The default is 1.

|| is an operator that indicate concatenation. So in Mark's example he is concatenating 2 matrices.

These functions and operators are fully documented in the Scripting Guide

Help==>JMP Document Library ….. Scripting Guide

Re: Calculating linear regression slope in formula

altug_bayram — Thu, 23 Apr 2020 19:17:51 GMT

hi ...

You mentioned "Linear Regression() function, which could be used in a formula" ...and getting a slope from a formula fitting to provided data set being the final objective, where would one go find Linear Regression() function or any help ?

Do you have any example ?

thanks

Re: Calculating linear regression slope in formula

Mark_Bailey — Thu, 23 Apr 2020 19:28:20 GMT

Select Help > Scripting Index. Enter "regression" in the filter box. You should find the Linear Regression() function in the second list. Select it to see information about this function on the right.

Re: Calculating linear regression slope in formula

altug_bayram — Thu, 23 Apr 2020 20:21:04 GMT

I am not looking for a script.
I am looking for a formula application (i.e. using formula builder)

Re: Calculating linear regression slope in formula

txnelson — Thu, 23 Apr 2020 20:46:42 GMT

A JMP formula can pretty much be anything you want it to be. My pointing you to the Linear Regression() function in the Scripting Index, happens to show it's example in the form of a script, but the function can also be used in a formula. Below is a silly example, of calculating the regression for first rows 1-3, then 1-4, 1-5, etc. for all of the rows in the Big Class data table. Here is the formula...….attached in the data table with the new formula column added

Re: Calculating linear regression slope in formula

Jeff_Perkinson — Thu, 23 Apr 2020 21:14:59 GMT

Can you give us some more background on what you're looking for?

Formulas, generally speaking, work across rows of a data table, evaluating one per row with column values from that row as inputs.

So, by asking for a regression function in the formula editor, one would expect that you had a matrix of input values in each row of your data table. Perhaps that's what you've got but that's an unusual structure.

So, with some more information we may be able to help you understand how you could use a formula for this, or perhaps, point you in a direction outside of the formula editor.