topic Re: Looking for a formula to find the "elbow" in my data in Discussions

Looking for a formula to find the "elbow" in my data

RZordich — Thu, 08 Aug 2024 00:12:44 GMT

Hi JMP friends,

(On regular JMP 18.0.1, example data attached)

I am analyzing some data wherein I need to find the time (x point) at which my data has an "elbow" or inflection. I have an initial mini-peak in my data and have been doing this work manually. However, this is slow and I would like to automate it, as I will be doing this for hundreds of tests that return graphs similar to the below, with varying peak times/amounts and varying 'flat' times. The 'flat' times are generally not very flat, but rather very gradually decreasing. You can see in the attached image of the graph (Example/dummy data) what spot/area I would like to find.

Any help on what I could do would be really helpful. I think setting up a flag column would be a good way to find this, but am not sure of the JSL or formula I can use to have the flag work.

Any help on figuring this out would be great, thank you!

Re: Looking for a formula to find the "elbow" in my data

txnelson — Thu, 08 Aug 2024 01:35:59 GMT

I approached finding the elbow as the data point, where the next 10 data points are all greater than the previous data point. This is absolutely a made up algorithm. But it looks like it might be a possible way to identify the elbow.

Here is the formula I created for the column I named Elbow

As Constant(
	flag = 0;
	lookAhead = 10;
);
val = .;
If( flag == 0 & Row() < N Rows( Current Data Table() ) - lookAhead,
	If(
		Min(
			:Counts[Index( Row() + 1, Row() + lookAhead + 1 )] - :Counts[
			Index( Row(), Row() + lookAhead )]
		) > 0,
		val = (flag = 1)
	)
);
val;

Re: Looking for a formula to find the "elbow" in my data

Byron_JMP — Wed, 14 Aug 2024 18:57:31 GMT

Ok, so this solution is way-WAY more complicated.

Imagine if you fitted a function to the peak curve, and then calculated the second derivative?

It's easier to see graphically. In the figure, the first trace is the original data, a model for a fit, the first and second derivatives of the fit. It might not be easy to see, but the maximum of the second derivative (Blue line) is the middle of the elbow on the first line (triangle marker).

Here's how I got there with JSL. (Note: I changed the name of the first column)
In this script I let JMP sort out all the calculus along with the curve fit.


Names Default To Here( 1 );
dt=Open( "$Examples/Example for Elbow.jmp" );

obj=dt <<Fit Curve( Y( :Counts ), X( :time), Fit Skew Normal Peak );
obj << (Fit["Skew Normal Peak"] << Save Prediction Formula);
obj << Close Window;

formula=column(3)<<get formula;
derivative = Derivative(name expr(formula), Time);
second = Derivative(name expr(derivative), Time);
dt<<new column("First Derivative", formula (name expr(derivative)));
dt<<new column("Second Derivative", formula (name expr(second)));
dt<<new column("Peak Start", formula( Col Maximum( :Second Derivative ) == :Second Derivative, 1 ) );
dt<<Select where(:Peak Start == 1)<<Colors( "Red" )<<Markers( "Filled Up Triangle" );

The last step turns the "elbow" row into a red triangle marker. But you could just as easily subset the selected row into a table and add it to a table with all your other fits. I didn't try this with hundreds of tables but I think it would execute fairly quickly if you put it in a loop to iterate through a lot of data tables.

Re: Looking for a formula to find the "elbow" in my data

MikeD_Anderson — Thu, 22 Aug 2024 14:21:00 GMT

You might have a look at the Spectris Add-in. It can be used to fit both the primary peak and the secondary peak (probably what's causing that shoulder.

https://community.jmp.com/t5/JMP-Add-Ins/Spectris-Graphical-spectral-data-processing/ta-p/650521

Re: Looking for a formula to find the "elbow" in my data

RZordich — Thu, 22 Aug 2024 17:37:26 GMT

Hi Byron,

Thanks for this one! I have tried 2nd derivative on a smoothed function, and it didn't seem to work. But I haven't tried the skew normal function. I'll give that a try and see what that catches for me! I'm also looking at figuring out a way to now script in the 'Kneedle' algorithm, which is something I found that finds the 'elbow' point in graphs really well.

Thanks!

Rochelle