topic Re: How to script calibration curve in Discussions

How to script calibration curve

JD23 — Wed, 03 Apr 2019 12:42:58 GMT

Hi, my level of scripting is really basic and I hope you can help me.

How do I write a script to achieve the following?

I have a table with the known concentrations of the calibrators. In each assay (one table per assay) I have the signal of several unknowns as well as the signal for the calibrators, it is basically an internal calibration curve for each assay.

Is it possible to create a script that takes the values from the table of the known concentrations, plots them against their signals for that specific assay (assay table) and then uses the linear regression to calculate the concentration of the unknowns?

Thanks in advance for the help!

Re: How to script calibration curve

Mark_Bailey — Thu, 04 Apr 2019 12:14:17 GMT

I suggest that you first learn how to perform this analysis interactively using JMP. See Help > Books > Predictive and Specialized Modeling > chapter about Fit Curve. It will explain how to use this platform for your purpose. It includes examples to guide you and help you practice before you attempt to analyze your own data.

What model do you currently use for your calibration curve? You mention linear regression. If this is the case, then click the red triangle next to Fit Curve and select Polynomials > Fit Linear. Now click the red triangle next to Linear and select Custom Inverse Prediction. A dialog appears in which you can enter the unknown responses.

Use the Fit Curve platform interactively first to be sure that this analysis is appropriate and yields the results you seek. Then we can help with a script.

Re: How to script calibration curve

JD23 — Thu, 11 Apr 2019 09:07:56 GMT

Thanks Mark.

I have followed the steps you mention to get the custom inverse prediction. Yes, it is a linear regression.

My problem with that is the limited number of unknows I can enter. The scripting question was asked having in mind that I have thousands of unknowns and I assumed that scripting the procedure would allow me to values for all those unknowns.

Re: How to script calibration curve

Mark_Bailey — Thu, 11 Apr 2019 11:15:45 GMT

Thanks for verifying the analysis method and process. Onward to the script!

Can you provide a sample of data? You can use generic names (anonymous data). That way I will understand what the script should expect from the data table.

Re: How to script calibration curve

JD23 — Fri, 12 Apr 2019 12:40:59 GMT

Here are two example tables. One is the calibrator table. The other table is the one that has the measured values (AU 1) and needs calibration to obtain the unknown values (AU 2). To complicate things a bit, this table has the measurements for 3 different samples and each of these 3 samples also contained the components that were used to make the calibrator. So in each sample we have an internal calibrator, sort of speak.

It would be great to know how to script a calibration that completely omits the fact that we have an internal calibrator (plain simple linear regression of what is on the calibrator table and use the expression to obtain the unknowns from samples table). And it would be even better to know how to script a calibration that uses the measured values (AU 1) of the calibrator components in each sample (samples table), plots it against the known values (AU 2) in the calibrator table and calculates the unknowns for that specific sample, and does the same for each sample present in the samples table.

English is not my native language so please let me know if you need me to be a bit clearer.

I hope you can help me.

Cheers.

Re: How to script calibration curve

Mark_Bailey — Fri, 12 Apr 2019 16:49:57 GMT

I think I understand the basic problem but I might have missed a detail. Try this script with both data tables open (calibration table and unknow table):

Names Default to Here( 1 );

// find all open data tables
tables = Get Data Table List();
If( N Items( tables ) < 2,
	Throw( "Two data tables are required for this analysis" )
);

dlg = New Window( "Assay Unknown Samples", << Modal,
	H List Box(
		lb = List Box( tables ),
		Line Up Box( N Col( 2 ),
			Button Box( "Calibrator Table",
				sel = lb << Get Selected;
				cal lb << Append( sel );
			),
			cal lb = List Box( {},
				N Lines( 1 )
			),
			Button Box( "Unknown Table",
				sel = lb << Get Selected;
				unk lb << Append( sel );
			),
			unk lb = List Box( {},
				N Lines( 1 )
			)
		)
	),
	H List Box(
		Button Box( "OK",
			cal table = cal lb << Get Items;
			Show( cal table );
			unk table = unk lb << Get Items;
			Show( unk table );
		),
		Button Box( "Cancel" )
	)
);

If( dlg["Button"] == -1,
	Throw( "User cancelled" )
);

// unload dialog
cal table = Data Table( cal table[1] );
unk table = Data Table( unk table[1] );

// perform the linear calibration
dlg = cal table << Fit Model(
	Y( :AU 1 ),
	Effects( :AU 2 ),
	Personality( "Standard Least Squares" ),
	Emphasis( "Minimal Report" )
);
fit = dlg << Run;

// perform inverse prediction on unknown
Current Data Table( unk table );
For Each Row(
	fit << Inverse Prediction( Response( :AU 1[] ) );
);

// unload unknown assays
assays = Report( fit )["Inverse Prediction"][TableBox(1)] << Make Combined Data Table;
fit << Close Window;
assays << Delete Columns( Column( assays, 1 ) );

// join with unknown table
unk table << Join(
	With( assays ),
	By Row Number
);
Close( assays, No Save );

// plot predictions with 95% confidence intervals versus actual
unk table << Graph Builder(
	Size( 534, 454 ),
	Show Control Panel( 0 ),
	Variables(
		X( :AU 1 ),
		Y( :Lower 95% ),
		Y( :Upper 95%, Position( 1 ) ),
		Y( :Predicted AU 2, Position( 1 ) )
	),
	Elements(
		Bar( X, Y( 1 ), Y( 2 ), Y( 3 ), Legend( 8 ), Bar Style( "Interval" ) )
	),
	SendToReport(
		Dispatch( {}, "AU 1", ScaleBox, {Format( "Best", 12 )} ),
		Dispatch( {}, "Lower 95%", ScaleBox, {Format( "Fixed Dec", 12, 0 )} )
	)
);

The script is not flexible but it is a start.

Re: How to script calibration curve

JD23 — Fri, 12 Apr 2019 19:14:23 GMT

Maybe it's a JMP version issue (I have JMP 13.1) but the script does not work. I used the JSL Debugger and I get an error right at the start: Name Unresolved: Get Data Table List in access or evaluation of 'Get Data Table List' , Get Data Table List() /*###*/

Re: How to script calibration curve

Mark_Bailey — Fri, 12 Apr 2019 19:43:47 GMT

It is a version issue. I opened JMP 13 and (1) the function was no longer highlighted as such and (2) it is not found in the Scripting Index.

I hate when that happens!

Try this work-around:

Names Default to Here( 1 );

// find all open data tables

/* for JMP 14 and later:
tables = Get Data Table List();
*/

n tables = N Table();

If( n tables < 2,
	Throw( "Two data tables are required for this analysis" )
);

tables = List();
For( t = 1, t <= n tables, t++,
	Insert Into( tables, Eval( Data Table( t ) << Get Name ) );
);

// present open tables for assignment as calibration or unknown.

dlg = New Window( "Assay Unknown Samples", << Modal,
	H List Box(
		lb = List Box( tables ),
		Line Up Box( N Col( 2 ),
			Button Box( "Calibrator Table",
				sel = lb << Get Selected;
				cal lb << Append( sel );
			),
			cal lb = List Box( {},
				N Lines( 1 )
			),
			Button Box( "Unknown Table",
				sel = lb << Get Selected;
				unk lb << Append( sel );
			),
			unk lb = List Box( {},
				N Lines( 1 )
			)
		)
	),
	H List Box(
		Button Box( "OK",
			cal table = cal lb << Get Items;
			Show( cal table );
			unk table = unk lb << Get Items;
			Show( unk table );
		),
		Button Box( "Cancel" )
	)
);

If( dlg["Button"] == -1,
	Throw( "User cancelled" )
);

// unload dialog
cal table = Data Table( cal table[1] );
unk table = Data Table( unk table[1] );

// perform the linear calibration
dlg = cal table << Fit Model(
	Y( :AU 1 ),
	Effects( :AU 2 ),
	Personality( "Standard Least Squares" ),
	Emphasis( "Minimal Report" )
);
fit = dlg << Run;

// perform inverse prediction on unknown
Current Data Table( unk table );
For Each Row(
	fit << Inverse Prediction( Response( :AU 1[] ) );
);

// unload unknown assays
assays = Report( fit )["Inverse Prediction"][TableBox(1)] << Make Combined Data Table;
fit << Close Window;
assays << Delete Columns( Column( assays, 1 ) );

// join with unknown table
unk table << Join(
	With( assays ),
	By Row Number
);
Close( assays, No Save );

// plot predictions with 95% confidence intervals versus actual
unk table << Graph Builder(
	Size( 534, 454 ),
	Show Control Panel( 0 ),
	Variables(
		X( :AU 1 ),
		Y( :Lower 95% ),
		Y( :Upper 95%, Position( 1 ) ),
		Y( :Predicted AU 2, Position( 1 ) )
	),
	Elements(
		Bar( X, Y( 1 ), Y( 2 ), Y( 3 ), Legend( 8 ), Bar Style( "Interval" ) )
	),
	SendToReport(
		Dispatch( {}, "AU 1", ScaleBox, {Format( "Best", 12 )} ),
		Dispatch( {}, "Lower 95%", ScaleBox, {Format( "Fixed Dec", 12, 0 )} )
	)
);

Re: How to script calibration curve

JD23 — Sat, 13 Apr 2019 10:18:34 GMT

Thanks Mark, it worked now.

From what I understood, looking at the script, this is for the straightforward calibration using AU 1 and AU 2 values from the calibrator table, yes?

How can we rearrange it so that the calibrator is the AU 1 values specified in the calibrator table vs the AU 2 from the unknown table (making it a different calibrator for each sample) and then predict the AU 2 values for the rest of the components in the sample?

Re: How to script calibration curve

Mark_Bailey — Sat, 13 Apr 2019 20:25:36 GMT

I am confused.

First of all, did it work or not? Did it work up to a point?

Second, yes, the calibration is performed entirely with the AU 1 and AU 2 values in the calibration table. Is that not correct?

Third, you don't have AU 2 values in the unknown table. Do you mean to use the predicted unknown AU 2 values?

Fourth, is this a change to the calibration definition or is it a new requirement (follow on step) after predicting the AU 2 in the unknown table?

Fifth, what is Sample? How does it relate to Peps?

Sixth, shouldn't the AU 1 in the calibrator table for Pep1 to Pep29 be correlated? They are not:

I am probably missing something very important and very basic.

Re: How to script calibration curve

JD23 — Sat, 13 Apr 2019 21:18:19 GMT

It did work perfectly. And it does what I asked help for. But there were two parts of my initial post. The first was to know how to script the calibration just like you showed me.

The second, which is not related to the first, so it would be a completely different script, is to do the following:

The unknown table has 3 samples, each sample was analyzed for 74 Peps. 29 of those Peps I want to use them for calibration of the other 45. And I want to do that by fetching the AU 2 values from the calibration table. So the script should fetch the AU 2 values from the calibration table and join them with the 1-29 Peps for each sample in the unknown table, after which should make a calibration curve (linear regression) for each of the samples and attribute AU 2 values to the remainder 45 Peps.

Thank you for bearing with me!

I know that the values do not correlate, these were randomly generated values with the purpose of learning how to script.

Re: How to script calibration curve

Mark_Bailey — Mon, 15 Apr 2019 12:12:59 GMT

OK, thanks for clarifying.

Names Default to Here( 1 );

// check for sufficient tables
n tables = N Table();

If( n tables < 2,
	Throw( "Two data tables are required for this analysis" )
);

tables = List();
For( t = 1, t <= n tables, t++,
	Insert Into( tables, Eval( Data Table( t ) << Get Name ) );
);

// identify the calibration and unknown data tables
dlg = New Window( "Assay Unknown Samples", << Modal,
	H List Box(
		lb = List Box( tables ),
		Line Up Box( N Col( 2 ),
			Button Box( "Calibrator Table",
				sel = lb << Get Selected;
				cal lb << Append( sel );
			),
			cal lb = List Box( {},
				N Lines( 1 )
			),
			Button Box( "Unknown Table",
				sel = lb << Get Selected;
				unk lb << Append( sel );
			),
			unk lb = List Box( {},
				N Lines( 1 )
			)
		)
	),
	H List Box(
		Button Box( "OK",
			cal table = cal lb << Get Items;
			Show( cal table );
			unk table = unk lb << Get Items;
			Show( unk table );
		),
		Button Box( "Cancel" )
	)
);

If( dlg["Button"] == -1,
	Throw( "User cancelled" )
);

// unload the dialog
cal table = Data Table( cal table[1] );
unk table = Data Table( unk table[1] );

// include AU 2 values for calibrators from calibration table
combined = cal table << Join(
	With( unk table ),
	Copy formula( 0 ),
	Select With( :Sample, :Peps, :AU 1 ),
	Select( :AU 2 ),
	By Matching Columns( :Peps = :Peps ),
	Drop Multiples( 0, 0 ),
	Include Nonmatches( 0, 1 ),
	Preserve Main Table Order( 0 )
);
Close( cal table, No Save );
Close( unk table, No Save );

// distinquish peps
Current Data Table( combined );
combined << New Column( "Status", "Character", "Nominal" );
For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
);


// fit the calibration curve
dlg = combined << Fit Model(
	Y( :AU 1 ),
	Effects( :AU 2 ),
	Personality( "Standard Least Squares" ),
	Emphasis( "Minimal Report" ),
	Where( :Status == "Calibrator" )
);
fit = dlg << Run;

// determine the assay for the unknown samples from the calibration curve
For Each Row(
	fit << Inverse Prediction( Response( :AU 1[] ) );
);

// extract the assays into a new data table
assays = Report( fit )["Inverse Prediction"][TableBox(1)] << Make Combined Data Table;
fit << Close Window;
assays << Delete Columns( Column( assays, 1 ) );

// add the assays to the unknown table
final = combined << Join(
	With( assays ),
	Copy formula( 0 ),
	By Matching Columns( :AU 1 = :Specified AU 1 ),
	Drop Multiples( 0, 0 ),
	Include Nonmatches( 1, 0 ),
	Preserve Main Table Order( 1 )
);
Close( combined, No Save );
Close( assays, No Save );

final << Sort(
	By( :Peps, :Sample ),
	Order( Ascending, Ascending ),
	Replace Table
);

Some of the predicted AU 2 values are missing due to the random measurements of AU 1.

Re: How to script calibration curve

JD23 — Mon, 15 Apr 2019 12:30:55 GMT

This is perfect! Thank you so much Mark!

Re: How to script calibration curve

JD23 — Mon, 15 Apr 2019 13:55:49 GMT

I think I spoke too soon. There is one issue Mark. Right now the script is pulling all AU 2 values from the calib table to the unk table fort the calibrators and that is fine. But then the new table is using all that were assigned with the calibrator designation to be used in the fit model and the unknowns are being predicted from that one model (that has the calibrators from sample 1, 2 and 3 pooled together), right?

What I need is a fit model per sample and that the unknowns of that sample be determined with the fit model associated with that sample only.

Re: How to script calibration curve

Mark_Bailey — Mon, 15 Apr 2019 16:20:46 GMT

So the three sample sets should be handled separate?

Re: How to script calibration curve

JD23 — Mon, 15 Apr 2019 16:37:28 GMT

Yes, they have the 29 peps as internal calibrators. Those specific 29 peps should fetch the AU 2 values from the calib table and then each set calibrate the remainder peps associated to the sample number.
I think I made this more confusing by asking for two different things in the same post. I’m sorry for that.

Re: How to script calibration curve

Mark_Bailey — Mon, 15 Apr 2019 18:13:47 GMT

Names Default to Here( 1 );

// check for sufficient tables
n tables = N Table();

If( n tables < 2,
	Throw( "Two data tables are required for this analysis" )
);

tables = List();
For( t = 1, t <= n tables, t++,
	Insert Into( tables, Eval( Data Table( t ) << Get Name ) );
);

// identify the calibration and unknown data tables
dlg = New Window( "Assay Unknown Samples", << Modal,
	H List Box(
		lb = List Box( tables ),
		Line Up Box( N Col( 2 ),
			Button Box( "Calibrator Table",
				sel = lb << Get Selected;
				cal lb << Append( sel );
			),
			cal lb = List Box( {},
				N Lines( 1 )
			),
			Button Box( "Unknown Table",
				sel = lb << Get Selected;
				unk lb << Append( sel );
			),
			unk lb = List Box( {},
				N Lines( 1 )
			)
		)
	),
	H List Box(
		Button Box( "OK",
			cal table = cal lb << Get Items;
			Show( cal table );
			unk table = unk lb << Get Items;
			Show( unk table );
		),
		Button Box( "Cancel" )
	)
);

If( dlg["Button"] == -1,
	Throw( "User cancelled" )
);

// unload the dialog
cal table = Data Table( cal table[1] );
unk table = Data Table( unk table[1] );

// include AU 2 values for calibrators from calibration table
combined = cal table << Join( With( unk table ),
	Copy formula( 0 ),
	Select With( :Sample, :Peps, :AU 1 ),
	Select( :AU 2 ),
	By Matching Columns( :Peps = :Peps ),
	Drop Multiples( 0, 0 ),
	Include Nonmatches( 0, 1 ),
	Preserve Main Table Order( 0 )
);
Close( cal table, No Save );
Close( unk table, No Save );

// distinquish peps
Current Data Table( combined );
combined << New Column( "Status", "Character", "Nominal" );
For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
);

// placeholders for matching information
combined
	<< New Column( "Predicted AU 2", "Numeric", "Continuous" )
	<< New Column( "Lower 95%", "Numeric", "Continuous" )
	<< New Column( "Upper 95%", "Numeric", "Continuous" );

// handle each sample set separately
For( s = 1, s <= 3, s++,
	// isolate sample
	ss = Char( s );
	combined
		<< Select Where( :Sample == ss )
		<< Invert Row Selection
		<< Exclude;

	// fit the calibration curve for this sample
	dlg = combined << Fit Model(
		Y( :AU 1 ),
		Effects( :AU 2 ),
		Personality( "Standard Least Squares" ),
		Emphasis( "Minimal Report" ),
		Where( :Status == "Calibrator" )
	);
	fit = dlg << Run;

	// determine the assay for the unknown samples from the calibration curve
	For Each Row(
		If( Not( Excluded( Row State( Row() ) ) ),
			fit << Inverse Prediction( Response( :AU 1[] ) );
		);
	);

	// access report layer
	fit rep = fit << Report;

	// remove extraneous string column box that interferes with making combined data table
	(fit rep << XPath( "//StringColBox[StringColBoxHeader=\!"Type of CI\!"]")) << Delete;

	// extract the assays into a new data table
	assays = fit rep["Inverse Prediction"][TableBox(1)] << Make Combined Data Table;
	fit << Close Window;

	// prepare predicted values for join
	assays
		<< Delete Columns( Column( assays, 1 ) )
		<< New Column( "Sample", "Character", "NominaL" )
		<< New Column( "Peps", "Character", "NominaL" );
	Current Data Table( combined );
	r = 1;
	For Each Row(
		If( Not( Excluded( Row State( Row() ) ) ),
			assays:Sample[r] = combined:Sample;
			assays:Peps[r] = combined:Peps;
			r++;
		);
	);
	combined << Clear Row States;
	combined << Update(
		With( assays ),
		Match Columns(
			:Sample = :Sample,
			:Peps = :Peps
		)
	);
	Close( assays, No Save );
);

Re: How to script calibration curve

JD23 — Wed, 17 Apr 2019 06:13:47 GMT

It works perfectly! Thank you.

I think I understand what each line of code is doing. I do have some doubts about these:

For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );

For( s = 1, s <= 3, s++,
	// isolate sample
	ss = Char( s );
	combined
		<< Select Where( :Sample == ss )
		<< Invert Row Selection
		<< Exclude;

Could you explain them to me, please?

Re: How to script calibration curve

Mark_Bailey — Wed, 17 Apr 2019 11:18:44 GMT

You doubt my code? Well, it is your prerogative.

The first example populates a new column without using a column formula property.

For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
);

The For Each Row() function iterates over the rows of the current data table. It handles indexing rows automatically. So the assignment occurs row-wise. The right side of the assignment is the If() function. The first argument is interpreted as a Boolean expression. If this expression evaluates to true, the the second argument ("Calibrator") is evaluated and returned. Otherwise, the third argument ("Unknown") is evaluated and returned. The first argument determines if the Peps number is in the range of 1 to 29. The number string is extracted with a regular expression and converted to a number. The regular expression matches a pattern of three word characters followed by one or more digit characters. The back reference returns the only capturing group.

The second example iterates over the three samples in the data set.

For( s = 1, s <= 3, s++,
	// isolate sample
	ss = Char( s );
	combined
		<< Select Where( :Sample == ss )
		<< Invert Row Selection
		<< Exclude;

It uses an integer index that is converted to a character string to match the data column attribute. It sends a cascade of messages to the combined data table to (1) select all the rows for the current sample, (2) change the selection to all the rows not for the current sample, and (3) excludes those rows. This approach is usually simpler and faster than subsetting data tables and later joining and concatenating the data tables with the results.

Re: How to script calibration curve

JD23 — Wed, 17 Apr 2019 11:41:16 GMT

I think you've misunderstood me. By doubts, I meant that I didn't understand the function of the code. I know very very little about coding and I wouldn't dare to doubt your code! I'm just trying to understand and learn from you.

I think I understand the first example.

In the second example, if instead of numbers the samples also have a text designation, how would the code be changed? For example, instead of sample 1, 2 and 3, we have sample 001-AAA, 002-AAA and 003-AAA?