BookmarkSubscribeRSS Feed
Choose Language Hide Translation Bar
JD23

Not applicable

Joined:

Apr 3, 2019

Re: How to script calibration curve

It did work perfectly. And it does what I asked help for. But there were two parts of my initial post. The first was to know how to script the calibration just like you showed me.

The second, which is not related to the first, so it would be a completely different script, is to do the following:

 

The unknown table has 3 samples, each sample was analyzed for 74 Peps. 29 of those Peps I want to use them for calibration of the other 45. And I want to do that by fetching the AU 2 values from the calibration table. So the script should fetch the AU 2 values from the calibration table and join them with the 1-29 Peps for each sample in the unknown table, after which should make a calibration curve (linear regression) for each of the samples and attribute AU 2 values to the remainder 45 Peps.

 

Thank you for bearing with me!

I know that the values do not correlate, these were randomly generated values with the purpose of learning how to script. 

 

markbailey

Staff

Joined:

Jun 23, 2011

Solution

Re: How to script calibration curve

OK, thanks for clarifying.

 

Names Default to Here( 1 );

// check for sufficient tables
n tables = N Table();

If( n tables < 2,
	Throw( "Two data tables are required for this analysis" )
);

tables = List();
For( t = 1, t <= n tables, t++,
	Insert Into( tables, Eval( Data Table( t ) << Get Name ) );
);

// identify the calibration and unknown data tables
dlg = New Window( "Assay Unknown Samples", << Modal,
	H List Box(
		lb = List Box( tables ),
		Line Up Box( N Col( 2 ),
			Button Box( "Calibrator Table",
				sel = lb << Get Selected;
				cal lb << Append( sel );
			),
			cal lb = List Box( {},
				N Lines( 1 )
			),
			Button Box( "Unknown Table",
				sel = lb << Get Selected;
				unk lb << Append( sel );
			),
			unk lb = List Box( {},
				N Lines( 1 )
			)
		)
	),
	H List Box(
		Button Box( "OK",
			cal table = cal lb << Get Items;
			Show( cal table );
			unk table = unk lb << Get Items;
			Show( unk table );
		),
		Button Box( "Cancel" )
	)
);

If( dlg["Button"] == -1,
	Throw( "User cancelled" )
);

// unload the dialog
cal table = Data Table( cal table[1] );
unk table = Data Table( unk table[1] );

// include AU 2 values for calibrators from calibration table
combined = cal table << Join(
	With( unk table ),
	Copy formula( 0 ),
	Select With( :Sample, :Peps, :AU 1 ),
	Select( :AU 2 ),
	By Matching Columns( :Peps = :Peps ),
	Drop Multiples( 0, 0 ),
	Include Nonmatches( 0, 1 ),
	Preserve Main Table Order( 0 )
);
Close( cal table, No Save );
Close( unk table, No Save );

// distinquish peps
Current Data Table( combined );
combined << New Column( "Status", "Character", "Nominal" );
For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
);


// fit the calibration curve
dlg = combined << Fit Model(
	Y( :AU 1 ),
	Effects( :AU 2 ),
	Personality( "Standard Least Squares" ),
	Emphasis( "Minimal Report" ),
	Where( :Status == "Calibrator" )
);
fit = dlg << Run;

// determine the assay for the unknown samples from the calibration curve
For Each Row(
	fit << Inverse Prediction( Response( :AU 1[] ) );
);

// extract the assays into a new data table
assays = Report( fit )["Inverse Prediction"][TableBox(1)] << Make Combined Data Table;
fit << Close Window;
assays << Delete Columns( Column( assays, 1 ) );

// add the assays to the unknown table
final = combined << Join(
	With( assays ),
	Copy formula( 0 ),
	By Matching Columns( :AU 1 = :Specified AU 1 ),
	Drop Multiples( 0, 0 ),
	Include Nonmatches( 1, 0 ),
	Preserve Main Table Order( 1 )
);
Close( combined, No Save );
Close( assays, No Save );

final << Sort(
	By( :Peps, :Sample ),
	Order( Ascending, Ascending ),
	Replace Table
);

 

Some of the predicted AU 2 values are missing due to the random measurements of AU 1.

Learn it once, use it forever!
JD23

Not applicable

Joined:

Apr 3, 2019

Re: How to script calibration curve

This is perfect! Thank you so much Mark!

Highlighted
JD23

Not applicable

Joined:

Apr 3, 2019

Re: How to script calibration curve

I think I spoke too soon. There is one issue Mark. Right now the script is pulling all AU 2 values from the calib table to the unk table fort the calibrators and that is fine. But then the new table is using all that were assigned with the calibrator designation to be used in the fit model and the unknowns are being predicted from that one model (that has the calibrators from sample 1, 2 and 3 pooled together), right?

 

What I need is a fit model per sample and that the unknowns of that sample be determined with the fit model associated with that sample only. 

markbailey

Staff

Joined:

Jun 23, 2011

Re: How to script calibration curve

So the three sample sets should be handled separate?

Learn it once, use it forever!
JD23

Not applicable

Joined:

Apr 3, 2019

Re: How to script calibration curve

Yes, they have the 29 peps as internal calibrators. Those specific 29 peps should fetch the AU 2 values from the calib table and then each set calibrate the remainder peps associated to the sample number.
I think I made this more confusing by asking for two different things in the same post. I’m sorry for that.
markbailey

Staff

Joined:

Jun 23, 2011

Solution

Re: How to script calibration curve

Names Default to Here( 1 );

// check for sufficient tables
n tables = N Table();

If( n tables < 2,
	Throw( "Two data tables are required for this analysis" )
);

tables = List();
For( t = 1, t <= n tables, t++,
	Insert Into( tables, Eval( Data Table( t ) << Get Name ) );
);

// identify the calibration and unknown data tables
dlg = New Window( "Assay Unknown Samples", << Modal,
	H List Box(
		lb = List Box( tables ),
		Line Up Box( N Col( 2 ),
			Button Box( "Calibrator Table",
				sel = lb << Get Selected;
				cal lb << Append( sel );
			),
			cal lb = List Box( {},
				N Lines( 1 )
			),
			Button Box( "Unknown Table",
				sel = lb << Get Selected;
				unk lb << Append( sel );
			),
			unk lb = List Box( {},
				N Lines( 1 )
			)
		)
	),
	H List Box(
		Button Box( "OK",
			cal table = cal lb << Get Items;
			Show( cal table );
			unk table = unk lb << Get Items;
			Show( unk table );
		),
		Button Box( "Cancel" )
	)
);

If( dlg["Button"] == -1,
	Throw( "User cancelled" )
);

// unload the dialog
cal table = Data Table( cal table[1] );
unk table = Data Table( unk table[1] );

// include AU 2 values for calibrators from calibration table
combined = cal table << Join( With( unk table ),
	Copy formula( 0 ),
	Select With( :Sample, :Peps, :AU 1 ),
	Select( :AU 2 ),
	By Matching Columns( :Peps = :Peps ),
	Drop Multiples( 0, 0 ),
	Include Nonmatches( 0, 1 ),
	Preserve Main Table Order( 0 )
);
Close( cal table, No Save );
Close( unk table, No Save );

// distinquish peps
Current Data Table( combined );
combined << New Column( "Status", "Character", "Nominal" );
For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
);

// placeholders for matching information
combined
	<< New Column( "Predicted AU 2", "Numeric", "Continuous" )
	<< New Column( "Lower 95%", "Numeric", "Continuous" )
	<< New Column( "Upper 95%", "Numeric", "Continuous" );

// handle each sample set separately
For( s = 1, s <= 3, s++,
	// isolate sample
	ss = Char( s );
	combined
		<< Select Where( :Sample == ss )
		<< Invert Row Selection
		<< Exclude;

	// fit the calibration curve for this sample
	dlg = combined << Fit Model(
		Y( :AU 1 ),
		Effects( :AU 2 ),
		Personality( "Standard Least Squares" ),
		Emphasis( "Minimal Report" ),
		Where( :Status == "Calibrator" )
	);
	fit = dlg << Run;

	// determine the assay for the unknown samples from the calibration curve
	For Each Row(
		If( Not( Excluded( Row State( Row() ) ) ),
			fit << Inverse Prediction( Response( :AU 1[] ) );
		);
	);

	// access report layer
	fit rep = fit << Report;

	// remove extraneous string column box that interferes with making combined data table
	(fit rep << XPath( "//StringColBox[StringColBoxHeader=\!"Type of CI\!"]")) << Delete;

	// extract the assays into a new data table
	assays = fit rep["Inverse Prediction"][TableBox(1)] << Make Combined Data Table;
	fit << Close Window;

	// prepare predicted values for join
	assays
		<< Delete Columns( Column( assays, 1 ) )
		<< New Column( "Sample", "Character", "NominaL" )
		<< New Column( "Peps", "Character", "NominaL" );
	Current Data Table( combined );
	r = 1;
	For Each Row(
		If( Not( Excluded( Row State( Row() ) ) ),
			assays:Sample[r] = combined:Sample;
			assays:Peps[r] = combined:Peps;
			r++;
		);
	);
	combined << Clear Row States;
	combined << Update(
		With( assays ),
		Match Columns(
			:Sample = :Sample,
			:Peps = :Peps
		)
	);
	Close( assays, No Save );
);
Learn it once, use it forever!
JD23

Not applicable

Joined:

Apr 3, 2019

Re: How to script calibration curve

It works perfectly! Thank you.

 

I think I understand what each line of code is doing. I do have some doubts about these:

 

For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
For( s = 1, s <= 3, s++,
	// isolate sample
	ss = Char( s );
	combined
		<< Select Where( :Sample == ss )
		<< Invert Row Selection
		<< Exclude;

 Could you explain them to me, please?

markbailey

Staff

Joined:

Jun 23, 2011

Re: How to script calibration curve

You doubt my code? Well, it is your prerogative.

 

The first example populates a new column without using a column formula property.

 

 

For Each Row(
	:Status = If( 1 <= Num( Regex( :Peps, "\w{3}(\d+)", "\1" ) ) <= 29, "Calibrator", "Unknown" );
);

 

The For Each Row() function iterates over the rows of the current data table. It handles indexing rows automatically. So the assignment occurs row-wise. The right side of the assignment is the If() function. The first argument is interpreted as a Boolean expression. If this expression evaluates to true, the the second argument ("Calibrator") is evaluated and returned. Otherwise, the third argument ("Unknown") is evaluated and returned. The first argument determines if the Peps number is in the range of 1 to 29. The number string is extracted with a regular expression and converted to a number. The regular expression matches a pattern of three word characters followed by one or more digit characters. The back reference returns the only capturing group.

 

The second example iterates over the three samples in the data set.

 

 

For( s = 1, s <= 3, s++,
	// isolate sample
	ss = Char( s );
	combined
		<< Select Where( :Sample == ss )
		<< Invert Row Selection
		<< Exclude;

 

 

It uses an integer index that is converted to a character string to match the data column attribute. It sends a cascade of messages to the combined data table to (1) select all the rows for the current sample, (2) change the selection to all the rows not for the current sample, and (3) excludes those rows. This approach is usually simpler and faster than subsetting data tables and later joining and concatenating the data tables with the results.

Learn it once, use it forever!
JD23

Not applicable

Joined:

Apr 3, 2019

Re: How to script calibration curve

I think you've misunderstood me. By doubts, I meant that I didn't understand the function of the code. I know very very little about coding and I wouldn't dare to doubt your code!  I'm just trying to understand and learn from you.

 

I think I understand the first example. 

In the second example, if instead of numbers the samples also have a text designation, how would the code be changed? For example, instead of sample 1, 2 and 3, we have sample 001-AAA, 002-AAA and 003-AAA?