topic Re: Some way to get constants from the Johnson Transformation in Discussions

Some way to get constants from the Johnson Transformation

shampton82 — Wed, 26 Mar 2025 02:33:08 GMT

Hey everyone,

I was trying to develop a script that would transform a column using the Johnson Transform and then be able to grab the constants from the formula to be able to put back into the inverse formula so I could use that to calculate the spec limits in their original units.

So the work flow will be like this

select a column-> transform using Johnson-> Run Control Chart-> see what the control limits are and enter into my user interface,-> have spec limits spit out in original units-> run control chart again in original units and input calculated from previous steps.

I think I can handle most of this but I'm stumped on how to grab the constants from the transform equation.

So from here

to put in here

Thanks for any ideas!

Steve

Re: Some way to get constants from the Johnson Transformation

jthi — Wed, 26 Mar 2025 05:25:27 GMT

Extract the formula into an expression and then take values from there. After you have the expression, you can then for example use Arg() to pick values from the formula

Names Default To Here(1);

dt = Open("$SAMPLE_DATA/Big Class.jmp");

new_cols = dt << New Formula Column(
	Operation(Category("Distributional"), "Johnson Normalizing"),
	Columns(:weight)
);

new_col = Remove From(new_cols, 1)[1];
f_expr = Column(dt, new_col << get name) << Get Formula;

Show(Arg(f_expr, 1), Arg(f_expr, 2));

There are few posts regarding this in community, here is one

Re: Some way to get constants from the Johnson Transformation

txnelson — Wed, 26 Mar 2025 10:22:15 GMT

One method that I have used successfully is to use Successive Approximations to find the value in the original distribution for a value from the transformed distribution. It basically takes values from the original distribution and passes it through the formula created by the transformation formula until it finds a match with the calculated value from the transform distribution. Here is an example that fits a Johnson distribution, finds the mean of the transformed column and then finds the value in the original distribution for that transform mean value. In the examples case, it just writes the back transformed value to the log.

Names Default To Here( 1 );
dt = 
// Open Data Table: Semiconductor Capability.jmp
// → Data Table( "Semiconductor Capability" )
Open( "$SAMPLE_DATA/Semiconductor Capability.jmp" );

theColumn = "NPN1";

Distribution(
	Continuous Distribution(
		Column( as column(theColumn) ),
		Process Capability( Use Column Property Specs ),
		Fit Johnson(save distribution formula)
	),
	SendToReport(
		Dispatch( {"NPN1"}, "Process Capability", OutlineBox, {Close( 1 )} )
	)
);

// Find the back transform of the calculated mean of the transformed distribution
theTargetCol = Column( dt, N Cols( dt ) ) << get name;
theFormula = Char( As Column( dt, theTargetCol ) << get formula );
theTarget = Col Mean( Column( dt, theTargetCol ) );

// Change the reference to the original column used in the transformation to _X_
// so it can be used in the back transform
Substitute Into( theFormula, ":"|| theColumn, "_X_" );
// The formula needs to create a memory variable to allow for comparison.
// Add the resulting variable name to be the resulting variable when the
// formula is executed
theFormula = "theResult = " || theFormula || ";";

// Successive Approximation will be used to get the back transform
// Use the min and max of the original column to start the approximations
theMax = Col Max(as column(theColumn) );
theMin = Col Min( as column(theColumn)  );

// Loop through the approximations for up to 100 loops to find the value
For( i = 1, i <= 100, i++,
	_X_ = Mean( theMax, theMin );
	Eval( Parse( theFormula ) );
	If(
		theResult > theTarget, theMax = _X_,
		theResult < theTarget, theMin = _X_,
		Break()
	);
);

// Display the findings in the log
Show( theTarget, theResult, _X_, i );

Re: Some way to get constants from the Johnson Transformation

shampton82 — Wed, 26 Mar 2025 16:15:21 GMT

Thanks @jthi !

I modified your script to pull out each individual parameter as such:

Names Default To Here(1);

dt = Open("$SAMPLE_DATA/Big Class.jmp");

new_cols = dt << New Formula Column(
	Operation(Category("Distributional"), "Johnson Normalizing"),
	Columns(:weight)
);

new_col = Remove From(new_cols, 1)[1];
f_expr = Column(dt, new_col << get name) << Get Formula;


test1=contains(char(arg(f_expr))," - (",1);
test2=contains(char(arg(f_expr)),"))",1);

c1=num(substr(char(arg(f_expr)),num(test1)+4,num(test2)-num(test1)-4));

test3=contains(char(arg(f_expr))," / ",1);
test4=contains(char(arg(f_expr)),") *",1);


c2=num(substr(char(arg(f_expr)),num(test3)+4,num(test4)-num(test3)-4));

test5=contains(char(arg(f_expr))," * ",1);
test6=contains(char(arg(f_expr)),"+",1);


c3=num(substr(char(arg(f_expr)),num(test5)+4,num(test6)-num(test5)-4));

c4=Arg(f_expr, 2);

Show(Arg(f_expr, 1), Arg(f_expr, 2));

Working great so far!

Steve

Re: Some way to get constants from the Johnson Transformation

shampton82 — Wed, 26 Mar 2025 16:21:03 GMT

Thats really cool @txnelson !

I'll play around with this some more but looks like I can use this to feed in a target from the transformed units (vs how the script just used the column mean) and it will output the value in original units (if I'm understanding this correctly).

Steve

Re: Some way to get constants from the Johnson Transformation

jthi — Wed, 26 Mar 2025 16:27:53 GMT

For something like this all the Expression functions can be very useful

N Arg(f_expr);
Head(f_expr);
Head Name(f_expr); 
Head(Arg(f_expr, 1));
Head(Arg(Arg(f_expr, 1), 1));

Sometimes you might want to add Extract Expr into the mix

Extract Expr(f_expr, (:weight - Wild()));

Re: Some way to get constants from the Johnson Transformation

txnelson — Wed, 26 Mar 2025 21:04:42 GMT

That is correct, or any other stat you get from the transformed distribution

Re: Some way to get constants from the Johnson Transformation

shampton82 — Thu, 27 Mar 2025 16:23:48 GMT

Hey @txnelson ,

I modified your script(try not to cringe to much lol) and made it so I could put in my spec limits and will get the transformed values of those limits, which is sweet. However, for my workflow I actually need to be able to put in the spec limits from the transformed data (so post running a control chart on the transformed data) and output them in the original data values. I follow most the code but not enough of it to feel confident trying to modify further myself so was wondering if this is a quick fix you could make in this code to basically reverse what it is currently doing?

Names Default To Here( 1 );

dt=current data table();

theColumn = dt<<get selected columns("string");

if(n items(theColumn)!=1,
	throw("Incorrect number of columns selected, must be just one");
	
	
);

		nw=new window("calculate control limits limits",
						Panel Box("actions",
							Hlistbox(
									bb1=Buttonbox("press to get Limits",
										nebvalue1=neb1<<get;
										nebvalue2=neb2<<get;

										obj=Distribution(
											Continuous Distribution(
												Column( as column(theColumn) ),
												Process Capability( Use Column Property Specs ),
												Fit Johnson(save distribution formula)
											),
											invisible(1),
											SendToReport(
												Dispatch( {eval(theColumn)}, "Process Capability", OutlineBox, {Close( 1 )} )
											)
										);

										obj<<close window;

										// Find the back transform of the calculated mean of the transformed distribution
										theTargetCol = Column( dt, N Cols( dt ) ) << get name;
										theFormula_lcl = Char( As Column( dt, theTargetCol ) << get formula );
										theTarget_lcl = nebvalue1;

										// Change the reference to the original column used in the transformation to _X_
										// so it can be used in the back transform
										Substitute Into( theFormula_lcl, ":"|| theColumn[1], "_X_" );
										// The formula needs to create a memory variable to allow for comparison.
										// Add the resulting variable name to be the resulting variable when the
										// formula is executed
										theFormula_lcl = "theResult_lcl = " || theFormula_lcl || ";";

										// Successive Approximation will be used to get the back transform
										// Use the min and max of the original column to start the approximations
										theMax = Col Max(as column(theColumn) );
										theMin = Col Min( as column(theColumn)  );

										// Loop through the approximations for up to 100 loops to find the value
										For( i = 1, i <= 1000, i++,
											_X_ = thetarget_lcl;
											Eval( Parse( theFormula_lcl ) );
											If(
												theResult_lcl > theTarget_lcl, theMax = _X_,
												theResult_lcl < theTarget_lcl, theMin = _X_,
												Break()
											);
										);

										// Display the findings in the log
										Show( theTarget_lcl, theResult_lcl);
										

										// Find the back transform of the calculated mean of the transformed distribution
										theTargetCol = Column( dt, N Cols( dt ) ) << get name;
										theFormula_ucl = Char( As Column( dt, theTargetCol ) << get formula );
										theTarget_ucl = nebvalue2;

										// Change the reference to the original column used in the transformation to _X_
										// so it can be used in the back transform
										Substitute Into( theFormula_ucl, ":"|| theColumn[1], "_X_" );
										// The formula needs to create a memory variable to allow for comparison.
										// Add the resulting variable name to be the resulting variable when the
										// formula is executed
										theFormula_ucl = "theResult_ucl = " || theFormula_ucl || ";";
										// Successive Approximation will be used to get the back transform
										// Use the min and max of the original column to start the approximations
										theMax = Col Max(as column(theColumn) );
										theMin = Col Min( as column(theColumn)  );

										// Loop through the approximations for up to 100 loops to find the value
										For( i = 1, i <= 1000, i++,
											_X_ = thetarget_ucl;
											Eval( Parse( theFormula_ucl ) );
											If(
												theResult_ucl > theTarget_ucl, theMax = _X_,
												theResult_ucl < theTarget_ucl, theMin = _X_,
												Break()
											);
										);

										// Display the findings in the log
										Show( theTarget_ucl, theResult_ucl);
										
										neb3<<set(theResult_lcl);
										neb4<<set(theResult_ucl);
									),
			
								Vlistbox(
									textbox("LCL"
										
									),
									textbox("UCL"
										
									)

									
								),
								Vlistbox(
									neb1=numbereditbox(
										
									),
									neb2=numbereditbox(
										
									),
								Vlistbox(
									neb3=numbereditbox(
										
									),
									neb4=numbereditbox(
										
									)
									
								)
							)
						)
					)
			
		);

Thanks for any input and/or help!

Steve

Re: Some way to get constants from the Johnson Transformation

txnelson — Fri, 28 Mar 2025 01:48:16 GMT

You have changed some of the statements that should not have been changed. I have commented out the inappropriate lines, and changed them back. I have also added in functionality that runs a control chart and strips off the Control Limits and uses them as the values to transform back into the original distribution.

Names Default To Here( 1 );

dt=current data table();

theColumn = dt<<get selected columns("string");

if(n items(theColumn)!=1,
	throw("Incorrect number of columns selected, must be just one");
	
	
);

		nw=new window("calculate control limits limits",
						Panel Box("actions",
							Hlistbox(
									bb1=Buttonbox("press to get Limits",
										nebvalue1=neb1<<get;
										nebvalue2=neb2<<get;

										obj=Distribution(
											Continuous Distribution(
												Column( as column(theColumn) ),
												Process Capability( Use Column Property Specs ),
												Fit Johnson(save distribution formula)
											),
											invisible(1),
											SendToReport(
												Dispatch( {eval(theColumn)}, "Process Capability", OutlineBox, {Close( 1 )} )
											)
										);

										obj<<close window;
										
// Run the Control Chart and strip off the LCL and UCL
ccb = /* Control Chart Builder(
	Variables( Y( column(ncols(dt)) ) ),
	Show Control Panel( 0 )
);*/
Control Chart Builder(
	Variables( Y( column(ncols(dt)) )  ) ,
	Set Subgroup Size( 5 ),
	Show Control Panel( 0 )
);

// Get the Target LCL and Target UCL from the Displayed Report
theTarget_lcl = (report(ccb)[NumberColBox("LCL")]<<get)[1];
theTarget_ucl = (report(ccb)[NumberColBox("UCL")]<<get)[1];
ccb << close window;

										// Find the back transform of the calculated mean of the transformed distribution
										theTargetCol = Column( dt, N Cols( dt ) ) << get name;
										theFormula_lcl = Char( As Column( dt, theTargetCol ) << get formula );
										//theTarget_lcl = nebvalue1;

										// Change the reference to the original column used in the transformation to _X_
										// so it can be used in the back transform
										Substitute Into( theFormula_lcl, ":"|| theColumn[1], "_X_" );
										// The formula needs to create a memory variable to allow for comparison.
										// Add the resulting variable name to be the resulting variable when the
										// formula is executed
										theFormula_lcl = "theResult_lcl = " || theFormula_lcl || ";";

										// Successive Approximation will be used to get the back transform
										// Use the min and max of the original column to start the approximations
										theMax = Col Max(as column(theColumn) );
										theMin = Col Min( as column(theColumn)  );

										// Loop through the approximations for up to 100 loops to find the value
										For( i = 1, i <= 5000, i++,
_X_ = Mean( theMax, theMin );
											//_X_ = thetarget_lcl;
											Eval( Parse( theFormula_lcl ) );
											If(
												theResult_lcl > theTarget_lcl, theMax = _X_,
												theResult_lcl < theTarget_lcl, theMin = _X_,
												Break()
											);
										);
neb1 << set( _X_ );
										// Display the findings in the log
										Show( i,_x_,theTarget_lcl, theResult_lcl);
										

										// Find the back transform of the calculated mean of the transformed distribution
										//theTargetCol = Column( dt, N Cols( dt ) ) << get name;
										theFormula_ucl = Char( As Column( dt, theTargetCol ) << get formula );
										//theTarget_ucl = nebvalue2;

										// Change the reference to the original column used in the transformation to _X_
										// so it can be used in the back transform
										Substitute Into( theFormula_ucl, ":"|| theColumn[1], "_X_" );
										// The formula needs to create a memory variable to allow for comparison.
										// Add the resulting variable name to be the resulting variable when the
										// formula is executed
										theFormula_ucl = "theResult_ucl = " || theFormula_ucl || ";";
										// Successive Approximation will be used to get the back transform
										// Use the min and max of the original column to start the approximations
										theMax = Col Max(as column(theColumn) );
										theMin = Col Min( as column(theColumn)  );
										// Loop through the approximations for up to 100 loops to find the value
										For( i = 1, i <= 15000, i++,
_X_ = Mean( theMax, theMin );
											//_X_ = thetarget_ucl;
											Eval( Parse( theFormula_ucl ) );
											If(
												theResult_ucl > theTarget_ucl, theMax = _X_,
												theResult_ucl < theTarget_ucl, theMin = _X_,
												Break()
											);
										);

										// Display the findings in the log
										Show( i,_x_,theTarget_ucl, theResult_ucl);
neb2 << set( _X_ );
										neb3<<set(theResult_lcl);
										neb4<<set(theResult_ucl);
									),
			
								Vlistbox(
									textbox("LCL"
										
									),
									textbox("UCL"
										
									)

									
								),
								Vlistbox(
									neb1=numbereditbox(
										
									),
									neb2=numbereditbox(
										
									),
								Vlistbox(
									neb3=numbereditbox(
										
									),
									neb4=numbereditbox(
										
									)
									
								)
							)
						)
					)
			
		);

Re: Some way to get constants from the Johnson Transformation

shampton82 — Fri, 28 Mar 2025 02:31:04 GMT

Thanks @txnelson !

Thanks for going above and beyond. This works great!

Steve