Scripters Club Recordings: Tips and Tricks

[#1]

Video 1 @thickey1 : Managing Datatables.  

• Modify Column Names.
• Alternative method of referencing column names.
• Add/Embed a script in the table. A user can then run it when ever they like.

(Files: Tips and Tricks and Probe)

Video 2 @Georg : General tips on how to write scripts and use JMP Script Editor

• code folding
• expression handling
• JSL in Formulas
• eval (eval expr(some code with expr(to be evaluated first)))
• test on type of variables
• associative array

(Files: 20241120_Scripters_Club_tipps_and_tricks-inspec)

Video 3 @Mauro_Gerber : Open to multiple import and Get path of a current script

(Files: SC-2024_ini and SC-2024_multi-import)

Video 4 @Daniel_Valente : JMP Marketplace overview

https://marketplace.jmp.com/

[#2]

Thanks @Georg for taking the time to show different approaches to calculate is in spec in JMP.
I feel very  honoured : )

I just tried a dynamic version of the approach with some locally defined lsl and usl values.
Please decide on your own - with the additional As constant() it is much faster, but the values don't get updated.
Without the As Constant() it's slower, but the user has the chance to update the column via

cols = Current Data Table(  ) << Get Selected Columns;
for each({col}, cols, col << eval formula())

View more...

Names Default To Here( 1 );

Ndoubled = 4; // more realistic: 10
dynamic = 1;



Try( Close( cdt, NoSave ) );
cdt = Open( "$SAMPLE_DATA/Semiconductor Capability.jmp" );
Wait( 0 );
bigger = Expr(
	cdt = cdt << concatenate( cdt, append to first table( 1 ) )
);
For( I = 1, i <= Ndoubled, i++,
	bigger
);
col_lst = cdt << get column group( "Processes" );

Wait( 0 );

cdt << New Table Variable( "cdt", "current data table();" );

cdt << begin data update();

t0 = hptime();

If( dynamic,
	col_lst = cdt << get column group( "Processes" );
	For Each( {col}, col_lst,
		col_Name = col << get name() || "_inSpecDynamic";
		Eval(
			Substitute(
					Expr(
						New Column( col_Name,
							Nominal,
							Formula(
									local({
									limits = As constant(_col_ << get property( "Spec Limits" )),
									usl = As Constant(limits["USL"]),
									lsl = As Constant(limits["LSL"])},
									_col_ > lsl & _col_ < usl)
								
							)

						)
					),
				Expr( _col_ ), Name Expr( col )
			)
		);
		cdt << move selected columns( {Column( col_name )}, after( col ) );
	);
, 

	For Each( {col}, col_lst,
		sl_lst = col << get column properties( "Spec Limits" );
		sl_AA = [=> .];
		For Each( {limit}, Arg( sl_lst[1], 2 ),
			Match( Head( limit ), Expr( USL ), sl_AA["USL"] = Arg( limit ), Expr( LSL ), sl_AA["LSL"] = Arg( limit ) )
		);
		col_name = Concat( col << get name(), "_inspec" );
		Eval(
			Eval Expr(
				cdt << New Column( col_name,
					"Ordinal",
					formula(
						If(
							And(
								As Column( Expr( col << get name() ) ) < Expr( sl_AA["USL"] ),
								As Column( Expr( col << get name() ) ) > Expr( sl_AA["LSL"] )
							),
							1,
							0
						)
					)
				)
			)
		);
		cdt << move selected columns( {Column( col_name )}, after( col ) );
	)
);

cdt << run formulas() << end data update();

Show((hptime()-t0)/1000000);


// TEST THE RESULT
// check result in table visually
t0 = hptime();
ps = cdt << Process Screening( Process Variables( Eval( col_lst ) ), Spec Limits Dialog( "No (skip columns with no spec limits)" ) );
ps << Color Out Of Spec Values( 1 );
ps << Close Window;
cdt << Clear Select << Clear Column Selection;
Print("\!NVia process screening: ");
Show((hptime()-t0)/1000000);

I did not dare to set Ndoubled to 10 : )
My dream: a fast, dynamic and easy to use 🙏 is in spec (value) 

Along the idea of "roast our code" - any ideas to improve the speed?
And how to make this approach accessible to other users with less JSL experience?

[#3]

this one is faster - but doesn't trigger an update when specs get adjusted.

So at the moment: no real benefit compared to 

for each (... dt[0,colname] =  ...)

 or  new column(set values())  

Names Default To Here( 1 );

Ndoubled = 10; 
dynamic = 1;

Try( Close( cdt, NoSave ) );
cdt = Open( "$SAMPLE_DATA/Semiconductor Capability.jmp" );
Wait( 0 );
bigger = Expr(
	cdt = cdt << concatenate( cdt, append to first table( 1 ) )
);
For( I = 1, i <= Ndoubled, i++, bigger );
col_lst = cdt << get column group( "Processes" );

Wait( 0 );

cdt << New Table Variable( "trigger update", 1 ); // can this be used to force an update?

cdt << begin data update();

t0 = HP Time();

If( dynamic,
	col_lst = cdt << get column group( "Processes" );
	For Each( {col}, col_lst,
		col_Name = col << get name() || "_inSpecDynamic";
		colnew = New Column( col_Name, Nominal );
		col_Name = colnew << get name();
		Eval(
			Substitute(
					Expr(
						Column( col_Name ) << set Formula(
							Local(
								{t0 = As Constant(
									
									dt = Current Data Table();
									limits = _col_ << get property( "Spec Limits" );
									nr = nrows(dt);
									usl = limits["USL"];
									lsl = limits["LSL"];
									myRows = Where( _col_ > lsl & _col_ < usl );
									
									NmyRows = N Items( myRows );
									dt[0, _col_Name_] = J( nr, 1, 0 );
									If( NmyRows,
										dt[myRows, _col_Name_] = J( NmyRows, 1, 1 )
									);
									1
								)},
								Stop();
							)
						)
					),
				Expr( _col_ ), Name Expr( col ), Expr(_col_name_), col_name
			)
		);
		cdt << move selected columns( {Column( col_name )}, after( col ) );
	);
	 

	
);

cdt << run formulas() << end data update();

Show( (HP Time() - t0) / 1000000 );

Wait( 0 );

Caption( "Process screening - don't dare  ..." );

Stop();
// TEST THE RESULT
// check result in table visually
ps = cdt << Process Screening( Process Variables( Eval( col_lst ) ), Spec Limits Dialog( "No (skip columns with no spec limits)" ) );
ps << Color Out Of Spec Values( 1 );
ps << Close Window;
cdt << Clear Select << Clear Column Selection;

[#4]

Der @hogi , thanks for looking into the code and the additional implementation. For the presentation I did not look too much into speed etc., my focus was to show different ways and approaches. In real life the best implementation depends on your boundary conditions, so it is case specific, as I think.

If you look at the program flow, it may be not ideal to pull out specification limits in each row using the formula approach, but in some cases it may be useful. However JMP may internally improve the program at runtime avoiding unneccessary calculations. It is impossible for me to know all these details, but that's why we need to try different approaches. 

Generally when speed comes into place, I would use the platform options, e.g. via colors in process screening, because I assume, that they are optimized for large data.

Maybe we can look into it again during roasting series. And yes again, I think that in_spec would be a very nice built in function. Only the implementation may become difficult when caring for all the details - hope the developers find a good way.

[#5]

To compare the performance with that of the official platform, please delete or comment out the stop().

[#6]

One thing to remember with spec limits are one sided specs. They can cause all sorts of fun issues

[#7]

yes, right. 
For sake of simplicity I thinks it's OK to use tw-sided specs.
Accounting for on-sided specs (or columns without any specs) is quite straight-forward - but indeed, one should not forget.

Do you have a trick how to speed up the calculation - and to make it dynamic?

[#8]

Most likely it would depend a lot on the problem which is being solved (so solution depends a lot on use case).

[#9]

use case: 
a new column with a formula to generate 0/1 values for out-of-spec/in-spec values.

If there is a typo in the specs, or the user wants to adjust the spec, the values of the column should automatically update (should not stay stuck with the old settings).

maybe for 10-30 columns.

100k - 10Mio rows.

A solution which can easily be shared with new colleagues who want to use JMP to analyze pass/fail information.

[#10]

This might go a bit off-topic:

You are missing few specifications(?) (I consider myself professional XY Problem solver and I don't really thing you want be one):

• Input: just a data table
• Use ALL columns which have spec limits set?
  • But keep in mind that these might be set incorrectly (much bigger problem in my opinion)
    • How incorrectly? Can we have missing values on both? Same values on both even if we have continuous measurement? Or just "buggy" values (you used to be able to have "non-evaluated" values as spec limits in JMP which still did show as the correct numeric values. Not sure if this is still the case with JMP18).
  • Separate case is that user might want to adjust these (I would consider this different use case than single column with 0/1)
• What is the use case? I'm pretty sure that users won't be analyzing 30 columns and 10million cells visually from a table
  • How are they using the results? Could the calculation be implemented there? -> Have a custom platform for this type of work?

If I were to "solve a problem" in this case, I would first start by solving the incorrect spec limits, or maybe rather the situation where they are changed. What are you really trying to see? I really doubt it is to have a column with 0/1 based on spec limits when you have 10 million rows of data.

You could (or I could) create a functionality to create such a column (not sure how fast you could make it) but is that really the solution to the problem? If you just want to have 0/1 column based on current spec limits for ALL columns, I don't think it has to be dynamic (update immediately). User can press press single button in menu to update results (I'm thinking add-in which uses specific column names or maybe table variables). If the purpose is to adjust specs, there is no value (in my opinion) seeing 0 changing to 1 when you have 10 rows (create special platform for JUST that use case, if nothing else and you "only" care about speed, it will most likely be faster).

Or is this just a theoretical question how to make this fast?

[#11]

Hi Jarmo, thank you for your comments.

You are right, with the help of a single  "click here to update" all the problems vanish.
But, to be honest, I love the "Graph Builder - just set it up and JMP will handle the details" approach.

No need to discuss/implement the details via the community (it's clear how to extend a formula such that it also works for 1-sided specs). Here I even prefer a short code which shows the concept.

Besides that, no need to doubt if a fast, dynamic and user friendly 🙏 is in spec (value) will be used for something useful.

So, till 🙏 is in spec (value) is available -- If somebody has a better solution, how to set it up manually, I will be extremely happy and use it right away. It's not a theoretical question