Tweak a model before running fit model, list box not respecting special column names

[#1]

I’m looking to build a copy of this model effects dialog from fit model:

Goal is to build a script that will read in this effects expression from the model script that DOE saves in the table, allow people to tweak the effect expression and then run some analysis using fit model with the new effects expression.

If you run the dialog in below script, you will see that it reads in the effects expression from the model script stored in the table. Puts that in the list box that resembles Fit Model, allows you to tweak it and then when you hit ok it attempts to parse the listbox into a new effects expression and start a custom fit model dialog with that new expression.

The problem/difference is in the special name column: in the original expression it is stored as :name(“Height-”) but when retrieving that expression from the list box, it is stored as “Height-” which becomes a incomplete mathematical operation instead of a column name. So how do I apply the special column name rules to items from this list box? The same applies to crossed and nested terms with the special name column.

dt=Open("$SAMPLE_DATA/Big Class.jmp");
dt:height << Set Name( "Height-" );

FitModelScript = dt << Get table variable( "Fit Model" );

i = 1;
try(
	While( !Is Empty( arg_i = Arg( parse(FitModelScript), i ) ),
		If( Starts With( Char( Name Expr( arg_i ) ), "Effects" ),
			EffectsExpr = Name Expr( arg_i );
			Break();
		,
			i++
		)
	);
);
New Window( "Test",
	Border Box( Left( 3 ), top( 2 ),
		v list box(
			H List Box(
				Panel Box( "Select Columns",
					v list box(
						colListData = Col list box( All, width( 253 ), grouped, nLines( 20 ) ),
						Spacer box(size(1,2))
					)	
				),
				Panel Box( "Cast Selected Columns into Roles",
					Lineup Box( N Col( 2 ), Spacing( 3 ),
						Button Box( "Response columns", colListYbox << Append( colListData << GetSelected ) ),
						colListYbox = Col List Box( width( 200 ), nLines( 20 ), numeric, MinItems( 1 ) ),
					)
				),
				Panel Box( "Model effects",
					H List Box(
						V List Box(
							Button Box( "Main", LB << Append( colListData << GetSelected ) ),
							Button Box( "Interact",
								SelectedColumns = colListData << GetSelected();
								For( i = 1, i <= N Items( SelectedColumns ), i++,
									For( k = 1, k <= N Items( SelectedColumns ) - i, k++,
										LB << Append( concat(SelectedColumns[i], " * ", SelectedColumns[i + k] ) )
									)
								);
							),
							Button Box( "Quadratic",
								SelectedColumns = colListData << GetSelected();
								For( i = 1, i <= N Items( SelectedColumns ), i++,
									LB << Append( concat(SelectedColumns[i], " * ", SelectedColumns[i] ) )
								);
							),
							Button Box( "Nest", 
								colListX = colListData << GetSelected();
								For( i = 1, i <= N Items( colListX ), i++,
									If(Column(colListX[i]) << Get Modeling Type() != "Nominal",
										New window("Error",<<modal,Text box("Nested factors need to be column type Nominal. Colum: " || colListX[i] ));
										Remove from(colListX, i);
									);
								);
								MainNest = LB << get selected();
								For( i = 1, i <= N Items( MainNest ), i++,
									If(Column(MainNest[i]) << Get Modeling Type() != "Nominal",
										New window("Error",<<modal,Text box("Nested factors need to be column type Nominal. Colum: " || MainNest[i] ));
										Remove from(MainNest, i);
									);
								);
								If(N items(colListX)>0,
									For( i = 1, i <= N Items( MainNest ), i++,
										If(Contains(colListX, MainNest[i])==0,
											LB << Append(concat(MainNest[i], "[", concat items(colListX, ", "), "]"));
											LB << Remove Item(contains(LB << Get items( ),MainNest[i]));
										,
											New window("Error", <<modal, Text box("Cannot nest column into itself, select different columns only. Column: " || MainNest[i]))
										);
									);
								);
							),
							Button Box( "Remove", LB << Remove Selected( ) ),
							Button Box("Reset",
								LB << Remove All();
								try(For( i = 1, i <= N Arg( EffectsExpr ), i++,
									LB << Append( Char( Arg( parse(substitute(char(eval expr(EffectsExpr)), expr(":"),expr(""))), i ) ) )
								))
							),
						),
						LB = List Box( {}, nlines( 23 ) ),
						Try(For( i = 1, i <= N Arg( EffectsExpr ), i++,
							LB << Append( Char( Arg( parse(substitute(char(eval expr(EffectsExpr)), expr(":"),expr(""))), i ) ) )
						))
					)
				),
				Panel Box( "Actions",
					Lineup Box( N Col( 1 ),
						Button Box( "OK",
							ResponseColumns = colListYbox << get items();
							newEffectsExpr = Concat Items(LB << GetItems, ", ");
							jsl = evalinsert("\[
								try(FitModelDialogWindow << closewindow());
								FitModelDialogWindow = Fit Model(
									invisible,
									Y( ^ResponseColumns^ ),
									Effects( ^newEffectsExpr^ ),
									Keep dialog open( 1 ),
									Personality( "Standard Least Squares" ),
									Emphasis( "Effect Leverage" )
								);
							]\");
							eval(parse(jsl));
						)
					)
				),
			)
		)
	)
)

[#2]

One way to do this is by building up expressions. A major benefit of this approach is that, unlike with string-based approaches, the code remains colored, easily read, and requires minimal escape characters. The flow in this particular case is to:

1. Define and build up the response expression,
2. Define and build up the effects expression,
3. Define a Fit Model expression with placeholders,
4. Substitute the response and effect expressions into the placeholders in the Fit Model expression, and
5. Execute the Fit Model expression.

Note: I've commented out the "invisible" option, and wrapped the column name change in a try() to allow repeated runs.

To give this a go, replace your buttonbox("OK" ... ) code with the code below. A column must be entered into the Response column's list box before clicking "OK".

Cheers,

Brady

						Button Box( "OK",

							respList = colListYbox << get items();
							respExpr = expr(Y());
							for(i=1, i<=nitems(respList), i++,
									insertinto(respExpr, parse(evalinsert(":name(\!"^respList[i]^\!")")))
							);

							effectsList = lb << get Items;						
							effectsExpr = expr(Effects());
							for(i=1, i<=nitems(effectsList), i++,
									insertinto(effectsExpr, parse(evalinsert(":name(\!"^effectsList[i]^\!")")))
							);
							
							jslExpr = expr(
								try(FitModelDialogWindow << closewindow());
								FitModelDialogWindow = Fit Model(
									//invisible,
									_Y_,
									_EFFECTS_,
									Keep dialog open( 1 ),
									Personality( "Standard Least Squares" ),
									Emphasis( "Effect Leverage" )
								);
							);
							substituteInto(jslExpr, 
								expr(_Y_), nameexpr(respExpr),
								expr(_EFFECTS_), nameexpr(effectsExpr)
							);
							jslExpr;
						)

[#3]

Thanks Brady, I will look into this. I'm mostly wondering how it will handle crossed, interacting and nested terms but will need to play with that a bit to check.

I also need to see if I can do something similair when importing an existing model with these terms into the list box as this shows the "Name(" stuff currently. I will report backs once I've had some time to dig in.

Thinking about it, it might actually be better to call/re-use the Construct Model Effects element from the Fit Model dialog by sending the user out to interact with that window and then read back in the effects expression from that window. Not as pretty though.

[#4]

@brady_brady I almost have it solved but there is still one problem that I'm running into: If column names contain the same characters that are used for crossing or nesting terms, how do you split them out into columns when comming back from a listbox?

dt = New Table( "Test",
	Add Rows( 3 ),
	Compress File When Saved( 1 ),
	New Column( "SCR[12]*34",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Set Values( [1, 2, 3] )
	),
	New Column( "SCR5[a]*",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Set Values( [3, 2, 3] )
	),
	New Column( "Y",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Set Values( [4, 1, 4] )
	)
);

new window("test", LB = list box());
LB << append((Column(1) << get name()) || " * " || (Column(2) << get name()));
LB << append((Column(1) << get name()) || "[" || (Column(2) << get name()) || "]");
LB << get items();

You'll notice that the items comming from the list box are just strings. If I use words() with splitter "*" or "[]" I get too many splits and the results are not column reference anymore. I need to split each item into its constituents so that I can wrap them in the :Name(" ") and re-nest/cross them and add them to the effects expression for fit model. I also need a list of all columns involved in any of the terms for another part of the script.

If you know of a solution, please consider that the fit model dialog allows nesting and crossing with more than 2 variables.

Going the other way, parsing the effects expression from the model script I could make use of Arg() function to extract columns and populate the list box, but going back this doesn't work as it is string type.

Also: is there a words() equivalent to extract all arguments of an expression type? Now I end up having to write loops to extract all arguments from an expression.

[#5]

Hi,

I'm wondering if it would be possible to add ":Name(ColumnName)" around each column name and if that would solve the problem.  One way to do this might be using Regex().  I've written something that I think may be a step in this direction but it's still not perfect.  Maybe someone with more knowledge of Regex could help with this?:

colnames=dt<<Get column Names(string);
lbstring=LB << get items();

for(i=1,i<=nitems(lbstring),i++,
for(ii=1,ii<=nitems(colnames),ii++,
lbstring[i]=regex(lbstring[i],colnames[ii],evalinsert(":name(\!"^colnames[ii]^\!")"),GLOBALREPLACE);
));

[#6]

If column names contain the same characters that are used for crossing or nesting terms...

The simplest way is to prevent this from occurring. If your script is creating such columns, consider, for example, using \{ \} in place of [ ] and \# in place of *. If on the other hand you are trying to support any possible column name, you could cycle through the column names, making substitutions like those mentioned above, after showing a modal warning window informing the user that the column names must be temporarily changed. You could then change them back when the script is finished.

Also: is there a words() equivalent to extract all arguments of an expression type? Now I end up having to write loops to extract all arguments from an expression.

I don't know of an analogous function to words for expression arguments. If you're building such lists that often, maybe a function would help? Note that you have to wrap the expression in nameexpr() as you pass it to the function.

Cheers,

Brady

exp = Expr( note( a, b, c ) );

argList = function({ex},
	{aList = {}, i=1},
	While( i <= N Arg( ex ), Insert Into( aList, Arg( ex, i++ ) ) );
	aList
);

argList(nameexpr(exp));

[#7]

Thanks!

Yes i'm trying to support every column name because I don't create them. Renaming is the only solution that I found that truly works but i don't like changing people's data tables even temporarily. Instead I'm going to see if I can get there by embedding Fit model directly in the new window dialog and let that handle the creation of the effects expression. I loose some functionality that way (recall, and other column boxes being fillable) but it is worth it to have a robust effects expression generator.