Scripting table scripts containing reference lines from a different dt

[#1]

Hello,

I have a task to script some visualizations as table scripts.  Normally this is simple. The rough part is the reference lines that need to reference other sources when the script is running.  Due to the references being contained in two functions that don’t normally accept variables from the outside, finding out how to use expression manipulations to insert these values is something that is currently beyond my skill level. There are multiple obstacles to overcome.

• Functions that don’t accept outside variables well
• var1 = 1;
  var2 = 2;
  Dt << New script( “example”, print( var1 + var2 ) );
• Note that the variables are defined out of the table script. This is a requirement, as the numbers will be referenced from a data table.
• Dispatch()
• For values inside of this, they need to be static, plainly stated values
• Working with expressions that do not contain functions
• example = expr( New Script() );
  title  = “test”;
  insert into( example, title );  // show(example) = New script( “test” )
  
  // Later on when inserting cols into the x axis, as on line 22 of the example…
  x_empty = expr( X() );
  x_cols = { :col1, :col2 };
  insert into( x_empty, x_cols[1] ); // errors out, JMP doesn’t know what X() is
• This tells me that somehow X() needs to be unevaluated, but still allow for item inserts, which is what the likes of name expr() and expr() are supposed to do, but they don’t work in this case

The example script contains a simplified example of what I want to accomplish. The titles and numbers in the newly generated table are to be inserted into the table script, before the function is run to be inserted into Big Class. I have a basic example of this at the bottom of the script.

Names default to here(1);
clear log();

// Reference table
// Used to populate data in table script below for dt( big )
dt = New Table( "Example",
	Add Rows(2),
	New Column( "Growth Label", Character, Nominal, Set Values( { "Tall", "Short" } ) ),
	New Column( "Height tag", Numeric, Continuous, Set Values( { 67.5, 60 } ) ),
);

// Example table
big = open( "$SAMPLE_DATA/Big Class.jmp" );

// Plain example - vis added as table script * works *
big << New Script( "Example",
	Graph Builder(
		Size( 534, 456 ),
		Show Control Panel( 0 ),
		Fit to Window,
		Set α Level( 0.05 ),
		Variables( X( :weight ), X( :age ), Y( :height ), Color( :sex ) ),
		Elements( Points( X, Y, Legend( 4 ) ) ),
		SendToReport(
			Dispatch(
				{},
				"height",
				ScaleBox,
				{Add Ref Line( 67.5, "Dashed", "Medium Dark Green", "Tall", 1 ),
				Add Ref Line( 60, "Dashed", "Medium Dark Red", "Short", 1 )}
			)
		)
	)
);

// Needed info for below
vis_x = { :weight, :age };
vis_y = { :height };

// Modified example - vis added as table script * not working *
big << New Script( "Example",
	Graph Builder(
		Size( 534, 456 ),
		Show Control Panel( 0 ),
		Fit to Window,
		Set α Level( 0.05 ),
		Variables( X( vis_x[1] ), X( vis_x[1] ), Y( vis_y[1] ), Color( :sex ) ),
		Elements( Points( X, Y, Legend( 4 ) ) ),
		SendToReport(
			Dispatch(
				{},
				"height",
				ScaleBox,
				{Add Ref Line( big:Height tag[1], "Dashed", "Medium Dark Green", big:Growth Label[1], 1 ),
				Add Ref Line( big:Height tag[2], "Dashed", "Medium Dark Red", big:Growth Label[2], 1 )}
			)
		)
	)
);

// Example: expression manipulations
var1 = 10; // represents some reference number, from somewhere
var2 = 1; // represents some reference number, from somewhere

expr_test = eval expr( expr(var1) + expr(var2) );
expr_begin = expr( New Script( "Var replacement" ) );
insert into( expr_begin, name expr( expr_test ) );
show( expr_begin );

eval( eval expr( big << expr( name expr( expr_begin ) ) ) ); // Script contains: 10 + 1
/* Notice how the table script "Var replacement" in big doesn't contain var1 & var2,
it contains the values they represent. This is what I want to do. */

close( dt, "No Save" );

Searching for "reference lines" yields quite a few results in the community forums. The closest example I have found, so far, are the links below.
https://community.jmp.com/t5/Discussions/Is-there-a-way-to-automatically-draw-ref-lines-at-the-peak-...

https://community.jmp.com/t5/Discussions/Fill-color-for-multiple-reference-lines/m-p/273205 

[#2]

Yesterday I found the following thread, which caused me to investigate parse(). 
https://community.jmp.com/t5/Discussions/evaluating-a-string-using-a-column-formula/m-p/38028 

The thought process is as follows:

• When using expr(), JMP expects functions as an argument, not other arguments. More on this later.
• If arguments can be saved to a variable as a string, and then placed into either another argument, or function, this will help code what's needed

Here's an example.

names default to here(1);
clear log();
clear symbols();

dt = open( "$SAMPLE_DATA/Big Class.jmp" );

dt << Graph Builder(
	Size( 517, 447 ),
	Show Control Panel( 0 ),
	Fit to Window,
	Set α Level( 0.05 ),
	Variables( X( :weight ), Y( :height ), Color( :sex ) ),
	Elements( Points( X, Y, Legend( 3 ) ) ),
	SendToReport(
		Dispatch(
			{},
			"height",
			ScaleBox,
			{Add Ref Line( 67.5, "Solid", "Black", "Tall", 1 )}
		)
	)
);

 The script opens Big Class.jmp and then generates a visualization with a reference line. When I want to pull data from other sources for the visualization, I could modify the script above to the following.

Names default to here(1);
clear log();

dt = open( "$SAMPLE_DATA/Big Class.jmp" );

// Example "other" source
ex = New Table( "Example",
	Add Rows(2),
	New Column( "Growth Label", Character, Nominal, Set Values( { "Tall", "Short" } ) ),
	New Column( "Height tag", Numeric, Continuous, Set Values( { 67.5, 60 } ) ),
);

dt << Graph Builder(
	Size( 517, 447 ),
	Show Control Panel( 0 ),
	Fit to Window,
	Set α Level( 0.05 ),
	Variables( X( :weight ), Y( :height ), Color( :sex ) ),
	Elements( Points( X, Y, Legend( 3 ) ) ),
	SendToReport(
		Dispatch(
			{},
			"height",
			ScaleBox,
			{Add Ref Line( ex:Height tag[1], "Solid", "Black", ex:Growth Label[1], 1 )} // 2x references to data table( "Example" )  as "ex"
		)
	)
);

The difference in the above script is that some of the reference line information is now pulling from the example table that was created.  What if I wanted to add the visualization in the script above as a data table script?  Assuming that we don't want to always be referencing another table for the values, how can we replace the references with the values they represent?  The answer is expression manipulations and, seemingly, parse().

To read more about expression manipulations, read this resource: https://www.jmp.com/support/help/en/17.0/#page/jmp/advanced-expressions-macros-and-lists.shtml# 

If we were to add the visualization as a table script to Big Class, our script window would have code like this.

// Visualization - No references to other data sources
dt << New script( "Vis 1",
Graph Builder(
	Size( 517, 447 ),
	Show Control Panel( 0 ),
	Fit to Window,
	Set α Level( 0.05 ),
	Variables( X( :weight ), Y( :height ), Color( :sex ) ),
	Elements( Points( X, Y, Legend( 3 ) ) ),
	SendToReport(
		Dispatch(
			{},
			"height",
			ScaleBox,
			{Add Ref Line( 67.5, "Solid", "Black", "Tall", 1 )}
		)
	)
);
);

// Visualization - With references to other data sources
// This will not work, as the data table reference "ex" is undefined
// We could replace ex with data table( "example" ), but that table would need to be open in the background 
dt << New script( "Vis 2",
Graph Builder(
	Size( 517, 447 ),
	Show Control Panel( 0 ),
	Fit to Window,
	Set α Level( 0.05 ),
	Variables( X( :weight ), Y( :height ), Color( :sex ) ),
	Elements( Points( X, Y, Legend( 3 ) ) ),
	SendToReport(
		Dispatch(
			{},
			"height",
			ScaleBox,
			{Add Ref Line( ex:Height tag[1], "Solid", "Black", ex:Growth Label[1], 1 )}
		)
	)
);
);

This is the fruit of my investigation and experimentation so far, with a focus on parse().  It doesn't work yet, but it's a step in the right direction!

names default to here(1);
clear log();
clear symbols();

// Opening & creating data tables for use
dt = open( "$SAMPLE_DATA/Big Class.jmp" );

ex  = New Table( "Example",
	Add Rows(2),
	New Column( "Growth Label", Character, Nominal, Set Values( { "Tall", "Short" } ) ),
	New Column( "Height tag", Numeric, Continuous, Set Values( { 67.5, 60 } ) ),
);

// Variables
graphing = expr( Graph Builder() );
size = "Size( 517, 447 )";
control_panel = "Show Control Panel( 0 )";
fit = "Fit to Window";
a_lvl = "Set α Level( 0.05 )";
vars = "Variables( X( :weight ), Y( :height ), Color( :sex ) )";
eles = "Elements( Points( X, Y, Legend( 3 ) ) )";
str = "SendToReport()";
disp = parse( "Dispatch()" );
curly = "{}";
label = "ex:Growth label[1]";
label2 = "height";
scle = "ScaleBox";
x_axis = "x()";
test_label = "ex:Height tag[1]";
mylabel = "ex:Growth label[1]";
ref_line = parse( "Add Ref Line()" );

insert into( graphing, parse( size ) );
insert into( graphing, parse( control_panel ) );
insert into( graphing, parse( fit ) );
insert into( graphing, parse( a_lvl ) );
insert into( graphing, parse( vars ) );
insert into( graphing, parse( eles ) );
show( graphing ); // Result:graphing = Graph Builder(Size(517, 447), Show Control Panel(0), Fit to Window, Set α Level(0.05), Variables(X(:weight), Y(:height), Color(:sex)), Elements(Points(X, Y, Legend(3))));

// add ref line
insert into( ref_line, eval( parse( test_label ) ) );
insert into( ref_line, "Solid" );
insert into( ref_line, "Black" );
insert into( ref_line, eval( parse( mylabel ) ) );
show( ref_line ); // Result: ref_line = Add Ref Line(67.5, "Solid", "Black", "Tall");

insert into( disp, parse( curley ) );
insert into( disp, eval( parse( mylabel ) ) );
insert into( disp, char( label2 ) );
insert into( disp, parse( scle ) );
insert into( disp, parse( curly ) ); // Note this line! More work needs to be done here
show( disp ); // Result: disp = Dispatch({}, "Tall", "height", ScaleBox, {});

The script above is very busy, and after figuring out the rest of this, I'll look for ways to optimize by shrinking the number of variables necessary, as this is quite a list.

Near the bottom of the script, more information on the note: ref_line needs to be inserted into curly BEFORE curly is inserted into disp. It would be easy to think that you could do this:

insert( curley, ref_line );

But, because the contents of ref_line is contained within an argument, not a function, JMP will error out.   Once a way is figured out around this, everything else should be easy.  My next guess is to try something like this:

// Note: char( ref_line ) will cause JMP to error, because it's an argument
new_parse = expr( parse() );
insert into( new_parse, char( ref_line) );

// Then somehow
eval( insert into( curley, eval expr( eval( new_parse ) ) ) );

After learning how to insert a non-character type argument into another argument, the rest of the Graph Builder() function will be simple to build. The final piece will be to insert everything as an expression, into dt << New Script( "Visualization", <expression here> );, and then use eval expr() and another eval() (?) to make JMP successfully execute the code and insert the visualization into dt.

[#3]

Here's something interesting I found.  When trying to insert an argument (non-character) into another argument (character), try() helped the non-character become a string.  Yet when I tested this a bit more, the results couldn't be replicated.  Also, those curly braces are quite difficult to insert anything into.

Names default to here(1);
clear log();

// This block demonstrates, at a basic level, what I'm trying to do
printing = Expr( print() );
hello = "hello world";
insert into( printing, hello );
eval( printing );
show( printing ); // Result = printing = Print("hello world");

// Test run 1 - Works
new_parse1 = Parse( "Variable()" );
mytest1 = "X(:height)";
insert into( new_parse1, parse( mytest1 ) );
show( new_parse1, type( mytest1 ) ); // Result: new_parse = Variable(X(:height)); Type(mytest) = "String";

// Test run 2 - Doesn't work
new_parse2 = Parse( "Variable()" );
mytest2 = X(:height); // "Name unresolved"" error! X is not recognized
insert into( new_parse2, parse( mytest2 ) );
show( new_parse2, type( mytest2 ) ); // Intended: new_parse = Variable(X(:height)); Type(mytest) = "String";

// Test run 3 - Works
new_parse3 = Parse( "Variable()" );
try( mytest3 = X(:height) );
insert into( new_parse3, parse( mytest3 ) );
show( new_parse3, type( mytest3 ) ); // Result: new_parse = Variable(X(:height)); Type(mytest) = "String";

// Test run 4 - Doesn't work
curly = "{}";
new_parse4 = Parse( "Variable()" );
x_test = parse( "X()" );
some_list = { "one", "two", "three" };
insert into( x_test, eval( some_list[1] ) );
show( x_test ); // Result: x_test = X("one");
insert into( new_parse4, x_test ); // Containing within try() avoids error, but x_test not added into new_parse3
show( new_parse3 ); // Result: new_parse3 = Variable(); <-- Intended: new_parse3 = Variable(X("one"));
insert into( curly, new_parse4 );
show( curly ); // Intended: new_parse3 = {Variable(X("one"))};

// Other attempts - Doesn't work
variable1 = Variable(X("one"));
variable2 = eval insert( "{^variable1^}" );

variable3 = "{" || char( variable1 ) || "}";

[#4 · Accepted solution]

You've got a lot of extra complication I believe.  I believe that something like this is what you're wainting:

Names default to here(1);
clear log();

// Reference table
// Used to populate data in table script below for dt( big )
dt = New Table( "Example",
	Add Rows(2),
	New Column( "Growth Label", Character, Set Values( { "Tall", "Short", "Thin" } ) ),
	New Column( "Height tag", Numeric, Set Values( { 67.5, 60, 56 } ) ),
	New Column( "color", Character, Set Values( { "medium dark green", "medium dark red", "purple" } ) )
);

// Example table
big = open( "$SAMPLE_DATA/Big Class.jmp" );

big << New Column( "Random", Numeric, <<Set Each Value( Random Integer( 10, 100 ) ) );

x cols = {"weight", "age", "Random"};
y cols = {"height"};

x col expr = {};
y col expr = {};

For Each( {x, i}, x cols,
	show( x, i );
	x col expr[i] = Eval Expr( X( Expr( As Name( x ) ) ) );
);
For Each( {y, i}, y cols,
	y col expr[i] = Eval Expr( Y( Expr( As Name( y ) ) ) );
);

variables = Insert( x col expr, y col expr );
Insert Into( variables, Expr( Color( :sex ) ) );
variables = Substitute( variables, {}, Expr( Variables() ) );

reference lines = {};
For Each Row( dt,
	Insert Into( reference lines, Substitute( Eval List( {:height tag, "Dashed", :color, :growth label, 1} ), {}, Expr( Add Ref Line() ) ) );
);

Eval( Eval Expr(
big << New Script( "Example",
	Graph Builder(
		Size( 534, 456 ),
		Show Control Panel( 0 ),
		Fit to Window,
		Set α Level( 0.05 ),
		Expr( Name Expr( variables ) ),
		Elements( Points( X, Y, Legend( 4 ) ) ),
		SendToReport(
			Dispatch(
				{},
				"height",
				ScaleBox,
				Expr( reference lines )
			)
		)
	)
);
) )

[#5]

@ErraticAttack THANK YOU! This is so much easier to understand and compact!