Dynamically created formula not evaluating

[#1]

Came across a problem when I tried to dynamically create formula columns. I did manage to solve it in the end, but I would like to know what is happening here. My guess is that it is somehow related to how I'm building the formula and that the "formula" gets evaluated before it is set to column and then it somehow messes up the column formula.

Here is an example: columns created by test1 and test2 do have issues and test3 works fine (test4 and test5 also seem to work fine, but I'm not sure I would feel confident in using them as I have no idea why they do get converted into column references from strings)

Names Default To Here(1);

test1 = function({dt, val_col, by_cols}, {Default Local},
	ref_list = Transform Each({col_name}, Insert(by_cols, val_col, 1),
		Name Expr(As Column(dt, col_name));
	);
	
	new_col = Eval(Substitute(
		Expr(dt << New Column("Formula", Numeric, Continuous,
			Formula(_new_formula_)
		)),
		Expr(_new_formula_), Substitute(ref_list, Expr(list), Expr(Col Median))
	));
	return(new_col);
);

test2 = function({dt, val_col, by_cols}, {Default Local},
	ref_list = Transform Each({col_name}, Insert(by_cols, val_col, 1),
		Name Expr(As Column(dt, col_name));
	);
	new_formula = Substitute(ref_list, Expr(list), Expr(Col Median));		
	new_col = Eval(Substitute(
		Expr(dt << New Column("Formula", Numeric, Continuous,
			Formula(_new_formula_)
		)),
		Expr(_new_formula_), Name Expr(new_formula)
	));
	return(new_col);
);

test3 = function({dt, val_col, by_cols}, {Default Local},
	ref_list = Transform Each({col_name}, Insert(by_cols, val_col, 1),
		Name Expr(As Column(dt, col_name));
	);
	
	new_formula = Char(Substitute(ref_list, Expr(list), Expr(Col Median)));
	new_col = Eval(Substitute(
		Expr(dt << New Column("Formula", Numeric, Continuous,
			Formula(_new_formula_)
		)),
		Expr(_new_formula_), Parse(new_formula)
	));
	return(new_col);
);

test4 = function({dt, val_col, by_cols}, {Default Local},
	new_col = Eval(Substitute(
		Expr(dt << New Column("Formula", Numeric, Continuous,
			Formula(Col Median(_new_formula_))
		)),
		Expr(_new_formula_), Insert(by_cols, val_col, 1)
	));
	return(new_col);
);

test5 = function({dt, val_col, by_cols}, {Default Local},
	new_col = Eval(Substitute(
		Expr(dt << New Column("Formula", Numeric, Continuous,
			Formula(_new_formula_)
		)),
		Expr(_new_formula_), Substitute(Insert(by_cols, val_col, 1), Expr(List), Expr(Col Median))
	));
	return(new_col);
);

dt = Open("$SAMPLE_DATA/Big Class.jmp");
by_cols = {"age", "sex"};
val_col = "height";

col1 = test1(dt, val_col, by_cols);
col2 = test2(dt, val_col, by_cols);
col3 = test3(dt, val_col, by_cols);
col4 = test3(dt, val_col, by_cols);
col5 = test3(dt, val_col, by_cols);

Show(col1 << get formula);
Show(col2 << get formula);
Show(col3 << get formula);
Show(col4 << get formula);
Show(col5 << get formula);

//dt1 = dt << Subset(All Rows, Selected columns(0), Suppress formula evaluation(0));
//Close(dt, no save);

If I for example open Formula 2's formula editor and press apply I might get an error

Is there any way to recover from the "broken" formulas and is there a possibility to see it during creation that it will break? I can use Parse(Char()) to get working formula and I can also create subset of the data and the formulas will work there.

Tested with JMP Pro 16.2.0 on Windows 10.

[#2 · Accepted solution]

try

Expr(Col Median())

with the extra ().

The parser sees the parens and sets the function flag on the node in the expression tree. Without parens, the Col Median node is just a name that happens to have children that won't be used ( except by arg() and char() conversion ). At run time the name is not in the name space of variables, and the name space of builtin functions won't be used, thus the message.

I think this explanation is consistent with the behavior of all 5 tests.

Formula(...) is different from ColMedian(...); ColMedian is a JSL function. Formula is a keyword that the NewColumn message recognizes. The keyword handling expects children, so it might not need expr(Formula()), but would probably work just fine with it.

edit: I'm pretty sure the function flag is not exposed in JSL via arg(), etc.