JSL Adding "And" condition to Data Filter

[#1]

Hello JMP Community

I was wondering if there is a way to add an "and" condition to a data filter in JSL.

as an example I made a dataset

If we make a simple model over just the mean,

model = Fit Model( Y( :Data ), Run);

We can restrict the model to just one group with a data filter

dtf = model << Local Data Filter(Columns(:Group), where(:Group == "A"));

However now if I want to add another filter the only JSL options i could find( "Add Filter", "Filter Columns", and similar) only seems to add as an "Or" condition.

dtf << Add Filter(columns(:Outlier), where(:outlier == 0));
dtf << Filter Columns(:Outlier);

The only option seem to be to remove the data filter, then add a new filter with all the conditions

model << Remove Local Data Filter;
dtf = model << Local Data Filter(Columns(:Group, :Outlier), where(:Group == "A"), where(Outlier == 0));

Am i missing a way to add "And" conditions or is this just not possible.

br.

Kasper Schlosser 

[#2]

Create the filter in JMP and copy the script

Fit Model(
	Y(:Data),
	Effects,
	Personality("Standard Least Squares"),
	Emphasis("Effect Leverage"),
	Run(
		:Data << {Summary of Fit(1), Analysis of Variance(1), Parameter Estimates(1),
		Lack of Fit(0), Plot Actual by Predicted(1), Plot Regression(0),
		Plot Residual by Predicted(1), Plot Studentized Residuals(0),
		Plot Effect Leverage(1), Plot Residual by Normal Quantiles(0),
		Box Cox Y Transformation(0)}
	),
	Local Data Filter(
		Add Filter(
			columns(:Group, :Outlier),
			Where(:Group == "A"),
			Where(:Outlier == 1)
		)
	)
);

This generally gives a good idea what you can then script

dt = Current Data Table();

model = Fit Model(Y(:Data), Run);

dtf = model << Local Data Filter(
	Columns(:Group, :Outlier),
	Where(:Group == "A"),
	Where(:Outlier == 1)
);

[#3]

Yes or the scripting index also gives an overview of what messages the a data filter accepts (though not always all).

Both of these seem to indicate that no, there is no way to add a "and" condition to a data filter through jsl.

This would mean the only way to add an "and" condition is in jsl is to extract the filter code, change the code, then parse and evaluate. which is messy and error prone.

[#4]

Make sure you add to the JMP Wish List this exposed limitation.

Also, I guess I have a different take on your mentioned parse and evaluate.  It gives me comfort that if I come up against a limitation JMP provides this base level capability that most times allows me a path to solution.  

[#5]

When you add new columns to filters, you are basically creating new filtering groups and by default those are OR separated. Depending on the use case, sometimes you might be able to use Grouped by And

Names Default To Here(1);
dt = Open("$SAMPLE_DATA/Cities.jmp");
obj = dt << Data Filter(
	Add Filter(columns(:Region, :POP), Where(:Region == {"C", "N"})),
	Mode(Select(0), Show(0), Include(1))
);
obj << Add Filter Columns(:State);
obj << Grouped by AND(1);

On the UI we have Add to Selected Group when you press AND but we do not have simple scripting option for that.

[#6 · Accepted solution]

---

@jthi wrote:

but we do not have simple scripting option for that.

---

I think we do have:

Names Default to Here(1);
dt = Open( "$SAMPLE_DATA/Big Class Families.jmp" );
gb = dt << Graph builder();

dtf = gb << Local Data Filter(
	Columns(:sex),
	Where(:sex == "F")
);

dtf << (Filter Column(:age) << Where(:age == 14));

[#7]

This seems to work

Quick testing indicates, this method  affects the first filter. ie. if we have 2 "or" conditions we can only affect the first one.

Names Default to Here(1);
dt = Open( "$SAMPLE_DATA/Big Class Families.jmp" );
gb = dt << Graph builder();

dtf = gb << Local Data Filter(
	Columns(:sex),
	Where(:sex == "F")
);

// adds or condition
dtf << Add Filter(Columns(:sex), Where(:sex == "M"))

// Only affects the female filter
dtf << (Filter Column(:age) << Where(:age == 14));

but for my use case this is perfect.

[#8]

Nice find!

This seems to work only if you also send the where message with the filter column. This might be unintended / bug or something but we are lucky we have it (it just might break any given time when JMP gets updated).

Names Default To Here(1);
dt = Open("$SAMPLE_DATA/Cities.jmp");
obj = dt << Data Filter(
	Add Filter(
		columns(:Region, :POP),
		Where(:Region == {"C", "N"})
	),
	Mode(Select(0), Show(0), Include(1))
);

obj << Filter Column(:State); // creates OR group
obj << (Filter Column(:State) << Where()); // Adds AND filter to first filter group
obj << (Filter Column(:CO) << Where()); // Adds AND filter to first filter group

Most likely it would be much better if we were somehow able to just add AND filter within existing groups and not just new filter OR groups.

@Craige_Hales any idea why this is happening / any idea who to tag from JMP's side?

[#9 · Accepted solution]

As @jthi pointed out earlier, filters are composed of "groups" and "items", and the filter group is not directly exposed to JSL as a scriptable object.  The syntax df << (Filter Column(:col) << Msg()) was the original way to "forward" messages to a group, though it is not well documented.  I will point out that this syntax is a very different path than the simple message df << Filter Column(:col), which is documented in the Scripting Index.  The former syntax does not have built-in JSL rules for processing, though it is used in similar ways in other objects, like Bivariate.  Essentially, the message being sent to the filter is an expression, and the syntax supported is custom to the filter.

A little more on the special syntax for forwarding the messages.  There is an optional second argument to specify the index of the filter group that should receive the message.  This can be important when you have multiple groups, which might even contain the same columns.  Without this, the command will be sent to the first matching column, or added to the first group.

You may also find that the Msg() being sent to the filter item can only be a simple message.  The processing here does not support sending a list of commands to be processed.  This is again due to the inner << being a custom execution and not directly using the core JSL rules.

dt = Open( "$SAMPLE_DATA/Cities.jmp" );
df = dt << Data Filter(
	Location( {1001, 218} ),
	Add Filter( columns( :POP ), Where( :POP >= 4461 ) ),
	Add Filter( columns( :OZONE ), Where( :OZONE <= 0.12581 ) )
);
// add a filter item to the first group
df << (Filter Column(:Region, 1) << Check Box Display);
// add a filter item to the second group
df << (Filter Column(:NO, 2) << Where(:NO <= 0.05));