topic Data filter: how to specify the limits in Discussions

Data filter: how to specify the limits

hogi — Fri, 16 Jun 2023 16:06:54 GMT

In a Data Filter, a user is limited by which values can be entered as limits.
If the keyed in limit is outside of the range of the available data points, the value is automatically replaced by min or max.
The user can change the JSL code - but when the graph opens, the values are again set back to min and max.

a) Is there a hidden trick how to key in any number ? shift enter did not work
b) Is there a trick how to generate a dashboard with user defined filter specs - which stay as they are?

Re: Data filter: how to specify the limits

PatrickGiuliano — Fri, 16 Jun 2023 18:28:01 GMT

@hogi someone on my team in Technical Support should be providing their assessment on question a) courtesy of your email to support@jmp.com (dated 06/16/23) in relation to question a), but for question b) "Is there a trick how to generate a dashboard with user defined filter specs - which stay as they are?"

It seems that the specs persist for me after defining them. Can you provide more information with an example to describe what you're seeing?

Re: Data filter: how to specify the limits

hogi — Fri, 16 Jun 2023 21:40:52 GMT

To generate a dashboard with a filter beyond the current data range, I generated a GraphBuilder plot with lower spec = 1 via

Graph Builder(
	Variables( X( :height ), Y( :weight ), Overlay( :sex ) ),
	Elements( Points( X, Y, Legend( 1 ) ), Smoother( X, Y, Legend( 2 ) ) ),
	Local Data Filter(
		Conditional,
		Add Filter( columns( :height ), Where( :height >= 1 ) )
	)
)

the result didn't look promising:

then I added it to a Dashboard

and checked the code - and indeed - Jmp did adjust my spec:

I set the limit back to 1 and used the code to generate a new dashboard, checked the code again and it was again 51.

Re: Data filter: how to specify the limits

hogi — Fri, 16 Jun 2023 21:51:52 GMT

Then I added a fake row with height =1, which allowed me to define a data filter with lower spec =1:

but as soon as I deleted the row, the data filter was changed again to 51 ...

A data filter with specs outside of the current data range seems to be a quite fragile object.
Up to now, I generated Dashboards, saved them as table scripts, loaded new data during the weeks, optimized the dashboards, saved them again as a table script ...
Dangerous if data filters adjust dynamically to the current data ranges.

Re: Data filter: how to specify the limits

danschikore — Fri, 16 Jun 2023 22:16:26 GMT

@hogi - In JMP 17, if you do not modify one of the min/max values:

They do not appear in the script:

Bivariate(
	Y( :weight ),
	X( :height ),
	Fit Line( {Line Color( {212, 73, 88} )} ),
	Local Data Filter( Add Filter( columns( :height ), Where( :height <= 65 ) ) )
);

This helps when you have changing data and need one-sided limits. However, if you modify both values and then drag them back to the extreme points, both values will appear in the script. I would suggest that after you do this you might "Clear" the filter item, which will result in no Where() clause being generated. Unfortunately the presence of the Where() clause is the indicator that tells us whether the filter conditions have been set, and that is why you see both values in the scenario that you show.

Re: Data filter: how to specify the limits

hogi — Tue, 27 Jun 2023 07:47:36 GMT

Thanks @danschikore for the expalantion.

When I posted my question I didn't think about opening one side of the filter.

The idea was:
if I want the data filter to automatically remove bad data points (<0) or some other well-defined data (missing data, indicator entry), such data has to be in the current data set - otherwise the data filter will automatically adapt.

In this case I want to remove all rows with names "reference" and missing names:

dt = Open( "$SAMPLE_DATA/Big Class.jmp" );
Data Table( "Big Class" ):name << Set Modeling Type( "Multiple Response" );
Graph Builder(
	Summary Statistic( "Median" ),
	Graph Spacing( 4 ),
	Variables( X( :height ), Y( :weight ), Overlay( :sex ) ),
	Elements( Points( X, Y, Legend( 1 ) ), Smoother( X, Y, Legend( 2 ) ) ),
	Local Data Filter(
		Add Filter(
			columns( :name ),
			Match None( Where( :name == "reference","" ) ),
			Display( :name, Find( Set Text( "" ) ) )
		)
	)
);

but if there is no such row in the current data set, the data filter will just be ignored - and the user will lose this part when he saves the current version of the graph or Dashboard. Just run the code and save the resulting script.
Next time the user opens the Graph, rows with missing name and "reference" data will be displayed by default and the user has to exclude them manually.

This issue will get more severe as soon as the option not gets available for more modeling types:

Re: Data filter: how to specify the limits

danschikore — Sat, 17 Jun 2023 14:00:44 GMT

@hogi - Thanks for the context - data cleaning is often performed outside the analysis, but one way you can build these conditions into the filter is to define a transform column with the conditions that you want to remove. This gives you the ability to use an arbitrary Where() clause as a boolean condition:

dt = Open("$SAMPLE_DATA/Big Class.jmp");
dt << Graph Builder(
	Transform Column(
		"Invalid Data",
		Nominal,
		Formula( :age > 18 | Starts With( :name, "Teacher" ) )
	),
	Show Control Panel( 0 ),
	Variables( X( :height ), Y( :weight ), Overlay( :sex ) ),
	Elements( Points( X, Y, Legend( 1 ) ), Smoother( X, Y, Legend( 2 ) ) ),
	Local Data Filter(
		Conditional,
		Add Filter(
			columns( :age, :Invalid Data ),
			Where( :Invalid Data == 0 ),
			Display( :age, N Items( 6 ) )
		)
	)
);

This does assume that you don't have a table with all invalid data. Another approach would be to use the LDF in combination with a global data filter to remove the invalid data from all of the reports. The global row-states will combine with the filter conditions used in each platform.

Re: Data filter: how to specify the limits

hogi — Sun, 18 Jun 2023 07:34:59 GMT

Thanks @danschikore , what an elegant workaround :)

The only case it will fail: if there is a dataset with no valid data, i.e. no data

Where( :Invalid Data == 0 )

but then anyways, nobody would dare to save the Dashboard :)

Re: Data filter: how to specify the limits

hogi — Sat, 24 Aug 2024 13:58:53 GMT

(aka Forced Values) will fix this issue.