speeding up heatmap graphs?

[#1]

In the exmple below, a data filter is used to pre-select "groups" of data points. These groups are plotted in Graph Builder.
The good thing: once the Graph Builder is available, the user can select different entries in the Data Filter and the graph will show up almost instantaneously.
The disadvantage:
It takes some time for the GraphBuilder to load. The Heatmap Plot takes almost a factor 10 longer than a Points Plot.
It's not the number of plotted data points which determines the loading time, it's the total number of data points in the table.

My workaround at the moment:
Reduce the number of data points in the table:
Use a list box instead of a data filter and use it to generate a subset of the original data set.
Then use the subset as input for the GraphBuilder. 
This solves the speed issue, but there are other drawbacks.

Why does it take so much longer to generate a "Heatmap"?
Is there some clever way to increase the speed?
At some time, I thought that switching from continuous to ordinal values on the axes reduces the plotting time. But in this example the loading time is quite similar for the 3 "variant" options.

Another thing that I noticed:
If I use Today() before and after a Graphbuilder command, the time difference is quite small, much smaller than the waiting time until the Graph Builder Window gets visible. A time measurement with a "new window" around the Graph Builder gives a more realistic value.

View more...

try Variant=1; // 1-3 (no influence on timing)
try Command=1; // 1: Heatmap, 2: Points (~ factor 10 faster)
groups = 500;


variant = {"Continuous", "Ordinal","Nominal"}[tryVariant];
myCommand={Expr(Elements( Heatmap( X, Y, Legend( 17 ) ) )),Expr(Elements( Points( X, Y, Legend( 17 ) ) ))}[tryCommand];


dt = New Table( "dt" );

Eval( Eval Expr( New Column( "X", Numeric, Expr( variant ), Format( "Best", 12 ), Formula( Floor( Modulo( (Row() - 1), 10000 ) / 100 ) ) ) ) );
Eval( Eval Expr( New Column( "Y", Numeric, Expr( variant ), Format( "Best", 12 ), Formula( Modulo( (Row() - 1), 100 ) ) ) ) );
New Column( "group", Numeric, "Nominal", Format( "Best", 12 ), Formula( Floor( (Row() - 1) / 10000 ) ) );
	
New Column( "value",
	Numeric,
	"Continuous",
	Format( "Best", 12 ),
	Formula( Random Normal( 0, 0.001 ) + If( Random Integer( 10000 ) < 5, 1, 0 ) ), 

);

dt << Add Rows( 10000 * groups, At End );


New Window( "wait ...", <<Modal, V List Box( Text Box( "wait until the data table is ready ..." ), Button Box( "OK" ) ) );
//don't click too early!
// If the Data Filter command in the Data Filter Context Box  is executed before the data table is  ready,  the "where"" clause will just be ignored --> all groups will be plotted
// surprising: the data filter inside the GraphBuilder doesn't show this effect ...

start1 = Today();
	
Eval(Eval Expr(New Window( "Data Filter Context Box",
	Data Filter Context Box(
		H List Box(
			dt << Data Filter(
				Local,
				Add Filter( columns( :group ), Where( :group == As List( 0 :: 4 )[1] ), Display( :group, N Items( 4 ), Find( Set Text( "" ) ) ) )
			),
			dt << Graph Builder(
				Size( 400, 400 ),
				Show Control Panel( 0 ),
				Graph Spacing( 5 ),
				Variables( X( :X ), Y( :Y ), Wrap( :group ), Color( :value ) ),
				Expr(myCommand),
				SendToReport(
					Dispatch( {}, "X", ScaleBox, {Min( 0 ), Max( 100 ), Inc( 10 ), Minor Ticks( 9 )} ),
					Dispatch( {}, "Y", ScaleBox, {Min( 0 ), Max( 100 ), Inc( 10 ), Minor Ticks( 9 )} ),
					Dispatch(
						{},
						"400",
						ScaleBox,
						{Legend Model(
							17,
							Properties(
								0,
								{gradient( {Color Theme( "Black Body" ), Scale Values( [0 1] ), Reverse Gradient( 1 )} )},
								Item ID( "value", 1 )
							)
						)}
					)
				)
			)
		)
	)
)));
stop1 = Today();
Print( "data filter context box: ", stop1 - start1 );

New Window( "wait ...", <<Modal, V List Box( Text Box( "generate 2nd graph ..." ), Button Box( "OK" ) ) );

start2 = Today();

//  "new window" around the GraphBuilder to get the actual time. Without the "new window", the >calculated< time difference is much smaller, but it takes the same time for the graph to show up
Eval(EvalExpr(New Window( "Data Filter inside GraphBuilder",
	dt << Graph Builder(
		Size( 400, 400 ),
		Show Control Panel( 0 ),
		Graph Spacing( 5 ),
		Variables( X( :X ), Y( :Y ), wrap( :group ), Color( :value ) ),
		Expr(myCommand),
		Local Data Filter(
			Add Filter( columns( :group ), Where( :group == As List( 0 :: 4 )[1] ), Display( :group, N Items( 15 ), Find( Set Text( "" ) ) ) )
		),
		SendToReport(
			Dispatch( {}, "X", ScaleBox, {Min( 0 ), Max( 100 ), Inc( 10 ), Minor Ticks( 9 )} ),
			Dispatch( {}, "Y", ScaleBox, {Min( 0 ), Max( 100 ), Inc( 10 ), Minor Ticks( 9 )} ),
			Dispatch(
				{},
				"400",
				ScaleBox,
				{Legend Model(
					17,
					Properties( 0, {gradient( {Color Theme( "Black Body" ), Scale Values( [0 1] ), Reverse Gradient( 1 )} )}, Item ID( "value", 1 ) )
				)}
			)
		)
	)
)));
stop2 = Today();
Print( "local data filter inside the Graphbuilder: ", stop2 - start2 );

[#2 · Accepted solution]

Maybe for some reason heatmap gets built without local filter applying and after it is ready local data filter is applied and that's why it is so slow. Maybe you could try to speed it up by first creating graph builder with points and then "changing" points to heatmap

Names Default To Here(1);

groups = 500;

dt = New Table("dt",
	Add Rows(10000 * groups),
	New Column("X", Numeric, Ordinal, Format("Best", 12), <<Set Each Value(Floor(Modulo((Row() - 1), 10000) / 100))),
	New Column("Y", Numeric, Ordinal, Format("Best", 12), <<Set Each Value(Modulo((Row() - 1), 100))),
	New Column("group", Numeric, Ordinal, Format("Best", 12), <<Set Each Value(Floor((Row() - 1) / 10000))),
	New Column("value", Numeric, "Continuous", Format("Best", 12), <<Set Each Value(Random Normal(0, 0.001) + If(Random Integer(10000) < 5, 1, 0)))
);

Wait(0);

start = HP Time();
gb = Graph Builder(
	Variables(X(:X), Y(:Y), Wrap(:group), Color(:value)),
	Elements(Points(X, Y, Legend(27))),
	Local Data Filter(Add Filter(columns(:group), Where(:group == 0), Display(:group, N Items(15), Find(Set Text(""))))),
	SendToReport(
		Dispatch({}, "X", ScaleBox, {Min(0), Max(100), Inc(10), Minor Ticks(9)}),
		Dispatch({}, "Y", ScaleBox, {Min(0), Max(100), Inc(10), Minor Ticks(9)}),
		Dispatch(
			{},
			"400",
			ScaleBox,
			{Legend Model(
				27,
				Properties(
					-1,
					{gradient({Color Theme("Black Body"), Scale Values([0 1]), Range Type("Exact Data Range"), Reverse Gradient(1), Width(12)})},
					Item ID("Count", 1)
				)
			)}
		)
	)
);

gbb = Report(gb)[Graph Builder Box(1)];
Wait(0);
Show((HP Time() - start) / 1e6);
gbb << Remove Element(1, 1, 1);
gbb << Add Element(1, 1, {Type("Heatmap"), X, Y});
Wait(0);
Show((HP Time() - start) / 1e6);

Write();

[#3]

amazing speed (feedback time and plotting time

surprising explanation

beautiful workaround

and a clear sign that the issue could get fixed in a future JMP release

[#4]

Thanks for fixing the issue in Jmp 17
https://community.jmp.com/t5/JMP-Wish-List/Heatmap-Data-filter-speed-up-the-graph-generation/idc-p/5...