topic Re: Data Filter in JSL script running slow in Discussions

Data Filter in JSL script running slow

andrey — Tue, 13 May 2025 01:50:04 GMT

I am trying to create a JSL script that will utilize data filter and select ~180 tests of interest in "ITEM_NAME" column and create a new table. When I use Data Filter and run it manually it doesn't have any delays. Though when I try to automate the process it takes ~15 minutes to create the new table. What can I do to make it run more efficiently?

Open( "$DOCUMENTS/test.jmp" );

ref10 = Data Table( "ABC" ) << Select Where(
	:ITEM_NAME == "test1" | 
	:ITEM_NAME == "test2" | 
	:ITEM_NAME =="test3" | 
	:ITEM_NAME ==
	...
	"test180"
	
) << Data View;

Data Table( ref10 ) << Save(
	"$DOCUMENTS/filtered.jmp"
);

Re: Data Filter in JSL script running slow

txnelson — Tue, 13 May 2025 01:59:30 GMT

Try this

Open( "$DOCUMENTS/test.jmp" );

items={"test1" "test2""test3"..........};

ref10 = Data Table( "ABC" ) << Select Where(
	contains(ITEMS) ) << Data View;

Data Table( ref10 ) << Save(
	"$DOCUMENTS/filtered.jmp"
);

Re: Data Filter in JSL script running slow

andrey — Tue, 13 May 2025 17:35:00 GMT

Thank you Jim. For some reason I am getting the error (likely syntax related).

Re: Data Filter in JSL script running slow

andrey — Tue, 13 May 2025 17:45:20 GMT

Seems to be running faster now and took around 3-4 minutes to complete but nowhere near as quick if I was to do this manually. Had to make a slight change to the code to avoid the error:

Open( "$DOCUMENTS/test.jmp" ); items={"test1", "test2", "test3"..........}; ref10 = Data Table( "ABC" ) << Select Where( contains(items, ITEM_NAME) ) << Data View; Data Table( ref10 ) << Save( "$DOCUMENTS/filtered.jmp" );

Re: Data Filter in JSL script running slow

txnelson — Tue, 13 May 2025 17:49:16 GMT

My error.....it was late at night....

Re: Data Filter in JSL script running slow

jthi — Tue, 13 May 2025 17:52:18 GMT

It is a bit difficult to understand why your script would be that slow without seeing/understanding your data (it shouldn't be).

You could try using << Get Rows Where and Subset instead of selecting those rows. In this example I have tried to replicate Data View by using linked subset and Untitled output table name

Names Default To Here(1); 

dt = open("$SAMPLE_DATA/Big Class.jmp");

items = {12, 13, 14, 15};
rows_of_interest = dt << get rows where(Contains(items, :age));

dt_subset = dt << Subset(Rows(rows_of_interest), Selected Columns(0), Link to original data table(1), Output Table("Untitled"));

Re: Data Filter in JSL script running slow

andrey — Tue, 13 May 2025 18:36:52 GMT

Thank you Jarmo. You solution shaved the processing time to about 1:35 seconds.

The JMP table I am using has 13 columns and 13127825 raws. The output file has 1274416 raws.

Re: Data Filter in JSL script running slow

jthi — Tue, 13 May 2025 20:30:11 GMT

There are most likely optimizations which could be done but if that speed is enough for your application, I wouldn't worry about those.