cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
  • Learn how to build custom Python data connectors and further customize JMP’s Data Connector Framework with the Python Data Connector Demo, available now in the JMP Marketplace!
  • See how to create experiments to support product design and ID useful product features. Register for June 12 webinar, 2pm US Eastern Time.

Discussions

Solve problems, and share tips and tricks with other JMP users.
Choose Language Hide Translation Bar
andrey
Level II

Data Filter in JSL script running slow

I am trying to create a JSL script that will utilize data filter and select ~180 tests of interest in "ITEM_NAME" column and create a new table. When I use Data Filter and run it manually it doesn't have any delays. Though when I try to automate the process it takes ~15 minutes to create the new table. What can I do to make it run more efficiently?

Open( "$DOCUMENTS/test.jmp" );

ref10 = Data Table( "ABC" ) << Select Where(
	:ITEM_NAME == "test1" | 
	:ITEM_NAME == "test2" | 
	:ITEM_NAME =="test3" | 
	:ITEM_NAME ==
	...
	"test180"
	
) << Data View;

Data Table( ref10 ) << Save(
	"$DOCUMENTS/filtered.jmp"
);
2 ACCEPTED SOLUTIONS

Accepted Solutions
txnelson
Super User

Re: Data Filter in JSL script running slow

Try this

Open( "$DOCUMENTS/test.jmp" );

items={"test1" "test2""test3"..........};

ref10 = Data Table( "ABC" ) << Select Where(
	contains(ITEMS) ) << Data View;

Data Table( ref10 ) << Save(
	"$DOCUMENTS/filtered.jmp"
);
Jim

View solution in original post

jthi
Super User

Re: Data Filter in JSL script running slow

It is a bit difficult to understand why your script would be that slow without seeing/understanding your data (it shouldn't be).

 

You could try using << Get Rows Where and Subset instead of selecting those rows. In this example I have tried to replicate Data View by using linked subset and Untitled output table name

Names Default To Here(1); 

dt = open("$SAMPLE_DATA/Big Class.jmp");

items = {12, 13, 14, 15};
rows_of_interest = dt << get rows where(Contains(items, :age));

dt_subset = dt << Subset(Rows(rows_of_interest), Selected Columns(0), Link to original data table(1), Output Table("Untitled"));
-Jarmo

View solution in original post

7 REPLIES 7
txnelson
Super User

Re: Data Filter in JSL script running slow

Try this

Open( "$DOCUMENTS/test.jmp" );

items={"test1" "test2""test3"..........};

ref10 = Data Table( "ABC" ) << Select Where(
	contains(ITEMS) ) << Data View;

Data Table( ref10 ) << Save(
	"$DOCUMENTS/filtered.jmp"
);
Jim
andrey
Level II

Re: Data Filter in JSL script running slow

Thank you Jim. For some reason I am getting the error (likely syntax related). 

andrey
Level II

Re: Data Filter in JSL script running slow

Seems to be running faster now and took around 3-4 minutes to complete but nowhere near as quick if I was to do this manually. Had to make a slight change to the code to avoid the error:

Open( "$DOCUMENTS/test.jmp" );

items={"test1", "test2", "test3"..........};

ref10 = Data Table( "ABC" ) << Select Where(
	contains(items, ITEM_NAME) ) << Data View;

Data Table( ref10 ) << Save(
	"$DOCUMENTS/filtered.jmp"
);

 

txnelson
Super User

Re: Data Filter in JSL script running slow

My error.....it was late at night....

Jim
jthi
Super User

Re: Data Filter in JSL script running slow

It is a bit difficult to understand why your script would be that slow without seeing/understanding your data (it shouldn't be).

 

You could try using << Get Rows Where and Subset instead of selecting those rows. In this example I have tried to replicate Data View by using linked subset and Untitled output table name

Names Default To Here(1); 

dt = open("$SAMPLE_DATA/Big Class.jmp");

items = {12, 13, 14, 15};
rows_of_interest = dt << get rows where(Contains(items, :age));

dt_subset = dt << Subset(Rows(rows_of_interest), Selected Columns(0), Link to original data table(1), Output Table("Untitled"));
-Jarmo
andrey
Level II

Re: Data Filter in JSL script running slow

Thank you Jarmo. You solution shaved the processing time to about 1:35 seconds.

The JMP table I am using has 13 columns and 13127825 raws. The output file has 1274416 raws.

jthi
Super User

Re: Data Filter in JSL script running slow

There are most likely optimizations which could be done but if that speed is enough for your application, I wouldn't worry about those.

-Jarmo

Recommended Articles