Subscribe Bookmark RSS Feed

Concatenate Multiple Data Tables [SOLVED]

mad

Community Trekker

Joined:

Jun 23, 2011

I have 605 data files I need to concatenate after filtering out the data I don't need from each file. Looping through each file, deleting the rows I don't need, concatenating the current file to the first file, then closing the current file works fine for the first 10+ files but gets exponentially slower with each additional file. Any suggestions?

I have noticed that each time a file is concatenated the source script gets added to the data table, could this be what is slowing things down? Is there a way to prevent this or delete them?

Message was edited by: MAD
5 REPLIES
mad

Community Trekker

Joined:

Jun 23, 2011

Each time a file is opened and after each concatenation operation add:

dt << Delete Table Property( "Source" );

This dramatically improved the speed.
chungwei

Staff

Joined:

Jun 23, 2011

what version of JMP are you running ?
mad

Community Trekker

Joined:

Jun 23, 2011

JMP 8.0
chungwei

Staff

Joined:

Jun 23, 2011

I think this was fixed in 8.0.1.
mad

Community Trekker

Joined:

Jun 23, 2011

You are correct. I installed the 8.0.2 update and removed my workaround. It works fine. Thank you for pointing me to the update.