Choose Language Hide Translation Bar
d_barnett
Community Trekker

Concatenation of a Defined number of .jmp Files in Folder

I have >600,000 jmp data tables in a single folder and each of these has 12,000 rows and the file size of each is ~100kb. I have tried without success to concat all of these together but, not unsurprisingly, I can only generate a file of ~15million rows ( approx 1200 files) before JMP runs out of steam and is unresponsive.

I used this script to perform the operation, which I copied from one of these discussions ( apologies but I couldn't find the actual board so can't link to it). I could copy 1,000 files into individual folders and then concat but that would be rather tedious.

 

 

mypath = Pick Directory( "Select a directory" );
Set Current Directory( mypath );
y = Files In Directory( mypath );
For( i = N Items( y ), i > 0, i--,
If( Ends With( y[i], "jmp" ),
,
Remove From( y, i )
)
);
n = N Items( y );
For( i = 1, i <= N Items( y ), i++,
sname = Left( y[i], Length( y[i] ) - 4 );
/* If the file is the first in the list, open it. Otherwise, open the table, concatenate with the main table, and close the table just opened. */
If( i == 1,
mainDt = Open( mypath || y[i] );

mainDt << New Column( "source", character, set each value( sname ) );
,
dt = Open( mypath || y[i], hidden, invisible );
dt << New Column( "source", character, set each value( sname ) );
mainDt = mainDt << Concatenate( dt, Append to First Table );
Close( dt, NoSave );
Wait( 0 );
);
);
/* Give the final table a name. */
mainDt << Set Name( mypath, "_Concatenated Files" );
dt << Save( mypath, "_Concatenated Files" );

 

 

This add-in doesn't work either https://community.jmp.com/t5/JMP-Add-Ins/Bulk-Concatenate-Add-in/ta-p/35435

 

Is there a way to only to concat the first 1000 files, save the concatenated table, close it then move onto the next set of 1,000?  I've tried a few things but they usually just stop after the first set is complete.

Regards

 

David

0 Kudos
1 REPLY 1
txnelson
Super User

Re: Concatenation of a Defined number of .jmp Files in Folder

Here is a modification to your current script that now saves every 1000 files into a concatenated file.  In the new script, I also corrected an error, where a few times you were not properly concatenating "mypath" with the "_Concatenated Files" string.  Additionally, in your Save statement, you were pointing to the wrong data table to save.

mypath = Pick Directory( "Select a directory" );
Set Current Directory( mypath );
y = Files In Directory( mypath );
For( i = N Items( y ), i > 0, i--,
	If( Ends With( y[i], "jmp" ),
		,
		Remove From( y, i )
	)
);
n = N Items( y );
counter=0;
For( i = 1, i <= N Items( y ), i++,
	sname = Left( y[i], Length( y[i] ) - 4 );
/* If the file is the first in the list, open it. Otherwise, open the table, concatenate with the main table, and close the table just opened. */
	If( counter == 0,
		mainDt = Open( mypath || y[i] );

		mainDt << New Column( "source", character, set each value( sname ) );
	,
		dt = Open( mypath || y[i], hidden, invisible );
		dt << New Column( "source", character, set each value( sname ) );
		mainDt = mainDt << Concatenate( dt, Append to First Table );
		Close( dt, NoSave );
		Wait( 0 );
	);
	counter = counter + 1;
	If(mod(counter,1000) == 0,
		mainDt << Set Name( mypath || "_Concatenated_Files_" || char(i) || "_" || char(i+counter) );
		mainDt << Save( mypath || "_Concatenated_Files_" || char(i) || "_" || char(i+counter) );
		counter = 0;
	)
);
/* Give the final table a name. */
mainDt << Set Name( mypath || "_Concatenated_Files_" || char(i) || "_" || char(i+counter) );

mainDt << Save( mypath || "_Concatenated_Files_" || Char( i ) || "_" || Char( i + counter ));
Jim