Choose Language Hide Translation Bar
Staff (Retired)

Concatenate data tables


Concatenate data tables using Files in Directory, without using Multiple File Import



Use Files in Directory to make a list and Concatenate to put them together.


// make a test directory
dt = New Table( "Untitled", Add Rows( 1 ), New Column( "a" ), New Column( "b" ) );
path = "$TEMP/test/";
Delete Directory( path ); // Be careful! This will delete your files!
Create Directory( path );
For( i = 1, i <= 3, i += 1,
	dt:a = i; // make each table a little different
	dt:b = 10 * i;
	dt << save( path || Char( 1000 + i ) ); // make the table names sort 
Close( dt, nosave );

// here's the code to open and concatenate...
namelist = Files In Directory( path );
Sort List Into( namelist ); // not really needed?
dtlist = {}; // hold the opened, hidden, tables here
While( N Items( filename = Remove From( namelist, 1 /*first item*/ ) ) == 1,
	filename = filename[1]; // {dt} -> dt
	filename = path || filename; // prepend path name
	// inserts opened table at end of dtlist
	Insert Into( dtlist, Open( filename, invisible ) );
rememberdt = dtlist; // need these later to close the hidden tables
firstdt = Remove From( dtlist, 1/*first item*/ )[1]; // need one to start
If( N Items( dtlist ), // concatenate needs more than one table
	resultdt = firstdt << Concatenate( dtlist, Create source column )
, // subset returns a fresh table, like concatenate, so both cases have same behavior
	resultdt = firstdt << Subset( All rows, Selected columns only( 0 ) )
// clean up the hidden tables
While( N Items( closedt = Remove From( rememberdt ) ) == 1, Close( closedt[1], nosave ) );
// clean up new table
resultdt << setname( "all my tables" );
resultdt << deletetableproperty( "Source" ); // from subset or concatenate
Print( "done", resultdt );



JMP 14 introduce Multiple File Import; @wu noticed it did not preserve some table attributes. (JMP 15 should do better, but is not available yet.)

The first part of this example uses DeleteDirectory and CreateDirectory to make a clean set of sample files for testing. You might discover DeleteDirectory fails if any of the JMP tables are still open from a previous attempt; you can close the tables manually, even if they are hidden, using the table widget


Click the table icon beside the widget to open the hidden tableClick the table icon beside the widget to open the hidden table

That same code adds 1000 to the file names to make them sort nicely ( not 1,10,2,3,4,5,6,7,8,9 ). The path variable has a trailing slash so the slash doesn't have to be added in other places.

namelist is initialized by Files In Directory; dtlist is initialized to an empty list. The names are removed, one by one, from namelist and opened (hidden). The opened tables are stored one by one in dtlist. (That's just a pointer, or handle, to the table, not the contents of the table.)

Then, if there is more than one dtlist item, concatenate makes a new table from them. (I was happy to find concatenate accepts a list of tables.) If there is only one, subset makes a copy. Making a copy makes an unsaved table, just like concatenate made, so the results are similar in both cases.

Clean up removes the hidden tables and renames the result table and removes the source script from the result. You might want to keep the source script. This example could also use Delete Directory at the end, but it is probably a bad idea to do that to real source files.


See Also




New user here and I found this to be very helpful when trying to consolidate data from many multiple sets. Thanks! 

@Alex-L_NC Thanks!

Article Labels

    There are no labels assigned to this post.

Article Tags