cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Submit your abstract to the call for content for Discovery Summit Americas by April 23. Selected abstracts will be presented at Discovery Summit, Oct. 21- 24.
Discovery is online this week, April 16 and 18. Join us for these exciting interactive sessions.
Choose Language Hide Translation Bar
Rajat
Level IV

Concatenate Multiple tables with different column order

Team,

I have 5 csv files each having 200column (not each file have same number of columns). Each file has columns in different order. I want to write a scirpt which will let user to select all 5 files then combine all 5 files (dropping mismatched columns) and save the combined csv file to the required location. I want to use command similar to (data = Open(Pick File("Choose the File", x)), x is the directory), to pick the files.

 

Thanks

11 REPLIES 11
txnelson
Super User

Re: Concatenate Multiple tables with different column order

Here is a rework of the script, selecting all of the files to be concatenated in one selection, and then running a little script to put them together, keeping only columns that find a match.  I have built it to your specifications, however it could easily be changed to allow the concatenation of any number of files

Names Default To Here( 1 );

inputDirectory = "C:\";

// Select the files to be read in......this can be any number of files
filePath = Pick File(
	"Choose the File to be Concatenated to:",
	inputDirectory,
	{"CSV Files|csv;", "All Files|*"},
	1,
	0,
	""
);
// Open the first file
dtMain = Open( filePath, invisible );


files = Pick File(
	"Choose the Additional 4 Files to Read In:",
	inputDirectory,
	{"CSV Files|csv;", "All Files|*"},
	1,
	0,
	"",
	multiple
);

// Validate that files have been read in
If( Try( N Items( files ), 0 ) == 4,
	Dialog( "Incorrect number of files were read in."
		"Only 4 additional files can be read in");
	Throw( "Incorrect number of files were read in" );
);

// Read in the remaining files and process them
For( i = 2, i <= N Items( Files ), i++,
	Try( dtConCat = Open( Files[i], invisible ) );
	
	mainColList = dtMain << Get Column Names( string );
	
	conCatColList = dtConCat << Get Column Names( string );
	
	// Delete the columns in the dtConCat table that do not exist in the dtMain table
	delList = {};
	For( k = 1, k <= N Items( conCatColList ), k++,
		If( N Rows( Loc( mainColList, conCatColList[k] ) ) == 0,
			Insert Into( delList, conCatColList[k] )
		)
	);
	If( N Items( delList ) > 0,
		dtConCat << delete columns( delList )
	);

	// Delete the columns in the dtMain table that were not found in the dtConCat table
	delList = {};
	For( k = 1, k <= N Items( mainColList ),
		k++,
		If( N Rows( Loc( conCatColList, mainColList[k] ) ) == 0,
			Insert Into( delList, mainColList[k] )
		)
	);
	If( N Items( delList ) > 0,
		dtMain << delete columns( delList )
	);
	
	// If any columns still remain in the two tables, then concatenate them together
	If( N Cols( dtMain ) > 0 & N Cols( dtConCat ) > 0,
		dtMain = dtMain << Concatenate( dtConCat, Append to First Table )
	);
	
	// Close the no longer needed file
	Close( dtConCat, nosave );
	
);

// Make the main file visible
dtMain << show window( 1 );

 

Jim
Rajat
Level IV

Re: Concatenate Multiple tables with different column order

Thanks @txnelson
You are always a great help.