cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Choose Language Hide Translation Bar
Rajat
Level IV

Concatenate Multiple tables with different column order

Team,

I have 5 csv files each having 200column (not each file have same number of columns). Each file has columns in different order. I want to write a scirpt which will let user to select all 5 files then combine all 5 files (dropping mismatched columns) and save the combined csv file to the required location. I want to use command similar to (data = Open(Pick File("Choose the File", x)), x is the directory), to pick the files.

 

Thanks

11 REPLIES 11
txnelson
Super User

Re: Concatenate Multiple tables with different column order

Here is a rework of the script, selecting all of the files to be concatenated in one selection, and then running a little script to put them together, keeping only columns that find a match.  I have built it to your specifications, however it could easily be changed to allow the concatenation of any number of files

Names Default To Here( 1 );

inputDirectory = "C:\";

// Select the files to be read in......this can be any number of files
filePath = Pick File(
	"Choose the File to be Concatenated to:",
	inputDirectory,
	{"CSV Files|csv;", "All Files|*"},
	1,
	0,
	""
);
// Open the first file
dtMain = Open( filePath, invisible );


files = Pick File(
	"Choose the Additional 4 Files to Read In:",
	inputDirectory,
	{"CSV Files|csv;", "All Files|*"},
	1,
	0,
	"",
	multiple
);

// Validate that files have been read in
If( Try( N Items( files ), 0 ) == 4,
	Dialog( "Incorrect number of files were read in."
		"Only 4 additional files can be read in");
	Throw( "Incorrect number of files were read in" );
);

// Read in the remaining files and process them
For( i = 2, i <= N Items( Files ), i++,
	Try( dtConCat = Open( Files[i], invisible ) );
	
	mainColList = dtMain << Get Column Names( string );
	
	conCatColList = dtConCat << Get Column Names( string );
	
	// Delete the columns in the dtConCat table that do not exist in the dtMain table
	delList = {};
	For( k = 1, k <= N Items( conCatColList ), k++,
		If( N Rows( Loc( mainColList, conCatColList[k] ) ) == 0,
			Insert Into( delList, conCatColList[k] )
		)
	);
	If( N Items( delList ) > 0,
		dtConCat << delete columns( delList )
	);

	// Delete the columns in the dtMain table that were not found in the dtConCat table
	delList = {};
	For( k = 1, k <= N Items( mainColList ),
		k++,
		If( N Rows( Loc( conCatColList, mainColList[k] ) ) == 0,
			Insert Into( delList, mainColList[k] )
		)
	);
	If( N Items( delList ) > 0,
		dtMain << delete columns( delList )
	);
	
	// If any columns still remain in the two tables, then concatenate them together
	If( N Cols( dtMain ) > 0 & N Cols( dtConCat ) > 0,
		dtMain = dtMain << Concatenate( dtConCat, Append to First Table )
	);
	
	// Close the no longer needed file
	Close( dtConCat, nosave );
	
);

// Make the main file visible
dtMain << show window( 1 );

 

Jim
Rajat
Level IV

Re: Concatenate Multiple tables with different column order

Thanks @txnelson
You are always a great help.