BookmarkSubscribeSubscribe to RSS Feed

Community Trekker


Dec 27, 2016

Selecting specific columns and concatenating

Hi All,


Right now, I open a bunch of data tables and put it in a list called datalist.


For(i = 2 ,i <= N Items(datalist) , i++,
fdt = datalist[1];
fdt<< Concatenate(Data Table(datalist[i]),append to first table(1));
close(Data Table(datalist[i]),nosave);
 fdt << set name( "Final_Table");



This concatenates everything and it takes a lot of time when I try to concatenate 100's of tables together.

I have index of a column and I know exactly what I want to concatenate.

Right now, I need to concatenate first 7 columns and index number(say param_index=10) specified( column 10) for all the tables.

So the final table will just have 8 columns with all data tables.


I have been stuck on this for a while. Any idea how to do this?




Is there a way to subgroup the columns (8 columns) and save it in the list in the first place, that it can be concatenated using the above script?




Thank you so much.



Community Trekker


Sep 15, 2014

Re: Selecting specific columns and concatenating

Hello @vishwasanj
      Try this 


Clear Globals() ; Clear Log(); 

// Open Sample Data 
dt = Open( "$SAMPLE_DATA/Air" );

// Generate Some random tables 
nSteps = N Rows(dt)/100; 

// Define Columns you want to use  
DesiredCols = {"Airline","Flight Number","Flight ID","Event"};
for( i = 1 , i < = N Row(dt), i++,
		dt << Select Randomly( 0.1 );
		dt1 = dt << Subset(Selected Rows(1), Columns(DesiredCols)); 
		dt1 << Show Window(0); 

This shows how to select only the columns you want. Once you have this - you can use the code you have to concatenate only the columns you needed. 





Super User


Jun 22, 2012

Re: Selecting specific columns and concatenating

I would suggest that as you open up each table, that you delete the unneeded columns.  I might also make all but the first table "Private".

You may also see if doing the concatenation of all of the tables in one step rather than looping through each table and concatenating it, and then moving on to the next table.  This may prove to be faster.




Jun 23, 2011

Re: Selecting specific columns and concatenating

Building on Jim's thoughts, perhaps the 'Select Columns' option for 'Open()' might also be useful.