Subscribe Bookmark RSS Feed

Concatenating large number of files

Hi Guys,

I am trying to concatenate several files in a folder as below:

7 REPLIES
mattf

Community Trekker

Joined:

Jun 23, 2011

See:
Jump into JMP Scripting
By: Wendy Murphrey and Rosemary Lucas
// http://www.sas.com/apps/pubscat/bookdetails.jsp?catid=1&pc=61733
Page 122, Soultion 5.16

Set Current Directory( "C:\temp2" );
myFiles = Files In Directory( "C:\temp2" );
For( i = 1, i <= N Items( myFiles ), i++,
/* If the file is the first in the list, open it. Otherwise, open the table, concatenate with the main table, and close the table just opened. */
If( i == 1,
mainDt = Open( "C:\temp2\" || myFiles ),
dt = Open( "C:\temp2\" || myFiles );
mainDt = mainDt << Concatenate( dt, Append to First Table );
Close( dt, NoSave );
Wait( 0 );
)
);
/* Give the final table a name. */
mainDt << Set Name( "Concatenated Files" );


Best regards,
-Matt
mattf

Community Trekker

Joined:

Jun 23, 2011

See:
Jump into JMP Scripting
By: Wendy Murphrey and Rosemary Lucas
// http://www.sas.com/apps/pubscat/bookdetails.jsp?catid=1&pc=61733
Page 122, Soultion 5.16

Set Current Directory( "C:\temp2" );
myFiles = Files In Directory( "C:\temp2" );
For( i = 1, i <= N Items( myFiles ), i++,
/* If the file is the first in the list, open it. Otherwise, open the table, concatenate with the main table, and close the table just opened. */
If( i == 1,
mainDt = Open( "C:\temp2\" || myFiles ),
dt = Open( "C:\temp2\" || myFiles );
mainDt = mainDt << Concatenate( dt, Append to First Table );
Close( dt, NoSave );
Wait( 0 );
)
);
/* Give the final table a name. */
mainDt << Set Name( "Concatenated Files" );
Thanks. Can you please use the following to post your code?

mpb

Super User

Joined:

Jun 23, 2011

You may find this trick useful:

After logging in, "reply" to the posting and "Quote Original". You will the have the complete code which you can copy / paste into a JMP script window. Then remove the first 2 characters on each line ("> ") to get the code as it was intended. Then cancel the reply.
Thanks Matt, that worked. I will take a look at the book.

Thanks, mpb for the tip.
mattf

Community Trekker

Joined:

Jun 23, 2011

 
jorgeramosdealm

Community Trekker

Joined:

Feb 20, 2012

I made a variation of that code that incorporates the Source table name for each file. In this case it takes all the txt files but you can also do it with csv or other format.


mypath ="C:\temp\";


Set Current Directory( mypath ); 


y = Files In Directory( mypath ); 


For( i = N Items( y ), i > 0, i--,


  If( Ends With( y[i], "txt" ),


  ,


  Remove From( y, i )


  )


);


n = N Items( y );


For( i = 1, i <= N Items( y ), i++,


  sname=left(y[i],length(y[i])-4);


/* If the file is the first in the list, open it. Otherwise, open the table, concatenate with the main table, and close the table just opened. */ 


    If( i == 1,


        mainDt = Open( mypath || y[i] );


        mainDt << New Column ("source", character, set each value ( sname )),


        dt = Open( mypath || y[i] ) ;


        dt << New Column ("source", character, set each value ( sname ));


        mainDt = mainDt << Concatenate( dt, Append to First Table );


        Close( dt, NoSave );


        Wait( 0 );


    )


  );   


/* Give the final table a name. */ 


mainDt<< Set Name("Concatenated Files");