cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Browse apps to extend the software in the new JMP Marketplace
Choose Language Hide Translation Bar
monir
Level I

Save memory while Concatenating vertically in loop

Greetings!

I am trying to Concatenate query results, from several tables in a directory, into one single table. I am using loop to read/query the tables and concatenate the results. The result table is fine but each intermediate concatenated tables keeps opening consuming the memory. 

 

Thank you for any suggestions to to avoid opening each intermediate result tables but just updating the data frame which can be displayed as the result table only after the loop is completed.

I am using JMP 17 Pro. 

 

names default to here(1);
dirLoc=Pick Directory("Select a directory", "myDirectory\"); fileList = Files In Directory(dirLoc); dt1 = open(dirLoc || fileList[1]); dt1 << Select Where( :Amount >= 500 ); x=dt1 << Subset ( Output Table Name("T1") ); Close(dt1, NoSave); For(i = 2, i <= N Items( fileList ), i++, dt = open(dirLoc || fileList[i] ); dt << Select Where( :Amount >= 500 ); y = dt << Subset ( Output Table Name("Temp")); x = x << Concatenate( y); Close(dt, NoSave); Close(y, NoSave); ); result = x<<Subset ( Output Table Name("Result"));

 

3 REPLIES 3
ErraticAttack
Level VI

Re: Save memory while Concatenating vertically in loop

You need to append the concatenated tables.  By default JMP concatenates tables into a new table.

 

names default to here(1);dirLoc=Pick Directory("Select a directory", "myDirectory\");
fileList = Files In Directory(dirLoc);
dt1 = open(dirLoc || fileList[1]);
dt1 << Select Where( :Amount >= 500 );
x=dt1 << Subset ( Output Table Name("T1") );
Close(dt1, NoSave);
For(i = 2, i <= N Items( fileList ), i++,
	dt = open(dirLoc || fileList[i] );
	dt << Select Where( :Amount >= 500 );
	y = dt << Subset ( Output Table Name("Temp"));
	x << Concatenate( y, "Append to First Table" );
	Close(dt, NoSave);
	Close(y, NoSave);
);
result = x<<Subset ( Output Table Name("Result"));

  

Jordan
monir
Level I

Re: Save memory while Concatenating vertically in loop

Thanks for your suggestion to append instead of concatenation. Actually I tried appending as well as shown below. The problem is, it pops up all the intermediate table every time it is appended in the loop.

 

x << Concatenate(x,y);

 Any thought?

ErraticAttack
Level VI

Re: Save memory while Concatenating vertically in loop

There might be some misunderstanding here.  Doing x << Concatenate( x, y ) will not update x, it will create a new table.

 

To update x with the values of y, use the literal expression x << Concatenate( y, "Append to First Table" )

 

Also, think about doing all this as private tables to conserve resources, then at the end show the table, like this:

 

Names Default To Here( 1 );
dirLoc = Pick Directory( "Select a directory", "myDirectory\" );
fileList = Files In Directory( dirLoc );
dt1 = Open( dirLoc || fileList[1], Private );
dt1 << Select Where( :Amount >= 500 );
x = dt1 << Subset( Output Table Name( "T1" ), Private );
Close( dt1, NoSave );

For( i = 2, i <= N Items( fileList ), i++,
	dt = Open( dirLoc || fileList[i], Private );
	dt << Select Where( :Amount >= 500 );
	y = dt << Subset( Output Table Name( "Temp" ), Private );
	x << Concatenate( y, "Append to First Table" );
	Close( dt, NoSave );
	Close( y, NoSave );
);

x << Set Name( "Result" );
x << New Data View;

 

Jordan