topic Save memory while Concatenating vertically in loop in Discussions

Save memory while Concatenating vertically in loop

monir — Fri, 26 Jan 2024 17:31:53 GMT

Greetings!

I am trying to Concatenate query results, from several tables in a directory, into one single table. I am using loop to read/query the tables and concatenate the results. The result table is fine but each intermediate concatenated tables keeps opening consuming the memory.

Thank you for any suggestions to to avoid opening each intermediate result tables but just updating the data frame which can be displayed as the result table only after the loop is completed.

I am using JMP 17 Pro.

names default to here(1);
dirLoc=Pick Directory("Select a directory", "myDirectory\");
fileList = Files In Directory(dirLoc);
dt1 = open(dirLoc || fileList[1]);
dt1 << Select Where( :Amount >= 500 );
x=dt1 << Subset ( Output Table Name("T1") );
Close(dt1, NoSave);
For(i = 2, i <= N Items( fileList ), i++,
	dt = open(dirLoc || fileList[i] );
	dt << Select Where( :Amount >= 500 );
	y = dt << Subset ( Output Table Name("Temp"));
	x = x << Concatenate( y);
	Close(dt, NoSave);
	Close(y, NoSave);
);
result = x<<Subset ( Output Table Name("Result"));

Re: Save memory while Concatenating vertically in loop

ErraticAttack — Fri, 26 Jan 2024 20:02:28 GMT

You need to append the concatenated tables. By default JMP concatenates tables into a new table.

names default to here(1);dirLoc=Pick Directory("Select a directory", "myDirectory\");
fileList = Files In Directory(dirLoc);
dt1 = open(dirLoc || fileList[1]);
dt1 << Select Where( :Amount >= 500 );
x=dt1 << Subset ( Output Table Name("T1") );
Close(dt1, NoSave);
For(i = 2, i <= N Items( fileList ), i++,
	dt = open(dirLoc || fileList[i] );
	dt << Select Where( :Amount >= 500 );
	y = dt << Subset ( Output Table Name("Temp"));
	x << Concatenate( y, "Append to First Table" );
	Close(dt, NoSave);
	Close(y, NoSave);
);
result = x<<Subset ( Output Table Name("Result"));

Re: Save memory while Concatenating vertically in loop

monir — Fri, 26 Jan 2024 20:20:54 GMT

Thanks for your suggestion to append instead of concatenation. Actually I tried appending as well as shown below. The problem is, it pops up all the intermediate table every time it is appended in the loop.

x << Concatenate(x,y);

Any thought?

Re: Save memory while Concatenating vertically in loop

ErraticAttack — Fri, 26 Jan 2024 22:24:43 GMT

There might be some misunderstanding here. Doing x << Concatenate( x, y ) will not update x, it will create a new table.

To update x with the values of y, use the literal expression x << Concatenate( y, "Append to First Table" )

Also, think about doing all this as private tables to conserve resources, then at the end show the table, like this:

Names Default To Here( 1 );
dirLoc = Pick Directory( "Select a directory", "myDirectory\" );
fileList = Files In Directory( dirLoc );
dt1 = Open( dirLoc || fileList[1], Private );
dt1 << Select Where( :Amount >= 500 );
x = dt1 << Subset( Output Table Name( "T1" ), Private );
Close( dt1, NoSave );

For( i = 2, i <= N Items( fileList ), i++,
	dt = Open( dirLoc || fileList[i], Private );
	dt << Select Where( :Amount >= 500 );
	y = dt << Subset( Output Table Name( "Temp" ), Private );
	x << Concatenate( y, "Append to First Table" );
	Close( dt, NoSave );
	Close( y, NoSave );
);

x << Set Name( "Result" );
x << New Data View;