topic Re: Memory usage when creating many tables in Discussions

Memory usage when creating many tables

Niko_Porjo — Sat, 10 Jun 2023 23:13:50 GMT

When creating many tables and then closing them JMP seems to keep the tables or some remnants in memory. The memory is sometimes released after a while. I ran into this when simulating a measurement situation with noise sources. JMP din't crash but after RAM ran out windows started swapping to HDD.

Below is the simplest example where I was able to repeat the problem.

Question: is this expected behaviour and/or did I do something funny?

Note to anyone encountering the same: uncommenting the Wait(0); removes the problem, but seems to have a small (~10 %) performance penalty at least in the case where this was a problem.

For (i = 1, i <= 100000, i++,
	dt = New Table("Name of the table",
		New Column("Name of the column", Set Values(1::20000)),
	);
	// Do stuff
	Close(dt, "No Save");	
	//Wait(0);
);

JMP 14.0.0 64-bit, Win 10

Re: Memory usage when creating many tables

uday_guntupalli — Wed, 09 Jan 2019 14:11:43 GMT

@Niko_Porjo,
Try to include an "invisible" or a "private" flag when you open so many data tables. Making Data Tables "invisible" will help you see some improvement and making it "private" will help you see significant improvements.

Re: Memory usage when creating many tables

Mark_Bailey — Wed, 09 Jan 2019 14:13:05 GMT

Why create a new table in each iteration? You could create a data table before iterating and the update the contents in each iteration.

Re: Memory usage when creating many tables

Niko_Porjo — Thu, 10 Jan 2019 09:23:15 GMT

Mostly because I used parts from scripts I had already written before and didn't want to change them. In many cases I prefer slow scripts over working on the script to make it faster.

In some cases the data tables get their data from matrices:

dt = As Table(matrixWithData, private);

Is there a way to easily put the data into an existing data table? That would make the changes relatively simple.

Re: Memory usage when creating many tables

ian_jmp — Thu, 10 Jan 2019 10:08:41 GMT

Regarding the last question, you could use data table subscripting:

NamesDefaultToHere(1);

// Make a starting table to update later
mat = J(10, 5, RandomNormal());
dt = AsTable(mat);
dt << setName("My Table");

// Update the last four columns of dt (using data table subscripting) with new values
Wait(3);
mat = J(10, 4, RandomNormal());
dt[1::10, 2::5] = mat;