Create subsets from huge table (3,632,135 rows)

giaMSU — Sat, 10 Jun 2023 23:28:13 GMT

Hi all,

I have a big dataset with more than 3 million rows. The table contains 1105 ID (0-1104). Each ID has 3287 rows. So, I want to create 3287 subsets, and each subset contains 1105 ID lines.

Could you please help me to find the solution?

I attached the part of the dataset.

Re: Create subsets from huge table (3,632,135 rows)

txnelson — Fri, 02 Apr 2021 21:45:01 GMT

If you have lots of memory, all you have to do is to select the Subset By check box, and to select id

Another method would be to save the subsets as they are created. This will allow the method to run without having to use up a lot of memory.

Names Default To Here( 1 );
dt = Current Data Table();
For( i = 0, i <= 1104, i++,
	dt << select where( :ID == i );
	dt1 = Data Table( "2017_18Year (1)" ) << Subset(
		Output Table( "ID = " || Char( i ) ),
		Selected Rows( 1 ),
		Selected columns( 0 )
	);
	Close( dt1, save( "path to the folder to save to" || "ID = " || Char( i ) || ".jmp" ) );
);

topic Re: Create subsets from huge table (3,632,135 rows) in Discussions

Create subsets from huge table (3,632,135 rows)

Re: Create subsets from huge table (3,632,135 rows)