cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Choose Language Hide Translation Bar
giaMSU
Level II

Create subsets from huge table (3,632,135 rows)

Hi all, 

I have a big dataset with more than 3 million rows. The table contains 1105 ID (0-1104). Each ID has 3287 rows. So, I want to create 3287 subsets, and each subset contains 1105 ID lines.

Could you please help me to find the solution?

I attached the part of the dataset. 

1 REPLY 1
txnelson
Super User

Re: Create subsets from huge table (3,632,135 rows)

If you have lots of memory, all you have to do is to select the Subset By check box, and to select id

subset.PNG

Another method would be to save the subsets as they are created.   This will allow the method to run without having to use up a lot of memory.

Names Default To Here( 1 );
dt = Current Data Table();
For( i = 0, i <= 1104, i++,
	dt << select where( :ID == i );
	dt1 = Data Table( "2017_18Year (1)" ) << Subset(
		Output Table( "ID = " || Char( i ) ),
		Selected Rows( 1 ),
		Selected columns( 0 )
	);
	Close( dt1, save( "path to the folder to save to" || "ID = " || Char( i ) || ".jmp" ) );
);
Jim