cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Browse apps to extend the software in the new JMP Marketplace
Choose Language Hide Translation Bar
hogi
Level XII

Why is Tables/Concatenate so slow?

Has this ever happened to you?
Sometimes it takes a very long time for Tables/Concatenate to finish merging two tables.

More than a minute for just a few columns and some 100k rows.

What could be the reason?

 

2 REPLIES 2
hogi
Level XII

Re: Why is Tables/Concatenate so slow?

I was wondering if it's because of the character columns - or because of the new compact columns (to compute the logic in the background).
Is it much faster to merge 2x the same table - instead of merging to "different" tables?

 

no - no - no

 

dt1 = new table("test",<< add rows(500000));
for(I=1, i<=20, i++,
dt1 << new column ("x", Character, set each value(Hex(random Integer(1000000))));
);
for(I=1, i<=20, i++,
dt1 << new column ("y", Character,Compact set each value(Hex(random Integer(1000))));
);

dt2 = dt1 << Subset( All rows, Selected columns only( 0 ) );

t0= hptime();
dt1 << Concatenate(	dt1);

time=(hptime()-t0)/1000000;
print("concat same file: ", time);

t0= hptime();
dt1 << Concatenate( dt2);

time=(hptime()-t0)/1000000;
print("concat 2 files: ", time);
hogi
Level XII

Re: Why is Tables/Concatenate so slow?

You may have noticed the FILES "scripts" - they look quite innocent:

hogi_0-1730845773110.png

 

but concerning speed, they can be devastating:

hogi_1-1730845938983.png

 

dt1 = new table("test",<< add rows(1), new column ("x",  set each value(1)));

t0=hptime();

Nmax=200;

for(i=1, i<100,i++,
filesN = As list(Index(1,Nmax)`);
filelist = transform each({i},filesN,Eval List({"myfile"||Char(i)||".txt", 1, 1}));
Eval (Eval Expr(
dt1 << New Table Script(	"Files",Expr(filelist)	)
)));
time=(hptime()-t0)/1000000;
print("add files info: ", time);

dt2 = dt1 << Subset( All rows, Selected columns only( 0 ) );

t0= hptime();
dt1 << Concatenate( dt2);

time=(hptime()-t0)/1000000;
print("concat 2 files: ", time);