Why is Tables/Concatenate so slow?

[#1]

Has this ever happened to you?
Sometimes it takes a very long time for Tables/Concatenate to finish merging two tables.

More than a minute for just a few columns and some 100k rows.

What could be the reason?

[#2]

I was wondering if it's because of the character columns - or because of the new compact columns (to compute the logic in the background).
Is it much faster to merge 2x the same table - instead of merging to "different" tables?

no - no - no

dt1 = new table("test",<< add rows(500000));
for(I=1, i<=20, i++,
dt1 << new column ("x", Character, set each value(Hex(random Integer(1000000))));
);
for(I=1, i<=20, i++,
dt1 << new column ("y", Character,Compact set each value(Hex(random Integer(1000))));
);

dt2 = dt1 << Subset( All rows, Selected columns only( 0 ) );

t0= hptime();
dt1 << Concatenate(	dt1);

time=(hptime()-t0)/1000000;
print("concat same file: ", time);

t0= hptime();
dt1 << Concatenate( dt2);

time=(hptime()-t0)/1000000;
print("concat 2 files: ", time);

[#3]

You may have noticed the FILES "scripts" - they look quite innocent:

but concerning speed, they can be devastating:

dt1 = new table("test",<< add rows(1), new column ("x",  set each value(1)));

t0=hptime();

Nmax=200;

for(i=1, i<100,i++,
filesN = As list(Index(1,Nmax)`);
filelist = transform each({i},filesN,Eval List({"myfile"||Char(i)||".txt", 1, 1}));
Eval (Eval Expr(
dt1 << New Table Script(	"Files",Expr(filelist)	)
)));
time=(hptime()-t0)/1000000;
print("add files info: ", time);

dt2 = dt1 << Subset( All rows, Selected columns only( 0 ) );

t0= hptime();
dt1 << Concatenate( dt2);

time=(hptime()-t0)/1000000;
print("concat 2 files: ", time);

[#4]

So from time to time, either delete the Files scripts - or merge them into one script *)

This will speed up the Concatenate step by orders of magnitude.

Even more important: if you trigger the Update manually:
With the new Preview Tab, the strange "processing" has to be done EVERY time again and again:
1) when the Concatenate menu opens

2) when tables are added via Add

3) after clicking on OK

This video illustrates the problem - to make it not extremely boring, I sped it up  by reducing the for loop to 50:

Concatenate_slow.mp4

Video Player is loading.

Play Video

Play

Mute

Current Time 0:00

/

Duration 1:22

Loaded: 0%

0:00

Stream Type LIVE

Seek to live, currently behind liveLIVE

Remaining Time -1:22

1x

Playback Rate

Chapters

• Chapters

Descriptions

• descriptions off, selected

Captions

• captions settings, opens captions settings dialog
• captions off, selected

Audio Track

• en (Main), selected

Picture-in-PictureFullscreen

This is a modal window.

Beginning of dialog window. Escape will cancel and close the window.

Text

ColorWhiteBlackRedGreenBlueYellowMagentaCyanTransparencyOpaqueSemi-Transparent

Background

ColorBlackWhiteRedGreenBlueYellowMagentaCyanTransparencyOpaqueSemi-TransparentTransparent

Window

ColorBlackWhiteRedGreenBlueYellowMagentaCyanTransparencyTransparentSemi-TransparentOpaque

Font Size

50%75%100%125%150%175%200%300%400%

Text Edge Style

NoneRaisedDepressedUniformDropshadow

Font Family

Proportional Sans-SerifMonospace Sans-SerifProportional SerifMonospace SerifCasualScriptSmall Caps

Reset restore all settings to the default valuesDone

Close Modal Dialog

End of dialog window.

Close Modal Dialog

This is a modal window. This modal can be closed by pressing the Escape key or activating the close button.

*) almost surprising: the same content in the table scripts - but orders of magnitude faster:

dt1 = new table("test",<< add rows(1), new column ("x",  set each value(1)));

t0=hptime();

Nmax=200;

for(i=1, i<100,i++,
filesN = As list(Index(1,Nmax)`);
filelist = transform each({i},filesN,Eval List({"myfile"||Char(i)||".txt", 1, 1}));
Eval (Eval Expr(
dt1 << New Table Script(	"Files",Expr(filelist)	)
)));
time=(hptime()-t0)/1000000;
print("add files info: ", time);

//merge the table scripts 
merged Script = dt1 << get script();
dt1 << delete scripts(dt1 << Get Table Script Names);
Eval(eval Expr(dt1 << new  table script ("merged",Expr(Name Expr(merged Script))))); 

dt2 = dt1 << Subset( All rows, Selected columns only( 0 ) );

t0= hptime();
dt1 << Concatenate( dt2);

time=(hptime()-t0)/1000000;
print("concat 2 files - merged scripts: ", time);

vs before:

View more...

Concatenate_merged__notslow.mp4

Video Player is loading.

Play Video

Play

Mute

Current Time 0:00

/

Duration 0:18

Loaded: 0%

0:00

Stream Type LIVE

Seek to live, currently behind liveLIVE

Remaining Time -0:18

1x

Playback Rate

Chapters

• Chapters

Descriptions

• descriptions off, selected

Captions

• captions settings, opens captions settings dialog
• captions off, selected

Audio Track

• en (Main), selected

Picture-in-PictureFullscreen

This is a modal window.

Beginning of dialog window. Escape will cancel and close the window.

Text

ColorWhiteBlackRedGreenBlueYellowMagentaCyanTransparencyOpaqueSemi-Transparent

Background

ColorBlackWhiteRedGreenBlueYellowMagentaCyanTransparencyOpaqueSemi-TransparentTransparent

Window

ColorBlackWhiteRedGreenBlueYellowMagentaCyanTransparencyTransparentSemi-TransparentOpaque

Font Size

50%75%100%125%150%175%200%300%400%

Text Edge Style

NoneRaisedDepressedUniformDropshadow

Font Family

Proportional Sans-SerifMonospace Sans-SerifProportional SerifMonospace SerifCasualScriptSmall Caps

Reset restore all settings to the default valuesDone

Close Modal Dialog

End of dialog window.

Close Modal Dialog

This is a modal window. This modal can be closed by pressing the Escape key or activating the close button.

[#5]

according to JMP support, the bad performance is no bug, but as expected: TS-00173141

Comparing all the text in all the script in the tables being concatenated takes a long time.  It is a lot of information to compare.  It is not surprising that the concatenation is so much slower with the large number of table scripts your example tables have.