Does JSL have a file stream?

[#1]

I would like to output a large amount of text data from a JMP Script. I can use string concatenation to build up the output and then write it all at once with the Save Text File command. However, this is very inefficient and gets infeasible as the string grows.

Does JMP support file streams that would let me direct string output to a buffer as it is created?

[#2]

My suggestion would be to use the "Append" qualifier on the Save Text File.

This will allow you to populate the output file on an ongoing basis

[#3 · Accepted solution]

Another strategy for working around the very long string slowdown:  build a list of shorter strings and use ConcatItems( list ) to join them into a single string.  Specify the separator you need, which might be "" if you don't need a separator added between items. Prior to JMP 13 you may need to build the list carefully as well if it will have thousands of items...see Fast List.​  This example shows the build-it-in-reverse approach. 

list = {};

insert into(list, "line1", 1);

insert into(list, "line2", 1);

insert into(list, "line3", 1);

list = reverse(list);

ConcatItems( list, "\!n");

"line1

line2

line3"

If you have a choice, keep the strings in the 10,000 to 100,000 character neighborhood.  Too short and you'll (maybe) run into issues with a huge number of list items, each with its own overhead.  Too long and you'll still have the slow concatenation issue (which is an N^2 issue, so managing the length of the strings this way will really help).

But for all that, JSL strings are limited to 2^31 (2 GB) characters.  I think Jim's append idea will work for more than 2GB.  You'll still want to append reasonably large blocks to the file because each save+append has overhead opening/writing/closing.  This example is writing 500,000,000 character blocks, until the disk filled up. 

txt = Repeat( "a", 5e8 );

Save Text File( "$temp\deleteme.txt", txt, Mode( "replace" ) );

Show( File Size( "$temp\deleteme.txt" ) );

For( i = 1, i <= 20, i++,

    Save Text File( "$temp\deleteme.txt", txt, Mode( "append" ) );

    Show( File Size( "$temp\deleteme.txt" ) );

);

File Size("$temp\deleteme.txt") = 500000000;

File Size("$temp\deleteme.txt") = 1000000000;

File Size("$temp\deleteme.txt") = 1500000000;

File Size("$temp\deleteme.txt") = 2000000000;

File Size("$temp\deleteme.txt") = 2500000000;

File Size("$temp\deleteme.txt") = 3000000000;

File Size("$temp\deleteme.txt") = 3500000000;

File Size("$temp\deleteme.txt") = 4000000000;

File Size("$temp\deleteme.txt") = 4500000000;

File Size("$temp\deleteme.txt") = 5000000000;

File Size("$temp\deleteme.txt") = 5500000000;

Unable to write.

There is not enough space on the disk.

[#4]

Thanks for the detailed approach. Combining the use of a list for binning and using Save Text File with Append gives a good approximation of a file stream. With this approach, I need to manually code for the bin size and periodic append operations but with those implemented the performance is reasonable.