topic Re: Java API for creating JMP files? in Discussions

Java API for creating JMP files?

bwadsworth — Thu, 22 Oct 2015 19:20:15 GMT

Is there a way to create JMP files in Java rather than creating CSV and then requiring users to do the conversion themselves using a JSL script?

Re: Java API for creating JMP files?

Jeff_Perkinson — Wed, 19 Oct 2016 01:02:00 GMT

There's no Java API but the JSL New Table() syntax is the best way to create a file that will be a JMP data table when opened.

You can put a //! on the first line of your script and it will run automatically when the file is opened. In this way, it will "look" like a JMP data table when opened.

You can get a template for the the New Table() syntax to create a data table by examining the table script for an existing data table. You'll find the table script in the hotspot menu in the upper left of the data table.

Attached here is an example JSL file to show this. When you open the file it will run automatically and create a data table.

-Jeff

Re: Java API for creating JMP files?

bwadsworth — Wed, 28 Oct 2015 18:41:45 GMT

Hi Jeff,

Thanks for offering this solution. So, the script would be a template and we would be dynamically adding the needed columns? Our data files can be very large. We can have a large number of columns and rows. What are the limits of the Set Values() method?

New Column(

["en" => "name", "ja" => "??", "x-id" => "S_name_Col", "zh-CN" => "??"],

Character,

"Nominal",

Set Property(

"Notes",

["en" => "...usually used as a label variable in plots",

"ja" => "?????????????????",

"x-id" => "S_usually_used_as_a_label_variable_Not",

"zh-CN" => "...???????????"]

Set Values(

{"KATIE", "LOUISE", "JANE", "JACLYN", "LILLIE", "TIM", "JAMES", "ROBERT",

"BARBARA", "ALICE", "SUSAN", "JOHN", "JOE", "MICHAEL", "DAVID", "JUDY",

"ELIZABETH", "LESLIE", "CAROL", "PATTY", "FREDERICK", "ALFRED", "HENRY",

"LEWIS", "EDWARD", "CHRIS", "JEFFREY", "MARY", "AMY", "ROBERT",

"WILLIAM", "CLAY", "MARK", "DANNY", "MARTHA", "MARION", "PHILLIP",

"LINDA", "KIRK", "LAWRENCE"}

)

Re: Java API for creating JMP files?

Craige_Hales — Wed, 19 Oct 2016 01:03:39 GMT

Try this to see how it will scale for you. This JSL will make a table of random data, some numbers, some strings. You should probably make it twice as big as you expect your data to be... The script then creates the script Jeff described above and re-imports the table. Two times are presented; the first is the creation time and the second is the re-import time. Below you can see 1,000,000 rows of 30 variables makes a 370MB text string that re-imports in 45 seconds. At the end, the compare tables platform verifies the result. For problems of this size you will need a 64 bit JMP most likely; a 370MB string is going to push the limits of a 32 bit JMP on windows.

start = Tick Seconds();
dt = New Table( "original", Add Rows( 1000000 ), );
For( i = 1, i <= 30, i++,
  If( Random Integer( 1, 2 ) == 1,
  x = dt << New Column( "c", Character, Formula( Substr( "abcdefghijklmnopqrstuvwxyz", Random Integer( 1, 20 ), Random Integer( 1, 20 ) ) ) ),
  x = dt << New Column( "c", Numeric, Formula( Random Integer( 1, 999999999 ) ) )
  );
  dt << runformulas;
  x << deleteformula;
);
stop = Tick Seconds();
Show( stop - start );
Wait( 1 );
text = Char( dt << getscript );
Show( Length( text ) );
Wait( 1 );
start = Tick Seconds();
dt2 = Eval( Parse( text ) );
stop = Tick Seconds();
Show( stop - start );
dt << Compare Data Tables( Compare with( dt2 ), Compare Data );

stop - start = 12.0166666666046;

Length(text) = 369241917;

stop - start = 45.1166666664649;

Compare Data Tables[]

Re: Java API for creating JMP files?

nascif_jmp — Tue, 10 Nov 2015 15:28:20 GMT

Hi bwadsworth,

You could use a binary data format that both Java and JMP can handle - for example, a SQLite table.

I made an experiment on Windows, using JMP13 and a 160MB CSV file (POD - Product Open Data - gs1_gcp.csv) with a mixture of strings and floats.

JMP can parse and load this CSV file in 37 seconds.

But it took only 20 seconds to load an equivalent SQLite database using JMP's Query Builder and the SQLite ODBC driver - or 54% of the original time.

Of course, these gains are nothing compared with the time to load a JMP table directly - in this case, just 4 seconds!

But if you want to output the data from Java directly into a format that doesn't require manual conversion, this might be a good approach.

Re: Java API for creating JMP files?

bwadsworth — Tue, 10 Nov 2015 17:13:11 GMT

Thanks for the suggestion. I’ll check into this approach.

Regards,

Barry Wadsworth

 Apple Inc., MQM Ops Presentation & Access

(m) 916-208-9394

bwadsworth@apple.com

Re: Java API for creating JMP files?

nascif_jmp — Tue, 10 Nov 2015 20:59:28 GMT

You might also want to try this script that Craige@JMP graciously provided.

It illustrates how to explicitly list the table schema when reading a CSV file, cutting the load time in half - about the same gains of the SQLite loading approach.