topic Re: Max File Size in JMP on 64bit? in Discussions

Max File Size in JMP on 64bit?

paul_nistler — Mon, 14 Mar 2016 20:32:38 GMT

Hi,

I had a laptop with 4GB ram and could open sas7bdat files close to ~1GB within JMP. Greedy me wanted to play with bigger datasets (bigger is always better, right?) - so I upgraded to a workstation with 64GB ram. I have JMP Pro 10 (64bit) installed - same as on the laptop - but I cannot open any larger files than I could on my laptop. What gives? I keep getting "invalid memory access" errors when I try to go to larger files. Any suggestions on how to debug?

Thanks,

Paul

Re: Max File Size in JMP on 64bit?

Craige_Hales — Tue, 15 Mar 2016 13:25:24 GMT

The last several releases of JMP have all improved JMP's ability to handle tables with more than 2,000,000,000 rows (2^31 was a limitation). If your data is bigger than 2 billion rows, JMP 10 may be the problem, try JMP 12.

If not 2 billion rows...

Is this on windows or mac? If windows, is it a 64-bit windows? is the error you get a crash, and were you getting the same crash before? Is the file you are opening a .JMP or .CSV or ??? Is the error message in the JMP log or a crash dialog from the operating system?

Invalid Memory Access probably means an error in JMP, but more information will get us closer to an answer.

You *should* be able to open some pretty large files on a 64GB ram 64-bit OS with 64-bit JMP 12.

Craige

Re: Max File Size in JMP on 64bit?

paul_nistler — Thu, 17 Mar 2016 20:03:25 GMT

So, I tried opening a different (3.4gb) file that I know is underneath the limit of 2 billion rows and the error is "JMP is unable to allocate enough memory to open this data set". I'm surprised though given my system specs. I am running on 64bit windows 7 pro. The file is a *.csv type.

Re: Max File Size in JMP on 64bit?

Craige_Hales — Wed, 19 Oct 2016 02:09:27 GMT

roughly, numbers are 8 bytes each, strings are about 32 bytes each, minimum. For your machine and a guess about your number of rows: 64GB/2B rows is 32 bytes per row. And, the row state information in the data table (selected, hidden, color, symbol, etc) uses 8 bytes per row. 3 columns of numbers, maybe.

It is possible to use short numeric integers of 1, 2, or 4 bytes in a data table; you'll have to enable them in preferences.

You probably have a page file of 64GB as well, which doubles the address space, but your data tables will be very sluggish once you start paging GBs of data.

Strings longer than a threshold of about 20 characters use additional memory.

JMP 12 has preferences that may help for importing the CSV. An extra pass is made to determine which columns can use the 1- 2- or 4-byte integer columns and which character columns have few enough distinct values for list check.