My user having issue to query about 2gb csv file took 2hrs, is there any recommeneded way or have any data to show how big file took how many minutes eg:1GB CSV -- 30min
I don't have performance data, I will let others share that.
Please know, though, that you can often speed up the text import process if you are fairly confident of the layout of the data ahead of time. If you open the file using the "Text Import Preferences" rather than "Best Guess", it will speed up import. Also, in the Preferences panel for Text Data Files, if you select "Scan for 5 seconds" in the "When determining column types" it can help if you don't have thousands of columns.
Yes we did ran through the 5secs options didn't only the scan went fast but import remain slow.
But I cant see the option "Text Import Preferences" in this JMP Pro 13.
You can use the source script from a previous import to import the file (or similar file) again and it will be faster than having JMP re-discover everything. The source script knows how many columns to expect and what the data types are.
A 2GB text file is likely to be even larger in memory; it is possible an 8GB machine is going to be using the disk, a lot, for paging memory.
There is a significant performance improvement in JMP 13.1 for MAC. If you are using JMP on MAC this may be what you need.
This should run in under 15 minutes on a machine with sufficient memory.
start=tickseconds(); data = repeat("1,2,3,4,5\!n",2e8); stop=tickseconds(); show(stop-start,length(data)); start=tickseconds(); file = savetextfile("$temp\deleteme.csv",data); stop=tickseconds(); show(stop-start,filesize(file));
data = 0; // release 2GB before opening file
start=tickseconds(); dt = Open( file, columns( New Column( "a", Numeric, "Continuous", Format( "Best", 12 ) ), New Column( "b", Numeric, "Continuous", Format( "Best", 12 ) ), New Column( "c", Numeric, "Continuous", Format( "Best", 12 ) ), New Column( "d", Numeric, "Continuous", Format( "Best", 12 ) ), New Column( "e", Numeric, "Continuous", Format( "Best", 12 ) ) ), Import Settings( End Of Line( CRLF, CR, LF ), End Of Field( Comma, CSV( 1 ) ), Strip Quotes( 0 ), Use Apostrophe as Quotation Mark( 0 ), Use Regional Settings( 0 ), Scan Whole File( 0 ), Treat empty columns as numeric( 0 ), CompressNumericColumns( 0 ), CompressCharacterColumns( 0 ), CompressAllowListCheck( 0 ), Labels( 0 ), Column Names Start( 0 ), Data Starts( 1 ), Lines To Read( "All" ), Year Rule( "20xx" ) ) ); stop=tickseconds(); show(stop-start,nrows(dt));
Here below the pc specs, the info you have provided is applicable only for MAC?
If is it for Windows as well required what platform to use for this script.
Please advice as we are new to JMP and assisting user reporting issue. thank you.
Since we are new to JMP i cant identify, have requested my user to check, thak you.
Will update the outcome.
The hardware list looks fine; the script above should run on JMP on either Windows or Mac. You should contact our technical support staff if the problem continues. They will need:
If the CSV contains characters that are not Unicode, fields that are quoted incorrectly, or other odd things, please include that in the CSV sample.
Since its confidential data may not able to share the file.
Is there any recommened clock speed or increase the clock speed performance from JMP application.