cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Check out the JMP® Marketplace featured Capability Explorer add-in
Choose Language Hide Translation Bar
shasheminassab
shasheminassab
Level IV

Import only a part of a large csv file to JMP

Created: Sep 12, 2020 04:40 PM  |  Last Modified: Jun 10, 2023 4:19 PM (1246 views)

I have a large csv file containing > 50million rows of data for all 50 states of the US. How can I import only part of the data to JMP (e.g., data for California only)?

One way to do it is to open the cvs file in JMP and then filter out the data for California, but this takes forever. So I was wondering if there is a way to filter the data *before* importing it to JMP? I guess this can be done with the query builder (right?) but can this be done on a csv file? Please advise.

0 Kudos
1 REPLY 1
Craige_Hales
Super User Craige_Hales
Super User

Re: Import only a part of a large csv file to JMP

Sep 12, 2020 08:25 PM (1224 views)  |  Posted in reply to message from shasheminassab 09-12-2020

Give multiple file import a try if you are seeing the csv import being slow. It won't solve the filtering issue but it might be quite a bit faster, even with just one file. The filtering should be quite fast once the data is loaded.

 

Or, keep the source script from the CSV import and use the script next time. CSV import (unlike multiple file import) makes an extra pass over the data to discover which columns are character and numeric. With the source script, it doesn't need to make the extra pass.

 

Also, if the data table is too big to fit in memory (8GB? 16GB?) then it is going to be slow and increasing memory might be the best choice. If you have a disk light, keep an eye on it, or use task manager, etc to see if the disk is 100% busy.

Craige
0 Kudos