topic Importing large dataset from SAS in Discussions

Importing large dataset from SAS

farinoushsharif — Thu, 29 Sep 2016 01:55:14 GMT

Well I have three large SAS datasets (each almost 6 GB), and I need to clean the datasets, somehow mix all of them, and take a sample of all.

my main problem is I even could not open them in JMP Pro. I think dividing each of them into smaller files might not be a good idea, as I have to take a sample of all.

1. So, how can I import them into JMP Pro 12? Is it possible on a 64 bit OS without dividing the dataset?

2. If I have to divide it, and there is no other way, how can I do that as I do not have the SAS and JMP installed on a same system?

Thank you!

Re: Importing large dataset from SAS

Craige_Hales — Wed, 19 Oct 2016 04:13:10 GMT

How many rows and columns? How did it fail, out of memory? The JMP log window may have a message. What file extension on the SAS datasets?

I'd guess a 32GB machine should handle the 6GB files one at a time, and probably all together, without paging too much. Might be a memory or speed issue with a second copy of an 18GB file though.

If you had a CSV file, text import offers some subsetting choices.

I think there are similar options in other import tools in JMP.

Re: Importing large dataset from SAS

Eric_Hill — Thu, 29 Sep 2016 12:12:26 GMT

Hey, farinoushsharifi,

I agree with Craige, a 64-bit Windows box with a minimum of 16 GB of physical RAM, would be necessary to successfullyl open a 6GB SAS data set in JMP.

Have you considered doing the cleaning in SAS, say with some DATA step code? JMP does have some great data cleaning tools, don't get me wrong, but having enough memory to open the data sets successfully is just the first problem. It doesn't guarantee a pleasant experience in JMP working with such large tables.

Is the machine that has SAS installed a server, allowing you to connect to it remotely? JMP has good tools for connecting to remote SAS servers - File > SAS > Server Connections is the place to start. You'd need to know the machine name and port number of the SAS server (or, if you also have a SAS Metadata Server, that makes connecting even easier).

HTH,

Eric