How to load a very large database

[#1]

I'm trying to load a very large database using JMP but I'm running out of memory. It contains almost 1k columns and millions of rows. The file size is almost 35 GB. Is there any way that I can open only selected columns to minimize memory usage? Any recommendations on how to open such a large file using JMP? Unfortunately, I don't have coding experience

[#2]

Using

    File=>Database=>Query Builder

you should be able to select only the columns you want, from the tables you want

I suspect, other Community Members may have additional suggestions.

[#3]

To add to Jim's post, Query Builder allows you to sample x% of the rows or choose the first (or random) x rows, as well.

[#4]

I have the data as .csv file. I couldn't open it using query builder

[#5]

Thanks, I have the data as .csv file. Is there a different way to open it? Or I need to upload the file to a specific server and then do query builder?

[#6]

You could also try Select Columns when you open the file

Script created by it

Open("$SAMPLE_DATA/Big Class.jmp", Select Columns("name", "age", "height"));

This will let you limit the column selection, but I'm not sure if it will work for you.

Open also has other options to limit the data, you can open first x rows, last x rows or random x rows

Names Default To Here(1);
dt = Open("$SAMPLE_DATA/CrimeData.jmp", First(10));
dt = Open("$SAMPLE_DATA/CrimeData.jmp", Random(10));
dt = Open("$SAMPLE_DATA/CrimeData.jmp", Last(10));

and instead of selecting specific columns, you can ignore specific columns

Names Default To Here(1);
dt = Open("$SAMPLE_DATA/Big Class.jmp", ignore columns("age"));

One of the Open items in scripting index has some of these examples

[#7]

I appreciate your effort in putting together this comprehensive explanation. The first option does not work unless the file is a JMP file. I've tried all other options and it's not executing the second part of the code. It only opens the file without selecting or ignoring the specific columns I want. I've tried the example you provided and it worked but when I choose the file I want, the select or ignore function does not work.

[#8]

You can edit the script that imports the file to omit columns and subset the data.

importText.mp4

Video Player is loading.

Play Video

Play

Mute

Current Time 0:00

/

Duration 1:13

Loaded: 0%

0:00

Stream Type LIVE

Seek to live, currently behind liveLIVE

Remaining Time -1:13

1x

Playback Rate

Chapters

• Chapters

Descriptions

• descriptions off, selected

Captions

• captions settings, opens captions settings dialog
• captions off, selected

Audio Track

• default, selected

Picture-in-PictureFullscreen

This is a modal window.

Beginning of dialog window. Escape will cancel and close the window.

Text

ColorWhiteBlackRedGreenBlueYellowMagentaCyanTransparencyOpaqueSemi-Transparent

Background

ColorBlackWhiteRedGreenBlueYellowMagentaCyanTransparencyOpaqueSemi-TransparentTransparent

Window

ColorBlackWhiteRedGreenBlueYellowMagentaCyanTransparencyTransparentSemi-TransparentOpaque

Font Size

50%75%100%125%150%175%200%300%400%

Text Edge Style

NoneRaisedDepressedUniformDropshadow

Font Family

Proportional Sans-SerifMonospace Sans-SerifProportional SerifMonospace SerifCasualScriptSmall Caps

Reset restore all settings to the default valuesDone

Close Modal Dialog

End of dialog window.

Close Modal Dialog

This is a modal window. This modal can be closed by pressing the Escape key or activating the close button.

[#9]

Thanks! It worked for me

[#10]

Ah, it's a .csv file.  Use "Open" and choose the "Data (Using Preview)" option.

Then choose your subset preferences.