topic (Re: Load Text File() File Size Limit? in Discussions

Load Text File() File Size Limit?

nikles — Fri, 09 Jun 2023 22:06:46 GMT

Hi. I was wondering if anyone knows what the file size limit is for using the Load Text File() function? I've imported text files up to 1GB, but noticed it fails somewhere at or above 2GB for my system. Knowing this limit would be nice so I could warn the user ahead of time that the file is too big to open. I understand it may be system dependent, but if anyone knows how this limit could be estimated, that would be great. Thanks.

Re: Load Text File() File Size Limit?

Craige_Hales — Tue, 23 Feb 2021 18:05:31 GMT

Text, and strings are limited to 2gig. You can load a blob as big as you need up to memory.

(Re: Load Text File() File Size Limit?

Monomorphist — Wed, 24 Feb 2021 11:37:12 GMT

I've loaded recently a csv file with 310 million rows (17GB)! Worked!! Quite impressive....

Cheers,

Kofi

Re: (Re: Load Text File() File Size Limit?

Craige_Hales — Wed, 24 Feb 2021 12:57:48 GMT

@Monomorphist using open() or multiple file import() ? Both of those read the file, incrementally, from disk, avoiding the 2G limit. Open() may read the file more than once. MFI could be a lot faster, especially if you have more than one file to load at the same time.

@nikles I probably should have asked: what sort of data is in the text file? Maybe there is another approach.