HTTP request get (file download): Operation timed out after 60000 milliseconds

[#1]

Dear all,

I want to import a csv file from an internal server ("http:....").

(JMP 14.0, Windows)

It generally worked quite well, but the file is getting larger and larger.

As I need to scan the full file (to get all data types right), I thought, it would be a good idea,

to first download the file to the $TEMP directory and than to import it.

So I implemented file download with http request.

For a smaller file it worked well again, but for the larger ones I get a timeout,

and I found no option to tell JMP to change that timeout,

except that I could change the registry, please see @landon  in the following link:

https://community.jmp.com/message/225323

Is downloading a large file from http:... by "http request" the right method?

Is there another method to modify timeout without changing registry (my script needs to work at other users also)?

Is downloading before importing (using full scan) faster than direct import from server?

Thanks and best regards,

   Georg

[#2 · Accepted solution]

I'm pretty sure JMP only downloads the file once for open("https...", type), even if the file needs to be scanned more than once (when type is "text" for csv files). I think the file is kept in memory for the text/csv case, which will limit the maximum size; for other types, the file may be saved to a temp file (the excel opener might need to work from a disk file, for example).

So it is possible that downloading csv to a disk file might be slower (probably not by a noticeable amount though). If the CSV file is approaching half your memory size (8GB file size on a 16GB machine...the JMP table will need memory too!) it might be necessary to download it first.

Not sure where the timeout happens, but an alternate approach to downloading could look like this:

// here's a way to download a URL to a local file
filename = Save Text File(
	"$temp/sample.csv", // extension is saved as CSV
	Load Text File(
		"https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv?accessType=DOWNLOAD",
		blob
	)
);

dt1 = Open( filename ); // "text" not needed because extension is CSV

// here's a way to open directly
dt2 = Open(
	"https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv?accessType=DOWNLOAD",
	"text" // here the extension might or might not be CSV, use "text" to make sure
);

// verify both ways make the same table
compare = dt1 << Compare Data Tables(
	Compare with( dt2 ),
	Compare Data,
	Show Difference Summary( 0 ),
	Show Difference Plot( 0 )
);

If( compare << Are Data Different,
	Write( "\!nwhy are they different?" ),
	Write( "\!nResults identical" )
);

Show( N Rows( dt1 ), N Cols( dt1 ), N Rows( dt2 ), N Cols( dt2 ) );

// clean up
compare << closewindow;
Close( dt1, "nosave" );
Close( dt2, "nosave" );

@bryan_boone 

[#3]

Thanks Craig for quick reply,

your approach generally works and gives similar results,

but in the resulting data tables the data types are different (due to decimal separator , / .).

So your information helps me to find a way for working.

BR, Georg

[#4 · Accepted solution]

HTTP Request has a Timeout message.  The default is 60 seconds.

Names Default To Here( 1 );
request = New HTTP Request(
	Url( "https://community.jmp.com/html/assets/community-icons/community-icon-mastering.png" ),
	Method( "GET" ),
	Timeout( 120 )
);
bytes = request << Send;
img = Open( bytes, jpg );
obj = New Window( "Mastering JMP", img );