topic Re: Multi file Import: Discovering Files takes too long in Discussions

Multi file Import: Discovering Files takes too long

hogi — Fri, 08 Sep 2023 10:02:21 GMT

MFI is great - I really love it.

But sometimes the low speed of the file search hurts.
If the data comes from a network drive and I want to access the folder from home office, it can take > 10 minutes for Discovering Files to find my 4 matching file out of thousand of folders/subfolders/files ...

Then it's faster to open Python in parallel, write a short GLOB search and load the files one by one via for loop and open(filename). Much faster than waiting for MFI to populate the Files list with all the file which fail the filename match.

The example with Python/glob shows that the file search can be orders of magnitude faster - Is there any trick in Jmp/MFI to speed this up?

Re: Multi file Import: Discovering Files takes too long

Craige_Hales — Mon, 11 Sep 2023 13:34:49 GMT

The glob search is probably faster because it does not retrieve date and size.

https://stackoverflow.com/questions/23430395/glob-search-files-in-date-order

There is no way to tell MFI to skip retrieving date and size (but it could be a wish.)

edit: this is much less noticeable on the machine's local file system; the network disk communication is probably 3X greater to get the name+date+size rather than just the name. Also, you might be able to use the recursive filesInDirectory, which should be similar to the glob search time, and pick the files you want from that full list with a pattern match. And be careful bench-marking; the 2nd time may be much faster if caching is involved--easily 100X if the communication time goes to zero.

(And change date+time to date+size.)

Re: Multi file Import: Discovering Files takes too long

hogi — Mon, 11 Sep 2023 15:43:08 GMT

I posted a wish to optimize the speed of MFI:

Re: Multi file Import: Discovering Files takes too long

hogi — Wed, 26 Jun 2024 15:38:47 GMT

Amazingly useful: the new fast mode of MFI in JMP18 :)

Re: Multi file Import: Discovering Files takes too long

Craige_Hales — Wed, 26 Jun 2024 17:30:05 GMT

@hogi Is that a button somewhere?

Re: Multi file Import: Discovering Files takes too long

hogi — Tue, 05 Nov 2024 07:58:16 GMT

I often used the Name Filter in MFI. The idea:
The user uses pick files to select some files, then the workflow uses MFI to import the files.

This is now possible via Select Files Manually: