topic Re: How can I loop through multiple csv files and select them base on specific file name? in Discussions

How can I loop through multiple csv files and select them base on specific file name?

Bin_JMPspace — Fri, 11 Oct 2024 12:53:54 GMT

Here is my code:

path = "/users/BOT/Test_files/";
Set File Search Path( mypath );

// filter out all csv files in the path that contains "Pass" &&  Not contains "Conditional_Pass" or "Fail" in file name
y = Files In Directory( mypath );

For( i = N Items( y ), i > 0, i--,
	If( (Ends With( y[i], ".csv" ) & Contains( y[i], "Pass" ) & Not( Contains( y[i], "Conditional_Pass" ) ) & Not( Contains( y[i], "Fail" ) )),
	,
		Remove From( y, i )
	)
);
n = N Items( y );

Keep getting below error as screenshot, is it because the Files In Directory() function being deprecated?

Funny thing is above codes used to work for JMP16, below error pops out after I upgrade to JMP17 yesterday.

Re: How can I loop through multiple csv files and select them base on specific file name?

jthi — Fri, 11 Oct 2024 12:57:22 GMT

You are most likely getting that error as you are removing same list you are looping over. For this type of filtering I would suggest using Filter Each instead of For loop.

Also, have you tried using Multiple File Import?

Re: How can I loop through multiple csv files and select them base on specific file name?

hogi — Fri, 11 Oct 2024 13:26:04 GMT

please check path vs. mypath

besides that you could use Filter Each instead of the for loop:

//path = "/users/BOT/Test_files/";
mypath = "/users/BOT/Test_files/";

y = Files In Directory( mypath );

y = filter each({fn}, y, Ends With( fn, ".csv" ) & Contains( fn, "Pass" ) & Not( Contains( fn, "Conditional_Pass" ) ) & Not( Contains( fn, "Fail" ) ) )

Re: How can I loop through multiple csv files and select them base on specific file name?

hogi — Tue, 15 Oct 2024 10:53:18 GMT

You could take the next step as well and upgrade to JMP18 - with the option to use a File List in MultiFileImport.

For some cases, this will speed up the data import by orders of magnitude!

y= Transform each ({fn}, y, mypath || fn);
Eval(Eval Expr (Multiple File Import(
	<<Set Use File List( 1 ),
	<<Set File List( Expr(y))
);

At the moment, Filelist + MultiFileImport needs the complicate Eval(Eval Expr(... Expr() ...)).
This will get fixed in a future release of JMP.

Re: How can I loop through multiple csv files and select them base on specific file name?

Craige_Hales — Mon, 14 Oct 2024 18:52:53 GMT

I think you have a data table open with a column named Y. JSL is trying to use the table's column, which is not what you want. Either close the table, or change the name in either the table or the script, or use scoping to tell JMP which you mean.

This example is starting JMP fresh before Y is in any namespace.

@Audrey_Shull - That error dialog would be improved 100% if it named the table the variable is coming from.

Re: How can I loop through multiple csv files and select them base on specific file name?

Bin_JMPspace — Tue, 15 Oct 2024 10:01:43 GMT

Thanks Craige, yes I found my bug a day ago and solved it, just like your description.

Re: How can I loop through multiple csv files and select them base on specific file name?

Bin_JMPspace — Tue, 15 Oct 2024 10:02:32 GMT

Thanks hog, the filter each function also gives me some inspiration.

Re: How can I loop through multiple csv files and select them base on specific file name?

Bin_JMPspace — Tue, 15 Oct 2024 10:03:42 GMT

Hey jthi, thank you.
Agree the Multi File Import is a good way to parse the files, faster than my current method.