How do I import only the latest files in JMP.

[#1]

Hi All,

I currently have a script that imports multiple files from a directory using the IMF function with a date and name filter enabled. I import about 4000 files and then clean up the data tables into one data table that is saved. The directory keeps getting updated with new files and I would like to import the new files only so I can append it to the last date table.  

1. I was wondering if I can obtain the timestamp at which the import was done from the JMP logs and use that to update the data filter to run only from the last date I imported.   
2. I tried getting the names of the files from the File Name column, create a list  and compare it to the directory, no luck there. 

your help is much appreciated. 

tblRef = Multiple File Import(

<<Set Folder( "C:\Docs\" ),

<<Set Show Hidden( 0 ),

<<Set Subfolders( 0 ),

<<Set Name Filter( "*IA-N5*;" ),

<<Set Name Enable( 1 ),

<<Set Size Filter( {27136, 110080} ),

<<Set Size Enable( 0 ),

<<Set Date Filter( {3772434440.006, 3786862040} ),

<<Set Date Enable( 0 ),

<<Set Add File Name Column( 1 ),

<<Set Add File Size Column( 0 ),

<<Set Add File Date Column( 1 ),

<<Set Import Mode( "CSVData" ),

<<Set Charset( "Best Guess" ),

<<Set Stack Mode( "Stack Similar" ),

<<Set CSV Has Headers( 1 ),

<<Set CSV Allow Numeric( 1 ),

<<Set CSV First Header Line( 1 ),

<<Set CSV Number Of Header Lines( 1 ),

<<Set CSV First Data Line( 2 ),

<<Set CSV EOF Comma( 1 ),

<<Set CSV EOF Tab( 0 ),

<<Set CSV EOF Space( 0 ),

<<Set CSV EOF Spaces( 0 ),

<<Set CSV EOF Other( "" ),

<<Set CSV EOL CRLF( 1 ),

<<Set CSV EOL CR( 1 ),

<<Set CSV EOL LF( 1 ),

<<Set CSV EOL Semicolon( 0 ),

<<Set CSV EOL Other( "" ),

<<Set CSV Quote( "\!"" ),

<<Set CSV Escape( "" ),

<<Set XML Method( "guess" ),

<<Set XML Guess( "huge" ),

<<Set XML Settings( XML Settings() ),

<<Set JSON Method( "guess" ),

<<Set JSON Guess( "huge" ),

<<Set JSON Settings( JSON Settings() ),

<<Set PDF Method( "guess" ),

<<Set PDF Settings( PDF All Tables( Combine( All ) ) ),

<<Set Import Callback( Empty() )

) << Import Data;

tblRef << Set Name(" Data");

[#2]

You could pick the maximum File Date from the MFI import result table and use that on the next import to limit the files?

[#3]

How can I pick the maximum File Date? 

Is there a way to obtain the timestamp from the logs?

[#4]

You should have File Date in your data table as you have Set Add File Date Column enabled. Then you can use for example Col Max()

Col Max(:File Date)

[#5]

It still importing everything. This is what I did:

dt = Open("C:\Users\");

LastImport = dt << Col Max(:"File Date");
maxDate = LastImport[1];
tblRef = Multiple File Import(
	<<Set Folder("C:\Docs"),
	<<Set Show Hidden(0),
	<<Set Subfolders(0),
	<<Set Name Filter("*IA-;"),
	<<Set Name Enable(1),
	<<Set Size Filter({27136, 110080}),
	<<Set Size Enable(0),
	<<Set Date Filter({maxDate, 3786862040}),
	<<Set Date Enable(0),
	<<Set Add File Name Column(1),
	<<Set Add File Size Column(0),
	<<Set Add File Date Column(1),
	<<Set Import Mode("CSVData"),
	<<Set Charset("Best Guess"),
	<<Set Stack Mode("Stack Similar"),
	<<Set CSV Has Headers(1),
	<<Set CSV Allow Numeric(1),
	<<Set CSV First Header Line(1),
	<<Set CSV Number Of Header Lines(1),
	<<Set CSV First Data Line(2),
	<<Set CSV EOF Comma(1),
	<<Set CSV EOF Tab(0),
	<<Set CSV EOF Space(0),
	<<Set CSV EOF Spaces(0),
	<<Set CSV EOF Other(""),
	<<Set CSV EOL CRLF(1),
	<<Set CSV EOL CR(1),
	<<Set CSV EOL LF(1),
	<<Set CSV EOL Semicolon(0),
	<<Set CSV EOL Other(""),
	<<Set CSV Quote("\!""),
	<<Set CSV Escape(""),
	<<Set XML Method("guess"),
	<<Set XML Guess("huge"),
	<<Set XML Settings(XML Settings()),
	<<Set JSON Method("guess"),
	<<Set JSON Guess("huge"),
	<<Set JSON Settings(JSON Settings()),
	<<Set PDF Method("guess"),
	<<Set PDF Settings(PDF All Tables(Combine(All))),
	<<Set Import Callback(Empty())
) << Import Data;

tblRef << Set Name("Data");

[#6 · Accepted solution]

I think you will have to evaluate the datetime value inside multiple file imports settings. Below is an example. See how Eval(EvalExpr()) and Expr() are used ( Insert one expression into another using Eval Insert, Eval Expr, Parse, and Substitute )

Names Default To Here(1);

dt = New Table("12345678_Cycle_123456781_Cycle",
	Add Rows(6),
	Compress File When Saved(1),
	New Column("Cycle",
		Numeric,
		"Continuous",
		Format("Best", 12),
		Set Values([1, 2, 3, 4, 5, 6])
	),
	New Column("File Date",
		Numeric,
		"Continuous",
		Format("yyyy-mm-ddThh:mm:ss", 20, 0),
		Input Format("yyyy-mm-ddThh:mm:ss", 0),
		Set Values(
			[3741613139.034, 3741613139.034, 3741613139.034, 3783350373.379, 3783350373.379,
			3783350373.379]
		)
	)
);

max_date = Col Max(Column(dt, "File Date"));

mfi = Eval(EvalExpr(Multiple File Import(
	<<Set Folder(<path>),
	<<Set Show Hidden(0),
	<<Set Subfolders(0),
	<<Set Name Filter("*;"),
	<<Set Name Enable(1),
	<<Set Size Filter({5700, 10079}),
	<<Set Size Enable(0),
	<<Set Date Filter({Expr(max_date), 3783350373.379}),
	<<Set Date Enable(1),
	<<Set Add File Name Column(0),
	<<Set Add File Size Column(0),
	<<Set Add File Date Column(1),
	<<Set Import Mode("CSVData"),
	<<Set Charset("Best Guess"),
	<<Set Stack Mode("Stack Similar"),
	<<Set CSV Has Headers(1),
	<<Set CSV Allow Numeric(1),
	<<Set CSV First Header Line(1),
	<<Set CSV Number Of Header Lines(1),
	<<Set CSV First Data Line(2),
	<<Set CSV EOF Comma(1),
	<<Set CSV EOF Tab(0),
	<<Set CSV EOF Space(0),
	<<Set CSV EOF Spaces(0),
	<<Set CSV EOF Other(""),
	<<Set CSV EOL CRLF(1),
	<<Set CSV EOL CR(1),
	<<Set CSV EOL LF(1),
	<<Set CSV EOL Semicolon(0),
	<<Set CSV EOL Other(""),
	<<Set CSV Quote("\!""),
	<<Set CSV Escape(""),
	<<Set XML Method("guess"),
	<<Set XML Guess("huge"),
	<<Set XML Settings(XML Settings()),
	<<Set JSON Method("guess"),
	<<Set JSON Guess("huge"),
	<<Set JSON Settings(JSON Settings()),
	<<Set PDF Method("guess"),
	<<Set PDF Settings(PDF All Tables(Combine(All))),
	<<Set Import Callback(Empty())
))) << Import data;

[#7]

It worked. Thank you soooo much! 

[#8]

You might have to add some time to your max time to avoid getting that max file date file couple of times. Depending on how often you get new files, adding a second could be enough

max_date = Col Max(Column(dt, "File Date")) + 1;

but most likely MFI also understands decimals, so you could add 0.01 for example (might be a good idea to test with small amounts of files)