Log Compiler

[#1]

I have a script that compiles a group of log files depending on the title of the files in the source directory.  This script reads in a Lot ID from an address table and uses that to search the path directory "dir" and compile the target files.  I need to repurpose this script where I would like the script to look inside the text of the available files to see if the lot ID shows up there and compiles accordingly.  I saw some references to this online but the idea of opening every file to make the determination seems like something that could take a long time.  Ideas appreciated.

Names Default To Here( 1 );
Clear Globals();

lst = Column( "lot" ) << Get Values;

n = N Items( lst );

dir = "\\Source_Directory\";

myfiles = {};

files = Files In Directory( dir, Recursive );

n_getfiles = N Items( files );

filteredFiles = {};

While( (testname = Remove From( files, 1 )) != {}, 

	Relevant = Char( Regex( testname[1], "/4(\d{6})R", "4\1" ) );

If( Contains( testname[1], Relevant ),

		Insert Into( filteredFiles, testname, 1 );

	);
);

Show( filteredFiles );

t_filterfiles = t4 - t3;

n_filterfiles = N Items( filteredFiles );

Show( t_getfiles, n_getfiles, t_filterfiles, n_filterfiles );

Show( filteredFiles );

compiler_function = Function( {lot_number}, 

	Try(
		use1 = lot_number;
		LotID = Char( use1 );

		For( i = 1, i <= N Items( filteredFiles ), i++, 
			If( Contains( filteredFiles[i], LotID ), 
				Insert Into( myFiles, filteredFiles[i] ))
		);
		final_files = myfiles;
	)
);

For( l = 1, l <= n, l++, //Show(l);
	compiler_function( lst[l] )
);

i = 1;

dtjoin = Open(

	dir || myfiles[i], 

	Import Settings(
		End Of Line( CRLF, CR, LF ),
		End Of Field( Tab, Comma, CSV( 1 ) ),
		Strip Quotes( 1 ),
		Use Apostrophe as Quotation Mark( 0 ),
		Use Regional Settings( 0 ),
		Scan Whole File( 1 ),
		Treat empty columns as numeric( 0 ),
		CompressNumericColumns( 0 ),
		CompressCharacterColumns( 0 ),
		CompressAllowListCheck( 0 ),
		Labels( 1 ),
		Column Names Start( 1 ),
		First Named Column( 1 ),
		Data Starts( 2 ),
		Lines To Read( "All" ),
		Year Rule( "20xx" )
	);
	
	
);
Wait( 0 );
Column( 1 ) << set name( "Column 1" );
dtjoin << set name( "Joined1" );
dtjoin << Add Rows( 1, after( N Rows( dtjoin ) ) );
For( i = N Rows( dtjoin ), i <= N Rows( dtjoin ), i++,
	:Column 1[i] = "File 1"
);

i = 2;

For( i = 2, i <= N Items( myfiles ), i++, 

	dt = Open(

		dir || myfiles[i], 

		Import Settings(
			End Of Line( CRLF, CR, LF ),
			End Of Field( Tab, Comma, CSV( 1 ) ),
			Strip Quotes( 1 ),
			Use Apostrophe as Quotation Mark( 0 ),
			Use Regional Settings( 0 ),
			Scan Whole File( 1 ),
			Treat empty columns as numeric( 0 ),
			CompressNumericColumns( 0 ),
			CompressCharacterColumns( 0 ),
			CompressAllowListCheck( 0 ),
			Labels( 1 ),
			Column Names Start( 1 ),
			First Named Column( 1 ),
			Data Starts( 2 ),
			Lines To Read( "All" ),
			Year Rule( "20xx" )
		);

	);
        
	Column( 1 ) << set name( "Column 1" );
	dt << Add Rows( 1, after( N Rows( dt ) ) );
	For( j = N Rows( dt ), j <= N Rows( dt ), j++,
		:Column 1[j] = ("File " || Char( i ))
	);
 
	dtjoin = dtjoin << Concatenate(
		dt, output table name( "Joined" || Char( i ) ), 
	);

	Close( dt, nosave );

	Close( "Joined" || Char( i - 1 ), nosave );

	Wait( 0.1 );

); 

dtjoin << set name( " Mask Data" );

[#2]

How large files do you have? How many files? Can the ID be anywhere in the file or in specific location/line? What have you tried (for example Load Text File())?

[#3]

Files come out at ~25kB, 300-400 lines long, number of files can vary, maybe 100 worst case.  Anything I considered would have had to have opened each file to decide relevance, I didn't write anything like that, I might expire before the data arrives.  Load Text File() looks interesting, I'll have a tinker with that.

[#4]

Multiple File Import might help, either by importing and stacking CSV or by loading the unprocessed text into a row per file. Play with the GUI then save the script if you get something you like. If you can use file time stamps, it will help there too.