Log Compiler

SpannerHead — Tue, 14 Jan 2025 16:14:44 GMT

I have a script that compiles a group of log files depending on the title of the files in the source directory. This script reads in a Lot ID from an address table and uses that to search the path directory "dir" and compile the target files. I need to repurpose this script where I would like the script to look inside the text of the available files to see if the lot ID shows up there and compiles accordingly. I saw some references to this online but the idea of opening every file to make the determination seems like something that could take a long time. Ideas appreciated.

Names Default To Here( 1 );
Clear Globals();

lst = Column( "lot" ) << Get Values;

n = N Items( lst );

dir = "\\Source_Directory\";

myfiles = {};

files = Files In Directory( dir, Recursive );

n_getfiles = N Items( files );

filteredFiles = {};

While( (testname = Remove From( files, 1 )) != {}, 

	Relevant = Char( Regex( testname[1], "/4(\d{6})R", "4\1" ) );

If( Contains( testname[1], Relevant ),

		Insert Into( filteredFiles, testname, 1 );

	);
);

Show( filteredFiles );

t_filterfiles = t4 - t3;

n_filterfiles = N Items( filteredFiles );

Show( t_getfiles, n_getfiles, t_filterfiles, n_filterfiles );

Show( filteredFiles );

compiler_function = Function( {lot_number}, 

	Try(
		use1 = lot_number;
		LotID = Char( use1 );

		For( i = 1, i <= N Items( filteredFiles ), i++, 
			If( Contains( filteredFiles[i], LotID ), 
				Insert Into( myFiles, filteredFiles[i] ))
		);
		final_files = myfiles;
	)
);

For( l = 1, l <= n, l++, //Show(l);
	compiler_function( lst[l] )
);

i = 1;

dtjoin = Open(

	dir || myfiles[i], 

	Import Settings(
		End Of Line( CRLF, CR, LF ),
		End Of Field( Tab, Comma, CSV( 1 ) ),
		Strip Quotes( 1 ),
		Use Apostrophe as Quotation Mark( 0 ),
		Use Regional Settings( 0 ),
		Scan Whole File( 1 ),
		Treat empty columns as numeric( 0 ),
		CompressNumericColumns( 0 ),
		CompressCharacterColumns( 0 ),
		CompressAllowListCheck( 0 ),
		Labels( 1 ),
		Column Names Start( 1 ),
		First Named Column( 1 ),
		Data Starts( 2 ),
		Lines To Read( "All" ),
		Year Rule( "20xx" )
	);
	
	
);
Wait( 0 );
Column( 1 ) << set name( "Column 1" );
dtjoin << set name( "Joined1" );
dtjoin << Add Rows( 1, after( N Rows( dtjoin ) ) );
For( i = N Rows( dtjoin ), i <= N Rows( dtjoin ), i++,
	:Column 1[i] = "File 1"
);

i = 2;

For( i = 2, i <= N Items( myfiles ), i++, 

	dt = Open(

		dir || myfiles[i], 

		Import Settings(
			End Of Line( CRLF, CR, LF ),
			End Of Field( Tab, Comma, CSV( 1 ) ),
			Strip Quotes( 1 ),
			Use Apostrophe as Quotation Mark( 0 ),
			Use Regional Settings( 0 ),
			Scan Whole File( 1 ),
			Treat empty columns as numeric( 0 ),
			CompressNumericColumns( 0 ),
			CompressCharacterColumns( 0 ),
			CompressAllowListCheck( 0 ),
			Labels( 1 ),
			Column Names Start( 1 ),
			First Named Column( 1 ),
			Data Starts( 2 ),
			Lines To Read( "All" ),
			Year Rule( "20xx" )
		);

	);
        
	Column( 1 ) << set name( "Column 1" );
	dt << Add Rows( 1, after( N Rows( dt ) ) );
	For( j = N Rows( dt ), j <= N Rows( dt ), j++,
		:Column 1[j] = ("File " || Char( i ))
	);
 
	dtjoin = dtjoin << Concatenate(
		dt, output table name( "Joined" || Char( i ) ), 
	);

	Close( dt, nosave );

	Close( "Joined" || Char( i - 1 ), nosave );

	Wait( 0.1 );

); 

dtjoin << set name( " Mask Data" );

Re: Log Compiler

jthi — Tue, 14 Jan 2025 16:18:32 GMT

How large files do you have? How many files? Can the ID be anywhere in the file or in specific location/line? What have you tried (for example Load Text File())?

Re: Log Compiler

SpannerHead — Tue, 14 Jan 2025 16:41:17 GMT

Files come out at ~25kB, 300-400 lines long, number of files can vary, maybe 100 worst case. Anything I considered would have had to have opened each file to decide relevance, I didn't write anything like that, I might expire before the data arrives. Load Text File() looks interesting, I'll have a tinker with that.

Re: Log Compiler

Craige_Hales — Tue, 14 Jan 2025 23:17:21 GMT

Multiple File Import might help, either by importing and stacking CSV or by loading the unprocessed text into a row per file. Play with the GUI then save the script if you get something you like. If you can use file time stamps, it will help there too.

topic Re: Log Compiler in Discussions

Log Compiler

Re: Log Compiler

Re: Log Compiler

Re: Log Compiler