cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Choose Language Hide Translation Bar
thedellette
Level II

Batch processing files to extract data into a summary table

I have multiple files that I want to perform some analysis on. I want to create a loop to open the file, perform the analysis, calculate a value "area under a curve" in this example and then make a summary table with the file name and area. I can open the files from a folder and perform the analysis. I am not sure how to create the summary table. I have given an example with 2 files but I have many more files I would like to process in this way. Any help or guidance is greatly appreciated! Also please let me know if this does not make sense or if more information is needed.

 

//point to file directory

path = Pick Directory( "Select Directory Containing Data Files" );

//Gets the list of files in the selected file folder
file_list = Files In Directory( path );

//i=0;open file
nfile_list = N Items( file_list );
For( i = 1, i <= nfile_list, i += 1, 
	//Set Current Directory( dir_DirectoryPath );
	dt = Open( path || file_list[i], );
	//Calculate area
	dt = Current Data Table();
	dt << clear row states;
	obj3 = dt << Bivariate(
		Y( :Name( "Data" ) ),
		X( :Time ),
		Fit Polynomial( 6, {Line Color( {212, 73, 88} )} ),
		SendToReport( Dispatch( {}, "Bivar Plot", FrameBox, {Frame Size( 449, 411 )} ) )
	);
	//save graph to data dable
	obj3 << Save Script to Data Table( "Area" );
	//get the prediction formula string and calculate the area under the curve for 30 minute adsoription
	fm = Report( obj3 )[Outline Box( "Polynomial ?" )][Text Edit Box( 1 )] << get text;
	//get the string to the right of the equal sign and convert the string to an expression using Parse()
	fm = Parse( Trim( Word( 2, fm, "=" ) ) );
	area = Integrate( fm, :Time, 0, 3600 );
	Caption( "The area under this curve is " || Char( area ) );
	Print( area );
);

Example of Final Table I would like to have:

thedellette_0-1659380880942.png

 

Any help with this would be greatly appreciated. Thanks

2 ACCEPTED SOLUTIONS

Accepted Solutions
jthi
Super User

Re: Batch processing files to extract data into a summary table

Before you start looping, create new table with File Name and Area columns and format them as you want them to be. Easiest way is to create new table and then use Copy Table Script. Also remember to assign reference variable to that table!

jthi_0-1659383448197.png

Then in the loop after you have calculated the area you can get  the table name by using << Get Name and then add the new rows to your collection table with << Add Rows(). Also remember to close unnecessary tables and reports after getting the relevant information from them.

Names Default To Here(1);

dt_collector = New Table("Untitled",
	Add Rows(0),
	New Column("File Name", Character, Nominal, Set Values({})),
	New Column("Area", Numeric, "Continuous",  Set Values([]))
);

// For(..
//	...
	//dt_name = dt << Get Name;
	dt_name = "table";
	area = 123;
	dt_collector << Add Row({File Name = dt_name, Area = area});
//);

 

-Jarmo

View solution in original post

txnelson
Super User

Re: Batch processing files to extract data into a summary table

I have added code that will create a new data table, and will write the file name and area to the new table

Names Default to Here( 1 );
//point to file directory

path = Pick Directory( "Select Directory Containing Data Files" );

//Gets the list of files in the selected file folder
file_list = Files In Directory( path );

// Crete beginnings of output data table
dtFinal = New Table( "Final", New Column( "File Name", Character ), New Column( "Area" ) );

//i=0;open file
nfile_list = N Items( file_list );
For( i = 1, i <= nfile_list, i += 1, 
	//Set Current Directory( dir_DirectoryPath );
	dt = Open( path || file_list[i], );
	//Calculate area
	dt = Current Data Table();
	dt << clear row states;
	obj3 = dt << Bivariate(
		invisible,
		Y( :Name( "Data" ) ),
		X( :Time ),
		Fit Polynomial( 6, {Line Color( {212, 73, 88} )} ),
		SendToReport( Dispatch( {}, "Bivar Plot", FrameBox, {Frame Size( 449, 411 )} ) )
	);
	//save graph to data dable
	obj3 << Save Script to Data Table( "Area" );
	//get the prediction formula string and calculate the area under the curve for 30 minute adsoription
	fm = Report( obj3 )[Outline Box( "Polynomial ?" )][Text Edit Box( 1 )] << get text;
	//get the string to the right of the equal sign and convert the string to an expression using Parse()
	fm = Parse( Trim( Word( 2, fm, "=" ) ) );
	area = Integrate( fm, :Time, 0, 3600 );
	Caption( "The area under this curve is " || Char( area ) );
	Print( area );
	
	// Add information to Final data table
	dtFinal << add rows( 1 );
	dtFinal:File Name[N Rows( dtFinal )] = file_list[i];
	dtFinal:area[N Rows( dtFinal )] = area;
);
Jim

View solution in original post

4 REPLIES 4
jthi
Super User

Re: Batch processing files to extract data into a summary table

Before you start looping, create new table with File Name and Area columns and format them as you want them to be. Easiest way is to create new table and then use Copy Table Script. Also remember to assign reference variable to that table!

jthi_0-1659383448197.png

Then in the loop after you have calculated the area you can get  the table name by using << Get Name and then add the new rows to your collection table with << Add Rows(). Also remember to close unnecessary tables and reports after getting the relevant information from them.

Names Default To Here(1);

dt_collector = New Table("Untitled",
	Add Rows(0),
	New Column("File Name", Character, Nominal, Set Values({})),
	New Column("Area", Numeric, "Continuous",  Set Values([]))
);

// For(..
//	...
	//dt_name = dt << Get Name;
	dt_name = "table";
	area = 123;
	dt_collector << Add Row({File Name = dt_name, Area = area});
//);

 

-Jarmo
thedellette
Level II

Re: Batch processing files to extract data into a summary table

Thank you! You gave a nice detailed explanation as well. Easy to follow

 

txnelson
Super User

Re: Batch processing files to extract data into a summary table

I have added code that will create a new data table, and will write the file name and area to the new table

Names Default to Here( 1 );
//point to file directory

path = Pick Directory( "Select Directory Containing Data Files" );

//Gets the list of files in the selected file folder
file_list = Files In Directory( path );

// Crete beginnings of output data table
dtFinal = New Table( "Final", New Column( "File Name", Character ), New Column( "Area" ) );

//i=0;open file
nfile_list = N Items( file_list );
For( i = 1, i <= nfile_list, i += 1, 
	//Set Current Directory( dir_DirectoryPath );
	dt = Open( path || file_list[i], );
	//Calculate area
	dt = Current Data Table();
	dt << clear row states;
	obj3 = dt << Bivariate(
		invisible,
		Y( :Name( "Data" ) ),
		X( :Time ),
		Fit Polynomial( 6, {Line Color( {212, 73, 88} )} ),
		SendToReport( Dispatch( {}, "Bivar Plot", FrameBox, {Frame Size( 449, 411 )} ) )
	);
	//save graph to data dable
	obj3 << Save Script to Data Table( "Area" );
	//get the prediction formula string and calculate the area under the curve for 30 minute adsoription
	fm = Report( obj3 )[Outline Box( "Polynomial ?" )][Text Edit Box( 1 )] << get text;
	//get the string to the right of the equal sign and convert the string to an expression using Parse()
	fm = Parse( Trim( Word( 2, fm, "=" ) ) );
	area = Integrate( fm, :Time, 0, 3600 );
	Caption( "The area under this curve is " || Char( area ) );
	Print( area );
	
	// Add information to Final data table
	dtFinal << add rows( 1 );
	dtFinal:File Name[N Rows( dtFinal )] = file_list[i];
	dtFinal:area[N Rows( dtFinal )] = area;
);
Jim
thedellette
Level II

Re: Batch processing files to extract data into a summary table

Thank you! I saw the other reply first but I like this one as well.