topic SQL Custom and Eval / Parse (and \!\!\!...) in Discussions

SQL Custom and Eval / Parse (and \!\!\!...)

Franck_R — Fri, 09 Jun 2023 22:00:00 GMT

Hi!

I do a lot of queries that can't use the JMP query builder and have to be written in SQL Custom.
But as I also want to generalize my code I have to integrate a lot of eval / inserts and \!

Here is an example:

Code not yet generalized

New SQL Query(
	Connection(
		"ODBC:blabla"
	),
	QueryName( "v_test_run" ),
	CustomSQL(
		"SELECT \!"t1\!".lot_id 
		FROM \!"public\!".\!"v_test_run\!"  \!"t1\!" 
		WHERE  (  (  ( \!"t1\!".test_run_timestamp >= '2020-11-01 15:25:11' )  )  ) ;"
	)
) << Modify;

Code Generalized to include a date:

gbStr = Eval Insert("

New SQL Query(
	Connection(
		\!"ODBC:blabla\!"
	),
	QueryName( \!"v_test_run\!" ),
	CustomSQL(
		\!"SELECT \!\!\!"t1\!\!\!".lot_id 
FROM \!\!\!"public\!\!\!".\!\!\!"v_test_run\!\!\!"  \!\!\!"t1\!\!\!" 
WHERE  (  (  ( \!\!\!"t1\!\!\!".test_run_timestamp >= '^date_deb^' )  )  ) ;\!"
)
) << Modify;


");
Eval( Parse( gbStr ) );

Is there any other method that would save me from having to juggle with these \!\!\! symbols?

On complex scripts it's really difficult to handle and debug!

Thank you

Re: SQL Custom and Eval / Parse (and \!\!\!...)

ih — Mon, 07 Dec 2020 18:09:56 GMT

There are a few options outlined in this article, I recommend starting with Eval( Eval Expr()) and then wrapping each piece you need to replace with Expr(), and using \[ ... ]\ to reduce the number of escaped characters. It could look something like this:

Eval( Eval Expr(
	New SQL Query(
		Connection(
			"ODBC:blabla"
		),
		QueryName( "v_test_run" ),
		CustomSQL(
			Expr( "\[SELECT "t1".lot_id 
			FROM "public"."v_test_run"  "t1" 
			WHERE  (  (  ( "t1".test_run_timestamp >= ']\" || date_deb || "' )  )  ) ;")
		)
	) << Modify
) );

Re: SQL Custom and Eval / Parse (and \!\!\!...)

Franck_R — Mon, 07 Dec 2020 18:14:50 GMT

Thank you very much! I had read this article and had tried but I still have difficulties in mastering this eval (eval expr (expr))

Re: SQL Custom and Eval / Parse (and \!\!\!...)

pmroz — Mon, 07 Dec 2020 18:15:08 GMT

I would use the delimiters \[ and ]\ to include double quotes in the string.

date_deb = "to_date('01-Mar-2019', 'dd-mon-yyyy')";
gbStr = Eval Insert("\[
New SQL Query(
	Connection(
		"ODBC:blabla"
	),
	QueryName( "v_test_run" ),
	CustomSQL(
		"SELECT "t1".lot_id 
FROM "public"."v_test_run"  "t1" 
WHERE  (  (  ( "t1".test_run_timestamp >= ^date_deb^ )  )  ) ;"
)
) << Modify;
]\");

write(gbstr);

Eval( Parse( gbStr ) );

Output:

New SQL Query(
	Connection(
		"ODBC:blabla"
	),
	QueryName( "v_test_run" ),
	CustomSQL(
		"SELECT "t1".lot_id 
FROM "public"."v_test_run"  "t1" 
WHERE  (  (  ( "t1".test_run_timestamp >= to_date('01-Mar-2019', 'dd-mon-yyyy') )  )  ) ;"
)
) << Modify;

Re: SQL Custom and Eval / Parse (and \!\!\!...)

ih — Mon, 07 Dec 2020 18:38:04 GMT

I agree that this syntax is only obvious once you understand it, and even then it is confusing. Check out this wish list item to introduce an Eval Expr First() function, maybe that would be more intuitive?

Re: SQL Custom and Eval / Parse (and \!\!\!...)

Georg — Mon, 07 Dec 2020 22:05:56 GMT

I like to follow a different path avoiding unintuitive eval parse combinations, where I find them quite useful for other purposes.

Often I have complex queries (built in plain SQL by other Tools like SQL developer). And I Need to modify some variables.

The attached example Shows how it works with big class datatable, howewer it would work for all ODBC Connections in a similar way.

Simply I prepare my Connection string (not used here) and the SQL Query in a string variable, given to the "New SQL Query".object.

I use "assign" (instead of "=") because it works nice for large queries and Code folding.

And I'm more flexible for also using open database function, or processing the SQL in other ways.

Usually the double quotes are not needed in a plain SQL (and I try to avoid also), unless you have names with spaces etc. like "Big Class".

Names Default To Here( 1 );

cdt = Open( "$Sample_Data\Big Class.jmp" );
age = "15";

// define custom SQL with placeholder
Assign( sql_str, "SELECT t1.name, t1.age, t1.sex, t1.height, 
	t1.weight 
FROM \!"Big Class\!"  t1 
WHERE  (  (  ( t1.age IN  ( &age )  )  )  ) ;" );

// fill placeholder
Substitute Into( sql_str, "&age", "15" );

// execute query
dt_age = New SQL Query( Version( 130 ), Connection( "JMP" ), JMP Tables( cdt ), QueryName( "aged" || "_" || age ), CustomSQL( sql_str ) ) << Run;

Re: SQL Custom and Eval / Parse (and \!\!\!...)

Franck_R — Tue, 08 Dec 2020 07:34:32 GMT

Great, that's really very useful! And since we've just created the database and I was the main actor of this creation, I've just made sure that no table name or variable name has parentheses or spaces, so it should simplify this generalisation.

Re: SQL Custom and Eval / Parse (and \!\!\!...)

ReliabilityWolf — Wed, 26 Jun 2024 15:09:36 GMT

@pmroz I tried the similar thing as above. after custom SQL run, returning the result is 1 in the end of log. could you please help explain more about the result of 1? is it Eval( Parse( gbStr ) ) return code that you mentioned in your script?

Re: SQL Custom and Eval / Parse (and \!\!\!...)

pmroz — Wed, 26 Jun 2024 15:14:54 GMT

Sorry I don't use Query Builder. I would guess that the return value of 1 is true indicating success. But that could be a wild surmise :)

Re: SQL Custom and Eval / Parse (and \!\!\!...)

ReliabilityWolf — Thu, 27 Jun 2024 02:24:08 GMT

@pmroz thanks for your reply. May I have your support to review the following script?

SQL Query usually takes a few minutes to access database. deleting rows (if :rank >1) sometimes doesn't execute but can do manually. I have no idea.... not sure if my script has any problem?

date_deb = "to_date('01-Mar-2019', 'dd-mon-yyyy')";
gbStr = Eval Insert("\[
dt1 = New SQL Query(
	Connection(
		"ODBC:blabla"
	),
	QueryName( "v_test_run" ),
	CustomSQL(
		"SELECT "t1".lot_id 
FROM "public"."v_test_run"  "t1" 
WHERE  (  (  ( "t1".test_run_timestamp >= ^date_deb^ )  )  ) ;"
)
) << Run foreground;
]\");

write(gbstr);

Eval( Parse( gbStr ) );

dt1 << New Column( "Rank", Numeric, "Continuous", Format( "Best", 12 ) );
dt1:Rank << set Formula( Col Rank( :DATE* -1, :ID ) );
dt1 << Row Selection(select where(:Rank > 1 ));
dt1 << delete rows;

Re: SQL Custom and Eval / Parse (and \!\!\!...)

Georg — Thu, 27 Jun 2024 06:56:41 GMT

I think formula evaluation takes some time,

a wait(0) would force JMP to finish until row selection is done (see script) and deletion can be done.

I would really encourage you to keep the string part as small as possible, as it is always error prone and difficult to read.

As well it is a great possiblity to use JMP Tables to figure out these kind of problems, and to write scripts for the community that everyone can run out of the box.

Edited: replaced "\C:\Program Files\JMP\JMPPRO\18\Samples\Data" by "$SAMPLE_DATA"

Names Default To Here( 1 );

// date_deb = "to_date('01-Mar-2019', 'dd-mon-yyyy')";

date_str = "2008-06-19";

/*
gbStr = Eval Insert("\[
dt1 = New SQL Query(
	Connection(
		"ODBC:blabla"
	),
	QueryName( "v_test_run" ),
	CustomSQL(
		"SELECT "t1".lot_id 
FROM "public"."v_test_run"  "t1" 
WHERE  (  (  ( "t1".test_run_timestamp >= ^date_deb^ )  )  ) ;"
)
) << Run foreground;
]\");
*/

dt1_sql = Eval Insert(
	"\[SELECT t1.Date, t1.State, t1."Flu Cases", t1.Region, 
	t1."Population (July 2009)" 
FROM "Seasonal Flu"  t1 
WHERE  (  (  ( t1.Date >= JMPDATE('^date_str^') )  )  ) ;]\"
);

dt1 = New SQL Query(
	Version( 130 ),
	Connection( "JMP" ),
	JMP Tables( ["Seasonal Flu" => "$SAMPLE_DATA\Seasonal Flu.jmp"] ),
	QueryName( "SQLQuery1" ),
	CustomSQL( dt1_sql )
) << run;

// Write( gbstr );
// Eval( Parse( gbStr ) );

dt1 << New Column( "Rank", Numeric, "Continuous", Format( "Best", 12 ) );
dt1:Rank << set Formula( Col Rank( :DATE * -1, /* :ID */ :State) );
wait(0);
dt1 << Row Selection( select where( :Rank > 1 ) );
dt1 << delete rows;

Re: SQL Custom and Eval / Parse (and \!\!\!...)

ReliabilityWolf — Thu, 27 Jun 2024 09:44:23 GMT

@Georg thanks for suggestion. it's really a good and smart idea for keeping the string part as small as possible. i tried it, and deletion still fail with wait(0). but it works for wait(1). however, when I access more size data from SQL datacenter (likely it takes a few mins to fetch data from server), it works until wait(5). my guess is col rank() usually takes some time to calculate. more data size, more time for calculation. when row selection execute before col rank calculation finish, deletion will fail. is there any way to determine the calculation complete? if yes, I can use IF function to run next step.

Re: SQL Custom and Eval / Parse (and \!\!\!...)

Georg — Thu, 27 Jun 2024 11:10:29 GMT

@ReliabilityWolf generally I would try to do the filtering at database, so it depends on the sql language you have how to do (you can use rank with subselect in sql as well). And you gain a lot of time for unnecessary traffic.

Additionally I would avoid using a formula for this, if you want to have unique values for the ID, you can sort in SQL and use select duplicate rows and delete, see below. Database usually is much faster doing these tasks than your client.

Regarding formulas, if you need, perhaps the table message "run formulas" or "rerun formulas" helps. Or you can delete formula, or write the results directly into the column ...

Names Default To Here( 1 );

date_str = "2008-06-19";

dt1_sql = Eval Insert(
	"\[SELECT t1.Date, t1.State, t1."Flu Cases", t1.Region, 
	t1."Population (July 2009)" 
FROM "Seasonal Flu"  t1 
WHERE  (  (  ( t1.Date >= JMPDATE('^date_str^') )  )  )
order by
t1.State, t1.date desc;]\"
);

dt1 = New SQL Query(
	Version( 130 ),
	Connection( "JMP" ),
	JMP Tables( ["Seasonal Flu" => "$SAMPLE_DATA\Seasonal Flu.jmp"] ),
	QueryName( "SQLQuery1" ),
	CustomSQL( dt1_sql )
) << run;

dt1 << Select Duplicate Rows( Match( :State ) ) << delete rows();

Re: SQL Custom and Eval / Parse (and \!\!\!...)

ReliabilityWolf — Thu, 27 Jun 2024 13:41:28 GMT

really appreciate your new direction on filtering at database. let me take a look at this. thanks again.