Check the Scripting Index example for the Box<<MakeRowStateHandler message. This allows you to hook up a JSL callback when the filter row states change. Unfortunately, the Triangulation object will use the global row states for the computation. I think you would have to get the subset of values and pass in arrays of coordinates instead of column names for the X's.
Or....
In JMP 15 Graph Builder the convex hull is directly available as part of the Contour element, when used with a color role. In this case the contour is based on a triangulation, and you can turn off the contours and show only the boundary. This script in JMP 15 will demonstrate the feature:
Open("$SAMPLE_DATA/Big Class.jmp");
Graph Builder(
Size( 534, 456 ),
Show Control Panel( 0 ),
Variables( X( :height ), Y( :weight ), Color( :height ) ),
Elements(
Contour( X, Y, Legend( 4 ), Fill( 0 ) ),
Points( X, Y, Color( 0 ), Legend( 5 ) )
),
Local Data Filter(
Add Filter(
columns( :age ),
Where( :age == 13 ),
Display( :age, Height( 102 ) )
)
)
);
One big advantage of the built-in version (if using JMP 15 of course...) is that all of the Graph Builder grouping computations are already taken care of. So, if you use another column in the overlay role, you will get a convex hull for each group: