topic Re: Using the Prediction Profiler to optimize for a certain range in Discussions

Using the Prediction Profiler to optimize for a certain range

Marco_ — Sun, 11 Jun 2023 11:25:37 GMT

Hi everyone,

I'm trying to use the prediction profiler in JMP in combination with the desirability function in order to optimize several models simulatanously.
There are several optimisation options in JMP implemented like maximizing, minimizing or matching a target. However, I need an option to match a range.

Since there are only 3 adjustable points in the desitability function and I need a desirability function where all values outside the target ranges are equal to 0 and within equal to 1, I need your help how I can achieve this ( the function should look similar to a rectangle)

Please note, I'm familiar with the option to set specification limits in order to perform simulations or to calculate defect rates. These static plots are unfortunately not a big help for me. What I need is the interactivity of the desirability plot.

I'm looking forward to your suggestions!

Thanks!

Re: Using the Prediction Profiler to optimize for a certain range

Phil_Kay — Mon, 11 Jul 2022 10:04:24 GMT

Hi,

I think the solution here is to define a custom desirability function as a formula column.

In the attached example using the Tiretread data I created a custom desirability function for ELONG which is 1 for the range 450 to 550 and 0 outside of that range.

If( 450 < :Pred Formula ELONG < 550,
	1,
	0
)

Then we profile the prediction formula column for ELONG and the custom desirability function. Set the desirability function for ELONG to none and Maximise for the custom desirability function.

I think that should work. Let me know.

Phil

Re: Using the Prediction Profiler to optimize for a certain range

Marco_ — Mon, 11 Jul 2022 11:03:35 GMT

That sounds super interesting. I will give it a try - thanks for your fast reply!

Re: Using the Prediction Profiler to optimize for a certain range

SamGardner — Mon, 11 Jul 2022 14:18:30 GMT

One thing you will need to careful of is that the custom desirability function that @Phil_Kay describes is flat over a range, so there are possibly many optimal solutions (with a maximum desirability). To visualize factor regions where the desirability is constant and at the maximum, you could consider using the "Output Random Table" command under the Profiler and then make a scatterplot matrix of your factors, and filter that plot on the desirability.

Re: Using the Prediction Profiler to optimize for a certain range

Marco_ — Wed, 13 Jul 2022 15:23:45 GMT

Hi Phil,
in principal your idea works for my probleme. However, reality is sometimes more complex than expected...
For my approach, I have to consider also the uncertainity interval around my model. JMP only allows me to use CIs. However, with a simple trick where I save the PredSE Formula and add the MSE to that formula (see picture below with the example of 9.1816) I can also approximate the 95% prediction interval (PI).

So what I actually want to know is, where does my PI intersect with my specification limits in order to define an acceptable input-parameter range.
I could do this with you approach by:

Create a new column where I have to multiply the "adjusted" PredSE column with the correct t value to consider the degrees of freedom of my model. Lets call this column "Interval"
Add the value from the Interval column to the :Pred Formula column and check whether the upper PI falls within the Specs
Substract from the :Pred Formula the value from the Interval column and check if also the lower PI falls within the specs.
If 2 and 3 are true => 1 and if not => 0

Since I have to perform this process on multiple outputs/models it is too much effort to this manually. Especially the degrees of freedom in each of my models are different and I would need to adapt the value manually. Do you have any idea how I can automate this process or do you have a different idea how I can tackle this probleme?

Re: Using the Prediction Profiler to optimize for a certain range

Phil_Kay — Thu, 14 Jul 2022 09:36:29 GMT

Interesting questions and challenge that you have there, @Marco. Unfortunately I don't think that I have time to fully explore the solution options at the moment. I think that @SamGardner 's suggestion of using Monte Carlo filtering might be a useful solution. I also think that the new design space capabilities that are currently being evaluated in the beta version of JMP 17 could be interesting for you. You might want to contact your JMP representative to ask about the Early Adopter program for JMP 17 in order to see if these capabilities can help in your situation and to provide useful feedback to the JMP Development team.

Re: Using the Prediction Profiler to optimize for a certain range

Marco_ — Thu, 14 Jul 2022 13:04:51 GMT

@Phil_Kay well thats a pitty! Hopefully you have more time in the future, since I would really appreciate a solution option from you.

By the way, I have already access to the JMP EA 17. However, you have to help me a bit where I can find your mentioned option "design space capabilities".

I have also investigated the solution from @SamGardner which is really interesting and might be handy for some other challenges I'm facing. However, it does not really tackle my problem since I'm looking for an analytical solution instead of a simulation driven one. Especially because JMP does simply use the RMSE in the Random by Model option and not the correlation structure. Thus, the simulation results will only be correct in cases where the investigated design is completly orthogonal.

Re: Using the Prediction Profiler to optimize for a certain range

Phil_Kay — Thu, 14 Jul 2022 13:14:50 GMT

Great news that you already have the JMP17EA.

When you log in to the Early Adopter web page look under Videos > EA2 > Design Space Profiler. That is a video from the developer that is working on that capability (one of most experienced developers ;)).

I have a long train journey tomorrow so I might get chance to think about this. I can't make any promises though. (I have promised to do too many things this week!)

Re: Using the Prediction Profiler to optimize for a certain range

Marco_ — Thu, 14 Jul 2022 14:42:53 GMT

@Phil_Kay
Just watched the video and it was really interesting. However, at first glance it seems that the model uncertainty is only implemented in form of the RMSE. Thus, no correlation with respect to the input parameters is taken into account. I need to look into this more closely and will let you guys know how helpful this feature is for my problems. Is there any chanche for an exchange with @John_Sall ?

Re: Using the Prediction Profiler to optimize for a certain range

Phil_Kay — Thu, 14 Jul 2022 15:22:57 GMT

He's quite busy! I think the best thing to do start with is to use jmpbeta@jmp.com to provide feedback and ask about how these capabilities might/might not address these needs.

Re: Using the Prediction Profiler to optimize for a certain range

SamGardner — Thu, 14 Jul 2022 17:46:46 GMT

@Marco_ if you could go to the JMP Wish List https://community.jmp.com/t5/JMP-Wish-List/idb-p/jmp-wish-list and provide your feature request, that would be helpful for use to keep track of this and to be able to better evaluate it. You can also reach out to me directly via PM on the Community and we could discuss this further. I am interested to find out more specifics about what you want to do.

Re: Using the Prediction Profiler to optimize for a certain range

Marco_ — Fri, 15 Jul 2022 09:09:03 GMT

@SamGardner
Instead of writing you a PM I thought it also might be more helpful for other JMP users if we keep the discussion public. I also google very frequently questions I have and I'm always happy if I find a post in the community ;)

I spent now some time with prediction profiler and you guys have done a really great job there. This feature is really great to tackle QbD or PAR requests from authorities such as the FDA or EMA. However, at the moment is see two big problems why I cannot use the feature.

1. As I already mentioned in my last post, I'm afraid that the model uncertainty in the simulation is underestimated. @John_Sall explained at the end of this video how to incorporate the model uncertainty within PredFormula. I could reproduce his example. But, the used RMSE of the model does not consider the correlation structure within the model. You anticipate that the model uncertainty across the whole investigated design space (lets call the design matrix X) is constant. This however is in general not true, since most of the models are not completely orthogonal regarding their input-parameters. See below a short equation how the uncertainty of an estimated coefficient is dependent on the design correlation structure.

Since JMP allows easily to save the PredSE formula of each model, where the correlation structure is considered. I wonder why the formula is not used for the simulation within the DesignSpace profiler? Instead of using a constant SD, simply connect the PredSE formula here and the issue should be solved.

2. The other point is that by using the RMSE of the model as the SD for the simulation within the DesignSpace Profiler, the uncertainty in the model mean prediction is neglected. From my understanding the simulation anticipates that the mean prediction of the model has no error and only considers the uncertainty around this model prediction. To make my point a bit clearer, see the following formula for the calculation of a prediction interval (just copy/pasted from a paper):

At the moment you are only considering the first part of the equation but not the second part (i.e. the standard error).

To sum up my points. I think at the present status, the DesignSpace profiler underestimates the true variability and thus allow to wide ranges for the input-parameters. I think both points could easily be fixed and I would much appreciate your feedback (also from @Phil_Kay if he has some time during his train trip ;) )