topic Re: How to force I-Optimal algorithm in Custom Design to choose integers only? in Discussions

How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Thu, 30 May 2019 16:38:32 GMT

Hello,

I am working on a custom I-optimal design with two factors (time and temperature). I would like to force the algorithm to choose an I-optimal design where all temperatures are based on integers (e.g., 0, 50, 75, 100) since I can only set my equipment to whole temperature values.

I haven't been able to find an option that would force the algorithm to do this. When I simply round the temperature values of the design that has been found to integer values and then look at the average variance of the prediction, it becomes much worse. For this reason, I'd like the optimization algorithm to choose the integers in the first place, in hopes that a better average prediction variance would be reached overall. Please let me know how this is possible, as it really seems like it should be, but I can't figure out how. Thank you!

Alan

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Thu, 30 May 2019 16:52:33 GMT

Define your factors to be Discrete Numeric type.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Thu, 30 May 2019 17:34:01 GMT

Hi Mark,

If I do that, then I have to tell the software that there are ~130 discrete levels to cover the range, and then I have to manually change each of them, but I can't actually see what each one is since all 130 are crammed into a small space of the GUI. Additionally, if I am using equipment that allows for 0.5 degree increments, that doubles the levels, making things even more difficult and inefficient to enter. One other substantial problem is that I have an irregular experimental region with several linear constraints, and when using discrete numeric factors for temperature, JMP no longer lets me include the temperature factor in my linear constraints. Isn't there another way to do this by keeping temperature set to continuous and just adding an argument somehow that selections must be integers?

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Dan_Obermiller — Thu, 30 May 2019 18:25:55 GMT

One possible approach: Make a data table that contains your possible temperature values. It seems like using a sequence would work well. Then in custom design include the temperature as a Covariate. JMP will ask for the data table column to use, so point to the one you made.

That being said, I would be concerned with the design if rounding the temperatures to an integer value drastically changes your prediction variance. With the extra information of the constraints you have in place, that tells me that you have a very unstable design space. You will REALLY need to be careful in validating your model once you have completed your design and analysis.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Thu, 30 May 2019 20:51:22 GMT

You misunderstood me. I apologize. I did not mean that your factor must be defined such that it includes all the discrete temperatures that are possible. It should only the ones that you want to test. Maybe four discrete levels?

Alternatively, why not specify temperature as a continuous factor, define the range of temperature with low and high, and then define the model with the potential terms? JMP will find the optimal design but if one of the temperature levels is not achievable, change it to the closest setting that you actually use before the analysis. This way works best if you have some linear constraints, too.

Really sorry about the confusion.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Thu, 30 May 2019 21:01:01 GMT

Hi Dan,

Thank you for your input. I'll keep your concerns in mind.

As for solving my immediate problem, as you suggest, I have added the temperature as a covariate based on a table of values, but JMP is now no longer allowing me to create linear constraints that include temperature, but only the continuous factor (time). I need to be able to set constraints based on temp x time combinations. Is there something obvious that I am missing?

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Thu, 30 May 2019 21:07:34 GMT

Hi Mark,

No worries. I played around with doing things this way at first, and it does avoid most of the problems. However, my concern was that in constraining the possible temperatures that the I-Optimal algorithm would be severly hampered in finding the lowest average prediction variance. Do you suspect that this will not be the case as long as it has the full range of values for the continuous factor (i.e. time)? I'll try to run a couple of tests for comparison to check it out.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Thu, 30 May 2019 21:25:10 GMT

Hi Mark,

I added temp as an 8-level categorical factor, but again, JMP will now not let me include linear constraints of time x temp combinations. Advice is welcome.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Fri, 31 May 2019 11:53:58 GMT

I would not worry about the optimization. Yes, by definition, adding linear constraints will constrain the optimizer, right? But it is built for such a case.

Not sure what you mean, though, by "severely hamper" but then I do not know what the set of constraints might be.

So, I think that the most straight-forward approach to designing your experiment is to enter both factors as continuous type and define them to have the full range of levels. Define your constraints. Define your model. This way is actually the normal approach. I think that using covariate or discrete numeric factors is 'over thinking' the problem or 'gaming' the optimizer in non-productive ways. It isn't wrong or illegal. It just doesn't work as well.

Let's use custom design as it was intended.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Fri, 31 May 2019 11:56:15 GMT

My advice is to scrap that approach and jettison the previous ventures. Start over. My previous post suggests that we take a straight-forward approach using custom design as it was intended. Use continuous factors and linear constraints. Specify your linear model. You can always round or change the levels from custom design to suit the practical limitations of your equipment later.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Fri, 31 May 2019 13:29:16 GMT

Hi Mark,

I am happy to do that, but this brings me back to where I started. To quantify what is going on in order to have you advise if I am simply being overly concerned, when I set both time and temperature to continuous and enter the linear model, along with the linear constraints, I end up with an average prediction variance of ~0.4. When I round the values in the table afterwards, it increases to ~2.8. This is what concerned me initially since the average prediction variance is ~7 times larger. I imagine that if the difference expected in the response is large that this may be less of an issue, but I am not certain that this will be the case. I am interested to hear your thoughts.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Dan_Obermiller — Fri, 31 May 2019 13:57:01 GMT

Personally, I agree with Mark's advice. The problem you are fighting is the definition of continuous. Is the factor continuous or not? It is, so it should be treated that way. I understand the average prediction variance increases quite a lot, but that is due to the constraints that you are putting on the system including the integer-based temperatures. What about the maximum prediction variance? What about the minimum prediction variance? More specifically, how do the Fraction of Design Space plots compare? For this situation, I think that picture will tell a better story than just the average prediction variance. You really want to see the "distribution" of prediction variances, not just a single point.

When you change from fractional temperatures to an integer, my guess would be that the maximum prediction variance change is large causing the average to go up. Remember, with the linear constraints your design space is smaller, so a change out at the high "fraction of space" will have a bigger impact on the average.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Fri, 31 May 2019 14:45:24 GMT

Can you show the I-optimal design from JMP data table before) and then the resulting design after you round the levels (data table after)? I am surprised that rounding to the nearest degree over such a wide range would cause the prediction variance to increase that much.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Fri, 31 May 2019 17:12:42 GMT

Hi Dan and Mark,

Thank you once again for your feedback. As I was preparing some screenshots, I noticed something odd that may be contributing to the problem. When I first generate the design, if I look at the design evaluation I get an average prediction variance calculation of ~0.4. However, if I then click on the "Make a Table" at the bottom of the dialog box and then click on "evaluate design" on the right, prior to rounding anything, then the calculated average prediction variance (APV) increases to ~1.4. I have reproduced this multiple times. Given that both designs are identical, I'm not sure why the APV is changing, let alone changing that much. I'm running short on time, so I'll have to post the screen shots later today, but if you have any idea where this changing APV issue could be coming from, maybe I need to solve it first.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Fri, 31 May 2019 18:15:07 GMT

I have not seen this behavior before. Then again, I wasn't looking for it like you!

I would report this anomaly to support@jmp.com for investigation. I expect adding constraints to introduce correlations among the estimates that affect the performance of the design. I do NOT expect making a data table with the design to affect the performance!

(One other thought: right-click on the table in the Design section of custom design and select Make Into Data Table. Use the design evaluation command in the DOE menu to see if you get the same results as in the custom design platform. Repeat this examination with the data table produced by clicking Make Table. What do you see?)

I would also proceed to use the design as it is in the data table. I am confident in the approach that you used. As always, be careful when performing each run and during the analysis.

Please report here anything that you might learn from JMP Technical Support. Of course, do not hesitate any more questions that you might have as you proceed.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Fri, 31 May 2019 19:00:08 GMT

Hi Mark,

This is really interesting. So, as I mentioned, the APV was ~0.4 initially in design dialog box right after the design was created. If I clicked the "Create a Table" button and then chose "Design Evaluation" on the left, I got an APV of ~1.4. However, if I do as you suggest and create a table by right clicking on the table in the design dialogue box, and then selecting "Evaluate Design" under DOE, the APV is then ~0.2. Three different APVs for identical designs. I think I will go ahead and report this to support@jmp.com to see what they have to say. While I'm not ruling out user error, I'm not sure what it could be at this point. I'll be back in touch once this is, hopefully, sorted out.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Sat, 01 Jun 2019 15:29:05 GMT

Hi all,

I have discovered a couple more interesting things, so I wanted to take a moment to update this thread in case it will help anyone in the future:

1) It turns out that when right-clicking to create the table and then selecting design evaluation from the DOE menu, somehow the higher order model terms are dropped. This is why the average prediction variance (APV) drops considerably to ~0.2. So obviously this figure is not correct. It is inefficient to have to add all the terms back, so this is probably not the best approach.

2) If I create the custom model and save it, then round all the values and save it, and use the "Compare Designs" feature under DOE < Design Diagnostics, then the two models are virtually identical in every respect, including APV, which in this case is about ~0.5. Of course they aren't exactly the same, but certainly the APV of the rounded model is _not_ 7 times larger. So, this appears to be the only reasonable way of comparing the APV of two models.

Ultimately, both of you are correct that sticking with "continuous" for the factors and rounding doesn't make a huge difference, but it isn't necessarily obvious given the APV weirdness unless the Compare Designs tool is used.

Anyway, thank you for your help. I doubt that I would have figured this out without all of the feedback.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

Mark_Bailey — Mon, 03 Jun 2019 11:21:32 GMT

Please report this phenomenon to JMP Technical Support as promised. This behavior is not good and such a work-around should not be necessary.

Re: How to force I-Optimal algorithm in Custom Design to choose integers only?

APM — Mon, 03 Jun 2019 12:45:09 GMT

I reported the problem(s) on Friday and also pointed them to this thread for more information.