Need Guidance on Analyzing a DSD and Transitioning to an RSM Model

[#1]

Hello everyone,

I hope you’re doing well.

I posted a question regarding DSD on JMP User community, but it was reported as spam for some reason? I will repost it here in hopes that someone might help.

I’m a master’s student working on a research project that uses a Definitive Screening Design (DSD) in JMP to optimize a Vertical Axis Wind Turbine (VAWT). I have completed all 22 runs for my 8-factor DSD, but I am struggling with the analysis and next steps.

The thesis deadline is in 2 weeks, and I am quite lost if it's correct to transform my DSD results into RSM without further augmentation.

Any brief insight or recommended resource would be helpful!

Thank you so much!

Regards,

[#2]

Hi @Mohnasre,

Yes, sometimes the automatic filtering on the Community is a bit hard, so your message may have been falsely tag as "spam".

Concerning your topic and DSD analysis, there are very few information to help you, and you particularly don't explain anything about the study context, objective of the study, and your experience with DoE.

Definitive Screening Designs are foremost screening designs (as the name suggests), but they are also able to detect strong interaction and quadratic effects. You can read more about their performance and their detection abilities (as well as the possibility to fit a full RSM model from them) here. The possibility to fit a full RSM model from your design depends on the number of runs (here 22, including the 4 extra runs for 8 factors), and the number of active main effects detected. From JMP Help documentation, if you have less than 4 significant main effects (from your 8 factors), you could fit a RSM model on your data.

Concerning the analysis, I would recommend starting with :
- Fit Definitive Screening platform, the default platform used for the analysis of DSD. A script to run this analysis is present in the data table when creating this design.
- Fit Two Level Screening platform, that can effectively detect main effects as well as interaction and quadratic effects.

There are more estimation methods available in the Fit Model platform to fit Least Squares models, as well as Generalized regression models (with JMP Pro) using Two Stage Forward selection or Pruned Forward selection estimation methods.

Compare the models found by these platforms and see where they agree and differ. If the models are different, verify that the regression models do respect regression assumptions (with residuals analysis) before comparing their performances in terms of explainative (with R2 and R2 adjusted metrics, to be maximised with the lowest difference between the two) and predictive (with RMSE metric, to be minimized) performances, as well as model complexity (with information criteria like AICc and BIC, to be minimized).

Depending on the number of main effects detected in the model and precision of the model, you'll be able to evaluate if you have sufficiently answered your objective through this DSD with a useful and relevant model, or if an augmentation phase may be required to fit a more complete and precise model.

Hope these few suggestions may help you. If you need more precise feedback and help, you can provide more information and context, and/or share your (anonimized) data table.

Best,

[#3]

Thank you so much @Victor_G  for the detailed response!

The goal of the thesis is to improve the power coefficient (efficiency) of the VAWT. This will be done by optimizing the parameters of tip devices added at the tip of each blade. There were initially a lot of parameters, so I had to fix some, and I was left with 8 (7 continuous and 1 categorial). The first idea was to apply a screening design to decrease the number of parameters, and then optimize them using RSM. DSD outperforms other screening designs, especially that it allows to fit full RSM when number of parameters is 4 or less, which is why I chose this design.

I don't have much knowledge about DOE or DSD, but I have learned few things by reading some webinars slides. 

Going back to my results, I have found the following:

1. When I fit the DSD with 95% confidence interval (p=0.05), I get 2 main factors only (let's call them A and B, where B is the categorial factor). R^2 is 0.56, and adj R^2 is 0.45
2. When I fit DSD with 90% CI (p=0.1), I get 4 main factors (A,B,C,D). R^2 is 0.89, and adj R^2 is 0.82. 

In both cases, it seems like no further augmentation is needed to optimize the parameters.

(Please note that I don't know how to use AIC or BIC that you mentioned)

But my questions are:

• Should I choose the 2nd option, where p = 0.1 ?
• And afterwards, how do I fit RSM? Do I use a platform other than DSD where I take the main factors I got? Or is there a way to do it directly from the DSD platform.
• Finally, lets say I finished the RSM fitting and I got the optimized values for parameters (A,B,C,D). If I want to verify the predicted power coefficient (response) by running few other CFD simulations on ANSYS, what level for the other parameters (E,F,G,H) should I choose? Should I choose the center values for these parameters while running the CFD simulations? (Note that I need all 8 parameters to draw the tip sails on the vertical axis wind turbines).

Thank you again for your help, I really appreciate it!

Best Regards,

[#4]

You can try the two platforms I mentioned to see if you can have additional models suggestions.
Concerning the Fit Definitive Screening and the two settings you have tested regarding the threshold for statistical significance, you can check this conversation that may help you :  Setting Stage 1 P value in Analyis of DSD at a high level to detect active effects

AICc and BIC are information criterion that evaluate the trade-off between the complexity of the model and it's accuracy. They can be helpful metrics when comparing several models, in order to find the one(s) that have a good fit with a low complexity (low value of AICc/BIC).

In your situation, as you're in a screening stage, I would consider the second model more useful as the first one, as R2/R2 adjusted are higher and the difference between the two seems low, indicating a well adjusted model. 
From a more practical perspective and answer to your question :

• You can fit the model found by the Fit Definitive Screening by clicking on "Run Model" in the "Combined Model Parameter Estimates". You also have a similar option in Fit Two Level platform to Make or Run the model with the effects detected. Once the model is run, you can check that it does respect regression assumptions.
• There is no definitive answer to this question, you can choose the levels so that they are the most practical to use, so middle levels or low levels (in case these low levels are at 0 for example and it may facilitate CFD calculations). Once you have found the optimal settings for your factors, run the CFD simulations to validate the settings do represent your optimum, and that your model is able to predict accurately the optimum.

Have fun !

[#5]

Interesting case! Great that you screened 4 main effects so you can fit a RSM. Are all interaction effects & quadratic significant? I would eleminate weak interactions and powers to avoid overfitting. More runs by augmenting are required to validate these effects Good luck with your master thesis! 

[#6]

I had a DSD case with 7 factors where I had to increase threshold p-value setting to 0,4 (!) to filter out the main effects. I double checked reliability of this screening using the Fit two level screening mentioned by Victor and results were the same;  so I was confident about the first DSD screening. Unfortunately I got 5 active factors so had to augment my DSD with 4 additional runs to de-alias interaction effects for upgrading the model to RSM. This worked out fine!

[#7]

Hello @frankderuyck ,

No, not all interactions are available in the 2nd model that I tried (p=0.1), there are only 2 interactions and 2 squared terms (A*B, B*C, B^2 , C^2) with the 4 main factors (A,B,C,D).

Since I got 3 continuous factors (A,C,D) and 1 categorial (B), I think there is no need for augmentation to fit a RSM, the high level (+1) of the categorial factor B is clearly better than the low level (-1) for all the cases I tried. 

Thank you for your insightful reply!! I appreciate it

[#8]

Thank you so much @Victor_G for your valuable help!  Your support is greatly appreciated!

I will definitely check today all the links you mentioned.

One last question please, is Box-Cox transformation any important in my case? It shows me that λ = -2 is the best value to reduce SSE as shown in screenshot attached. I tried it and it gave R^2 = 0.92 and R^2 adj = 0.85

[#9]

You can try the Box-Cox transformation with lambda=-2 and compare the original and transformed model to see which one better respects the regression assumptions: plot actual by predicted and residuals analysis to see if the residuals have a random and normal pattern with homogeneous variance (homoscedasticity). You can then compare the performances of the models.
Transformations are helpful to better align the models with regression assumptions and to simplify them.

With a lambda value of -2, this is equivalent to transforming Y into 1/Y2 : https://www.statisticshowto.com/probability-and-statistics/normal-distributions/box-cox-transformati...

Hope these answers will help you analyze your results !

[#10]

Hello again @Victor_G ,

I tried the Box-Cox transformation and it gave better results in terms of R^2, R^2 adj, RMSE, AICc and BIC.

Now I want to fit the RSM, if I go to analyze-->Fit model  what should I choose as the y variable? Do I choose the original power coefficient I got from CFD simulations? Or do I choose the predicted expression that I got from Box-Cox?

I appreciate your help and support! I don't know what I would've done without your help :)

[#11]

Hello @Mohnasre,

Ok, it sounds like the Transformation may be helpful in your case. Did it affect positively the residuals (random pattern, homoscedasticity, normal distribution of the residuals) ?

Not sure to fully understand your question ? Once you have detected the terms in your model through the Fit Definitive Screening and/or Fit Two Levels Screening platforms, you can make or run directly the model through the integrated Fit Model platform. From there, you can evaluate which lambda value is the most appropriate in Box-Cox transofrmation, and choose the option "Replace with Transform". Now you should have the final model you mentioned, and from there, you can save the Prediction Formula to have the equation directly in your data table. Note that you don't need any "back"-transformation or anything to come back to the original unit, as JMP already does it when saving the Prediction Formula. It's also done directly when checking the Prediction Profiler. In your case, you will see in the Prediction Formula column that if you have chosen a lambda value of -2, JMP will add the term "1/square(expression)" (or simply square(expression) with exposant -2) before the equation to do this back-transformation in original units.

If you are mentioning the Show Prediction Expression available in Fit Model, this equation is the equation fitted on the data after the transformation, so if you're planning to use it, don't forget to back-transform it in your original unit.

Hope this answer will help you finalize your subject,

[#12]

Hello @Victor_G ,

Yes the transformation did improve residual behavior as well.

Ok, so just to clarify, there is no need to use the Fit Model platform with the RSM macros, correct?

I was under the impression that the DSD platform is mainly for screening, and that after identifying the active factors I would need to build a separate RSM model using "Analyze → Fit Model" and the “Response Surface” macro.

So if I understand you correctly, after applying the Box–Cox transformation in the DSD platform and accepting the transformed model, I already have the equivalent of an “RSM-level” model for my active factors, and there is no additional RSM fitting step needed?

Sorry for the basic question, I just want to make sure I fully understand the correct workflow.

Thanks again for your valuable help!

[#13]

By the way, I tried using analyze --> fit model with RSM macros. I used the "predicted response" I got from the Box-Cox transformation as the y response.

This fitted model had 2 more active interaction terms (compared to the final DSD model), with higher R^2, R^2 adj, and lower AICc, BIC and RMSE. 

But I felt something is wrong using the "predicted response" as the Y variable to be predicted again. That's why I asked you if there is a need to fit RSM on another platform after the Box Cox transformation.

Thank you

[#14 · Accepted solution]

Hi @Mohnasre,

The Fit Model platform with the RSM macro will launch the fitting of a model with all main effects, interaction effects and quadratic effects of the factors you have. If you already have worked on the identification of active terms through any other platform, there is no need to repeat this terms identification/filtering and model fitting operation. However, you can try several platforms in paralell to see how the models agree and differ.

Yes, DSD is mostly for screening, but in situations where you have a relatively low number of active effects and/or high number of factors to screen, the DSD may be sufficient to fit a custom RSM model on the most important factors. 

So to answer your question directly, there is no additional RSM fitting step needed once you already built your custom model through other platform. The use of other platforms like Fit DSD or Fit Two Level to fit models is more appropriate or easier when facing different design situations.

Concerning your second message, I wouldn't try to model a predicted response, as you won't model any variability coming from the CFD calculations. So you might end up with an apparently better model, as you're fitting a model on a predicted "perfect" response coming already from a model. But you can't compare a model predicting a "pure" perfect predicted response, and a model predicting a measured response from CFD calculations. You can still try to use other modeling platform (did you use Fit Two Levels ? It's very effective at identifying interaction and quadratic effects), to see if you have not missed any terms during model building.

Hope this answer will help you get your final result.

[#15]

Thank you so much for your time and support!

I appreciate it!

Have a great day @Victor_G