Choose Language Hide Translation Bar
Highlighted
mandeson91
Community Trekker

Analysis of Opt DOE simulation results

Dear JMPer, 

 

Currently, I'm under analysis a Opt DOE simulation results. However, I don't get consistent result form simulations and also get to different trends during counter profile analysis. I wonder if any methodology or analysis could use to understand which simulation result is valid. 

 Here is my simulation process. The DOE matrix is in Table 1

  1. Fit model (Simulation 1):
    1. Put 4 inputs in Simulation 1 and select surface response mode in macro functions.
    2. Remove interaction option from same factor, eg input A * input A
    3. Run the simulation with standard least squares and effect leverage.
    4. Table 2 shows Rsquare of summary of fit and Output trend based on Input A/C (Input A/C is key factors for the process)
  2. Fit two level screening (Simulation 2):  
    1. Select Output A to run fit two level screening.
    2. Use defined important factors, such as output's interaction and individual outputs. 
    3. Run the simulation with standard least squares and effect leverage. 
    4. Table 2 shows Rsquare of Summary Of Fit and Output trend based on Input A/C (Input A/C is key factors for the process)

Output B and Output C is able to predict the trend based on higher input or lower input (due to physic). However, Output A is the important response that I would like to see what simulation would show the trend. Based on Rsquare of Summary Of Fit, Simulation 1 seems batter than Simulation 2. However, output trend of Simulation 2 is more logical to me than Simulation 1. Therefore, I wonder if anyone could share some input to have better analysis for both simulation results. Please kindly share me your inputs. Much appreciate your time and help. 

 

Thank you, 

 

Bests,
Andy

 

 

Table 1. DOE matrix

Leg#PatternInput AInput BInput CInput DInput EInput FOutput AOutput BOutput C
1+−−++−30020010032510.0%        2,463                 61
2−+−+−+20040010031550.0%        3,572                 45
3++−−++30040010012550.0%        3,990                 47
4−−−−−−20020010011510.0%        3,245                 31
5+−−−−+30020010011550.0%        3,017                 34
6−−−+++20020010032550.0%        3,842                 76
7−+−−+−20040010012510.0%        3,431                 48
8++−+−−30040010031510.0%        2,409                 51
900000025030011022030.0%        3,336                 66
1000000025030011022030.0%        2,564                 66
11−++++−20040012032510.0%        3,289                 62
12−−+−++20020012012550.0%        2,698                 53
13+−++−+30020012031550.0%        3,017                 47
14+++−−−30040012011510.0%        3,460                 31
15−++−−+20040012011552.8%        3,503                 28
16−−++−−20020012031510.0%        2,835                 42
17+−+−+−30020012012510.0%        2,304                 42
18++++++30040012032550.0%        3,431                 69
19POR1503001200007.1%        3,642                 16
20POR1503001200008.3%        3,093                 15

 

Table 2. Simulation 1 and Simulation 2 Results. 

JMP analysisSimulation 1Simulation 1Simulation 1Simulation 2Simulation 2Simulation 2
MatrixOutput AOutput BOutput COutput AOutput BOutput C
Input A highReduce reducetwo trendsincreaseIncreasereduce 
Input A lowincreaseincreasetwo trendsreduceReduceIncrease
Input C highIncreaseIncreasereducereduceIncreaseReduce
Input C lowreducereduceincreaseincreasereduceincrease
Summary of fit
Rsquare
0.960.780.840.950.590.85
0 Kudos
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Analysis of Opt DOE simulation results

You asked, "Based on your experience, should the simulation be focused on main effects which has defined by fit two level screening?"

 

First of all, this is NOT a simulation. It is a regression analysis to fit a linear model.

 

There are several common ways to approach the analysis of an experiment based on a fractional factorial design. You can approach it, as I said, using two stages in which you first select the main effects, believing that the principles of hierarchy holds, and then, add interaction terms for the active factors, believing that the principle of heredity holds. This approach is often a successful way to select the model. This model, like all others, must be verified with new empirical observations (augmented runs).

 

You asked, "Could you share me script for stepwise platform simulation?"

 

You do not need a script. Start your analysis with Fit Model as usual. Specify the response data column in the Y role. Specify the terms in the linear predictor for the Effects. But switch the Personality from Standard Least Squares to Stepwise..

 

Please see JMP > Books > Fitting Linear Models > Stepwise chapter.

Learn it once, use it forever!

View solution in original post

0 Kudos
5 REPLIES 5
Highlighted

Re: Analysis of Opt DOE simulation results

So you designed an experiment for a computer simulation? That is, the factor levels are used as parameter values in the computer simulation?

 

I instead interpret your question to be about regression results. The two-level fractional factorial design in 16 runs with 2 center points and 2 ad hoc runs does not support estimating the full quadratic model. In this case, you must not try to estimate the parameters for the quadratic terms.

 

The Screening platform is intended only for two-level factors. You used a two-level design but you added a third or fourth level with the center points and ad hoc runs. This platform won't necessarily give you wrong information but it won't necessarily give you correct information when there are more than two levels.

 

Still, it can be useful as an exploration tool and a starting point leading to a full regression analysis. I used it to find a model for Output A with the linear predictor including Input E, Input D, Input A, and Input E*Input E. So it detected a non-linear response but attributing it to Input E is arbitrary. It is based on the heredity principle and nothing more. It cannot be established with this experiment. Output B does not appear to have any fixed effects from these factors. Output C has the most complex model. It includes two quadratic effects, which is the most that might be estimated and the choice of which terms is again arbitrary. The usual action at this stage is to augment the two-level design specifically to estimate all the quadratic terms for a final decision.

 

Another approach is to use the Stepwise platform starting with all the potential terms (main effects and interactions) that might be estimated. Here is the prediction profiler launched with the model formulas saved for Output A and Output C using stepwise regression to select the model.

 

Capture.JPG

 

Another approach is to start with the main effects only, remove the unimportant factors, then add and test interactions involving the important factors. This approach is based on the hierarchy and heredity principles.

 

Just remember that there are many confounded and correlated effects using this design. It provides incomplete information about the response. So augmentation might be the best next step to clarify the important effects. Regardless, the final model selected here or after augmentation must be empirically verified with new, independent runs for as yet untested treatments.

Learn it once, use it forever!
0 Kudos
Highlighted
mandeson91
Community Trekker

Re: Analysis of Opt DOE simulation results

@markbailey  Thanks a lot for your detail explanation. I acknowledged inputs for the current matrix design. 

The DOE matrix is designed for fractional factorial with some 2-factor interactions and I tried to define key factors for the process. 

A quick question. Based on your experience, should the simulation be focused on main effects which has defined by fit two level screening? I notice that you would analyze the defined effects and plot prediction profile. 

 

Also, I haven't use stepwise platform before. Could you share me script for stepwise platform simulation? 

 

Once again, thanks for your thoughts and inputs.

 

Thank you, 

 

Bests,
Andy 

0 Kudos
Highlighted

Re: Analysis of Opt DOE simulation results

You asked, "Based on your experience, should the simulation be focused on main effects which has defined by fit two level screening?"

 

First of all, this is NOT a simulation. It is a regression analysis to fit a linear model.

 

There are several common ways to approach the analysis of an experiment based on a fractional factorial design. You can approach it, as I said, using two stages in which you first select the main effects, believing that the principles of hierarchy holds, and then, add interaction terms for the active factors, believing that the principle of heredity holds. This approach is often a successful way to select the model. This model, like all others, must be verified with new empirical observations (augmented runs).

 

You asked, "Could you share me script for stepwise platform simulation?"

 

You do not need a script. Start your analysis with Fit Model as usual. Specify the response data column in the Y role. Specify the terms in the linear predictor for the Effects. But switch the Personality from Standard Least Squares to Stepwise..

 

Please see JMP > Books > Fitting Linear Models > Stepwise chapter.

Learn it once, use it forever!

View solution in original post

0 Kudos
Highlighted
mandeson91
Community Trekker

Re: Analysis of Opt DOE simulation results

Thank you for correction. Again, appreciate for your suggestion.
0 Kudos
Highlighted
statman
Community Trekker

Re: Analysis of Opt DOE simulation results

An alternate approach to Mark's method of model building (starting with 1st order then adding 2nd order based on excellent principles of scarcity, hierarchy and heredity) is to start with a saturated model.  Use Daniel plots (normal/half normal), Pareto plots (making sure the estimates are of practical significance) and Bayes plots (if your into that sort of thing).  Remove insignificant terms (consider Rsquare vs. Rsquare adjusted deltas), and re-run the model paying particular attention to residuals to ensure your model meets the usual assumptions.  

 

One of my favorite quotes:

"Two equally competent investigators presented with the same problem would typically begin from different starting points, proceed by different routes, and yet could reach the same answer. What is sought is not uniformity but convergence" (Box, Hunter Hunter "Statistics for Experimenters").

0 Kudos