Designing Process Optimization Experiments Using Models Based on Production Data - Mastering JMP

Video was recorded in January 2025 using JMP 18.

Do you have historical or real time production data? Have you wondered how to leverage these data to design high value, efficient, cost-effective experiments?

In this session, we will use the following steps to find an optimal process recipe for a plasma enhanced chemical vapor deposition (PECVD) tool that is used to deposit an insulative layer on a wafer in semiconductor manufacturing:

• Derive and refine a JMP Pro penalized regression model from production data combined with process and scientific knowledge.
• Use the regression model as a basis for an economical experiment design using the Design of Experiments (DOE) methodology.
• Find an optimal process recipe using a regression model of the experiment data.

 Suggested Prerequisites:

• Some experience evaluating process data.
• Basic familiarity with Design of Experiments and statistical models.

For additional detailed information about topics covered in the video, consider:

• Developer Tutorial: Using JMP Pro Generalized Regression to Better Understand Observational Data
• Developer Tutorial: Using JMP Pro Generalized Regression to Analyze Designed Experiments

Some questions answered by @Jason_Wiggins, @Lovy_Singh and @charlie_whitman at the live webinar:

Q: Why are interactions and squared terms centered?

A: JMP will automatically center the estimates unless you request/specify otherwise. For a main effects model, there is no centering. For interactions and squared terms, etc., JMP will automatically center the estimates. You can "uncheck" this option from the red triangle menu dialog when using Fit Model. Also, centering can help reduce collinearity between factors.

 See a Tech Support note on the advantages of centering.  

Q: Why not use a Definitive Screening Design instead of Custom Design?

A: This is using historical data to help find what active terms might be present first, and then designing an experiment to lock in those terms.  Also,  this method helps find the important  factors we should be using. A DSD is typically used to remove unimportant factors.

Q: Observational data will likely have outliers. How would one address outlier data to make sure that the outliers don't skew the model?

A: A good way to highlight outliers is using Cook's distance. Then, after identifying any possible outliers, you can remove them from the model using a "Hide and Exclude" row state.

Q: is the goal to have the lowest AICc possible?

A: When comparing different models, a lower AICc helps determine the better model. A rule of thumb is an AICc difference value of ~5 between one model and another basically tells you both are pretty equivalent.

Q: Why did you remove pressure from the model when the p-value less than 0.0001?

A: This is a good example of incorporating process knowledge to refine model. In this example we knew, gas flow and pressure are related.  So, perhaps gas flow is enough to help measure the process. 

Q: You mentioned we can use process understanding, for example eliminate two closely related terms that remain in the model. Is there a way to identify multicollinear factors using a mathematical tool in JMP (VIF, etc.) if you may not know about every term in the process?

A:  Yes, here is a quick video on this:  https://community.jmp.com/t5/Statistical-Thinking-for/Assessing-Multicollinearity/ta-p/271968 and also some short  demos by one of our JMP Statistical Educators: https://community.jmp.com/t5/Deeper-Dives/A-Deeper-Dive-Into-Multicollinearity/ta-p/674508.

Q: Why don't you use Full Factorial design?

A; Full Factorial is very rigid in the number of runs you have to run and is going to be looking for terms that might not be significant at all. Using custom design, we can chose the type of model we want to build and can constrain the number of runs based on our business needs and limitations.

Q: Are there any tools or methods to help identify and/or visualize the space explored by the existing data, like a scatter plot tool for more than 2 or 3 dimensions?

A: Certainly. Graph>Scatterplot 3D. And for 4D you could use colored row states to highlight a 4th dimension on a 3d plot.

Q: With evaluation of process data on the fly we are certainly a slave to the variability /control of the hardware used across the process selected per variable monitored.  Less variability or No variability could be a function of better control.  The other issue is gage based -> How well does the tool measure these variables? That will likely come out in the overall signal to noise. It is a little dangerous to

Q: Could we use the model RMSE from the observational data to help inform our estimate of the anticipated RMSE in the Design Evaluation>Power Analysis?

A: That is not appropriate in this context.  Walds qui square or t ratio are better for that. RSME in DOE is an expectation and useful when doing experiment design.

Q: Is Stepwise as an alternative method for picking significant terms.?

A: Stepwise, can be powerful. But, essentially you're still building a standard least squares model and selectively bringing terms in and out based on some criteria. The shrinkage methods and the methodology that we used are estimation methods  better suited to deal with some of the issues that we have with observational data. If you want to do due  diligence, of course you can try Stepwise also, but it carries some of the same challenges as Standard Least Squares.

Q: Because we use a tight range of factors to draw a model and identify the active term, do we have to test a similar range in the Custom Design? Should we build the model to  model out to what we're actually going to test in the DOE?

A: No. We  are extremely limited with observational data, for the reasons that I mentioned. But that doesn't mean that we need to be limited in our design experiment. Really, all we're trying to figure out in the model is what assumed model needs to look like. In the experiment, we still want to apply all the best practices for designed experiments and make sure that our levels for our factors are as wide apart as possible.

Some comments from attendees during the session:

• If you want to replicate your Predictor Screening output, you need to set/save a random seed.
• When using observational data, be super careful about missing values.  Depending on the platform/method being used, you can unintentionally drop entire runs/batches from your analysis. JMP also offers Informative Missing.