topic Definitive Screening Design DOE for a large number of factors in Discussions

Definitive Screening Design DOE for a large number of factors

hanyu119 — Thu, 08 Jun 2023 21:10:15 GMT

Dear JMP experts,

I'm doing a dual-lens design DOE to couple lights into a fibre array. My goal is to look at the yield of the system when taking into account both the manufacturing tolerances and decenter and tilt of both lenses. I have 15 factors and 4 responses in the DOE. The 4 responses CE1, CE2, CE3 and CE4 correspond to the coupling efficiency of each fibre in the fibre array.

I tried to use the Definitive Screening Design to do the initial screening due to its unique advantage compared to the traditional fractional factorial design. As can be seen from the simulation data, the difference between CE1 and CE4 should be negligible and the difference between CE2 and CE3 should be negligible. However, the DSD screening gives me different collection of statistically significant factors for all four responses.

Especially for the factor of l1_tilt_y, I knew for a fact that it's a dominant factor for the response of CE1 and CE4 and it has a nonlinear relationship for both of the responses, but the DSD screening cannot even pick that up for CE4.

The JMP report file with data table embedded is attached.

Can you please kindly share with me your thought?

Thank you.

Best regards,

Eric

Re: Definitive Screening Design DOE for a large number of factors

Mark_Bailey — Tue, 28 Jun 2022 13:41:42 GMT

The data table that you provided contains the design columns but no response data. The evaluation of the design is as expected. The evaluation of the model is not possible, though.

I am not familiar with this lens design, but I often encountered pre-conceived ideas about what was or was not known from previous experience. Many such situation were based on anecdotal stories that were not actually testable or scrutinized with a statistical approach. So I am not concerned yet that the outcome surprised you. Still, let's see what we can do to sort it out.

Re: Definitive Screening Design DOE for a large number of factors

statman — Tue, 28 Jun 2022 14:28:43 GMT

Mark's points are valid. Some additional thoughts: Before analyzing the relationships between the factors and the responses, look at the response variables. Are the values reasonable? Do they make sense? Was the variation in the response variables practically significant? Look at the multivariate relationships (Analyze>Multivariate Methods>Multivariate). Do the relationships/correlations make sense? I always recommend that experimenters predict the data prior to running the experiment to establish context for the results. The ability to determine relationships can be effected by the amount of variation captured in the study.

Re: Definitive Screening Design DOE for a large number of factors

hanyu119 — Tue, 28 Jun 2022 14:59:21 GMT

Dear Mark,

Many thanks for the prompt reply.

Apologize for attaching the wrong JMP file.

I've reattached a JMP report file with the data table embedded.

Hope that's alright now.

Look forward to hearing your thought.

Cheers.

Re: Definitive Screening Design DOE for a large number of factors

hanyu119 — Tue, 28 Jun 2022 15:05:48 GMT

Dear Statman,

Many thanks for sharing your thought.

I've reattached the correct JMP report file with the data table embedded. Can you please kindly have a look?

As you can see, the response of CE1 and CE4 is not much different, but JMP somehow gives a different screening result. This doesn't make sense to me.

What's your thought on this?

Thanks.

Re: Definitive Screening Design DOE for a large number of factors

Mark_Bailey — Tue, 28 Jun 2022 16:10:18 GMT

You have some unusual observations for CE1 and CE4. The residuals are unusual. For example, here is the start of the plots for CE1:

These observations can have unusually high influence on the estimates of the model parameters and, therefore, the hypothesis tests that the estimates is not zero.

Re: Definitive Screening Design DOE for a large number of factors

statman — Tue, 28 Jun 2022 17:11:09 GMT

Much as Mark found, there are some unusual data points. These may be due to noise rather than treatment effects.

Re: Definitive Screening Design DOE for a large number of factors

hanyu119 — Wed, 29 Jun 2022 06:27:55 GMT

The Multivariate is a great method to detect outliers. Thanks for sharing!

I've validated my data with the simulation and all the data is real.

Row 15 & Row 16: since l1_tilt y is the most dominant factor in the system, when l1_tilt_y is 0, which is the case for Row 15 & Row 16, the coupling efficiency of CE1 and CE4 becomes very high compared to the other runs. That's why JMP thinks they're outliers.

Row 37 is the centre point. Again, since l1_tilt_y is 0, CE1 and CE4 are high.

Row 14 is a special case. I've triple-checked my simulation and it sort of makes sense, so I can't remove this point either.

Based on this, does it mean DSD is not the right approach to use for this specific case?

What's the sensible next step that you recommend?

Thanks.

Re: Definitive Screening Design DOE for a large number of factors

hanyu119 — Wed, 29 Jun 2022 06:33:12 GMT

That's a great finding Mark! I never pay too much attention to the Main Effects Residual Plots, now I know how to use them.

I've validated my simulation data and all those outliers seem to be real, so I can't really remove them from my DOE.

What's the sensible next step that you recommend?

Is there a more reliable screening approach that you can share with me?

Thanks.

Re: Definitive Screening Design DOE for a large number of factors

statman — Wed, 29 Jun 2022 15:00:40 GMT

I'm confused by your statement: "I've validated my data with the simulation and all the data is real.". How can you possibly validate the real data with simulation? Is this experiment run completely by simulation?

There are two distinctly different unusual components of your situation:

1. There may be unusual data points for each treatment as identified by the multivariate analysis (these are likely impacted by noise). Your DSD does not have any strategy to handle noise (blocking, repeats, split-plots, etc.). I suggest you spend some time understanding noise.

2. The model is not adequate (this is identified by the resulting issues with residuals as Mark points out). There are times when the model doesn't do a very good job predicting the results. Why? This is where your investigation should head.

Re: Definitive Screening Design DOE for a large number of factors

hanyu119 — Thu, 30 Jun 2022 04:01:28 GMT

This experiment runs completely by simulation. It's a conceptual design. I don't have an actual product to verify the simulation. Having said that, the simulation should be pretty accurate since it's a very straightforward design.

Clearly, the DSD model is not adequate. What next?

Let's say you have 15 factors to start with for a screening DOE, you know the response is nonlinear with some of the factors. How will you approach this problem?

Thanks.

Re: Definitive Screening Design DOE for a large number of factors

statman — Thu, 30 Jun 2022 13:42:15 GMT

Unless running the simulation takes a significant amount of time, you might as well run full factorials with as many levels as you want. Of course, the problem is noise...not sure how you simulate this? I actually don't understand why you would run simulation experiments. The algorithm is already known in the software. The order of the model is already known. If variables (or their non-linear components) are not in the algorithm, they can't possibly affect the response of the algorithm.

In real world experiments, we build models in hierarchical order using the fundamental principles or sparsity, hierarchy and heredity. Typically we would start with determining linear main effects and add hierarchy as the experiment space approaches what we deem as optimal. This work is always sequential and iterative. Don't spend a lot of time and money learning about unimportant factors at less than optimum levels. Center point designs are efficient at testing the curvature inside the experiment space.

Re: Definitive Screening Design DOE for a large number of factors

Victor_G — Thu, 30 Jun 2022 14:43:41 GMT

Hello @hanyu119,

Other members of the community have bring a lot of very important notions, like how to handle noise in DoE.

If you're running a simulation experiment with non-linear effects, maybe a space-filling design would suit your needs : you will be able to fit a lot of different models through the construction points and handle non-linear effects through different analysis technique (Tree-based methods, neural networks, SVM, etc... based on the response type). I'm however a bit skeptical about the number of factors you have (15) and the number of experiments this type of design will generate with so many factors (for 15 factors, JMP recommends by default 150 runs with various Space-Filling designs). Maybe the number of runs is not so important for you, since you don't use materials/ressources (it's only a computation/simulation experiment) ?

This JMP quizz would be a good starting point to have a first recommendation about which type of design to try first : DOE Quiz (jmp.com)

Hope this will help you in your reflexion and simulation experiment,

Re: Definitive Screening Design DOE for a large number of factors

Mark_Bailey — Thu, 30 Jun 2022 16:13:49 GMT

You might consider using 'space filling designs' to collect the responses from a computer simulations. Also, all you want is a interpolator for rapid prediction, not for assessing factor importance. (You can just look at the simulation code to see the importance and relationship.) JMP provides the Gaussian Process platform for modeling computer simulation data.

Re: Definitive Screening Design DOE for a large number of factors

David_Burnham — Thu, 30 Jun 2022 21:04:57 GMT

I deleted my comments about non-DSD strategies for simulation data because I saw others have made the same point ... although I often run my experiments against a simulator prior to doing it in real life, perhaps that's what you were doing?