Definitive Screening Design DOE for a large number of factors

[#1]

Dear JMP experts,

I'm doing a dual-lens design DOE to couple lights into a fibre array. My goal is to look at the yield of the system when taking into account both the manufacturing tolerances and decenter and tilt of both lenses. I have 15 factors and 4 responses in the DOE. The 4 responses CE1, CE2, CE3 and CE4 correspond to the coupling efficiency of each fibre in the fibre array. 

I tried to use the Definitive Screening Design to do the initial screening due to its unique advantage compared to the traditional fractional factorial design. As can be seen from the simulation data, the difference between CE1 and CE4 should be negligible and the difference between CE2 and CE3 should be negligible. However, the DSD screening gives me different collection of statistically significant factors for all four responses.

Especially for the factor of l1_tilt_y, I knew for a fact that it's a dominant factor for the response of CE1 and CE4 and it has a nonlinear relationship for both of the responses, but the DSD screening cannot even pick that up for CE4. 

The JMP report file with data table embedded is attached.

Can you please kindly share with me your thought?

Thank you.

Best regards,

Eric

[#2]

You might consider using 'space filling designs' to collect the responses from a computer simulations. Also, all you want is a interpolator for rapid prediction, not for assessing factor importance. (You can just look at the simulation code to see the importance and relationship.) JMP provides the Gaussian Process platform for modeling computer simulation data.

[#3]

I deleted my comments about non-DSD strategies for simulation data because I saw others have made the same point ... although I often run my experiments against a simulator prior to doing it in real life, perhaps that's what you were doing? 

[#4]

Mark's points are valid.  Some additional thoughts: Before analyzing the relationships between the factors and the responses, look at the response variables.  Are the values reasonable?  Do they make sense?  Was the variation in the response variables practically significant? Look at the multivariate relationships (Analyze>Multivariate Methods>Multivariate).  Do the relationships/correlations make sense? I always recommend that experimenters predict the data prior to running the experiment to establish context for the results.  The ability to determine relationships can be effected by the amount of variation captured in the study.

[#5]

Dear Statman,

Many thanks for sharing your thought.

I've reattached the correct JMP report file with the data table embedded. Can you please kindly have a look?

As you can see, the response of CE1 and CE4 is not much different, but JMP somehow gives a different screening result. This doesn't make sense to me.

What's your thought on this?

Thanks.

[#6]

Hello @hanyu119,

Other members of the community have bring a lot of very important notions, like how to handle noise in DoE.

If you're running a simulation experiment with non-linear effects, maybe a space-filling design would suit your needs : you will be able to fit a lot of different models through the construction points and handle non-linear effects through  different analysis technique (Tree-based methods, neural networks, SVM, etc... based on the response type). I'm however a bit skeptical about the number of factors you have (15) and the number of experiments this type of design will generate with so many factors (for 15 factors, JMP recommends by default 150 runs with various Space-Filling designs). Maybe the number of runs is not so important for you, since you don't use materials/ressources (it's only a computation/simulation experiment) ?

This JMP quizz would be a good starting point to have a first recommendation about which type of design to try first : DOE Quiz (jmp.com)

Hope this will help you in your reflexion and simulation experiment,