Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

[#1]

I would like to evaluate the influence of 8 factors on the response. There are 1 continuous factor, 7 other categorical factors, and 1 categorical response, and 2 of the categorical factors have 4 levels, one of the categorical factors has 8 levels. Can we use the classic screening design method to design experiments?

We want to know the most influential factors among the 8 factors, where we could analyze the screening experimental results? When we add more than 6 factors or the levels of factors exceed 3, the "Screening" tab disappeared where we cannot analyze the screening design results. What are the problems? Does that mean the classical screening design cannot be used in this case? Or we should analyze the screening design results separately? 

Thanks.

[#2]

I do not think we disagree. I just wanted to be sure of the nature of the Location factor. The description "best" and "worst" made it sound like a fixed effect. I see that it is two random locations that will each provide half of the runs. Then it should be a random block. But I totally agree with view of Location.

[#3]

Mark, I often "treat" noise as a fixed effect during the experiment and during analysis. To do this requires the experimenter understand what the noise is (it must be identified) and can manage it (or sample it) during the experiment. This allows for the estimation of noise-by-factor interactions (are the design factor effects consistent over changing noise) which is a necessary component of creating robust designs.
Reference:
Sanders, D., Leitnaker M., and McLean R. (2002) “Randomized Complete Block Designs in Industrial Studies” Quality Engineering, Vol. 14, Issue 1

[#4]

@Mark_Bailey Based on what you mentioned, that's always the case that I will need a lot of runs to estimate the logistic regression or the binomial GLM  model well. Is there any standard on the number of runs that normally could provide good estimation results? Or it only depends on whether we get good results? I know there are a lot of ways to evaluate the goodness of a model if the response is continuous, such as Pareto plot, etc. What are the methods we could use for evaluating whether the model is good or not when a response is binary?

[#5]

There is no standard that I am aware of. You can use the Simulate feature in JMP Pro to evaluate any performance of the analysis for a given design.

Yes, it depends on if you get good results. But there is a rigorous method behind this approach. For example, you could create a design and select Simulate Response from the Custom Design platform menu. Notice that you can simulate a binomial response. You can enter the minimum coefficients or expected coefficients and generate the response Perform the analysis for the first simulated sample. Then right click on the numeric column of interest (e.g., parameter estimate, p-value, or another) and select Simulate. You can then adjust the design as indicated by the simulation results.

Please see the documentation for Simulate.

[#6]

@P_Bartell I think it should be the main effect only previously, but I may mess something up. I recheck the model and make sure the model to be the main effect only, and it seems the error goes away now. 

Since I am a starter of DOE and JMP, I chose the Elastic Net fitting personality just because I follow the example that was provided in the JMP Documentation. I will try to use the basic Logistic Regression estimation method as a starting point then. Thank you for your advice. When we chose the estimation methods, how do we decide what estimation methods we should choose, when to choose a basic logistic regression estimation method, when we should choose others, like Lasso, Elastic Net, Ridge, etc. 

And from your first reply, you said sometimes we need lots of runs to tease a signal from the noise for a categorical response... Could you please explain a little bit more about that? What could we do to check whether we need more runs? 

Thank you for your time again. 

[#7]

@yiyichu It is beyond the scope of a general discussion forum such as this to give even incomplete guidance on which modeling technique to use when. Practical problems, data types, data gathering methods, the influence of noise factors, software capability, capabilities of people involved in the project, and probably some things I fail to mention, all play a role in helping one determine HOW to solve a problem. So in that regard, I really think the best avenue for you to pursue is to consider a more holistic approach to building your problem solving skills through education and practice. One great way to accomplish this, and it's free (as in no charge) is to complete the SAS "Statistical Thinking for Industrial Problem Solving" curriculum. Much of the course is devoted to marrying data collection, data types, and analytic strategies to solve problems. Here is a link to the main web page for course information: Statistical Thinking for Industrial Problem Solving 

I'll try and give you an example of what I mean by with a small number of runs, it may be difficult to tease a signal from the noise. It's a bit of a mental exercise...so here goes. Let's start out by saying you'd like to determine if a coin flip is fair and balanced. That is there is a 0.5 chance of getting a head or tails with the flip of a coin. Now suppose I come to the 'experiment' with a coin that is NOT 50/50, but your null hypothesis IS that it's a 50/50 coin. But in reality I'm cheating and have brought a coin to the experiment that has a 0.7 chance of a heads, and a 0.3 chance of a tails. If I flip the coin once...I get a head. Are you willing to reject your hypothesis that the coin is 50/50? Probably not on the basis of one toss. How about a second toss? Now you get a head again. Willing to reject yet? After all, if your hypothesis is true, there is a 0.25 chance of two consecutive heads. I think most people would still not reject the null hypothesis. So you flip a third time...this time you get a tail. How many flips and outcomes will it take for you to finally reject the null? Therein lies the issue that in my practical experience happens sometimes with binary outcomes.

I hope this helps?

[#8]

@P_Bartell It really helps. I already took some of the DOE training, I will take the others too.

I think I understand what you are trying to say in the example, but when it goes to screening design, what we really care about is which factor would have a significant influence on the response, we don't really know how the number of runs would impact the modelling results, right? Let's say, we design an experiment using screening design, and choose the main effect model, after we do the experiments, we collect actual response data and choose a model to do the main effect analysis. Then we will get the results telling us which factor is significant according to their P-values. How do we know whether the results are correct or not? Or maybe after a certain number of runs, we found that none of the factors is significant by analyzing the results, so we need more experimental runs? Maybe I am wrong, or my question is kind of silly, but it is just hard for me to relate this to screening experimental design...

[#9]

My example is in fact a screening design. Only one factor...but the principles apply for one to many factors and their appropriate levels. Yes to some degree you'll use p values...but don't fall into the p-value is a cliff trap. Suppose you adopt, and it's entirely up to you the specific critical p value needed to reject the null hypothesis for any effect estimate...so which value you gonna pick? 0.05? 0.01?  0.10? and on and on. And what happens if you pick 0.05 as your critical p value for a parameter estimate, and you have one effect's p value = 0.049 and another that is 0.051. By the cliff mentality you'd be forced to say the factor reflecting the 0.049 estimate is 'significant'...and Lord I hate that word...but that's another rant for another time, and the other factor is 'not significant' because it's p value is higher than the critical value.

The number of runs gives power to be able to detect effects. My example bears that out. Two runs...you'd be hard pressed to reject the null hypothesis...but flip that coin, say 1,000 times and I'll bet my house you'd reject the null hypothesis. Screening designs are intentionally very sparse wrt to number of runs...so there may be an effect present...you just haven't gathered enough evidence to convince the jury.

[#10]

@P_Bartell So it seems there is no easy way or single answer for this problem. I will keep this in mind when I use the screening design method to design experiments and analyze the results. Thank you for your time and patience.

[#11]

I would suggest using Daniel plots (normal plots), Pareto plots of effects and possibly Bayes plots vs. p-values for assessing interesting factors from a screening design. I am aways concerned the estimates of MSE in your screening design are poor, not representative and biased. The plots I suggest are less likely biased. You are basically comparing all effects to each other to determine which are assignable and of practical significance.