topic Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response? in Discussions

Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Thu, 08 Jun 2023 21:02:22 GMT

I would like to evaluate the influence of 8 factors on the response. There are 1 continuous factor, 7 other categorical factors, and 1 categorical response, and 2 of the categorical factors have 4 levels, one of the categorical factors has 8 levels. Can we use the classic screening design method to design experiments?

We want to know the most influential factors among the 8 factors, where we could analyze the screening experimental results? When we add more than 6 factors or the levels of factors exceed 3, the "Screening" tab disappeared where we cannot analyze the screening design results. What are the problems? Does that mean the classical screening design cannot be used in this case? Or we should analyze the screening design results separately?

Thanks.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

P_Bartell — Mon, 11 Jan 2021 16:46:55 GMT

Once you find yourself in a situation where the levels of factors is anything other than 2, the classic screening designs, that is, two level fractional factorial designs are not applicable or even doable. In your case with some factors having levels other than 2 you are forced into using an optimal DOE scenario...which in JMP takes you straight to the Custom Design platform. Since you seem to want to run a screening design I would use the Custom Design platform to build a design that is supported by a main effects only model. With an eye towards minimizing the number of runs. You may want to add a few runs over and above the JMP specified minimum to allow for some degrees of freedom for error.

Good luck trying to find active effects with a categorical response...my past experience is sometimes you need lots of runs to tease a signal from the noise...but you'll never know unless you try.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Tue, 12 Jan 2021 17:38:05 GMT

Thank you very much. And I have another follow-up question.
I used the Custom Design platform. Since I have a binary response, when I fit the model, I chose Personality: Generalized Regression, Distribution: Binomial, Estimation Method: Elastic Net -- Adaptive, Validation Method: AICc. But when I click on the "Go" button, it shows an alert which is "Fitting terminated early because of failure to maintain heredity. Consider running again without enforcing heredity". Is this a severe problem I have to solve, how could I solve the problem, or should I choose some other methods?

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

Mark_Bailey — Tue, 12 Jan 2021 17:57:15 GMT

This refers to the fact that the model hierarchy is important. JMP strongly encourages you to maintain the hierarchy, but does not require it.

Model hierarchy means that if you have X1*X2 in the model, then X1 and X2 should also be in the model, even if they are not significant alone.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

P_Bartell — Tue, 12 Jan 2021 22:49:02 GMT

@yiyichu Great that you used the Custom Design platform. Can you share the specific model you articulated for the design construction? Was it main effects only? Or something different?

In my first reply, I suggested you specify a main effects only model...the error message you are seeing and based on @Mark_Bailey 's correct reply, it sure looks like you are trying to fit a model with more than just main effects? Does your design support at least some level of estimation for at least some two (or heaven forbid) even higher order factor interactions? With categorical factors at more than 2 levels, to estimate interaction effects containing those factors your design can get big in a hurry. Especially with the one factor with 8 levels.

And I'm also just a bit curious regarding why you chose the Elastic Net fitting personality? I guess, in a screening modeling mode, I might have started out with perhaps just a simple nominal logistic regression model. Elastic Net isn't an incorrect choice...but it might tend to zero out some of the parameter estimates since that's it's very nature and you might miss effect level information content. Just curious more than anything?

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

statman — Tue, 12 Jan 2021 23:56:15 GMT

I am also curious, but I'm curious why you are looking at 4 level and 8 level categorical factors at all. This sounds more like a pick the winner test than a screening design? My thoughts: Screening is not meant to test all possible combinations. Can you pick the extremes of the 4 or 8 levels? The idea would be to compare all of the factors (equally, without bias) and determine which of those are worthy of further investigation. By testing at more than 2 levels for some of your factors, you have more information about those factors and have therefore biased your study.

But, of course, I don't know your specific situation, so you may want to ignore me.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Wed, 13 Jan 2021 19:17:21 GMT

@P_Bartell I think it should be the main effect only previously, but I may mess something up. I recheck the model and make sure the model to be the main effect only, and it seems the error goes away now.

Since I am a starter of DOE and JMP, I chose the Elastic Net fitting personality just because I follow the example that was provided in the JMP Documentation. I will try to use the basic Logistic Regression estimation method as a starting point then. Thank you for your advice. When we chose the estimation methods, how do we decide what estimation methods we should choose, when to choose a basic logistic regression estimation method, when we should choose others, like Lasso, Elastic Net, Ridge, etc.

And from your first reply, you said sometimes we need lots of runs to tease a signal from the noise for a categorical response... Could you please explain a little bit more about that? What could we do to check whether we need more runs?

Thank you for your time again.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Wed, 13 Jan 2021 19:23:06 GMT

@statman The 4-level factor here is "motion level", including Major motion, Minor motion, Fine motion, and none motion. The 8-level factor here is to randomly choose 8 different locations in a room for our case. But you are right. Actually, I am also considering picking the extremes of the 4 level and 8 level factors to make all the factors at 2 levels. Then we may also choose the definitive screening design method. I am just not sure which way could be better. Now your advice makes me more confident to do it this way. Thank you.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

P_Bartell — Wed, 13 Jan 2021 19:41:18 GMT

@yiyichu It is beyond the scope of a general discussion forum such as this to give even incomplete guidance on which modeling technique to use when. Practical problems, data types, data gathering methods, the influence of noise factors, software capability, capabilities of people involved in the project, and probably some things I fail to mention, all play a role in helping one determine HOW to solve a problem. So in that regard, I really think the best avenue for you to pursue is to consider a more holistic approach to building your problem solving skills through education and practice. One great way to accomplish this, and it's free (as in no charge) is to complete the SAS "Statistical Thinking for Industrial Problem Solving" curriculum. Much of the course is devoted to marrying data collection, data types, and analytic strategies to solve problems. Here is a link to the main web page for course information: Statistical Thinking for Industrial Problem Solving

I'll try and give you an example of what I mean by with a small number of runs, it may be difficult to tease a signal from the noise. It's a bit of a mental exercise...so here goes. Let's start out by saying you'd like to determine if a coin flip is fair and balanced. That is there is a 0.5 chance of getting a head or tails with the flip of a coin. Now suppose I come to the 'experiment' with a coin that is NOT 50/50, but your null hypothesis IS that it's a 50/50 coin. But in reality I'm cheating and have brought a coin to the experiment that has a 0.7 chance of a heads, and a 0.3 chance of a tails. If I flip the coin once...I get a head. Are you willing to reject your hypothesis that the coin is 50/50? Probably not on the basis of one toss. How about a second toss? Now you get a head again. Willing to reject yet? After all, if your hypothesis is true, there is a 0.25 chance of two consecutive heads. I think most people would still not reject the null hypothesis. So you flip a third time...this time you get a tail. How many flips and outcomes will it take for you to finally reject the null? Therein lies the issue that in my practical experience happens sometimes with binary outcomes.

I hope this helps?

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

statman — Wed, 13 Jan 2021 19:45:32 GMT

For Motion, pick extremes. If that is significant, then you can "fine tune". For location in room...Can you actually control this or is it noise? If noise, consider blocking.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Wed, 13 Jan 2021 20:08:15 GMT

@P_Bartell It really helps. I already took some of the DOE training, I will take the others too.

I think I understand what you are trying to say in the example, but when it goes to screening design, what we really care about is which factor would have a significant influence on the response, we don't really know how the number of runs would impact the modelling results, right? Let's say, we design an experiment using screening design, and choose the main effect model, after we do the experiments, we collect actual response data and choose a model to do the main effect analysis. Then we will get the results telling us which factor is significant according to their P-values. How do we know whether the results are correct or not? Or maybe after a certain number of runs, we found that none of the factors is significant by analyzing the results, so we need more experimental runs? Maybe I am wrong, or my question is kind of silly, but it is just hard for me to relate this to screening experimental design...

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Wed, 13 Jan 2021 20:16:32 GMT

@statman Generally, our goal is to decide whether the location of occupants in a room would influence the performance of occupancy sensors to detect occupants. So we would mark a 3x3 ft grid on the floor first, and randomly choose 8 locations using some random number generation tools, and occupants will stand in these 8 different locations for testing. Is this kind of like we could control the location? Actually, since we know that the sensor may not detect occupants when one person is standing at the corner of the room, does it mean we could also choose the peak extremes for the location? For example, one peak extreme is the person standing in the center of the room so that the sensor has the largest probability to detect occupants, the other would be the person standing in the corner of the room so that the sensor has the smallest probability to detect occupants?

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

P_Bartell — Wed, 13 Jan 2021 21:27:47 GMT

My example is in fact a screening design. Only one factor...but the principles apply for one to many factors and their appropriate levels. Yes to some degree you'll use p values...but don't fall into the p-value is a cliff trap. Suppose you adopt, and it's entirely up to you the specific critical p value needed to reject the null hypothesis for any effect estimate...so which value you gonna pick? 0.05? 0.01? 0.10? and on and on. And what happens if you pick 0.05 as your critical p value for a parameter estimate, and you have one effect's p value = 0.049 and another that is 0.051. By the cliff mentality you'd be forced to say the factor reflecting the 0.049 estimate is 'significant'...and Lord I hate that word...but that's another rant for another time, and the other factor is 'not significant' because it's p value is higher than the critical value.

The number of runs gives power to be able to detect effects. My example bears that out. Two runs...you'd be hard pressed to reject the null hypothesis...but flip that coin, say 1,000 times and I'll bet my house you'd reject the null hypothesis. Screening designs are intentionally very sparse wrt to number of runs...so there may be an effect present...you just haven't gathered enough evidence to convince the jury.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Wed, 13 Jan 2021 21:40:50 GMT

@P_Bartell So it seems there is no easy way or single answer for this problem. I will keep this in mind when I use the screening design method to design experiments and analyze the results. Thank you for your time and patience.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

statman — Thu, 14 Jan 2021 15:10:05 GMT

In this case, location in the room is noise. You have no control over where someone will be in the room yet you still want your sensor to detect they are there. The location in the room CAN be controlled during the experiment, but not in real life. This type of factor can be handled a number of different ways:
1. Repeats: Keep the treatment combination constant and take repeated measures at different locations. The locations can be random, but you will get more information if there are specific hypotheses about WHY there would be differences in the locations (e.g., proximity to sensor, angle from sensor, corner of room) and therefore sample systematically.
2. Randomized replicates: I believe this is how you are handling this factor. While this increases inference space and provides a theoretically unbiased estimate of the MSE, you don't know the effect of the location and you may compromise design factor effect detection precision.
3. RCBD: In this case, you would select best location (close right in front of the sensor) as 1 level and worst location (far and at an extreme angle) as a 2nd level. Replicate the treatments over the 2 blocks. In this case you could treat the location as a fixed effect. This allows for increased inference space, as well as the ability to estimate the Block (location) and all block-by-factor interaction effects (a measure of robustness of your sensor) with increased precision.
4. Split-plot (cross product array): Either put the treatments in the whole plot and noise in the subplot or noise in the WP and treatments in the SP. This would improve the efficiency of the design and likely increase precision of detecting design factor and noise by factor interaction effects.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

statman — Thu, 14 Jan 2021 15:15:04 GMT

I would suggest using Daniel plots (normal plots), Pareto plots of effects and possibly Bayes plots vs. p-values for assessing interesting factors from a screening design. I am aways concerned the estimates of MSE in your screening design are poor, not representative and biased. The plots I suggest are less likely biased. You are basically comparing all effects to each other to determine which are assignable and of practical significance.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Thu, 28 Jan 2021 19:54:21 GMT

@statman Thank you for your advice.

If I chose to use Method 3: RCBD as you mentioned to replicate the treatments over 2 blocks (best and worst locations), it is like the following:

The default number of runs is 16, and then the "Color Map on Correlations" is as follows:

From the above color map, it seems the model is not the main effect model. In general terms, the color map for a good design shows a lot of white off the diagonal, indicating orthogonality or small correlations between distinct terms. Does this mean that this is not a good design? And if I increase the number of runs, it will show most of white off the diagonal, whether it means I need to increase the number of runs when I do experimental design?

And one more question here:

Our response is a binary value (0/1), 0 indicates non-presence of occupants, 1 indicates presence of occupants, for the response, I used "Maximum" as the Goal, as follows:

Is this proper, or should I use "None" as the Goal?

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

yiyichu — Thu, 28 Jan 2021 19:59:26 GMT

@statman Thank you for the suggestions.

I tried to find these plots, including Daniel plots (normal plots), Pareto plots of effects and possibly Bayes plots, but it seems they are under the "Effect Screening", and only available if you use "Standard Least Squares" as the "Personality", which means the response should be continuous. There is no such choice if I chose the "Generalized Linear Model" with the "Binomial" response. Am I wrong, or do I need to find this somewhere else?

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

statman — Thu, 28 Jan 2021 23:26:00 GMT

While both @P_Bartell and I are trying to give you some advice, you should realize there is no "right" design. Every design will potentially vary in in the resources required, the precision of detecting design factor effects, the resolution, the model that can be estimated, the number of factors that can be estimated, etc. What I recommend is to design multiple experiments and evaluate them against the set of criteria and your situation. I also suggest you predict all possible outcomes and what you will do in each outcome. You have 8 design factors and at least 1 noise variable. The purpose of this 1st experiment is to design a better experiment (move the design space and identify the factors that are most interesting). Usually a lower resolution will work to identify main effects, but you should rank order the model effects up to 2nd order to determine whether you need to bump the resolution. Also when you have JMP block the design, the software treats the block as a random effect by default. However, when you know what noise is making up the block, you can treat it as a fixed effect and quantify block-by-factor interactions. I'm a bit old school you might say. I want to know the aliasing and don't like to do partial aliasing as this is more difficult to interpret and determine what the subsequent experiment should be. But I am probably in the minority here. Many folks like the optimal designs for their efficiency. I also don't fill in any Y's as I don't want the software to "control" my analysis (in some cases, limit my analysis options).
Regarding your response variable, have you thought about other ways to quantify the response? The binary response requires large sample sizes to detect differences. Could you vary the size of the "object" or "person" and record what size object gets detected? Or could you vary the amount of "motion" it takes to get detected? If distance from sensor matters, instead of blocking on location in the room, could you determine what distance from the sensor the object is detected? In any case, I would recommend more thought on response variables.

Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?

statman — Thu, 28 Jan 2021 23:27:00 GMT

Yes, you are correct...sorry for missing that the response was binary. I really suggest you create a better response variable (see previous note).