Hi, I'm Chris Gotwalt with JMP, and I'm presenting with Fangyi Luo of Procter & Gamble, and her colleague, Beatrice Blum, who'll be joining us for the in- person presentation at the Discovery Conference in Spain.
Today, we are talking about how to model data from designed experiments when the response is detection limited. This is an important topic because on the one hand, detection limits are very common, especially in industries that do a lot of chemistry, like the pharmaceutical and consumer products industries.
While on the other hand, the consequences of ignoring detection limits leads to seriously inaccurate conclusions that will not generalize. This leads to lost R&D time and inefficient use of resources. The good news that we are here to show today is that getting the analysis right is trivially easy if you are using generalized regression in JMP Pro and know how to set up the detection limits column property.
In this talk, we're going to give a brief introduction to sensor data, explaining what it is, what it looks like in histograms and a brief description of how you analyze it a little bit differently. Then Fangyi is going to go into the analysis of some designed experiments from Procter & Gamble. Then I'm going to go through an analysis of a larger data set than the one that Fangyi introduced. Then we're going to wrap up with a summary and some conclusions.
What are detection limits? Detection limits are when the measurement system is unable to measure, at least reliably, when the actual value is above or below a particular value. If the actual value is, say, above an upper detection limit, the measured value will be observed as being at that limit. For example, if a speedometer in a vehicle only goes to 180 kilometres an hour, but you are driving 200 kilometres an hour, then the speedometer will just read 180 kilometres an hour.
In the graphs above, we see another example. We see five histograms of the same data. The true or actual values are over here on the left, and moving to the right, we see what results when you apply an increasing detection limit to this data.
What happens is we see this characteristic bunching at the detection limit. When you see this pattern, it's a really good sign that you may need to think about taking detection limits into account in your distributional or regression analysis.
Why should we care about detection limits in a data analysis? Well, if you don't take your detection limits into account properly, you'll end up with very heavily biased results, and this leads to very poor model generalization. The regression coefficients will be way off. You'll have an incorrect response surface, which leads to matched targets with the Profiler being way off.
I think the situation is a little bit less dire when maximizing a response, but there's still quite a lot of opportunity for things to go wrong. In particular, Sigma, your variance estimate will still be way off, which leads to much lower power, you have completely unreliable p- values. The tendency is that variable selection methods will heavily under select important factors . The actual impact that a factor has on your response will be dramatically understated if you don't take the detection limits into account.
The two tables of parameter estimates that we see here illustrate this very nicely. On the left are the parameter estimates from a detection- limited LogN ormal analysis of a regression problem. On the right, they are the resulting parameter estimates when we ignore the detection limit. We see that the model on the left is a lot richer and that a lot of our main effects, interactions, and quadratic terms have been admitted into the model.
Whereas on the right, when we ignore the detection limit, we're only able to get one main effect and its quadratic term included in the model, and the quadratic term is heavily overstated with a value of negative 11.5 about relative to the value in the proper analysis where that quadratic term is equal to just negative 3.
We see that we're really missing it on a lot of the other parameters here as well. When we take a look at this in the Profiler, this becomes really apparent. O n the left, we have the Profiler of the model correctly analyzing with the Limit of Detection, and we see that all the factors are there, and overall, the response surface is pretty rich- looking.
On the right, we see that only the one factor, dichloromethane, has been included in the model. T he solution to the problem that you would get with the problem on the left is likely rather different from the one that you would get on the right.
Thanks, Chris. Now I'm going to share with you a little bit background on the experiment of the data mentioned by Chris, the time to bacterial detection. The objective of that experiment was to understand hostility impacts of our formulation ingredients or factors on a liquid consumer cleaning formulation.
The experiment was a micro- hostility Design of Experiment with 36 samples, and we have five key formulation factors A, B, C, D, E. W e have two responses from this experiment. They're from microbial testing. The first one was the one mentioned by Chris. It is time to bacteria detection in two days, and it was measured by hour. I f we are not able to detect the bacteria in two days, then time to bacteria detection is right censored at 48 hours. So the Limit of Detection for this endpoint is 48 hours.
Another endpoint is log reduction in mode from Micro Susceptibility Testing. For this endpoint, what we did is that we add certain amount of mold to the formulation, wait for two weeks, and measure amount of mold in the product after two weeks. T hen we calculate the reduction in log base time mold, and this is the second endpoint.
Limit of Detection for this endpoint is six unit. T his shows you the detailed data from the experiment, the first 15 samples. Y ou can see the formulation factors A, B, C, D, E, and they were from response surface design. W e have two endpoints, the bacteria detection time in hours and the log reduction in mold. The data highlighted in red are right censored data.
We can use histograms and scatterplots to visualize our data as well as factor versus censoring relationship. As you can see from the histogram, more than 50 % of samples are right censored at 48 hours. If an observation is not censored, then most of them will be below 15 hours.
O n the right, we have the scatter plot. The red circle indicates the censored data points. You can see that we have censoring at all levels of the factors except for factor C. We don't have the censoring at higher level of C, but we observe censoring at all level of the factors.
In JMP Pro 16 and higher, we can specify column properties for detection limit. W hen you go to column property, you find detection limits, and then you can specify the lower detection limit and upper detection limit. I f a data point is below the lower detection limit, that means it's less censored at the lower detection limit. If a data point is higher than the upper detection limit, then it means that it's right censored at the upper detection limit.
For the bacterial detection time, we have an upper detection limit and it's 48 hours. W e put 48 hours in the upper detection limit box. After we specified detection limit on the column property in JMP, then we can use JMP generalized regression modeling to analyze the data by taking into account the Limit of Detection. So this is a new feature in JMP Pro 16 and higher.
For this type of analysis, we need to first specify the distribution for a response and estimation method. W e try different distribution for the data and use the forward selection method, and we found Normal distribution fits the data the best because it has lowest AICc.
We can also analyze data ignoring the detection limit. Y ou can see that we will have a much smaller model with five factors left in the final model. T he model ignoring Limit of Detection will have much less power to detect significant factors.
This showed you the factors left in the final model from the generalized regression modeling. If we take into account Limit of Detection for the response, or if we ignore Limit of Detection in the response. As you can see, if we take into account Limit of Detection, then we have much more significant factors in the model. W e can only detect the effect of C and D and their quadratic effect in the model if we ignore Limit of Detection for our response.
Again, this is comparison of the parameter estimate from the model if we consider Limit of Detection in the modeling or ignoring Limit of Detection in the modeling. Ignoring Limit of Detection in the modeling would give us the bias estimate of the parameter as well.
This slide shows you the prediction Profiler of the response if we perform the modeling by considering the Limit of Detection versus ignoring the Limit of Detection. If we consider the Limit of Detection in the modeling, then we get a model with all the terms in the model, the main effects as well as some of the interaction and quadratic terms. T his model makes much more sense to our collaborators.
Remember that at lower level of C and at higher level of D, we have more censoring data. That means the detection time is longer and the prediction Profiler showed that at lower level of C and a higher level of D, the predicted detection time is longer. A lso because we have more censored data in those region, so the confidence interval for the prediction P rofiler is wider.
If we ignore the Limit of Detection in the analysis, we get much less significant factors. Only C and D showed up in the model, and the parameter estimate is also biased. This one shows you the diagnostic plotting of observed data on the y- axis versus predicted data on the x- axis.
If we consider Limit of Detection in the generalized regression modeling, it gives correct prediction. But if we ignore Limit of Detection in the modeling, then it will give incorrect prediction for your data.
In addition to the prediction Profiler, JMP generalized regression modeling would also give you two profilers similar to those from Parametric Survival M odeling platform. Those are the Distribution Profiler and Quantile Profiler. The distribution profiler will give you the failure probability at a certain combination of our formulation factors and a certain detection time.
The Quantile Profiler will give you the quantile of the detection time at a certain combination of our formulation factors and the specified failure probability. T hese two profilers are available in JMP under the Generalized Regression Modeling.
But one advantage of using Generalized R egression Modeling to analyze time to failure type of data is that it would provide you the Prediction Profiler, and this type of profiler is much more easier for our collaborator to understand. I t's much harder to explain the Distribution Profiler and Quantile Profiler to our collaborators.
Now it comes to the analysis of the second endpoint, the log reduction in mold. Again, we can use histogram and the scatterplot to visualize our data and visualize the factor versus censoring relationship. As you can see from the left histogram, you can see that we have a lot of data that are right censored at six unit.
We can see censoring at all level of our formulation factors, except at higher level of C and lower level of E. T his is the region of concern. We have seen a lot of censoring at lower level of C and higher level of E. That means at lower level of C and higher level of E, it's good for the product. We have higher log mold reduction.
Again, we can use detection limit on the column property to specify the Limit of Detection for this endpoint. W e used upper detection limit of six in this column property. N ow the next step is to analyze this data using the Generalized R egression modeling by taking into account the Limit of Detection. W e use LogN ormal distribution and forward selection.
Interestingly, we found that the RS quare is one and this is very suspicious. A lso, we see some red flag. The AICc had a severe drop after step 17. T he standard error of the estimate as well as the estimate for the scale parameter seems to be extremely small. A lso, the diagnostic plot showed perfect prediction from the model. W e know that the model has overfit.
This is the Prediction Profiler, and they showed very narrow confidence interval for the prediction, and we knew that our model is overfit. So what we did for the modeling is that we tried a simpler model by removing the quadratic terms from the initial response surface model.
We found that LogN ormal with forward selection model fits the data the best because it has a lowest AICc and BIC. T his time, the solution path looks more reasonable as well as the standard error estimate of our parameters and estimate of the scale parameter of the LogN ormal distribution. T he diagnostic plot looks more reasonable now.
This is the Prediction Profiler of the final model after we removed the quadratic terms. This Prediction Profiler makes a lot more sense. Recall that at lower level of C and at higher level of E, we have more censored data you can see here. That means at lower level of C and higher level of E, we have higher log mold reduction.
It showed on the Prediction Profiler because it has more censored data in this region and the confidence interval for the prediction is wider. We can also compare the final model Prediction Profiler if we ignore Limit of Detection in the modeling.
If we ignore Limit of Detection in the modeling, we got less significant model factors as well as biased results. If we ignore Limit of Detection in the Generalized Regression modeling, then the second model, which is incorrect and is trying to use the quadratic term to predict in the lower level of C and higher level of E. So t rying to get the predictive value close to the Limit of Detection, and we knew that this result is biased.
Fangyi has nicely shown here that the incorrect analysis, ignoring the Limit of Detection, leads to some seriously biased results. And that getting the analysis right is easy if you set up the detection limits in either the custom designer or as a column property.
I'm going to go through one more example that has measurements at different times, which adds a little bit more complexity to the model set up, and in our case, required some table manipulation to get the data in the right format.
Here is the data table of the second DOE in basically the form that it originally came to us. In this data, we have 8 factors, A through H, and the data has measurements at 1 day, 2 days, and 7 days. Originally, our intent was to analyze the 3 days separately, but when we fit the day 7 data, the confidence intervals on the predictions were huge.
It was apparent that there was so much censoring that we were unable to fit the model, and so we were either going to have to come up with another strategy or back away from some of our modeling goals. What we ended up doing was we used a stack operation from under the tables menu so that the responses from different days would be combined together into a single column, and we added day as a column that we could use as a regression term.
In the histogram of log reduction, we see the characteristic bunching at the detection limit of five. Combining the data like this certainly seems to have improved the impact of censoring on the design and hopefully allows us to make more effective use of all the data that we have.
As in the previous examples, we start off fitting a full RSM model, but in this case, because we have day as a term, we add a day and interact all of the RSM terms with day in the Fit Model Launch Dialog prior to bringing up the generalized regression platform. Again, we're going to use the LogN ormal distribution as our initial response distribution.
Because this is a large model, we can't use best subset selection, so we used pruned forward selection as our model selection criterion. We try the LogN ormal, Gamma, and Normal distributions, and clearly the LogN ormal comes out as the best distribution because its A ICc is 205.3, which is more than 10 less than the second best distribution, which was the Normal, whose A ICc was 257.
Here, the model fit looks really reasonable with nothing suspicious. The solution path standard errors, scale parameter, and the actual by- predicted plots all look pretty good and realistic. There's a little bit of bunching down at the low end of the responses, but the thinking is that wasn't due to a detection limit, just a part of the discreetness of the measurement system at lower levels of reduction.
Now, if we repeat this analysis, ignoring the detection limit, it guides us towards the normal distribution. Here we see the Profilers for the model that incorporated the detection limit on the top and the model that ignored the detection limit on the bottom.
As in the other examples, we see that the size of the effects are dramatically muted when we ignore the detection limit and we get quite a different story as there's a strong relationship between log reduction in factor E when we take the detection limit into account properly, and that effect is seriously muted when we ignore the detection limit.
If we compare the actual by- predicted plots for the two models, the model with the Limit of Detection taken to account properly is tighter around the 45- degree line for the uncensored observations. W e see that the model ignoring the detection limit is just generally less accurate as the observations are more spread out across the 45- degree line.
Those are our two case studies. In summary, I want to reiterate that detection limits are very common in comical and biological studies. As we've seen in our case studies, ignoring detection limits introduces severe model biases. T he most important message is that using the column property or setting up the detection limits in the custom designer make analyzing detection- limited data much easier to get correct.
There are some pitfalls to watch out for in that if you see standard errors that are unrealistically small, or models are unrealistically accurate, you may need to back off from the quadratic terms or possibly even interaction terms.
We've shown how histograms can be used to identify when we have a detection limit situation. It's useful to see the censoring relationship between different factors, because if there are big corners of the factor space where all the observations are missing, then we may not be able to fit interactions in that region of the design space.
A gain, if the model looks too good to be true, go ahead and try a simpler model, back off a bit. That's all we have for you today. I want to thank you for your attention.
