topic ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value in Discussions
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194041#M41486
<P>I fit model in JMP. Everything but my graph looks very strange. </P><P>THe model is significant at 0.0476</P><P>RSquare 0.951662<BR />RSquare Adj 0.794562<BR />Root Mean Square Error 0.928013<BR />Mean of Response 88.68333<BR />Observations (or Sum Wgts) 18</P><P>The pink area seems to cover everything as an error. Has anyone else Encounter this?<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Capture_WEIRD MODEL GRAPH.JPG" style="width: 681px;"><img src="https://community.jmp.com/t5/image/serverpage/image-id/16956iA875CF717EE7E0FB/image-size/large?v=v2&px=999" role="button" title="Capture_WEIRD MODEL GRAPH.JPG" alt="Capture_WEIRD MODEL GRAPH.JPG" /></span></P><P> </P><P> </P>Mon, 22 Apr 2019 19:05:18 GMTnguyendv06102019-04-22T19:05:18ZERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194041#M41486
<P>I fit model in JMP. Everything but my graph looks very strange. </P><P>THe model is significant at 0.0476</P><P>RSquare 0.951662<BR />RSquare Adj 0.794562<BR />Root Mean Square Error 0.928013<BR />Mean of Response 88.68333<BR />Observations (or Sum Wgts) 18</P><P>The pink area seems to cover everything as an error. Has anyone else Encounter this?<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Capture_WEIRD MODEL GRAPH.JPG" style="width: 681px;"><img src="https://community.jmp.com/t5/image/serverpage/image-id/16956iA875CF717EE7E0FB/image-size/large?v=v2&px=999" role="button" title="Capture_WEIRD MODEL GRAPH.JPG" alt="Capture_WEIRD MODEL GRAPH.JPG" /></span></P><P> </P><P> </P>Mon, 22 Apr 2019 19:05:18 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194041#M41486nguyendv06102019-04-22T19:05:18ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194051#M41487
<P>I think a few more details might be needed. Typically, these strange types of results occur because of very high multi-collinearity in the model terms.</P>
<P>What do the leverage plots look like?</P>
<P>If you right-click on the Parameter Estimates table and add the VIF column, what do those numbers look like?</P>Mon, 22 Apr 2019 19:13:25 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194051#M41487Dan_Obermiller2019-04-22T19:13:25ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194060#M41490
<P>Hi Dan, Thank you for your quick response.</P><P>The Leverage Plots all looks pretty good. Example Pics attached here. </P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Capture_WEIRD Leverage graph.JPG" style="width: 999px;"><img src="https://community.jmp.com/t5/image/serverpage/image-id/16961i87B4B11A595CECA5/image-size/large?v=v2&px=999" role="button" title="Capture_WEIRD Leverage graph.JPG" alt="Capture_WEIRD Leverage graph.JPG" /></span></P><P>The VIF Is pretty high (>5):</P><P>VIF<BR />12.19448<BR />11.013741<BR />16.435892<BR />28.099305<BR />1.1546501<BR />1.2200138<BR />7.8651502<BR />14.560553<BR />5.9118511<BR />10.052586<BR />8.2589787<BR />5.4652854<BR />5.4652854<BR />1.0590402</P><P>However, even when I step thru the model to eliminate some of these factors, a refined version of the model with VIF less than 3 still has the same shaded plot. (attached here). This is also a mixture study so X1, 2, 3, and X4 are inherently correlated (add up to a sum of 100%). </P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Capture_WEIRD refineGRAPH.JPG" style="width: 706px;"><img src="https://community.jmp.com/t5/image/serverpage/image-id/16962iC9D2EC36CE71A6D3/image-size/large?v=v2&px=999" role="button" title="Capture_WEIRD refineGRAPH.JPG" alt="Capture_WEIRD refineGRAPH.JPG" /></span></P>Mon, 22 Apr 2019 19:37:46 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194060#M41490nguyendv06102019-04-22T19:37:46ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194062#M41492
<P>Ah, the mixture situation explains alot of this. You are fitting a Scheffe model. They will typically have more error associated with the prediction because of the built-in collinearity of the mixture factors. </P>
<P> </P>
<P>Expand your y-axis scale on the observed versus predicted plot. I think you will see the error bounds appear. There will likely be high error because you really do not have many significant terms in this model. Perhaps only one term, X6.</P>
<P> </P>
<P>The significance of your X1-X4 terms is misleading. The p-values are associated with the test comparing the parameter estimate to 0. But you are fitting a Scheffe model. The parameter estimate for a Scheffe mixture term is the combination of the mean of the data AND the effect of the component. So a more meaningful test would be to compare the parameter to the overall mean (88.68 for your scenario). Considering that, you will see that none of the X1 through X4 terms look to be "significant". You can verify by adding the 95% confidence interval on for each of the parameter estimates. If the confidence intervals for X1 - X4 contain the overall mean, that component is not having much of an effect. </P>
<P> </P>
<P>Stated another way, it looks like you are getting about the same response everywhere within the mixture space. X5 does not seem to have an impact, but X6 does.</P>Mon, 22 Apr 2019 20:03:14 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194062#M41492Dan_Obermiller2019-04-22T20:03:14ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194194#M41513
<P>Thank you Dan. That makes sense. </P><P>Given this interpretation, my model is still significant at alpha <0.05 correct? Even if it is just base on the value of X6?</P><P> </P><P>Thanks again</P>Tue, 23 Apr 2019 18:33:39 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194194#M41513nguyendv06102019-04-23T18:33:39ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194309#M41543
<P>You are correct that the overall model is likely significant, based on the X6 term.</P>
<P> </P>
<P>Realize that because of the issues with the main effects of a Scheffe model, the overall model test may show as significant even if the model is not.</P>Wed, 24 Apr 2019 15:48:58 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/194309#M41543Dan_Obermiller2019-04-24T15:48:58ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/248540#M48769
<P>Hi Dan,</P><P>I have a follow up to the <SPAN>Scheffe model fit issue in the fit-model platform. Is there a way to fit this model appropriately where the parameter estimate is comparing the estimate to null=mean instead of comparing it to null=0?<BR />In other word, is there a better way to fit this model to get a more correct output and transfer equation?</SPAN></P><P> </P><P><SPAN>Thank you,</SPAN></P><P> </P>Thu, 20 Feb 2020 18:47:05 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/248540#M48769nguyendv06102020-02-20T18:47:05ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/248553#M48773
<P>A colleague of mine wrote a script to provide the proper testing. However, I always caution anyone working with formulations: what will you do with this information? Suppose a component is not significant. What does that really mean? Does it mean you could remove that component from the formulation? No, it does not mean that. You MIGHT be able to remove it from the formulation, but Scheffe models are NOT cause-and-effect. So, use this with caution. Finally, this script will add columns to an existing Parameter Estimates table (even if that report is closed). You will see testing on all of the model effects, the report is only valid for the main effects.</P>
<P> </P>
<PRE><CODE class=" language-jsl">Names Default to Here( 1 );
// find current Fit Least Squares platform
window = Get Window List();
n windows = N Items( window );
For( w = 1, w <= n windows, w++,
window ref = Get Window( w );
title = window ref << Get Window Title;
If( Contains( title, "Fit Least Squares" ),
Break();
);
);
// obtain values for proper t-test of estimates
mean = window ref["Summary of Fit"][NumberColBox(1)] << Get( 4 );
err df = window ref["Analysis of Variance"][NumberColBox(1)] << Get( 2 );
estimate = window ref["Parameter Estimates"][NumberColBox(1)] << Get As Matrix;
std err = window ref["Parameter Estimates"][NumberColBox(2)] << Get As Matrix;
// compute t-ratios
t ratio = (estimate - mean) :/ std err;
// compute p-value for each t-ratio
p val = 2 * (1 - Students t Distribution( Abs( t ratio ), err df ));
// add proper test to report
window ref["Parameter Estimates"][TableBox(1)]
<< Append( Number Col Box( "Proper t Ratio", t ratio ) )
<< Append( Number Col Box( "Proper Prob>|t|", p val ) );
window ref["Parameter Estimates"][NumberColBox("Proper Prob>|t|")] << Set Format( 8, 97) << Set Conditional Format( "PValue" );
window ref["Parameter Estimates"] << Append(
Text Box( "Proper tests valid for linear terms only!" );
);
</CODE></PRE>Thu, 20 Feb 2020 19:18:36 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/248553#M48773Dan_Obermiller2020-02-20T19:18:36ZRe: ERROR: 0.05 significant Curve shaded error plot even though my model is significant with high R2 value
https://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/248564#M48778
Thanks Dan for your detail answer.<BR />I am not looking to remove any component from the formulation. We actually have 4 different responses for this mixture DOE and 4 models are constructed. I am hoping to identify the parameter that truly matters in each model and ultimately combine the prediction equations from the 4 models to maximize certain desirability.Thu, 20 Feb 2020 20:31:24 GMThttps://community.jmp.com/t5/Discussions/ERROR-0-05-significant-Curve-shaded-error-plot-even-though-my/m-p/248564#M48778nguyendv06102020-02-20T20:31:24Z