Hi Everyone!  Thank you to those of you in attendance live today for our Mastering Session on Basic Statistical Analysis, taught by the excellent @MarilynWheatley and moderated by my colleagues @gail_massari and @Jeff_Upton 

Here is the article that I shared in today's Mastering Webinar during our Q&A Session: Moving to a World Beyond “p < 0.05”

It's basically a call to move away from automatic, threshold-based statistical decision making and towards a more transparent, and context-aware, uncertainty-embracing statistical reasoning framework.

The idea here is not to admonish the use of p-values with strict cut-offs for significance, but to stop letting them stand in for real scientific judgment.  Among other ideas and suggestions for thoughtful analysis, the use of p-values is suggested if at all as a descriptive, rather than a gate-keeping mechanism. 

Other directions include the suggestion to embrace uncertainty in statistical decision-making rather than trying to eliminate it. 

Statistical inference is not equivalent to scientific inference, and limitations should be acknowledged in both with careful analysis and a thoughtful risk-based framework.

To summarize the question from the audience today: "What is a good cut-off for the p-value?"

The p<0.05 threshold is a holdover with historical underpinnings, and while this article suggests that small differences in p-values (e.g., 0.049 vs 0.051) do not justify categorical differences in interpretation, I recognize that there are many contexts (business and regulatory environments) where a cut-off decision is required for operational consistency and for careful and 'honest' analysis based on committed acceptance criteria. 

Above all, a rigorous way to establish the p-value threshold is by establishing a so-called "standard of evidence" (alpha) before we run our study.  Alpha allows us to specify in advance how often we are tolerating a false-alarm (where a false-alarm is  a false-rejection of the null hypothesis, that is, a situation where we detected something that isn't actually a real signal).  

The p-value indicates how likely we are to observe the result that we got under the null hypothesis. Our cut-off, alpha (often historically by convention equal to 0.05) is what we compare it to. If p≥alpha, we do not reject the null, that is, we retain the null as a plausible explanation.  If p<alpha, then we reject the null, that is, we assert the alternate.

But critically, the p-value tells us how likely we are to obtain our result, under the null hypothesis.

As always we are here to assist you with questions you may have about the use of JMP Software. 

-JMP Technical Support (support@jmp.com)