Much has been written and discussed about how hypothesis testing is controversial / confusing / slanted toward "Accepting the null". Practitioners find the approach very confusing to implement well and interpret. Nathaniel Stevens introduced the idea of Probability of Agreement (PoA) as an alternative to this prickly problem. The method is flexible for many different situations, has different options to look at one- and two-sided agreement, and does not start with a predisposed idea of whether the items being compared should or should not be considered equal. This provides a balance between Type I and Type II errors in a way that P-values often fail to do. The method adjusts automatically to different sample sizes to reflect the added uncertainty associated with smaller samples where traditional hypothesis testing fails to alert the user to a lack of power.
This would be an excellent addition to JMP as. it provides a more flexible, more easily interpreted version of how to compare populations in many different arenas. It matches the general approach fostered in JMP of encouraging exploration and flexible assessment of patterns in data.
I think the philosophy of PoA will resonate with practitioners and provide them with tools for making comparisons in a way that feels intuitive and easy to interpret.
References:
Stevens N.T., Steiner S.H. and MacKay R.J. (2017). Assessing agreement between two measurement systems: An alternative to the limits of agreement approach. Statistical Methods in Medical Research, 26(6): 2487–2504.
Stevens N.T., Rigdon S.E. and Anderson-Cook C.M. (2018). Bayesian probability of predictive agreement for comparing the outcome of two separate regressions. Quality and Reliability Engineering International, 34(6): 968–978.
Stevens N.T. and Anderson-Cook C.M. (2019). Design and analysis of confirmation experiments. Journal of Quality Technology 51(2): 109–124.
Stevens N.T. and Lu L. (2020). Comparing Kaplan-Meier curves with the probability of agreement. Statistics in Medicine 39(30): 4621–4635.