I agree with @Craige_Hales and @statman. You can do much better than A/B testing. We use custom design and choice design in JMP as the basis for extensive and quantitative usability studies of JMP features and user experiences. I know that A/B testing is the gold standard in practice today, but better methods are available to follow leaders such as Ben Schneiderman
It sounds like you might already have an A/B test in hand and want to analyze it with JMP. What types of results are you looking for? JMP can provide statistical and graphical analyses to understand your data. Let us know what you want to do and we will help you along the way.