If your case is 2x2, you can use McNemar's test in JMP. Select Analyze Fit Y by X, assign your X and Y colums, click OK, and the click the red triangle and select Exact Agreement.
One other thought for you is did you look in the JMP User Community File Exchange Add Ins and Script libraries? Lots of people place Add Ins or scripts in the File Exchange when they are willing to share something that JMP doesn't do 'out of the box.'
