I'm trying to do permutation tests and Monte Carlo Cross Validations with the Iris sample dataset as a MWE for our dataset. I'm not sure if I'm doing things and interpreting output correctly.
1) Create New Formula Column (Random->Sample Without Replacement) for the Species Column for Permutation Test.
2) Create a Validation Column (0.75/0.25 split) and a new Formula Column (Random->Sample Without Replacement) using this for Monte Carlo Cross Validation
3) Run the Discriminant Platform with the Validation column and display the ROC curves. Use the Simulate Platform on the Area of the ROC Curve
4.1) For the Permutation Test-Select Species as column to switch out and Shuffle[Species] as column to Switch In. Enter the desired number of random sampling and random seed and run the simulation.
4.2) View the Distributions script and take note of the empirical p-value.
5.1) For the Monte Carlo Cross Validation-Select Validation as column to switch out and Shuffle[Validation] as column to Switch In. Enter the desired number of random sampling and random seed and run the simulation.
5.2) View the Distributions script and take note of the empirical p-value.
6) Assuming what I did is correct (testing the Area of ROC curve), the low p-value from the Permutation test is expected, but what about the large p-value for the Monte Carlo cross validation? How should this be interpreted?
Thanks