Hi Victor_G,

Thanks for your help. Either Model Screening platform with hyperparameters tuning function or all individual model platforms with K-fold function can easily solve my problem. It is a pity that the two platforms are lacking some functions.

After some thought, I'd also like to ask if the following steps are appropriate:

(1) stratified sampling with 80% training and 20% test set.

(2) In training set (80%), I trained multiple machine learning algorithms with K-fold cross-validation in Model Screening platform. Of course, this step uses the default hyperparameters. After comparing, I found part of the algorithm to be optimal, such as Bootstrap Forest.

(3) Then, I clarified the optimal hyperparameters of Bootstrap Forest through the Tuning Design Table.

(4) In training set (80%), I re-fitted the Bootstrap Forest algorithm with optimal hyperparameters. Of course, I can also fit other algorithms.

(5) In test set (20%), I further assessed model performance of Bootstrap Forest and other algorithms.

Are the above steps appropriate in JMP？

In step 2, can I just report the results of model comparisons (results of K-fold cross-validation) with default hyperparameters in the article? Of course, we know that many articles start with step 3 and then step 2, so they report the results of model comparisons with optimal hyperparameters.

In step 4, I didn't set up any cross-validation method, is this correct? Or is there another way?

Looking forward to hearing from you.