Hi Victor_G,
Thanks for your help. Either Model Screening platform with hyperparameters tuning function or all individual model platforms with K-fold function can easily solve my problem. It is a pity that the two platforms are lacking some functions.
After some thought, I'd also like to ask if the following steps are appropriate:
(1) stratified sampling with 80% training and 20% test set.
(2) In training set (80%), I trained multiple machine learning algorithms with K-fold cross-validation in Model Screening platform. Of course, this step uses the default hyperparameters. After comparing, I found part of the algorithm to be optimal, such as Bootstrap Forest.
(3) Then, I clarified the optimal hyperparameters of Bootstrap Forest through the Tuning Design Table.
(4) In training set (80%), I re-fitted the Bootstrap Forest algorithm with optimal hyperparameters. Of course, I can also fit other algorithms.
(5) In test set (20%), I further assessed model performance of Bootstrap Forest and other algorithms.
Are the above steps appropriate in JMP?
In step 2, can I just report the results of model comparisons (results of K-fold cross-validation) with default hyperparameters in the article? Of course, we know that many articles start with step 3 and then step 2, so they report the results of model comparisons with optimal hyperparameters.
In step 4, I didn't set up any cross-validation method, is this correct? Or is there another way?
Looking forward to hearing from you.