topic Re: Using XGBoost Model to Predict on a hold-out test set in Discussions

Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Fri, 09 Jun 2023 00:36:04 GMT

I am using JMP 16. I installed the XGBoost addin. I developed a xgbbost model using this. I saved the model in the data table using 'Save Prediction formulae'. Now, I have a separate hold-out test set on which I want to run the formulae to get the prediction. If I click the formulae (see attached figure), I found that the formulae depend on the training and validation set, which seems odd to me. This also hinders me to run the model for the hold-out test set.

Re: Using XGBoost Model to Predict on a hold-out test set

statman — Tue, 06 Jul 2021 15:59:33 GMT

I'm not sure I know the answer to your question, but it appears you do not have any variables in your model (the column names are validation and training and shrinkage predator validation)? So the formula is a function of those column names. Either re-write the model in terms of variables or re-name the columns of your hold out data set?

Re: Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Tue, 06 Jul 2021 17:20:41 GMT

I added the jmp file. If you look into the XGBoost model, you can see the details. I have 17 variables.

Re: Using XGBoost Model to Predict on a hold-out test set

statman — Tue, 06 Jul 2021 19:55:00 GMT

Your data set is the same as the one used in the help menu for the XGBoost add-in. Did you follow the steps shown in the application of the platform?

Re: Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Tue, 06 Jul 2021 22:54:21 GMT

Yes. I did. I was not able to still get the formulae.

Re: Using XGBoost Model to Predict on a hold-out test set

statman — Wed, 07 Jul 2021 19:22:42 GMT

Sorry I haven't had time to look at your data. I'm also not very experienced with the platform, but my guess is you should use the platform to find the significant factors and then re-write your model in fit model and run it and save the prediction formula...but again, I'm not experienced here. Perhaps there is an easier way to get the actual model in terms of the variables in the data table...

Re: Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Thu, 08 Jul 2021 14:47:54 GMT

Can you show me over a webex call?

Re: Using XGBoost Model to Predict on a hold-out test set

statman — Fri, 09 Jul 2021 11:13:23 GMT

I suggest getting in touch with Russ:

Re: Using XGBoost Model to Predict on a hold-out test set

russ_wolfinger — Fri, 09 Jul 2021 11:43:21 GMT

Hi @sukrit2020 ,

A recommended approach is to append your separate hold-out test set to the main data as new rows with the Y target values set to missing. Then the formula will automatically create predictions on those rows.

Also, it's important to note that XGBoost handles validation differently than other JMP platforms. If the validation column is Nominal, it will automatically do full k-fold with each of the levels. This is why the formula looks like it does. This can be confusing if your validation column has two levels, e.g. "Validation" and "Training". In this case XGBoost would actually do 2-fold, holding out each subset regardless of their values.

A better way to set up your folds is to use the Make K-Fold Columns utility, and then use those as your Validation columns. I feel repeated k-fold is a much better way to validate your model than using a single holdout set. If you really want to only do a single holdout, you must set the type of the Validation column to Continuous, with value 0 corresponding to training and 1 to validation.

Re: Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Sat, 10 Jul 2021 16:33:12 GMT

Hi Russ,

Can we set up a webex where you can show me the procedure? I am little new to JMP and thus could not fully follow your instructions.

thanks,

sukrit

Re: Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Thu, 15 Jul 2021 12:46:49 GMT

Hi Russ,

I have not heard back from you regarding the previous message. PLease let me know whether you are amenable to a short phone call or not.

thanks,

sukrit

Re: Using XGBoost Model to Predict on a hold-out test set

lwx228 — Fri, 23 Jul 2021 09:44:53 GMT

Re: Using XGBoost Model to Predict on a hold-out test set

sukrit2020 — Tue, 27 Jul 2021 20:51:33 GMT

Can you show me how you got this? I could not get this.