topic Re: LDA cross-validation in Discussions

LDA cross-validation

Adias — Fri, 17 Mar 2017 18:17:20 GMT

Hello,

I'm a new user of JMP (JMP 13.1.0). I have a question please: Is it possible to perform an linear discriminant analysis with cross-validation (leave-one-out method) in this version of the software?

Thank you,

Adias

Re: LDA cross-validation

jiancao — Fri, 07 Apr 2017 20:01:37 GMT

Leave-one-out cross validation is not available in LDA, but it is in JMP Pro's GenReg. If your outcome is binary and you have JMP Pro, consider running a logistic regression from GenReg, which is considered as an alternative approach to LDA and is often preferred. http://www.jstor.org/stable/2286261?seq=1#page_scan_tab_contents

Re: LDA cross-validation

Bill_Worley — Mon, 10 Apr 2017 12:11:54 GMT

Hello,

You will need JMP Pro 13 to run Discriminant Analysis and Leave-one-out cross-validation is an option there as well. You would run this through the Fit Model platform.

HTH

Re: LDA cross-validation

AN1 — Sat, 10 Feb 2018 16:44:45 GMT

Hello: There are other great features in the discriminant platform like regularized discriminant analysis, canonical plots, interpretation of dimensions, etc. If I want to stay in the discriminant platform and perform leave-one-out validation what are the choices?

Re: LDA cross-validation

susan_walsh1 — Fri, 03 Aug 2018 15:51:40 GMT

Within the Discriminant platform there is not a way to get leave-one-out validation. Within the Discriminant platform in JMP 14, there is an option "Cross Validate by Excluded Rows". In JMP Pro, you can create a Validation column with no more than 3 values to be used for Training, Validation, and Test sets.

Re: LDA cross-validation

mkachl01 — Tue, 26 May 2020 16:30:19 GMT

Dear all,

I'm trying to validate my LDA analysis, but I can't seem to find the "Cross Validate by Excluded Rows" option in my JMP15 (not PRO). Could you please help me to navigate and find that function?

I have already prepared a Validation Column and Excluded appropriate rows.

Thanks

Re: LDA cross-validation

Mark_Bailey — Wed, 27 May 2020 14:48:09 GMT

The Discriminant platform in JMP Pro provides the Validation role to which you can assign the validation data column that you created.

(Please do not start another discussion when you do not get an answer right away.)

Re: LDA cross-validation

mkachl01 — Wed, 27 May 2020 17:01:12 GMT

Dear Mark,

Thank you so much for responding so quickly and sorry for a naive question - but where do I find this 'Validation role' in the LDA platform? I'm quite new to JMP, and not everything is clear to me.

The only thing I found is the option on the main tab where I select variables for LDA. There I can tick the box 'Validate by excluded rows' which selects Train/Test data based on the validation column I've created. But this offers only a one-off split based on this column, not any iterative validation options (like leave-one-out or k-folds or Monte Carlo I'd like to run). But again, maybe I'm missing something very obvious here?

I didn't want to duplicate the posts, sorry about that! I wasn't sure if anyone will answer to my response to a very old post!

Re: LDA cross-validation

Mark_Bailey — Wed, 27 May 2020 17:03:21 GMT

Select Analyze > Multivariate Methods > Discriminant. You will see a button to assign the column to the Validate role.

Re: LDA cross-validation

mkachl01 — Wed, 27 May 2020 20:24:33 GMT

Thank you! I don't know how I missed that!

But, apart from being easier to use, doesn't it have the same effect as the function "Validate by excluding rows" mentioned earlier? Is that right?

Maybe I'm doing something wrong, but it looks like I can use only one validation column at the time and 'Validation role' doesn't give me any option to select the validation method or generate multiple validation columns without repetition in the test set (i.e., so that the variable selected as Test in the first column will not be selected anymore in the following validation columns). Is it possible to do that in JMP?

Once again, sorry for posting such naive questions, but I got totally lost in this!

Re: LDA cross-validation

Mark_Bailey — Thu, 28 May 2020 11:43:38 GMT

It is similar to excluding rows, but you can also specify a hold out set for testing the selected model.

You can create multiple data columns for the validation role, but you can only use one at a time. The idea is that once you decide on the size of the hold out sets, any of them are equally useful. Also, you can use the same validation column in more than one modeling platform for a fair and valid comparison of models. Why would you need more than one validation column? How would you use multiple validation columns?

I do not know what you mean by "option to select the validation method." There is only one cross-validation method with hold out sets represented by the Validation analysis role.

Re: LDA cross-validation

mkachl01 — Thu, 28 May 2020 15:47:32 GMT

I see, thanks a lot for explaining this!

By 'validation options' I meant different methods of validating the models like k-fold or Monte Carlo cross-validation explained here: https://www.statisticshowto.com/cross-validation-statistics/

As far as I understand, to run 5-fold cross-validation, I'd need to have five validation columns. Each column would represent the 80-20 split, but where the 20% dedicated for the test in each of the 5 folds is different every time. Is there a way I can do this in JMP?

Re: LDA cross-validation

Mark_Bailey — Thu, 28 May 2020 16:45:11 GMT

JMP provides K-fold cross-validation in some platforms but not all of them. You would have to implement your own version of it. Here is one way to do it:

Create a new data column called Fold that uses the Nominal modeling type and with a formula of Random Integer( 1, 5 ). This column will identify the five folds for you.
Create a series of five more data columns called Validation i, also using the Nominal modeling type, and each with a formula of Fold == i, where i changes from 1 to 5 as you go from the first to the last of these new columns.
Launch the Discriminant platform five times using the succession of Validation i columns in the Validation analysis role.

You should have new columns for 5-fold cross-validation in your data table like this.

This first iteration of your model fitting might look like this:

Use the cross-validation information for each of the folds, such as:

You can then combine the five sets of results into the overall training and validation results as you see fit such as described in your cited reference.

Re: LDA cross-validation

mkachl01 — Mon, 01 Jun 2020 08:54:19 GMT

That's exactly what I was looking for, thank you so much for this!