topic Re: JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets? in Discussions

JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

Thierry_S — Fri, 09 Jun 2023 16:00:02 GMT

Hi JMP Community,

I need to assess the correlation between ~10,500 biomarkers measured in 150 patients at baseline. I tried a simple JSL script based on the fit Y by X and the Multivariate Correlation platform. Both approaches take a long time to compute on my machine (Windows 10 Pro, Intel i-7 2.50 GHz, 32 GB RAM, JMP 16). Is there a faster approach to generate and retrieve Pearson's r and associated p Values? Of note, while the matrix-based approach seems attractive, it does not appear to return the p-values.

I appreciate your help.

Best,

Re: JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

Chris_Kirchberg — Mon, 10 Oct 2022 19:50:48 GMT

@Thierry_S ,

How long does it take now? Have you compared it to Response Screening? Put all Biomarkers in Y and X and use the Corr (Pearsons Product Moment Correlation) advanced option?

Re: JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

Thierry_S — Mon, 10 Oct 2022 19:52:33 GMT

Hi Chris,

I did not think of that approach. Let me try ASAP.

Thanks,

Re: JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

Thierry_S — Mon, 10 Oct 2022 19:56:03 GMT

Hi @Chris_Kirchberg ,

Beautiful. The processing time has been cut by 5 to 10 folds.

Best,

Re: JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

Chris_Kirchberg — Mon, 10 Oct 2022 20:12:23 GMT

@Thierry_S ,

I have a data set that is 18856 mRNA relative abundances and 65 samples. Response Screening in JMP Pro 16 seems to take 120 GB of memory ( I only have 64 so caching is occurring). Might not be a good idea now that I mentioned it.

UPDATE: I cut that down to about 9000 and it took less than a minute. So there is definitely a threshold on the number of biomarkers you will be able to use.

Re: JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

Chris_Kirchberg — Mon, 10 Oct 2022 19:58:16 GMT

YEAH! I had the p-value table on launch option turned on so maybe that is my cause of so much memory use.