JMP > Multivariate Analysis > Correlation> How to Handle Large Datasets?

[#1]

Hi JMP Community,

I need to assess the correlation between ~10,500 biomarkers measured in 150 patients at baseline. I tried a simple JSL script based on the fit Y by X and the Multivariate Correlation platform. Both approaches take a long time to compute on my machine (Windows 10 Pro, Intel i-7 2.50 GHz, 32 GB RAM, JMP 16). Is there a faster approach to generate and retrieve Pearson's r and associated p Values? Of note, while the matrix-based approach seems attractive, it does not appear to return the p-values. 

I appreciate your help.

Best,

TS

[#2 · Accepted solution]

@Thierry_S ,

How long does it take now?  Have you compared it to Response Screening? Put all Biomarkers in Y and X and use the Corr (Pearsons Product Moment Correlation) advanced option?

[#3]

Hi Chris,

I did not think of that approach. Let me try ASAP.

Thanks,

TS

[#4]

Hi @Chris_Kirchberg ,

Beautiful. The processing time has been cut by 5 to 10 folds.

Best,

TS

[#5]

YEAH! I had the p-value table on launch option turned on so maybe that is my cause of so much memory use.

[#6]

@Thierry_S ,

I have a data set that is 18856 mRNA relative abundances and 65 samples. Response Screening in JMP Pro 16 seems to take 120 GB of memory ( I only have 64 so caching is occurring).  Might not be a good idea now that I mentioned it.

UPDATE:  I cut that down to about 9000 and it took less than a minute. So there is definitely a threshold on the number of biomarkers you will be able to use.