topic Data mining with Small samples...Regression with Small Samples? in Discussions

Data mining with Small samples...Regression with Small Samples?

triunk — Thu, 09 May 2013 19:00:48 GMT

Dear all JMP experts, I know that this kind of discussion might be both a silly...very silly and a hot issue simultaneously...but I thought it would be nice to raise it : Which types of linear AND non-linear regression does JMP offer when you have a small sample size (e.g. 50-70 records) and you want to predict a continuous outcome from 5-7 predictors (both categorical and continuous) Can k-fold cross-validated stepwise linear regression OR Partial Least Regression is a remedy to this problem? Are boosted trees OR Neural networks just insane even to think about them? Your responses are GREATLY WELCOMED and VERY MUCH APPRECIATED. Respectfully, Chris

Re: Data mining with Small samples...Regression with Small Samples?

paigemiller — Thu, 09 May 2013 20:15:13 GMT

Whether these procedures will work or not at those sample sizes depends on signal-to-noise. If you have lots of signal and low noise, you probably won't even need 50 records. Other way around, low signal and high noise, and you're probably wasting your time.

Re: Data mining with Small samples...Regression with Small Samples?

pgstats — Thu, 09 May 2013 20:21:22 GMT

The first thing I do with such skimpy datasets is a regression tree analysis. It is a quick and intuitive way to explore the relationship between your dependent and predictor variables. The first three or four nodes are generally meaningful. - PG

Re: Data mining with Small samples...Regression with Small Samples?

triunk — Thu, 09 May 2013 22:50:34 GMT

First of all thank you very much for your prompt reply. I really appreciate it

PGStats do you mean Classification and Regression Tree analysis? CART?

Alternatively do you think that multivariate adaptive regression splines could work also for small sample size?

Many thanks again!

Re: Data mining with Small samples...Regression with Small Samples?

triunk — Thu, 09 May 2013 22:52:14 GMT

P.S. I know that MARS is not offered in JMP but it is offered from SAS...so scipt might help.

Re: Data mining with Small samples...Regression with Small Samples?

triunk — Thu, 09 May 2013 23:12:13 GMT

Hi PaigeMiller

Would you please be so kind and elaborate a little bit more about signal and noise? (just one sentence)

Please accept my apologies but for me these two terms are pretty allegorical to my silly mind.

THANK YOU!

Chris

Re: Data mining with Small samples...Regression with Small Samples?

pgstats — Fri, 10 May 2013 01:29:32 GMT

Yes I meant CART-type analysis, called Partition in JMP. When dependent variable is continuous, the result is a regression tree, otherwise, it's a classification (or decision) tree. I think regression splines eat up too many degrees of freedom to be applicable to your size of dataset. - PG

Re: Data mining with Small samples...Regression with Small Samples?

triunk — Fri, 10 May 2013 01:54:21 GMT

Thanks PG, this helps. I will check it out.

Re: Data mining with Small samples...Regression with Small Samples?

paigemiller — Fri, 10 May 2013 16:02:38 GMT

Would you please be so kind and elaborate a little bit more about signal and noise? (justone sentence)

Please accept my apologies but for me these two terms are pretty allegorical to my silly mind.

Most statistical modeling attempts to determine if there is a signal (a real relationship) that is larger than the variability of the errors (noise). Each modeling technique that I am aware of gives a measure of this signal-to-noise; for example, in standard regression is would be the overall F-test.

There's no reason you can't have a very large signal and very low noise in 50 data points. It depends on the data.

Re: Data mining with Small samples...Regression with Small Samples?

triunk — Fri, 10 May 2013 19:52:32 GMT

That's much clearer. Thank you VERY VERY much

Kind Regards

Re: Data mining with Small samples...Regression with Small Samples?

martindemel — Mon, 12 Oct 2020 16:31:57 GMT

Though it is quite long ago this question got ansked I want to chime in here as I think the discussion does not talk about some specifics which would be important to decide if you will be able to get some meaningful information.

1. Is the data from a DOE? Then 50-70 observations will be usually way more as sufficient to have enough degrees of freedom to detect the underlying behaviour.

2. If from a DOE what kind of. It is important if the DOE can estimate the model effects you want to take a look in your regression or other models. If it is only a main effect model the Design can estimate you will have again uncertainty if the created model will tell you the truth

3. If these are just observations, exploratory data analysis can help you to identify potential patterns, using then different modeling strategies liek previously mentioned and compare these will help you to see if you can trust the models (are these telling the same story or completely different ones?)

If the latter you may want to get a better understanding from a DOE.

4. Neural nets are oftaen said are not applicable with such few data ponits. Newer research shows that with certain strategies you will be able to gain still acceptable results. However, you never should trust one model alone. Test out different ones. Check how the model behaves when you simulate small variation of the actual settings, does the simulation results stay in the confidence limits or does it break out heavily.

Before you do that, you should check the quality of the data, are there outliers, missing values, are the columns correlated due to some previously known relationships like A is k*B or similar, are there many duplicates. Are the observations measurements of the same thing over time, are they repeated measures, ... All this will tell you what you can or what you cannot do with your data.

So, what I want to say, is you can create models for all, but if they are useful depends on your question, where the data comes from, and how it has been measured, and not necessarily on how much observations you have.