Hi @ivanpicchi,
I am confused and may not directly answer your questions, as I'm not sure what is/are your objective(s) in this plan : are H/W and FN MED the two responses ?
Also, what is your objective :
- Explaining your system based on your factors ?
- Screen important factors ?
- Create a predictive model ?
What is your knowledge about the system ? and many other questions ... Here are some first thoughts, but there is much more to discuss (but this may take too long to write on a forum).
Using Neural Network for a model-based DoE seems completely off for me, as you only have 3 levels for each of your factor, and you use a highly flexible model that can approximate any complex functions... Seems like using a bazooka to kill a fly : overly complex and unsuitable approach. So one or several NN are completely premature at this stage. There are other (more simple) algorithms that you could try : Decision Tree/Random Forests, Support Vector Machines, ...
Also the validation column is not set properly for Machine Learning, as the experiments (through replicates) can be both in training, validation and/or test sets. This results in overfitting of models, and overconfidence in the prediction results. For the validation of predictive model on small-size dataset, you have to keep the same treatment in the same set to avoid data leakage, and overfitting in your model. Cross-validation approach, like K-folds and Leave-One-Out, might be interesting to consider (with the same precaution with the use of replicates).
Anyway, a general advice is to visualize your data before modeling, and use Occam razor principle when modeling : Start your modeling with a simple model, and then iterate and add complexity (if needed !). Compare your new model with the previous one, and assess how much improvement you gained (and how much the complexity has increased). I would recommend to start simple, use regression models first and evaluate properly the pro's and con's from your models.
I don't know which kind of regression model and the terms you included, but the response H/W can be quite good modelized through Fit Least Squares model (see script "Fit Group"). For the response FN, it seems you have a lot of noise in the measurements, so maybe a further work on the ranges, experimental protocol, factors (control nuisance factors, and/or use blocking, ...), measurement device, and maybe on the a priori model may be considered to improve results.
Fitting a complicated model on this response won't help: there are no big differences between the results of your best NN (R² = 0,772 - RASE = 0,174 (with overfitting...)) and a simple regression model (R² = 0,758 - R² adjusted = 0,715 - RMSE = 0,586).
Take also in consideration that the modeling approach, metric(s) and design used should be defined accordingly. For example, in a predictive objective focusing on the inside of the experimental space, a Space-Filling design is suitable to use Machine Learning algorithms, with possible metrics like RMSE/RASE/MSE focussing on the accuracy of the predictions.
I hope these first discussion points will help you,
Victor GUILLER
L'Oréal Data & Analytics
"It is not unusual for a well-designed experiment to analyze itself" (Box, Hunter and Hunter)